You can send MediationZone log files to an external log collector by configuring the properties described in this section. All log files generated in MediationZone will be forwarded when log forwarding is enabled, including System, Platform, and EC logs.
The log structure, fields, and severity follow the JSON and RFC5424 specifications and it is a prerequisite to know one of these specifications to use the log forwarding functionality.
The following properties should be added to the platform configuration file to forward the logs to the external log collector:
Properties | Description | Default Value |
|---|---|---|
| This property is used to enable or disable the logging. | false |
| This property defines the protocol to use, either UDP or TCP. | UDP |
| This property defines the hostname or IP address of the target log collector. | localhost |
| This property defines the port of the target log collector. | 514 |
| This property defines the layout type based on the supported specifications. You can select either RFC5424 or JSON. | RFC5424 |
| This property is used to include a full stack trace in your log messages. | false |
| This property is used to generate a debug log file, useful when you need to determine any issues when configuring the log forwarding. | false |
| This property defines the name of the debug log file. | syslog-debug.log |
| This property defines the location of the debug log file. | /tmp/syslog |
| This property defines the number of debug log file to be generated. | 5 |
Platform Configuration
To enable syslog logging to connect to external log collector.
mzsh topo set topo://container:main1/pico:platform/val:config.properties.mz.syslog.logging.enable true
To set external log collector connection port. For instance, port 5140 is connection port for Fluent-bit
mzsh topo set topo://container:main1/pico:platform/val:config.properties.mz.syslog.port 5140
The rest of the properties that are not set explicitly will follow their default value in the tables.
Changes made to the properties above only take effect after platform restarted.
mzsh restart platform
Log Forwarding With Fluent-bit:
Fluent-bit is one of the external log collector that is widely used by major cloud providers, banks, and companies world-wide. Fluent-bit can read from local files and network devices, and can scrape metrics in the Prometheus format from your server. Furthermore, Fluent Bit supports a vendor-neutral approach, seamlessly integrating with other ecosystems such as Prometheus and OpenTelemetry.
Click here to install Fluent-bit for your environment if you have not.
Fluent-bit Configuration File
Fluent-bit configuration file allows user to configure what kind of data source (input) to read from and where to forward the data to, i.e., data destination (output).
Edit Fluent-bit configuration file with the following contents.
[SERVICE]
Flush 1
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name syslog
Mode udp
Buffer_Chunk_Size 32000
Buffer_Max_Size 64000
Receive_Buffer_Size 512000
[OUTPUT]
Name stdout
Match *
[OUTPUT]
Name file
Match *
Path <your-directory-path>
[OUTPUT]
name syslog
match *
host <your-host-name-or-ip-address>
port 514
mode udp
syslog_format rfc5424
syslog_maxsize 2048
syslog_severity_key severity
syslog_facility_key facility
syslog_hostname_key host
syslog_appname_key appname
syslog_procid_key procid
syslog_msgid_key msgid
syslog_sd_key sd
syslog_message_key message
then startup Fluent-bit with configuration file manually
fluent-bit -c /opt/homebrew/etc/fluent-bit/fluent-bit.conf
on Linux environment with Systemd, startup Fluent-bit with systemctl
sudo systemctl start fluent-bit
then do a status check, you should see a similar output like this:
$ systemctl status fluent-bit
● fluent-bit.service - Fluent Bit
Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2016-07-07 02:08:01 BST; 9s ago
Main PID: 3820 (fluent-bit)
CGroup: /system.slice/fluent-bit.service
└─3820 /opt/fluent-bit/bin/fluent-bit -c etc/fluent-bit/fluent-bit.conf
...
Fluent-bit started up manually with these output on the screen
Fluent Bit v3.1.9 * Copyright (C) 2015-2024 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io ______ _ _ ______ _ _ _____ __ | ___| | | | | ___ (_) | |____ |/ | | |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __ / /`| | | _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / \ \ | | | | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /.___/ /_| |_ \_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ \____(_)___/ [2024/11/05 18:31:45] [ info] [fluent bit] version=3.1.9, commit=, pid=17946 [2024/11/05 18:31:45] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128 [2024/11/05 18:31:45] [ info] [cmetrics] version=0.9.6 [2024/11/05 18:31:45] [ info] [ctraces ] version=0.5.6 [2024/11/05 18:31:45] [ info] [input:syslog:syslog.0] initializing [2024/11/05 18:31:45] [ info] [input:syslog:syslog.0] storage_strategy='memory' (memory only) [2024/11/05 18:31:45] [ info] [in_syslog] UDP server binding 0.0.0.0:5140 [2024/11/05 18:31:45] [ info] [output:syslog:syslog.2] setup done for 192.168.64.2:514 (TLS=off) [2024/11/05 18:31:45] [ info] [output:stdout:stdout.0] worker #0 started [2024/11/05 18:31:45] [ info] [output:file:file.1] worker #0 started [2024/11/05 18:31:45] [ info] [sp] stream processor started
Fluent-bit Outputs
Fluent-bit can be configured to route collected data to multiples destinations. The fluent-bit configuration file listed in previous section routes collected data to standard output, local file and remote rsyslog server.
Refer to official documentation to know more about output plugins supported by Fluent-bit.
Standard Output
[0] syslog.0: [[1730863047.926000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:17:27.926+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"System", "extradata"=>"[2@32473 UUID="3ff217ee-d18c-45ca-9a3b-7907923e9e26" host="localhost:6790" picoName="platform" severity="Information" type="System"]", "message"=>"The pico instance platform (platform) at host localhost has connected."}]
[1] syslog.0: [[1730863047.927000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:17:27.927+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"System", "extradata"=>"[2@32473 UUID="e3ac6725-23cf-4e56-b483-803e3dfaaf90" host="localhost:6790" severity="Information" type="System"]", "message"=>"System message: The system started"}]
[0] syslog.0: [[1730863057.633000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:17:37.633+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"System", "extradata"=>"[2@32473 UUID="2b4d0683-3a6d-442e-9d45-c5089159d6e0" host="localhost:6790" picoName="platform" severity="Information" type="System"]", "message"=>"The pico instance ec1 (ec) at host localhost has connected."}]
[0] syslog.0: [[1730863068.462000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:17:48.462+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"System", "extradata"=>"[2@32473 UUID="819b616b-504a-42fc-a271-907fa9e801b6" host="localhost:6790" picoName="platform" severity="Information" type="System"]", "message"=>"The pico instance ec2 (ec) at host localhost has connected."}]
[0] syslog.0: [[1730863179.066000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:19:39.066+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"System", "extradata"=>"[2@32473 UUID="9b3191a2-d90b-441e-9602-8a2d99371301" host="localhost:6790" picoName="platform" severity="Information" type="System"]", "message"=>"The pico instance desktop-0 (desktop) at host 192.168.100.3 has connected."}]
[1] syslog.0: [[1730863179.085000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:19:39.085+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"User", "extradata"=>"[3@32473 UUID="3ebb0e90-a3db-40b3-b11a-f7cd23fb6ff1" host="192.168.100.3:52018" severity="Information" type="User" user="mzadmin"]", "message"=>"mzadmin, Desktop user logged in"}]
[0] syslog.0: [[1730863752.812000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:29:12.812+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"User", "extradata"=>"[3@32473 UUID="6e360bcb-fee7-433e-a964-cb3e547786fe" host="diameter-client.digitalroute.com:52018" severity="Information" type="User" user="mzadmin" workflow="perf_http2_client_server.wf_http2_server_perf.workflow_1"]", "message"=>"mzadmin, Debug has been enabled on workflow perf_http2_client_server.wf_http2_server_perf.workflow_1."}]
[1] syslog.0: [[1730863752.822000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:29:12.822+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"System", "extradata"=>"[2@32473 UUID="c19103ec-bbe9-48b2-9c79-7f0f03de9727" host="localhost:6790" severity="Information" type="System"]", "message"=>"System message: Workflow perf_http2_client_server.wf_http2_server_perf.workflow_1 is being manually started by user: mzadmin"}]
[2] syslog.0: [[1730863752.823000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:29:12.823+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"User", "extradata"=>"[3@32473 UUID="955166fe-87ac-4ced-8a51-c17888431bd3" host="diameter-client.digitalroute.com:52018" severity="Information" type="User" user="mzadmin" workflow="perf_http2_client_server.wf_http2_server_perf.workflow_1"]", "message"=>"mzadmin, Workflow perf_http2_client_server.wf_http2_server_perf.workflow_1 (1) started."}]
Local File
syslog.0: [1730863047.926000000, {"pri":"14","time":"2024-11-06T11:17:27.926+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"System","extradata":"[2@32473 UUID=\"3ff217ee-d18c-45ca-9a3b-7907923e9e26\" host=\"localhost:6790\" picoName=\"platform\" severity=\"Information\" type=\"System\"]","message":"The pico instance platform (platform) at host localhost has connected."}]
syslog.0: [1730863047.927000000, {"pri":"14","time":"2024-11-06T11:17:27.927+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"System","extradata":"[2@32473 UUID=\"e3ac6725-23cf-4e56-b483-803e3dfaaf90\" host=\"localhost:6790\" severity=\"Information\" type=\"System\"]","message":"System message: The system started"}]
syslog.0: [1730863057.633000000, {"pri":"14","time":"2024-11-06T11:17:37.633+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"System","extradata":"[2@32473 UUID=\"2b4d0683-3a6d-442e-9d45-c5089159d6e0\" host=\"localhost:6790\" picoName=\"platform\" severity=\"Information\" type=\"System\"]","message":"The pico instance ec1 (ec) at host localhost has connected."}]
syslog.0: [1730863068.462000000, {"pri":"14","time":"2024-11-06T11:17:48.462+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"System","extradata":"[2@32473 UUID=\"819b616b-504a-42fc-a271-907fa9e801b6\" host=\"localhost:6790\" picoName=\"platform\" severity=\"Information\" type=\"System\"]","message":"The pico instance ec2 (ec) at host localhost has connected."}]
syslog.0: [1730863179.066000000, {"pri":"14","time":"2024-11-06T11:19:39.066+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"System","extradata":"[2@32473 UUID=\"9b3191a2-d90b-441e-9602-8a2d99371301\" host=\"localhost:6790\" picoName=\"platform\" severity=\"Information\" type=\"System\"]","message":"The pico instance desktop-0 (desktop) at host 192.168.100.3 has connected."}]
syslog.0: [1730863179.085000000, {"pri":"14","time":"2024-11-06T11:19:39.085+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"User","extradata":"[3@32473 UUID=\"3ebb0e90-a3db-40b3-b11a-f7cd23fb6ff1\" host=\"192.168.100.3:52018\" severity=\"Information\" type=\"User\" user=\"mzadmin\"]","message":"mzadmin, Desktop user logged in"}]
syslog.0: [1730863752.812000000, {"pri":"14","time":"2024-11-06T11:29:12.812+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"User","extradata":"[3@32473 UUID=\"6e360bcb-fee7-433e-a964-cb3e547786fe\" host=\"diameter-client.digitalroute.com:52018\" severity=\"Information\" type=\"User\" user=\"mzadmin\" workflow=\"perf_http2_client_server.wf_http2_server_perf.workflow_1\"]","message":"mzadmin, Debug has been enabled on workflow perf_http2_client_server.wf_http2_server_perf.workflow_1."}]
syslog.0: [1730863752.822000000, {"pri":"14","time":"2024-11-06T11:29:12.822+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"System","extradata":"[2@32473 UUID=\"c19103ec-bbe9-48b2-9c79-7f0f03de9727\" host=\"localhost:6790\" severity=\"Information\" type=\"System\"]","message":"System message: Workflow perf_http2_client_server.wf_http2_server_perf.workflow_1 is being manually started by user: mzadmin"}]
syslog.0: [1730863752.823000000, {"pri":"14","time":"2024-11-06T11:29:12.823+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"User","extradata":"[3@32473 UUID=\"955166fe-87ac-4ced-8a51-c17888431bd3\" host=\"diameter-client.digitalroute.com:52018\" severity=\"Information\" type=\"User\" user=\"mzadmin\" workflow=\"perf_http2_client_server.wf_http2_server_perf.workflow_1\"]","message":"mzadmin, Workflow perf_http2_client_server.wf_http2_server_perf.workflow_1 (1) started."}]
Remote rsyslog Server
Nov 6 03:17:27 server-host.local - The pico instance platform (platform) at host localhost has connected. Nov 6 03:17:27 server-host.local - System message: The system started Nov 6 03:17:37 server-host.local - The pico instance ec1 (ec) at host localhost has connected. Nov 6 03:17:48 server-host.local - The pico instance ec2 (ec) at host localhost has connected. Nov 6 03:19:39 server-host.local - The pico instance desktop-0 (desktop) at host 192.168.100.3 has connected. Nov 6 03:19:39 server-host.local - mzadmin, Desktop user logged in Nov 6 03:29:12 server-host.local - mzadmin, Debug has been enabled on workflow perf_http2_client_server.wf_http2_server_perf.workflow_1. Nov 6 03:29:12 server-host.local - System message: Workflow perf_http2_client_server.wf_http2_server_perf.workflow_1 is being manually started by user: mzadmin Nov 6 03:29:12 server-host.local - mzadmin, Workflow perf_http2_client_server.wf_http2_server_perf.workflow_1 (1) started