Log Forwarding
You can send MediationZone log files to an external log collector by configuring the properties described in this section. All log files generated in MediationZone will be forwarded when log forwarding is enabled, including System, Platform, and EC logs.
Prerequisites
The log forwarding function supports two separate, mutually exclusive log formats, selected using the mz.syslog.layouttype property described below. Before using log forwarding, you should be familiar with the format you plan to use:
JSON - MediationZone's own JSON log format
RFC5424 - the IETF syslog protocol specification
The following properties should be added to the platform configuration file to forward the logs to the external log collector:
Properties | Description | Default Value |
|---|---|---|
| This property is used to enable or disable the logging. | false |
| This property defines the protocol to use, either UDP or TCP. | UDP |
| This property defines the hostname or IP address of the target log collector. | localhost |
| This property defines the port of the target log collector. | 514 |
| This property enables TCP keepalive on the connection used to forward logs to the external log collector. Only applicable when | false |
| This property defines the layout type based on the supported specifications. You can select either RFC5424 or JSON. | RFC5424 |
| This property is used to include a full stack trace in your log messages. | false |
| This property is used to generate a debug log file, useful when you need to determine any issues when configuring the log forwarding. | false |
| This property defines the name of the debug log file. | syslog-debug.log |
| This property defines the location of the debug log file. | /tmp/syslog |
| This property defines the number of debug log files to be generated. | 5 |
Platform Configuration
To enable syslog logging to connect to an external log collector.
mzsh topo set topo://container:main1/pico:platform/val:config.properties.mz.syslog.logging.enable trueTo set the external log collector connection port. For instance, port 5140 is the connection port for Fluent-bit
mzsh topo set topo://container:main1/pico:platform/val:config.properties.mz.syslog.port 5140The rest of the properties that are not set explicitly will follow their default value in the tables.
Changes made to the properties above only take effect after the platform restarts.
mzsh restart platformLog Forwarding With Fluent-bit:
Fluent-bit is one of the external log collectors that is widely used by major cloud providers, banks, and companies worldwide. Fluent-bit can read from local files and network devices and can scrape metrics in the Prometheus format from your server. Furthermore, Fluent Bit supports a vendor-neutral approach, seamlessly integrating with other ecosystems such as Prometheus and OpenTelemetry.
Click here to install Fluent-bit for your environment if you have not.
Fluent-bit Configuration File
Fluent-bit configuration file allows you to configure what kind of data source (input) to read from and where to forward the data to, i.e., the data destination (output).
Edit the Fluent-bit configuration file with the following contents.
[SERVICE]
Flush 1
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name syslog
Mode udp
Buffer_Chunk_Size 32000
Buffer_Max_Size 64000
Receive_Buffer_Size 512000
[OUTPUT]
Name stdout
Match *
[OUTPUT]
Name file
Match *
Path <your-directory-path>
[OUTPUT]
name syslog
match *
host <your-host-name-or-ip-address>
port 514
mode udp
syslog_format rfc5424
syslog_maxsize 2048
syslog_severity_key severity
syslog_facility_key facility
syslog_hostname_key host
syslog_appname_key appname
syslog_procid_key procid
syslog_msgid_key msgid
syslog_sd_key sd
syslog_message_key messageThen start Fluent-bit with the configuration file manually
fluent-bit -c /opt/homebrew/etc/fluent-bit/fluent-bit.confIn a Linux environment with Systemd, start Fluent-bit with systemctl
sudo systemctl start fluent-bit Then do a status check, you should see a similar output like this:
$ systemctl status fluent-bit
● fluent-bit.service - Fluent Bit
Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2016-07-07 02:08:01 BST; 9s ago
Main PID: 3820 (fluent-bit)
CGroup: /system.slice/fluent-bit.service
└─3820 /opt/fluent-bit/bin/fluent-bit -c etc/fluent-bit/fluent-bit.conf
...Fluent-bit started up manually with these output on the screen
Fluent Bit v3.1.9
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
______ _ _ ______ _ _ _____ __
| ___| | | | | ___ (_) | |____ |/ |
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __ / /`| |
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / \ \ | |
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /.___/ /_| |_
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ \____(_)___/
[2024/11/05 18:31:45] [ info] [fluent bit] version=3.1.9, commit=, pid=17946
[2024/11/05 18:31:45] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/11/05 18:31:45] [ info] [cmetrics] version=0.9.6
[2024/11/05 18:31:45] [ info] [ctraces ] version=0.5.6
[2024/11/05 18:31:45] [ info] [input:syslog:syslog.0] initializing
[2024/11/05 18:31:45] [ info] [input:syslog:syslog.0] storage_strategy='memory' (memory only)
[2024/11/05 18:31:45] [ info] [in_syslog] UDP server binding 0.0.0.0:5140
[2024/11/05 18:31:45] [ info] [output:syslog:syslog.2] setup done for 192.168.64.2:514 (TLS=off)
[2024/11/05 18:31:45] [ info] [output:stdout:stdout.0] worker #0 started
[2024/11/05 18:31:45] [ info] [output:file:file.1] worker #0 started
[2024/11/05 18:31:45] [ info] [sp] stream processor startedFluent-bit Outputs
Fluent-bit can be configured to route collected data to multiple destinations. The fluent-bit configuration file listed in the previous section routes collected data to standard output, a local file, and a remote rsyslog server.
Refer to official documentation for further information on output plugins supported by Fluent-bit.
Standard Output
[0] syslog.0: [[1730863047.926000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:17:27.926+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"System", "extradata"=>"[2@32473 UUID="3ff217ee-d18c-45ca-9a3b-7907923e9e26" host="localhost:6790" picoName="platform" severity="Information" type="System"]", "message"=>"The pico instance platform (platform) at host localhost has connected."}]
[1] syslog.0: [[1730863047.927000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:17:27.927+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"System", "extradata"=>"[2@32473 UUID="e3ac6725-23cf-4e56-b483-803e3dfaaf90" host="localhost:6790" severity="Information" type="System"]", "message"=>"System message: The system started"}]
[0] syslog.0: [[1730863057.633000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:17:37.633+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"System", "extradata"=>"[2@32473 UUID="2b4d0683-3a6d-442e-9d45-c5089159d6e0" host="localhost:6790" picoName="platform" severity="Information" type="System"]", "message"=>"The pico instance ec1 (ec) at host localhost has connected."}]
[0] syslog.0: [[1730863068.462000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:17:48.462+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"System", "extradata"=>"[2@32473 UUID="819b616b-504a-42fc-a271-907fa9e801b6" host="localhost:6790" picoName="platform" severity="Information" type="System"]", "message"=>"The pico instance ec2 (ec) at host localhost has connected."}]
[0] syslog.0: [[1730863179.066000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:19:39.066+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"System", "extradata"=>"[2@32473 UUID="9b3191a2-d90b-441e-9602-8a2d99371301" host="localhost:6790" picoName="platform" severity="Information" type="System"]", "message"=>"The pico instance desktop-0 (desktop) at host 192.168.100.3 has connected."}]
[1] syslog.0: [[1730863179.085000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:19:39.085+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"User", "extradata"=>"[3@32473 UUID="3ebb0e90-a3db-40b3-b11a-f7cd23fb6ff1" host="192.168.100.3:52018" severity="Information" type="User" user="mzadmin"]", "message"=>"mzadmin, Desktop user logged in"}]
[0] syslog.0: [[1730863752.812000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:29:12.812+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"User", "extradata"=>"[3@32473 UUID="6e360bcb-fee7-433e-a964-cb3e547786fe" host="diameter-client.digitalroute.com:52018" severity="Information" type="User" user="mzadmin" workflow="perf_http2_client_server.wf_http2_server_perf.workflow_1"]", "message"=>"mzadmin, Debug has been enabled on workflow perf_http2_client_server.wf_http2_server_perf.workflow_1."}]
[1] syslog.0: [[1730863752.822000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:29:12.822+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"System", "extradata"=>"[2@32473 UUID="c19103ec-bbe9-48b2-9c79-7f0f03de9727" host="localhost:6790" severity="Information" type="System"]", "message"=>"System message: Workflow perf_http2_client_server.wf_http2_server_perf.workflow_1 is being manually started by user: mzadmin"}]
[2] syslog.0: [[1730863752.823000000, {}], {"pri"=>"14", "time"=>"2024-11-06T11:29:12.823+08:00", "host"=>"server-host.local", "ident"=>"DRX-9.3.1.0", "pid"=>"3795", "msgid"=>"User", "extradata"=>"[3@32473 UUID="955166fe-87ac-4ced-8a51-c17888431bd3" host="diameter-client.digitalroute.com:52018" severity="Information" type="User" user="mzadmin" workflow="perf_http2_client_server.wf_http2_server_perf.workflow_1"]", "message"=>"mzadmin, Workflow perf_http2_client_server.wf_http2_server_perf.workflow_1 (1) started."}]Local File
syslog.0: [1730863047.926000000, {"pri":"14","time":"2024-11-06T11:17:27.926+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"System","extradata":"[2@32473 UUID=\"3ff217ee-d18c-45ca-9a3b-7907923e9e26\" host=\"localhost:6790\" picoName=\"platform\" severity=\"Information\" type=\"System\"]","message":"The pico instance platform (platform) at host localhost has connected."}]
syslog.0: [1730863047.927000000, {"pri":"14","time":"2024-11-06T11:17:27.927+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"System","extradata":"[2@32473 UUID=\"e3ac6725-23cf-4e56-b483-803e3dfaaf90\" host=\"localhost:6790\" severity=\"Information\" type=\"System\"]","message":"System message: The system started"}]
syslog.0: [1730863057.633000000, {"pri":"14","time":"2024-11-06T11:17:37.633+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"System","extradata":"[2@32473 UUID=\"2b4d0683-3a6d-442e-9d45-c5089159d6e0\" host=\"localhost:6790\" picoName=\"platform\" severity=\"Information\" type=\"System\"]","message":"The pico instance ec1 (ec) at host localhost has connected."}]
syslog.0: [1730863068.462000000, {"pri":"14","time":"2024-11-06T11:17:48.462+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"System","extradata":"[2@32473 UUID=\"819b616b-504a-42fc-a271-907fa9e801b6\" host=\"localhost:6790\" picoName=\"platform\" severity=\"Information\" type=\"System\"]","message":"The pico instance ec2 (ec) at host localhost has connected."}]
syslog.0: [1730863179.066000000, {"pri":"14","time":"2024-11-06T11:19:39.066+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"System","extradata":"[2@32473 UUID=\"9b3191a2-d90b-441e-9602-8a2d99371301\" host=\"localhost:6790\" picoName=\"platform\" severity=\"Information\" type=\"System\"]","message":"The pico instance desktop-0 (desktop) at host 192.168.100.3 has connected."}]
syslog.0: [1730863179.085000000, {"pri":"14","time":"2024-11-06T11:19:39.085+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"User","extradata":"[3@32473 UUID=\"3ebb0e90-a3db-40b3-b11a-f7cd23fb6ff1\" host=\"192.168.100.3:52018\" severity=\"Information\" type=\"User\" user=\"mzadmin\"]","message":"mzadmin, Desktop user logged in"}]
syslog.0: [1730863752.812000000, {"pri":"14","time":"2024-11-06T11:29:12.812+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"User","extradata":"[3@32473 UUID=\"6e360bcb-fee7-433e-a964-cb3e547786fe\" host=\"diameter-client.digitalroute.com:52018\" severity=\"Information\" type=\"User\" user=\"mzadmin\" workflow=\"perf_http2_client_server.wf_http2_server_perf.workflow_1\"]","message":"mzadmin, Debug has been enabled on workflow perf_http2_client_server.wf_http2_server_perf.workflow_1."}]
syslog.0: [1730863752.822000000, {"pri":"14","time":"2024-11-06T11:29:12.822+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"System","extradata":"[2@32473 UUID=\"c19103ec-bbe9-48b2-9c79-7f0f03de9727\" host=\"localhost:6790\" severity=\"Information\" type=\"System\"]","message":"System message: Workflow perf_http2_client_server.wf_http2_server_perf.workflow_1 is being manually started by user: mzadmin"}]
syslog.0: [1730863752.823000000, {"pri":"14","time":"2024-11-06T11:29:12.823+08:00","host":"server-host.local","ident":"DRX-9.3.1.0","pid":"3795","msgid":"User","extradata":"[3@32473 UUID=\"955166fe-87ac-4ced-8a51-c17888431bd3\" host=\"diameter-client.digitalroute.com:52018\" severity=\"Information\" type=\"User\" user=\"mzadmin\" workflow=\"perf_http2_client_server.wf_http2_server_perf.workflow_1\"]","message":"mzadmin, Workflow perf_http2_client_server.wf_http2_server_perf.workflow_1 (1) started."}]Remote rsyslog Server
Nov 6 03:17:27 server-host.local - The pico instance platform (platform) at host localhost has connected.
Nov 6 03:17:27 server-host.local - System message: The system started
Nov 6 03:17:37 server-host.local - The pico instance ec1 (ec) at host localhost has connected.
Nov 6 03:17:48 server-host.local - The pico instance ec2 (ec) at host localhost has connected.
Nov 6 03:19:39 server-host.local - The pico instance desktop-0 (desktop) at host 192.168.100.3 has connected.
Nov 6 03:19:39 server-host.local - mzadmin, Desktop user logged in
Nov 6 03:29:12 server-host.local - mzadmin, Debug has been enabled on workflow perf_http2_client_server.wf_http2_server_perf.workflow_1.
Nov 6 03:29:12 server-host.local - System message: Workflow perf_http2_client_server.wf_http2_server_perf.workflow_1 is being manually started by user: mzadmin
Nov 6 03:29:12 server-host.local - mzadmin, Workflow perf_http2_client_server.wf_http2_server_perf.workflow_1 (1) started