¿Journald'ing Remotely? Mucho Sí!
If you're running Linux and using systemd then you've likely needed to deal with systemd-journald to stream system logs, or check on the general health of the node. In the course of administering your systemd hosts it's likely you've thought, it sure would be nice if I could ship journals natively within systemd. Well, I'm here to tell you, you can, and if you've had this nightmare, keep reading, if not close the browser tab and consider yourself lucky.
In this post, I'm going to cover how to journal remotely and stream logs from a remote host. The systemd-journald functionality provides quite a bit of great functionality but leveraging can be a bit cryptic. While the documentation on optionality is generally complete, the implementation of some features is less than stellar. This post aims to shine a light on the remote journaling capabilities built into systemd and how to consume the remotely journaled logs.
Setup the Remote (Journal Receiver)
The first thing needed is to get remote journaling functional. To do this, setup ONE node that receives journals. Most installations come with systemd-journal-remote already installed however if the executable /lib/systemd/systemd-journal-remote is not present, it'll need to be installed.
On Ubuntu 16.04 install the following package.
apt install systemd-journal-remote
Create the journal-remote configuration file.
cat > /etc/systemd/journal-remote.conf <<EOF
[Remote]
SplitMode=host
EOF
Create the systemd-journal-remote service unit override file.
The following override file is using unencrypted HTTP to transfer the journals. In a production environment, this should use HTTPS. For a full rundown of all of the available options see the upstream documentation.
cat > /etc/systemd/system/systemd-journal-remote.service <<EOF
[Unit]
Description=Journal Remote Sink Service
Documentation=man:systemd-journal-remote(8) man:journal-remote.conf(5)
Requires=systemd-journal-remote.socket
[Service]
ExecStart=/lib/systemd/systemd-journal-remote --listen-http=-3 --output=/var/log/journal/remote/
User=systemd-journal-remote
Group=systemd-journal-remote
PrivateTmp=yes
PrivateDevices=yes
PrivateNetwork=yes
WatchdogSec=3min
[Install]
Also=systemd-journal-remote.socket
EOF
Create the directory where the remote journals will be placed.
mkdir -p /var/log/journal/remote
chown -R systemd-journal-remote:systemd-journal-remote /var/log/journal/remote
Enable and start the systemd-journal-remote.socket.
systemctl enable systemd-journal-remote.socket
systemctl start systemd-journal-remote.socket
systemctl status systemd-journal-remote.socket
● systemd-journal-remote.socket - Journal Remote Sink Socket
Loaded: loaded (/lib/systemd/system/systemd-journal-remote.socket; enabled; vendor preset: enabled)
Active: active (running) since Mon 2018-02-19 13:34:24 CST; 8h ago
Listen: [::]:19532 (Stream)
Feb 19 13:34:24 mining-logger systemd[1]: Listening on Journal Remote Sink Socket.
Enable the systemd-journal-remote service and start it.
systemctl enable systemd-journal-remote.service
systemctl start systemd-journal-remote.service
systemctl status systemd-journal-remote.service
● systemd-journal-remote.service - Journal Remote Sink Service
Loaded: loaded (/etc/systemd/system/systemd-journal-remote.service; indirect; vendor preset: enabled)
Active: active (running) since Mon 2018-02-19 13:34:31 CST; 7h ago
Docs: man:systemd-journal-remote(8)
man:journal-remote.conf(5)
Main PID: 287 (systemd-journal)
Status: "Processing requests..."
CGroup: /machine.slice/systemd-nspawn@mining\x2dlogger.service/system.slice/systemd-journal-remote.service
└─287 /lib/systemd/systemd-journal-remote --listen-http=-3 --output=/var/log/journal/remote/
Feb 19 13:34:31 mining-logger systemd[1]: Started Journal Remote Sink Service.
Setup the Upload (Journal Sender)
Once the remote receiver is online it's time to setup the remote hosts. As with the remote receiver, the first thing needed is to install/setup systemd-journal-remote. If it's not already installed, install it.
On Ubuntu 16.04 install the following package.
apt install systemd-journal-upload
Create the journal-upload.conf file.
The following override file is using unencrypted HTTP to transfer the journals. In a production environment, this should use HTTPS. For a full rundown of all of the available options see the upstream documentation.
cat > /etc/systemd/journal-upload.conf <<EOF
[Upload]
URL=http://${IP_OR_DOMAIN_OR_RECIEVER}:19532
EOF
Create the systemd-journal-upload service unit override file.
cat > /etc/systemd/system/systemd-journal-upload.service <<EOF
[Unit]
Description=Journal Remote Upload Service
Documentation=man:systemd-journal-upload(8)
After=network.target
[Service]
ExecStart=/lib/systemd/systemd-journal-upload --save-state
User=systemd-journal-upload
SupplementaryGroups=systemd-journal
PrivateTmp=yes
PrivateDevices=yes
WatchdogSec=3min
# Add reset/restart options
TimeoutSec=120
Restart=on-failure
RestartSec=2
# Add accounting options
CPUAccounting=true
BlockIOAccounting=true
MemoryAccounting=false
TasksAccounting=true
# If there are many splits up journal files we need a lot of file descriptors to access them all and combine
LimitNOFILE=16384
[Install]
WantedBy=multi-user.target
EOF
Enable the systemd-journal-upload service and start it.
systemctl enable systemd-journal-upload.service
systemctl start systemd-journal-upload.service
systemctl status systemd-journal-upload.service
● systemd-journal-upload.service - Journal Remote Upload Service
Loaded: loaded (/etc/systemd/system/systemd-journal-upload.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2018-02-19 17:40:27 CST; 3h 35min ago
Docs: man:systemd-journal-upload(8)
Main PID: 869 (systemd-journal)
Status: "Processing input..."
Tasks: 1
CPU: 13.814s
CGroup: /system.slice/systemd-journal-upload.service
└─869 /lib/systemd/systemd-journal-upload --save-state
Feb 19 17:40:27 txrig2 systemd[1]: Started Journal Remote Upload Service.
Once the service is started the journals should immediately start streaming to the remote (server) host.
When running on operating systems running Firewalld a port will need to be opened. One port for the upload the other for the gateway.
firewall-cmd --add-port=19531/tcp # for the upload firewall-cmd --add-port=19532/tcp # for the gateway-- Thanks Dan Berger for the notes regarding Firewalld.
Journal Processing
On the receiving host login, and check the remote directory. If everything is working as expected the journal files will start coming saving the streams at /var/log/journal/remote/remote-${HOST_OR_ADDRESS}.journal.
Example from a running journal server.
ls -1 /var/log/journal/remote/
remote-172.16.24.254.journal
remote-172.16.24.255.journal
remote-172.16.24.255@359a897567f3440ea87e860f4f145d28-0000000000000001-0005657e971d3848.journal
remote-172.16.24.255@359a897567f3440ea87e860f4f145d28-000000000002657e-00056587fbb2e63c.journal
remote-172.16.24.255@359a897567f3440ea87e860f4f145d28-000000000004d0a1-0005659192286c8c.journal
In order to stream, parse or otherwise consume the remote journals use the --file switch with the journalctl command. For an entire breakdown of available options see the upstream documentation.
journalctl --file /var/log/journal/remote/remote-172.16.24.255.journal -n 3
-- Logs begin at Mon 2018-02-19 19:57:26 CST, end at Mon 2018-02-19 21:33:08 CST. --
Feb 19 21:33:07 txrig2 start-miner.sh[866]: m 21:33:07|ethminer Speed 139.79 Mh/s gpu/0 28.89 69C 27% gpu/1 28.81 69C 27% gpu/2 28.97 69C 36% gpu/3 26.31 69C 25% gpu/4 26.80 69C 33% [A486+0:R0+0:F0] Time: 03:52
Feb 19 21:33:08 txrig2 start-miner.sh[866]: m 21:33:08|ethminer Speed 139.79 Mh/s gpu/0 28.89 70C 27% gpu/1 28.81 69C 27% gpu/2 28.97 69C 36% gpu/3 26.31 69C 25% gpu/4 26.80 69C 33% [A486+0:R0+0:F0] Time: 03:52
Feb 19 21:33:08 txrig2 start-miner.sh[866]: m 21:33:08|ethminer Speed 139.79 Mh/s gpu/0 28.89 69C 27% gpu/1 28.81 69C 27% gpu/2 28.97 69C 36% gpu/3 26.31 69C 25% gpu/4 26.80 69C 33% [A486+0:R0+0:F0] Time: 03:52
That's all Folks
I hope you've enjoyed this post and it puts to bed any journal nightmares of you might have had. The setup is simple, useful, and quite powerful. If you've liked this post reach out, hit me up on twitter, let me know.