Replace Journald in Fedora or CentOS

Journald – a contentious subject in the Linux community.  On one side, proponents tout advanced features and security. On the other, die-hard *nix users decry yet another move away from the simplicity of text files and more towards the Windows way of obfuscated binary file based systems.  After using journald (and systemd) for a couple of years under Fedora,we’re unconvinced despite being a user of journal signing in security-conscious applications.

With Red Hat shipping journald in 7.0, right or wrong, journald is here to stay.

Fedora 20 no longer includes syslog in new installations. The log files you expect to be there – aren’t. As heavy users of OSSEC, fail2ban, and other log-reading security applications, this doesn’t fly.

It’s simple enough to install rsyslog on Fedora, and it defaults to reading journald.

We’ve encountered two frequent problems with this approach on smaller VMs:

1. journald uses an enormous amount of space for the journals. It’s actually storing more information than syslog, and you can modify journald.conf to limit space usage. However, it’s still a concern on space-constrained VMs.

2. On I/O or CPU constrained VMs, journald causes a surprising amount of disk i/o for an otherwise quiet system.

What to do?

The title of this post is misleading. You cannot effectively remove journald from a system using systemd. You can, however alleviate both of the primary problems we encounter.

First, reconfigure journald’s storage mode to ‘volatile’. This means it will not consume massive amounts of disk space in /var/log/journal, but will instead store logs temporarily in tmpfs on /run in /run/logs/journal. This solves our disk space problem and our disk i/o problem.

Modify /etc/systemd/journald.conf:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# See journald.conf(5) for details

[Journal]
#Storage=auto
Storage=volatile

Restart journald:

# systemctl restart systemd-journald
#

Second, install and configure rsyslog to start automatically.

# yum -y install rsyslog
Loaded plugins: langpacks, refresh-packagekit
atlassian-hipchat                                                                                                 2/2
Resolving Dependencies
--> Running transaction check
---> Package rsyslog.x86_64 0:7.4.8-1.fc20.1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================
 Package                   Arch                     Version                           Repository                 Size
======================================================================================================================
Installing:
 rsyslog                   x86_64                   7.4.8-1.fc20.1                    updates                   546 k

Transaction Summary
======================================================================================================================
Install  1 Package

Total download size: 546 k
Installed size: 1.9 M
Downloading packages:
rsyslog-7.4.8-1.fc20.1.x86_64.rpm                                                              | 546 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction (shutdown inhibited)
  Installing : rsyslog-7.4.8-1.fc20.1.x86_64                                                                      1/1 
  Verifying  : rsyslog-7.4.8-1.fc20.1.x86_64                                                                      1/1 

Installed:
  rsyslog.x86_64 0:7.4.8-1.fc20.1                                                                                     

Complete!

# systemctl enable rsyslog
# systemctl start rsyslog

On Fedora, rsyslog comes pre-configured to load the imjournal module that will read logs from /run/log/journal. Rsyslog will then write out /var/log/messages, /var/log/maillog, /var/log/secure and all the other logs you’ve come to depend on.

You can now remove your journal files in the subdirectory under /var/log/journal.

On the 1GB VM we tested this on, this cut our disk i/o in half and allowed us to remove 4GB of journal files from a cramped disk.

Is journald really the future of Linux?