Replace Journald in Fedora or CentOS

journaldJournald – a contentious subject in the Linux community.  On one side, proponents tout advanced features and security. On the other, die-hard *nix users decry yet another move away from the simplicity of text files and more towards the Windows way of obfuscated binary file based systems.  After using journald (and systemd) for a couple of years under Fedora,we’re unconvinced despite being a user of journal signing in security-concious applications.

With Red Hat shipping journald in 7.0, right or wrong, journald is here to stay.

Fedora 20 no longer includes syslog in new installations. The log files you expect to be there – aren’t. As heavy users of OSSEC, fail2ban, and other log-reading security applications, this doesn’t fly.

It’s simple enough to install rsyslog on Fedora, and it defaults to reading journald.

We’ve encountered two frequent problems with this approach on smaller VMs:

1. journald uses an enormous amount of space for the journals. It’s actually storing more information than syslog, and you can modify journald.conf to limit space usage. However, it’s still a concern on space-constrained VMs.

2. On I/O or CPU constrained VMs, journald causes a surprising amount of disk i/o for an otherwise quiet system.

What to do?

The title of this post is misleading. You cannot effectively remove journald from a system using systemd. You can, however alleviate both of the primary problems we encounter.

First, reconfigure journald’s storage mode to ‘volatile’. This means it will not consume massive amounts of disk space in /var/log/journal, but will instead store logs temporarily in tmpfs on /run in /run/logs/journal. This solves our disk space problem and our disk i/o problem.

Modify /etc/systemd/journald.conf:

Restart journald:

Second, install and configure rsyslog to start automatically.

On Fedora, rsyslog comes pre-configured to load the imjournal module that will read logs from /run/log/journal. Rsyslog will then write out /var/log/messages, /var/log/maillog, /var/log/secure and all the other logs you’ve come to depend on.

You can now remove your journal files in the subdirectory under /var/log/journal.

On the 1GB VM we tested this on, this cut our disk i/o in half and allowed us to remove 4GB of journal files from a cramped disk.

Is journald really the future of Linux?