Skip to main content

Memcached logging (and others) under Systemd on RHEL7

I've been getting into RHEL7 lately (yay, a valid reason to get into at work!) and this means learning about systemd. This post is not about systemd... at least its not another systemd tutorial. This post is about how I got memcached to emit its logging to syslog while running under systemd, and how to configure memcache to sanely log to a file that I can expose via a file share.

Its 2:21am, so this is gonna be quick. I wrote this because I needed memcached, and in response to some bad advice on Stack Overflow.

The usual configuration... it hasn't changed much BUT do be aware that OPTIONS doesn't appear to allow a shell script (so no shell redirections) and that I've specified -vv in order to get some logs.

# cat /etc/sysconfig/memcached
OPTIONS="-l -vv"

First off, note that we can actually get information from using journalctl -- anything sent to a systemd service's stdout, by default and depending on the service. Although after this body of work, the logs will be sent to a syslog, instead of to the journal, so this will be the last we'll see of them (in the journal).

# journalctl --since '2012-01-01' _SYSTEMD_UNIT=memcached.service
-- Logs begin at Fri 2015-07-10 11:00:21 NZST, end at Tue 2015-08-04 02:23:05 NZST. --
Aug 03 23:36:49 HOSTNAME memcached[4318]: slab class   1: chunk size        96 perslab   10922
Aug 03 23:36:49 HOSTNAME memcached[4318]: slab class   2: chunk size       120 perslab    8738
Aug 03 23:36:49 HOSTNAME memcached[4318]: slab class   3: chunk size       152 perslab    6898
Aug 03 23:36:49 HOSTNAME memcached[4318]: slab class   4: chunk size       192 perslab    5461
Aug 03 23:36:49 HOSTNAME memcached[4318]: slab class   5: chunk size       240 perslab    4369
Aug 03 23:36:49 HOSTNAME memcached[4318]: slab class   6: chunk size       304 perslab    3449

Here is the default / base service definition for memcached. We don't touch it. Note that it still includes /etc/sysconfig/memcached

# cat /usr/lib/systemd/system/memcached.service

ExecStart=/usr/bin/memcached -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS


Our changes can be dropped into a file under /etc/systemd/...

# cat /etc/systemd/system/memcached.service.d/local.conf

Reload the systemd configuration and restart the service that it is about.

# systemctl daemon-reload
# systemctl restart memcached.service

Restart and test that its running and its picked up our file.

# systemctl status memcached
memcached.service - Memcached
   Loaded: loaded (/usr/lib/systemd/system/memcached.service; enabled)
  Drop-In: /etc/systemd/system/memcached.service.d
           └─local.conf        <---------- b="" style="background-color: yellow;">NOTE
   Active: active (running) since Tue 2015-08-04 01:07:50 NZST; 7s ago
 Main PID: 3842 (memcached)
   CGroup: /system.slice/memcached.service
           └─3842 /usr/bin/memcached -u memcached -p 11211 -m 64 -c 1024 -l -vv

... a tail of log messages show here

Note: rsyslogd uses its imjournal module to read logs from journald; make sure you have this configured if you've brought your rsyslog config from a previous version of RHEL. I fell into this trap and I noticed I could see logs from 'logger', but not from journald. My method of prompting memcache to emit some logs was simply to restart it (with -vv you get plenty)

# echo "local1.debug  /var/log/memcached/memcached.log" >> /etc/rsyslog.d/memcached.conf
# mkdir /var/log/memcached
# systemctl restart rsyslog.service
# systemctl status rsyslog.service

Don't forget log rotation

# cat /etc/logrotate.d/memcached
/var/log/memcached/memcached.log {
    rotate 3
    create 0640 root root
        /bin/kill -HUP `cat /var/run/ 2> /dev/null` 2> /dev/null || true


Popular posts from this blog

Use IPTables NOTRACK to implement stateless rules and reduce packet loss.

I recently struck a performance problem with a high-volume Linux DNS server and found a very satisfying way to overcome it. This post is not about DNS specifically, but useful also to services with a high rate of connections/sessions (UDP or TCP), but it is especially useful for UDP-based traffic, as the stateful firewall doesn't really buy you much with UDP. It is also applicable to services such as HTTP/HTTPS or anything where you have a lot of connections...

We observed times when DNS would not respond, but retrying very soon after would generally work. For TCP, you may find that you get a a connection timeout (or possibly a connection reset? I haven't checked that recently).

Observing logs, you might the following in kernel logs:
kernel: nf_conntrack: table full, dropping packet. You might be inclined to increase net.netfilter.nf_conntrack_max and net.nf_conntrack_max, but a better response might be found by looking at what is actually taking up those entries in your conne…

ORA-12170: TNS:Connect timeout — resolved

If you're dealing with Oracle clients, you may be familiar with the error message
ERROR ORA-12170: TNS:Connect timed out occurred I was recently asked to investigate such a problem where an application server was having trouble talking to a database server. This issue was blocking progress on a number of projects in our development environment, and our developers' agile post-it note progress note board had a red post-it saying 'Waiting for Cameron', so I thought I should promote it to the front of my rather long list of things I needed to do... it probably also helped that the problem domain was rather interesting to me, and so it ended being a late-night productivity session where I wasn't interrupted and my experimentation wouldn't disrupt others. I think my colleagues are still getting used to seeing email from me at the wee hours of the morning.

This can masquerade as a number of other error strings as well. Here's what you might see in the sqlnet.log f…

Getting MySQL server to run with SSL

I needed to get an old version of MySQL server running with SSL. Thankfully, that support has been there for a long time, although on my previous try I found it rather frustrating and gave it over for some other job that needed doing.

If securing client connections to a database server is a non-negotiable requirement, I would suggest that MySQL is perhaps a poor-fit and other options, such as PostgreSQL -- according to common web-consensus and my interactions with developers would suggest -- should be first considered. While MySQL can do SSL connections, it does so in a rather poor way that leaves much to be desired.

UPDATED 2014-04-28 for MySQL 5.0 (on ancient Debian Etch).

Here is the fast guide to getting SSL on MySQL server. I'm doing this on a Debian 7 ("Wheezy") server. To complete things, I'll test connectivity from a 5.1 client as well as a reasonably up-to-date MySQL Workbench 5.2 CE, plus a Python 2.6 client; just to see what sort of pain awaits.

UPDATE: 2014-0…