Skip to main content

Memcached logging (and others) under Systemd on RHEL7

I've been getting into RHEL7 lately (yay, a valid reason to get into at work!) and this means learning about systemd. This post is not about systemd... at least its not another systemd tutorial. This post is about how I got memcached to emit its logging to syslog while running under systemd, and how to configure memcache to sanely log to a file that I can expose via a file share.

Its 2:21am, so this is gonna be quick. I wrote this because I needed memcached, and in response to some bad advice on Stack Overflow.



The usual configuration... it hasn't changed much BUT do be aware that OPTIONS doesn't appear to allow a shell script (so no shell redirections) and that I've specified -vv in order to get some logs.

# cat /etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-l 127.0.0.1 -vv"

First off, note that we can actually get information from using journalctl -- anything sent to a systemd service's stdout, by default and depending on the service. Although after this body of work, the logs will be sent to a syslog, instead of to the journal, so this will be the last we'll see of them (in the journal).

# journalctl --since '2012-01-01' _SYSTEMD_UNIT=memcached.service
-- Logs begin at Fri 2015-07-10 11:00:21 NZST, end at Tue 2015-08-04 02:23:05 NZST. --
Aug 03 23:36:49 HOSTNAME memcached[4318]: slab class   1: chunk size        96 perslab   10922
Aug 03 23:36:49 HOSTNAME memcached[4318]: slab class   2: chunk size       120 perslab    8738
Aug 03 23:36:49 HOSTNAME memcached[4318]: slab class   3: chunk size       152 perslab    6898
Aug 03 23:36:49 HOSTNAME memcached[4318]: slab class   4: chunk size       192 perslab    5461
Aug 03 23:36:49 HOSTNAME memcached[4318]: slab class   5: chunk size       240 perslab    4369
Aug 03 23:36:49 HOSTNAME memcached[4318]: slab class   6: chunk size       304 perslab    3449

Here is the default / base service definition for memcached. We don't touch it. Note that it still includes /etc/sysconfig/memcached

# cat /usr/lib/systemd/system/memcached.service
[Unit]
Description=Memcached
Before=httpd.service
After=network.target

[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/memcached
ExecStart=/usr/bin/memcached -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS

[Install]
WantedBy=multi-user.target

Our changes can be dropped into a file under /etc/systemd/...

# cat /etc/systemd/system/memcached.service.d/local.conf
[Service]
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=memcached
SyslogFacility=local1
SyslogLevel=debug
SyslogLevelPrefix=false

Reload the systemd configuration and restart the service that it is about.

# systemctl daemon-reload
# systemctl restart memcached.service

Restart and test that its running and its picked up our file.

# systemctl status memcached
memcached.service - Memcached
   Loaded: loaded (/usr/lib/systemd/system/memcached.service; enabled)
  Drop-In: /etc/systemd/system/memcached.service.d
           └─local.conf        <---------- b="" style="background-color: yellow;">NOTE
   Active: active (running) since Tue 2015-08-04 01:07:50 NZST; 7s ago
 Main PID: 3842 (memcached)
   CGroup: /system.slice/memcached.service
           └─3842 /usr/bin/memcached -u memcached -p 11211 -m 64 -c 1024 -l 127.0.0.1 -vv

... a tail of log messages show here

Note: rsyslogd uses its imjournal module to read logs from journald; make sure you have this configured if you've brought your rsyslog config from a previous version of RHEL. I fell into this trap and I noticed I could see logs from 'logger', but not from journald. My method of prompting memcache to emit some logs was simply to restart it (with -vv you get plenty)

# echo "local1.debug  /var/log/memcached/memcached.log" >> /etc/rsyslog.d/memcached.conf
# mkdir /var/log/memcached
# systemctl restart rsyslog.service
# systemctl status rsyslog.service

Don't forget log rotation

# cat /etc/logrotate.d/memcached
/var/log/memcached/memcached.log {
    daily
    rotate 3
    dateext
    missingok
    create 0640 root root
    compress
    delaycompress
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}



Comments

Popular posts from this blog

ORA-12170: TNS:Connect timeout — resolved

If you're dealing with Oracle clients, you may be familiar with the error message
ERROR ORA-12170: TNS:Connect timed out occurred I was recently asked to investigate such a problem where an application server was having trouble talking to a database server. This issue was blocking progress on a number of projects in our development environment, and our developers' agile post-it note progress note board had a red post-it saying 'Waiting for Cameron', so I thought I should promote it to the front of my rather long list of things I needed to do... it probably also helped that the problem domain was rather interesting to me, and so it ended being a late-night productivity session where I wasn't interrupted and my experimentation wouldn't disrupt others. I think my colleagues are still getting used to seeing email from me at the wee hours of the morning.

This can masquerade as a number of other error strings as well. Here's what you might see in the sqlnet.log f…

Getting MySQL server to run with SSL

I needed to get an old version of MySQL server running with SSL. Thankfully, that support has been there for a long time, although on my previous try I found it rather frustrating and gave it over for some other job that needed doing.

If securing client connections to a database server is a non-negotiable requirement, I would suggest that MySQL is perhaps a poor-fit and other options, such as PostgreSQL -- according to common web-consensus and my interactions with developers would suggest -- should be first considered. While MySQL can do SSL connections, it does so in a rather poor way that leaves much to be desired.

UPDATED 2014-04-28 for MySQL 5.0 (on ancient Debian Etch).

Here is the fast guide to getting SSL on MySQL server. I'm doing this on a Debian 7 ("Wheezy") server. To complete things, I'll test connectivity from a 5.1 client as well as a reasonably up-to-date MySQL Workbench 5.2 CE, plus a Python 2.6 client; just to see what sort of pain awaits.

UPDATE: 2014-0…

From DNS Packet Capture to analysis in Kibana

UPDATE June 2015: Forget this post, just head for the Beats component for ElasticSearch. Beats is based on PacketBeat (the same people). That said, I haven't used it yet.

If you're trying to get analytics on DNS traffic on a busy or potentially overloaded DNS server, then you really don't want to enable query logging. You'd be better off getting data from a traffic capture. If you're capturing this on the DNS server, ensure the capture file doesn't flood the disk or degrade performance overmuch (here I'm capturing it on a separate partition, and running it at a reduced priority).

# nice tcpdump -p -nn -i eth0 -s0 -w /spare/dns.pcap port domain

Great, so now you've got a lot of packets (set's say at least a million, which is a reasonably short capture). Despite being short, that is still a massive pain to work with in Wireshark, and Wireshark is not the best tool for faceting the message stream so you can can look for patterns (eg. to find relationshi…