Friday, January 14, 2011

SASL client library...

There is a lot of software out there that allows you to plug in SASL authentication if you got a SASL client library on your system. Some operating systems allows an easy download of a SASL library for their platform, but I have not seen any for Windows yet.


Membase supports SASL authentication, so when I started to implement libmembase I decided that I wanted to treat libsasl as a required dependency. One of my goals with libmembase is that it should be easy to compile as a dll for Windows (I'm not there yet), so I needed a libsasl dll for Windows.


Earlier today I fixed and pushed a (client side) SASL library to https://github.com/membase/libisasl/. If you're running on a Unix-like system you should build the library by using autotools, but I've added a Makefile you may use to build the dll on Windows with the following command:

nmake -f NMakefile


There is no install-target in the Makefile, so you need to copy the header-files from "include", libsasl.dll and libsasl.lib to the desired location on your machine when you're done.

Tuesday, January 11, 2011

Developing with Membase

As a developer I need to be able to start my processes in a certain way. One way to do that may be to modify the startup code we've got in our management system, but I found it way more flexible and easy to just replace our binaries with wrapper scripts that starts up our binaries.

Please note that this is something I do when I try to track down a certain bug, and not something I recommend in your production environment.

I've created my own little script that installs the wrapper script:

#! /bin/ksh

cat > /opt/membase/bin/launcher.sh <
#! /bin/ksh
logfile=/tmp/membase.log
binary=\`basename \$0\`
echo pid \$\$ : \$0 \$* >> \${logfile}
exec \${0}.bin "\$@" 2>&1 | awk "{printf(\"%d: %s\n\", $$, \\\$0); }" >> \${logfile}
EOF

chmod a+x /opt/membase/bin/launcher.sh

for f in memcached vbucketmigrator moxi
do
   mv /opt/membase/bin/${f}/${f} /opt/membase/bin/${f}/${f}.bin
   ln -s ../launcher.sh /opt/membase/bin/${f}/${f}
done

As an extra bonus this redirects all of the output from the processes to /tmp/membase.log, so that I can just check there for the error text instead of running browse_logs and start decoding the output there.

The above script use the same wrapper script for all processes, but sometimes I want to add extra options to one of the processes (like enabling verbosity for vbucketmigrator). All I need to do is just to replace the link with a copy of the file:

root@ubuntu:/opt/membase/bin# rm vbucketmigrator/vbucketmigrator
root@ubuntu:/opt/membase/bin# cp -p launcher.sh vbucketmigrator/vbucketmigrator

and edit the file. Since I'm going to add extra command line options, I'm most likely expecting more output so normally I store the output in its own file as well:

#! /bin/ksh
logfile=/tmp/vbucketmigrator.$$
binary=`basename $0`
echo pid $$ : $0 $* >> ${logfile}
exec ${0}.bin "$@" -vv 2>&1 >> ${logfile}

The next time vbucketmigrator starts it will dump the message traffic to /tmp/vbucketmigrator.pid

Monday, January 10, 2011

Dumping stats from a memcached server...

I normally dump the stats from my memcached servers with the following command:

trond@opensolaris> echo stats | nc localhost 11211

But if you start memcached with the ascii protocol disabled it becomes hard to dump the stats. I just created a small tool named mcstats that dumps the stats in the same format as you would get from the above command.

Example:
trond@opensolaris> ./mcstat
STAT evictions 0
STAT curr_items 0
STAT total_items 0
STAT bytes 0
STAT reclaimed 0
STAT engine_maxbytes 67108864
STAT pid 15436
STAT uptime 682
STAT time 1294693339
STAT version 1.3.3_499_g580ae55
STAT libevent 1.4.13-stable
STAT pointer_size 32
STAT rusage_user 0.012261
STAT rusage_system 0.014169
STAT daemon_connections 10
STAT curr_connections 11
STAT total_connections 16
STAT connection_structures 14
STAT cmd_get 0
STAT cmd_set 0
STAT cmd_flush 0
STAT auth_cmds 0
STAT auth_errors 0
STAT get_hits 0
STAT get_misses 0
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT bytes_read 106
STAT bytes_written 5656
STAT limit_maxbytes 67108864
STAT rejected_conns 0
STAT threads 4
STAT conn_yields 0

By default mcstat will connect to "localhost:11211", but you may tell it to go somewhere else by using -h host.

You'll find the tool in my git branch of the memcached source repository


Happy new year btw.