Running Membase on OpenSolaris

I've been wanting to run membase on my OpenSolaris server for a long time, but I haven't had the time to look into all of the details on getting it there. All of the command line tools compile and works perfectly well on OpenSolaris already, but I haven't been able to get the web ui and cluster management up and running. One of the bugs preventing an easy install of membase on OpenSolaris (and other unsupported platforms) is that the makefile for the admin ui don't have an install target (This is tracked in MB-2700 ).

Earlier today I cloned the layout of the web application on disk from the Ubuntu installation to my OpenSolaris box, and with some work I was able to get the system up and running on my OpenSolaris box. Just check it out:

trond@storm> svcs membase
STATE          STIME    FMRI
online         13:37:26 svc:/application/database/membase:membase

So let's assume you've got all of the dependencies you need to build membase installed on your OpenSolaris (I'll come up with this list later on and create an IPS package for you to install from my IPS server).

Due to MB-3227 we can't use the build method outlined in "Building on OSX from source" to build membase, but I've updated my build environment I blogged about a while ago so that it will build and install everything for you! So let's go ahead and build the stuff!

trond@storm> git clone git://github.com/trondn/tools.git
Initialized empty Git repository in /tmp/tools/.git/
remote: Counting objects: 168, done.
remote: Compressing objects: 100% (163/163), done.
remote: Total 168 (delta 89), reused 0 (delta 0)
Receiving objects: 100% (168/168), 157.56 KiB | 231 KiB/s, done.
Resolving deltas: 100% (89/89), done.
trond@storm> cd tools/membase/smf/membase
trond@storm> ./setup -u -s -z scratch               (scratch is the name of my zfs pool)
trond@storm> cd ../..
trond@storm> chown -R trond:staff /opt/membase
trond@storm> ./setup.sh -d /opt/membase/dev membase
Download commit hook - Ok.
Checking out libmemcached (Bazaar) - Ok.
Checking out bucket_engine (git) - Ok.
Checking out ep-engine (git) - Ok.
Checking out libconflate (git) - Ok.
Checking out libvbucket (git) - Ok.
Checking out memcached (git) - Ok.
Checking out moxi (git) - Ok.
Checking out vbucketmigrator (git) - Ok.
Checking out membase-cli (git) - Ok.
Checking out ns_server (git) - Ok.
Checking out memcachetest (git) - Ok.
Checking out mc-hammer (git) - Ok.
Checking out libmembase (git) - Ok.
Configure build for SunOS
trond@storm> cd membase
trond@storm> gmake install

Due to MB-2926 we need to change the file layout to move the files so that management process in membase can locate the tools. In theory we shouldn't need to create links for the libraries as well (because the runtime path of the binary should pick up the correct one), but unfortunately ns_server use the absolute path for the library.

trond@storm> cd /opt/membase/bin/dev
trond@storm> rm memcached vbucketmigrator moxi
trond@storm> mkdir memcached vbucketmigrator moxi bucket_engine ep_engine
trond@storm> cd memcached
trond@storm> ln -s ../amd64/memcached
trond@storm> ln -s ../../lib/amd64/default_engine.so
trond@storm> ln -s ../../lib/amd64/stdin_term_handler.so
trond@storm> cd ../vbucketmigrator
trond@storm> ln -s ../amd64/vbucketmigrator
trond@storm> cd ../moxi
trond@storm> ln -s ../amd64/moxi
trond@storm> cd ../bucket_engine
trond@storm> ln -s ../../lib/amd64/bucket_engine.so
trond@storm> cd ../ep_engine
trond@storm> ln -s ../../lib/amd64/ep.so
trond@storm> chown -R membase:membase /opt/membase

So let's go ahead and start the service by running:

trond@storm> svcadm enable membase

And navigate your browser to http://localhost:8091/ and start configuring your server. I was able to let my OpenSolaris server join my Ubuntu cluster.

Happy holidays!

Building libmembase

I guess I was a bit too excited yesterday when I announced libmembase, because I completely forgot to mention where to get and build it.

You will find the code at: https://github.com/trondn/libmembase, and building it should be pretty easy once you've installed all of the required dependencies. I've tried to keep the list of dependencies as short as possible (so you might think I'm suffering from the "not invented here syndrome" since I implemented the REST support myself instead of using cURL, but the main reason for that was to keep the list short ;-)):

• git
• Auto tools (automake, autoheader, autoconf, libtoolize)
• A C99 compiler
• libvbucket
• libevent
• SASL library

I have integrated building of libmembase in the build system I'm using to build membase making it easy for you to get the code and run configure with the correct set of options (see this blog entry for a description).

If you want to build it yourself all you need is:

$ git clone git://github.com/trondn/libmembase.git
$ cd libmembase
$ ./config/autorun.sh
$ ./configure --prefix=/opt/membase
$ make all install

libmembase - a C interface to Membase

Membase is "on the wire" compatible with any memcached server if you connect to the standard memcached port (registered by myself back in 2009), so that you should should be able to access membase with any "memcapable" client. Backing this port is our membase proxy named moxi, and behind the scene it will do SASL authentication and proxy your requests to the correct membase server containing the item you want. One of the things that differs Membase from Memcached is that we store each item in a given vbucket that is mapped to a server. When you grow or shrink the cluster, membase will move the vbuckets to new servers.

There is no such thing as a free lunch, so accessing membase through moxi "costs" more than talking directly to the individual nodes yourself. We like to refer to such clients as "smart clients". As a developer on Memcached I need to test various stuff, so I went ahead and hacked together a quick prototype of such a library to ease my testing. Initially I wanted to extend libmemcached with this functionality, but that seemed to be a big (and risky) change I didn't have the guts to do at the time.

The current state of the library is far from production quality, and with a minimal list of supported features. So why announce it now? Well I don't think I'll find the time to implement everything myself, so I'm hoping that people will join me in adding features to the library when they need something that isn't there...

I've designed the library to be 100% callback based and integrated with libevent, making it easy for you to plug it into your application.

So let's say you want to create a TAP stream and listen to all of the modifications that happens in your cluster. All you need to do would be:

struct event_base *evbase = event_init();

   libmembase_t instance = libmembase_create(host, username, passwd, bucket, evbase);
   libmembase_connect(instance);

   libmembase_tap_filter_t filter;
   libmembase_callback_t callbacks = {
      .tap_mutation = tap_mutation
   };
   libmembase_set_callbacks(instance, &callbacks);
   libmembase_tap_cluster(instance, filter, true);

Then you would implement the tap callback function as:

static void tap_mutation(libmembase_t instance, const void *key, size_t nkey, const void *data, size_t nbytes, uint32_t flags, uint32_t exp, const void *es, size_t nes)
{
   // Do whatever you want with the object
}

And thats all you need to do to tap your entire cluster :-) Let's extend the example to tap multiple buckets from the same code.

struct event_base *evbase = event_init();

   libmembase_t instance1 = libmembase_create(host, username, passwd, bucket1, evbase);
   libmembase_t instance2 = libmembase_create(host, username, passwd, bucket2, evbase);
   libmembase_connect(instance1);
   libmembase_connect(instance2);

   libmembase_tap_filter_t filter;
   libmembase_callback_t callbacks = {
      .tap_mutation = tap_mutation
   };
   libmembase_set_callbacks(instance1, &callbacks);
   libmembase_set_callbacks(instance2, &callbacks);
   libmembase_tap_cluster(instance1, filter, false);
   libmembase_tap_cluster(instance2, filter, false);

   event_base_loop(evbase, 0);

The instance handle is passed to the callback function so you should be able to tell which bucket each mutation event belongs to.

As I said all of the functions in the API is callback based, so if you want to retrieve an object you have to register a callback for get before calling libmembase_mget. Ex:

libmembase_callback_t callbacks = {
        .get = get_callback
    };
    libmembase_set_callbacks(instance, &callbacks);
    libmembase_mget(instance, num_keys, (const void * const *)keys, nkey);

    // If you don't want to run your own event loop, you can call the following method
    // that will run all spooled commands and wait for their replies before breaking out
    // of the event loop
    libmembase_execute(instance);

The signature for the get callback looks like:

void get_callback(libmembase_t instance, libmembase_error_t error, const void *key, size_t nkey, const void *bytes, size_t nbytes, uint32_t flags, uint64_t cas)
{
   // do whatever you want...
}

So what is missing from the library right now?

• Proper error handling. Right now I'm using asserts and abort() to handle error situations, causing your application to crash... you don't want that in production ;-)
• Timeouts.. Right now it will only time out on TCP timeouts../
• A lot of operations! I'm only supporting get/add/replace/set...
• Fetch replicas..
• Gracefully handle change in the vbucket list
• +++

Do you feel like hacking on some of them?