Monday, October 4, 2010

Writing your own storage engine for Memcached

I am working full time on membase, which utilize the "engine interface" we're adding to Memcached. Being the one who designed the API and wrote the documentation, I can say that we do need more (and better) documentation without insulting anyone. This blog entry will be the first entry in mini-tutorial on how to write your own storage engine. I will try to cover all aspects of the engine interface while we're building an engine that stores all of the keys on files on the server.

This entry will cover the basic steps of setting up your development environment and cover the lifecycle of the engine.

Set up the development environment

The easiest way to get "up'n'running" is to install my development branch of the engine interface. Just execute the following commands:

$ git clone git://github.com/trondn/memcached.git
$ cd memcached
$ git -b engine origin/engine
$ ./config/autorun.sh
$ ./configure --prefix=/opt/memcached
$ make all install
     

Lets verify that the server works by executing the following commands:

$ /opt/memcached/bin/memcached -E default_engine.so &
$ echo version | nc localhost 11211
VERSION 1.3.3_433_g82fb476     ≶-- you may get another output string....
$ fg
$ ctrl-C
     

Creating the filesystem engine

You might want to use autoconf to build your engine, but setting up autoconf is way beyond the scope of this tutorial. Let's just use the following Makefile instead.

ROOT=/opt/memcached
INCLUDE=-I${ROOT}/include

#CC = gcc
#CFLAGS=-std=gnu99 -g -DNDEBUG -fno-strict-aliasing -Wall \
# -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations \
# -Wredundant-decls \
# ${INCLUDE} -DHAVE_CONFIG_H
#LDFLAGS=-shared

CC=cc
CFLAGS=-I${ROOT}/include -m64 -xldscope=hidden -mt -g \
      -errfmt=error -errwarn -errshort=tags  -KPIC
LDFLAGS=-G -z defs -m64 -mt

all: .libs/fs_engine.so

install: all
 ${CP} .libs/fs_engine.so ${ROOT}/lib

SRC = fs_engine.c
OBJS = ${SRC:%.c=.libs/%.o}

.libs/fs_engine.so: .libs $(OBJS)
 ${LINK.c} -o $@ ${OBJS}

.libs:; -@mkdir $@

.libs/%.o: %.c
 ${COMPILE.c} $< -o $@   clean:  $(RM) .libs/fs_engine.so $(OBJS)       

I am doing most of my development on Solaris using the Sun Studio compilers, but I have added a section with settings for gcc there if you're using gcc. Just comment out lines for CC, CFLAGS and LDFLAGS and remove the # for the gcc alternatives.

In order for memcached to utilize your storage engine it needs to first load your module, and then create an instance the engine. You use the -E option to memcached to specify the name of the module memcached should load. With the module loaded memcached will look for a symbol named create_instance in the module to create an handle memcached can use to communicate with the engine. This is the first function we need to create, and it should have the following signature:

MEMCACHED_PUBLIC_API
ENGINE_ERROR_CODE create_instance(uint64_t interface, GET_SERVER_API get_server_api, ENGINE_HANDLE **handle);
     

The purpose of this function is to provide the server a handle to our module, but we should not perform any kind of initialization of our engine yet. The reason for that is because the memcached server may not support the version of the API we provide. The intention is that the server should notify the engine with the "highest" interface version it supports through interface, and the engine must return a descriptor to one of those interfaces through the handle. If the engine don't support any of those interfaces it should return ENGINE_ENOTSUP.

So let's go ahead and define a engine descriptor for our example engine and create an implementation for create_instance:

struct fs_engine {
  ENGINE_HANDLE_V1 engine;
  /* We're going to extend this structure later on */
};

MEMCACHED_PUBLIC_API
ENGINE_ERROR_CODE create_instance(uint64_t interface,
                                 GET_SERVER_API get_server_api,
                                 ENGINE_HANDLE **handle) {
  /*
   * Verify that the interface from the server is one we support. Right now
   * there is only one interface, so we would accept all of them (and it would
   * be up to the server to refuse us... I'm adding the test here so you
   * get the picture..
   */
  if (interface == 0) {
     return ENGINE_ENOTSUP;
  }

  /*
   * Allocate memory for the engine descriptor. I'm no big fan of using
   * global variables, because that might create problems later on if
   * we later on decide to create multiple instances of the same engine.
   * Better to be on the safe side from day one...
   */
  struct fs_engine *h = calloc(1, sizeof(*h));
  if (h == NULL) {
     return ENGINE_ENOMEM;
  }

  /*
   * We're going to implement the first version of the engine API, so
   * we need to inform the memcached core what kind of structure it should
   * expect
   */
  h->engine.interface.interface = 1;

  /*
   * Map the API entry points to our functions that implement them.
   */
  h->engine.initialize = fs_initialize;
  h->engine.destroy = fs_destroy;

  /* Pass the handle back to the core */
  *handle = (ENGINE_HANDLE*)h;

  return ENGINE_SUCCESS;
}
     

If the interface we provide in create_instance is dropped from the supported interfaces in memcached, the core will call destroy() immediately. The memcached core guarantees that it will never use any pointers returned from the engine when destroy() is called.

So let's go ahead and implement our destroy() function. If you look at our implementation of create_instance you will see that we mapped destroy() to a function named fs_destroy():

static void fs_destroy(ENGINE_HANDLE* handle) {
  /* Release the memory allocated for the engine descriptor */
  free(handle);
}
     

If the core implements the interface we specify, the core will call a the initialize() method. This is the time where you should do all sort of initialization in your engine (like connecting to a database, initializing mutexes etc). The initialize function is called only once per instance returned from create_instance (even if the memcached core use multiple threads). The core will not call any other functions in the api before the initialization method returns.

We don't need any kind of initialization at this moment, so we can use the following initialization code:

static ENGINE_ERROR_CODE fs_initialize(ENGINE_HANDLE* handle,
                                      const char* config_str) {
  return ENGINE_SUCCESS;
}
     

If the engine returns anything else than ENGINE_SUCCESS, the memcached core will refuse to use the engine and call destroy()

In the next blog entry we will start adding functionality so that we can load our engine and handle commands from the client.

5 comments:

  1. Hello Trondn,

    My name is Rain , and I come from Taiwan.

    I just read the Memcached for a week, and have some questions about memcached...

    Could you help me to answer this questions?

    In the Gear6 web site, it compare with Memcached & Gear6 Web Cache features..
    and Memcached V1.2.x is already finished 30 to 40% in the feature of "Dynamic Slab Allocation" , (the hyperlink: http://www.gear6.com/memcached-product/compare-web-cache-memcached ) I don't understand the 30-40% means what.. Can you try to explain that 30-40% Dynamic Slab Allocation ?

    Will Memcached Team finish the "Dynamic Slab Allocation" function in the future?
    How long is expected to be completed?

    I sended the same mail to your github mailbox..I'm so sorry about this..

    I appreciate your help...

    ReplyDelete
  2. You would have to ask Gear6 about their maketing info. It is possible to "tweak" the current slabber so that it fits your item size better (look at the man page), and we will improve the slabber inside memcached (we just don't know when yet)

    ReplyDelete
  3. Hi Trond, we've been using Memcached for a while and recently started testing Membase in production. We're testing a single instance of Membase 1.6.0 with 5GB RAM, 750GB disk. We have an issue with the current storage manager which is why I'm asking this question here.

    We've noticed that SQLite seems to block on eviction purges on an hourly basis when expiryPagerSleeptime wakes up.

    Although it's clear that SQLite locks the database when writes occur, it was unanticipated that Membase as a whole would appear to completely block. It seems that SQLite is deleting old keys, Membase operations / sec falls to near zero for several minutes. After eviction has finished, the Membase server quickly recovers. I would have anticipated that reads from Membase RAM would still proceed while SQLite was locked.

    I would appreciate your thoughts. Is there a different storage engine that we should use instead? Are there any recommendations that you would make to prevent Membase from blocking on evictions?

    Thank you.

    ReplyDelete
  4. openid: Hi, Please post your question in the forums on http://forums.membase.org/ That makes it easier to keep track of the question, and for others who might have the same question to get the answer :)

    Cheers,

    Trond

    ReplyDelete
  5. Done: http://forums.membase.org/thread/membase-blocking-key-eviction

    Thanks again...

    ReplyDelete