Building the PHP extension for Couchbase on Microsoft Windows!

I've got a pretty good background from doing cross platform work. My first job as a software developer was to port a system to Trusted Solaris (and add privilege bracketing), but I've been porting stuff to Windows, HP-UX, Linux etc to name a few later on. It is important to me that the software should be portable, and that was one of the fixed requirement I had when I started libcouchbase back in the days.

Two days ago I sat down with Matt planning what I should focus on for the SDK team, and we both felt that we had to give the Windows support for some of our projects a makeover. Given PHPs popularity I figured that I should start with that.

I was kind of surprised to see how easy it was to build PHP with support for Couchbase on windows, but relax; we are going to ship binary versions for you so that you won't have to do this yourself, but being a geek I thought you might find it interesting to read how to do it yourself.

The first thing you have to do is to get set up a development environment. I initially used the description from PHP wiki, with some minor modifications. Feel free to use the steps outlined in that link, but I'll this is however how I did it. First I installed the following software:

• Windows server 2008 r2
• Visual studio 2008 professional
• Windows SDK 6.1
• git source control system
• winrar

With that installed, you can open up a "windows sdk" shell, and get execute the following commands:

setenv /x86 /xp /release
mkdir c:\php-sdk

By using winrar I then extracted the php-sdk-binary-tools-20110915.zip into that directory. I could then create the directories and get ready to build the source.

cd c:\php-sdk
bin\phpsdk_setvars.bat
bin\phpsdk_buildtree.bat php-src

Start up a "git-bash shell", and navigate to c:\php-src\vc9\x86, and check out the code. Unfortunately this won't work as of today for you because not all of the patches for the source code have been pushed through yet:

git clone git://github.com/php/php-src
git clone git://github.com/couchbase/libcouchbase
cd php-src/ext
git clone git://github.com/couchbase/php-ext-couchbase couchbase

We're now ready to build the source! The first thing we need to do is to build libcouchbase (the C library that provides the functionality to talk to the Couchbase cluster):

cd libcouchbase
nmake -f NMakefile all install

With that installed we're ready to build the PHP module that expose the PHP functionality:

cd ..\php-src
buildconf
configure --disable-all --enable-cli --with-couchbase=shared
nmake all install
copy ../deps/bin/libcouchbase.dll c:\php

As you see I'm disabling all other modules than the Couchbase plugin (and I'm making that as a shared object). You would have to populate the "deps" directory with other libraries if you wanted to add support for them (but that's beyond the intention of this blog post).

So let's go ahead and try our new php extension. If you don't have a Couchbase cluster running already, now is the time to install one :) With the Couchbase cluster running, you can create a simple php.ini file that tells php to load our module:

extension=c:\php\php_couchbase.dll

I'm pretty sure you all know PHP way better than me, so please forgive me for the stupid code we'll use to show that it's working. I created the file test.php with the following content:

<?php
   $cb = new Couchbase("","","","default");
   $cb->set("hello", "world");
   var_dump($cb->get("hello"));
?>

Running the program with:

cd c:\php
php -f test.php

Should print out:

string(5) "world"

Thats all for this time! Happy hacking!

Shift of focus!

I really like going to conferences, not necessarily because I want to attend the different sessions, but because its a great arena to meet up with users and people interested in the same stuff as I am. Yesterday I attended CouchConf in Berlin, which was a lot of fun. I got to meet old friends, people I've only talked to on IRC (so fun to get a face behind the nick) and new people!

As part of attending CouchConf I spent some time talking to Matt Ingenthron about our SDKs, so I'm happy to announce that as of today I'm going to spend a significant amount of time working on our clients. Up until now I've done most of my contributions to the clients on my spare time, so I'm really looking forward to be able to spend my entire day trying to make your life easier.

YANFS - Yet Another Network File System

In the last few blog posts we've been exploring different kinds of how you might want to utilize libcouchbase from your app. To me it has always been really hard to come up with good example apps, so I always end up creating pretty boring applications.

Those of you who attended CouchConf in San Francisco really missed out if you didn't watch Couchbase Labs demo sessions, where they showed off CBFS. I'm not going to go into any details about what CBFS is (hopefully Dustin Sallings will create a blog post about that), but you can think cbfs as a large object store analogous to S3 (but meant to run in your own environment).

CBFS is implemented in the go language (if you haven't looked at that yet, put that on your todo list!), and it's shipped with it's own client to perform operations (upload files, list files etc). This gave me a great idea for todays blog post! FUSE is a project that allows you to create a filesystem implementation in user space, so the topic for today is an example where we utilize libcouchbase in our implementation for our FUSE driver.

As usual I won't explain the entire code, but I'll try to comment on the important bits. You'll find the source code for the entire project at http://github.com/trondn/mount_cbfs

The readme in the project should tell you how to install fuse and build the stuff, so in this blog post I'm only going to focus on how it works.

Creating a full featured filesystem driver is a lot of work, so I figured it would be better to start with a minimal implementation with a lot of limitations we could fix up later on (or I would never get around building it ;))

• It is fully single threaded. This should be pretty easy to fix up later on by using a pool of libcouchbase instances, no shared data etc..
• Read Only. This limits the number of entry points we need to implement.

So let's start looking at the code and how it all fits together. In main we register the struct containing all of the function pointers to the functions our filesystem support:

static struct fuse_operations cbfs_oper = {
    .getattr = cbfs_getattr,
    .open = cbfs_open,
    .read = cbfs_read,
    .readdir = cbfs_readdir,
};

This means that whenever someone tries to open a file in our filesystem, cbfs_open is called etc. So let's walk through an example here: What is the flow when you're typing "ls -l" in your shell.

The first things that happens is that cbfs_getattr is called. Its responsibility is to populate a stats struct with information about the file. Let's skip the details for now, and just assume that we return that this is a directory. Now cbfs_readdir is called with the directory to receive all of the entries in the directory, before cbfs_gettattr is called for every file to get the details of the files.

So how does our cbfs_readdir work like (after all this blog is about libcouchbase, not fuse ;)). CBFS use a http interface so we can request the list of files in a directory through the following URL:

http://cbfsserver:8484/.cbfs/list/path

It returns a JSON document looking something like:

{"files":{"file1":{}},"dirs":{},"path":"/foo"}

All we need to do is to decode the JSON and populate the information to FUSE. So how do we do this through libcouchbase? Through the http interface:

static lcb_error_t uri_execute_get(const char *uri, struct SizedBuffer *sb) {
    lcb_http_cmd_t cmd = {
        .version = 1,
        .v.v1 = {
            .path = uri,
            .npath = strlen(uri),
            .body = NULL,
            .nbody = 0,
            .method = LCB_HTTP_METHOD_GET,
            .chunked = 0,
            .content_type = "application/x-www-form-urlencoded",
            .host = cfg->cbfs_host,
            .username = cfg->cbfs_username,
            .password = cfg->cbfs_password
        }
    };

    return lcb_make_http_request(instance, sb, LCB_HTTP_TYPE_RAW, &cmd, NULL);
}

Since I'm using the synchronous interface to libcouchbase that call will block until we've received the response from the server. In my response handler I just copy whatever data the server returned to me:

static void complete_http_callback(lcb_http_request_t req, lcb_t instance,
                                   const void *cookie, lcb_error_t error,
                                   const lcb_http_resp_t *resp)
{
    struct SizedBuffer *sb = (void*)cookie;

    if (error == LCB_SUCCESS) {
        /* Allocate one byte extra for a zero term */
        sb->data = malloc(resp->v.v0.nbytes + 1);
        sb->size = resp->v.v0.nbytes;
        memcpy(sb->data, resp->v.v0.bytes, resp->v.v0.nbytes);
        sb->data[resp->v.v0.nbytes] = '\0';
    }
}

Both cbfs_read and cbfs_getattr utilize the same function from libcouchbase, but they're only hitting different URL's to get their data.

Adding supports for write operations should be no worse than finding the correct URL and do a PUT etc.

Happy Hacking :)

Extend a Couchbase Server with your own commands!

I was having a beer with Sergey after yesterday's CouchConf in San Francisco and we started discussing command extensions for memcached. I guess I haven't been very good at advertising about what I've been doing, because I added this to memcached a couple of years ago as part of the plugin extension framework.

I thought that it might be interesting to write up a small blog post that shows you how you may write your own commands to access a Couchbase cluster. Please note that what I'm going to show you in this blog post is abusing Couchbase Server, and nothing you should be doing in production. If you have special needs you should connect with us to let us work with you to make it happen (and ensure that you don't break stuff).

So lets imagine that you're storing relatively big objects with data into a cluster, and you only need fragments of each object at a time. The current SDK only allows you to read and write a complete item at a time, so we're going to extend our Couchbase cluster to be able to give us a fragment of the data at a time.

Luckily, I wrote this extension for memcached a couple of years ago, and it is actually bundled with the Couchbase Server. This does however not imply that it is a supported extension to be using from your Couchbase Server. So let's go ahead and enable the extension on the server:

# cd /opt/couchbase/bin
# mv memcached memcached.bin
# cat > memcached << ...
#! /bin/sh
exec `dirname $0`/memcached.bin \
            -X /opt/couchbase/lib/memcached/fragment_rw.so "$@"
...
# chmod +x memcached

If you are running a cluster you need to do this on all of your nodes (and you need to restart all of your nodes for them to load the module).

You don't learn anything unless we're walking through how the extension actually works. You'll find the module at https://github.com/trondn/memcached/tree/engine-pu/extensions/protocol in the files fragment_rw.c and fragment_rw.h.

So let's look into the details there. During startup memcached will load all shared objects specified with the -X parameter, and try to look up the function named memcached_extensions_initialize. This is where you should initialize your engine, and add it to the system. If you look at the signature for the method, you might wonder where you may specify the configuration. The "per engine" configuration is specified as part of the -X parameter. The fragment module allows you to specify the command id for read and write through "r" and "w", so we could write -X fragment.so,r=5,w=6 to tell it to use the command code 5 and 6 (please note that it would be a conflict with the existing commands ;))

Setting up a command extension happens in two different phases. The first thing you need to do is to registering your command descriptor:


server->extension->register_extension(EXTENSION_BINARY_PROTOCOL, &descriptor);

The descriptor for protocol extensions is pretty simple:

static EXTENSION_BINARY_PROTOCOL_DESCRIPTOR descriptor = {
    .get_name = get_name,
    .setup = setup
};

get_name is a function that just returns the name of the extension, whereas setup is the interesting function:


static void setup(void (*add)(EXTENSION_BINARY_PROTOCOL_DESCRIPTOR *descriptor,
                              uint8_t cmd,
                              BINARY_COMMAND_CALLBACK new_handler))
{
    add(&descriptor, read_command, handle_fragment_rw);
    add(&descriptor, write_command, handle_fragment_rw);
}

So what is happening here. The function is called with a parameter that is a function you may call in order to register handlers for various command codes. The first parameter is you should pass to this callback is the descriptor you registered. The second parameter is the command opcode you want to install a handler for, and the third parameter is the function that implements the logic.

So let's go ahead and look at how the function looks:

static ENGINE_ERROR_CODE handle_fragment_rw(EXTENSION_BINARY_PROTOCOL_DESCRIPTOR *descriptor,

                                            ENGINE_HANDLE* handle,

                                            const void* cookie,

                                            protocol_binary_request_header *request,

                                            ADD_RESPONSE response)

The descriptor is the descriptor you registered, the handle is the handle to the engine keeping all of the data (so that you may interact with the backing store). Cookie is a reference to the client who connected to you. Request is the complete package the client sent for the request, and response is a callback function you may call to send data back to the client. If you look at the implementation for handle_fragment_rw you'll see that it's not that hard to interact with the engine.

The only thing left for us now is to create create client functionality to connect to a server and send such a packet :) I leave that up to you folks for the moment (or if people would like me to post it here I'd be happy to do so. let me know).

Happy hacking!

libcouchbase is async, but my app isn't...

In the previous example I showed you how to hook libcouchbase into your own event loop, but not everyone wants to use it asynchronously. The best part of an asynchronous library is that it's pretty easy to make a synchronous library on top of it. All you need to do is to call the function and wait for it to complete, but in libcouchbase I went the extra mile. You may use it fully asynchronous, you may use it fully synchronous or you may use it somewhat in the middle.

The easiest way to use libcouchbase in a synchronous way is to toggle it's internals to the syncmode setting by calling:

lcb_behavior_set_syncmode(instance, LCB_SYNCHRONOUS);

One "drawback" with the syncmode setting is that you can't execute a chunk of commands concurrently. Lets look at the following example:

lcb_store(instance, NULL, 15, store_commands);
lcb_get(instance, NULL, 2, get_commands);

With syncmode enabled all of the store commands must have been executed and the response received before any of the get commands is sent (even if they don't hit the same servers). It would improve the latency of your application if you could shuffle as much data as possible over the network before blocking to wait for the result. That's when you want to set the syncmode to LCB_ASYNCHRONOUS (the default) and use lcb_wait() instead:

lcb_store(instance, NULL, 15, store_commands);
lcb_get(instance, NULL, 2, get_commands);
lcb_wait(instance);

when lcb_wait() returns we know that all of the above commands are executed (and the appropriate callbacks called).

So let's whip up a small program that shows you how to use the syncmode setting. To keep the example as small as possible I'm going to skip error recovery etc. Let's jump straight to the main() method:

int main(void)
{
    lcb_error_t error;
    lcb_t instance = create_instance();

    if ((error = lcb_connect(instance)) != LCB_SUCCESS) {
        fprintf(stderr, "Failed to connect to cluster: %s\n",
                lcb_strerror(instance, error));
        exit(EXIT_FAILURE);
    }

    set_key(instance);
    get_key(instance);

    lcb_destroy(instance);
    exit(EXIT_SUCCESS);
}

As you see, the program is pretty simple. First we create a libcouchbase instance and connect to the cluster; set and get a key before we clean up and exits.

Let's look inside create_instance:

static lcb_t create_instance(void)
{
    lcb_t instance;
    struct lcb_create_st copt;
    lcb_error_t error;

    memset(&copt, 0, sizeof(copt));
    copt.v.v0.host = "myserver:8091";

    if ((error = lcb_create(&instance, &copt)) != LCB_SUCCESS) {
        fprintf(stderr, "Failed to create libcuchbase instance: %s\n",
                lcb_strerror(NULL, error));
        exit(EXIT_FAILURE);
    }

    lcb_behavior_set_syncmode(instance, LCB_SYNCHRONOUS);
    lcb_set_error_callback(instance, error_handler);
    lcb_set_store_callback(instance, store_handler);
    lcb_set_get_callback(instance, get_handler);

    return instance;
}

This looks pretty much like how we normally do stuff except for the line I've marked as bold. From that line all of the lcb_ functions will block until we've received a reply from the server for the requested operation. You might be curious about how my handler functions looks like, and they are pretty boring:

static void error_handler(lcb_t instance, lcb_error_t err, const char *info)
{
    fprintf(stderr, "FATAL! an error occured: %s (%s)\n",
            lcb_strerror(instance, err), info ? info : "none");
    exit(EXIT_FAILURE);
}

static void store_handler(lcb_t instance, const void *cookie,
                          lcb_storage_t operation, lcb_error_t error,
                          const lcb_store_resp_t *resp)
{
    (void)cookie; (void)operation;

    if (error != LCB_SUCCESS) {
        fprintf(stderr, "Failed to store the key on the server: %s\n",
                lcb_strerror(instance, error));
        exit(EXIT_FAILURE);
    }

    if (resp->version == 0) {
        fprintf(stdout, "Successfully stored \"");
        fwrite(resp->v.v0.key, 1, resp->v.v0.nkey, stdout);
        fprintf(stdout, "\"\n");
    }
}

static void get_handler(lcb_t instance, const void *cookie,
                        lcb_error_t error, const lcb_get_resp_t *resp)
{
    (void)cookie;

    if (error != LCB_SUCCESS) {
        fprintf(stderr, "Failed to read the key from the server: %s\n",
                lcb_strerror(instance, error));
        exit(EXIT_FAILURE);
    }

    /* Validate that I read the correct key and value back */
    if (resp->version != 0) {
        fprintf(stderr,
                "WARNING: I don't support this version of libcouchbase\n");
        exit(EXIT_FAILURE);
    }

    fprintf(stdout, "I received \"");
    fwrite(resp->v.v0.key, 1, resp->v.v0.nkey, stdout);
    fprintf(stdout, "\" with the value: [");
    fwrite(resp->v.v0.bytes, 1, resp->v.v0.nbytes, stdout);
    fprintf(stdout, "]\n");
}

If we jump back to our main function the next instructions to execute would be:

    if ((error = lcb_connect(instance)) != LCB_SUCCESS) {
        fprintf(stderr, "Failed to connect to cluster: %s\n",
                lcb_strerror(instance, error));
        exit(EXIT_FAILURE);
    }

Now that we've enabled the LCB_SYNCHRONOUS mode in libcouchbase this will be a blocking call (I fixed that bug today Sept 17th), so that when the method returns we know the topology of the Couchbase cluster and we're about to store our object there in the set_key method that looks like:

static const char const *key = "mykey";
static const char const *value = "myvalue";

static void set_key(lcb_t instance)
{
    lcb_store_cmd_t cmd;
    lcb_error_t error;
    const lcb_store_cmd_t * const commands[] = { &cmd };

    memset(&cmd, 0, sizeof(cmd));
    cmd.v.v0.key = key;
    cmd.v.v0.nkey = strlen(key);
    cmd.v.v0.bytes = value;
    cmd.v.v0.nbytes = strlen(value);
    cmd.v.v0.operation = LCB_SET;

    if ((error = lcb_store(instance, NULL, 1, commands)) != LCB_SUCCESS) {
        fprintf(stderr, "Failed to store key: %s\n",
                lcb_strerror(instance, error));
        exit(EXIT_FAILURE);
    }
}

This looks exactly like how we would have done it in our asynchronous application, and here is a very important detail. The return code of the function is not the return code of the actual operation! It means the exact same thing as it used to: if we encountered any problems initiating the operation. It is the callback function that will tell you the result of the operation from the server!

To make the example complete let's look at the get_key function as well:

static void get_key(lcb_t instance)
{
    lcb_get_cmd_t cmd;
    lcb_error_t error;
    const lcb_get_cmd_t * const commands[] = { &cmd };

    memset(&cmd, 0, sizeof(cmd));
    cmd.v.v0.key = key;
    cmd.v.v0.nkey = strlen(key);

    if ((error = lcb_get(instance, NULL, 1, commands)) != LCB_SUCCESS) {
        fprintf(stderr, "Failed to get key: %s\n",
                lcb_strerror(instance, error));
        exit(EXIT_FAILURE);
    }
}

Pretty simple and straight forward.

Happy hacking!

How do I use libcouchbase with my own event loop?

Writing asynchronous programs seems to be popular these days, so I thought I should whip up an example that shows you how you may utilize libcouchbase with your own event loop.

libcouchbase performs it's IO through an "IO handle". This is a plugin system so you should be able to use whatever mechanism you want. Adding support for a new system is nothing more than implementing a handfull of function calls and place them in a shared object. When I wrote libcouchbase I created a version for libevent and a small version that use select for Windows just to ensure that it should be possible to write such plugins. Later Mark Nunberg wrote the plugin we're using for node.js that is based on libuv. I'm not going to cover how to write your own plugin in this blog post (perhaps I'll get around to do that at a later time), but I'll show you how to let libcouchbase utilize the same event base from libevent that you're using for something else.

To keep the example as short as possible I'm only going to do the very basic error handling: print out an error message and terminate the program. This means that we can start writing our first function; our error handler:

static void error_callback(lcb_t instance, lcb_error_t error, const char *errinfo) {
    fprintf(stderr, "ERROR: %s %s\n", lcb_strerror(instance, error), errinfo);
    exit(EXIT_FAILURE);
}

I'm not very good at coming up with interesting examples, so today we'll just create a small program that connects to a Couchbase cluster, stores a key and then reads the key back out again.

So let's take a look at our main function:

int main(int argc, char** argv) {
    struct event_base *evbase = event_base_new();

    if (create_libcouchbase_handle(evbase) == -1) {
        exit(EXIT_FAILURE);
    }

    event_base_loop(evbase, 0);
    event_base_free(evbase);
    exit(EXIT_SUCCESS);
}

As you see there is nothing fancy there.. We're creating the event base that the rest of our application should use (I just don't use it for anything else in this example to make the example simple), before we create our libcouchbase handle and associate it with the newly created event base (we'll look at that shortly). We then start the event loop that will run the rest of the example through the callbacks.

So how does the create_libcouchbase_handle function look like:

static int create_libcouchbase_handle(struct event_base *evbase) {
    struct lcb_create_io_ops_st ciops;

    memset(&ciops, 0, sizeof (ciops));
    ciops.v.v0.type = LCB_IO_OPS_LIBEVENT;
    ciops.v.v0.cookie = evbase;

    lcb_io_opt_t ioops;
    lcb_error_t error = lcb_create_io_ops(&ioops, &ciops);
    if (error != LCB_SUCCESS) {
        fprintf(stderr, "Failed to create an IOOPS structure for libevent: %s\n",
                lcb_strerror(NULL, error));
        return -1;
    }

Let's stop there for a moment and look what we're doing. The first thing we're doing is that we're creating an instance of the lcb_create_io_ops_st structure. Like the rest of libcouchbase we're using a versioned struct for creating objects. The "constructor" for the IO handle on top of libevent allows us to pass the event base to use as the cookie.

We can now go ahead and create the libcouchbase instance "the normal way, with the only difference that we specify the io member in the create structure.:

    struct lcb_create_st copts;
    memset(&copts, 0, sizeof (copts));
    copts.v.v0.host = "localhost:8091";
    copts.v.v0.user = "Administrator";
    copts.v.v0.passwd = "secret";
    copts.v.v0.bucket = "default";
    copts.v.v0.io = ioops;

    lcb_t instance;
    if ((error = lcb_create(&instance, &copts)) != LCB_SUCCESS) {
        fprintf(stderr, "Failed to create a libcouchbase instance: %s\n",
                lcb_strerror(NULL, error));
        return -1;
    }

The next thing we'll do is to set up the different callbacks we're going to use in our program and call lcb_connect to initiate the connect sequence:

    lcb_set_error_callback(instance, error_callback);
    lcb_set_configuration_callback(instance, configuration_callback);
    lcb_set_get_callback(instance, get_callback);
    lcb_set_store_callback(instance, store_callback);

    if ((error = lcb_connect(instance)) != LCB_SUCCESS) {
        fprintf(stderr, "Failed to connect libcouchbase instance: %s\n",
                lcb_strerror(NULL, error));
        lcb_destroy(instance);
        return -1;
    }
}

You might be curious why I'm adding a "configuration callback"? This is actually a small trick :-) You might have tried to use libcouchbase yourself and had problems that your operations failed because you forgot to do a lcb_wait() after calling connect. The thing is that libcouchbase needs to know the topology of your cluster before it may perform any operations, and it cannot do that until it receives the first configuration from the server.

So how does this configuration callback look like in our example:

static void configuration_callback(lcb_t instance, lcb_configuration_t config) {
    if (config == LCB_CONFIGURATION_NEW) {
        // Since we've got our configuration, let's go ahead and store a value
        lcb_store_cmd_t cmd;
        const lcb_store_cmd_t * cmds[] = {&cmd};
        memset(&cmd, 0, sizeof (cmd));
        cmd.v.v0.key = "foo";
        cmd.v.v0.nkey = 3;
        cmd.v.v0.bytes = "bar";
        cmd.v.v0.nbytes = 3;
        cmd.v.v0.operation = LCB_SET;
        lcb_error_t err = lcb_store(instance, NULL, 1, cmds);
        if (err != LCB_SUCCESS) {
            fprintf(stderr, "Failed to set up store request: %s\n",
                    lcb_strerror(instance, err));
            exit(EXIT_FAILURE);
        }
    }
}

As the comment tells you, we've received the configuration from the server. This means that it's safe to start using the library. In a real world application you would probably have a more advanced logic here (please note that this callback will be called if you add/remove nodes etc, so it's not safe to assume that it will be called only once!)

When we receive the result for the store command from the server, our store_callback is called:

static void store_callback(lcb_t instance, const void *cookie, lcb_storage_t operation, lcb_error_t error, const lcb_store_resp_t *resp) {
    if (error != LCB_SUCCESS) {
        fprintf(stderr, "Failed to store key: %s\n",
                lcb_strerror(instance, error));
        exit(EXIT_FAILURE);
    }

    /* Time to read it back */
    lcb_get_cmd_t cmd;
    const lcb_get_cmd_t * cmds[] = {&cmd};
    memset(&cmd, 0, sizeof (cmd));
    cmd.v.v0.key = "foo";
    cmd.v.v0.nkey = 3;
    if ((error = lcb_get(instance, NULL, 1, cmds)) != LCB_SUCCESS) {
        fprintf(stderr, "Failed to setup get request: %s\n",
                lcb_strerror(instance, error));
        exit(EXIT_FAILURE);
    }
}

If the value was successfully stored on the server, we're issuing a single get command to the server to verify that it's there. When we receive the get result from the server, our get_callback is called and we're terminating the program:

static void get_callback(lcb_t instance, const void *cookie, lcb_error_t error, const lcb_get_resp_t *resp) {
    if (error != LCB_SUCCESS) {
        fprintf(stderr, "Failed to get key: %s\n",
                lcb_strerror(instance, error));
        exit(EXIT_FAILURE);
    }

    fprintf(stdout, "I stored and retrieved the key \"foo\". Terminate program");
    exit(EXIT_SUCCESS);
}

This wasn't the most exciting example, but it shows you the basics on how you may utilize libcouchbase with your own event loop. None of the above functions will cause your application to block waiting for socket IO. If that happens I'd be more than happy to fix it if you send me a bug report (unless you wrote your own IO plugin and forgot to enable nonblocking IO ;)

Happy hacking!

How do I utilize Couchbase from my node.js app?

In yesterdays post I told you that I had created an npm for our Couchbase driver, but I didn't give any examples on how you could utilize it so I figured I should create a short blog post about that. I'm the kind of guy who like real world examples instead of a lot of text, so I went ahead and created a small application for this purpose.

I didn't have any good ideas about what I could create, so I decided I should create a version of the vacuum example I did for libcouchbase a while back. The application is a small daemon that will move all JSON documents you store in a given directory into a Couchbase server by using the _id field in the JSON document as the key, and the entire file as the value. Its a pretty simple example, but should show us some of the concepts. You'll find the entire program (including installation notes) at https://github.com/trondn/vacuum.js.

I won't be going through the entire program, but only mention the specifics needed to talk to Couchbase.

The first thing we need to do is to create an instance of the Couchbase driver:

var couchnode = require('couchbase');
var cb = new couchnode.Couchbase(config.hostname,
    config.username,
    config.password,
    config.bucket);

Here we pass the address of the couchbase cluster (normally: hostname:8091), the username to log in with (normally the name of the bucket) with the corresponding password and the bucket we want to connect to. Unlike the C version we may start to use the instance immediately from JavaScript, and it will buffer all of our operations until it's connected.

If you look in the function process_file you'll find the first usage of the Couchbase instance:

cb.set(obj._id, String(data), 0, undefined, set_handler, fullname);

So what happens here? We try to store an object in Couchbase with the key being the content of the "_id" field in the JSON document. The current version of the driver only allows you to store Strings, so we need to create a String of the data. The next field is the expiration time of the object (0 means that the object will never expire). The undefined parameter is for the CAS field, which would allow us to say that we would only set this parameter if the object already exist in Couchbase with that identifier. The set_handler is the callback function when the operation is finished, and fullname is the absolute name of the file we just stored. But wait, why would the Couchbase driver need the absolute name of the file we just stored? The answer to that is that it doesn't. It is just a data field get back in the callback.

So let's look at the callback:

function set_handler(data, error, key, cas) {
    if (error) {
        console.log('Failed to store object: %s', key);
    } else {
        fs.unlink(data);
        process_next();
    }
}

It looks pretty simple? The first parameter is the "fullname" field we passed to the set parameter. It contains the absolute name of the file we wanted to store. The error parameter tells you if the operation succeeded or not. The key is the key we wanted to store, and the last field is the CAS identifier the server assigned to the object (which you may use in successive calls to set if you want to replace this exact version of the object.


Happy hacking!!!!