Friday, January 27, 2012

So whats the story about libcouchbase and Windows?

A couple of days ago I showed you an example program using libcouchbase to create a small application to put data into a Couchbase cluster, but the code wouldn't compile on Windows. That does by no means imply that libcouchbase doesn't work on Windows, its more that I was in a hurry writing the blog post so I didn't have the time fixing everything up in time for the blog post.

In this blog post I'll show you how easy it is to get everything up'n'running using Windows 7 and Microsoft Visual Studio 2010. In addition to that you need to download and install git to be able to check out the source code (select the option that you want to put git in the path (not the full msys suite, but just git)).

I have to admit that I am far from a "hardcore Windows developer", so there is a lot of things I don't know about the platform. For instance I don't know where I should install third party header files and libraries, so I just decided that I'm going to install all of them into C:\local (with an install, lib and bin directory). I'd be happy if someone could tell me how I'm supposed to do this ;-)

So let's open up the Visual Studio Command Prompt and start building everything:

Setting environment for using Microsoft Visual Studio 2010 x86 tools.
C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC> cd %HOMEPATH%
C:\Users\Trond> mkdir build
C:\Users\Trond> cd build

Since we're going to build dll's you need to set C:\local\bin into your path so that the runtime linker finds the dll's:

C:\Users\Trond\build> set PATH=c:\local\bin;%PATH%

We need to install two dependencies before we can compile libcouchbase itself. Let's check out all of the source code we're going to use:

C:\Users\Trond\build> git clone git://
C:\Users\Trond\build> git clone git://
C:\Users\Trond\build> git clone git://
C:\Users\Trond\build> git clone git://
C:\Users\Trond\build> git clone git://

The first dependency we're going to build is the SASL library. This is the library libcouchbase use for authenticating to the Couchbase servers. To build and install the library, simply execute:

C:\Users\Trond\build> cd libisasl
C:\Users\Trond Norbye\build\libisasl> nmake -f NMakefile install

That will install libisasl with its header files and libraries into c:\local.

The next library we need to build is libvbucket; the library libcouchbase use to figure out where a a vbucket is located (if you don't know what a vbucket is, you don't really need to know). It is just as easy as libvbucket to build:

C:\Users\Trond\build\libisasl> cd ..\libvbucket
C:\Users\Trond\build\libvbucket> nmake -f NMakefile install

The next thing we need to do is to install some headerfiles libcouchbase needs during build time. These header files contains the protocol definitions libcouchbase needs (but it is not needed by the application). So let's go ahead and install them (to make it easier for us to build libcouchbase)

C:\Users\Trond\build\libvbucket> cd ..\memcached
C:\Users\Trond\build\memcached> git checkout -b branch-20 origin/branch-20
C:\Users\Trond\build\memcached> mkdir c:\local\include\memcached
C:\Users\Trond\build\memcached> copy include\memcached c:\local\include\memcached

So let's go ahead and build libcouchbase! 

C:\Users\Trond\build\memcached> cd ..\libcouchbase
C:\Users\Trond\build\libcouchbase> nmake -f NMakefile install

I guess that most Windows developers don't use nmake during their development, but use the full IDE instead. That's why I've created a project you may open in the vacuum project. So feel free to open that project now, and it should build without any problems. 

Now we're going to need a Couchbase server we can connect to. If you don't have any running, you should download and install one now. 

Let's go ahead and create the spool directory and start the vacuum server...

C:\Users\Trond\build\vacuum\Debug> mkdir c:\vacuum
C:\Users\Trond\build\vacuum\Debug> vacuum -h

And you can start copy JSON files into C:\vacuum and see them being added to the Couchbase cluster!

Tuesday, January 24, 2012

So how do I use this "libcouchbase"?

Some of you may have noticed that we released Couchbase 1.8 earlier today, and a new set of smart clients for various languages. For me personally this is a milestone, because libcouchbase is now a
supported client for the C language.

So why do I care about that? Well, libcouchbase started out of my needs to easily test various components of the server. Since I did most of my development on the components on the server implemented in C, it made sense for me to use C for my testing.

I've received some questions on how libcouchbase work in a multithreaded context, so I should probably start off by clarifying that: libcouchbase doesn't use any form of locking to protect it's internal data structures, but it doesn't mean you can't use libcouchbase in a multithreaded program. All it means is that you as a client user must either use locking to protect yourself from accessing the libcouchbase instance from multiple threads at the same time, or just let each thread operate on it's own instance of libcouchbase. One easy way to solve this is to have a "pool" of libcouchbase instances each thread pop and push its instance to whenever they need to access a Couchbase server. Access to this pool should be protected with a lock (but I guess you figured that out ;-)

In this blog post I'll create a demo program you may use to upload JSON documents into a Couchbase server. You'll find the complete source available at if you would like
to try the example.

The idea of this program is that it will "monitor" a directory and upload all files appearing there into a Couchbase cluster. I'm pretty sure most of you start thinking: "how do we do that in a portable way?". That's not an easy task to do, so I'm not even going to try to do that. I'll try to write it in a semi-portable way so that it shouldn't be that hard to implement on other platforms. That means that I'm using the following limitations:

  • I'm using opendir and readdir to traverse the directory. This can easily be reimplemented with FindFirst and FindNext on Microsoft Windows.
  • Monitor of the directory means that I'm going to scan the directory, then sleep a given number of seconds before running another scan. I know some platforms supports subscribing of changes to the filesystem, but I'm not going to spend time on that (at least not right now ;-)).
  • To avoid file locking or accessing the file while others are writing the file, the clients should write the file into the directory with a leading "dot" in the filename, and then rename the file when they are done. The program ignores all files starting with a dot.

So let's jump to the code. The first piece of code that might be interesting to look at would be where we create the libcouchbase instance in main():

    instance = libcouchbase_create(host, user, passwd, bucket, NULL);
    if (instance == NULL) {
        fprintf(stderr, "Failed to create couchbase instance\n");

The above code snippet creates the libcouchbase instance. There is no way you can use a static structure for this, because doing so will make it incredible hard to maintain binary compatibility. I like to be able to fix bugs within the library and release new versions you may use without having to recompile your program, and by hiding the internal datastructures from the clients makes it easier to ensure that the client don't depend on their size. The first parameter to libcouchbase_create is the name (and port) of the REST port for the couchbase server (default: localhost:8091). The second and third parameter is the credentials you'd like to use to connect to the REST port to get the pool information (default is to not authenticate). The forth parameter is the bucket you'd like to connect to, and if you don't specify a bucket you'll end up in the "default bucket". The fifth argument is a special object you may want to use if you are going to use "advanced" features in libcouchbase. Most users will probably just use the defaults and pass NULL here.

The next thing we need to do is to set up some callback handlers to be able to figure out what happens. In the example we're only going to use one operation (to load data into the cache) so we'll need to set up a handler to catch the result of storage operations. Unfortunately we may also encounter problems, so we need to set up an error handler (we'll get back to work in a bit).

    libcouchbase_set_storage_callback(instance, storage_callback);
    libcouchbase_set_error_callback(instance, error_callback);

Now that we've created and initialized the instance, we need to try to connect to the Couchbase cluster:

    libcouchbase_error_t ret = libcouchbase_connect(instance);
    if (ret != LIBCOUCHBASE_SUCCESS) {
        fprintf(stderr, "Failed to connect: %s\n",
                libcouchbase_strerror(instance, ret));

Due to the fact that libcouchbase is fully asynchronous, all that happened above was that we initiated the connect. That means that we need to wait for the server to be connected to the Couchbase cluster and connect to the correct bucket. If our program should do other stuff now would be the time to do so, but since we don't have any other initialization to do we can just wait for it to complete:


One of the "cool" features we've got in libcouchbase is that it provides an internal statistics interface, so we may tell it to collect timing information of the operations with the following snippet:

   if ((ret = libcouchbase_enable_timings(instance) != LIBCOUCHBASE_SUCCESS)) {
      fprintf(stderr, "Failed to enable timings: %s\n",
              libcouchbase_strerror(instance, ret));

Our program is now fully initialized, and we can enter the main loop that looks like pretty much like:

   while (forever)

So how does our process_files() look like? I'm not going to make the example too big by pasting all of it, but the first piece in there looks like:

   if (de->d_name[0] == '.') {
       if (strcmp(de->d_name, ".dump_stats") == 0) {
           fprintf(stdout, "Dumping stats:\n");
           libcouchbase_get_timings(instance, stdout, timings_callback);
           fprintf(stdout, "----\n");

As you see from the above code snippet we'll ignore all files that starts with a '.' except for the file named ".dump_stats". Whenever we see that file we dump the internal stats timings by using the timings_callback (I'll get back to that later).

The next thing we do is to try to read the file into memory and decode it's JSON before we try to get the "_id" field to use as a key. If all of that succeeds, we try to store the data in Coucbase with:

      int error = 0;
      ret = libcouchbase_store(instance, &error, LIBCOUCHBASE_SET,
                               id->valuestring, strlen(id->valuestring),
                               ptr, size, 0, 0, 0);
      if (ret == LIBCOUCHBASE_SUCCESS) {
      } else {
         error = 1;

The &error piece here is quite interesting. It is a "cookie" passed to the callback, so that I may know if I encountered a problem or not. You'll see how I'm using it when I discuss the storage_callback below.

This is basically all of the important logic in the example. I promised that I would get back to the different callbacks, so let's start by looking at the error callback:

   static void error_callback(libcouchbase_t instance,
                              libcouchbase_error_t error,
                              const char *errinfo)
       /* Ignore timeouts... */
       if (error != LIBCOUCHBASE_ETIMEDOUT) {
           fprintf(stderr, "\rFATAL ERROR: %s\n",
                   libcouchbase_strerror(instance, error));
           if (errinfo && strlen(errinfo) != 0) {
               fprintf(stderr, "\t\"%s\"\n", errinfo);

As you see from the above snippet libcouchbase will call the error_callback whenever a timeout occurs, but we just want to retry the operation. If we encounter a real error we print out an error message and terminate the program.

The next callback we use is the storage_callback. It is called when the store operation completed, so it is the right place for us to figure out if an error occured while storing the data. Our callback looks like:

   static void storage_callback(libcouchbase_t instance,
                                const void *cookie,
                                libcouchbase_storage_t operation,
                                libcouchbase_error_t err,
                                const void *key, size_t nkey,
                                uint64_t cas)
      int *error = (void*)cookie;
       if (err == LIBCOUCHBASE_SUCCESS) {
           *error = 0;
       } else {
           *error = 1;
           fprintf(stderr, "Failed to store \"");
           fwrite(key, 1, nkey, stderr);
           fprintf(stderr, "\": %s\n",
                   libcouchbase_strerror(instance, err));

As you see we're storing the result of the operation in the integer passed as the cookie. The observant reader may see that we might as well could unlink the file and remove the memory from within the callback (if we provided that information as the cookie instead ;))

The last callback to cover is the timings callback we're using to dump out the timing statistics.

   static void timings_callback(libcouchbase_t instance, const void *cookie,
                                libcouchbase_timeunit_t timeunit,
                                uint32_t min, uint32_t max,
                                uint32_t total, uint32_t maxtotal)
      char buffer[1024];
      int offset = sprintf(buffer, "[%3u - %3u]", min, max);
      switch (timeunit) {
         offset += sprintf(buffer + offset, "ns");
         offset += sprintf(buffer + offset, "us");
         offset += sprintf(buffer + offset, "ms");
         offset += sprintf(buffer + offset, "s");

      int num = (float)40.0 * (float)total / (float)maxtotal;
      offset += sprintf(buffer + offset, " |");
      for (int ii = 0; ii < num; ++ii) {
         offset += sprintf(buffer + offset, "#");

      offset += sprintf(buffer + offset, " - %u\n", total);
      fputs(buffer, (FILE*)cookie);

When you request the timings from libcouchbase it reports all of the timing metrics collected by calling the timings callback. As you can see from the API you'll get the minimum, maximum value for the range, and the number of operations performed within that range. These metrics are not to be considered as exact numbers, because they depend on when what you do in your client code from the time you call the operation until you call libcouchbase_wait for the operation to complete.

So let's run the go ahead and run the program. I've prepopulated /var/spool/vacuum with a number of JSON files, to have the program do something.

trond@illumos ~> ./vacuum
sleeping 3 secs before retry..

From another withdow I execute the command:

trond@illumos ~> touch /var/spool/vacuum/.dump_stats

And when the timer expires in first window, it prints out:

Dumping stats:
[ 60 -  69]us |######################################## - 18
[ 70 -  79]us |## - 1
[240 - 249]us |## - 1
sleeping 3 secs before retry..

Hopefully this blog revealed how easy it is to use libcouchbase to communicate with a Couchbase cluster. We've got various clients for other programming languages like PHP and Ruby built on top of libcouchbase, so I can promise you that you'll see more functionallity added!

Thursday, January 12, 2012

Couchbase Server meets SmartOS!

I've loved Solaris since we first met back in 95. Since then I've been using Solaris as my primary os (including desktop). For a period of time I even ran Trusted Solaris 2.5 on my SS5 ;-) The kernel and all of the fantastic tools available makes it a superior platform for software development.

Ever since I started working for NorthScale (now Couchbase) I've done my very best to ensure that our stuff works on Solaris (if not I'd have a hard time doing any development ;), so getting it to work on SmartOS shouldn't be a big problem.

So far I've only been using the Sun Studio tools (I can't help it but I still find dbx superior to gdb...), but a few days ago I added a slave running SmartOS to my Jenkins cluster to see how much work it would be to get everything built with the tools and libraries I could install with pkgin. We do compile our software on different platforms and with different compilers, so I didn't expect too much trouble.

The "biggest" problem I had was to get libtool to stop trying to link 32bit object files into my 64 bit binaries. The workaround so far is to use:
CXX="g++ -m64 -L/opt/local/lib/amd64"
CC="gcc -m64 -L/opt/local/lib/amd64"

This was a small goal for me, but it will really ease my testing of a full cluster :-)

Now that I've got everything built on SmartOS, OpenIndiana, Soalris 10 I guess I should create packages for pkgin, IPS and SVr4 :)

Monday, January 9, 2012

Thank you Joyent, I love you!

I’ve always been a strong believer of that people should be able to choose the platform they feel suits their needs the most. If people want to make stupid decisions and not choose my beloved Solaris, I’m not going to stop them. This means that I need to ensure that the software I’m working on not only compiles, but also works on multiple platforms. In “the old days” I used to install a new os on my desktop box every time I upgraded my desktop (which always ran the latest bits of Solaris), but lately I’ve been using virtual machines to make it easier for myself ;)

You may wonder why I don’t just set up virtual machines in the cloud? The answer is pretty simple. I’m working from home, and I’ve had my share of problems with my ISP. I don’t want to end up in a situation where I can’t do my work just because my ISP fails to keep me connected with the rest of the world...
I guess it must be roughly a year ago since I decided to replace all of my VirtualBox instances running on a handful of old machines with KVM on top of Debian on my server. With the server running there I moved more and more stuff off my desktop box, and I ended up in a situation I really didn’t like. I had the services I “needed” running on top of a server without mirrored disks. To save money I had ordered that dell server with just a single disk. I’ve had enough disks dying on me over the years, so to me this feels like hiking in the middle of the freeway at midnight.. To get out of this situation I created an iSCSI share on my OpenIndiana box on top of a ZFS mirror that I connected to the Debian box. Now I could sleep a little bit better at night...

I had noticed the releases of SmartOS, and it looked really cool! Unfortunately for me I didn’t have a machine I could try it on, but luckily for me Debian gave me a helping hand in December to speed up the process! After upgrading packages with aptitude my box would no longer boot! I don’t have a keyboard/monitor attached to the box, so I had to bring the server into my office to “debug” the issue. I didn’t figure out why it wasn’t booting with the new kernel, but I was able to boot it with the old kernel and get the stuff I needed off the box. I could have spent more time trying to figure out why it was failing, but instead I took this as a golden opportunity to try out SmartOS. Doing so brought nothing but joy into my life!!!!!

After booting off the USB stick I had to answer a couple of questions to configure the system for the first time, and I was ready to create my first machine. I followed the instructions in "How to create a Virtual Machine in SmartOS" and I had my first VM up’n running in less than a minute! So simple, and yet so powerful!

Roughly a week later my box looks like:
[root@00-26-b9-85-bd-92 ~]# vmadm list
UUID                                  TYPE  RAM      STATE             ALIAS     
80658f56-0eb3-405f-a6eb-690461c2d9ce  OS    256      running           -       
9c441211-bb77-446f-abec-2291039aeca2  OS    256      running           smartos64 
b0e34876-cd7a-4922-ad6e-921452d34359  OS    512      running           jenkins   
13f4223b-a2c1-400a-b682-79372c3ba846  KVM   1024     running           solaris11 
7bad78f1-d202-4dfe-97f6-e421e8da8d58  KVM   1024     running           ubuntu64  
8e042001-bc79-48eb-a0bc-704ca64f20e0  KVM   1024     running           debian    
e1710b31-0270-43a3-89dc-71398ba3630a  KVM   1024     running           windows  
f165261a-6a27-4ee3-a6d3-f3814cf69bd6  KVM   1024     running           ubuntu32  

My "jenkins" vm is just running the Jenkins CI web application for, and it connects via ssh into the other VMs (and a couple of other machines) to build software there. This is part of my automatic build process. I've got it on my ever growing todo list to set up NIS/LDAP, but in the mean time I'm just sync'ing the user definitions around so that I can log into all of the machines.

But wait, I said I didn’t like to keep stuff on filesystems that isn’t mirrored, and choosing another OS doesn't change this? Right now I don’t care if the disk dies, because I can easily recreate all of the vm’s from scratch (i’ve got the descriptions I used to create the vm’s stored somewhere else). All “users” on the vm’s mount their home directory from my NFS server, so none of those files would get lost. I guess I could use zfs send/receive to back up the vm itself while I’m waiting for another disk for the box. 

So is there anything I miss from the current release, or is it perfect? There is one thing I really miss, and that is the ability to use the alias instead of the uuid when I’m using vmadm. I know that the alias doesn’t have to be unique, but if the alias is unique it would be a lot easier to use (instead of having to do a vmadm list first).

The second thing I’d love to see would be something like:

# vmadm bootinstall name iso-file

it would expand into something like:

cp iso-file uuid/root/cdrom.iso
vmadm boot uuid order=cd,once=d cdrom=/cdrom.iso,ide
vmadm info uuid vnc

But hey, if this is the biggest problems I’ve got with SmartOS I must be pretty happy with it. After all how often do you really create and install new VMs? The fact that I may use the tools I know and love (dtrace, zfs, smf etc) is just awesome!

Thank you Joyent for bringing this to my fingertips!