Wednesday, July 28, 2010

libmemcached on win32

It's a fact that people love their development platform and want to stick with it. I'm a die hard Solaris fan, and would never dream of switching to something else. I've heard that there is a crowd out there that likes to work on other systems like Windows, MacOSX, BSD and Linux. That the developer use a platform during development doesn't necessary mean that the target product will run on the platform, but the developer is more productive on that platform.

People can argue as much as they want, but there is a large crowd of developers using Windows. There is also a large number of systems running some version of Windows, so enabling them to use the projects I'm working on is a good thing. Earlier today I pushed a branch that adds support for building libmemcached into a dll on Windows.

90% of the source code in libmemcached is just "logic" that applies for all platforms, but there is a small part of the code that interacts tightly with the operating system. "Everything" is a file descriptor on Unix systems, but Windows got their own subsystem for sockets called WinSock. In order to avoid getting tonns of #ifdefs all over the code, I defined memcached_socket_t to represent a socket object, and used "the WinSock way" to implement the code. It is pretty easy to map the WinSock code to work on Unix systems with a couple of macros.

Well, enough talk. If you're interested in the details you can check out the branch on Launchpad.

The easiest way for you to test out the code is to install the fullinstall of msysgit. Unfortunately it doesn't come with all of the tools needed to build libmemcached (you can't generate a configure script and generate the documentation). This means that you cannot build the development branch unless you got another machine where you can generate the configure script. I am exporting parts of my ZFS filesystem via CIFS, so I generated the configure script on Solaris.

Building libmemcached with mingw is just as easy as on your favorite platform:

$ ./configure --with-memcached= --without-docs
$ make all install

I haven't fixed the test suite yet, so you have to wait a bit longer before you can run make test ;)

21 comments:

  1. Great stuff, Trond! Unfortunately, I can't manage to generate the missing configure script myself (on my Windows-only setup), so I'm stuck...

    Could someone who successfully built the libmemcached.dll for Win 32-bit share it, please? And/or the used configure script for MinGW? Many thanks in advance!

    ReplyDelete
  2. send me an email to trond dot norbye at gmail dot com and I'll send you one :-)

    ReplyDelete
  3. I had to issue the following command to build configure script:

    $ ./configure --with-memcached=memcached --without-dics

    first placing memcached.exe in the directory. Otherwise configure stopped with error, complaining that it couldn't find memcached for tests (as far as I remember).

    I managed to build dll's (3 of them), but had no time yet to integrate them with MS C++.

    I'm wondering why the target libraries are so big? limemcached-5.dll has size of 18 MB (!), libmemcachedprotocol-0.dll size is 2.5 MB and libmemcachedutil-1.dll is over 1 MB.

    Anyway - thank you very much for this work!

    ReplyDelete
  4. Jacek, You need to add ".exe" to the memcached binary (I assume the dics is just a typo here). C

    The correct configure line would be:
    ./configure --with-memcached=memcached.exe --without-docs

    I have no idea why the size is so big, but please note that I'm not done with my win32 (and win64) efforts for libmemcached. Getting the test suite up'n'running is a high priority ;-)

    ReplyDelete
  5. Hi Trond, thank you for your suggestion (and sorry for my late response...).

    Shortly after posting my first comment, I found an alternative that will do fine for me for the time being (also because you mention your win32 libmemcached port is still work in progress).


    I found it via http://stackoverflow.com/questions/1511045/php-memcache-dll-for-php-5-3

    This php_memcache.dll PHP extension (about 50kB) conforms to the (less extended) Memcache API (http://www.php.net/manual/en/book.memcache.php) (vs. the Memcached API as libmemcached does).

    However, the nice thing is that a lot of prebuilt variants of the library are directly available
    for download (x86 vs. x64/PHP 5.2 vs. 5.3/thread-safe vs. non-thread-safe) here:

    http://downloads.php.net/pierre/

    After selecting the right variant for your setup, it works like a charm with the NorthScale Memcached Server.

    The existence of those build variants (apart from the obvious x86 vs. x64 flavour) indicates
    that it is not that easy to get a self-built library working with a specific setup. So, that's
    another reason, if you don't mind, why I prefer to stick with the php_memcache.dll for now
    (although I hope one day libmemcached will be available for download the same way ;-) , as it
    opens up the full potential of Memcached).

    Again, thanks a lot for your efforts!

    ReplyDelete
  6. First steps to integrate libmemcached.dll with MSVC++ 2008 have been done. I created import libraries according to the procedure described here:

    http://www.coderetard.com/2009/01/21/generate-a-lib-from-a-dll-with-visual-studio/

    (Unfortunately Microsoft does not provide a utility to create import library from a dll in a single step.)

    To compile libmemcached headers with MSVC++ some additional definitions were necessary - types like uint32_t, and to compile inttypes.h from MinGW it was necessary to #define __restrict__ as empty string. MSVC++ (in C++ mode) doesn't also allow returning user defined types from extern "C" functions, so I had to exclude a function returning a structure.

    Now it's time to see if it works :-)

    ReplyDelete
  7. Hello Trond & Jacek, I'm just (very) curious about your progress with the libmemcached win32 build in the meantime.

    Honestly, once you're familiar with Memcached, you outgrow the API of the available php_memcache.dll rather quickly. More in particular, the need for access to the GetMulti(ByKey) functionality libmemcached offers becomes clear. I'm convinced that with me you will help out many many developers tremendously with a win32 release of libmemcached!!

    Good luck & many thanks again! J.

    ReplyDelete
  8. Jozz and Jacek, I've been a bit busy doing other (but related) tasks lately.. I will be back with more info later on :)

    ReplyDelete
  9. Any news on this? Would really like to get my Windows dev machine to use libmemcached.

    ReplyDelete
  10. Holm: Is there any reason why you can't use the current version?

    ReplyDelete
  11. Hi,
    I am not able to compile it under WIN32 getting this error.
    I have installed MinGW and
    ./libmemcached/options/server.h:42:23: fatal error: arpa/inet.h: No such file or directory

    ReplyDelete
  12. Hi,
    I am not able to compile it under WIN32 getting this error.
    I have installed MinGW and
    ./libmemcached/options/server.h:42:23: fatal error: arpa/inet.h: No such file or directory
    Can you please help me fix this error?

    ReplyDelete
  13. I was able to get it working after making some changes and using platform.h. Thanks.

    ReplyDelete
  14. Do you have any documentation on how to compile libmemcached on visual studio?

    ReplyDelete
  15. It is great to try this branch. I really like to use libmemcached in win32. I have downloaded the code and started to play with it. However, it seems that tests still fail. In particular, when I run test-atom, the memcached_set() always returns MEMCACHED_HOST_LOOKUP_FAILURE.

    Are those tests supposed working, or still not? Thanks.

    BTW, I use the memcached binary which is downloaded from http://downloads.northscale.com/memcached-1.4.5-x86.zip (got the link from http://blog.elijaa.org/index.php?post/2010/08/25/Memcached-1.4.5-for-Window).

    ReplyDelete
  16. Please send me compiled DLL? My mail: m1u1n2kh2 at gmail dot com

    ReplyDelete
  17. I was able to generate the libmemcached-6.dll, am not sure how to proceed after this. How do I enable it on PHP with this dll, I tried setting this dll on the extensions section of php.ini, but no luck. Has anyone succeeded?

    ReplyDelete
  18. Hi,

    I am trying to build this dll but get an error during the "make all install" step. I am trying to build a 32-bit memcached dll file on Windows 7 using 64-bit cygwin. My memcached.exe is 32-bit.

    Here are the results of my build steps:

    Results of executing "./configure" command
    https://babycenter.box.com/s/iq3vus03ux060a8vkk45

    I also get this warning:
    configure: WARNING: libmemcached requires at least version 1.0.6 of Embedded InnoDB

    Results of executing "make" command
    https://babycenter.box.com/s/kyv6ns3m9ponjkivt0h4

    Any help you can provide would be appreciated.

    Also, if you have a win32 memcached dll already built that you could just email me (cboudrea at babycenter.com), I would also accept that. ;)

    ReplyDelete
  19. I haven't used libmemcached for a couple of years.

    ReplyDelete
  20. Hi Trond,

    We need a small help from you for building auth_memcookie module for apache 2.4.

    We are trying to build auth-memcookie for apache 2.4 on windows x64 box. We tried to build libEvent, memcached and libMemcached which are dependencies for building auth_memcookie. We were able to build libMemcached using Visual Studio 2012 and generate libMemcached.lib file, but when we try to build mod_auth_memcookie we are getting linking errors for the functions that are defined in libMemcached. Errors we are getting are as follows:

    LNK2001: unresolved external symbol __imp_memcached
    LNK2001: unresolved external symbol __imp_memcached_free
    LNK2001: unresolved external symbol __imp_memcached_get
    LNK2001: unresolved external symbol __imp_memcached_last_error_message
    LNK2001: unresolved external symbol __imp_memcached_set

    Also, we tried to generate libMemcached.dll but we were getting compilation error for the same.

    Can you please guide us on these issues.
    Thanks.

    ReplyDelete