A faster MySQL driver for Python and gevent

During the later development and optimization phases of our real-time web framework PLANET we at ESN began looking more at the possibility of doing spot on C/C++ optimizations of our Python code. Not being the typical Python hacker of ESN I was asked to put together a proof of concept MySQL driver for Python written in pure C/C++.

Lucky for me I didn’t have to start from scratch, the gevent port of the Concurrence project already had done most of the research of the protocol and their source code acted as the perfect blue print. My work would become more of porting Python and Pyrex to C/C++ while adding my own touch for low level optimizations.

Integration code and the base driver

The base driver and the language specific integration code exchange a set of function pointers which allows the base driver to call into the integration code. The integration code is responsible for creating sockets, handling blocking I/O, result sets, character set conversions and type conversions. The base drivers deals with the MySQL protocol and may I say in a very performant matter

Not only for gevent and Python!

While the emphasis has been on creating a Python driver for gevent I’m also shipping integration code for “normal” blocking Python called simply CPython. Beyond that there’s nothing that will stop anyone from developing bindings to other languages like Ruby for instance. I’m looking very much forward to seeing that happening!

Fibers, threads or greenlets

Depending on which approach the I/O specific integration code takes it’s possible to adapt the base driver to any type offiber, thread or greenlet enviorment provided that they either switch back to the operating system upon blocking for I/O or yield to another user space thread (like fibers and greenlets) and while doing so preserves the stack.

Performance benchmarks

I’ll publish more detailed statistics later on but my initial benchmarks showed a 3-8 times performance gain over using the gevent-mysql driver. Until I find the time for putting up better benchmarks you’re just gonna have to take my word for it or feel free to contribute your own results!
Check out the project on github here!