Writing native database drivers - and a new MySQL driver port

Sun, 29 Apr 2012 After we published the vibe.d project, one reaction that came quite often was: "It's nice to see MongoDB and Redis support, but I need support for classical SQL databases". This post shows an example on how to port an existing driver to vibe.

If there is an open source driver already available in C or D, the process is straight forward. Since vibe.d uses a blocking I/O model and almost any database driver is written using blocking sockets, all that is needed is to replace any socket by a TcpConnection. The following table shows the most common socket functions next their vibe.d equivalent.

vibe.d D sockets C sockets
TcpConnection Socket/TcpSocket SOCKET/int
TcpConnection.tcpNoDelay Socket.setOption(SocketOption.TCP_NODELAY) setsockopt(TCP_NODELAY)
connectTcp(host, port) Socket.connect(addr) connect(sock, addr)
TcpConnection.write(buf) Socket.send(buf) send(sock, buf, len, flags)
TcpConnection.read(buf) Socket.receive(buf) recv(sock, buf, len, flags)
TcpConnection.close() Socket.shutdown(which), Socket.close() shutdown(which), close()

Note that unlike the C and D versions, TcpConnection.read() and write() will always read/write the whole buffer that is given. If this is not possible for some reason, they will throw an exception. Because of this, it is often possible to simplify the code a bit and remove the otherwise needed loops.

As an example, the following list shows the changes that were necessary to port the MySQL driver originally written in D by Steve Teale to the vibe framework. MySQL uses a packet based serial protocol over a TCP connection to communicate between the server and its clients.

  1. The first part was to replace the import and the socket declarations:

    import std.socket;
    // ...
    class Connection {
        // ...
        Socket _socket;
        ubyte[] _packet;
        // ...
    

    was changed to:

    import vibe.core.tcp;
    // ...
    class Connection {
        // ...
        TcpConnection _socket;
        ubyte[] _packet;
        // ...
    
  2. Then the connection sequence had to be adjusted:

    _socket = new TcpSocket();
    Address a = new InternetAddress(InternetAddress.PORT_ANY);
    _socket.bind(a);
    a = new InternetAddress(_host, _port);
    _socket.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVBUF, (1 << 24)-1);
    _socket.connect(a);
    

    this was replaced by the following line. The RECVBUF option is not available in vibe, but is also not needed because its effects are mostly hidden inside the system.

    _socket = connectTcp(_host, _port);
    
  3. Next, there were a lot of single calls to _socket.receive() and _socket.write(). These were replaced by calls to read()/write() with the same parameters. Note that the code actually was made more robust here, because the return value of the original calls was seldomly checked and this way data could get lost (this probably did not actually happen because the read/write buffers of the socket were large enough).

    However, there was an exception in the code, where this behavior was actually wanted. The transformed code simulates the behavior using _socket.leastSize - this waits until data is available an then returns the size of this data.

    _packet.length = 512;
    _socket.receive(_packet);
    

    was changed to

    _packet.length = _socket.leastSize;
    _socket.read(_packet);
    
  4. The main getPacket() function was performing a loop to read a full protocol packet that was possibly fragmented:

    _packet.length = pl;
    ubyte[] buf;
    buf.length = (pl > _rbs)? _rbs: pl;
    uint got = 0;
    size_t n = _socket.receive(buf);
    for (;;)
    {
       _packet[got..got+n] = buf[0..n];
       got += n;
       if (got >= pl)
          break;
       if (pl-got < _rbs)
       buf.length = pl-got;
       n =_socket.receive(buf);
    }
    

    This code block could also be replaced by a single line:

    _packet.length = pl;
    _socket.read(_packet);
    
  5. Finally, the call to _socket.shutdown() could be removed because TcpConnection.close() does this implicitly.

    _socket.shutdown(SocketShutdown.BOTH);
    _socket.close();
    

    became:

    _socket.close();
    

And that's all. The complete converted code is available as a github fork. You can include it in your project by putting it as a dependency in your package.json (See the docs):

{
    ...
    "dependencies": [
        "mysql-native": ">=0.0.1"
    ]
}

It seems to be able to perform a successful login by now, but I have not really tested it any further, yet. It will also need some refactoring and a connection pool to make it more efficient and better fitting into the framework. Thanks to Steve Teale for making the original code available!

Posted at 08:37:13 GMT by Sönke Ludwig

Comments

Abscissa on Sun, 03 Jun 2012, 01:50:13 GMT:
When using vibe.d, is there a problem with using a MySQL lib that's not ported to vibe? Do the standard D and C sockets conflict with vibe.d's sockets?
Sönke Ludwig on Wed, 06 Jun 2012, 05:55:34 GMT:
Sorry for the late answer. But yes the sockets need to be non-blocking and they need to be able to wake the event loop of the active event driver (libevent currently). With standard sockets, the application performance would come to a crawl as soon as many connections are active.

Commenting is currently disabled for this article.