Inter-Thread Communication Without a Mutex

One common problem that comes up in multithreaded code is the need to pass information between one thread and another. Solutions to this using the pthread API normally involve using some flavor of mutex. One potential problem with a mutex-based approach is that, while often the initial code is designed to always avoid deadlocks, during later maintenance changes are made which introduce deadlock issues. Code with mutexes can also experience race conditions or performance issues if the critical sections are not chosen with care. And let’s not forget the most popular mutex bug of all — forgetting to unlock a mutex in a rarely taken error path.

A simple way to avoid mutex-locking issues is to not use mutexes. This post describes a simple technique for passing objects between threads in a thread-safe manner without mutexes. It is especially well suited for code which follows an event-driven paradigm. This technique makes the entire class of possible mutex-related errors impossible to implement!

The usual case is where we have (possibly) multiple producer threads that need to pass data to a single consumer thread. As an example, say the producer threads are each off talking to other devices over the network, gathering information, and passing it to some master thread for display.

The approach I often take to this problem exploits the fact that UNIX pipe writes are defined to be atomic by POSIX. The Linux pipe(7) man page points this out:

POSIX.1-2001 says that write(2)s of less than PIPE_BUF bytes must be atomic: the output data is written to the pipe as a contiguous sequence. Writes of more than PIPE_BUF bytes may be nonatomic: the kernel may interleave the data with data written by other processes. POSIX.1-2001 requires PIPE_BUF to be at least 512 bytes. (On Linux, PIPE_BUF is 4096 bytes.) The precise semantics depend on whether the file descriptor is nonblocking (O_NONBLOCK), whether there are multiple writers to the pipe, and on n, the number of bytes to be written.

To do this, I create a pipe with the pipe(2) system call. In the consumer thread, I add one end of the pipe to the set of file descriptors monitored by poll(2) in its event loop. In the producing thread, I can allocate an object on the heap via malloc or new, and write the address of the object I want to pass to the consuming thread down the pipe, and discard the pointer. The consuming thread wakes up out of the poll, reads the pointer to the object from the pipe, and now owns the pointer. At this point, it can use the data, delete or free the object, or even pass it to another thread.

Because pipe writes are atomic, I can have multiple producer threads writing down the same pipe, without worrying that the bytes specifying the address of an object being sent from one thread will be intertwingled with the bytes specifying the address of another object from another thread.

The overhead of this approach is a trip into the kernel for write() of sizeof(pointer) bytes on the producer side. I am assuming the consumer thread is already blocked in a poll() event loop. So this approach is quite lightweight and makes locking issues into non-issues.

An example producer and consumer implementation may be found here.