We have provided an Eclipse Formatter file formatter.xml with all the formatting conventions currently used in the project. Highlights include no tabs, 4-space indentation, and 120-char width. Please respect this so as to reduce the amount of formatting-related noise produced in commits.

Static Analysis

We would like to use static analysis tools PMD and FindBugs to maintain code quality. However, we have not yet arrived at a consensus on what rules to adhere to, and what to ignore.

Netty Notes

The asynchronous network IO infrastructure that Hedwig uses is Netty. Here are some notes on Netty's concurrency architecture and its filter pipeline design.

Concurrency Architecture

After calling ServerBootstrap.bind(), Netty starts a boss thread (NioServerSocketPipelineSink.Boss) that just accepts new connections and registers them with one of the workers from the NioWorker pool in round-robin fashion (pool size defaults to CPU count). Each worker runs its own select loop over just the set of keys that have been registered with it. Workers start lazily on demand and run only so long as there are interested fd's/keys. All selected events are handled in the same thread and sent up the pipeline attached to the channel (this association is established by the boss as soon as a new connection is accepted).

All workers, and the boss, run via the executor thread pool; hence, the executor must support at least two simultaneous threads.

Handler Pipeline

A pipeline implements the intercepting filter pattern. A pipeline is a sequence of handlers. Whenever a packet is read from the wire, it travels up the stream, stopping at each handler that can handle upstream events. Vice-versa for writes. Between each filter, control flows back through the centralized pipeline, and a linked list of contexts keeps track of where we are in the pipeline (one context object per handler).


This summarizes the control flow through the system.


Need to document


Need to document

ReadAhead Cache

The delivery manager class is responsible for pushing published messages from the hubs to the subscribers. The most common case is that all subscribers are connected and either caught up, or close to the tail end of the topic. In this case, we don't want the delivery manager to be polling bookkeeper for any newly arrived messages on the topic; new messages should just be pushed to the delivery manager. However, there is also the uncommon case when a subscriber is behind, and messages must be pulled from Bookkeeper.

Since all publishes go through the hub, it is possible to cache the recently published messages in the hub, and then the delivery manager won't have to make the trip to bookkeeper to get the messages but instead get them from local process memory.

These ideas of push, pull, and caching are unified in the following way: - A hub has a cache of messages

Scalability Bottlenecks Down the Road