Event Sourcing from the Trenches: Projections

While visiting QCon New York this year, I realized that a lot of the architectural problems that were discussed there could benefit from the Event Sourcing architecture style. Since I've been in charge of architecting such a system for several years now, I started to reflect on the work we've done, e.g. what worked for us, what would I do differently next time, and what is it that I still haven't made my mind up about. So after having discussed the domain events, let me share some of my thoughts on projections.

Optimized for querying
Projections, a denormalized aggregation of events, have only one purpose: be optimized for reading. So if your API or user interface needs data to be grouped or aggregated in a certain way, project the events like that into whatever storage mechanism you use. The work load should be focusing on the projection logic, not on the reading side.

Projections should not enforce constraints
Projections should be seen as an aggregated cache of events and consequently shouldn't be used to enforce any constraints. In fact, projection code should never crash, ever. This may sound trivial, but if you're code base has been evolving for the a couple of years and events and the underlying constraints have changed several times, bugs are inevitable. So make sure you projection code is resilient. An (bit of naïve) example would be to always cut a string value to the maximum length of the underlying database column, even though you know the event value never exceeds it…now. That's why I love NoSQL databases so much….

Don't share projections
A logical consequence of optimizing queries for a single purpose is that they are optimized for a single purpose…. Even though the persisted projection schema looks like it is a good fit for another kind of query, don't reuse it. Even though they may look similar now, they are inevitably going to deviate. The worst problem you can have is that you need to add all kinds of alternate executing paths in your logic to make sure both interests are served equally. Just duplicate the data.

Keep projections close to the consumer
If you persist the projections to a persistent store like a database, don't assume that the projection belongs to something like a data access layer. As I said before, they should be seen as a local query cache for a very particular purpose. So keep it as close to the consumer as possible. If you use it in a particular HTTP API implementation, move the projection code next to the API code. In fact, I would be going so far by saying that you should test the two together. Such a test should use the events as input and observe the HTTP response as output.

Allow each projector to decide on the persistency store.
Another logical result of all those earlier statements is that each projection can be persisted to whatever storage fits best. Sure, you can store it to a relational database, but I highly recommend to use a NoSQL database instead. However, you could even build up your specific projection into memory as soon as the system starts. And what about projecting your events directly to a local HTML or JSON file that is served by an HTTP API or web site? That's the beauty of Event Sourcing.

Track progress locally
So if each projection can be using a different storage mechanism, you need to be able to have the projection code to track progress themselves. This is something that NEventStore got wrong. It relied on a central structure for determine whether an event was dispatched to all projections. Instead, make sure you design your projection logic to track progress yourself. It gives the projection logic a lot of autonomy, including the possibility to rebuild itself when code changes deem that necessary. Notice that this does mean your event store should allow arbitrary subscribers, each interested in a different starting point.

Asynchronous projection as a first-class concern
Another consequence of all that autonomy is that it essentially means all that projection logic should run asynchronously. We kind of started the notion of having the projection logic run as part of the same thread of control that caused the action on the domain. This allowed us to avoid having to change the UI to deal with the fact that projections are eventually consistent. But a lot of projections don't have to be up-to-date all the time can be run in the background perfectly. This scales much better, in particular when a new version of the code base needs a rebuild of the persisted projections. We added this possibility afterwards, which means it needs to work around the existing code base. So if you can, make this a primary architectural principle. And if you really need a 'synchronous' behavior, have your projection logic expose an interface that you can observe to see if it caught up.

Feedback, please!
So what do you think? Do my thoughts make sense? Am I too pragmatic here? Are you using Event Sourcing yourself? If so, care to share some experiences? Really love to hear your thoughts by commenting below. Oh, and follow me at @ddoomen to get regular updates on my everlasting quest for better solutions.