Independent IT Professional
Data continuously flows through organisations, and on its way, it's transformed many times to serve different purposes. Functional design emphasises data transformation—projections— rather than repeatedly mutating state in a single database. One particularly useful form of projection is the creation of read-optimised data structures tailored to well-defined use cases.
In my article Kafka Versioning I have briefly covered different ways of handling versions and using schemas to manage changes to the data structures. However, merely understanding these concepts falls short when operating within an environment characterized by numerous teams and bounded contexts. It is necessary to identify patterns and principles for navigating model changes successfully.
I have worked with DDD for many years, and learned to not share databases and to use anti-corruption layers between systems and services. Therefore, I was at first very reluctant to connect Kafka directly to the databases. Wouldn't that recreate the monolith that we have been working so hard to get rid of?
Defining an event may seem straightforward, and when I wrote my first Kafka article I thought of all events as business events. I've now realized that there are various types, with Change Data Capture (CDC) events emerging as one of the most prevalent.
While working with domain-driven design and event-driven development, I've every now and then stumbled over the event streaming platform named Kafka. When asking what it is, I've not fully understood. I've also been warned that it should only be used for very specific cases.