Technical Debt and Ageing Systems

When developers take shotcuts or decrease the quality to gain time, they will cause a technical debt. By doing nothing and letting the system age, the technical debt is growing too. Actually, as soon as we’ve implemented the code, it starts to age.

The code that is tightly connected to a specific technology ages more than code that is written purely in its own, e.g. business logic. But sometimes it is the language itself that is ageing. Or rather, it’s impossible to separate the code itself from the infrastructure.

Developers’ view

Developers often use the term technical debt to describe what happens when they are forced to implement a design that will decrease code maintainability. Let’s say it takes three weeks to develop something properly, but because of time pressure they have to do a bad design that take two weeks instead. Later on, when the feature is in production and there might have been more changes to it, it would take them 3 weeks to refactor.

DevelopmentGood design Shortcut
Develop feature3 weeks2 weeks
Refactor to reduce technical debt3 weeks
Total3 weeks5 weeks
Technical debt example

By making a lot of shortcuts, the technical debt grows and there will be a future cost to make the code maintainable. Changes will take longer and there is a higher risk for defects.

Technical debt is the coding you must do tomorrow because you took a shortcut in order to deliver the software today.

Technical Debt: The Ultimate Guide

Ageing systems

I would like to widen the definition a bit. When building new software there is a lot of focus on features. This is necessary to get more customers and take market shares. There are also technical and architectural decisions about how to fulfil non-functional requirements like scalability and maintainability. All systems have been through this process, even the old ones. Decisions have been made based on what was available and popular at that time.

As the world change, your system will age. It might not keep up with new requirements when it comes to performance, scalability and usability. I consider this a technical debt as well, since it means that you, by doing nothing, push a growing cost to the future.

What problems does technical debt cause in ageing systems?

What will happen when you let your system age:

  • Users are limited when using the system. Your system can’t be used on a wide range of devices or operating systems. The system can’t take advantage of new technology and frameworks provided for, e.g., browsers, which might allow only a very basic user experience.
  • The system can’t scale. The technology does not support scaling or performance requirements. Maybe you use technology that can’t be moved to the cloud.
  • Unable to improve business model. The technology does not support the business model you want to implement. For example, you install the system on each customer’s site, but you actually want to offer a SaaS (Software as a Service).
  • Code changes take time and are costly. The code base has grown large without refactoring and time to delivery is huge for each change.
  • Lack of development resources. It’s hard to find developers that want to work with old technology.

How should companies handle the technical debt?

What happens if the company doesn’t handle the technical debt? Probably not so much, until it’s too late, which might take many years. As long as the competitors have a similar technical debt the company can be successful. But if you can’t run your system in the cloud, and suddenly a competitor launches a new SaaS solution, you might find yourself in a very difficult situation.

A company must fund money to handle the technical debt. The debt is a loan from the future, that will need to be paid back. An alternative is of course to continuously evolve the business and find other ways to earn money, and after that scrap the old solutions. The nature of a technical debt is that it can never be fully paid, just kept at a reasonable level. To find that level, the management have to weight different factors like competitors, risk, finance etc.

There is no simple answer to the question about how to handle technical debt. But a general advice would be to admit that it exists, and to take informed decisions. It is a much lower risk to spread out the investments over many years, doing stepwise improvements, rather than doing nothing for twenty years and then perform a complete rewrite of the whole system.

Conclusion

The technical debt is very unlikely to ever be fully paid off. It can be higher or lower, and have more or less implications on the company. Defining the amount of the technical debt is a strategic decision that has to weigh in knowledge about technology, market and finance.

Knowledge about this is usually spread over different roles in a company, and people tend to have little understanding beyond their own areas of expertise. Awareness and discussions will significantly increase the chances for a company to successfully handle their technical debt.

https://martinfowler.com/bliki/TechnicalDebt.html

Code structure – The House Metaphor

Reading code vs. writing code

In a legacy system, what is it that you spend absolutely most time on?

I would say, understanding code and finding out where to do a change. Most of us have been in the situation that we spend hours or days looking for exactly one single line that would be changed in a million line code base. And of course trying to figure out what effects that little change would get. Expected and unexpected.

In an old Cobol system I came across, even the most experienced developers often spent two weeks analysing what consequences a change would get. Of course that was before the time of unit tests and modularisation of code into microservices, but the correlation of the time it took to write code compared to change it was very obvious.

What are we looking for in a code base?

Let’s say I’ve got at task at my new job. I’ve got requirements that needs a change in the code. I have to find the exact lines to change. Being in this situation, I’m often confused. I can’t immediately see the overall structure and don’t know where to look for the code I’m about to change. I try to figure out certain things, for example:

Where is the code..

  • ..providing the UI or API?
  • ..that call the infrastructure like database or file system?
  • ..handling integrations with other systems or services?
  • ..handling the business logic?
  • ..defining configuration?

Also, I’m looking for the code that handles the business domain that I want do my changes in? In a monolith there might not be any separations by domains. In a microservice architecture there might be a better separation. For example, managing hotels is handled in the Hotel Management Service. Price changes are handled in the Pricing Service. And so on. Preferably I would like some form of graphical overview that could map the business to the system.

The house metaphor

When talking about this, my and my teammates often use the house metaphor. Many of us have owned or lived in a house, we know how they usually are structured and we get help from craftsmen to do maintenance and changes on our houses. There is a lot of similarities with a system.

A house is divided into separate rooms. Each room has a purpose; cooking, sleeping, storage, bathing and so on. Each room is optimised for the users’ (people who live there) needs.

Craftsmen come to the house with a mission to improve something or fix stuff that has broke. Depending on what they’re going to fix, they go to different rooms. They immediately know which room they are looking for. If they are about to install a new shower, they will go to the bathroom, if they are fixing the dishwasher machine they go to the kitchen. Nobody would look for a shower in the bedroom. There might of course occasionally be exceptions, I heard about someone having a TV built in to the bathroom floor.

If there isn’t a map, they will make a quick glance in each room to find out what type of rooms it is or ask the owner. Within a few minutes they grasp the location of everything in the house.

Create 2D Floor Plans easily with RoomSketcher. Draw yourself or order. Perfect for real estate, home design and office projects. High-quality for print & web.
roomsketcher.com

Before looking for the appropriate room, the craftsmen must of course find the correct house. In a city of millions of houses. First they might ask which district the house belongs to, and after that which address.

Here the craftsmen has an advantage compared to developers. We rarely get the “address” of the service we’re going to change together with the requirements. Sometimes we can start out in the UI and follow the code to find out in which are the changes should be done, and sometimes we simply have to ask someone in our team. Preferably, we always go through the changes with someone that is experienced within the system before we start to find out where to make the change.

What can we learn from managing houses?

If we could significantly decrease the time a developer spend on trying to find out what the system does, there would be a huge benefit for companies. Time could be spent on new features, and more features. So how do we do that?

  1. Define what it is you’re building. Is it a storehouse, a bungalow or a skyscraper? Sometimes you start with a bungalow and end up with a skyscraper, it’s pretty obvious that the architecture is not clear and easy to follow in that case. When this happens, refactoring is crucial to build a maintainable system.
  2. Define responsibilities. Define responsibilities of services (e.g. hotel management), layers (e.g. domain layer) and classes (e.g. controller, repository). Then developers immediately will understand where to look for the code they want to change. Is the change about the contract between the client and the API, then look in the controller classes in the API layer. Is it about the communication to the database, then look in the repository classes. The shower should be in the bathroom!
  3. Use commonly used patterns and guidelines. Most houses have a similar set of rooms, with similar purpose. This makes it easy for us to quickly understand the usage of each room. I your system follows a well-known way of structure the code, it will be easier for new users to start working with it.

And one more thing. Draw a picture of the system and make sure that the system’s file structure represents that picture. If there is a box named Hotel Management on the picture, make a folder named Hotel Management. If there is a layer named domain layer, make sure the developers can find a folder with that name in the code.

I think most developers find the above ideas pretty obvious. Despite that, many systems end up with an unclear structure. There are many reasons for that; tight time schedules, bad communication in the team, problems in combining helicopter perspective with frog perspective.

Conclusion

Systems often live up to 20 years and during that time a certain amount of developers are passing through. All of them with different experiences from previous systems they have worked with. If they could be up and running quickly, if they know what they are doing when making a change and if they easily can find a specific place in the code then time to delivery will be shortened and the quality improved. The changes will be more aligned with existing code and cause less confusion for other developers. The risk of causing bugs with the change will be lowered.

Companies will have everything to win, and nothing to lose, on encouraging the development team to keep a good code structure!

Same theme, another metaphor: https://medium.com/nick-tune-tech-strategy-blog/the-chocolate-sauce-design-heuristic-3cdc143dd2ea

KAFKA – Turning systems inside out and upside down

Inside out upside down: If something such as a system or way of life is turned inside out or upside down, it is changed completely, making people confused or upset.

collinsdictionary.com

While working with domain-driven design and event-driven development, I’ve every now and then stumbled over the event streaming platform named Kafka. When asking what it is, I’ve not fully understood. I’ve also been warned that it should only be used for very specific cases.

So, I’ve been kind of reluctant to dig into it until I started to read about Azure Event Hubs. The first thing that came up when starting to type it on Google was “Azure Event Hub vs Kafka”! So, being a person that wants to be up-to-date (well…) with Azure, I’ve now decided that I need to learn more about this.

I downloaded a book with the title Designing Event-Driven Systems from the prominent company Confluent. After reading it, I was really impressed! It confirms so many problems we have by building our systems as separate islands trying to communicate to each other. There were definitely a lot of new ideas of how to solve these problems!

What about the warnings?

Event streaming, as well as other event technologies or patterns as CQRS and Event sourcing, can add a lot of extra overhead if not used for an explicit purpose. You should consider the business you’re in, how they make money, weighing pros and cons. Bringing in Kafka to an organisation is a huge step, and it is important that you get return of investment.

But with that said, how could we know why we shouldn’t use it, if we don’t know what it its. So let’s get started!

What is Kafka?

Event streaming is the digital equivalent of the human body’s central nervous system. It is the technological foundation for the ‘always-on’ world where businesses are increasingly software-defined and automated, and where the user of software is more software.

Apache

Kafka is an event streaming platform, optimised for stream processing and high throughput. If you want to read the hard facts you can continue at Apache. If you want to know why it turns your view of software development upside down, preferably continue with this post!

Before I read the book, I thought of Kafka as an advanced service bus (ESB), but there is a main difference. Organisations using ESB tend to have a centralised approach with central teams that decide about schemas, transformation and message flows. This slows down the setup of new integrations and changes of existing. This also causes systems and services to evolve at a slower pace. With Kafka the services themselves are encouraged to provide their data to others, who are free to “act, adapt and change” according to their business needs.

Centralize an immutable stream of facts. Decentralize the freedom to act, adapt, and change.

Core mantra of event-driven services

Kafka has two APIs for stream processing, Kafka Streams and KSQL. With these you can filter, join streams and run arbitrary functions on the data. There is also an API for connecting to other systems, for example databases optimised for specific purposes like search. This API can connect to legacy systems that you are moving away from.

Kafka is saving events in streams which can also be seen as logs. Keeping data as logs means that doing updates is amazingly fast, just appending a new event to the end of the log. Logs also makes it possible for Kafka to scale linearly. It is typically installed on at least three machines but can be scaled up to hundreds. When reading and writing to a log, your data is written on all machines, which makes it easy to just add one. I is nearly impossible to hit a scalability wall with Kafka, the book is describing this in more detail.

A business consists of events

A lot of things (everything?) that happens in a business can be seen as events; hotels prices are changed, bookings cancelled, refunds performed. Even when a misspelled name is corrected, it is an event. In traditional systems there are few traces of these events, since only the latest state is stored in the database. Data that could be used for historical analysis is gone, or is hard to find. The events are not visible in the system, and throughout the flow between the systems and services.

Two things that are affected by this is:

  • It’s harder to analyse historical data to make business decisions, because a lot of data is not there anymore.
  • It’s hard to find out what is causing a bug and exactly where it happens.

“Work with the system, not against it”

Keynote at #kafkasummit

Instead, we should embrace the fact that things that happens in a business can be stored as events. These events flow in streams, and are projected to other events. (The present state is often cached to improve performance.) When data is handled this way, there is a transparency of what has happened in the system and the flow can be followed throughout different services. By analysing the data in the streams, valuable information can be collected which can serve as foundation for important business decisions.

The outside data is considered as a single source of truth

There is a distinction between the data on the inside and the data on the outside of the service or system. The data on the outside is much harder to change since a lot of other services are depending on it. But the fact that many are dependent of it makes it also much more important than the encapsulated data inside the service. Kafka is storing the entire event log outside the service, fully available to other services and systems! This makes the outside data a first-class citizen and the single source of truth.

Make data on the outside a first-class citizen

Ben Stopford

The database is turned inside out

Many of us recognise the problem of synchronising changes over several teams. E.g. one team wants to try out a brilliant idea that the customers would love. To do that they need data from a system that is maintained by another team. The other team is busy for months with changes requested by other parts of the business, and can’t help out. This slows the whole organisation down and causes endless priority discussions and a fight for resources.

When the source of truth is placed outside systems and services, it is available for anyone. It is stored as a stream of events. This makes it possible for anybody to travel in time, choosing exactly the data that would be the best for their purpose. This is referred to as “turning the database inside out”! To improve performance, data is also cached in Kafka tables, refreshed asynchronously. These tables can be defined for different purposes and are queried using the KSQL language.

This decouples the data in an organisation and keeps it as a shared single source of truth.

Less data needs to be stored inside

When the data is stored in shared logs outside the system or service, there is less need to store data inside. Data can be kept in memory, or read from the log or views whenever needed. That would reduce the storage required when data is stored more than once. It also lowers the risk that the data in different systems or services starts to differ.

Conclusion

Is Kafka the answer to my naive thought about streaming data through a system in my blog post Functional Domain Modelling? I can be. Kafka works very well with functional programming. But streaming data throughout a system can be done in many ways. Some of them certainly more lightweight and less complicated than Kafka. There is a lot more to learn in that area!

Increased interest from developers

To develop the best possible services to their customers, many companies need to handle a lot of data, e.g. collected in apps or in IOTs. Huge amounts of data must be processed with exceptional performance. More and more developers realise that they must find new ways to tackle these requirements. I recently joined the (on-line) Kafka summit together with 25000 other developers. Most of the participants described themselves as beginners which indicates an increased interest in this technology. Of course, there is a huge step to take from building traditional systems with REST APIs and request/reply approach, to doing it with event streaming. It’s a completely different way of thinking.

Kafka is also a very complex platform and it requires specialised persons only to manage the hosting. This indicates a large-scale use to make it profitable. It seems like they are working on that, though. Both by improving the product itself, and by making it easier to host in the cloud e.g. by using Azure Event Hubs. Techniques for event streaming and event sourcing in general tends to drive complexity. Especially as it is less intuitive and you have to find different solutions than you normally do.

Not mature enough for all of us…yet!

I’m quite impressed with Kafka and find it very interesting. Is mainly because I’ve been working at large companies with hundreds of systems. It is quite obvious that the data flows in the organisation, rather than is stored in separate databases.

One question that I ask myself is: Would we benefit from using Kafka even in applications that does not have enormous amounts of data in combination with extreme performance demands? I think we might…eventually. These use cases might be handled by other tools than Kafka, but a guess from my side is that a transformation to a more common usage of event streaming will probably be performed during the next ten years.

I’ve used event sourcing and CQRS in several projects. From quite pragmatic solutions within a single microservice, to fully implemented throughout the whole system. There were lessons learned and we can discuss whether it was the right way to go or not. Nevertheless, it gave me a new dimension to software development that I can’t move away from. The technologies are not mature enough yet, and we as developers are still very much stuck in the existing paradigm. But I think that event streaming is something that we’ll have to learn and get used to, in most applications. Exactly in what forms, we don’t know yet.

But, for sure…

To boil a whole book down to one tiny blog post is of course impossible. There are so much more that I want to squeeze in, so many more smart conclusions and so many quotes. But one thing I’m pretty sure of:

Kafka turns the database inside out and the way we build systems upside down!

What about the connection to Azure Event Hubs? Please read more in the links below!

Links:

Apaches description of Kafka: Apache

What Azure Event Hubs are: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview
How Azure Event Hubs relates to Kafka: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview
Kafka .Net Client: https://docs.confluent.io/current/clients/dotnet.html

Pat Helland: Data on the Inside and Data on the Outside

How to work with software architecture?

Are there any good reasons to not continuously work with a system’s architecture? I don’t think so. In my opinion it should be part of every development team’s work to know in what direction they’re heading, and also to try to influence that direction. I’ll try to help by describe how it can be done.

In most teams I’ve been working with there are a lot of complaints about the current code but there are very little money for refactoring. The system is slowly degrading, changes become expensive, and random refactorings (if the team can for once persuade their management) are done without getting any real improvements.

I usually follow the same steps to get in control of the system’s architecture and find a way forward:

  1. Describe the current architecture
  2. Define the target architecture
  3. Sell in to management
  4. Implement with team

It’s absolutely fundamental that the team starts to cooperate during this process, because to be successful you have to get the whole team running in the same direction. Quite often there are different opinions that has to be discussed and considered, to eventually agree on a way forward.

Describe current architecture

It is very common that the developers can’t describe their system, or draw it on the board. They know exactly where they need to do their changes, and they read the code efficiently, but they can’t give an overview of the system. This makes it very hard to describe it to new developers, and to discuss it with other team members. Which part of the system are we actually talking about? Which are the responsibilities and purpose of that part?

So the first step to improve the system is to describe the current architecture, to draw a picture of it as-is. When describing the system, don’t get into too much details. One page is usually enough. Of course this depends of the size of your system, but be rather too brief than too detailed. I would recommend something like this:

As-is architecture

The different clients and servers are described, and which technology/tool they are developed with. There is also an overview of other systems that this system is integrating with, and how.

When you’ve finished this description, put it on the wall and use it when you talk about the system, or when there are developers that are new to the team. Even project leaders and product owners can understand this kind of picture.

Define a target architecture

Next step, now when you know what you have, is to define the target architecture; how you want the system to be.

Now, if not done earlier, and even if you’re an appointed architect, you need to start discussions within the team about the future. Because it is now you have the best chance to involve the team members, listen to them and make them part of the decisions.

Involved team members will be dedicated when the architecture is to be implemented later on!

This takes time. It is also a phase full of learning, discussions and compromises. It is important that you understand as much as possible about the business:

  • What is the business making money of? (The vacuum cleaners or the dust bags? The cars or the parts?)
  • What part of the system is core, i.e. supports the business domains that makes the most money?
  • Which parts of the system is in most need of improvement, from the business perspective?
  • How would the business see the system evolve?

If your system is not very small, you need to do some domain modeling here to divide the system into smaller parts and structure the business logic into domains where it can be found, changed and reused. Also try to define non-functional requirements like availability, performance requirements and usability. These might already be known, but it’s good to write them down and agree about them.

Preferably, do proof of concepts of the parts of the architecture that you feel uncertain about. But since you have not convinced your management about this new architecture yet, you might not have time or resources to do technical investigations at this point. Remember that you can do the target architecture on the basis of what you know at this moment, and adjust it later as you learn more.

It can also be a good idea to learn more general things about architecture and coding conventions together with the team; read books, watch videos, have study groups. This makes it easier to get out of “deadlocks” when it comes to different opinions.

When you’ve gone through this phase you should hopefully end up with a target architecture more or less different from the current architecture:

To-be architecture

Try to not get started on the technical choices made in the target architecture above. It’s hard, I know, but this is just an example!

Sell in to management

Now you know your current architecture and you have defined the target architecture. It’s time to decide how to reach the target architecture.

To implement the target architecture in one big bang release, or to rewrite the system part by part must be decided from case to case. I always recommend to do it one part at a time because then you get payback from you investment sooner. That also significantly lower the risk.

What should you start with? From my experience, the best is if you can find something that would benefit both the business and the system itself. You can argue that to rewrite something without improving the business functionality is also good for the business because it will save money in maintenance, improve automated testing etc, which is probably true. But it can be hard to convince management to do that kind of investment.

In short, when moving towards the target architecture:

Start with the parts where the business wants a lot of functionality improvement.

Use the architecture pictures that you’ve made when discussing with your management, but leave technical details out of the discussion as much as possible. Try to see it from their perspective.

Implement with team

Hopefully, the work with the target architecture has resulted in a good team spirit because that’s what you need when you start to implement the target architecture. The way forward is now clear to everyone, but there are still a lot of details that must be solved and maybe a lot of new technology to be dealt with.

It is now important to keep focus on the benefits that you did sell in to the management, and make sure that you’re deliver something that fulfills the requirements. Keep the risks and the scope down. What you deliver now will hopefully build confidence and open up new possibilities for the future. If you fail, which we all do sometimes, be clear and honest of what failed and what you’ve learned from that.

There is a lot to write about the implementation of the architecture, too much for this blog post. I would recommend you to read more about hexagonal architecture or onion architecture, business logic and layering. It is also a good idea to read about business events, and event-driven development. Some of these I might have happened to write about in other blog posts!

Read my blog posts about:
Functional Domain Modeling
Why Micro services
What is Business logic

This is a very interesting article about legacy architecture modernisation that I wish I had written myself:
Legacy Architecture Modernisation with Strategic Domain-Driven Design

Also, read more about architecture and modeling from these famous writers:
Domain-driven design by Martin Fowler
Clean architecture by Uncle Bob

Functional domain modelling

Most of us have an idea of what functional programming is, and have at least used some of the functional features of our best known programming language such as Lamba or LINQ in .Net.

When I’ve tried to learn more about functional programming, I’ve always felt that something is missing. There is a lot focus on the functions themselves, but less on where to put them in a structure. I’ve asked fellow programmers, but even if they seem very confident in writing functions, they get quite fuzzy when it comes to implementation of a real business problem, and how to structure the code.

Since I’ve used reactjs for many years I have one good example of how to implement a functional code design. But I have troubles applying these principles to the server side, or to see them as a more common approach. So I applied to a workshop at DDDEurope: Lean and functional modelling, with Marcello Duarte. This is what I learned:

You have to start with the programming

To understand the functional design you need to start with the programming. By understanding some of the most fundamental concepts in functional programming it will be easier to grasp the modelling part later. Now, I’ll probably not be the best to explain this after just two days of learning, but I can give you an overview of what you need to dig into. We used Scala for the programming exercises, but I think this is something that you can find in several languages and some of it maybe even in C#.

I found some descriptions with examples in Javascript, which might be easier to grasp for most of us:

Higher order function: “A higher order function is a function that takes a function as an argument, or returns a function. Higher order function is in contrast to first order functions, which don’t take a function as an argument or return a function as output”. Higher order functions.

Composition and currying: “A curried function is a function that takes multiple arguments one at a time. Given a function with 3 parameters, the curried version will take one argument and return a function that takes the next argument, which returns a function that takes the third argument. The last function returns the result of applying the function to all of its arguments.”
Curry and function composition

Functor: “A functor supplies a box with zero or more things inside, and a mapping interface. An array is a good example of a functor, but many other kinds of objects can be mapped over as well, including promises, streams, trees, objects, etc. JavaScript’s built in array and promise objects act like functors.”
Functors and Categories

Monads: “A monad is a way of composing functions that require context in addition to the return value, such as computation, branching, or I/O. Monads type lift, flatten and map so that the types line up for lifting functions a => M(b), making them composable. It’s a mapping from some type a to some type b along with some computational context, hidden in the implementation details of lift, flatten, and map.”
Monads made simple

Abstractions and compositions: “Software solutions should be decomposable into their component parts, and recomposable into new solutions, without changing the internal component implementation details.”
Abstraction and composition

Thanks to Eric Elliot for the descriptions and examples above!

When trying to write clean code there are things that I sometimes stumble on, e.g. keeping the number of parameters down and writing pure functions without side effects. To use a more functional approach is making it easier to write clean code.

Functional modelling

Functional architecture and design is often based on events and event-driven development. Concepts like event sourcing and CQRS can be considered as functional architectures. These architectures can of course be implemented (and is implemented) with any programming language and a wide range of technologies, but functional programming is very well suited for implementing this kind of architecture.

With event-driven development you identify a business flow and let the system reflect that flow. Data might change shape during this flow; for example in a purchasing domain you talk about a price of a product you buy, but in the finance domain the same amount is considered a cost of the same product. It is not just a different status of the product; it is considered a totally different thing.

Event Storming

A common way of identifying the business flow, is to do an event storming. At the event storming business events are identified, and also roles that are involved in firing the events, and the data that is sent. The team can do the event storming by themselves, but if possible, it is good to involve the business. The events, roles and data is written on post-its and put on the wall to illustrate the flow.

During the event storming it is important to avoid too much talk about implementation details (which is hard), and use cases (also hard). Try to focus on how the business work, the flow, and why things are done in a certain way. Look out for the core domain, and try to understand how it contributes to the business. It is important to understand what the business makes money of!

When the flow is defined it is possible to identify the services. One service should reflect one part of the business (domain). The data flows between the services in form of events, and very often the user roles indicate where it is suitable to draw the boundary of the service:

The services can be implemented in many different ways and they have their own databases or data stores. E.g. one is using a SQL database, while the other is using Eventstore and a CQRS approach. Since the services in this example is part of the same deployable unit there are some limitations when it comes to choosing languages and technology. Each service is responsible for its business rules, and don’t know anything about the others. Each service owns its own data.

Separate Responsibilities

In functional programming it is crucial to separate responsibilities into different parts of the code. This has spread to the object oriented systems too, so for many of us this comes very natural. The business logic is considered as the core of the system, and if you change technical implementations around it (database, client api’s etc.), it should be possible to leave the business logic untouched.

The flow throughout the system

The business logic implementation is composed of many independent functions which are called in a fluent way. I like to think of a system in the form of streams where the data flow and via projections and usage of different functions the data take different forms depending on what is should be used for. The flow sometimes temporarily stop waiting for input and is then saved in queues or databases. With event sourcing the flow could be reset and replayed at any time.

As a simple example, think about using LINQ or Lambda for filtering and mapping the the data that flows throughout the system. Then you’ve taken a big step. But try to think beyond that; can your own functions be written and applied to the flow in an even better way? Maybe, if we learn how to use the monads, functors etc. that was described earlier in this text.

Immutability

As an object oriented programmer I have some difficulties to stop thinking in objects with methods and properties. One problem with objects is that the functions often have side-effects, such as reading or modifying a property. Then the outcome of that function might not always be the same, with the same in-data. Side-effects must be avoided to make the functions composable.

The data that shall be modified in a function must be sent in as a variable, be recreated with the changes applied, and then sent back as the result of the function. The data is immutable. For those of us that has worked with reactjs, we’re quite familiar with this.

Conclusion

To fully understand functional programming is not easy. How to design composable functions using higher order functions, functors and monads is quite hard to grasp. On the other hand, functional programming has influenced us a lot already, and given us the thinking of responsibility, events, event sourcing, and event-driven design. Many of us are eager to learn how to use the reduce function in Javascript, how to implement reactjs in a better way, or writing LINQ and Lambda in C# and Java. All that work takes us closer to the paradigm shift toward functional programming.

Why Microservices?

Microservices is a way of separating a system into smaller autonomous parts. This can be done in many ways, but I’m most into the event-driven architecture and think that is at least a good start.

Every system I’ve been working with has started as a monolith. Some of them written in old languages like Cobol or Delphi, but some of them also in C#. What in a business that drives rewrite of a system is a separate discussion, but if these systems had been modularised in any way, it would have been so much easier (and cheaper) to step-by-step modernize them.

To be able to deal with technology changes of a system is a good reason to use a microservice architecture. To be able to rewrite business logic without affecting the whole system is another.

The possibility to deploy each service separately, has not made sense practically to me yet. Of course it depends on what type of system you’re working with. I think that to make sure that the services communicate using events (e.g. with pub-sub) and have separate data sources is much more important. This makes it easier to split the system if or when it is needed.

Event-driven architecture

In an event-driven architecture, business events are identified e.g. using event storming. A business event can be something like when a booking is done or cancelled, or a hotel is available for booking. When a business event is published other subscribing services are triggered and performing their tasks.

The publishing service don’t know anything more than that e.g. a booking is done. And the subscribing service, let’s say an accounting service, has no idea of how a booking is made, only that it is done and it is time to perform the accounting tasks that it is responsible for.

An event-driven system is loosely coupled and asynchronous. The different services does not now anything about each others, and have no dependencies. In this way logic is isolated and has clear responsibilities. They can easily be deployed and scaled separately.

Read more about event-driven architecture and event storming.

CQRS and Event sourcing

CQRS and event sourcing are ways of taking event-driven design further and think of, and store, every performed task in the system as an event. All except the read of the CRUD operations are saved as created, updated and deleted events. Of course business events are saved as events as well. You get the traceability of everything that has happened in the system. You also get the possibility to generate and regenerate read data in different forms depending on what’s needed.

Microservices is the way forward

Microservices tend to decrease complexity in systems when performing changes in each service. When you look at the whole, though, it might increase complexity. Especially if you’re going to deploy them separately. The infrastructure gets more complex and, for example, if you have different languages in the services, the developers need a broad range of skills.

I suggest that you try to think in generations if you have services in different technologies. Try to keep the number of different technologies down. Develop one bunch of services that are very similar to each other when it comes to structure and technology. When that technology is getting less popular or starts to feel outdated, begin a new generation of services with a new set of technology choices and continue with these for the next period of time. Maybe rewrite some of the old ones, if needed. And continue like this over the years.

From my perspective, microservices (either compiled together or completely separated) is the only way of making a system survive over time. Even the most modern, high tech technology eventually gets old-fashioned. If we don’t want to completely rewrite the whole system every 20th year, we have to split them into smaller units.

One benefit of continuously rewrite the parts of a system instead of rewrite it all at once is that it is easier to attract developers who tends to choose jobs where they get the possibility to work with new technology. I’m sure it is easier to find a python developer rather than a COBOL developer today, for example.

Want to read more? Start here.

What is business logic?

I found it rather common that developers have only a vague idea of what business logic is. I also sometimes find it hard to describe it myself, even though I’ve spent many hours of my career working with it.

So let’s try to clarify things. Business logic is core logic, that doesn’t have anything to do with technology. It is the logic that can be found as rules in the business even though there were no digital system implemented. What is considered as core to a business, is the activities that they gain money of.

Strictly speaking, business rules are rules or procedures that make or save the business money

Uncle Bob

Let’s take the example of a booking that gets cancelled. Business logic are the rules that kicks in with freeing up resources (hotels, cars or whatever is booked), confirm cancellation to customer, and refund money. Technically how this is done; by updating a database, which technology that is used to send an email or which payment solution that is called to refund the money, is not business logic. The technology should be exchangeable.

Business logic should be kept separate from the other logic in the system and it is worth a considerable effort to have a lot of automated testing here to verify that the rules are not broken.

Business rules vs. Use case

Business logic is often divided into two different types: Business rules and Use cases. First, I will address that there might be some naming confusion here, when it comes to business rule vs. business logic. The business rule is something that exists in the business even without a computer, while the business logic is the code that implements the rule.

..the application of VAT on invoices is a business rule but the calculations involved in applying it are implemented as business logic

Ben Morris

When implementing business rules, it is very common that it is some kind of flow in which the rules are called. This flow can be implemented in something called Use case. Other names might be Service or Domain service.

Use cases are easily mapped to the requirements which are preferably defined as test cases. If implementing automated acceptance tests following the test cases you get a good assurance that the functionality that you have agreed with the customer or business is working.

Implementation

Use cases are classes containing the flow for a use case. They can call repositories to save data, or they might call the domain model classes that performs the business rules.

E.g. when a booking is cancelled the hotel rooms should be made available to other bookings. It is the responsibility of the use case to make sure that this is included in the flow, but it is the domain model’s responsibility to know how a hotel is marked as available (maybe by changing the status).

As I wrote earlier, there is much value in automated testing of business logic to make sure that these rules work according to spec. Write acceptance tests testing the Use cases, and write unit tests that tests the business rules:

As we know reality is often a lot more complex that these simple examples. But this can be used as a starting point from which we continuously search for more knowledge. Why not continue by reading this blogpost by Uncle Bob?