Unit testing with F#

I recently heard about a project where the code was written in C#, but where they implemented the unit tests in F#. They did so because they wanted to learn and evaluate F# in a protected environment. I thought that sounded like a very good idea and wanted to try.

I picked the unit tests used in builder pattern blog post. My idea was that the builder pattern would be replaced by abilities built into the language itself. Let’s see if that is a correct assumption.

From Fluent to Composition

This C# code is taken from my builder pattern blog post. It dynamically builds an order to use in the unit test, and is written with a fluent syntax:

            var order = new OrderBuilder()
                .WithSingle()
                .WithStatusInProgress()
                .Build()
                .First();

But, how do we write fluent code in F#. The answer is, we don’t:

Now the concept of “fluent interfaces” and “method chaining” is really only relevant for object-oriented design. In a functional language like F#, the nearest equivalent would be the use of the pipeline operator to chain a set of functions together.

F# for fun and profit

F# is a composable language, using functions as building blocks. Composition can be done in different ways, with function pipelining (|>) or function composition (>>). By defining functions that has the same type as argument and return type, they can be chained together in a very clean and decent way. Let’s see how that works.

The C# model

The C# model that we will test from our F# tests is just a simple order class with two methods:

    public class Order
    {
        public OrderStatus Status { get; set; }

        public void Cancel ()
        {
            Status = OrderStatus.Canceled;
        }

        public void StartProcessing()
        {
            Status = OrderStatus.InProgress;
        }
    }

It is stateful and the methods return void. This type of classes can be written in many ways, but I think this is rather common so I’ll use it even though it’s not perfect for the F# code which we’ll see later.

Create the builder

The builder class in C# is stateful (see blog post referred above), because it has class variable to which each function adds data. Using class variables means that the methods has a side-effect, and side-effects are something that we’ve learned that we should avoid if possible.

The functions in the F# builder are very simple and with no side effects as they operate on their indata (or create new data). The first one creates the order itself and returns it. The next one takes an order as an argument, changes it and returns it:

let withSingle () = 
    Order()

let withStatusInProgress (order : Order) = 
    order.StartProcessing()
    order

In F# we don’t write “return”, the value of the last line is automatically returned. Order() is the same as “new Order()” in C#. Note that if the method “order.StartProcessing” had returned the order, we could have omitted the last line of the second function. But I wanted to keep the C# code as we usually write it, like it would be in a real case.

So, this is the builder for now. We’ll keep it simple and find out later how to handle lists (which we do in the C# builder).

Function pipelining

This is what a test can look like in F#, FsUnit is used to get the nice assert syntax. Here we use pipelining to combine different functions that produces and manipulates the order.

[<Fact>]
let ``Cancel order - pipe test`` () =
    
    //Arrange
    let order = 
        OrderBuilder.withSingle()
        |> OrderBuilder.withStatusInProgress

    //Act
    order.Cancel()

    //Assert
    order.Status |> should equal OrderStatus.Canceled

“OrderBuilder.withSingle” is called to get an order that is used as input the next function “OrderBuider.withStatusInProgress”. The order that is sent to, and received from the functions is not visible in the code. This might be confusing at first, but when you get used to it it’s nice because it keeps you code cleaner.

..the function parameters can often be ignored when doing function composition, which reduces visual clutter.

F# for fun and profit

Function composition

With function composition it’s possible to combine functions in many different ways, and store them in variables. These function variables can then be combined with each other. If we use function composition in our test it would look like this instead:

[<Fact>]
let ``Cancel order composition test`` () =
    
    //Arrange
    let createOrder = 
        OrderBuilder.withSingle 
        >> OrderBuilder.withStatusInProgress

    let order = createOrder()

    //Act
    order.Cancel()

    //Assert
    order.Status |> should equal OrderStatus.Canceled

In this test there is an extra line to actually run the function. The best written tests should be both readable and short, so probably pipelining would be the best choice in this specific example.

Build lists

How do we build lists in a clean and simple way in our tests? First let’s add another function to our builder, “withStatusCancelled”:

let withSingle () = 
    Order()

let withStatusInProgress (order : Order) = 
    order.StartProcessing()
    order

let withStatusCancelled (order : Order) = 
    order.Cancel()
    order

Let’s say the test should build a list with one order that is in progress, and one that is cancelled. By using composition, two functions are defined by combining two other functions:

    //Arrange
    let createInProgressOrder = OrderBuilder.withSingle >> OrderBuilder.withStatusInProgress
    let createCancelledOrder = OrderBuilder.withSingle >> OrderBuilder.withStatusCancelled
    
    let orders = [createInProgressOrder(); createCancelledOrder()] 

To make the code even more compact, the following can be done without losing readability:

    //Arrange
    let orders = [
        (OrderBuilder.withSingle >> OrderBuilder.withStatusInProgress)()
        (OrderBuilder.withSingle >> OrderBuilder.withStatusCancelled)()
    ]

What do you think? Which one is the best? Or does anyone have a better suggestion?

Conclusion

Since F# is a language that is built for composition, it’s great for providing flexible data to tests. The amount of code needed for the OrderBuilder is significantly reduced compared to in C#. Handling lists is very easy and can be kept outside the builder, in the test itself.

Using F# for the unit tests is not only a good and safe way to get used to a functional language. It also makes the tests simpler and reduce boilerplate and the number of lines in your code.

After getting used to it, I also think that it makes the code more readable! (If written in a good way, but that applies to C# too). I recommend you to read more about how to use it as a Domain-Specific Language to take this one step further!


Download the complete code here.

Confused? Want to learn more?
Spend 60 seconds to better understand F#
Want to learn more about building blocks?


Letter to the coding newbie

Dear Newbie,

I know you struggle a bit with your new role and how to become a successful professional programmer. I think the best would be if I just told you, already now at start, what you need to know!

Many newbies are worried that they are not smart enough, but this is rarely the problem. What I’ve seen though, is lack of communication and humbleness. So, when you get stuck, don’t wait too long before you ask. Just do it! Asking questions means you’re smart! And even smarter is to ask the one in the team who you suspect might criticize your code later!

One advantage of being a newbie, is that you’re not supposed to know so much. I mean, after all, you’re a newbie! So instead of pretending to know stuff, be curious and try to learn all the time, every day. Being a person who wants to learn, is always appreciated by others. You’ll show the team that you’re genuine and you will learn and become more productive.

If the customer is happy, then I’m happy, you may think. And while this is of course true in one sense, it’s not the whole story. Because almost equally important is being a good team member. If the team agrees about something, you need to comply. Make friends with the team members and if something seems important to them try to work in that direction too. They might know something that you haven’t understood yet. Assume you work with smart people and try not to be the smartest person in the room.

I’m very glad that you’ve chosen this career path and I promise you a lot of great moments! Even if we old dinosaurs seems stressed sometimes, we love your enthusiasm and drive, and to relive memories from the time when we were newbies ourselves!

Yours sincerely,

Christina

KAFKA – Turning systems inside out and upside down

Inside out upside down: If something such as a system or way of life is turned inside out or upside down, it is changed completely, making people confused or upset.

collinsdictionary.com

While working with domain-driven design and event-driven development, I’ve every now and then stumbled over the event streaming platform named Kafka. When asking what it is, I’ve not fully understood. I’ve also been warned that it should only be used for very specific cases.

So, I’ve been kind of reluctant to dig into it until I started to read about Azure Event Hubs. The first thing that came up when starting to type it on Google was “Azure Event Hub vs Kafka”! So, being a person that wants to be up-to-date (well…) with Azure, I’ve now decided that I need to learn more about this.

I downloaded a book with the title Designing Event-Driven Systems from the prominent company Confluent. After reading it, I was really impressed! It confirms so many problems we have by building our systems as separate islands trying to communicate to each other. There were definitely a lot of new ideas of how to solve these problems!

What about the warnings?

Event streaming, as well as other event technologies or patterns as CQRS and Event sourcing, can add a lot of extra overhead if not used for an explicit purpose. You should consider the business you’re in, how they make money, weighing pros and cons. Bringing in Kafka to an organisation is a huge step, and it is important that you get return of investment.

But with that said, how could we know why we shouldn’t use it, if we don’t know what it its. So let’s get started!

What is Kafka?

Event streaming is the digital equivalent of the human body’s central nervous system. It is the technological foundation for the ‘always-on’ world where businesses are increasingly software-defined and automated, and where the user of software is more software.

Apache

Kafka is an event streaming platform, optimised for stream processing and high throughput. If you want to read the hard facts you can continue at Apache. If you want to know why it turns your view of software development upside down, preferably continue with this post!

Before I read the book, I thought of Kafka as an advanced service bus (ESB), but there is a main difference. Organisations using ESB tend to have a centralised approach with central teams that decide about schemas, transformation and message flows. This slows down the setup of new integrations and changes of existing. This also causes systems and services to evolve at a slower pace. With Kafka the services themselves are encouraged to provide their data to others, who are free to “act, adapt and change” according to their business needs.

Centralize an immutable stream of facts. Decentralize the freedom to act, adapt, and change.

Core mantra of event-driven services

Kafka has two APIs for stream processing, Kafka Streams and KSQL. With these you can filter, join streams and run arbitrary functions on the data. There is also an API for connecting to other systems, for example databases optimised for specific purposes like search. This API can connect to legacy systems that you are moving away from.

Kafka is saving events in streams which can also be seen as logs. Keeping data as logs means that doing updates is amazingly fast, just appending a new event to the end of the log. Logs also makes it possible for Kafka to scale linearly. It is typically installed on at least three machines but can be scaled up to hundreds. When reading and writing to a log, your data is written on all machines, which makes it easy to just add one. I is nearly impossible to hit a scalability wall with Kafka, the book is describing this in more detail.

A business consists of events

A lot of things (everything?) that happens in a business can be seen as events; hotels prices are changed, bookings cancelled, refunds performed. Even when a misspelled name is corrected, it is an event. In traditional systems there are few traces of these events, since only the latest state is stored in the database. Data that could be used for historical analysis is gone, or is hard to find. The events are not visible in the system, and throughout the flow between the systems and services.

Two things that are affected by this is:

  • It’s harder to analyse historical data to make business decisions, because a lot of data is not there anymore.
  • It’s hard to find out what is causing a bug and exactly where it happens.

“Work with the system, not against it”

Keynote at #kafkasummit

Instead, we should embrace the fact that things that happens in a business can be stored as events. These events flow in streams, and are projected to other events. (The present state is often cached to improve performance.) When data is handled this way, there is a transparency of what has happened in the system and the flow can be followed throughout different services. By analysing the data in the streams, valuable information can be collected which can serve as foundation for important business decisions.

The outside data is considered as a single source of truth

There is a distinction between the data on the inside and the data on the outside of the service or system. The data on the outside is much harder to change since a lot of other services are depending on it. But the fact that many are dependent of it makes it also much more important than the encapsulated data inside the service. Kafka is storing the entire event log outside the service, fully available to other services and systems! This makes the outside data a first-class citizen and the single source of truth.

Make data on the outside a first-class citizen

Ben Stopford

The database is turned inside out

Many of us recognise the problem of synchronising changes over several teams. E.g. one team wants to try out a brilliant idea that the customers would love. To do that they need data from a system that is maintained by another team. The other team is busy for months with changes requested by other parts of the business, and can’t help out. This slows the whole organisation down and causes endless priority discussions and a fight for resources.

When the source of truth is placed outside systems and services, it is available for anyone. It is stored as a stream of events. This makes it possible for anybody to travel in time, choosing exactly the data that would be the best for their purpose. This is referred to as “turning the database inside out”! To improve performance, data is also cached in Kafka tables, refreshed asynchronously. These tables can be defined for different purposes and are queried using the KSQL language.

This decouples the data in an organisation and keeps it as a shared single source of truth.

Less data needs to be stored inside

When the data is stored in shared logs outside the system or service, there is less need to store data inside. Data can be kept in memory, or read from the log or views whenever needed. That would reduce the storage required when data is stored more than once. It also lowers the risk that the data in different systems or services starts to differ.

Conclusion

Is Kafka the answer to my naive thought about streaming data through a system in my blog post Functional Domain Modelling? I can be. Kafka works very well with functional programming. But streaming data throughout a system can be done in many ways. Some of them certainly more lightweight and less complicated than Kafka. There is a lot more to learn in that area!

Increased interest from developers

To develop the best possible services to their customers, many companies need to handle a lot of data, e.g. collected in apps or in IOTs. Huge amounts of data must be processed with exceptional performance. More and more developers realise that they must find new ways to tackle these requirements. I recently joined the (on-line) Kafka summit together with 25000 other developers. Most of the participants described themselves as beginners which indicates an increased interest in this technology. Of course, there is a huge step to take from building traditional systems with REST APIs and request/reply approach, to doing it with event streaming. It’s a completely different way of thinking.

Kafka is also a very complex platform and it requires specialised persons only to manage the hosting. This indicates a large-scale use to make it profitable. It seems like they are working on that, though. Both by improving the product itself, and by making it easier to host in the cloud e.g. by using Azure Event Hubs. Techniques for event streaming and event sourcing in general tends to drive complexity. Especially as it is less intuitive and you have to find different solutions than you normally do.

Not mature enough for all of us…yet!

I’m quite impressed with Kafka and find it very interesting. Is mainly because I’ve been working at large companies with hundreds of systems. It is quite obvious that the data flows in the organisation, rather than is stored in separate databases.

One question that I ask myself is: Would we benefit from using Kafka even in applications that does not have enormous amounts of data in combination with extreme performance demands? I think we might…eventually. These use cases might be handled by other tools than Kafka, but a guess from my side is that a transformation to a more common usage of event streaming will probably be performed during the next ten years.

I’ve used event sourcing and CQRS in several projects. From quite pragmatic solutions within a single microservice, to fully implemented throughout the whole system. There were lessons learned and we can discuss whether it was the right way to go or not. Nevertheless, it gave me a new dimension to software development that I can’t move away from. The technologies are not mature enough yet, and we as developers are still very much stuck in the existing paradigm. But I think that event streaming is something that we’ll have to learn and get used to, in most applications. Exactly in what forms, we don’t know yet.

But, for sure…

To boil a whole book down to one tiny blog post is of course impossible. There are so much more that I want to squeeze in, so many more smart conclusions and so many quotes. But one thing I’m pretty sure of:

Kafka turns the database inside out and the way we build systems upside down!

What about the connection to Azure Event Hubs? Please read more in the links below!

Links:

Apaches description of Kafka: Apache

What Azure Event Hubs are: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview
How Azure Event Hubs relates to Kafka: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview
Kafka .Net Client: https://docs.confluent.io/current/clients/dotnet.html

Pat Helland: Data on the Inside and Data on the Outside

I’m now London based

A few weeks ago, me and my family took a huge step in our lives, we relocated from Gothenburg to London!

My everyday life has totally changed; a new house, a new city, not knowing anybody around me. I find myself going shopping at the butcher shop, greengrocery, fish store and bakery rather than driving to a supermarket. Amazon is my best friend; I can order almost anything and get it delivered two hours later. I walk in forests that look like they were taken from a Robin Hood movie instead of the enormous fir forests I’m used to in Sweden. Three weeks into my new life, after some ups and downs during the actual move, I’m realizing it has started out very well!

“Real change is difficult at the beginning, but gorgeous at the end. Change begins the moment you get the courage and step outside your comfort zone; change begins at the end of your comfort zone.”

Roy T. Bennett

Relocating to a new city, and country, might also turn things upside down in your career. I’ve usually got my jobs and contracts in Gothenburg via my social network and previous customers. In London I have to join the big job market competing with a lot of other equally skilled applicants, which is both exciting and challenging. I definitely have to step out of my comfort zone, which will hopefully make me grow and enable further personal development.

The Corona pandemic is not making things easier, or, maybe it is? These days we’ve all got used to work remotely which means that more companies than ever have their teams distributed over short and long distances. From that perspective I could still find a contract in Sweden and be part of a Swedish based team although I’m London based!

I have a lot of different options of how to proceed from now, and I’m so excited to see where this will take me. If anyone has any input or idea, I would be curious to hear about it!

View from London Bridge on our first visit to the city after moving to our new house in Barnes.

Teamwork – Complement instead of Compete

In an article called “The Secrets of Great Teamwork”, published in HBR (June 2016), Haas and Mortensen claim that an enabling condition for successful teamwork is that the team has a strong structure:

High-performing teams include members with a balance of skills. Every individual doesn’t have to possess superlative technical and social skills, but the team overall needs a healthy dose of both.

Harvard Business Review

This is something that I’ve experienced a lot and learned to appreciate. In many cases, however, the different personality and skills are perceived the opposite way, and problems occur accordingly. People are getting on each other’s nerves and team friction appears.

Agile methodologies are not enough

As you probably already know, there are several methodologies that provide a good foundation for effective teamwork, for example Agile, Lean and Kanban. These methodologies are very good, and should be used adapted to the specific conditions of the team.

My experience is, though, that methodology can never replace soft skills, and believe me, I’ve been in projects where we tried – a lot! I think that the first step to improve the soft skills is to start look at each other in a different way.

Different personalities

Many conflicts and lack of cooperation derives from the fact that we simply are different. For example, some people want to move fast, other are slow but thoughtful, some want to take risks while other want to be safe. And so on.

It is good that we have different personalities, since it makes the team stronger.

Manager

Next time you get annoyed by a colleague who doesn’t think like you, try to appreciate the other person’s perspective. If you’re fast and tend to take risks, it might be good for you to think everything over once more. If you’re slow and always want to take a safe approach, you might need someone to help you getting forward.

Everybody has both good and bad sides – even you and me! If you’re very good at details, you might not have the best overview of things. Your good sides often come with bad ones.

Good qualities come with bad ones.

Manager
Different skills

The same is true for different skills in a team. One teammate might be very experienced within the technologies that the system is built with. Another has worked a lot with newer technologies. Excellent, they complement each other! Let the one who knows new technologies evaluate whether they can be useful, but let the other guy balance it all up to make sure it really applies to the business problem that the system solves.

Make the differences an advantage

So, we should see differences as an advantage instead of getting competitive against each other. It makes the team stronger. Everybody is needed and should be encouraged to contribute as much as they can, bringing out their good sides.

Strive for a complementary team!

As 4D (dynamic, diverse, dispersed, and digital) teams become more and more common, I think that one key to success will be to change the view of each other to complementary instead of competitive!

Read the Harvard article here: https://hbr.org/2016/06/the-secrets-of-great-teamwork

Unit testing – Provide flexible data

It’s rather easy to find information about how to write a good unit test in the means of the test itself. But how should we handle the data that we need to test? How can we create test data structures in a flexible way?

As a DDD (Domain-Driven Development) person, I like to be able to read the code as I read a book. This is of course very hard, but something that I strive for. Therefore I often use the builder pattern for unit tests.

Builder pattern

Generally, a good approach is to create the data within the test itself. This makes it easy to see exactly what the test do. But after some tests you often find the same code repeated many times so you start looking for a generic way of generating the data suitable for each little thing you want to test. The builder pattern comes in handy here.

Let’s say you want to test the following test case:

* When you cancel an order, it gets the status cancelled. *

The test

You need to create an order with the status InProgress. Your test will then change the order to Canceled an verify that it worked. In a real system, you would probably have a lot of tests around status changes. Therefore it would be suitable to have code that creates an order in different statuses.

With the builder pattern to generate the order the test might look something like this:

        [Fact]
        public void CancelOrderTest()
        {
            //Arrange
            var order = new OrderBuilder()
                .WithSingle()
                .WithStatusInProgress()
                .Build()
                .First();

            //Act
            order.Cancel();
            
            //Assert
            order.Status.Should().Be(OrderStatus.Canceled);
        }

Let’s start with the “Arrange” part. A new OrderBuilder is created and asked to create a single order with the status InProgress. When looking into the builder later on you’ll see why the .Build() and .First() methods are needed.

The “Act” part here is intended to cancel the order, and the “Assert” checks whether the order has got the status Canceled. The FluentAssertions package is used to get a nice and fluent syntax of the assertions.

The builder

The OrderBuilder is a class with methods that returns the class itself. This is what makes it possible to use the fluent syntax in the test. Only the Build() method returns the actual list of orders:

    public class OrderBuilder
    {
        private List<Order> _orders = new List<Order>();

        public OrderBuilder WithSingle()
        {
            _orders.Add(new Order());
            return this;
        }

        public OrderBuilder WithStatusInProgress()
        {
            _orders.Last().StartProcessing();
            return this;
        }

        public List<Order> Build()
        {
            return _orders;
        }
    }

I chose to prepare the builder to handle many orders, by adding a private list variable in the top of it. Of course that could also be a single order.

The WithSingle and WithStatusInProgress methods both operates on the private class variable and step by step build up the order that the caller of the builder wants to have. More methods can be added and freely combined and called after each other to e.g. build up a list of orders:

        //Arrange
        var orders = new OrderBuilder()
            .WithSingle().WithStatusInProgress()
            .WithSingle().WithStatusNew()
            .Build();

The Build method returns the orders and breaks the chain by not returning the TestBuilder class.

A functional approach

Another approach to the problem is to use a functional way of building up the test data. Sometimes you have a lot of properties, and want to change them in a controlled way to get exact the data you want to test. You can add a method to the builder that lets you create an order this way:

    [Fact]
    public void CancelOrderTest2()
    {
        //Arrange
        var order = new OrderBuilder()
            .WithSingle(o => o.Status = OrderStatus.InProgress)
            .Build()
            .First();

        //Act
        order.Cancel();

        //Assert
        order.Status.Should().Be(OrderStatus.Canceled);
    }

You can from the test set which properties you want to change on the order, and combine with other builder methods if you want. The WithSingle builder method that takes a function parameter looks like this:

    internal OrderBuilder WithSingle(Action<Order> action)
    {
        _orders.Add(new Order());

        action?.Invoke(_orders.Last());

        return this;
    }

It takes a function as an argument, creates an order and applies the function to it.

Conclusion

The builder pattern has helped me a lot and it is always fun to use it! I show it here as I learned it from start, but in every project I use it in a special flavor suitable exactly for that special environment.

How to work with software architecture?

Are there any good reasons to not continuously work with a system’s architecture? I don’t think so. In my opinion it should be part of every development team’s work to know in what direction they’re heading, and also to try to influence that direction. I’ll try to help by describe how it can be done.

In most teams I’ve been working with there are a lot of complaints about the current code but there are very little money for refactoring. The system is slowly degrading, changes become expensive, and random refactorings (if the team can for once persuade their management) are done without getting any real improvements.

I usually follow the same steps to get in control of the system’s architecture and find a way forward:

  1. Describe the current architecture
  2. Define the target architecture
  3. Sell in to management
  4. Implement with team

It’s absolutely fundamental that the team starts to cooperate during this process, because to be successful you have to get the whole team running in the same direction. Quite often there are different opinions that has to be discussed and considered, to eventually agree on a way forward.

Describe current architecture

It is very common that the developers can’t describe their system, or draw it on the board. They know exactly where they need to do their changes, and they read the code efficiently, but they can’t give an overview of the system. This makes it very hard to describe it to new developers, and to discuss it with other team members. Which part of the system are we actually talking about? Which are the responsibilities and purpose of that part?

So the first step to improve the system is to describe the current architecture, to draw a picture of it as-is. When describing the system, don’t get into too much details. One page is usually enough. Of course this depends of the size of your system, but be rather too brief than too detailed. I would recommend something like this:

As-is architecture

The different clients and servers are described, and which technology/tool they are developed with. There is also an overview of other systems that this system is integrating with, and how.

When you’ve finished this description, put it on the wall and use it when you talk about the system, or when there are developers that are new to the team. Even project leaders and product owners can understand this kind of picture.

Define a target architecture

Next step, now when you know what you have, is to define the target architecture; how you want the system to be.

Now, if not done earlier, and even if you’re an appointed architect, you need to start discussions within the team about the future. Because it is now you have the best chance to involve the team members, listen to them and make them part of the decisions.

Involved team members will be dedicated when the architecture is to be implemented later on!

This takes time. It is also a phase full of learning, discussions and compromises. It is important that you understand as much as possible about the business:

  • What is the business making money of? (The vacuum cleaners or the dust bags? The cars or the parts?)
  • What part of the system is core, i.e. supports the business domains that makes the most money?
  • Which parts of the system is in most need of improvement, from the business perspective?
  • How would the business see the system evolve?

If your system is not very small, you need to do some domain modeling here to divide the system into smaller parts and structure the business logic into domains where it can be found, changed and reused. Also try to define non-functional requirements like availability, performance requirements and usability. These might already be known, but it’s good to write them down and agree about them.

Preferably, do proof of concepts of the parts of the architecture that you feel uncertain about. But since you have not convinced your management about this new architecture yet, you might not have time or resources to do technical investigations at this point. Remember that you can do the target architecture on the basis of what you know at this moment, and adjust it later as you learn more.

It can also be a good idea to learn more general things about architecture and coding conventions together with the team; read books, watch videos, have study groups. This makes it easier to get out of “deadlocks” when it comes to different opinions.

When you’ve gone through this phase you should hopefully end up with a target architecture more or less different from the current architecture:

To-be architecture

Try to not get started on the technical choices made in the target architecture above. It’s hard, I know, but this is just an example!

Sell in to management

Now you know your current architecture and you have defined the target architecture. It’s time to decide how to reach the target architecture.

To implement the target architecture in one big bang release, or to rewrite the system part by part must be decided from case to case. I always recommend to do it one part at a time because then you get payback from you investment sooner. That also significantly lower the risk.

What should you start with? From my experience, the best is if you can find something that would benefit both the business and the system itself. You can argue that to rewrite something without improving the business functionality is also good for the business because it will save money in maintenance, improve automated testing etc, which is probably true. But it can be hard to convince management to do that kind of investment.

In short, when moving towards the target architecture:

Start with the parts where the business wants a lot of functionality improvement.

Use the architecture pictures that you’ve made when discussing with your management, but leave technical details out of the discussion as much as possible. Try to see it from their perspective.

Implement with team

Hopefully, the work with the target architecture has resulted in a good team spirit because that’s what you need when you start to implement the target architecture. The way forward is now clear to everyone, but there are still a lot of details that must be solved and maybe a lot of new technology to be dealt with.

It is now important to keep focus on the benefits that you did sell in to the management, and make sure that you’re deliver something that fulfills the requirements. Keep the risks and the scope down. What you deliver now will hopefully build confidence and open up new possibilities for the future. If you fail, which we all do sometimes, be clear and honest of what failed and what you’ve learned from that.

There is a lot to write about the implementation of the architecture, too much for this blog post. I would recommend you to read more about hexagonal architecture or onion architecture, business logic and layering. It is also a good idea to read about business events, and event-driven development. Some of these I might have happened to write about in other blog posts!

Read my blog posts about:
Functional Domain Modeling
Why Micro services
What is Business logic

This is a very interesting article about legacy architecture modernisation that I wish I had written myself:
Legacy Architecture Modernisation with Strategic Domain-Driven Design

Also, read more about architecture and modeling from these famous writers:
Domain-driven design by Martin Fowler
Clean architecture by Uncle Bob

Hackathon at Systemite!

This year’s hackathon at Systemite is now finished! The challenge trophy is handed over to a new winner, Thorsten Jakobsson, and we’ve had 24 hours with a lot of fun!

For you who wonder what a hackathon is, please have a look at the first minutes of the greeting that Mark Zuckerberg sent us (didn’t he, Thorsten?):

The purpose of the Hackathon is to give the employees the possibility to explore and do hacks in any area, technically or businesswise. It is a way to encourage innovation and build a positive team culture. To give some inspiration we’ve chosen to have a special theme, and this year it was:

Take SystemWeaver to new platforms!

Explore the possibilities of the new api’s!

During last six months we’ve developed a REST API making it possible to access most of the features in the system from any platform. We’ve also ported the C# API to .Net Core so it can be accessed from e.g. Linux-based platforms.

SystemWeaver is opened up in many new ways, and we’re very curious to find out how this will be used both inside and outside the company!

Below you find the time schedule during the hackathon: From forming teams, to announce a winner:

Several ideás came up, and there were two that seemed to attract most participants:

  • Integrate SystemWeaver with test rigs using Ubuntu running on Docker and the new .Net Core API.
  • “Not only SystemWeaver at your hands – but in you hand”. A SystemWeaver mobile app.

Participants were joining the both teams, a great mix of developers, business analysts and business consultants, and the work begun!

The next day, everybody presented their hacks! The app team, which was the larger one, had managed to develop an app that used the REST API with the following features:

  • Display a dashboard with issues and items assigned to you.
  • View tests and fill in test cases
  • A bot (“Anna”) answering questions about SystemWeaver.
  • Providing help and information about releases (prototype)

The Docker team showed an integration with a test rig:

  • Export tests from SystemWeaver to the integration service on the Docker instance.
  • Import the tests to the rig, and send them back.

When everybody voted the Docker team turned out to be the winners and will get their names engraved on the challenge trophy!


~ A great thank you everyone for an exciting and productive Hackathon! ~

Extra thanks to Thorsten Jakobsson, my co-organiser, and the excellent help with food and snacks from Jan Söderberg.

Functional domain modelling

Most of us have an idea of what functional programming is, and have at least used some of the functional features of our best known programming language such as Lamba or LINQ in .Net.

When I’ve tried to learn more about functional programming, I’ve always felt that something is missing. There is a lot focus on the functions themselves, but less on where to put them in a structure. I’ve asked fellow programmers, but even if they seem very confident in writing functions, they get quite fuzzy when it comes to implementation of a real business problem, and how to structure the code.

Since I’ve used reactjs for many years I have one good example of how to implement a functional code design. But I have troubles applying these principles to the server side, or to see them as a more common approach. So I applied to a workshop at DDDEurope: Lean and functional modelling, with Marcello Duarte. This is what I learned:

You have to start with the programming

To understand the functional design you need to start with the programming. By understanding some of the most fundamental concepts in functional programming it will be easier to grasp the modelling part later. Now, I’ll probably not be the best to explain this after just two days of learning, but I can give you an overview of what you need to dig into. We used Scala for the programming exercises, but I think this is something that you can find in several languages and some of it maybe even in C#.

I found some descriptions with examples in Javascript, which might be easier to grasp for most of us:

Higher order function: “A higher order function is a function that takes a function as an argument, or returns a function. Higher order function is in contrast to first order functions, which don’t take a function as an argument or return a function as output”. Higher order functions.

Composition and currying: “A curried function is a function that takes multiple arguments one at a time. Given a function with 3 parameters, the curried version will take one argument and return a function that takes the next argument, which returns a function that takes the third argument. The last function returns the result of applying the function to all of its arguments.”
Curry and function composition

Functor: “A functor supplies a box with zero or more things inside, and a mapping interface. An array is a good example of a functor, but many other kinds of objects can be mapped over as well, including promises, streams, trees, objects, etc. JavaScript’s built in array and promise objects act like functors.”
Functors and Categories

Monads: “A monad is a way of composing functions that require context in addition to the return value, such as computation, branching, or I/O. Monads type lift, flatten and map so that the types line up for lifting functions a => M(b), making them composable. It’s a mapping from some type a to some type b along with some computational context, hidden in the implementation details of lift, flatten, and map.”
Monads made simple

Abstractions and compositions: “Software solutions should be decomposable into their component parts, and recomposable into new solutions, without changing the internal component implementation details.”
Abstraction and composition

Thanks to Eric Elliot for the descriptions and examples above!

When trying to write clean code there are things that I sometimes stumble on, e.g. keeping the number of parameters down and writing pure functions without side effects. To use a more functional approach is making it easier to write clean code.

Functional modelling

Functional architecture and design is often based on events and event-driven development. Concepts like event sourcing and CQRS can be considered as functional architectures. These architectures can of course be implemented (and is implemented) with any programming language and a wide range of technologies, but functional programming is very well suited for implementing this kind of architecture.

With event-driven development you identify a business flow and let the system reflect that flow. Data might change shape during this flow; for example in a purchasing domain you talk about a price of a product you buy, but in the finance domain the same amount is considered a cost of the same product. It is not just a different status of the product; it is considered a totally different thing.

Event Storming

A common way of identifying the business flow, is to do an event storming. At the event storming business events are identified, and also roles that are involved in firing the events, and the data that is sent. The team can do the event storming by themselves, but if possible, it is good to involve the business. The events, roles and data is written on post-its and put on the wall to illustrate the flow.

During the event storming it is important to avoid too much talk about implementation details (which is hard), and use cases (also hard). Try to focus on how the business work, the flow, and why things are done in a certain way. Look out for the core domain, and try to understand how it contributes to the business. It is important to understand what the business makes money of!

When the flow is defined it is possible to identify the services. One service should reflect one part of the business (domain). The data flows between the services in form of events, and very often the user roles indicate where it is suitable to draw the boundary of the service:

The services can be implemented in many different ways and they have their own databases or data stores. E.g. one is using a SQL database, while the other is using Eventstore and a CQRS approach. Since the services in this example is part of the same deployable unit there are some limitations when it comes to choosing languages and technology. Each service is responsible for its business rules, and don’t know anything about the others. Each service owns its own data.

Separate Responsibilities

In functional programming it is crucial to separate responsibilities into different parts of the code. This has spread to the object oriented systems too, so for many of us this comes very natural. The business logic is considered as the core of the system, and if you change technical implementations around it (database, client api’s etc.), it should be possible to leave the business logic untouched.

The flow throughout the system

The business logic implementation is composed of many independent functions which are called in a fluent way. I like to think of a system in the form of streams where the data flow and via projections and usage of different functions the data take different forms depending on what is should be used for. The flow sometimes temporarily stop waiting for input and is then saved in queues or databases. With event sourcing the flow could be reset and replayed at any time.

As a simple example, think about using LINQ or Lambda for filtering and mapping the the data that flows throughout the system. Then you’ve taken a big step. But try to think beyond that; can your own functions be written and applied to the flow in an even better way? Maybe, if we learn how to use the monads, functors etc. that was described earlier in this text.

Immutability

As an object oriented programmer I have some difficulties to stop thinking in objects with methods and properties. One problem with objects is that the functions often have side-effects, such as reading or modifying a property. Then the outcome of that function might not always be the same, with the same in-data. Side-effects must be avoided to make the functions composable.

The data that shall be modified in a function must be sent in as a variable, be recreated with the changes applied, and then sent back as the result of the function. The data is immutable. For those of us that has worked with reactjs, we’re quite familiar with this.

Conclusion

To fully understand functional programming is not easy. How to design composable functions using higher order functions, functors and monads is quite hard to grasp. On the other hand, functional programming has influenced us a lot already, and given us the thinking of responsibility, events, event sourcing, and event-driven design. Many of us are eager to learn how to use the reduce function in Javascript, how to implement reactjs in a better way, or writing LINQ and Lambda in C# and Java. All that work takes us closer to the paradigm shift toward functional programming.

Why Microservices?

Microservices is a way of separating a system into smaller autonomous parts. This can of course be done in many ways, but I’m most into the event-driven architecture and think that is at least a good start.

Every system I’ve been working with has started as a monolith. Some of them written in old languages like Cobol or Delphi, but some of them also in C#. What in a business that drives rewrite of a system is a separate discussion, but if these systems had been modularised in any way, it would have been so much easier (and cheaper) to step-by-step modernize them.

To be able to deal with technology changes of a system is a good reason to use a microservice architecture. To be able to rewrite business logic without affecting the whole system is another.

The possibility to deploy each service separately, has not made sense practically to me yet. Of course it depends on what type of system you’re working with. I think that to make sure that the services communicate using events (e.g. with pub-sub) and have separate data sources is much more important. This makes it easier to split the system if or when it is needed.

Event-driven architecture

In an event-driven architecture, business events are identified e.g. using event storming. A business event can be something like when a booking is done or cancelled, or a hotel is available for booking. When a business event is published other subscribing parts of the system is triggered and performing their tasks.

The publishing part don’t know anything more than that e.g. a booking is done. And the subscribing part, let’s say an accounting service, has no idea of how a booking is made, only that it is done and it is time to perform the accounting tasks that it is responsible for.

An event driven system is loosely coupled and asynchronous. The different services does not now anything about each others, and has no dependencies. In this way logic is isolated and has clear responsibilities. They can easily be deployed and scaled separately.

Read more about event-driven architecture and event storming.

CQRS and Event sourcing

CQRS and event sourcing is a way of taking event-driven design further and think of, and store, every performed task in the system as an event. All except the read of the CRUD operations are saved as created, updated and deleted events. Of course business events are saved as events as well. You get the traceability of everything that has happened in the system. You also get the possibility to generate and regenerate read data in different forms depending on what’s needed.

Microservices is the way forward

Microservices tend to decrease complexity in systems when performing changes in each service. When you look at the whole, though, it might increase complexity. Especially if you’re going to deploy them separately. The infrastructure gets more complex and, for example, if you have different languages in the services, the developers need a broad range of skills.

I suggest that you try to think in generations if you have services in different technologies. Try to keep the number of different technologies down. Develop one bunch of services that are very similar to each other when it comes to structure and technology. When that technology is getting less popular or starts to feel outdated, begin a new generation of services with a new set of technology choices and continue with these for the next period of time. Maybe rewrite some of the old ones, if needed. And continue like this over the years.

From my perspective, microservices (either compiled together or completely separated) is the only way of making a system survive over time. Even the most modern, high tech technology eventually gets old-fashioned. If we don’t want to completely rewrite the whole system every 20th year, we have to split them into smaller units.

One benefit of continuously rewrite the parts of a system instead of rewrite it all at once is that it is easier to attract developers who tends to choose jobs where they get the possibility to work with new technology. I’m sure it is easier to find a python developer rather than a COBOL developer today, for example.

Want to read more? Start here.