Spring Boot Chaos Monkey

You are done with your application and it is ready to go into production, but are you confident in it? Spring Boot Chaos Monkey will help you verify that your application flies smoothly in production where network delays and outages occur.

Chaos Engineering

Chaos engineering is something very intriguing. The basic principle is quite straightforward. You know that your application will run good in a steady sunny weather, but are you certain that it will hold up when traffic comes in, and with networking between certain servers, databases and other components? Do you trust that your application will handle timeouts and outages in a good way?

When applying the principles of chaos engineering when testing the robustness of your application you start to simulate real-world events such as server failures, network delays, and even server crashes. The point is to make sure that your application handles this the way that you want it to. If your application performs correctly even after you have exposed it to chaos engineering, then you will certainly feel much more confident about deploying it into production.

In this article, I am going to explain about a favorite library of mine that I and my team uses for an application which is in production. I am going to give a couple of examples of what we use it so that you, hopefully, can get some inspiration.

The project is located here, and it is inspired by Chaos Monkey, initially developed by Netflix.

Spring Boot Chaos Monkey is extremely easy to get started with. Add the latest dependency, at the time of writing the latest released version, is 2.0.0, with 2.0.1 coming very soon.


    de.codecentric
    chaos-monkey-spring-boot
    2.0.0

However, Chaos Monkey will not run unless you activate the profile for it.

-Dspring.profiles.active=chaos-monkey

When starting with that profile activated, you will see that chaos monkey launches when booting your application. However, it is still only in an idle state so far. If you want it to be enabled it can be done in two ways, either adding by adding a property.

chaos.monkey.enabled: true

It can also be enabled via an actuator REST API, which does require you to enable another property for that.

management.endpoint.chaosmonkey.enabled: true

This will now expose an actuator REST API, which enables you to start/stop, checking status and configuring Chaos Monkey.

POST http://127.0.0.1:8080/actuator/chaosmonkey/enable

POST http://127.0.0.1:8080/actuator/chaosmonkey/disable

POST http://127.0.0.1:8080/actuator/chaosmonkey/assaults

{
        "level": 1,
        "latencyRangeStart": 100,
        "latencyRangeEnd": 600,
        "latencyActive": true,
        "exceptionsActive": false,
        "killApplicationActive": false,
        "restartApplicationActive": false,
}

In the request above, level 1 means that every request will be assaulted. Level 10 means that every 10th request would be assaulted.

There are different types of watchers; service, repository, controller, rest controller, and components. If no watcher is enabled, then no assaults will be performed. Sadly, it is not possible to change these at runtime because they are Spring AOP components that have to be created at the start of the application. If you enable the service watcher all Service beans will be watched.

chaos.monkey.watcher.controller: false
chaos.monkey.watcher.restController: false
chaos.monkey.watcher.service: true
chaos.monkey.watcher.repository: false
chaos.monkey.watcher.component: false

I would recommend creating an application-chaos-monkey.yml in your resources folder, where you add the chaos monkey configuration that you want. The reason for this is that the spring profile chaos-monkey will then enable your configuration, a profile that you anyway need to enable in order to run chaos monkey.

Watched Custom Services

Something new that is coming in Spring Boot Chaos Monkey 2.0.1 is the possibility to configure watched custom services. Prior to 2.0.1, if you enable services to be watched, then all of the services would be attacked. But my team had the need to target only specific services from time to time, and therefore, I contributed with that functionality to the opensource project. The configuration is done in the assaults configuration request. Where you can specify the classes that should be watched in a list.

{
        "level": 1,
        "latencyRangeStart": 100,
        "latencyRangeEnd": 600,
        "latencyActive": true,
        "exceptionsActive": false,
        "killApplicationActive": false,
        "restartApplicationActive": false,
        "watchedCustomServices": [
                "org.thecuriousdev.chaosmonkey.MyService"
        ]
}

With the configuration above, the only service that will be watched and attacked is the one specified in the list of watched custom services.

An example of a real-life usage from my own personal experience is that our application relies on a REST API from another component. This other component has had tendencies to not behave that well when a lot of stress is put on it. Therefore, I added a circuit breaker so that we don’t continue putting stress on the component, and instead return a 503 Service Unavailable to the clients calling us. With Spring Boot Chaos Monkey it is extremely easy to test this functionality, simply add that service as a watched custom service with a delay that will trigger the circuit breaker, and then you are good to go.

Final Words

Spring Boot Chaos Monkey is an extremely useful tool to verify that your application will run smoothly in production. There are loads of interesting things that you can do with it, and it enables you to test robustness in your application that otherwise would be very difficult to test.

I really recommend that you give it a shot, it is extremely simple to get started with and it will make you sleep a lot better at nights, knowing that your application will handle production.

Spring Boot Chaos Monkey

Chaos Engineering