Kafka

The Apache Kafka instance used in DE4A is an Open-Source distributed event streaming platform. It is used to receive messages from the different services used in the project. The received messages can be viewed online as they are received in the Package Tracker made available for everyone. The main purpose is to be able to quickly see if there are any problems between services and what type of problem (e.g. connection problems) to help pilot partners and developers find those issues. Therefore, the main use case is distributed event logging and not digital traffic reporting where full logs are sent and stored off-site. The reasoning for this is that the Kafka instance on the Aeioros servers is fully open, meaning that anyone can read and write messages on that service. To reduce security risks, the Kafka instance does not store messages sent for extended periods of time, i.e. messages are essentially ephemeral.

The particular configuration used for the DE4A project uses ZooKeeper as the coordinator, Kafka as the message broker and the Package Tracker as the web service configured to consume and display any messages received by the Kafka broker. The Package Tracker can only display "live" messages, i.e. any messages received in the past will never be displayed.