DE4A Playground

The Playground is a set of predefined and pre-deployed components to be used as is. The “DE4A Playground” was designed to help pilot partners to onboard the DE4A network. It is essentially a temporary infrastructure with two purposes:

• It simulates a real scenario where the User can request data (evidence) about a subject (citizen or company). Thus, it allows testing the performance of the DE4A Connector by using mocked components and fake data.

• It allows DE4A pilot participants to deploy their own infrastructure in an orderly manner. A three-stage process was set out for the pilot participants to deploy their software components: first their own developed entities (DE/DO), secondly the DE4A Connector, and finally the SMP of the eDelivery infrastructure. At the end of each stage, the pilot partner can test their deployment by making used of the rest of the Playground components.

The target audience for this chapter are primarily software developers that try to connect their DE/DR/DT/DO system to DE4A. Bellow, a quick introduction to the main structure and list of components involved:

DE4A Playground Big Picture

The Playground consists of the following components:

• Data Evaluator
• DE4A Connector with the role of Data Requestor, using the phase4 AS4 Gateway
• DE4A Connector with the role of Data Transferor
• Data Owner
• Domibus AS4 Gateway
• SMP
• Mocked IDK
• Kafka Server

The DO and the DT are deployed at SGAD (ES). All other components, including the Kafka Server, are deployed at SGAD subcontractor facilities (ES).

Electronic Certificates

Most of the components used require the HTTPS protocol, so TLS certificates are needed. A separate document, “Policy for Transport Security”, describes the requirements on them. Additionally, the SMP and the AS4 gateways require each a separate certificate for signing and encrypting the payload. For this purpose, we required two separate Certificate Authorities (CAs) – one for the AS4 Gateway and one for the SMP component.

Certificate Authorities in the DE4A Playground

The above figure depicts the layout of the CAs. For the “DE4A Test AS4 CA”, WP5 created a new root CA that provides AS4 certificates. This simple CA can be used to issue as many certificates as we like, but it’s only suitable for piloting purposes. For a production exchange DE4A would need to search a commercial solution. For the SMP CA we needed to use a CA that matches the requirements of CEF, as the maintainers of the Service Metadata Locator (SML). To fulfil our urgent needs for a suitable CA, CEF was kind to offer us 10 SMP certificates based on their trusted “Connectivity Test CA”. In parallel to that, we promised CEF to search a commercial CA that we could use as a sustainable solution for SMP certificates. The requirements are outlined in the “Trust Models Guidance” document of CEF.

Data Evaluator and Data Owner

The DE and DO are both set up in the DE4A Playground. These components are deployed to act as a starting point where the pilots can interact with the Playground.

Data Evaluator

The Data Evaluator deployed in the Playground was developed as a separate piece of software. The source code can be found at https://github.com/de4a-wp5/wp5-demo-ui and the deployed instance of this software is available at https://de4a-demoui.informatika.uni-mb.si/.

All the IEM handling functionality is taken from the shared “de4a-commons” library available at https://github.com/de4a-wp5/de4a-commons.

Note: the functionality of this tool may be extended during the further development of the project. The primary goal is to support Member States in onboarding as good as possible.

Note: the terminology used in the demo application is very simplified and may not suffice the needs of real-world applications. Real pilot applications should align the texts with their local needs.

System requirements

The DE4A WP5 Demo UI is a web application that exclusively runs in the browser. The user interface is based on the Bootstrap 4.6 framework and the system requirements for using it are documented at https://getbootstrap.com/docs/4.6/getting-started/browsers-devices/

User interface

Basic user interface of the DE4A WP5 Demo UI

The basic user interface of the DE4A WP5 Demo UI is separated in 4 different parts:

1. The “header” that contains the application title as well as the breadcrumbs
2. The “menu” that shows the available functionality – each of the menu items brings up different functionality and is described below.
3. The “content area” that contains the main interaction area with the user and shows content, depending on the chosen menu item (see step 2).
4. The “footer” contains static information and a brief disclaimer

Functionality overview

The Data Evaluator is separated into two main areas:

• the “Data Evaluator” testing part, that can be used to send real test messages

• the “Demo UI” part, that contains mainly support functionality for software developers

An overview over the available functionality is provided below, listed per menu item

• “Data Evaluator” – contains a tree with all child pages
  • “IM Exchange (User)” – initiate a new evidence request using the Intermediation (IM) pattern. The necessary field values are collected in a wizard-style approach. This is one of the key functionalities and described in further detail below. This functionality is only compatible with the Doing Business Abroad (DBA) pilot.
  • ”IM Exchange (Expert)” – initiate a new evidence request using the IM pattern. This page requires the user to provide the final XML payload to be send and offers no graphical guidance. By default, a random message is provided as the basis.
  • “USI Exchange (User)” – initiate a new evidence request using the User Supported Intermediation (USI) pattern. The necessary field values are collected in a wizard-style approach. This is one of the key functionalities and described in further detail below. This functionality is compatible with the Studying Abroad (SA) and Moving Abroad (MA) pilots.
  • “USI Exchange (Mock)” – this is a relic of an early development stage, in which we tested the redirect functionality. This page does NOT use the Connector for message exchange but uses a proprietary interface of the Mock DO. This legacy will be removed for the next iteration.

• “Demo UI” – contains a tree with all child pages
  • “Send Random Message” – this page lets you create a random message for all known piloting scenarios and send it to an arbitrary endpoint URL. This can be used to test inbound messages on DE and DO side. One key drawback here is that the message itself is not visualized.[GS3] [PH4]  This functionality will be removed for the next iteration.
  • “Send Message” – this page lets you send a user provided XML message to a customizable endpoint URL. Each provided XML message is validated against the XML schemas of the selected pilot scenario.
  • “Create Random Message” – on this page, random messages according to specific pilot scenarios can be created. The resulting XML message is displayed to the user for further use. The output of this page can e.g. be used on the “Send Message” page to send it to an HTTP endpoint.
  • “Validate Message” – on this page arbitrary XML messages can be validated against selectable pilot scenarios. This page only verifies if a message is valid or invalid according to the defined XML schemas.

The content of the pages “IM Exchange (User)” and “USI Exchange (User)” are now elaborated in more detail. As they are very similar, only the differences are outlined. The general flow for the user is like this:

1. Enter all necessary data in a web form
2. Preview the collected data before sending to give explicit consent
3. Send the request to the DO
4. Preview the received data (on DE side for the IM exchange, on DO side for the USI exchange)

The process starts with the selection of the Use Case to be performed:

Select the Use Case to perform

The next step is the selection of the Data Evaluator the user wants to test. For simplicity a set of Mock DEs is provided, from which the user can select one. Alternatively, each field can also be filled manually by the user. In a real-world application these data elements will most likely be provided as constant values by the DE application.

Provide the DE details

After the DE details have been provided, the DO details must also be provided. Here the user again has the possibility to select from a set of Mock DOs or provide the details manually. Additionally, the Mocked IDK can be queried, based on the country code of the DO. Note: on the USI exchange, also the “Data Owner redirect URL” of the DO must be provided. This field is not available for the IM exchange.

Provide the DO details

The following steps deals with the details of the “Data Request Subject”. Depending on the chosen use case this can be either a Company (Legal person) consisting of a Company ID and a Company name or a Natural person consisting of Natural Person ID, First name, Family Name and Birthday. In both cases the respective ID elements should be formatted in the eIDAS style (<Data Owner Country Code>/<Data Evaluator Country Code>/<ID>).ç

Provide the Data Request Subject details

Now all the necessary data is collected and is shown to the user for approval. This view shows both the provided data in a human readable format as well as the created XML document to help developers to create the correct field mapping. The destination to send to is especially queried in the “Target URL” field – the provided URL is the endpoint URL of the DR where it takes requests from DEs. The provided default value is the URL of the Test DR of the Playground.

Show collected data before the request is send

Before the user is able to send the request, he must check the checkbox “Confirmation to send request”, otherwise the request is not sent. This should mimic the “Explicit Request” functionality as described by the SDGR. The exact implementation of this consent functionality may also vary from DE to DE.

After providing the necessary consent, the request is sent via the DE4A Connector to the DO. Now the biggest difference between the IM and the USI pattern becomes obvious:

• When using the IM pattern, the data is transferred from the DO to the DE and then visualized at the DE, meaning that the data is transferred anyway. Hence, for this exchange pattern the Demo UI will show the preview.

• When using the USI pattern, the user is redirected to a website acting on behalf of the DO and the preview is shown there, meaning that in case of the user not approving the data usage, the data will not be transferred from DO to DE. This implies that the user needs to most likely identify himself a second time in the systems of the DO. Hence, for this exchange pattern the Playground DO will show the preview.

The following figure depicts the preview of an IM pattern request in the Demo UI. It contains the header data of DE, DO and Data Request Subject but also the “Canonical Evidence” retrieved from the DO. The Canonical evidence is again shown in both a human readable format as well as in an XML format to support software developers.

Preview of an IM pattern request in the Demo UI

Show preview

The “Accept data” and “Reject data” buttons shown in the above figure have no impact in the Demo UI, but in the real-world scenario, where this will be the indicator, if the user approves the usage of the data or not.

Data Owner

The Playground DO is deployed with example evidences that it can receive queries for. These are example identifiers from multiple different DOs. For the USI pattern the mocked DO provides a preview functionality, where the user has to consent to sending the evidence before it is sent back to the DE. Together with the DE it provides the user interfaces making the full user journey possible.

The DO preview page

Technology used

The DO uses (like the DE) the de4a-commons library for handling all the XML related things.

The DO is deployed as a stand-alone Java 11 application using the Spring Boot framework. The frontend web application needed for the preview functionality is a React.js application bundled by webpack and served by the Spring boot backend.

Data used

The DO uses a dataset of example evidence provided by the pilots. Pilot testing dataset matrix.

Deployment and configuration

Data Evaluator

The mocked DE is deployed together with the Playground DR Connector at Maribor University at https://de4a-demoui.informatika.uni-mb.si/. It is deployed behind an Nginx reverse proxy. The mock DE is configured to send messages to the Kafka server to make it easy to follow the processes when testing.

Point to TLS configuration

The Nginx reverse proxy is set up to terminate the TLS connection. The TLS certificate is from Let’s encrypt and is automatically renewed by the Certbot service.

Data Owner

The mocked DO functionality is performed by an instance of the de4a-connector-mock project [7]. The application allows to configure different parameters related to each role that the mock could assume. In this case, the DO related ones will be listed:

• Endpoints configuration

mock.do.endpoint.im=/requestExtractEvidenceIM
mock.do.endpoint.usi=/requestExtractEvidenceUSI

• USI pattern configuration

#The url to where the mock is deployed.
# Currently only used to generate the redirect url for the do preview
mock.baseurl=https://pre-smp-dr-de4a.redsara.es/de4a-mock-connector/

#the base of the path to the preview pages
mock.do.preview.endpoint.base=/preview/

#path to the preview index page, append to the base path to get the full path
mock.do.preview.endpoint.index=index

#paths to the preview websocket endpoints, append to the base path to get the full path
mock.do.preview.endpoint.websocket.socket=ws/socket
mock.do.preview.endpoint.websocket.mess=ws/messages

#paths to the preview rest server endpoints, append to the base path to get the full path
mock.do.preview.evidence.requestId.all.endpoint=request/all
mock.do.preview.evidence.redirecturl.endpoint=redirecturl/{requestId}
mock.do.preview.evidence.get.endpoint=request/{requestId}
mock.do.preview.evidence.accept.endpoint=request/{requestId}/accept
mock.do.preview.evidence.reject.endpoint=request{requestId}/reject
mock.do.preview.evidence.error.endpoint=request{requestId}/error

#path to send the dt to send the request from the do
mock.do.preview.dt.url=http://localhost:31036/de4a-connector/requestTransferEvidenceUSIDT

The configuration above will determine the paths and URLs used in the USI pattern to the previews, redirections and behaviour itself.

The mocked DO is running over an Apache Tomcat 9 server located at: https://pre-smp-dr-de4a.redsara.es/de4a-mock-connector/ For the DO the outbound connection are HTTP POST of ResponseUserRedirection data structures to the URL supplied by the DE in the RequestUserRedirection HTTP POST and if configured to send logs to the Kafka server, either directly via TCP or using the HTTP proxy.

Data Requestor

There are two Connectors deployed as a part of the Playground infrastructure. The Connector acting as DR is deployed on the University of Maribor servers and accessible at https://de4a-connector.informatika.uni-mb.si/.

Deployment and configuration

The DR connector deployed in the Playground is done so with mainly default settings and uses the built in Phase4 AS4 gateway. The only local configurations needed are the certificates used by the AS4 gateway, the URL of the local DE/DO and the Kafka logging specifics. The routing information is fetched from the IDK and SMP. The AS4 certificates for the Playground are all signed by the DE4A self-signed test CA.

Point to TLS configuration

The DR Connector is run behind a reverse proxy like Nginx that terminates the SSL/TLS connection. The certificate is a Let us Encrypt certificate that is kept up to date with Certbot. The DR has been configured to allow self-signed certificates to be able to connect to the DT.

Kafka client configuration

The DR is configured to send logs to the Kafka server directly, not using the http proxy. The topic has been set to de4a-um-si to distinguish it on the Package Tracker.

Data Transferor

The Connector acting as DT is deployed on the SGAD servers and accessible at https://pre-as4gw-dt-de4a.redsara.es/de4a-connector/.

Deployment and configuration

The DT connector deployed in the Playground is done with mainly default settings. The only local configurations needed are the certificates used by the AS4 gateway, the URL of the local DE/DO and the Kafka logging specifics. The routing information is fetched from the IDK and SMP. The AS4 certificates for the Playground are all signed by the DE4A self-signed test CA.

Point to HTTP Proxy

Due to the outgoing communication infrastructure, the Connector DT needs to establish the outgoing connections via proxy, for that purpose a proper configuration is provided by the Connector, to create the HTTP connections invoking a proxy server located at the SGAD (ES Government) systems. To perform that functionality, the following properties are configured:

http.proxy.enabled=true
http.proxy.address=proxy.redsara.es
http.proxy.port=8080
http.proxy.non-proxy=localhost|*.redsara.es

Point to TLS configuration

To create the SSL context for secure connections, the Connector provides the parameters, whose are properly configured on the deployed Connector DT instance to perform the connections though a secure tunnel, using a self-signed certificate expedited by the corresponding CA of the country.

ssl.context.enabled=true
ssl.keystore.type= PKCS12
ssl.keystore.path= ../../
ssl.keystore.password= *****
ssl.truststore.path= ./../..
ssl.truststore.password= ******

Kafka client configuration

In the Data Transferor Connector instance, a via proxy Kafka producer is configured, which mean that the messages sent must be through the HTTP protocol instead of the TCP as usually.

The HTTP protocol allows to establish a via proxy communication with the Kafka server, which is getting the proxy configuration from the parameters described above. Also, there are specific Kafka parameters on the configuration:

de4a.kafka.enabled=true
de4a.kafka.http.enabled=true
de4a.kafka.url= https://de4a.simplegob.com/kafkak-rest/
de4a.kafka.topic=de4a-es-sgad

All messages sent by the Kafka Client will be viewable by anyone going to the Package Tracker URL https://de4a.simplegob.com/package-tracker/#!detail.

Mocked IDK

Testing component used to simulate the Information Desk behaviour and provide APIs to the Playground based on the IDK definition by WP3. This mocked component comes from the necessity of the a IDK funcionality and the lack of developments around this centraliced component. Therefore WP5 mocked the funcionality to be used in the playground of the iteration 1 and 2.

To have a more realistic scenario around the Connector, a mocked IDK component has been developed to simulate the data exchanged with the component. It is just a web application that stores information about the data providers (DOs) of each evidence and country, modelled on tables. It also provides the corresponding interfaces to consume that information by an external component, following the current functional and technical definition of the IDK.

Simulated functionalities

The mocked component has no business logic, so it is basically an interface to the stored data. To describe the mocked IDK it is useful to be aware of the data model and the API REST exposed, so we present them next.

Mocked IDK data model

The mocked IDK has the following API REST interfaces available:

• /ial/{CanonicalEvidenceTypeId}: return the list of Sources for the canonical evidence type queried.
• /ial/{CanonicalEvidenceTypeId}/{CountryCode}: return the list of Sources for the canonical evidence type and country code queried.
• /provision?canonicalEvidenceTypeId={canonicalEvidenceTypeId}&dataOwnerId={dataOwnerId}: return the Provision according with the canonical evidence type and data owner identifier arguments.

Data management and dataset

The data stored in the mocked IDK is a static dataset provided from an imports SQL file inside the project. The model and data are dynamically created on each mocked IDK deployment, since it has an in-memory database. It is possible to access and modify this database accessing to a web UI console which access can be configured in the mocked IDK properties.

Playground dataset

For the Pilots on the Playground, the mocked IDK collects a combination of Data Owners with the evidence types that they provide, and the rest of the associated information. That information comes from the testing dataset

Deployment and configuration

Regarding the deployment of the mocked IDK, it is not necessary to consider any extra configuration. It is just a standalone web application that can be deployed on any application server once the war file is generated. [6] The mocked IDK deployed instance is at the UM servers, and it can be reached by accessing the URL: https://de4a-idk.informatika.uni-mb.si/

eDelivery

SML

The Service Metadata Locator (SML) is a singleton instance in the DE4A network. It is operated by CEF, a key facility of the European Commission, and provided to the DE4A project at no cost. DE4A is currently operating on the test instance of the SML, called “SMK”, and was assigned the DNS zone de4a.acc.edelivery.tech.ec.europa.eu. for the project.

Additionally, CEF provided the project 10 test SMP X.509 certificates based on the “DE4A_TEST_SMP_CA” which is based on CEFs “Connectivity Test Component CA”. Only certificates issued by the “DE4A_TEST_SMP_CA” are allowed to register in the DE4A SML DNS zone.

Every SMP (see below) that wants to join the DE4A network needs to register once at the SML using the specific DNS zone and a certificate based on the “DE4A_TEST_SMP_CA”.

Note: See the document “Request for DE4A Pilot eDelivery Certificates” on how to request an SMP certificate.

SMP

The task of a Service Metadata Publisher (SMP) is to create the link between the technical identifiers and the effective endpoint URL to exchange messages with. Additionally, it provides X.509 certificates that will be used to encrypt messages for a specific recipient. Each participant has a list of so called “Endpoints” that are uniquely identified by a combination of a document type identifier, a process identifier and a transport profile. An SMP makes it possible for two parties in the domain to exchange documents with dynamic discovery and utilizing the 4 Corner Model used for eDelivery within the DE4A project.

The DE4A Playground contains a single SMP that is deployed at the URL https://pre-smp-dr-de4a.redsara.es/sgad-smp, operated by SGAD. It is an instance of the Open Source SMP solution phoss SMP that was configured according to the project needs. At the time of writing of this document, check the participant identifiers registered at the SMP instance

Each of the participants has registered for different endpoints, and this document will not duplicate the information easily retrievable from the SMP.

All of the participants are also properly registered in the SML, so that the default CEF eDelivery Dynamic Discovery process can be used to find from the participant ID to the SMP.

The SMP uses the SMP X.509 certificate with this subject name:

1.2.840.113549.1.9.1=#16224345462d4544454c49564552592d535550504f52544065632e6575726f70612e6575,CN=SMP_DE4A_ACC_001,OU=CEF,O=DE4A,ST=Brussels-Capital,C=BE

Deployment and configuration

The Playground SMP is the phoss SMP hosted on SGAD servers at https://pre-smp-dr-de4a.redsara.es/sgad-smp. It is configured for the Playground and holds Metadata about what documents and where to send/request them among the Playground participants. It also has a certificate for signing all responses sent for requests that can be matched to a root or intermediate cert in the SMP trust store included in all deployed connectors for verification.

Point to TLS configuration

phoss SMP does not have a HTTPS mode built in but relies on the deployment server to handle SSL/TLS. In SGAD backend the phoss SMP runs in a appliaction server behind a Apache reverse proxy that handles all the SSL/TLS encryption for all requests.

phase4 AS4 Gateway

The DE4A Connector’s built-in AS4 gateway, phase4, is deployed for the Playground at UM servers and is accessible at https://de4a-connector.informatika.uni-mb.si/ and assumes the role of Data Requestor (DR) within the Playground.

Find below the relevant configurations for the Connector DR at UM side:

• application.properities

..........

# Spring As4 gateway implementation bean (provided: phase4GatewayClient and domibusGatewayClient).Implements eu.toop.as4.client.As4GatewayInterface
as4.gateway.implementation.bean=phase4GatewayClient
..........

# What AS4 implementation to use?
toop.mem.implementation = phase4

• phase4.properties

# (string) - the AS4 To/PartyId/@type value. E.g. urn:oasis:names:tc:ebcore:partyid-type:unregistered
phase4.send.toparty.id.type=urn:oasis:names:tc:ebcore:partyid-type:unregistered

# (string) - the AS4 From/PartyId/@type value. E.g. urn:oasis:names:tc:ebcore:partyid-type:unregistered
phase4.send.fromparty.id.type=urn:oasis:names:tc:ebcore:partyid-type:unregistered

When the phase4 Gateway receives a message, it starts a discovery process for the receiving party according to the eDelivery specifications with SML and SMP configured. Once found it delivers the messages to the relevant receiving party DE4A Connector.

• Phase4 gateway Specifics
  • It uses a certificate provided by the DE4A CA for signing messages.
  • It uses a certificate provided by Lets Encrypt for TLS encryption.
  • It uses the default DE4A AS4 trust store including the public CA and CA Intermediate certificates.
  • Version deployed: 1.3.0.
  • Dynamic discovery mode.
  • DE4A SML configured.

Domibus AS4 Gateway

An instance of Domibus is deployed for the Playground at SGAD servers, accessible by the URL: https://pre-as4gw-dt-de4a.redsara.es/domibus/.

In order to have a mixed environment combining both available gateways, the Connector (DT) deployed at SGAD facilities is configured to use this Domibus instance as eDelivery exchange platform.

Find bellow the configuration for the Connector DT at the SGAD side:

• application.properties

..........

# Spring As4 gateway implementation bean (provided: phase4GatewayClient and domibusGatewayClient).Implements eu.toop.as4.client.As4GatewayInterface
as4.gateway.implementation.bean=domibusGatewayClient
..........

# Domibus properties
domibus.endpoint=https://pre-as4gw-dt-de4a.redsara.es/domibus/services/backend

• phase4.properties

# (string) - the from party ID to be used for outgoing messages. Previous versions need to use toop.mem.as4.tc.partyid - starting from RC3 this property is still used as a fallback)
phase4.send.fromparty.id=pre-as4gw-dt-de4a.redsara.es

# (string) to party ID to be used for outgoing messages. Configure it in case of using Domibus without dynamic participant discovery or in the phase4 side in a mixed AS4 implementations phase4<->Domibus
phase4.send.toparty.id=

# (string) - the AS4 To/PartyId/@type value. E.g. urn:oasis:names:tc:ebcore:partyid-type:unregistered
phase4.send.toparty.id.type=urn:oasis:names:tc:ebcore:partyid-type:unregistered

# (string) - the AS4 From/PartyId/@type value. E.g. urn:oasis:names:tc:ebcore:partyid-type:unregistered
phase4.send.fromparty.id.type=urn:oasis:names:tc:ebcore:partyid-type:unregistered

Notice that the phase4.send.toparty.id is not configured, that is because is not needed when Domibus is using the Dynamic Discovering of participants. Once it receives a suitable message with the request and all the required parameters are properly configured, Domibus will start a discovering process where it asks to the SML which SMP contains the service information associated to a certain participant, so once it gets the service information, it will send out the message to that service (endpoint + certificate).

Also, it is important to mention that Domibus expose two different services:

• /services/backend

This endpoint should ONLY be exposed to the backend client(s) within the trusted zone, and it should not be exposed to the internet.

• /services/msh

Domibus exposes the Message Service Handler to be reachable by the other AS4 Access Points and it is typically exposed on the internet.

So, that is why it is configured the backend service on the Connector DT (SGAD), meanwhile on the SMP will be settled the msh service for the participants to whose Connector will be the SGAD one.

• Domibus instance characteristics
  • It uses his own certificate provided by the DE4A CA.
  • It uses the default DE4A AS4 trust store including the public CA and CA Intermediate certificates.
  • Version deployed: 4.1.6
  • Dynamic discovery mode
  • DE4A SML configured

Kafka server

The Kafka instance running on the SGAD backend is an open source distributed event streaming platform. In the DE4A project it is used to receive messages from the different services used in the project. The received messages can be viewed online as they are received in the Package Tracker made available for everyone. This is mainly for the purpose of quickly seeing if there are any problems between services and what type of problem i.e. connection problems to help finding issues for pilot partners and developers.

The main use case for this is thus distributed event logging and not digital traffic reporting where entire logs are sent and stored off site. The reasoning for this is that the Kafka instance at SGAD servers is fully open, meaning anyone can read and write messages to that service. To reduce the security risks the Kafka instance on SGAD premises does not store the sent messages for any longer periods, i.e. the messages are essentially ephemeral.

Deployment and configuration

At SGAD the Kafka instance and other services are deployed behind an Nginx reverse proxy and SSL/TLS terminator in Docker containers. For the DE4A project the particular setup uses ZooKeeper as the coordinator, Kafka as the broker of messages and the Package Tracker as a web service configured to consume and display any messages received by the Kafka broker. The Package Tracker can only display “live” i.e. any messages received in the past will never be displayed. Here a screenshot from the real system currently running on the SGAD servers:

Kafka Package Tracker

The Kafka instance is configured to accept new topics automatically to reduce manual handling of new services/pilots/partners getting involved and/or deployed within the DE4A project.

Usage of a REST API for receiving messages from a Connector through a proxy

By default, the Kafka instance at SGAD servers only communicates via the TCP protocol which created problems for SGAD. Their service runs behind a proxy server which only allows outbound HTTP(S) traffic. The resolution to this problem was to set up an additional REST interface to Kafka on SAGD backend, so that the messages can also be transmitted via the HTTP protocol.