DE4A Playground it2

The Playground is a set of predefined and pre-deployed components to be used as is. The “DE4A Playground” was designed to help pilot partners to onboard the DE4A network. It is essentially a temporary infrastructure with two purposes:

• It simulates a real scenario where the User can request data (evidence) about a subject (citizen or company). Thus, it allows testing the performance of the DE4A Connector by using mocked components and fake data.

• It allows DE4A pilot participants to deploy their own infrastructure in an orderly manner. A three-stage process was set out for the pilot participants to deploy their software components: first their own developed entities (DE/DO), secondly the DE4A Connector, and finally the SMP of the eDelivery infrastructure. At the end of each stage, the pilot partner can test their deployment by making used of the rest of the Playground components.

The target audience for this chapter are primarily software developers that try to connect their DE/DR/DT/DO system to DE4A. Bellow, a quick introduction to the main structure and list of components involved:

Playground components

The Playground consists of the following components:

• Data Evaluator (DEMO-UI)
• DE4A Connector with the role of Data Requestor, using the phase4 AS4 Gateway
• DE4A Connector with the role of Data Transferor
• Data Owner (MOCK-DO)
• SMP
• Mocked IDK
• Kafka Server

The DO and the DT are deployed at SGAD (ES). All other components, including the Kafka Server, are deployed at SGAD subcontractor facilities (ES).

Data Evaluator Mock (DEMO-UI)

The Data Evaluator deployed in the Playground infrastructure has been developed as a standalone piece of software. It comprises an interface to which every pilot infrastructure can make test calls.

Its codebase can be found at https://github.com/de4a-wp5/wp5-demo-ui/ also the deployed instance of the software is available at https://de4a-demoui-dev.informatika.uni-mb.si/

All the IEM handling functionality is taken from the shared “de4a-commons” library available at https://github.com/de4a-wp5/de4a-commons

System requirements

The DE4A WP5 Demo UI is a web application that exclusively runs in the browser. The user interface is based on the Bootstrap 4.6 framework and the system requirements for using it are documented at https://getbootstrap.com/docs/4.6/getting-started/browsers-devices/

Supported interaction patterns

The Data Evaluator was designed to simulate all the iteraction patterns available. For more information Common behaviour and interactions between components

Intermediation (IM) Pattern

The request allows to send some additional parameters, specifically requested by each DO in order to access to the corresponding evidence. This pattern is intended to exchange pieces of Canonical Evidence.

User-Supported Intermediation (USI) Pattern

This pattern is also intended to exchange pieces of Canonical Evidence, while no additional parameters are needed since the User will have the opportunity to interact directly with system of the DO.

Subscription and Notification (S&N) Pattern

Instead of evidence, a subscription message asks the DO to subscribe the DE to a Catalogue of Events in relation to the Subject of the request. The user will be notified uppon the evidence is available.

Lookup (LU) Pattern

Very similar to the IM pattern, the request allows to send some additional parameters too, and also allows to exchange pieces of Canonical Evidence.

Intermediation IM Pattern - Backwards compatibility

It provides compatibility message exchange towards iteration 1 Data Owners, so it is possible to send request from it2 Data Evaluators to it1 Data Owners.

Uses of Playground

Playground can be mainly used in two different ways:

1. For the exchange of messages between DemoUI and Mocked DO as participants involved. For this both ends will be mocked or simulated.
2. For the use of the infrastructure by a partner to act only as Data Consumer or Data Provider.

For example, if a partner wants to send a message from a mocked DE to a mock DO, a complete Data Consumer infrastructure set-up needs to be configured beforehand, i.e.:

1. An available SMP for that partner.
2. A DE4A Connector (DR)
3. A Data Evaluator (DE).

By other hand, if a partner needs to receive a message in its DO from the DemoUI, the partner needs to have been configured a full Data Consumer infrastructure, i.e.:

1. An available SMP for that partner.
2. A DE4A Connector (DT)
3. A Data Owner (DO).

Electronic Certificates

Most of the components used require the HTTPS protocol, so TLS certificates are needed. A separate document, “Policy for Transport Security”, describes the requirements on them. Additionally, the SMP and the AS4 gateways require each a separate certificate for signing and encrypting the payload. For this purpose, we required two separate Certificate Authorities (CAs) – one for the AS4 Gateway and one for the SMP component.

Certificate Authorities in the DE4A Playground

The above figure depicts the layout of the CAs. For the “DE4A Test AS4 CA”, WP5 created a new root CA that provides AS4 certificates. This simple CA can be used to issue as many certificates as we like, but it’s only suitable for piloting purposes. For a production exchange DE4A would need to search a commercial solution. For the SMP CA we needed to use a CA that matches the requirements of CEF, as the maintainers of the Service Metadata Locator (SML). To fulfil our urgent needs for a suitable CA, CEF was kind to offer us 10 SMP certificates based on their trusted “Connectivity Test CA”. In parallel to that, we promised CEF to search a commercial CA that we could use as a sustainable solution for SMP certificates. The requirements are outlined in the “Trust Models Guidance” document of CEF.

Data Evaluator

The DE and DO are both set up in the DE4A Playground. These components are deployed to act as a starting point where the pilots can interact with the Playground.

Data Evaluator

The Data Evaluator deployed in the Playground was developed as a separate piece of software.

The source code can be found at https://github.com/de4a-wp5/wp5-demo-ui and the deployed instance of this software is available at https://de4a-demoui.informatika.uni-mb.si/.

All the IEM handling functionality is taken from the shared “de4a-commons” library available at https://github.com/de4a-wp5/de4a-commons.

Note: the functionality of this tool may be extended during the further development of the project. The primary goal is to support Member States in onboarding as good as possible.

Note: the terminology used in the demo application is very simplified and may not suffice the needs of real-world applications. Real pilot applications should align the texts with their local needs.

System requirements

The DE4A WP5 Demo UI is a web application. The user interface is based on the Bootstrap 4.6 framework and the system requirements for using it are documented at https://getbootstrap.com/docs/4.6/getting-started/browsers-devices/

The backend was deployed with Java11 and Spring Framework. And is deployed on an Apache Tomcat 9.

User interface

Two type of "modes" are available for some of the patterns: Guided or Expert mode.

1. Guided mode: Allow the user to generate the request with a guided steps process.
2. Expert mode: The user can use a pregenerated XML as a request.

Guided mode steps

Main menu: it allow to select the pattern to be used.

IM Exchange (Guided) - Pilot selection.

This step allow the user to choose witch Pilot test wants to perform.

IM Exchange (Guided) - Data evaluator selection.The next step is the selection of the Data Evaluator the user wants to test.

For simplicity a set of Mock DEs is provided, from which the user can select one. Alternatively, each field can also be filled manually by the user. In a real-world application these data elements will most likely be provided as constant values by the DE application.

IM Exchange (Guided) - Data owner selection.

The next step is the selection of the Data Owner the user wants to test. Here the user again has the possibility to select from a set of Mock DOs or provide the details manually. Additionally, the Mocked IDK can be queried, based on the country code of the DO

IM Exchange (Guided) - Identity data. The following steps deals with the details of the “Data Request Subject”. Depending on the chosen use case this can be either a Company (Legal person) consisting of a Company ID and a Company name or a Natural person consisting of Natural Person ID, First name, Family Name and Birthday. In both cases the respective ID elements should be formatted in the eIDAS style (<Data Owner Country Code>/<Data Evaluator Country Code>/<ID>).

IM Exchange (Guided) - Summary and generated XML

Expert mode

Allow the user to directly paste an pregenerated request XML. The next image show an example of a USI Exchange (Expert mode).

Data Owner

The Playground DO is deployed with example evidences that it can receive queries for. These are example identifiers from multiple different DOs. For the USI pattern the mocked DO provides a preview functionality, where the user has to consent to sending the evidence before it is sent back to the DE. Together with the DE it provides the user interfaces making the full user journey possible.

The DO preview page

Technology used

The DO uses (like the DE) the de4a-commons library for handling all the XML related things.

The DO is deployed as a stand-alone Java 11 application using the Spring Boot framework. The frontend web application needed for the preview functionality is a React.js application bundled by webpack and served by the Spring boot backend.

Data used

The DO uses a dataset of example evidence provided by the pilots. Pilot testing dataset matrix.

Deployment and configuration

Data Evaluator

The mocked DE is deployed together with the Playground DR Connector at Maribor University at https://de4a-demoui.informatika.uni-mb.si/. It is deployed behind an Nginx reverse proxy. The mock DE is configured to send messages to the Kafka server to make it easy to follow the processes when testing.

Point to TLS configuration

The Nginx reverse proxy is set up to terminate the TLS connection. The TLS certificate is from Let’s encrypt and is automatically renewed by the Certbot service.

Data Owner

The mocked DO functionality is performed by an instance of the de4a-connector-mock project [7]. The application allows to configure different parameters related to each role that the mock could assume. In this case, the DO related ones will be listed:

• Endpoints configuration

mock.do.endpoint.im=/requestExtractEvidenceIM
mock.do.endpoint.usi=/requestExtractEvidenceUSI

• USI pattern configuration

#The url to where the mock is deployed.
# Currently only used to generate the redirect url for the do preview
mock.baseurl=https://pre-smp-dr-de4a.redsara.es/de4a-mock-connector/

#the base of the path to the preview pages
mock.do.preview.endpoint.base=/preview/

#path to the preview index page, append to the base path to get the full path
mock.do.preview.endpoint.index=index

#paths to the preview websocket endpoints, append to the base path to get the full path
mock.do.preview.endpoint.websocket.socket=ws/socket
mock.do.preview.endpoint.websocket.mess=ws/messages

#paths to the preview rest server endpoints, append to the base path to get the full path
mock.do.preview.evidence.requestId.all.endpoint=request/all
mock.do.preview.evidence.redirecturl.endpoint=redirecturl/{requestId}
mock.do.preview.evidence.get.endpoint=request/{requestId}
mock.do.preview.evidence.accept.endpoint=request/{requestId}/accept
mock.do.preview.evidence.reject.endpoint=request{requestId}/reject
mock.do.preview.evidence.error.endpoint=request{requestId}/error

#path to send the dt to send the request from the do
mock.do.preview.dt.url=http://localhost:31036/de4a-connector/requestTransferEvidenceUSIDT

The configuration above will determine the paths and URLs used in the USI pattern to the previews, redirections and behaviour itself.

The mocked DO is running over an Apache Tomcat 9 server located at: https://pre-smp-dr-de4a.redsara.es/de4a-mock-connector/ For the DO the outbound connection are HTTP POST of ResponseUserRedirection data structures to the URL supplied by the DE in the RequestUserRedirection HTTP POST and if configured to send logs to the Kafka server, either directly via TCP or using the HTTP proxy.

Data Requestor

There are two Connectors deployed as a part of the Playground infrastructure. The Connector acting as DR is deployed on the University of Maribor servers and accessible at https://de4a-connector.informatika.uni-mb.si/.

Deployment and configuration

The DR connector deployed in the Playground is done so with mainly default settings and uses the built in Phase4 AS4 gateway. The only local configurations needed are the certificates used by the AS4 gateway, the URL of the local DE/DO and the Kafka logging specifics. The routing information is fetched from the IDK and SMP. The AS4 certificates for the Playground are all signed by the DE4A self-signed test CA.

Point to TLS configuration

The DR Connector is run behind a reverse proxy like Nginx that terminates the SSL/TLS connection. The certificate is a Let us Encrypt certificate that is kept up to date with Certbot. The DR has been configured to allow self-signed certificates to be able to connect to the DT.

Kafka client configuration

The DR is configured to send logs to the Kafka server directly, not using the http proxy. The topic has been set to de4a-um-si to distinguish it on the Package Tracker.

Data Transferor

The Connector acting as DT is deployed on the SGAD servers and accessible at https://pre-as4gw-dt-de4a.redsara.es/de4a-connector/.

Deployment and configuration

The DT connector deployed in the Playground is done with mainly default settings. The only local configurations needed are the certificates used by the AS4 gateway, the URL of the local DE/DO and the Kafka logging specifics. The routing information is fetched from the IDK and SMP. The AS4 certificates for the Playground are all signed by the DE4A self-signed test CA.

Point to HTTP Proxy

Due to the outgoing communication infrastructure, the Connector DT needs to establish the outgoing connections via proxy, for that purpose a proper configuration is provided by the Connector, to create the HTTP connections invoking a proxy server located at the SGAD (ES Government) systems. To perform that functionality, the following properties are configured:

http.proxy.enabled=true
http.proxy.address=proxy.redsara.es
http.proxy.port=8080
http.proxy.non-proxy=localhost|*.redsara.es

Point to TLS configuration

To create the SSL context for secure connections, the Connector provides the parameters, whose are properly configured on the deployed Connector DT instance to perform the connections though a secure tunnel, using a self-signed certificate expedited by the corresponding CA of the country.

ssl.context.enabled=true
ssl.keystore.type= PKCS12
ssl.keystore.path= ../../
ssl.keystore.password= *****
ssl.truststore.path= ./../..
ssl.truststore.password= ******

Kafka client configuration

In the Data Transferor Connector instance, a via proxy Kafka producer is configured, which mean that the messages sent must be through the HTTP protocol instead of the TCP as usually.

The HTTP protocol allows to establish a via proxy communication with the Kafka server, which is getting the proxy configuration from the parameters described above. Also, there are specific Kafka parameters on the configuration:

de4a.kafka.enabled=true
de4a.kafka.http.enabled=true
de4a.kafka.url= https://de4a.simplegob.com/kafkak-rest/
de4a.kafka.topic=de4a-es-sgad

All messages sent by the Kafka Client will be viewable by anyone going to the Package Tracker URL https://de4a.simplegob.com/package-tracker/#!detail.

Mocked IDK

Testing component used to simulate the Information Desk behaviour and provide APIs to the Playground based on the IDK definition by WP3. This mocked component comes from the necessity of the a IDK funcionality and the lack of developments around this centraliced component. Therefore WP5 mocked the funcionality to be used in the playground of the iteration 1 and 2.

To have a more realistic scenario around the Connector, a mocked IDK component has been developed to simulate the data exchanged with the component. It is just a web application that stores information about the data providers (DOs) of each evidence and country, modelled on tables. It also provides the corresponding interfaces to consume that information by an external component, following the current functional and technical definition of the IDK.

Simulated functionalities

The mocked component has no business logic, so it is basically an interface to the stored data. To describe the mocked IDK it is useful to be aware of the data model and the API REST exposed, so we present them next.

Mocked IDK data model

The mocked IDK has the following API REST interfaces available:

• /ial/{CanonicalEvidenceTypeId}: return the list of Sources for the canonical evidence type queried.
• /ial/{CanonicalEvidenceTypeId}/{CountryCode}: return the list of Sources for the canonical evidence type and country code queried.
• /provision?canonicalEvidenceTypeId={canonicalEvidenceTypeId}&dataOwnerId={dataOwnerId}: return the Provision according with the canonical evidence type and data owner identifier arguments.

Data management and dataset

The data stored in the mocked IDK is a static dataset provided from an imports SQL file inside the project. The model and data are dynamically created on each mocked IDK deployment, since it has an in-memory database. It is possible to access and modify this database accessing to a web UI console which access can be configured in the mocked IDK properties.

Playground dataset

For the Pilots on the Playground, the mocked IDK collects a combination of Data Owners with the evidence types that they provide, and the rest of the associated information. That information comes from the testing dataset

Deployment and configuration

Regarding the deployment of the mocked IDK, it is not necessary to consider any extra configuration. It is just a standalone web application that can be deployed on any application server once the war file is generated. [6] The mocked IDK deployed instance is at the UM servers, and it can be reached by accessing the URL: https://de4a-idk.informatika.uni-mb.si/

eDelivery

SML

The Service Metadata Locator (SML) is a singleton instance in the DE4A network. It is operated by CEF, a key facility of the European Commission, and provided to the DE4A project at no cost. DE4A is currently operating on the test instance of the SML, called “SMK”, and was assigned the DNS zone de4a.acc.edelivery.tech.ec.europa.eu. for the project.

Additionally, CEF provided the project 10 test SMP X.509 certificates based on the “DE4A_TEST_SMP_CA” which is based on CEFs “Connectivity Test Component CA”. Only certificates issued by the “DE4A_TEST_SMP_CA” are allowed to register in the DE4A SML DNS zone.

Every SMP (see below) that wants to join the DE4A network needs to register once at the SML using the specific DNS zone and a certificate based on the “DE4A_TEST_SMP_CA”.

Note: See the document “Request for DE4A Pilot eDelivery Certificates” on how to request an SMP certificate.

SMP

The task of a Service Metadata Publisher (SMP) is to create the link between the technical identifiers and the effective endpoint URL to exchange messages with. Additionally, it provides X.509 certificates that will be used to encrypt messages for a specific recipient. Each participant has a list of so called “Endpoints” that are uniquely identified by a combination of a document type identifier, a process identifier and a transport profile. An SMP makes it possible for two parties in the domain to exchange documents with dynamic discovery and utilizing the 4 Corner Model used for eDelivery within the DE4A project.

The DE4A Playground contains a single SMP that is deployed at the URL https://pre-smp-dr-de4a.redsara.es/sgad-smp, operated by SGAD. It is an instance of the Open Source SMP solution phoss SMP that was configured according to the project needs. At the time of writing of this document, check the participant identifiers registered at the SMP instance

Each of the participants has registered for different endpoints, and this document will not duplicate the information easily retrievable from the SMP.

All of the participants are also properly registered in the SML, so that the default CEF eDelivery Dynamic Discovery process can be used to find from the participant ID to the SMP.

The SMP uses the SMP X.509 certificate with this subject name:

1.2.840.113549.1.9.1=#16224345462d4544454c49564552592d535550504f52544065632e6575726f70612e6575,CN=SMP_DE4A_ACC_001,OU=CEF,O=DE4A,ST=Brussels-Capital,C=BE

Deployment and configuration

The Playground SMP is the phoss SMP hosted on SGAD servers at https://pre-smp-dr-de4a.redsara.es/sgad-smp. It is configured for the Playground and holds Metadata about what documents and where to send/request them among the Playground participants. It also has a certificate for signing all responses sent for requests that can be matched to a root or intermediate cert in the SMP trust store included in all deployed connectors for verification.

Point to TLS configuration

phoss SMP does not have a HTTPS mode built in but relies on the deployment server to handle SSL/TLS. In SGAD backend the phoss SMP runs in a appliaction server behind a Apache reverse proxy that handles all the SSL/TLS encryption for all requests.

phase4 AS4 Gateway

The DE4A Connector’s built-in AS4 gateway, phase4, is deployed for the Playground at UM servers and is accessible at https://de4a-connector.informatika.uni-mb.si/ and assumes the role of Data Requestor (DR) within the Playground.

Find below the relevant configurations for the Connector DR at UM side:

• application.properities

..........

# Spring As4 gateway implementation bean (provided: phase4GatewayClient and domibusGatewayClient).Implements eu.toop.as4.client.As4GatewayInterface
as4.gateway.implementation.bean=phase4GatewayClient
..........

# What AS4 implementation to use?
toop.mem.implementation = phase4

• phase4.properties

# (string) - the AS4 To/PartyId/@type value. E.g. urn:oasis:names:tc:ebcore:partyid-type:unregistered
phase4.send.toparty.id.type=urn:oasis:names:tc:ebcore:partyid-type:unregistered

# (string) - the AS4 From/PartyId/@type value. E.g. urn:oasis:names:tc:ebcore:partyid-type:unregistered
phase4.send.fromparty.id.type=urn:oasis:names:tc:ebcore:partyid-type:unregistered

When the phase4 Gateway receives a message, it starts a discovery process for the receiving party according to the eDelivery specifications with SML and SMP configured. Once found it delivers the messages to the relevant receiving party DE4A Connector.

• Phase4 gateway Specifics
  • It uses a certificate provided by the DE4A CA for signing messages.
  • It uses a certificate provided by Lets Encrypt for TLS encryption.
  • It uses the default DE4A AS4 trust store including the public CA and CA Intermediate certificates.
  • Version deployed: 1.3.0.
  • Dynamic discovery mode.
  • DE4A SML configured.

Domibus AS4 Gateway

An instance of Domibus is deployed for the Playground at SGAD servers, accessible by the URL: https://pre-as4gw-dt-de4a.redsara.es/domibus/.

In order to have a mixed environment combining both available gateways, the Connector (DT) deployed at SGAD facilities is configured to use this Domibus instance as eDelivery exchange platform.

Find bellow the configuration for the Connector DT at the SGAD side:

• application.properties

..........

# Spring As4 gateway implementation bean (provided: phase4GatewayClient and domibusGatewayClient).Implements eu.toop.as4.client.As4GatewayInterface
as4.gateway.implementation.bean=domibusGatewayClient
..........

# Domibus properties
domibus.endpoint=https://pre-as4gw-dt-de4a.redsara.es/domibus/services/backend

• phase4.properties

# (string) - the from party ID to be used for outgoing messages. Previous versions need to use toop.mem.as4.tc.partyid - starting from RC3 this property is still used as a fallback)
phase4.send.fromparty.id=pre-as4gw-dt-de4a.redsara.es

# (string) to party ID to be used for outgoing messages. Configure it in case of using Domibus without dynamic participant discovery or in the phase4 side in a mixed AS4 implementations phase4<->Domibus
phase4.send.toparty.id=

# (string) - the AS4 To/PartyId/@type value. E.g. urn:oasis:names:tc:ebcore:partyid-type:unregistered
phase4.send.toparty.id.type=urn:oasis:names:tc:ebcore:partyid-type:unregistered

# (string) - the AS4 From/PartyId/@type value. E.g. urn:oasis:names:tc:ebcore:partyid-type:unregistered
phase4.send.fromparty.id.type=urn:oasis:names:tc:ebcore:partyid-type:unregistered

Notice that the phase4.send.toparty.id is not configured, that is because is not needed when Domibus is using the Dynamic Discovering of participants. Once it receives a suitable message with the request and all the required parameters are properly configured, Domibus will start a discovering process where it asks to the SML which SMP contains the service information associated to a certain participant, so once it gets the service information, it will send out the message to that service (endpoint + certificate).

Also, it is important to mention that Domibus expose two different services:

• /services/backend

This endpoint should ONLY be exposed to the backend client(s) within the trusted zone, and it should not be exposed to the internet.

• /services/msh

Domibus exposes the Message Service Handler to be reachable by the other AS4 Access Points and it is typically exposed on the internet.

So, that is why it is configured the backend service on the Connector DT (SGAD), meanwhile on the SMP will be settled the msh service for the participants to whose Connector will be the SGAD one.

• Domibus instance characteristics
  • It uses his own certificate provided by the DE4A CA.
  • It uses the default DE4A AS4 trust store including the public CA and CA Intermediate certificates.
  • Version deployed: 4.1.6
  • Dynamic discovery mode
  • DE4A SML configured

Kafka server

The Kafka instance running on the SGAD backend is an open source distributed event streaming platform. In the DE4A project it is used to receive messages from the different services used in the project. The received messages can be viewed online as they are received in the Package Tracker made available for everyone. This is mainly for the purpose of quickly seeing if there are any problems between services and what type of problem i.e. connection problems to help finding issues for pilot partners and developers.

The main use case for this is thus distributed event logging and not digital traffic reporting where entire logs are sent and stored off site. The reasoning for this is that the Kafka instance at SGAD servers is fully open, meaning anyone can read and write messages to that service. To reduce the security risks the Kafka instance on SGAD premises does not store the sent messages for any longer periods, i.e. the messages are essentially ephemeral.

Deployment and configuration

At SGAD the Kafka instance and other services are deployed behind an Nginx reverse proxy and SSL/TLS terminator in Docker containers. For the DE4A project the particular setup uses ZooKeeper as the coordinator, Kafka as the broker of messages and the Package Tracker as a web service configured to consume and display any messages received by the Kafka broker. The Package Tracker can only display “live” i.e. any messages received in the past will never be displayed. Here a screenshot from the real system currently running on the SGAD servers:

Kafka Package Tracker

The Kafka instance is configured to accept new topics automatically to reduce manual handling of new services/pilots/partners getting involved and/or deployed within the DE4A project.

Usage of a REST API for receiving messages from a Connector through a proxy

By default, the Kafka instance at SGAD servers only communicates via the TCP protocol which created problems for SGAD. Their service runs behind a proxy server which only allows outbound HTTP(S) traffic. The resolution to this problem was to set up an additional REST interface to Kafka on SAGD backend, so that the messages can also be transmitted via the HTTP protocol.