Difference between revisions of "DE4A Playground it2"
Hugo.berdon (talk | contribs) |
Hugo.berdon (talk | contribs) |
||
| Line 242: | Line 242: | ||
== IAL Directory == | == IAL Directory == | ||
| − | + | Issuing Authority Locator (IAL) component helps Data Consumers (DC) to find out the issuing authority within a particular country either to obtain a canonical evidence type or to subscribe a canonical event catalogue, and know the characteristics of the evidence provision or the subscription provision, respectively. | |
| − | + | For more information, [[DE4A Issuing Authority Locator (IAL)]] | |
| − | |||
| − | |||
| − | |||
| − | [[ | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
== eDelivery == | == eDelivery == | ||
| Line 328: | Line 309: | ||
** Dynamic discovery mode. | ** Dynamic discovery mode. | ||
** DE4A SML configured. | ** DE4A SML configured. | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
== Kafka server == | == Kafka server == | ||
Revision as of 12:44, 10 November 2022
The Playground is a set of predefined and pre-deployed components to be used as is. The “DE4A Playground” was designed to help pilot partners to onboard the DE4A network. It is essentially a temporary infrastructure with two purposes:
- It simulates a real scenario where the User can request data (evidence) about a subject (citizen or company). Thus, it allows testing the performance of the DE4A Connector by using mocked components and fake data.
- It allows DE4A pilot participants to deploy their own infrastructure in an orderly manner. A three-stage process was set out for the pilot participants to deploy their software components: first their own developed entities (DE/DO), secondly the DE4A Connector, and finally the SMP of the eDelivery infrastructure. At the end of each stage, the pilot partner can test their deployment by making used of the rest of the Playground components.
The target audience for this chapter are primarily software developers that try to connect their DE/DR/DT/DO system to DE4A. Bellow, a quick introduction to the main structure and list of components involved:
Playground components
The Playground consists of the following components:
- Data Evaluator (DEMO-UI)
- DE4A Connector with the role of Data Requestor, using the phase4 AS4 Gateway
- DE4A Connector with the role of Data Transferor
- Data Owner (MOCK-DO)
- SMP
- IAL Directory
- Kafka Server
The DO and the DT are deployed at SGAD (ES). All other components, including the Kafka Server, are deployed at SGAD subcontractor facilities (ES).
Data Evaluator Mock (DEMO-UI)
The Data Evaluator deployed in the Playground infrastructure has been developed as a standalone piece of software. It comprises an interface to which every pilot infrastructure can make test calls.
Its codebase can be found at https://github.com/de4a-wp5/wp5-demo-ui/ also the deployed instance of the software is available at https://de4a-demoui-dev.informatika.uni-mb.si/
All the IEM handling functionality is taken from the shared “de4a-commons” library available at https://github.com/de4a-wp5/de4a-commons
System requirements
The DE4A WP5 Demo UI is a web application that exclusively runs in the browser. The user interface is based on the Bootstrap 4.6 framework and the system requirements for using it are documented at https://getbootstrap.com/docs/4.6/getting-started/browsers-devices/
User interface
Supported interaction patterns
The Data Evaluator was designed to simulate all the iteraction patterns available. For more information Common behaviour and interactions between components
Intermediation (IM) Pattern
The request allows to send some additional parameters, specifically requested by each DO in order to access to the corresponding evidence. This pattern is intended to exchange pieces of Canonical Evidence.
User-Supported Intermediation (USI) Pattern
This pattern is also intended to exchange pieces of Canonical Evidence, while no additional parameters are needed since the User will have the opportunity to interact directly with system of the DO.
Subscription and Notification (S&N) Pattern
Instead of evidence, a subscription message asks the DO to subscribe the DE to a Catalogue of Events in relation to the Subject of the request. The user will be notified uppon the evidence is available.
Lookup (LU) Pattern
Very similar to the IM pattern, the request allows to send some additional parameters too, and also allows to exchange pieces of Canonical Evidence.
Intermediation IM Pattern - Backwards compatibility
It provides compatibility message exchange towards iteration 1 Data Owners, so it is possible to send request from it2 Data Evaluators to it1 Data Owners.
Uses of Playground
Playground can be mainly used in two different ways:
- For the exchange of messages between DemoUI and Mocked DO as participants involved. For this both ends will be mocked or simulated.
- For the use of the infrastructure by a partner to act only as Data Consumer or Data Provider.
For example, if a partner wants to send a message from a mocked DE to a mock DO, a complete Data Consumer infrastructure set-up needs to be configured beforehand, i.e.:
- An available SMP for that partner.
- A DE4A Connector (DR)
- A Data Evaluator (DE).
By other hand, if a partner needs to receive a message in its DO from the DemoUI, the partner needs to have been configured a full Data Consumer infrastructure, i.e.:
- An available SMP for that partner.
- A DE4A Connector (DT)
- A Data Owner (DO).
Electronic Certificates
Most of the components used require the HTTPS protocol, so TLS certificates are needed. A separate document, “Policy for Transport Security”, describes the requirements on them. Additionally, the SMP and the AS4 gateways require each a separate certificate for signing and encrypting the payload. For this purpose, we required two separate Certificate Authorities (CAs) – one for the AS4 Gateway and one for the SMP component.
The above figure depicts the layout of the CAs. For the “DE4A Test AS4 CA”, WP5 created a new root CA that provides AS4 certificates. This simple CA can be used to issue as many certificates as we like, but it’s only suitable for piloting purposes. For a production exchange DE4A would need to search a commercial solution. For the SMP CA we needed to use a CA that matches the requirements of CEF, as the maintainers of the Service Metadata Locator (SML). To fulfil our urgent needs for a suitable CA, CEF was kind to offer us 10 SMP certificates based on their trusted “Connectivity Test CA”. In parallel to that, we promised CEF to search a commercial CA that we could use as a sustainable solution for SMP certificates. The requirements are outlined in the “Trust Models Guidance” document of CEF.
Data Evaluator
The DE and DO are both set up in the DE4A Playground. These components are deployed to act as a starting point where the pilots can interact with the Playground.
Data Evaluator
The Data Evaluator deployed in the Playground was developed as a separate piece of software.
The source code can be found at https://github.com/de4a-wp5/wp5-demo-ui and the deployed instance of this software is available at https://de4a-demoui.informatika.uni-mb.si/.
All the IEM handling functionality is taken from the shared “de4a-commons” library available at https://github.com/de4a-wp5/de4a-commons.
Note: the functionality of this tool may be extended during the further development of the project. The primary goal is to support Member States in onboarding as good as possible.
Note: the terminology used in the demo application is very simplified and may not suffice the needs of real-world applications. Real pilot applications should align the texts with their local needs.
System requirements
The DE4A WP5 Demo UI is a web application. The user interface is based on the Bootstrap 4.6 framework and the system requirements for using it are documented at https://getbootstrap.com/docs/4.6/getting-started/browsers-devices/
The backend was deployed with Java11 and Spring Framework. And is deployed on an Apache Tomcat 9.
User interface
Two type of "modes" are available for some of the patterns: Guided or Expert mode.
- Guided mode: Allow the user to generate the request with a guided steps process.
- Expert mode: The user can use a pregenerated XML as a request.
Guided mode steps
Main menu: it allow to select the pattern to be used.
IM Exchange (Guided) - Pilot selection.
This step allow the user to choose witch Pilot test wants to perform.
_-_Pilot_selection..png)
IM Exchange (Guided) - Data evaluator selection.The next step is the selection of the Data Evaluator the user wants to test.
For simplicity a set of Mock DEs is provided, from which the user can select one. Alternatively, each field can also be filled manually by the user. In a real-world application these data elements will most likely be provided as constant values by the DE application.
_-_Data_Evaluator_selection..png)
IM Exchange (Guided) - Data owner selection.
The next step is the selection of the Data Owner the user wants to test. Here the user again has the possibility to select from a set of Mock DOs or provide the details manually. Additionally, the Mocked IDK can be queried, based on the country code of the DO
_-_Data_Owner_selection..png)
IM Exchange (Guided) - Identity data.
The following steps deals with the details of the “Data Request Subject”. Depending on the chosen use case this can be either a Company (Legal person) consisting of a Company ID and a Company name or a Natural person consisting of Natural Person ID, First name, Family Name and Birthday. In both cases the respective ID elements should be formatted in the eIDAS style (<Data Owner Country Code>/<Data Evaluator Country Code>/<ID>).
_-_Data_indetity_data..png)
IM Exchange (Guided) - Summary and generated XML
_-_Summary_and_generated_XML.png)
Expert mode
Allow the user to directly paste an pregenerated request XML. The next image show an example of a USI Exchange (Expert mode).
_.png)
Data Owner
The Playground DO is deployed with example evidences that it can receive queries for. These are example identifiers from multiple different DOs. For the USI pattern the mocked DO provides a preview functionality, where the user has to consent to sending the evidence before it is sent back to the DE. Together with the DE it provides the user interfaces making the full user journey possible.
Technology used
The DO uses (like the DE) the de4a-commons library for handling all the XML related things.
The DO is deployed as a stand-alone Java 11 application using the Spring Boot framework. The frontend web application needed for the preview functionality is a React.js application bundled by webpack and served by the Spring boot backend.
Data used
The DO uses a dataset of example evidence provided by the pilots. Pilot testing dataset matrix.
Deployment and configuration
Data Evaluator
The mocked DE is deployed together with the Playground DR Connector at Maribor University at https://de4a-demoui.informatika.uni-mb.si/. It is deployed behind an Nginx reverse proxy. The mock DE is configured to send messages to the Kafka server to make it easy to follow the processes when testing.
Point to TLS configuration
The Nginx reverse proxy is set up to terminate the TLS connection. The TLS certificate is from Let’s encrypt and is automatically renewed by the Certbot service.
Data Owner
The mocked DO functionality is performed by an instance of the de4a-connector-mock project [7]. The application allows to configure different parameters related to each role that the mock could assume. In this case, the DO related ones will be listed:
- Endpoints configuration
mock.do.endpoint.im=/requestExtractEvidenceIM mock.do.endpoint.usi=/requestExtractEvidenceUSI
- USI pattern configuration
#The url to where the mock is deployed.
# Currently only used to generate the redirect url for the do preview
mock.baseurl=https://pre-smp-dr-de4a.redsara.es/de4a-mock-connector/
#the base of the path to the preview pages
mock.do.preview.endpoint.base=/preview/
#path to the preview index page, append to the base path to get the full path
mock.do.preview.endpoint.index=index
#paths to the preview websocket endpoints, append to the base path to get the full path
mock.do.preview.endpoint.websocket.socket=ws/socket
mock.do.preview.endpoint.websocket.mess=ws/messages
#paths to the preview rest server endpoints, append to the base path to get the full path
mock.do.preview.evidence.requestId.all.endpoint=request/all
mock.do.preview.evidence.redirecturl.endpoint=redirecturl/{requestId}
mock.do.preview.evidence.get.endpoint=request/{requestId}
mock.do.preview.evidence.accept.endpoint=request/{requestId}/accept
mock.do.preview.evidence.reject.endpoint=request{requestId}/reject
mock.do.preview.evidence.error.endpoint=request{requestId}/error
#path to send the dt to send the request from the do
mock.do.preview.dt.url=http://localhost:31036/de4a-connector/requestTransferEvidenceUSIDT
The configuration above will determine the paths and URLs used in the USI pattern to the previews, redirections and behaviour itself.
The mocked DO is running over an Apache Tomcat 9 server located at: https://pre-smp-dr-de4a.redsara.es/de4a-mock-connector/ For the DO the outbound connection are HTTP POST of ResponseUserRedirection data structures to the URL supplied by the DE in the RequestUserRedirection HTTP POST and if configured to send logs to the Kafka server, either directly via TCP or using the HTTP proxy.
Data Requestor
There are two Connectors deployed as a part of the Playground infrastructure. The Connector acting as DR is deployed on the University of Maribor servers and accessible at https://de4a-connector.informatika.uni-mb.si/.
Deployment and configuration
The DR connector deployed in the Playground is done so with mainly default settings and uses the built in Phase4 AS4 gateway. The only local configurations needed are the certificates used by the AS4 gateway, the URL of the local DE/DO and the Kafka logging specifics. The routing information is fetched from the IDK and SMP. The AS4 certificates for the Playground are all signed by the DE4A self-signed test CA.
Point to TLS configuration
The DR Connector is run behind a reverse proxy like Nginx that terminates the SSL/TLS connection. The certificate is a Let us Encrypt certificate that is kept up to date with Certbot. The DR has been configured to allow self-signed certificates to be able to connect to the DT.
Kafka client configuration
The DR is configured to send logs to the Kafka server directly, not using the http proxy. The topic has been set to de4a-um-si to distinguish it on the Package Tracker.
Data Transferor
The Connector acting as DT is deployed on the SGAD servers and accessible at https://pre-as4gw-dt-de4a.redsara.es/de4a-connector/.
Deployment and configuration
The DT connector deployed in the Playground is done with mainly default settings. The only local configurations needed are the certificates used by the AS4 gateway, the URL of the local DE/DO and the Kafka logging specifics. The routing information is fetched from the IDK and SMP. The AS4 certificates for the Playground are all signed by the DE4A self-signed test CA.
Point to HTTP Proxy
Due to the outgoing communication infrastructure, the Connector DT needs to establish the outgoing connections via proxy, for that purpose a proper configuration is provided by the Connector, to create the HTTP connections invoking a proxy server located at the SGAD (ES Government) systems. To perform that functionality, the following properties are configured:
http.proxy.enabled=true http.proxy.address=proxy.redsara.es http.proxy.port=8080 http.proxy.non-proxy=localhost|*.redsara.es
Point to TLS configuration
To create the SSL context for secure connections, the Connector provides the parameters, whose are properly configured on the deployed Connector DT instance to perform the connections though a secure tunnel, using a self-signed certificate expedited by the corresponding CA of the country.
ssl.context.enabled=true ssl.keystore.type= PKCS12 ssl.keystore.path= ../../ ssl.keystore.password= ***** ssl.truststore.path= ./../.. ssl.truststore.password= ******
Kafka client configuration
In the Data Transferor Connector instance, a via proxy Kafka producer is configured, which mean that the messages sent must be through the HTTP protocol instead of the TCP as usually.
The HTTP protocol allows to establish a via proxy communication with the Kafka server, which is getting the proxy configuration from the parameters described above. Also, there are specific Kafka parameters on the configuration:
de4a.kafka.enabled=true de4a.kafka.http.enabled=true de4a.kafka.url= https://de4a.simplegob.com/kafkak-rest/ de4a.kafka.topic=de4a-es-sgad
All messages sent by the Kafka Client will be viewable by anyone going to the Package Tracker URL https://de4a.simplegob.com/package-tracker/#!detail.
IAL Directory
Issuing Authority Locator (IAL) component helps Data Consumers (DC) to find out the issuing authority within a particular country either to obtain a canonical evidence type or to subscribe a canonical event catalogue, and know the characteristics of the evidence provision or the subscription provision, respectively.
For more information, DE4A Issuing Authority Locator (IAL)
eDelivery
SML
The Service Metadata Locator (SML) is a singleton instance in the DE4A network. It is operated by CEF, a key facility of the European Commission, and provided to the DE4A project at no cost. DE4A is currently operating on the test instance of the SML, called “SMK”, and was assigned the DNS zone de4a.acc.edelivery.tech.ec.europa.eu. for the project.
Additionally, CEF provided the project 10 test SMP X.509 certificates based on the “DE4A_TEST_SMP_CA” which is based on CEFs “Connectivity Test Component CA”. Only certificates issued by the “DE4A_TEST_SMP_CA” are allowed to register in the DE4A SML DNS zone.
Every SMP (see below) that wants to join the DE4A network needs to register once at the SML using the specific DNS zone and a certificate based on the “DE4A_TEST_SMP_CA”.
Note: See the document “Request for DE4A Pilot eDelivery Certificates” on how to request an SMP certificate.
SMP
The task of a Service Metadata Publisher (SMP) is to create the link between the technical identifiers and the effective endpoint URL to exchange messages with. Additionally, it provides X.509 certificates that will be used to encrypt messages for a specific recipient. Each participant has a list of so called “Endpoints” that are uniquely identified by a combination of a document type identifier, a process identifier and a transport profile. An SMP makes it possible for two parties in the domain to exchange documents with dynamic discovery and utilizing the 4 Corner Model used for eDelivery within the DE4A project.
The DE4A Playground contains a single SMP that is deployed at the URL https://pre-smp-dr-de4a.redsara.es/sgad-smp, operated by SGAD. It is an instance of the Open Source SMP solution phoss SMP that was configured according to the project needs. At the time of writing of this document, check the participant identifiers registered at the SMP instance
Each of the participants has registered for different endpoints, and this document will not duplicate the information easily retrievable from the SMP.
All of the participants are also properly registered in the SML, so that the default CEF eDelivery Dynamic Discovery process can be used to find from the participant ID to the SMP.
The SMP uses the SMP X.509 certificate with this subject name:
1.2.840.113549.1.9.1=#16224345462d4544454c49564552592d535550504f52544065632e6575726f70612e6575,CN=SMP_DE4A_ACC_001,OU=CEF,O=DE4A,ST=Brussels-Capital,C=BE
Deployment and configuration
The Playground SMP is the phoss SMP hosted on SGAD servers at https://pre-smp-dr-de4a.redsara.es/sgad-smp. It is configured for the Playground and holds Metadata about what documents and where to send/request them among the Playground participants. It also has a certificate for signing all responses sent for requests that can be matched to a root or intermediate cert in the SMP trust store included in all deployed connectors for verification.
Point to TLS configuration
phoss SMP does not have a HTTPS mode built in but relies on the deployment server to handle SSL/TLS. In SGAD backend the phoss SMP runs in a appliaction server behind a Apache reverse proxy that handles all the SSL/TLS encryption for all requests.
phase4 AS4 Gateway
The DE4A Connector’s built-in AS4 gateway, phase4, is deployed for the Playground at UM servers and is accessible at https://de4a-connector.informatika.uni-mb.si/ and assumes the role of Data Requestor (DR) within the Playground.
Find below the relevant configurations for the Connector DR at UM side:
- application.properities
.......... # Spring As4 gateway implementation bean (provided: phase4GatewayClient and domibusGatewayClient).Implements eu.toop.as4.client.As4GatewayInterface as4.gateway.implementation.bean=phase4GatewayClient .......... # What AS4 implementation to use? toop.mem.implementation = phase4
- phase4.properties
# (string) - the AS4 To/PartyId/@type value. E.g. urn:oasis:names:tc:ebcore:partyid-type:unregistered phase4.send.toparty.id.type=urn:oasis:names:tc:ebcore:partyid-type:unregistered # (string) - the AS4 From/PartyId/@type value. E.g. urn:oasis:names:tc:ebcore:partyid-type:unregistered phase4.send.fromparty.id.type=urn:oasis:names:tc:ebcore:partyid-type:unregistered
When the phase4 Gateway receives a message, it starts a discovery process for the receiving party according to the eDelivery specifications with SML and SMP configured. Once found it delivers the messages to the relevant receiving party DE4A Connector.
- Phase4 gateway Specifics
- It uses a certificate provided by the DE4A CA for signing messages.
- It uses a certificate provided by Lets Encrypt for TLS encryption.
- It uses the default DE4A AS4 trust store including the public CA and CA Intermediate certificates.
- Version deployed: 1.3.0.
- Dynamic discovery mode.
- DE4A SML configured.
Kafka server
The Kafka instance running on the SGAD backend is an open source distributed event streaming platform. In the DE4A project it is used to receive messages from the different services used in the project. The received messages can be viewed online as they are received in the Package Tracker made available for everyone. This is mainly for the purpose of quickly seeing if there are any problems between services and what type of problem i.e. connection problems to help finding issues for pilot partners and developers.
The main use case for this is thus distributed event logging and not digital traffic reporting where entire logs are sent and stored off site. The reasoning for this is that the Kafka instance at SGAD servers is fully open, meaning anyone can read and write messages to that service. To reduce the security risks the Kafka instance on SGAD premises does not store the sent messages for any longer periods, i.e. the messages are essentially ephemeral.
Deployment and configuration
At SGAD the Kafka instance and other services are deployed behind an Nginx reverse proxy and SSL/TLS terminator in Docker containers. For the DE4A project the particular setup uses ZooKeeper as the coordinator, Kafka as the broker of messages and the Package Tracker as a web service configured to consume and display any messages received by the Kafka broker. The Package Tracker can only display “live” i.e. any messages received in the past will never be displayed. Here a screenshot from the real system currently running on the SGAD servers:
The Kafka instance is configured to accept new topics automatically to reduce manual handling of new services/pilots/partners getting involved and/or deployed within the DE4A project.
Usage of a REST API for receiving messages from a Connector through a proxy
By default, the Kafka instance at SGAD servers only communicates via the TCP protocol which created problems for SGAD. Their service runs behind a proxy server which only allows outbound HTTP(S) traffic. The resolution to this problem was to set up an additional REST interface to Kafka on SAGD backend, so that the messages can also be transmitted via the HTTP protocol.