Difference between revisions of "Usage of third party specifications and components"

From DE4A
Jump to navigation Jump to search
 
(6 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== Overview ==
+
The DE4A project is based on some specifications and components that allow building new software with some inherited capabilities such as security, reliability, integrity and scalability. Some of the main technologies involved are described here.
DE4A project is based on some specifications and components that allows to build new software with some inherited capabilities as security, reliability, integrity and scalability.
 
 
 
At this point some of the main technologies involved will be described.
 
  
 
== eDelivery ==
 
== eDelivery ==
The CEF [https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/How+does+eDelivery+work eDelivery] Building Block helps users to exchange electronic data and documents with one another in a reliable and trusted way.
+
The CEF [https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/How+does+eDelivery+work eDelivery] Building Block helps users to exchange electronic data and documents with each other in a reliable and trusted way.
  
The CEF eDelivery solution is based on a distributed model called the “4-corner model”. In this model, the back-end systems of the users don't exchange data directly with each other but do this through [https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/Access+Point+software Access Points]. These Access Points are conformant to the same [https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/Access+Point+specifications technical specifications] and therefore capable of communicating with each other.
+
The CEF eDelivery solution is based on a distributed model called the “4-corner model”. In this model, users' back-end systems do not exchange data directly with each other, but via [https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/Access+Point+software Access Points]. These Access Points comply with to the same [https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/Access+Point+specifications technical specifications] and are therefore able to communicate with each other.
  
As a result of this, users adopting CEF eDelivery can easily and safely exchange data even if their IT systems were developed independently from each other.
+
In this way, users adopting the CEF eDelivery can exchange data easily and securely, even if their IT systems were developed independently.
  
 
===SML===
 
===SML===
The Service Metadata Locator (SML) is a singleton instance in the DE4A network. It is operated by CEF, a key facility of the European Commission, and provided to the DE4A project at no cost. DE4A is currently operating on the test instance of the SML, called "SMK", and was assigned the DNS zone de4a.acc.edelivery.tech.ec.europa.eu. for the project.
+
The Service Metadata Locator (SML) is a singleton instance in the DE4A network. It is operated by CEF, a key facility of the European Commission, and is provided to the DE4A project free of charge. DE4A is currently operating on the test instance of the SML, called "SMK", and was assigned the DNS zone of4a.acc.edelivery.tech.ec.europa.eu. for the project.
  
Additionally, CEF provided the project 10 test SMP X.509 certificates based on the "DE4A_TEST_SMP_CA" which is based on CEFs "Connectivity Test Component CA". Only certificates issued by the "DE4A_TEST_SMP_CA" are allowed to register in the DE4A SML DNS zone.
+
In addition, CEF provided the project with 10 SMP X.509 test certificates based on the "DE4A_TEST_SMP_CA", which is based on CEF's "Connectivity Test Component CA". Only certificates issued by the "DE4A_TEST_SMP_CA" can be registered in the DNS zone of the DE4A SML.
  
Every SMP (see below) that wants to join the DE4A network needs to register once at the SML using the specific DNS zone and a certificate based on the "DE4A_TEST_SMP_CA".
+
Each SMP (see below) that wants to join the DE4A network needs to register once in the SML using the specific DNS zone and a certificate based on the "DE4A_TEST_SMP_CA".
  
 +
In the case of DE4A project, the Connector handles the interaction with the ''"service metadata resources"'', there are two function modes depending of the configuration in the Connector:
 +
 +
'''Establishing a certain SMP'''
 +
*The Connector will request all the services information to the SMP configured.
 +
'''Without SMP definition'''
 +
*The Connector will request to the SML in order to get the SMP location which contains services information for the certain participant given.
 
===SMP===
 
===SMP===
The task of a Service Metadata Publisher (SMP) is to create the link between the technical identifiers and the effective endpoint URL to exchange messages with. Additionally, it provides X.509 certificates that will be used to encrypt messages for a specific recipient. Each participant has a list of so called “Endpoints” that are uniquely identified by a combination of a document type identifier, a process identifier and a transport profile. An SMP makes it possible for two parties in the domain to exchange documents with dynamic discovery and utilizing the 4 Corner Model used for eDelivery within the DE4A project.
+
The task of a Service Metadata Publisher (SMP) is to create the link between the technical identifiers and the URL of the actual endpoint with which to exchange messages. In addition, it provides the X.509 certificates that will be used to encrypt the messages for a specific recipient. Each participant has a list of so-called "Endpoints" that are uniquely identified by a combination of a document type identifier, a process identifier and a transport profile. An SMP makes it possible for two domain parties to exchange documents with dynamic discovery and utilizing the 4-Corner Model used in eDelivery within the DE4A project.
  
The information has been structured as follows, based on the SMP schemas outlines:
+
The information has been structured as follows, based on the SMP schemas outlined:
 
{| class="wikitable"
 
{| class="wikitable"
 
! SMP filed name
 
! SMP filed name
Line 36: Line 39:
 
|Canonical Evidence Type
 
|Canonical Evidence Type
 
|Type of canonical evidence
 
|Type of canonical evidence
|Scheme: urn:de4a-eu:CanonicalEvidenceType - Value: CompanyRegistration
+
|Scheme: <nowiki>urn:de4a-eu:CanonicalEvidenceType</nowiki> - Value: CompanyRegistration
 
|-
 
|-
 
|ProcessIdentifier
 
|ProcessIdentifier
 
|Message Type
 
|Message Type
 
|Action type (request/response) for a message
 
|Action type (request/response) for a message
|Scheme: urn:de4a-eu:MessageType - Value: request
+
|Scheme: <nowiki>urn:de4a-eu:MessageType</nowiki> - Value: request
 
|-
 
|-
 
|EndpointURI
 
|EndpointURI
 
|AS4 endpoint
 
|AS4 endpoint
 
|Endpoint where the AS4 message will be sent for a certain participant
 
|Endpoint where the AS4 message will be sent for a certain participant
|https://de4a-dev-connector.egovlab.eu/phase4
+
|https://de4a-connector.informatika.uni-mb.si/
 
|-
 
|-
 
|Certificate
 
|Certificate
Line 53: Line 56:
 
|MIIFfzCCA2egAwIBAgICEAkwDQYJKoZIhvcNAQELBQAwajELMAk...(more)
 
|MIIFfzCCA2egAwIBAgICEAkwDQYJKoZIhvcNAQELBQAwajELMAk...(more)
 
|}
 
|}
 
+
The requests to the SMP/SML components are performed using the [https://github.com/phax/peppol-commons peppol-commons] ''(by Phax'') libraries, based on the CEF Specifications:
 +
*[https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/eDelivery+BDXL eDelivery BDXL]
 +
*[https://oasis-open.org/committees/tc_home.php?wg_abbrev=bdxr OASIS BDXR]
 +
===AS4 Gateway===
 +
The message exchange is built on the AS4 protocol, which is an open standard for secure, payload-agnostic business-to-business document exchange via web services. Secure document exchange is governed by WS-Security aspects, including XML Encryption and XML Digital Signatures. There are many implementations of AS4. The Connector includes an implementation of an AS4 gateway ('''phase4''') using the following third-party libraries and components:
 +
*[https://github.com/TOOP4EU/toop-connector-ng Toop Connector NG]
 +
*[https://github.com/phax/phase4 phax/phase4 libraries]
 +
Based on the specifications:
 +
*[https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/eDelivery+AS4+-+1.14 eDelivery AS4]
 +
*[https://docs.peppol.eu/edelivery/as4/specification/ PEPPOL AS4 Profile]
 
==RegRep==
 
==RegRep==
 
ebXML RegRep is a standard defining the service interfaces, protocols and information model for an integrated registry and repository. The repository stores digital content while the registry stores metadata that describes the content in the repository.
 
ebXML RegRep is a standard defining the service interfaces, protocols and information model for an integrated registry and repository. The repository stores digital content while the registry stores metadata that describes the content in the repository.
 
An ebXML Registry is capable of storing any type of electronic content such as XML documents, text documents, images, sound and video. Instances of such content are referred to as a RepositorytItems.  RepositorytItems are stored in a content repository provided by the ebXML Registry.
 
An ebXML Registry is capable of storing any type of electronic content such as XML documents, text documents, images, sound and video. Instances of such content are referred to as a RepositorytItems.  RepositorytItems are stored in a content repository provided by the ebXML Registry.
  
RegRep usage at DE4A project has been limitated to a minimal approach due to the low benefits against the "plain" message, also, until the moment, it there was not definition about the RegRep usage. Nevertheless, the eDelivery implementation at the Connector includes the the message wrapping into a RegRep specification with basic slots organization and the message obtained is sended via AS4 as a payload.
+
RegRep usage at DE4A project has been limited to a minimal approach due to the low benefits compared to a "plain" message. Nevertheless, the eDelivery implementation at the Connector includes the message wrapping into a RegRep specification with basic slots organization and the message obtained is sent via AS4 as a payload.
  
 
==Kafka==
 
==Kafka==
The [https://www.confluent.io/what-is-apache-kafka/?utm_medium=sem&utm_source=google&utm_campaign=ch.sem_br.nonbrand_tp.prs_tgt.kafka_mt.xct_rgn.emea_lng.eng_dv.all_con.kafka-general&utm_term=apache%20kafka&creative=&device=c&placement=&gclid=CjwKCAjw95yJBhAgEiwAmRrutDb2Nv4YGzhiVGoi0LGOmtnAC1kWCAgrggycC6EB-brGTfPJt1PughoCHT0QAvD_BwE Apache Kafka] instance running at the DE4A facilities is an open source distributed event streaming platform. In the DE4A project it is used to receive messages from the different services used in the project. The received messages can be viewed online as they are received in the Package Tracker made available for everyone. This is mainly for the purpose of quickly seeing if there are any problems between services and what type of problem i.e. connection problems to help finding issues for pilot partners and developers.
+
The [https://www.confluent.io/what-is-apache-kafka/?utm_medium=sem&utm_source=google&utm_campaign=ch.sem_br.nonbrand_tp.prs_tgt.kafka_mt.xct_rgn.emea_lng.eng_dv.all_con.kafka-general&utm_term=apache%20kafka&creative=&device=c&placement=&gclid=CjwKCAjw95yJBhAgEiwAmRrutDb2Nv4YGzhiVGoi0LGOmtnAC1kWCAgrggycC6EB-brGTfPJt1PughoCHT0QAvD_BwE Apache Kafka] instance used in DE4A is an open source distributed event streaming platform. It is used to receive messages from the different services used in the project. The received messages can be viewed online as they are received in the Package Tracker made available for everyone. The main purpose is to be able to quickly see if there are any problems between services and what type of problem (e.g. connection problems) to help pilot partners and developers find those issues.
The main use case for this is thus distributed event logging and not digital traffic reporting where entire logs are sent and stored off site. The reasoning for this is that the Kafka instance at eGovlab servers is fully open, meaning anyone can read and write messages to that service. To reduce the security risks the Kafka instance on eGovlab premises does not store the sent messages for any longer periods, i.e. the messages are essentially ephemeral.
+
Therefore, the main use case is distributed event logging and not digital traffic reporting where full logs are sent and stored off-site. The reasoning for this is that the Kafka instance on the eGovlab servers is fully open, meaning that anyone can read and write messages on that service. To reduce security risks, the Kafka instance on the eGovlab premises does not store messages sent for extended periods of time, i.e. messages are essentially ephemeral.
  
For the DE4A project the particular setup uses ZooKeeper as the coordinator, Kafka as the broker of messages and the Package Tracker as a web service configured to consume and display any messages received by the Kafka broker. The Package Tracker can only display “live” i.e. any messages received in the past will never be displayed.
+
The particular configuration used for the DE4A project uses ZooKeeper as the coordinator, Kafka as the message broker and the Package Tracker as the web service configured to consume and display any messages received by the Kafka broker. The Package Tracker can only display "live" messages, i.e. any messages received in the past will never be displayed.
 +
[[Category:Wip]]

Latest revision as of 13:55, 22 February 2022

The DE4A project is based on some specifications and components that allow building new software with some inherited capabilities such as security, reliability, integrity and scalability. Some of the main technologies involved are described here.

eDelivery

The CEF eDelivery Building Block helps users to exchange electronic data and documents with each other in a reliable and trusted way.

The CEF eDelivery solution is based on a distributed model called the “4-corner model”. In this model, users' back-end systems do not exchange data directly with each other, but via Access Points. These Access Points comply with to the same technical specifications and are therefore able to communicate with each other.

In this way, users adopting the CEF eDelivery can exchange data easily and securely, even if their IT systems were developed independently.

SML

The Service Metadata Locator (SML) is a singleton instance in the DE4A network. It is operated by CEF, a key facility of the European Commission, and is provided to the DE4A project free of charge. DE4A is currently operating on the test instance of the SML, called "SMK", and was assigned the DNS zone of4a.acc.edelivery.tech.ec.europa.eu. for the project.

In addition, CEF provided the project with 10 SMP X.509 test certificates based on the "DE4A_TEST_SMP_CA", which is based on CEF's "Connectivity Test Component CA". Only certificates issued by the "DE4A_TEST_SMP_CA" can be registered in the DNS zone of the DE4A SML.

Each SMP (see below) that wants to join the DE4A network needs to register once in the SML using the specific DNS zone and a certificate based on the "DE4A_TEST_SMP_CA".

In the case of DE4A project, the Connector handles the interaction with the "service metadata resources", there are two function modes depending of the configuration in the Connector:

Establishing a certain SMP

  • The Connector will request all the services information to the SMP configured.

Without SMP definition

  • The Connector will request to the SML in order to get the SMP location which contains services information for the certain participant given.

SMP

The task of a Service Metadata Publisher (SMP) is to create the link between the technical identifiers and the URL of the actual endpoint with which to exchange messages. In addition, it provides the X.509 certificates that will be used to encrypt the messages for a specific recipient. Each participant has a list of so-called "Endpoints" that are uniquely identified by a combination of a document type identifier, a process identifier and a transport profile. An SMP makes it possible for two domain parties to exchange documents with dynamic discovery and utilizing the 4-Corner Model used in eDelivery within the DE4A project.

The information has been structured as follows, based on the SMP schemas outlined:

SMP filed name DE4A parameter Description Example
ParticipantIdentifier Participant ID Unique identifier assigned to each entity involved (DE's/DO's) Scheme: iso6523-actorid-upis - Value: 9999:egov
DocumentIdentifier Canonical Evidence Type Type of canonical evidence Scheme: urn:de4a-eu:CanonicalEvidenceType - Value: CompanyRegistration
ProcessIdentifier Message Type Action type (request/response) for a message Scheme: urn:de4a-eu:MessageType - Value: request
EndpointURI AS4 endpoint Endpoint where the AS4 message will be sent for a certain participant https://de4a-connector.informatika.uni-mb.si/
Certificate AS4 public key certificate Base64 X.509 certificate for the AS4 gateway MIIFfzCCA2egAwIBAgICEAkwDQYJKoZIhvcNAQELBQAwajELMAk...(more)

The requests to the SMP/SML components are performed using the peppol-commons (by Phax) libraries, based on the CEF Specifications:

AS4 Gateway

The message exchange is built on the AS4 protocol, which is an open standard for secure, payload-agnostic business-to-business document exchange via web services. Secure document exchange is governed by WS-Security aspects, including XML Encryption and XML Digital Signatures. There are many implementations of AS4. The Connector includes an implementation of an AS4 gateway (phase4) using the following third-party libraries and components:

Based on the specifications:

RegRep

ebXML RegRep is a standard defining the service interfaces, protocols and information model for an integrated registry and repository. The repository stores digital content while the registry stores metadata that describes the content in the repository. An ebXML Registry is capable of storing any type of electronic content such as XML documents, text documents, images, sound and video. Instances of such content are referred to as a RepositorytItems. RepositorytItems are stored in a content repository provided by the ebXML Registry.

RegRep usage at DE4A project has been limited to a minimal approach due to the low benefits compared to a "plain" message. Nevertheless, the eDelivery implementation at the Connector includes the message wrapping into a RegRep specification with basic slots organization and the message obtained is sent via AS4 as a payload.

Kafka

The Apache Kafka instance used in DE4A is an open source distributed event streaming platform. It is used to receive messages from the different services used in the project. The received messages can be viewed online as they are received in the Package Tracker made available for everyone. The main purpose is to be able to quickly see if there are any problems between services and what type of problem (e.g. connection problems) to help pilot partners and developers find those issues. Therefore, the main use case is distributed event logging and not digital traffic reporting where full logs are sent and stored off-site. The reasoning for this is that the Kafka instance on the eGovlab servers is fully open, meaning that anyone can read and write messages on that service. To reduce security risks, the Kafka instance on the eGovlab premises does not store messages sent for extended periods of time, i.e. messages are essentially ephemeral.

The particular configuration used for the DE4A project uses ZooKeeper as the coordinator, Kafka as the message broker and the Package Tracker as the web service configured to consume and display any messages received by the Kafka broker. The Package Tracker can only display "live" messages, i.e. any messages received in the past will never be displayed.