Difference between revisions of "DE4A Connector it2"

From DE4A
Jump to navigation Jump to search
 
(13 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
The DE4A Connector is a technical proxy that allows the final participants (DE or DO) to send requests for evidence or responses to other final participants over an [[Usage of third party specifications and components|eDelivery]] communication environment. In addition, to handle the message exchange process, the Connector is responsible for obtaining the message routing information, by exchanging information with external components such as the IDK, the [[Usage of third party specifications and components|SML/DNS]] or the [[SMP]].
+
The DE4A Connector is a technical proxy that allows the final participants (DE or DO) to send requests for evidence or responses to other final participants over an [[Usage of third party specifications and components|eDelivery]] communication environment. In addition, to handle the message exchange process, the Connector is responsible for obtaining the message routing information, by exchanging information with external components such as the Central IAL, the [[Usage of third party specifications and components|SML/DNS]] or the [[SMP]].
  
 
To do so, it provides a common interface to DEs and DOs, making the complexity of the system transparent to the final participants and integration easier.
 
To do so, it provides a common interface to DEs and DOs, making the complexity of the system transparent to the final participants and integration easier.
  
−
The Connector component provides the [[AS4 Gateway]] functionality, so it can assume both the role of Data Requestor and Data Transferor. This first approach makes the Connector a stand-alone web application that can be deployed on any suitable application server.
+
The Connector component provides the [[AS4 Gateway]] functionality, so it can assume both the role of Data Requestor and Data Transferor. This approach makes the Connector a stand-alone web application that can be deployed on any suitable application server.
  
−
The security and integrity of messages, as well as the unique identification of the participants involved, are the cornerstones of the Connector component.
+
The security and integrity of messages, as well as the unique identification of the participants involved, are the cornerstones of the Connector component.[[File:DE4A network IT2 v2-Target infrastructure it2.drawio.png|alt=|center|frame|Conceptual schema of the target DE4A system for the second iteration.]]
−
 
 
−
Follow the link to the [[Installation and configuration guide of the DE4A Connector it2|Installation and configuration]] guide of the DE4a Connector.
 
−
 
 
−
[[File:De4a-schema-it1.png|Conceptual schema of the target DE4A system|alt=|center|frame]]
 
  
 
== Functionalities provided ==
 
== Functionalities provided ==
−
The main purpose of the DE4A Connector is sending and receiving of evidence requests and their responses. The message exchange process is described in the DE4A deliverables D2.4 Project Start Architecture and D5.3 Initial technical design of interfaces.
+
The main purpose of the DE4A Connector is sending and receiving evidence requests and their responses. The message exchange process is described in the DE4A deliverables D2.4 Project Start Architecture and D5.3 Initial technical design of interfaces.
  
 
=== Routing information lookup ===
 
=== Routing information lookup ===
−
The Connector is responsible for obtaining the Data Provider information from the IDK. It exposes a REST API <code>/lookupRoutingInformation</code> to get information about the Data Owners that provide a specific Canonical Evidence Type and further related information. That information is used to construct a request message to be sent through the Connector. Thus, the Data Evaluator is the only consumer of the mentioned API method.
+
The DE4A Connector is responsible for obtaining the Data Provider information from the Central IAL. The Central IAL collects information from all the SMPs deployed at each partner infrastucture and offers it in a centralized way through the REST API <code>/service/ial/</code>. When queried about a Canonical Item (Evidence or Event Catalogue) and a country, it returns the identifier of the Data Owner to request that Canonical Item from in that country. Therefore, through this functionality, the DE4A Connector implements the [[DE4A Issuing Authority Locator (IAL)|IAL]] semantic tool.
  
−
=== Dynamic discovery of Services ===
+
=== Dynamic discovery of services ===
−
In order for the Connector to be able to send a message to the corresponding endpoint, the [[Usage of third party specifications and components|eDelivery]] dynamic discovery mode of operation is used. This operation mode is based on the use of the [[DE4A Issuing Authority Locator (IAL)|IAL]] and [[Usage of third party specifications and components|SML/DNS]] and SMP components of the eDelivery infrastructure.  
+
In order for the Connector to be able to send a message to the corresponding endpoint, the [[Usage of third party specifications and components|eDelivery]] dynamic discovery mode of operation is used. This operation mode is based on the use of the components of the eDelivery infrastructure: [[Usage of third party specifications and components|SML/DNS]] and SMPs. Through this functionality, the DE4A Connector implements the [[DE4A Evidence Service Locator (ESL)|ESL]] semantic tool.  
−
 
 
−
The IAL is a new component for it2, it offers centralized information of all the SMPs deployed at each partner infrastucture. When a request is performed to the IAL it gathers the distributed information en each SMP to enroute to the correct destination.
 
  
 
The main elements stored in the SML/DNS and SMPs for this purpose are the following:
 
The main elements stored in the SML/DNS and SMPs for this purpose are the following:
  
−
* ''ParticipantIdentifier'': The Data Owner/Data Evaluator identifier who is publishing its [[Usage of third party specifications and components|AS4]] communication point (of the Connector linked to it).
+
* ''ParticipantIdentifier'': The Data Owner/Data Evaluator identifier who is publishing its [[Usage of third party specifications and components|AS4]] communication point (or the Connector linked to it).
−
* ''DocumentTypeIdentifier'': Canonical evidence type.
+
* ''DocumentTypeIdentifier'': Canonical Evidence Type or Canonical Event Catalogue.
−
* ''ProcessIdentifier'': Orchestration type (request/response).
+
* ''ProcessIdentifier'': Orchestration type: request, response or notification.
 
* ''AS4'' endpoint: AS4 service endpoint URL.
 
* ''AS4'' endpoint: AS4 service endpoint URL.
 
* ''Certificate'': The X.509 certificate of the AS4 server, used to encrypt the transmitted data for this specific participant.
 
* ''Certificate'': The X.509 certificate of the AS4 server, used to encrypt the transmitted data for this specific participant.
Line 37: Line 31:
 
* Response signature validation is mandatory.
 
* Response signature validation is mandatory.
 
* The communication between SMP and SML requires the usage of a client certificate.
 
* The communication between SMP and SML requires the usage of a client certificate.
−
The service metadata lookup will be performed as a step prior to the [[Usage of third party specifications and components|AS4]] message exchange. Therefore, the participant IDs and other related information must be known by the Connector in advance.
+
The service metadata lookup will be performed as a step prior to the [[Usage of third party specifications and components|AS4]] message exchange. Therefore, the participant IDs and other related information must be known by the DE4A Connector in advance.
  
 
=== Supported interaction patterns ===
 
=== Supported interaction patterns ===
−
Interaction patterns define the flow of data through the Connector and the intercommunication between the different components. Each pattern exchanges certain types of messages, and the incoming/outgoing information will depend on the processes occurring in the external components [3].
+
Interaction patterns define the flow of data through the Connector and the intercommunication between the different components. Each pattern exchanges certain types of messages, and the incoming/outgoing information will depend on the processes occurring in the external components.
  
−
The Connector currently supports three interaction patterns:
+
The Connector currently supports four interaction patterns: [[Intermediation Pattern|Intermediation (IM)]], [[User-supported Intermediation Pattern|User-Supported intermediation (USI)]], [[Subscription and Notification Pattern|Subscription and Notification (S&N)]] and [[Lookup Pattern|Lookup (LU)]] patterns. The [[Verifiable Credentials Pattern|Verifiable Credentials (VC)]] pattern is not supported by the DE4A Connector since that interaction pattern has benn implemented through its own components: the [[DE4A SSI Authority Agent|SSI Authority Agent]] and the [[DE4A SSI Edge Agent|SSI Edge Agent]].
−
 
 
−
* '''Intermediation (IM) pattern'''
 
−
** ­Asynchronous communication between the Connector and final participant (DE or DO).
 
−
** Data Owner endpoint must be known by the Data Transferor.
 
−
** Since the communication is synchronous, the Data Requestor does not need to know the identifier and endpoint of the Data Evaluator.
 
−
 
 
−
* '''User supported intermediation (USI) pattern'''
 
−
** Asynchronous communication between the Connector and final participant (DE or DO).
 
−
** Data Owner endpoint must be known by the Data Transferor.
 
−
** Since the communication is asynchronous, the Data Evaluator endpoint must be known by the Data Requestor. In addition, the Data Evaluator identifier is recovered from the request, since it is not sent in the response from the Data Transferor.
 
−
 
 
−
* '''Subscription and Notification (S&N) pattern'''
 
−
** Asynchronous communication between the Connector and final participant (DE or DO).
 
−
** Data Owner endpoint must be known by the Data Transferor.
 
−
** Since the communication is asynchronous, the Data Evaluator endpoint must be known by the Data Requestor. In addition, the Data Evaluator identifier is recovered from the request, since it is not sent in the response from the Data Transferor.
 
  
 
Most of the specific behaviour of each interaction pattern is independent of the Connector itself, as the Connector component is just designed to exchange messages and the main differences between the patterns take place in the external components such as the Data Evaluator and the Data Owner.
 
Most of the specific behaviour of each interaction pattern is independent of the Connector itself, as the Connector component is just designed to exchange messages and the main differences between the patterns take place in the external components such as the Data Evaluator and the Data Owner.
  
−
Additionally, the it2 connector offers a '''backwards compatibility feature''' for the IM pattern of it1. It can handle old messaging structure so implementation for it1 will be fully compatible with the it2 connector. In it1 IM pattern worked as a synchronous communication, this backwards compatibility also works with synchronous communication.
+
Additionally, the DE4A Connector offers a '''backwards compatibility feature''' for the former synchronous IM pattern. It can handle the old messaging structure, so the synchronous implementation is fully compatible with the current asynchronous version.
  
 
=== Connector roles ===
 
=== Connector roles ===
Line 69: Line 48:
 
* Data Transferor (DT)
 
* Data Transferor (DT)
  
−
No configuration is needed to differentiate the roles, it only depends on the usage, i.e., the behaviour will be according to the messages sent.
+
No configuration is needed to differentiate the roles, it only depends on the usage, i.e., the behaviour will be according to the messages sent. Moreover, in a functional scenario, the SMP data will determine which Connector is playing the DR or DT role.
−
 
 
−
Also, in a functional scenario the SMP data will determine which Connector is playing the DR or DT role.
 
−
 
 
−
=== AS4 implementations ===
 
−
The message exchange is built on the [[Usage of third party specifications and components|AS4]] protocol, which is an open standard for secure, payload-agnostic business-to-business document exchange via web services. Secure document exchange is governed by WS-Security aspects, including XML Encryption and XML Digital Signatures.
 
  
−
==== Phase4 ====
+
=== AS4 implementations supported ===
 +
The message exchange is built on the [[Usage of third party specifications and components|AS4]] protocol, which is an open standard for secure, payload-agnostic, business-to-business document exchange via web services. Secure document exchange is governed by WS-Security aspects, including XML Encryption and XML Digital Signatures.
  
−
* '''Main features'''
+
The DE4A Connector implements phase4 as an AS4 gateway to perform the AS4 message exchange. Its main features are:
−
** It is a generic AS4 solution.
+
* It is a generic AS4 solution.
−
** It can be integrated in basically any Servlet-based application server.
+
* It can be integrated in basically any Servlet-based application server.
−
** It can easily be used with Peppol.
+
* It can easily be used with Peppol.
−
** It can be used with other Dynamic Discovery solutions.
+
* It can be used with other Dynamic Discovery solutions.
−
** It supports a high degree of customization.
+
* It supports a high degree of customization.
−
** It is not limited to the CEF/e-SENS profile.
+
* It is not limited to the CEF/e-SENS profile.
−
 
+
All necessary configuration parameters are provided by the properties file of the DE4A Connector.
−
* '''Implementation'''
 
−
** The Connector implements phase4 as a gateway to perform the AS4 message exchange.
 
−
 
 
−
All necessary configuration parameters are provided by the properties file of the Connector.
 
  
 
=== Error handling ===
 
=== Error handling ===
 
Since the Connector performs multiple communications between different external components and some data and structure validations are performed, the Connector needs to monitor all failure points and be able to identify them in order to build the corresponding messages and warnings to inform each external component. When an error happens, the corresponding component creates an error message with the information about the error to be sent back to the entity that sent the failed message.
 
Since the Connector performs multiple communications between different external components and some data and structure validations are performed, the Connector needs to monitor all failure points and be able to identify them in order to build the corresponding messages and warnings to inform each external component. When an error happens, the corresponding component creates an error message with the information about the error to be sent back to the entity that sent the failed message.
 +
 +
The error handling is based on the set of [[DE4A Logs and error messages it2|log and error messages]] defined for DE4A.
  
 
=== Kafka messages ===
 
=== Kafka messages ===
Line 99: Line 72:
 
This feature is an advantage from a technical and business point of view, as the Connector performs the message exchange transparently to the other components.
 
This feature is an advantage from a technical and business point of view, as the Connector performs the message exchange transparently to the other components.
  
−
It should be noted that the messages sent to the Kafka server are hardcoded, it is not a parameterizable feature, so any enhancements must be hardcoded again and deployed. It has been developed in such a way because the necessity of a Kafka server is not expected outside the DE4A project. In a real scenario, an alternative way of collecting logs should be implemented.
+
The messages sent to the Kafka server are based on the set of [[DE4A Logs and error messages it2|log and error messages]] defined for DE4A.
 +
 
 +
===Data management===
 +
DE4A Connector can take the DE4A roles of Data Requestor and Data Transferor. This means that a Connector might provide service to multiple DEs and DOs at the same time. Therefore, the DE4A Connector needs to store the base endpoint (without path name) of each DE and DO it is providing service. Each record is identified by the participant ID of the DEs and DOs, so that when a message arrives to the DE4A Connector, whose recipient is identified by the participant ID, the DE4A Connector can forward the message to the corresponding endpoint service. This information is configured in the '''de-do.json''' file. More information on this is available at [[Connector iteration 2 installation and configuration guide]].
 +
==Security==
 +
The message exchange performed by the Connector has been conceived to ensure different security aspects as:
 +
 
 +
*'''Authenticity''': ensures the identity of the participant entities and any common component involved in the message exchange. It is achieved through the participant identifiers definition, [[Usage of third party specifications and components|eDelivery]] specification and [[Usage of third party specifications and components|AS4]] implementations.
 +
 
 +
* '''Integrity''': refers to the accuracy and completeness of data. Security controls focused on integrity are used to prevent data from being modified or misused by an unauthorized party, provided by the WS-security and XML message signature.
 +
 
 +
*'''Confidentiality''': data confidentiality at message level could be achieved by taking advantage of WS-Security encryption feature in User messages. This message encryption is performed via asymmetric keys: the sender of the message encrypts with the public key of the recipient, so the message can only be decrypted by the recipient, who owns the private key. Those keys should be configured either in the Connector in case of phase4 or in any external AS4 gateway.
  
−
==== Message types ====
+
===Communications===
−
The Connector implements a [[Usage of third party specifications and components|Kafka]] message producer through the de4a-kafka-client library of the de4a-commons package. This producer provides several types of messages or severity levels:
+
The Connector creates a secure context to establish in/outgoing connections with external components.
  
−
* '''Success'''
+
The main elements for such communication securitisations are:
−
* '''Info'''
 
−
* '''Warn'''
 
−
* '''Error'''
 
−
* '''Fatal error'''
 
  
−
Those message categories can be easily used to specify the severity level of the message sent to the Kafka server to track and identify them.
+
*'''Private key''': private part of private-public key pair is used in the message exchange between your server and the connecting clients. This private key must be configured in the Connector properties in order to create the secure context as well in the web server, in the case of reverse proxy or any similar structure. The keys configured on each component are generated under the DE4A management.
  
−
==== List of messages ====
+
*'''Truststore''': used for the storage of certificates from the trusted Certificate Authority (CA), which is used in the verification of the certificate provided by the server in an SSL connection. The Connector provides a default truststore which allows to trust on certificates generated under the main DE4A CA.
−
The messages currently sent from the Connector to the Kafka server are as follows:
 
  
−
'''Services''' – ''(Info level)''
+
====HTTP Proxy mode====
 +
Sometimes, the environments have certain communication structure where the outgoing connections must be performed through an HTTP proxy server. The Connector provides de functionality and proper configuration to perform the communications via an HTTP proxy server.
 +
===Message cryptographic protection ===
 +
The Connector messages exchange can be separated into:
  
−
* RequestTransferEvidenceIM message received - RequestId: <code>{0}</code>, CanonicalEvidenceType: <code>{1}</code>, DataEvaluator: <code>{2}</code>, DataOwner: <code>{3}</code>
+
====REST APIs messages====
−
* RequestTransferEvidenceUSI message received - RequestId: <code>{0}</code>, CanonicalEvidenceType: <code>{1}</code>, DataEvaluator: <code>{2}</code>, DataOwner: <code>{3}</code>
+
Once the SSL connection is established, all the data exchanged through it, will be encrypted on transport level based on the SSL context configuration settled, either managed by a reverse proxy on the server environment or configuring the proper application properties.
−
* RequestTransferEvidenceUSIDT message received - RequestId: <code>{0}</code>
 
−
* RequestLookupRoutingInformation message received - CanonicalEvidenceType: <code>{0}</code>, CountryCode: <code>{1}</code>, DataOwnerId: <code>{2}</code>
 
−
* Sending request to IDK - URL: <code>{0}</code>
 
−
* ­Looking for Data Owner address - DataOwnerId: <code>{0}</code>
 
  
−
'''Client''' – ''(Info level)''
+
====AS4 messages====
  
−
* Requesting to SMP - ParticipantId: <code>{0}</code>, DocumentTypeId: <code>{1}</code>, MessageType: <code>{2}</code>
+
*'''Phase4''': internally the message is encrypted using a specific keystore configuration only used for this purpose; also the cross-gateways communications are additionally encrypted over the HTTPS secure protocol.
−
* Sending RequestForwardEvidence to the Data Evaluator - RequestId: <code>{0}</code>, DataEvaluatorId: <code>{1}</code>, DataOwnerId: <code>{2}</code>, Endpoint: {3}
 
−
* Sending RequestExtractEvidence IM to the Data Owner - RequestId: <code>{0}</code>, CanonicalEvidenceType: <code>{1}</code>, DataEvaluatorId: <code>{2}</code>, DataOwnerId: <code>{3}</code>, Endpoint: <code>{4}</code>
 
−
* ­Sending RequestExtractEvidence USI to the Data Owner - RequestId: <code>{0}</code>, CanonicalEvidenceType: <code>{1}</code>, DataEvaluatorId: <code>{2}</code>, DataOwnerId: <code>{3}</code>, Endpoint: <code>{4}</code>
 
  
−
'''AS4''' – ''(Info level)''
+
===Message validation===
 +
Message validation is performed at different stages of the Connector data flow, from message structure compliance to content integrity.
  
−
* Sending request message via AS4 gateway - DataEvaluatorId: <code>{0}</code>, DataOwnerId: <code>{1}</code>, CanonicalEvidenceType: <code>{2}</code>
+
====Schema validation====
−
* Sending response message via AS4 gateway - DataEvaluatorId: <code>{0}</code>, Message tag: <code>{1}</code>, CanonicalEvidenceType: <code>{2}</code>
+
All the messages produced and consumed by the Connector are performing an XML Schema validation once it is received or sent out. This validation is done by the object conversion library that keeps the schema model including all the structure and content constraints.
−
* Processing the request received via AS4 gateway - RequestId: <code>{0}</code>
 
−
* ­Processing the response received via AS4 gateway - RequestId: <code>{0}</code>
 
  
−
'''Errors''' – ''(Error level)''
+
====Signature validation====
 +
As well as the XML Schema validation, the Connector performs validations to ensure the content and reliability on certain phases of the message exchange process:
  
−
* The corresponding request to the received response is not found on database - RequestId: <code>{0}</code>
+
*Services metadata querying
−
* RequestForwardEvidence has not been sent, unknown Data Evaluator endpoint - RequestId: <code>{0}</code>, DataEvaluatorId: <code>{1}</code>
+
*In the SMP data exchange a message signature validation is being performed to maintain the content reliability. This validation consists of verifying the embedded signature on the message against the trust certificate configured in the Connector for that purpose.
−
* RequestTransferEvidence not found on AS4 incoming message
+
The signature of incoming AS4 messages is verified against the central [[Usage of third party specifications and components|AS4]] Gateway CA globally used in DE4A project, internally validated by the implementation libraries that handle the reception and delivery.
−
* Error processing incoming message from AS4 gateway
 
−
* Error building URI from Data Evaluator endpoint: <code>{0}</code>
 
−
* ­Error sending message to Data Requestor via AS4 gateway - RequestId: <code>{0}</code>
 
−
* Data Owner address not found - DataOwnerId: <code>{0}</code>
 
−
[1] The <code>{x}</code> symbols are placeholders for dynamic text to be logged.
 
  
−
===Data management===
+
==Technology used==
−
The Connector stores and manages certain information such as DE endpoints (not to be confused with SMP endpoints), DO endpoints, certain request records for asynchronous response matching, etc. All this data and where to find it is described in the following subsections.
+
 
 +
===System core architecture===
 +
The Connector works as standalone application that runs different web services according to the RESTful API architecture principles. The application is built with the following tools:
 +
 
 +
*Spring Framework version 5.x.
 +
*Java 11
 +
*Apache HTTP Client 4.x
 +
 
 +
In addition to the core architecture, the XML Schemas defined to model the exchanged information, data constraints, interfaces, etc. are also important. All this is part of the Connector core through the de4a-commons library, which contains the above-mentioned model as well as utilities and conversion tools.
 +
 
 +
*Author: ''DE4A (WP5)''
 +
*Repository: https://github.com/de4a-wp5/de4a-commons
 +
 
 +
===Third party libraries===
 +
As part of the libraries used by the Connector some are related with the core features and represent a starting point for the functionalities provided by the Connector.
 +
 
 +
==== TOOP Connector====
 +
The TOOP Connector is a set of shared utility functions used in the Connector to perform common tasks that are required for a safe and interoperable data exchange. In the initial iteration the latest version of the TOOP Connector technical components were reused mainly for the usage of the built-in phase4 AS4 Gateway. Other elements of the TOOP Connector are currently ignored.
 +
 
 +
*Author: ''TOOP Project''
 +
*Repository: https://github.com/de4a-wp5/toop-connector-ng
 +
 
 +
====ph-oton libraries====
 +
Set of Java libraries to build Java web applications.
 +
 
 +
*Author: ''Philip Helger (phax)''
 +
*Repository: https://github.com/phax/ph-oton
 +
 
 +
====Peppol commons libraries====
 +
They include the SMP client library used by the Access Points to retrieve service metadata. This library supports the Peppol SMP specification, the OASIS BDXR SMP v1 and OASIS BDXR SMP v2 specification. This project uses Apache HTTP client to perform the REST lookups on foreign SMPs.
 +
 
 +
*Author: ''Philip Helger (phax)''
 +
*Repository: https://github.com/phax/peppol-commons
 +
 
 +
=== Data management===
 +
To manage the model and the data stored by the Connector the following technologies are used:
 +
 
 +
*'''JPA:''' The Java Persistence API is a specification of Java. It is used to persist data between Java object and relational database. JPA acts as a bridge between object-oriented domain models and relational database systems. As JPA is just a specification, it does not perform any operation by itself.
 +
 
 +
*'''Hibernate:''' An object–relational mapping tool for the Java programming language. It provides a framework for mapping an object-oriented domain model to a relational database. Hibernate handles object–relational impedance mismatch problems by replacing direct, persistent database accesses with high-level object handling functions.
 +
 
 +
===Utilities libraries===
 +
The project uses several libraries and utilities to process and transform the data. They can be divided according to their nature:
 +
 
 +
*'''Commercial libraries'''
  
−
====Data Owner addresses====
+
To perform common and non-business operations on web and data exchange projects, the DE4A Connector does not use any commercial libraries as it is published as an Open Source project, all the used libraries are embedded in the Java development kit.
−
Once each Data Owner is publishing his Connector service information on the SMP, the request will arrive to that Connector service configured on the SMP (via the AS4 Gateway), and considering that one Connector could be serving to multiples DOs, this Connector has to know the addressing (base endpoint, without path name) information related to a specific participant identifier (e.g. iso6523-actorid-upis::9999:egov) to send forward the request to the corresponding Data Owner, so due to that, the Connector maintains a table, named  '''''<code>owner_addresses</code>''''', with two columns:
 
  
−
*'''AgentUrn''': participant identifier in the DE4A format, ''e.g., iso6523-actorid-upis::9999:ess2833002e –'' see the “DE4A Policy for use of identifiers” for further information about the participant identifiers policy.
+
*'''In-house solutions'''
−
*'''Endpoint''': base endpoint URL of the Data Owner who is exposing the service <code>/requestExtractEvidence</code>
+
The Connector project includes some utilities that allow the data processing and internal tools to perform all the Connector tasks. Those utilities are within the Connector project as a module called ''de4a-commons.''
−
The information above will be used by a Connector playing the Transferor role when the ''RequestExtractEvidence'' message is being sent.
+
[[Category:Wip]]

Latest revision as of 14:06, 27 March 2023

The DE4A Connector is a technical proxy that allows the final participants (DE or DO) to send requests for evidence or responses to other final participants over an eDelivery communication environment. In addition, to handle the message exchange process, the Connector is responsible for obtaining the message routing information, by exchanging information with external components such as the Central IAL, the SML/DNS or the SMP.

To do so, it provides a common interface to DEs and DOs, making the complexity of the system transparent to the final participants and integration easier.

The Connector component provides the AS4 Gateway functionality, so it can assume both the role of Data Requestor and Data Transferor. This approach makes the Connector a stand-alone web application that can be deployed on any suitable application server.

The security and integrity of messages, as well as the unique identification of the participants involved, are the cornerstones of the Connector component.

Conceptual schema of the target DE4A system for the second iteration.

Functionalities provided

The main purpose of the DE4A Connector is sending and receiving evidence requests and their responses. The message exchange process is described in the DE4A deliverables D2.4 Project Start Architecture and D5.3 Initial technical design of interfaces.

Routing information lookup

The DE4A Connector is responsible for obtaining the Data Provider information from the Central IAL. The Central IAL collects information from all the SMPs deployed at each partner infrastucture and offers it in a centralized way through the REST API /service/ial/. When queried about a Canonical Item (Evidence or Event Catalogue) and a country, it returns the identifier of the Data Owner to request that Canonical Item from in that country. Therefore, through this functionality, the DE4A Connector implements the IAL semantic tool.

Dynamic discovery of services

In order for the Connector to be able to send a message to the corresponding endpoint, the eDelivery dynamic discovery mode of operation is used. This operation mode is based on the use of the components of the eDelivery infrastructure: SML/DNS and SMPs. Through this functionality, the DE4A Connector implements the ESL semantic tool.

The main elements stored in the SML/DNS and SMPs for this purpose are the following:

  • ParticipantIdentifier: The Data Owner/Data Evaluator identifier who is publishing its AS4 communication point (or the Connector linked to it).
  • DocumentTypeIdentifier: Canonical Evidence Type or Canonical Event Catalogue.
  • ProcessIdentifier: Orchestration type: request, response or notification.
  • AS4 endpoint: AS4 service endpoint URL.
  • Certificate: The X.509 certificate of the AS4 server, used to encrypt the transmitted data for this specific participant.

The information described above is managed by the SML/DNS and SMP components and is used by the Connector when working with the phase4 implementation of AS4.

Focusing on the SML/DNS/SMP data retrieval, the process will take place according to the following features:

  • SSL/TLS communication is mandatory.
  • Response signature validation is mandatory.
  • The communication between SMP and SML requires the usage of a client certificate.

The service metadata lookup will be performed as a step prior to the AS4 message exchange. Therefore, the participant IDs and other related information must be known by the DE4A Connector in advance.

Supported interaction patterns

Interaction patterns define the flow of data through the Connector and the intercommunication between the different components. Each pattern exchanges certain types of messages, and the incoming/outgoing information will depend on the processes occurring in the external components.

The Connector currently supports four interaction patterns: Intermediation (IM), User-Supported intermediation (USI), Subscription and Notification (S&N) and Lookup (LU) patterns. The Verifiable Credentials (VC) pattern is not supported by the DE4A Connector since that interaction pattern has benn implemented through its own components: the SSI Authority Agent and the SSI Edge Agent.

Most of the specific behaviour of each interaction pattern is independent of the Connector itself, as the Connector component is just designed to exchange messages and the main differences between the patterns take place in the external components such as the Data Evaluator and the Data Owner.

Additionally, the DE4A Connector offers a backwards compatibility feature for the former synchronous IM pattern. It can handle the old messaging structure, so the synchronous implementation is fully compatible with the current asynchronous version.

Connector roles

A Connector instance can play two different roles:

  • Data Requestor (DR)
  • Data Transferor (DT)

No configuration is needed to differentiate the roles, it only depends on the usage, i.e., the behaviour will be according to the messages sent. Moreover, in a functional scenario, the SMP data will determine which Connector is playing the DR or DT role.

AS4 implementations supported

The message exchange is built on the AS4 protocol, which is an open standard for secure, payload-agnostic, business-to-business document exchange via web services. Secure document exchange is governed by WS-Security aspects, including XML Encryption and XML Digital Signatures.

The DE4A Connector implements phase4 as an AS4 gateway to perform the AS4 message exchange. Its main features are:

  • It is a generic AS4 solution.
  • It can be integrated in basically any Servlet-based application server.
  • It can easily be used with Peppol.
  • It can be used with other Dynamic Discovery solutions.
  • It supports a high degree of customization.
  • It is not limited to the CEF/e-SENS profile.

All necessary configuration parameters are provided by the properties file of the DE4A Connector.

Error handling

Since the Connector performs multiple communications between different external components and some data and structure validations are performed, the Connector needs to monitor all failure points and be able to identify them in order to build the corresponding messages and warnings to inform each external component. When an error happens, the corresponding component creates an error message with the information about the error to be sent back to the entity that sent the failed message.

The error handling is based on the set of log and error messages defined for DE4A.

Kafka messages

Within the data flow and message exchange, there are many key points where it is important to know how the data is being managed, as well as identifying intermediate errors and unhandled system states. In this respect, the Connector can send messages to a Kafka server to track the data flow and trace the state of the system at certain points.

This feature is an advantage from a technical and business point of view, as the Connector performs the message exchange transparently to the other components.

The messages sent to the Kafka server are based on the set of log and error messages defined for DE4A.

Data management

DE4A Connector can take the DE4A roles of Data Requestor and Data Transferor. This means that a Connector might provide service to multiple DEs and DOs at the same time. Therefore, the DE4A Connector needs to store the base endpoint (without path name) of each DE and DO it is providing service. Each record is identified by the participant ID of the DEs and DOs, so that when a message arrives to the DE4A Connector, whose recipient is identified by the participant ID, the DE4A Connector can forward the message to the corresponding endpoint service. This information is configured in the de-do.json file. More information on this is available at Connector iteration 2 installation and configuration guide.

Security

The message exchange performed by the Connector has been conceived to ensure different security aspects as:

  • Authenticity: ensures the identity of the participant entities and any common component involved in the message exchange. It is achieved through the participant identifiers definition, eDelivery specification and AS4 implementations.
  • Integrity: refers to the accuracy and completeness of data. Security controls focused on integrity are used to prevent data from being modified or misused by an unauthorized party, provided by the WS-security and XML message signature.
  • Confidentiality: data confidentiality at message level could be achieved by taking advantage of WS-Security encryption feature in User messages. This message encryption is performed via asymmetric keys: the sender of the message encrypts with the public key of the recipient, so the message can only be decrypted by the recipient, who owns the private key. Those keys should be configured either in the Connector in case of phase4 or in any external AS4 gateway.

Communications

The Connector creates a secure context to establish in/outgoing connections with external components.

The main elements for such communication securitisations are:

  • Private key: private part of private-public key pair is used in the message exchange between your server and the connecting clients. This private key must be configured in the Connector properties in order to create the secure context as well in the web server, in the case of reverse proxy or any similar structure. The keys configured on each component are generated under the DE4A management.
  • Truststore: used for the storage of certificates from the trusted Certificate Authority (CA), which is used in the verification of the certificate provided by the server in an SSL connection. The Connector provides a default truststore which allows to trust on certificates generated under the main DE4A CA.

HTTP Proxy mode

Sometimes, the environments have certain communication structure where the outgoing connections must be performed through an HTTP proxy server. The Connector provides de functionality and proper configuration to perform the communications via an HTTP proxy server.

Message cryptographic protection

The Connector messages exchange can be separated into:

REST APIs messages

Once the SSL connection is established, all the data exchanged through it, will be encrypted on transport level based on the SSL context configuration settled, either managed by a reverse proxy on the server environment or configuring the proper application properties.

AS4 messages

  • Phase4: internally the message is encrypted using a specific keystore configuration only used for this purpose; also the cross-gateways communications are additionally encrypted over the HTTPS secure protocol.

Message validation

Message validation is performed at different stages of the Connector data flow, from message structure compliance to content integrity.

Schema validation

All the messages produced and consumed by the Connector are performing an XML Schema validation once it is received or sent out. This validation is done by the object conversion library that keeps the schema model including all the structure and content constraints.

Signature validation

As well as the XML Schema validation, the Connector performs validations to ensure the content and reliability on certain phases of the message exchange process:

  • Services metadata querying
  • In the SMP data exchange a message signature validation is being performed to maintain the content reliability. This validation consists of verifying the embedded signature on the message against the trust certificate configured in the Connector for that purpose.

The signature of incoming AS4 messages is verified against the central AS4 Gateway CA globally used in DE4A project, internally validated by the implementation libraries that handle the reception and delivery.

Technology used

System core architecture

The Connector works as standalone application that runs different web services according to the RESTful API architecture principles. The application is built with the following tools:

  • Spring Framework version 5.x.
  • Java 11
  • Apache HTTP Client 4.x

In addition to the core architecture, the XML Schemas defined to model the exchanged information, data constraints, interfaces, etc. are also important. All this is part of the Connector core through the de4a-commons library, which contains the above-mentioned model as well as utilities and conversion tools.

Third party libraries

As part of the libraries used by the Connector some are related with the core features and represent a starting point for the functionalities provided by the Connector.

TOOP Connector

The TOOP Connector is a set of shared utility functions used in the Connector to perform common tasks that are required for a safe and interoperable data exchange. In the initial iteration the latest version of the TOOP Connector technical components were reused mainly for the usage of the built-in phase4 AS4 Gateway. Other elements of the TOOP Connector are currently ignored.

ph-oton libraries

Set of Java libraries to build Java web applications.

Peppol commons libraries

They include the SMP client library used by the Access Points to retrieve service metadata. This library supports the Peppol SMP specification, the OASIS BDXR SMP v1 and OASIS BDXR SMP v2 specification. This project uses Apache HTTP client to perform the REST lookups on foreign SMPs.

Data management

To manage the model and the data stored by the Connector the following technologies are used:

  • JPA: The Java Persistence API is a specification of Java. It is used to persist data between Java object and relational database. JPA acts as a bridge between object-oriented domain models and relational database systems. As JPA is just a specification, it does not perform any operation by itself.
  • Hibernate: An object–relational mapping tool for the Java programming language. It provides a framework for mapping an object-oriented domain model to a relational database. Hibernate handles object–relational impedance mismatch problems by replacing direct, persistent database accesses with high-level object handling functions.

Utilities libraries

The project uses several libraries and utilities to process and transform the data. They can be divided according to their nature:

  • Commercial libraries

To perform common and non-business operations on web and data exchange projects, the DE4A Connector does not use any commercial libraries as it is published as an Open Source project, all the used libraries are embedded in the Java development kit.

  • In-house solutions

The Connector project includes some utilities that allow the data processing and internal tools to perform all the Connector tasks. Those utilities are within the Connector project as a module called de4a-commons.

Retrieved from ‘https://wiki.de4a.eu/index.php?title=DE4A_Connector_it2&oldid=5971’