Lookup Pattern
Functional Variants of the Lookup Pattern
The basic logic is the Lookup pattern is a simple Request-Response interaction between DC and DP without any user involvement. We now identified two functional variations.
Evidence Lookup
This variant is for looking up a complete Evidences. Once is established that an update of the evidence is needed, e.g. via a notification from the DP to DC, the evidence can be looked up in its entirety.This can also be used for integration in public service (back-office) processes for cases where a legal basis for data sharing exists (e.g: bilateral agreement or publicly available data).
Request: Evidence type ID
Attribute Lookup (i.e. using API)
This variant is for getting updates for specific attribute(s) as well as addressing the need for an API approach. Reusing existing APIs that already exist in MSs and providing a light-weight alternative for eDelivery.
Request: (array of) attribute(s) [canonical of domestic (point-to-point)].
Alternative Solution Approaches
Evidence Lookup
The proposed solution approach for DBA second iteration is the Evidence Lookup. Simple put, the intermadiation pattern but in a simplified form.
- Leveraging the AS4-infrastructure and message definitions (cf. Intermediation Pattern) which are already in place
- Simplified: no user intervention (i.e. no explicit request, no preview)
Attribute Lookup
An interesting development is the piloting of an API approach in ISA2. This project investigates new patterns of data access by request and data sharing. The initiative will facilitate design choices on the legal, organizational, semantic and technical level necessary for setting up APIs. It includes the piloting of such an approach through a combination of the CEF eDelivery building block and a REST-based profile (a.k.a. the APIs approach). This looks like a promising inititative and an interesting development for the future. At this point in time however, there is no mature BB for DE4A to be used. This is one of the reasons why we recommend the Evidence Lookup based on IM as solution direction for the DBA PIlot.
Working hypotheses and implementation principles
| Interdisciplinary Topic | Hypothesis / Principle | Implications and Limitations |
| Orchestration / Choreography | The DC is orchestrating the overall flow. This means that the process on DP side is a child processes of the process on the DC side. | The DC controls the status of all DP evidence retrieval processes. The DC can retain overal control by reacting to responses of the DP (evidence or error) and monitoring the a response is received in a reasonable amount of time (i.e. SLA) |
| Complementary, overlapping or conflicting evidence equivalents | Cases of ambiguous evidences must in principle be supported by the technical system. These cases are expected to be rare for lookup, because it is always related to a single Evidence request, single Evidence Type and single DP in contrast to the Intermediation Pattern, that by definition needs to be able to handle multiple Evidence requests to multiple DP in potentially different countries. | The DE4A pilot cases appear not to suffer from this issue and the canonical evidence approach also means that this issue is usually resolved at the DP-side. |
| Interrupted vs. Uninterrupted exchange | The whole lookup is handled in an uninterrupted manner. This means that any exception during the lookup leads to its termination, potentially to be repeated at a later time as a new attempt. | |
| Explicit request and transitivity between actors | After 2023 (with SDGR as legal basis), the DP does not need to re-validate the explicit user request, they can rely on the DC to have done so. It is questionable whether this is presently possible in the Pilots, as the SDGR Article 14 enters into force after the Pilot timeline (Article 39). The assumption is, however, that piloting for the SDGR is part of the public authority tasks related to the SDGR (i.e. fall under the application of Article 14 (11)). | We need the MS participating in the pilots to sustain this interpretation, i.e. in a MoU and accept the limitation that the pilot solution cannot transition to full production on grounds of this legal basis, before the full Article 14 of the SDGR enters into force on 12.12. 2023. |
| Preview & Approval UI | The preview can be provided, and the user approval collected, by the DC, prior to the evidence being used in eProcedure. It is well understood that the data processing of the evidence on the part of the DC is restricted to providing the preview to the user. This entails the risk that operators of the receiving competent authority could gain, either by accident or (disingenuous) intent, access to the evidence data prior to user authorisation, i.e. for example by using administrator rights on the underlying ICT infrastructure. | There are legal, privacy and security concerns with this hypothesis and there are indications that not all MS are prepared to accept these. A preview provided by the DC would for instance break the privacy-by-design principle.
It is also noteworthy that the DP does not know about the outcome of a DC-side preview or would need to be explicitly informed about it. The preview at the DC side must be able to show previews of evidences from multiple countries and must be implemented for every DC. The DC has in any case to implement a solution guaranteeing that “the data included in the preview should not be stored longer than is technically necessary” (recital 47 SDGR)[3] if the user decides not to reuse or to submit the data. |
| Identity and Record Matching | From experience on MS-level we see that a reasonably good match can result from the use of the (mandatory) eIDAS attributes. The working hypothesis is that this insight can be generalised to all pilot MSs. Two consequences of this hypothesis are that a) the user does not need to provide supplementary attributes and b) a second eIDAS authentication at the DP (potentially multiple DP) is not required. | As the matching based on eIDAS attributes is never 100% it is only considered sufficient from a piloting perspective, where an unsuccessful match could be dropped from the pilot population.
Most MS consider current examples of implementation of record matching as insufficiently matured and scalable across all EU MS. A process has to be defined, for example, to manage the situations where this automatic matching doesn’t work. The Intermediation pattern has limitations for catching these exceptions especially in case of an unsuccessful match at the DP, as no direct interaction between U and DP is foreseen. |
| Transitivity of user identity | After 12.12.2023, the SDGR and the legal task of the DC provide the legal basis for the exchange of user or data subject data from DC to DP. We assume that the development in preparation of the SDGR (i.e. piloting) is part of the public authorities’ tasks covered by the SDGR (i.e. Article 14 (11)), hence that the SDGR provides the legal basis for the pilots.
Adding a GDPR consent in the explicit request is not a valid legal basis for the case that the identification does include personal data of other data subjects than the requestor (e.g. change of address for families). |
IF the intermediation pattern is used in in teh citizen domain, we need the MS participating in the pilots adopting the intermediation pattern to sustain this interpretation. |
| Hand-on of UI between actors | The DC handles all user interaction of the eProcedure, including the OOP transfer of evidence, thus foreclosing the need to hand-over user sessions across MSs. | This means that the pilot cases do not include additional information, other than included in the initial request and (mandatory) eIDAS attributes, to be used by the DP. |
| Mandate and Proxy | The mandate and proxy challenge can be resolved by an extension of the eIDAS node.
A simple solution can be build on the "full powers"-assumption with current eIDAS functionality. |
The results from SEMPER can be adopted for piloting. It is expected that solutions based in this approach cannot go production live within the timelines of DE4A, as it would require an adjustment of the eIDAS Regulation. |
| Encryption Gap | OOP in the public sector does not require true E2E encryption. The exchange between DR and DP must be encrypted and signed, as well as the transfers (if applicable on national level) between DR and DE on DC side and DT and DO on DP side (i.e. using the national OOP layer), but the encryption gap within the systems of the DR and DT is acceptable. | This might not hold for cases where the gateway would be outsourced to a private sector subcontractor, which is not foreseen for the DE4A pilots. |
| Structured data vs. unstructured data | Evidence is handled as structured data. This is not contradicting the addition of an unstructured or scanned document/certificate as part of the structured data transfer (hybrid approach) for reasons of legal validity. | |
| Automated re-use of data | Evidence and its use in public service procedures has legal consequences. We assume that automated re-use without premediated harmonization of evidence data definitions is not applicable for the OOP transfer of evidence between MS. | To facilitate automated re-us of data requires establishing canonical evidence definitions. |
| Production system and real-life cases | With reference to SDGR Article 14(11), pilots based on the intermediation pattern can interface with productive systems and use real-life cases (if participants are made aware that they are participating in a DE4A pilot). | Pilots considering the intermediation pattern must align with their participating MS that they accept the interpretation of the Article 14(11) as legal basis of the pilot even before the full Article 14 of the SDGR enters into force on 12.12. 2023.
The situation in the business domain is differnt as the company registration data is already publicly available. |
| Payment for evidence | In the context of the pilots we assume that no payments are required. | This can restrict transition of pilot solutions to production in cases that compenent authorities require payment for issuing evidence. |
| BRIS integration | A technical re-use or bridge to BRIS is not possible because of differences in scope and accessibility by competent authorities other then business registers. The semantic definitions of BRIS can be largely reused. | The pilot system for the Doing Business Abroad Pilot need to be set-up separte from BRIS. |
| Matching evidences between Member States | The final system should support both harmonized and harmonized evidence type and the architecture is taking account of both bases. In the pilot context, focus will be put on establishing deep semantic interoperability through the definition of canonical evidences | Heterogenous, national evidence types do not need to be matched in run-time.
For all evidence types in DE4A, a canonical form must be defined an agreed between the the pilot partners. Each partner needs to implement a transformation from national to canonical evidence. |
| Multi-evidence Cases | The system should support all four multi-evience cases, which means that an array of evidence types and evidences must be included in a single OOP request/response. | The second iteration should expand the MVP restriction to a single request to single evidence cases, which requires an update of the Exchange Information Model. It is likely that piloting would focus on simpler cases to show the inclusion of multiple evidences in a single evidence response.
The multi-evidence cases are likely not relevant for the Doing Business Abroad Pilot. Theoretically, the 'Multi Evidence Types'-case could be applied in the second iteration to request e.g. company registration evidence and annual financial statement in a single request. |
Business Process of the Evidence Lookup
Business Process Collaboration View
The Figure below shows the BPMN Business Process Collaboration view of the Evidence Subscription Process, which is either triggered because a Notification was interpreted to require an evidence update, or it is triggered by a Public Service procedure that requires an evidence that can be fetched based in bilater agreement or Union law. Please note that this pattern is not triggered by the user. The Evidence Lookup could therefor also be used in a traditional procedure based on a physical transaction with the user.
As you can see in the Business Process Collaboration view above, the process of looking up an evidence for the first time or looking up a new version of the evidence is essentially identical. These variants have, however, different legal implications and might consequently differ in the authorization aspect of the Evaluate Evidence Request activity. The process is also very similar to the Intermediation Pattern, even through not all activities listed below are equally relevant for all use cases. The Establish Subject Identity activity, for example, is not relevant for all business use-cases that can base identification on an European unique identifier. The DC looks up the correct DP, which might simplified for pilot purposes, and sends an Evidence request to the DP. The DP checks the request, extracts the evidence and returns the Evidence response that is then saved by the DC.
Activity Table
| Activity / UC | Role | Type | Description |
| Determine required cross-border evidence | DE | Service | This step makes sure that the DE always requests the recent version of the Evidence type (cf. canonical evidences); in the evidence update case, for example, the evidence type definitions might have changed since the last lookup.
In cases where the evidence type is not harmonized, the required evidence type (in terms of the DC country) is translated into equivalent evidence types that are issued in a lawful way in the DP country indicated by the user. |
| Lookup routing information | DR | Service | The DR retrieves the technical routing information (e.g. eDelivery rooting identifier or URL of the webservice provider), based on the evidence type (in terms of DP country) and the issuing competent authority (or geographic scope of authority). Note that the Evidence Lookup is used in DE4A in combination with the Subscription and Notification Pattern, so as long as the subscription and lookup service is provided by the same DC, the participant ID can be assumed to be known and be included in the Evidence update requiement. |
| Request evidence | DR | Service | The DR encrypts, signs and sends the evidence request to the identified technical data service interface of the DP. The evidence request must include subject (i.e. company) information that enables the DP to identify for which subject be issued. Companies already have a European unique identifier available (EUID), which is sufficient identification information. |
| Evaluate evidence request | DT | Service | The DT receives and decrypts the request and checks whether the request meets formal requirements and can be accepted. It should be checked whether the requesting competent authority can reasonably and rightfully request that specific type of evidence (The authority check is not piloted in DE4A) |
| Establish subject identity | DO | Service | This activity is only relevant in absence of a European unique identidier. The DO matches identification information about the subject (i.e. equivalent to eIDAS mandatory and optional attributes) with the DP country’s records to identify the subject in their systems. This amounts to matching the eIDAS attributes to a national identification number. This is a Data Owner activity, because in a distributed scenario the data transferor might not have a legal basis to do so. |
| Communicate non-availability of OOP | DT | Service | This exception handling activity is only relevant in absence of a European unique identifier: The DT informs the DR that the subject cannot be identified unequivocally and the system cannot be used to transfer the evidence. |
| Extract evidence | DO | Service | The DO extracts the requested evidence form their registry and forwards it to the DT. |
| Communicate non-availability of evidence | DT | Service | Exception handling activity: The DT informs the DR that the requested evidence cannot be provided or cannot be provided within the agreed SLA. |
| Establish non-availability of OOP | DR | Service | Exception handling activity: The DR catches the negative (non-evidence) response from the DT and establishes the reason in terms of the DC country system and language:
There are potentially several reasons why an OOP transfer of evidence is not available. The DT communicates these reasons to the DR in all cases that the evidence request cannot be fulfilled (i.e. by sending the digitally available evidence within the agreed SLA as described above). At the moment we expect at least the following reasons for such an exception that should be framed in standard error messages or codes, each one with a corresponding recommendation.
|
| Compose evidence response | DO | Service | The DO prepares the extracted evidence to be send as an evidence response. Depending on the level of harmonization of the evidence type this task can differ in complexity. If a canonical evidence definition is agreed, this task includes the translation of the national definitions into the canonical evidence. |
| Transfer evidence | DT | Service | The DT creates the evidence response message (compliant to agreed message format), encrypts and signs the message and sends it to the DR. |
| Forward evidence | DR | Service | The DR registers the receipt, decrypts the message and in many cases encrypts the message in a MS specific format to hand it on to the DE. |
| Evaluate evidence | DE | Service | The DE validates that the evidence conforms to the evidence type requested and stored or updates the evidence. If it is a new evidence that was requested as part of a public service procedure, the availability of the evidence is signalled to the active procedure. |
Process Realisation of the Evidence Lookup
The figure below shows how application services serve the Data Consumer process. The application services are realized by application collaborations.
The process starts by an external business trigger identifying the neeed for an evidence or update thereof. With he help of the Information Desk the required cross-border evidence is determined and the relevant routing information is looked up.
Next the Evidence can be requested, the request message is encrypted and digitally signed using the Trust Architecture. The evidence is exchanged using Data Logistics and can be tracked using Evidence Interchange Management. The signature of the received message is validated and the message decrypted (Trust Architecture). Next the evidence can be evaluated by the DC (EProcedure Portal) and if all is well the public service can be (contunued to be) provided.
The figure below shows how application services serve the Data Provider process. The application services are realized by application collaborations.
The Evidence request is received via Data Logistics and with the help of Trust Architecture the DP checks the signature of the request and decrypts it. An Authority check may be performed using the Information Desk establishing that the DC is allowed to request the evidence type. Next the user identity is established using Trust Architecture. If this successful the evidence is extracted by Evidence Retrieval Retrieval and transformed to cannonical form (Evidence Portal). Various exceptions like non-availability of OOP or the delay or non-availability of evidence are handled by Data Logistics and Evidence Portal. If all is well the Evidence response is composed is prepared for transfer (Evidence Portal), encrypted and digitally signed using Trust Architecture and ultimately exchanged using Data Logistics.
Application Collaborations
Evidence Interchange Management
Future Extension: Attribute Lookup Using API
TODO Functional - what does it do.
Typical tech. choice: API
-> ISA2 / eDelivery API
ISA2: REST API extension of CEF eDelivery
Specifically for "Light Context"
Pilot project underway
Complexity, many considerations (legal, technical: #corners, security, protocols, signing etc.)
Unfortunately BB immature, probably not in time ready for DE4A
EC Important document ISA2 Action ‘Innovative Public Services’: Piloting a REST API extension of CEF eDelivery, 30/10/2020 v1.1 [1]
API project in ISA2 Action INNOVATIVE PUBLIC SERVICE
| Source | European Commission |
| Action Owner | CONNECT (DIGIT, JRC). |
| Objectives &
scope |
Develop relevant legal, organizational and technical artefacts trialled through a combination of the CEF eDelivery building block with blockchain-based transactions’ log and a REST-based profile (a.k.a. APIs approach), that support new patterns of data access by request and data sharing. The initiative will facilitate design choices on the legal, organizational, semantic and technical level necessary for setting up APIs. |
The REST-based profile is relevant for DE4A Lookup pattern; however the scope of the API project is wider. The envisaged implementation is an extension of the eDelivery BB.
Piloting a REST API extension of CEF eDelivery [1] contains a lot of useful information. What follows is an extract.
Lookup Pattern D2.1:
Business need
A BB with a profile to cater for the REST API architectural style primarily addressing different architectures and communication patterns than those already supported by the eDelivery AS4 profile. The data exchange would operate in a light context.
Light Context
The term “light context” refers to a set of constraints and circumstances applying to organisations or environments that do not run (in) an enterprise IT data centre (non-limitative):
- Organisational constraints
- Hardware and IT infrastructure constraints
- “Low throughput” scenarios
- Limitations introduced by sandbox environments
Requirements
Requirements envisaged specification:
- Simple or automatic installation of the software
- Minimal or zero configuration that assumes no advanced knowledge of the used technology
- Minimal operation and maintenance
- Ease of use with immediate start and no complicated enrolment
- Reduced requirements on the hardware resource
- Reduced access privileges on the host
Legal Basis
Legal basis: carried out under the ISA² action on Innovative Public Services, legal artefacts are also envisaged.
Analysis - Checklist of Required Decisions for Applying API-Approach
Things to consider:
- Number of corners, 2 or more (even >4)
The specification/profile could consider a variable number of corners, starting with as few as two and extending the model to support an arbitrary number greater than four (interoperability with other existing protocols and message/data exchange networks).
- 2-corner – traditional client-server call (proposed for DE4A as simplifying assumption)
- 3-corner – a reduced version of the 4-corner where corners C1 and C2 are collapsed into a single corner, C1+2, or corners C2 and C3 are collapsed into a single corner, C2+3
- Four corners or more, in particular in the sense of not introducing accidental barriers to interoperability between the REST API profile and other existing protocols and message/data exchange networks is concerned (CEF eDelivery AS4, SDG, X-Road, GAIA-X). The profile should strive to minimise the need for a conformant API to be adapted for use in different such networks.
- Communication patterns
Various communication patterns can be considered:
- Synchronous business response (the sending corner (C1) sends a business message to the receiving corner (C2) via an http request and expects a business response. The http response it receives from C2 contains a business message and completes the exchange) (proposed for as simplifying assumptionDE4A)
- Asynchronous business response (the sending corner (C1) sends a business message to the receiving corner (C2) via an http request and expects a business response. The http response it receives from C2 contains no business message, but only an acknowledgment of receipt. The business response will be obtained at a later time, e.g., through a pull or web socket).
- No business response (the sending corner (C1) sends a business message to the receiving corner (C2) via an http request and does not expect a business response. The http response it receives from C2 contains only an acknowledgment of receipt and competes the exchange).
- reliable delivery (in a 3-corner model, by enabling retry calls from C2 to C3)
- broadcast (in a 3-corner model, by forwarding the call to a list of recipients)
- asynchronous send buffer / streaming (send buffer instead of full message)
- correlated calls to transmit multi-part messages
- etc.
- Identity
Direct management of certificates is impractical in a “light context”, alternative authorisation approaches relying on protocols designed for the web/mobile application world are required.
- OAuth 2.0 / OpenID Connect
- JSON Web Token
- SAML
- Web authentication
- FIDO 2
- potentially others (e.g., EU Login)
TODO proposal for DE4A
- Transport protocols
DE4A proposal HTTP/JSON
- Integrity & confidentiality
TLS (see WP5 recommendation)
message signing option (TODO DE4A recommendation)
- (Q)ERDS = Qualified Electronic Registered Delivery Service
DE4A not required