Second phase

Building Block Assessment - 2nd Phase

The second phase of the assessment of the architecture building blocks (BBs) used in the DE4A project comes after the cataloging of an initial set of relevant BBs for the Project Start Architecture (PSA). While in the previous phase the BBs were assessed for their technical, administrative and operational maturity, in this phase a more constrained set of BBs actually implemented by the pilots was evaluated from a wider perspective.

Methodology

The methodology for evaluation was designed following general systemic principles, with a set of indicators including: usability, openness, maturity, interoperability, etc. The identification of the evaluation criteria and the analysis of the results have been approached from several perspectives: literature survey and thorough desktop research; revision and fine-tuning of the initial BB set and evaluation methodology, and an online questionnaire designed for gathering feedback by the relevant project partners:

• Member States (MSs)
• WP4 partners - Pilots (Doing Business Abroad - DBA, Moving Abroad - MA, and Studying Abroad - SA)
• WP5 partners - for specific components

To revise and fine-tune the evaluation methodology developed in the first phase, as well as the set of BBs selected for assessment as a result, several meetings and consultations were held with the aforementioned project partners. This led to a narrower set of relevant BBs consisting of a subset of the ones assessed in the first phase, and news ones resulting from the piloting needs and implementation. To capture these specificities, a wide set of questions from the survey capture all aspects important for the decisions made and for future reuse. Several iterations over the initial set of questions were performed, determining the relevance according to the roles of the project partners that provided feedback. Finally, a process of feedback coordination was also determined for the pilot leaders, who gathered additional information by the pilot partners’ implementation of some of the BBs (or important aspects of them).

It is important to note that the division of partners according to their role in the project was done due to the difference in the set of BBs that they had experience with. This implies that not each partner evaluated the whole set of BBs, but only the BBs relevant for their practice and research in the DE4A project.

The final set of relevant BBs per partner type is given in Table 1, together with statistics on the number of assessments obtained for each BB.

Table 1 List of building blocks evaluated in the second phase, per relevant partner

#	Building Block	# of assessments on the BB	Relevant partner(s)
	Common Components
1	eDelivery (data exchange)	3	WP5/MSs
2	SMP/SML	2	MSs via pilots
3	DE4A Connector	2	MSs via pilots
4	DE4A Playground	6	DBA/SA/MA
	Semantic
5	Information Exchange Model	7	DBA/SA/MA/WP5
6	Canonical Data Models	6	DBA/SA/MA DE and DO pilot partners
7	ESL (implemented as part of SMP/SML)	1	WP5
8	IAL	1	WP5
9	MOR	1	MA pilot partners
	eID/PoR
10	SEMPER	1	DBA
	VC Pattern
11	SSI Authority agent	4	SA
12	SSI User agent (mobile)	4	SA
13	EBSI-ESSIF (CEF Blockchain)	1	WP5 (T5.4)

Questionnaire design

This questionnaire aims to analyze the extent and the ways of employment of a list of BBs, whichwere cataloged during the first assessment phase of this task. Through a number of assessment categories and indicators assigned to them, the questionnaire supports a methodology that allows us to qualify and quantify the applicability, functionality, maturity and potential for reusability of each of the BBs, especially in view of the DE4A project.

The questionnaire contains 9 sections, i.e. categories, inquiring on:

• Inventory of BBs and functionalities;
• Interoperability;
• Maturity;
• Openness;
• Ease of implementation;
• Meeting pilot requirements;
• Performance (Non-functional requirements);
• Patterns; and
• Trust, Identity, Security, Privacy, Protection.

The questionnaire was distributed among the relevant (aforementioned) partners, and the feedback has been obtained in the period 28.11.2022 - 12.12.2022. Based on the insights from the provided feedback, we are able to also extract valuable lessons learned through the implementation practices, as well as recommendations for future reuse of the analyzed BBs.

Note: The BB assessment includes the BBs used in both iterations of the pilots. It also includes BBs used in the playground and the mocked DE/DO.

Data and preprocessing

After the data gathering period, a total of 10 surveys were received with valid input, representing full coverage of the BBs by foreseen parties (See Table 2). This provides sufficient statistical significance to proceed with the analysis of the results and finalize the BB assessment.

Table 2.Distribution of received valid feedback per partner type

Partner type	# of assessments done
1 (WP5 representative)	2
2 (DBA pilot representative)	1
3 (SA pilot representative)	4
4 (MA pilot representative)	1
5 (MS representative)	2
Valid	10

Data analysis

In this section, we present the analysis of the obtained data and put the results in both the DE4A and the wider context relevant for reusing the analyzed BBs. We then provide comparative analysis between the two evaluation phases of the BBs carried out in the DE4A project.

Inventory of BBs and functionalities

In this part of the survey, we inquired on the new functionalities and requirements defined for the BBs during the project life-time.

In 9 of the 10 cases, there were new functionalities defined for one or more of the implemented BBs, and in 7 of the 10 cases, new requirements as well. This is shown in Figure 1a) and b), respectively.

Of the functionalities, most noticeable were:

• Iteration 2: definition of notification request and response regarding the Subscription and Notification (S&N) pattern;
• Evidence request, DE/DO mocks;
• Average grade element was added to the diploma scheme, due to requirements at the Universitat Jaume I (UJI) to rank applicants;
• Automatic confirmation of messages;
• Canonical data models: additional information to generate other types of evidence;
• Capacity to differentiate between "environments" (mock, preproduction and piloting) in the information returned by the IAL, since in the second iteration we had just one playground for all the environments; and
• Deregistration, multi evidence.

Figure 1. a) New functionalities; b) New requirements for the existing BBs defined in DE4A lifetime

Of the new requirements[1], the following were noted:

• Those necessary to define and implement the above functionalities;
• SSI authority agent and SSI mobile agent were updated to simplify the SA UC3 ("Diploma/Certs/Studies/Professional Recognition") service;
• Subscription and Notification pattern, Evidence Request, DE/DO mocks (see project’s wiki solution architecture iteration 2, for requirements regarding Subscription and Notification); and
• Deregistration.

In addition to the new functionalities, we also inquired about the redundant ones, those that were already defined, but found unusable in the context of DE4A. Thus, in 4 of the (10) cases there were redundant functionalities found for the implementation of the pilot.

Despite the existing BBs that were on the list for evaluation, 4 of the partners also pointed out the following additional BBs or components that were employed in the pilots:

• Logs and error messages (in the MA-pilot);
• IAL / IDK (used by the Member States); and
• eIDAS (used by the SA pilot and the MSs).

With their use, the following additional functionalities were provided:

• Cross-border identification of students;
• Clarity and understanding; and
• Authentication and lookup routing information.

For the additional BBs, one new functionality was also defined during piloting: Common Errors.

Finally, the survey also inquired on the possibility of completely new BBs being defined during the lifespan of the project. According to the partners’ feedback, in 4 of the 10 cases such BBs were also defined, all of which were also regarded as reusable by other projects and in other cases as well, such as:

• For any SSI or verifiable credential like pattern;
• MOR semantics in EBSI and SDGR; and
• By IAL: for lookup routing information.

BB Interoperability

EIF/EIRA defines 4 dimensions with respect to interoperability: Legal, Organizational, Semantic and Technical. Thus, in addition to analyzing the contribution of each BB to interoperability (as defined by EIF/EIRA), we also delved into more granular inquiry on how each of the interoperability dimensions was addressed in the DE4A project (with the implementation of the given set of BBs). This was done both for the cases of the current implementation of the BBs, as well as in view of the potential of each BB to contribute to (the dimensions of) interoperability in contexts other than DE4A.

Figure 2a) shows the contribution of each BB to interoperability in the context of DE4A, while Figure 2b) granulizes this contribution per each interoperability dimension.

Furthermore, Figure 3a) provides insight into the potential of each BB to contribute to interoperability in contexts other than DE4A, while Figure 3b) depicts the contribution of the analyzed BBs per each interoperability dimension, in general. It is important to note that this latter analysis (of the BBs potential) also takes into account the additional functionalities of the BBs defined during DE4A lifespan, through their piloting, and with the updated set of requirements. Thus, these figures also provide a proof of the DE4A contribution in the improvement of BB potential to respond to a wider set of interoperability requirements along each of the four dimensions: Legal, Organization, Semantic and Technical.

Finally, as part of the analysis of BB interoperability, the survey asked the relevant partners to indicate if a BB could help enable other technologies or standards. In that sense, the DE4A-VC pattern makes use of a Wallet, which could be an enabler for the EUDI Wallet. Moreover, it was also denoted as having the potential for reuse in order to enable take up of EBSI. Furthermore, the DBA pilot makes use of SEMPER, which could facilitate the adoption and spreading of SEMPER as a standard for authentication and powers/mandates. Finally, the CompanyEvidence model could be used as evidence standard with implementation of SDG OOTS.

BB Maturity

In the first phase of the BB assessment, domain experts provided qualitative assessment of the maturity of the BB along the following dimensions:

1.   Technical

2.   Administrative

3.   Operational

The aim of this initial feedback was to provide the grounds for a comparative analysis over the same dimensions, after the current (second) phase of the BB assessment (i.e. check if anything changed meanwhile).

The following scale was used to provide input on the level of maturity of the given set of BBs by each relevant partner: 0 - Discarded; 1 - Useful, 2 - Acceptable, 3 - Recommended). This is the identical scale employed for the assessment of the BBs in the first phase. As a reference, the semantics behind these scores is also again given here on Figure 4 below.

However, there is one important aspect on which the BB assessment in this phase differs from the previous. Whereas previously we obtained a single evaluation score to determine the general maturity of a BB, in the current iteration, the 13 BBs were evaluated for each type of maturity (Technical, Administrative, Operational). This is actually fine-tuning of the initial methodology, which appeared insufficiently granular to provide correct output. Due to this lack of granularity and the inability to capture some contextual specificities, in the first phase the SEMPER building block was denoted as Immature, but was still Recommended for use in the DE4A.

As Figure 5 below show, none of the 13 BBs analyzed in this phase was Discarded for future implementation and reuse. The highest level of maturity was achieved Administrative context, where almost half of the BBs are considered to be completely aligned with current EU policies and only one BB (IAL) is denoted as unstable. From a technical maturity aspect, most of the BBs are implemented and running, while 2 (IAL and MOR) are still unstable and under development. Similar is the case with operational maturity, where 3 BBs are deemed as unstable: SMP/SML[1] , IAL and MOR. Hence, it is reasonable to pinpoint IAL as the least mature BB, which is however not to be discarded for reuse and re-uptake by other projects.

Figure 6 provides insight into the general state of maturity of each of the BBs, showing that each of the 13 building blocks integrates all aspects of maturity in its overall maturity. Furthermore, the figure also makes it evident that some of the BBs are more advanced in one aspect and less in others. For instance, the DE4A Playground demonstrates higher technical maturity, whereas its operational maturity has been reached to a lesser extent. Similarly, MOR’s maturity is mainly in administrative sense, and to a lesser extent in the technical and operational sense.

BB Openness

In this section, we analyzed the openness of each of the 13 BBs along several dimensions, i.e. properties: Extensibility, Customizability and Modularity. As Figure 7 shows, the most prevalent of all is Extensibility, present in most of the BB except the SSI User and Authority agent. Most of the BBs lend themselves to Customization, and more than half are also modular.

It is important to note, however, that 75% of the partners stated that the implementation, i.e. the use of some of the BBs is either technology, platform or solution dependent. For instance, IEM is DE4A specific, while CompanyData model is usable beyond DE4A. Furthermore, many components depend on a one-man company, introducing risks in terms of support and maintenance. Similarly, EBSI and federated services were denoted as solution/platform dependent, especially in the context of mixed environments .

Finally, half of the BBs were also marked as sector-specific. Thus, some are only relevant for public administrations, others for certain education environments (e.g., canonical data models in the SA pilot are specific to higher education), and some - for the business domain.

Table 3. BB usability traits in the context of DE4A

#	Building Block	Ease of deployment	Ease of configuration	Ease of integration with other BBs/existing SW	Barriers for implementation
1	eDelivery (data exchange)	It is embedded into the Connector; Easy	3/5; Tricky (certificates)	3/5; Easy	Certificates; Support by one-man company
2	SMP/SML	No Easy	No; Difficult	No Easy	Certificates; Support by one-man company
3	DE4A Connector	No Easy	No; Difficult	Fairly easy integration with data evaluator	No
4	DE4A Playground	N/A	N/A	Fairly easy integration with data evaluator	No
5	Information Exchange Model	Yes	4/5; Yes	4/5, yes, Transparent to data evaluators	No
6	Canonical Data Models	Fairly easy	yes	CompanyEvidence was easy to use with integration with BusReg; Fairly easy integration with data evaluator	No
7	ESL (implemented as part of SMP/SML)	5/5	3/5	2/5	No
8	IAL	5/5	2/5	2/5	Yes. Business cards from TOOP were reused, whose data model did not completely meet DE4A needs for the IAL functionality. However, a working solution was provided
9	MOR	N/A	N/A	N/A	N/A
10	SEMPER	Normal	Normal	Normal	No
11	SSI Authority agent	Yes	Yes	Yes	No
12	SSI User agent (mobile)	Yes	yes	Yes	No
13	EBSI-ESSIF (CEF Blockchain)	Normal	Normal	Normal	No

As a result of these experiences, practical recommendations for use and implementation of the BBs are also provided (see Table 4 below), together with pointers to some of the successful uses in the context of DE4A.

Table 4. Recommendations for use and implementation of the BBs

#	Building Block	Practical recommendation for use	Used in:
1	eDelivery (data exchange)	To make certificate management easier	Used by multiple Interaction Patterns (IM, USI, S&N, L, DReg?); Can be considered common infrastructure
2	SMP/SML	To make certificate management easier	Part of eDelivery (idem)
3	DE4A Connector	The concept of a Connector with an integrated AS4 gateway allowed for easier integration of DEs and DOs in the USI pattern. Decoupling the exchange and business layers allows for abstraction and adds flexibility to the exchange model. The DE4A Connector reference implementation helped MS to integrate their data evaluators and data owners and enable connectivity between the states more easily; Make certificate management easier.	Used for easier integration of data evaluators and data owners
4	DE4A Playground	The DE4A playground with mock DE, mock DO test connectors and shared test SMP proved successful for validating national connectors and SMP installations and easier DE and DO integration.	Used for easier integration of DEs and DOs
5	Information Exchange Model	N/A	To fully understand the XSDs; Transparent to pilots
6	Canonical Data Models	Assess fit for use in SDG OOTS; Extend with additional attributes/data; It was difficult to find a common denominator of higher education evidence from the three MS for applications to higher education and applications for study grants (some data required by one MS might not be obtainable from another MS). This will become even more difficult when doing the same among all MS. Many MS also have difficulties to provide the evidence required for certain procedures, e.g. non-academic evidence for the applications for study grants that can contain privacy sensitive data. The pilot suggested to the DE4A Semantic Interoperability Solutions (WP3) the Europass data model as the basis for the higher education diploma scheme in order to be able to use the same schema for both USI and VC pattern (and thus between SDG OOTS and revised eIDAS regulation).	Used for canonical evidence exchange
7	ESL (implemented as part of SMP/SML)	Documented in the proper guidelines[1]	Extension of SMP/SML, i.e. business cards
8	IAL	Documented in the proper guidelines	Conceptually part of IDK, central component providing routing information

Meeting pilot requirements

After BBs piloting implementation, we asked pilot partners to also document their experience in terms of BBs meeting the initial requirements for the particular piloting context. These experiences are listed in Table 5, which shows that most of the piloting requirements were met, although for some of the BBs new functionalities were provided (as discussed in the first subsection of this analysis) in order to achieve the piloting objectives.

Table 5. Inventory of piloting requirements related to each building block

#	Building Block	Piloting requirements that were met	Requirements that were not met
1	eDelivery (data exchange)	Message exchange; All	None
2	SMP/SML	All	None
3	DE4A Connector	Met (4.75 on a 1-5 scale; see D4.3); All	None
4	DE4A Playground	N/A	N/A
5	Information Exchange Model	Include the information to request and send evidence, even multiple evidence; All; Met (4.5 on a 1-5 scale; see D4.3)	None
6	Canonical Data Models	All; Met (4.5 on a 1-5 scale; see D4.3)	None; Missing element was added to the diploma scheme for the final phase
7	ESL (implemented as part of SMP/SML)	Identify the DO’s evidence service	None
8	IAL	Identify the evidence DO provider	The data model of the IAL does not support all the information on Administrative Territorial Units designed by WP3
9	MOR	N/A	N/A
10	SEMPER	N/A	N/A
11	SSI Authority agent	Met (4 on a 1-5 scale; see D4.3)	None
12	SSI User agent (mobile)	Met (4.5 on a 1-5 scale; see D4.3)	None
13	EBSI-ESSIF (CEF Blockchain)	N/A	N/A

As the barriers to meeting piloting requirements and smooth implementation may be of different nature, we have analyzed them in a finer granularity. This systematization of barriers has been rationed and defined in WP1, and can be found in Deliverable 1.8 – “Updated legal, technical, cultural and managerial risks and barriers”. Table 6, provides the partners experiences in terms of implementation barriers, describing them in the context in which they were encountered.

Table 6. Barriers to BB implementation

Type of barrier	Description of barrier
Legal	All MSs need to accept DLT; How to authorize a DE to request a canonical evidence type about a specific subject.
Organizational	Hire/Engage internal staff; The collection of signed information from partners to create the first PKI with Telesec, which took a lot of time and was very burdensome; Manual trust management, horizontal trust model
Technical	Invest in EBSI infrastructure; Allow federated until 2027; The need of knowing different external technologies and protocols in a very short time, such as eDelivery; maintainability
Semantic	Terms get antiquated need synonym hoover text; No major issues; Lack of canonical models at semantic level, lack of official pairings from local models to canonical models so translation can be done for each pair of MS
Business	Allow for private service providers to improve Usability UI; No major issues related to BBs.
Political	AI and ML in reuse of the Data; The obligation of using RegRep.; Following global standards
Human factor	Too little resources to SW dev; The availability of the personnel assigned to DE4A, which more often than not has not been the expected one. Additionally, the withdrawal of some partners (such as eGovlab); usability.

Figure 8 further details the level of criticality of the listed barrier in the context of DE4A.

All types of barriers have had elements of high criticality to be addressed, although the technical barriers had the highest criticality during implementation. These were mainly related to the need of knowing different external technologies and protocols in a very short time, such as in the case of eDelivery, but also to maintainability and sustainability of certain regulatory requirements related to technical implementation. High accent for all types of barriers was put on the time-frames needed to meet certain requirements, which were often in discord with the technical readiness of the current infrastructure and the human ability to respond to the required changes. Finally, the amount of available resources was present in all barriers - in terms of scarce human resources, technical expertise, administrative procedures, legislative readiness, infrastructure and staff availability, etc.

Performance and Non-Functional Requirements

The same systematization of the types of barriers investigated above is also ascribed to the various aspects (dimensions) of the building blocks through which we can analyze the BBs performance at a more granular level.

In that sense, Figure 9 shows the importance of each of these aspects (legal, organizational, technical, semantic, business, political and human factors) in the context of DE4A. In other words, the responding partners articulated the aspects that appeared as most important in their experience with piloting and implementation. The assessed BBs have mainly performed satisfactory across all dimensions, and even exceeded expectations on some technical and semantic traits. However, there is a (small) subset of traits across most dimensions on which the BBs also underperformed. These mainly refer to the barriers and the unmet requirements discussed earlier in this analysis. Some of them are also enlisted later in this section.

In terms of overall performance of each building block in the context of DE4A, Figure 10 shows that the only BB who was claimed as Underperforming is IAL, which we also pin-pointed in the maturity analysis as well, in each of the Technical, Administrative and Operational aspects.

Although there are no specific non-functional requirements (NFRs) with respect to performance, availability and scalability, and no performance tests were performed for DE4A, the idea of this section is to get a perception of the partners’ experience with non-functional requirements in the context of DE4A. Table 1 provides an overview of the partners’ claims.

Table 7. Partners’ experience with non-functional requirements

Type of NFR issue	Encountered	Foreseen
Performance	Response times and uptime for some components; Frequent issues with "external service outage" that would lower availability of services largely.	Connectors and SMP/SML can become single points of problems if having performance issues in the future when many DEs and DOs join; Hard to say as the number of users may be much higher in production; Possible - the BB should be stress-tested by EU/MS before recommendation.
Availability	Yes, due to software and configuration issues; Several components were not available (regularly); Yes, frequently, IP-changes led to connectivity issues; Yes, some services were unstable, but during piloting the situation improved.	Yes, including external services
Scalability	Yes. Components not designed for high-availability deployments, nor for flexible escalation.	Depending on use of components, support and development may depend on a one-man company.

Patterns

Some of the building blocks are used only for specific interaction patterns, while others span multiple patterns. In this section, we analyze the correlation between DE4A patterns and each of the 13 BBs, per pilot. As Figure 11In terms of overall performance of each building block in the context of DE4A, shows, the most employed is the User Supported Intermediation, closely followed by Verifiable Credential, whereas the least used are the Push [1] and Business patterns. However, this does not imply that these patterns are less useful, but only that they were exploited to a lesser extent in the DE4A depending on the pilots’ needs. For more documentation and guidelines on when and how each of these patterns can be used and implemented, please see the relevant pilot wiki.

---

This is apprently the deregistration thing?

S&N and LKP are used in DBA but should also be applicable to non-business context

The use of each of these patterns per building block is shown in Figure 12.