Goals and success criteria
Back to Doing Business Abroad main page
Back to main page of D4.7 Initial Running Phase Report
Back to Previous Chapter: 2. Current Status of Pilot
[Work in progress]
3.1 Goals and pilot success criteria
Objectives and how they are satisfied in relation to success criteria. Use D4.6 section 2.1 (Final version of success criteria and Common Pilot Criteria) as a basis.
GOALS (BLUE TEXT IS COPIED FROM D4.6)
| Actor | ID | Goal |
|---|---|---|
| Public authorities | A | Improve the quality of Company data within the service fulfilment process by re-using data from authentic sources, thereby reducing manual work and lowering processing costs. |
| Companies | B | Reduce manual work, lower transaction costs and improving enrolment speed for the company when using the Once Only Principle |
| Project | C | Evaluate the OOP-components supporting the cross-border information flow:
- Assess (technical) impact on national services/registers already in place - Evaluate connections of national systems to the OOP TS |
| D | Evaluate whether the solutions designed to the DBA specific challenges have proven adequate in piloting the DBA eProcedures:
- Usability of harmonised Company Evidence model - Degree to which powers must be validated - Scalability of solution for powers validation - Usability and security of Explicit Request and Preview - Need for record matching on Natural Persons - Adequacy of patterns to keep data up-to-date |
- Success Criteria for Public Authorities
| - ID | - Criterion | - Technical Common Criteria | - Principles |
|---|---|---|---|
|
Pilot goal A: Improve the quality of Company data within the service fulfilment process by re-using data from authentic sources, thereby reducing manual work and lowering processing costs | |||
| - A1 |
- The DE recognizes the company data is of higher quality, more reliable and easier to process when using the OOP TS to retrieve company data directly from the DO. (e.g. can data is available in an electronic and structured format for easy processing in the systems of the DE, data requires less correcting, data is kept up to date automatically, data is reliable and leads to less exceptions when processing, data is more meaningful, has less inconsistencies and errors, is more complete). |
- Reusability, Transparency, Effectiveness & Efficiency, Administrative Simplification |
- U, A, L, V |
| - A2 |
- The DE recognizes the method of powers validation to provide data of higher quality and reliability, proving that the representative is sufficiently authorized to represent the company (e.g. authorisation data is easier to interpret, authenticity is clear, data is trustworthy, there is less manual work in validating the users powers to represent the company with documents proving the relationship of the user to the company, authorization data requires less correcting, verification is easier). |
- Reusability, Transparency, Effectiveness & Efficiency, Administrative Simplification |
- U, A, L, V |
Success criteria for Companies applying for a service
| ID | Criterion | Technical Common Criteria | Principles |
|---|---|---|---|
|
Pilot goal B: Reduce manual work, lower transaction costs and improving enrolment speed for the company when using the Once Only Principle | |||
| B1 |
The user acknowledges the procedure for applying for a service to be effective and efficient (e.g. the procedure requires acceptable effort and cost, the procedure is not complex, has no language barriers, no interruptions. The user spends little manual time to correct company data, and experiences no errors after finishing the enrolment process). |
Reusability, Effectiveness & Efficiency, Administrative Simplification, Transparency |
U, A, L, V |
| B2 |
The user acknowledges the method to proof their authorisation as effective and efficient (e.g. requires little effort, is established with simple and effective communication, is reliable). |
Reusability, Effectiveness & Efficiency, Transparency, Security and Privacy |
U, A, L, V |
| B3 |
The user acknowledges the duration of completing the online eProcedure activities to apply for a service as acceptable. |
Effectiveness & Efficiency, Administrative Simplification |
V, A |
| B4 |
The user saves time and/or cost when completing the eProcedure using the OOP TS. |
Effectiveness & Efficiency |
V, A |
Success criteria and research questions for Pilot Technical Goals
| ID | Criterion | Technical Common Criteria | Principles |
|---|---|---|---|
| Pilot goal C: Evaluate the OOP-components supporting the cross-border information flow:
· Assess technical impact on national services/registers already in place · Evaluate connections of national systems to the OOP TS | |||
| C1 | The DO believes the cost and effort for integrating to the DE4A Connector will eventually be outweighed by the benefits. | Openness, Technical Neutrality and Data Portability | U, A, V |
| C2 | The DE believes the cost and effort for integrating to the DE4A Connector will eventually be outweighed by the benefits. | Openness, Technical Neutrality and Data Portability | U, A, V |
| C3 | The DO believes the cost and effort for integrating to the Mandate Management System will eventually be outweighed by the benefits. | Openness, Technical Neutrality and Data Portability | U, L, V |
| C4 | The participating Member States believe the cost and effort for setting up and deploying the DE4A Connector in their national infrastructure will eventually be outweighed by the benefits. | Openness, Technical Neutrality and Data Portability | U, L, V |
| Pilot goal D: Evaluate whether the solutions designed to the DBA specific challenges have proven adequate in piloting the DBA eProcedures | |||
| D1 | Has the Company Evidence Model proven adequate for cross-border exchange of information on companies for the DBA eProcedures? | Openness, Neutrality and Data Portability, Reusability | U, V, L |
| D2 | Have the solutions to validate powers proven adequate for the eProcedures involved in piloting? | Reusability, Administrative Simplification | U, L |
| D3 | Have the explicit request and preview requirements as specified in the SDGR proven suitable for company eProcedures (representation scenarios)? | Administrative Simplification, User Centricity, Inclusion and Accessibility | U, L |
| D4 | Have the mechanisms for record matching at the DC proven adequate for the DBA eProcedures? | Administrative Simplicity | U, L |
| D5 | Have the mechanisms to keep the company information up-to-date (second pilot iteration) proven adequate | Administrative Simplicity, Effectiveness & Efficiency | U, V |
3.2 Pilot dimensions
Qualitative description on lessons learned (technical, functional, proess, data, usability etc) and preliminary conclusions on these dimensions, based on metrics, questiuonnaires and interviews? The dimensions target the scope of the piloted functionality and patterns (until delivery of the report)
3.2.1 Use
- Overview
- Initial feedback from focus group and real users
- Initial results from use related metrics (logs)
- Usefulness of DE4A patterns and components related to internal stakeholders take-up
- Strategy on pilot use until final report
3.2.2 Value
- Verified benefits with users and DEs / DOs
- Contribution of pilot to DE4A benefits and to external community of SDG stakeholders
- Pilot specific benefits
3.2.3 Learning towards Adoption
- Approach to knowledge-building
| topic | observations | lessons for wider adoption and implementing SDG | |
|---|---|---|---|
| 1 | design process | Designing national integration required in depth knowledge of both eIDAS and OOTS. This knowledge (specifically the combination of both) is not broadly available in Member States. Knowledge of both domains should be brought together in order to prevent designing based on false assumptions of the other domain. | DBA advises Member States to invest time to bring together the eIDAS and OOTS knownledge. This requires organising and prioritising as this knowledge is scarse. |
| scoping | The project ran into a lot of complex topics that needed to be solved in the pilot desing. The pilot lead has organised a lot of meetings discussing these topics requiring a lot of resources.
In the process the pioot partners agreed to simplify whenever possible, e.g. focussing at most important evidence type instead of all possible types, accepting request for one single evidence type at the time (instead of allowing requests for multiple evidence types), limiting to full powers validation to start with. The pioot greatly benefitted from strict scoping. |
DBA advises the European Commission and Member States not to solve all user scenario's at once, but to focus on the most frequently used ones. Please focus on core functinality. And at the same time organise follow-ups on improvements and additions to address later on. | |
| company representation | Use of eIDAS including legal entity attributes (company representation) is not widespread in the EU. Currently, there are just two eID scheme's notified including legal person attributes. In preparing for the pilot, Member States turned out to communicate company representation in different ways. Especially regaring the use of the eIDAS representative attributes (representative prefix). During pilot preparation eIDAS node 2.4 became available. This version of the CEF reference software enforced the eiDAS 1.2-specification that turned out to be in conflict with the agreed use of eIDAS attributes in the DBA pilot. As a result, this raised confusion and led to inconsistent deployments. The eIDAS 1.2 specification regarding representation is not necesarilly backwards compatible. | DBA advises the European Commission to clarify in advance which version of the eIDAS specificaton should be implemented for SDGR to prevent incompatibility between Member State communicating legal person attributes. | |
| powers validation | Validating full powers has been proven to be a first (and good) step in implementing cross-border OOP for businesses (requiring company representation). It allows for moving ahead with eIDAS as is available today, is acceptable to the DE's participating in the pilots and seems fitting for SME's (it will be an official representative initiating doing business abroad most of the time). | DBA advises Member States to focus at implementing full-powers validation flows to start with. Adding more fine grained powers validation is required for 100% implementing the ePorcedures, but also adds complexity to the solutions.
| |
| record matching | The pilot partners agreed to provide the national company registry numbers as eIDASLegalIdentifier. This diminished the need to do record matching on companies at the data owner. | DBA advises Member States to use the national company id's as eIDASLegalIdentifiers when extending the pilot to SDG-wide implementation. | |
| explicit request | In some cases, users need to express consent for the retrieval of attributes. In almost all cases when using the OOTS, the user needs to express explicit request. Although legally sound, in practise the difference between both is difficult to understand for data evaluators. DE's furthermore expect that users might ignore such requests and just click "ok". | DBA advises data evaluators to integrate request to consent and explicit request into one joint question to the user to prevent adding to the confusion. | |
| Multiple-MS scenario's | Three- or multiple Member State scenario's (add examples) tend to get really complex, requiring disproportionate resources to analyse, design and implement. | DBA advises Member States to start SDG-implementation with the simplest interaction patterns involving just two Membert States. | |
| eIDAS non-notified eID | Most of the participating Member States dind't operate a notified eID at the moment of piloting. On a bilateral basis non-notified eID's were accepted for piloting purposes, although pilot partners expressed there doubts regarding acceptance of non-notified eIDs for large scale SDG. Notification of eID's is a strong prerequisite for implementing SDG. Mandatory eID-notification as expected under the new eIDAS regulation (eIDAS revision) will not be in time for SDG-implementation. | DBA advises The European Commission and the Member States without notified eID's to agree on a temporarily solution for using non-notified eID's in SDG-procedures. | |
| sector specific systems | For the DBA pilot alignment to - or integration with - BRIS has been an important topic from the start of the project. A lot of time has been spent on workshops, desk research and analysis. In the end, re-use of BRIS has been limitied to semantics. Re-use of information flows, building blocks, etc. was not possible due to difference in legal framework, solution implemented, authorities involved, governance, etc. | Integration of the OOTS with sectoral systems (BRIS in this pilot) has proven to be not so straight forward as many expexted at the start of the project. | |
| user interaction design | Several data evaluators needed to implement the same logic in their specific systems, including user iteraction (general explanaition, explicit request, preview). The user interaction design across participating Member States turned out to show great differences in formulating texts, detail of explanation, use of buttons, etc. This may lead to confusion for the user that deals with multiple data evaluators as well as a slow learning curve. DBA decided to provide a pilot-wide reference in the form of wireframes. | DBA advises the Europeon Commissin to provide wireframes in order to have generic steps (like Explicit Request and Preview) implemented in a similar way in all MS. |
| topic | observations | lessons for wider adoption and implementing SDG | |
|---|---|---|---|
| 1 | planning and organising tasks | components to be used (in the pilot) are sometimes distributed over several authorities in a Member State, requiring the commitment from all authorities. This commitment is not obvious and must be secured beforehand. Also, as the systems are distributed, the teams working on the systems are distributed. Collaboration takes more time and in each team, a battle for prioritizing needs to be fought. | DBA advises to allocate a multi-month phase for involving all organisations involved, aligning between them, prioritising, allocating financial means, etc.
Finally, it is necessary to have a coordinating team (having sufficient knowledge about the solution) in each Member State to make sure that planning and communication issues are being resolved. |
| 2 | handing over | Design documents and specification can often be interpreted in several ways. During preperation of the pilot or during interoperabuility testing such differences surface. It would be better to have a common understanding of all the design details priori to the testing phase. Lesson learned within DBA: take the time for handing over Solution Architecture to WP3 and 5, and make sure that everything is understood. | DBA advises the European Commision to put additional efforts in explaining the workings of the SDG OOTS components to public authorities involved. The better the solution is understood by all, the smooher the SDG implementation will be. |
| 3 | documenting | For developers of the common components, there's a lot of logic behind its function, structure, configuration, etc. Deploying these components by the Member States raised several questions regarding specifically the use of Docker images, configuration and required firewall and DNS settings. Things were not always as straight forward as expected. | DBA advises the European Commission to invest in proper and clear documentation for developers in MS, so they can get the DE4A Connector up and running, as well as integrate to the DE4A Connector wit minimal effort. Documentyatkion should not be too cryptic and short, but definately also not too extensive. Feedback from first movers has proven to be very useful in the DBA pilot. |
| 4 | configuring | The components needed for SDG rely heavily on use and exchange of certificates for server authentication, signing, etc. The process of acquiring the certificates required turned out to be very time-consuming and error-prone (all details must be in order when requesting the certificates). Furthermore, the procedure of requesting certificates is regulated in a way it requires signatures of repsonsible people within the requesting institution that do not on a daily basis work with - and understand the use of - certificates. Or people that are not available at any point in time (company executives). | DBA advises Member States to prepare for the steps to be taken before receiving the required certificates needed to operate the OOTS.
DBA advises the European Commission to investigate whether the process for acquiring the certificates for OOTS can be simplified or streamlined. DBA advises the European Commission to design a procedure for communication between Member States in case of change of certificates and to provide for certificate-rollover to guarantee OOTS connectivity. |
| 5 | integrating DE and DO |
|
DBA advises Member States to take impact on existing systems into account. |
| 6 | interopability testing | The speed of development varies per Member State. Therefore readiness for cross-border testing (and piloting, for that matter) is also distributed in time. MS A can have their DE ready months before MS B has (due to several impediments). Testing on fixed moments in time for all DEs and all DOs has proven not realistic, as does starting pilots at one moment in time. | Wider OOTS implementation requires inter-Member State coordination regarding exchange of connectivity details, configuration and cross-border interoperability testing. Planning of these activities requires additional attention and flexibility of Member States. DBA advises to take this into account when connecting the decentralised SDG OOTS components.See also We refer also to the eIDAS lessons learned with regards to exchange fo certificates etc. on EU-wise scale. |
| 7 | interopability testing | The DBA pilot has proven that a lot of settings need to be configured corectly to allow succesful cross-border evidence exchange. During interopatbility testing (connecthatons) Member States sometimes had different views on what components or parameters had to be set in order to start testing. As a result, not in all cases the complete flow could be tested at once. | Establish clear readiness criteria for DE/DO/DE4A Connector before starting connectathons. |
| 8 | interoperability testing | Proper interoperability testing is only possible with the required test eID means. These national eID means have proven not always to be easily available (depending on the MS-specific situation). The effect of missing test credentials will be much greater in case of large scale implementing the SDGR. | DBA advises the European Commission to coordinate exchange of test credentials between Member States. Many-to-many requesting and sending of eID's on a bilateral basis should be prevented. |
| 9 | reliance on eIDAS | DBA pilotting - just as SDG implementation - relies on use of eIDAS. Unformtunately, eIDAs is not fully up and running in alle Member States. In preparing to for the DBA pilot, Member States had to setup eIDAS as well. In interoperasbility testing several eIDAS related setup-issues needed to be solved. | DBA advises Member States to setup and test national eIDAS deployment as soons as possible, to prevent this delaying SDG. |
| 10 | SDG implementing acts | DBA pilot implementation has been delayed by numerous discussions (within Member States and between Member States) on alignment with the SDG OOTS that was being sketched at the same time. Although this approach has been deliberately chosen and agreed upon at the start of the DBA project (to enable real piloting and provide input to SDG), in practise discussions were raised over and over again. | DBA advises the European Commission and Member States to be aware no such thing as 'a final version' exists in the area of inter-Member State information exchange. Moving forward step-by-step with versions currently available is crucial to progress. Note that continuous alignment with all European initiatived during single steps is not feasible and will delay each initiative started. |
- Keep audit-trail and error-logging simple, considering the limited number of participating companies and the highly controlled fashion of the pilot.
- Collect pilot data in the most simple way possible, without risking data loss or integrity-loss. But don't set up complex systems to collect data for metrics.
| topic | observations | lessons for wider adoption and implementing SDG | |
|---|---|---|---|
- Prepare the creation of an animation by setting up a good storyline and slides that illustrate the flow of the animation.
- Slack seems to be a good means to have developers of different MS / WPs collaborate
- Also use input from additional questionnaire regarding technical issues and process of connecting (connectathons) and documentation etc
| topic | observations | lessons for wider adoption and implementing SDG | |
|---|---|---|---|
| 1 | harmonisation | For fine-grained powers validation, Member States need to agree on a harmonised set of services, e.g. the SDG procedures of annex II. | DBA advises the European Commission to organise the process of harmonising services for cross-border powers validation (see SEMPER project results and DBA pilot for sample harmonisation). |
| 2 | harmonisation | For subscribing and notifying on company events / changes there needs to be a specified set of harmonised company event types. | DBA advises the European Commission to organise the process of harmonising event types for cross-border subscription & notification. See DBA pilot for example of harmonisation. |
- Lessons being learned from users (questionnaires & interviews)
- Lessons being learned from DEs and DOs (results and outputs questionnaires & interviews)
-
- Bank account information is not part of the CompanyEvidence. But it can be requested to provide afterwards, in screens in the eProcedure after exchange of evidence by the OOP TS. It seems that BIC-codes are unique, but BANK-codes (like INGB etc) are not unique across Europe (but are unique within one country). The DE might do some kind of validation on the bankcode and might then run into issues because of new bank-codes (from Romania, for example) that are identical to bank-codes in the country of the DE (The Netherlands for example). DE systems should therefore consider closely how to work with bank information (when entered manually, but also in case of future extension of CompanyEvidence).
- Other lessons from interaction with other initiatives (SEMPER, EBSI…)
- The added value of the OOP TS compared to other developments / pilots / experiments in the EU must be broadcasted continuously in order to keep authorities involved and secure commitment.
- (Potential) differences between the DE4A scope and the Implementing Act distract and demotivate participants from prioritizing development and deployment of changes in their national infrastructures.
3.3 Technical common criteria (questionnaire for evaluation?)
[Qualitative description of preliminary conclusion per criterium. Explanation (in written) on how success criteria /metrics are related with Technical common criteria, distributed by pilot dimensions]
Openness => U, A
Transparency => U, V
Reusability => V, L
- eIDAS tech profile 1.2 is more prescriptive than profile 1.1. Depending on the way profile 1.1 is implemented in a MS, it might not work with the way 1.2 prescribes the implementation of this version, introducing an error in the authentication process between MS A (using 1.1) and MS B (using 1.2).
Technological neutrality and data portability => L
User-centricity => V
Inclusion and accessibility => U
Security and privacy => U
Administrative simplification => A, V
Effectiveness and efficiency => A, V
- Not sure if this is the right pace, but we should address the design decisions we made too. And for that matter, also say something about how we comply to the SDG (or where we deviate and why, for example we work with 1 company evidence type while the SDG states that a DE should not receive more attributes than they strictly need for the procedure, which might not always be the case in DBA). Another example is that the preview is provided by the DE, meaning that the transfer has already taken place.
[Qualitative comments, and follow-up with quantitative comments on second iteration]