EDAMIS How to exchange files with EDAMIS 4

EDAMIS How to exchange files with EDAMIS 4 Javier MANSO DEL VALLE Unit A. 3 - IT solutions for statistical production Eurostat October 22 nd, 2019

Agenda • EDAMIS and the different networks • The transmission methods of EDAMIS • How to choose the right transmission method • How to exchange files in each method

Also check • "How to exchange files using EDAMIS 4" • Transmission methods explained and compared • Step-by-step instructions • Further details

EDAMIS and the different networks

EDAMIS and the networks • Internet • Used by EDAMIS for many years • Global network, low cost INTERNET • TESTA • Objective: exchange of data between public administrations in Europe TESTA • CCN • Objective: exchange of data in the domain of customs and taxation CCN

Member State European Commission CUSTOMS CCN INTERNET TESTA NATIONAL NETWORK

The transmission methods of EDAMIS

Member State European Commission WEB PORTAL s. FTP server ESDEN Client AS 4 Access Point EDAMIS AS 4 access point

WEB PORTAL • Needs standard software • Web browser • How it works • User logs in to a web page • Navigates using a GUI

s. FTP • Needs standard software • s. FTP client • Compression software that supports ZIP • Encryption software that supports PGP • How it works • Create XML file with metadata • Package data and XML file in a ZIP • Transmit the ZIP with s. FTP to a Eurostat server

AS 4 • The AS 4 protocol • New standard protocol for file exchange • Oriented to business-to-business • Interoperable • Based on proven technologies: web services, MIME • Reliable • Guarantees once-and-only-once delivery • In the roadmap for EDAMIS

AS 4 • Needs standard software • An AS 4 access point • How it works • • Configure the access point Create an XML file with the metadata Package data and metadata in a zip file Send the file using the AS 4 access point

ESDEN Client • Needs non-standard software • ESDEN Client, developed by Eurostat • How it works (option 1) • Create XML file with metadata • Package data and XML file in a ZIP • Transmit the ZIP with s. FTP to a Eurostat server • How it works (option 2) • Use the EDAMIS naming convention for the file • Place it in a specific folder

How to choose the right transmission method

Transmission methods are equivalent • There are different transmission methods • Different software needed • Different characteristics • Each is a better fit for different scenarios WEB PORTAL s. FTP AS 4 ESDEN CLIENT • They are all equivalent and coherent • The processing of the file is independent from the transmission method used

Standard vs. non standard methods • Standard methods • Can be used with any software that supports the standard • IT departments are free to select the product, acquire it and support it following internal rules WEB PORTAL s. FTP AS 4 • Non standard methods • Need specific software provided by Eurostat • Dependency: only Eurostat can provide support ESDEN CLIENT

Why non-standard methods? • To facilitate access to the CCN network • Adapt to proprietary CCN protocols • To facilitate adoption of EDAMIS 4 • Replacement of EWA • Recommendation • Privilege standard transmission methods

Manual vs. automatic data exchange • Manual data exchange • Intended to be used by persons • Needs a human: uses password, keyboard, mouse • For individual files WEB PORTAL ESDEN CLIENT • Automatic data exchange • Machine to machine communication • For mass transfer of files or repetitive tasks s. FTP AS 4 ESDEN CLIENT

Support for large files • Indicated for large files s. FTP AS 4 ESDEN CLIENT • Not recommended for large files WEB PORTAL

Facilities for encryption • Encryption must be performed by the sender • User selects the right encryption key • User encrypts with a standard tool s. FTP AS 4 • Provide facilities for encryption • Automatic selection of the right encryption key WEB PORTAL ESDEN CLIENT

Local installation and maintenance • The software needs significant effort for installation and maintenance s. FTP ESDEN CLIENT AS 4 • The software needs low effort for installation and maintenance WEB PORTAL

Accessible through secure networks INTERNET WEB PORTAL s. FTP AS 4 ESDEN CLIENT TESTA WEB PORTAL s. FTP AS 4 ESDEN CLIENT CCN WEB PORTAL ESDEN CLIENT

Transmission methods compared WEB PORTAL Machine to machine Supports large files Encryption facilities Local installation needed Standard method Accessible through Internet Accessible through TESTA Accessible through CCN s. FTP AS 4 ESDEN CLIENT

How to exchange files using each method

WEB PORTAL • Open the URL in your browser • Internet: https: //webgate. ec. europa. eu/edamis 4 • TESTA: https: //webgate. ec. testa. eu/edamis 4 • CCN: https: //webgate. estat. ccncsi. int/edamis 4 • Authenticate using EU Login • WARNING: change of URL • Estimated: december 2019 • CCN: https: //webgate. taxud. ccncsi. int/edamis 4

WEB PORTAL • Option 1 • Sending the file via the widgets on the main page

Dashboard configuration

WEB PORTAL • Option 2 • Sending the file via the « Send datafile » page

Click on the "Transmissions" menu

In the menu, select "Send datafile"

Drag and drop your file in this area

The file follows the EDAMIS naming convention… … so all metadata fields are filled automatically Click here to perform the official transfer

s. FTP • The first time: register with Eurostat • Organisation provides • • Name of the organisation Network that will be used (Internet or TESTA) PGP key to enable signature A contact e-mail address • Organisation receives • User name and key to connect with s. FTP

s. FTP Member State European Commission INTERNET s. FTP Client s. FTP server TESTA EDAMIS s. FTP server

<metadata> <format. Version>2. 1</format. Version> <transmission> Version of the XML Only changes with new versions of EDAMIS <transmission. Id>EU-ESTAT-SFTP 03_201810191452</transmission. Id> <from> <country>EU</country> <organisation. ID>4 D 0_B 3</organisation. ID> <sender. Email>user. name@company. com</sender. Email> </from> <dataset. Id>EDAMIS 1_A</dataset. Id> <year>2018</year> <period>0000</period> <file. Format>txt</file. Format> <data. Filename>example. txt</data. Filename> <is. Encrypted>false</is. Encrypted> </data> </transmission> </metadata>

<metadata> <format. Version>2. 1</format. Version> Unique ID for each transmission <transmission> <transmission. Id>EU-ESTAT-SFTP 03_201810191452</transmission. Id> <from> <country>EU</country> <organisation. ID>4 D 0_B 3</organisation. ID> <sender. Email>user. name@company. com</sender. Email> </from> <dataset. Id>EDAMIS 1_A</dataset. Id> <year>2018</year> <period>0000</period> <file. Format>txt</file. Format> <data. Filename>example. txt</data. Filename> <is. Encrypted>false</is. Encrypted> </data> </transmission> </metadata>

<metadata> <format. Version>2. 1</format. Version> <transmission. Id>EU-ESTAT-SFTP 03_201810191452</transmission. Id> <from> <country>EU</country> <organisation. ID>4 D 0_B 3</organisation. ID> <sender. Email>user. name@company. com</sender. Email> </from> <dataset. Id>EDAMIS 1_A</dataset. Id> <year>2018</year> <period>0000</period> <file. Format>txt</file. Format> <data. Filename>example. txt</data. Filename> <is. Encrypted>false</is. Encrypted> </data> </transmission> </metadata> Sender information

<metadata> <format. Version>2. 1</format. Version> <transmission. Id>EU-ESTAT-SFTP 03_201810191452</transmission. Id> <from> <country>EU</country> <organisation. ID>4 D 0_B 3</organisation. ID> <sender. Email>user. name@company. com</sender. Email> </from> <dataset. Id>EDAMIS 1_A</dataset. Id> <year>2018</year> <period>0000</period> <file. Format>txt</file. Format> <data. Filename>example. txt</data. Filename> <is. Encrypted>false</is. Encrypted> </data> </transmission> </metadata> File info: dataset, reference period, format, encryption

s. FTP • To send a file • • • If confidential, encrypt the data file Create an XML file with the metadata (Optional) Sign the file Package the XML and the data file in a ZIP file Connect to the s. FTP server of Eurostat Upload it to the folder /from. User/ • To receive a file • Connect regularly to the s. FTP of Eurostat • Check the contents of folder /to. User/ • Unpack the ZIP files received

AS 4 • In the roadmap for EDAMIS • Data Transmission Coordinators group will be consulted about AS 4 in its 2020 meeting

ESDEN Client • The first time: register with Eurostat • Organisation provides: • • Name of the organisation Network that will be used (Internet, TESTA, CCN) PGP key to identify the ESDEN Client A contact e-mail address • In addition, if using CCN: • Organisation contacts national CCN administrator • Requests CCN configuration • Receives a configuration file

ESDEN Client • The first time: install and configure the ESDEN Client • • Verify network connectivity Receive the software from Eurostat Unpack in an empty folder Modify configuration file for the individual case • Eurostat provides support for the installation

ESDEN Client • To send a file (option 1) • Create an XML file with the metadata • Package the XML and the data file in a ZIP file • Copy the file to: …/outbox/user@company. org/ • To send a file (option 2) • Use the EDAMIS naming convention for the file • Copy the file to: …/outbox/user@company. org/

ESDEN Client • To receive a file • Files appear in the inbox folder • …/inbox/<domain>/<dataset>/<transmission>/ • Maintenance needed • Outbox: files ending in. retry and. error • Inbox and Sent: cleanup historical files