MDG DATA AND METADATA EXCHANGE AMONG NATIONAL AGENCIES

MDG DATA AND METADATA EXCHANGE AMONG NATIONAL AGENCIES AND WITH INTERNATIONAL ORGANIZATIONS The Experience of Mexico Enrique Ordaz 19 - 21 October, 2011 Manila

Statistics and Geographic Information Law

National Institute of Statistics and Geography, INEGI COORDINATES THE SYSTEM AUTONOMY PRODUCES INFORMATION

The new statistical (and geographic information) system NATIONAL COUNCIL SUBSYSTEM Socio Demographic Information Economic Information Geographic and Enviromental Information Specialized Technical Committees Working groups UE UE Working gropus National Information Programs INEGI Catalog of National Indicators SUBSYSTEM UE UE Goverment, Public Security and Justice Information Specialied Technical Committees Working gropus Norms UE UE Coordination Data dissemination Society State

MDG Indicators Committee Ø A Specialized Technical Committee was set up in February 2010 to coordinate the integration of the Millennium Development Goals Indicators.

Mexico’s MDG Indicators Lista oficial de los indicadores de la ONU Objetivo Total 1. Erradicar la pobreza extrema y el hambre 2. Lograr la enseñanza primaria universal 3. Promover la igualdad de género y el empoderamiento de la mujer 4. Reducir la mortalidad de los niños menores de 5 años 5. Mejorar la salud materna 6. Combatir el VIH/SIDA, el paludismo y otras enfermedades 7. Garantizar la sostenibilidad del medio ambiente 8. Fomentar una alianza mundial para el desarrollo Sistema de Información de los Objetivos de Desarrollo del Milenio, México 2010 Indicadores Metas De Más allá Preexistent Reformulad Propuestos de las Metas Total e os por México del Milenio 22 70 22 80 40 8 12 20 3 9 5 14 8 1 0 5 1 5 3 14 5 0 1 8 1 7 5 1 1 0 1 3 1 6 3 0 0 3 2 6 4 2 0 0 3 13 4 15 4 1 7 3 4 12 4 13 8 3 1 1 6 16 1 3 3 0 0 0

General schedule of the work program 2010 Activities Enero Febrero Marzo Abril Mayo Junio Julio 2011 Agosto Septiem- Octubre Noviem- Diciembre bre Enero Febrero 1234123412341234123412341234 • Committee Meetings • Data integration by working groups MDG System 1 st Phase 2 nd Phase 3 rd Phase • Drafting of the 2010 country report

Coordination Working groups 1. Poverty, income and nutrition 2. Poverty, employment 3. Education and gender equality 4. Maternal and child mortality, rep. health 5. VIH/AIDS, malaria, other diseases 6. Environment and natural resources 7. Improved water access, and sanitation Ministries responsible for reviewing and updating metadata; updating basic data and indicators 8. Global partnerships for development General Coordination: Chrístel Rosales (DGPAE) Agreements follow up: César Garcés (CONAPO) INEGI: technical assistance; data integration and development and management of the website.

Process for updating the MDG system oceso de actualización Sistema ODM

Activities for the operation of the project by INEGI 1. Prepare manuals of procedure and rules. 2. Create an FTP site for data exchange. 3. Backing up time series and metadata from the FTP site. 4. Classify, analyze and assess the data and metadata to identify possible inconsistencies. 5. Ask for clarifications from each responsible agency when problems are found in the data. 6. Update the MDG system.

Form to incorporate data and indicators in the MDG system

Metadata form Concept Name of the indicator Definition Algorithm Meaning of acronyms Source of primary data Geographic coverage Frequency Updating date Name of agency responsible for the indicator Importance and usefulness of the indicator International reference Remarks

Procedure to compile and disseminate the information 1. The ftp: //200. 23. 8. 226/ site contains the following information: Ø Formal the creation of the MDG Technical Committee. Ø MDG 2006 Manual; draft of the 2009 manual; Report from de Secretary General. Ø Mexico’s reports from 2005, 2006. Ø Relationship of responsible agencies, indicators and data sources. Ø Terms of reference of the MDG Technical Committee. Ø Dossiers for each one of the working groups. Each dossier contains: ü Indicators’ metadata and the rules for updating it on line. ü Raw statistics produced or integrated by INEGI for updating the indicators. ü A document with the method of calculation for each indicator

Procedure to compile and disseminate the information 2. Within each working group a person is designated to calculate the indicator. 3. INEGI provides access to FTP: //200. 23. 8. 226, with different rights.

Procedure to compile and disseminate the information 4. Once the work has been finished and authorized, the designated person will up load on the FTP site all the statistical series of raw data to calculate the indicator, as well as the series of the indicator itself, with the corresponding metadata. 5. INEGI reviews and assesses the data and the indicators, and asks for clarifications. 6. Once the data are cleared they are put on an internal server for final review by the Technical Committee. 7. Data are published on the website.

Weekly report

Observations Typology of comments made by INEGI on the data produced bay other agencies 1. Differences in name of indicators: system vs. working groups. 2. Differences or missing metadata. 3. Lack of URL links in the metadata. 4. Acronyms not explained. 5. Inconsistencies in the raw data 6. Differences between the data produced by INEGI and the data published by the government agencies. 7. Unexplained breaks in time series of indicators or the raw data.

Moving MDG to SDMX • We used a preliminary version of the DSD for MDG and we made the following changes and we finally have a NEW DSD for MDG • It was very useful, but we had to make the following changes in order to adapt it to our information and tools. So we have a NEW DSD for MDG: – We changed the encoding XML ISO-8859 -1 per utf-8 because our tools use encoding = "utf-8“ – The XML namespaces are generated directly from our Web Service for each type of message. So it can be seen that some namespaces (i. e. the required for data messages as "compact", "generic", etc. ) do not appear in the DSD proposed by INEGI, but on the other side, it is a new one that is the xmlns: registry that is required in a DSD.

Moving MDG to SDMX • 34 MDG indicators. • We have adjusted some code lists: 9 were used, of which 6 are consistent with the UNSD and 3 were included by the INEGI. • The codes are alphabetically sorted in Spanish. – Two age groups were added because of the particularities of the indicators presented by Mexico. – One class added: ND Not determined

Progress in the conversion of MDGs to SDMX The names of the catalogs were changed, removing _SDMX or _ODM, because by recommendation the identification should be as generic as possible so they can be used by other flows. ANTES AHORA CL_UNIT_MULT_SDMX CL_UNIT_MULT CL_NATURE_MDG CL_NATURE CL_UNIT_MDG CL_UNIT_MEASURE CL_SOURCE_TYPE_MDG CL_SOURCE_TYPE CL_LOCATION_MDG CL_LOCATION CL_AGE_GROUP_MDG CL_AGE_GROUP CL_SEX_MDG CL_SEX CL_SERIES_MDG CL_SERIES CL_REF_AREA_MDG CL_REF_AREA CL_FREQ_MDG CL_FREQ CL_UNIT_MULT_SDMX CL_UNIT_MULT CL_NATURE_MDG CL_NATURE CL_UNIT_MDG CL_UNIT_MEASURE CL_SOURCE_TYPE_MDG CL_SOURCE_TYPE

Progress in the conversion of MDG to SDMX • • Added Concept. Scheme CS_MDG – This component (is required for SDMX 2. 0), it contains a list of concepts applicable to the DSD Key Family MDG was changed to DSD_MDG because it is recommended that the nomenclature of the id is based on the first initials of the type of device (DSD Data. Structure, Codelist CL, Scheme Concept CS, etc) followed by the identifier. The following 3 attributes changed from observation to sets level, because for Mexico, in the retrieval systems and original databases, these features are specified for each series. This change produces smaller SDMX files. – UNIT_MULT – TIME_DETAIL – FOOTNOTES The next three code lists were updated because the INEGI information could not be classified with the original code lists: – CL_FREQUENCY, – CL_AGE_GROUP – CL_UNIT_MEASURE

United Nations Statistics Division DSD for the MDG DSD STRUCTURE • DIMENSIONS CODELIST (CL) 1. 2. 3. 4. 5. 6. Time period Frequency Series Location Sex Age group CL_FREQ_MDG CL_SERIES_MDG CL_LOCATION_MDG CL_SEX_MDG CL_AGE_GROUP_MDG 7. 8. 9. Reference area Units of measurement Source Type CL_REF_AREA_MDG CL_UNIT_MDG CL_SOURCE_TYPE_MDG APPLICATION Yes Yes- CL Supplemented Yes ATTRIBUTES 1. 2. 3. 4. 5. Unit multiplier Time period details Nature of data points Source Detail Footnotes CL_UNIT_MULT_SDMX CL_NATURE Yes Yes Yes

CODE LIST FOR UNITS OF MEASUREMENT CODE DESCRIPTION SOURCE 1 Not applicable ONU 2 Deaths of children under five per thousand live births INEGI-New 3 Deaths of children under one year per thousand live births. INEGI-New 4 USD in end-2006 net present value terms ONU 5 USD ONU 6 Children per thousand women INEGI-New 7 Number ONU 8 Kilograms person INEGI-New 9 Kg oil equivalent per USD 1, 000 constant 2005 PPP GDP ONU 10 Square kilometers ONU 11 Local currency ONU 12 Local currency per USD (PPP) ONU 13 Women ONU 14 Women for men INEGI-New 15 Live Births ONU 16 Population ONU 17 Per hundred thousand INEGI-New 18 Percent ONU 19 Per 1 USD GDP (PPP) ONU 20 Metric tons ONU APPLICATION Not used Used Used Not used Not used Used Not used Used Nota: For the Mexico’s DSD of 34 MDG indicators, 9 classes were used (in purple), of which 3 are consistent with the Catalogue of the UN and 6 were included by the INEGI. About the codes, were used consecutive numbers, arranged sorted alphabetically terms in Spanish

CODE LIST FOR AGE GROUPS CODE DESCRIPTION Z Not applicable 1 Under 1 year olds 2 Under 5 year olds 3 6 -11 year olds 4 10 -14 year olds 5 14 and more year olds 6 15 -19 year olds 7 15 -24 year olds 8 15 -49 year olds 9 All age ranges SOURCE ONU ONU INEGI ONU ONU APPLICATION Yes Yes Not used Yes Yes Yes Note: Two age groups were added because of the particularities of the indicators presented by Mexico.

CODE LIST FOR FREQUENCY CODE DESCRIPTION A Annual SOURCE APPLICATION ONU Yes 2 A Two-year average ONU Yes 3 A Three-year average ONU Not used S Half-yearly, semester ONU Not used Q Quarterly ONU Not used M Monthly ONU Not used Yes ND Not determined INEGI – New Note: Added one class to characterize the statistics that have no clearly defined periodicity: ND: Not determined

SDMX – MDG Consulta del DSD desde el Web Service de INEGI Data. Flows de INEGI (Para saber qué flujos disponibles hay y sus correspondientes DSD se hace primero una consulta dinámica de todos los Data. Flows publicados por INEGI) http: //www. sdmx. snieg. mx/sistemas/sdmx/restsdmx/Dataflow/ALL/ALL (o si ya se sabe el nombre del Data. Flow se puede acotar por ejemplo a DF_MDG) http: //www. sdmx. snieg. mx/sistemas/sdmx/restsdmx/Dataflow/ALL/DF_MDG/ALL El nombre del DSD de MDG se puede ver en el resultado de la consulta anterior. Por ejemplo, para el caso de los MDG el nombre es DSD_MDG. Para consultarlo se construye la URL como sigue: Data. Structure http: //www. sdmx. snieg. mx/sistemas/sdmx/restsdmx/Data. Structure/ALL/DSD_MDG/ALL Data. Structure con artefactos de referencias (codelist, consept scheme) http: //www. sdmx. snieg. mx/sistemas/sdmx/restsdmx/Data. Structure/ALL/DSD_MDG/ALL? reference s=shallow CODELISTS (Ejemplo para consultar dinámicamente el código CL_FREQ, los nombres de los códigos los sabemos en el DSD de la consulta anterior) http: //www. sdmx. snieg. mx/sistemas/sdmx/restsdmx/Codelist/IAEG/CL_FREQ/ALL

SDMX – MDG Consulta de Datos desde el Web Service de INEGI en diferentes DATOS del Flujo ODM INEGI formatos Datos xml (Se pueden visualizar en el Explorer o en cualquier visualizador como XML-Marker) http: //www. sdmx. snieg. mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI? Key=ALL&format=com pact Datos xml filtrados, Ejemplo (Trae exclusivamente el indicador de MEXICO cuya SERIE es SL_EMP_TOTL) KEY = [FREQ]. [REF_AREA]. [SERIES]. [SEX]. [AGE_GROUP]. [LOCATION]. [SOURCE_TYPE]. [UNIT] http: //www. sdmx. snieg. mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI? Key=. MEX. SL_EMP_TO TL. . . &format=compact Datos csv (Archivo separado por comas, reduciendo mas del 50% de su tamaño xml, y se puede visualizar en varias herramientas como Excel, ó llenar directamente una tabla de BD) http: //www. sdmx. snieg. mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI? Key=ALL&alt=csv Datos chart (Trae una imagen, este es el mismo ejemplo anterior de datos filtrados, que trae sólo el indicador de MEXICO cuya SERIE es SL_EMP_TOTL, para Total, Hombres y Mujeres, en una imagen) http: //www. sdmx. snieg. mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI? Key=. MEX. SL_EMP_TOTL. . . &format=co mpact&alt=chart Datos json y jsonp (Para utilizar los datos del flujo en aplicaciones WEB de terceros como Facebook, Twitter, etc. -OPEN DATA-) http: //www. sdmx. snieg. mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI? Key=ALL&format=compact&alt=json&ca llback=jsonp

SDMX – MDG CAMBIOS EN EL DSD • Dynamic chart sourced by a SDMX flow

Proposed Timeline for Conversion to SDMX the MDG indicators Actividades 1. Analysis of the DSD - MDG provided by the UN and its applicability to integrated indicators in the Mexican project A A S S OOOO Area 2 3 4 5 1 2 3 4 responsibl e * * DGAI – DGAII 2. Meeting with conceptual and database staffs to present the structure of DSD (dimensions and attributes) and catalogs, in order to analyze the concepts used in the DSD and to define from where will be taken * DGAI – DGAII 3. Establishing the equivalence between the dimensions, attributes and catalogs of DSD vs Project database * * DGAII 4. Analysis of the database for the transformation process 5. Amendment proposed to the DSD * * * DGAI – DGAII DGAI

Proposed Timeline for Conversion to SDMX the MDG indicators Actividades 6. Generating of a SQLServer scheme database and migrate data from Access, or agreement the access if there is a copy 7. Generating a SQL query to the database to obtain the specific information for the required flow 8. Loading the modified DSD and the query to the Mapping Assistant and make appropriate correlations, saving it with its stream identifier 9. Run the flow and validate the data against the original data file, and make corrections where necessary 10. Plenary meeting with conceptual staff in order to present them the results 11. Developing a plenary meeting with conceptual staff in order to present the results. 12. Sending the flow to the UN for validation A A S SOOOO Area 2 3 4 5 1 2 3 4 responsibl e * * DGAI-DGAII * DGAI * * * * DGAI - DGAII DGAI * DGAI - DGAII * DGIAI

MDG DATA EXCHANGE AMONG NATIONAL AGENCIES AND WITH INTERNATIONAL ORGANIZATIONS The Experience of Mexico Enrique Ordaz 19 - 21 October, 2011 Manila