Open Standard Voting Localization with CAM Localization Mechanisms

Open Standard Voting Localization with CAM - Localization Mechanisms - Publishing Localizations - Leveraging Open Standards - XSD ingesting David RR Webber Chair OASIS CAM TC (Content Assembly Mechanism) E-mail: drrwebber@acm. org http: //wiki. oasis-open. org/cam

Overview n n n Goal is to promote the use of open public standards for voting systems Develop in-country localizations from standard international base set Provide common open public lexicon for elections management and verification Leverage XML and open standards approach, including NIEM* Provide developers with tools to aid delivery, documenting and testing of solutions *NIEM – National Information Exchange Model 2

Approach n Voting Localizations built from OASIS EML § EML has 29 pre-built components (schema) for voting processes n Use desktop visual editor to pick components § Trim and select only what you need from each standard schema § Desktop toolkit makes templates for the XML required n Generate localization details § Automatically build XML components to use § Build your own dictionary, compare to EML standard § Create readable documentation of details n Test, Package and Publish § The CAM desktop toolkit environment supports NIEM IEPD approach n Available as Open Source on Source. Forge. net § Implementation of the public OASIS CAM standard for schema templates (camprocessor) 3

Localization Mechanisms Creating a Country Voting Localization Package

Localization Key Components XML Schema 1 EML Schema XSD aligns voting information model, processes and domain dictionary 2 Documentation 3 XML artifacts 4 Examples & Test Cases Readable information use details (HTML), EML cross-reference Spreadsheet The localized schema and picklist. xml Key componen ts (aka IEPD*) Realistic test XML instances for conformance and interoperability testing *IEPD – Information Exchange Package Documentation 5

Localization Approach Steps EML 4 Voting Exchange 1 develop 2 XML XSD ingest Base CAM Template 3 localize Local XML XSD Generate Revised CAM Template 5 Compare Manage 6 Publish Steps: EML Dictionary Documentation + Examples 1. OASIS develops standard international XSD schema § Implements the voting process, management and information exchange needs 2. Ingest the standard XSD into base CAM template 3. Apply localizations to CAM template § § § Exclude unwanted optional components Align local exchange needs to standard Extend for local needs 4. Generate new localized XSD schema that conform to EML standard 5. Build dictionary cross-reference of localization usage 6. Publish documentation and examples 6

Creating a Package for Localization Step 1 - Ingest EML XSD – rules, structure, annotations n Step 2 - Select & mark out your use model in visual editor - Save your “pick list” selections - Compress template to desired structure n Step 3 - Generate your localized use XSD schema subset n Step 4 - Generate and test rich concrete data examples n (complete with content hints / pass / fail / random options) - Run rules engine - Verify operation and outcomes n Step 5 - Build localization documentation of structure elements, usage, rules and cross-reference map n Step 6 – Package, Publish and Share (IEPD) 7

Challenge: XSD schema has limitations! n The schema contains the superset of voting exchange component variations for every country n XSD syntax does not have direct context mechanisms n Hence people make everything in schema optional n Dependencies are not clear n It is difficult to understand the constructs and to document the rules clearly for election authorities to verify n It is hard to create test cases and instances (the “pick list” tough to visualize) n These are all limitations inherent in W 3 C Schema language itself 8

Completing Localization Picture: § Base EML XSD schema structure model § CAM template + rules / refinements § Documentation of use patterns (aka “pick list” + subset XSD) § Test cases and examples § Content hinting (localization use data) § Validation engine for unit and regression testing § Open standard, open platform and open source allows consistent agreements between participants 9

Localization Package Use Scenario Publish Structure Rules Context Examples Documentation XML Templates Package Test Cases 2 3 1 Validate Report html 4 XML Test Localization Content Hints Testing, Agreement / Alignment. Results Verify Result Outcomes Developer Uses & Creates XML Pass / Fail 10

Tutorial n Base CAM templates – (ingesting XSD schema ) § step by step example n Documenting the Localization Patterns § Creating “pick list” selections § Compressing template structure § Subset XSD generation n Publishing Localization package § Cross-reference spreadsheet § Documentation reporting options § Generate use dictionary n Testing and Conformance § Creating Test Case examples § Content Hinting § Running Test Cases n Advanced Techniques 11

Base CAM Templates (ingesting XSD Schema) Using j. CAM editor Wizard (http: //www. jcam. org. uk)

EML V 6. 0 provides base CAM templates n n Templates come pre-packaged in the EML V 6 download However, for completeness we show to generate one… 13

Ingesting XSD to CAM template format Structure OASIS EML XSD Collection XML Schema Common xsd Components XSLT tools process entire EML XSD collection Rules CIQ xsd INGEST Namespace Localization EML xsd Documentation EML xml CAM = Content Assembly Mechanism 14

Step 1 & 2 – Pick the XSD schema to ingest Choose File / New Template 1 Specify XSD Location root directory 2 Pick XSD First location is the root folder for the XSD collection. Typically this is the same as the location for the XSD schema you want to ingest. (Complex XSD can have this in separate folder; hence need for option) 15

Step 3 – Choose the XSD parent element 3 Choose XSD parent element Confirm and start the XSD ingesting From the dropdown list pick the correct root element you wish to use (EML). XSD’s may have more than one collection in them (as shown here) or may not have the parent node element cleanly defined. The Wizard shows you the list of all possible ones it finds – so you can select and confirm the right one. 16

Step 4 – Ingesting complete – Save Results Completed template is loaded and save dialogue appears 4 4 Processing usually takes a few seconds, some larger EML XSD can take a few minutes to process. Tip: j. CAM runs the ingesting as a background task – so you can continue to use the computer while such long ingesting is proceeding. 17

Step 5 – Review ingested structure Completed template is shown in the structure navigator panel for review and editing 5 Template contains all the default content model and structure rules ingested from the XSD. All annotations and documentation from XSD also ingested (show as “paperclip” symbol). Code lists and typical content values inserted for easy visual reference. 18

Documenting the Localization Patterns “Pick lists”, documentation and XSD subset generation

Building a pick list Make pick list Structure EML structure is all inclusive!!! Rules MARK WHAT IS NOT NEEDED Documentation excluded items DESIRED RESULTS 20

Marking Items for exclude - pick list n Can exclude at all levels within the structure § exclude. Tree() § exclude. Element() § exclude. Attribute() n Use XPath operators to control scope: § § n Specific node Group of nodes Anywhere occurs in structure Contextually based on condition Can add new domain elements with own namespace to EML template 21

Using Editor to mark Exclude items Structure Editor Tools Select focus on item Invoke action menu (right mouse click) Pick action Tip: exclude rule display is context sensitive and only available on optional items Tip: use “Add New Rule” mode to specify different XPath for exclude (quick mode assumes “current path”) 22

Export and Save completed pick list File Menu Option Select Export / Pick List and then specify filename of destination. Excluded items are designated with red “dot” in structure editor and italics font with no bold highlight 23

Pick List Details Pick lists provide a handy way to catalogue the localization model and can be re-used later by importing into other templates EXCLUDE FLAG VALUE Exported Example 24

Compress Operation File Menu Option Select option and specify filename for new copy of your template. Compress process removes all rules and structure items marked with an exclude statement. Note: ignores excludes that have a conditional context expression (advanced section technique). Note: You can retain both original and new compressed templates 25

Extending the base EML schema In addition to excluding existing optional components you may need to add new pieces n New pieces should be assigned their own extension namespace prefix – to identify them from the base EML components n Typically the OASIS EML committee will review new pieces as candidates for adding to future versions of the standard n • Send comments and suggestions to election-comment@lists. oasis-open. org 26

Adding New Domain Elements 1 2 Select root element node in structure; right mouse click for context menu; select “Add Namespace”; then enter prefix and URI Select element node in structure; right mouse click for context menu; select “Add Child Attribute / Element”; then enter prefix and name 27

Generating your EML use schema Once your localization details are completed you are ready to complete the remainder of the process n This starts with generating your own new subset schema n Then once schema details are OK – creating test cases, cross-reference spreadsheet and documentation n 28

Generating subset use schema File Menu Option Select Export CAM as XSD menu option Confirm the location and filename, and namespace mode. 1 2 Select ‘default’ for Mode to write a regular import set of XSD schema for each namespace 29

Schema subset generated Set of XSD files with filename and namespace suffix Each namespace file is import for those specific type definitions Reviewing XSD results in a schema visual editor tool 30

Information Restriction Considerations n The CAM template allows full use of XPath conditional expressions and a rich set of over 30 functions including: § set. Number. Range(), set. Length(), set. Limit(), set. Date. Mask(), set. Number. Mask(), restrict. Values(), lookup() § Those that are compatible with XSD constraints will cause constraint schema assertions to be written out when exporting to schema n In the advanced topics section we will look at cross field validations using XPath conditional rules 31

Generating Testing and Conformance Examples Generating base examples Customizing content with Hints Selecting valid and invalid modes Running rules validation check

Test Case Generation Quick Start File Menu Option Default directory to write examples into Name to be used for the examples How many examples to create Repeating elements count for Quick Test – just click “OK” to use default settings 33

Test Case Results Active links to view the generated examples 34

Advanced Generation Options Optional schema file validation link; use this to have example validate with EML schema or local subset schema Use content type or item name (name is useful for checking backend transform processing) How to handle optional items: all | random | none If you want deliberate errors for fail testing; (will give variety of data and structure errors) Use namespaces or not; if ‘false’ is selected – then XML instances are created with minimized namespace usage. Use slider to pick a specific seed value – or leave blank for random seed Optional content hints (explained next ) 35

Test Case Generator Feature Summary n Make both Pass / Fail testing examples n Content hinting so examples use real not fake data n Test optional item logic with: all / random / none n Uses exclude() assertions so does not include those items – makes realistic examples of your use pattern n Can pass in seed value – use when adding and testing hints (each test case is labelled with its seed value) n Make hundreds of test cases without manual editing n Can link test case to XSD schema for structure tests n You can modify XSLT tool for own testing needs 36

Run CAM Rules Check on Examples Run Menu Option 1 Pick Test Case Example to VALIDATE; click Finish to run validation rules 2 Review validation results 3 37

Content Hinting Mechanisms n n n Designed to create realistic data examples Hints can be provided in two ways Firstly - using ‘Value’ notes in annotations on specific items in the structure editor Second – create your own Hints XML file and add matching rules to globally apply across your template(s) – e. g. First. Name, Last. Name, Address, Birth. Date, etc. Can export from one template, import into another 38

First Approach: annotation Value Hints Select focus on structure item Invoke action menu (right mouse click) Choose “Edit Annotations” 39

Then add Value annotation item 3 Select “Value” tab 2 4 Enter values terminated with “|” character 1 Click on “Add New”, then enter “Value” as Type and confirm “OK” 5 Re-run Example export to see results 40

Second: Hints File Mechanism (XML file) 1 like / with partial name matching use for component match on items – e. g. first with name matches <nxnl: first_name> 2 key matching on tag name use for exact match of items 3 key / parent path matching use when same name occurs within different parents – e. g. Country and Person / Name. Element with different content and context Note: matching is case sensitive but ignores namespaces TIP: can use Export Hints to create initial XML file for editing 41

A- Using Examples Generator with Hints Select XML hints file to be used here 42

B- Import Hints into Annotations (merge) set and select as needed Option to autoreload new anno file into current template 43

Documentation Cross-Reference Spreadsheet Default reporting options

Generating Cross-Reference Spreadsheet n Use Compare to Dictionary tool § Compare to default EML-V 6 -dictionary Drag and Drop XML file into Microsoft Excel n Review and save spreadsheet workbook n 45

Dictionary Compare Tool Select compare tool menu option Browse and choose EML dictionary 46

Drag and Drop to Excel Spreadsheet Click OK 47

Documentation Layouts n Five options § § § Source XML Component details (XML) Tabular format (HTML) Interactive web page (wiki) Code list Tabular format page layout for data analyst use and designed to make rules and use patterns clear n Each documentation layout XSLT script can be customized as desired n 48

Tabular Documentation Menu Option Select Format, resulting HTML opens in browser viewer 49

Tabular HTML Content Details Clear use pattern Extended Code list handling Annotations XPath references and functions Enhanced Data type Logic 50

Summary Ingesting XSD n Creating use pattern (aka pick list) n Generating test examples n Hints system n Generate XSD schema subset n Running tests n

Localization Checklist Review XML Schema 1 EML Schema XSD aligns voting information model, processes and domain dictionary 2 Documentation 3 XML artifacts 4 Examples & Test Cases Readable information use details (HTML), EML cross-reference Spreadsheet The localized schema and picklist. xml Realistic test XML instances for conformance and interoperability testing 52

Summary n Capabilities covered § § § § Ingesting XSD to Template Creating use pattern (aka pick list) Generating test examples Hints system Generate local use XSD schema subset Running tests Generating documentation and spreadsheets Allows rapid localization development n Enables consistent implementations n Documentation provides clear details on structures, content, rules and definitions n Cross-reference to EML standard n 53

Advanced Techniques Extending rules for actual use cases n Using XPath selector wizard n Handling ingestion recursion issues n Source. Forge XSLT repository n

Importing pick list operation File Menu Option Select Import and specify filename of your existing pick list xml. Import process matches the path expressions in your pick list to the XPath expressions in the template. Exclude statements generated for matching items. Makes it easy to re-apply a pick list on new versions of schemas, or on similar schemas with same blocks of content – address, company, person, etc. 55

Illustrative EML requirements n When Ballot type > 0 - AND Language code > 1 Resource Matching Region Restriction n Require Ballot State = CA, NV, NM n When Registered Voters Total < 1000 - AND Ballots > 1000 Audit alerting 56

Template rules syntax <as: Business. Use. Context> <as: Rules> <as: default> <as: context> <as: constraint condition="//Cast. Ballots /Total > ‘ 1000’ and( // Registered. Voters /Total < ‘ 1000’)" action="restrict. Values(// Cast. Ballots/Total, ‘Alert – possible audit required’) "/> <as: constraint action="restrict. Values(//eml: Administrative. Area /eml: Name. Element , 'CA|NV|NM')"/> </as: context> </as: default> </as: Rules> </as: Business. Use. Context> 57

XPath selector wizard XPath syntax is automatically generated for you Check boxes allow user to select scope of rule action Conditional context rules can be optionally used Documentation notes can be added here 58

“CAM Kit” of XSLT tools used n XSD 2 CAM § extracts structure and rules and builds template from schema n XML 2 pick list § builds pick list from any XML instance (uses Level Depth setting) n CAM 2 Examples § Generates a collection of XML instance test cases n Import/Export Hints § Manage and apply content hinting across CAM templates n CAM 2 XSD export § Creates a subset XSD from CAM template 59

Source. Forge XSLT svn repository n Using any svn client the XSLT scripts can be retrieved from: § https: //camprocessor. svn. sourceforge. net/svnroot/camprocessor/c amed/uk. org. jcam. camed/trunk/xsl/ 60

Resources Selection of useful links and additional technical details

Simple EML 440 Example using the EML 440 cast vote schema n Simplified to minimum requirements n Example can be downloaded here: n § http: //markmail. org/message/qyrpfq 75 mtctblyq 62

Resources: www. oasis-open. org/committees/election docs. oasis-open. org/election www. oasis-open. org/committees/cam wiki. oasis-open. org/cam docs. oasis-open. org/cam www. jcam. org. uk 63