Starlink VOTable software Open source Java software for

Starlink VOTable software Open source Java software for table manipulation Author: Mark Taylor (m. b. taylor@bristol. ac. uk)

Software Features ● Java – ● Open source – ● Licenced under the GPL Full documentation – ● J 2 SE 1. 4/1. 5, portable, easy deployment Tutorial/reference user documents, javadocs Scalable – Designed for use with large tables, not memory-limited – Switch between memory & disk caching of bulk data using pluggable Storage Policies

STIL Starlink Tables Infrastructure Library ● ● ● Generic table I/O & processing library Presents abstract view of table to programmer Format-neutral I/O – VOTable, FITS, SQL, ASCII, CSV. . .

VOTable FITS SQL table SQL query ASCII HTML CSV La. Te. X . . . Mirage. . . Input handlers random-access sequential Table Processing Output handlers

STIL VOTable features ● ● Fully implements VOTable standard – Understands TABLEDATA, BINARY, FITS (inline or by URL) – VOTable 1. 0 or 1. 1 versions Flexible options for writing VOTables – ● ● Can write TABLEDATA, BINARY, FITS (inline or by URL) Full access to structure of VOTable document – Table-aware SAX (streamed) parsing – Table-aware DOM (memory-based) processing Best-efforts parsing of non-conforming documents – Processing is mainly FIELD and DATA elements – Non-catastrophic schema changes easy to cope with

Table-aware DOM ● ● ● Parser builds normal org. w 3 c. dom document in memory Only small (non-DATA) elements are included For table data, use ((Table. Element)el). get. Data() ● ● This gives you a Star. Table which provides table elements Don't need to worry wheter it's TABLEDATA, BINARY or FITS <VOTABLE> <RESOURCE type=”data”> <INFO name=”author” value=”Mark Taylor”/> <TABLE> <FIELD datatype=”int” name=”index”/> <FIELD datatype=”double” name=”RA”/> <FIELD datatype=”double” name=”Dec”/> <DATA> <TABLEDATA> <BINARY> <TR> <STREAM encoding=”base 64”> <TD>1</TD> AAAD 0 No. YXJs. ZXMg. TWVzc 2 llc <TD>202. 43</TD> g. AAAAt. NYXJr. IFRhe. Wxvcg== <TD>47. 22</TD> </STREAM> </TR> </BINARY> <TR> <TD>2</TD> <TD>168. 63</TD> <TD>55. 03</TD> </TR> <TR>. . . </TR> </TABLEDATA> </TABLE> </RESOURCE> <RESOURCE type=”meta”> <COOSYS equinox=”J 2000. 0” epoch=”J 2000. 0” system=”eq_FK 5”/>. . . </RESOURCE> </VOTABLE>

Table-aware SAX ● ● Extends SAX (streaming) XML parser with VOTableaware callback methods Parsing code receives notification for start, end and each row of TABLE data as well as XML elements, entities, etc Standard SAX handler public interface Content. Handler { void start. Document(); void end. Document(); void start. Element( name, atts. . . ); void end. Element( name, . . . ); void characters( char-data, . . . ); } STIL extension for VOTables public interface Table. Handler { void start. Table( Star. Table metadata ); void row. Data( Object[] row ); void end. Table(); }

Axis VOTable de/serialization ● ● STIL provides custom serializer/deserializers for use with Axis web services JAX-RPC calls send Star. Table objects over the wire as VOTable documents In principle, streaming means this works with large tables (but there are problems at Axis 1. 1)

TOPCAT Tool for OPerations on Catalogues And Tables ● ● View/analyse/edit/write tabular data – Manipulate table data and metadata – Plot, match, sort, calculate, select, . . . – Powerful and extensible expression language Based on STIL – Multi-format, including VOTable (all encodings) & SQL – Large tables (10^6 rows x 10^2 columns)

STILTS STIL Tool Set ● ● Command-line tools for table manipulation Generic table tools: – – ● VOTable structure-sensitive tools – – ● tcopy: format conversion tpipe: table pipeline processing votcopy: DATA encoding conversion votlint: VOTable document validation Current release is beta

votcopy ● ● Converts VOTable DATA encodings Can write all defined encodings: – ● TABLEDATA, BINARY, FITS Choose STREAM element output: – inline (encoding='base 64') – external (href='<local-file>') <VOTABLE> <RESOURCE type=”data”> <INFO name=”author” value=”Mark Taylor”/> <TABLE> <FIELD datatype=”int” name=”index”/> <FIELD datatype=”double” name=”RA”/> <FIELD datatype=”double” name=”Dec”/> <DATA> <TABLEDATA> <TR> <TD>1</TD> <TD>202. 43</TD> <TD>47. 22</TD> </TR> <TD>2</TD> <TD>168. 63</TD> <TD>55. 03</TD> </TR> <TR>. . . </TR> </TABLEDATA> </TABLE> </RESOURCE> <RESOURCE type=”meta”> <COOSYS equinox=”J 2000. 0” epoch=”J 2000. 0” system=”eq_FK 5”/>. . . </RESOURCE> </VOTABLE> ● All non-data elements are untouched ● Useful for: – humans looking at STREAM data – conversion to STREAM prior to transmission – . . .

votlint ● ● ● Many VOTables are in violation of the standard – Makes it hard for parsing software – Data might get garbled DTD/Schema validation misses many errors: – Content of TD/STREAM elements can't be checked against FIELDs – arraysize attribute values may be meaningless – ref/ID crossreferences might make no sense – etc, etc. . . votlint parses a VOTable document and warns of trouble – ● ● Doesn't pick up everything, but it's a lot better than not using it Useful for VOTable authors and parser developers If you are producing VOTables or VOTable-writing software please use it to check your VOTables are legal!

votlint example Running votlint on this valid document: <VOTABLE xmlns: xsi=”http: //www. w 3. org/2001/XMLSchema-instance“ xsi: schema. Location=”http: //www. ivoa. net/xml/VOTable/v 1. 1” xmlns=”http: //www. ivoa. net/xml/VOTable/v 1. 1” version=” 1. 1”> <RESOURCE> <TABLE nrows=” 2”> <DESCRIPTION>A very small table</DESCRIPTION> <FIELD name=”Identifier” datatype=”char”/> <FIELD name=”RA” datatype=”double”/> <FIELD name=”Dec” datatye=”double”/> <DATA> <TABLEDATA> <TR> <TD>Fomalhaut</TD> <TD>344. 48</TD> <TD>-29. 618 d 0</TD> </TR> </TABLEDATA> </TABLE> </RESOURCE> </VOTABLE> gives these warnings: INFO (l. 4): No arraysize for character, FIELD implies single character WARNING (l. 15): Characters after first in char scalar ignored (missing arraysize? ) ERROR(l. 17): Bad double string -29. 618 d 0 WARNING (l. 19): Wrong number of TDs in row (expecting 3 found 4) ERROR (l. 22): Row count (1) not equal to nrows attribute (2)

Summary ● ● STIL http: //www. starlink. ac. uk/stil/ – Table-aware DOM – Table-aware SAX STILTS http: /www. starlink. ac. uk/stilts/ – – ● ● votcopy votlint Other features available – TOPCAT, generic parts of STIL & STILTS – Applications session tomorrow, or talk to me The future is uncertain. . .