Automating electronic data sheets CCSDS Spring 2013 Sam

Automating electronic data sheets CCSDS Spring 2013 Sam Price

Navigator Background • Missions – GPM – MMS – High Altitude 12 Re – Hubble servicing mission • Reflected GPS signals • Embedded orbit propagator – Geons – 3 rd Party software – 3 rd party commands/telemetry • 45 commands • 43 telemetry messages Background C++ NASA GSFC – CCSDS Spring 2013 XML CTypes Summary 2 / 23

Navigator Electronic Documents for Command / Telemetry Document Count Document 1, 2 MMS, GPM ICD MSWord 3, 4 GPM, MMS Telemetry mockup (Ada) 5, 6 MMS, GPM Ground system (asist) 7 C++ Headers 8 3 rd Party C++ Orbit Prop Yes 3 rd party 9 3 rd party ICD Orbit Prop Yes 3 rd party 10, 11 MMS, GPM Bandwidth Rates (excel) (5) Bandwidth person 12 C++ Command Verification (1) Device - Algorithms 13 CSV Logging (1) Device - Logging 14 Analysis (Matlab) (1) Device - Analysis 15 Message Size Descriptions Member Layout Meta data units, polys, limits TLM Rate Per Mission Responsible Person 5+ people (1) Device Maybe? ? ? (2) Mockup (3) Ground System owner Yes Doxygen Endianess converters for NASA GSFC CCSDS Spring 2013 CSV –Logging Yes Doxygen Yes 3 rd party (1) Device (4) 3 rd Party C/C++ Yes 3 rd party (4) 3 rd Party ICD (1) Device - Logging 3 / 23

Multiple documents are expensive $$$$ • 15 different electronic versions, – manually created • Same data across all of the documents • 5+ different people maintaining separate documents. • Changes propagate slowly throughout documents ($$$) • Errors occur from manual changes. ($$$$) • Naming conventions are not the same Background C++ NASA GSFC – CCSDS Spring 2013 XML CTypes Summary 4 / 23

3 rd party C Headers A streamlined approach Some tools C++ headers Telemetry/ Commands • C/C++ – Endianess conversion routines – Command validation – Header files • Human readable Interface Control Document – Latex – HTML – Maybe MS Word … XML Database Why C/C++ as starting point – Widespread existence • All of my Telemetry is in C • More people know C – – Machine readable Exactly what is implemented 3 rd party libraries are in C Many more reasons Background C++ NASA GSFC – CCSDS Spring 2013 • • Ground system definitions Bandwidth spreadsheet Ada mockup of telemetry for testing. Logging tools – CSV Logging – Matlab import – Database logging XML CTypes Summary 5 / 23

Workflow 3 rd party C Headers Convert C/C++ to XML Latex ICD xml_to_latex. py Asist C++ headers Telemetry/ Commands xml_to_asist. py Asist RDLS XML Database Python xml_to_ctypes. py Python CTypes Background C++ NASA GSFC – CCSDS Spring 2013 XML CTypes Summary 6 / 23

What does it take to mimic C/C++ in XML • Basic Types – bools – integers -- Use stdint. h • int 8_t, int 16_t, int 32_t, int 64_t – unsigned Integers • uint 8_t, uint 16_t, uint 32_t, uint 64_t – floating point memory • float, double, long double – strings • char arrays – Arrays • Multidimensional – void pointer, pointers (16, 32, 64) sizes – Structures/classes – unions – enums Background C++ NASA GSFC – CCSDS Spring 2013 • No child / parents / inheritance • No templates • C/C++ can have alignment/padding problems • Metadata stored in doxygen style comments – Units, Polynomials, warning ranges, etc • Bit fields defined in comments • No C bitfield yet… XML CTypes Summary 7 / 23

Use a compiler for parsing • Clang/llvm – C/C++ Compiler similar to GCC – Cross platform compatible(Windows, Linux, Mac) • Lib. Clang – Pass arguments to change C/C++ structure padding. • -m 32 , -m 64 – Cross platform compatible (Linux, Mac, Windows) – Apple uses it in their developer tools for code completion, syntax highlighting, etc… – Use it to find comments, structures, types, padding in structure, etc. • http: //clang. llvm. org/docs/Tooling. html • http: //clang. llvm. org/get_started. html • https: //github. com/trolldbois/clang <- padding/size info Background C++ NASA GSFC – CCSDS Spring 2013 XML CTypes Summary 8 / 23

C Example CCSDS Header /*! * @brief CCSDS Telemetry Header * @Telemetry. Packet. Header * */ typedef struct CCSDS_Telemetry_Header { /*! * @brief Destination address */ char p_sp_dest; /*! * @brief Protocol ID * @default 0 xf 2 */ char p_sp_prot; /*! *@bitarray msbf * @bits 3 * @name packet_version_number * @brief Pacet Version Number * @default 0 * @bits 1 * @name packet_type * @brief Packet type * @enum ENUM_PACKET_TYPE * @bits 1 * @name secondary_header_present * @brief Secondary Header flag * @detailed The Secondary Header Flag shall indicate the presence or absence of the Packet Secondary Header within this Space Packet. * It shall be 1 if a Packet Secondary Header is present; it shall be 0 if a Packet Secondary Header is not present. * @enum CCSDS_Secondary_Header * @bits 11 * @name apid * @brief Application Process Identifier (APID) * @detailed via 135. 0 -b-1 application ID's 2040 - 2047 are reserved and should not be used. * The APID (possibly in conjunction with the optional APID Qualifier that * identifies the naming domain for the APID) shall provide the naming mechanism for the LDP. */ uint 16_t p_ident; /*!. . */ uint 16_t p_seq_cont; /*!. . */ uint 16_t p_data_len; /*!. . */ uint 64_t s_time; } CCSDS_Telemetry_Header; Doxygen comments start with @marker_name marker value Indented comments belong to parent comment NASA GSFC – CCSDS Spring 2013 Could have used C bitfields (See backup slides) 9 / 23

C/C++ -> clang_to_json. py -> json_to_xml. py -> XML • Current status – 500 lines of code to go from C Header to XML – Need to add enums, meta data, bit fields. – Implemented structures, unions, bit structures, arrays, padding bytes, integers, unsigned integers, floating point numbers, pointers, comment parsing. • https: //github. com/nasa/SOIS-CCSDS-XML – Non. Nasa Contact for access, currently private project until XML format is well defined. – NASA: fill out a NAMS/IDMAX request Background C++ NASA GSFC – CCSDS Spring 2013 XML CTypes Summary 10 / 23

C Example p_ident /*! *@bitarray msbf * @bits 3 * @name packet_version_number * @brief Pacet Version Number * @default 0 * @bits 1 * @name packet_type * @brief Packet type * @bits 1 * @name secondary_header_present * @brief Secondary Header flag * @detailed The Secondary Header Flag shall indicate the presence or absence of the Packet Second * It shall be 1 if a Packet Secondary Header is present; it shall be 0 if a Packet Secondary Header is no * @bits 11 * @name apid * @brief Application Process Identifier (APID) * @detailed via 135. 0 -b-1 application ID's 2040 - 2047 are reserved and should not be used. * The APID (possibly in conjunction with the optional APID Qualifier that * identifies the naming domain for the APID) shall provide the naming mechanism for the LDP. */ uint 16_t p_ident; NASA GSFC – CCSDS Spring 2013 11 / 23

Clang to JSON Output for p_ident > python clang_to_json. py. . /examples/src_cpp/ccsds. h -m 32 –I. . /examples/src_cpp …. . {'array': [], 'basetype': 'uint 16_t', 'basetype_name': None, 'bitarray': True, 'data': <clang. cindex. Cursor object at 0 x 10 a 7969 e 0>, 'is_pointer': False, 'kind': Cursor. Kind. FIELD_DECL, 'name': 'p_ident', 'offset': 16, 'size': 2 L, 'comments': [{'bitarray': 'msbf', 'children': [{'bits': '3', 'children': ['name': 'packet_version_number'}, {'brief': 'Pacet Version Number’}, {'default': '0'} ] }, ], …. More Member Comments …. . ] NASA GSFC – CCSDS Spring 2013 'members': [{'array': [], 'basetype_name': None, 'bitarray': False, 'bits': 3, 'comments': [{'name': 'packet_version_number'}, {'brief': 'Pacet Version Number’}, {'default': '0'}], 'data': None, 'is_pointer': False, 'kind': None, 'members': [], 'name': 'packet_version_number', 'size': -1}, … More Members. . ], }, • Offset, Sizes include padding information. • Clang object used for further processing if needed. • Bit arrays become children for bitstructs. 12 / 23

XML Layout • Type database – Basic types • Integers, floats – Structures – Unions – Bit structures • Telemetry database – Similar to structures • Command database – Similar to structures Background C++ NASA GSFC – CCSDS Spring 2013 <Type. Database> <type. Int/> <type. Float/> <type. Bit. Struct> <Members> <variable/> </Members> <type. Bit. Struct> <type. Struct> <Members> <variable/> </Members> <type. Struct/> </Type. Database> <Telemetry> </Telemetry> <Commands> </Commands> XML CTypes Summary 13 / 23

C Example /*! * @brief CCSDS Telemetry Header * @Telemetry. Packet. Header * */ typedef struct CCSDS_Telemetry_Header { /*! * @brief Destination address */ char p_sp_dest; /*! * @brief Protocol ID * @default 0 xf 2 */ char p_sp_prot; /*! *@bitarray msbf * @bits 3 * @name packet_version_number * @brief Pacet Version Number * @default 0 * @bits 1 * @name packet_type * @brief Packet type * @enum ENUM_PACKET_TYPE * @bits 1 * @name secondary_header_present * @brief Secondary Header flag * @detailed The Secondary Header Flag shall indicate the presence or absence of the Packet Secondary Header within this Space Packet. * It shall be 1 if a Packet Secondary Header is present; it shall be 0 if a Packet Secondary Header is not present. * @enum CCSDS_Secondary_Header * @bits 11 * @name apid * @brief Application Process Identifier (APID) * @detailed via 135. 0 -b-1 application ID's 2040 - 2047 are reserved and should not be used. * The APID (possibly in conjunction with the optional APID Qualifier that * identifies the naming domain for the APID) shall provide the naming mechanism for the LDP. */ uint 16_t p_ident; /*!. . */ uint 16_t p_seq_cont; /*!. . */ uint 16_t p_data_len; /*!. . */ uint 64_t s_time; } CCSDS_Telemetry_Header; NASA GSFC – CCSDS Spring 2013 14 / 23

Generated XML <Type. Database> <type. Int bits="3" endianess="big. Endian" name="int 3_be" signed="true"/> …. More small types for int 1, int 11, int 8, int 16. . etc …. <type. Bit. Struct name="CCSDS_Telemetry_Header_p_ident" type="uint 16_be"> <Members> <variable name="packet_version_number" type="int 3_be"/> <variable name="packet_type" type="int 1_be"/> <variable name="secondary_header_present" type="int 1_be"/> <variable name="apid" type="int 11_be"/> </Members> </type. Bit. Struct> …. More type. Bit. Struct for p_seq_cont, s_time …. <type. Struct name="CCSDS_Telemetry_Header"> <Members> <variable bits="8" name="p_sp_dest" offset="0" type="int 8_be"/> <variable bits="8" name="p_sp_prot" offset="8" type="int 8_be"/> <variable bits="16" name="p_ident" offset="16" type="CCSDS_Telemetry_Header_p_ident"/> <variable bits="16" name="p_seq_cont" offset="32" type="CCSDS_Telemetry_Header_p_seq_cont"/> <variable bits="16" name="p_data_len" offset="48" type="uint 16_be"/> <variable bits="64" name="s_time" offset="64" type="CCSDS_Telemetry_Header_s_time"/> </Members> </type. Struct> </Type. Database> >NASA python json_to_xml. py. . /examples/src_cpp/messages. h -m 32 –I. . /examples/src_cpp 15 / 23 GSFC – CCSDS Spring 2013

Writing XML with python from xml. etree. Element. Tree import Element. Tree parent_node = Element('EDS') Messages = Element('Telemetry') parent_node. append(Messages) #All attributes/text must text Messages. attrib['Count'] = str(5) Messages. text = “Hello” document = Element. Tree(parent_node) #See json_to_xml for prettify function print prettify(document. getroot()) NASA GSFC – CCSDS Spring 2013 16 / 23

XML Output <? xml version="1. 0" ? > <EDS> <Telemetry Count="5"> Hello </Telemetry> </EDS> Or Root = document. getroot() print xml. etree. Element. Tree. tostring(Root, 'utf-8') '<EDS><Telemetry Count="5">Hello</Telemetry></EDS>' Background C++ NASA GSFC – CCSDS Spring 2013 XML CTypes Summary 17 / 23

• XML to Python CTypes • Decode telemetry/send commands 3 rd party C Headers clang_to _json. py json_to_xml. py xml_to_ctypes. py C++ headers Telemetry/ Commands Background C++ NASA GSFC – CCSDS Spring 2013 Python XML Python CTypes XML CTypes Summary 18 / 23

Reading XML with python import xml. etree. Element. Tree as ET doc = ET. parse(filename) root = doc. getroot() for node in root. findall('Type. Database/type. Struct'): print node. attrib['name’] print node. text See xml_to_ctypes. py for complete listing 200 lines of code to convert from XML to python ctypes Or http: //docs. python. org/2/library/xml. etree. elementtree. html Background C++ NASA GSFC – CCSDS Spring 2013 XML CTypes Summary 19 / 23

Python ctypes example Ø python xml_to_ctypes. py test. xml > test_ctypes. py Ø 205 lines of code import ctypes class CCSDS_Telemetry_Header_p_ident ( ctypes. Little. Endian. Structure ): """""" #_packed_ _fields_ = [ ("apid" ("secondary_header_present” ("packet_type" ("packet_version_number" , ctypes. c_uint 16 , 11 ), , ctypes. c_uint 16 , 3)] class CCSDS_Telemetry_Header ( ctypes. Little. Endian. Structure ): ""”This is a comment about the CCSDS Packet Header""" _pack_ = 1 _fields_ = [ ("p_sp_dest" , ctypes. c_int 8 ("p_sp_prot" , ctypes. c_int 8 ("p_ident" , CCSDS_Telemetry_Header_p_ident ("p_seq_cont" , CCSDS_Telemetry_Header_p_seq_cont ("p_data_len” , ctypes. c_int 16 ("s_time" , CCSDS_Telemetry_Header_s_time )] NASA GSFC – CCSDS Spring 2013 ), ), ), 20 / 23

Using python ctypes to decode / encode data import test_ctypes as tc fid = open('log. raw’) #Buffer big enough to hold any command / telemetry data big_buffer = bytearray(1024*10) write_buffer = memoryview(big_buffer) #header data directly into the buffer. 8 bytes. fid. readinto(write_buffer[0: 8]) #big_buffer[0: 8] #bytearray(b'x 00xf 2 Enx 00xc 0x 17x 00') header = tc. CCSDS_Telemetry_Header. from_buffer(big_buffer) #Read in the rest of the CCSDS packet. fid. readinto(write_buffer[8: (header. p_data_len+1)]) print "Apid is %d decode it. " % header. p_ident. apid #changing ctypes changes the binary data. Good for commanding header. p_sp_dest = 5 #big_buffer[0: 8] #bytearray(b'x 05xf 2 Enx 00xc 0x 17x 00') fid. close() NASA GSFC – CCSDS Spring 2013 21 / 23

ctypes cont… class CCSDS_Telemetry_Header ( ctypes. Little. Endian. Structure ): "””Documentation about CCSDS Telemetry Header""" _pack_ = 1 _fields_ = [ ("p_sp_dest" , ctypes. c_int 8 ), ("p_sp_prot" , ctypes. c_int 8 ), ("p_ident" , CCSDS_Telemetry_Header_p_ident ), ("p_seq_cont" , CCSDS_Telemetry_Header_p_seq_cont ), ("_p_data_len" , ctypes. c_int 16 ), ("s_time" , CCSDS_Telemetry_Header_s_time )] @property def p_data_len(self): CCSDS data length is 1 minus actual length “””More Documentation””” return self. _p_data_len + 1 of the message. @p_data_len. setter def p_data_len(self, new_len): Provide documentation via docstring on self. _p_data_len = new_len – 1 telemetry value. h = CCSDS_Telemetry_Header h. p_data_len = 8 print h. _p_data_len 7 print h. p_data_len 8 NASA GSFC – CCSDS Spring 2013 Telemetry: Polynomials, conversions, etc Commanding: limit values, throw errors if not within ranges. 22 / 23

Lessoned learned • Provide all of the standard C/C++ types – As listed on “mimic C/C++ in XML” slide • Standardized naming of Integers, floats, bools, pointers. (int 32, int 16, int 8, …) – Allows for creating dictionary mappings between XML and desired language • Mimic the structure of existing languages. – Makes mapping from XML to existing language more strait forward • Keep it simple • Develop tools with XML standard – Kept having to change scheme while making tools to make processing easier Background C++ NASA GSFC – CCSDS Spring 2013 XML CTypes Summary 23 / 23

Backup slides NASA GSFC – CCSDS Spring 2013 24 / 23

C bit field example (Not supporting, yet …) struct CCSDS_Telemetry_header_p_ident { }; Least significant field first uint 16_t apid: 11; uint 16_t secondary_header_present: 1; uint 16_t packet_type: 1; uint 16_t packet_version_number: 1; http: //msdn. microsoft. com/enus/library/ewwyfdbe(v=vs. 71). aspx NASA GSFC – CCSDS Spring 2013 25 / 23

C Example cont /*! * @ingroup MESSAGES * @brief An example message for Power point. * @apid 135 */ typedef struct Basic_M{ CCSDS_Telemetry_Header packet_header; //CCSDS Packet Header /*! * @brief Temp of the baseplate the cpu is attached to. * @polynomial [1566. 7676, -18. 85822, -. 17372, 0. 00229, 0. 00001] * @unit fahrenheit * @mode nominal * @red_limit [-Inf, -20), (120, Inf] * @yellow_limit [-20, -15), (115, 120] Doxygen comments start with */ @marker_name marker value int 32_t cpu_temp; /*! Indented comments belong to * @brief Receiver Position parent comment * @coordinate_frame ECEF * @unit meter */ double position[3]; }; NASA GSFC – CCSDS Spring 2013 26 / 23

Example basic example <Telemetry> <Message name="Basic_M"> <Members> <variable bits="128" name="packet_header" offset="0" type="CCSDS_Telemetry_Header"/> <variable bits="32" name="cpu_temp" offset="128" type="int 32_be"/> <variable bits="192" name="position" offset="192" type="float 64_be"> <array> <size> 3 </size> </array> </variable> </Members> </Message> </Telemetry> NASA GSFC – CCSDS Spring 2013 27 / 23