Cathode Strip Chamber CSC Raw Data Unpacking and
Cathode Strip Chamber (CSC) Raw Data Unpacking and Packing using bit field data classes n Alex Tumanov, Rick Wilkinson n September 2007
CMS Muon Endcap n n 468 Cathode Strip Chambers send data to Front End Drives (FEDs) 8 FEDs with the capacity of 200 MByte/s each A typical CSC event size is 10 -100 Kb The L 1 event rate is therefore 10 -100 KHz Alex Tumanov Rice University September 2007 2
CMS HLT trigger basics n n Every 25 ns 20 pp collisions occur 800 million pp collisions per second L 1 event rate of 100 KHz The CMS HLT trigger which is software based and its output is 100 Hz Alex Tumanov Rice University September 2007 3
CSC DAQ schematic Global Event HLT Farm Builder 100 KHz (runs unpacking) CSCs with on-chamber electronics CF FEB EBCFEB CF C A EB T LCTA EB C F L LC A C CT T L A n FED CSC Raw data unpacking is a part of the HLT: è è n 100 Hz Runs online Must be very fast in order to handle 100 KHz->100 Hz CSC unpacking is also used in offline Reconstruction and local “spy” data acquisition Alex Tumanov Rice University September 2007 4
CSC Event Data Format 8 DCC slinks DCC Data 36 DDUs 468 CSC chambers Chamber Data DDU Data Chamber Data Event #1 DCC Data DDU Data Contains data from different boards corresponding to each chamber (ALCT, DMB, CFEB, TMB) Chamber Data DDU Data . . . … … Alex Tumanov . . . … … Rice University . . . … … September 2007 5
Unpacking classes CSCDCCEvent. Data DCC bytestream CSCStatus. Digi up to 10 DDUs CSCDDUEvent. Data CSCEvent. Data up to 15 CSCs CSCEvent. Data CSCCFEBData CSCCFEBData CSCDMBHeader CSCALCTHeader CSCAnode. Data CSCTMBHeader CSCWire. Digi CSCALCTDigi CSCCLCTData CSCTMBTrailer CSCTMBScope CSCRPCData CSCCLCTDigi CSCStrip. Digi CSCCorrelated. LCTDigi CSCComparator. Digi Alex Tumanov CSCDMBTrailer Rice University CSCRPCDigi September 2007 6
Taking a closer look at one class CSCDCCEvent. Data DCC bytestream CSCDDUEvent. Data standard. CSCDDUEvent. Data approach for up to 10 DDUs CSCStatus. Digi B Id M CSCStatus. Digi D CSCStatus. Digi er The umb N L 1 A extracting meaningful CSCEvent. Data up to 15 CSCs variables out CSCEvent. Data of. CSCEvent. Data sea of CSC Id CSCDMBTrailer binary data are bitshifts CSCCFEBData and masks CSCCFEBData CSCDMBHeader CSCALCTHeader CSCAnode. Data CSCTMBData CSCCFEBData n This is what was used in … … … CSCTMBHeader CSCTMBTrailer L 1 A number early versions of the CSC //intthe L 1 A = (the. Data[i] >> 12) & 0 x. F; CSCTMBScope CSCWire. Digi CSCCLCTData unpacking code: // CSC Id CSCStrip. Digi n CSCALCTDigi CSCCorrelated. LCTDigi CSCRPCData int CSCId = (the. Data[i] >>6) & 0 x 3 F; //DMB Id int DMBId = (the. Data[i+2] >> 8) & 3 F; ……… CSCRPCDigi CSCComparator. Digi Alex Tumanov Rice University September 2007 7
Binary data formats can be complicated: These are our simpler data objects There are 29 variables in this class We have 5 (and counting) different versions of the data formats Alex Tumanov Rice University September 2007 8
Better way to unpack data n We use bit field classes: class CSCDMBHeader { public: ……… unsigned L 1 ANumber() const {return L 1 A_; } unsigned CSCId() const {return csc. Id_; } unsigned DMBId() const {return dmb. Id_; } ……… private: … …. . . unsigned L 1 A_ : 4; unsigned cfeb. Id_ : 6; unsigned dmb. Id_ : 6; ……… }; The example of the data class This is how it gets unpacked CSCDMBHeader(unsigned short * the. Data) { memcpy(this, buf, size. In. Words()*2); } Alex Tumanov Rice University September 2007 9
Old way of unpacking binary data Binary Data 101110001011 L 1 A CSC Id DMB Id L 1 A Number = Fill variables one by one CSCId = DMBId = Alex Tumanov Rice University September 2007 10
Bit Field based unpacking Binary Data 101110001011 L 1 A Number = L 1 A Fill variables at once!!! CSCId = CSC Id DMBId = DMB Id Alex Tumanov Rice University September 2007 11
Why bit field unpacking is faster n n CSCDMBHeader class contains 29 variables that are filled from binary data Using bitshifts & masks that would result in 29 lines of code like this: int L 1 A = (the. Data[i] >> 12) & 0 x. F; n Using bit field based classes all 29 variables are filled during just one line: CSCDMBHeader(unsigned short * the. Data) {memcpy(this, the. Data, size. In. Words()*2); } n n Because of this we are able to achieve an order of magnitude of improvement in the average unpacking time (currently around 3 ms/event) Total number of bit field variables in our package – 355 Alex Tumanov Rice University September 2007 12
Understanding memcpy() n n Memcpy implementation in assember loads two addresses and the size into registers and does the copying in a single hardware instruction Assignment operator in L 1 A = (the. Data[i] >> 12) & 0 x. F; is at least one copy operation Alex Tumanov Rice University // C++ code #include <memory. h> const int MAXSIZE = 1000; int main() { int *a, *b, i; a = new int[MAXSIZE]; b = new int[MAXSIZE]; memcpy(b, a, sizeof(int) * MAXSIZE); return 0; } // Assembly code // the address of a is in eax // and the address of b in edx mov ecx, 1000 mov esi, eax mov edi, edx rep movsd September 2007 13
Memcpy() timing benchmarks #include <memory. h> CPU Time, const int MAXSIZE = 1000; seconds per 1 mln int main() repetitions { int *a, *b, i; a = new int[MAXSIZE]; b = new int[MAXSIZE]; const int MAXSIZE = 1000; int main() { int *a, *b, i; a = new int[MAXSIZE]; b = new int[MAXSIZE]; memcpy(b, a, sizeof(int) * MAXSIZE); return 0; } for (long j = 0; j < MAXSIZE; ++j) { *a++ = *b++; } return 0; } Alex Tumanov MAXSIZE X 1000 Rice University September 2007 14
A word about raw data packing n n Data stored in persistent objects (digis) can be converted into raw binary bite stream (Digi->Raw packing) Because each class stores data as bit fields all that is needed to create raw bit stream is this pointer and size() method that returns the size of the object: unsigned short * data() {return (unsigned short *) this; } boost: : dynamic_bitset<> raw. DMBbits = Ushort. To. Bitset(the. DMBHeader. size(), the. DMBHeader. data()); n We use boost: : dynamic_bitset<> class to handle bit streams Alex Tumanov Rice University September 2007 15
Conclusions n n n An order of magnitude in data unpacking speed can be gained by using bit fields instead of conventional bit shifts and masks The bit field data classes are usually easier to read and write – a huge advantage to help avoid probably the biggest source of unpacking problems – errors/typos/bugs in bit definitions Particular attention should be paid to questions related to memory and 32 bit vs 64 bit CPU architecture and endian-ness issues: è è Alex Tumanov Recently CSC unpacking code was successfully moved from 32 bit Scientific Linux 3 (SLC 3) machines to 64 bit SLC 4 Extensive testing by Valgrind shows no memory problems Rice University September 2007 16
Appendix Alex Tumanov Rice University September 2007 17
Pouring wine (data) slowly Inefficient way: pour one at a time Alex Tumanov Rice University September 2007 18
Pouring wine (data) fast! Efficient way: prearrange glasses and pour into all at once! Alex Tumanov Rice University September 2007 19
- Slides: 19