Supplement 145 Whole Slide Imaging background and design

Supplement 145 Whole Slide Imaging – background and design decisions Harry Solomon GE Healthcare

DICOM BASICS 2 Sup 145 WSI Harry Solomon

DICOM Image Information Object Definition DICOM Composite Information Model Hierarchy Patient Information Patient Module Study Information General Study Module Series Information Image (Instance) Information General Series Module General Image Module Modality Image Module 3 Sup 145 WSI Harry Solomon Patient Name Patient ID Patient Sex Patient Birthdate General Frame of Reference Equipment Module Image Plane Module Multiframe Module Study Unique ID Accession Number Study Date/Time Study Description Referring MD Patient Study Module Contrast/ Bolus Module VOI LUT Module Rows/Columns Bits per Pixel Photometric Image Pixel Module … SOP Common Module Dwight Simon

Data Element Encoding Data Set order of transmission Data Elem. Data Element Value Tag Representation Value Length Data Elem. Value Field Attributes are the logical concepts associated with an information entity Data elements are how attributes are encoded in an information object Similar to TIFF optional field - dependent on negotiated Transfer Syntax 0020000 Dhex UI Study Instance Unique Identifier (0020, 000 D) 4 Sup 145 WSI Harry Solomon 26 hex 1. 2. 840. 1. 113709. 9. 0. 0. 5743. 14575602. 1 Instance UID encoded as “dotted decimal”

Part of a DICOM object (0008, 0005) CS 30 ISO 2022 IR 13ISO 2022 IR 87 Series Information (0008, 0008) CS 22 ORIGINALPRIMARYAXIAL (0008, 0016) UI 26 1. 2. 840. 10008. 5. 1. 4. 1. 1. 2 Image (Instance) Information (0008, 0018) UI 58 1. 2. 392. 200036. 9116. 2. 6. 1. 48. 1211393615. 1211874194. 564494 (0008, 0020) DA 8 20080527 (0008, 0021) DA 8 20080527 (0008, 0022) DA 8 20080527 (0008, 0023) DA 8 20080527 (0008, 0030) TM 10 163836. 000 (0008, 0031) TM 10 164306. 390 (0008, 0032) TM 10 164039. 850 (0008, 0033) TM 10 164040. 397 (0008, 0050) SH 8 5162581 (0008, 0060) CS 2 CT (0008, 0070) LO 8 TOSHIBA Tags in increasing numeric order Value length always an even number Attributes related to modules and information model levels all jumbled up … (7 FE 0, 0010) 5 Patient Information Study Information OW Sup 145 WSI Harry Solomon 524288 00 00 ff ff 00 0 f 4 c 4 a 49 46 00 01. . .

Attributes Logical concepts in the description of an Information Entity May have 0, 1 or many Values • 0 (empty) means the creating application doesn’t know the value of the attribute, e. g. Accession Number (0008, 0050) • Multi-value example: Specific Character Set (0008, 0005) • Value Multiplicity (VM) specified in Part 6 (possibly further constrained in Part 3) Attribute value will be a complex data structure for a Sequence attribute 6 Sup 145 WSI Harry Solomon

Sequence Attributes and Items Sequence attribute has a “value” of a structure of subsidiary attributes • Sequence Attribute name typically includes word “Sequence” • Subsidiary attributes specified in Part 3 with > character Each instantiated set of attributes is a Sequence Item Number of allowed Items specified in Part 3 For editorial convenience the attributes of a Sequence are often documented in a separate Table as a Macro • Include ‘x Macro’ Table m-n • Facilitates reuse of structure in other sequence attributes 7 Sup 145 WSI Harry Solomon

Example: Scheduled Protocol Code Sequence attribute 8 Sup 145 WSI Harry Solomon

Scheduled Protocol Code Sequence attribute Scheduled Protocol Code Sequence (0040, 0008) >Code Value (0008, 0100) >Coding Scheme Version (0008, 0103) >Coding Scheme Designator (0008, 0102) >Code Meaning (0008, 0104) >Protocol Context Sequence (0040, 0440) >>Value Type (0040, A 040) >>Concept Name Code Sequence (0040, A 043) >>>Code Value (0008, 0100) >>>Coding Scheme Designator (0008, 0102) >>>Coding Scheme Version (0008, 0103) >>>Code Meaning (0008, 0104) >>Date. Time (0040, A 120) >>Person Name (0040, A 123) >>Text Value (0040, A 160) >>Concept Code Sequence (0040, A 168) >>>Code Value (0008, 0100) … 9 Sup 145 WSI Harry Solomon >>Content Item Modifier Sequence (0040, 0441) >>>Value Type (0040, A 040) >>>Concept Name Code Sequence (0040, A 043) >>>>Code Value (0008, 0100) >>>>Coding Scheme Designator (0008, 0102) …

Sequence attribute encoding Sequence Items are the “values” of Sequence attributes • Structure placed in the Data Element Value Field Item structure is a “nested data set” of attributes • Attributes in each Item in tag order • Item “wrapped” using special data elements specified in Part 5 Sequence attributes and wrappers may have an “undefined length” flag • Length of Sequence or Item terminated by explicit Delimiter data elements May be “undefined length” Sequence Data Element Tag Value Represen- tation SQ Value Length Item Introducer Value Field Attribute 1 Attribute 2 Specifies length of Item, or may say “undefined length” Item Delimiter Item Introducer Attribute 1 Required if “undefined length” Item Attribute 2 Item Delimiter Sequence Delimiter Required if “undefined length” Sequence Attribute

Image Compression Pixel data can be monochrome, color (RGB or YCb. Cr), or palette color (monochrome colorized through LUT) • No definitions yet for hyperspectral, but it has been discussed Pixel data can be ‘native DICOM’ (with color by-plane or bypixel) Pixel data can be compressed using standard compression schemes, and compressed stream put in pixel data element • JPEG, JPEG-LS, JPEG 2000 (each lossy or lossless) • MPEG 2 • Run-Length Encoding (Packbits) Private compression schemes can also be used 11 Sup 145 WSI Harry Solomon

Compressed Image Encoding Uses structure similar to Sequence attribute • Allows “undefined length” attribute – eliminates 232 byte limitation • 1 st Item is ‘Basic Offset Table’ - pointers to individual frames of a multi-frame image (optional) JPEG and JPEG 2000 Part 1 encode each frame of a multi-frame image in a separate Encapsulated Stream Fragment JPEG 2000 Part 2 (multicomponent) allows arbitrary mapping of frames to stream fragments to allow component collections (inter-frame compression) May be “undefined length Pixel Data Element Value Tag Represen(7 FE 0, 0010) tation OB Value Length Item Introducer Basic Offset Table Specifies length of Basic Offset Table Value Field Item Introducer Encapsulated Stream Fragment 1 Specifies length of Stream Fragment Item Introducer Encapsulated Stream Fragment 2 Sequence Delimiter Required if “undefined length” Pixel Data Attribute

MULTIFRAME IMAGES 13 Sup 145 WSI Harry Solomon

Enhanced Multi-frame paradigm Basic concept used for all new multiframe IODs • MR (Image and Spectroscopy), CT, XA, US, PET Multi-frame object to support 1000+ image studies • Dynamic image header supports functional or acquisition attributes changing during scan • Dimensions allow multiple views of data File size flexibility through concatenations 14 Sup 145 WSI Harry Solomon

Single-frame to Multi. Frame N Objects, N Headers N Frames, One Header Fixed Header 15 Sup 145 WSI Harry Solomon Per-frame header Dimension data Pixel data

Functional Groups and the Per-Frame Header 16 Sup 145 WSI Harry Solomon

Functional Groups Collection of set of closely related attributes • A “mini Module” Structured as a sequence of (usually 1) item under a main Sequence attribute Invoked as a ‘Macro’ in either Shared Functional Groups Sequence or Per-Frame Functional Groups Sequence Keeps items together in encoding under the main Sequence attribute 17 Sup 145 WSI Harry Solomon

Dimensions – properties that may change echo cardiac phase b-value orientation time position volume time 18 Sup 145 WSI Harry Solomon

Multi-phase / Multi-slice Phase (Time) Position Index 1 Slice Order for phase 1 2 3 Phase order for slice 2 Physical Location (Stack) Index 6 5 6 4 3 2 1 Frame number 1 -6 5 6 4 3 2 5 1 Frame number 13 -18 4 3 2 1 Frame number 7 -12 Image frames can be sorted/displayed independent of encoded frame order 19 Sup 145 WSI Harry Solomon

Concatenations What is a concatenation? • • • set of image objects in the same series with the same dimension indexes uniquely identified with a Concatenation UID (0020, 9161) “contained” image objects must have the same Instance Number Why? • file system limits – e. g. , 600 MB CD-R • pseudo real-time transfer of a stream of images – workstation needs to post process images in near real time to figure out when the scan is to be terminated 20 Sup 145 WSI Harry Solomon

Concatenations Legend: Fixed Header Per-frame header Dimension data (not on scale) Pixel data (not on scale) An object may be split up into two or more SOP Instances, using the same concatenation UID 21 Sup 145 WSI Harry Solomon

IMAGE RETRIEVAL 22 Sup 145 WSI Harry Solomon

DICOM Query/Retrieve Allows a system to query another system for a list of available images (query) Also allows a system to request another system to send images (retrieve) Workstation Query Request PACS Query Match(es) Retrieve Request Image(s) Send Query/Retrieve SCU Store Response(s) Retrieve Response 23 Sup 145 WSI Harry Solomon Query/Retrieve SCP

Hierarchical Query DICOM query is not a full SQL-type feature • Limited attributes, no Join capability • Directed toward production imaging department requirements Hierarchical data structure • (Patient), Study, Series, Image levels – Patient attributes typically subsumed in Study level • Query at any level requires specification of unique entity at each higher level 24 Sup 145 WSI Harry Solomon

Typical Hierarchical Query Workstation Level: STUDY Patient ID: D 73001 Date: 20090521 -20090524 Study ID: 09 -35541 Study UID: 1. 2. 789. 45. 63 Patient ID: D 73001 Date: 20090521 Query/Retrieve SCU PACS Study ID: 09 -35602 Study UID: 1. 2. 789. 87. 11 Patient ID: D 73001 Date: 20090522 Study ID: 09 -35819 Study UID: 1. 2. 789. 154. 3 Patient ID: D 73001 Date: 20090524 Level: SERIES Study UID: 1. 2. 789. 87. 11 Series Num: 1 Series UID: 1. 2. 405. 31. 1 Modality: CT Study UID: 1. 2. 789. 87. 11 Series Num: 2 Series UID: 1. 2. 405. 31. 2 Modality: CT Query/Retrieve SCP Level: IMAGE Study UID: 1. 2. 789. 87. 11 Series UID: 1. 2. 405. 31. 1 Image UID: 1. 2. 405. 31. 1. 99. 1 25 Sup 145 WSI Harry Solomon Study UID: 1. 2. 789. 87. 11 Series UID: 1. 2. 405. 31. 1 Image UID: 1. 2. 405. 31. 1. 99. 2 Study UID: 1. 2. 789. 87. 11 Series UID: 1. 2. 405. 31. 1 Image UID: 1. 2. 405. 31. 1. 99. 3

Classical Hierarchical Retrieve can be at any hierarchical level • (Patient), Study, Series, Image • Retrieve at any level requires unique ID of entity at each higher level Object transfer can be on separate Association (C-MOVE) or on same Association (C-GET) • C-MOVE object transfer can be directed to third party Examples: • Retrieve all objects under Study UID 1. 2. 789. 87. 11 / Series UID 1. 2. 405. 31. 1 • Retrieve single object Study UID 1. 2. 789. 87. 11 / Series UID 1. 2. 405. 31. 1 / Instance UID 1. 2. 405. 31. 1. 99. 1 Retrieved objects sent and confirmed as wholes 26 Sup 145 WSI Harry Solomon

Interactive JPIP Retrieve Image Store SCU and SCP can negotiate a JPEG 2000 Interactive Protocol (JPIP) Transfer Syntax • Image header (i. e. , entire object minus pixel data) transferred and confirmed as usual • Pixel data replaced by URL to JPIP service for this image Limitations • Pixel data must be in JPEG 2000 format • Storage Commitment not allowed • Duration of availability of JPIP not specified or guaranteed Capabilities • Retrieve subset of image (ROI) • Retrieve at a lower resolution (e. g. , for quick navigation) 27 Sup 145 WSI Harry Solomon

Frame-based retrieve in w Ne 9 20 m 0 ent 119 le p Sup Retrieve subset of frames from a multi-frame image • • Selected frames of a volumetric stack (ROI) Decimated volume (e. g. , every 10 th slice) Single dimension of a multi-dimensional image Time snippet of motion image (video) SCU & SCP negotiate “Instance Root Retrieve” SOP Class SCU specifies selected frames or time interval SCP creates new multi-frame image with derivation attributes • Frame Derivation Module and Contributing Equipment Sequence • Correct subset of Functional Group Sequence Items 28 Sup 145 WSI Harry Solomon

VOCABULARY AND STRUCTURED REPORTING 29 Sup 145 WSI Harry Solomon

Vocabulary-intensive messaging There’s a lot of things we want to say about imaging that cannot be pre-defined in fixed DICOM attributes • E. g. , specimen processing How do we define message attributes to handle what we need to say? 30 Sup 145 WSI Harry Solomon

Name-value pairs 00180015 ABDOMENPELVIS 00180015 = Body Part Examined < Body. Part. Examined “ABDOMENPELVIS” /> <el> <name “Body. Part. Examined” /> <value “ABDOMENPELVIS” /> </el> <name code=00180015 system=DICOM meaning=“Body Part Examined” /> <value code=R-FAB 57 system=SNOMED meaning=“Abdomen and pelvis” /> Why would we</el> want to do this? 31 Sup 145 WSI Harry Solomon

External coded/concept terminologies Flexibility and extensibility Leverage externally defined/maintained concepts Semantic rigor through referenced dictionary/ ontology General structure – higher layer of abstraction Allows generalized messaging applications Shared vocabulary across disparate systems 32 Sup 145 WSI Harry Solomon

SNOMED Systematized Nomenclature of Medicine Most comprehensive clinical healthcare terminology • 375, 000 concepts; 900, 000 relationships between concepts • Multi-hierarchically organized Primary external vocabulary system for DICOM • Anatomy • Procedures (including radiographic views and methods) • Clinical findings Originally developed by the College of American Pathologists, now managed by an international consortium of governmental agencies (IHTSDO) 33 Sup 145 WSI Harry Solomon

LOINC Logical Observation Identifier Names and Codes Standard coding system for laboratory and clinical observations • Hosted by Regenstrief Institute • Supported by National Library of Medicine Particularly focused on names of laboratory and clinical tests • 50, 000 codes; over 275, 000 relationships Major external code system for DICOM and HL 7 34 Sup 145 WSI Harry Solomon

Code Sequences DICOM Part 3 “Triplet coding” : code value, scheme, meaning (version seldom used)

Context Groups (Value Sets) DICOM Part 3 DICOM Part 16

Content Items Generic Name: Value pair using external coding for Name concept Encoded as Item in Sequence attributes: • • Acquisition Context Sequence (in image IODs) Protocol Context Sequence (in Modality Worklist) Content Sequence (in Structured Reporting IODs) Specimen Preparation Step Sequence (in Specimen Module) Content Item Concept Name Sequence (0040, A 043) Code (0008, 0100) Date. Time Value (0040, A 120) Person Name Value (0040, A 123) Text Value (0040, A 160) UID Value (0040, A 124) Referenced SOP Sequence (0008, 1199) SOP Class UID (0008, 0050) SOP Instance UID (0008, 0055) Value Type (0040, A 040) Scheme (0008, 0102) Concept Value Sequence (0040, A 168) Numeric Value (0040, A 30 A) Measurement Units Sequence (0040, 08 EA) Meaning (0008, 0104) Code (0008, 0100) Scheme (0008, 0102) Meaning (0008, 0104)

Templates Structure for Content Items - like Modules are a structure for Attributes Specified in DICOM Part 16 Coding Scheme Code Value Code Meaning HL 7 v 3 Act. Class equivalent SRT P 3 -02000 Specimen collection SPECCOLLECT SRT P 3 -05013 Specimen receiving CONTREG SRT P 3 -4000 A Sampling of tissue specimen PROC SRT P 3 -00003 Staining SPCTRT

ANNOTATION AND SEGMENTATION 39 Sup 145 WSI Harry Solomon

DICOM annotation principles Annotations are conveyed in information objects separate from the original image Annotations may be created at a time much later than the image acquisition, and in a completely different environment Multiple annotation objects can reference the same image Selection of an annotation object for display implicitly invokes display of the referenced image 40 Sup 145 WSI Harry Solomon

Annotation types Presentation States Structured Reporting Segmentation 41 Sup 145 WSI Harry Solomon

Presentation State Softcopy Presentation States define how referenced image(s) will be displayed • • Transforms to device independent grayscale/color space (LUTs) Selection of display area (ROI) of the image Image rotate or flip Graphical and textual annotations, overlays, shutters Grayscale, color, and pseudo-color SPSs Blending SPS overlays a pseudo-color image on a grayscale image • E. g. , for PET/CT • Blending on grayscale originals (currently no standard for blending of color originals) 42 Sup 145 WSI Harry Solomon

Structured Reporting Presentation State annotations are for human reading, not interoperable for automated applications • No controlled and coded vocabulary, no structural semantics (relationships between annotations) SR important for (semi-)automated imaging analysis and review processes 43 Sup 145 WSI Harry Solomon

Key Image Note SR-type object that provides a classification and a textual comment for a referenced object • Formally known as “Key Object Selection”, but commonly denoted “Key Image Note” after IHE use case and profile Classifications typically identify intended subsequent use of referenced objects • “For Referring Provider”, “For Research”, “For Report Attachment” • “Rejected for Quality Reasons”, “Signed Complete Study Content” 44 Sup 145 WSI Harry Solomon

Segmentation Derived image object • Uses enhanced multi-frame mechanism Multiple segments per object • Each segment linked to a categorization • Pixels show presence of category at pixel location • Binary (1 -bit/pixel) or fractional (probability or occupancy) Segmentation object is typically in same Frame of Reference as source image Segments can be displayed as overlays on source image 45 Sup 145 WSI Harry Solomon

Segmentation Example 46 Sup 145 WSI Harry Solomon

PATHOLOGY IN DICOM – SPECIMEN AND WORKFLOW 47 Sup 145 WSI Harry Solomon

What’s NOT in Sup 145 All the modules already standardized • Patient, Study, Series, Equipment, General Image • Multi-Frame Functional Groups and Dimensions • Sup 122 Specimen Module Explicit description of workflow • Use of Modality Worklist, Modality Performed Procedure Step, Image Availability Notification, etc. 48 Sup 145 WSI Harry Solomon

Sup 122 Specimen Identification Support for pathology lab workflow, specimen-based imaging • Gross specimens, blocks, vials, slides • Image-guided biopsy samples Specimen Module at image level of hierarchy • Identification, processing history • May be used with current Visible Light image object definitions Update to Modality Worklist to convey Specimen Module • Enables automated slide scanning devices to fully populate header Update to Modality Performed Procedure Step to identify imaged specimen • Allows LIS/APLIS to track images for specimens 49 Sup 145 WSI Harry Solomon

Specimen Imaging Information Model Basic DICOM Information Model Disambiguates specimen and container Container is target of image Container may have more than one specimen Specimens have a physical derivation (preparation) from parent specimens When more than one specimen in an imaged container, each specimen is distinguished (e. g. , by position or color-coding) 50 Sup 145 WSI Harry Solomon

Preparation Step 0 -n Preparation Steps per Specimen Each Preparation Step described by 1 -n structured Content Items (name: value pairs) • Acquisition Context plus structuring into steps DICOM Template 8001 Specimen Preparation Coding Scheme Code Value Code Meaning HL 7 v 3 Act. Class equivalent SRT P 3 -02000 Specimen collection SPECCOLLECT SRT P 3 -05013 Specimen receiving CONTREG SRT P 3 -4000 A Sampling of tissue specimen PROC SRT P 3 -00003 Staining SPCTRT SRT P 3 -05000 Specimen processing SPCTRT

Preparation steps example

Managed Workflow Concepts (IHE) ORDER : A request for departmental service REQUESTED PROCEDURE : Report Unit of work resulting in one with associated codified, billable acts PROCEDURE STEP : Scheduled Procedure Step: Performed Procedure Step: 53 Sup 145 WSI Harry Solomon The smallest unit of managed work in the workflow ‘A unit of work to do’ ‘A unit of work done’

Simple Workflow One Order – One Procedure – One Study – One Report ORDER A request for Departmental Service DICOM Modality Worklist Scheduled Procedure Step Requested Procedure Imaging Department Performed Procedure Step One or more series of images Report Set of Codifiable, Billable, Acts Acquisition Modality 54 Sup 145 WSI Harry Solomon Charles Parisot - IHE

Multiple Modality Steps ORDER A request for Departmental Service DICOM Modality Worklist. Imaging Department Scheduled Procedure Step A Requested Procedure Scheduled Procedure Step B DICOM Modality Worklist Performed Procedure Step P 1 One or more series of images Performed Acquisition Procedure Step P 2 Modality One or more series of images Acquisition Modality Report Set of Codifiable, Billable, Acts

Anatomic Pathology Imaging Workflow Interpretation Worklist by accession Pathology order Slide preparation history data Specimen accessioning data Modality Worklist Query by slide barcode Gross specimen accessioning Images w/ slide prep history Images – X-ray, U/S, optical, etc. 56 Imaging task completion w/ list of images and specimen IDs Whole Slide Scanner Surgical or biopsy procedure Sup 145 WSI Harry Solomon Workstation Imaging task w/ slide preparation history data Images

SUP 145 WHOLE SLIDE IMAGING PROPOSAL 57 Sup 145 WSI Harry Solomon

Sup 145 multi-frame tiling concept Use multi-frame image objects (not object per tile) Thumbnail Image Intermediate Image Tiles Baseline Image Tiles Single frame image Multi-frame image (single object) may include multiple Z-planes, color planes In 1 or more DICOM Series 58 Sup 145 WSI Harry Solomon

Alternate approach (not in draft!) Remove 64 k 2 image matrix restriction Can leverage JPEG 2000 Part 2 multi-component compression Use JPEG Interactive Protocol capabilities JPIP low-res view of baseline image JPIP medium-res view of baseline image Baseline Image 59 Sup 145 WSI Harry Solomon Multi-frame (Z-planes, colors) image (single object)

Total Pixel Matrix Origin Rows Columns → Total pixel matrix origin at top left hand corner of imaged volume ↓ Frame (tile) rows and columns align with total pixel matrix rows and columns Frames limited to 216 columns and rows Frame Pixel Matrix Origin 60 Sup 145 WSI Harry Solomon Total pixel matrix limited to 232 columns and rows

Z-planes are identified as nominal physical height of image focal plane above reference surface (μm) Z-plane information is used for relative spatial positioning of image planes, and nominal interplane distance ↑ Z Cover slip ↑ Z 61 Sup 145 WSI Harry Solomon Specimen Slide substrate (glass) An image plane may track variable specimen thickness / surface contour, but only one Zvalue used

Z planes track curved surface Z plane 1, Z plane 2, Z plane 3, Z plane 4 62 Sup 145 WSI Harry Solomon Viktor Sebestyén Varga – 3 DHISTECH Ltd.

Organization of tiles into objects All valid: Single Multi-frame image per Z-plane Multi-frame image per spatial region 63 Sup 145 WSI Harry Solomon

Sparse tiling Multi-frame med-res image Multi-frame hi-res image Only selected tiles encoded Full image matrix might be encoded at lower resolution 64 Sup 145 WSI Harry Solomon

Localizer Image Flavor Thumbnail image (single frame) plus multi-resolution navigation links Each tile of other resolution images has its corresponding area identified in thumbnail Full description of target tiles • Object UID and frame # • Resolution • Z-plane Multiple target frames can overlap • Different resolution, Z-plane, color, etc. Presentation and any interactive behavior is not defined in standard 65 Sup 145 WSI Harry Solomon

Label Image Flavor Label Purpose is to capture slide label • Any specimen captured is irrelevant Image IOD includes Slide Label Module • Barcode (if deciphered) • Label Text (if deciphered) Burned In Annotation (0028, 0301) might be “NO” if the label includes only a specimen identifier and not patient identifying data 66 Sup 145 WSI Harry Solomon

C. 8. 12. 2 Slide Coordinates Used in VL Slide-Coordinates Microscopic Image IOD Label • Single frame image, typically from microscope-mounted camera ↑ Specimen Y X → Slide Coordinates Origin ↑ Z 67 Sup 145 WSI Harry Solomon VL SCM Image area Cover slip Specimen Slide substrate (glass) Used to localize center of VL SCM Image DICOM Frame of Reference associated with slide corner origin Reproducibility not guaranteed across different mountings of slide, even on same equipment

C. 7. 4. 1 Frame Of Reference Module When a Frame of Reference is identified, it is not important how the [imaging target] is positioned relative to the imaging equipment or where the origin of the Frame Of Reference is located. It is important that the position of the [imaging target] and the origin are constant in relationship to a specific Frame Of Reference The Position Reference Indicator may not coincide with the origin of the fixed frame of reference related to the Frame of Reference UID. The Position Reference Indicator shall be used only for annotation purposes and is not intended to be used as a mathematical spatial reference. 68 Sup 145 WSI Harry Solomon

WSI Image Pixel Matrix Slide. Coordinates Origin ↑ Z 69 Sup 145 WSI Harry Solomon Y ↓ ↑ Image Matrix Origin X → Label Columns → Rows Image Matrix not necessarily aligned to slide edge, nor to Slide-Coordinates Image Matrix origin (top left hand corner) located relative to Slide-Coordinates Frame of Reference origin (X, Y in mm) Direction of rows and columns given as cosines in Slide-Coordinates Frame of Reference Cover slip Specimen Slide substrate (glass) Each tile (frame) TLHC located relative to Image Matrix origin (column, row) Each tile center located relative to Slide. Coordinates origin (X, Y in mm)

Optical paths Each combination of light source, lenses, illumination method, detected wavelengths, etc. used in an acquisition is an optical path • Each path described in an Item of the Optical Path Sequence Examples: • Full spectrum light, transmission, RGB color sensors • u. V light, excitation, blue monochrome sensor Each frame may specify a different optical path • Allows different colors in a single object, including hyperspectral (n monochrome planes) • Identified in Optical Path Functional Group by reference to Optical Path Sequence Item 70 Sup 145 WSI Harry Solomon

Optical paths What parameters are needed? • To be added in Part 16 Context Groups Is a “macro image” simply a selected optical path? Illumination: Color(s) Intensity Type (laser) 71 Sup 145 WSI Harry Solomon Filters: Color(s) Polarization Lens: Illumination Method: Transmission Reflection Scatter Excitation Lens: Filters: Color(s) Polarization Sensor: Color(s)

WSI Functional Groups Standard • Pixel Measures (pixel spacing, layer thickness) – shared • Frame Content (datetime, dimensional location) – per-frame • Referenced Image, Derivation (if required for individual frames) WSI Specific • Plane Position (relative to total matrix and to SCM Fo. R) • Optical Path • Specimen Reference (if multiple specimens on slide are automatically distinguishable) 72 Sup 145 WSI Harry Solomon

Pixel Measures functional group Attribute Name Pixel Measures Sequence Tag (0028, 9110) >Pixel Spacing (0028, 0030) Type Attribute Description 1 Identifies the physical characteristics of the pixels of this frame. Only a single Item shall be permitted in this sequence. 1 C Physical distance in the imaging target (patient, specimen, or phantom) between the centers of each pixel, specified by a numeric pair - adjacent row spacing (delimiter) adjacent column spacing in mm. See 10. 7. 1. 3 for further explanation of the value order. Note: In the case of CT images. . . Required if Volumetric Properties (0008, 9206) is other than DISTORTED or SAMPLED. May be present otherwise. >Slice Thickness (0018, 0050) 1 C Nominal reconstructed slice thickness (for tomographic imaging) or depth of field (for optical imaging) in mm. See C. 7. 6. 2. 1. 1 and C. 7. 6. 16. 2. 3. 1 for further explanation. Required if Volumetric Properties (0008, 9206) is VOLUME or SAMPLED. May be present otherwise. 73 Sup 145 WSI Harry Solomon Redefinition

Plane Position (Slide) functional group Attribute Name Plane Position (Slide) Sequence Tag (gggg, nn 1 A) Type 1 >Position In Image Pixel Matrix (gggg, nn 1 F) 1 >Image Center Point (0040, 071 A) Coordinates Sequence 1 >>X Offset in Slide Coordinate System >>Y Offset in Slide Coordinate System >Z Offset in Slide Coordinate System 74 Sup 145 WSI Harry Solomon (0040, 072 A) 1 (0040, 073 A) 1 (0040, 074 A) 1 Attribute Description Describes position of frame in the Total Pixel Matrix and in the Slide Coordinate System Frame of Reference. Only a single Item may be present in this Sequence. The coordinate of the top left pixel of the frame in the Total Pixel Matrix (see C. 8. 12. X 3. 1. 1), given as columnrow. Column is the horizontal position and row is the vertical position. The coordinate of the top left pixel of the Total Pixel Matrix is 11. Identifies the coordinates of the center point of this frame in the Slide Coordinate System Frame of Reference. Only a single Item shall be permitted in this sequence. See Section C. 8. 12. 2. 1. 1 for further explanation. Note: This attribute allows simplified transformation of a single frame of a multi-frame VL WSI SOP Instance into an instance of the VL Slide Coordinates Microscopy SOP Class. The X offset in millimeters from the Origin of the Slide Coordinate System. See Figure C. 8 -16. The Y offset in millimeters from the Origin of the Slide Coordinate System. See Figure C. 8 -16. The Z offset in microns from the Origin of the Slide Coordinate System, nominally the surface of the glass slide substrate. See Figure C. 8 -17 Note: Required even if only a single focal plane was acquired. Do we need to separate to two attributes to support independent dimensions? Do we really need this sequence introducer (consistency w/ C. 8. 12. 2)? Could just specify X and Y as center point coordinates

Dimensions Based on attributes in functional groups (i. e. , values that change on a perframe basis) Typical dimensions for WSI: • • Total Matrix Column Origin Total Matrix Row Origin Z-Plane Optical Path (color/polarization) Attributes used for Dimensions specified in Multi-frame Dimension Module Each frame specifies its dimensional indexes in Frame Content functional group • Index values (ordinals) mapped to dimensional attribute values • E. g. , with (Column, Row) dimensions, and 40962 pixel frames, frame with index value (2, 3) would have origin columnrow values of 819312289 75 Sup 145 WSI Harry Solomon

Annotations of WSI - Segmentations can be created frame-by-frame / pixel-by-pixel against selected frames of original image • Reference through Derivation Image Functional Group • 1 -bit/source-pixel, or 8 -bits/source-pixel Segmentations can be created against arbitrary areas within a specified Frame of Reference • Requires Plane Position and Plane Orientation Functional Groups – may not be usable with slide coordinates Frame of Reference Display of segmentation can implicitly invoke a nonstandardized overlay or blending with source image • Segmentation can specify its preferred color 76 Sup 145 WSI Harry Solomon

Annotations of WSI - Presentation States Color Presentation State supports annotation of a source image Displayed Area Selection allows up to 231 -1 rows/columns, currently relative to frame-based rows/columns • Proposed enhancement (with new attribute and new enumerated value) to allow Displayed Area Selection and annotation location relative to WSI total matrix, rather than to frame • Implicitly applies to all dimensions (Z-planes, colors), only constrained by explicit frame numbers; should there be a general mechanism to limit by dimension (as is done for segments)? Placement of annotations limited to 24 -bit precision (IEEE 754 32 -bit float) • May be image relative or selected display area relative • Allows sub-pixel resolution up to 8 M rows/columns for image relative 77 Sup 145 WSI Harry Solomon

Presentation State - Displayed Area Selection Attribute Name Displayed Area Selection Sequence Tag (0070, 005 A) Type 1 Attribute Description A sequence of Items each of which describes the displayed area selection for a group of images or frames. Sufficient Items shall be present to describe every image and frame listed in the Presentation State Relationship Module. One or more Items shall be present. >Referenced Image (0008, 1140) 1 C Sequence >>Include ‘Image SOP Instance Reference Macro’ Table 10 -3 >Pixel Origin (gggg, bb 01) 1 C For a referenced multi-frame image, specifies whether the Displayed Interpretation Area Top Left Hand Corner (0070, 0052) and Displayed Area Bottom RIght Hand Corner (0070, 0053) are to be interpreted relative to the individual frame pixel origins, or relative to the Total Pixel Matrix origin (see C. 8. 12. X 3. 1. 1). Required if the Referenced Image Sequence (0008, 1140) >Referenced SOP Class UID (0008, 1150) value is 1. 2. 840. 10008. 5. 1. 4. 1. 1. xxx (VL Whole Slide Microscopy Image). May be present otherwise. Enumerated Values: FRAME VOLUME If not present, TLHC and BRHC are defined relative to the frame pixel origins. 78 >Displayed Area Top Left Hand Corner >Displayed Area Bottom Right Hand Corner … Sup 145 WSI Harry Solomon (0070, 0052) 1 (0070, 0053) 1

Presentation State - Graphic Annotation Units Attribute Name Graphic Annotation Sequence Tag (0070, 0001) Type 1 Attribute Description A sequence of Items each of which represents a group of annotations composed of graphics or text or both. One or more Items shall be present. … >>Bounding Box Annotation Units (0070, 0003) 1 C Units of measure for the axes of the text bounding box. Defines whether or not the annotation is Image or Displayed Area relative. Both dimensions shall have the same units. Enumerated Values: PIXEL = Image relative position specified with sub-pixel resolution such that the origin at the Top Left Hand Corner (TLHC) of the TLHC pixel is 0. 0. 0, the Bottom Right Hand Corner (BRHC) of the TLHC pixel is 1. 01. 0, and the BRHC of the BRHC pixel is ColumnsRows (see figure C. 10. 5 -1). The values must be within the range 0 to ColumnsRows. DISPLAY = Fraction of Specified Displayed Area where 0. 0. 0 is the TLHC and 1. 01. 0 is the BRHC. The values must be within the range 0. 0 to 1. 0. MATRIX = Image relative position specified with sub-pixel resolution such that the origin at the Top Left Hand Corner (TLHC) of the TLHC pixel of the Total Pixel Matrix is 0. 0. 0, the Bottom Right Hand Corner (BRHC) of the TLHC pixel is 1. 01. 0, and the BRHC of the BRHC pixel of the Total Pixel Matrix is Total Pixel Matrix ColumnsTotal Pixel Matrix Rows (see C. 8. 12. X 3. 1. 3). The values must be within the range 0 to Total Pixel Matrix ColumnsTotal Pixel Matrix Rows. This value is valid only if the Referenced Image Sequence (0008, 1140) >Referenced SOP Class UID (0008, 1150) value is 1. 2. 840. 10008. 5. 1. 4. 1. 1. xxx (VL Whole Slide Microscopy Image). Required if Bounding Box Top Left Hand Corner (0070, 0010) or Bounding Box Bottom Right Hand Corner (0070, 0011) is present. 79 Sup 145 WSI Harry Solomon

Modality Worklist Scheduled Specimen Sequence added to MWL in Sup 122 • Allows query by Container ID (slide barcode) • Allows return from SCP of complete Specimen Module (slide processing history to be used for imaging set up and/or inclusion in WSI header Other parameters can be passed in Protocol Context Sequence • Template specification for Content Items Proposed Protocol Optical Paths Sequence • Parallel to Protocol Context Sequence • General VL attributes 80 Sup 145 WSI Harry Solomon