Open Storage API part II Open Storage API

Open. Storage API part II Open. Storage API Training 1

Image Creation and Retrieval stsp_create_image() stsp_write_image() stsp_close_image() stsp_open_image() stsp_flush() Open. Storage API Training 2

Image Creation and Retrieval • How a new image is initialized • stsp_create_image(). Pass in sts_lsu_t (server handle + LSU name), sts_image_def_t described previously, and pendingflag. Pendingflag is a hint (not definitive) – indicates whether this process intends to create another image after this one on same LSU. • In the initialized image’s sts_image_info_t, set STS_II_IMAGE_PENDING, STS_II_BUSY_WRITE. Ignore STS_II_FILES_PENDING and STS_II_FILES_CREATED. Open. Storage API Training 3

Image Creation and Retrieval • How image data is written • stsp_write_image(). Passed in: image handle, stat structure, buffer ointer, length, offset in image. Return bytes successfully written hrough bytewritten. See Section VI, Image Content Description, for discussion of stat structure. • Argument len is n * STS_BLOCK_SIZE. • stsp_write_image() need not be synchronous, despite appearance. • Optimize data transmission to storage server! If data is buffered, and an error subsequently occurs, return error from stsp_write_image() or stsp_close_image(), whichever occurs first. • stsp_flush() (optional) – flush any buffered data to storage server • As image is written, update size in sts_image_info_t. imo_size. Update statistics in sts_lsu_info_t (lsu_used, lsu_used_phys, possibly lsu_capacity). Open. Storage API Training 4

Image Creation and Retrieval • Normal completion, complete image • stsp_close_image() is called with flag argument STS_CLOSEF_IH_COMPLETE. • Clear STS_II_IMAGE_PENDING, set STS_II_IMAGE_CREATED. Clear STS_II_BUSY_WRITE. • Image cannot be reopened for writing. Open. Storage API Training 5

Image Creation and Retrieval • Normal completion, incomplete image • Occurs during checkpoint/restart. • stsp_close_image() is called with flag either clear (STS_CLOSEF_IH_INCOMPLETE). • Clear STS_II_BUSY_WRITE. • Image can be reopened for writing. Open. Storage API Training 6

Image Creation and Retrieval • Abnormal completion • Connection dropped before stsp_close_image() is called. • Clear STS_II_BUSY_WRITE. If image is broken (cannot be written to any more) clear STS_II_IMAGE_PENDING. Open. Storage API Training 7

Image Deletion – stsp_delete_image() • How an image is deleted • At present, image deletion can only be initiated by the DPA. • stsp_delete_image(). Pass in sts_lsu_t (server handle + LSU name), sts_image_def_t containing image name, asyncflag means deletion can be asynchronous (no return code from actual deletion). Open. Storage API Training 8

Image Deletion – stsp_delete_image() • Space reclamation associated with image deletion • Defined by LSU properties described previously (STS_LSUF_STORAGE_*). Open. Storage API Training 9

Image Retrieval – stsp_read_image() • Steps in image retrieval • stsp_open_image(). Image must be complete (STS_II_IMAGE_COMPLETE). Pass sts_lsu_t (server handle + LSU name), sts_image_def_t, mode STS_O_READ. Return image handle. • Iterate stsp_read_image(). Pass image handle, buffer pointer, offset, length. Return bytes successfully read through bytesread. Argument len is n * STS_BLOCK_SIZE • stsp_close_image(). Open. Storage API Training 10

Image Duplication (Optimized) stsp_open_target_server() stsp_copy_extent() stsp_copy_image() Open. Storage API Training 11

Image Duplication (Optimized) • Concept of optimized duplication • Image copy is implemented by storage server, instead of application reading source image and writing to target image. • Image data flows directly from source storage server to target storage server. Open. Storage API Training 12

Image Duplication (Optimized) • How optimized image copy differs from normal image creation • Image creation: stsp_open_server(), stsp_create_image(), iterate stsp_write_image(), stsp_close_server(). • Optimized copy: either • a. stsp_open_target_server(), stsp_create_image(), stsp_open_server(), stsp_open_image(), iterate stsp_copy_extent(), stsp_close_image(source), stsp_close_image(target), stsp_close_server(source), stsp_close_server(target). This copies image extent by extent. Or • b. stsp_open_target_server(), stsp_open_server(), stsp_copy_image(), stsp_close_server(source), stsp_close_server(target). This copies entire image in one operation. • c. Whole-image copy can also use asynchronous interfaces stsp_async_copy_image() and stsp_named_async_copy_image() (see Part III, Events and Asynchronous Operations). Open. Storage API Training 13

Image Duplication (Optimized) • stsp_open_target_server() similar to stsp_open_server(), but contains server handle for the source storage server. The server parameters refer to the target storage server. This allows the source storage server to open a server handle to the target storage server. The returned server handle is only valid for stsp_create_image(), stsp_open_image(), stsp_copy_image(), stsp_async_copy_image(), stsp_named_async_copy_image(), and stsp_close_server(). • stsp_copy_extent(): arguments are source and target image handles, source and target offset, length. Return bytes successfully copied through bytescopied. Target image handle may refer to image on local or remote storage server. Argument len is n * STS_BLOCK_SIZE. flags argument allows to_offset and len arguments to be overridden; STS_CX_APPEND means append to target image, STS_CX_REMAINDER means that len encompasses all of the source image beyond from_offset. These are conveniences for the OST application. • If image duplication is supported, stsp_open_target_server() and either stsp_copy_extent() or whole_image copy must be implemented. Set server capability flags in sts_server_info_t. srv_cap[], and set srv_flags. STS_SRV_IMAGE_COPY. Open. Storage API Training 14

Virtual Image Creation – stsp_include_in_image() • Concept of virtual images, and how a virtual image differs from a normal image • A virtual image consists of references to other images, which must be stored in the same storage server. • Likely implementation: the storage server creates a map for the virtual image, which contains pointers to extents in other images. Extents have reference counts so that if an image is deleted and some of its extents are referenced by other images, those extents are not freed. • Don't implement stsp_include_in_image() by copying data. Do that with stsp_copy_extent() instead. Open. Storage API Training 15

Virtual Image Creation – stsp_include_in_image() • Creating a virtual image • Initialize with stsp_create_image() in the usual way. • Specify extents with stsp_include_in_image(). This is similar to stsp_copy_extent(), with source + target image handles and offsets, and length. Argument length is n * STS_BLOCK_SIZE. • stsp_close_image() when done. Open. Storage API Training 16

Image Meta. Data Don't confuse with image content description, described in Section 6; image content description is also known as metadata. stsp_read_image_meta() stsp_write_image_meta() Open. Storage API Training 17

Image Meta. Data • Image metadata can be written during, before, or after image data. • Metadata is always r/w irrespective of image access mode. • In particular this means metadata can be updated after image is complete. • Also metadata can be written if image is open for read only. • Metadata content is defined by application. • Typically not more than a few MB. • Obvious implementation is to store metadata in a file, associated with image (no need to dedup). Open. Storage API Training 18

Concurrent Image Update • Storage server/plugin is not responsible for protecting badly behaved applications. • For example: two processes write same image concurrently. • Or: one process deletes image while another reads it. • Storage server should execute all calls as received and return errors as necessary. Open. Storage API Training 19

Building a Plugin and Storage Server • Header Files • Plugin only needs to explicitly include stspi. h. • stspi. h includes stsi. h. • stsi. h includes stsplat. h, which is platform-specific. • To get correct stsplat. h, put a –I in makefile for the correct platform under sts/include/platforms, e. g. sts/include/platforms/linux. • STS_VERSION (defaulted in stsi. h) determines build version. • Must define STS_EXTERNAL_VENDOR in build environment Open. Storage API Training 20

Required Interfaces • All those discussed in Part I. All others are optional. See Plugin Writer's Guide (SDK) for authoritative list. Open. Storage API Training 21

Option Interfaces Currently used by Net. Backup • 6. 5. 3: stsp_open_target_server(), stsp_copy_extent() • 6. 5. 4: (subject to change) stsp_include_in_image(), stsp_named_async_copy_image(), stsp_open_evchannel(), stsp_get_event(), stsp_delete_event(), stsp_close_evchannel(), stsp_named_async_wait(), stsp_named_async_cancel(), stsp_named_async_copy_image(), stsp_label_lsu() Open. Storage API Training 22

Major Design Decisions – Storage Server • All of these are strongly vendor-dependent. • Nature of LSUs – relationship with physical storage. • How are LSUs configured? Tool and documentation. STS-P-I does not support LSU configuration, at present. • What LSU properties will be advertised (sts_lsu_info_t). • How to represent dedup attributes of storage, if any. Open. Storage API Training 23

Major Design Decisions – Storage Server • • • How images will be stored and accessed. Files, blocks, blobs, …? How storage is reclaimed when an image is deleted. How to store LSU information and image information. What server properties will be advertised (sts_server_info_t). Storage server naming convention. Which optional services in STS-P-I to support. Which event model to support (push/pull/stored/volatile), if any. How to implement optimized image copy, if supported. How to implement virtual synthetic images, if supported. Handling abnormal conditions from plugin e. g. dropped connection. Handling concurrent connections. Open. Storage API Training 24

Major Design Decisions – Plugin • Also strongly vendor-dependent. • RPC mechanism to storage server. • Write buffering for stsp_write_image() – performance is critical. • Try not to store state information on plugin host. Instead store state in the storage server. Plugin may store non-critical state information that is meaningful only on that host (however, must be synchronized for multiple plugin instances on same host). • Where possible, considering allowing different versions of the same API interface to share some common source. • Multithreading considerations. • Which API version(s) to support. • For multiple versions, all in one binary, or one binary each version? Open. Storage API Training 25

Thank you! Copyright © 2011 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U. S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice. Presentation Identifier Goes Here 26