IT 343 Week 3 S 3 CRR Cloud

IT 343 Week 3 –S 3: CRR, Cloud. Front, Securing Buckets, Storage Gateway, Snowball, Transfer Acceleration

S 3 – Cross Region Replication • Replicate data from one AWS region to another AWS region • Version must be enabled on both the source and destination buckets • Regions must be unique • Cannot replicate in same bucket • Files in existing bucket are not replicated automatically. • All the following updates to files will be replicated automatically

S 3 – Cross Region Replication • To copy existing files • Google: aws cli tools bundled • (https: //docs. aws. amazon. com/cli/latest/userguide/install-windows. html) • Run bundled installer and then: • Open command prompt • Type aws configure • You will have to create a user using IAM and use the programmatic access • Access key ID and secret access key • Use aws s 3 ls to show buckets info command • Use aws s 3 cp command

S 3 – Cross Region Replication • Delete markers are not replicated • Deleting individual version or delete markers will not replicate

Cloud. Front – CDN • CDN - Content Delivery Network • Is a system of distributed servers • Network that delivers webpages & other web content to a user based on the geographic location of the user, the origin of the webpage and the content delivery server.

Cloud. Front - Terminology • Edge Location • Cache location • Separate from Region/AZ • Origin of all the files that the CDN will distribute • Can be S 3 Bucket, EC 2 instance, an Elastic Load Balancer, or Route 53 • Can be outside server (non-AWS) • Distribution • Name given the CDN which consists of a collection of Edge locations

Cloud. Front

Cloud. Front • Can be used to deliver entire website • • Dynamic content Static content Streaming Interactive content • Uses a global network of edge locations • Request for content are routed to nearest edge locations • Content is delivered with best possible performance

Cloud. Front • Optimized to work with other Amazon webservices • Can be S 3 Bucket, EC 2 instance, an Elastic Load Balancer, or Route 53 • Also works seamless with any non-AWS origins • Store the original and version of your files

Cloud. Front - Distributions • Web Distribution • Typically used for websites • RTMP • Used for Media Streaming

Cloud. Front – Edge Locations • Are not just for read • Can put an object on one of them • Objects are cache for the life of the TTL (Time To Live) • You can clear cached object • You will be charged to clear immediately

Securing Buckets • By Default • All newly created buckets are private • Access to buckets can be controlled using: • Bucket Policies • Access Control List • S 3 Buckets can be configure to create access logs • Log all request to a bucket. • This can be done to another bucket.

S 3 – Encryption • In transit (to & from bucket) • SSL / TLS • At rest • Server Side Encryption • S 3 Managed Key – SSE-S 3 (AES-256) • AWS Key Management Service, Managed Keys (SSE-KMS) • Audit trail • Envelope Key • Server Side Encryption with Customer Provided Keys (SS 3 -C) • Client Side Encryption

Storage Gateway • Connects on-premise software appliance with cloud-based storage. • Provides seamless and secure integration between an organization’s on-premises IT environment and AWS’s storage infrastructure. • The service enables you to securely store data to AWS cloud for scalable and cost-effective storage.

Storage Gateway • Software Appliance – available for download as a virtual machine image you install on a host in a datacenter. • Supports Vmware ESXi and Microsoft’s Hyper-V • Install gateway, associate with AWS account through an activation process. • Can use AWS Management Console to create the storage gateway option.

Storage Gateway • File Gateway (NFS) • Volumes Gateway (i. SCSI) • Stored Volumes • Cached Volumes • Tape Gateway

File Gateway • Files are stored in S 3 buckets through a Network File System (NFS) mount-point. • Ownership, permissions, and timestamps are stored durably in S 3 usermetadata.

Volume Gateway • The volume interface presents your applications with disk volumes using the i. SCSI block protocol. • Data written to these volumes can be asynchronously backed up as point-in-time snapshots of your volumes and stored in the cloud as Amazon EBS snapshots. • Snapshots are incremental backups that capture only changed blocks. All snapshot storage is also compressed to minimize storage charges.

Volume Gateway – Stored Volumes • Store primary data locally, while asynchronously backing up data to AWS • Provide on-premises application with low-latency access to entire datasets, while provide durable, off-site backups. • Can create storage volumes and mount them as i. SCSI devices from your on-premises application servers. Data written to your stored volumes are stored locally and backed up to S 3 in the form of Elastic block store snapshots. • 1 GB – 16 TB in size for stored volumes

Volume Gateway – Stored Volumes

Volume Gateway – Cached Volumes • Allows S 3 as primary data storage while retaining frequently accessed data locally in you storage gateway • Minimizes the need to scale your on-premise storage infrastructure, while still providing applications with low-latency access to frequently accessed data. • Volumes up to 32 TB in sized can be created and attached to i. SCSI devices from on-premise application servers • Gateway stores data to volumes in S 3 and retains recently read data in on-premises storage gateway’s cache and upload buffer storage. • 1 GB – 32 TB in size for Cached Volumes

Volume Gateway – Cached Volumes

Volume Gateway – Tape Gateway • Durable, cost-effective solution to archive data in AWS Cloud • Virtual Tape Library (VTL) Interface provides allows you to leverage existing tape-based backup application infrastructure to store data on virtual tape cartridges created on you tape gateway • Each tape gateway is preconfigured with a media changer and tape drives, which are available to existing client backup application as i. SCSI devices • You add tape cartridges as you need to archive data • Supported by Net. Backup, Backup Exec, Veeam…

Snowball • AWS Import/Export Disk • Precursor to snowball • Accelerates moving large amount of data into and out of AWS cloud using portable storage devices • Basically you could send in a storage device to Amazon

Snowball • Types • Snowball Edge • Snowmobile

Snowball • Petabyte-Scale data transport solution using secure appliances to transfer • Simple, fast, Secure, can cost a little as one-fifth the cost of highspeed internet. • 80 TB Snowball is available in all regions. • Multiple layers of security • Tamper resistant enclosures • 256 bit encryption • Industry standard Trusted Platform Module (TPM) designed to ensure both security and full chain-of-custody of your data. • Once the data has been processed and verified, AWS performs a erasure of the snow appliance

Snowball

Snowball Edge • On board storage and compute capabilities • 100 TB • Temporary storage tier to support local workload in remote or offline locations • Connect to existing applications and infrastructure using standard storage interfaces • Can be clustered to for local storage and processing data onpremises

Snowball Edge

Snowmobile • Exabyte-scale data transfer • Moves extremely large amounts of data to AWS • 100 PB per snowmobile • 45’ foot shipping container

Snowmobile

S 3 Transfer Acceleration • Utilizes the Cloud. Front Edge Network to accelerate uploads to S 3 • Instead of uploading directly to S 3 Bucket, use a distinct URL to upload to edge location which transfers to S 3

S 3 – Static websites • S 3 can be used to host static websites • Serverless • Very Cheap, Scales automatically • Static only

S 3 Review • • • Object Based, allows you to upload files Files can be – 0 Bytes to 5 TB Unlimited Storage Files are stored in Buckets S 3 universal namespace, name must be unique globally Example URL: • https: //s 3 -eu-west-1. amazonaws. com/it 203 test • Read after Write consistency for PUTS of a new object • Eventual Consistency for overwrite PUTS and DELETES • Takes time to propage

S 3 Review – Storage Tiers • S 3 Standard – • 99. 99% availability • 99. 99999% durability • Stored redundantly across multiple devices in multiple facilities, and designed to sustain the loss of 2 facilities concurrently. • S 3 IA (Infrequently Accessed) • For data that is accessed less frequently but requires rapid access when needed. • Lower fee than S 3 standard but there is a charge for retrieval

S 3 Review – Storage Tiers • S 3 One-Zone-IA • Lower cost option for infrequently accessed data, but do not require the multiple Availability Zone data resilience. • Glaciers • • Very Cheap Used for archival only Expeditated, Standard, or Bulk. Standard retrieval time takes 3 -5 hours.

S 3 Review • Key (Name) • Value (Data) • Version ID • Metadata • Not suitable to install an OS

S 3 Review - Versioning • Stores all version of an object • Including all writes even if you delete an object • • • Great backup tool Once enabled, versioning cannot be disable, only suspended Integrates with Lifecyle rules Versioning MFA delete capability Cross Region Replication, requires versioning enabled on the source bucket

S 3 Review – Lifecycle Management • Can be used in conjunction with versioning • Can be applied to current versions and previous version • Following actions can be done • Transition from standard to infrequently accessed • Archive to Glacier storage • Permanently delete

S 3 Review - Cloud. Front • Edge Location – location where content is cached. • Separate from AWS Region and Availability Zone • Origin – this is the origin of all the files that the CDN will distribute. • Can be S 3 Bucket, EC 2 Instance, Elastic Loud Balancer, or Route 53 • Distribution – name given the CDN which consists of a collection of Edge Locations • Web Distribution – Typically for websites • RTMP – Media Streaming • Edge Location – not just read only, writes are allowed • Objects are cached for TTL • Can clear cached objects (will be charged)

S 3 Review – Securing Buckets • By default, all newly created Buckets are private • You can setup access control • Bucket Policies • Access Control List • S 3 bucket can have access logs. • Can be saved in the bucket or another bucket

S 3 Review – Encryption • In Transit • SSL/TLS • At Rest • Server Side Encryption • S 3 managed keys – (SSE-3) • AWS Key Management Service, Managed Keys (SSE-KMS) • Server Side Encryption with Customer Provided Keys (SS 3 -C) • Client Side Encryption

S 3 Review - Gateways • File Gateway – for flats files stored directly on S 3 • Volume Gateway • Stored volumes – entire dataset is stored on site and async backups to S 3 • Cached Volumes – entire dataset is on S 3 and mostly frequently accessed data is cached on site • Gate Virtual Tape Library (VTL) – Used for backup and uses popular backup apps like Net. Backup, Backup Exec, Veeam…

S 3 Review • Snowball – storage and compute • Snowmobile

S 3 Review • Transfer Acceleration • Speed up transfer to S 3. • Cost extra, greatest impact on people who are in a far away location • Static Websites • • S 3 can host static websites Serverless Very cheap, scales automatically Static only