Digital Imaging and Processing Is seeing believing Lecture

Digital Imaging and Processing: Is seeing, believing? Lecture 15 Digital Imaging

The Nature of Visible Light z A very small part of the total spectrum of electromagnetic waves z Unlike sound, electromagnetic waves can travel through a vacuum z They include the categories of Radio, Microwave, and Visible light waves z They vary in frequency and amplitude

Electromagnetic Spectrum

What is light? z Normally when we use the term "light, " we are referring to a type of electromagnetic wave which stimulates the retina of our eyes. In this sense, we are referring to visible light, a small spectrum of the enormous range of frequencies of electromagnetic radiation.

What is light? z This visible light region consists of a spectrum of wavelengths, which range from approximately 700 nanometers (abbreviated nm) to approximately 400 nm; z that would be 7 x 10 -7 meter to 4 x 10 -7 meter. This narrow band of visible light is affectionately known as ROYGBIV

Fundamental Colors z Dispersion of visible light (through) a prism for instance) produces the colors red (R), orange (O), yellow (Y), green (G), blue (B), indigo (I), and violet (V). It is because of this that visible light is sometimes referred to as ROY G. BIV

The visible light spectrum

White and Black z When all of the colors strike our eye at the same time, we perceive that as WHITE z Black is defined as the absence of light. It is actually not a real color

Our eyes z The retinas of our eyes contain cells called Rods and Cones. Rods are sensitive to intensity while cones are sensitive to wavelength (color) z As it turns out our cones are sensitive to Red, Green and Blue above all else

Relative Sensitivity of our eyes

Photography Timeline z 1822 – Nicéphore Niépce takes the first fixed, permanent photograph, of an engraving of Pope Pius VII z 1826 – Nicéphore Niépce takes the first fixed, permanent photograph from nature a landscape that required an eight hour exposure z 1839 - William Fox Talbot invented the positive / negative process widely used in modern photography z 1861 – The first color photographis shown by James Clerk Maxwell z 1887 – Celluloid film base introduced z 1888 – Kodak n° 1 box camera is mass marketed; first easy-to-use camera.

Timeline cont. z 1891 – William Kennedy Laurie Dickson develops the "kinetoscopic camera" (motion pictures) while working for Thomas Edison z 1902 – Arthur Korn devises practical phototelegraphy technology (enabling the electronic transmission of pictures) z 1939 – Agfacolor negative-positive color material, the first modern "print" film z 1948 - Edwin H. Land introduces the first Polaroid instant image camera.

Timeline cont. z 1973 – Fairchild Semiconductor releases the first large image forming CCD chip; 100 rows and 100 columns z 1986 – Kodak scientists invent the world's first megapixel sensor z 1994 -1995 First consumer digital cameras introduced (Apple, Casio, and Kodak) z 2008 – Polaroid announces it is discontinuing the production of all instant film products, citing the rise of digital imaging technology. z 2009 - Kodak announces the discontinuance of Kodachrome film

Digital Imaging Basics z z Image Acquisition Digital Image Representation Storage Implications and Compression Image Processing

Charged Coupled Devices z Invented over 40 years ago z Consists of an array of transistors and capacitors (pixels) that are very sensitive to light z Photons hit the array which creates and stores electrical charges proportional to intensity of the light z The values for each pixel are then converted to binary numbers and stored in memory in the camera/computer

CCDs Continued z Originally used in spy satellites and astronomy applications due to high sensitivity z Recent popularity for consumer applications has resulted in dramatic cost reduction z Now used in every type of imaging z Replacing film in many applications z Higher equipment cost, lower operational cost

Kodak Digital Camera 1975 Steve Sasson CCD Imager Black+white 23 sec record 18

A Charged Coupled Device (CCD) A Outputs an analog electrical signal that must be sampled and converted to digital

CMOS Sensor Outputs a digital binary signal for every pixel

A Digital Camera has predefined Pixels Image is projected onto Camera’s sensor By camera lens Each pixel is then assigned a numeric value in binary which corresponds to color and luminence Sensor consists of an array of Millions of light sensitive transistors and capacitors

Image Acquisition Delivery PC CAMERA I/O Interface (USB/ Firewire) running Photoshop Or similar program Disk

Analog Images are represented by waves of photons traveling through space z a natural image is typically represented by a continuous or analog signal (such as a photograph, video frame, etc. )

Analog into Digital

Image Acquisition z Acquisition determines ultimate resolution z Remember, you cannot “create” resolution after the fact z The more samples “acquired” the better the resolution (accuracy) z The higher the resolution, the more data acquired, hence more storage required

Representing Digital Images Digital images are composed of PIXELS (or picture elements) z digitizing samples the natural image into discrete components

Representing Digital Images Digital images are composed of PIXELS (or picture elements) z each discrete sample is averaged to represent a uniform value for that area in the image

Representing Digital Images Digital images are composed of PIXELS (or picture elements) z PICTURE RESOLUTION is the number of pixels or samples used to represent the image

Representing Digital Images Digital images are composed of PIXELS (or picture elements) z ASPECT RATIO expresses this resolution as the product of the no. of horizontal pixels by the no. of vertical pixels

Representing Digital Images Digital images are composed of PIXELS (or picture elements) z this image is square, 50 X 50 z typical ratios are 320 X 200 or 1. 6: 1, 640 X 480, 800 X 600, and 1024 X 768 --all of which are 1. 33: 1

Pixels and Resolution z Images are represented (ultimately) as arrays of pixels (picture elements). z Image resolution is the number of pixels in the image (e. g. , 600 x 1000) z Display resolution is the number of pixels in the display device (often expressed in dots per square inch, or dpi).

Representing Digital Images Picture resolution determines both the amount of detail as well as its storage requirements z here is a (edited) digitized image with a resolution of 272 X 416

Representing Digital Images Picture resolution determines both the amount of detail as well as its storage requirements z notice the changes when the resolution is reduced (136 X 208)

Representing Digital Images Picture resolution determines both the amount of detail as well as its storage requirements z notice more changes when the resolution is reduced (68 X 104)

Representing Digital Images QUANTIZING a sampled image refers to representing each discrete sample by a set of numbers chosen from a given scale z imagine a simple image with a bright object in the foreground surrounded by a dark background

Representing Digital Images QUANTIZING a sampled image refers to representing each discrete sample by a set of numbers chosen from a given scale z suppose that we sampled the signal horizontally across the middle of the image

Representing Digital Images QUANTIZING a sampled image refers to representing each discrete sample by a set of numbers chosen from a given scale z if we assigned a numeric scale for the signal it might look like this

Representing Color z The RGB (red, green, blue) color system represents color by specifying the intensity of red, green, and blue light. z 24 bit color would use 8 bits (one byte) for each color. z In this scheme we specify 8 numbers in base 16 (hexadecimal) = rrggbb.

Representing Grayscale z For black and white images we need to represent the shade. z A binary image would represent only white or black pixels. z Four bits per pixel would allow “ 16 shades of gray”

Representing Digital Images DYNAMIC RANGE refers to the number of values for the measuring scale used in quantizing z Here is an intensity or graylevel image with 256 levels (i. e. , 0 to 255 scale)

Representing Digital Images DYNAMIC RANGE refers to the number of values for the measuring scale used in quantizing z Here is an intensity or graylevel image with 16 levels (i. e. , 0 to 15 scale)

Representing Digital Images DYNAMIC RANGE refers to the number of values for the measuring scale used in quantizing z Here is an intensity or graylevel image with 4 levels (i. e. , 0 to 3 scale)

Representing Digital Images DYNAMIC RANGE refers to the number of values for the measuring scale used in quantizing z Here is an intensity or graylevel image with 2 levels (i. e. , 0 to 1 scale or a binary image)

JPEG and GIF Storage Formats z JPEG (Joint Photographic Experts Group) is a set of lossy image compression techniques. z GIF (Graphic Interchange Format) uses a combination of color tables and lossless compression.

Image Modification Original Image Computer Program Revised Image

Global Intensity Modification z Let us just consider black and white images (so each pixel is represented in, say, one byte = 256 possibilities). z A global intensity modification technique would change, say, all pixels with intensity 111 to intensity 158. z Why would one want to do such a thing?

Making a Picture Brighter To make an overly dark picture brighter, generally raise the light intensity numbers. Output light intensity Make brighter No modification Input light intensity

Increasing Contrast

Histograms

Processing Digital Images z digital images are often processed using “digital filters” z digital filters are based on mathematical functions that operate on the pixels of the image

Processing Digital Images z there are two classes of digital filters: global and local z global filters transform each pixel uniformly according to the function regardless of its location in the image z local filters transform a pixel depending upon its relation to surrounding ones

Global Filters z z z Brightness and Contrast control Histogram thresholding Histogram stretching or equalization Color corrections Hue-shifting and colorizing Inversions

Global Filters z a histogram is a graph depicting the frequency distribution of pixel values in the image z thresholding creates a binary image by converting pixels according to a threshold value

Global Filters z Histogram stretching redistributes pixel values in the image that has poor contrast z Equalization improves images with poor contrast

Global Filters z Hue-shifting is used to modify the color makeup of an image z Pseudo-coloring assigns hues to intensity ranges for better rendering of details Colorized image of Mississippi at Vicksburg

Local Filters z z z Sharpening Blurring Unsharp Masking Edge and line detection Noise filters

Local Filters z Edge and line detection filters subtract all parts of the image except edges or boundaries between two different regions z edge detection is often used to recognized objects of interest in the image edges and lines detected in an image of toy blocks

Editing Images z editing or retouching an image involves selecting a region of the digital image for processing using some special effect z image compositing combines components of two or more images into a single image z painting (or rotoscoping) an image is to edit the image by hand with graphic tools that alter color and details

Editing Images z compositing images involves combining separate image layers into one image z layers may be moved and arranged

Computer Animation z Computer animation is simply computer graphics for sequences of scenes designed to be viewed in rapid succession. z Commercial computer animation is very labor intensive.

Animation and Physics z The goal of computer animation research is to model not just how the world looks, but how it changes. z For example, how do clothes fold when the body inside moves, or how do the limbs of a person (or a dog) move when the person/dog is walking.

Graphics and Image Processing z The distinction between computer graphics and image processing is becoming increasingly blurry. z This is because many of the most advanced image processing techniques employ computer graphics ideas like modeling and rendering.

Noise Reduction Techniques Noise in an image is the insertion of random, spurious pixel values because of non-image events like the decay of a photograph, or errors in the transmission of the image (as when a picture is transmitted from a satellite to the ground station).

How Can One Remove Noise? z One can simply smooth pixel values so that, say, white spots become closer in value to the surrounding pixels. But this removes contrast generally. z Better is to locate surface boundaries and remove abrupt intensity changes that do not correspond to boundaries. z This requires building up an image model.

Graphics and Scene Recognition z These techniques require (to a greater or lesser degree) scene recognition - the ability to infer from one or more images what is in the scene, and where. z Scene recognition is normally considered to be part of AI (Artificial Intelligence - the study of how to make computers behave “intelligently”).

Indexed Color z INDEXED COLOR images are derived from full color images z INDEXED COLOR images are smaller or more compact in storage z are composed of pixels selected from a limited palette of colors or shades z They are “browser safe”

Digital Image Files z Digital images are converted to files for storage and transfer z The file type is a special format for ordering and storing the bytes that make up the image z File types or formats are not necessarily compatible z You must often match the file type with the application (. psd = photoshop)

Storing Digital Images z TIFF (Tagged Image File Format) y used by most document preparation programs y has optional lossless compression y Windows and Macintosh formats differ z GIF (Graphic Interchange Format) y indexed color image (up to 256 colors) y compressed y used in Web applications

Storing Digital Images z JPEG (Joint Photographic Experts Group) y lossy compression with variable controls y also used in Web applications z WMF (Windows Metafile Format) y “metafile” formats permit a variety of image types z PICT y the metafile format for Macintosh apps

With Digital Imaging You can create just about anything…. .

911 Accidental Tourist

Great White Taken in South Africa

Rescue Diver Drill Under the Golden Gate

Shark attacking rescue diver in San Francisco Bay!

Quick Review z We convert analog image information into digital format by sampling and analog to digital conversion (Quantizing) z The more samples, the better the resolution hence, more accuracy z We can reduce resolution but we cannot create it after the fact z Once in digital form, we can easily modify the image, store it, and send it anywhere in the world!

Lecture 16 Digital Video !

Digital Video Introduction Video Timeline ‘ 60’s -70’s ’ 70’s 80’s 2009 2005 ‘ 90’s 2003 2000’ Improved Accessibility Increased Technical Complexity 2011 77

Film vs. Video • Film captures motion at 24 frames (Images) per second • Video typically operates at 30 frames per second • Video inherits many of its characteristics from broadcast television, developed in the 1930’s – 40’s

Video Starts off as Analog Information • Just as in Imaging, the original information contained in video is analog by nature – Intensity – Color – Speed / Motion (30 Frames per Second) • Digital technology allows us to convert it to bits, store it and manipulate it much easier than its analog counterpart

The Cmos Video Imager CCD 500, 000 to 20, 000 Pixels

Producing Digital Video • Video capture • Editing • Playback

Converting the Video Frame to Bits • Think of Video Frames as individual Images, stacked front to back • 110001111110000001111100 0000011111000 000000

DVD – Digital Versatile Disk • Up to 133 minutes of medium resolution video, with 720 dots of horizontal resolution X 480 dots of vertical resolution (The video compression ratio is typically 40: 1 using MPEG-2 compression. ) • Soundtrack presented in up to eight languages using 5. 1 channel Dolby digital surround sound • 4. 7 Gb of storage total per disk

Blu-Ray High Definition DVD • 10 times the capacity of std DVD • Higher resolution: – Up to 1920 X 1080 dots of resolution • Up to 50 GB of storage! • Uses a blue laser as opposed to a red one (shorter wavelength) • The current high def standard

Video Aspect Ratios

Advantages and Disadvantages of Digital Video • Advantages – – – Scalable to different playback systems Random access to frames Easy to Edit More playback options Potential for interactivity • Disadvantages – High playback and storage requirements • ( Who Cares!!)

But, there is soooo much data! • If we didn’t have a way to efficiently compress all of the information in the video frames, we would quickly run out of cost effective storage capacity for consumer applications. • Thus the need for Compression/Decompression

Digital Compression Concepts • Compression techniques are used to replace a file with another that is smaller • Decompression techniques expands the compressed file to recover the original data -- either exactly or in facsimile • A pair of compression/decompression techniques that work together is called a codec for short

Redundancy • Data compression is possible because most messages (images, etc. ) are redundant, and they can theoretically be reconstructed from a smaller set of bits.

Types of Data Compression We can divide up data-compression techniques in many different ways: • Lossy as opposed to lossless compression • Syntactic as opposed to semantic compression.

Assumptions • One way to look at data compression techniques is to ask what fact about the world they assume. • Syntactic techniques make very broad assumptions, semantic techniques can depend on very specific ones.

Run Length Encoding (RLE) • Achieves modest savings with a Syntactic method • Based on the assumption that redundancy is is present in certain repetitions of ASCII characters or numbers • ABBCCDDDDDEEFGGGGG becomes • ABBCCD#9 EEFG#5

Image Compression • The basic assumption of image compression is that pixel intensity values do not change much between neighboring pixels. • So record, say, the center pixel, and work out in a spiral. For each new pixel, just record the difference between it and the previous one.

JPEG • JPEG is set of lossy image compression standards. • JPEG combines a lossy scheme much like the one we just described, and then further compresses the data using a lossless scheme. If we have a long string of 0’s (no change) this could be represented by a pointer back to a previous such string or the use of Run Length Encoding • JPEG results in some loss of detail due to averaging as well as slight discoloration

Video Compression: Coping with Large Files • Video Compression is an encoding process that filters the original file in several successive stages • Without powerful compression we would NOT be able to produce CDs, DVDs, or Video Downloads over the Internet

Types of Codecs • Codecs that upon decompression always reproduce the original file exactly are called lossless codecs • Codecs that reproduce only an approximation of the original file upon decompression are called lossy codecs • Codecs that take approximately the same amount of time to compress and decompress a file are referred to as symmetric codecs • By contrast, codecs that feature simple fast decompression but significantly slower compression are called asymmetric codecs

Codec Methods • Syntactic encoding methods attempt to reduce the redundancy of symbolic patterns in a file without any regard to the type of information represented • Semantic methods consider special properties of the type of information represented to reduce nonessential information in a file • Hybrid methods combine both syntactic and semantic methods

Compressing Video • Video compression employs both spatial and temporal compression techniques – spatial techniques compress individual frames – temporal methods compress data in frames over time • Quick. Time and AVI (Audio Video Interleaved) are two popular (and incompatible with each other) compression formats used on PCs

Temporal Compression in Video • Lossy strategies for eliminating redundancy of information between frames employ temporal compression -- referred to as interframe compression • Sequence of frames are considered together – key frames – difference frames

Other Brute Force Methods for Reducing Demands • Frame rate adjustment – slow it from 30 to 24 fps • Lower resolution on individual frames – sometimes hard to notice by average viewer

MPEG 2 - The Mother of all Video Compression!…. so far • Uses : Temporal and Spatial Redundancy • Basically it predicts what subsequent frames of video are going to be based on previous and future frames • It encodes that knowledge such that only one out of 12 frames has a complete set of digital binary information…. the others have a combo of binary and vector information • 40: 1 Compression Ratio…. . makes DVDs possible

The Desktop Video System Basic Components • • • Analog Source Video Capture Card CPU Secondary Storage Monitor Edit and Playback Control

Editing Digital Video • Clip Logging • Assembling • Transitions – dissolves – wipes, etc. • Rotoscoping – Frame Editing (Digital Effects) • Compositing – keying – titling

Compositing…. . First we have a Mountain

Plane

Mountain and Plane…. . Together !!

Digital Video The Entire Process Illustrated

Video Resolution • Standard definition video was typically delivered at 440 X 320 or 720 X 480 depending upon whether it was broadcasted, stored on VHS videotape or standard DVD • High definition video is delivered at: 1280 × 720 pixels Or 1920 × 1080 pixels 3 -5 fold increase in pixel resolution…lot’s more data… How is it possible that we can afford to transmit this over cable, satellite and over the air given this drastic increase in resolution

Digital Cinema • Has replaced traditional film in all major movie theatres • Movies are shipped in encrypted memory packs or downloaded to theatres • Ensures that every viewing is at the same level of quality • Prevents counterfeiting

So What does Digital Video make possible? • Anyone can produce, direct, shoot, edit and publish a hi-def video • Portability • Self publishing over the net • Video on Demand – Downloading – Streaming – Purchasing And it gets cheaper every day!

Summary • Digital video is: – scalable – allows unlimited editing – has interactive potential • Digital video can be produced with desktop systems • Flexible editing and playback options are major advantages • Storage requirement is biggest challenge – But, Remember Moore’s Law !!

• Do Not Try This Unsupervised • Don’t Be Afraid of the Technology • Take the Plunge! 114

Digital Video 101 Digital Video Workflow Step 1: Produce a Great Video Step 2: Encode Files Step 3: Store Files Step 4: Deliver a Great Video to Any Device 115

Questions? 116