Introduction to Computing and Programming in Python A

Introduction to Computing and Programming in Python: A Multimedia Approach Chapter 13: Creating and Modifying Movies

Chapter Objectives

Movies, animations, and video …oh my! �We’re going to refer generically to captured (recorded) motion as “movies. ” �This includes motion entirely generated by graphical drawings, which are normally called animations. �This also includes motion generated by some kind of photographic process, normally called video.

Psychophysics of Movies: Persistence of Vision �What makes movies work is yet another limitation of our visual system: Persistence of vision �We do not see every change that happens in the world around us. �Instead, our eye retains an image (i. e. , tells the brain “This is the latest! Yup, this is still the latest!”) for a brief period of time. �If this were not the case, you would be aware of every time that your eye blinks because the world would “go away” for a moment.

16 frames and it’s motion �If you see 16 separate pictures in one second, and these pictures are logically sequenced, �That is, #2 could logically follow from the scene in #1. � 16 pictures of completely different things doesn’t work, �You will perceive the pictures as being in motion. � 16 frames per second (fps), 16 pictures in a second, is the lower bound for the sensation of motion.

Beyond 16 fps �Early silent pictures were 16 fps. �Motion picture standards shifted to 24 fps to make sound smoother. �Videocameras (digital video) captures 30 fps �How high can we go? �Air force experiments suggest that pilots can recognize a blurb of light in 1/200 th of a second! �Video game players say that they can discern a difference between 30 fps and 60 fps. �Bottomlines: �Generate at least 16 fps and you provide a sense of motion. �If you want to process video, you’re going to have 30 fps to process (unless it’s been modified elsewhere for you. )

Processing movies �Our frames are going to be JPEG pictures. �One JPEG file per frame. �So, if we’re going to be processing movies, we’re going to generating or processing sequences of JPEG files. �Three tools for manipulating movies <-> JPEGs �Media. Tools �Quick. Time Pro (free Quick. Time won’t do it) �Windows Movie Maker (for converting image sequences to movies)

Using Media. Tools To generate a series of frame pictures in a folder from an MPEG file. To play a folder of frame pictures and to save it as a JMV file. (JPEG Movie format. ) To play JMV or MPEG movies.

What the other tools can do �Quick. Time Pro (http: //www. apple. com/quicktime) can read a sequence of JPEG images and produce MPEG, AVI, or Quick. Time movies. �Windows Movie Maker can create WMV (Windows Media Player movies) from image sequences. �Image. Magick (open source toolkit) can also read a sequence of JPEG images and produce MPEG movies.

Quick. Time Pro: Making a Movie From Images • Open an image sequence • Choose the first image in the sequence. • Specify a frame rate • POOF! You get a movie!

Quick. Time Pro: Make images from movie �Choose “Export” from File menu. �Choose as Image Sequence. �Click “Options” to choose image format (PNG, JPEG) and frames per second. �This will save a numbered sequence of images.

Windows Movie Maker: Making a movie from images �Free with most Windows installations. �Choose “Import Pictures” and select all the images in your sequence.

Windows Movie Maker: Creating the Movie �Set the “Options” (Tools menu) so that there is a small duration between pictures. �Drag all the pictures into the timeline. �Play and export your movie!

MPEG? Quick. Time? AVI? JMV? �MPEG, Quick. Time, and AVI are compressed movie formats. �They don’t record every frame. �Rather, they record some key frames, and then store data about what parts of the screen change on intervening frames. �MPEG is an international standard, from the same people who invented JPEG. �AVI is a Microsoft standard. �Quick. Time is an Apple standard. �JMV is a file consisting of JPEG frames in an array. �All frames represented

Why do we compress movies? �Do the math: �One second of 640 x 480 pixels at 30 fps � 30 (frames) * 640 * 480 (pixels) = 9, 216, 000 pixels �With 3 bytes of color per pixel, that’s 27, 648, 000 bytes or 27 megabytes of information per second. �For a 90 minute feature movie (short), that’s 90 * 60 * 27, 648, 000 = 149, 299, 200, 000 bytes (149 gigabytes) �A DVD stores 6. 47 gigabytes of data. �So even on a DVD, the movie is compressed.

MPEG movie = MPEG frames plus MP 3 soundtrack �An MPEG movie is actually a series of MPEG frames composed with an MP 3 soundtrack. �It’s literally two files stuck together in one. �We’re not going to deal with sound movies for now. �The real challenge in doing movie processing is generating and manipulating frames.

Get the frames in order �Many tools (including os. listdir()) can process frames in order if the order is specified. �We specify the order by encoding the number of the frame into the name. �If you put in leading zeroes so that everything is the same length, the order is alphabetical as well as numerical.

Movies in JES �make. Movie. From. Initial. File(first. File) will create a movie object from the image sequence starting from that file. �play. Movie(movie) opens a movie player on the movie object. You can write out Quick. Time or AVI movies from there.

Simple Motion def make. Rect. Movie(directory ): for num in range (1 , 30): #29 frames (1 to 29) canvas = make. Empty. Picture (300 , 200) add. Rect. Filled(canvas , num * 10, num * 5, 50, red) # convert the number to a string num. Str=str(num) if num < 10: write. Picture. To(canvas , directory+"\ frame 0"+num. Str+". jpg") if num >= 10: write. Picture. To(canvas , directory+"\ frame"+num. Str+". jpg") movie = make. Movie. From. Initial. File(directory+"\ frame 00. jpg"); return movie

A Few Frames frame 00. jpg frame 02. jpg frame 50. jpg

Making and Playing the Movie >>> rect. M = make. Rect. Movie("c: \ Temp \ rect") >>> play. Movie(rect. M)

Important cool thing: You can draw past the end of the picture! �add. Text, add. Rect, and the rest of the drawing tools will work even if you go beyond the edge of the drawing. �Drawings will clip what can’t be seen in them, so you don’t get an array out of bounds error. �This is a big deal, because it means that you don’t have to do complicated math to see when you’re past the end of the drawing. � But only for the drawing functions. � If you set pixels, you’re still on your own to stay in range.

Making a tickertape def tickertape(directory, string): for num in range(1, 100): #99 frames canvas = make. Empty. Picture(300, 100) #Start at right, and move left add. Text(canvas, 300 -(num*10), 50, string) # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently num. Str=str(num) if num < 10: write. Picture. To(canvas, directory+"//frame 0"+num. Str+". jpg") if num >= 10: write. Picture. To(canvas, directory+"//frame"+num. Str+". jpg")

Playing the tickertape movie

Can we move more than one thing at once? Sure! def moving. Rectangle 2(directory ): for num in range (1 , 30): #29 frames canvas = make. Empty. Picture (300 , 250) # add a filled rect moving linearly add. Rect. Filled(canvas , num*10, num*5, 50, red) # Let’s have one just moving around blue. X = 100+ int (10 * sin(num)) blue. Y = 4*num+int (10* cos(num)) add. Rect. Filled(canvas , blue. X , blue. Y , 50, blue) # Now , write out the frame # Have to deal with single digit vs. double digit num. Str=str(num) if num < 10: write. Picture. To(canvas , directory +"// frame 0 "+ num. Str +". jpg") if num >= 10: write. Picture. To(canvas , directory +"// frame "+ num. Str +". jpg")

Moving two things at once

Moving a clip from a picture def move. Head(directory ): mark. F=get. Media. Path("blue -mark. jpg") mark = make. Picture(mark. F) head = clip(mark , 275 , 160 , 385 , 306) for num in range (1 , 30): #29 frames print. Now("Frame number: "+str(num)) canvas = make. Empty. Picture (640 , 480) # Now , do the actual copying copy(head , canvas , num*10, num *5) # Now , write out the frame # Have to deal with frame # digits num. Str=str(num) if num < 10: write. Picture. To(canvas , directory+"// frame 0"+num. Str+". jpg") if num >= 10: write. Picture. To(canvas , directory+"// frame"+num. Str+". jpg") def clip(picture , start. X , start. Y , end. X , end. Y ): width = end. X - start. X + 1 height = end. Y - start. Y + 1 res. Pict = make. Empty. Picture(width , height) res. X = 0 for x in range(start. X , end. X ): res. Y =0 # reset result y index for y in range(start. Y , end. Y ): orig. Pixel = get. Pixel(picture , x, y) res. Pixel = get. Pixel(res. Pict , res. X , res. Y) set. Color(res. Pixel , ( get. Color(orig. Pixel ))) res. Y=res. Y + 1 res. X=res. X + 1 return res. Pict Clip() function returns part of another picture. Using general copy() function we defined earlier.

Moving around Mark’s head

What if we have over 100 frames? def write. Frame(num, directory, framepict): # Have to deal with single digit vs. double digit frame numbers differently framenum=str(num) if num < 10: write. Picture. To(framepict, directory+"//frame 00"+framenum+". jpg") if num >= 10 and num<100: write. Picture. To(framepict, directory+"//frame 0"+framenum+". jpg") if num >= 100: write. Picture. To(framepict, directory+"//frame"+framenum+". jpg") This will make all our movie-making easier — it’s generally useful

Rewriting moving Mark’s head def move. Head 2(directory ): mark. F=get. Media. Path("blue -mark. jpg") This code is much easier mark = make. Picture(mark. F) to read and understand face = clip(mark , 275 , 160 , 385 , 306) with the subfunctions. for num in range (1 , 30): #29 frames print. Now("Frame number: "+str(num)) canvas = make. Empty. Picture (640 , 480) # Now , do the actual copying copy(face , canvas , num*10, num *5) # Now , write out the frame write. Frame(num , directory , canvas)

Using real photographs �Of course, we can use any real photographs we want. �We can use any of the techniques we’ve learned previously for manipulating the photographs. �Even more, we can use the techniques in new ways to explore a range of effects.

Slowly making it (very) sunset �Remember this code? �What if we applied this to create frames of a movie, but slowly increased the sunset effect? def make. Sunset(picture): for p in get. Pixels(picture): value=get. Blue(p) set. Blue(p, value*0. 7) value=get. Green(p) set. Green(p, value*0. 7)

Slow. Sunset Just one canvas repeatedly being manipulated def slowsunset(directory): canvas = make. Picture(get. Media. Path("beach-smaller. jpg")) #outside the loop! for frame in range(0, 100): #99 frames print. Now("Frame number: "+str(frame)) make. Sunset(canvas) # Now, write out the frame Not showing you write. Frame(frame, directory, canvas) write. Frame() because you know how that def make. Sunset(picture): works. for p in get. Pixels(picture): value=get. Blue(p) set. Blue(p, value*0. 99) #Just 1% decrease! value=get. Green(p) set. Green(p, value*0. 99)

Slow. Sunset frames

Fading by background subtraction def swapbg(person, bg, newbg, threshold): for x in range(1, get. Width(person)): Remember background for y in range(1, get. Height(person)): subtraction? person. Pixel = get. Pixel(person, x, y) One change here is that bgpx = get. Pixel(bg, x, y) the threshold is now an person. Color= get. Color(person. Pixel) input. bg. Color = get. Color(bgpx) if distance(person. Color, bg. Color) < threshold: bgcolor = get. Color(get. Pixel(newbg, x, y)) set. Color(person. Pixel, bgcolor)

Use the frame number as the threshold def slowfadeout(directory): bg = make. Picture(get. Media. Path("wall. jpg")) jungle = make. Picture(get. Media. Path("jungle 2. jpg")) for frame in range(0, 100): #99 frames canvas = make. Picture(get. Media. Path("wall-two-people. jpg")) print. Now("Frame number: "+str(frame)) swapbg(canvas, bg, jungle, frame) # Now, write out the frame write. Frame(frame, directory, canvas)

Slow. Fadeout

Different images, with subfunctions def swap. Back(pic 1 , back , new. Bg , threshold ): for x in range(0, get. Width(pic 1 )): for y in range(0, get. Height(pic 1 )): p 1 Pixel = get. Pixel(pic 1 , x, y) back. Pixel = get. Pixel(back , x, y) if (distance(get. Color(p 1 Pixel), get. Color(back. Pixel )) < threshold ): set. Color(p 1 Pixel , get. Color(get. Pixel(new. Bg , x, y))) return pic 1

Different images, with subfunctions def slow. Fadeout(directory ): orig. Back = make. Picture(get. Media. Path("bgframe. jpg")) new. Back = make. Picture(get. Media. Path("beach. jpg")) for num in range (1 , 60): #59 frames # do this in the loop kid = make. Picture(get. Media. Path("kid -in -frame. jpg")) swap. Back(kid , orig. Back , new. Back , num) # Now , write out the frame write. Frame(num , directory , kid)

Cool effect!

Dealing with real video �We really can’t deal with live video. �Dealing with each frame takes a lot of processing. �If you were going to process each frame as fast as it was coming in (or going out), you’d have 1/30 th of a second to process each frame! �We cheat by �Saving each frame as a JPEG image �Processing the JPEG images �Convert the frames back to a movie

The original kid-in-bg-seq movie

Let’s have Mommy “watching” �We’ll paste Barb’s head into each frame. �We’ll use os. listdir to process all the frames of the kid sequence.

Mommy. Watching import os def mommy. Watching(directory): kid. Dir="C: /ip-book/mediasources/kid-in-bg-seq" barb. F=get. Media. Path("barbara. S. jpg") barb = make. Picture(barb. F) We process each frame, and face = clip(barb , 22 , 93 , 97) copy Mommy’s head to the frame, just like we animated in a num = 0 line before onto a blank canvas. for file in os. listdir(kid. Dir ): if file. endswith(". jpg"): num = num + 1 print. Now("Frame number: "+str(num)) frame. Pic = make. Picture(kid. Dir+“/"+file) # Now , do the actual copying copy(face , frame. Pic , num*3, num *3) # Now , write out the frame write. Frame(num , directory , frame. Pic)

Mommy. Watching

Lightening a picture �I took some video of a puppet show in black light. �Very hard to see the puppets. �Your eye can pick them up, but the camera can’t. �Recall earlier discussion: Your eye can detect luminance changes that no media can replicate.

Dark-fish 2 sequence

How I did the processing �First try, lighten every pixel. �Didn’t work. �Made all the black whiter as well as the colors �No improvement in contrast �Second try, explore under Media. Tools first �Black parts are really black �Lighter parts have really low number values �So: � Look for any pixel less black than black (threshold=8) � Lighten it a couple values

Lightenfish import os def lighten. Fish(directory): framenum = 0 for framefile in os. listdir(get. Media. Path("dark-fish 2")): framenum = framenum + 1 print. Now("Frame: "+str(framenum)) if framefile. endswith(". jpg"): frame=make. Picture(get. Media. Path("dark-fish 2")+"//"+framefile) for p in get. Pixels(frame): color = get. Color(p) if distance(color, black)>8: color=make. Lighter(color) set. Color(p, color) write. Frame(framenum, directory, frame)

Original sequence again

Same frames after lightening

Putting kids on the moon �Took a video of our kids crawling past a blue sheet. �Unfortunately, did it in front of electric light, not daylight. �Not really blue. �If you chromakey against black, pants and eyeballs go away.

Code for putting kids on moon import os def kids. On. Moon(directory ): kids="C: //ip-book//mediasources//kids-blue" moon=get. Media. Path("moon-surface. jpg") back=make. Picture(moon) num = 0 for frame. File in os. listdir(kids): num = num + 1 print. Now("Frame: "+str(num)) if frame. File. endswith(". jpg"): frame=make. Picture(kids+"//"+frame. File) for p in get. Pixels(frame ): if distance(get. Color(p), black) <= 100: set. Color(p, get. Color(get. Pixel(back , get. X(p), get. Y(p)))) write. Frame(num , directory , frame)

Making underwater movies look better Before: �Water filters out red and yellow light. �We can color-correct underwater footage by increasing red and green. After:

Code for fixing underwater footage import os def change. Red. And. Green(pict , red. Factor , green. Factor ): for p in get. Pixels(pict ): Creating a useful function set. Red(p, int(get. Red(p) * red. Factor )) to make the task easier. set. Green(p, int(get. Green(p) * green. Factor )) def fix. Underwater(directory ): num = 0 dir="C: //ip -book//mediasources //fish" for frame. File in os. listdir(dir): num = num + 1 print. Now("Frame: "+str(num)) if frame. File. endswith(". jpg"): frame=make. Picture(dir+"//"+frame. File) change. Red. And. Green(frame , 2. 0 , 1. 5) write. Frame(num , directory , frame)

Building an effect from the bottom up �Notice that the underwater footage code was made cleaner and clearer through use of an extra, “helper” function. �Made the main function easier to read and shorter to write. �We can build visual effects “bottom-up” by building helper functions first, then assembling them all.

Drawing with light �Many commercials feature actors “drawing” with light. �Light beams that seem to hang in the air. �How could we do that?

Our Algorithm �The light should create high luminance pixels. 1. From frame 1, for each pixel of high luminance, copy the color to frame 2. � Now frame 2 contains the high luminance from frame 1 and from frame 2 2. Go on to frame 2 and 3, and back to step 1. �Each frame now contains the “trace” of light from all the previous frames.

Input �Having my kids draw in darkness (to make sure luminance difference is large) with flashlights and light sticks.

What do we need? First step: Compute luminance def luminance(apixel ): return (get. Red(apixel )+ get. Green(apixel )+ get. Blue(apixel ))/3. 0

Test the pieces �As we build each piece, we test it. �You don’t want to build more on top of it until you know this works! We make a small picture so that we can a pixel to known colors and check its luminance. >>> pict = make. Empty. Picture (1, 1) >>> pixel=get. Pixel. At(pict , 0) >>> white Color (255 , 255) >>> set. Color(pixel , white) >>> luminance(pixel) 255. 0 >>> black Color(0, 0, 0) >>> set. Color(pixel , black) >>> luminance(pixel) 0. 0

Is that bright enough? def bright. Pixel(apixel , threshold=100): if luminance(apixel) > threshold: return true return false This could also be written: def bright. Pixel(apixel , threshold=100): return luminance(apixel) > threshold Using a Python feature that allows you to specify an optional parameter with a default value. We can specify a threshold, but if we don’t, it will be 100.

Testing our brightness function >>> red Color (255 , 0, 0) >>> set. Color(pixel , red) >>> luminance(pixel) 85. 0 >>> bright. Pixel(pixel) 0 >>> bright. Pixel(pixel , 80) 1 >>> bright. Pixel(pixel , threshold =80) 1 >>> set. Color(pixel , white) >>> bright. Pixel(pixel , threshold =80) 1 >>> bright. Pixel(pixel) 1 >>> set. Color(pixel , black) >>> bright. Pixel(pixel , threshold =80) 0 >>> bright. Pixel(pixel) 0

Walking through the list of files import os def all. Files(from. Dir ): list. Files = os. listdir(from. Dir) list. Files. sort () return list. Files def first. File(filelist ): return filelist [0] def rest. Files(filelist ): return filelist [1: ] #returns after [1]

Testing the file list functions >>> files = all. Files("/") >>> files [’Recycled ’, ’_314109_ ’, ’bin’, ’boot ’, ’cdrom ’, ’dev’, ’etc’, ’home ’, ’initrd. img’, ’initrd. img. old’, ’lib’, ’lost+found ’, ’media ’, ’mnt’, ’opt’, ’proc ’, ’root ’, ’sbin ’, ’srv’, ’sys’, ’tmp’, ’usr’, ’var’, ’vmlinuz. old’] >>> first. File(files) ’Recycled ’ >>> rest. Files(files) [’_314109_ ’, ’bin’, ’boot ’, ’cdrom ’, ’dev’, ’etc’, ’home ’, ’initrd. img’, ’initrd. img. old’, ’lib’, ’lost+found ’, ’media ’, ’mnt’, ’opt’, ’proc ’, ’root ’, ’sbin ’, ’srv’, ’sys’, ’tmp’, ’usr’, ’var’, ’vmlinuz. old’]

Now, putting it all together! def bright. Combine(from. Dir , target ): file. List = all. Files(from. Dir) from. Pict. File = first. File(file. List) from. Pict = make. Picture(from. Dir+from. Pict. File) for to. Pict. File in rest. Files(file. List ): print. Now(to. Pict. File) # Copy all the high luminance colors from. Pict to to. Pict = make. Picture(from. Dir+to. Pict. File) for p in get. Pixels(from. Pict ): if bright. Pixel(p): c = get. Color(p) set. Color(get. Pixel(to. Pict , get. X(p), get. Y(p)), c) write. Picture. To(to. Pict , target+to. Pict. File) from. Pict = to. Pict

Final frame of a light-drawing movie

Why? �Why does movie processing take so long? �Why does sound processing seem to go so fast? �Why can Photoshop do these things faster than we can in Python? �What makes software fast, or slow? �Coming soon…