Android Media Player Development Suyash Gupta Android Developer

Android Media Player Development Suyash Gupta Android Developer Talentica Software

Agenda 1. Introduction 2. Characteristics of Video streams 3. Android Media Player Development 4. Advanced Media Player Development

1. Introduction • Video - Most Popular and common way of entertainment. • Boom in mobile market in past 5 years. • 3 g services revolutionized the telecommunication industry.

1) Video - Journey So Far ● ● Video technology was first developed for (CRT) televisions. Video recorders were sold for $50, 000 in 1956, and videotapes cost $300 per one-hour reel. ● Sony introduced VCR in 1971. ● Then came CD's in 1997. ● Followed by blue ray disc in 2006. ● Computers to capture, store, edit and transmit video clips. ● Smartphones users grew to a increase to a limit of 80%. ● Video watching became very user friendly.

Note: In Real-time entertainment the major share is made by Youtube, Netflix and Pandora.

2. Characteristics of video streams a) Frame rate b) Interlaced VS progressive c) Aspect ratio d) Color depth e) Digital video f) Video compression method (digital only) g) Containers for Codecs

a) Frame Rate The number of still pictures per unit of time of video.

a) Frame Rate • To identify each frame individually min 10 -12 FPS. • Early silent films had a frame rate from 14 to 24 FPS but it was perceived as jerky motion. • There are three main frame rate standards in the TV and digital cinema business: 24 p, 25 p, and 30 p. • 48 p is a progressive format and is currently being trailed in the film industry. (The Hobbit)

b) Video can be interlaced or progressive. Interlaced video: ● ● ● Every frame has horizontal lines. Each frame is divided into 2 fields. Odd field and even field. Interlacing is a technique of doubling the FR within the limitations of a narrow bandwidth. Image Detail is sacrificed.

b) Interlaced and progressive frames Odd Field Even Field

b) Interlaced video fields

b) Interlace video Frame

b) Progressive video ● ● Progressive is a method for displaying, storing or transmitting moving images. Each Frame resembles an image. Progressive scan is used in most CRTs used as computer Monitors. It is also becoming increasingly common in highend Television equipment, which is often capable of performing de-interlacing so that Interlaced video can still be viewed.

b) Progressive video Frame

b) Progressive Video Advantages of progressive scan include: 1. Better quality of video. 2. No flickering of narrow horizontal patterns. 3. Simpler video processing equipment. 4. Easier compression than interlaced.

c) Aspect ratio ● ● Aspect ratio describes the dimensions of video screens. Traditional television screen is 4: 3. High definition televisions(720 p and 1080 p) 16: 9. AR 1. 85: 1 and 35 mm US and UK widescreen standard for theatrical film.

c) Some common aspect ratios

d) Color depth The number of bits used to indicate the color of a single pixel, in a video frame buffer. Bit is a storage value of a pixel. 1 -bit color (2^1 = 2 colors): monochrome, black and white 2 -bit color (2^2 = 4 colors): CGA, gray-scale early Ne. XTstation, color Macintoshes ect. 3 -bit color (2^3 = 8 colors)

d) Color depth 6 -bit color (2^6 = 64 colors): 8 -bit color (2^8 = 256 colors): most early color Unix workstations, VGA at low resolution, video games. 16 -bit color (2^16 = 32768 colors): some Silicon Graphics systems True color supports 24 -bit for three RGB colors.

d) Color depths examples 1 bit (2 colors) 2 bits (4 colors) 4 bits (16 colors)

d) Color Depths 8 bits (256 colors) 24 bits (16, 777, 216 colors, "truecolor")

e) Digital Video ● ● Analog video stores video signal on magnetic tape. When copies are made of an Analog signal, degradation is introduced due to loss of video signal in the copying. ● Whereas in digital data every bit has a binary value. ● So copying is 100% precise. Analog solution to video recording Digital video storage medium

e) Size of raw digital video Digital Video in its raw form can be huge. An example video can have a duration of (3600 sec), a frame size of 640 x 480 (Wx. H) at a color depth of 24 bits and a frame rate of 25 fps. ● ● pixels per frame = 640 * 480 = 307, 200 bits per frame = 307, 200 * 24 = 7, 372, 800 = 7. 37 Mbits bit rate (BR) = 7. 37 * 25 = 184. 25 Mbits/sec video size (VS) = 184 Mbits/sec * 3600 sec = 662, 400 Mbits = 82, 800 Mbytes = 82. 8 Gbytes

e) Issues Leads to two interrelated problems: 1) Large storage requirements. 2) High Bandwidth requirement during playback and recording.

f) Video Compression 1. Conversion of data to a format that requires fewer bits. 2. The inverse process is known as decompression. 3. Decompression may or may not yield an exact copy of the original data.

f) Video Codec A codec is a compression algorithm, used to reduce the size of a stream. The compression is usually lossy. • • Lossless compression: It is a class of data compression algorithms that allows the original data to be perfectly reconstructed from the compressed data. “Lossy" compression is a data encoding method that compresses data by discarding (losing) some of it.

f) Comparison of lossy compression 80% compression 96% compression

f) Different Codecs MPEG-1(1993)(Moving Pictures Expert Group) [Old, supported by everything (at least up to 352 x 240)] MPEG-2 codecs A good quality of compression. This is what is used for DVD. MPEG-4 (1998)AVC (Div. X codec) A good quality compression with maintaining the quality of video. H. 263/MPEG-4 Part 2 codecs Div. X Pro Codec: Compresses video down to very small file sizes- excellent for web use but loses quality. Thats why not used for the television purpose Xvid: Free/open-source implementation of MPEG-4 ASP, originally based on the Open. Div. X project. Similar as Div. X Pro Codec.

FFmpeg MPEG-4: open-source libavcodec library Used by default for decoding or encoding in many open-source video players such as MXPlayer, VLC, MPlayer and GStreamer. WMV (Windows Media Video): Microsoft's family of proprietary video codec designs including WMV 7, WMV 8, and WMV 9. Google (On 2) codecs: VP 6, VP 6 -E, VP 6 -S, VP 7, VP 8 VP 9 : Proprietary high definition video compression formats and codecs developed by On 2 Technologies

g) Containers for Codecs The container describes the structure of the file: where the various pieces are stored and how they are made. I) AVI (standard Microsoft Win container) (mostly used in digital cameras) II) MOV (standard Quick. Time container) Carries the widest range of codecs III) MPEG-2 TS (acronym of Transport Stream, standard container for digital broadcasting)

g) Containers for Codecs IV) MP 4 (standard container for the MPEG-4 multimedia portfolio) V) Real. Media (standard container for Real. Video and Real. Audio) VI) 3 gp (used by many mobile phones)

3. Android Media Player a)Android supported formats b)Media player API and Sample Code c)Limitations of Media Player

3. a) Android supported formats Type Format / Codec Vide o Encoder H. 263 H. 264 AVC MPEG-4 SP VP 8 Decoder Details Supported File Type(s) / Container Formats • 3 GPP (. 3 gp) • MPEG-4 (. mp 4) • • • (Android 3. 0+) • Baseline Profile • 3 GPP (. 3 gp) (BP) • MPEG-4 (. mp 4) • MPEG-TS (. ts, AAC audio only, not seekable, Android 3. 0+) • (Android 4. 3+) 3 GPP (. 3 gp) • Streamable • Web. M (. webm) (Android 2. 3. 3+) only in Android • Matroska 4. 0 and above (. mkv, Android 4. 0+)

3. b) Media Player API Audio / Video Playback Record android. media. Media. Player android. media. Media. Recorder

b) Playback: Media. Player Basics Play Media From: • Resource Folder • File System Path • URL Basically • Create new instance of Media. Player • Call prepare() • Call start()

b) Playback View Playing a video File

b) Layout View <? xml version="1. 0" encoding="utf-8"? > <Relative. Layout xmlns: android="http: //schemas. android. com/apk/res/android" android: id="@+id/Relative. Layout 1" android: layout_width="match_parent" android: layout_height="fill_parent" android: orientation="horizontal" > <Surface. View android: id="@+id/surface. View 1" android: layout_width="match_parent" android: layout_height="wrap_content" android: layout_above="@+id/seek. Bar 1" android: layout_align. Parent. Top="true" android: layout_weight="0. 67" /> </Relative. Layout>

b) Media. Player Class Create Using: Get: • URI • Resource ID • Position • Duration • is. Playing Set: Output Display Looping • Data Source • Wake Mode States: • • Start Prepare Pause Release Reset Seek. To Stop Error Inherits from java. lang. Object Listeners: • Buffering Update • Completion • Error • When Prepared

b) Activity Code public class Main. Activity extends Activity implements Surface. Holder. Callback, On. Prepared. Listener, On. Error. Listener, On. Info. Listener { Media. Player media. Player; Surface. Holder surface. Holder; Surface. View player. Surface. View; String video. Src = "rtsp: //v 6. cache 1. c. youtube. com/Cj. YLENy 73 w. Ia. LQk. Ds. LHya 4 Z 9 h. MYDSANFEIJb. XYt. Z 29 v. Z 2 xl. SARSBXdhd. GNo. YKX 4 k 4 u. Bjb. Oi. UQw=/0/0/0/video. 3 gp"; //String data. Src = "http: //clips. vorwaerts-gmbh. de/big_buck_bunny. mp 4"; //int res. Id = R. raw. sample; //. mp 4 //int String data. Src = "/sdcard/music/sample_music. mp 3"; @Override protected void on. Create(Bundle saved. Instance. State) { super. on. Create(saved. Instance. State); set. Content. View(R. layout. activity_main 2); player. Surface. View = (Surface. View) find. View. By. Id(R. id. playersurface); surface. Holder = player. Surface. View. get. Holder(); surface. Holder. add. Callback(this); }

b) Activity Code @Override public void surface. Created(Surface. Holder arg 0) { try { } media. Player = new Media. Player(); media. Player. set. Display(surface. Holder); media. Player. set. Data. Source(video. Src); media. Player. set. On. Prepared. Listener(this); media. Player. set. On. Error. Listener(this); media. Player. set. Audio. Stream. Type(Audio. Manager. STREAM_MUSIC); media. Player. prepare(); } catch (Illegal. Argument. Exception e) { e. print. Stack. Trace(); } catch (Security. Exception e) { e. print. Stack. Trace(); } catch (Illegal. State. Exception e) { e. print. Stack. Trace(); } catch (IOException e) { e. print. Stack. Trace(); }

b) Activity Code @Override public void on. Prepared(Media. Player mp) { media. Player. start(); } @Override public boolean on. Error(Media. Player mp, int what, int extra) { Log. e("Main. Activity", "what="+what+" extra="+extra); return false; } @Override public boolean on. Info(Media. Player mp, int what, int extra) { Log. i("Main. Activity", "what="+what+" extra="+extra); return false; }

Media Player State Diagram

c) Limitations of Media Player • Any time media player changes state, check before selecting any action or you may fall into Illegal. State. Exception. • Error when the orientation of screen is change, even if you manage the tag screen. Orientation=“orientation” in manifest. xml • When activity goes in background and Media. Player is playing, even if you have handled on. Pause and on. Resume in some devices it goes to error state. • There are 30 to 34 types of possible error starting from -1 to -34. For e. g. Error: the video container is not valid for progressive playback (-32); Error due to resource being in wrong state to handle request (-14) Error due to general data processing(-18)

c) Limitations of Media Player • public static void play. Music(int id) { // Not a good practice Media. Player media. Player = Media. Player. create(context, id); media. Player. set. Looping(true); media. Player. start(); } // Here the media player will be removed by GC even if it is in play mode. • The error value (1, -2147483648), the '1' value corresponds to the constant in Media. Player. MEDIA_ERROR_UNKNOWN. (This might happen) -2147483648 corresponds to hexadecimal 0 x 80000000 which is defined as UNKNOWN_ERROR

4. Advanced Media Player Development a) Customization Features a) Benefits of Customization a) Drawbacks a) FFMPEG Library

a) Customization Features We can Customize the Media Player and achieve : We can enhance security. a) Playing video in chunks b) Prevent video from getting stolen. c) Encrypt the Input. Stream with Cipher. • Real Time manipulation. (Play ads in between) Adaptive Streaming. (Configurable bandwidth) • Multi Format Playing. (discussed later)

b) Benefits of Customization • Your video content is secure. • Easily applicable to other java platforms too for e. g. J 2 ME • Saved video can be played only by our media player. • Configurable bandwidth according to network speed. • Since playback is chunk by chunk you can show video ads in between them too. • Event handling during playback is possible for e. g. at what point the user stops the video or how much percentage watched.

c) Drawbacks • Decoding of Android media player is bad, may fall into lot of errors and you might not get possible solution anywhere. (Limited number of file formats) • The total chunk playback duration and the clip playback duration should match or the player will again throw errors like Chunk. Not. Found (Custom exception).

d) FFMPEG Library • Almost every android phone user is friendly with MX Player which is a pretty famous player. • MX Player is build using FFMpeg library which we will discuss later. • It is user-friendly, plays almost all formats and possesses great build-in features.

d)Features of mx player • Plays almost every movie files including. 3 gp. avi. divx . f 4 v. flv. mkv. mp 4. mpeg. mov. vob. wmv. webm. xvid and many more. • Hardware decoding (with h/w+ decoder) • Processor optimization • High speed rendering for ARM® NEON™ compliant processors. • Provides processor specific Codecs including Tegra 2 processor. • Move forward/backward (Optional) • Aspect ratio selection.

How to build FFmpeg for Android Following conditions are required to run included shell script: • Linux or Cygwin. • Latest Google NDK. Download ffmpeg source code 1)In config-ffmpeg. sh, change "NDK" variable to point NDK path. 2)Add this line NDK=$HOME/Desktop/adt/android-ndk-r 9 3)In build. sh, change "MAKE" variable to point ndk-build script of NDK directory. 4) Run the shell scripts by command. /build. sh 5) The build can take a while to finish depending on your computer speed. 6) Make ffmpeg Libraries available for Your Projects

FFFMPEG Project Live Demo

Thank You Questions? ?

References Video Characteristics: http: //en. wikipedia. org/wiki/Video Android Developer Site: http: //developer. android. com/reference/android/media/Media. Player. html FFMPEG Build: http: //www. roman 10. net/how-to-build-ffmpeg-with-ndk-r 9/