Creating the Embedded Media Processing Ecosystem Neil Trevett
Creating the Embedded Media Processing Ecosystem Neil Trevett Vice President Embedded Content, NVIDIA President, Khronos Chairman, Open. GL ES Working Group
Mobile Convergence • Handsets are becoming the predominant personal computing platform - Sophisticated media processing will be central to this revolution • 50% of phones will have multimedia hardware acceleration by 2008 - Source - Jon Peddie & Associates 100 Ms / year • Significant opportunity for games developers - Handhelds are the largest market for a 10 Ms manufactured device – ever – period / year 100 Ks / year 80’s Workstatio ns Page 3 90’ s PCs 00’s Handhelds
Media APIs Enable Market Growth • An API is a contract between hardware and software worlds - Enabling both - everyone wins • ISVs see reduced variability across multiple platforms - More software can reach market faster at a better level of functionality and quality • Hardware vendors can accelerate many applications - Adding value to their platform Hardware Provider Software Developer Hardware Provider Software Developer Khronos develops “Foundation-Level” APIs As close-to-the-metal” as possible while providing portable access to hardware acceleration. Great performance. Page 4 Good foundation for higher-level engines and
APIs Enable Mobile Acceleration Faster Performance at Higher Quality Hardware delivers at least 10 times the performance of software – even on low-cost systems with low-end CPUs POWER EFFICIENCY State of the art user interfaces Smaller screens need more advanced graphics processing per pixel Page 5 Less Power Hardware accelerators exploit parallelism in a media pipeline to give a x 10 increase in power efficiency over software
Creating Open API Standards Open Membership Any company is welcome Funded by membership dues - $5 K / year Open Standards Publicly available on web-site Royalty-free Open Standard Platform for Embedded Rich Media Acceleration Cross Platform Enabling diverse handheld and embedded markets Promoting Ecosystem Conformance tests, tools, developer materials and outreach Khronos has a PROVEN reputation for the TIMELY creation of HIGH-QUALITY, ROYALTY-FREE standards Page 6
Over 100 companies creating media authoring and acceleration standards Page 7
How Does Khronos Make Money? • It doesn’t! • Khronos is purely a non-profit organization - Funded by member dues – to cover costs • Our members make money by selling PRODUCTS enabled by standards - NOT trying to charge for the standard itself $ An open, royalty free API standard creates much larger market opportunities Selling an API would generate relatively small amounts of revenue Page 8 $ Our members cooperate to create standards – and compete in the marketplace with products that use Khronos standards
Khronos API Stack The Khronos API family provides a complete ROYALTY-FREE, cross-platform media acceleration platform Applications or middleware libraries (JSR 184 engines, Flash players, media players e AL 3 D Small footprint 3 D for embedded systems EGL Vector 2 D Low-level vector acceleration API Abstracted Access to OS Resources Fast mixed mode 2 D/3 D rendering SOUND Low-level gaming audio acceleration API Abstracted component controls Media Frameworks Component interfaces for IL codec integration Image Libraries, Video Codecs, Sound Libraries DL Accelerated media primitives for codec development Media Engines – CPUs, DSP, Hardware Accelerators etc. Khronos defines low-level, FOUNDATION-level APIs. “Close to the hardware” abstraction provides portability AND flexibility Page 9
Adoption of Khronos APIs. We Are Here Market adoption in mediaaccelerated Widespread, handsets cross-platform 100% - Open. GL ES is widespread - Open. VG is in rapid adoption - Open. MAX is being implemented - Open. SL ES is being designed availability Rapid adoption when silicon ships Silicon Designs in progress Mid-2004 Open. GL ES 1. 1 Spec release Page 10 Mid-2005 Beginning-2006 End-2006 Open. VG 1. 0 Open. Max IL 1. 0 Open. SL ES 1. 0 Spec release Time
Accelerated Vector Graphics
Vector Graphics for Mobile Devices • Vector graphics used in many popular formats such as Flash, SVG, PDF - High-quality user interfaces, screen savers, 2 D Games - Portable mapping and GPS applications, E-book readers and text packages • High-quality 2 D vector graphics use scalable Bezier curves - Path based for scaling and positioning at full quality - not polygon based - Easy porting of full quality 2 D content to different screen sizes • But all 2 D vector graphics usually run un-accelerated! - Works OK on high-performance PCs - Not effective on low-powered handset CPUs • We need to enable accelerated mobile 2 D vector graphics! Page 12
Open. VG – Accelerated Vector Graphics • Open. VG ACCELERATES existing formats – such as Flash - NOT a competitor to Flash, SVG etc. - Enables popular vector formats to run with faster performance and less power • Open. VG 1. 0 released at Siggraph 2005 - Open, royalty-free standard - Developed in just 12 months • Uses Open. GL-style syntax - Easy to learn for Open. GL developers • Will be supported by graphics silicon - Dedicated 2 D engines AND full 3 D engines • Conformance tests under construction Applications SVG/Flash/ Font Packages etc. . 2 D Hardware Acceleration Page 13
Streaming Media Portability
Open. MAX - Three Layer Solution AL ”Application Level” Media Application Portability Applications programmed using cross-vendor interfaces IL Streaming Media Infrastructure Portability Open, royalty-free standard “Integration Level” Media Graph Portability Integrate media networks using standard interconnect protocols DL “Development Level” Media Component Portability Develop portable media components using low-level media APIs More media applications available on more platforms for more end-user value! Portable and powerful media processing graphs can flexibly leverage available platform media components Component vendors can ship more advanced functionality across more processors. Media silicon vendors reduce costs and time to market Open. MAX defines three holistically designed media open standards to provide complete streaming media infrastructure portability Page 15
Open. MAX IL Example Graph • Standardized component interfaces enable flexible media graphs • Includes multi-stream synchronization Open. MAX IL Component Interfaces Enables components to be flexibly connected in any graph topology AAC Audio File Clock for AV Sync *. mp 4 / *. 3 gp File Reader MPEG 4/ H. 264 Video Decoder Audio Renderer Speakers Video Scheduler Video Renderer Display Time Data Decompressed Video Example: MPEG-4 video synchronized with AAC audio decode Page 16
Open. MAX Summary • Created with strong industry consensus and participation - ARM, ATI, Beatnik, Broadcom, Emuzed, Fraunhofer, Freescale, Infineon, Intel, Motorola, Nokia, NVIDIA, Philips, SKY Mobile. Media, Samsung, Sasken, Siemens, STMicroelectronics, Symbian, Texas Instruments • Specification is open and royalty-free using Khronos IP framework - Delivered with sample implementations and conformance tests • Available on wide variety of architectures and operating systems - To enable Open. MAX IL 1. 0 true Specification and Conformance Process Released 4 Q 05 streaming media portability Open. MAX IL 1. 0 Sample 1 Q 06 Open. MAX DL 1. 0 Specification Released Page 17 Implementation to be released 2 Q 06 Open. MAX IL 1. 0 Conformance Tests to be released Open. MAX AL 1. 0 Specification Target Release Date 3 Q 06 4 Q 06
Accelerated Embedded Audio
Open. SL ES – Solving an Audio Crisis • Many different proprietary audio APIs - Even playing a simple sound is different on different platforms • No standard way to access any available hardware - Lots of work for developers to re-write code for every platform • Need a unified native audio acceleration API - Targeting handheld devices • API for games developers - low-latency sound generation for games - Advanced audio functionality: 3 D positional audio, reverb, SP-MIDI • Cross-platform foundation for a wide range of higher-level audio APIs Application - Open. AL, Java sound APIs (JSR-135 and JSR-234) s High-level Audio Libraries Audio Hardware Page 19
Working Group Milestones Draft Specification External review First face-toface meeting 4 Q 05 1 Q 06 2 Q 06 Release Open. SL ES 1. 0 Specification 3 Q 06 4 Q 06 Strong industry quorum of leading audio hardware and software companies. Open call for participation Page 20
Industry Standard Embedded 3 D
Open. GL ES API Standard • Small-footprint subset of Open. GL - Created with the blessing and cooperation of the Open. GL ARB • Powerful, low-level API with full functionality for 3 D games - Available on all key platforms • Fully extensible - Enables vendor differentiation and exploration of new functionality Feedback and Ratification Embedded Focus Strong functionality in a small footprint e. g. <100 KB software engines ARB Workstation Focus Eliminate Redundancy Eliminate Workstation Functionality Page 22
Open. GL ES – Central to Mobile 3 D JSR 239 Defining official Java Bindings to Open. GL ES Java Application s Usable directly by applications Usable by higher-level graphics libraries “Close to the hardware” API provides portability AND flexibility Brings advanced 2 D/3 D graphics to a wide range of OS platforms Page 23 C/C++ Application s Scenegraph APIs M 3 G (JSR 184) Hardware Open. GL ES Engines Games Middlewa Engines re Libraries Software Open. GL ES Engines
Open. GL ES – Two Track Standard • Two tracks - manage mobile graphics through programmable transition - With maximized portability and minimized platform costs • Open. GL ES 2. 0 ruthlessly eliminates redundancy – just like 1. X - Deprecates all fixed functionality that can be replaced by shaders - Significant reduction in engine cost and driver complexity • Platforms can ship either or both 1. X and 2. X libraries - Cheaper, more flexible than one large driver with both fixed and programmable functions - With full backwards compatibility maintained in each track • Open. GL ES 1. X – Fixed Function Acceleration ES 1. 1 Open. GL ES 2. X does. Open. GL NOT replace Open. GL ES 1. X - For software and fixed functionality hardware - Will always need lowest -cost, non-programmable hardware for certain high. All 1. X specifications are backwards compatible volume devices Open. GL ES 2. X – Programmable Acceleration Open. GL ES 2. 0 - Vertex & pixel shaders through GLSL ES shading language - All 2. X specifications will be backwards compatible Page 26
Open. GL ES DNA Spec release Open. GL 1. 3 Streamlined Subset 2003 Open. GL 1. 5 New Functionality 2004 Open. GL 2. 0 GLSL M 3 G (JSR 184) acceleration requirement s Open. GL ES 1. 0 Eliminate redundancy and workstation functionality -> simple, streamlined API 2005 Open. GL ES 2. 0 Provisional Specification Open. GL ES 1. 1 Minimized Fixed Function Open. GL ES 1. 1 Extension Pack Enhanced functionality for emerging hardware, seamless acceleration for M 3 G, additional power management Shader programs using GLSL. Eliminate redundant fixed functionality –> simple, streamlined API. Supports remote compilation Roadmap track for programmable hardware Open. GL ES 2. X Roadmap track for fixed function Open. GL ES 1. X hardware Package of extensions that enhance functionality and reduce variability Minimize differences between 1. X and 2. X tracks to ease programmable transition Page 27
Open. GL ES Roadmap • Stability and reducing fragmentation is currently the key concern - More important than new functionality in the current phase of market development - Industry is still absorbing and implementing current Open. GL ES 2. 1 Open. GL ES 2. 2 Open. GL ES 2. 0 specifications Absorbs proven New functionality to Provisional specification finalized extensions IF needed meet PROVEN market needs Open. GL ES 2. 1 Open. GL ES accelerated products extensibility begin to ship enables new functionality to Open. GL ES 2. 0 be explored accelerated products before adoption begin to ship into the core specification Open. GL ES 1. 1 with. Open. GL ES 1. 1 will continue to be used in low-cost devices. hardware acceleration – 1. X specification will absorb proven extensions if needed “Sweet Spot” Shipping Products Page 28 2006 2007 2008
Open. GL ES – More than Handhelds • Open. GL ES will be available for Playstation 3 - Sony made public announcement at GDC in San Francisco in March 2005 - All interactive demos at E 3 were using Open. GL ES • Powerful portability for console and handset titles - Previous generation console games can be deployed on 100 s of millions of cell phones Application portability across a wide range of platforms THE POWER OF COOPERATIVE OPEN STANDARDS A growing infrastructure of Open. GL ES tools and platforms are creating one of the world’s largest opportunity for games developers Page 29
Mobile 3 D Graphics Ecosystem Synergistic Development of authoring and acceleration standards under one body. Third party tools and debuggers. g. DEBugger ES announced at GDC 2006 Khronos drives Open. GL ES roadmap to meet market needs Market demand for Handheld 3 D Industry Cooperatio n Open API Standards Conformance Tests Open. GL ES 1. 1 Conformance Tests released Aug’ 05 with peer review conformance process. Working Group is tightening conformance and Khronos is developing Open. GL ES 2. 0 tests Page 30 Great 3 D Applications Tools Benchmark s Highquality platforms Futuremark 3 DMark. Mobile 06 JBenchmark 3 D. Soon see consumer benchmark interest as on the PC?
Native Gaming Platform
Good Titles Gaining Attention • Users place a premium on compelling media-enabled handsets - Smooth 3 D gaming functionality, MP 3 audio and TV in the palm of your hand - 3 D benchmarks to enable consumer comparisons now appearing • Users pay more for premium 3 D gaming content - 50 -100% price increase for 3 D games over 2 D games is common - 3 D UI/Avatar/messaging applications will enable new levels of personalization • Multi-user content encourages higher network usage BUT – compelling media applications not happening - OTA provisioning 3 D fastapplications enough. and levels -Systemic Real-time, industry multi-player networking issues threaten to stall mobile Spider-Man 2 3 D: NY media Subway Spider-Man 2 Activision Sony Pictures Spider-Man 2: The Hero Returns Sony Pictures Page 34 2 D 3 D Software 3 D with HW
Barriers to Great Mobile Content Barrier #1 - Platform Variability Each phone supports different selection of APIs Often available APIs do not work reliably - needs workarounds Need to port to and support 100 s of platform variants “Native Content Platform” 1. A cross-platform set of native APIs for media application development Titles are failing to meet customer 2. A consistent method to expectations provision, install and run and cost more to port than to develop native C applications across Price per title is being held artificially low diverse platforms Difficult for ISVs to get return on investment Holding back development of innovative content Barrier #2 - Performance “Khronos Open Java is architected for safety not speed Development Most handsets have no way to deploy a C native title Environment” The capability of Open. GL ES silicon is not being unleashed Page 35
Open. KODE Native API Platform • Enable high-performance, source-portable, native media applications - Through a coherently designed API platform • Open. KODE collects Khronos media APIs into a single platform - Rigorous conformance testing to ensure reliable API implementations • Add “missing” APIs needed that prevent portable media applications - Input, Networking and OS resource abstraction • Functionally similar to the Direct. X platform - Except cross-platform, royalty-free and streamlined for mobile platforms • Enable a native content platform familiar to. New PCAPIs and console New APIs provide complete a abstracted coherent development 2 D/3 D 2 D developers access to: environment for source Vector - Encourage application porting and Input reduced learning curve for many portable media Networking applications Gaming Streaming developers OS Resources Audio Page 36 Media
Need for Native Deployability • Open. KODE needs to be available on maximum number of OS - For widest application portability • Smartphone OS are able to download and link native applications - Windows Mobile, Symbian, Linux • But >80% of the handheld market today uses feature phone OS - Many RTOS do not have the capability to link and load a native application • Open. KODE considering defining formats to enable a onhandset linker - Enabling feature phones to download and link Open. KODE applications • Include standardized dependency information - Enable dynamic download of Open. KODE libraries - Accelerate market ecosystem velocity Open. KODE needs to enable smart phones AND feature phones with a native media platform to provide the maximum market for Open. KODE applications Page 38
Open. KODE and Java • Open. KODE complements Java – low-level access to media acceleration - Use all Java infrastructure – provisioning, billing, key JSRs • Possibly looking to leverage OSGi and JSR 232 - For flexible component provisioning • Open. KODE needs high-efficiency Java/native process communications - Possibly will use CHAPI Java can use state-of-art Complete Java Environment media acceleration with portability across different silicon vendors Open standard APIs enable silicon vendors to export media acceleration capability Page 39 Higher- level native gaming engines and middleware JSR Bindings Media Acceleration Silicon
Open. KODE Native Deployment Over-the-air download uses standard Java or OMA provisioning and billing Interface with existing signature checking and DRM management to enable operators to control distribution domain Great 3 D Content Dependency Check Verify Link JAR (ZIP) file Contains binary file Download dependency information to enable dynamic download of required Open. KODE API binaries and application binary Page 40 Link application and Open. KODE API binaries to create single binary – works even on RTOS with no native linking capability
Possible Consortium Liaisons Device Interfaces e. g. camera OSGi – for flexible Java component provisioning KWISF WIPI 3. 0 Adoption of Open. KODE Carrier platform recommendatio ns Provisionin g, DRM, Device Manageme nt Page 41 Provisionin g, Java interfaces
Open. KODE Milestones • Aiming for Open. KODE 1. 0 release in 2006 - Keeping things as simple while delivering significant ISV benefit • Restrict scope to media applications and games - Narrow focus will enable rapid development • Maximize leverage of existing mobile infrastructure - E. g. enable flexible Java invocation of native libraries, applications and engines • Open. KODE is NOT: - An operating system, DRM scheme, provisioning scheme or a carrier certification process - But it will interface will all of the above • Working for wide industry adoption on many platforms - Java, Symbian, WIPI, Linux, Windows (through 3 rd parties), Brew, Nucleus Call for participation and contributions First F 2 F meeting 1 Q 06 Page 42 2 Q 06 Scope Agreed F 2 F April 24 th 3 Q 06 First public deliverable s 4 Q 06 1 Q 07
Why Use Khronos Standards? • Khronos is creating a complete, coherent media acceleration platform - To reduce development and deployment costs and increase market opportunity • “Foundation Level” APIs - Close to the silicon – fundamental, flexible functionality needed on every platform • Designed by industry experts - The industry leaders in media silicon, platforms and software all Khronos members • Flexible, fast-track roadmap evolution - Effective and streamlined process – specification updates every 12 months if needed • Royalty-free - Khronos is committed to generating market opportunities for its members and the industry • Any company is welcome to join Khronos! - Only $5, 000 / year membership fees Page 43
- Slides: 38
