Advanced Game Development with the Mobile 3 D

Advanced Game Development with the Mobile 3 D Graphics API Tomi Aarnio, Kari Pulli Nokia Research Center 1 | 2004 Java. One. SM Conference | Session TS-2121 java. sun. com/javaone/sf

Gain an insight to the Mobile 3 D Graphics API and take your games to the next level 2 | 2004 Java. One. SM Conference | Session TS-2121

Prerequisites • Fundamentals of 3 D graphics • Open. GL® or some other modern 3 D API • Java™ technology, preferably MIDP 3 | 2004 Java. One. SM Conference | Session TS-2121

Objectives • Get a quick overview of the whole API • Gain a deeper understanding of selected topics • Learn practical tricks not found in the spec 4 | 2004 Java. One. SM Conference | Session TS-2121

Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation 5 | 2004 Java. One. SM Conference | Session TS-2121

Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation 6 | 2004 Java. One. SM Conference | Session TS-2121

Mobile 3 D Graphics API (M 3 G) • Also known as JSR-184 • Designed for mobile devices ─ Primarily CLDC / MIDP ─ But also CDC Mobile 3 D Graphics API Graphics Hardware 7 | 2004 Java. One. SM Conference | Session TS-2121 Midlets MIDP Java Virtual Machine (CLDC 1. 1)

Overcome the performance barrier Native (C/C++) vs. Java on mobiles Benchmarked on an ARM 9 processor 8 | 2004 Java. One. SM Conference | Session TS-2121

Leverage hardware acceleration • Benefits for developers ─ Mask hardware differences (3 D HW, FPU, DSP, …) ─ Exploit any new hardware automatically • Benefits for hardware vendors ─ Gives a concrete target which features to offer 9 | 2004 Java. One. SM Conference | Session TS-2121

Speed up development cycles • Separate content from code ─ ─ Quick prototyping without programming Artists create looks and behavior Export all in a binary file format Load the file, populate scene graph • Commonly needed functionality built in ─ High-level functionality such as scene graph, keyframe animation 10 | 2004 Java. One. SM Conference | Session TS-2121

Why a new standard? • Open. GL (ES) is too low-level ─ Lots of Java code needed for simple things • Java 3 D™ API is too bloated ─ A hundred times larger than M 3 G ─ Does not fit together with MIDP ─ Tried and failed, but… • Now knew what we wanted! ─ Basic Java 3 D™ ideas: nodes, scene graph ─ Add file format, keyframe animation ─ Remain compatible with Open. GL ES 11 | 2004 Java. One. SM Conference | Session TS-2121

Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation 12 | 2004 Java. One. SM Conference | Session TS-2121

Key classes Graphics 3 D Loader 3 D graphics context Performs all rendering Can load individual objects and entire scene graphs (M 3 G and PNG files) Scene graph root node World 13 | 2004 Java. One. SM Conference | Session TS-2121

Graphics 3 D • Contains global state ─ Target surface, viewport, depth buffer ─ Camera, light sources ─ Rendering quality hints • Each renderable object has its own local state ─ Geometry and appearance (material, textures, etc. ) ─ Transformation relative to parent or world 14 | 2004 Java. One. SM Conference | Session TS-2121

Graphics 3 D: Rendering modes • Retained mode ─ Render a scene graph, rooted by the World ─ Take the Camera and Lights from the World • Immediate mode ─ Render a branch or an individual node at a time ─ Explicitly give the Camera and Lights to Graphics 3 D 15 | 2004 Java. One. SM Conference | Session TS-2121

Graphics 3 D: How-To-Use • Bind a target to it, render, release the target • Tip: Never mess with a bound target • Tip: Graphics 3 D is a singleton (threads!) void paint(Graphics g) { my. Graphics 3 D. bind. Target(g); my. Graphics 3 D. render(world); my. Graphics 3 D. release. Target(); } 16 | 2004 Java. One. SM Conference | Session TS-2121

Graphics 3 D: Rendering targets Can render to textures also World M 3 G Graphics 3 D Image 2 D Graphics Canvas Image Custom. Item 17 | 2004 Java. One. SM Conference | Session TS-2121 MIDP

Things to keep in mind • Everything is synchronous ─ A method returns only when it’s done ─ No separate thread for renderer or loader • There are no callbacks ─ No abstract methods, interfaces, or events ─ Tip: Do not even try to override any methods • Scene update is decoupled from rendering ─ render : draws the scene, no side-effects ─ animate : updates the scene to the given time ─ align : applies auto-alignments, e. g. , billboards 18 | 2004 Java. One. SM Conference | Session TS-2121

“Hello, World” A simplified animation player import javax. microedition. midlet. MIDlet; javax. microedition. lcdui. Display; javax. microedition. lcdui. game. Game. Canvas; javax. microedition. m 3 g. *; public class Player extends MIDlet { public void pause. App() {} public void destroy. App(boolean b) {} } 19 public void start. App() { Player. Canvas canvas = new Player. Canvas(true); Display. get. Display(this). set. Current(canvas); try { canvas. run(); } catch (Exception e) {} notify. Destroyed(); } | 2004 Java. One. SM Conference | Session TS-2121

“Hello, World” class Player. Canvas extends Game. Canvas { Player. Canvas(boolean suppress){super(suppress); } } 20 public void run() throws Exception { Graphics 3 D g 3 d = Graphics 3 D. get. Instance(); World w = (World) Loader. load("/file. m 3 g")[0]; long start, elapsed, time = 0; while (get. Key. States() == 0) { start = System. current. Time. Millis(); g 3 d. bind. Target(get. Graphics()); try { w. animate(time); g 3 d. render(w); } finally { g 3 d. release. Target(); } flush. Graphics(); elapsed = System. current. Time. Millis()-start; time += (elapsed < 100) ? 100 : (int)elapsed; if (elapsed < 100) Thread. sleep(100 -elapsed); } } | 2004 Java. One. SM Conference | Session TS-2121

Demo Animation playback 21 | 2004 Java. One. SM Conference | Session TS-2121

Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation 22 | 2004 Java. One. SM Conference | Session TS-2121

Renderable Objects Mesh Sprite 3 D 23 | 2004 Java. One. SM Conference | Session TS-2121 Made of triangle strips Base class for meshes (two subclasses) 2 D image placed in 3 D space Good for labels, etc.

Mesh Composed of submeshes • Common buffer of vertex data • Submesh has triangle strip indices to vertices • One Appearance for each submesh Mesh Vertex. Buffer Index. Buffer Appearance 24 | 2004 Java. One. SM Conference | Session TS-2121 coordinates normals colors texcoords

Appearance components Material Compositing. Mode Polygon. Mode Fog Texture 2 D 25 | 2004 Java. One. SM Conference | Session TS-2121 Material colors for lighting Can track per-vertex colors Blending, depth buffering Alpha testing, masking Winding, culling, shading Perspective correction hint Fades colors based on distance Linear and exponential mode Texture matrix, blending, filtering Multitexturing: One Texture 2 D for each unit

Sprite 3 D 2 D image with a position in 3 D space • Scaled mode for billboards, trees, etc. • Unscaled mode for text labels, icons, etc. Sprite 3 D 26 Appearance Compositing. Mode Image 2 D Fog | 2004 Java. One. SM Conference | Session TS-2121

Rendering tricks • Use layers to impose rendering order ─ Appearance contains a layer index (integer) ─ Defines a global ordering for submeshes and sprites ─ Tip: Disable z-buffering for sky boxes, use layers • Tip: Maximize triangle strip length ─ Even if it requires adding degenerate triangles ─ Better benefits from vertex caching 27 | 2004 Java. One. SM Conference | Session TS-2121

Example: Rotating cube Define raw data for a cube // Corners of a cube as (X, Y, Z) triplets static short[] cube. Vertices = { -1, 1, 1, -1, -1, -1, }; 1, 1, 1, -1 // A color for each corner as an (R, G, B) triplet static byte[] cube. Colors = { (byte) 255, (byte) 0, // red … (byte) 0, (byte) 255, (byte) 0, // green }; // Define the cube as a single triangle strip static int[] indices = { 6, 7, 4, 5, 1, 7, 3, 6, 2, 4, 0, 1, 2, 3 }; static int[] strip. Len = { 14 }; 28 | 2004 Java. One. SM Conference | Session TS-2121

Example: Rotating cube Copy raw data into Objects // Fill in Vertex. Arrays from short[] and byte[] int num. Vertices = vertices. length/3; Vertex. Array pos = new Vertex. Array(num. Vertices, 3, 2); Vertex. Array col = new Vertex. Array(num. Vertices, 3, 1); pos. set(0, num. Vertices, cube. Vertices); col. set(0, num. Vertices, cube. Colors); // Attach the Vertex. Arrays to a Vertex. Buffer // Note the scale (1. 0) and bias (0, 0, 0) vertices = new Vertex. Buffer(); vertices. set. Positions(pos, 1. 0 f, null); vertices. set. Colors(col); // Fill in the triangle strip triangles = new Triangle. Strip. Array(cube. Indices, strip. Len); // Create a Mesh with default Appearance cube = new Mesh(vertices, triangles, new Appearance()); 29 | 2004 Java. One. SM Conference | Session TS-2121

Example: Rotating cube Set up a Camera cam = new Camera(); // 60 -degree field of view, screen aspect ratio // Near clipping plane at 1. 0, far plane at 1000 float aspect = (float) get. Width() / (float) get. Height(); cam. set. Perspective(60. 0 f, aspect, 1. 0 f, 1000. 0 f); // Place the camera at z=5. 0 in world space // View vector is along the negative Z axis transform = new Transform(); transform. post. Translate(0. 0 f, 5. 0 f); g 3 d = Graphics 3 D. get. Instance(); g 3 d. set. Camera(cam, transform); 30 | 2004 Java. One. SM Conference | Session TS-2121

Example: Rotating cube Clear the buffers, render, animate g 3 d = Graphics 3 D. get. Instance(); g 3 d. bind. Target(get. Graphics()); try { g 3 d. clear(null); g 3 d. render(cube, transform); } finally { g 3 d. release. Target(); } x. Angle += 1; y. Angle += 2; z. Angle += 3; transform. set. Identity(); transform. post. Rotate(x. Angle, 1. 0 f, 0. 0 f); transform. post. Rotate(y. Angle, 0. 0 f, 1. 0 f, 0. 0 f); transform. post. Rotate(z. Angle, 0. 0 f, 1. 0 f); 31 | 2004 Java. One. SM Conference | Session TS-2121

Demo Rotating cube 32 | 2004 Java. One. SM Conference | Session TS-2121

Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation 33 | 2004 Java. One. SM Conference | Session TS-2121

Add bitmap graphics • 3 D graphics is mostly vector graphics ─ Mathematical shape definitions ─ Define camera, lights, simulate ─ “Computer graphics look”, pretty boring • Bitmap graphics adds visual richness ─ Background images ─ Texture maps on surfaces ─ Sprites 34 | 2004 Java. One. SM Conference | Session TS-2121

Texturing tricks • Tip: Use light maps for lighting effects ─ Usually faster than per-vertex lighting ─ Use luminance textures, not RGB ─ Multitexturing is your friend • Tip: Use the perspective correction hint ─ Almost always faster than adding more vertices ─ But first check if the implementation supports it! 35 | 2004 Java. One. SM Conference | Session TS-2121

More bitmap tricks • Tip: Use background images ─ Can be scaled, tiled and scrolled very flexibly ─ Generally much faster than sky boxes or similar • Tip: Use sprites as impostors, particles ─ Generally (much) faster than textured quads ─ Unscaled mode is (much) faster than scaled • Tip: Load your images with the Loader ─ Loader. load(“/img. png”) outputs Image 2 D ─ Going through MIDP Image is a waste of memory 36 | 2004 Java. One. SM Conference | Session TS-2121

Demo Backgrounds and texturing 37 | 2004 Java. One. SM Conference | Session TS-2121

Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation 38 | 2004 Java. One. SM Conference | Session TS-2121

The scene graph Actually, it’s just a tree World Group Camera Skinned. Mesh Morphing. Mesh Not allowed! Group 39 | 2004 Java. One. SM Conference | Session TS-2121 Mesh Sprite Light

How to find an object in the scene World 0 User. ID defaults to 0 0 Background find(4) Group 1 Image 2 D 6 Group Mesh 2 44 Light 3 Camera 4 User. IDs not necessarily unique 5 Vertex. Buffer Mesh 7 Mesh 8 6 Vertex. Array 9 Appearance 10 Texture 2 D 1 Diagram courtesy of Sean Ellis, Superscape 40 | 2004 Java. One. SM Conference | Session TS-2121 Index. Buffer 11 Image 2 D

Node transformations • From this node to the parent node ─ Ignored for the root node • Composed of four parts ─ ─ World Translation T Orientation R Non-uniform scale S Generic 3 x 4 matrix M Group • Composite: C = T R S M Group 41 | 2004 Java. One. SM Conference | Session TS-2121 C C C Sprite C Mesh C C

Node transformations • Tip: Keep the transformations simple ─ Favor the T R S components over M ─ Avoid non-uniform scales in S • Tip: How to scale along an arbitrary axis ─ There is no direct support (C = T R N-1 S N M) ─ Method 1: R’ = R N-1 and M’ = N M ─ Method 2: Use an extra Group node • Tip: How to rotate about an arbitrary point ─ Again, no direct support (C = T P-1 R P S M) ─ Method 1: Use an extra Group node ─ Method 2: If S is identity, combine into T and M 42 | 2004 Java. One. SM Conference | Session TS-2121

Example: Set up a hierarchy private … cube 1 = cube 2 = cube 3 = Group group 1, group 2; Mesh cube 1, cube 2, cube 3; new Mesh(cube. VB, triangles, appearance); group 1 = new Group(); group 2 = new Group(); group 1. add. Child(cube 1); group 1. add. Child(cube 2); group 1. add. Child(group 2); group 2. add. Child(cube 3); group 1 group 2 cube 3 43 | 2004 Java. One. SM Conference | Session TS-2121 cube 2

Animate and render the hierarchy g 3 d. render(group 1, null); … cube 1. set. Orientation( y. Angle, 0. 0 f, 1. 0 f, 0. 0 f ); cube 1. set. Translation( 0. 0 f, 2. 0 f, 0. 0 f ); cube 2. set. Orientation(-y. Angle, 0. 0 f, 1. 0 f, 0. 0 f ); cube 2. set. Translation( 0. 0 f, -2. 0 f, 0. 0 f ); group 2. set. Orientation(-y. Angle, 0. 0 f, 1. 0 f, 0. 0 f ); group 2. set. Translation( -2. 0 f, 0. 0 f ); cube 3. set. Orientation( z. Angle, 0. 0 f, 1. 0 f ); cube 3. set. Translation( -2. 0 f, 0. 0 f ); cube 3. set. Scale( 0. 5 f, 0. 5 f ); 44 | 2004 Java. One. SM Conference | Session TS-2121

Demo Hierarchic transformations 45 | 2004 Java. One. SM Conference | Session TS-2121

Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation 46 | 2004 Java. One. SM Conference | Session TS-2121

Morphing. Mesh For vertex morphing animation • A base mesh and several morph targets • Result = weighted sum of morph deltas • Can morph any vertex attribute ─ ─ 47 Vertex positions Colors Normals Texture coordinates | 2004 Java. One. SM Conference | Session TS-2121

Morphing. Mesh Example: Animating a rabbit’s face Base 48 Target 1 eyes closed | 2004 Java. One. SM Conference | Session TS-2121 Target 2 mouth closed Animate eyes and mouth independently

Skinned. Mesh Articulated characters without cracks at joints • Stretch a mesh over a hierarchic “skeleton” ─ The skeleton consists of scene graph nodes ─ Each node (“bone”) defines a transformation ─ Each vertex is linked to one or more bones 49 | 2004 Java. One. SM Conference | Session TS-2121

Skinned. Mesh Weighted skinning Neutral pose, bones at rest 50 | 2004 Java. One. SM Conference | Session TS-2121

Skinned. Mesh Weighted skinning Bone B rotated 90 degrees 51 | 2004 Java. One. SM Conference | Session TS-2121

Skinned. Mesh Example: Animating an arm No skinning 52 Local skinning one bone per vertex | 2004 Java. One. SM Conference | Session TS-2121 Smooth skinning two bones per vertex

Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation 53 | 2004 Java. One. SM Conference | Session TS-2121

Animation: Relevant classes 54 Keyframe. Sequence Storage for keyframes Defines interpolation mode Animation. Controller Controls the playback of one or more animations Animation. Track A link between sequence, controller and target Object 3 D Base class for all objects that can be animated | 2004 Java. One. SM Conference | Session TS-2121

Keyframe. Sequence Keyframe is a time and the value of a property at that time Can store any number of keyframes Several keyframe interpolation modes Can be open or closed (looping) v t Diagram courtesy of Sean Ellis, Superscape 55 | 2004 Java. One. SM Conference | Session TS-2121 sequence time

Animation. Controller 0 00 Can control several keyframed animations together Determines relationship between world time and sequence time sequence d time 0 s t t Diagram courtesy of Sean Ellis, Superscape 56 | 2004 Java. One. SM Conference | Session TS-2121 d d world time

Animation Call to animate(world. Time) Object 3 D Apply value to animated property Calculate sequence time from world time Identifies animated property on this object Animation. Controller Animation. Track Relates animation controller, keyframe sequence, and object property together. Keyframe. Sequence 0 sequence time d v Look up value at this sequence time Diagram courtesy of Sean Ellis, Superscape 57 | 2004 Java. One. SM Conference | Session TS-2121 s

Animation • Tip: You can read back the animated values ─ Use the animation engine for anything you want ─ Much faster than doing interpolation in Java ─ Especially in case of quaternions and splines 58 | 2004 Java. One. SM Conference | Session TS-2121

Example: Set up keyframes Keyframe. Sequence create. Keyframe. Sequence() { // 10 keys, 3 components, spline interpolation Keyframe. Sequence ks; ks = new Keyframe. Sequence(10, 3, Keyframe. Sequence. SPLINE); // x grows linearly, float[] tb = { 0. 0 f, tb[0] = -4. 0 f; tb[1] tb[0] = -3. 0 f; tb[1] … tb[0] = 5. 0 f; tb[1] } 59 y wiggles up-down, z remains 0. 0 f, 0. 0 f }; = 0. 0 f; ks. set. Keyframe(0, 0, tb); = 2. 0 f; ks. set. Keyframe(1, 10, tb); = 0. 0 f; ks. set. Keyframe(9, 90, tb); ks. set. Duration(100); // sequence duration in local time ks. set. Valid. Range(0, 9); // all 10 keyframes are valid ks. set. Repeat. Mode(Keyframe. Sequence. LOOP); return ks; | 2004 Java. One. SM Conference | Session TS-2121

Example: Set up controller = new Animation. Controller(); // The animation is active between these world times controller. set. Active. Interval(0, 1500); // Set speed, scale “around” given time (0) controller. set. Speed(2. 0 f, 0); // Create animation track with keyframes & controller ks = create. Keyframe. Sequence(); at = new Animation. Track(ks, Animation. Track. TRANSLATION); at. set. Controller(controller); // Attach it to our cube. add. Animation. Track(at); 60 | 2004 Java. One. SM Conference | Session TS-2121

Demo Keyframe animation 61 | 2004 Java. One. SM Conference | Session TS-2121

Summary • M 3 G is the 3 D API for J 2 ME™ technology ─ Will be supported in millions of devices • Not just an API ─ M 3 G also defines a binary file format • M 3 G is flexible ─ Using the immediate mode gives you full control ─ High-level features make your code faster and smaller 62 | 2004 Java. One. SM Conference | Session TS-2121

Demo 3 D snowboarding 63 | 2004 Java. One. SM Conference | Session TS-2121

For More Information • M 3 G specification www. forum. nokia. com/java/jsr 184 • SDKs, additional documentation www. forum. nokia. com/ (click “Series 40”) developer. superscape. com/ (registration needed) • M 3 G content creation tools www. discreet. com/ www. superscape. com/ 64 | 2004 Java. One. SM Conference | Session TS-2121

Q&A Thanks: Kari Kangas, Sean Ellis, Nokia M 3 G team, JSR-184 Expert Group 65 | 2004 Java. One. SM Conference | Session TS-2121

Advanced Game Development with the Mobile 3 D Graphics API Tomi Aarnio, Kari Pulli Nokia Research Center 66 | 2004 Java. One. SM Conference | Session TS-2121 java. sun. com/javaone/sf