Confidential 2009 Teleca AB Dual Display for Android

Confidential Agenda Introducing Teleca History of Multi Display Dual Display Mobile Devices Teleca’s experience

Confidential © 2009 Teleca AB Introducing Teleca is the only player in segment with comprehensive onshore/offshore global delivery footprint across all major markets We have 2000 mobile software engineers available worldwide 3

Confidential History of Multi-Monitor (1) – PC and TV © 2009 Teleca AB Dual Monitors at Work Place Picture in Picture Multi monitor use case ¡ Engineer needs as bigger screen possible to enhance productivity ¡ HW Support – two video ports, or add second video card ¡ OS Support – Adjust display properties PIP use case ¡ Two programs side–by–side and audio output of one program going into speaker and another going to headset ¡ Two tuner built into TV set OR one tuner being external source like DVD player. 5

Confidential © 2009 Teleca AB History of Multi-Monitor (2) – Mobile Phones Multi display Mobile Phone usecases ¡ The secondary display is used typically for notifications ( messages, incoming call ) and Clock ¡ HW Support – two display drivers ¡ OS Support – Adjust display properties ¡ Separate applications running on two screen Disadvantages ¡ Cannot write applications for secondary display ¡ Only one display is active at any time. ¡ Low resolutions 6

Samsung Continuum Acer Iconia © 2009 Teleca AB Confidential Dual Display Mobile Devices (1) Samsung Continuum Use cases Acer Iconia ¡ Dedicated Ticker Display for displaying Music player controls, RSS feeds, Social network site updates, Incoming call notification. ¡ Concept laptop. ¡ The idea seems to be use the second display for dynamically changing data without affecting the activities user does on main screen ¡ Used as e-Reader ¡ One of the display can become Soft. Keyboard. 8

Confidential Dual Display Mobile Devices (2) HW Support ¡ Two different displays ¡ Two different frame buffers and drivers ¡ One display virtually split into two displays ¡ Single virtual frame buffer to support two independent displays © 2009 Teleca AB SW Support ¡ Framework changes to support dual displays ¡ SDK changes to program applications specific to Dual display ¡ Currently not available, mostly proprietary 9

© 2009 Teleca AB Confidential Teleca’s experience in Dual Display for Android (1) Teleca

Confidential Teleca’s experience in Dual Display for Android (2) Teleca built Dual Display for Android with Texas Instruments Demonstrated at MWC 2009 Development Environment ¡ TI Zoom 3(OMAP 4430) with Dual Display Support (each display 864 x 480) © 2009 Teleca AB ¡ Android 2. 0 ( Éclair ) 12

Confidential Teleca’s experience in Dual Display for Android (3) - Components Android Components involved in Dual Display ¡ Applications - need to define onto which display their Activities should go. ¡ Activity Manager – launch application’s activities onto the right display. ¡ Window Manager – needs to properly handle two Activity stacks and two window stacks © 2009 Teleca AB ¡ Surface Flinger – needs to handle layer composition for two displays. ¡ Open. GL / EGL – needs to support one context for each display ¡ Linux Kernel / Driver – support for two secondary display for output and touch input 13

Confidential Teleca’s experience in Dual Display for Android (4) - Design Approaches for Window Manager Support Single Window Manager concept ¡ This concept is based on an extension of the existing Window Manager for the second display. The Window. Manager. Service would be extended to support a second stack of activities and a second stack of windows. © 2009 Teleca AB ¡ The Activity Manager would –based- on the applications preference or the preference of an Activity create the Activity on the right stack of the Window Manager. Besides the changes to the Activity. Manager the main changes need to be done in the Window. Manager. Service and the IWindow. Manager. aidl. Pros Interface to only one Window Manager No additional thread Cons Very complex 11 K lines of code Massive design changes to central Android component 14

Confidential Teleca’s experience in Dual Display for Android (5) - Design Approaches for Window Manager Support Second Window Manager ¡ The concept proposes to duplicate the Window. Manager. Service code, rename it and instantiate an instance of this which handles the Activities and Windows for the second display. © 2009 Teleca AB ¡ All Stake holders need to interface with two Window Managers 15

Confidential Pros Simple concept with only minor changes to the actual implementation of the Window. Manager. Service; Standard Window. Manager. Service requires nearly no changes © 2009 Teleca AB Easy to migrate to next version of Android Cons Synchronization of both Window. Manager. Service instances required in some parts Code duplication Increased memory usage due to a second instance of the service A number of stakeholders of Window. Manager. Service need to be modified Not too attractive solution from architecture point of view 16

Confidential Teleca’s experience in Dual Display for Android (6) - Design Approaches for Window Manager Support Second Window Manager Service Instance © 2009 Teleca AB ¡ This concept proposes to modify the Window. Manager. Service and the IWindow. Manager. aidl code so that it can be instantiated twice. Each instance of the service would be responsible for one display then. The required changes haven’t been analysed in detail as this seemed to be more effort compared to previous approach. However from an architecture point of view this seems to be preferred. ¡ The problem with this approach might be that the decision which Window. Manager. Service needs to be used might not always be possible and thus requiring modifications in the stakeholders. As a result the other approach was chosen 17

Confidential Agenda Introducing Teleca History of Multi Display Dual Display Mobile Devices Teleca’s experience in Dual Display for Android Devices © 2009 Teleca AB ¡ Specific Implementation Details QA 20

Confidential Teleca’s experience in Dual Display for Android - High Level Block Diagram Applications Home Phone Gallery App. on second display Home 2 Open. GL Application … Application Framework Activity Manager … © 2009 Teleca AB Window Manager 2 Window Manager View System Libraries Android Runtime bionic Surface Flinger Open. GL Hardware Abstraction Layer Linux Kernel Display Driver 1 Modified Display Driver 2 New Pico DLP Keyboard Driver Not feasible Touch Input 1 Touch Input 2 … 21

Confidential Teleca’s experience in Dual Display for Android Changes across the Android Stack Applications ¡Extensions to the Android. Manifest. xml The Android. Manifest. xml was extended to support the attribute “android: display” for the <application> and the <activity> tag. The possible values are integer values, usually only used when the application or an Activity of the application wants to go on the © 2009 Teleca AB second screen ¡Sample: <application android: label="@string/app_label" android: icon="@drawable/ic_launcher_alarmclock" android: display="1"> 22

Confidential ¡ Extension to the applications resources ¡The interface of the resources was extended to allow an application to set the display programmatically. This is usually done in the on. Create() of an Activity: protected void on. Create(Bundle icicle) { super. on. Create(icicle); get. Resources(). set. Application. Display(Display. DISPLAY 2); © 2009 Teleca AB } ¡The class Resource was extended with two interfaces: public int get. Application. Display(); public void set. Application. Display(int display); ¡The display values are defined in Display. java as: Display. DEFAULT_DISPLAY Display. DISPLAY 2 23

Confidential Activity Manager ¡ The Activity Manager starts an Activity and as such needs to know onto which display an application wants to go. The Activity Manager then adds the Activity onto the Window Manager’s proper Activity stack. ¡ The Activity. Manager. Service was extended by a number of changes: ¡Additional list of history entries for the second display which stores the stack of launched activities ¡Reference to Window. Manager. Service for the second display © 2009 Teleca AB ¡Support to start, stop and resume activities for the second display ¡Support for separate Home Screen on second display. The second Home Screen itself was not implemented during the project. 24

Confidential Window Manager ¡ The Window Manager needs extensions to support two application/Activity stacks and two Window stacks. Additionally it needs to be made second display-aware. ¡Touch Events ¡Key Events © 2009 Teleca AB ¡Rendering 25

Confidential Surface Flinger Applications Application 1 Application 2 Application 3 Application Framework © 2009 Teleca AB Surface Surface Libraries Surface Flinger Hardware Abstraction Layer Linux Kernel Clamshell Driver Display Driver 1 /dev/graphics/fb 0 Display Driver 2 /dev/graphics/fb 1 26

Confidential The standard Surface Flinger was extended two handle two stacks of layers, one for each display. Internally the Layers contain the buffer to be rendered. Additionally all the calls to Open. GL set the right Open. GL context before so that the Open. GL calls go to the right display. From the applications point of view the decision onto which display is drawn is done by the display parameter set in the Surface object. The applications don’t do this directly though. They define this either in the Android. Manifest. xml or by setting the display in the associated resources by a direct API call. © 2009 Teleca AB The Activity Manager then decides onto which display an Activity wants to go and adds it to the correct Window Manager. The Window Manager then sets the proper display in the Surface object. Access permissions ¡ To access the Surface Flinger the following permission is defined in the platforms Android. Manifest. xml (in frameworks/base/core/res/): ¡ android. permission. ACCESS_SURFACE_FLINGER ¡ The following public constructor of Surface is hidden from the SDK and requires this permission: ¡ public Surface( Surface. Session s, int pid, int display, int w, int h, int format, int flags) throws Out. Of. Resources. Exception; 27

Confidential Linux Kernel / Drivers ¡ /dev/graphics/fb 0 – Framebuffer for default display ¡ /dev/graphics/fb 1 – Framebuffer for second display ¡ /dev/input/event 0 – HW keyboard ¡ /dev/input/event 1 – Touchable first display © 2009 Teleca AB ¡ /dev/input/event 2 – Touchable second display 28