I MX ANDROID INTRODUCTION CHEN XINYU 2020 PUBLIC

I. MX ANDROID INTRODUCTION CHEN XINYU 2020 PUBLIC

Agenda Overview • System Architecture • Booting • Android Framework • − Application − Graphics − Multimedia − Power Management − Camera Compatibility • Debug Method • 1 PUBLIC

OVERVIEW 2 PUBLIC

Basic Introduction • Android is an open source software stack created for a wide array of devices with different form factors: − Mobile phone (where start from) − Wear − Tablet − TV − Auto (Automotive) License - Majority of the Android software is licensed with Apache 2. 0 • Versions – Cupcake to Pie and 10 • 3 PUBLIC

i. MX Releases Cupcake 1. 5 (April 2009) Eclair 2. 1 (Oct 2009) i. MX 5 x R 3 (Jun 2009) i. MX 5 x R 7 (Jan 2010) Donut 1. 6 (Sept 2009) i. MX 5 x R 5 (Sept 2009) Ice Cream Sandwich 4. 0 (Oct 2011) i. MX R 13. 4 GA (Sept 2012) 4 PUBLIC Gingerbrea d 2. 3 (Dec 2010) i. MX 5 x R 10 (Jan 2011) Froyo 2. 2 (May 2010) i. MX 5 x R 9 (Aug 2010) Honeycomb 3. 2 (Feb 2011) i. MX 5 x R 11 i. MX 6 x R 12 (Sept 2011) Marshmall ow 6. 0 (Oct 2015) Kit. Kat 4. 4 (Oct 2013) KK 4. 4. 2 GA (Jun 2014) Oreo 8. 0 (Aug 2017) O 8. 1 GA (April 2018) M 6. 0 GA (July 2016) Jelly. Bean 4. 3 (July 2012) Lollipop 5. 1 (Nov 2014) JB 4. 3 GA (Nov 2013) L 5. 1. 1 (Jan 2016) Nougat 7. 1 (Aug 2016) N 7. 1 GA (May 2017) Q 10 (Sept 2019) Q 10. 0 (Feb 2020) Pie 9. 0 (Aug 2018) P 9. 0 (Jan 2019)

SYSTEM ARCHITECTURE 5 PUBLIC

Android System Architecture • • • 6 PUBLIC Kernel - The foundation of the Android platform is the Linux kernel. HAL - The hardware abstraction layer (HAL) provides standard interfaces that expose device hardware capabilities to the higherlevel Java API framework. Android Runtime - For devices running Android version 5. 0 (API level 21) or higher, each app runs in its own process and with its own instance of the Android Runtime (ART). ART is written to run multiple virtual machines on low-memory devices by executing DEX files, a bytecode format designed specially for Android that's optimized for minimal memory footprint. Native Libraries - Many core Android system components and services, such as ART and HAL, are built from native code that require native libraries written in C and C++. Java API Framework - The entire feature-set of the Android OS is available to you through APIs written in the Java language. These APIs form the building blocks you need to create Android apps by simplifying the reuse of core, modular system components and services. System App - Android comes with a set of core apps for email, SMS messaging, calendars, internet browsing, contacts, and more. Apps included with the platform have no special status among the apps the user chooses to install.

Source Code Structure • art – target runtime library and host tools for ART • bionic – bionic C library (Apache license), smaller and faster than glibc • bootable - uboot • build – build system for android source • cts – cts test suite package and app • development – host and target development tools (gdb, apkcheck), script, app(Sdk. Setup) and example app(Open. GL example), etc • device – i. MX soc/board configurations (init/mount/audio/sepolicy. . ), Makefiles and framework overlay files (config. xml). • external – external open source projects like sqlite, skia library etc. • frameworks – Android framework core source code. § frameworks/av – native av frameworks. media/camera/radio services (stagefright) § frameworks/base – runtime, java core frameworks (services, managers) • • • 7 PUBLIC hardware – HAL HIDL and legacy libcore – core java library packages – default applications (email, dialer. . ), providers prebuilt – prebuilt tools, like gcc cross compiler system – system components like bluedroid, init, adb, netd, vold. vendor – NXP vendor implementations, kernel, uboot, ATF, HAL, proprietary binary and codecs, etc.

ART (Android Runtime) – Lollipop 5. 0 Dalvik - Just In Time (JIT) With the Dalvik JIT compiler, each time when the app is run, it dynamically translates a part of the Dalvik bytecode into machine code. As the execution progresses, more bytecode is compiled and cached. ART - Ahead Of Time (AOT) ART is equipped with an Ahead-of-Time compiler. During the app’s installation phase, it statically translates the DEX bytecode into machine code and stores in the device’s storage. This is a one-time event which happens when the app is installed on the device. With no need for JIT compilation, the code executes much faster. As ART runs app machine code directly (native execution), it doesn’t hit the CPU as hard as just-in-time code compiling on Dalvik. Because of less CPU usage results in less battery drain. 8 PUBLIC

Binder IPC • Binder allows the application framework to cross process boundaries and call into the Android system services code. • At the application framework level, this communication is hidden from the developer and things appear to "just work". • High performance through shared memory • Reference counting, and mapping of object references across processes • Synchronous calls between processes 9 PUBLIC

Typical calling flow • • • 10 Example. Activity directly call get. System. Service() to get Location. Manager proxy instance through Service. Manager by Binder Location. Manager invoke functions in Location. Service through Binder Location. Service call Location. Provider for access the HAL. Location. Provider call the provider C++ code by JNI Provider call the HAL. so directly to access UART for GPS location PUBLIC

BOOTING 11 PUBLIC

Booting Flow • • 12 PUBLIC ROM load U-boot, U-boot load kernel, kernel start Kernel start the “init” process (PID 1) “Daemons” for handling low level hardware interfaces are started up (vold, netd, ueventd, logd, adbd, etc). Init initiates the "Service Manager", a key element for "Binders" and IPC communication. app_process runs "Zygote“, which start a new instance of Dalvik/ART for running the "System. Server“ Important servers like “surfaceflinger”, “mediaserver” and “cameraserver” are startup by init (defined in /etc/init/*. rc, startup by start class) All the others android components (Services and their Managers, like Activity. Manager, Power. Manager and Window. Manager) are then started by “System. Server” Zygote is a special Android OS process that enables shared code across Dalvik/Art VM in contrast with Java VM where each instance has its own copy of core library class files and heap objects.

Booting Flow – bootloader uboot> boota <addr> [mmc 0 | mmc 1 [<partition>]] struct andr_img_hdr { char magic[ANDR_BOOT_MAGIC_SIZE]; /* “ANDROID!” */ u 32 kernel_size; /* size in bytes */ u 32 kernel_addr; /* physical load addr */ z. Image u 32 ramdisk_size; /* size in bytes */ u 32 ramdisk_addr; /* physical load addr */ u 32 second_size; /* fdt size */ ramdisk. img mkbootimg <board>. dtb u 32 second_addr; /* fdt address */ boot. img u 32 tags_addr; /* physical addr for kernel tags */ u 32 page_size; /* flash page size we assume */ u 32 unused[2]; /* future expansion: should be 0 */ cmdline baseaddr char name[ANDR_BOOT_NAME_SIZE]; /* asciiz product name */ char cmdline[ANDR_BOOT_ARGS_SIZE]; u 32 id[8]; /* timestamp / checksum / sha 1 / etc */ }; 13 PUBLIC

Booting Flow – init • 14 init. <*>. rc file identify which services and daemons started up, and by which priority. PUBLIC Action / Service Description on early-init Set init and its forked children's oom_adj. Set the security context for the init process. Start ueventd Mount debugfs on init setup the global environment Create cgroup mount point for cpu accounting Create folders used later. on fs mount partitions (system/data/cache/misc. . ) on post-fs change permissions of system directories on post-fs-data change permission of /data folders and sub folders on early-boot Anything earlier than boot phase, but after fs mounted. on boot basic network init , setprop, change /sys nodes ownership etc. Start all the core class “service” service start daemons, servers and services. Priority by the class attrib “core/main/late_start”

ANDROID FRAMEWORK 15 PUBLIC

App Framework An activity has essentially four states: • If an activity is in the foreground of the screen (at the top of the stack), it is active or running. • If an activity has lost focus but is still visible (that is, a new non-full-sized or transparent activity has focus on top of your activity), it is paused. A paused activity is completely alive (it maintains all state and member information and remains attached to the window manager), but can be killed by the system in extreme low memory situations. • If an activity is completely obscured by another activity, it is stopped. It still retains all state and member information, however, it is no longer visible to the user so its window is hidden and it will often be killed by the system when memory is needed elsewhere. • If an activity is paused or stopped, the system can drop the activity from memory by either asking it to finish, or simply killing its process. When it is displayed again to the user, it must be completely restarted and restored to its previous state. 16 PUBLIC

UI Framework 3 or 5 App 0 Surface Flinger Framebuffer Frame buffer UI Draw Layers App 1 Surface HWUI Open. GL ES SKIA Hardware Composer libg 2 d 3 D GPU AGL alloc. Memory Gralloc HAL Reserved 17 PUBLIC CMA ION buddy 3 D GPU 2 D GPU Optional DRM/fb

Surface. Flinger Composition Surface. Flinger Display. Device Construct( ) rebuild. Layer. Stacks( ) set. Visible. Layers. Sorted. By. Z() Layer EGL Framebuffer. Surface HWC HAL egl. Create. Window. Surface() producer consumer Get layers attribute do. Composition() For each display do. Composition. Surface( ) get. Visible. Layers. Sorted. By. Z() For each layers draw(Display. Device) draw. With. Open. GL() swap. Buffers( ) Compose to framebuffer egl. Swap. Buffers( ) advance. Frame() next. Buffer() post. Framebuffer() 18 PUBLIC set. Client. Target() present. And. Get. Release. Fences() hwc 2_set_cli ent_target() Set framebuffer to m. Render. Target hwc 2_prese nt_display() g 2 d compose DRM commit (m. Render. Target)

Gralloc • gralloc. imx 8. so − Graphic buffers -> GPU gralloc RGBA(x)8888/888 BGRA 8888 RGB 565 YUV Frame. Buffer gralloc_viv. imx 8. so i. MX gralloc − Video buffers, Frame buffers -> ION • GPU gralloc go by viv DRM driver − Allocating pools Reserved -> Virtual -> libion libdrm ion drv viv drm drv CAM − Surface layer buffers directly allocated from virtual pool − compression TITLED buffer start from reserved pool User space Kernel • Surface layer ION − Allocate from buddy or CMA depends Buddy 19 PUBLIC CMA Reserve on the contiguous or cache flags

Multimedia Framework Audio. Flinger Audio. Track Tiny. Alsa Local File Media. Player Alsa Driver Stagefright App 0 Surface Video Surface VPU OMX Component VPU 20 PUBLIC Surface. Flinger SAI

Power Framework • Kernel CPU Governor - Interactive Governor This governor is designed for latency-sensitive workloads, such as interactive user interfaces. The interactive governor aims to be significantly more responsive to ramp CPU quickly up when CPUintensive activity begins. 21 PUBLIC

Media System Overview • • • Application framework – at the application framework level is application code that utilizes android. media APIs to interact with the multimedia hardware. Binder IPC - The Binder IPC proxies facilitate communication over process boundaries. They are located in the frameworks/av/media/libmedia directory and begin with the letter "I“. Native Multimedia Framework - At the native level, Android provides a multimedia framework that utilizes the Stagefright engine for audio and video recording and playback. Stagefright comes with a default list of supported software codecs and you can implement your own hardware codec by using the Open. Max integration layer standard. For more implementation details, see the Media. Player and Stagefright components located in frameworks/av/media. Open. MAX Integration Layer (IL) - The Open. MAX IL provides a standardized way for Stagefright to recognize and use custom hardware-based multimedia codecs called components. You must provide an Open. MAX plugin in the form of a shared library named libstagefrighthw. so. This plugin links Stagefright with your custom codec components, which must be implemented according to the Open. MAX IL component standard i. MX OMX Codecs • Binary: device/fsl-codec/lib/ • Source: external/fsl_imx_omx/ 22 PUBLIC

Audio System Overview • • 23 PUBLIC Application framework − The application framework includes the app code, which uses the android. media APIs to interact with audio hardware. Internally, this code calls corresponding JNI glue classes to access the native code that interacts with audio hardware. JNI − The JNI code associated with android. media calls lower level native code to access audio hardware. JNI is located in frameworks/base/core/jni/ and frameworks/base/media/jni. Native framework − The native framework provides a native equivalent to the android. media package, calling Binder IPC proxies to access the audio-specific services of the media server. Native framework code is located in frameworks/av/media/libmedia. Binder IPC − Binder IPC proxies facilitate communication over process boundaries. Proxies are located in frameworks/av/media/libmedia and begin with the letter "I". Media server − The media server contains audio services, which are the actual code that interacts with your HAL implementations. The media server is located in frameworks/av/services/audioflinger. HAL − The HAL defines the standard interface that audio services call into and that you must implement for your audio hardware to function correctly. The audio HAL interfaces are located in hardware/libhardware/include/hardware. For details, see audio. h. Kernel driver − The audio driver interacts with your hardware and HAL implementation. You can use Advanced Linux Sound Architecture (ALSA), Open Sound System (OSS), or a custom driver (HAL is driver-agnostic).

Camera System Overview • App framework − At the app framework level is the app's code, which uses the Camera 2 API to interact with the camera hardware • AIDL − The binder interface associated with Camera. Service can be found at frameworks/av/camera/aidl/android/hardware. The generated code calls the lower level native code to obtain access to the physical camera and returns data that is used to create the Camera. Device and eventually Camera. Capture. Session objects at the framework level. • Native framework − This framework residing in frameworks/av/ provides a native equivalent to the Camera. Device and Camera. Capture. Session classes. See also NDK camera 2 reference. • binder IPC interface − The IPC binder interface facilitates communication over process boundaries. There are several camera binder classes located in the frameworks/av/camera/aidl/android/hardware directory that call into camera service. ICamera. Service is the interface to the camera service; ICamera. Device. User is the interface to a specific opened camera device; and ICamera. Service. Listener and ICamera. Device. Callbacks are the respective Camera. Service and Camera. Device callbacks to the application framework. • camera service − The camera service, located in frameworks/av/services/camera/libcameraservice/Camera. Service. cpp, is the actual code that interacts with the HAL. • HAL − The hardware abstraction layer defines the standard interface that the camera service calls into and that you must implement to have your camera hardware function correctly. 24 PUBLIC

Camera System Overview (HAL) Android Camera HAL i. MX Camera HAL Implement Camera. cpp Camera. HAL. cpp Ov 564 x (MIPI/CSI) Max 9286 (MIPI) MMAP Streamer UVC (MJPG) UVC (YUV) MJPG Streamer Ov 5640 (8 Q/8 MN) DMA Streamer VPU V 4 L 2_MEMORY_DMABUF V 4 L 2_MEMORY_MMAP (Allocate from dma_alloc in driver) V 4 l 2 driver 25 PUBLIC V 4 L 2_MEMORY_DMABUF (Allocate from ION)

Bluetooth System Overview • Application framework - At the application framework level is application code, which utilizes the android. bluetooth APIs to interact with the Bluetooth hardware. Internally, this code calls the Bluetooth process through the Binder IPC mechanism. • Bluetooth system service - The Bluetooth system service, located in packages/apps/Bluetooth, is packaged as an Android app and implements the Bluetooth service and profiles at the Android framework layer. • JNI - The JNI code associated with android. bluetooth is located in packages/apps/Bluetooth/jni. The JNI code calls into the HAL layer and receives callbacks from the HAL. • Bluetooth stack - The default Bluetooth stack is provided for you and is located in system/bt. The stack implements the generic Bluetooth HAL and customizes it with extensions and configuration changes. • Vendor implementation - Vendor devices interact with the Bluetooth stack using the Hardware Interface Design Language (HIDL). 26 PUBLIC

COMPATIBILITY 27 PUBLIC

Why Compatibility matters Application API compatibility. • Ecosystem Fragment! • Updating existing devices to a new version of Android is incredibly time consuming and costly!!! • 28 PUBLIC

Treble Project & HIDL Project Treble is trying to resolve the versions fragment issue by separating the hardware-specific code from the main Android OS code • HIDL is an HAL interface description language (IDL) to specify the interface between a HAL and its users. It allows specifying types and method calls, collected into interfaces and packages. • 29 PUBLIC

Document & Test Suites • The Compatibility Definition Document (CDD), representing the "policy" aspect of compatibility − It defines devices type, hardware requirement − Software/Multimedia/Performance/Security compatibility The Compatibility Test Suite (CTS), representing the "mechanism" of compatibility for Developer API • Vendor Test Suite (VTS), analogous to the CTS, to ensure forward compatibility of the vendor implementation, bring by the Treble Project • 30 PUBLIC

CTS Make sure release compatible with Android’s CDD (Compatibility Definition Document) • The Compatibility Test Suite (CTS) is a free, commercial-grade test suite. The CTS represents the "mechanism" of compatibility. • Area Signature tests Description The CTS contains a utility to check those API signatures against the APIs available on the device. The results from signature checking are recorded in the test result XML file. Platform API Tests Dalvik Tests Test the platform (core libraries and Android Application Framework) APIs as documented in the SDK Class Index to ensure API correctness. The tests focus on testing the Dalvik Executable Format. Platform Data Model The CTS tests the core platform data model as exposed to application developers through content providers. Platform Intents The CTS tests the core platform intents, as documented in the SDK Available Intents. Platform Permissions The CTS tests the core platform permissions, as documented in the SDK Available Permissions. Platform Resources The CTS tests for correct handling of the core platform resource types, as documented in the SDK Available Resource Types. 31 PUBLIC • The CTS is an automated testing harness that includes two major software components: The CTS tradefed test harness runs on your desktop machine and manages test execution. − Individual test cases are executed on the Device Under Test (DUT). The test cases are written in Java as JUnit tests and packaged as Android. apk files to run on the actual device target. −

VTS & GSI VTS consists of a set of testing frameworks and test cases, designed to help enhance the robustness, reliability, and compliance of 1) the Android system (e. g. , Core HALs and libraries) 2) low-level system software (e. g. , kernel, module, and firmware). A Generic System Image (GSI) is a pure Android implementation with unmodified Android Open Source Project (AOSP) code, runnable on a variety of Android devices. GSIs are used for running VTS and CTS-on-GSI tests. The system image of an Android device is replaced with a GSI then tested with the Vendor Test Suite (VTS) and the Compatibility Test Suite (CTS) to ensure that the device implements vendor interfaces correctly with the latest version of Android. 32 PUBLIC

DEBUG METHOD 33 PUBLIC

General Debug Method • Log system • − Android Studio − Logcat for almost all logs − dumpsys [ <service> | -l ] for service log • • PRINTF − systrace (ftrace for kernel profiling, atrace app) LOG_TAG macro − Log. x(TAG, . . ) in Java code Process tracing § Application tracing (Java API) § Native tracing § Native crash − Debuggered (logcat and /data/tombstones/<dump file>) − Strace (attach process, before run. trace system calls and signals ) • Native debugging − Gdb 34 PUBLIC Performance issue − Bootchart for boot timing − ALOGx() in C/C++ native code, with • App debugging − Strace/Valgrind (memory) for app profiling • Hardware Debug tool − TRACE 32

How to Debug System unstable issue? • • 35 Clarify failure details including symptom, reproducible steps, logs. Check whether voltage setting is aligned with datasheet. If power ripple exceeds the spec, try to increase voltages according to datasheet and then test. Check whether DDR is stable if random panic is observed. Memtester tool can be used for DDR stress test under OS. CPU frequencies can be scaled automatically depending on the system load. Please try to set as fixed max frequency to identify whether the issue is due to CPU frequency switch: echo performance > /sys/devices/system/cpu. X/cpufreq/scaling_governor Android system automatically adjusts the bus frequency (DDR, AHB, etc. ) for power saving based on the devices that are active. To disable busfreq: echo 0 > /sys/bus/platform/drivers/imx_busfreq/busfreq. x/enable Try to disable watchdog to capture failure symptom. Under Android N 7 release, it can add command option “androidboot. watchdogd=disabled” into kernel command line to disable watchdog. JTAG, Sys. Rq can be used to identify whether the system hang issue. Analysis kernel panic (if there’s panic) PUBLIC

How to Debug Performance issue? • Frame(UI) jank? App launch slow? • General tools − Logcat (ANR? ) − top, gputop − dumpsys activity/surfaceflinger/meminfo Application level – Android Studio for CPU/GPU profiler • System level – Android systrace • − Java code, native code: atrace − Kernel space: ftrace (with report trace file modification) 36 PUBLIC

How to Debug GPU relative issue? • There are three parts using GPU acceleration on android, you can disable some of them separately to narrow down issue. − Disable HWComposer: in Setting apk, Settings -> {} Developer options ->Disable HW overlays. − Disable Open. GL Renderer: to force use SKIA to draw by set “setprop sys. viewroot. hw false” and kill surfaceflinger thread. − Disable Open. GL 3 D draw can only be done after Disable Open. GL Renderer as this operation will totally disable all 3 D Open. GL acceleration. You can do it by “mv /system/lib/egl/lib. GLES_android. so /system/lib/egl/lib. GLES. so” and kill surfaceflinger thread. • APITrace − Trace Open. GL ES APIs call into file − Replay the recorded calls from file • 37 GPU CORE DUMP PUBLIC

How to Debug Multimedia issue? Analyze Clip information - Using PC client tool “Media. Info” to get codec info of the clip. Then check supported codec matrix from release doc. • Check behavior of that issue clip in latest android release. If no issue, try to replace parser library from latest android release to your android version. • Use android logcat to get issue case log. Please consider to change debug log level to generate more log sometimes. • OMX log • − touch /data/omx_log_level − echo 5 > /data/omx_log_level − chmod 777 /data/omx_log_level • OMX memory leakage − Change macro in external/fsl_imx_omx/Android. mk from OMX_MEM_CHECK to OMX_MEM_CHECK for memoy alloc/free trace 38 PUBLIC

Reference Android official site: https: //www. android. com • Android Source: https: //source. android. com • Application development: https: //developer. android. com/index. html • 39 PUBLIC