Framework for digital camera in Linux Dongsoo Kim

Framework for digital camera in Linux Dongsoo Kim (dongsoo 45. kim@samsung. com) Heungjun Kim(riverful. kim@samsung. com) Samsung Electronics

Contents Introducing mobile camera devices - So. C camera subsystem - Mobile camera module devices Framework for digital camera - Video 4 Linux 2 and camera device - New APIs for high-end mobile camera devices - Dual camera framework for 3 G handset Porting issues in So. C camera subsystem 2

Terms AP : Application Processor Camera interface : interface device which is syncing image data from external camera device and gives some commands to them Camera module : external camera device packaged with lens, sensor and ISP on it ISP : Image Signal Processor. Also supports lots of additional functions V 4 L 2 : Video for Linux 2 3

This document contains some animations. Slideshow mode is recommended 4

What makes camera “a camera”? Exposure control - Aperture (Iris) - Shutter - ISO Strobe support - Various program mode : slow sync, red-eye reduction and on…. Lens control - Focus - Zoom 5

Introducing mobile camera devices 6

History of camera phone 1. 3 MP in 2003 First Camera Phone 5 MP in 2004 3 MP 7 MP with in 2004 2005 2 MP in 12 MP in 2009 in 10 MP 2000 3. 5 MP 2006 7

Mobile camera module devices Old and low-end mobile camera modules Feature Exposure Lens Strobe Low end camera Aperture Not supported Shutter Just change frame rate ISO Low sensitivity & High noise level Focus Pan focus Zoom Digital zoom Just LED light most of time 8

Mobile camera module devices Brand new and high-end mobile camera modules Feature Exposure Lens Strobe High end camera Aperture Supported with lens packaging Shutter Supported with lens packaging ISO High sensitivity & Low noise level Focus Movable focus point (Auto/manual) Zoom Optical zoom & Digital zoom XENON flash is also supported 9

So. C camera subsystem What is So. C camera subsystem? ARM So. C solution Peripherals Camera interface Peripheral 3 (USB) ARM Core External Camera module Peripheral 2 (MMC) Peripheral 1(Display) 10

So. C camera subsystem Request for data Image data in expected format Camera interface Command interface Resolution handling Data format handling (YUV/RAW/JPEG …) Buffer handling I 2 C RAW DATA Camera module Exposure control Lens control White balance control Effect control Face detect Zoom control Strobe control . . . 11

So. C camera subsystem Various ways of preview (Live view) in So. C camera subsystems DATA ARM core Camera DATA Module Camera Interface User Application M e m o r y COPY! Frame buffer OK Cancel 12

So. C camera subsystem Various ways of preview (Live view) in So. C camera subsystems User Application DATA ARM core Camera DATA Module Camera Interface M e m o r y D M A COPY! Save as file Through DMA Frame buffer OK Cancel 13

So. C camera subsystem How to capture still shots in So. C camera subsystem - Your camera module has a dedicated JPEG encoder on it Camera module ARM core DATA JPEG PROCESS MEMORY DATA RAW DATA PROCESS DATA Camera Interface User Application M e m o r y COPY! Save as JPEG file 14

So. C camera subsystem How to capture still shots in So. C camera subsystem - No JPEG encoder in your camera module User Application DATA ARM core Camera DATA Module Camera Interface M e m o r y COPY! RAW DATA Codec S/W Save as JPEG file 15

So. C camera subsystem How to take motion pictures in So. C camera subsystem COPY! DATA ARM core Camera DATA Module Camera Interface M e m o r y Frame buffer User Application OK Codec S/W Cancel Save as… COPY! 16

What we’ve got in Linux to control camera In Kernel V 4 L 2 (AKA Video For Linux 2) - The second version of the Video For Linux API - Kernel interface for analog radio and video capture and output drivers In user space Multimedia middleware Gstreamer - Pipeline based multimedia framework written in the C programming language with the type system based on GObject Open. MAX - Cross-platform set of C-language programming interfaces that provides abstractions for routines especially useful for audio, video, and still images 17

Mobile camera module devices Is V 4 L 2 enough for mobile digital camera modules? No way 18

Framework for digital camera 19

Video 4 Linux 2 and camera device Camera subsystem in Linux kernel aspect V 4 L 2 videobufcore V 4 l 2 dev videobufsg-dma V 4 l 2 ioctl videodev V 4 l 2 -device V 4 l 2 -intdevice (old) V 4 l 2 subdev (new) Camera interface V 4 L 2 driver Command interface Resolution handling Data format handling (YUV/RAW/JPEG…) Buffer handling Camera module V 4 L 2 driver Exposure control Lens control White balance control Effect control Face detect Zoom control Strobe control . . . 20

Abstract view of V 4 L 2 working with video capture device How does it work with V 4 L 2? Application or Middleware Copy memory to userspace Open device VIDIOC_QUERYCAP VIDIOC_S_INPUT VIDIOC_QBUF VIDIOC_REQBUF VIDIOC_DQBUF VIDIOC_QUERYBUF VIDIOC_STREAMON 21

New V 4 L 2 APIs for digital camera Exposure control 22

New V 4 L 2 APIs for exposure control Need for detailed control in exposure - We’ve got camera devices with mechanical shutter - Iris could be handled also - ISO controlled output is somehow useful Why not using enhanced exposure control like a regular digital camera? 23

New V 4 L 2 APIs for exposure control What we’ve got in mainline V 4 L 2 right now? #define V 4 L 2_CID_EXPOSURE_AUTO (V 4 L 2_CID_CAMERA_CLASS_BASE+1) enum v 4 l 2_exposure_auto_type { V 4 L 2_EXPOSURE_AUTO = 0, V 4 L 2_EXPOSURE_MANUAL = 1, V 4 L 2_EXPOSURE_SHUTTER_PRIORITY = 2, V 4 L 2_EXPOSURE_APERTURE_PRIORITY = 3 }; #define V 4 L 2_CID_EXPOSURE_ABSOLUTE (V 4 L 2_CID_CAMERA_CLASS_BASE+2) #define V 4 L 2_CID_EXPOSURE_AUTO_PRIORITY (V 4 L 2_CID_CAMERA_CLASS_BASE+3) No way to control iris No way to control exposure metering method 24

New V 4 L 2 APIs for exposure control Additional exposure control enum v 4 l 2_exposure_auto_type { V 4 L 2_EXPOSURE_AUTO = 0, V 4 L 2_EXPOSURE_MANUAL = 1, V 4 L 2_EXPOSURE_SHUTTER_PRIORITY = 2, V 4 L 2_EXPOSURE_APERTURE_PRIORITY = 3, /* Additional features for digital camera */ V 4 L 2_EXPOSURE_ISO_PRIORITY =4, + + }; #define V 4 L 2_CID_EXPOSURE_ABSOLUTE (V 4 L 2_CID_CAMERA_CLASS_BASE+2) #define V 4 L 2_CID_EXPOSURE_AUTO_PRIORITY (V 4 L 2_CID_CAMERA_CLASS_BASE+3). . . +#define V 4 L 2_CID_EXPOSURE_LOCK (V 4 L 2_CID_CAMERA_CLASS_BASE+18) 25

New V 4 L 2 APIs for exposure control +#define V 4 L 2_CID_CAM_APERTURE(V 4 L 2_CID_CAMERA_CLASS_BASE+19) +#define V 4 L 2_CID_CAM_SHUTTER (V 4 L 2_CID_CAMERA_CLASS_BASE+20) +#define V 4 L 2_CID_CAM_ISO (V 4 L 2_CID_CAMERA_CLASS_BASE+21) Supported aperture stages, shutter speed, and ISO speed could be different between every different camera module products Supported values for aperture, shutter and ISO should be made in V 4 L 2_CTRL_TYPE_MENU { . id = V 4 L 2_CID_CAM_APERTURE, . type = V 4 L 2_CTRL_TYPE_MENU, . name = "Aperture", . minimum = 0, . maximum = 4, . step = 1, . default_value = 0, }, static const char *camera_iris_stages[] = { /* This module supports 5 Iris stages * on it's own but F number depends * how you package Lens module */ "2. 8“, "4“, "5. 6“, "8“, "11“, NULL }; 26

New V 4 L 2 APIs for exposure control New exposure metering control +/* Exposure Methods */ +#define V 4 L 2_CID_PHOTOMETRY (V 4 L 2_CID_CAMERA_CLASS_BASE+17) +enum v 4 l 2_photometry_mode { + V 4 L 2_PHOTOMETRY_MULTISEG = 0, + V 4 L 2_PHOTOMETRY_CWA = 1, + V 4 L 2_PHOTOMETRY_SPOT = 2, + V 4 L 2_PHOTOMETRY_AFSPOT = 3, +}; 27

New V 4 L 2 APIs for digital camera Lens control 28

New V 4 L 2 APIs for lens control Need for enhanced Focus mode API - Current V 4 L 2 framework just slightly covers MANUAL & AUTO FOCUS mode for usb webcam - High-end camera modules support for various focus mode presets like • AF Macro • AF-S / AF-C For compatibility Focus control API should be defined 29

New V 4 L 2 APIs for lens control Focus Mode & Enumeration values #define V 4 L 2_CID_FOCUS_ABSOLUTE (V 4 L 2_CID_CAMERA_CLASS_BASE+10) #define V 4 L 2_CID_FOCUS_RELATIVE (V 4 L 2_CID_CAMERA_CLASS_BASE+11) +#define V 4 L 2_CID_FOCUS_MODE (V 4 L 2_CID_CAMERA_CLASS_BASE+13) +/* Focus Methods */ +enum v 4 l 2_focus_mode { + V 4 L 2_FOCUS_AUTO + V 4 L 2_FOCUS_MANUAL + V 4 L 2_FOCUS_MACRO + V 4 L 2_FOCUS_CONTINUOUS +}; = = 0, 1, 2, 3, 30

New V 4 L 2 APIs for digital camera Object recognition 31

What is object detection? Object detection Gee! It recognized my face! Hello everyone 32

What should we consider? What do we set? - Detect mode - Type of object - How many objects to detect - Triggered action What do we get? - How many objects are detected - Detected object’s type - Detected object’s coordinates 33

New V 4 L 2 APIs for object recognition New API for object recognition - New capability #define V 4 L 2_CAP_ASYNCIO #define V 4 L 2_CAP_STREAMING 0 x 02000000 /* async I/O */ 0 x 04000000 /* streaming I/O ioctls */ +#define V 4 L 2_CAP_OBJ_RECOGNITION 0 x 10000000 - New IOCTL +#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v 4 l 2_detect) +#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v 4 l 2_detect) 34

New V 4 L 2 APIs for object recognition New API for object detection +/* Object detection and triggered actions */ +enum v 4 l 2_recog_mode { + V 4 L 2_RECOGNITION_MODE_OFF = 0, + V 4 L 2_RECOGNITION_MODE_ON = 1, + V 4 L 2_RECOGNITION_MODE_LOCK = 2, +}; + +enum v 4 l 2_recog_action { + V 4 L 2_RECOG_ACTION_NONE = 0, /* only detection */ + V 4 L 2_RECOG_ACTION_BLINK = 1, /* Capture on blinking */ + V 4 L 2_RECOG_ACTION_SMILE = 2, /* Capture on smiling */ +}; + +enum v 4 l 2_recog_pattern { + V 4 L 2_RECOG_PATTERN_FACE = 0; /* Face */ + V 4 L 2_RECOG_PATTERN_HUMAN = 1; /* Human */ + V 4 L 2_RECOG_PATTERN_CHAR = 2; /* Character */ +} 35

New V 4 L 2 APIs for object recognition New API for object detection +struct v 4 l 2_recog_rect { + enum v 4 l 2_recog_pattern p; /* What pattern detected */ + struct v 4 l 2_rect o; /* detected area */ + __u 32 reserved[4]; +} + +struct v 4 l 2_recognition_data { + __u 32 detect_cnt; /* detected object counter */ + struct v 4 l 2_recog_rect obj; /* detected area */ + __u 32 reserved[4]; +}; + +struct v 4 l 2_recognition { + enum v 4 l 2_recog_mode; + enum v 4 l 2_recog_pattern; /* What pattern to detect */ + __u 32 obj_num; /* How many object to detect */ + __u 32 detect_idx; /* select detected object */ + struct v 4 l 2_recog_data; /* read only */ + enum v 4 l 2_recognition_action; + __u 32 reserved[4]; +}; 36

What should we consider? What do we set? - Detect mode - How many objects to detect - Triggered action What do we get? - How many objects are detected - Detected object’s coordinates 37

New V 4 L 2 APIs for object recognition New API for object recognition - New capability #define V 4 L 2_CAP_ASYNCIO #define V 4 L 2_CAP_STREAMING 0 x 02000000 /* async I/O */ 0 x 04000000 /* streaming I/O ioctls */ +#define V 4 L 2_CAP_OBJ_RECOGNITION 0 x 10000000 - New IOCTL +#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v 4 l 2_detect) +#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v 4 l 2_detect) 38

New V 4 L 2 APIs for object recognition New API for object detection +/* Object detection and triggered actions */ +enum v 4 l 2_recognition_mode { + V 4 L 2_RECOGNITION_MODE_OFF = 0, + V 4 L 2_RECOGNITION_MODE_ON = 1, + V 4 L 2_RECOGNITION_MODE_LOCK = 2, +}; + +enum v 4 l 2_recognition_action { + V 4 L 2_RECOG_ACTION_NONE = 0, /* only detection */ + V 4 L 2_RECOG_ACTION_BLINK = 1, /* Capture on blinking */ + V 4 L 2_RECOG_ACTION_SMILE = 2, /* Capture on smiling */ +}; + +struct v 4 l 2_recognition_data { + __u 8 detect_cnt; /* detected object counter */ + struct v 4 l 2_rect o; /* detected area */ +}; + +struct v 4 l 2_recognition { + enum v 4 l 2_recognition_mode; + __u 8 obj_num; /* How many object to detect */ + __u 8 detect_idx; /* select detected object */ + struct v 4 l 2_recognition_data; /* read only */ + enum v 4 l 2_recognition_action; +}; 39

New V 4 L 2 APIs for digital camera Dual camera 40

Dual camera for 3 G handset What is dual camera? Flip side camera : for still shots Front side camera : for special purpose Dual camera applications - Video telephony - Camera applications • Taking self portrait shots 41

Dual camera for 3 G handset H/W restrictions in Application Processor’s camera peripheral interface Mega camera VGA camera Data path 42

Why should we make single device node? H/W restrictions in Application Processor’s camera peripheral interface “SINGLE” camera interface on Application Processor ! Only one camera could be handled at a time Only one camera could be using data pins at a time Each camera is different in Working MCLK Working resolution (SYNC) Working PCLK 43

What do we have in V 4 L 2? We already have API for dual camera - Enumerate video inputs VIDIOC_ENUMINPUT - Query or select the current video input VIDIOC_G_INPUT VIDIOC_S_INPUT 44

What do we need to consider for dual camera? What do we need to consider? • Let user space application get noticed about camera input devices • Don’t forget VIDIOC_ENUMINPUT • Check out supported pixel format using VIDIOC_ENUM_FMT and VIDIOC_ENUM_FRAMESIZES • Context of each camera module • Capability : Supported color spaces, resolutions, additional functions… • Latest working status : resolution, color space, effect and so on • Context of camera interface • Working information of the camera module getting switched : Clock information and so on 45

Example of switching camera How to switch active camera Condition 1 How to enable camera module: Give power source Enable expecting camera Give MCLK Condition 2 Shared things Camera power enable pin Camera data pins MCLK Mega camera VGA camera Mega Enable VGA Enable Mega Reset VGA Reset MCLK DATA PINS CAM POWER CPU Condition 3 Dedicated things Camera enable pins Camera reset pins 46

Abstract work flow of switching camera Cam power off STREAMON Camera (A) initialize Start preview Camera (A) enable pin off Camera (B) Enable pin on Cam power on Switch to camera STREAMOFF (B) Stop preview G_INPUT/ S_INPUT Camera (B) initialize Start preview Camera interface reset & setting Stop preview Stop DMA Reset camera interface’s current setting Make a proper MCLK for camera (B) Setup resolution, color-space, DMA 47

Porting issues in So. C camera subsystem 48

I 2 C issue I 2 C protocol has start and stop condition Some camera modules generate unexpected noise until their power up sequence - Makes I 2 C bus in busy state 49

I 2 C issue How to prevent this malfunctioning issue Use a “noise free on power up” camera module OR Use a level shifter to block and ignore noise from camera module OR Change functionality of I 2 C pins while turning on camera module 50

Resolution issue All about between camera interface and camera module - Expected resolution from camera interface - Served resolution from camera module - Symptoms depend on camera interface Expecting resolution is bigger than served resolution - A “select time out” error in user space application or - Preview image is floating in some direction Serving resolution is bigger than expected resolution - A “select time out” error in user space application or - Preview angle looks like to be zoomed in 51

Color space issue Color order mismatched between camera interface and camera module 52

Lens control issue Preview freezing while checking lens response - Using while() loop while it checks response, preview will freeze until focusing job gets finished - If the ISP driver use the kernel thread when it check response from current lens status could make smooth preview on the LCD while lens is on focusing. 53

Lens control issue Solving preview freezing while checking lens status struct platform_isp { const struct platform_data *pdata; struct v 4 l 2_int_device *v 4 l 2_int_device; struct i 2 c_client *i 2 c_client; int reschk; . . } static int ioctl_s_ctrl(struct v 4 l 2_int_device *s, struct v 4 l 2_control *vc) { struct platform_isp *isp = s->priv; struct i 2 c_client *client = isp->i 2 c_client; switch (vc->id) { case V 4 L 2_CID_FOCUS_AUTO: isp->vfm = vc->value; if (vc->value == V 4 L 2_CID_FOCUS_MACRO) { err = i 2 c_command_for_isp( client, focus_macro ); isp->reschk = 1; } If isp->reschk is Set, break; The kernel thread is active. } } 54

Lens control issues Solving preview freezing while checking lens status int pseudo_reschk_thread (void *data) Platform Data Vairable for isp, or static variable. { int err = 0; This function return when the ISP is ……… matched the focus(+ return) or do { timeout(- return). switch (isp->reschk) { case 1: err = i 2 c_command_for_isp(client, focus_check); if (err < 0) { isp->reschk = 1; /* i 2 c command function failed. */ } else { isp->reschk = 0; /* if succeed, reschk ended. */ } break; …… This is able to re-check } to the next check time. } while(1); return 0; } 55

JPEG data synchronization issues JPEG has no synchronization protocol when it gets transferred All about camera interface and camera module’s data synchronization - When camera interface supports JPEG sync • Follow user manual and make camera module to follow that protocol - When camera interface does not support JPEG sync • Make synchronization protocol: make JPEG data to be synchronized with VSYNC 56

Q&A 57

Thank you 58