Final Year Project 20042005 LYU 0402 Augmented Reality

Final Year Project 2004/2005 LYU 0402 Augmented Reality Table for Interactive Card Games Supervisor: Prof Michael Lyu Presented by: Kevin Chow, Albert Lam

Overview n n n n Introduction Objective Trading Card Game Architecture of ART Implementation in First Term Experiment Conclusion Future Work

Introduction n What is Augmented Reality? • Mixed Reality combines the content from the real world with virtual imaginary • Augmented Reality is a subset of this where virtual content is overlaid into real objects of the world

Introduction (Con’t) • Paul Milgram characterized Mixed Reality interfaces on his “Reality. Virtuality Continuum” in 1994 Mixed Reality (MR) Real Environment Augmented Reality (AR) Augmented Virtuality (AV) Reality-Virtuality (RV) Continuum Virtual Environment

Introduction (Con’t) n An AR system has the following three main characteristics: • Combines real and virtual objects in a real environment • Runs interactively, and in real time • Registers virtual objects onto the real world

Introduction (Con’t) n ART stands for Augmented Reality Table • User can play Card Games on the ART. • ART can enhance the visual effect during the playing of the card game.

Objective n n Develop a generic Interactive ART Trading Card Game application. Enhance the traditional card games: • Visual effect • Sound effect • Complex calculation

Trading Card Game n Trading Card Games • Two players play a match. • Players play cards to fight with each other. It includes summoning monster, casting magic, or setting traps, etc. • We choose “YU-GI-OH” as our implementation

Trading Card Game (Con’t) n Trading card game includes five kinds of actions • • • Restore Draw cards Play cards Attack/Challenge Discards

An Example - YU-GI-OH n In YU-GI-OH, Different information are stored on the card.

An Example - YU-GI-OH (Con’t) n n Game Mat of YU-GI-OH Different kinds of card must be put inside particular card zones

An Example - YU-GI-OH (Con’t) 1200 – 500 = -700 ATK : 500 ATK : 1200 LP : 1300 2000 LP : 2000

An Example - YU-GI-OH (Con’t) 1300 – 1200 = -100 DEF : 1300 ATK : 1200 LP : 1300 LP : 1900 2000

Architecture n Hardware Setup • Plasma monitor • Overhead camera n System architecture • Augmented Reality Perception • Game Core • Database • Game Enhancement

Hardware Setup

Hardware Setup (Con’t) n Computer • • • n processes perceiving image generates visual and audio enhancement controls game flow Plasma Monitor. • acts as the Game Table • displays computer-generated scene n Overhead mounted camera • captures the cards and the screen of the plasma. • only input of the system

System Architecture n n Game Core Module Perception Module Database Module Game Enhancement Module

System Architecture (Con’t)

System Architecture (Con’t) n Microsoft Direct. X SDK • A set of low-level APIs for highperformance multimedia applications • directly access the hardware • provides device independent through HAL • We will use Direct. Show, Direct 3 D and Direct. Sound in our implementation

Implementation n Perception Module • Calibration • Search window locator • Card Recognizer n Database Module • Identify card type • Image retrieval • Image matching

Perception Module n n n Read and process the raw video perceived from the camera Detect input card and command Some assumptions • fixed camera, fixed table • camera is approximately right above the table • cards can only be placed in predefined region called card zones

Calibration n Why? • Captured image varies with environmental lighting condition • Take advantage of the assumptions made

Calibration (Con’t) n How? • Set the card zone position • Set the card area threshold for each card zone • Calibrate the colors • Calibrate card type colors

Search Windows Locator n n n Find region to search for cards The screen is partitioned into some fixed small search windows Search windows which just stop changing will be processed

Search Windows Locator (Con’t) n Located search windows that have changes • Image different • Pixel changed if • A Search window is said to be changed if n Activate search window from “change” to “unchange” state

Card Locator n Locate four corner card positions given a search window • Apply Canny Algorithm • Extract Contours • Search for contours which contain 4 points n contain nearly right angle corners n have area within a certain threshold n

Card Locator (Con’t) n Canny Algorithm • Remove noise by Gaussian Filter • Compute edge strength and edge direction by Sobel Operator • Apply Nonmaximum Suppression to trace edges • Use Hystersis to eliminate streaking

Card Recognizer n Card orientation • Compare the lengths of the four edges • Compare color difference between top and bottom parts of the card

Card Recognizer (Con’t) n Get Card Image • Extract the undistorted card image from a distorted image

Card Recognizer (Con’t) • Pixel coordinate transformation n Map a point (x, y) on the distorted image to a point (x’, y’) on the undistorted image (x 2, y 2) (x 3, y 3) . (x’, y’) . (x, y) h (x 1, y 1) (x 0, y 0) w

Card Recognizer (Con’t) • Brightness interpolation Assign brightness value to the point (x, y) n Apply Bilinear Interpolation n

Card Recognizer (Con’t) n Identify the card uniquely • Query the Image Database with the card image for unique card Id • Query the Card Database with the card Id for card information and details

Database Module n n Contains all game information Identify card image by 3 steps: • Identify card type n Classify card by card type • Image retrieval n Retrieve several similar candidate cards • Image matching n Find the best match card

Identify Card Type • Compare the background color of the card to the calibrated color

Image Retrieval n n n Retrieve similar images of same card type. Use color-based retrieval method due to low resolution Color Histogram method select cards with largest color histogram intection

Image Retrieval (Con’t) n Color histogram intersection

Image Matching n Select the best matched card from the candidates • Split the captured image into 4 channels • Compare only inner image • Compare the different channels to the database separately

Image Matching • Find the pixel difference by • Reject the image if the pixel difference is larger than the threshold • Accept the image with minimum pixel difference

Experiment n n Match 15 cards to a set of different cards with threshold set to 1500 Here, we have chosen results of three cards for discussion

Experimental Result

Experimental Result (Con’t)

Experimental Result (Con’t)

Conclusion n n We have implemented a simplified version of the ART system Concerning low resolution of the image, we have develop an algorithm to recognize card with high accuracy but less efficiency

Future Work n n Color Calibration Develop an efficient card recognition algorithm while retaining high accuracy Game Enhancement Interactive input detector

End of Presentation Thank you very much

Q & A Session