Grammar of Image Zhaoyin Jia 03 30 2009
- Slides: 62
Grammar of Image Zhaoyin Jia, 03 -30 -2009
Problems Enormous amount of vision knowledge: …… Computational complexity Classificatio n, Recognition Semantic gap
Task of image parsing
Objectives in this paper Framework for vision Algorithm for this framework Top-down/bottom-up computation Generalization of small sample And-Or Graph Use Monte Carlos simulation to synthesis more configurations Fill the semantic gap
Grammar Language: co-occurance of s is more than chance CONSTANTINOPLE Image: Parallel; T-junction
Formulation of grammar Start symbol: S Non-terminal nodes: VN Reproduction Rule: R Terminal nodes: VT
Formulation of grammar Start symbol: S Non-terminal nodes: VN Reproduction Rule: R Terminal nodes: VT
Formulation of grammar Start symbol: S Non-terminal nodes: VN Reproduction Rule: R Terminal nodes: VT S VP VP PP VP …… NP VP V NP
Formulation of grammar Start symbol: S Non-terminal nodes: VN Reproduction Rule: R Terminal nodes: VT
Formulation of grammar Start symbol: S Non-terminal nodes: VN Reproduction Rule: R Terminal nodes: VT
Image grammar Start symbol: S Reproduction Rules Non-terminal nodes: VN Terminal nodes: VT
Overlapping parts/Ambiguity
Overlapping parts/Ambiguity Similar color, occlusion, etc.
Stochastic Context Free Grammar For each VN , we have reproduction rules: with a probability associated with each one: Probability of parsing tree: Probability of sentence:
Stochastic Grammar with Context From left to right: bi-gram model (Markov chain) a sentence with n words: Non-local relations: tree model
New issues in Image Grammar Loss of “left to right” order: region adjacency graph
New issues in Image Grammar Scaling makes different terminal in parsing tree
New issues in Image Grammar Switch between texture and structure
Building the image grammar Visual Vocabulary: primitives, sketch graph, textons… Relations and configurations: co-occurance, attached, hinged, supported, occluded… And-or Graph representation embedding image grammar Learning /testing the parse graph find the possible inference
Database Lotus Hill Institute Dataset Benjamin Yao, Xiong Yang, and Song-Chun Zhu, “Introduction to a large scale general purpose ground truth dataset: methodology, annotation tool, and benchmarks. ” EMMCVPR, 2007 http: //www. imageparsing. com/ 636, 748 images, 3, 927, 130 Physical Objects A few hundred are free
Free Data http: //yoshi. cs. ucla. edu/yao/data/ 6 categories, 145 subsets Manmade Object 75 Nature Object 40 Objects in Scene 6 Transportation 9 UCLA Aerial Image 5 UIUC Sport Activity 10 Outline & segmentation of the object
Free Data http: //yoshi. cs. ucla. edu/yao/data/ 6 categories, 145 subsets Manmade Object 75 Nature Object 40 Objects in Scene 6 Transportation 9 UCLA Aerial Image 5 UIUC Sport Activity 10 Segmentation of a scene (street)
Free Data http: //yoshi. cs. ucla. edu/yao/data/ 6 categories, 145 subsets Manmade Object 75 Nature Object 40 Objects in Scene 6 Transportation 9 UCLA Aerial Image 5 UIUC Sport Activity 10 Physical parts of the object
Visual Vocabulary The “Lego Land” Language
Visual Vocabulary : function of image primitives : a) geometry transformation b) appearance : bond between each primitives
Visual Vocabulary Sketch and Texture S. C. Zhu, Y. N. Wu, and D. B. Mumford, “Minimax entropy principle and its applications to texture modeling, ” Neural Computation, vol. 9, no. 8, pp. 1627– 1660, November 1997
Primal sketch model Sketch graph Input image Texture pixels C. E. Guo, S. C. Zhu, and Y. N. Wu, “Primal sketch: Integrating texture and structure, ” in Proceedings of International Conference on Computer Vision, 2003.
Primal sketch model C. E. Guo, S. C. Zhu, and Y. N. Wu, “Primal sketch: Integrating texture and structure, ” in Proceedings of International Conference on Computer Vision, 2003.
High level visual vocabulary Cloth: collar, left/right sleeves, hands H. Chen, Z. J. Xu, Z. Q. Liu, and S. C. Zhu, “Composite templates for cloth modeling and sketching, ” in Proceedings of IEEE Conference on Pattern Recognition and Computer Vision, New York, June 2006
Relations and configurations Definition of relation: bonds: relations: , compatibility Three types of relations Bonds and connections Joints and junctions Object interactions/semantics Definition of configurations: : structure, :
Relations Bonds and connections connects primitives into bigger graphs intensity/color compatibility
Relations Joint and junctions
Relations Object interactions
Configuration Spatial layout of entities at a certain level Primal sketch – parts – object – scene
Reconfigurable graphs Treat bonds as random variables: address nodes
Inference of the configuration Have the primal sketch of the image Detect the ‘T-junction’ Simulated annealing to infer the Gestalt Law Red dot: connect region Black line: known edge Green line: inferred connection R. X. Gao and S. C. Zhu, “From primal sketch to 2. 1 D sketch, ” Technical Report, Lotus Hill Institute, 2006
Reconfigurable graphs Source image T-junction Inferred connection Layer extraction Ru-Xin Gao 1, Tian-Fu Wu, Song-Chun Zhu, and Nong Sang, “Bayesian Inference for Layer Representation with Mixed Markov Random Field ”
Reconfigurable graphs R. X. Gao and S. C. Zhu, “From primal sketch to 2. 1 D sketch, ” Technical Report, Lotus Hill Institute, 2006
And-Or Graph Parse graph of the image pt: parse tree of vocabulary E: relations Inference the parse graph: Z. J. Xu, L. Lin, T. F. Wu, and S. C. Zhu, “Recursive top-down/bottom up algorithm for object recognition, ” Technical Report, Lotus Hill Research Institute, 2007.
And-Or Graph Contain all the valid parse graphs And node, Or node, leaf -node Relation between children of And node Parse tree: assigning label on Or node Z. J. Xu, L. Lin, T. F. Wu, and S. C. Zhu, “Recursive top-down/bottom up algorithm for object recognition, ” Technical Report, Lotus Hill Research Institute, 2007.
And-Or Graph Definition: image primitives relations at all level : probability model defined on the And-Or graph : valid configuration of terminal nodes
Stochastic Model on And-Or graph Terminal (leaf) node: And-Or node: Set of links: Switch variable at Or-node: Attributes of primitives:
Stochastic Model on And-Or graph Terminal (leaf) node: And-Or node: Set of links: Switch variable at Or-node: Attributes of primitives: SCFG: weigh the frequency at the children of ornodes
Stochastic Model on And-Or graph Terminal (leaf) node: And-Or node: Set of links: Switch variable at Or-node: Attributes of primitives: Weigh the local compatibility of primitives (geometric and appearance)
Stochastic Model on And-Or graph Terminal (leaf) node: And-Or node: Set of links: Switch variable at Or-node: Attributes of primitives: Spatial and appearance between primitives (parts or objects)
Learning And-Or Graph Learning the vocabulary Learning the relation set R, given Learning the parameters , given R and
Learning And-Or Graph Learning the vocabulary , and hierarchic And-Or Graph Discussed in the Learning the relation set R, given paper Learning the parameters , given R and
Learning And-Or Graph Observation: Learning model: Learning and Pursuing Relation Set R: Start from Stochastic Context Free Graph (a) Learn the relations that maximally reduce the KL divergence to the observation (b-e) J. Porway, Z. Y. Yao, and S. C. Zhu, “Learning an And–Or graph for modeling and recognizing object categories, ” Technical Report, Department of Statistics, 2007
Learning And-Or Graph Learning graph parameter Approximating to Similar to texture synthesis S. C. Zhu, Y. N. Wu, and D. B. Mumford, “Minimax entropy principle and its applications to texture modeling, ” Neural Computation, vol. 9, no. 8, pp. 1627– 1660, November 1997
Case I: Rectangle Nodes: Rectangle Two vanishing points, four edge direction Rules: F. Han and S. C. Zhu, “Bottom-up/top-down image parsing by attribute graph grammar”. Proceedings of International Conference on Computer Vision, Beijing, China, 2005.
Case I: Rectangle Get the primal sketch of the scene Find the ‘strong’ rectangular (bottomup, red) Weigh (score) different hypothesis (top-down, blue) Weight is the compatibility of the image with the proposed rectangular (primalsketch) Accept the best one Do the previous 3 steps until all the weigh is small. (negative) F. Han and S. C. Zhu, “Bottom-up/top-down image parsing by attribute graph grammar”. Proceedings of International Conference on Computer Vision, Beijing, China, 2005.
Case I: Rectangle Inference process
Case I: Rectangle F. Han and S. C. Zhu, “Bottom-up/top-down image parsing by attribute graph grammar”. Proceedings of International Conference on Computer Vision, Beijing, China, 2005.
Case II: Human Cloth Use And-Or graph to generate a matching model Matching using the And-or Graph Vocabulary (training dataset)
Case II: Human Cloth The And-Or Graph Novel Configuration H. Chen, Z. J. Xu, Z. Q. Liu, and S. C. Zhu, “Composite templates for cloth modeling and sketching, ” in Proceedings of IEEE Conference on Pattern Recognition and Computer Vision, New York, June 2006.
Case II: Human Cloth Inference process Top-down: refine the matching using the relation Localize face, then estimate the parts of the body Bottom-up: a coarse matching of the parts H. Chen, Z. J. Xu, Z. Q. Liu, and S. C. Zhu, “Composite templates for cloth modeling and sketching, ” in Proceedings of IEEE Conference on Pattern Recognition and Computer Vision, New York, June 2006.
Case II: Human Cloth Inference result H. Chen, Z. J. Xu, Z. Q. Liu, and S. C. Zhu, “Composite templates for cloth modeling and sketching, ” in Proceedings of IEEE Conference on Pattern Recognition and Computer Vision, New York, June 2006.
Case II: Human Cloth Inference result Hands are not exactly the same: find the best matching in the dataset H. Chen, Z. J. Xu, Z. Q. Liu, and S. C. Zhu, “Composite templates for cloth modeling and sketching, ” in Proceedings of IEEE Conference on Pattern Recognition and Computer Vision, New York, June 2006.
Case III: Recognition Z. J. Xu, L. Lin, T. F. Wu, and S. C. Zhu, “Recursive top-down/bottomup algorithm for object recognition, ” Technical Report, Lotus Hill Research Institute, 2007.
Conclusion Enormous amount of vision knowledge: (Add-Or graph) ……
Conclusion Computational complexity : Remain open for scheduling bottom-up/top-down procedure Semantic Gap Learning the And-Or Graph Learning the vocabulary , and its attributes After all, we are not supposed to define so many things: ideal vision words: what we have now:
Thank you Zhaoyin Jia
- Zhaoyin jia
- How to convert right linear grammar to left linear grammar
- Weaknesses of traditional grammar
- Unrestricted grammar is also known as
- Right linear grammar to left linear grammar
- Harmony hui
- Ni jia you ji ge ren
- Ta zai
- Translate to chinese pinyin
- Jia zhu de rong yao
- 8hui jia.com
- 2ming2
- Jiangyong jia
- Zhihao jia
- Ni zai nar in english
- Boon lay primary school
- Nán tóngxué yǒu duōshǎo rén
- Oligoartiküler jia
- Ai jia
- Lim jia sheng
- Jia et al
- Jiabin chords
- Jia-bin huang
- Jiangyong jia
- Jiangyong jia
- Adjectives shifted out of order
- Image grammar examples
- Grammar brushstrokes
- Image grammar examples
- Adjectives shifted out of order
- Image grammer
- What are real and virtual image
- Real vs virtual images
- Image transform in digital image processing
- Optimum notch filter in image processing
- Fundamentals of image compression
- Image segmentation in digital image processing
- Analog image and digital image
- Variable length coding in digital image processing
- Image sharpening and restoration
- Static image vs dynamic image
- Image geometry in digital image processing
- Area of convergence
- Isopreference curve
- Appuyez sur l’image qui est dans le bon sens
- Difference between logical file and physical file
- Perturbação
- Image restoration is to improve the dash of the image
- Digital image processing
- Maketform matlab
- Image restoration in digital image processing
- 余采樺
- Generative adversarial networks
- 2008-2009 school year
- 2009 mathematics standards of learning answers
- Sistema integral de servicio social
- Uma máquina fotográfica custava r$ 400
- Legible meaning
- Cais.ecustoms
- Plan nacional del buen vivir 2009 al 2013
- Permenkes 51 tahun 2009
- R.t.t. 2009
- January 25 2018 chemistry regents answers