Spatiotemporal Relational Constraint Calculi Debasis Mitra Florida Institute

Spatio-temporal Relational Constraint Calculi Debasis Mitra Florida Institute of Technology Melbourne, USA

Abstract It all began within the Natural Language Processing. We often use temporal expression like, "Maurya dynasty in India was before or overlapped the period of Qin Dynasty in China. " Only qualitative relation, no numbers, are involved, and the relation is disjunctive: “before or overlapped”. Take another example, “Heart is located above and left of liver. ” Formalizing such notions over both space and time led to a set of beautiful Spatio-temporal Qualitative Calculii. A few operators involved in reasoning with such qualitative spatio-temporal relations generated a class of relational algebras. In this talk I will briefly introduce some of these algebras and delve into a few projects that we have had an opportunity to contribute. I will also briefly raise some questions for future and mention some possibilities on how to utilize these results. 10/26/2021 AI Class 2

Motivation… n n n Maurya emperors ruled “before” or “overlapped” Qin dynasty Tang dynasty was “after” Qin dynasty Huen-Tsang lived “during” Tang dynasty Huen-Tsang visited India “during” or “overlapping” king Harsha’s rule What is the relationship between Maurya rule and the Tang dynasty? Answer: Maury “before” Tang 10/26/2021 AI Class 3

Constraint Network before | overlap Maurya Qin after Harsha Tang during | overlap during Huen-Tsang 10/26/2021 AI Class 4

Constraint Network n n n Network may be Inconsistent, and “local consistency” not enough Reasoning = Implied information (Inferencing) Reasoning involve operations: Converse, Compose, Intersect, Union before | overlap Maurya Qin ? (implied) after Harsha Tang after (implied) during | overlap during Huen-Tsang 10/26/2021 AI Class 5

Basics of the Calculi n (S, E, B, CT) • • 10/26/2021 S: Continuous, and Dense Space E: Basic entity B: Atomic relations CT: Composition table of basic relations AI Class 6

1. Point-based Calculus Vilain and Kautz, AAAI 1986 Space: One-dimensional time-line (R) Entity: Time point in the space Atomic/ Basic relations: { <, =, > } (B < A ) B 10/26/2021 A AI Class 7

Composition table of Point-calculus = B<C A<B = > 10/26/2021 > A <C < {<, =, >} < = > {<, =, >} > > AI Class 8

2. Interval Calculus n Space: Time-line (R) n Entity: Interval on R (Ordered 2 -tuple of points) n Atomic Relations: Allen, CACM 1983 10/26/2021 AI Class 9

Before Ab. B Before-inverse A b~ B Meet Am. B Meet-inverse A m~ B Overlap Ao. B Overlap-inverse A o~ B Start As. B A B B A A B Start-inverse A s~ B During Ad. B During-inverse A d~ B Finish Af. B Finish-inverse A f~ B 10/26/2021 Equal A eq B B A AI Class A 10 B

Interval Composition Table n n A 13 x 13 table Sample: ( A overlaps B ) & (B overlaps C) (A before | meets | overlaps C) A before meet overlap B C 10/26/2021 AI Class 11

Interval Composition Table. p m o F D s e S d f O M P p (p) (p) (pmosd) full m (p) (p) (p) (m) (m) (osd) (Fef) (DSOMP) o (p) (pmo) (pmo. FD) (o) (o. FD) (osd) concur (DSO) (DSOMP) F (p) (m) (o) (F) (D) (osd) (Fef) (DSO) (DSOMP) D (pmo. FD) (o. FD) (D) concur (DSO) (DSOMP) s (p) (pmo) (pmo. FD) (s) (se. S) (d) (df. O) (M) (P) e (p) (m) (o) (F) (D) (s) (e) (S) (d) (f) (O) (M) (P) S (pmo. FD) (o. FD) (D) (se. S) (S) (df. O) (O) (M) (P) d (p) (pmosd) full (d) (d) (df. OMP) (P) f (p) (m) (osd) (Fef) (DSOMP) (d) (f) (OMP) (P) O (pmo. FD) (o. FD) concur (DSO) (DSOMP) (df. O) (O) (OMP) (P) M (pmo. FD) (se. S) (df. O) (M) (P) (P) P full (df. OMP) (P) (P) 10/26/2021 (df. OMP) https: //www. ics. uci. edu/~alspaugh/cls/shr/allen. html 12

3. Cyclic-time… A B A overlaps B 10/26/2021 AI Class 13

4. Cardinal-directions Calculus Ligozat: AAAI 1998 n n n Space: R 2 Entity: Point Atomic relations (9) North EQ Northwest West Eq Northeast East Southwest Southeast South 10/26/2021 AI Class 14

5. Star-calculi Renz and Mitra: PRICAI 2004 n n Space: R 2 (4 n +1) atomic relations generated by n concurrent infinite lines 0 22 2 1 4 23 3 5 19 6 eq 18 7 8 17 9 15 11 10/26/2021 20 21 10 16 13 AI Class 12 14 15

Use of Star-calculus May be used for approximate shape representation, e. g. a 3 D protein’s backbone Cα‘s 20 16 0 10/26/2021 AI Class 16

Star-calculus to Approximately represent tracks Track Representation by sequence 9, 6, 0, … 10/26/2021 AI Class 17

Star-calc to Locate Organs Automatically • Need 3 D Star-calc • We have atlas, a standard anatomical map • Motivation: Seed segmentation algorithm 10/26/2021 AI Class 18

6. Region-connection Calculi (RCC-x) Randell, Cui, Kohn: KR 1992 n n Space: R 2 Atomic relations of RCC-5: Five basic relations between 2 closed sets A disjoint B 10/26/2021 A overlaps B A contains B AI Class A inside B A equal B 19

RCC-8 A contains B 10/26/2021 A contains-and-touches B AI Class 20

7. Region-connection Calculi (RCC-8) Renz: LNCS-2293 n Atomic relations: 8, by splitting RCC-5 relations • Distinguish between “inside” and “boundary” of a set n Problem with limits! What is “boundary”? • Disjoint, & Touches-from-outside • Inside, & Touches-from-inside • Contains, & Contains-and-touches 10/26/2021 AI Class 21

Reasoning b. o |m. o |b. d |m. d 10/26/2021 AI Class 22

Spatio-temporal Reasoning (STR) Problem n Input: Constraint network (V, E) • V: set of entities • E: labeled binary relations between entities (v 1, v 2, R 12) • R 12: disjunctive set of relations B n Output / Inference: Satisfiable/ Unsatisfiable n Output 2: In case of satisfiability: a “solution” n [Output 3: In case of unsatisfiability, “culprit” constraints] 10/26/2021 AI Class 23

Reasoning operators n Disjunctive composition B b |m o |d C A b. o |m. o |b. d |m. d 10/26/2021 AI Class 24

Reasoning operators n Converse B b |m B A 10/26/2021 b~ |m~ A AI Class 25

Reasoning/ Inferencing operators n Set Intersection b |m A B o |d b. o |m. o |b. d |m. d C … |… |… |… D 10/26/2021 AI Class 26

Reasoning/ Inferencing operators Set Intersection – leads to Path Consistency B b |m A o |d b. o |m. o |b. d |m. d C {b, m, o} o o D 10/26/2021 AI Class 27

Spatio-temporal Reasoning (STR) Problem n n Satisfiability: Does there exist a satisfying assignment? (NP-hard, except point-calc) Find one assignment: Singleton network (one constraint only on each arc) n All possible assignments / minimal network n Do: filter the network = constraint propagation • PC ≠> GC, but close (~90% in Interval Reasoning) 10/26/2021 AI Class 28

8. Quantitative Temporal Constraint Network (TCN) n n n Space: 1 D Time-line (global clock) Entity: Time-point Constraints: (1) Domain constraint, (2) Sets of convex intervals 10/26/2021 AI Class 29

Consistency issues n General case is NP-hard, • General case: not-necessarily convex, • e. g. , [42, 62] U [70 -75] Dechter-Meiri-Pearl, AI Jnl. 1991 n n n Simple Temporal Problem or STP: Constraints as only single convex intervals STP is tractable (≡ P-class) Floyd-Warshall algorithm works for STP with some preprocessing 10/26/2021 AI Class 30

Preprocessing [42, 62] to t 3 62 -42 t 0 10/26/2021 -11 44 -14 21 t 4 25 -5 t 2 AI Class 31

Reasoning on STP by Floyd-Warshall for All-pairs Shortest-path t 3 62 -42 t 0 10/26/2021 -11 44 -14 21 t 4 25 -5 t 2 AI Class 32

Improving Formal Verification with Timedautomata (TA) by using TCN-representation (with Dr. Bhattacharyya) 10/26/2021 AI Class 33

Merci! Debasis Mitra dmitra@cs. fit. edu • Mitra D, and Launay F. (2010) "Explanation Generation over Temporal Interval Algebra. " ISP Book: Ed. Hazarika • • Renz J, and Mitra D. (2004) “Qualitative Direction Calculi with Arbitrary Granularity. ” Pacific Rim Intl. Conf. on AI 2004 Ligozat G, Mitra D, Condotta JF. (2004) “Spatial and Temporal Reasoning: Beyond Allen's Calculus. ” AI Communications 10/26/2021 AI Class 34