Exploring Workflow Enactments Business Intelligence Revisited through Querying

Exploring Workflow Enactments Business Intelligence Revisited through Querying Execution Logs Jianwen Su Dept. of Computer Science UC Santa Barbara (Joint work with Yan Tang)

Outline n Business Workflow and Analytics (=BI) n Classical Approach to BI n Workflow Logs n Exploration n Research CBD'17 as An Example Challenges 2017/8/13 2

Business Processes & Workflow Management n A BP is an assembly of tasks v Eg: Obtaining a Permit Application BP Init review Review to accomplish an objective Approval Fee Certificate Delivery workflow Workflow Management (Wf. M) System n Each workflow model matches a BP model n Each workflow activity (☐) is a software program that interfaces one task in the BP n A Wf. M system manages executions, resources, documents, etc. CBD'17 2017/8/13 3

BP and Workflow: Another View Biz Proc Model Biz Proc Execution On-site Repair Application Repair Assignment documents, people, devices, resources, collaboration, … Post-repair Visit Application Review Doc. Archive Only software activities development are present Workflow Model Workflow Execution flow mgmt, services, db’s, software, resources, … CBD'17 2017/8/13 4

BP/Workflow: Change is Essential n Causes of changes: policy/regulation change, environment change, market, improvements, . . . n To incorporate changes, need to modify: v BP/workflow models v Databases v Wf. M systems v… Very hard problem but not the focus of here n To find opportunity for change from past executions v Business intelligence (BI) Needs data, where are data? CBD'17 2017/8/13 5

An Example from Univ of California n TC Travel Council oversees travel policies, programs, and travel related business services n Allowing Air. Bn. B: what steps in e. g. , reimbursement are affected? v State funding v Federal funding v Gifts and donations n Defined procedures in 10 campuses, 5 hospitals, & Office of President n Actual practices vary, need to find from past cases, e. g. travel reimbursement ? Where are data CBD'17 2017/8/13 6

BP and Workflow: Another View Biz Proc Model Biz Proc Execution On-site Repair Application Repair Assignment documents, people, devices, resources, collaboration, … Post-repair Visit Application Review Doc. Archive Only software activities development are present Workflow Model Workflow Execution data & documents flow mgmt, services, db’s, software, resources, … n Workflow execution generates a lot of data: biz data, execution status, resource usages, correlations, … CBD'17 2017/8/13 7

Workflow Management System (John Doe, 2017, UCORP meeting, Oakland, …) business processes . . . workflow manageme (John Doe, 2017, UCORP meeting, Oakland, …) data management CBD'17 2017/8/13 8

Where Are Data in Wf. M Systems n Typical Wf. MS [van der Aalst-van Hee 2004] (Pre-architecture [Bussler 1997]) architecture: Includes all data required for control flow decisions, correlations, … Task wrapper . . . Local data store Execution Engine Task wrapper Enterprise database Task wrapper n Logging often an option: v Workflow instance, activity (task), variables, …, etc all logged separately, sometimes in difference dbs CBD'17 2017/8/13 9

Outline n Business Workflow and Analytics (=BI) n Classical Approach to BI n Workflow Logs n Exploration n Research CBD'17 as An Example Challenges 2017/8/13 10

Traditional Approach to BI (Big Data) n Biz analytics (intelligence): Extract-Transform-Load inventory Transactions catalog Transactions activities Data Warehouse Analysis Biz Processes are missing! Transactions cust_db 11

A Few Details on ETL-OLAP n Typically, extract via relational queries (SQL, relational algebra, etc. ) on other relational databases v Could be from other types n Relational queries may be chained together and combined, results loaded into data warehouse n The types of queries are predetermined: v What data to get v How data from different sources are combined v Fixed semantics and representations of data CBD'17 2017/8/13 12

Example: Tao. Mart n Whole sales, dominant online sales n Could like to do market predictions based on past sales n Fact table: sales transactions n Dimensions: v Shopper demographics v Locations v Seasons v Classifications of goods v Possible others n OLAP: data cube operations n Does CBD'17 an excellent job for the types of queries, but… 2017/8/13 13

Hospital Referring Example [Tang-S. PED 17] receipt *Update referral Get reimbursement Complete referral get referral Check in See doctor Pay fees Q 1: Are there any students update referral? Q 2: Any students update referral then get reimbursement? CBD'17 2017/8/13 14

Weaknesses of ETL-OLAP n Analysis only limited to data extracted n Hard to support exploration/ad hoc queries n Loss of enactment information v Only some actions are represented in the loaded data v Temporal information is lost v Usually no correlations n When processes change, ETL must be reconfigured n Also, CBD'17 process mining techniques: no data 2017/8/13 15

Traditional BI Framework is NOT Flexible Lack flexibility ETL processes need to change, when data needed changes or the process model changes [Chaudhuri, Dayal SIGMODrecord 97] [Vassiliadis IJDWM 09] 16 CBD'17 2017/8/13

Outline n Business Workflow and Analytics (=BI) n Classical Approach to BI n Workflow Logs n Exploration n Research CBD'17 as An Example Challenges 2017/8/13 17

A Framework for Flexible Process Analytics General-purpose log structure No data selection Easy-to-use query language CBD'17 2017/8/13 18

Workflow Logs n Faithfully captures workflow executions n Includes the following information v workflow id v workflow instance id v activity instance id v timestamps v data (read/written) v correlation information v. . . n Granularity of logging 19 CBD'17 2017/8/13

Example Log: A Concrete Log Record n One log record per activity execution n Logical timestamp based log sequence numbers lsn wid is-lsn task name input map output map 4 1 3 Check. In refer. Id=034 d 1, refer. State=start, bal refer. State=acti ance=1000 ve Equivalent json: { "refer. Id": "034 d 1", "refer. State": "start", "balance": 1000 } Equivalent json: { "refer. State": "active", } 20 CBD'17 2017/8/13

Log Records within One Instance lsn wid is-lsn task name input map output map 1 2 1 start - - 5 2 2 Get. Refer - hospital=”People Hospital”, refer. Id=022 f 3, refer. State=start, balance=2000 8 2 3 Check. In refer. Id=022 f 3, refer. State=start, balance=2000 refer. State=active 13 2 4 See. Doctor refer. Id=022 f 3, refer. State=active - 14 2 5 Update. Refer refer. Id=022 f 3, refer. State=active, balance=2000 balance=5000 17 2 6 See. Doctor refer. Id=022 f 3, refer. State=active - 18 2 7 Pay. Fees refer. Id=022 f 3, refer. State=active receipt 1=4560, receipt 1 State=active 19 2 8 Take. Treatmen refer. Id=022 f 3, receipt 1=4560 t 20 2 9 Get. Reimburse CBD'17 - refer. State=active, balance=5000, amount=6560, balance=0, receipt 1=6560, receipt 1 State=active reimburse=5000, 21 receipt 1 State=complete 2017/8/13

Log Example: College Hospital Referring Application lsn wid is-lsn task name input map output map 1 1 1 start - - 2 2 1 start - - 3 1 2 Get. Refer - hospital=”Public Hospital”, refer. Id=034 d 1, refer. State=start, balance=1000 4 1 3 Check. In refer. Id=034 d 1, refer. State=start, balance=1000 refer. State=active 5 2 2 Get. Refer - hospital=”People Hospital”, refer. Id=022 f 3, refer. State=start, balance=2000 6 3 1 start - - 7 3 2 Get. Refer - hospital=”Public Hospital”, refer. Id=048 s 1, refer. State=start, balance=500 8 2 3 Check. In refer. Id=022 f 3, refer. State=start, balance=2000 refer. State=active 9 1 4 See. Doctor refer. Id=034 d 1, refer. State=active - 10 1 5 Pay. Feest refer. Id=034 d 1, refer. State=active receipt 1=560, receipt 1 State=active 11 1 6 See. Doctor refer. Id=034 d 1, refer. State=active - 12 1 7 Pay. Fees refer. Id=034 d 1, refer. State=active receipt 2=460, receipt 2 State=active 13 2 4 See. Doctor refer. Id=022 f 3, refer. State=active - 14 2 5 Update. Refer refer. Id=022 f 3, refer. State=active, balance=2000 balance=5000 15 1 8 Get. Reimburse refer. State=active, balance=1000, receipt 1=560, receipt 1 State=active, receipt 2=460, receipt 2 State=active amount=1020, balance=0, reimburse=1000, receipt 1 State=complete, receipt 2 State=complete 16 1 9 Complete. Refer refer. State=active, balance=0 refer. State=complete . . . . CBD'17 2017/8/13 22

Outline n Business Workflow and Analytics (=BI) n Classical Approach to BI n Workflow Logs n Exploration n Research CBD'17 as An Example Challenges 2017/8/13 23

Example Queries [Tang-S. 17] n Find instance ids where activity Update. Refer occurs before activity Check. In n Find the value of attribute balance where the activity Get. Refer occurs before Check. In with balance greater than 500 right after the get. Refer activity CBD'17 2017/8/13 24

Incident Query Language [Tang-S. 17] n Basic query language implemented n Optimization based on costs n Preliminary evaluation study (on the cost most and optimization n Still to do: v Multi-instance, multi-log queries v Aggregates CBD'17 2017/8/13 25

Outline n Business Workflow and Analytics (=BI) n Classical Approach to BI n Workflow Logs n Exploration n Research CBD'17 as An Example Challenges 2017/8/13 26

Research Problems and Challenges n Workflow n Query logs and (re-)construction languages for workflow logs n Process mining with data n Application CBD'17 techniques and tools 2017/8/13 27

Workflow Logs and (Re-)construction n No current standards on workflow logs/logging n Is there a universal model for logging? n Existing systems all have logging utility but v Spread through several types of logs (activity, instance, variables, etc. ) v Sometimes incomplete n It seems possible to construct workflow log from these logs, general tools and techniques? CBD'17 2017/8/13 28

Query Languages for Logs n Development of query languages v Understanding the properties, expressiveness, usefulness, etc. v Equivalence of log query languages v Indexing and optimization techniques n Aggregation? Multi-instances? Multi-log queries? CBD'17 2017/8/13 29

Process Mining with Data n Existing process mining techniques not compatible with data n Expressions in log query languages provide abstractions for data v Possibly combined with existing process mining algorithms CBD'17 2017/8/13 30

Application Techniques and Tools n Many existing tools are developed for ad hoc environment n Reporting tools, e. g. , v Ministry of Housing & Urban-Rural Development needs reports from local Housing Management agencies v Could be helped by reporting tools based on logs n Medical fraud detection n Staff training n Many other possibilities CBD'17 2017/8/13 31

Conclusions n Growing need for business intelligence beyond the traditional types (retail transactions) n ETL-OLAP is limited n Workflow logs lead to a general framework for business analytics n Many research problems and challenges CBD'17 2017/8/13 32