Watson Robert Yates Watson Core Development Agenda A
Watson Robert Yates Watson Core Development
Agenda A brief History of Watson What is it good for? How does it work? Current Focus
A Brief History of Watson
A Brief History of Watson • Deep Blue Ended in 1997 • Looking for a new research challenge • 2004, IBM Research manager Charles Lickel, over dinner with coworkers, noticed that the restaurant they were in had fallen silent • Ken Jennings • Started in Research in 2005 • Won Jeopardy In Feb 2011 • SW Division in Aug 2011
What is it good for?
Watson’s real value proposition: Efficient decision support over unstructured (and structured) content Qu es Op tio en n-A -D ns oma we in rin Re g lev Ra an nk ce ing Ke y. W Se ord arc h Deeper Understanding, Higher Precision and Broader, Timely Coverage at lower costs Jeopardy! Challenge Shallow Understanding Low Precision Broad Coverage Unstructured Data Broad, rich in context Rapidly growing, current Invaluable yet under utilized I is Ex B ing t s le Ru / ce n e r e Inf ry ue Q L/X Q S Deeper Understanding but Brittle High Precision at High Cost Narrow Limited Coverage Structured Data Precise, explicit Narrow, expensive 7
How does it work?
Learning To Rank Systems Training data is used by a learning algorithm to produce a ranking model which computes relevance of responses for actual queries. Typically, users expect a search query to complete in a short time, which makes it impossible to evaluate a complex ranking model on each document in the corpus, and so a two-phase scheme is used. First, a small number of potentially relevant documents are identified using simpler retrieval models which permit fast query evaluation. In the second phase, a more accurate but computationally expensive machine-learned model is used to rerank these documents.
Learning To Rank Systems Training data is used by a learning algorithm to produce a ranking model which computes relevance of responses for actual queries. Typically, users expect a search query to complete in a short time, which makes it impossible to evaluate a complex ranking model on each document in the corpus, and so a two-phase scheme is used. First, a small number of potentially relevant documents are identified using simpler retrieval models which permit fast query evaluation. In the second phase, a more accurate but computationally expensive machine-learned model is used to rerank these documents.
Learning To Rank (Basic Architecture) Trained Models Keywords Search Hypothesis Evidence Scoring Final Merging Ranking Results
Learning To Rank (Watson) Wikipedia etc. Primary Search Candidate Answer Generation Answer Contextual Answer Scoring Answer Scoring Evidence Retrieval Trained Models Question Analysis Search Scoring Final Merging Ranking Answer, Confidence, Evidence
th 44 Who is the President of the United States?
Who is the 44 th President of the United States? Question Analysis
Who is the 44 th President of the United States? 'Who' is the '44 th' 'President' of the 'United States'? Who is the 44 th President of the United States? Question Analysis Lexical Answer Type → Person Focus Can be replaced by the correct answer to make a true statement Keywords
Who is the 44 th President of the United States? Wikipedia etc. Primary Search Candidate Answer Generation Answer Contextual Answer Scoring Answer Scoring Evidence Retrieval Trained Models Question Analysis Search Scoring Final Merging Ranking Answer, Confidence, Evidence
Who is the 44 th President of the United States? Keywords: 44 th President United States Primary Search Question Analysis Search
Who is the 44 th President of the United States? Keywords: 44 th President United States Primary Search Question Analysis Search
Who is the 44 th President of the United States? Wikipedia etc. Primary Search Candidate Answer Generation Answer Contextual Answer Scoring Answer Scoring Evidence Retrieval Trained Models Question Analysis Search Scoring Final Merging Ranking Answer, Confidence, Evidence
Who is the 44 th President of the United States? Primary Search Candidate Answer Generation Question Analysis Search
Who is the 44 th President of the United States? Barack Obama George W. Bush Harvard Law School Illinois Primary Search Candidate Answer Generation Question Analysis Search
Who is the 44 th President of the United States? Wikipedia etc. Primary Search Candidate Answer Generation Answer Contextual Answer Scoring Answer Scoring Evidence Retrieval Trained Models Question Analysis Search Scoring Final Merging Ranking Answer, Confidence, Evidence
Who is the 44 th President of the United States? Barack Obama Who is the 44 th President of the United States? George W. Bush 44 th Who is the President of the United States? Harvard Law School Who is the 44 th President of the United States? Illinois Answer Scoring Answer Contextual Scoring Answer Scoring Question Analysis Search Scoring
Who is the 44 th President of the United States? Barack Obama Who is the 44 th President of the United States? George W. Bush Who is the 44 th President of the United States? Harvard Law School Who is the 44 th President of the United States? Illinois Who is the 44 th President of the United States? → Person Answer Scoring Answer Contextual Scoring Answer Scoring Question Analysis Search Scoring
Who is the 44 th President of the United States? Barack Obama Who is the 44 th President of the United States? George W. Bush Who is the 44 th President of the United States? Harvard Law School Who is the 44 th President of the United States? Illinois Who is the 44 th President of the United States? → Person Answer Is Barack Obama a Person? . 90 Scoring Answer Contextual Scoring Answer Is George W. Bush a Person? . 90 Scoring Answer Scoring Is Harvard Law School a Person? . 10 Scoring Is Illinois a Person? . 15 Scoring Question Analysis Search Scoring
Who is the 44 th President of the United States? Wikipedia etc. Primary Search Candidate Answer Generation Answer Contextual Answer Scoring Answer Scoring Evidence Retrieval Trained Models Question Analysis Search Scoring Final Merging Ranking Answer, Confidence, Evidence
Who is the 44 th President of the United States? Contextual Answer Scoring Question Analysis Search Scoring
Who is the 44 th President of the United States? Barack Obama is the 44 th President of the United States George W. Bush is the 44 th President of the United States Harvard Law School is the 44 th President of the United States Illinois is the 44 th President of the United States Question Analysis Search Scoring Contextual Answer Scoring
Who is the 44 th President of the United States? Barack Obama is the 44 th President of the United States George W. Bush is the 44 th President of the United States Harvard Law School is the 44 th President of the United States Illinois is the 44 th President of the United States Question Analysis Search Scoring Contextual Answer Scoring
Who is the 44 th President of the United States? Barack Hussein Obama II (i/bəˈrɑːk huːˈseɪn oʊˈbɑːmə/; born August 4, 1961) is the 44 th and current President of the United States. George Walker Bush (born July 6, 1946) is an American politician who served as the 43 rd President of the United States from 2001 to 2009 and the 46 th Governor of Texas from 1995 to 2000. Barack Obama is the 44 th President of the United States George W. Bush is the 44 th President of the United States Harvard Law School is the 44 th President of the United States Illinois is the 44 th President of the United States Question Analysis Search Scoring Contextual Answer Scoring Barack Obama. 95 George W. Bush. 80 Harvard Law School. 05 Illinois. 10
Who is the 44 th President of the United States? Wikipedia etc. Primary Search Candidate Answer Generation Answer Contextual Answer Scoring Answer Scoring Evidence Retrieval Trained Models Question Analysis Search Scoring Final Merging Ranking Answer, Confidence, Evidence
Who is the 44 th President of the United States? Candidate Answer Scoring Contextual Answer Scoring Confidence Barack Obama 0. 90 . 95 George W. Bush 0. 90 0. 80 . 65 Harvard Law School 0. 10 0. 05 Illinois 0. 15 0. 10 Evidence Retrieval . 05. 10 Trained Models Question Analysis Search Scoring Final Merging Ranking
Who is the 44 th President of the United States? Candidate Answer Scoring Contextual Answer Scoring Confidence Barack Obama 0. 90 . 95 George W. Bush 0. 90 0. 80 . 65 Harvard Law School 0. 10 0. 05 Illinois 0. 15 0. 10 Evidence Retrieval . 05. 10 Trained Models Question Analysis Search Scoring Final Merging Ranking
Who is the 44 th President of the United States? Wikipedia etc. Primary Search Candidate Answer Generation Answer Contextual Answer Scoring Answer Scoring Trained Models Question Analysis Search Scoring Final Merging Ranking Answer, Confidence, Evidence
IEEE Xplore Library: “This is Watson” • http: //ieeexplore. ieee. org/xpl/tocresult. jsp? isnumber=6177717 • Recommended – “This is Watson”, For all: at least read this thoroughly. – “Question Analysis, How Watson Reads a Clue” – “Finding needles in the haystack, Search and Candidate Generation” – “Automatic Knowledge Extraction from Documents” – “Structured Data and Inference in Deep. QA” – “A framework for merging and ranking of answers in Deep. QA” • Read the others to broaden your knowledge 35
Current Focus
Current Customer Engagements • Medicine – Challenge: Body of Knowledge doubles every 5 years – Best Practice / Evidence Based Medicine – Oncology Treatments • Finance
Evidence Based Medicine • http: //www. anthem. com/medicalpolicies/mp_pw_a 053323. htm • Varicose Veins – Medically Necessary: – Endoluminal radiofrequency ablation or endoluminal laser ablation, of the greater saphenous vein (GSV) or lesser saphenous veins (LSV) is medically necessary when the following criteria are met: • Saphenofemoral or saphenopopliteal junction incompetence (i. e. , reflux) is demonstrated by Doppler or duplex ultrasound scanning; and • One or more of the following criteria (a, b, or c) are met: a) – – Symptoms of venous insufficiency or recurrent thrombophlebitis (including but not limited to: aching, burning, itching, cramping, or swelling during activity or after prolonged sitting) which: i. are interfering with activities of daily living; and ii. persist despite appropriate non-surgical management; and iii. persist despite a trial of properly fitted gradient compression stockings for at least six weeks; or There is ulceration secondary to stasis dermatitis; or There is hemorrhage from a superficial varicosity.
Treatment Example (Not Oncology) • A 44 -year-old businessman presents to a physician because of a markedly inflamed and painful right great toe. He states that he just returned from a convention, and noticed increasing pain in his right foot during his plane trip home. Physical examination is remarkable for swelling and erythema of the right great toe as well as small nodules on the patient's external ear. Aspiration of the metatarsalphalangeal joint of the affected toe demonstrates needleshaped negatively birefringent crystals. • What’s wrong with this patient? • How do you treat it?
Questions?
- Slides: 40
