Data acquisition and FIRST datasets Miha Grar Joef

Data acquisition and FIRST datasets Miha Grčar, Jožef Stefan Institute FIRST Y 3 Review Meeting

Activity in Y 3 Ontology evolution Data acquisition software (Dacq. Pipe) FIRST dataset of news & blogs FIRST Y 3 Review Meeting 2 Luxembourg, Nov 2013

Ontology evolution Dynamic part (Nearly) Static part Semantic & lexical resources, IDMS API Indices, stocks, companies, geo-entities, actors… Topic detection & tracking Topic taxonomies Active learning Sentiment vocabulary FIRST ontology FIRST Y 3 Review Meeting 3 Luxembourg, Nov 2013

Ontology evolution Dynamic part (Nearly) Static part Semantic & lexical resources, IDMS API Topic detection & tracking Active learning* FIRST ontology Models for canyon flow visualization Models for sentiment classification* “Knowledge base” *Smailović, Grčar, Lavrač, Žnidaršič: Stream-based active learning for sentiment analysis in the financial domain (to appear)

Data acquisition pipeline (Dacq. Pipe) Read & parse Syntactic analysis Clean RSS reader HTML tokenizer B'plate remover & duplicate detector Language detector Filter NLP pipe Semantic preprocessing OBIE Store Emit DB writer 0 MQ channel DB writer DB Resembles big data streaming architectures such as Twitter Storm Running continuously since April 2011 Several scientific contributions Boilerplate remover & gold standard dataset Ontology & ontology-based information extractor Executable available at http: //first. ijs. si/software/Dacq. Pipe. Jun 2013. zip Source code: https: //github. com/project-first/dacqpipe FIRST Y 3 Review Meeting 5 Luxembourg, Nov 2013

Data acquisition pipeline (Dacq. Pipe) Read & parse Syntactic analysis Clean RSS reader HTML tokenizer B'plate remover & duplicate detector Language detector Filter NLP pipe Semantic preprocessing OBIE Store Emit DB writer 0 MQ channel DB writer DB Resembles big data streaming architectures such as Twitter Storm Running continuously since April 2011 Several scientific contributions Boilerplate remover & gold standard dataset Ontology & ontology-based information extractor Executable available at http: //first. ijs. si/software/Dacq. Pipe. Jun 2013. zip Source code: https: //github. com/project-first/dacqpipe FIRST Y 3 Review Meeting 6 Luxembourg, Nov 2013

Boilerplate removal FIRST Y 3 Review Meeting 7 Luxembourg, Nov 2013

Streaming setting FIRST Y 3 Review Meeting 8 Luxembourg, Nov 2013

Hypothesis Web pages at similar Web addresses share common boilerplate, while main content is unique FIRST Y 3 Review Meeting 9 Luxembourg, Nov 2013

URL Tree “About us” http: //www. bbc. co. uk/sports/story 2371. html How many times did I see “About us” in this part of the tree? Stream FIRST Y 3 Review Meeting 10 Luxembourg, Nov 2013

Evaluation Dataset 569, 583 time-stamped documents (stream) 292, 053 documents after URL normalization Oct 24 – Dec 19, 2011; 31 Web sites Part of the FIRST dataset of news & blogs Gold standard 56, 436 documents annotated with manually designed regex tailored for specific Web sites FIRST Y 3 Review Meeting 11 Luxembourg, Nov 2013

Evaluation Reset FIRST Y 3 Review Meeting 12 Luxembourg, Nov 2013

Gold standard dataset http: //first. ijs. si/urltreedataset FIRST Y 3 Review Meeting 13 Luxembourg, Nov 2013

Conclusion: Final results of WP 3 Data acquisition pipeline software (Dacq. Pipe) Since April 2011 https: //github. com/project-first/dacqpipe FIRST dataset of news & blogs 219 Web sites; ~15 million unique documents http: //first. ijs. si/FIRSTDataset FIRST ontology Semantic + lexical part Information extraction + sentiment analysis http: //first. ijs. si/FIRSTOntology/y 3 FIRST Y 3 Review Meeting 14 Luxembourg, Nov 2013

Technical Presentations and Demos - Sentiment Analysis Achim Klein (UHOH), 20 November, Luxembourg

Knowledge-based Sentiment Extraction a) Direct sentiment Example: „I expect the S&P 500 to rise“ positive sentiment Addressed by rules b) Indirect sentiment, using indicators Example: „I think U. S. interest rates will rise“ negative sentiment Addressed by ontology

UC Retail Brokerage/Market Surveillance: Economic Indicators Debt to Equity Dividend Yield Earnings to Price Ratio New Products Profit Margin Sales … Interest Rate Inflation M 2 Change Rate Durable Goods Orders Unemployment Private Housing New Building Permits … Advance/Decline Ratio Bear Flag Break Out Double Bottom RSI Support Resistance …

Example Insights: Unemployment Indicator 180 160 Official US unemployment statistics release dates. Record Greek unemployment numbers released. 140 120 100 80 Unemployment Indicator Volume 60 40 20 0 1. 1. 2013 4. 1. 2013 7. 1. 2013 10. 1. 2013

UC Reputational Risk: Reputation Indicators (Y 3) Reputation Indicator Social Responsibility Human Resources Business Behavior Corporate Governance Exposure on Critical Markets Charity Positive Donation Correlation Education Professional Positive Talent Correlation Manpower Transparent Positive Responsible Correlation Campaign Accountable Positive Tier 1 ratio Correlation AML Subsidy Positive Liquidity Correlation Customers Crime Negative Bullying Correlation Slave Lay off Negative Job cuts Correlation Wrongdoers Debt Negative Foreclosure Correlation Price-fixing Breach Negative Shady funds Correlation Law suit Subprime Negative Mortgage Correlation CDS spread Total number of indicators: 1451 Positive and negative sample indicators per reputation topic

Reputation Sentiment Classification Performance 90% Without Indicators 80% 70% 67, 7% 67. 7% With Indicators 71. 2% 71, 2% 60% 50% 44, 9% 44. 9% 40% 30% 23, 7% 23. 7% 20% 10% 0% Precision Recall Higher recall of (indirect) sentiments by means of indicators

13. 09. 2013 14. 09. 2013 15. 09. 2013 16. 09. 2013 17. 09. 2013 18. 09. 2013 19. 09. 2013 20. 09. 2013 21. 09. 2013 22. 09. 2013 23. 09. 2013 24. 09. 2013 25. 09. 2013 26. 09. 2013 27. 09. 2013 28. 09. 2013 29. 09. 2013 30. 09. 2013 01. 10. 2013 02. 10. 2013 03. 10. 2013 04. 10. 2013 05. 10. 2013 06. 10. 2013 07. 10. 2013 08. 10. 2013 09. 10. 2013 11. 10. 2013 12. 10. 2013 13. 10. 2013 14. 10. 2013 15. 10. 2013 16. 10. 2013 17. 10. 2013 18. 10. 2013 19. 10. 2013 20. 10. 2013 21. 10. 2013 22. 10. 2013 23. 10. 2013 24. 10. 2013 25. 10. 2013 26. 10. 2013 27. 10. 2013 28. 10. 2013 29. 10. 2013 30. 10. 2013 31. 10. 2013 01. 11. 2013 02. 11. 2013 03. 11. 2013 04. 11. 2013 “Scandals cost 250 JPMorgan $1 billion in fines” 200 [REUTERS] Volume of Corporate Governance Reputational Insights: JPMorgan Corporate 300 Governance 19. 09. 2013 Corporate Governance 11. 10. 2013 “JPMorgan’s Dimon Posts First Loss on $7. 2 Billion Legal Cost” [BLOOMBERG] 150 100 50 0

Fuzzy Sentiment Classification 1. Extract sentiment objects „Apple‘s earnings are rising“ „Sales might decrease because of the financial crisis“ 2. Classify sentiment per object in each sentence 3. Generate machine-learning input: Sentiments and words of all sentences that refer to the same object 4. Two separate documentlevel machine-learning fuzzy classifiers with 5 degrees of … (1) positive, (2) negative

Enhanced Gold Standard Corpus (Y 3): Retail Brokerage/Market Surveillance Corpus size 1200 80% 1021 70% 1000 60% 800 50% 62, 4% 62. 4% 54, 2% 54. 2% 40% 600 400 62, 6% 62. 6% 69, 0% 69. 0% 30% 409 20% 10% 200 0% Precision 0 Y 2 Y 3 Y 2 Recall Y 3 Improved hybrid sentiment classifier performance

Main Results Deep knowledge-based sentiment analysis Specific to a feature of an object using rules (e. g. , reputation of a company) Economic and reputation indicators improve classifier performance and provide valuable insights for users Glass-box approach with drill-down capabilities Best paper award at IEEE Fuzzy classifier with 5 degrees of positivity and negativity for better decision making Fuzzy-level Gold Standard Corpus Analyzed >3 million documents Open source available git: //github. com/project-first/semanticinformationextraction. git

Thank you

WP 6 Technical Presentation & Demos Marko Bohanec, Miha Grčar, Jan Muntermann, Michael Siering Luxembourg, November 20 th, 2013

WP 6 Status End of Y 2 19 20 21 22 23 24 25 26 27 • Mainly presenting basic stand-alone prototypes • Presentation of the first models • First visualisation components FIRST Y 3 Review Meeting 28 29 30 31 32 33 34 35 36

WP 6 Achievements Y 3 19 20 T 6. 2 / T 6. 3 T 6. 4 21 22 Machine Learning & Qualitative Models Visualisation Components 23 24 25 26 28 29 30 31 32 33 34 35 36 • Refinements of qualitative models based on domain experts’ feedback • Highly scalable implementations • FIRST pipeline integration • Delivery of D 6. 3 in M 33 • Development of additional and revised visualisation components based on domain experts’ feedback • Highly scalable implementations • Delivery of D 6. 4 in M 34 FIRST Y 3 Review Meeting 27

Agenda Qualitative and quantitative models Reputational Risk Management Market Surveillance Visualizations Retail Brokerage 29 FIRST Y 3 Review Meeting

Reputational Risk Problem Formulation (1/2) General Area: Production and distribution of investment products and services by banks and other financial institutions. Specific Use Case: Assessment of reputational risk (RI) based on assessments of MPS counterparties. Reputational Risk: Risk arising from negative perception on the part of customers, counterparties, shareholders, investors, debt-holders, market analysts, other relevant parties or regulators that can adversely affect a bank’s ability to maintain existing, or establish new, business relationships and continued access to sources of funding. FIRST Y 3 Review Meeting

Reputational Risk Problem Formulation (2/2) Goal: to develop • a multi-criteria model for the assessment of MPS reputational risk (RIM) • that serves as the main component of corresponding DSS novelties Approach: expert modeling, qualitative multi-attribute modeling (method DEX) FIRST Y 3 Review Meeting

RIM: Main Components FIRST Y 3 Review Meeting

RIM: Basic Data Processing FIRST Y 3 Review Meeting

RIM: Qualitative Evaluation Aim: qualitative assessment of Reputational Index for one customer and product Model: qualitative hierarchical rule-based DEX model FIRST Y 3 Review Meeting

RIM: Aggregation Aim: gradually aggregate q. RI 1 into the overall Reputation Risk Index (RI): • hierarchical aggregation: Customer → Product → Counterpart → Bank • taking into account relative product volumes and relative customer numbers C/P → PRODUCT q. RI 1 FIRST Y 3 Review Meeting PRODUCT → COUNTERPART → BANK

RIM Reports: Topmost Level (Bank) FIRST Y 3 Review Meeting

RIM Summary Developed and implemented a decision support model component for the assessment of bank reputational risk Approach: expert modeling using a variety of modeling methods (qualitative, quantitative, hierarchical, relational) Novel aspects: taking into account sentiment assessments of counterparts advancing the present RI assessment model Benefits for the users: obtaining a comprehensive RI as time series for different groups (customers, products, counterparts, bank) ability to analyse and explain assessments at different levels by drilling down through the RIM hierarchy FIRST Y 3 Review Meeting

Agenda Qualitative and quantitative models Reputational Risk Management Market Surveillance Visualizations Retail Brokerage FIRST Y 3 Review Meeting

Problem Formulation: Market Surveillance Pump & Dump market manipulation: Manipulation of the share price by the dissemination of false positive information in order to take profit from an increased price level. FIRST Y 3 Review Meeting

Pump & Dump Example (1/2) „Shares can multiply dramatically in value over short time periods. “ „Could this company be the next blockbuster? “ „Thursday's pick is a story straight out of Hollywood!“ „SAPX - Wake Up, Put It On Your Screen NOW“ Source: http: //newsletter. hotstocked. com/newsletters/view/Could_this_company_be_the_next_blockbuster_-92301 FIRST Y 3 Review Meeting

Pump & Dump Example (2/2) Seven Arts Entertainment, Inc. (SAPX) Shares Purchased Shares Sold Pump & Dump campaign July, 24 th – 28 th 2011 > 30 different recommendations Source: Yahoo! Finance FIRST Y 3 Review Meeting

How to address Pump & Dump Manipulations? Qualitative Modeling Quantitative Modeling Based on expert knowledge Based on machine learning Qualitative attributes algorithms Quantitative attributes Goal: assessment of single documents Decision problem divided into sub problems Goal: daily assessments FIRST Y 3 Review Meeting

Qualitative Multi-Attribute Model Development Country Black List Industry Black List Company Age Bankrupt Market Segment History Comp_Fin. Inst Market Capitalization Trading Volume Number of Trades Sentiment Content FIRST Y 3 Review Meeting Financial Instrument Trading News Pump & Dump

From initial DEXi Model (M 24) to Processing of Data Stream (M 33) Initial development of the model structure was distributed as DEXi- files. Models can be applied within DEXi-environment only (M 24). To address of the models capability to process large-scale data streams, a JAVA-based prototype was implemented (M 33). FIRST Y 3 Review Meeting

Definition of Data Sources Regulatory Authorities web pages FIRST Y 3 Review Meeting

Model Configuration and Evaluation (M 24) Model Configuration • V-high: 3 configurations • High: 9 • Medium: 7 • Low: 5 • V-low: 1 Evaluation • 1700 OTC-traded companies • Dataset: 01. 2012 to 06. 2013 (370 trading days) • on average 157 alerts per day for vhigh and high Evaluation based on predefined configuration: FIRST Y 3 Review Meeting v-high Number of P&D Percentage values 482 0. 66 57588 78. 75 med low v-low 12342 2498 215 16. 88 3. 42 0. 29 Sum 73125 100

Reconfiguration of the Rules in Y 3 FIRST Y 3 Review Meeting

Model Configuration and Evaluation (M 33) Configuration: • V-high: 3 configurations • High: 7 • Medium: 8 • Low: 6 • V-low: 1 Evaluation: • 1700 OTC-traded companies • Dataset: 01. 2012 to 09. 2013 (435 trading days) • on average 53 alerts per day for v-high and high Evaluation results based on reconfigured model: v-high med low v-low Sum FIRST Y 3 Review Meeting Number of Percentage P&D values 982 0. 8 22215 18. 8 92049 2779 57 78. 0 2. 4 0. 0 118082 100

Research Impact Alic, I. ; Siering, M. ; Bohanec, M. (2013) Hot Stock or Not? A Qualitative Multi. Attribute Model to Detect Financial Market Manipulation Proceedings of the 26 th Bled e. Conference; Bled, Slovenia FIRST Y 3 Review Meeting

How to address Pump & Dump Manipulations? Qualitative Modeling Quantitative Modeling Based on expert knowledge Based on machine learning Qualitative attributes algorithms Quantitative attributes Goal: assessment of documents Decision problem divided into sub problems Goal: daily assessments FIRST Y 3 Review Meeting

Research Objective: Development of a Pump & Dump Classifier Learning phase: Labeled documents Training algorithm ++ - Evaluation of Application Training phase: Documents ? Classifier Support Vector Machine Evaluation of Machine Learning Algorithms Integration in FIRST Pipeline not suspicious ? New documents FIRST Y 3 Review Meeting Classifier suspicious Predictions

Evaluation of Training Documents Event study: capital market reaction during / after pump and dump campaign significant abnormal returns during campaign price decrease after campaign has ended Siering, M. (2013) All Pump, No Dump? The Impact of Internet Deception on Stock Markets In: Proceedings of the 21 st European Conference on Information Systems; Utrecht, Netherlands FIRST Y 3 Review Meeting

Evaluation of Machine Learning Algorithms Evaluation of different machine learning algorithms Decision Tree Accura cy 95. 10 Class suspicious Precis Recall F 1 ion 94. 65 95. 60 95. 12 Class non-suspicious Precis Recall F 1 ion 95. 56 94. 60 95. 08 Naïve Bayes 97. 30 96. 28 98. 40 97. 33 98. 36 96. 20 97. 27 k-NN, k =1 k-NN, k=2 k-NN, k=3 k-NN, k=4 k-NN, k=5 Neural Network SVM 78. 10 73. 60 75. 30 74. 20 75. 10 97. 00 80. 09 68. 67 77. 32 71. 68 77. 34 96. 81 74. 80 86. 80 71. 60 80. 00 71. 00 97. 20 77. 35 76. 68 74. 35 75. 61 74. 03 97. 00 76. 36 82. 07 73. 56 77. 38 73. 20 97. 19 81. 40 60. 40 79. 00 68. 40 79. 20 96. 80 78. 80 69. 59 76. 18 72. 61 76. 08 96. 99 99. 30 99. 20 99. 40 99. 30 99. 40 99. 20 99. 30 Neural Network: reduced feature set SVM: parameter optimisation according to 500 450 400 350 300 250 200 150 100 50 0 Decision Tree Naive Bayes k-NN Neural Network Computing requirements of different learners (10 -Fold Cross Validation, in sec. ) Hsu et al. (2003) Hsu, C. W. , Chang, C. C. , & Lin, C. J. (2003). A practical guide to support vector classification. National Taiwan University, http: //www. csie. ntu. edu. tw/~cjlin/papers/guide. pdf (accessed on 10/16/2011). FIRST Y 3 Review Meeting Support Vector Machine

Integration in FIRST Pipeline Internal Data Sources Database External Data (Web) Sources HTML preproc. FIRST Y 3 Review Meeting Boilerplate removal Language detection Nearduplicate removal Quant. Pump and Dump Model Clean text HTML pages Data sources

Qualitative Models: Multi Classifier Approach Country Black List Industry Black List Company Age Bankrupt Market Segment History Comp_Fin. Inst Market Capitalization Trading Volume Number of Trades Sentiment Content FIRST Y 3 Review Meeting Financial Instrument Trading News Pump & Dump

Market Surveillance Summary Developed and implemented multi-classifier decision support component for the assessment of information-based market manipulation Approach: expert modeling using a variety of modeling methods (qualitative, quantitative, hierarchical, relational) and machine learning Novel aspects: taking into account sentiment assessments of published documents multi-classifier component integrates qualitative and quantitative model Benefits for the users: obtaining the ability to monitor information-based market manipulation in market segments with a large number of financial instruments. FIRST Y 3 Review Meeting

Agenda Qualitative and quantitative models Reputational Risk Management Market Surveillance Visualizations Retail Brokerage FIRST Y 3 Review Meeting

Visualisation Components emo D e Onlin o Vide FIRST Y 3 Review Meeting

Retail Brokerage Summary Developed and implemented visualisation components providing the basis for data/document-driven DSS in the Retail Brokerage use case scenario. Approach: Clustering of document topics, aggregation of document sentiments and publication statistics. Novel aspects: Visualisation components that condense social media activity Aggregation of media topics to explore social media contents Benefits for the users: Explore activity, sentiment and topics of social media in a userfriendly way FIRST Y 3 Review Meeting

Thank you for your attention! Questions? FIRST Y 3 Review Meeting