Applying Text Analytics to Examination of End Users

Agenda 1. 2. 3. 4. 5. 6. 7. 8. Motivation Applications of Mental Models

Motivation • Capture end user’s knowledge of cybersecurity using mental models based on written

Applications of Mental Models to Cybersecurity • human-centered security knowledge - behavior • cybersecurity

Research Questions • How text analytics can be used to elicit and compare the

Methodology • Represent mental models as a collection of adjacent concepts (bigrams) present in

Methodology: Individual Mental Model protection information computer hackers data systems protection 0 1 0

Methodology: Group Mental Model protection information computer hackers data systems protection 0 1 1

Measures to Differentiate Mental Models of End Users • Lexical [bigram] frequency to measure

Mental Model as a Predictor for Perceived Security • Perceived security to capture the

Hypotheses for Differentiating Mental Models of End Users • H 1 Measuring bigrams will

Hypotheses for Determining the Impact of Mental Models on Perceived Security • H 2

Survey Instrument OPEN-ENDED QUESTIONS to capture knowledge of cybersecurity concepts: Q 1 definition: What

Data Collection • Electronic survey • Administered from September 2017 to December 2018 •

Results: H 1, H 1. 1 and H 1. 2 Cosine Similarity of the

Results: H 2. 1 and H 2. 2 Linear Regression Results to Predict Perceived

Discussion and Implications • Inform cybersecurity awareness training – Learners with informal exposure to

Conclusions • Elicited end user cybersecurity knowledge using mental models • Compared end user

Limitations and Future Work • Collect larger data sets by leveraging on open data

Thank You! Contact us with questions or suggestions: acotoranu@pace. edu lchen@pace. edu Americas Conference

Slides: 31

Download presentation

Applying Text Analytics to Examination of End Users’ Mental Models of Cybersecurity Andreea Cotoranu and Li-Chiou Chen, Ph. D Seidenberg School of Computer Science and Information Systems Pace University Americas Conference on Information Systems, August 2020

Agenda 1. 2. 3. 4. 5. 6. 7. 8. Motivation Applications of Mental Models Research Questions Methodology Experiment Design and Data Collection Results Conclusions Limitations and Future Work Americas Conference on Information Systems, August 2020 2

Motivation • Capture end user’s knowledge of cybersecurity using mental models based on written responses to domainlevel questions • Provide a method to elicit and compare end user mental models of cybersecurity using text analytics • Test whether end user’s knowledge of cybersecurity, captured with our method, correlates with their perceived security Americas Conference on Information Systems, August 2020 3

Applications of Mental Models to Cybersecurity • human-centered security knowledge - behavior • cybersecurity training novice - experts • risk communication individuals - groups Americas Conference on Information Systems, August 2020 5

Applications of Mental Models to Cybersecurity • human-centered security knowledge - behavior • cybersecurity training novice - experts • risk communication individuals - groups need a method to automate the elicitation and analysis of mental models Americas Conference on Information Systems, August 2020 6

Research Questions • How text analytics can be used to elicit and compare the knowledge level of end users in cybersecurity topics by extracting their mental model? • How mental models can be used as a predictor of the end users’ perceived security? Americas Conference on Information Systems, August 2020 7

Methodology • Represent mental models as a collection of adjacent concepts (bigrams) present in the written text. • Example: End User Text Response Cybersecurity is the protection of data and information on computer systems. End User Mental Model cybersecurity-protection-data-information bigram 1 bigram 2 bigram 3 information-computer-system bigram 5 bigram 6 Americas Conference on Information Systems, August 2020 9

Methodology: Individual Mental Model protection information computer hackers data systems protection 0 1 0 0 information 0 0 1 0 0 0 computer 0 0 0 1 systems 0 0 0 Americas Conference on Information Systems, August 2020 11

Methodology: Group Mental Model protection information computer hackers data systems protection 0 1 1 0 0 0 information 0 0 1 1 0 0 computer 0 0 0 1 0 2 systems 0 1 0 hackers 0 0 0 Americas Conference on Information Systems, August 2020 13

Measures to Differentiate Mental Models of End Users • Lexical [bigram] frequency to measure lexical richness as an indicator of levels of complexity and elaboration in understanding of the topic • Shannon entropy to estimate the average number of bits needed to encode the mental model as a string of symbols based on the size of the dictionary and the frequency of the words. • Cosine similarity to measure the similarity between two groups of learners who have different levels of exposure to cybersecurity Americas Conference on Information Systems, August 2020 15

Mental Model as a Predictor for Perceived Security • Perceived security to capture the end user’s belief regarding the security of product, service or situation – we measure the end user’s perception of security as it relates to transactions in an online store – we generate a perceived security score based on three five-point Likert scale statements; an individual-based score is obtained by averaging the individual selections for the three statements. Americas Conference on Information Systems, August 2020 17

Hypotheses for Differentiating Mental Models of End Users • H 1 Measuring bigrams will show that learners with informal exposure to cybersecurity have statistically significantly different mental models from those with formal exposure to the topic. – H 1. 1 Learners with informal exposure to cybersecurity have statistically significantly different frequencies of lexical bigrams from those with formal exposure to the topic. – H 1. 2 The mental model of learners with informal exposure to cybersecurity has statistically significantly different entropy from those with formal exposure to the topic. Americas Conference on Information Systems, August 2020 18

Hypotheses for Determining the Impact of Mental Models on Perceived Security • H 2 Knowledge has an inverse correlation with perceived security. The more knowledge of cybersecurity a learner can articulate the lower their perceived security. – H 2. 1 Knowledge, as evidenced by lexical-based mental models (frequency), has an inverse relationship with perceived security. – H 2. 2 Knowledge, as evidenced by lexical-based mental models (entropy), has an inverse relationship with perceived security. Americas Conference on Information Systems, August 2020 19

Survey Instrument OPEN-ENDED QUESTIONS to capture knowledge of cybersecurity concepts: Q 1 definition: What is cybersecurity? [Q 1 -def] KNOWLEDGE TYPE Declarative Q 2 cyber hygiene: What makes a strong password? [Q 2 -hyg] Declarative Q 3 procedure: How is security achieved when shopping in an online store, such as Amazon. com? [Q 3 -pro] Procedural LIKERT-SCALE STATEMENTS to capture perceived security: SCALE S 1: Shopping on Amazon. com is secure. 1 Strongly Disagree - 5 Strongly Agree S 2: Amazon. com keeps its website secure. 1 Strongly Disagree - 5 Strongly Agree S 3: My information on Amazon. com cannot be compromised. 1 Strongly Disagree - 5 Strongly Agree Americas Conference on Information Systems, August 2020 20

Data Collection • Electronic survey • Administered from September 2017 to December 2018 • Participants – 136 undergraduate students enrolled at a private university in the Northeast • Participant Groups – identify two distinct groups within the respondents in order to study the effect on mental models of formal exposure to cybersecurity topics – Group A (n=66): either new or had no experience in the discipline or topic; enrolled in “Introduction to Computing” – Group B (n=70): had some experience in the discipline or topic; enrolled in “Overview of Computer Security” Americas Conference on Information Systems, August 2020 22

Results: H 1, H 1. 1 and H 1. 2 Cosine Similarity of the Mental Model of Groups A and B to examine Hypothesis 1 Mann-Whitney two-tailed U-test Results to Compare Learner Groups using Lexical Frequency and Lexical Entropy Americas Conference on Information Systems, August 2020 23

Results: H 2. 1 and H 2. 2 Linear Regression Results to Predict Perceived Security based on Mental Model using Lexical Frequency and Lexical Entropy Americas Conference on Information Systems, August 2020 25

Discussion and Implications • Inform cybersecurity awareness training – Learners with informal exposure to cybersecurity have statistically significantly different mental models from those with formal exposure to the topic. • Inform organizational risk management – Knowledge has an inverse correlation with perceived security. The more knowledge of cybersecurity a learner can articulate the lower their perceived security. Americas Conference on Information Systems, August 2020 27

Conclusions • Elicited end user cybersecurity knowledge using mental models • Compared end user mental models of cybersecurity given the end users’ exposure to cybersecurity topics (formal vs informal) • Predicted end users’ perceived security based on cybersecurity mental models Americas Conference on Information Systems, August 2020 29

Limitations and Future Work • Collect larger data sets by leveraging on open data sources • Capture different end user populations • Broaden the methodology to include machine learning algorithms Americas Conference on Information Systems, August 2020 30

Thank You! Contact us with questions or suggestions: acotoranu@pace. edu lchen@pace. edu Americas Conference on Information Systems, August 2020 31