Learning Hub user research Forums The Learning Hub

Learning Hub user research: Forums The Learning Hub will be the place to go to access and record learning from a broad range of shareable resources, including existing e-Learning for Healthcare (e-Lf. H) programmes, and resources contributed and uploaded from the user community. The Learning Hub will also encourage discussions to take place around the uploaded learning content and networks to form around areas of shared interest to maximise the opportunities for collaboration and realise the potential of the resources. Over the past few months key areas of the Learning Hub’s design have been shared with users from all over the UK and from a variety of NHS, health and care backgrounds. The focus on user research ensures that users are central to the platform’s design and by listening to the needs of the users we can deliver a Learning Hub that meets the learning requirements of the health and care workforce. This pack provides an overview of both the process and findings of this piece of user research, which focussed on forums within the Learning Hub. January 2020 @HEE_TEL https: //telblog. hee. nhs. uk

Key questions • What form do users expect a forum to take in the Learning Hub? (Q&As on resources/Forum discussion board/Communities of practice groups etc. ) • • • How do users expect to interact with a forum? How often do users currently interact with forums? What are the main concerns around a Learning Hub forum? What is the ‘best’ design and the ‘worst’ design for a forum? Why would a user engage with a forum? What governance issues come into play? What type of moderation is expected? What are the key features? What are the needs for groups and how does this relate to privacy? @HEE_TEL https: //telblog. hee. nhs. uk

Participants • Total: 11 participants • 6 female, 5 male • Range of educator and learner backgrounds • Digital capabilities: – – – @HEE_TEL Advanced: Intermediate: Novice: Getting started: Unknown: 2 3 0 0 6 • Organisations include: – NHS (Education and Non-clinical) – HEE – e-Lf. H • Locations include: – East & West Midlands – London – Wales https: //telblog. hee. nhs. uk

Method • • Timeframe: Three weeks interviews and workshop, one week analysis Workshop (in-person in London offices) Half-day workshop using Despicable Design framework Remote follow-up interviews (conducted via Web. Ex) 45 minutes - 1 hour sessions Q&As based on responses from workshop For interviews, participants were in their work environments using their own devices. • Limitations: – Relatively low attendance for workshop (six participants) necessitated follow-up interviews – Working without prototypes meant clarifying user expectations forums and using real-life examples. @HEE_TEL https: //telblog. hee. nhs. uk

Findings @HEE_TEL https: //telblog. hee. nhs. uk

Key issues from workshop @HEE_TEL https: //telblog. hee. nhs. uk

In their own words… What would be your worst forum experience? ‘No ratings or comments or way to ensure evidence base’ ‘Poor working platform – no real layout, not user friendly, can’t search, etc. ’ ‘No filter for who can join forums, especially for closed/private groups’ ‘Quality of information: lack of accuracy, lack of evidence base, bad practice being shared and not picked up on’ @HEE_TEL ‘Unmoderated: bullying behaviour, inappropriate content’ What would be your best forum experience? ‘Pre-established forum threads/groups even when system is new’ ‘Merging of threads which are asking the same questions or asking for the same info’ ‘Reminders of Code of’ Conduct' ‘Filters: chronological, most responses, most liked, evidence-based, experience, etc. ’ ‘Log-in is email, password and ‘remember me’ – no need for a user ID’ ‘Well-trained moderators’ https: //telblog. hee. nhs. uk

Key issues from workshop • People want to see that moderation is taking place and how they are supposed to engage with it e. g. control of inappropriate content, avoiding duplicate threads, etc. This moderation should be both proactive and positive. There should be a clearly visible/accessible code of conduct for users. • Ability to create closed groups, invite users to those groups and for those groups to remain visible to the community at large with a 'request to join' option. Similarly, they should be able to direct message other users. • Search needs to be present and easy to use. Filters should be helpful and allow users to separate out expert content and user contributed/experience-based content/discussions. Being able to browse content was also a requirement as long as search was available to 'deep dive' into topics. • Usability was a big issue emphasised by the group. They wanted a clear, straightforward user interface that worked across multiple devices/browsers/systems. It should clearly direct users on how to search/post/create group etc. • Quality of information/content: this covered a wide range of features including proactive moderation, ratings on content and ability to feed back, badging of 'experts' or evidence-based posts, displaying accurate information about the poster (name, job role, etc), means of validating content or challenging inappropriate content. @HEE_TEL https: //telblog. hee. nhs. uk

Must-haves @HEE_TEL Search function Simple and accessible interface Pre-populated with content Ability to set notification preferences Usable on multiple platforms/devices/browsers Moderation and Help/Support https: //telblog. hee. nhs. uk

Appendix @HEE_TEL https: //telblog. hee. nhs. uk

Follow-up Q & As Question Answer Used some very basic, open forums to test this (e. g. https: //www. writingforums. com/forum. php). Layout of the forum homepage: what is expected? Majority responded positively to the familiarity of this design: • Topics and subtopics laid out down the page • Click in to see more subtopics/threads • Short info blurb about topics beneath each • Stats on numbers of threads/posts, last post highlighted, etc. . However, this design does not fit with the 'Communities of Practice' (which is primarily group-based) or the 'Q & A on resources' (where forums expand out from individual resources). Expected code of conduct (or a summarised version of it) to be displayed to users as they sign up to the platform. Users should also have to tick a box to agree to adhere to the code. Where should the code of conduct be visible and in what format? Highlighted that it's unlikely users will read at this point. Many suggested having a (small) link to the code of conduct available on every page/the majority of pages on the Learning Hub to allow users to check on the rules if they need to. Some suggestions for having short, snappy reminders of sections of the code of conduct displayed throughout the forum. When content is removed for breaking the code of conduct, a link to the code should be included with an explanation of why it's been removed. @HEE_TEL https: //telblog. hee. nhs. uk

Follow-up Q & As Question Answer Would users expect threaded conversations? No to threading. They would expect to be able to quote another user’s post when replying but their response would appear at the bottom of the thread as normal. Should there be collaboration on resources? Potentially, but participants highlighted that this was dependent on how the question was framed to users. If it was ‘Ask a question’ then they would expect a Q & A on resources but not long discussions. If it was ‘Start a debate’ this might be different. Q & A on resources was seen as distinct from a forum area for discussion. What is the user’s understanding of private and closed groups? @HEE_TEL Closed = visible to all (potentially with a description of purpose), any user can 'Request to join', users can also be invited from within the group. Private = invisible unless you are a member, users join by invite only (from admin/creator). https: //telblog. hee. nhs. uk

Follow-Up Q & As Question Answer Should there be 'direct messaging’ (DM) to groups of people or is this covered by group forum posts? Most agreed there should be one-to-one DMs but didn't see a need for sending a DM to multiple users as this overlapped with the purpose of groups/forums generally. Two main places where users thought they would access DMs: Clicking somewhere around another user's name (either on their name to open their profile or on a message button below their name in the forum) and having an option to send a DM. How do you access direct messaging? Where is it found on the main page? How do you create a new direct message? A 'messages' inbox or similar accessed either directly from the main menu of the Learning Hub (an envelope icon etc. ) or from within a user’s profile that allowed them to view all their DM conversations. Also, users expected that a DM to them would create a notification - on the platform and via email - to alert them, then clicking this would take them to the message. NB: Users expected to be able to turn off email notifications via settings. What key information do they need on a post e. g. to assure them that it is a valid/quality post? Important details on a post included full name of the user who posted it, their profession/job role and their organisation (if relevant). Profession was seen as particularly important as this applies to the subject being discussed and whether they have relevant expertise. On a thread/group, it should be clear who created it, when, and who is responsible for moderating it. Clicking into names should bring up something like a profile to give you more details, if available. @HEE_TEL https: //telblog. hee. nhs. uk

Follow-up Q & As Question Answer Happy for group creators to automatically become moderators of a group, as long as they could assign other moderators and as long as there were still 'system wide' moderators who users could go to if they had a problem with a group moderator. If someone has set up their own group, are they happy to self-moderate? Are they happy to be moderated by the creator of a group? Group creators moderating their groups was seen as a positive as it got more moderator oversight on more sections of the forums. Several highlighted that users might not want to be 'in charge' of a group for a very long time so there needed to be mechanisms for other admins to take over. Several said there was a possibility of bias with group creator moderation but that this would be mediated by users being able to go "further up the hierarchy" to systems admins. Moderators in a group should be clearly highlighted to users of that group. Most expected to be able to upload files to forums (separate from linking to a resource). There was a definite expectation to allow links to websites and uploading images. Would users expect to be able to upload documents into forums and share files that way? Where would they expect to be able to find these files? There wasn't an expectation for the forums to have its own 'file section' though a file uploaded to forums should be discoverable through search (provided it's not in a private group of which you are not a member). Lots of discussion around how users manage file uploads. A few suggestions that group moderators should be able to set when/if files can be uploaded by non-admin users of the group. Reason for being able to share files instead of links to resources: a group may be highly specialised/focused and the content being uploaded might not be of use outside of that group of people, so turning it into a learning resource wouldn't be appropriate e. g. minutes from a meeting. @HEE_TEL https: //telblog. hee. nhs. uk

Follow-up Q & As Question Answer Expected moderators to begin with a light touch (direct messaging the person who posted unacceptable content, negotiating). What actions should a moderator be able to take on a post or thread? Agreed they needed to: • Edit posts/threads • Remove a post/thread (leaving explanatory text in its place) • Remove users from a group • Merge threads/link relevant threads. Moderators of groups should also have responsibility for accepting/rejecting applications to join a group and sending invitations to potential members. Moderators should also be clearly labelled as such so that when they add a post about behaviour it's clear it's coming from a place of authority. Do users want/need alternate views of the forums page e. g. seeing only those discussions posted in the last week, only those they've posted in, etc. If these are required, what should they include and where should they be visible? @HEE_TEL Most didn't see a need for alternate views of the forum, as long as search allowed filtering by different options. Some suggested having 'recommended threads' or other highlighted content, alongside the structured topic-based layout of the forum. There may be a need to have a separate 'Groups' area in a tab. https: //telblog. hee. nhs. uk

Follow-up Q & As Question Answer It was expected that selecting a name in the forum would open that user's profile. Expected to access their own profile through a 'profile' button/icon in the main header menu of the platform. Where do you find your profile? What can you do on your own or someone else's profile? How much of this information is displayed to others? Expected basic information (full name, role, organisation) to be displayed to all and there were different views on what other information should be visible. Very few thought that platform usage information (e. g. what they've looked at, etc. ) would be useful to others. There were several suggestions for being able to include any publications/research you had been involved in. Most expected there to be some kind of free text section that you could choose to add to that would be visible to anyone visiting your profile. It was suggested that users could edit their own profile but not others, however they should be able to direct message other users via their profile. Is there an expectation that there is one search that covers resources and forums or separate searches, each with filters specific to either resources or forums? @HEE_TEL Most expected that if they use search (anywhere in the platform) that it should search everywhere. However, they agreed that the number of filters for different sections might increase the complexity of search - didn't think this was a problem as long as the filter options were under an 'advanced' label and laid out clearly. Many suggested a quick checkbox or equivalent beneath the search bar to quickly deselect a section of the platform to not search in (e. g. forums). It was also important how results were laid out for users, as well as making it clear what type of result each was (and again an easy way to filter down by type). Relevance of results was the core concern. https: //telblog. hee. nhs. uk

Follow-up Q & As Question Answer Explore the need for quotes, @messages, etc. Quoting posts/sections of posts created by other users was seen as very useful in order to give context to discussion and is expected from a forum. @messages were seen as a nice-to-have and have proven useful in other learning platforms (e. g. Future. Learn). However, direct messaging would cover its functionality at the beginning. Expertise was seen as very context specific. The important thing was to understand the person's role and experience. Profession/organisation was seen as important. Most didn't see a 'points based' system for highlighting experts as useful (i. e. earning an 'expert' label by your number of posts, upvoting etc. ) as this became a popularity contest that could be easily gamed and didn't take account of lurkers (i. e. quiet experts). Further detail on how experts would be chosen as an expert: is it user rating, number of posts, association to a particular body? What is the criteria for being labelled an expert? Multiple comparisons to the Future. Learn set up where 'subject matter experts' are chosen for a particular course and they are labelled as such. Inconclusive on whethere should be experts assigned by the platform/organisations on the platform - it would have to be clear what they were experts on and why they had been chosen. Profiles could help with this. Experts should be encouraged to have a more complete and engaging profile detailing their experience. Experts made more sense within a specialised group or attached to a particular learning resource. @HEE_TEL https: //telblog. hee. nhs. uk