Sage Innovators
A How-To Webinar Series
Teaching Research Methods to Prepare Students for Careers in Behavioral Science and Data Science with Gorilla
Teaching Research Methods to Prepare Students for Careers in Behavioral Science and Data Science with Gorilla Experiment Builder
In this webinar, Johanna Tomczak, Research Specialist at Gorilla Experiment Builder, shares insights into the most cutting-edge approaches and practical examples for teaching research methods, illustrating where and how these techniques can be applied. She discusses how online research has enabled both staff and students to reach larger samples and gather data more quickly, positively impacting both teaching and career preparation. By embedding research directly into teaching methods, students experienced the full scope of behavioral science research firsthand - from study design and participant recruitment to data analysis. Watch on to discover innovative strategies and best practices to create a dynamic learning environment where students are actively engaged in the research process, equipping them with valuable skills for their professional futures.
About the Speakers
Johanna Tomczak works as a Research Specialist at Gorilla Experiment Builder. She completed her degree in Mathematics, Computer Science and Cognitive Science at the University of Grenoble (France), before obtaining her PhD in Cognitive Psychology from the University of Leeds (UK) in 2022. Her thesis, which examined the impact of bilingualism and ageing on the communication between the cerebral hemispheres, has shown her the benefits of online research and she decided to join Gorilla to work on improving behavioral science as a whole.
Additional Resources
Teaching Research Methods with Gorilla Blog
What over 1,000,000 participants tell us about online research protocols
The Kings Business School class of 2021
UCL charitable giving project
Gorilla Grants
Gorilla Presents Webinars
BeOnline Conference
Support:
Previous Sage Partnered Gorilla Webinars:
Q&A
+ Does Gorilla provide tools to simplify data analysis for students?**
Yes, Gorilla’s platform is designed to make data analysis approachable. It encourages students to think about their research questions and hypotheses first, which naturally informs their data analysis plans. This approach can make statistics more intuitive, as students see how their choices relate directly to their research questions.
+ What are some best practices for engaging students in research through teaching?**
A hands-on approach, where students can create their own research tasks, tends to be effective. We’ve found that students enjoy the applied aspect of Gorilla, and it helps reduce "stats anxiety" by making data analysis more accessible and relevant to their research.
+ Does Gorilla provide resources for creating practical teaching simulations with dummy data?**
Yes, Gorilla offers curricula and guides, particularly in psychology. These materials help teachers set up courses, analyze data, and streamline the learning process. We also have an online support system that answers students’ questions, reducing the burden on instructors.
+ Can Gorilla capture audio files, such as gathering verbal responses from participants?**
Yes, Gorilla can capture audio responses. You can set up tasks to record participants’ voices, and once you download your data, these audio files will be included. Learn more here and here.
+ How easy is it to import experiments from other software, like E-Prime, into Gorilla?**
Currently, Gorilla doesn’t have an E-Prime integration. However, there is a Qualtrics importer for survey and questionnaire imports. We’re also working on integrations with other task-building tools. If there’s significant demand for E-Prime integration, users can suggest it for our development roadmap.
+ Can Gorilla be used in clinical research settings, like medicine or surgery?**
Absolutely. An example is a study where radiology students used Gorilla to analyze CT scans, identifying potential medical issues. Gorilla recreated the CT scan environment, allowing for interactive tasks similar to what they’d use on medical devices. This setup can be adapted to various clinical training scenarios.
+ Can Gorilla be used in GIS (Geographic Information System) applications, such as environmental health data collection?**
While Gorilla primarily supports cognitive psychology and related fields, it may support integration with GIS tools if they can connect via USB. For specialized setups, please contact Gorilla for more guidance.
+ How does Gorilla’s pricing work, and does it cover participant recruitment?**
Gorilla has different subscription tiers, including department and lab packages. Recruitment fees for services like Prolific or MTurk are separate. Alternatively, users can distribute a study link for unpaid participants at no extra cost.
+ Is there a university-wide subscription option available for Gorilla?**
Yes, Gorilla offers bespoke university-wide subscriptions in addition to department, lab, and individual researcher plans. Find out more about pricing here.
+ Does the researcher subscription limit data collection to 200 participants, and can additional responses be purchased?**
The researcher subscription includes data for 200 participants, but additional “top-up” packs for more participants are available.
Explore How To Webinar Series
Online research advanced tools: Multiplayer, gamified experiments and shopping simulations with Gorilla
Online research advanced tools: Multiplayer, gamified experiments and shopping simulations with Gorilla Experiment Builder
In this webinar, Johanna Tomczak, Research Specialist at Gorilla Experiment Builder, showcases powerful tools like Game Builder, Multiplayer, and Shop Builder. Discover how advanced online research tools are transforming the way behavioral science experiments are conducted! Learn how these tools create engaging, ecologically valid experiments without needing coding skills. Whether you're studying consumer behavior or cognitive processes, these features can help elevate your research by motivating participants and ensuring higher-quality data.
About the Speakers
Johanna Tomczak works as a Research Specialist at Gorilla Experiment Builder. She completed her degree in Mathematics, Computer Science and Cognitive Science at the University of Grenoble (France), before obtaining her PhD in Cognitive Psychology from the University of Leeds (UK) in 2022. Her thesis, which examined the impact of bilingualism and ageing on the communication between the cerebral hemispheres, has shown her the benefits of online research and she decided to join Gorilla to work on improving behavioral science as a whole.
Additional Resources
Gorilla Game Builder:
Gorilla Shop Builder (a winner of the 2023 Sage Concept Grant):
Gorilla Grants
For £50 research credit
Gorilla Presents Webinars
BeOnline Conference
Support:
Previous Sage Partnered Gorilla Webinars:
Q&A
+ How can researchers recruit participants for their studies on Gorilla?
Gorilla offers various recruitment services that integrate natively, such as Prolific, Amazon Mechanical Turk, and Sonar systems. If you prefer not to use an external service, you can create a link to your study and share it via social media or email. This method allows anyone with access to the link to participate in your study. Many students prefer this approach as it doesn't involve additional fees.
+ Can Gorilla be used for both qualitative and quantitative research?
Absolutely! Gorilla supports both qualitative and quantitative research. We have a survey tool for creating qualitative surveys, and participants can record responses via webcam. For quantitative research, Gorilla captures reaction times, accuracy, and interaction data, such as mouse clicks. All participant actions are logged as part of the dataset, providing a comprehensive range of data.
+ What research domains can benefit from using Gorilla?
Gorilla can be applied across many domains, including military research, teaching, economics, and security analysis. For example, we’ve had researchers study attention in military pilots using online experiments. The Shop Builder tool is great for examining consumer behavior, and Game Builder allows for gamified research. Social behavior can also be studied using multiplayer features. Gorilla can even simulate websites, such as online shops, to study user interactions and reaction times.
+ How is statistical significance determined when using Gorilla?
Determining statistical significance depends on your specific research design. For example, in a study comparing three different shop designs, you might calculate average reaction times and use ANOVA or t-tests to compare the results across participant groups. While the specifics vary by field, Gorilla provides guides to help users with data analysis.
+ Are there resources available for learning how to create gamified experiments in Gorilla?
Yes, we have a recorded webinar on our website that walks you through setting up a game using Gorilla’s Game Builder tool. This hands-on tutorial explains everything from start to finish. Additionally, our support page contains extensive resources on all Gorilla tools, making it easy to find the information you need.
+ What is Gorilla's fee structure?
Gorilla has a tiered pricing model. It’s free to use for building and testing experiments. When you start collecting data, pricing kicks in, and you're charged per dataset (participant). We offer a pay-as-you-go option or subscription plans for labs and departments. Many UK universities already have subscriptions in place, but pricing details are available on our website if needed.
Explore How To Webinar Series
Online Research Advanced Tools: Social media & Website simulations with Gorilla
Online Research Advanced Tools: Social media & Website simulations with Gorilla Experiment Builder
In our latest webinar, Johanna Tomczak and Max Mawby shared their expertise on combating misinformation through cutting-edge inoculation methods. Johanna, a Research Specialist at Gorilla Experiment Builder with a background in Cognitive Psychology, and Max, a leading practitioner in behavioral science with extensive experience in both the private and public sectors, provided valuable insights into their approaches. If you missed it, watch the full video recording and review the Q&A session to learn about their strategies for enhancing digital literacy and influencing positive change.
About the Speakers
Johanna Tomczak works as a Research Specialist at Gorilla Experiment Builder. She completed her degree in Mathematics, Computer Science and Cognitive Science at the University of Grenoble (France), before obtaining her PhD in Cognitive Psychology from the University of Leeds (UK) in 2022. Her thesis, which examined the impact of bilingualism and ageing on the communication between the cerebral hemispheres, has shown her the benefits of online research and she decided to join Gorilla to work on improving behavioral science as a whole.
Max Mawby is a leading practitioner in applying insights from behavioral science to drive positive change. He heads Thinks Applied Behavioral Science team with extensive experience of designing and running online experiments for government and industry clients. Max is passionate about applied behavioral science and how this can improve our lives. In particular, he is interested in driving positive financial behaviors, sustainable decisions, and using behavioral insight to make design inclusive for all.
Most recently, Max conducted an online randomized controlled trial (RCT) to explore the threat posed by election disinformation ahead of the 2024 UK general election. This focused on a behavioral approach to testing and addressing simple interventions with the potential to help limit the spread of mis and disinformation.
Additional Resources
Thinks Insight and Strategy: Addressing disinformation online case study
Due diligence support information
What over 1,000,000 participants tell us about online research protocols
Webinar about improving participant engagement
Webinar about gamification with Dr Drew McLaughlin
Previous Sage Partnered Gorilla Webinars:
Q&A
+ What is the best approach to rolling out an inoculation intervention to effectively reach people and change their behavior on social media?
The key challenge with rolling out inoculation interventions is effectively reaching people in a way that actually influences their behavior on social media. While online research offers a controlled environment, applying these findings in real-world settings remains complex. The real challenge lies in scaling and distributing these interventions to achieve meaningful impact.
+ Can you discuss the concept of confounding variables and how they can affect the validity of research findings?
Confounding variables are factors that might influence the outcome of a study but are not controlled or accounted for. In behavioral research, it's nearly impossible to control every variable, and some may not even be recognized until after the study is conducted. This inherent limitation means that researchers must acknowledge the potential for confounding variables and the compromises they make in controlling their research environment.
+ How can researchers address the challenge of confounding variables when conducting online research?
Online experimentation and simulation offer a controlled environment that helps minimize confounding variables. By using platforms like Gorilla, researchers can manage and control various aspects of the study, leading to cleaner data sets and more reliable results. Although some confounds may still exist, online research generally provides a clearer and more manageable setting compared to field studies.
+ How do online research methods compare to traditional lab-based research in terms of controlling the research environment?
Online research methods, while offering a controlled environment, often create an illusion of control compared to traditional lab settings. In labs, researchers can more directly observe participants and control variables. However, online platforms allow for a wide reach and can manage many variables effectively, making them a useful tool for obtaining clean data and conducting research at scale.
+ How do you ensure data quality and participant authenticity in online research?
Ensuring data quality and participant authenticity online involves implementing checks and attention measures within the research design. Despite not being able to physically observe participants, online platforms can use various techniques to verify participant identity and engagement, leading to more accurate and reliable data.
+ Are there tutorials on Youtube?
Yes, here.
Explore How To Webinar Series
Teaching Research Methods in a Digital World with Gorilla
Teaching Research Methods in a Digital World with Gorilla Experiment Builder
Dive into the future of research education with our latest webinar Teaching Research Methods in a Digital World with Gorilla. Led by Jo Evershed, Johanna Tomczak, and Dr. Miles Tufft, this session highlights innovative strategies and concrete examples reshaping research methodologies for the digital age. An essential watch if you’re an educator training the next generation of researchers: stay at the forefront of behavioral science education!
About the Speakers
Jo Evershed is the Founder CEO of Gorilla Experiment Builder, a powerful, flexible and intuitive platform for running behavioural research online that allows researchers to go far beyond surveys. She also convenes and hosts the Behavioural Science Online Conference. As an Innovate UK Women in Innovation Award Winner and Computer Weekly Rising Star Women In Tech, Jo is on a mission to provide behavioural scientists with tools to liberate their work from the lab and accelerate research initiatives that can be tested rigorously at scale. Jo leads a multidisciplinary team of software engineers and psychologists focused on creating powerful and accessible experimental research infrastructure.
Johanna Tomczak works as a Research Specialist at Gorilla Experiment Builder. She completed her degree in Mathematics, Computer Science and Cognitive Science at the University of Grenoble (France), before obtaining her PhD in Cognitive Psychology from the University of Leeds (UK) in 2022. Her thesis, which examined the impact of bilingualism and ageing on the communication between the cerebral hemispheres, has shown her the benefits of online research and she decided to join Gorilla to work on improving behavioural science as a whole.
Dr Miles Tufft is a Lecturer in Experimental Psychology and co-director of the Eyethink Lab at University College London. He graduated with a BA in Natural Sciences from the University of Cambridge and holds an MSc in Cognitive Neuroscience and a PhD in Cognitive Psychology from UCL. Prior to academia, he worked for over a decade in industry including seven years as a management consultant where he applied data science to inform strategic and organisational change. His research examines the perceptual and attentional mechanisms that underpin distributed cognitive systems. Currently, he investigates the ways in which information deemed to be the responsibility of others in joint contexts is actively prioritised or de-prioritised to optimise individual or team performance (social offloading). He has received a Provost Teaching Award from UCL and currently convenes research methods on the BSc Psychology programme. He also works extensively with industry partners to help embed enterprise and knowledge exchange activities into the undergraduate curriculum.
Additional Resources
Gorilla Grants - expression of interest
Due diligence support information
Webinar about improving participant engagement
What over 1,000,000 participants tell us about online research protocols
Teaching Faculty mentioned:
Gillian Brooks: Students’ Presentation and her Module
Miles Tufft: Open Teaching Materials
Rachel Theodore: Her Experience
Previous Sage Partnered Gorilla Webinars:
Q&A
+ Which operating system or browser do I need to use for Gorilla?
Gorilla operates seamlessly in web browsers, accessible from various devices, including smartphones, for both study creation and data collection. Comprehensive support resources are available, including tutorials on YouTube and documentation on the Gorilla website. These resources provide thorough guidance for users, ensuring smooth navigation and utilization of the platform.
+ Are participants restricted by location, or can you recruit internationally?
Participants can be anywhere globally.
+ Is Gorilla applied in qualitative research?
This platform offers a range of survey components for qualitative research, including open text responses, multiple choice questions, and image, audio and video stimuli. It allows for various data collection methods, such as talk-out-loud protocols and video recordings, even enabling the creation of sign language corpora. Subsequent analysis can be conducted outside the platform, offering flexibility for researchers.
+ How is data security ensured?
We meet stringent data security standards required by universities worldwide and are fully GDPR-compliant, ensuring robust protection for human data. With extensive experience across 1,000 universities, we undergo rigorous assessments and maintain top-tier servers subject to annual penetration testing. Our platform offers comprehensive data processing agreements, privacy policies, and tools for anonymous data collection and minimization. We prioritize data security, providing customizable options to safeguard sensitive information effectively.
+ How do you ensure that the participants are not bots or bad actors?
We offer various options to control for factors like attention checks in tasks, allowing researchers to filter out incomplete data effectively. Using Gorilla, you only pay for complete participants, minimizing costs associated with incomplete responses. Online research with Gorilla generally shows minimal evidence of checkbox or bot issues, ensuring data quality.
+ In using Gorilla for teaching, do you have sort of a user manual that people can use to get started?
Our platform supports student success with onboarding tutorials and resources, including the Gorilla Academy and Gorilla support pages. Many universities use Gorilla for undergraduate teaching, enabling students to engage in research projects and gain valuable skills. We offer ample samples and materials, alongside webinars teaching effective navigation and modification. Gorilla Open Materials provide additional content for exploration and extension of research projects.
+ How does utilizing Gorilla assist students in grasping research methods such as counterbalancing, randomization, and experimental controls? Specifically, how does this interaction facilitate understanding, and what benefits does it offer in terms of comprehension?
Engaging students with Gorilla effectively enhances their understanding of research methods such as counterbalancing and randomization. Rather than solely conceptualizing these complex ideas, interacting with them in a tangible way on the platform allows students to grasp them more deeply. By actively manipulating elements on the screen, students can experiment, make mistakes, and refine their understanding, leading to a more embodied comprehension of research design concepts. This approach is vital as it shifts the focus from mere mechanics to broader thinking about good research design and question formulation, essential aspects of experimental psychology. Ultimately, by offloading technical aspects onto the platform, students can concentrate on developing robust research questions and collecting meaningful data, which is crucial for their growth as experimental psychologists.
+ Can you give some more information about eye-tracking?
Yes, eye tracking in Gorilla is feasible through webcam technology. It allows researchers to measure participants' gaze location on the screen, albeit with some differences compared to in-lab eye tracking. Despite potential noise in the data, online eye tracking still yields comparable results to lab-based studies. This method is particularly useful for experiments involving visual stimuli, such as comparing participant gaze between different images on the screen. Comparisons between lab-based and online eye tracking data show similar trends, with variations in noise levels across different environments, like classrooms and home settings.
+ Are there tutorials on Youtube?
Yes, here.
Explore How To Webinar Series
Online Research for PhDs and PIs - New Possibilities - with Gorilla Experiment Builder
Online Research for PhDs and PIs - New Possibilities - with Gorilla Experiment Builder
Welcome to this post dedicated to exploring the transformative potential of online research, in partnership with Gorilla Experiment Builder. In the wake of the COVID-19 pandemic, the landscape of behavioral research has evolved dramatically, with digital platforms offering unprecedented opportunities for innovation and collaboration. Join us as we delve into the dynamic world of online research, highlighting the invaluable insights shared by experts from Gorilla Experiment Builder. From cutting-edge tools to addressing common concerns, this post serves as a guide for researchers seeking to harness the full capabilities of digital methodologies. Let's revolutionize the way we conduct behavioral science research in the 21st century.
About the Speakers
Jo Evershed is the Founder CEO of Gorilla Experiment Builder, a powerful, flexible and intuitive platform for running behavioural research online that allows researchers to go far beyond surveys. She also convenes and hosts the Behavioural Science Online Conference. As an Innovate UK Women in Innovation Award Winner and Computer Weekly Rising Star Women In Tech, Jo is on a mission to provide behavioural scientists with tools to liberate their work from the lab and accelerate research initiatives that can be tested rigorously at scale. Jo leads a multidisciplinary team of software engineers and psychologists focused on creating powerful and accessible experimental research infrastructure.
Johanna Tomczak works as a Research Specialist at Gorilla Experiment Builder. She completed her degree in Mathematics, Computer Science and Cognitive Science at the University of Grenoble (France), before obtaining her PhD in Cognitive Psychology from the University of Leeds (UK) in 2022. Her thesis, which examined the impact of bilingualism and ageing on the communication between the cerebral hemispheres, has shown her the benefits of online research and she decided to join Gorilla to work on improving behavioural science as a whole.
Additional Resources
Due diligence support information
Webinar about improving participant engagement
Previous Sage Partnered Gorilla Webinars:
Q&A
+ Is Gorilla applied in qualitative research?
This platform offers a range of survey components for qualitative research, including open text responses, multiple choice questions, and image, audio and video stimuli. It allows for various data collection methods, such as talk-out-loud protocols and video recordings, even enabling the creation of sign language corpora. Subsequent analysis can be conducted outside the platform, offering flexibility for researchers.
+ How is data security ensured?
We meet stringent data security standards required by universities worldwide and are fully GDPR-compliant, ensuring robust protection for human data. With extensive experience across 1,000 universities, we undergo rigorous assessments and maintain top-tier servers subject to annual penetration testing. Our platform offers comprehensive data processing agreements, privacy policies, and tools for anonymous data collection and minimization. We prioritize data security, providing customizable options to safeguard sensitive information effectively.
+ How do you ensure that the participants are not bots or bad actors?
We offer various options to control for factors like attention checks in tasks, allowing researchers to filter out incomplete data effectively. Using Gorilla, you only pay for complete participants, minimizing costs associated with incomplete responses. Online research with Gorilla generally shows minimal evidence of checkbox or bot issues, ensuring data quality.
+ In using Gorilla for teaching, do you have sort of a user manual that people can use to get started?
Our platform supports student success with onboarding tutorials and resources, including the Gorilla Academy and Gorilla support pages. Many universities use Gorilla for undergraduate teaching, enabling students to engage in research projects and gain valuable skills. We offer ample samples and materials, alongside webinars teaching effective navigation and modification. Gorilla Open Materials provide additional content for exploration and extension of research projects.
+ From a Research Ethics perspective, do you have automatic debrief material following the use of deception during research data collection, built into the software?
Researchers are responsible for debriefing participants appropriately after experiments, given the vast range of methods and experiments conducted. The Experiment Builder allows researchers to create debriefing materials to be presented at the end of the experiment, enhancing the participant experience and gathering valuable feedback. It's good practice to solicit feedback from participants on their experience to improve future experiments. Utilizing the debrief screen for feedback can greatly enhance the quality of research outcomes.
+ Do you provide official statements on participant confidentiality for IRB purposes?
Our data security policies and confidentiality can be found on this due diligence page.
+ How does Gorilla access participant data and is an additional disclaimer necessary for participant consent?
As a data processor, we only access data with explicit permission granted by users. This access is limited to situations where users request assistance, implicitly granting permission to our support desk. Users retain control over their data, and we only access it to support research purposes with explicit user consent. We adhere strictly to contractual agreements and ethical guidelines regarding data access and privacy.
+ When studying reaction time, how do you ensure/control for internet speed?
On Gorilla, researchers can restrict participants to have certain connection speeds to be able to access their experiments. The online timing accuracy page provides a useful resource to review the differences in terms of timing between online and lab-based research.
+ What are some innovative online research methodologies that you've found effective for PhDs and PIs?
Replicating lab studies online ensures data quality and comparison. Utilize resources like onboarding webinars for tool familiarization. Researchers can then explore theoretical extensions, like age or cultural differences, using multiplayer tools or gamified simulations to enhance research depth and applicability.
+ Can you share examples of successful online research projects conducted by PhDs or PIs?
A lot of them can be found on the BeOnline conference page, hosted by Gorilla.
+ What advice do you have for PhD students and PIs looking to integrate online research into their projects?
Exploring diverse experiments on platforms like Gorilla offers valuable insights. Utilize resources like onboarding webinars and support documentation to familiarize yourself with tools and capabilities. Start by replicating lab studies online to ensure data quality and comparison. Then, explore theoretical extensions using multiplayer tools or gamified simulations to broaden studies and enhance research findings.
+ Could I do a longitudinal study where the same participants log-in and complete a task once a week for many weeks and it can keep track of these participants (mixed model design, both between and within-subjects)?
Of course, this is definitely possible. Researchers can design multi-session studies on Gorilla and participants will be prompted to complete the next session once the time has come for them to be invited back.
+ It seems your tool can eliminate, for example, lab bias (people often behave differently in lab environments than in natural environments - and today, most of us feel naturally in front of a screen). On the other hand, the ability to control the course of the experiment could be minimal. How do we know the respondent conducted the experiment as it should have been? In other words, does Gorilla offer tools to eliminate respondents' biases? In the case of eye-tracking, how are measurement calibration and accuracy ensured?
A good starting point could be to think about the illusion of control we usually have in lab-based experiments. More specifically, there are several researchers who have already tried eye-tracking in the browser – successfully! Gorilla offers calibration.
+ Could we “manipulate” other parts of the 'shop builder' e.g. the Privacy notice?
Most things in Gorilla are customisable, including the Shop Builder settings. Have a look at the complete guide with Gorilla’s tools and services.
Explore How To Webinar Series
Connecting Your Research Findings with Protolyst
Connecting your rsearch findings with Protolyst
In this tutorial, we delve into the world of Protolyst, a powerful networked knowledge app designed to revolutionize the way you collect and organize your research findings. Protolyst allows you to connect individual pieces of knowledge, enabling you to identify patterns and themes in your data effortlessly. By developing a coding frame and collating crucial insights from various sources, you can build an interconnected network of knowledge that accelerates progress in your projects. Protolyst is a must-try tool for researchers seeking to streamline their qualitative analysis and make data management more efficient.
About the Speakers
Dr. Maddy Nichols is the Commercial Director & Cofounder of Protolyst. Following a PhD Maddy moved into startup operations, experiencing first hand the challenges of efficiently organising knowledge and insights.
Richard Barnes is the CEO & Cofounder of Protolyst. Richard built the Protolyst app and continues to lead development working closely with researchers, including through workshops in collaboration with the University of Southampton.
Additional Resources
Protolyst is free to try and you can get 20% off a Pro Subscription with the code SAGE20.
For Workshops, 1-2-1 Walkthroughs or Feedback, please click HERE, or email Protolyst.
Q&A
+ With citing and referencing, does it work with reference managers like EndNoote, Zoterro and RefWorks?
Protolyst does have doi and bibtex import, together with automatic citation extraction, and we are interested in exploring direct integrations with these platforms.
+ Can you sort the data by age of participant and another demographic like gender? I'm trying to see how this compares to NVivo?
Yes, that's the beauty of the Tables in Protolyst - you can use a range of property types such as numbers, dropdowns and text associated with your pages. You can then add sorts and filters that leverage these properties so you can work with your data in distinct "views".
+ Would the highlight option work with pdf files, such as papers?
Yes, it will work with PDF files, they are our primary file type. Specific to papers, we have citation functionality that automatically extracts citations from papers so you can also generate bibliographies using Protolyst. These citations are also reflecting in your outputted reports like just showed. Other file types supported: YouTube videos (so you can "atomise" the recording of this session!), web pages, word docs, PowerPoint files.
Explore How To Webinar Series
Reading to Write Effective Research Papers with Scrintal
Revolutionize the way you organize and extract insights from research papers by harnessing the Zettelkasten method with Scrintal.
In today's fast-paced world of academia, the sheer volume of research papers can be overwhelming. It's not just about collecting data; it's about effectively organizing the insights we extract. Enter the Zettelkasten method, a hidden gem in the realm of knowledge organization, originally pioneered by the prolific sociologist Niklas Luhmann. We're excited to join forces with Scrintal to unveil the remarkable effectiveness of the Zettelkasten method in mapping your ideas and the knowledge you extract from the literature. In this blog post, we'll guide you through the art of connecting your research findings using the Scrintal and Zettelkasten note-taking method. Step by step, you’ll be shown how to transform your initial ideas into meticulously crafted research papers.
The speaker, Ece (the CEO, and founder of Scrinta), showcases how this method can revolutionize the way you organize and extract insights from your growing pile of research papers. She demonstrates how to harness the Scrintal and Zettelkasten note-taking method to connect your research findings seamlessly. She walks you through each step, transforming your initial ideas into well-crafted research papers.
Explore the power of knowledge organization in this tutorial as we delve into the Zettelkasten method, a brilliant concept developed by renowned sociologist Niklas Luhmann.
About the Speaker
Ece Kural is the CEO and founder of Scrintal; she started the company while writing her dissertation and out of frustration with current tools that didn’t save any time nor facilitate the overwhelm of information. Ece earned her PhD in International Relations from Stockholm University. You can follow Ece on Twitter.
Additional Resources
For 10% off a Scrintal purchase, click HERE
Alternatively, visit www.scrintal.com and use the code SAGE10
Q&A
+ Does Scrintal require a subscription? Are there discounted plans for academics/ students? Is there an institutional access option?
Please use the code SAGE10 for 10% discount or click on this link to have 10% discount automatically added to your purchase.
+ Is it compatible with the iPad pencil?
Scrintal doesn’t have a native tablet app, but if you open Scrintal in the browser on your iPad, you can use your pencil.
+ How is Scrintal different from Miro?
In Scrintal, the notes or cards are not just documents, they can turn into full documents, you can also embed anything into each of these cards. A key difference between the two applications is that in Scrintal, you can easily search through all your notes.
+ Can you import notes from Endnote for example into Scrintal?
Yes.
+ How can you keep track of references in the notes? Does Scrintal integrate with Zotero for example?
You can add your reference to every note you create on Scrintal. The integration with Zotero and other reference management tools is on the roadmap and will be prioritized based on demand.
Explore How To Webinar Series
Master Scientific Writing with Wisio AI Assistance
This tutorial introduces Wisio, an AI-powered application that enables you to improve your research paper or an academic article.
This tutorial introduces Wisio, an AI-powered application that enables you to improve your research paper or an academic article.
Watching the recording below, you will learn how to use the application to improve a research paper you are currently writing. It benefits researchers and students whose primary language is not English or those learning the structure of academic writing. Wisio is based on OpenAI’s GPT models and can be used successfully for any discipline.
How to use wisio.app to improve your research paper.
About Wisio
Wisio is an AI-powered writing assistant for scientists. Get AI suggestions, find relevant papers, translate your text, and more an AI-powered platform for scientific writing.
About the Speakers
Antonio Carlos Filho is a full-stack engineer and founder of wisio.app. You can find him on twitter or LinkedIn.
Q&A
+ Is there a discount code for accessing wisio?
Yes, please use SAGE50 whene you sign up.
+ How can wisio be used to support graduate students in particular?
Yes, the wisio.app platform can be used to support every type of student interested in writing any scientific document.
+ How can wisio be used to teach (AI, academic writing, etc.)?
The platform can be used to write essays and papers for undergrad classes, but we don’t offer tools for teachers, like evaluation, collaboration, etc.
+ How can wisio help me write in English?
wisio.app has a translation and grammar correction tool that improves and translates from any language to scientific English.
+ How can research articles be published using Wisio.AI app?
The platform offers a general AI that is trained on scientific articles, aiding in the overall scientific writing process. Based on this, the platform can help authors improve grammar, find references, and add new content for scientific papers aiming to be published. Other formats like grants, projects, etc., can benefit from the Magic Mode, where you add a PDF and the AI will offer suggestions on the paper’s writing style and content.
We are working on adding functionalities that can offer templates for journals and connect directly to journals.
+ How does the technology help with standard literature review?
Literature review can benefit from reference finding capabilities inside wisio.app’s platform.
+ What are the costs, availability of using wisio.app?
wisio.app has a free tier, and a Premium tier ($16/month or $109/year). Monthly, we run a one-week student campaign where anyone with a valid university email receives 50% off on the yearly subscription. Also, a code SAGE50 gives 50% off for the yearly subscription.
+ How can AI support academic integrity and arguments for its adoption?
All content produced by wisio.app’s AI is original and free from plagiarism. Most journals nowadays ask for a declaration of AI use, so, it can be cited that wisio.app was used. Other than that, no need to further cite our platform.
+ How can AI be applied in different fields and research support for scientists?
The platform offers a general AI that is trained on scientific articles, aiding in the overall scientific writing process. Based on this, the platform can help authors improve grammar, find references, and add new content for scientific papers aiming to be published. Other formats like grants, projects, etc., can benefit from the Magic Mode, where you add a PDF and the AI will offer suggestions on the paper’s writing style and content.
We are working on adding functionalities that can offer templates for journals and connect directly to journals.
+ How does Wisio compare to other AI assistants, and who owns the generated results?
wisio.app is the only writing assistant focused on scientific writing. All the data generated on the platform belongs to the user. We don’t use or share any info from projects to train any AI model.
+ How can AI be used for generating statistical analysis and simple reports?
We are developing a feature called Statwizard, where all statistical analysis will be conducted by AI, still in development.
Explore How To Webinar Series
Build Multiplayer Experiments with Empirica
How to create multiplayer, interactive, online experiments with Empirica
In this tutorial, we introduce one of the Sage Concept Grant winners, Empirica. Empirica enables you to design and conduct synchronous experiments with groups of human participants in a virtual lab setting. The software is open-source and extensible.
Watching the recording below, you will learn how to install and run Empirica, modify interfaces, and adapt server-side logic. You will also learn about Empirica’s purpose and structure, its capabilities through a real-world case study, and some tips on building your first multi-player experiment.
How to create multiplayer, interactive, online experiments with Empirica.
About Empirica
Empirica is a free, open-source, virtual lab platform for developing and conducting synchronous and interactive human-participant experiments.
About the Speakers
Mohammed Alsobay is a PhD candidate in the Information Technology group at MIT Sloan. His research focuses on the design and analysis of systems in which humans and algorithmic agents interact, with the goal of achieving collective outcomes that exceed what is achievable by either type of agent on its own..
James Houghton is a postdoctoral researcher with the University of Pennsylvania’s Computational Social Science Lab, where he uses high-throughput online experiments to study small-group deliberation.
Additional Resources
The project demo used during the tutorial.
Install Empirica following the instructions here. Windows users must install the "Windows Subsystem for Linux" (WSL 2) to run the installation script, but MacOS and Linux users can run the script in their terminal directly.
Join over 200 members on the Empirica Slack channel using this invitation! You can use the #sage-webinar-2023 channel to discuss the webinar, and we'll aim to provide support for those running through the tutorial at their own pace afterward.
Explore How To Webinar Series
Map Causality in Qualitative Data with CausalMap
How to code, analyse and visualise qualitative information about causal connections
In this webinar, the Causal Map team will introduce you to causal mapping, a way to find out about people’s mental models of the world: what they think causes what, based on interview transcripts or other documents. We will show you that Causal Map does what no other software can: it enables you to directly code, organise, consolidate and understand the causal claims contained within narrative information and present the results as a variety of compelling graphics which we call “maps”.
How to code, analyse and visualise qualitative information about causal connections with Causal Map
Jaimie was carrying out research on social capital in Mali. She wanted to know what kind of contribution informal tea clubs might make to different aspects social capital, and via what pathways or mechanisms. She used Causal Map to code her interview transcripts using causal Qualitative Data Analysis and generate causal maps to visualise the causal pathways. It worked great, and her paper is currently in the “revise and resubmit” stage.
About the Tool
Causal mapping has been used since 1976 in areas from ecology to business management. It is a rigorous, qualitative research approach which helps researchers quickly cut to the chase in answering some of the most important research questions: what causes what, in the minds of key stakeholders? The Causal Map web app (which is free to use for small studies) is a new way to make causal mapping more accessible to researchers.
About Authors
Steve Powell - Co-founder and Director
Steve has led and contributed to research and evaluation projects in many countries around the world over the last 25 years. He has worked on a wide range of topics, from psychosocial programming after the 2004 tsunami and community resilience in East Africa to counting stray dogs in Sarajevo. Steve has expertise in both quantitative and qualitative research and evaluation approaches. He gained his PhD in psychology researching post-traumatic stress after the war in Bosnia-Herzegovina.
This research and evaluation work left Steve longing for a better way to collect and synthesise people’s ideas about ‘what influences what’. This inspired Steve to co-found Causal Map Ltd.
Fiona Remnant - Co-founder and Director
Fiona is a communications and research professional, with a special interest in the practical application of academic research in the international development sector. She has worked in communications in the private and NGO sector, in both regional and international roles.
Fiona was co-author of the Qualitative Impact Protocol (QuIP) whilst working at the Centre for Development Studies at the University of Bath, and founded Bath Social and Development Research (Bath SDR) Ltd in 2016 to promote more and better use of the QuIP.
Hannah Mishan - Outreach Specialist
Hannah is a project manager at Bath SDR, who works with Steve to improve educational materials and support users of the app. She also works to promote use and understanding of the tool through the creation of online content.
Hannah has previously held communications and outreach roles in the charity sector and brings this experience to her role with Casual Map. They studied International Development at both undergraduate and Masters level with a particular interest in sustainability. Throughout this time she became increasingly interested in the theories and tools surrounding qualitative data analysis.
Other Resources
The causalmap.app website, where you can sign up for updates and use code SAGECM23 to get a 50% discount.
A more in-depth demo of Causal Map
Q&A
+ Is Causal Map free or do you need a subscription?
There are free and paid for versions for the app, you can find more information on the website. You can also use the code SAGECM23 to get 50% off when you sign up.
+ Could you recommend readings/literature for those of us starting out with qualitative causal maps?
You can find a few resources including a zotero library here
+ Are there more details about the study you talk about in the tutorial?
You can find more information about this study here.
+ Have you used Causal Map with social media data?
Yes, it is possible to use Causal Map with social media data, you can simply import it and code it. With Twitter data, consider whether you can realistically identify causal relationships in the text - this will work better if you select your hashtags carefully. We would not recommend coding tens of thousands of tweets with Causal Map as that would be very labour-intensive.
+ How does Causal Map compare with other qualitative data analysis tools like Atlas.ti and NVIVO?
There are a lot of excellent qualitative data analysis tools that can be used for a variety of techniques. Causal Map specifically addresses causal connections and simplifies the job of a researcher to code these. We are not aware of other qualitative data analysis tools that can do this without additional effort. Of course there may be and we might not have come across them.
+ Is it easy to use Causal Map to construct tables for qualitative data compared to manually doing it?
We used Excel to create manual links for a few years - it’s possible, but it’s much harder and slower, and you don’t have the benefit of the algorithms to help with the more sophisticated analysis.
+ Can I import already coded data?
Yes you can. You can upload both the links and the relevant statements (with the highlighted coded claim).
+ Can I use text files?
Yes, you can directly upload text files including docx files for coding.
+ Can I delete the data I uploaded?
You can delete the data or file from your dashboard.
+ What about data privacy?
It is generally recommended that any qualitative data that requires anonymisation is anonymised before analysis, so we would expect anyone using Causal Map to have done that in advance. In terms of storage, all data is encrypted and we use Amazon cloud locations in Europe.
+ Could we use the source count to essentially argue the strength of causation?
There is a place for both qualitative and quantitative analysis when evaluating causal relationships. With Causal Map we are aiming to help researchers understand and describe and even count the causal links that the groups they are studying make. But we would not use source count as a proxy for the strength of the relationship.
+ Are there any papers using Causal Map for their analysis?
There are plenty of research reports which used Causal Map. Academic papers are still in the pipeline, watch this space. Here is a preprint.
Explore How To Webinar Series
Manage your References with SciWheel
Manage your references with Sciwheel, from discovery to collaboration and writing your research.
In this tutorial we introduce Sciwheel, the award-winning reference management system. Sciwheel provides customizable, intuitive, and accessible features to progress research and discovery. From reference management with smart citation suggestions to writing, annotating, and team collaboration, Sciwheel is where research truly comes together.
In this tutorial, our presenters discuss and demonstrate the main benefits of using the Sciwheel suite of tools to collect, organize, discover, read, share, and cite references for researchers and students.
We covered:
A brief overview of reference management
Saving and annotating references with the browser extension
Organizing and discovering relevant content
Reading, discussing and sharing references
Citing references and annotations with the citation tools for Microsoft Word and Google Docs
About the Speakers
João Peres - Head of Product
João is the Head of Product for Sciwheel and has been involved in its development since the beginning, always with the researcher's needs at its core. Having done a PhD, postdoctoral research in developmental (neuro)biology and a stint in graphic design, he is now focusing on bringing people together to build great products at Technology from Sage.
Chris Smith - Product Specialist
Chris is the resident product specialist for Sciwheel, where he's worked since 2016 providing support to individual users, librarians and others via the help chat, hosting webinars, creating videos, and various other miscellaneous duties. Since moving from Australia in 2015, where he was primarily employed as a care worker for people with Autism Spectrum Disorder, and casually as a musician as well as doing audio production, he's worked in the UK recording and mixing audio for documentaries and websites prior to his appointment at Sciwheel.
Additional Resources
Q&A
+ Is Sciwheel a free software?
There is a freemium model. You can start using the tool for free and then to add more projects, suggestions, and other features; you will need a premium subscription.
+ Is there a student discount?
Currently we're unable to provide individual paid subscriptions while we review our subscription model. We will provide more information in early 2023.
+ How compatible is Sciwheel with other tools, like Zotero, Mendeley, EndNote and DevonThink? Can I import/ export data from one to other?
Yes, you can export your references to a file and then import it on another reference manager. You can also easily import all your references to Sciwheel from Zotero and Mendeley. Demonstrated HERE. However, the Sciwheel browser extension is really powerful and easy to use, so that is a key advantage over other reference managers.
+ Is Sciwheel housed on my computer (Mac/ PC) or shared across (mobile) devices? Or is it on a website, that I will not actually own?
Sciwheel is web-based software and exists entirely in the cloud, so there's nothing installed on your computer. You can make local backups of reference metadata and PDF files if needed.
+ Does Sciwheel allow for different reference styles?
Our style library has over 7000 styles covering all major journals, publishers, and standard styles for various purposes. Custom styles can be created for premium subscribers on request.
+ Is Sciwheel firewall friendly? Regrettably my institution often blocks add-ins
You should contact your institution’s IT department. Sciwheel is happy to engage with them to answer any security question that they might have. The reason we are moving to the word add-in is to support word online and word for iPad, which we couldn’t do with the plugin.
+ Can you group/ categorize your articles?
Yes, with projects, subprojects, and tags.
+ Does the algorithm consider the reputation of journals when it suggests readings?
No. We believe that users can judge the quality of the articles independently where they are published.
Answered live HERE.
+ How do I organise the citations without losing track of each one of them ?
Inside of your library, using projects, subprojects and tags is a good way to keep them organised. Plus, Sciwheel has a very good search capability.
+ To see if a have read an article in the past should I use a tag (e.g., “read” tag)? If I have found an article relevant?
Yes. Articles to read can easily be added to the reading list (book icon). Then using tags is an easy and handy way to “classify” the article. for example, “Very Important”, “Method” or “not important”.
+ Does Sciwheel provide any bibliometric analysis?
In the reference page we show how many articles cited it with a link to Europe PMC (life science only) to see them. We are looking at other collaborations to increase our coverage.
+ Can we use specific journal or conference template in manuscript section?
There are no word templates at the moment. Our current Manuscript tab only shows Google Docs documents linked to the project.
+ Does this not lead to plagiarism?
No. The notes and highlighted text when inserted in a document are clearly lable in italic and with quotation marks. Plus the source is cited automatically.
Answered live HERE.
+ Is the browser extension only available on Google Chrome?
It is available for Chrome, Firefox, Safari and Edge.
+ How well does Sciwheel locate open access versions of journal articles. For journals like PLoS, this is straightforward, but how about such as on institutional repositories as flagged by Open Access Button?
Currently we do not cover institutional repositories. However, we are looking in integrating with Unpaywall to bring more open access content.
Explore How To Webinar Series
Take Better Research Notes with Scrintal
Scrintal is a web app that combines mind mapping with the power of networked note-taking. So you get crystal clear in your thinking, write creatively and share your findings easily. Scrintal works best when you use the zettelkasten method developed by Niklas Luhmann.
In this tutorial we introduce Scrintal as a new tool to organize literature reviews. Scrintal is a web-based application which allows for taking extensive literature notes. Using the tool, researchers can organize, and visually connect the literature notes with bidirectional links. Scrintal is built upon the Zettelkasten and Evergreen note-taking techniques which aim at building an interconnected personal knowledge system, rather than a collection of ideas and notes.
This tutorial goes through the process of how to organize the literature and research notes according to the three principles of Zettelkasten: 1) creating atomic notes which include only one main idea, 2) creating hyper-textual notes which are linked to the other notes, 3) creating personal notes which are blends of the literature and personal thoughts.
About the Speakers
Ece Kural is the CEO and founder of Scrintal; she started the company while writing her dissertation and out of frustration with current tools that didn’t save any time nor facilitate the overwhelm of information. Ece earned her PhD in International Relations from Stockholm University last year. You can follow Ece on Twitter.
Additional Resources
Sign up for Scrintal HERE
Q&A
+ How does Scrintal work?
Scrintal is a web app that combines mind mapping with the power of networked note-taking. So you get crystal clear in your thinking, write creatively and share your findings easily. Scrintal works best when you use the zettelkasten method developed by Niklas Luhmann.
+ What are literature notes, fleeting notes and permanent notes?
Literature notes are the notes you take while you read the literature. Fleeting notes are essentially anything that pops to your mind, and where you should not worry about format, just note it down before you forget. You don’t have to keep all fleeting notes. Once you have your literature and fleeting notes, you can draft the more robust notes, so the permanent notes that you will use in your paper. Main take-away is that every note you create should contain a single idea.
+ How does Scrintal help me deal with an overwhelming amount of notes?
It uses the basic pillar of Zettlekasten - notes should be retrievable. Scrintal helps you both visualise your notes on the boards, organically organize and connect them by linking the notes with each other. Using Scrintal and Zettlekasten will help you build up your knowledge with each note, you will be placing the note on your board not based on where an idea came from, but rather where it is fitting within all the other notes you’ve already taken. You can also use colors and hashtags to organize the notes further into clusters, boards and metaboards.
+ What happens when you have notes that might be relevant to different projects?
By keeping one idea per card you can then tag that card with different topic names, and link or connect the card with different cards or ideas that belong to different projects.
+ Can you include links on the Scrintal cards?
Yes!
+ Is there a way to add hand written notes either directly via an iPad app with the pencil or import from e.g. GoodNotes?
We don’t have a native tablet app, but if you open Scrintal in the browser on your iPad, you can use your pencil.
+ How is Scrintal different from Miro?
In Scrintal, the notes or cards are not just documents, they can turn into full documents, you can also embed anything into each of these cards. A key difference between the two applications is that in Scrintal, you can easily search through all your notes.
+ How is Scrintal different from qualitative analysis tools like Atlas.ti and Nvivo?
You should not use Scrintal for qualitative analysis, where you need to code reports or interviews line by line. Scrintal is more flexible when creating notes for your literature review. With qualitative analysis tools, the coding is quite specific as you tag line by line, often hirerarchically, with a schema that you might have to set up before reading the literature. As a result it can be difficult to synthesise and move these around as you shelf your ideas as they fit into a structure you are trying to grow organically.
+ Can you export the notes to Microsoft Word?
Not right now. But you can already export to markdown and PDF.
+ Can you import notes from Endnote for example into Scrintal?
Yes.
+ How can you keep track of references in the notes? Does Scrintal integrate with Zotero for example?
You can add your reference to every note you create on Scrintal. The integration with Zotero and other reference management tools is on the roadmap and will be prioritized based on demand.
+ Is Scrintal a web-based or desktop-based system? Is there a Linux version?
Scrintal is a web app. Scrintal works on every browser, but is best on Chrome and Safari. There is a non-native Mac app. All the data is saved on AWS
+ Is there an official Scrintal roadmap?
You can find out all about what we are working on and connect with like minded individuals that use Scrintal by joining our community on slack
+ If your computer gets damaged, are you still able to retrieve all your notes?
Yes. All your notes are saved to the cloud.
+ How can I sign up to use Scrintal and how much does it cost?
Sign up here for the $5/month access. We expect to launch the free version in Q2 next year. If you are working in a team and would like to help us test and improve the collaborative features, you can apply to become a beta tester here. If you want to find out more, check this board.
Explore How To Webinar Series
Gamifying Cognitive Tasks with Gorilla
How to gamify cognitive tasks to increase participant engagement - A Gorilla Masterclass
How to gamify cognitive tasks to increase participant engagement is an online tutorial that we held in May 2022. Here we share the webinar video with you. We have also added the questions that were asked during the live session and their responses. If you have a question, please send it through using the form below, and we will follow up with a response and any other resources.
More engaging experiments means better quality data - participants are more motivated and attentive and are less likely to underperform due to boredom or fatigue. Moreover, visually rich experiences help you verify your findings persist in richer (i.e. more ecologically valid) settings. With Gorilla's Game Builder, rich games and game-like experiences are accessible to research scientists without touching a line of code.
About the Speakers
Jo Evershed - Founder CEO
Jo is the Founder CEO of Gorilla Experiment Builder, a powerful, flexible and intuitive platform for running behavioural research online. An Innovate UK Women in Innovation Award Winner, Jo is on a mission to provide behavioural scientists with tools to liberate their work from the lab and accelerate research initiatives that can be tested rigorously at scale. Jo leads a multidisciplinary team of software engineers and psychologists focused on creating powerful and accessible experimental research infrastructure.
Nick Hodges - Founder CTO
Nick is the Founder CTO of Gorilla and has been building platforms for online research for nearly ten years. Before that he worked in the videogame industry on titles such as Call of Duty, Resident Evil and Lara Croft, and has always been passionate about enabling people to build rich and compelling interactive experiences.
Additional Resources
Participant Engagement Webinar
This is Sarah Jayne Blakemore's Director Game for studying ToM in Adolescence
For 20% discount on a 1 year subscription, use the code: GORILLASAGE2022
Q&A
+ How is analysis done with games in research?
You can create whatever game you need for your research questions. Analysis is the same as for non-game tasks. You'll get the data file of all the participant responses and then you can analyse them to look at accuracy and reaction time.
+ Does your platform handle audio files?
Yes - you can have audio playback, and (soon) also audio recording.
+ What kind of research questions you can answer with this game?
This is a classic go-no-go task - so often used to look at inhibitory control. The important thing is not the mechanics of this game, but how you can add images, control stimuli, add animations.
+ Can you apply gaming techniques to online survey completion?
Potentially yes - the example with the crime scene could work (e.g. even N questions they get another clue).
+ Can we gamify physical tasks? Are such gamification complicated (e.g., requiring IoT, etc)?
It would probably require your participants to have whatever hardware you require, which would likely be prohibitively complicated/expensive.
+ Is gorilla providing different gamified experience for every experiment/task?
Gorilla includes tools that allow you to build your own games - I'll be demoing them in the second half of this session. Stay tuned!
+ I would be interesting in exploring whether this could be applied to more complex thought processes. Can you run sequential or choice dependant programmes - leading to different outcomes?
Yes - you can create games that have more complex sequences of screens depending on the choices made. Reach out and we can chat further. Feel free to send a message to @EvershedJo on Twitter.
+ Will some of those games you’ve shown be available for replication?
The games I've shown belong to researchers, so I can't share them. Some have shared them to Gorilla Open Materials to make it easy for other researchers to use. This is Sarah Jayne Blakemore's Director Game for studying ToM in Adolescence
+ Does Gorilla provide templates for various cognitive tasks, or do we have to "hard-code" the tasks into the gamification?
There are lots of samples here.
And many researchers publish their tasks to Gorilla Open Materials
So yes - often you can find the cognitive task you need, and just change the spreadsheet.
+ Can you possibly do a demo for the multiplayer? Or a “step by step” tutorial?
Here's a video of an ultimatum game: https://www.loom.com/share/b9353962a2c04874b5b2cf0739789b56.
+ Is multi-character input (e.g. words, phrases) an option for responses you can collect?
Yes - you can collect a wide range of responses including words!
+ Please explain how you’d collect data/ demographics from the game.
When you collect online data you usually have anonymous or pseudonymous IDs for participants to alleviate data security concerns. So, you’d need to add a questionnaire before the game. Essentially each response is captured - so in the treasure game with the dragon, we log whenever they press the space bar (together with whether that was the correct response or not). If they don't respond, we log a 'no-go' response at the end of the trial. The researchers can then download a CSV file where each row is one response, and also contains timing information, the participant ID, etc.
+ How to get permission from the users to gather all that data, is it okay to use a disclaimer page so users click accept on them?
You normally put a consent form at the start of your study. You’d put that in first, before they start the game. You can wire this up easily in Gorilla.
Explore How To Webinar Series
Assimilate Literature with Scholarcy
This session showed how you can extract and assimilate key information in a more systematic way and critically analyse the text by easily identifying:
how the author positions their work in relation to previous studies;
what the key findings of any cited studies are;
other indicators of the quality of the research.
How to screen and assimilate scholarly literature in a more systematic way is an online tutorial that we held in April 2022. Here we share the webinar video with you. We have also added the questions that were asked during the live session and their responses. If you have a question, please send it through using the form below, and we will follow up with a response and any other resources.
This session showed how you can extract and assimilate key information in a more systematic way and critically analyse the text by easily identifying:
how the author positions their work in relation to previous studies;
what the key findings of any cited studies are;
other indicators of the quality of the research.
About the Speakers
Emma Warren-Jones - Co-Founder
Emma has 20 years’ experience in the EdTech, academic publishing, and information industries, launching content & discovery platforms and analytics tools to the global research community.
Scholarcy
Scholarcy is a service that uses machine learning and artificial intelligence (AI) to read and condense academic papers into referenced summaries. Either upload your PDFs to Scholarcy, or give Scholarcy the public URL of a PDF and it will read the paper, break it down into readable chunks, and summarise it automatically. It will also read the bibliography and generate a link for each reference so that you can download cited papers.
Additional Resources
Individual and Institutional Subscriptions
For 30% off an individual subscription, please click here
If you would like to enquire about institutional subscription, please email: info@scholarcy.com, or complete the contact form here
Q&A
+ Can institutions buy licences, or is it just individuals?
Scholarcy offers both institutional and individual licenses. For institutional licences, please email info@scholarcy.com and for individual purchase, please click here for 30% off.
+ How much will an individual Scholarcy subscription cost after the first year?
£72
+ Are there videos aimed at institutions considering buying a licence?
Not as such, but we can set-up demos for interested institutions as well as one-month trials. We also have promotional documentation and flyers that we’d be happy to share with interested stakeholders. Feel free to get in touch at info@scholarcy.com if you’d like more information on this.
+ Is the Scholarcy app available in Apple app?
Scholarcy is a web application and so is compatible with any web browser, including the web browser on your iPhone or iPad, but there is no specific app as such.
+ If scholars use Scholarcy to produce the literature review for their PhD thesis, do they need to declare this?
No. Scholarcy does not produce the literature review for you, it helps you to organise and analyse the sources. It is a productivity tool. It is still up to you to write the literature review itself.
+ Are there other tools beyond Scholarcy and reference managers to assist in the analysis of literature?
There are other tools available, depending on the type of literature review, such as Rayyan and Pico portal.
+ Is this system compatible with Mendeley reference manager?
Yes, you can export to BibTeX or RIS file format and those file formats will go straight into Mendeley. The reverse is also possible – so you can export from Mendeley in RIS or BibTeX file format and import that into Scholarcy. If you’d like to import the PDFs that are currently in your Mendeley library, you’d have to locate those on your computer and drag them into Scholarcy or import them via Google Drive. We are looking at a direct integration with Mendeley in the future so that this could be a bit more automated.
+ Does the application import literature from the web?
Scholarcy can import open-access literature from the web, via RSS feeds, search engine exports, or directly via URL, as we demonstrated.
+ Is the indexing multi-dimensional? Could I classify a paper as being about a person, a topic, a method, a country, a language, an institution, etc.? Does it have Tags like Mendeley, for example?
Scholarcy primarily focuses on content extraction to identify the key information from each study. You can organise papers into folders and libraries. You can add key terms to each study, but we don’t currently have a tagging system to organise papers by tags, but it is something we could add in future.
+ Is Scholarcy compatible with Endnote?
Scholarcy will work with any reference manager that can import RIS or BibTeX files. This includes Endnote, Mendeley, Zotero, and many others.
+ Can we import existing collections of articles from Endnote to Scholarcy?
Yes. You would need to export as a RIS file and then import that RIS file directly into Scholarcy.
+ Will these RIS documents generated contain my previous notes added (when convert from Endnote to Scholarcy)?
RIS does have a field for storing notes, so if Endnote stores them, then yes they will be carried over.
+ Can I import an Endnote Library and its files?
You’d need to export them as a RIS file and then import them into Scholarcy.
+ What is the relationship between Endnote and Scholarcy? Do you think that Scholarcy can kind of replace the citation organiser like Endnote? Or need to work together?
Endnote is a reference manager, it’s good at creating bibliographies. They work together, complementing each other. Scholarcy is for extracting the key facts and findings, and then to create the bibliography, you’d export to Endnote or another reference manager.
+ Pdf documents will need to be added manually when importing from Endnote. Does this mean you need to attach them one-by-one?
No, you can drag and drop a whole folder of pdfs. You can also upload multiple pdfs at one time. It can upload up to 128 at once, or you can import them from your Google Drive or DropBox.
+ How can one be sure that you have the most up to date research at your fingertips?
Subscribing to publisher RSS feeds can help.
+ Please explain: a) Optimizing scholarly journal searches when searching within a set basket of journals, b) Doing ""forward citation"" searches and downloading the results (to CSV formatted file?) to track reading progress in a spreadsheet (unless there is a better way!) and c) Setting up alerts for articles to be notified when they are cited?
We cover the reading and analysis part in this webinar, so post-search and post-discovery. However, there are many resources available online on literature searching and discovery. For b) you could use ConnectedPapers, ResearchRabbit. For c) you could use Scite.ai, Dimensions, Web of Science.
+ What is best practice for paraphrasing ideas?
By reading many articles you can get an idea of how other authors paraphrase when citing. There are also tools from Writefull that can help with writing academic English.
+ Can we place the highlights ourselves, in case we find additional points that we find useful?
Yes, we integrate with Hypothes.is so you can add your own highlights and annotations and store these.
+ What if the article is not open access, or we do not have a subscription?
If the article is not open access or you do not have a subscription then the link will direct to the publisher’s page for that article (usually the abstract).
+ How do I link this to my commercial databases? Do I need to retrieve articles first?
You’d need to export the data into a file format compatible with Scholarcy and then import it to Scholarcy. For Scholarcy to locate them, they’d need to be Open Access. If they are not, you’d need to upload the pdfs to Scholarcy manually.
+ Can you integrate pdfs of book chapters to generate the flash cards?
Yes, you can import book chapters. There’s a different summarisation engine that you can switch on for book chapters.
+ How does one create the filtering tables in the exported Excel spreadsheet, as shown?
Creating Excel slicers is covered in these videos: https://www.youtube.com/watch?v=Y40Wy1guAiQ and https://www.youtube.com/watch?v=T_bHx5hGMq8
+ Would Excel work with the manual highlights and annotations as well?
You can add manual highlights and annotations. We use a system called Hypothes.is, which is the industry standard annotation system for the web. The manual annotations are stored outside of Scholarcy, so they don’t form part of the Excel export. However, the top 5 highlights and the summary flashcards can be edited, so you can add your own notes in there, which will get exported to Excel. Here’s a link to a video that shows how.
+ What does an export to Word look like?
You can export (multiple) flashcards to Word. It’ll export as a structured summary of the paper, with its highlights, with references at the end. You can also tailor the settings before you export to Word, so you can customise what you export. So, you could have just the highlights, and not the full structured summary, for example. Unfortunately, Word won’t give you the same side-by-side view that Excel can.
+ Are there more tutorials available on how to use Scholarcy?
Yes, please click through to our YouTube channel for more.
Explore How To Webinar Series
Analyse Online Networks with VOSON Lab Tools
This session provided an overview of methods and research used to study online networks of political discussion on social media (Twitter, Hyperlinks, Reddit), using data collected with the VOSON Lab suite of open-source R tools
Analysing online networks with VOSON Lab tools is an online tutorial that we held in March 2022. Here we share the webinar video with you. We have also added the questions that were asked during the live session and their responses. If you have a question, please send it through using the form below, and we will follow up with a response and any other resources.
This session provided an overview of methods and research used to study online networks of political discussion on social media (Twitter, Hyperlinks, Reddit), using data collected with the VOSON Lab suite of open-source R tools: vosonSML, VOSON Dashboard and voson.tcn. A live demo of VOSONDash, an interactive R Shiny web application for the collection (via vosonSML), visualisation and analysis of social media network data, was also presented.
About the Tool
The VOSON Lab Virtual Observatory for the Study of Online Networks is located in the Research School of Social Sciences at the Australian National University. We are advancing the Social Science of the Internet through an innovative programme of research, research tool development, and teaching and training. The VOSON tools have been publicly available since 2006. The current R tools are available on CRAN and GitHub, with over 61K downloads to date, and are downloaded over 1K times per month.
About the Speakers
Prof Robert Ackland - VOSON Lab School of Sociology and ANU Centre for Social Research and Methods, Australian National University.
Robert works at the intersection of empirical social science and computer science, developing new approaches (involving information retrieval, data visualisation and social network analysis) for studying networks on the World Wide Web. He has been a chief investigator on five Australian Research Council grants and under a 2005 ARC Special Research Initiative (e-Research Support) grant, he established the Virtual Observatory for the Study of Online Networks project. Robert has co-organised symposia focusing on e-Social Science (2004) and the social impact of nanotechnology (2006) and in 2007, he spent six months at the Oxford Internet Institute under a UK National Centre for e-Social Science Visiting Fellowship and a University of Oxford James Martin Visiting Fellowship.
Robert has degrees in economics from the University of Melbourne, Yale University (where he was a Fulbright Scholar) and the ANU, where he completed his PhD in economics (on index number theory and international comparisons of income) in 2001. Prior to commencing his PhD, Robert gained extensive experience in applied economic and statistical analysis in the government and non-government sectors. From 1991-1993, he worked as a senior researcher in the Bureau of Immigration Research (Commonwealth Department of Immigration). He worked as a World Bank consultant (based in Washington DC, 1995-1997) in the area of poverty analysis and has also consulted on AusAID and Asian Development Bank projects in this area. Robert teaches courses on the social science of the Internet and online research method in the Master of Social Research and his book Web Social Science: Concepts, Data and Tools for Social Scientists in the Digital Age was published by SAGE in 2013.
Francisca Borquez - Research Assistant VOSON Lab
Francisca graduated from ANU Master of Social Research in 2010. Francisca has been involved in various research related work for the last 11 years in both academia and industry, with a focus on Social Network Analysis (SNA), computational social science as well as quantitative and qualitative methods. As part of the VOSON Lab, she has assisted in diverse research projects and has collaborated with open-source software developed at the lab. Her research interests are online social and organisational networks, online behaviour, computational methods and experimental social research.
Bryan Gertzel - Research Programmer VOSON Lab
Bryan is an Information technologist with interests in the Internet, cyber security and online social networks. He graduated from the ANU Master of Social Research in 2012 and is a Research Programmer for the VOSON Lab. Bryan is the main developer and maintainer of the VOSON Lab suite of tools. He has collaborated in large-scale data collection projects and is involved in the research project Unbiased Bots That Build Bridges (U3B): Technical Systems that Support Deliberation and Diversity as a Chance for Political Discourse, led by the University of Bielefeld, Germany.
Additional Resources
vosonSML: Documentation - GitHub page and vosonSML vignette
VOSONDash: Documentation - GitHub page and VOSONDash Userguide
voson.tcn: Guide for Collecting and Constructing Twitter Conversation Networks
Teaching and Training
•ANU undergraduate and masters courses in Online Research Methods, Social Science of the Internet, Economic Analysis of the Digital Economy
•PhD studies
•Online short courses and masterclasses via ACSPRI
References
Analyzing Social Networks Using R, by Stephen P. Borgatti, et al. here.
Web Social Science, by Robert Ackland, here.
Q&A
+ Are data able to be archived and reused? Or is the data collected dynamically as part of the analysis?
Collected data can be downloaded as an R data frame to RDS file or exported from the data tables, for example as CSV. Networks can be downloaded as data frames of nodes and edges, or in GraphML format (that can be imported later). The current state of a network graph from the analysis section can be downloaded as GraphML and then imported later via the open GraphML control. Data collection is performed prior to analysis, however the VOSONDash interface makes it easy to iteratively collect and examine data as part of exploratory analysis.
+ Best practice in using Reddit datasets, especially API data and Pushshift.io data
vosonSML retrieves JSON data from subreddit threads using unauthenticated requests. The data retrieved is public, however this method is very limited and may be removed by Reddit at any time. Best practice would be to use another library that supports authenticated access to the API until we support this in vosonSML. We hope to support an authenticated API approach with a more comprehensive, standardised data collection and network creation in the future. It is possible to generate networks from JSON data retrieved from Pushshift.io: this can be done using igraph in R and we are planning a blogpost on this topic.
+ In what ways is VOSON helpful in literary studies and research?
VOSON is designed to enable research into online networks. The main reason for using VOSON is if you are interested in understanding how actors are interacting with one another via e.g., replies or retweets on Twitter, or comments on Reddit, and where it is useful from a research perspective to use network analysis to study this behaviour. Another reason for using VOSON is if you are interested in collecting and analysing text data from social media, and where you would like to know the actors who are authoring the text, and how these actors connect with one another. If you are not interested in networks, then here are other complementary open-source tools available for text analysis within the R environment, such as Quanteda and tidytext, which are used for the quantitative analysis of text data.
+ Cost of VOSON? Is there a free option?
VOSON R tools are Free and Open-Source Software (FOSS) released under the GPL-3 licence. They are publicly available via The Comprehensive R Archive Network (CRAN) and from VOSON Lab GitHub repositories.
+ What's the difference between the VOSON tools and other social network analysis tools (e.g., NodeXL)?
The VOSON tools are released as open-source R packages and hence they make use of, and can be used in addition to, other packages within the R environment such as rtweet, igraph, statnet, visnetwork, quanteda, tidytext etc. VOSON is complementary to the R packages igraph and statnet and indeed, VOSON makes extensive use of igraph for network analysis functionality. But igraph and statnet do not enable data collection from social media or the web (that is VOSON’s speciality). While we think NodeXL is great software (and indeed there used to be a VOSON plugin to NodeXL, for hyperlink network collection), some users may find it a limitation that NodeXL only runs in Windows. R works on the major operating systems: Windows, MacOS, Linux. We would like to draw your attention to Gephi for large-scale network visualisation. It is possible to create a network in VOSONDash and then export it to graphml and import it into Gephi for visualisation. Why would you do this, and not simply make use of the visualisation capabilities of VOSONDash? Well, while we are very proud of network visualisation in VOSONDash (and we build on igraph and visnetwork for this), the fact is that since VOSONDash is a web application, it is not capable of visualising very large networks. Gephi is the specialist tool for network visualisation, and we use it extensively in the VOSON Lab. Finally, we’d like to mention two other software tools that are very prominent in social network analysis (SNA): UCINET and Pajek. Again, these tools do not provide functionality for collecting network and text data from social media, but it is possible to use UCINET and Pajek to analyse networks created in VOSON.
+ Is there a possibility to filter the data related to a profile of users, for example concerning their age or location?
The fields available for analysis are those provided by the APIs, and they differ for each data source. After you have collected your data, you will be able to see what fields are available in the data table. For Twitter, for example, there are around 80 fields that are available including profile information (such as location, if the user provided it). Note that by default, only a subset of available fields is included in the network as node or edge attributes. It is possible to include additional fields as node/edge attributes, but that will require some simple R/igraph coding.
+ Can this software be used for analysing social media contents other than political discussion?
Yes. In the VOSON Lab we tend to focus on analysis of political discussion, but it is possible to use VOSON tools to study any public social media activity. By “public” we mean that users have not changed their privacy settings such that their behaviour is hidden (such private activity is not available for collection via APIs). So VOSON can be used studying activity on social media related to any topic that you are interested in, as long as there are people on social media who are talking about it.
+ How reliable is data collection online?
VOSON will collect whatever the APIs allow it to collect. APIs have well-known restrictions or limitations. For example, with the Twitter API there might be limitations associated with sampling of data (when you are collecting on a hashtag that has high volume), and collection of historical Twitter data is only available if you have Academic Track access to the API. If a user deletes their tweets or Twitter suspends a user account, then the data will no longer be available via the API. Another issue of “reliability” of social media data is: how representative is the data of the population you are interested in studying? Social media data are typically not representative of the general public. Appropriate research design can help address such restrictions and limitations. For example, in our analysis of the 2020 US presidential debate Twitter data we do not make claims about what the US voting public thought about the candidates, rather our population of interest is people who were on Twitter talking about the debates.
+ Does text analysis (sentiment analysis) in Dash work in different languages or just in English?
We have spent a lot of time to ensure that the VOSON tools collect and store text data in an appropriate manner. So, for example, if the Twitter data you collect contains non-ASCII characters (e.g., Chinese language) then the text data will be stored correctly for further analysis. However, the VOSONDash text analysis tools (frequency analysis, word clouds, sentiment analysis) will possibly not handle the text correctly. With regard to the frequency analysis and word clouds, the approach we use relies on using spaces for tokenisation of words, and that is not appropriate for all languages. The sentiment analysis in VOSONDash is using an English lexicon. So, our recommendation is that if you are wanting to conduct text analysis for a language other than English, then you are probably best using VOSON just for the data collection and network construction. Then you can export your data (including networks, if useful to you) and analyse your text using R packages that are designed for handling the language you are working with. That is the beauty of working in the R environment: there is almost certainly going to be an R package to help you.
+ In which format can you download the data?
The raw data (what VOSON collects from the APIs) can be downloaded as a data frame (rds format for storing R objects). Network data can also be downloaded as a data frame, csv or Excel format. Network graphs can be downloaded as GraphML files.
+ Is there a way to override the API restrictions via brute force scraping?
If you are wanting to scrape a social media platform, then VOSON is not the tool for you. VOSON allows you to collect via APIs (for Twitter, Reddit, and YouTube) and hence whatever the API will allow you to collect, then you can collect it via VOSON. By supporting collection via APIs (rather than web scraping) we contend that VOSON is a tool for ethical research into online behaviour. However, there will still be ethical considerations with what you do with the data collected using VOSON, and this will be something that the human research ethics committee at your university will have something to say about. Finally, while we do provide a web crawler within vosonSML (WWW hyperlink networks are one of the data sources that you can collect on), it is designed to crawl websites, not social media platforms, and further: it obeys the robots.txt protocol, so it only collects the data that webmasters are making visible to crawlers.
+ Could you please share some sample research articles, which employed the VOSON app?
There are some research examples in the slides we are providing as part of this webinar. Also, please see the VOSON Lab website for nearly twenty years of research using and producing the VOSON tools. If you want to find research by other people where VOSON tools are used, sometimes authors forget to cite us, but google searching for VOSON can turn up some papers.
+ Can you do search query without the hashtag? Can we search for certain words in the tweet, for example?
Yes, it is possible to search a tweet for any term (a word, a hashtag) or a combination of terms (including boolean searches). Twitter allows for sophisticated search queries (see standard search operators) https://developer.twitter.com/en/docs/tweets/rules-and-filtering/overview/standard-operators and so whatever the API allows, you can do this in VOSON. Additionally, the collection may be filtered by, for example, type of Twitter activity (e.g., to include retweets only), number of collected tweets, or language of tweet. See our vignette for more information: https://cran.r-project.org/web/packages/vosonSML/vignettes/Intro-to-vosonSML.html
+ What is the maximum number of tweets you can collect in a network?
It depends on the Twitter API rate limits. With the standard v1.1 Twitter API, there is a limitation of 18,000 collected in a 15-minute period. If your collection is going to exceed this rate limit, it is possible to set VOSON so that the collection will pause or sleep if the limit is reached, and then it will automatically start up again. The VOSON Lab conducted large-scale Twitter collections (over 1 million tweets collected) during the debates of the 2020 U.S. presidential election. For details, see this blog post: https://vosonlab.github.io/posts/2021-06-03-us-presidential-debates-2020-twitter-collection/
+ Can you look at changes over time? Is it possible to build an author network scraping data in reddit based on date? For example, from day x to day y?
For three of the data sources (Twitter, Reddit, and YouTube) there is timestamp information indicating when a tweet was authored, or when a comment on Reddit or YouTube was written. VOSON includes the timestamp data in the network as a node or edge attribute. Hence, it is possible to conduct dynamic network analysis. Also, it is quite common to undertake Twitter collections over a period of time e.g., collecting on a particular hashtag every week. This leads to a series of dynamic networks which can be analysed separately or merged into a single large dynamic network. We are currently exploring ways to integrate dynamic network analysis and visualisation into VOSONDash, but the data for dynamic network analysis are being collected and are available. It is currently not possible to use VOSON to collect comments that were authored during a particular time period. What you would need to do is collect the entire thread (or post) and then you can later filter out comments based on date of creation (this would require that you download the data and work with it directly in R).
+ What types of training/ workshops do you offer? For researchers and educators?
- The VOSON Lab contributes to undergraduate and master’s courses at the Australian National University in the following areas: Online Research Methods, Social Science of the Internet, Economic Analysis of the Digital Economy.
- We encourage applications from suitably qualified students to undertake PhD studies in the School of Sociology, where the VOSON Lab is located.
- We run online short courses and masterclasses via the Australian Consortium for Social and Political Research Inc. (ACSPRI).
+ Currently VOSON searches Twitter, YouTube, and Reddit. Is one able to search Facebook, Instagram, LinkedIn, and Tik Tok? Might this be possible in the future?
If a social media platform affords networked behaviour (e.g., conversations, commenting, liking of posts, sharing of posts) and has a publicly available API, then the VOSON Lab might be interested and available to extend the VOSON tools to collect the data. In the past VOSON was able to collect data from both Facebook and Instagram, but the changes to the API that Facebook enacted after the Cambridge Analytica data scandal meant that it was no longer possible to collect network data from these platforms. We are always looking to integrate other data sources via their APIs, that can be used for social network analysis, but please remember we are a small team so we might need to seek resourcing for any major software development.
+ Is VOSON tool able to crawl the hyperlink and content of a website/page?
Yes. Hyperlink collection is available via vosonSML. See the following blogpost: https://vosonlab.github.io/posts/2021-03-15-hyperlink-networks-with-vosonsml/
+ Is this only for user networks? Or can I use this for co-hashtag network visualization?
The VOSON software designed for the analysis of networks, and the software currently produces the following networks:
- Reddit: actor network (nodes are Reddit users who have commented, and the author of the post); activity network (nodes are the comments, and the top-level post).
- YouTube: actor network (nodes are users who have commented on a YouTube video, and the channel that uploaded the video); activity network (nodes are the comments, and video).
- Twitter: actor network (nodes are Twitter users who have e.g. authored tweets containing a hashtag or are mentioned/replied to/retweeted in tweets containing a particular hashtag); activity network (nodes are the tweets); two-mode network (two actor types – user and hashtag – and there is an edge from user i to hashtag j if user i authored a tweet containing hashtag j; semantic networks (nodes are entities extracted from the tweet text - words, hashtags and usernames and edges reflect co-occurrence i.e. there is an edge between entities i and j if they both occurred in the same tweet).
- WWW hyperlink: actor network (nodes are website domains e.g., www.anu.edu.au); activity network (nodes are web pages).
More details on the network types can be found in our vignette: https://cran.r-project.org/web/packages/vosonSML/vignettes/Intro-to-vosonSML.html
But remember: if there is a particular network type that you wish to work with, and it is not currently provided by VOSON, then it is always possible to export the VOSON network as graphml and then use igraph in R to construct whatever network type you would like. We do this in the VOSON Lab, and we are planning future blogposts on this topic.
Explore How To Webinar Series
Qualitative Analysis with Quirkos
This tutorial shows how to do Constant Comparative Analysis and Grounded Theory for teams of researchers small and large, and working together in real time with the simple qualitative data analysis software, Quirkos.
Making Qualitative Analysis Constant With Quirkos is an online tutorial that we held in February 2022. Here we share the webinar video with you. We have also added the questions that were asked during the live session and their responses. If you have a question, please send it through using the form below, and we will follow up with a response and any other resources.
This session shows how to do Constant Comparative Analysis and Grounded Theory for teams of researchers small and large, and working together in real time with the simple qualitative data analysis software, Quirkos.
About the Tool
Quirkos is a research software tool helping manage small qualitative text analysis. You can get 25 percent off using the code SAGE25.
About the Speakers
Dr Daniel Turner is the founder and director of Quirkos, which he left academia to start after a decade focusing on qualitative health research. He runs workshops and training on all kinds of qualitative methods, and writes a popular qualitative blog.
Additional Resources
The Coding Manual for Qualitative Researchers, by Johnny Saldana, 2021 edition here.
Q&A
+ What is Quirkos?
Quirkos is a software tool for helping manage small qualitative text analysis. It is designed to be easy to use, visual and affordable.
+ How does it Compare to other software packages (ATLAS, NVIVO and QDA specifically)?
Quirkos does not support multimedia data like these packages can, nor mixed-method or quantitative data. However, it has a more visual interface, is cheaper and has a lower learning curve.
+ Can you create sub-codes in Quirkos?
Yes! Just drag and drop codes onto each other to create sub-categories.
+ Can you keep two separate coding sessions for a project so that you can compute inter-rater reliability?
You can keep separate projects for each coder, and then merge together to see the differences. However, Quirkos doesn’t have built in quantitative metrics for calculating inter-rater reliability, but has exports to SPSS/R where you can run these statistical tests.
+ What about multiple subsets at once?
Yes, you can work on multiple subsets at once in Quirkos: the Query view will let you see work from one (or more) subset at any time.
+ How do you standardize the interview schedule or how can you check the reliability and validity of the interview schedule?
Check your epistemological stance here: structured and standardised interviews are rarely used in this type of qualitative research. Usually unstructured or semi-structured guides are used to cover the main topics but to allow for deviation and discussion to get more detail. It’s more important to train interviewers to not ask leading questions, but be able to formulate new questions on the fly to unexpected answers.
+ Could you explain how to analyze multi qualitative sources like dairy, interview, social media?
All these sources can be brought into software like Quirkos, where you should try and code them with the same code set. You would then use the Query view to see just results from one data type (if necessary).
+ Can Quirkos read languages besides English?
Yes, Quirkos can display any language or character set (including non-Latin like Korean or Chinese).
+ Your top two tips on recruiting and scheduling interviews?
I think sampling and recruitment are two separate and important considerations, this blog post gives more detail My two tips would be to try and recruit in different ways to make sure you get diverse respondents, and continuously check your sample to make sure you are talking to the right people.
+ Can this method be used only when your research questions seek to compare?
No – I think Constant Comparative Method is useful even when comparing within your internal coding and framework process, the aim doesn’t have to be to compare one group with another, or against different research findings.
+ Where are Quirkos data stored?
You can choose the offline storage (where everything is only stored on your computer), or the Cloud subscription, which saves project data on a global network of secure, encrypted servers across the world. We store the data as best we can for local data protection requirements (eg a separate server for Canada, as they do not allow international data transfer).
+ How do you access social media especially with issues of ethical consent and how can you use these without identifying users?
Is social media considered public data? It’s a good question, and one that your IRB might have strong views on. It’s always better to get explicit rather than implied consent, and remove any unnecessary identifying information - not just usernames, but mentioned locations etc.
+ Does Quirkos offer training to institutions or individuals interested in subscribing?
Yes! We provide free training to groups when they register, and also free introductory sessions for groups interested in evaluating or learning Quirkos for the first time. Just contact us to set one up.
+ If using Quirkos to work with a large amount of data (e.g. 500 sources), what will happen?
It will not be a great experience – some searches will be slow, but mostly there isn’t an easy way to browse hundreds of sources, there’s just one long list. Our interface is designed to work mostly with smaller datasets.
+ Is Quirkos accessible to blind users who use screen readers?
We have several visually impaired users, in our experience it depends on the screen reader and hardware they are using, as some work better than others. We are always happy to help get a particular tool working.
+ Does Quirkos work for phenomenological analysis?
Yes, Quirkos supports memos and reflexive writing often used in line-by-line approaches like IPA and in-vivo coding, which can be later added to codes and themes.
+ How might coding themes be co-produced and refined in process?
Good question – one that depends a lot on the analytic design. It’s usually a good idea to co-develop rough guidelines for when codes will be created, how they will be described (with usage guides and definitions). Often some basic codes will be agreed in advance through the use of a codebook, like this
+ Do you have any discounts?
Absolutely. You can sign up for a free trial and get 25 percent off with the code SAGE25.
Explore How To Webinar Series
Introduction to Online Behavioural Research with Gorilla
Watch our webinar, hosted by Dr Catriona Silvey from Gorilla Experiment Builder; Introduction to Online Behavioural Research: Successes, Challenges, and How to Get Started. The webinar covers the benefits of online behavioural research, how to maintain data quality and a practical demo showcase of the Gorilla tool.
Introduction to Online Behavioural Research with Gorilla Experiment Builder is an online tutorial on how to design and run your own experiments. Here we share the webinar video and slides with you. We will add some of the questions that were asked during the live session and their responses. If you have a question, please send it through using the form below, and we will follow up with a response and any other resources.
Learn about the benefits of online behavioural research, how to collect and maintain data quality when designing experiments. Dr Catriona Silvey shares a few fascinating research projects that used Gorilla, and walks through a practical showcase of the tool. This tutorial is perfect for PhD and postdoctoral researchers wanting to take Reaction Time tasks online quickly and easily without learning JavaScript or other programming languages.
About Gorilla
Gorilla Experiment Builder is a cloud-based research platform that allows researchers and students to quickly and easily create and deploy behavioural (reaction-time) experiments online. Get 50 tokens FREE when you buy 50 tokens with the promo code GORILLASAGE2021. Offer ends January 31st 2022. Find out more and follow Gorilla on twitter.
About the Speakers
Dr Catriona Silvey is a former academic researcher with 10 years of experience running linguistics and psychology experiments both in the lab and online. At Gorilla, she focuses on introducing new users to the possibilities of the platform and the benefits of online research.
Q&A
+ How can I start using Gorilla?
You can create an account and build an experiment for free with Gorilla, and use the promo code GORILLASAGE2021 to get 50 free tokens when you buy 50. With Gorilla you only pay for the data collection. See more on our flexible pricing here: https://app.gorilla.sc/pricing
+ Is there a user manual on Gorilla software?
We've got a wide range of support documentation available for users to get started. I'd recommend our get started guide or one of our onboarding webinars as great places to start.
+ Who has access to the actual data that is collected?
At Gorilla we are very aware of the importance of keeping your data secure. The data that's collected is owned by the researcher and we never access it (unless explicitly asked to do so by the researcher e.g. to answer a support question). We have a comprehensive due diligence information about data security here.
+ Is Gorilla GDPR compliant?
Gorilla is fully GDPR compliant, and you can find more about this here.
+ How do you ensure that the participants are really eligible to participate in your study?
Gorilla is a platform for creating your tasks and experiments rather than finding participants. However, we do offer integration with a number of popular recruitment services, such as Prolific, Sona Systems and MTurk. You can find more information about recruitment options here. If you were using Prolific, for example, you can check out this extensive step-by-step guide on how to connect it with Gorilla.
+ Is it possible to include survey questionnaires before the experiment?
Yes! Gorilla includes a Questionnaire Builder tool which allows you to collect a wide variety of survey data from participants. Find out more in our how-to guide.
As well as a survey tool, Gorilla also includes a task builder, code editor, game builder and shop builder. To create an experiments, the individual tools are combined using our experiment tree tool. This allows for an immense range of experimental designs, while making it easy to work with each bit of the experience.
+ Is there a way to run multi-participant experiments (i.e., economic games)?
One of our upcoming features in 2022 is Multiplayer, which will allow multiple participants to interact within the same task; if you are interested in trying it out, contact us on info at gorilla dot sc.
+ Does Gorilla support other languages?
Our documentation and guides for using Gorilla aren't yet available in other languages; however, if you are able to use Gorilla, we have localisation options so you can customize both the instructions and the automatic text within the Task Builder to be in the language your participants will be using. Find out more about localisation in Gorilla here.
We have researchers using Gorilla all over the world including: The Netherlands, Germany, France, Brazil, Israel, Japan, China and UAE. So we're confident that the Gorilla participant experience, can be customised for a wide range of languages and scripts.
+ Does Gorilla support two different tasks on one screen, for instance can a participant see an image and hear an audio file simultaneously?
Yes, you can present an image and an audio file on the same screen as multiple stimuli by adding both an Image Zone and a Web Audio Zone to the same screen in your Task. Find out more about incorporating different Zones in our Task Builder Zones Tooling Reference Guide.
+ Can I design my own tasks if these aren't available?
Yes! The Gorilla task builder has lots of different zones that can be configured in an wide range of different combinations to bring you study design to life. Essentially, we provide the LEGO bricks and you combine them to create what you need.
If we don't have the right 'lego brick', you can you extend the functionality of our tools using using scripting. So you'll always have a way through. An advanced webinar on this topic is coming soon!
You can also program your own tasks in the Code Editor. You can import third-party libraries to build your own task from scratch, or alternatively import tasks built in other frameworks (e.g. jsPsych and PsychoJS). Find our list of example tasks built in the Code Editor here.
Explore How To Webinar Series
Grounded Theory with Delve
Learn how to discover new theories based on the collection and analysis of real world qualitative data. LaiYee Ho, the co-founder of Delve, talks through the process of grounded theory analysis with a practical focus on using the qualitative data analysis tool, Delve.
Introduction to Grounded Theory Data Analysis With Delve is an online tutorial that we held in October 2021. Here we share the webinar video with you. We have also added the questions that were asked during the live session and their responses. If you have a question, please send it through using the form below, and we will follow up with a response and any other resources.
Learn how to discover new theories based on the collection and analysis of real world qualitative data. LaiYee Ho, the co-founder of Delve, talks through the process of grounded theory analysis with a practical focus on using the qualitative data analysis tool, Delve.
About the Tool
Delve is an online software that helps researchers analyze qualitative data. Delve is easy to learn, and great for anybody that is new to qualitative research or teaching a qualitative methodology course. In the lead-up to the webinar, you can start a free trial of Delve and get 50 percent off your first three months.
About the Speakers
LaiYee Ho, a researcher and designer, builds online software for qualitative coding and has published the Practical Guide to Grounded Theory, which is valuable even for those not currently using Delve. You can find it here.
Additional Resources
Constructing Grounded Theory, by Kathy Charmaz, 2014 edition here.
Doing Grounded Theory, by Uwe Flick, 2018 edition here.
Q&A
+ How can I start using Delve?
Go to the Delve website and start a free trial. You can get 50 percent off your first three months using this link.
+ Can Delve be used for other qualitative methods besides grounded theory?
Yes! Delve can be used for a number of qualitative methods such as thematic analysis, narrative analysis, discourse analysis, content analysis and more. Delve is not exclusive to a particular qualitative method, and can support any analysis that involves qualitative coding.
+ What is the difference between Delve and other qualitative coding software, like NVivo?
If you are new to qualitative analysis, Delve is the right choice for you. Delve is simpler to use and easier to learn, so you can focus on analyzing your data rather than learning how to use software. It is also web-based and collaborative, so students are able to share their projects for instructors to review. Delve also includes responsive, human customer support and flexible pricing options. See our comparison chart here.
+ Does Delve work with other languages like Malayalam, Kannada or Tamil?
Delve works with most languages. If you want to double-check that Delve is compatible with the languages you need to use, feel free to reach out to us.
+ How does Delve protect the security and privacy of our data?
Delve follows the best practices in the industry to protect your data and privacy. For details on the services, policies, processes, and procedures we have in place, see our security and privacy page.
+ Can you collaborate with multiple people when using Delve?
Yes! Since Delve is cloud-based, you can have multiple collaborators located in different places all code the same project at the same time using Delve. Any changes your collaborators make will live update in real-time on your computer! This is one of the main benefits to using Delve in your classroom. Students can collaborate and also share their projects with their instructors to review.
+ Does the coding have to be done by paragraphs or is it done line by line?
Both are absolutely acceptable. It's very much what you like as a researcher. Some researchers do every single line, while others just select whatever group of text that they think encompasses an idea. In Delve you can select a paragraph or sentence in one click.
+ Is it possible to create tables on Delve?
Yes. You can use Delve to create tables such as code co-occurrence matrices, as well as other tables that show various dimensions of your data.
+ Does Delve have any special pricing for students, or use in education?
Yes, Delve offers a reduced price for use in education.
+ Do you have any discounts?
Absolutely. You can sign up for a free trial and get 50 percent off your first three months using this link.
Explore How To Webinar Series
Designing Trust with Knowsi
Knowsi is a privacy-first platform for researchers to collect, track and manage participant consent. Consent is vital to performing ethical research, but the operational difficulty of tracking consent and the need to adhere to privacy regulations presents barriers for researchers. In the webinar, Andrew covers top tips and key pitfalls to look out for.
We hosted a tutorial with our SAGE 2020 Concept Grant winner, Andrew Lovett-Barron, on how to design trust relationships with participants in research using Knowsi. Here we share the webinar video with you. We have also added the questions that were asked during the live session and their responses. If you have a question, please send it through using the form below, and we will follow up with a response and any other resources.
Designing a research program is a difficult task. So, it comes as no surprise that often the participant experience gets cut in favor of focusing time and resources on refining your interview guide, recruiting more participants, or wrangling your Research Ethics Board. In this tutorial, Andrew Lovett-Baron proposes a structure for applying the front-stage/back-stage model articulated in Service Design to designing the participant experience.
About the Tool
Knowsi is a portal to manage research participant consent in compliance with data privacy regulations. Enjoy the post and get started with Knowsi today by signing up for a free account. If you want to upgrade to the paid-for version, use the code SAGE to get two months free.
About the Speakers
Andrew Lovett-Barron, a software designer and entrepreneur, launched the consent management platform Knowsi in 2019 and recently launched the open source outfit blog, Stupid Fits. You can find his brilliant newsletter and other projects he is working on here.
Q&A
+ Does Knowsi work with all computer platforms?
Yes. Knowsi is a progressive web application, so it lives in your browser like a normal site. You can also add Knowsi as an “app” on your mobile device by clicking the Add to Homescreen link on IOS or Android.
+ Can you generate consent forms in different languages?
There is a still beta version of form translation in the system right now. When you are editing a form, there is a tab for Translations that allow you to create a duplicate but linked version of that form in a different language. You’re ultimately responsible for providing the proper translations for your documents, but we try to support as much as possible with headers and similar. Currently, this feature is incomplete — but if it’s something that you’d like to use in the near term, please get in touch and we can move it up on the roadmap.
+ What are the key benefits of Knowsi in building a trust relationship in research?
Knowsi provides a consistent platform for participants to manage their consent and relationship with the researcher — respecting their agency in the process and putting actions behind the privacy rights and expectations they have under law and social norms. Perhaps most importantly, Knowsi also takes the guesswork out of good privacy practice in a research team. Forgotten forms, unclear communication, and complexity of regulatory compliance make for considerable liability concerns that any research team needs to navigate. Knowsi automates the complexity of compliance, and helps you and your team focus on the content and the research.
+ Does the Knowsi platform come with detailed instructions/guidelines on how to use it?
For the most part, Knowsi relies on a clear user experience to guide both researchers and participants through the application. However, there is a growing set of documentation that lives in the question mark icon on the bottom of the screen. You can also use this feature to directly contact the Knowsi team for any kind of assistance.
+ Sometimes we need to use deceptive strategies in research so participants do not change their behaviour. Is it always necessary to obtain full consent from participants while doing research on them?
Knowsi allows you to collect consent both remotely and in person - but it is incumbent on you to not be deceptive in how you collect that consent. Basically, consent needs to be informed, freely given, and generally unambiguous when it comes to data collection about individuals. You can update consent after data has been provided though, and send users the appropriate consent form to use the gathered data in this new way. I’d suggest engaging with your REB to specifically workshop the language of your form before sharing it with your participants, as frequently you can gather specific consents while also maintaining the integrity of the study.
+ Does Knowsi work directly with any Ethics Review Boards at universities? As Knowsi is GDPR-compliant, I think it could make getting approval through Ethics Review Boards easier.
Absolutely. Please contact andrew@knowsi.com to arrange a meeting.
+ The excel tracker template shown in the webinar was great! If I sign up to Knowsi can I download this?
You can download the template [here] 1. I share templates like this and other handy assets in my newsletter. You can sign up to the newsletter by registering for Knowsi or simply from the form at the bottom of our website.