Lab 1: HRI Study Design

To Complete Before Lab

Read the paper: Hoffman, G., & Zhao, X. (2020). A primer for conducting experiments in human–robot interaction. ACM Transactions on Human-Robot Interaction (THRI), 10(1), 1-31.

Learning Goals

Students will learn how to design an HRI experiment by going through the an example HRI study design process.
Students will gain experience working in collaborative groups to develop, discuss, and evaluate HRI experiment designs.
Students will gain knowledge of and experience with tools used in experiment design (e.g., running an a-priori power analysis).

Working in Groups

During this lab, you will work collaboratively in a group of 3 students. To allow you to get to know your classmates better, we have made random group assignments for this lab that can be found in this Google spreadsheet.

Lab 1 Deliverables & Submission

Your work on Lab 1 will involve designing an HRI experiment by filling out this Lab 1 HRI Study Design Worksheet. Please open the Google Doc worksheet, copy its contents, and start a new document where you and your group members can contribute to it.

To receive credit for this lab, your group will need to submit your completed study design worksheet to Canvas by Monday, March 31 at 1:30pm.

Note: For labs with deliverables, the deadline for submitting lab deliverables will normally be the following day (Friday) at 6pm. However, for this first lab, we're giving a bit extra time due to this being the first week of the quarter.

Lab 1 Exercise: HRI Study Design

The goal of this lab is to learn how to design an HRI experiment. Our main reference material for this lab is the paper: Hoffman, G., & Zhao, X. (2020). A primer for conducting experiments in human–robot interaction. ACM Transactions on Human-Robot Interaction (THRI), 10(1), 1-31.

You are expected to have read this paper before lab and use this paper as a reference as you go through each of the study design steps below.

Your Research Goal & Guiding Research Question

Your goal for this lab is to design social robot companions that improve people's everyday lives. Social robots have the potential to provide high-quality and affordable academic tutoring to children, enable older adults to remain self-sufficient by providing assistance with daily living tasks, and provide companionship for an increasing number of people facing loneliness and isolation. Despite the incredible potential of these social robots and the recent technical advances that have made these social robot platforms a possibility, we have yet to see successful social robot platforms in U.S. markets (e.g., Jibo, Anki, Kuri) or in research settings that people want to interact with over the long-term (i.e., months or years).

Your goal as an HRI researcher is to investigate how a robot can be designed to sustain long-term human-robot interactions. You are interested in pursuing this research goal in the context of an at-home homework assistant robot for 4th grade students that can both help answer questions and (more importantly) keep the child engaged and motivated to complete their homework.

The big-picture research question we are providing you as a starting point for this lab is: How can an at-home homework assistant robot for 4th grade students be designed to sustain long-term interactions?

Developing Your Own Study Design

At this point, you'll start following, discussing as a group, and filling out your group's copy of the Lab 1 HRI Study Design Worksheet.

Study Design Resources

Questionnaire Measures

When I develop my questionnaires, I typically consider including the following types of questions:

Manipulation checks: These questions allow you to verify that participants perceived the robot in the way you intended. For example, if I have 3 experimental conditions where a robot is expressing different personality types (e.g., high neuroticism, high extraversion, control), my manipulation check would be having participants assess the robot's personality, to ensure that the high neuroticism robot is truly perceived as higher in neuroticism.
Psychometric Scales: The best questionnaire measure to use is one that has been developed by someone else and already validated to measures a particular construct. This Finding the "Perfect" Scale database is a brilliant resource for HRI researchers looking for scales to use for their research.
Your Own Scale Questions/Items: I will often need to design unique questions for each of my studies that are tailored to the exact study and context I'm evaluating. If I cannot find a preexisting scale, I will develop my own questions participants rate using a Likert scale.
Open-Ended Response Questions: In addition to rating items on a Likert scale, I also always have open-ended text-box response questions. This allows participants to express observations about the experiment that may not be captured in a more constrained Likert scale. For example, I'll generally start out with a broad question, such as, "What was your impression of the robot?" and then gradually get more and more specific. I typically don't ask any more than 3 open-response questions.

Conducting a Post-Experiment Interview

One practice I've recently adopted in my own research is ALWAYS conducting post-experiment interviews in addition to collecting questionnaire data. This is a great way to get a more in-depth understanding of the participant's experience and can help you better understand why participants behaved the way they did in the experiment.

I typically suggest asking about 3 questions in the post-experiment interview and leaving flexibility for the interviewer to go into greater depth in each question as needed.

Conducting an A-Priori Power Analysis to Determine Experiment Sample Sizes

Conducting a power analysis is the best way to answer the question, "How many participants should I recruit for my HRI study?" To conduct a power analysis, I'd suggest using the free and well-used G*Power tool.

When calculating sample size using a power analysis, G*Power is pretty easy to follow. You select the statistical test you plan to run, the type of power analysis you want to conduct, and input some parameters -- and voila, you have the number of participants you should run in order to ensure that your study is sufficiently powered. Now, here are some notes on how to determine your parameter values:

Effect size: The best way to determine an effect size is to find a human subjects study examining a similar research question to yours and use the effect size reported in that other study. If you cannot find a listed effect size, it's best practice to assume a medium effect size. I've found this Rules of thumb on magnitudes of effect sizes web page to be helpful in determining small/medium/large effect sizes and also converting between effect sizes.
α err prob: Typically, we use a value of 0.05 here, representing the fact that statistical significance is determined by p-values less than 0.05.
Power (1-β err prob): This represents the desired statistical power of the study. For in-person human subjects studies, I recommend going with a power of 0.80 (or 80%). For online studies (e.g., Prolific, AMT), you may want to go with a higher power of 0.90-0.95 (09%-95%).
Number of groups: This represents the number of between-subjects conditions your study contains.
Numerator df: You may be asked to detail the number of degrees of freedom your study contains. If you have 3 conditions, for example, your number of degrees of freedom is 2 (number of conditions - 1). If you have a 2x2 experimental design, your number of degrees of freedom is 1. This is because as one condition is fixed, only one can vary.
Number of measurements: If you are conducting a study where you repeatedly take measurements of the same participant, this value will represent how many measurements each participant receives.

The following screenshot represents an example of a power computation for a 1-way ANOVA analysis with a medium effect size and 80% power for a 3 condition between subjects design. Based on this analysis, I should have a total sample size of 159. 159/3 = 53, so I should aim to recruit 53 participants in each of my 3 between subjects conditions.

Pre-registration

Pre-registration is a practice where you submit your study design, hypotheses, and analysis plan to a third-party website before you collect data. This is a great way to ensure that you are not engaging in HARK-ing (Hypothesizing After the Results are Known). I recommend using aspredicted.org for pre-registration.