Course Project

The course project is your chance to gain hands on HRI research experience. I highly recommend that you complete this project in a group of 2 or 3. If you wish to complete your project individually, please email me to discuss further. This project must be focused on the interaction between physically embodied robots and 1 or more people. This page contains details on the project deliverables, due dates, as well as suggestions for possible projects (at the bottom of the page).

Project Deliverables & Due Dates

Project Timeline

Due Date	Time Due	Project Deliverable
Fri, Jan 12	9:30am CST	Pitch 1-3 Project Ideas
Wed, Jan 17	9:30am CST	Draft Project Proposal
Wed-Mon, Jan 17-22	9:30am CST	Refine Project Proposal
Mon, Jan 22	9:30am CST	Finalized Project Proposal
Mon, Feb 5	9:30am CST	Progress Report #1
Mon, Feb 19	9:30am CST	Progress Report #2
Wed, Feb 28 & Fri, Mar 1		Project Presentations to Class
Thu, Mar 7	5:00pm CST	Project Final Report & Teaming Survey

Submission of Project Deliverables

Put your project pitches in this Google doc.

Once you have formed your team, make a Google Drive folder for your project team that you share with the teaching team (sarahsebo@uchicago.edu and llwright@uchicago.edu). You'll turn in of all your project deliverables to this Google Drive folder.

Project Pitches

Before pitching project ideas, read this entire web page thoroughly to get the full picture of what this project will entail.

Although the project will be completed in teams of 2-4, these pitches are made by individuals. The purpose of these project pitches is to aid in the formation of project teams (your project proposal can and will deviate from these project pitches, again, the project pitches are meant to serve as a starting point for team formation). In this Google document you’ll pitch 1-3 project ideas that you’d be interested in pursuing for the course project. It does not need to be fully formed, but enough to give your classmates a good idea of what the project will entail.

Draft Project Proposal

The draft proposal should be 1-2 pages in length and describe:

The research question(s) you are investigating and any related hypotheses
The methods you expect to use to answer your research question (e.g., experimental conditions, experiment protocol, questionnaire measures, measurements of human behavior)
How you plan to evaluate whether or not you've answered your research question (e.g., statistical analysis, thematic analysis, comparison against a state-of-the-art approach)

Please share your project proposal with me by putting it in your group's Google Drive folder (see "Submission of Project Deliverables" above).

Finalized Project Proposal

The finalized project proposal, in addition to what is contained in the draft proposal, should detail a detailed timeline with corresponding intermediate deliverables. Before submitting your final project proposal, please set up a meeting with either me or Lauren in the range of Wed-Fri, Jan 18-20. You'll receive an email from me on how to sign up for a time to meet with me to discuss your project proposal.

Progress Reports

The progress report should contain:

An update on your progress relative to your project proposal timeline (1 paragraph)
The accomplishments you've made since the last project deliverable (1 paragraph or a bullet point list)
Any unexpected roadblocks you've encountered (1-2 paragraphs or a bullet point list)
An updated timeline of how you plan to complete the project

Project Presentation

You will present the work that you've done in your course project during our last two class meetings of the quarter. Half of the project teams will present on each date. You should aim for your presentation to take 8-10 minutes, leaving 3-5 minutes for Q&A and transitioning to the next group (I will plan to cut you off if your presentation goes beyond 10 minutes). All members of your team are expected to participate in the presentation. Your slides should be uploaded to your team's Google Drive folder by the start of class on the day you present. Your presentation should highlight the following:

Motivation: Detail the research question(s) you were interested in investigating and why it's important to explore.
Background: Describe any relevant background literature and tell us what unique and novel contributions your study is making beyond what already exists in the literature.
Methods: Describe your study design (experimental conditions), hypotheses, study protocol (videos are great for this), and the measures you used to assess your hypotheses.
Results: Report on the results you found. Please include all of the main measures, regardless of whether or not you found significant differences.
Discussion/Implications: Reflect on the broader implications and takeaways from your results.
Challenges: Describe your main challenges in conducting your research project and how you addressed them.
Learning Outcomes: Enumerate some "lessons learned" from this project, what you learned by going out and actually conducting HRI research, beyond what you've learned from reading the papers in this class.

Your presentation will also be graded for its design and style. Please ensure that your slides are NOT too texy heavy. If the audience is spending their time trying to read a lot of text on a slide, it's hard for them to pay attention to you. Use visuals and limited text on your slides, describing the details verbally during your presentation.

Final Report

The project report should be written in the style of an academic conference paper (like the papers we've been reading for our class discussions), feel free to use the ACM templates for conferences. The report should have the following sections: abstract, introduction, background, methods, results, discussion, conclusion, and contributions. The abstract - conclusion sections should resemble the papers we have read and discussed in this course. The contributions section should detail the contributions of each team member, detailing which portions of the project were completed by each team member. The report should be approximately 6 pages long without references. Please both 1) send the teaching team an email of your final report and 2) upload a copy to your team's Google Drive folder.

Teaming Survey

Due at the same time as your final report, ensure that you've filled out the course project teaming survey. Assuming that all team members make approximately equal contributions on the course project, all team members will receive the same grade. However, if certain team member(s) do not contribute a sufficient amount of work to the course project, each team member may receive a different grade that reflects their individual contributions.

Project Grading

The grading for the course project will be as follows:

15% intermediary deliverables - project pitches, proposals, progress reports
10% final project presentation
25% final report
50% project work - your grade based on the work that you accomplished during the class project is not dependent on your project being a 'success', your project work will be graded based on the project's scope, novelty, and the investigative work that you put into pursuing your research questions

For further details on project grading, please check out the final project grading rubric.

Getting Started

Here are some different types of projects you could consider:

An in-person human-subjects study: Similar to most of the papers we read in class, this option gives you the chance to conduct a human-subjects study of your own design. You can work with any of the robots in the Human-Robot Interaction lab (see the available robots below) and you will be expected to recruit and run your own human participants.
A human-subjects study evaluated on an online crowd-sourced platform: Running a human-subjects study on an online crowd-sourced platform (e.g., Prolific, Amazon Mechanical Turk) has advantages over in-person human subjects studies because recruitment is significantly easier and data can be collected quickly. However, the downside is that the human-robot interaction is not in-person, and thus, a bit less realistic. Here are some recent examples of recent papers published at the HRI conference that relied on crowd-sourced human-subjects studies:
- Reig et al. (2021) - A study exploring people's perceptions of multi-robot recovery strategies when one robot experiences a failure.
- Hedaoo et al. (2019) - A study examining people's impressions of a robot making a comment to multiple people in different settings (potential romantic partners, roommates, or job colleagues) and with different valences (going well or going poorly).
- Chita-Tegmark et al. (2019) - A study examining people's perceptions of a robot's gender and emotional intelligence in one-on-one interactions.
- de Graaf and Malle (2019) - A study of how people ascribe mental states to robots through their explanations of robot behavior.
Analysis of a human-robot interaction in a pre-recorded dataset: Two datasets I'm aware of are the Vernissage Dataset and the UE-HRI dataset. For a larger picture view on the use of datasets in HRI, I'd recommend checking out the Working Group on Datasets with Humans and Agents in Groups section of the Dagstuhl Seminar I recently attended on "Social Agents for Teamwork and Group Interactions".
A literature review: You could take a dive into the published work on a particular topic, analyze it, critique it, and suggest avenues for future research. There are some excellent examples of literature reviews within HRI on robots for education, gaze, and robot physical embodiment. Or you can propose a framework that summarizes and puts into perspective the work done within a particular area of HRI (e.g., this recent work on trust-relevant failures and mitigation strategies).

Examples Projects from Prior Topics in HRI Student Teams

The Influence of Ingroup/Outgroup Robots on Participant Compliance to Robot Instructions to Undermine Another Robot

As robots become increasingly capable and widespread, they may be placed into roles of relative authority where they give people instructions (e.g., directing human coworkers in a warehouse). Given people's tendency to comply with either robot or human instructions, it is important to better understand the factors that may influence human compliance to robot instructions, especially if these instructions may undermine another person or robot. In this work, this project team investigated to what extent an established robot-robot relationship would impact a person's choice to comply with instructions from one robot to undermine another robot. They ran a between-subjects study (N = 10) where participants collaborated with one a partner robot to build a series of towers at the direction of a manager robot. These two robots were either presented as an ingroup with a shared history and preferential treatment of one another (ingroup condition) or as an outgroup without shared history and negative treatment of one another (outgroup condition). In response to the manager robot's instructions to the human participant to undermine the partner robot, their results show that participants in the ingroup condition are more likely to abuse the partner robot per the manager robot's commands than in the outgroup condition. The ingroup robots were perceived as more warm, more competent, and causing less discomfort.

The students who worked on this project continued working on this project after the quarter ended, re-vamped the study design, ran 50 new participants, and are working to get the resulting paper submitted to an academic conference or journal.

Benefits of an Interactive Robot Character in Immersive Puzzle Games

Robots are becoming increasingly prominent in the entertainment sphere, where they interact with guests in themed environments to tell stories, often in place of human characters. To evaluate the potential benefits of robots in these contexts compared to humans, this team created an interactive puzzle game where either a robot or a human actor serves as a diegetic “game guide” character that is both a cooperative partner and an omniscient game master. In the game, participants solve a crime mystery by asking the game guide for information to complete tasks and for hints to solve puzzles. They conducted a between-subjects study (n = 25) to investigate how players’ game experiences differed when the game guide was a human compared to an embodied robot. Their results showed that participants playing with a robot had more fun, felt more comfortable and less awkward, and made more progress in solving the tasks compared to those playing with a human. These results suggest that robots can be effective alternatives to human actors in broader immersive entertainment contexts such as escape rooms to provide greater enjoyment and promote more social interaction with in-game characters.

The students who worked on this project continued working on this project after the quarter ended and published a paper on this project at the 2022 RO-MAN conference.

Be Nice to Garbage Bot

This project explored how the addition of eyes to a non-humanoid robot affects human responses to requests for assistance from the robot. The experiment was executed in a public setting (near Peaches on the first floor of JCL), with the Stretch robot asking passers-by for help picking up an item of garbage. Participants interacted with a robot with and without eyes by assisting it to pick up a piece of trash that it audibly requested help with. Most people from whom the robot requested help paused to assess the situation, however, the team found far more participants in the eye condition actually intervening. The no eyes condition had similar rates of participants pausing to view the robot, but these participants more often described confusion about the source of the voice or the intentions of the robot. The team also recorded whether the people that approached the robots were individuals or small groups. They found that all the small groups that they recorded chose to intervene, they believe, because members of small groups were more likely to stop and discuss the robot, as well as to reassure each other about how to assist it.

You can find a fun YouTube video demonstrating this project at this link.

Observing Robotic Touch: Effects of Appearance, Personality, and Intent on Quality of Care

Human-robot touch offers a compelling medium for communication; touch can be used both to perform tasks (instrumental touch) and provide comfort (affective touch). Deliberate, physical contact with humans is especially applicable for healthcare robots because touch is frequently employed and often necessary to provide patients with medical aid and emotional support. To understand the factors that influence how people perceive robotic caregivers, these students conducted a 2x2x2x2 crowd-sourced study on Prolific, where each participant viewed a video recording of one of the students being given a medical exam by a robot and then filled out a Qualtrics survey based on the observed video clip. The students examined how the perceived quality of care is affected by the presence of touch (present or absent), type of touch (instrumental or affective), robot appearance (Nao or Stretch), and robot tone (empathetic or serious). Their findings confirmed that instrumental touch is more important to quality of care than affective touch and were able to suggest several guidelines for designing robot personality and appearance to maximize the warmth, competence, and comfort of a patient’s experience.

The students who worked on this project continued working on this project after the quarter ended and published a paper on this project at the 2022 RO-MAN conference.

Note: This study was conducted on Prolific (a crowd-sourcing platform) because the covid-19 pandemic prevented these students from running any in-person human subjects.

Robots

If you are interested in using a physical robot as a part of your project, here are the robots available for you to use in my lab:

Nao V6

Franka Emika

Misty II

Stretch

Turtlebot3

ClicBot

Once you are ready to start working with one or more of these robots, please reach out to me for resources and example code for programming them and also to request access from me (Sarah) to the Human-Robot Interaction (HRI) lab. The HRI lab is located in JCL 379 and is the place where you can find all of the robots and will be a place where we recommend that you spend your time programming the robot(s).

Helpful Information for Running a Human-Subjects Study

Human Subjects Protection Training

If you plan to conduct a human subjects study in your class project, regardless of whether it is in-person or online (e.g., Prolific), you will need to submit an IRB (Internal Review Board) Application and have completed Human Subjects Protection Training (← that link contains all the info you need to complete the training). Before you submit your IRB application, you must have already completed the human subjects protection training.

UChicago Internal Review Board (IRB) Application

In order to conduct human subjects research in affiliation with UChicago, you must submit and get approved an Internal Review Board (IRB) application. This means that BEFORE you run any human subjects, you MUST have your IRB application approved. For HRI related research, we submit our IRB applications to UChicago's Social and Behavioral Sciences Institutional Review Board (SBS IRB). Their main website, https://sbsirb.uchicago.edu/, contains helpful information, templates, and guidance for completing an IRB application.

The purpose of the IRB is to ensure that UChicago research is being conducted ethically and with proper respect to human participants. Your IRB application will include:

a high-level description of the research you plan to conduct
a protocol specifying what each human participant will do in your research study
a consent form that each participant will read and agree to before beginning your research study
a description of how you plan to protect any identifiable data you collect from human participants and how you plan to store the data
any questionnaires you plan to administer to the human participants
any recruitment methods you plan to use (e.g., flyers, social media posts, emails)
details on how participants are to be paid and how much they will be paid

To create, edit, submit, and amend your IRB application go to https://aura.uchicago.edu/ and click on the "IRB" button.

Note: The first time you access the IRB (by going to https://aura.uchicago.edu/ and clicking on the "IRB" button), you will have the opportunity to register for an AURA account. ALL of your team members must register this way in order to be added to your IRB protocol.

It is most helpful to model your IRB application after one that has already been approved. Please ask the teaching staff to provide you with model IRB application examples to help you as you create your own.

When you complete your own IRB, please put me (Sarah) as the PI and all other teaching team members as collaborators on the personnel list. Since I (Sarah) will be the PI for the IRB, this means that I must be the one to hit the "submit" button whenever you submit the IRB application or revisions to your IRB application. Please email me when you are ready to make an IRB submission, and I will hit that "submit" button.

Determining the Number of Participants to Recruit for Your Study

Ideal World Scenario. If you have unlimited time and resources, you determine the number of participants you should recruit for your study by conducting a statistical power analysis. You can find detailed information on how to conduct a power analysis by going to my "Conducting Statistical Analyses of HRI Data" guide that can be found on the For Lab Members page on my HRI lab website. Click the "Access Lab Wiki" button. You will need to enter the password to enter (please ask the teaching staff for this password). Then find the wiki entry titled "Conducting Statistical Analyses of HRI Data" and the section on "How Many Participants Should I Recruit? A Guide to Conducting a Power Analysis to Determine Experiment Sample Sizes."

Real World Scenario. If you are limited by time and/or resources, then you may be not be able to recruit as many participants as you wish (especially given the short time frame of UChicago's academic quarters). For those running in-person human-subjects experiments, we recommend you try to recruit the following minimum number of participants, dependent on your group size:

Group of 2: at least 10 participants
Group of 3: at least 16 participants
Group of 4: at least 24 participants

If you have any questions about how many participants we deem as acceptable for you to achieve a good grade on the project, please chat with us about it.

Qualtrics Questionnaires

If your study involves asking human participants to fill out a questionnaire, I recommended that you use Qualtrics. In order to get all the nice bells and whistles that comes with Qualtrics, either 1) you need a pro account or 2) you can ask me (Sarah) to start a Qualtrics survey on my pro account and I can share it with you. I am more than happy to help you out with option #2, please email me.

Commonly Used Questionnaire Measures in HRI

These are the most commonly used and recommended pre-existing questionnaires that you might consider using for your human subjects experiment:

RoSAS - measures perceptions of the robot on the dimensions of warmth, competence, and discomfort
- this is my FAVORITE and most used questionnaire to assess how people perceive robots
- Carpinella, C. M., Wyman, A. B., Perez, M. A., & Stroessner, S. J. (2017, March). The robotic social attributes scale (rosas) development and validation. In Proceedings of the 2017 ACM/IEEE International Conference on human-robot interaction (pp. 254-262).
MDMT - measures trust in the robot on 4 dimensions: reliable, capable, ethical, and sincere
- Ullman, D., & Malle, B. F. (2018, March). What does it mean to trust a robot? Steps toward a multidimensional measure of trust. In Companion of the 2018 acm/ieee international conference on human-robot interaction (pp. 263-264).
- Ullman, D., & Malle, B. F. (2019, March). Measuring gains and losses in human-robot trust: Evidence for differentiable components of trust. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI) (pp. 618-619). IEEE.
Godspeed - measures perceptions of the robot on the dimensions of anthropomorphism, animacy, likability, perceived intelligence, and perceived safety
- Bartneck, C., Kulić, D., Croft, E., & Zoghbi, S. (2009). Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International journal of social robotics, 1(1), 71-81.

Running an Online/Crowd-Sourced Human Subjects Study

If you are interested in running an online or crowd-sourced human subjects study, I would recommend using Prolific. The easiest way to set this up is to create a Qualtrics questionnaire and link to that questionnaire on Prolific.

Timing. Before you launch your study, you will want to have a friend or two take your survey to get a sense of approximately how much time it will take participants to complete your survey. This will also determine how much you will pay each of your participants.

Prescreening Participants. I typically use the following criteria:

Current Country of Residence: United States (I only use this criteria if I think that someone's cultural background will influence their responses)
Fluent languages: English
Approval Rate: 90-100

Payment. We will pay our participants at the rate of $8.00 per hour. When you are ready to launch your study, you will email me (Sarah) with your Prolific login credentials (please ensure that your password is something you do not mind sharing with me), and I will go into your account and add the appropriate amount of money to your account. Then you can launch your study.

Piloting. It is very important to pilot your study before going ahead and running it. We recommend that you have one or two friends go through your study to ensure that everything makes sense and to catch any typos. Then, ask me (Sarah) to go through your study, I would love to give you feedback on it. Piloting is essential to work out any kinks and ensure that your study is set up to adequately answer your research questions.

Staging Study Launch. We HIGHLY RECOMMEND that you start out by running 20 or so participants in your study first, so that you can ensure that your study is working properly, before running the rest of your planned participants. To ensure that your study is working, please go through the data you collected in Qualtrics to ensure that participants make it through your survey as you intend and answer the questions as you intend them to.

Running an In-Person Human Subjects Study

If you want to run an in-person human subjects study, here's an example timeline of what your project work might look like:

Weeks 1-3: Form project team, finalize study design (main research questions, experimental conditions)
Week 4: Develop study protocol, questionnaires, consent forms, etc. and submit your IRB application
Week 5: Program the robot and go back-and-forth with the IRB if they request any changes to your IRB application
Week 6: Pilot the study
Weeks 7-8: Run human subjects
Week 9: Conduct data analysis and put together final presentation
Week 10: Write final report

Please note that the quarter is extremely short, you will have to be on top of it from day 1.

Recruitment. You will be responsible for recruiting your own participants. You can post on social media, send emails, post on UChicago organization sites, etc. Just be sure that you include all forms of recruitment in your IRB application that you plan to use. We recommend that you aim to recruit 10-20 participants for each of your experimental conditions if you are running a between-subjects study.

Payment. For in-person human participants, we will pay them at the rate of $12.00 per hour. We will pay them in the form of Amazon gift cards. To facilitate this payment, please email me (Sarah) daily when you are running your study with a list of email addresses that correspond with your study participants from that day. I will ensure that they are sent their Amazon gift card for participating in your study.

Piloting Your Study. It is SO important before you start running real human subjects that you practice, practice, practice executing your human subjects study. We recommend:

Running the study several times yourselves from top to bottom, AND
Running the study one one or two friends (including having them fill out the questionnaires) and asking them if anything was confusing or if they noticed any typos in your questionnaire, AND
Running the study on Sara and getting her feedback.

Piloting is essential to work out any kinks and ensure that your study is set up to adequately answer your research questions.

Statistical Data Analysis

I (Sarah) have created a guide for conducting statistical analysis on human subjects data, complete with a thorough answer to the question "what statistical test should I run?" and examples in R of how to run each statistical test. You can find this guide by navigating to the For Lab Members page on my HRI lab website. Click the "Access Lab Wiki" button. You will need to enter the password to enter (please ask the teaching staff for this password). Then find the wiki entry titled "Conducting Statistical Analyses of HRI Data", which will contain everything you need. If you have any questions about conducting your data analysis, please do not hesitate to reach out to me.