Course Project

The course project is your chance to gain hands on HRI research experience. I highly recommend that you complete this project in a group of 2 or 3. If you wish to complete your project individually, please email me to discuss further. This project must be focused on the interaction between physically embodied robots and 1 or more people. This page contains details on the project deliverables, due dates, as well as suggestions for possible projects (at the bottom of the page).

Project Deliverables & Due Dates

---

Project Timeline

Submission of Project Deliverables

Put your project pitches in this Google doc.

Once you have formed your team, make a Google Drive folder for your project team that you share with the teaching team (sarahsebo@uchicago.edu and vzhao@uchicago.edu)). You'll turn in of all your project deliverables to this Google Drive folder.

Project Pitches

Before pitching project ideas, read this entire web page thoroughly to get the full picture of what this project will entail.

Although the project will be completed in teams of 2-3, these pitches are made by individuals. The purpose of these project pitches is to aid in the formation of project teams (your project proposal can and will deviate from these project pitches, again, the project pitches are meant to serve as a starting point for team formation). In this Google document you’ll pitch 1-3 project ideas that you’d be interested in pursuing for the Topics in HRI course project. It does not need to be fully formed, but enough to give your classmates a good idea of what the project will entail.

Draft Project Proposal

The draft proposal should be 1-2 pages in length and describe:

1. the research question you are investigating and any related hypotheses,
2. the methods you expect to use to answer your research question, and
3. how you plan to evaluate whether or not you've answered your research question.

Please share your project proposal with me by putting it in your group's Google Drive folder (see "Submission of Project Deliverables" above).

Finalized Project Proposal

The finalized project proposal, in addition to what is contained in the draft proposal, should detail a detailed timeline with corresponding intermediate deliverables. Before submitting your final project proposal, please set up a meeting with the teaching team in the range of Tue-Fri, Oct 12-15. You'll receive an email from me (Sarah) on how to sign up for a time to meet with the teaching team to discuss your project proposal.

Progress Reports

The progress report should contain:

1. An update on your progress relative to your project proposal timeline
2. Any accomplishments you've made since the last project deliverable
3. Any unexpected roadblocks you've encountered
4. An updated timeline of how you plan to complete the project

Project Presentation

You will present the work that you've done in your course project on either December 1 or December 3. Half of the project teams will present on each date. You should aim for your presentation to take 12 minutes, leaving 3 minutes for Q&A (I will plan to cut you off if you go beyond 12 minutes). All members of your team are expected to participate in the presentation. Your slides should be uploaded to your team's Google Drive folder by the start of class on the day you present. Your presentation should highlight the following:

• Motivation: Detail the research question(s) you were interested in investigating and why it's important to explore.
• Background: Describe any relevant background literature and tell us what unique and novel contributions your study is making beyond what already exists in the literature.
• Methods: Describe your study design (experimental conditions), hypotheses, study protocol (videos are great for this), and the measures you used to assess your hypotheses.
• Results: Report on the results you found. Please include all of the main measures, regardless of whether or not you found significant differences.
• Discussion/Implications: Reflect on the broader implications and takeaways from your results.
• Challenges: Describe your main challenges in conducting your research project and how you addressed them.
• Learning Outcomes: Enumerate some "lessons learned" from this project, what you learned by going out and actually conducting HRI research, beyond what you've learned from reading the papers in this class.

Final Report

The project report should be written in the style of an academic conference paper (like the papers we've been reading for our class discussions), feel free to use the ACM templates for conferences. The report should have the following sections: abstract, introduction, background, methods, results, discussion, conclusion, and contributions. The abstract - conclusion sections should resemble the papers we have read and discussed in this course. The contributions section should detail the contributions of each team member, detailing which portions of the project were completed by each team member. The report should be approximately 6 pages long without references. Please both 1) send the teaching team an email of your final report and 2) upload a copy to your team's Google Drive folder.

Project Grading

---

The grading for the course project will be as follows:

• 15% intermediary deliverables - project pitches, proposals, progress reports
• 10% final project presentation
• 25% final report
• 50% project work - your grade based on the work that you accomplished during the class project is not dependent on your project being a 'success', your project work will be graded based on the project's scope, novelty, and the investigative work that you put into pursuing your research questions

For further details on project grading, please check out the final project grading rubric.

Getting Started

---

Here are some different types of projects you could consider:

• An in-person human-subjects study: Similar to most of the papers we read in class, this option gives you the chance to conduct a human-subjects study of your own design. You can work with any of the robots in the Human-Robot Interaction lab (see the available robots below) and you will be expected to recruit and run your own human participants.
• A human-subjects study evaluated on an online crowd-sourced platform: Running a human-subjects study on an online crowd-sourced platform (e.g., Prolific, Amazon Mechanical Turk) has advantages over in-person human subjects studies because recruitment is significantly easier and data can be collected quickly. However, the downside is that the human-robot interaction is not in-person, and thus, a bit less realistic. Here are some recent examples of recent papers published at the HRI conference that relied on crowd-sourced human-subjects studies:
  • Reig et al. (2021) - A study exploring people's perceptions of multi-robot recovery strategies when one robot experiences a failure.
  • Hedaoo et al. (2019) - A study examining people's impressions of a robot making a comment to multiple people in different settings (potential romantic partners, roommates, or job colleagues) and with different valences (going well or going poorly).
  • Chita-Tegmark et al. (2019) - A study examining people's perceptions of a robot's gender and emotional intelligence in one-on-one interactions.
  • de Graaf and Malle (2019) - A study of how people ascribe mental states to robots through their explanations of robot behavior.
• Analysis of a human-robot interaction in a pre-recorded dataset: Two datasets I'm aware of are the Vernissage Dataset and the UE-HRI dataset. For a larger picture view on the use of datasets in HRI, I'd recommend checking out the Working Group on Datasets with Humans and Agents in Groups section of the Dagstuhl Seminar I recently attended on "Social Agents for Teamwork and Group Interactions".
• A literature review: You could take a dive into the published work on a particular topic, analyze it, critique it, and suggest avenues for future research. There are some excellent examples of literature reviews within HRI on robots for education, gaze, and robot physical embodiment.Or you can propose a framework that summarizes and puts into perspective the work done within a particular area of HRI (e.g., this recent work on trust-relevant failures and mitigation strategies).

Examples Projects from Prior Topics in HRI Student Teams

---

Robots

---

If you are interested in using a physical robot as a part of your project, here are the robots available for you to use in my lab:

Once you are ready to start working with one or more of these robots, please reach out to the teaching team for resources and example code for programming them and also to request access from me (Sarah) to the Human-Robot Interaction (HRI) lab. The HRI lab is located in JCL 379 and is the place where you can find all of the robots and will be a place where we recommend that you spend your time programming the robot(s).

Helpful Information for Running a Human-Subjects Study

---

Human Subjects Protection Training

If you plan to conduct a human subjects study in your class project, regardless of whether it is in-person or online (e.g., Prolific), you will need to submit an IRB (Internal Review Board) Application and have completed Human Subjects Protection Training (← that link contains all the info you need to complete the training). Before you submit your IRB application, you must have already completed the human subjects protection training.

UChicago Internal Review Board (IRB) Application

In order to conduct human subjects research in affiliation with UChicago, you must submit and get approved an Internal Review Board (IRB) application. This means that BEFORE you run any human subjects, you MUST have your IRB application approved. For HRI related research, we submit our IRB applications to UChicago's Social and Behavioral Sciences Institutional Review Board (SBS IRB). Their main website, https://sbsirb.uchicago.edu/, contains helpful information, templates, and guidance for completing an IRB application.

The purpose of the IRB is to ensure that UChicago research is being conducted ethically and with proper respect to human participants. Your IRB application will include:

• a high-level description of the research you plan to conduct
• a protocol specifying what each human participant will do in your research study
• a consent form that each participant will read and agree to before beginning your research study
• a description of how you plan to protect any identifiable data you collect from human participants and how you plan to store the data
• any questionnaires you plan to administer to the human participants
• any recruitment methods you plan to use (e.g., flyers, social media posts, emails)
• details on how participants are to be paid and how much they will be paid

To create, edit, submit, and amend your IRB application go to https://aura.uchicago.edu/ and click on the "IRB" button.

It is most helpful to model your IRB application after one that has already been approved. Please ask the teaching staff to provide you with model IRB application examples to help you as you create your own.

When you complete your own IRB, please put me (Sarah) as the PI and all other teaching team members as collaborators on the personnel list. Since I (Sarah) will be the PI for the IRB, this means that I must be the one to hit the "submit" button whenever you submit the IRB application or revisions to your IRB application. Please email me when you are ready to make an IRB submission, and I will hit that "submit" button.

Determining the Number of Participants to Recruit for Your Study

Ideal World Scenario. If you have unlimited time and resources, you determine the number of participants you should recruit for your study by conducting a statistical power analysis. You can find detailed information on how to conduct a power analysis by going to my "Conducting Statistical Analyses of HRI Data" guide that can be found on the For Lab Members page on my HRI lab website. Click the "Access Lab Wiki" button. You will need to enter the password to enter (please ask the teaching staff for this password). Then find the wiki entry titled "Conducting Statistical Analyses of HRI Data" and the section on "How Many Participants Should I Recruit? A Guide to Conducting a Power Analysis to Determine Experiment Sample Sizes."

Real World Scenario. If you are limited by time and/or resources, then you may be limited to recruit the maximum number of participants you can with the time and money available to you. There is no shame in this, especially given the short time frame of UChicago's academic quarters. If you have any questions about how many participants we, the teaching team, deem as acceptable for you to achieve a good grade on the project, please chat with us about it.

Qualtrics Questionnaires

If your study involves asking human participants to fill out a questionnaire, I recommended that you use Qualtrics. In order to get all the nice bells and whistles that comes with Qualtrics, either 1) you need a pro account or 2) you can ask me (Sarah) to start a Qualtrics survey on my pro account and I can share it with you. I am more than happy to help you out with option #2, please email me.

Commonly Used Questionnaire Measures in HRI

These are the most commonly used and recommended pre-existing questionnaires that you might consider using for your human subjects experiment:

• RoSAS - measures perceptions of the robot on the dimensions of warmth, competence, and discomfort
  • this is my FAVORITE and most used questionnaire to assess how people perceive robots
  • Carpinella, C. M., Wyman, A. B., Perez, M. A., & Stroessner, S. J. (2017, March). The robotic social attributes scale (rosas) development and validation. In Proceedings of the 2017 ACM/IEEE International Conference on human-robot interaction (pp. 254-262).
• MDMT - measures trust in the robot on 4 dimensions: reliable, capable, ethical, and sincere
  • Ullman, D., & Malle, B. F. (2018, March). What does it mean to trust a robot? Steps toward a multidimensional measure of trust. In Companion of the 2018 acm/ieee international conference on human-robot interaction (pp. 263-264).
  • Ullman, D., & Malle, B. F. (2019, March). Measuring gains and losses in human-robot trust: Evidence for differentiable components of trust. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI) (pp. 618-619). IEEE.
• Godspeed - measures perceptions of the robot on the dimensions of anthropomorphism, animacy, likability, perceived intelligence, and perceived safety
  • Bartneck, C., Kulić, D., Croft, E., & Zoghbi, S. (2009). Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International journal of social robotics, 1(1), 71-81.

Running an Online/Crowd-Sourced Human Subjects Study

If you are interested in running an online or crowd-sourced human subjects study, I would recommend using Prolific. The easiest way to set this up is to create a Qualtrics questionnaire and link to that questionnaire on Prolific.

Timing. Before you launch your study, you will want to have a friend or two take your survey to get a sense of approximately how much time it will take participants to complete your survey. This will also determine how much you will pay each of your participants.

Prescreening Participants. I typically use the following criteria:

• Current Country of Residence: United States (I only use this criteria if I think that someone's cultural background will influence their responses)
• Fluent languages: English
• Approval Rate: 90-100

Payment. We will pay our participants at the rate of $8.00 per hour. When you are ready to launch your study, you will email me (Sarah) with your Prolific login credentials (please ensure that your password is something you do not mind sharing with me), and I will go into your account and add the appropriate amount of money to your account. Then you can launch your study.

Piloting. It is very important to pilot your study before going ahead and running it. We recommend that you have one or two friends go through your study to ensure that everything makes sense and to catch any typos. Then, ask one of the teaching team members to go through your study, we would love to give you feedback on it. Piloting is essential to work out any kinks and ensure that your study is set up to adequately answer your research questions.

Staging Study Launch. We HIGHLY RECOMMEND that you start out by running 20 or so participants in your study first, so that you can ensure that your study is working properly, before running the rest of your planned participants. To ensure that your study is working, please go through the data you collected in Qualtrics to ensure that participants make it through your survey as you intend and answer the questions as you intend them to.

Running an In-Person Human Subjects Study

If you want to run an in-person human subjects study, here's an example timeline of what your project work might look like:

• Weeks 1-3: Form project team, finalize study design (main research questions, experimental conditions)
• Week 4: Develop study protocol, questionnaires, consent forms, etc. and submit your IRB application
• Week 5: Program the robot and go back-and-forth with the IRB if they request any changes to your IRB application
• Week 6: Pilot the study
• Weeks 7-8: Run human subjects
• Week 9: Conduct data analysis and put together final presentation
• Week 10: Write final report

Please note that the quarter is extremely short, you will have to be on top of it from day 1.

Recruitment. You will be responsible for recruiting your own participants. You can post on social media, send emails, post on UChicago organization sites, etc. Just be sure that you include all forms of recruitment in your IRB application that you plan to use.

Payment. For in-person human participants, we will pay them at the rate of $12.00 per hour. We will pay them in the form of Amazon gift cards. To facilitate this payment, please email me (Sarah) daily when you are running your study with a list of email addresses that correspond with your study participants from that day. I will ensure that they are sent their Amazon gift card for participating in your study.

Piloting Your Study. It is SO important before you start running real human subjects that you practice, practice, practice executing your human subjects study. We recommend:

1. Running the study several times yourselves from top to bottom, AND
2. Running the study one one or two friends (including having them fill out the questionnaires) and asking them if anything was confusing or if they noticed any typos in your questionnaire, AND
3. Running the study on one of the members of the teaching team and getting their feedback.

Piloting is essential to work out any kinks and ensure that your study is set up to adequately answer your research questions.

Statistical Data Analysis

I (Sarah) have created a guide for conducting statistical analysis on human subjects data, complete with a thorough answer to the question "what statistical test should I run?" and examples in R of how to run each statistical test. You can find this guide by navigating to the For Lab Members page on my HRI lab website. Click the "Access Lab Wiki" button. You will need to enter the password to enter (please ask the teaching staff for this password). Then find the wiki entry titled "Conducting Statistical Analyses of HRI Data", which will contain everything you need. If you have any questions about conducting your data analysis, please do not hesitate to reach out to the teaching team.