| Notes |
Jill Watson:
A Virtual Teaching Assistant for Online Education
Ashok K. Goel and Lalith Polepeddi
Design & Intelligence Laboratory,
School of Interactive Computing, Georgia Institute of Technology
goel@cc.gatech.edu, lpolepeddi@gatech.edu
Abstract
MOOCs are rapidly proliferating. However, for many MOOCs, the effectiveness of learning is
questionable and student retention is low. One recommendation for improving the learning and
the retention is to enhance the interaction between the teacher and the students. However, the
number of teachers required to provide learning assistance to all students enrolled in all
MOOCs is prohibitively high. One strategy for improving interactivity in MOOCs is to use virtual
teaching assistants to augment and amplify interaction with human teachers. We describe the
use of a virtual teaching assistant called Jill Watson (JW) for the Georgia Tech OMSCS 7637
class on Knowledge-Based Artificial Intelligence. JW has been operating on the online
discussion forums of different offerings of the KBAI class since Spring 2016. By now some 750
students have interacted with different versions of JW. In the latest, Spring 2017 offering of the
KBAI class, JW autonomously responded to student introductions, posted weekly
announcements, and answered routine, frequently asked questions. In this article, we describe
the motivations, background, and evolution of the virtual question-answering teaching assistant.
1. Motivations: Learning Assistance in Online Education
Massively Open Online Courses (MOOCs) are rapidly proliferating. According to Class Central1
,
in 2016 more than fifty eight million (>58,000,000) students across the world together registered
for more than six thousand and eight hundred (>6,800) MOOCs offered by more than seven
hundred (>700) institutions. Further, these numbers continue to grow rapidly. Today MOOCs
cover almost all disciplines and education levels, and the students cut across most
demographics groups such as gender, age, class, race, religion, nationality, etc.
However, the effectiveness of learning in many MOOCs is questionable, and the student
retention ratio typically is less than 50% and often less than 10% (Yaun & Powell 2013).
Although there are several reasons for the low student retention, a primary reason is the lack of
interactivity in MOOCs (Daniel 2012). Thus, one of the principle recommendations for improving
the effectiveness of learning in MOOCs, and thereby also improving student retention, is to
enhance the interaction between the teacher and the students (Hollands & Tirthali 2014).
1 https://www.class-central.com/report/mooc-stats-2016/
This is a DRAFT. Please do NOT distribute.
As an example, consider Georgia Tech’s recently launched online section of CS 1301:
Introduction to Computing2 based on the Python programming language. This online section is
in addition to traditional, residential sections of the Introduction to Computing class. The online
class itself has two sections. In Spring 2017, the accredited section is available only to forty five
selected Georgia Tech students who have access to three teaching assistants (TA) in addition
to course materials provided by the instructor. The three TAs provide several kinds of support to
the online students, such as answering questions, tutoring on the course materials, evaluating
student progress, etc. The open and non-credited section of the online Introduction to
Computing class – the MOOC – currently has more than forty thousand registered students.
The students in the MOOC have access to all the same course materials as the students in the
other online section. However, the forty thousand MOOC students do not have access to any
TA (or the instructor, except indirectly through the standard course materials). Given that
computer programming is a technical skill that many students find difficult to master on their
own, it is unclear what percentage students in the MOOC section will successfully complete the
course. It seems safe to say the percentage of students who successfully complete the MOOC
section without any teaching assistance will be significantly lower than the students in the online
section with teaching assistants.
Of course most humans are capable of learning some knowledge and some skills by
themselves. However, reliable estimates of autodidacts with the capacity to learn advanced
knowledge and complex skills are not readily available. For the purposes of the present
discussion, let us posit that a vast majority of learners can benefit from learning assistance:
perhaps more than 90% of the fifty eight million students taking a MOOC worldwide may need
or want some learning assistance and perhaps as many as 99% may significantly benefit from
learning assistance. If we assume just one teaching assistant (TA) for fifty students for a typical
MOOC, then we need at least one million TAs for supporting the fifty eight million students
registered for a MOOC! It is highly doubtful that anyone can organize or afford such a large
army of human TAs. The Georgia Tech CS 1301 MOOC itself will need about eight hundred
TAs to support the forty thousand students, more than the number of TAs in all other Georgia
Tech classes in computing combined. This raises a profound problem: how can we provide
meaningful learning assistance to the tens of millions of learners taking MOOCs?
In response to this question, MOOC teachers, researchers, and service providers are building
on several technologies for automated or interactive learning assistance such as E-Learning
(e.g., Clark & Mayer 2003), interactive videos (e.g., Kay 2011; Koumi 2006), intelligent books
(Chaudhri et al. 2013), intelligent tutoring systems (e.g., Azevedo & Aleven 2013; Polson &
Richardson 2013; VanLehn 2011), peer-to-peer review (e.g., Faltchikov & Goldfinch 2000;
Kulkarni, Berstein & Klemmer 2015), and autograding. Of course many of these technologies
were developed prior to the start of the modern MOOC movement with the Stanford University’s
MOOC on artificial intelligence in 2011 (Leckart 2012; Raith 2011). Nevertheless, MOOCs too
are extensively developing and deploying these technologies to assist the online education.
2 http://www.cc.gatech.edu/academics/degree-programs/bachelors/online-cs1301
This is a DRAFT. Please do NOT distribute.
One strategy for improving interactivity in MOOCs is to use virtual teaching assistants to
augment and amplify interaction with human teachers. In this article, we describe a virtual
teaching assistant called Jill Watson for the Georgia Tech OMSCS 7637 class on KnowledgeBased Artificial Intelligence. Jill Watson (JW) has been operating on the online discussion
forums of different offerings of the KBAI class since Spring 2016. By now some 750 students
and some 25 (human) TAs have interacted with different versions of JW. In the latest, Spring
2017 offering of the KBAI class, JW autonomously responded to student introductions, posted
weekly announcements, and answered routine, frequently asked questions. Thus, JW is a
partially automated, partially interactive technology for providing online assistance for learning at
scale. In this first scientific article on JW, we describe the motivation, background and evolution
of the virtual question-answering teaching assistant, focusing on what JW does rather than how
she does it.
2. Background: An Online Course on Artificial Intelligence
In January 2014, Georgia Tech launched its Online Masters of Science in Computer Science3
program (OMSCS for short). OMSCS is a fully accredited Georgia Tech graduate degree
offered to highly selected students from across the world. The online courses are developed by
Georgia Tech faculty in cooperation with Udacity staff, offered through the Udacity platform4
,
and supported by a grant from AT&T, The goal of the OMSCS program is to offer the same
courses and programs online that are offered through the on-campus Masters program while
maintaining equivalent depth and rigor (Joyner, Goel & Isbell 2016). In Spring 2017, the
OMSCS program currently has enrolled an order of magnitude more students (approximately
4500) than the equivalent residential program (approximately 350) and costs almost an order of
magnitude (approximately $7000) less than the residential program (approximately $30,000)
(Carey 2016; Goodman, Melkers & Pallais 2016). By now a few hundred students have
successfully completed the OMSCS program, and the diploma awarded to them does not
mention word “online” anywhere anyhow.
As part of the OMSCS program, in 2014, we developed a new online course called CS7637:
Knowledge-Based Artificial Intelligence: Cognitive Systems5 (KBAI for short). The first author of
this article (Goel) had been teaching an earlier KBAI course on Georgia Tech campus for more
than a decade. While the online KBAI course builds on the contents of the earlier on-campus
KBAI course, we rethought the course for the new medium and developed many of the course
materials from scratch (Goel & Joyner 2016). The second author (Polepeddi) took the online
KBAI course in Summer 2015 and was a TA for the course in Spring 2016.
The online semester-long KBAI course consists of 26 video lessons developed from scratch that
help teach the course material (Ou et al. 2016), a digital forum6 where students ask questions
and participate in discussions as illustrated in Figure 1, a learning management system through
3 http://www.omscs.gatech.edu/
4 https://www.udacity.com/courses/georgia-tech-masters-in-cs
5 https://www.omscs.gatech.edu/cs-7637-knowledge-based-artificial-intelligence-cognitive-systems
6 Sankar, P. (2013). Piazza: Our Story. Retrieved from https://piazza.com/about/story
This is a DRAFT. Please do NOT distribute.
which students submit assignments and receive grades7
, a proprietary peer feedback tool
developed at Georgia Tech where students read and submit feedback on each other's
assignments, and a proprietary autograder tool developed by Udacity that helps grade the
source code of programming projects. The course is administered by the instructor (typically
Goel), who is assisted by a small team of TAs. The TAs typically answer questions and facilitate
discussions on the digital forum, and grade assignments, projects, and examinations.
Figure 1. While the video lessons in the OMSCS KBAI course are like a textbook, the
class forum is like a virtual classroom where students ask questions, discuss ideas, and
give feedback. Here, a student asks a question about whether there is a word limit on an
assignment.
Since Fall 2014, we have offered the OMSCS KBAI course each fall, summer and spring term.
Initial enrollment in the class has ranged from about 200 to about 400 students each term so
that by now about 2000 online students have enrolled in the course. For the most part, student
surveys of the online KBAI course have been very positive (Goel & Joyner 2016; Ou et al.
2016). In addition, in the fall terms of 2014, 2015 and 2016, we have offered the same KBAI
course to residential students at both graduate and undergraduate levels. The performance of
the online students on the same set of assessments using blind grading has been comparable
to that of the residential students (Goel & Joyner 2016, 2017). The retention ratio in the online
section has been 75-80%, only slightly lower than the 80-85% in the residential sections.
The OMSCS KBAI course has provided us with a research laboratory for conducting
experiments in pedagogy for online education. For example, we have experimented with
7 The Sakai Project. (2014). Sakai 10. Retrieved from https://sakaiproject.org/sakai-10
This is a DRAFT. Please do NOT distribute.
programming projects based on real AI research to promote authentic scientific practices (Goel
et al. 2013) as well as use of peers as reviewers and TAs as meta-reviewers (Joyner et al.
2016). We also developed and deployed about a hundred nanotutors for teaching domain
concepts and methods (Goel & Joyner 2017). A nanotutor is small, focused AI agent that
models students’ reasoning on a particular problem engaging a domain concept or method to be
learned. Given a student’s answer to the problem, a nanotutor first classifies the answer as
correct or incorrect, and then provides an explanation on why the answer is (in)correct.
3. A Challenge in Scaling Online Education: Responding to Student Questions
Teaching the OMSCS KBAI class in the Fall 2014 and Spring 2015 terms revealed a new
challenge for the teaching staff: the discussion forum for the online class was very active and
thus took a large amount of staff time to monitor and respond. Table 1 provides the data from
the discussion forums for the online and residential sections from Fall 2016. As Table 1
indicates, the discussion forum for the online section had >12,000 contributions compared to
<2,000 for the residential class. One obvious reason for this six-fold increase is that online class
had three times as many students as the residential class. Another, perhaps less obvious
reason is that discussion forum acts as the virtual classroom for the online class (Joyner, Goel &
Isbell 2016). It on the discussion forum that the online students ask questions and get (and give)
answers, discuss the course materials, learn from one another, and construct new knowledge.
Table 1: The level of participation of online students in the OMSCS KBAI class on the
digital forum is much higher than that of residential students. Table 1 compares four
participation metrics between online students and on campus students during the Fall
2016 offering of KBAI class.
Residential
(Fall 2016)
Online
(Fall 2016)
Number of students 117 356 +3x
Total threads 455 1201 +2x
Total contributions 1,838 12,190 +6x
While the abundant participation on the discussion forum of the online class likely is an
indication of student motivation, engagement and learning, and thus is very welcome, the higher
levels of participation create a challenge for the teaching staff in providing timely, individualized,
and high quality feedback. On one hand, the quality and timeliness of TAs’ responses to
students’ questions and discussions is an important element of providing learning assistance
and thus plays a part in the success of student learning and performance. On the other, given
the high rate of student participation on the discussion forum, the TAs may not have time to
respond to each message with a high quality answer in a timely manner.
This is a DRAFT. Please do NOT distribute.
4. A Potential Answer: Virtual Teaching Assistants
In reading through the students’ questions on the online discussion forums of the OMSCS KBAI
class in Fall 2014 and Spring 2015, we recognized (as many-a-teacher has done in past), that
students often ask the same questions from one term to another, and sometimes even from one
week to another within a term. For example, questions about length and formatting of the
assignments, allowed software libraries for the class projects, and class policies on sharing and
collaborating have been asked in different ways every semester since January 2014. Perhaps
more importantly that from the online discussion forums of the Fall 2014 and Spring 2015
OMSCS KBAI classes, we had access to a dataset of questions students had generated and
the answers TAs had given.
Thus, in summer 2015, we wondered if we could construct a virtual teaching assistant that could
use the available dataset to automatically answer routine, frequently asked questions on the
online discussion forum? We posited that if we could create a virtual TA that could answer even
a small subset of students' questions, then it would free the human TAs to give more timely,
more individualized, and higher quality feedback to other questions and the human TAs may
have more time to engage in deeper discussions with the students.
Our thinking about the virtual teaching assistant was also inspired by IBM’s Watson system
(Ferruci 2012; Ferruci et al. 2009). Independently of the OMSCS KBAI class, in Fall 2014, IBM
had given us access to its Watson Engagement Manager8 for potential use in support of
teaching and learning. We successfully used the Watson Engagement Manager for teaching
and learning about computational creativity in a residential class in Spring 2015 (Goel et al.
2016). Building on this educational experience with the Watson Engagement Manager, in Fall
2015, IBM gave us access to its newer Bluemix9 toolkit in the cloud. Thus, we were familiar with
both the paradigm of question answering and some of the Watson tools.
5. Jill Watson and Family
Starting in Fall 2015, we have developed three generation of virtual teaching assistants. We
have also deployed these virtual teaching assistants in the discussion forums of the online KBAI
classes in Spring 2016, Fall 2016, and Spring 2017, as well as the residential class in Fall 2016.
All actual experiments with the virtual teaching assistants have been in compliance with an IRB
protocol to safeguard students’ rights and to follow professional and ethical norms and
standards.
We call our family of virtual teaching assistants Jill Watson because we developed the first
virtual teaching assistant using IBM Watson APIs. However, the names and tasks of specific
virtual teaching assistants have evolved from generation to generation as described below.
More importantly, starting with the second generation, we have used our own proprietary
software and open-source libraries available in the public domain instead of IBM Watson APIs
8 IBM Watson Engagement Manager. Retrieved from http://m.ibm.com/http/www03.ibm.com/innovation/us/watson/watson_for_engagement.shtml
9 IBM Bluemix. Retrieved from https://www.ibm.com/cloud-computing/bluemix/
This is a DRAFT. Please do NOT distribute.
(or any other external tool). We made this shift to cover a larger set of questions as well as a
larger set of tasks.
5.1 Jill Watson 1.0
5.1.1. Design
In January 2016, we deployed the first version of Jill Watson, Jill Watson 1.0 (or JW1 for short)
to the Spring 2016 offering of the OMSCS KBAI class. Although we included JW1 in the listing
of the teaching staff, initially we did not inform the online students that JW1 was an AI agent. As
noted above, we built JW1 using IBM Watson APIs. JW1 is essentially a memory of questionanswer pairs from previous semesters organized into categories of questions. Given a new
question, JW1 classifies the question into a category, retrieves an associated answer, and
returns the answer if the classification confidence value is >97%.
Initially, we deployed JW1 on the discussion forum with a human-in-the-loop; if she was able to
answer a newly asked question, then we would manually check that her answer was correct
before letting her post that answer to the class forum in reply to the question. In March 2016, we
removed the human-in-the-loop and let JW1 post answers autonomously.
Every 15 minutes between 9am and 11pm, JW1 checked the discussion forum for newly asked
student questions. We chose this time interval to mimic the working hours for most human TAs
as well as to monitor to JW1’s performance throughout the day. If there was a question that
JW1 could answer and that another human TA had not already answered, she would post an
answer.
5.1.2. Performance
Figures 2, 3, 4 and 5 illustrate some of JW1 interactions with the online students on the
discussion forum of the OMSCS KBAI class in Spring 2015. (Note that we have blackened
some portions of the exchanges to maintain student confidentiality.)
Figure 2. In this question about a class project with a coding component, the student
asks whether there is a limit to their program’s run time. Jill Watson 1.0 correctly
answers that there is a soft 15 minute run time limit.
This is a DRAFT. Please do NOT distribute.
Figure 3. In this question about a class assignment involving a writing component, the
student asks whether there is a maximum word limit. Jill Watson 1.0 correctly answers
that there is no strict word limit. Another student then has a follow up question asking
for elaboration, which a human TA handles. After this exchange, one student in the class
speculates whether Jill Watson is human.
Figure 4. In this question about submitting a class project, a student asks about resubmitting with the correct file. Jill answers the question as if the student was asking
about submitting the class project for the first time. However, the student accepts the
answer and asks for further instructions.
This is a DRAFT. Please do NOT distribute.
Figure 5. In this question about a class project, a student asks about his program’s
performance on a problem set. Jill incorrectly answers the question, and the student
asks whether the answer was meant for another question.
We found that while JW1 answered only a small percentage of questions, the answers she gave
were almost always correct or almost correct. We wanted to both increase the range of
questions covered by JW as well as the task she addresses. The latter goal led us to develop
the next generation of Jill Watson.
5.2. Jill Watson 2.0
5.2.1. Design
In the first week of the KBAI class, we ask students to introduce themselves on the discussion
forum by posting a message with their name, their location, why they're taking KBAI this
semester, other OMS classes they've taken, activities outside of school, and one interesting fact
about them. Human TAs then reply to each student, welcoming him/her to the class. However, it
is time consuming to respond individually to 200-400 students within one week. Thus, we built
the second generation of Jill Watson, Jill Watson 2.0 (or JW2) to autonomously respond to
student introductions.
Unlike JW1 that was built using IBM Watson APIs, we developed the software for JW2 in our
laboratory from scratch, using only open-source external libraries available in the public domain.
Further, unlike JW1 that used only an episodic memory of question-answer pairs from previous
semesters, JW2 used semantic processing based on conceptual representations. Given a
student’s introduction, JW2 first mapped the introduction into relevant concepts and used the
concepts as an index to retrieve an appropriate precompiled response.
In August 2016, we deployed two separate virtual TAs to the discussion forums of the Fall 2016
offerings of the KBAI class that included both an online section and a residential section. We
redeployed JW1 to answer routine, frequently answered questions as a TA named "Ian Braun"
and we deployed JW2 to respond to student introductions as a TA named "Stacy Sisko."
This is a DRAFT. Please do NOT distribute.
Just like Ian Braun, every 15 minutes between 9am and 11pm, Stacy checked for newly posted
student introductions. Just as with routine questions, if there was a student introduction that
Stacy could reply to and that another TA hadn't already replied to, she would autonomously post
a welcome message.
Once again while we listed both Ian Braun and Stacy Sisko among the teaching staff, we did not
inform the students that were AI agents. To prevent students from identifying the human TAs
among the teaching staff through internet searches, all human TAs operated on the discussion
forum under pseuodnyms.
5.2.2. Performance
Stacy Sisko autonomously replied to >40% of student introductions. Figures 6, 7 and 8 illustrate
Stacy’s responses to student introductions.
Figure 6. In this introduction, the student expresses interest in learning more about
artificial intelligence. Stacy responds that she also shares a similar interest in AI.
Figure 7. In this introduction, the student shares that he took another OMS course called
Introduction to Operating Systems. Stacy responds that she took the course as well and
asks the student whether he been able to apply what he had learned in the class, to
which the student replies.
This is a DRAFT. Please do NOT distribute.
Figure 8. In this introduction, a student shares that he took another OMSCS course called
Software Development Process. Stacy responds that she took the course as well, but
now asks the student what he thought about the professor of the class.
Figure 9. In this question about a class project with a coding component, the student
asks whether they can use the Python library SciPy. Ian correctly replies with the course
policies on using external libraries.
Figure 10. In this question about a class assignment, the student asks whether there is a
preferred way to name their submission. Ian correctly replies that there isn’t a specific
naming convention, and the same student thanks Ian for the answer.
This is a DRAFT. Please do NOT distribute.
Figure 11. In this question about a class project with a coding component, the student
asks whether they can upload additional files that their program needs to run. Ian
correctly replies that additional files are allowed.
Figure 12. In this post about a class project with a coding component, the student shares
their current progress and asks for feedback. Ian incorrectly answers the question as if
the student was asking about how to get started with the provided code.
Figures 9, 10, 11 and 12 illustrate Ian Braun’s interactions with students on the online
discussion forum. We found that although Ian Braun was a redeployment of JW1, he performed
better in the Fall 2016 KBAI class than JW1 did in the Spring 2016 class both in the coverage of
routine, frequently asked questions and the proportion of correct answers. This improvement
This is a DRAFT. Please do NOT distribute.
likely was because by Fall 2016, we had a larger dataset of question-answer pairs because by
then the class has been offered a few more times.
5.3. Jill Watson 3.0
5.3.1. Design
Given the success of Stacy Sisko in using semantic processing to reply to student introductions,
we created a third generation of Jill Watson, Jill Watson 3.0 (or JW3 for short) that uses
semantic processing for answering questions. Unlike JW1, JW3 does not use IBM Watson APIs.
Instead JW3 relies solely on an episodic memory. Given a student’s question, JW3 first maps
the question into relevant concepts and uses the concepts as an index to retrieve an associated
answer from the episodic memory of questions organized into categories.
In January 2017, we deployed two separate virtual TAs to the Spring 2017 offering of the
OMSCS KBAI class. We redeployed version JW2 (or Stacy Sisko) to respond to student
introductions as a new virtual TA named "Liz Duncan" and we deployed version JW3 to answer
routine questions as a virtual TA named "Cassidy Kimball." Once again while we listed both and
Liz Duncan and Cassidy Kimball among the teaching staff, we did not inform the students that
were AI agents. To prevent students from identifying the human TAs among the teaching staff
through internet searches, all human TAs operated on the discussion forum under pseuodnyms.
We also increased the time interval during which Cassidy checked for newly asked questions to
6am and 1159pm based on our observations of the activity on the discussion forum.
5.3.2. Performance
Liz Duncan replied to 60% of all student introductions, a performance superior to that of Stacy
Sisko in the earlier generation. Figures 13, 14, 15 and 16 illustrate Liz’s interactions with the
online students.
This is a DRAFT. Please do NOT distribute.
Figure 13. In this introduction, a student shares that they took another OMS course
called Computer Vision. Liz responds by recommending that the student share their
insights throughout the course. After Liz’s initial response, other students respond to
other parts of the student’s introduction.
Figure 14. In this introduction, the student shares that they just started the OMS
program. Liz Duncan responds by commenting that KBAI is a good first class to enter
the OMS program.
Figure 15. In this introduction, the student shares that they live in Atlanta. Liz responds
by inviting the student to visit Georgia Tech in person if they are in the area.
Figure 16. In this introduction, the student shares that they are currently taking another
OMS course called Computer Architecture in addition to KBAI. Liz incorrectly processes
that the student took Computer Architecture in a previous semester, and responds and
asks what they thought of the class, prompting the student to reiterate that they are
currently taking the class.
This is a DRAFT. Please do NOT distribute.
We found that Cassidy Kimball performed much better than JW1 and Ian Braun. For example,
of the questions that students asked about KBAI's three class assignments, Cassidy
autonomously answered 34%, and of all the answers Cassidy gave, 91% were correct. Figures
17 through 25 illustrate Cassidy’s interactions on the online discussion forum.
Figure 17. In this question about a class assignment involving a written component, the
student asks whether there is a preferred format for citations. Cassidy correctly
responds to part of the student’s question that the APA format is recommended. A
human TA responds to the other part of the student’s question.
Figure 18. In this question about a class assignment involving a written component, the
student asks about the level of detail they should include in their paper. Cassidy
correctly replies that assignments can be at a high level of detail and don’t need to get
into low-level implementation.
Figure 19. In this question about a class project involving a coding component, the
student asks whether they can use the Python library SciPy in their code. Cassidy
correctly replies that external libraries are not allowed. Another student asks a follow up
question about the reason why this decision was made, which another human TA
answers.
This is a DRAFT. Please do NOT distribute.
Figure 20. In this question about a class project involving a coding component, the
student asks for more feedback after submitting their assignment to the automated
grading system. Cassidy incorrectly answers this question as if the student was asking
about which problem sets are graded. The student asks someone else to help, to which a
human TA responds.
Figure 21. In this question about a class assignment involving a writing component, the
student asks about whether there is a preferred format to name files. The student also
inserts a sentence asking human TAs not to respond, possibly in an attempt to discover
the identity of the virtual TA. Cassidy correctly responds to this question.
Figure 22. In this question about the class midterm involving a written component, the
student asks about the level of detail they should include in their responses. Cassidy
correctly replies to the question, but the student second-guesses her answer and asks
another human TA for confirmation.
This is a DRAFT. Please do NOT distribute.
Figure 23. In this question about a class assignment, the student asks whether they can
reuse content from a previously submitted assignment. Cassidy could have answered
this question, but did not because the question was asked outside the time interval in
which she checks the class forum for new questions. Since a human TA answered the
question by the time Cassidy checked the class forum again, Cassidy did not answer this
question.
Figure 24. In this question about a class project involving a coding component, the
student asks whether they can discuss ideas with other students. Cassidy could have
answered this question, but did not because another human TA Quentin Washington
answered the question within 15 min. As Cassidy checks the discussion forum every 15
minutes, she did not have a chance to respond. Therefore, the next time she checked the
class forum, since a another TA had already answered the question, she did not answer.
Figure 25. In this question about the class midterm, the student asks about whether they
can submit the midterm more than once. While Cassidy could have answered this
question, we deliberately prevented her from answering questions about class
submissions - those questions are among the most important that students ask, and for
now we feel more comfortable that a human handles them.
8. Student Reaction
In the KBAI classes in Spring 2016, Fall 2016, and Spring 2017, we shared the true identities of
the virtual as AI agents towards the end of the term. Student reactions to our use of virtual
teaching assistants in online discussion forums have been uniformly and overwhelmingly
positive. Figure 26 illustrates a small sample of student reactions from the KBAI class in Spring
2016 after the students learned about the true identity of Jill Watson towards the end of April
2016.
This is a DRAFT. Please do NOT distribute.
Figure 26. Students react to our class post in at the end of KBAI Spring 2016 class
announcing the true identity of Jill Watson.
9. Discussion
There are several questions about the virtual teaching assistants that we have not fully
answered in this article. The first question is how does Jill Watson work? As we briefly indicated
above, Jill Watson 1.0 uses an episodic memory of questions and their answers from previous
episodes. We developed JW1 using the IBM Bluemix toolsuite. In the second generation of Jill
Watson, Ian Braun was a redeployment of JW1 for answering questions. However, Stacy Sisko
used semantic information processing technology developed in our laboratory to reply to student
introductions. In the third generation of Jill Watson, Cassidy Kimball too uses semantic
information processing technology developed in our laboratory for answering questions as does
Liz Duncan for replying to student answers.
Second, is the Jill Watson technology transferrable to other classes with different student
demographics and using different educational infrastructures? To answer this question, we are
presently building a new version of Jill Watson for a new Georgia Tech CS 1301 Introduction to
Computing MOOC that presently has forty thousand students but no TA support whatsoever.
Third, is the Jill Watson technology effective in lowering the demands on the teaching staff?
While it is too early to determine the answer to this question for the task of question answering,
anecdotally there is some evidence to suggest that Jill Watson did reduce the load on the
teaching staff for responding to student introductions and for posting messages to the class.
Fourth, is the Jill Watson technology effective in enhancing student performance and improving
student retention. We are presently conducting studies and collecting data to answer this
question about student engagement, learning and performance; it is too early to have insights
into the issue of student retention.
Fifth, what ethical issues arise in using Jill Watson as an educational technology in an online
classroom? As we mentioned above, we obtained IRB approval in advance of the Jill Watson
experiments. Nevertheless, these experiments have raised several additional ethical issues. For
This is a DRAFT. Please do NOT distribute.
example, when it is appropriate to use AI agents without telling human subjects about them?
Does the use of a feminine name for an AI agent implicitly promote gender stereotypes? Might
the use of AI agents as virtual teaching assistants eventually result in reduced employment
opportunities for human teachers? These are serious questions that require investigation.
10. Conclusions
We may view the Jill Watson experiments from several perspectives. First, we may view Jill
Watson as an educational technology for supporting learning at scale. In fact, this was our
primary initial motivation for developing Jill Watson and this is also how we motivated the
discussion in this chapter. As indicated above, Jill Watson uses AI technology for supporting
learning at scale by automatically answering a variety of routine, frequently asked questions,
and automatically replying to student introductions.
Second, we may view Jill Watson as an experiment in developing AI agents so that for highly
focused technical domains, highly selected subject demographics, and highly targeted context
of human-computer interaction, it is difficult for humans to distinguish between the responses of
AI and human experts. We found that in order to improve coverage, the design of Jill Watson
gradually moved from using an episodic memory of previous question-answer pairs to using
semantic processing based on conceptual representations.
Third, we may view Jill Watson as an experiment in human-AI collaboration. The KBAI class has
became a microsociety in which humans and AI agents collaborate extensively and intensively,
living and working together for long durations of time.
Acknowledgements
We thank IBM for its generous support of this work both by providing us access to IBM Bluemix
and through multiple IBM Faculty Awards. We are grateful to Parul Awasthy for her contributions
to the development of Jill Watson 1.0 in Spring 2016. We thank members of the Design &
Intelligence Laboratory for many discussions on the Jill Watson project. We also thank the
students and (human) teaching assistants of the KBAI course in Spring 2016, Fall 2016, and
Spring 2017. This research has been supported by multiple Georgia Tech seed grants for
research and education.
References
Azevedo, R., & Aleven, V. (2013) International Handbook of Metacognition and Learning
Technologies. Springer.
Carey, K. (2016, September 28). An Online Education Breakthrough? A Master's Degree for a
Mere $7,000. The New York Times. Retrieved from
http://www.nytimes.com/2016/09/29/upshot/an-online-education-breakthrough-a-mastersdegree-for-a-mere-7000.html
Chaudhri, V., Cheng, B., Overholtzer, A., Roschelle, J., Spaulding, A., Clark, P., Greaves, M., &
This is a DRAFT. Please do NOT distribute.
Gunning, D. (2013a). Inquire Biology: A Textbook that Answers Questions. AI Magazine, 34.
Clark, R., & Mayer, R. (2003). E-Learning and the Science of Instruction. San Francisco, CA.
Pfeiffer.
Daniel, J. (2012), Making Sense of MOOCs: Musings in a Maze of Myth, Paradox and
Possibility. Journal of Interactive Media in Education.
Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A metaanalysis comparing peer and teacher marks. Review of Educational Research, 70(3), 287-
322.
Ferrucci D, Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A., Lally, A., Murdock,
J.W., Nyberg, E., Prager, J., Schlaefer, N., & Welty, C. (2010) Building Watson: An overview
of the DeepQA project. AI Magazine, 31:59–79.
Ferrucci, D. (2012) Introduction to ”This is Watson”. IBM Journal of Research and Development
56(3):1-15.
Goel, A., Anderson, T., Belknap, J., Creeden, B., Hancock, W., Kumble, M., Salunke, S.,
Sheneman, B., Shetty, A., & Wiltgen, B. (2016) Using Watson for Constructing Cognitive
Assistants. In Proceedings Fourth Annual Conference on Cognitive Systems, Chicago, May
2016.
Goel, A. & Joyner, D. A. (2016) An Experiment in Teaching Cognitive Systems Online. In
Haynes, D. (Ed.) International Journal for Scholarship of Technology-Enhanced Learning
1(1).
Goel, A. & Joyner, D. (2017) Using AI to Teach AI. AI Magazine, Summer 2017.
Goel, A., Kunda, M., Joyner, D., & Vattam, S. (2013) Learning about Representational Modality:
Design and Programming Projects for Knowledge-Based AI. In Procs. Fourth Symposium on
Educational Advances in Artificial Intelligence, Bellevue, Washington, July 2013.
Goodman, J., Melkers, J., & Pallais, A. (2016). Does Online Delivery Increase Access to
Education? Harvard University Kennedy School Faculty Research Working Paper Series
RWP16-035.
Hollands, F., & Tirthali, D. (2014). MOOCs: Expectations and reality: Full report. Center for
Benefit-Cost Studies of Education, Teachers College, Columbia University, NY. Retrieved
August 1, 2015, from http://cbcse.org/wordpress/wpcontent/uploads/2014/05/MOOCs_Expectations_and_Reality.pdf
Joyner, D., Ashby, W., Irish, L., Lam, Y., Langston, J., Lupiani, I., ... & Goel, A. (2016, April).
Graders as Meta-Reviewers: Simultaneously Scaling and Improving Expert Evaluation for
Large Online Classrooms. In Proceedings of the Third (2016) ACM Conference on Learning
@ Scale (pp. 399-408). ACM.
Joyner, D., Goel, A., & Isbell, C. (2016). The Unexpected Pedagogical Benefits of Making
Higher Education Accessible. In Proceedings of the Third Annual ACM Conference on
Learning at Scale. Edinburgh, Scotland.
Kay, R. (2012). Exploring the use of video podcasts in education: A comprehensive review of
the literature. Computers in Human Behavior, 28(3), 820-831.
Koumi, J. (2006). Designing video and multimedia for open and flexible learning. Oxford, UK:
Routledge Falmer.
Kulkarni, C., Bernstein, M., & Klemmer, S. (2015). PeerStudio: Rapid Peer Feedback
Emphasizes Revision and Improves Performance. In Proceedings from The Second ACM
This is a DRAFT. Please do NOT distribute.
Conference on Learning @ Scale. ACM.
Leckart, S. (2012) The Stanford education experiment could change higher learning forever.
Wired, March 2012.
Li, L., Liu, X., & Steckelberg, A. (2010). Assessor or Assessee: How student learning improves
by giving and receiving peer feedback. British Journal of Educational Technology, 41(3), 525-
536.
Ou, C., Goel, A., Joyner, D., & Haynes, D. (2016). Designing Videos with Pedagogical
Strategies: Online Students' Perceptions of Their Effectiveness. In Procs. Learning @Scale
2016, pp.141-144
Polson, M., & Richardson, J. (2013). Foundations of intelligent tutoring systems. Psychology
Press.
Raith, A. (2011) Stanford for Everyone: More Than 120,000 Enroll in Free Classes. MindShift.
Retrieved from http://ww2.kqed.org/mindshift/2011/08/23/stanford-for-everyone-more-than120000-enroll-in-free-classes/
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems,
and other tutoring systems. Educational Psychologist, 46(4), 197-221.
Van Zundert, M., Sluijsmans, D., & Van Merriënboer, J. (2010). Effective peer assessment
processes: Research findings and future directions. Learning and Instruction, 20(4), 270-279.
Yuan, L., & Powell, S. (2013). MOOCs and open education: Implications for higher education
[White Paper]. Retrieved from Google Scholar. |