Clicker Implementation and
Assessment
Department of Communication
Michele H. Jackson April R. Trees
jackson@colorado.edu april.trees@colorado.edu
Submitted to
Information and Technology Services
and to the Faculty
Teaching Excellence Program
December 19, 2003
Report Of 1st Semester Clicker Use Within Department Of Communication p.2
Background
Technology selection and implementation
Training
Infrastructure and Support
Nature of clicker “questions”
Classroom dynamics
Method and Procedure
Results
Follow-up Analyses
Conclusions and Recommendations p.21
For New Instructors
For Instructors Already Using Clickers
For Administrators and Technology Support
Summary p.24
Appendices:
A. Survey
B. Faculty Survey Responses
C. Student Open-ended Comments (not yet available Dec 19)
An increasing number of instructors in several departments are experimenting with the H-ITT student response system informally known as “clickers.” At this point, little systematic information exists regarding the socio-technical infrastructure needed to support clicker use across campus, or the effect of clickers on the learning environment. This report represents a first attempt to provide that information. If sufficient interest and funding are available, we are interested in expanding beyond this exploratory study, by conducting a follow up study and analysis in Spring-Summer 2004.
Our main finding is that while clicker technology is simple, its effects on the classroom are complex and can be challenging to manage. The effectiveness of a “clicker classroom” depends on a combination of technical, social, and pedagogical factors. This study is a first attempt to unpack the nature of this combination.
A central implication of this study is the importance for providing support and resources to instructors to learn how to use clickers in ways that can accomplish their learning goals. One approach is Concept Tests, but additional approaches need to be developed for instructors who either do not wish to use Concept Tests, or whose material is not well suited to that approach.
Our observations are based on two sources of data. First, we draw on the experiences of two instructors who piloted clickers Fall 2003, April Trees and John Jackson. Communication is the first department outside of the sciences to implement clickers. Our experiences are instructive to other non-science departments interested in adopting clickers, such as the social sciences, business, and humanities.
Second, we administered a survey to over 1,500 students enrolled in 7 “clicker courses.” Though the survey is limited and exploratory, the results question some of the assumptions that typically guide clicker adoption and use. The results suggest actions for improving the potential for clicker success.
This document first reports the use of clickers in the Communication Department during Fall 2003. The majority of the report follows, in which we detail the survey we administered at the end of Fall 2003, presenting the rationale for our approach, the method of the survey, results, and analyses. We end with recommendations for Spring-Summer 2004.
REPORT OF 1ST
SEMESTER CLICKER USE WITHIN DEPARTMENT OF COMMUNICATION
Overall, our experience differed from the informal reports of other uses of clickers on campus. If clickers were to be adopted widely, others may share various aspects of our experience. The lessons learned from our experience can impact positively future distribution and implementation of the technology.
In spring 2003, instructors of the two largest courses offered by the Department of Communication decided to adopt on a trial basis a “student response system.” These courses are COMM1210, Perspectives on Human Communication, and COMM1300, Public Speaking. Each course consists of 50-minute large lecture sessions and small (20-25 student) recitations. In COMM1210, the division is 2 lecture/1 recitation and in COMM1300 the division is 1 lecture/2 recitation.
Each course enrolls over 320 students. Both were scheduled for CHEM140 during Fall 2003, which was not equipped with a “clicker” system. After investigating several options, we decided to adopt the H-ITT system. M. Jackson approached ITS in Summer 2003, suggesting a partnership to install the H-ITT system in CHEM140. At about that same time, it was learned that Chemistry and Chemical Engineering also had interest in outfitting CHEM140. With financial and institutional support from ITS leadership, and financial support from Communication, Chemistry, and Chemical Engineering, CHEM140 was professionally outfitted over the summer and first used as a clicker classroom Fall 2003.
We support the selection of H-ITT as a campus standard. First, the classroom software provided by H-ITT is superior to other student response systems (SRS). In addition, the transmitter hardware is relatively inexpensive and represents only a one-time cost to the student. Some alternative systems require students to lease the equipment on a semester basis (which is an unreasonable cost for students). Others require “hardwiring” transmitters into a single classroom (which poses unreasonable risk of equipment theft and vandalism). The professional installation of the system in CHEM140 meant that the classroom was free of dangling wires and the cable for the instructor’s computer was well placed and easily accessible.
In contrast to several other clicker courses, our instructors used only their laptops in the classroom. We did not use a central computer to run the H-ITT acquisition or analyzer software. This was a completely satisfactory arrangement. After completing an initial learning curve, instructors had no difficulty running the H-ITT software. All student data were captured without problems, and the instructors backed up their data in their normal fashion.
We had one unexpected problem with the technical configuration. H-ITT requires a serial port for input, but J. Jackson had a brand new Dell laptop computer that did not come equipped with a serial port. We purchased an adapter, but it did not arrive for a few weeks. J. Jackson used his old laptop, transferring data to his new computer in his office after class.
The H-ITT software is fairly complex with a less than friendly interface. It is also quite powerful, including a number of different options that can be configured in real-time by the instructor wanting to customize the classroom experience. Further, some of the ways in which H-ITT works are actually counterintuitive, such as the complete independence of the content of clicker questions and the answers collected.
Some basic instructions for H-ITT are available online. Even though our instructors possess an average level of computer literacy, the instructions were difficult to follow and not very useful. Therefore, approximately 3 weeks before the beginning of the semester, M. Jackson provided two half-day training sessions to J. Jackson and Trees. The training focused on how the software worked and how to operate it in the classroom, but included no pedagogical component. This is because each instructor had an idea of how they would incorporate clicker questions into their preferred pedagogical styles and learning goals.
Because of a lack of time and resources, there was not enough training in a number of issues that turned out to have important consequences for the courses: how to handle student registration, how to handle data collected during classes, how to merge clicker data with other course data. Training on how to handle data should occur throughout the semester.
Both instructors faced a learning curve not only in learning the technology, but also in how to incorporate the technology into their pedagogy. Our instructors would have benefited from pedagogical training. Student response systems are particularly well suited to a technique called “concept tests,” pioneered by Eric Mazur. Concept Tests are well suited to the sciences and collections of concept tests exist for classes in physics and chemistry. However, according to our research, none yet exist for the social sciences or humanities. In other words, outside of the sciences (or perhaps engineering) integrating clickers into the classroom is more challenging, because research and practice for standard uses of clickers is not as developed.
The basic principles of Concept Tests reflect common pedagogical techniques: active learning, course engagement, peer instruction, improved student motivation, and real-time feedback on student understanding. Student response is core to achieving these goals. The use of clickers—whether or not through the use of Concept Tests—is strongly associated with the learning goal of changing the dynamic of the large class, away from a “lecture” style to an engaged style reminiscent of a small seminar (or what Waite, Jackson, & Diwan have called the “Conversational Classroom”).
At the beginning of a semester, students register their clickers so their answers can be linked to them. H-ITT has an email utility to handle registration, but it is complex and prone to user error. We built a web-based form for students to enter their name, course, and clicker serial number. We did not have time to debug the form before classes started, and students had several interface problems that could have been avoided with better design. As a consequence, student anxiety increased because they worried that their clicker answers wouldn’t be registered (even though this wasn’t true). Whatever registration procedure is used, it must well so that students trust it. Registration is a student’s first interaction with clickers, and it is very important that it work seamlessly.
Our instructors received no technical support in the classroom and a minimal amount of support outside of the classroom. Support for the other media in the classroom (sound, video) was provided by ITS. The Communication Department’s general IT specialist provided ad hoc clicker support. Trees reported experiencing technical problems in about 80% of the class periods, while J. Jackson reported problems in about 40% of the class periods.
The most significant technical problem was overload of the H-ITT receivers caused by too many students answering simultaneously. After a few weeks, the instructors adopted an “alphabetical rolling response” strategy, in which students answered clicker questions in stages. In J. Jackson’s class, this solved the problem. However, in Trees’ class, students in one section of the classroom refused to follow procedure and so repeatedly overloaded the receiver in that part of the room. Although a problem easily solved, receiver overload also raises student anxiety about the clickers.
Until an instructor becomes competent with the technology, technical support must be available in real-time. For our instructors, when a technical problem surfaced, it often could not be resolved immediately. This meant that sometimes the clickers could not be used that day, or they could not be used as planned. This resulted in frustration not only for the instructor, but for the students as well. It was not uncommon for some students, upon hearing that there would be no clicker questions for that day, to get up and leave the classroom.
Any clicker classroom, particularly a large lecture hall, must have all other technology in working condition. Students see clickers as part of a larger “educational technology” package. When one technology doesn’t work, it affects how students respond to the other technologies. Our courses rely heavily on multimedia. Often throughout the semester, our instructors show videos or photos, or visit websites. Both regularly use PowerPoint. But there were often problems with these other technologies. On days that the microphone did not work for Trees, she basically needed to yell to be heard, which works against creating a “small and engaged” classroom environment. When showing video clips from a laptop does not work, time is taken up to solve “technology problems.” Clicker malfunctions contribute to a broader negative attitude about technology in the classroom.
Student anxiety about the technology is a real and significant problem with clickers. For many students, this is the first time they have had their grade depend on a piece of equipment. The more students worry about whether their answers are being captured, the less they will be able to concentrate on learning. Sometime after the semester started, we learned that other classes regularly give feedback to students regarding their personal clicker question data. This seems a good strategy for reducing anxiety.
Also, instructors must be sure of the technology themselves. This could be accomplished by prior training if it included a very strong hands-on practicum. For naïve or inexperienced adopters, the experience is quite unlike the use of any other technology. Training should simulate the environment—and the potential problems—as closely as possible.
Data management in H-ITT is cumbersome. Although H-ITT offers an analyzer program, our instructors did not attempt to use it. During training, we demonstrated the program and they did not find its functionality useful. The procedures for porting data from H-ITT to another program and then making that data meaningful are confusing for instructors who do not have experience with this type of data set. Our IT specialist closely assisted our instructors in extracting and managing the collected H-ITT data. In future semesters, this will be a priority for us. We would like to integrate this with the regular feedback to students, as mentioned above.
Our instructors used clickers in a number of different ways. For example, the instructor would introduce a concept, and then ask a clicker question to gauge the extent to which students understood the concept. Another common use was exam preparation. The instructor would show students questions that might typically appear on an exam. In addition to questions testing students understanding of the material, questions with no right answer were sometimes included to generate discussion or illustrate a concept. An interesting use, but one that was not used as often, was to ask for student input that might then direct what was done next in the class period. For example, asking students which of two example videos they wanted to see.
Although H-ITT supports impromptu questions, this was not used. All questions were prepared beforehand. A limitation of relying on clicker questions is they are limited to a 5-response multiple choice format. Typically, instructors would ask 1-3 questions in a 50-minute period. Both report using them in about 70% of the class periods.
Most questions worked well. Questions that were complex or complexly worded, however, did not work well. In part this was because of how the questions had to be displayed. Questions are shown at the top of a display that shows student answers. Because of the high enrollment, the instructors often removed the question from the display (to allow the student numbers to be larger). By the end of the semester, the instructors instead showed the clicker questions on an overhead so they could be displayed at the same time as the clicker answers. The same effect could be achieved by writing them on the chalkboard.
Our instructors graded clicker questions differently. Trees gave points for clicker use, but did not differentiate between right and wrong answers (indeed, some of the questions she used did not have a right or wrong answer). J. Jackson gave extra credit points. For both instructors, the number of points possible was a very small percentage of the overall course grade (1-4%). Because they were experimenting with the technology, they did not want to make student grades too dependent on clickers. However, as discussed below, having so few clicker points may work against their effectiveness.
The change in classroom dynamics was perhaps the most interesting result from our experience. Some changes were expected, but a number we did not anticipate.
One expected change was that posing clicker questions and waiting for responses would require extra time. However, the receivers in CHEM140 were typically “overloaded” in that too many clickers were transmitting at once. Adopting a “rolling answer” strategy prevented overload, but still required more time to answer a question than would be ideal if there were sufficient number of receivers. Considering 3-4 minutes to pose a question and collect answers, just 3 questions can take over 20% of a class period.
An unexpected change was the disturbance this caused in the classroom. The simple process of answering a question caused classroom “chatter” to increase. Both instructors report that students would then respond audibly when the graph of the results was displayed. And both report that it would take some effort to quiet the course back down. This was a distinct problem in CHEM140, where the background noise from the ventilation system is quite loud.
This change to a louder, more chaotic, classroom dynamic is related to another expected change: an increase in attendance. In both courses, student attendance increased dramatically. Of course, any time a group is larger, it is likely to be louder. However, the students who were now attending classes for the “clicker points” were students who otherwise would not attend. Many of these students were seniors taking these entry-level courses out of sequence. These students were often inattentive or disruptive in class, and would resent feeling that they had to attend or risk “losing points,” even when the points were extra credit. For some class periods, the additional noise was high enough that the instructor had to stop and wait for the class to quiet down before continuing. Student reports suggest this then had the additional unintended consequence of frustrating the students who did want to come to class and who were attentive.
To follow up on these experiences, and to provide data that would allow us to better assess the effects of clickers on the learning environment, we created and administered a survey to students in several courses using clickers during Fall 2003. That survey is discussed in detail in the next section.
From a technological point of view, clicker systems are simple. Indeed, such systems have been used for at least 30 years. In its most basic form—such as is represented by the H-ITT systems—the technology has only one core functionality, which is to register, aggregate, and display the opinions of each individual in the group or audience. Thus, the environment becomes “interactive,” because the speaker can solicit and reliably collect audience input, regardless of the size of that audience. Increasing interactivity is also at the center of most reasons to adopt clicker systems within a classroom. Our experiences suggested to us that there are a number of assumptions behind any decision to adopt clickers. These are assumptions about how clickers will affect the learning environment.
To explore this more deeply, we conducted an exploratory study that looked at four important elements of a learning environment that clickers might affect. The four elements of the learning environment that we investigated were (1) student learning processes, (2) classroom culture, (3) classroom relational messages, and (4) student motivation. We recognize that these elements are interrelated. However, separating them is valuable for heuristic purposes in thinking about our instructional goals in using clickers.
1. Student learning processes. A common assumption is that a clicker system is beneficial to student learning because it can provide immediate feedback. Because many implementations of clicker use also involve peer teaching, there is an additional assumption that students learn well from their peers.
2. Classroom culture. A common goal in using clickers is to provide an alternative to the supposedly impersonal and anonymous traditional “large lecture.” Indeed, creating a small group or discussion atmosphere is an explicit goal of Concept Tests. We wanted to investigate whether students agreed this was a valuable goal. Accomplishing this change depends heavily on whether students “go along” with this goal and change their behavior accordingly.
3. Classroom relational messages. Prior research suggests that strategies that encourage student engagement and involvement improve student learning. One assumption instructors might make is that students want to be engaged and involved during class (as opposed to passive and distant), but find that they can’t in the large lecture format. If this true, clickers have the potential to capitalize on this unmet need. If it is not, clickers may actually increase student apathy and engender resistance.
4. Student motivation. General campus reports indicate that clickers increase classroom attendance. However, our experiences suggested that might not be desirable in terms of the effects on the learning environment. We were interested in how clicker behavior might be related to overall student motivation in the course.
For each of these elements, we looked at assumptions, perceptions, and behaviors. First, we isolated assumptions instructors at CU might be making about what is valuable in a learning environment, that would prompt them to adopt clickers. Then we asked whether or not students agreed with these values. A second interest is whether or not students perceived any relation between those elements of the learning environment and the use of clickers. Third, we asked students to describe their behaviors in using clickers, to see whether or not students acted in ways that would help create the desired learning environment.
RQ1: Do students share the common value assumptions motivating clicker adoption?
RQ2: Do students perceive that clickers are capable of achieving these values?
RQ3: What are the student behaviors that may impact elements of the learning environment?
We conceptualized each of the learning elements as a logical whole. In other words, if a student agreed with the value assumptions, and if the clickers were capable of achieving those values, then the student would also exhibit positive clicker-use behavior.
H1: The level to which students agree with instructional assumptions directly correlates with students’ perceptions of clickers and with their level of desirable clicker behavior.
Using an email list previously compiled by Michael Dubson (Physics) of instructors using clickers, we sent a message offering to distribute surveys to anyone teaching with clickers. We received 7 responses outside of the department. Two declined to participate in the survey this semester, but expressed willingness to participate in the Spring (Weimann from Physics and Asirvatham/Bierbaum from Chem/ChemEng). Of the 5 who agreed, 3 teach physics courses and 2 teach astronomy courses.
All surveys were administered in the last week of class; often during the period FCQs were completed. Care was taken to ensure that students understood this to be a research project specifically relating to clicker use, and not associated with FCQs. Survey answers were anonymous. In completing the survey, students circled numbers corresponding with their level of agreement with a statement (1=strongly disagree, 5=strongly agree). We also collected demographic data including their major, class standing, projected course grade relative to other courses, if they’d taken any other clicker courses, and how many large lecture courses they’d enrolled in at CU. A copy of the survey is included as an appendix to this report.
Although technically possible, we consciously did not administer the survey using clickers, in order to avoid technology bias (and to get answers from students who may not have brought their clickers with them). Additionally, we did not use machine-readable “bubble sheets” in order to avoid students who did not have pencils self-selecting out of the survey. This proved an important decision as many of the surveys were completed in ink.
The last question of the survey invited student comments. The percentage of comments was quite low, however, when they did appear these were coded for evaluations of instructor, class, or clicker. Distinctive information (information not captured by the survey items) was also recorded in a text file and used to supplement statistical analysis. Finally, each faculty member completed a short survey to collect additional information about the clicker classroom experience. Summaries of open-ended responses, and faculty survey responses are both included as appendices to this report.
Data were
collected from six different courses representing three different departments
(Astronomy, Communication, Physics) with seven different instructors. Two instructors (Pollock and Beale) provided
data from two different sections of the same course, and these data were
collapsed for analysis. Two instructors
(Schneider and Colwell) taught different sections of the same course, and these
data were analyzed separately, because of potential instructional differences.
|
Course |
Instructor |
Enrollment |
N |
% Response |
|
Astronomy
1110-01 |
Nick
Schneider |
315 |
191 |
60.6 |
|
Astronomy
1110-02 |
Josh
Colwell |
320 |
165 |
51.6 |
|
Astronomy
1120 |
Doug
Duncan |
206 |
53 |
25.7 |
|
Communication
1210 |
April
Trees |
350 |
282 |
80.6 |
|
Communication
1300 |
John
Jackson |
320 |
171 |
53.4 |
|
Physics
1110 |
Steve
Pollock |
622 |
336 |
54.0 |
|
Physics
2010 |
Paul
Beale |
504 |
345 |
68.5 |
Total
|
|
2637 |
1543 |
58.5 |
Despite the fact that these were lower
division courses, only 40.5% of the students reported being freshman (with
30.3% reporting sophomore, 19.0% junior, and 9.8% senior). Sixteen percent had enrolled (or were
enrolled) in at least one other clicker classes. We also asked students how many “large lecture courses they had
completed. Responses were fairly well
distributed (0 = 3.7%, 1-2 = 32.3%, 3-4 = 30.6%, 5-6 = 17.4%, 7+ = 14.9%). We combined over 80 reported majors into 10
groups. Not surprisingly, natural sciences and Communication were the highest
reported majors.
|
Major |
N |
%
Response |
|
Communication |
288 |
18.9 |
|
Journalism |
67 |
4.4 |
|
Arts |
56 |
3.7 |
|
Humanities |
57 |
3.7 |
|
Social Science |
99 |
6.5 |
|
Natural Science |
343 |
22.6 |
|
Engineering |
209 |
13.7 |
|
Business |
90 |
5.9 |
|
Architecture |
90 |
5.8 |
|
Open Option / Undecided |
222 |
14.6 |
Answers to all survey items showed
variation across courses. Further, group comparisons of the courses with the
highest and lowest mean for each item indicate that the difference between the
two groups was significant for each item.
This data do not support the idea of a universal—or even typical—clicker
classroom experience. Many variations
exist in the learning environment.
RQ1: Do students share the common value
assumptions motivating clicker adoption?
The first research question asked if there was support for the common value assumptions motivating clicker adoption. We did not include items to test the assumption that students generally should be motivated to learn because we determined responses to these items would not have enough variation to be instructive. Assumptions concerning learning were generally supported across courses. Also, as expected, students indicated a preference to be involved and engaged in small courses.
Responses indicate very strong support for the value of feedback to student learning and only somewhat lesser support for the preference to be involved and engaged in large courses. This is an important finding, as feedback is the primary functionality of clickers. At the same time, a number of students express a preference to remain anonymous in large courses.
Finally, there appears to be no clear preference for or against the traditional lecture style. This challenges a common bias against lecturing expressed in discourse about clicker classrooms. The relationship between preference for lecture style and clicker use is explored more fully below.
Though we were not surprised, it is useful to learn that some considerable numbers of students in each course indicate that they think they already know the material (though the means remain below the midpoint, the standard deviations for all courses are over 1.0).
|
Assumptions |
||||||||
|
|
Overall |
ASTRO 1110-1 |
ASTRO 1110-2 |
ASTRO 1120 |
COMM 1210 |
COMM 1300 |
PHYS 1110 |
PHYS 2010 |
|
Learning Processes |
||||||||
|
1. Getting feedback on my
ideas helps me learn better. |
||||||||
|
M |
3.78 |
3.77 |
3.79 |
3.91 |
3.40 |
3.51 |
4.15 |
3.84 |
|
SD |
.97 |
.96 |
.95 |
.88 |
1.04 |
.98 |
.78 |
.95 |
|
2. My expectation before I
started this course was that I knew a majority of the material already. |
||||||||
|
M |
2.61 |
2.26 |
2.25 |
2.34 |
2.64 |
2.92 |
3.04 |
2.42 |
|
SD |
1.25 |
1.10 |
1.21 |
1.06 |
1.31 |
1.07 |
1.30 |
1.20 |
|
Classroom Culture |
||||||||
|
1. Classroom experiences in large courses should be different than classroom experiences in small courses. |
||||||||
|
M |
3.54 |
3.48 |
3.58 |
3.51 |
3.81 |
3.60 |
3.43 |
3.40 |
|
SD |
1.15 |
1.21 |
1.05 |
1.19 |
1.09 |
1.08 |
1.13 |
1.22 |
|
2. If I had a choice, I would
avoid classes where the instructor just lectures. |
||||||||
|
M |
3.09 |
3.17 |
3.09 |
2.77 |
2.91 |
2.87 |
3.26 |
3.18 |
|
SD |
1.24 |
1.31 |
1.16 |
1.24 |
1.25 |
1.29 |
1.17 |
1.23 |
|
3. The best way to teach large
lecture courses is with the traditional lecture style. |
||||||||
|
M |
2.63 |
2.31 |
2.75 |
2.64 |
3.32 |
2.93 |
2.29 |
2.38 |
|
SD |
1.15 |
1.09 |
1.06 |
1.06 |
1.21 |
1.08 |
1.00 |
1.06 |
|
Classroom Relational Messages |
||||||||
|
1. I prefer to be anonymous in
large classes. |
||||||||
|
M |
3.26 |
3.50 |
3.25 |
3.09 |
3.38 |
3.10 |
3.04 |
3.33 |
|
SD |
1.19 |
1.13 |
1.25 |
1.20 |
1.23 |
1.12 |
1.17 |
1.17 |
|
2. In large classes, I prefer
to be involved and engaged. |
||||||||
|
M |
3.19 |
3.01 |
3.02 |
3.04 |
2.58 |
2.85 |
3.69 |
3.58 |
|
SD |
1.19 |
1.27 |
1.15 |
1.04 |
1.18 |
1.09 |
1.03 |
1.06 |
|
3. In small classes, I prefer
to be involved and engaged. |
||||||||
|
M |
4.04 |
4.00 |
3.80 |
4.15 |
4.14 |
3.98 |
4.08 |
4.05 |
|
SD |
1.01 |
1.04 |
1.09 |
.87 |
1.04 |
.99 |
.99 |
.96 |
RQ2: Do students perceive that clickers are
capable of achieving these values?
One important finding is that, overall, students clearly perceived the ability of clickers to improve their learning. More important, however, is the differentiation across courses. Students agreed most strongly for PHYS1110, which follows a structure based closely on Mazur’s Concept Tests. The instructor, Steve Pollock, reports never having given a traditional lecture in this course. He also posts the clicker questions online on the course website, extending the idea and usefulness of “clickers” beyond the transmitter device toward an integrated pedagogical approach. In contrast, in Communication courses, where clickers augment a traditional lecture style, the perceived value of clickers to learning is markedly lower (with the mean below the midpoint). This reinforces the importance of developing “clicker pedagogy” and of helping instructors frame the usefulness of clickers to their courses.
Contrary to our expectations (and to open-ended comments on several surveys), responses do not indicate an overall strong opinion as to whether clickers were used to “track” or “watch” students. However, as a corollary, neither did clickers generally make students feel either more or less anonymous.
We asked whether or not students perceived that the use of clickers helped the class to be more like the experience of a small class. Students enrolled in courses following a Concept Test format, in which students do engage for the most part in small group discussion, gave the strongest agreement, though the mean falls below a 4 (out of 5). Thus, it may be that, when used in this way, clickers remove structural constraints imposed by a large class size, allowing activities typically appropriate for smaller classes.
For the most part, students saw clicker teaching styles as more immediate (warm, friendly, close).
Finally, we were interested in what motivated students to attend class. Results suggest that students are generally motivated to attend class but they vary considerably on whether or not they are internally motivated to attend. Thus, it may not be desirable to use clickers as a tool to motivate students to attend class. Strategies such as “clicker participation” points are an external motivator. Below, we explore motivation in more detail.
|
Results: Perceptions |
||||||||
|
|
Overall
|
ASTRO 1110-1 |
ASTRO 1110-2 |
ASTRO 1120 |
COMM 1210 |
COMM 1300 |
PHYS 1110 |
PHYS 2010 |
|
Learning
Processes
|
||||||||
|
1. By using clickers in this
class, I got feedback on my understanding of class material. |
||||||||
|
M |
3.62 |
3.99 |
3.69 |
3.55 |
2.46 |
2.85 |
4.43 |
3.92 |
|
SD |
1.26 |
1.02 |
.99 |
1.17 |
1.19 |
1.24 |
.80 |
1.06 |
|
2. Clicker questions helped me to know how well I
was learning the material. |
||||||||
|
M |
3.57 |
3.88 |
3.70 |
3.48 |
2.60 |
2.86 |
4.30 |
3.79 |
|
SD |
1.23 |
1.04 |
.98 |
1.13 |
1.17 |
1.19 |
.83 |
1.17 |
|
3. Clicker questions were helpful for preparing
me for the exams in class. |
||||||||
|
M |
3.27 |
3.66 |
3.24 |
2.81 |
2.29 |
2.63 |
4.02 |
3.50 |
|
SD |
1.29 |
1.20 |
1.14 |
1.13 |
1.15 |
1.17 |
.95 |
1.22 |
|
Classroom
Culture
|
||||||||
|
1. The use of clickers helped my experience in this class to be more
like the experience of a small class. |
||||||||
|
M |
3.10 |
3.54 |
3.17 |
2.94 |
1.89 |
2.53 |
3.76 |
3.46 |
|
SD |
1.26 |
1.20 |
1.07 |
1.15 |
.98 |
1.08 |
.95 |
1.15 |
|
2. The use of clickers in this
class made me feel like I was being “watched”. |
||||||||
|
M |
2.53 |
3.12 |
2.58 |
2.51 |
3.15 |
2.45 |
2.16 |
2.45 |
|
SD |
1.28 |
1.16 |
1.33 |
1.34 |
1.37 |
1.18 |
1.14 |
1.22 |
|
Classroom
Relational Messages
|
||||||||
|
1. The use of clickers in this
class has made me feel less anonymous. |
||||||||
|
M |
2.83 |
3.12 |
2.78 |
2.68 |
2.39 |
2.54 |
3.05 |
3.03 |
|
SD |
1.16 |
1.16 |
1.09 |
1.19 |
1.18 |
1.08 |
1.12 |
1.13 |
|
2. Clickers made the teaching
style seem more immediate: combined 3
items – warm, friendly, close. |
||||||||
|
M |
3.51 |
3.74 |
3.61 |
3.51 |
2.73 |
3.06 |
3.96 |
3.75 |
|
SD |
.94 |
.84 |
.86 |
.87 |
.85 |
.81 |
.72 |
.87 |
|
Student Motivation |
||||||||
|
1. I attend this class because I experience pleasure and
satisfaction in learning new things here. |
||||||||
|
M |
3.27 |
3.59 |
3.42 |
3.72 |
2.54 |
2.86 |
3.70 |
3.31 |
|
SD |
1.15 |
1.09 |
1.16 |
1.18 |
1.08 |
1.10 |
.96 |
1.09 |
|
2. I attend this class for the satisfaction I feel when I am in
the process of accomplishing difficult activities. |
||||||||
|
M |
2.92 |
3.06 |
2.93 |
3.25 |
2.31 |
2.74 |
3.25 |
3.05 |
|
SD |
1.09 |
1.03 |
1.03 |
1.16 |
.99 |
1.03 |
1.02 |
1.09 |
|
3. Honestly, I don’t know why I attend this class, I really feel
that I am wasting my time here. |
||||||||
|
M |
2.06 |
1.80 |
1.88 |
1.94 |
2.81 |
2.72 |
1.64 |
1.77 |
|
SD |
1.23 |
1.12 |
1.10 |
1.29 |
1.32 |
1.25 |
.99 |
1.05 |
RQ3: What are the student behaviors that
may impact elements of the learning environment?
If clickers are to be an effective learning tool, students must use them in a serious, reflective manner. Responses indicate that students pay more attention to clicker questions and clicker answers in courses that are more heavily structured around clicker use or where discussions are based in clicker answers. Students also expressed stronger agreement in courses where instructors had previous experience with clickers. Responses clearly indicate that students take clickers seriously: they pay attention to clicker questions, and they think about their answers.
Another surprising finding is that, even in courses using small-group exercises and discussions, students do not consistently agree that they are learning from their peers. It would be interesting to examine interaction within these groups to see the level to which the discussion processes are deliberative.
Generally, students give only slight agreement to items asking if they actively participate during class. Students may not perceive clicker use as engaged behavior; perhaps the use of the transmitter is too close to the use of a TV remote control. In some courses, students reported strong agreement that their instructor is more responsive than those in other courses, but open-ended comments temper our interpretation. Several students indicated that the instructor was more responsive irrespective of the use of clickers.
We asked students to report if they saw other students using more than one clicker during class. Cheating is an important concern for two reasons. The first, obviously, is that clicker use is usually tied to student grades. The second is more central to our interest in the classroom culture. If students see that their peers routinely cheat and get away with it, we could expect negative effects, including lower student morale.
Our results only indicate the level of observations of cheating, not the level of cheating itself. In order to encourage honest answers, we did not ask students directly if they cheated. This could be explored in a follow up study. Looking only at responses above the scale midpoint, results show that at least 20% and up to 58% of the students report observing cheating. (In one course, students extended the scale, and circled higher numbers 6 and 10). These percentages refer only to the number of students who completed the survey, so the actual number of students who reported observing cheating is instructive.
|
Observations
of Cheating by Course, Percentage (N) I
personally saw students use more than one clicker during class |
|||||||
|
|
ASTRO 1110-1 |
ASTRO 1110-2 |
ASTRO 1120 |
COMM 1210 |
COMM 1300 |
PHYS 1110 |
PHYS 2010 |
|
4 (agree) |
15.1 (28) |
20.6 (34) |
9.4 (5) |
13.7 (38) |
14.4 (24) |
7.9 (26) |
17.2 ( 58) |
|
5
(strongly agree) |
15.1 (28) |
37.6 (62) |
22.6 (12) |
12.9 (36) |
12.6 (21) |
13.9 (46) |
16.0 ( 54) |
|
Total |
30.2 (56) |
58.2 (96) |
32.0 (17) |
36.6 (74) |
37.0 (45) |
21.8 (72) |
33.2 (112) |
Finally, we were interested in whether clickers served as an external motivator for student attendance. Overall, results are above the midpoint, suggesting that clicker points are a motivator. Understandably, percentages are high in courses where points were a significant part of the overall course grade. But we found high percentages even in courses where the points are either a minimal amount of the grade, or only extra credit. Offering clicker points changes student behavior. Students attend class, but it may be that for some students there is no corresponding change in their internal motivation.
|
Clicker
Points as Motivation to Attend Class For me,
earning “clicker” points motivates me to come to class |
|||||||
|
|
ASTRO 1110-1 (10% of grade) |
ASTRO 1110-2 (10% of grade) |
ASTRO 1120 (10% of grade) |
COMM 1210 (<4% of grade) |
COMM 1300 (extra credit) |
PHYS 1110 (extra credit) |
PHYS 2010 (15% of grade) |
|
4 (agree) |
29.3 % |
27.3 % |
24.5 % |
28.7 % |
27.5 % |
28.3 % |
24.3 % |
|
5
(strongly agree) |
55.0 % |
50.3 % |
41.5 % |
22.3 % |
28.1 % |
47.3 % |
62.3 % |
|
Total |
84.3 % |
77.6 % |
66 % |
51.0 % |
55.6 % |
75.6 % |
86.6 % |
In analysis presented below, we examine whether students of different profiles (for example, class standing) exhibit more or less positive behavior, or demonstrate different motivation.
Results: Behaviors |
||||||||
|
Learning
Processes
|
||||||||
|
|
Overall |
ASTRO 1110-1 |
ASTRO 1110-2 |
ASTRO 1120 |
COMM 1210 |
COMM 1300 |
PHYS 1110 |
PHYS 2010 |
|
1. I chose my answer to each clicker question carefully. |
||||||||
|
M |
3.85 |
4.35 |
4.04 |
4.21 |
2.96 |
3.38 |
4.14 |
4.09 |
|
SD |
1.08 |
.79 |
.94 |
.87 |
1.19 |
1.10 |
.82 |
.95 |
|
2. I pay attention to whether or not my answer to a clicker
question was right or wrong. |
||||||||
|
M |
4.34 |
4.59 |
4.45 |
4.60 |
3.70 |
3.92 |
4.61 |
4.56 |
|
SD |
.98 |
.83 |
.94 |
.53 |
1.14 |
1.07 |
.80 |
.75 |
|
3. I’m learning more from my peers in this class than I do in
other large classes. |
||||||||
|
M |
3.13 |
3.30 |
2.95 |
3.09 |
2.19 |
2.38 |
3.59 |
3.83 |
|
SD |
1.28 |
1.15 |
1.21 |
1.17 |
1.08 |
1.02 |
1.11 |
1.12 |
|
Classroom Culture |
||||||||
|
1. I actively participate during class. |
||||||||
|
M |
3.29 |
3.15 |
2.83 |
2.94 |
2.89 |
2.93 |
3.72 |
3.71 |
|
SD |
1.11 |
1.12 |
1.05 |
1.09 |
1.07 |
1.10 |
.98 |
1.00 |
|
2. I behave differently in this class than I do in other large
classes. |
||||||||
|
M |
2.58 |
2.47 |
2.45 |
2.32 |
2.11 |
2.38 |
2.75 |
3.05 |
|
SD |
1.31 |
1.30 |
1.25 |
1.30 |
1.12 |
1.26 |
1.32 |
1.33 |
|
3. The instructor is more responsive during this class than
instructors in other large classes. |
||||||||
|
M |
3.56 |
3.86 |
3.95 |
3.57 |
2.52 |
2.98 |
4.12 |
3.81 |
|
SD |
1.16 |
1.03 |
.99 |
1.08 |
1.08 |
1.00 |
.87 |
1.03 |
|
Classroom Relational Messages |
||||||||
|
1. Clicker questions encouraged
me to be more engaged in the classroom process. |
||||||||
|
M |
3.69 |
4.00 |
3.87 |
3.66 |
2.49 |
3.10 |
4.34 |
4.08 |
|
SD |
1.25 |
1.08 |
1.02 |
1.29 |
1.17 |
1.18 |
.82 |
1.06 |
|
2. I responded to clicker questions without really thinking about
the answer. |
||||||||
|
M |
1.90 |
1.60 |
1.75 |
1.60 |
2.54 |
2.21 |
1.70 |
1.71 |
|
SD |
1.03 |
.85 |
.94 |
.77 |
1.19 |
1.10 |
.90 |
.87 |
|
3. Often I feel withdrawn or distant during interactions in this
class. |
||||||||
|
M |
2.57 |
2.40 |
2.62 |
2.65 |
3.07 |
2.80 |
2.42 |
2.25 |
|
SD |
1.07 |
1.07 |
1.07 |
1.22 |
1.10 |
1.07 |
.95 |
.97 |
|
4. Sometimes during a class, students were asked to discuss
material with each other. Because the
class used clickers, we discussed the material more seriously. |
||||||||
|
M |
3.36 |
3.88 |
3.78 |
3.75 |
2.00 |
2.46 |
3.83 |
3.89 |
|
SD |
1.27 |
1.01 |
1.00 |
1.02 |
.99 |
1.06 |
.96 |
1.02 |
|
Student Motivation |
||||||||
|
1. For me, earning “clicker” points motivates me to come to class. |
||||||||
|
M |
3.96 |
4.29 |
4.17 |
3.83 |
3.23 |
3.46 |
4.08 |
4.42 |
|
SD |
1.23 |
.99 |
1.05 |
1.30 |
1.41 |
1.36 |
1.12 |
.92 |
|
2. I attended class when I otherwise would not have because of the
clickers. |
||||||||
|
M |
3.32 |
3.59 |
3.32 |
2.96 |
3.27 |
3.12 |
3.16 |
3.54 |
|
SD |
1.38 |
1.30 |
1.35 |
1.32 |
1.37 |
1.36 |
1.39 |
1.40 |
H1:
The level to which students agree with instructional assumptions
directly correlates with students’ perceptions of clickers and with their level
of desirable clicker behavior.
We assessed whether or not students’ assumptions concerning ideal learning processes, classroom culture, and classroom relational messages predicted both positive perceptions of clickers and positive clicker behaviors. First, we identified four different items that clearly indicated positive perception and four different items that clearly indicated positive behavior, from an instructor’s perspective. These items were chosen from three elements of the learning environment: learning processes, classroom culture, and relational messages. We used these items as criterion variables for eight different standard regression analyses. We used as predictor variables all items assessing students’ agreement with clicker instructional assumptions (as listed above, with the exception of the item concerning assumptions about small classes). We recoded items as needed so that higher scores reflected stronger agreement with assumptions motivating clicker use in the classroom.
Results demonstrate support for the hypothesis. All regression equations were significant at p<.001 (see Table), indicating a clear relationship between agreement with assumptions and positive perceptions and behavior. Two of the seven assumptions consistently and strongly predict positive perceptions and behaviors: 1) I prefer to being involved and engaged in large classes, and 2) getting feedback on ideas helps me learn better. Not surprisingly, students who indicated a higher preference for being involved and engaged in large lecture were more likely to perceive clickers positively and engage in desirable clicker behaviors. Additionally, students with a greater assumption that feedback on ideas helps them learn better predicted seven of the eight dependent variables.
Two additional assumptions are also statistically significant predictors: 1) The traditional lecture style is not the best way to teach large courses, and 2) I do not prefer to be anonymous. This means that a student’s attitudes with regard to these four assumptions are mostly likely be reflected in the level to which his or her perceptions and behaviors toward clickers are positive.
|
Criterion
Variable |
R |
R2 |
F |
Percent
Variance Accounted for by Agreement with Assumptions |
|
Clicker questions helped me to know how well
I was learning the material |
.68 |
.46 |
24.71** |
53 |
|
Clicker questions encouraged me to be more engaged
in the classroom process. |
.70 |
.49 |
28.14** |
49 |
|
I actively participate during class. |
.67 |
.45 |
23.38** |
45 |
|
The use of clickers helped my experience in this
class to be more like the experience of a small class |
.63 |
.40 |
19.07** |
40 |
|
By using clickers in this class, I got feedback on
my understanding of class material |
.62 |
.39 |
18.41** |
39 |
|
Sometimes during a class, students were asked to
discuss material with each other.
Because the class used clickers, we discussed the material more
seriously. |
.58 |
.34 |
14.96** |
34 |
|
I chose my answer to each clicker question
carefully |
.53 |
.28 |
11.24** |
28 |
|
The use of clickers in this class has made me feel
less anonymous |
.34 |
.12 |
3.86** |
12 |
** p<.001
Regression Analyses:
Significant Predictor Variables
|
Criterion Variable (Perceptions and Behaviors) |
Predictor Variables (Assumptions) |
|||
|
|
I prefer
to be involved and engaged in large classes |
Getting
feedback on ideas helps me learn better |
Traditional
lecture style is not the best way to teach large courses |
I do not
prefer to be anonymous |
|
Clicker questions helped me to know how well
I was learning the material |
β =
.24 *** |
β =
.39 *** |
β =
.26 *** |
β =
.12 * |
|
Clicker questions encouraged me to be more engaged
in the classroom process. |
β =
.28 *** |
β =
.36 *** |
β =
.20 *** |
|
|
I actively participate during class. |
β =
.45 *** |
β =
.22 *** |
|
|
|
The use of clickers helped my experience in this
class to be more like the experience of a small class. |
β =
.41, *** |
β =
.23 *** |
β =
.18 ** |
β =
-.21 *** + |
|
By using clickers in this class, I got feedback on
my understanding of class material |
β =
.26 *** |
β =
.32 *** |
β =
.23 *** |
|
|
Sometimes during a class, students were asked to
discuss material with each other.
Because the class used clickers, we discussed the material more
seriously. |
β =
.30 *** |
β =
.24 *** |
β =
.15 * |
β =
-.19 *** |
|
I chose my answer to each clicker question
carefully |
β =
.31 *** |
β =
.28 *** |
|
β =
-.16 * + |
|
The use of clickers in this class has made me feel
less anonymous |
β =
.28 ** |
|
|
|
* p <= .05
** p < .01
*** p <= .001
+ This negative relationship likely reflects a suppression effect since the zero-order correlation between the criterion and predictor variable was small, but positive.
The use of clickers assumes that students will find the kind of feedback that clickers provide to be helpful for the learning process. The pattern of findings in this analysis suggests that clicker feedback will be perceived as more useful if the student already understands and values feedback in general. Whether or not students see the value of feedback generally also may motivate careful, considered clicker-related behavior and engagement in the classroom activity. Students need to see the relevance of the clicker activity for their learning.
This pattern of findings also suggests that the degree to which the behaviors required in a “clicker classroom” violate students’ expectations and preferences for how a large lecture class should operate impacts their perceptions and (more importantly) their classroom behavior. The clicker itself does not ensure engaged, active students in the classroom, but rather is a tool that may facilitate that process, depending in part upon the expectations that students bring to the large lecture class.
Finally, we note that the one item concerning students’ assumptions about course content—the degree to which they expected to have a lot to learn—did not predict any of the clicker perceptions or behaviors. Follow-up correlations confirm a lack of relationship between this item and the consistency of student assumptions. This suggests that class content (at least in this dimension) is less relevant than more general preferences for nontraditional practices in large lecture classes for predicting positive attitudes and behaviors in regard to clickers.
For follow-up investigations, we created two different composite variables. The first, desirable learning processes, included five items clearly assessing perceptions of and reported behaviors regarding clickers and learning (α = .86). The second, student engagement, included six clearly positive items within classroom culture and relational message categories that measured students’ perceptions of and reported behaviors regarding active involvement and participation in the class and included six items (α = .80). Where relevant, items were recoded so that when the composites were created higher scores reflected more positive learning perceptions and behaviors or greater active involvement perceptions and behaviors.
The amount of clicker use within classes. Faculty reported using clickers to a substantially varying degree. We investigated whether this level of use could be related to whether students responded favorably to clickers. Using the two scales identified above as dependent variables, we conducted two separate one-way ANOVAs with courses grouped according to the number of clicker questions typically used within a class day (1-3, 4-6, 7-9).
Results indicate a significant difference among groups for both reports of learning processes, F (2, 1504) = 155.32, p < .001, and reports of involvement, F (2, 1479) = 149.61, p < .001. Post hoc analyses indicated a significant difference in reports of learning-related perceptions and behaviors among classes using 1-3 questions (M = 3.30), 4-6 questions (M = 4.14), and 7-9 questions (M = 3.97). For involvement, classes using 1-3 questions (M = 3.07) differed significantly from classes using 4-6 questions (M = 3.65) and 7-9 questions (M = 3.72), but the latter two groups did not differ significantly from one another.
Additionally, we investigated the degree to which students’ reports of clickers motivating attendance and attending when they otherwise would not were related to the number of clicker questions used in a class. This one-way ANOVA was also significant for both points motivating attendance, F (2, 1539) = 57.09, p < .001, and attending when otherwise would not, F (2, 1531) = 6.11, p< .01. Classes with 1-3 questions (M = 3.52), 4-6 questions (M = 4.01), and 7-9 questions (M = 4.32) all differed significantly from one another on students’ reports that points motivated attendance. For students reports that they attended when they otherwise would not, classes with 1-3 questions (M = 3.20) and classes with 4-6 questions (M = 3.09) differed significantly from classes with 7-9 questions (M = 3.39).
The incorporation of clicker points in grades. To explore this more deeply, we looked at whether or not there was a difference if clicker points were extra credit or part of the grade. We excluded Communication 1210 from the analysis because clickers were incorporated into the grading policies, but on a much smaller level than other courses. No significant differences in the t-tests were identified between courses with extra credit and those with points incorporated into the class grade for both learning perceptions and involvement scales. One t-test indicated significant differences in the degree to which students reported that points motivated their attendance, t (1258) = 6.62, p < .001 (extra credit, M = 3.87; part of grade, M = 4.29) . A second t-test, with students’ report of the degree to which they attended class when they otherwise would not as the dependent variable was also significant, t (1252) = 4.03, p < .001 (extra credit, M = 3.15; part of grade, M = 3.49).
Higher numbers of clicker questions used in a class increase the visibility and centrality of clickers in the classroom. This may effect students’ assessments of their learning and involvement as it pertains to clickers.
In terms of learning, students reported more positive assessments of learning processes in classes in which 4-6 questions typically were asked, followed by students in classes with 7-9 questions and then 1-3 questions. One possible explanation for this would be that students are more likely to see the usefulness of clickers for their learning when they are more frequently incorporated into the activities of the classroom. Responses to the open-ended question at the end of the survey suggest it is also possible that those in the group with the greater number of questions see that as eliminating any time for lecture.
In terms of involvement, clearly students saw themselves as more active participants in classes with a greater number of questions (both 4-6 and 7-9), something that is not surprising since more clicker questions both give them more opportunities to participate and place more responsibility on students to contribute to the classroom process. Finally, it seems clear from the pattern of results that clickers function as a stronger motivation for attending class if they are used more extensively during the class period.
Not surprisingly, both the ways in which clickers are incorporated into the grade and the number of clicker questions used in class predicted the degree to which clickers worked as external motivators for attendance.
Differences Across
Student Demographics
We looked at differences in evaluations of clickers between students taking classes in different disciplines, grouping astronomy, communication, and physics courses separately. One-way ANOVAs for both learning and involvement scales were significant (learning: F (2, 1504) = 333.51, p < .001; involvement: F (2, 1479) = 300.59, p < .001). Post hoc analyses indicate that astronomy classes (M = 3.93), communication classes (M = 2.92), and physics classes (M = 3.72) all differed in students’ reports of learning processes. Courses in the three disciplines also differed significantly on the involvement scale as well, astronomy (M = 3.49), communication (M = 2.80), and physics classes (M = 3.75).
Clickers are embraced or resisted by students of a specified profile. In addition to ways in which the course structure may influence students’ evaluations of clickers, it is also possible that certain types of students may be more likely to positively evaluate clickers in the large course. Two dimensions of student experience that seem likely to affect their assessment of clickers in the classroom are their class standing and the number of large lecture courses that they have taken. We expect these two dimensions to be significant because students who are seniors may have a stronger sense of how things should be in a university large lecture course. Also, the more lecture courses students have experienced the less likely they may be to be open to alternative ways of doing lecture.
To test this, two separate factorial ANOVAs were run with reports of learning and reports of involvement again as the dependent variables. Students’ report of learning processes differed significantly for both class standing (F (3, 1468) = 2.78, p < .05 and number of large lecture classes students had (F (4, 1468) = 2.44, p < .05). Post hoc analyses indicated that freshman (M = 3.87) and sophomores (M = 3.81) differed from juniors (M = 3.44) and seniors (M = 3.39) in their evaluation of clicker’s contribution to learning. Additionally, students with 5- 6 (M = 3.52) and 7+ (M = 3.50) lecture classes differed significantly from students who had taken 0 (M = 3.92), 1-2 (M = 3.88) or 3-4 (M = 3.75) large courses. A second factorial ANOVA indicated no significant differences in students’ assessment of involvement by class standing or experience with large lectures.
We also considered that students’ assessment of their grade in the class compared to their grades in other classes and their major would influence their perceptions of the learning processes and their involvement. A factorial ANOVA with students’ assessment of learning as the dependent variable indicated differences across majors (F = 19.84, p < .001) and among students expecting lower (M = 3.37), similar (M = 3.83), or higher (M = 4.12) grades (F= 33.03, p < .001). With involvement as the dependent variable, differences were also identified among students expecting lower (M = 3.15), similar (M= 3.48), and higher (M = 3.70) grades in this class (F = 25.11, p < .001). A main effect was also found for major, F (9, 1421) = 17.59, p < .001.
As expected, freshman and sophomores perceived greater learning with clickers than did juniors and seniors. Additionally, more experience with lecture classes meant that students had a lower assessment of the learning being done with clickers. Both of these findings suggest it is possible that as students become socialized into the university and experience traditional large course structures, they may be more resistant to changes in learning processes. It is also possible, given the fact that all of these classes are lower division courses, that seniors in particular may have a negative response to the fact that attending lecture impacts their grade. In fact, two one-way ANOVA’s with the two “clicker motivation” items as dependent variables indicated clicker points were less likely to motivate attendance for seniors than for freshmen and sophomores, and seniors were less likely than sophomores to attend class when they otherwise would not. While these differences were small, they do suggest further attention might be paid to the ways in which students past experiences with classes impact their evaluation of clicker classrooms, particularly in terms of understanding whether or not learning is occurring as a result.
Finally, due to the ways in which students’ majors were also linked to the disciplinary divisions of the courses being analyzed (i.e., 282 of the 288 communication and pre-communication majors were in communication courses), it was impossible to tease out the ways in which students’ clicker perceptions and behaviors may differ across majors, separate from the influences of the courses they were in. Students were also asked, however, to assess whether or not they thought their grade in the clicker course would be lower than, similar to, or higher than their grades in other courses. Students expecting lower grades had a lower average on both assessments of learning and assessments of involvement. There are several possible explanations for this, including the expectation that involvement and engagement in class should lead to better performance in the course and that treating clickers seriously as a learning tool (i.e., taking clicker questions seriously, seeing the ways in which they contribute feedback about understanding) should also lead to a better performance in the course. It is also possible that students who do not do as well as they typically perform in other courses were disengaged. It is possible that at least some of these students had expectations about what a 1000 or 2000 level large lecture course should look like and are resistant to a change in that.
This study has focused on the learning environment of “clicker classrooms,” in order to gain insights into the ways in which clickers could be seen to present new potentials (or problems) for teaching large courses. We recognize, however, that this study has a number of limitations. First, the survey was given only at the very end of the semester. It would be very instructive to track the learning environment throughout the semester, and to survey students at least at the beginning and end of the course. Second, this scale was not tested for reliability, and the number of items testing each variable were very low (1-2). However, our results suggest that our approach—using key variables to measure and assess the learning environment—is instructive. Further study should refine the survey instrument. Third, the contents of the survey were limited by the late start date of the study. The items emerged from our interests in the learning environment, and from discussions of the “clicker user group” FTEP sponsored during Fall 2003. We did not review existing instruments or the literature on SRS in order to compose the survey. Open-ended comments on collected surveys suggested a number of questions that should have been included as items, such as a general score for satisfaction with clickers, questions concerning levels of students anxiety with the technology and their learning, and questions relating to the technology itself (including cost and perceived reliability).
Clickers are an unusual educational technology. Unlike web sites, PowerPoint, or even webCT, the success of clickers depends less on the instructor and more on the student. In other words, no matter how skillfully the instructor incorporates clicker questions into the classroom, the final effectiveness of the clicker will rest with each student accepting the potential of clickers to positively affect their learning. Students must use them skillfully and faithfully. Students must also embrace the ways that clickers will change their learning experience. Moreso than any technology since the video (which transformed the classroom into a movie theatre), clickers transform the classroom experience. As an educational tool, clickers do not simply augment the classroom, they transform it because of the new responsibilities placed upon the students to guide and affect classroom dynamics.
1. Ensure that clickers integrate seamlessly with the class. Because students still do not have extensive experience with clickers, from the beginning of your course students will be forming opinions and judging the effectiveness of the clickers. Early technical or pedagogical problems will lead students to develop negative opinions that will be difficult to overcome. Suggestions: Practice with clickers in a simulated environment before you begin to teach with them. Visit the classroom of a teacher who uses the clickers in the same manner you will use them. Ensure that you have adequate real-time technical support to back you up in the event of a problem.
2. Take care to frame early and often how clickers fit into the course and your learning goals. Students who recognize the value of feedback and who prefer to be involved and engaged in the classroom are more likely to be positive toward the clickers. In the beginning of class, emphasize these as general aspects of learning: the value of feedback, and the value of involvement. Making these assumptions explicit and placing them at the center of the class will help students see clearly the ways that clickers help you to achieve your goals. Place statements in your syllabus, and incorporate both feedback and involvement into your courses in ways beyond just the use of clicker questions. In other words, recognize the deeper issue of pedagogical commitments. Students may not respond positively if they don’t see the use of clickers as necessary to your pedagogical style.
3. Make clicker questions a central component of the learning experience in the classroom. Although on the surface, making clickers worth a significant part of the grade may seem to provide this centrality, the deeper issue is that students need to be able to accomplish different things through the use of clicker questions than could be accomplished by simply listening to a lecture of the same material. This is similar to the challenge of making lecture extend beyond material found in the textbook.
4. Be ready for the transformation of the classroom dynamics. Clickers are an invitation to (and perhaps a requirement for) students to be active. Because the norms of this type of classroom are not well established, students may exhibit surprising behavior, some of it undesirable. Your classroom will be louder, more students will be present (often significantly more), it will less controlled and seemingly more chaotic. The clicker classroom requires strong classroom management skills.
1. Reduce student experience of technical problems. Even though we did not include a survey item that asked students to report their level of technical problems with the clickers, we received many open ended comments that suggested they experienced problems. Whether or not these were “legitimate” problems, a perception that the technology can be trusted is critical for extending clicker effectiveness to more students.
2. Target clicker cheating behavior. This is another example of how clickers are a distinctive educational technology. Students are graded on clicker use. Thus, clickers are not only a tool for teaching; they are also a tool for assessment. From the student’s perspective, these are not separated. They occur simultaneously. Students need to trust that their assessment is fair and depends on each student’s honorable behavior. Until cheating is curbed, it is unlikely that clickers will achieve their full potential as a pedagogical tool. Regardless of how experienced the instructor is with using clickers, a substantial portion of students report observing cheating behavior.
3. Recognize the centrality of feedback to clicker classroom learning. This suggestion applies primarily to “concept test” courses. Our results suggest that clickers are useful when students acknowledge that they learn from feedback. However, students expecting a lower grade in the course do not respond positively to clickers. This may reflect a learning style that is not being met. Open-ended survey comments suggest that not all students find clicker feedback sufficient. They express the need for some explanation or lecture. Consider whether there are opportunities to augment the feedback provided by clicker use with feedback in the form of an explanation, a mini-lecture, or a guided large group discussion.
1. Structured training is crucial for new adopters. The basic concept of clickers is straightforward and easy for interested instructors to “get” or understand. It is also easy for a new instructor to envision potential ways to pose clicker questions. However, the simplicity of clicker technology is deceptive and dangerous. Our results suggest that the actual dynamics of clicker classrooms are surprising and challenging without advance training. We also found that effective clicker use is bound up with a pedagogy that logically and consistent calls for the kind of feedback made possible by clickers. To be effective in their first semester, new instructors require structured training not only on H-ITT software, but also how to construct questions, and how to manage a clicker classroom. Training should include a strong component that realistically simulates the behaviors of an actual clicker classroom.
2. The technology must be seamless. This is a basic principle of educational technology, but magnified by the presence of hundreds of pieces of technology within a single classroom. At the least, attention should be given to preventing installations where the receivers are overloaded. Students should have central support resources available to test their clickers to see if they are functional. Each room might have a few extra clickers in the room for students to use if theirs is not functioning.
3. Develop a central registration site for the campus. Develop procedures for instructors to interface with the site and access their class data. An effective registration is critical to student trust of clickers. As more departments adopt clickers, it makes sense for students to have a central, secure, location to register their clickers (perhaps on the student portal).
4. Investigate further the patterns of how students learn and respond to clickers. Our results suggest that while there are some strong predictors for positive student behavior, there are substantial students who respond to the clickers in different ways. Future research should explore these other patterns.
5. Collect or develop pedagogical techniques that are alternatives to Concept Tests. The Concept Test approach, although well developed and tested, is not appropriate for all course material or all pedagogical approaches. Clicker success also seems to be related to how well the instructor frames student perception of clickers. Future study should investigate more closely (through observation and fieldwork) how this framing is done. If clickers are to be adopted more widely, it is important that instructors have resources available that will help them to structure their use in ways that will improve effectiveness.
6. Develop a comprehensive “checklist” for new instructors. Our instructors experienced a number of surprises that created challenges for their teaching. In hindsight, several of these surprising could have been anticipated. Further work should collect these experiences and create “tips and tricks” that can also improve effectiveness by reducing the learning curve for the new instructor.
Contrary to the marketing discourse surrounding student response systems, clickers do not necessarily tap an unmet need or desire in students to avoid the experience of the large classroom. Yet, from an instructional viewpoint, there are advantages to developing alternatives to the traditional lecture. Complexity arises because students are not only accustomed to large lecture (and their responsibilities therin), they may prefer it for the way it fits with their learning style. From this study, it appear that while “clicker classrooms” are certainly not large lectures, neither do they provide the experience of a small seminar. In fact, they do not appear to be thing at all. Clickers are a tool that can support many pedagogical approaches. If we are going to make them effective, we need to learn more about these variations.
Survey on “Clicker” Use
Thank you for taking time to
complete this survey. Many instructors
at CU-Boulder are experimenting with “clickers” and the university is
considering whether to promote their use extensively in other classes. Your
answers are very important because they will help the campus decide how (or
whether) to continue using them.
All your answers are
confidential. This means that your
survey will never be shared with your instructors, and only summaries and
statistics will be shared with them.
Nothing will be shared with them until after the semester is over and final
grades are assigned. This study is being conducted by Professor Michele
Jackson, in the Department of Communication, with the support of the Faculty
Teaching Excellence Program. If you
have any questions or concerns, please contact Prof. Jackson at 303-492-8139 or
jackson@colorado.edu.
Demographic
Information
What is your current major? _______________________________
What is your class standing? (circle one) Freshman Sophomore Junior Senior or higher
How does your grade so far in this class compare to your
other classes? (circle one)
Lower Similar Higher
What other clicker classes are you enrolled in (if
any)? _____________________________
In your opinion, how many of the courses you have taken at
CU are “large lecture” courses (not including this course)? 0 1-2 3-4 5-6 7+
Clickers
as an Instructional Tool
All these questions are about the lecture portion of this course only (not recitations or labs).
|
|
Strongly Strongly Disagree Agree |
|
I attend this class because I experience pleasure and
satisfaction in learning new things here. |
1 2 3
4 5 |
|
By using clickers in this class, I got feedback on my
understanding of class material. |
1 2 3
4 5 |
|
Classroom experiences in large courses should be
different than classroom experiences in small courses. |
1 2 3
4 5 |
|
Clicker questions encouraged me to be more
engaged in the classroom process |
1 |