An advisory committee is investigating the effectiveness of online course evaluations after complaints that they have led to lower response rates and more vicious comments from students, administrators said.

Professors say online course and teacher evaluations may be drawing responses skewed against them. Some departments are even discouraging the use of online CTECs because student evaluations can factor into tenure and promotion decisions.

The committee is formulating plans for a study and hopes to release results by next fall.

Since Spring Quarter 2000, about 75 percent of CTECs have been administered online, although faculty members can request paper versions.

Many teachers worry that online CTECs are attracting a different pool of students than those tested when evaluations are given in class.

“The reason why people go to complete (CTECs) online is because they have a beef,” said Wesley Skogan, a professor in the political science department who uses paper CTECs.

Prof. Charles Moskos said he uses in-class evaluations for his Introduction to Sociology class.

“People are more likely to write comments,” Moskos said. “And the response rate is everyone in class except those who are cutting.”

Nedra Hardy, senior assistant registrar, said faculty members who request paper CTECs are primarily concerned about low online response rates.

Hardy said average response rates for paper CTECs are 70 to 75 percent, while online rates average 50 percent.

But Ken Bain, director of the Searle Center for Teaching Excellence, said these numbers are “misleadingly high.”

Bain said the classes with high online response rates are dragging up the average and, in the majority of classes, rates are too low to provide results that can be trusted.

“The number of classes with sufficient response rates (between 70 and 80 percent) to consider the results reliable and valid are depressingly low,” Bain said.

Bain is part of the CTEC advisory committee that is conducting a study to investigate the low response rates. Bain said he hopes results will be released by next fall.

The committee also will investigate concerns that students are rating the wrong classes, which Bain said anecdotal evidence verifies.

The CTEC committee will research the possibility that online evaluations draw biased samples.

Cristina Traina, chairwoman of the religion department, said the department has found that people who dislike classes are more likely to respond online than those who had neutral or positive reactions.

Traina said the department investigated one case in which a professor received online ratings that were lower than in-class ratings.

The department interviewed students from the class in detail and found that “really, people were highly in favor of the class,” Traina said.

Traina said she was especially worried that online CTECs draw especially negative written comments.

“I would say that when students are looking for a course, they look at the written evaluations, which are much less balanced online,” she said. “There are a higher proportion of negative responses.”

But Bain said results could be distorted either way.

“The evidence is not in but because response rates are substantially lower, one might hypothesize that (ratings) could be up or down if the (online CTECs) draw the most pleased or least pleased people,” Bain said.

The possibility of a negatively biased sample worries faculty members who rely on teaching evaluations for jobs.

Skogan said, “It doesn’t matter to me because I’m an old guy.” But he said some assistant professors are advised against using online evaluations.

Traina said the religion department advises graduate students, lecturers and non-tenured professors to use paper evaluations.

“If it’s not possible to get representative samples, these documents can be unreliable or harmful,” she said. “It does hobble us in judging teaching.”

Other faculty members said online administration did not affect their ratings.

Prof. Gadi Barlevy said he obtained enough “usable” responses from his Introduction to Macroeconomics class because it is so large.

“My situation might be different because even if there’s a drop-off in response rates, the average ratings aren’t going to be any different,” Barlevy said.

Traina, who usually uses paper evaluations, switched to online CTECs for one small class she taught. She said the online results were similar to the paper ones because the students felt obligated to respond.

“There was a sense of community that doesn’t develop in medium to large classes,” she said.

Hardy said there may be no differences between online and paper results regardless of class size.

She said she conducted a “very unscientific survey” comparing online and written responses for about 15 classes and found no significant differences in the responses.

Hardy also said there are many advantages to the online evaluations.

“Results are available to instructors as soon as grades are submitted, so they have them if they need to prepare for class (the following quarter),” Hardy said.

Hardy also said online evaluations don’t take up class time and are done in private, allowing students to write longer responses.

But some professors prefer paper CTECs for more personal reasons.

“I like to hold the surveys in my hand,” Moskos said. “I’m so old-fashioned, I only use a blackboard. I don’t like these smart classrooms. Except for the microphone, I’d be comfortable teaching in the 19th century.”