Taylor, S. S. (2011). "I really don't know what he meant by that": How well do engineering students understand teachers' comments on their writing? Technical Communication Quarterly, 20(2), 139-166. doi:10.1080/10572252.2011.548762

12:59, In this study, Smith tries to ascertain written comments' helpfulness in the field of engineering by analyzing comments from both engineering and English faculty. The study is mixed methods because she has to interview both the engineering and English faculty in order to see what they meant by particular comments, but then she interviews students too in order to see what they thought. So I say mixed methods because Smith has to analyze a corpus of comments quantitively, then cross reference the numbers with students and teachers intentions and understandings. 

She compared a lot of her results to an earlier study she had done: 

Smith , S. ( 2003a ). The role of technical expertise in engineering and writing teachers' evaluations of students' writing . Written Communication , 20 , 37 – 80 . [Crossref], [Web of Science ®], [Google Scholar]

What did she find? 

Focus

Reason

Mode

Development 

Content/form 

Mechanics 

This study identified two components of students' understanding of comments: recognition of the focus of the comment and comprehension of the reason for the comment.

Students incorrectly guessed a focus 18% of the time

 Unfortunately, the students were unable to venture any guess about the focus of 8% of the comments, and in fact, this “I have no idea” response was the most common response if the student did not recognize the focus correctly.

The comments least likely to be recognized were coherence (40%), validity (45%), and effort (0%, but the sample included only two such comments). These were three of the four least common focuses, so students were less accustomed to seeing and recognizing them. 

Coherence comments tended to be confused with development of ideas; 83% of the unrecognized coherence comments were interpreted as development. This finding suggests that students had difficulty recognizing the difference between level of detail (development) and relevance or internal consistency of details (coherence). 

This tendency of students not to recognize validity comments suggests that teachers should more clearly highlight the topic of validity comments as the truth of the information and perhaps elaborate more fully on the concept that the student needs to learn. Comments that included phrases such as “can't be right” or “not so!” were recognized by students as validity comments, giving the students a better opportunity to identify mistakes and learn from them.

Only 55% (388) of the 708 comments in the study were well understood, with both focus and reason recognized. (See Table 4, which shows the extent of understanding of the reason behind comments in each focus category.) If we include comments for which the student partially understood the reason, the percentage increases to only 68% (481) of the 708 comments in the sample. 

 Illegible handwriting was a pervasive problem in this sudy, as was also the case in Taylor's research of student mis-readings of faculty comments (2011). Over and over, students confessed that they could not read some of the faculty comments. Other "fixable" problems were failure to respond in a timely manner, failure to prioritize comments, failure to qualify comments (and consequently overgeneralizing context-specific advice), and, most importantly, failure to have a long-term view of revision. All of these can be addressed to some degree in faculty development workshops and in brown bag lunches.

For example, Mary Hayes and Donald Daiker (1984) determined that students frequently misinterpreted and even ignored the feedback they received. More recently, Williams (1997), whose research is described above, found that 25% of written comments were not understood as teachers intended, and in Taylor’s 2011 study, only 55% of 708 comments were well understood. These studies all focus their analysis on how students interpret each comment individually. 

Yet, this study focuses on perceptions, rather than on the comments

themselves, so it tells us more about how students process comments than

about the content of the comments. Simply assigning a writing task may shift

students’ attention away from comprehending material and toward the style

or clarity of their own writing. The idea that there may be confusion is suggested by Taylor’s 2011 study of how engineering students understood written

comments on their papers. In separate interviews, she asked teachers and their

students to explain what comments meant. These interviews indicate that

students may focus more on issues of expression in their writing rather than

logic or accuracy, even when they receive comments that teachers explained

in interviews were pointing to misunderstandings of material. For example,

a comment one teacher had written about something that was “completely

inconsistent with the rest of the paper” was described by the student as saying “we should have elaborated and explained more” (149). Another student

responded to a comment by saying “I think that is just a difference in the way

we’re writing”; however, the teacher explained that “[it is] a correction because

they said the computer was used to set the steam pressures. You’re not actually

setting the steam pressure” (149). Furthermore, when asked to explain comments that targeted the validity of information, students declined even to guess

18% of the time. 

so this is the one where students are not perceiving that teachers are calling the logic sequence of their ideas coherent, rather, the students are thinking that they're talking about their writing 

lit review 

Such research on student reception of response is less common than are studies of the comments themselves.

Regarding preferences, Straub (1997) found that students prefer comments that state the reason for an evaluation and that tie the reason to the student's ideas or words, thus confirming an established best practice. 

Similarly, Patchan, Charney, and Schunn (2009) found that students' comments on peers' papers tended to offer more praise and directive solutions than did faculty comments, and Patchan, Charney, and Schunn speculate that the student practices reflect their preferences. Kind of like the advice thing for basic writers, 

the disciplinary element of context has received little attention.

context has a strong effect on both the writing and the reading of comments

misc. 

But teachers in engineering aren’t going to grade with a contract, so validity comments matter/…

But that’s not you. You’re not asking whether it leads them to revise, you’re asking how it changes the quality of their writing, but would that have any implications for revision? 

Revision studies, 

Studies that measure the extent to which comments lead students to revise or to achieve improved learning outcomes have attempted to address such concerns by examining the effectiveness of comments as teaching tools. 

genre, script: it's interesting to think about how students need a script for interpreting comments, that is, the comments they don't recognize are the ones they aren't used to seeing

&&&

x