Evaluating Collaborative Projects and Evaluation Tools: Putting the Pieces of the Collaborative Puzzle Together

This article will suggest that collaborative projects, as common as they may be in the workplace (or even in the technical communication class itself), still pose real problems in the classroom. These problems must be explored if collaborative projects are to be truly effective teaching tools. One ef these problems is evaluation: ef the group, ef the individual student, and ef the processes ef collaboration and composing. Not surprisingly, evaluating the final product is relatively easy, but evaluating the process that produced it is much more difficult to do. This article first looks at two seminal models ef evaluation: the Purdue model (articulated by Morgan, Allen, Moore, Atkinson, and Snow), and the IMlyne State model (explicated by Beard, Rymer, and Williams). Second, the article looks at the evaluation tools these models use and, third, it assesses the models and the tools. In answer to the fundamental question as to whether these models ef evaluation (or, indeed, any model) can accurately gauge anything other than the final product, it is suggested that the tools can do so, at least in part, though more work needs to be done in the vital area ef evaluating the collaborative project.


100
Evaluating Collaborative Projects and Evaluation Tools assignment of varying length and complexity. Moreover, the sheer volume of work done on the subject of collaboration both in the classroom and in the workplace suggests that more and more researchers are turning their attention to this important topic in the field, as shown by the special issues of journals looking exclusively at collaboration [such as Technical Communication and recent ATIW anthologies].
The literature cites many reasons why collaborative projects are a pedagogically sound idea (see, for example, Parker). For one thing, collaborative projects take advantage of a wide variety of expertise and abilities, such as computing, editing, graphics, even the writing itself For another, collaborative projects are time-efficient; that is, they make the most oflimited time because several people are involved who can divide the tasks among them and share the load. Considerable time and effort can be saved since one individual does not have to be responsible for absolutely everything to do with the project.
Another compelling reason why collaborative projects are considered worthwhile is that the quality of the solutions proposed and, indeed, the quality of the document itself tend to be higher than that which a single person, working alone, can achieve.
But as common as collaborative projects may be in the workplace, in the classroom many problems associated with this type of work still remain, particularly since the classroom cannot exactly duplicate the world of work. The classroom imposes numerous restrictions, such as the space available where groups can effectively function undisturbed; the time available during class for groups to work together; and the resources available for teams (for example, access to computers and printers).
However, the major stumbling block, and the major reason why instructors introduce collaborative projects only to abandon them later, is evaluation. As instructors, we need to (indeed, we must) assess our students' work. Additionally, institutions still insist on each student's being assigned an individual grade, even if that grade has been calculated on the basis of group work. Satisfying these requirements can be a very real constraint on evaluating collaborative projects.
The problem of evaluation, then, is many-sided, and the following questions reflect the kinds of issues instructors face when considering collaborative projects as possible classroom activities and assignments. How do you assign an individual grade when the work has been done collaboratively? Has the student in fact developed his/her communication skills? How do you keep track of what an individual student is doing? In other words, how do you translate a group effort into an individual grade which will truly represent that student's mastery of the course? Indeed, how do you keep track of what a team is doing? And the most obvious question is, of course, how do you evaluate a collaborative project in the first place?
Clearly, these questions are not as relevant in a workplace setting, where the emphasis is on the group and maintaining its function rather than on the individual (Beard et al.,32). But they are of paramount importance in a classroom where evaluation is part of the job.
In the past, evaluation was based on individual assignments, so the process tends to be more linear, as shown below:

Returned
But in the class where collaborative projects are the norm, although an instructor still receives assignments and submits grades, the usual way of doing so needs to be modified. The linear configuration changes to one where several processes impinge on each other and, while they are relatively self-contained, as the circles show in Figure 2 below, they are nevertheless interconnected. Proc/ For this reason, because of the "interconnectedness" of the parts, Mary Lay compares collaboration to a web, where each strand is dependent on all the other strands. Thus, anything that affects one strand will affect every other strand in the web, and the entire structure must be able to withstand any force, such as conflict, which can threaten to destroy it. This image suggests the fragility at the same time as it suggests the resiliency of the collaborative effort, and instructors must be aware of this tension.
Collaboration can also be compared to a multi-piece puzzle with interconnecting parts where, if one piece is missing or faulty, the picture is incomplete. This analogy illustrates just how difficult it can be to introduce, implement, and administer collaborative projects in the classroom, where all the pieces must somehow be linked together to form a whole. Doing this can be an onerous, time-consuming task for the instructor.
On the one hand, little of the research on collaboration offers concrete examples oflww to introduce group projects, and there are few illustrations ofhow to evaluate the group's finished product and the group's collaborative effort. Evaluating a collaborative project is a very complex process because an instructor must somehow assess the process that produced the product of collaboration, the document, which must also be evaluated in its turn. For the evaluation to be fair, an instructor must grapple as well with evaluating both the group and the individual members of that group.
All of this is even more complicated by the lack, on the one hand, of tested and fully developed assessment models (Beard et al.,31) and, on the other, of a well-developed model of the collaborative writing process itself (33). These kinds of models are necessary if the evaluation process is to be less problematic. As Beard says, models for such an assessment system have, to date, been fairly "exploratory" (31) simply because collaboration itself is still a fairly new thing.
But two useful models do exist and are worth looking at. One is the Wayne State model, explicated by Beard, Rymer, and Williams, where the focus is more on interaction, and the other is the Purdue model, articulated by Morgan, Allen, Moore, Atkinson, and Snow, where the focus is more on writing. Both models can be considered seminal works when it comes to the problem of evaluating collaborative projects because both models are quite detailed in their study of collaboration and evaluation. Secondly, these models provide a clear set of objectives to be met and a clear set of evaluation tools with which to meet those objectives. Finally, and perhaps most importantly, these models have been tested in the classroom and have been shown to be workable for any type of collaborative assignment the instructor might choose.

Objectives of the Assessment Models: Theoretical Constructs
The theoretical basis of collaborative projects, while still relatively new, nonetheless is an important starting point when we consider either collaborative projects or their evaluation. These models assess both the product of the collaboration and the process that generated it, and that means both the group itself and the individual member must also be evaluated. As well, both these models are related to the social theory of learning, a two-pronged theory that stresses, first, that students must be active and participating members of a discourse community and, secondly, that "communal learning, especially about composing," has value (Beard et al.,29). Out of this theoretical basis, the two models establish the objectives to be met by any model for collaboration.
I have identified eight objectives which these models propose for any assessment process: [ 1] An assessment model evaluates the individual's skills at interaction and at facilitating both the group process and the collaborative process. That means that students: always give their best to the project and the group; become more "active, responsible, and cooperative participants through small-group discussion" (Beard et al.,29); demonstrate a responsibility to the group to the group and the project.
[2] The evaluation model encourages the reward of individual effort, and students are likewise encouraged to reward others for their efforts. In addition, the model rewards the group's performance (Beard et al.,30).
[3] Students develop their communication skills, which would include interpersonal skills, oral skills, writing skills, and management skills. In other words, the goals of the technical communication course itself should be enhanced. To this end, students demonstrate a willingness to help others develop their skills as communicators; for example, they are sensitive to others and show respect for others.
[ 4] Students become more aware of the writing process itsel£ Morgan et al.argue that, because of the group's frequent discussions about it~ the entire writing process becomes more "externalized" for individual students (1987,25). The evaluation process likewise assesses the writing process and the individual student's involvement in each step of that process, such as the planning, drafting, and revising of either the whole document or specific sections of the document.
[ 6] The model provides feedback on both the product and the process involved leading to that product; that is, the way the group works together and how the group eventually produces the document. However, instuctors also provide some kind ofjudgment on a student's work, both as a writer and as a member of a group, and this is unique to the classroom, not the workplac::e. So, to do so, the instructor assigns both individual and team grades.
[7] The model gives the student some sense of control and power over what happens to him/her in the group. Therefore, each student on a team plays a role in the evaluation process.
[8] The model tries to encourage students to value the collaborative project beyond just the final product; that is, the model encourages students to function successfully as a group. To that end, grade incentives help the collaborative process to work.
These objectives stress the process of collaboration, and part of that process is the generation of a product, so any assessment must include not just the individual's contribution but also the quality of the product and the performance of the group itsel£ As a result, any document produced by the team would receive a team grade. Both models emphasize the importance of recognizing the group, and the team grade helps to do this. Individual grades would be assigned for other components, such as recognition of an individual's contribution, or recognition of the planning and revision that went into the final product (Morgan et al.,1987, 24).
Clearly, evaluating collaborative projects is no easy matter. For all these goals to be achieved and for the assessment to be equitable, the assessor needs tools by which to evaluate all these different factors of the individual, the group, the product, and the process.

Evaluation Tools: Practical Constructs
These evaluation tools, which Morgan et al. call "instruments" (1989, p. 84 ), form the basis of the evaluation strategy. Indeed, the tools provide fairly complete information about group processes, such as planning, writing and decision-making, and about individual contributions and participation (Morgan et al., 1989, 84). In both models, all the tools taken together help to determine a student's individual grade for a collaborative project.
The following diagram shows the various evaluation tools as parts of the puzzle.

Figure 3: Evaluation Tools
An analysis of each of the tools will show how they help an instructor assess a group and the individuals in that group. First, I will describe the tool and then show how it is used in the evaluation process.
[1) The Log/journal Many scholars have commented on the value of the student journal, among them Goldstein and Malone (1985, 24). Specifically,journals become like a "private conversation" between a student and the instructor (Morgan et al., 1989, 85). Rymer and Beard compare it to the instructor's "reading over [the student's] shoulder" (127). But, for the journal to serve its purpose, it must always be a confidential record between student and instructor. As well, it must figure in the calculation of a student's grade.
The journal becomes a way for the student to communicate his/her perceptions about the team and its functioning (or dysfunctioning) as well as a chance to judge how well the group is working now, or will work over time. For example, a journal entry can describe what happens at meetings, and comment on individual contributions to the discussion and the team's decision-making.
The Purdue model (Morgan et al.) incorporates two important goals into its use of the journal. First, the journal is designed to help students learn the value of keeping a detailed record of a project (1989,85) and, secondly, the journal gives the instructor important information about what the group is actually doing. Journal entries, for example, will signal potential problems or, alternatively, give the instructor an insight into what students consider important issues on the team (85). Students are encouraged to just "free write," so the writing itself is not graded, although instructors can make comments and offer suggestions. What the instructor will especially watch for is the "completeness and substantive nature of the content" (85), since the log is an important component in the evaluation of the collaborative process (93 ). This evaluation is ongoing since the log is submitted to the instructor every time a group submits a revised document.
Similarly, the Wayne State model uses the journal so that students can identify problems associated with group work (Rymer and Beard,118); in their entries, students "describe and analyze the interaction process of the team, focusing on their own participation" (124). Here, too, the journal is not assessed as a finished product but, rather, it provides important information about the process of collaboration. Students must demonstrate that they have been thoughtful and have made the effort when it comes to exploring the group's interaction.
Particularly noteworthy in this model, however, is the stress on how, and how much, the student has "used writing to develop greater self-awareness (128), including awareness of group roles and skills", becoming a "professional participant in groups," facilitating the group process, and developing interpersonal skills (127).

[2 J Conferences and Observations in the Class
Clearly, observing groups in action can only be done during a class, and only when time has been set aside for the teams to hold their meetings. As well, observing a group in action presupposes that the students can behave naturally in their groups and that the instructor has a keen insight into group dynamics. Both of these potential pitfalls, however, can be alleviated by experience and familiarity with the other people in the group. These restrictions aside, observation can nonetheless give some clues as to the group dynamic, although to be useful other tools must be used in conjunction with observation. To some extent, at least, both the Purdue model and the Wayne State model use this tool.
Conferences can: be scheduled on a regular basis, or they can be arranged as needed. But conferences can be a form of intervention in the running of the group. Indeed, Morgan et al. use conferences to solve problems detected either in the student logs or sometimes during classroom observation. It is important to note that instructors cannot be seen as problem-solvers; the group should be the problem-solvers. But the instructor can offer advice and suggestions; in other words, serve as a "facilitator" (1987, 23).

[3] Student Assessment Sheets/Peer Evaluations
Evaluating their peers in the group should be seen as a safeguard for individual students. Not only do peer evaluations help students to feel that they have input into the evaluation process, but also these sheets help to prevent poor performance and ensure increased commitment for the project and the team. The goal of the assessment sheets is the evaluation of the performance of other team members; for example, a team member will comment on another group member's attendance at team meetings, contributing ideas to the group, helping with writing and revision, as well as how well a student has performed as a member of a group (Morgan et al., 1989, 87-88).
In the Purdue model (Morgan et al.), the individual student evaluates each team member in several categories according to a 3-point scale of"high," "medium," or "low" (1989,92). In the Wayne State model (Beard et al. and Rymer and Beard), peer evaluations have two components: first, the confidential memo, which describes who actually did what (Rymer and Beard,118), the team's process and the contributions of each to the process (124); and, secondly, the peer evaluation sheet, where individual students assess their own contribution and also assess other team member's contributions (124). In this model, students use a 5-point scale, where "1" is the lowest and "5" is the highest.
[ 4 J Response Sheets/"Working Papers"/"Paper Trail" Morgan et al. ( 1989, 86, 91) ask students to submit drafts of an entire document or assignment. Other teams, as well as the instructor, then respond to the draft, on what are called "response sheets," according to five categories of Purpose, Development (or Content), Organization (or arrangement and format), Style, and Mechanics (spelling, punctuation, capitalization, and the like). The Wayne State model adds a twist: here, students are not responsible for a discrete section of the document. Rather, all tasks are rotated so that no one student "owns" a section (Rymer and Beard,118). The group as a whole participates in the planning and reviewing, but it is individuals who assume the task of drafting the purpose, analyzing the audience, generating ideas, drafting outlines, writing and editing drafts. Individuals may even prepare drafts for group discussion, and the model refers to these as "working papers" (121). These tasks are then rotated, so that no single person ever has to do more than another.
But, because of this rotation, each student must keep a record of all drafts, notes, and outlines (that is, everything) that he or she has done for the team and for the project. These are then all submitted as part of the "paper trail" for the document, and one copy must be submitted to the instructor for credit.

[5] The Final Product/ Final Evaluation
Finally, there is the document itself, and both models assign a team grade for the finished product. In the Purdue model, the evaluation of this final product uses the same categories as the response sheet has used; each category is marked and awarded so many points. The finished product includes a "packet" for evaluation, which is submitted at the same time as the assignment is (Morgan et al., 1989, 87). This packet includes (87): the first draft of any collaborative assignment that has been submitted for the purposes of feedback; [b] the final draft; that is, the version that has been revised on the basis of the comments received; [ c] the student logs that contain each individual's "personal account of a group's progress and process"; [ d] student assessment sheets, where each student evaluates all the other team members' performance in the group; [ e] response sheets that another group has submitted with the team's first draft, and the feedback upon which the team can then revise the team draft.
In this model, all students receive the same grade for the final product, but they receive individual grades for the process. As can be seen in the sample below, the sum of both the document component and the process component will then be the individual's final grade for the collaborative assignment (Morgan et al., 1989, 93):

Figure 4: Sample Evaluation Sheet
Assignment: DOCUMENT (60 points) 2 Purpose (10) Development (15) Organization (15) Style (10) Mechanics (10) PROCESS ( 40 points In the Wayne State model, the components for the final evaluation of the collaborative process include: the working papers, the peer evaluation of other members, the log, and the confidential memos. These four components would receive half the final grade, while the final report itself would receive half the final grade.

Assessment of the Models and the Tools
As can be seen, both of these models rely on a very complicated array of evaluation tools. What strikes most instructors is the sheer "messiness" of the practice as opposed to the relative "neatness" of the theory (Weiner,60), and that translates into a very real need for providing students with lots and lots of guidance, as can be seen in the directions for a meeting schedule (Rymer and Beard,123 ), given in the Appendix.
The points listed here are an example of one way to distribute points using this sheet. The number of points per category actually awarded for any assignment is determined by the instructor. Every piece of information the students receive on this kind of collaborative project must, of necessity, be detailed. And anything this detailed requires preparation time for the instructor. In my experience, it also means a constant process of re-assessing and polishing every piece of information, including assignments, that the students receive.
Another factor is what Beard et al. calls the "massive" amounts of material (34) and what Morgan et al. refers to as the "overwhelming" volume of papers to be reviewed ( 1989,89). And this is true for even the shortest, most straightforward collaborative project. The result is that the instructor can soon be buried under this mass of paper, all which must be read, assessed and assigned a grade.
Clearly; then, two critical points must now be considered, and we might as well begin with what are really negative features of the collaborative project in the classroom: first, the time involved for any instructor attempting to introduce and administer a team project; and secondly, the procedure to be used to convert prose into a grade or, in the case of these models, into points.

[1 J The Time Factor
Morgan et al. claim that collaborative projects do not require more work, just adijferent kind of work. And Beard et al. claim that determining a student's "process" grade takes no longer than it does to grade a final report. In my experience, I have found that marking the written document, the final report produced by the team, takes on average one hour. I use two of the tools, namely, a "paper trail" component and a "peer evaluation" component, to determine the "process" grade, and each takes approximately one hour. Thus, these components of the "process" grade actually equal two hours, or double the time it takes to mark the written document.
But even if fewer components were used and the time required for marking both the report and the "process" were in fact equal, it is still the instructor who must somehow construct, grasp, and grade the larger picture; that is, the instructor must be the one who can (and indeed who must) put all these pieces of the collaborative puzzle together. That means reviewing and synthesizing all the data; reconstructing the group's history; and determining the role played by each team member. As Beard et al. state, this translates into "building up, layer by layer, a view of the group" (34), a very daunting task, to say the least. Added to the burden of the sheer paper work involved is the inevitable reliance on what students have to say about the project and each other. So, in addition to the time factor is the factor of the procedure to be used to evaluate all of these components.

[2] The Evaluating Procedure
There is no easy or clear-cut way to convert prose into numbers (or points), nor can the processes of writing and collaboration be transformed into a neat formula. At issue here is the question of subjectivity, of whether this entire evaluation process is, or could be perceived to be, far too reliant on "iffy" methods and unreliable tools. Neither of the models provides much help or guidance. Certainly the need for flexibility is great, but so, too, is the need for definition. We do, after all, work in a learning environment where grades are expected.
But perhaps the most damning criticism to be made is that an instructor must still rely on artifacts, response sheets in some form or other, in order to gain insight into a process. Aside from locking a team in a room and then observing them through a one-way mirror, much as we do when we conduct psychological studies, there is no effective way for an instructor to watch a group in action and see firsthand exactly what the group dynamic is. Two conditions must be met, difficult to achieve in a classroom setting, in order for instructors to do this: first, students must behave naturally during group work in class; and secondly, the instructor must be very perceptive and knowledgeable about what to look for (Goldstein and Malone,24). The bottom line is that instructors must inevitably rely on what students say about the group and about the group members.
Another factor, just to complicate things further, is the many intangibles at play in any group setting, things like attitudes or gender (to name just two), which affect the group process. Measuring these, let al.one trying to assess their impact, is a very real problem.
However, having shown how vulnerable the evaluation process can be to these intangibles, I can now move on to show how these tools, as part of a thoughtful strategy, can offset, if not eliminate, the effects of these intangibles on the evaluation process.
One of the ways these tools minimize the impact of such intangibles is their flexibility. Any evaluative model can use any or all of the tools, and the value assigned to each tool can vary, according to the needs of the instructor and the goals of the course. Of importance, too, is that there is always overlap; an instructor will need one of the other tools simply to "corroborate the data from the different sources" (Beard et al.,35).Taken together, they help to put the various pieces of the collaboration puzzle together.
Another reason why the tools can help an instructor form a fairly complete picture of the group is the participation of the individual student in the evaluation. Each student's individual responses represent that individual's control over the evaluation process. The likelihood of a student's providing the necessary information is therefore greater than it might be, were the responses to be less critical to the evaluation strategy.
Of course, there is always some arbitrariness about the evaluation process. It is hard, after all, to separate or measure every kind of behaviour in a group, and it would be unrealistic to expect to do so (Beard et al.,34). Additionally, the relationship between collaboration and composing has not been that well-defined. But the tools can work because, first of all, discrete actions such as brainstorming or outlining or writing can be separated and, second of all, participation in both the composing and the collaboration processes is encouraged; indeed, it is expected.

Conclusion
In a way, there is no conclusion that can be firmly and finally made about collaborative projects. They are as various as people in classrooms. But one can say that there are many benefits to introducing collaborative projects into the technical writing classroom, not the least of which is the fewer number of written assignments to mark, although the procedure for evaluating the reports that must be reviewed can be very involved and time-consuming. There are also other benefits, especially for the students, who have the chance to practice the principles taught in a technical writing class, and who can learn valuable skills, such as interpersonal skills, time management skills, and oral communication skills.
Nevertheless, much more needs to be done, especially in terms of establishing models to evaluate the "group process" and of defining more precisely what these two processes of collaboration and evaluation really involve. But these two seminal models discussed here provide at least a starting point for the instructor who is willing to try putting all the pieces of the puzzle together.

Sample Meeting Schedule
Meeting 1-Discuss the assignment, the case or communication problem, and allow time for all members to meet. Make sure everyone has read over the entire assignment and knows what is expected of the team and of each participant. Discuss the case and make sure everyone understands it. Establish your group as a working team, making plans for meeting, exchanging telephone numbers and schedules, and learning to know each other. Assign each member to prepare a working document analyzing the audience and purpose for the report and planning the message and major points for the next class meeting. (For all subsequent meetings, each member should bring enough copies of his/her working papers for all members, plus one copy for the instructor.) Meeting 2-Define the objectives and plan your document. Develop your purpose, analyze the audience and context, and determine the group's stance as "writer." Then determine your overall strategy in the report and set up some guidelines you will follow.
Finally, decide what you want to say, develop your ideas, and organize the information into an overall plan. For your next meeting, assign each member to draft a sentence outline with supporting evidence for the whole document.
Meeting 3-Exchange individual outlines for the whole document, discuss the differences among them, and create a consensus team outline. (One member should wordprocess the team outline for all members after the meeting and distribute it as soon as possible.) For the next meeting, assign each member to prepare either a rough draft or detailed outline of some section of the report; assign all segments except the beginning and the ending. (Attempt to assign these drafting responsibilities equitably, subdividing long segments and allotting several short segments to one person.) Technostyle Vol. 12, No. 11995 Spring Meeting 4-review the individual drafts or outlines for the report segments, working out any problem in meaning, overlapping information, or lack of congruence. Rationalize differences and create a new consensus team outline that notes exactly what will be in each section and exactly how the various rough drafts will be incorporated into the report. For your next meeting, assign each member to complete a fully revised draft of a section initially drafted or outlined by someone else.
Meeting 5-Review the discussion sections and refine plans for revising. After a thorough discussion of the views of the writers, exchange these sections so that each drafter's work will now be revised by someone else. Plan the report's overall formatting, revising, editing. For the next meeting, assign these tasks to various individuals, and assign all members to prepare a rough draft of the beginning and the ending of the report.
Meeting 6--Review the drafts of the beginning and end of the report and the overall plans for unifying the document (e.g. through formatting). For the next meeting, assign individuals to make consensus revisions as necessary and work on any weak sections of the report.
Meeting 7-Review the revised, wordprocessed drafts of all sections of the report. Correlate the pieces of the report and make decisions about overall revision. Set objectives for final revisions, and assign revising, editing, final formatting, and production tasks, such as merging files.
Meeting 8--Review the final manuscript and make any final editing changes. All members should review the final document. (The whole team could meet at the computer and have one member enter changes that meet group consensus.)