Gabelica et al

Catherine Gabelica^a, Piet Van den Bossche^ab, Mien Segers^a, Wim Gijselaers^a

Article received 15 January 2014 / revised 11 February 2014 / accepted 5 June 2014 / available online 18 June 2014

A great deal of work has been generated on feedback in teams and has shown that giving performance feedback to teams is not sufficient to improve performance. To achieve the potential of feedback, it is stated that teams need to proactively process this feedback and thus collectively evaluate their performance and strategies, look for alternatives, and make clear decisions about ways to tackle their task. This concept of team reflexivity has been commonly described as a sequence of behaviours, which relative importance has not been demonstrated. Further, empirical research investigating the dynamic aspects of reflexivity has been scarce. This study sought to explore how reflexivity evolves over time and at which moments of the team interaction it is related to team performance. Thirty-two student dyads participated to a cognitively complex task (flight simulation) over four performance episodes comprising action phases followed by transition (feedback) phases. High interdependence between participants (pilots and co-pilots) was ensured through the distribution of complementary knowledge in the dyads. The results showed that teams seldom engaged in full cycles of reflective behaviours. When looking into individual behaviours, teams exhibited more reflective behaviours during action over time, while their reflective behaviours during feedback did not change, demonstrating a suboptimal feedback processing as time goes by. Additionally, it was demonstrated that teams were capable to learn from their past and act upon feedback to better subsequent team performance but also that initial performance acts as a trigger to future reflective behaviours.

Corresponding author: Catherine Gabelica, Department of Educational Research and Development, School of Business and Economics, Maastricht University, Tongersestraat 53, 6211LM Maastricht, The Netherlands, Phone: +31 43 38 82587, Email: C.Gabelica@maastrichtuniversity.nl

Small group work has gradually progressed to being one of the dominant approaches in the domain of learning and instruction and professional development (e.g., Kirschner, 2009). Collaborative learning is one of the most successful and widespread instructional practice implemented in schools and universities (e.g., Dillenbourg, 1999, Johnson & Johnson, 1992). Similarly, work teams have become a central element in the functioning of organisations in many domains (e.g., health care, military, and aviation) (Salas, Stagl, Burke, & Goodwin, 2007). Both professional teams and learning teams face similar challenges inherent to collaboration and joint understanding (Barron, 2000; Järvelä, Volet, & Järvenoja, 2010). Specifically, in both environments, interdependent team members need to interact and communicate effectively, share knowledge and experiences, and capitalise each other’s skills and resources to successfully complete a common task (e.g., Johnson, Johnson, & Stanne, 2000; Salas, Dickinson, Converse, & Tannenbaum, 1992). Crucially, recent work has shown that teams that engage in team learning processes and learn how to work effectively are more likely to succeed (e.g., Dochy, Gijbels, Raes, & Kyndt, 2014; Van der Haar, Segers, & Jehn, 2013, Veestraeten, Kyndt, & Dochy, 2014). Team learning has been defined as “an ongoing process of reflection and action’’ (Edmondson, 1999, p.353) during which teams reflect on their own prior activities and consequently plan adjustments for future practice (see Decuyper et al., 2010, for a review). Although scholars in these areas have tended to remain isolated within their own disciplines despite obvious overlaps in research interests, they generally agree that team learning processes do not occur naturally (Johnson & Johnson, 1992; Rummel & Spada, 2005; Sims, Salas, & Burke, 2005). The awareness that not all teams learn, and as a consequence may reach substandard group performance, raises the need to outline deliberate interventions to build learning in teams. More and more, new research interests focus on what can be done to leverage learning in teams and improve their performance (e.g., Decuyper, Dochy, & Van den Bossche, 2010; Salas, Stagl, & Burke, 2004). Despite these renewed efforts, it seems that potential leverage points (such as training or the provision of feedback) calibrated for teams need to be better specified and validated (Kozlowski & Ilgen, 2006).

Giving teams feedback on their team process and performance has been identified as a leverage point that shapes team learning and can improve team performance (Gabelica, Van den Bossche, Segers, & Gijselaers, 2012; Johnson & Johnson, 2002; London & Sessa, 2006; Phielix, Prins, & Kirschner, 2010). In school and beyond, teams need feedback to monitor and regulate their work (Hattie & Timperley, 2007). Previous theoretical work on feedback provided by external agents at the team level of analysis (e.g., Goodman, Wood, & Hendrickx, 2004; London & Sessa, 2006) suggests that to achieve changes in team learning and performance, teams need to process received feedback, be receptive to this feedback, understand its value, and actively engage in collaborative activities during which they use feedback cues to make improvements. Nevertheless, empirical work on the value of active feedback processing to the mere reception of feedback in teams has considerably lagged behind theoretical development (Hattie & Timperley, 2007). This feedback processing has yet to be empirically examined (Gabelica et al., 2012). More specifically, the more interesting question about feedback effectiveness is rather how learning naturally happens during the team feedback process and how effective are these learning processes (e.g., Adcroft, 2011). Moreover, previous work in both team and collaborative learning research leaves much about the dynamics of feedback processing in teams unspecified, such as 1) how do teams respond to repeated (external) feedback in dialogue over the course of ongoing activities, and 2) when (i.e., at which point in time) are these behaviours related to effective learning and performance. There is a general agreement across disciplines that we should consider feedback loops in which behavioural changes resulting from each cycle are inputs in cycles that follow (e.g., Soller, Monés, Jermann, & Mühlenbrock, 2005) but this is rarely reflected in research designs (e.g., Ilgen, Hollenbeck, Johnson, & Jundt, 2005).

Concerning how teams process feedback, we propose that they do so by performing shared reflective activities, that is by collectively discussing and reflecting upon their functioning (e.g., Schippers, Den Hartog, & Koopman, 2007). These activities are core building blocks of team learning. Specifically, it has been shown that reflective teams evaluate their performance and strategies, look for alternatives to consider situations, and make decisions about new ways to tackle their task. The concept of team reflexivity, as proposed in organisational psychology, mirrors these activities (West, Garrod, & Carletta, 1997). In educational settings, generic forms of intra-group reflection such as collective/social metacognition (McCarthy & Garavan, 2008), reflection (Edmondson, 1999), collaborative reflection (Morris & Stew, 2007; Yukawa, 2006), peer reflection, or reflective self-explanation (Rummel, Spada & Hauser, 2009) have been increasingly used. This recent research area is an extension of the work on individual reflection or reflective practice (e.g., Boud, Keogh, & Walker, 1985) that adds interactions and communication with peers to the learning process. Many authors agree that team reflexivity (in any generic sense) allows teams to reach a more accurate understanding of their task and, as a result, better performance (e.g., McCarthy & Garavan, 2008; Schippers, Homan, & Van Knippenberg, 2013). Although the very recent research strand on team reflexivity acknowledges the importance of the dynamics of team performance when considering team reflexivity, the empirical work is only beginning to consider under which circumstances team reflexivity relates to changing performance, but not in contexts with systematic performance data on which to reflect (Schipper et al., 2013). Across disciplines, external and specific feedback is not systematically part of the reflective process while it is usually agreed that reflection can only occur if people have accurate knowledge about their current and desired learning state (Hattie, 2013). Also, the relation between time and timing of reflexivity and team performance remains in question, such as does reflecting right from the start of a team activity help the team get started and allows later success or does sustained reflection after events later in a team’s life also matter for sustained performance?

Thus, when teams process feedback appears as a gap in both feedback and reflexivity research. Previous research on feedback and team performance suggests that feedback effects are not static but dynamic (e.g., McGrath, 1993); it cannot be understood as a single-cycle linear path from inputs (e.g., feedback) through outcomes (e.g., team performance). In the same vain, how teams learn from feedback should also be considered with a dynamic glance (Ilgen et al., 2005)

The purpose of the present study is therefore to address the above-mentioned gaps by shedding light on how reflective behaviours relate to performance over a period of time in a complex, fast-paced, and high-workload situation in which two individuals with distributed information have to keep on learning to achieve success. Specifically, the following questions are explored: 1) how do teams naturally overtly reflect when provided with feedback depicting their performance, 2) how do reflective behaviours grow over time during and/or after action, 3) how does the timing at which team reflexivity occurs relate to performance?

Prior to addressing feedback in team settings, it is critical to briefly solicit input from multiple disciplines to better understand how the much more substantial body of research on feedback given to individuals have shaped the feedback concept in teams.

In the learning sciences, feedback is an instructional practice that is expected to enhance motivation and learning (Mulder & Ellinger, 2013; Shute, 2008). Learning scientists have acknowledged feedback as a key characteristic of quality teaching decades ago in non-team settings (e.g., Mory, 2003; Shute, 2008, Yang & Carless, 2013). Much of the extensive work on feedback given to individuals has come to two main conclusions: 1) learners should be given feedback containing learning information (e.g., Duijnhouwer, Prins, & Stokking, 2012; Gibbs & Simpson, 2004) and 2) researchers should consider feedback from the perspective of the feedback receiver and thus incorporate the uptake and the receptivity of feedback in the feedback process (e.g., Boud & Molloy, 2013; Eva et al., 2012). This also introduces the idea of a feedback process that goes beyond the provision of feedback (Mulder, 2013). Since feedback is traditionally part of instructional programs, the drawback of multi-component interventions is that it is not always possible to assign behavioural changes to feedback interventions (e.g., van der Pol, van den Berg, Admiraal, & Simons, 2008). Further, most studies concern primary school and high school students (e.g., Johnson & Johnson, 1993), which raises the question of the generalizability of findings to higher education or workplace (adult) learning.

By contrast, organisational, social, and behavioural psychology have incorporated feedback delivery in many (semi)experimental research and extensively investigated its added value with or without other components (such as goal setting) to human performance (e.g., Kluger & DeNisi, 1996) while the feedback process has largely remained a black box. Furthermore, feedback is often mere “knowledge of performance or results” (e.g., performance data of a company or score on a simulation game) instead of elaborated informational feedback (e.g., Austin, Kessler, Riccobono, & Bailey, 1996). The learning value of feedback seems to be a consistent omission. Moreover, this research tradition has primarily focused on post-secondary levels.

Taken together these disciplines have given rise to a new question transcending the simple question of whether or not feedback is truly effective: how and under which conditions feedback improves learning and performance. This concern has been echoed in the relatively smaller research strand on feedback to teams.

Feedback at the team level of analysis is defined as the communication of information provided by (an) external agent(s) concerning actions, events, processes, or behaviours relative to task completion or teamwork (Gabelica et al., 2012, London, 2003).

It is widely accepted that feedback can provide teams with accurate information on their performance and may steer, motivate, support, and reinforce future team behaviour. Feedback is considered as a leverage point in the team's development of a collective view of expectations and awareness about its behaviours, capabilities, and skills (London & Sessa, 2006; Prins, Sluijsmans, & Kirschner, 2006). Research in collaborative learning environments has highlighted that feedback has the power to draw the team’s attention to specific aspects of its task and hence encourage task-related discussion (Johnson & Johnson, 2002). In the workplace, feedback can also serve as a motivational trigger. For example, Scott-Young and Samson (2009) showed that providing teams of managers with performance feedback reinforced teams’ confidence in themselves and in turn, their performance.

Despite many potential benefits of feedback delivery, a recent review by Gabelica et al. (2012) integrated findings from fifty-nine empirical studies investigating the effects of feedback in teams in educational and professional settings and showed mixed results. Approximately one third of the studies did not find support for its expected positive effect on performance. For example, in a field experiment, Jung and Sosik (2003) found that giving feedback to teams performing decision-making tasks had positive benefits on group members’ collective confidence (i.e., collective efficacy and group potency) but not on team performance. Based on these inconsistent results, analogue to feedback research in non-team settings, Gabelica and colleagues (2012) concluded that the key question of whether team feedback is effective depends on the conditions under which feedback is given, and not only on feedback as such (e.g., its quality).

Based on educational research, it can be argued that in addition to factors related to the feedback giver and environment, feedback receivers have a critical role to play. Research on team feedback suggests that teams given feedback will only change if they perceive a learning need and opportunity and if they attend, interpret, and act upon feedback (e.g. London & Sessa, 2006; Phielix et al., 2010). In other words, teams need to proactively process the content of feedback, and thus invest time and effort into actively building content-oriented reactions if we expect visible changes in the way they perform. Yet, how teams process information cues contained in feedback, and thus what specific processing behaviours and activities are dynamically related to performance remains largely unknown (Gabelica et al., 2012). Although there are few studies on peer feedback exploring the role of feedback receivers during the feedback process in teams (e.g. Prins et al., 2006), the interconnections between uptakes of feedback receivers and ongoing performance are still unclear. Also, since the success of feedback in terms of an effective uptake from the receivers depends at least partially on the feedback quality provided by others, studying the uptake of standardised feedback (i.e., of constant quality and constant source) would allow us to isolate the learning effects of providing feedback.

In sum, there seems to be an agreement that reaching an intersubjective understanding of the content of the feedback in teams by discussing what can be learned and worked out from past experiences is a potent factor augmenting feedback effectiveness (e.g., Boud et al., 1985). Despite a lack of direct evidence establishing the benefits of feedback processing behaviours, the consensus appears to be that the construct holds enough potential to warrant further investigation. The recent research strand on team reflexivity depicting the extent to which teams reflect upon and modify their functioning informs our understanding of these processing behaviours (Schippers et al., 2013).

In pedagogy, individual reflection- or reflective practice- can be traced back to the early 1900s (Dewey, 1910, 1997) but has been introduced more extensively into the field of professional learning by Schön (1983) as professionals’ critical consideration of what they are working on while they are working on it. On a simple level, one can consider reflection in the past, present, and future tense. Schön refers to ‘reflection-in-action’ as analysis in the present tense (i.e., reflection on the spot) and ‘reflection-on-action’ as analysis in the past tense (i.e., review of past actions). Killion and Todnem (1991) underlie a lack of forward thinking implicit in Dewey’s work and propose that reflection should also inform future action. Thus, they added ‘reflection-for-action’ as reflection oriented towards the future (i.e., identification of guidelines to follow to succeed in the future).

Reflection as an individual critical thinking process has been recently extended to a view of reflection as a collaborative critical thinking process consisting of cognitive and social interactions between two or more individuals who examine their experiences to construct novel intersubjective understandings (Boud et al., 1985; Yukawa, 2006). As such, it is considered as a core team learning process. Work on team learning has demonstrated that collective learning can be realised through iterative sequences of action, reflection, and implementation (Dochy et al., 2014; Edmondson, 1999). In the learning sciences, there are multiple labels denoting this concept of reflection at the team level. For example, the following terms have been used: collaborative reflection (Morris & Stew, 2007; Yukawa, 2006), peer reflection, reflective self-explanation (Rummel, Spada, & Hauser, 2009), or collective or social metacognition (McCarthy & Garavan, 2008). In small group research, principally one label “team reflexivity” has been introduced by West (1996) as a set of collaborative reflective behaviours and activities during which the team objectives, strategies, and processes are discussed openly. We use the term “team reflexivity” throughout this paper as a unique label for reflection at the team level.

Originally, the concept of team reflexivity has not been explicitly connected to the feedback process. However, it is generally acknowledged that reflection is enabled by feedback to ensure accuracy in learning (Hattie, 2013). As a result, team reflexivity can be conceptualised as ways teams collectively try to extract meaning and cues for future behaviours from received feedback, generate intentions and plans, and ultimately decide to act upon feedback. Thus, when performance feedback that is merely evaluative is given to teams, the process that follows this feedback moment might be shared reflection on the task and the team process. The underlying assumption is that team feedback gives goal-oriented information but teams are still responsible for its mindful uptake. It can be argued that reflective teams consider reasons, rationales, and evidence for this evaluation of past performance, weigh alternative perspectives to construct a better understanding of their collective experience that, in turn, better guides their future action (Yukawa, 2006).

Three behaviours that reflect complementary dimensions of team reflexivity can hence be derived from previous work on team reflexivity across disciplines (e.g., Schippers et al., 2007; Yukawa, 2006): (a) evaluating present and past performance and strategies, (b) looking for alternatives, and (c) making decisions. Evaluating refers to team members reviewing their goals, performance, strategies, and possible reasons behind success or failures. Looking for alternatives occurs when teams make an inventory of possible ways to achieve the task. Finally, making decisions consists of clearly stating a decision about how to handle the task differently and acting upon it. Evaluating and looking for alternatives reflect the capability of the team to be self-aware of its behaviours and the necessity to make changes. According to Schippers and colleagues (2007), this is necessary but not sufficient to engage in change. Teams also need to implement the adapted actions. This is reflected by our conceptualisation of ‘’making decisions’’ that depicts both the ‘’intention to act’’ and ‘’carrying out the decision’’. Hence, this suggests a time-ordered sequence of reflective behaviours that might constitute reflective cycles, although no empirical work supports the necessity of full three-phase sequences.

Overall, reflective teams have the ability to uncover why they succeeded or failed, solve misunderstandings, and correct their future approaches as new challenges emerge (Tschan, Semmer, Nägele, & Gurtner, 2000; Wills & Clerkin, 2009). As a consequence, team reflexivity has been recognised as an important contributor to effective collaboration and performance (e.g., Kramarski, 2004; Rummel, Mullins, & Spada, 2012; Schippers, Den Hartog, Koopman, & van Knippenberg, 2008; Tjosvold, Tang, &West, 2004; van Ginkel, Tindale, & Knippenberg, 2009). However, in their review of small group research Moreland and McMinn (2010) draw attention to 1) the lack of significant relation between reflexivity and team performance found in some studies (e.g., Edmondson, Bohmer, & Pisano, 2001; Savelsbergh, van der Heijden, & Poell, 2009) and 2) relatively limited evidence of the effect of reflexivity on team performance (e.g., Lewis, Belliveau, Herndon, & Keller, 2007; Müller, Herbig, & Petrovic, 2009). They concluded that reflexivity could be beneficial to team performance under certain circumstances. In the learning sciences, a similar trend has been observed: although team reflexivity in collaborative teams is highly important for the learning process, it does not always yield better learning gains (e.g., Prinsen, Terwelb, Zijlstrac, & Volman, 2013).

Given these mixed results, limitations of the small but growing research strand on team reflexivity need to be synthesised. First, reflexivity does not happen in a vacuum. Teams will eventually adapt their operating methods and ways of working based on feedback cues from their environment. We could expect reflexivity to only improve team performance when teams have access to feedback describing their objective and accurate performance (Schippers et al, 2013). Yet, reflexivity is seldom conceptualised as a process augmenting the effect of feedback on performance (Seibert, 1999). Moreover, little is known about how people reflect on feedback at the team level, while there has been empirical evidence of the effect of reflection upon feedback at the individual level (e.g., Duijnhouwer et al., 2012). For example, Anseel, Lievens, and Schollaert (2009) have tested the effect of feedback augmented with reflection at the individual level and demonstrated that the combined use of individual-level feedback and reflection improved performance better than individual feedback alone. At the team-level, only one series of studies isolated the effect of feedback from its combination with reflection in computer-supported collaborative learning in high-school teams (Phielix, Prins, & Kirschner, 2010; Phielix, Prins, Kirschner, Erkens, & Jaspers, 2011). The authors expected shared self and peer assessment and shared reflection to have complementary effects. They did not find any significant effect of reflection alone or of the combined use of feedback and reflection on objective performance (i.e., grade), but demonstrated that the joint use of feedback and reflection lead to higher group process satisfaction and social and cognitive behaviour. Interestingly, they draw attention to the fact that feedback (based on peer and self-perceptions) and reflection did not decrease unrealistic positive perceptions teams generally have about their own and other performance. This could be a reason for a lack of effect on objective performance. We do not know what are the effects of external feedback based on objective criteria for task achievement.

Second, as in most research in organisational psychology, the vast majority of small group research measuring team reflexivity in relation to team performance has used self-report instruments. Self-report measures are limited by team members’ level of awareness of their own behaviours and states and distorting biases such as social desirability. Calls for studying reflective behaviours in teams have generally gone unheeded (West, 1996). In the learning sciences, Dillenbourg, Baker, Blaye, and O’Malley (1996) have advised researchers to zoom in the ‘black box’ of collaborative processes. Subsequent to this call, there has been a recent proliferation of process-oriented research on collaboration (e.g., De Wever, Schellens, Valcke, & Van Keer, 2005) that advanced our understanding on interaction features contributing to more effective learning. Most of this collaborative learning research strand has focused on individual learning without explicitly investigating how collaborative processes influence team performance (e.g., Janssen, Kirschner, Erkens, Kirschner, & Paas, 2010). Nevertheless, these insights underscore the value of observational methods to provide crucial information about the context in which reflective behaviours occur and relate to team performance (e.g., Chi, 1997; Leicht, Hunter, Saluja, & Messner, 2010).

Finally, one area in which our understanding is incomplete across disciplines concerns the role of time and timing of reflective behaviours and their relation with team performance (e.g., Ballard, Tschan, & Waller, 2008; Janssen et al., 2010; Okhuysen & Waller, 2002; Reimann, 2007; Waller, 1999). There is a general agreement in the team literature that team performance is the product of ongoing and recurrent processes and actions (McGrath, 1993). Marks and colleagues (2001) conceptualise these cycles as performance episodes. Performance episodes consist of repeated cycles of action (i.e., when teams perform an activity) and transition (or ‘interrupts’) phases between actions. These interrupts are opportunities for teams to stop and reflect about their progresses for engaging in change (Okhuysen & Waller, 2002). The most common conceptualisation of team reflexivity does stipulate that shared reflection can occur before, during, and after a task (West, 2000; Schippers et al., 2007). However, scholars have generally measured it as an overall working style (Gurtner, Tschan, Semmer, & Nagele, 2007) or as an aggregated measure of collaborative activities and have not differentiated the moments at which it occurs (e.g., Lajoie & Lu, 2012). Specifically, they have not tested whether team reflexivity was more or less beneficial in certain phases, in relation to team performance dynamics, as suggested by certain authors (Hoeg & Parboteeah, 2006; Janssen et al., 2010; Schippers et al., 2003). This time-related issue of team reflexivity is elaborated upon in the following section.

We identify three primary issues in understanding the dynamic aspects of team reflexivity. First, although the importance of dynamic conditions experienced by teams over time is widely accepted (e.g., Waller, 1999), empirical work on how team reflexivity changes over time is missing (e.g., Janssen et al., 2010). On the one hand, it may be that overt communication is no longer needed as teams improve their implicit coordination over time, thus decreasing reflective interactions (e.g., Entin & Serfaty, 1999). Additionally, teams tend to define their goals and strategies at an early stage of their work and not to deliberatively review them after some work has been accomplished (Argote, 1989; Hackman & Wageman, 2005). Also, in line with arguments from Schippers and colleagues (2003), reflexivity might decline over time in diverse teams as viewpoints and perspectives become incompatible. On the other hand, they suggest that this declining effect might be reduced by the provision of feedback. It may be that the availability of accurate performance data highlighting deficiencies and a sustained task complexity trigger learning needs for teams perhaps calling for more reflection over time (Rulke & Rau, 2000). As such, feedback provision occurring during transition can act as a formal mechanism, a temporal punctuation likely to encourage reflection without necessarily giving a predetermined framework to follow (Okhuysen & Waller, 2002).

Second, the role of timing of reflective behaviours is similarly not well understood. The scarce previous research on the question ‘’does reflection during action and/or transition lead to better performance’’ has shown mixed results in contexts without explicit feedback. For example, in a study conducted by Moreland and McMinn (2010), none of the (scarce) reflective behaviours occurring during transition was significantly related to changes in team performance. By contrast, research looking into the impact of "interrupts" on group processes concluded that these were triggers of change in groups (Okhuysen & Eisenhardt, 2002). Team members appear to naturally interrupt their work around the midpoint of the allocated time for task completion and be more likely to put into practice strategies they set during these time outs (Gersick, 1989). Concerning reflection during action, Moreland and his colleague suggest that it might have more impact on performance. This reflection-in-action would be more directly related to the activities team members perform and thus prevent errors from being committed in real time. Conversely, it is likely that reflection while performing has a cost especially in a task that combines active processing of information and coordinated actions (Kirschner, Paas, & Kirschner, 2009; Schippers et al., 2013). That is, reflecting while executing a complex task places an extra burden on teams which may overload their working memory occasioning less optimal performance (Kirschner et al., 2009).

Furthermore, it is generally recognised that what happens in the early part of the team interaction might provide insight into subsequent effectiveness (e.g., Eriksen & Dyer, 2004; Kaplan, Laport, & Waller, 2013). Team decision-making literature has provided preliminary insights into this issue of timing of behaviours. It was shown that in teams with distributed information (i.e., comprising team members holding unique information), early agreements might harm decision quality because teams are less focused on exchanging and integrating distributed information (van Ginkel et al., 2009; van Ginkel & van Knippenberg, 2009). Accordingly, jumping too early into task completion might lead to process losses and performance decline (Mathieu & Rapp, 2008). On the contrary, it was demonstrated that effective teams share and elaborate upon distributed information at the beginning of interaction (van Ginkel at al., 2009). Rulke and Rau (2000) came to a similar conclusion when examining how teams develop a shared understanding of ‘who knows what’ in the team (i.e., transactive memory systems), proved to be an important factor to achieve better success. They observed that teams with high transactive memory systems were those whose members shared understandings and evaluated each other’s expertise early in their team interactions. In another example from computer-supported collaborative learning research, Kapur, Voiklis, and Kinzer (2008) demonstrated that a high quality contribution at the beginning of a problem solving process had more impact than those occurring later during team interactions. Therefore, the temporal pattern within reflective interactions should be taken into account in further understanding team reflexivity upon feedback. Also, presently, in the team reflexivity literature the question ‘’do the three behaviours making up reflexivity have differential effects on subsequent performance at an early stage of team interaction’’ remains unanswered. No empirical work has shown the necessity of a certain order of these reflective behaviours nor whether, and if so when, certain reflective behaviours were more conducive to better performance. For example, if we look into team reflective behaviours individually, we do not know if evaluating and looking for alternatives during teams’ first moment of interaction promote elaboration and understanding of the task whereas making decisions at an early stage is detrimental to subsequent performance.

Third, the direction of the relation between reflexivity and performance can be questioned (e.g., Janssen et al., 2010). In line with the core assumption of previous research on team reflexivity, does reflexivity lead to subsequent better performance? Alternatively or at the same time, do teams learn from previous performance, and thus reflect more as a consequence of how they performed previously? Research still has to prove the theoretical claim that teams can learn from the past through reflection with clear sight of performance criteria and information about their attainment. Only recently, Schippers and colleagues (2013) have given indirect evidence in this regard. In this study, self-report reflexivity was measured at two points in time in teams of students working on their bachelor thesis. This study showed that low-performing teams had the capability to translate information from performance feedback into effective task approaches. However, students were only given a grade and not feedback describing attainment of specific performance criteria. As suggested by Waller, Gupta, and Giambatista (2004) the timing of errors and subsequent behaviours has to be recorded to answer the question of causality. Further, investigating the timing of reflective activities in teams can help detect the points at which team reflexivity occurs and may need to be supported (Lajoie & Lu, 2012).

In the present study, we seek to understand the dynamics of team reflexivity and the relation (uni or bi-directional) between team reflexivity and performance. Therefore, we explore the two following questions. 1) Does the occurrence of team reflexivity augment or decline over time during action and transition phases of teamwork? 2) How is the timing at which reflective behaviour occurs related to performance? Specifically, is reflexivity during action and/or during transition related to higher performance (a)? Does each behaviour making up team reflexivity have the same impact on performance when occurring during teams’ first moment of interaction (b)?

Sixty-four students (32 male and 32 females) were recruited from a university in the Netherlands and randomly assigned to thirty-two dyads (N = 32). Their ages ranged from 18 to 29 years, M = 22.3, SD = 2.4. Participants were not eligible if any of the following exclusion criteria were present: experience in flight or related simulations and familiarity with each other. They were either paired with a same-gender partner (female and male teams, n = 11 and n = 11 respectively) or different-gender partner (mixed teams, n = 10). By random assignment, half of the sample was assigned to a role of pilot and the other half to a role of co-pilot. Subjects participated voluntarily in exchange for vouchers.

Participants in the role of pilot and co-pilot were required to complete four landing missions of the computer simulation “Microsoft Flight Simulator X”. The task, a complex, fast-paced, and high-workload situation, was chosen to stimulate ongoing learning in a controlled environment. Cognitively complex and interactive simulation tasks, such as flight-simulations, are commonly used in team research to investigate processes related to team performance (e.g., Bowers, Salas, Prince, & Brannick, 1992; Villado & Arthur, 2013). We did not use this computer simulation to mimic real-work team environments but rather to examine a set of theoretical relations (i.e., nomological network) among constructs within specific and controlled boundaries: a complex, fast-paced, and high-workload situation in which team members with unequally distributed information have to learn from each other to achieve their team goal and extend their learning to more complex variations of the task (Marks, 2000). To avoid that good-performing teams would have less need to learn as a consequence of their reflection on performance (Schippers et al. 2013), the level of complexity of the missions increased gradually over time. The abundance of information teams received before and during the missions and the high level of interdependence between pilot and co-pilot ensured a high level of complexity across performance episodes. In each mission, teams had to follow a predetermined traffic pattern during which they were required to maintain appropriate levels of speed, altitude, and a correct configuration of the airplane. The missions were completed when the team managed to land safely on the runway. The computer was connected to a whiteboard on which the game was screened.

The whole session lasted approximately two and a half hours. After introduction to the procedure and random assignment to the role of pilot or co-pilot, participants were individually trained during forty-five minutes. Items of information necessary for achieving a good landing were distributed between the team members. Pilots and co-pilots were seated in separate rooms to study the task material containing critical role-specific knowledge of piloting or monitoring the aircraft. The task of the pilot was to fly the plane and operate the joystick. For that purpose, pilots received an additional 10-minute hands-on training to practice. The task of the co-pilot was to control the gas of the plane and provide the pilot with indications and directions. Only the co-pilot had the access to the air traffic control (ATC)’s instructions, given through headphones, and knew how to interpret the cockpit instruments. After the training, participants were seated together to complete four landing missions. Teams had up to fifteen minutes to complete each mission and were also allowed to restart a mission if they had crashed. Before each mission the team received a written description of the flight objectives and the general mission scenario. Moreover, before starting missions 2, 3, and 4, teams were given specific performance feedback about their previous performance. Performance feedback described the attainment of success criteria such as speed, altitude, rate of descent, pitch, touchdown, and traffic pattern. The participants were allowed to communicate freely with one another. All teams were videotaped.

Two performance scores were computed: the total number of errors during a mission and the number of times teams crashed. The number of errors was derived from an instrument rating objective performance criteria (e.g., speed, altitude, activation of flaps and landing gear, landing position) of a good landing approach. This instrument was based on two sources: firstly, to identify key factors of a good flight, we performed a task analysis with a flight expert. Secondly, we used tests that the game itself provides its players to refine these criteria. Examples of deficiencies (i.e., errors) included failure to extend the flaps before landing, to maintain a certain speed interval during descent, to reduce the speed before touchdown, to keep a constant rate of descent, to align with the runaway, or to have one touchdown on the runaway. The total number of potential errors varied in the four missions. This variation reflects the increasing level of difficulty of the missions. We chose the number of crashes to depict one of the most salient manifests of performance for participants. There were four measurement times in total (i.e., T1, T2, T3, and T4).

Team communication was coded to identify representative behaviours that could be taken as evidence of team reflexivity (Rourke & Anderson, 2004). We developed the coding scheme of the present study through a series of steps assuring its validity and reliability (Schippers et al., 2007). First, we determined the granularity of the unit of analysis. The unit of meaning was applied (Rourke, Anderson, Garrison, & Archer, 2001). Specifically, to consider a verbal statement a significant unit, we decided that utterances had to be individual messages (questions or statements) that 1) were expressed by one team member, 2) dealt with one topic, idea, or argument chain, 3) reflected one unique behaviour, and 4) related to the topic at hand or the team. Thus, one semantic feature (unit of meaning) and one activity feature (team member speaking) were used for segmentation of the communication content into units (Chi, 1997). As such, as soon as the topic or the speaker changed, a new behaviour was coded (Visschers-Pleijers, Dolmans, de Leng, Wolfhagen, & van der Vleuten, 2006). In addition to verbal statements, one unambiguous non-verbal behaviour was set as an evidence of one of the reflective behaviours.

Second, we discriminated verbal interactions types that typified reflexivity. To do this, we adapted and expanded the initial framework of team reflexivity (West, 1996, 2000) and an existing questionnaire from Schippers and colleagues (2003). Reflexivity was originally defined as an iterative process including three broader behaviours, namely reflection, planning, and acting/adapting. As shown in Table 1, the coding scheme covers three reflective behaviours: evaluating or reviewing present or past team performance and strategies, looking for alternatives, and making decisions. Information directly forwarded from the ATC (repetitions) and the literal reading of the feedback form were excluded from the coding.

Third, we ran a pilot study to test and validate the coding scheme. This lead to adaptations, clarifications of the reasoning behind the framework definitions and the boundaries of the units, and the addition of typical examples. Fourth, we extensively trained two coders, each blind to the hypotheses of the study, to optimise reliability and consequently reduce errors in observation. They were provided with clear examples (of inclusion and exclusion) of the manifestation of the behaviours and had rating exercises with multiple rounds and discussions to attain consistency among coders. Fifth, videotapes were coded with the newly developed coding scheme. Finally, the two coders coded independently one-third of the videotapes to estimate interrater reliability (Cohen’s kappa). Kappas were calculated for all the categories. These kappas ranged from 0.65 to 0.88, with an average of 0.78, indicating a ‘substantial’ to ‘almost perfect agreement’ across the two coders as to the occurrence of the specific behaviours (Landis & Koch, 1977). Coders and two trainers resolved any discrepancies.

Figure 1. Overview of the phases of the study and behaviours measured at each wave of data. (see pdf file)

For the coding, we used The Observer® XT 10.5, a computer software aimed for quantitative analysis of observational data. Videotapes were directly coded without transcripts. The extent to which teams engaged in each behaviour was expressed in terms of frequencies of occurrence of the behaviour of its members for each mission (i.e., action phase) and for each feedback (transition) phase. Additionally, we computed an overall team reflexivity score (i.e., aggregation of the three behaviours) for each time measurement for action and feedback.

We coded acts on the basis of utterances of reflective behaviours occurring at the team level of analysis (i.e., aggregation of individual utterances for each team). Besides utterances, we examined phases (or ‘’when’’, specifically action or transition, earlier interaction or later interaction) in which some performance events (i.e., crashes and errors) were related to team reflective behaviours. Behaviours were coded at four points in time during action and feedback phases (Time 1, Time 2, Time 3, and Time 4) for each team (N = 32).

In the following section, we first present an overview of the frequencies of behaviours considered individually and of the frequencies of sequences comprising two or three behaviours. Second, we test whether reflective behaviours change over time during action and feedback using repeated-measures analyses of variance. Third, correlations between team reflective behaviours and performance are examined, more specifically (a) the relations between prior performance and subsequent team reflexivity and (b) prior team reflexivity and subsequent performance.

Figures 2, 3, and 4 depict means and standard deviations of the reflective categories across time during action and feedback phases. It can be seen that evaluating is the most frequent reflective behaviour. Looking for alternatives during action at Time 1 appears very scarce. During feedback, it seems that looking for alternatives is not a frequent practice. The same trend can be noted for making decisions. It is more frequent during missions than after feedback reception. It has to be noted that standard deviations reflect important differences between teams. In sum, reflective behaviours, when examined individually, tend to follow a similar pattern: they appear more frequent during action while they are low during feedback.

Figure 2. Means and standard deviations of Evaluating for each measurement time and phase (N = 32). (see pdf file)

Figure 3. Means and standard deviations of Looking for alternatives for each measurement time and phase (N = 32). (see pdf file)

Figure 4. Means and standard deviations of making decisions for each measurement time and phase (N = 32). (see pdf file)

The coded reflective acts described above are single communication behaviours. In the present study, team reflexivity was conceptualised as a collection of three behaviours. We explored whether teams actually completed full ‘’reflective cycles’’ comprising all three behaviours in a sequence. Since there has been no empirical work demonstrating the necessity of all three behaviours we also considered the most basic behavioural patterns consisting of two subsequent reflective behaviours. As can be seen in Table 2, most reflective communication across teams can be summarised by two main two-behaviour sequences: sequences starting with evaluating and looking for alternatives and ending with clear decisions about a different way to handle the task. In these sequences, teams “skipped” the evaluation or search for alternatives phases. While sequences of looking for alternatives followed by decisions seemed to grow over time during action, except for the last mission, the frequencies of the sequences starting with an evaluative comment and ending with a decision stayed relatively stable over time, except for a drop at Time 2. During feedback, the same trend than for individual behaviours is observed; sequences are very scarce. Importantly, full cycles were almost never completed, suggesting that teams were not naturally systematic in their reflective process. The absence of reflective cycles does not allow us to further investigate their change over time and relatedness to team performance. Still, how individual reflective behaviours evolve over time and are related to team performance is of importance to map 1) how teams naturally respond to feedback and 2) whether some behaviours appear more important than others in the feedback process.

Frequencies of Sequences of Reflective Behaviours at each Wave of Data (N = 32)

Sequences	Time 1		Time 2		Time 3		Time 4
	Time 1
	Action	Feedback	Action	Feedback	Action	Feedback	Action

B1-B2	2	2	3	0	15	1	9

B1-B3	30	14	15	7	29	0	29

B2-B3	2	2	18	0	30	3	23

B1-B2-B3	0	0	1	0	2	0	0

To test for significant changes of team reflexivity behaviours (considered individually) over time, we computed repeated-measures analyses of variance with a Greenhouse-Geisser correction (as sphericity was violated for all behaviours) and with the four times each behaviour was measured as a within-team factor. Pairwise comparisons with Bonferroni corrections controlling for inflation of Type I error were also computed. Evaluating during action changed significantly from Time 1 to Time 4, F (2.26, 65.67) = 5.04, p = .05, with pairwise comparisons showing that evaluating at Time 4 was significantly more frequent than evaluating at Time 2 (p = .019). In contrast, evaluating during feedback did not change over time. Regarding looking for alternatives during action there was also an overall significant difference between the means at the different time points, F (2.24, 64.84) = 5.76, p = .004. A pairwise comparison confirmed a difference between Time 1 and Time 2 (p = .034), Time 1 and Time 3 (p = .000), and Time 1 and Time 4 (p = .0006), signifying an increase of the behaviour over missions. Looking for alternatives during feedback, making decisions during action and during feedback did not change significantly over time with a Bonferroni correction. With Tukey’s test, making decisions during action at Time 4 was significantly higher than at Time 3 (p = .043) and Time 2 (p = .045). Finally, while overall reflexivity across feedback did not change significantly over time, the mean scores for overall reflexivity across missions (i.e., aggregated behaviours) were significantly different, F (2.33, 67.56) = 5.46 p = .004. Specifically, reflexivity during mission at Time 4 was higher than reflexivity during mission at Time 1 (p = .027) and Time 2 (p = .035). It is worthwhile noting that if the less conservative Tukey post hoc test is used, the score in mission 3 is higher than at Time 1 (p = .015) and Time 2 (p = .043).

Intercorrelations between reflective behaviours and performance measures are presented per time period in Table 3.

Correlations between Coded Categories and Performance Measures at each Point in Time (N = 32) (see pdf file)

Note. *p < .05. ** p < .01. T1= Time 1; T2=Mission 2; T3=Mission3; T4=Mission 4. Only significant correlations are indicated

At first glance, initial errors seem to be beneficial to subsequent reflective behaviours, while the trend that errors trigger reflection tends to decline with time. Specifically, the number of errors teams made during the first action phase was positively correlated with numerous reflective behaviours and to all reflective categories. The number of crashes teams initially faced followed the same path. Teams did not seem to be discouraged by their first experience with crash. For example, the initial number of crashes was positively related to more decision-making behaviours after it occurred, both during the feedback phase following the “failure” (r = .39, p < .05) and during the next action at Time 2 (r = .47, p < .01). In contrast, errors committed at Times 2 and 3 were not related to higher subsequent reflective behaviours, with the exception of errors at Time 2 appearing as a trigger for decision making during the transition phase immediately following that mission (r = .43, p < .05). Conversely, errors at Times 2 and 3 seem to hamper decision making during action at Time 4 (respectively r = - .42, p < .05 and r = - .43, p < .05), while the number of crashes at Time 2 was related to less frequent evaluative behaviours during feedback at Time 2 (r = -.45, p < .05) and during action at Time 4 (r = - .42, p < .05). In sum, these first results seem to suggest that initial failure acts as an eye opener to evaluate what went wrong, look for alternative ways of approaching the task, and make more decisions, while later, as task becomes more complex, it might relate to less subsequent reflection.

Concerning the impact of team reflexivity on subsequent performance, the correlations indicate a differential impact depending on the type of behaviour considered. The extent to which teams evaluated their performance and strategies during feedback at Time 3 was related to lower number of crashes at Time 4 (r = -.45, p < .05). Similarly, the extent to which teams engaged in evaluative behaviours during feedback at Time 2 significantly related to fewer errors at Time 3 (r = -.42, p < .05). What is particularly noticeable is that these significant relationships concern 1) evaluative behaviours and 2) periods of team transition, suggesting a positive effect of processing feedback on subsequent performance. Reflection during action does not seem to be significantly related to subsequent performance when bivariate correlations are computed.

In contrast, the extent to which teams made decisions during action at Time 1 was related to more crashes at Time 3 (r = .48 p < .01), suggesting that making decisions during the initial moments of team development on this novel task (with its specific characteristics) may impede subsequent performance.

To uncover when giving teams feedback about their performance creates an opportunity for learning, it has been posited that it is important to examine how teams actively process feedback and thus collaboratively evaluate information about past activities and derive alternative recommendations for next action. However, research on this feedback processing has been scarce in the learning sciences and organisational psychology (Gabelica et al., 2012; London & Sessa, 2006; Phielix, et al., 2010). It is shown that the specific activities teams perform to deal with feedback and when these activities are related to performance remain to be considered.

Following from these perceived gaps, we conducted a study attempting to build upon and extend research in this domain by 1) identifying actual behaviours enabling feedback processing (i.e., team reflexivity) and 2) providing a more fine-grained analysis of dynamic aspects of team reflexivity in a context with systematic and explicit feedback. While theoretical work seems to suggest team reflexivity is an iterative three-step cycle that involves evaluating performance and strategies, looking for alternatives, and making decisions (e.g., Schippers et al., 2007; Yukawa, 2006), we do not know if to perform well, it is necessary to follow this series of steps and if some steps are more dominant and influential than others in relation to team performance. We explored the development of team reflective behaviours and the relation between timing of reflective behaviours and performance during four performance episodes.

Following conclusions could be drawn. Firstly, teams never completed full cycles of evaluating, looking for alternatives, and making decisions. They did, however, completed two behaviour-sequences starting with evaluating or looking for alternatives and ending with a clear decision. Moreover, these sequences were very scarce during transition (i.e., interrupts during which team performance feedback was delivered). While the reflective cycle is usually described sequentially, teams seem to rarely follow a rigid series of steps to deal with feedback. Instead, it seems like they often skip steps or even go back through steps several times.

When taken individually, team reflective behaviours were overall more frequent and increased over time during action, whereas team reflexivity during transition was relatively less frequent and did not change. Looking for alternatives was very scarce but also increased over time during action only. There might be two reasons for this growth of reflection-in-action. First, natural reflection might arise as a response to an immediate learning need when a team observes cues of ineffective behaviours or experience misunderstanding or uncertainty (e.g., they get lost) while completing the task. Additionally, this learning need could have been triggered by the increasing complexity of the task (i.e., more cues to understand and interpret). Second, preceding feedback, which has been advanced as a way to counteract the natural decline of team reflexivity (Schippers et al., 2003), could also have had a delayed effect on next reflection-during action. In the learning sciences, feedback has been shown to impact subsequent learning (e.g., Mory, 2003). It may be that teams only see the learning content of a feedback when they have to deal with a similar situation. This high reflexivity-in-action suggests that teams are more reactive than proactive and adaptive to anticipated circumstances when faced with higher complexity and workload. This is in line with a common rationale some authors have been using to speculate on the causes of a lack of actual reflection in teams (Arvaja, Häkkinen, Eteläpelto, & Rasku-Puttonen, 2000; Morris & Stew, 2007). They state that teams are more driven by their results (i.e., producing and performing) than the learning they can gain, especially when they are under pressure and despite the obvious benefits of strategy development (Gurtner et al., 2007; Karau & Kelly, 1992). A possible explanation of why teams did not use transition phases to increase reflection in the same way lies in teams’ tendency to set their goals and strategies early and not to actively question them later (Argote, 1989; Hackman & Wageman, 2005; Weingart, 1992). Further, it may be that the concurrent cognitive demands of collaborating on the complex task, making sense of the received feedback, and trying to prepare for the next task were too high for teams to make the most of their learning experience (Kirschner et al., 2009; Rummel & Spada, 2009). Reflecting is a challenging high-order activity (Jay & Johnson, 2001). The concept of reflection assumes individuals have the capacity to engage in self-examination and open-minded analysis of their own knowledge. Additionally, even if team members can reflect in solo-learning situations, they may not be able to coordinate and co-reflect in a team, communicate their reflective thoughts, nor agree on ways to address the task (Chan, 2012).

Secondly, we uncovered patterns that transcended the straightforward question of whether team reflexivity can change performance in teams given explicit feedback. It seems that the key question is rather when improvements occur. (Lajoie & Lu, 2012) As signified in recent research on team reflexivity (e.g., Moreland & MacMinn, 2010; Schippers et al, 2013), reflexivity was not uniformly beneficial. We showed instances in which timing of reflective behaviours determined its effect (positive, negative, or neutral) on performance. First, early decision-making was related to lower subsequent outcomes. These findings are in accordance with previous studies on timing of decision-making indicating that early decision making might prevent deep processing and sharing of unevenly distributed information (van Ginkel, et al., 2009; van Ginkel & van Knippenberg, 2008). Second, after that first experience with their task, teams were able to derive insights from past performance (depicted by feedback) and correct misunderstandings that prevented effective action (Tjosvold et al., 2004). However, only evaluative behaviours performed during feedback phases were related to later improved performance. For example, teams were able to reduce their errors in the last high-workload task with more preparation (i.e., evaluating during preceding feedback time). As such, the effect of evaluative behaviours seems to be contingent on the phase during which they are performed (during transition rather than during action). This points a paradox: though they could experience positive consequences of reflecting during these time outs, teams did not increase reflection during feedback over time. Third, reflection during action does not seem to be significantly related to better subsequent performance when bivariate correlations are computed. As such, no empirical support could be found for Moreland and McMinn’s (2010) proposition that reflecting during the task could be more beneficial to teams due to the immediate possibility to adjust to the situation. However, this (increasing) reflection during action did not harm team performance either and showed that, in general, teams remained connected despite the increasingly higher workload of their task. Communication breakdowns could have been expected. Previous team research on non-routine working situations (e.g., Waller et al., 2004) has demonstrated that low-performing teams become cognitively overwhelmed in case of high workload and consequently tend to focus more on their individual sub-task instead of collaborating (e.g., Salas, Rosen, & King, 2007).

Finally, initial errors appeared to be a driver of subsequent team reflexivity while later errors mostly did not have this motivational role. These preliminary findings open up the possibility that there might be some time-specific effects of previous errors that are determinant to trigger motivation to improve. Maybe the first mistake does not really hurt while repeated errors would be more detrimental to performance?

Importantly, these results raise the question of the causality of the relation between team reflexivity and performance (e.g., Janssen et al., 2010). These analyses seem to rather show a more dynamic and retroactive relation between past and future performance and team reflexivity, raising the need for time-series-analyses. This is however behind the scope of the present explorative study. This suggests that the question “does prior performance trigger team reflexivity or does prior team reflexivity generate better performance” should rather be changed into: when do performance and team reflexivity dynamically interact to trigger team learning and better subsequent performance. As we found a reversed effect as well, it could be formulated that initial errors do matter and that teams have the capability to learn from them under certain circumstances.

Taken together, these findings underscore the importance of a careful evaluation of how the team is doing and why during transition phases corresponding to feedback reception. We could not empirically test the effects of three-step reflective cycles since they did not naturally occur but we provided evidence that teams using the feedback opportunity to stop and analyse their performance and strategies were able to translate information about performance into corrective behaviours since their performance got improved. These results are line with studies on the impact of interrupts (e.g., Okhuysen & Waller, 2002) and theories on feedback in teams that stipulate that feedback receivers’ involvement plays a critical role to explain feedback effectiveness in teams (e.g., Gabelica et al., 2012).

Although we have specified behaviours signifying team reflexivity, we have not explored depth of processing (Volet, Summers, & Thurman, 2009). It is likely that deeper reflection (e.g., reflective statements conveying inferences) generates better insight into the feedback content and use in subsequent tasks (Anseel et al., 2009).

Also, previous research has acknowledged that developing strategies for action is important and necessary but does not always ensure actual strategy implementation (Gurtner et al., 2007; Marks et al., 2001; Tschan et al., 2000). Our third reflective category (i.e., making decisions) encompassed both strategy development (i.e., clear decisions about new strategies and actions) and strategy implementation (i.e., clear gesture or overt behaviour showing the team acting upon a decision). Despite the fact that teams were provided with feedback based on specific criteria, improvement strategies defined by teams could still have been too general or abstract to be directly put into action, or coordination problems could have render the well-defined plans unused. A further investigation of implementation strategies and their quality seems warranted.

Overall, this empirical study was designed to meet methodological requirements of a rigorous design with high internal validity, based on its temporal sequencing and the collection of objective performance data. Capturing fundamental processes in a controlled environment is a first step to better understand complex phenomena. To that purpose, we used a flight simulation as a research platform to simulate and control task and team features. Simulations constitute an interesting environment that offers standardised performance measures with possibilities of controlling complexity, information overload, and cues available from the environment (Mathieu, 2000). In a learning perspective, they also allow learners to apply their knowledge and understanding to a task and observe the effects of their decisions in a reactive environment that offers real-time feedback (Bronack, Riedl, & Tashner, 2006; Gredler, 2004). Prior work on simulations has shown that they stimulate numerous cognitive processes, such as higher-level reasoning or creative thinking (Moreno & Mayer, 2005; Moreno, Mayer, Spires, & Lester, 2001). As a trade-off, laboratory environments overlook natural factors in real-world contexts that may mediate learning. As such, the extent to which the results from our controlled design can be generalised to real had-hoc teams has to be considered with cautious. The artificial and temporal nature of the team and the limited number of team members must be acknowledged. Teams of more than two team members and/or knowing each other before completing a team task might exhibit more complex interaction patterns. In this regard, we acknowledge that there has been a recent debate about whether findings from research on dyads can be simply generalised to larger teams (Moreland, 2010; Williams, 2010). We are aware that certain aspects of team processes and dynamics (e.g., group socialization) can hardly be grasped by the use of dyads and that the addition of team members increases complexity of team communication and coordination (Michinov & Michinov, 2009; Noroozi et al. 2012). However, since research into the dynamic aspects of co-reflection in teams is a relatively new area, the present study looking into the timing of basic team behaviours in the smallest form of teams provides a very good start for further research. A replication study with triads and larger groups is needed to corroborate our results and explore the relationship between team size and successful team reflexivity. Moreover, although novelty of the task was controlled, we did not account for group-ability composition while research has demonstrated its influence on the accuracy and quality of explanations in teams (Webb et al., 1998). Finally, the use of students is sometimes considered as a possible limitation. Nevertheless, previous studies have established that little difference holds between the use of students and professional teams when using problem solving and decision-making scenarios (Balijepally et al., 2009; Yoo & Alavi, 2001). Still, further research exploring the effects of reflexivity with explicit feedback with more team members and in different settings will be needed to understand the complexity of how team members with different expertise, knowledge, and possibly high diversity deal with feedback that describes the aggregated group effort towards a shared goal. Furthermore, future field studies obviously need to consider critical contextual factors that influence and constraint team behaviours (Kozlowski & Ilgen, 2006).

Another limitation of the study is the relatively lower frequency of some reflective behaviours (e.g., looking for alternatives). Similarly, sequences of reflective behaviours did not occur very often, limiting the possible analyses relating these to performance improvements. This necessitates caution about drawing premature conclusions and underlines the need for replication studies with larger samples. Additionally, this limitation could be overcome in future research by stimulating or training teams to become reflexive (e.g., King, 1991) and comparing occurrences of reflective behaviours and their relation with team performance with a no- training condition. Another challenging issue is motivational: we do not know why teams did not frequently reflect after feedback. Motivational factors behind a lack of reflection and receptivity to feedback should be investigated in further research.

Finally, the relation between team performance (signified by accurate and specific feedback) and team reflexivity should be analysed in longitudinal designs. Additional measurement points of team reflexivity spaced over time would provide a more fine-tuned understanding of under which circumstances previous performance has more impact on learning. Feedback loops in which previous performance acts as an input for determining subsequent processes and performance have been recently forwarded as relevant models to understand team dynamics (llgen et al., 2005).

The present study suggests that to be effective, feedback requires high levels of cognitive engagement from learning teams. However, discussion of and reflection on underlying reasons for success or failure, alternatives, and improvement strategies does not seem to happen spontaneously in teams, which in turn brings out the need to provide them with appropriate external support. This has potential implications for education. First, teachers should uncover whether a lack of engagement in thoughtful analysis of team experience is due to a lack of ability to perform shared reflection (i.e., availability deficiency) or a lack of execution of available skills (i.e., production deficiency). If students know how to reflect as a team, prompts (i.e., scenarios indicating how learners should interact) designed to induce inferences and deep-oriented processing of the content of the feedback appear an appropriate intervention to enhance teams’ motivation to engage in team reflexivity and to elicit learning strategies which teams would not naturally demonstrate (Veenman & Elshout, 1999). If students have not yet acquired the skills needed to perform shared reflection, teachers may first model and organise repeated practice with reflective cycles and provide more guidance and structure in their prompts (King, 2007). However, it may be that when students have attained a high level of reflective skills, prompts need to be withdrawn or faded-out to facilitate internalization (Dillenbourg & Tchounikine, 2007). Second, teachers should plan time and space for feedback, in which errors are considered as learning opportunities, and provide tools enabling students to actually perform reflective activities on feedback (Gan & Hattie, 2014).

The authors are very grateful to the two coders of the present study, Claudia Baudewijns and Lubomira Nikolova.

Anseel, F., Lievens, F., & Schollaert, E. (2009). Reflection as a strategy to enhance task performance after feedback. Organizational Behavior and Human Decision Processes, 110, 23-35. doi:10.1016/j.obhdp.2009.05.003

Adcroft, A. (2011). The mythology of feedback. Higher Education Research and Development 30(4), 405-419. doi:10.1080/07294360.2010.526096

Argote, L. (1989). To centralize or not to centralize: the effects of uncertainty and threat on group structure and performance. Organizational Behavior and Human Decision Processes, 43, 58–74. doi:10.1016/j.bbr.2011.03.031

Arvaja, M., Häkkinen, P., Eteläpelto, A., & Rasku-Puttonen, H. (2000). Collaborative processes during report writing of a science learning project: The nature of discourse as a function of task requirements. European Journal of Psychology of Education, 15(4), 455-466. doi:10.1007/BF03172987

Austin, J., Kessler, M. L., Riccobono, J. E., & Bailey, J. S. (1996). Using feedback and reinforcement to improve the performance and safety of a roofing crew. Journal of Organizational Behavior Management, 16(2), 49-75. doi:10.1300/J075v16n02_04

Ballard, D. I., Tschan, F., & Waller, M. J. (2008). All in the timing: Considering time at multiple stages of group research. Small Group Research, 39, 328–351. doi:10.1177/1046496408317036

Balijepally, V., Mahapatra, R., Nerur, S., & Price, K. H. (2009). Are two Heads better than one for Software Development? The Productivity Paradox of Pair Programming. MIS Quarterly, 33(1), 91-118.

Barron, B. (2000). Achieving coordination in collaborative problem-solving groups. Journal of the Learning Sciences, 9, 403–436. doi:10.1207/S15327809JLS0904_2

Boud, D., Keogh, R., & Walker, D. (1985). Promoting reflection in learning: A model. In D. Boud, R. Keogh, & D. Walker (Eds.), Reflection: Turning experience into learning (pp. 18-40). London: Kogan Page.

Boud, D., & E. Molloy. (2013). Rethinking Models of Feedback for Learning: The Challenge of Design. Assessment & Evaluation in Higher Education, 38(6), 698-712. doi:10.1080/02602938.2012.691462

Bowers, C. A., Salas, E., Prince, C., & Brannick, M. T. (1992). Games teams play: A methodology for investigating team coordination and performance. Behavior Research Methods, Instruments, and Computers, 24, 503-506. doi:10.3758/BF03203594

Bronack, S., Riedl, R., & Tashner, J. (2006) Learning in the zone: A social constructivist framework for distance education in a 3D virtual world. Interactive Learning Environments, 14(3), 219-232. doi:10.1080/10494820600909157

Chan, C. K. K. (2012). Co-regulation of learning in computer-supported collaborative learning environments: a discussion, Metacognition and Learning, 7(1), 63-73. doi:10.1007/s11409-012-9086-z

Chi, M. T. H. (1997). Quantifying qualitative analysis of verbal data: A practical guide. The Journal of the Learning Sciences, 6, 271-315. doi:10.1207/s15327809jls0603_1

Decuyper, S., Dochy, F., & Van den Bossche, P. (2010). Grasping the dynamic complexity of team learning: An integrative model for effective team learning in organizations. Educational Research Review, 5, 111-133. doi:10.1016/j.edurev.2010.02.002

De Wever, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review. Computers & Education, 46(1), 6–28. doi:10.1016/j.bbr.2011.03.031

Dillenbourg, P. (1999). Collaborative Learning: Cognitive and Computational Approaches. Advances in Learning and Instruction Series. New York, NY: Elsevier Science, Inc.

Dillenbourg, P., Baker, M., Blaye, A, & O’Malley, C. (1996). The evolution of research on collaborative learning. In E. Spada & P. Reiman (Eds), Learning in Humans and Machine: Towards an interdisciplinary learning science (pp. 189-211). Oxford: Elsevier.

Dillenbourg, P., & Tchounikine, P. (2007). Flexibility in macro-scripts for computer-supported collaborative learning. Journal of Computer Assisted Learning, 23(1), 1–13. doi:10.1111/j.1365-2729.2007.00191.x

Dochy, F., Gijbels, D., Raes, E., & Kyndt, E. (2014, forthcoming). Team learning: Research in education and professional contexts. In S. Billett, C. Harteis, and H. Gruber (eds.), International Handbook of Research in Professional Practice-based Learning. Springer.

Duijnhouwer, H., Prins, F. J., , Stokking, K. M. (2012). Feedback providing improvement strategies and reflection on feedback use: Effects on students’ writing motivation, process, and performance. Learning and Instruction, 22(4), 171-184. doi:10.1016/j.learninstruc.2011.10.003

Edmondson, A. C. (1999). Psychological safety and learning behaviour in work teams. Administrative Science Quarterly, 44(2), 350–383. doi:10.2307/2666999

Edmondson, A. C, Bohmer, R. M, & Pisano, G. P. (2001). Disrupted Routines: Team Learning and New Technology Implementation in Hospitals. Administrative Science Quarterly, 46, 685-716. doi:10.2307/3094828

Edmondson, A. C., Dillon, J., & Roloff, K. S., (2007). Three perspectives on team learning outcome improvement, task mastery, and group process. The Academy of Management Annals, 1, 269-314. doi:10.1080/078559811

Entin, E. E., & Serfaty, D. (1999). Adaptive team coordination. Human Factors, 41(2), 312-325. doi:10.1518/001872099779591196

Eriksen, J., & Dyer, L. (2004). Right from the start: Exploring the effects of early team events on subsequent project team development and performance. Administrative Science Quarterly, 49, 438–471. doi:10.2307/4131442

Eva, K. W., Armson, H., Holmboe, E., Lockyer, J., Loney, E., Mann, K., et al. (2012). Factors influencing responsiveness to feedback: On the interplay between fear, confidence, and reasoning processes. Advances in Health Sciences Education: Theory and Practice, 17(1), 15–26. doi:10.1007/s10459-011-9290-7

Gabelica, C., Van den Bossche, P., Segers, M., & Gijselaers, W. (2012). Feedback, a Powerful Lever in teams: a Review. Educational Research Review, 7(2), 123-144. doi:10.1016/j.bbr.2011.03.031

Gan, M. J. S., & Hattie, J. (2014). Prompting secondary students’ use of criteria, feedback specificity and feedback levels during an investigative task. Instructional Science. doi:10.1007/s11251-014-9319-4.

Gersick, C. J. G. (1989). Marking time: Predictable transition in task groups. Academy of Management Journal, 32, 274-309. doi:10.2307/256363

Goodman, J. S., Wood, R. E., & Hendrickx, M. (2004). Feedback specificity, exploration, and learning. Journal of Applied Psychology, 89(2), 248-262. doi:10.1037/0021-9010.89.2.248

Gredler, M. E. (2004). Games and simulations and their relationships to learning. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology (2nd ed., pp. 571-82). Mahwah, NJ: Lawrence Erlbaum Associates.

Gurtner, A., Tschan, F., Semmer, N. K., & Nagele, C. (2007). Getting groups to develop good strategies: Effects of reflexivity interventions on team process, team performance, and shared mental models. Organizational Behavior and Human Decision Processes, 102, 127–142. doi:10.1016/j.bbr.2011.03.031

Hackman, J. R., & Wageman, R. (2005). A theory of team coaching. Academy of Management Review, 30, 269-287. doi:10.5465/AMR.2005.16387885

Hattie, J. (2013). Calibration and confidence: Where to next? Learning and Instruction, 24, 62-66. doi:10.1016/j.bbr.2011.03.031

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112. doi:10.3102/003465430298487

Hoegl, M., & Parboteeah, K. P. (2006). Team goal commitment in innovative projects. International Journal of Innovation Management, 10(3), 299–324. doi:10.1142/S136391960600151X.

Ilgen, D. R., Hollenbeck, J. R., Johnson, M., & Jundt, D. (2005). Teams in organizations: From I-P-O models to IMOI models. Annual Review of Psychology, 56, 517-543. doi:10.1146/annurev.psych.56.091103.070250

Jay, J. K., & Johnson, K. L. (2001). Capturing complexity: A typology of reflective practice for teacher education. Teaching and Teacher Education, 18(1), 73-85. doi:10.1016/j.bbr.2011.03.031

Janssen, J., Kirschner, F., Erkens, G., Kirschner, P. A., & Paas, F. (2010). Making the black box of collaborative learning transparent: Combining process-oriented and cognitive load approaches. Educational Psychology Review, 22,139-154. doi:10.1007/s10648-010-9131-x

Järvelä, S., Volet, S., & Järvenoja, H. (2010). Research on motivation in collaborative learning: Moving beyond the cognitive-situative divide and combining individual and social processes. Educational Psychologist, 45, 15–27. doi:10.1080/00461520903433539

Johnson, D. W., & Johnson, R. T. (1992). Key to effective cooperation. In R. Hertz-Lazarowitz, & N. Miller (Eds.), Interaction in cooperative groups. The theoretical anatomy of group learning (pp. 174–199). New York, NY: Cambridge University Press.

Johnson, D. W., & Johnson, R. T. (1993). What we know about cooperative learning at the college level. Cooperative Learning, 13(3), 17–18.

Johnson, D. W., Johnson, R. T., & Stanne, M. E. (2000). Cooperative Learning Methods: A meta-analysis. University of Minnesota, Minneapolis: Cooperative Learning Center. Retrieved from: http://www.co-operation.org/pages/cl-methods.html.

Jung, D. I., & Sosik, J. J. (2003). Group potency and collective efficacy: Examining their predictive validity, level of analysis, and effects of performance feedback on future group performance. Group & Organization Management, 28, 366-391. doi:10.1177/1059601102250821

Kaplan, S., LaPort, K., & Waller, M. J. (2013). The role of positive affect in team effectiveness during crises. Journal of Organizational Behavior, 34(4), 427-580. doi:10.1002/job.1817

Kapur, M., Voiklis, J., & Kinzer, C. K. (2008). Sensitivities to early exchange in synchronous computer-supported collaborative learning (CSCL) groups. Computers & Education, 51(1), 54-66. doi:10.1016/j.compedu.2007.04.007

Karau, S. J., & Kelly, J. R. (1992). The effects of time scarcity and time abundance on group performance quality and interaction process. Journal of Experimental Social Psychology, 28, 542–571. doi: 10.1016/j.bbr.2011.03.031

Killion, J., & Todnem, G. (1991). A process for personal theory building. Educational Leadership, 48(6), 14-16.

King, A. (2007). Scripting collaborative learning processes: A cognitive perspective. In F. Fischer, I. Kollar, H. Mandl, & J. M. Haake (Vol. Eds.), Computer-supported collaborative learning series: Vol. 6. Scripting computer-supported collaborative learning (pp. 13–37). New York: Springer. doi:10.1007/978-0-387-36949-5.

King, A. (1991). Effects of training in strategic questioning on children’s problem-solving performance. Journal of Educational Psychology, 83(3), 307–317.

Kirschner, P. A. (2009). Epistemology or pedagogy, that is the question. In: S. Tobias & T.M. Duffy (eds.), Constructivist theory applied to instruction: Success or failure? (pp. 144-157). New York: Routledge.

Kirschner, F., Paas, F., & Kirschner, P. A. (2008). A cognitive load approach to collaborative learning: United brains for complex tasks. Educational Psychology Review, 21, 31-42. doi:10.1016/j.chb.2008.12.008

Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254-284. doi:10.1037/0021-9010.73.1.87

Kozlowski, S. W. J., & Ilgen, D. R. (2006). Enhancing the effectiveness of work groups and teams. Psychological Science in the Public Interest, 7, 77–124. doi:10.1111/j.1529-1006.2006.00030.x

Kramarski, B. (2004). Making sense of graphs: Does metacognitive instruction make a difference on students’ mathematical conceptions and alternative conceptions. Learning and Instruction, 14, 593-619. doi:10.1016/j.bbr.2011.03.031

Lajoie, S. P., & Lu, J. (2012). Supporting collaboration with technology: Does shared cognition lead to co-regulation in medicine. Metacognition and Learning, 7, 45-46. doi:10.1007/s11409-011-9077-5

Landis, J. R., & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data, Biometrics, 33, 159-174.

Leicht, R. M., Hunter, S. T., Saluja, C., & Messner, J. I. (2010). Implementing observational research methods to study team performance in construction management. Journal of Construction Engineering and Management, 136(1), 76–86. doi:10.1061/(ASCE)CO.1943-7862.0000080

Lewis, K., Belliveau, M., Herndon, B, & Keller, J. (2007). Group cognition, membership change, and performance: Investigating the benefits and detriments of collective knowledge. Organizational Behavior and Human Decision Processes, 103(2), 159-178. doi: 10.1016/j.bbr.2011.03.031

London, M. (2003). Job feedback: Giving, seeking, and using feedback for performance improvement (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.

London, M., & Sessa, V. I. (2006). Group feedback for continuous learning. Human Ressource Development Review, 5(3), 1-27. doi:10.1177/1534484306290226

Marks, M. A., Mathieu, J. E., & Zaccaro, S. J. (2001). A temporally based framework and taxonomy of team processes. Academy of Management Review, 26,356–376. doi: 10.2307/259182

Mathieu, J. E., & Rapp, T. L. (2009). Laying the foundation for successful team performance trajectories: The role of team charters and performance strategies. Journal of Applied Psychology, 94, 90 –103. doi:10.1037/a0013257

McCarthy, A., & Garavan, T. (2008). Team learning and metacognition: A neglected area of HRD research and practice. Advances in Developing Human Resources, 10, 509-524. doi:10.1177/1523422308320496

McGrath, J. E. (1993). The JEMCO workshop: Description of a longitudinal study, Small Group Research, 24, 285–306. doi:10.1177/1046496493243002

Michinov, N., & Michinov, E. (2009). Investigating the relationship between transactive memory and performance in collaborative learning. Learning and Instruction, 19(1), 43–54. doi:10.1016/j.learninstruc.2008.01.003

Moreland, R. L., & McMinn, J. G. (2010). Group reflexivity and performance. In S. R. Thye & E. Lawler (Eds.), Advances in group processes (Vol. 27, pp. 63-95). Bingley, UK: Emerald Press.

Moreno, R., Mayer, R. E., Spires, H. A., & Lester, J. C. (2001). The case for social agency in computer-based teaching. Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction, 19(2), 177-213. doi:10.1207/S1532690XCI1902_02

Mory, E. H. (2003). Feedback research revisited. In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology (pp. 745-783). Mahwah, NJ: Erlbaum.

Morris, J., & Stew, G. (2007). Collaborative reflection: how far do 2:1 models of learning in the practice setting promote peer reflection, Reflective Practice, 8(3), 419-432. doi:10.1080/14623940701425220

Mulder, R. H., & Ellinger, A. D. (2013). Perceptions of quality of feedback in organizations: Characteristics, determinants, outcomes of feedback, and possibilities for improvement: Introduction to a special issue. European Journal of Training and Development, 37(1), 4-23. doi:10.1108/03090591311293266

Mulder, R. H. (2013). Exploring feedback incidents, their characteristics and the informal learning activities that emanate from them. European Journal of Training and Development, 37(1), 49-71.

Müller, A., Herbig, B. , & Petrovic, K. (2009). The explication of implicit team knowledge and its supporting effect on team processes and technical innovations. An action regulation perspective on team reflexivity. Small Group Research, 40, 28-51. doi:10.1177/1046496408326574

Mullins, D., Rummel, N., & Spada, H. (2011). Are two heads always better than one? Differential effects of collaboration on students' computer-supported learning in mathematics. International Journal of Computer Supported Collaborative Learning, 6(3), 421-443. doi:10.1007/s11412-011-9122-z

Noroozi, O., Weinberger, Biemans, H. J. A., Mulder, M., & Chizari, M. (2012). Argumentation-based computer supported collaborative learning (ABCSCL). A systematic review and synthesis of fifteen years of research. Educational Research Review, 7(2), 79–106. doi:10.1016/j.learninstruc.2008.03.001

Okhuysen, G. A., & Eisenhardt, K. M. (2002). Integrating knowledge in groups: How simple formal interventions help. Organization Science, 13, 370-386. doi:10.1287/orsc.13.4.370.2947

Okhuysen, G., & Waller, M. J. (2002). Focusing on midpoint transitions: An analysis of boundary conditions. Academy of Management Journal, 45, 1056-1065. doi:10.2307/3069330

Phielix, C., Prins, F. J., & Kirschner, P. A. (2010). Awareness of group performance in a CSCL environment: Effects of peer feedback and reflection. Computers in Human Behavior, 26, 151-161. doi:10.1016/j.bbr.2011.03.031

Phielix, C., Prins, F. J., Kirschner, P. A., Erkens, G., & Jaspers, J. G. M. (2011). Group awareness of social and cognitive performance in a CSCL environment: Effects of a peer feedback and reflection tool. Computers in Human Behavior, 27(3), 1087-1102. doi:10.1016/j.bbr.2011.03.031

Prins, F. J., Sluijsmans, D. M. A., & Kirschner, P. A. (2006). Feedback for general practitioners in training: quality, styles, and preferences. Advances in Health Sciences Education, 11, 289-303. doi:10.1007/s10459-005-3250-z

Prinsen, F. R., Terwel, J., Zijlstra, B. J. H., & Volman, M. M. L. (2013). The effects of guided elaboration in a CSCL programme on the learning outcomes of primary school students from Dutch and immigrant families. Educational Research and Evaluation, 19(1), 39–57. doi:10.1080/13803611.2012.744694

Reimann, P. (2007). Time is precious: Why process analysis is essential for CSCL (and can also help to bridge between experimental and descriptive methods). In C. Chinn, G. Erkens & S. Puntambekar (Eds.), Mice, minds, and society. Proceedings of the computer-supported collaborative learning conference (CSCL 2007) (pp. 598–607). New Brunswick, NJ: International Society of the Learning Sciences.

Rourke, L., & Anderson, T. (2004). Validity in quantitative content analysis. Education Technology Research and Development, 52(1), 5-18. doi:10.1007/BF02504769

Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts. International Journal of Artificial Intelligence in Education, 12, 8-22

Rulke, D. L., & Rau, D. (2000). Investigating the encoding process of transactive memory development in group training. Group & Organization Management, 25, 373-396.

Rummel, N., Mullins, D., & Spada, H. (2012). Scripted collaborative learning with the Cognitive Tutor Algebra. International Journal of Computer Supported Collaborative Learning, 7(2), 307-339. doi:10.1007/s11412-012-9146-z

Rummel, N., & Spada, H. (2005). Learning to Collaborate: An Instructional Approach to Promoting Collaborative Problem Solving in Computer-Mediated Settings. Journal of the Learning Sciences, 14(2), 201-241. doi:10.1207/s15327809jls1402_2

Rummel, N., Spada, H., & Hauser, S. (2009). Learning to collaborate from being scripted or from observing a model. International Journal of Computer-Supported Collaborative Learning, 4(1), 69-92. doi: 10.1007/s11412-008-9054-4

Salas, E. Dickinson, T. L., Converse, S. A., & Tannenbaum, S. I. (1992). Toward an understanding of team performance and training. In R. W. Swezey & E. Salas (Eds.), Teams: Their training and performance (pp. 3-29). Norwood, NJ: Ablex.

Salas, E., Rosen, M. A., & King, H. (2007). Managing teams managing crises: Principles of teamwork to improve patient safety in the emergency room and beyond. Theoretical Issues in Ergonomics Science, 8(5), 381-394. doi:10.1080/14639220701317764

Salas, E., Stagl, K. C., & Burke, C. S. (2004). 25 years of team effectiveness in organizations: Research themes and emerging needs. In C. L. Cooper & I. T. Robertson (Eds.), International review of industrial and organizational psychology (pp. 47–91). New York: Wiley.

Savelsbergh, C., Heijden, B. I. J. M. van der, & Poell, R. F. (2009). The development and empirical validation of a multi-dimensional measurement instrument for team learning behaviors. Small Group Research, 40(5), 578-607. doi:10.1177/1046496409340055

Schippers, M. C., Den Hartog, D. N., & Koopman, P. L. (2007). Reflexivity in teams: A measure and correlates. Applied Psychology-An International Review, 56(2), 189-211. doi:10.1111/j.1464-0597.2006.00250.x

Schippers, M. C., Den Hartog, D. N., Koopman, P. L., & Wienk, J. A. (2003). Diversity and team outcomes: The moderating effects of outcome interdependence and group longevity and the mediating effect of reflexivity. Journal of Organizational Behavior, 24, 779-802. doi:10.1002/job.220

Schippers, M. C., Den Hartog, D. N., Koopman , P. L., & van Knippenberg, D. (2008). The role of transformational leadership in enhancing team reflexivity. Human Relations, 61, 1593-1616. doi:10.1177/0018726708096639

Schippers, M. C., Homan, A. C., & van Knippenberg, D. (2013). To reflect or not to reflect: Prior team performance as a boundary condition of the effects of reflexivity on learning and final team performance. Journal of Organizational Behavior, 34(1), 6-23. doi:10.1002/job.1784

Schön, D. (1983). The reflective practitioner: How professionals think in action. New York: Basic.

Scott-Young, C., & Samson, D. (2009). How team efficacy beliefs impact project performance: An empirical investigation of team potency in capital projects in the process industries. World Academy of Science, Engineering and Technology, 3, 6-28. doi:10.1.1.308.8306

Seibert, K. W. (1999). Reflection-in-action: Tools for cultivating on-the-job learning conditions. Organizational Dynamics, 27, 54-65. doi:10.1016/S0090-2616(99)90021-9

Soller, A., Monés, A. M., Jermann, P., & Mühlenbrock, M. (2005). From Mirroring to Guiding: A Review of State of the Art Technology for Supporting Collaborative Learning. International Journal on Artificial Intelligence in Education, 15(5), 261– 290. doi:10.1.1.26.2186

Sims, D. E., Salas, E., & Burke, C. S. (2005). Promoting effective team performance through training. In S. A. Wheelan (Ed.), The Handbook of Group Research and Practice (pp. 407-425). Thousand Oaks, CA: Sage.

Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78, 153-189. doi:10.3102/0034654307313795

Tschan, F., Semmer, N. K., Nägele, C., & Gurtner, A. (2000). Task adaptive behavior and performance in groups. Group Processes and Intergroup Relations, 3(4), 367–386. doi:10.1177/1368430200003004003

Tjosvold, D., Tang, M. M. L., & West, M. (2004). Reflexivity for team innovation in China: The contribution of goal interdependence. Group & Organization Management, 29(5), 540–559. doi:10.1177/1059601103254911

van Ginkel, W., Tindale, R. S., & van Knippenberg, D. (2009). Team reflexivity, development of shared task representations, and the use of distributed information in group decision making. Group Dynamics, 13, 265-280. doi:10.1037/a0016045

van Ginkel, W. P., & van Knippenberg, D. (2009). Knowledge about the distribution of information and group decision making: When and why does it work? Organizational Behavior and Human Decision Processes, 108(2), 218-229. doi:10.1016/j.obhdp.2008.10.003

Van der Haar, S., Segers, M. R. S, & Jehn, K. A. (2013). Towards a contextualized model of team learning processes and outcomes. Educational Research Review, 10, 1-12. doi:10.1016/j.edurev.2013.04.001

Van der Pol, J., Van den Berg, I., Admiraal, W. F., & Simons, P. R. J. (2008). The nature, reception, and use of online peer feedback in higher education. Computers & Education, 51, 1804-1817. doi: 10.1016/j.bbr.2011.03.031

Veenman, M. V. J., & Elshout, J. J. (1999) Changes in the relation between cognitive and metacognitive skills during the acquisition of expertise. European Journal of Psychology of Education, 14, 509-523. doi:10.1007/BF03172976

Veestraeten, M., Kyndt, E., & Dochy, F. (2014). Investigating Team Learning in a Military Context. Vocation and Learning, 7(1), 75-100. doi:10.1007/s12186-013-9107-3

Villado, A. J., & Arthur, W. Jr. (2013). The comparative effect of subjective and objective after-action reviews on team performance on a complex task. Journal of Applied Psychology, 98, 514-528. doi:10.1037/a0031510

Visschers-Pleijers, A. J. S. F., Dolmans, D. H. J. M., De Leng, B. A., Wolfhagen, H. A. P., & Van der Vleuten, C. P. M. (2006). Analysis of verbal interactions in tutorial groups: A process study. Medical Education, 40, 129-137. doi:10.1111/j.1365-2929.2005.02368.x

Volet, S., Summers, M., & Thurman, J. (2009). High-level co-regulation in collaborative learning: how does it emerge and how is it sustained? Learning and Instruction, 19, 128–143. doi:10.1016/j.bbr.2011.03.031

Waller, M. J. (1999). The timing of adaptive group responses to nonroutine events. Academy of Management Journal, 42, 127–137. doi:10.2307/257088

Waller, M. J., Gupta, N., & Giambatista, R. C. (2004). Effects of adaptive behaviors and shared mental models on control crew performance. Management Science, 50, 1534–1544. doi:10.1287/mnsc.1040.0210

Weingart, L. R. (1992). Impact of group goals, task component complexity, effort, and planning on group performance. Journal of Applied Psychology, 77(5), 682–693. doi:10.1037/0021-9010.77.5.682

West, M. A. (Ed.) (1996). Handbook of work group psychology, Chichester: John Wiley & Sons, Ltd.

West, M. A. (2000). Reflexivity, revolution and innovation in work teams. In M. M. Beyerlein, D.A. Johnson & S.T. Beyerlein (Eds), Product development teams, Vol. 5. (pp. 1-29). Stamford, CT: JAI Press.

West, M. A., Garrod, S., & Carletta, J. (1997). Group decision-making and effectiveness: unexplored boundaries. In C.L. Cooper & S.E. Jackson (Eds.), Creating tomorrow’s organizations: A handbook for future research in organizational behavior (pp. 293- 316). Chichester: John Wiley & Sons Ltd.

Williams, K. D. (2010). Dyads can be groups (and often are). Small Group Research, 41, 268-274. doi:10.1177/1046496409358619

Wills, K. V., & Clerkin, T. A. (2009). Incorporating reflective practice into team simulation projects for improved learning outcomes. Business Communication Quarterly, 72(2), 221 227. doi:10.1177/1080569909334559

Yang, M., & Carless, D. (2013). The feedback triangle and the enhancement of dialogic feedback processes. Teaching in Higher Education. 18(3), 285-297. doi:10.1080/13562517.2012.719154

Yoo, Y., & Alavi, M. (2001) Media and group cohesion: Relative influences on social presence, task participation, and group consensus, MIS Quarterly, 25(3), 371-390. doi:10.2307/3250922

Yukawa, J. (2006). Co-reflection in online learning: Collaborative critical thinking as narrative. Journal of Computer-supported Collaborative Learning, 1, 203–228. doi:10.1007/s11412-006-8994-9

Categories	Description	Examples
Evaluating or reviewing performance or strategies	Statements or questions about team performance (e.g., whether the team does/did well, is/was on the right track according to plans or received instructions), the goal of the mission and its requirements, actions and strategies (mis)used, reasons behind success, failure, or problems (e.g., he/she gives examples of behaviours, task or team strategies that may explain why they achieved success or encountered problems during this mission).	“We are going in the wrong direction.’’ “We crashed because we were always too fast.” “Something went wrong, maybe the nose of the plane went too low?”
Looking for alternatives	Suggestions or discussions of alternatives in how they approached the task (at the task or team levels) and of the sequence of actions undertaken. In other words, teams discuss how they could do or could have done differently.	“We could have reduced the speed by pitching up or reducing the throttle.” “We could lower the speed by extending the flaps, pitching up, or lower the gas.” “We could either make a U-turn either still try to lower speed and make a sharp descent.”
Making decisions	Statements clearly depicting a decision about a new direction to take or observable behaviours following a decision. Team members’ utterances depicting very explicit decisions about the way they were going to approach the task or work as a team, explicit statements about the intention to follow decisions made within the team, and explicit reaction to a decision by an action (e.g., by pressing the flaps, pulling the gas controller).	‘’We are going to make a U-Turn” or “This time, you look at the speed indicator and I will pitch down”.