[1] Corresponding author:  Heidelberglaan 1, 3584 CS Utrecht, Utrecht, The Netherlands, email: l.h.bronkhorst@uu.nl DOI: http://dx.doi.org/10.14786/flr.v4i3.198

1.          Introduction

Educational design research is described as a challenging research approach (e.g. Collins, Joseph, & Bielaczyc, 2004). Different aspects of this complexity are discussed in literature, typically tracing the origin of this complexity back to the multiple aims of educational design research, namely contributing to the general understanding of teaching and learning and creating a viable contextualized design to solve a local problem (Anderson & Shattuck, 2012). Some stress how educational design research might be especially challenging for early career researchers (Herrington, McKenney, Reeves, & Oliver, 2007), defined as researchers with up to ten years of experience (Andres, Bengtsen, Castaño, Crossouard, Keefer, & Pyhältö, 2015), including PhD students. The challenges of educational design research for early career researchers are described as procedural, whereas case studies (e.g. Akkerman, Bronkhorst, & Zitter, 2013) and experience suggest that educational design research might also be challenging for different reasons. While some studies suggest that such challenges can lead to PhD students experiencing dissonance (e.g. Wisker, Robinson, Trafford, Creighton, & Warns, 2003), others suggest that while challenges can be burdensome for PhD students, they can also be experienced as empowering (Stubb, Pyhältö, & Lonka, 2011), benefitting the learning process involved in becoming a researcher (Hall & Burns, 2009).

Being early career researchers with experience in educational design research, we conducted a self-study exploring PhD candidates’ experiences with educational design research in terms of the challenges as well as the learning outcomes. Self-study is an unconventional and relatively unknown method, gaining popularity in research on teacher education (Zwart, Smit, & Admiraal, 2015) as a powerful way of providing insights complementary to those gained from other research methods (Bullough & Pinnegar, 2001; Loughran, 2007). As such, this article can be appreciated as a potentially thought-provoking example of using self-study methodology for studying researcher practices, illustrating the methodology’s potential and pitfalls to critically analyse early career researchers’ developing research practices.

1.1       Educational design research (EDR)

The origin of educational design research (EDR) is often traced back to the works of Brown (1992). Different authors use different terms, including design research, design-based research (or the abbreviation DBR) and design experiments; also, the specific methods used in EDR studies differ (Engeström, 2011a; Reeves, Herrington, & Oliver, 2005). Barab and Squire (2004, p.2) typify EDR as “a series of approaches, with the intent of producing new theories, artifacts, and practices that account for and potentially impact learning and teaching in naturalistic settings”. In their review of the last decade of EDR research, Anderson and Shattuck (2012) characterize EDR as research situated in a real educational context, concentrating on testing a significant intervention, in collaboration with practitioner(s), informed by theories and an assessment of the local context as well as practices in other contexts. Accordingly, EDR uses mixed methods, involves multiple iterations to perfect the design, and requires a collaborative partnership between researcher(s) and practitioner(s), as it focuses on theory development and overcoming a problem in local practice, typically culminating in design principles.

EDR is generally acclaimed as it is assumed to have the potential to enhance theoretical knowledge development while also having public educational value (van den Akker, 1999) and therefore resonates with wider calls for increasing the relevance and impact of educational research (Anderson & Shattuck, 2012). Baptista, Frick, Holley, Remmik, Tesch, and Åkerlind (2015) describe how these calls for relevance are also being voiced in relation to research conducted as part of PhD dissertations, where the usefulness of the knowledge gained by the research is increasingly considered. Increasing attention to and application of EDR is demonstrated by the special issues devoted to this topic by leading educational research journals, including Journal of the Learning Sciences (Barab & Squire, 2004), Educational Researcher (Kelly, 2003) and Educational Psychologist (Sandoval & Bell, 2004).

More and more, EDR is not only applauded, but also critically assessed (Svihla, 2014). Several authors have pointed at potential weaknesses in EDR methodology (Dede, 2004; Shavelson, Phillips, Towne, & Feuer, 2003), questioning EDR’s potential to draw causal claims in natural settings and EDR’s tendency to generalize small scale studies (Kelly, 2004). Akkerman, Bronkhorst, and Zitter (2013) maintain that pursuing concurrent goals in EDR requires immediate and sometimes intuitive actions and decisions, as accepted systems of quality assurance are lacking. Barab and Squire (2004) highlight how EDR requires researchers to carefully balance insider and outsider perspectives: “If a researcher is intimately involved in the conceptualization, design, development, implementation, and re-searching of a pedagogical approach, then ensuring that researchers can make credible and trustworthy assertions is a challenge” (p.10). Additionally, several scholars have addressed the challenges of analysis in EDR, given the large amounts of data that are usually collected (Collins, Joseph, & Bielaczyc, 2004; Kelly, 2004).

Herrington, McKenney, Reeves, and Oliver (2007) have argued that specifically for beginning researchers, EDR might be too challenging. For one, the longitudinal nature of most design studies might extend the four-year period in which most PhD students are expected to complete their studies (Evans, 2010). But even if the data collection itself can be completed, the richness of the data collected might extend the time and especially the skill needed for analysis. These challenges are presented and interpreted as procedural and thereby manageable, as can be deduced from Anderson and Shattuck’s (2012) solution in terms of multi-year multi-actor research agendas. Such solutions help advance EDR, but fail to appreciate the experience of engaging in EDR and its challenges for early career researchers. For instance, a case study of EDR conducted by a PhD candidate stresses the consequences in terms of feelings of insecurity and misfit of a seemingly procedural challenge—namely assuring quality in EDR studies (Akkerman et al., 2013). Recently, Castelló, Kobayaski, McGinn, Pechar, Vekkaila, and Wisker (2015) have called attention to feelings of misfit as a PhD student, as they impact present and future professional aspirations and can lead students to abandoning the field of learning and instruction. Others have also cautioned against the consequences of feelings of dissonance as a PhD student (Wisker et al., 2003).

1.2       Learning to do research

In contrast, Lee and Roth (2003) argue that the learning potential of research actually lies in working through the challenges engaging in research is likely to generate. Others have also hinted at the learning potential of challenging circumstances for learning to do research, especially for early career researchers (e.g. Haigh, 2012; Hopwood, 2010). In such cases, learning is conceptualized as participation (González-Ocampo et al., 2015) and learning outcomes are studied in terms of becoming a researcher, emphasizing not only the technical, but also the personal nature of learning to do research (Barnacle, 2005). “Transitioning […] to the role of researcher is not as simple as acquiring a new set of skills and expanding one’s knowledge of scholarship” (Hall & Burns, 2009, p.53). Departing from this conceptualization, the biographical or narrative process of learning is usually studied, as well as the open-ended and potentially transformative outcomes for researcher identity of engaging in research (e.g. Lee & Roth, 2003). Such studies are in line with widespread calls for more in-depth, longitudinal research to explore what it means to become a researcher (Hall & Burns, 2009; Stubb et al., 2011).

In terms of specific learning outcomes, Herrington and colleagues (2007) also report potential learning outcomes of conducting EDR specifically for PhD students. Most importantly, PhD students can learn to see practitioners as partners in research instead of beneficiaries of the outcomes of their research, thereby potentially increasing the impact of educational research on educational practice. Moreover, PhD students might benefit from learning about the ways in which EDR differs from other research methods. Also stressing the importance of awareness of the implications of methodological decisions made during research, Newbury (2002, p.156, emphasis added) states that “[p]erhaps most important is research students’ exposure to alternative approaches”. Similarly, Pallas (2001) emphasizes how preparing doctoral students for an essentially unpredictable future entails acquainting them with epistemological diversity as consumers (i.e. reading research grounded in diverse epistemologies) as well as producers (i.e. engaging in research with diverse epistemologies).

Summarizing, despite the fact that the challenges and learning potential of engaging in EDR for early career researchers are acknowledged, there is still a great deal to explore about the actual experience of engaging in EDR. In this study, we explore what challenges and learning outcomes PhD students experience when engaging in EDR, as such insight could not only support early career researchers and their supervisors in making their EDR studies successful, but also provide an in-depth insider perspective, informing a wider audience about what it means to become an (educational design) researcher.

2.          Methods

2.1       Context of the study

Cognizant of the differences in early career researcher education across countries (Andres, Bengtsen, Castaño, Crossouard, Keefer, & Pyhältö, 2015), we detail the specific characteristics of PhD trajectories in educational and learning sciences in the Netherlands, where this study was conducted. First of all, PhD trajectories are jobs; PhD candidates do not pay tuition, but are paid for doing research. As such, the phrase ‘PhD student’ is not used in Dutch, as candidates are not seen as students, although they are supervised by at least one full professor and one so-called ‘daily supervisor’ – usually an assistant or associate professor. Additionally, the PhD dissertation consists of four semi-independent articles, written in English and preferably published internationally. Hence, although coursework is included in this trajectory, it usually does not extend throughout the trajectory, as the focus of the four-year trajectory is the research project. Consequently, a completed (research) Master’s degree is a prerequisite to enter a four-year PhD trajectory.

2.2       Self-study

Given the exploratory and interpretative research aim, and conceptualizing PhD learning as participation, we chose to conduct a collaborative self-study, also referred to as auto-ethnography (e.g. Holt, 2003). Self-study is typified as a methodology for studying professional practices that stems from a desire “to be more fully informed about the nature of a knowledge of practice” (Loughran, 2007, p.14). In the field of learning and instruction, self-study is an approach mainly used in research on teacher education and teaching (Zwart, Smit, & Admiraal, 2015). Self-study has become recognized as a powerful methodology to promote critical reflective attitudes, understand the relationship between theory and practice more profoundly, and develop knowledge from an insider perspective (Loughran, 2007; Petrarca & Bullock, 2014; Williams & Ritter, 2010). The increasing use of self-study, reflected in the creation of the self-study journal Studying Teacher Education, can be understood in the light of current debates on methodology—more specifically, debates on how to take into account the contextualization of human thinking and acting, and the importance of both outsider and insider understanding in research on teacher education (e.g., Hamilton, Smith, & Worthington, 2008) and in educational research in general (e.g., Maxwell, 2004a).

2.3       Participants

The authors of this paper are the two participants in this self-study. For clarity, we use pseudonyms and refer to them in the third person, and only use ‘we’ to refer to ourselves as authors of this article. At the time of the study, both participants were in the penultimate year of their PhD trajectory. They had started their PhD trajectories at the same university department in 2008, having completed the same research Master’s degree in two consecutive cohorts. The full professors who supervised their PhD research differed, but the PhD candidates had the same daily supervisor.

Mary was 25 years old at the time of data collection. In her EDR study, she originally aimed to design a tool to support the goal-relatedness of the Master’s thesis supervision by conducting group discussion meetings and individual interviews with five supervisors with a locally good reputation (see also de Kleijn, Meijer, Brekelmans, & Pilot, 2015). Erica was 28 years old. She studied how student teachers’ meaning-oriented learning and deliberate practice could be fostered by collaboratively re-designing two year-long courses in the teacher education program with two pairs of teacher educators, based on design principles developed in prior research[2] (see also Bronkhorst, Meijer, Koster, & Vermunt, 2011; Bronkhorst, Meijer, Koster, Akkerman, & Vermunt, 2013). Mary and Erica informally discussed their progress in their EDR studies and it seemed that they had quite different experiences, which they found striking given their similar background. This triggered the desire for a more systematic exploration of their experiences in EDR by means of a collaborative self-study.

2.4       Interviews

We assumed that a probing interview might help in the explication of challenges and learning outcomes, as self-narratives have been shown to be a powerful methodology for self-studies (Haigh, 2012). Such interviews require well established interview skills and can benefit from the interviewer and interviewee being acquainted (Lichtman, 2006). Therefore, the interviews were conducted by their daily supervisor, Christine, as she knew both Mary and Erica and had extensive experience in conducting qualitative research in general and open interviews specifically.

Christine was asked to conduct individual in-depth interviews with the PhD candidates revolving around broadly defined topics: (1) the EDR research that they had conducted; (2) the challenges they had experienced in their research and how they had dealt with these; and (3) the learning outcomes in the process of becoming a researcher that they attributed to engaging in EDR. These themes were to be addressed longitudinally (i.e. in terms of their development over time).

As a potential learning outcome of engaging in EDR concerns a different relationship with participants (Herrington et al., 2007), Mary and Erica had interviewed the participants in their EDR studies upon completing their studies. Selected fragments concerning the EDR participants’ perception of the PhD candidates from these interview were provided to Christine, to inform her about the participants’ perspectives. Christine designed an open interview structure (see Table 1) adhering to this input and these guiding principles. She explicitly used her knowledge of the PhD candidates to have them explicate more. Both interviews lasted about an hour and a half and were fully transcribed.

Table 1.

Interview themes and example questions

2.5       Analysis

A cross-case analysis of experiences of engaging in EDR was performed. This was preceded by a within-case, narrative, connective analysis (Maxwell, 2004b) of each PhD candidate individually. First, drawing on critical incident technique (Meijer, de Graaf, & Meirink, 2010), in both interviews fragments were selected where challenges and/or (learning) outcomes were substantially discussed. We verified our selections by scrutinizing the transcripts for words that indicated emotions (e.g. ‘doubt’), struggles (e.g. ‘difficult’) and/or words that indicated changes (e.g. ‘different’) or time differences (e.g. ‘now’). Secondly, we traced each of these key experiences backwards and forwards: in the transcripts, we identified the processes by which they came about and how they subsequently developed, as well as factors or individuals that had influenced their origin, development or outcome. In order to triangulate the findings from these interviews with the PhD candidates, the interviews with the participants of both PhD candidates’ EDR studies were also scrutinized for confirming and disconfirming evidence. Thirdly, we compared and discussed our individual findings from the previous steps until a consensus on the relevant themes and their relationships was reached. The quality of this last step was enhanced by a ‘peer-debriefing’ (Guba, 1981), in which a colleague, unfamiliar with the study, read the data and analysis and critically questioned the initial findings.

Based on these steps, we created two descriptions which are presented chronologically in the results section in order to increase legibility and understanding for readers. These descriptions are based on and contain illustrative quotes from the interview transcripts of the interviews with the PhD candidates and of the data from their participants. We used these descriptions for the cross-case analysis. The cross-case analysis was sensitized by our theoretical framework, focusing on developing views on EDR methods and quality, the role of a researcher in EDR, and the process of becoming a researcher.

3.          Results

3.1       Mary

Although an EDR study had been included in Mary’s research plan from the start, she kept postponing it. Her hesitation mainly resulted from the lack of guidelines, protocols or general conventions Mary found in the literature on EDR. She herself considered reliability, in terms of reproducibility, a key quality criterion for scientific research, for which clear and shared conventions were necessary. She generally preferred to be in control of what she was doing:

“This is what makes me insecure and creates chaos in my head. Because there are no guidelines to hold on to and ‘everything goes’. And that I find very difficult. I obviously need boundaries and limitations.”

Additionally, at that time she thought research was about answering questions and proving or demonstrating theories.

Yet, when an EDR study was no longer necessary for completing her thesis, she made the conscious decision to engage in EDR. She knew that EDR was well outside her comfort zone and thus would be fairly challenging for her, but she wanted to be(come) an all-round researcher and, being a PhD student, she counted on her supervisors’ support.

For her EDR study, Mary invited expert thesis supervisors to three collaborative design meetings. She prepared these meetings extensively, but she reasoned that she could not completely control (“board up”) the research, as she sought her participants’ expertise on the topic of thesis supervision, for which she needed their ownership, expertise and creativity. This conscious lack of control over how the process unfolded was a recurring challenge for her, before and between these meetings. She also experienced an ethical dilemma in relation to the time investment she asked of her participants:

“How will I ever tell them that they invested three times two hours, almost an entire working day? How will I ever tell them that I am not able to write [an article] about it? That it eventually does not lead to something that will be part of my dissertation. That was the biggest stress [factor], so to say.”

During the collaborative design meetings, she was immediately confronted with things that did not go as she had imagined, but she surprised herself by being able to adapt to unexpected circumstances successfully:

“I thought: ‘I want to understand what is happening here.’ […] At that time I did not worry at all, as in: ‘Gosh, where is this going?’ It was more in looking back and when I started preparing the next meeting, that I panicked. Because that brought me back in the research mind set. But during the meetings I was mainly curious.”

The fact that she saw how she could relate her participants’ contributions to the literature greatly supported this process, as did their general enthusiasm:

“They were also really captivated by the issue and also appreciated discussing it so much that my fear of ‘They are wasting their time here’ lessened a little. And I also really observed during those meetings that [engaging in the study] brought about things for them as well.”

The planning, time-management and enthusiasm was also recognized by one of her participants, who indicated in the final interview:

“Well, I really liked it, the way that you handled it. I mean, you are enthusiastic but also focused and flexible in the way you handled things. I found that very pleasant.” (EDR participant)

These meetings, the data collection, became a collaborative exploration and Mary’s role as a researcher changed in this process; she was seen more as an expert on the literature of supervision, which contrasted with her earlier experiences with questionnaires and interviews, where she felt like a “nitwit”. Yet, it also meant that sometimes others took on the leadership role and determined the course of action in the meetings. Moreover, Mary did not only ask questions, but was also asked questions in return. She saw this as a sign of ownership on the part of her participants, necessary for the research, and welcomed it as such.

Looking back, engaging in EDR brought about a number of changes for Mary. First, her view on research and research quality has shifted. She would no longer claim reproducibility is a key criterion for scientific research. Instead, transparency has become crucial and the dialogue with theory is now her first and foremost connotation with scientific research. Consequently, she would design future studies differently, leaving room to deviate from her original plan. Similarly, she would now say that research is about understanding and asking questions, rather than answering them.

By engaging in EDR, she now has a clearer view on it, but she still perceives confusion between different perspectives on EDR:

“I kind of have the impression that we have yet to agree on what [educational design research] actually is. Or at least that there are a lot of different perspectives on it. So it has become clearer for me what I would consider good design research.”

As her EDR was actually a quest for understanding and did not result in a design, she doubts whether she has actually engaged in EDR, an idea with which she now feels comfortable:

“I don’t mind that at all. That was not my first priority, or goal. I consider such a design a means. What this resulted in—what I had not imagined in advance—is the [increased] understanding.”

Now, she would question the time investment asked of participants in completely controlled research:

“Can you ask people to…what does it entail to ask people to participate in such a highly structured interview, in which there is hardly any room for their own input? In such a way that you as a researcher are only ‘taking’?”

Moreover, she now sees the dialogue with theory, present already during data collection, as a key characteristic for research quality in general.

These shifting perspectives also concern how she sees herself as a researcher. She no longer believes in the need to choose between the paradigms or ‘teams’ she perceived before, but feels comfortable as a multi-faceted and most of all curious researcher. She prefers certain types of research, including a preference for control and statistics, without considering these to be better types of research.

“I immediately get the jibbers when I am assigned to a team, whether that is qualitative or quantitative. I like to think that the [research] question is leading, so to speak. The research design is then a means to answer the question, to put it like that.”

She now knows she is capable of doing different types of research to satisfy her curiosity.

3.2       Erica

Erica’s research goal was to study an intervention in educational practice, by having the educators (i.e. the participants in her study) experiment and explore the effects of that experimentation. The two main guiding principles for her research were that, first, educators should have agency in shaping educational design, as she believed that researcher control is not desirable nor possible in general, and, second, that the agency and expertise of the educators would actually make the design better.

“So in that sense, I hoped that they would try out things for me, but I also hoped that the things they would then try out would be enriched with what they knew. So not just [based on] my theoretical knowledge.”

Based on the literature she had read about EDR, she thought her research would be a somewhat structured research endeavour. She started her project combining ideas resulting from her Master’s with her intuitive ideas about how “the world works”. Initially, these ideas were almost contradictory.

“If you look at my research design, it really falls between two stools. The one is how I was educated, with large scale and more objective instruments on which I would have no further influence. And [on the other hand] apparently my intuitive ideas about how such a thing works and which data you need to collect for that.”

In her research design, she mainly attributed the valuable expertise required in this research to the educators, as she felt she had very little knowledge of the educational context, also calling herself a “nitwit”.

Upon engaging in the research, Erica noticed how the process took its own turn, which she—contrary to her expectations—could not really characterize as structured nor cyclical, because every decision was grounded in a prior decision. Moreover, she noticed how she herself felt reluctant to exert agency. At some point, her participants asked her to share more of her knowledge and expertise, as they indicated in their interview:

“At some point we said: […] ‘We want to hear your opinion. […] Especially [as it is] a different perspective. That is the added value.’” (EDR participant)

This made her realize that her research would benefit from combining different expertise, including her own. She became more proactive and felt more at ease with steering the collaborative design according to her own agenda. She noticed how she recognized possibilities in which she could influence the educators’ engagement, relying on communicative techniques she normally did not associate with research (e.g. purposeful small talk) and which she had acquired elsewhere.

“I had not considered [….] that I would apply those things. And that I would consider that they are part of research, [which] I had not really imagined. I thought it would be […] much less interaction, or actually, much less personal.”

This led to ethical dilemmas as she questioned if it “was allowed” to act in this way deliberately in research.

“But because I feel as though the values of education and research are different, and people always think that research is objective, and spotless and ethical and responsible, that makes it feel worse when you apply these [communicative techniques] as a researcher.”

All in all, this meant that her research design relied more and more on her intuition and less on what she assumed other educational researchers would prefer. Intuitively she thought her research “could not be done differently”, but rationally she feared the response of the educational and learning sciences community at large. Especially at conferences, Erica often realized how her research differed from research presented:

“Actually at each research conference I attended, where a different type of research was presented, each time I thought: ‘I do that completely differently…why do I do that so differently?’ Also because I had trouble indicating what it was exactly that I was doing, and why I thought it was important.”

This in turn led to doubts about herself as a researcher, which she hid from her supervisors, along with specific details about what she was really doing in her research.

“During the year I had my doubts [about] if I was a researcher after all, or someone who participated. Or someone who was very meaningful for the educators, but who doesn’t amount to anything in the research context.”

Her participants recognized these doubts, as they indicated during their interview:

“In that sense, I did not envy her [Erica], the past year. I was aware that she put herself in a difficult position, by choosing this [research] approach. […] Because it means that she has a lot to legitimize in the research domain.” (EDR participant)

In dealing with these challenges, she relied on two strategies. First, she made an effort to explicate her intuition(s), which was rather implicit at the start of her study. Putting her rationale for the research in writing enabled her to have faith in the study’s value for research alongside its value for educational practice. Secondly, she purposefully shared this vision, first at conferences and later also with her supervisors. The feedback was positive, which made her trust her intuition more.

“In that sense the AERA was also important. [...It was where] the keynote of Engeström [2011b] took place and then I felt supported in: Okay, it might not be common what I assert, but there are people who’d like to hear about it.”

As such, she feels that perhaps her vision on research has not changed much, but she now understands how more collaborative research designs, and the active role assigned to participants in these, benefit research next to educational practice, which was a crucial outcome for her.

In terms of research and research quality, she assigns increased importance to (ecological) validity: measuring what one intends to measure in context. She describes how she might not feel comfortable doing educational research in decontextualized settings any more. In general, she would now argue that research control over a situation, often employed to yield reliability, might produce unnatural behaviour. She prefers to rely more on transparency and the available knowledge on human behaviour in designing her studies:

“In that sense I started thinking less about how a research intervention ought to be and more about, ‘What do we know about how people learn?’”

Looking back, Erica now thinks that her perspective on EDR or intervention research in general may not be shared by all, but is supported by some whom she holds high. Consequently, she would now call herself a researcher, albeit one with a specific perspective, namely: ‘We have as much to learn from practice as they do from us’. This in turn has increased her confidence towards the research community and her participants.

3.3       Cross-case comparison

3.3. 1   Challenges of EDR

Mary and Erica both describe how engaging in EDR was challenging, especially with respect to two archetypal characteristics of EDR: the cyclical process, including multiple iterations, and the relationship with participants. Both describe how their experience with EDR differs from how EDR is presented in the literature, specifically as being simplified and structured. Mary had expected this in advance, which was one of the main reasons for postponing her EDR study, whereas Erica was somewhat surprised, despite the fact that the complexity of EDR is described in the literature. Apparently, reading about EDR’s complexity—including the necessity of ongoing decisions and the challenges of balancing the multiple stakes and stakeholders involved—does not fully account nor prepare for the experience of EDR. Only when actually engaging in the research did it become clear to both PhD students what a cyclical research process entailed, indicating that learning by doing seems necessary for learning to conduct EDR.

            Despite having taken rather different approaches, Erica and Mary both experienced ethical dilemmas with respect to their role in relation to the role of the participants. The nature of their dilemmas differed. Erica struggled with the fact that she purposely invested in social interaction with her participants in order for them to be committed to her study, which she found ethically questionable in conducting research. The participants in her EDR study do not mention having noted such behaviour when describing the working relationship, which they felt was appropriate. Mary, on the other hand, struggled with the fact that her participants invested time in her study while she herself was not even sure about whether the study would result in a chapter in her dissertation. In contrast, the participants in her EDR study only mentioned how inspiring their participation in the EDR study had been for them.

3.3.2    Becoming a researcher

EDR necessitated dealing with challenges and the resulting feeling of misfit. Mary had anticipated that she personally would not be a good fit with EDR; she questioned whether she would be able to cope with scant guidelines about how to carry out the study, mainly in light of her prior experience with more controlled research and personal preference for structured activities. Erica experienced a lack of fit between her EDR approach and what she considered to be the “general research community” as she questioned whether her collaborative approach to research, initially mainly informed by previous experiences outside academia, would be accepted and understood by the general educational research community, where other standards appeared to exist.

Conducting their EDR studies involved working through these challenges and dealing with their insecurities, thereby fuelling reflections on becoming and being a researcher. Ironically in light of the insecurities, the most salient outcome is that in the end both PhD students describe a development from seeing themselves as ‘nitwits’ or novice researchers to being seen as an expert in their specific fields of study. In both cases, the participants in their EDR studies played an important part in the transition, as they explicitly asked the PhD candidates to take an expert role and to share their knowledge, even before the candidates themselves felt comfortable doing so.

3.3.3    Quality of educational research

Next to reflections on their own expertise, the challenges experienced in engaging in EDR triggered contemplations on the quality of educational research. Both PhD candidates described that after having engaged in EDR, they regarded transparency as one of the most important criteria for educational research, more than replicability. This appeared to result from the fact that replicability was impossible to achieve in EDR, which would imply that their EDR was not scientific. Yet, in hindsight, the PhD candidates do not question their research, as the ongoing dialogue with theory and the general quest for understanding made it scientific, in their opinion (and the acceptance of their papers in international journals supports that assertion). Both PhD candidates do, however, question the feasibility of replicability when it is operationalized in terms of complete researcher control.

4.          Conclusion and discussion

This study departed from contrasting findings of previous studies concerning the challenges of PhD students when engaging in educational design research (EDR). For one, the challenges of EDR are described as procedural, but also as fundamental in the literature. Moreover, the consequences of dealing with such challenges are evaluated as burdensome, but also as empowering, benefitting the process of becoming a researcher. To explore these contrasting findings, this explorative study examined the experiences of PhD candidates engaging in EDR, focusing on challenges and learning outcomes.

Our findings show that the challenges experienced concerned two typical aspects of EDR, namely the cyclical nature of an EDR process and the role of participants (e.g., Collins et al., 2004). More specifically, our analysis shows how the cyclical nature of EDR required ongoing decisions within limited time. Yet, it was not the procedural aspect of making these decisions, but the candidates’ awareness of the significant implications of these decisions for the research (cf. Newbury, 2002) that was experienced as challenging. Similarly, our findings reaffirm the potential EDR to support PhD students in learning to see practitioners as partners in research, rather than beneficiaries of the outcomes of their study (Herrington et al., 2007). Yet, for the PhD candidates not interacting with participants, but balancing the multiple stakes and stakeholders involved – including the research community - was troubling and caused a lot of doubt and insecurity. It follows that while the iterative nature and the role of the participants are manageable for PhD candidates engaging in EDR, the (perceived) conflicts with accepted quality standards of research can make them disruptive (as suggested by Akkerman, et al., 2013).

The PhD candidates’ experiences in EDR contained both ambiguity and novelty, a combination that often proves to be quite challenging (cf. Wisker et al., 2003). However, the analysis illustrates how working through the challenges actually made the engagement in EDR educative, as this necessitated learning about alternative perspectives on and approaches to research and research quality. This resulted in a more elaborated perspective on EDR and research quality in general, and a more pronounced understanding of what kind of researcher the PhD candidates hoped to become. Finding that the learning outcomes are intertwined with the challenges extends previous findings on PhD learning, wherein PhD experiences were categorized as burdensome or empowering (Stubb et al., 2011), by illustrating that burdensome experiences can become empowering over time. More specifically, our findings show that in the end both PhD candidates describe having developed a more refined perspective on research quality, alongside a new understanding of what it means to be a researcher. The latter finding is in line with a conceptualization of PhD learning as a process of participation (González-Ocampo et al., 2015) and learning outcomes in terms of becoming (Hall & Burns, 2009). This resonates with Pallas (2001) who, among others, stresses the importance of learning about and from epistemological diversity in research preparation. Hall and Burns (2009, p.61) draw attention to how learning about multiple and even contrasting epistemological approaches can inform PhD students about the researcher they want to become, as “becoming a professional researcher requires students to negotiate new identities and reconceptualize themselves both as people and professionals in addition to learning specific skills” (p.49).

Generally speaking, the results of this study indicate that describing the challenges of EDR for early career researchers in terms of the necessary time and technical skills needed for the data collection and analysis, is insufficient. Alternatively, we would ascribe at least some of EDR’s complexity and learning potential for PhD students to two other aspects: EDR’s relatively new and minority position in educational sciences, and the role a researcher needs to assume to make EDR a success.

First, EDR was not a mainstream research design in the Netherlands at the time of study and was only marginally discussed in doctorate curricula (cf. Wilhelm, Craig, Glover, Allen, & Huffman, 2000, for a similar discussion of qualitative research). The PhD candidates therefore did not have experience in designing, conducting or even reading about EDR. Moreover, some EDR characteristics differ from what the PhD candidates learned about research, as EDR aims to generate hypotheses rather than to prove them; is interactive rather than objective or distant; and relies on emergent instead of completely controlled designs (Collins et al., 2004). Secondly, the relationship with the participants in EDR also differs in many respects from other types of research, notably from those in which these PhD candidates were educated. The relationship with participants is a cornerstone of EDR and required Mary and Erica to take on a specific role as a researcher. Our findings indicate that two aspects of this role can be quite challenging. First, in EDR the participants and researchers are assumed to collaborate. It follows that part of the control over the research process is handed over to the participants (Edwards, Sebba, & Rickinson, 2007). Sharing the control over the research process is precisely what differentiates interventions that seek to build on participant agency and those that solely recognize researcher expertise, according to Engeström (2011a). As the implications of shared research control have only recently become topic of debate, the PhD candidates had few examples or guidelines to build on. Secondly, the participants acknowledged and addressed the PhD researchers as knowledgeable or even experts on their research topic. This expert role not only disagreed with the PhD candidates’ perception of themselves as novices and their role as junior in the research domain, it it also implied that they as researchers might have had influence over the research content – something they have learned to avoid at all cost. These differences in perspectives highlight the merit of checking participant perspectives on the research, also as a way of quality assurance (see also Bronkhorst et al., 2013).  

Additionally, our analysis suggests that not only the causes, but also the experience of EDR’s complexity, especially for early career researchers, is not fully represented entirely in the literature. The EDR literature describes how EDR is and should be a cyclical process and a collaborative exploration for both researcher and participants, but the specifics of such processes and collaborations are typically not discussed extensively. Akkerman and colleagues (2013) even suggest that there might be a difference between how EDR studies are reified as structured while “a lot of design researchers in praxis act differently” (p.422). For those learning to do research, being aware of potential differences between EDR practice and its reification might prove valuable.

4.1       Limitations

In light of calls for more in-depth research (Stubb et al., 2011), attending to experiences of engaging in EDR from a contextualized insider perspective was our main argumentation for choosing to conduct a self-study. While our findings highlight self-study’s possibilities for increased understanding of research practices, there are some pitfalls that deserve attention. For one, seeing as the content of the interviews was partly determined by the PhD candidates, ways in which EDR was not experienced as challenging or did not carry learning potential have received limited attention. In terms of interpretation of findings, distinguishing personally relevant findings from findings relevant for the field requires an outsider perspective in self-study. Therefore, we asked a colleague to be involved in verifying our analysis, supporting us in being less biased. Seeing as the anonymous reviewers also played a vital role in interpreting our findings from a wider perspective, we underscore the importance of debating self-study methods and findings publicly—in line with calls from the self-study community of teacher education (see, for instance, Loughran, 2007).

4.2       Implications

The findings illustrate how the experience of engaging in research in general, and EDR in particular, deserves more attention and support, which holds significant implications for designing and supervising early career researchers. In both cases studied, the EDR experience evolves from challenging to carrying learning potential, but not without effort. Therefore, for those debating whether or not EDR is (too) challenging for PhD students, we would like to stress how our study shows that engaging in EDR can challenge PhD students to develop and extend their methodological competence, to adapt and develop methodologies and perceive methodology as a field of study in itself. Few would not consider these to be valuable outcomes. Evans (2010), for instance, holds that they align with an extended understanding of professionality of early career researchers and Newbury (2002) claims that they lie at the core of methodological reflexivity, which he considers key to researcher preparation. However, for these challenges to be educative, our findings suggest there needs to be space, not only in terms of time and resources, but also conceptually and methodologically, and support for PhD students to work through such challenges. Unfortunately, such space is not always available in our age of efficiency and accountability (Biesta, 2010).

In terms of support, our findings also draw attention to how engaging in EDR plays out differently in light of PhD students’ participation in other communities, echoing the results of Castelló and colleagues (2015). For supervisors, it is important to realize that conducting EDR as an early career researcher brings about a range of insecurities and challenges that can differ depending on researcher personalities, previous experiences with research, and participation in other communities. Given the important role supervisors play in facilitating the successful completion of a PhD trajectory (González-Ocampo et al., 2015), we suggest that supervisors need to adapt their supervision to the specific challenges that PhD students experience, referred to as “adaptivity” in the context of in the context of Master’s thesis supervision (de Kleijn, Bronkhorst, Meijer, Pilot, & Brekelmans, 2014). Yet, PhD students and supervisors should not avoid insecurities altogether, as they can carry learning potential.

A final implication concerns the methodology used in our study. Typically, the professional practices under scrutiny in self-study concern teaching, with the exception of studies focusing on self-study as a method. There are very few examples in which research practices are the object of study (for an example, see Bronkhorst, van Rijswijk, Meijer, Koster, & Vermunt, 2013). In research on teacher education, self-study has become recognized as a powerful methodology for teachers and teacher educators to promote local professional (knowledge) development, as well as develop relevant knowledge for the field from an insider perspective (Petrarca & Bullock, 2014; Williams & Ritter, 2010). Our experiences in this study suggest that invoking self-study to understand research practices can enrich our understanding in more than one way. Not only by means of our findings, foregrounding aspects of engaging in EDR undisclosed in the literature, but also by critically analysing and openly discussing the challenges involved in conducting research and becoming an educational researcher.

Keypoints

       Studies PhD students’ experiences of engaging in educational design research.

       Finds specific EDR challenges and learning outcomes pertaining to early career researchers.

       Relates these challenges to EDR’s position in educational sciences.

       Advances self-study as a way for PhD students to learn about research practices.

Acknowledgements

References

Akkerman, S. F., Bronkhorst, L. H., & Zitter, I. (2013). The complexity of educational design research. Quality & Quantity, 47(1), 421-439. doi: 10.1007/s11135-011-9527-9

Anderson, T., & Shattuck, J. (2012). Design-based research: a decade of progress in education research? Educational researcher, 41(1), 16-25. doi: 10.3102/0013189X11428813

Andres, L., Bengtsen, S. S., Castaño, L. G., Crossouard, B., Keefer, J. M., & Pyhältö, K. (2015). Drivers and interpretations of doctoral education today: National comparisons. Frontline Learning Research, 3(3), 1-18. doi: 10.14786/flr.v3i3.177

Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1), 1-14. doi: 10.1207/s15327809jls1301_1

Barnacle, R. (2005). Research education ontologies: Exploring doctoral becoming. Higher Education Research & Development, 24(2), 179-188. doi: 10.1080/07294360500062995

Baptista, A., Frick, L., Holley, K., Remmik, M., Tesch, J., & Åkerlind, G. (2015). The doctorate as an original contribution to knowledge: Considering relationships between originality, creativity, and innovation. Frontline Learning Research, 3(3), 51-63. doi: 10.14786/flr.v3i3.147

Biesta, G. J. (2010). Why ‘what works’ still won’t work: From evidence-based education to value-based education. Studies in Philosophy and Education, 29(5), 491-503. doi: 10.1007/s11217-010-9191-x

Bronkhorst, L. H., Meijer, P. C., Koster, B., & Vermunt, J. D. (2011). Fostering meaning oriented learning and deliberate practice in teacher education. Teaching and Teacher Education, 27, 1120-1130. doi: 10.1016/j.tate.2011.05.008

Bronkhorst, L. H., Meijer, P. C., Koster, B., Akkerman, S. F., & Vermunt, J. D. (2013). Consequential research designs in research on teacher education. Teaching and Teacher Education, 33, 90-99. doi: 10.1016/j.tate.2013.02.007

Bronkhorst, L. H., van Rijswijk, M. M., Meijer, P. C., Koster, B., & Vermunt, J. D. (2013). University teachers’ collateral transitions: continuity and discontinuity between research and teaching. Infancía y Aprendizaje, 36, 293-308. doi: 10.1174/021037013807532972

Brown, A.L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2(2), 141-178. doi: 10.1207/s15327809jls0202_2

Bullough, R. V., & Pinnegar, S. (2001). Guidelines for quality in autobiographical forms of self-study research. Educational researcher, 30(3), 13-21. doi: 10.3102/0013189X030003013

Castelló, M., Kobayaski, S., McGinn, M., Pechar, H., Vekkaila, J., & Wisker, G. (2015). Researcher identity in transition: Signals to identify and managesSpheres of activity in a risk-career. Frontline Learning Research, 3(3), 35-50. doi: 10.14786/flr.v3i3.149

Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. Journal of the Learning Sciences, 13(1), 15-42. doi: 10.1207/s15327809jls1301_2

Dede, C. (2004). If design-based research is the answer, what is the question? A commentary on Collins, Joseph, and Bielaczyc; diSessa and Cobb; and Fishman, Marx, Blumenthal, Krajcik, and Soloway in the JLS special issue on design-based research. Journal of the Learning Sciences, 13(1), 105-114. doi: 10.1207/s15327809jls1301_5

Edwards, A., Sebba, J., & Rickinson, M. (2007). Working with users: Some implications for educational research. British Educational Research Journal, 33(5), 647-661. doi: 10.1080/01411920701582199

Engeström, Y. (2011a). From design experiments to formative interventions. Theory & Psychology, 21, 598-628. doi:10.1177/0959354311419252

Engeström, Y. (2011b). Intervening to shape the future. Keynote given at the Annual AERA Conference, New Orleans, LO.

Evans, L. (2010). Developing the European researcher: ‘extended’ professionality within the Bologna process. Professional Development in Education, 36, 663-677. doi:10.1080/19415251003633573

González-Ocampo, G., Kiley, M., Lopes, A., Malcolm, J., Menezes, I., Morais, R., & Virtanen, V. (2015). The curriculum question in doctoral education. Frontline Learning Research, 3(3), 19-34. doi: 10.14786/flr.v3i3.191

Guba, E. G. (1981). Criteria for assessing the trustworthiness of naturalistic inquiries. Educational Technology Research and Development, 29(2), 75-91. doi:10.1007/BF02766777

Haigh, N. (2012). Historical research and research in higher education: reflections and recommendations from a self-study. Higher Education Research & Development, 31(5), 689-702. doi: 10.1080/07294360.2012.689955

Hall, L., & Burns, L. (2009). Identity development and mentoring in doctoral education. Harvard Educational Review, 79(1), 49-70. doi: 10.17763/haer.79.1.wr25486891279345

Hamilton, M. L., Smith, L., & Worthington, K. (2008). Fitting the methodology with the research: An exploration of narrative, self-study and auto-ethnography. Studying Teacher Education, 4(1), 17-28. doi: 10.1080/17425960801976321

Herrington, J., McKenney, S., Reeves, T., & Oliver, R. (2007). Design-based research and doctoral students: Guidelines for preparing a dissertation proposal. In C. Montgomerie, & J. Seale (Eds.), Proceedings of EdMEdia 2007: Wolrd conference on education multimedia, hypermedia & telecommunications (pp. 4089-4097). Chesapeake, VA: AACE.

Holt, N. L. (2003). Representation, legitimation, and autoethnography: An autoethnographic writing story. International Journal of Qualitative Methods, 2(1), 18-28. doi: 10.1177/160940690300200102

Hopwood, N. (2010). Doctoral students as journal editors: non‐formal learning through academic work. Higher Education Research & Development, 29(3), 319-331. doi: 10.1080/07294360903532032

Kelly, A. E. (2003). Theme issue: The role of design in educational research. Educational Researcher, 32(1), 3-4. doi: 10.3102/0013189X032001003

Kelly, A. E. (2004). Design research in education: Yes, but is it methodological? Journal of the Learning Sciences, 13(1), 115-128. doi: 10.1207/s15327809jls1301_6

Kleijn, R. A. M. de, Bronkhorst, L. H., Meijer, P. C., Pilot, A., & Brekelmans, M. (2014). Understanding the up, back, and forward-component in master's thesis supervision with adaptivity. Studies in Higher Education, 1-17. doi: 10.1080/03075079.2014.980399

Kleijn, R. A. M. de, Meijer, P. C., Brekelmans, M., & Pilot, A. (2015). Adaptive research supervision: exploring expert thesis supervisors' practical knowledge. Higher Education Research & Development, 34(1), 117-130. doi: 10.1080/07294360.2014.934331

Lee, S., & Roth, W. (2003). Becoming and belonging: Learning qualitative research through legitimate peripheral participation. Forum: Qualitative Social Research, 4(2), Available online at: http://www.qualitative-research.net/index.php/fqs/article/view/708 (accessed October 26, 2012).

Lichtman, M. (2006). Qualitative research in education. A user's guide. Thousand Oaks, London, New Delhi: Sage Publications, Inc.

Loughran, J. (2007). Researching teacher education practices responding to the challenges, demands, and expectations of self-study. Journal of Teacher Education, 58(1), 12-20. doi: 10.1177/0022487106296217

Maxwell, J. A. (2004a). Causal explanation, qualitative research, and scientific inquiry in education. Educational researcher, 33(2), 3-11. doi: 10.3102/0013189X033002003

Maxwell, J. A. (2004b). Using qualitative methods for causal explanation. Field Methods, 16(3), 243-264. doi:10.1177/1525822X04266831

Meijer, P. C., de Graaf, G., & Meirink, J. (2011). Key experiences in student teachers’ development. Teachers and Teaching: theory and practice, 17(1), 115-129. doi: 10.1080/13540602.2011.538502

Newbury, D. (2002). Doctoral education in design, the process of research degree study, and the ‘trained researcher’. Art, Design and Communication in Higher Education, 1(3): 149-159. doi: 10.1386/adch.1.3.149

Pallas, A. M. (2001). Preparing education doctoral students for epistemological diversity. Educational researcher, 6-11. doi: jstor.org/stable/3594455

Penuel, W. R. (2014). Emerging forms of formative intervention research in education. Mind, Culture, and Activity, 21(2), 97-117. doi: 10.1080/10749039.2014.884137

Petrarca, D., & Bullock, S. M. (2014). Tensions between theory and practice: Interrogating our pedagogy through collaborative self-study. Professional Development in Education, 5, 265-281. doi:10.1080/19415257.2013.801876

Reeves, T. C., Herrington, J., & Oliver, R. (2005). Design research: A socially responsible approach to instructional technology research in higher education. Journal of Computing in Higher Education, 16(2), 96-115. doi:10.1007/BF02961476

Shavelson, R. J., Phillips, D. C., Towne, L., & Feuer, M. J. (2003). On the science of education design studies. Educational Researcher, 32(1), 25-28. doi: 10.3102/0013189X032001025

Sandoval, W. A., & Bell, P. (2004). Design-based research methods for studying learning in context: Introduction. Educational Psychologist, 39(4), 199-201. doi:10.1207/s15326985ep3904_1

Svihla, V. (2014). Advances in design-based research. Frontline Learning Research, 2(4), 35-45. doi: 10.14786/flr.v2i4.114

Stubb, J., Pyhältö, K., & Lonka, K. (2011). Balancing between inspiration and exhaustion: PhD students' experienced socio-psychological well-being. Studies in Continuing Education, 33(1), 33-50. doi: 10.1080/0158037X.2010.515572

van den Akker, J. (1999). Principles and methods of development research. In J. van den Akker, N. Nieveen, R. M. Branch, K. L. Gustafson & T. Plomp (Eds.), Design methodology and developmental research in education and training (pp. 1-14). The Netherlands: Kluwer Academic Publishers.

Wilhelm, R. W., Craig, M. T., Glover, R. J., Allen, D. D., & Huffman, J. B. (2000). Becoming qualitative researchers: A collaborative approach to faculty development. Innovative Higher Education, 24(4), 265-278. doi: 10.1023/B:IHIE.0000047414.56668.5b

Williams, J., & Ritter, J. K. (2010). Constructing new professional identities through self‐study: From teacher to teacher educator. Professional Development in Education, 36, 77-92. doi:10.1080/19415250903454833

Wisker, G., Robinson, G., Trafford, V., Creighton, E., & Warnes, M. (2003). Recognising and overcoming dissonance in postgraduate student research. Studies in Higher Education, 28(1), 91-105. doi: 10.1080/03075070309304

Zwart, R. C., Smit, B., & Admiraal, W. F. (2015). A closer look at teacher research: a review study into the nature and value of research conducted by teachers. Pedagogische Studien, 92(2), 131-149.