Benefits of the Person-Oriented Perspective for Program Evaluation: Analyzing the Differential Treatment Effects of the Vienna E-Lecturing Program

Abstract

Background: In program evaluation interventions are usually examined for global effects but not always for differential effects. The reasoning behind the focus on global effects is that most of the concepts applied in data analyses, and most of the hypotheses being assessed, are variable-oriented. Consequently, existing differential effects will not be detected through global analyses based on a variable-oriented perspective. Accordingly, the intervention under evaluation will be wrongly considered to be ineffective.

Purpose: This contribution, therefore, argues for the application of person-oriented analyses in addition to the common variable-oriented approach in order to identify the differential effects of an intervention during evaluation.

Setting: Both approaches will be applied in the evaluation of a showcase program conducted in real life setting. The results of the two perspectives will be compared.

Intervention: An internet supported program for the promotion of self-regulated learning in an academic context called Vienna E-Lecturing (VEL).

Research Design: The evaluation of the VEL was conducted with a treatment-control-design. Data was collected at the beginning as well as at the end of the intervention.

Data Collection and Analysis: Data regarding motivation as well as factual knowledge was collected. All motivational variables were assessed using previously published scales consisting of 3-4 items. Factual knowledge in statistics and methodology was tested regarding the aspects of reproduction, comprehension and the production of methodological content.

Findings: The application of a variable-oriented perspective showed small effects with low practical relevance. Through the application of a person-oriented approach, and taking into account the theoretical framework of the intervention, three students groups which differed systematically in their learning motivation could be identified. For two of these groups the intervention was very successful. The discussion provides recommendations on how person-oriented approaches can be used systematically in program evaluation.

 Keywords: person-oriented approach; program evaluation; differential effects; learning motivation; variable-oriented approach; academic context