Modeling and Measuring Domain-Specific Quantitative Reasoning in Higher Education Business and Economics
Main Article Content
Abstract
Quantitative reasoning is considered a crucial prerequisite for acquiring domain-specific expertise in higher education. To ascertain whether students are developing quantitative reasoning, validly assessing its development over the course of their studies is required. However, when measuring quantitative reasoning in an academic study program, it is often confounded with other skills. Following a situated approach, we focus on quantitative reasoning in the domain of business and economics and define domain-specific quantitative reasoning primarily as a skill and capacity that allows for reasoned thinking regarding numbers, arithmetic operations, graph analyses, and patterns in real-world business and economics tasks, leading to problem solving. As many studies demonstrate, well-established instruments for assessing business and economics knowledge like the Test of Understanding College Economics (TUCE) and the Examen General para el Egreso de la Licenciatura (EGEL) contain items that require domain-specific quantitative reasoning skills. In this study, we follow a new approach and assume that assessing business and economics knowledge offers the opportunity to extract domain-specific quantitative reasoning as the skill for handling quantitative data in domain-specific tasks. We present an approach where quantitative reasoning – embedded in existing measurements from TUCE and EGEL tasks – will be empirically extracted. Hereby, we reveal that items tapping domain-specific quantitative reasoning constitute an empirically separable factor within a Confirmatory Factor Analysis and that this factor (domain-specific quantitative reasoning) can be validly and reliably measured using existing knowledge assessments. This novel methodological approach, which is based on obtaining information on students’ quantitative reasoning skills using existing domain-specific tests, offers a practical alternative to broad test batteries for assessing students’ learning outcomes in higher education.
Article Details
FLR adopts the Attribution-NonCommercial-NoDerivs Creative Common License (BY-NC-ND). That is, Copyright for articles published in this journal is retained by the authors with, however, first publication rights granted to the journal. By virtue of their appearance in this open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.
References
AERA (American Education Research Association), APA (American Psychological Association), & NCME (National Council on Measurement in Education). (2014). Standards for educational and psychological testing. AERA.
Alexander, P. A., & Jetton, T. L. (2003). Learning from traditional and alternative texts: New conceptualization for an information age. In A. C. Graesser, M. A. Gernsbacher & S. R. Goldman (Eds.), Handbook of Discourse Processes (pp. 199–241). Lawrence Erlbaum Associates.
Anderson, J. R. (2005). Cognitive Psychology and its Implications (6th ed.). Worth.
Association of American Colleges and Universities (AAC&U). (2008). College learning for the new global century. https://secure.aacu.org/AACU/PDF/GlobalCentury_ExecSum_3.pdf
Bagozzi, R. P., & Yi, Y. (1988). On the evaluation of structural equation models. Journal of the Academy of Marketing Science, 16(1), 74–94. https://doi.org/10.1007/BF02723327
Ball, D. L. (2003). Mathematical proficiency for all students: Toward a strategic research and development program in mathematics education. RAND Mathematics Study Panel.
Ballard, C. L., & Johnson, M. F. (2004). Basic math skills and performance in an introductory economics class. The Journal of Economic Education, 35(1), 3–23. https://doi.org/10.3200/JECE.35.1.3-23
Bleske-Rechek, A., & Browne, K. (2014). Trends in GRE scores and graduate enrollments by gender and ethnicity. Intelligence, 46, 25–34. https://doi.org/10.1016/j.intell.2014.05.005
Brückner, S., Förster, M., Zlatkin-Troitschanskaia, O., Happ, R., Walstad, W.B., Yamaoka, M., & Asano, T. (2015a). Gender effects in assessment of economic knowledge and understanding: Differences among undergraduate business and economics students in Germany, Japan, and the United States. Peabody Journal of Education, 90(4), 503–518. https://doi.org/10.1080/0161956X.2015.1068079
Brückner, S., Förster, M., Zlatkin-Troitschanskaia, O., & Walstad, W. B. (2015b). Effects of prior economic education, native language, and gender on economic knowledge of first-year students in higher education. A comparative study between Germany and the USA. Studies in Higher Education, 40(3), 437–453. https://doi.org/10.1080/03075079.2015.1004235
Brückner, S., & Pellegrino, J. W. (2016). Integrating the analysis of mental operations into multilevel models to validate an assessment of higher education students’ competency in business and economics. Journal of Educational Measurement, 53(3), 293–312. https://doi.org/10.1111/jedm.12113
Byrne, M., & Flood, B. (2008). Examining the relationship among background variables and academic performance of first year accounting students at an Irish University. Journal of Accounting Education, 26(4), 202–212. https://doi.org/10.1016/j.jaccedu.2009.02.001
Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. Cambridge University Press. https://doi.org/10.1017/CBO9780511571312
CENEVAL (Centro Nacional de Evaluación para la Educación Superior). (2011). Examen General para el Egreso de la Licenciatura en Administración. EGEL-ADMON. CENEVAL.
Davidson, M., & McKinney, G. G. R. (2001). Quantitative reasoning: An overview. Dialogue, 8, 1–5.
ETS (Educational Testing Service). (2016). GRE: Guide to the use of scores 2016-17. https://www.ets.org/s/gre/pdf/gre_guide.pdf
Elrod, S. (2014). Quantitative Reasoning: The Next "Across the Curriculum" Movement. Peer Review, 16(3), 4–8. https://search.proquest.com/docview/1698319962?accountid=14632
Follette, K., Buxner, S., Dokter, E., McCarthy, D., Vezino B., Brock, L., & Prather, E. (2017). The Quantitative Reasoning for College Science (QuaRCS) Assessment 2: Demographic, Academic and Attitudinal Variables as Predictors of Quantitative Ability. Numeracy, 10(1), 1–33. https://doi.org/10.5038/1936-4660.10.1.5
Förster, M., Brückner, S., & Zlatkin-Troitschanskaia, O. (2015). Assessing the financial knowledge of university students in Germany. Empirical Research in Vocational Education and Training, 7(6), 1–20. https://doi.org/10.1186/s40461-015-0017-5
Frey, A., Hartig, J., & Rupp, A. A. (2009). An NCME Instructional Module on Booklet designs in large-scale assessments of student achievement: Theory and practice. Educational Measurement: Issues and Practice, 28(3), 39–53. https://doi.org/10.1111/j.1745-3992.2009.00154.x
Fritsch, S., Berger, S, Seifried, J., Bouley, F., Wuttke, E., Schnick-Vollmer, K., & Schmitz, B. (2015). The impact of university teacher training on prospective teachers’ CK and PCK – a comparison between Austria and Germany. Empirical Research in Vocational Education and Training, 7(4), 1–20. https://doi.rog/10.1186/s40461-015-0014-8
Gaze, E. C., Montgomery, A., Kilic-Bahi, S., Leoni, D., Misener, L., & Taylor, C. (2014). Towards developing a quantitative literacy/reasoning assessment instrument. Numeracy, 7(2), 4. https://doi.org/10.5038/1936-4660.7.2.4
Happ, R., Zlatkin-Troitschanskaia, O., & Förster, M. (2018). How prior economic education influences beginning university students’ knowledge of economics. Empirical Research in Vocational Education and Training, 10(5), 1–20. https://doi.org/0.1186/s40461-018-0066-7
Hart Research Associates. (2009). Learning and assessment: Trends in undergraduate education. A survey among members of the Association of American Colleges and Universities. http://www.aacu.org/membership/documents/2009MemberSurvey_Part1.pdf
ITC (International Test Commission). (2005). ITC Guidelines for Translating and Adapting Tests. https://www.intestcom.org/files/guideline_test_adaptation.pdf
Kuncel, N. R., Wee, S., Serafin, L., & Hezlett, S. A. (2010). The validity of the Graduate Record Examination for master’s and doctoral programs: A meta-analytic investigation. Educational and Psychological Measurement, 70(2), 340–352. https://doi.org/10.1177/0013164409344508
Lusardi, A., & Wallace, D. (2013). Financial literacy and quantitative reasoning in the high school and college classroom. Numeracy, 6(2), Article 1. https://doi.org/10.5038/1936-4660.6.2.1
Madison, B. L. (2009). All the more reason for QR across the curriculum. Numeracy, 2(1), Article 1. https://doi.org/10.5038/1936-4660.2.1.1
Mehler, A., Zlatkin-Troitschanskaia, O., Hemati, W., Molerov, D., Lücking, A., & Schmidt, S. (2018). Integrating computational linguistic analysis of multilingual learning data and educational measurement approaches to explore student learning in higher education. In O. Zlatkin-Troitschanskaia, G. Wittum & A. Dengel (Eds.), Positive learning in the age of information (pp. 145–193). Springer VS. https://doi.org/10.1007/978-3-658-19567-0_10
Mislevy, R. J., & Haertel, G. D. (2006). Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 25(4), 6–20. https://doi.org/10.1111/j.1745-3992.2006.00075.x
Muthén, L. K., & Muthén, B. O. (2012). Mplus user’s guide (7th ed.). Muthén & Muthén.
NRC (National Research Council). (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. https://nap.nationalacademies.org/catalog/13398/education-for-life-and-work-developing-transferable-knowledge-and-skills
OECD (Organisation for Economic Co-operation and Development). (2013). The Survey of Adult Skills: Reader’s Companion. https://www.oecd.org/skills/piaac/Skills%20(vol%202)-Reader%20companion--v7%20eBook%20(Press%20quality)-29%20oct%200213.pdf
O’Neill, P. B., & Flynn, D. T. (2013). Another curriculum requirement? Quantitative reasoning in economics: Some first steps. American Journal of Business Education, 6(3), 339–346. https://doi.org/10.19030/ajbe.v6i3.7814
Owen, A. L. (2012). Student characteristics, behavior, and performance in economics classes. In G. M. Hoyt & K. M. McGoldrick (Eds.), International Handbook on Teaching and Learning Economics (pp. 341–350). Edward Elgar.
Rocconi, L. M., Lambert, A. D., McCormick, A. C., & Sarraf, S. A. (2013). Making college count: an examination of quantitative reasoning activities in higher education. Numeracy, 6(2), 1–20. https://doi.org/10.5038/1936-4660.6.2.10
Roohr, K. C., Graf, E. A., & Liu, O. L. (2014). Assessing quantitative literacy in higher education: An overview of existing research and assessments with recommendations for next-generation assessment. ETS Research Report Series, 2014(2), 1–26. https://doi.org/10.1002/ets2.12024
Satorra, A., & Bentler, P. M. (2010). Ensuring positiveness of the scaled difference chi-square test statistic. Psychometrika, 75(2), 24–248. https://doi.org/10.1007/s11336-009-9135-y
Schreiber, J. B., Nora, A., Stage, F. C., Barlow, E. A., & King, J. (2006). Reporting structural equation modeling and confirmatory factor analysis results. A review. The Journal of Educational Research, 99(6), 323–338. https://doi.org/10.3200/JOER.99.6.323-338
Schuler, H., Funke, U., & Baron-Boldt, J. (1990). Predictive validity of high-school grades: A meta-analysis. Applied Psychology: An International Review, 39(1), 89–103. https://doi.org/10.1111/j.1464-0597.1990.tb01039.x
Shavelson, R. J. (2008). Reflections on quantitative reasoning: An assessment perspective. In B. L. Madison & L. A. Steen (Eds.), Calculation vs. context. Quantitative literacy and its implications for teacher education (pp. 27–44). Mathematical Association of America.
Shavelson, R. J., & Huang, L. (2003). Responding responsibly to the frenzy to assess learning in higher education. Change. The Magazin of Higher Learning, 35(1), 10–19. https://doi.org/10.1080/00091380309604739
Shavelson, R. J., Marino, J. P., Zlatkin-Troitschanskaia, O., & Schmidt, S. (2019). Reflections on the assessment of quantitative reasoning. In B. L. Madison & L. A. Steen (Eds.), Calculation vs. context: Quantitative literacy and its implications for teacher education. Mathematical Association of America.
Tiffin, P. A., McLachlan, J. C., Webster, L., & Nicholson, S. (2014). Comparison of the sensitivity of the UKCAT and A levels to sociodemographic characteristics: A national study. BMC Medical Education, 14(7), 1–12. https://doi.org/10.1186/1472-6920-14-7
Walstad, W. B., Watts, M. W., & Rebeck, K. (2007). Test of understanding in college economics: Examiner’s manual (4th ed.). National Council on Economic Education.
Williams, M. L., Waldauer, C., & Duggal, V.G. (1992). Gender differences in economic knowledge: An extension of the analysis. The Journal of Economic Education, 23(3), 219–231. https://doi.org/10.1080/00220485.1992.10844756
Yamaoka, M., Walstad, W. B., Watts, M. W., Asana, T., & Abe, S. (Eds.). (2010). Comparative studies on economic education in Asia-Pacific region. Shumpusha.
Zahner, D. (2013). Reliability and validity of CLA +. Council for Aid to Education.
Zlatkin-Troitschanskaia, O., Förster, M., Brückner, S., & Happ, R. (2014). Insights from a German assessment of business and economics competence. In H. Coates (Ed.), Higher education learning outcomes assessment (pp. 175–200). Peter Lang.
Zlatkin-Troitschanskaia, O., Förster, M., Schmidt, S., Brückner, S., & Beck, K. (2015). Erwerb wirtschaftswissenschaftlicher Fachkompetenz im Studium. Eine mehrebenenanalytische Betrachtung von hochschulischen und individuellen Einflussfaktoren [Acquisition of economic competence over the course of studies. A multilevel consideration of academic and individual determinants]. In S. Blömeke & O. Zlatkin-Troitschanskaia (Eds.), Kompetenzen von Studierenden (pp. 116–134). Beltz Juventa. https://doi.org/10.25656/01:15506
Zlatkin-Troitschanskaia, O., Shavelson, R. J., & Pant, H. A. (2017). Assessment of learning outcomes in higher education – international comparisons and perspectives. In C. Secolsky & B. Denison (Eds.), Handbook on measurement, assessment and evaluation in higher education (2nd ed.). Routledge.