Behind the numbers. A mixed-methods study of the CLASS tool application in Italy
DOI:
https://doi.org/10.7346/sird-012021-p46Keywords:
quality assessment; mixed-methods; standardized instruments; ECEC; cross-cultural research.Abstract
The increasing globalization of tools used to assess ECEC quality requires reflection on their cultural coherence and ecological validity when applied in cultural contexts other than those of origin, as well as on the meaning of inconsistencies at the psychometric level that sometimes emerge - inconsistencies that quantitative methods alone often struggle to explain adequately. This paper proposes an integrated framework that combines the statistical analysis of quality assessment tools with a qualitative exploration of the meanings and interpretations that accompany their use in other cultures. This mixed-methods study, analyzing the implications of the CLASS Pre-K tool application in Italian preschools, offers a broader reflection on the importance and the need to adopt a critical, culturally-sensitive approach when using assessment tools outside their cultural cradle.
References
Araujo, M. C., Carneiro, P., Cruz-Aguayo, Y., & Shady, N. (2014). A Helping Hand? Teacher Quality and Learning Outcomes in Kindergarten. Working paper, version: October 4, 2014. Retrieved from http://economics.yale.edu/sites/default/files/carneiropedroahelpinghand_final.pdf (accessed on 26th August 2020).
Bouchard, C., Cantin, G., Charron, A., Crépeau, H., & Lemire, J. (2014). Validation du Classroom Assessment Scoring System (CLASS) en maternelle 4 ans à mi-temps au Québec. Paper presented at the 82nd Congres de l’Acfas, Colloque 519, in Montréal, Canada, May 15.
Boyatzis, R. E. (1998). Transforming qualitative information: Thematic analysis and code development. Thousand Oaks, CA: Sage.
Braun, V., & Clarke, V. (2006). Using Thematic Analysis in Psychology. Qualitative Research in Psychology, 3(2), 77-101.
Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.),Testing structural equation models. Newbury Park, CA: Sage.
Cadima, J., Aguiar, C., & Barata, M. C. (2018). Process quality in Portuguese preschool classrooms serving children at-risk of poverty and social exclusion and children with disabilities. Early Childhood Research Quarterly, 45, 93-105.
Creswell, J. W., & Plano Clark, V. L. (2011). Designing and Conducting Mixed Methods Research (2nd ed.). Thousand Oaks, CA: Sage.
Dahlberg, G., Moss, P., & Pence, A. (2007). Beyond Quality in Early Childhood Education and Care: Languages of Evaluation (2nd ed.). London: Falmer Press.
Declercq, B., & Laevers, F. (2015). Measuring ‘Pedagogical’ Quality with an Observation Tool: Insights from the Pilot Study. Paper presented at the 25th EECERA conference, in Barcelona, Spain, September 7-10.
Dessus, P., Cosnefroy, O., & Joët, G. (2014). Evaluer la qualité des interactions enseignant-élèves en début de scolarité:Qualités psychométriques d’une adaptation française du CLASS. Paper presented at the 82nd Congres de l’Acfas, Colloque 519, in Montréal, Canada, May 15.
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological methods, 4, 272-299.
Fenech, M. (2011). An Analysis of the Conceptualisation of ‘Quality’ in Early Childhood Education and Care Empirical Research: Promoting ‘Blind Spots’ as Foci for Future Research. Contemporary Issues in Early Childhood, 12(2), 102-117.
Fenech, M., Sweller, N., & Harrison, L. (2010). Identifying high-quality centre-based childcare using quantitative data-sets: what the numbers do and don’t tell us. International Journal of Early Years Education, 18(4), 283-296.
Field, A. (2009). Discovering statistics using SPSS (3rd ed.). London: Sage publications.
Gagné, P., & Hancock, G. R. (2006). Measurement model quality, sample size, and solution propriety in confirmatory factor models. Multivariate Behavioral Research, 41(1), 65-83.
Geisinger, K. F. (1994). Cross-cultural normative assessment: translation and adaptation issues influencing the normative interpretation of assessment instruments. Psychological assessment, 6(4), 304-312.
Grammatikopoulos, V., Gregoriadis, A. & Zachopoulou, E. (2015). Evaluation of Early Childhood Evaluation Environments and Professional Development: Current Practices and Implications. In O. N. Saracho (Ed.), Contemporary Perspectives on Research in Testing and Evaluation in Early Childhood Education, 153-172. Maryland: Information Age Publishing.
Hamre, B. K., Goffin, S. & Kraft-Sayre, M. (2009). Classroom Assessment Scoring System (CLASS) Implementation Guide: Measuring and Improving Classroom Interactions in Early Childhood Settings. Charlottesville: Teachstone.
Hamre, B. K., Pianta, R. C., Downer, J. T., DeCoster, J., Mashburn, A. J., Jones, S. M., Brown, J. L. et al. (2013). Teaching Through Interactions. The Elementary School Journal, 113(4), 461-487.
Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185.
Hu, B. Y., Fan, X., Gu, C., & Yang, N. (2016). Applicability of the Classroom Assessment Scoring System in Chinese preschools based on psychometric evidence. Early Education and Development, 27(5), 714-734.
Ishimine, K., & Tayler, C. (2014). Assessing Quality in Early Childhood Education and Care. European Journal of Education, 49(2), 272-290.
Kline, R. B. (2011). Principles and practice of structural equation modelling (3rd ed.). New York, NY: Guilford Press.
Lemay, L., Lehrer, J., & Naud, M. (2017). Le CLASS pour mesurer la qualité des interactions en contextes culturels variés. Les dossiers des sciences de l’éducation, 37, 15-34.
Leyva, D., Weiland, C., Barata, M., Yoshikawa, H., Snow, C., Treviño, E., & Rolla, A. (2015). Teacher-child Interactions in Chile and Their Associations with Prekindergarten Outcomes. Child development, 86(3), 781-799.
Limlingan, M. C. (2011). On the Right Track: Measuring Early Childhood Development Program Quality Internationally. Current Issues in Comparative Education, 14(1), 38-47.
MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological methods, 4(1), 84-99.
Malaguzzi, L. (1993). For an Education Based on Relationships. Young Children, 49, 9-12.
Mathers, S., Linskey, F., Seddon, J., & Sylva, K. (2007). Using quality rating scales for professional development: Experiences from the UK. International Journal of Early Years Education, 15(3), 261-274.
Pakarinen, E., Lerkkanen, M. K., Poikkeus, A. M., Kiuru, N., Siekkinen, M., Rasku-Puttonen, H., & Nurmi, J. E. (2010). A Validation of the Classroom Assessment Scoring System in Finnish Kindergartens. Early Education and Development, 21(1), 95-124.
Pagani, V. (2016). Teacher-child relationship quality in ECEC settings. A mixed-methods study of the CLASS tool application in Italy. PhD diss., University of Milano-Bicocca.
Pagani, V., & Pastori, G. (2019). Coniugare valutazione standardizzata e approccio partecipativo-riflessivo. Un’esperienza di ricerca collaborativa nei servizi per l’infanzia italiani. RicercAzione, 11(2), 103-120.
Pastori, G., & Pagani, V. (2017). Is validation always valid? Cross-cultural complexities of standard-based instruments migrating out of their context. European Early Childhood Education Research Journal, 25(5), 682-697.
Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2008). Classroom Assessment Scoring System Pre-K Manual. Baltimore: Brookes Publishing.
Rogoff, B. (2003). The Cultural Nature of Human Development. Oxford: University Press.
Sandilos, L. E., Di Perna, J. C. & The Family Life Project Key Investigators (2014). Measuring Quality in Kindergarten Classrooms: Structural Analysis of the Classroom Assessment Scoring System (CLASS K–3). Early Education and Development, 25(6), 894-914.
Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics (4th ed.). Boston, MA: Allyn and Bacon.
Tobin, J. (2005). Quality in Early Childhood Education: An Anthropologist’s Perspective. Early Education and Development, 16(4), 421-434.
van de Vijver, F., & Poortinga, Y. H. (1997). Towards an integrated analysis of bias in cross-cultural assessment. European journal of psychological assessment, 13(1), 29-37.
Downloads
Published
How to Cite
License
Copyright (c) 2021 Valentina Pagani
This work is licensed under a Creative Commons Attribution 4.0 International License.
The authors who publish in this magazine accept the following conditions:
- The authors retain the rights to their work and give the magazine the right to first publish the work, simultaneously licensed under a Creative Commons License - Attribution which allows others to share the work indicating the intellectual authorship and the first publication in this magazine.
- Authors may adhere to other non-exclusive license agreements for the distribution of the version of the published work (eg deposit it in an institutional archive or publish it in a monograph), provided that the first publication took place in this magazine.
- Authors can disseminate their work online (eg in institutional repositories or on their website) before and during the submission process, as it can lead to productive exchanges and increase citations of the published work.
