Colloquium "Methods of Empirical Educational Research” 2020
Once again the annual colloquium "Methods of Empirical Educational Research" was held from the 8th to 15th of March 2020 in Hirschegg, Austria. In the course of the colloquium, 13 experts discussed contributions on current methodological developments within the scope of educational research.
On behalf of the work unit, talks were held by:
- Andreas Frey on taking item position effects into account in the continuous calibration of large-scale assessments
- Christoph König on the question, if theoretical assumptions towards the parameter precision in Bayesian hierarchical models would also apply to corresponding IRT models
- Aron Fink on the consideration of the uncertainty in item parameter estimates in computerized adaptive testing
We look once again back at a very stimulating time in Kleinwalsertal and give our thanks to all participants!
March 23th, 2020
Andreas Frey appointed as expert for research methods
Andreas Frey was appointed as expert for research methods to the advisory board of the research project "Promotion of social and emotional competences in vocational education and training: A study on medical assistants in training" at the Federal Institute for Vocational Education and Training (Bonn).
December, 17th, 2019
Workshop on the improvement of university exams available as video-on-demand
The workshop on the KAT-HS concept is now available free of charge via our website https://kat-hs.uni-frankfurt.de as video-on-demand.
Our workshop on the improvement of university exams (Introduction to the construction, administration and evaluation of competence-oriented e-exams) took place on 8th of October 2019 and was recorded on video. We have cut, edited and supplemented the videos to create a compact and informative online workshop on the construction of fair exams.
The video workshop provides a comprehensive overview of directly implementable procedures for construction, administration and evaluation of high-quality competence-oriented electronic exams. It covers all the steps necessary for a scientifically proven measurement of competence using written examinations: specification of learning objectives, transfer of learning objectives into competence requirements, item construction, setting of grades and passing thresholds, exam evaluation, identification of problematic items, maintenance of a constant evaluation scale over multiple examinations, and reporting of examination results. Additionally, insights into the computer-based exam application KAT-HS and its configuration is given.
The video workshop was created by Patrick Naumann, Andreas Frey, Christian Spoden and Aron Fink. Registered users of the website https://kat-hs.uni-frankfurt.de can watch the videos free of charge in the download section of that page.
December, 16th, 2019
Christmas get-together 2019
This year, our annual Christmas get-together was spent in the Escape Room in “Cooky's”. Split up into two groups, each group was determined to solve the riddles of the “laboratory”, resulting in a tie. The second portion of the night was spent with Äppler (cider) and regional specialities in the traditional cider restaurant “Fichtekränzi” in Sachsenhausen, Frankfurt.
We spent a great evening together to ring in the festive season. The entire department wishes on this occasion everyone dropping by our page a happy, festive season and a successful new year! (A.S.)
December, 10th, 2019
Workshop on the improvement of university exams at the Goethe University Frankfurt
Our workshop on the improvement of university exams (Introduction to the construction, administration and evaluation of competence-oriented e-exams) took place on 8th of October 2019 at the Interdisciplinary College for University Teaching at the Goethe University Frankfurt. In the workshop, a comprehensive overview of procedures for construction, administration and evaluation of high-quality competence-oriented electronic exams was provided.
The workshop covered all the steps necessary for a scientifically proven measurement of competence using written examinations: specification of learning objectives, transfer of learning objectives into competence requirements, item construction, setting of grades and passing thresholds, exam evaluation, identification of problematic items, maintenance of a constant evaluation scale over multiple examinations, and reporting of examination results. Additionally, insights into the computer-based exam application KAT-HS and its configuration were given. The participants had the opportunity to carry out steps of the KAT-HS concept practically for an exam from their own field of activity.
The workshop was organized by Patrick Naumann and Andreas Frey and was supported by Christian Spoden, Aron Fink and Irina Kuzmina. Registered users of the website https://kat-hs.uni-frankfurt.de/ can download workshop materials such as the presentation slides.
October, 08th, 2019
Presentations at the 14th Conference of the Subdivision “Methods and Evaluation”
On the 18th of September 2019 the symposium “Applications of the Item-Response-Theory (IRT)” took place during the 14th Conference of the Subdivision “Methods and Evaluation” of the German Psychological Society (DGPs) in Kiel, Germany. The symposium gave insight into the applications of IRT in different diagnostic application contexts. Patrick Naumann presented the results of a simulation study entitled “Item Parameter Drift in Small Samples”, which compared the performances of three methods for the detection of item parameter drift using the Continuous Calibration Strategy (CCS). In his talk “Consideration of Item Position Effects in Computerized Adaptive Testing with the Continuous Calibration Strategy” Prof. Dr. Andreas Frey presented comprehensive results on the effectiveness of the CCS considering item position effects. Dr. Christian Spoden gave a talk entitled “Adaptive and Partially Adaptive Measurement of Individual Change”, in which he reported on the use of CAT for measuring change of learning processes. Further information about the FGME can be found here.
September, 18th, 2019
Presentations at the 18th Biennial EARLI Conference
From 12th to 16th August the 18th conference of the European Association for Research on Learning and Instruction (EARLI) took place in Aachen. The conference (“Thinking Tomorrow's Education: Learning from the past, in the present and for the future”) focussed on results and ideas on state-of-the-art research in learning and instruction. Aron Fink held a talk entitled “Testing an explanatory model for the intention to use adaptive and non-adaptive e-exams by the university teaching staff”. This presentation focused on the factors influencing the intention of university lecturers to use (adaptive) e-examinations. Dr. Christoph König “Reducing sample size requirements in large-scale assessments for improving test design flexibility” presented an approach for increasing the flexibility of international large-scale assessments with Fixed Item Parameter Calibration (FIPC). In this approach the parameters of already previously calibrated items (such as the trend items in PISA) are fixed to their known estimates, and only the new item parameters (such as the newly developed items in PISA) are estimated.
August, 16th, 2019
Our work unit at the Annual Meeting of the European Mathematical Psychology Group (EMPG)
The 50th Meeting of the European Mathematical Psychology Group (EMPG) took place in Heidelberg from 5th to 7th August 2019. Dr. Christoph König had a talk entitled “Alternatives to the Inverse Wishart distribution in Bayesian hierarchical IRT models”. Based on results of a simulation study, the standard Inverse Wishart distribution (the most popular prior distribution for covariance matrices in Bayesian hierarchical Item Response Theory models) was compared to the performance of three alternative specifications (scaled IW, hierarchical IW, and a separation strategy based on the Cauchy distribution and the Cholesky factor of the correlation matrix) in terms of parameter recovery (Bias and RMSE) in the hierarchical two-parameter logistic (2PL) IRT model. The results show a superior performance of the alternative specifications.
August, 8th, 2019
Nicola Hericks newest member of the work unit
The work unit Educational Psychology with focus on Counselling, Measurement, and Evaluation welcomes Nicola Hericks as its newest member!
July, 1st, 2019
Presentation on item position effects of the continuous calibration strategy at the IACAT-conference
The conference of the International Association for Computerized Adaptive Testing (IACAT) took place in Minneapolis from 10th to 13th June 2019. Andreas Frey presented results on item position effects in the continuous calibration strategy, which he examined together with Aron Fink and Christian Spoden. The presentation was entitled “Consideration of Item Position Effects in CAT with the Continuous Calibration Strategy”. The continuous calibration strategy (Fink et al., 2018) allows the use of computerized adaptive tests even in small samples. At the IACAT conference, the results of a Monte Carlo simulation study were presented. In this study, the suitability of the continuous calibration strategy was examined in the presence of item position effects.
Further information on the IACAT conference can be found at http://iacat.org/.
June, 14th, 2019
Paper on Equating Setups in the Continuous Itempool Calibration for Computerized Adaptive Testing published in “Frontiers in Psychology”
Sebastian Born, Aron Fink, Christian Spoden and Andreas Frey have published an article on equating methods in the continuous calibration strategy in the journal “Frontiers in Psychology”. They investigated different methods for scale transformation using the continuous calibration strategy (Fink et al., 2018) in computerized adaptive tests regarding the estimation accuracy of the item parameters (mean/mean, mean/sigma, Haebara and Stocking-Lord). The characteristic curve methods, in particular Stocking-Lord, turned out to be suitable methods for scale transformation in the continuous itempool calibration for computerized adaptive testing. The article can be found by clicking on the link in the doi.
Born, S., Fink, A., Spoden, C., & Frey, A. (2019). Evaluating Different Equating Setups in the Continuous Item Pool Calibration for Computerized Adaptive Testing. Frontiers in Psychology, 10, 1277. doi: 10.3389/fpsyg.2019.01277.
June, 6th, 2019
New DFG project
The German Research Foundation (DFG) is funding the project "Online Calibration of Item Response Time Models" with their program “Initiation of International Collaborations”. As part of the project, Prof. Dr. Alison Cheng and Prof. Dr. Cheng Liu from the University of Notre Dame (USA) will be visiting the department Educational Psychology with focus on Counselling, Measurement, and Evaluation in June and July 2019. We are looking forward to our guests!
May 13th, 2019
Patrick Naumann newest member of the work unit
The work unit Educational Psychology with focus on Counselling, Measurement, and Evaluation welcomes Patrick Naumann as its newest member!
April 2nd, 2019
Colloquium "Methods of Empirical Educational Research”
From 10.3.-15.3.2019 the annual colloquium "Methods of Empirical Educational Research" was held in Hirschegg, Austria. During the colloquium, 14 experts discussed contributions on current methodological developments. Our department was represented by contributions from Andreas Frey, Christoph König and Aron Fink. We look back again on a very stimulating time in Kleinwalsertal!
March 26th, 2019
Work unit at the GEBF 2019
The work unit Educational Psychology with its focus on Counselling, Measurement, and Evaluation contributed three talks to this year’s Conference of the Society for Empirical Educational Research (GEBF) in Cologne. Andreas Frey gave a talk about the use of computerized adaptive testing in repeated small sample surveys, Shirin Esmaeili-Bijarsari gave a talk about the acceptance of criterion-referenced computerized adaptive university exams from students, and Aron Fink gave a talk about influencing factors on the intention of lecturers to use (adaptive) computerized university exams.
February 28th, 2019
Paper on the consideration of multiple item context effects in computerized multidimensional tests published
A paper on how to consider multiple item context effects in computerized multidimensional tests was published recently in Frontiers in Psychology. Norman Rose, Gabriel Nagy, Benjamin Nagengast, Andreas Frey and Michael Becker describe a generalized linear mixed model (GLMM) parameterization of the multidimensional Rasch model that allows considering multiple item position effects. They illustrate this approach with data from an item calibration study of a multidimensional computerized test assessing student competencies in mathematics, science, and reading. The paper is available as a download on the homepage of the journal: https://doi.org/10.3389/fpsyg.2019.00248.
February 25th, 2019
Andreas Frey Appointed to the KERMIT-Online Advisory Board
Andreas Frey was appointed to the advisory board of the KERMIT-Online project of the Free and Hanseatic City of Hamburg and the Hanseatic City of Bremen.
February 20th, 2019
Andreas Frey contributed a chapter for the new Handbook of School Education
Andreas Frey, in cooperation with Johannes Hartig, contributed a chapter on competence diagnostics to the new Handbook of School Education, edited by Marius Harring, Carsten Rohlfs and Michaela Gläser-Zikuda. The authors introduce the basic questions of competence diagnostics and illustrate the typical planning and administration of a competence test. The Handbook of School Education is available for purchase at http://www.utb-shop.de/handbuch-schulpadagogik-9518.html.
December 5th, 2018
Paper on a novel response-time based method for modeling omitted responses in computer-based testing published
A paper on the illustration of a novel response-time based method for modeling omitted responses in computer-based testing was published recently in Behaviormetrika. Andreas Frey, Christian Spoden, Frank Goldhammer und S. Franziska C. Wenzel derive the method from the theory of missing data problems of Rubin and colleagues and embed it in an item response theory framework. They illustrate the novel method with responses given by students to items of a computer-based ICT skills test. The paper is available as a download on the homepage of the journal: https://rdcu.be/bczWd.
November 20th, 2018
Second part of the Special Topic „Advances in Educational Measurement” published in Psychological Test and Assessment Modeling
The second part of the Special Topic “Advances in Educational Measurement”, edited by Andreas Frey, Christoph König and Christian Spoden, has been published in Psychological Test and Assessment Modeling. The Special Topic presents a series of studies investigating means and possibilities to make educational measurement, and its statistical modeling approaches, more personalized and flexible. The increasing use of digital devices in educational processes requires educational measurement to be able to adapt accordingly. The Special Topic is available at https://tinyurl.com/y7m77jk9. The first part, published in June, is available at https://tinyurl.com/ychg9p3y.
October 18, 2018
A position paper was presented by Andreas Frey at the 51. DGPs-Congress
At the 51. Congress of the German Psychological Society (DGPs) in Frankfurt am Main, Andreas Frey presented a position paper concerning “University Exams in the 21st Century”.
October 2nd, 2018
Paper on the development of a computerized adaptive test published in the Journal of Computerized Adaptive Testing
A paper on the development of a computerized adaptive test was published recently in the Journal of Computerized Adaptive Testing. Christian Spoden, Andreas Frey, and Raphael Bernhard show that the development of such tests, which is widely regarded as time-consuming and resource-intensive, can be accomplished in a relatively short timeframe. They illustrate this with three tests (measuring student competencies in reading, mathematics, and science), which had to be developed and piloted within an 18-month period, and were used directly afterward in a large nationwide research initiative. The paper is available as a download on the homepage of the journal: http://iacat.org/jcat/index.php/jcat/article/view/70.
October 2nd, 2018
Andreas Frey Appointed Professor
Today, Andreas Frey has been officially appointed professor of the work unit for Educational Psychology with focus on Counselling, Measurement and Evaluation. He will take up his position at the Goethe University Frankfurt on September 1, 2018.