WORKSHOPS
The AEA-Europe Conference is pleased to offer five highly relevant pre-conference workshops that address some of the most important developments, challenges, and innovations currently shaping educational assessment. Designed for researchers, practitioners, policymakers, and assessment professionals, these interactive sessions provide opportunities to explore emerging ideas, strengthen practical skills, and engage with leading experts in the field.
The 2026 workshop programme includes:
- RWorkshop 1: Identifying and analysing evidence to determine whether tasks elicit the intended constructs: bridging the gap between modern validity theory and innovative validation practic.
- RWorkshop 2: Boost your assessment literacy with your best friend, the psychometrician.
- RWorkshop 3: Building Assessment Literacy That Lasts: From Effective Training Design to System-Level Reform
- RWorkshop 4: Comparing assessment cultures in Europe.
- RWorkshop 5: AI Literacy for assessment practitioners in the age of Large Language Models.
Details
- November 25, 2026
- }Full-Day
- Room: TBA
Workshop #1
Identifying and analysing evidence to determine whether tasks elicit the intended constructs: bridging the gap between modern validity theory and innovative validation practice.
Establishing that assessment tasks elicit performances that reflect the intended constructs is a fundamental component of assessment validation. However, literature on this question is dominated by theoretical perspectives, with little practical guidance for practitioners. In addition, recent developments in the assessment landscape, including the emergence of technological advances (e.g. process data) and novel forms of assessment (including 21st-century skills such as collaboration and self-reflection), mean that established guidance may not adequately address contemporary requirements. Through a combination of presentations, discussion and group activities, this workshop explores the challenges in identifying and analysing validation evidence to determine whether assessment tasks elicit performances that reflect the intended constructs. Participants will explore a range of validation evidence sources, from traditional statistical and qualitative approaches to innovative methods like process data and eye-tracking. Attendees will work through real-world scenarios, critically examining the alignment between task design and cognitive processes, and considering the implications of emerging technologies on validation practices. Participants will leave with a deeper understanding of how to strengthen their validation arguments, navigate trade-offs in assessment design, and leverage both classic and emerging techniques to enhance validity in diverse educational contexts.
This workshop is intended to make the complexities of validation theory and practice less challenging and more readily operational. Accordingly, the workshop is relevant to many key actors in educational assessment.
- Gain a clearer understanding of modern validity theory and its practical application.
- Explore a range of validation evidence sources, including statistical, qualitative, process data, and eye-tracking approaches.
- Examine how assessment tasks can be designed to better reflect intended cognitive processes and learning outcomes.
- Work through real-world validation scenarios and discuss common challenges faced by assessment practitioners.
- Learn how emerging technologies are influencing validation practices and assessment design.
- Strengthen your ability to build convincing validity arguments for diverse educational contexts.
- Consider trade-offs and decision-making strategies when evaluating assessment quality.
- Leave with practical ideas and tools that can be applied immediately to assessment development and validation activities.
![]() |
Stuart ShawStuart Shaw is an educational assessment consultant, researcher, and author, and Honorary Professor at University College London’s Institute of Education. With over 20 years of experience in international awarding organisations, his work has focused on ensuring the validity, reliability, and fairness of educational, psychological, and vocational assessments. He has published extensively in educational assessment and language testing and is Chair of the Board of Trustees of the Chartered Institute of Educational Assessors (CIEA). Stuart is a Fellow and Council member of the Association for Educational Assessment in Europe (AEA-Europe), where he also chairs the Scientific Programme Committee. He additionally serves on the Board of the e-Assessment Association and chairs its Research Awards judging panel.
_ |
![]() |
Ezekiel SweiryEzekiel Sweiry is an Associate Director at Ofqual, a non-governmental department that regulates qualifications, examinations and assessments in England. He has over 25 years’ experience in assessment, with the majority of those years spent in paper-based and digital test development and item validity research. He has worked for the three largest UK awarding bodies, where he has been involved in the development of a range of high-stakes assessments and responsible for the training of examiners. His particular research interests include the factors that affect the difficulty and accessibility of test items, the item and mark scheme features that affect marking reliability, and the comparability of paper-based and digital assessments. |
Details
- November 25, 2026
- }Full-Day
- Room: TBA
Workshop #2
Boost your assessment literacy with your best friend, the psychometrician.
Assessment literacy refers to the competence needed to design, interpret, and use assessments effectively to support learning. It is a multi-faceted and dynamic construct that continuously evolves for individuals, due to new experiences with educational activities and educational measurement.
This workshop will engulf you in new experiences with educational assessment. The interaction with your friendly neighbourhood psychometrician will help you boost your assessment literacy. In this interactive session we discover together how the timely involvement of a psychometrician can refresh your assessment knowledge, assessment skills, and attitudes toward assessment of all stakeholders involved in testing in your particular context.
During the workshop we’ll follow a scheme of activities needed to develop high-quality assessments, for anybody involved in setting up (new) exams, tests, and testing programs. We’ll cover several psychometric topics without formulas and the need to know R. The workshop helps broadening your assessment literacy. We do so by linking psychometrics to the practical decisions and challenges you’ll face, and show how broader assessment literacy helps address and solve these issues. In each block, we’ll give guidelines and illustrate best practices. We’ll invite you to share your experiences with the topics and ask us for advice.
- Develop a deeper understanding of assessment literacy and its role in effective educational assessment.
- Discover how psychometricians can support the development of high-quality exams, tests, and assessment programs.
- Explore key psychometric concepts through practical examples, without formulas or technical programming requirements.
- Learn how to make informed assessment decisions based on evidence and best practices.
- Gain insights into common challenges faced when designing and evaluating assessments.
- Connect assessment theory with real-world testing scenarios and practical solutions.
- Exchange experiences and ideas with fellow participants and assessment professionals.
- Receive practical guidance that can be applied immediately within their own educational or testing context.
|
|
Bas HemkerBas Hemker has been the head of the psychometric department of Cito. He worked as an assistant professor at Vrije Universiteit Brussel. He currently works at Cito, specialized as an educational measurement researcher, working on the quality of school exams and novel approaches to measuring reading ability. He is actively involved in the work of CitoLab International, Cito’s international department. He is also a member of the Dutch Committee on Test Matters (COTAN) and was a member for many years of the Professional Development Committee of AEA Europe. _ |
|
|
Marieke VanonnaMarieke Vanonna worked for a few years as an assistant professor for statistics and methodology in psychology before joining Cito. At Cito she coordinates all psychometric analyses that are necessary for the national exams in Dutch secondary education. She has ample experience with standard settings, in national and international contexts? She also works on a variety of smaller assessment and research programs. _ |
|
|
Cor SluijterCor Sluijter is a former head of the psychometric department of Cito and a former head of the department for middle and higher vocational education. He currently works as an independent consultant on test quality and test use and is a lecturer in educational measurement at Fontys University of Applied Sciences. He is an external member of several Exam Boards of various universities in the Netherlands. He is also the Vice-President of AEA-Europe since November 2024. _ |
Details
- November 25, 2026
- }Full-Day
- Room: TBA
Workshop #3
Building Assessment Literacy That Lasts: From Effective Training Design to System-Level Reform.
Assessment literacy becomes valuable when it is embedded it into institutions and assessment systems. Well-designed training programmes can often fail due to systemic pressures, changes of leadership or due to accountability misalignment. This workshop addresses these issues directly, by exploring the factors that underpin success in educational assessment related training, and how they are realised utilising a series of case studies. Delegates will leave with a practical framework for designing assessment training that is genuinely differentiated to stakeholder need, a focused analytical tool for identifying the conditions that will determine whether their own reform takes root, and a structured approach to evaluation that goes beyond measuring knowledge gain to asking whether assessment practice has actually changed.
The first section will focus on understanding the stakeholders involved in developing assessment literacy and how this understanding transfers into the design of the professional development intervention. The second section will focus on implementation strategies including the optimal approaches for evaluation and impact analysis.
This workshop is designed for practitioners and researchers who are actively engaged in assessment literacy development at scale including assessment professionals within national and regional assessment bodies, policymakers and teacher educators
- Learn how to design assessment literacy initiatives that address the needs of different stakeholder groups.
- Explore the institutional and system-level factors that influence the success of assessment reform efforts.
- Examine real-world case studies highlighting effective approaches to assessment literacy development.
- Gain practical tools for identifying barriers and enablers within your own organizational context.
- Develop strategies for implementing assessment training that leads to sustainable change.
- Discover methods for evaluating impact beyond knowledge acquisition and measuring changes in assessment practice.
- Understand how leadership, accountability structures, and organizational culture affect reform outcomes.
- Leave with a practical framework for planning, implementing, and evaluating assessment literacy initiatives at scale.
|
|
Simon ChildSimon Child is Head of Assessment Training at The Assessment Network at Cambridge, part of Cambridge University Press & Assessment. He is also the co-course director for the Postgraduate Advanced Certificate in Educational Studies: Educational Assessment at the University of Cambridge. He works with schools, universities, ministries and professional bodies on developing assessment practices for the benefit of learners. In 2021, he co-authored the What, why and how of assessment: A guide for teachers and school leaders with Paul Ellis. Previously, Simon was Senior Research Officer in the Assessment Research and Development Division of Cambridge University Press & Assessment. His research interests include assessment ‘literacy’, formative assessment practices and professional development methodologies. His background is in developmental psychology. In 2011, he received his Ph.D. from the University of Manchester, which focused on the development of symbolic cognition in pre-school children.
_ |
|
|
Brooke WyattBrooke Wyatt is Head of Assessment Reform at Cambridge University Press & Assessment, bringing over a decade of specialist experience in assessment reform to her work. With a foundation built on a distinguished career in teaching, Brooke transitioned into assessment reform ten years ago, developing expertise in assessment training and large-scale reform projects. In her current role, Brooke leads the Assessment Reform Group, an experienced team responsible for the end-to-end delivery of complex, multi-faceted assessment projects. The group's portfolio spans the design and development of assessment materials through to capacity building initiatives to a range of stakeholders, supporting organisations and systems to strengthen their assessment practices at scale. Brooke brings both strategic leadership and hands-on expertise to her work in this field. |
Details
- November 25, 2026
- }Full-Day
- Room: TBA
Workshop #4
Comparing assessment cultures in Europe.
This workshop will raise questions about ways of carrying out education assessment in different countries. The concept of ‘Assessment Cultures’ opens the door to such questioning and to celebration of differences. The thinking for the workshop is underpinned by ideas from Cristina Alarcon Lopez, the first person to present a webinar for our SIG. She is working with the team that is planning and will present this workshop.
In her webinar Cristina counselled against neglecting ‘the cultural and historical conditions of assessment’. But how can we develop the language to describe what it is that we are seeking to nurture, and to avoid?
Success for the workshop will be seen if participants achieve a better understanding of how their culture differs from others, and why. They will take away a draft of elements of a paper which they might further develop with their colleagues.
- Explore the concept of assessment cultures and how they influence educational assessment practices across Europe.
- Gain a deeper understanding of the cultural and historical factors that shape assessment policies and approaches.
- Compare your own assessment context with those of other countries and identify key similarities and differences.
- Develop a richer language for discussing assessment values, traditions, and reform priorities.
- Engage in meaningful discussions with colleagues from diverse educational and professional backgrounds.
- Reflect critically on practices that should be preserved, adapted, or avoided within different assessment systems.
- Contribute to collaborative activities aimed at analysing and interpreting assessment cultures.
- Leave with a draft framework or paper outline that can be further developed with colleagues after the workshop.
![]() |
Andrew WattsAndrew Watts was one of the initiators of the Assessment Cultures Special Interest Group within the Association and continues to serve on the Steering Group for that SIG. He has taken a leading part in a number of pre-conference workshops.
_ |
|
|
Cristina Alarcón LópezCristina Alarcón López is a postdoctoral researcher and lecturer at the Institute of Education at the University of Vienna. Her publications include "Assessment Cultures" (Peter Lang), which she co-edited with Martin Lawn in 2018.
_ |
|
|
Isabel NisbetIsabel Nisbet was the founding CEO of Ofqual, the regulator of examinations and qualifications in England. She brings her academic background in philosophy to her current work on educational assessment, and has co-authored (with Stuart Shaw) two books on assessment.
_ |
|
|
Dr. Christoph SchneiderDr. Christoph Schneider is a full professor of educational sciences at the University of Trier, Germany. Within teacher education programmes, he is responsible for a study module on educational assessment. He has presented at several AEA-Europe conferences.
_ |
Details
- November 25, 2026
- }Full-Day
- Room: TBA
Workshop #5
AI Literacy for assessment practitioners in the age of Large Language Models.
AI literacy is becoming a necessity for navigating many aspects of the modern world. Nowhere is this more true than in assessment, where it is vital that human oversight and ethical and responsible use is front and centre.
This workshop will equip attendees to judge when and how it is (and is not) appropriate to use AI in their work and daily lives. By building a practical understanding of how these systems work, participants will make those decisions with greater confidence. Aimed at a non-technical audience, we will explain and demystify the underlying technology, what drives error and inconsistency, and how those behaviours follow from the way these systems operate.
We will also facilitate discussions on the ethics, advantages and pitfalls of their use, with the aim of helping participants look past the hype and recognise when to trust, when to double check, and when to walk away.
- Develop a practical understanding of how AI and Large Language Models operate.
- Learn when AI can be a useful tool and when human judgment remains essential.
- Explore the strengths, limitations, and common sources of error in AI-generated outputs.
- Gain confidence in evaluating the reliability and appropriateness of AI-assisted work.
- Discuss ethical considerations, responsible use, and the importance of human oversight in assessment.
- Examine real-world examples of AI applications within educational and assessment contexts.
- Learn strategies for identifying risks, biases, and potential misuse of AI technologies.
- Build the skills needed to critically assess AI outputs rather than simply accept them at face value.
![]() |
Alex DunhillAlex Dunhill is a data scientist with an interest in applying machine learning and AI techniques to assessment data and processes in novel ways. Having joined AQA in 2019, his work focuses on how we can use these approaches to improve assessment quality, including research into using AI to support marking quality and item writing. With a background in computational astrophysical simulations, he has experience communicating complex concepts and results through outreach talks.
_ |
|
|
Daryl StevensDaryl Stevens leads the Data Science & Innovation team. He has a background in computer science, machine learning and software engineering, with 12 years’ experience across assessment research, standards setting and technical analysis. He works with his team to apply data science and machine learning to deepen AQA’s understanding of assessment. Alongside this, Daryl is engaged in the technical, societal and ethical impacts of modern AI, and aims to support careful, evidence-led and positive use of AI in assessment.
_ |
|
|
Victoria TassieVictoria Tassie is a data scientist at AQA, having joined the awarding body in 2019. She is particularly interested in natural language processing (NLP) and in the ethical implications of artificial intelligence and machine learning models. Her recent work has primarily focused on handwriting-related NLP, including developing and comparing approaches for machine transcription of handwritten text, alongside methods for post-hoc, context-driven correction of those transcriptions and clustering methods for student responses.
_ |
|
|
Rob CarruthersRob Carruthers previously taught Music and Mathematics in UK secondary schools for 10 years, across the independent and state education sectors, and led on data in several roles. He became a full-time data scientist at AQA in 2024 and, in this role, he has led work on large-scale document classification and analysis using machine-learning methods. His current work centres around ethical human-centric AI to assist with assessment delivery, as well as research into synthetic response generation. |
SPACES ARE LIMITED!
Workshops are expected to fill up quickly. Secure your spot today.
TBA
TBA
TBA
TBA
Establishing valid qualification equivalency with qualitative judgement.
Stuart Gallagher and Georgie Billings
Where statistical equating methods are not available, through lack of common items or common candidates, but equivalency between two qualifications is required, it can be difficult to provide robust evidence.
Session 1 of the workshop focuses on a novel standard setting methodology that allowed for IGCSE scores to be translated into Mississippi end-of-course performance levels and integrated into the state accountability system. The method draws on aspects of both Body of Work (BoW) and Bookmarking methods to create an operationally feasible process.
Section 2 explores why there is a need for qualification equivalency, how a qualification can be broken down into content, demand and awarding standards, and some possible methodologies for establishing standards equivalency when the data for psychometric equating is not available.
The practical activity will involve delegates becoming comfortable with using the CRAS framework to evaluate the demand of questions, and how to set up, run and evaluate a comparative study using the No More Marking platform. We will discuss the benefits and limitations of this methodology for establishing demand equivalency, and what follow up work can usefully done with the results.
APPENDIX A: Template pre-conference workshop
WORKSHOP TITLE:
Establishing valid qualification equivalency with qualitative judgement
Presenters:
Stuart Gallagher and Georgie Billings
Presenters’ Bios (500 words max per presenter):
Stuart Gallagher:
Stuart Gallagher is a Principal Assessment Advisor in the International Education group at Cambridge University Press & Assessment. His responsibilities include advising on assessment issues for the US market, evidencing and developing the technical quality of large-scale, high-stakes international assessments, and innovating practice around reporting, analysing and articulating candidate performance and awarding standards.
He has wide experience of UK assessment, having worked in Research and Technical Standards and as a Chair of Examiners for OCR. This includes assessment design and development, general qualification reform, awarding standards in vocational and technical qualifications, and major process change for key operational activities.
Before moving into educational assessment, Stuart taught French and Spanish, having gained a Postgraduate Certificate in Education and Qualified Teacher Status at the Faculty of Education in Cambridge. As a secondary school MFL practitioner and head of department, his key interests were innovating and improving curriculum design and delivery, developing use of data for reporting and analysis, and introducing immersion-style teaching of modern languages.
Stuart holds a BA in French and Linguistics from the University of York and an MEd in Researching Practice from the University of Cambridge.
Georgie Billings
Georgie Billings is Head of Assessment Quality within the Assessment Reform Group at Cambridge University Press & Assessment. The team enacts assessment reform projects for ministries, school groups and NGOs globally. Her responsibilities include heading up the quality assurance and research programme for accredited qualifications, advising on standard setting across a vast range of tests, including Africa, Asia and the Middle East, and heading up reform projects in contexts such as Cox’s Bazar refugee camps.
Georgie has a wealth of prior experience, including teaching, working for the educational UK charity The Prince’s Trust and working for the Department for Education.
Georgie holds a BAHons in English, PGCSE (Cantab) in Secondary English, MEd(Cantab) in Researching Practice, and holds RITTech accreditation as a data analyst. She has completed post-graduate studies in Creative Writing and Forensic Chemistry, and is currently finishing her MSc in International Development at University of Edinburgh.
Why AEA-E members / conference delegates should attend this workshop:
Where statistical equating methods are not available, through lack of common items or common candidates, but equivalency between two qualifications is required, it can be difficult to provide robust evidence. This workshop explores some methodologies for doing this, based on real experiences in the field, and will provide practical tools that can be applied to other qualifications.
Who this Workshop is for:
Anyone interested in establishing comparability or equivalency in the demand or awarding standard of qualifications through non-psychometric methodologies, particularly across different international contexts.
Overview of workshop (500-600 words):
Part 1: In 2022, the US Department of Education approved Cambridge IGCSE assessments in English Language, Mathematics and Biology to be administered in place of the corresponding high school end-of-course tests within the Mississippi state testing program, known as the MAAP. Cambridge IGCSE and the MAAP represent the different educational testing traditions in the UK and the US: Cambridge IGCSE assessments include a significant number of constructed response and multi-part items and have a strong focus on evaluating candidate-generated evidence, while MAAP end-of-course tests are grounded in psychometric tradition and make considerable use of field-tested banks of selected-response items.
This section of the workshop focuses on a novel standard setting methodology that allows for IGCSE scores in these subjects to be translated into Mississippi end-of-course performance levels and integrated into the state accountability system. The method draws on aspects of both Body of Work (BoW) – a holistic standard-setting method often used when an assessment contains many open-ended or constructed response items[1] – and Bookmark methods to create an operationally feasible process.
Delegates will learn about how this blended approach was designed and operationalised, including the statistical methodology used to develop ordered item booklets for constructed response items, and consider the challenges at each stage. They will have the opportunity to compare and contrast the BoW and Bookmark aspects of the process side-by-side, and explore the contributions made by these activities to the overall validity of the outcome.
Part 2: Cambridge International accredits a variety of qualifications globally which are owned and administered by other awarding organisations, resulting in co-certification and a statement of equivalency with a Cambridge qualification.
The session will explore why there is a need for qualification equivalency, how a qualification can be broken down into content, demand and awarding standards, and some possible methodologies for establishing standards equivalency when the data for psychometric equating is not available.
The practical activity will involve delegates becoming comfortable with using the CRAS framework[2] to evaluate the demand of questions, and how to set up, run and evaluate a comparative study using the No More Marking platform. We will discuss the benefits and limitations of this methodology for establishing demand equivalency, and what follow up work can usefully be done with the results.
Preparation for the workshop:
While no explicit knowledge or prior reading is required, delegates will need to have a laptop with the ability to connect to WiFi at the venue. They may also wish to bring two question papers which they are interested in comparing the demand of.
Tentative Schedule
|
Time |
Session |
Presenter |
|
9.30- 12.00 (Inc. a 15 min break 10:30-11:00) |
Block I Blending Bookmark and Body of Work standard setting methods: a practical solution to linking grading scales across testing traditions. This will include discussion of the problem statement and the development of the methodology. Delegates are then invited to undertake a critique of the method, exploring the impact with real data. |
Stuart Gallagher |
|
12.00- 13.00 |
Lunch |
|
|
13.00- 14.30 |
Block II Discussion of the need for comparability Exploring the concepts of content, demand and awarding standards Development of qualitative methods to establish equivalency |
Georgie Billings |
|
14.30- 14.45 |
Tea/coffee break |
|
|
14.45- 16.30 |
Block III Introducing the CRAS framework Practical task setting up a demand standard comparative task in No More Marking and analysing the results. |
Georgie Billings |
[1] Wyse, A. E., Bunch, M.B., Deville, C., and Viger, S. G. (2014). A Body of Work Standard-Setting Method With Construct Maps. Educational and Psychological Measurement, 74(2) pp.236–262
[2] Johnson, M. and Mehta,S. (2011). Evaluating the CRAS Framework: Development and recommendations. Research Matters(12) pp. 27-32
Network Analysis for the investigation of Rater Effects (using R).
Iasonas Lamprianou.
This workshop introduces the application of Network Analysis (NA) to rater-mediated assessments. NA analyzes rating datasets by considering pairwise comparisons between raters.
Participants will learn how to detect and interpret key rater behaviors, including severity/leniency, inconsistency (misfit), halo effects, bias, drift (changes over time), and the formation of rater sub-communities. A key feature of the workshop is the comparison of NA results with those from traditional approaches such as the Rasch model.
NA is a flexible method that can handle nominal, dichotomous, ordinal, and numeric data. Unlike traditional models that rely on strong assumptions (e.g., local independence or unidimensionality), NA operates with minimal requirements, making it especially suitable for complex or non-standard rating contexts. Visualizations further enhance interpretability.
The workshop emphasizes hands-on experience using open-source R code and real datasets from published studies. A brief theoretical overview will also be provided. Participants are encouraged to bring their laptops and follow along.
Reading list:
Lamprianou (2018) in Educational and Psychological Measurement, 78(3), 430–459.
Lamprianou (2023) in Sociological Methods and Research, 55(1), 525–553.
Lamprianou (2025, in press) in Research Methods in Applied Linguistics.
Lamprianou et al. (2023) in Assessing Writing, 56, 100713.
APPENDIX A: Template pre-conference workshop
WORKSHOP TITLE:
Network Analysis for the investigation of rater effects
Presenters:
Iasonas Lamprianou, University of Cyprus
Presenters’ Bios (500 words max per presenter):
Iasonas Lamprianou is an Associate Professor of Quantitative Methods at the Department of Social and Political Sciences, University of Cyprus. His methodological interests include rater and coding effects, Rasch models, and appropriateness measurement (person-fit).
His recent books include ‘A Step-by-Step Guide to Applying the Rasch Model Using R: A Manual for the Social Sciences’ (2024, 2nd Edition) and ‘Network Analysis for Rating Datasets in R: A Multi-Disciplinary Perspective’ (2025, in press), both by Routledge.
Iasonas has published in journals such as Educational and Psychological Measurement, Sociological Methods and Research, International Journal of Social Research Methodology, Journal of Educational Measurement, Research Methods in Applied Linguistics, Language Testing, Assessing Writing, Assessment in Education, International Journal of Testing, and others.
Why AEA-E members / conference delegates should attend this workshop:
Participants will develop data analysis skills using open-source, cutting-edge tools that can be used either as stand-alone approaches or in complement to existing methods (e.g., Rasch models).
Who this Workshop is for:
Researchers, academics, students, and practitioners who want to enhance their empirical analysis toolbox, particularly in rater designs and rater effects within the broader field of social sciences.
Overview of workshop (500-600 words):
This workshop introduces the application of Network Analysis (NA) to rater-mediated assessments. NA analyzes rating datasets by considering pairwise comparisons between raters.
Participants will learn how to detect and interpret key rater behaviors, including severity/leniency, inconsistency (misfit), halo effects, bias, drift (changes over time), and the formation of rater sub-communities. A key feature of the workshop is the comparison of NA results with those from traditional approaches such as Rasch measurement.
NA is a flexible method that can handle nominal, dichotomous, ordinal, and numeric data. Unlike traditional models that rely on strong assumptions (e.g., local independence or unidimensionality), NA operates with minimal requirements, making it especially suitable for complex or non-standard rating contexts. Visualizations further enhance interpretability.
The workshop emphasizes hands-on experience using open-source R code and real datasets from published studies. A brief theoretical overview will also be provided. Participants are encouraged to bring their laptops and follow along.
Reading List:
Lamprianou, I. (2018). Investigation of rater effects using Social Network Analysis and Exponential Random Graph Models. Educational and Psychological Measurement, 78(3), 430-459.
Lamprianou, I. (2023). Measuring and visualizing coders’ reliability: New approaches and guidelines from experimental data. Sociological Methods and Research, 55 (1), 525-553.
Lamprianou (2025, in press). Network Analysis for the investigation of rater effects: a comparison of ChatGPT vs human raters. Research Methods in Applied Linguistics.
Lamprianou, I., Tsagari, D., & Kyriakou, N. (2023). Experienced but detached from reality: Theorizing and operationalizing the relationship between experience and rater effects. Assessing Writing, 56, 100713.
Preparation for the workshop:
Laptop with R, RStudio. More information will be provided.
Tentative Schedule
|
Time |
Session |
Presenter |
|
9.30- 12.00 (Inc. a 15 min break 10:30-11:00) |
Block I: · Theory of Rater Effects · Introduction to Network Analysis · Severity/Leniency |
Iasonas Lamprianou |
|
12.00- 13.00 |
Lunch |
|
|
13.00- 14.30 |
Block II · Consistency / Inconsistency · Halo Effects |
Iasonas Lamprianou |
|
14.30- 14.45 |
Tea/coffee break |
|
|
14.45- 16.30 |
Block III · Differential Functioning (drift) · Detection of sub-communities |
Iasonas Lamprianou |














