The role of the student in formative assessment
Methods
Three consecutive groups of six students were employed to write items with response contingent feedback. The item writing environment (organizational procedures, quality control procedures, software systems) was redesigned after each iteration.
The test was delivered online with Question Mark Perception software. The routinely collected data from the test was exported into Excel and SPSS and analyzed for the number of students that logged on to take the test and the score on the first attempt.
This project demonstrates the 5 characteristics that are exhibited by a design-based research method (Barab & Squire, 2004; Design-Based Research Collective, 2003). Firstly there is the dual purpose of designing and evaluating the learning environment and of developing the theory of learning and assessment. On the one hand, we will explore the local impact of both the item writing environment and of the test on learning. On the other hand we will develop our understanding of student construction of knowledge, of the relationship between assessment and learning and of the role of the student in the assessment cycle. This project will not only explore the impact of the formative assessment on learning, but also develop and locate the theoretical perspective of Learning Oriented Assessment firmly in the theoretical tradition of formative assessment (Black & Wiliam, 1998; Sadler, 1989; Roos & Hamilton, 2005; Smith & Gorard, 2005; Taras, 2002).
Second, there is an iterative process of design, enactment, analysis and re-design in a naturalistic setting. Initial design efforts focused on developing software, item writing training and quality control procedures. This process improved the quality of the items created and refined the process of creating the items and delivering the test. It also resulted in an awareness of the quality of the item writing environment for the development of both basic knowledge and clinical reasoning and also higher cognitive functions such as learning skills, communication skills, and cognitive strategies of how to write clear feedback to teach effectively etc. Now that the groundwork has been laid we are in a position to systematically adjust aspects of the designed context and explore the impact of the adjustment.
Third, the theoretical developments arising from the Learning Oriented Assessment will inform other practitioners and educational designers. It is hoped that other institutions stimulated by the local impact of this work and inspired by the theoretical developments it embodies, will design an intervention within their own specific context. It is hoped that this process will produce 'lethal mutations' and 'productive adaptations' and so uncover some of the variables that are in play. Thus this design-based research project aims to explore the hypothesis that students benefit from the process of item writing, articulate the mechanisms at work and demonstrate an impact on learning. The model will be available for adapting to alternative environments and for further developments of theory and practice. The validity of the theory therefore is demonstrated by its ability to do work in the world (Barab & Squire, 2004).
Fourth, the research will account for how the design functions in an authentic setting. It will not only document the success or failure of the item writing process, of the tests and of the student performance on the test, it will also refine our understanding of Learning Oriented Assessment as a learning environment and lead to revisions of practice on the ground.
Fifth, it will do this through methods that 'document and connect processes of enactment to outcomes of interest' (Design-Based Research Collective, 2003). We aim to triangulate multiple kinds and sources of data from the item writing environment and from the tests, to enquire into both knowledge acquisition and the theoretical nature of learning. Qualitative and quantitative data sets from item writers (about what were the strengths and weaknesses of the item writing environment), and from the test (student, item, and test performance) can be combined to inform both the local project and the wider debate. Reliability will be enhanced through triangulation of different data sources, repetition of analysis across cycles of enactment, and the standardization of measures. Validity will be enhanced through iterations of the Learning Oriented Assessment project to increase alignment of theory, design, practice and measurement over time.
