Formative Assessment for Progress Tests of Applied Medical Knowledge
Rationale
The progress test has a high utility for an integrated PBL course (Verhoeven et al., 2002; Verhoeven, 1998; Albano et al.,1996; van der Vleuten et al., 1996). It samples the whole domain of knowledge appropriate to a newly qualified doctor and avoids test directed 'cramming and dumping', it encourages clinical reasoning rather than factual recall, takes a frequent look allowing rapid remediation, and is cost effective. In the early years however students choose the 'don't know' option as they have little knowledge at the assessed level. Test data shows that the number of times a 'don't know' option is chosen declines as the student progresses through the course. One advantage of the progress test is that it focuses the student on applied medical knowledge that is aligned with intended course outcomes and avoids the test driven 'cramming and dumping' of detailed information. The disadvantage however is that in the early years the students have only studied a small number of case units, so a test that samples the whole domain of knowledge appropriate to a newly qualified doctor is too broad and not focused on material the students have studied on the course so far.
The Formative Assessment was therefore set up to provide more information about knowledge acquisition in the early years. We developed a bank of formative items relevant to the first 2 years of the program, classified by case unit (19 over 2 years) and 5 curriculum themes. We were keen to align the formative assessment with the summative assessment in order to stay faithful to the pedagogic rationale and utility of the progress test.
At the PMS students are not allowed to take copies of the Progress Test exam paper out of the examination room. This is because some of the questions are drawn from the Hong Kong IDEAL database which must remain secure. Students receive item specific feedback in the form of the name of the broad topic which the item addressed (e.g. 'diagnosis of chest pain') and verification of whether their response to the item was correct or incorrect.
The key feature of the formative items however, is that the on-line assessment delivery enables the designer to provide response contingent feedback (Manson & Bruning, 2005) which explains the reasoning behind each choice and directs the student to additional learning resources (e.g. texts, workbooks, images, websites) to encourage further self-directed learning.
Feedback is one of the key principles of formative assessment (Manson & Bruning, 2005; Black & Wiliam, 1998; Sadler, 1989; Roos & Hamilton, 2005; Natriello, 1987). Our main aim was to provide the students with more specific feedback than that received after the progress test. The term 'feed-forward' might be more appropriate as it emphasizes that the purpose of the feedback is to improve performance if a similar situation is encountered in the future.
