Delphi method
Stage 2 Assessment rubric
Development of the rubric
Educational research using Delphi as the research method usually stops at the point of reaching the consensus. Two examples are Nolan's 'A Delphi study of the future of education' (1994) and Jacobs' 'Essential assessment criteria for physical education teacher education programs: A Delphi study' (1996). Yet, some may argue that consensus arrived through Delphi may sustain only on paper but not necessary work when put into practice. To investigate if the consensus can be put into practice, stage 2 and 3 of this study went further and actually applied the results in practice. The 11 assessment criteria agreed upon in the Delphi were used to develop a rubric which was then used to grade 23 students' film projects for creativity.
A usual grading scale from "A" to "F" was adopted when designing the performance grid, in which the level "A" indicates the best result and the level "F" indicates failure in performance. To simplify the process, only levels "A", "C" and "F" were described in detail. Levels "B" and "D" were left blank. However, the Delphi expert agreed that assessors should be able to determine the different quality of each level from the adjacent descriptions.
Film work is an art form that combines many different elements. It can be a work using only visuals to tell a story, such as in the silent cinema of the past, or it can be a composite of sound, music, visuals, drama, animation, etc., such as the recent Hollywood blockbuster "Lord of the Rings". Each individual work can be a combination of different elements. Therefore, a separate column "Not significant to the project" was included in the performance grid in case any stated criterion was not applicable in assessing a particular piece of work. For a similar purpose, space for further comment was also included in the rubric.
The descriptions of different performance levels were selected from responses and phases used by panelists when they discussed criteria assessing creativity as well as terms commonly used within the context of the Film/ TV School.
The assessment grid together with the statistical results from the Delphi survey, were sent to all panelists for comments and recommendations.
The final version of the rubric contains 3 categories: innovation of content, innovation of form and craftsmanship. Each category has 4 to 5 items which adds up to 13 items. Space is left for additional comments under each category. Please contact the first author for the full version of the assessment rubric if interested.
