Innovative Item Development at ATP

Just lefta couple of great sessions at ATP 2010 this morning.  I was able to attend the Innovative Item Type Development panel with some looks as to what some cutting-edge performance-based programs are doing (AICPA, ARDMS, and the NBME) as well as some lessons from practioners on a process for correctly executing an innovative item type program.

Met lots of folks who were very interested in these topics.  I had a few thoughts along the way...

1.  The state of the art is pretty dated.  I have worked firsthand with some of Craig Mills' great people at the AICPA.  The items they showed from their current testing program delivered under Prometric have a number of usability issues that this team had bumped into and resolved years ago.  It is obvious that they are struggling under the limitations of their current exam delivery platform and it is certainly a governor on their ability to deliver innovative content to their community.

2.  The NBME showed where the trend is headed - away from recall questions and towards items that can assess dimensions of ability that are not easily factored into MCQs.  These dimensions include judgement, research, synthesis, analysis and a dozen others.  Utilizing cinical simulators is certainly a trend we'll be seeing more of in the future.

3.  Innovative item development is not just presenting and scoring an item.  It was obvious from a mature innovative program like the AICPA that they have had to turn their attentions and pocketbooks towards solving secondary innovatiion problems.  They have developed and maintain a proprietary CMS called CCMS - Complex Content Management System.  They use this as an environment for managing the innovative item types they support, creating and publishing new items and sample items, and probably accessing the rather customized packets of result data that their testing vendors provide.  We at Breakthrough believe that we will all see real innovation in the extension of these tools and proceses as they are stretched to accommodate innovative item types.

4.  Also the consulting duo of Cynthia Parshall and Lynn Webb presented a six-step framework that is a must-read for anyone considering venturing off down the non-MCQ path.

Anyway, some great sessions today, looking forward to more tonight and tomorrow.