News from the Exams Institute, Fall 2011

The Exams Institute always seems to find ways to fill the summer with activity. One thing that often occurs is a large return of data from people who administered ACS Exams at the end of the spring semester, so we will be finishing up several norms before the fall semester begins. Still, there are some new exams that will still be in need of data. If you go to the web site at http://chemexams.chem.iastate.edu/ you can click on the “Exam Statistics” link and see which exams still need data (and how much data is already in hand.) If you have given an ACS Exam, please consider returning data from your student performances for the purposes of norm calculation.

The project to deliver ACS Exams via an on-line format is moving forward. Several additional pilot tests have been held to determine how well the platform works, and the number of possible concerns continues to shrink, even as more students use the system and inevitably invent new ways to make errors that challenge the recovery capability of the platform. We are also going to continue to conduct pilot tests for placement exams, general chemistry and organic chemistry as we make sure we have (a) very robust software system and (b) understand how the electronic delivery of exam items influences how they measure student performance. Thus the exam items for the current set of on-line delivered tests will be from previously released exams, but the trial will focus on the on-line delivery and any changes that may happen in the ways the psychometric characteristics of those items change when they are delivered via computer.

One completely new product that made possible by the on-line delivery system and is reaching a more advanced stage is the Lab Assessment exam for General Chemistry. This exam will have two portions – an on-line system that will include features such as full-motion video to provide information pertaining to students in the laboratory. There will also be a laboratory practical exam for which student performances will be collected for nationally normed scoring. Initial testing of this system was carried out during the spring. While the system generally performed well, we will be taking some time to make adjustments and improvements to the items and how they are asked, so that full scale trial testing can be undertaken in the upcoming fall semester. Because of the nature of the scenarios that drive the assessment, there will likely be more than two trial test versions that contain different laboratory tasks to drive the questions being asked. We will be happy to communicate with interested parties about this level of content if people are interested in serving as trial testers.

Another emerging project from our research and development efforts is the ability to conduct norms with partial-credit scoring for ACS Exams. This possibility is predicated on the fact that the disctractors for the multiple-choice questions from our exams are derived from known mistakes that student make. We are working with instructors to ask, “if a student made the same error on an open response question, would it receive partial credit?” If it does, we can devise a partial credit system using polytomous scoring models. (Usually, we have dichotomous scoring models, 1 = correct, 0 = wrong.) Research done at the Institute has shown how this type of model changes score distributions for student performances on a previously released exam. Current research is focused on building a system to obtain consensus for which wrong answers would receive partial credit. From a research perspective, the instructors who look at the items are raters, and we are currently establishing acceptable methods to gain strong scores for inter-rater reliability. Essentially, we are asking: Do different instructors score the group of items in very similar ways? When we have established this method and the polytomous scoring models that result, the Institute will be able to generate norms for future exams with both the traditional scoring and the partial-credit scores, and instructors will be able to choose which system they prefer to use with their own students.

Finally, there are also a number of new exams being developed at this time, and several will need trial testers this fall. If you are teaching First-term General Chemistry, General-Organic-Biochemistry, Analytical Chemistry, Physical Chemistry, Biochemistry or the Diagnostic of Undergraduate Chemistry Knowledge (DUCK), which is commonly used as an exit exam, and are interested in serving as a trial tester, you can contact me at taholme@iastate.edu.

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer