Busy Chairperson's Guide to Assessment

Written by Dr. Dennis Holt
Vice Provost at Southeast Missouri State University

1. What is a departmental assessment program?
A departmental assessment program evaluates the effectiveness of its undergraduate and graduate programs in terms of measurable student outcomes. The program consists of (a) lists of educational objectives for each of the department’s major programs expressed in terms of student learning outcomes; (b) measures of student achievement for each of the objectives; (c) methods of collecting data; (d) procedures for involving departmental faculty in reviewing and using the results of assessment, including revision of the assessment plan when necessary; and (e) annual collection, analysis, and reporting of the results of assessment.

2. What should be included in the department’s list of educational objectives?
The list of educational objectives for each academic program in the department should include knowledge, skills, and attitudes specific to the major plus, at a minimum, the three core University Studies objectives: locating and gathering information, written and oral expression, and critical thinking and reasoning.

While the list of educational objectives should be comprehensive and detailed, it should still be feasible to assess every stated objective. Formulate each objective so that there is a credible connection between the objective and the method of assessing it.

3. Who should be assessed?
North Central Evaluators and Missouri CBHE staff emphasize that all majors should be assessed--typically as they near completion of their program. Thus, voluntary testing in which only some majors participate is no longer acceptable. Though some departments may wish to assess the learning of non-majors in their service courses and students transferring to other colleges and universities, the focus of assessment is generally on graduates from the department’s major programs.

4. How can the department insure that all students are assessed? What if students don’t want to participate?
The most viable solution is to integrate assessment into the curriculum. For example, a department might design internship evaluations so that they provide useful information about student performance on key objectives, or incorporate senior projects and exit exams into a capstone course, or pre-test students in an introductory course.

Students will engage in assessment activities that are an integral, logical part of their education. In any case, since voluntary participation is highly unlikely to produce satisfactory levels of student involvement, both the benefits of participating and the costs of abstaining need to be made evident to the students in terms that make sense to them.

5. What if an appropriate nationally normed test of achievement in the major is not available?
Departments are not required to use nationally normed tests. In fact, North Central Evaluators discourage the use of nationally normed tests if they do not provide relevant information about student achievement in the major. One advantage of nationally normed tests is that they provide a comparative standard of performance; a disadvantage is that they often do not relate directly to a department’s program objectives.

Popular alternatives to the nationally normed exam are locally developed exams and performance-based assessments (a capstone project or a portfolio, such as the senior project in Geosciences or senior recital in Music). Locally developed exams are scored "objectively." Performance-based assessments typically use a criterion-referenced rating system.

6. Can you be more specific about acceptable and unacceptable measures of student learning?
North Central Evaluators distinguish direct from indirect measures of student learning. Direct measures include the capstone experience, portfolio assessment, standardized tests, certification and licensure exams, locally developed exams, essay exams blind scored by multiple scorers, juried review of student performances and projects, external evaluation of student performance in internships.

Indirect measures include surveys, exit interviews, retention and transfer rates, length of time to degree, SAT and ACT scores, graduation rates, and placement and acceptance data. Grade point averages, grades in the major, faculty/student ratios, curriculum review documents, accreditation reports, demographic data, and other administrative data are not acceptable measures of student outcomes.

7. Is one good measure of student learning enough to satisfy the assessment requirement?
No. Departments are expected to use multiple measures of student learning. For example, a department might employ a capstone project, internship evaluations, writing test scores, ACT scores, California Critical Thinking Test scores, placement and acceptance data, exiting student interviews, and alumni survey data in its assessment program. One measure can be used for several objectives; a capstone project, for example, might be used to measure knowledge in the major, research skills, and communication skills.

Departments are especially encouraged to use several measures for one objective; skill in writing, for example, might be measured by performance on the University’s writing test, performance on the program’s capstone project, and grades of the department’s majors compared with other students in English composition.

8. Whatever happened to value-added?
"Value added" is still with us. NCA Evaluators would be delighted to find that we are demonstrating value-added by testing both entering and exiting students. But, as Lopez writes, "pre-testing is not necessary if one is highly confident that students know little or none of the content they are to master through completing the degree program." Lopez stresses that pre-testing is particularly appropriate for transfer and graduate students, but the ongoing assessment of majors through various levels of development is always impressive.

The national model is Alverno College, where students are assessed continuously throughout their college careers. More fundamental than demonstrating growth, however, is the need to measure student achievement against clearly stated standards. Though it is quite difficult to demonstrate that a department’s programs are the primary contributor to student learning, it is less difficult to show that students are completing the department’s programs having reached an acceptable level of achievement relative to specific educational objectives.

9. Can you be more specific about standards? How do you set up and apply standards in
assessment?
Standards constitute performance goals and should be defined in terms appropriate to the relevant method of measurement. Where comparative data are available, a department might define standards in terms of the percentage of students at or above a particular percentile. An individual department might have good reasons to state that all of its students should score above the 50th (or 65th, 70th, etc.) percentile on a standardized test in the major--provided that this is a meaningful expression of standards.

Departments with licensure exams might want to state that no fewer than 95% of its students will pass the exam on the first attempt. And departments with a criterion-referenced capstone project (or internship evaluations based on specified criteria) might want to state that all students will receive at least a satisfactory score in each criterial area with 30% performing at a level higher than satisfactory.

Performance-based assessments present specific problems. Though standards are usually written into scoring criteria, performance-based assessments have little credibility unless results are analyzed by comparison to performance of students outside the department, by external review, or through conscientious discussion among faculty of the relative strengths and weaknesses of student performance.

The Art Department’s evaluation of senior projects is strengthened by the fact that it employs an external judge, as is the evaluation of a department that identifies areas of weakness indicated by particular measures (for example, relatively weak understanding of the hypothetical method as indicated by performance in the senior portfolio) and proposes actions to strengthen them (for example, holding a department faculty workshop on teaching the hypothetical method). Whatever your approach, remember that statements such as "All graduating students passed the department’s exit exam" are not credible indicators of standards unless supplemented with appropriate analysis, interpretation and follow-up.

10. How can I add assessment to the already busy schedules of my faculty and students?
To the extent that one can incorporate assessment into daily practice, assessment will not appear as an additional burden. We need to find creative ways to incorporate assessment into curriculum and instruction so that it is part of our normal work load. The burden will seem unbearable to a chairperson who tries to pull together disparate elements of an uncoordinated assessment program on the weekend before the Departmental Annual Report is due. For the chairperson who plans ahead and fully involves faculty in the collection, interpretation, and use of assessment data, the burden will be less onerous.

17. Do I have to use the results of assessment for the purpose of improvement?
In any given year, it may not be necessary or appropriate to launch a program improvement initiative based on assessment results. Still, NCA Evaluators have consistently faulted assessment programs on the grounds that the results of assessment are not being used to improve curriculum and instruction.

The UARC hopes to see a significant increase in the number of departments using assessment for improvement, and if there are any efforts to improve departmental programs connected with assessment they should be reported. Reports of efforts to improve programs are telling indicators of a vital, ongoing assessment program. If your assessment program is not giving you useful information for program improvement, then this information (that the information is not useful) should be used to improve your assessment program. By a curious twist of logic, useless information thereby becomes useful.

Written by Dr. Dennis Holt, Vice Provost
Copyright @ Southeast Missouri State University

Copyright © DWU
Website by: DaveV
Last updated: 1/17/07
605-995-2600