Equipped for Living, Equipped for Learning:
Assessment,
Initiative, and Student Learners at Union College
Union College
Quality Enhancement Plan
August 31, 2005
Table of Contents
Foreword.......................................................................................................................... 1
Chapter 1: The Problem................................................................................................ 4
Figure 1.1 Praxis I Results....................................................................................... 11
Table 1.1 CAAP 2005 Results............................................................................... 14
Figure 1.2 CSI Summary Observations.................................................................... 16
Chapter 2: Review of Best Practices...................................................................... 18
Remedial Education................................................................................................. 18
Best Practices for Achieving Faculty Support........................................................... 24
Best Practices in Assessment................................................................................... 27
Summary................................................................................................................. 32
Chapter 3: ARIES Plan for Quality Enhancement at Union College............... 34
Assessment Program
Roundtable Review
Initiatives
o Initiatives I: What already works
o Initiatives II: Developing Coursework
· Example: Transitional Writing
· Example: Strategies for Success course
Education of Faculty and Staff
Strategies for Success
Chapter 4: Resource Allocation and Governance............................................... 50
Resource Allocation................................................................................................. 50
Table 4.1 Assessment Budget Summary................................................................... 50
Governance and Administrative Infrastructure........................................................... 51
Chapter 5: Program Evaluation................................................................................ 55
Goals for Basic Skills Areas..................................................................................... 55
Goals for Union College Assessment Program ......................................................... 56
References..................................................................................................................... 60
Appendices.................................................................................................................... 63
Appendix A: Working Model for QEP: “ARIES”..................................................... 63
Appendix B: Subcommittee Completion Timeline...................................................... 64
Appendix C: ARIES Overview and Timeline............................................................ 65
Appendix D: Assessment Measures......................................................................... 66
Appendix E: Flow Charts for Mathematics, Reading, Writing.................................... 69
Appendix F: Organizational Charts........................................................................... 72
Appendix G: Union College Learning Outcomes ...................................................... 75
Appendix H: Program Assessment Plan Elements..................................................... 76
Appendix I: Sample Program Assessment Report .................................................... 77
Appendix J: Comments from Assessment Workshop................................................ 79
Equipped for
Living, Equipped for Learning:
Assessment,
Initiative, and Student Learners at Union College
Foreword
The goal of
the Quality Enhancement Plan (QEP) of
Union College is to enhance student learning by providing every undergraduate
student the opportunity to strengthen his or her core competencies in reading,
writing, and mathematics as outlined in the Goals for General Education stated
in the 2005-06 Union College Undergraduate Academic Catalog (pp. 6-7).
The strengthening of these core competencies will be accomplished through an
improved system of assessment and initiatives designed to address student
deficiencies in these areas.
Initial discussions regarding the selection of QEP
topics in 2003 indicated that a sizeable number of Union College students
showed deficiencies in the writing, reading, and mathematics sections of the
Praxis I examination. With these data as points of departure, this QEP develops
a set of methods for assessment of core competency levels and for strengthening
students’ core competencies across the curriculum.
This set of methods, the ARIES plan, proposes a
structured process of reviewing, recommending, and implementing assessment
measures and initiatives based on prevailing best practices. Assessment
measures include both nationally recognized and normed instruments and locally
developed instruments; initiatives include curricular changes and faculty
development activities—for example, activities which might increase awareness
of the role advising plays in enhancing student learning. As the working model
(Appendix A) shows, nationally normed academic assessments such as COMPASS and
the Collegiate Assessment of Academic Proficiency (CAAP) will be complemented
by non-academic assessments such as the College Student Inventory (CSI) and
National Survey of Student Engagement (NSSE). Review of these results should
enhance understanding of the differences between students who are academically
successful and those who are not and lead to development of initiatives
designed to address the differences and enhance student learning at Union
College. The effectiveness of the ARIES plan will be evaluated through a
systematic process of review involving both scores on nationally-normed
instruments and internal measures such as course grades, writing samples, and
other measures identified in each academic program’s assessment plan.
People representing all constituencies of the
college community have been involved in the planning process that led to this
QEP. Membership of the planning subcommittee, which met several times in May,
June, and July 2005, included student, staff, administrative, and faculty
representatives. Versions of the proposal have been sent to the full faculty,
staff, students, and trustees for comment on two occasions. Formal approval of
the plan was obtained from the Administrative Core on June 23, from the Faculty
Summer Senate on July 7, and from the full faculty on August 18.
Representatives from the Student Government met on July 15 and offered valuable
feedback that is included in this document. Members of the Board of Trustees
have also had an opportunity to review the document and comment. An immediate
result of the planning discussions has been the administrative appointment of
qualified individuals representing campus constituencies to an Assessment
Council, which will oversee immediate and future implementation of the QEP (see
Appendix B).
The work of the QEP subcommittee benefited greatly
from the consultancy of Dr. Sandra Bowles, who chairs the University of Charleston
(WV) Assessment Committee, and from attendance of three Union College faculty
and staff at the University of Charleston Assessment Conference at Charleston,
West Virginia on June 6-7, 2005, where the keynote speaker was Dr. Peggy Maki.
Insights gained from these sources and affirmed by the QEP committee’s review
of best practices in assessment, academic program evaluation, and remediation
have been central to the development of Union College’s ARIES plan.
The first components of the QEP have been
implemented for the fall semester of the 2005-2006 school year, the Baseline
Year. Initiatives related to mathematics and writing are being implemented in
the fall of 2005; the reading component will be implemented in the fall of
2006. By the academic year 2009-2010, the ARIES assessment-initiative mechanism
will be fully operative and functioning smoothly for reviewing assessment
results in the core competencies and implementing appropriate ongoing
initiatives (see Appendix C).
Chapter 1: The Problem
Where We Are, Where We’re Going, and How
ARIES Will Get Us There
The Union College Quality Enhancement Plan will improve student learning in three core competencies—reading, writing, and mathematics—by developing and implementing a cycle of Assessment, Review, Initiatives, Education, and Strategies. The plan is designed to renew itself: initial assessments will lead to academic initiatives which will, in turn, lead to further assessments, and so on. This is not to say that Union College will engage in assessment for its own sake, but rather that it will ensure that self-examination serve the college’s practical concerns. Assessment is one of many important steps in an ongoing process of improvements that will benefit all student learners.
Throughout their time at Union College, students should own those key skills that will help them succeed in the classroom, in the careers they are preparing for, and in the lives they lead and will lead. If students do not already have these skills, Union must offer opportunities to develop them. If, like most students, they do own key skills, but can benefit from fine-tuning them, Union must help students identify and develop these essential abilities. And Union College must be prepared to recognize, inspire, and enable its many talented students as well. The rhetorician Kenneth Burke spoke of “equipment for living” (p. 253), of owning “strategies for dealing with situations” (p. 254). This QEP statement borrows his deceptively simple language and extends his vision. Through a Quality Enhancement Plan that emphasizes the symbiosis of assessment and initiative, Union College will make certain that its students have strategies for dealing with situations. Union will equip its students for living and learning.
In its Handbook for the Reaffirmation of Accreditation, the Southern Association of Colleges and Schools (SACS) stipulates that a QEP must offer a “definition of student learning [italics added] appropriate to the focus of” that document (p. 25). Union College places that definition in a pre-existing matrix of expected outcomes. Its mission statement outlines the objectives the college sets for itself: Union College will “creat[e] personal, intellectual, and social transformation through the pursuit of knowledge, wisdom, service, and spiritual growth” (Catalog 2005-06, p. 3). Further catalog statements of “Vision,” “Core Values,” and “Goals for General Education” reflect discussions among faculty, administration, and students which have identified learning outcomes. Thus, the category personal growth comprises (1) spiritual quest; (2) wellness; (3) philosophy of life; and (4) interpersonal growth. Union’s dedication to social responsibility encompasses (1) ethical reasoning; (2) service; (3) community and civic responsibility; (4) leadership; (5) cultural awareness; (6) world citizenship; and (7) ecological responsibility. And outcomes proposed for intellectual enhancement include (1) communication skills; (2) quantitative skills; (3) technological literacy; (4) career preparation; and (5) lifelong learning.
Taken
together, these statements emphasize the college’s dedication to learning as
transformation; however, many of the more qualitative outcomes have proven
difficult, if not impossible, to measure. The QEP will therefore focus on
enhancing student learning in the key competencies for which there is much more
agreement about measurable outcomes, affirming that writing, reading, and
quantitative skills are the foundations for the other kinds of learning and
growth. Attention to students’ “equipment for learning” will continue to
promote their personal and social development, their “equipment for living.”
In its first year the QEP will direct its efforts to students whose needs are most readily identifiable and for whom innovative learning strategies are most available; thus, it will focus on transitional (a preferred term to “remedial”) students in this baseline year, teaching them not simply the “basic skills” of reading, writing, and mathematics, but an even more basic skill: how to learn. But under the QEP, in later years learning how to learn will not be limited just to students who arrive on campus poorly prepared. As Gerald Graff (2003) has noted, first-year students of all abilities usually come to colleges ill-prepared for the conversations of academia: “The college curriculum says to students, in effect: ‘Come and get it, but you’re on your own as to what to make of it all’” (p. 3). In its first year, QEP’s cycle of action and assessment will train the faculty and administration of Union College to develop initiatives that can be extended beyond transitional learners to students at large. Thus, as it evolves, the plan will extend its goals, eventually reaching all students with every level of ability. In short, through a program of assessment and initiative, the QEP will enhance every student’s core skills within five years of its implementation.
In its first year, however, the QEP will limit its scope to first-year students in General Education courses. Their abilities will be assessed, both before attending classes at Union and after. Students who demonstrate the need for transitional coursework will be assigned to appropriate sections of composition (ENCO 101) or mathematics (MATH 101) or both. These sections will carry the same academic credit, three hours, as other sections of the courses. Students in these sections will complete the same kinds of assignments as students in earlier years. Under the QEP, however, they will be engaged to a greater extent, both in the classroom and out, than their predecessors. Thus, designated sections of the first semester composition and literature course, ENCO 101, will use the rhetorical praxes of Gerald Graff and Cathy Birkenstein-Graff to prepare transitional students for academic as well as everyday writing and the well-tested writing workshop model proposed by such theorists as Peter Elbow to confirm students’ ownership of their own writing and thought. Likewise, in its baseline year the QEP will focus specifically on transitional mathematics students, not simply for the readily evident reason of helping them succeed but also to help Union College develop a culture of assessment and intervention that will address the needs of all mathematics students.
The following document provides details supporting the Union College QEP. First, it offers a history of the present QEP, including a narrative of how the present document evolved with the support of faculty, administration, students, and trustees as well as a synopsis of the previous QEP proposal. Second, a review of best practices in education and assessment demonstrates the theoretical basis for the plan. Third, this document details the measures Union will take under each segment of the ARIES plan: to assess the success of its programs in General Education, both as the QEP takes effect and then later, after its initiatives have produced measurable results; initiatives in place for the baseline year, including transitional courses, tutorials, and workshops for student learners; review of the success of its initiatives followed by whatever changes are necessary to ensure students’ intellectual growth; education of faculty and staff; and strategies for further assessing students’ growth, broadening the scope of QEP assessment from academic skills to other measures of student success and satisfaction. Fourth, the document details the allocation of resources needed to implement the plan. And fifth, an evaluation plan and detailed timeline will insure that the QEP continues on schedule.
Assessing General Education: A First Attempt and What We Learned When
It Failed
In
Spring 2005, Union College presented the Southern Association of Colleges and
Schools accreditation committee with a Quality Enhancement Plan that the
committee later rejected. This proposal, “Improving Student Performance on the
Praxis I Examination,” sought to identify and correct a problem afflicting many
Union College students who sought teacher certification. Because many
prospective education majors failed the Praxis I examination, Union College
proposed to plan for their remediation. However, the number of students
affected, while large, did not meet the condition, stated in the Handbook for the Reaffirmation of
Accreditation, that the “topic or issue to be developed in the plan should
be sufficiently broad in scope to be viewed as significant to the institution
and a major enhancement of the learning experience for students” (p. 25). Nor,
after citing the need for remediation, did the plan actually propose any ways
of doing so. It was, thus, a “plan for a plan.” While its rejection was not, of
course, pleasant or widely foreseen, Union College now recognizes that it will
more successfully work toward a viable QEP by learning the lessons of our
failure and heeding the advice of that committee.
The
review committee’s reasons for rejecting the plan have proven eminently helpful
in drafting the present QEP. Specifically, the committee asked that several
specific standards be met, and Union College has complied. First, it
recommended that Union College establish a timeline based on expected learning
outcomes, strategies, assessment, and evaluation of outcomes the college
expects to achieve from 2005 to 2010 and to develop a QEP to address those
issues. The present document contains the timeline (Appendix C) and its basis.
Second, the committee specified that qualified people oversee the QEP project,
and that their professional apointments be related to the plan. Thus, the
current QEP subcommittee includes, among others, members of the social sciences
department who have worked diligently to evaluate and interpret assessments
from the institutional past, members of the humanities departments who, as the
major providers of General Education courses on campus, have the greatest
responsibility for guiding initiative, and members of the staff who will
provide much needed support services and coursework as the QEP takes effect. As
Chapter 4 indicates, several of these qualified people will continue the work
as members of the Assessment Council. Third, the committee recommended that the
college earmark financial, physical, academic, support, and administrative
resources, systems, and processes to ensure the plan’s success. Details of the
college’s compliance are also provided in Chapter 4. Further, the committee
recommended that every constituency on campus agree to the terms of the QEP and
commit to its development and success. The present document reflects the
participation of those constituencies.
The other two recommendations of the SACS committee concerned assessment. First, “The Committee recommends that Union College develop specific means for assessing the achievement of QEP learning outcomes that lead to the success of the Plan, and that relevant detailed internal and external measures be identified to evaluate the Plan.” Second, “The Committee recommends that Union College identify an internal system for monitoring progress, and describe in detail how the results of evaluation will be used to improve student learning.” Again, the following pages demonstrate that Union College has learned from its failure. In brief, it has determined what methods will best measure the success of student learning and the success of this plan in enhancing student learning.
Still,
research for the earlier plan provided the genesis of the current QEP. Although
the data were flawed in several respects, as the following synopsis details,
analysis of those data led to conclusions that agreed with many faculty, staff,
and student observations of Union’s most pressing problems. Thus, the following
“Praxis I QEP Review” prefaces later pages that describe what Union College has
learned about assessment and initiative from its Praxis I study. Specifically,
under the ARIES program, Union College will assess using several measures, both
external and internal, will assess regularly and publicly, and will make
practical use of assessment to create and refine a broadbased set of
initiatives.
Praxis I QEP review, with suggestions for getting the most out of a
mishap
Passing
the Praxis I examination is one of several criteria for admission into the
teacher education program at Union College. The examination requires that
students demonstrate proficiency in three primary core competencies
(mathematics, reading, and writing) to be admitted to the program. The
criterion for admission to the program is established by the Kentucky State
Board of Education and differs somewhat from test to test. In prior years,
Union College students’ pass rates on the Praxis I were apparently low compared
to Kentucky and national norms, raising questions about the efficacy of general
education at Union. In addition, study of the Praxis I scores indicated that
transfer students had lower pass rates than students who began and were continuing
their education at Union. The initial proposal suggested that to enhance the
quality of learning for both groups of students and thus improve pass rates on
the Praxis I, the college should consider such intervention strategies as
curricular improvements (revising the core curriculum or adding remedial or
test-taking courses) and improvements in advising and other support services.
Several
procedural errors clouded the analysis, however. For example, the initial
review of Praxis I performance focused on students at Union who took the paper
form of the test. When students who were taking the computerized version of the
test were included in the analysis, the disparity between performance levels at
Union and the Kentucky and national norms was reduced. Secondly, review of the
data disclosed that it included information from students who never attended
Union College or who were counted twice due to name changes (following marriage
or divorce). In addition, the college counted every attempt to pass the test
while Kentucky counted only the most recent pass-fail performance of each
student. Using the state method of counting, the pass rates of Union College
students were at or slightly above Kentucky averages. Another potential source
of error involved a change in the requirements for taking the Praxis I
series—that is, beginning in Spring 2003, students with an ACT comprehensive
score of 21 or higher were exempt from taking the Praxis I. Thus, while pass
rates through 2001-02 included students with ACT scores of 21 or higher, those
from Spring 2003 on did not.
Figure 1.1
Such internal validity errors seriously compromise
the conclusions that can be drawn from analysis of the data.When they were
corrected or removed as a source of variation, some interesting results were
noted: first, that the number of students taking Praxis I declined from a high
of about 275 in 2001-2002 to a low of 152 students in 2003-2004; and second,
while reading and writing failures increased in years 2001-2003, failures in
mathematics declined in 2003 compared to earlier years. Still, the overall
failure rate for Union College students was roughly twice the national failure
rates for 2003-2004, as shown in the chart below (UC Expected Failures were
defined by the national proportion of students who failed each test).
If these
teacher education students reflect the wider student population at Union,
enhancing learning in the core competency areas of mathematics, reading, and
writing should be a priority.
At
the same time, the design of the initial QEP studies did not consider other
data available for analysis. For example, most students entering Union College
have ACT assessment scores (or SAT scores) and high school GPAs, both of which
provide a basis for quantitative assessment of core competencies. Second, by
the time students reach the point of taking Praxis I, they have had several
semesters of academic work, primarily in general core courses designed
specifically to deal with content areas being assessed by the Praxis I. Third,
the high failure rates on the Praxis I might be predictable if an analysis of
ACT, GPA, and academic course work grades were conducted, and if predictable
would identify a potential issue that might be addressed in the QEP—for
example, a need for revision of the core curriculum in math, reading, and
writing courses. Through such revision, or through remedial courses early in a
student’s academic career, failure rates on the Praxis I examination should be
reduced. However, leadership and personnel changes in the offices of Special
Programs and Administrative Systems have left gaps in data collection and
analysis, so that reliable data on student ACT scores were not available.
Thus,
the initial Praxis I study revealed a need for better assessment of all
students’ performance in core competencies and of their progress toward the
skill levels expected of a Union College graduate. While Praxis I is an
effective measure for assessing students’ readiness for the teacher education
program, other instruments could provide more reliable information about
general student performance. For example, the Collegiate Assessment of Academic
Proficiency (CAAP) is specifically designed to address proficiency at certain
benchmarks in students’ academic careers and could also be used as a basis for
assessing performance at other times during their educational experience at the
college. Such a test could provide a more complete picture of student
competency, early, middle and late in the educational process. When coupled
with other assessment instruments at the beginning of the educational process
(the COMPASS, for example), or its end (terminal performance measures such as
the Graduate Record Examinations or a terminal CAAP assessment) a full picture
of general core competency as well as proficiency in major curricular study
areas may be obtained.
A preliminary step toward this broader assessment occurred in March 2005 with administration of the CAAP to student volunteers. A total of 34 students participated, with 33 of those being freshmen. All 34 students were full-time students, the majority of whom (32) planned to return to Union in the fall. All but two of the students had GPAs of 2.00 or better. Self-reported performance efforts indicated that 24 students took both the writing skills and reading portions of the CAAP, while 11 took the science segment. Only 4 students took the mathematics segment. No students engaged the Critical Thinking or Writing (Essay) sections of the CAAP. All but one student failed to provide a response to questions regarding the effort they put forth while taking the CAAP, and the only student who provided a response indicated that he/she gave the test little effort. Students (n =24) taking the Writing Skills assessment scored from the 1st through 85th percentiles against national norms. Only 7 students attained scores above the 70th percentile point. Students who took the Mathematics segment (n = 4) scored from the 55th through 84th percentile in the reference group. Students scored in the first through 97th percentile on the Reading segment, with the majority scoring below the 70th percentile. Students (n =11) who took the Science portion of the CAAP scored from the 17th through the 97th percentile, with the majority scoring at or below the 70th percentile. Beyond a rough delineation of percentile ranges, little additional information can be gleaned from the CAAP report. What can be said is that generally, despite the small n, the results fell within the semi-interquartile range on the Writing Skills and Reading assessments, while the results of the Mathematics assessment taken by n = 4 students were in the average to above average range. Science scores (n = 11) were relatively average. The table below summarizes the results.
|
Table 1.1 |
|
CAAP Spring 2005 Results |
|
|
|
|
|
|
|
|
|
Area |
Union College |
National PB |
Union Z-Scores |
Comments |
|
Writing Skills |
59.8 |
64.5 |
-1.00 |
n =24 |
|
Math |
55.8 |
58.1 |
-0.58 |
n = 4 |
|
Reading |
58.1 |
62.5 |
-0.83 |
n = 24 |
|
Critical Thinking |
0 |
62.4 |
Not assessed |
n = 0 |
|
Science |
59.3 |
61.1 |
-0.40 |
n = 11 |
|
|
|
|
|
|
|
Area |
Union College |
National PB |
Z-scores |
Comments |
|
Writing Skills |
59.80 |
64.50 |
-1.00 |
n = 24 |
|
Usage/Mechanics |
15.60 |
17.20 |
-0.73 |
n = 24 |
|
Rhetorical |
14.90 |
17.30 |
-1.00 |
n = 24 |
|
Math |
55.80 |
62.40 |
-0.58 |
n = 4 (estimated) |
|
Basic Algebra |
20.29 |
15.30 |
2.08 |
n = 4 (estimated) |
|
College Algebra |
20.29 |
15.20 |
1.96 |
n = 4 (estimated) |
|
Reading |
58.10 |
62.50 |
-0.83 |
n = 24 |
|
Arts/Literature |
14.00 |
15.60 |
-0.64 |
n = 24 |
|
Social Sciences |
14.20 |
16.50 |
-0.85 |
n = 24 |
|
Critical Thinking |
0.00 |
62.40 |
Not assessed |
n = 0 |
|
Science |
59.30 |
61.10 |
-0.67 |
n = 11 |
|
|
|
|
|
|
|
Composite Scores Not Reported |
|
|
|
|
|
|
|
|
| |