FAQ | Plan for Assessment | Implementation Schedule | Assessment Strategies | Model

Comprehensive Assessment Program
 

Overview 

·        The program provides information that is meaningful for students.  Students participate in a variety of assessment activities, and the focus is to ensure that these activities positively affect students and their needs or expectations.

·        The program focuses on assessing and improving programs and services as opposed to individuals. 

·        External controls such as academic accreditation and program reviews will continue to be a major focus of academic assessments, but emphasis will also be placed upon achieving internal commitment to the assessment program by all participating units. 

·        The program is comprehensive in that more than 160 academic, administrative, and support units participate.  While facilitating internal improvements of programs and services, this single program is intended to serve a multitude of both internal and external reporting and accountability requirements.

·        The program is integrated across all campus units as the mechanism for aligning unit planning with institutional planning.  Furthermore, it is integrated as a critical component of the planning, evaluation, and budgeting cycle and is the basis for change and improvement.  As a planned and regularly scheduled activity shared by all units and communicated across various layers of the organizational structure, it ensures cohesiveness in purpose and direction.  

·        The program is systematic.  All units are expected to apply the same data-driven assessment philosophy and practices as they engage in assessment activities.

·        The program is ongoing.  Assessment is a never-ending process that requires annual participation. 

·        The program is decentralized.  Each unit determines and conducts most of its assessment activities.      

·        The program is evolving.  Assessment activities allow flexibility and change in response to changing or environmental needs, results of assessment activities, and the identification of improvement opportunities.

·        The program is unit or discipline-specific.  Each unit or academic discipline has a unique purpose and goals, which are assessed separately from other units. 

·        The program is cyclic.  Assessment activities occur in cycles, generally in response to planning and resource requirements.

·        The program involves use of multiple measures.  Assessment activities may require use of several methods, instruments, and measures to gauge progress toward some intended outcomes.

·        The program uses results of assessments as the means for making improvements rather than as an end. 

• The program is supported by personnel, training, and an easy-to-use electronic data base (TracDat).  This database is used to facilitate systematic data collection, storage, and reporting.  An Assessment Coordinator is housed in the central administration who oversees and coordinates the program.
   

Benefits Expected from the Program

Though most benefits of the program are at least implicit in the characteristics described above, it may be helpful to focus on answers to specific questions such as:

Why does ETSU need a systematic assessment process?

A systematic process will ensure that all units use the same assessment plan components, definitions of terms, database, calendar used in the assessment cycle, and reporting designs.  This consistent and unified approach will result in a better campus-wide understanding of assessment and the database.  Reports can easily be produced and understood by internal and external audiences because all units will use the same input formats.  Strategic planning will be enhanced through the availability of assessment data provided on a regular cycle.  

How does the assessment program relate to planning, evaluation, and budgeting?

As shown in the attached schematic, strategic planning, evaluation, and budgeting are linked.  Planning is an ongoing process designed to identify the direction the university and its units intend to follow in accomplishing their mission and goals.  Evaluation (assessment program) is needed to determine progress toward attaining these goals as well as to facilitate development of action plans and corrective actions.  Budgeting is needed to identify costs/other resources needed to support implementation of planning and evaluation activities.

How will we know we are successful in implementing the program?

Success Indicators: 

• Assessment plans will be well written and maintained by all units.
• Assessment data will be available in unit assessment plans, and units will demonstrate how the data were used to make improvements.
• There will be evidence that significant resource allocations are based on data provided through the assessment process.
• Internal and external accountability reports will be met with greater success because of participation in this systematic, integrated process, and the demonstrated use of results.
• Significant improvements in programs and services will be evidenced and promoted.
• All units will have effective assessment plans housed in an electronic data base.  Useable is defined as “demonstrating evidence of facilitating ongoing planning and improvement of expected outcomes, processes, or services through the use of assessment results.” 

How will units benefit from the program?

The most obvious benefits of the program are:  (1) units will achieve a better understanding of how to refine their assessment plan to produce data that drive improvement and support resource decisions, (2) units will incorporate assessment activities into their annual calendar, (3) units will see a improvement in morale and teamwork not only by involving their faculty/staff in planning but as result of improvements stemming from the use of assessment data, and (4) units will find that less effort is needed to prepare for reaccreditations and program reviews and to meet a host of other external reporting requirements as a result of their ongoing participation in assessment activities.

How does the electronic data base (TracDat) fit in all of this?

The focus of the model is to facilitate the refinement and/or development of effective assessment plans.  Although the data base is critical to the assessment program, it is not the emphasis.  We have jumped that hurdle through training provided as Phase I, and adequate understanding has been demonstrated by a data base audit.  Participating units know that TracDat is the current data base into which unit assessment plans will be housed electronically so that they can easily produce a history of their progress to facilitate improvements and to serve a multitude of purposes and audiences for accountability using a variety of reporting designs. 

What kinds of reports will units be required to produce from assessment data?

Most units will find that many of their internal reporting needs are met by simply hitting one keystroke.  Currently, users of the assessment data base can select from up to 11 report designs.  The query builder allows users to produce additional reports.  The Office of Institutional Effectiveness and Planning will assist campus units in aggregating data and preparing summary reports that will meet reporting needs for accreditation, program reviews, performance funding, internal planning, and other external agencies.

Current Status of Assessment Plans

Currently, approximately 150 units have developed assessment plans, and they have entered them into the data base.  However, like other initiatives that involve use of new technologies, time will be needed to achieve campus-wide success in developing effective assessment plans and in achieving a better understanding of features and capabilities of the data base.  The key to improving understanding is effective teaching and use of instructional materials on assessment, continued training, and experience.  Based on a recent review by the Assessment Coordinator of progress that has been made by units this year in the development and use of assessment data for improvement, many units have a good understanding of assessment and are positioned to move forward.  Most will profit from additional instruction and training.

Proposed Training Calendar by Unit 

The attached schedule for instruction and training that will be provided between 2001 and 2005 is shown for more than 190 units or programs.  It lists each department by year, along with appropriate program review/accreditation dates.   The intent was to schedule academic departments for training a year or two in advance of their next program review or reaccreditation visit.  This proposed schedule will serve approximately 40 units or programs each year.  Some units may need to either advance or delay their participation within this calendar and will be permitted to do so. 

Model Used to Refine Assessment Plans

The suggested model for units to use in refining their assessment plans calls for use of a more deliberate approach in which individualized instruction and training will be provided to selected units over the next four years.  It is especially important that priorities for this assistance be given to academic units since they serve students directly in the teaching and learning process.  The basis for determining these priorities was the calendar cycle of accreditations and program reviews.  Approximately 20 academic units/programs have been selected each year to work closely with the Assessment Coordinator, the Director of Outcomes Assessment, or the Director of Institutional Effectiveness and Planning.  It is further anticipated that another 20 administrative and support units will also receive this assistance.  The intent and scope of this assistance will be three-fold:  (1) to assist units in preparing effective assessment plans that serve internally in providing data for improvements in teaching, learning, research, or service, (2) to assist units in developing assessment activities for inclusion in their plan that will lead to subsequent preparation of documents needed for a program review, reaccreditations, or other external accountability needs, and (3) train units how to use features of the data base that will meet their unique needs.  Additional details regarding these three areas of assistance are provided below.  

1.  Assistance Intended to Increase Understanding of Assessment and How to Refine Plans That Will Produce Data Used Internally for Improvement

This type of assistance is recommended for most units and will be the primary focus.  However, several units have already developed effective assessment plans intended primarily for internal use that will only need to continue making needed refinements.  Reports produced from the data base demonstrate clearly that many academic and administrative support units already understand how to develop effective assessment plans.  Units are expected to follow the lead of these departments by taking advantage of individualized instruction and training that will be provided.  Instruction and training in the form of intermittent meetings, appropriate handout materials, and e-mail correspondence will be used to achieve the results desired.

2.  Assistance Intended for Academic Units to Increase Understanding of How to Refine Plans That Will Facilitate Preparation for Program Reviews and Reaccreditations      

The Office of Outcomes Assessment maintains a calendar of scheduled program reviews and reaccreditations.  This calendar will be used to determine the order in which academic departments are expected to receive assistance in developing or refining their assessment plans.  Resources needed to conduct assessments will also be addressed and resolved, as will issues involving data collection, analysis, and reporting.   Instruction and training will occur through intermittent meetings, appropriate handout materials, and e-mail correspondence.  Follow-up meetings will be needed on an ongoing basis  to make any needed corrections and to ensure progress and achievement.

3.  Assistance Intended to Increase Understanding of the Electronic Data Base by Tailoring Training to the Needs of the Department 

The Assessment Coordinator will continue to provide training and assistance through one-on-one meetings and periodic group training provided in Gilbreath Hall.  The focus will be upon exploring ways to use the appropriate data base features that are most suited to the needs of the unit.  It is anticipated that several meetings may be needed.  While priority will be given to selected academic units, the Assessment Coordinator will work closely with any department head throughout the year to help them improve their understanding and use of the data base.  It is further anticipated that units will provide some useful ideas that will be incorporated into new report designs.  

Major Components of an Effective Assessment Plan

ETSU’s Comprehensive Assessment Program requires campus-wide participation.  Each unit or cost center develops and maintains a unit assessment plan.  These assessment plans vary in accordance with the unique stated purpose of each unit, but they contain the same basic components.  That is, each is derived from a unit mission and goals.  Underlying the goals are carefully written objectives or intended outcomes along with an explanation of methods or procedures that will be used in conducting the assessment, an action plan indicating intended use of assessment results, actual results or data obtained from the assessment, and an explanation of how results were used (for improvement).   Separate from but linked to the model is a budget plan that identifies any resources needed to meet objectives.  Participating units will be given one-on-one instruction and appropriate handout materials to assist them in making refinements.  It is anticipated that this process, in conjunction with the kinds of assistance provided to support program reviews and individualized assessment data base training will require up to two months.

Steps Used to Design/Refine an Assessment Plan

Update the Mission.  (TracDat – Mission)

The first step in strategic planning is to update the unit’s mission statement (will be available on the web profile and in TracDat).  The mission is the purpose and scope of the unit, and it describes the direction or lays the foundation for developing goals and objectives. 

Update Goals and Objectives. (TracDat – Goals, Objectives)

The second step is to update goals and objectives.  Department goals focus the direction of  assessment activities to be conducted by the department, while academic program goals are specific to a degree program (will be available on the web profile and in TracDat).   Goals relate directly to the mission and describe the ends toward which action is directed.  Objectives describe actions that are needed to achieve the goal.  They usually contain an action verb and are measurable, attainable, contain a performance standard, and identify a time frame.  Since multiple measures or indicators of goal attainment are usually more effective than a single measure, more than one objective is often used with each goal.  Goals and objectives are necessary for relating HOW to achieve the unit’s purpose and IF it is being achieved. 

Verify Usefulness of Data. (TracDat – Feedback Loop)

The third step is to determine that assessments are directed toward collecting the kinds of data that are the most useful and pertain to the most critical needs of the unit or program.  For some units there will be several (3-7) needs that might be considered equal in importance.  All faculty/staff in the unit should be engaged in determining these needs and their priorities.  For academic units especially, the most critical needs will reasonably focus on intended program or learning outcomes.  This approach in an assessment model assures that appropriate data are collected as opposed to data being used that are simply available.  Questions need to be asked such as:  What are we assessing?  Are we assessing learning outcomes in the major/minor, general education courses, graduate courses, or performance outcomes/demonstrated skills, or attitudes/beliefs/values?  Are we conducting the assessments to achieve desirable program outcomes, improve programs/courses, in conjunction with a program review/reaccredidation, or an accountability measure/productivity measure, or for process improvement, sustained desired performance or achievement.  How will assessment data be used?  e.g. internally, externally, curricular revisions, or as a basis for requesting additional resources.     

Determine Assessment Methods and Procedures. (TracDat – Assessment Method)

The fourth step is to identify and describe procedures to be used in conducting the assessments.  Some units may decide to include criteria for success in their assessment procedures.  Data collection and analysis procedures are also included.  At a minimum, units must commit to completing at least one assessment during the first year that will include data collection, analysis, preparation of results, and demonstrating how results were used for improvement.  Academic units will be expected to address at least one student-learning outcome in meeting this requirement.  Ask questions such as:  Who will be asked to participate in the assessments?  Will assessment activities include new students, returning students, transfer students, alumni, graduate students, employers, administrators or faculty?  Are appropriate assessment tools available, or will we need to devise surveys, achievement tests, standardized tests, portfolios, performance measures, focus groups, capstone courses, unobtrusive measures, or capture anecdotal records?  Should all students in a class participate or only a random sample?  Who will analyze the data?  What statistical package and procedures should be used?  Who will conduct the assessments?  Who will prepare reports?  Are any costs involved?  Do offices exist on campus that provide services that involve assessment?  Will the assessments be conducted each semester, once a year, every two years, daily, end of the semester, or prior to graduation?

Prepare results. (TracDat – Observation)

The fifth step is to prepare results of data analysis procedures.  Results can be quantitative or qualitative,  e.g., statistics, ratios, rankings, percentages, dichotomies, ratings, or counts.

Demonstrate How Results were Used. (TracDat – Remedy)

The sixth step is to explain how assessment results were used,  e.g., to  make an improvement, change the curriculum, change a process, meet or satisfy an externally imposed standard, satisfy an accountability measure, or meet a program review requirement.       

 Proposed Assessment Calendar

Units that have developed and implemented functional assessment plans will determine appropriate dates and procedures to conduct their assessments.  Whenever feasible, units should produce “progress reports” for internal use only at the end of six months into the assessment cycle.  The purpose of these reports is to monitor progress toward important objectives and also to assist in identifying problem areas that need to be addressed and corrected mid-year.  Typically, considering ETSU’s strategic planning/budgeting cycle, these mid-year assessments and subsequent reports need to be produced by October.  In addition, units will be encouraged to provide any updates to their assessment plans throughout the year that could impact assessment outcomes, such as budgeting or staffing issues.  End-of-year assessments are also needed along with assessment reports and any budgetary requests by March.
 

Glossary of Terms

(Note:  equivalent TracDat terms are shown in parentheses)

Academic Program Assessment - evaluation of the performance and quality of credit-producing programs leading to a degree or certificate.

Academic Program Review – internal and external evaluation of all academic programs on a continuous five-year cycle.

Academic Units – units that offer credit-producing educational programs or certificates, such as colleges and academic departments.

Accreditation – external review by national, designated professional accrediting bodies.

Actual Outcomes (Observations) – results that were actually achieved.

Administrative Units – units that provide administrative or support services but do not offer credit-producing degree programs.  

 Annual Plan – one-year institutional objectives and strategies which are expected to be achieved so that priorities and timetables of the strategic plan can be realized.

 Assessment – the systematic process of gathering, interpreting, and using information to make improvements.

 Assessment Procedures (Assessment Methods) – description of how the level of accomplishment of a specific objective will be determined.

 Criterion for Success – specific requirements of an assessment outcome needed to attain a predetermined performance standard.

 Effectiveness Indicator – Indirect measure, statistic, or metric, used to assess the extent to which a unit is achieving its objectives. 

 Expected Outcomes (Objectives) – Desired results.

 Goals (Goals) – Compelling, clear statements of what the institution, unit, or program intends to do.

 Improvement Plan – Remedy or actions the unit will take if actual outcomes fall short of expected outcomes.

 Institutional Effectiveness – a qualitative indicator that the institution’s programs and services  are being provided in accordance with acceptable standards of quality and efficiency.  

 Intended Use of Results (Feedback Loop) – description of how units plan to use the results of assessments.

 Mission – (Mission) - expanded statement of purpose of the institution or unit.

 Objectives (Objectives) – Intended outcomes toward which the institution and its units are striving.

 Program Goals – statements of what learners should know or be able to do at the completion of their course of study.

 Results – Findings or what happened based on analysis of data.

 Strategic Plan – the long-range iterative plan that guides and directs the efforts of the institution toward achieving its goals.

 Student Learning Outcomes – specific kinds of knowledge, skills, attitudes, or beliefs that students are expected to demonstrate as a result of completing an academic, instructional, or experience-based program.

 Unit Plan – one-year objectives and assessment activities conducted by academic, administrative, and support units which need to be achieved if the Annual Plan is to be accomplished.

 Use of Results (Remedy) – description of how assessment results were used.

Search IEP website

Amy D. Goodson, Webmaster