Page 121

CHAPTER 9

Evaluating Learners, the Learning Process, and Yourself

As an instructional designer for a local community college, Mark Hayes knew that evaluation was one of the most important aspects of any program. After all, the information obtained from evaluations lead to various improvements in the program, instruction, and decision-making process. But Mark was concerned by a recent phone conversation with Miriam Rashad, an instructor in the Psychology Department, who believed that evaluation was a waste of time. Miriam believed that teaching evaluations were more of a popularity contest than anything else. Worst yet, she felt they did not seem to provide any meaningful information about how to improve the instructional process or the learning experience.

Mark decided to investigate why Miriam had such negative views about evaluation by paying her a personal visit. He quickly learned that Miriam viewed evaluation very narrowly; its purpose was perceived to be that of evaluating students not instructors. It also was something that occurred at the end of a learning experience so how could instruction be improved once it was over? Mark showed Miriam how

Page 122

evaluation could be used as a tool for improving the learning experience by using a sentence completion tool midway through the course. He also discussed various evaluation techniques that could yield fair and objective information about her teaching performance and the conduct of the course. Miriam thanked Mark for sharing some of his ideas and views about evaluation and promised to consider some of the techniques he had suggested.

As Mark pointed out, a key feature of any educational endeavor is the process of determining what is to be accomplished. Our society places great value on how well we are able to attain goals and objectives. The skill that we exhibit as instructors in providing feedback to learners, guiding the teaching and learning transaction, and evaluating instruction is central to our overall success (Steele & Brack, 1973).

Learners, too, often need to develop skills in evaluating and validating their own learning experiences. The process we use to assess teaching and learning outcomes is typically referred to as evaluation. This chapter discusses what evaluation is and how we can use it effectively in the context of individualizing instruction.

What Is Evaluation?

The concept of evaluation is not new. As early as 2,000 B.C., Chinese officials were conducting civil service examinations. The famed Greek teacher and philosopher, Socrates, reputedly used verbally mediated evaluations as part of the learning process. In the United States, the first reported user of program evaluation was Joseph Rice, who in 1897-1898 compared the spelling performance of 33,000 students in a large city school system. With the advent of IQ testing and other standardized achievement measures in the 1920's and 1930's, evaluation practices flourished in the United States. More recently, such evaluation measures as the Scholastic Aptitude Test (SAT) and the Graduate Record Examination (GRE) are in common use in most postsecondary educational institutions today as evaluative measures of aptitude and ability (Worthen & Sanders, 1973).

Page 123

Some typical forms that evaluation takes in instructional settings are course grades, scores on standardized measures such as an IQ test or the measures noted above, and performance feedback from a supervisor. So pervasive is the need for or use of evaluation today that it can be found in nearly every type of organizational setting.

Most instructors would define evaluation as the process of judging the quality of an educational enterprise is judged. Put another way, the term "evaluation" means appraising the value or worth of some educational undertaking such as a curriculum, a particular instructional procedure, or an individual performance in some area of learning (Popham, 1972). Generally, most evaluation efforts in education are undertaken with the aim of improving a given instructional situation through informed decision-making. For example, evaluation can be used to answer such questions as how effective is the use of discussion with a group of experienced adults? Or, in a more personal way, how well am I communicating with the learners with whom I work?

1. The two main approaches to analyzing educational activities are summative and formative evaluation.

There are many different types or models of evaluation. Some emphasize process, others product, while still others focus on learner outcomes. Despite this array, there are generally two main approaches to evaluation. The first of these is called summative evaluation. Summative evaluation is usually conducted for the purpose of determining learner performance, justifying the worth of a program, or as a means of ensuring accountability. Such evaluation forms usually are administered after a learning experience is completed. Some common examples used to determine how well participants performed include final examinations, term projects, oral presentations, and supervisor ratings. We ourselves use summative procedures to determine how well we did as instructors and how well the course or training session was received. These procedures include instructor evaluation forms, questionnaires used to obtain reactions to a particular session or technique, and course or workshop evaluation forms.

A second approach is called formative evaluation where the entire instructional process is evaluated during its implementation with the aim of using any resulting information for ongoing improvements or changes. This evaluation approach seeks to assess determine

Page 124

how well the instructor and learners are doing so that any problem or shortcoming can be remedied. Examples of formative evaluation include an end-of-session evaluation slip, a sentence-completion form that asks among other things, what is the most interesting part of the learning experience as well as what is least interesting, and informal interviews with selected learners.

One of our colleagues also has had success with an evaluation team made up of class members who provide periodic feedback on the course, instruction, and learning materials. We have found that all these formative evaluation techniques work well with adults since they are most responsive to their needs while at the same time being consistent with our goal of ensuring learner success.

2. Effective instruction results from evaluating both how well learners are performing and how well you are doing as an instructor.

One of the key characteristics of good instructors is that they are constantly finding ways to improve their instructional skills. Even the best instructor realizes there is always room for improvement. But exemplary instructors go even further. They realize that the true test of their instructional effectiveness is how well each learner performs. To some instructors, this view may seem threatening since it is customary to place the burden of learning entirely on the learners. However, in our view, instructors have a responsibility to organize and deliver their instruction in such a way that optimum learner performance is assured. One of the most effective ways to ensure that this occurs is by using both instructor and learner evaluations that follow the formative approach described above.

Instructor evaluation involves direct feedback from learners. As will be described in more detail later in the chapter, we ask learners to provide both formative and summative evaluation of our facilitation efforts. We also help learners design evaluation strategies through the learning contract and assist them in thinking about evaluation in terms of planning efforts. We also encourage them to become more adept at assessing their own learning activities.

We have found that some instructors carry out evaluations

Page 125

only because they are required. At the same time, we have found many instructors to be extremely interested in receiving feedback about how they can further improve their instructional methods. There are many reasons for these varied views of evaluation but most relate to the instructor's evaluation philosophy and to institutional constraints with which the instructor must deal.

3. Evaluation efforts should be designed in light of both the instructor's personal philosophy and the sponsoring institution's philosophy.

In a number of chapters throughout this book, we have discussed the importance of understanding your own instructional philosophy and that of the sponsoring agency or institution in which you work. We have noted that there should be some consistency between your philosophy and your classroom practices (Cervero, 1988; Schon, 1987). As we have noted, too, our own philosophies of instruction have evolved from a point where we tried to control all aspects of the instructional process to our present position of sharing that responsibility with learners. Evaluation feedback from learners about our instructional skills and procedures have been instrumental in helping us make these changes. We hope that reading this book will prompt you to take a hard look at your own instructional practices and associated philosophy, and urge you to use various evaluative strategies as you carry out such reflection.

4. Evaluation should be put in the context of learning goals and objectives.

One of the most important aspects of the individualizing process is linking evaluation to what is to be learned or accomplished. This can be initiated by having the learners set forth their learning intentions in the form of objectives. Clearly stated objectives that emphasize such terms as writing, solving, and listening, which emphasize certain behaviors or goals, are desirable since they can be used to assess learners' progress in achieving minimal to optimal proficiencies (Knox, 1986). They also provide the basis for specifying the resources that will be used to accomplish the learning goals, the various activities that will be employed in the learning effort, the resulting products or evidence, and any assessment procedures.

Page 126

Grading and Testing

Perhaps nothing is more sacred in most educational settings than the assignment of grades. Nearly everyone has an opinion on the subject, although people often accept prevailing views uncritically. We have agonized over the appropriateness of grading and have concluded that it is acceptable as long as learners know what is expected and have some say in what grade they are working toward. Thus, as we noted in Chapter Eight, in our credit courses we ask participants to specify on the learning contract the grade that they are working toward. Our experience with this approach has consistently been that adults appreciate this grading practice once they become familiar with it

At the same time, conflicts can arise between institutional expectations and those of the learners or the individual instructor. Certain educational agencies or institutions might view as scandalous the idea of granting learners some role in determining their grades. If this is the case, you will need to reckon with this philosophy and exercise some caution in allowing learners to choose the grades they are working toward. With careful planning and a solid rationale for why and how you are implementing your grading practice, however, few educational agencies or institutions will put too many roadblocks in your way.

5. The grade a person earns should be based on criteria negotiated between the learner and the instructor.

Although grades are something we all must live with, they often present a dilemma for instructors who use the individualizing approach. Thus, grades are actually normative measures designed to assess an individual's performance with respect to other individuals who have been measured similarly. For example, an instructor might use standardized midterm and final examination to compare and grade learners (Popham, 1972).

The individualizing process advocates a different kind of evaluation procedure. Referred to as criterion-referenced evaluation, it is designed to help learners assess personal progress according to some criteria or standards of performance. The performance criteria often are self-established and based on the needs and competence

Page 127

levels of each learner, who works individually at demonstrating mastery or accomplishment. In this form of evaluation, each learner indicates on the contract what will be learned by listing learning objectives, the activities that will be used to achieve these objectives, and the criteria that will be used to demonstrate mastery of them. Thus, in such an evaluation effort the instructor's role becomes one of helping learners know when they have accomplished their learning objectives while at the same time helping them to become more competent at assessing their own learning.

6. Differences between an instructor and learner in respect to grading and evaluation plans should be negotiated early in the instructional process.

Where have occasionally encountered differences between learner expectations and our own. For example, certain learners have felt that they deserved a higher grade than ultimately received for a particular learning experience. The reason for this discrepancy can best be described in terms of the difference between summative and formative evaluation measures.

For example, we attempt to link grades with learning activities as early as possible in a course. We provide thorough explanations of each learning activity and what the criteria are for demonstrating mastery. We also grade on a pass/incomplete basis, issuing incompletes for those learners who do not provide evidence of mastery or competence as initially set forth in the contract. We then encourage learners to follow a formative evaluation approach by turning in materials on a regular basis as the learning experience unfolds. We are thus able to offer any necessary suggestions for improvement so that modifications can be made, critical thinking can be demonstrated, and ultimate mastery can be achieved. Although some learners do challenge our assessments, in general we are able to negotiate an acceptable compromise through individual conferences and discussions.

Unfortunately, what happens in a few cases is that learners fail to adhere to their contracted time lines and wait until the learning experience is over to turn in assignments. We remind learners when assignments are due through announcements in class or short memos. However, if someone falls way behind a reasonable schedule, we must move from a formative evaluation approach to a summative process. This necessitates evaluating the quality of the various

Page 128

written materials or other evidence of learning received. In many cases, learners who have procrastinated receive a lower grade than expected unless it is possible to work out with the learner and within institutional guidelines an incomplete grade that can be improved at a later date.

In order to cut down on those situations, we frequently discuss our evaluation practices during the first few hours together with learners, noting the differences between formative and summative evaluation and the consequences of waiting until the end to turn in learning activity materials. We also stress that individualizing their learning and subsequent evaluation activities requires advance planning and careful time management. Despite such interventions, we still get an occasional learner who has a problem completing the learning activities on time.

Tests are, of course, another aspect of the evaluation process that can be used, depending on the instructor's philosophy or the institution's requirement. We generally view testing as an appropriate tool for evaluating competence, but we tend not to use it in our own graduate classes or training workshops unless a particular learner requests it on the learning contract. In most cases, learners select other means for demonstrating mastery, such as simulation exercises, interactive reading and writing activities, or some sort of project.

We feel strongly that if you use testing, you should provide opportunities for follow-up discussion and debriefing. Tests should be developmental tools that give people the opportunity to learn from the mistakes they make. Too often, especially with summative forms of assessment and evaluation, learners simply take a test and then have no opportunity to know how they did other than their given grade or score.

Learner Assessment

7. There are any number of techniques that can be used to assess learner progress and competence, ranging from observations, to interviews, to written reactions to materials that are submitted.

We encourage a mixture of assessment practices or tools organized around a set of suggested learning activities. We have successfully used these learning activities many times in our courses, and

Page 129

we present them as guides for thinking about possibilities or as presumptive devices for designing contracts or learning approaches. However, such activities are only suggestions, and alternative activities may be selected and completed through the learning contract.

We also encourage participants to select activities that reflect different levels of learning, ranging from basic knowledge or comprehension to higher order synthesizing and analyzing. Such a selections process includes thinking creatively about how their learning activities will be assessed and evaluated. The evaluation process can also involve two or more learners working together on various learning activities, where they can carry out self-evaluations through synergistic exchanges.

Initially, learners tend to rely on familiar assessment measures such as tests or final exams, and ,in some cases, expect us to tell them what assessment techniques to select. We spend considerable time discussing such alternatives as peer evaluations and professional assessments from experts in the community or their place of work. Following this kind of discussion learners usually begin to think critically about the relationship between their intended learning objectives and how these objectives will be assessed. This typically leads to enhanced responsibility for and greater confidence in their learning.

We also use a number of less formal assessments in tracking learner progress. These often take the form of in-class discussions with the learners, personal observations, and informal interviews with learners during a break period. What we are typically looking for is whether learners are internalizing the course-related information and how this is affecting them as developing professionals. We are also constantly on the lookout for nonverbal cues such as puzzled facial expressions and angry gestures. These sources of information provide us with further data about how each learner is absorbing the material and the resonance they feel with it.

During breaks we try to make contact with as many learners as possible. We ask participants informally about any problems they may be encountering, how their learning activities are progressing, or what they feel is a high-point thus far. Inevitably, various problems emerge with acquiring particular resources, or

Page 130

there may be misunderstandings about a learning assignment. With this information at hand, we are better able to help learners find acceptable solutions so that they can focus more of their energies on their various learning activities.

8. It is important to obtain formative and summative evaluation feedback on how the conducted your course or learning experience.

We also make extensive use of course evaluations in our individualizing process. We typically use two procedures that are in keeping with the formative and summative evaluation approaches described earlier. The first is a midterm evaluation instrument that consists of five open-ended questions (Exhibit 3).

We can adapt this form to workshops, conferences, and even staff training settings. The mid-point evaluation effort allows us to gather information about the progress of the learning experience so that adjustments can be made to improve it. On many occasions, such evaluation efforts have turned up some important items that we had simply overlooked. For example, in one graduate course, participants asked for more lectures and less group discussion in response to the question, "I wish we had . . . ." In a training workshop, learners indicated that one particular instructional topic was so meaningful that they wanted additional group time spent on it. In both instances, the information provided by the mid-point activity resulted in corresponding changes in the remaining learning experiences.

A second approach that we have used with considerable success is an evaluation instrument administered at or near the end of the instructional experience (an example of this instrument is reproduced in the resources section). It consists of a variety of items designed to gauge the overall effectiveness of the learning experience. We ask participants not to include their name unless they have a particular reason for letting us know they have raised certain points. We are interested in receiving anonymous feedback on the various techniques used to present information during the experience, whether it was a lecture, a group discussion, or a guest speaker. We also want feedback on the various topics explored during the learning experience as well as on the materials utilized, such as handouts and workbooks or study guides. Additionally, we ask for each learner's assessment of the textbooks, the learning activities,

Page 131

Exhibit 3. Midterm Evaluation Form

Please Complete Each of the Following Statements

This class has been........................................................................ . . . .	Other comments............................................ . . . .
The most interesting part of this class has been............................... . . . .	Why was it interesting to you?........................ . . . .
The least interesting part of this class has been................................ . . . .	Suggestions and recommendations................. . . . .
I learned........................................................................................ . . . .	I wish I had learned....................................... . . . .
I wish we had................................................................................ . . . .	Other comments............................................ . . . .

Page 132

and the learning contract. This evaluation procedure follows the summative approach. We conduct an extensive analysis of the information provided by each participant and use this information to improve subsequent offerings.

9. We also recommend that you ask for an evaluation of yourself as a learning facilitator.

In addition to receiving information about the conduct of a learning experience, we also believe that some appraisal of our performance as instructors is vitally important. For many instructors, this area may be a highly sensitive one. Yet, receiving feedback on your teaching is essential if you are to improve your performance.

Many instructors must also solicit some sort of evaluative information because their sponsoring agency or institution requires this of them. For example, as a requirement for tenure, junior faculty at most colleges and universities must demonstrate instructional effectiveness. In business and industry, trainers must show instructional competence if they are to keep their jobs. In adult basic education, literacy instructors are evaluated on how well they can teach, and their continued employment is often tied to this appraisal.

For our classes, we use an instructor evaluation form that consists of a number of items arranged on a scale of 1 to 5 with opportunities to provide open-ended comments. (A sample of this form also is reproduced in the resources section.) It provides substantial information about how effective group time was in promoting learning, the instructor's interest in learners, and the instructor's mastery of the content. Additionally, the instrument helps assess whether the material was presented at an appropriate level, the instructor's control of the group and tolerance for varied opinions, and how effective the instructor was overall. We take this information very seriously and incorporate the findings into our instructional activities. In fact, such an evaluation procedure was very instrumental in helping us refine the individualizing process.

The importance of conducting both formative and summative evaluations can not be overstated. Both types of evaluation provide information that can make the difference between success and failure as you employ the individualizing process.

_______________________________

Go to the bibliography.

Return to the book's table of contents.

Return to the book's index.

Return to the first page.