Reprinted from the Handbook
of Contemporary Education by permission of R. R. Bowker Company, New York, NY.
Copyright
© 1976 by Xerox Corporation.
PROGRAM PLANNING AND EVALUATION
by Roger Hiemstra, Professor
Department of Adult and Continuing Education
Teachers College
University of
Nebraska
Lincoln, Nebraska
The term Program
Planning and Evaluation does not have a commonly agreed-upon meaning. The three words--program, planning, and evaluation--are commonly thought of together as part of an educational process
for organizing and administering
programs; however, an understanding of this process begins with separate definitions:
Program -a single activity, or sequential series of activities, designed to achieve one or more educational
objectives determined as necessary or desired in
promoting change in people.
Planning-the procedures or steps
utilized in designing a program.
Evaluation-an assessment of a
program's effectiveness,
usually in terms of its achievement of objectives.
The implementation of
a program is considered as part of the total process.
Planning and evaluation are widely
discussed topics in education, with a variety
of procedures being utilized to plan and implement educational activities.
Added to this diversity in procedures
are large numbers of commercialized
approaches to planning. Literature on planning and evaluation, gaming and simulation devices on problem
solving, courses and workshops to improve planning skills, and related
research reports are readily available. Much
of this information and many of these procedures are referenced at the end of
this chapter.
This abundance of
information is one indication that the design of effective education is a complex
process. In
addition, the process is dynamic in nature, in that developed plans are frequently subject to change
based on new and evolving information. Thus, there is no simple recipe tor successful program planning. Perhaps it
is this complexity and dynamic
quality that have made a simple definition of program planning and evaluation
so elusive.
However,
there are procedures and steps commonly employed in the systematic development of
educational activities. For example, the design of instruction, the development of an
annual conference program, the planning of an in-service training session for
teachers, or the development of a workshop for the continuing education of a group of
professionals, will be based on a process made up of differentiated but interrelated
steps. The focus of this chapter will be to describe this process and its various
components.
Finally, any
discussion of planning and evaluation will normally be based around the assumption
that some
type of change-be it change in knowledge, skill, or attitude-is an expected educational
program result. However, the focus of this chapter's discussion is the planning and
evaluating of primarily short-term programs based on some determined need for
change. Consequently, curriculum planning which tends to focus on anticipated or
long-range needs does not relate directly to the process to be described. Parts
of the process certainly are related to, and can be used for, curriculum planning, but the
sequencing of, and emphasis placed on, the various components usually are different.
RELATED LITERATURE
Literature on program
planning and evaluation is as diverse as are the approaches or models utilized
for planning
purposes. Some literature centers on the level at which one designs programs. Verner
(1964) has suggested that at least two levels should be considered: the first level deals with fulfilling assigned
social roles of the institution organizing
the program: the second deals with
activities, learning tasks, and needs at (lie client or program participant level. Knowles (1970) and Theide
(1964) support this contention, but
add that community or societal goals
also need to be considered in the planning
process.
Another portion of
the literature relates to establishing program objectives. Bloom (1956),
Krathwohl, Bloom, and Masia (1964), Mager (1962), Popham and Baker (1970), and
Tyler (1950) have been some of the pioneers in encouraging the utilization of
"instructional" or "behavioral" objectives in educational
programs. Although variations exist regarding what encompasses an educational
objective, most educators agree that desired
. . learner behaviors should be specified. Furthermore, the task of designing appropriate learning
activities is facilitated by precise
behavior specifications.
Somewhat related is
the discussion by Delbecq and Van de Ven (1970), Dutton (1 970), and Knowles
(1970) suggesting
that program participants should be involved in the planning process, which
includes the specification of
objectives. This assumes that involving the participant in the planning process will build personal interest in resulting
educational activities, and promote learning more central to actual need. This assumption is
supported by McLoughlin (1971); he
found that experimental subjects who
had been involved in objective-setting and program-planning had more positive attitudes toward the educational experience, and achieved us well
as control subjects.
The conceptualization
and development of planning models is another topic found frequently in the
literature.
For example. Beal, Ross, and Towers (1966) approached planning from a sociological
view, and designed a community level model that includes such functions as determining groups
to be involved in the planning process, obtaining legitimation for the
developed plan, and seeking community commitment. Variations on this
community-oriented approach can be found in Biddle and Biddle (1965), Lindeman
(1959), Sower (l957), and Thelen (1954). More recently, Boone, Dolon, and Shearon (1971) used these various
community-level models to develop a
conceptual scheme for planning that focuses on the decision-making process, Houle
(1972) has developed some thoughts toward using a decision-making process and planning system in
redesigning current educational programs.
Another approach to
model development for educational planning has stemmed from the systems analysis procedures used by business and industry.
Several authors (Carter, 1969; Churchman,
1968; Lehmann, 1968; Silvern, 1969, 1972) have developed educationally-related
systems analysis models. The critical path analysis approach (Cook, 1966; Justus, 1967; Kaimann, 1966) and Program-Planning-Budgeting Systems (Hartley, 1968; Stauber, 1968) are two additional systematic planning
approaches that have received considerable attention.
The assessment of needs prior to designing
educational programs has received
attention, too. Individual needs and
how to assess or define them (Dobbs, 1966; Johnston, 1963; Knowles, 1970; Knox,
1965; Leagans, 1964; Sheasga, 1961),
needs of groups or certain population segments (Dowling, 1969; Dublin, 1972; Hiemstra, 1972a; Long, 1972), and needs at a community or institutional level (Baumel, Hobbs, and Powers,
1964; Habib, 1970; Hiemstra, 1972b; McMahon, 1970) are the more common
categories around which this portion
of the literature has been organized.
The
literature on evaluation is immense. It ranges from textbooks or general
informational books on evaluation (Beatty, 1969; Bym, 1959; Gottman and Clasen, 1972; Suchman, 1967;
Thorndike and Hagen, 1969; Tyler, 1969) to specific evaluation materials or topics (Stake, 1970;
Wedemeyer, 1969), to critiques or discussions of evaluation problems
(Alexander, 1965; Cohen, 1970; Cuba, 1969; Scriven, 1972; Tyler, 1967). The
information on educational evaluation is so broad am! extensive that the above
citations provide only a starting-point.
A final body of
literature to be noted here centers on a growing awareness of research needs in
program planning
and evaluation. Caro (1969), Hemphill (1969), Merriman (1970), Mezirow (1971), Verner
(1962),
and Worthen (1968) are but a few of the authors attempting to apply research
techniques and findings systematically in building a comprehensive body of knowledge
regarding planning. These efforts should lead to a better understanding of the
total process.
NEEDS ASSESSMENT
Need can be defined as a discrepancy between what is known or can be
determined about a behavior and what should be (Gottman & Clasen, 1972;
Knowles, 1970).
The educator's role is to create an environment in which learning activities are utilized to reduce or remove the discrepancy.
Thus, needs
assessment becomes a necessary and crucial aspect of the planning and evaluating
process. A
determined need becomes the basis for one or more objectives. Without
needs assessment the planned educational activities could promote behavioral
changes with limited or no relationship to existing discrepancies.
Various tools and
techniques are available at the individual, group, institutional, and
community levels. These include observational techniques, projective
techniques, questionnaire approaches, community surveys, and job performance
analysis procedures. Space limitations do not permit their description, but
McMahon (1970)
provides an extensive bibliography regarding needs assessment. In addition,
references are included at the end of the chapter relating to each component in the planning and
evaluation process.
DEVELOPMENT OF OBJECTIVES
Most educational
programs revolve around assessed change requirements of three types: (1)
attitude modification, (2) behavioral change, and (3) skill or content mastery. An objective
is the description of an educational intent !o bring about one or more of these
changes. Identification
and specification of desired change are prerequisites to the design of
appropriate learning activities.
Wording objectives so
that they are precise enough to be useful is a frustrating and often poorly done part of educational program
planning. However, designing those appropriate learning activities, and
measuring whether
or not they have met some determined need, are dependent upon clearly stated objectives.
Mager (1962) suggests that a useful objective must identify the expected change,
define the important conditions under which the change is to occur, and define
the related
criteria for measuring the change achievement. Some of this chapter's
references will provide direction on writing and using objectives.
PLANNING THE
ACTIVITIES
Planning appropriate
learning activities entails the following: the identification of learning
tasks necessary for achieving a desired performance or change, the sequencing of these
tasks, and the determination of instructional techniques appropriate for both
the tasks and the program participant.
The program planning
and evaluation process can, and often does, begin at this component in the model. This occurs, for
example, when a program is built around a desired speaker or topic. However, including the earlier described needs assessment and objective selling
components can strengthen the
resulting programs. Thus, a program can be made more relevant if actual needs
are being met; the measurement of a
program's success can be facilitated if precise objectives have been
identified; and the actual design of learning activities can determine if objectives are realistic or achievable.
(Various tools, procedures, and techniques are available to facilitate this component's activities. Note the cited sources at
the end of the chapter.)
IMPLEMENTATION OF
ACTIVITIES
Operating the learning environment and
activities, although not unimportant to the
planning process, simply means to
carry out the activities planned during the three components of the model
described above. This function can
include such activities as managing the instructional effort, guiding learning
activities, collecting evaluative
information, meeting the comfort needs of the program participants, and making
appropriate decisions as situations
warrant them. Various references at
this chapter's conclusion provide information pertaining to this component.
EVALUATION
The importance of
evaluation cannot be overstressed; however, the activities required for
effective evaluation cannot be discussed adequately in the space allotted to
this chapter. The reader should refer to a variety of sources, such as Byrn
(1959), Gottman and Clasen (1972), Tyler (1969) and others included at the
chapter's conclusion, to obtain some understanding of how effective evaluation can be
incorporated into the program planning process.
Evaluation is crucial to successful
program planning if educators are to know
how effective their programs have been. Consequently, evaluation must be
included throughout the planning
process. Evaluation is depicted in the model as taking place near the end of
the process (summative evaluation),
because the analysis of whether or not a program's objectives have been met
usually cannot be determined until the learning activities have been
completed. However, evaluative information often is obtained both before and during the learning
activities (formative evaluation), in
order to measure actual change. Thus,
information gathered from evaluation efforts becomes useful in improving and supplementing ongoing programs, as
well as in planning subsequent programs.
FEEDBACK AND
MODIFICATION
This feature of the planning
process provides a means for utilizing
evaluative findings in strengthening the planning efforts, in making adjustments in methods or techniques used
throughout the process, and in providing
information for future programs regarding educational needs, feedback is the utilization of evaluative information to modify functions within other components, and as a means whereby adjustments and decisions
can he made by the educator even before formal evaluation procedures are
carried out.
For example, the educational
planner might find that certain objectives were not achieved by a majority of
participants.
In a subsequent program, different or supple-menial instructional techniques could
be planned for the objectives. Another use might involve discovering that it was impossible
to design appropriate instructional activities because of unclear objectives.
The planner would return to the objective writing component and re-examine the related
objective. As a final example, evaluative information could be used for needs
assessment; this would occur when the educational planner discovered that
participants would have liked more learning activities on a certain topic. That
topic could become the partial need basis for a follow-up or improved program.
CONTROVERSIES
One of the current
controversial issues in planning and evaluation involves the function of
behavioral objectives. It has been suggested that behavioral objectives
restrict the learning activities in a program to a narrow focus, and that an impersonal,
mechanistic measuring of objective achievement in effect reduces participants to pawns
of the teacher's will (Arnstine, 1964). This argument has some merit, in that
the behavioral
objective approach can be misused or misunderstood.
Another argument
relates to goal-free evaluation of programs. Scriven (1972) has suggested that
program goals arc often vague
or unrealistic; therefore, evaluative confirmation
of their achievement means little. This indicates the necessity of
well-designed objectives based on the real
need to prevent, as Kneller (1972) suggests, a poorly designed program plan.
Another issue involves the question of
when a program should be evaluated, Dressel
(1971) and Tuckman (1972) have suggested two types of evaluation: formative evaluation,
the on-going assessment of a program which aids in its development, and summative evaluation, the assessment of a
program's overall effectiveness.
Controversy also
exists because of the varied approaches to evaluation, only a few of which
will be described here. One approach is the school accreditation model
(National Study, 1960), which involves outside experts making judgments about
effectiveness through onsite visitations. Another approach is the focus on
analyzing the achievement of objectives (Tyler, 1950). The utilization of
collected information for decision- making (Stufflebeam, 1968, 1969) is another
variation described in the literature. Still another version is the judgment
based approach (Stake, 1967), where the worth of a program is determined
through a very complex data collection and analysis procedure.
These controversies
will not be resolved easily or quickly. Each points to the fact that program planning and evaluation is a complex and
time-consuming process. Consequently, to be successful
the educational planner must be systematic in the development of a program. At
the same time, the development of programs for people means that flexibility must be a necessary feature of the
planning process.
REFERENCES
Alexander, F.D. A
Critique of Evaluation. Journal of
Cooperative Extension, 1965, 3,
2005-212.
Ainstine, D. G. The
language and values of programmed instruction: Part 2. The Educational Forum, 1964, 28, 337-346.
Baumel, C. R.,
Hobbs. D. J., and Powers, R. C. The Community Survey. Sociology 15. Ames. Iowa: Iowa State University, Cooperative Extension Service, 1964.
Beal, G.
M., Ross. C., and Powers, R. C. Social Action and Interaction in Program
Planning, Iowa State University Press, 1966.
Beatty,
W, H. , ed. Improving Educational Assessment and an Inventory of Measures of Affective Behavior. Washington, D.C.:
Association for Supervision and Curriculum Development, 1969.
Biddle,
W. W., and Biddle. L. J. The Community Development Process: The Rediscovery of Local
Initiative. New York: Holt, Rinehart, and Winston,
Inc., 1965.
Bloom, B. S., ed. Taxonomy
of Educational Objectives: The Classification of Educational Coals.
Handbook 1. Cognitive Domain. New York: David McKay Co., 1956.
Boone, E., Dolan. R.
J., and Shearon. R. W. Programming in
the Cooperative Extension Service: A Conceptual
Scheme. Miscellaneous Extension Publication 72. Raleigh, North Carolina: North
Carolina State University, Agricultural Extension Service, 1971.
Byrn,
D., ed. Evaluation in Extension Topeka. Kansas: H. M. Ives & Sons. Inc.,
1959.
Caro, F.
G. Approaches to Evaluative Research: A Review. Human Organization, 1969, 25. 87-99.
Carter,
L. F. The Systems Approach to Education: Mystique and Reality. Educational Technology, 1969, 9(4),
22-31.
Churchman.
C. W. The Systems Approach. New York: Dell, 1968.
Cohen.
D. K. Politics and Research: Evaluation of Social Action Programs in Education.
Review of Educational Research, 1970, 40, 213-238.
Cook, D. L. Program Evaluation and Review Technique:
Applications in Education. U.S. Office of Education Cooperative Research Monograph No,
17. OE-12024. Washington. D.C.: U.S. Government Printing Office, 1966.
Delbecq.
A. L.. and Van de Ven. A. G. A Group Process Model for Problem Identification and
Program Planning. Journal of Applied Behavioral Science, 1970, 7, 466-492.
Dobbs, R. C. Self-Perceived Educational Needs of Adults. Adult Education, 1966, 16, 92-100.
Dowling,
W. D. How to Identify Needs of Groups
for Continuing Education. American
Journal of Pharmaceutical Mutation, 1969,
33, 721-728.
Dressel,
P. E. ed. The New Colleges: Toward an Appraisal. Iowa City, Iowa: American College Testing
Program and the American Association for Higher Education, 1971.
Dubin, S. S. Obsolescence or Lifelong Education A Choice
for the
Professional. American Psychologist, 1972, 27, 486-498.
Dutton,
D. Should the Clientele be Involved in Program Planning? Adult Leadership, 1970,
19, 181-182.
Gottman.
J. M., and Clasen. R. E. Evaluation in Education: A Practitioner's Guide.
Itasca.
Illinois: F. K. Peacock, 1972.
Guba, E. G. The Failure of Educational Evaluation. Educational Technology. A Special Report, 1969, 9(5). 29-3K.
Habib.
W. Problems in Determining Training Needs in an Organization. Training
end Development Journal, 1970,
24, (7), 44-48.
Hartley,
H. J. Educational Planning, Programming, Budgeting: A Systems Approach. Englewood Cliffs. N.J.: Prentice-Hall, 1968.
Hemphill.
J. K. The Relationships Between Research and Evaluation Studies. In R.
W. Tyler, ed.. Educational Evaluation: New Roles, New Means. 68th Yearbook of the
National Society for the Study of Education. Part 2. Chicago: University of Chicago
Press, 1969.
Hiemstra. R. P. Continuing
Education for the Aged: A Survey of Needs and Interests of Older People. Adult
Education, 1972, 22, 100-109. (a)
Hiemstra,
R. P. The Educative Community. Lincoln. Nebraska: Professional Educators Publications, Inc.,
1972. (b)
Houle, C. O. The Design of Education. San Francisco:
Jossey-Bass, 1972.
Johnston, J. W. C. The
Educational Pursuits of American Adults. Adult
Education, 1963, 13, 217-221.
Justus,
J. E. PERT. School Management, 1967, 11(12), 24-29.
Kaimann. R. A.
Educators and PERT. Journal of Educational Data Processing, 1966, 3, 43-57.
Kneller, G. F. Goal-Free
Evaluation. Evaluation Comment, 1972, 3(4),
7-8.
Knowles,
M. S. Modern Practice of Adult Education. New York: Association Press,
1970.
Knox.
A. B. Clientele Analysis. Review of Educational Research. 1965, 35, 231-239.
Krathwohl, D. R., Bloom, B.
S., and Masia, B. B. Taxonomy of Educational Objectives: The Classification
of Educational Goals. Handbook II.
Affective Domain. New York: David McKay Company, Inc., 1964.
Leagans,
J. P. A Concept of Needs. Journal of Cooperative Extension, 1964,
2, 89-96.
Lehmann,
H. The Systems Approach to
Education. Audiovisual Instruction, 1968, 13, 144-148.
Lindeman,
E. C. Ten Steps in Community
Action. In E. Harper and A Dunham, eds., Community
Organization in Action. New York: Association Press, 1959.
Long.
R. W. Continuing Education for Physical Therapists in Nebraska: A Survey of Current Practices
and Self-Expressed Needs with Recommendations for Program Development.
Unpublished doctoral dissertation. University of Nebraska, 1972
Mager, R. Preparing Instructional Objectives. Palo
Alto, California: Fearon Publishers, 1962.
McLoughlin, D. Participation of the Adult
Learner in Program Planning. Adult
Education, 1971, 22, 30-35.
McMahon, E. E. Needs of People and Their Communities and the Adult Educator. Washington,
D.C.: Adult Education Association of the
USA, 1970.
Merriman, H. O. Evaluation Theory into Practice. Journal of
Research and Development in Education,
1970, 3(4), 48-58.
Mezirow, J. Toward a Theory of Practice. Adult
Education, 1971, 21, 135-147.
National Study of Secondary School
Evaluation. Evaluation Criteria. 1960 edition. Washington, D.C.:
National Study of Secondary School
Evaluation. 1960.
Popham, W. J., and Baker. E. L. Systematic
Instruction. Englewood Cliffs, N.J.:
Prentice-Hall, Inc., 1970.
Scriven, M. Pros
and Cons about Goal-Free Evaluation. Evaluation Comment, 1972, 3(4),
1-4.
Sheasga. T. A.
Definition of Needs and Wants. Adult Education, 1961, 12, 52-53.
Silvern, L. C. Systems Engineering of Education IV: Systems Analysis
and Synthesis Applied Quantitatively to Create an Instructional System. Los
Angeles: Educational and Training Consultants Co., 1969.
Silvern, L. C. Systems
Engineering of Education V: Quantitative Concepts for Education Systems.
Los Angeles: Educational and Training Consultants Co., 1972.
Sower. C. et al. Community Involvement:
The Webs of Formal and Informal Ties the Make for Action. Glencoe, Illinois: The
Free Press, 1957.
Stake, R. E. The Countenance of
Educational Evaluation. Teachers College
Record, 1967, 58, 523-540.
Stake, R. E. Objectives, Priorities, and
Other Judgment Data. Review of
Educational Research, 1970, 40,
181-212.
Stauber, R. I. PPBS
for Extension? Journal of Cooperative Extension, 1968, 6, 229-235.
Stufflebeam, D. L. Evaluation as
Enlightenment for Decision-Making. In W. H. Beatty, Improving Educational Assessment
and an Inventory of Measures of Affective Behavior. Washington, D.C.:
Association for Supervision and Curriculum Development, 1969.
Stufflebeam, D. L. Toward a Science of
Educational Evaluation. Educational
Technology, 1968, 5(14), 5-17.
Suchman, E. A. Evaluative Research: Principles and
Practices in Public Service and Social Action Programs. New York: Russell
Sage Foundation, 1967.
Theide, W. Evaluation and Adult
Education. In G. E. Jensen, A. A. Liveright, and W. Hallenbeck, eds. Adult Education:
Outlines of an Emerging Field of University Study. Washington,
D.C.: Adult Education Association of the USA, 1964.
Thelen, H. A. Dynamics of Groups at Work. Chicago:
The University of
Chicago Press, 1954.
Thorndike, R. L.,
and Hagen, E. Measurement and Evaluation in Psychology and Education. New York: John Wiley and Sons, 1969.
Tuckman, B. W. Conducting
Educational Research. New York: Harcourt,
Brace, Jovanovich, 1972.
Tyler, R. W. Basic
Principles of Curriculum and Instruction. Chicago: University of Chicago Press, 1950.
Tyler, R. W., ed. Educational Evaluation: New Roles, New Means. 68th Yearbook of the
National Society for the Study of Education,
Part 2. Chicago: University of Chicago Press, 1969.
Tyler, R. W., Gagne, R. M., and Scriven, M. Perspectives
of Curriculum Evaluation.
AERA Monograph
Series on Curriculum Evaluation, No, 1.
Chicago: Rand McNally & Company, 1967.
Verner, C. A. A Conceptual Scheme for the Identification and Classification of
Processes of Adult Education. Chicago: Adult Education Association of the
USA, 1962.
Verner, C. A. Definition of Terms. In G.
E. Jensen, A. A. Liveright, and W.
Hallenbeck, eds. Adult Education: Outlines of an Emerging Field of University Study. Washington, D.C.: Adult Education
Association of the USA, 1964.
Wedemeyer, C. A.
Evaluation of Continuing Education Programs.
American Journal of Pharmaceutical Education, 1969, 33, 793-810.
Worthen, H. R.
Toward a Taxonomy of Evaluation Designs. Educational
Technology, 1968, 8, (15), 3-9.