Reprinted from the Handbook of Contemporary Education by permission of R. R. Bowker Company, New York, NY.
Copyright © 1976 by Xerox Corporation.
PROGRAM PLANNING AND EVALUATION
by Roger Hiemstra, Professor
Department of Adult and Continuing Education
University of Nebraska
The term Program Planning and Evaluation does not have a commonly agreed-upon meaning. The three words--program, planning, and evaluation--are commonly thought of together as part of an educational process for organizing and administering programs; however, an understanding of this process begins with separate definitions:
Program -a single activity, or sequential series of activities, designed to achieve one or more educational objectives determined as necessary or desired in promoting change in people.
Planning-the procedures or steps utilized in designing a program.
Evaluation-an assessment of a program's effectiveness, usually in terms of its achievement of objectives.
The implementation of a program is considered as part of the total process.
Planning and evaluation are widely discussed topics in education, with a variety of procedures being utilized to plan and implement educational activities. Added to this diversity in procedures are large numbers of commercialized approaches to planning. Literature on planning and evaluation, gaming and simulation devices on problem solving, courses and workshops to improve planning skills, and related research reports are readily available. Much of this information and many of these procedures are referenced at the end of this chapter.
This abundance of information is one indication that the design of effective education is a complex process. In addition, the process is dynamic in nature, in that developed plans are frequently subject to change based on new and evolving information. Thus, there is no simple recipe tor successful program planning. Perhaps it is this complexity and dynamic quality that have made a simple definition of program planning and evaluation so elusive.
However, there are procedures and steps commonly employed in the systematic development of educational activities. For example, the design of instruction, the development of an annual conference program, the planning of an in-service training session for teachers, or the development of a workshop for the continuing education of a group of professionals, will be based on a process made up of differentiated but interrelated steps. The focus of this chapter will be to describe this process and its various components.
Finally, any discussion of planning and evaluation will normally be based around the assumption that some type of change-be it change in knowledge, skill, or attitude-is an expected educational program result. However, the focus of this chapter's discussion is the planning and evaluating of primarily short-term programs based on some determined need for change. Consequently, curriculum planning which tends to focus on anticipated or long-range needs does not relate directly to the process to be described. Parts of the process certainly are related to, and can be used for, curriculum planning, but the sequencing of, and emphasis placed on, the various components usually are different.
Literature on program planning and evaluation is as diverse as are the approaches or models utilized for planning purposes. Some literature centers on the level at which one designs programs. Verner (1964) has suggested that at least two levels should be considered: the first level deals with fulfilling assigned social roles of the institution organizing the program: the second deals with activities, learning tasks, and needs at (lie client or program participant level. Knowles (1970) and Theide (1964) support this contention, but add that community or societal goals also need to be considered in the planning process.
Another portion of the literature relates to establishing program objectives. Bloom (1956), Krathwohl, Bloom, and Masia (1964), Mager (1962), Popham and Baker (1970), and Tyler (1950) have been some of the pioneers in encouraging the utilization of "instructional" or "behavioral" objectives in educational programs. Although variations exist regarding what encompasses an educational objective, most educators agree that desired . . learner behaviors should be specified. Furthermore, the task of designing appropriate learning activities is facilitated by precise behavior specifications.
Somewhat related is the discussion by Delbecq and Van de Ven (1970), Dutton (1 970), and Knowles (1970) suggesting that program participants should be involved in the planning process, which includes the specification of objectives. This assumes that involving the participant in the planning process will build personal interest in resulting educational activities, and promote learning more central to actual need. This assumption is supported by McLoughlin (1971); he found that experimental subjects who had been involved in objective-setting and program-planning had more positive attitudes toward the educational experience, and achieved us well as control subjects.
The conceptualization and development of planning models is another topic found frequently in the literature. For example. Beal, Ross, and Towers (1966) approached planning from a sociological view, and designed a community level model that includes such functions as determining groups to be involved in the planning process, obtaining legitimation for the developed plan, and seeking community commitment. Variations on this community-oriented approach can be found in Biddle and Biddle (1965), Lindeman (1959), Sower (l957), and Thelen (1954). More recently, Boone, Dolon, and Shearon (1971) used these various community-level models to develop a conceptual scheme for planning that focuses on the decision-making process, Houle (1972) has developed some thoughts toward using a decision-making process and planning system in redesigning current educational programs.
Another approach to model development for educational planning has stemmed from the systems analysis procedures used by business and industry. Several authors (Carter, 1969; Churchman, 1968; Lehmann, 1968; Silvern, 1969, 1972) have developed educationally-related systems analysis models. The critical path analysis approach (Cook, 1966; Justus, 1967; Kaimann, 1966) and Program-Planning-Budgeting Systems (Hartley, 1968; Stauber, 1968) are two additional systematic planning approaches that have received considerable attention.
The assessment of needs prior to designing educational programs has received attention, too. Individual needs and how to assess or define them (Dobbs, 1966; Johnston, 1963; Knowles, 1970; Knox, 1965; Leagans, 1964; Sheasga, 1961), needs of groups or certain population segments (Dowling, 1969; Dublin, 1972; Hiemstra, 1972a; Long, 1972), and needs at a community or institutional level (Baumel, Hobbs, and Powers, 1964; Habib, 1970; Hiemstra, 1972b; McMahon, 1970) are the more common categories around which this portion of the literature has been organized.
The literature on evaluation is immense. It ranges from textbooks or general informational books on evaluation (Beatty, 1969; Bym, 1959; Gottman and Clasen, 1972; Suchman, 1967; Thorndike and Hagen, 1969; Tyler, 1969) to specific evaluation materials or topics (Stake, 1970; Wedemeyer, 1969), to critiques or discussions of evaluation problems (Alexander, 1965; Cohen, 1970; Cuba, 1969; Scriven, 1972; Tyler, 1967). The information on educational evaluation is so broad am! extensive that the above citations provide only a starting-point.
A final body of literature to be noted here centers on a growing awareness of research needs in program planning and evaluation. Caro (1969), Hemphill (1969), Merriman (1970), Mezirow (1971), Verner (1962), and Worthen (1968) are but a few of the authors attempting to apply research techniques and findings systematically in building a comprehensive body of knowledge regarding planning. These efforts should lead to a better understanding of the total process.
Need can be defined as a discrepancy between what is known or can be determined about a behavior and what should be (Gottman & Clasen, 1972; Knowles, 1970). The educator's role is to create an environment in which learning activities are utilized to reduce or remove the discrepancy.
Thus, needs assessment becomes a necessary and crucial aspect of the planning and evaluating process. A determined need becomes the basis for one or more objectives. Without needs assessment the planned educational activities could promote behavioral changes with limited or no relationship to existing discrepancies.
Various tools and techniques are available at the individual, group, institutional, and community levels. These include observational techniques, projective techniques, questionnaire approaches, community surveys, and job performance analysis procedures. Space limitations do not permit their description, but McMahon (1970) provides an extensive bibliography regarding needs assessment. In addition, references are included at the end of the chapter relating to each component in the planning and evaluation process.
DEVELOPMENT OF OBJECTIVES
Most educational programs revolve around assessed change requirements of three types: (1) attitude modification, (2) behavioral change, and (3) skill or content mastery. An objective is the description of an educational intent !o bring about one or more of these changes. Identification and specification of desired change are prerequisites to the design of appropriate learning activities.
Wording objectives so that they are precise enough to be useful is a frustrating and often poorly done part of educational program planning. However, designing those appropriate learning activities, and measuring whether or not they have met some determined need, are dependent upon clearly stated objectives. Mager (1962) suggests that a useful objective must identify the expected change, define the important conditions under which the change is to occur, and define the related criteria for measuring the change achievement. Some of this chapter's references will provide direction on writing and using objectives.
PLANNING THE ACTIVITIES
Planning appropriate learning activities entails the following: the identification of learning tasks necessary for achieving a desired performance or change, the sequencing of these tasks, and the determination of instructional techniques appropriate for both the tasks and the program participant.
The program planning and evaluation process can, and often does, begin at this component in the model. This occurs, for example, when a program is built around a desired speaker or topic. However, including the earlier described needs assessment and objective selling components can strengthen the resulting programs. Thus, a program can be made more relevant if actual needs are being met; the measurement of a program's success can be facilitated if precise objectives have been identified; and the actual design of learning activities can determine if objectives are realistic or achievable. (Various tools, procedures, and techniques are available to facilitate this component's activities. Note the cited sources at the end of the chapter.)
IMPLEMENTATION OF ACTIVITIES
Operating the learning environment and activities, although not unimportant to the planning process, simply means to carry out the activities planned during the three components of the model described above. This function can include such activities as managing the instructional effort, guiding learning activities, collecting evaluative information, meeting the comfort needs of the program participants, and making appropriate decisions as situations warrant them. Various references at this chapter's conclusion provide information pertaining to this component.
The importance of evaluation cannot be overstressed; however, the activities required for effective evaluation cannot be discussed adequately in the space allotted to this chapter. The reader should refer to a variety of sources, such as Byrn (1959), Gottman and Clasen (1972), Tyler (1969) and others included at the chapter's conclusion, to obtain some understanding of how effective evaluation can be incorporated into the program planning process.
Evaluation is crucial to successful program planning if educators are to know how effective their programs have been. Consequently, evaluation must be included throughout the planning process. Evaluation is depicted in the model as taking place near the end of the process (summative evaluation), because the analysis of whether or not a program's objectives have been met usually cannot be determined until the learning activities have been completed. However, evaluative information often is obtained both before and during the learning activities (formative evaluation), in order to measure actual change. Thus, information gathered from evaluation efforts becomes useful in improving and supplementing ongoing programs, as well as in planning subsequent programs.
FEEDBACK AND MODIFICATION
This feature of the planning process provides a means for utilizing evaluative findings in strengthening the planning efforts, in making adjustments in methods or techniques used throughout the process, and in providing information for future programs regarding educational needs, feedback is the utilization of evaluative information to modify functions within other components, and as a means whereby adjustments and decisions can he made by the educator even before formal evaluation procedures are carried out.
For example, the educational planner might find that certain objectives were not achieved by a majority of participants. In a subsequent program, different or supple-menial instructional techniques could be planned for the objectives. Another use might involve discovering that it was impossible to design appropriate instructional activities because of unclear objectives. The planner would return to the objective writing component and re-examine the related objective. As a final example, evaluative information could be used for needs assessment; this would occur when the educational planner discovered that participants would have liked more learning activities on a certain topic. That topic could become the partial need basis for a follow-up or improved program.
One of the current controversial issues in planning and evaluation involves the function of behavioral objectives. It has been suggested that behavioral objectives restrict the learning activities in a program to a narrow focus, and that an impersonal, mechanistic measuring of objective achievement in effect reduces participants to pawns of the teacher's will (Arnstine, 1964). This argument has some merit, in that the behavioral objective approach can be misused or misunderstood.
Another argument relates to goal-free evaluation of programs. Scriven (1972) has suggested that program goals arc often vague or unrealistic; therefore, evaluative confirmation of their achievement means little. This indicates the necessity of well-designed objectives based on the real need to prevent, as Kneller (1972) suggests, a poorly designed program plan.
Another issue involves the question of when a program should be evaluated, Dressel (1971) and Tuckman (1972) have suggested two types of evaluation: formative evaluation, the on-going assessment of a program which aids in its development, and summative evaluation, the assessment of a program's overall effectiveness.
Controversy also exists because of the varied approaches to evaluation, only a few of which will be described here. One approach is the school accreditation model (National Study, 1960), which involves outside experts making judgments about effectiveness through onsite visitations. Another approach is the focus on analyzing the achievement of objectives (Tyler, 1950). The utilization of collected information for decision- making (Stufflebeam, 1968, 1969) is another variation described in the literature. Still another version is the judgment based approach (Stake, 1967), where the worth of a program is determined through a very complex data collection and analysis procedure.
These controversies will not be resolved easily or quickly. Each points to the fact that program planning and evaluation is a complex and time-consuming process. Consequently, to be successful the educational planner must be systematic in the development of a program. At the same time, the development of programs for people means that flexibility must be a necessary feature of the planning process.
Alexander, F.D. A Critique of Evaluation. Journal of Cooperative Extension, 1965, 3, 2005-212.
Ainstine, D. G. The language and values of programmed instruction: Part 2. The Educational Forum, 1964, 28, 337-346.
Baumel, C. R., Hobbs. D. J., and Powers, R. C. The Community Survey. Sociology 15. Ames. Iowa: Iowa State University, Cooperative Extension Service, 1964.
Beal, G. M., Ross. C., and Powers, R. C. Social Action and Interaction in Program Planning, Iowa State University Press, 1966.
Beatty, W, H. , ed. Improving Educational Assessment and an Inventory of Measures of Affective Behavior. Washington, D.C.: Association for Supervision and Curriculum Development, 1969.
Biddle, W. W., and Biddle. L. J. The Community Development Process: The Rediscovery of Local Initiative. New York: Holt, Rinehart, and Winston, Inc., 1965.
Bloom, B. S., ed. Taxonomy of Educational Objectives: The Classification of Educational Coals. Handbook 1. Cognitive Domain. New York: David McKay Co., 1956.
Boone, E., Dolan. R. J., and Shearon. R. W. Programming in the Cooperative Extension Service: A Conceptual Scheme. Miscellaneous Extension Publication 72. Raleigh, North Carolina: North Carolina State University, Agricultural Extension Service, 1971.
Byrn, D., ed. Evaluation in Extension Topeka. Kansas: H. M. Ives & Sons. Inc., 1959.
Caro, F. G. Approaches to Evaluative Research: A Review. Human Organization, 1969, 25. 87-99.
Carter, L. F. The Systems Approach to Education: Mystique and Reality. Educational Technology, 1969, 9(4), 22-31.
Churchman. C. W. The Systems Approach. New York: Dell, 1968.
Cohen. D. K. Politics and Research: Evaluation of Social Action Programs in Education. Review of Educational Research, 1970, 40, 213-238.
Cook, D. L. Program Evaluation and Review Technique: Applications in Education. U.S. Office of Education Cooperative Research Monograph No, 17. OE-12024. Washington. D.C.: U.S. Government Printing Office, 1966.
Delbecq. A. L.. and Van de Ven. A. G. A Group Process Model for Problem Identification and Program Planning. Journal of Applied Behavioral Science, 1970, 7, 466-492.
Dobbs, R. C. Self-Perceived Educational Needs of Adults. Adult Education, 1966, 16, 92-100.
Dowling, W. D. How to Identify Needs of Groups for Continuing Education. American Journal of Pharmaceutical Mutation, 1969, 33, 721-728.
Dressel, P. E. ed. The New Colleges: Toward an Appraisal. Iowa City, Iowa: American College Testing Program and the American Association for Higher Education, 1971.
Dubin, S. S. Obsolescence or Lifelong Education A Choice for the Professional. American Psychologist, 1972, 27, 486-498.
Dutton, D. Should the Clientele be Involved in Program Planning? Adult Leadership, 1970, 19, 181-182.
Gottman. J. M., and Clasen. R. E. Evaluation in Education: A Practitioner's Guide. Itasca. Illinois: F. K. Peacock, 1972.
Guba, E. G. The Failure of Educational Evaluation. Educational Technology. A Special Report, 1969, 9(5). 29-3K.
Habib. W. Problems in Determining Training Needs in an Organization. Training end Development Journal, 1970, 24, (7), 44-48.
Hartley, H. J. Educational Planning, Programming, Budgeting: A Systems Approach. Englewood Cliffs. N.J.: Prentice-Hall, 1968.
Hemphill. J. K. The Relationships Between Research and Evaluation Studies. In R. W. Tyler, ed.. Educational Evaluation: New Roles, New Means. 68th Yearbook of the National Society for the Study of Education. Part 2. Chicago: University of Chicago Press, 1969.
Hiemstra. R. P. Continuing Education for the Aged: A Survey of Needs and Interests of Older People. Adult Education, 1972, 22, 100-109. (a)
Hiemstra, R. P. The Educative Community. Lincoln. Nebraska: Professional Educators Publications, Inc., 1972. (b)
Houle, C. O. The Design of Education. San Francisco: Jossey-Bass, 1972.
Johnston, J. W. C. The Educational Pursuits of American Adults. Adult Education, 1963, 13, 217-221.
Justus, J. E. PERT. School Management, 1967, 11(12), 24-29.
Kaimann. R. A. Educators and PERT. Journal of Educational Data Processing, 1966, 3, 43-57.
Kneller, G. F. Goal-Free Evaluation. Evaluation Comment, 1972, 3(4), 7-8.
Knowles, M. S. Modern Practice of Adult Education. New York: Association Press, 1970.
Knox. A. B. Clientele Analysis. Review of Educational Research. 1965, 35, 231-239.
Krathwohl, D. R., Bloom, B. S., and Masia, B. B. Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook II. Affective Domain. New York: David McKay Company, Inc., 1964.
Leagans, J. P. A Concept of Needs. Journal of Cooperative Extension, 1964, 2, 89-96.
Lehmann, H. The Systems Approach to Education. Audiovisual Instruction, 1968, 13, 144-148.
Lindeman, E. C. Ten Steps in Community Action. In E. Harper and A Dunham, eds., Community Organization in Action. New York: Association Press, 1959.
Long. R. W. Continuing Education for Physical Therapists in Nebraska: A Survey of Current Practices and Self-Expressed Needs with Recommendations for Program Development. Unpublished doctoral dissertation. University of Nebraska, 1972
Mager, R. Preparing Instructional Objectives. Palo Alto, California: Fearon Publishers, 1962.
McLoughlin, D. Participation of the Adult Learner in Program Planning. Adult Education, 1971, 22, 30-35.
McMahon, E. E. Needs of People and Their Communities and the Adult Educator. Washington, D.C.: Adult Education Association of the USA, 1970.
Merriman, H. O. Evaluation Theory into Practice. Journal of Research and Development in Education, 1970, 3(4), 48-58.
Mezirow, J. Toward a Theory of Practice. Adult Education, 1971, 21, 135-147.
National Study of Secondary School Evaluation. Evaluation Criteria. 1960 edition. Washington, D.C.: National Study of Secondary School Evaluation. 1960.
Popham, W. J., and Baker. E. L. Systematic Instruction. Englewood Cliffs, N.J.: Prentice-Hall, Inc., 1970.
Scriven, M. Pros and Cons about Goal-Free Evaluation. Evaluation Comment, 1972, 3(4), 1-4.
Sheasga. T. A. Definition of Needs and Wants. Adult Education, 1961, 12, 52-53.
Silvern, L. C. Systems Engineering of Education IV: Systems Analysis and Synthesis Applied Quantitatively to Create an Instructional System. Los Angeles: Educational and Training Consultants Co., 1969.
Silvern, L. C. Systems Engineering of Education V: Quantitative Concepts for Education Systems. Los Angeles: Educational and Training Consultants Co., 1972.
Sower. C. et al. Community Involvement: The Webs of Formal and Informal Ties the Make for Action. Glencoe, Illinois: The Free Press, 1957.
Stake, R. E. The Countenance of Educational Evaluation. Teachers College Record, 1967, 58, 523-540.
Stake, R. E. Objectives, Priorities, and Other Judgment Data. Review of Educational Research, 1970, 40, 181-212.
Stauber, R. I. PPBS for Extension? Journal of Cooperative Extension, 1968, 6, 229-235.
Stufflebeam, D. L. Evaluation as Enlightenment for Decision-Making. In W. H. Beatty, Improving Educational Assessment and an Inventory of Measures of Affective Behavior. Washington, D.C.: Association for Supervision and Curriculum Development, 1969.
Stufflebeam, D. L. Toward a Science of Educational Evaluation. Educational Technology, 1968, 5(14), 5-17.
Suchman, E. A. Evaluative Research: Principles and Practices in Public Service and Social Action Programs. New York: Russell Sage Foundation, 1967.
Theide, W. Evaluation and Adult Education. In G. E. Jensen, A. A. Liveright, and W. Hallenbeck, eds. Adult Education: Outlines of an Emerging Field of University Study. Washington, D.C.: Adult Education Association of the USA, 1964.
Thelen, H. A. Dynamics of Groups at Work. Chicago: The University of Chicago Press, 1954.
Thorndike, R. L., and Hagen, E. Measurement and Evaluation in Psychology and Education. New York: John Wiley and Sons, 1969.
Tuckman, B. W. Conducting Educational Research. New York: Harcourt, Brace, Jovanovich, 1972.
Tyler, R. W. Basic Principles of Curriculum and Instruction. Chicago: University of Chicago Press, 1950.
Tyler, R. W., ed. Educational Evaluation: New Roles, New Means. 68th Yearbook of the National Society for the Study of Education, Part 2. Chicago: University of Chicago Press, 1969.
Tyler, R. W., Gagne, R. M., and Scriven, M. Perspectives of Curriculum Evaluation. AERA Monograph Series on Curriculum Evaluation, No, 1. Chicago: Rand McNally & Company, 1967.
Verner, C. A. A Conceptual Scheme for the Identification and Classification of Processes of Adult Education. Chicago: Adult Education Association of the USA, 1962.
Verner, C. A. Definition of Terms. In G. E. Jensen, A. A. Liveright, and W. Hallenbeck, eds. Adult Education: Outlines of an Emerging Field of University Study. Washington, D.C.: Adult Education Association of the USA, 1964.
Wedemeyer, C. A. Evaluation of Continuing Education Programs. American Journal of Pharmaceutical Education, 1969, 33, 793-810.
Worthen, H. R. Toward a Taxonomy of Evaluation Designs. Educational Technology, 1968, 8, (15), 3-9.