Reprinted from the Handbook of Contemporary Education by permission of R. R. Bowker Company, New York, NY.

Copyright © 1976 by Xerox Corporation.

PROGRAM PLANNING AND EVALUATION

by Roger Hiemstra, Professor

Department of Adult and Continuing Education

Teachers College

University of Nebraska

Lincoln, Nebraska

The term Program Planning and Evaluation does not have a commonly agreed-upon meaning. The three words--program, planning, and evaluation--are commonly thought of together as part of an educational process for organizing and administering programs; however, an understanding of this process begins with separate defini­tions:

Program -a single activity, or sequential series of activities, designed to achieve one or more educa­tional objectives determined as necessary or desired in promoting change in people.

Planning-the procedures or steps utilized in de­signing a program.

Evaluation-an assessment of a program's effective­ness, usually in terms of its achievement of objectives.

The implementation of a program is considered as part of the total process.

Planning and evaluation are widely discussed topics in education, with a variety of procedures being utilized to plan and implement educational activities. Added to this diversity in procedures are large numbers of com­mercialized approaches to planning. Literature on plan­ning and evaluation, gaming and simulation devices on problem solving, courses and workshops to improve plan­ning skills, and related research reports are readily avail­able. Much of this information and many of these pro­cedures are referenced at the end of this chapter.

This abundance of information is one indication that the design of effective education is a complex process. In addition, the process is dynamic in nature, in that developed plans are frequently subject to change based on new and evolving information. Thus, there is no simple recipe tor successful program planning. Perhaps it is this complexity and dynamic quality that have made a simple definition of program planning and evalu­ation so elusive.

However, there are procedures and steps commonly employed in the systematic development of educational activities. For example, the design of instruction, the development of an annual conference program, the plan­ning of an in-service training session for teachers, or the development of a workshop for the continuing education of a group of professionals, will be based on a process made up of differentiated but interrelated steps. The focus of this chapter will be to describe this process and its various components.

Finally, any discussion of planning and evaluation will normally be based around the assumption that some type of change-be it change in knowledge, skill, or attitude-is an expected educational program result. However, the focus of this chapter's discussion is the planning and evaluating of primarily short-term pro­grams based on some determined need for change. Con­sequently, curriculum planning which tends to focus on anticipated or long-range needs does not relate directly to the process to be described. Parts of the process cer­tainly are related to, and can be used for, curriculum planning, but the sequencing of, and emphasis placed on, the various components usually are different.

RELATED LITERATURE

Literature on program planning and evaluation is as diverse as are the approaches or models utilized for planning purposes. Some literature centers on the level at which one designs programs. Verner (1964) has sug­gested that at least two levels should be considered: the first level deals with fulfilling assigned social roles of the institution organizing the program: the second deals with activities, learning tasks, and needs at (lie client or program participant level. Knowles (1970) and Theide (1964) support this contention, but add that community or societal goals also need to be considered in the plan­ning process.

Another portion of the literature relates to establish­ing program objectives. Bloom (1956), Krathwohl, Bloom, and Masia (1964), Mager (1962), Popham and Baker (1970), and Tyler (1950) have been some of the pioneers in encouraging the utilization of "instructional" or "behavioral" objectives in educational programs. Al­though variations exist regarding what encompasses an educational objective, most educators agree that desired     . . learner behaviors should be specified. Furthermore, the task of designing appropriate learning activities is facili­tated by precise behavior specifications.

Somewhat related is the discussion by Delbecq and Van de Ven (1970), Dutton (1 970), and Knowles (1970) suggesting that program participants should be involved in the planning process, which includes the specification of objectives. This assumes that involving the participant in the planning process will build personal interest in re­sulting educational activities, and promote learning more central to actual need. This assumption is supported by McLoughlin (1971); he found that experimental sub­jects who had been involved in objective-setting and program-planning had more positive attitudes toward the educational experience, and achieved us well as con­trol subjects.

The conceptualization and development of planning models is another topic found frequently in the literature. For example. Beal, Ross, and Towers (1966) ap­proached planning from a sociological view, and designed a community level model that includes such functions as determining groups to be involved in the planning process, obtaining legitimation for the developed plan, and seeking community commitment. Variations on this community-oriented approach can be found in Biddle and Biddle (1965), Lindeman (1959), Sower (l957), and Thelen (1954). More recently, Boone, Dolon, and Shearon (1971) used these various community-level models to develop a conceptual scheme for planning that focuses on the decision-making process, Houle (1972) has developed some thoughts toward using a decision-making process and planning system in rede­signing current educational programs.

Another approach to model development for educational planning has stemmed from the systems analysis procedures used by business and industry. Several au­thors (Carter, 1969; Churchman, 1968; Lehmann, 1968; Silvern, 1969, 1972) have developed educationally-related systems analysis models. The critical path analy­sis approach (Cook, 1966; Justus, 1967; Kaimann, 1966) and Program-Planning-Budgeting Systems (Hartley, 1968; Stauber, 1968) are two additional systematic planning approaches that have received considerable attention.

The assessment of needs prior to designing educa­tional programs has received attention, too. Individual needs and how to assess or define them (Dobbs, 1966; Johnston, 1963; Knowles, 1970; Knox, 1965; Leagans, 1964; Sheasga, 1961), needs of groups or certain popu­lation segments (Dowling, 1969; Dublin, 1972; Hiemstra, 1972a; Long, 1972), and needs at a community or institutional level (Baumel, Hobbs, and Powers, 1964; Habib, 1970; Hiemstra, 1972b; McMahon, 1970) are the more common categories around which this por­tion of the literature has been organized.

The literature on evaluation is immense. It ranges from textbooks or general informational books on evalu­ation (Beatty, 1969; Bym, 1959; Gottman and Clasen, 1972; Suchman, 1967; Thorndike and Hagen, 1969; Tyler, 1969) to specific evaluation materials or topics (Stake, 1970; Wedemeyer, 1969), to critiques or discus­sions of evaluation problems (Alexander, 1965; Cohen, 1970; Cuba, 1969; Scriven, 1972; Tyler, 1967). The information on educational evaluation is so broad am! extensive that the above citations provide only a starting-point.

A final body of literature to be noted here centers on a growing awareness of research needs in program planning and evaluation. Caro (1969), Hemphill (1969), Merriman (1970), Mezirow (1971), Verner (1962), and Worthen (1968) are but a few of the authors attempting to apply research techniques and findings systematically in building a comprehensive body of knowledge regarding planning. These efforts should lead to a better understanding of the total process.

 


NEEDS ASSESSMENT

Need can be defined as a discrepancy between what is known or can be determined about a behavior and what should be (Gottman & Clasen, 1972; Knowles, 1970). The educator's role is to create an environment in which learning activities are utilized to reduce or re­move the discrepancy.

Thus, needs assessment becomes a necessary and crucial aspect of the planning and evaluating process. A determined need becomes the basis for one or more objectives. Without needs assessment the planned educa­tional activities could promote behavioral changes with limited or no relationship to existing discrepancies.

Various tools and techniques are available at the in­dividual, group, institutional, and community levels. These include observational techniques, projective techniques, questionnaire approaches, community surveys, and job performance analysis procedures. Space limita­tions do not permit their description, but McMahon (1970) provides an extensive bibliography regarding needs assessment. In addition, references are included at the end of the chapter relating to each component in the planning and evaluation process.

DEVELOPMENT OF OBJECTIVES

Most educational programs revolve around assessed change requirements of three types: (1) attitude modi­fication, (2) behavioral change, and (3) skill or content mastery. An objective is the description of an educational intent !o bring about one or more of these changes. Identification and specification of desired change are prerequisites to the design of appropriate learning ac­tivities.

Wording objectives so that they are precise enough to be useful is a frustrating and often poorly done part of educational program planning. However, designing those appropriate learning activities, and measuring whether or not they have met some determined need, are dependent upon clearly stated objectives. Mager (1962) suggests that a useful objective must identify the expected change, define the important conditions under which the change is to occur, and define the related criteria for measuring the change achievement. Some of this chapter's references will provide direction on writing and using objectives.

PLANNING THE ACTIVITIES

Planning appropriate learning activities entails the following: the identification of learning tasks necessary for achieving a desired performance or change, the sequencing of these tasks, and the determination of in­structional techniques appropriate for both the tasks and the program participant.

The program planning and evaluation process can, and often does, begin at this component in the model. This occurs, for example, when a program is built around a desired speaker or topic. However, including the earlier described needs assessment and objective selling com­ponents can strengthen the resulting programs. Thus, a program can be made more relevant if actual needs are being met; the measurement of a program's success can be facilitated if precise objectives have been identified; and the actual design of learning activities can determine if objectives are realistic or achievable. (Various tools, procedures, and techniques are available to facilitate this component's activities. Note the cited sources at the end of the chapter.)

IMPLEMENTATION OF ACTIVITIES

Operating the learning environment and activities, although not unimportant to the planning process, simply means to carry out the activities planned during the three components of the model described above. This function can include such activities as managing the in­structional effort, guiding learning activities, collecting evaluative information, meeting the comfort needs of the program participants, and making appropriate de­cisions as situations warrant them. Various references at this chapter's conclusion provide information per­taining to this component.

EVALUATION

The importance of evaluation cannot be overstressed; however, the activities required for effective evaluation cannot be discussed adequately in the space allotted to this chapter. The reader should refer to a variety of sources, such as Byrn (1959), Gottman and Clasen (1972), Tyler (1969) and others included at the chapter's con­clusion, to obtain some understanding of how effective evaluation can be incorporated into the program plan­ning process.

Evaluation is crucial to successful program planning if educators are to know how effective their programs have been. Consequently, evaluation must be included throughout the planning process. Evaluation is depicted in the model as taking place near the end of the process (summative evaluation), because the analysis of whether or not a program's objectives have been met usually can­not be determined until the learning activities have been completed. However, evaluative information often is obtained both before and during the learning activities (formative evaluation), in order to measure actual change. Thus, information gathered from evaluation efforts be­comes useful in improving and supplementing ongoing programs, as well as in planning subsequent programs.

FEEDBACK AND MODIFICATION

This feature of the planning process provides a means for utilizing evaluative findings in strengthening the planning efforts, in making adjustments in methods or techniques used throughout the process, and in provid­ing information for future programs regarding educa­tional needs, feedback is the utilization of evaluative information to modify functions within other com­ponents, and as a means whereby adjustments and decisions can he made by the educator even before formal evaluation procedures are carried out.

For example, the educational planner might find that certain objectives were not achieved by a majority of par­ticipants. In a subsequent program, different or supple-menial instructional techniques could be planned for the objectives. Another use might involve discovering that it was impossible to design appropriate instructional ac­tivities because of unclear objectives. The planner would return to the objective writing component and re-examine the related objective. As a final example, eval­uative information could be used for needs assess­ment; this would occur when the educational planner discovered that participants would have liked more learning activities on a certain topic. That topic could become the partial need basis for a follow-up or im­proved program.

CONTROVERSIES

One of the current controversial issues in planning and evaluation involves the function of behavioral objectives. It has been suggested that behavioral ob­jectives restrict the learning activities in a program to a narrow focus, and that an impersonal, mechanistic measuring of objective achievement in effect reduces participants to pawns of the teacher's will (Arnstine, 1964). This argument has some merit, in that the be­havioral objective approach can be misused or mis­understood.

Another argument relates to goal-free evaluation of programs. Scriven (1972) has suggested that program goals arc often vague or unrealistic; therefore, evaluative confirmation of their achievement means little. This in­dicates the necessity of well-designed objectives based on the real need to prevent, as Kneller (1972) suggests, a poorly designed program plan.

Another issue involves the question of when a program should be evaluated, Dressel (1971) and Tuckman (1972) have suggested two types of evaluation: formative evalu­ation, the on-going assessment of a program which aids in its development, and summative evaluation, the assessment of a program's overall effectiveness.

Controversy also exists because of the varied ap­proaches to evaluation, only a few of which will be de­scribed here. One approach is the school accreditation model (National Study, 1960), which involves outside experts making judgments about effectiveness through onsite visitations. Another approach is the focus on analyzing the achievement of objectives (Tyler, 1950). The utilization of collected information for decision- making (Stufflebeam, 1968, 1969) is another variation described in the literature. Still another version is the judgment based approach (Stake, 1967), where the worth of a program is determined through a very complex data collection and analysis procedure.

These controversies will not be resolved easily or quickly. Each points to the fact that program planning and evaluation is a complex and time-consuming process. Consequently, to be successful the educational planner must be systematic in the development of a program. At the same time, the development of programs for people means that flexibility must be a necessary feature of the planning process.

 

REFERENCES

Alexander, F.D. A Critique of Evaluation. Journal of Cooperative Extension, 1965, 3, 2005-212.

Ainstine, D. G. The language and values of programmed instruction: Part 2. The Educational Forum, 1964, 28, 337-346.

Baumel, C. R., Hobbs. D. J., and Powers, R. C. The Community Survey. Sociology 15. Ames. Iowa: Iowa State University, Cooperative Extension Service, 1964.

Beal, G. M., Ross. C., and Powers, R. C. Social Action and Inter­action in Program Planning, Iowa State University Press, 1966.

Beatty, W, H. , ed. Improving Educational Assessment and an Inventory of Measures of Affective Behavior. Washington, D.C.: Association for Supervision and Curriculum Develop­ment, 1969.

Biddle, W. W., and Biddle. L. J. The Community Development Process: The Rediscovery of Local Initiative.  New York: Holt, Rinehart, and Winston, Inc., 1965.

Bloom, B. S., ed. Taxonomy of Educational Objectives: The Classification of Educational Coals. Handbook 1. Cognitive Domain. New York: David McKay Co., 1956.

Boone, E., Dolan. R. J., and Shearon. R. W. Programming in the Cooperative Extension Service: A Conceptual Scheme. Miscellaneous Extension Publication 72. Raleigh, North Carolina: North Carolina State University, Agricultural Extension Service, 1971.

Byrn, D., ed. Evaluation in Extension Topeka. Kansas: H. M. Ives & Sons. Inc., 1959.

Caro, F. G. Approaches to Evaluative Research: A Review. Human Organization, 1969, 25. 87-99.

Carter, L. F. The Systems Approach to Education: Mystique and Reality. Educational Technology, 1969, 9(4), 22-31.

Churchman. C. W. The Systems Approach. New York: Dell, 1968.

Cohen. D. K. Politics and Research: Evaluation of Social Action Programs in Education. Review of Educational Re­search, 1970, 40, 213-238.

Cook, D. L. Program Evaluation and Review Technique: Appli­cations in Education. U.S. Office of Education Cooperative Research Monograph No, 17. OE-12024. Washington. D.C.: U.S. Government Printing Office, 1966.

Delbecq. A. L.. and Van de Ven. A. G. A Group Process Model for Problem Identification and Program Planning. Journal of Applied Behavioral Science, 1970, 7, 466-492.

Dobbs, R. C. Self-Perceived Educational Needs of Adults. Adult Education, 1966, 16, 92-100.

Dowling,  W. D. How to Identify Needs of Groups for Continuing Education. American Journal of Pharmaceutical Mutation, 1969, 33, 721-728.

Dressel, P. E. ed. The New Colleges: Toward an Appraisal.  Iowa City, Iowa: American College Testing Program and the American Association for Higher Education, 1971.

Dubin, S. S. Obsolescence or Lifelong Education A Choice for the Professional. American Psychologist, 1972, 27, 486-498.

Dutton, D. Should the Clientele be Involved in Program Planning? Adult Leadership, 1970, 19, 181-182.

Gottman. J. M., and Clasen. R. E. Evaluation in Education: A Practitioner's Guide. Itasca. Illinois: F. K. Peacock, 1972.

Guba, E. G. The Failure of Educational Evaluation. Educational Technology. A Special Report, 1969, 9(5). 29-3K.

Habib. W. Problems in Determining Training Needs in an Organi­zation. Training end Development Journal, 1970, 24, (7), 44-48.

Hartley, H. J. Educational Planning, Programming, Budgeting: A Systems Approach. Englewood Cliffs. N.J.: Prentice-Hall, 1968.

Hemphill. J. K. The Relationships Between Research and Evalua­tion Studies. In R. W. Tyler, ed.. Educational Evaluation: New Roles, New Means. 68th Yearbook of the National So­ciety for the Study of Education. Part 2. Chicago: Univer­sity of Chicago Press, 1969.

Hiemstra. R. P. Continuing Education for the Aged: A Survey of Needs and Interests of Older People. Adult Education, 1972, 22, 100-109. (a)

Hiemstra, R. P. The Educative Community. Lincoln. Nebraska: Pro­fessional Educators Publications, Inc., 1972. (b)

Houle, C. O. The Design of Education. San Francisco: Jossey-Bass, 1972.

Johnston, J. W. C. The Educational Pursuits of American Adults. Adult Education, 1963, 13, 217-221.

Justus, J. E. PERT. School Management, 1967, 11(12), 24-29.

Kaimann. R. A. Educators and PERT. Journal of Educational Data Processing, 1966, 3, 43-57.

Kneller, G. F. Goal-Free Evaluation. Evaluation Comment, 1972, 3(4), 7-8.

Knowles, M. S. Modern Practice of Adult Education. New York: Association Press, 1970.

Knox. A. B. Clientele Analysis. Review of Educational Research. 1965, 35, 231-239.

Krathwohl, D. R., Bloom, B. S., and Masia, B. B. Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook II. Affective Domain. New York: David McKay Company, Inc., 1964.

Leagans, J. P. A Concept of Needs. Journal of Cooperative Extension, 1964, 2, 89-96.

Lehmann, H. The Systems Approach to Education. Audiovisual Instruction, 1968, 13, 144-148.

Lindeman, E. C. Ten Steps in Community Action. In E. Harper and A Dunham, eds., Community Organization in Action. New York: Association Press, 1959.

Long. R. W. Continuing Education for Physical Therapists in Nebraska: A Survey of Current Practices and Self-Expressed Needs with Recommendations for Program Development. Unpublished doctoral dissertation. University of Nebraska, 1972

Mager, R. Preparing Instructional Objectives. Palo Alto, California: Fearon Publishers, 1962.

McLoughlin, D. Participation of the Adult Learner in Program Planning. Adult Education, 1971, 22, 30-35.

McMahon, E. E. Needs of People and Their Communities and the Adult Educator. Washington, D.C.: Adult Education Associa­tion of the USA, 1970.

Merriman, H. O. Evaluation Theory into Practice. Journal of Research and Development in Education, 1970, 3(4), 48-58.

Mezirow, J. Toward a Theory of Practice. Adult Education, 1971, 21, 135-147.

National Study of Secondary School Evaluation. Evaluation Criteria. 1960 edition. Washington, D.C.: National Study of Secondary School Evaluation. 1960.

Popham, W. J., and Baker. E. L. Systematic Instruction. Englewood Cliffs, N.J.: Prentice-Hall, Inc., 1970.

Scriven, M. Pros and Cons about Goal-Free Evaluation. Evaluation Comment, 1972, 3(4), 1-4.

Sheasga. T. A. Definition of Needs and Wants. Adult Education, 1961, 12, 52-53.

Silvern, L. C. Systems Engineering of Education IV: Systems Analysis and Synthesis Applied Quantitatively to Create an Instructional System. Los Angeles: Educational and Training Consultants Co., 1969.

 Silvern, L. C. Systems Engineering of Education V: Quantitative Concepts for Education Systems. Los Angeles: Educational and Training Consultants Co., 1972.

Sower. C. et al. Community Involvement: The Webs of Formal and Informal Ties the Make for Action. Glencoe, Illinois: The Free Press, 1957.

Stake, R. E. The Countenance of Educational Evaluation. Teachers College Record, 1967, 58, 523-540.

Stake, R. E. Objectives, Priorities, and Other Judgment Data. Review of Educational Research, 1970, 40, 181-212.

Stauber, R. I. PPBS for Extension? Journal of Cooperative Extension, 1968, 6, 229-235.

Stufflebeam, D. L. Evaluation as Enlightenment for Decision-Making. In W. H. Beatty, Improving Educational Assessment and an Inventory of Measures of Affective Behavior. Wash­ington, D.C.: Association for Supervision and Curriculum Development, 1969.

Stufflebeam, D. L. Toward a Science of Educational Evaluation. Educational Technology, 1968, 5(14), 5-17.

Suchman, E. A. Evaluative Research: Principles and Practices in Public Service and Social Action Programs. New York: Russell Sage Foundation, 1967.

Theide, W. Evaluation and Adult Education. In G. E. Jensen, A. A. Liveright, and W. Hallenbeck, eds. Adult Education: Outlines of an Emerging Field of University Study. Wash­ington, D.C.: Adult Education Association of the USA, 1964.

Thelen, H. A. Dynamics of Groups at Work. Chicago: The Uni­versity of Chicago Press, 1954.

Thorndike, R. L., and Hagen, E. Measurement and Evaluation in Psychology and Education. New York: John Wiley and Sons, 1969.

Tuckman, B. W. Conducting Educational Research. New York: Harcourt, Brace, Jovanovich, 1972.

Tyler, R. W. Basic Principles of Curriculum and Instruction. Chicago: University of Chicago Press, 1950.

Tyler, R. W., ed. Educational Evaluation: New Roles, New Means. 68th Yearbook of the National Society for the Study of Education, Part 2. Chicago: University of Chicago Press, 1969.

Tyler, R. W., Gagne, R. M., and Scriven, M. Perspectives of Curriculum Evaluation. AERA Monograph Series on Curriculum Evalua­tion, No, 1. Chicago: Rand McNally & Company, 1967.

Verner, C. A. A Conceptual Scheme for the Identification and Classification of Processes of Adult Education. Chicago: Adult Education Association of the USA, 1962.

Verner, C. A. Definition of Terms. In G. E. Jensen, A. A. Liveright, and W. Hallenbeck, eds. Adult Education: Outlines of an Emerg­ing Field of University Study. Washington, D.C.: Adult Edu­cation Association of the USA, 1964.

Wedemeyer, C. A. Evaluation of Continuing Education Pro­grams. American Journal of Pharmaceutical Education, 1969, 33, 793-810.

Worthen, H. R. Toward a Taxonomy of Evaluation Designs. Educational Technology, 1968, 8, (15), 3-9.