Evaluation Methods and Techniques

We employed a number of evaluation methods, techniques, and strategies during the project to meet both the Kellogg Foundation requirements and our own desires to continuously improve what we were doing.

  1. During the first year we acquired the consultive services of Dr. Nick Smith, Professor, Instructional Development, Design, and Evaluation, Syracuse University, and a nationally recognized evaluation expert. He provided considerable advice on gathering evaluation data, how to work with those associated in some way with the project in terms of collecting information, and in assessing our evaluation information. He was instrumental in our creating an annual project vita and he provided feedback on early drafts of our annual reports.
  2. We created annual vitas of our project activities, results, and products during the first five years of the project. This involved compiling in a binder information pertaining to such activities or events as descriptions of project visitors, abstracts of related publications, details on presentations by project personnel, records of project endeavors, and information on any publicity the project received. Thus, there exists a useful history and record of project results.
  3. Maureen Goodman and I carried out an ongoing assessment of the project's accomplishments, progress toward goals, and evolving directions. We discussed such issues regularly in our frequent meetings together, examined quarterly reports, and used our regular full staff meetings to discuss evaluative issues. We worked together with able support from Irene Quinlan, the project's administrative coordinator, in developing each annual report through the first five years of the project. During the last two years of the project I coordinated the evaluation efforts and maintained the practice project personnel submitting quarterly reports.
  4. One of the roles for both internal and external advisory council members was that of providing evaluative advice and recommendations. Portions of advisory council meetings typically were used for obtaining such information. Sub-groups from both councils also were used to provide specialized advice on technology, information dissemination, library processing, and historical research issues. External advisory council members also helped develop some policy during the first two years pertaining to the collections and information access that guided much of our subsequent actions.
  5. One of the critical activities during the first years was securing the services of a vendor to build the computer-based system we envisioned necessary for providing world-wide access to the Collections. Thus, we created a campus-wide committee to conceptualize what was needed, developed a call for proposals from vendors, evaluate the proposal received, help select the most appropriate vendor, and provide advice during subsequent developmental activities. Approximately a dozen people from various campus units served admirably on this committee over a several month period.
  6. The campus-wide committee process was also used later in the project when we conceptualized and developed the important template necessary to portray and arrange the searchable information for the system described in number five. Under the capable leadership of Beth Oddy, an active committee created a whole new way of thinking about accessing and disseminating information. This effort resulted in several conference presentations and publications documented in prior reports.
  7. We utilized a mid-project evaluation team made up of Dr. David Bearman, Washington, DC, information and archive specialist, Dr. Ralph Brockett, Knoxville, TN, adult education history specialist and past user of the archival collection, and Dr. Sharan Merriam, Athens, GA, general adult education specialist. Team members spent two days on campus, examined project documentation, and interviewed various people associated with the project. They provided a report and presented a series of recommendations to project personnel. Information pertaining to their efforts was included in a prior report.
  8. Special information collection and analysis efforts also were carried out by individual project personnel and the results used to affect project activities in various ways. For example, Barbara Florini gathered actual conversations of our distance education enrollees to better understand the computer mediated conferencing system. Maureen Goodman maintained paper copies of email conversations pertaining to the project so records of important dialogue could be retained. Mary Beth Hinton carried out evaluations of the project's publications and public relations efforts. Rae Rohfeld maintained records of all visiting scholar debriefings to help us improve our services to scholars and better understand the value of the collection to them. Ollie Owen kept track of developments related to the Computers and the Elderly project. Various research assistants evaluated aspects of our conferences, distance education efforts, and New Horizons journal, providing written reports that informed subsequent actions in multiple ways.
  9. Finally, we kept an archive of our products and project processes. In addition to the project vita material described above, we kept computer printouts, correspondence, meeting agendas, meeting minutes, and planning documents. All of these materials are now a part of the Adult Education Collection and a permanent record for future researchers to examine.

PART TWO: THE SEVENTH-YEAR ACTIVITIES

Introduction

The final year was seen as a period for wrapping-up the project and making an effort to institutionalize as many of the remaining components as possible. It also was a period of considerably fewer financial resources than in previous years. Thus there were fewer personnel involved.

Nevertheless, two people were employed full-time part of the year and several were involved on a part-time or volunteer basis. Marcia Henry and Chris Darby provided secretarial and administrative assistance during portions of the year. Five graduate students (Annamary Allen, Charles Awasu, Indra Balachandran, Ina Sue Brown, and Sue Slusarski) were employed or volunteered on a part-time basis in a variety of functions. Dan Eastmond was employed at approximately a 75% basis for much of the year to coordinate distance education efforts and provide other technological support. Tim O'Shea, an undergraduate student, provided some short-term consultation and support pertaining to some of our optical scanning efforts. Preston Clark and Yuming Tung provided some short-term consultation related to our various computer efforts. Rae Rohfeld and I provided various kinds of support in our faculty roles.

Following is a brief summary of some activities pertaining to our objectives for the year:

Objective 1: AEDNET

The project created a national electronic network, AEDNET, that commenced soon after the project began and continued until the summer of 1993. AEDNET continued to grow during much of the seventh year under the capable management of Preston Clark (AEDNET SysOp and moderator--Fall and Spring semesters). AEDNET was expanded to be seen globally by a variety of listservs. The number of members grew to 544, approximately 11% more than the previous year. In addition to that, an unknown number of readers within at least nine known member newservers (gateways to listservs around the world containing hundreds of additional subscribers) and people from a total of 31 nodes world-wide participated.

An important activity during the year was finding another home for AEDNET (and New Horizons) as the project came to a close. Under my guidance, Dan Eastmond disseminated a call for proposals from institutions interested in assuming management of the system. In January, 1993, we chose Nova Southeastern University as the new site for AEDNET and New Horizons because of the institution's commitment to house, staff, and enhance the resources and the fit of both activities within the programs at Nova. The change over took place in June, 1993.

Objective 2: New Horizons in Adult Education

Our electronic journal, New Horizons, the first refereed electronic journal in the country, also was published over AEDNET. The journal promoted scholarship in the field and provided an opportunity for adult education graduate students throughout the world to obtain publishing and editing experience.

New Horizons continued to mature during the year and find new audiences. The fall issue was devoted to an experimentation with two interactive pieces to stimulate discussion among readers. The spring issue contained four articles.

Dan Eastmond and Charles Awasu coordinated the transfer of related journal manuscripts and files to Nova personnel in the late spring, 1993, as the institution accepted responsibility for it as well as the parent network. Dan Eastmond and I met with Dr. Nancy Gadbow, former member of the internal advisory council and former Adult Education Graduate Program faculty member, and currently a member of the Nova faculty. She was assuming editorship of New Horizons and we provided some advice and ideas to aid her efforts.

Objective 3: MacLars and System Development

One of the project's hallmarks was the various efforts we made through computer technology to develop various systems and networks. As was described in last year's report, we developed the MacLars prototype system as a replacement for the initially conceived KLARS system. During this final year we added into the system an optical scanning device which enabled us to incorporate new biographical and bibliographical material from text material created earlier in the project.

During this final year we also maintained our local area network which connected all project staff and faculty electronically. In addition, my office was established as a model for distance education.

Objective 4: Historical Research

The original plan was to process new and previously unprocessed materials (more than an 800 linear-foot manuscript collection), review and refine a collection policy, buy an optical disk system with telecommunications links, and scan materials on to disks for researchers to access via computer. We learned in the first year that existing optical disk systems could not accommodate our needs, so we designed our own system with Plexus Computers, Inc. In previous reports we described the difficulties involved in working with that company and we never realized that particular goal. However, as noted in the previous section we did develop a Macintosh computer-based version of the originally conceived system.

Much of our promotion of historical research would have been fruitless if we had not also developed and added to the Adult Education Collection. Collection development and management has become routinized through the Bird Library policy and personnel. Former Kellogg Project Adult Education Librarian, Terry Kennan, now is a full-time staff member of the Syracuse University Library and has integrated all Adult Education Archive and Collection activities into his daily responsibilities [Note: See the postscript in the first portion of this report about Terry's unfortunate decline in health a decade after the project was completed]. The collection now is regularly used by both local and visiting scholars for historical and other research purposes.

The final history conference was held in the spring of 1993. Rae Rohfeld and a team of colleagues developed and coordinated an excellent event. Several papers were presented (see Appendix F).

Objective 5: Visiting Scholars

The visiting scholar component was originally designed to support individual scholars and this past year there were eight of them:

  1. Ralph Brockett, Associate Professor, Adult Education, University of Tennessee, Knoxville, TN.
  2. Paul Edelson, Dean, School of Continuing Education, State University of New York-Stony Brook, Stony Brook, NY.
  3. Nancy Lee Karlovic, Assistant Professor, Adult Education, Western Washington University, Bellingham, WA.
  4. Huey B. Long, Professor, Adult Education, University of Oklahoma, Norman, OK.
  5. Fred Schied, Assistant Professor, Adult Education, Penn State University, University Park, PA.
  6. Robert Shen, Visiting Fellow, University of Maryland, College Park, MD.
  7. Ruth Queen Smith, Graduate Assistant, Adult Education, University of Tennessee, Knoxville, TN.
  8. Melody Thompson, Graduate Student, Penn State University, Boalsburg, PA.

Objective 6: Distance Education

The initial plan was to develop an independent study master's degree program in adult education. However, the School of Education faculty did not approve the plan. Instead it was decided that we should start on a modest scale with distance education efforts using computer mediated conferencing as our delivery tool.

Two courses were offered during the fall of 1992, "Lifelong Learning: The Adult Years" (ATE 711) taught by Dr. Nancy Gadbow, and "Continuing Professional and Higher Education" (ATE 600) taught by Dr. Rae Rohfeld. An additional two courses were offered during the spring of 1993, "Survey of Adult Education" (ATE 601) taught by Dr. Nancy Gadbow, and I taught "Community and the Adult Educator" (ATE 604). Finally, I worked with a group of students from Toronto, Canada, during the summer of 1993 using the electronic communication software to help acquaint them with distance education approaches. Dan Eastmond was also involved in evaluating the distance education courses.

Factors Affecting Goal Attainment

There were three factors that somewhat hindered the achievement of project goals and objectives during the final year:

  1. Because of declining resources both for the project in terms of Kellogg Foundation funding and for the Graduate Program in terms of Syracuse University support, there were fewer faculty and staff involved. This meant that several responsibilities were assumed by only a few people. Fortunately, hard work by part-time personnel and even several who volunteered much of their time facilitated our still achieving project goals. The respect that these fine individuals maintained for the project and their commitment to improving adult education were instrumental in our having a successful final year.
  2. Although the actual announcement of the Graduate Program's closure was made earlier than this year, there were several corresponding administrative and coordination functions requiring time and commitment by several people affiliated in some way with the project during this final year. Thus, at times it was difficult to keep the various priorities in perspective. However, constant support of each other and a "regenerative spirit" possible because of periodic contact with visiting scholars, history conference participants, and others who gave encouragement helped us maintain our momentum.
  3. Concomitantly, there was somewhat of a declining interest in the project both on and off campus. We had fewer resources to spend on publicity, for example, and were unable to spread the word about our efforts as much as we would have liked. There were fewer of us to promote involvement in project efforts so maintaining steady interest in research activities, the conference, our visiting scholar program, and the collections required extra efforts. Fortunately, the international audience we had via AEDNET assisted us in this regard.

There also were three factors that helped us achieve our goals and objectives:

  1. Most important was the continued good services of several excellent research assistants and staff members. Putting in long hours, often on a volunteer basis, was the hallmark of these fine people. Even though it was a final year, everyone seemed to take their roles very seriously and worked with dedication and conscientiousness.
  2. Correspondingly, there was continuing dedication by Adult Education faculty and students. Besides the fine work of Rae Rohfeld already mention, all the adjunct faculty provided some type of support. In addition, many Adult Education graduate students helped in some manner by either volunteering their time or by studying the project so subsequent understanding could influence their work as professionals.
  3. Finally, the contributions to knowledge made by our visiting scholars and conference participants added to the legacy we hope this project has achieved in terms of enhancing the adult education field. Their papers, presentations, and current and future publications are important to development of the knowledge base in adult education.

PART THREE: IMPORTANT OUTCOMES AND LESSONS

Based on my experience with the project throughout its entire period and as a professional who has worked in the adult education field for thirty years, I believe there were several important project outcomes. Many of these were cutting edge activities in terms of using technology or conceptualizing new ways for thinking about information transfer. In many ways, we were forerunners of some concepts being discussed today as part of the information super highway. We also learned some important lessons that can serve to inform the Foundation in various ways pertaining to future grants and contracts. Each outcome or lesson will be lettered and will contain one or more related recommendations.

A. One of the project's intents was to create new models for information access and management that can be applied in other fields. We believe we did this in various ways as pointed out in this and prior reports. AEDNET as a discipline-specific world-wide network, New Horizons (the first peer-reviewed electronic journal), the computers and the elderly project, MacLars, the documentation strategy work, and IISN are examples of our cutting edge efforts.

Recommendation 1: The Kellogg Foundation should seek proposals from other disciplines and institutions that specifically attempt to duplicate or improve on several of the project's models.

Recommendation 2: The Kellogg Foundation should commission a project in approximately five years for follow-up research on the Syracuse University Kellogg Project to determine if such models influenced the development of knowledge or were adapted by others in some way.

B. We also made a special effort to ensure the continuation of various project components and activities. Some of them were incorporated within the Syracuse University environment while others moved to new institutional sites:

Recommendation 3: The Kellogg Foundation should encourage in future proposals the development of plans to continue important project components either within the original institution or by migrating them to other institutions at the conclusion of a several year effort.

C. Those associated in some way with conceptualizing the project were convinced of the value associated with understanding, clarifying, and enhancing the historical knowledge base of a field. Thus, an important and substantial by-product of the Syracuse University project was enhancement of our adult education archival collection. Not only have all materials been processed for use by historical researchers, many things were added. The end result is a resource that can truly be called a national treasure for the Adult Education profession.

Recommendation 4: The Kellogg Foundation should examine ways to better understand the history of other disciplines, perhaps through specific calls for proposals that will build discipline-specific archival collections.

D. Related to the above point, an important outcome of the project was the idea and success of our working history conferences under the capable leadership of Rae Rohfeld. Many conference participants were afforded a rare opportunity to not only carry out research in a discipline-specific archival collection, they also received immediate feedback on preliminary findings as a means for enhancing their understanding and ability to communicate such knowledge to others. Other participants were able to hear about several "works in progress" and contribute their own ideas and recommendations to researchers.

Recommendation 5: The Kellogg Foundation should consider encouraging future projects to include working conferences as part of their implementation strategies whenever they are appropriate.

E. Another important project outcome was our approach to determining a strategy for documenting historical and other information pertaining to Adult Education. By pulling together a focus group of experts in adult education, library science, archival work, and information dissemination for discussion and planning activities, Terry Keenan and other project personnel developed a documentation strategy that has impacted on the Syracuse University library and that is available to others in the field. Prior reports have presented information pertaining to our documentation strategy.

Recommendation 6: The Kellogg Foundation should consider including support for other disciplines to develop documentation strategies, policies, and procedures.

Recommendation 7: The Kellogg Foundation should examine the use of focus groups consisting of interdisciplinary representatives as a mechanism for developing important policy that impact on or result from future funded projects.

F. An important outcome was that we helped to change the attitudes of many adult educators about technology and its various uses for communication, dissemination, and networking. We believe we helped move several individuals and organizations forward in their incorporation of computer technology into daily activities. We also helped to make electronic networking common among many people. In 1989, for example, I visited a colleague at Iowa State University. He had programmed his computer so that each morning when he turned it on any AEDNET messages were the first thing that crossed his screen. I believe many others now think of electronic communication as an invaluable professional tool.

Recommendation 8: The Kellogg Foundation should fund additional projects aimed at helping various disciplines incorporate technology more fully into the daily work of professionals, especially its use in information access and dissemination.

G. We learned an important lesson during the project because of problems we had with the Plexus Corporation, the outside vendor we had selected to develop our optical scanning, storage, and retrieval system. An interdisciplinary committee made up of computer, education, library, information, and administration representatives selected Plexus out of several who put in proposals to build the system our vision for the library required. The company built an acceptable prototype and were in the process of developing the final system when they were forced to file for protection under bankruptcy laws. Unfortunately, all work on the system stopped. This was, of course, a huge disappointment and an unexpected turn of events. It curtailed our intent to build a complete system, networkable from a distance, for information access, retrieval, and dissemination of digitized information from our collection. Instead we developed a much smaller system on our own, MacLars, based on standard scanners and Macintosh computers us hypertext software (see prior reports for information on the original Plexus efforts and our more recent MacLars efforts. In essence, a big risk of being on the cutting edge of knowledge development, as we believe we were, is that occasionally things don't work out as anticipated. We simply were slightly ahead of our time with our plans. Today other systems are accomplishing much of what we had envisioned at a fraction of the cost. In addition, massive amounts of information are now accessible throughout the world via the Internet and other electronic networks demonstrating that what we had in mind is clearly possible and, in many cases, now taking place.

(Revised, April, 2005)

Return me to Roger Hiemstra's opening web page.

Return me to Section One of this report

Return me to the distance education section of Roger Hiemstra's web page

I would like to continue reading this report.