DATA COLLECTION TECHNIQUES

Roger Hiemstra

Revised and Updated

2010

Data Collection Techniques

Key:

I -- Individual level N -- Primarily useful for needs assessment G -- Group level E -- Primarily useful for evaluation

O -- Organizational level C -- Community level

Mailed Devices

Checklist/Diagnostic Form (N,E)

Forms given to respondents individually or in groups where answers are checked on a list of statements (I/G).

Uses:

1. For groups of people in a meeting.

2. For individuals randomly selected from a group meeting who will complete them at home.

3. To collect evidence of progress made or practices adopted.

4. To assess a perception of individual need or interest on a topic.

Variation:

A group discussion of diagnostic form results can be carried out for purposes of further clarification, building consensus, determining new needs, and providing input for further program planning.

Limitations:

1. Can be difficult to interpret.

2. Difficult to obtain a good list of names.

References:

Hiemstra, R., & Sisco, B. (1990). Individualizing instruction. San Francisco: Jossey-Bass.

Knowles, M. S. (1980). The modern practice of adult education (2nd Edition). Chicago: Association Press.

Community Survey (N)

An analysis of various aspects of behavior and social interaction within a given community (C).

Uses:

1. To examine intergroup relations.

2. To study the physical aspects of communities.

3. To obtain an historical perspective relative to a community.

4. To examine population mobility.

5. To examine technological changes.

6. To examine changes in status and values.

Variations:

1. Can be combined with a larger community study effort.

2. Public opinion surveys or polls.

References:

· American Library Association. (1960). Studying the community. Washington, DC: American Library Association.

· Baumel, C. P., Hobbs, D. J., & Powers, R. C. (1964). The community survey (Soc. 16). Ames, IA: Iowa State University, Cooperative Extension Service.

· Hampton, K., Sessions, L., Her, E. J., & Rainie, L. (2009). Social isolation and new technology: Methodology and design. Washington, DC: Pew Internet & American Life Project. Retrieved September 1, 2010, from http://www.pewinternet.org/Reports/2009/18--Social-Isolation-and-New-Technology/Appendix-C/Methodology-and-Design.aspx

· Warren, R. L. (1965). Studying your community. New York: Russell Sage Foundation.

Interest or Attitude Inventory or Survey (N)

A device for finding out in what participants or potential participants are interested (see mailed questionnaires) (I/G).

Uses:

1. Study of continuing education interests and interest areas.

2. Study of attitudes toward learning.

3. Study of attitudes on a particular subject.

Mailed Questionnaire (N,E)

A mailed survey form used to obtain a broad analysis of some social phenomenon or problem (I/G/0).

Uses:

1. For reaching a wide geographic distribution of people.

2. For reaching a relatively homogeneous, fairly well-educated group.

3. For understanding some current situations, attitudes, and/or interests.

4. To determine factual material.

5. For making a survey of employee needs, problems, or interests.

Variations:

1. Delphi Technique.

2. Q-sort or card sort.

3. Picture sort.

Limitations:

1. The reliability of the results can be quite low at times, as well as the rate of return.

2. Any open-ended responses or added comments may be difficult to interpret.

References:

· Byrn, D. (Ed.). (1973). Evaluation extension. Topeka, KS: Ives Publications.

· Leece, P., Bhandari, M., Sprague, S., & Swiontkowski, M. F. (2004). Internet versus mailed questionnaires: A controlled comparison. Journal of Medical Internet Research, 6(4), e39. Retrieved September 1, 2010, from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1550620/

· Oppenheim, A. M. (1966). Questionnaire design and attitude measurement. New York: Basic Books.

· Sudman, S., & Bradburn, N. M. (1983). Asking questions: A practical guide to questionnaire design. San Francisco: Jossey-Bass.

Also see the Delbecq citation under the nominal group technique.

Verbal Devices

Informal Interview (N,E)

An unstructured and unstandardized method of obtaining answers to various questions and gaining information on various topics (I/G).

Uses:

1. In beginning discussion on a topic.

2. For small group discussion.

3. For gathering qualitative information.

4. In obtaining insight on a problem or program's progress.

Variation:

A probing interview with prompt sheets or cue cards combines both the informal and the more structured interview format.

References:

· Bogdan, R. C., & Biklen, S. K. (1982). Qualitative research for education. Boston: Allyn and Bacon, Inc.

· Merriam, S. B., & Simpson, E. L. (1984). A guide to research for educators and trainers of adults. Malabar, FL: Krieger Publishing.

· Tough, A. (1979). Adults' learning projects. Austin, TX: Learning concepts.

Personal Interview (N,E)

The collection of data through direct verbal interaction between individuals, usually formal in nature - the data collection takes place face to face or via a phone (I/G).

Uses:

1. For obtaining specific facts and opinions.

2. To measure attitudes and interests.

3. For an understanding of current situations.

4. When a high percentage of participation is needed.

References:

· Bingham, W. V. D., & Moore, B. V. (1959). How to interview. New York: Harper & Row.

· Bradburn, N. M., & Sudman, S. (1979). Interview method and questionnaire design: Response effects to threatening questions in survey research. San Francisco: Jossey-Bass publishers.

· Cannel, C. F., Marquis, K. H., & Laurent, A. (1977). A summary of studies of interviewing methodology (Vital and health statistics, series 2, No. 69). Rockville, MD: U.S. National Center for Health Statistics.

· Hyman, H. (1970). Interviewing in social research. Chicago: University of Chicago Press.

· Ipathia, Inc (2005). Personal interview surveys vs. web surveys: A comparison. Retrieved September 1, 2010, from http://knowledge-base.supersurvey.com/in-person-vs-web-surveys.htm

Phone Survey (N,E)

The collection of data through a series of patterned questions asked orally via a telephone (I/O).

Uses:

1. For obtaining specific facts and opinions.

2. Allows interviewer to ask additional follow-up or clarifying questions.

3. When personal interview is not possible but a mailed device is not practical.

Variation:

There is a growing use of email, computer conferencing software, and online survey programs such as SurveyMonkey.

Limitations:

1. Difficult to understand non-verbal reactions.

2. Can be expensive if long-distance charges are incurred.

3. Phone surveys are not always the wisest choice for various reasons.

References:

· Fink, A., & Kosecoff, J. (1985). How to conduct surveys: A step-by-step guide. Beverly Hills, CA: Sage Publications.

· Henning, J. (2009). The phone survey in decline (Voice of Vovici Blog). Retrieved September 1, 2010, from http://blog.vovici.com/blog/bid/19381/The-Phone-Survey-in-Decline

Team Interview (N)

Collection of data through direct verbal interaction between two interviewers (can be more) and (usually) one respondent (I).

Uses:

1. When time is a factor.

2. When interviewing those with verbal skills, much experience, and/or educational backgrounds.

3. In exploratory studies.

Telephone Interview (N,E)

A method of collecting information quickly and relatively inexpensively (I/G).

Uses:

1. Where good rapport with respondents has been or can be established.

2. For gathering factual information.

3. For gathering opinions, suggestions, and ideas.

4. For obtaining information about feelings and attitudes.

Systematic or Structured Devices

Advisory Council/Committee Input (N,E)

The use of a council or committee to provide advice, ideas, and experiences pertaining to needs or evaluation. Can be intuitive, experiential, or come from data bases (O/C).

Uses:

1. To obtain advice, insight, or factual information from people knowledgeable about an area.

2. To evaluate ongoing or completed educational programs.

3. Public hearings.

4. Town or neighborhood meeting or block organizations/clubs.

Reference:

· Hiemstra, R. (2002). The educative community: Linking the community, education, and family. Fayetteville, NY: Syracuse University Adult Education Publications. Retrieved September 1, 2010, from /lll.html

Anecdotal Records (N,E)

Observations and descriptions of behaviors deemed typical of an individual (also known as skill inventories or task analysis records) (I).

Uses:

1. To study human behavior.

2. To determine individual performance problems.

Content Analysis (N,E)

The objective, systematic, and quantitative description of the manifest content of communication on a particular subject wither directly or indirectly (I/G).

Uses:

1. For the analysis of propaganda.

2. To examine the treatment of a particular subject in books, media, etc.

3. For the analysis of readability of various materials.

4. For the development of need or objective statements.

5. In historical studies.

References:

· Holsti, O. R. (1968). Content analysis. In B. Lindsey & E. Aronson (Eds). The handbook of social psychology (2nd Edition). Reading, MA: Addison-Wesley.

· Kerlinger, F. N. (1999). Foundations of behavioral research (4th. Edition). New York: Holt, Rinehart, & Winston.

Critical Incidents Technique (N)

An interview with a supervisor, judge, or someone knowledgeable about an individual to determine specific behavior patterns that are considered critical to the skills or areas of behavior being studied; sometimes referred to as job analysis or task analysis (I).

Uses:

1. In studies of leadership ability.

2. For determining qualifications of individuals for certain jobs or duties.

3. In studies of individual behavior or on the job behavior.

4. In efforts to examine education or training need in relation to job performance.

Variation:

The critical incidents technique also can be used as a research tool or to provide feedback to an individual. For example, the technique has been used to determine critical teaching incidents or to provide a mirror for the improvement of instruction.

Limitation:

It is past or present oriented, not future oriented.

References:

· Argyris, C., Putnam, R., & Smith, D. L. (1990). Action science. San Francisco: Jossey-Bass.

· Merriam, S. B. (1991). Case study research in education. San Francisco: Jossey-Bass.

· Nadler, L. (1982). Designing training programs: The critical events model. Reading, MA: Addison-Wesley.

· Rigors, P. (1971). Case methods in human relations: The incident process. New York: McGraw-Hill.

Panel Survey (N,E)

The interview and study of a selected sample of respondents at two different times: Panels are picked and know of the task ahead of when the data are actually collected (C/O).

Uses:

1. To study changes in behavior.

2. To study changes in feelings and attitudes.

3. To study needs and interests.

4. To evaluate programs or materials.

Variation:

A survey form also could be mailed out before and after some experience; however, the results may not have the reliability and validity of an actual interview.

Limitations:

1. If panel members drop out between the two contact times the results can be affected.

2. A low return rate on any mailed out forms will affect the results.

Peer Evaluating (N,E)

The evaluation of behavior by an individual's peers (usually voluntary); this technique initially was used for evaluation purposes but now is used in business and industry as a feedback device for ascertaining employee training needs (also known as peer review) (I/G).

Uses:

1. To assess some aspect of human behavior.

2. To evaluate a person's job performance

Limitations:

1. The technique requires trained observers with considerable skill and sensitivity.

2. Peers may not always report/perceive information accurately.

Reference:

· American Medical Association. (1971). Peer review manual. New York: American Medical Association.

Performance Audit (N,E)

A broad approach to solving some problem by comparing performances against some competency model or selected performance criteria. Collected data often are analyzed through sophisticated statistical procedures (I/G/O).

Uses:

1. For comparing local practices or behaviors against national, industry, or selected norms.

2. For examining a broad problem area that exists within a particular organization or setting.

Limitations:

1. Usually requires a large data collection and analysis team so can be time consuming and expensive.

2. It can be difficult to agree on an appropriate competency model.

References:

· Mager, R, & Pipe, P. (1970). Analyzing performance problems. Belmont, CA: Fearon.

· Sargent, A. (1983). Competency-based management. In F. Ulschak (Ed.), Human resource development. Reston, VA: Reston Publishing.

· Zemke, R. (1982). Job competencies: Can they help you design better training? Training, 19(5), 28-31.

Power Structure Analysis (N)

A determination of the manner in which individual power actors in a social system relate to each other (although this is not a standard needs assessment technique, it provides useful information to better understand needs, program planning strategies, blocking groups, etc.--also known as community leader analysis) (C).

Uses:

1. For community analysis efforts.

2. To understand formal organization leadership.

3. To assess leadership skills and experience.

References:

· Domhoff, G. W. (2009). How to do power structure research. Retrieved September 1, 2010, from http://sociology.ucsc.edu/whorulesamerica/methods/how_to_do_power_structure_research.html

· Powers, R. C. (1965). Identifying the community power structure (North Central Regional Extension Publication No. NCRS-5, Leadership Series No. 2, Soc. 18). Ames, IA: Iowa State University, Cooperative Extension Service.

· Tait, J. L., Bokemeier, J., & Bohlen, J. M. (1976). Identifying the community power actors: A guide for change agents (North Central Regional Extension Publication 59). Ames, IA: Iowa State University Cooperative Extension Publications.

Supervisory Ratings (N,E)

Ratings of an individual made by someone in a supervisory capacity (also known as job analysis performance appraisal, performance review, supervisory) (I/C/O).

Uses:

1. To analyze individual behavior, performance, and training need.

2. To determine where performance gaps can be interpreted as educational need.

References:

· Bernardin, H., & Beatty, R. (1983). Performance appraisal: Assessing human behavior at work. Boston: Kent Publishing.

· Mager, R., & Piper, P. (1970). Analyzing performance problems. Belmont, CA: Fearon Publishers.

Also see the references cited for the critical incidents technique.

Systematic Study of Available Records (N)

An analysis of available records on a particular subject or need area (I/C).

Uses:

1. When an interview or questionnaire procedure is impossible.

2. For information on the past.

3. For use in a historical study of needs.

4. When a case study technique is desired.

Variation:

An analysis of census records to determine demographic changes, the concentration of specific characteristics (such as people with lower levels of education by census tract), and the changing work force.

Task Analysis (N)

The observation of a job, performance, or task to determine the relevant sub-tasks, components, or responsibilities (I).

Uses:

1. For distinguishing between individuals as they perform various tasks.

2. To examine specific job training needs.

3. To plan for job enrichment programs.

Limitations:

1. Primarily behavioral-based and does not place much value on cognitive or attitudinal differences.

2. Requires very good observational and assessment skills.

References:

· Carlisle, K. (1983). Improving task analysis in the nuclear utility industry. Performance and Instruction, 22(2), 8-9, 27.

· Gael, S. (1983). Job analysis: A guide to assessing work activities. San Francisco: Jossey-Bass.

Test, Diagnostic Tools, Pretests (N,E)

Group or individual completion of a device designed to test or measure some aspect of behavior or knowledge (I/G).

Uses:

1. To determine needs through an assessment of deficiencies.

2. To measure performance/status on some task, attribute, or attitude.

Variation:

A group discussion of test results can be carried out for purposes of further clarification, building consensus, or elaboration.

Limitations:

1. There are potential problems with validity, standardization, and measurement.

2. Frequently, this technique will need to be combined with other techniques.

Reference:

· Hopkins, K. D., & Stanley, J. C. (1990). Educational and psychological measurement and evaluation (7th Edition). Englewood Cliffs, NJ: Prentice-Hall, Inc.

Gaming or Group Interaction Devices

Brainstorming (N)

Group (usually small) members spontaneously generate a wide variety of ideas, interests, etc.; clarifying and follow-up techniques also are typically required (I/G).

Uses:

1. Where quick responses are desired.

2. When some initial ideas or categories of needs are required.

Limitations:

1. Responses obtained quickly or spontaneously may not always reflect reality.

2. Some people may not desire to participate in a brainstorming activity.

References:

· Brainstorming. Retrieved September 1, 2010, from http://www.brainstorming.com/

· Clark, C. H. (1958). Brainstorming. New York: Doubleday.

· Clark, C. H. (1980). Idea management. New York: Amacon.

Delphi Technique (I,G,O)

A structured approach to collecting information, often of a forecasting nature, from several independent experts or judges. Such experts participate through two or more rounds of responses where assessing others' responses in comparison with your own eventually leads to some convergence toward common findings.

Uses:

1. For obtaining the opinions of experts through a fairly simplistic procedure.

2. Can be used to obtain information about present or projected conditions.

Variation:

It can be carried out by fax or electronic mail to speed the process of gathering information.

Limitations:

1. It can result in more responses and ideas than easily manageable in moving toward common themes.

2. It will generally consume considerable time if responses are obtained through the mail.

3. A high dropout rate by participants between rounds can limit the procedure's usefulness.

References:

· Custer, R. L., Scarcella, J. A., & Stewart, B. R. (1999). The modified Delphi technique – A rotational modification. Journal of Vocational and Technical Education, 15(2). Retrieved September 1, 2010, from http://scholar.lib.vt.edu/ejournals/JVTE/v15n2/custer.html

· Rath, G., & Stoyanoff, K. (1983). The Delphi technique. In F. Ulschak (Ed.), Human resource development. Reston, VA: Reston Publishing.

See also Delbecq (1975) noted for the Nominal Group Technique.

Gaming and Stimulation Device (N,E)

Role playing facilitated by some sort of a gaming board or tool; needs either personal observation or self-evaluation tied to it (I/G).

Uses:

1. To determine the participant's knowledge about some topic.

2. To facilitate interest in or practice with some topic.

References:

· Horn, R., & Cleaves, A. (Eds.). (1980). The guide to simulation/games for education and training (4th Edition). Beverly Hills, CA: Sage Publications.

· Simulation and Games. An international journal of theory development and research. Published quarterly by Sage Publications of Beverly Hills, CA.

Nominal Group Technique (N)

Group decision making where all suggestions are recorded and ranked (I/G).

Uses:

1. When contributions need to be encouraged.

2. When synergistic results from group involvement or commitment are desired.

References:

· CreatingMinds.org. (2010). Nominal group technique. Retrieved September 1, 2010, from http://creatingminds.org/tools/ngt.htm

· Delbecq, A. (1975). Group techniques for program planning: A guide to nominal group and delphi process. Glenview, IL: Scott Foresman Company.

· Scholtes, P. R. (1988). The team handbook. Madison, WI: Joiner Associates.

Team or Group Problem Solving (N)

The attempt to solve a particular problem through team action (also known as task force analysis) (I/G).

Uses:

1. In studies of human interaction within group activities.

2. In studies of work groups.

3. When a group facilitator can be employed to assist with assessment efforts.

References:

· Drucker, P. F. (1974). Management. New York: Harper & Row.

Outside Consultants

Consultants or Outside Experts (N,E)

A professional helper's advice (usually tied to assessing needs and evaluating programs) (I/C/G).

Uses:

· When participant observations are appropriate.

· When outside advice can be combined with evaluations.

Limitation:

Reliance solely on an outside expert for advice on some project or future activity may reveal only a partial picture of reality.

Systematic or Personal Observations (N,E)

Recommended practices observed, adoptions recorded, and recommendations made (I).

Uses:

1. In the study of practice adoption.

2. In the study of needed change in behavior.

Reference:

· Hiemstra, R., & Long, R. (1974). Survey of "felt" versus "real" needs of physical therapists, Adult education, 24, 270-279.

Selected References

Altschuld, J. W., & Kumar, D. D. (2009). Needs assessment: An overview. Thousand Oaks, CA: Sage publications.

Barbazette, J. (2006). Training needs assessment: Methods, tools, and techniques. Hoboken, NJ: Pfeiffer.

Brackhaus, B. (1984). Needs assessment in adult education: Its problems and prospects. Adult Education Quarterly, 34, 233-239.

Brown, F. G., & Wedel, K. R. (1974). Assessing training needs. Washington, DC: National Training and Development Service Press.

Calsyn, R. J. (1992). Acquiescence in needs assessment studies of the elderly. The Gerontologist, 32, 246-252.

Carlisle, K. (1986). Analyzing jobs and tasks. Englewood Cliffs, NJ: Educational Technology Publications.

Claydon, P. D., & Johnson, M. E. (1985). An instrument for needs assessment and evaluation of alcohol education programs. Journal of Alcohol and Drug Education, 31, 51-64.

Coffing, R. T. (1977). Client need assessment. In G. Zaltman & Others (Eds.), Dynamic educational change. New York: Macmillan.

Dick, W., & Carey, L. (1990). The systematic design of instruction (Third Edition). New York: Harper-Collins Publishers.

Evans, N. J. (1985). Needs assessment methodology: A comparison of results. Journal of College Student Personnel, 26, 107-114.

Fisher, J. C. (1986). Participation in educational activities by active older adults. Adult Education Quarterly, 36, 202-210.

Forlizzi, L. A. (1990). An assessment of the educational needs and interests of older, low-literate adults. Adult Literacy and Basic Education, 14, 204-211.

Griffith, W. S. (1978). Needs: Definition, assessment, and utilization. School Review, May, 382-394.

Harless, J. H. (1975). An ounce of analysis (is worth a pound of objectives. McLean, VA: Harless Performance Guild.

Hunt, G. J. F. (1986). Needs assessment in adult education: Tactical and strategic considerations. Instructional Science, 15, 287-298.

Johnson, D. E. (1987). Needs assessment: Theory and methods. Ames, IA: Iowa State University Press.

Jonassen, D., Hannum, W., & Tessmer, M. (1989). Handbook of task analysis procedures. New York: Praeger.

Kaufman, R. (1975). Needs assessment--what is it and how to do it. San Diego, CA: United States International University, UCIDT.

Kaufman, R. (1977). A possible taxonomy of needs assessments. Educational technology, 17(11), 60-66. (Note: This entire issue of the journal is devoted to the topic of needs assessment.)

Kaufman, R. A. (1979). Needs assessment: Concept and application. Englewood Cliffs, NJ: Educational Technology Publications.

Kaufman, R. (1988). Planning educational systems: A results-based approach. Lancaster, PA: Technomic Publishing.

Kaufman, R. A. (1993). Needs assessment: A user's guide. Englewood Cliffs, NJ: Educational Technology Publications.

Kaufman, R., & Bowers, D. (1990). Proactive and reactive planners: An even closer look at needs. Performance and instruction, 29(5), 7-10.

Kaufman, R., & English, F. W. (1979). Needs assessment: Concept and application. Englewood Cliffs, NJ: Educational Technology Publications.

Kaufman, R., Rojas, A. M., & Mayer, H. (1993). Needs assessment: A user's guide. Englewood Cliffs, NJ: Educational Technology Publications.

Kemerer, R. W., & Schroeder, W. L. (1983). Determining the importance of community-wide adult education. Adult Education Quarterly, 33, 201-214.

Lareau, L. S. (1983). Needs assessment of the elderly: Conclusions and methodological approaches. The Gerontologist, 23, 518-525.

Lee, W. (1973). The assessment, analysis, and monitoring of educational needs. Educational Technology, 13(4), 28-32.

Lee, W. W., & Rodman, K. H. (1991). Linking needs assessment to performance based evaluation. Performance and instruction, 30(6), 4-6.

Ludman, E. K., & Newman, J. M. (1986). Frail elderly: Assessment of nutrition needs. The Gerontologist, 26, 198-202.

McKillip, J. (1987). Needs analysis: Tools for the human services and education (Applied Social Research Methods Series, Volume 10). Newbury Park, CA: Sage Publications.

Messner, A. (2009). Needs assessment and analysis. Retrieved September 1, 2010, from http://www.uwstout.edu/static/lib/thesis/2009/2009messnera.pdf

Mocker, D. W., & Spear, G. (1979). Needs assessment. In P. D. Langerman & D. H. Smith, (Eds.), Managing adult and continuing education programs and staff. Washington, DC: National Association for Public Continuing and Adult Education.

Monette, M. (1977). The concept of educational need: An analysis of selected literature. Adult Education, 27, 116-127.

Monette, M. (1977). Need assessment: A critique of philosophical assumptions. Adult Education, 29, 83-95.

Pennington, F. C. (1980). Needs assessment: Concepts, models, and characteristics. In F. C. Pennington, (Ed.), Assessing educational needs of adults (New Directions for Continuing Education, No. 7). San Francisco: Jossey-Bass.

Rodriguez, S. R. (1988). Needs assessment and analysis: Tools for change. Journal of Instructional Development, 11, 23-28.

Rossett, A. (1987). Training needs assessment. Englewood Cliffs, NJ: Educational Technology Publications.

Rossett, A. (1990). Overcoming obstacles to needs assessment. Training, 27(3), 36-41.

Scissons, E. H. (1984). Needs assessment in adult education (Brackhaus, 1984): A reaction. Adult Education Quarterly, 35, 105-108.

Stuffflebeam, D. L., McCormick, C. H., Brinkerhoff, R. O., & Nelson, C. O. (1989). Conducting educational needs assessment. Boston: Kluwer Nijhoff Publishing.

Thomas, K.J. & Kellerman, D.K. (1995). Ask a silly question, get a silly answer: Designing effective questionnaires for needs assessment instruments. Performance and instruction, 34(4), 4-6

Zemke, R., & Kramlinger, T. (1982). Figuring things out: A trainer's guide to needs and task analysis. Reading, MA: Addison-Wesley.

________________________

Additions or corrections are gladly welcomed: rogerhiemstra@gmail.com

September 1, 2010