Page 78

Chapter Six

Experimental Research

Huey B. Long

As adult educators investigate increasingly complex issues, they are likely to use different parts of the descriptive-correlational-experimental research paradigm to study different kinds of questions. And as descriptive studies become more sophisticated, along the lines suggested by Dickinson and Blunt in their chapter, they should yield substantial information about significant relationships among those elements important to adult educators. This development, too, should stimulate greater use of the experimental method, as we attempt to clarify those relationships.

The research design described in this chapter is that labeled the true experimental design by Campbell and Stanley (1963), to distinguish it from a variety of other designs that will be briefly identified later in the chapter. Although this method is perceived in

Page 79

certain circles as the highest form of scientific inquiry, I am not prepared to accept a hierarchical ordering of research methods, since I believe a variety of approaches is desirable for expanding knowl­edge. Hence, I regard the experimental research design discussed here as, but one available means of testing claims against evidence for the purpose of progressively approximating reality or truth.

Basic Logic and Mill's Canons

Experimental research is enhanced by an awareness of the relationships among logic, Mill's five rules of experimental research, and hypothesis testing. An understanding of logic is necessary in designing a theoretically sound experimental research design. The researcher analyzes and synthesizes theory until he explicates a proposition. That proposition may be defined as a truth statement or a logical consequence. The primary consideration in judging such a proposition is whether or not it conforms to the canons of logic for determining what is true and what is false (Dubin, 1969). Two kinds of propositions have been discussed in the literature-cate­gorical and hypothetical (Dubin, 1969).

The most common form of propositional logic is concerned with establishing the class, or set, of which certain elements are members. The classical categorical syllogism illustrates this kind of analysis, and its use is frequently referred to as the deductive method. When one reasons deductively, one holds that what is true of all instances of a class must also be true of any single instance that comes within its limits. Therefore, one tries to demonstrate that a particular instance under consideration logically falls within the instances of an entire class. To accomplish the deductive objective, one uses the device known as the syllogism.

A syllogism provides a logical means of determining the validity of a particular conclusion or fact. It is an argument of three propositions. The first two statements are premises and furnish the grounds for the conclusion, the third or final proposition. Four kinds of syllogisms have been identified: alternative, cate­gorical, disjunctive, and hypothetical. Here is an example of each kind:

Page 80

1. Alternative

Either I will work all day, or I will not finish my work.

I will not work all day.

Therefore, I will not finish my work.

2. Categorical

All humans are mammals (major premise).

She is a human (minor premise).

Therefore, she must be a mammal (conclusion).

3. Disjunctive

It is not the case that one is both a good man and a good politician.

He is a good man.

Therefore, he is not a good politician.

4. Hypothetical

If one jumps into a pool of water, he will get wet.

He jumped into a pool of water.

Therefore, he got wet.

Each kind of syllogism is characterized by and labeled ac­cording to the type of proposition in the major premise. The hypothetical syllogism is especially useful in the search for scientific laws by means of stating and testing hypotheses (Van Dalen, 1973). The hypothetical syllogism requires that the investigator have enough knowledge of an event to discard irrelevant bits of informa­tion, to select previously unconnected facts critically, and to com­bine such facts in a way that implies a conclusion. Deductive reasoning provides a structure for the organization of premises into patterns that provide evidence for the validity of a conclusion.

Even though they differ in appearance and context, syl­logisms and statistical procedures share the common functions of classification and inference. For example, statistical procedures are useful to the researcher in determining the appropriate set from which a statistic is derived. Tests of significance are concerned with locating the universe of which a specific statistic is a member. Thus, the means of two samples can be compared statistically to determine the probability that they are from different populations or sets. If the difference between the means exceeds specified limits, then the

Page 81

difference is said to be significant statistically and the two samples are considered to be from different sets.

A theoretical-hypothetical proposition is specifically con­cerned with a theory in operation, usually with the values of units or concepts in that theory. Such a propositional statement is a pre­diction, because it tells what must be true about the theory in operation if other elements of the theory are known. Usually the pro­position states the values of two critical theoretical units or concepts in the classic "if-then" format. For example: "If an individual has reached a high level of education, then he will participate in a learning activity." This proposition states that a positive value of education is associated with a positive value of learning. Pro­positions derived from theory provide the basis for hypothetical statements. And every proposition is potentially capable of generat­ing a large number of hypotheses. More is said about hypotheses after a brief look at Mill's canons of experimental research.

The conduct of experimental research is clarified by an understanding of the five methods of John Stuart Mill (Hillway, 1974), whose analysis of experimental research generated four major principles, to which a fifth is added by combining the first two. They are useful guides in the development of a research design, but they should not be dogmatically accepted as rigid rules that must be applied in all cases. These five are (1) the method of agreement, (2) the method of difference, (3) the joint method, ( 4) the method of residues, and (5) the method of concomitant variations. According to the principle of agreement, if the phe­nomena leading up to a given event always have only one factor in common, that factor is probably the cause of the specified event. Or when the idea is expressed in the negative, it may be observed that in the absence of a specific factor (X), no combination of phenomena (A + B + C) can be the cause of the identified event (E). The method of difference applies if two or more sets of cir­cumstances are identical except for one factor, and if a specific result occurs only when that factor is present; in this case the factor in question may be the cause of the result. Stated negatively, no factor can be the cause of a phenomenon if the phenomenon occurs in the absence of that factor. The joint method, which in­-

Page 82

corporates both of the first two methods, perhaps generates more reliable results than either of its components used independently. According to this third method, if the conditions of agreement and the conditions of difference are met, the cause of a given event should be fairly evident (Hillway, 1974). The third principle then requires that one factor common to all instances in which the given phenomenon occurs is present and that the phenomenon never occurs when that particular factor is absent.

Examples of the application of these principles in adult education research are readily available. For instance, Buckholdt and Ferritor (1974) report an investigation that illustrates the idea of the method of agreement. Their study was intended to determine whether the use of a token reinforcement procedure would change two specific behaviors of adult students. And their data suggest that this procedure did effectively explain the observed behavior change. The second method is illustrated by the work of Cole and Glass (1977), who set up an experiment to explain the difference between two groups in terms of a specific variable. They developed an experimental group and a control group to determine how students' participation in program planning affected their achieve­ment, retention, and attitude. The difference between the two groups was subsequently explained by the student-participation variable.

Boshier's study (1975) of the effect of a behavior modifica­tion program in a graduate program was designed in such a way as to illustrate the joint method of agreement and disagreement. This canon requires at least two groups with only one circumstance in common and at least two instances in which this circumstance does not occur in order that the two groups will have nothing in common except the absence of that circumstance. Boshier met this requirement by manipulating his sample so that two subgroups each experienced the "treatment" and each served as a control at some point during the study. As a result, Boshier could explain his findings in the presence or absence of the treatment condition. The following diagram illustrates the joint method of agreement as used in Boshier's study. The symbol (+) indicates the presence of the desired behavior; (-) indicates the absence of the desired behavior.

Page 83

In both instances where the treatment was present, the desired behavior occurred; in both instances where the treatment was absent, the desired behavior failed to occur.

The fourth method, the method of residues, is based on the recognition that some problems cannot be solved by the application of the first three principles. The method of residues may be expressed as follows: "Remove from any phenomena such action as is known by previous analysis to be the effect of certain antecedents, and the residue of the phenomena is the effect of the remaining antecedents" (Cohen and Nagel, 1934, p. 264). The discovery of the planet Neptune is cited as an example of the application of this method. The movements of Uranus had been studied with the help of Newton's theories. Accordingly, the orbit of Uranus was projected on the assumption that the sun and the planets within Uranus's orbit were the only bodies determining its motion. But calculations revealed that Uranus's observed position was different from the theoretical position. It was therefore assumed that the differences could be accounted for only by the gravitational action of another planet. The perturbations in the motion of Uranus were then used to calculate the position of the hypothetical planet. Neptune was subsequently discovered in the vicinity of the place calculated for it (Cohen and Nagel, 1934).

According to the method of concomitant variations, when two phenomena consistently vary together, either the change in the first is caused by the variation in the second or both are changed by some common factor. To illustrate the fifth method, Mill cited the relationship between the moon and earth's tides. The factors, in this case, the moon and tides, are beyond man's ability to manipulate, and thus the methods of agreement, method of differ­ence, and the joint method cannot be used to examine the relation­ship. However, the method of concomitant variation provides a framework wherein the variations in tides may be compared with the variations in the moon's position relative to the earth. Such

Page 84

observation suggests three possible conclusions: (1) the tides affect the motion of the moon, or (2) the changes in the moon's relationship to the earth affect the tides, or (3) the variations in the moon's position and the changes in the tides are caused by some element affecting both.

Mill's fourth and fifth principles have become preferred bases of analysis in many of the social sciences and in education. Modem developments in statistical analysis, such as factor analysis and multiple correlations, with corollary developments in computer science and technology have increased the utility of the latter principles. Nevertheless, four limitations restrict the use of Mill's principles in modem research. First, they are designed for qualita­tive dichotomous variables, those which are either usually present or usually absent. Second, they are based on the assumption that no interaction occurs among the variables. Third, they are not efficient, because study is limited to one causal variable at a time. And finally, they fail to take account of random variations in measurement. The simple logic of the principles is both an ad­vantage and disadvantage. The simplicity is attractive, but it some­times fails to cover adequately complex interactions that exist in reality. For example, causal relations are seldom clear. Furthermore, the qualitative form of the methods may not allow them to be applied effectively to quantitatively determined variables. Modern statistical techniques seem to be related only to the use of Mill's fifth principle, that of concomitant variations, which is seen by some as the logical precursor of modern correlation procedures.

Despite these limitations, however, Mill's canons are in­structive in developing an attitude toward the eliminative nature of the experimental research design. The eliminative or disconfirm­ing characteristic of experimental research seems to be closely re­lated to the use of hypothetical statements that lend themselves to testing within the logical framework explicated earlier. Conse­quently, the logical principle of the excluded middle is that any proposition must be either true or false (Phillips, 1971), and the convention of testing a null hypothesis in order to prove it true or untrue serves the elimination needs of experimental research very well.

Page 85

Hypotheses

Hypotheses are useful devices to all kinds of individuals seeking solutions to problems. The mechanic, housewife, and carpen­ter may frequently employ informally stated hypotheses to help organize their thinking in order to solve a problem. When used in experimental research, hypotheses are often formally stated; that is, they are expressed as statements or questions in order to explain, describe, or predict conditions or events that have not been confirmed by facts. The organizational properties of hypotheses require the researcher to restrict the scope of the question under investigation. Through the use of hypotheses the researcher brings together isolated data into some unifying framework. Finally, hypotheses provide the researcher with some direction in identifying relevant information.

Hypotheses may be derived from theory--deductive--or they may be ad hoc, inductive hypotheses. Different scholars prefer one over the other. Many of the hypotheses appearing in adult education research are ad hoc. Such hypotheses are frequently associated with the situation in which an investigator discovers a measuring device, proceeds to use it, and then asks, after identifying some statistical relationship between the values measured, What theory can I relate these findings to? A sustained inductive pursuit of knowledge based on empirical conclusions may be fruitful if it is carried out with reasonable logic. The University of Chicago School of Sociology was characterized by this kind of research be­tween the two world wars (Dubin, 1969).

Hypotheses have been described as bold, descriptive, ex­planatory, and substantive; a fifth category contains research hypo­theses. So-called "bold" hypotheses are those that go beyond avail­able facts in an effort to answer a question. Descriptive hypotheses are intended to answer such questions as what, when, who, and where. In contrast, explanatory hypotheses seek to provide answers to how and why concerns. A substantive hypothesis is a conjectural statement that may be derived from the theory being tested. The substantive hypothesis is not testable until each of its terms is de­fined operationally. Statistical research hypotheses are commonly tested against a null hypothesis, which states that there is no relation

Page 86

between the variables of a problem. The null hypothesis is a state­ment of chance expectation. It is never completely disproved and its retention or nonrejection is always tentative.

Invalid hypotheses may also prove to be useful in the process of discovering the correct solution to a given problem. In­deed, most problems of consequence have not been resolved by the formulation and testing of a single hypothesis. Theory building usually proceeds slowly through a process of eliminating a series of hypotheses.

To be useful, a hypothesis must have certain characteristics. First, it must be testable. Valid hypotheses are those that have been tested and confirmed by evidence. The difficulty of confirming a hypothesis depends to a large degree on the precision and clarity of the statement. Second, a hypothesis is expected to account for all the facts it purports to cover. Third, it should be able to explain what is claimed for it. And finally, a hypothesis should be internally consistent (Bledsoe, 1972).

The null hypothesis is formulated and tested according to the following procedure (Bledsoe, 1972, p. 56) :

1.      Report the null hypothesis.

2.      Select a statistical technique based on the test that most closely approximates the conditions of the research and whose measure­ment requirements are met by the research.

3.      Select the significance level for rejecting the null hypothesis.

4.      Specify the sample size.

5.      Find or assume the sampling distribution of the statistical test.

6.      On the basis of the chosen statistical technique, the risk of Type I error, the sample size, the sampling distribution, and the alternative hypothesis, define the region of rejection.

7.      Determine the value from the statistical test. If the value is in the region of rejection, the null hypothesis should be rejected.

8.      Rejection of the null hypothesis results in the acceptance of the alternative hypothesis.

Researchers are constantly alert to two kinds of erroneous conclusions that may result from their work: accepting a null

Page 87

hypothesis that should be rejected, and rejecting a null hypothesis that should be accepted. Because of the implications of such errors, experimental designs are constructed to reduce threats to internal and external validity as much as possible. Concern about internal validity is based on the need to be able to observe that any change in the subject can be explained by the treatment. In contrast, concern about external validity arises from the need to be able to report the populations, settings, treatment, and variables to which the findings can be generalized.

Different kinds of extraneous variables that threaten internal validity have been identified by Campbell and Stanley (1963). Those variables are due to a number of factors:

1.      History--the unanticipated and uncontrolled events that occur between the first and subsequent measurement, in addition to those events due to the independent variables.

2.      Maturation--biological processes operating as a function of the passage of time.

3.      Testing--how the first test affects people's scores on subsequent tests.

4.      Instrumentation--changes in the calibration of a measuring de­vice or scale and changes in observers that may influence changes in scores.

5.      Statistical regression--the possible effects of the selection process.

6.      Selection bias--the impact of sampling procedures.

7.      Experimental mortality--the differential loss of respondents from comparison groups.

8.      Selection--maturation-interaction-a problem in certain multiple­ group designs.

9.      Instability--the instability or unreliability of measures.

External validity or representativeness may be threatened by (1) the reactive or interaction effect of testing; (2) the interac­tion effects of biases in sample selection and the experimental vari­able; (3) the reactive effects of experimental arrangements; (4) multiple treatment interference; (5) the irrelevant responsiveness of measures; and (6) the irrelevant replicability of treatments.

Page 88

Kinds of Experimental Designs

Campbell and Stanley (1963) cogently describe the varia­tions available in designing an experimental research project. Tuckman (1972) and Kerlinger (1964) also provide insights on this subject. Reference to those publications should be a normal prelude to planning an experimental research project. Because of the impact of Campbell and Stanley's classification of experimental research designs in educational research, their main ideas are sum­marized below. They describe sixteen designs divided into four classes: preexperimental (designs 1-3), true experimental (4-6), quasi-experimental (7-10), and correlational and ex post facto designs (11-16).

The three pre experimental designs have severe limitations. The one-shot case study is really a descriptive analysis of a group following some treatment. There is no control group or pretest. The one-group pretest-posttest design also fails to provide a control group. Furthermore, the experimenter has no control over the effects that are due to history, maturation, testing, and instrumenta­tion; and statistical regression may contribute to another expla­nation of the findings. In the static-group comparison design, an "untreated" group is compared with an experimental group. The results are confounded because of the inability of the researcher to determine that the two groups are equivalent except for the treatment.

The three true experimental designs meet all the first eight threats to internal validity identified by Campbell and Stanley. In contrast, the quasi-experimental designs are characterized by partial control by the researcher. He may have full control over the "when" and "who" measurement elements, but lacks full control over the scheduling of the treatment, including the timing of the treatment as opposed to the timing of the measurement and the random assignment of subjects.

Experimental research is distinguished from other kinds of research by the use of designs that are primarily intended to elimi­nate rival hypotheses or explanations of cause and effect (invariant relations). In addition, the experimental design is characterized by control groups, randomized samples of subjects (elements), and

Page 89

manipulation (treatment) of independent variables in order to con­trol pertinent factors as much as possible. Furthermore, the independent and dependent variables are frequently operationally defined so they can be described according to a given set of criteria.

A strong experimental design must be carefully conceived and rigorously monitored and controlled because each of its characteristics (control group, randomization, manipulation, and opera­tional definition) affects the validity of the findings. Weaknesses often develop in one of the following ways: the experimenters fail in their efforts to control the environment of the control group adequately; they fail to randomize the experimental group; the manipulated variable is unsatisfactorily handled; or the key vari­ables are not sufficiently operationalized for accurate quantification. Boyd's chapter in this volume points out related issues.

The characteristics of the experimental method (control, randomization, manipulation, and operational definition) are re­lated to two crucial elements of the experiment: (1) the experiment must focus on some specific change that takes place, whether in­troduced by the investigator or by the environment; and (2) the experimenter must be able to achieve a high degree of control over the experimental condition or to understand the situation well enough to make allowances for relevant factors (Phillips, 1971).

There are basic differences between the class of true experi­mental designs and the research methods discussed in earlier chapters. For example, the survey method seeks to generate information on the status of a phenomenon, whereas the experimental method is generally concerned with determining the existence of a phe­nomenon, the cause of the phenomenon, under what kinds of' conditions the phenomenon under study may exist, and how a phenomenon may vary systematically in quantity or quality with another phenomenon. Using the grounded-theory method one may attempt to determine similar things, but one would' use the inductive approach rather than the deductive approach of the experimental method, and one would be generating hypotheses instead of testing them. The historical method also differs from the experimental method, principally with respect to the researcher's control over the subjects. For example, the experimental researcher usually manipu­-

Page 90

lates some variable in the environment so that the first condition is different from the second condition in a planned and controlled way. The researcher's ability to control selected variables and to account for differences between two groups based on that control is central to the experimental design. But the historian cannot con­trol the subjects or the environment, which have only historical visibility and cannot be manipulated.

Thus, the experimental method cannot be easily con­fused with other research methods unless one uses the ex post facto or quasi-experimental designs described by Campbell and Stanley (1963). The user of these designs lacks control and usually seeks to explain changes that occur without the investigator's manipulation.

Handling the Critical Elements: Examples

A few examples of how the critical elements-control, ran­domization, manipulation, operational definition, and the testing of hypotheses-have been handled in adult education studies should prove to be instructive. Because I assume that articles appearing in Adult Education are fairly representative of research in the field, I have chosen such articles for illustrative purposes. But since these articles are frequently summarized reports of larger projects and since it is difficult to determine whether the weaknesses noted are products of the report procedure or part of the design, I have not identified the articles and their authors.

The experimenter has available several procedures for con­trolling the assignment of subjects to appropriate groups. These procedures include administrative selection procedures and statistical techniques. The physical control represented by the administrative selection procedures appears to be generally preferred. But in situa­tions where such control is not possible, the experimenter must use a statistical treatment, such as a partial correlation technique, to hold one variable constant.

One study reported in Adult Education achieved control through an administrative procedure. In this study, designed to measure differences in students' achievement produced by two instructional methods, among other things, the control was provided

Page 91

by limiting all enrollment to the experimental and control courses. Thus, students registered in 1966-67 in an experimental class (un­knowingly) and in 1967-68 in a control class. Courses at two levels were used; all lower-level courses were experimental one year and control the second year. The opposite sequence was followed in the higher-level courses. There were 322 students in the experi­mental group and 308 in the control group. The outcomes were analyzed according to sex, credit-hours attempted, semester grade­point average, cumulative gradepoint average, and pretest scores.

Up to this point, the approach appears to be proper. But another element that needs to be controlled--the similarity of treatment--was apparently not carefully handled. Although each of the nine instructors was to teach both kinds of sections, experi­mental and control, only five actually did so. And the article does not discuss how the instructors were prepared for the experiment. Their participation, personality, and related classroom interactions thus appear (from the available data) to have been loosely controlled.

This study illustrates how one can provide control in some areas only to lose it in others. Because the instructor variable was apparently poorly controlled, any difference in achievement could be explained as well by the instructor variable as by the experi­mental treatment. Thus the researcher must examine all elements of any control efforts to ensure the validity of the collected data and the corresponding analyses. Many research problems also require statistical control to make up for administrative deficiencies.

The manipulation of variables is illustrated by a study in which the investigator manipulated several variables at strategic points. First, he obtained one set of data by asking the subjects, on whom he exerted no overt pressure, to respond to a written instru­ment. Later, he obtained oral responses to identical stimuli when the subjects were under group pressure. He also manipulated the in­formation presented so that each subject responded to two different classes of stimuli and two different kinds of content in each class. Manipulating one variable, such as the instructional method, is usually not overly difficult. But manipulating several variables, such as the kind of information, the group size, the kind of group, and personality characteristics, becomes more difficult. Statistical as well

Page 92

as administrative manipulation is usually required in such complex experiments.

Another study is instructive in the use, or lack of use, of operational definitions. According to the article, the central hypo­thesis was that the "personality characteristics of participants in management development programs influence the success of the teaching methods used." The investigator then predicted certain results, such as, "The case-discussion method would be more suc­cessful than the lecture method for more extroverted people." Although several key words in this sentence are basic to the fundamental objectives and design of the study-"case discussion method," "more successful," "lecture method," "extroverted," and "people"--the author failed to define any of those terms opera­tionally. By studying the procedures, the reader can deduce how extroversion and success were measured and who the people were. However, such failure to define key terms operationally puts a heavy burden on the reader and may affect the interpretation of the results. Even though differences between the lecture technique and case-discussion technique were central concerns, they were not defined. Clearly, operationalism is extremely important in experi­mental research. Phillips (1971) points out that operationalism is constructive in calling attention to the importance of clarity, pre­cision, and definition.

The stating and testing of hypotheses, the last of the "critical elements" discussed in this section, are not handled consistently in the experimental studies reported in Adult Education. Hypotheses have been referred to as "questions put to nature" and as the basic tools of experimental research. And thus most experimental studies are designed to test one or more hypotheses. However, in these articles the number and quality of the hypotheses vary widely. For example, one study contained five hypotheses stated in the null form. The writer also provided the acceptable level of probability. In contrast, another article provided only a substantive hypothesis without indicating the acceptable level of probability.

Only a few adult education dissertations demonstrate the use of the experimental method. For instance, a review of twelve abstracts provided by ERIC revealed few salient characteristics of that method. Five of the dissertations (Bodenhamer, 1964; Dollins,

Page 93

1967; Etter, 1969; Grotelueschen, 1967; Lupton, 1967) dealt with instruction and three with attitudinal questions (Peters, 1968; Rhyne, 1968). Each of the others took up a different kind of problem: comparing the musical pitch discrimination of adults and children (Maltzman, 1964) ; examining the relationship between rates of parental participation in education (Majure, 1972); group size and divergent thinking (Shah, 1966); and the information-processing capacity of persons of different ages (Car­penter, 1967). These dissertations are cited not as examples of good or poor design, only as examples of the kinds of dissertations written by adult education students. The reader should review the studies to determine their strengths and weaknesses.

As a consulting editor for Adult Education, I have had frequent opportunities to assess research reports from the field. Articles submitted for consideration often suffer from one or more weaknesses with respect to the critical elements of experimental research: (1) External validity is often weakened by the sample selection procedures. So-called random samples often fail to provide all potential subjects an equal chance of selection. (2) Design difficulties frequently include a lack of control over additional variables (other than treatment), such as education, exposure, maturation, or unique experience. (3) Interpretation error is sometimes in evidence; the experimenter fails to use an appropriate statistical treatment, goes beyond the data, or applies findings to an inappropriate population. One example was provided by a recent manuscript wherein the data called for multivariate an­alysis, but the experimenter used only univariate analysis. An ex­ample of going beyond the data was provided by a manuscript in which the writer arrived at conclusions that did not appear to be supported in the reported findings and that were not related to the design.

Explanations for Limited Use of Experimental Method

Some fields of study and practice, such as psychology, physics, and chemistry, are especially suited to the use of the experimental method. Others-sociology, history, and education, for instance-are not always so amenable to the method. But because

Page 94

of the prestige of such fields as psychology and the natural sciences in academic circles, researchers in other fields are constrained to generate experimental studies as a kind of reputation-maintenance act.

Thus it is no surprise that adult educators, who have at best a marginal status in higher education power circles, have sometimes felt they had to defend their intelligence, scholarship, and analytical rigor by trying to use the experimental method. Knowles (1973) has written pointedly on this situation, and little profit would result from expounding further. However, it is important to observe that adult educators should not feel compelled to select the experimental method just because it is the "in thing." Like any other researchers, they should select the method most appropriate to the problem they are studying. And in fact, they have not felt so compelled, if we are to judge by the assertions of four major sources (DeCrow and Loague, 1970; Dickinson and Rusnell, 1971; Grabowski and Loague, 1970; Long and Agyekum, 1974), who indicate that adult educators have used the experimental research method much less frequently than other methods.

One explanation for this finding may be the humanistic orientation of most adult educators, which may be somewhat in­consistent with the use of the experimental method. Humanists hold that the individual is unique and that efforts to reduce human differences to a quantified continuous scale present logical and philosophical problems. Some adult educators may believe that measures of central tendency fail to measure any individual, and as a result, such measures are not useful. When such limited use is compared with the possible costs of such data, such as the cost of violating the researcher's philosophy regarding the integrity of the individuals tested or treated in an experimental design, the costs may exceed the reward for the researcher. Hence, the experimental method is seldom used by people with such beliefs.

The developmental stage of the field, at which the problems of practitioners are emphasized, may provide a second explanation. It is possible that as the field changes, adult educators will more often find the experimental method appropriate for the sorts of issues they want to investigate. For example, many of the studies concerning participation have been based on descriptive surveys. But

Page 95

to explain how and why participation occurs, investigators must use experimental methods.

Somewhat related to both of the foregoing possibilities is a third explanation: the philosophical orientation and professional concerns of practitioners. During the past half century adult educa­tors have stressed pragmatic and often programmatic matters. As a result, they have been highly interested in describing the dynamic field, whose scope and character appear to require continuing reassessment. Similarly, many of their programmatic concerns appear to be more related to descriptive research methods than to experi­mental methods, especially when cast against a humanistic orienta­tion. For example, how does one establish or prove the need for continuing education or assess the efficacy of certain administrative procedures by using an experimental design? Until basic questions such as these are more fully understood, adult education may employ experimental research less than other methods.

Another explanation is the kind of research training adult educators receive. Many are not taught how to use experimental designs in adult education research. Most research-design courses the graduate student takes are based on examples and problems more closely related to public schools than to adult education, and sometimes they concern even less meaningful (to adult education students) subjects. As a result, the graduate student in adult educa­tion often does not learn about the variety of research models. A related factor is that many graduate advisors are themselves in­experienced in experimental work. This problem can be overcome only as more professors of adult education become competent at experimental research.

Difficulties in formulating theory in adult education may provide the fifth explanation. In the fields in which the experimental design is frequently used, physics and chemistry particularly, theory is highly developed. In contrast, adult education and its generic area, education, have few theoretical foundations. The experimental method as a hypothesis-testing procedure is especially well suited to theoretical fields. Conversely, one could argue that the less theoret­ical fields are better served by other research designs.

Problems related to the testing of adults provide a sixth potential explanation for the limited use of the experimental method

Page 96

in adult education. Any experimental design that uses human sub­jects contains a particular set of problems. The problems of control may be described as a continuum on which inanimate subjects present the least difficulties, animals more, children even more, and finally, adults most. Why? For one thing, obtaining permission and coopera­tion from adults in experimental studies is sometimes difficult. Often, such subjects are a fortuitous voluntary sample, are compensated, and may come from an institutionalized population. All three possibilities have some potential impact on external validity or encourage the selection of another research method. Finally, many adults may find it unrewarding or overly time-consuming to par­ticipate in an experimental study. As a result, they may depart from the scene physically or psychologically. Even in those instances when adults are not strongly opposed to the activity, their relative sophis­tication may motivate them to seek and discover the intent of the experimenter and to behave in such a way as to affect the outcomes. Consequently, the validity of the findings may be low.

A study cited earlier (the one involving the use of nine instructors each year for two years) also illustrates another problem: research that requires an extended time. In this case, the research plan was aborted when four of the nine instructors left before the study was completed.

Factors Favoring Use of the Experimental Method

Whatever the explanations for its limited use at present, the experimental method should prove more attractive to adult educa­tors in the future for four main reasons.                         .

First, the field has, I believe, reached a state of development that requires the testing of hypotheses. Although the dynamic character of adult education will probably necessitate further efforts to describe and define its characteristics and boundaries, we also need to examine deeply specific topics as a means to better understanding and increased knowledge. And such in-depth study will require manipulation of some variables.

The special subjects needing examination include adult learning, instructional strategies and modes, delivery systems, and learner motives. The literature is replete with case analyses and

Page 97

descriptions of teaching techniques, but almost barren of experi­ments measuring interactions among learning styles, methods and techniques, and learner motives.                                            .

The increased use of the experimental method in adult educa­tion has important implications for the theoretical development of the field, since that growth may very well depend on how much scientific inquiry contributes to the discovery of useful ideas and to their evaluation. Adult educators are interested in strategies that will help to develop ideas contributing to scientific progress. And because not every idea that is generated is worthwhile, they are also interested in evidence and judgments concerning the value of ideas (Phillips, 1971). Tests (experimental research) within the latter context sug­gest the direction of further inquiry. Evidence presented in this evaluative context not only may indicate that the ideas are poten­tially fruitful, but may also suggest what kinds of additional data are urgently needed. Conversely, if evidence indicates that the ideas are incorrect, the experimenter is driven to make discoveries.

The experimental method and theory formulation are closely related. The experimental design is perhaps of greatest value when the investigator has developed a theory and wishes to validate it. If this perception is correct, efforts to develop theory in adult educa­tion should be followed by efforts to validate that theory. Thus, the advancement of a validated theoretical base in adult education re­quires the execution of experimental research.

Economics is a second factor favoring the use of the experi­mental research methods by graduate students in adult education. If the researcher can readily identify and secure access to a group of adults, the research with these people may be less expensive in time and money than a study using historical or grounded theory methods. However, the costs will vary among the research methods as well as within any method, according to the design and nature of the study.

The relative ease of reporting the findings of an experimental study also favors an increase in such work. As indicated previously, a characteristic of the experimental method is the generation of quantified data that are often used to test hypotheses. In most dissertation studies the decision to reject a hypothesis is easily made following statistical analyses. That decision, in turn, usually leads directly to conclusions that can be supported by the quantified data.

Page 98

There is a certain degree of security in such data that may be es­pecially appealing to the graduate student. Conclusions in studies based on other methods are not always so clean-cut. For example, the historian's data may always be subject to the question whether the sources are biased or whether the researcher was biased in select­ing and interpreting the material. Similar questions of interpretation also apply to other research methods.

A warning may be required here, however. There is a rather close relationship among hypotheses, theory, and explanation. Scientific explanations usually describe the relationship between at least two variables. These explanations are based on the evidence for such relationships (Phillips, 1971). Explanation is thus hindered or assisted by the evidence generated and the theoretical framework guiding the development and implementation of the research design. Therefore, although the presentation of findings may be relatively easy, suitable explanation and interpretation may be more difficult.

A fourth factor favoring the use of the experimental method is the clarity of the design. The experimental design is usually easy to describe. The graduate student can tell in parsimonious terms how he will treat group A as opposed to group B and how the quantified differences between the two groups will be analyzed. The task of writing a dissertation based on the experimental method may therefore be less difficult than writing one based on other designs.

Thus, there are several acceptable explanations for the limited number of adult education dissertations based on the experimental design. And there are also a number of rather good reasons why graduate students in adult education may wish to consider an experimental design.

Need for Experimental Research

Experimental research in adult education does not appear to suffer more from design weaknesses than do other forms of research. So I want to point out once again that adult education will probably be best served by the use of a variety of techniques. The selection of, the appropriate method needs to be determined by several factors. For example, if independent variables can be manipulated or con­-

Page 99

trolled, the experimental method might be appropriate. If only highly select volunteer subjects are available, or if manipulation is impossible, the experimental method is probably inappropriate.

The number of experimental studies in adult education has gradually increased in recent years. The increase parallels developments occurring among the practitioners, as well as changes in adult educators' research competence. These kinds of modifications will probably speed up too, thus contributing to further use of the ex­perimental method. And the development of inferential statistics and multivariate techniques of analysis, accompanied by the develop­ment of computer technology, including packaged computer pro­grams, may help overcome some of the problems of control and analysis.

Unfolding areas in adult education call for the use of all available research skills and methods. If the field is to develop in substance as well as size, such variety is imperative. Adult educators are called on to be realistic in their assessment of research needs. Careful, rigorous, ethical experimental research is necessary (in combination with other kinds of research) if adult education is to develop. Descriptive research can go only so far before continued application becomes redundant. Unprofitable redundancy may be prevented if there is close interaction among the research methods and mutual respect among the practitioners. Thus, graduate stu­dents, practitioners, and professional researchers in adult education are encouraged to carefully examine the experimental method in their attempts to extend knowledge of adult education.

January, 2005

Return to the top of this page.

Return to the first page.

Go to the Preface, Contents, Chapter One, Chapter Two, Chapter Three, Chapter Four, Chapter Five, Chapter Seven, Chapter Eight, the Epilogue, References, or the Index