Development of an Instrument for Identifying Groups of Learners
Gary J. Conti, Oklahoma State University
Rita C. Kolody, Medicine Hat College
Abstract
ATLAS (Assessing The Learning Strategies of AdultS) has been developed to quickly identify the learning strategy group to which the respondent belongs. The validation process involved the use of past learning strategy studies and multivariate statistical procedures.
Introduction
The concept of lifelong learning suggests that adults need to acquire a variety of process skills to enable them to address their constantly changing learning needs. This approach to learning has stimulated interest in the learning strategies that an individual elects to use in order to accomplish specific learning tasks. The Self-Knowledge Inventory of Lifelong Learning Strategies (SKILLS) has been developed to measure learning strategies in the areas of metacognition, metamotivation, memory, critical thinking, and resource management (Conti & Fellenz, 1991). In this instrument, each of the five areas consists of three specific learning strategies: Metacognition-- Planning, Monitoring, and Adjusting; Metamotivation--Attention, Reward/Enjoyment, and Confidence; Memory--Organization, Use of External Aids, and Memory Application; Critical Thinking--Testing Assumptions, Generating Alternatives, and Conditional Acceptance; and Resource Management--Identification of Resources, Critical Use of Resources, and Use of Human Resources.
Since its development in 1991, numerous studies with diverse populations have been conducted using SKILLS. Collectively, these studies have found that selected demographic variables are not useful in discriminating among different groups in their learning strategy usage. However, the same studies have consistently found that distinct groups of learners exist when they are identified by the pattern of the learning strategies which they use. Together these two types of findings indicate that the patterns of learning strategy use cut across the variables such as gender and age which are typically used to group people in educational studies. Instead, the distinct groups which are inherent among people are independent of demographic labeling. Anyone can be in any group. Placement in a learning strategy group is dependent upon the strategies one chooses to use rather than being predetermined by other factors. Despite this independence, however, there are clear patterns in the learning strategies which people have a propensity to use when initiating a learning activity.
Therefore, the purpose of this study was to develop an instrument for identifying the pattern of learning strategy usage of learners and to establish the validity for this instrument. The goal was to produce an instrument which was easy to administer, which could be completed rapidly, and which could be used immediately by both facilitators and learners. The instrument which was created has been entitled ATLAS (Assessing The Learning Strategies of AdultS). ATLAS utilizes a flow-chart design. Items are printed on colored cards which are one-quarter sheets of a standard-sized, 8.5" x 11" page. Sentence stems, which are in the top box on the page, lead to options in other boxes which complete the stem. Connecting arrows direct the respondent to the options. Each option leads the respondent to another box which either instructs the respondent to proceed to another colored card or which provides information about the respondent's correct group placement. Five colored cards constitute the entire packet for the instrument. Depending upon reading level, ATLAS can be completed in approximately one to two minutes.
Construct Validity
Validity is concerned with what a test actually measures; while there are several types of validity, the three most important types recognized in educational research are construct, content, and criterion-related validity (Kerlinger, 1973, p. 457). These may be established in a variety of ways; however, they should be compatible with the overall purpose of the test (Borg & Gall, 1983, p. 275; Van Dalen, 1979, pp. 135-136). Because establishing validity is essential to the credibility of any test and because it involves several steps, "the validation of a test is a long process rather than a single event" (Tyler & Walsh, 1979, p. 29).
Construct validity assesses the underlying theory of the test. It is the extent to which the test can be shown to measure hypothetical constructs which explain some aspect of human behavior (Borg & Gall, 1983, p. 280; Van Dalen, 1979, p. 137). It is the element that allows for the assigning of "meaning" to the test (Kerlinger, 1973, p. 461). The process of establishing construct validity for ATLAS was to synthesize the results of the numerous research studies using SKILLS and to consolidate these results.
Much of the learning strategy research using SKILLS has been coordinated with the Center for Adult Learning Research at Montana State University. Thirteen doctoral dissertations have been completed using the instrument, at least three are currently underway in Montana and Ohio, and another (Uhland, 1995) used the conceptual basis from SKILLS for data gathering. SKILLS has also been used in a nationwide study using American Express financial planners (Conti, Kolody, & Schneider, 1997). The dissertation studies have involved diverse populations in various states and Canada in the areas of two-year college students (Hays, 1995; Kolody, 1997; Kolody & Conti, 1996; Strakal, 1995), the business community (Courtnage, 1998; Gehring, 1997), tribal communities (Bighorn, 1997; Hill, 1992), nursing (Lockwood, 1997), the military (Korinek, 1997; Yabui, 1993), public school administration (McKenna, 1991), students concurrently enrolled in high school and college (Ungricht, 1997), and volunteer leadership (Moretti, 1994). Collectively, these studies have produced a data set of 3,070 cases in which the data were in similar form.
Studies coordinated through the Center for Adult Learning have utilized a similar research design which was recommended by the staff at the center. This design consisted of describing the learning strategy profile of the participants, conducting discriminant analysis to determine if the respondents differed in learning strategy usage in any way on selected demographic variables, and conducting cluster analysis to uncover inherent learning strategy groupings within the sample. Several of the studies involved interviews and focus groups with the various cluster groupings to elicit qualitative data to better describe the groups.
Because it had the most general sample, the study by Kolody (1997) has been viewed by those using the instrument as the most universal, and the findings of other studies have been related to it. In order to check this assumption, a cluster analysis was performed on the entire data set of 3,070 cases. This analysis with an even larger and more diverse sample did not support the assumption that the Kolody study should serve as the conceptual basis for a general instrument for associating an individual with patterns of learning strategy use. Although many of the characteristics of the groups have been similar, the various studies using specific populations have found differing numbers of clusters among the sample: Five clusters--Gehring, Hays, Kolody, Strakal, and Ungricht (1997); four clusters--Bighorn, Courtnage, Korinek, and Lockwood; and three clusters--Conti, Kolody, and Schneider. The cluster analysis of the aggregate data set from the various studies revealed three distinct clusters.
"The key to using cluster analysis is knowing when these groups are 'real' and not merely imposed on the data by the method" (Aldenderfer & Blashfied, 1984, p. 16). Although the use of multivariate analysis of variance or discriminant analysis as a means of performing significance tests on the clusters is inappropriate statistically because of the invariably high significance results (pp. 64-65), discriminant analysis is a useful tool for exploring if a clear process exists which separates the groups (Conti, 1996, p. 71). Therefore, analyses were conducted for five-cluster, four-cluster, and three-cluster solutions using the Quick Cluster program of SPSS. Although the structure matrixes were similar for all three analyses, the discriminant functions produced by each differed greatly in their ability to correctly place learners in their correct group. The correct placement percentage for each solution was as follows: Five clusters--62.5%, four clusters--73.9%, and three clusters--96.1%. Because ATLAS is concerned with correct placement in the groups formed by SKILLS, because it is very accurate, and because it is much more accurate than the other two solutions, the three-cluster solution was selected to serve as the conceptual basis for ATLAS.
Thus, the construct validity for ATLAS was established by reviewing the literature of studies actually using SKILLS in field-based research and by consolidating the similar data from many of these studies. This resulted in the identification of three groups with similar patterns of learning strategy usage. Because of their similarity to groups in the studies which were reviewed, these groups have tentatively been named Navigators, Problem Solvers, and Engagers. The distribution of the respondents among the three groups was relatively equal: Navigators--36.5%, Problem Solvers--31.7%, and Engagers--31.8%.
Content Validity
Content validity refers to the sampling adequacy of the content of the instrument (Kerlinger, 1973, p. 458). For ATLAS, content validity is concerned with the degree to which the items are representative of learning strategy characteristics of the three groups identified in the SKILLS' research. A series of discriminant analyses were conducted to determine the differences between each grouping. At each stage of this analysis, the findings from the structure matrix for the discriminant analysis were used to determine the wording of the items.
The structure matrix of the discriminant analysis for these three groups revealed that the major process that separated the groups related to how each groups sought to accomplish the learning task. The Navigators and Problem Solvers initiate a learning task by looking externally to themselves at the utilization of resources that will help them accomplish the learning. Engagers, on the other hand, involve themselves in the reflective process of determining internally that they will enjoy the learning task enough to finish it. The learning strategies associated with the Navigators and Problem Solvers are Identification of Resources and Critical Use of Resources. Those used more extensively by the Engagers are Confidence and Reward. This process was 96.1% accurate in discriminating between the Navigators and Problem Solvers as one group and the Engagers as another group. Therefore, the items on the first card of ATLAS other than the directions card requires the respondent to choose between these concepts related to how they initiate a learning task.
Since the Navigators and Problem Solvers are grouped together on the first card, a second card is used to separate them. Since the responses are structured in a flow-chart format, the Engagers neither see nor respond to this card. The structure matrix analysis of the discriminant analysis using only those in these two groups revealed that the process that separated the Navigators from the Problem Solvers involved the way they focussed on the learning task. Navigators are much more concerned than Problem Solvers with identifying exactly what needs to be learned and on designing a plan for the learning. In contract, Problem Solvers are more concerned with identifying a variety of solutions for the learning task. The learning strategies associated with the Navigators are Attention and Planning while the Problem Solvers utilize Generating Alternatives. This process was 98.3% accurate in discriminating between the Navigators and Problem Solvers.
Since several members of three groups from the Kolody study collapsed into the Navigators group, an additional discriminant analysis was performed to investigate the structure of this group. This revealed that two subgroups compose the overall group of Navigators. One group (45%) has a strong preference for the use of human resources while the other group (55%) is more concerned with the organization of material into meaningful patterns. While these two subgroups do not constitute a separate group when they are combined with the Problem Solvers and Engagers, they do provide greater clarity into the pattern of learning strategy usage of the Navigators. Therefore, after Navigators are identified as a separate group, they are directed to a card for further uncovering this distinction.
The discriminant analysis on the Navigators, Problem Solvers, and Engagers produced two discriminant functions. Both had high enough eigenvalues (1.34 and 1.09) to be judged useful. Since the strongest function was used for the first card of ATLAS, the second function was used to create an item to check the accuracy of the responses elicited by the items which were based on the first discriminant function. The structure matrix for the second function indicated that each of the groups had one learning strategy with they preferred over the others. These learning strategy preferences were as follows: Navigators--Attention, Problem Solvers--Generating Alternatives, and Engagers--Confidence. After identifying their group placement on the cards based on the first discriminant function, the respondents are all directed to the card which instructs them to choose between these three learning strategies.
Thus, content validity was established by using discriminant analysis to determine the exact pattern of learning strategies used by each group when it was compared to the other groups. Since the three groups were originally identified by a multivariate process, the items were arranged so that respondents follow a track of questions. Qualitative data collected during field-testing to determine the best wording for items revealed that respondents might find options for distinguishing between other groups appealing to them if they saw them. Therefore, the tracks were divided and placed on smaller sheets so that the respondent could only see one item at a time. All cards are kept face down until they are used. Through this procedure, the respondents do not have access to the items that do not apply to them because they have identified themselves as belonging in another track. While ATLAS has only a few items, each item was based on the powerful multivariate procedure of discriminant analysis. Instead of using an approach which involves summing multiple attempts to identify a characteristic, ATLAS uses discriminant analysis to precisely describe the content for each item. The last item serves as a check on the accuracy of the previous items in identifying the correct group placement.
Criterion-Related Validity
Criterion-related validity compares an instrument's scores with external criteria known or believed to measure the attribute under study (Kerlinger, 1973, p. 459). Criterion-related validity was established by comparing ATLAS scores to actual group placement using SKILLS.
Groups of adult learners in Alberta, Montana, and Oklahoma were administered both SKILLS and draft versions of ATLAS. After completing the instruments, comments concerning ATLAS were gathered by means of individual interviews and group discussions. Suggestions were taken into consideration in improving ATLAS. The current version of ATLAS correctly places approximately 70% of the respondents in their corresponding SKILLS group. Focus groups are currently being conducted with each group of learners in order to gather qualitative data to describe the exact ways members of each group go about learning, the barriers they face in the learning process, and the things that facilitators do to help and hinder them in the learning process. Based upon this information, the wording of each item will be reviewed and adjusted to be extremely compatible with the comments of the group members. Once this process is completed, a criterion-related validity check will be made on the final form of the instrument.
Conclusion
One of the most exciting results of recent research related to learning strategies is the confirmation that distinct groups of learners who use clear patterns of learning strategies can be identified. These findings have been consolidated to produce an instrument, ATLAS, which can be used to quickly identify a person's group membership for learning strategy usage. ATLAS was purposely designed to be easy to use and quick scoring. Although it appears to be a very simple instrument, its contents are based on powerful multivariate statistical procedures. The final form of the instrument is available from the authors.
References
Aldenderfer, M., & Blashfield, R. (1984). Cluster analysis. Beverly Hills, CA: Sage Publications.
Bighorn, R. (1997). Learning strategies in the Fort Peck Reservation community. Unpublished doctoral dissertation, Montana State University, Bozeman.
Borg, W. R., & Gall, M. D. (1983). Educational research. New York: Longman.
Conti, G. J. (1996). Using cluster analysis in adult education. Proceedings of the 37th Annual Adult Education Research Conference. University of South Florida, Tampa.
Conti, G. J., & Fellenz, R. A. (1991). Assessing adult learning strategies. Proceedings of the 32nd Annual Adult Education Research Conference. University of Oklahoma, Norman.
Conti, G., Kolody, R., & Schneider, B. (1997). Learning strategies in the corporate setting. Proceedings of the 38th Annual Adult Education Research Conference (pp. 67-72). Oklahoma State University, Stillwater.
Courtnage, L. A. (1998). Advertising industry survey: Learning strategies of advertising sales people. Unpublished doctoral dissertation, Montana State University, Bozeman.
Gehring, C. M. (1997). Factors affecting learning strategies in the professional workplace. Unpublished doctoral dissertation, Montana State University, Bozeman.
Hays, P. A. (1995). Learning strategies and the learning-disabled adult student. Unpublished doctoral dissertation, Montana State University, Bozeman.
Hill, M. (1992). Learning strategies used in real life and achievement of adult Native American tribal college students. Unpublished doctoral dissertation, Montana State University, Bozeman.
Kerlinger, F. N. (1973). Foundations of behavioral research. New York: Holt, Rinehart, & Winston.
Kolody, R. C. (1997). Learning strategies of Alberta college students. Unpublished doctoral dissertation, Montana State University, Bozeman.
Kolody, R. C., & Conti, G. J. (1996). The use of learning strategies: Do distinctive groups of learners exist? Proceedings of the 37th Annual Adult Education Research Conference (pp. 199-204). University of South Florida, Tampa.
Korinek, D. L. (1997). An investigation of learning strategies utilized by Air Force officers. Unpublished doctoral dissertation, Montana State University, Bozeman.
Lockwood, S. (1997). An investigation of learning strategies utilized by nursing students in Montana. Unpublished doctoral dissertation, Montana State University, Bozeman.
McKenna, R. J. (1991). The influences of personal and professional learning situations on real-life learning strategy utilization by school administrators in Wyoming. Unpublished doctoral dissertation, Montana State University, Bozeman.
Moretti, R. (1994). Executive level decision styles and learning strategies of volunteer leaders. Unpublished doctoral dissertation, Montana State University, Bozeman.
Strakal, D. J. (1995). The use of real-life learning strategies in personal and career development situations by students at Eastern Idaho Technical College. Unpublished doctoral dissertation, Montana State University, Bozeman.
Tyler, L. E., & Walsh, W. B. (1979). Tests and measurements. Englewood Cliffs, NJ: Prentice-Hall.
Uhland, R. L. (1995). Learning strategy behaviors demonstrated by low-literate adults engaged in self-directed learning. Unpublished doctoral dissertation, Pennsylvania State University, University Park.
Ungricht, T. R. (1997). Learning strategies of concurrent enrollment students at Utah Valley State College. Unpublished doctoral dissertation, Montana State University, Bozeman.
Van Dalen, D. B. (1979). Understanding educational research. New York: McGraw-Hill.
Yabui, A. E. (1993). Reflective judgement and the adult learner's use of metacognitive learning strategies. Unpublished doctoral dissertation, Montana State University, Bozeman.