Hostname: page-component-7c8c6479df-r7xzm Total loading time: 0 Render date: 2024-03-28T20:52:56.378Z Has data issue: false hasContentIssue false

Is It Better to Be Feared Than Loved? Investigating the Efficacy of Different Teaching Methods on the Learning of Specific Content

Published online by Cambridge University Press:  06 October 2014

Robert P. Amyot*
Affiliation:
Hastings College
Rights & Permissions [Opens in a new window]

Abstract

The use and efficacy of problem-based learning (PBL) exercises during single class periods, especially in political science courses, have largely been unstudied. This article explores the ability of a PBL exercise to provide better learning outcomes than a standard lecture-discussion presentation on a particular topic in an introductory political science course. The author finds that the PBL approach, as a one-time pedagogical exercise, is not more effective at producing learning. In exploring the data, however, the author finds that differences in course structure and methods of learning assessment can have a larger impact on learning outcomes than specific teaching techniques.

Type
The Teacher
Copyright
Copyright © American Political Science Association 2014 

When first designing a group exercise in my introduction to political science course, my sole aim was to provide some variety for my students. A typical 50-minute class consisted of a mix of discussion and oral quiz questions, in which I posed a content-based, analytical, or more provocative question to the class—based on the daily reading assignment—and then randomly chose a student to answer it. I used that initial Q&A to further explore the students’ understanding of the concept, explaining the finer points and relating the content to the day’s news. Although this technique certainly kept students engaged, previous student evaluations indicated that they found it rather monotonous. They enjoyed something different now and then to “spice up” the class.

Hence, in considering how to make a chapter on constitutions more interesting, I devised a way to seize their imagination and encourage the students to apply the chapter content to a fictional problem: they would develop a constitution for the spaceship Aurora, which would take an international group of colonists on a 200-year-long journey to another planet.

Because my introductory classes typically have 20 to 25 students, it was obvious to me that smaller groups would be required. Therefore, I divided the students into four groups so they could tackle the problem and collaborate on a report at the end of the class session. That report would count as a quiz grade for all of the students involved in the project.

At this point, I had not read any scholarship on the use of simulations, problem-based learning (PBL), or group-based learning and therefore was operating solely on instinct. With a minimum of instructions, I left the groups to their own devices to outline a basic constitution, taking into account the assigned chapter’s discussion of successful constitution-building characteristics. Students seemed energized and engaged during the exercise, and they demonstrated a solid understanding of the main points of the chapter in their reports. I judged the exercise to be a success and used it in future sections of the course.

After attending my first APSA Teaching and Learning Conference and reading a study presented there about a similar exercise (Centellas and Love Reference Centellas and Love2011), I realized that much more was at stake. Beyond simply providing variety in the classroom, this technique could be more effective in helping students learn course content. Moreover, it might help them acquire skills that would be useful in other classes and after graduation, such as the abilities to collaborate, problem solve, and present as a group. If one technique could do all of this, then should I be using it more often? Should I incorporate more of these exercises into my courses?

BACKGROUND: TWO APPROACHES TO GROUP-BASED LEARNING

Even a cursory survey of pedagogical techniques shows that there are many reasons and many ways to incorporate groups into active learning. The techniques vary in the specificity of the roles given to students, degree of situational detail provided, type and degree of the instructor’s involvement, and pedagogical goals of the exercise.

In general, the group-learning approach has been shown to be very effective, as found by a meta-analysis of 168 studies (Johnson et al. Reference Johnson, Johnson and Smith1998). Group-learning pedagogies are classified generally as either cooperative team learning or collaborative learning (Bruffee Reference Bruffee1995; Matthews et al. Reference Matthews, Cooper, Davidson and Hawkes1995; Smith Reference Smith1995). Cooperative team learning has the clearest evidence of its efficacy in the educational literature (Johnson et al. Reference Johnson, Johnson and Smith1998). In cooperative team learning, the instructor groups students, assigns roles, provides all background information necessary, and outlines specific procedures to be followed (Bruffee Reference Bruffee1995). Most political science simulations are in this category, with students playing national leaders, United Nations negotiators, members of parliament, or senators. They interact with one another in well-defined ways, and the instructor interacts with the players and provides information, feedback, and guidance.

Collaborative learning is much less structured. This PBL approach presents a group of students with a goal to reach or a problem to solve; however, it does not provide all of the information needed or set procedures to follow in solving the problem (Bruffee Reference Bruffee1995). Although these types of exercises sometimes are labeled as simulations, the lack of set roles and elaborate procedures makes them more akin to classic PBL exercises. Footnote 1 (Project-based learning, a similar pedagogy, emphasizes the final product rather than the problem or process [Bransford and Stein Reference Bransford and Stein1993].)

The use and efficacy of these types of exercises during single class periods, especially in political science courses, have largely been unstudied. The quasi-experiment reported in this article is a small step in the exploration of the limits of PBL collaborative group learning.

In the literature, and for both types of learning, it is almost always recommended that the exercise last for long periods—multiple class periods, for example, or even several weeks. The use and efficacy of these types of exercises during single class periods, especially in political science courses, have largely been unstudied. The quasi-experiment reported in this article is a small step in the exploration of the limits of PBL collaborative group learning.

GROUP-BASED LEARNING IN POLITICAL SCIENCE

Evidence on the efficacy of PBL collaborative group learning in political science courses is mixed. Recent studies using this technique to enhance student learning in introductory-level political science courses attempted to measure its effect on student learning in different ways. Raymond (Reference Raymond2010), for example, examined the effect of a simulation exercise on exam scores; Williamson and Gregory (Reference Williamson and Gregory2010) sought evidence that multiple PBL exercises lead to better critical thinking, use of sources, and understanding of political institutions in student papers. In a simpler approach, Centellas and Love (Reference Centellas and Love2012) looked for improved performance on a battery of multiple-choice questions testing specific knowledge that should have been reinforced in a four-week PBL exercise. The results of these studies are mixed: Raymond (Reference Raymond2010) found no direct evidence of improved learning, whereas Williamson and Gregory (Reference Williamson and Gregory2010) and Centellas and Love (Reference Centellas and Love2012) did.

THE ORIGINAL STUDY: DOES PROBLEM-BASED LEARNING REALLY TEACH BETTER THAN LECTURES?

This research was originally modeled on an earlier study (Centellas and Love Reference Centellas and Love2011). In that study, Miguel Centellas and Gregory J. Love evaluated their attempt to use a PBL approach to enhance student learning in an introductory-level political science course. After administering a pretest, the authors randomly grouped students to write a new constitution for the Land of Oz, a project that lasted several weeks. The authors then analyzed the students’ performance on a multiple-choice battery embedded in the final exam. They found that students who had taken part in the PBL exercise scored significantly better than the control group on the battery of items related to constitution building.

However, a careful reading of Centellas and Love’s methodology suggested an alternate explanation for the treatment group’s higher scores: the amount of time spent working with the tested content. In Centellas and Love’s quasi-experiment, the treatment groups spent additional time working on the project for about four weeks, whereas the control group did not have any extra assigned work outside of the classroom. I wondered if it was the extra time spent by the treatment groups—rather than the PBL approach—that led to their better learning of specific content than the control group.

The question I intended to answer, then, was whether a collaborative group exercise is more efficient in improving student recall of specific content than a lecture-discussion session taking the same amount of time. In other words, is it PBL or the time spent working with the material that improves the learning and retention of specific content?

RESEARCH DESIGN

Seven sections of introduction to political science during two semesters, with a total of 137 students, were included in this study, which was approved by the institution’s Committee for Human Subject Research (on file; available on request). Three of the classes were taught by Professor A and four by Professor B. Both instructors used the same textbook for all their sections: Power and Choice by W. Phillips Shively. On the day that each instructor was scheduled to discuss constitutional design, students were administered a short, three-item, multiple-choice pretest (see the appendix for this and other study documents) to ascertain how much they understood of that day’s reading assignment. Students were then assigned to groups using a count-off technique (to separate seatmates); two groups were sent to other rooms where they would do a PBL exercise, and the remainder stayed in their seats. After giving brief oral instructions to each PBL group, the instructor returned to the main classroom and used the standard lecture-discussion format to cover material on constitutional design.

In addition to the brief oral instructions, PBL groups were given a one-page written description of their task that encouraged them to use their notes and textbook to design a “good” constitution for the space colony. The PBL groups spent the next 35 minutes brainstorming and outlining a constitution, and then submitted their work at the end of the class period.

In the next class period, all students were tested—using a reordered version of the pretest instrument—to measure any differences in learning effects. Two weeks later, a different reordered version of the instrument was administered to measure any differences in retention. Students’ scores ranged from zero (none correct) to 3 (all correct) on each instrument. Footnote 2

FINDINGS OF THE ORIGINAL STUDY

Overall, the PBL exercise appeared to have no significant effect on the learning of content. Footnote 3 For the 67 students who were exposed to the standard lecture-discussion format, their scores increased an average of 0.60 points at the first retesting; the PBL students averaged a gain of 0.64 points. Differences between treatment groups were more pronounced when retention was measured two weeks later. While the results again did not reach statistical significance, this time they ran counter to the expected direction: students exposed to the lecture-discussion format averaged 0.68 points higher than their pretest scores, whereas those in the PBL group achieved a mean that was only 0.58 points higher (table 1).

Table 1 Results by Treatment

A linear-regression model that incorporates students’ final grades as a control variable confirmed these findings: the results yielded very low adjusted R-squares and no significant effects for the PBL dummy variable on posttest or retention-test scores (table 2).

Table 2 Linear-Regression Model: Treatment and Final Grade

** p<0.005

These results suggest that PBL exercises during a single class period are not more effective than the lecture-discussion format in helping students learn specific content. Moreover, the PBL exercises may be less effective than the lecture-discussion format in promoting the retention of knowledge during a period of several weeks.

The differences were not between the treatment and control groups but rather appeared to be related to the principal instructor.

IS IT BETTER TO BE FEARED? THE IMPORTANCE OF COURSE STRUCTURE AND METHODS OF LEARNING ASSESSMENT

Although these results provided a straightforward answer to the original research question, further examination of the data revealed significant differences between groups of students in the amount of learning produced. The differences were not between the treatment and control groups but rather appeared to be related to the principal instructor.

There were two major differences in how the two instructors structured this phase of the course. The first difference involved expectations of students’ preparation for the class. In Professor A’s classes, student learning was assessed through the use of scheduled quizzes and exams, none of which took place during the study period. Professor B, however, administered daily random oral quizzes on the assigned readings throughout the semester, with each quiz worth 2% of the final grade. Footnote 4 Students who were not prepared for class risked a lower grade and embarrassment in front of their peers.

The second difference between the two instructors entailed the requirement of further work with the tested content. The students in Professor A’s classes moved on to other topics, whereas Professor B’s students—after the posttest was administered—were assigned a version of the group exercise as an individual paper. This assignment was due one week before the retention test and was worth 10% of the final grade.

There are specific reasons why these two distinctive approaches could lead to significant differences in the amount of learning produced in each instructor’s class. First, the possibility of a daily quiz likely encouraged Professor B’s students to increase the time they spent on the assigned reading before class, resulting in higher pretest scores. Second, the assignment of a substantial paper on constitutional design due before the retention test likely reinforced Professor B’s students’ learning of the material, thereby yielding much higher retention scores.

FINDINGS: COURSE STRUCTURE AND METHODS OF LEARNING ASSESSMENT MATTER

As expected, Professor B’s students were significantly better prepared for the initial assessment instrument, scoring 0.86 points higher on the pretest (i.e., p <0.005) than Professor A’s students (table 3)—a difference that held steady in the posttest assessment. This is an important point: whereas the scores of Professor B’s students were higher on both instruments, there was no difference in pretest or posttest gain between the two instructors’ students. Any advantage enjoyed by Professor B’s students was a result of their being better prepared, not from any difference in the instructor’s efficacy in the classroom.

Table 3 Results by Instructor

Also as expected, there was a significant difference in retention-test scores between the two instructors’ students (i.e., 1.26 points, p<0.005). In addition, the gain in retention scores of Professor B’s students was much higher than that of Professor A’s students: 0.89 versus 0.49 points, a near-significant difference of 0.40 points (i.e., p <0.10). Footnote 5

…this study simply confirms the obvious: when students are forced to read, think, and write about a topic, they learn it better than if they only listen to lectures about it.

These findings suggest that differences in course structure and methods of learning assessment contributed to how much students were learning and retaining during the study. A second linear-regression model using a Professor B dummy variable to account for the difference in pedagogical approaches, as well as a control for the final grade, supported these findings. The results are shown in table 4.

Table 4 Linear-Regression Model: Instructor and Final Grade

+ p <0.10, **p <0.005

DISCUSSION AND CONCLUSION

These findings suggest several lessons that can be applied to introductory political science courses in general. First, one-time PBL collaborative group exercises should not be expected to be more effective than the standard lecture-discussion format in helping students to learn specific content. The PBL exercises may be useful in varying the classroom routine and they may help to develop skills or encourage other types of learning. However, the findings in this study suggest that they do not produce more learning of specific content—and may even produce less.

This does not mean that PBL as a technique is ineffective. As discussed in this article, other studies (i.e., Centellas and Love Reference Centellas and Love2012; Vaidyanathan and Rochford Reference Vaidyanathan and Rochford1998; Williamson and Gregory Reference Williamson and Gregory2010) reported higher rates of learning when PBL group-based techniques were used during multiple class periods or when students were required to spend significant time working together outside of class. This study suggests, however, that those higher rates of learning are not due to the use of the PBL technique but rather to the requirement that students spend extra time and energy working with the content the mastery of which was being assessed in the study.

Other findings from this study, although not surprising, support this assertion. By establishing that students who face daily quizzes and are assigned a substantial paper learn specific content better, this study simply confirms the obvious: when students are forced to read, think, and write about a topic, they learn it better than if they only listen to lectures about it.

The main result of this study is that when we, as educators, can convince students to spend more time reading and working with specific content, the students apparently learn that content better. Finding the best way to motivate our specific student populations to do the extra work is perhaps the most important task we can do. PBL group learning can be an excellent way to accomplish this goal, but many other techniques may work just as well. v

Notes

APPENDIX A: Questionnaire

Name: _______________________

Class time: 8 am 9 am 11 am 1 pm

Later questionnaires rearranged both question order and answer order.

Shively Ch. 9 Pre-Treatment Quiz

This question was later dropped from the analysis because about 90% of students consistently answered it correctly at each administration of the questionnaire.

Please do NOT use notes or textbook when taking this quiz.

1. A federal state is one in which:

A) Local/regional political bodies may be overruled by the central government.

B) No other governmental body but the central government has any areas of policy that are exclusively under its control.

C) Two governments control the same group of people but with regard to different political questions.

D) The government and bureaucracy are relatively insulated from political pressures of groups in the society.

2. A constitution usually includes all of the following EXCEPT:

A) How the people holding those positions are to be chosen.

B) How government revenues are to be spent.

C) Who is to be in charge during an emergency.

D) Who is to carry out the major functions of politics.

3. A good constitution is one that:

A) Assumes that the holders of power will seek the common good for its own sake.

B) Holds with long-standing traditions.

C) Is difficult to change.

D) Is as specific as possible on major points.

4. Which of the following options best defines a constitution?

A) A group of people who have the ultimate authority to act on behalf of a state.

B) A set of rules (formal or informal) by which power is distributed among the members of a political group.

C) A collection of rules laid down by the government, binding all members of the state, including members of the government itself.

D) The basic unit by which people are organized politically.

APPENDIX B: For Instructors Working with the Collaboration/PBL Treatment Groups

Instructions for “Collaboration/PBL Treatment” Groups

In each room where the “collaboration/PBL treatment” groups will be gathering, distribute enough copies of the Briefing Sheet for each group and a few blank sheets for them to use for their outline. At the start of class, administer the pretreatment assessment tool. When finished, count off the students as they sit in class, “1-2-3-4,” moving left to right, front to back. Send Groups 1 and 3 to nearby rooms; when they are settled, brief them by reading the following instructions. (Groups 2 and 4 should stay in the classroom and wait for you to return to give a normal lecture on the topic.)

Verbal Instructions Treatment Group

“Congratulations! You have been selected to write a constitution for a group of colonists! They and their descendants will be stuck on a spaceship for the next 200 years, and we need to make sure they arrive at their destination in good shape – without experiencing civil war or genocide, for example.

“In the next 35 minutes, your group needs to lay out a basic outline of what that constitution should look like, based on Shively’s ideas of a ‘good’ constitution, and sketch it out on the paper provided.

“Please read the briefing sheet carefully before you begin. Good luck!”

APPENDIX C: Briefing Sheet for Collaboration/PBL Treatment Groups

Shively Ch. 9 Constitutional Design Exercise: The Colonization Ship Aurora

Welcome aboard the Aurora! You and your team of classmates need to come up with a constitution for the orderly self-government of a group of colonists.

Here are the details:

  • - The Aurora carries 10,000 colonists of varying ethnicities, skills, levels of education, religion, etc.

  • ° Their countries of origin are: U.S. and Canada (3,000), Russia (2,000), Saudi Arabia and UAE (1,000), China (2,000), and Japan (2,000).

  • - Colonists will be on the ship for 200 years before it arrives at its destination.

  • - The ship itself is segmented into a main hub (with automated piloting, energy, and propulsion systems, all not reachable by the colonists) and 20 linked pods:

  • ° 10 residential pods of 1,000 people each.

  • ° 10 specialized pods that include facilities to which all colonists have access:

  • - administration, education, and entertainment

  • - physical conditioning and recreation

  • - medical care and biological research

  • - physics and chemistry research

  • - industrial production

  • - food production and agricultural research (2 pods)

  • - water purification and aquaculture production (2 pods)

  • - storage of colonization supplies

INSTRUCTIONS

  1. 1. Based on Shively’s discussion of what makes a “good” constitution, your team must outline a constitution for the Aurora colonists to use to govern themselves while onboard.

  2. 2. Each team has 35 minutes.

  3. 3. At the end of the exercise, hand in your outline on a separate sheet, marked with your class time, group number, and group members’ names.

APPENDIX D: Paper Assignment and Rubric [assigned to all Professor B’s students]

Shively Ch. 9 Paper 3: Constitutional Design Exercise: The Colonization Spaceship Aurora

The bad news: Scientists have discovered that a comet will hit the Earth, causing massive devastation and perhaps the extinction of the human race, unless it can be diverted. A UN mission has been put together to try to change the comet’s path, but we have no idea if it will succeed.

The good news: Various countries have banded together to build “ark” ships that will take human beings safely to other planets, where they can colonize them. The United States has partnered with Canada and Japan for their technological expertise, Russia with its heavy-lift rocket capabilities, China for its manufacturing and financial stake in the United States, Saudi Arabia for its wealth and access to petroleum, and Ghana for its access to important high-tech minerals needed for the “hyper-ion” propulsion system, to build a colonization ship called the Aurora.

(The ship will be built in orbit by assembling various components and pods carried up via rocket. Once fully assembled, it will depart months before the comet hits the Earth, to be safely away when the time comes. Also, assume that our colonists will not run into other intelligent beings, human or otherwise, on the way to their new home, or once there.)

Your job: As a political scientist, you have been asked to create a Constitution for the orderly self-government of a group of colonists (initially numbering 10,000 and potentially growing to 10 times that number) that will be confined on this ship for about 200 years. You need to design a set of political institutions that will keep harmony among a diverse group of people who have to share close quarters and scarce resources, AND that will allow their descendants to prosper once the ship reaches its target planet.

Here are the details:

  • - The Aurora carries 10,000 colonists of varying ethnicities, skills, levels of education, religion, etc.

    • ° Their countries of origin are United States and Canada (3,000), Russia (2,000), Japan (2,000), China (1,000), Saudi Arabia (1,000), and Ghana (1,000).

    • ° The colonists will be between the ages of 20 and 65, in good health, and not carriers of major genetic diseases. While almost all colonists will be chosen for their physical, intellectual, or educational qualifications by a multinational committee of experts, 10% will be chosen by the donor-country governments. (For example, members of the Saudi royal family, the Canadian Olympic hockey and curling teams, and the top management of several American corporations will be included among the colonists.)

  • - Colonists will be on the ship for 200 years before it arrives at its destination.

    • ° The colonists will be expected to live normal life cycles during the voyage, procreating, aging, and dying. The ultimate capacity of the ship is 100,000, assuming all of its agricultural and aquacultural facilities stay functional. (That means that major birth-control efforts won’t be necessary for many generations.)

  • - The piloting, energy, and propulsion systems are all automated. Colonists will not be able to access these parts of the ship; instead, they will be confined to their own series of linked pods:

    • ° Ten residential pods, each able to hold 1,000 colonists. (Extra pods are available for future population growth but will initially be kept nonoperational until they’re needed, in order to save on power/resources.)

    • ° Ten specialized pods that include facilities to which all colonists must share access:

      • - administration, education, and entertainment

      • - physical conditioning and recreation

      • - medical care and biological research

      • - physics and chemistry research

      • - industrial production

      • - zoological preservation (animals and insects, some live, some in hypersleep)

      • - food production and agricultural research (2 pods initially; more available as population grows)

      • - water purification and aquaculture production (2 pods initially; more available as population grows)

    • ° In addition, a colonization pod containing transport ships preloaded with supplies and equipment is attached at the head of the ship, to be accessed once the Aurora enters orbit around its intended target.

Footnotes

1. “What differs [in PBL exercises] from many standard political science simulations is the level of detail that students receive for the steps they must take from that point forward….PBL scenarios emphasize the importance of the process of finding a solution as much as the solution itself” (Williamson and Gregory Reference Williamson and Gregory2010, 278–9).

2. Originally, there were four questions, but one was dropped due to a lack of variability because approximately 9 of 10 students scored correctly in each iteration.

3. This study does not examine the development of other knowledge or skills, although PBL exercises very well may help with them. “Problems provide students with direction and framework but do not provide all of the information necessary for ‘solving’ the problem; students must become active participants in their own learning by figuring out which questions to ask to obtain the needed information and by doing necessary research.…PBL scenarios should include opportunities for students to practice listening, writing, verbal and presentation skills and negotiation skills” (Williamson and Gregory Reference Williamson and Gregory2010, 276).

4. For more information about how to use oral quizzes to increase student preparation and understanding of the material, enliven the classroom, and allow better use of instructor resources, please contact the author.

5. It is regrettable that the study did not include a group of students who did not face the possibility of a quiz but who wrote the paper, or vice versa; thus, the effect that each separately would have had on retention scores cannot be established.

References

Bransford, John D., and Stein, Barry S.. 1993. The IDEAL Problem Solver (2nd ed.). New York: W. H. Freeman Publishers.Google Scholar
Bruffee, Kenneth A. 1995. “Sharing our Toys: Cooperative Learning Versus Collaborative Learning.” Change 27 (1) (January/February): 12–18.CrossRefGoogle Scholar
Centellas, Miguel, and Love, Gregory J.. 2011. “We’re Off to Replace the Wizard: Lessons from a Collaborative Group Project Assignment.” APSA Teaching and Learning Conference, Albuquerque, NM, February.CrossRefGoogle Scholar
Centellas, Miguel, and Love, Gregory J.. 2012. “We’re Off to Replace the Wizard: Lessons from a Collaborative Group Project Assignment.” PS: Political Science and Politics 45(3): 506–12.Google Scholar
Johnson, David W., Johnson, Roger T., and Smith, Karl A.. 1998. “Cooperative Learning Returns to College: What Evidence Is There That It Works?Change 30 (4) (July/August): 27–35.CrossRefGoogle Scholar
Matthews, Robert S., Cooper, James L., Davidson, Neil, and Hawkes, Peter. 1995. “Building Bridges between Cooperative and Collaborative Learning.” Change 27(4): 3440.CrossRefGoogle Scholar
Raymond, Chad. 2010. “Do Role-Playing Simulations Generate Measurable and Meaningful Outcomes? A Simulation’s Effect on Exam Scores and Teaching Evaluations.” International Studies Perspectives 11: 5160.CrossRefGoogle Scholar
Smith, Karl A. 1995. “Cooperative vs. Collaborative Learning Redux.” Change 27(3): 6.CrossRefGoogle Scholar
Vaidyanathan, Rajiv, and Rochford, Linda. 1998. “An Exploratory Investigation of Computer Simulations, Student Preferences, and Performance.” Journal of Education for Business 73(3): 144–50.CrossRefGoogle Scholar
Williamson, Jonathan, and Gregory, Alison S.. 2010. “Problem-Based Learning in Introductory American Politics Classes.” Journal of Political Science Education, 6: 274–96.CrossRefGoogle Scholar
Figure 0

Table 1 Results by Treatment

Figure 1

Table 2 Linear-Regression Model: Treatment and Final Grade

Figure 2

Table 3 Results by Instructor

Figure 3

Table 4 Linear-Regression Model: Instructor and Final Grade