The Effect of Embedding Visual Devices with Three Types of Semantic Prompts on Students’ Relational Understanding of History

Steve Wills & Edwin Ellis

This study compared the effectiveness of SMARTsheets from Makes Sense Strategies with three categories semantic prompts relative to traditional text-based instructional tools:

1. Information Structure (IS) Organizer SMARTsheets (visual devices that illustrate four basic ways to structure information: hierarchic, compare/contrast, cause/effect, and/or sequence.
2. Essential Understanding / Information Processing (EU/IP) SMARTsheets (topic-specific visual devices designed to (a) focus attention on what is essential to understand about each of 15 of the most commonly taught social studies topics, and (b) prompt student engagement in use of specific information processing strategies.
3. Generative-idea (GI) SMARTsheets (visual devices designed to focus attention on generative ideas related to specific social studies topics commonly addressed in secondary history classes.

Subjects consisted of 96 10th grade students enrolled in one of 8 American History classes in an urban high school taught by 4 “highly qualified” history teachers with at least 7 years teaching experience.
STUDENTS
32 = High achieving
32 = Typical achieving, including 4 classified as LD
32 = Low achieving, including 12 classified as LD

Each teacher used each of the four instructional techniques to teach American history mini-units, but to control for order effects, the sequence in which a given procedure was used differed across teachers. For example, for the first mini-unit, each of the four teachers taught a mini-unit on the Great Depression, but one teacher used traditional text-based techniques, another used the EU/IP SMARTsheets, another used the IS Organizer SMARTsheets, while the fourth teacher used GI SMARTsheets. For the second mini-unit, each of the teachers used a different instructional technique than they had previously used, and so forth. See Figure 1

A mixed method research design was used to collect and analyze quantitative and qualitative data.

QUANTITIVE DATA

Analysis of pre- and post-mini-unit student generated concept maps relative to pre-established criterion maps provided three relational understanding scores (Ruiz-Primo, Shavelson & Shultz, 1997):

• Accuracy of Relational Understanding Score
• Breadth of Relational Understanding Score
• Depth of Relational Understanding Score

These scores were derived from student oral explanations of ideas depicted on concept maps they generated before and after each mini-unit; students orally explained links between the terms and were asked to elaborate on concepts relating to the proposition identified. Students were provided three types of oral prompts to encourage elaboration:

• Prompts to summarize important ideas about the topic
• Prompts to relate or apply ideas
• Prompts to think about the idea in a new way (what if…)

Each proposition received a Proposition Score (0-3) depending on whether the proposition appeared on the criterion map, whether the student’s explanation was accurate, factually correct and complete, and the degree of elaboration provided.

Analysis of variance (ANOVA) was used to determine the relative impact that each of the four instructional approaches evaluated by level of student achievement. Tukey’s HSD was used for post hoc analysis.

QUANTITIVE RESULTS

The above data indicate that significant differences were found between the four different instructional techniques. The EU/IP SMARTsheets and IS SMARTsheets had the greatest impact on students’ depth of relational understanding. Use of traditional-text-based instruction was the least effective.

The above data indicate that significant differences were found between the four different instructional techniques relative to disabilities status. For students with LD, the EU/IP SMARTsheets and IS SMARTsheets had the greatest impact on their depth of relational understanding. Use of traditional-text-based instruction was the least effective.

The above data indicate that significant differences were found between the four different instructional techniques. The EU/IP SMARTsheets had the greatest impact on students’ depth of relational understanding, followed by IS SMARTsheets. Use of GI SMARTsheets and traditional-text-based instruction were the least effective.

This data indicates that the affects of the four instructional techniques on students’ accuracy of knowledge affected all of the sub-groups equally. In other words, if a given technique worked well for one group, it worked equally well for all of the other groups with regard to knowledge accuracy. If a technique worked poorly with one group, it was equally poor with the other groups.

QUALITATIVE DATA

Qualitative data, via semi-structured interviews, were collected and analyzed to examine teachers’ and students’ perceptions of the use of different types of instructional techniques.

Common Themes from Semi-structured Interviews

The visual devices with embedded prompts about the information structure (hierarchic, compare/contrast, cause/effect, and/or sequence) as were perceived as useful by teachers because they help differentiate the curriculum and organize material.

Sample comments:

Teacher: “I could use these every day. They really help organize the material. When the students used these to take notes, it really helped them see what was important and how things relate”

Student: “The IS SMARTsheets point to the main topics instead of going into long, long notes. It is better than pages of notes.”

Both teachers and students perceived that the IS SMARTsheets with embedded prompts related to topic-specific essential understandings and cues to engage in specific information processing skills help make the concepts addressed during the history instruction applicable for the students.

Sample comments:

Teacher: “These (EU/ IP IS SMARTsheets) make the information real and personal to the student. It reminds them that history is interconnected and can even apply to their own lives.”

Student: “These (EU/IP SMARTsheets) make it so that you don’t forget what it’s all about. You look at the questions, then the whole topic comes back to you and you say ‘oh yea, I remember that from our notes’.”

Teachers viewed the EU/IP SMARTsheets as flexible tools that can be applied in a range of instructional contexts.

Sample comment:

Teacher: “(EU/IP SMARTsheets) can be used before taking notes to get the students thinking, while taking notes, or to take notes, or after taking notes to get the students’ reflections. You can also use these individually, in groups, or as a whole class.”

Teachers and students viewed the IS SMARTsheets featuring generative ideas somewhat differently. Teachers perceived them as complex and perceived that their students did not understand the “big ideas”.

Sample comment:

Teacher: “Maybe they would be good for twelfth grade second semester students; but my students did not understand these. They did not get the big picture that was trying to be expressed.”

The novelty of teaching generative ideas and perceived misalignment between them and what they perceived to be objectives in the state course of study impacted teachers’ value of the GI SMARTsheets.

Teacher: “The concepts in these (GI SMARTsheets) were not difficult, they were just new. We’ve never taught anything like this.”

Teacher: “I personally don’t have time to teach concepts not on the graduation exam.”

Teacher: “They (the students) needed more experience working with these (GI SMARTsheets) and these concepts.”

Students, however, valued the GI SMARTsheets and appreciated the direct instruction in the generative ideas related to a topic.

Sample comment:

Student:: “These (GI SMARTsheets) tell you what is important to understand. It spells it right out.”

SUMMARY & IMPLICATIONS

Quantitative data validate that IS SMARTsheets with embedded semantic prompts designed to focus attention about what is essential to understand about a topic and prompt use of specific information processing strategies and IS SMARTsheets depicting hierarchic, compare/contrast, cause/effect, and/or sequence information structures were found to significantly impact depth and breadth of relational understanding. The tools provide immediate benefit to students of all ability levels in secondary classrooms with little classroom time devoted to explicitly teaching the structure and use of the visual devices. These tools were significantly superior to the traditional text-based instructional techniques used by history teachers. Qualitative data, overall, reinforce the quantitative findings and provide social validity of the quantitative data for use of the tools among both teachers and students.

IS SMARTsheets depicting generative ideas, however, did not produce similar dramatic changes in students’ depth and breadth of knowledge. Qualitative data provides some insight as to why. Although students valued them, teachers’ perceived them as too complex and that the GI SMARTsheets were not aligned with the state course of study.