| Summary
of Pilot Evaluation - May, 1999
Summaries
| Front-end | Pilot Project | Summative
| Supplemental
METHODOLOGY
The data were collected from on-site observations, face-to-face and telephone interviews, pre- posttest self-report comparisons, and narratives written by the In-Touch Science (ITS) activity leaders, and "I wonder" statements written by the children and recorded by adults. Twenty-one sites were included in the sample, with a total of approximately 200 children. The
paper-and-pencil instruments are included in the appendix.
Reporting the findings for the children will consist of combined discussion of children's
self-reported statements, and their observed behaviors, along with the adult leaders' comments about children's behaviors and statements that they noted.
Reporting the findings for the adults will be based on their self-reported pre-post feelings of confidence and concern, interspersed with quotes from their interviews. (The interviews were recorded so the quotations are verbatim.)
Reporting the curriculum itself will focus on answering these questions: Did the
adults like the manual? Was there enough information presented and in a way that untrained adults could comfortably implement the activities? Were the activities interesting to the children? Was the level of difficulty well-matched with the children's ages and the settings? These questions were answered by asking leaders to complete evaluation forms for each individual activity.
ADULTS
In evaluating the adults' responses to ITS, we wanted to know about leaders' concerns with their background knowledge, their time constraints, their discipline and behavior concerns, and so we asked them to complete a "Leadership concerns" form before and after each pair of activities. The leaders' responses here (as they indicated them on a Likert scale) are augmented by quotes from interviews and impressions by the evaluator.
Are you concerned about having background knowledge;
Are you concerned about being familiar with science teaching strategies
(1 = not at all concerned; 5 = extremely concerned)
Before implementing ITS: 3.06; After implementing ITS: 1.67
Leaders expressed their personal concern about implementing science activities. The leaders' range of "expertise" in science varied. There were some people who had extensive backgrounds in science, or education, but there were also leaders who reported that they had no background, and it was these individuals who, before they implemented ITS, expressed concerns about being able to lead science activities. In general, their concerns were lessened after they had an opportunity to implement ITS. There was a range of concern, and a range of comfort. For example, these two
perspectives were shared by interviewees (recorded in interviews):
Leader A: "I was concerned about background knowledge. I wasn't aware of what the kids would be asking and how many questions they would come up with. They come up with a lot of questions. I answer what I can and tell them we'll look it up."
Leader B: "I was concerned going into it because I was always, you're always afraid, do you have enough knowledge to be able to present this and present it well. But the book - the manual - was terrific. I mean it gave me everything I needed. It really did. I found it very, very easy to follow. You know, the hard work was already done for me. I just adapted what was in front of me to meet the needs of the group that was in front of me."
It was interesting that some of the leaders who seemed less confident relied on teen helpers, or (in mixed-age groups of children) on the older, more knowledgeable participants. On the other hand, the leaders who were confident of their science backgrounds seemed to not want older children to take leadership roles in the activities. These conflicting attitudes were clearly stated by two interviewees:
A: "Like I said, the ones who knew a little bit more about it - and these were experiments they'd never done before, so it was new for everyone there - but those who knew a little more about it helped younger ones or those that were struggling."
B: "I thought it was good to have the same age kids... the older girl was impatient because I wouldn't call on her more to answer because she knew it on a different level than they did and she got frustrated."
Are you concerned about having time to organize materials;
Are you concerned about obtaining hands-on materials
(1 = not at all concerned; 5 = extremely concerned)
Before implementing ITS: 2.95; After implementing ITS: 2.40
It can be difficult to "do" informal science activities. Most of the leaders work in settings that are less than optimal. It can be seen from the pre-post survey, that leaders were moderately concerned about the organizational aspects of ITS before they implemented the activities, and that their concerns did not diminish significantly. Concerns varied; some of the ITS units require more materials (or more specialized materials). One of the leaders who was doing Chemistry and Environment activities expressed concern about obtaining the chemical needed for the water-absorbing activity. Two of the leaders found that the daisies they purchased for one of the Plant Science activities were not fresh enough to pull up any colored water.
The ITS settings ranged from outdoor picnic tables to classrooms to shared spaces and hallways.
It is a compliment to the flexibility of ITS, and to the leaders' skills that the activities "worked" in all of these environments. One leader reported that:
"The shared space sessions were a challenge. We were impressed by the fact that the activities kept the youths' attention even with the many distractions and noise... since we were in a small room with 50 kids! We did a wrap-up in the hall after A and B to provide quiet closure and found
that the youth really did grasp the concept.
Of course, not all challenges were met successfully. We sympathized with reported failures
such as this:
"After working all day to prepare a lesson and to try it out, I got to the center and was told
that the teacher went home sick." This leader also shared with us the comment that "The supervisor was not even aware of what I was doing at the center. She knew that I came on Thursday afternoon, but that was the extent of it.
In some instances the external constraints kept activities from being undertaken. For example
at two sites, the cheese-making activity could not be done because there was no hot-plate. "Name the Flame" could not be done at some sites because of school rules against having open flame within the building
Are you concerned about meeting the needs of a range of kids;
Are you concerned about group management/maintaining discipline
(1 = not at all concerned; 5 = extremely concerned)
Before implementing ITS: 2.62; After implementing ITS: 1.53
Children with Special Needs: One leader (4-H club of children with special needs) pointed out
that ITS was so successful with his group because there was no reading required. He demonstrated the activities and the children did it themselves. He said that it was a real self-esteem builder. Another leader who worked with special needs children said that he had been concerned beforehand because of "the vast age range accompanied by an even larger cognitive range, as there were many disability 'labels' represented". This leader has since moved to a new special education teaching position and said that he felt that his experience with ITS may have helped him get the job, and he indicated that he was planning to make ITS a major part of his science teaching planning.
Another leader whose group included children with special needs said that:
"It went well. We had to do more one-on-one, and we had the teen volunteers there to help me
with that area. We kept it more hands-on and it brought them all together as equals I felt. And so it really improved their self-esteem of some of our kids who are more academically
challenged."
Native American children: It may have been a coincidence, but the sites that involved Native American children shared the characteristic that activities were not time-limited. For example, the Seneca Nation leader reported that she ended up letting the projects go as long as the children wanted to work on them. This allowed for enrichment that would not have been possible had the activities been limited to the set amount of time. One leader took an entire afternoon for the activity that involved feathers ("Fantastic Feathers"). Another leader pointed out that:
"The students loved the project 'Name that Flame' and I could have made it easily into a
two-hour class. They first were competitive in guessing the materials everyone brought in. I had two tables with 5 tins burning. The students would even wait to watch the others burn their materials. They compared the material burning with the clothes they were wearing to see how it
would burn. It did turn into a safety lesson when someone noticed the flannel burning and mentioned pajamas."
Girls: As one leader pointed out that, "Girls felt like scientists doing science stuff in
these activities." It was an opportunity for them to participate fully, not as observers, but with hands-on opportunities for participation. One leader said that:
"The interactive science, in small group activities of In Touch Science at Camp Brookledge reached some often looked-over girls. Ebony loved doing the science activities... being told she is smart. She often had great questions and ideas. This approach worked for this child who has
many obstacles to overcome in her life."
Another leader said that "Using these materials and activities really made an impact on these girls that science was fun, exciting and something they liked to do." She informed us that as they went through ITS, that "the girls had advanced in their skills and ability to
question, observe in these science activities."
In Touch Science offered an opportunity for one leader to invite representatives from Penn State Women in Engineering organization to speak to the girls who had been doing the "Plants and Engineering" ITS unit. She said "The girls had a chance to learn about engineering and science on a different level... we had pooled resources from the community to tie everything together."
Children of varying ages: The utilization of older youth volunteers was a familiar theme in the ITS implementations. One leader (one of the people who was less confident about her ability to lead science activities) said that:
"Teen leaders assisted also. The sixth graders enjoyed helping. They liked sharing their knowledge with the younger ones. It was a nice thing to see happen and I'm hoping to pull some of them back as helpers when they're in seventh grade... sort of mentoring the younger children. The ITS was one of the avenues that helped make that happen." In part, using teen volunteers was possible because, as this leader pointed out, the manual "...was just so well presented and broken down for anyone to understand... Even the teens could follow and they could feel comfortable having the book there and reading over it."
Are you concerned about incorporating activity into overall program;
(1 = not at all concerned; 5 = extremely concerned)
Before implementing ITS: 2.43; After implementing ITS: 1.53
Are you concerned about incorporating activity into available time blocks
Before implementing ITS: 2.57; After implementing ITS: 2.47
Although it had been something of a concern beforehand, leaders found that ITS fit well with other activities. Several leaders pointed to instances in which children made connections between what they were doing in an ITS activity and things that they had read or learned about or seen. As one leader said, "The kids were very enthusiastic about it. It depended on when we caught them as to what they knew about it already. We just did the erosion one (in Plants and Engineering) and that very morning they had watched the Magic School Bus do a whole program on erosion."
Another leader said that when her group of girls was doing the activity
where "we take the chemical and we add the water to it... at the same time, I had been talking to
somebody and they showed me a newspaper article in our local paper - the day before -- about a fireman who has come up with this same chemical and they're spraying it on houses - we tied it in that way."
However, concerns about incorporating the activities into available time blocks were not lessened. From observation and interview, it was clear that the children would become so involved in the activities that the time available (a half-hour squeezed into an after-school child care program) was sometimes not enough time. Not all leaders had strict time constraints. And in fact, 71 percent of indicated that the activities were "just right" in their length. However, for those
leaders who were under time pressure, ITS may be somewhat problematic.
Each activity was evaluated individually by each leader. Tabulation of these individual
evaluation forms gave the following information.
We wanted to know about:
The children's level of interest (84 percent high interest)
The amount of conversation among the children (82 percent high level of conversation)
The amount of conversation between children and adults (71 percent high level of conversation with adults)
We wanted to know whether leaders would do these activities again:
90 percent said yes.
We wanted to know what the leaders thought about the instructions in the manual:
4 percent said "too short"
2 percent said "too long"
94 percent said "just right"
We wanted to know what the leaders thought about the complexity of instructions:
4 percent said "too complicated"
nobody said "too simple"
96 percent said "just right"
We asked leaders what they thought about the activity for children in their
group:
17 percent said "too short"
13 percent said "too long"
71 percent said "just right"
We asked leaders what they thought about the difficulty for the
children:
20 percent said "too difficult"
7 percent said "too easy"
73 percent said "just right"
We asked leaders what they thought about the number of steps for the children:
11 percent said "too complicated"
9 percent said "too simple"
80 percent said "just right"
CHILDREN
Observational data were recorded by scanning the children while they were participating in the ITS activities. The observer sat in a corner of the room and observed individual children at ten second intervals with five minutes of observation and five minutes of rest. The observer noted whether the child was "talking/not talking", "involved in a hands-on activity/not involved in a hands-on activity", and if not doing something hands-on, then was the child "attending/not attending". Coding was only "yes" or "no" so that reporting is in percentages of time that the children were observed talking, being involved, and being attentive.
One of the ways in which we tried to evaluate whether the children were "getting" the conceptual connections between the two associated activities was that the children were asked to write questions they had - "I wonder..." before the activity and then again afterward and determine whether the "post" comment expressed a more complex understanding of the concept than the "pre" comment. We used a reader (an educator) blind to whether the "I wonder" statement had been
written before or after, and asked her which one of each pair was more complex. Another way we evaluated the children's enjoyment and understanding of the activities was to have the adult leaders write "I wonder..." comments as the children were doing the activities themselves. These were comments that reflected curiosity and thoughtfulness about the activity's concepts and the connection with everyday aspects of the children's lives.
We were especially interested in the response of children with special needs, girls, and Native American children. Therefore, special attention was paid to the sites in which ITS implementation included these populations. Data were collected from ITS with special education children an in-school setting, and in a summer day camp setting. ITS with girls was evaluated through its use in Girls Incorporated day camps and with girl scouts. Native American children tried ITS in an
in-school setting and in a summer day camp.
Youth's responses to ITS were measured in three ways -- their self-reported "I wonder" statements, observations made of their behaviors, and "I wonder" comments recorded by leaders and volunteers. Children's satisfaction was measured by the leaders writing down statements made by the children about level of satisfaction.
Observations of behaviors were kept to a simple 10-second scanning technique of each child once every five minutes. Only three variables were used to evaluate ITS's claim that these were hands-on activities, that talking was encouraged because it meant that children were involved, and that "sitting and listening" time would be minimized. Sixty children were observed in all, and of the children who were observed and scanned, we found that children who were:
observed talking 25 percent of the time
observed doing hands-on activities 75 percent of the time
observed not doing hands-on activities were attending 80 percent of the time
Here are a list of I wonder questions that were recorded by teen-age volunteers at the
Girls Incorporated Camp Brookledge girls day camp.
I wonder why the soap made the colors move?
What would happen if we added all the soap?
Ooh look, the soap is like strings? Why does it stay separate from the milk?
I wonder why oil and water don't like to mix but they do if you add soap?
What happens when we add milk to the oil/water/soap mixture?
What animals does oil effect? What problems happen during oil
spills?
What is in soap that helps clean the animals?
Can we pour milk in our lakes? Is that pollution?
I love science.
I thought this would be boring but it was mad fun.
Here are some randomly selected "Before" and "After" I wonder questions written by children
themselves.
Before: I wonder if the cream will turn into butter
After: It turns clumpy and then turns into butter.
Before: I thought it will be hard
After: I thought it was hard then I thought it was cool.
Before: I wonder if my cooler will work.
After: Until I made my cooler and it melted more.
Before: I wonder if the lard will be greasy.
After: I wonder if the balloon will get warm.
Before: I wonder what texture we are going to make.
After: Today I mixed heavy cream and warm milk and closed it and
shaked.
Before: I wonder if it would be a good lesson
After: I learned that there all kinds of feathers.
Before: I wonder if my ice will melt.
After: I wonder if you put jean material [sic] in it if it will be cold or warm.
Before: I wonder who invented plastic?
After: I wonder how do we unmake a plastic?
Before: I wonder how plastics work?
After: I wonder how do molecules work?
Before: I wonder why it's so sticky
After: I wonder why all plastics start with polly [sic]. What is the use of all different plastics?
Before: I wonder why water is clear
After: I wonder how salt gets in the ocean to make salt water.
These "I wonder" statements were not easy to evaluate in terms of complexity. An educator was given the statement pairs (those that were legible - many were not) and asked which one of each pair seemed to reflect a greater level of complexity or more developed thought about scientific concepts. The educator sorted the statement pairs into "Before" and "After" and two-thirds of the time (63 percent of the time, to be exact), she said that the "After" statement was more complex/developed than the "Before" statement.
-Submitted by Dr. Doreen
Greenstein
|
Looking
for extra activities to enhance your program?
Click
here for Helpful Hints submitted by In-Touch Science
users.
|
|
|
|
|
