|
|
American Institutes for
Research®
Validating the Impact of Strategic Learning Initiatives’
Focused Instruction Process (FIP)
Model
Steven Leinwand
Sarah Edwards
Submitted to:
Strategic Learning Initiatives
954 W. Washington Blvd. Suite
600
Chicago, Illinois 60607-2224
Submitted by:
American Institutes for
Research
1000 Thomas Jefferson
St., N.W.
Washington, DC 20007-3835
July 29, 2009
“American
Institutes for Research” is a registered trademark. All other brand, product,
or company names are trademarks or registered trademarks of their respective
owners.
|
|
|
Introduction
This
report arises from discussions between John Simmons, President of Strategic
Learning Initiatives (SLI) and the American Institutes for Research (AIR) to
provide an external validation of the impact data in ten Chicago Public Schools
elementary schools that participated in SLI’s Focused Instruction Process
between 2006 and 2008.
AIR
agreed to complete two tasks:
·
Rerun and
validate the ISAT-Reading data for the 10 FIP schools; and
·
Compare the ISAT
scores between the 10 FIP schools and a matching set of 10 non-FIP schools.
The results
that SLI has achieved, and that AIR has validated, are very impressive and
suggest that well before decisions are made to reconstitute schools under the
mandates of NCLB, school districts would be wise to consider far less drastic,
but clearly powerful, interventions such as the Focused Instruction Process.
Findings
1.
How well did the 10 FIP schools do during the first
two years of intervention when compared with the Chicago city average?
Figure 1
shows that for the two-year period from 2006 to 2008 all but two of the 10 FIP
schools had ISAT Reading gains in the percent of students at or above
proficient that exceeded the Chicago city average and that as a group, the
gains in the FIP schools were nearly twice the city average (11.4 percentage
points vs. 6.3 percentage points).
Figure 1
Two-Year
Gains in Percent of Students Meeting/Exceeding Reading Proficiency for 10 FIP Schools During
FIP (2006-08)
School
|
Gains
|
Cather
|
16.1
|
Faraday
|
14.2
|
Goldblatt
|
10.8
|
Morton
|
2.1
|
Tilton
|
14.2
|
Cardenas
|
18.7
|
Castellanos
|
6.0
|
Finkl
|
15.3
|
Gary
|
8.3
|
Kanoon
|
8.5
|
|
|
10 FIP Schools
|
11.4
|
|
|
Chicago City
Average
|
6.3
|
Figure 2
shows the annual average gains in the FIP schools during the two-year period
(2006-2008) during which the intervention was in place when compared with the
annual average gains for the four-year period (2001-2005) prior to the
intervention. Once again, the gains made during the period of the intervention
are impressive relative to the gains prior to the invention and relative to
non-FIP schools.
Figure 2
Annual
Average Gains in Percent of Students Meeting/Exceeding Reading Proficiency Before (2001-2005) and During
(2006-2008) the Intervention
|
Before FIP (2001-2005)
|
During FIP (2006-2008)
|
5 Area
7 FIP schools
|
0.8
|
5.7
|
5 Area
10 FIP schools
|
3.6
|
5.7
|
10 FIP
schools
|
2.2
|
5.7
|
Non-FIP
schools
|
2.1
|
3.2
|
2.
How well did the 10 FIP schools do when compared with
a set of matching schools?
AIR asked SLI to identity a set of ten matching
schools to the ten FIP schools. SLI used the following process to make these
matches:
1.
Data from publicly available data bases from Chicago Public Schools
and Illinois State Department of Education was extracted for the 10 FIP schools
and all other elementary schools in Chicago,
Illinois. This data included:
a.
Percent low income as indicated by participation in
free or reduced price lunch program.
b.
Percent of enrolled students who were identified as
being in any of six specific racial/ethnic groups: White, Black, American Indian, Asian,
Hispanic, Multi-Racial.
c.
ISAT Percent Meeting or Exceeding Proficiency in Reading for grade level in
the years 2001-2008. It should be noted
that standards of proficiency were altered between academic years ending in
2005 and 2006.
d.
From the ISAT data, Average Annual Gains/Losses in
Percent Proficient were calculated for the 2001 to 2005 (4 year) difference and
separately for the 2006-2008 (2 year) period.
2.
A computer program was written to initially screen for
matches to the 10 FIP schools. The
program ignored a criterion that was originally considered: No consideration was given to whether the
schools matched were in the same city neighborhood (“Area”).
3.
The program initially searched for schools that had a
Low Income Percent within 1.0 percent of the Low Income Percent of the FIP
school to which they were to be matched.
In some cases, this resulted in short list of schools which would be
difficult to match to the FIP school using the additional criteria to be
applied. Accordingly, a second run of
the program was performed to select schools that would match within 2.0 percent
of the Low Income Percent of the FIP schools.
This resulted in a longer list.
In subsequent matching, preference was given to schools that appeared on
the “within 1.0 percent” match list.
4.
Using the schools that matched on the Low Income
criteria, a statistic designated as “stress” was calculated consisting of the
sum of squared differences in racial/ethnic percentages between the FIP school
and each potential matching school.
5.
The list of matching schools for each of FIP schools
was sorted low to high on this statistic.
The lower the “stress” statistic, the closer the match on racial/ethnic
composition.
6.
Further matching was performed manually with the
following criteria:
a.
Percent Meeting or Exceeding Proficiency in 2005 was
roughly equal in FIP school and matched school.
This criterion was applied because improvement due to interventions will
depend upon starting point prior to the intervention.
b.
Average Annual Percent Gain/Loss for the period 2001 to
2005, prior to FIP intervention, was roughly equal.
7.
Finally, some schools which would otherwise be
considered as possible matches were excluded because they had participated in
other Strategic Learning Initiatives programs.
Figure 3 shows the results of this matched pair
analysis of FIP vs. non-FIP schools prior to and during the intervention. As would be expected given the selection
process, before the intervention there is a very small difference in the
average annual increase in the percent of students meeting or exceeding reading
proficiency between the FIP and non-FIP schools (2.24 percentage points vs.
2.45 percentage points respectively for the period 2001-2005). However, during the intervention there is a
significant difference in the average annual increase – 5.71 percentage points
for the FIP schools vs. 1.14 percentage points for the non-FIP schools – for
the period 2006-2008. (See Appendix,
Tables 1, 2, and 3 for summary statistics and p-values for the t tests.) Before the intervention, the mean difference
in the average annual increase of percent of students meeting or exceeding
reading proficiency between matched FIP and non-FIP schools was -0.21. The mean difference in average annual
increase between matched FIP and non-FIP schools during the intervention (4.57)
was significantly greater. (See
Appendix, Tables 4 and 5 for summary statistics and p-value for the t-test.)
Figure 3
Average Annual Increases and Differences in Test Scores
for Matched Pair FIP and Non-FIP Schools Before and During FIP
Pair
|
FIP School
|
Non-FIP School
|
FIP School: Average Annual
Increase Before FIP (2001-2005)
|
Non-FIP School: Average Annual
Increase Before FIP (2001-2005)
|
FIP School: Average Annual
Increase During FIP (2006-2008)
|
Non-FIP School: Average Annual
Increase During FIP (2006-2008)
|
Difference
in Average Annual Increase in Matched FIP and Non-FIP School
Before FIP
|
Difference
in Average Annual Increase in Matched FIP and Non-FIP School
During FIP
|
1
|
CATHER
|
DEPRIEST
|
2.78
|
2.90
|
8.05
|
2.40
|
-0.13
|
5.65
|
2
|
FARADAY
|
O'TOOLE
|
1.25
|
2.23
|
7.10
|
-1.40
|
-0.98
|
8.50
|
3
|
GOLDBLATT
|
HERZL
|
-0.13
|
0.45
|
5.40
|
3.70
|
-0.58
|
1.70
|
4
|
MORTON
|
O'KEEFFE
|
0.10
|
0.05
|
1.05
|
0.25
|
0.05
|
0.80
|
5
|
TILTON
|
OWENS
|
0.20
|
-0.93
|
7.10
|
1.65
|
1.13
|
5.45
|
6
|
CARDENAS
|
LARA
|
2.78
|
2.45
|
9.35
|
2.50
|
0.33
|
6.85
|
7
|
CASTELLANOS
|
COOPER
|
7.30
|
3.65
|
3.00
|
-0.55
|
3.65
|
3.55
|
8
|
FINKL
|
STOWE
|
3.15
|
4.33
|
7.65
|
2.10
|
-1.18
|
5.55
|
9
|
GARY
|
MCCORMICK
|
2.90
|
4.55
|
4.15
|
-2.15
|
-1.65
|
6.30
|
10
|
KANOON
|
WHITNEY
|
2.08
|
4.85
|
4.25
|
2.90
|
-2.78
|
1.35
|
|
All FIP schools
|
Matched non-FIP schools
|
2.24
|
2.45
|
5.71
|
1.14
|
-0.21
|
4.57
|
Figure 4
shows the average change in the percent of students meeting or exceeding
reading proficiency between each year in the 10 FIP schools and the 10 matched
non-FIP schools for the entire 2001-2008 period and reveals graphically the
apparent impact of the FIP intervention.
Figure 4
Note: The test format changed significantly from
2005 to 2006 so the 2005 to 2006 test score change is not included
Conclusion
It is
clear, on the basis of the ISAT Reading scores for the percent of students
meeting or exceeding proficiency in ten Chicago elementary schools for the
period 2001-2008, that the Focused Instruction Process intervention has had a
positive and significant impact on student achievement in the cohort of ten
schools that participated in the FIP model from 2006 to 2008. Whether compared to pre-intervention
achievement, or to the entire set of Chicago
elementary schools, or to a carefully selected set of matched schools, the data
suggest that FIP has resulted in gains that are very unlikely to have occurred without
the intervention.
APPENDIX
Table 1:
Mean Average Annual Increase in Scores Before and During FIP
for 10 FIP Schools
Variable
|
N
|
Mean
|
Std
Dev
|
Minimum
|
Maximum
|
Average annual increase
before FIP (2001-2005)
|
10
|
2.24
|
2.18
|
-0.13
|
7.30
|
Average annual increase
during FIP (2006-2008)
|
10
|
5.71
|
2.58
|
1.05
|
9.35
|
Table 2:
Mean Average Annual Increase in Scores Before and During FIP
for 10
Matched Non-FIP Schools
Variable
|
N
|
Mean
|
Std
Dev
|
Minimum
|
Maximum
|
Average annual increase
before FIP (2001-2005)
|
10
|
2.45
|
2.01
|
-0.93
|
4.85
|
Average annual increase
during FIP (2006-2008)
|
10
|
1.14
|
1.98
|
-2.15
|
3.70
|
Table 3:
t-test for Mean Average Annual Increase in Scores Before and During FIP
for 10 FIP Schools
and 10 Matched Non-FIP Schools
Statistics
|
|
|
|
|
|
|
|
|
|
|
|
Variable
|
FIP
|
N
|
Lower
CL
|
Mean
|
Upper
CL
|
Lower
CL
|
Std Dev
|
Upper
CL
|
Std Err
|
Minimum
|
Maximum
|
|
|
|
Mean
|
|
Mean
|
Std
Dev
|
|
Std
Dev
|
|
|
|
beforeave
|
0
|
10
|
1.01
|
2.45
|
3.89
|
1.39
|
2.01
|
3.68
|
0.64
|
-0.93
|
4.85
|
beforeave
|
1
|
10
|
0.68
|
2.24
|
3.80
|
1.50
|
2.18
|
3.98
|
0.69
|
-0.13
|
7.30
|
beforeave
|
Diff (1-2)
|
|
-1.76
|
0.21
|
2.18
|
1.59
|
2.10
|
3.10
|
0.94
|
|
|
afterave
|
0
|
10
|
-0.28
|
1.14
|
2.55
|
1.36
|
1.98
|
3.61
|
0.63
|
-2.15
|
3.70
|
afterave
|
1
|
10
|
3.86
|
5.71
|
7.56
|
1.78
|
2.58
|
4.72
|
0.82
|
1.05
|
9.35
|
afterave
|
Diff (1-2)
|
|
-6.73
|
-4.57
|
-2.41
|
1.74
|
2.30
|
3.40
|
1.03
|
|
|
T-Tests
|
|
|
|
|
|
Variable
|
Method
|
Variances
|
DF
|
t Value
|
Pr > |t|
|
beforeave
|
Pooled
|
Equal
|
18
|
0.23
|
0.8235
|
afterave
|
Pooled
|
Equal
|
18
|
-4.44
|
0.0003
|
Table 4:
Mean Difference in Average Annual Increase
in
Matched Pair FIP and Non-FIP Schools Before and During FIP
Variable
|
N
|
Mean
|
Std Dev
|
Minimum
|
Maximum
|
Difference in Average Annual Increase in Matched
FIP and Non-FIP
School Before FIP
|
10
|
-0.21
|
1.74
|
-2.78
|
3.65
|
Difference in Average Annual Increase in Matched
FIP and Non-FIP
School During FIP
|
10
|
4.57
|
2.59
|
0.80
|
8.50
|
Table 5:
Paired t-test for Mean Difference in Average Annual Increase
in
Matched Pair FIP and Non-FIP Schools Before and During FIP
Statistics
|
|
|
|
|
|
|
|
|
|
|
Difference
|
N
|
Lower
CL
|
Mean
|
Upper
CL
|
Lower
CL
|
Std Dev
|
Upper
CL
|
Std Err
|
Minimum
|
Maximum
|
|
|
Mean
|
|
Mean
|
Std
Dev
|
|
Std
Dev
|
|
|
|
afterdiff
- beforediff
|
10
|
2.56
|
4.78
|
7.01
|
2.14
|
3.11
|
5.67
|
0.98
|
-0.10
|
9.48
|
T-Tests
|
|
|
|
Difference
|
DF
|
t Value
|
Pr > |t|
|
afterdiff
- beforediff
|
9
|
4.87
|
0.0009
|
