Results

Report from American Institute for Research

Validating the impact of Strategic Learning Initiatives' Focus Instruction Model (FIP)

 

 

 Steven Leinwand

Sarah Edwards

 

 

Submitted by:

American Institutes for Research

1000 Thomas Jefferson St., N.W.

Washington, DC 20007-3835

 July 29, 2009

 

 

 

Introduction

 

This report arises from discussions between John Simmons, President of Strategic Learning Initiatives (SLI) and the American Institutes for Research (AIR) to provide an external validation of the impact data in ten Chicago Public Schools elementary schools that participated in SLI’s Focused Instruction Process between 2006 and 2008. 

 

AIR agreed to complete two tasks:

 

·         Rerun and validate the ISAT-Reading data for the 10 FIP schools; and

·         Compare the ISAT scores between the 10 FIP schools and a matching set of 10 non-FIP schools.

 

The results that SLI has achieved, and that AIR has validated, are very impressive and suggest that well before decisions are made to reconstitute schools under the mandates of NCLB, school districts would be wise to consider far less drastic, but clearly powerful, interventions such as the Focused Instruction Process.

 

Findings

 

1.      How well did the 10 FIP schools do during the first two years of intervention when compared with the Chicago city average?

 

Figure 1 shows that for the two-year period from 2006 to 2008 all but two of the 10 FIP schools had ISAT Reading gains in the percent of students at or above proficient that exceeded the Chicago city average and that as a group, the gains in the FIP schools were nearly twice the city average (11.4 percentage points vs. 6.3 percentage points).

 

Figure 1

Two-Year Gains in Percent of Students Meeting/Exceeding Reading Proficiency for 10 FIP Schools During FIP (2006-08)

 

School	Gains
Cather	16.1
Faraday	14.2
Goldblatt	10.8
Morton	2.1
Tilton	14.2
Cardenas	18.7
Castellanos	6.0
Finkl	15.3
Gary	8.3
Kanoon	8.5
10 FIP Schools	11.4
Chicago City Average	6.3

Figure 2 shows the annual average gains in the FIP schools during the two-year period (2006-2008) during which the intervention was in place when compared with the annual average gains for the four-year period (2001-2005) prior to the intervention. Once again, the gains made during the period of the intervention are impressive relative to the gains prior to the invention and relative to non-FIP schools.

 

Figure 2

Annual Average Gains in Percent of Students Meeting/Exceeding Reading Proficiency Before (2001-2005) and During (2006-2008) the Intervention

 

	Before FIP (2001-2005)	During FIP (2006-2008)
5 Area 7 FIP schools	0.8	5.7
5 Area 10 FIP schools	3.6	5.7
10 FIP schools	2.2	5.7
Non-FIP schools	2.1	3.2

 

 

2.      How well did the 10 FIP schools do when compared with a set of matching schools?

 

AIR asked SLI to identity a set of ten matching schools to the ten FIP schools. SLI used the following process to make these matches:

1.        Data from publicly available data bases from Chicago Public Schools and Illinois State Department of Education was extracted for the 10 FIP schools and all other elementary schools in Chicago, Illinois.  This data included:

a.       Percent low income as indicated by participation in free or reduced price lunch program.

b.      Percent of enrolled students who were identified as being in any of six specific racial/ethnic groups:  White, Black, American Indian, Asian, Hispanic, Multi-Racial.

c.       ISAT Percent Meeting or Exceeding Proficiency in Reading for grade level in the years 2001-2008.  It should be noted that standards of proficiency were altered between academic years ending in 2005 and 2006.

d.      From the ISAT data, Average Annual Gains/Losses in Percent Proficient were calculated for the 2001 to 2005 (4 year) difference and separately for the 2006-2008 (2 year) period.

2.        A computer program was written to initially screen for matches to the 10 FIP schools.  The program ignored a criterion that was originally considered:  No consideration was given to whether the schools matched were in the same city neighborhood (“Area”).

3.        The program initially searched for schools that had a Low Income Percent within 1.0 percent of the Low Income Percent of the FIP school to which they were to be matched.  In some cases, this resulted in short list of schools which would be difficult to match to the FIP school using the additional criteria to be applied.  Accordingly, a second run of the program was performed to select schools that would match within 2.0 percent of the Low Income Percent of the FIP schools.  This resulted in a longer list.  In subsequent matching, preference was given to schools that appeared on the “within 1.0 percent” match list.

4.        Using the schools that matched on the Low Income criteria, a statistic designated as “stress” was calculated consisting of the sum of squared differences in racial/ethnic percentages between the FIP school and each potential matching school.

5.        The list of matching schools for each of FIP schools was sorted low to high on this statistic.  The lower the “stress” statistic, the closer the match on racial/ethnic composition.

6.        Further matching was performed manually with the following criteria:

a.       Percent Meeting or Exceeding Proficiency in 2005 was roughly equal in FIP school and matched school.  This criterion was applied because improvement due to interventions will depend upon starting point prior to the intervention.

b.      Average Annual Percent Gain/Loss for the period 2001 to 2005, prior to FIP intervention, was roughly equal.

7.        Finally, some schools which would otherwise be considered as possible matches were excluded because they had participated in other Strategic Learning Initiatives programs.

 

Figure 3 shows the results of this matched pair analysis of FIP vs. non-FIP schools prior to and during the intervention.  As would be expected given the selection process, before the intervention there is a very small difference in the average annual increase in the percent of students meeting or exceeding reading proficiency between the FIP and non-FIP schools (2.24 percentage points vs. 2.45 percentage points respectively for the period 2001-2005).  However, during the intervention there is a significant difference in the average annual increase – 5.71 percentage points for the FIP schools vs. 1.14 percentage points for the non-FIP schools – for the period 2006-2008.  (See Appendix, Tables 1, 2, and 3 for summary statistics and p-values for the t tests.)  Before the intervention, the mean difference in the average annual increase of percent of students meeting or exceeding reading proficiency between matched FIP and non-FIP schools was -0.21.  The mean difference in average annual increase between matched FIP and non-FIP schools during the intervention (4.57) was significantly greater.  (See Appendix, Tables 4 and 5 for summary statistics and p-value for the t-test.)

 

Figure 3

Average Annual Increases and Differences in Test Scores

for Matched Pair FIP and Non-FIP Schools Before and During FIP

Pair	FIP School	Non-FIP School	FIP School: Average Annual Increase Before FIP (2001-2005)	Non-FIP School: Average Annual Increase Before FIP (2001-2005)	FIP School: Average Annual Increase During FIP (2006-2008)	Non-FIP School: Average Annual Increase During FIP (2006-2008)	Difference in Average Annual Increase in Matched FIP and Non-FIP School Before FIP	Difference in Average Annual Increase in Matched FIP and Non-FIP School During FIP
1	CATHER	DEPRIEST	2.78	2.90	8.05	2.40	-0.13	5.65
2	FARADAY	O'TOOLE	1.25	2.23	7.10	-1.40	-0.98	8.50
3	GOLDBLATT	HERZL	-0.13	0.45	5.40	3.70	-0.58	1.70
4	MORTON	O'KEEFFE	0.10	0.05	1.05	0.25	0.05	0.80
5	TILTON	OWENS	0.20	-0.93	7.10	1.65	1.13	5.45
6	CARDENAS	LARA	2.78	2.45	9.35	2.50	0.33	6.85
7	CASTELLANOS	COOPER	7.30	3.65	3.00	-0.55	3.65	3.55
8	FINKL	STOWE	3.15	4.33	7.65	2.10	-1.18	5.55
9	GARY	MCCORMICK	2.90	4.55	4.15	-2.15	-1.65	6.30
10	KANOON	WHITNEY	2.08	4.85	4.25	2.90	-2.78	1.35
	All FIP schools	Matched non-FIP schools	2.24	2.45	5.71	1.14	-0.21	4.57

 

Figure 4 shows the average change in the percent of students meeting or exceeding reading proficiency between each year in the 10 FIP schools and the 10 matched non-FIP schools for the entire 2001-2008 period and reveals graphically the apparent impact of the FIP intervention.  

 

 

Figure 4

 

Note: The test format changed significantly from 2005 to 2006 so the 2005 to 2006 test score change is not included

 

Conclusion

 

It is clear, on the basis of the ISAT Reading scores for the percent of students meeting or exceeding proficiency in ten Chicago elementary schools for the period 2001-2008, that the Focused Instruction Process intervention has had a positive and significant impact on student achievement in the cohort of ten schools that participated in the FIP model from 2006 to 2008.  Whether compared to pre-intervention achievement, or to the entire set of Chicago elementary schools, or to a carefully selected set of matched schools, the data suggest that FIP has resulted in gains that are very unlikely to have occurred without the intervention.  

                       

APPENDIX

 

 

Table 1: Mean Average Annual Increase in Scores Before and During FIP

 for 10 FIP Schools

Variable	N	Mean	Std Dev	Minimum	Maximum
Average annual increase before FIP (2001-2005)	10	2.24	2.18	-0.13	7.30
Average annual increase during FIP (2006-2008)	10	5.71	2.58	1.05	9.35

 

 

Table 2: Mean Average Annual Increase in Scores Before and During FIP

for 10 Matched Non-FIP Schools

Variable	N	Mean	Std Dev	Minimum	Maximum
Average annual increase before FIP (2001-2005)	10	2.45	2.01	-0.93	4.85
Average annual increase during FIP (2006-2008)	10	1.14	1.98	-2.15	3.70

 

 

Table 3: t-test for Mean Average Annual Increase in Scores Before and During FIP

for 10 FIP Schools and 10 Matched Non-FIP Schools

Statistics
Variable	FIP	N	Lower CL	Mean	Upper CL	Lower CL	Std Dev	Upper CL	Std Err	Minimum	Maximum
			Mean		Mean	Std Dev		Std Dev
beforeave	0	10	1.01	2.45	3.89	1.39	2.01	3.68	0.64	-0.93	4.85
beforeave	1	10	0.68	2.24	3.80	1.50	2.18	3.98	0.69	-0.13	7.30
beforeave	Diff (1-2)		-1.76	0.21	2.18	1.59	2.10	3.10	0.94
afterave	0	10	-0.28	1.14	2.55	1.36	1.98	3.61	0.63	-2.15	3.70
afterave	1	10	3.86	5.71	7.56	1.78	2.58	4.72	0.82	1.05	9.35
afterave	Diff (1-2)		-6.73	-4.57	-2.41	1.74	2.30	3.40	1.03

 

 

T-Tests
Variable	Method	Variances	DF	t Value	Pr > |t|
beforeave	Pooled	Equal	18	0.23	0.8235
afterave	Pooled	Equal	18	-4.44	0.0003

 

 

Table 4: Mean Difference in Average Annual Increase

in Matched Pair FIP and Non-FIP Schools Before and During FIP

Variable	N	Mean	Std Dev	Minimum	Maximum
Difference in Average Annual Increase in Matched FIP and Non-FIP School Before FIP	10	-0.21	1.74	-2.78	3.65
Difference in Average Annual Increase in Matched FIP and Non-FIP School During FIP	10	4.57	2.59	0.80	8.50

 

 

 

Table 5: Paired t-test for Mean Difference in Average Annual Increase

in Matched Pair FIP and Non-FIP Schools Before and During FIP

Statistics
Difference	N	Lower CL	Mean	Upper CL	Lower CL	Std Dev	Upper CL	Std Err	Minimum	Maximum
		Mean		Mean	Std Dev		Std Dev
afterdiff - beforediff	10	2.56	4.78	7.01	2.14	3.11	5.67	0.98	-0.10	9.48

 

T-Tests
Difference	DF	t Value	Pr > |t|
afterdiff - beforediff	9	4.87	0.0009