High School to CSUN: Study of Multiple Measures for Math and Writing Placement and Implications of the EO 1110 Guidelines

This study is an effort to understand large-scale patterns in high school records of students who have attended CSUN. We want to understand how well multiple measures (grades earned in A-G high school courses and SAT/ACT test scores) are related to placement and success at CSUN.  We characterize the difference in the student populations who were placed in 100-level math GE baccalaureate level courses versus those who were placed in developmental math courses based on their high school records. We discuss the problems with the data, and the implications and implementation of the  California State University Executive Order EO 1110 and EO 1110 Guidelines, particularly for CSUN students and faculty, but also for the CSU system.

We examine high school application data of students who matriculated at CSUN as FTF from March 12, 2005 – March 1,  2016,  AY 2006- 2015, a total of 24,819 student records. During this time, the (old) SAT had a consistent scale. Early Assessment proficiency data was missing from our dataset.

Summary of Results

  • Based on past application data,  more than 50%  of FTF would have been placed in Category  IV if HS Math GPA. Using total 11th grade GPA, about 28% of students will be placed in Category IV (required to take stretch math courses). 

Estimate of student placement by EO 1110 by Categories I-IV on AY 2006-2015 data based on Total GPA:

Category Percent (%)
I       4.2
II     40.6
III     27.1
IV     28.1

Estimate of student placement by EO 1110 by Categories I-IV on AY 2006-2015 data based on Math GPA and requiring :

  •  33.4% of entering FTF have high school math weighted GPAs above 3.0.
  • Only 18% of FTF scored above 550 and 19.5% scored between 490-540 on the old Math SAT. About 26% of CSUN admits had SAT Math score above 500.
  • High school math grades, when available, are not a good measure of student success in math courses at CSUN without considering the high school of origin.
  • Students who list Algebra II as their highest level high school math course are overwhelmingly from URM populations.  Historically, they have been placed in developmental math, and often fail it at much higher rates.
  • SAT/ACT scores vary by highest level math course attempted in high school. Students who complete Algebra II as their highest level math course have the lowest scores and are placed in developmental math at the highest rate.
  • Math SAT scores have not gone up since 2000, but CSUN Math proficiency rates have increased dramatically.
  • Currently, not all transcripts are submitted electronically, and course placement has to be decided in advance on senior year grade submissions in June. Meanwhile, class schedules are set up in February. The timeline for admissions and placement make it difficult to effectively use multiple measures.
  • Only 9% of all students reported grades in their senior year, so we could not measure true success based on all 9-12 grades.

EO 1110 Implementation Problems

  • EO 1110 implementation is rushed;  CSUN EPC will not approve the curriculum.
  • There are no good estimates of FTE who will be moved to stretch math.  These elective units will come out of other departments. The impact on budget and positions is not well understood. The instructional value of the courses created for stretch math is difficult to evaluate because some of them do not exist.
  • The EO 1110 Guidelines do not match up well with our student population. Relative placement scores should be determined locally rather than systemwide due to variation in student preparation at each campus. The EO 1110 Guidelines set relative placement scores or cutoff scores for Category I-IV which appear to be too high for CSUN students.

Problems with Admissions, the EO 1110 Policy, Eligibility Index, and the Higher Education Master Plan 

  • There are problems with the eligibility index. Over 41% of graduating seniors in Los Angeles County are eligible for the CSUN, although the CSU was designed to admit the top third of students under the Master Plan for Higher Education.
  • CSUN has a chronic problem of exceeding targets. Many of these students are underprepared in math. They have low Math SAT scores and low math GPAs (below 2.0 GPA). The eligibility index needs to be reconsidered. For example, consider a minimum GPA of 2.0 in English/Language Arts and Mathematics.
  • The CSU campuses which admit students with the highest SAT scores and GPAs also spend the most on direct instruction per FTE.  This sets up a system which is separate, but not equal.  EO 1110 seeks to cut the time and money spent on math and writing preparation at high Pell campuses. We believe that there are better solutions. The Legislative Analyst Office has proposed supplemental funding for low-income, first-generation college students: each campus receives a base level of funding for every student plus a 20% supplement for each high-need student.

Comments on Writing Placement are in progress.

CSUN Admission requirements:  For in-state students, SAT/ACT  test scores are required unless the grade point average (GPA) is above 3.0. The CSU uses a calculation called an Eligibility Index that combines high school grade point average with the score on either the SAT or ACT tests. Impacted campuses and impacted first-time freshmen (FTF) enrollment categories often include test scores among the supplemental criteria required of all applicants to those campuses and enrollment categories. These criteria are so high at some campuses that eligible regional high school graduates can not attend. 

Admissions Data (2016) from ThoughtCo.

Simply based on this data,  we see that the more than a quarter of FTF may be placed in Category IV under the EO 1110 Guidelines. However, these students on Category IV will be placed in 100-level non-GE credit stretch or supported courses rather than developmental math courses and Early Start. Students in Category II and III are strongly recommended to follow the path of Category IV students. This means that less than 3% of CSUN FTF could be directly placed into 100-level GE math courses without reservation.  Thus, the EO 1110 Guidelines greatly expands remediation (now called stretch because it earns 100-level credit). Note: many units will be lost from other departments.

This report goes on to give a fuller picture of the variety of courses CSUN students take in high school and how that predicts placement and student success.

CSUN is a Low SAT/ACT Score, Low GPA School with High Percent of Students on Pell Grants 

In the scatterplot below, each campus is represented by a red dot. The trend line shows a linear relationship between Math (old) SAT scores and CSU Graduation Rates by campus (each red dot)Campuses with high Pell populations have more students with weaker high school preparation as measured by lower Math SAT scores and lower math GPAs. For more, see CSU Systemwide Impact of EO 1110.

A few facts about CSUN students pre-EO 1110:

  • 32% of CSUN’s 2016 freshmen were placed in developmental math courses.
  • Failure in math is known to be a leading factor in longer time to degree and  failure to graduate.
  • CSUN’s 6-year graduation rate of first time freshmen FTF is around 50%  for 2009 cohort.

Why is math placement important to get right?  Students who started in  the lowest level developmental math class M092 had a 40% chance of graduating if they passed the class; this increased to 50% if they start in and pass M093. Students who started in M092 or M093 and fail the first time have only a 9% or 16% change of graduating, respectively. We examined the most commonly failed courses taken in the last term before students drop out.  Math courses make up four of the five most frequently failed courses attempted in their final term before dropping out.  In order of frequency, the most commonly failed courses are  M102, M093, M092, Soc 150, and M140.  Proper placement of students into classes which they can pass is very important for student success, see [1].  Note, M092 and M093 will not be offered after Spring 2018; instead Early Start, 100-level courses to be described below.

Percent of Students with HS Weighted Math GPA above 3.0  

This histogram shows the distribution of weighted High School Math GPA. Notice that 33.7% of student have a GPA above 3.0; the majority of students have a Math GPA below 3.0.

Basic GPA Weighted Math HS GPA Stats:

Mean 2.7
Mode 2.5
25th Percentile 2.3
50th Percentile (median) 2.7
75th Percentile 3.1
95th Percentile 3.75
Weighted High School Math GPA  Percent (%)
 3.0 – 3.29     15.8
 3.3 – 3.49     5.7
 3.5 –  3.69      6.4
 3.7 – 4.0 or above      5.8
  • 90.2% of students have a weighted math GPA  over 2.0.
  • 33.4% of students have a weighted math GPA over 3.0.
    Only 2% of students did not take a math class in both junior and senior year.  For these students, we do not calculate a weighted math GPA.

High school math courses:  The math courses most commonly taken during the senior year are Math Analysis, AP Calculus AB or BC, Algebra II, AP Statistics, Precalculus, and Trigonometry. Less commonly attempted course include Discrete Math, Algebra I, Advanced Math, Accounting, Finite Math, and Probability.

A further breakdown by senior year course is given in the table below. Precalculus includes Math Analysis and Trig, Calculus includes AP, AB, and BC, and not AP, Statistics includes AP and non-AP.

HS Senior Year Math Course Number of students Percent (%)
Algebra II 6334  10.8
Remedial 2965   5.1
No math 16142  27.6
Precalculus 13709  23.4
Calculus 6451  11.0
Statistics 5961  10.2
Other 7008  12.0
Total 58570 100.0

From the above table we can conclude that, if Math SAT or ACT scores or grades are not high, as many as 43.5% of FTF could be placed in Category IV simply because they did not take one year of math beyond  Algebra I, Geometry, or Algebra II in their senior year. We note that some students who took no math in their senior year already completed Precalculus or above as juniors.

What is the breakdown of senior year high school math course by ethnicity?  The proportion of ethnicities is not evenly distributed in all senior high school math courses.  The majority of students taking Algebra I, Algebra II, Geometry, or Remedial courses in their senior year are from URM groups.

Percent Math Course by Ethnicity  

                   

Raw Count Ethnicity by Senior Year Math Course 

 

What do high school math course records tell us about placement and success in CSUN math courses?  There were no statistical differences in the pass rates of students with or without senior year math who took Developmental Math classes in the Fall term.  There was a statistical difference in pass rates of students who took Developmental Math in the Spring term in CSUN and Math in their senior year in high school: students were less likely to pass. 

Box plot of GPA in  100-level math courses at CSUN based on math background (no math or math in 12th grade and no Developmental Math or  Developmental math).

Pass rates for any first math course (Developmental Math or GE) varies by ethnicity:

Ethnicity Pass rate in first math course (%)
African American 59
Asian 76
Latina/o 68
Multi-race 73
White 78

There is an opportunity gap starting from the first math class attempted.

Time to Degree

The last high school math course attempted was a significant factor in the average time spent on finishing a college degree. High school students who took Algebra II or Geometry average degree completion in 10.7 or 10.8 terms, while students who took Trig/Math Analysis, Calculus, or Statistics finished in 10.5, 10.2, or 10 terms, respectively. This is not surprising, since a far higher percent of students who took Algebra II in their senior year placed into developmental math. We note that missing data made it difficult to accurately predict the completion times.* (For details, see Problems with CSUN Application Data and Student Records.) We will later discuss the implications of time to degree for the majority students under EO 1110.

Analysis of High School to CSUN Math Pathways

Students who took Calculus or Statistics in their senior year generally placed out of developmental math courses. 37% of students who took  Precalculus placed in developmental math (which contains material well below Precalculus course level). About 45% of those who took no math place into Developmental Math.  The highest percent of students who placed into CSUN’s developmental math courses took remedial math or Algebra II during their senior year (51% and 60%, respectively).  The claim that students who take math as seniors are less likely to need remediation is simply not true.  In general, the need for remediation depends on the highest level of  math course attempted. Students who are behind in high school (according to California State Standards**) tend to need to remediate in college. These students generally lack fundamental math skills from K-8. For more, see Sankey Diagrams of Flow from High School to CSUN Math Courses.

Pass rates in Developmental Math and some 100-level math courses for 12th grade students who took math are the same or worse than than students who did not take math.  Students who started in the lowest level Developmental Math course, M092, continue to underperform in later math courses. For more, see CSUN Math Pathways from Developmental Math to some 100-level Courses.

Pre- 2016, most CSUN students place into Developmental Math or Early Start. For more, see Visualization of High School Math Courses and Placement in CSUN Math Courses.

High school of origin matters.  Although we only looked at a small sample, it appears that high school math grades alone are not an equal predictor of student success in math courses at CSUN.  For more, see Does High School of Origin Matter?

How good a predictor is the math SAT/ACT scores in CSUN math courses? Are the EO 1110 Guideline cut scores in line with CSUN student performance?

  1. Performance on standardized tests such as the Math SAT (old version) and Math ACT depends on the highest math course taken in high school.
  2. The cut scores set by EO 1110 are not well correlated with CSUN student profiles for passing various courses.
  3. The cut scores set by EO 1110 will continue to place many students in stretch or other supported math courses, so there is likely to be little decrease in the time to degree.

How many CSUN students scored above 500 on the old Math SAT?  We use a cumulative histogram to determine how many data values lie above or below a particular value in a data set. (Cumulative frequency can also defined as the sum of all previous frequencies up to the current point.)

The specific cutoff scores listed in the EO 1110 guidelines for placement have not been piloted. We wonder what analysis, if any,  supports the specific guidelines.

Overall the SAT Math scores trend has not been uphill.

 

Perhaps the SAT changed to a new format in 2016 to increase SAT scores as part of the national initiative to increase the college graduation rates? For more, see Study of SAT/ACT Math scores by high course, placement and pass rates at CSUN, and EO 1110 cut scores.

Interesting Comparison

In contrast the Chancellor’s Office reports that proficiency rates have been on a steep rise, as have the Mean High School GPAs  of FTF Proficient students.

For more on what’s wrong with this picture, read on.

The EO 1110 Guidelines are not really very good indicators of our student success path.  After looking at our students records at CSUN, and using the students highest level of math at high school, we came up with our own guidelines.  Our sense is that it would be best if each campus designed its own guidelines based on their typical incoming student population and knowledge of their own courses and level of difficulty.

Discussions about Implementation and Implications of EO 1110 

The more “competitive” entry becomes, the more likely the “opportunity gap” will align with admissions criteria.  The system is aiming for a ten-year increase in the six-year rate from 57% to 70%, and, of course, policymakers want to see a boost in the four-year rate, too. The Senate Select Committee on Student Success is likely to ask for more curricular changes in the future. Faculty do not feel that sound or equitable academics can be dictated from Sacramento or the Chancellor’s Office.  For more, see Broader considerations in terms of shared governance and EO 1110.

The CSU system has 23 separate campuses where direct instruction is not equitably funded. Students with the highest level of preparation can attend impacted campuses where there is higher cost for instruction per FTE.  Those with weak high school preparation can attend campuses with lower retention and graduation rates where there is less funding for instruction per FTE. There is an overall inequity in the system which will not be solved by EO 1110. Separate, but not equal. For more, read  CSU Systemwide Impact of EO 1110.

Students feel more positive about their collegiate experience and have a psychological edge by taking stretch courses rather than being placed in Developmental Math. Students may prefer the content of the courses which is problem solving in practical situations. Therefore, some of the goals of EO 1110 should be given serious consideration.

Presumably, the CSU Chancellor’s Office envisioned some cost savings of 25-33%; otherwise, EO 1110 fails to work for a budgetary standpoint. However, we do not see much cost-savings, nor a significant reduction in time to degree because the majority of students will be placed in  Category IV.

Rushed implementation “forces” CSUN Math Department to pass huge curriculum changes with less than an hour discussion. The Math Department Implementation Plan consists of new courses which do not yet even exist or have not been redesigned. For more, read An Example of “Shared Governance”: CSUN’s Math Department’s EO 1110 Implementation Plan . There was no time to discuss the impact on FTF, or on instructional positions, much less to gather evidence of student success that should contribute to the Math Department discussion, such as the one on 12/6/2017.  For more, see CSUN Math Budget Considerations and Instructional Positions 

Discussion of the Data Problems

The authors of this document started with 7.7 million data points from the CSUN application data and student records. We filtered on the data for only students who matriculated at CSUN, took a first math course and had a HS GPA that we could calculate. This reduced more than 54,000 distinct student records to around 24,819 records. We list some of the problems we encountered with the data:

  1. Vast amounts of data are missing. 90% of high school students do not report their senior year grades. (The grades are self-reported.) EAP records are missing.
  2. Many course names are non-standardized. A new application is being used this year, 2017-2018; we hope this solves some of the problems listed. However, not all high schools participate by sending in their course names.
  3. A new SAT was introduced in 2016. All our estimates use (old) SAT Math.
  4. Currently, not all transcripts are submitted electronically and verified against the application data, and course placement has to be decided in advance on senior year grade submissions. The timeline is problematic. For more, see Problems with CSUN Application Data and Student Records.

Conclusion

The Executive Order EO 1110 attempts to provide “for the broadest utilization of multiple measures in assessing academic readiness and determining course placement for first-year students. The Early Start Program is recast to allow students to focus on a single discipline and acquire necessary foundational content at the same time they earn baccalaureate credit.”  The manner in which it was introduced triggered a backlash from the faculty.  The CSUN Faculty Senate (9/28/2017),  the only official voice of the Faculty at CSUN, passed the following motion:

The Faculty Senate of CSU Northridge and its Standing Committees will not participate in the implementation of Executive Order 1100 (revised) and Executive Order 1110.

On every level other than political expediency, the Executive Order 1110 falls short. Vast amounts of “multiple measure” data are missing. There may be serious problems with our own institutional records. The new CSU application is not tested. The new SAT scores are not understood. The  EO 1110 Guidelines are not adequately vetted. Cutoff scores for the SAT/ACT are questionable; SAT/ACT scores vary greatly from campus to campus.  EAP data is missing.  The Executive Order 1110 violates shared governance.   Large scale curricular changes are not going through the normal curriculum process. The EO 1110 effect on instructional positions or cost may be neutral but that depends on each campus’ implementation plan. The short timeline will stress campus resources and instructional staff, Institutional Research staff, Undergraduate Studies staff, Admissions and Records staff, and scheduling and room placement staff with unknown impact on student success. At best, the implementation of EO 1110 is rushed – unnecessarily rushed.

We surmise that the Chancellor’s Office has farmed much of the multiple measures work to external consulting firms who work with the California Community College System: the RP Group, Educational Results Partnership, and CalPass Plus. These groups have no expertise in the placement of CSU students whose students’ preparation varies greatly from campus to campus. The Chancellor’s Office recently signed a systemwide contract with EAB, as if they can solve all the problems mentioned in this report.(At least now, the Chancellor’s Office has some other organization to blame for all the problems it failed to solve over the years.)  Course placements and participation in the Early Start Program will be communicated to a student via updates made to PeopleSoft CMS.  Student placement in Category IV  (Early Start required) will be available in late-March/early-April 2018;  therefore, neither transcripts or senior grades will be used for placement determination.

The CSU project to design remediation in order to improve the graduation rate is long overdue. However, the system and state still have to confront major problems to succeed. The CSU severely under-funds direct instruction at those campuses with the students most in need of additional academic attention and support. There is troubling evidence that gains in graduation rates do not address persistent gaps in basic skills by  race and ethnicity. We strongly support LAO’s 12/20/2017 suggestion that the Legislature consider weighted student formula which  includes a base per‑student funding amount, supplemental rates for each low‑income and first‑generation student.

We are all deeply concerned with and invested in student success, quality education, and efficient and effective use of state funding.  Let’s spend a little more time thinking about the data. Then we should spend a lot more time connecting the data to student learning and teaching practices.  More inclusion at the level of the faculty and staff will achieve better results.

Our future work will include formulating a clearer picture of student profiles and pathways with lead to optimal success and creating better advisement tools.

Acknowledgements

References

 

 

Leave a Reply