Jack Schneider - Beyond Test Scores: A Better Way to Measure School Quality

THE IDEA that I would have to move was ridiculous, but thats what the numbers kept suggesting.

In the spring of 2013, the Boston Globe created an online tool called the Dreamtown Finder, which proposed to help people choose a place to live. The tool prompted users to rank values in six categories, and then produced a customized list of Massachusetts cities and townsostensibly ordered by fit. Naturally, one of the categories was schools.

Like many people, I rated the schools category as very important. I did the same for fun, location, and housing cost. I enjoy living in a diverse neighborhood, so I toggled the people like me bar down. I was neutral on the remaining variablehipster.

The site told me to pack up my family and movea quarter-mile west, into Cambridge. I clicked the back button, determined to stay in Somerville, where I currently reside and am quite happy. I altered my rankings, and resubmitted, but I couldnt make it work. That is, not until I dropped my concern with school quality down to zero.

The Somerville schools certainly arent perfect. Parents, teachers, and principals all have wish lists and complaints. However, if you spend enough time in any school district in America, you will learn that every community has its own set of priorities and concerns. People care enough about education to want more from it.

Yet few people in Somerville seemed unhappy with the schools. Like Americans in general, residents of the city tend to express relatively high levels of confidence in the schools their children attenda surprising fact given the doom-and-gloom rhetoric that dominates headlines about the nations educational woes. From what I could see, their general satisfaction seemed warranted. Still a relative newcomer to the city, I had yet to visit every school, but I did know several of them, and I had already begun to work closely with the school across the street from our housea K8 elementary school that my daughter now attends. We were also, at the time, preparing to enroll our daughter in the districts early childhood centera diverse school with what seemed to us a solid mix of traits. The faculty was warm and caring, children seemed happy, the building was well kept and decorated with student artwork, and students appeared engaged in activity as we walked through on a tour.

Of course, judging a school from the outside is incredibly difficult. In lieu of decent information, parents often rely on imperfect measures of qualitya new computer lab, for instance, an array of impressive-sounding classes, or a list of prestigious colleges attended by graduates. No one knows this better than the leaders of private schools, which must sell their clients on the idea that educationfree to all children in the United Statesshould actually come with a hefty price tag. Go on a private school tour, and you will see lots of bells and whistles, but you will learn less than you imagine about what your childs actual experience at that school will be.

My wife is a teacher and the daughter of a retired administrator. I am a former high school teacher who left the classroom to become a professor of education; much of my research is in school quality. As such, we thought of ourselves as relatively savvy. Still, we wondered: Why would anyone think the Somerville schools are bad?

There were a few reasons.

Most obviously, the districts raw standardized test scores were relatively low.

Many parents also tend to use race as a proxy for school quality. Knowing that students of color have long been denied equal educational opportunities, middle-class white parents often shy away from schools with large concentrations of black and brown students. In so doing, they exacerbate segregation, take their high levels of capital elsewhere, and ensure that people like them continue to avoid schools with large populations of color.

I hoped that the Globe wasnt using raw test scores to measure school quality, and though I was relatively certain they werent using race as a variable, I was curious enough that I dug into the methodology.

As it turned out, schools were being ranked by SAT scores and teacher-student ratios. The former, as research has borne out, is more strongly correlated with socioeconomic status than it is with college performance. The latter measure, beyond being generally misleadingit counts the number of adults in the building rather than the average class sizeis often not meaningfully different across schools. Consequently, it can lead to perceived differences where there may be none.

These were bad measures of school quality. But I wasnt particularly surprised. Most available measures of school quality are weak, and user-friendly data interpreted by third parties tend to be even weaker.

I wasnt in a position to offer much more than criticism. Nevertheless, I began using my newly minted Twitter account to go after the Globe. Tools like this, I tweeted, could be powerful if the methodology werent so simplistic. I included the link to the Globe website.

It wasnt much in terms of public engagement. But to my surprise, it led to several interesting conversations, and eventually produced a response from the designer of the tool, who contacted me directly. His question, in essence, was a challenge: Do you think you can do better?

Two weeks later we were sitting in my office at the College of the Holy Cross, a small liberal arts college forty miles west of Boston. I suggested a few measures I thought were better gauges of school quality, as well as some ways of adjusting for the different kinds of populations various schools work with. We looked at the statewide data available to us, and though they were limited, we agreed that we could certainly do better than what had been included in the Dreamtown Finder.

Using educational research to inform our decisions, we picked several measures that captured more about school quality. And recognizing the fact that rating schools is an inherently subjective enterprise, we designed a model that allowed parents to customize rankings based on personal valuesvalues that, in our version of the tool, included academic growth, school resources, college readiness, school culture, and diversity. Rather than assume a single vision of a good school, our tool asked users to distribute an allotment of one hundred points, in five-point increments, according to their own personal values. Thus, a parent who values the diversity of the student population and also wants to assure that his child will be college-ready at the end of high school could weight these two variables more heavily than the others. Alternatively, a parent concerned first and foremost about academic growth could allocate all of her points to that category.