A Single Point of Data: Avoiding the Telephone Game

Cartoon courtesy of thadguy.com

Over the last couple of years, I have seen the following quote used to bolster arguments against data: "There are lies, damned lies, and statistics".  The quote is often attributed to mark Twain, however, the original source is contested.  The actual quote that Mark Twain made in "Chapters from My Autobiography" (1906) was "Figures often beguile me particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: 'There are three kinds of lies: lies, damned lies, and statistics.'"

Like this quote, we often only depend on part of the data when we do interpretations, leading to unnecessary misunderstanding and caustic arguments.  The simple fact is that there is not one single data point that can tell a story on it's own.  Some people will say that the argument is about the numbers and will follow up with several anecdotes: the issue with this is that anecdotes are data themselves.

As administrators, our role is to create a story about our school's performance, using multiple data points and to ensure that these data points accurately portray our schools.  As we build our stories, we should make sure that we are clear about:

1.  The meaning of the data point.  Whenever data points are published, a definition is also published.  The interpretation of that data point is limited to that definition.  A single data point can contribute to an evaluation, but cannot serve as an evaluation itself.

Our role as administrators is to make sure that everyone understands the definitions of the data points that are used.

2.  Clusters of data tell stories, individual data points do not.  Summative data is a great starting point for understanding your school, but it most likely does not tell the whole story.  Use summative data as a starting point to find out your school's story.

I used to crunch data for a group of schools, and you would be surprised how far off base people's beliefs were about a school that they were sitting in based on the summative data that they received. They accepted the summative data even though their day to day experiences contrasted with the data.

Is your attendance really low, or are there data entry errors?  Do you have a large number of students cutting, or are your offices forgetting to submit attendance for them?  

How many of your students were within 1-2 questions of meeting proficiency? 

What percentage of your students have been disciplined and what are they being disciplined for? 

Our role as the administrator is to tell a story with the data, not just to report out what is given to us.

3.  Alignment: Systems, resources/training, processes, THEN people.  Summative data is OUTCOME data.  It is a reflection of SYSTEMS, not people.  This is why the role of the principal is so important - we lead the design of the systems.   

It is important to make sure that the data points you use align to the appropriate level (i.e. standardized test scores can be used to identify curriculum issues (system), but not resource or teaching issues), so it makes sense to people when you explain your strategies and to your staff as they work day to day. Weaknesses in resources/training, processes, or people all point to some system flaw.  Correcting at any of these levels may create short-term gains, but only system changes create long-term gains. Strategies based on changes or replacement of people are the riskiest and can cost you the most - that's why it's important that data is not used to blame people but to correct structures.

Failure to ensure the correct alignment leads to an over dependence on individual data points and misinterpretation.

Our role as the administrator is to make sure that the main thing is actually the main thing.

Many people are intimidated by statistics and this can lead to a multitude of issues.  As administrators, becoming data proficient can be a big support to our stakeholders and help everyone stay focused on improvement rather than blame.

If you're new to working with data, you may want to check out my post about managing your school data, "Getting Muddy: Personalizing Your School's Data",
http://principalinstruction.blogspot.com/2014/12/how-to-personalize-your-school-data-get.html