Some thoughts about strategy-based test prep

When proclaiming that the SAT and the ACT are not tests that can be effectively coached, the College Board and the ACT like to trot out the following statistic, courtesy of the National Association for College Admissions Counseling Existing academic research suggests average gains as a result of commercial test preparation are in the neighborhood of 30 points on the SAT and less than one point on the ACT, substantially lower than gains marketed by test preparation companies. Lets take a moment and unpack this assertion. First, one of the key words here is commercial test prep (e.g. Kaplan and Princeton Review); nowhere is tutoring through boutique companies or private tutoring mentioned. As someone who has helped more than one student increase their verbal  scores alone by 350+ points on the SAT and 10+ points on the ACT, I have some grounds for disputing the idea that the shortcomings of commercial test-prep should not be extended to test-prep in general. Thats not, however, what I really want to focus on here. What interests me, rather, is the idea of average gains and the way in which that average was determined. Ive been thinking about this thanks to Debbie Stier, who put up a very interesting blog link to the following article by cognitive psychologist Daniel Willingham, a professor at the University of Virginia. Willingham makes the point that: When a teacher presents a reading strategy to students, we can assume that there are three types of students in the class: students who have already discovered the strategy (or something similar) on their own, students who are not fluent enough decoders to use the strategy, and students who are good decoders but don’t know the strategy. Only the last group of students will benefit from reading strategy instruction. When a researcher finds an average effect size of d=0.33 for teaching students the strategy, that effect is probably actually composed of many students who showed no benefit and a smaller number of students who showed a large benefit. I think that something very similar is going on in many strategy-based prep classes. My guess is that only around half  of the people who take those tests (those scoring 500+) actually have solid enough literal comprehension skills to even make any sort of strategy-based prep worthwhile. What this means is that if someones comprehension skills are truly up to par (meaning, more or less, that they can pick up a College Board Critical Reading passage at random, understand the gist of it, and summarize the main point and tone), they actually stand to benefit immensely from strategy-based prep. It probably wont help for the ones scoring 750+ from the start because theyre already using many of the standard strategies, even unconsciously, but for many of the still-small percentage scoring in the 600 to low 700 range, the increase can be very substantial. Many of the ones who persistently score in the 500s, however, wont succeed in raising their scores at all because they lack the core skills on which to base the strategies they learn. This points to a disturbing conclusion: the real problem isnt that people can game the test by learning strategies (aka tricks) but rather that many test-takers dont even even have strong enough comprehension skills to be helped by those strategies in the first place.