9.26.2009

Are Oklahoma Students Really This Dumb? Or Is Strategic Vision Really This Stupid?

Although the evidence continues to mount that there is something very funny about Strategic Vision's process, for the most part there has not been too much reason to question their results themselves, which have tended to play it safe and straight down the fairway. Strategic Vision was rated as a pollster of roughly average accuracy in our pollster rankings, which were based on results through the 2008 primaries.

As Tom Jensen at Public Policy Polling notes, however, it would not be that hard to manufacture the results of an election poll. Just look up the average at RCP or Pollster.com, or 538, tweak upward or downward a couple of points depending on your whim, and you're good to go. But once you venture outside of the bubble of electoral politics and into an area where you can't copy off your neighbor, there is potentially more room for a dishonest pollster to get themselves into trouble. Here, then, are a few oddities from a poll that Strategic Vision recently conducted for an educational thinktank.

The poll in question comes from the Oklahoma Council of Public Affairs (OCPA), a conservative-leaning thinktank that recently commissioned Strategic Vision, LLC to conduct a poll of 1,000 Oklahoma high school students. (A similar poll has previously been conducted by Strategic Vision, LLC in Arizona). The poll asked ten relatively basic political knowledge questions that were drawn the U.S. Citizenship Test, such as: "How many justices are on the Supreme Court".

Only 2.8 percent of Oklahoma's high school students passed the test, claim OCPA and Strategic Vision, which is defined by having gotten at least 6 of the 10 answers right. Moreover, the results to some particular questions were strikingly low. Ostensibly, only 23 percent of the students correctly identified George Washington as the first President, and only 43 percent correctly named the Democrats and Republicans as the two major political parties (11 percent of the students, COPA and Strategic Vision claim, provided the answer "Communist and Republican").

For me, some of these results don't pass the smell test. I agree that public schooling in the United States needs to be improved, particularly in the areas of government and citizenship. But only 23 percent of high school students in Oklahoma knew that George Washington was the first President? Really? I have difficulty accepting that claim at face value. In 2008, 68 percent of Oklahoma fifth graders passed the Oklahoma Core Curriculum Social Studies Test. You can read some of the questions on that test beginning on page 50 of this PDF; they're generally quite a bit more difficult than the ones that Strategic Vision asks. (For instance, "Which was the most profitable export of the Jamestown settlement?" and "Which group would most likely agree with ideas presented in Common Sense?"). So either those smart fifth graders were really forgetful by the time they got to high school, or there's something very wrong with this poll.

But let's put that aside for a moment and do an examination of the math. Here are the results of the poll, as taken from OCPA's website:



When I first saw these results a couple weeks ago, they really got my spidey sense tingling. Forget about the overall level of knowledge being low -- what I found strange was that there were no students, out of 1,000, who answered as many of eight out of the ten questions correctly. Isn't there some total nerd in Tulsa, some AP Honors student in Stillwater, who was able to answer at least eight of these ten very basic questions correctly? The distribution seems to be too compact.

Let's run a couple of simulations to test the robustness of these results. In the first simulation, I'll assume that:

(i) the student body is homogeneous -- everyone is as knowledgeable as everyone else, and
(ii) the questions are independent of one another; so knowing, say, who wrote the Declaration of Independence doesn't make you any more (or less) likely to know what the Bill of Rights is.

These are completely unrealistic assumptions, which, as you'll see, is the whole point.

But stay with me for a minute. To conduct the simulation, we'll create 50,000 "students", and they'll randomly get the questions right or wrong based on the percentages in Table 1. So, when we ask, for instance, who was the first President of the United States, they have a 23 percent chance of correctly guessing George Washington and a 77 percent chance of getting the question wrong. Then we'll add up the results for each student and see how they did.

When we do this, the results are strikingly close to the ones Strategic Vision produced in Table 2:



But, here's the problem: these are not realistic assumptions. Students in public high schools do not all have the same achievement levels. Moreover, the fact of having gotten one question right almost certainly does have some bearing on your odds of getting another right.

So, let's undertake a more realistic set of assumptions. To do, we'll divide our simulated students into thirds. First, there's a low-knowledge group; these students' chances of getting each question right are diminished by 50 percent. Then, there's a high-knowledge group; these students' chances are increased by 50 percent. Finally, there's a medium-knowledge group; these students' chances are exactly as listed in Table 1. So, for instance, for the question about correctly identifying the two major political parties, students in the low-knowledge group have a 22 percent chance of getting it right, the medium-knowledge group a 43 percent chance, and the high-knowledge group a 65 percent chance.

If we simulate the results again with our now more heterogeneous student body, here is what we get:



In this case, the results provided by Strategic Vision do not do a very good job of capturing the likely distribution of responses. The simulated distribution is more spread out, with more students getting 0's and 1's but also more getting 6's and 7's, etc. Meanwhile, the peak around 2-4 answers correct is less prominent.

A slightly more robust procedure might be to assume that students' aptitude is normally distributed. In this last simulation, we will assign a bonus or penalty to each student's chances of getting the questions right, where one standard deviation is equal to a bonus or penalty of +/- 40% on the chances of getting a particular question right (so a student one standard deviation above the norm would have a 32 percent chance of getting the "first President" question right, rather than a 23 percent chance). This produces a graph very much like the last one:



I'm not sure if there's any a priori way to know what the underlying distribution of responses "should" be. As a very rough guide, on the reading portion of the 2008 SAT, a single standard deviation corresponded to a difference of about 17 questions out of a 67 question test (ignoring the penalty for wrong answers), or about 25 percent of the total. The standard deviation implied by Strategic Vision on this citizenship test is only about half that -- 1.3 questions out of a 10 question test, or 13 percent. But the two tests, of course, are not directly comparable.

It seems quite strongly possible, nevertheless, that the students polled for this survey don't exist anywhere in Oklahoma but instead on a hard drive somewhere in Atlanta. This is a valuable exercise undertaken by the OCPA. But they owe it to the hardworking students of Oklahoma to make sure that their contractor, Strategic Vision, didn't flunk its own citizenship test.

187 comments

shiloh said...

Yes and yes!

carry on

Matt said...

Nate - I'm not really sure what you're going for here. It seems to me that the case for SV LLC's funny business was much stronger on the national issues polls.

It very well might be the case that they actually do conduct fair polling on *some* of their polls - like the one for Oklahoma schools, but fudge the national political issue ones (b/c they take a wide range of jobs).

Matt said...

Perhaps to make my point in a more effective way....if the students were not given a multiple-choice set, and were instead prompted to write the answer essay-style, I think it's plausible that no one got more than 7 correct. Oklahoma has a dismal secondary education system, as do practically all states where the Republican takes more than 58 percent in a presidential election.

Mongo said...

I do find it EXTREMELY unlikely that not one Oklahoma student got more than 7 out of 10 answers correct on this test. I am not even an American, but I knew all ten answers without having to think all that hard.

I would think that there would be SOME non-zero number of kids who had an interest in the US political system, and who would therefore be able to correctly answer these ten questions.

Quadrivium said...

Somewhat off-topic, but here's a testimonial from the Strategic Vision website that I find a bit amusing:

"Strategic Vision allowed me to stay competitive even when underfunded 2 to 1 by my opposition."
--Judy Youngblood, Candidate, School Board, Duval County, FL

"Underfunded by my opposition"? That would seem to mean the opposition spent less money than she did...

http://strategicvision.biz/political/contact_us.html

MondSemmel said...

"(11 percent of the students, COPA and Strategic Vision claim, provided the answer "Communist and Republican""
That really says it all. There's no way whatsoever that this poll had real people answering it.

Smitty said...

If, by some exceptionally odd chance, this data represents a genuine poll...what a sad, sad statement on the educational system!

Matthew said...

@Matt, no matter how bad Oklahoma's education system is, there has got to be at least one student out of 1000 who can get more than 7 questions correct.

BrettT2466 said...

We don't need to guess about the knowledge of Oklahoma student. Under No Child Left Behind, states have to test students and report statistics. Picking at random the George Washington question...

Oklahoma does state testing for history after fifth and eighth grade. The core curriculum guide for fifth grade is here:

http://sde.state.ok.us/AcctAssess/pdf/Core/itemspecs/Gr5SocialStu.pdf

... and for eighth grade is here:

http://sde.state.ok.us/AcctAssess/pdf/Core/testspecs/Gr8History.pdf

Both expressly cover George Washington. (He is also covered in the second grade guide, but they don't test history after second grade.)

The Oklahoma Department of Education standardized test reports for elementary and middle school can be found here...

http://sde.state.ok.us/AcctAssess/pdf/Core/testresults/StateResults2008.pdf

Pass rates for history were 68% in Fifth Grade and an identical 68% in Eighth Grade.

So it's a pretty fair guess that at least 68% of Oklahoma Fifth graders and Eighth Graders know that George Washington was our first president.

And SV would have us believe only 23% of Oklahoma's high school students knew that George Washington was our first presidents?

Juris said...

Medical labs that cheat in performing tests sometimes use what's called a "sink test." They dump the sample down the sink and write up the results of the test any way they wish.

SOMEWHERE in Oklahoma high schools there is a Mark Grebner or a Nate Silver. But they couldn't get more than 7 of the 10 questions right?

(Well maybe they decided to blow off the test. And maybe if I lived in Oklahoma and got such a simple test, I'd blow if off, too.)

Anon said...

WHy not send this post to Markos and ask him to commission a Research 2000 poll of the same questions. I think at least 1/1000 OK students will get 100%

eddie said...

This is such exciting work you're doing. You should be commended for pushing through to this story.

Woody (Tokin Librul/Rogue Scholar/ Helluvafella!) said...

Were there any consequences for not answering correctly?

Were there any rewards for success?

American students are very test savvy. They know if there is something at stake and if there is not, they're likely to just blow it off...

It's a way of saying "Fuck you" to the testers... Indigenous peoples used to do the same thing with intrusive anthropologists...

You have no idea how many tests kids today take, and how overwhelmingly unimpressed they are about them...

Dan "The Man" said...

Oklahoma public school student here (well 13 years anyways). Yes they are that dumb, and it is the worst kind of dumb, willful. The state is an entire society of willful idiots waiting for what they think is coming, the rapture.

Steven said...

I think this is the best post on this site since the election (and I don't mean that as an insult). Great work.

Liano said...

Anecdotal information:

I happen to have three HS students in the house tonight (two 2nd years and one senior) and gave them all the 10 question test. It wasn't clear if the SV LLC poll was multiple choice or not, so I gave it to them without giving them any answers to select from.

The results:

One got all 10 right.

One got 8 right (missed the first question and the 6th)

One got 7 right (missed the 1st, 5th and 8th questions).

So in my small sample of 3, 2/3rds did better than all 1000 supposedly in the SV LLC poll, and 1/3rd did better than all but 6 out of that 1000.

I would note that the one who got all 10 right clearly has an interest in politics (I would have predicted he would get all of the right), the other two do not have any such interest. But even they did much better than the SV LLC poll would suggest.

Again, purely anecdotal, but I thought you might find it interesting.

Mike in Maryland said...

MondSemmel said...
"(11 percent of the students, COPA and Strategic Vision claim, provided the answer "Communist and Republican""
That really says it all. There's no way whatsoever that this poll had real people answering it
.

Considering the political situation in Oklahoma (Senators are Coburn and Inhofe, Representatives Sullivan, Boren (Democrat, but a blue dog of a blue dog), Lucas, Cole and Fallin), I would think many would think of the Democratic Party to be 'the Communist Party'. What I wonder about on that account is how many would have answered 'Socialist', surely if 11% responded with 'Communist', at least a few would have responded with 'Socialist', IMO.

But what's even more puzzling to me is that less than three per cent of the students got more than 50% of the questions correct (2.2% got six correct, .6% got seven correct)? When I was in school (granted, during the late 50s through the 60s), you had to answer correctly at least 70% of the questions to even get a D- grade - passing, but just barely. And the vast majority of the students I went to school with got at least a passing grade and advanced to the next grade level (and no, it was not because the teachers took pity on the poor students and passed them - the students had to actually earn their passing grades). I would expect that at least 50% of the Jr. High students would have answered at least 60% of the questions correctly, prior to taking a high school level civics and government class(es).

Mike in Maryland

Wayne said...

I just gave my Michigan high school son the quiz (B student -- hasn't had a government class yet) and he got 8 out of 10 right. Either he's smarter than any Oklahoma high school student or the results are fraud.

Paul said...

Doing a survey of high school students is not like doing a general population survey--how did they draw the sample? The OCPA website indicates a phone survey, so where did they get the numbers to draw the sample?

Veek said...

There was a post I read that pointed out other clues suggesting it was faked.

http://www.dailykos.com/storyonly/2009/9/18/213027/943

One of the points from the post:
1. There are answers all over the map on the number of Supreme Court Justices, but absolutely no one out of 1000 guessed 7 or 11. Every other number from 5-12 were guessed.

My own thought: Regarding the question about the two political parties: Either the question was a fill-in-the-blank, in which case it makes no sense that the only answers provided were Democrat and Republican, Communist and Republican, and Don't Know (These three add up to 100%), or it was a multiple choice and these were the only three choices presented by the pollster, which makes it a incompetently designed question.

jbz7890 said...

I wonder how black kids in Detroit would do on this test.

Nick said...

My guess would be that, in the apparently unlikely event this survey is real, it would have been multiple choice. The fact that none of the sets of answers has an "other" category seems pretty conclusive on this.

Adam said...

Nice data there, Brett.

As an aside, here's an anti-Howard Dean op-ed piece that SV LLC mass-mailed some years back. Class acts, these guys.

http://www.democraticunderground.com/discuss/duboard.php?az=view_all&address=296x2175

Clark said...

It might be the case that almost none of the students took it seriously. I remember when we had to take various survey "tests" in high school about health and lifestyle that a sizable portion of us students represented ourselves as sexual deviants shooting up with our parents, just to be irritating and screw with the results. If it didn't count as a grade, I could easily see my high-school self putting down "Big Bird" or "Homer Simpson" as our first president out of anger that someone was wasting my time with such obvious (and, in that, somewhat condescending) questions.

dbuck said...

Random observations.

First, the OCPA is not exactly friendly to public education, and thus has a vested interest in making publlic schools look bad.

Second, the survey was done by telephone, so how did the pollsters have any idea who they were speaking with?

Third, citizenship applicants pass this same test with at a 92% rate.

Fourth, don't discount the prank answer theory. (Who was the first precident of the United States? Alfred E. Neuman. Etc.)

Fifth, on the other hand, apparently the same survey in Arizona returned similarly dreary results.

Dan

P.S. In spite of OCPA's conservative mien, it does not seem to find that the parents are to blame.

Adam said...

Heck of a tangent, but do you suppose thisis the Judy Youngblood giving the testimonials? Appears to take place in southern GA, not far from Jacksonville, FL (Duval Co.).

Back on-topic, do we have any idea if the answers were free response or multiple choice? The initial reports I saw about this didn't have any of that info, either.


The mother of a kindergarten student, whose paddling led to the arrest of an assistant principal, said Wednesday she's disappointed the official hasn't been suspended.

Inferno said...

@jbz: Classy.

To answer your question: At least 1 out of 1000 would probably get a 10, I think. Even though school populations are self-selecting in most urban settings (for example, in Newark, NJ, there are specialized target public schools like Arts High and Technology High), it'd be hard as hell to find NO students that scored higher than a 7 on a test that's pretty freaking basic.

Even if you just went to the worst general high schools in Detroit (and I'm not talking about special needs schools, such as - say - a juvie-style high), there would be at least one kid that would know - say - at least 8 out of 10 answers.

Statistically, it's pretty impossible in a sample of 1000 to not have at least one who is in the upper 90% of all students, I think. I did it with the upper 99%. The chance that all students were in the lower 99% purely by chance is something like...0.004%. Or, one out of every 25,000 cases. The chance that none of the students were in the top 10% of all students is astronomical (quite literally).

Simply put: Regardless of what our coastal posters (and I myself, as a coastal guy) might say, the people of Oklahoma are not that stupid. It's mathematically impossible.

Which is ironic, because given Johnson's personal politics, I'd think that he'd do this for a poll of inner-city students. ;)

Alex said...

If the questions were asked free response and multiple choice then it is very odd that (or seem so anyway) that 7 and 11 were never picked when at least 50 kids picked every other number between from 5 to 12.

Also on the length of senate term nobody picked an odd length of term? Sure, if you know anything in general about politics and terms in this country but didn't know a specific term you'd guess even years. But the survey results supposedly show that they don't know anything about politics in general.

5 seems a perfectly good clueless guess to me.

shiloh said...

Re: Question #5 as Jefferson based his D of I on George Mason's "Virginia Declaration of Rights" much of it word for word, Mason at least deserves partial credit, I digress ...

carry on

Chris P. said...

Nate: As a fairly recent survivor of public high school, my guess would be that the students did not take this seriously and randomly filled out the answers without reading the questions. Unless this was a graded assignment for a class, they have no incentive to actually put forth any effort.

Or, more sinisterly, they purposefully filled out incorrect answers in an effort to screw with the statistics.

Justin said...

Regarding Tyler's picure
That may be the address at the top of the door but i can't clean it up enough to tell. i don't have the graphic expertise but i'm sure someone here does.

JSZ said...

One of the options for the "supreme law of the land" was the Bill of Rights, which as part of the Constitution should also get some credit.

It seems from the details that are posted on some blog posts that the quiz was actually multiple choice, and that leaves it very poorly designed at least: for the party question, for example, "Democrat and Republican" and "Communist and Republican" were apparently the only options.

Quixote said...

The flawed assumption in most discussion of this sort of "testing" is that the students are genuinely attempting to provide accurate answers to the pollster's questions. I can't understand why anyone with any recollection of high school would believe this for an instant. Perhaps because it makes for a much nicer media froth about the results?

Nate said: "But only 23 percent of high school students in Oklahoma knew that George Washington was the first President? Really? I have difficulty accepting that claim at face value." Me too. However, I have no difficulty at all accepting that only 23 percent of them ANSWERED that GW was the first president. You assume that these two things (knowing and answering) are the same and they are most certainly not in the circumstances of this poll.

If you give a meaningless joke of a test to high school students (or adults for that matter) you will tend to get meaningless joke responses. In addition, of course, to the baseline of incorrect answers due to mistakes and ignorance that you would get from a real test. That failure to take the poll seriously may take the form of answering "C" to every question, or of haphazard responses, or of humor (e.g., the alternative to GOP is Commie). (There were plenty of tempting jokey answers provided in the "multiple choice, " by the way).

The degree to which this warps the results (and just how puzzling it remains that there wasn't even one decent score) depends on the details. There may well be a case to be made for irregularities, but if it is to have any worth it simply cannot include the assumption that the polled students would have diligently attempted to provide accurate answers.

It is both hilarious and frustrating to see so many people, who otherwise seem reasonably savvy, completely flummoxed by a bunch of high school students having a bit of fun with a stupid and arguably insulting poll.

JSZ said...

The suggestion that the results were due to students not answering the questions faithfully may have some merit, but I think this may be less of a problem when the survey is conducted by phone than when it is on paper.

Furthermore, I don't think we'd expect a distribution like that seen in the survey, which as Nate shows would be expected when all students have the same (independent) chance to answer each question correctly. "Don't know" was usually the most frequent response. When students who intentionally gave wrong answers really distorted the survey, we'd perhaps rather expect a somewhat bimodal distribution of the results (one mode for the prankers, one for the people who gave real answers). And I still find it hard to accept that there was no student who answered even 8 questions correctly.

(I also saw that 2% actually said that the Monroe Doctrine was the supreme law of the land. I think that's additional argument that it was multiple choice, as I can't believe 2% would say that when they'd get asked what the supreme law is in an open question.)

John said...

There is no way this poll is credible. No way. The only question is how they cooked it. Meaning did they conduct the poll and then change the numbers or did they just make them up whole-cloth.

And how do you even do a phone poll of high school students?

Re: prank theory... the problem is that you assume that every single smart kid was in on it. Knowing smart kids, it is not remotely believable that they would all intentionally do poorly.

This whole thing is just ridiculous.

dondiaglo said...

John, I've been thinking the same thing about the prank theory. Smart kids love the chance to prove their intelligence, even if just to a pollster.

Ricardo said...

Nate, your methodology is off. You are looking at a posteri statistics, but treating them as a priori. Of course you get the same distribution that they get if you assume their numbers. This will work with any poll like this.

There may be plenty of correlations in this poll. You point out that you expect some students to be "knowledgeable" and will do 50% better on all questions. The problem with this is that you assume the percentage of students who fall in this class and how much better they do. You don't even say how many fall into each class. You have 5 parameters (percentage in each class gives you 2 - the third is free since you need the total to be 100%; and how well each class does relative to the average - the average students don't have to get the average number right! The distribution can be skewed), which is a lot for 10 questions... especially given your assumptions.

Without knowing how each student did, you can't back these things out in a robust way (you could if you had subsamples with Bayesian statistics).

All of that said, I also don't think that this passes the smell test. But your analysis proves nothing.

John said...

"It is both hilarious and frustrating to see so many people, who otherwise seem reasonably savvy, completely flummoxed by a bunch of high school students having a bit of fun with a stupid and arguably insulting poll."

It is both hilarious and frustrating to see so many people, who otherwise may be reasonably savvy, argue something as ridiculous as an entire sampling of high school students chose independently to intentionally screw up a test.

To argue that some high school kids would not take it seriously is to argue the obvious. To argue, as you have to be, that EVERY SINGLE smart kid decided to throw the test is absolutely absurd.

You've come up with a reasonable, if totally obvious explanation for those the 4.6% that got zero right. And you've come up with no explanation at all for the stuff we are actually talking about.

Just John said...

@Quixote:

How quixotic of you to claim that the high schoolers just pulled a fast one on us... yet in their quest to answer the questions wrong on purpose, NONE of them guessed the wrong answers "7" or "11" on the SCOTUS question. Those're some very organized rebels ya got there.

B Elliott Edwards said...

Have you ever been to Oklahoma, I went to elementary school in that god forsaken wasteland of a state. I do not find there results hard to believe at all. I'd be surprised in 23% of HS students in that state could count to ten using their fingers...

dondiaglo said...

Well Elliott, no one has yet been able to explain how SV managed to poll 1,000 students without finding even one who could answer at least eight questions correctly.

Russ said...

It's highly unlikely they didn't find one smart student eager to show off his ability to answer all these questions correctly in a 1000 person sample... I understand the poor results, high schoolers have no incentive to do well on this, however for not one student to care is hard to believe

Ken said...

Well anecdotal information is wonderful, but since 538 seems to be based on hard numbers, let's take a quick gander at some real numbers. According to the ACT testing organization (http://www.act.org/news/data/09/pdf/states/Oklahoma.pdf), Oklahoma students aren't dumb. The percentage of OK students taking the ACT that are ready for college level social science courses is 52% compared to 53% nationwide.

Either SV was AMAZINGLY good at randomly selecting stupid students, or someone cooked the books. I would be interested in seeing what a nationwide survey of high school students showed and then doing an analysis comparing the nationwide numbers to the SV results.

C khripin said...

There is something VERY strange going on with this sort of polling. Check this out: a poll was conducted in November 2008 by Goldwater Institute in AZ. They asked the same 10 questions - and got almost exactly the same answers. This is the article: http://www.azstarnet.com/metro/299259

How could two polls conducted by different people in different states give exactly the same result?? How can the 10 "randomly" chosen questions be the same twice?

Conservative political groups DEPEND on people's outrage with the government, and they are spreading panic by claiming public education does not work!

Nate better start looking at these Goldwater Institute polls too!

tbkent said...

while i find the mathematical analysis of this poll to be interesting, i think the most pertinent question is: wouldn't it be incredibly difficult to fake this poll and avoid the schools detecting the ruse? i mean, presumably this poll would've been administered in the schools themselves... so if some poll results about how stupid the students in Oklahoma are came out, and no students or teachers or administrators remember the poll actually being done, wouldn't someone say something?

Inferno said...

Pretty much that re: they didn't ALL throw the test.

Yeah, some would have. But unless it was taken at one high school, I find it very difficult to see that they all would have thrown it - the chances of them all independently deciding to goof off are astronomical, even if you give a 50-50 chance to a student deciding to goof off.

Hell, even if you assume a student decided to goof off independently on each question, there's still a 62.4% chance that at least ONE student would have decided to attempt to answer all 10 questions honestly. And that's a rough approximation - students who take at least one question seriously are more likely to take the others seriously, and the ones who say that Perez Hilton was the first president of the US are more likely to say that Taylor Swift's VMA speech is the supreme law of the land, senators are elected for life, so on and so forth.)

I wasn't in high school that long ago either. I'm not quite that jaded about the state of public education, though. (Disclaimer: I'm from New Jersey. My school district, while not a top-ranked one like Millburn, was still ranked fairly highly within the state (like top 20%). I'm slightly less jaded about public ed than I should be.)

loner said...

In my experience, there's always at least one smart ass who screws up the curve. Always.

That's some random sample they came up with.

A. Smith said...
This post has been removed by the author.
A. Smith said...

An earlier poster said that you had a one in 25,000 chance to not get a student in the top 1%, and that the chances of not getting someone in the top 10% were astronomical... You said it. the chance that, if you pick randomly 1000 students, you will get none out of the top 10% is of 0.0000000000000000000000000000000000000000000175%, or 1.75*10^-44 %. Ridiculous.

I refuse to believe that there was not one single student, in 1000 polled, that didn't answer seriously.

This coming from someone that, in the US, would still be in High School (Quebec's weird educational system makes it so I'm in college).

Evan said...

I think you can generally safely assume that anyone who gives a joke answer for one question will give a joke answer for all of the questions. So if the bad results were due to people not taking the poll seriously, you would expect a lot of people to get zero answers correct, however according to the poll only 46/1000 students got all of the questions wrong.

Quixote said...

The problem I have is not with the notion that there are irregularities in the poll, it is with the utter foolishness of taking these answers at face value as indicating knowledge. (Or in this case, with the idea that the only obvious alternative to doing so is to think that the pollster cooked the books.)

Also, I am not suggesting a conspiracy or large-scale "prank" or even an either-or response. I am suggesting that none of the students had any incentive to answer any of the questions accurately, and that there are a number of ways in which high school students are likely to respond. To reiterate, a couple of the most obvious responses are to provide the same answer for each question, to "randomly" select an answer for each question, to provide ironic or jokey answers for some or all of the questions (some lend themselves to it better than others).

(Personal experience suggests to me that the most knowledgable kids, the ones who could've done best on a real test, are among the most likely to be tempted to nominate Bush or Obama as the first POTUS, or the like. And the most likely to find the questions insulting and timewasting. But that's just speculation.)

I would agree that the lack of even a single good score raises some questions. The answer to them, however, cannot being with the assumption that the questions were answered in good faith.

Quixote said...

er, that's "begin", not "being"

C khripin said...

Its a ridiculously small sampe, but here is the trailing digit count for the Goldwater poll:
1
2
3x
4x
5xxx
6x
7
8x
9
0xxx

This does not look very random to me, though of course sample is tiny.

Nate should look at this outfit more closely!

dondiaglo said...

tbkent - The answer to your question is no. Even if the test was "issued" in schools and not done over the "phone", unless the pollster released the names of the schools where the poll was administered, I don't see why anyone would say anything. If you're an OK high school teacher and you read this poll, you're just going to assume this was done at another school. Every high school teacher in the state could simply make this assumption and none of them would realize that none of them ever administered such a test.

dondiaglo said...

tbkent - The answer to your question is no. Even if the test was "issued" in schools and not done over the "phone", unless the pollster released the names of the schools where the poll was administered, I don't see why anyone would say anything. If you're an OK high school teacher and you read this poll, you're just going to assume this was done at another school. Every high school teacher in the state could simply make this assumption and none of them would realize that none of them ever administered such a test.

Peter` said...

Curiouser and curiouser

The Blog for Arizona weren't convinced by the GI survey

http://arizona.typepad.com/blog/2009/07/fools-gold-the-goldwater-institute-civics-test.html

What jumped out at me was this ...

"We hired a firm to interview 1,140 Arizona high school students and ask 10 questions drawn at random from the exam given to applicants for United States citizenship."

... which firm was that I wonder?

Juris said...

Come to think of it, I've seen test scores that are skewed like that before -- when the opscan isn't programmed right or reading correctly.

If we assume these results truly weren't derived from a sink test (as I described earlier), and if we assume Oklahoma High Schoolers are normal (will stipulate that), then there could well have been a read error in the optical scanning.

In which case, Strategic Vision is truly stupid if it didn't question the results.

Peter said...

I find it surprising that the two percent which knows of the existence of the Monroe doctrine would be unable to answer 8 questions correctly

Tyler said...

Erm, that Arizona poll that was mentioned IS a Strategic Vision LLC poll. Goldwater was the commissioning organization - "Strategic Vision, LLC conducted the poll of 1,350 Arizona public high school students on November 21-23, 2008."

The pdf of those results is here. The results are disturbingly similar, with some notable exceptions - for example, 11.9 percent of respondents said something other than Democrats and Republicans to the major parties question. There's no breakdown for communists.

Also, the methodology explanation here seems to suggest the questions were open-ended: "The telephone surveyor called a sample of Arizona high school students and read the following statement: "On the next 10 questions, I will be asking you questions about American government and history. Give me your best answer, and it is permissible to respond 'I don’t know.'""

That raises some questions about the complete lack of respondents who said there were 11 Supreme Court justices, or that no one in Arizona thought Obama or Michael Jackson wrote the Declaration of Independence. (There's only a small 1.9 percent "other" category.)

Michael (mbw) said...

So far I think every poster has somewhat missed the main point.

The problem with the reported results is not some particular detail about lack of anyone getting a perfect result, or a disagreement with some particular toy models chosen by Nate. It is that the student-to-student variance (in fact the whole distribution) is identical to what you'd get for zero inter-question correlation. That's an insanely unrealistic condition, though a very easy one to put in an amateur Monte-Carlo program.

All tests of this sort show large positive inter-question correlations, which rigorously lead to larger inter-student variance than reported. That's true regardless of whether or not students are on the average knowledgeable, motivated, serious, etc. Therefore these results are fake. That's a statistically robust conclusion.

John said...

The issue brought up by Mond and Mike and expanded on by Veek with this:

"Regarding the question about the two political parties: Either the question was a fill-in-the-blank, in which case it makes no sense that the only answers provided were Democrat and Republican, Communist and Republican, and Don't Know (These three add up to 100%), or it was a multiple choice and these were the only three choices presented by the pollster, which makes it a incompetently designed question."

This is, to me, independently damning. Absent every other thing that is wrong with this "poll," this single issue would still invalidate the poll and, by extension, the poller. There is no acceptable answer for those results. Any explanation I can possibly dream up leads straight to fraud or utterly crippling incompetence.

The fact that this poll got A LOT of media play should utterly shame the media that played it.

Tyler said...

Oh, and in Strategic Vision LLC's Arizona poll, not a single public school student got more than seven questions right. But those smart private school students did at least somewhat better, with seven getting eight questions right and one getting nine questions right.

Danny Tarlow said...

Hey Nate,
Very interesting articles. I did a bit more tinkering with your three-class-of-achiever model, and I come to the conclusion that if you would have chosen .7, 1, 1.3 instead of .5, 1, 1.5 as the relative achievement of your groups, then the more realistic model actually matches SV's data pretty well. I wrote up the details and posted code, so maybe somebody can double check me:
http://blog.smellthedata.com/2009/09/analysis-of-pollster-fraud-and-oklahoma.html

Norman Kittrell said...

Here is an interesting set of numbers. If the test was a 4 option multiple choice and 1000 people took it blind we would expect these results:
# Correct:
0: 56.3
1: 187.7
2: 281.5
3: 250.2
4: 145.9
5: 58.3
6: 16.2
7: 3.0
8: .3
9: .02
10: .0009

Looks alot like the results of the poll.

Inferno said...

...and I just jumped over to Pollster and saw that I got quoted and then I damn near fell out of my chair.

...'sup Mark. Anyway.

@A. Smith - I will put the caveat on it that it depends on the size of the pool. If the size of the pool is all Oklahoma HS students, then yeah, that holds up. If the size of the pool is HS students at Charlton Heston County High (pop: 1000), then it becomes...less so.

Also, it becomes easier to coordinate a faking with smaller pools. At Heston High, it may be that they all decide to throw the test just to mess with everyone.

That said:

@Quixote - Subtle difference. A lot of us are not saying that OK HS students as a whole can't possibly be that dumb. What we're saying is that they can't uniformly be that dumb so that not a single one - out of a thousand - answered all 10 fairly simple questions correctly. Or, alternately, they can't all have independently decided to screw with the pollster's head.

That's where you're stubbornly refusing to see the point - it is almost statistically certain that at least one egghead would have answered all 10 questions correctly. Most teenagers are assholes, yes (and I was an asshole when I was a teenager, too), but you're making the leap from "most" to "all." It becomes more of a possibility when you're selecting from smaller pools, but even in those cases it's still fairly unlikely. (Less outlandish, but still unlikely.)

tl;dr - You're essentially arguing that every single student polled messed with the pollster, or is a dumbass. Excuse me if I find that hard to believe even of flyover country.

dondiaglo said...

It seems the OK poll was open answer:

"In considering the profoundly awful results of this survey, it is important to bear in mind that an open-answer format represents a much higher standard than a multiple-choice-format exam, even with high-quality exams such as the NAEP. After all, in a multiple-choice exam, the correct answer is sitting right in front of you."

...which only makes the "results" all the more unbelievable.

Tyler said...

@Norman - And yet the Arizona version of the poll seems to suggest the questions were open-ended, which really calls things like the "Communists and Republicans" responses in Oklahoma into question.

Pragmatus said...

Off-topic but important…

A little run down of the bullshit the GOP is spreading about the proposed cuts in Medicare.

Juris said...

Come to think of it, I went to school in Arizona for a year, in middle school. They were about 1 year behind the curriculum of the L.A. city schools. That says a lot.

Nevertheless, this test sucks and something is wrong somewhere to produce these results.

Inferno said...

And yeah, I know, my point towards A. Smith seems to contradict that towards Quixote. But I guess what I'm trying to say is that while it's more likely that if the test was conducted using a few high schools (or even just one), it'd have a higher chance of producing numbers like that...it's still rather unlikely that it'd go that way. Some nerd might have missed the plot, that same nerd might have heard the plot but thought it was dumb instead of funny, so on and so forth.

Pragmatus said...

I think the answer to the funny OK numbers can be found by thinking about who hired SV to conduct its “research”. A conservative think tank would be very interested in finding miserable results from an assessment of the effectiveness of public schools. They could then harpoon not only the teachers’ unions but the entire idea of public education.

SV says itself that it tailors its results to its clients’ expectations. So they either rigged up a poll to produce these dismal results, or they just cooked numbers up without bothering about the polling part.

Case closed.

William said...
This post has been removed by the author.
Juris said...

I went to that FB page with the address in Blairsville. So much political activity going on there! All organized by UCR(a)P -- Union County Republican Party.

Linda Zoe said...

In the Arizona survey, the responses to "name the two parts of congress" were either the right answer, or "don't know." I think it's virtually impossible that no one gave a wrong answer. I think SV LLC realized some problems with the Arizona results and gave a few more wrong answers for the Oklahoma survey to make it more believable.

Norman Kittrell said...

You can say it was open ended but assuming each question has 4 multiple choice answers and we adminstered the test with blank answers were all you can do is guess A, B, C, or D the distribution of the number of correct answers looks alot like the polls distribution of correct answers.
#Correct - Random - Poll
0)-------- 56.31--- 46
1)-------- 187.71-- 158
2)-------- 281.57-- 246
3)-------- 250.28-- 265
4)-------- 146.00-- 177
5)-------- 58.40--- 80
6)-------- 16.22--- 22
7)-------- 3.09---- 6
8)-------- 0.39---- 0
9)-------- 0.03---- 0
10)------- 0.00---- 0

Tyler said...

There's also a lack of responses for the supreme law of the land question in Arizona - it's either the Constitution, the Declaration of Independence of don't know. It's missing all the other things students supposedly answered in Oklahoma (Bill of Rights, Gettysburg Address, etc.)

Tyler said...

@Norman - I'm not sure what your point is. That breakdown doesn't actually mean anything other than getting 100 percent with pure chance is hard. Students are supposedly smarter than pure chance.

John said...

I know many of you are bending over backwards to avoid a rush to judgment, but...

If someone is trying to tell me that out of 1000 thousand students asked to name the two main parties, 430 said democrats and republican, 460 said they didn't know, 111 said republicans and communists and there were NO OTHER ANSWERS... than that someone is lying.

Among the other million things wrong with this is this first question.

"What is the supreme law of the land.

26% The constitution
17% D. Independence
5% B. Rights
4% G. Address
3% E. Proclomation
2% M. Doctrine
41% Don't know"

Bullshit. You are asking me to believe that the question was open-ended but 2% said the Monroe Doctrine and NO ONE IN OKLAHOMA said "The Ten Commandments?"

Bullshit.

PeterAtJET said...

Oh dearie me.

I'm starting to feel a twinge of sympathy for their customers. Outfits like GI have been paying good money for these polls.

Only a twinge though.

shiloh said...

John said...

Among the other million things wrong with this is this first question.

Bullshit.

Bullshit.
~~~~~~~~~~


John, thanx for the hyperbole and reasoned, scientific analysis. Calm down buddy as you definitely sound like a Rep winger who's upset Oklahomans are dumb as a rock! But not to worry as OK should remain solid Rep for quite some time to come, eh. ;)

take care

Norman Kittrell said...

@John Wow good point. How does no one say the 10 Commandments with an open-ended question.

@Tyler My point is that if you assigned the probability of a correct answer and then 'generated' the results you would expect data similar in shape to what they got.

Inferno said...

OT: So I went over to Tom Jensen's entry at PPP's blog, and the first comment?

"YEAH BUT SELZER FUCKED UP IOWA"

Yes, and birds go "tweet."

I do disagree with ducdebradant when he/she says that it'd have been in SV's interest to fake a Hillary third-place showing. I think the opposite would have been true - it should have been in SV's interest to show Hillary coming first as I believe she was not seen so much as the strongest Presidential candidate as the inevitable one. And, more importantly, one of the most vulnerable as she was considered a highly polarizing figure. (The fact that Obama became a polarizing figure himself notwithstanding.)

Anyway.

@Tyler - Norman's point was that the curve matches up fairly well to a binomial distribution of a multiple-choice test where there are 4 answers, one of which is correct.

@John - Hey, separation of church and state. ;) Then again, the adults from that state often seem a bit lost on that concept.

Milltycoon said...

I agree with the above. Case closed. If this was a fill-in-the-blank test in Oklahoma, I would find it very hard to believe that 5 students independently came up with the satirical answer that the "Communist Party" is one of the two major parties. 110 students had the same snarky answer? Absolutely impossible to believe. Even if these kids' parents regularly say that the Democrats are a "bunch of Commies," there is no chance that these kids wouldn't know that the party was still officially called the Democratic Party (Or "Democrat Party" or "Dumbocrat Party" or some other variant.) The _only_ way this answer could be enough in the bloodstream of Oklahoma teenagers is if some Conservative local pundit there is satirically referring to the Democratic Party as the Communist Party, and this has become a common usage term. But if these kids don't know about the Bill of Rights or the Supreme Court or the Executive Branch, they are _NOT_ listening to Conservative talk radio.

This is obviously another fact-inventing snowjob by the Right to discredit public schools.

shiloh said...

Inferno said...
~~~~~~~~~~


Re: Iowa, ah the memories, just want to thank the voters of Iowa once again for making the best choice in their Dem primary, hence, ergo, therefore stopping the inevitable Clinton machine.

Yes we can!

Of course Obama being the best candidate and best campaigner had an effect ;) also.

carry on

shiloh said...

Caucus, not primary ...

ArcadeFire said...

What I don't understand is why they would make the survey so obviously fake? You don't have to be a statistician to instantly realize it's utter bull shit.
I agree with Pragmatus that they would definitely have an interest in making the public school system seem awful, but making it so blatantly facetious seems to undercut their entire goal.

ArcadeFire said...

Also, I just realized, when Strategic Vision is googled, Nate's article is now the third link displayed!
Haha. I'm sure David E. Johnson is none too pleased.

Pragmatus said...

ArcadeFire…

I would guess the “think tank” was relying on the inherent laziness of the American political public, especially those on the right, who can swallow no end of nonsense (death panels, anyone?) without question. How many of those people are going to put down their beer cans and flip off the TV in order to dig out the original source of a particular bit of data?

The organs used to transmit “information” on the right never have to break a sweat in order to get their points across.

John said...

"John, thanx for the hyperbole and reasoned, scientific analysis. Calm down buddy as you definitely sound like a Rep winger who's upset Oklahomans are dumb as a rock! But not to worry as OK should remain solid Rep for quite some time to come, eh. ;)"

Well, you've left me a little confused, shiloh.

I think perhaps you read my post completely incorrectly or you are trying some form of stealth comedy.

Either way, I'm nice and calm, I assure you. Emphatic statements don't have to imply strength of emotion, they can also imply strength of position.

In this case, the emotion is "chill" and the position is "This poll is bullshit." Bullshit being a descriptor that even Jimmy Buffet on Xanax could use.

LinCA said...

Let's not forget that because he published this bullshit, ANYTHING ever written by Matthew Ladner, Ph.D is, at the very least, suspect.

He either told SV LLC to make up this crap, he willfully accepted their garbage or was stupid enough not to question it. Either way, somebody should take away his Masters and Ph.D.

beavis said...

You can say it was open ended but assuming each question has 4 multiple choice answers and we adminstered the test with blank answers were all you can do is guess A, B, C, or D the distribution of the number of correct answers looks alot like the polls distribution of correct answers.
#Correct - Random - Poll
0)-------- 56.31--- 46
1)-------- 187.71-- 158
2)-------- 281.57-- 246
3)-------- 250.28-- 265
4)-------- 146.00-- 177
5)-------- 58.40--- 80
6)-------- 16.22--- 22
7)-------- 3.09---- 6
8)-------- 0.39---- 0
9)-------- 0.03---- 0
10)------- 0.00---- 0


This pretty much says it all. The "poll" is probably fraudulent.

beavis said...

John, thanx for the hyperbole and reasoned, scientific analysis. Calm down buddy as you definitely sound like a Rep winger who's upset Oklahomans are dumb as a rock! But not to worry as OK should remain solid Rep for quite some time to come, eh.

I think in your rush to flame someone you forgot to read what was written. Most likely you read the part where he said people may be rushing to judgment and you stopped there.

Take a deep breath and reread:


If someone is trying to tell me that out of 1000 thousand students asked to name the two main parties, 430 said democrats and republican, 460 said they didn't know, 111 said republicans and communists and there were NO OTHER ANSWERS... than that someone is lying.

Among the other million things wrong with this is this first question.

"What is the supreme law of the land.

26% The constitution
17% D. Independence
5% B. Rights
4% G. Address
3% E. Proclomation
2% M. Doctrine
41% Don't know"

Bullshit. You are asking me to believe that the question was open-ended but 2% said the Monroe Doctrine and NO ONE IN OKLAHOMA said "The Ten Commandments?"

Linda Zoe said...

The question is, how to get the major newspapers that ran the article about the "ignorant Oklahoma students" to publish a follow up article questioning the validity of the data. We can bash the data all we want to here, but most of the US will never hear about it. They will just remember the survey results. People love a scandal, so I think they would be eager to read about possibly shady doings.

Pragmatus said...

Yep shiloh, I think John is on the side of reason. He was calling the “poll” bullshit, and makes an excellent point that I didn’t even think of. How on earth could Oklahoma students come up with the Monroe Doctrine as the highest law of the land, yet none would suggest the Bible or the Ten Commandments?

As ’twas said, Bullshit.

Animal Spirit said...

Tyler (10:26). Couldn't see more myself, but there are also some nice documents in view (and on screens) that someone with top-end graphics and character recognition software could probably decode. Not good to put this type of thing on the web.

As for the test: b.s.

ArcadeFire said...

@Pragmatus
I definitely agree with you on that.

But this did not get a whole lot of coverage when it was released, and would have served quietly and perfectly as a think-tank talking point had it merely made Oklahoma public-school students look stupid.

But all it takes is one popular blogger's analysis of the impossibility of the results for the shit to hit the fan. It will probably be on Huffington Post by tomorrow, possibly the cable networks by next week (minus Fox of course), and with any luck, have some long-lasting consequences for Strategic Vision.

Matt said...

Having had time to reflect a bit, I'm still going with my original answer given at the beginning of this comment thread. If you look at the questions, some of them are worded in a very vague and unhelpful manner. For example: "what is the supreme law of the land?".

That's just an inane and confusing question. I could very well see plenty of people missing that one - especially if it is open-ended. Think about the context here. You're being cold-called by a pollster and someone asks a question about the Constitution in a bizarre fashion. I think it's entirely possible that the poll is real (but just ill-conceived and very poorly worded).

That said, Nate's earlier analysis of SV LLC's national issue polls is still troubling, and still needs to be countered by evidence. I'm assuming that since Johnson hasn't done it yet, the evidence will not be forthcoming.

Toby said...

I remember around the fourth grade, there was a survey asking what we thought were the effects of various drugs (alcohol, marijuana, heroin, etc.). I checked "Very Harmful" on all of them, as any good little boy would.

But one of the items was "Cocaine (Coke)". And I thought, "I drink that stuff all the time!" So I checked the box labeled "Not Harmful At All".

I wonder what the evaluator must've thought.

Jonathan Greenberg said...

So apparently this whole 538 pwns SV LLC has made the CEO claim "...the series of events has caused a panic at his firm and that employees are fearful for their safety."

http://thehill.com/homenews/campaign/60443-national-polling-firm-battling-back-amid-controversy

"...their receptionist was verbally accosted in their parking lot by someone who referenced Silver’s blog."

Personally, I'm blaming Mule Rider for trying to get Nate in trouble...

shiloh said...

John said...

Well, you've left me a little confused, shiloh.
~~~~~~~~~~


Obviously

take care

Quixote said...

@Inferno:

"You're essentially arguing that every single student polled messed with the pollster, or is a dumbass."

I am arguing that these results could not be taken at face value in these circumstances even in the absence of pollster misconduct. The responses to this sort of poll just cannot reasonably be thought to accurately represent student knowledge. (And once again, strawman folk, I am not even remotely talking about some vast prankster conspiracy.)

Might Nate reach the same conclusion taking this into account? Of course. Is Nate taking it into account? No. Naive assumptions like the one I quoted in my first post render his analysis flawed regardless of how intuitively likely you find his conclusion.

Admittedly, I brought this up primarily because I am in general sick of seeing media outlets take such obviously silly results seriously, as though they indicate that the respondents really are so ignorant. It is just annoying to see Nate and others here making simliarly naive assumptions. It doesn't mean they're wrong about Strategic Vision.

PS: I was under the impression that this poll was multiple choice, but others suggest no ... can someone confirm?

Inferno said...

...I'm beginning to wonder whether something's up with Shiloh.

Anyway. So, I did a little detective work, and:

1) I checked the BBB and there are no BBB reports for a Strategic Vision in Blairsville, Atlanta, or any of its other locations.

2) I checked Google Maps, and the one review was...less than glowing. Essentially, an author hired SV to do work for them (PR work) and while initially the customer service was adequate, they stopped returning e-mails near the end of the term of the contract.

This was written in 2007. Sound familiar?

Mike in Maryland said...

Anyone else notice that all the usual suspects are still attempting to defend SV LLC?

They are starting to give (but only slightly) on one point or another, not whole cloth rejecting everything, but in the end, they are still attempting to defend SV LLC, at least how I perceive it.

Mike in Maryland

James Kennard said...

Nate you're still the second best thing to come out of the 2008 elections.

Inferno said...

@Quixote: Normally, I try to be a fairly diplomatic guy, but...are you deliberately missing the point?

It's not about the end results - which is the same reason why my response re: Selzer in IA (the most common defense of SV) was to sarcastically quote The Simpsons. I'm not arguing that the results on their face are wrong (though I despair for our country if they actually are reflective).

I'm arguing that it's damn near impossible for not a single student to get more than 7/10 questions right, and less than 1% to hit that low mark, especially given the difficulty of the test.

True, students would screw with the test. But all of them? Or even all of the smart ones, if we assume that...say, 1% know the answer to all 10 questions, 2% know the answer to 9, and 5% know the answer to 8?

You can't be serious.

Mike in Maryland said...

LinCA said...
Let's not forget that because he published this bullshit, ANYTHING ever written by Matthew Ladner, Ph.D is, at the very least, suspect.

I'm sure what the Goldwater Institute is trying to do is to get out the meme that 'private schools are ALWAYS better than public schools'. What I find extremely amusing about Ladner's write-up is when he states that:

1. When immigrants attempting to pass the citizenship test must answer correctly 6 of 10 of the questions, and that "the United States Citizenship and Immigration Services . . . recently reported a first-try
passing rate of 92.4 percent). But then he goes on to state:

2. [I'm inserting comments, indicated by italicization "Only 3.5 percent of Arizona high school students attending public schools passed the citizenship test. The passing rate for charter school students was about twice as high as for public school students. [in other words, somewhere around 6%, not quite 7%] Private school students passed at a rate almost four times higher than public school students." [3.5% x 4 equals 14%, so 'almost four times' is less than 14% - a far cry from the USCIS reported rate of more than 92% for first-time test takers attempting to gain US citizenship[.

Looks to me that even private schools (at least in Arizona) are a far, far, far distance from doing their job of teaching the basics of civics, EVEN if we presume that the poll is valid and able to hold up to scrutiny. And the evidence keeps mounting that such scrutiny may cause the test and/or testing method and/or polling method to receive a FAIL.

Mike in Maryland

Mike in Maryland said...

Quixote said...
PS: I was under the impression that this poll was multiple choice, but others suggest no ... can someone confirm?

SV LLC says the questions are taken directly from the Citizenship test on Civics, and the company followed the same procedures as when they conducted the test in Arizona. From the Goldwater Institute report, authored by Ladner:

"We designed a survey instrument containing 10 of the USCIS questions (chosen at random) and administered the survey to Arizona high school students." The report give some description of who took the test (although I didn't see any information on exactly how the test subjects were chosen).

It then goes on to state:
"The telephone surveyor
called a sample of Arizona high school students and read the following statement:
“On the next 10 questions, I will be asking you questions about American government and history. Give me your best answer, and it is permissible to respond “I don’t know.”

No indication of how many, and what, questions were asked prior to the 'questions about American government and history. But look at the final part of the second sentence - it is permissible to respond “I don’t know.”

So, if SV LLC patterned its poll on the USCIS test, it was an open ended poll, not multiple choice. With all the questions that have been raised about the firm, and still very few answers to those questions, one has to wonder if, in fact, the poll WAS given as open answer questions as the USCIS test. For as John asked, if this poll was given as an open answer poll, and actually performed in Oklahoma, why no mention of the 10 Commandments for the question asking ""What is the supreme law of the land"?

I guess you have to draw your own conclusions on whether the poll was open question, or was multiple choice.

Mike in Maryland

Bob said...

I'm surprised that no one has mentioned that the same guy (Matt Ladner) that writes here about the Oklahoma Test

http://www.ocpathink.org/publications/perspective-archives/september-2009-volume-16-number-9/?module=perspective&id=2321

Is also the the VP of Research at the Goldwater Institute? If you compare the PDF from the GI report in Arizona showing a 3.5% passing rate on the test there, which has been linked above in the comments, but here is the link:

http://regulus2.azstarnet.com/pdf/pdfs/560.pdf

parts of it are taken whole cloth from the other.

He uses the exact same quotes from Benjamin Franklin, Charles N. Quigley, Richard Riley, Patrick Wolf, the NAEP (two whole paragraphs of which were word-for-word on both papers), and the USCIS. He also forgot to mark his endnotes in the PDF, and there are some small formatting errors.

I only bring this up because it is interesting, and to better point out Ladner's connection to the two surveys. I don't know how GI and the OCPA are connected other than a few posts by Ladner on the OCPA website(one of which lists him as an 'education scholar'), but if someone else wants to look it up, be my guest.

Ladner himself is also connected with a report from a Michigan Conservative think-tank, link here:

http://www.mackinac.org/archives/2000/s2000-04.pdf

Which lists him as the president of Capitol Research and Consulting in Austin Texas, of which I cannot find any information.

Here's the % correct per question for the OCPA survey, then the GI survey:

1 | 28
2 | 26
3 | 27
4 | 10
5 | 14
6 | 61
7 | 43
8 | 11
9 | 23
10 |29

Arizona Survey:

1 | 29.5
2 | 25
3 | 23
4 | 9.4
5 | 25.3
6 | 58.8
7 | 49.6
8 | 14.5
9 | 26.5
10 |26

And now I am going to bed. Someone wanna pick up the trail on this one?

quiet_celt said...

Haven't commented in a long time, but am a parent with w child just entering the PS system. I must say that my experience with the current system bears no resemblance to that of my era. So, comparisons are out. Still, Having been one of those Nerds and also one of those jokesters (from my time in HS), I am uniquely qualified to comment.

Had you polled me in the 9th grade, you'd have gotten a perfect 10 (along with pretty much every Nerd I knew), even though I was never into politics. You see the thing with Nerds is they are condescending, but fiercely proud of their intellect. To have not one one nerd in 1000 people who would not get a 10 is ludicrous to expect even if you polled only one school. The reason being this being a phone test, and hence there would be zero pressure for any one nerd to answer incorrectly. Nerds are notoriously anti-social to the general school population, because of similar attitude returned.

Now had you caught me in the 11th grade, You;d have gotten a radically different set of answers, maybe. I dropped out in 12th, because the school had nothing left to teach me. Literally. Just got a GED and went to college.

So I can see the joke answers, but I see a disturbing lack of what I like to call the Family Feud factor, that of there seems to be a major lack of the top five answers on the board. ?You take any survey of anyone anywhere you're going to see a distribution of answers.

Peter Wolf said...

I'm not American, but British. The only question I got wrong was the US senate (guessed 4 years not 6).

If I can get 9 right on that sheet, there is no way that out of 1000 students none of them could get 8.

And sorry - only 27% know that the US Congress is the Senate & the House? That seems completely unbelievable. Only 43% know Democrat & Republican after a media-heavy general election and Obama being in the news so much? Bullsh*t.

Blatantly a fake - think Nates first graph of the distributions shows it all.

Might be worth contqacting whoever commisioned the report with this. They can probably take Strategic Vision to court and get their money back unless Strategic Vision can actually provide proof they conducted the interviews (for instance, telephone records of the calls if it was a telephone interview).

Polymeron said...

Unbelievable.

There is NO reason for such a poll to have results distributed as though there's no correlation between correct answers.

Sloppy, SV. Very sloppy indeed. Nate caught them red-handed on this one...

Quixote said...

@Inferno:

As the title of Nate's post suggests, his analysis focuses on a false dichotomy (more or less) between unbelievably dumb students and a "stupid" (misbehaving) pollster. He repeatedly invokes the assumption that the students polled would have tried to answer the questions correctly - that the responses would be a matter of "knowledge" or lack thereof on the part of the students. I have claimed that this assumption is unrealistic. I don't think you've disputed that.

You have claimed that correcting this would ultimately make no difference to the conclusion: that it seems nigh impossible for these particular results to occur naturally. I don't think I've disputed that.

Surtur said...

Do Oklahoma's brighest go to public HS? Checking out the 2009 Presidential scholar list (www.edu.gov\programs\psp\candidates.doc) of the 48 highest ACT/SAT scorers statewide: 5 private schools, 2 sectarian, 5 unknown. Looks like at least 75% of the nerdiest Sooners are from public HS.

Peter Wolf said...

@ Quixote: "that the responses would be a matter of "knowledge" or lack thereof on the part of the students. I have claimed that this assumption is unrealistic."

it's completely realistic. A student who is deliberately doing badly is, in effect, an extremely stupid student - who will correlate getting one question wrong with another one. The correlation remains.

Fact remains, 1000 students wouldn't all be that stupid.

Scott Yates said...

Nate,

You have the power to answer this question!

Unfortunately the statistical analysis couldn't do it, but your followers could. You certainly have enough readers to conduct your own poll.

Ask each of them to give the test to any high school student they know. Give them some basic instructions in being fair, and then build a quick poll on your site so your readers can quickly enter the results.

It won't be totally valid, but it will be at least as valid as the one from Oklahoma, and will give you more data to compare their data to.

(I'll be blogging about this in a minute on my blog, Scott Yates.)

Peter Ammon said...

How do we know that this is the Atlanta-based Strategic Vision LLC, and not the well regarded California based Strategic Vision?

harold said...

Nate -

It would be intriguing to know whether the client who paid for this poll is an advocate of, or especially, would benefit financially from, legislation related to vouchers, charter schools, or the like in Oklahoma.

Jarod said...

So what would it take to defeat the truth defense in a class action libel suit against the original publisher of this poll?

Thatcher said...

Tyler,
Regarding the photo of their office at http://www.facebook.com/photo.php?pid=2831370&id=66419995840

If you examine the name on the window (photoshop or just zoom on the jpeg) you will see that it is pixillated differently than the surrounding window. An obvious explanation might be that a blank window was cut out in photoshop, the name inserted and replaced in an amateur way. The door shows some signs of this as well. I wonder if someone with better image skills than mine could confirm the phenomenon. Is the photograph manipulated?

BruinKid said...

Haven't read all the comments, but for those asking, it was free response, not multiple choice.

JSZ said...

BruinKid: It was really free response, the reported results seem impossible, as noted by several posters here:
-The only two responses given to the party question (in addition to "Don't know") are "Dem and Rep" and "Communist and Rep". I find it hard to believe even that 11% of Oklahoma students would give the nonsensical response and harder that none would give other equally nonsensical reponses (Liberal and Conservative? Republican and Socialist?).
-For the supreme law question, it is odd that no one answered "the Bible" or the "Ten Commandments" and 2% percent answered the "Monroe Doctrine". I'd expect that if you're smart enough to know the Monroe Doctrine, you'd know that it is not the supreme law.
-For the Senate term length question, no one answered five or seven years, even though all other answers close to six had some responses. That is implausible.

If this poll was truly free response, its results seem utterly unlikely even when not looking at the distribution of the results. If it was multiple choice, it was poorly designed, and the problem with the distribution of the results remains.

Michael (mbw) said...

With the exception of Danny Tarlow, nobody here is getting what the main statistical issue is. Sure there are lots of things about the individual question results that look weird. However, we don't know enough about the students or the test to know how serious they were, so the means are very sensitive to subjective priors. Strong positive inter-question correlations for similar questions are ubiquitous, however, regardless of what the means are or the reasons some students do well or poorly. The net variance (and higher moments) of the SV 'results' are far too small to be consistent with any realistic positive inter-question correlations. One of the results of this reduction in the moments is the loss of the high-end tail, which many have commented on but only in an informal anecdotal way. Anecdotes aside, the results are fake.

Danny- I think you've picked an extremely narrow distribution. When you say the results are similar, how do the higher moments compare?

Thad said...

With respect to thatcher's question about the facebook image

http://www.facebook.com/photo.php?pid=2831370&id=66419995840

That pixelization around the logo is a typical characteristic of the discrete cosine transform underlying the Jpeg format. Where there are high frequencies in the image, you get more noise. The DCT uses an 8x8 block, and you will notice that the noise level abruptly changes on 8-pixel boundaries.

Can't deny it as real on that basis. I'm no photoanalyst, but am a visual effects supervisor in real life

Ian said...

No one has really harped on the fact that so many people said I don't know. Could they have loaded questions and told people not to guess? Its absurd to have 40+% of people saying I won't even try something I think is wrong.

Bradford said...

I gotta disagree with you on this one, remember that Texas and OK texrbooks are designed to mislead, and are government approved to do so. I am surprised any of them got 8 right.

Kris said...

I know you were just making the comment for comparison, but the SAT isn't really a fair comparison - the non-college bound students have generally selected out of that. Most states have standardized testing. At least here in CA, it starts in 5th grade, and social studies is a testable subject. I don't know about OK, but you could look at those results (they are released after some time) and see how they relate for SD's and other stats.

Tyler said...
This post has been removed by the author.
rc said...

My twelve-year-old son took the test and got nine answers right. (Missed number of SC justices.)

Of course, he's brilliant and has never set foot in Oklahoma.

infinitum17 said...

I'm relieved to find out that this was faked--the fact that nobody got over 7 questions right gives it away completely. Imagine my surprise on finding out that the same pollster Nate was investigating was the one that conducted the Oklahoma poll that I had been so horrified at. Keep it up, Nate!

Mad Joy said...

Thanks, Nate, for doing these additional analyses. I'm definitely more convinced than before that something very fishy is going on.

However, as much as I love a scandal, I also love the concept of innocent-until-proven-guilty, and I'm not convinced that the survey results were intentionally doctors as much as that the survey and its methodology were just utterly incompetently designed and administered. Plenty of incompetent authors with PhDs do sloppy work with major methodological flaws and wind up getting published in peer-reviewed journals (albeit not the best ones) and all over. And a pollster is usually under considerably less scrutiny than that, other than when being analyzed by fivethirtyeight or reprimanded by polling organizations. Newspapers reporting poll results don't request the same methodology standards as peer-reviewed journals!

So, the argument being made is that the distribution of answers is too close to what you'd get if you chose the results according to a (simplistic and poorly-thought-out) distribution. Still, I think a reasonable explanation of these irregularities could be found from incompetence rather than completely making things up:

- There is still no explanation given about sampling methods. Maybe the pollsters happened to know some kids in Oklahoma, and asked to poll them, and then used an incompetent version of snowballing to ask for their friends' #s, etc - nerds tend to be friends with other nerds, and you could wind up with no "smart, nerdy kids" in your sample. Or maybe they had a friend who worked at a very low-performing school, or for a troubled youth center, or something, and received contact information from them. Now, some of these would be past incompetent and closer to trying to influence the results negatively, but still not to the point of *making up results out of thin air*. I want to believe that students were actually polled, in some way.

- There is also no explanation about an interview script or protocol. A lot of people have been asking whether this poll is open-ended or multiple choice, and have been arguing (rightfully) that the distribution of answers makes no sense for either type of test. Well, maybe it really was in between. Maybe the person on the phone doing the polling was some kid hired out of high school, and not really trained. Maybe students were asked, "What are the two major political parties in the United States?" and then if they answered "uhh.. I don't know... what do you mean" the pollster said something like, "For example, Communists and Republicans" and the student could be like "Oh yeah, that".

Anyway, my point is not for these specific possibilities, but rather that there are plenty of things Strategic Vision, LLC could have done that would have been *just about as bad scientifically* as making up results completely, without just making up the results completely. I think incompetence is playing more of a part here than deliberate corruption.

Danny Tarlow said...

Michael(mbw) -- I might not be understanding you correctly, but the underlying achievement distribution I'm assuming is just a 1/3, 1/3, 1/3 mixture of point masses where the distance from the mean of the lower and upper masses is controlled by the parameter, h. Setting h=0 gives you Nate's first model, and setting h=.5 gives you his second model.

So correlation _is_ induced between student's answers, but there is no correlation when conditioning on the achievement group a given student belongs to. As long as you assume the order of questions doesn't matter, though, I think there's a statistical argument around de Finetti's therorem to be made that this is a reasonable assumption.

Having said that, I'll be the first to agree that there are many other unrealistic assumptions in the model, namely that there are only three groups and that 1/3 of students belong to each group. This is just following Nate's procedure in the first two parts, though.

lovebroker said...

LOL. The "correct" answer to which are the two national political parties is Republican and Democrat. I guess an answer including the DemocratIC party is incorrect according to SV.

Alan said...

Surely somebody on this site is either in the market research racket or teaches a related course. Call 10 or 20 Oklahoma kids. Ask two of the questions. When you're looking at response rates this small you don't need a big sample to reject it. I lack the energy to do the (simple) math, but if the survey shows a 10% correct rate and our idiot survey shows as 30% correct rate--or, better, what we really expect which is close to 70% for George Washington--that's pretty solid proof.

Come on, someone; bully a supplier. Tell him it will be great publicity.

Michael said...

I suspect Nate has uncovered a secret new business model for conservative polling groups.
Since conservatives know ahead of time what results they want (rather than wanting to find out the truth) and the actual work of polling people is a significant expense, it only make sense to fabricate the poll results and save the money that would have been used for the polling.

Justin said...

Most of what's going on in this comment thread is fantastic, but people, PLEASE stop indicating how you (or your children) performed on the test as if it had any value whatsoever, even anecdotal.

If you read this blog you should know enough to know that you've self-selected your way toward much higher probabilities of correct answers simply by being someone who READS political blogs (or for your children, having a parent who does).

This connotes a certain level of interest in the social sciences and could probably be shown, given data, to correlate strongly with correct responses.

Huh said...

How much does it cost to order your own poll from a reliable pollster? Surely there is another polling group out there that could conduct an identical poll with honest methodology and you could compare those answers to the real ones.

I'm sure it costs thousands of dollars, but maybe the readers would chip in with paypal or something. I'd throw you five bucks just because I'm fascinated and want to see how this all turns out.

shiloh said...

Huh said...

How much does it cost to order your own poll from a reliable pollster? Surely there is another polling group out there that could conduct an identical poll with honest methodology and you could compare those answers to the real ones.

I'm sure it costs thousands of dollars, but maybe the readers would chip in with paypal or something. I'd throw you five bucks just because I'm fascinated and want to see how this all turns out.
~~~~~~~~~~


But w/all this publicity, how reliable would a redo w/the same questions be. OK, we're talking Oklahoma, which is probably like N. Korea when it comes to reliable news reporting, but OK probably is hooked up to the internet lol but hs students probably don't surf political sites or read newspapers.

OK, KS, NE, AR, MS, AL ie the birthers, deathers, 10thers, secessionists, folks who watch fixednoise misinformation religiously. ;)

'nuf said!

Grinsnchuckles said...

I'm not a pollster, but I'm reasonably intelligent, and it looks like this is going to end very badly for Strategic Vision LLC once people dig a litle deeper.

I checked the organiation's website and found their office addresses still listed on one of the political pages. I used Google Maps to research those locations, and Stategic Vision does not appear to have offices at any of the locations indicated, except in Atlanta, and that office is listed as closed. Toll free, non local numbers are listed for each location.

I live in Tallahassee, so I'm going to go check that location myself to see if there is a Suite 5 and whether Strategic Vision ever rented it.

2451 Cumberland Parkway
Suite 3607
Atlanta, GA 30339
877-556-0004

1360 Regent Street
Suite 152
Madison, WI 53715
877-287-2141

800 5th Ave
Suite 101-387
Seattle, WA 98104
877-245-0098

2892 Park Ave
Suite 5
Tallahassee, FL 32301
877-878-2680

Juris said...

@Grinsnchuckles: It's been investigated already (see previous threads on this general subject, as well as article by Mark Blumenthal on pollster.com). These addresses are all just mailboxes at best, i.e., mailboxes at UPS Stores.

I think you can look for the one in TLH on google maps in streetview and see the UPS name on it.

didaskalos said...

As an educator, these results do not surprise me in the least. We do a good amount of internal "assessment" (read "testing") in our College, and we regularly get very surprising and counter-intuitive results. E.g. our sophomores regularly perform more poorly on their western civ assessment, even when the freshmen have not had half the material. No doubt many students just don't give a flip about the test by the second year and they just throw it.

The comments re how unimpressed students are with the array of tests they are forced to take should be seriously considered.

Inferno said...

Okay, so I just went back and read a few posts - namely, my own and Quixote's.

While I will give him credit for noting that none of them got a decent score in his first post, the thrust of his argument seems to be that you can't use this as indicative of knowledge. While, yeah, the title of the post states that, and Nate does question the results on their own in paragraph 5...

...the major thrust of his argument seems to be that the answers correlate strikingly to a distribution where there was no correlation whatsoever between right and wrong answers, and the student body is homogeneous. That is, each question is independent. See the curve of his first graph and how well it lines up with a truly homogeneous, independent distribution - and how even just making the sample slightly non-homogeneous (while retaining independence) alters the curve substantially.

The thrust of my argument throughout this thread has been tangentially similar - that it would have been impossible for even the smartest OK civics students to not score higher than a 7, and it would have been very unlikely for all of them to screw with the pollster's heads.

There is a point about poor methodology made by other posters. But on its face...I'll apologize if I've misunderstood him, but it seems as if his argument is that these results can't be taken at face value because kids will be kids. I see his point, but that's not the point of this analysis at all.

Grinsnchuckles said...

@juris: thanks for pointing out the article on pollster. I hadn't seen that one. But I had read the suggestion that each location is a UPS store, and I'm not sure that's true. I checked the addresses when I read Ben Smith's interview of CEO David Johnson, in which stated he took the locations offline because "we've had people show up at our physical offices unexpectedly, including a religious fanatic who had appeared at one office after seeing Johnson on TV."

That statement of course suggests there are actual offices to show up at, not UPS boxes.

Anywho - it all just seems very odd.

Inferno said...

@Grins: Well, read down a bit to the comments section. SV, LLC does have an office...in Blairsville. Which is on the state line with North Carolina and Georgia. And almost 100 miles (and 2 hours away) from Atlanta driving.

(I actually got driving directions off of Google Maps from it and decided to compare. Blairsville is approximately the same distance from Atlanta as Haddonfield, NJ is from New York City. Haddonfield is a Philadelphia suburb. Southeast of Philly.)

Anyway. On topic to the math geek thing: I was kind of thinking of a simulation, and I want others' opinions to see how well it would work. In rough form:

1) Randomly select one of 10 questions out of the list given.

2) Have the simulation select the correct answer by chance. (Where p is the proportion of students who answered the question correctly.)

3) If the answer is correct, apply a bonus of (1-p)/2 to the next question. If the answer is incorrect, apply a penalty of p/2. So, a student that answered the number of Supreme Court justices correctly would get a 45% bonus to the probability of answering subsequent questions correctly, and a student that answered it incorrectly would receive a 5% penalty.

4) Remove the answered question from the pool.

5) Repeat steps 1-4 until all questions are answered.

It's hard to explain; I might write it up in mathematical form later. But that's a rough idea of what I was thinking. So, how would it work?

Mike in Maryland said...

Grinsnchuckles,

Are you trying to say that someone who might be conducting questionable polls, and who is saying that everyone who is accusing them of conducting questionable polls is full of male bovine droppings, is not capable of lying about an employee lying?

Johnson might think everyone else is naive, but just because he thinks so does not mean that everyone else IS naive.

Mike in Maryland

Michael (mbw) said...

@Danny Tarlow- I don't have any major quibble with your methods. It's just the plain parameter that seems off. I think your 0.3 parameter corresponds to inter-item correlation about r=0.06. (But check my math- that's just off the top of my head.) That sounds low for correlation between a math question and a vocabulary question, It sounds unbelievably low for two questions from a narrow family.

Basically the inter-student variance here will be about the value for r=0 times (1+9r). As your simulations with slightly larger r show, the fit to the SV report quickly becomes terrible.

So the question is whether r < 0.1 is plausible for these sorts of items. I don't think so. Do you know any explicit data from relevant tests?


It's still disappointing that most of the other comments don't seem to even follow what the statistical question is.

Michael (mbw) said...

Another free-association:

The inter-item correlations on various tests provides the data that are used to justify the so-called general-intelligence "g" on IQ tests. The construction of "g" from the data and its interpretation are highly debatable, but the inter-item correlations are just a fact. Of course those include items far les closely related than the ones on this alleged poll.

Has SV proved the non-existence of "g"? Some of their conservative sponsors would be highly disappointed with that result. I'm afraid that what SV has done is somewhat less significant.

Michael (mbw) said...

I'm trying to come up with a consistent conservative account of the SV results. These show essentially no variance in g, or even in narrower categories. Now it is important to standard conservative ideology that variance in g be almost all hereditary. So SV has shown that a random selection of 1000 Oklahoma students are genetically identical, i.e. are a homogeneous in-bred strain. Oh wait...

Grinsnchuckles said...

Mike of Maryland,

lol - I'm just saying that I'm a social worker, not a pollster. And in my field, when a person defends himself with statements that don't ring true, my alarm bell goes off. If Mr Johnson had nothing to hide, I would expect him to say that the offices are just UPS boxes, or to say they only have one physical office, or at least not to make it sound like they have many. His response is just puzzling, and not very brght since it just adds to the suspicion that something is amiss.

Inferno said...

So, anyway. Since we have a math geek here and I've been waiting for an answer...

Michael, how do you think the simulation I proposed would work? I was trying to get an inter-question dependence, which is what your objection to this entire discussion is. So far, you've made three posts consecutively about that point, yet you haven't addressed my attempt to address it.

Ryan said...
This post has been removed by the author.
Gary said...

They directly copied the Goldwater test.

All of their results are also very close to the Goldwater test.

The only answer that was more than 4% off was on Jefferson where Oklahoma is about half as smart as Arizona.

Could they have taken the Goldwater test and the known results from the Goldwater test, did a little randomization so they weren't identical, and then passed it off as a new test in Oklahoma? I don't know what is the deal with Jefferson then.

This seems like polls and results with an agenda produced for any group of right wing boobs.

It is much more profitable just to make up the results then go to all the work conducting valid scientific surveys.

Mo said...

Nate, you are dead on right.

I did a similar study with my stat students and the "knowledge" of students did follow a normal distribution - questions and scores are not independent. It's obvious to me that SV faked their data, and poorly at that.

The companies that paid them have a case for fraud.

Michael (mbw) said...

@inferno- Sure, your algorithm would generate some inter-item correlation. It's kind of specific in how it does it, so it might be taken as a bit artificial. Why not try it anyway?

I'm focusing on the second moment of the score distribution because it's very simple and easy to express in terms of the typical inter-item correlation coefficient.

Danny's simulations also get at this in a pretty accessible way. They're a bit artificial (trimodal distribution) but that's not especially important.

My wife has suggested some anecdotes to help bring some others into the conversation. (Inferno, and Danny can skip this.) The point is that the SV results look like a student who knew how many judges were on the SC was not much more likely to know how long a senator served than a student who didn't, and so on for the other pairs of questions. Does that seem plausible? Think of your own high school!

Mongo said...

@mo "The companies that paid them have a case for fraud."

Although as far as I can see, the results that SV LLC provided are exactly what their clients want. Being mostly right-wing pressure groups, they are looking for ammunition against non-right-wing institutions, in the form of 'statistics' that show that their opponents' policies are not working.

And that is what SV LLC has provided. The minor detail that the data are fabricated appears to be of little consequence or interest to their clients.

Dwight said...

And that is what SV LLC has provided. The minor detail that the data are fabricated appears to be of little consequence or interest to their clients.

It is of interest if the fabrication becomes wildly known publicly. :) Although that said the public mess of suing SV, and consequently having to prove that they weren't co-consirators to ginned up data, could be a strong reason that they wouldn't pursue that.

They just never work with SV, LLC again. Mrs and Mr Johnson rename their business to something else, potetially with someone else's name on the incorp papers, and shell game picks up and moves on.

Quixote said...

@Inferno

"the major thrust of his argument seems to be that the answers correlate strikingly to a distribution where there was no correlation whatsoever between right and wrong answers, and the student body is homogeneous"

Yes, understood. On a second look, I think I overstated my point earlier.

Now, before Nate gets to the math, he spends a paragraph saying how hard it is to believe that OK students are so stupid. And he continues to state the statistical argument in terms of "knowledge" and "aptitude" and "nerd" and "achievement" and reference to the SAT. It gives the impression that, were the distribution not suspicious, he would happily accept this sort of poll as indicative of student knowledge.

As I've said, I brought this up because it is irritating to see someone like Nate falling into the same gullible mindset that I see in the media every time someone publishes this kind of nonsense poll.

Call it a pet peeve. I know it doesn't go to the crux of the case against SV.

Over and out.

T.J. said...

Having read some of these comments, there clearly wasn't enough diversity in wrong answers. Anyone who has graded any sort of fill-in-the blank exam knows that you will get a range of educated guesses, attempts to be cute and sarcastic, and the occasional "completely misunderstood the question" kind of responses.

Things are not looking good for SV.

nethead said...

The domain record for STRATEGICVISION.BIZ returns 907 Cove Place, ATL which is a few blocks away from the listed ATL address (Publix Market) and looks to be in some gated condo community.

Bill Butler said...

Actually I find the entire test dead-on-balls-accurate.

My wife teaches 4th graders in the Philadelphia, PA area. Over 70% of her class comes in thinking that Philadelphia is the "state" they live in.

Sean said...

@jbz7890,

The good old implied, "We'll 'we're' pretty bad but at least we're not as bad as those coloreds".

Such neanderthal logic in the middle of such a great blog sigh.

AxmxZ said...

@Veek: That was my post over on DailyKos. Frankly, I'm surprised that Nate hasn't mentioned the Surpreme Court Justice question.

I mean, come on.

5 Justices - 5% (50 people said this)
6 Justices - 11% (110 people said this)
7 Justices - 0% (no one said this)
8 Justices - 15% (150 people said this)
9 Justices - 10% (100 people said this)
10 Justices - 21% (210 people said this)
11 Justices - 0% (no one said this)
12 Justices - 7% (70 people said this)

How on earth would this have happened in an open-ended test?

T. J. Hairball said...

So when I read this, I wonder to myself how many of the polls of "the public" or of "public school students" are similarly complete garbage that the media pick up and run with for a couple days.

goodeda1122 said...

自慰套,自慰器,情趣,自慰套,情趣,視訊交友,充氣娃娃,
性感丁字褲,AV,按摩棒,電動按摩棒,情趣按摩棒,

按摩棒,變頻跳蛋,跳蛋,無線跳蛋,G點,
男女,潤滑液,
SM,情趣內衣,內衣,性感內衣,

角色扮演,角色扮演服,吊帶襪,丁字褲,情趣用品,情趣用品,飛機杯,
自慰套

btilly said...

At a 99% confidence level we can reject the theory that a simulation of the form described was not used to generate the numbers. See this post for details.

That said, I have trouble believing that the presented numbers are true. But they weren't made up with a completely naive simulation.

markymark said...

The thing that would send my spidey senses tingling is that 60% of the students knew that the Atlantic was on the East Coast, and yet only 23% of the poll knew that Washington was the first President.

I suppose if you are guessing the oceans question a basic level of US geography might mean you know its either atlantic of pacific, so you are down to a 50-50 if you are guessing, but even still it seems a slightly odd number.

But the problem that Nate points to, that surely some people would know all the answers, seems sound. Surely someone who knows there are 9 SCOTUS judges would be able to answer most, if not all, of the other questions. It does seem very odd to me as a school teacher, that out of a sample of 1000 kids, not 1 could answer all the question (as badly worded as some of them where!)

Peter Wolf said...

@btilly

@99% confidence interval it isn't a simple simulation...ok, fine. But that doesn't preclude a simple simulation with some small variance being used, ie some hand numbers used for working out the individual percentages for each question correct. Then THOSE numbers could be being plugged into a simple simulation with no correlation but + a bit of variance.

sorry but your analysis doesn't preclude that, and I'm guessing there are other possibilities.

Peter Wolf said...

@markymark

"It does seem very odd to me as a school teacher, that out of a sample of 1000 kids, not 1 could answer all the question (as badly worded as some of them where!)"

not an english teacher I'm guessing :) heh, only kidding...

if you're a school teacher, why don't you set a pop quiz of those 10 questions and see what the results are?

markymark said...

Peter Wolf, sorry not good at proof reading posts to websites lol!

Anyhoo, the one slight note of concern with bashing our good friends at SV, LLC that I would suggest is that how seriously were the tests taken (if they were indeed taken!) I mean its at least possible kids taking the test they were not taking the test very seriously, and putting joke answers down or something similar.

Wilson said...

I'm new to this site, and I'm not too familiar with different polling techniques, but does anyone else find it suspicious that the sample size was exactly 1000?

I can imagine them doing the poll one of two ways (considering they're based in Georgia, surveying students in Oklahoma): either distributing this test to various high school students, or spending a lot of time on the phone calling up each household with a high school student.

It should be very unlikely to get exactly 1000 students using the first method, since classes are likely to have odd sizes. With the second method, I imagine poll response rates are going to be very low, so a polling firm would want to split up the phone calls between multiple callers, and just have them go through a list of names. It should be difficult to stop all the calls at exactly 1000 responses.

Any thoughts?

Juris said...

@Wilson: this has been commented on on previous articles here in the last few days. Some firms seem to report such even numbers regularly, such as Rasmussen. They use a "robocall" method that essentially shuts down the automatic interviews when they've reached their desired total. I don't know what they do if they have more than one such call active when the target (say 500) is reached.

It would be conceivable for this to happen even if they had live (phone) callers. But that would be unusual.

It seems possible they may be editing out some of the completed or partially completed interviews -- but using what method?

I've run surveys using a CATI before, and out of dozens of surveys we may hit a round target a handful of times. If we exceed our target, we keep all completed interviews; never edit any out to move down to some desired number.

John said...

My first inclination when I saw those numbers was that they were clearly grossly incompetent. Not one kid in a thousand got eight or more? And if the kids were just bombing the test on purpose, you would see a whole lotta 0's. However Nate's analysis seems to show that the most likely analysis isn't incompetence but fraud, they made these numbers up for an agenda.

Ickey said...

As someone who's been a professional in the realm of testing schoolkids, I can say with certainty that any test that is not part of a kid's grade is going to get horrible results at the Jr High level but not at the grade school level. Why? Because as kids get older they learn how to become lazy and not care about tests that aren't a part of their all-important grade. It matters what states the kids are in, too. Here in IA, they try on every test, even trivia quizes. In TX, not so much. These test results have absolutely nothing to do with whether kids actually know who Washington was. It has everything to do with teachers who'd rather take naps than teach kids in the poorest, rural districts.
-Jeff
P.S. I've graded essay tests by kids with every fake name you can think of including one "Barry McCaw-Kinner"

Kathleen said...

I think the results are fake, but I would like to point out that telephone polls are a terrible way to get a random sample of teenagers, especially if they don't include cell phones. Wealth is highly correlated with academic achievement, and wealth is also correlated with teenagers having their own cell phones. And you would be excluded the poorest teens who do have phones at all.

MidPointMan said...

Nate -

I think your witch hunt against Strategic Vision is perhaps a bit off the mark.

Your main line of evidence is that the trailing numbers skew in a non-random direction.

This is not particularly compelling because of the following:


1) Trailing digits is largely dictated by how you treat undecideds (how hard you push them to commit) and how far in advance of the election you poll. Far in advance will produce more undecideds and thus impact trailing digits.

2) Pollsters vary GREATLY in their effort to classify undecided voters.

- Some word the question loosely to encourage soft commitment
- Some as a "leaner" follow-up (Rasmussen)

It is the strength of effort to classify undecideds and the method in doing so that will cause a non-random skew in the data.

Your singling out of Quinnipiac Polls felt very much like cherry-picking.

You should have run the analysis for all major pollsters and showed us the distribution for each.

Why did you not do that?

Because some have much greater skews that Strategic Vision.

Here is my experiment.

I took all presidential approval polls from George W. Bush as archived by the Roper Center.

http://webapps.ropercenter.uconn.edu/CFIDE/roper/presidential/webroot/presidential_rating_detail.cfm?allRate=True&presidentName=Bush

This produced 2,894 trailing digits.

What is good about this is that it is that pollsters are measuring:

- The same thing
- In the same geography
- Under similar conditions
- Over the same time period

The result?

The SKEW differed wildly:

FIRM N %0-4 %5-9 Spread
Fox/OpDyn 282 49 51 1
Gallup 226 46 54 7
Gallup/CNN/USA 218 42 58 17
Pew 200 50 51 1
Newsweek 186 46 54 8
ABC/WP 172 52 48 5
CBS 162 49 51 2
Democracy Corp 154 52 48 4
ARG 138 38 62 25
NBC/WSJ 138 41 59 17
CBS/NYT 132 58 42 15
All 2,894 48 52 3

Even among these firms measuring...

- The same thing
- In the same geography
- Under similar conditions
- Over the same time period

We saw a spread on %0-4 vs. %5-9 as high as

ARG: 25 points
NBC/WSJ: 17 points
Gallup/CNN: 17 points
CBS/NYT: 15 points

Strategic Vision's spread was 10 points, and this was...

- Different candidates
- Different geographies
- Different time periods

YOU WOULD EXPECT MORE VARIATION.

Your methodology and threshold indict about 1/3 of pollsters as frauds.

J2 said...

Did the polll really have "Communist and Republican" as a choice? If so, it may have appeared like a Republican Push Poll and elicited a largely Republican response? Which speaks poorly of the knowledge of Republican-leaning students, if this were the case.

Were the "2 major parties" options write-in or preformed?

Graham said...

The numbers seem plausible to me. Nate, you and your readers are government geeks so it seems weirder to you and your readers. Would have missed most of those questions without multiple choice.
Let me try some basic science natural science question on you all:

"Why is it hotter in summer than in the winter?"

"Why are there high and low tides? (Hint: Remember there are two high and low tides in 24 hours)"

"Why is the sky blue?"

Try answering those and then look up the answers...

Also if some one called up and asked me "What is the supreme law of the land?" with no context I might say "Gravity" or "The Golden Rule".

Harold said...

I sent the questions to my daughter (a high school student in Oklahoma) without any of the list of answers. She got all 10 correct. She also wrote back, "We did this test in government last year."

John said...

The results of this poll are not improbable or unlikely - they are impossible. There is simply *no* possibility that a random sample of 1000 HS students (even located in Oklahoma) could not have *any* students who get 7 or more answer right. First, these are easy questions - I got 9 out of 10 right (I'll admit I answered "Supreme Court" on the first "law of the land" question). You're telling me there isn't *one* randomly-selected HS that is smarter than me? Or even anywhere near the same ballpark? Hogwash.

These answers were just made up out of thin air - or the "real" answers were simply "fixed" afterwards to match the desired results. No question.

Graham said...

Okay you guys changed my mind. The results are not plausible. The fact that no one out of 1000 guessed 7 supreme court justices tipped the scales for me, especially cause that how many I thought there were.
;)

TB said...

I just took the test and got 10 out of 10. And I'm an Oklahoma public high school grad!

Pat said...

Mond is right about the "Communist & Republican" finding. Graduating Seniors in 2008 would have been born some time in 1991. After the fall of the Berlin wall, after (or very near) the collapse of the USSR. Knowledge about communism would probably be more difficult to acquire for them than knowledge about US senators (also an 11% figure). Old people made up these numbers.

Rob said...

Just ran a small scale test in my office. None of the people are in high school, with ages ranging from 30-60. I realize the above poll was for high schoolers, but my logic is that someone who learned this information a year or two ago should be able to answer the questions more easily (or at least more accurately) than people who haven't actively learned it for multiple decades.

All of the people came from different backgrounds with regards to their schooling and upbringing. I asked them the 10 questions and they were required to answer them using a free response (either orally or written down). The only thing that might skew their answers is that I was essentially "grading" them so they all actually tried to answer the questions accurately (therefore no joke answers like the high school kids).

The results:
One person scored a 10, one person 9, two people 8, and one person 7.

Interestingly the person who scored a 7 has regularly claimed to be "dumb" and have stated that they did poorly in high school and didn't learn much. I guess they should have gone to school in Oklahoma, they would have been valedictorian!

(You could argue that this just shows the quality of education that people used to get, and how poorly today's public schools teach kids. I think it shows just how terrible the OK poll is and how if 5 more or less randomly selected adults could score 7 or better then surely at least 1 high school kid could have.

Raz said...

Hey Nate,

I really appreciate you picking this one up. I'm an Oklahoma educated student, and while I agree that we're drastically underfunded, we're not all idiots. I saw this when it was posted on fark, and it set my spidey sense tingling, as well!

I assumed it was because they were doing phone (presumably land line) interviews, and that most teenagers are probably pretty anxious to get a random pollster off the phone ASAP, and will answer "don't know" to get them to go away immediately. But I was particularly suspicious because they didn't describe how the 1000 teenagers polled were selected!

Thank you for going to bat for Oklahoma!

Rachel Shadoan

Smith said...

I'm a sophomore in High School in Maine and I know scores of people in my small (600) school who would easily answer all ten of those correctly.
just saying.

BarackStar said...

First off, these students are all under 18, so none of them really vote or have voted at least in past elections...

Yeah, this is pretty sad still...

Oklahoma is the most conservative state in the country, so who knows what they are teaching these kids...I don't think High Schools have civics classes any more, which they should...

Farley Balanced said...

I question their assertion that there are two major parties names the "Democrat" and Republican Party. Only the wingnuts on the right call the Democratic Party the "Democrat" Party. Sounds to me as if they are not exactly unbiased.
Let's hear it for the Republic Party!

Freedom's Truth said...

"I gotta disagree with you on this one, remember that Texas and OK texrbooks are designed to mislead, "

This is insulting and totally 100% wrong. The SBOE in Texas has enabled a strong social studies curriculum w.r.t. our nation's founding and strong civics and history knowledge.
Texas also requires in their college core curriculum a study of the US Constitution and history.

The poll is not trustworthy. My five year-old first grader knows the first President of the United States.