Commentary on TOEIC Booster column of April 30, "A new-look TOEIC possible," from The Daily Yomiuri

TOEIC test takers who follow the TOEIC Booster column in The Daily Yomiuri must have been fairly stunned to read that Stimulus-Response-type questions do not yet exist on Part II of the TOEIC test. Anyone who has taken the test in the past several years has certainly come across this type of test item, and TOEIC preparation materials often include at least one example of this type question in their sample tests and exercises. It's a mystery why Lynn Yilmaz would make a claim that anyone familiar with TOEIC knows is clearly false.

Before I go any further, let me say this about Lynn Yilmaz. She seems like a very nice person. She took the time to respond to my letter to the Yomiuri, has corresponded with me a number of times by e-mail, and has seen to it that her publisher has sent me a complimentary copy of her TOEIC Bridge preparation book, Cross the TOEIC Bridge, published by McGraw-Hill. Very nice lady, Lynn Yilmaz. Unfortunately (for a TOEIC expert), she doesn't seem to know very much about TOEIC.

So why this web page? I'm realistic about the chances that the Yomiuri will publish my letter setting the record straight - about 0%. Lynn Yilmaz has a potential audience of thousands through her column (though far fewer actually read it, I suppose). If anyone happens to stumble across this web page, at least they can know that Ms Yilmaz is not getting a free pass to dispense misleading information about TOEIC - someone is paying attention and calling her to task, for whatever it's worth. As for Ms Yilmaz's feelings about this page, I am actually concerned, because she is a nice person, like I said. But she also seems to be quite resilient, and I'm hoping that she may just be happy to know that someone is paying attention.

Examples from published TOEIC preparation materials of Stimulus-Response-type items:

I'd appreciate it if you could fax this document before you leave.
a. No problem. I'll take care of it right away.
b. I don't think it can be fixed so easily.
c. In fact, I'm afraid it's expected to depreciate.

From 7 Hours to a Higher TOEIC Test Score, by Matsutani, Hilke & Wadden, published by Goken, 2001, p. 68

Let's get something to eat.
a. Good idea. I'm starved.
b. I got him something to read.
c. It's made of wheat.

From Oxford preparation course for the TOEIC test, Teacher's Book, 2002, p. 93 (tapescript)

Let me know if you can't make the deadline.
a. Get off at the last stop.
b. I didn't know the phone wasn't working.
c. Everything is going right on schedule.

From Thomson Preparation Course for the TOEIC Test 1, by Roberta Steinberg, published by Thomson, 1999, p. 44

Please fill out this form in pencil.
a. I'll have to borrow one of yours.
b. No, I didn't.
c. I don't draw very well.

From Thomson Preparation Course for the TOEIC Test 2, by Roberta Steinberg, published by Thomson, 1999, p. 99

We'll arrive in a few minutes.
a. Oh, that's good to hear.
b. Are they arriving so soon?
c. No, she didn't have to.

From Listening Pointer for the TOEIC Test, by Yamaguchi and Guire, published by Seibido, 2003, p. 58 (teacher's manual tapescript)

Now I should mention that just because many TOEIC materials writers include such items in their publications doesn't mean that they can be trusted to mirror TOEIC item writing perfectly. My experience is that many materials writers don't really understand TOEIC item writing conventions. However, at least I can say that the above authors, and no doubt others as well, were on the ball enough to include Stimulus-Response-type items, and they did so because these items have appeared on TOEIC for years. Meanwhile, Lynn Yilmaz has been asleep at the wheel.

A few more ramblings about the "new-look TOEIC" column of April 30...

Lynn Yilmaz makes the assertion in her column that "responding to statements is an important language skill - one that is now recognized by the new form of TOEIC Bridge Stimulus-Response question."

I agree that responding to statements is an important language skill, but I have my doubts that TOEIC Bridge or TOEIC actually tests this skill. What TOEIC does is present a highly artificial, abbreviated exchange stripped of all context and pragmatic information. In the real world, one is never confronted with a statement out of context and asked to make an appropriate response to it. In my experience writing test items for our university entrance exam, I have noticed that almost any sentence can follow a statement, so it is extremely difficult to write items of this type and be certain that the distractors would not be correct in some hypothetical exchange.

Let's look at Example 2 from the TOEIC Booster article:

The office is closed until one o'clock.
A. The office manager wants to.
B. We'll just have to come back.
C. Then I'll close it.

Lynn Yilmaz says the correct response is B, and that's certainly a good choice, based on the limited information available. But try giving the dialog a little more context. Imagine what might have transpired before this exchange, such as:

Speaker #1: We'll have a little extra time to prepare on Monday morning, if anyone wants to. The office is closed until one o'clock.

Speaker #2: The office manager wants to.

It may be unlikely that a test taker could imagine such a dialog, but that's a moot point. If the dialog is even remotely possible, then how can anyone declare answer A to be an incorrect response to the comment? To determine that one response is correct while another is incorrect may not reflect a test-taker's language ability so much as his or her ability to think like the test writer. You may even have the paradoxical situation where the higher the level of the test taker, the more attractive some of the distractors will appear, as it becomes easier to imagine a hypothetical context for the exchange.

Lynn Yilmaz suggests in her column that ETS will move towards greater use of Stimulus-Response-type items, and she implies that this is a good thing. Personally, I say any move toward greater use of this type of item is going to create more headaches for ETS in terms of test validity.

Commentary on TOEIC Booster column of Oct. 24, "Vocabulary Muscle Counts," from The Daily Yomiuri

As a TOEIC teacher, I am interested in what tips and examples TOEIC materials writers offer. I have read the TOEIC Booster column in The Daily Yomiuri from time to time, and have generally thought the advice to be sound until I read the Oct. 24 article. I then realized that TOEIC Booster column writer, Lynn Stafford-Yilmaz, has probably never seen an actual TOEIC test, or if she has, it was from many years ago. Of course, most native English speakers haven't seen the test, so that's not unusual, but Stafford-Yilmaz holds herself out to be an expert on TOEIC and purports to offer examples of TOEIC that are "typical." Make no mistake about it, they are NOT typical. Now as far as crimes against humanity go, this one ranks fairly low on the seriousness list, but why should Stafford-Yilmaz get a pass? Anyway, I've taken on the responsibility of calling her to task. What follows is the case against her questionable examples of Oct. 24.

Click here to read:
Kevin's Oct. 28 letter to The Daily Yomiuri
Reply from TOEIC Booster writer, Lynn Stafford-Yilmaz

Stafford-Yilmaz is correct in stating that the Listening Part I Photos section does not test knowledge of grammar. The majority of the sentences are present-progressive "doing" type sentences, as in "The man is reading a newspaper." Sentences of this type are fairly easy to grasp, and indeed most sentences on this section would not greatly challenge a test taker grammatically. Contrary to Stafford-Yilmaz's view, however, the sentences do not greatly challenge the test taker vocabulary-wise either. Most of the sentences are simple and straightforward, and use everyday, non-technical vocabulary. While Stafford-Yilmaz may believe words as far down the word frequency list as "aviary" and "calibrating" are typical of this section, she is mistaken. These words are highly unlikely to appear in the Photos section, though they cannot be ruled out on the Short Talks or Reading Comprehension sections.

Stafford-Yilmaz submits four example questions for the Photos section in her article of October 24, 2003. From her examples, it is clear that she has little understanding of the TOEIC item writing conventions for this section. She claims that Example 1 is a "typical TOEIC photograph question." In fact, this example is extremely atypical. The TOEIC Photos section would not list four identical sentences, changing just one word in each so that the identification of that single word is the key to answering the question. In TOEIC, whenever the verb changes, the object changes as well. TOEIC test takers must listen to and understand the entire sentence, not just the verb. In Stafford-Yilmaz's example, she is essentially asserting that the item in the picture can be nothing other than a salad, removing that portion of the sentence from consideration as true or false.

In place of Stafford-Yilmaz's Example 1 sentences, I would submit the following as more typical of TOEIC for the photo in question:

1. The woman is cutting her food.
2. The woman is sipping her drink.
3. The woman is reaching for a glass.
4. The waiter is taking the woman's order.

Where Stafford-Yilmaz goes wrong is in not following the item writing conventions of TOEIC. The conventions for writing distractors are:

1) They must be clearly and unambiguously false in relation to the photo.

2) They should be true for a hypothetical photo. In other words, if the sentences cannot be pictured, they should not appear as distractors.

3) The should pass what I call the "blindfold test." If a person of native-speaker ability were blindfolded (i.e., prevented from looking at the photos) and listened to a tape of the sentences, he or she should not be able to eliminate any distractors. If any distractors can be eliminated, the sentences do not pass. Sentences that do not pass the blindfold test include:

a) absurd sentences
b) ungrammatical sentences
c) sentences about peoples thoughts, feelings, or intentions (there is some flexibility here, but not much)
d) unpicturable sentences (see convention #2 above)

Item by item commentary on the Stafford-Yilmaz examples

Example 1 [Photo shows a woman cutting something in a bowl.]

(A) The woman is cutting her salad.

• The problem is not so much this sentence, but the fact that "her salad" is used in all the other choices, removing this part of the sentence from consideration as true or false. Within Stafford-Yilmaz's logic for this question, the sentence appears to be OK. We have been told that it's a salad, and yes, she appears to be cutting it. However, if this sentence appeared on an actual TOEIC test, and all other sentences had different direct objects, then we would expect the photo to be clear enough to show that the item on the plate is salad, and not something else. In this particular case, and perhaps because it's a newspaper quality photo, the photo is not clear enough to show that she is cutting salad or some other kind of food. If test takers could reasonably imagine this food to be anything other than salad, then they would logically assume the sentence to be false. TOEIC would likely avoid ambiguity here by replacing the word "salad" with "food." The sentence would then read, "The woman is cutting her food," a sentence which conceivably could appear on TOEIC.

(B) The woman is chewing her salad.

• Maybe some folks masticate their food with their mouths wide open so everyone can verify that they're chewing, but for many people, it would be difficult to know if they are chewing or not from a photo. Can anyone honestly look at that photo and state with 100% certainty that the woman is NOT chewing? Granted, "the woman is cutting her salad" is a more obvious answer, but TOEIC would not offer a distractor like B which is virtually unknowable.

(C) The woman is dressing her salad.

• Normal, spoken American English* doesn't use the word "dress" as a verb to mean "put salad dressing on" something. The transitive verb "dress" is most frequently used to mean "put clothes on someone," as in the sentence, "The woman is dressing her child." Granted, "dress" is listed in some dictionaries as a secondary meaning to "put dressing on" something. For example, we "dress a wound" and a cookbook might instruct one to "dress a turkey" or something similar. However, the photos section of TOEIC uses sentences that are in general usage and would not use "dress" in this slightly obscure sense. Stafford-Yilmaz has locked herself into a pattern where she wants to keep the original sentence intact, replacing only the verb, which forces her to make this rather unusual combination of verb and direct object.

* For better or worse, TOEIC follows typical American usage.

(D) The woman is ordering her salad.

• The possessive pronoun "her" seems strange here, and should be replaced with the more natural article "a", but since Stafford-Yilmaz has locked herself into a decision to change only the verb, she is left with this somewhat unlikely wording. Beyond that, this sentence fails the blindfold test. Without looking at the picture, if one heard and understood this sentence, one would have to wonder how TOEIC could picture her specifically ordering a salad rather than some other item. Since this sentence can't be pictured hypothetically, it would not likely appear as a distractor on TOEIC.

Example 2 [Same photo]

(A) The lady is honored by her meal.

• This example clearly fails the blindfold test as it is both absurd and unpicturable.

(B) The lady is upset about her meal.

• Granted, she doesn't appear to be upset about anything, but the TOEIC photos section does not ask test takers to imagine what people are thinking or feeling. Even if a photo showed someone who appeared to be upset, it would be difficult to indicate clearly that she was upset about her meal rather than something else. This sentence fails the blindfold test.

(C) The lady is intent on her meal.

• While I hesitate to endorse this sentence as a likely TOEIC answer choice, I can't categorically disallow it either. The woman in the photo does seem to be intent on her meal, so perhaps this sentence could appear on a TOEIC test.

(D) The lady is distracted from her meal.

• Perhaps OK.

Example 3 [Photo shows two men in a room with some technical equipment, sitting at desks, facing their computer monitors.]

(A) The workers are in a lavatory.

• Many TOEIC materials writers seem to think that it's common for TOEIC distractors to feature similar sounds in order to confuse test takers. In fact, on the TOEIC photos section, choosing the correct answer rarely depends on one's ability to distinguish between minimal pairs or similar sounding words. Here "lavatory" is given as a distractor for "laboratory." I can't state with certainty that this sentence would not appear as a distractor, but I find it somewhat unlikely, with or without the presence of "laboratory" as the intended answer.

(B) The workers are in a gymnasium.

• OK.

(C) The workers are in an aviary.

• This sentence is an unlikely distractor, as the word "aviary" is too uncommon a word to appear in the photos section.

(D) The workers are in a laboratory.

• OK.

Example 4 [same photo]

(A) The geologists are measuring atmospheric pressure.

• Stafford-Yilmaz is wrong in thinking that the photos section requires test takers to understand difficult vocabulary. I find "atmospheric pressure" to be too difficult for this section, though not for the Short Talks or the Reading Comprehension sections. Aside from that, it strikes me as possible that nowadays, geologists might measure atmospheric pressure by computer. If so, can anyone say with certainty that the people in the photo are not geologists and that they are not measuring atmospheric pressure? Because the sentence is not clearly, unambiguously false, it would not appear on TOEIC.

(B) The technicians are concentrating on a task.

• OK.

(C) The veterinarians are calibrating instruments.

• My thinking on this sentence is similar to that of (A) above. First, "calibrating instruments" is too difficult for this section. And since it's hard to imagine what "calibrating instruments" should look like, how can we say with certainty that the two men in the photo are not doing it? I don't know whether Stafford-Yilmaz thinks "veterinarians" is the key point here or not, but I certainly can't tell from the photo what line of work these men are in.

(D) The electricians are disassembling equipment.

• OK.

Top

Care to comment or rebut? I will post your comment at the bottom of this page, provided you give your name and affiliation. You may contact me by e-mail at: kmiller@shikoku-u.ac.jp

Return to Kevin Miller's webpage

		Return to Eng. Dept. homepage