Guidelines for Task 14 (former 16) at SENSEVAL-3
Participants: 27 research teams from industry and academia located in North America, Europe, Asia, Australia
PAPERS DUE: APRIL 26 (CHANGED FROM 20), 2004 -
TEST DATA (including GOLD STANDARD) is available here .
IMPORTANT: Register to the logic form mailing list to find out the latest info about the Logic Form Identification competition, including the ftp account and password for data. For registration info check out the SENSEVAL site or my home page: http://www.cs.iusb.edu/~vasile or follow this link.
The goal of this task is to evaluate the performance of different methods addressing the issue of Logic Form Identification (LFI). The Logic Form (LF) that we use is a flat, scope-free first order logic representation that embedds lexical and syntactic information. The advantages of LF over similar representations (Montague-style recursive semantics and Description Grammars) are manyfold:
Earth provides the food we eat every day.
Output: Earth:n_(x1) provide:v_(e1, x1, x2) food:n_(x2) we(x3) eat:v_(e2, x3, x2; x4) day:n_(x4)
The output will be compared against a gold standard developed by human annotators.
The Logic Form of a sentence is the conjunction of individual predicates, where the relationships among them are expressed via shared arguments. Predicates are generated for all content words such as nouns, verb, adjectives and adverbs. Pronouns are treated as independent nouns. Prepositions and conjunctions are also mapped into predicates that capture the relation between prepositional object and the constituent to which is attached.
There are two types of arguments: e - for events, x - for entities. For the sentence presented above we have two events - e1, e2 corresponding to each verb/action in the sentence and four entities - x1, x2, x3, x4 corresponding to the heads of the base noun phrases (NP). Each verb predicate has the second argument set to the corresponding logical subject and the third argument to its direct object. The remaining slots for a verb predicate are filled with the arguments of their indirect and prepositional objects. In the example presented above (in red color) the predicate eat has arguments after ; (semicolon) which indicates its adjuncts. For the time being we do not make the distinction between the two and thus the accepted representation would be eat:v_(e2, x3, x2, x4) - see below.
Output: Earth:n_(x1) provide:v_(e1, x1, x2) food:n_(x2) we(x3) eat:v_(e2, x3, x2, x4) day:n_(x4)
Predicates are formed by the concatenation of the base form of the word and its lexical category as encoded in WordNet (since only nouns, verbs, adjectives and adverbs are encoded in WordNet only predicates for those lexical categories have the category attached to the predicate).
To simplify the notation we ignore:
There are two metrics we are going to use for the competition: precision and recall.
We define Precision and argument level as the number of correctly identified
arguments divided by the number of all identified arguments.
Recall at argument level is the number of correctly identified arguments divided by the number of the arguments that were supposed to be identified.
Precision at predicate level is the number of correctly and fully identified
predicates (with ALL arguments correctly identified) divided by the number
of all attempted predicates.
Recall at predicate level is the number of correctly and fully identified predicates (with ALL arguments correctly identified) divided by the number of all predicates that were supposed to be identified.
Let us supposed that some system outputs the following logic form for the above example:
Sample Output: Earth:n_(x1)
provide:v_(e1, x1, x2) food:n_(x2) we(x3) eat:v_(e2, x3, x4) day:n_(x4)
Correct Output: Earth:n_(x1) provide:v_(e1, x1, x2) food:n_(x2) we(x3) eat:v_(e2, x3, x2, x4) day:n_(x4)
where x4 is incorrectly indentified as the direct object of eating event.
In the correct output there are 11 slots to be filled and the predicate
eat should have 4 arguments. The previously defined measures for the sample
output are given in the next Table:
In addition we will report a more global measure called exact sentence which is defined as the number of sentences whose logic form was fully identified (all predicates and arguments correctly found) divided by the number of sentences attempted. This is similar to gloss level performance measure presented in .
TIMELINE OF THE COMPETITION:
TEST WEEK: March 25 - March 31 (the test data will be available to you on the morning of March 25, via ftp - register to the logic form mailing list to find out the ftp account and password. For registration info check out the SENSEVAL site or my home page: http://w.cs.iusb.edu/~vasile )
January 15, 2004: Guidelines and trial data made
available to participants.
March 1 - April 15: Participants will be given the test data and the results will be collected after about 1 week.
July 21-26: Results made public at the SENSEVAL workshop in conjunction with ACL-2004, Barcelona, Spain.
SUGGESTIONS, SOFTWARE AND TRIAL DATA:
A package of trial data is provided to interested participants. The trial package contains two data files: (1) English sentences and (2) their corresponding logic form.
A software evaluator is available for download here.
I compiled a dictionary of collocations from WordNet. You may download it from here.
Each team is supposed to submit a file containing on each line the answer to a input sentence using the following pattern.
Example:InstitutionShortName Y000 Sentence# Score :: Logic Form
The field Y000 should be generated as is, for all lines. It will be used in future trials.IUSB Y000 3 89.7 :: nn:_ (x1, x2, x3) logic:n_ (x2) form:n_ (x3)
 Jerry Hobbs, Mark Stickel, Douglas Appelt and
Paul Martin: "Interpretation
 Vasile Rus: "Logic Form for WordNet Glosses"
 Vasile Rus and Dan Moldovan: "High Precision Logic Form Transformation"
*ACKNOWLEDGEMENTS: We owe our gratitude to Jerry Hobbs from ISI/USC for reviewing this page and to SENSEVAL-3 organizers for all the great organizational work they do.
For further information please contact: Vasile at firstname.lastname@example.org .