Logic Form Identification*
Coordinator: Vasile Rus

Guidelines for Task 14 (former 16) at SENSEVAL-3

Participants: 27 research teams from industry and academia located in North America, Europe, Asia, Australia

PAPERS DUE: APRIL 26 (CHANGED FROM 20), 2004 - see SENSEVAL-3.


TEST DATA (including GOLD STANDARD) is available here .

IMPORTANT: Register to the logic form mailing list to find out the latest info about the Logic Form Identification competition, including the ftp account and password for data. For registration info check out the SENSEVAL site or my home page: http://www.cs.iusb.edu/~vasile  or follow this link.

GOAL:

The goal of this task is to evaluate the performance of different methods addressing the issue of Logic Form Identification (LFI). The Logic Form (LF) that we use is a flat, scope-free first order logic representation that embedds lexical and syntactic information. The advantages of LF over similar representations (Montague-style recursive semantics and Description Grammars) are manyfold:

Briefly, the participants will be given a set of English sentences and they are supposed to return the sentence in Logic Form as in the example below.

Input:       The Earth provides the food we eat every day.
Output:    Earth:n_(x1) provide:v_(e1, x1, x2) food:n_(x2) we(x3) eat:v_(e2, x3, x2; x4) day:n_(x4)

The output will be compared against a gold standard developed by human annotators.

GENERAL GUIDELINES:

The Logic Form of a sentence is the conjunction of individual predicates, where the relationships among them are expressed via shared arguments. Predicates are generated for all content words such as nouns, verb, adjectives and adverbs. Pronouns are treated as independent nouns.  Prepositions and conjunctions are also mapped into predicates that capture the relation between prepositional object and the constituent to which is attached.

There are two types of arguments: e - for events, x - for entities. For the sentence presented above we have two events - e1, e2 corresponding to each verb/action in the sentence and four entities - x1, x2, x3, x4 corresponding to the heads of the base noun phrases (NP). Each verb predicate has the second argument set to the corresponding logical subject and the third argument to its direct object. The remaining slots for a verb predicate are filled with the arguments of their indirect and prepositional objects. In the example presented above (in red color) the predicate eat has arguments after ; (semicolon) which indicates its adjuncts. For the time being we do not make the distinction between the two and thus the accepted representation would be eat:v_(e2, x3, x2, x4) - see below.

Output:    Earth:n_(x1) provide:v_(e1, x1, x2) food:n_(x2) we(x3) eat:v_(e2, x3, x2, x4) day:n_(x4)

Predicates are formed by the concatenation of the base form of the word and its lexical category as encoded in WordNet (since only nouns, verbs, adjectives and adverbs are encoded in WordNet only predicates for those lexical categories have the category attached to the predicate).

To simplify the notation we ignore:

For details about the principles of Logic Forms read Chapter 2 in [2] and [3]. We also reccomend the reading of [1].

METRICS:

There are two metrics we are going to use for the competition: precision and recall.

Argument Level

We define Precision and argument level as the number of correctly identified arguments divided by the number of all identified arguments.
Recall at argument level is the number of correctly identified arguments divided by the number of the arguments that were supposed to be identified.

Predicate Level

Precision at predicate level is the number of correctly and fully identified predicates (with ALL arguments correctly identified) divided by the number of all attempted predicates.
Recall at predicate level is the number of correctly and fully identified predicates (with ALL arguments correctly identified) divided by the number of all predicates that were supposed to be identified.

Let us supposed that some system outputs the following logic form for the above example:

Sample Output:    Earth:n_(x1) provide:v_(e1, x1, x2) food:n_(x2) we(x3) eat:v_(e2, x3, x4) day:n_(x4)
Correct Output:   Earth:n_(x1) provide:v_(e1, x1, x2) food:n_(x2) we(x3) eat:v_(e2, x3, x2, x4) day:n_(x4)

where x4 is incorrectly indentified as the direct object of eating event. In the correct output there are 11 slots to be filled and the predicate eat should have 4 arguments. The previously defined measures for the sample output are given in the next Table:
 
 

Metric\Level
Argument Level
Predicate Level
Precision
9/10
5/6
Recall
9/11
5/6

In addition we will report a more global measure called exact sentence which is defined as the number of sentences whose logic form was fully identified (all predicates and arguments correctly found) divided by the number of sentences attempted. This is similar to gloss level performance measure presented in [2].

TIMELINE OF THE COMPETITION:

TEST WEEK: March 25 - March 31 (the test data will be available to you on the morning of March 25, via ftp - register to the logic form mailing list to find out the ftp account and password. For registration info check out the SENSEVAL site or my home page: http://w.cs.iusb.edu/~vasile )

January 15, 2004: Guidelines and trial data made available to participants.
March 1 - April 15: Participants will be given the test data and the results will be collected after about 1 week.
July 21-26: Results made public at the SENSEVAL workshop in conjunction with ACL-2004, Barcelona, Spain.

SUGGESTIONS, SOFTWARE AND TRIAL DATA:

A package of trial data is provided to interested participants. The trial package contains two data files: (1) English sentences and (2) their corresponding logic form.

A software evaluator is available for download here.

I compiled a dictionary of collocations from WordNet. You may download it from here.

SUBMISSION FORMAT:

Each team is supposed to submit a file containing on each line the answer to a input sentence using the following pattern.

InstitutionShortName Y000 Sentence# Score :: Logic Form
Example:
IUSB Y000 3 89.7 :: nn:_  (x1, x2, x3) logic:n_ (x2) form:n_ (x3)
The field Y000 should be generated as is, for all lines. It will be used in future trials.

REFERENCES:

[1] Jerry Hobbs, Mark Stickel, Douglas Appelt and Paul Martin: "Interpretation as Abduction"
[2] Vasile Rus: "Logic Form for WordNet Glosses"
[3] Vasile Rus and Dan Moldovan: "High Precision Logic Form Transformation"

*ACKNOWLEDGEMENTS: We owe our gratitude to Jerry Hobbs from ISI/USC for reviewing this page and to SENSEVAL-3 organizers for all the great organizational work they do.


For further information please contact: Vasile at vasile@cs.iusb.edu .