The Flinders University of South Australia presents

the 1999 Loebner Prize Competition

January 22th

Can Computers Think? (Click for a map of the debate)

photograph of Loebner side of medal The Loebner Prize is a search for the thinking computer based on the famous Turing Test... Can a computer think?

Following negotiations with the creator of the Loebner Prize for Artificial Intelligence, Dr Hugh Gene Loebner, the Flinders University of South Australia agreed to assume the role of Official Host of the Prize in 1998 and 1999. The competitions were co-ordinated by Associate Professor David Powers from the University's Department of Computer Science. The 1998 contest was held on Sunday 11 January 1998 at the Powerhouse Museum in Sydney and the 1999 contest was hosted by Flinders University on 22nd January 1999. For the first time entrants from around the world were able to watch the competition live whereever they were. Not only could they view it on location at Flinders (see program/venue information) they could watch by IRC or via the WWW (see below). The Australian entrants and the Australian media attended in person while other entrants and journalistsfrom around the world watched the competition live via IRC, on a series of "chat room" channels. In addition, in a separate "chat room", participants were able to discuss the conversations as they happened, and discussion continued well after the official close of the competition, which was in the early hours of the morning for most participants. Furthermore, one of the human confederates was also participating live from the US, using a telnet connection over the internet. Amazingly this connection survived the seven hour marathon session without a glitch.

The University welcomes to host the Prize, and expects to have a continuing association with the competition. In 2000 the Loebner Prize will be organize by Dr James Moor at Dartmouth College in New Hampshire, USA, whilst in 2001 the competition move to the London Science Museum. The US confederate was in fact Dr Moor, connecting from Dartmouth to learn first hand about the competition he is taking over next year. In a suprise session at the end of the competition, Dr Loebner himself was set chatting to Dr Moor, and gradually discovered whose conversation he'd been monitoring and who it was that was talking to him.

This initiative is the first step in a new direction for the Loebner Prize competitions, and is designed to provide increased interaction with the Artificial Intelligence and Computational Linguistics communities.

Rules and Instructions for Entrants

Instructions for Confederates

Instructions for Judges

Result Tables for 1998

Result Tables for 1999

Experience the Loebner Prize Live

IRC Chat Room - live!

Connect to ai.ist.flinders.edu.au port 6667 using any standard IRC client, e.g. mIRC, ircII, etc. Join (Remember to include the # in each case.)

1999 Transcripts - terminal by terminal and round by round!

Click on one of the following to get the transcript uptodate as far as the last round for one program or person (computer or confederate):

Open a window on each and review the conversations which will be updated after each round (every 15 minutes apart the breaks - see program ). Netscape should hold your place when you refresh the windows each round - dunno about other browsers...

Note for technical reasons one or two terminals may not be able to be shown on IRC. Some sessions/judges may be rescheduled - due to software or operator error. Such sessions will be held at the end of the current bracket, in a scheduled break.

The Turing Test

photograph of Turing side of medal In a classic paper from 1950, Alan Turing posed the question "Can a Computer Think?" and decided that the answer came down to what we think "think" means? If you decide that thinking is something that only people can do, you have answered the question before you even build the computer! So Turing came up with a behavioural test - if you can't tell the difference between a computer and a person, that is if the each win 50% of the time, then you have no basis for saying that one can think and the other can't. Turing predicted that by the end of the century, a computer would be identified as human 30% of the time if limited to 5 minute conversations.

The Loebner Prize competition in 1998 saw the best program selected by judges 15% of the time, being award the (US) $2000 Annual Prize and Bronze Medal for being the highest ranked computer program. Once entrants reach the 30% level, the competition will change in nature and there will need to be a playoff for the $25000 Turing Prize and Silver Medal. Entrants in this class will also be able to compete for the $100000 Grand Prize and Gold Medal, which requires demonstrating audio-visual as well as conversational capabilities.

History

Reminiscent of the first Loebner Competition, held at the Boston Computer Museum in 1991, the 1998 competition took place at Australia's premier museum, the Powerhouse Museum in Sydney, which has a long tradition of promotion and presentation of advanced technologies.

Over the last nine years, the Loebner Prize has generated considerable interest - and controversy - in the media, the professional Artificial Intelligence research community and industry, and the broader community of internet newsgroups, hobbyists and industry observers. The entrants have been programs written by people from across the full spectrum - entries have been marketed as commercial programs, written as a personal challenge, or offered as outgrowth of a research program. However, we are not really seeing research systems, and the Loebner Prize has not been seen by the research community as providing any kind of useful boost to the community, but rather has been critized as a distraction, a publicity stunt which has little to do with the state of the art in AI or Human-Computer Conversation.

Moreover, the authors have usually not had the opportunity to attend the competitions, and demonstration and explanation of the systems outside the competition has not been possible. By bringing the competition together with international Natural Language and Computational Linguistics events, we provide the opportunity for direct evaluation and cross-fertilization amongst the authors and other interested researchers, and we have the opportunity to establish a richer connection between the competition and the researchers whose work will eventually produce a Turing-level Artificial Intelligence.

In addition, previous Loebner Prize competitions have been judged by a panel who were supposed not to have any particular expertise which would assist them in the task of distinguishing Human and Computer. Now we are deliberately seeking to make the panel of judges representative of different facets of life and different kinds of relevant expertise from across the spectrum of Cognitive Science, Artificial Intelligence and Computational Lingusitics.

Associating the Loebner Prize with a Human-Computer Conversation workshop also provided the opportunity to see something of where the real issues in AI and computer conversation lie, as well as for researchers to interact with each other and their systems.

For further information on the debate around thinking computers,there is an interactive map

For further information on the competition please refer to the Rules.


This webpage is http://www.cs.flinders.edu.au/research/AI/LoebnerPrize/.

 

Copyright © 1998 Department of Computer Science, Flinders University, dmwp
Email: powers@cs.flinders.edu.au