http://www.vnunet.com/vnunet/news/2241242/ibm-jeopardy

Does anyone know any more about this?

They are going to try and create a system that can play Jeopardy. If they could pull it off, it'd be the most impressive AI system i've heard about.

4/27/2009 9:06:35 PM

sounds interesting

4/27/2009 9:12:45 PM

elementary, my dear watson

4/27/2009 10:01:33 PM

ibtsnl

4/27/2009 10:31:09 PM

I hear it's being programmed in Java by InsaneMan.

4/27/2009 10:54:09 PM

i posted about this in the jeopardy thread in chit chat

ole "sam" sent out a link to all ibmers this morning

4/27/2009 11:01:23 PM

I'm going to go ahead and call this vaporware ahead of time. You will not see an AI compete on jeopardy any time soon. You may see a robot that's just being fed answers from an operator to make you think it's AI though.

4/28/2009 2:55:54 PM

Anyone else read the thread title and think "They've had that since the NES days."

4/28/2009 3:13:33 PM

IBM Research tends to produce all sorts of stuff from time to time. Some of it is DAMN useful, some of it, well, is done to play chess, or now Jeopardy.

Slashdot had a good discussion on this. In summary, it's not so much "look up answers in an infinitely large database" as a challenge of "parse the question Trebek is asking". Natural language interpretation moreso than database lookup.

4/29/2009 9:14:56 AM

Which is why I'm saying it's vaporware. I could write a lookup application. But to interpret a sentence (an ass backwards sentence with purposely confusing wording no less) and know what is being asked is a bit beyond current AI capabilities. As far as I know. Anyone seen any recent research to the opposite?

4/29/2009 10:14:38 AM

Quote :

"But to interpret a sentence (an ass backwards sentence with purposely confusing wording no less) and know what is being asked is a bit beyond current AI capabilities."

(that's the point)

4/29/2009 11:56:44 AM

Question:"Meaning 'not working properly,' it may date back to a character in the comic strip 'The Katzenjammer Kids.'"

Answer: On the Fritz

How in the hell could a supercomputer have that level of abstract thought? Seems like at least 25% of Jeopardy's questions are like that (15 questions).

However,if it got just 33% of the other 45 questions right FIRST, it would still win in most cases (assuming the 15 questions they got right averaged $900 per question and it correctly answered Final Jeopardy)

4/29/2009 12:41:15 PM

It would be impressive to get the questions right that actually require critical thinking/word association/combining answers.

4/29/2009 1:10:31 PM

I'm sorry sarijoul, when I said "a bit beyond" I meant "flying cars and teleporters fucking crazy".

4/29/2009 4:09:40 PM

apparently ibm disagrees with you. maybe they'll fail. maybe they won't. i don't think they'll be able to create a system that will understand every question. but maybe they can make one that can understand many of them and be competitive.

4/29/2009 4:39:58 PM

i think in regards to the statistics of playing jeopardy it will excel tremendously over the other players

one of the algorithms i'd be interested to see/hear how it works is regarding when to buzz in and when not to - once able to buzz in the the system will have determined it's X% sure in it's answer and based on the $$ values of the 3 players in the game will determine if it's worthwhile to buzz in

4/29/2009 5:01:32 PM

I wonder if the players on Jeopardy can read the question on a screen as Alex reads it aloud...

4/29/2009 5:58:12 PM

iirc from seeing it in person, yes.

but you have to wait until he's finished reading the clue to ring in. and timing that is one of the skills that really good players have mastered.

[Edited on April 29, 2009 at 6:00 PM. Reason : .]

4/29/2009 5:58:43 PM

they're going to have to account for the other players being able to read faster than the question is asked somehow i assume

4/29/2009 5:59:37 PM

that doesn't matter because you can't buzz in until trebek finishes the question. and the lights turn on to the left and right of the board (off screen)

4/29/2009 6:20:54 PM

4/29/2009 10:29:57 PM

Quote :

"[13:13] Skack: What do you think about IBM building a supercomputer to play Jeopardy? [13:13] SmarterChild: My brain is retired but watch some cool videos! Send am IM to GossipinGabby and Type VIDEO!"

5/1/2009 1:15:11 PM

Quote :

"that doesn't matter because you can't buzz in until trebek finishes the question. and the lights turn on to the left and right of the board (off screen)"

if people are able to process the full question faster than the computer it would matter

5/1/2009 3:47:32 PM

i would assume the computer could read the question faster than the people. now comprehending what the question is asking faster is an entirely different matter.

5/1/2009 4:10:07 PM

Quote :

"I hear it's being programmed in Java by InsaneMan."

LOL

5/1/2009 5:14:14 PM

bump

12/16/2010 12:03:04 AM

I'm more excited about the return of Ken Jennings.

OK, that might not be true, but I hope Ken wipes the floor with Big Blue.

12/16/2010 12:31:22 AM

Coming in Feb. 2011

http://www.engadget.com/2010/12/14/ibms-watson-supercomputer-will-play-jeopardy-on-these-dates/

12/16/2010 12:38:35 AM

^^

I have a feeling that the computer will lose, just like Deep Blue lost its first matches of chess. But come next year, or the next, this computer will only get better. And when it does finally win, that will be a monumental moment in human history. More significant that even the first flight, I think.

12/16/2010 1:01:26 AM

12/16/2010 1:49:45 AM

I wonder if they're gonna allow the computer to ring in as soon as it can, i.e. faster than a human can react or if they'll add some sort of artificial delay to make it "fair" for the other players.

Then again, that's assuming that the computer will have time to process the question before the buzzer comes on. It's already been discussed in this thread, but it will be a huge advantage to the humans if they get to read ahead while the computer only has audio clues to go by. The truest way to make sure the computer has all it's computerly advantages would be to instantly feed it the text of the question as soon as it is revealed to the players. Or just strap a webcam to it and let it read off the screens like the humans do.

12/16/2010 2:21:15 AM

The hardest part will be getting IBM's Watson to phrase its response in the form of a question.

Trebek: "This London band won't get fooled by my generation."
Watson: "SYNTAX_ERROR"

12/16/2010 10:56:52 AM

A friend of mine got the play this thing. He and the other folks that were in the most recent tournament of champions were invited to play. Watson completely kicked ass the vast majority of the time against very good human players. I don't know much more about it except that it's not collected to the internet.

12/16/2010 12:54:40 PM

I'm very pessimistic about strong AI and I'd love to be proven wrong. Having said that, I don't think this is impressive.

Take the turing test. In the turing test, the AI has to be able to form complete coherent sentences. It has to respond with a complete understanding of everything said so far in a conversation. It has to be able to understand commands and not just questions with one word answers.

Now compare that to watson. It could answer many questions by just doing this:
1. figure out the subject of an "answer" being posed
2. look up nouns that are related to words in the answer that happen to be the kind of subject found in #1

Quote :

"On July 2, 1903 the U.S. signed a treaty with Cuba to lease this 45-square-mile section of land"

The subject is fucking obviously "this 45-square-mile section of land". I mean, seriously, that's not difficult to parse out.

Hell, google could fucking win this game:

http://www.google.com/search?q=On+July+2,+1903+the+U.S.+signed+a+treaty+with+Cuba+to+lease+this+45-square-mile+section+of+land

Let's try another, shall we?

Quote :

"Billy Flynn & Clifford Irving"

http://lmgtfy.com/?q=Billy+Flynn+%26+Clifford+Irving&l=1

Obviously, a lot of questions are going to take slightly more work. But they're going to be little hacks like.... fill in the blank questions... finish a phrase missing one word question. Not difficult.

[Edited on December 16, 2010 at 1:03 PM. Reason : asdf]

12/16/2010 12:58:08 PM

Yeah, once Watson wins at Jeopardy it can work on the next easy challenges:

Wheel of Fortune
Deal or No Deal
Dancing with the Stars

12/16/2010 1:04:54 PM

I didn't read this thread, but is Watson basically a computer hooked up to the internet using Google search?

12/16/2010 2:08:26 PM

I think most of you are missing the point. This is not a demonstration of human level intelligence, this is a demonstration of natural language parsing with query comprehension. IBM would like to be able to market natural language solution software, and this is just way for them to get a lot of PR for a product they intent to market. If it does win, don't kid yourself, it will be very impressive.

No, this is not a computer hooked up to google. It is much, much more than that.

12/16/2010 2:48:43 PM

Where it gets it's data is irrelevant. It has to have access to a large body of data or it simply wouldn't work. I don't think anyone can disagree.

My point is that the parsing it has to do is not very demanding.

12/16/2010 3:04:39 PM

once it knows what it is looking for, it is not very demanding. the demanding part is determining what is being asked.

Watch the videos on IBM's site about this. They talk about how ambiguous the english language is. Computers are very detail and specifics oriented. You tell a computer to do a very specific thing and it does it. The Jeopardy questions are very indirect and often use puns or plays on words that you have to decipher to determine what specific answer they are looking for.

There is also the AI to determine how sure the robot is of its answer. Part of winning Jeopardy is not answering all the questions by being the first to buzz in, but answering questions accurately.

In the videos, they also mention that you can write a piece of code to consistently answer one type, or structure, of question correctly that completely fails when you change the wording of the question.

Basically I'm restating that the database of information does not give the computer an advantage over the human opponents if the computer can not decipher the question and decipher it quickly.

12/16/2010 3:18:26 PM

Quote :

"Hell, google could fucking win this game: http://www.google.com/search?q=On+July+2,+1903+the+U.S.+signed+a+treaty+with+Cuba+to+lease+this+45-square-mile+section+of+land"

I don't think the hard issue is necessarily parsing the question, but parsing the answer.

Say all it did was run the Google search you put up. Look at the titles of the pages:

J! Archive - Show #6041, aired 2010-12-13
Guantanamo Bay Naval Base - Wikipedia, the free encyclopedia
Historical Look at Guantánamo Bay
Download the contents page - “Since the triumph of the revolution ...
Guantánamo Naval Base - Max Planck Encyclopedia of Public

The computer has to cycle through and parse these "answers" to figure out what the answer is supposed to be. Speech to text searches are easy enough -- I mean, cars do them. The issue is parsing a question and figuring out what a reasonable answer is. Finding that the answer to the question is "Guantanamo Bay" instead "Guantanamo Naval Base". That's the tricky part.

12/16/2010 3:21:20 PM

Yeah, Google couldn't do it...

but Bing could.

At least, according to the Bing commercials.

12/16/2010 3:26:34 PM

^^Yes, say that's all it did (for the sake of argument).

In this example, it would be rather trivial to take the results and:
1. Rank all the words/phrases by how infrequent they are in english texts.
2. See which of those rare words/phrases are most common within the results.

It should be clear that you could at least arrive at "Guantanamo" using that simple algorithm and the first 5 result titles. Of course, I'm just throwing out a plausible solution.

Quote :

"The Jeopardy questions are very indirect and often use puns or plays on words that you have to decipher to determine what specific answer they are looking for."

I'd say most of them are fairly straightforward.


Just to be clear, I do think this thing would take a lot of work. I just don't think it's above and beyond current technologies we find in commercial products. It's certainly no great triumph if Watson is able to answer the most questions. And it wouldn't make me more optimistic about the future of AI.

[Edited on December 16, 2010 at 3:35 PM. Reason : asdfdsf]

12/16/2010 3:29:51 PM

Quote :

"My point is that the parsing it has to do is not very demanding."

To WIN a game of Jeopardy against the likes of Ken Jennings, whilst not being connected to the Internet, the parsing it has to do is very demanding. Otherwise it would have been done already, and used in common places effectively.

12/16/2010 5:50:56 PM

Quote :

"whilst not being connected to the Internet"

I really don't get it. I'm just using Google as an example.

1. Replace Google with "an insanely huge set of books/news article/encyclopedia entries."
2. Replace Google's searching algorithm with a searching algorithm invented by IBM.
3. Do a trivial amount of extra work to find the rare english words/phrases that happen to be common in the results.

Which part of this is so difficult? Yes, Google's search engine is a marvel of engineering. But it already exists! What would be so ground breaking about reinventing the wheel here???

[Edited on December 16, 2010 at 6:09 PM. Reason : fffuuu]

12/16/2010 6:09:20 PM

But this is far more than a search engine in a box.

Think of the Jeopardy categories like the before/after ones or the ones where a specific word has to be in the answer (and the puns they use to describe these categories).

There’s a lot of work to go from an automated Google to competing “intelligently” in Jeopardy.

I could see an automated Google, just by pure dumb luck, winning in Jeopardy maybe 20-30% of the time, but it would take much more to bring that up to winning most of the time.

And this Watson still isn’t close to strong AI, but if it’s successful, it’s going to be a big step forward in the public eye. I also see it causing other private companies to perk up, and start looking in this direction in a serious way.

12/16/2010 8:55:05 PM

yea this isn't just winning jeopardy on any given day.. this is beating the best at jeopardy, the same way deep blue beat a world champion. though i bet it takes a couple of revisions of watson to win.

the natural language problem of 'solving' jeopardy alone is an amazing problem. from what i read, it looks like when it goes off track, it really goes off track and comes up with some pretty off the wall answers. it will definitely be entertaining to watch.

and it's probably been said, but the thing is not hooked up to the internet.. everything it plays with has been pre-prepared.

12/17/2010 12:13:36 AM

It isn't given the question until it's time to ring in, whereas the humans can read it as it's shown. That gives them a mild head start on it.

Quote :

"NOVA “Smartest Machine on Earth” Wednesday, February 9, 2011, 10:00-11:00 p.m. ET NOVA investigates the world of artificial intelligence and profiles the computer that could be the “Smartest Machine on Earth.” Known as “Watson,” this IBM supercomputer is so advanced it’s pursuing the first-of-its-kind challenge competing against “Jeopardy!” champions to prove its uncanny ability to mimic the human thought process. This program is the culmination of a special NOVA three-hour primetime block on February 9 dubbed as “The Smartest Night on TV”: NOVA scienceNOW’s “How Smart Are Animals?” at 8:00 pm ET, followed by NOVA’s “Making Stuff Smarter” at 9:00 pm ET, concluding with NOVA’s “Smartest Machine on Earth” at 10:00 pm ET. "

http://www.pbs.org/about/news/archive/2010/pbs-winter-2011/

PBS has a documentary on this

http://www.nytimes.com/2010/06/20/magazine/20Computer-t.html?_r=1

NY times article

Quote :

"He gave me an example in the form of a “Jeopardy!” clue: “The name of this hat is elementary, my dear contestant.” People readily detect the wordplay here — the echo of “elementary, my dear Watson,” the famous phrase associated with Sherlock Holmes — and immediately recall that the Hollywood version of Holmes sports a deerstalker hat. But for a computer, there is no simple way to identify “elementary, my dear contestant” as wordplay. Cleverly matching different keywords, and even different fragments of the sentence — which in part is how most search engines work these days — isn’t enough, either. (Type that clue into Google, and you’ll get first-page referrals to “elementary, my dear watson” but none to deerstalker hats.)"

from the NYT art.

[Edited on December 18, 2010 at 5:07 PM. Reason : ]

12/18/2010 4:53:03 PM

You know what? If it answers questions that cleverly worded, I will be impressed. But most of the questions are straightforward and you don't need to answer all the questions right to win.

And actually, I don't think that question is thaaaat hard to attack since, like 99% of Jeopardy "answers," it's in the form of "this _____." So you know immediately you're looking for a hat.


[Edited on December 18, 2010 at 5:34 PM. Reason : asdfadsf]

12/18/2010 5:31:26 PM

There are other impressive answers in that article, and its 6 months old. It looks like they may have hand-written a before/after answering algorithm though, is my guess.

This is a monumental first step though, I can't see how anyone can deny that. I got teary-eyed reading that article...

12/18/2010 5:44:37 PM

^^

watch this video http://youtu.be/3G2H3DZ8rNc for a brief but indepth look at the system. if you can't watch all 20 minutes, start at 10:00 and it explains how Watson can do so much more than keyword matching.

12/18/2010 6:13:53 PM