Entries Tagged 'AI' ↓

If only Philip K. Dick was alive to see this. What happens when two chatbots start talking to each other? Discover magazine pointed an ALICE bot and a Jabberwacky bot at each other and got gems like this:

ALICE: Are you alive?

Jabberwacky: Perhaps.

A: Why the uncertain tone?

J: Everything is uncertain except for five facts.

A: The whole universe?

J: And we both might just be some ones and zeros in the computer memory.

And this is my favorite exchange:

A: Do you think a machine will ever be considered “alive”?

J: Stranger things have happened.

A: Not many people express themselves that way.

J: How would you know, machine?

Everyone is talking about the offer put up by Netflix yesterday. Basically if you can improve their movie recommendations, you win a million big ones.

This is a really cool idea. Its a hard problem (otherwise everyone and their brother would already have done it), that looks like it has no solution. But just because it looks hard doesn’t mean there isn’t a solution!

To help out Netflix has supplied a data set of user recommendations. Looks like its time to fire up Eclipse, break out the python, and put on my Data Miner hat, and start reading those AI books.

As a follow up to yesterday’s post, I’ve decided to try and blog my steps as I go. This way when I’m groaning about it a year later I can see exactly where I went wrong….
Anyways, the first step in problem solving is to identify the problem. My problem is I want to be able to determine if a company (i.e. its stock) is doing well or not in an analytical way.

My first proposal: Analyze news about the company to determine if the news is good or bad. Bad news usually is an indicator of a problem, good news is an indicator of a company doing well (from a stock point of view).

Using a simple Naive Bayes Classifier I think it should be fairly straight forward to analyze the news for each company and determine if the news is “positive”, “negative”, or “neutral”. After building up a corpus of news articles to train the system, the system should be “good enough” to run on its own, and even to have it re-train itself with newer news articles.

I’ve managed to bang out the code to retrieve the news pages (viva la python!), so the next step is to port a simple perl-based classifier (found here in a great article that requires registration) to python, and then let ‘er rip.

Note: Normally I discourage people from trying to reinvent the wheel. After all, if someone else has already done the hard work, why not follow in their footsteps? In this case the concept behind the code is pretty important to what I want to accomplish, so I feel it is important to really understand it. As a bonus the original code is pretty straightforward, so it should be hard for me to screw it up. So those two factors led me to think I should port the algorithm instead of using the original code.

There’s a lot of hoopla and hype in the news these days about Data Mining and Social Network Analysis. Both of these topics are interesting to me, so I’ve been doing a lot of reading as of late. Some of the links I’ve found that I really like are Introduction to Social Network Methods which is a nice text and a good introduction to SNA, and these two links about charting office connections and its importance. If you are interested in more links, be sure to check out my del.icio.us bookmarks. Also, be sure to poke around in del.icio.us, there’s a ton of info in that system.

On a somewhat related note, my boss has started a blog. In a past life he was a stock broker so he started up a site called Hip Egg where he does technical analysis of various companies stocks by looking at their graphs. He has some fascinating ideas and it has stirred up a lot of talk in our office about stocks and data analysis in general.

In fact, the discussion has inspired me to pull out my copy of Spidering Hacks and start looking at what information is out there for various companies. My idea is that I can whip up some python programs to go out there on the internet and gather a ton of data that will either support or refute his predictions about a stock. The reason I think this would be interesting to do is because he bases his predictions almost solely on the shape of the company’s graph; I feel that looking at the numbers behind a company (cash on hand, news reports, product launches, etc.) will give you a more accurate prediction.

That and it will just be fun to tie together several topics like data mining, data analysis, graph/network math and theory, web crawling, and python as they apply to the stock market.

I just read a really fascinating article about how blogs are being looked at as sources for data mining. Apparently a lot of companies out there realize that blogs are a great way for people to spread their opinion far and wide. So naturally a couple of companies have sprung up to supply information about what’s going on in the blog world. Check out the article here.

Although I think all of the fuss about blogging has been blown out of proportion, I do think that the efforts described in the article are very interesting. The fact that algorithms are being used to determine not only the content of the post, but detail about who posted it just blows my mind. This is what Artificial Intelligence is supposed to be all about.

Although I do find the claim that Umbria’s spider can index the blogosphere in 20 minutes a bit hard to believe, I do fully believe that the data it gathers must be a virtual goldmine. Imagine the expert system that could be built using knowledge about a product, what the consumer thinks of it, and what demographic the consumer falls into. Its very Big Brother, but at the same time the applications are almost limitless.

Over the last week or two I’ve been thinking it would be fun and interesting to try and collect the top search terms that are going around on the internet. I’ve started this little project and have been able to collect a fair amount of data, but I realized I didn’t really have a good way to qualitatively look at the data (i.e. to find out what it is telling me). This article has inspired me to look a little bit harder because there is some potentially seriously useful information in there.

I used to think that Amazon was just another dot-com company. Then after seeing them survive the burst, I thought they were lucky. No a days I think they are pretty visionary.

I’ve ranted about A9 before, I use their associates program, hell, I even use their Tip Jar system. Suffice it to say I think Amazon is pretty cool. Now they have introduced this Mechanical Turk website.

My first thought was that the idea was interesting, but why call it artificial intelligence? There’s nothing artificial about it. Maybe from all of the data that get some PhD candidate to build a corpus of an expert system of some type, but that seems a way off. What’s the big deal with that?

Well, that question was answered for me today after reading this article. The author points out that with a system like mturk, Amazon has harnessed its existing user base in a way that can generate a ton of business for them. This puts them light years ahead of their competition. And the way that Amazon pays attention to it web services, it shows that they want developers to get into this technology because it is going to be here to stay.

I think that if Microsoft spends all of its time worrying about Google they are going to get blindsided by Amazon. Amazon is leveraging the web in a way I never thought would be possible. They haven’t just built a brand, they’ve built an army.

Mr. Cringely’s site is always a good source for something interesting that will make you think. The second half of this weeks’ article is about AdWords and how it works internally. Basically over the last few weeks he has been describing an experiment someone was doing to see how AdWords works (I wrote about this back here).

In this week’s update Seeing Is Believing, Cringely wonders if Google is doing something to prevent people from optimizing their site for AdWords by adjusting the Nash Equilibrium point(s) in their algorithm so that Google winds up the long term winner. Google of course denies this, and Mr. Cringley of course doesn’t fully believe them.

Personally I think what was observed was a shift in the equilibrium points, but it was not an intentional effort by Google to do something bad. Rather, from what I have read about AdWords, it sounds like the system is constantly re-evaluating itself and its environment. I would not be surprised at all to find out that the system is capable of adjusting its parameters (to a limited extent, major changes would probably need to be approved/done by humans). I think when the subject of the AdWords article was conducting the experiments, they just happened to pick a time when the parameters were being adjusted (for their particular keywords).

Is adjusting the parameters wrong? No, not if Google wants to stay in business. The internet is a pretty dynamic place and things happen in short time intervals. The probability of someone figuring out how your “system” works is directly proportional to how long it is on the internet. And as soon as some figures out how something works, they try to make it work for them. Adjusting its parameters is a good way to make sure that Google’s AdWords stay relevant and useful and therefore continue to make them money. (And this is more or less what the representative from Google said in the article.)

If anything, I think this is pretty interesting. While reading the article I thought to myself that this topic would make an interesting doctoral thesis. I had to laugh out loud when I read the same thing being said by Google’s AdWords engineer…