With customers wanting to process more and more content (and more and more content being available) and wanting to do that in an a near real time manner as possible, the throughput speed of our various components becomes something that we are very interested in. Of course customers always want more features added as well (Pronominal CoReference, Sentiment Concepts etc.) which in turn require more complex engineering solutions (grammatical engines) which in turn require more processing time. Trading off this increased complexity with the need to keep the document throughput up can be a challenging task. My internal metric has always been the ability to process 2 normal sized (1000 -1500 words) documents a second in a single thread. If we can keep to that then scaling can become a hardware problem, add more cores and you get more throughput, but with all the changes we’ve made over the last couple of releases, it was time to see if we were still keeping up with that. To this end, last week I had one of our engineers profile the various bits and pieces that make up Salience internally and we came up with some interesting conclusions

• Machines based around a Core 2 architecture completely trounce the Pentium 4 architecture with Salience. A 3.4Ghz Pentium D was only 20% faster than a 1.66Ghz Core 2 machine and was slower than a 2.5Ghz Xeon based machine.

• Perhaps not surprisingly, the 64 bit versions of the software were substantially faster than the 32bit versions with a document taking 338ms on 64bit compared with 458ms on a 32bit platform.

• Linux and Windows performance was pretty much the same, so no adding to platform debates there 

The testing also identified (as we hoped it would) a couple of areas that we could make some more speed improvements in the upcoming release so we should be able to get the numbers down further by the time 4.1 goes out the door. And as for the internal metric. Well as you can see for the numbers above, even before we do some more performance enhancements we are still at the 2 docs a second mark, and on the 64bit platform we are nearly at 3, which all in all makes me happy and a happy Mike is a good thing.

Now I’m big fan of the Drama 2.0 blog as I think he generally calls it right about most of the Web 2.0 and Social Media news around, but this time I think he’s got it dead wrong. As I’m sure a lot of you are aware the World Economic Forum took place in Davos and Robert Scoble scored a chat with Facebook Honcho Mark Zuckerberg. Amongst the usual fluff this interesting titbit came out

Facebook is, he told me, studying “sentiment” behavior. It hasn’t yet used that research in its public service yet, but is looking to figure out if people are having a good day or bad day. He said that already his teams are able to sense when nasty news, like stock prices are headed down, is underway. He also told me that the sentiment engine notices a lot of “going out” kinds of messages on Friday afternoon and then notices a lot of “hungover” messages on Saturday morning. He’s not sure where that research will lead. We talked about how sentiment analysis might lead to a new kind of news display in Facebook. Knowing whether a story is positive or negative would let Facebook pick a good selection of both kinds of news, or maybe even let you choose whether you want to see only “happy” news.

Now Drama sees this news as a excuse to poke fun at Zuckerberg but I actually see this as a really big thing and it could just be the final piece of the puzzle that makes automated sentiment analysis an accepted technology in the same way that search has. Of course at the moment Sentiment Analysis is used in several ‘Voice of the Customer’ type applications and also in various trading systems around the world but most of that analysis occurs on news data, which is generally well structured and well written but it hasn’t quite made the breakthrough to being a standard technology. As Jeff alluded to in a previous entry, there is still a lot of smoke and mirrors in the industry and whilst corporate customers get the advantages automated sentiment analysis can often bring to their business, they still have a mental hurdle to get over as to its effectiveness and ROI - of course 10 or 12 years ago Search was in much the same place! If Facebook can get it right and actually show on the site that determining sentiment enables them to do x and y better, then it makes it a win for all of us. The challenge they face is that its data tends to be a small number of words and contains both text speak and spelling errors, which tend to make a mess of automated tagging systems etc. Of course it would be even better if they opened up there (the users?) content and let the rest of us play with it as well, but I realise that’s just a pipedream - ah well maybe one day.

Given the continuing stream of bad news that assaults us each and every day… 5000 layed off here, 8000 there, and the political parties battling over what shade of lipstick to apply to the pig, you wouldn’t be alone in feeling down and uncertain. There are however some bright spots out there, and thankfully we seem to be dead in the middle of one of them. There was a very interesting article in PRWeek that we came across today discussing the success that a number of the vendors in Social Media Monitoring are having in this economy, and thankfully some of them are our customers. While this may come as a suprise to some, its something I expected to see, because one thing companies can’t afford to do in this economy is ignore the bad news. They need to know what’s being said about their companies so they can minimize panic and damage around their brand. Of course this means they have to look into more places (blogs, twitter, facebook, etc) and have less people to do the looking (remember the 5000 layoffs at the top of this post), so they are having to look to companies like dna13, Evolve24 and Cymfony to do the digging for them. The PRWeek article spells out how well many of these guys are doing in this tough economy. My general opinion is that the companies that help companies handle the ever increasing glut of information that’s out there are going to thrive in this economy, and thankfully we’re one of those.

I’m a techie at heart. And there are few things that interest a techie more than the shiny new things. This week’s shiny new thing of course is Google Latitude. The basic idea is that your location can now be shown on Google Maps through geolocating your IP address or smart phone and shared with your friends. One initial response to this release has been the concerns about privacy. On Wednesday evening’s ABC News, John Berman reported on the “privacy paradox going mobile”, raising the possibility that “there are times when you don’t want parents or bosses knowing where you are, in the middle of the day, when you’re supposed to be writing a story” as he walks into a movie theater. Mr. Berman does go on to describe the controls that Google has put in to restrict who can see your location and to what accuracy. To me, it all goes back to the core struggle in social networking, which is how much information do you want to give to your network of friends, and how close-knit do you keep that network. Some things that Latitude is not going to let someone do, if Google keeps to their statement about sensitivity of location data:

1. Display where you have been, only where you are right now.
2. Display where you are to anyone that you have not previously expressly stated you wanted to give that information to.

How this is going to benefit the social networking world, in a way is yet to be seen. Currently I have my location shared with one friend, who is 200 miles away in New York City. And if my boss wanted to know where I was at, he probably wouldn’t look on Latitude to see where my smartphone was located, he would just call me on it. I hope it is something more useful than helping me find friends close to me that want to grab lunch or a latte, which seems to be the main benefit to a similar application for the iPhone called loopt. Jordan McCollum’s “Stalk Your Friends with Google’s Latitude” asks the question, “Will you be using Latitude?” But I think the more interesting question is what will you be using it for?