I recently attended the Sentiment Symposium in NYC that Seth Grimes puts on a couple of times per year. This one was interesting for two reasons: the first being the increase in amount of heavy duty academic research on sentiment scoring being done by some of the big universities (specifically University of Pittsburgh), and secondly for some astoundingly absurd accuracy claims that are being made by some of the new tech companies trying to make a name for themselves in Sentiment Analytics, which is what I’m going to focus on here.
There were at least 2 different talks that claimed sentiment accuracies of better than 90%, which if you know anything about scoring sentiment is totally absurd because humans under the best of circumstances (when they’re well trained and understand the domain of the content they’re scoring) can’t quite achieve 90% accuracy. For commercial systems to claim accuracies of 93% and better puts them squarely in the “snake oil salesman” camp in my book. Perhaps the most interesting part of these talks was the reaction of the audience and our moderator Seth Grimes. The audience clearly had a difficult time not audibly snickering at these presentations that were essentially claiming that their systems were smarter than humans. Even more entertaining was watching Seth Grimes try and be a good host and not visibly mock these speakers. For those of you that know Seth, you can just imagine how tough this must have been, as Seth is, if nothing else, a “straight from the hip” kind of guy who pretty much tells you what he thinks. He mostly succeeded, though he did cut off one of the talks that was trying to go over, much to the speakers visible dismay 🙂
So, what’s the point of this… It’s that anyone looking for sentiment scoring needs to be a smart consumer, don’t believe wild claims of accuracy. Make the vendors give you the software and run your own tests and see for yourself what the accuracy is. My suspicion is that you’ll rarely see accuracy numbers above 80%, and that’s OK because software simply isn’t as smart as people and anyone that tells you differently is _______________ (fill in your favorite word here).