Datasift & Twitter Firehose

  1 m, 40 s

Here’s the third video taken from our LUG conference.

Lorenzo Alberton of Datasift discusses their overall architecture for processing the entire twitter firehose. Specifically, they use the Lexalytics Salience engine for sentiment analysis, topic extraction, and named entities, in all 4 languages covered by Salience. In Lorenzo’s words: “we chose Lexalytics because it is the only product that can handle this load at the speed we want.”

Lorenzo goes on to show the incredibly large amount of metadata that they can extract from just 140 characters (plus the other stuff that Twitter is providing). This data includes but is not limited to sentiment analysis, entities (people, companies, dates, places, etc), each user’s profile, as well as how influential they are and what topics they discuss, demographics (gender, age, etc), trend analysis by location, and much much more.

Once Datasift has this data, they use Salience to build filters that allow them to pinpoint specific information, based on location, place, language, position, follower number, etc. In the later half of the presentation, Lorenzo demonstrates the practical applications of their work. One specific example involved a news break, followed by a surge in popular opinion about the news, followed a few minutes later by changes in relevant stock prices. It was very cool example of cause and effect in action, which would have gone completely unnoticed without Datasift’s intervention of Twitter sentiment analysis.

DataSift is the real time social media filtering engine from MediaSift Ltd, which builds upon the success of TweetMeme, the consumer facing brand of MediaSift Ltd. DataSift provides very granular and modular ‘sifting’ functions from across a wide range of social and web input feeds, augmenting them with sentiment analysis, storage and analytics to offer an unrivalled service platform, which leverages the cloud and scales infinitely. DataSift uses Salience to help their clients manage their brand, by monitoring what people are saying about them across the Internet. Twitter sentiment analytics, blog and news monitoring, and entity recognition come included with their services package.

Categories: Events, Sentiment Analysis