On Saturday April 27th, viafoura is hosting a hack-a-thon to see what interesting ideas developers, product designers, and data scientists can come up with when given large data sets.

Hackers are armed with 10 years worth of  news stories from the Guardian (one of UK’s largest news publications), computing power from Amazon ($250 credit for each person), a natural language processing tool (Lexalytics-Semantria) and space donated by Ryerson’s Digital Media Zone. Mix all this together and you have all the building blocks to come up with some amazing big data projects.

Amaze us!  Wow us!  Amuse us!  Oleg Rogynskyy, the CEO of Semantria and I will judge and are both super excited to see what you can come up with!

Here at Lexalytics, we’re excited to be in beta with Salience Text Analysis for Chinese.

There are many features in our toolkit – sentiment, topic detection, summarization, theme extraction – but sentiment is what we’ve been best known for. With our beta release of text analytics of Chinese content, we decided to measure our document-level sentiment results against human annotations of sentiment, and compare to another public engine recently released that also provides automated sentiment analysis of Chinese.

What started as a basic measurement of precision and recall turned into a deeper effort to quantitatively determine how closely our sentiment analysis matches the sentiment judgment of multiple humans.

Setup

We gathered 109 pieces of Chinese content from Weibo and blog discussion forums each of which we annotated as positive or negative. The content was filtered to items that were clearly positive or negative to a human. Our intent was not to measure ability to detect subtle cases of sentiment.

Even though we marked content as only positive or negative, Salience, being phrase-based, can return a neutral result if it is unable to detect any sentiment at all. We think this is a valid approach – sometimes there really isn’t any sentiment.

The other sentiment engine that was tested, Chatterbox, was not observed to return a neutral result; it appears that all content is categorized as positive or negative, with an associated strength score.  It could be argued that you could consider a polar result with a small strength score to be "neutral"

Additionally, the phrase-based approach developed for Salience to assess sentiment in Chinese was developed from longer form text, but shorter text was needed for this test to accommodate a content length constraint imposed by the Chatterbox API.

Precision, Recall, and F1

The table below gives the precision, recall, and the weighted average (F1) for positive and negative sentiment within the set. The F1 score for positive sentiment and F1 score for negative sentiment are combined to calculate an overall accuracy measure.

These scores are quite good for both engines, which can be attributed in part to the polarity in the test content selected. Salience performs comparative to Chatterbox in terms of positive sentiment, with slightly better performance on the detection and identification of negative sentiment.

As we have developed support for non-English languages, we have done so at a core level of deconstructing the language, and developing the support needed to handle the language natively. We feel this is a much better approach to NLP of non-English languages than using machine translation techniques to apply techniques developed for English to translated text. In order to test this, we took the Chinese content and ran it through the Google Translate API, and ran it through Salience's standard distribution for English.

As you can see from the table above, a machine translation approach suffers from any translation issues, particularly when working with phrase-based detection of sentiment where other linguistic modifiers such as negations and intensifiers are taken into account. A model-based sentiment approach may be less affected by these issues, but will be less flexible to use across varied content domains and require more technical tuning effort.

Inter-rater agreement

To me, this is the most interesting part of the experiment. Automated sentiment analysis is often compared to human sentiment analysis through precision and recall tests. But that assumes that across humans there would be 100% agreement. In reality, there are discrepancies in the sentiment annotation across multiple humans, and in many cases the same human can mark the same set of documents with slight differences from one day to the next. So we want to measure the consistency of multiple human annotations of the same content, and calculate the consistency of the two automated sentiment approaches to human judgment. After all, if you can’t agree with another human, why expect the machine to agree with you? 

The same content was also annotated by an external contractor, a native Mandarin speaker located in China. Two inter-rater agreement measures were calculated, Krippendorff’s alpha and Cohen’s kappa. These two measures were also featured in an analysis of inter-rater agreement presented by Maritz Research at the Text Analytics World seminar in the fall of 2012.

For our dataset, both Krippendorff's and Cohens'indicated an agreement of 94% across the two humans, showing that even for a relatively small set of very polar content there is not absolute agreement between two humans.

In one particular example, we marked an article that was very prejudicial against Japan and pro-Chinese as negative, because of the emphasis of the prejudice. The contractor based in China, however, considered this to be positive content. Judging sentiment can be tricky, even for humans.

So how did the computers do?

Calculating Cohen’s kappa requires the results to be fully annotated, so for cases in which Salience returns an inconclusive result we considered what agreement would be if that result was taken to be a positive or negative result.   In other words "0=neg" means that if Salience returned a result of 0 (one that we would normally consider to be neutral), we will consider this results to be "negative".

Cohen’s kappa also only allows us to calculate agreement between two raters. The conclusion we can draw from this calculation is that humans agree 94% of the time on their ratings of the content, Salience agrees between 74% and 84% of the time, slightly better with one human than another and slightly better when inconclusive results are considered negative (or not-positive). Chatterbox fares worse with about 62 to 64% agreement with humans.

The calculation of Krippendorff’s alpha is more flexible, allowing for gaps in the annotations which accommodate cases in which Salience did not detect sentiment and allowing for determining sentiment across a group of more than two raters.

The most interesting chart is below - so, for multiple raters (more than two), what are the best combinations. Because we're not forcing a "0" from Salience into either positive or negative, the agreement numbers end up better.   We're about 75% agreement across humans + Salience + Chatterbox.   (Which says good things about the state of sentiment analysis!)   We're, of course, happiest with the results between two humans and Salience, where we're pushing 90%.   

These results show that Salience’s analysis of sentiment in Chinese content correlates well with human judgment. Perhaps not quite well enough to satisfy a Turing condition of generating results which are indistinguishable from those of a human, but certainly close enough to serve as a good starting point from which further phrase-based sentiment tuning can refine results.  

Conclusion

We think these results validate our approach of native NLP phrase based sentiment analysis for Chinese over a machine translation approach or classification model and show that on tonal content, Salience is a compelling option. At present, our attention is focused on including named entity recognition for our general release of support for Chinese. We’re pleased with the results of this assessment of our initial document sentiment analysis, and looking forward to bringing the full product to market. 

Carl Lambrecht, one of our many language geniuses, wrote up a post on the Lexalytics Dev blog about how dialect differences affect text analytics, especially when taking into consideration the fact that Lexalytics only uses Native Language Packs.

He touches on a couple of great points. For instance, most dialect differences are only apparent during speech and pronunciation, not in writing, and when they do appear, Salience can often figure out what the strange new words are based on context. 

Head on over to our Dev site and check it out for the full scoop.

At Lexalytics, we know it’s not only a global marketplace, but a multi-lingual global marketplace. It’s this understanding which has driven us to extend the capabilities of Salience beyond analysis of English.

Salience was originally developed by Lexalytics to analyze English text, traditionally from online news articles and blogs. Over the continued development and maturation of the product, more and more complex analysis of online content, including Twitter, has been added and improved.

The first non-English language we tackled was French. Our first release of support for French provided almost all of the features found in our analysis of English; part-of-speech tagging, novel named entity extraction, theme extraction, summarization, and model-based sentiment analysis. The only feature in our English language support that was not available in this initial release was phrase-based sentiment analysis. As opposed to model-based sentiment analysis, phrase-based sentiment is much more tunable and transparent: you know exactly which phrases influenced the results as well how, and can tune the phrases to your needs accordingly. An update followed shortly which added an  improved handling of French social media content.

The next languages we tackled were Spanish and Portuguese, again providing model-based sentiment analysis. These languages have not yet undergone an update to specifically address unique social media complications in those languages, yet they do benefit from the techniques built into Salience itself for adjusting to handle short content.

With Salience 5.0, Lexalytics introduced the core feature of the Concept Matrix™, a model of semantic relationships derived from Wikipedia. At the same time, we were developing support for German. Additional research and development allowed us to provide phrase-based sentiment analysis for German, our first non-English language pack to include this capability.

Over the course of the summer of 2012, we worked to develop updates to our existing language packs that would bring all of them up to the same feature level. Updates have been released for French (June 2012), Spanish (September 2012), and Portuguese (October 2012) that bring all currently supported languages up to the same level of functionality.

While our current language packs will see ongoing improvements (better novel entity extraction, better quotation extraction), we have our sights firmly set outside the Romance and Germanic languages. We are hard at work at applying our experience and techniques to Chinese, looking to release our initial support in early 2013.