The problem. Tag list used to classify chats is not useful.

We had 1400 different tags to label chats. And believe it or not, we just regularly used a small fraction of them. More than 500 of them were used exactly 0 times during the last month (900+ were used less than once a week). This seems helpful to capture all the nuances but in fact, it causes more issues that it solves.

We had an automatic system to label chats that learned from human tagging. Lots of tags don’t help an automatic tagging bot to work. The bot could discriminate which tag to use because it extracted knowledge from about 2 months of chats to understand how tags are used. What if a given tag was not used during that period? The bot could not understand how we use that tag so (1) the tag could not be suggested by the bot, (2) this creating a bias.

Even if a given support agent knows all these tags, having so many makes these tags more likely to be forgotten. If the human support representative cannot remember the tag name, then the bot cannot learn and consequently could not suggest it… the bias we cause is even bigger.

Tags’ definitions were inaccurate. At first glance, we saw lots of overlapping, duplicated or fuzzily defined tags. Also, tags having to do with one-off campaigns or retired features. Many of them are too generic to provide any valuable information or too specific to generate enough data to be valuable for further analysis. The icing on the cake: We had 1400, but a tag called “other” has been used 7K times over time.

Moral: Cleaning up the list of HappyChat tags is something we will need to do as a first step.

Solution. Cleaning the tags list

Our target was to get a list of 400 tags from these 1400. We also have an almost 100 parent tags list that we would like to reduce to about 30.

Parent tags are used to group tags. For instance, “domain” is the parent category of about 35 more detailed tags (#). We should be able to cover any scenario with these detailed tags. However bare “domains” generic tag has been used 30K times. Our plan is to have about 30 tag groups that will be called categories instead of tags. We change the name to make explicit that these groups cannot be used to tag chats: we will use them to group tagged chats or extractions. In other words: we will be able to extract aggregated data from groups of tags (now called categories) but we will not allow these generic tags to be used to tag chats.  

Solution. The process.

We prepared a spreadsheet to keep track of any changes we suggested together with the reason. We identified 3 main scenarios for each tag we analyzed.

1 We suggest removing the tag. In these cases, we added a reason and when it makes sense the substitute tag or a longer explanation. We had a list of possible reasons to choose from:

Some examples:

2 We could keep the tag. And when we kept it, sometimes we renamed it to follow some naming conventions, we could change the parent tag (Now category) or it could need a new/changed definition. So we added these fields to the table. Some examples:

3 We also created some tags. In this case as with those we kept, a name, category and description is needed. Some examples:

What’s the result? A curated list of 400 tags grouped into 39 categories.

We ended up removing 1000 tags. The leading reason was that most of these tags were too specific. Too specific tags are not frequently used, so data extractions are not that useful, they are likely forgotten and the tagging bot cannot learn from them. Other frequent reasons were not being used, having to do with a retired theme or feature, being too generic to be useful or having an overlapping meaning:

• 

• 

We finally got a list of 400 tags grouped into 39 categories. Some notes to take into account:

• Parent tags (themes, domains or atomic) were now real tags that were used to tag chats (in some cases intensively used). Our proposal included categories that are just groups of tags. Categories can be used to extract and aggregate data but cannot be used to label interactions.
• Remove a tag doesn’t mean to remove the current database information for that tag. It’s just that its usage would be disabled.
• We created a category to group tags (e.g. quality_review, training, WPcom_woo…). Flags are not topics but could be used to extract information or prepare samples to be studied or compared.
• The naming format included the name of the category. E.g. the above flags names would be flag_quality_review…

Next steps: Tag Keepers

My suggestion was to create a group of ‘tag keepers’ to keep the tags list and structure updated and homogeneous. The idea would be to give just to this group of people permission to create, update, disable/enable or remove tags. If we do this, they (1) could analyze the usage of the possible tags and also could receive feedback/suggestions from HEs or other stakeholders to periodically update the list of possible tags. This meaning disabling no longer used tags but also adding new tags. They would also (2) keep naming, description and specificity level consistency (avoiding having very detailed tags living with ultra generic ones, for instance).

#data-wrangling #tagging