We're really excited to say that our machine learning keyword classification tool has just been shortlisted for the UK Digital Growth Innovation award!
[EDIT: as of 31st July, the tool has also been shortlisted in the Best Use of AI category at the Digital Impact Awards! Woohoo.]
We wanted to share a bit more about the tool, how we made it and how it's helping us with keyword research at Three Whiskey.
Keyword research is an essential part of how we do search for our clients. It helps us understand how people use search engines to research given topics, and reveals insights about audience needs across sectors, geographic locations and specific companies. At Three Whiskey, we conduct in-depth keyword research as part of developing every strategy, to help us understand the landscape our clients and their competitors are operating in.
As part of the process, we classify keywords into thematic groups, and by audience type. This lets us understand audience behaviour and search themes even more comprehensively.
We’re always thinking about ways to improve how we do integral work like this, and that’s how our innovative tool Classy - as we like to call her - was born.
Our research, analytics and data (RAD) team knew their SEO colleagues were classifying thousands of keywords a month, and they were certain, with a bit of RAD magic, they could find a time saving solution using machine learning.
Over a relatively short time, using open source code and with no cost other than the time we invested, we were able to create a solution that will save hundreds of days’ work for our busy SEO team over the next year and beyond. And that’s time we can re-invest into our clients’ projects, to help them achieve more for their businesses.
The nitty gritty
Using a supervised machine learning model, we fed keyword training data into an open source algorithm. The data consisted of keywords and associated categories, based on manual keyword research from the SEO team’s archive. To start with, we trained Classy in keywords from the healthcare industry, teaching her to recognise keywords based on whether they would be used by an audience of either ‘patients’ or ‘healthcare professionals.’ We did this training over 5 data sets, feeding in 14,569 keywords in total. After this training had been absorbed, unclassified keywords could now be run through Classy.
The feature extractor in the algorithm only extracts the information we’ve told the model is relevant to the classification problem. So how well the feature extractor works is paramount to how accurate the output classifications will be.
The extractor attributes a set of features to the unclassified keywords (that are either stemmed words, word N-grams, character N-grams and others). Finally, Classy gives each unclassified keyword a label, and a measure of certainty about the label being correct.
Since starting this project, we’re seeing a consistent 80% success rate for Classy accurately classifying the keywords we feed in. Where Classy does not have enough information for classification, we manually categorise the keyword and feed it back into the model. This means it’s constantly learning and improving in accuracy.
In this video, you can see the process of adding training data, and then running Classy to classify a sample of keywords.
The Classy effect
Since we started using Classy in 2019, she has classified tens of thousands of new keywords, and saved the agency over 85 days of manual classifications. As well as aiding efficiency, Classy has helped standardise keyword research internally and eliminated human error - meaning we can deliver a higher consistency of work to our clients.
We are continuing to teach Classy all the time, so we can use her abilities across multiple verticals and projects. We’ve already started to differentiate the data used to train Classy, going deeper into therapy areas in healthcare to improve accuracy, as well as branching out into keyword classification in travel and ecommerce.
Classy started as a clever time saving tool for our team. But with the additional data we’ve trained her on so far, and by developing her user interface, she soon turned out to be an essential keyword knowledge resource that will only become more powerful, valuable and accessible across our agency, as time goes on.
Got an open mind? Want to keep it that way? Sign up to our newsletter. It's full of thought-provoking insights from the front line of performance marketing.