The digital revolution has given us an abundance of data during the last decade or so. This is interesting for a variety of reasons, but most notably, in terms of how AI will continue to disrupt business. However, in the realm of B2B — the industry in which I work — we still have a data scarcity, owing to the fact that the number of transactions is much smaller than in B2C. As a result, in order for AI to fulfill its promise of transforming the company, it must also be able to handle these tiny data issues. It can, thankfully.
The problem is that many data scientists use terrible techniques, which creates self-fulfilling prophesies, reducing AI’s efficacy in small data settings – and eventually hindering AI’s influence in organizational advancement. The key to effectively applying AI to minor data challenges is to follow good data science methods while avoiding poor ones.
The term “self-fulfilling prophecy” is used in psychology, finance, and other fields, but it may simply be characterized as “predicting the obvious” in data science. We see this when businesses develop a model that forecasts what currently works for them, sometimes “by design,” and then apply it to other circumstances. For example, if a retailer decides that customers who put items in their shopping cart online are more likely to buy than those who do not, they would extensively promote to that group. They are foreseeing what will happen!
Instead, companies should use models to improve what is not working, such as converting first-time purchasers who do not have anything in their basket. This retail organization will be considerably more likely to affect sales and attract new consumers instead of merely maintaining the existing ones if it solves for the latter — or predicts the non-obvious. To avoid falling into the trap of making self-fulfilling prophesies, here is how to use AI to minor data problems:
Enrich your data: If you do not have many existing data to work with, the first thing you should do is enrich the data you do have. This may accomplished by using external data to perform look-alike modeling. We are seeing this more than ever before, owing to the growth of recommendation systems like Amazon, Netflix’s, and Spotify’s.
Even if you have just made one or two purchases on Amazon, they have so much data on items throughout the world and the individuals who buy them that they can forecast your next purchase very accurately. If you are a B2B firm that categorizes your deals using a “single dimension” (for example, “big corporations”), follow Pandora’s lead and dissect each customer to the greatest extent possible (e.g., song title, artist, singer gender, melody construction, beat, etc.).
The more you learn about your data, the more valuable it becomes. You can get from low-dimensional data to high-dimensional knowledge with powerful prediction and recommendation models by starting with low-dimensional data and making basic predictions.
