AI in Impact Investing: Goodhart’s Law, Unintended Consequences, and the Dangers of Blunt Metrics

This post is the intro to a series on AI for Good in financial services. At Silo.AI we are always looking for opportunities to combine our financial AI expertise with our ethical commitments. What better way to do that than through engaging with our partners in the growing field of Impact Investing - funding projects that generate investment returns, alongside socially and environmentally beneficial outcomes. Follow us on this journey and get in touch to know more.

In my former life in sales and trading at an investment bank, my employer introduced a new Customer Relationship Management tool. In a curious but welcome departure from the prevailing conventions in finance, this system was given a name that was neither ludicrous, nor opaque. The Business Management team missed a major opportunity to call it something intense and creepy, like the trading algos that get called “Nighthawk”, “Knife” or “Guerrilla”. I won’t give you the exact name in case it’s protected, but it’s close enough to say it was called CustomerTracker. CustomerTracker was, in theory, as brilliant as its name was prosaic. It had all the CRM functionality you could want: as relationship managers we could now monitor our clients’ interactions across the bank with incredible granularity. By automatically scraping our chats, messages, phone calls, emails, calendar entries, and so on, the system built complex maps of the bank’s engagement with its client base. We could see which of our clients we should be collaborating to cross-sell to, which we needed to engage more, and which should be cut off. When you have a platform as big, diffuse and expensive as that of an investment bank, it pays to know which clients consume resources way in excess of their contribution to the bottom line.Gaming the systemHowever, despite the promise, in practice, CustomerTracker created some extremely counterproductive effects. Management understandably wanted to encourage and reward the most “intense” salespeople, and decided that the best way to monitor that was the number of total interactions. Well, very quickly, even the more conscientious salespeople quickly noticed that they could game that system pretty easily. Why on earth would you have one big, extremely productive and efficient conference call with dozens of your contacts at a client and several of your specialist colleagues in the line, when you could call each of them in turn and rack up a load of discrete interactions, causing management to see you as extremely industrious? It’s all about those interactions! So, once CustomerTracker went live, client engagement by one blunt metric went up, but the value of interactions to clients arguably plummeted.Goodhart's LawThis is a clear-as-day example of a phenomenon called Goodhart’s Law. Goodhart’s Law states that “when a metric becomes a target, it ceases to be a valuable metric”. In other words, when you track a system’s performance using one variable, participants in the system will adjust their behaviour in order to maximise solely that metric. Simply put, when people know exactly how they’re being assessed, they will inevitably game the system. In the case of CustomerTracker, what we care about is engagement, but we measure that using the metric of frequency of interactions as a proxy. So what do we get? Piles of low quality interactions. Thousands of hours of employee time spent sub optimally. A value alignment problem.The introduction of CustomerTracker and the “interactions” metric was actually, in some ways, a very intelligent attempt on the part of our Business Management teams to counter some effects of Goodhart’s Law that were already in operation. Amongst other things, it aimed to inhibit existing negative behaviour within the sales force. Previously, the bank only had a single measure of a salesperson’s contribution - the annual revenues on their client accounts. But this hadn’t accounted for salespeople who were, in economic parlance, engaging in “rent-seeking” - sitting on annuity revenue streams, which required little or no effort to maintain them, and using those revenues to cultivate protected positions within the organisation. Surely, CustomerTracker would finally expose that lazy behaviour! And indeed we quickly found there was always someone who somehow managed to have fewer interactions with their clients during an average week than you racked up last week while you were away on holiday.Complex systems can't be monitored with one or two metricsNote how this neatly demonstrates the problem of monitoring and managing complex systems. Even when intelligent leaders with the best intentions introduce additional metrics to increase oversight, a sufficiently complex environment will still leave room for the system to be gamed. Lazy salespeople with no interactions might be exposed by CustomerTracker, but employees who were previously effective are now incentivised to pile up low quality interactions.And with every additional metric introduced, the system becomes harder to monitor, and it becomes harder to know how to balance the interplay of the variables. Optimisation in multivariate systems can be a tough, tough gig. All the more so when autonomous and self-interested agents are involved, and especially when those agents are salespeople with a high proportion of discretionary, performance-related pay.So, how does this relate to AI? And why should we care?Well, let’s start with why we should care, and let’s hone in on one of our favourite topics here at Silo.AI - Impact Investing. Impact Investing is a new, but massively important and rapidly growing field within finance. Impact is all about allocating capital, not just on the basis of financial returns, but with the aim of supporting projects that deliver social and environmental benefits.How do we measure those benefits, and what are the risks when we assign metrics to them? Let’s imagine we have the aim of zero carbon dioxide emissions in an exciting new clean energy infrastructure project. How could Goodhart’s Law hurt us? Well, maybe through relentless fixation on the goal of carbon neutrality, the project’s manager ends up favouring processes that emit tons of sulphur dioxide instead. Or maybe our carbon capture systems pollute a nearby water supply, or destroy local wilderness.These are obviously cartoonish examples. But they capture a real world problem. We need to get this right - or at least we need to get it as right as we can in an imperfect world. And even then we’ll probably have to keep an eye on it and adjust our processes as we go along. Luckily, we aren’t alone in this. We now have a helping hand from technology that holds huge promise in reducing complexity in multivariate systems, and supporting human experts while they monitor Impact projects, in the form of Artificial Intelligence. In future blog posts we are going to consider some of the proposed solutions to Goodhart’s Law, and the ways in which AI can support those efforts, both on the Impact side, and on the Investing side. When it’s the future of the planet and our societies that we’re talking about, we really can’t afford to get this stuff wrong.

This post is the intro to a series on AI for Good in financial services. At Silo.AI we are always looking for opportunities to combine our financial AI expertise with our ethical commitments. What better way to do that than through engaging with our partners in the growing field of Impact Investing - funding projects that generate investment returns, alongside socially and environmentally beneficial outcomes. Get in touch to know more.