Why Is Explainability Important and How Can It Be Achieved?

Picture this: You are a data scientist and you are reviewing a neural network architecture built by one of your colleagues. The model is highly predictive and would cause significant uplift to the business. “That’s great,” you say.  “How does the model arrive at that decision?” Before finishing the question, you already know the answer, “I’m not quite sure, it just does.” Unfortunately, this is a common scenario in data science labs the world over. And hence the reason that explainability has risen to the fore.

But what exactly is explainability? Its definition is quite straightforward — it is the ability to explain in human terms what is going on within the internal mechanics of a Machine Learning system. However, unfortunately, its application is anything but simple, but more on that later.

So, why is explainability so important? There are a number of reasons, but the most significant is the recent change in data protection legislation, particularly in Europe. Under GDPR, companies are now required to know the provenance of the data they hold and process. Furthermore, consumers have the right to know exactly what their personal information is being used for, and why decisions have been made as a result of their data. Given that customer data is a primary source of fuel for the algorithms constructed using Machine Learning, organizations have a legal responsibility to understand these models.

Read More: Smart Managers Know There’s More to Leading a Team Than Simply Being a Boss

Moreover, GDPR notwithstanding, explainability is a case of plain and simple ethics. Understanding a model to anticipate any unintended consequences and potential bias that could impact vulnerable customers (or indeed any customer) is morally the right thing to do. Data ethics are increasingly coming under scrutiny and several Think Tanks and organizations around the world are creating ethical frameworks to enable data science to move forward but in a responsible and answerable manner.

The problem is models are becoming more complex, with larger numbers of features and increased feature engineering, which make explainability much more of an issue. Transparency is not just important legally and ethically but also practically, for instance, for software debugging or certification. In recent years, Neural Networks have made great advances in multiple application areas, however, they are notoriously opaque and therefore it often isn’t clear which feature is important or has had an impact on the outcome of a model.

Read More: AI in Cybersecurity: Applications in Various Fields

Explainability can be achieved in the following ways. One is through feature replacement and reverse engineering. Models can be retrained without a key feature and the impact of removing this variable on the outcome of the model is analyzed. Alternatively, the use of a simpler model, such as a regression model, can be used to predict the outcome of the more complex one.  If your simpler model can achieve a relatively high ROC (Receiver Operating Characteristic) score, it is possible to evaluate the most important drivers within the model, even if the detailed permutations are out of reach.

Both methods take time, patience and skill and the temptation will always be to skip this step, in order to move on to the more exciting job of building the next impressive predictive model. However, if data science is to truly become the powerhouse that is being predicted, it is crucial that it does so in a responsible way. And explainability will be key to this.

Read More: How to Ensure AI Doesn’t Make Your Customers Hate You