In a world awash in cultural relativism there remains one absolute commandment: Thou shalt not be -ist. Thou shalt not be sexist, ageist, racist, specieist, ableist, intellectualist, sizeist, capableist, healthist, wealthist nor wiseist (actually, I made that last one up). Nor shalt thou be -phobic: thou shalt not be homophobic, islamophobic, biphobic, transphobic, xenophobic and certainly not blackaphobic (yes, apparently it is a thing). In short, thou shalt not be judgmental – and certainly not pre-judgmental or prejudiced. The motivation behind the anti-ists is to have us act as if we are colour blind, gender blind and blind to all the other traits which may differentiate us one from another, which seems to me a little extreme.

At the same time, I find blatant racism, for example, to be extremely distasteful. Close-minded hatred of a large group of strangers simply on the basis of a single trait such as the colour of their skin or the nature of their religious beliefs clearly has no place in a rational and productive life. But there is a middle way. It is possible to navigate between the Charybdis of fearing the phobias on the the one hand and the Scylla of extreme prejudice on the other. And strangely enough, it is in the discipline of machine learning and artificial intelligence that the nature of this middle way may be found.

In in the modern world of so-called ‘big data’, corporations large and small develop a plethora of statistical models to ‘score’ their customers on a range of issues. They develop ‘propensity-to-churn’ models, ‘next-best-offer’ models, ‘cross-sell’ and ‘up-sell’ models, ‘propensity-to-complain’ models and ‘credit risk’ models. Insurance companies develop models to price insurance products based on the characteristics of their customers. Many immigration departments score individuals who seek to cross their borders on the likelihood that the person is up to no good. Those with low risk scores breeze through without let or hindrance while those with suspiciously high risk scores are detained for questioning. In short, these models judge individuals based on the available information they have and assign a propensity or risk score based on their characteristics. It will come as no surprise that the key variables which distinguish individuals one from another are more often than not exactly those characteristics which we as humans are forbidden to admit we take into consideration: gender, race, age, religion, socio-economic-status and level of education.

The way these models make their decisions is important to understand. Good predictive models are multivariate, meaning that they take into consideration a large number of variables simultaneously and score individuals accordingly. They ascertain which variables are significant, not by considering each variable one-by-one in isolation, but simultaneously taking into account the effect of all the variables. Furthermore, it usually turns out that even the best single multivariate model can be improved upon. This is achieved by combining a number of models which use slightly different algorithms to predict the same outcome into an ensemble. For example, the modeler might develop a logistic regression model, a support vector machine mode, a random forest model and a naive Bayes model, all of which are attempting to solve the same problem. The ensemble is then formed by taking the predictions from each and combining them into one master prediction. It turns out that ensembling is a highly effective method to develop accurate predictive models. Almost all winners of Kaggle predictive modeling competitions, including the $US3 million Heritage Health Prize, use ensembles to optimise predictions.

So how does this apply to everyday life? One of the crowning achievements of modern capitalism under the rule of law is that it facilitates commercial interactions between strangers. Our business dealings are not based on family, tribe, kith or kin. Rather, we do business with people who are more or less total strangers. When you meet a stranger, you need to assess and make judgments about them very quickly based on limited information. You need to decide if you trust the person, how much information you’re willing to share, how much time and effort you should invest building a relationship and so on. To do so you need to mimic the propensity modelling approach of a multivariate predictive algorithm as closely as possible; you need to estimate the characteristics of interest about the person you are dealing with, and to update those estimates as new information comes to hand. To do so you need to take into account everything you know about the person, including their race, gender, age and appearance. To willfully exclude information on the basis of ideology undermines your ability to make effective decisions.

As Malcolm Gladwell argues in Blink, the best decisions are made by those who are proficient in ‘thin-slicing’ – the ability to identify the few factors that are particularly pertinent in a given situation. Excluding potentially significant variables a priori will inevitably degrade the decision making process.

However, in one way, being both anti ist and anti phobic is perfectly rational and an intelligent way to deal with the world. If, when they say ‘don’t be homophobic’ the anti-phobes mean, ‘don’t solely take into account the sexual orientation of the person’ they are right. This would be the univariate approach. Which is why diversity targets and gender quotas are so wrong headed. They give undue weight to a single aspect of a person’s identity and view the individual solely as a representative of a single class – female, indigenous, disabled – minimising all other factors. If, on the other hand, the anti-racism brigade mean we should completely ignore the race of a person they are making us exclude a variable which is almost always significant; clearly not a good idea.

Just as machine learning ensembles work well because they pool the collective opinions of multiple algorithms, each of which looks at the data available in a slightly different way, we too can make judgements by talking with and listening to people which whom we do not agree and subconsciously pooling multiple conflicting opinions when developing our own views and judgments.

So next time you’re accused of being racist, sexist or any other flavor of ist, defend yourself by explaining that if multivariate ensembling is good enough to win multi-million-dollar data mining competitions, then it’s good enough for you.

The post The case for Isms and Phobias appeared first on The Spectator.