Category Archives: Psychology

The mystical underpinnings of Facebook's anti-fake news algorithms

Imagine you're Rene Descartes, data scientist at Facebook in charge of saying true things. Problem is, your only input is what people say, and everybody lies.

It's not a moral failure, it's just that you've built Facebook into the world's largest integrated content distribution machine, and convincing people of things that aren't true is where the money is. So, how will you synthesize a truthful news feed about the world from the reports of people who are as likely to be trying to deceive you as not?

You can't. There's no algorithm that'll allow you, on its own, to reverse-engineer truth from an active opponent that controls, directly or indirectly, what you see. The original Descartes got away from this problem through a theological leap of faith, but that's not going to help you here.

For all the quantitative processes that make the practice of science possible and fruitful, the roots of it are, fundamentally, social. Science doesn't work without scientists, people who, as a group, are socially and personally committed to, basically, not trying to con you into believe something they know is false. Algorithms, heuristics, big data sets, and all of the other machinery is meant to deal with errors, shortcomings, and the occasional bad egg, but not with systematic active deception from an entire sub-community. To try to extract empirical non-mathematical truth from fundamentally suspect numbers is an exercise in numerology in its most mystical sense.

In order to deal with fake news, Facebook has to engage with other actors that can be trusted to have some sort of epistemological commitment to truth. The problem, of course, is that people interested in pushing specific non-true ideas (e.g. that climate change isn't a thing) have managed, after some systematic work amplifying other people's non-factual epistemological commitments, to paint as suspect the entire social machinery of truth-seeking (e.g. that climatology is a giant worldwide hoax). Facebook can't feed their algorithms with mostly non-adversarial (non-demonic, the Descartes would say) without making an active choice as to what organizations and processes are relatively trustworthy. It can't leave that choice to "society", because for most aspects of the world that matter there's a significantly motivated and funded side of society that has gone off the factual rails.

Can algorithms help? Yes. Can it be done, at the speed and scale required by Facebook's unique position in many societies, without algorithms? No. But algorithms are essentially fast, scalable ways of implementing an epistemological choice (who are you going to believe about what's around Jupiter, philosophy books or your lying eyes?), not magical oracles that make that choice for you.

It's not impossibly hard — it's also what journalists do when they evaluate their sources for both their likelihood of lying and their likelihood of having factual knowledge about facts the journalist wants to report on — but it's a choice that was deemed political at the time that Galileo made it, and it has never ceased to be political since then.

Facebook's position of we're a social network, our job is to help people communicate with each other, not to help them know true things is understandable in the abstract; its concrete moral validity, as it's always the case, depends on context and consequences. History is filled with atrocities made possible because people and organizations didn't make choices framed as political that weren't part of their basic business model, but that should've been part of a more basic moral boundaries. Facebook's history already contains atrocities — not just problematic political developments, but literal mass killings — made possible by their choice not to make one. No algorithmic sophistication will get you out of making that choice, or of your the responsibility for the consequences.

Using Artificial Intelligence to Understand Brands

Artificial Intelligence techniques behind automated translation and other NLP (Natural Language Processing) applications doesn't just work at the level of words and phrases, but give us quantitative data about the meaning people assign to different concepts, including brands. We can then measure relationships between different brands in a space, understanding what distinguishes how people talk about, or rather with, them.

As an example, we'll use data from the GloVe project at Stanford — in particular their Twitter- and Wikipedia-derived sets — to look at the main smartphone brands: Samsung, Apple, Nokia, Sony, LG, HTC, Motorola, and Huawei. It's a dictionary, but a very special one: instead of taking a term in English and mapping it to a Chinese one, it takes terms in English, Chinese, etc, and translates them to points in an abstract space, essentially sets of apparently meaningless numbers. But they are far from arbitrary; the software learns to map terms that are used in similar ways, regardless of how they are written, into points that are close in this abstract space. So bicycle, bicicleta, and 自行车 are translated to points that are close to each other, which is how a system like Google Translate knows how to go from the English term to the Spanish one — it just looks for the closest point that comes from a Spanish term (using neural networks to figure out this map is where the real difficulty lies, but that's for a different post).

We can leverage this into an intuitive but data-driven way of looking at the relationship between multiple brands. Just as words that have similar meanings are closer to each other than those that have different ones, brands that people think of as similar will be talked about in the same way, and so, because they are similar as words, will end up having close points in the abstract space. It sounds a bit... abstract, but here's how it looks for the main smartphone brands, using the mapping based on GloVe's crawling of about two billion tweets:

What this map is telling us is that, based on the way people actually use the brands as words when tweeting — in some senses, the "real" content of the brand — most smartphone brands are pretty much identical, with Apple (as expected), Huawei, and LG the odd ones out. An algorithm that told us that Samsung and Motorola are identical as brands wouldn't be a very perceptive one, and in fact that's not what's happening here. If we zoom into that cluster of brands, we see they are quite separate from each other:

What we're seeing is that, simply put, compared with Apple (and Huawei and LG) Samsung and Motorola are identical as brands; it's just when you zoom into the area of "major Android brands" that you can see that Samsung and Nokia are quite similar — compared to Motorola. It's all relative, but not in an arbitrary way.

So we've used AI to put in perspective the success of Samsung's branding efforts, or, in a more positive way, to highlight how Apple and a couple of other brands are on their own class, each of them separated as words from the pretty much homogeneous (unless you forget the competitors and zoom into them) central core of Android brands. But can we say something about the semantics of that difference?

Surprisingly, we can! Sort of. The biggest surprise of this algorithm — and it caused quite a stir in the AI community when it was first published — is that there's meaning not just in the distance between points, but also in their specific geometric relationships. The best way to explain it is by showing it:

Each of the words king, queen, man, woman, son, daughter gets its own point, as expected. The fascinating thing is that the arrow between king and queen is almost the same in length and direction as the arrow between man and woman, and they are also almost identical to the arrow between son and daughter! Somehow, the algorithm doesn't just learn a way to represent words as points, but also the abstract relationship female version of, which is "translated" into an arrow of specific angle and length; to know what the female version of son is, you just start from that point, do the same jump as you'd do to go from man to woman... and reach the point for daughter.

This is an incredibly powerful capability, because now we can ask not just which brands are comparatively similar or different, but in what way. We took a list of fifty common adjectives in English as our vocabulary, and asked the data

if king is to queen as man is to woman then
the generic Android brand is to Apple as... ? is to ?

The closest metaphors we got?

the generic Android brand is to Apple as black is to white

the generic Android brand is to Apple as international is to national

the generic Android brand is to Apple as special is to great

This doesn't mean people use the word national a lot when talking about Apple. It's subtler and more powerful: the "national-ness" that is the difference between the words international and national is similar to the difference between the way people use the words for the major Android brands, e.g. Samsung, and Apple. The white cases and Apple being an US company (and, arguably, their being great) are part of the difference in meaning between Apple and Samsung.

The most important thing about the above is how unsurprising it is if you pay attention to the brands; it's part of the discourse, yes, but the algorithm automatically teased those differences out and simplified them to the starkest meaningful metaphor. Apple is the white smartphone — you push billions of tweets in one side, process it carefully enough, and out comes a conceptual observation.

Artificial intelligence: it's not just about numbers anymore (and it never was).

Applying the same analysis to compare LG and Huawei against the rest of the Android brands gives us consistently the terms good (for LG) and strong (for Huawei), perhaps an indication of solid if not brilliant conceptual branding (remember, we're quantifying metaphors, not counting mentions — it's not the words in the advertising copy, but the way people use the brand in their own tweets).

But the important aspect of this technique is that we could just as easily have used a completely different vocabulary to query the relationship between brands — feelings instead of adjectives, or terms related to prices, or whatever vocabulary helps answer the specific question we wanted to ask. Brands, like every other word, are deeply multidimensional, and so are their relationships; rather than attempting to oversimplify them through the narrow lens of a specific survey, pulling vast amounts of actual usage data allows us to look at concrete answers to specific questions about the relationship between brands in all the messy complexity of the real world, yet distill that complexity into conceptually usable semantic relationships.

Forget counting retweets and classifying mentions: we now have the tools to look into the living reality of brands as part of our continuously shifting languages, and to apply quantitative methods to elucidate, and eventually shape, their conceptual and emotional overtones. As happened with metrics-driven, quantitatively optimized advertising, leveraging these tools will require expanding the conceptual and strategic toolsets of organizations in ways that to many will feel too alien to attempt, but which will eventually become part of the basic practices of the industry. Marketing, I believe, will become the richer for that, not just through increased transparency and effectiveness, but also by making possible the development of completely new means to achieve the oldest ends.

After all, what has marketing always been, if not the engineering of hidden metaphors?

The normalcy of online learning: the more you study, the better you do

Online learning, after all, is just a form of learning: time spent studying is one of the best predictors of success.

Both the pattern (and the exceptions) can be seen quite clearly on the Open University Learning Analytics dataset, which collects anonymized data about the personal characteristics and, crucially, interactions with the Open University's Virtual Learning Environment (as counts of clicks by date) of 32,593 students registered in 22 courses; see the linked entry on Nature for a detailed description of the data set. For this quick exploratory analysis I chose to focus on students that either passed or failed their courses, ignoring those who withdrew along the way; the latter is a very frequent outcome in this kind of setting (31% of cases in the data set), but one that merits a separate analysis.

Of those students that completed the course, 68.6% passed it (13.5% of them with Distinction), and 31.4% failed. To what degree was this a matter of sheer effort?

Here the data supports what teachers and parents always say. Only about a third of the students who interacted with the learning platform between 10 and 23 days (the second decile of activity) passed the course, while 94% of those who did it between 120 and 155 days (the ninth decile) did. This is perhaps an obvious effect, but it's noteworthy that even among the highest deciles of activity, more activity leads to a better result: moving from the eight to the ninth decile of activity — from, say, 110 days of activity to 140 — raises the probability of passing the test an extra five percent.

There are things we can say about the probability of somebody passing the course before it begins. Most significantly, the probability of passing the course among students who finish it grows strongly with the already achieved educational level of the student (note that this date refers to the UK educational system).

There's nothing mysterious about the mechanics of it. By and large, better-educated students interact more often with the platform, and the extra days explain much of the variability in outcome.

This is the point where reading the data becomes tricky, and domain experience and a healthy dosis of skepticism become useful. There's both a correlation and a reasonable mechanism of influence between studying more days and getting a better outcome, which — as an hypothesis to guide interventions — suggests we should attempt to get students to interact with the platform more often. But understanding why they already don't do it on their own is critical to understanding what would help, and that's not necessarily obvious from this data. For example, one possibility is that students simply underestimate how many days of study they'll need in order to get a reasonable chance of passing the course; if that's the case, then explicit, dynamic guidance on this could be of use (including something like a regular, model-based Estimated Probability of Passing alert).

On the other hand, the data does suggest that more exogenous constraints probably play a role. To their credit, and that's something that every educational system should attempt to replicate, this Open University data set also includes socio-economic information in the form of the student's approximate Index of Multiple Deprivation, an statistical proxy — based on a ranking comparison between places in England — to issues like crime prevalence, unemployment, education, income, etc, of the place where the student lived during the course.

This index is correlated with the outcome of the course, as would be expected (a higher IMD band indicates a more favorable socio-economic context):

But also with, and arguably through, the number of days students interact with the platform:

So there are factors, which could be cultural but might as well be, and we could easily imagine are, related to constraints in resources, time, energy, support networks, etc., of students living in more deprived areas. If or to the degree to which the latter is the cause, "gamification" features like the one described above would at best be useless and at worst a mockery. The point of data-driven analysis is to be able to determine what's going on, in order to guide our intuition about what could help; this data set suggests possibilities, but that's as far as we can get with it.

Of course, that in this post we're playing at reinventing the wheel — poorly. Education experts are deeply familiar with everything we've discussed so far, from the impact of study time on outcomes to the effect of socioeconomic constraints. The point isn't that we have found anything new, but rather to show how already-known things surface very quickly and obviously whenever data is gathered in a sufficiently comprehensive and open way, and the possibilities for personalized diagnostics and scalable assistance that this might offer as a way of assisting and helping educational systems.

On the topic of things already well-known, we've seen that putting in days of interaction with the platform improves students' chances of passing the course, and that better-educated students have a higher a priori chance of doing it. Is the increased time all of it? In other words, do higher educational achievements, besides being correlated with exogenous and endogenous factors related to being able to study more, also enable students to do it better? Do students with different educational backgrounds get different amounts of value from a day of interacting with the system?

The data set only offers indirect clues to this, but as far as we can see, this is true pretty consistently. For each intensity of interaction with the platform, students with a higher level of education will, generally speaking, do better (click on the graph for a larger version):

We can't distinguish with this data, of course, the details of the mechanics of how this happens; "better study habits" can include anything from a larger store of previous knowledge to draw relationships from, to a better physical environment in which to study. The often large correlations between different factors are part of what makes research in social sciences both difficult and important. But we see there's a difference, which means there's also potential for improved outcomes.

Online learning isn't, in many ways, a radical departure from traditional education: we can see how the traditional issues of socio-economic context, educational history, and effort continue to play the roles they always have. However, the increased legibility of the online process, and the enormous flexibility it offers for interventions and experiments, make it not just a powerful teaching mechanism on its own, but also a tool to help us understand and improve learning in general.

Deep(ly) Unsettling: The ubiquitous, unspoken business model of AI-induced mental illness

"The junk merchant," wrote William S. Burroughs, "doesn't sell his product to the consumer, he sells the consumer to his product. He does not improve and simplify his merchandise. He degrades and simplifies the client.” He might as well have been describing the commercial, AI-mediated, social-network-driven internet.

The emotionally and politically toxic effects of the ecosystem of platforms like Facebook and Twitter, together with the organizations leveraging them, might not be their intended goals, but they aren't accidents either. If you configure a data-driven system to learn the best way to induce users to stay on the platform and interact with it and its advertisers, it'll simply do that. It just so happens that the ideally compulsive, engaged user of a game and or a social network, the one every algorithm is continuously trying to train by what content and rewards it offers, isn't the emotionally healthy one.

Maximizing engagement is the explicit optimization goal of contemporary online businesses. They just rediscovered and implemented, quickly and efficiently, the time-honored tools of compulsive gambling, gaslighting, and continuous emotional manipulation. They aren't tools that make the user mentally healthier, quite the opposite, but nobody programmed to algorithms to even measure, much less take into account, this side effect.

And the impacts they've had so far have been achieved with technology that's already conceptually obsolete. Picture the greatest chess player in history, retrained using the knowledge of the day-to-day experience and reactions of billions of people into the world's most effective and least ethical behavioral therapist, fed in real time every scrap of information available about you, constantly interacting with each digital device, service, and information source you are in direct or indirectly contact with, capable of choosing what it's suggested to you to see and do — even of making up whatever text, audio and video it thinks it'll work best — and dedicated exclusively to shaping your emotions and understanding of the world, with no regard at all for your well-being, according to the preferences of whoever or whatever is paying it the most at the moment or is best exploiting its own technological vulnerabilities.

Rephrased in an allegorical way, it could be an updated version of one of Philip K. Dick's Gnostic nightmares. A video designed by a superhumanly capable AI to exploit every one of your emotional weak spots — a murder victim with a face that reminds you of a loved one, a politician's voice slightly remodulated to make it subliminally loathsome, a caption that casually inserts an indirect reference to a personal tragedy at the exact moment of the day where you're most tired and your defenses at their lowest — wouldn't be out of place in one of his stories, but it's also just a few years' away from being technologically feasible, and very explicitly in the industry's R&D roadmap. Change the words used to describe it, changing nothing of what it describes, and it's a pitch Silicon Valley investors hear a dozen times a month.

It'd be absurd to pretend we've always been sane and well-informed. Every form of media carries opportunities for both information and manipulation, for smarter societies and collective insanity. But getting things right is always a challenge. This one is ours, and it might be one of the most difficult we have ever faced. The amount of information and sheer cognitive power bent on manipulating each of us, individually, at any given minute of the day is growing exponentially, and our individual and collective ability to cope with these attempts certainly isn't. Whether and how we react to this will be a subtle but powerful driver of our societies for decades to come.

When devops involves monitoring for excess suicides

There is strong observational evidence of prolonged social network usage being correlated with depression and suicide — enough for companies like Facebook to deploy tools to attempt to predict and preempt possible cases of self-harm. But taken in isolation, these measures are akin to soda companies sponsoring bicycle races. For social networks, massive online games, and other business models predicated on algorithmic engagement maximization, the things that make them potentially dangerous to psychological health — the fostering and maintenance of compulsive behaviors, the systemic exposure to material engineered to be emotionally upsetting — are the very things that make them work as businesses.

Developers, and particularly those involved in advertising algorithms, content engineering, game design, etc, have in this a role ethically similar to that of, say, scientists designing new chemical sweeteners for a food company. It's not enough for a new compound to have an addictive taste and be cheap to produce — it has to be safe, and it's part of the scientist and the company's responsibility to make sure it is. If algorithms can affect human behavior — and we know they do — and if they can do so in deleterious way — and we also know this to be true — then developers have a responsibility to account for this possibility not just as a theoretical concern, but as a metric to monitor as closely as possible.

Software development and monitoring practices are the sharp end of corporate values for technology companies. You can tell what a company really values by noting what will force an automated rollback of new code. For many companies this is some version of "blowing up," for others it's a performance regression, and for the most sophisticated, a negative change in a business metric. But any new deployment of, e.g., Facebook's feed algorithms or content filtering tools has the potential of causing a huge amount of psychological and political distress, or worse. So their deployment tools have to automatically monitor and react to not just the impact of new code on metrics like resource usage, user interface latencies, or revenue per impression, but also the psychological well-being of those users exposed to the newest version of the code.

I don't know whether companies like Facebook treat those metrics as first-order data input to software engineering decisions; perhaps they do, or are beginning to. The ethical argument for doing so is quite clear, and, if nothing else, it should be a natural first step in any goodwill PR campaign.

There are only two emotions in Facebook, and we only use one at a time

We have the possibility of infinite emotional nuance, but Facebook doesn't seem to be the place for it. The data and psychology of how we react emotionally online are fascinating, but the social implications, although not specific to social networks, are rather worrisome.

A good way to explore our emotional reaction to Facebook news is through Patrick Martinchek's data set of four million posts from mainstream media during the period 2012-2016. I focused on news posts during 2016, most (93%) of which had received one or more of the emotional reactions in Facebook's algorithmic vocabulary: angry, love, sad, thankful, wow, and, of course, like.

In theory, an article could evoke any combination of emotions — make some people sad, others thankful, others a bit angry, and yet in others call for a simple "wow" — but it turns out that our collective emotional range is more limited. Applying to the data a method called Principal Component Analysis, we see that we can predict most of the emotional reactions to an article as a combination of two "hidden knobs":

  • There's a knob that increases the frequency of both love and wow reactions. We can just call that knob love.
  • The other knob increases the frequency of wows as well, but also, more significantly, the frequency of angry and sad, both in almost equal measure.

And that's it. Thankfulness, likes, even that feeling of "wow," are distributed pretty much at random through our reaction to news. What makes one article different to another to our eyes (or, more poetically, to our hearts) are something that makes us love them, and something else that makes us, with equal strength or probability, feel angry or sad about them.

Despite their names, it's not logically necessary for the "strength" of love to be low when anger/sadness is high, or vice versa. Remember that they measure the frequency of different emotional responses; it's easy to imagine news that half of its readers will love, yet will make the other half angry or sad.

Remarkably, that's not the case:

The graph shows how many news posts, relatively speaking, show different combinations of strength in the (horizontal) love and (vertical) angry/sad dimensions (click on the graph to expand it). Aside from a small group of posts that have zero strength in either dimension, and another, smaller group of more anomalous posts, most posts lie in a straight line between the poles of love and angry/sad: the stronger the love dimension of a post, the weaker will be its angry/sad dimension, and vice versa.

Different people have different, often opposite reactions to the same event. Why is our emotional reaction to news about them so relatively homogeneous? The answer is likely to be audience segmentation: each news post is seen by a rather homogeneous readership (that media source's target audience), so their reaction to the article will also be homogeneous.

In other words, a possible indicator that people with different preferences and values do read different media (and/or are shown different media posts by Facebook) is that the reactions to each post, either love of its statistical opposite, are statistically more homogeneous than they'd otherwise be. If everybody at a sports game are either cheering or booing at the same time, you can tell only one group of fans is watching it.

It's common, but somewhat disingenuous, to blame the use of recommendation algorithms for this. As soon as there are two TV stations in an area or two newspapers in a city, they have always tended to get each their own audience, and shape themselves to their interests as much as they influence them. The fault, such as it is, lies not in our code, but in ourselves.

Two things make algorithmic online media in general, and social networks in particular, different. First, while resistant to certain classic forms of manipulation and pressure (e.g. censure by phone call to TV network owner, except in places like China, where censorship mechanisms are explicitly built in both technology and regulations) they are vulnerable to new ones (content farms, bots, etc).

Second — and this is at the root of the current political kerfuffle around social networks — they need not be. Algorithmic recommendation is increasingly flexible and powerful; while it's unrealistic to require things like "no extremist content online, ever," the dynamics of what gets recommended and why can and are continuously modified and tweaked. There's a flexibility to how Facebook, Twitter, or Google work and could work that newspapers don't have, simply because networked computers are infinitely more programmable than printing presses and pages of paper.

This puts them in a bind that would deserve sympathy if they weren't among the most valuable and influential companies in the world, and utterly devoid of any sort of instinct for public service until their bottom line is threatened: whatever they do and not do risks backlash, and there's no legal, political, or social agreement as to what they should do. It's straightforward to say that they should censor extremist content and provide balanced information about controversial issues — in a way, we're asking them to fix not bugs in their algorithms, but in our own instincts and habits — but there are profound divisions in many societies about what counts as extremism and what's controversial. To focus on the US, when first-line universities sometimes consider white supremacism a legitimate political position, and government officials in charge of environmental issues consider the current global scientific consensus on climatology a very undecided matter, there's no politically safe algorithmic way to de-bias content... and no politically safe way to just wash your hands off the problem.

Social networks aren't powerful just because of how many people they reach, and how much, fast, and far they can amplify what they say. They are are unprecedentedly powerful because they have an almost infinite flexibility on what they can show to whom, and how, and new capabilities can always unsettle the balance of power. Everywhere, from China to the US to the most remote corners of the developing world, we're in the sometimes violent process of re-calculating how this new balance will look like.

"Algorithms" might be the new factor here, but it's human politics what's really at stake.

Any sufficiently advanced totalitarianism is indistinguishable from Facebook

Gamification doesn't need to be enjoyable to be effective.

You're more likely to cheat on your taxes than to walk barefoot into a bank, even if it's summer and your feet hurt. That's because we don't just care about how bad the consequences of something could be, but also how certain they are to happen, and, illogically but consistently, how soon they will happen.

That's what makes Facebook so addictive. Staying another minute isn't going to make you happy, but it guarantees a small and immediate dose of socially-triggered emotion, and that's an incredibly powerful driver of behavior. The business of Facebook is to know enough about you, and have enough material, to make sure it can keep that subliminal promise while showing you targeted ads.

Governments' tools are noticeably blunter. Most of the laws that are generally respected reflect some sort of pre-existing social agreement. Conversely, where that social agreement doesn't exist (e.g., the legitimacy of buying dollars in Argentina, or the acceptability of misogyny pretty much everywhere), laws can only be enforced sporadically and with delay, and hence are seldom effective.

What the ongoing deployment by totalitarian governments — and the totalitarian arms of not-entirely-totalitarian governments — is making possible is the recreation of Facebook, but one co-founded by Foucault. The granularity, flexibility, and speed of perception and action, once a State is digitized enough, is unfathomable by the standards of any State in history. You can charge a fine, report a behavior to a boss, inconvenience a family member, impact a credit score, or notify a child's school the very moment a frowned-upon action was performed, with (sufficiently) total certainty and visibility. It doesn't have to be a large punishment or a lavish reward, or even the same for everybody: just as Facebook knows what you like, a government good enough at processing the data it has can know what you care about, and calibrate exactly how to use it so even small transgressions and small "socially beneficial activities" will get a small but fast and certain reward. Small but fast and certain is a cheap and effective way of shaping behavior, as long as it's something you do care about, and not generic "points" or "achievements." It can be your children's educational opportunities, your job, your public image, anything — governments, once they develop the right process and software infrastructure, can always find buttons to push.

This kind of detail-oriented totalitarianism only used to be possible in the most insanely paranoid societies (the Stasi being a paradigmatic example) but it escalated very poorly, and with ultimately suicidal economic and social costs.

Doing it with contemporary technology, on the other hand, scales very well, as long as a government is willing to cede control of the "last mile" of carrots and sticks to software. You would be very surprised if you entered Facebook one day and saw something as impersonal and generic, or at best as fake-personalized, as most interactions with the State are now. A government leveraging contemporary technology has a some significant computing power constantly looking at you and thinking about you — what you're doing, what you care about, what you're likely to do next — and instead of different parts of the government keeping their own files and dealing with you on their own time, everything from the cop on your street to your grandparents' pharmacist is integrated into that bit of the State that is exclusively and constantly dedicated to nudging you into being the best citizen you can possibly be.

It won't just be a cost-effective way of social control. Everything we know of psychology, and our recent experience with social networks and other mobile games, suggests it'll be an effective way of shaping our decisions before we even make them.

Big Data, Endless Wars, and Why Gamification (Often) Fails

Militaries and software companies are currently stuck in something of a rut: billions of dollars are spent on the latest technology, including sophisticated and supposedly game-changing data gathering and analysis, and yet for most victory seems a best to be a matter of luck, and at worst perpetually elusive.

As different as those "industries" are, this common failure has a common root; perhaps unsurprisingly so, given the long and complex history of cultural, financial, and technological relationships between them.

Both military action and gamified software (of whatever kind: games, nudge-rich crowdsourcing software, behaviorally intrusive e-commerce shops, etc) are focused on the same thing: changing somebody else's behavior. It's easy to forget, amid the current explosion — pun not intended — of data-driven technologies, that wars are rarely fought until the enemy stops being able to fight back, but rather until they choose not to, and that all the data and smarts behind a game is pointless unless more players do more of what you want them to do. It doesn't matter how big your military stick is, or how sophisticated your gamified carrot algorithm, that's what they exist for.

History, psychology, and personal experience show that carrots and sticks, alone or in combination, do, work. So why do some wars take forever, and some games or apps whimper and die without getting any traction?

The root cause is that, while carrots and sticks work, different people and groups have different concepts of what counts as one. This is partly a matter of cultural and personal differences, and partly a matter of specific situations: as every teacher knows, a gold star only works for children who care about gold stars, and the threat of being sent to detention only deters those for whom it's not an accepted fact of life, if not a badge of honor. Hence the failure of most online reputational systems, the endemic nature of trolls, the hit-and-miss nature of new games not based on an already successful franchise, or, for that matter, the enormous difficulty even major militaries have stopping insurgencies and other similar actors.

But the root problem behind that root problem isn't a feature in the culture and psychology of adversaries and customers (and it's interesting to note that, artillery aside, the technologies applied on both aren't always different), but in the culture and psychology of civilian and military engineers. The fault, so to speak, is not in our five-stars rating systems, but in ourselves.

How so? As obvious as it is that achieving the goals of gamified software and military interventions requires a deep knowledge of the psychology, culture, and political dynamics of targets and/or customer bases, software engineers, product designers, technology CEOs, soldiers, and military strategists don't receive more than token encouragement to develop a strong foundation in those areas, much less are required to do so. Game designers and intelligence analysts, to mention a couple of exceptions, do, but their advice is often given but a half-hearted ear, and, unless they go solo, they lack any sort of authority. Thus we end, by and large, with large and meticulously planned campaigns — of either sort — that fail spectacularly or slowly fizzle out without achieving their goals, not for failures of execution (those are also endemic, but a different issue) but because the link between execution and the end goal was formulated, often implicitly, by people without much training in or inclination for the relevant disciplines.

There's a mythology behind this: they idea that, given enough accumulation of data and analytical power, human behavior can be predicted and simulated, and hence shaped. This might yet be true — the opposite mythology of some ineffable quality of unpredictability in human behavior is, if anything, even less well-supported by facts — but right now we are far from that point, particularly when it comes to very different societies, complex political situations, or customers already under heavy "attack" by competitors. It's not that people can't be understood, and forms of shaping their behavior designed, it's that this takes knowledge that for now lies in the work and brains of people who specialize in studying individual and collective behavior: political analysts, psychologists, anthropologists, and so on.

They are given roles, write briefs, have fun job titles, and sometimes are even paid attention to. The need for their type of expertise is paid lip service to; I'm not describing explicit doctrine, either in the military or in the civilian world, but rather more insidious implicit attitudes (the same attitudes the drive, in an even more ethically, socially, and pragmatically destructive way, sexism and racism in most societies and organizations).

Women and minorities aside (although there's a fair and not accidental degree of overlap), people with a strong professional formation in the humanities are pretty much the people you're least likely to see — honorable and successful exceptions aside — in a C-level position or having authority over military strategy. It's not just that they don't appear there: they are mostly shunned, and implicitly or explicitly, well, let's go with "underappreciated." Both Silicon Valley and the Pentagon, as well as their overseas equivalents, are seen and see themselves at places explicitly away from that sort of "soft" and "vague" thing. Sufficiently advanced carrots and sticks, goes the implicit tale, can replace political understanding and a grasp of psychological nuance.

Sometimes, sure. Not always. Even the most advanced organizations get stuck in quagmires (Google+, anyone?) when they forget that, absent an overwhelming technological advantage, and sometimes even then (Afghanistan, anyone?) successful strategy begins with a correct grasp of politics and psychology, not the other way around, and that we aren't yet at a point where this can be provided solely by data gathering and analysis.

Can that help? Yes. Is an organization that leverages political analysis, anthropology, and psychology together with data analysis and artificial intelligence like to out-think and out-match most competitors regardless of relative size? Again, yes.

Societies and organizations that reject advanced information technology because it's new have, by and large, been left behind, often irreparably so. Societies and organizations that reject humanities because they are traditional (never mind how much they have advanced) risk suffering the same fate.

This screen is an attack surface

A very short note on why human gut feeling isn't just subpar, but positively dangerous.

One of the most active areas of research in machine learning is adversarial machine learning, broadly defined as the study of how to fool and subvert other people's machine learning algorithms for your own goals, and how to prevent it from happening to yours. A key way to do this is through controlling sampling; the point of machine learning, after all, is to have behavior be guided by data, and sometimes the careful poisoning of what an algorithm sees — not the whole of its data, just a set of well-chosen inputs — can make its behavior deviate from what its creators intended.

A very public example of this is the nascent tradition of people collectively turning a public Microsoft demonstration chatbot into a bigot spouting conspiracy theories, by training it with the right conversations, last year with "Tay" and this week with "Zo." Humans are obviously subject to all sorts of analogous attacks through lies, misdirection, indoctrination, etc, and a big part of our socialization consists on learning to counteract (and, let's be honest, to enact) the adversarial use of language. But there's a subtler vector of attack that, because it's not really conscious, is extremely difficult to defend from.

Human minds rely very heavily on what's called the availability heuristic: when trying to figure out what will happen, we tend to give more weight to possibilities we can easily recall and picture. This is a reasonable automatic process in stable environments and first-hand observations, as it's fast and likely to give good predictions. We easily imagine the very frequent and the very dangerous, so our decision-making follows probabilities, with a bias towards avoiding that place where a lion almost ate us five years ago.

However, we don't observe most of our environment first-hand. Most of us, thankfully, have more exposure to violence through fiction than through real experience, always in highly memorable forms (more and better-crafted stories about violent crime than about car accidents), making our intuition misjudge relative probabilities and dangers. The same happens in every other area of our lives: tens of thousands of words about startup billionaires for every phrase about founders who never got a single project to work, Hollywood-style security threats versus much more likely and cumulatively harmful issues, the quick gut decision versus the detached analysis of multiple scenarios.

And there's no way to fix this. Retraining instincts is a difficult and problematic task, even for very specific ones, much less for the myriad different decisions we make in our personal and professional lives. Every form of media aims at memorability and interest over following reality's statistical distribution — people read and watch the new and spectacular, not the thing that keeps happening — so most of the information you've acquired during your life comes from an statistically biased sample. You might have a highly accurate gut feeling for a very specific area where you've deliberately accumulated an statistically strong data set and interacted with it in an intensive way, in other words, where you've developed expertise, but for most decisions we make in our highly heterogeneous professional and personal activities, our gut feelings have already been irreversibly compromised into at best suboptimal and at worst extensively damaging patterns.

It's a rather disheartening realization, and one that goes against the often raised defense of intuition as one area where humans outperform machines. We very much don't, not because our algorithms are worse (although that's sometimes also true) but because training a machine learning algorithm allows you to carefully select the input data and compensate for any bias in it. To get an equivalently well-trained human you'd have to begin when they are very young, put them on a diet of statistically unbiased and well-structured domain information, and train them intensively. That's how we get mathematicians, ballet dancers, and other human experts, but it's very slow and expensive, and outright impossible for poorly defined areas — think management and strategy — or ones where the underlying dynamics change often and drastically — again, think management and strategy.

So in the race to improve our decision-making, which over time is one of the main factors influencing our ultimate success, there's really no way around substituting human gut feeling with algorithms. The stronger you feel about a choice, the more likely it is to be driven by how easy it is to picture, and that's going to have more to do with the interesting and spectacular things you read, watched, and remember than with the boring or unexpected things that do happen.

Psychologically speaking, those are the most difficult and scariest decisions to delegate. Which is why there's still, and might still be for some time, a window of opportunity to gain competitive advantage by doing it.

But hurry. Sooner or later everybody will have heard about it.

The Mental Health of Smart Cities

Not the mental health of the people living in smart cities, but that of the cities themselves. Why not? We are building smart cities to be able to sense, think, and act; their perceptions, thoughts, and actions won't be remotely human, or even biological, but that doesn't make them any less real.

Cities can monitor themselves with an unprecedented level of coverage and detail, from cameras to government records to the wireless information flow permeating the air. But these perceptions will be very weakly integrated, as information flows slowly, if at all, between organizational units and social groups. Will the air quality sensors in a hospital be able to convince most traffic to be rerouted further away until rush hour passes? Will the city be able to cross-reference crime and health records with the distribution of different business, and offer tax credits to, say, grocery stores opening in a place that needs them? When a camera sees you having trouble, will the city know who you are, what's happening to you, and who it should call?

This isn't a technological limitation. It comes from the way our institutions and business are set up, which is in turn reflected in our processes and infrastructure. The only exception in most parts of the world is security, particularly against terrorists and other rare but high-profile crimes. Organizations like the NSA or the Department of Homeland Security (and its myriad partly overlapping versions both within and outside the United States) cross through institutional barriers, most legal regulations, and even the distinction between the public and the private in a way that nothing else does.

The city has multiple fields of partial awareness, but they are only integrated when it comes to perceiving threats. Extrapolating an overused psychological term, isn't this an heuristic definition of paranoia? The part of the city's mind that deals with traffic and the part that deals with health will speak with each other slowly and seldom, the part who manages taxes with the one who sees the world through the electrical grid. But when scared, and the city is scared very often, and close to being scared every day, all of its senses and muscles will snap together in fear. Every scrap of information correlated in central databases, every camera and sensor searching for suspects, all services following a single coordinated plan.

For comparison, shopping malls are built to distract and cocoon us, to put us in the perfect mood to buy. So smart shopping malls see us like customers: they track where we are, where we're going, what we looked at, what we bought. They try to redirect us to places where we'll spend more money, ideally away from the doors. It's a feeling you can notice even in the most primitive "dumb" mall: the very shape of the space is built as a machine to do this. Computers and sensors only heighten this awareness; not your awareness of the space, but the space's awareness of you.

We're building our smart cities in a different direction. We're making them see us as elements needing to get from point A to point B as quickly as possible, taking little or no care of what's going on at either end... except when it sees us, and it never sees or thinks as clearly and as fast, as potential threats. Much of the mind of the city takes the form of mobile services from large global companies that seldom interact locally with each other, much less with the civic fabric itself. Everything only snaps together with an alert is raised and, for the first time, we see what the city can do when it wakes up and its sensors and algorithms, its departments and infrastructure, are at least attempting to work coordinately toward a single end.

The city as a whole has no separate concept of what a person is, no way of tracing you through its perceptions and memories of your movements, actions, and context except when you're a threat. As a whole, it knows of "persons of interest" and "active situations." It doesn't know about health, quality of life, a sudden change in a neighborhood. It doesn't know itself as anything else than a target.

It doesn't need to be like that. The psychology of a smart city, how it integrates its multiple perceptions, what it can think about, how it chooses what to do and why, all of that is up to us. A smart city is just an incredibly complex machine we live in and whom we give life to. We could build it to have a sense of itself and of its inhabitants, to perceive needs and be constantly trying to help. A city whose mind, vaguely and perhaps unconsciously intuited behind its ubiquitous and thus invisible cameras, we find comforting. A sane mind.

Right now we're building cities that see the world mostly in terms of cars and terrorism threats. A mind that sees everything and puts together very little except when it scares it, where personal emergencies are almost entirely your own affair, but becomes single-minded when there's a hunt.

That's not a sane mind, and we're planning to live in a physical environment controlled by it.

When the world is the ad

Data-driven algorithms are effective not because of what they know, but as a function of what they don't. From a mathematical point of view, Internet advertising isn't about putting ads on pages or crafting seemingly neutral content. There's just the input — some change to the world you pay somebody or something to make — and the output — a change in somebody's likelihood of purchasing a given product or voting for somebody. The concept of multitouch attribution, the attempt to understand how multiple contacts with different ads influenced some action, is a step in the right direction, but it's still driven by a cosmology that sees ads as little gems of influence embedded in a larger universe that you can't change.

That's no longer true. The Internet isn't primarily a medium in the sense of something that is between. It's a medium in that we live inside it. It's the atmosphere through which the sound waves of information, feelings, and money flow. It's the spacetime through which the gravity waves from some piece of code shifting from data center to data center according to some post-geographical search of efficiency reach your car to suggest a route. And, on the opposite direction, it's how physical measurements of your location, activities — even physiological state — are captured, shared, and reused in ways that are increasingly more difficult to know about, and much less to be aware of during our daily life. Transparency of action often equals, and is used to achieve, opacity to oversight.

Everything we experience impacts our behavior, and each day more of what we experience is controlled, optimized, configured, personalized — pick your word — by companies desperately looking for a business model or methodically searching for their next billion dollars or ten.

Consider as a harbinger of the future that most traditional of companies, Facebook, a space so embedded in our culture that people older than credit cards (1950, Diners) use it without wonder. Among the constant experimentation with the willingly shared content of our lives that is the company, they ran an experiment attempting to deliberately influence the mood of their users by changing the order of what they read. The ethics of that experiment are important to discuss now and irrelevant to what will happen next, because the business implications are too obvious not to be exploited: some products and services are acquired preferentially by people in a certain mood, and it might be easier to change the mood of an already promising or tested customer than to find another new one.

If nostalgia makes you buy music, why wait until you feel nostalgic to show you an ad, when I can make sure you encounter mentions of places and activities from your childhood? A weapons company (or a law-and-order political candidate) will pay to place their ad next to a crime story, but if they pay more they can also make sure the articles you read before that, just their titles as you scroll down, are also scary ones, regardless of topic. Scary, that is, specifically for you. And knowledge can work just as well, and just as subtly: tracking everything you read, and adapting the text here and there, seemingly separate sources of information will give you "A" and "B," close enough for you to remember them when a third one offers to sell you "C." It's not a new trick, but with ubiquitous transparent personalization and a pervasive infrastructure allowing companies to bid for the right to change pretty much all you read and see, it will be even more effective.

It won't be (just) ads, and it won't be (just) content marketing. The main business model of the consumer-facing internet is to change what they consume, and when it comes down to what can and will be leveraged to do it, the answer is of course all of it.

Along the way, advertising will once again drag into widespread commercial application, as well as public awareness, areas of mathematics and technology currently used in more specialized areas. Advertisers mostly see us — because their data systems have been built to see us — as black boxes with tagged attributes (age, searches, location). Collect enough black boxes and enough attributes, and blind machine learning can find a lot of patterns. What they have barely begun to do is to open up those black boxes to model the underlying process, the illogical logic by which we process our social and physical environment so we can figure out what to do, where to go, what to buy. Complete understanding is something best left to lovers and mystics, but every qualitative change in our scalable, algorithmic understanding of human behavior under complex patterns of stimuli will be worth billions in the next iteration of this arms race.

Business practices will change as well, if only as a deepening of current tendencies. Where advertisers now bid for space on a page or a video slot, they will be bidding for the reader-specific emotional resonance of an article somebody just clicked on, the presence of a given item in a background picture, or the location and value of an item in an Augmented Reality game ("how much to put a difficult-to-catch Pokemon just next to my Starbucks for this person, whom I know has been out in this cold day enough for me to believe it'd like a hot beverage?"). Everything that's controlled by software can be bid upon by other software for a third party's commercial purposes. Not much isn't, and very little won't be.

The cumulative logic of technological development, one in which printed flyers co-exist with personalized online ads, promises the survival of what we might call by then overt algorithmic advertising. It won't be a world with no ads, but one in which a lot of what you perceive is tweaked and optimized so it's collective effect, whether perceived or not, is intended to work as one.

We can hypothesize a subliminally but significantly more coherent phenomenological experience of the world — our cities, friendships, jobs, art — a more encompassing and dynamic version of the "opinion bubbles" social networks often build (in their defense, only magnifying algorithmically the bubbles we had already built with our own choices of friends and activities). On the other hand, happy people aren't always the best customers, so transforming the world into a subliminal marketing platform might end up not being very pleasant, even before considering the impact on our societies of leveraging this kind of ubiquitous, personalized, largely subliminal button-pushing for political purposes.

In any case, it's a race in and for the background, and once that already started.