by Samar Farage
Long before ChatGPT, Ivan Illich issued a warning:
[I] foresee that machines which ape people are tending to encroach on every aspect of people’s lives, and that such machines force people to behave like machines. The new electronic devices do indeed have the power to force people to ‘communicate’ with them and each other on the terms of the machine. Whatever structurally does not fit the logic of machines is effectively filtered from a culture dominated by their use.The machine-like behavior of people chained to electronics constitutes a degradation of their well-being and of their dignity, which, for most people in the long run, becomes intolerable. Observations on the sickening effects of programmed environments show that people in them become indolent, impotent, narcissistic, and apolitical. The political process breaks down because people cease to be able to govern themselves; they demand to be managed.((Ivan Illich, “Silence is a Commons,” in *In the Mirror of the Past*, (London: Marion Boyars, 1992), p.47.))
Illich pronounced these words in 1982 at the Asahi Symposium: Science and Man—the computer-managed society. Every line of this excerpt can bear pages of commentary and discussion, but here I will focus on smartphones to show how we are chained to electronics, how we engage in machine-like behavior, and have slouched from the age of tools into the age of systems. In his later years, Illich said he had intuited the end of the tool or instrument as such. He felt Western culture was crossing over a watershed into a new historical era that he named the Age of Systems. I argue that the smartphone exemplifies the portal through which people are seduced by design, into degrading and undignified machine-like behavior. I do not want to speculate on how the people will cast off their attachment to this intolerable condition. Instead, I limit my attention to show that the smartphone cannot be understood as an instrument and to therefore substantiate Illich’s prescient claim that we are living through a historical discontinuity.
I am both mesmerized and repulsed by the new addition to most people’s hands. A plastic outgrowth now seems permanently attached to our palms: its size and sophistication varying only by the wealth and status of the hand to which it is connected. It is an object that most people can neither put down nor put away. The contemporary philosopher Byung Chul Han has called it “a mobile confessional box” and “a subjugation device”. “It acts like a rosary and its beads; this is why we keep a smartphone constantly at hand. The ‘like’ is a digital amen. We keep going to confession. We undress by choice. But we don’t ask for forgiveness, instead we call for attention.”1)
As a devotional object, it induces liturgical gestures–the prayerful bent head over the glowing screen, the scrolling thumb as if working over a rosary or misbaha, and the periodic exchange of access to include another into one’s sectarian fold. It not only defines the rhythm and structure of social relations but also shapes the sense of self and identity. It is not accidental that the most popular smartphone is named the I-phone. As one student said when I tried to take away his smartphone: “you cannot do that, it is my whole life.”
It is well-known that it was the iPhone, announced in 2007, that began the world-wide contagion of the smartphone. More than half the world’s population now uses smartphones and are active on some social media. Teens spend more than half their waking hours glued to their phones while children under the age of 12 in the US spend roughly a quarter of their day watching screens.2) Smartphones combine formerly distinct devices on one platform: the internet-connected computer, the mobile telephone, the video machine, the camera, and the music player are all integrated on one interface. Additionally, a host of specialized apps attend to almost every need and desire—social media to keep up with family, friends, and the news; weather and navigation apps to help plan for the summer holiday or weekend trek; apps to access podcasts, movies, and songs that entertain; even apps that function as so-called personal assistants to help with schedules, appointments, and emails. The smartphone accompanies us everywhere—our location is tracked, our steps counted, our purchases recorded, even our heartbeats and sleep cycles monitored. It anticipates our preferences and acts on them. It speaks, cajoles, and advises us. It has all the answers in one package. It was once called an intimate machine, and never has there been a more intimate machine!
Illich identified a fundamental difference between the instrument and a system. He argued that the instrument presupposed a distality or separation between the device and the user which a system collapsed. Tools, he said, “mean something that incorporates, materializes or formalizes a human intention and can be picked up or not picked up by the person who wants to pursue the goal that corresponds to his intention.”3) Tools are designed by the tool maker for specific uses. The axe is designed to cut but it can be used for chopping down a tree or killing a man. The purpose of its user depends on the intention of the user. In this sense, the tool or instrument materially confronts the user who conceives of herself as standing apart from and distinct from it. The instrument requires or calls forth the intention of its user.
In contrast, according to Illich, a system’s immateriality integrates the user into itself. It permits no conceptual or felt difference between a device and the user–the first is understood as a part, extension, or continuation of the second. Systems are designed by presuming human actions and feelings can be redescribed as feedback. In systems theory, a person is reconstructed as a manageable and self-managing element encased within a larger system. From the viewpoint of the system analyst, the bodily self-perception of a person which permits the distality between and instrument and the user is an illusion. Instead, says Illich, for the system designer, “a man walking a dog is a man-dog-system — a cyborg, as one would say today.” Accordingly, self-perception, he continues, is the result of what is designed by “the system analyst who imputes to the [person] what he or she is…”4)
Insofar as the users are seduced into taking the viewpoint of the system analyst as their own, they integrate themselves into the system. Unlike tools, systems can neither be put down nor rejected. Unlike tools, systems do not transmit the intention of the purported user but enact the commands of the programmer. Most of humanity is now hardwired into the system, wirelessly! The systemic dimension of the smartphone is not only that it integrates humans as nodes in an ever-growing network, but also that self-perception is shaped by the system analyst. It is in these ways that systems become all-encompassing and integrate us into a man-machine environment.
The voluntary integration into networked systems was first dreamed up before it was designed. The loss of distality through computer networks today relies heavily on three conceptual shifts. First, Gerd Gigerenzer has traced the history of the metaphor of the mind as computer to the late 17th century during which so-called Enlightenment thinkers distinguished both reason and moral sentiment from calculation but only to reduce the former to the latter. From then on, it became the enduring part of the modern scientific drive to shrink ideas and sensations to the combinations and permutations of numbers. By the 19th century, calculation became the dull, repetitive, monotonous work that could be performed by machines, as suggested by Babbage’s Difference machine.
By the second half of the 20th century, the parallel notion of the computer as mind became widely accepted, signaling the second conceptual shift behind the design of networked systems. In the 1930s, Alan Turing’s work on the algorithm led him to suggest that the human mind was in some sense a computer. This conceit was only deepened by John von Neumann in 1943 when he explicitly stated that the neuron was the fundamental element of the brain and that its nervous pulses could be understood as a digital 0/1 code. Accordingly, the entire nervous system, he proposed, had a digital character.5) By the 1970s, Joseph Weizenbaum discovered his secretary treated his program DOCTOR that mimicked a Rogerian psychologist as if it were a human being. Until then, Weizenbaum had not realized “that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.”6) Yet, this replacement of human judgment by mindless calculation would be precisely what animated the third conceptual shift behind the fabrication of man-machine systems.
Though the dream of a man-machine symbiosis is old, the integration of the human and the computer takes concrete shape with JCR Licklider who was nicknamed the Johnny Appleseed of modern computing. His paper titled “Man-Computer symbiosis” in 1960 foresaw “the very close coupling between the human and the electronic members of the partnership.”7) Symbiosis refers to a “cooperative interaction between men and machines. Man-computer symbiosis is a subclass of man machine systems…” Licklider made a clear distinction between instruments that mechanically extended man’s abilities and artificial intelligence. The aim of the latter, he said, is not to mechanically extend one’s capacities but to replace one’s intelligence and thought. Soon after, Doug Engelbart, who invented the mouse, opened his Augmentation Research Lab in Stanford to develop “principles and techniques for an augmentation system” that would enhance the human intellect through interactive multi-console computer displays. It took two decades to domesticate the public to these new computers, which was mostly achieved by anthropomorphizing them. Computers were given human features and machine-like characteristics were imputed to humans. It is telling that in 1984, the Man of the Year on the cover of TIME magazine was the Mac computer, and in its advertisement to the world, the computer itself introduced its inventor, Steve Jobs, as “a father to me.”
The human-computer symbiosis began to be realized because computers, unlike other instruments, could be designed to be interactive. In the 1950s and 1960s, computers were already conceived as partners, tutors, or replacements. By the 1970s and early 1980s, the new field of Human- Computer interactions (HCI) emerged to study such interactions. The earlier applications of HCI included hypertext research, text editing, spreadsheets, computer aided design and video games. Over the next three decades, psychology departments were enlisted and disciplines such as behavioral economics and recently neuroscience began to shape the intricate features of such interactions. The world wide web of the 1990s clearly demonstrated the interactive capacities of computer systems that would culminate, by 2022, in the artificially intelligent systems of ChatGPT.
However, by the early 1990s, the notion of the man-computer symbiosis was incrementally naturalized by Clifford Nass, who promoted the notion of computers as social actors based on experimental evidence of HCI research. He characterized the computer as a social actor, implicitly consigning the human part of the equation to an engineerable element. His student, BJ Fogg took his idea one step further by insisting that computers could be “persuasive social actors.” According to Fogg, the most likely explanation for people responding to computers as though they were human was that humans were “hardwired to respond to cues in the environment.” He thought that “social cues from computing products could trigger automatic responses from humans that were “instinctive rather than rational”, that were “automatic and natural.”
BJ Fogg founded the Persuasive Technologies Lab at Stanford in 1997 for such purposes. The lab remains at the heart of the tech industries’ design and BJ Fogg’s students have included inventors of Instagram, Snapchat, Facebook among others. His work was a quantum jump from earlier instances of using the interactivity of computers to persuade users to do things. In the 1970s and 1980s, a few systems were designed to promote health and increase workplace productivity. One of the earliest was the Body Awareness Resource Network (BARN), designed to teach adolescents about health issues such as smoking, drugs, exercise, and more. Gradually other interactive programs of this nature followed, most designed to address adolescent health issues or to treat psychological disorders. But it was only with the Stanford Persuasive Lab that a host of concerted efforts were deployed to design computers as persuasive social actors “to change what we think and do.”8)
At Stanford, BJ Fogg was not interested in computers themselves but in how such interactive machines could influence human behavior. His aim was to study and improve the man-computer symbiosis by making the interfaces as interactive as possible, which is to say, to design feedback loops into devices by which to integrate them with humans. The basic template for such feedback loops used by most persuasive engineers exploits one or more of the five kinds of cues that trigger human behavior as described by B.J Fogg. These include physical cues (movement of eyes, face, body); psychological cues (preferences, humor, feelings); language; social cues (taking turns, praise, answering questions) and finally, enacting social roles (authority, experts, clients).
To this end, BJ Fogg invented a field of study he called Captology (an acronym for Computers as Persuasive technologies). Captology was the study of “all methods and techniques, apps, designs used to enthrall us, seduce for the purpose of changing behaviors and attitudes without coercion or deception.” Believing that technology could rival language in its persuasive powers, he wanted to design technologies that “allows you to trigger behaviors in a new way, motivate them a new way and then simplify, automate and scale up.” The goal was to make digital products irresistible by capturing our attention and increasing users’ engagement adding a sense of willingness and voluntarism to such manipulations.
Fogg had a simplistic algorithm to capture human behavior, one which has been adapted and adopted in many tech designs. He stipulated that B=MAT, or Behavior equals the sum of motivation, ability, and the trigger. For him, humans were nodes motivated by the desire or need for something. They seek positive sensations and avoid negative ones. Motivation also includes the element of anticipation whereby expectations, hope and fears about the future are brought into the present. Sociability and a sense of belonging are elements of the variable called motivation. Ability, in contrast, refers to the ease of use or convenience by which the user is captured to remain attached to the machine. User convenience is measured according to six categories in Fogg’s model: time, money, mental effort, physical effort, social deviance, and non-routine acts. Promoting the ease of use requires designing “user friendly interfaces.” Finally, the triggers or prompts are the enticements that keep the human part of the interface coming back for more. They are the reminders or commands that spark, facilitate, or signal the designed behavior depending on the motivation of the user. A forgetful or inactive user might need a “spark” whereas a highly motivated user could use a “signal”. Triggers engineer “tiny habits” which are intended to determine “what you think and do.”
Nir Eyal was one of the best known students of BJ Fogg. He is a behavioral engineer and consumer psychology professor at Stanford who popularized the telling word hooks to indicate what were once understood as the system feedback loops.9) Explicitly designed to build habit forming products, Eyal proposed a four-phase model to inculcate specific behavior patterns in humans. As he said, “The Hooked Model is a way of describing a user’s interactions with a product as they pass through four phases: a trigger to begin using the product, an action to satisfy the trigger, a variable reward for the action, and some type of investment that, ultimately, makes the product more valuable to the user. As the user goes through these phases, he builds habits in the process.” The hooks in Eyal’s model creates persistent routines or behavioral loops by inscribing the neural network with new pathways. Both teacher and student thereby intended to engineer humans with automated behavior.
Such persuasive technologies or habit-forming apps derive from Skinnerian behaviorist psychology even when they appear in academic fields seemingly distant from psychology. For example, Skinner’s operant conditioning which uses “intermittent reinforcement” not only appears in the “variable reward schedule” of Fogg and Eyal, but also in the vastly more popular “nudges” of behavioral economics.10) Nudges are easy, low-cost interventions that are designed to alter people’s decision making. They are easily applied and spread through mobile devices and online communications that speed the adoption of new designed behaviors. Designers of Digital nudges are called “choice architects” because they design the environment in which the user must behave. Digital nudges include a variety of techniques ranging from simply changing a background color or photo to constructing decoy effects, middle option bias, scarcity effects, and so on.
Similarly, lessons from the gambling industry have proven useful to the design of interactive computer interfaces that determine behavior. Social media users are often modeled on gamblers in a casino not only because of their obsessive behavioral patterns but also because of the tricks and techniques used to keep the gambler playing. Casinos block out daylight and offer continuous refreshments and meals to create a trance like flow state that maximizes the time spent in them. Similarly, the smartphone is designed to enhance the experience of “dropping out of time”, of creating a feeling of being in the “machine zone” that maximizes time on device.”11) One can think here of the time experience on the smartphone–the many hours spent as if in a daze while going down one rabbit hole after another. Is it any wonder that another student of BJ Fogg –Tristan Harris– called the smartphone “the slot machine in your pocket”?
Programmed Environments, Managed Behaviors
It was also Tristan Harris, a Stanford computer scientist and Google ethicist who in 2013 blew the whistle on persuasive technologies. A couple of years later, he was the first of many inventors of persuasive technologies to apologize for the nefarious effects of their creations that “hijacked our attention and brains.” Through the Center for Human Technology which he co-founded with Aza Raskin and the film The Social Dilemma, he documented how our behaviors are being programmed by algorithms and invisible engineers behind the scenes. Tristan Harris knew these tricks well also because he himself was a magician who gained the confidence of the crowd by playing on human emotions and weaknesses. Harris explained how social media operated by manipulating human emotions whether social anxiety, the need for love, admiration and recognition, jealousy, or envy. These emotions are used as punishments and rewards to hook you to your device. These systems are designed to connect with the brain and its “reptilian” pleasure centers: a well-designed social media system releases timed dopamine squirts, as do drugs such as cocaine and chocolate, to keep us coming back for more. As Harris said in his testimony to Congress: “…in an attention economy, there is only so much attention and the business model always wants more. So, it becomes a race to the bottom of the brain stem [and] as attention becomes more competitive, we must crawl deeper down the brain stem to your identity and get you addicted to get attention from other people.”
Sean Parker, an early president of Facebook, confessed that the designs of interactive apps deliberately “exploit human vulnerabilities.” Social media were offered as a cure for the loss of communities and loneliness and infinite entertainment was offered as a solution to drudgery and boredom. But as Illich understood half a century ago, such technologies only escalate what they pretend to eliminate. Examples of the design features of interactive apps that keep humans glued to their machines include:
1.The touch screen, the smooth swiping and the infinite scroll are designed to ensure that users stay online as long as possible. Ezra Raskin who invented the infinite scroll and is now a crusader at the Center for Humane Technology, said that “seamless experience was deliberately engineered to take away any stopping cues and breaks so that users do not realize when to stop.”
2.The ubiquitous “like button” makes us gluttons for social approval and social validation. We measure our worth and our popularity by numbers of likes. Likes are also given as “intermittent rewards” known to work better in enticing people to return for more. For Nir Eyal, the dopamine hits from such rewards keep us searching for more by inducing a semi-stressful response we call “desire”. It is for this reason that he insists he is in the business of “manufacturing desires.”
3.Others habit-forming features include: “Push notifications” that intrude and interrupt when your attention is distracted; “Auto play” on movie streaming platforms to prompt our continuous engagement–this is why we end up consuming two or three hours of mindless content on Netflix having started with a short three-minute video; “streaks” which reward us for staying on a game or learning language using sounds, colors, and icons that make us feel victorious; “Pull to refresh” button which updates content continuously in the style of a slot machine, giving the illusion of control and “the allure of unpredictable rewards.
4.Customization and personalization, such as Spotify’s slogan “soundtrack your life with Spotify”; You tube’s “broadcast yourself”; or Tik Tok’s “for your page” makes the user’s experience appear more intimate. The system now integrates us by appearing to be made just for us. It can tailor ads, newsfeeds and recommendations that stroke our egos and blur the distance between us and the device. So, designers mostly use three criteria– speed, ease of use, and emotional manipulation –to keep users “more invested” and “engaged.” People feel flattered and special because the machine now knows them and caters to their interests. If you watch cat videos, you will get more cat videos on your search screen. If you “like” stories of local foods or global wars, your news feed will adapt accordingly.
5.Gamification according to which apps use game designs that make them “fun.” We voluntarily now submit ourselves to these apps because of the multiple pings, rewards, dancing images that compliment us at every step. These features are part of a larger gamification of everything with points, rewards, and a leaderboard. For instance, the language app Duolingo and the exercise app Fitbit use all these features to keep users attached to their devices.
The principal reason for the development and spread of these interactive features is that people spend as much time on their devices as possible so that eyeballs, clicks, swipes could be harvested as data sold to advertisers. The business model of the giant tech companies is to give their product for free in exchange for ad revenues from other giant companies. As is widely derided, when the product is free, you are the product.
Is Addiction another name for System integration?
In Tools for Conviviality, Illich remarked that “in a consumer society there are inevitably two kinds of slaves: the prisoners of addiction and the prisoners of envy.”12)The smartphone with its interactive apps is designed to addict. In fact, the addictive features of computer networks were already known by 1995 when Ivan Goldberg made up a fictitious sickness he called “Internet addiction disorder.” During the same year, a young doctorate student, Kimberley Young, started her studies on “the addictive use of the internet as a pathological condition” and Mark Griffith wrote a path breaking paper titled “Technological Addictions.” Today there are thousands of scholarly articles on the internet and related addictions.13) In this respect, we are all now prisoners of addiction. But equally, we are prisoners of envy because social media are designed to fuel constant comparison with strangers.
However, is the word addiction appropriate to describe our inextricable attachment to the smartphone? Is addiction sufficient to describe how humans were swept right into the system, how we became enmeshed in the man-computer symbiosis?
Etymologically, the word “addict” derives from the Latin addictus, past particle of addicere meaning, “to deliver, to give one’s assent to” which combines ad “to” and dicere “to say, declare.” Figuratively, to be addicted meant “to devote, consecrate, or abandon oneself to” and in this sense from the 16th century meant “to devote or give oneself to a habit or occupation.” Interestingly, this sense is a softer form of the earlier meaning of addictio. As technical term in Roman law between the 5th and 3rd centuries BCE, addictio named the judicial proclamation by which the judge gave up a debtor to his creditor, the pronouncement by which the debtor was made the prisoner or slave of his creditor.14) Not a few of these debt slaves had incurred gambling debts. Therefore, the association between gambling, enslavement, and addiction has a long history. The contemporary meaning of addiction as a compulsive habit medicalizes and mutes the hard edge of addiction as a condition of enslavement.
The word “user” itself tells a story of addiction. “Users” also name those addicted to heroin, opioids, and other substances. Drug addicts come back for the dopamine surge of the next hit. And like gamblers, those seeking the pleasure from their social media clicks are driven by a behavioral addiction, defined as “any behavior that a person craves, finds temporary relief or pleasure in but suffers negative consequences as a result of, and yet has difficulty giving up.” The social media addict is similarly called a user. Each time we download an app, we sign user agreements which we never read, we are described as end-users, we flock towards user-friendly programs, and we also have usernames.
However, the word user also has the older sense of autonomy as in “tool user.” But the smartphone is not a tool, as I have just argued. In the sense of Ellul, smartphones are the ultimate “technique” that entrenches us completely into the technological system. In the preface to The Technological Society, Jacques Ellul described technique as” the totality of methods, rationally arrived at and having absolute efficiency (for a given stage of development) in every field of human activity.” The adoption of technique has prevailed not only in our digital and technical devices but infiltrated all aspects of human life. The technological system promises productivism, efficiency and maximizing outputs with minimum inputs. Feedback loops or hooks are necessary to ensure the best functioning of such a system. Not only do we “love our cell phones”, but we constantly justify this integration in the name of convenience, ease, speed, productivity, the elimination of boredom, and so on. We continue to pretend that systems are like tools that serve us. Therefore, we ask only what they can do for us, not what they do to us or say about us.
In truth, as Jean Robert pointed out, systems are pseudo tools. Under the pretense of being tools, a whole reengineering of humanity is underway whereby behavior, feelings, and thoughts are now designed, directed, and integrated into the cybernetic system. As Nir Eyal said about the networked systems that are our smartphones, they are built specifically to persuade people “to do what we want them to do. We call these people users and even if we don’t say it aloud, we secretly wish everyone of them would become fiendishly addicted…”
When addicted, we become enslaved subunits in systems because there is no distality between us and the system that integrates us with its dings of pleasure and seductively accessible portals. The smooth screens are portals that wire us to virtual networks and clouds of data beyond the felt world. We are being designed to interface with avatars, profiles, and ghostly connections rather than allowed to live with and among each other. Systemic pseudo-tools cannot convey or carry our intentions but only encode the commands and instructions of their programmers.
By design, systems permit neither autonomous action nor freedom. B.F. Skinner, the father of behaviorism and the inspiration for persuasive technologies stated the matter simply: “it is in the nature of experimental analysis of human behavior that it should strip away the functions assigned to autonomous man and transfer them one by one to the controlling environment.”15)
I believe one should draw a clear distinction between autonomous action and designed behavior. A behavior is an engineered habit that is built on compelled obedience to triggers and prompts. Autonomous action is thoughtful, rooted in the senses, and presumes the possibility of resistance. It allows intention and uncertainty. It bespeaks the contingency of the world. The solidity of the world that we encountered through our bodies and senses was our haven from the informational hell we inhabit today. An addict craving continuous stimulation is not free. To understand the deepest consequence of our integration into systems, let us hear from one of those who addicted us to them. The inventor of virtual reality (VR), Jared Lanier, has put it starkly. Our Freedom itself is at stake, he says, because “addiction turns you into a zombie. And Zombies don’t have free will.” 16)