designing design

I adapted the title of this talk from a book by Japanese designer Kenya Hara that is now almost 20 years old: Designing Design.

Within the book Hara asked various designers to redesign objects so familiar they had stopped being visible — toilet paper, a tea bag, matches, a diaper.

His approach was simple yet striking. Take something we think we know and make it unknown.
His premise was that the most designed things are the ones we can no longer see.

In my opinion, the HCI community can learn from Hara. Interfaces are the most designed and least visible objects in existence. You have all done a great job of turning the strange, complex concept of computation into something we use every day without thinking about it.

How individuals find love has completely shifted into an interface operation within the past 30 years.

The way we consume news and with that how we know about the world around us has changed.

New forms of media have emerged, introducing novel mechanisms of authorship and consumption.
There are hundreds of charts hinting at the vast societal shift that has occurred since the 1990s.

I don't consider myself a human-computer interaction researcher. However, I work, teach, and conduct research in a field closely related to HCI: information design.
For the past decade and a half, I have been working on making data visible, understandable, and tangible.

My work ranges from scientific and investigative visualizations. Such as this publication in the Journal Nature this spring.

To exhibitions and installations. Here a project exhibited at the Berlin Art Week 2025 on turning weather data into tattoos.

All the way to Theater Performances.

My students at Folkwang University of the Arts explore data and information to investigate a wide range of topics:

• From AI Slop,
• Femicides,
• scientific visualizations
• Investigative research
• to large tree diagrams of ambivalent sexism.

The visual representation of data has a long history.
One well-known example is Dr. John Snow's cholera map from 1854.

Today, these tools have become an important medium for representing scientific and journalistic knowledge. One example is Ed Hawkins's "Warming Stripes," which visualizes temperature change from 1850 through 2025.

Visualization and interfaces work together to track global incidents. Take, for example, the Johns Hopkins Coronavirus dashboard from 2020.

All these examples follow a similar pattern and goal:
1. They use data and information.
2. Perform a computation, by hand or through digital methods.
3. To arrive at a humanly perceivable form.
4. This form point towards the world, a reality out there.
5. That reality is tied to the data it represents.

Over the past decade, I have researched ways to break with this pattern by using visualizations to point not primarily toward the world, but toward the systems, settings, and infrastructures that make data, information, and computation possible.

I visualize the meta perspectives of interfaces.

This is a change in perspective, not a change in tools or methods.
Through this, my questions, interests, and motives are shifting.

You might ask, "Why?" Why take such a shortcut by changing the reference point from the world to systems?

The urge to investigate the systems we construct around us is inspired by a theory called ontological design.

In philosophy, the term "ontology" refers to the possibilities and conditions of being.
In computer science, ontology is used in a more narrowly defined way. It is the task of representing and communicating what has been observed or conceived.

The basic idea of ontological design is that what defines us as human beings is not inside our bodies or brains, but rather inside the artifacts, systems, and structures in which we embed ourselves.

“We literally live inside design, like the spider lives inside the web constructed from inside its own body.”

This concept is not new in Human–Computer Interaction. Winograd and Flores have made a similar claim in their book “Understanding Computers and Cognition” 40 years ago:
“In designing tools, we are designing ways of being.”

The digital world is shaped by the social constructs that define our perception of reality. For me, information design is a way to explore the ontological spaces in which we situate ourselves.

There is a circular relationship between design and being human. Everything we design, in return, designs us back.

I do not consider the relationship between humans and design to be a single system. Rather, I see it as a vast, complex network of interacting, overlapping, and alternating loops.
Within this structure, visualization can trace the systems that shape us.

Throughout this talk I will introduce five projects from which I will explain and expand this shift in perspective.

1. From mapping a design space
2. Inspecting how the map is shaping us
3. To contesting the map
4. To investigating new layers of design
5. To end with ontologies of Infiltration

In 2015, I worked with Laszlo Barabasi, a well-known figure in network science. I joined a research project with a simple question: What is the U.S. government doing for its citizens?

To answer the question, we investigated government websites and their links.

For example, the Massachusetts website directly links to unemployment benefits and a page on climate action. Government websites' digital footprints tell us something about what states, schools, and universities are doing for their citizens and how these functions are interlinked.

1. This became a massive undertaking, as we scraped over 30 million .gov web pages from nearly 50,000 websites.
2. We crawled the pages to cluster, categorize, and compare them to legal mandates.
3. The result of this undertaking was simple. It works. The internet footprint provides a good indication of what the government is providing its citizens.

Here, you can see the full network of all 50,000 websites and how they are interlinked. The pages cluster heavily around specific states. However, we can also observe how certain states cluster closer together, are more interlinked with other states, or exhibit unusual patterns compared to others.

Through this, data visualization becomes a method of meta-reflection on a subsection of the Internet. The way we design the internet, in return, designs us back in how we can use and access services.

Coming back to this loop between human and design.

Mapping, diagramming, and visualization are capable of tracing the structures we are embedded in within the architecture of designed systems.

These maps are not monolithic; they offer different perspectives and are subjective representations of the loop. Maps enable us to examine systems from specific viewpoints.

For a project like this, I create hundreds of maps of the data from different angles and perspectives. I view this process as drawing distinctions within the design space.

My point for this talk is that visualization can give us the opportunity to investigate the designs we embed ourselves in.

Second: The map is shaping us

The Schlesinger Library at Harvard University archives documents on the history of women in America. In 2017, the library began collecting tweets with the hashtag “MeToo” and related hashtags. By 2020, they had collected over 30 million tweets and offered a grant to study the data.

Our proposed idea was simple: Investigate how individual tweets spread within the system. How do likes, retweets, and comments evolve after a #MeToo story is shared? Do the comments express support, hate, or dismissal?

We won the grant, and one of the first things I did was analyze a random sample of one million tweets from 2017 to 2020.

To identify the most circulated tweets, I filtered out all tweets with fewer than 100 retweets. There were 892 tweets in total. Then, we read through all of them.

153 of the tweets violated Twitter's rules. Since these tweets show similar patterns in retweet count and messaging, I am confident that they are the result of bot activity. Approximately 17% of the highly calculated tweets were created and circulated by machines.

Various tweets were temporarily unavailable.

A large number of tweets used the #MeToo, but were politically oriented. Discussing Donald Trump, Brett Kavanaugh, and the GOP.

Only 9 tweets, 1% were actual tweets of sharing stories about #MeToo.

From here I build an interface of investigate the phenomena further. My colleagues Catherine D'Ignazio, Cole Martin, Matthew Battles, and I read though filtered subsections of tweets.

What we found was that a vast number of tweets, in which individuals, mainly women, shared their story, were never seen. No likes, retweets, comments.

A common method when working with large datasets is to filter out the unconnected. But what if the important information is the non-connected?

Data Science commonly removes the non-networked, non-ranked.
Superlatives drive the exploration of data.

The final result of this project was a set of four interactive graphics that visualized the neglected, and showcased tweets that never had an interaction in likes, retweets, or comments. Giving form to the unseen.

These mappings did not only allow us to investigate the feedback between humans and design, but also what has been neglected in the system. Visualization can not only look at the structures, but also the shadows they leave behind.

Third: Contesting the Map

What if the network, in its temporal emergence, is not about the points and lines it is seemingly abstracted as?

What if the network is, rather, about the negative space it creates? The holes, the non-spaces, the void, and in-betweenness. Community might exist as the non networked, the gap between the lines, filled, extracted, and enhanced.

Within this visualization, not the network becomes visible, but its past traces. This changes the concept from nodes connected by edges to overlapping areas that merge into one another over time.

Metaphors are extremely powerful. They shape how we situate ourselves. Visualization can draw new worlds into perspective.

Second-order observations can call into question the systems we design and that design us. Visualization enables us to draw new distinctions based on these observations.

Fourth: New Layers of Design

In her book “Atlas of AI,” Kate Crawford discusses artificial intelligence as a registry of power.

I am asking: Can we visualize these registries of power by investigating the outcome of these systems?

I am conceptualizing systems like ChatGPT as a megastructure ranging from the data of all scrapable websites, abusive labor practices, to massive interventions into energy sources, usage of water, and material. At the center of this undertaking lie computational prediction algorithms that feel like magic when neglecting their infrastructural components.

When ChatGPT got released in November 2022, I quickly realized that it can not only generate linear text, but also create small, structured datasets. I had these conversations about categorizing the knowledge of the world into data.

I began requesting categories of knowledge from ChatGPT in order to observe the system's default values.

What came back were long lists of topics and subtopics.

These lists remind me of short stories by Borges or Kafka. Strange, categorical creatures of inclusion and exclusion. Why are accommodation, activities, and adventure part of travel and other things are not?

I used these topics to make further requests to the system, not in the web interface, but via the programming interface.

I asked about important people and personalities. Objects, things, and ideas that are connected to these topics.

In the end, I made almost 1,800 requests to the system.

I created my own little database of what knowledge is, from the perspective of ChatGPT.

I analyzed this data by creating a co-occurrence network, which connects categories to subcategories, and items.

I then calculated the optimal positions of all the objects in relation to each other. Connected items are closer together, while unrelated items are farther apart.

As we can see here, psychology, medicine, and health are placed close to one another in the bottom right, while literature, film, and the arts are on the opposite side of the map. This is how GPT related these categories of knowledge to one another.

The result of this process is what I call "Artificial Worldviews.” A mapping of algorithmic conversations with ChatGPT.

Machine Learning is referred to as a black box. But we can access this black box, very well. By talking to it. Individual queries can only lead to anecdotes. However, structured inquiries in large quantities can give us new insights into what these systems produce.

The aesthetics of the interface act as a Trojan horse, leading the viewer into a world of categorization and relationality of the machine learning dispositif.

What we see here are all the people in the 'Director' category.

Why these ones? Who or what determines the inclusion and exclusion of certain directors? If a majority of texts are soon written by these systems, who or what decides what is included?

The map becomes a representation of power structures between humans, data, and algorithms.

Within the project, I asked for the gender of each individual.
Is this the representation we want?

If, dissertations are written with ChatGPT from now on. Do we want a system that names men three times more often than women?
And almost neglects everything in-between and beyond?

Here is a list of the 26 most frequently mentioned things, ideas, and people from the knowledge map. I would like to draw attention to just four points:
The list consists only of people.
23 of the 26 people are male.
The three women are all at the top of the list, between Aristotle.
The list is led by a zoologist, biologist, science journalist, and non-fiction author whose magnum opus, Silent Spring, from 1962, is often cited as the starting point of the US environmental movement.

The map is freely accessible online and was well received. In December 2023, the scientific journal Nature published the graphic on its cover.

Investigative data practices can play a role in keeping these systems accountable. While I cannot untangle the web of training data from post-training labor, we can investigate the outcomes and reflect on whether what we see is the representation we are looking for within these systems. This will become ever more important as machine learning becomes something society is increasingly dependent on.

In 1710 the German mathematician, philosopher, and scientist Gottfried Wilhelm Leibniz published a book with the title Theodizee, meaning "Justice of God.”

Within the book, Leibniz attempts to answer the question of how suffering in the world can be reconciled with under the assumption that God is omnipotent, omniscient, and good. The question is: how to solve the problem of evil?
—
His answer was that God is constantly calculating the best of all possible worlds. God as a computer.

Silicon Valley is currently selling us a similar but different idea. We are not getting the best of all possible worlds, we are getting the most average of all possible worlds.

And this might not be enough.

Today, it's not only humans who design. But machines start designing, too. The circle duplicates and shifts in these new transitions. Visualization can investigate the loops, shadows, and strange overlaps that happen in-between.

Lastly: Ontologies of Infiltration
This is a current and ongoing book project. I hope that the book will be ready in the beginning of 2027.

On June 9th 2022 a blogpost appeared on the Company Website of the digital advertising company Xandr called “Data Marketplace – Buyer Overview”. Only days before Microsoft acquired Xandr for roughly 1 Billion dollars.

Xandr was an online platform, for buying and selling consumer-centric digital advertising. A marketplace for data brokers.

One year after the acquisition Microsoft dissolved the brand into other products.

Data brokers buy, scrape, and collect data from a variety of sources.

They clean and connect these data sources to create profiles and segments.

Then, they sell these segments for advertising, finance, employment checks, research, government, and law enforcement.

Data brokers is a multi-billion dollar industry.

Acxiom managed 3.7 billion consumer profiles. That is almost half of all humans on earth. Oracle aggregated three billion user profiles and is analyzing 700 million social media messages, blogs, and consumer reviews daily for clustering and segmentation.

Data brokers operations are so powerful that NATO published an report saying that they are among the most powerful organizations in today’s digital world.

At the same time data brokers operate in the dark. In “The Age of Surveillance Capitalism” (2019), one of the most influential accounts of digital capitalism, Shoshana Zuboff mentions data brokers only once in over 500 pages. We are all tracked, segmented and targeted with almost no discussion.

The combination of power, reach, opacity, and neglect turns data brokers into one of the least scrutinized yet most consequential forces in the contemporary information economy.

This brings me back to this blogpost. Within the blogpost someone from within the company published an excel sheet with over 650.000 segments.

Data brokerage segments suddenly became public. What is usually hidden behind interface operations became visible.

While a couple of journalists reported on this. The content of the dataset is largely unexplored.

I started mapping and categorizing the structure of the dataset and it’s actors.

Here an overview of all the 93 data brokers contained in the dataset.

The book is becoming an atlas of data brokers. Visualizing the categorical trees we are all categorized by while using our digital devices.

Not everything about this dataset is evil, nor it is systematic. The data is messy, strange but always with a clear intent, to make money.

The dataset is now five years old. The industry is in a major transition due to the vast changes in machine learning. But make no mistake, this dataset is far from irrelevant.

The main idea of the book is to depict and expose the ontology of commercial classification.

In 2025 Larry Ellison the CTO of Oracle, one of the largest Providers in the dataset, said this:

“So if you look at ChatGPT or Anthropic, Grok, Llama, what have you. They're all trained on all of the data on the internet. In other words, publicly available data. But for these models to reach their peak value, you need to not just train them on publicly available data, but you need to make private, privately owned data available to those models as well.”

I am concerned that Ellison is discussing the segments I am researching, as well as their extensive connections to us. These companies probably train machines to understand our needs, wants, and desires.

The loop between design, human and machine is strangely wrapped. It is not linear, it is temporal. Data from the past reappears in new form through design. AI and big data are not different things but the same ghost reappearing under new names.

In the beginning of this talk I introduced the notion of Ontological Design, that what makes us human is dependent on the structures and systems we embed ourselves in. But maybe Ontological Design is not fully grasping the situation.

In 1993 Jacques Derrida coined the term hauntology. Reality is never simply the present moment. It is always full of traces, ghosts, and history. The question is: What are the ghosts from the past that are haunting us into the future?
I am wondering if we should rather talk about Hauntological Design, rather than Ontological Design.

I started this talk with the circular relationship between design and being human. Everything we design, in return, designs us back.

This system is a vast, complex of interacting, overlapping, and alternating loops between humans and design.

Visualization has the ability to trace the structures of these loops.

These visual representations are not monolithic; they have perspectives and are subjective representations.

These mappings enable us to examine both the visible and the obscured layers of interfaces.

Currently, machines start entering these loops, too. The circle duplicates and shifts.

The relationship between design, humans, and machines is not linear; it is temporal. Data from the past reappears in new forms through design.

Kenya Hara, whom I introduced at the beginning of this talk, coined another useful term: Exformation: Make the known unknown again.

The interfaces and interactions we use are designed by you. You have the power to transform the distinctions that users, consumers, and humans can make. This makes your role very powerful. All of us in this room are designers of the web we live in.

In order to do so, we must critically examine the transformative power of the technologies themselves. The verb cannot exist without the adjective; individual action is embedded within the collective structure.

Within the context of the conference theme of “Transforming Interactions”, I would like to leave you with three questions:
- What should be visible?
- What becomes invisible?
- What should stay invisible?

Thank you