tfS ( ᐛ )و

Sorting Song

Year: 2021
Format: HD Video, 3D Animation
Duration: 6 min 50 sec
Audio: Stereo Sound
Produced with the support of the Pax Media Art Award 2020.

In the video work Sorting Song a selection of domestic objects are displayed in perpetual ambiguity. Morphing between a vase and a bowl, a bench and a couch, a chair and a toilet, the fuzzy borders between language, categorisation and objects are explored. “Must a name mean something?” asks Alice in Lewis Carroll’s Through the Looking Glass. The world is constructed through constant negotiation. Where does the bowl end and the vase begin? Perhaps seemingly trivial, these object boundaries are essential to computer vision training datasets that aim to sort objects into neat categories. With whom does the machine negotiate?

Sorting Song features objects from the SceneNet RGB-D indoor training dataset. This dataset by Imperial College London is a large scale repository of 3D meshes, floorplans and objects, compiled to develop computer vision for future domestic robots. The difficulties of sorting objects by their form alone becomes apparent when a wheelchair, toilet and electric chair are all found within the category “chair”. Context matters, yet this data is stripped of it, and training robots that will ultimately cohabit with humans.

As a succession of the video piece Homeschool, Sorting Song picks up where Homeschool left off: at the frustration with the limitation of computed language and the finite world it produces. The work makes visible the protocols and data that shape the digital representation of the world. Taking the format of an educational kids song, the video piece Sorting Song aims to push beyond what appears innocent and naïve, unsettling a world of coded assumptions.

Related writing


↑ **Sorting Song** at the *Tschumi Pavillion Groningen* as part of Noorderlicht Photofestival 2021. Photo by Hanne van der Velde.

Sorting Song at the Tschumi Pavillion Groningen as part of Noorderlicht Photofestival 2021. Photo by Hanne van der Velde.


⦁ Research, Script & Animation: Simone C Niquille
⦁ Music: Jeff Witscher & Peter Rylander
⦁ Voice Over: Emma Prat & Julian Murray
⦁ 3D Assets: SceneNet-RGBD, Dyson Robotics Lab at Imperial College London
→ With a nod to Sesame Street's Sorting Song.

Credits Cinekid 2021

⦁ Development: Eurico Sá Fernandez
⦁ Emojis: OpenMoji
Commissioned by Cinekid Festival 2021.

Tomatoes are fruits and other tales from category-land

Simone C Niquille in conversation with Eleanor Rosch

A fruit, botany tells us, is any body that develops as the result of a flower’s ovary being fertilised. Botany also tells us that a vegetable is any edible segment of a plant that is not a fruit. A tomato is then, botanically speaking, a fruit. However, tomatoes’ longstanding culinary use alongside vegetables has led to their common labelling as a vegetable. In this essay-interview between designer and researcher Simone C Niquille and professor of psychology Eleanor Rosch, this seemingly mundane example of understanding and negotiating categories becomes a wormhole for exploring the limits and fuzzy borders of categorisation in the world. Seeking to make legible the otherwise hidden frameworks of knowledge production embedded in machine learning, Niquille turns to Rosch, whose prototype theory involves a “graded degree of belonging rather than strict boundaries”. Together, in a conversation that centres on Niquille’s video work Homeschool, they refuse “categorisation as a tool to control a system of relations” and instead propose “a network of objects, things, and concepts that are all connected and yet more or less alike.”

William Labov 1975 \
Where does the cup end and the vase begin? \

William Labov 1975
Where does the cup end and the vase begin?
"boundaries are vague, meaning is context sensitive"

‘When I use a word,’ Humpty Dumpty said, in a rather scornful tone, ‘it means just what I choose it to mean, neither more nor less.’ Lewis Carroll, Through the Looking-Glass

Eyes don’t know what they are looking at. Cameras don’t know what they are looking at. To make sense of visual input, the brain (or neural network) organises these signals into categories. Chairs are ‘furniture,’ an apple is ‘fruit.’ Like a paint-by-number landscape, the world is constantly segmented into digestible parts by machines and humans alike. To the human mind, the scientific accuracy of such mental sorting isn’t crucial. In the world of botany, a tomato is classified as fruit. In the supermarket, to most nutritionists, and in my mind, it is a vegetable. Organising what one sees into mental groups is essential to understanding the surroundings. Categorisation simplifies complexity by drawing comparison and pointing out similarity. In doing so, it also defines what belongs – and what doesn’t. Without categories, there is no anomaly, no possibility to fall outside of their scope. Where a category’s borders are drawn is deeply personal and communal; it reveals cultural context, a set of references and personal beliefs. Categories are a reflection of someone’s worldview.

While communicating with others, categories are constantly negotiated, allowing for approval, synchronisation, disagreement. I refuse to be (mis)gendered based on my hair length. All food can be a snack. My friend’s cat is a human. It is in systems of control and organisation that categories are fixed containers. I am thinking of predefined drop-downs to select one’s gender in online profiles, the taxonomy of plants and animals, the classification of ethnicity through race with a few commonly used labels. In computer vision, with whom does the machine negotiate? The definition of categories and where a category’s border is drawn matters. In supervised machine learning, categories are clearly defined in a labelled training data set. Huge sets of images are assembled to teach computers what they are looking at; these images are segmented and labelled by content, and then they are sorted into categories. This grouping produces an average that simplifies complexity: one does not need to know all of the possible content of a category to process the world. One does not need to have seen all of the chairs of the world to understand what a chair is. Yet, what a chair is depends not on what you use for sitting, but what was defined as such in the training data set. What – and more importantly, whose – world model is used to categorise training data? It takes a position of lazy comfort to make sense of the world by subtracting everything from what one already knows, rather than wanting to understand a thing for what it reveals.

technoflesh Studio ( ᐛ )و

Design & Research practice
of Simone C Niquille

Located in Amsterdam, NL

studio at