From Abstraction to Realism and Back

Hi, I’m Marcin Ignac from studio Variable. Today, I would like share with you a few thoughts on relationship between computer graphics and data visualization and how it influences my practice.

Let’s start with a little bit history! This photo is from 1992, when I was a 9 years old kid, playing with his first computer. Amiga 500.

At that age, half of my computer time I was spending playing computer games like Sid Meier’s Civilisation 1. Trying to understand what do those icons mean as I barely spoke any words in English back then.

The other the half of the time I was spending in a software called Deluxe Paint III. I must say a fairly advanced pixel graphics program. Using few tools and limited color palette one could create digital artworks like this ancient portrait which was the cover image to dpaint.

Even though the drawing tools were simple I was able to start learning how to draw…

.. pixel by pixel and line by line bringing my ideas onto the computer screen. As as you can see the results were not pretty…

Everything has changed next year in June 1993 when in one of the issues of Amiga magazine I saw a tutorial on isometric painting.

I was blown away. Like, WAT! How even!

This 3d, those shadows, the perspective. Just wow. These are not a just buildings. They are so mathematically precise constructions that I couldn’t imagine how they could be done by a human alone. At this point I think I literally threw mouse in the air and went for a bike ride or play with LEGO.

The article says “The artistic quality of an artwork is independent from the tool and the material in which it is created. But both the tool and the material decide on it’s final form.” It’s clear that I didn’t have the tools back then at my disposal and any material to work with.

Ludwig Wittgenstein an Austrian - British philosopher once said “The limits of my language are the limits of my world”. Both my language and therefore the world I can describe with was very small at that time.

I never stopped pixel pushing though. Throughout the high school I was learning new software, copying, experimenting, “wasting time”.

Moving to vector graphics allowed me to achieve the precision I was so amazed in the first place. Learn more about composition, colors, shadows and playing with ideas.

But math and in particular geometry was my true passion. I find geometry as way to describe relationships between objects frankly quite beautiful.

That passion evolved into studying computer science, learning about algorithms, code, debugging and speaking C++ to the computer.

Finally I was able to pick up computer graphics and start realizing my ideas in 3d space.

Being computer graphics programmer at that time in Europe.. sooner or later brings you to the demoscene - a computer art subculture focused around producing demos..

• small non interactive computer animations made with code.., featuring electronic music.. and artworks made by fellow group members. There is number of demo categories but my favourite is definitely 64kb where the size limitation of the executable file forces you to generate most of the content and animations in code. Just like we can see on the screen here.

This is where the math comes in again. Taking basic concepts like a torus and adding more complicated formulas combined with procedural colors and textures I was able to create new forms and bring them to life.

My group was called DMA and here you see our first production we have made. A 64kb intro with all the visual content procedurally generated. I think it took me about half a year of 4h a day of learning, coding, building tools and experimenting to make it. Still it was just a hobby…

Back at the Computer Science uni being part of Intelligent Decision Support Systems major meant that you spend your time learning about Java and writing web crawlers. But I didn’t stop there. I could combine my computer graphics interest with the collected data. Here is what I consider my first data visualization I have ever made. It uses data from a now gone polish social network called “Grono" that predates Facebook by one month.

After finishing Computer Science I had one year break studying new media art but soon I dropped out and moved to Denmark to join the pilot year at Copenhagen Institute of Interaction Design. At CIID I learned how to use software & hardware prototyping combined with user research and data to solve complex problems.

It is CIID that finally opened my eyes to the possibilities of using data as a material and the power of interactivity for exploration.

After graduating I eventually joined CIID Research and practiced more traditional forms of data visualisation like charts, maps and diagrams. The design process combined with analytical mindset had big influence on my personal work. For example …

Here is a jellyfish. If were given a task to program realistic computer generated jellyfish it’s an incredibly difficult one. Just getting the transparency right could take weeks of work. I actually tried once and failed so badly I won’t even show you any screenshots today..

But let’s think for a moment what I’m actually trying to do here?… Jellyfish are made out of 3 distinct parts: We have the wavy head, loose hair and curly tentacles.

And here are the geometric primitives I knew how to create in code at that time. Points, lines, polygons, solids and some basic GPU shaders, like procedural noise.

Having those basic concepts we can build more complex structures. Let’s start with a head… On the top.. we have a subdivided circle which radius is modulated by a sine function. On the bottom.. we mirror the vertices after reaching certain angle that creates a half moon shape that’s later subdivided and rounded into a croissant.

These 2d constructs can be then expanded into 3rd dimension be it the time where we again use sine function to modify the path or depth where we revolve the shape 360 degrees.

We can then modify the geometry of those 3d objects but combining them into even more complex structures or we make them alive by applying physics and behaviours.

To escape the sterile vacuum of pure geometry I started adding shading and think of the textures I would like to achieve.

Looking for inspiration I encountered a book called Kunstformen der Natur (artforms in nature) by German biologist Ernst Haeckel from the beginning of the 20th century. The book is full of hand drawn illustrations of underwater life.

I immediately knew that instead of going for a realistic style I wanted more stylized version with multiple layers of ink, chalk and crayon like textures.

The final results exceeded my expectations and I was honoured to be included in Written Images book where every printed page was generated and unique for every copy of the book.

Even though the Cindermedusae is known for its rendering style… the more interesting part for me was the system of interconnected parameters that produces it. Each parameter was continuous and animatable allowing for seemingly infinite number of variations or species. This brings a number of questions:

What else can we describe as a parametric system?

Can everything be deconstructed into basic elements? What are those?

What is the role of a designer / artist?

These questions eventually led me to leaving CIID and in 2012 starting my own studio: Variable. Working with companies, researchers and artist we explore new ways of experiencing data and systems through artworks, installations and visualizations.

Our work is blending of design, software and aesthetic emerging from data itself, processes and human behaviour.

One of my favourite Variable projects is a piece we did for Nike called Fibers. It’s an exercise data visualisation using their Nike Fuelbands that tracks how much you move through out the day. Not matter if you run, dance, or do the dishes you collect so called fuel point. On the left we can see an example week of data with inactive periods of time during sleep, 4 sharp spikes caused by the morning runs and low level of activity during the day.

If you think about 24h day as a cycle you can represent it as a circle. We can then put the fuel points you have collected next to the time of the activity. This looks quite cool but i was more interested in the total amount of points that you collected that day.

As you collect more and more points i extrude the line higher and higher. As it keeps growing it starts looking similar to a branch of a tree.

When we combine 7 of them, one for each day of the week… for me they started to look like a muscle, hence the name of the final artwork Fibers. Structurally the project is very similar to Cindermedusa. I have a set of parameters like number of fibers, thickness and the length of them, ratios of colors etc. But this time instead of choosing random numbers I use the data to control the final result.

Here are some first renders. The shapes are great but the visual style was not really going anywhere. They felt disconnected from the data. I wanted to work with real colors of real materials.

So I decided to steal again. This time it was colors from Nike gear for women. I wrote a piece of software that analize pixels of the photo and extract colors from it. This way I’ve got a color palette that I could work with.

Here are the final final visualizations. What’s interesting here is that these visualisations are true to the data. You can easily see which day was the most successful as the biggest workouts have the thickest fibers.. You can tell if the person has a schedule where most of the days look similar like on the right or if she more spontaneous like on the left.

Here is a short animation of this visualization explaining again how the form were created. The 24 hour circle, the growth, thicker parts for the bigger workouts and the seven days of data for 7 different runners. Even though the final result was static images I created those animations to show the evolution of data throughout the week. It is worth noting that those animations are not renders in Cinema 4D or Maya but completely created in code and webgl. To achieve that level of quality you need tools.

Our tool at variable is called PEX. PEX is a set of JavaScript modules for computational thinking for Node.js and WebGL in the browser. Even though PEX is open source you probably have never heard of it before. You might ask: Why write my own? Why don’t just use ThreeJS or Unity?

Well.. I believe that Richard Feynman was right by writing just before his death “What I cannot create, I do not understand”. I feel the same about software and tools I’m using. I want to understand now they work.

..but if you go to my twitter… you will be confused. Most of the time I talk about webgl.., rendering algorithms.., driver bugs… and new tools I've been building,,. Barely any data visualization at all it seems….

It’s because I'm trying to understand. What happens from the moment a stroke of light hits the surface of an object.

And how the GPU rendering pipeline works.

I build tools to understand the structure of my own code… like this diagram explaining shadow mapping algorithm. It wasn’t created by hand but by intercepting the commands I’m sending to the GPU.

I build tools to visualize how those algorithms see the 3d worlds I'm creating. I want to unpack as many of the black boxes as I can…

While learning the theory of modern rendering techniques.

That allows me to achieve physically correct and realistic renders in realtime. From plastic and chrome to wood and rubber.

It’s not only about the rendering quality. By choosing materials and controlling realistic lighting I can start referring to the emotions `and moods those images are creating.. like this virtual sunset bringing warm and relaxed atmosphere.

In the end it’s all about expanding your vocabulary

So we can go from a rough stream of cubes

To a mesmerising artworks that are truthful to you intention and allow for new forms of expression.

Inflows and Outflows is a data driven artwork based on population flows in the middle east.

There is four of them in the series. Each representing the same data set of years 2006 - 2014 of how many people came and left to different parts of the region. The readability of the data was not a goal here, quite the opposite but the concept as you will see it’s quite simple. Each strand represent one month of data. It’s height represent how many people came or left that month. But it’s only one of possible representations of the data.

Here is a better one. We can immediately which region is the most active and where the financial crisis in 2008 had the biggest impact.

When working with this kind of project I always start with data visualization in order to understand the data I'm dealing with. What are the ranges of values, how do they change?

And here is another view that despite being less readable than the previous one it uncovers a massive spike which could be an interesting point for further investigation.

Perhaps instead if data visualization a better way to describe to what I do is through computational design and the definition by Lorena Barba: “solving problems by exploiting concepts fundamental to computer science: abstraction, decomposition, recursion, separation of concerns”

I like to think about it as fluctuating between building systems and decomposing existing ones.

Like Pablo Picasso in his Bull series moving from a realistic drawing of a bull.. to more and more abstract one.

If we push the abstraction one level further to a “data bull”. We get a full spectrum that explains the title of this talk.

Starting from the basic elements, the lines and points and moving towards realism we get SYNTHESIS (or what I'm trying to do with graphics programming)

While moving from left to right we get data visualization where we try to explain concepts with more and more abstract representations. And most of the time I'm somewhere in between the real and the abstract.

Let’s talk about maps.

Maps fascinate me. They are probably one of the most common type of visualizations we encounter in our every day life. Maps are also all about moving between different levels of abstration. From realistic satellite photos and street views.

To the geometric shapes of streets, parks and rivers. Maps are made of data, the geo locations of places & monuments, boundaries of districts and countries.

Or like this LIDAR scan of my neighbourhood. If you look at the top left corner you can easily notice a rugby stadium. It was there on the two previous screens.. but their representation wasn’t optimized to show height of the building.

Therefore we can say that all maps are interpretations. We believe in them. We trust their GIS 1 inch accuracy. But that precision can be also very sterile. If you think of City of London what’s the first image that comes to your mind? Before moving there I always though of London as the city of fog and dancing in the rain. Maps can’t capture that.

So I experimented with adding fog to hide the imperfect abstractions and add a layer of realism and sense of belonging. What was an empty Lego city suddenly becomes a place with a story. Cars stuck in the traffic… and people being late…

Here is another one. [Marcin takes sip of water]

Another thing maps are very good at is showing context By putting data in a familiar environment we allow the spectator to draw in their own stories and make connections. This is a map of all my gps positions in London the last 4(?) years, places i walked, run or cycled. You can probably guess what are the three neighbourhoods that i lived in so far.

Another thing maps are very good at is showing context By putting data in a familiar environment we allow the spectator to draw in their own stories and make connections. This is a map of all my gps positions in London the last 4(?) years, places i walked, run or cycled. You can probably guess what are the three neighbourhoods that i lived in so far. [need another map with smaller zoom, I wish this was all animation..]

Another thing maps are very good at is showing context By putting data in a familiar environment we allow the spectator to draw in their own stories and make connections. This is a map of all my gps positions in London the last 4(?) years, places i walked, run or cycled. You can probably guess what are the three neighbourhoods that i lived in so far. [need another map with smaller zoom, I wish this was all animation..]

Another thing maps are very good at is showing context By putting data in a familiar environment we allow the spectator to draw in their own stories and make connections. This is a map of all my gps positions in London the last 4(?) years, places i walked, run or cycled. You can probably guess what are the three neighbourhoods that i lived in so far. [need another map with smaller zoom, I wish this was all animation..]

If we zoom out we can see all the places i travelled to in UK and Ireland. And this point I asked a question what would happen if we add height information.

This is of course not up to scale but I wanted to highlight the differences which if you go on a road trip are very apparent. It’s still not the UK I know.

Here is the same map with vegetation index data from US National Oceanic and Atmospheric Administration. Finally a map that resembles UK from my experience, green, wast and waiting to you to go out there.

Let's plot all the uk cities above 1000 inhabitants. Suddenly places like Newcastle in the East and Whales in the West pop up. Beside context maps are great at visualizing geo located data. Lets try another thing. Which cities are the most innovative?

Plotting all the projects that received research funding money from the UK gov in the last 10y gives us another picture. Suddenly Liverpool seems like a place to go to…

Here is a visualization of all the partner organizations of University College London. Our metaphor breaks and the geographical context is no longer useful. We need to go couple of abstraction levels deeper.

A better way to represent reaserch network in the UK could look like this. A graph.. starting with the organization of interest at the bottom we see projects as squares linking to other collaborator and projects the are working on. Buy using this tool and exploring the links we can build better mental models and draw our own maps. I had a chance to challenge the relationship between maps and the data while running a workshop with students at UCL bartlett school of architecture.

The starting point were LIDAR point clouds and how can we use them as a raw material to tell stores about parts of London.

Surprisingly most students were not that interested in the physical structure of the building that the LIDAR scans are capturing but in the digital layer, the invisible, the immaterials: tweets, Instagram photos, air pollution levels and algorithmically calculated paths of the Uber cabs. We realized that we had more questions than aswers. If you order a food via an app while walking down the Old Street to get you take away from Islington being delivered by a curier from Bethnal Green that got a notification from a cloud server in California. How do you visualize that?

I want to talk about maps but not in a geographical sense. We use maps for navigation but what are the navigational tools to find our way in the digital domain? Should we visualize the data itself or the metadata that aggregated is perhaps more valuable?

Here is my soon to become wife working with me from a cafe in Jeju island in South Korea couple of years ago. I love working from places like this, cafes, libraries, museums… I feel a bit like in an aquarium… observing surrounding me environment and people passing by. It’s very calming… making you aware and focused at the same time. One of those times I asked myself very simple question: why this website loads so slow?

You immediately start blaming wifi. What can be so difficult? We click a link, our browser sends a request and the server responds back with a website.

Having done web development for quite a while I knew that the reality is a bit more complicated. We click a link and our browser sends a request packages (together with IP, cookies, user agent information etc) and the server responds with data packages (text, photos, video files etc)

To my surprise what is really happening nowadays is far more complex: Today the websites are built from many separate blocks so we actually talk to many servers at once. Photos might come from a cloud storage provider, friend list from your social network, ads from an ad broker. Each of those files can include scripts loading even more bits and pieces from other servers. What do all those servers do and which information they retain is not clear or transparent. So I started digging.

Let’s see a real example. This a tech blog called The Verge loading in Google Chrome with DevTools opened on the right. You can see ads kicking in… [scrolling] It’s a half a page article with one image but the whole thing took almost 500 requests to different servers.

Let’s compare two websites: If we visit infoesthetics.com we will do a moderate 100 requests. It’s seems a lot but it's a highly visual blog about data visualisation, they use a couple of social media buttons, and ad providers. So it’s ok.

On the other hand if you visit a clickbait website like timetobreak.com you get whooping 1500 requests.

Data as a Place

I like to think about data as a place. Somewhere I can visit, observe and look around.

So I have built a prototype. The different colors represent different types of files like code, images etc. What’s important though are the files highlighted in red. Those are classified as trackers. One of the reasons why I'm working on this is that I think projects like Disconnect me or Mozilla Lightbeam fail to capture my imagination and tent be either scary or simplistic.

Here is how another clickbait website looks like. About 80% of loaded content doesn’t serve any value to the visitor. I felt like this representation is a dead end so I’ve put the project on hold for quite a long time. But the question on how to visualize our interactions online kept combing back.

While preparing my talk for Eyeo I decided to revisit the the project by looking at the last 6 months of my browsing history.

Worst offender, up to 2000, same as cooking websites.

This website is build from thousands of JS files. Turns out WIX.com doesn't bundle their application.

And here is my bank's website.

[Video]

t-SNE of website requests.

“We Are Data: Algorithms and The Making of Our Digital Selves” by John Cheney-Lippold Joke: Lets pause on the usage of floppy discs on that image… The book talks about “Algorithmic identities” “who we are in terms of data” “algorithmic interperetations behind a black box” and “datafication: the transformation of part, if not most, of our lives into computable data”. What does it mean to be man? Or woman?

Enter algorithmic cubism

These are categories Google has assigned to me to serve me personalized ads.

Some of them makes sense like Parenting, Home & Garden, Cooking & Recipes.

But Google also thinks I'm interested in Rugby. I've nerve played rugby, i've never been to a rugby game and I don't even know the rules. But I happend to live in Twickenham which is home to biggest rugby stadium in the UK, hence according to Google I must love Rugby...

“Our datafied lives, when aggregated and transcoded into quotation-marked categories, increasingly define who we are and who we can be.” - by John Cheney-Lippold / We Are Data

Here is a 3d scan of me from 2010. A digital representation of myself. I had beard and was single. Nowadays I’m married but that beard still regrows from time to time before deadlines. This is me how I perceive myself: the same geeky guy for years, nickname VORG, reading about computer graphics, listening to techno, and still pulling out all nighters on a weekly basis.

If you look at my lunch and leisure time browsing and compare that with publicly available data you might see completely average polish immigrant living in London.

Male 25-35. Frantically refreshing The Guardian and BBC websites scavenging for newest information about Brexit.

During the day though I’m more “real” me. A 30+ developer, company owner. Perhaps 75% polish or 10% danish as I lived in Denmark for 5 years before and still have many friends on Facebook there.

During the day though I’m more “real” me. A 30+ developer, company owner. Perhaps 75% polish or 25% danish as I lived in Denmark for 5 years before and still have many friends on Facebook there.

My geolocation data is confusing though. I live in UK, do most of my projects for US clients, I use ‘z’ insted of ’s’ when I spell “data visualization” (which drops me to the second page of Google results btw).

Maybe I’m 40% American then?

Some website seem to disagree.
Where would all those ads of prams, nursing bras, baby photographers?

Maybe I'm 50% korean woman then?

“As subjects to algorithmic interpretation, who we are also changes “from point to point,” moving from ‘woman’ to ‘man’ or ‘old’ to ‘young.’”

Here are initial explorations how the transitions between different identities could look like.

There is no data plugged in yet but I want to classify website I'm visiting and assign them to different profiles to speculate how those identities could be shaped.

In a way we are made of data.

Who am I?

Am I all those people or not of them? These are 3d scans of people I will never meet. Approximations of my online identities that I’ll never see.

Maybe it’s like cubism with multiple contradicting perspectives embedded within one picture. Data visualization and data art can become a new lens onto the world allowing us to see things differently and from many points of view at the same time.

I call this project you are I/O. Because we are at the center of this data infrastructure that’s both fed by our actions and influences what we can do.

We need tools to reverse engineer the data reality around use

“maybe if you see art as a reverse engineering problem it becomes more interesting” - Mario Klingemann

Building alternative representations of the underlying systems and our data identity

So we can move between different levels of abstraction and draw our own bull.

Thank You