Marcin Ignac

From Abstraction to Realism and Back

This is transcribed version of a talk I gave at Eyeo Festival in Minneapolis in June 2017. The talk is about the relationship between computer graphics and data visualization and how it influences my practice.

Video of the talk is available on Vimeo

Note: You can click on the images to see higher res versions.


slide

Hi, I’m Marcin Ignac from studio Variable. Today, I would like share with you a few thoughts on relationship between computer graphics and data visualization and how it influences my practice.


slide

Let’s start with a little bit history! This photo is from 1992, when I was a 9 years old kid, playing with his first computer. Amiga 500.


slide

At that age, half of my computer time I was spending playing computer games like Sid Meier’s Civilisation 1. Trying to understand what do those icons mean as I barely spoke any words in English back then.


slide

The other the half of the time I was spending in a software called Deluxe Paint III. I must say a fairly advanced pixel graphics program. Using few tools and limited color palette one could create digital artworks like this ancient portrait which was the cover image to dpaint.


slide

Even though the drawing tools were simple I was able to start learning how to draw…


slide

.. pixel by pixel and line by line bringing my ideas onto the computer screen. As as you can see the results were not pretty…


slide

Everything has changed next year in June 1993 when in one of the issues of Amiga magazine I saw a tutorial on isometric painting.


slide

I was blown away. Like, WAT! How even!


slide

This 3d, those shadows, the perspective. Just wow. These are not a just buildings. They are so mathematically precise constructions that I couldn’t imagine how they could be done by a human alone. At this point I think I literally threw mouse in the air and went for a bike ride or play with LEGO.


slide

The article says “The artistic quality of an artwork is independent from the tool and the material in which it is created. But both the tool and the material decide on it’s final form.” It’s clear that I didn’t have the tools back then at my disposal and any material to work with.


slide

Ludwig Wittgenstein an Austrian - British philosopher once said “The limits of my language are the limits of my world”. Both my language and therefore the world I can describe with was very small at that time.


slide

I never stopped pixel pushing though. Throughout the high school I was learning new software, copying, experimenting, “wasting time”.


slide

Moving to vector graphics allowed me to achieve the precision I was so amazed in the first place. Learn more about composition, colors, shadows and playing with ideas.


slide

But math and in particular geometry was my true passion. I find geometry as way to describe relationships between objects frankly quite beautiful.


slide

That passion evolved into studying computer science, learning about algorithms, code, debugging and speaking C++ to the computer.


slide

Finally I was able to pick up computer graphics and start realizing my ideas in 3d space.


slide

Being computer graphics programmer at that time in Europe.. sooner or later brings you to the demoscene - a computer art subculture focused around producing demos..


slide


slide

This is where the math comes in again. Taking basic concepts like a torus and adding more complicated formulas combined with procedural colors and textures I was able to create new forms and bring them to life.


slide

My group was called DMA and here you see our first production we have made. A 64kb intro with all the visual content procedurally generated. I think it took me about half a year of 4h a day of learning, coding, building tools and experimenting to make it. Still it was just a hobby…


slide

Back at the Computer Science uni being part of Intelligent Decision Support Systems major meant that you spend your time learning about Java and writing web crawlers. But I didn’t stop there. I could combine my computer graphics interest with the collected data. Here is what I consider my first data visualization I have ever made. It uses data from a now gone polish social network called “Grono" that predates Facebook by one month.


slide

After finishing Computer Science I had one year break studying new media art but soon I dropped out and moved to Denmark to join the pilot year at Copenhagen Institute of Interaction Design. At CIID I learned how to use software & hardware prototyping combined with user research and data to solve complex problems.


slide

It is CIID that finally opened my eyes to the possibilities of using data as a material and the power of interactivity for exploration.


slide

After graduating I eventually joined CIID Research and practiced more traditional forms of data visualisation like charts, maps and diagrams. The design process combined with analytical mindset had big influence on my personal work. For example …


slide

Here is a jellyfish. If were given a task to program realistic computer generated jellyfish it’s an incredibly difficult one. Just getting the transparency right could take weeks of work. I actually tried once and failed so badly I won’t even show you any screenshots today..


slide

But let’s think for a moment what I’m actually trying to do here?… Jellyfish are made out of 3 distinct parts: We have the wavy head, loose hair and curly tentacles.


slide

And here are the geometric primitives I knew how to create in code at that time. Points, lines, polygons, solids and some basic GPU shaders, like procedural noise.


slide

Having those basic concepts we can build more complex structures. Let’s start with a head… On the top.. we have a subdivided circle which radius is modulated by a sine function. On the bottom.. we mirror the vertices after reaching certain angle that creates a half moon shape that’s later subdivided and rounded into a croissant.


slide

These 2d constructs can be then expanded into 3rd dimension be it the time where we again use sine function to modify the path or depth where we revolve the shape 360 degrees.


slide

We can then modify the geometry of those 3d objects but combining them into even more complex structures or we make them alive by applying physics and behaviours.


slide

To escape the sterile vacuum of pure geometry I started adding shading and think of the textures I would like to achieve.


slide

Looking for inspiration I encountered a book called Kunstformen der Natur (artforms in nature) by German biologist Ernst Haeckel from the beginning of the 20th century. The book is full of hand drawn illustrations of underwater life.


slide

I immediately knew that instead of going for a realistic style I wanted more stylized version with multiple layers of ink, chalk and crayon like textures.


slide

The final results exceeded my expectations and I was honoured to be included in Written Images book where every printed page was generated and unique for every copy of the book.


slide

Even though the Cindermedusae is known for its rendering style… the more interesting part for me was the system of interconnected parameters that produces it. Each parameter was continuous and animatable allowing for seemingly infinite number of variations or species. This brings a number of questions:


slide

What else can we describe as a parametric system?

Can everything be deconstructed into basic elements? What are those?

What is the role of a designer / artist?


slide

These questions eventually led me to leaving CIID and in 2012 starting my own studio: Variable. Working with companies, researchers and artist we explore new ways of experiencing data and systems through artworks, installations and visualizations.


slide

Our work is blending of design, software and aesthetic emerging from data itself, processes and human behaviour.


slide

One of my favourite Variable projects is a piece we did for Nike called Fibers. It’s an exercise data visualisation using their Nike Fuelbands that tracks how much you move through out the day. Not matter if you run, dance, or do the dishes you collect so called fuel point. On the left we can see an example week of data with inactive periods of time during sleep, 4 sharp spikes caused by the morning runs and low level of activity during the day.


slide

If you think about 24h day as a cycle you can represent it as a circle. We can then put the fuel points you have collected next to the time of the activity. This looks quite cool but i was more interested in the total amount of points that you collected that day.


slide

As you collect more and more points i extrude the line higher and higher. As it keeps growing it starts looking similar to a branch of a tree.


slide

When we combine 7 of them, one for each day of the week… for me they started to look like a muscle, hence the name of the final artwork Fibers. Structurally the project is very similar to Cindermedusa. I have a set of parameters like number of fibers, thickness and the length of them, ratios of colors etc. But this time instead of choosing random numbers I use the data to control the final result.


slide

Here are some first renders. The shapes are great but the visual style was not really going anywhere. They felt disconnected from the data. I wanted to work with real colors of real materials.


slide

So I decided to steal again. This time it was colors from Nike gear for women. I wrote a piece of software that analize pixels of the photo and extract colors from it. This way I’ve got a color palette that I could work with.


slide

Here are the final final visualizations. What’s interesting here is that these visualisations are true to the data. You can easily see which day was the most successful as the biggest workouts have the thickest fibers.. You can tell if the person has a schedule where most of the days look similar like on the right or if she more spontaneous like on the left.


slide

Here is a short animation of this visualization explaining again how the form were created. The 24 hour circle, the growth, thicker parts for the bigger workouts and the seven days of data for 7 different runners. Even though the final result was static images I created those animations to show the evolution of data throughout the week. It is worth noting that those animations are not renders in Cinema 4D or Maya but completely created in code and webgl. To achieve that level of quality you need tools.


slide

Our tool at variable is called PEX. PEX is a set of JavaScript modules for computational thinking for Node.js and WebGL in the browser. Even though PEX is open source you probably have never heard of it before. You might ask: Why write my own? Why don’t just use ThreeJS or Unity?


slide

Well.. I believe that Richard Feynman was right by writing just before his death “What I cannot create, I do not understand”. I feel the same about software and tools I’m using. I want to understand now they work.


slide

..but if you go to my twitter… you will be confused. Most of the time I talk about webgl.., rendering algorithms.., driver bugs… and new tools I've been building,,. Barely any data visualization at all it seems….


slide

It’s because I'm trying to understand. What happens from the moment a stroke of light hits the surface of an object.


slide

And how the GPU rendering pipeline works.


slide

I build tools to understand the structure of my own code… like this diagram explaining shadow mapping algorithm. It wasn’t created by hand but by intercepting the commands I’m sending to the GPU.


slide

I build tools to visualize how those algorithms see the 3d worlds I'm creating. I want to unpack as many of the black boxes as I can…


slide

While learning the theory of modern rendering techniques.


slide

That allows me to achieve physically correct and realistic renders in realtime. From plastic and chrome to wood and rubber.


slide

It’s not only about the rendering quality. By choosing materials and controlling realistic lighting I can start referring to the emotions `and moods those images are creating.. like this virtual sunset bringing warm and relaxed atmosphere.


slide

In the end it’s all about expanding your vocabulary


slide

So we can go from a rough stream of cubes


slide

To a mesmerising artworks that are truthful to you intention and allow for new forms of expression.


slide

Inflows and Outflows is a data driven artwork based on population flows in the middle east.


slide

There is four of them in the series. Each representing the same data set of years 2006 - 2014 of how many people came and left to different parts of the region. The readability of the data was not a goal here, quite the opposite but the concept as you will see it’s quite simple. Each strand represent one month of data. It’s height represent how many people came or left that month. But it’s only one of possible representations of the data.


slide

Here is a better one. We can immediately which region is the most active and where the financial crisis in 2008 had the biggest impact.


slide

When working with this kind of project I always start with data visualization in order to understand the data I'm dealing with. What are the ranges of values, how do they change?


slide

And here is another view that despite being less readable than the previous one it uncovers a massive spike which could be an interesting point for further investigation.


slide

Perhaps instead if data visualization a better way to describe to what I do is through computational design and the definition by Lorena Barba: “solving problems by exploiting concepts fundamental to computer science: abstraction, decomposition, recursion, separation of concerns”


slide

I like to think about it as fluctuating between building systems and decomposing existing ones.


slide

Like Pablo Picasso in his Bull series moving from a realistic drawing of a bull.. to more and more abstract one.


slide

If we push the abstraction one level further to a “data bull”. We get a full spectrum that explains the title of this talk.


slide

Starting from the basic elements, the lines and points and moving towards realism we get SYNTHESIS (or what I'm trying to do with graphics programming)


slide

While moving from left to right we get data visualization where we try to explain concepts with more and more abstract representations. And most of the time I'm somewhere in between the real and the abstract.


slide

Let’s talk about maps.


slide

Maps fascinate me. They are probably one of the most common type of visualizations we encounter in our every day life. Maps are also all about moving between different levels of abstration. From realistic satellite photos and street views.


slide

To the geometric shapes of streets, parks and rivers. Maps are made of data, the geo locations of places & monuments, boundaries of districts and countries.


slide

Or like this LIDAR scan of my neighbourhood. If you look at the top left corner you can easily notice a rugby stadium. It was there on the two previous screens.. but their representation wasn’t optimized to show height of the building.


slide

Therefore we can say that all maps are interpretations. We believe in them. We trust their GIS 1 inch accuracy. But that precision can be also very sterile. If you think of City of London what’s the first image that comes to your mind? Before moving there I always though of London as the city of fog and dancing in the rain. Maps can’t capture that.


slide

So I experimented with adding fog to hide the imperfect abstractions and add a layer of realism and sense of belonging. What was an empty Lego city suddenly becomes a place with a story. Cars stuck in the traffic… and people being late…


slide

Here is another one. [Marcin takes sip of water]


slide

Another thing maps are very good at is showing context By putting data in a familiar environment we allow the spectator to draw in their own stories and make connections. This is a map of all my gps positions in London the last 4(?) years, places i walked, run or cycled. You can probably guess what are the three neighbourhoods that i lived in so far.


slide

Another thing maps are very good at is showing context By putting data in a familiar environment we allow the spectator to draw in their own stories and make connections. This is a map of all my gps positions in London the last 4(?) years, places i walked, run or cycled. You can probably guess what are the three neighbourhoods that i lived in so far. [need another map with smaller zoom, I wish this was all animation..]


slide

Another thing maps are very good at is showing context By putting data in a familiar environment we allow the spectator to draw in their own stories and make connections. This is a map of all my gps positions in London the last 4(?) years, places i walked, run or cycled. You can probably guess what are the three neighbourhoods that i lived in so far. [need another map with smaller zoom, I wish this was all animation..]


slide

Another thing maps are very good at is showing context By putting data in a familiar environment we allow the spectator to draw in their own stories and make connections. This is a map of all my gps positions in London the last 4(?) years, places i walked, run or cycled. You can probably guess what are the three neighbourhoods that i lived in so far. [need another map with smaller zoom, I wish this was all animation..]


slide

If we zoom out we can see all the places i travelled to in UK and Ireland. And this point I asked a question what would happen if we add height information.


slide

This is of course not up to scale but I wanted to highlight the differences which if you go on a road trip are very apparent. It’s still not the UK I know.


slide

Here is the same map with vegetation index data from US National Oceanic and Atmospheric Administration. Finally a map that resembles UK from my experience, green, wast and waiting to you to go out there.


slide

Let's plot all the uk cities above 1000 inhabitants. Suddenly places like Newcastle in the East and Whales in the West pop up. Beside context maps are great at visualizing geo located data. Lets try another thing. Which cities are the most innovative?


slide

Plotting all the projects that received research funding money from the UK gov in the last 10y gives us another picture. Suddenly Liverpool seems like a place to go to…


slide

Here is a visualization of all the partner organizations of University College London. Our metaphor breaks and the geographical context is no longer useful. We need to go couple of abstraction levels deeper.


slide

A better way to represent reaserch network in the UK could look like this. A graph.. starting with the organization of interest at the bottom we see projects as squares linking to other collaborator and projects the are working on. Buy using this tool and exploring the links we can build better mental models and draw our own maps. I had a chance to challenge the relationship between maps and the data while running a workshop with students at UCL bartlett school of architecture.


slide

The starting point were LIDAR point clouds and how can we use them as a raw material to tell stores about parts of London.


slide

Surprisingly most students were not that interested in the physical structure of the building that the LIDAR scans are capturing but in the digital layer, the invisible, the immaterials: tweets, Instagram photos, air pollution levels and algorithmically calculated paths of the Uber cabs. We realized that we had more questions than aswers. If you order a food via an app while walking down the Old Street to get you take away from Islington being delivered by a curier from Bethnal Green that got a notification from a cloud server in California. How do you visualize that?


slide

I want to talk about maps but not in a geographical sense. We use maps for navigation but what are the navigational tools to find our way in the digital domain? Should we visualize the data itself or the metadata that aggregated is perhaps more valuable?


slide

Here is my soon to become wife working with me from a cafe in Jeju island in South Korea couple of years ago. I love working from places like this, cafes, libraries, museums… I feel a bit like in an aquarium… observing surrounding me environment and people passing by. It’s very calming… making you aware and focused at the same time. One of those times I asked myself very simple question: why this website loads so slow?


slide

You immediately start blaming wifi. What can be so difficult? We click a link, our browser sends a request and the server responds back with a website.


slide

Having done web development for quite a while I knew that the reality is a bit more complicated. We click a link and our browser sends a request packages (together with IP, cookies, user agent information etc) and the server responds with data packages (text, photos, video files etc)


slide

To my surprise what is really happening nowadays is far more complex: Today the websites are built from many separate blocks so we actually talk to many servers at once. Photos might come from a cloud storage provider, friend list from your social network, ads from an ad broker. Each of those files can include scripts loading even more bits and pieces from other servers. What do all those servers do and which information they retain is not clear or transparent. So I started digging.


slide

Let’s see a real example. This a tech blog called The Verge loading in Google Chrome with DevTools opened on the right. You can see ads kicking in… [scrolling] It’s a half a page article with one image but the whole thing took almost 500 requests to different servers.


slide

Let’s compare two websites: If we visit infoesthetics.com we will do a moderate 100 requests. It’s seems a lot but it's a highly visual blog about data visualisation, they use a couple of social media buttons, and ad providers. So it’s ok.


slide

On the other hand if you visit a clickbait website like timetobreak.com you get whooping 1500 requests.


slide

Data as a Place


slide

I like to think about data as a place. Somewhere I can visit, observe and look around.


slide

So I have built a prototype. The different colors represent different types of files like code, images etc. What’s important though are the files highlighted in red. Those are classified as trackers. One of the reasons why I'm working on this is that I think projects like Disconnect me or Mozilla Lightbeam fail to capture my imagination and tent be either scary or simplistic.


slide

Here is how another clickbait website looks like. About 80% of loaded content doesn’t serve any value to the visitor. I felt like this representation is a dead end so I’ve put the project on hold for quite a long time. But the question on how to visualize our interactions online kept combing back.


slide

While preparing my talk for Eyeo I decided to revisit the the project by looking at the last 6 months of my browsing history.


slide


slide


slide

Worst offender, up to 2000, same as cooking websites.


slide

This website is build from thousands of JS files. Turns out WIX.com doesn't bundle their application.


slide

And here is my bank's website.


slide

[Video]


slide

t-SNE of website requests.


slide

“We Are Data: Algorithms and The Making of Our Digital Selves” by John Cheney-Lippold Joke: Lets pause on the usage of floppy discs on that image… The book talks about “Algorithmic identities” “who we are in terms of data” “algorithmic interperetations behind a black box” and “datafication: the transformation of part, if not most, of our lives into computable data”. What does it mean to be man? Or woman?


slide

Enter algorithmic cubism


slide

These are categories Google has assigned to me to serve me personalized ads.


slide

Some of them makes sense like Parenting, Home & Garden, Cooking & Recipes.


slide

But Google also thinks I'm interested in Rugby. I've nerve played rugby, i've never been to a rugby game and I don't even know the rules. But I happend to live in Twickenham which is home to biggest rugby stadium in the UK, hence according to Google I must love Rugby...


slide

“Our datafied lives, when aggregated and transcoded into quotation-marked categories, increasingly define who we are and who we can be.” - by John Cheney-Lippold / We Are Data


slide

Here is a 3d scan of me from 2010. A digital representation of myself. I had beard and was single. Nowadays I’m married but that beard still regrows from time to time before deadlines. This is me how I perceive myself: the same geeky guy for years, nickname VORG, reading about computer graphics, listening to techno, and still pulling out all nighters on a weekly basis.


slide

If you look at my lunch and leisure time browsing and compare that with publicly available data you might see completely average polish immigrant living in London.


slide

Male 25-35. Frantically refreshing The Guardian and BBC websites scavenging for newest information about Brexit.


slide

During the day though I’m more “real” me. A 30+ developer, company owner. Perhaps 75% polish or 10% danish as I lived in Denmark for 5 years before and still have many friends on Facebook there.


slide

During the day though I’m more “real” me. A 30+ developer, company owner. Perhaps 75% polish or 25% danish as I lived in Denmark for 5 years before and still have many friends on Facebook there.


slide

My geolocation data is confusing though. I live in UK, do most of my projects for US clients, I use ‘z’ insted of ’s’ when I spell “data visualization” (which drops me to the second page of Google results btw).


slide

Maybe I’m 40% American then?


slide

Some website seem to disagree. Where would all those ads of prams, nursing bras, baby photographers?


slide

Maybe I'm 50% korean woman then?


slide

“As subjects to algorithmic interpretation, who we are also changes “from point to point,” moving from ‘woman’ to ‘man’ or ‘old’ to ‘young.’”


slide

Here are initial explorations how the transitions between different identities could look like.


slide

There is no data plugged in yet but I want to classify website I'm visiting and assign them to different profiles to speculate how those identities could be shaped.


slide

In a way we are made of data.


slide


slide


slide


slide


slide

Who am I?


slide

Am I all those people or not of them? These are 3d scans of people I will never meet. Approximations of my online identities that I’ll never see.


slide

Maybe it’s like cubism with multiple contradicting perspectives embedded within one picture. Data visualization and data art can become a new lens onto the world allowing us to see things differently and from many points of view at the same time.


slide

I call this project you are I/O. Because we are at the center of this data infrastructure that’s both fed by our actions and influences what we can do.


slide

We need tools to reverse engineer the data reality around use

“maybe if you see art as a reverse engineering problem it becomes more interesting” - Mario Klingemann


slide

Building alternative representations of the underlying systems and our data identity


slide

So we can move between different levels of abstraction and draw our own bull.


slide

Thank You