Teaching a computer to draw

There must be hundreds of artists trying to mix art and computing. I have always wanted to try to perform some experiments on this topic. It is interesting computationally speaking (there are tons of math and engineering involved), visually (computed generated art is usually impressive) and conceptually (is it actually art? how we define art?).

This @anastasiaopara’s tweet has inspired me to write this post. It is a project to make computer draw images. She has published her code so that we all can play around with it. I wanted to understand how it works and try my own ideas, so that I ended forking and refactoring her code https://github.com/sgmoratilla/genetic-drawing.

This is not an essay about genetic algorithms or function optimizations. There is a lot to know about these subjects. You can find several introductions to genetic algorithms on the Internet. If you want my advice, read some of them and jump to scholar papers. For example, this thesis of my former investigation partner Rubén (backup here). Part I is about Optimization, Section 2.3.2 about Genetic Algorithms.

Anyway, if you don’t understand something about what I wrote here, feel free to ask: @sgmoratilla

##What are genetic algorithms? To be brief, optimizations are search problems. You have a space of possible solutions, a function whose maximum (or minimum) you want to know. Sometimes you are lucky and your target function has nice properties as being differentiable and smooth, so that you can use some traditional approaches as gradient descent or Newton’s methods. The most interesting problems I’ve worked on weren’t differentiable or their solutions must have to fulfill some restrictions. That’s were genetic and evolutionary (they are not the same) algorithms shine.

##Can a computer paint?

So.. let’s say we want to solve the “problem of painting”.

What is a painting?
It is a succession of brush strokes on a canvas.

How can we model that into a computer?
Let’s say a stroke starts are some point of the canvas (x,y) and goes in some direction.
Painters are able to make a infinite number of strokes depending on how they spin their hands or how much pressure they apply. We could probably model that too, but keep it simple and let’s say that we give to the computer a list of brushes that it can use (brushes as in Photoshop). Painters mix different colors as well. We could model that too, but we are going to simplify assuming only black and white paintings.

In summary, we can model a stroke as:

• 2 integer numbers x,y that represents where the stroke starts.
• 1 integer number that represent which brush to use.
• 1 float number that represent the rotation of tha brush.
• 1 float number that represent the size of the brush in % of the original brush.
• 1 integer number that represent the color (0 = black, 255 = white, any number in between is a scale of grey).

We can call that set of 6 numbers a gene. A painting is just a collection of these genes applied in order. For example, let’s say we can draw a painting with 1000 strokes. A solution is a collection of 1000 genes, that is, a collection of 6000 numbers. This is called the genotype in Genetic Algorithms frameworks. Isn’t it magical how can we code the process of painting with these simple numbers?

How good is this solution of 6000 numbers? How can we measure it? We can apply these strokes on an empty canvas. This is our painting. The fenotype. Then we subtract the original image from this painting (pixel by pixel). The sum of the differences represents how different it is from the original one.

Our aim is to minimize that number. How can we do that using a genetic algorithm?

1. Generate a random population of possible solutions.
2. Create a children population by mixing those candidates.
3. Alter randomly some of the genes.
4. Evaluate how good these solutions are.
5. Discard the worse ones and keep the best ones.
6. Repeat several times.

At some point, we stop and choose the best solution. That’s our final painting.

This is how it works on one photo of my own:

##But… is it actually learning?

Sorry for the catchy headline. We are not actually teaching a computer to draw. This algorithm does not learn anything O:) If you think of the algorithm, we have actually programmed how to paint by trial and error given some image as inspiration.

There are thousands of possible optimizations. For example, use fewer strokes (faster iterations) and resetting the algorithm using the last image as seed of the next iteration (a greedy approach).
We are not discussing them here, you can see some of them in my repo.

I know there are several algorithms to transfer style. You can teach a neural network to copy the style of a painter (given a list of painting with the same style) and transfer that style to a image or photo. In my opinion and from an artistic/romantic point of view, I feel that is not painting but applying a filter to the image.

I am not promissing anything but I have plans to write more about this subject soon.

##Links Github Project
Anastasia’s Github