Pythagoras Bitmap Mode: How It's Done

Introduction

Bitmap mode involves shading in the color of images, as opposed to just making the outlines. For some images, this makes vastly superior output compared to outlines. Faces are a prime example. When outlined, edges of shadows appear the same as edges of objects, causing a mess of lines to be drawn on a face. With shading, shadows and lower contrast features remain in the image, improving the familiarity of the face.

Its my face!

The downside is that the output resolution is greatly reduced, and the time it takes to draw is greatly increased.

The Concept

How do I generate levels of grey scale when my pen only creates a line of constant darkness? The answer: squiggles. Each pixel forms a 2D square on the paper. The more that square is shaded in by the pen, the darker the pixel. This idea can be clearly seen in one of the earlier test pictures. Ramp function patterns are drawn on the paper to fill the square. Then I tested with square functions, then triangle waves.

Since the shading depends on the width of the pen stroke, the pixel size on paper becomes a factor. Smaller pixels make smoother borders around regions on the output image, but also decreases the number of squiggles that can be fit into a pixel before the pixel becomes solid dark. So at one extreme, the pixel is completely filled by a single stroke of the pen, while at the other extreme, the entire image is a single pixel.
Through testing, I found the best compromise between greyscale levels and output resolution is to have an image width of between 120 and 160 pixels, for a 7.5 inch (19 cm) square drawing area.

Tweaking Waveforms

First was experimenting with different waveforms. After creating ramp functions, I tried to do square waves. However, this created odd behavior whenever the squares shifted in and out of phase with the rows above and below them. It was quickly abandoned, apologies for the lack of pictures. What I settled on was triangle waves, which were always in phase like the ramp functions, yet more uniform like square waves, closer to the ideal of sine waves.

In an attempt to squeeze out more greyscale levels, I considered sharing squiggles across multiple pixels, up to four pixels. This way I could get even less dense pen strokes and therefore lighter colored intensities. The results were not so great though. This image allowed up to four combined pixels.

The pixel combining can be observed near the center of the drawing.

The pixel combining created awkwardly shaded areas in an image. Even when combining less pixels.

A series of test drawings under various brightness settings. Scribbling in the corner is unrelated.

Pixel combining didn't create evenly lighter areas, just awkward gaps. The solution to this is to use amplitude modulation. To make a lighter region, the amplitude of the squiggle is reduced, instead of the number of squiggles. This can be seen clearly below.

Just below the center is an example of amplitude modulation of the squiggle.

This fills the gap in shading between one squiggle per pixel and no squiggle, making nicer transitions for lighter colors.

Color Mapping

Originally, color was mapped linearly to number of squiggles in each pixel. This caused for pretty dark images, as seen below.

Can barely make out differences in the greys.

The first step to fixing this is to add gamma correction. Since human eyes are more sensitive to darker colors than lighter colors, color values in images are shifted toward the darker regions according to an exponential rule. This way, more bits are devoted toward the darker colors, instead of highlights that eyes cannot differentiate anyways. 

To correct for gamma, simply raise the pixel value, mapped from 0 to 1, to the (1/2.2) power to get the actual color intensity. This has an effect of making the output image brighter. Technically, any image processing should be done after gamma correction, for proper mixing of pixel values. However, this will involve a lot of going between gamma encoded and decoded, so I let it slide. I do scaling before gamma correction, and gamma correct once, right before converting the pixel value to squiggles. This makes the picture a little brighter.

The same picture, with gamma correction. A little brighter.

The next step in color mapping is to fix the mapping of intensity to squiggles. The mapping is not linear, and is depending on the size of the pixel. The mapping I got from experimenting is intensities 0.7 and less are linearly mapped to squiggle density. Between 0.85 and 0.7 intensity is the region for amplitude modulation of the squiggles. Testing was done with printing gradients and comparing the result to the computer image.


Software

The image processing software was written in Python, using the OpenCV libraries for image processing and simple GUIs. One slider lets the user change the output resolution until the aliasing looks alright, and a second slider lets the user change the gamma value from the default of 2.2, to change the brightness of an image.

There is a subtlety in the image preview. While the screen can display the full 256 values of greyscale, the robot can only print a much smaller range of greyscale, which varies as a function of pixel size. This can be especially misleading for lighter colored regions, since full white pixels cover color intensities of 0.85 and 0.7, and for darker colored regions, where seemingly differently colored regions are both shaded the same.

To fix this, the output pixel values are grouped into buckets based on the number of squiggles in that pixel. This way, the output image has the same number of greyscale levels as the printed image. With some funky mapping, the colors on screen are closer to the colors as printed. This way, it is much easier to adjust the brightness and scaling and predict the actual results. Following are screenshots of the program in action after bucketing. You can see how there are significantly less than 256 levels of grey.

Default scaling of 120 pixels and gamma of 2.2 (220 / 100).

Darker image.

Brighter image.

Appendix

• Code to convert image to a list of squiggles for the robot: https://github.com/aaronbot3000/deltadraw/blob/master/vectorizer/rasterize.py
• Web album of Pokemon drawn by Pythagoras: https://picasaweb.google.com/aaron.y.fan/PythagorasPokemon?authuser=0&feat=directlink

Pythagoras: Vectorizing Software

Having to program points in the robot is annoying at best and horribly inelegant and time consuming. What's better is the computer generating all these fine points, so all I have to do is say, "Robot! Draw!" and it does.

This program vectorizes raster images, but doesn't quite process (read: not at all) vector images, such as SVG or DXF. That is for a later project.

OpenCV

OpenCV is a really dandy computer vision library originally headed by Intel. It has since been open sourced and freely available to the public in C/C++, Python, Android, GPUs, and a work-in-progress iOS port. The library implements just about every computer vision algorithm you can think of. For the vectorizer, I use the Canny edge detection, contour finding, and polygonal approximation of curves.

Take Note

This guide is written assuming you know how to fiddle with OpenCV and program. If not, there are many other websites that teach you how to program and beginner's tutorials to OpenCV that do a better job than I would have.

TL; WR (Too Long, Won't Read)

Resize the image to around 800 pixels, apply Gaussian blur and Canny Edge Detection. Fiddle with thresholds and blurs until the edge image looks right. Run findContours, then approxPolyDP (polygon approximation). Increase approximation accuracy until output looks about right. Stream polygon points to robot over serial or your favorite communication protocol.

Links to source code in the appendix.

Start: Acquire Image and Resize

Why resize your source image? After all, if its big, then more detail, great. If its small, then process faster, great.

If the source image is too big, say greater than 1200 pixels across, then there is too much detail. Since the image processing algorithms are run only once, the speed of processing at is no issue, but the delta robot can only plot so fast. At larger sizes, very small, insignificant features will show up, creating a large number of points to plot, slowing down the drawing and cluttering the image. Of course, unless you want that kind of precision for a very feature full image.

If the source image is too small, what ends up happening is basically plotting at low resolution. The manipulator positions are quantized based on the resolution of the input image. Low res input, low res output. So even if the input image is small, resizing it to a larger image makes better results.

I have found the optimal image size of lower precision delta robots (such as version 1) to be around 400 pixels, and for higher precision delta robots (such as version 2) to be around 800 to 1200 pixels, depending on the input image.

Finally, make it grayscale, either by loading the image greyscale or by using cvtColor to pull out the green channel. CV algorithms love grayscale images. This is the source image I'm using.

yay Lena

Next: Blur and Edge Detect

Exactly as it says, run Gaussian blur and then Canny edge detection on the image, to get a binary image of pure, one pixel width lines representing all the detected edges. Why blur? Because Canny edge detection will find any and all lines in the image, regardless of the content or "importance" of the line. Without blur, the edge image looks like this.

Too much blur, and you lose too much data, and get an edge image like this.

A happy medium is nice, though it differs depending on the picture. For this image I used a seven pixel radius (not sure what the sigma is though, sorry) blur to get this.

Canny edge detection also has variables to adjust the hysteresis of the found lines, a high and low threshold. High threshold is the minimum "strength" of the edge for it to appear in the edge image. Low threshold is the minimum "strength" of the edge for it to continue being a line in the edge image. Drop below the low threshold, and the line stops in the edge image.

Finally: Contour Finding and Polygon Approximation

Currently, all the edges are just white pixels on a black image. They need to be grouped into lines. This is where the function FindContours comes in. It goes and finds all the groups of pixels that can become contours and makes them into unified contours. It has a bunch of fine ways to store the contours, such as in a hierarchy, showing which contours are inside other contours, or only finding outlines of things. In this case we want all the contours in any order, so we can pass it CV_RETR_LIST to get a list of contours.

We aren't done yet. Contours aren't nice straight lines that can be linearly approximated. The last step is ApproxPoly, that is, polygonal approximation. Give it the list of contours, and it will give you back a list of polygons, each a list of points. Exactly what the robot needs. There is one adjustable value: the precision of approximation. Low approximation gives very "shapy" images, such as below.

High approximation approaches an exact match to the edge image, but it can add many many extra points in the polygons. For example, the below image contains 23,000 points, while for a typical drawing I adjust for no more than 3000 points.

With some adjustment, I achieved a 2115 point image while still remaining quite faithful to the original edge image.

So now we have all these points, it is just a simple matter of mapping pixel values to real world inches (or {milli,centi,}meters). Of course, make sure you maintain aspect ratio, or you will get stretched out images as such.

One more thing to consider. If you flipped your coordinate axes like I did, with the Z axis pointing down, your images are mirrored about the Y axis. Simply flip back.

One more thing

Since the robot interpolates between points, extra points will need to be added to be sent. Traverse points. One to traverse to the start of the polygon, and one to raise the pen to prepare for traversing to the next polygon.

In the next post

Getting the data to the robot.

Appendix

Source image Lena: http://sipi.usc.edu/database/database.php?volume=misc&image=12#top

Implemented vectorizer code in Python (warning: dirty, dirty, hacked together code): https://github.com/aaronbot3000/deltadraw/tree/master/vectorizer