Grep and Sed, Demystified

tutorial

I’ve kind of half understood grep for a while, but assumed that I didn’t really get it at all. I thought I knew nothing at all about sed. I took some time this weekend to sit down and actually learn about these two commands and discovered I already knew a good deal about both of them and filled in some of what I didn’t know pretty easily. Both are a lot more simple and straightforward than I thought they were.

Grep

grep comes from “global regular expression print”. This is not really an acronym, but comes from the old time ed line editor tool. In that tool, if you wanted to globally search a file you were editing and print the lines that matched, you’d type g/re/p (where re is the regular expression you are using to search. The functionality got pulled out of ed and made into a standalone tool, grep.

Basics

grep, in its simplest use, searches all the lines of a given file or files and prints all the lines that match a regular expression. Syntax:

grep <options> <expression> <file(s)>

So if you want to search the file animals.txt for all the lines that contain the word dog, you just type:

grep dog animals.txt

It’s usually suggested that you include the expression in single quotes. This prevents a lot of potential problems, such as misinterpretations of spaces or unintentional expansion:

grep 'flying squirrel' animals.txt

It’s also a good idea to explicitly use the -e flag before the expression. This explicitly tells grep that the thing coming next is the expression. Say you had a file that was list of items each preceded with a dash, and you wanted to search for -dog

grep '-dog' animals.txt

Even with the quotes, grep will try to parse -dog as a command line flag. This handles it:

grep -e '-dog' animals.txt

You can search multiple files at the same time with wildcards:

grep -e 'dog' *

This will find all the lines that contain dog in any file in the current directory.

You can also recurse directories using the -r (or -R) flag:

grep -r -e 'dog' *

You can combine flags, but make sure that e is the last one before the expression:

grep -re 'dog' *

A very common use of grep is to pipe the output of one command into grep.

cat animals.txt | grep -e 'dog'

This simple example is exactly the same as just using grep with the file name, so is an unnecessary use of cat, but if you have some other command that generates a bunch of text, this is very useful.

grep simply outputs its results to stdout – the terminal. You could pipe that into another command or save it to a new file…

grep -e 'dog' animals.txt > dogs.txt

Extended

When you get into more complex regular expressions, you’ll need to start escaping the special characters you use to construct them, like parentheses and brackets:

grep -e '\(flying \)\?squirrel' animals.txt

This can quickly become a pain. Time for extended regular expressions, using the -E flag:

grep -Ee '(flying )?squirrel' animals.txt

Much easier. Note that -E had nothing at all to do with -e. That confused me earlier. You should use both in this case. You may have heard of the tool egrep. This is simply grep -E. In some systems egrep is literally a shell script that calls grep -E. In others it’s a separate executable, but it’s just grep -E under the hood.

egrep -e '(flying )?squirrel' animals.txt

Other Stuff

The above covers most of what you need to know to use basic grep. There are some other useful flags you can check into as well:

-o prints only the text that matches the expression, instead of the whole line

-h suppresses the file name from printing

-n prints the line number of each printed line.

Simple Text Search

If the thing you are searching for is simple text, you can use grep -F or fgrep. All the same, but you can’t use regular expressions, just search for a simple string.

grep -Fe 'dog' animals.text
fgrep -e 'dog' animals.text

Perl

There’s also grep with Perl syntax for regular expressions. This is a lot more powerful than normal grep regex syntax, but a lot more complex, so only use it if you really need it. It’s also not supported on every system. To use it, use the -P flag.

grep -Pe 'dog' animals.text

In this simple case, using Perl syntax gives us nothing beyond the usual syntax. Also note that pgrep is NOT an alternate form of grep -P. So much for consistency.

Sed

I thought that I knew next to nothing about sed, but it turns out that I’ve been using it for a few years for text replacement in vim! sed stands for “stream editor” and also hails from the ed line editor program. The syntax is:

sed <options> command <file(s)>

The two main options you’ll use most of the time are -e and -E which work the same way they do in grep.

The most common use of sed is to replace text in files. There are other uses which can edit the files in other ways, but I’ll stick to the basic replacement use case.

Like grep, sed reads each line of text in a file or files and looks for a match. It then performs a replacement on the matched text and prints out the resulting lines. The expression to use for replacement is

s/x/y/

where x is the text you are looking for, and y is what to replace it with. So to replace all instances of cat with the word feline in animals.txt

sed -e 's/cat/feline/' animals.txt

Note that sed will print every line of text in the file, whether or not it found a match or not. But the lines that it matched will be changed the way you specified.

After the final slash in the expression, you can add other regex flags like g for global or i for case insensitivity.

Like grep, sed just outputs to stdout. You can redirect that to another file using > or pipe it to another process using |. But do NOT save the output back to the original file. Try it out some time on a test file that you don’t care about and see what happens.

There are lots of other options you can use with sed but the above will probably get you by for a while to come. As you need more, just read up.

Summary

The biggest thing I took away from the couple of hours I spent on this was actually how easy these two commands were to learn. I’d been avoiding doing so for a long time and now wish that I had spend the effort on this much earlier.

New Shell Script Shortcut

tutorial

I’m often making shell scripts for various things – sometimes just a quick one for a specific task, sometimes just to test something out, sometimes for some kind of workflow task that I plan to keep around.

It’s always the same steps:

  1. Create the file.
  2. Add the header: #! /bin/bash
  3. Write the code.
  4. Save it.
  5. Exit the editor.
  6. Make it executable with chmod +x <filename>
  7. Run, test, edit, etc.

When you’re repeating yourself, time for some automation. So I wrote a shell script that creates shell scripts.

#! /bin/bash
if [ -f "$1" ]
then
  echo "$1 already exists"
exit 1
fi
echo '#! /bin/bash' > $1
chmod +x $1
$EDITOR $1

Create a file, add that code, make it executable (sound familiar?) and put it somewhere in your path. I named mine newss for new shell script.

Now you can just say:

newss test.sh

And you are editing an executable shell script. As you can see, it checks to see if the file exists as a minimum level of safety. It will follow any paths, such as:

newss foo/bar/test.sh

But only if the directories exist. Creating non-existent directories wouldn’t be too hard if that’s important to you.

Animated Sinusoidal Cardiods

tutorial

I think I just made up a thing. Usually when I think that, it just means fewer than a few hundred people have thought about it before me, so who knows.

Let’s start with cardioids. A cardioid is a heart-shaped curve. One way to create a cardiod is to roll a circle around another circle of the same size, tracing the path of a single point on the moving circle. Like so:

https://commons.wikimedia.org/wiki/File:Cardiod_animation.gif

I discovered another neat way to create a cardioid while checking out the math art challenge.

#MathArtChallenge Day 7: Cardioids!

In this method, you divide a circle into an arbitrary number of points around the radius. Then, for each point n, you draw a line from point n to point n*2. Point 1 to 2, point 2 to 4, point 3 to 6, etc.

Here’s some JavaScript/Canvas code showing this in action:

context.translate(400, 400);
const radius = 350;
const res = 100;
const slice = Math.PI * 2 / res;
const mult = 2;

context.beginPath();
for(let i = 1; i < res; i++) {
let a1 = slice * i;
let a2 = slice * i * mult;
context.moveTo(Math.cos(a1) * radius, Math.sin(a1) * radius);
context.lineTo(Math.cos(a2) * radius, Math.sin(a2) * radius);
}
context.stroke();

And here’s what that gives you:

You’ll notice in the code that I’ve divided the circle into 100 points, which is is rather low-res. If I up that to 360, we get something nicer:

So I’m calculating the points by getting two angles, a1 and a2, calculating those with slice * i and slice * i * mult as described above. slice being a full circle divided by res and mult equals 2 for now.

   let a1 = slice * i;
let a2 = slice * i * mult;

What if I change that to mult = 3 instead?

Or, mult = 4 ?

You see that for each multiplier, m, you get m-1 nodes in the cardioid. Let’s just go crazy and see what happens when we set res to 1440 points and mult to 25:

Here’s the code for those of you following along at home:

context.translate(400, 400);
const radius = 350;
const res = 1440;
const slice = Math.PI * 2 / res;
const mult = 25;

context.lineWidth = 0.25;
context.beginPath();
for(let i = 1; i < res; i++) {
let a1 = slice * i;
let a2 = slice * i * mult;
context.moveTo(Math.cos(a1) * radius, Math.sin(a1) * radius);
context.lineTo(Math.cos(a2) * radius, Math.sin(a2) * radius);
}
context.stroke();

All very interesting, but I wanted to start changing things up even more. I decided that rather than using a simple circle, what if I varied the radius of the circle with a sine wave? Here’s the code I came up with:

context.translate(400, 400);
const radius = 300;
const res = 1440;
const slice = Math.PI * 2 / res;
const mult = 5;
const waves = 6;

context.lineWidth = 0.25;
context.beginPath();
for(let i = 1; i < res; i++) {
let a1 = slice * i;
let a2 = slice * i * mult;
let r1 = radius + Math.sin(a1 * waves) * 100;
let r2 = radius + Math.sin(a2 * waves) * 100;
context.moveTo(Math.cos(a1) * r1, Math.sin(a1) * r1);
context.lineTo(Math.cos(a2) * r2, Math.sin(a2) * r2);
}
context.stroke();

First I created a waves constant that controls how many sine waves will go around the circle. Then an r1 variable that is based on radius, multiplied by the sine of a1 * waves times 100. And an r2 variable based on a2 * waves. So for each point, it’s radius will get larger and smaller as they progress around the circle. The result (setting mult back to 5):

You can get all kinds of interesting shapes by varying how many nodes and how many waves and the size of the waves and the resolution.

Of course, I had to have a go at animating these. The first idea was to vary the height of that radial wave. Here, it’s going back and forth from -80 to +80:

I’m not going to give the source for the animation examples, because it was written in another system entirely, but if you’ve followed along so far, you’ll be able to figure it out.

Next, I thought about varying the phase of that radial wave, so that the wave itself seemed to be animating around in a circle. This produced some really striking animations. I’ll close the article by posting animations for 2, 3, 4, 5 and 6 wave animated sinosoidal cardiodids. Enjoy!

Perlinized Hexagons

tutorial

In my last post, Hexagons, I talked a bit about how cool hexagons are, and gave one method of creating a hex grid. I also used this image as a header:

This was a bit of a tease, as I didn’t show how to create such an image. But I promised I’d come back around and give some code for it, so here we go.

First, some refactoring

As I’m sure most of you guessed, this image uses Perlin noise, but you could use any other function to get different textures. Check out my articles on flow fields (parts one and two) for more ideas along these lines. I’ll also be using the same Perlin noise library mentioned in the second of those articles.

I’m going to take the basic code from the tiled hexagons in the last article and change it up a bit. Mainly all I’m doing here is removing the translation code and calculating each x, y drawing point directly. Here’s what that looks like:

const canvas = document.getElementById("canvas");
const context = canvas.getContext("2d");
context.lineWidth = 0.5;

function hexagon(x, y, r) {
  let angle = 0;
  for(let i = 0; i < 6; i ++) {
    context.lineTo(x + Math.cos(angle) * r, y + Math.sin(angle) * r);
    angle += Math.PI / 3;
  }
  context.closePath();
}

const radius = 10;
const ydelta = Math.sin(Math.PI / 3) * radius;
let even = true;

for(let y = 0; y < 900; y += ydelta) {
  let offset = 0;
  let yy = y;
  if(even) {
    offset = radius * 1.5;
  }
  for(let x = 0; x < 900; x += radius * 3) {
    context.beginPath();
    hexagon(x + offset, y, radius);
    context.stroke();
  }
  even = !even;
}

Here’s what that gives us. Notice that I made the radius much smaller and stroked it instead of filling it.

Again, this all builds off of the last article. I had to remove the translation code because I need the exact x, y coordinates of every point of every hexagon, so I can feed that into a Perlin noise function to know how much to shift it.

Adding some noise

The Perlin function is going to make use of this library: https://github.com/josephg/noisejs. Feel free to add that to your project however you’d like. Or add a different noise function. It doesn’t really matter which one you use.

I need a function that’s going to take an x, y coordinate, calculate an offset based on Perlin noise, and return a new, shifted x, y coordinate. Since JavaScript doesn’t let us return multiple values, I’ll return and object with x, y properties. Here’s the function:

function perlinize(x, y) {
  const scale = 0.01;
  const strength = 20;
  const angle = noise.perlin2(x * scale, y * scale) * Math.PI;
  return {
    x: x + Math.cos(angle) * strength,
    y: y + Math.sin(angle) * strength,
  }
}

To revisit how a Perlin flow field works, we’re going to use 2-dimensional Perlin noise, feeding it an x, y value. Since the numbers we’re using will be in the hundreds or even thousands for pixel values, we’ll scale that down a bit with a scale value. This function returns a value from -1 to +1. Other Perlin noise functions sometimes return values from 0 to 1. Either will work just fine, but might need some tweaking. I take that value and multiply it by PI, which will give me a number from -PI to +PI. We’ll call that an angle in radians. In degrees, it would be -180 to +180. And we’ll use the sine and cosine of that angle to create an offset based with a customizable strength. Then, we return the original x, y coordinate + that x, y offset.

Now we just need to alter our code to use the perlinize function. I’ll just show the relevant function:

function hexagon(x, y, r) {
  let angle = 0;
  for(let i = 0; i < 6; i ++) {
    const p = perlinize(x + Math.cos(angle) * r, y + Math.sin(angle) * r);
    context.lineTo(p.x, p.y);
    angle += Math.PI / 3;
  }
  context.closePath();
}

And this is where this gets us.

Each hexagon still perfectly tiles because the Perlinized offset for each shared point of each hexagon should be exactly the same, at least down to an adequate degree of precision.

Shading

Now let’s add some shading. Basically we just want a grayscale value for each hexagon. To enhance the effect, the shading should be coordinated with the offset flow field values, so we’ll use the same Perlin noise settings. Here’s a function that will accomplish that:

function getShading(x, y) {
  const scale = 0.01;
  const value = (noise.perlin2(x * scale, y * scale) + 1) / 2;
  const shade = 255 - value * 255;
  return "rgb(" + shade + "," + shade + "," + shade + ")";
}

This takes an x, y coordinate and returns and rgb color string. Remember that this particular noise function returns -1 to +1. I’ll add 1 to that to have a range of 0 to 2, then divide by 2 to get 0 to 1. Then I’ll calculate a shade value by subtracting that normalized value by 255 and subtracting from 255. A bit convoluted, I know. This is all made a lot easier if you have a map range function. It would look something like this:

const n = noise.perlin2(x * scale, y * scale);
const shade = map(n, -1, 1, 255, 0);

To see how a map function would work, check out this video.

Finally, we just need to alter our drawing code to set the fill color based on the x, y of each hexagon, and do a fill before the stroke:

  for(let x = 0; x < 900; x += radius * 3) {
    context.beginPath();
    hexagon(x + offset, y, radius);
    context.fillStyle = getShading(x + offset, y);
    context.fill();
    context.stroke();
  }

And here is the result:

Summary

Don’t stop here. There are all kinds of variables and different rendering techniques to experiment with here. This is all just about taking two different concepts – hex grids and Perlin noise – and combining them in a creative way. This worked out pretty well, but there are an infinite other combinations waiting for you to discover.

Late breaking comment…

One thing I had in writing this but totally spaced out on while writing the article is that this code is extremely unoptimized. Every single x, y point here is calculated and then “perlinized” three times. OK, some of the ones on the edges only get calculated only once or twice, but you get the point.

Strategies for optimizing would probably involve creating a single point grid one time and then using a loop to draw using the pre-calculated points in that grid.

This is left as an exercise for the reader.™

Hexagons

tutorial

I know what you’re thinking. I’m thinking it too. So let’s stop tiptoeing around the subject and just say it out loud:

Hexagons are freaking cool.

Oh… you weren’t thinking that? Oops. My bad. I just assumed. Must be just me. So let me state my case on why hexagons are so damn neat.

First, let’s get some definitions out of the way. A hexagon is a six-sided polygon. And just to be clear, the hexagon I’m talking about is the regular, convex polygon type of hexagon. Which is to say that all of its sides are the same length and all of its angles are the same (120 degrees).

Cool things about hexagons

  1. A hexagon is made up of six equilateral triangles. Each triangle is composed of three 60-degree angles, and three sides that are the same length, which is the radius of the hexagon.
  2. This means that each side of the hexagon is the same length as its radius.
  3. Hexagons are one of the three regular tessellations. That means you can take a bunch of hexagons and pack them together perfectly with no space left over. The only other two regular polygons you can do that with are squares and triangles. And in a sense a hexagon tiling and triangle tiling wind up being exactly the same thing. See #1 above.
  4. Hexagons appear pretty regularly in nature. The bounds of any snowflake, the cells of a beehive, basalt columns, the compound eyes of insects, the packing of bubbles, the cloud formations on the north pole of Saturn.

    By NASA/JPL-Caltech/Space Science Institute – http://www.nasa.gov/sites/default/files/pia18274_full.jpg, Public Domain, https://commons.wikimedia.org/w/index.php?curid=33952464

  5. When you tile hexagons, exactly six hexagons will surround every other hexagon. If you connect the center points of the six surrounding hexagons, you get another, larger hexagon.
  6. If you connect every other point of a hexagon to its center, you have a perfect isometric cube.

All of these points, and more, make hexagons a great shape to play with if you’re doing any kind of creative coding.

So, you want to draw a hexagon…

Now you’re saying, “Gosh, Keith. These hexagons sound wicked pissah! How can I draw one???” Well, let me tell you.

You need three parameters, the x, y position and a radius. You can also throw in a rotation parameter, but I’ll start without that. I’m going to do this in JavaScript, and assume there’s an 800×800 canvas element on the page with an id of “canvas”. You should be able to easily convert this code to whatever language you are working with.

const canvas = document.getElementById("canvas");
const context = canvas.getContext("2d");

function hexagon(x, y, r) {
  context.save();
  context.translate(x, y);
  let angle = 0;
  for(let i = 0; i < 6; i ++) {
    context.lineTo(Math.cos(angle) * r, Math.sin(angle) * r);
    angle += Math.PI / 3; // 60 degrees
  }
  context.closePath();
  context.restore();
}

context.beginPath();
hexagon(400, 400, 100);
context.stroke();


Run that, and we get a very nice hexagon. You can fill it instead of stroke it, make a bunch at different positions and of different sizes and colored differently. I’m not going to dive into the code there much, as it’s pretty simple.

Note, that I’m keeping it super simple here. You can optimize this, you can rework this, make it more reusable, make it draw regular polygons of any number of sides, pass in a rotation value and set angle to that instead of zero, and on and on. Maybe you want to pass in the context, or make the function a method of the context object. And you should do all of that. Or some of that. Or however much of that you want to do. But I’m in charge today, and we’re doing hexagons. And we’re doing them simple.

What about that tessellation stuff?

Yeah, now let’s make a bunch of hexagons and stick them together perfectly to see them tile. With squares, this is really easy. Loop through on the y and x axes and draw a square at each point. With hexagons, it’s a bit trickier. There are certainly multiple ways to go about this. I’ll give you one method, but if you come up with something that fits your needs better, go for it.

First of all, note that with the hexagon we drew above, it’s a bit wider than it is tall. Here, I’ve draw lines from the center to each point of the hexagon.

We can see that the width is two times the radius. Simple. But the height is a bit less. Knowing a bit of trigonometry, we can figure out that the distance from the center to the top of the hexagon is the sine of 60 degrees times the radius. So the height of the hexagon is twice that.

Here’s a hex grid.

The main thing to note is that unlike a grid of squares or rectangles, each alternate row or column overlaps with the previous and next row or column. If you consider we have even rows and odd rows, the even rows are horizontally offset from the odd rows by 1.5 times the radius. To see that, consider the horizontal distance from the center of one hexagon to the center of one of the hexagons immediately to its left or right, but in an alternate row.

Within each row, the center-to-center horizontal distance between each hexagon is 3 times the radius. To see that, consider the distance from the center of one hexagon to the center of the hexagon directly to its left or right in the same row.

And vertically, each row is offset from the previous row by one half a hexagon height, or sin(60) * radius.

Knowing all of that, we can put together a double for loop that draws a grid.

const radius = 50;
const ydelta = Math.sin(Math.PI / 3) * radius;
let even = true;

for(let y = 0; y < 900; y += ydelta) {
  context.save();
  if(even) {
    context.translate(radius * 1.5, 0);
  }
  for(let x = 0; x < 900; x += radius * 3) {
    context.beginPath();
    hexagon(x, y, radius);
    context.stroke();
  }
  context.restore();
  even = !even;
}

I’ll walk through this code quickly. First we set the radius to whatever. 50 in this case. Then we calculate how far apart to place each row. We said that was sin(60) * radius. 60 degrees is PI / 3 radians. And I’ll set an even variable to true to know whether we are on an even or odd row.

Then we create a double for loop on y and x. I have an 800 pixel canvas, but I looped through to 900 on each axis so I didn’t wind up with any empty space on the right or bottom edges. The y loop increments by ydelta and the x loop increments by radius * 3, all as discussed above.

If we’re on an even row, I’ll translate the canvas by radius * 1.5. Note that I did a context.save() before that, and follow up with a context.restore() at the end of the loop.

Inside the inner x loop, I just draw the hexagon. And at the very end of the loop I set even to !even to make the next row be odd, meaning it will not get that horizontal translation applied.

And now you can make a hex grid. Of course, you can fill each hexagon instead of stroking it. If you used different colors, that works out well, but if you are using all the same color, they’re all going to blend in together. A trick to handle that is to reduce the radius you pass into the hexagon function like so:

context.beginPath();
hexagon(x, y, radius - 5);
context.fill();

This gives you a 10 pixel border between each hexagon (5 pixels on each side).

Another strategy for filling a space with hexagons is to start with a single hexagon at the center and draw six new hexagons around that one and continue out in rings from there.

Hexagons are also a common tiling pattern in various strategy games. In that case, you need a bit more logic to figure out which six tiles are adjacent to any given tile and how to move from one to the next. Beyond the scope of this tutorial.

But what about that cool image at the top with the warped, 3d-looking hexagons?

Yeah, that one is pretty neat, isn’t it? Try it for yourself. I’ll write up another article on how I did that soon.

Intro to Cairo Graphics in Rust

tutorial

In the past few weeks, I’ve been itching to learn some new stuff. Those who know me even just a little bit know that “new stuff” is generally going include “new ways of creating graphics and/or animations”. And so it does this time.

I’ve been creating graphics in HTML5 Canvas for quite a few years now, and that’s cool. But there are a few pain points with its workflow:

1. It sometimes seems silly to pipe everything through the browser all the time. I’d like to just be able to run a program and have it create a file and display that file.

2. Running in the browser via the file system introduces various security issues, so you need to run some kind of local server, and even then you run into issues with saving files to disk automatically, so you need to right click and save or set up some kind of button to save an image.

3. At work, I’m using React with Webpack and Babel, so I get to use the latest and greatest ECMAScript with classes and all the other syntactic sugar. It’s really nice, but setting all of that up is not trivial. For a one-time large project, it’s not a huge deal to spend some time on creating all that config. But if you’re going to be doing random experiments all the time, it’s a ton of overhead. I get stuck with whatever the current browsers implement.

So I’ve been looking for some other language / drawing api I could learn.

After a short amount of searching, I came across Cairo Graphics.

https://cairographics.org/

It’s a 2D graphics library, written in C, but has bindings for several other languages. Well, that sounds pretty cool. Looking over the tutorials and api documents (https://cairographics.org/manual/), I see that you create a surface, then get a context from that surface, then call various commands on the context such as move_to and line_to, and then follow up with stroke and/or fill. Well that sounds awfully familiar. In fact, I’m going to guess that HTML5’s Canvas is either backed up by Cairo at some level, or is at the very least inspired by Cairo’s api. At any rate, it sounded worth looking into.

I wasn’t about to dive into C programming, but as I mentioned, there are bindings for several other languages. The first I tried was Python, and that worked great. I’ll probably do another tutorial on pyCairo at some point, but I wanted to take it a little further and really dive into something new. I’ve had my eye on a couple of languages – Rust and Go. Both have Cairo bindings, so I flipped a coin and decided to try out Rust.

https://www.rust-lang.org/en-US/

Rust is a pretty interesting programming language. It’s built around speed and safety – preventing you from doing bad things. As such, it has some programming paradigms that seem really strange at first. But there’s a lot of logic behind those decisions. I’m not going to do a Rust tutorial here. But the documentation on how to learn the language is some of the best learning material I’ve seen on any site for any language or system.

Just start reading “The Book”:

https://doc.rust-lang.org/book/

I’m going through the Second edition, which is still listed as “under construction”, but I haven’t run into any flaws or incompleteness as of yet.

What I do want to cover in this tutorial is using Cairo in Rust. There’s a lot resources on how to use Cairo, lots of stuff on how to use Rust. But the center part of the Venn diagram is rather thin.

So, here’s what you need to do:

1. Install Rust and learn how to program in it. At least the basics. I had to read maybe 1/3 of “The Book” before I felt I was ready to start trying to pull in the Cairo bindings and create some graphics.

2. Install Cairo. This is the C library itself. It needs to be on your system so that the Rust bindings for Cairo have something to bind to.

https://cairographics.org/download/

I’m on Ubuntu, so this was really easy, just:

sudo apt install libcairo2-dev

On MacOS, it seems you’re supposed to use MacPorts, which I’m not really familiar with, or fink, which I’ve never heard of.

On Windows, it looks like you just have to add a few dlls to your path.

OK, we’ll carry on, assuming you got Cairo installed.

3. Create a project and set up the dependencies.

Again, this is not a Rust tutorial, but to create a Rust project, you can use cargo new.

cargo new cairo_stuff --bin

This creates a new project named cairo_stuff, which will be a binary executable, (bin) as opposed to a library project.

You’ll see a new folder with the project name. Inside that will be a src folder and a .toml file. It also sets up the project to use git by default, so you get a .git folder and .gitignore file. The .toml file is the project configuration file. It should look something like this:

[package]
name = "cairo_stuff"
version = "0.1.0"
authors = ["Your Name <you@domain.com>"]

[dependencies]

The authors data is taken from your .gitconfig file if you have one. We need to add our Cairo binding as dependencies for this project. To do that, we list them under dependencies. We’ll need two packages or “crates” as they are called in Rust.

We’ll need the png crate so we can save images, and the cairo-rs crate which contains the actual Cairo bindings. We also need to add png as a feature to cairo-rs, so that cairo-rs itself knows how to save pngs.

Whenever you’re doing dependency configuration in any language, it’s important to have a way to know what versions of different dependencies are available. In Rust, you can go to https://crates.io and search for the package you want to add, in this case, “png”.

The first match tells us that we want version 0.11.0.

And for cairo-rs, it’s 0.2.0.

Note, these will definitely and continuously change, so actually go to the site and check the numbers.

For png, we can just add a line in the dependencies section that says

[dependencies]
png = "0.11.0"

Simple.

But the cairo-rs line is a bit more complicated because we need to pass png into it as a feature. So we say

[dependencies]
png = "0.11.0"
cairo-rs = { version = "0.2.0", features = ["png"] }

Again, read “The Book” to fully understand all the syntax here. But this will get you up and running.

Now, if you look at the main source file in the project, it’s src/main.rs and it looks like this:

fn main() {
    println!("Hello, world!");
}

If you’ve done any kind of programming, all that should be pretty self-explanatory. You have a function called main. Rust follows the pattern of many other languages where main is the function that is called when the program is executed.

In main, we call println, passing a string. Other than the exclamation point there, it should be straightforward. (The ! indicates that println is implemented as a macro, not a function, but … whatever.)

At this point you can call cargo run from the main project directory. This will satisfy all your dependencies, build the executable, and run it. It should print out the message “Hello, world!”. Great! It probably took a while to download everything it needed. But if you run it again, it should complete almost instantly.

If you now look in your project folder, you’ll see you have a Cargo.lock file and a target folder. The target is where your final executable and all the build artifacts are. The lock file lists all the other dependencies that got pulled in when you added png and cairo-rs. For me, this file wound up at 171 lines, so there were quite a few other libraries it needed to pull down to satisfy those two that we added.

Let’s Draw Some Stuff

Now that we’ve set things up, let’s draw something.

Here are the basic steps:

1. Create an image surface.

2. Create a 2D drawing context that references that surface.

3. Draw things.

4. Create a file.

5. Write the surface to the file as a png.

For most of your projects, steps 1, 2, 4 and 5 are going to be boilerplate. The meat is going to be step 3 where you draw the actual content you want to draw.

First, we’re going to need to import the cairo-rs bindings. That’s an external crate that we downloaded in our dependencies. So up at the top of the file we say

extern crate cairo;

Note that although the crate name was cairo-rs, we just link to it as cairo.

We’ll be using three modules from the cairo library: ImageSurface, Format, and Context. We could access these like so:

cairo::ImageSurface
cairo::Format
cairo::Context

But repeating that leading cairo:: every time is unnecessary. Instead we can say:

use cairo::{ ImageSurface, Format, Context };

This would be equivalent to something like the following in the latest versions of JavaScript:

import { ImageSurface, Format, Context } from "cairo";

Now we can use those modules directly. Next, let’s make our surface. Delete the println line inside of main and add this instead:

let surface = ImageSurface::create(Format::ARgb32, 600, 600);

So ImageSurface has a method named create which gets a format, width and height and creates a new surface. The format will have four channels – a, r, g, b – each 32 bits.

But actually, this is not quite right as it stands. The ImageSurface::create method does not directly return a surface, it returns a result because this call may fail. Maybe you don’t have enough memory to make a surface the size that you requested for example. So you need to check the result to make sure that it’s valid before proceeding.

There are a few ways to handle that result and get at the surface. Rust provides a quick shortcut to let us do this in one step: just call expect, passing in a message string. If the call succeeds, it will pass through the newly created surface. If it fails, the program will crash, displaying your message.

let surface = ImageSurface::create(Format::ARgb32, 600, 600)
    .expect("Couldn't create a surface!");

Now, obviously, this is not great error handling for a program you’re going to release into the world, but it’s good enough for a first pass of something you’re testing out.

Now we create a context based on this surface, assuming it gets created.

let context = Context::new(&surface);

Context has a new method which takes a reference to a surface, hence the ampersand. (Read “The Book”!)

Now we have a context to draw stuff on! Let’s start by painting it red.

context.set_source_rgb(1.0, 0.0, 0.0);
context.paint();

set_source_rgb takes three normalized floating point values. paint fills the context with the current source value.

Where is My Image?

So how do we see this thing? The program we are creating has no UI. It just runs on the command line. We can’t just display the image in a window, because we have no window. We can save it to a file though.

First we need to create a file object. That would be std::fs::File. We can use use up at the top to shorten that in our code.

use std::fs::File;

After the context.paint() line, create the file:

let mut file = File::create("output.png")
    .expect("Couldn't create file.");

The mut there means this is a mutable variable. It can be changed. By default, variables are immutable in Rust. Our next line will require a mutable reference though, so we make file mutable. Again, this call may fail, so we tack on expect with a message string so that we get a file not a result, and we’ll crash if this fails.

Now we can write the surface as a png to this file:

surface.write_to_png(&mut file)
    .expect("Couldn't write to png");

The write_to_png method needs a mutable reference to a file object. &mut file is how we specify that (see “The Book”). And once again, we check the result.

And that’s that. Here’s the whole program:

extern crate cairo;
use cairo::{ ImageSurface, Format, Context };
use std::fs::File;

fn main() {
    let surface = ImageSurface::create(Format::ARgb32, 600, 600)
        .expect("Couldn't create surface");
    let context = Context::new(&surface);
    context.set_source_rgb(1.0, 0.0, 0.0);
    context.paint();
    let mut file = File::create("output.png")
        .expect("Couldn't create file");
    surface.write_to_png(&mut file)
        .expect("Couldn't write to png");
}

When you run that, you should wind up with a file called ouput.png in the main project directory. It should be a 600×600 red square.

Wait, That’s Not Drawing!

OK, OK, filling a png with a single color is not really drawing. Let’s get a taste of the drawing api itself. I’m just going to throw the whole file at you and let you figure it out.

extern crate cairo;
extern crate rand;

use cairo::{ ImageSurface, Format, Context };
use std::fs::File;

fn main() {
    let surface = ImageSurface::create(Format::ARgb32, 600, 600)
        .expect("Couldn't create surface");
    let context = Context::new(&surface);

    // paint canvas white
    context.set_source_rgb(1.0, 1.0, 1.0);
    context.paint();

    // draw 100 random black lines
    context.set_source_rgb(0.0, 0.0, 0.0);
    for _i in 0..100 {
        let x = rand::random::<f64>() * 600.0;
        let y = rand::random::<f64>() * 600.0;
        context.line_to(x, y);
    }
    context.stroke();

    let mut file = File::create("output.png")
        .expect("Couldn't create file");
    surface.write_to_png(&mut file)
        .expect("Couldn't write to png");
}

Again, the top and bottom of this file is mostly boilerplate. After we create the surface and context, we paint the surface white this time, and use a for loop to draw 100 lines. This is done with context.line_to.

Note that we’re using a random number generator here. That’s another crate. Up top I’ve imported that with extern crate rand; I’ll leave it up to you to search https://crates.io to find the current version and add it to your .toml file.

Now the random functionality gets a bit wordy here, with specifying the type each time (f64 = 64-bit floating point number). But you’re probably going to want to create your own random range function anyway, so you can say random(min, max) and get a result in that range. When you do that, you can just abstract the actual random call within that function.

What Next?

I’ve already said it enough, but read “The Book” if you want to learn more about Rust. It’s really that good.

In terms of cairo-rs, all the api docs are here: http://gtk-rs.org/docs/cairo/

And all the actual drawing commands are on the context object here: http://gtk-rs.org/docs/cairo/

A lot of those will look really familiar if you’ve been working in HTML5 Canvas. move_to, line_to, curve_to, arc, rectangle, etc.

And once you’re familiar with Rust and cairo-rs, you should be able to go to any C-specific Cairo documentation and figure out how that can be translated over to Rust.

One point I want to address briefly is that I said in the beginning that setting up a Webpack/Babel project in order to use the latest and greatest JavaScript features is cumbersome. After reading this article, you might be thinking that setting up a Rust project is just as cumbersome, maybe more so.

But it’s not really. Installing Rust and Cairo itself may have taken a bit, but those are one time actions. Once they’re on your system, you’re done. From there, a new project is as simple as running cargo new and adding two lines for the dependencies. For the source file, you can create some kind of snippet or template for all that boilerplate stuff and you’re off to the races, writing drawing commands in under a minute.

Anyway, hopefully this starts you on some kind of journey of learning something new.

Flow Fields, Part II

tutorial

In Flow Fields, Part I, we covered what a flow field is and looked at a few different formulas to create them. And we rendered flow fields in a various ways. We even animated particles being influenced by fields. In this article we’ll cover a couple more ways to generate flow fields and some new ways to render them. As I said in the first part, the possibilities are endless. I’m just pointing you in a few directions to get started.

Perlin Noise

Using simple math and minimal trigonometry gave us some pretty interesting patterns, but they wound up being very regular and repeating. The more complex function we used (the strange attractor) made for a fairly uninteresting field by itself, but was beautiful with particles tracing lines across it.

What would be nice is something complex but non-repeating. Perlin noise fits the bill, which is why it is used so often to create flow fields. Perlin noise was invented by Ken Perlin and was initially used to generate natural looking textures for computer generated special effects in movies and video games. It gives you random values across a two- or three-dimensional field, and ensures that there is a nice, smooth gradient between the various values. With different settings, scaling and colors applied, you can get textures that look like smoke, fire, liquid, wood grain, rusty metal, etc.

We can tap into Perlin noise to create a flow field that varies all over the place and never repeats. And we can animate it as well by creating a 3D Perlin noise field and moving through slices of it on the z-axis. More on that soon.

If you’re using Processing or some other platform, you may have a Perlin noise function built in. If you’re just using plain vanilla JavaScript like me, however, you’ll need a Perlin noise library to generate the values. There are a few out there. I’m going to use one by Seph Gentle (josephg on github) at https://github.com/josephg/noisejs. I’ve downloaded that into my project folder.

I’m going to start with the same basic HTML file as last time, and add the perlin.js library in the script tag.

And here’s the first script file:

The first block is the same as before. Then we seed the Perlin noise library with a random number so we’ll get a new image each time, and call render.

The render function loops through the canvas in a grid-like fashion as before, getting a value and drawing a line there. All pretty much the same as last time.

It’s the getValue function that changes. Now we’re going to get the value directly from the Perlin noise function. This noise library has a few different functions. We’ll use the 2d Perlin noise function, perlin2, for now. This just takes an x and a y. We’ll scale it down somewhat to make a more interesting pattern. The result of this is a number from -1 to +1. So we can multiply that by 2 PI to get a direction in radians. That’s all, folks. And here’s what it gives us:

Pretty cool, eh? Let’s change the res variable to 2. This makes a much tighter grid:

And now we can see what happens if we change the scale, to maybe 0.004.

It’s like zooming in.

Now, consider 3d Perlin noise as a stack of these 2d Perlin flow fields layered on top of each other, each one varying just a bit. If we could flip through them, one by one, we could see them animating. That’s exactly what our next file does, using the noise.perlin3 function:

Here, we add a z variable set to zero. Then we clear the canvas at the start of render so we can redraw each time. Finally, we increase z by a very small amount and call requestAnimationFrame(render) to set up the next frame of the animation.

The getValue function calls noise.perlin3, passing in the current z. No other changes. A still image would look about the same as the first image in this article, but if you actually run this one, you’ll see a lovely, flowing animation, reminiscent of seaweed, or amber waves of grain.

OK, what’s next? Well, last time we moved on to having the flow field influence the movement of some particles. We’ll do the same thing here, using the Perlin-based flow field.

Again, we create a column of particles on the left edge of the screen, and on each frame, update each particle’s position and draw a line from its last position to where it currently is. This gives us a trail of its movement. The major change here is, again, that we’re using Perlin noise to calculate the flow field. Here’s what we get:

And of course, this looks even cooler as you watch it animate and build up.

One last Perlin example. We’ll use 3d Perlin noise again. This time, on each frame, we’ll create a column of particles down the center of the canvas and iterate each one’s path 500 times. That will create a bunch of flowing lines. Then we’ll update z and draw the next frame. Now, the path of each particle is altered a bit differently on each frame, so you get an eerie, undulating flow going on. The further out from their source each particle gets, the more its path differs from the last frame. Sometimes it jumps off in a totally different direction, giving it a weird, glitchy feeling. The code:

And a single frame of the animation, which you’ll want to see in motion:

What else?

Well, we’ve created flow fields with simple and complex formulas and used third party libraries to create them. You might want to play with the simplex noise contained in that noise library we were using. It’s similar to Perlin noise, also created by Ken Perlin, but has a bit of a different feel to it.

But really we can use anything we want to create a flow field. All we need is something that maps an x, y coordinate to a value. So where could we get a source of values that are mapped to x, y coordinates? Like a map of different… bits of information… like a bit… map. OK, I guess you see where I’m going here. So yeah, any image is simply a 2d map of values. If we can take an image and pass in x and y to some function to get the value at that location, we are golden. And in HTML canvas, we can. And it’s likely you can do the same thing if you are using some other system.

I’ve created a Bitmap module. I’m not going to show it or go into it much here, but you can check out the source at the repo I’ll post at the end of the article. The module takes a url and an onComplete callback that gets called when it’s ready. It then has a getValue method that takes x, y parameters and returns a value between 0 and 255. This is an average of the red, green and blue values of that pixel and basically equates the brightness of the image at that point.

Here’s how we use it:

We create the initial canvas, but we don’t size it full screen like we did before. Then we create the bitmap and wait for it to complete. When it does, we set the canvas to be the same size as the bitmap, create 1000 random points, and call render.

Most of render should look pretty familiar. Get the value at the current location and use that to influence the particle’s velocity. Note that I extracted a couple of values, force and friction. Playing with these can give you very different behaviors. Try out different values. The other main difference here is that if a particle goes off the canvas, we set it back to a random location again.

Finally, the getValue function calls bitmap.getValue. Remember, that will return a value from 0 to 255. So we convert that to be in the range of 2 PI. And that’s it.

All you have to do is supply it an image. Note that for security reasons, you’re not going to just be able to link to any image from anywhere on the web, and you’re probably not going to be able to run this example from the file system. I just set up a simple node.js based server using the http-server package and serve the app and image from there as localhost:8080.

Supplying a random cat image (because this is the Internet, after all), we get the following:

You might even be able to see a hint of the cat there. But if you uncomment line 12 in the code, it will draw the original image before starting to animate the particles. With that, you see the following:

So, there’s a few more ideas. Different images, different settings, different ways of rendering or animating, all make for infinite possibilities. Now you can go explore some of them. Have fun.

The code for all of this can be found at: https://github.com/bit101/tutorials . Note that the code here is not necessarily super clean or ruggedly tested. I’m just having fun here and pretty much just stopped when things looked good.

Useful, or at least fun?

I plan to use the money I make from these posts to finance my presidential campaign. Or at least pay my Netflix bill for a month or two. If you’re down for helping me with either of those two things…

Flow Fields, Part I

tutorial

Maybe you’ve heard the term “flow field” and maybe you’ve seen some neat pictures or animations with the term attached. In this article, I’ll cover what a flow field is, show how to create one, and give a few examples of experimenting with them. Flow fields are also known as vector fields.

The Pre-Game Show

I’ll be using HTML5, JavaScript and Canvas for this article, but the concepts will apply to Processing, p5js, WebGL, or just about any other graphics programming platform. That being the case, I’m going to try to focus less on the canvas specific stuff and more on the core concepts.

But, to just lay the foundations, I’m going to go over the files we’re using. It’s all going to be super basic. An HTML file and a JavaScript file. Here’s the HTML:

A bit of CSS, my QuickSettings Panel, which might come in handy eventually, a canvas element and a link to the main script. The script is going to start out like this:

All we’re doing here is getting a reference to the canvas and its context, then resizing the canvas so it fills the browser window, saving the width and height values for later use. Some of the CSS in the HTML file aids in all this being fully possible.

The Main Event: Flow Fields

So, what is a flow field? Well, you can think of a field as just a two-dimensional area. Of course, you could have a 3D flow field, but let’s save that for another day. You can also think of a magnetic field.

In this image of iron filings revealing a magnetic field, you see various lines and loops. The strength and orientation of the magnetic force is different in different areas of the field. You can imagine that an object moving through this field would be influenced by it and tend to move along those visible lines.

Basically, that’s what we want to do – create a two-dimensional area where each point has a different value. But these aren’t just random values. A particular point has a particular value and as you move out from that point to neighboring points, you get similar, but gradually changing values. In flow fields, these values are usually interpreted as directions. So if you map your values so they are between 0 and 360, they can be directly used as degrees of direction. Of course, we’ll probably map them between 0 and 2PI radians because the computer likes that better.

Once we have values for each point, we can graphically render each one based on it value. For example, we can draw a line pointing in the direction associated with that value.

So that brings us to two different tasks – how to come up with the different values for the field, and how to render the different values on the field. The great thing is, there are no correct answers for either one of those questions. In fact, there are nearly infinite ways of doing either one, so you can explore this technique for quite a while.

Let’s start out really simple, to get the idea of what’s going on. I’m just going to loop through from left to right and top to bottom, every 10 pixels. The value for that point on the grid will simply be (x + y) * 0.01 % Math.PI * 2. Just adding x and y together, scaling it down and modding it by 360 degrees (Math.PI * 2 radians). Then we render that value by translating to that point, rotating by the value and drawing a short line at the point.

Simple as that all is, we already have the start of something interesting going on.

You might now see why this is starting to be something you’d call a “flow field”. As you follow the directions of the lines, you start to see some very directional motion going on.

We can start cleaning this up a bit by extracting the two things that we’ve already determined are going to change a lot – the calculation of the value and how each value point is rendered. We’ll put each one of those into its own function and call them where appropriate.

No change in behavior, but we’ve isolated the things that we now want to change. Furthermore, we don’t necessarily always want this to be in a strict grid. Instead we can grab any number of random points on the grid and render those.

I’ve arbitrarily defined a value of 20,000 for number of points to render. Just looping through that many times, getting a random point and finding its value and rendering it. I’ve changed a few other values as well, such as the length of the line and the scale value that (x + y) is multiplied by. Now we’re getting something a bit more natural looking.

At this point, we can just start messing with the formula we use to get the value. Generally, you’re going to want to use the x and y inputs somehow, but really, just do whatever you want here. Here’s a somewhat interesting one I came up with.

Just taking the sine of x and y and adding them together. Some rendering changes too: reduced the line width and varied the line length at each point. But with just those few small changes, we’re onto something really hairy looking.

Now I could spend all day iterating on this – different formulas, different parameters for the lines it’s drawing etc. I know I could spend all day on it because I have spent whole days doing just that in the past. But let’s move on to something else.

Earlier I mentioned the idea that you could possibly imagine an object moving through a flow field and being influenced by those flows. Let’s simulate that. I’ll start with a single random point. Call it a particle. This particle will actually be an object with not only position, but velocity on the x and y axes. Initially these will be set to zero. On each iteration, wherever the particle is, we’ll get the value at that location – which is a direction, remember – and use that value to influence the particle’s velocity. Then we’ll add that velocity to the position to get a new position. And repeat.

Hopefully the comments explain a bit more what’s going on here. This gives us the following:

What’s interesting in this case is that we no longer actually see the field. We only see the result that the field has on the motion of the particle.

Do run this one on your own. It’s pretty neat to watch the drawing build up over time. You can see that the flow field is influencing the way the particle moves. You can change some of those hard-coded values to see what they do to the motion. Even in just our fifth iteration here, we have a ton of things to experiment with.

Let’s do one more for this article. Where one particle was fun, more particles is… more fun.

Here, I just made an array of particles and arranged them down the left side of the screen. For the algorithm, I used a strange attractor called the “Clifford Attractor”, published by Paul Bourke and attributed to Clifford Pickover. You can see the code for that here: http://paulbourke.net/fractals/clifford/ This gives you a very complex field. The parameters, a-d, are randomized at the top of the file, so you’ll get a different pattern every time.

Again, this is one you want to run on your own and play around with. It’s really quite beautiful to see in action. If you’re curious what the flow field actually looks like for one of these attractors, it’s something like this:

OK, that’s enough for one article. In part two, we’ll look at some other ways to create fields and other ways to render them. If you want to play around with the code itself, you can either grab the gists I’ve embedded here, or just check out the full tutorial repo itself at https://github.com/bit101/tutorials.

Pay(pal) it Forward?

If you’ve found this post useful or at least entertaining and want to contribute in some way, I will gladly accept your kindness.

It’s not my plan to make any kind of premium, subscriber-only content. And the Patreon model doesn’t really cut it for me. I just want to write tutorials for a while. If I make a few bucks here and there, I will probably want to write tutorials even more.