Thoughts on Art

misc

I’ve been thinking a lot about “art” recently. Specifically, what people call “generative art”, “algorithmic art”, “code art”, “math art”, etc. Here are some random thoughts.

Message

I’ve never been one to try to communicate some message through the things I create. I really only try to create things that are visually interesting. I can get very excited about the way a piece looks and I just want to show others and hope that they get a taste of that excitement too. Sometimes things I create can evoke various emotions – they can look ominous, dark, scary, energetic, fun, etc. I often find that when I feel a certain way about a piece, others tend to experience that same feeling. I guess you could call that a message if you want. But I don’t know that I’ve ever sat down to create some digital art with the thought, “I feel like creating something dark and ominous today,” or intentionally tried to create any emotion before I started.

Discovery

For me, creating art with code is way more about discovery. I play around with some technique, or combine one technique with another dis-related technique just to see what happens. Then I see something interesting and I zoom in on it. Sometimes this means literally zooming into some detail in an image. More often it means focusing on some combination of parameters that have created a particular result. There’s a promise of something even more interesting there. I start tweaking numbers to see if I can bring out more of that something. I increment a parameter a few times and that thing I saw goes away. So I decrement it and maybe that thing becomes bigger, or more clear, or more detailed. It’s more like mining than creating.

Math

I love math, or “maths” if that’s how you think of it. I love finding some new interesting formula. I subscribe to recreational math blogs, YouTube channels, Twitter feeds. I go to the math section of bookstores and libraries. I scroll through Wolfram and Wikipedia looking for new ideas. Old copies of Scientific American, Omni, Quantum and other math and science magazines. I get sucked in by anything that has a graph or an interesting diagram. It’s got to be visual. For me, math is the ultimate creative tool. It’s the canvas, it’s the paint, it’s the brush. Really, it’s the artist. All the images are already there. I’m just carefully extracting a few of them out of the sea of numbers. If I have to have a message in my art it’s “Look how amazing math is.”

Random

Random is evil. Random is lazy. Random is OK when you’re starting a new piece. It’s OK when you have a formula and you’re searching for an interesting range of input parameters. But once you find something interesting, lock in those parameters and start focusing. My code framework is set up to allow me to easily create animations by changing parameters over each of many frames. Sometimes I’ll generate several hundred frames all with random parameters. I’ll print the parameters right on the piece. Then I’ll sift through the frames one by one and find those that have something I like. I’ll grab those parameters and hard code them and start tweaking them as described above. But random should come at the start, not at the end. I never continue to use random parameters to generate a finished piece. I’m totally lying. I do it often enough. But I feel lazy when I do it.

Technology

People often ask me what technology I used to create my images. I think they expect that I’m using some app or framework that has all these kinds of image generation tools built into it. I don’t. I write mostly in Go (Golang). I have some custom Go bindings for the C library, cairographics. Cairo is a pretty powerful drawing API, but that’s all it is – it draws lines and arcs curves and rectangles, sets colors, etc. Nearly identical in most important ways to the HTML Canvas drawing API. That’s all I need – those 2D drawing primitives. I have a framework of my own that I’ve built up over many years that does all the complex fancy stuff. But it’s all based on those 2d primitive drawing actions.

Color

I love monochrome. Black and white. I love to be able to bring out form and emotion just by the relative brightness of black and white pixels. Sometimes I’ll experiment with color. I have a random RGB function that makes me feel guilty when I use it. I like using HSV colors better. You can create nice gradients. You can keep one hue and vary the saturation or value. You can create a range of colors with a similar hue. I’d like to take a deep dive into learning color theory some time. But I’m very comfortable with black and white.

Combination

One of my best techniques is a meta-technique. I mentioned it above. Every time I learn some new technique or formula, I do a mash up with that and some other technique I already know. I know I’ve talked about that in other posts in more detail. It’s the best way to discover something that has a chance of the elusive “nobody has done this before”.

Dislikes

  • AI/ML/GAN stuff. It just does nothing for me. To me it just looks like Photoshop filters on steroids. I know there’s a lot of impressive stuff going on behind the scenes, but the result is not interesting to me.
  • Glitch art. I think this is due to the fact that I grew up in the 60s and 70s and 80s. Everything was glitchy. I didn’t like it then and I don’t like it now. I like that things are not glitchy now. I like that I turn on the TV and see a clear picture without bending the coat hanger stuck in the back into some new shape.
  • 3D. By this I mean slick, realistic 3D stuff done in professional 3D rendering packages. I like occasionally hand-coding lofi 3D stuff.
  • Shaders. In either 2D or 3D. I probably should like shaders. You can do some impressive math-based stuff with them. I’ve used them. I even written about them and touched on them in some videos, but they never stuck as something I wanted to continue working with.

OK, now here’s where you’re going to say, “You should check out _____ (AI/Glitch/3D/Shader thing).” And I’ll say, “Thanks, that looks cool. Maybe I’ll check it out.” But I never will. But really, thanks! I appreciate your enthusiasm. I’m not saying any of these things are inherently bad. Just sharing my tastes.

My Music Appreciation Project

misc

Not a big fan of music streaming.

I have a good size, custom curated, very well organized library of music on my hard drive. It’s taken me years to create, it’s all backed up. I have a subsonic server behind Wireguard so I can listen to it from my phone or other devices, and it’s all also synced to my Sony Walkman NW-A55 hi-res digital audio player that I use with my Ikko OH1 IEMs.

I’m really very satisfied with this setup.

But still, I find myself not being able to decide what to listen to way too often. The paradox of choice. Fairly often I’ll just throw the whole thing, or maybe some specific genre on shuffle, and that’s pretty good. But I also like listening to albums. When I’m just choosing an album myself, I’ll gravitate to the same few dozen or so artists. Other stuff might go months or years without being listened to.

I think my devices and software let me shuffle albums, but I wanted to create a long range plan to get to all of it over time. Here’s what I did:

  1. Printed out a list of all artists. Literally just went to the terminal and did an ls in my music folder, directing it to a file.
  2. Pulled that file into a Google spreadsheet, one artist per line.
  3. Shuffled them.

Now I’m just working down the list. I just grab the next artist in the list and choose an album and start playing.

If I’m not feeling the vibe with that particular artist or album, I just move on. But I try to listen to at least one or two songs anyway.

When I move on to the next artist, I mark the previous one as done.

Here’s what’s on my playlist for today:

  • Ministry
  • Van Morrison
  • Harry Nillson
  • The White Stripes
  • John Lee Hooker

I’ve really been enjoying this strategy. Been listening to music that I haven’t in a long time, not just the same albums over and over. And it’s all content that I’ve chosen to put in my library, so it’s all artists I generally like to some degree. And sometimes get totally different genres up against each other, which is fun.

This is going to take months to go through, but it’s been keeping things fresh.

Strange Attractor Flow Fields

tutorial

One of my tricks when coming up with new generative ideas is to take two (or three) completely unrelated techniques and combine them. I even created an app once where I had a long list of concepts and it would randomly combine them and come up with things like, “circle packing with diffuse limited aggregation”. (I just made that up and I’m not sure how it would work, but it’s intriguing, isn’t it?) I can’t say that the app ever sparked any projects that I ended up using, but I’ll probably go back to it some time and see if I can make it better.

This post was one of those successful combinations that evolved on its own and produced something that I have never seen anyone explore before. It produces a visualization of the forces behind a strange attractor that I find fascinating and beautiful. It feels like putting on special glasses that let you see the invisible workings of the universe.

Sketching in eInk

misc

In my last post I talked about my newest eInk tablets from Onyx Boox. Beyond reading ebooks and saving articles, the Note Air is a fantastic sketching tool. The Wacom layer and stylus is one of the big features that led me to look beyond Kindle devices. It was nice on the 7.8″ Nova Pro, but on the 10.3″ Note Air, I’ve really fallen in love with the drawing feature.

Preliminary Pen Points

The Note Air comes with a stylus that attaches to the side of the device via a magnet. All I can say is that it makes a good backup stylus.

Issues with the stylus:

  1. The magnet is not strong enough to trust, so you wind up having to figure out your own way to keeping track of the stylus. Because it WILL fall off and get lost.
  2. The stylus does not have an eraser feature. some styluses put an eraser function in the top of the stylus, where you’d find an eraser on a regular pencil. Others add a button that switches the stylus into erase mode. The default Note Air stylus does neither. So when you want to erase something, you have to choose the eraser tool from the tool bar, erase, then re-choose the drawing tool you were drawing with.

I wound up getting a Wacom One stylus.

It looks nice, feels nice, has an eraser button, and doesn’t break the bank.

I read a lot of reviews about the Remarkable 2 Marker stylus that is made for the Remarkable 2 eInk tablet. It’s apparently really good, but pretty pricey. It does work on the Boox devices. Maybe I’ll spring for one some day, but I’m happy with the Wacom One for now. One of the pros of the Remarkable stylus is that the tips are made of a different substance than a lot of other styluses, which gives it a good amount of drag and makes it feel like a regular pen, rather than sliding a piece of plastic across glass. Remarkable sells replacement tips, which fit the Wacom One, so I got a set of them, and I can attest it does add to the experience.

To hold the pen when I’m not using it, I got some of these Ringke pen holders.

These just stick to the cover and have an elastic loop. Been working out perfectly.

Seriously Sketching Stuff

Now onto the sketching functionality of the device. The built-in notes app is where you do the sketching and it’s one of the main apps on the Note Air. This is what it looks like:

You can name your notes, organize them in folders, rename them, view them as a list or thumbnails, etc.

Notes can have multiple pages. You can see above the Notepad1(1) has a little “6” in the corner. This means that this note has 6 pages. While in the note, you can choose to see an overview of all of the pages.

You can delete, add, rearrange, export, and share pages here.

There are multiple sketch tools you can use:

I most often just use the default pen tool. All of the tools allow you to adjust their width. All but the ballpoint pen and marker are also pressure sensitive. You can change colors – various shades of gray mostly, but also a few real colors. Though you won’t be able to see these on a black and white eInk device until you export the note.

There’s all the usual other tooling, if all a bit basic. A few basic shapes, several different kinds of erasers, lasso tool for selecting and moving parts of your sketch, zoom, text, handwriting recognition.

It even has layers, which is pretty damn cool.

You can add layers, rearrange them, hide them, delete them. The bottom layer is a background template. It can be blank, our you can set it to just about anything. Above you can see a basic grid template. There are a bunch built in:

Three pages worth, for drawing, writing, penmanship, accounting, whatever. There are also ones you can download from the Boox cloud:

And you can add your own, just by putting the into the right folder on the device. Just do an internet search for PDF graph paper or pdf templates and you’ll find all kinds of useful stuff. Or you can just create any PDF you want and put it in there. Here are some I added:

You can turn the template’s visibility on and off from the layers tool, which is really quite nice.

On for drawing:

And then off:

I’m constantly using this to figure out stuff that I want to code. I have pages and pages of sketches like this. Most are just brainstorming or figuring out the math about something.

The touch layer of the device is totally separate from the Wacom drawing layer, so there’s no need to worry about putting your hand on the device as you’re drawing. When you want a new page in a particular sketch, just swipe with your finger, right to left, and you’re on a new page. Go back and forth to existing pages by swiping right or left.

You can set up notes to automatically sync to the cloud, using Onyx’s own cloud sync, Youdao (a Chinese service), Evernote, Dropbox and/or OneNote. Synced notes are available in the cloud as PDFs, but you can also export them as PNGs.

I always have the Note Air nearby when I’m doing some creative coding, so it’s easy to just grab it and work out some idea. I just create note after note, page after page. Space is no worry. They’re backed up and easily available on Dropbox, not floating around on pieced of dead tree pulp.

While the sketch/notes app is probably not something you’re going to create amazing art with, it’s super useful for sketching and notes. I also use it to take notes in meetings and interviews. Very nice for that.

I have to give a warning though. While the built-in notes app is super responsive and a joy to use, the Wacom layer is not optimized for third party drawing, sketching or note-taking applications on the web or Android apps. It will be a bad experience using apps like that. It would be nice if that got fixed, but not holding my breath. I did see where someone has put together their own libraries that make the experience much better in third party apps, but I have not tried it myself.

20 years

misc

TWENTY

DAMN

YEARS

September 11, 2001 is quite a memorable day for the obvious reasons. But for me, it holds an additional significance. Because on September 10, 2001 I registered the domain bit-101.com and on the morning of September 11 the first version of the site went live, only to be massively overshadowed by other events just a couple of hours later.

Initially the site was a single page with a Flash application containing a calendar that linked to various interactive experimental pieces. I’d started doing the experiments in late August, so I was able to launch BIT-101 with fourteen experiments. It ultimately grew to over 600.

This was the previous site that was retired on 9/11/01, also fully Flash:

That KP logo came in with a really cool animation and there was a funky 5-second free music loop that I snagged off of FlashKit, which got really annoying after roughly 10 seconds.

A later version of BIT-101:

Yeah, I liked the Papyrus font back then. Also… what are lower case letters? All those sections were draggable and closable windows. Peak 2002 “web design”.

BIT-101 lasted in this general form, with various interface changes up until the end of 2005. There were many months I posted something new every day. Towards the end, it got a bit slower.

While all this was going on, near the end of 2003, I started the first BIT-101 blog. I say the “first” one because in late 2017 I did a blog reboot, to the new blog that you are reading here. The old one had a good 14 year run though. And is immortalized here: http://www.bit-101.com/old/. Amazing to think that the blog reboot is now almost 4 years old, which is about as long as the first old Flash site lasted. Time keeps moving faster.

Changes

Things sure have changed since that first site 20 years ago. Back then it was all about Flash for me. I was not working full time as a programmer, but I had a steady flow of side jobs doing Flash work. I’d written a few Flash tutorials on the KP Web Design site and those had done really well. In fact it led me to contributing to my first book, Flash Math Creativity.

This led to many more books, mostly with Friends of ED and Apress, but also OReilly.

In 2003 I was invited to FlashForward in NYC where I was a finalist for their Flash awards ceremony in the Experimental category. I remember being so starstruck meeting all my Flash heroes there – many of whom I consider good friends to this day. As it turns out I won the award, which was amazing. I went back to my day job the following Monday. I was working in the estimation department of a mechanical contracting company. I hated that job. I was thinking, “Why am I here? I am a published author and I just won an award for Flash. That’s what I should be doing.” Amazingly, when the boss came in, he called me into his office. Apparently I had screwed up delivering an estimate the previous week and he fired me. What I remember most clearly about that conversation was trying not to smile as I realized I was free. The next day I went to talk to a company in Boston that I had been talking to about doing some Flash work on the side and said I was ready to go full time. They hired me and thus began my official career as a “professional” developer.

Of course, Flash officially died earlier this year. But I had really moved on from it in early 2011, when I did my “31 days of JavaScript” series on the old blog. The inaugural post here: http://www.bit-101.com/old/?p=3030. This series got a lot of attention and by the end of it I had personally switched over to doing all my personal creative coding using HTML5 Canvas.

In 2018 I started looking for some other platforms for creative code. I discovered Cairo Graphics, a C library that is pretty similar to the canvas api in JavaScript. It has bindings for many other languages. I tried it with Python and liked it, but wanted to learn a new language. I’d been interested in both Rust and Golang. I converted my JS library over to Rust and got it working well. But Rust is a pretty exacting language. I found it hard to work with for something like creative coding. I spent more time trying to satisfy the compiler than I did writing any interesting code. So I tried Go and that really hit the spot. It’s been the mainstay language for my creative work for the last three and a half years, though I still keep active in JavaScript as well.

Work-wise, starting from first job in 2003:

  • Exit 33 / Xplana Learning
  • Flash Composer
  • Brightcove
  • Infrared5
  • Disney
  • Dreamsocket
  • Notarize

I started all of those jobs as a Senior Developer/Engineer/Programmer. At Notarize I am now an Engineer Manager, managing 10 other engineers and not really doing any hands-on coding myself. That’s fine with me. It’s a totally new challenge and I’m enjoying it, especially seeing and helping new grads out of school growing into amazing engineers. Interestingly, only two of those jobs required a formal interview. The rest of them were almost straight to offer from people I had gotten to know well through the Flash community.

Summary

It’s been an amazing 20 years. I had no idea where this was going when I randomly came up with “bit-101” and registered the name back then. But it’s worked out pretty damn well. What about the next 20 years? If I’m still breathing and able to type coherent code, I’ll be cranking out something for sure.

More gif-making tips and tools

misc, tutorial

I’ve been continuing my search on the ultimate gif-making workflow and came across two more tools.

gifsicle

and

gifski

Both of these are command line tools available across platforms.

gifsicle

I first heard about gifsicle a while ago as a tool to optimize gifs. I tried it on some of the larger ffmpeg-created gifs and it didn’t seem to do a whole lot. You can specify three levels of optimization. The lowest level didn’t have any effect. The highest level took a single megabyte off of a 27 mb gif. Not really worth it.

You can also use gifsicle to reduce colors in an existing gif. This got considerable results, but at a loss of quality. I think it would be better to do this in the palette creating stage.

But gifsicle can also create gifs from a sequence of images, just like ImageMagick and ffmpeg. Big drawback on this workflow though: the source images have to be gifs themselves. OK, I was able to put together a quick ImageMagick script to convert all my pngs to gifs. But that took quite a while. I didn’t time it, but I feel like it was more than a minute for 300 frames. As for size, it was almost right in between the sizes produced by ImageMagic and ffmpeg.

But the additional conversion process put this one out of the running for me.

gifski

I think this is a very new solution. And it’s written in Rust, which tends to give projects a lot of street cred these days.

After using it, I am impressed.

The syntax is pretty straightforward:

gifski --fps 30 frames/*.png -o out.gif

I don’t think I really need to explain any of that.

Performance-wise, it hits a pretty sweet spot. Not as fast as ffmpeg, but image sizes are way smaller. Not as small as ImageMagick’s output, but way faster.

Here’s the results I got:

Time:

FFMPEG:       5.780 s
gifski:      19.341 s
ImageMagick: 43.809 s
gifsicle:    with image conversion needed, way too long

Size:

ImageMagick: 13 mb
gifski:      16 mb
gifsicle:    18 mb
FFMPEG:      27 mb

Summary

I feel like it’s pretty straightforward.

If you are going for size, nothing beats ImageMagick, but it takes forever.

If you are going for speed, nothing beats ffmpeg.

If you are dealing with gifs as your source image sequence, gifsicle might be a good compromise.

But I think the overall winner is gifski in terms of hitting that sweet spot. I’ll be using it a lot more in coming weeks and days and update with any new findings.

I should also note that all my tests have been on grayscale animations. Full color source images could change everything.

A note on quality and duration

All of the gifs produced seemed to me to be of very comparable quality. I didn’t see any quality issues in any of them. To my eye, they seemed like the same gif, with one exception – duration.

Actually I discovered today that ImageMagick’s delay property will truncate decimal arguments. Or maybe round them off. I’ve gotten conflicting info. Anyway, I’ve been using a delay of 3.33 to make them run at 30 fps. But it turns out it just puts a delay of 3/100’s of a second. So they’ve actually been running a bit faster than 30fps. Somehow, the gifs created with ffmpeg and gifski do seem to run at the exact fps specified. Specifically, a 300 frame animation set to run at 30 fps should run for 10 seconds, as the ffmpeg and gifski gifs do. But ImageMagick’s finishes in 9 seconds.

I tried some other formats for the delay parameter. Apparently you can specify units precisely, like -delay 33,1000 for 33/1000’s of a second, or even -delay 333,10000 for 333/10000’s of a second. But this doesn’t make a difference. From what I understand, the gif format itself does the 100th of a second rounding. If so, I’m not sure what ffmpeg and gifsicle are doing to make it work correctly.

Big deal? Maybe, maybe not. Worth noting though.

More ffmpeg tips

tutorial

Palettes

In a recent post, I shared making animated gifs with ffmpeg. I hadn’t been doing it very long so I wasn’t sure how it would work in the long run. Lo and behold, I ran into a problem. I was making black and white (actually monochrome grayscale) gifs and for the most part it was going well. But then I saw some yellow getting in there!

None of the still images had anything but grayscale values. So how was I getting yellow? To be honest, I’m not sure of the details 100%, but it’s got to do with palettes. Gifs generally have just 256 available colors. There are tricks to make animated gifs use more than that, but let’s stick with the basic case. 256. The frames you create for your animations will likely be pngs, which mean they can have millions of colors. Somehow, ffmpeg needs to take all those millions of colors and choose just 256 for the gif.

Last time, I posted this command:

ffmpeg -framerate 30 -i frames/frame_%04d.png out.gif

This was working pretty well for my grayscale gifs, but if you tried using it for full color animations, there’s a good chance you ended up with a mess. Because you basically got a random palette. I don’t know how the palette is chosen in that case, but there’s a damn good chance it’s not going to be right. So you need to create a decent palette. That’s done with a palettegen filter. First you run ffmpeg to generate a new 16×16 pixel image (256 pixels) that contains the palette it thinks it should use. That looks like this:

ffmpeg -i frames/frame_%04d.png -vf palettegen palette.png

I’ll keep the same flow of going through each parameter:

-i frames/frame_%04d.png – chooses the frames that contain the palette data.

-vf palettegen – a video filter. The filter is palettegen which generates a palette.

palette.png – the output image holding the palette.

The result, as I said, is just a 16×16 png image. You can open it up and see your palette. This has been done by analyzing all the images in the sequence and determining which 256 colors can be used across the entire animation to make things look decent.

Now you use this palette image in another call to ffmpeg:

ffmpeg -framerate 30 -i frames/frame_%04d.png -i palette.png -filter_complex paletteuse out.gif

Once again, I’ll break it down.

-framerate 30 – the fps just like before.

-i frames/frame_%04d.png – the input frames, just like before.

-i palette.png – yet another input, this time, the palette image.

-filter_complex paletteuse – this is where the magic happens.

out.gif – the output image, just like before.

So the filter_complex one is pretty complex, especially if you try to look up the documentation or examples. You’ll find examples like this (IGNORE THESE!):

ffmpeg -i input.mp4 -filter_complex "[0:v]scale=iw:2*trunc(iw*16/18), \
boxblur=luma_radius=min(h\,w)/20:luma_power=1:chroma_radius=min(cw\,ch)/20: \
chroma_power=1[bg];[bg][0:v]overlay=(W-w)/2:(H-h)/2,setsar=1" output.mp4

or…

ffmpeg -i bg.mp4 -i video1.mp4 -i video2.mp4 -filter_complex \
"[0:v][1:v]setpts=PTS-STARTPTS,overlay=20:40[bg]; \
 [bg][2:v]setpts=PTS-STARTPTS,overlay=(W-w)/2:(H-h)/2[v]; \
 [1:a][2:a]amerge=inputs=2[a]" \
-map "[v]" -map "[a]" -ac 2 output.mp4

You’ll generally find these in Stackoverflow, with instructions like, “Just do this…” and no explanations at all on why you should JUST do that.

If you’re lucky, you’ll at least find something like…

 -filter_complex "fps=24,scale=${SIZE}:-1:flags=lanczos[x];[x][1:v]paletteuse"

I stripped away everything but the filter_complex part there. This one actually came from a good friend, Kenny Bunch, who saw my last article and happened to be digging in to the exact same stuff at the same time, but with full color animations.

This one was still more complex than I needed, but it was the simplest example I found, so I was very thankful.

So back to basics. filter_complex is a way of defining, a… well, a complex filter. You define the filter in a string and can chain together multiple actions to do all kinds of fancy things. The way it works is a series of filter actions that result in an output, and then you can use that output in further actions. Like this, broken down per action:

do first filter action[a]; does the first filter action and stores the output in a variable a.

[a] do another filter action[b]; feeds the output a into the next action, and saves that as b.

[b] yet another action[x]; you get the idea.

So in the above example:

fps=24,scale=${SIZE}:-1:flags=lanczos[x]; Sets the fps to 24, scales the gif to a given size on x and keeps the aspect ratio on y (the -1 param), uses Lanczos resampling to do the scaling. Stores the result in x.

[x][1:v]paletteuse Takes the data in x and uses input 1 (the palette image) as a palette using the paletteuse filter. In short, uses the palette image as the palette for the animation.

In my case, I’d already set the framerate. And I wasn’t scaling anything. so I could get rid of that whole first action. And apparently, ffmpeg was smart enough to figure out that input 0 was the frames and input 1 was the palette, So I could shorten the entire thing down to :

-filter_complex paletteuse

Magical.

Using these two commands, I was able to get a correctly paletted animation, with no yellow:

I still don’t know exactly why I was getting yellow, since all the source frames were completely grayscale. I guess it was just the complexity of calculating the values of all the channels of all the pixels from all the frames. Somewhere along the line, it wound up with a bit less in the blue channel for some reason. And once it started, it just multiplied.

Anyway, couldn’t help thinking of this…

Speed and Size

Other considerations when deciding whether to use ffmpeg or ImageMagick are rendering speed and file size.

ffmpeg will create gifs way faster than ImageMagick. A quick test:

Input: 300 frames, 500×500 pixels each.

ImageMagick: 29.985 seconds

ffmpeg: 4.301 seconds

And the ffmpeg test included generating the palette as well as the animation.

File size is not so good a story though. Same animation:

ImageMagick: 13 Mb

ffmpeg: 27 Mb

Other examples weren’t quite that bad, but ImageMagick always wins handily in the size category.

Of course, the other thing I mentioned last time was that ImageMagic will consistently use so much memory that the process just crashes and fails, whereas ffmpeg never has a problem in that area.

Just Say Yes

One last trick if you’re scripting these commands and you’re constantly having your script stop and ask you if you want to override your previous output gif with a new animation. Just add the -y parameter into your command and it won’t bother you anymore.

Animation Cookbook for ffmpeg and ImageMagick

tutorial

I create a lot of animated gifs and I’m getting into creating videos as well. All of these are initially created by rendering a series of png files into a folder called frames. Over the last few years I’ve figured out some pretty good recipes for converting these frames into animated gifs and videos. So I’m sharing.

Both ImageMagick and ffmpeg are extremely complex programs with tons of parameters that allow you to do all kinds of image, video and audio manipulation. I’m not any kind of an expert on any of these. And this post is not meant as thorough documentation on these programs. It’s more of a cookbook of commands that handle specific use cases well. I’ll also try to explain what each of these parameters does, which might help you to come up with your own commands.

Nothing beats thoroughly reading the official docs, but if you just want to get something done quickly, these recipes should help you.

ImageMagick

It confused me at first, but ImageMagick is not a single executable. It’s more of a suite of graphics utilities. Maybe under the hood it’s all one executable, but it presents as different commands you can call. For generating an animated gif, you’ll be using the convert command.

As I said, I keep my frames in a folder called frames. They are named sequentially like so:

  • frame_0000.png
  • frame_0001.png
  • frame_0002.png

I use four digits, which gives me up to 10,000 frames. At 30 fps, that gives me about 333 seconds worth of video, or about five and a half minutes. Good enough for my needs now. If you go with three digits, you’ll only get 33 seconds worth. If I ever need to do more than five minutes in a single clip, five digits will give me almost an hour.

convert uses simple wildcard matching, so my input is simply frames/*.png and the output can be something like out.gif

Next you need to figure out the frame rate. I usually go with 30 fps. You don’t enter frame rate directly with convert though. You need to specify a delay between each frame. And here’s where things get really weird. The unit you use for the delay parameter is hundredths of a second. So if you want 30fps, you’d specify 100 / 30 or 3.333… Not exactly intuitive, but you can get used to anything. Put it all together and you get:

convert -delay 3.333 frames/*.png out.png

But there’s one more parameter I put in there, which is -layers Optimize

convert has a whole ton of stuff you can do with it to affect how the gif is put together. -layers Optimize combines a few of the most useful ones into a single command. I found that before I started using this, I’d get an occasional animation that just totally glitched out for a frame or two. Actually, it was more than occasional. Not every time, but often enough to be a problem. Using this parameter completely solved it, so I highly recommend it. If you want to read more up on it: https://legacy.imagemagick.org/Usage/anim_opt/#optimize

So the final command I use is:

convert -delay 3.333 -layers Optimize frames/*.png out.png

This has been my bread and butter command for a long time now.

But lately, I’ve been running into problems using ImageMagick to create longer, larger format gifs. This hasn’t been an issue at all when I was just posting to Twitter, which has something like a 15mb size limit. But as I’ve been posting gifs in other places that can be up to 100mb, I’ve started making larger and longer animations. And ImageMagick has been crashing pretty regularly while generating these. I’ve watched a graph of memory use and I’m pretty sure that it’s just running out of memory. It uses up all my physical memory, then uses up all my swap, then crashes. On at least one occasion I got it to work by closing down every other program that was running on the computer. That freed up just enough memory to get through. But it doesn’t always work. So that’s… less than ideal.

ffmpeg for gifs

[Update] Before using the command below, read this post, which has to do with color palettes: More ffmpeg Tips

I’ve been using ffmpeg for a while now to create videos, and I knew it could create gifs as well, so I’ve recently given it a try and it seems to do just as good a job as ImageMagick. And it can handle those high res, long gifs as well. So I’m pretty happy so far. Creating gifs with ffmpeg is also pretty straightforward, except for the input filename specification, which I’ll cover next. Here’s the command I use:

ffmpeg -framerate 30 -i frames/frame_%04d.png out.gif

Breaking that down, the framerate param is just the number of frames per second. Simple. -i is for input, and you point it to your sequential frames. and finally, the output filename.

The only complexity is the frame_%04d.png part. This is a printf type of string formatting convention. The %04d part means that you have four digits (4d), and they will be padded with 0s. So frame 37 will be 0037.

As I said, this has been working really well for me so far, but I haven’t been using it very long, so if there are any issues, I might come upon them later.

Videos

Video creation is what ffmpeg is usually used for. It’s an amazingly powerful and complicated tool. Even just figuring out all of the possible parameters can be daunting. So I’ll go over some of the common use cases I have and what each parameter does.

Mostly I’ve been optimizing the videos I create for use on YouTube. This uses h264 encoding and a specific pixel format. Here’s the basic command I use for creating a video from a sequence of frames:

ffmpeg -framerate 30 -i frames/frame_%04d.png -s 1920x1080 -c:v libx264 \
 -preset veryslow -crf 20 -pix_fmt yuv420p out.mp4

(That should be all one line. I wrapped it to make it more readable.)

Yeah, a lot to take in there. So let’s go through it param-by-param.

-framerate 30 is just the fps setting.

-i frames/frame_%04d.png specifies the input files as described earlier.

-s 1920x1080 sets the size of the video in the format of widthxheight. I believe that if you leave this param off, it will create the video based on the size of the source images. But you can use this to scale up or down as well.

-c:v libx264 OK, this will take a little background. ffmpeg can be used for creating or processing both audio and video – or both at the same time. Some of the parameters can be applied to audio or video. In this case, -c is letting you specify what codec to use for encoding. But is it a video codec or an audio codec? By adding :v we’re specifying that it’s a video codec. Fairly obvious here because there is no audio involved in this case, but explicit is always good. I’ve seen examples where people use -s:v to specify the size of the video, which seems overkill to me. But ffmpeg can also handle things like subtitles, and maybe those can be sized? Again, explicit is good. Anyway, here we are saying to use the libx264 codec for encoding the video. Which is good for Youtube.

-preset veryslow Here we have a tradeoff – speed vs size. These are the presets you can use:

  • ultrafast
  • superfast
  • veryfast
  • faster
  • fast
  • medium – default preset
  • slow
  • slower
  • veryslow

The faster you go, the larger your file size will be. It’s recommended that you use the slowest preset you have patience for.

-crf 20 is the constant rate factor. There are two ways of encoding h264 – constant rate factor (CRF) and average bit rate (ABR). Apparently CRF gives better quality generally. This parameter can go from 0 to 51, where 0 is lossless and 51 is crap. 23 is default and anything from around 17 to 28 is probably going to be generally acceptable. The CRF setting also affects file size. Lower CRF means a higher file size. So if you need the lowest possible file size, use a high CRF and a slow preset. It will take a long time and look like crap, but it will be small! More info on all this here: http://trac.ffmpeg.org/wiki/Encode/H.264

-pix_fmt yuv420p is the pixel format. You might be more used to RGB888 where you have red, green and blue channels, and 8 bits per each of those channels. YUV is just an alternate way of formatting colors and pixels. Mind-numbingly detailed description over at the old wikipedia: https://en.wikipedia.org/wiki/YUV . But this is a good format for Youtube.

out.mp4 is the file that is created after applying everything else.

So there you go. my bread and butter video creation command.

But there are a few more tidbits I use:

Fades

I usually try to make my animated gifs loop smoothly. But for longer form videos this is not always what you are going for. Still, you usually don’t want the video to just get to the end and stop. A fade out to black is a nice ending effect. You could build that into your source animation code, or you could do it in post of course. But ffmpeg has fades built in. Here’s the altered command:

ffmpeg -framerate 30 -i frames/frame_%04d.png -s 1920x1080 -c:v libx264 \
 -preset veryslow -crf 24 -pix_fmt yuv420p -vf "fade=t=out:st=55:d=5" out.mp4

Here, I’ve added the param -vf "fade=t=out:st=55:d=5"

The vf is for video filter, I believe. This takes a string in a very compact format. It starts by specifying the type of filter and its definition: "fade=...". After the equals sign is a list of parameters in the format: a=x:b=y:c=z. We’ll step through those:

t=out means the type of fade out.

st=55 tells the fade to start at 55 seconds into the video.

d=5 means the fade will last 5 seconds.

In most cases the st and d parameters will add up to the total video length, but you can do all kinds of things here. Like fading out mid video and then fading back in later.

Obviously t=in would execute a fade in. You might want to do that at the start of your video.

Although I have not had the need for it, I assume you can use -af to create an audio fade.

Adding Audio

One of the big benefits of video over animated gifs is the possibility of adding music or sound effects to the animation. As mentioned, ffmpeg can do that too.

I believe it’s possible to add the soundtrack to the video as you are creating it from the source frames, but this does not seem like a good workflow to me. I might generate dozens of videos before coming up with one I decide to go with. So it makes more sense to me to add the audio when I’m done. Here’s my command for adding audio to an existing video:

ffmpeg -i video.mp4 -i audio.wav -map 0:v -map 1:a -c:v copy output.mp4

Again, let’s go through each param.

-i video.mp4 is the video you’ve already created that you want to add audio to.

-i audio.wav is the sound file you want to add to the video. Yes, we have two inputs here. That’s how it works.

-map 0:v The map parameter tells ffmpeg which input is what. Here we’re saying the first input (0) is video (v).

-map 1:a As you can probably guess, this says that input 1 is audio.

-c:v copy As covered before, this is the video codec. In this case though, we don’t want to encode the video all over again. We just want to copy the video over and add the audio to it.

output.mp4 is the final combined audio/video file.

So that’s pretty straightforward and works great. But this kind of assumes that your video and audio are the same size. They might be, which is nice, but they might not be, which raises the question of how long to make the final video? Do you make it the same length of the video or the length of the audio?

I believe by default, it will choose the longest option. So if your video is one minute long and you throw a five minute song on top of it, you’ll wind up with a five minute video. Honestly, I’ve never tried this so I don’t know what happens. I assume you get four minutes of blackness. Or maybe four minutes of the last frame of the source video. Or maybe it loops?

On the other hand, if your video is longer than your audio, you will probably wind up with some amount of silence after the audio finishes.

So the other option here is the shortest param:

ffmpeg -i video.mp4 -i audio.wav -map 0:v -map 1:a -c:v copy -shortest output.mp4

All this does is make the output video the same length as the shortest input.

I’m sure you can figure out all the different use cases here. And you probably want to think about fading your audio out along with a video fade so the sound/music doesn’t just suddenly stop.