As mentioned previously, I have started working on self-publishing my Playing With Chaos book on the Amazon Kindle self publishing service. I quickly got the outline, first chapter, part of the second, some code and images done. Then last weekend I decided I better check into the whole publishing process in a bit more detail. I went through all the material on Amazon’s site, as well as several other tutorials. I even downloaded a free ebook on Kindle publishing and paid for another 99 cent book on the subject. I learned a lot, but none of it made me very happy to begin with. Here’s why:

Pretty much every reference starts out telling you to write your book in Microsoft Word. Now don’t get me wrong – I think Word is a great tool as long as you are writing in word and then printing your documents out, or sending them to someone else who will view them in Word. It’s a super powerful application with every tool you could possibly imagine for writing almost anything. But where Word gets really ugly is its export functionality. I’ve done some professional work trying to convert exported Word docs into other formats. It was one long, horrible headache.

Workflow #1

The simplest possible workflow is:

1. Write your book in Word.
2. Upload to Kindle and let Amazon turn it into a Kindle ebook.

I suggest you try this. When you’re done looking at the results and wiping up your own vomit, we can move on.

Workflow #2

The next workflow, which is what most resources will recommend is:

1. Write your book in Word.
2. Save it as a filtered web page.
   2a. Optionally clean up the HTML.
3. Zip and upload the HTML and other files to Amazon.
   3a. Alternately, run it through another program to create a .mobi file and upload that to Amazon.

Doing steps 1, 2 and 3, without the optional/alternate steps will essentially give you the result of the first workflow. But I also suggest you do steps 1 and 2 and look at the results. This will give you an idea of why it’s such a mess. For my 1.5 chapter book in progress, the resulting HTML file was composed of 50% CSS, 50% HTML. All kinds of custom styles going on there. 99% of which will be ignored.

For step 2a, several pages I found offered advice on the various things to clean up and delete or change from the HTML/CSS. Nobody offered an automated solution. This gives you a workflow like this:

A. Write some of your book.
B. Export it to HTML.
C. Go through a lengthy, manual, painful process of cleaning up the HTML by hand.
D. Convert it to .mobi or upload it to Amazon.
E. Preview it.
F. Write some more of your book.
G. Do B-E again.

No thanks.

Now let’s move on to step 3a. There are three main tools used for converting HTML to .mobi.

– kindlegen

This is Amazon’s command line tool, so it must be good, right? Wrong. It’s known to be buggy and its output is just incorrect. It was my first choice, despite the warnings I’d read on various sites. There are Windows, Mac and Linux versions. And it’s command line! How could I not try it. But I soon found myself agreeing with the warnings and abandoned it.

– Mobipocket Creator

This is a Windows-only tool that will take documents of various types and convert them into ebooks. I tried this, but didn’t like it too much. It has a very consumery feel to it. Seems like it tries to over-automate the process for you, making all the decisions for you. Probably with a bit more investment of time, I could learn to control it more, but I was also not thrilled about the Windows-only part. And the fact that it’s a graphic interface tool, which I’ll discuss more in depth shortly.

– Calibre

The third choice is mentioned by a few, but seems to scare away most people. It’s triple platform – a big plus. In fact, I think it began as a linux tool. It also exposes just about every possible option you could ever want to tweak on your import/convert/output workflow. A bit overwhelming at first, but I liked that. While the main program is a graphical tool, I soon found out that it all just sits over a suite of command line tools that expose all those possible options. I was onto something big here!

Workflow #3

I quickly abandoned the whole idea of writing in Word. For one, it means jumping back and forth in a VM. Most Kindle books do not actually have so much formatting that you’re really getting that much benefit out of working in a word processor program like Word anyway. But more importantly, Word was just making me crazy with all that extra markup and styling. Furthermore, it was destroying my code formatting. I’d have a line of code with 4 spaces of indenting. It would sometimes do 3 spaces, sometimes 4, sometimes 5! No lie. True story. Trust me, I spent a couple of hours trying to just get a consistent 4 spaces = 4 spaces conversion. No go.

So I decided to just write my book in Sublime Text 2. Plain text baby!

OK, to be honest, it’s not plain text, but plain HTML. Yes, I’m writing my book with HTML markup. Before you start groaning, let me re-iterate that there is very little formatting that needs to go on in a Kindle book. Really all I’m doing is putting p tags around my paragraphs, and h tags around headings. And the occasional bold/italic/code/pre tag here and there. And images, yeah. But honestly, 95% of it is p’s and h’s.

I’d like to figure out some way to make that a bit cleaner eventually. Maybe some kind of markdown to HTML conversion. Something to research…

See my update on using markdown and pandoc here: http://www.bit-101.com/2003/3502

Anyway, I’m writing the code for the book in Sublime, so it’s nice to have the text and the code in one project so I can just copy and paste.

Now, the workflow in most of the examples I read was something like this:

1. Write.
2. Open MobiPocket Creator.
3. Import your files.
4. Tweak the following settings…
5. Export your book.
6. Close MobiPocket Creator.
7. Open an ebook previewer.
8. Load your exported book into the previewer.

Here’s the workflow I wanted to have:

1. Write.
2. Press F7.

And I have achieved exactly that! Yes, from within Sublime Text 2, I can change some text, hit F7, sit back, and in within a few seconds, I have an ebook reader open with my published .mobi format book loaded up.

The Setup

In the root project folder I have the following:

– book
– code
– images
build.xml

book is a folder containing the chapter HTML files, plus a header and footer.
code is a folder with HTML and .js files that are the example code files I’m building throughout the book.
images is a folder. Guess what’s in it.
build.xml is an ant build file. What? You hate ant? Sorry to hear that. You lose. 🙂

Anyway, if you really don’t like ant, what I have is not rocket science. Should be easy enough to convert into your superior build system. But let’s look at what it does.

[php lang=“xml”]

We start out with some various properties that will get used throughout the build targets. The default target is preview while I’m writing the book, or code while I’m writing code. The preview target depends on publish, which depends on concat. The concat file takes all the chapters, wrapping them in an HTML header and footer, creating one large HTML file with the whole book. The header.html looks like this:

[php lang=“html”]

[/php]

And the footer just closes things up:

[php lang=“html”]

[/php]

All the chapter files just go in the body section. Chapter 1 looks like this:

[php lang=“html”]

Chapter 1: Introduction

Fractals, Strange Attractors and Chaos Theory

Blah blah blah

and blah.

Another Section

more blah

Here’s some code:

function foo() {
  blah();
}

mbp:pagebreak/
[/php]

The is a special tag that Kindle will use to indicate the start of the book. If a user navigates “Go to… Beginning”, this is where they will wind up. So that’s only in Chapter 1. Chapter titles are h1. Sections are h3. Paragraphs are in p’s. Code blocks in pre’s. Note that any < or > in code (or anywhere for that matter) will need to be converted to HTML entities. Pain in the neck there, but whatever. To be honest, it’s less work than I had to manually do to format this blog post. Finally, there’s a custom Kindle tag that inserts a page break, useful for end of chapters.

Again, see the update at http://www.bit-101.com/2003/3502. No manual HTML needed at this point with markdown.

OK, back to the build. After concat, publish runs. Let’s look at that again…

[php lang=“xml”]














[/php]

This calls up the Calibre command line tool, ebook-convert. The command line tools should be automatically installed when you install Calibre on Windows or Linux. Apparently there is one extra step to install them on Mac. The first two args are the input HTML file, and the output .mobi file. Then there are a whole bunch of options.

I’m saying chapters are h1’s and should get a page break. I’m not sure how that differs from the custom pagebreak tag I added manually, but doing both doesn’t seem to hurt anything. The table of contents should be built from h1’s and h3’s.

I’m setting metadata for the author name and cover image. The metadata for the book title will come from the HTML title tag.

I’m setting the output profile as Kindle. Not sure exactly what this does. Seems to work fine without it, but seems sensible to set it.

The last arg is –smarten-punctuation. This turns regular apostrophes, single quotes, and double quotes to their curly equivalents. So I can simply type

"foo"

and get

“foo”

in the output. Also converts single dashes to en dashes and double dashes to em dashes. Brilliantly, this option ignores anything in pre tags!

Once the book is published, the preview target will run. This runs another Calibre command line tool, ebook-viewer, passing in the name of the .mobi file that was just created. This pops open an ebook reader with your book preloaded. This is NOT a Kindle simulator by any means, but it will let you verify that your cover is set, your text looks like what it should, images show up, TOC works, etc.

So again, write something, throw some tags around it, hit F7 and there it is in ebook form. Couldn’t be better. You can then take this compiled .mobi and upload it to Amazon for publishing.

Additional testing. If I’m doing something that I want to make sure looks just right, I have a VM open with Windows running in it. There is a Kindle preview tool that you can download from Amazon that lets you preview how your .mobi will look in a regular Kindle, Kindle DX, Kindle Fire, iPad, iPhone, etc. On Windows and Mac you can also install the Kindle Desktop Reader and make sure your book looks OK there too.

Finally, nothing beats dragging the .mobi into right into a real live Kindle and seeing how it looks on the real live device. There’s actually a Calibre command line tool for transferring a file to a device as well. But with the Kindle, it needs to be plugged in and mounted for transfer, then unmounted to actually read the book. And I don’t test on device every time I build, so I did not make that part of my process.

Here’s the book as seen on a real device:

Oh, and there’s also the code target, which simply launches the specified HTML file in the browser. I just change the default target to code while I’m coding up an example, and back to preview when I’m writing the book.

Summary

Overall, I’m super happy with how this is working out. The big things I’d like to improve are 1. as mentioned, figuring out some possible markdown conversion to simplify formatting and 2. something to possibly automate replacing angle brackets in pre code tags.

Here’s a link to the Calibre command line tools:

http://manual.calibre-ebook.com/cli/cli-index.html

And all the possible options for publishing HTML -> MOBI:

http://manual.calibre-ebook.com/cli/ebook-convert-4.html#html-input-to-mobi-output

I’ve pored over that last link, but I’m sure there’s still some things I could use in there that I haven’t investigated.