My Kindle Authoring Setup

May 13 2012 Published by under Kindle, Uncategorized

Send to Kindle

As mentioned previously, I have started working on self-publishing my Playing With Chaos book on the Amazon Kindle self publishing service. I quickly got the outline, first chapter, part of the second, some code and images done. Then last weekend I decided I better check into the whole publishing process in a bit more detail. I went through all the material on Amazon’s site, as well as several other tutorials. I even downloaded a free ebook on Kindle publishing and paid for another 99 cent book on the subject. I learned a lot, but none of it made me very happy to begin with. Here’s why:

Pretty much every reference starts out telling you to write your book in Microsoft Word. Now don’t get me wrong – I think Word is a great tool as long as you are writing in word and then printing your documents out, or sending them to someone else who will view them in Word. It’s a super powerful application with every tool you could possibly imagine for writing almost anything. But where Word gets really ugly is its export functionality. I’ve done some professional work trying to convert exported Word docs into other formats. It was one long, horrible headache.

Workflow #1

The simplest possible workflow is:

1. Write your book in Word.
2. Upload to Kindle and let Amazon turn it into a Kindle ebook.

I suggest you try this. When you’re done looking at the results and wiping up your own vomit, we can move on.

Workflow #2

The next workflow, which is what most resources will recommend is:

1. Write your book in Word.
2. Save it as a filtered web page.
2a. Optionally clean up the HTML.
3. Zip and upload the HTML and other files to Amazon.
3a. Alternately, run it through another program to create a .mobi file and upload that to Amazon.

Doing steps 1, 2 and 3, without the optional/alternate steps will essentially give you the result of the first workflow. But I also suggest you do steps 1 and 2 and look at the results. This will give you an idea of why it’s such a mess. For my 1.5 chapter book in progress, the resulting HTML file was composed of 50% CSS, 50% HTML. All kinds of custom styles going on there. 99% of which will be ignored.

For step 2a, several pages I found offered advice on the various things to clean up and delete or change from the HTML/CSS. Nobody offered an automated solution. This gives you a workflow like this:

A. Write some of your book.
B. Export it to HTML.
C. Go through a lengthy, manual, painful process of cleaning up the HTML by hand.
D. Convert it to .mobi or upload it to Amazon.
E. Preview it.
F. Write some more of your book.
G. Do B-E again.

No thanks.

Now let’s move on to step 3a. There are three main tools used for converting HTML to .mobi.

- kindlegen

This is Amazon’s command line tool, so it must be good, right? Wrong. It’s known to be buggy and its output is just incorrect. It was my first choice, despite the warnings I’d read on various sites. There are Windows, Mac and Linux versions. And it’s command line! How could I not try it. But I soon found myself agreeing with the warnings and abandoned it.

- Mobipocket Creator

This is a Windows-only tool that will take documents of various types and convert them into ebooks. I tried this, but didn’t like it too much. It has a very consumery feel to it. Seems like it tries to over-automate the process for you, making all the decisions for you. Probably with a bit more investment of time, I could learn to control it more, but I was also not thrilled about the Windows-only part. And the fact that it’s a graphic interface tool, which I’ll discuss more in depth shortly.

- Calibre

The third choice is mentioned by a few, but seems to scare away most people. It’s triple platform – a big plus. In fact, I think it began as a linux tool. It also exposes just about every possible option you could ever want to tweak on your import/convert/output workflow. A bit overwhelming at first, but I liked that. While the main program is a graphical tool, I soon found out that it all just sits over a suite of command line tools that expose all those possible options. I was onto something big here!

Workflow #3

I quickly abandoned the whole idea of writing in Word. For one, it means jumping back and forth in a VM. Most Kindle books do not actually have so much formatting that you’re really getting that much benefit out of working in a word processor program like Word anyway. But more importantly, Word was just making me crazy with all that extra markup and styling. Furthermore, it was destroying my code formatting. I’d have a line of code with 4 spaces of indenting. It would sometimes do 3 spaces, sometimes 4, sometimes 5! No lie. True story. Trust me, I spent a couple of hours trying to just get a consistent 4 spaces = 4 spaces conversion. No go.

So I decided to just write my book in Sublime Text 2. Plain text baby!

OK, to be honest, it’s not plain text, but plain HTML. Yes, I’m writing my book with HTML markup. Before you start groaning, let me re-iterate that there is very little formatting that needs to go on in a Kindle book. Really all I’m doing is putting p tags around my paragraphs, and h tags around headings. And the occasional bold/italic/code/pre tag here and there. And images, yeah. But honestly, 95% of it is p’s and h’s.

I’d like to figure out some way to make that a bit cleaner eventually. Maybe some kind of markdown to HTML conversion. Something to research…

See my update on using markdown and pandoc here: http://www.bit-101.com/blog/?p=3502

Anyway, I’m writing the code for the book in Sublime, so it’s nice to have the text and the code in one project so I can just copy and paste.

Now, the workflow in most of the examples I read was something like this:

1. Write.
2. Open MobiPocket Creator.
3. Import your files.
4. Tweak the following settings…
5. Export your book.
6. Close MobiPocket Creator.
7. Open an ebook previewer.
8. Load your exported book into the previewer.

Here’s the workflow I wanted to have:

1. Write.
2. Press F7.

And I have achieved exactly that! Yes, from within Sublime Text 2, I can change some text, hit F7, sit back, and in within a few seconds, I have an ebook reader open with my published .mobi format book loaded up.

The Setup

In the root project folder I have the following:

- book
- code
- images
build.xml

book is a folder containing the chapter HTML files, plus a header and footer.
code is a folder with HTML and .js files that are the example code files I’m building throughout the book.
images is a folder. Guess what’s in it.
build.xml is an ant build file. What? You hate ant? Sorry to hear that. You lose. :)

Anyway, if you really don’t like ant, what I have is not rocket science. Should be easy enough to convert into your superior build system. But let’s look at what it does.

<project default="preview">
    <!--
        Targets:
            code:    executes code_file property in browser
            publish: creates ebook
            preview: creates and previews ebook
    -->
    <property name="browser" location="/usr/bin/google-chrome"/>
    <property name="code_file" location="code/sierpinski.html"/>

    <property name="book_name" value="playing_with_chaos"/>
    <property name="author" value="Keith Peters"/>
    <property name="cover_img" value="images/cover.jpg"/>

    <property name="html_file" value="${book_name}.html"/>
    <property name="mobi_file" value="${book_name}.mobi"/>
    <property name="meta_file" value="${book_name}.opf"/>
    
    
    <target name="concat">
        <concat destfile="${html_file}">
            <filelist dir="book">
                <file name="header.html"/>
                <file name="pwc_ch01.html"/>
                <file name="pwc_ch02.html"/>
                <file name="footer.html"/>
            </filelist>
        </concat>
    </target>

    <target name="publish" depends="concat">
        <exec executable="ebook-convert">
            <arg value="${html_file}"/>
            <arg value="${mobi_file}"/>
            <arg value="--chapter=//h:h1"/>
            <arg value="--chapter-mark=pagebreak"/>
            <arg value="--level1-toc=//h:h1"/>
            <arg value="--level2-toc=//h:h3"/>
            <arg value="--authors=${author}"/>
            <arg value="--cover=${cover_img}"/>
            <arg value="--output-profile=kindle"/>
            <arg value="--smarten-punctuation"/>
        </exec>
    </target>

    <target name="preview" depends="publish">
        <exec executable="ebook-viewer" spawn="true">
            <arg value="${mobi_file}"/>
        </exec>
    </target>

    <target name="code">
        <exec executable="${browser}">
            <arg value="${code_file}"/>
        </exec>
    </target>

</project>

We start out with some various properties that will get used throughout the build targets. The default target is preview while I’m writing the book, or code while I’m writing code. The preview target depends on publish, which depends on concat. The concat file takes all the chapters, wrapping them in an HTML header and footer, creating one large HTML file with the whole book. The header.html looks like this:

<html>
	<head>
	  <title>Playing with Chaos</title>
	  <meta http-equiv="content-type" content="text/html; charset=iso-8859-1">
	  <style type="text/css">
	    p {
	      margin-bottom: 10pt;
	    }
	    h1 {
	    	margin-top: 120px;
	    	margin-bottom: 20px;
	    	text-align: center;
	    }
	    pre {
	    	font-size: 10px;
	    }
	    .fig {
	    	font-style: italic;
	    	font-size: 10px;
	    }
	  </style>
	</head>
	<body>

And the footer just closes things up:


	</body>
</html>

All the chapter files just go in the body section. Chapter 1 looks like this:

<a name="start"></a>
<h1>Chapter 1: Introduction</h1>
<h3>Fractals, Strange Attractors and Chaos Theory</h3>

<p>Blah blah blah</p>

<p>and blah.</p>

<h3>Another Section</h3>

<p>more blah</p>

<p>Here's some code:</p>

<pre>
function foo() {
  blah();
}
</pre>
<mbp:pagebreak/>

The <a name=”start”></a> is a special tag that Kindle will use to indicate the start of the book. If a user navigates “Go to… Beginning”, this is where they will wind up. So that’s only in Chapter 1. Chapter titles are h1. Sections are h3. Paragraphs are in p’s. Code blocks in pre’s. Note that any < or > in code (or anywhere for that matter) will need to be converted to HTML entities. Pain in the neck there, but whatever. To be honest, it’s less work than I had to manually do to format this blog post. Finally, there’s a custom Kindle tag that inserts a page break, useful for end of chapters.

Again, see the update at http://www.bit-101.com/blog/?p=3502. No manual HTML needed at this point with markdown.

OK, back to the build. After concat, publish runs. Let’s look at that again…

    <target name="publish" depends="concat">
        <exec executable="ebook-convert">
            <arg value="${html_file}"/>
            <arg value="${mobi_file}"/>
            <arg value="--chapter=//h:h1"/>
            <arg value="--chapter-mark=pagebreak"/>
            <arg value="--level1-toc=//h:h1"/>
            <arg value="--level2-toc=//h:h3"/>
            <arg value="--authors=${author}"/>
            <arg value="--cover=${cover_img}"/>
            <arg value="--output-profile=kindle"/>
            <arg value="--smarten-punctuation"/>
        </exec>
    </target>

This calls up the Calibre command line tool, ebook-convert. The command line tools should be automatically installed when you install Calibre on Windows or Linux. Apparently there is one extra step to install them on Mac. The first two args are the input HTML file, and the output .mobi file. Then there are a whole bunch of options.

I’m saying chapters are h1′s and should get a page break. I’m not sure how that differs from the custom pagebreak tag I added manually, but doing both doesn’t seem to hurt anything. The table of contents should be built from h1′s and h3′s.

I’m setting metadata for the author name and cover image. The metadata for the book title will come from the HTML title tag.

I’m setting the output profile as Kindle. Not sure exactly what this does. Seems to work fine without it, but seems sensible to set it.

The last arg is –smarten-punctuation. This turns regular apostrophes, single quotes, and double quotes to their curly equivalents. So I can simply type

"foo"

and get

“foo”

in the output. Also converts single dashes to en dashes and double dashes to em dashes. Brilliantly, this option ignores anything in pre tags!

Once the book is published, the preview target will run. This runs another Calibre command line tool, ebook-viewer, passing in the name of the .mobi file that was just created. This pops open an ebook reader with your book preloaded. This is NOT a Kindle simulator by any means, but it will let you verify that your cover is set, your text looks like what it should, images show up, TOC works, etc.

So again, write something, throw some tags around it, hit F7 and there it is in ebook form. Couldn’t be better. You can then take this compiled .mobi and upload it to Amazon for publishing.

Additional testing. If I’m doing something that I want to make sure looks just right, I have a VM open with Windows running in it. There is a Kindle preview tool that you can download from Amazon that lets you preview how your .mobi will look in a regular Kindle, Kindle DX, Kindle Fire, iPad, iPhone, etc. On Windows and Mac you can also install the Kindle Desktop Reader and make sure your book looks OK there too.

Finally, nothing beats dragging the .mobi into right into a real live Kindle and seeing how it looks on the real live device. There’s actually a Calibre command line tool for transferring a file to a device as well. But with the Kindle, it needs to be plugged in and mounted for transfer, then unmounted to actually read the book. And I don’t test on device every time I build, so I did not make that part of my process.

Here’s the book as seen on a real device:

Oh, and there’s also the code target, which simply launches the specified HTML file in the browser. I just change the default target to code while I’m coding up an example, and back to preview when I’m writing the book.

Summary

Overall, I’m super happy with how this is working out. The big things I’d like to improve are 1. as mentioned, figuring out some possible markdown conversion to simplify formatting and 2. something to possibly automate replacing angle brackets in pre code tags.

Here’s a link to the Calibre command line tools:

http://manual.calibre-ebook.com/cli/cli-index.html

And all the possible options for publishing HTML -> MOBI:

http://manual.calibre-ebook.com/cli/ebook-convert-4.html#html-input-to-mobi-output

I’ve pored over that last link, but I’m sure there’s still some things I could use in there that I haven’t investigated.

Send to Kindle

10 responses so far. Comments will be closed after post is one year old.

  • keith says:

    Another thing I didn’t mention about this setup, but which you might have gleaned from the screenshot, is that the whole thing, book, code, build, is all checked in to a git local repository that is pushed to a remote repo.

  • Josh Tynjala says:

    Great stuff. I don’t see myself writing a book any time soon, but this is exactly the sort of info I’d be looking for.

    I say definitely take a moment to try using Markdown. It’s nice and clean, and I bet that HTML conversion will be a piece of cake.

  • keith says:

    yeah, the markdown was an idea I had while I was writing this article. I’ve already started looking into it using pandoc, another cross-platform command line tool. Looking extremely positive! I’ll probably have a followup post some time tomorrow.

  • [...] I was writing the previous post on my Kindle Authoring Setup, it occurred to me that markdown might be a solution to having to mark [...]

  • Matita says:

    Mark Pilgrim (peace be with his web soul) said he writes books in plain HTML, but his books context is a bit different…
    Here is the interview http://mark.pilgrim.usesthis.com/

    Whatever, thanks for sharing!

  • Mario Pozzo says:

    I’m using Sigil https://code.google.com/p/sigil/ to create an epub from the text, and then converting with Calibre.

    • keith says:

      Interesting, Mario. The name Sigil sounds familiar, but I hadn’t actually checked it out before. Thanks.

  • [...] While I was at it, I also used this experiment to try my hands on a Kindle ebook. I used a setup almost identical to what Keith at BIT-101 is using. [...]

  • Akshay says:

    Wow! We have very similar opinion. I almost though of doing the same thing. I love handcrafting everything in Sublime Text 2. Just curious to know why you didn’t go for grunt? You guessed it write I don’t like Apache ant.

  • Karola says:

    I want to build and publish a photo heavy ebook – about 200 photos going with 200 bits of text. I am not a programmer of any kind so the Sublime Text 2 is probably not for me (hats off to you though for working it all out). I’ve been reading and reading all about Amazon Kindle Publishing and this has been one of the most helpful pages so far (and hasn’t given me migraine). I know Kindle Fire is coming out but now I’m wonding if I should bother working on putting something out that’s so photo heavy.

    I’m:
    A) confused about how to start (have looked at Pressbooks, but no way of previewing it seems, which is crucial to me).
    8) Thinking I might try to look at the Sigl/Calibre route. I have all the content, but alongside the photos there’s information/data that would probably benefit from a table formatting (made to look invisible to the reader). Does Sigl/Calibre sound good? Previewing as I go along is essential. I just need someone to say ‘use this!’ and I will!

    I’m working in Apple’s pages as am on an Apple Mac laptop.

    PS I know about iBooks and am quite happy about getting on with that but wanted to go onto Kindle too. I found Amazon’s manual very hard-going – they talk about paragraph formatting right from the start!