Writing "Hands-On Concurrency with Rust"

Hey, I wrote a book!

From late November 2017 to early May 2018 I did very little other than write my book, Hands-On Concurrency with Rust. In this post I'd like to tell you a little bit about the book itself—which, miracrulously, is available for purchase—but also the process of writing it. I've done a number of elaborate conference talks over the years and written a good deal of software but this is the first time I'd seriously attempted a book. Truthfully, I didn't know what I was doing at the outset but I did it with gusto and kept at it and here we are. There's a book now. Let me tell you all about the work.

What's the book about?

First, what's Hands-On Concurrency with Rust about? Hopefully you can guess some of the subject from the title. It's a Rust-focused book that's meant to teach you, as of 2018, what you can do in Rust to fiddle with modern, commodity parallel machines. The ambition I had at the outset of the project was to lay out all the work I do professionally when I'm doing parallel programming. How do you go about imagining the machine itself and structure your data effectively for it? How do you validate the good function of concurrent implementations, especially across different CPUs? How do you debug these things? What crates in the Rust ecosystem ought you to know about? How do these crates actually work?

The book aims to teach mechanical sympathy for modern machines. We do a deep-dive into the Rust compiler, we use atomic primitives to build up mutexes and the like, examine atomic-safe garbage collection methods, investigate the internal mechanism of Rayon and a ton more. Here's the chapter titles, which I tried to make as on-the-nose as possible:

1. Preliminaries: Machine Architecture and Getting Started with Rust
2. Sequential Rust Performance and Testing
3. The Rust Memory Model: Ownership, References and Manipulation
4. Sync and Send: the Foundation of Rust Concurrency
5. Locks: Mutex, Condvar, Barriers and RWLock
6. Atomics: The Primitives of Synchronization
7. Atomics: Safely Reclaiming Memory
8. High-Level Parallelism: Threadpools
9. FFI and Embedding: Combining Rust and Other Languages
10. Futurism: Near-Term Rust

When writing out the book outline, I imagined the concepts arranged in an inverted pyramid: the first chapter—Preliminaries—is the base of everything. In this chapter I discuss the CPU itself, how it operates and what features of that operation are especially apropos to the book. The second chapter introduces the consequences of, say, cache hierarchies introduced in Preliminaries and motivates perf, valgrind and the use of lldb/gdb from that. The chapter Locks discusses the semantics of coarse locking and Atomics: The Primitives of Synchronization then takes all that materials and re-builds it from scratch. But, once you start doing this it becomes clear that memory reclamation is a hard problem, motivating Atomics: Safely Reclaiming Memory. In this fashion, the book builds on itself.

Okay, well, I mentioned an outline? How exactly does one go about writing a technical book?

How 'Hands-On Concurrency' was written.

The kind folks at Packt Publisher reached out to me in late November 2017 wanting to develop a video course on concurrent programming in Rust. Putting together conference presentations is something I adore doing so I was immediately interested in putting together a video script. I found, also, through my work on cernan that I was having repeat conversations about the structure and validation of parallel Rust systems and been (unsuccessfully) putting notes together on the topic. The lack of success was a result of size limitations: I figured a note on the subject suitable for cernan development should be no more than 50 pages or so. This very blog contains some of that work in the hopper series, which needs but one final article to be complete.

Anyhow, the topic is one I'd been thinking on for a while so that a video course was falling into my lap seemed real opportune.

From video course to book.

Why did I end up writing a book? I had a very enjoyable Skype conversation with my initial acquisition editor—this is the person that scouts new authors and shops subjects to the same—and it became clear to them that my personal ambitions for a video course was overly ambitious for such a thing. By, like, a lot. Here's what I wrote in my journal on the subject:

I aim to produce a path into the field. Each chapter should demand the next. Each chapter, it should be clear, must sit on top of a great body of research.

The eventual book came to just under 100k words—Thoreau's Walden, for reference, is 114k—and 7k lines of source code. Imagine how horrible it would have been to sit through the videos of all that. (The audiobooks of Walden, for reference, run around 11 hours.) The acquisition editor was forthright and said that it was plainly too much material for a video course but that Packt was interested in developing the project as a book as well.

In 2015 I put together one talk a month for six months, flying around to present them in the process. I remember that time as difficult, but doable. "How hard can a book be?" I asked myself. "Treat a chapter like a talk and work them one after the other. Doable."

Here's the secret about my talks: they are simple subjects presented in great detail. Each talk is one clean and small idea supported by careful selection of photographs, production of illustrations and typographic design. The longest to prepare talk I've ever done, Getting Uphill on a Candle, took around 200 hours, give or take. It's also the most complex I've put together, as well, being that it's a history of aeronautics research through the 20th century. Most every other talk takes around 100 hours which I put together over weekends and nights after work. That's not a modest amount of time but is possible to turn-around in a month with familiar subject material. If you were to include the background research time into the preparation estimate, well, you'd be looking at closer to 1k hours, easy. Most are the result of years long cycles of reading, trips to the book store and talking over the particulars with friends.

I forgot that.

Here's my first naive mistake with the book and I did it in the initial phone call: I assumed that because I could use a body of knowledge in my work I was familiar with the material enough to teach it.

"Sounds fun. Let me put together an outline." I told the editor.

What is an outline?

At least with Packt, the book outline serves two purposes. Firstly, it informs the editors of the intention of your book project: these are the readers I expect, these are the things I intend to cover, these are the things I don't intend to cover. Secondly, the outline serves to structure the writing project itself. I wrote a summary of each chapter, describing how the chapters would call back to previous ones and influence those to come. This was very valuable four months in when I was tired and maybe had lost a bigger sense of the project.

The outline didn't end up being the book I wrote. I'd assumed that the seventh chapter, Atomics: Safely Reclaiming Memory, would be a smaller section in the sixth, Atomics: The Primitives of Synchronization. As it happens, chapter seven is one of the longest in the book!

If you're interested, you can find the outline we went with on Dropbox, here. The book eventually shipped with ten chapters, not twelve, and the working title Rust Concurrency didn't survive through to the end of the project.

But! By December 23 I had word the publisher was enthusiastic for the book idea, contracts were signed and I was off.

Laying down chapters

To help me keep tempo on the project I wrote a little program called coach that would inspect a yaml file with chapter/date goals and tell me how much I needed to write each day. Here's the schedule file:

---
global:
  total_pages: 350
  words_per_page: 300
chapters:

sequence_number: 01
total_pages: 20
preliminary_draft: 2018-01-10
final_draft: 2018-04-29
sequence_number: 02
total_pages: 35
sufficient_wordcount: true
preliminary_draft: 2018-01-22
final_draft: 2018-05-02
sequence_number: 03
total_pages: 35
preliminary_draft: 2018-02-01
final_draft: 2018-05-05
sequence_number: 04
total_pages: 25
preliminary_draft: 2018-02-09
final_draft: 2018-05-08
sequence_number: 05
total_pages: 30
preliminary_draft: 2018-02-19
final_draft: 2018-05-11
sequence_number: 06
total_pages: 40
preliminary_draft: 2018-03-04
final_draft: 2018-05-14
sequence_number: 07
total_pages: 20
preliminary_draft: 2018-03-11
final_draft: 2018-05-17
sequence_number: 08
total_pages: 15
preliminary_draft: 2018-03-16
final_draft: 2018-05-20
sequence_number: 09
total_pages: 45
preliminary_draft: 2018-03-29
final_draft: 2018-05-23
sequence_number: 10
total_pages: 20
preliminary_draft: 2018-04-05
final_draft: 2018-05-26
sequence_number: 11
total_pages: 40
preliminary_draft: 2018-04-18
final_draft: 2018-05-29
sequence_number: 12
total_pages: 20
preliminary_draft: 2018-04-25
final_draft: 2018-06-01

The schedule here is intense in a way I didn't fully appreciate at the outset. But, the general notion of doing a little bit of work each day—usually in the morning—is a style of work I find very comfortable. Up through chapter three, approximately, I managed to keep up with the schedule. coach would say it was a 500 word day and I'd crank through that before work, over lunch breaks (carting my personal laptop back and forth to work) and again after work. Saturdays were fairly well taken over as uninterrupted writing days. This pattern held all through the project—I don't have children and my wife works long hours as a chef with weekends offset from my own—but broke down once I'd got through chapter three.

I forgot about software.

If you have a look at the book's code repository and take a peak in the chapter sub-directories you'll find that the amount of software I wrote custom for the chapters increased markedly between chapter three and four and then again from chapter four to five. The software written for the book is tested (with quickcheck, often enough), probed with fuzzers and verified in a variety of other ways I go into in the text. Meaning, I found a good deal of bugs in the custom software and spent much more time than expected on, say, a single section out of a chapter. I wrote the book serially—hah!—and without any co-authors: if I couldn't figure out a bug I couldn't advance the book.

Judging by my commits, here's how the book actually proceeded:

Each time you see a big jump between the end of a chapter and beginning of the next, that's me struggling to get the software for that chapter ready. Without the software, there's not much to talk about in Sync and Send. Each time there's a chapter that goes on for a while, like the Atomics chapters, I'm moving slowly from one section to the next as I produce the software to write about it.

Ultimately, the schedule the publisher and I worked up was just not relevant to the day to day work of putting the book together and I started ignoring it for my own well-being. I'm sure that my editors would be not at all surprised by that information.

My writing environment

Packt Publishers has a workflow built around a thing called Type Cloud, which, near as I can tell, is built on top of Wordpress. Type Cloud is a WYSIWYG browser-based editor and ended up being very effective for the final proofing stage of the book, going through and fixing up formatting issues or incorporating technical reviewer feedback. For writing, at least for me, Type Cloud did not present a good working environment. Part of the project deal was that I'd send specially formatted HTML-like markup that Type Cloud uses internally to my content editor—more on the different kinds of editors below—and they'd import it into the WYSIWYG environment.

My understanding is that other publishers work end-to-end in Asciidoc, LaTeX or similar. That sounds nice.

I wrote in Pandoc markdown, one file per chapter. The source code lived in a separate directory structure from the chapter source and I included it with pandoc-include-code. I'm not at all sure how I would have produced the book without this plugin.

A great deal of care was made to ensure that readers wishing to type out the source code of a book could get a working project on the other end. Me, I find it beneficial to type-along to a text and get really frustrated when the source code is missing something important. Also, it's not impossible that the source code for a book won't be available online a few decades after having been initially published: even worse! I, my editors and the technical reviewers all went through and checked that you could read the book in this way.

The editing team

There were a good number of other people that worked on this book with me, each with different jobs. I worked with most everyone through email or Type Cloud's inline commenting scheme. The roles:

• acquisition editor, discussed the initial project with me, worked to finalize the outline
• content editor, my day to day contact and helped shepherd the overall tone of the book
• technical editor, where the 'content' editor was concerned with the prose of the book the 'technical' editor was concerned with the software side of the house
• copy editors, this small crowd of people went through the text, correcting typos, asking about awkward wording, ensured the typographic conventions were uniform throughout and asked me not to begin sentences with "Which, ..."
• technical reviewer, this person typed out all the book source code, called out any questionable assertions in the text and generally read the book as a reader would, advocating their position

Each of these people had a different take on the book. Some takes I agreed with, some strayed from my original intentions. The copy edit stage—this happens after the first draft is submitted and before the draft is considered final—was kind of rough in this regard. I fully admit I am particular about how I word things and generally read material aloud to make sure the language flows in the way I'd say it. (I will have read this article aloud a few times before you've read it.) My other major naive mistake was to believe that my sense of a chapter would make it through from the first draft to the final. It mostly did, but if a couple of copy editors insisted on a phrase change I'd have to ask myself if this was vital to my intention or just part of my offbeat idiolect.

Final thoughts

Now that I've gone through all the work to write a book it's clear to me that I would love to write another. I'm not sure on what subject but I guess we'll see here in the next five years or so. I think I could, very reasonable, take a few of the chapters from Hands-On Concurrency with Rust and expand them into something stand-alone.

That said, I absolutely will not write one in five/six months again and probably in the future cannot. I bet the life circumstances that allowed the book to be written in as brief a time as I took are temporary. In retrospect I can't rightly understand how the book was written so quickly and I hope, as people go through it, they don't come away feeling that it was rushed. Me, I've read the book through several times and don't think it reads rushed but I also don't have much space from it.

The process of producing a book in secret and then gating access to the material behind payment seems odd to me. I know that's how it's mostly done. But, a model like Learn You Some Erlang for great good!, Real World Haskell or Crafting Interpreters—where the text of the book is available online during the writing and you can contact the authors meanwhile but if you want a physical copy you gotta pony up—seems like a better way to spread the ideas of a book and maybe a way to build a broader audience. This impression is driven partly by the collaborative, open model of software development I've grown up around and a sense that I won't make a living writing books. My motivation is to disseminate ideas. Making money sweetens the pot. It's similar for talks: I want to get some idea across and being able to travel around to do so is a really nice treat on the side.

Anyhow, if you have read or intend to read Hands-On Concurrency with Rust I sure do thank you. It was a real pleasure to put it together and I hope it has or will manage to teach you something.