Active Code

"Active code" is the term used by the Babel system, part of Emacs's Org-mode. It refers to authoring systems which can execute code embedded in the documents they're rendering. This page documents the active code system I use to write articles, whether they're published as HTML on my Web site or PDFs in a journal.

Why not Babel?

Babel, Org mode and Emacs are all wonderful things, however there are a few reasons why we may want to avoid them, which can be summarised by saying "they're not UNIX":

The Alternative: Pandoc

Pandoc is a great document conversion program by John MacFarlane. It can convert between various markup languages, including HTML, LaTeX and Markdown. We can also mix and match the formats, for example embedding a mixture of HTML and LaTeX in a Markdown document and rendering it to a PDF.

In particular, most of the source of my Web site is written in Markdown and converted to HTML using Pandoc. I use Make to orchestrate the process.

Embedded Code

The following examples are all written in Pandoc's Markdown, but apply to the other formats supported by Pandoc too.

Pandoc supports code blocks, which are written like this in Markdown:

`echo "Inline code"`
    echo "Indented code"
```
echo "Fenced code"
```

As you can see, these get rendered in monospaced fonts. Code blocks can have "attributes", "classes" and an "id", like this:

```{#SomeID .SomeClass .AnotherClass Attribute1="Value1" Attribute2="Value2"}
Some content
```

This lets us manipulate blocks, for example if we're rendering to HTML we might use these IDs, classes and attributes from some associated Javascript. Pandoc also uses classes to apply syntax-highlighting, based on language descriptions from Kate.

This is the standard, off-the-shelf way to embed code snippets in a document. However, such code is not active.

PanPipe

PanPipe is a Pandoc filter which walks the document tree looking for code blocks annotated with a pipe attribute, like this:

```{pipe="sh"}
echo "Hello world!"
```

When such a block is found, the following steps take place:

For example, running the above through pandoc --filter panpipe gives:

Hello world!

PanHandle

PanHandle is a Pandoc filter which looks for code in an unwrap class. It extracts the code, which is assumed to be in 'Pandoc JSON' format, and splices it into the surrounding document.

We can turn any Pandoc-supported format into Pandoc JSON by piping it through pandoc -t json

For example, if we take the JSON for this Markdown table:

X NOT(X)
- ------
T F
F T

and wrap it in an unwrap code block, we get:

```{.unwrap}
[{"unMeta":{}},[{"t":"Table","c":[[],[{"t":"AlignDefault","c":[]},{"t":"AlignDefault","c":[]}],[0,0],[[{"t":"Plain","c":[{"t":"Str","c":"X"}]}],[{"t":"Plain","c":[{"t":"Str","c":"NOT(X)"}]}]],[[[{"t":"Plain","c":[{"t":"Str","c":"T"}]}],[{"t":"Plain","c":[{"t":"Str","c":"F"}]}]],[[{"t":"Plain","c":[{"t":"Str","c":"F"}]}],[{"t":"Plain","c":[{"t":"Str","c":"T"}]}]]]]}]]
```

When we send our document through pandoc --filter panhandle, the table will be spliced into the document, like this:

X NOT(X)
T F
F T

Examples

Some non-toy examples of this system in action:

This Site

This whole site is static HTML generated from Markdown with these tools. Not every page takes advantage of these capabilities, but it's nice to know they're available when I need them. You may like to browse this page's source to see how the example output is derived straight from the examples themselves (note that this requires meta-programming, which complicates things a little).

Fibonacci Post

I wrote PanPipe and PanHandle after trying and failing to integrate Babel into my site's build process. My Fibonacci Sequence in PHP post was an experiment with Babel, so porting that post over to Pandoc was the motivating use-case for these scripts. Thankfully the port was a success, and that post is now managed by Make like the rest of the site.

If you compare it to the source you'll see a few of the required features which influenced my thinking:

Useful Tricks

These simple scripts let us call out to the UNIX shell from our documents. This lets us recreate many of the active code features of Babel, just by piping between programs and reading/writing files. Here are some common tasks you may want to solve:

Hiding Output

You may want a code block to execute, but not show up in the output. The easiest way is to pipe the output to /dev/null, or an actual file if we plan to use it later:

```{pipe="sh > /dev/null"}
ls /
```

```{pipe="sh > contents"}
ls /
```

This works well for HTML, and results in:

<pre><code></code></pre>
<pre><code></code></pre>

Sometimes these empty elements may have undesirable effects, e.g. interacting badly with some styling rule. If this is the case, you might try using inline snippets instead, eg. `ls / `{pipe="sh > /dev/null"} , which gives <p><code></code></p> .

Splicing Nothing

To eliminate the code block takes a little more effort, but might be necessary in some cases. To remove a code block, we can use panhandle to splice an empty document in its place.

Remember that panhandle accepts JSON, which we can generate using pandoc:

```{.unwrap pipe="sh | echo '' | pandoc -t json"}
ls /
```

Here's the result when converting to HTML:


Ta da! If our code block has any extra attributes, etc. then a div will be left behind to catch them, for example:

```{.unwrap pipe="sh | echo '' | pandoc -t json" myattr="myvalue"}
ls /
```

This gives:

<div myattr="myvalue">

</div>

Format-specific

If you're targetting a specific output format, you can use techniques specific to that format.

For example, if you're rendering to HTML, you can hide code blocks with CSS:

```{pipe="sh" style="display: none;"}
ls /
```

This results in:

<pre style="display: none;"><code>bin
boot
cfg
dev
etc
home
lost+found
nix
proc
root
run
sys
tmp
usr
var
</code></pre>

If you're using LaTeX you can use if statements to skip over the block:

\iffalse

```{pipe="sh"}
ls /
```

\fi

Showing Code and Output

We can use tee to save a copy of our code into a file, then run it in another code block:

```{.php pipe="tee script.php"}
<?php
echo 10 + 20;
```
```{pipe="sh"}
php script.php
```

This results in:

<?php
echo 10 + 20;
30

Tangling

Use tee -a to append to a file. Make sure to include extra newlines as needed:

```{.haskell pipe="tee -a tangled.hs"}
foo = "Hello"

```

```{.haskell pipe="tee -a tangled.hs"}
bar = "World"

```

```{.haskell pipe="ghci -v0"}
:load tangled.hs
print (foo ++ " " ++ bar)
```

This gives:

foo = "Hello"
bar = "World"
"Hello World"

Execution Order

Blocks are executed from the top of a document to the bottom. We can change the order they're displayed in by capturing their output to files and dumping them out later. For example, to show a program listing after its results:

```{pipe="cat > code.sh"}
echo "Hello"
echo "World"
```

```{pipe="sh"}
sh code.sh
```

```{.bash pipe="sh"}
cat code.sh
```

This produces:

Hello
World
echo "Hello"
echo "World"

Procedural Documents

We can generate content using PanPipe, send it through Pandoc to get JSON, then use PanHandle to splice it into the document. For example:

```{.unwrap pipe="php | pandoc -t json"}
<?php
foreach (range(1, 10) as $x) {
  echo " - Element $x\n";
}
```

This produces:

Importing Sub-Documents

We can use PanPipe to dump the contents of files and PanHandle to combine them together. We can even call Pandoc recursively:

```{.unwrap pipe="sh"}
pandoc -t json header.md
```

```{.unwrap pipe="sh"}
pandoc -t json footer.md
```

Including Images

We can obtain image files using PanPipe, then encode them in data URIs. PanHandle will splice these into the document:

```{pipe="php > carpet.pbm"}
<?php
$scale = 5;
$dim   = pow(3, $scale);
$max   = ($dim * $dim) - 1;

function carpet($x, $y) {
  if ($x % 3 == 1 && $y % 3 == 1) return 0;
  return ($x || $y)? carpet(intval($x / 3),
                            intval($y / 3))
                   : 1;
}

$colour = function($c) use ($dim) {
  $x =  $c       % $dim;
  $y = ($c - $x) / $dim;
  return carpet($x, $y);
};

echo "P1 $dim $dim\n";
echo implode("\n", array_map($colour, range(0, $max)));
```

```{.unwrap pipe="sh | pandoc -t json"}
convert carpet.pbm carpet.png
echo -n '<img alt="Sierpinski Carpet" src="data:image/png;base64,'
base64 -w 0 carpet.png
echo -n '" />'
```

This results in:

Sierpinski Carpet

Handling Errors

In general, errors should abort the rendering. We would rather have no document than an erroneous one.

If you want to trigger an error from a script, just have it return a non-zero exit code:

```{pipe="sh"}
if [ ! -e "foo" ]
then
  exit 1
fi
cat foo
```

If you want to carry on rendering in the presence of errors, you must implement some kind of error handling. For example:

```{pipe="sh"}
./dodgyCommand || echo "dodgyCommand didn't work; oh well!"
```

Anything printed to stderr by a shell command will appear in the stderr of PanPipe. Likewise, when used as a Pandoc filter, PanPipe's stderr will appear in Pandoc's stderr. Note that Pandoc may buffer the stderr stream, which prevents content showing immediately (eg. progress bars and such). To prevent this, you can use pandoc -t json | panpipe | panhandle | pandoc -f json rather than pandoc --filter panpipe --filter panhandle.