panpipe: 18fc204800828dcd542af227bd7f87be984d8615

     1: # PanPipe #
     2: 
     3: This software is released into the Public Domain
     4:   -- Chris Warburton <chriswarbo@gmail.com>, 2014-09-28
     5: 
     6: ## Usage ##
     7: 
     8: Tested on GNU/Linux, might work on other POSIX systems.
     9: 
    10: You'll need some way to run Haskell. Check your package manager or go to
    11: https://www.haskell.org/platform/ to get a compiler or a `runhaskell`
    12: interpreter.
    13: 
    14: You'll also need Pandoc available as a library, which you can get from your
    15: package manager or with `cabal install pandoc`, and will probably want the
    16: `pandoc` command available too.
    17: 
    18: To use PanPipe, invoke it as a Pandoc "filter", like this:
    19: 
    20: `pandoc --filter ./panpipe input_file > output_file`
    21: 
    22: You can also run `panpipe` as a standalone command, but note that its stdio
    23: should be in PanDoc's JSON format, rather than e.g. raw Markdown. You can
    24: convert *to* PanDoc JSON using a command like `pandoc -t json`, and convert
    25: *from* PanDoc JSON using `pandoc -f json`. The `--filter` argument automates
    26: this plumbing.
    27: 
    28: ## Intro ##
    29: 
    30: PanPipe is a simple Haskell script using PanDoc. It allows code blocks in
    31: PanDoc-compatible documents, eg. Markdown, to be sent to external programs for
    32: processing.
    33: 
    34: Any code blocks or lines with a "pipe" attribute will have the contents of that
    35: attribute executed as a shell command. The body of the block/line will be piped
    36: to that command's stdin, and the stdout will replace the body of that
    37: block/line. A non-zero exit code will cause PanPipe to exit with that code;
    38: stderr will be sent to PanPipe's stderr.
    39: 
    40: For example, we can execute shell scripts by piping them to "sh":
    41: 
    42: ````
    43: ```{pipe="sh"}
    44: echo "Hello world"
    45: ```
    46: ````
    47: 
    48: This will cause "sh" to be called, with 'echo "Hello world"' as its stdin.
    49: It will execute the echo command, to produce 'Hello world' as its stdout. This
    50: will become the new contents of the code block, so in the resulting document
    51: this code block will be replaced by:
    52: 
    53: ````
    54: ```
    55: Hello world
    56: ```
    57: ````
    58: 
    59: ## Usage Notes ##
    60: 
    61: ### Attributes ###
    62: 
    63: The "pipe" attribute is removed, but other attributes, classes and IDs remain:
    64: 
    65: ````
    66: ```{#foo .bar baz="quux" pipe="sh"}
    67: echo 'Hello'
    68: ```
    69: ````
    70: 
    71: Will become:
    72: 
    73: ````
    74: ```{#foo .bar baz="quux"}
    75: Hello
    76: ```
    77: ````
    78: 
    79: ### Execution Order ###
    80: 
    81: PanPipe uses two passes: in the first, all code *blocks* are executed, in the
    82: order they appear in the document. Hence later blocks can rely on the effects of
    83: earlier ones. For example:
    84: 
    85: ````
    86: ```{pipe="sh"}
    87: echo "123" > /tmp/blah
    88: echo "hello"
    89: ```
    90: 
    91: ```{pipe="sh"}
    92: cat /tmp/blah
    93: ```
    94: ````
    95: 
    96: Will become:
    97: 
    98: ````
    99: ```
   100: hello
   101: ```
   102: 
   103: ```
   104: 123
   105: ```
   106: ````
   107: 
   108: The second pass executes *inline* code, in the order they appear in the
   109: document.
   110: 
   111: ### Environment ###
   112: 
   113: Commands will inherit the environment from the shell which calls panpipe, except
   114: they will all be executed in a temporary directory. This makes it easier to
   115: share data between code, without leaving cruft behind:
   116: 
   117: ````
   118: ```{pipe="sh"}
   119: echo "hello world" > file1
   120: echo "done"
   121: ```
   122: 
   123: ```{pipe="sh"}
   124: cat file1
   125: ```
   126: ````
   127: 
   128: Will become:
   129: 
   130: ````
   131: ```
   132: done
   133: ```
   134: 
   135: ```
   136: hello world
   137: ```
   138: ````
   139: 
   140: The temporary directory will contain a symlink called "root" which points to
   141: wherever Pandoc was called from. This allows resources to be shared across
   142: invocations (although it's not recommended to *modify* anything in root).
   143: 
   144: ### Imperative Blocks ###
   145: 
   146: If you want to execute a block for some effect, but ignore its output, you can
   147: hide the result using a class or attribute:
   148: 
   149: ````
   150: ```{.hidden pipe="python -"}
   151: import random
   152: with open('entropy', 'w') as f:
   153:     f.write(str(random.randint(0, 100)))
   154: ```
   155: ````
   156: 
   157: When rendered to HTML will produce:
   158: 
   159: ```html
   160: <pre class="hidden"><code></code></pre>
   161: ```
   162: 
   163: ### Program Listings ###
   164: 
   165: A common use-case is to include a program listing in a document *and* show the
   166: results of executing it. You can do this by passing the source code to the Unix
   167: "tee" command, then using a subsequent shell script to run it:
   168: 
   169: ````
   170: ```{.python pipe="tee script1.py"}
   171: print "Foo bar baz"
   172: ```
   173: 
   174: ```{pipe="sh"}
   175: python script1.py
   176: ```
   177: ````
   178: 
   179: Will become:
   180: 
   181: ````
   182: ```{.python}
   183: print "Foo bar baz"
   184: ```
   185: 
   186: ```
   187: Foo bar baz
   188: ```
   189: ````
   190: 
   191: ### Changing Block Order ###
   192: 
   193: Blocks will always be executed in document-order, so you must arrange dependent
   194: blocks appropriately. However, we can display blocks in any order by saving them
   195: to files and dumping them later.
   196: 
   197: For example, to show the output of a program *before* its source code listing,
   198: we can define the program first, using "tee" to save it to a file and a HTML
   199: class to hide the listing in the resulting document:
   200: 
   201: ````
   202: ```{.hidden pipe="tee script2.py"}
   203: print "Hello world"
   204: ```
   205: ````
   206: 
   207: Next we can include a block which executes the file we created:
   208: 
   209: ````
   210: ```{pipe="sh"}
   211: python script2.py
   212: ```
   213: ````
   214: 
   215: Finally we can include a listing by having a block dump the contents of the file
   216: (using ".python" for syntax highlighting):
   217: 
   218: ````
   219: ```{.python pipe="sh"}
   220: cat script2.py
   221: ```
   222: ````
   223: 
   224: ### Inline Snippets ###
   225: 
   226: PanPipe also works on inline code snippets; for example, my root filesystem is
   227: currently at `` `df -h | grep "/$" | grep -o "[0-9]*%"`{pipe="sh"} `` capacity.
   228: 
   229: ### PanHandle ###
   230: 
   231: PanPipe keeps the results of script execution inside code blocks/lines, where
   232: they can't interfere with the formatting. If you want to splice some of these
   233: results back into the document, you can use the
   234: [PanHandle](http://chriswarbo.net/git/panhandle/) script which was written to
   235: complement PanPipe.
   236: 
   237: To prevent ambiguity, PanHandle requires data to be in PanDoc's JSON format. We
   238: can convert things to that format using the `-t json` option to the `pandoc`
   239: command. For example, to generate a Markdown list and insert it into the
   240: document, we can do this:
   241: 
   242: ````
   243: ```{.unwrap pipe="python - | pandoc -t json"}
   244: for n in range(5):
   245:     print " - Element " + str(n)
   246: ```
   247: ````
   248: 
   249: If we run this document through PanPipe, the Python code will output the
   250: following to its stdout:
   251: 
   252: ```
   253:  - Element 0
   254:  - Element 1
   255:  - Element 2
   256:  - Element 3
   257:  - Element 4
   258: ```
   259: 
   260: This will be transformed by `pandoc -t json` into the following:
   261: 
   262: ```
   263: [{"unMeta":{}},[{"t":"BulletList","c":[[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"0"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"1"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"2"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"3"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"4"}]}]]}]]
   264: ```
   265: 
   266: Hence PanPipe will end up giving out a document equivalent to the following:
   267: 
   268: ````
   269: ``` {.unwrap}
   270: [{"unMeta":{}},[{"t":"BulletList","c":[[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"0"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"1"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"2"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"3"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"4"}]}]]}]]
   271: ```
   272: ````
   273: 
   274: Running *that* code through PanHandle will splice the contents into the document
   275: to give (a JSON equivalent of):
   276: 
   277: ```
   278:  - Element 0
   279:  - Element 1
   280:  - Element 2
   281:  - Element 3
   282: ```
   283: 
   284: Note that this is no longer in a codeblock, so it will render like this:
   285: 
   286:  - Element 0
   287:  - Element 1
   288:  - Element 2
   289:  - Element 3
   290: 
   291: Hence we can use PanPipe to obtain or generate data, and PanHandle to splice it
   292: into the document in a sensible way. For example, it's easy to include one
   293: Markdown document inside another:
   294: 
   295: ````
   296: ```{.unwrap pipe="sh"}
   297: cat /some/file.md | pandoc -t json
   298: ```
   299: 
   300: ```{.unwrap pipe="sh"}
   301: wget -O- http://some.site/some/markdown | pandoc -t json
   302: ```
   303: ````