panpipe: 18fc204800828dcd542af227bd7f87be984d8615
1: # PanPipe #
2:
3: This software is released into the Public Domain
4: -- Chris Warburton <chriswarbo@gmail.com>, 2014-09-28
5:
6: ## Usage ##
7:
8: Tested on GNU/Linux, might work on other POSIX systems.
9:
10: You'll need some way to run Haskell. Check your package manager or go to
11: https://www.haskell.org/platform/ to get a compiler or a `runhaskell`
12: interpreter.
13:
14: You'll also need Pandoc available as a library, which you can get from your
15: package manager or with `cabal install pandoc`, and will probably want the
16: `pandoc` command available too.
17:
18: To use PanPipe, invoke it as a Pandoc "filter", like this:
19:
20: `pandoc --filter ./panpipe input_file > output_file`
21:
22: You can also run `panpipe` as a standalone command, but note that its stdio
23: should be in PanDoc's JSON format, rather than e.g. raw Markdown. You can
24: convert *to* PanDoc JSON using a command like `pandoc -t json`, and convert
25: *from* PanDoc JSON using `pandoc -f json`. The `--filter` argument automates
26: this plumbing.
27:
28: ## Intro ##
29:
30: PanPipe is a simple Haskell script using PanDoc. It allows code blocks in
31: PanDoc-compatible documents, eg. Markdown, to be sent to external programs for
32: processing.
33:
34: Any code blocks or lines with a "pipe" attribute will have the contents of that
35: attribute executed as a shell command. The body of the block/line will be piped
36: to that command's stdin, and the stdout will replace the body of that
37: block/line. A non-zero exit code will cause PanPipe to exit with that code;
38: stderr will be sent to PanPipe's stderr.
39:
40: For example, we can execute shell scripts by piping them to "sh":
41:
42: ````
43: ```{pipe="sh"}
44: echo "Hello world"
45: ```
46: ````
47:
48: This will cause "sh" to be called, with 'echo "Hello world"' as its stdin.
49: It will execute the echo command, to produce 'Hello world' as its stdout. This
50: will become the new contents of the code block, so in the resulting document
51: this code block will be replaced by:
52:
53: ````
54: ```
55: Hello world
56: ```
57: ````
58:
59: ## Usage Notes ##
60:
61: ### Attributes ###
62:
63: The "pipe" attribute is removed, but other attributes, classes and IDs remain:
64:
65: ````
66: ```{#foo .bar baz="quux" pipe="sh"}
67: echo 'Hello'
68: ```
69: ````
70:
71: Will become:
72:
73: ````
74: ```{#foo .bar baz="quux"}
75: Hello
76: ```
77: ````
78:
79: ### Execution Order ###
80:
81: PanPipe uses two passes: in the first, all code *blocks* are executed, in the
82: order they appear in the document. Hence later blocks can rely on the effects of
83: earlier ones. For example:
84:
85: ````
86: ```{pipe="sh"}
87: echo "123" > /tmp/blah
88: echo "hello"
89: ```
90:
91: ```{pipe="sh"}
92: cat /tmp/blah
93: ```
94: ````
95:
96: Will become:
97:
98: ````
99: ```
100: hello
101: ```
102:
103: ```
104: 123
105: ```
106: ````
107:
108: The second pass executes *inline* code, in the order they appear in the
109: document.
110:
111: ### Environment ###
112:
113: Commands will inherit the environment from the shell which calls panpipe, except
114: they will all be executed in a temporary directory. This makes it easier to
115: share data between code, without leaving cruft behind:
116:
117: ````
118: ```{pipe="sh"}
119: echo "hello world" > file1
120: echo "done"
121: ```
122:
123: ```{pipe="sh"}
124: cat file1
125: ```
126: ````
127:
128: Will become:
129:
130: ````
131: ```
132: done
133: ```
134:
135: ```
136: hello world
137: ```
138: ````
139:
140: The temporary directory will contain a symlink called "root" which points to
141: wherever Pandoc was called from. This allows resources to be shared across
142: invocations (although it's not recommended to *modify* anything in root).
143:
144: ### Imperative Blocks ###
145:
146: If you want to execute a block for some effect, but ignore its output, you can
147: hide the result using a class or attribute:
148:
149: ````
150: ```{.hidden pipe="python -"}
151: import random
152: with open('entropy', 'w') as f:
153: f.write(str(random.randint(0, 100)))
154: ```
155: ````
156:
157: When rendered to HTML will produce:
158:
159: ```html
160: <pre class="hidden"><code></code></pre>
161: ```
162:
163: ### Program Listings ###
164:
165: A common use-case is to include a program listing in a document *and* show the
166: results of executing it. You can do this by passing the source code to the Unix
167: "tee" command, then using a subsequent shell script to run it:
168:
169: ````
170: ```{.python pipe="tee script1.py"}
171: print "Foo bar baz"
172: ```
173:
174: ```{pipe="sh"}
175: python script1.py
176: ```
177: ````
178:
179: Will become:
180:
181: ````
182: ```{.python}
183: print "Foo bar baz"
184: ```
185:
186: ```
187: Foo bar baz
188: ```
189: ````
190:
191: ### Changing Block Order ###
192:
193: Blocks will always be executed in document-order, so you must arrange dependent
194: blocks appropriately. However, we can display blocks in any order by saving them
195: to files and dumping them later.
196:
197: For example, to show the output of a program *before* its source code listing,
198: we can define the program first, using "tee" to save it to a file and a HTML
199: class to hide the listing in the resulting document:
200:
201: ````
202: ```{.hidden pipe="tee script2.py"}
203: print "Hello world"
204: ```
205: ````
206:
207: Next we can include a block which executes the file we created:
208:
209: ````
210: ```{pipe="sh"}
211: python script2.py
212: ```
213: ````
214:
215: Finally we can include a listing by having a block dump the contents of the file
216: (using ".python" for syntax highlighting):
217:
218: ````
219: ```{.python pipe="sh"}
220: cat script2.py
221: ```
222: ````
223:
224: ### Inline Snippets ###
225:
226: PanPipe also works on inline code snippets; for example, my root filesystem is
227: currently at `` `df -h | grep "/$" | grep -o "[0-9]*%"`{pipe="sh"} `` capacity.
228:
229: ### PanHandle ###
230:
231: PanPipe keeps the results of script execution inside code blocks/lines, where
232: they can't interfere with the formatting. If you want to splice some of these
233: results back into the document, you can use the
234: [PanHandle](http://chriswarbo.net/git/panhandle/) script which was written to
235: complement PanPipe.
236:
237: To prevent ambiguity, PanHandle requires data to be in PanDoc's JSON format. We
238: can convert things to that format using the `-t json` option to the `pandoc`
239: command. For example, to generate a Markdown list and insert it into the
240: document, we can do this:
241:
242: ````
243: ```{.unwrap pipe="python - | pandoc -t json"}
244: for n in range(5):
245: print " - Element " + str(n)
246: ```
247: ````
248:
249: If we run this document through PanPipe, the Python code will output the
250: following to its stdout:
251:
252: ```
253: - Element 0
254: - Element 1
255: - Element 2
256: - Element 3
257: - Element 4
258: ```
259:
260: This will be transformed by `pandoc -t json` into the following:
261:
262: ```
263: [{"unMeta":{}},[{"t":"BulletList","c":[[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"0"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"1"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"2"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"3"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"4"}]}]]}]]
264: ```
265:
266: Hence PanPipe will end up giving out a document equivalent to the following:
267:
268: ````
269: ``` {.unwrap}
270: [{"unMeta":{}},[{"t":"BulletList","c":[[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"0"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"1"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"2"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"3"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"4"}]}]]}]]
271: ```
272: ````
273:
274: Running *that* code through PanHandle will splice the contents into the document
275: to give (a JSON equivalent of):
276:
277: ```
278: - Element 0
279: - Element 1
280: - Element 2
281: - Element 3
282: ```
283:
284: Note that this is no longer in a codeblock, so it will render like this:
285:
286: - Element 0
287: - Element 1
288: - Element 2
289: - Element 3
290:
291: Hence we can use PanPipe to obtain or generate data, and PanHandle to splice it
292: into the document in a sensible way. For example, it's easy to include one
293: Markdown document inside another:
294:
295: ````
296: ```{.unwrap pipe="sh"}
297: cat /some/file.md | pandoc -t json
298: ```
299:
300: ```{.unwrap pipe="sh"}
301: wget -O- http://some.site/some/markdown | pandoc -t json
302: ```
303: ````
Generated by git2html.