Understanding ob-shell.el

Why isn't async the default?

As far as I can tell, it's because Carsten used shell-command-on-region in the original ob-eval.el.

(defun org-eval-run (cmd code)
  (with-temp-buffer
    (insert code)
    (shell-command-on-region (point-min) (point-max) cmd nil 'replace)
    (buffer-string)))

In the early days of Babel, collaboration notes were kept. This started as rorg.org and was later renamed to org-babel.org.

The rorg.org in commit e770100 mentions development for an ob-eval-light on March 31, 2009:

org-eval was written by Carsten. It lives in the org/contrib/lisp directory because it is too dangerous to include in the base. Unlike org-eval-light org-eval evaluates all source blocks in an org-file when the file is first opened, which could be a security nightmare for example if someone emailed you a pernicious file.

All evaluations at this time happen with shell-command-on-region:

• org-eval-run May 6, 2008 commit 62fc2c1
• org-eval-light-run March 31, 2009 commit e7701000
• all the "execute:lang"-type functions (see How blocks execute to obtain results)

The last known update to rorg.org (later renamed to org-babel.org) 2d2a2a19 says,

,*** TODO evaluation of shell code as background process? After C-c C-c on an R code block, the process may appear to block, but C-g can be used to reclaim control of the .org buffer, without interrupting the R evalution. However I believe this is not true of bash/sh evaluation. [Haven't tried other languages] Perhaps a solution is just to background the individual shell commands.

The other languages (aside from emacs lisp) are run through the shell, so if we find a shell solution it should work for them as well.

Adding an ampersand seems to be a supported way to run commands in the background (see external-commands). Although a more extensible solution may involve the use of the call-process-region function.

Going to try this out in a new file org-babel-proc.el. This should contain functions for asynchronously running generic shell commands in the background, and then returning their input.

I can find no record of org-babel-proc.el:

cd /tmp/org-mode && pwd && git log --all -- '*org-babel-proc.el*'
cd /tmp/org-mode && pwd && git log --all --full-history -- org-babel-proc.el
cd /tmp/org-mode && pwd && git log --all --full-history -- **/org-babel-proc.*
cd /tmp/org-mode && pwd && git log --all --full-history -- *org-babel-proc*
cd /tmp/org-mode && pwd && git log --all --full-history -- org-babel-proc*

/tmp/org-mode
/tmp/org-mode
/tmp/org-mode
/tmp/org-mode
/tmp/org-mode

org-babel.org (bb2969ef) (quoted above) suggests "a more extensible solution may involve the use of the call-process-region function". The thought was sap that "adding an ampersand seems to be a supported way to run commands in the background." This appears based on information from the EmacsWiki.

Unfortunately, call-process-region is synchronous and the suggestion is appears ill-informed (although that may simply be the fog of time). See How many different ways are there to execute a shell block?

Today, process-file is used instead of call-process-on-region. Yet process-file is a wrapper around call-process and is still synchronous.

I think an investigation was made, no clear answer or path figured out, and focus shifted elsewhere. After the initial development push, async has come up a few times on the mailing list:

• (February 29, 2012) babel, executing code in background process
• (September 4, 2013) Re: asynchronous code evaluation
• (November 18, 2015) long running processes
• (November 20, 2015) asynchronous python code blocks in org-mode
• (April 6, 2016) [PATCH] new :async feature for org-babel-clojure
• (July 2, 2019) Asynchronous session evaluation
• (March 20, 2023) ob-async
• (October 25, 2020) [PATCH] Async session eval (2nd attempt)
• (September 26, 2021) [PATCH] async process in R
• (November 22, 2021) [PATCH] ob-shell-test, test-ob-shell and introduction

Ultimately, I would like ob-shell to be more robust and have more functionality. It's limited presently by ob-comint.el and ob-eval.el.

This is an ob-comint/ob-eval issue.

Async depends on make-process not on comints. Currently, sessions are the only way to run something asynchronously. That doesn't make sense for some languages. That doesn't mean they can't work asynchronously.

This is an ob-comint/ob-eval/ob-core issue.

The only documentation is ob-template.el.

On-hold until I can find the brain power to dig into kernel drivers and terminal I/O more.

The following is Linux centric because that's what information is available. Other resources included MINIX. All of these are inspired by UNIX. Maybe it's more accurate to say it leans POSIX.

A "shell" is a command driven process launcher. It reads input, evaluates it, prints results, and loops back to reading input. The following demonstrates how a simple shell operates. It's easy to imagine the example extended to receive input in batch. Such batch input to a shell is a "script."

/*
  * https://stevens.netmeister.org/631/
  */

#include <sys/types.h>
#include <sys/wait.h>

#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sysexits.h>
#include <unistd.h>

char *
getinput (char *buffer, size_t buflen)
{
  printf ("$$ ");
  return fgets (buffer, buflen, stdin);
}

int
main (int argc, char **argv)
{
  char buf[BUFSIZ];
  pid_t pid;
  int status;

  /* cast to void to silence compiler warnings */
  (void) argc;
  (void) argv;

  while (getinput (buf, sizeof (buf)))
    {
      buf[strlen (buf) - 1] = '\0';

      if ((pid = fork ()) == -1)
        {
          fprintf (stderr, "shell: can't fork: %s\n", strerror (errno));
          continue;
        }
      else if (pid == 0)
        {                       /* child */
          execlp (buf, buf, (char *) 0);
          fprintf (stderr, "shell: couldn't exec %s: %s\n", buf,
                   strerror (errno));
          exit (EX_UNAVAILABLE);
        }

      /* parent waits */
      if ((pid = waitpid (pid, &status, 0)) < 0)
        {
          fprintf (stderr, "shell: waitpid error: %s\n", strerror (errno));
        }
    }

  exit (EX_OK);
}

POSIX defines a shell as:

A program that interprets sequences of text input as commands. It may operate on an input stream or it may interactively prompt and read commands from a terminal.

– https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_347

A "terminal" is an interface to a shell. A shell simply "interprets sequences of text" for the purpose of starting new processes. A terminal manages the sequences of text sent and received from a shell. The terms "console" and "command-line" are also used.

Historically, a terminal was a physical device. Early computers repurposed Teletypes, electrical mechanical devices for sending and receiving telegrams, for input and output. As computers developed, so did terminals. Gradually, terminals went from recycled devices resembling typewriters to simple computers with memory, screens, and basic computational ability. Still, the primary purpose of a terminal is preparing text for a shell and outputting text from the shell to the user.

Terminals prepare text in one of two ways: character-wise or line-wise. For example, if the user inputs "dsta", realizes their mistake, presses backspace three times, and then inserts "ata" before pressing enter, the shell receives 11 characters. If the terminal instead handled this as a line, then only the 5 characters are necessary. Which approach to use depends on the hardware and requirements. Yet even within this simple example is hidden complexity. What constitutes the end of a line? A carriage return? A line feed? Both? If both, must the user type both? This and many other tasks like echoes, signal handling, jobs control, and special characters processing are managed by the terminal. Such tasks constitute what's called "line discipline."

Terminals act as an interface in other ways. The POSIX standard has this gem of a definition for "terminal,"

A character special file that obeys the specifications of the general terminal interface.

– https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_401

A "character special file" is a file that refers to a device (such as a terminal device file) or that has special properties (such as /dev/null).

– https://pubs.opengroup.org/onlinepubs/9699919799/nframe.html

So, according to POSIX, a terminal is a file that refers to a device such as a terminal. What that means is a terminal a file, as well as a physical device. Unfortunately, several "slights of hand" happen with the words "file", "device", and "terminal."

POSIX is a standardization of operating systems based on UNIX. If modular design is "the UNIX philosophy", then UNIX's motto must be "everything is a file." More accurately, "everything has a file descriptor." This means that (most) objects in the system have a numeric reference like 0, 1, or 2, which allow them to work with the system calls "open", "close", "read", and "write" as a stream of bytes, just like a file. Saying something is a "file" means it has the same interface as a file regardless of whether it's an electronic document, a keyboard, a mouse, or a terminal.

Any device we attach to our computer must communicate with the system. Data must flow from the device into our system and the system may send data back. Processes related to this I/O are often called "device drivers." Typically, there is one process, or driver, per device. UNIX-like operating systems drivers are represented by special files. The fact they're not typical files like a document makes them special; they're an interface. So, "device", "driver", and "file" often refer to the same thing: the interface the operating system presents for a connected device.

We've already seen that a file is another way to refer to a stream of bytes. Different devices group data within a stream in different ways. Some devices, such as a keyboard, operate one byte at a time. Other devices, like hard disks, use blocks of data. Since one byte corresponds to a character, such devices are called "character devices." Connecting our terms, "a character special file" is another way to say, "a driver associated with a device that sends and receives one byte at a time." Generally, devices which send and receive one byte at a time are called "serial" devices.

Physical terminals were some of the first devices ever connected to a computer. Indeed, the famous picture of Ken Thompson and Dennis Richie in front of a PDP-11 shows Ken at a Teletype Model 33 ASR teleprinter (probably interacting with UNIX). It's no surprise then that the device file used to represent a Teletype in UNIX is named "tty," a common abbreviation for (T)ele(ty)pes. TTY files have come to mean any character device.

In Linux, processes are grouped into sets of "sessions" in order to facilitate process management. Every session is tied to a terminal, called the "controlling terminal," from which processes in the session get their input and to which they send their output. A TTY device file corresponds to any device that could act as a controlling terminal. These are either serial devices or virtual consoles (full-screen terminal displays on the system video monitor). There are various naming conventions for these devices/files. However, they always correspond to a serial device or virtual console.

Sometimes it's desirable to enforce a line discipline without a physical device. For example, we may want to automate shell interaction using a program or we may want to connect Emacs to a shell. Linux provides a generalization of TTY drivers called pseudoterminals, abbreviated by PTY.

The man page for "pty" says it best,

A pseudoterminal (sometimes abbreviated "pty") is a pair of virtual character devices that provide a bidirectional communication channel. One end of the channel is called the master; the other end is called the slave.

The slave end of the pseudoterminal provides an interface that behaves exactly like a classical terminal. A process that expects to be connected to a terminal, can open the slave end of a pseudoterminal and then be driven by a program that has opened the master end. Anything that is written on the master end is provided to the process on the slave end as though it was input typed on a terminal. For example, writing the interrupt character (usually control-C) to the master device would cause an interrupt signal (SIGINT) to be generated for the foreground process group that is connected to the slave. Conversely, anything that is written to the slave end of the pseudoterminal can be read by the process that is connected to the master end.

Data flow between master and slave is handled asynchronously, much like data flow with a physical terminal. Data written to the slave will be available at the master promptly, but may not be available immediately. Similarly, there may be a small processing delay between a write to the master, and the effect being visible at the slave.

– https://www.man7.org/linux/man-pages/man7/pty.7.html

Regurgitating some information, on a PTY, the master end sends it's input to the slave's output:

                    PTY
            +--------+-------+
     input  |        |       | output
      -------->==============+----->
            |        |       |
e.g. emacs  | master | slave | e.g. shell
            |        |       |
      <-----+==============<--------
     output |        |       | input
            +--------+-------+

I don't actually know what this diagram means.

For C examples of working with a PTY, see:

• https://www.rkoucha.fr/tech_corner/pty_pdip.html
• The Linux Programming Interface, Kerrisk - Chapter 64

rkoucha's example is good yet it's interactive and not easily embedded within an org document. I could not automate it and must return to it.

PTYs are compared to bidirectional pipes. Here's an example of implementing bidirectional pipes (AFAIU) between two processes.

/* gcc -o pipe3-example pipe3-example.c && ./pipe3-example

   This example creates a "bidirectional" pipe.

   A pipe enables unidirectional data channel between processes.
   Using two pipes allows two processes to each send and receive data.

   The pipe() Linux system command (man 2 pipe) says,

   "The array pipefd is used to return two file descriptors referring
   to the ends of the pipe.  pipefd[0] refers to the read end of the
   pipe.  pipefd[1] refers to the write end of the pipe.  Data written
   to the write end of the pipe is buffered by the kernel until it is
   read from the read end of the pipe."

   fd[0]          fd[1]
   read --------> write
   in             out

   Reading happens before writing.  Something is put in the pipe
   before it is taken out.

   So, given two pipes, A and B, and two processes (c)hild and
   (p)arent,

                 A
   cread a[0]<------->a[1] pwrite
   pread b[0]<------->b[1] cwrite
                 B

   The child reads from a[0] after the parent writes to a[1].
   The parent reads from b[0] after the child writes to b[1].

 */

#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>


int
main ()
{
  int pipe_A[2];
  int pipe_B[2];
  ssize_t size_parent_read;
  ssize_t size_child_read;

  char parent_buffer[100];
  memset (parent_buffer, '\0', 100);

  char child_buffer[100];
  memset (child_buffer, '\0', 100);

  if (pipe (pipe_A) == 0 && pipe (pipe_B) == 0)
    {
      pid_t pid = fork ();
      if (pid == (pid_t) - 1)
        {
          fprintf (stderr, "Fork failed");
          exit (EXIT_FAILURE);
        }

      else if (pid > (pid_t) 0)
        {                       // Parent
          printf ("Parent writing...\n");
          write (pipe_A[1], "hello", sizeof("hello"));

          size_parent_read = read (pipe_B[0], parent_buffer, 99);
          printf ("Parent received: '%s'\n", parent_buffer);
        }

      else if (pid == (pid_t) 0)
        {                       // Child
          size_child_read = read (pipe_A[0], child_buffer, 99);
          printf ("Child received: '%s'\n", child_buffer);

          printf ("Child writing...\n");
          write (pipe_B[1], "world", sizeof("world"));
        }
    }
  exit (EXIT_SUCCESS);
}

• https://unix.stackexchange.com/a/4132
• https://tldp.org/LDP/lpg/node10.html#SECTION00721000000000000000
• https://www.youtube.com/watch?v=07Q9oqNLXB4&pp=ygUZYnJpYW4gd2lsbCB0ZXJtaW5hbHMgdW5peA%3D%3D
• Operating Systems Design and Implementation, Tanenbaum 1997
• http://www.haifux.org/lectures/86-sil/kernel-modules-drivers/node10.html
• https://web.archive.org/web/20231230182026/https://raw.githubusercontent.com/torvalds/linux/master/Documentation/admin-guide/devices.rst
• https://web.archive.org/web/20231230204511/https://raw.githubusercontent.com/torvalds/linux/master/Documentation/admin-guide/devices.txt
• https://web.archive.org/web/20231208020059/https://www.informit.com/articles/article.aspx?p=397655&seqNum=6
• https://www.rkoucha.fr/tech_corner/pty_pdip.html

Shell blocks have two basic execution models depending on whether it's a session or not (that is, a persistent environment or a temporary environment). In each case, a subprocess call is made to the corresponding shell program.

For non-sessions, the final lisp call is to call-process (via process-file). This runs the shell in a synchronous process. Stdin and stderr are output to a buffer when the process completes. The output is scraped, cleaned, and returned for the block results.

For sessions, the final lisp call is to process-send-string (via comint-send-string). When the first session block is run, a process buffer is created to collect output from a synchronous process (see Output from Processes). Block source is sent to the buffer process along with command to echo delimiters. Output is filtered until the ending delimiter appears. When the ending delimiter is found, output is scraped, cleaned, and returned for the block results.

NOTE Sessions are currently entangled with async! See "session" and "non-session" vs. "synchronous" and "asynchronous".

See https://lists.gnu.org/archive/html/emacs-orgmode/2023-11/msg00242.html

"Session" means a shell environment is "persistent." Each call is executed in the same environment. State exists between calls. In the early-history of Babel, this was called the "imperative" style.

"Non-session" means a shell environment is "temporary." Each call is executed in an independent environment. State does not exist between calls. In the early-history of Babel, this was called the "functional" style.

"synchronous" means that execution prevents the user from editing the document while results are obtained.

"asynchronous" means that execution does not prevent the user from editing the document while results are obtained.

These concepts are conflated (or entangled) in org-babel.

• Why can't non-session blocks run asynchronously?
• Why not make asynchronous the default?

Currently, ob-shell and org-babel provide a set of functions and variables that implement subprocess calls. org-babel, along with ob-template, imply a framework for creating new integrations with different applications. However, in practice, are those functions and variables set up to be composed? Not really.

Asynchronous execution was introduced by ob-python and adopted by ob-shell. Why can't ob-C execute asynchronously? Because the implementation is not set up as a framework.

The following hangs Emacs:

while (1) {}

Executing a shell block requires starting a process.

Processes are synchronous or asynchronous.

Three primitives exist in Emacs for making processes:

1. make-process (asynchronous)
2. call-process (synchronous)
3. call-process-region (synchronous)

There exist several convenience wrappers for these.

Output from a process typically goes to a buffer. This may be changed and instead handled by a filter. call-process has an option to directly send output to a file.

Subprocesses inherent the default-directory and the environment from Emacs. The environment may be changed using process-environment.

There are two types of asynchronous connections: "pty" ("pseudoterminal") and "pipe". The main difference is that "pty" provides a terminal-like connection which allows for things like job control (C-c, C-z, etc.). See Difference between a "shell" and a "terminal".

Shells have two modes. A third mode may exist that blends these.

1. (required) command pipeline
2. (required) shell scripting
3. (optional) shebangs (scripts executed as commands)

Shebangs are not available for all shells (such as cmd.exe)

Files paths have two forms:

1. absolute
2. relative ("expanded" (converted to absolute) using exec-path)

File paths may convert to absolute paths (be "expanded"). In Emacs, this happens using exec-path. exec-path is initialized using PATH and serves a similar role to PATH within Emacs.

Files may be local or remote.

Executing a block can happen by:

• via pty or pipe
• passing it to the shell as an argument (e.g. bash -c)
• putting the source in a file and running it (e.g. bash my-file.sh)

Commands may execute in the current environment or in a child environment.

• source environment
• subshell

Processes have several return values:

• success or failure of the process
• stdout of the process
• stderr of the process

org buffer
+-------+    process buffer     pty       shell process
|       |       +------+      +------+      +------+
|       |       |      |      |      |      |      |
|       | <---> |      | <--> |      | <--> |      |
|       |       |      |      |      |      |      |
|       |       +------+      +------+      +------+
+-------+

org buffer
+-------+          shell process
|       |            +------+
|       |    pipe    |      |
|       | <--------> |      |
|       |            |      |
|       |            +------+
+-------+

• [X]

  How is the word "shell" used within the ob-shell API?

  "shell" is the Babel package name. Babel packages have the form ob-<language>. However, "<language>" isn't always one-to-one. For example, ob-C defines functionality for D and C++.

  #+begin_src shell
  echo "hi"
  #+end_src
  

  It's confusing because it's also possible to call "sh". However, "sh" is a specific shell application, the Bourne shell! Unfortunately, "sh" is often linked to bash which confuses this point.

  The "shell" word within the ob-shell source code is used as a common form in which specific shell APIs are based. For example, org-babel-shell-initialize creates functions of the form "org-babel-execute:template" where "template" is a specific shell. The specific execute functions call the generic org-babel-execute:shell function.

• [X]

  Should ob-shell and ob-eshell be separate packages?

  In some sense they represent two different execution paths. Personally, I think so. However, this will require effort.

• [X]

  What is the execution path for inline versus non-inline blocks?

  Ultimately, org-babel-execute-src-block is the dispatcher for the appropriate org-babel-execute:template function.

  These functions call it:

  • org-babel-execute-buffer
  • org-ctrl-c-ctrl-c
  • org-babel-exp-results
  • org-babel-lob-execute-maybe
• [X]

  What is "posh"?

  See https://list.orgmode.org/18cfe5d0cea.b1bf508e1171442.6917906320380891277@excalamus.com/T/#m759874922a8447246fbc2346fc167993d0a43a5f

  See https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/

• [X]

  Would it make sense for sessions to have the process buffer exist without a prompt? Would this fix problems we see of prompts being returned in the results?

  This is an issue for ob-comint.

  Such a change would require updating org-babel-comint-with-output.

  See ob-blub.

• [X]

  What is meant by "support"?

  Not all functionality is consistent nor available for some shells. This is not necessarily because of technical limitations of the shell. The biggest culprit is probably Powershell. Obviously, the software comes with no warranty or guarantee.

  However, I think it would be nice to provide as consistent behavior as possible. So, I would say "support" means "we'll try to make things consistent but no guarantee :)"

• [X]

  Is there a better way to handle the "variable quoting"

  1. Convert to string
  2. Quote
  3. Use quoted version to define a shell variable

  This seems reasonable.

  Looking at this months after it was written, I don't know what it's referring to. Marking as done.

• [X]

  How do (or even do) :hlines and :sep or :separator work with ob-shell?

  These are strewn throughout the code base. However, none of them are documented, nor is their use clear. They should be fully implemented and documented or removed.

  There is no mention of the :separator keyword anywhere in the manual. There is mention of a :sep keyword in the documentation for Texinfo export. The string ":sep" appears twice in the ob-shell.el source:

  2 matches for ":sep" in buffer: ob-shell.el
     211:  (let ((sep (cdr (assq :separator params)))
     241:      (orgtbl-to-generic var  (list :sep (or sep "\t") :fmt echo-var
  

  These appear to perform the same role: specify how to separate items in a table. However, the syntax is different and (as we shall see) the functionality not quite implemented.

  :hlines twice in in the manual. First, with respect to columnview blocks and again with Python blocks. columnview blocks are not source blocks, so while the concept is probably similar, it's not clear how it translates (if at all) to source blocks. Regarding, Python, the manual says,

  In-between each table row or below the table headings, sometimes results have horizontal lines, which are also known as "hlines". The 'hlines' argument with the default 'no' value strips such lines from the input table. For most code, this is desirable, or else those 'hline' symbols raise unbound variable errors. A 'yes' accepts such lines, as demonstrated in the following example.

  Adapting the (only) example from the info manual, I can sort of get an answer:

  #+NAME: many-cols
  | a | b | c |
  |---+---+---|
  | d | e | f |
  |---+---+---|
  | g | h | i |
  
  #+BEGIN_SRC sh :var tab=many-cols :hlines yes
  echo $tab
  #+END_SRC
  
  #+RESULTS:
  : a b c hline d e f hline g h i
  
  #+BEGIN_SRC sh :var tab=many-cols :hlines yes
  echo "$tab"
  #+END_SRC
  
  #+RESULTS:
  | a     | b | c |
  | hline |   |   |
  | d     | e | f |
  | hline |   |   |
  | g     | h | i |
  

  Looking at the source code and playing around with it more, here is an example using :separator and :hlines. When :hlines is "yes", it requires the use of :hline-string to specify what the hline looks like, otherwise it defaults to "hline" (as we saw above).

  #+NAME: many-cols
  | a | b | c |
  |---+---+---|
  | d | e | f |
  |---+---+---|
  | g | h | i |
  
  #+BEGIN_SRC sh :var tab=many-cols :separator -- :hline-string ++ :hlines yes
  echo "$tab"
  #+END_SRC
  
  #+RESULTS:
  | a--b--c |
  | ++      |
  | d--e--f |
  | ++      |
  | g--h--i |
  

  If the :separator is a pipe, then the result is the usual table:

  #+NAME: many-cols
  | a | b | c |
  |---+---+---|
  | d | e | f |
  |---+---+---|
  | g | h | i |
  
  #+BEGIN_SRC sh :var tab=many-cols :separator | :hline-string ++ :hlines yes
  echo "$tab"
  #+END_SRC
  
  #+RESULTS:
  | a  | b | c |
  | ++ |   |   |
  | d  | e | f |
  | ++ |   |   |
  | g  | h | i |
  

  So, we see that the :hlines and :separator keywords produce some result. However, result leaves a lot to desire and it appears like the functionality is not fully implemented. The results are not meaningful and :hlines requires an undocumented header argument :hline-string.

• [ ] Do people use :separator outside of Texinfo exports?

• [ ]

  Make all symbols follow the convention of org-babel-X where X is the specific language. "shell" is the generic name whereas something like "bash" or "sh" refers to a specific shell implementation.

  There are three names used:

  • org-babel-shell
  • org-babel-sh
  • ob-shell

  This module was originally called ob-sh and later changed to ob-shell. When this happened, the meaning of the sh changed. Before, it meant a generic shell. "shell" now means a generic "shell" changing the meaning of "sh". "sh" refers now to a specific shell (/bin/sh) since that's the call that's actually made when executing. Are "sh" functions specific to the "sh" shell or are they generic?

  It looks like when I submitted my async changes, I kept the ob-shell prefix. My preference is for ob-shell since to matches the filename. However, it shouldn't be there if it's not consistent with the rest of the file (or babel system). The problem with introducing ob-shell as a prefix is that none of the other babel files use that convention.

  It looks like ob-C uses org-babel-X where X is the language. This is the only other Babel module to support multiple languages in a single library.

  See https://list.orgmode.org/87ttndl8wk.fsf@localhost/T/#t

  • [ ] non-private functions and variables need alias or made obsolete
  • [ ] can change variable names in main yet not released
  • [ ] API functions
    • org-babel-default-header-args:<lang>
    • org-babel-execute:<lang>
    • org-babel-expand-body:<lang>
    • org-babel-variable-assignments:<lang>
    • org-babel-header-args:<lang>
    • org-babel-load-session:<lang>
    • org-babel-<lang>-initiate-session
    • org-babel-prep-session:<lang>
    • org-babel-edit-prep:<lang>
    • org-babel-<lang>-associate-session

  Org needs to work for Emacs 29, 28, and 27.

  Third-party code that references public symbols in the current stable release needs to continue working for the next release. The subsequent release may break. Child must be stable. Grand-child can break.

  "Some Org components also depend on third-party packages available through package archives. Org is only guaranteed to be compatible with the latest stable versions of these third-party packages." https://orgmode.org/worg/org-maintenance.html#emacs-compatibility

  How to handle strict renaming?

  Problem: If the new release doesn't contain old names, then third-party code that references the old name will break when the Org version is updated.

  Solution: Make a 1:1 mapping from new name to old name

  Emacs	Org
  29.1	9.6.6
  28.2	9.5.5
  27.2	9.4.4

  How to handle change in API, functionality, or use?

  Problem: If the name is the same yet the arguments

  Seems like aliases are placed in org-compat.el.

  • define-obsolete-function-alias
  • define-obsolete-variable-alias
  • make-obsolete

  Consider org-babel-sh-eoe-indicator. The new name would be org-babel-shell-eoe-indicator.

  ;;; need to make alias/obsolete
  (defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'"
  (defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
  (defvar org-babel-sh-prompt "org_babel_sh_prompt> "
  
  (defun org-babel-sh-var-to-sh (var &optional sep hline)
  (defun org-babel-sh-var-to-string (var &optional sep hline)
  (defun org-babel-sh-initiate-session (&optional session _params)
  (defun org-babel-sh-evaluate (session body &optional params stdin cmdline)
  (defun org-babel-sh-strip-weird-long-prompt (string)
  
  ;;; Change
  ;; private
  (defun org-babel--variable-assignments:sh-generic
  (defun org-babel--variable-assignments:fish
  (defun org-babel--variable-assignments:bash_array
  (defun org-babel--variable-assignments:bash_assoc
  (defun org-babel--variable-assignments:bash (varname values &optional sep hline)
  
  ;; not released in 9.6.17
  ;; make private?
  (defconst ob-shell-async-indicator "echo 'ob_comint_async_shell_%s_%s'"
  (defun ob-shell-async-chunk-callback (string)
  
  ;;; do not touch!
  ;; Babel API
  (defvar org-babel-default-header-args:shell '())
  (defun org-babel-variable-assignments:shell (params)
  (defun org-babel-execute:shell (body params)
  (defun org-babel-prep-session:shell (session params)
  (defun org-babel-load-session:shell (session body params)
  
  ;; Already fits convention
  (defconst org-babel-shell-set-prompt-commands
  (defcustom org-babel-shell-names
  (defcustom org-babel-shell-results-defaults-to-output t
  (defvar org-babel-shell-names)
  (defun org-babel-shell-initialize ()
  

  (define-obsolete-variable-alias
    'org-babel-sh-eoe-indicator
    'org-babel-shell--end-of-output-indicator
    "9.7")
  
  (define-obsolete-variable-alias
    'org-babel-sh-eoe-output
    'org-babel-shell--end-of-output
    "9.7")
  
  (define-obsolete-variable-alias
    'org-babel-sh-prompt
    'org-babel-shell--prompt
    "9.7")
  

• [X]

  Add file local variable for nameless

  -- nameless-current-name: "org-babel-shell"; --

  This should be handled by a .dirlocals. It's not proper to have third party code referenced in the mainline repo. The .dirlocals should also be local and not part of the repo.

• [ ]

  Fixed mixed tabs/spaces and remove trailing whitespace

  Both issues make preparing and reviewing commits needlessly complex. Executive decision: use spaces and remove trailing whitespace. We're not doing embedded work, so there's no point to space saving with tabs. It should be one or the other. Since the rest of the code is indented by spaces and the gnu standard is two spaces, I think we should enforce only spaces.

• [ ]

  Rearrange definitions. The order of definitions doesn't assist the process of learning what this library does. Aside from the entangled org-babel-shell-initialize / org-babel-shell-names, order doesn't matter for execution. So, we should optimize the order for understanding.

  Helper terms defined after usage Organization of the ob-shell hinders understanding. "Helper functions" are defined at the end of the file. This makes reading the code difficult; each function is defined in terms only introduced later on.

  Solution: Reorder function declarations

  1. org-babel-sh-eoe-output
  2. org-babel-sh-eoe-indicator
  3. org-babel-shell-names
  4. org-babel-default-header-args:shell
  5. org-babel-shell-results-defaults-to-output
  6. org-babel-sh-initiate-session
  7. org-babel-sh-var-to-string
  8. org-babel-sh-var-to-sh
  9. org-babel–variable-assignments:sh-generic
  10. org-babel–variable-assignments:bash_array
  11. org-babel–variable-assignments:bash_assoc
  12. org-babel–variable-assignments:bash
  13. org-babel–variable-assignments:fish
  14. org-babel-variable-assignments:shell
  15. org-babel-prep-session:shell
  16. org-babel-load-session:shell
  17. org-babel-sh-strip-weird-long-prompt
  18. org-babel-sh-evaluate
  19. org-babel-execute:shell
  20. org-babel-shell-initialize
• [ ]

  Remove needless helper functions

  The way the code is split up doesn't seem to really warrant being split up. The subdivisions are too small. Most of the helper functions are only used once. If that's the case, is it really helpful to break them out separately? My opinion is no. That might be the case if the function names and docstrings helped understanding the use and context. In my opinion, they don't. If they were instead not broken out and appeared in the context they are used, the context would be clear. The role of the function names could be satisfied by comments. This may be worth refactoring, considering it took me several hours to make sense of things.

• [ ]

  Implement a logical separation of concerns

  Aside from the useless helper functions, it's not clear why functions like org-babel-execute:shell and org-babel-sh-evaluate do exactly what they do. There's initiating the process, getting the inputs, dispatching, and then providing output. All of this appears to split, ad hoc, between (at least) these two functions. Review and consider a different break down of functionality. See the individual functions for details on how this might be done.

• [X] Fix confusion about what "posh" is
• [ ]

  Remove :sep, :separator, and :hlines code

  First, they are not documented. I doubt they are, or ever have been, used.

  Second, they do not appear to be implemented correctly. Specifying :hlines along with :hline-string produces a result that doesn't appear useful.

  One option would be to implement these. Doing this would require 1) understanding the tooling that currently exists and 2) integrating it into the shell context.

  Another option is to remove it until someone requests it.

  Given all the problems that exist currently with shells and the refactoring that needs to happen to make a consistent framework, I would prefer to simply remove this code.

No commentary, just a standard Emacs header. Included for validation.

;;; ob-shell.el --- Babel Functions for Shell Evaluation -*- lexical-binding: t; -*-

;; Copyright (C) 2009-2023 Free Software Foundation, Inc.

;; Author: Eric Schulte
;; Maintainer: Matthew Trzcinski <matt@excalamus.com>
;; Keywords: literate programming, reproducible research
;; URL: https://orgmode.org

;; This file is part of GNU Emacs.

;; GNU Emacs is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; GNU Emacs is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.

;;; Commentary:

;; Org-Babel support for evaluating shell source code.

;;; Code:

(declare-function org-babel-comint-in-buffer "ob-comint" (buffer &rest body)
                  t)
(declare-function org-babel-comint-wait-for-output "ob-comint" (buffer))
(declare-function org-babel-comint-buffer-livep "ob-comint" (buffer))
(declare-function org-babel-comint-with-output "ob-comint" (meta &rest body)
                  t)
(declare-function orgtbl-to-generic "org-table" (table params))

These declarations exist to satisfy the byte compiler (or to help it find the definitions of functions defined in other files).

(defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
  "String to indicate that evaluation has completed.")

Token used by org-babel-comint-with-output (within org-babel-sh-evaluate) to indicate that execution has completed. Used only by sessions. See How blocks execute to obtain results.

(defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'"
  "String to indicate that evaluation has completed.")

Shell command used by org-babel-comint-with-output within org-babel-sh-evaluate. It echoes org-babel-sh-eoe-output within the comint, that being the token used to indicate that execution has completed. See How blocks execute to obtain results. It is used only with sessions.

#+begin_src sh :session *example*
echo "hello, world"
#+end_src

sh-5.1$ PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2=
org_babel_sh_prompt> echo "hello, world"      <------ This is the org-babel-sh-eoe-indicator
echo 'org_babel_sh_eoe'
hello, world
org_babel_sh_prompt> org_babel_sh_eoe
org_babel_sh_prompt>

(defvar org-babel-sh-prompt "org_babel_sh_prompt> "
  "String to set prompt in session shell.")

Prompt for shell sessions. This is set during org-babel-sh-initiate-session using the org-babel-shell-set-prompt-commands corresponding to the block language.

In practice, the prompt is given by the comint-prompt-regexp which looks like:

(concat "^" (regexp-quote org-babel-sh-prompt)     " *")
(concat "^" (regexp-quote "org_babel_sh_prompt> ") " *")
"^org_babel_sh_prompt>  *"

comint-prompt-regexp is required by org-babel-comint-with-output.

(defconst ob-shell-async-indicator "echo 'ob_comint_async_shell_%s_%s'"
  "Session output delimiter template.
See `org-babel-comint-async-indicator'.")

Similar to org-babel-sh-eoe-indicator.

Used by org-babel-comint-async-delete-dangling-and-eval within org-babel-sh-evaluate.

Template for a shell command. Used to construct strings that delimit results within the comint process. Used only with async sessions. See How blocks execute to obtain results.

org-babel-sh-evaluate formats two indicators, a "start" and an "end", along with a Universally Unique IDentifier (UUID). The UUID is a placeholder used until results are available. When the process returns, the UUID is replaced with the results.

(defcustom org-babel-shell-names
  '("sh" "bash" "zsh" "fish" "csh" "ash" "dash" "ksh" "mksh" "posh")
  "List of names of shell supported by babel shell code blocks.
Call `org-babel-shell-initialize' when modifying this variable
outside the Customize interface."
  :group 'org-babel
  :type '(repeat (string :tag "Shell name: "))
  :set (lambda (symbol value)
         (set-default-toplevel-value symbol value)
         (org-babel-shell-initialize)))

Used to define shell languages supported by ob-shell. Each item in the list corresponds to the shell binary.

The following three forms automate variable definitions for the various supported shells.

(defvar org-babel-default-header-args:shell '())

This comes from ob-template. The only documentation for it says, "optionally declare default header arguments for this language". The variable is not used anywhere in the ob-shell code base. Maybe it's intended for end-users rather than developers? Searching for "org-babel-default-header-args" within the Org source, I see that other languages use it in their code base. This seems like something that should stay and something that should be documented.

Later, org-babel-shell-initialize will define a similar variable for each of the other supported shells.

(defvar org-babel-shell-names)

This is a variable to hold the list of shell names supported by ob-shell. org-babel-shell-names is actually defined later on. This definition acts as a forward declaration so that org-babel-initialize works.

(defconst org-babel-shell-set-prompt-commands
  '(;; Fish has no PS2 equivalent.
    ("fish" . "function fish_prompt\n\techo \"%s\"\nend")
    ;; prompt2 is like PS2 in POSIX shells.
    ("csh" . "set prompt=\"%s\"\nset prompt2=\"\"")
    ;; PowerShell, similar to fish, does not have PS2 equivalent.
    ("posh" . "function prompt { \"%s\" }")
    ;; PROMPT_COMMAND can override PS1 settings.  Disable it.
    ;; Disable PS2 to avoid garbage in multi-line inputs.
    (t . "PROMPT_COMMAND=;PS1=\"%s\";PS2="))
  "Alist assigning shells with their prompt setting command.

Each element of the alist associates a shell type from
`org-babel-shell-names' with a template used to create a command to
change the default prompt.  The template is an argument to `format'
that will be called with a single additional argument: prompt string.

The fallback association template is defined in (t . \"template\")
alist element.")

Introduced as a result of the issue babel output seems to drop anything before % (in session).

This variable is only used with blocks containing the :session header. It is used in org-babel-sh-initiate-session with org-babel-sh-prompt to reset the prompt. Each command in org-babel-shell-set-prompt-commands removes the prompt and replaces it with whatever is in org-babel-sh-prompt, for the first two levels of prompts (if the shell supports multiple shell level prompts).

Running the commands alone looks like this:

ahab@pequod ~ $ guix shell fish
ahab@pequod ~ [env]$ fish
ahab@pequod ~> function fish_prompt\n\techo \"%s\"\nend
ahab@pequod ~> function fish_prompt
                   echo "%s"
               end
%secho "hi"
hi
%s

ahab@pequod ~ $ guix shell tsch
ahab@pequod ~ [env]$ tcsh
> set prompt="%s"
set prompt2=""
exit

It replaces the prompt with "%s". The "%s" is a format expression.

When org-babel-shell-set-prompt-commands is run by org-babel-sh-initiate-session, the command for corresponding shell gets fed into the lisp format command. For example, the default (which corresponds to bash) is

"PROMPT_COMMAND=;PS1=\"%s\";PS2="

this is used by format:

(format "PROMPT_COMMAND=;PS1=\"%s\";PS2=" org-babel-sh-prompt)

this results in the following being called in a shell:

"PROMPT_COMMAND=;PS1=\"org_babel_sh_prompt> \";PS2="

(defcustom org-babel-shell-results-defaults-to-output t
  "Let shell execution defaults to \":results output\".

When set to t, use \":results output\" when no :results setting
is set.  This is especially useful for inline source blocks.

When set to nil, stick to the convention of using :results value
as the default setting when no :results is set, the \"value\" of
a shell execution being its exit code."
  :group 'org-babel
  :type 'boolean
  :package-version '(Org . "9.4"))

Used to define the results header default.

I'm confused by this because it feels like there are several places in the codebase which define "defaults" rather than one. This concerns me because it's not clear how they're incorporated into the broader system. The concern is basically having a state machine spread across the system and running into issues where one clobbers the other.

(defun org-babel-sh-initiate-session (&optional session _params)
  "Initiate a session named SESSION according to PARAMS."
  (when (and session (not (string= session "none")))
    (save-window-excursion
      (or (org-babel-comint-buffer-livep session)
          (progn
            (shell session)
            ;; Set unique prompt for easier analysis of the output.
            (org-babel-comint-wait-for-output (current-buffer))
            (org-babel-comint-input-command
             (current-buffer)
             (format
              (or (cdr (assoc (file-name-nondirectory shell-file-name)
                              org-babel-shell-set-prompt-commands))
                  (alist-get t org-babel-shell-set-prompt-commands))
              org-babel-sh-prompt))
            (setq-local comint-prompt-regexp
                        (concat "^" (regexp-quote org-babel-sh-prompt)
                                " *"))
            ;; Needed for Emacs 23 since the marker is initially
            ;; undefined and the filter functions try to use it without
            ;; checking.
            (set-marker comint-last-output-start (point))
            (get-buffer (current-buffer)))))))

The main purpose of this function is to start a Process Buffer and set it up so that we can scrape output from it.

When a initiating a new session, create a new process buffer with shell. Loop until the prompt appears. When the prompt appears, the shell is ready to receive input. Then, change the shell prompt and update the comint-prompt-regexp. Return the buffer associated with process.

Note that for Emacs 23 compatibility, we must manually set comint-last-output-start.

(defun org-babel-sh-var-to-string (var &optional sep hline)
  "Convert an elisp value to a string."
  (let ((echo-var (lambda (v) (if (stringp v) v (format "%S" v)))))
    (cond
     ((and (listp var) (or (listp (car var)) (eq (car var) 'hline)))
      (orgtbl-to-generic var  (list :sep (or sep "\t") :fmt echo-var
                                    :hline hline)))
     ((listp var)
      (mapconcat echo-var var "\n"))
     (t (funcall echo-var var)))))

Convert a "value" to a string. First define a helper function echo-var to keep strings as strings and convert anything else using the "%S" option of format. You can read the documentation for those, but it basically boils down to return primitive types like numbers as-is and evaluate symbols first.

org-babel-sh-var-to-string behaves like follows:

#+name: my-table
| col1  | col2   |
|-------+--------|
| test1 | test 2 |

#+begin_src sh :stdin my-table :results output :eval never-export
cat
#+end_src

#+RESULTS:
: col1	col2
: test1	test 2

For a table, var comes in as:

(("col1" "col2") hline ("test1" "test 2"))

This is the result of org-babel-ref-resolve. Unfortunately, org-babel-ref-resolve doesn't document what that value will look like. This is what a table apparently looks like. A raw Org table is converted to a list of lists such that rows are nested lists or the symbol 'hline.

The first condition of the cond checks for a table by way of "is it a list and is the first element another list or the symbol 'hline?" When true, the table contents are passed to orgtbl-to-generic which converts it to a string of some sort according to some given parameters. In our case, it's hard coded as tab delimited strings and no hlines.

The second condition checks for a list:

#+NAME: my-list
- simple
- list
- without
- nesting

#+begin_src sh :results output :stdin my-list :eval never-export
cat
#+end_src

#+RESULTS:
: simple
: list
: without
: nesting

In the case of a list, var comes through as a list of items:

("simple" "list" "without" "nesting")

org-babel-sh-var-to-string concatenates each element of the list together with a newline.

Finally, the third condition is a catch-all that simply does the conversion:

#+name: text
The third case

#+begin_src sh :results output :stdin text :eval never-export
cat
#+end_src

#+RESULTS:
: The third case

Used in org-babel-execute:shell for stdin (to allow named tables as inputs). Also used in org-babel-sh-var-to-sh.

(defun org-babel-sh-var-to-sh (var &optional sep hline)
  "Convert an elisp value to a shell variable.
Convert an elisp var into a string of shell commands specifying a
var of the same value."
  (concat "'" (replace-regexp-in-string
               "'" "'\"'\"'"
               (org-babel-sh-var-to-string var sep hline))
          "'"))

This function quotes its argument. The docstring inaccurately says, "Convert an elisp value to a shell variable." This is not true. Yes, it's used in a process of converting a lisp value to a shell variable. However, that's not what this function does.

This function does two things: 1) each ' quote in the argument is replaced by '"'"' and 2) the whole result is surrounded in single quotes. Step 1 is done to prevent single-quotes contained in the argument from breaking the outer quotes added by step 2. Step 2 is done to ensure the quoted argument is interpreted as a literal.

This is easiest to see with a non-trivial example:

(org-babel-sh-var-to-sh "it'd") -> 'it'\"'\"'d'

See how the single quote is replaced by '"'"'? In this case, the double-quotes are escaped with backslashes. It's maybe easier to see if we separate the quoted single-quote with spaces:

'it '\"'\"' d'

All of this is made even more confusing because when these values are viewed in Emacs, double-quotes are added around it (again to represent literal).

Most likely this quoting is shell specific. I assume it only applies to Bourne-like shells. In truth, I'm not sure.

This function is a helper used in the following:

• org-babel--variable-assignments:sh-generic
• org-babel--variable-assignments:fish
• org-babel--variable-assignments:bash_array
• org-babel--variable-assignments:bash_assoc

(org-babel-sh-var-to-sh (org-babel-ref-resolve "pointless-sep-hline") "--" "++")

(defun org-babel--variable-assignments:sh-generic
    (varname values &optional sep hline)
  "Return a list of statements declaring the values as a generic variable."
  (format "%s=%s" varname (org-babel-sh-var-to-sh values sep hline)))

Create a string like "%s=%s" for use in creating variables. See Shell Parameters.

(defun org-babel--variable-assignments:bash_array
    (varname values &optional sep hline)
  "Return a list of statements declaring the values as a bash array."
  (format "unset %s\ndeclare -a %s=( %s )"
          varname varname
          (mapconcat
           (lambda (value) (org-babel-sh-var-to-sh value sep hline))
           values
           " ")))

Create a string for defining Bash indexed arrays.

Indexed arrays are lists of values accessed by their position in the list. See Arrays.

unset is a Bash builtin to destroy arrays. Use declare -a myarray to create a new array. There are two ways to define elements of an array. One of them is myarray[index]=value. Another is NAME=(VALUE1 VALUE2 ... ) where each VALUE may be of the form '[SUBSCRIPT]='STRING.

Given a 'varname' for the array name and a list of pairs, construct an the string:

unset VARNAME
declare -a VARNAME=( VARNAME[key1]=value1 VARNAME[key2]=value2 ... VARNAME[keyN]=valueN )

This function is called when using the :var header argument.

echo ${myfriends[2]} is a friend of mine

(defun org-babel--variable-assignments:bash_assoc
    (varname values &optional sep hline)
  "Return a list of statements declaring the values as bash associative array."
  (format "unset %s\ndeclare -A %s\n%s"
          varname varname
          (mapconcat
           (lambda (items)
             (format "%s[%s]=%s"
                     varname
                     (org-babel-sh-var-to-sh (car items) sep hline)
                     (org-babel-sh-var-to-sh (cdr items) sep hline)))
           values
           "\n")))

Create a string for constructing an associative array in Bash syntax.

Bash supports "associative" arrays. Associative arrays are key-value stores (sometimes called "dictionaries"). See Arrays.

unset is a Bash builtin to destroy arrays. Use declare -A myarray to create a new array. There are two ways to define elements of an array. One of them is myarray[key]=value.

Given a 'varname' for the array name and a list of pairs, construct an the string:

unset VARNAME
declare -A VARNAME
VARNAME[key1]=value1
VARNAME[key2]=value2
...
VARNAME[keyN]=valueN

This function is called when using the :var header argument.

echo I have ${myarray[apple]} apples

(defun org-babel--variable-assignments:bash (varname values &optional sep hline)
  "Represent the parameters as useful Bash shell variables."
  (pcase values
    (`((,_ ,_ . ,_) . ,_)               ;two-dimensional array
     (org-babel--variable-assignments:bash_assoc varname values sep hline))
    (`(,_ . ,_)                         ;simple list
     (org-babel--variable-assignments:bash_array varname values sep hline))
    (_                                  ;scalar value
     (org-babel--variable-assignments:sh-generic varname values sep hline))))

Call appropriate assignment function:

• org-babel–variable-assignments:bash_assoc
• org-babel–variable-assignments:bash_array
• org-babel–variable-assignments:sh-generic

Uses pcase which does pattern matching on the form of a value and performs an action accordingly.

(defun org-babel--variable-assignments:fish
    (varname values &optional sep hline)
  "Return a list of statements declaring the values as a fish variable."
  (format "set %s %s" varname (org-babel-sh-var-to-sh values sep hline)))

This is basically the same function as org-babel--variable-assignments but for fish's syntax.

(defun org-babel-variable-assignments:shell (params)
  "Return list of shell statements assigning the block's variables."
  (let ((sep (cdr (assq :separator params)))
        (hline (when (string= "yes" (cdr (assq :hlines params)))
                 (or (cdr (assq :hline-string params))
                     "hline"))))
    (mapcar
     (lambda (pair)
       (if (string-suffix-p "bash" shell-file-name)
           (org-babel--variable-assignments:bash
            (car pair) (cdr pair) sep hline)
         (if (string-suffix-p "fish" shell-file-name)
             (org-babel--variable-assignments:fish
              (car pair) (cdr pair) sep hline)
           (org-babel--variable-assignments:sh-generic
            (car pair) (cdr pair) sep hline))))
     (org-babel--get-vars params))))

Call appropriate variable assignment function depending on shell type (i.e. suffix is bash or not). Each element of a :var header arg is processed.

(defun org-babel-prep-session:shell (session params)
  "Prepare SESSION according to the header arguments specified in PARAMS."
  (let* ((session (org-babel-sh-initiate-session session))
         (var-lines (org-babel-variable-assignments:shell params)))
    (org-babel-comint-in-buffer session
      (mapc (lambda (var)
              (insert var) (comint-send-input nil t)
              (org-babel-comint-wait-for-output session))
            var-lines))
    session))

Cruft left over from the original implementation of ob-sh. It was used to start and return the comint associated with the session.

Originally, the comint associated with a session was actually started by a function (that no longer exists) called org-babel-sh-initiate-session-by-key. The org-babel-sh-initiate-session returned only the buffer after it was created:

;; from commit 0fa68a53
(defvar org-babel-sh-buffers '(:default . nil))

(defun org-babel-sh-session-buffer (session)
  (cdr (assoc session org-babel-sh-buffers)))

(defun org-babel-sh-initiate-session-by-key (&optional session)
  "If there is not a current inferior-process-buffer in SESSION
then create.  Return the initialized session."
  (save-window-excursion
    (let* ((session (if session (intern session) :default))
           (sh-buffer (org-babel-sh-session-buffer session))
           (newp (not (org-babel-comint-buffer-livep sh-buffer))))
      (if (and sh-buffer (get-buffer sh-buffer) (not (buffer-live-p sh-buffer)))
          (setq sh-buffer nil))
      (shell sh-buffer)
      (when newp
        (setq sh-buffer (current-buffer))
        (org-babel-comint-wait-for-output sh-buffer))
      (setq org-babel-sh-buffers (cons (cons session sh-buffer)
                                       (assq-delete-all session org-babel-sh-buffers)))
      session)))

(defun org-babel-sh-initiate-session (&optional session)
  (unless (string= session "none")
    (org-babel-sh-session-buffer (org-babel-sh-initiate-session-by-key session))))

Over time, the functionality of creating the comint process was moved into org-babel-sh-initiate-session. Variable assignment was moved from org-babel-prep-session:sh to org-babel-sh-variable-assignments in 8e151c06 October 14, 2010. It has been untouched since then (for 13 years).

As you can see, org-babel-prep-session duplicates that of org-babel-sh-initiate-session-by-key (and the modern org-babel-sh-initiate-session). That is, it starts a comint.

(defun org-babel-load-session:shell (session body params)
  "Load BODY into SESSION."
  (save-window-excursion
    (let ((buffer (org-babel-prep-session:shell session params)))
      (with-current-buffer buffer
        (goto-char (process-mark (get-buffer-process (current-buffer))))
        (insert (org-babel-chomp body)))
      buffer)))

This is cruft from the original development. I highly doubt it's been used in the 13 years since it was last changed. Everything this function does has been reimplemented in other functions.

It starts a process if none exists and inserts the body into the associated process buffer. Initiate a comint, if needed. Define any variables that are given in the header args. Return buffer associated with the session.

It's intended for use with the (largely undocumented) org-metaup.

(defun org-babel-sh-strip-weird-long-prompt (string)
  "Remove prompt cruft from a string of shell output."
  (while (string-match "^% +[\r\n$]+ *" string)
    (setq string (substring string (match-end 0))))
  string)

Used only in org-babel-sh-evaluate and only for sessions. Removes leading white space for strings starting with "%".

The name concerns me. It sounds an awful lot like the problem it tries to solve wasn't fully understood. That is, it's a work around, not a solution.

Introduced in commit 9623b169.

(defun ob-shell-async-chunk-callback (string)
  "Filter applied to results before insertion.
See `org-babel-comint-async-chunk-callback'."
  (replace-regexp-in-string comint-prompt-regexp "" string))

As the docstring says, the filter applied to results before insertion. Replace the comint-prompt-regexp ("^org_{babelshprompt}> *") with nothing.

Used in org-babel-sh-evaluate. Provided as the CHUNK-CALLBACK argument of org-babel-comint-async-register. Called after the ob-shell-async-indicator is found and the results scraped from the comint buffer.

This is where all the functionality of ob-shell lives. Understand this and you'll have the bulk of what ob-shell is.

(defun org-babel-sh-evaluate (session body &optional params stdin cmdline)
  "Pass BODY to the Shell process in BUFFER.
If RESULT-TYPE equals `output' then return a list of the outputs
of the statements in BODY, if RESULT-TYPE equals `value' then
return the value of the last statement in BODY."
  (let* ((shebang (cdr (assq :shebang params)))
         (async (org-babel-comint-use-async params))
         (results-params (cdr (assq :result-params params)))
         (value-is-exit-status
          (or (and
               (equal '("replace") results-params)
               (not org-babel-shell-results-defaults-to-output))
              (member "value" results-params)))
         (results
          (cond
           ((or stdin cmdline)         ; external shell script w/STDIN
            (let ((script-file (org-babel-temp-file "sh-script-"))
                  (stdin-file (org-babel-temp-file "sh-stdin-"))
                  (padline (not (string= "no" (cdr (assq :padline params))))))
              (with-temp-file script-file
                (when shebang (insert shebang "\n"))
                (when padline (insert "\n"))
                (insert body))
              (set-file-modes script-file #o755)
              (with-temp-file stdin-file (insert (or stdin "")))
              (with-temp-buffer
                (with-connection-local-variables
                 (apply #'process-file
                        (if shebang (file-local-name script-file)
                          shell-file-name)
                        stdin-file
                        (current-buffer)
                        nil
                        (if shebang (when cmdline (list cmdline))
                          (list shell-command-switch
                                (concat (file-local-name script-file)  " " cmdline)))))
                (buffer-string))))
           (session                     ; session evaluation
            (if async
                (progn
                  (let ((uuid (org-id-uuid)))
                    (org-babel-comint-async-register
                     session
                     (current-buffer)
                     "ob_comint_async_shell_\\(.+\\)_\\(.+\\)"
                     'ob-shell-async-chunk-callback
                     nil)
                    (org-babel-comint-async-delete-dangling-and-eval
                        session
                      (insert (format ob-shell-async-indicator "start" uuid))
                      (comint-send-input nil t)
                      (insert (org-trim body))
                      (comint-send-input nil t)
                      (insert (format ob-shell-async-indicator "end" uuid))
                      (comint-send-input nil t))
                    uuid))
              (mapconcat
               #'org-babel-sh-strip-weird-long-prompt
               (mapcar
                #'org-trim
                (butlast ; Remove eoe indicator
                 (org-babel-comint-with-output
                     (session org-babel-sh-eoe-output t body)
                   (insert (org-trim body) "\n"
                           org-babel-sh-eoe-indicator)
                   (comint-send-input nil t))
                 ;; Remove `org-babel-sh-eoe-indicator' output line.
                 1))
               "\n")))
           ;; External shell script, with or without a predefined
           ;; shebang.
           ((org-string-nw-p shebang)
            (let ((script-file (org-babel-temp-file "sh-script-"))
                  (padline (not (equal "no" (cdr (assq :padline params))))))
              (with-temp-file script-file
                (insert shebang "\n")
                (when padline (insert "\n"))
                (insert body))
              (set-file-modes script-file #o755)
              (if (file-remote-p script-file)
                  ;; Run remote script using its local path as COMMAND.
                  ;; The remote execution is ensured by setting
                  ;; correct `default-directory'.
                  (let ((default-directory (file-name-directory script-file)))
                    (org-babel-eval (file-local-name script-file) ""))
                (org-babel-eval script-file ""))))
           (t (org-babel-eval shell-file-name (org-trim body))))))
    (when (and results value-is-exit-status)
      (setq results (car (reverse (split-string results "\n" t)))))
    (when results
      (let ((result-params (cdr (assq :result-params params))))
        (org-babel-result-cond result-params
          results
          (let ((tmp-file (org-babel-temp-file "sh-")))
            (with-temp-file tmp-file (insert results))
            (org-babel-import-elisp-from-file tmp-file)))))))

The org-babel-sh-evaluate function has two parts, calculating the results and formatting the results. Calculating the results makes up most of the function. It's a conditional for each case where each case corresponds to possible header arguments. Formatting is more-or-less off-loaded to core Org functionality.

There are several ways in which a shell block may evaluate. Many of these overlap and influence each other. Each, more-or-less, corresponds to a case in the conditional.

These are the current cases:

1. stdin
2. cmdline
3. shebang
4. session
5. async
6. general

• [X]

  defun and org-babel-sh-evaluate-varlist-shebang

  (defun org-babel-sh-evaluate (session body &optional params stdin cmdline)
    "Pass BODY to the Shell process in BUFFER.
  If RESULT-TYPE equals `output' then return a list of the outputs
  of the statements in BODY, if RESULT-TYPE equals `value' then
  return the value of the last statement in BODY."
    (let* ((shebang (cdr (assq :shebang params)))
  

  The arguments are several parameters (session, stdin, cmdline, async, etc) and the source block body. For some reason, the parameters are split between being explicitly passed and passed within the params list. See Questions and Refactor for discussion.

  Params is an alist of data. condition See "What is params?" in defun org-babel-execute:shell for more information.

  A shebang is a line like "#!/bin/bash" which does two things:

  1. Allows a script file to execute as a command
  2. Specifies which binary to use when executing

  This let statement binds the :shebang header argument.

• [X]

  org-babel-sh-evaluate-varlist-async

  (async (org-babel-comint-use-async params))
  

  Determine whether to use async.

  Async depends on 1) :session not being "none" and 2) the :async header not being "no". See Questions section below on toggle (in)consistencies. org-babel-use-async moves this logic to a separate module for other async implementations to use.

• [X]

  org-babel-sh-evaluate-varlist-results-params

  (results-params (cdr (assq :result-params params)))
  

  The :result-params key in params holds the header values for the :results header.

  It looks something like,

  (:result-params "replace" "output")
  

  or

  (:result-params "replace" "value" "table")
  

  Results are presentable in several formats and :result-params holds the keywords for deciding what the output should look like.

• [X]

  org-babel-sh-evaluate-varlist-value-is-exit-status

  (value-is-exit-status
   (or (and
        (equal '("replace") results-params)
        (not org-babel-shell-results-defaults-to-output))
       (member "value" results-params)))
  

  Block results are stdout, stderr, or an exit code. Typically, an exit code of 0 means success, non-0 means failure. value-is-exit-status helps figure out which result to return.

  This is something that needs refactoring. See Refactoring notes.

• [X]

  org-babel-sh-evaluate-varlist-results

  (results
   (cond
  

  Results are calculated using a cond. Various combinations of headers make up the different cases. See Question asking whether all combinations are represented.

• [X]

  org-babel-sh-evaluate-varlist-results-stdin-cmdline

  ((or stdin cmdline)            ; external shell script w/STDIN
   (let ((script-file (org-babel-temp-file "sh-script-"))
         (stdin-file (org-babel-temp-file "sh-stdin-"))
         (padline (not (string= "no" (cdr (assq :padline params))))))
     (with-temp-file script-file
       (when shebang (insert shebang "\n"))
       (when padline (insert "\n"))
       (insert body))
     (set-file-modes script-file #o755)
     (with-temp-file stdin-file (insert (or stdin "")))
     (with-temp-buffer
       (with-connection-local-variables
        (apply #'process-file
               (if shebang (file-local-name script-file)
                 shell-file-name)
               stdin-file
               (current-buffer)
               nil
               (if shebang (when cmdline (list cmdline))
                 (list shell-command-switch
                       (concat (file-local-name script-file)  " " cmdline)))))
       (buffer-string))))
  

  This condition handles the stdin, cmdline, and (sometimes) shebangs headers. Note: there's a separate branch condition for shebang that's only called when there's no stdin or cmdline header. So, this is for:

  stdin cmdline stdin, shebang cmdline, shebang stdin, cmdline, shebang

  A shebang is a line at the top of a file like "#!/bin/bash" which allows a shell script to run as a command. The name "shebang" comes from combining the names of the first two characters, the "hash" (#) and the "bang" (!).

  It works by calling process-file which has form:

  (process-file PROGRAM &optional INFILE BUFFER DISPLAY &rest ARGS)
  

  The stdin header is put into a temporary file whose path is passed in as INFILE.

  The block code is also put into a temporary file. The block code is either executed as a shebang or called as part of the argument to the shell binary. The shebang line is inserted when a :shebang is given.

  The temporary script file is made executable with 755 permissions. This corresponds to chmod a+rwx,o-w which sets permissions so that the group and owner can read, can write and can execute. Others can read and execute, but can't write.

  When the block script is executed, it's run using process-file in a temporary buffer. The results of the process are placed in the buffer and, finally, the contents of the buffer given for the results.

  For some reason, a newline is inserted if :padline is given. See Questions.

• [X]

  org-babel-sh-evaluate-varlist-results-session

  (session                        ; session evaluation
   (if async
       (progn
         (let ((uuid (org-id-uuid)))
           (org-babel-comint-async-register
            session
            (current-buffer)
            "ob_comint_async_shell_\\(.+\\)_\\(.+\\)"
            'ob-shell-async-chunk-callback
            nil)
           (org-babel-comint-async-delete-dangling-and-eval
               session
             (insert (format ob-shell-async-indicator "start" uuid))
             (comint-send-input nil t)
             (insert (org-trim body))
             (comint-send-input nil t)
             (insert (format ob-shell-async-indicator "end" uuid))
             (comint-send-input nil t))
           uuid))
     (mapconcat
      #'org-babel-sh-strip-weird-long-prompt
      (mapcar
       #'org-trim
       (butlast ; Remove eoe indicator
        (org-babel-comint-with-output
            (session org-babel-sh-eoe-output t body)
          (insert (org-trim body) "\n"
                  org-babel-sh-eoe-indicator)
          (comint-send-input nil t))
        ;; Remove `org-babel-sh-eoe-indicator' output line.
        1))
      "\n")))
  

  The "session" condition has two parts: async and non-async.

  Async handling comes first. The async API requires "registering" certain information: the session buffer, the org buffer the block lives in, a regexp for matching the beginning and end of evaluation, and a user-defined filter function. The registration process adds a special (non-user-defined) filter function to the session buffer (org-babel-comint-async-filter). This filter looks at output sent to the session buffer from the process for the end of evaluation regex.

  After the registration is made, the "start" indicator is sent, the block code sent, and finally the "end" indicator. Because this is all sent to a shell session, it runs in the background until the filter finds the "end" indicator. Until that happens, the result of the async branch is the uuid and that's what appears for the user as the result. (If the "end" indicator is found, the uuid is located and replaced with the text occurring between the "start" and "end" indicators. But that's all handled by things in ob-comint.el.)

  Non-async sessions are handled by the next bit of code. Basically, it sends the block code to the session and waits for the org-babel-sh-eoe-indicator. How this happens is determined by code in ob-comint.el.

• [X]

  org-babel-sh-evaluate-varlist-results-shebang

  ;; External shell script, with or without a predefined
  ;; shebang.
  ((org-string-nw-p shebang)
   (let ((script-file (org-babel-temp-file "sh-script-"))
         (padline (not (equal "no" (cdr (assq :padline params))))))
     (with-temp-file script-file
       (insert shebang "\n")
       (when padline (insert "\n"))
       (insert body))
     (set-file-modes script-file #o755)
     (if (file-remote-p script-file)
         ;; Run remote script using its local path as COMMAND.
         ;; The remote execution is ensured by setting
         ;; correct `default-directory'.
         (let ((default-directory (file-name-directory script-file)))
           (org-babel-eval (file-local-name script-file) ""))
       (org-babel-eval script-file ""))))
  

  Check if shebang is non-empty; check for padline Write shebang and body to temp file

  The temporary script file is made executable with 755 permissions. This corresponds to chmod a+rwx,o-w which sets permissions so that the group and owner can read, can write and can execute. Others can read and execute, but can't write.

  Check if the script file is remote. This is because, ultimately, process-file is called. Two things must happen:

  1. strip the remote information from the filename
  2. make sure the file runs on the remote

  process-file says to use file-local-name for remote files. This strips the remote information from the filename. However, without this information, process-file will execute the file on the local machine. We change this behavior by setting the default-directory to make the file relative to the remote machine.

  For example, when processing a remote file, the script-file variable looks like:

  "/ssh:ahab@localhost:/tmp/sh-script-3qPNNc"

  The file-local-name strips the "ssh" stuff off the front to make it: /tmp/sh-script-3qPNNc. Originally, the file was relative to "/ssh:ahab@localhost:/tmp/". After file-local-name, it looks like it's relative to "/tmp" (on the local machine).

  org-babel-eval then tries to run the result of file-local-name. Without setting the default-directory, it uses the current default-directory. This is the the local default-directory which is "/tmp". When this happens, the script can't be found. We must set the default-directory to the remote (/ssh:ahab@localhost:/tmp/). Then it's found.

• [X]

  org-babel-sh-evaluate-varlist-results-general

  (t (org-babel-eval shell-file-name (org-trim body))))))
  

  This handles any of the cases not covered.

  It trims the body and runs it through process-file. See "How Shell works at a deep level" in the Mailing list items.

• [X]

  org-babel-sh-evaluate-varlist-return

  (when (and results value-is-exit-status)
    (setq results (car (reverse (split-string results "\n" t)))))
  (when results
    (let ((result-params (cdr (assq :result-params params))))
      (org-babel-result-cond result-params
        results
        (let ((tmp-file (org-babel-temp-file "sh-")))
          (with-temp-file tmp-file (insert results))
          (org-babel-import-elisp-from-file tmp-file)))))))
  

  Shells can return two types of values, exit codes and standard output.

  The Babel results are the exit code when using ":results value":

  echo "hi"
  

  The results look like:

  "hi
  0
  "
  

  (car (reverse (split-string "hi\n0" "\n" t)))
  

  The final branch takes the result-params (which control output formatting) and

  echo "hello"
  echo "world"
  

  org-table-result-cond handles the different result cases. For a table, it needs a sexp. It gets this from org-babel-import-elisp-from-file:

  (("hello") ("world"))
  

  This is what's returned to the caller. Why? That's not exactly clear. But there you have it.

(defun org-babel-execute:shell (body params)
  "Execute Shell BODY according to PARAMS.
This function is called by `org-babel-execute-src-block'."
  (let* ((session (org-babel-sh-initiate-session
                   (cdr (assq :session params))))
         (stdin (let ((stdin (cdr (assq :stdin params))))
                  (when stdin (org-babel-sh-var-to-string
                               (org-babel-ref-resolve stdin)))))
         (results-params (cdr (assq :result-params params)))
         (value-is-exit-status
          (or (and
               (equal '("replace") results-params)
               (not org-babel-shell-results-defaults-to-output))
              (member "value" results-params)))
         (cmdline (cdr (assq :cmdline params)))
         (full-body (concat
                     (org-babel-expand-body:generic
                      body params (org-babel-variable-assignments:shell params))
                     (when value-is-exit-status "\necho $?"))))
    (org-babel-reassemble-table
     (org-babel-sh-evaluate session full-body params stdin cmdline)
     (org-babel-pick-name
      (cdr (assq :colname-names params)) (cdr (assq :colnames params)))
     (org-babel-pick-name
      (cdr (assq :rowname-names params)) (cdr (assq :rownames params))))))

The fundamental entry point for ob-shell when a source block is evaluated.

When a user executes a block, the ob-core function org-babel-execute-src-block is called. It dynamically binds "org-babel-execute:lang" to a variable "cmd" where "lang" corresponds to whatever follows "#+begin_src" in the block header ("lang" corresponds to what we've been calling "template").

;; org-babel-execute-src-block

...
           d))))
     (cmd (intern (concat "org-babel-execute:" lang)))  ; dynamically bind symbol
     result exec-start-time)
...
(setq exec-start-time (current-time)
      result
      (let ((r (save-current-buffer (funcall cmd body params))))  ; call org-babel-execute:lang
        (if (and (eq (cdr (assq :result-type params)) 'value)
...

Recall that org-babel-shell-initialize generated functions called "org-babel-execute:template", one for each of the shells given in org-babel-shell-names. Each of these calls org-babel-execute:shell.

BODY is the inside of the source block. PARAMS is an alist (see below) of header arguments and other information.

This is function doesn't have a clear purpose. It just happens to be the entry point.

It parses the block header and passes some of these to is the eventual execution function org-babel-sh-evaluate. However, it also passes the full params which is re-parsed out again.

It sort of handles assembling results. For example, it reassembles the table. However, some of that is also done in org-babel-sh-evaluate.

(defun org-babel-shell-initialize ()
  "Define execution functions associated to shell names.
This function has to be called whenever `org-babel-shell-names'
is modified outside the Customize interface."
  (interactive)
  (dolist (name org-babel-shell-names)
    (let ((fname (intern (concat "org-babel-execute:" name))))
      (defalias fname
        (lambda (body params)
          (:documentation
           (format "Execute a block of %s commands with Babel." name))
          (let ((shell-file-name name))
            (org-babel-execute:shell body params))))
      (put fname 'definition-name 'org-babel-shell-initialize))
    (defalias (intern (concat "org-babel-variable-assignments:" name))
      #'org-babel-variable-assignments:shell
      (format "Return list of %s statements assigning to the block's \
variables."
              name))
    (funcall (if (fboundp 'defvar-1) #'defvar-1 #'set) ;Emacs-29
             (intern (concat "org-babel-default-header-args:" name))
             nil)))

Several different shells are supported.

Each needs the following functions:

1. An alias org-babel-execute:template for org-babel-execute:shell
2. An alias org-babel-variable-assignments:name for org-babel-variable-assignments:shell
3. org-babel-default-header-args:name

These allow people to use any of the org-babel-shell-names as the source block language. For example, the ob-core functionality that runs the source block (org-babel-execute-src-block) expects a "org-babel-execute:template" function.

The way it works is this. It iterates through the org-babel-shell-names list. For each element, it defines a symbol for org-babel-execute:template according to the current list item (name). It makes this symbol an alias for a function that calls org-babel-execute:shell such that shell-file-name is assigned to the current list item. For example, when processing "fish", a call to org-babel-execute:fish will call org-babel-execute:shell yet any call to an inferior shell process will use "fish" rather than the shell stored in the SHELL environment variable at start up. (It's assumed that the command, such as "fish", is accessible through PATH). A "put" call makes org-babel-execute:template accessible from the built in help. It next creates an alias for org-babel-variable-assignments:name corresponding to org-babel-variable-assignments:shell. Finally, it creates a symbol for org-babel-default-header-args:name, using a different function (either set or defvar-1) depending on the Emacs version.

Some of ob-shell.el is dedicated to undocumented behavior. The org-babel-load-in-session function which is bound to the org-metaup-hook, M-up. The source body will be inserted in a new session process buffer.

For this is the code path (M-up):

• ob-core:org-babel-load-in-session
• ob-shell:org-babel-prep-session:shell

Now documented, yet a good example of this, is the :dir header. There have been many questions on the mailing list regarding shell blocks and ssh. The :dir header works with TRAMP to create a remote connection. This works seamlessly with Babel. However, it was not clearly documented (the only example showed an R code block and is located in a separate part of the manual).

In the early days of Babel, collaboration notes were kept. Eric makes a distinction between (what he calls) "functional" and "imperative". It's not clear to me what's meant by these terms. AFAICT, "functional" means "non-persistent" and "imperative" means "persistent". "persistent" means here that when you run a block twice, you always get the same result with a "functional" block.

See:

• https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/contrib/babel/org-babel.org?id=c0554c775344207e6adaf191258312fb8d1b6a15
• https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/org-babel.org?id=b1c103890c1523f99e380d88ed684454d902414e

On September 11, 2009, org-babel.org was moved to contrib.

org-babel.org was deleted in eec9c235 and its contents put into a new file babel.org. It was deleted the next day 2d2a2a19.

org-babel.org commit bb2969ef

Org Babel started as a package called "litorgy" around February 22, 2009. Support for shells, litorgy-shell.el, was added in commit e7701000 on March 31, 2009. The current file lisp/ob-shell.el was created in commit 583e7ab1 on December 13, 2013.

An incomplete history of ob-shell is here:

Backwards compatibility: https://list.orgmode.org/8734ur6dk4.fsf@localhost/T/#t

Discussion on refactoring all non-session calls to be run as scripts, including bugs to fix

• https://lists.gnu.org/archive/html/emacs-orgmode/2023-11/msg00137.html
• https://lists.gnu.org/archive/html/emacs-orgmode/2023-11/msg00142.html

How Shell works at a deep level

• https://lists.gnu.org/archive/html/emacs-orgmode/2023-11/msg00242.html

Bug report and patches for :cmdline failings.

• https://lists.gnu.org/archive/html/emacs-orgmode/2023-11/msg00263.html

Handling ANSI characters

• https://list.orgmode.org/orgmode/87msvcgjgv.fsf@gmail.com/

Misc:

• babel output seems to drop anything before % (in session).
• :results value vs. :results output