Digging into xonsh history backends


I want to write a per-directory-history xontrib for xonsh, like the one I used for zsh. The piece of information that that I need to do this is missing from xonsh history entries right now: the working directory a historical command was executed in. jimhester/per-directory-history tracks this by hooking into zsh's history searching and creation commands and putting each command in two history files, a global one and one specifically for the directory the command was executed in. I want to write my xontrib in the most xonshious (my word for if something works with the xonsh philosophy) way, so I don't want to rip this implementation scheme from per-directory-history and jam it into xonsh where there's a better way. So I have to see where I have to collect, store, and read this metadata.

Some ideas I have so far are:

  1. add a new history backend that writes entries with the additional metadata of wherever the command was executed
  2. essentially step (1), but instead of adding a new history backend, augment whatever history backend is in use with this new functionlity (composition by way of monkey-patching)
  3. listen to some existing hooks/events for xonsh history lookup and additions and add the functionality there

Or some combination of the above, depending on what I find.

This post documents my exploration of how history is implemented in xonsh.

I'll pay particular attention to how history backends work with the shell backend abstraction so that I can write a xontrib that is as agnostic as possible about the shell implementation in use (ptk, ptk2, readline, jupyter, etc.).

Reading the existing documentation first

So I figured that I should read any existing documentation first, since it's possible that:

  1. The xonsh docs already include a section that tell me how to do this or something close to it
  2. I might find learn something that I realize is undocumented, and then I can contribute that back to the project docs

I found three documents dealing with history on xon.sh:

While each of these is good at doing what it says, notice that none of them discusses how history backends are instantiated or how history entries are constructed during shell execution. The History API docs come closest, but that's cheating because those docs are autogenerated from docstrings in the Python source for xonsh.

How history entries are managed

Since there is no smoking gun in the Xonsh docs talking about how history backends are created and where the components of history entries come from, I decided I have to dig into the xonsh code now rather than later.

Rather than just explain when and how xonsh creates new history entries (which I will do some of), I also want to explain how I came to this understanding, since it's incredibly unlikely you're reading this doc just to learn how to write a clone of jimhester/per-directory-history`.

xonsh has support for multiple history backends, as we know. It ships with 3 backend implementations: history.json.JsonHistory and history.sqlite.SqliteHistory.

These backends are implementations of the history backend abstraction history.base.History. history.base.History doesn't do anything useful on its own - it is just inherited by implementations and defines the things the xonsh shell expects a history backend to be able to do:

The fact that history backends implement history.base.History is our first clue into how xonsh backends work. This fact means the xonsh shell does not interact directly with a history backend, so the shell's code doesn't know what backend it's working with - this is handled by our good friend polymorphism. For understanding how history entries are created, this establishes some constraints on what a history backend can accept as input - if the central part of the xonsh shell's code is interacting with a unique history backend through a generic abstraction, that unique history backend cannot use input that isn't passed into the generic abstraction. In other words, the xonsh shell gives a particular history item data structure to every history backend, no matter how special that history backend is, and if we want the history backend to be able to act on some other piece of data (such as the working directory the history item was executed in!), we have to alter that data structure.

The history entry data structure

I had trouble finding where these entries were defined and where they were appended to the history backend, but I soon realized I could drop an ipdb break statement into my active history backend's append method (JsonHistory.append) and use the debugger's where command to get a stacktrace, leading me directly to where xonsh appends history to the backend. I started up my debuggified xonsh, ran a command, watched as it paused in ipdb, and got the traceback:

(Note that you should make sure $XONSH_DEBUG is on or, alternatively, install xonsh as an editable package to avoid almalgamation and can see your changes right away without re-running setup.py.)

eddie@eddie-ubuntu ~ $ echo 'hey'                                                                                                                 
> /home/eddie/source/xonsh/xonsh/history/json.py(353)append()
    352         import ipdb; ipdb.set_trace()
--> 353         self.buffer.append(cmd)
    354         self._len += 1  # must come before flushing

ipdb> where
      5 __file__ = '/home/eddie/source/xonsh/scripts/xonsh'
      6 with open(__file__) as f:
----> 7     exec(compile(f.read(), __file__, 'exec'))

      3 from xonsh.main import main
----> 4 main()

    401         args = premain(argv)
--> 402         return main_xonsh(args)
    403     except Exception as err:

    430             try:
--> 431                 shell.shell.cmdloop()
    432             finally:

    193                     line = self.precmd(line)
--> 194                     self.default(line)
    195             except (KeyboardInterrupt, SystemExit):

    374             tee_out = tee.getvalue()
--> 375             self._append_history(inp=src, ts=[ts0, ts1], tee_out=tee_out)
    376             self.accumulated_inputs += src

    409         if hist is not None:
--> 410             hist.append(info)
    411             hist.last_cmd_rtn = hist.last_cmd_out = None

> /home/eddie/source/xonsh/xonsh/history/json.py(353)append()
    352         import ipdb; ipdb.set_trace()
--> 353         self.buffer.append(cmd)
    354         self._len += 1  # must come before flushing

Maybe that wouldn't have been so hard to track down manually, but history is appended to in BaseShell.default() with a method called BaseShell._append_history().

So what kind of information is passed to _append_history?

def default(self, line):
    """Implements code execution."""
    line = line if line.endswith("\n") else line + "\n"
    src, code = self.push(line)
    if code is None:


    env = builtins.__xonsh__.env
    hist = builtins.__xonsh__.history  # pylint: disable=no-member
    ts1 = None
    enc = env.get("XONSH_ENCODING")
    err = env.get("XONSH_ENCODING_ERRORS")
    tee = Tee(encoding=enc, errors=err)
        ts0 = time.time()
        run_compiled_code(code, self.ctx, None, "single")
        ts1 = time.time()
        if hist is not None and hist.last_cmd_rtn is None:
            hist.last_cmd_rtn = 0  # returncode for success
    except XonshError as e:
        print(e.args[0], file=sys.stderr)
        if hist is not None and hist.last_cmd_rtn is None:
            hist.last_cmd_rtn = 1  # return code for failure
    except Exception:  # pylint: disable=broad-except
        if hist is not None and hist.last_cmd_rtn is None:
            hist.last_cmd_rtn = 1  # return code for failure
        ts1 = ts1 or time.time()
        tee_out = tee.getvalue()
        self._append_history(inp=src, ts=[ts0, ts1], tee_out=tee_out)
        self.accumulated_inputs += src
        if (
            and env.get("XONSH_APPEND_NEWLINE")
            and not tee_out.endswith(os.linesep)
            print(os.linesep, end="")
    if builtins.__xonsh__.exit:  # pylint: disable=no-member
        return True

In the finally block, we see inp is src, which, after digging around a big into what happens above this call, appears to be the string that was typed into the command prompt, as opposed to the code, which is the xonsh code that was compiled and run (successfully or not) from compiling this src. Interestingly, this means we are typing in source code each time we enter text the xonsh REPL, and xonsh is compiling/running it. The essential piece of a history entry is a bit of uncompiled source code (like ls -alh or import sys)!

Let's follow an ls command entry from the prompt through BaseShell.default() and the code that appends the entry to history.

The code block picks up just after I've entered the ls command at the prompt.

> /home/eddie/source/xonsh/xonsh/base_shell.py(348)default()
    347         src, code = self.push(line)
--> 348         if code is None:
    349             return

ipdb> code                                                                                                                                                                                                                                                                                            
<code object <module> at 0x7f3682be44b0, file "/home/eddie/.virtualenvs/xonsh/lib/python3.7/site-packages/xontrib/fzf-widgets.xsh", line 1>
ipdb> src                                                                                                                                                                                                                                                                                             

Note that code is apparently wrapped in some non-ls xontrib code I have installed. I'm unsure exactly why right now.

But note that src is the ls command I typed in, followed by a newline.

Once we get down to the actual appending, we see that ts0 and ts1 are the start and end timestamps of the code's execution. tee_out is simply the output of the command.

--> 377             self._append_history(inp=src, ts=[ts0, ts1], tee_out=tee_out)
    378             self.accumulated_inputs += src
    379             if (
    380                 tee_out
    381                 and env.get("XONSH_APPEND_NEWLINE")
    382                 and not tee_out.endswith(os.linesep)
    383             ):
    384                 print(os.linesep, end="")
    385             tee.close()
    386             self._fix_cwd()
    387         if builtins.__xonsh__.exit:  # pylint: disable=no-member
    388             return True

ipdb> src                                                                                                                                                                                                                                                                                             
ipdb> ts0                                                                                                                                                                                                                                                                                             
ipdb> ts1                                                                                                                                                                                                                                                                                             

Let's step into _append_history():

def _append_history(self, tee_out=None, **info):
    """Append information about the command to the history.

    This also handles on_postcommand because this is the place where all the
    information is available.
    hist = builtins.__xonsh__.history  # pylint: disable=no-member
    info["rtn"] = hist.last_cmd_rtn if hist is not None else None
    tee_out = tee_out or None
    last_out = hist.last_cmd_out if hist is not None else None
    if last_out is None and tee_out is None:
    elif last_out is None and tee_out is not None:
        info["out"] = tee_out
    elif last_out is not None and tee_out is None:
        info["out"] = last_out
        info["out"] = tee_out + "\n" + last_out
        cmd=info["inp"], rtn=info["rtn"], out=info.get("out", None), ts=info["ts"]
    if hist is not None:
        hist.last_cmd_rtn = hist.last_cmd_out = None

It isn't the most exciting code. It is really just a matter of adding return code information for failed commands and, if available, the output of the command, to the info (history entry) provided to the backend. As a funny aside most of this method is a heuristic for deciding whether to use tee output or last_cmd_out from the history backend, which last_cmd_out seems to be an unused property in at least all the built-in history backends. Would be interesting to know why it ever existed at all!

The crucial thing we learn here, though, is that info is effectively what we've been calling the history entry. It is the "packet" (concretely, a dict) of information that is appended to the history. It defines what our history backend can save, delete, search, manipulate, etc. So any additional information we would need to add for our history backend would have to be added to info.

Let's take a look at the info object for two different cases. In the first, I'll call ls in a directory with exactly one empty regular file: test, and in the second I'll call grep something test in the same directory. The ls call will provide a successful return value and the grep call will not (since test will be empty).

Calling ls:

{'inp': 'ls\n', 'ts': [1560285242.9592671, 1560285243.0506482], 'rtn': 0, 'out': 'test\n'}

Calling grep something test:

{'inp': 'grep something test\n', 'ts': [1560285373.0232306, 1560285373.125136], 'rtn': 1}

There you have it - all the information available to a history backend's append() method as far as I can tell.

Thoughts on where I should go

So I've been digging around in here to ultimately change what history items are loaded when a user interactively scrolls through the history, uses the history command, etc., with the aim of showing only the history items that are associated with the current working directory. To do that, I have to get cwd information into each history item.

To fast-forward a bit, I've now done that, and it's pretty simple, though it did require a change to the xonsh source code:

diff --git a/xonsh/base_shell.py b/xonsh/base_shell.py
index b7e9aff2..7088427f 100644
--- a/xonsh/base_shell.py
+++ b/xonsh/base_shell.py
@@ -393,6 +393,8 @@ class BaseShell(object):
         hist = builtins.__xonsh__.history  # pylint: disable=no-member
         info["rtn"] = hist.last_cmd_rtn if hist is not None else None
+        if builtins.__xonsh__.env.get("XONSH_STORE_CWD"):
+            info['cwd'] = os.getcwd()
         tee_out = tee_out or None
         last_out = hist.last_cmd_out if hist is not None else None
         if last_out is None and tee_out is None:

Luckily, when I had asked whether such a xontrib as per-directory-history existed yet, xonsh creator Anthony Scopatz told me he'd be up for modifying the history mechanism to support this kind of xontrib, so we're good here.

The next question I had was how I could make history:

To make history aware of this new info, I had to alter the history backends - each history backend has a different way of handling the attributes of history items. I decided to follow a depth-first way of experimenting, hoping that if I got my history functionality working with JsonHistory, probably the most commonly used backend, I could either figure out how to get it working with other backends, or (less good) just make my xontrib available to people using the JsonHistory backend.

Next, I looked at where history strings are loaded by xonsh, so that I could start limiting the items loaded to those that matched by cwd. My thought was that each time history was searched by the user, by whatever mechanism, if I found the point where history strings were loaded, I could filter out those that didn't match.

I thought that creating a history backend with an overridden method would help since custom history backends are easily pluggable with the XONSH_HISTORY_BACKEND environment varible, thus making any solution that used a custom one pretty easily installable via a xontrib.

Unfortunately there was no clear and easy way to override the history backend functionality to filter out history entries on arbitrary criteria, so I added yet another thing to the xonsh source:

diff --git a/xonsh/history/json.py b/xonsh/history/json.py
index 50b6326b..7313cfc9 100644
--- a/xonsh/history/json.py
+++ b/xonsh/history/json.py
@@ -328,6 +328,7 @@ class JsonHistory(History):
--- a/xonsh/history/json.py
--- a/xonsh/history/json.py
+++ b/xonsh/history/json.py
@@ -328,6 +328,7 @@ class JsonHistory(History):
         self.tss = JsonCommandField("ts", self)
         self.inps = JsonCommandField("inp", self)
         self.outs = JsonCommandField("out", self)
+        self.cwds = JsonCommandField("cwd", self)
         self.rtns = JsonCommandField("rtn", self)

     def __len__(self):
@@ -382,10 +383,11 @@ class JsonHistory(History):
     def items(self, newest_first=False):
         """Display history items of current session."""
         if newest_first:
-            items = zip(reversed(self.inps), reversed(self.tss))
+            items = zip(
+                reversed(self.inps), reversed(self.tss), reversed(self.cwds))
-            items = zip(self.inps, self.tss)
-        for item, tss in items:
+            items = zip(self.inps, self.tss, self.cwds)
+        for item, tss, _ in items:
             yield {"inp": item.rstrip(), "ts": tss[0]}

     def all_items(self, newest_first=False, **kwargs):
@@ -413,10 +415,16 @@ class JsonHistory(History):
             if newest_first:
                 commands = reversed(commands)
             for c in commands:
-                yield {"inp": c["inp"].rstrip(), "ts": c["ts"][0]}
+                if self._include_history_item(c):
+                    yield {"inp": c["inp"].rstrip(), "ts": c["ts"][0]}
         # all items should also include session items
         yield from self.items()

+    def _include_history_item(self, item):
+        """Whether to include the history item.
+        Allows filtering history results by subclass."""
+        return True

In short, this diff just adds a method that checks whether a history item should be used, and in the default case (the JsonHistory base class), it simply allows all history items.

In my xontrib, I created a custom history backend that performed the filtering I wanted:

class JsonPerDirectoryHistory(JsonHistory):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.use_local_history = True

    def _include_history_item(self, item):
        run_in_terminal(lambda: print(f'Got item {item}'))
        run_in_terminal(lambda: print(f'Use local history: {self.use_local_history}'))
        if self.use_local_history and item.get('cwd') and os.getcwd() == item.get('cwd'):
            run_in_terminal(lambda: print('Using item'))
            return True
        run_in_terminal(lambda: print('Not using item'))
        return False

Notice in this that there are some Prompt Toolkit-specific run_in_terminal print calls, which I've added just for very verbose logging while developing. I'd remove these when releasing this xontrib, assuming I keep support for non-Prompt Toolkit shells.

In the xontrib, I set a prompt_toolkit2 keybinding to switch this functionality on and off, and to tell the user what mode they've switched to:

import os
from sys import stdout

from prompt_toolkit import keys, print_formatted_text
from prompt_toolkit.application import run_in_terminal

from builtins import __xonsh__
from xonsh.history.json import JsonHistory
from xonsh.platform import ptk_shell_type

def toggle_per_directory_history():
    if isinstance(__xonsh__.history, JsonPerDirectoryHistory):
        hist = __xonsh__.history
        hist.use_local_history = not hist.use_local_history
        if hist.use_local_history:
            return 'local'
            return 'global'

def custom_keybindings(bindings, **kw):
    def do_nothing(func):

    if ptk_shell_type() == 'prompt_toolkit2':
        binder = bindings.add
        binder = bindings.registry.add_binding

    key = ${...}.get('PER_DIRECTORY_HISTORY_TOGGLE')

    def switch_between_global_and_local_history(_):
        new_hist_type = toggle_per_directory_history()
        run_in_terminal(lambda: print(f'Switching to {new_hist_type} history.'))

Finally, I set my history backend to my custom one in my .xonshrc and turned on per-directory history:

from xontrib.per_directory_history import JsonPerDirectoryHistory
$XONSH_HISTORY_BACKEND = JsonPerDirectoryHistory

Initial results

Failure, mostly. Upon opening a gnome-terminal instance, I saw the debugging messages printed from my history backend, which was nice:

Got item {'cwd': '/home/eddie/source/xonsh', 'inp': 'ls\n', 'rtn': 0, 'ts': [1560543198.0524652, 1560543198.1038635]}
Use local history: True
Not using item
Got item {'cwd': '/home/eddie/source/xonsh', 'inp': 'z xonsh\n', 'rtn': 0, 'ts': [1560543197.440549, 1560543197.4461908]}
Use local history: True
Not using item
Got item {'cwd': '/home/eddie', 'inp': 'cd ..\n', 'rtn': 0, 'ts': [1560543047.8660042, 1560543047.8693159]}
Use local history: True
Using item
Got item {'cwd': '/home/eddie/test', 'inp': 'fancy mccheeese\n', 'rtn': 1, 'ts': [1560543038.735667, 1560543039.5000844]}
Use local history: True
Not using item
Got item {'cwd': '/home/eddie/test', 'inp': 'cd test\n', 'rtn': 0, 'ts': [1560543024.288666, 1560543024.2920816]}
Use local history: True
Not using item
Got item {'cwd': '/home/eddie', 'inp': 'ls\n', 'rtn': 0, 'ts': [1560543021.7835386, 1560543021.8042111]}
Use local history: True
Using item
Got item {'cwd': '/home/eddie', 'inp': 'ls\n', 'rtn': 0, 'ts': [1560543020.0776978, 1560543020.1177895]}
Use local history: True
Using item
Got item {'cwd': '/home/eddie/test', 'inp': 'cd test\n', 'rtn': 0, 'ts': [1560542986.1126633, 1560542986.1164727]}
Use local history: True
Not using item

My history backend was apparently being used to load existing history strings, and it was only returning those that matched the cwd, which, in a new gnome-terminal for me is /home/eddie. Notice how commands with cwd info that matches /home/eddie are the only history items being used.

Cool. So I decided to switch to another directory. If things are working as expected, I should be able to enter history commands, go back through the history, and only get commands for this new directory. Before entering any commands in this directory, I shouldn't see any history items!

But I did. I could scroll back through all the history items that my command output just said were being used. What's more, if I switched to yet another directory, I could access all the commands I'd entered in the current prompt session since opening it.


This may be specific to Prompt Toolkit, but I don't know yet. It appears xonsh uses history backends in conjunction with prompt toolkit like this:

  1. xonsh spins up a new shell instance in xonsh/shell.py when a new shell process is started
  2. xonsh creates a Prompt Toolkit instance and hands it all the history data up to this point, which, in the case of JsonHistory, is all the history in our JSON history files
  3. Prompt Toolkit runs the prompt
  4. Each time a command is entered, Prompt Toolkit feeds this command to the xonsh shell, which appends the command history to the history backend in use
  5. Indepedently, Prompt Toolkit maintains its own searchable/scrollable record of the history since the Prompt Toolkit instance was created
  6. The next time a shell is loaded, the previous history is given to Prompt Toolkit from the xonsh history backend.

Do you see my problem? I had indeed changed what history items are loaded by the history backend, but I hadn't changed anything about what the Prompt Toolkit history mechanism does once a shell is up and running.

Where do I go from here?

I am going to bring this post to a close since it contains lots of cool info about how xonsh history backends work, but I will pick back up on my own historical exploits in another post.