Jordan Torbiak

bash extglob: Tips for using negation/!()

Mon, 07 Apr 2025 00:00:00 +0000

Intro

bash’s extglob feature enables some additional pattern-matching syntax and brings globbing capabilities a lot closer to typical popular regular expression dialects. In my experience, though, the only extglob operators I’ve ever had occasion to use are @(<pat>|...), which matches one of the given subpatterns, and !(<pat>|...), which matches anything except one of the given subpatterns. @() is intuitive enough, but in my early experiences with !() it didn’t always behave as I expected, I didn’t find any more about it in the docs, and so I mostly gave up on it except for simple cases where it’s the only thing in the pattern. And, based on the answers to this superuser question, it seems I’m not alone.

!() is both non-greedy and can accept the empty string

The key things to realize about !() is that it can accept the empty string and that unlike the other operators it’s non-greedy. The combination of these attributes means that unless it’s at the end of the pattern you need to be careful to force it to eat some of the haystack, by having the following atom not match the empty string.

For example:

!(jtor*)  # ok: reject files starting with "jtor"
!(jtor*)*  # wrong: accepts anything
!(jtor*)!(*.pdf)  # wrong: equivalent to !(*.pdf)
!(jtor*).!(*.pdf)  # ok: reject files starting with "jtor" or ending with ".pdf"

Following a !() with * or another !() will always make it accept the empty string. When globbing files, bash tries to match !() against as little of the filename as possible, giving it one more character at a time. For each character it:

checks if the filename can be rejected at this position by testing each subpattern in !() against the rest of the haystack
if it can’t be rejected at this position of the haystack, then it checks if the filename can be accepted at this position by checking if the rest of the pattern after !() matches the rest of the haystack

So, if you want !() to eat more, you have to make the following atom fail at earlier positions in the haystack, and thus !(jtor*)*.pdf accepts anything and effectively ignores the !(), while with !(jtor*).pdf the period is going to fail to match in a filename like jtorbiak_resume.pdf until the !() subexpression eats jtorbiak_resume.

Use cases

non-test TypeScript files: !(*.test).ts
without a given prefix or suffix: !(jtor*).!(*.pdf)
without either substring: !(*italic*|*thin*)
directories that don’t start with given prefix: !(dir*)/
- Sadly, it doesn’t seem like you can use a trailing / to match only dirs inside of !().
with none of these extensions: !(*.svg|*.gp|*.png)
with no extension: !(*.*)
with no extension and without some prefix: !(*.*|jtor*)

A particularly neat but less-readable use of !() is to filter out a subset of a glob by nesting another !(). For example, to find files with an extension but that also don’t have some prefix, we can do !(!(*.*)|jtor*). The double-negative is awkward, but I find it’s not so bad if I read it like: exclude files without an extension or that start with “jtor”.

Performance and other options

While extglob is convenient for interactive use, I would generally avoid using it in scripts since it’s not enabled by default and it’s more likely to be confusing for others. Also, it’s not well-suited for dealing with lots of files since it produces filenames on the command-line, and the max length of a command line isn’t huge—it’s only a couple megabytes on my system. And, its performance is poor compared to other options. Some regex engines are optimized such that they have linear time complexity as the haystack scales, by trading off memory to construct a DFA or by doing a Thompson simulation on an NFA (see Regular Expression Matching Can Be Simple And Fast), but extglob in bash is implemented in a simple recursive way and !() is always going to multiply the amount of backtracking that the rest of the pattern needs to do by the number of characters that it needs to eat. Using find/fd or filtering an overly-accepting glob with a for-loop are often better options.

I was curious about what scale the performance of globbing could become an issue at. For finding files without extensions in my home directory, looking at ~100k files and matching 33k of them, using find is noticeably faster. I’m explicitly pruning dot-dirs from the find command since the glob is implicitly doing so.

~$ time find . -name ".?*" -prune -o -not -name "*.*" -print | wc -l
32911

real    0m0.149s
user    0m0.053s
sys     0m0.099s

~$ time printf '%s\n' **/!(*.*) | wc -l
32911

real    0m0.568s
user    0m0.133s
sys     0m0.447s

Appendix: bash source code

The code for extglob is in the bash repo in lib/glob/sm_loop.c. There’s a lot of indirection around sm_loop.c, since it gets included by smatch.c twice, with different preprocessor macro definitions for single and multi-byte characters.

GMATCH and EXTMATCH are mutually recursive, with GMATCH asking EXTMATCH to handle any extglob subexpressions, and EXTMATCH then asking GMATCH to check if each subpattern in an extglob subexpression matches a substring of the haystack. Note that in the outer loop for !() here, srest iterates from s (rest of the haystack) to se (end of the haystack) and GMATCH is asked to check if the subpattern matches from srest (EXTMATCH’s new idea of the rest of the haystack) to s, so on the first iteration it’s going to be checking the empty string from s..s.

// In EXTMATCH(), from sm_loop.c
case '!':           /* match anything *except* one of the patterns */
  for (srest = s; srest <= se; srest++)
    {
      m1 = 0;
      for (psub = p + 1; ; psub = pnext)
        {
          pnext = PATSCAN (psub, pe, L('|'));
          /* If one of the patterns matches, just bail immediately. */
          if (m1 = (GMATCH (s, srest, psub, pnext - 1, NULL, flags) == 0))
            break;
          if (pnext == prest)
            break;
        }
      ...
      if (m1 == 0 && GMATCH (srest, se, prest, pe, NULL, xflags) == 0)
        return (0);
    }
  return (FNM_NOMATCH);

Compiling the test harness in `glob.c`

If you need to convince yourself that extglob works how you think it does, you can compile bash with DEBUG_MATCHING defined to get debug input whenever GMATCH or EXTMATCH are called. You could use bash as a shell with DEBUG_MATCHING defined but that resulted in a lot of debug messages that I didn’t want, so I used the little test harness in glob.c, which just prints the results of matching the patterns given on the command line with the contents of the current directory. glob.c depends on a lot of the same stuff that bash itself does, so it’s a lot easier to compile it in the same way that you would bash instead of trying to isolate it. Here’s what worked for me.

define DEBUG_MATCHING by giving CFLAGS to configure via an environment variable:

$ CFLAGS=’-g -O2 -DDEBUG_MATCHING’ ./configure
Comment out main() in shell.c, and uncomment main() in glob.c
fix iteration over the matched values in main() in glob.c by using a different loop variable, since i is already being used to iterate over argv

Then, run make and maybe cp bash glob to give the binary a more accurate name. Then it can be run like:

$ ./glob 'g!(lob)'
...
gmatch: string = general.h; se =
gmatch: pattern = g!(lob); pe =
extmatch: xc = !
extmatch: s = eneral.h; se =
extmatch: p = (lob); pe =
extmatch: flags = 33
gmatch: string = eneral.h; se = eneral.h
...

Using the same colors for the terminal and Vim

Thu, 25 Jan 2024 00:00:00 +0000

In short, I’ve settled on a dark 4-bit terminal palette that is readable for the default and black background colors and use other techniques to workaround unreadable foreground/background combinations, which are mostly when two relatively light colors are put together. I do :set t_Co=16 in my vimrc so themes that check it use the terminal palette instead of the absolute 8-bit color model, and use autocommands to modify Vim highlight groups that colorschemes often set to an unreadable combination of my colors. And in the end I wrote a basic colorscheme to suit the terminal colors I chose.

Warning, I just learned a lot of the stuff in this post, and it’s difficult to make definitive statements about terminals due to their long history and multitude of implementations, so I bet I’m kinda wrong or at least not-technically-correct about a lot of this. Don’t trust me too much. Also, I’m writing this from a perspective of using classic Vim on Linux.

Motivation

My primary goal with terminal colors is for text to be readable, with a medium amount of contrast for the most common combinations of foreground and background colors. Years ago I remember often needing to switch Vim colorschemes or turn off syntax highlighting to avoid some highlight groups from being unreadable for certain filetypes or when I was doing a diff, which was a pain. And to save effort, if I come up with colors that are comfortable for me, I’d like to be able to use them in my most commonly used programs; at the very least in the shell and Vim.

Choosing a color model

Wikipedia has some good information on the color models used in terminals. My impression is that most of the popular terminal emulators now support most of the color models that have been developed. This table isn’t exhaustive.

bits	colors	notes
1-bit	2	monochrome
3-bit	8	superceded by 4-bit, but its influence remains visible
4-bit	16+2	widely supported
7-ish?	88	4x4x4 color cube, used mostly by older terminals like rxvt and xterm-88color, designed to save memory in the X server colormap vs 8-bit color
8-bit	256	6x6x6 color cube, typically defined the same as in xterm. Depending on the terminal it can be possible to redefine these colors using escapes or X resources, but generally they’re considered to be “absolute”/fixed.
24‑bit	lots	“true color”, with each of RGB specified from 0-255. Definitely absolute. Fairly well-supported in terminals, but less so in terminal-based programs.

4-bit color is the obvious choice of color model when designing a custom palette, since it has been widely supported by terminals and terminal-based programs for a long time now, most terminals let you change the colors easily, and picking 18 colors doesn’t take too long. With it we can pick a default foreground color, a default background color, 8 normal colors that can always be used for the foreground or backgorund, and 8 bold/bright colors whose use can sometimes be limited to when bold is selected for the foreground, but usually can be used whenever.

To understand how the bold/bright colors are used, we need to get into the weeds a bit. The related ANSI/ECMA/ISO standards from the 1970s don’t say much on how to implement SGR 1, the “bold” escape sequence, describing it as “BOLD OR INCREASED INTENSITY”, so a common approach is for terminals to both use a bolder font as well as to change the foreground to the corresponding bright color, and in some terminals this is the only way to access the bright colors. Later terminals introduced the non-standard escape sequences SGR 90-97 and SGR 100-107 to use the bright colors independently of setting bold; xterm supports these, and for a long time xterm was both popular and the reference implementation of a terminal emulator for the X Window System, and so my impression is that most modern terminals try to be xterm-compatible, to some extent, and also support them. When I queried terminfo with tput -T"${term:?}" setab, the popular terminals that I could think of (and that also exist in terminfo) apparently support them too, including: konsole, kitty, alacritty, st, and VTE-based terminals like GNOME, xfce, guake, sakura, and terminator.

Choosing colors

There’s a million ways to design a color scheme, but I went with 4bit Terminal Color Scheme Designer, which I’ll abbreviate as 4TCSD. It doesn’t let you control everything, but it is great for quickly choosing some colors that are collectively cohesive and individually still have the intended meaning/symbolism: for example, errors are often displayed in red, so I want my red to be red enough to convey connotations of danger. I particularly recommend playing with the “Dye” tab, which adds a color cast to a desired subset of colors, and can have a big effect on how cohesive the theme looks.

There’s also terminal.sexy, which is very flexible, has lots of preview facilities, included themes, and export formats. terminal.sexy is good for tweaking individual colors, but note that while it shows bright colors in the preview templates it doesn’t bold the font, so with an imported scheme from 4TCSD the bright colors probably won’t look quite how you’d expect. Still, using the /misc/rows and /misc/columns templates to preview the colors can help stay true to the color names while also quickly checking readability for various foreground/background combinations:

It doesn’t seem possible to have every foreground color be readable on every background color, so for a dark theme I tried to optimize having the foreground colors be readable on the default and black background colors while also keeping the constrast somewhere in the middle. If programs define funky color combos like yellow on green, I’m resigned to working around that as needed by choosing alternate program-specific themes or disabling color for them.

I ended up with something like in the image of 4TCSD above. Unfortunately, 4TCSD doesn’t support importing themes, so recreating something you’ve made before can be a challenge.

Configuring the terminal

Permanently changing the color palette is different for each terminal. My terminal’s config format wasn’t supported by 4TCSD and I initially couldn’t find any other tools to convert between formats on the command line so I wrote a quick-and-dirty script in Python called conv4bit. Later I realized that the converter for terminal.sexy also has a command-line interface: termcolors. terminal.sexy supports different export formats than 4TCSD, so exporting for xterm from 4TCSD and importing that as Xresources in terminal.sexy might be helpful.

A lot of terminals support using xterm-style OSC 4/10/11/12 escapes to change the 4-bit palette on-the-fly, which is especially convenient for trying out and editing themes. I had fun cloning the Gogh repo, applying one of its themes with conv4bit -ofmt osc "themes/${theme:?}" -, and then attaching to an existing tmux session to preview the theme in a variety of situations.

Configuring Vim

Back in Vim v7.0, the builtin colorschemes used color names from :help cterm-colors for color terminals, which are interpreted differently depending on t_Co (explained shortly), and 24-bit color for the GUI. And in Dec 2023 (Vim v9.1) the builtin themes were rewritten using the colortemplate plugin, and now each has specific support for 256, 16, 8, and 2 colors. The builtin themes still choose how many colors to use based on the t_Co option, which represents the max number of colors that the terminal supports (up to 256) and is retrieved from the terminfo db (see the Co capability in terminfo(5)) based on the value of the TERM environment variable, but you can override t_Co in your vimrc to get themes to use a lower-fidelity color model if desired. (24-bit color, on the other hand, is advertised by terminals in various ways and can be enabled in vim with :set termguicolors.) So, the easiest way to get Vim to use the terminal’s 4-bit palette is to do :set t_Co=16 in your vimrc.

When testing out Vim color themes I’d recommend using :help highlight-groups to identify unreadable groups early, and :help group-name to see the hierarchy of groups commonly used for programming language syntax. Note that most of the default highlight groups have a help entry under hl-<name> (eg hl-DiffAdd), which describes what it’s for. Highlight groups for specific languages will mostly link to the ones defined by default.

Also, it’s quite helpful to see what syntax/highlight group is under the cursor, and the following function can be bound to a key to do that. A simplified description is that it shows the matching and linked syntax groups. The effective highlight group could be different from what SynGroup() shows, though, such as when doing a diff, using visual mode, etc. Note that syntax and highlight group names are kind of shared, and syntax group names can be used with the :highlight command to create a corresponding highlight group or link to one, like with :hi link pythonStatement Statement, which links the pythonStatement syntax group to the Statement highlight group.

function! SynGroup()
    let id = synID(line('.'), col('.'), 1)
    echo synIDattr(id, 'name') . ' -> ' . synIDattr(synIDtrans(id), 'name')
endfunction

There’s a few highlight groups that I dislike in a lot of the builtin themes. I almost always want my default background color to be used, so I override Normal to have ctermbg=NONE when 4-bit color is being used. It can be tricky to get the search-related groups emphasized enough while also being readable, and striking the right balance really depends on what your terminal colors are. And for diff, putting red/green/yellow on black for removed/added/changed seems to work well compared to using more colorful combinations, assuming black is easily distinguished from the default background color. In the snippet below, the ColorScheme event fires after any (*) colorscheme is loaded. See :help colorscheme-override for more info on overriding colorschemes, and you may need to read the docs for autocommands as well depending on your familiarity with them. With Vim9 script you can put multiple autocommands in curly braces, but I like to keep my vimrc compatible with somewhat older Vim versions too, so I’m defining a function instead. In my vimrc:

function ModColorScheme()
    " Customize colorscheme when using 4-bit color.
    if str2nr(&t_Co) == 16
        hi Normal ctermfg=NONE ctermbg=NONE

        hi Visual ctermfg=Black ctermbg=Cyan cterm=NONE
        hi Search ctermfg=Red ctermbg=Black cterm=bold
        hi IncSearch ctermfg=White ctermbg=DarkRed cterm=bold

        hi DiffAdd ctermfg=DarkGreen ctermbg=Black cterm=NONE
        hi DiffChange ctermfg=DarkYellow ctermbg=Black cterm=NONE
        hi DiffDelete ctermfg=DarkRed ctermbg=Black cterm=NONE
        hi DiffText ctermfg=Black ctermbg=DarkYellow cterm=bold
    endif
endfunction
augroup color_mods
    au!
    au ColorScheme * call ModColorScheme()
augroup END

For diffs this results in:

One downside to using terminal colors is that themes on the opposite side of the light/dark spectrum probably won’t work well . My terminal colors don’t work well with light colorschemes, so if I did want to use a light theme I’d either use 8 or 24-bit color or choose a light set of terminal colors.

Vim’s cterm-colors

When inspecting Vim colorthemes or highlight groups, it’s helpful to know what the color names and numbers mean. As explained at :help cterm-colors, Vim uses MS Windows names for colors, which is confusing if you’re thinking in terms of the ANSI names. And also, later in the Vim help it’s explained that ANSI-style terminals use the NR-8 column of numbers in the table below and add 8 for the bright variants, and this includes xterm, and thus likely also includes most terminals on Linux. Here’s the table from :help cterm-colors, with a column of ANSI-ish color names added.

NR-16  NR-8  Vim color name                    ANSI color name
0      0     Black                             black
1      4     DarkBlue                          blue
2      2     DarkGreen                         green
3      6     DarkCyan                          cyan
4      1     DarkRed                           red
5      5     DarkMagenta                       magenta
6      3     Brown, DarkYellow                 yellow
7      7     LightGray, LightGrey, Gray, Grey  white
8      0*    DarkGray, DarkGrey                bright black
9      4*    Blue, LightBlue                   bright blue
10     2*    Green, LightGreen                 bright green
11     6*    Cyan, LightCyan                   bright cyan
12     1*    Red, LightRed                     bright red
13     5*    Magenta, LightMagenta             bright magenta
14     3*    Yellow, LightYellow               bright yellow
15     7*    White                             bright white

Basically, DarkRed maps to ANSI red (1), while LightRed and Red map to ANSI bright red (9), and this pattern holds for all the other colors too except for white and black.

Clearly the number 9 doesn’t match the ANSI-style SGR escape sequence to set the foreground to bright red, which is SGR 91. 9 will get translated to the appropriate escape sequence for the terminal during display, likely by ncurses in concert with terminfo. For example, the terminfo setaf or “Set ANSI Foreground” capability gives a recipe in a simple stack-based programming language (described in terminfo(5)) to translate the color number to the right escape sequence for the current terminal, and the tput program that comes with ncurses can interpret that recipe.

tput_demo.sh:

#!/bin/bash
set -eu
setaf=$(tput setaf)
set_fg_red=$(tput setaf 1)
set_fg_bright_red=$(tput setaf 9)
clear_attrs=$(tput sgr0)

declare -p setaf set_fg_red set_fg_bright_red clear_attrs
echo "${set_fg_red}hi ${set_fg_bright_red}there ${clear_attrs}again"

And in a terminal with different colors defined for red and bright red:

Writing a Vim colorscheme

I’ve been pretty happy using the pablo or default builtin colorschemes along with my set of terminal colors since 2016, but after writing the first version of this post and learning more about how colors and themes work in Vim, I started overriding more and more highlight groups that I wasn’t totally happy with until it was obvious that I should just write my own theme. For people writing configurable themes with light and dark variants for multiple color models and both Vim and Neovim it makes sense to script the theme generation, but since I just wanted a 4-bit dark theme for Vim and the palette is already set, I didn’t need any indirection and simply modified a list of :highlight commands based on the default scheme.

I used colortemplate to generate a clone of the default theme using its default_clone.colortemplate and then removed almost everything except the header and 4-bit if s:t_Co >= 16 section, but you could also just take the 4-bit section from any other builtin theme in Vim v9.1+, since they’re all generated with colortemplate. Prior to v9.1 some of the builtin themes don’t override all of the groups that default defines, so they’re less useful as templates.

I ended up with forbit.vim, which starts like this:

<snip header>

set background=dark
hi clear
let g:colors_name = 'forbit'

hi Normal ctermfg=NONE ctermbg=NONE cterm=NONE
hi Comment ctermfg=lightblue ctermbg=NONE cterm=NONE
hi Constant ctermfg=darkmagenta ctermbg=NONE cterm=NONE
<continues...>

To apply a colorscheme they just get sourced, so you could put it anywhere, but Vim looks for colorschemes in a few locations by default, including ~/.vim/colors, so after putting it there I could simply run :colorscheme forbit.

forbit doesn’t pass the $VIMRUNTIME/colors/tools/check_colors.vim script, which is recommended for colorschemes that are intended to be shared, but it’s good enough for my purposes. The point is to make it easy for me to customize colors as needed, which I wish I had started doing earlier.

Appendix: query if bright background is supported

#!/bin/bash
set -euo pipefail

terms=(
    terminator
    alacritty
    konsole-256color
    gnome-256color
    kitty
    st-256color
    xterm-256color
    vte-256color
)
for t in "${terms[@]}"; do
    setab=$(tput -T"$t" setab) || continue
    # Look for the "then" (%t) part of a conditional which does:
    # - 10: print "10"
    # - %p1: push param1 on the stack
    # - %{8}: push a literal 8 on the stack
    # - %-: pop the top two values and push their difference
    # - %d: pop a value and print it as a int
    [[ "$setab" = *'%t10%p1%{8}%-%d'* ]] && bright_bg=yes || bright_bg=no
    echo "$t $bright_bg"
done

Addendum: changelog

2024-02-12

Remove paragraph about 4TCSD themes having the same color for the normal and bright variants; somehow I didn’t notice my color lightness sliders were overlapping. Update the 4TCSD screenshot, too.
Update vimrc snippets to avoid Vim9 script for now.
Mention better ways to check highlight groups: :help group-name and :help highlight-groups.
Add section on writing my own Vim colorscheme.
Mention conv4bit, a script to convert between 4-bit theme formats.
Move the terminfo querying script into the appendix instead of linking to the file.

Normalize volume system-wide with PulseAudio

Wed, 20 Dec 2023 00:00:00 +0000

Motivation

I usually watch YouTube videos while I wash dishes, and if videos have very different volume levels it can be either annoying to not understand what’s being said or painful to endure loud sounds while I dry my hands so that I can adjust the volume. So I’ve been delighted over the past couple years using a compressor filter from Steve Harris’ LADSPA plugin suite with PulseAudio to normalize volume for all audio output from my laptop. I’m quite willing to trade dynamic range for listening comfort, and for me even music doesn’t sound noticeably worse with the compressor filter.

In case you’re wondering, LADSPA is the Linux Audio Developer’s Simple Plugin API, and basically it’s an interface for shared libraries that allows plugins to define some control ports/params and give callbacks to process samples. It’s like a simpler version of VST, if you’ve ever used synth or effects plugins in a DAW and have heard of that.

PulseAudio configuration

For configuring PulseAudio I referred to this answer on the Ask Ubuntu StackExchange. On Arch I installed the swh-plugins package. I then added a PulseAudio config drop-in under /etc/pulse/defaultpa.d, although you could instead add it to a per-user config in $XDG_CONFIG_HOME/pulse/client.conf, as described on the Arch wiki. For PulseAudio CLI syntax and commands, see the pulse-cli-syntax(5) manpage.

Sorry about the long lines in this and several other snippets, but neither PulseAudio CLI syntax or udev rules support line continuations and the likely confusion over including “fake” ones doesn’t seem worth it.

/etc/pulse/defaultpa.d/compressor.pa:

.ifexists module-ladspa-sink.so
.nofail
load-module module-ladspa-sink sink_name=compressor plugin=sc4_1882 label=sc4 control=1,1.5,400,-30,20,5,24
set-default-sink compressor
.fail
.endif

module-ladspa-sink takes a sink_master param (previously named master) to define where the processed audio goes, but I’m leaving it as the default. See the PulseAudio builtin module docs. If you wanted to route the output to a different sink, you can find its name with pactl list short sinks.

I’m still using the SC4 filter settings from that Ask Ubuntu answer, except that I’ve increased the makeup gain so there’s a smaller volume difference between having the filter on/off, to avoid having sound blaring if I ever accidentally turn off the filter. Mostly the knobs are cranked to always be compressing as soon and hard as possible, except knee radius, which is set in the middle of the allowed values, but when adjusting it I can’t notice a difference. With these settings it feels like the volume is always being reduced, which seems good to me.

RMS/peak ratio: 1
Attack time (ms): 1.5
Release time (ms): 400
Threshold level (dB): -30
Gain reduction ratio (1:n): 20
Knee radius: 5
Makeup gain (dB): 24

If you want to get a better sense of how the filter params work and see their allowed ranges, here’s the code for the SC4 filter. And if you’re looking at the code you might also want to peek at the little LADSPA spec.

After writing the PulseAudio config, I use pulseaudio -k to kill the server, and then the stock systemd config for my system immediately restarts it for me with the new settings. I can see any errors with my PulseAudio config by following syslog with journalctl -f.

The above config might be all you need, but I needed to do a bit more to have things work with my Bluetooth headphones.

Manually route a LADSPA sink to Bluetooth headphones

When you connect Bluetooth headphones a new sink gets added to PulseAudio, but unless we do something our compressor is going to stay routed to our ALSA output or wherever we pointed it at PulseAudio startup, and the output to the headphones won’t be filtered. For a few years I was keeping a pavucontrol window open and manually rerouting the compressor’s output to my headphones after connecting them, but when I set up my new laptop pavucontrol no longer gave me a dropdown to do manual routing for LADSPA sinks, and I haven’t figured out why yet. So, I started running a little script every time I connected my headphones, which adds another LADSPA sink with the output routed to my headphones. The sink name is based on the headphones’ MAC address, and is constant, so I just needed to look it up once using pactl list short sinks and provide it as the sink_master param.

pacmd <<EOF
load-module module-ladspa-sink sink_name=bluetooth_compressor plugin=sc4_1882 label=sc4 sink_master=bluez_sink.00_1B_66_A1_45_12.a2dp_sink control=1,1.5,400,-30,20,5,24
set-default-sink bluetooth_compressor
EOF

After disconnecting the headphones the LADSPA sink gets destroyed, so back when I was doing the routing in pavucontrol I would either need to change the LADSPA sink’s master back to my soundcard before disconnecting my headphones to avoid it getting destroyed, or I’d need to recreate the sink by restarting pulseaudio. With my new laptop my problem was that after the LADSPA sink for my headphones was destroyed, PulseAudio would choose my soundcard as the default sink instead of the compressor that outputs to it. I could run another script after disconnecting my headphones, but instead I finally had the motivation to automate the routing. I couldn’t figure out how to fix this with the builtin PulseAudio modules, and I considered writing a PulseAudio module to allow prioritizing sinks, but running shell scripts on Bluetooth connection/disconnection events seemed easier and more flexible.

Automatically route a LADSPA sink to Bluetooth headphones

Find udev info

udev is part of systemd and lets us run shell scripts or start systemd services on hardware events. udev(7) is a useful reference when writing rules, and the Arch wiki page on it has some good info, too. Before writing the rule I needed to find some way to identify my headphones. By running udevadm monitor before connecting my headphones, I could see the related devices the kernel makes on connection:

$ udevadm monitor
...
KERNEL[80470.275026] add      /devices/pci0000:00/0000:00:08.1/0000:c1:00.3/usb1/1-5/1-5:1.0/bluetooth/hci0/hci0:50 (bluetooth)
UDEV  [80470.276714] add      /devices/pci0000:00/0000:00:08.1/0000:c1:00.3/usb1/1-5/1-5:1.0/bluetooth/hci0/hci0:50 (bluetooth)
KERNEL[80472.139125] add      /devices/virtual/input/input28 (input)
KERNEL[80472.139222] add      /devices/virtual/input/input28/event13 (input)
UDEV  [80472.140796] add      /devices/virtual/input/input28 (input)
UDEV  [80472.172162] add      /devices/virtual/input/input28/event13 (input)

And I could get more information about devices with udevadm info -ap:

$ udevadm info -ap /devices/virtual/input/input28
...
  looking at device '/devices/virtual/input/input28':
    KERNEL=="input28"
    SUBSYSTEM=="input"
    DRIVER==""
    ...
    ATTR{id/bustype}=="0005"
    ATTR{id/product}=="004b"
    ATTR{id/vendor}=="0082"
    ATTR{id/version}=="0103"
    ATTR{inhibited}=="0"
    ATTR{name}=="HD1 M2 AEBT (AVRCP)"
    ATTR{phys}=="14:ac:60:46:87:9e"
    ...

I don’t know how to write a udev rule that would match all Bluetooth headphones, but this is enough information to write a rule for my specific ones, at least. An important thing to note now, though, is that the pulseaudio daemon is per-user, so I need to run pacmd as my user. I could use the RUN operator in the udev rule along with sudo -u <user> and have separate rules for ACTION=="add" and ACTION=="remove", but hard-coding my username feels kinda bad, so instead I went with using ENV{SYSTEMD_USER_WANTS} and writing a device-bound systemd user service that’ll automatically be stopped when the device goes away. To make that work, I need to TAG+="systemd" the device so systemd picks it up, and somehow give the device name that systemd uses to a parametrized service (“instantiated” in systemd parlance), so it can be bound to it.

With a udev rule like this in /etc/udev/rules.d/99-sennheiser.rules:

ACTION=="add", ATTR{name}=="HD1 M2 AEBT (AVRCP)", TAG+="systemd"

And after reloading the udev rules:

sudo udevadm control --reload

When I connect the headphones I see a corresponding device in my systemctl output:

UNIT                                      LOAD    ACTIVE  SUB      DESCRIPTION
sys-devices-virtual-input-input28.device  loaded  active  plugged  /sys/devices/virtual/input/input28

Which is cool, but not terribly useful yet. We’ll come back to writing the udev rule. First I need to figure out what the systemd user service will be like, so I know what parameters it needs.

systemd user service

Instantiated systemd services are named such that there’s an @ before the unit extension and then, when being started, they can be given an argument in their name after the @. For example, if you run systemctl you’ll probably see a getty@tty1.service, where getty@.service is the unit name and tty1 is the argument. I put my service file at $XDG_CONFIG_HOME/systemd/user/bt-compress@.service. After modifying that file I need to run systemctl --user daemon-reload to load the changes and use journalctl when connecting my headphones to look for errors in syslog. There’s not much point “installing” our service in the systemd sense, since we can’t start it unless the headphones are connected, so our service file doesn’t have an [Install] section, and it won’t show up in systemctl --user output unless it’s running or failed.

$XDG_CONFIG_HOME/systemd/user/bt-compress@.service:

[Unit]
BindTo=%i.device
After=%i.device

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=%h/bin/bt-compress start
ExecStop=%h/bin/bt-compress stop

By default a systemd service is Type=simple and the ExecStart option will run a long-running process, but we just need to run some short shell scripts at the beginning and end of the device’s life. Looking through systemd.service(5), we see that for Type=oneshot the service is considered up after the ExecStart program exits for the purpose of starting dependencies, but it will soon transition to the “dead” state unless we also give RemainAfterExit=yes. We need it to “remain after exit” so that systemd has an active service to stop when the device goes away.

The escapes available in systemd units can be found in systemd.unit(5). %h expands to the user’s home directory and %i expands to the string between @ and .service in the service name. So, we’d like to pass the extension-less systemd device unit name in there, which we saw in the last section was sys-devices-virtual-input-input28. But if I reconnect my headphones I see that the number at the end of the name increments each time I connect, so it seems like we’ll need to get the device path and convert it to the systemd device unit name in the udev rule somehow.

PulseAudio configuration script

But before we get to the udev rule, here’s the script that our systemd service calls. It’s pretty straightforward:

~/bin/bt-compress:

#!/bin/bash
set -euo pipefail

log() {
    local lvl=${1:?No level given}; shift
    local msg=${1:?No message given}; shift
    logger -p "$lvl" -t bt-compress "$msg"
}

usage="bt-compress start|stop"
if [[ $# -ne 1 ]]; then
    echo "$usage" >&2
    exit 1
fi
cmd=${1}; shift
case "$cmd" in
start)
    log info 'set bluetooth_compressor as default sink'
    pacmd <<EOF
load-module module-ladspa-sink sink_name=bluetooth_compressor plugin=sc4_1882 label=sc4 sink_master=bluez_sink.00_1B_66_A1_45_12.a2dp_sink control=1,1.5,400,-30,20,5,12
set-default-sink bluetooth_compressor
EOF
;;
stop)
    log info 'set compressor as default sink'
    pacmd 'set-default-sink compressor'
;;
*)
    echo "unexpected cmd: $cmd" >&2
    exit 1
;;
esac

udev rule

Finally, let’s wire up udev and systemd. According to systemd.device(5), the SYSTEMD_USER_WANTS udev device property adds a Wants= dependency from a device to a service, and is read by user service manager instances. And according to udev(7), we can set device properties using ENV{<key>}=<value>. And if we use udevadm test to inspect the device properties, we can see that the DEVPATH property is included:

$ sudo udevadm test --action="add" /devices/virtual/input/input28
...
DEVPATH=/devices/virtual/input/input28
PRODUCT=5/82/4b/103
NAME="HD1 M2 AEBT (AVRCP)"
PHYS="14:ac:60:46:87:9e"
PROP=0
EV=100007
KEY=...
REL=0
MODALIAS=...
ACTION=add
SUBSYSTEM=input
TAGS=:seat:systemd:
ID_INPUT=1
ID_INPUT_KEY=1
ID_BUS=bluetooth
CURRENT_TAGS=:seat:systemd:
SYSTEMD_USER_WANTS=bt-compress@.service
USEC_INITIALIZED=259580475046

Apparently we’re doing what systemd considers the usual thing, since if we just set SYSTEMD_USER_WANTS to our base service name, bt-compress@.service, it’ll automatically pick up the DEVPATH from the udev event, prefix it with the sysfs mount point, escape it, and give the escaped device path as the argument to our service. So our udev rule can just be this:

ACTION=="add", ATTR{name}=="HD1 M2 AEBT (AVRCP)", TAG+="systemd", ENV{SYSTEMD_USER_WANTS}="bt-compress@.service"

If you have trouble getting this to work, using udevadm test as above can be useful, as it prints a bunch of diagnostics, which I have elided here.

But if we needed to do the escaping ourselves, we could use systemd-escape:

$ systemd-escape --template bt-compress@.service --path /sys/devices/virtual/input/input28
bt-compress@sys-devices-virtual-input-input28.service

And surprisingly to me, there’s a udev key for running shell commands and capturing their output: the PROGRAM key. The results of PROGRAM are then available via the %c or $result escape, and while the manpage doesn’t currently include the ENV key in the list of keys where escapes are available, it apparently works. Note that $devpath doesn’t include the sysfs mount point, so we need to add it when running systemd-escape.

ACTION=="add", ATTR{name}=="HD1 M2 AEBT (AVRCP)", TAG+="systemd", PROGRAM="/usr/bin/systemd-escape -p --template=bt-compress@.service /sys$devpath", ENV{SYSTEMD_USER_WANTS}+="$result"

Good luck

I doubt you got this far unless you’re motivated enough to set something similar up on your own machine. Good luck, and I hope this helped.

Addendum: change log

2024-03-23: Change release time from 401ms to 400ms. I must have hit Ctrl-A in Vim and incremented it by one.

Revisiting pruned approaches

Fri, 29 Sep 2017 00:00:00 +0000

Recently I’ve noticed a few cases where I’ve been solving a problem and determined that some approach doesn’t work, only to get stuck on other approaches too and finally return to the supposedly broken approach and realize that it actually works. Occasionally I’ll explicitly set out to rewrite a piece of broken code using the same approach and it’ll work and I can compare the two, but for most problems the history will be gone and I’m left wondering what I was doing wrong before.

This doesn’t only happen for programming problems, either. I was searching through my journal looking for something else yesterday and happened to see an unsuccessful skin-care regime I undertook about a year ago. It called for the use of two products and I followed it for a week but it worsened my skin’s condition so much that I stopped. A few months ago I started using one of the products alone, having completely forgotten about my prior experience, and it proved super effective. I likely could have seen those results a year earlier if I’d changed one variable at a time and hadn’t conflated the failure of the combo with the failure of each individually.

These false negatives for good approaches aren’t surprising: I fail to solve many problems on the first try even when I’m using an approach that’s been shown by others to work. And when it’s unclear why something is broken, as it often is, it’s easy to get the impression that the approach is hopeless without having enough information to confidently rule it out. The best strategy I can think of to mitigate the cost of these false negatives is to take better notes. I often take notes of the approaches I’m trying, but clearly I need to be more disciplined in noting why I abandon an approach as well so I can judge later whether I did so mistakenly.

See also: Nelson Elhage’s Lab Notebooking for the Software Engineer, which gave more purpose to my existing notetaking habit.

Histograms of timestamped data: gnuplot vs matplotlib

Sun, 16 Jul 2017 00:00:00 +0000

Introduction

I wanted to learn a charting tool that is:

convenient enough to quickly create rough data visualizations without it feeling like an indulgent digression
fast enough to visualize large datasets, maybe millions of points
scriptable, so chart source code plays well with version control

Without doing much research on the options I decided to start with the venerable gnuplot. Data I’ve needed to analyze at previous jobs have always been time series or event streams, so I chose visualizing my bash history as a practice task. After working through a series of charts of increasing sophistication with gnuplot I started worrying that other tools might be more convenient, so I replicated my gnuplot journey with matplotlib. In the end my worries were unfounded and I’m actually pretty happy with gnuplot, especially for quick-and-dirty jobs. matplotlib certainly has some advantages, like having a popular, sensible language like Python as its interface, and possibly being an alternative to R when used along with pandas, but gnuplot’s brevity is difficult to resist.

I’ve been disappointed with the resources I’ve found for charting timestamped data in both gnuplot and matplotlib, motivating me to share my experience.

If you’re less interested in the progression and just want to see how I’d recommend making a chart like the below from a timestamped log, skip here.

The data

I’m examining my bash history to see when I’ve been most active over the past couple days. To get some data we can set HISTTIMEFORMAT so bash’s history command annotates entries with timestamps:

$ HISTTIMEFORMAT="%Y-%m-%dT%H:%M:%S " history >shellhist
$ tail shellhist
10215  2017-07-01T09:52:36 make dist
10216  2017-07-01T09:52:40 git tag
10217  2017-07-01T09:52:44 git tag v0.002003
10218  2017-07-01T09:52:57 cpan-upload -u TORBIAK App-Git-Autofixup.tar.gz
10219  2017-07-01T09:53:08 git push
10220  2017-07-01T09:53:13 git push --tags
10221  2017-07-01T09:53:31 make realclean
10222  2017-07-01T09:53:36 rm App-Git-Autofixup-0.002003.tar.gz
10223  2017-07-01T09:53:38 fg

gnuplot

A rough start: points

To get a rough visualization of when commands were entered we can tell gnuplot we’re dealing with time data on the x-axis by setting xdata time and timefmt and plot each command as a point, spreading the points across the y-axis at random to make it easier to get a sense of their density. gnuplot interprets using 2:(rand(0)) to mean that we’re using the second column of shellhist for the x-values, and random floats in the interval [0:1] as the y-values. The parentheses around rand(0) are necessary to signal that we want to use the value of an expression instead of selecting a column from the data file.¹

shellhist_gp_points.gp:

set xdata time
set timefmt "%Y-%m-%dT%H:%M:%S"
plot 'shellhist' using 2:(rand(0)) with points

Plot the script from the gnuplot prompt with load 'shellhist_gp_points.gp' or using a shell command like:

GNUTERM=svg gnuplot shellhist_gp_points.gp >shellhist_gp_points.svg

Binning times

The above chart gives a rough idea of when I was executing commands, but it’d be interesting to get more specific and quantify how many commands I was running per hour or minute. To do that we need to put entries in bins, mapping the command timestamps to intervals of the desired length. If we’re glancing at the manual we might think that the histograms plotting style does what we want, but it doesn’t: it expects data to already be binned. We need to bin the data some other way; we could use a different tool or programming language, or we can use the smooth frequency modifier for gnuplot’s using clause, which adds ups the y-values for each unique x-value encountered.² So if we can get the time values as epoch seconds, round them to the start of the interval³ they’re in, and set the y-values as 1.0, then smooth frequency will add up those ones, creating the needed mapping of interval start times to number of commands executed.

gnuplot represents times as epoch seconds⁴ and since we’ve set timefmt and xdata time we might expect that referencing column 2 in the expression for the x-values would evaluate to epoch seconds, but it actually follows the usual behaviour and evaluates to the first number present in the field, 2017 in our case. We’ve specified that xdata is time-based, not that column 2 is only to be interpreted as a time.

# WRONG. $2 evaluates to 2017, not epoch seconds.
set xdata time
set timefmt "%Y-%m-%dT%H:%M:%S"
binwidth = 3600
plot 'shellhist' using ($2 - ($2 % binwidth)):(1.0) with impulses

Instead we use the timecolumn function to get epoch seconds as a float. Most of the time we can rely on type coercion but to use the modulus (%) operator t needs to be converted to an int. Since timecolumn takes a format there’s no need to set timefmt anymore. Also, we can avoid parsing the date twice and make the script more readable by defining a user-defined function we’ll call bin.

shellhist_gp_line.gp:

set xdata time

binwidth = 3600 # 1h in seconds
bin(t) = (t - (int(t) % binwidth))

plot 'shellhist' using (bin(timecolumn(2, "%Y-%m-%dT%H:%M:%S"))):(1.0) \
    smooth freq with linespoints

Discontinuities

If we look closely at the chart above there’s a discontinuity on June 28, from 1 to ~30. This is due to multiline commands in my shell history: lines after the first don’t match the column specification given in the using clause and column 1 for these lines ends up being interpreted as NaN, resulting in a discontinuity in the data and multiple points being drawn for the same time interval. Instead of filtering the data with a separate script it was easier to ignore these lines using set datafile missing NaN. Sometimes discontinuities are easier to notice when plotted with the boxes plot style, where they show up as multiple lines within a box.

Improve readability

If we wanted to share this chart there’s a number of other worthwhile improvements:

Add a title.
Hide the legend (“key” in gnuplot parlance), which is more distracting than useful in this case.
Include time of day in the x-axis labels.
Remove extraneous xtics, ytics, and border lines, depending on our preference.

Histograms are commonly plotted using boxes, which is particularly nice when the boxes cover the x-axis intervals they represent. By default boxes are centered on their x-value, so we need to change the bin function to offset them slightly. Also, we’ll need to set boxwidth, since by default adjacent boxes are extended until they touch.

shellhist_gp_bars.gp:

binwidth = 3600 # 1h in seconds
bin(t) = (t - (int(t) % binwidth) + binwidth/2)

set xdata time
set datafile missing NaN

set boxwidth binwidth

set xtics format "%b %d %H:%M" time rotate
set xtics nomirror
set ytics nomirror
set key off
set border 1+2 # Set the bits for bottom and left borders.
set title 'Commands Run Per Hour'

plot 'shellhist' using (bin(timecolumn(2, "%Y-%m-%dT%H:%M:%S"))):(1.0) \
    smooth freq with boxes

Multiple distributions with lines

Plotting multiple distributions with lines is only slightly trickier. For this data plotting all possible series would clutter the chart severely, so we instead find the most frequent commands using gnuplot’s system function and a pipeline and only plot those. Then we use the plot command’s for clause to iterate over the data for each of the top commands, binning each separately by using ternary operator in the using clause to count one command per iteration.

If a line is missing columns then stringcolumn(3) will evaluate to NaN and a type error will be thrown when comparing it as a string to cmd. To avoid this we can clean up the file or use valid to check that a line has usable fields. valid only checks that columns aren’t NaN, though, so we can’t use it on string columns, as I was initially tempted to. valid(3) is always false since a string that don’t contain numbers converts to NaN when coerced to a float.

While I don’t want to plot all the series for the sake of readability, I would like to know how many command executions aren’t covered by the top few, so I’ve also plotted a line showing the overal number of commands run per hour. As shown in the last line of the example the previous datafile given to a plot command can be reused by specifying the empty string (''),

Most gnuplot command names and modifiers can be abbreviated as long as they’re unambiguous, and I’ve taken advantage of this by writing u instead of using for the first plot clause to help reduce the line length. Abbreviations are regularly used in examples I’ve seen on the web.

shellhist_gp_stackedlines.gp:

fmt = "%Y-%m-%dT%H:%M:%S"

binwidth = 3600 # 1h in seconds
bin(t) = (t - (int(t) % binwidth) + binwidth/2)

set xdata time
set datafile missing NaN

set xtics format "%b %d %H:%M" time rotate nomirror
set ytics nomirror
set border 1+2 # Set the bits for bottom and left borders.
set title 'Commands Run Per Hour'


top_cmds = system("awk '$3 ~ /^[a-zA-Z0-9_-]+$/{print $3}' shellhist \
    | sort | uniq -c | sort -rn | awk '{print $2}' | sed 10q")

plot for [cmd in top_cmds] 'shellhist' \
    u (bin(timecolumn(2, fmt))):((valid(1) && strcol(3) eq cmd) ? 1 : NaN) \
        smooth freq with lines title cmd, \
    '' using (bin(timecolumn(2, fmt))):(1.0) \
        smooth freq with lines title "sum" dashtype '..'

Multiple distributions with stacked bars

Plotting distributions using lines is convenient when making the chart, but if many distributions are involved the chart gets uncomfortably busy. Plotting the chart with stacked bars can make it easier to read.

Inline transformations

Here we’re using a similar approach as for the previous chart, using gnuplot expressions to transform the data so it’s suitable for use with smooth frequency.

The main complication is that we need to stack the bars ourselves, plotting sums of comamnd frequencies in descending order so taller bars don’t completely obscure the short ones. To acheive that we plot the sum of a decreasing number of commands using a sum⁵ expression. Each box will obscure just the bottom of the previous one, with the unobscured portion—that is, the difference between the sums—representing the frequency of the command.

There are some minor changes, too, like the formatting options for the boxes, and setting the xtics to be out so they aren’t obscured by the boxes. Also, since the series with the smallest overall frequencies are plotted first so they’re at the top of the stack, I’ve explicitly chosen which linetypes to use with linetype cmd (cmd being used as a linetype index) so the nicer colours at the start of the sequence are used for the biggest boxes.

shellhist_gp_stackedbars.gp:

binwidth = 3600 # 1h in seconds
bin(t) = (t - (int(t) % binwidth) + binwidth/2)

set xdata time
set datafile missing NaN
set boxwidth binwidth
set xtics format "%b %d %H:%M" time rotate out nomirror
set ytics nomirror
set style fill solid border lc rgb "black"
set border 3
set title "Commands run per hour"

top_cmds = system("awk '$3 ~ /^[a-zA-Z0-9_-]+$/{print $3}' shellhist \
    | sort | uniq -c | sort -rn | awk '{print $2}' | sed 10q")

time = '(bin(timecolumn(2, "%Y-%m-%dT%H:%M:%S")))'

plot for [cmd=words(top_cmds):1:-1] 'shellhist' \
    u @time:(!valid(1) ? NaN : sum [i=1:cmd] strcol(3) eq word(top_cmds, i)) \
    smooth freq w boxes t word(top_cmds, cmd) linetype cmd, \
    '' u @time:(valid(1)) smooth freq w lines t 'sum' dashtype '..'

External reshaping

An alternative method is to reshape the data using some other tool so gnuplot’s histogram plot style can be used. For this data this approach is a lot more work than using inline transformations, but there are certainly situations where it’s appropriate.

We’re taking the data from this shape:

time        cmd
<timestamp> sed
<timestamp> awk
...

to this:

time  sed awk
<bin>   0   5
<bin>   5   2
...

I was almost able to reshape the data with mlr but ended up using perl. It’s mostly a matter of counting frequencies per command for each bin, sorting the commands by frequency using a Schwartzian transform so it’s easy to print the top N commands, and then printing out all the bins. See the addendum about pandas for a far more concise way to do this.

bin.pl:

#!/usr/bin/perl

use strict;
use warnings FATAL => 'all';

use Time::Piece;
use List::Util qw(sum0 min max);

my $bin_width = 3600;

my %bins;
my %cmd_seen;
for my $line (<>) {
    my ($index, $time_str, $cmd) = split ' ', $line;
    next if $cmd !~ /^[a-zA-Z0-9_-]+$/;
    my $time = Time::Piece->strptime($time_str, "%Y-%m-%dT%H:%M:%S");
    my $interval = $time->epoch() - $time->epoch() % $bin_width;
    $bins{$interval}{$cmd}++;
    $cmd_seen{$cmd}++;
}

my %total_for;
for my $bin (values %bins) {
    for my $cmd (keys %{$bin}) {
        $total_for{$cmd} += $bin->{$cmd};
    }
}
my @cmds = map {$_->[0]}
           sort {$b->[1] cmp $a->[1]} # hi to lo
           map {[$_, $total_for{$_}]}
           keys %total_for;

my $ofs = ' ';
print join($ofs, 'time', 'sum', @cmds), "\n";

my ($start, $end) = (min(keys %bins), max(keys %bins));
for (my $interval = $start; $interval <= $end; $interval += $bin_width) {
    my @row;
    push @row, gmtime($interval)->strftime("%Y-%m-%dT%H:%M:%S");
    push @row, sum0(values %{$bins{$interval}});
    for my $cmd (@cmds) {
        push @row, $bins{$interval}{$cmd} // 0;
    }
    print join($ofs, @row), "\n";
}

Producing data like:

$ perl bin.pl shellhist | tee shellhist_binned
time sum gnuplot fg qiv man go less perl vim echo dicedist cat history find printf ls sort wc meh cd nl tac mv rm sed ll uniq pf wget mkdir help reset seq for
2017-06-27T20:00:00 10 0 0 0 5 0 0 0 0 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2017-06-27T21:00:00 11 0 0 0 1 0 0 3 2 1 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2017-06-27T22:00:00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2017-06-27T23:00:00 2 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2017-06-28T00:00:00 10 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 1 0 0 0 0
...

Now the histogram plot type can be used. First off, we’ll set style histogram rowstacked to get stacked bars instead of the default clustered bars, which don’t work well with a large number of bins like we have here.

The x-axis labels require a different approach from before. Lines of histogram data are implicitly plotted at x = 1, 2, 3, etc., so set xdata time isn’t applicable and the data needs to contain all the bins we want plotted, including empty ones. If the times in the datafile aren’t in the presentation format desired, the x-axis tick labels can be formatted by giving an xticlabels() specification in the using clause and parsing and reformatting the dates using the strptime and strftime functions. Since the x-axis range doesn’t represent times we need to use xticlabels() to get useful labels from the data itself. Unfortunately, this means there won’t be automatically-spaced major ticks, which are required for the automatic placement of minor ticks, and placing minor ticks manually is tricky. They can be placed using:

set for x[<start>:<end>:<incr>] xtics add ('' x 1)

but if an increment greater than 1 is used care must be taken to set the start so the minor ticks are aligned with the major ones, unless the data happens to start at a bin that’s a multiple of the increment. For example, if we’re using hour-long bins, major ticks are at midnight and noon, the first line of data (x=0) is for 8 o’clock, and we want a minor tick every 3 hours, we need to start the minor ticks at x=1—9 o’clock—instead, not at x=0, and calculating this programmatically is awkward. Alternatively, we can fake minor ticks by placing major ticks without labels, giving the empty string to xticlabels() at appropriate positions.

The second plot clause prints the overall command frequency stored in the sum column and takes care of the xticlabels. Since the rest of the columns are command frequencies sorted in decreasing order we can easily plot the top N commands by changing the iteration conditions of the first clause.

shellhist_gp_histogram.gp:

# Autotitle series in the key using columnheaders from the data file.
set key autotitle columnheader

set style histogram rowstacked
set style fill solid border lc rgb "black"

set xtics nomirror rotate out

set title "Commands run per hour"

xlabel(time) = ( \
    t = strptime("%Y-%m-%dT%H:%M:%S", time), \
    int(tm_hour(t)) % 12 == 0  ? strftime("%b %d %H:00", t) \
    : int(tm_hour(t)) % 3 == 0 ? '' \
    :                            NaN \
)

topN = 10
plot for [i=3:(3+topN-1)] 'shellhist_binned' using i with histogram, \
    '' u "sum":xtic(xlabel(strcol(1))) w lines dashtype '..'

matplotlib

Dates, times, and timezones

matplotlib includes some convenient date-handling code, particularly for locating and formatting axis ticks, but if we want to take advantage of this we need to give matplotlib our dates in its own format, a float representing the number of days since 0001-01-01 in UTC. Because the format is in UTC and timezone-naïve datetime.datetime assume the local timezone, any datetime.datetime objects converted using matplotlib.dates.date2num must be either timezone-aware or in UTC themselves.

The standard datetime module provides the tzinfo interface to add timezone information to datetime.datetime instances, but only provides a concrete implementation for simple timezone offsets that don’t take DST or other changes into account. Also, datetime.datetime.strptime can only parse timezone offsets formatted as [-+]HHMM. For other cases additional libraries will likely be needed. In particluar, if timezone abbreviations are being parsed other libraries like dateutil, a dependency of matplotlib, can be used to help disambiguate them, seeing as there are many collisions.

Also, unless we want to display UTC times in our charts we need to give a timezone to matplotlib to use for formatting. We can change matplotlib’s runtime configuration setting by giving an Olson timezone name in our matplotlibrc file, or we can override the rc value in code by using matplotlib.rcParams or by passing an object that implements the datetime.tzinfo interface to all our Locator and Formatter objects.⁶ I’d rather not override anyone’s settings, so I’ve just set timezone: America/Edmonton in my ~/.config/matplotlib/matplotlibrc.

Points

Let’s start with the same type of chart as with gnuplot, a point for each command executed, randomly-scattered from [0,1] across the y-axis. This gives a rough idea of when commands were executed. Getting the data out of the shellhist file is straight-forward but verbose compared to gnuplot, and numpy is convenient for generating a large array of random y-values. Note I’ve been careful to convert the datetime.datetime objects resulting from parsing times to Unix time in UTC before converting them to matplotlib’s time representation.⁷

shellhist_mpl_points.py:

from datetime import datetime
import matplotlib.pyplot as plt
from matplotlib.dates import epoch2num
import numpy as np

times = []
for line in open('shellhist'):
    fields  = line.split()
    if len(fields) < 3:
        continue
    _, time_str, cmd = fields[:3]
    unixtime = datetime.strptime(time_str, "%Y-%m-%dT%H:%M:%S").timestamp()
    times.append(epoch2num(unixtime))

fig, ax = plt.subplots()
fig.autofmt_xdate() # Rotate and right-align xtic labels.

ax.plot_date(times, np.random.rand(len(times)), '+')

plt.savefig('shellhist_mpl_points.svg')

Binning times

matplotlib doesn’t have anything like gnuplot’s smooth frequency filter (AFAIK), making it less convenient to do a quick-and-dirty line chart. It seems necessary to preprocess the data before plotting it. I looked at using numpy’s histogram function or matplotlib.axes.Axes.hist but it seems easier to count the frequency per bin using a defaultdict.

For bin widths that evenly divide a day, timestamps are most easily binned when represented as Unix time since every day is defined to have exactly 86400 seconds according to POSIX regardless of leap seconds and it’s easy to use the modulus operator, %, on an integer to find the start of some number of seconds, minutes, or hours. Wheras rounding datetime.datetime instances seems to require cascading through the different units, and rounding matplotlib’s days-since-0001-01-01 representation involves dealing with floats, which are tricky to compare or round reliably and thus a bad choice for dictionary keys.

shellhist_mpl_binned_overall.py:

from collections import defaultdict
from datetime import datetime, timedelta
import matplotlib.pyplot as plt
from matplotlib.dates import epoch2num

bin_width = timedelta(hours=1)

freq_for = defaultdict(lambda: 0)
for line in open('shellhist'):
    fields  = line.split()
    if len(fields) < 3:
        continue
    _, time_str, cmd = fields[:3]
    unixtime = datetime.strptime(time_str, "%Y-%m-%dT%H:%M:%S").timestamp()
    bin_start = unixtime - unixtime % bin_width.seconds
    freq_for[bin_start] += 1


unixtimes = list(sorted(freq_for.keys()))
times = [epoch2num(t) for t in unixtimes]
freqs = [freq_for[bin] for bin in unixtimes]

fig, ax = plt.subplots()
fig.autofmt_xdate()

ax.plot_date(times, freqs, linestyle='-')

plt.savefig('shellhist_mpl_overall_line.svg')

Improve readability

Assuming we liked how the gnuplot single-distribution line chart was styled, we can create a similar chart in matplotlib.

First, use DateFormatter and HourLocator objects instead of Figure.autofmt_xdate to choose xtick placement and labels:

format the xtick labels as “Jan DD HHHH” and only label midnight and noon
add minor xticks every 3 hours

We also:

hide the top and right “spines”, as matplotlib calls the Axes borders
rotate the xticklabels to vertical. It’s harder to read the label itself but clearer which point on the axis the label refers to.
add a title

shellhist_mpl_binned_overall_nicer.py:

from collections import defaultdict
from datetime import datetime, timedelta
import matplotlib.pyplot as plt
from matplotlib.dates import epoch2num, DateFormatter, HourLocator

bin_width = timedelta(hours=1)

freq_for = defaultdict(lambda: 0)
for line in open('shellhist'):
    fields  = line.split()
    if len(fields) < 3:
        continue
    _, time_str, cmd = fields[:3]
    unixtime = datetime.strptime(time_str, "%Y-%m-%dT%H:%M:%S").timestamp()
    bin_start = unixtime - unixtime % bin_width.seconds
    freq_for[bin_start] += 1


unixtimes = list(sorted(freq_for.keys()))
times = [epoch2num(t) for t in unixtimes]
freqs = [freq_for[bin] for bin in unixtimes]

fig, ax = plt.subplots()

plt.xticks(rotation='vertical')
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.xaxis.set_major_formatter(DateFormatter("%b %d %H00"))
ax.xaxis.set_major_locator(HourLocator(byhour=range(0, 24, 12)))
ax.xaxis.set_minor_locator(HourLocator(byhour=range(0, 24, 3)))
ax.set_title("Commands Run Per Hour")

ax.plot_date(times, freqs, linestyle='-')

plt.savefig('shellhist_mpl_overall_line_nicer.svg')

Multiple distributions

In matplotlib we can work from the same binned data whether we’re plotting lines or bars so I’m generating both from the same script here. The script is pretty long so I’ve organized it into functions.

Binning is mostly the same as before, except we’re filtering out commands that don’t look like words using a regular expression and using nested defaultdicts to also keep track of per-command frequency.

The styling code in save_chart is almost the same as before, except Axes.legend is called between plotting all the series and saving the figure, since it generates a legend based on the series that have already been plotted.

I’ve set the line/bar colours to be tab10 using Axes.set_prop_cycle in save_chart just to show one way to change to them. This example show some other ways. tab10 is the default colour cycle, so it’s not actually doing anything.

Plotting lines is straightforward, but plotting stacked bars requires a few extra steps. As with gnuplot we’ve offset the times by half a bin_width and set the bar width so the bars cover the time interval on the x-axis they represent instead of being centered over it. Also, we need to keep track of the where to plot the bottom of the bars and add the frequencies of each series to it; a numpy array is convenient and efficient for this since operations on an array apply to all of its elements.

shellhist_mpl_multi.py:

from collections import defaultdict, namedtuple
from cycler import cycler
from datetime import datetime, timedelta
from matplotlib.dates import epoch2num, DateFormatter, HourLocator
import matplotlib.pyplot as plt
import numpy as np
import re

# cmds: list of command names in decreasing order
# unix_times: bin start times
# freqs_for: dict of command names to numpy arrays of bin frequencies
# totals: numpy array of overall command frequencies for each bin
# bin_width: bin width as datetime.timedelta
CmdFreq = namedtuple('CmdFreq',
    ['cmds', 'unix_times', 'freqs_for', 'totals', 'bin_width'])

plt.rcParams['image.cmap'] = 'tab10'

def save_chart(plot_func, data, filename):
    fig, ax = plt.subplots()

    plt.xticks(rotation='vertical')
    ax.spines['right'].set_visible(False)
    ax.spines['top'].set_visible(False)
    ax.xaxis.set_major_formatter(DateFormatter("%b %d %H00"))
    ax.xaxis.set_major_locator(HourLocator(byhour=range(0, 24, 12)))
    ax.xaxis.set_minor_locator(HourLocator(byhour=range(0, 24, 3)))
    ax.set_prop_cycle(cycler('color', plt.get_cmap('tab10').colors))
    ax.set_title("Commands Run Per Hour")

    plot_func(ax, data)

    ax.legend() # Called after all data has been plotted.

    plt.savefig(filename)

def plot_bars(axes, data):
    bar_offset = data.bin_width.seconds/2
    offset_times = [epoch2num(t + bar_offset) for t in data.unix_times]
    bar_width = data.bin_width.seconds / 86400.0 # Fraction of a day.

    axes.plot(offset_times, data.totals,
        linestyle='--', linewidth=0.4, label='sum')

    bottom = np.zeros(len(offset_times))
    for cmd in data.cmds[:10]:
        axes.bar(offset_times, data.freqs_for[cmd],
            bottom=bottom, width=bar_width, label=cmd)
        bottom += data.freqs_for[cmd]

def plot_lines(axes, data):
    float_times = [epoch2num(t) for t in data.unix_times]
    axes.plot(float_times, data.totals,
        linestyle='--', linewidth=0.4, label='sum')

    for cmd in data.cmds[:10]:
        axes.plot(float_times, data.freqs_for[cmd], label=cmd)


def get_cmd_data(bin_width):
    with open('shellhist') as shellhist:
        bin_for, bin_total_for = bin_cmds(shellhist, bin_width)

    cmds = cmds_in_decreasing_freq(bin_for)

    unix_times = np.array(sorted(bin_for.keys()), int)

    freqs_for = {}
    for cmd in cmds:
        freqs_for[cmd] = np.array([bin_for[t][cmd] for t in unix_times])

    totals = np.array([bin_total_for[t] for t in unix_times])

    return CmdFreq(cmds, unix_times, freqs_for, totals, bin_width)

def bin_cmds(event_lines, bin_width):
    bin_for = defaultdict(lambda: defaultdict(lambda: 0))
    bin_total_for = defaultdict(lambda: 0)
    for line in event_lines:
        fields  = line.split()
        if len(fields) < 3:
            continue
        _, time_str, cmd = fields[:3]

        if not re.search(r'^[a-zA-Z0-9_-]+$', cmd):
            continue

        date = datetime.strptime(time_str, "%Y-%m-%dT%H:%M:%S")
        unixtime = int(date.timestamp())
        bin_start = unixtime - unixtime % bin_width.seconds

        bin_for[bin_start][cmd] += 1
        bin_total_for[bin_start] += 1

    return bin_for, bin_total_for

def cmds_in_decreasing_freq(bin_for):
    freq_for = defaultdict(lambda: 0)
    for bin_ in bin_for.values():
        for cmd, freq in bin_.items():
            freq_for[cmd] += freq
    pairs = sorted(freq_for.items(), key=lambda pair: pair[1])
    return list(reversed([p[0] for p in pairs]))

if __name__ == '__main__':
    data = get_cmd_data(timedelta(hours=1))
    save_chart(plot_bars, data, 'shellhist_mpl_stacked_bars.svg')
    save_chart(plot_lines, data, 'shellhist_mpl_stacked_lines.svg')

Best resources for learning

For gnuplot I’d definitely recommend looking at Gnuplot in Action, which is an amazing tutorial that teaches all the basics and a bunch of practical tricks. If you don’t want to buy a book I’d recommend reading the manual, skimming less relevant sections but paying particular attention to most of part 1 and the description of the plot command in part 3. Once these fundamental topics are understood I think the rest can be looked up as needed. At first glance the manual doesn’t look like it’s organized for a linear read but I was well-served by that approach. The help command seems to provide all the content in the manual but conveniently indexed by topic and command, and I wish that I realized how good it was earlier, as I’m finding it far more convenient than referring to the manual. Looking through the collection of demos may also be helpful.

I learned what I know about matplotlib by stumbling through the official docs, which was painful and inefficient. They’re clearly written and comprehensive but it was difficult to find what I needed to do practical things. I suspect having such comprehensive API docs creates a Worse is Better situation making the creation of tutorial-style documentation seem like a relatively low-value activity since all the needed information is already out there somewhere and duplicating it in a tutorial increases the maintenance burden. I’d recommend starting with Nicolas Rougier’s tutorial over the ones on matplotlib.org.

Conclusion

I’m impressed with both tools; they both render beautiful charts. As I mentioned in the intro I’d lean towards gnuplot for simpler, quicker tasks since it so convenient and concise, and towards matplotlib for fancier stuff, since it seems like everything imaginable is customizable if you pound your head against the docs for a bit. When each chart is defined in a separate script gnuplot is kicky fast compared to matplotlib, I think largely due to the time it takes to import all the matplotlib libraries; it probably doesn’t matter but it still bolsters my warm feelings of convenience for gnuplot.

Addendum: pandas

pandas is a data analysis toolkit for Python and it has some particularly convienient functions for pivoting and rolling up the data. Assuming the times and commands have already been extracted from the datafile as equal-length lists of strings named times and cmds, the data can be binned and pivoted into a wide format like this:

import pandas as pd

df = pd.DataFrame({'cmd': cmds, 'freq': 1}, index=pd.DatetimeIndex(times))
piv = df.pivot(columns='cmd', values='freq')
binned = piv.resample('{}S'.format(seconds)).sum().fillna(0)

pandas provides a wrapper for matplotlib that works very well for simple cases but isn’t flexible enough to make a chart similar to the stacked-bar histograms above. Here’s a script where I use matplotlib directly instead: shellhist_pandas.py.

Addendum: changelog

2017-08-15: Added an addendum discussing the use of pandas.

2017-10-09:

Added a nice way to make a stacked bar chart for multiple distributions using gnuplot, after discovering the sum [min:max:step] <expr> syntax.
Added help commands to be entered at the gnuplot prompt for easier access to relevant documentation.
Added the excellent Gnuplot in Action as an important resource for learning gnuplot.
Removed discussion of colour in gnuplot as it’s mostly irrelevant to the task at hand and there’s several good ways to change colours that aren’t difficult to find. See help colors.
Made the “sum” line more distinct by making it dotted.

2017-10-11: Previously I was using the term “time series” loosely to refer to timestamped data in general, but it usually refers to data that are equally spaced in time. “Event stream” is more applicable to the data used here.

A description of the using clause of the plot command can be found by entering help using at the gnuplot prompts or in the “Commands > Plot > Data > Using” section of gnuplot’s manual. The manual is an excellent reference but isn’t obviously a tutorial, and before reading most of it I had trouble finding the information I needed. If it was published in HTML I’d link directly to relevant sections, but it’s only published in PDF so I’ve provided section breadcrumbs and help command keywords. ↩︎
help smooth freq and the “Commands > Plot > Data > Smooth > Frequency” section of the gnuplot manual have an example of plotting a histogram with the lines plotting style. ↩︎
Note that one limitation of rounding the epoch seconds time representation like this is that leap seconds aren’t taken into account. If leap second accuracy is required for your application a different approach is needed. ↩︎
See the “Gnuplot > Time/Date Data” section of the manual or help time/date. ↩︎
See the “Gnuplot > Expressions > Summation” section of the manual or help sum. ↩︎
dateutil.tz.tzlocal returns a datetime.tzinfo-conforming object for the local time zone. ↩︎
Alternatively, it’s straightforward to stick with Unix times and use matplotlib.dates.FuncFormatter and MultipleLocator instead, but I’m trying to go with the grain of matplotlib. ↩︎

Absorb changes across a topic branch in git

Sun, 04 Jun 2017 00:00:00 +0000

Motivation: tedious fixups

Here’s the situation. I’m working on a feature and end up with a chain of commits that depend on each other. It’d be easier for me as the author to put all the work in a single commit, but I like telling stories with my commits to (hopefully) make reviewing the branch easier and the history more meaningful in general. Anyway, the review goes back and forth for a while, motivating a bunch of small edits that are logically most connected to various commits. I commit these edits separately and then squash/fixup them all with an interactive rebase. Now, assigning changes to commits representing logical groups of changes was fun/challenging the first time, but assigning these small edits born of review feedback is nearly a mechanical process of scanning through the list of topic branch commits and copy-pasting SHAs. [Or, more conveniently, using git commit --fixup=:/<regex>, although I haven’t thought to do that until recently.]

Personal and team git workflows apparently vary wildly, so not everyone has dealt with this situation, but I’ve encountered it with enough regularity that when I read the description of Facebook’s hg absorb command in these Mercurial sprint notes I was super envious. To cure my envy I wrote git-autofixup.

How it works

git-autofixup parses hunks of changes in the working directory out of git diff output and uses git blame to assign those hunks to commits in <revision>..HEAD, which will typically represent a topic branch, and then creates fixup commits to be used with git rebase --interactive --autosquash.

By default a hunk will be included in a fixup commit if the hunk’s context shows there’s an unambiguous target topic branch commit. There are two situations where a target commit is considered unambiguous:

When it’s the only topic branch commit the hunk is near. More precisely, when it’s the only topic branch commit appearing in the blame output of all the hunk’s context lines.
It’s blamed for all the lines that the hunk changed, even if changes from other topic branch commits are nearby. More precisely, it’s blamed for all the removed lines and at least one of the context lines adjacent to added lines, and no context lines adjacent to added lines are blamed on any other topic branch commits.

Slightly stricter assignment criteria are also available for when you’re untangling fixups from changes for a new commit: see the description of the --strict option in the --help.

Example

git-autofixup is most useful on big projects, in big teams, on long-lived topic branches, but I’ve tried to concoct a small example that motivates its use. Say we have a little python library that for whatever reason transforms a given name so the letters of the last word of the name alternate between upper and lower case:

def last_name_alternating_case(name):
    """Return name, but with the last word in aLtErNaTiNg case."""
    words = name.split()
    letters = list(words[-1])
    for (i, char) in enumerate(letters):
        if i % 2 == 0:
            letters[i] = char.lower()
        else:
            letters[i] = char.upper()
    words[-1] = ''.join(letters)
    return ' '.join(words)

A new function is required that alternates the case of the letters in every other word of a given string, so we start a topic branch and have it track master. Before we start writing odd_words_alternating_case, though, we realize some of the logic needed can be factored out of last_name_alternating_case. So we do that and make a commit with the summary Factor out alternating_case function. The file is now:

def last_name_alternating_case(name):
    """Return name, but with the last word in aLtErNaTiNg case."""
    words = name.split()
    words[-1] = alternating_case(words[-1])
    return ' '.join(words)

def alternating_case(s):
    letters = list(s)
    for (i, char) in enumerate(letters):
        if i % 2 == 0:
            letters[i] = char.upper()
        else:
            letters[i] = char.lower()
    return ''.join(letters)

Now we’re ready to write odd_words_alternating_case using alternating_case, and commit it as Add odd_words_alternating_case:

def odd_words_alternating_case(s):
    """Returns the string with alternating words in alternating case."""
    words = s.split()
    for (i, word) in enumerate(words):
        if i % 2 == 0:
            continue
        words[i] = alternating_case(word)
    return ' '.join(words)

Looking the code over, we realize we want to give alternating_case a docstring, change the tense/mood of the odd_words_alternating_case docstring so it’s consistent with last_name_alternating_case, and we decide the first character of alternating-case words should be uppercase. Here’s the diff of these unstaged changes:

diff --git a/ex.py b/ex.py
index 2ed7e63..2a5e73b 100644
--- a/ex.py
+++ b/ex.py
@@ -7,2 +7,3 @@ def last_name_alternating_case(name):
 def alternating_case(s):
+    """Return s with its characters in aLtErNaTiNg case."""
     letters = list(s)
@@ -10,5 +11,5 @@ def alternating_case(s):
         if i % 2 == 0:
-            letters[i] = char.upper()
-        else:
             letters[i] = char.lower()
+        else:
+            letters[i] = char.upper()
     return ''.join(letters)
@@ -16,3 +17,3 @@ def alternating_case(s):
 def odd_words_alternating_case(s):
-    """Returns the string with alternating words in alternating case."""
+    """Return s with odd words in alternating case."""
     words = s.split()

We’d like to squash these changes into the previous two commits. In this particular instance it’d be quite easy to do with two rounds of git add --patch followed by git commit --fixup=:/<regex>. but if the topic branch had more commits and we were fixing up more areas this process would get tedious. Let’s see what git-autofixup does with it:

$ git-autofixup -vv @{upstream}
ex.py @@ -5,16 +5,17 @@ has multiple targets
656a790f|   5|    return ' '.join(words)    |     return ' '.join(words)
656a790f|   6|                              |
656a790f|   7|def alternating_case(s):      | def alternating_case(s):
        |    |                              |+    """Return s with its char
656a790f|   8|    letters = list(s)         |     letters = list(s)
^       |   9|    for (i, char) in enumerate|     for (i, char) in enumerat
^       |  10|        if i % 2 == 0:        |         if i % 2 == 0:
^       |  11|            letters[i] = char.|-            letters[i] = char
656a790f|  12|        else:                 |-        else:
656a790f|  13|            letters[i] = char.|             letters[i] = char
        |    |                              |+        else:
        |    |                              |+            letters[i] = char
656a790f|  14|    return ''.join(letters)   |     return ''.join(letters)
5be3a3b9|  15|                              |
5be3a3b9|  16|def odd_words_alternating_case| def odd_words_alternating_cas
5be3a3b9|  17|    """Returns the string with|-    """Returns the string wit
        |    |                              |+    """Return s with odd word
5be3a3b9|  18|    words = s.split()         |     words = s.split()
5be3a3b9|  19|    for (i, word) in enumerate|     for (i, word) in enumerat
5be3a3b9|  20|        if i % 2 == 0:        |         if i % 2 == 0:

We’re using high verbosity (-vv) so that the “blamediff” gets printed and we can see how the hunks are being handled. Our changes are close enough together that they all get put into the same hunk with the default number of diff context lines (3), and then that single hunk is related to both of our topic branch commits. When we reduce the number of context lines to get more hunks the unstaged changes are isolated enough to be assigned to their respective commits and two fixup! commits are created:

$ git-autofixup --context=1 -vv @{upstream}
ex.py @@ -7,2 +7,3 @@ fixes 656a790f Factor out alternating_case function
656a790f|   7|def alternating_case(s):      | def alternating_case(s):
        |    |                              |+    """Return s with its char
656a790f|   8|    letters = list(s)         |     letters = list(s)

ex.py @@ -10,5 +11,5 @@ fixes 656a790f Factor out alternating_case function
^       |  10|        if i % 2 == 0:        |         if i % 2 == 0:
^       |  11|            letters[i] = char.|-            letters[i] = char
656a790f|  12|        else:                 |-        else:
656a790f|  13|            letters[i] = char.|             letters[i] = char
        |    |                              |+        else:
        |    |                              |+            letters[i] = char
656a790f|  14|    return ''.join(letters)   |     return ''.join(letters)

ex.py @@ -16,3 +17,3 @@ fixes 5be3a3b9 Add odd_word_alternating_case function
5be3a3b9|  16|def odd_words_alternating_case| def odd_words_alternating_cas
5be3a3b9|  17|    """Returns the string with|-    """Returns the string wit
        |    |                              |+    """Return s with odd word
5be3a3b9|  18|    words = s.split()         |     words = s.split()

[topic 44cadf7] fixup! Add odd_word_alternating_case function
 1 file changed, 1 insertion(+), 1 deletion(-)
[topic bde4ca5] fixup! Factor out alternating_case function
 1 file changed, 3 insertions(+), 2 deletions(-)

Finally, we do a git rebase --interactive --autosquash and see git has set the fixup commits to be squashed into their targets:

pick 656a790 Factor out alternating_case function
fixup cfa466e fixup! Factor out alternating_case function
pick 5be3a3b Add odd_word_alternating_case function
fixup 1a6c084 fixup! Add odd_word_alternating_case function

Where to get it

Check it out on GitHub or the CPAN. It can be installed using a CPAN client or by simply downloading the self-contained script, git-autofixup, to a directory in PATH.

Finding find: grasping its simple expression language

Wed, 08 Mar 2017 00:00:00 +0000

Confusion

I remember the first time I tried to use the find command on Linux, over a decade ago. I knew a substring of the name of a file I wanted to find, had found something on the web suggesting I use find, and was so disappointed when I couldn’t just run find $SUBSTR to get my desired result. “Surely such a command exists,” I thought. And it basically does, with locate, but I was searching directories that hadn’t been indexed by updatedb and didn’t know enough about globbing and quoting in bash to effectively give patterns to it. So I did some more searching on the web, learned I needed to give a -name predicate to find, and typed a lot of commands like the following over the next few years:

find . -name "*$SUBSTR*"

[Here I’m using “predicate” to mean a term of an expression. It’s the word the GNU manpage uses.]

Much later, after reading the bash manpage a few times, among other things, I wrote a shell function to make this specific task a bit shorter to type, giving me the command I had wished for years before:

# find files under current directory containing a pattern
function ff {
    pattern=${1:?No pattern given}; shift
    find . -iname "*${pattern}*" "$@" 2>/dev/null
}

Beyond names

Eventually I needed to find files based on their modification times and I started using the -mtime predicate. For example, to find files modified within the past 24 hours (-mtime takes units of 24 hours):

find . -mtime -1

When using a condition like this that can match large numbers of uninteresting files it becomes obvious that further filtering is necessary. egrep -v can be used for this, but to avoid needlessly traversing directories and wasting time, using the -prune predicate for find is desirable. It was difficult to use properly without understanding find’s little expression language, though.

Essence of expressions

The DESCRIPTION in the manpage for GNU find makes sense now but it meant nothing to me in 2005. “…evaluating the given expression from left to right, according to the rules of precedence, until the outcome is known, at which point find moves on to the next file name.” All it means, though, is that you give find a boolean expression composed of predicates and operators, and each file will get tested against it. Each predicate evaluates to true or false, and they are combined with the AND (-a) and OR (-o) operators. I don’t remember how I came to understand this, but the best explanation I’ve seen is in chapter 9 of Unix Power Tools.

A few pages down in the manpage, in the EXPRESSION section, we see there are several classes of predicates: tests, actions, options, and operators. -mtime and -name are examples of tests, -print is the default action, -maxdepth is an example of an option, and -a (AND) is the default operator. It might seem weird to include actions and options in a boolean expression of tests, and it is, but it works well. They can be shoehorned in with the tests because they also return boolean values: most actions always return true, and options always return true. -print, the most commonly used action, always returns true.

After grasping this key point, that find is just evaluating a boolean expression, it’s easy to write elaborate commands. Starting with something simple, though:

$ ls
bar  foo
$ find . -name foo -a -print
./foo

# Using defaults to save typing:
$ find . -name foo
./foo

This expression and its evaluation can be represented in a C-inspired syntax:

Testing "bar":
name == "foo" && print
false && print
false
continue to the next file

Testing "foo":
name == "foo" && print
true && print
print
true
continue to the next file

find reads the current directory, gets bar, tests it against -name foo, which evaluates to false, and short-circuits on the -a operator, continuing on to foo, which tests true with -name foo and so gets printed by -print.

Prune

Back to -prune. We’re still searching for foo, but let’s say there’s also a directory we don’t want to descend into called cache. There’s a file called foo in it so it’s easier to tell if it’s getting searched.

# List files (-type f) under the working directory.
$ find . -type f
./cache/foo
./bar
./foo

$ find . -name cache -prune -o -name foo -print
./foo

Here’s the expression in C-inspired syntax:

Testing "cache":
(name == "cache" && prune) || (name == "foo" && print)
(true && prune) || (name == "foo" && print)
prune || (name == "foo" && print)
true || (name == "foo" && print)
true
continue to the next file

Testing "foo":
(name == "cache" && prune) || (name == "foo" && print)
(false && prune) || (name == "foo" && print)
name == "foo" && print
true && print
print
true
continue to the next file

Let’s run through this. When find is testing cache, -name cache returns true, so -prune gets run, which removes cache from the list of directories to descend into and returns true. The return value of the whole expression is then known because the left side of the OR (-o) is true, so find moves onto the next file. When testing foo, -name cache returns false, failing the left side of the OR, so find moves to the right side where -name foo returns true, resulting in ./foo being printed.

Default action

If an expression doesn’t print or execute anything find treats it as if it were surrounded in parentheses and followed by a print action: ( EXPR ) -print. For example, if we remove -print from the previous command:

$ find . -name cache -prune -o -name foo
./cache
./foo

cache gets printed because -prune returns true, making the overall expression true for it.

Parens

As with many expression languages, parentheses can be used to force precedence. They need to be escaped or quoted so the shell doesn’t treat them specially:

$ find . -name cache -prune -o \( -name foo -o -name bar \) -print
./bar
./foo

Troubleshooting

If an expression isn’t behaving as expected -exec or -printf can be used to visualize what’s actually happening. If portability is important, note that -printf is a GNU extension and isn’t specified in POSIX.

$ find . -name cache -printf "pruning %p\n" -prune -o -name foo -print
pruning ./cache
./foo

$ find . -name cache -exec echo pruning {} \; -prune -o -name foo -print
pruning ./cache
./foo

There’s more

Once I understood the basic concept of the expression language the manpage became a good reference. Check it out to discover other useful tests and actions. Some of my favourite POSIX-specified predicates are -perm, -user, and -size, and I frequently use the GNU extensions -maxdepth, -mmin, -regex, and -ls. Happy finding.

Failing to write useful mouse keys for X11

Tue, 07 Mar 2017 00:00:00 +0000

Motivation

I’ve stuck with Apple laptops primarily because I haven’t had a pleasant trackpad experience on anything else. But about a year ago I noticed I was spending all my time in a web browser and tmux and had disabled or was avoiding many of OSX’s features, like the Dock, Dashboard, full-screen apps (due to the lengthy animations), Spotlight (because mdworker indexing processes have a history of going out of control and pinning CPUs), and iTunes. So I thought I might as well switch to Linux and the suckless.org desktop (dwm, dmenu, st), as long as I could get the trackpad working well enough.

Initially I had the synaptics touchpad driver installed, but after several hours of fruitless tweaking I gave up on making a usable configuration. It seemed impossible to get enough sensitivity while also adequately ignoring accidental touches using the given configuration parameters. So I tried the evdev and libinput drivers and thankfully the libinput driver gives very acceptable behaviour, though it’s still a ways off from the magic of Apple’s driver, which has never failed to behave as expected when pointing or scrolling.

Anyway, all this fussing over trackpads made me wonder if controlling the pointer using the keyboard could be a better experience. An efficient enough mouse keys implementation could also be useful for people with trackpad-specific repetitive strain injuries, or in the admittedly rare situations where both a trackpad and mouse are unavailable or infeasible, like if you wanted to control a media PC with only a keyboard.

Existing solution

I tried out the keypad pointer keys in X11 (setxkbmap -option keypad:pointerkeys), but they aren’t fluid when changing direction and it’s a pain to set up pointer keys on keys other than the keypad. Worst of all, accelerating the pointer after a movement key has been pressed down for a while is counter to how I usually point at things: flinging the pointer over to the general area of my target before carefully zeroing in on it. So I started writing my own, calling it ptrkeys.

Inspiration

Having decided acceleration was the wrong way to make mouse keys efficient, I tried out some schemes where the speed could be changed instantly. My first idea was to have “dual-sticks”, two sets of directional keys, one for coarse and the other for fine control, but this was surprisingly difficult to use. It was far more intuitive to use WASD for directional control while using the thumb and pinky or the other hand to change the speed using speed multiplier keys.

dwm served as a model for most of the Xlib interactions I needed to do. Its method of defining X11 key bindings at compile time using C99 struct initializations is surprisingly simple and flexible, and using a similar method has allowed me to try out totally different binding schemes with minimal changes.

The tricky part was understanding exactly how X11 keyboard grabs work. X generally directs keyboard events to the focused window, but a keyboard can be “grabbed” as a whole or on a per-key basis so that its events are sent elsewhere. For example, pressing a key implicitly sets up a single-key grab so the subsequent release event is received by the same window that got the keypress. ptrkeys doesn’t create a window that can be focused, so a single-key grab is necessary to setup a “global hotkey” that can be used to enable ptrkeys by grabbing the entire keyboard and thus “activating” the rest of its configured key bindings.

Downfall

Grabs also ended up being the downfall of the project: any full-featured desktop environment grabs the whole keyboard when opening a menu for a taskbar, menubar, or system tray, so that it can be navigated with the keyboard without changing the currently focused window. If ptrkeys already has the keyboard grabbed, the desktop environment’s attempt to grab it fails and the menu typically doesn’t open. This severely limits the applicability of ptrkeys. And it doesn’t seem like there’s a way around grabbing the whole keyboard either since a single-key grab becomes a whole-keyboard grab while it’s active.

So, while ptrkeys likely isn’t useful for anyone using a desktop environment like GNOME, KDE, or XFCE, it might be for those using tiling window managers. Check it out on github.

File management on Windows with Proto

Mon, 06 Mar 2017 00:00:00 +0000

Dissatisfied with explorer.exe? Find drag-and-drop and manual window placement tedious? Like using the keyboard? Don’t want to choose between the hundreds of Norton Commander clones? You should try Proto.

Proto is a fast, weird, original, keyboard-controlled file manager written by Mieszko Lassota. You get tabs, incremental subsequence filtering everywhere (eg. ‘mp3’ matches things like ‘my panda <3’ and ‘badtimes.mp3’), renaming using regexes, an application launcher, and a nifty way to select groups of files, even if they’re across filters or directories.

Install

Proto doesn’t seem to use the registry so just put it somewhere comfortable and run the executable.

Help

We’re on Windows, so press F1 to get help in the form of key bindings for the current tab. There are different types of tabs, like file manager, program launcher, calculator, etc., and help is context sensitive, bringing up the commands for the current tab. Type some characters to filter the commands. Press Escape to escape help.

Unlike in most of the other tab types which filter by subsequence, in help filtering is done by substring.

Invoke

Once it’s running you can call up the Proto window by pressing Alt-` (that’s a backtick). This key can be changed in $PROTODIR/Settings/Proto.xml.

Tabs

Ctrl-t opens a new file manager tab. Ctrl-shift-t duplicates the current tab. Shift-return opens a directory in a new tab.

Ctrl-w closes a tab.

Ctrl-tab, ctrl-shift-tab, ctrl-<arrow>, and ctrl-<number> let you select tabs. Ctrl-shift-<arrow> reorders a tab.

Navigate

Use the arrow keys to change which item in the tab is highlighted. To descend into a highlighted directory, press enter. Press backspace to go up a directory. Pressing enter will also open a highlighted file with its default program. Press tab to bring up the launcher and open the highlighted item with a different program. If any items are selected they will be opened instead of the highlighted item.

Select

Selected items are prepended by a bullet. If no items are selected the currently highlighted item is used as the selection for commands that take a filepath.

Space selects the currently highlighted item. Ctrl-a selects all the items, and pressing it again when everying is already selected deselects everything. Alt-a toggles between selecting all files and directories. Shift-down selects the current item and highlights the next one. Shift-right behaves similarly, but on columns.

Filtering

You can type a few characters to filter the contents of a tab using subsequence search. Backspace does what you’d expect. Ctrl-Backspace clears the filter. You can’t usually have spaces in your filter because the space key selects the current item in the tab.

File management

Cut, copy, and paste behave differently in Proto than they do in Explorer. Copy (Ctrl-c) puts filepaths in the clipboard, and then the clipboard is used as input for the cut and paste commands. Cut (Ctrl-x) moves the files in the clipboard to the tab’s current location, while paste (Ctrl-v) copies files. So instead of choosing up front whether you’re going to move or copy files like in Explorer, in Proto you first choose which paths you want to work with, and then choose whether to copy or move after navigating to the destination.

Alt-c appends the selection to the clipboard.

Delete deletes the selection.

Ctrl-n creates a new directory. Ctrl-space displays the size of the selection in the lower right.

Rename

Ctrl-r lets you rename the highlighted item. Ctrl-Shift-r lets you rename the selection using regular expressions. Captured groups can be accessed in the replacement using $1, $2, etc.

Launcher

The launcher can be invoked from inside or outside of Proto using Alt-space. To get Proto to index the Start Menu so the launcher is useful, ensure settings.links.indexMenuStart is set to True in $PROTODIR/Settings/AppLauncher.xml.

Bookmarks

Alt-1 opens a tab of bookmarks to your favourite directories and files. Ctrl-b adds a bookmark to the highlighted directory.

Find

Ctrl-f searches recursively for files using wildcards (* and ?) or regular expressions.

The grep (“Regex search IN files”) function doesn’t seem to work, unfortunately. It always crashes for me.

Other shortcuts I use

Ctrl-s: Change sort key.
Ctrl-.: Open the windows context menu.
Ctrl-End: Kill proto job, such as calculating file size. The current job shows up in the lower left corner of the Proto window.
Ctrl-h: edit text file, quickview image (use arrows for folder slideshow)
Ctrl-g: disk space chart

Jordan Torbiak

bash extglob: Tips for using negation/!()

Intro

!() is both non-greedy and can accept the empty string

Use cases

Performance and other options

Appendix: bash source code

Compiling the test harness in glob.c

Using the same colors for the terminal and Vim

Motivation

Choosing a color model

Choosing colors

Configuring the terminal

Configuring Vim

Vim’s cterm-colors

Writing a Vim colorscheme

See also

Appendix: query if bright background is supported

Addendum: changelog

2024-02-12

Normalize volume system-wide with PulseAudio

Motivation

PulseAudio configuration

Manually route a LADSPA sink to Bluetooth headphones

Automatically route a LADSPA sink to Bluetooth headphones

Find udev info

systemd user service

PulseAudio configuration script

udev rule

Good luck

Addendum: change log

Revisiting pruned approaches

Histograms of timestamped data: gnuplot vs matplotlib

Introduction

The data

gnuplot

A rough start: points

Binning times

Discontinuities

Improve readability

Multiple distributions with lines

Multiple distributions with stacked bars

Inline transformations

External reshaping

matplotlib

Dates, times, and timezones

Points

Binning times

Improve readability

Multiple distributions

Best resources for learning

Conclusion

Addendum: pandas

Addendum: changelog

Absorb changes across a topic branch in git

Motivation: tedious fixups

How it works

Example

Where to get it

Finding find: grasping its simple expression language

Confusion

Beyond names

Essence of expressions

Prune

Default action

Parens

Troubleshooting

There’s more

Failing to write useful mouse keys for X11

Motivation

Existing solution

Inspiration

Downfall

File management on Windows with Proto

Install

Help

Invoke

Tabs

Navigate

Select

Compiling the test harness in `glob.c`