Use with git-filter config - filter.*.clean and filter.*.smudge

[jinnko]

I've been able to integrate sops with git such that files are decrypted/encrypted on checkout/commit. This was achieved like this:

1. Set up git-filter config

   git config --local filter.sops-json.clean "sops --input-type json --output-type json --encrypt /dev/stdin"
   git config --local filter.sops-json.smudge "sops --input-type json --output-type json --decrypt /dev/stdin"
   git config --local filter.sops-json.required true
   

2. Set up .gitattributes to pass files through the filter

   *.json filter=sops-json diff=sops-json
   

3. Have a .sops.yaml configuration with default creation_rules:

   creation_rules:
     - kms: arn:aws:kms:...:...:key/2305235902
   

Checkout and commit work well. Unfortunately the files are always considered changed, I believe because the IV is new on every pass.

Is it necessary for the IV to be ephemeral? Is there a way the random IV could be avoided so this workflow is viable - i.e. so the file isn't always marked as modified by git?

[mtoohey31]

This is the best I could come up with:

#!/usr/bin/env -S bash -euo pipefail

# we need $1 to be the path of the file so we can check the previous version
# via git-show to prevent the encryption's non-determinism from resulting in
# unnecessary changes
if test $# -ne 1; then
  echo "Usage: $0 FILE" >&2
  exit 1
fi

if ! git cat-file -e "HEAD:$1" &>/dev/null; then
  # if git cat-file -e fails, then the file doesn't exist at HEAD, so it's new,
  # meaning we need to encrypt it for the first time
  echo "$0: no previous version found while cleaning $1" >&2
  sops --input-type binary --output-type binary --encrypt /dev/stdin

  # TODO: figure out a better way to open fd 3
elif exec 3< <(echo -n) && diff \
  <(git cat-file -p "HEAD:$1" | sops --input-type binary --output-type binary --decrypt /dev/stdin) \
  <(cat /dev/stdin | tee /dev/fd/3) >/dev/null; then
  # if there's no difference between the decrypted version of the file at HEAD
  # and the new contents, then we re-use the previous version to prevent
  # unnecessary file updates
  echo "$0: no changes found while cleaning $1" >&2
  git cat-file -p "HEAD:$1"
else
  # if there is a difference then we re-encrypt it from fd 3, where we
  # duplicated stdin to
  echo "$0: found changes while cleaning $1" >&2
  sops --input-type binary --output-type binary --encrypt /dev/fd/3
fi

Basically, what this does is check if a previous version exists, and if it does, compares the decrypted contents of the previous version with the latest version being fed to stdin. If there's no difference, then it re-uses the old version. Otherwise, it re-encrypts the file.

If you swap out your clean command to be the path to this script, plus a %f, then you shouldn't get unecessary update commits. For example, something like the following:

[filter "sops"]
	required = true
	smudge = sops --input-type binary --output-type binary --decrypt /dev/stdin
	clean = scripts/git-filter-sops-clean %f

You might want to swap out the *-type binary arguments for JSON if that's what your use-case requires.

If someone has pointers on how I can resolve that TODO about the wierd opening of fd 3, please let me know. I don't really know what I'm doing with bash file descriptors.

Edit: just realized, this won't behave properly if you change the keys without changing the file contents, so keep that in mind.

[archite]

Thanks, @mtoohey31. I was trying to get something working for age and your script was the answer. I did away with creating file descriptors base populating a variable with stdin. This would work equally well with sops which I'm also using.

#!/usr/bin/env bash

PS4='${LINENO}: '

set -euo pipefail

# Exit if no file given
test $# -eq 1

# Exit if no stdin
test -t 0 && exit 1

decrypt() {
  age -d -i ~/.config/sops/age/keys.txt
}

encrypt() {
  age -r someagekey -a
}

show() {
  printf "%s\n" "${@}"
}

INPUT=$(cat)
: ${ENCRYPTED:=$(encrypt <<<${INPUT})}
: ${CONTENTS:=$(git cat-file -p "HEAD:${1}" 2>/dev/null)}
: ${DECRYPTED=$(decrypt <<<${CONTENTS} 2>/dev/null)}

if [[ -z "${CONTENTS}" || "${DECRYPTED}" != "${INPUT}" ]]
then
  show "${ENCRYPTED}"
else
  show "${CONTENTS}"
fi

[prskr]

I built a prototype for something similar based on age: git-age. If anyone is brave enough to give it a try I'd highly appreciate feedback 😄

I only started last week and I intentionally built a rough PoC without any tests (so far) but I hope I can clean out the rough edges "soon" ™️

[archite]

@prskr looks interesting. I'll checkout when I have some free cycles!

[bphenriques]

Was someone able to use sops --input-type json --output-type json --encrypt /dev/stdin with a --filename-override ?

It seems to be a illegal argument if first argument:

$ cat Makefile | sops --input-type binary --filename-override Makefile --encrypt /dev/stdin
...
FATA[0000] flag provided but not defined: -filename-override 

But seems to accept, but ignore when last:

$ cat Makefile | sops --input-type binary --encrypt /dev/stdin --filename-override Makefile

creation_rules:
  - path_regex: Makefile
    key_groups:
      - age:
          - *desktop
          - *bphenriques

Does not seem to possible 🤔 I am trying to make this work with path_regex as I am using in my dotfiles to setup different machines with different sets of secrets.

[felixfontein]

@bphenriques --filename-override is only availae on the main branch, not yet in any release. It will appear in the 3.9.0 release (whenever that will happen). (Feature added in #1332.)

But seems to accept, but ignore when last:

$ cat Makefile | sops --input-type binary --encrypt /dev/stdin --filename-override Makefile

Basically sops [some opions here] filename ignores everything after the filename. That's an unfortunate result of how the argument parsing currently works. The next release (3.9.0) will print some warnings in case something is found after the filename. (Warning added in #1342.)