Tagging tests

docs/CHANGELOG.md

@@ -23,6 +23,8 @@ The format is based on [Keep a Changelog][kac] and this project adheres to
 * `BATS_TEST_TIMEOUT` variable to force a timeout on test (including `setup()`) (#491)
 * also print (nonempty) `$stderr` (from `run --separate-stderr`) with
   `--print-output-on-failure` (#631)
+* `# bats test_tags=<tag list>`/`# bats file_tags=<tag list>` and
+  `--filter-tags <tag list>` for tagging tests for execution filters (#642)
 
 #### Documentation
 

docs/source/writing-tests.md

@@ -12,6 +12,85 @@ For sample test files, see [examples](https://github.com/bats-core/bats-core/tre
 
 [bats-eval]: https://github.com/bats-core/bats-core/wiki/Bats-Evaluation-Process
 
+## Tagging tests
+
+Startig with version 1.8.0, Bats comes with a tagging system, that allows users
+to categorize their tests and filter according to those categories.
+
+Each test has a list of tags attached to it. Without specification, this list is empty.
+Tags can be defined in two ways. The first being `# bats test_tags=`:
+
+```bash
+# bats test_tags=tag:1, tag:2, tag:3
+@test "second test" {
+  # ...
+}
+
+@test "second test" {
+  # ...
+}
+```
+
+These tags (`tag:1`, `tag:2`, `tag:3`) will be attached to the test `first test`.
+The second test will have no tags attached. Values defined in the `# bats test_tags=`
+directive will be assigned to the next `@test` that is being encountered in the
+file and forgotten after that. Only the value of the last `# bats test_tags=` directive
+before a given test will be used.
+
+Sometimes, we want to give all tests in a file a set of the same tags. This can
+be achieved via `# bats file_tags=`. They will be added to all tests in the file
+after that directive. An additional `# bats file_tags=` directive will override
+the previously defined values:
+
+```bash
+@test "Zeroth test" { 
+  # will have no tags
+}
+
+# bats file_tags=a:b
+# bats test_tags=c:d
+
+@test "First test" { 
+  # will be tagged a:b, c:d
+}
+
+# bats file_tags=
+
+@test "Second test" {
+  # will have no tags
+}
+```
+
+Tags are case sensitive and must only consist of alphanumeric characters and `_`,
+ `-`, or `:`. They must not contain whitespaces!
+The colon is intended as a separator for (recursive) namespacing.
+
+Tag lists must be separated by commas and are allowed to contain whitespace.
+They must not contain empty tags like `test_tags=,b` (first tag is empty),
+`test_tags=a,,c`, `test_tags=a,  ,c` (second tag is only whitespace/empty),
+`test_tags=a,b,` (third tag is empty).
+
+Every tag starting with a `bats:` (case insensitive!) is reserved for Bats'
+internal use.
+
+### Filtering execution
+
+Tags can be used for more finegrained filtering of which tests to run via `--filter-tags`.
+This accepts a comma separated list of tags. Only tests that match all of these
+tags will be executed. For example, `bats --filter-tags a,b,c` will pick up tests
+with tags `a,b,c`, but not tests that miss one or more of those tags.
+
+Additionally, you can specify negative tags via `bats --filter-tags a,!b,c`,
+which now won't match tests with tags `a,b,c`, due to the `b`, but will select `a,c`.
+To put it more formally, `--filter-tags` is a boolean conjunction.
+
+To allow for more complex queries, you can specify multiple `--filter-tags`.
+A test will be executed, if it matches at least one of them.
+This means multiple `--filter-tags` form a boolean disjunction.
+
+A query of `--filter-tags a,!b --filter-tags b,c` can be translated to:
+Execute only tests that (have tag a, but not tag b) or (have tag b and c).
+
 ## Comment syntax
 
 External tools (like `shellcheck`, `shfmt`, and various IDE's) may not support

lib/bats-core/common.bash

@@ -95,4 +95,119 @@ bats_binary_search() { # <search-value> <array-name>
 
   # did not find it -> its not there
   return 1
-}
+}
+
+# store the values in ascending order in result array
+# Intended for short lists!
+bats_sort() { # <result-array-name> <values to sort...>
+  local -r result_name=$1
+  shift
+
+  if (( $# == 0 )); then
+    eval "$result_name=()"
+    return 0
+  fi
+
+  local -a sorted_array=()
+  local -i j i=0
+  for (( j=1; j <= $#; ++j )); do
+    for ((i=${#sorted_array[@]}; i >= 0; --i )); do
+      if [[ $i -eq 0 || ${sorted_array[$((i-1))]} < ${!j} ]]; then
+        sorted_array[$i]=${!j}
+        break
+      else
+        sorted_array[$i]=${sorted_array[$((i-1))]}
+      fi
+    done
+  done
+
+  eval "$result_name=(\"\${sorted_array[@]}\")"
+}
+
+# check if all search values (must be sorted!) are in the (sorted!) array
+# Intended for short lists/arrays!
+bats_all_in() { # <sorted-array> <sorted search values...>
+  local -r haystack_array=$1
+  shift
+
+  local -i haystack_length # just to appease shellcheck
+  eval "local -r haystack_length=\${#${haystack_array}[@]}"
+
+  local -i haystack_index=0 # initialize only here to continue from last search position
+  local search_value haystack_value # just to appease shellcheck
+  for (( i=1; i <= $#; ++i )); do
+    eval "local search_value=${!i}"
+    for ((; haystack_index < haystack_length; ++haystack_index)); do
+      eval "local haystack_value=\${${haystack_array}[$haystack_index]}"
+      if [[ $haystack_value > "$search_value" ]]; then
+        # we passed the location this value would have been at -> not found
+        return 1
+      elif [[ $haystack_value == "$search_value" ]]; then
+        continue 2 # search value found  -> try the next one
+      fi
+    done
+    return 1 # we ran of the end of the haystack without finding the value!
+  done
+
+  # did not return from loop above -> all search values were found
+  return 0
+}
+
+# check if any search value (must be sorted!) is in the (sorted!) array
+# intended for short lists/arrays
+bats_any_in() { # <sorted-array> <sorted search values>
+  local -r haystack_array=$1
+  shift
+
+  local -i haystack_length # just to appease shellcheck
+  eval "local -r haystack_length=\${#${haystack_array}[@]}"
+
+  local -i haystack_index=0 # initialize only here to continue from last search position
+  local search_value haystack_value # just to appease shellcheck
+  for (( i=1; i <= $#; ++i )); do
+    eval "local search_value=${!i}"
+    for ((; haystack_index < haystack_length; ++haystack_index)); do
+      eval "local haystack_value=\${${haystack_array}[$haystack_index]}"
+      if [[ $haystack_value > "$search_value" ]]; then
+        continue 2 # search value not in array! -> try next
+      elif [[ $haystack_value == "$search_value" ]]; then
+        return 0 # search value found
+      fi
+    done
+  done
+
+  # did not return from loop above -> no search value was found
+  return 1
+}
+
+bats_trim () { # <output-variable> <string>
+  local -r bats_trim_ltrimmed=${2#"${2%%[![:space:]]*}"} # cut off leading whitespace
+  # shellcheck disable=SC2034 # used in eval!
+  local -r bats_trim_trimmed=${bats_trim_ltrimmed%"${bats_trim_ltrimmed##*[![:space:]]}"} # cut off trailing whitespace
+  eval "$1=\$bats_trim_trimmed"
+}
+
+# a helper function to work around unbound variable errors with ${arr[@]} on Bash 3
+bats_append_arrays_as_args () { # <array...> -- <command ...>
+  local -a trailing_args=()
+  while (( $# > 0)) && [[ $1 != -- ]]; do
+    local array=$1
+    shift
+
+    if eval "(( \${#${array}[@]} > 0 ))"; then
+      eval "trailing_args+=(\"\${${array}[@]}\")"
+    fi
+  done
+  shift # remove -- separator 
+
+  if (( $# == 0 )); then
+    printf "Error: append_arrays_as_args is missing a command or -- separator\n" >&2
+    return 1
+  fi
+
+  if (( ${#trailing_args[@]} > 0 )); then
+    "$@" "${trailing_args[@]}"
+  else
+    "$@"
+  fi
+}

lib/bats-core/preprocessing.bash

@@ -9,7 +9,7 @@ bats_preprocess_source() {
 	# export to make it visible to bats_evaluate_preprocessed_source
 	# since the latter runs in bats-exec-test's bash while this runs in bats-exec-file's
 	export BATS_TEST_SOURCE="${BATS_TMPNAME}.src"
-	bats-preprocess "$BATS_TEST_FILENAME" >"$BATS_TEST_SOURCE"
+	CHECK_BATS_COMMENT_COMMANDS=1 bats-preprocess "$BATS_TEST_FILENAME" >"$BATS_TEST_SOURCE"
 }
 
 bats_evaluate_preprocessed_source() {

libexec/bats-core/bats

@@ -255,6 +255,10 @@ while [[ "$#" -ne 0 ]]; do
     shift
     flags+=('--filter-status' "$1")
     ;;
+  --filter-tags)
+    shift
+    flags+=('--filter-tags' "$1")
+    ;;
   -*)
     abort "Bad command line option '$1'"
     ;;

libexec/bats-core/bats-exec-suite

@@ -7,6 +7,7 @@ num_jobs=${BATS_NUMBER_OF_PARALLEL_JOBS:-1}
 bats_no_parallelize_across_files=${BATS_NO_PARALLELIZE_ACROSS_FILES-}
 bats_no_parallelize_within_files=
 filter_status=''
+filter_tags_list=()
 flags=('--dummy-flag') # add a dummy flag to prevent unset variable errors on empty array expansion in old bash versions
 setup_suite_file=''
 BATS_TRACE_LEVEL="${BATS_TRACE_LEVEL:-0}"
@@ -16,6 +17,9 @@ abort() {
   exit 1
 }
 
+# shellcheck source=lib/bats-core/common.bash
+source "$BATS_ROOT/lib/bats-core/common.bash"
+
 while [[ "$#" -ne 0 ]]; do
   case "$1" in
   -c)
@@ -47,6 +51,19 @@ while [[ "$#" -ne 0 ]]; do
     shift
     filter_status="$1"
     ;;
+  --filter-tags)
+    shift
+    IFS=, read -ra tags <<<"$1" || true
+    if (( ${#tags[@]} > 0 )); then
+      for (( i = 0; i < ${#tags[@]}; ++i )); do
+        bats_trim "tags[$i]" "${tags[$i]}"
+      done
+      bats_sort sorted_tags "${tags[@]}"
+      IFS=, filter_tags_list+=("${sorted_tags[*]}")
+    else
+      filter_tags_list+=("")
+    fi
+    ;;
   --dummy-flag)
     ;;
   --trace)
@@ -91,6 +108,7 @@ fi
 TESTS_LIST_FILE="${BATS_RUN_TMPDIR}/test_list_file.txt"
 
 bats_gather_tests() {
+  local line test_line tags 
   all_tests=()
   for filename in "$@"; do
     if [[ ! -f "$filename" ]]; then
@@ -105,6 +123,43 @@ bats_gather_tests() {
       fi
       line="${line%$'\r'}"
       line="${line#* }"
+      TAG_REGEX="--tags '(.*)' (.*)"
+      if [[ "$line" =~ $TAG_REGEX ]]; then
+        IFS=, read -ra tags <<<"${BASH_REMATCH[1]}" || true
+        line="${BASH_REMATCH[2]}"
+      else
+        tags=()
+      fi
+      if [[ ${#filter_tags_list[@]} -gt 0 ]]; then
+        local match=
+        for filter_tags in "${filter_tags_list[@]}"; do
+            # empty search tags only match empty test tags!
+            if [[ -z "$filter_tags" ]]; then
+              if [[ ${#tags[@]} -eq 0 ]]; then
+                match=1
+                break
+              fi
+              continue
+            fi
+            local -a positive_filter_tags=() negative_filter_tags=()
+            IFS=, read -ra filter_tags <<< "$filter_tags" || true
+            for filter_tag in "${filter_tags[@]}"; do
+              if [[ $filter_tag == !* ]]; then
+                bats_trim filter_tag "${filter_tag#!}"
+                negative_filter_tags+=("${filter_tag}")
+              else
+                positive_filter_tags+=("${filter_tag}")
+              fi
+            done
+            if bats_append_arrays_as_args positive_filter_tags -- bats_all_in tags &&
+              ! bats_append_arrays_as_args negative_filter_tags -- bats_any_in tags; then
+              match=1
+            fi
+        done
+        if [[ -z "$match" ]]; then
+          continue
+        fi
+      fi
       test_line=$(printf "%s\t%s" "$filename" "$line")
       all_tests+=("$test_line")
       printf "%s\n" "$test_line" >>"$TESTS_LIST_FILE"
@@ -145,8 +200,6 @@ fi
 bats_gather_tests "$@"
 
 if [[ -n "$filter_status" ]]; then  
-  # shellcheck source=lib/bats-core/common.bash
-  source "$BATS_ROOT/lib/bats-core/common.bash"
   case "$filter_status" in
     failed)
       bats_filter_test_by_status() { # <line>