How to export list of directories as JSON to be used in Matrix

[gaby]

I've been struggling for over an hour trying to get a list of changed directories as JSON to be used in a Matrix. I have been following this CI file: https://github.com/tj-actions/changed-files/blob/main/.github/workflows/matrix-test.yml but it seems very convoluted. There used to be a flag to auto escape JSON from this action and it was removed a few versions ago.

This is what I have:

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Changed Files
        uses: tj-actions/changed-files@v40
        id: changed-files
        with:
          json: true
          files_ignore: |
            .github/**
            **.md
          dir_names: true
          dir_names_exclude_current_dir: true
      - name: List all changed directories
        run: echo '${{ steps.changed-files.outputs.all_changed_files }}'
      - name: Output list of changed directories
        id: set-matrix
        run: echo "matrix={\"directories\":${{ steps.changed-files.outputs.all_changed_files }}}" >> "$GITHUB_OUTPUT"

  gosec-scan:
    runs-on: ubuntu-latest
    needs: detect-changes
    env:
      GO111MODULE: on
    strategy:
      matrix:
        modules: ${{ fromJSON(needs.detect-changes.outputs.matrix) }}
    steps:
      - name: Fetch Repository
        uses: actions/checkout@v4
      - name: Install Go
        uses: actions/setup-go@v4
        with:
          go-version: '^1.19.x'
          check-latest: true
          cache: false
      - name: Install gosec
        run: go install github.com/securego/gosec/v2/cmd/gosec@latest
      - name: Run gosec
        working-directory: ${{ matrix.modules }}
        run: gosec ./...

Basically I want to detect which directories changed, except for .github/ and changes to .md files. I want to use this directory list as my working-directory in a follow-on Job.

Most of the solutions listed on the previous issues/discussions are using super old versions of this action with options no longer supported.

Looking at the logs from the CI, even though i'm specifying json: true, the output says this:

Step after changed-files:

Run echo '[\"redis\"]'
  echo '[\"redis\"]'
  shell: /usr/bin/bash -e {0}
[\"redis\"]

Run echo "matrix={\"directories\":[\"redis\"]}" >> "$GITHUB_OUTPUT"
  echo "matrix={\"directories\":[\"redis\"]}" >> "$GITHUB_OUTPUT"
  shell: /usr/bin/bash -e {0}

[gaby]

Got it to work using this:

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Changed Files
        uses: tj-actions/changed-files@v40
        id: changed-files
        with:
          json: true
          files_ignore: |
            .github/**
            **.md
          dir_names: true
          dir_names_exclude_current_dir: true
      - name: List all changed directories
        run: echo '${{ steps.changed-files.outputs.all_changed_files }}'
      - name: Output list of changed directories
        id: set-matrix
        run: echo "matrix={\"directories\":${{ steps.changed-files.outputs.all_changed_files }}}" >> "$GITHUB_OUTPUT"

  gosec-scan:
    runs-on: ubuntu-latest
    needs: detect-changes
    env:
      GO111MODULE: on
    strategy:
      matrix: ${{ fromJSON(needs.detect-changes.outputs.matrix) }}
    steps:
      - name: Fetch Repository
        uses: actions/checkout@v4
      - name: Install Go
        uses: actions/setup-go@v4
        with:
          go-version: '^1.19.x'
          check-latest: true
          cache: false
      - name: Install gosec
        run: go install github.com/securego/gosec/v2/cmd/gosec@latest
      - name: Run gosec
        working-directory: ${{ matrix.directories }}
        run: gosec ./...

Is there a way to simplify this? Why do we need to echo the JSON, instead of changed-files? There's some bug with json and dir_names since the output is just a List, not a JSON.

[jackton1]

Hi @gaby, can you elaborate on what’s different between this and the earlier version you posted?

[gaby]

I changed this:

    strategy:
      matrix:
        modules: ${{ fromJSON(needs.detect-changes.outputs.matrix) }}

To this:

    strategy:
      matrix: ${{ fromJSON(needs.detect-changes.outputs.matrix) }}

[gaby]

@jackton1 I was trying to use your suggestion from here #1094 (comment) but json_raw_format is not an option anymore?

[jackton1]

Hi @deronnax, the json_raw_format has been renamed to escape_json .

See: #1094 (comment)

[gaby]

I'm using this now, and it still doesn't work:

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.changed-files.outputs.all_changed_files }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Changed Files
        uses: tj-actions/changed-files@v40
        id: changed-files
        with:
          files_ignore: |
            .github/**
            **.md
          json: true
          dir_names: true
          dir_names_exclude_current_dir: true

  gosec-scan:
    runs-on: ubuntu-latest
    needs: detect-changes
    env:
      GO111MODULE: on
    strategy:
      matrix:
        modules: ${{ needs.detect-changes.outputs.matrix }}
    steps:
      - name: Fetch Repository
        uses: actions/checkout@v4
      - name: Install Go
        uses: actions/setup-go@v4
        with:
          go-version: '^1.19.x'
          check-latest: true
          cache: false
      - name: Install gosec
        run: go install github.com/securego/gosec/v2/cmd/gosec@latest
      - name: Run gosec
        working-directory: ${{ matrix.modules }}
        run: gosec ./...

[gaby]

I just tried changing the second job to:

matrix:
  modules: ${{ fromJSON(needs.detect-changes.outputs.matrix) }}

And it still doesn't work, the detecte-changes actions is not returning JSON. It just returns a list

[gaby]

Output from workflow:

[Gosec Security Scan: .github#L1](https://github.com/gofiber/storage/pull/1091/files#annotation_15775771792)
Error when evaluating 'strategy' for job 'gosec-scan'. .github/workflows/gosec.yml (Line: 49, Col: 18): Error parsing fromJson,.github/workflows/gosec.yml (Line: 49, Col: 18): Unexpected character encountered while parsing value: \. Path '', line 1, position 1.,.github/workflows/gosec.yml (Line: 49, Col: 18): Unexpected value ''

[gaby]

if i remove the fromJSON this is the output:

Error when evaluating 'strategy' for job 'gosec-scan'. .github/workflows/gosec.yml (Line: 49, Col: 18): Unexpected value '[\"redis\"]'

[gaby]

Got it to work now in a cleaner setup. The escape_json has to be set to false for this to work:

Working solution:

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.changed-files.outputs.all_changed_files }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Changed Files
        uses: tj-actions/changed-files@v40
        id: changed-files
        with:
          files_ignore: |
            .github/**
            **.md
          json: true
          escape_json: false
          dir_names: true
          dir_names_exclude_current_dir: true

  gosec-scan:
    runs-on: ubuntu-latest
    needs: detect-changes
    env:
      GO111MODULE: on
    strategy:
      matrix:
        modules: ${{ fromJSON(needs.detect-changes.outputs.matrix) }}
    steps:
      - name: Fetch Repository
        uses: actions/checkout@v4
      - name: Install Go
        uses: actions/setup-go@v4
        with:
          go-version: '^1.19.x'
          check-latest: true
          cache: false
      - name: Install gosec
        run: go install github.com/securego/gosec/v2/cmd/gosec@latest
      - name: Run gosec
        working-directory: ${{ matrix.modules }}
        run: gosec ./...