Skip to content

Commit

Permalink
ci(check-urls): use a matrix to speed-up execution (#7060)
Browse files Browse the repository at this point in the history
* ci(check-urls): use a matrix to speed-up execution

The matrix strategy creates 10 max-parallel workers with a disabled fail-fast parameter

Examples
- Before: https://github.com/davorpa/free-programming-books/actions/runs/2999590952
- After: https://github.com/davorpa/free-programming-books/actions/runs/3007199364

Running the workflow when -langs or -subject files are involved could exhaust the runner execution max-time.

Using this implementation... the elapsed time is of the longest execution time, normally no more than 30min on the worst of cases

* fix: parallel strategy when matrix files.length == 0

* security: apply #9287

::set-output --> GITHUB_OUTPUT env file
  • Loading branch information
davorpa committed Feb 24, 2023
1 parent 44dd203 commit 62d466d
Showing 1 changed file with 66 additions and 34 deletions.
100 changes: 66 additions & 34 deletions .github/workflows/check-urls.yml
Expand Up @@ -5,6 +5,7 @@ on:
pull_request:

permissions:
# needed for checkout code
contents: read

# This allows a subsequently queued workflow run to interrupt/wait for previous runs
Expand All @@ -13,69 +14,100 @@ concurrency:
cancel-in-progress: false # true = interrupt, false = wait

jobs:
check-urls:
runs-on: ubuntu-latest

outputs:
changed-files: ${{ steps.changed-files.outputs.all_changed_files }}

steps:

# NOTE: tj-actions/changed-files.
# For push events you need to include fetch-depth: 0 | 2 depending on your use case.
# 0: retrieve all history for all branches and tags
# 1: retrieve only current commit (by default)
# 2: retrieve until the preceding commit
- name: Determine workflow parameters
id: init-params
get-changed-files:
name: Get changed files
runs-on: ubuntu-latest
outputs:
fetch-depth: ${{ steps.set-params.outputs.fetch-depth }}
files: ${{ steps.set-files.outputs.files }}
files-len: ${{ steps.set-files.outputs.files-len }}
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- name: Determine workflow params
id: set-params
run: |
echo "fetch_depth=0" >> $GITHUB_OUTPUT
if [ "${{ github.event_name }}" == "pull_request" ]; then
echo "fetch_depth=0" >> $GITHUB_OUTPUT
fi
- uses: actions/checkout@v3
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: ${{ steps.init-params.outputs.fetch_depth }}

fetch-depth: ${{ steps.set-params.outputs.fetch-depth }}
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@v35.5.5
with:
separator: " "
json: true
- id: set-files
run: |
echo "${{ steps.changed-files.outputs.all_changed_files }}" \
| jq --raw-output '. | join(" ")' \
| sed -e 's/^/files=/' \
>> $GITHUB_OUTPUT
echo "${{ steps.changed-files.outputs.all_changed_files }}" \
| jq --raw-output '. | length' \
| sed -e 's/^/files-len=/' \
>> $GITHUB_OUTPUT
- id: set-matrix
run: |
echo "{\"file\":${{ steps.changed-files.outputs.all_changed_files }}}" \
| sed -e 's/^/matrix=/' \
>> $GITHUB_OUTPUT
- uses: ruby/setup-ruby@v1
check-urls:
name: Check @ ${{ matrix.file }}
if: ${{ fromJSON(needs.get-changed-files.outputs.files-len) > 0 }}
needs: [get-changed-files]
runs-on: ubuntu-latest
strategy:
matrix: ${{ fromJSON(needs.get-changed-files.outputs.matrix) }}
max-parallel: 10
fail-fast: false
steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: ${{ needs.get-changed-files.outputs.fetch-depth }}
- name: Setup Ruby v2.6
uses: ruby/setup-ruby@v1
with:
ruby-version: 2.6

- run: |
- name: Install awesome_bot
run: |
gem install awesome_bot
- name: Check each changed file
- name: "Check URLs of file: ${{ matrix.file }}"
run: |
# Set field separator
IFS=$' ';
# Processing loop
for file in ${{ steps.changed-files.outputs.all_changed_files }}; do
echo;
echo "::group::Processing file... $file";
awesome_bot "$file" --allow-redirect --allow-dupe --allow-ssl || true;
echo "::endgroup::";
done
# Unset field separator
unset IFS;
awesome_bot "${{ matrix.file }}" --allow-redirect --allow-dupe --allow-ssl || true;
- uses: actions/upload-artifact@v3
with:
name: awesomebot-results
path: ${{ github.workspace }}/ab-results-*.json

- name: Generate Summary Report using AwesomeBot results

reporter:
name: GitHub report
needs: [get-changed-files, check-urls]
runs-on: ubuntu-latest
steps:
- name: Checkout # for having the sources of the local action
uses: actions/checkout@v3
# download and unzip the ab-results-*.json generated by job-matrix: check-urls
- uses: actions/download-artifact@v3
with:
name: awesomebot-results
- name: Generate Summary Report
uses: ./.github/actions/awesomebot-gh-summary-action
with:
ab-root: ${{ github.workspace }}
files: ${{ steps.changed-files.outputs.all_changed_files }}
files: ${{ needs.get-changed-files.outputs.files }}
separator: " "
append-heading: ${{ true }}

0 comments on commit 62d466d

Please sign in to comment.