Fix caching for cargo cross CI tests #1287

Job	Run time
test_other_archs (powerpc-unknown-linux-gnu)	4m 37s
test_other_archs (i686-unknown-linux-gnu)	4m 16s
test_toolchains (1.67.1)	35s
Fuzz targets (afl)	2m 17s
build_benchmarks	54s
test_toolchains (nightly)	1m 46s
Fuzz targets (cargo-fuzz)	47s
test_toolchains (beta)	1m 45s
cargo-deny	19s
clippy	1m 33s
public_private_dependencies	25s
rustfmt	7s
test_avif_decoding	1m 19s
test_features	15s
test_features (default)	2m 23s
test_features (rayon)	32s
test_features (avif)	1m 6s
test_features (bmp)	25s
test_features (dds)	26s
test_features (exr)	33s
test_features (ff)	16s
test_features (gif)	49s
test_features (hdr)	19s
test_features (ico)	26s
test_features (jpeg)	31s
test_features (png)	28s
test_features (pnm)	24s
test_features (qoi)	20s
test_features (tga)	22s
test_features (tiff)	43s
test_features (webp)	47s
	31m 45s

Provide feedback