CI: conda-forge focused tests should respect pinnings

[jaimergp]

(Comes from #2964 (comment)).

Conda Forge maintains their version pinnings in this repository, mainly to ensure ABI compatibility and prevent other conflicts. When we install our build environments we are not constraining the installed versions to these pinnings, so we might get too recent ones.

I guess this only affects us for compilers, swig and fftw, but still, it'd be nice to ensure the conda-forge focused tests accurately describe the build environment in the feedstock for the sake of catching errors before they get to conda-forge. Still, the feedstock has now a way to run full CI there (sans GPU), so this would get uncovered in the RC phase.

As a result, this is not a priority, but a nice-to-have CI feature.