[Bug]: Nix installation failures on self-hosted GitHub runners

[platten]

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

Current Behaviour

The release.yml workload is routinely failing on the cachix/install-nix-action step due to previous state on self-hosted GitHub runners and requires routine manual intervention. This is not sustainable and has blocked support for ARM64 based runners #1931 and necessitated #1936

Expected Behaviour

cachix/install-nix-action should be more resilient and support cleanup/uninstall after executing the job.

There may be three ways of addressing this:

1. Fix Nix/cachix-install-nix-action to work well on self-hosted runners. I've taken a stab at this, but I'm a bit out of my depth and will require @rvolosatovs insight to resolve this. Preferably we would like to clean up between runs for this.
2. Remove Nix from the equation (my preferred approach). This isn't that much work and is straight forward.
3. Make self-hosted runners stateless. This may require us to move over to docker-ized runners running in K8s and managed by something like actions runner controller or github actions runner operator. While I'm afraid that this is the right long term strategy, it will be quite a heavy lift.

Environment Information

All self-hosted runners affected. Including self-hosted aarch64-ubuntu and aarch64-darwin runners.

Steps To Reproduce

No response

[rvolosatovs]

Refs cachix/install-nix-action#98 (please check the thread for possible solutions, if any)