Push test-results to circle for nightly tests

[raubitsj]

Description

We want to save the junit-xml files for nightly tests so we can use circleci test insights and we want to push it as an artifact so we can do our own test insight pipeline.

To accomplish this:

• A Persistent volume needed to be created for cloud-test pods
• At the conclusion of the test in tox.ini we run pod-store which copies data to the persistent volume
• The circleci job needs to grab this data to a local dir but the pod is already complete so it has to launch a new attach-pod to access the persistent volume
• Use kubectl to copy data out
• Use circleci store_* flows to save the test results and the artifacts
• profit

Testing

Tested with:

./tools/circleci-tool.py trigger-nightly --shards standalone-cpu

Successful nightly full trigger:
https://app.circleci.com/pipelines/github/wandb/wandb/15130/workflows/32637510-9d52-4e06-9ff3-9f3ff3264f11

[codecov]

Codecov Report

Merging #4153 (6c4289f) into master (bea40b2) will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4153      +/-   ##
==========================================
+ Coverage   82.71%   82.73%   +0.01%     
==========================================
  Files         256      256              
  Lines       32534    32534              
==========================================
+ Hits        26912    26916       +4     
+ Misses       5622     5618       -4     

Flag	Coverage Δ
functest	55.01% <ø> (+<0.01%)	⬆️
unittest	73.45% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
wandb/sdk/lib/git.py	79.01% <0.00%> (-0.62%)	⬇️
wandb/sdk/internal/meta.py	90.79% <0.00%> (+3.06%)	⬆️

[dmitryduev]

This is sooooo awesome, thank you Jeff!!

Any comments on the current nightly setup re: its usability/debugability? One thing I have on the list is more robust cluster management