Update all submodules that rustbuild doesn't depend on lazily

[jyn514]

This only updates the submodules the first time they're needed, instead
of unconditionally the first time you run x.py.

Ideally, this would move all submodules to rustbuild and not exclude some tools and
backtrace. Unfortunately, cargo requires all Cargo.toml files in the
whole workspace to be present to build any crate.

On my machine, this takes the time for an initial submodule clone (for
x.py --help) from 55.70 to 15.87 seconds.

Helps with #76653. Builds on #86015 and should not be merged before (only the last commit is relevant).

[rust-highfive]

r? @Mark-Simulacrum

(rust-highfive has picked a reviewer for you, use r? to override)

[jyn514]

This is mostly working, but it now tries to update stdarch 3 times in a row, which is slow especially on windows.

$ x check
Updating only changed submodules
Submodules updated in 0.17 seconds
    Finished dev [unoptimized + debuginfo] target(s) in 0.69s
Updating submodule library/stdarch
Updating submodule src/doc/embedded-book
Updating submodule src/tools/rust-analyzer
Submodule path 'src/tools/rust-analyzer': checked out 'b82458818d44dfe5b4b5db38d8113e3f3194506e'
Updating submodule library\stdarch
Updating submodule library\stdarch
Checking stage0 std artifacts (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)

[jyn514]

This is mostly working, but it now tries to update stdarch 3 times in a row, which is slow especially on windows.

cc https://rust-lang.zulipchat.com/#narrow/stream/182449-t-compiler.2Fhelp/topic/x.2Epy.20always.20updates.20LLVM.20submodule

[bors]

☔ The latest upstream changes (presumably #85711) made this pull request unmergeable. Please resolve the merge conflicts.

[petrochenkov]

@jyn514
Can the updates be skipped only for submodules that are not yet initialized?

For submodules that are already initialized this results in a dirty git tree with out-of-date submodules - this is what I got after the recent LLVM changes.
Then you have to update manually with git submodule update instead of just running any x.py command which you are going to run anyway.
Partial updates usually require fetching only a couple of commits (except for perhaps LLVM version updates) and are therefore quick.

[jyn514]

For submodules that are already initialized this results in a dirty git tree with out-of-date submodules - this is what I got after the recent LLVM changes.

Hmm, I can't replicate this behavior:

$ x check
Updating only changed submodules
Submodules updated in 0.01 seconds
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
Updating submodule src/llvm-project
Submodule path 'src/llvm-project': checked out '5f67a5715771b7d29e4713e8d68338602d216dcf'
Checking stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

Do you remember the command you ran?

[jyn514]

No longer blocked since #86015 :)

[bors]

⌛ Testing commit 04718038eead5ee061f8d538b8f52f2e5ec50a17 with merge d869ed04dc9c324669fe2f5e4d97c4107e27aeb6...

[bors]

💔 Test failed - checks-actions

[jyn514]

@bors r=Mark-Simulacrum rollup=iffy

[bors]

📌 Commit 2ac0e9b has been approved by Mark-Simulacrum

[bors]

⌛ Testing commit 2ac0e9b with merge ac575b6...

[bors]

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing ac575b6 to master...

[semarie]

While building -nightly from tarball, the build fails with:

running: /data/semarie/build-rust/build_dir/build/bootstrap/debug/bootstrap dist --jobs=4
warning: x.py has made several changes recently you may want to look at
help: consider looking at the changes in `src/bootstrap/CHANGELOG.md`
note: to silence this warning, add `changelog-seen = 2` at the top of `config.toml`
finding compilers
CC_x86_64-unknown-openbsd = "cc"
CFLAGS_x86_64-unknown-openbsd = ["-O2", "-ffunction-sections", "-fdata-sections", "-fPIC", "-m64", "-O2", "-pipe"]
CXX_x86_64-unknown-openbsd = "c++"
CXXFLAGS_x86_64-unknown-openbsd = ["-O2", "-ffunction-sections", "-fdata-sections", "-fPIC", "-m64", "-O2", "-pipe"]
AR_x86_64-unknown-openbsd = "ar"
running sanity check
learning about cargo
thread 'main' panicked at 'command did not execute successfully: "git" "config" "--file" "/data/semarie/build-rust/build_dir/rustc-nightly-src/.gitmodules" "--get-regexp" "path"
expected success, got: exit status: 1', src/bootstrap/lib.rs:568:22
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Traceback (most recent call last):
  File "/data/semarie/build-rust/build_dir/rustc-nightly-src/x.py", line 27, in <module>
    bootstrap.main()
  File "/data/semarie/build-rust/build_dir/rustc-nightly-src/src/bootstrap/bootstrap.py", line 1209, in main
    bootstrap(help_triggered)
  File "/data/semarie/build-rust/build_dir/rustc-nightly-src/src/bootstrap/bootstrap.py", line 1195, in bootstrap
    run(args, env=env, verbose=build.verbose, is_bootstrap=True)
  File "/data/semarie/build-rust/build_dir/rustc-nightly-src/src/bootstrap/bootstrap.py", line 153, in run
    raise RuntimeError(err)
RuntimeError: failed to run: /data/semarie/build-rust/build_dir/build/bootstrap/debug/bootstrap dist --jobs=4
Thu Jul 22 18:07:30 CEST 2021: task not finished: see build.log for detail

it seems the build tries to run git config --file path/to/.gitmodules --get-regexp path whereas the tarball doesn't contains .gitmodules file.

[jyn514]

I have to look into how tarballs work - is there a reason they don't set submodules = false?

[Mark-Simulacrum]

We don't put any new default config in the tarballs (and I don't think we should); I imagine the right check here is something like is_git() and if so defaulting submodules to false.

[semarie]

I confirm that setting submodules = false is enought to make the build to not fail.

[petrochenkov]

@jyn514
As I previously complained in #82653 (comment) this PR results in having a bunch of out-of-date submodules that are no longer synchronized on any x.py run, but require manually running git submodule update.

How to reproduce:

• Have some moderately old (e.g. two weeks) branch you've been previously working on as a current branch.

   $ git status
   On branch my_old_branch

• Make a fresh branch from upstream/master (assuming upstream/master already refers to newer versions for some submodules)

  git fetch --all --prune
  git checkout -b my_fresh_branch upstream/master

• Run x.py to prepare for compiler work and watch youtube for some time

  ./x.py test src/test/ui --test-args zzzzzz

• Have a bunch of dirty submodules that pollute git gui and git status even after running x.py

   $ git status
   On branch my_fresh_branch
   Your branch is up to date with 'upstream/master'.
  
   Changes not staged for commit:
     (use "git add <file>..." to update what will be committed)
     (use "git restore <file>..." to discard changes in working directory)
   		modified:   src/doc/book (new commits)
   		modified:   src/doc/edition-guide (new commits)
   		modified:   src/doc/embedded-book (new commits)
   		modified:   src/doc/reference (new commits)
   		modified:   src/doc/rust-by-example (new commits)
   		modified:   src/doc/rustc-dev-guide (new commits)
   		modified:   src/tools/rust-analyzer (new commits)
  

If the submodule was already initialized, then updating it is very quick and can be done unconditionally, even if the given submodule is not strictly necessary for the given x.py command.
x.py did the right thing previously.
I agree that not cloning/initializing submodules that were never initialized previously is a viable optimization, but not updating already initializing submodules only results in inconveniences.

[jyn514]

If the submodule was already initialized, then updating it is very quick and can be done unconditionally, even if the given submodule is not strictly necessary for the given x.py command.
x.py did the right thing previously.
I agree that not cloning/initializing submodules that were never initialized previously is a viable optimization, but not updating already initializing submodules only results in inconveniences.

Contrary to what you seem to believe, I'm not trying to break your workflow. Not updating the submodules is just a bug, it's not intentional. I think #87443 should fix it.