Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

integrate gix-negotiate #861

Merged
merged 16 commits into from Jun 6, 2023
Merged

integrate gix-negotiate #861

merged 16 commits into from Jun 6, 2023

Conversation

Byron
Copy link
Owner

@Byron Byron commented May 23, 2023 •
edited

Follow-up of #855

Tasks

  • port multi-round test
  • use shared Graph in algorithms and re-add early .parsed check equivalents where feasible.
  • pass shared graph as &mut to each trait method
  • use negotiator and assure there is an object cache available for the graph traversal. Note that handle should not retry on misses (and do the same for describe). Assure we feed it dereferenced (i.e. peel to commit) commits only.
  • support for adding tips of alternate dbs.
  • support for replacement objects in commit-graph lookups. - this is implicitly implemented, at least so I think.
  • handle window size
  • make sure that missing objects (like when we check for remote-objects and their presence) don't trigger re-reading of the ODB structure
  • if there is ref-in-want, use ref-in-want to better handle busy HTTP repos
  • protocol.version as part of config tree, use here
  • fix all tests
  • be sure Arguments work as we think - I think haves have to be re-added each time, which means we can remove the crate-status comment if we re-add acks only on every trip.
  • make V1 stateless multi-round negotiation work
  • fetch a repo that has alternates (add alternate to clone so there is nothing to fetch from the remote as remote is the alternate)

Out of scope (for follow-up PR)

Research

  • Naive is really the first part of the 'common' algorithm which sends most of the refs as specified by the refspec. What's sent next is implemented by the algorithm, which we don't really implement at all.
  • with git fetch --negotiate-only it's possible to setup baseline expectations as we see the common commits produced by an algorithm, given a specific input of --negotiation-tips, which is equivalent to controlling the starting point of the negotiation
  • noop truly does nothing and is supposed to be used for refetching certain objects if a filter was used previously. It by itself wouldn't even trigger a full pack to be fetched as it also won't put any want line.
  • mark_complete_and_common_ref, mark_tips and for_each_cached_alternate (add tips of alternates) is the same sequence both in V1 and V2, then it sense HAVE lines as produced by the negotiator, feed ACK lines to it.
  • the first batch is definitely 16, but the window size adapts also depending on the protocol version (to be investigated)
  • --negotiate-only does its very own negotiation and thus isn't comparable - probably for V1 compatibility, we want to packet line tracing. However, negotiate_using_fetch provides a clean overview of what negotiation should be.
  • The window size for HAVE lines starts at 16, and is then increased with each round according to the rules in next_flush.
  • When traversing the commit-graph, which we probably have to do to some extend, the use of generation numbers is optional (and it defaults to INFINITY which effectively turns them off).
  • transport-actual-expected.zip - this recording of GIT_TRACE_PACKET=1 output shows a failure mode with multi-round negotiation. We hang because we keep reading acknowledgements past the NAK

@Byron Byron force-pushed the integrate-gix-negotiate branch 3 times, most recently from 46a1e86 to 3256523 Compare May 26, 2023 08:53
@g2p g2p mentioned this pull request May 26, 2023
Closed
1 task
@Byron Byron force-pushed the integrate-gix-negotiate branch 2 times, most recently from ce26df5 to 1c135e8 Compare June 1, 2023 19:49
@Byron Byron changed the title integrate gix negotiate integrate gix-negotiate Jun 1, 2023
@Byron Byron force-pushed the integrate-gix-negotiate branch 13 times, most recently from ca2c746 to 4289f20 Compare June 6, 2023 08:30
Byron added 10 commits June 6, 2023 15:42
@Byron
thanks clippy
9525ac8
@Byron
fix journey test (remove usage of tree)
3c1bfd5 
It seems homebrew `tree` keeps messing up directory counts, which may make it incompatible
with CI and generaly makes it impossible to use in a cross-platform manner.

Thus it was replaced with `find`.
@Byron
Add note about corrected generation dates
f3193c9 
Right now we don't handle them at all, which might lead to issues in huge repos.
@Byron
feat!: respect core.useReplaceRefs and remove `gitoxide.objects.noR…
e011e36 
…eplace`.

The gitoxide specific variable wasn't needed in the first place.
@Byron
Keep track of core.alternateRefsCommand
d925896
@Byron
feat!: Add version of Graph that handles fully-parsed commits
11ad8a8 
This renames `graph::Commit` to `graph::LazyCommit` to make space for `graph::Commit` to be a fully owned.
`LazyCommit::to_owned()` was added to obtain fully owned `Commit` instances.
Rename `Graph::try_lookup_and_insert()` to `Graph::try_lookup_or_insert()` and
`Graph::try_lookup_and_insert_default()` to `Graph::try_lookup_or_insert_default()`

Additionally, add the `peek()` and `iter_unordered()` method to the `PriorityQueue`, along with an implementation for `Clone`
Rename `PriorityQueue::iter_random()` to `::iter_unordered()`.
@Byron
feat: allow graph sharing by unifying Flags type.
1bd93be 
This makes the graph used in `gix-negotiate` shareable by callers, which can
do their own traversal and store their own flags. The knowlege of this traversal
can be kept using such shared flags, like the `PARSED` bit which should be set whenever
parents are traversed.

That way we are able to emulate the algorithms git uses perfectly, as we keep exactly the
same state.
@Byron
feat: add Store::alternate_db_paths().
8c72a23 
Provides a way to learn about loose database paths that are provided by
git alternates.
@Byron
fix!: parse 'handshake' of file:// based protocol version 0.
aa2d064 
This special protocol kicks in when `git` serves `file://` directly
and no version number is specified. Then it doesn't advertise capabilities
at all, but shows 0000 right away.
Make sure we can parse it, and show it by adding `Version::V0` as well.
@Byron
adapt to changes in gix-transport
60eaceb
Byron added 3 commits June 6, 2023 15:42
@Byron
feat: Add fetch::Arguments::is_stateless() to aid proper use of arg…
877aa29 
…uments.

When arguments are used, haves are reset every round in stateless protocols, while
everything else is repeated. However, this also means that previously confirmed common
commits aren't repeated unless this is specifically implemented by the user of `Arguments`.

That caller can now easily determine if negotiations have to be compensated for.

Please note that `Arguments` explicitly doesn't implement repeating of all prior arguments, which
would also repeat a lot of *in-vain* haves.
@Byron
fix: include-tag is now properly handled in V1 fetch arguments
c5dc7b4 
Previously it would be added like it's V2 arguments, which makes using it
in V1 impossible.
@Byron
fix!: make V1 stateless negotations work.
6a3c021 
This is done by working around another V1 negotiation oddity
by exploiting client-side knowledge. For very old servers, we probably
wouldn't be able to do multi-rounds without dead-locking, but with
recent-enough (probably 10 years or so) old git servers all should
work fine.

All this to not actually have to implement the V1 strangeness, allowing
our code to work smoothly with all permutations of stateless/stateful connections
and V1/V2 interactions, with a single high-level implementation essentially.
Byron added 3 commits June 6, 2023 16:29
@Byron
feat: use gix-negotiate in fetch machinery.
af0ef2f 
Thanks to it we are finally able to do pack negotiations just like git can,
as many rounds as it takes and with all available algorithms.

Works for V1 and V2 and for stateless and stateful transports.
@Byron
Add test to validate alternates in the context of fetching
ae1bc41
@Byron
adapt to changes in gix
7983f6f
@Byron Byron mentioned this pull request Jun 2, 2023
Open
27 tasks
@Byron Byron merged commit ae845de into main Jun 6, 2023
16 checks passed
@Byron Byron deleted the integrate-gix-negotiate branch June 6, 2023 15:34
This was referenced Jun 7, 2023
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@Byron