Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MultiPack index support #279

Closed
6 tasks done
Byron opened this issue Dec 20, 2021 · 1 comment
Closed
6 tasks done

MultiPack index support #279

Byron opened this issue Dec 20, 2021 · 1 comment

Comments

@Byron
Copy link
Owner

Byron commented Dec 20, 2021 •
edited

MultiPack indices are saving resources and speed up object access, they are less common locally but great on the server.

Tasks

  • multi_index::File as a way to read the file format
  • multi-index verification #282
  • multi-index writing
  • integration into git_odb::Store
  • assure packs can be built when a multi-pack index is present
  • optional light-weight verification similar to what git does
@Byron Byron created this issue from a note in Collaboration Board (In progress) Dec 20, 2021
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
update crate status with more details about midx (#279)
439fed8
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
frame for instantiation of multi-pack-index (#279)
5e085ec
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
only load multi-pack indices if allowed (#279)
b22e146
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
Respect core.multiPackIndex option (#279)
1495efc
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
refactor (#279)
4e89d8d
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
feat: Kind::len_in_bytes() method (#279)
9a0d8b8 
It yields the amount of bytes needed to store the hash.
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
First pieces of header parsing; allow to respect multi-index desired …
1a2a049 
…hash kind in git-odb (#279)
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
basic midx header parsing (#279)
edf02ae
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
frame for git-chunk crate to share among git-pack and git-commitgraph (…
b2d2ae2 
…#279)
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
Load chunk index of midx file (#279)
fac8efa
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
Read all mandatory and optional chunks (#279)
99023bb
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
read and validate fanout chunk (#279)
3ca04e3
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
read and validate index names contained in the multi-pack index (#279)
24a9790
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
completely validate and parse multi-index file (#279)
e7e40c3
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
remove empty tests (#279)
e30dcea
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
update changelog prior to release (#279)
6ae49e3
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
remove unnecessary test dependencies (#279)
463afcc
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
refactor (#279)
7a9e628
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
feat: new file::Index::highest_offset() method (#279)
d0fab1e 
With it it's simpler to figure out from where to read trailing
checksums.
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
Provide multi-index checksum (#279)
a363de9
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
feat!: Use [u8;4] as chunk id (#279)
9d9f2ee 
This allows to remove the additional string to describe the ids, which
are usually ascii anyway.
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
Adapt to latest changes to git-chunk (#279)
743d696
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
refactor (#279)
ac46765
Byron added a commit that referenced this issue Dec 20, 2021
@Byron
update changelog (#279)
099f055
Byron added a commit that referenced this issue Dec 21, 2021
@Byron
change!: rename Kind to Id (#279)
dda26a4 
This is more in line with gits terminology and feels quite natural to
use as well.
Byron added a commit that referenced this issue Dec 21, 2021
@Byron
Adapt to changes in git-chunk (#279)
44ea5c3
Byron added a commit that referenced this issue Dec 21, 2021
@Byron
change!: rename into_usize_range() to range::into_usize() (#279)
9328015
Byron added a commit that referenced this issue Dec 21, 2021
@Byron
First round of introducing git-chunk (#279)
51b991b
Byron added a commit that referenced this issue Dec 21, 2021
@Byron
feat: add file::Index::usize_offset_by_id() and `range::into_usize_…
e14096e 
…or_panic()` (#279)
Byron added a commit that referenced this issue Dec 31, 2021
@Byron
Writing of chunk index (#279)
17a93c3
Byron added a commit that referenced this issue Dec 31, 2021
@Byron
Write multi-index header along with path-names chunk (#279)
2fc6751
Byron added a commit that referenced this issue Dec 31, 2021
@Byron
fix docs (#279)
b61a920
Byron added a commit that referenced this issue Dec 31, 2021
@Byron
refactor (#279)
93dc660
Byron added a commit that referenced this issue Dec 31, 2021
@Byron
Also consider the size of the fanout table as part of the min size (#279
8190708 
)
Byron added a commit that referenced this issue Dec 31, 2021
@Byron
Write the fanout table (#279)
6a68ed7
Byron added a commit that referenced this issue Dec 31, 2021
@Byron
Add chunk for oids (#279)
565a7ae
Byron added a commit that referenced this issue Dec 31, 2021
@Byron
write pack-ids and offsets (#279)
bfc8069
Byron added a commit that referenced this issue Dec 31, 2021
@Byron
multi-pack index writing complete with large-offset support (#279)
f7d5c7f
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
feat: Facilities to write chunk files (#279)
389fea2 
This includes utilities to plan chunks for writing the table of
contents, and to get a hand when actually writing the chunks themselves
while assuring they are written into the correct spot.
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
progress for chunk writing (#279)
50fde01
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
change!: renamed progress::Read::reader -> progress::Read::inner (#…
61e5cfe 
…279)
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
adapt to changes in git-features (#279)
542c0df
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
feat: Add progress::Write to automatically pass bytes written to a …
0a749a2 
…progress instance (#279)
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
write progress for multi-pack writing (#279)
1bea1d4
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
add missing docs (#279)
4137327
@Byron
Copy link
Owner Author

Byron commented Jan 1, 2022

@avoidscorn I thought I'd share with you some improvements made to the handling of chunk files. Now there is a git-chunk crate which facilitates reading and writing of chunk files. The facilities for reading are already used by git-commitgraph, while the ones for writing are demoed when writing multi-pack indices. In case commitgraph writing of of interest to you, I highly recommend taking a look at git-chunk as it should make that easier. In any case, have a great new year 2022.

Byron added a commit that referenced this issue Jan 1, 2022
@Byron
'index' with its own sub-commands (#279)
c4c5678 
Cleanup to prepare for multi-index
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
Basic multi-pack index creation (#279)
89428b2 
Could imagine providing statistics as part of the creation result.
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
Fix progress and handling of large of multi-pack index offsets (#279)
5dc1f81
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
Handle large multi-pack indices correctly (#279)
4f6b030
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
update stress test to create multi-pack index with large offsets and …
4b6794a 
…verify it (#279)
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
better multi-pack verification progress (#279)
2e16f13
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
also test pack-creation with multi-index repo (#279)
235a27a
@Byron Byron closed this as completed Jan 1, 2022
Collaboration Board automation moved this from In progress to Done Jan 1, 2022
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
Allow interrupting multi-index creation more often (#279)
f223ecb
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
refactor (#279)
91e6d38
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
Add a less thorough and faster way of verifying multi-indices (#279)
7517482
Byron added a commit that referenced this issue Jan 1, 2022
@Byron
Fast-path multi-pack index verification in the CLI (#279)
bcde935
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Byron