Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build 'discovery' ODB with auto-consistency for on-disk files #266

Closed
5 tasks done
Byron opened this issue Nov 29, 2021 · 0 comments
Closed
5 tasks done

build 'discovery' ODB with auto-consistency for on-disk files #266

Byron opened this issue Nov 29, 2021 · 0 comments

Comments

@Byron
Copy link
Owner

Byron commented Nov 29, 2021 •
edited

The DB spiked in the discovery has all strengths and seemingly no weaknesses (except for complexity). Let's build it to be able to automatically adapt to changes on disk and to handle server loads efficiently.

The radicle-link ODB shows how a fully sync ODB can look like, and maybe the initial design can be simplified now that it's clear that fully Sync is very fast, too.

Or said differently, it appears that thread-local caches can be used but they probably only make sense for object buffers and pack-caches, less for accelerating object access.

Ideally, a quick-version can be drafted to have benchmark results quickly to even see if the path is right.

What I wish for is more dynamic handling of open file handles, ideally there is a trigger to release unused ones, and generally have a way of handling file limits somewhat gracefully (at least on server/caller level).

Tasks

  • Refactor Find traits #267
  • PoC of new ODB design #273
  • docs
  • make the above production ready
    • tests for parallel pack creation with changes made to the disk state while a pack is being created
    • auto-refresh of disk-state when packs are merged, new packs are added.
  • delete odb-design experiment
@Byron Byron created this issue from a note in Collaboration Board (In progress) Nov 29, 2021
Byron added a commit that referenced this issue Nov 30, 2021
@Byron
Add 'contains()' method to Find (#266)
dd386ce 
The plan is to separate pack location entirely from Object and put
the location specific functions into a separate trait.
Byron added a commit that referenced this issue Nov 30, 2021
@Byron
feat: add Data object (#266)
e8f1912 
This is typed data baked by a slice for conversion into parsed ObjectRef's
for example.

This is usually the result of a `Find` operation on an object database.
Byron added a commit that referenced this issue Nov 30, 2021
@Byron
feat: A simplified version of the Find trait (#266)
f788310 
It's meant for the next generation of object db handles which keep a
local cache of all the details of the actual object database.
Byron added a commit that referenced this issue Nov 30, 2021
@Byron
refactor!: move git_pack::data::Object to git_object::Data, massively…
e22a710 
… alter git_odb::Find trait (#266)

This will break a lot, but has to happen to prepare these traits for the
next generation of object databases.
Byron added a commit that referenced this issue Dec 1, 2021
@Byron
Add 'contains()' method to Find (#266)
532e9e0 
The plan is to separate pack location entirely from Object and put
the location specific functions into a separate trait.
Byron added a commit that referenced this issue Dec 1, 2021
@Byron
feat: add Data object (#266)
004120f 
This is typed data baked by a slice for conversion into parsed ObjectRef's
for example.

This is usually the result of a `Find` operation on an object database.
Byron added a commit that referenced this issue Dec 1, 2021
@Byron
feat: A simplified version of the Find trait (#266)
c2ded9d 
It's meant for the next generation of object db handles which keep a
local cache of all the details of the actual object database.
Byron added a commit that referenced this issue Dec 1, 2021
@Byron
refactor!: move git_pack::data::Object to git_object::Data, massively…
ccabe2f 
… alter git_odb::Find trait (#266)

This will break a lot, but has to happen to prepare these traits for the
next generation of object databases.
Byron added a commit that referenced this issue Dec 1, 2021
@Byron
fix docs (#266)
83e1845
Byron added a commit that referenced this issue Dec 1, 2021
@Byron
feat: linked::Store sorts bundles by modification date, newest first (#…
b54a985 
…266)
Byron added a commit that referenced this issue Dec 1, 2021
@Byron
Clarify that we really need stable pack ids (#266)
fab5fe7
Byron added a commit that referenced this issue Dec 1, 2021
@Byron
refactor!: remove pack-cache from Find::try_find(…) (#266)
3bd9261 
With the new architecture this can be an implementation detail without
forcing it to be Sync.
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
Add 'contains()' method to Find (#266)
cc9df78 
The plan is to separate pack location entirely from Object and put
the location specific functions into a separate trait.
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
feat: add Data object (#266)
adfbcee 
This is typed data baked by a slice for conversion into parsed ObjectRef's
for example.

This is usually the result of a `Find` operation on an object database.
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
feat: A simplified version of the Find trait (#266)
fcc1c31 
It's meant for the next generation of object db handles which keep a
local cache of all the details of the actual object database.
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
refactor!: move git_pack::data::Object to git_object::Data, massively…
8f6ff71 
… alter git_odb::Find trait (#266)

This will break a lot, but has to happen to prepare these traits for the
next generation of object databases.
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
fix docs (#266)
b557550
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
feat: linked::Store sorts bundles by modification date, newest first (#…
be124ce 
…266)
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
Clarify that we really need stable pack ids (#266)
108c77b
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
refactor!: remove pack-cache from Find::try_find(…) (#266)
87930a1 
With the new architecture this can be an implementation detail without
forcing it to be Sync.
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
Inform when multi-threaded counting is rejected (#266)
bed0e92
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
Use Deref instead of Borrow in linked ODB iterator (#266)
0f8a9d1 
Borrow is the manual form, Deref has allows for more automatic use
and more idiomatic looking code.
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
feat: add linked::Store::rc_iter() (#266)
565b7d9 
For completeness in case of single-threaded operations
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
git-odb::Find implementation for linked::Store (#266)
11a4550
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
refactor (#266)
e357ba4
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
refactor (#266)
f43d985
Byron added a commit that referenced this issue Dec 2, 2021
@Byron
A first tiny stab at Handle for linked store (#266)
f47b175 
It could easily be general over all kinds of store as long as there
is support for pack-caches, which might be helpful later for the 'final'
store type.

Ideally, this one now shows how to do it.
Byron added a commit that referenced this issue Dec 3, 2021
@Byron
Add 'contains()' method to Find (#266)
dfdd6fb 
The plan is to separate pack location entirely from Object and put
the location specific functions into a separate trait.
Byron added a commit that referenced this issue Dec 3, 2021
@Byron
feat: add Data object (#266)
a0bb652 
This is typed data baked by a slice for conversion into parsed ObjectRef's
for example.

This is usually the result of a `Find` operation on an object database.
Byron added a commit that referenced this issue Dec 18, 2021
@Byron
Make single-threaded programs possible to use with git-repository (#266)
dde5c6b 
It's a bit tricky to use the right kind of handle and transform the
Rc<Store> back into an Arc<Store>, but it works.
Byron added a commit that referenced this issue Dec 18, 2021
@Byron
fix docs (#266)
360bf9d
Byron added a commit that referenced this issue Dec 18, 2021
@Byron
minor improvements to module layout, docs (#266)
0364f48
Byron added a commit that referenced this issue Dec 18, 2021
@Byron
change!: move loose::iter::Iter to loose::Iter (#266)
8bb5c9a
Byron added a commit that referenced this issue Dec 18, 2021
@Byron
adapt to changes in git-odb (#266)
a44dd4b
Byron added a commit that referenced this issue Dec 18, 2021
@Byron
dynamic store module cleanu (#266)
494772c
Byron added a commit that referenced this issue Dec 18, 2021
@Byron
change!: move sink::Sink to the top-level exclusively (#266)
ab4e726
Byron added a commit that referenced this issue Dec 18, 2021
@Byron
refactor (#266)
3da91ce
Byron added a commit that referenced this issue Dec 18, 2021
@Byron
refactor (#266)
52a4dcd
Byron added a commit that referenced this issue Dec 18, 2021
@Byron
refactor (#266)
b88f253
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
chore: update sha-1 dependency to 0.10 (#266)
361892c
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
upgrade git-ref's os_str_bytes crate to 6.0.0 (#266)
0cfba57
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
upgrade dashmap to latest version (#266)
52d4fe5
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
chore: upgrade all dependencies (#266)
a3caf39
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
chore: upgrade dependencies (#266)
322b290
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
chore: remove unused dependencies (#266)
c800fdd
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
upgrade dependencies (#266)
8adf0d8
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
upgrade dependencies (#266)
c301abe
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
Default handle refresh mode is the least surprising, with option to c…
1b74c14 
…onfigure (#266)
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
Revert "chore: upgrade all dependencies (#266)"
0dfe4a7 
This reverts commit a3caf39.
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
upgrade dependencies except for crates-index (#266)
c77c0d6 
The latter needs the notion of the index not existing, and shouldn't
fetch it by default or else each sanbox run fetches the entire index.
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
feat!: upgrade to crates-index 0.18 (#266)
15e60b2 
It now assumes that the crates-index must exist, which migth not always
be the case and rightfully so. Now we wrap it to get back to the
original behavior.
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
docs for dynamic object store (#266)
2c2a2e9
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
refactor (#266)
c499843
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
a failing test to show the handle-stability doesn't quite work yet (#266
5562e88 
)
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
assure stable handles can actually access the indices hey need (#266)
9474a43 
This works by bypassing the central index, which doesn't know about the
garbaged indices anymore, to obtain the indices directly.
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
More explicit information about how much garbaged is in the slotmap (#…
cfd36ee 
…266)
Byron added a commit that referenced this issue Dec 19, 2021
@Byron
delete now unused experiment (#266)
c44f72e
@Byron Byron closed this as completed Dec 19, 2021
Collaboration Board automation moved this from In progress to Done Dec 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Byron