Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature branch: storage & sync revamp #446

Merged
merged 85 commits into from Jan 27, 2023
Merged

Feature branch: storage & sync revamp #446

merged 85 commits into from Jan 27, 2023

Conversation

@dholms dholms changed the title Repo storage & Sync revamp Feature branch: storage & sync revamp Jan 6, 2023
@dholms dholms marked this pull request as ready for review January 18, 2023 23:27
}
},
"output": {
"encoding": "application/cbor"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any thoughts on whether we should consider application/vnd.ipld.car for this? Popped into my head, looking at these re: mimes of blocks and CARs: ipfs/in-web-browsers#192 ipfs/kubo#8758 https://www.iana.org/assignments/media-types/application/vnd.ipld.car .

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup great point

fixed here: #485

@dholms dholms mentioned this pull request Jan 24, 2023
dholms and others added 8 commits January 24, 2023 16:26
* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts
* return suggestions by post count

* pr feedback

* fix up PG pagination issue
Partition commit-history & commit-blocks by user did
@dholms dholms merged commit 75e14ae into main Jan 27, 2023
@dholms dholms deleted the sync-revamp branch January 27, 2023 00:09
@dholms dholms mentioned this pull request Jan 27, 2023
mloar pushed a commit to mloar/atproto that referenced this pull request Nov 15, 2023
* wip

* fleshing out repo storage

* fleshing out sql storage

* cleaning things up

* fix up tests

* dumb bug - commit log reversed

* rm staging in favor of commiting diffs to blockstore

* clean up benches

* fixing up sql storage

* some caching for sql repo store

* pr feedback

* migration

* wip

* migraiton test

* unclear param

* sql repo storage tests

* rm unused code

* fix up some diff code

* pr feedback

* enum for action types

* missed some

* wip

* ripping out auth lib

* more auth cleanup

* another lurker

* wip better sync primitives

* wip

* improving diffs & sync

* tests working!

* actually implemented checkout lol

* simplify interface & improve error handling

* writing sql storage code

* fixing up tests

* testing & bugfixes

* checkouts return records instead of cids

* one last refactor lol

* missed one

* handle other cid codecs on incoming car verification

* tests + tricky bugs

* unneeded blockstore method

* trim mst on del instead of save

* cleanup comment

* dont resolve did for every commit

* use "commit" instead of "root"

* getRoot -> getHead

* pr feedback

* very silly bug fix

* improve sync output

* reorging + sync of particular records

* serve & verify proofs. also rename some ipld methods

* fix up sync issue in mst

* find reachable records form carfile

* getRecord xrpc method

* pr feedback

* better migration test

* check migraiton result

* fixing up a couple things for pg

* explicit migrateTo

* async exceptions

* ipld car mimetype + remove updateRepo

* Update module publish scripts (bluesky-social#478)

* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts

* Sort "suggested follows" by number of posts (bluesky-social#477)

* return suggestions by post count

* pr feedback

* fix up PG pagination issue

* partiion commit-history & commit-blocks by user did

* fix migration type

* switch pk indexes

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants