-
Notifications
You must be signed in to change notification settings - Fork 89
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add database interface for blob paths + DynamoDB integration + refactor BoltDB to use interface #588
Conversation
af1d119
to
f0e76e7
Compare
why so? |
@@ -29,7 +29,7 @@ jobs: | |||
s3mock: | |||
image: ghcr.io/project-zot/localstack/localstack:0.13.2 | |||
env: | |||
SERVICES: s3 | |||
SERVICES: s3,dynamodb |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This makes us a user of another aws service? :\
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes..I propose to keep backing up boltdb in s3, and locally we just download it to a temp dir, use it, and keep backing it up?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this will work for one zot but not for multiple, for multiple we should do the same for all GETs but that will add a lot of overhead as GETs are very frequent....
0016f01
to
3aaa919
Compare
Codecov Report
@@ Coverage Diff @@
## main #588 +/- ##
==========================================
- Coverage 88.05% 87.18% -0.88%
==========================================
Files 61 62 +1
Lines 11630 11712 +82
==========================================
- Hits 10241 10211 -30
- Misses 1090 1190 +100
- Partials 299 311 +12
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
0d8300d
to
c575305
Compare
What happens when we interface with GCP instead? is there a dynamoDB equivalent for GCP? |
5f4432a
to
30eacda
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
approved by mistake
cde8aaa
to
2c338a8
Compare
@@ -1032,7 +1041,7 @@ func (is *ImageStoreLocal) FinishBlobUpload(repo, uuid string, body io.Reader, d | |||
|
|||
dst := is.BlobPath(repo, dstDigest) | |||
|
|||
if is.dedupe && is.cache != nil { | |||
if is.dedupe && fmt.Sprintf("%v", is.cache) != fmt.Sprintf("%v", nil) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what if we use a switch where we use type check for each implementation we know we have, now we only have dynamodatabase and boltdb, if it's not one of these types assume cache is nil
later when adding more databases we add checks for them.
Thoughts?
d40fa6e
to
42f322a
Compare
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Added proper endpoint handling, logging, tests Implemented more functions, added more tests, updated errors Added aws databases documentation Added DB driver and factories, need to implement Implemented Factory and Driver interfaces for dynamodb Refactor cache.go and changed some stuff Rewrote some stuff, TODO: fix PutBlob Fixed dynamoDB functions Refactor boltDB to use new interface Fixed gotcha Lint Point S3 to use DynamoDB database instead of boltdb Reverted s3 use only boltdb Added more dynamodb tests, fixed errors lint Added database factory tests, removed Scan and DescribeTable functions Added loading of db parameters from either config or env Renamed packages and moved files around to avoid cyclic import Added more coverage tests Removed env var usage for config Added benchmark tests Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Closed as I split this PR into two: #667 and chofnar#18
What type of PR is this?
enhancement
Which issue does this PR fix:
#564
What does this PR do / Why do we need it:
This PR adds a database interface that allows the usage of vendor-specific database services by implementing their drivers.
The usage of BoltDB was refactored to use this new interface.
Added AWS DynamoDB driver.
Will this break upgrades or downgrades?
If a client uses DynamoDB and decides to downgrade, they must make sure that the database is migrated to BoltDB to avoid data loss.
Does this PR introduce any user-facing change?:
Adds the possibility to use AWS DynamoDB instead of BoltDB
Problems
Right now, there are several problems with this PR:
Other comments
Inspiration from https://github.com/distribution/distribution
To do
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.