HBASE-28568 Fix incremental backup set shrinking #5876
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir). The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink.
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
rmdmattingly
reviewed
May 7, 2024
|
@rmdmattingly added a test. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
rmdmattingly
reviewed
May 8, 2024
hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupDelete.java
Outdated
Show resolved
Hide resolved
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
The test failure looks relevant |
ndimiduk
approved these changes
May 10, 2024
| @@ -129,20 +128,15 @@ public int deleteBackups(String[] backupIds) throws IOException { | |||
| } | |||
| snapshotDone = true; | |||
There was a problem hiding this comment.
That snapshot logic looks problematic. Is it okay to continue using an existing snapshot? If an existing snapshot exists then we don't know what it contains and this operation should fail. Separate issue I guess?
There was a problem hiding this comment.
I agree it could be problematic. Perhaps the backup repair takes care of this kind of situations, I'm not sure.
But this is a separate issue indeed.
There was a problem hiding this comment.
FYI @rmdmattingly this is one to look into.
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupAdminImpl.java
Outdated
Show resolved
Hide resolved
| // Keep only the tables that are present in other backups | ||
| incrTableSet.retainAll(tableMap.keySet()); | ||
|
|
||
| table.deleteIncrementalBackupTableSet(backupRoot); |
There was a problem hiding this comment.
This API on BackupSystemTable is pretty ham-handed. Would be better if we could delete on the table entries that are being removed, rather than deleting and re-creating the row. I assume the lock taken via startBackupExclusiveOperation() at the beginning of deleteBackups prevents another actor from modifying that row concurrently...
|
Test failure looks like a failure of the backup system. It's attempting to take an incremental backup, which assumes the existence of a WAL that no longer exists. The test harness ran the unit test 3 times (once for each of our tested combinations of JDK version + hadoop version) and it failed once in three. I think the failure is not related to this patch but should be tracked as a flakey. @DieterDP-ng are you familiar with this test failure mode? Do we have this issue tracked in an existing ticket? |
|
I've encountered that failure while running tests (on the master branch) locally. I thought they always occurred for specific tests for me (because re-runs gave the same outcome), so I thought it had to do with my local setup. But just now I tried running all hbase-backup tests again so I could list the tests that fail for me, but now I only had a single failure for I don't know if there's an existing ticket for them, and haven't done any investigation for it. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
@DieterDP-ng I filed this as https://issues.apache.org/jira/browse/HBASE-28602 |
ndimiduk
pushed a commit
to ndimiduk/hbase
that referenced
this pull request
May 17, 2024
…5876) The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir). The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink. Reviewed-by: Ray Mattingly <rmdmattingly@gmail.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
to ndimiduk/hbase
that referenced
this pull request
May 17, 2024
…5876) The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir). The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink. Reviewed-by: Ray Mattingly <rmdmattingly@gmail.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
to ndimiduk/hbase
that referenced
this pull request
May 17, 2024
…5876) The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir). The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink. Reviewed-by: Ray Mattingly <rmdmattingly@gmail.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
to HubSpot/hbase
that referenced
this pull request
May 17, 2024
…5876) The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir). The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink. Reviewed-by: Ray Mattingly <rmdmattingly@gmail.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
to HubSpot/hbase
that referenced
this pull request
May 17, 2024
…5876) The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir). The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink. Reviewed-by: Ray Mattingly <rmdmattingly@gmail.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
rmdmattingly
pushed a commit
to HubSpot/hbase
that referenced
this pull request
May 17, 2024
…ly shrink (apache#5876) The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir). The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink. Reviewed-by: Ray Mattingly <rmdmattingly@gmail.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
rmdmattingly
pushed a commit
to HubSpot/hbase
that referenced
this pull request
May 17, 2024
…ly shrink (apache#5876) The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir). The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink. Reviewed-by: Ray Mattingly <rmdmattingly@gmail.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
reviewed
May 17, 2024
| @@ -30,6 +31,7 @@ | |||
| import org.apache.hadoop.hbase.testclassification.LargeTests; | |||
| import org.apache.hadoop.hbase.util.EnvironmentEdge; | |||
| import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; | |||
| import org.apache.hadoop.thirdparty.com.google.common.collect.Sets; | |||
There was a problem hiding this comment.
This should be using org.apache.hbase.thirdparty.com.google.common.collect.Sets, not the hadoop one. Backports to branch-2 fail because hadoop didn't have shaded jars in Hadoop 2.x
ndimiduk
pushed a commit
to ndimiduk/hbase
that referenced
this pull request
May 17, 2024
…5876) The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir). The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink. Reviewed-by: Ray Mattingly <rmdmattingly@gmail.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
to ndimiduk/hbase
that referenced
this pull request
May 17, 2024
…5876) The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir). The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink. Reviewed-by: Ray Mattingly <rmdmattingly@gmail.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
to ndimiduk/hbase
that referenced
this pull request
May 17, 2024
…5876) The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir). The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink. Reviewed-by: Ray Mattingly <rmdmattingly@gmail.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
that referenced
this pull request
May 18, 2024
The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir). The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink. Reviewed-by: Ray Mattingly <rmdmattingly@gmail.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
that referenced
this pull request
May 18, 2024
The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir). The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink. Reviewed-by: Ray Mattingly <rmdmattingly@gmail.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
that referenced
this pull request
May 18, 2024
The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir). The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink. Reviewed-by: Ray Mattingly <rmdmattingly@gmail.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The incremental backup set is the set of tables included when an incremental backup is created, it is managed per backup root dir and contains all tables that are present in at least one backup (in that root dir).
The incremental backup set can only shrink when backups are deleted. However, the implementation was incorrect, causing this set to never be able to shrink.