Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(auto-rules): PeriodicArchiver scans archives on startup #551

Merged
merged 82 commits into from
Aug 11, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
3ef6270
Retrieve list of recordings saved in archive
Jul 5, 2021
20681a8
Start parsing archived recordings JSON
Jul 6, 2021
d8c8377
Refactor previous recordings extraction to be outside of the constructor
Jul 6, 2021
5440f4a
Iterate through archive to get previous recordings for the current rule
Jul 6, 2021
ffd5040
Extract archived recordings listing into refactored chunk
Jul 7, 2021
20f840b
Clean up unneeded imports and class variables
Jul 8, 2021
0ee3e0f
Start iteration of archived recordings
Jul 8, 2021
33586e2
Continue iteration of archived recordings
Jul 8, 2021
fa1de8d
Begin archive directory restructuring
Jul 8, 2021
429af70
Implement get archive listing using Pattern matching
Jul 9, 2021
1ea75f7
Begin refactoring related files to accommodate archive directory change
Jul 9, 2021
19acec9
Implement nested directory structure using Path instead of String
Jul 12, 2021
be17e5c
Update archived recording(s) GET handlers
Jul 12, 2021
8e32a76
Create new class to represent archived recording information
Jul 14, 2021
085812e
Update recording archival and GET handlers to work with new class
Jul 14, 2021
ac3aa92
Fix compilation errors related to exception handling and class instan…
Jul 14, 2021
8f64c8c
Remove unnecessary Future creation/handling in PeriodicArchiver
Jul 14, 2021
5917da2
fixup! Remove unnecessary Future creation/handling in PeriodicArchiver
Jul 14, 2021
2b76d01
Update archived recordings GET and DELETE handlers to work with direc…
Jul 14, 2021
49d1a07
Update archived recordings POST handlers to work with directory restr…
Jul 15, 2021
2eec9cc
Begin updating unit tests
Jul 15, 2021
ee5d527
Fix test fields and comment out certain tests to allow debugging
Jul 15, 2021
39ea44e
Update archived recordings GET handler test
Jul 15, 2021
ae6b137
Extract ArchivedRecordingInfo custom JSON serializer into its own cla…
Jul 15, 2021
343e8d7
Update old PeriodicArchiver unit tests to better reflect its class st…
Jul 15, 2021
d15b01c
fixup! Extract ArchivedRecordingInfo custom JSON serializer into its …
Jul 20, 2021
0651f44
Change the RecordingArchiveHelper implementation to be Future-based a…
Jul 20, 2021
7efb192
Update RecordingArchiveHelperTest
Jul 21, 2021
69c5e6c
Update RecordingsPostHandlerTest
Jul 21, 2021
e2ccc0b
Fix broken unit tests
Jul 22, 2021
6145d14
Update PeriodicArchiverTest to include archive scanning test
Jul 22, 2021
0a66b3f
Test JSON serialization of archived recordings
Jul 22, 2021
a1a8c2b
Decouple target recording deletion from the RecordingArchiveHelper
Jul 22, 2021
451f2fd
Extract the correct archived recording deletion code from the Recordi…
Jul 22, 2021
2908c7f
Run mvn spotless:apply
Jul 27, 2021
eea762d
fixup! Run mvn spotless:apply
Jul 27, 2021
d504b63
Fix IOException due to not creating the encoded service URI directory
Jul 27, 2021
cda9d4f
Fix unit test broken by addition of encoded service URI directory
Jul 28, 2021
126a475
Update archived recordings GET implementation and testing to reflect …
Jul 28, 2021
7bd7694
Update archived recordings DELETE implementation to reflect proper di…
Jul 28, 2021
0f1881b
Update archived recordings DELETE testing to reflect proper directory…
Jul 29, 2021
327365e
Extract single recording GET into the RecordingArchiveHelper and upda…
Jul 29, 2021
ef674fd
Update RecordingWorkflowIT to verify target and saved recordings dele…
Jul 29, 2021
7e587a1
Run mvn spotless:apply
Jul 29, 2021
7e8b814
Remove reliance on default encoding
Jul 29, 2021
bf59b83
Remove unnecessary saved recordings path from archived recording GET …
Jul 29, 2021
6ca47d5
Update archived recordings POST handler to reflect proper directory h…
Jul 30, 2021
81d76c9
Update archived recordings POST handler testing to reflect proper dir…
Aug 2, 2021
36fe483
Update archived recordings UPLOAD POST handler implementation and tes…
Aug 2, 2021
d414de9
fixup! Update archived recordings UPLOAD POST() handler implementatio…
Aug 2, 2021
7f75d99
Run mvn spotless:apply
Aug 2, 2021
cbd2a0b
Remove unnecessary IOException handling
Aug 2, 2021
0dbd45c
Update archived recording GET testing
Aug 2, 2021
c618ed1
Add missing status code field to archived recordings GET handler
Aug 2, 2021
2db2999
Continue removing unnecessary IOException handling
Aug 2, 2021
643a7cc
Remove deprecated import
Aug 3, 2021
6d9acdd
Fix unit test errors due to addition of default HTTP content type
Aug 3, 2021
8f2643f
Fix incorrect CompletableFuture.get() error handling
Aug 3, 2021
a4ed433
Run mvn spotless:apply
Aug 3, 2021
56f2921
fixup! Fix incorrect CompletableFuture.get() error handling
Aug 3, 2021
96d187a
Remove unnecessary status code setting and extract PATCH/DELETE notif…
Aug 3, 2021
204a586
Change Base32 instances from local to class level
Aug 3, 2021
491b991
Normalize and convert to absolute path beforehand; refactor recording…
Aug 3, 2021
f5f510c
fixup! Normalize and convert to absolute path beforehand; refactor re…
Aug 4, 2021
ff0c60f
Delete deprecated saved recording descriptor class
Aug 4, 2021
255b9e2
Revert changes made to RecordingWorkflowIT that cause intermittent fa…
Aug 4, 2021
deff3c4
Replace iterator with for-each loop
Aug 4, 2021
ee2a671
Remove unnecessary custom JSON serialization
Aug 4, 2021
29f886b
Ensure handlers throw HttpStatusExceptions containing the most inform…
Aug 4, 2021
7bfb903
Encapsulate specific Exceptions inside Future(s) in the RecordingArch…
Aug 4, 2021
53338a9
Clean-up exception handling
Aug 4, 2021
a3ce1c2
Include deleted recording's path in returned Future
Aug 4, 2021
a043d13
Start fixing incorrect directory search during archived recording rep…
Aug 5, 2021
ab24e9a
Extract archived recording report deletion into the RecordingArchiveH…
Aug 5, 2021
10495d9
fixup! Clean-up exception handling
Aug 5, 2021
a240d50
Fix incorrect subdirectory creation in handler without access to targ…
Aug 6, 2021
0e4930a
Clean-up exception handling
Aug 9, 2021
c4c9aef
Remove unecessary HTTP response end()
Aug 9, 2021
cb7c0f5
Run mvn spotless:apply
Aug 9, 2021
60fefbf
fixup! Remove unecessary HTTP response end()
Aug 9, 2021
97572d4
Fix incorrect AuthManager mocking in unit tests
Aug 11, 2021
7a48fbf
Run mvn spotless:apply
Aug 11, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -47,105 +47,74 @@
import javax.inject.Named;
import javax.inject.Provider;

import io.cryostat.MainModule;
import io.cryostat.core.log.Logger;
import io.cryostat.core.sys.FileSystem;
import io.cryostat.net.reports.ReportService.RecordingNotFoundException;
import io.cryostat.net.web.WebModule;
import io.cryostat.net.web.http.generic.TimeoutHandler;
import io.cryostat.recordings.RecordingArchiveHelper;

class ArchivedRecordingReportCache {

protected final Path savedRecordingsPath;
protected final Path archivedRecordingsReportPath;
protected final FileSystem fs;
protected final Provider<SubprocessReportGenerator> subprocessReportGeneratorProvider;
protected final ReentrantLock generationLock;
protected final Logger logger;
protected final RecordingArchiveHelper recordingArchiveHelper;

ArchivedRecordingReportCache(
@Named(MainModule.RECORDINGS_PATH) Path savedRecordingsPath,
@Named(WebModule.WEBSERVER_TEMP_DIR_PATH) Path webServerTempPath,
FileSystem fs,
Provider<SubprocessReportGenerator> subprocessReportGeneratorProvider,
@Named(ReportsModule.REPORT_GENERATION_LOCK) ReentrantLock generationLock,
Logger logger) {
this.savedRecordingsPath = savedRecordingsPath;
this.archivedRecordingsReportPath = webServerTempPath;
Logger logger,
RecordingArchiveHelper recordingArchiveHelper) {
this.fs = fs;
this.subprocessReportGeneratorProvider = subprocessReportGeneratorProvider;
this.generationLock = generationLock;
this.logger = logger;
this.recordingArchiveHelper = recordingArchiveHelper;
}

Future<Path> get(String recordingName) {
CompletableFuture<Path> f = new CompletableFuture<>();
Path dest = getCachedReportPath(recordingName);
Path dest = recordingArchiveHelper.getCachedReportPath(recordingName);
if (fs.isReadable(dest) && fs.isRegularFile(dest)) {
f.complete(dest);
return f;
}

try {
generationLock.lock();
// check again in case the previous lock holder already created the cached file
if (fs.isReadable(dest) && fs.isRegularFile(dest)) {
f.complete(dest);
return f;
}
logger.trace("Archived report cache miss for {}", recordingName);

fs.listDirectoryChildren(savedRecordingsPath).stream()
.filter(name -> name.equals(recordingName))
.map(savedRecordingsPath::resolve)
.findFirst()
.ifPresentOrElse(
recording -> {
logger.trace("Archived report cache miss for {}", recordingName);
try {
Path saveFile =
subprocessReportGeneratorProvider
.get()
.exec(
recording,
dest,
Duration.ofMillis(
TimeoutHandler.TIMEOUT_MS))
.get();
f.complete(saveFile);
} catch (Exception e) {
logger.error(e);
f.completeExceptionally(e);
try {
fs.deleteIfExists(dest);
} catch (IOException ioe) {
logger.warn(ioe);
}
}
},
() ->
f.completeExceptionally(
new RecordingNotFoundException(
"archives", recordingName)));
} catch (IOException ioe) {
logger.warn(ioe);
f.completeExceptionally(ioe);
Path archivedRecording = recordingArchiveHelper.getRecordingPath(recordingName).get();
Path saveFile =
subprocessReportGeneratorProvider
.get()
.exec(
archivedRecording,
dest,
Duration.ofMillis(TimeoutHandler.TIMEOUT_MS))
.get();
f.complete(saveFile);
} catch (Exception e) {
logger.error(e);
f.completeExceptionally(e);
try {
fs.deleteIfExists(dest);
} catch (IOException ioe) {
logger.warn(ioe);
}
} finally {
generationLock.unlock();
}
return f;
}

boolean delete(String recordingName) {
try {
logger.trace("Invalidating archived report cache for {}", recordingName);
return fs.deleteIfExists(getCachedReportPath(recordingName));
} catch (IOException ioe) {
logger.warn(ioe);
return false;
}
}

protected Path getCachedReportPath(String recordingName) {
String fileName = recordingName + ".report.html";
return archivedRecordingsReportPath.resolve(fileName).toAbsolutePath();
return recordingArchiveHelper.deleteReport(recordingName);
}
}
13 changes: 0 additions & 13 deletions src/main/java/io/cryostat/net/reports/ReportService.java
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,6 @@

import io.cryostat.net.ConnectionDescriptor;

import org.apache.commons.lang3.tuple.Pair;

public class ReportService {

private final ActiveRecordingReportCache activeCache;
Expand All @@ -70,15 +68,4 @@ public Future<String> get(ConnectionDescriptor connectionDescriptor, String reco
public boolean delete(ConnectionDescriptor connectionDescriptor, String recordingName) {
return activeCache.delete(connectionDescriptor, recordingName);
}

// FIXME This is basically duplicated from UploadRecordingCommand
public static class RecordingNotFoundException extends RuntimeException {
public RecordingNotFoundException(String targetId, String recordingName) {
super(String.format("Recording %s not found in target %s", recordingName, targetId));
}

public RecordingNotFoundException(Pair<String, String> key) {
this(key.getLeft(), key.getRight());
}
}
}
13 changes: 5 additions & 8 deletions src/main/java/io/cryostat/net/reports/ReportsModule.java
Original file line number Diff line number Diff line change
Expand Up @@ -47,13 +47,12 @@
import javax.inject.Provider;
import javax.inject.Singleton;

import io.cryostat.MainModule;
import io.cryostat.core.log.Logger;
import io.cryostat.core.reports.ReportTransformer;
import io.cryostat.core.sys.Environment;
import io.cryostat.core.sys.FileSystem;
import io.cryostat.net.TargetConnectionManager;
import io.cryostat.net.web.WebModule;
import io.cryostat.recordings.RecordingArchiveHelper;
import io.cryostat.util.JavaProcess;

import dagger.Module;
Expand Down Expand Up @@ -126,19 +125,17 @@ static SubprocessReportGenerator provideSubprocessReportGenerator(
@Provides
@Singleton
static ArchivedRecordingReportCache provideArchivedRecordingReportCache(
@Named(MainModule.RECORDINGS_PATH) Path savedRecordingsPath,
@Named(WebModule.WEBSERVER_TEMP_DIR_PATH) Path webServerTempDir,
FileSystem fs,
Provider<SubprocessReportGenerator> subprocessReportGeneratorProvider,
@Named(REPORT_GENERATION_LOCK) ReentrantLock generationLock,
Logger logger) {
Logger logger,
RecordingArchiveHelper recordingArchiveHelper) {
return new ArchivedRecordingReportCache(
savedRecordingsPath,
webServerTempDir,
fs,
subprocessReportGeneratorProvider,
generationLock,
logger);
logger,
recordingArchiveHelper);
}

@Provides
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@
import io.cryostat.core.sys.FileSystem;
import io.cryostat.net.ConnectionDescriptor;
import io.cryostat.net.TargetConnectionManager;
import io.cryostat.net.reports.ReportService.RecordingNotFoundException;
import io.cryostat.recordings.RecordingNotFoundException;
import io.cryostat.util.JavaProcess;

import org.apache.commons.lang3.builder.EqualsBuilder;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,6 @@
*/
package io.cryostat.net.web.http.api.v1;

import io.cryostat.messaging.notifications.NotificationFactory;
import io.cryostat.net.TargetConnectionManager;
import io.cryostat.net.web.http.RequestHandler;
import io.cryostat.recordings.RecordingArchiveHelper;
Expand All @@ -46,6 +45,7 @@
import dagger.Module;
import dagger.Provides;
import dagger.multibindings.IntoSet;
import org.apache.commons.codec.binary.Base32;

@Module
public abstract class HttpApiV1Module {
Expand Down Expand Up @@ -92,9 +92,8 @@ abstract RequestHandler bindTargetRecordingPatchBodyHandler(

@Provides
static TargetRecordingPatchSave provideTargetRecordingPatchSave(
RecordingArchiveHelper recordingArchiveHelper,
NotificationFactory notificationFactory) {
return new TargetRecordingPatchSave(recordingArchiveHelper, notificationFactory);
RecordingArchiveHelper recordingArchiveHelper) {
return new TargetRecordingPatchSave(recordingArchiveHelper);
}

@Provides
Expand Down Expand Up @@ -135,6 +134,11 @@ static TargetRecordingPatchStop provideTargetRecordingPatchStop(
@IntoSet
abstract RequestHandler bindRecordingsPostHandler(RecordingsPostHandler handler);

@Provides
static Base32 provideBase32() {
return new Base32();
}

@Binds
@IntoSet
abstract RequestHandler bindTargetsGetHandler(TargetsGetHandler handler);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,49 +37,32 @@
*/
package io.cryostat.net.web.http.api.v1;

import java.io.IOException;
import java.nio.file.Path;
import java.util.EnumSet;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ExecutionException;

import javax.inject.Inject;
import javax.inject.Named;

import io.cryostat.MainModule;
import io.cryostat.core.sys.FileSystem;
import io.cryostat.messaging.notifications.NotificationFactory;
import io.cryostat.net.AuthManager;
import io.cryostat.net.reports.ReportService;
import io.cryostat.net.security.ResourceAction;
import io.cryostat.net.web.http.AbstractAuthenticatedRequestHandler;
import io.cryostat.net.web.http.HttpMimeType;
import io.cryostat.net.web.http.api.ApiVersion;
import io.cryostat.recordings.RecordingArchiveHelper;
import io.cryostat.recordings.RecordingNotFoundException;

import io.vertx.core.http.HttpMethod;
import io.vertx.ext.web.RoutingContext;
import io.vertx.ext.web.handler.impl.HttpStatusException;
import org.apache.commons.lang3.exception.ExceptionUtils;

public class RecordingDeleteHandler extends AbstractAuthenticatedRequestHandler {

private final ReportService reportService;
private final FileSystem fs;
private final Path savedRecordingsPath;
private final NotificationFactory notificationFactory;
private static final String NOTIFICATION_CATEGORY = "RecordingDeleted";
private final RecordingArchiveHelper recordingArchiveHelper;

@Inject
RecordingDeleteHandler(
AuthManager auth,
ReportService reportService,
FileSystem fs,
NotificationFactory notificationFactory,
@Named(MainModule.RECORDINGS_PATH) Path savedRecordingsPath) {
RecordingDeleteHandler(AuthManager auth, RecordingArchiveHelper recordingArchiveHelper) {
super(auth);
this.reportService = reportService;
this.fs = fs;
this.savedRecordingsPath = savedRecordingsPath;
this.notificationFactory = notificationFactory;
this.recordingArchiveHelper = recordingArchiveHelper;
}

@Override
Expand Down Expand Up @@ -110,34 +93,14 @@ public boolean isAsync() {
@Override
public void handleAuthenticated(RoutingContext ctx) throws Exception {
String recordingName = ctx.pathParam("recordingName");
fs.listDirectoryChildren(savedRecordingsPath).stream()
.filter(saved -> saved.equals(recordingName))
.map(savedRecordingsPath::resolve)
.findFirst()
.ifPresentOrElse(
path -> {
try {
if (!fs.exists(path)) {
throw new HttpStatusException(404, recordingName);
}
fs.deleteIfExists(path);
notificationFactory
.createBuilder()
.metaCategory(NOTIFICATION_CATEGORY)
.metaType(HttpMimeType.JSON)
.message(Map.of("recording", recordingName))
.build()
.send();
} catch (IOException e) {
throw new HttpStatusException(500, e.getMessage(), e);
} finally {
reportService.delete(recordingName);
}
ctx.response().setStatusCode(200);
ctx.response().end();
},
() -> {
throw new HttpStatusException(404, recordingName);
});
try {
recordingArchiveHelper.deleteRecording(recordingName).get();
ctx.response().end();
} catch (ExecutionException e) {
if (ExceptionUtils.getRootCause(e) instanceof RecordingNotFoundException) {
throw new HttpStatusException(404, e.getMessage(), e);
}
throw e;
}
}
}