Skip to content

Caseflow eFolder Monitoring

cdetlefva edited this page Jun 7, 2023 · 11 revisions

Current Monitoring and Metrics

Raven/Sentry

Direct Raven/Sentry Calls

Where in Code Key ID Method Used Description
ExceptionLogger
.capture()
n/a Raven.capture_exception General use for logging exceptions in the application to Sentry
JobDataDogMetricMiddleware
.call()
job_class queue current_user.id current_user.email current_user.ip_address current_user.station_id Raven.capture_exception Logs errors to Sentry when an error occurs while attempting to send out a Datadog gauge message
VeteranFinder
.find_duplicate_bgs_rec()
bgs_rec_numbers Raven.capture_exception Logs an error when the BGS lookup number does not match any SSN or claim numbers

ExceptionLogger Calls

Where in Code Key ID Method Used Description
Api::V1::ApplicationController
rescue_from StandardError
n/a ExceptionLogger.capture Logs all StandardErrors that occur within the API controller
ManifestFetcher
.process()
n/a ExceptionLogger.capture Logs any exceptions that occur while processing Manifests

Datadog

Direct Datadog Calls

Where in Code Key ID Method Used Description
DataDogService
.increment_counter()
stat_name
tags
Datadog::Statsd.increment Increments the given stat counter in Datadog
DataDogService
.emit_gauge()
stat_name
metric_value
tags
Datadog::Statsd.gauge Handles sending metric data to Datadog

DataDogService Calls

Where in Code Key ID Method Used Description
ApplicationJob
.datadog_report_runtime()
metric_group_name DataDogService.record_runtime Records how long a job took to run within a given metric group
*never used
ApplicationJob
.datadog_report_time_segment()
segment DataDogService.emit_gauge Records how long a segment of code took to run
*never used
CollectDataDogMetrics
.emit_datadog_point()
type
db_name
DataDogService.emit_gauge Records a count of how many database connections are: active, dead, or idle
config.ru
.emit_datadog_point()
type DataDogService.emit_gauge Records the number of Puma worker threads that are: idle or active
JobDataDogMetricMiddleware
.call()
job_class DataDogService.emit_gauge Logging how long each job takes and tagging that job with a job_class.
MetricsService
.record()
service DataDogService.emit_gauge Records the latency of a given service when used in Production
MetricsService
.increment_datadog_counter()
metric_name
service
DataDogService.increment_counter Keeps a count of how many times a request for a service has been called, and how many times those requests have resulted in an error

CollectDataDogMetrics Calls

Where in Code Key ID Method Used Description
HealthChecksController:2 db_name include CollectDataDogMetrics Calls an module that has a callback method that sends monitoring data to Datadog

MetricsService Calls

Where in Code Key ID Method Used Description
ExternalApi::BGSService
.fetch_veteran_info()
file_number MetricsService.record Records the time it takes to fetch veteran info
ExternalApi::BGSService
.fetch_poa_by_file_number()
file_number MetricsService.record Records the time it takes to fetch a POA by file number
ExternalApi::BGSService
.fetch_poa_by_participant_id()
participant_id MetricsService.record Records the time it takes to fetch a POA by participant ID
ExternalApi::BGSService
.fetch_poa_org_record()
participant_id MetricsService.record Records the time it takes to fetch a POA Org Record by participant id
ExternalApi::BGSService
.fetch_poas_by_participant_ids()
participant_ids MetricsService.record Records the time it takes to fetch a list of POAs that are associated with the given participant IDs
ExternalApi::BGSService
.fetch_claims_for_file_number()
file_number MetricsService.record Records the time it takes to fetch a list of claims associated with a given file number
ExternalApi::BGSService
.fetch_person_info()
participant_id MetricsService.record Records the time it takes to fetch a data for a given participant
ExternalApi::BGSService
.fetch_person_by_ssn()
ssn MetricsService.record Records the time it takes to fetch a person for a given SSN
ExternalApi::BGSService
.check_sensitivity()
file_number MetricsService.record Records the time it takes to check whether or not a given file number can be accessed
ImageConverterService
.convert_tiff_to_pdf()
tiff_to_convert MetricsService.record Records the time it takes to convert a tiff to a pdf
ManifestFetcher
.documents_from_service_for()
file_number MetricsService.record Records the time it takes to fetch documents
RecordFetcher
.content_from_va_service()
record.file_number
record.manifest_source.name
MetricsService.record Records the time it takes to fetch a document file from a VA manifest source
RecordFetcher
.content_from_va_service()
record.s3_filename MetricsService.record Records the time it takes to convert an S3 image file
RecordFetcher
.content_from_va_service()
record.s3_filename MetricsService.record Records the time it takes to store an S3 file
RecordFetcher
.content_from_s3()
Record.s3_filename
record.file_number
MetricsService.record Records the time it takes to fetch an S3 file
ExternalApi::VBMSService
.send_and_log_request()
id
request.class
MetricsService.record Shared method for recording the time it takes for a request to run
ExternalApi::VBMSService
.call_and_log_service()
vbms_id
service.class
MetricsService.record Shared method for recording the time it takes for a service to run
ExternalApi::VBMSService
.fetch_documents_for()
download.file_number
request.class
self.send_and_log_request Records the time it takes to fetch documents for a download
ExternalApi::VBMSService
.v2_fetch_documents_for()
veteran_file_number
request.class
self.send_and_log_request Records the time it takes to fetch documents associated with a given veteran file number
ExternalApi::VBMSService
.fetch_delta_documents_for()
veteran_file_number
request.class
self.send_and_log_request Records the time it takes to fetch documents associated with a given veteran file number and time period
ExternalApi::VBMSService
.fetch_document_file()
document.document_id
request.class
self.send_and_log_request Records the time it takes to fetch a document
ExternalApi::VBMSService
.v2_fetch_document_file()
document.document_id
request.class
self.send_and_log_request Records the time it takes to fetch a document
ExternalApi::VBMSService
.v2_fetch_documents_for()
Veteran_file_number
service.class
self.call_and_log_service Records the time it takes to find documents associated with a given veteran file number
ExternalApi::VVAService
.fetch_documents_for()
download.file_number MetricsService.record Records the time it takes to fetch documents associated with a given claim number
ExternalApi::VVAService
.v2_fetch_documents_for()
file_number MetricsService.record Records the time it takes to fetch documents associated with a given claim number
ExternalApi::VVAService
.fetch_document_file()
document_id MetricsService.record Records the time it takes to fetch data for a given document

Rails Logs / Shoryuken

Direct Rails Logger Calls

Where in Code Key ID Method Used Level Description
ApplicationController
.authenticate()
request.original_url
request.referer
Rails.logger info Logs page that was trying to be accessed and the page the user was coming from when user is not authenticated
Api::V2::ApplicationController
.fetch_veteran_by_file_number()
file_number
current_user
Rails.logger debug Logs the current user and file number they are attempting to fetch
Config.ru:44 Process.pid
thread_count
backlog
waiting
@workers
Rails.logger info Logs Puma stats
DocumentCreator
.create()
ft_delta_documents
css_id
Rails.logger info Logs whether or not the current user can use feature toggle
DocumentCreator
.create()
manifest_source Rails.logger info Logs the manifest source id of a record that was trying to download if that record was already downloaded
V2::DownloadManifestJob
.perform()
manifest_source
manifest
file_number
Rails.logger error Logs the error that occurred while a user was attempting to download a file with the name of the error and the appeal file number
V2::DownloadManifestJob
.perform()
file_number
docs.map(&:id)
log_info / Rails.logger info Logs the file number, documents in the file, and zipfile size when a user attempts to download a file
ExceptionLogger
.capture()
n/a Rails.logger error General use for logging exceptions in the application to Rails Logs
ManifestFetcher
.fetch_documents_or_
delta_documents_for()
file_number
docs
ft_delta_documents
Rails.logger info Logs the file number, documents in the file, and file size of documents fetched from VBMS
MetricsService
.record()
description<
stopwatch
Rails.logger info Shared method which logs the start of a task description sent to it and the finish with time taken, or logs that the task was rescued
PdfService
.write()
size Rails.logger warn Logs that a pdf of the size passed cannot be written along with the pdf path, filename, and an error
RailsVBMSLogger
.log()
response_code
response_body
request_class_name
Rails.logger error Logs the HTTP response code along with the request class name and the response body
RecordFetcher
.process()
n/a process error Logs an error if records cannot be fetched from VBMS or VVA
SessionsController
.create()
user
css_id
Rails.logger info Logs the user who successfully signed in and the path they are being redirected to
SessionsController
.create()
n/a Rails.logger error Logs the error that occurred while a user was attempting to log in with the code location it failed
Shoryuken.rb n/a Shoryuken.configure_server n/a Sets the Rails logger to use Shoryuken Logging
ExternalAPI::VBMSService
.fetch_documents_for()
download Rails.logger info Logs the VBMS downloaded document’s list length
ExternalAPI::VBMSService
.v2_fetch_documents_for()
veteran_file_number Rails.logger info Logs the VBMS document list length from a veterans file number
ExternalAPI::VVAService
.fetch_documents_for()
download Rails.logger info Logs the VVA document list length for the download
ExternalAPI::VVAService
.v2_fetch_documents_for()
file_number Rails.logger info Logs the VVA document list length for the download

MetericsService Calls

Where in Code Key ID Method Used Description
ExternalApi::BGSService
.fetch_veteran_info()
file_number MetricsService.record Logs that fetching veteran info by file number has started and then finished with the time taken, or that there was an error
ExternalApi::BGSService
.fetch_poa_by_file_number()
file_number MetricsService.record Logs that fetching a poa has started and then finished with the time taken, or that there was an error
ExternalApi::BGSService
.fetch_poa_by_participant_id()
participant_id MetricsService.record Logs that fetching a poa by claimant and participant id has started and then finished with the time taken, or that there was an error
ExternalApi::BGSService
.fetch_poa_org_record()
participant_id MetricsService.record Logs that fetching a poa by org and participant id has started and then finished with the time taken, or that there was an error
ExternalApi::BGSService
.fetch_poas_by_participant_ids()
participant_ids MetricsService.record Logs that fetching poas by org and participant ids has started and then finished with the time taken, or that there was an error
ExternalApi::BGSService
.fetch_claims_for_file_number()
file_number MetricsService.record Logs that fetching claims by file number has started and then finished with the time taken, or that there was an error
ExternalApi::BGSService
.fetch_person_info()
participant_id MetricsService.record Logs that fetching a person's info by participant id has started and then finished with the time taken, or that there was an error
ExternalApi::BGSService
.fetch_person_by_ssn()
ssn MetricsService.record Logs that fetching a person by ssn has started and then finished with the time taken, or that there was an error
ExternalApi::BGSService
.check_sensitivity()
file_number MetricsService.record Logs that checking if a client can access a given file number started and then finished with the time taken, or that there was an error
ImageConverterService
.convert_tiff_to_pdf()
image_magick MetricsService.record Logs that converting a tiff to pdf has started and then finished with the time taken, or that there was an error
ManifestFetcher
.documents_from_service_for()
manifest_source.name
file_number
MetricsService.record Logs that fetching documents or delta documents for a file number has started and then finished with the time taken, or that there was an error
RecordFetcher
.content_from_va_service()
manifest_source.name
file_number
s3_filename
MetricsService.record Logs that fetching a record from a manifest source for a file number has started and then finished with the time taken, or that there was an error
RecordFetcher
.content_from_s3()
s3_filename
file_number
MetricsService.record Logs that fetching a record from s3 for a file number has started and then finished with the time taken, or that there was an error
ExternalAPI::VBMSService.
send_and_log_request()
request
id
MetricsService.record Logs that sending a request for a given id has started and then finished with the time taken, or that there was an error
ExternalAPI::VBMSService
.call_and_log_service()
service
vbms_id
MetricsService.record Logs that calling a service for a vbms id has started and then finished with the time taken, or that there was an error
ExternalAPI::VVAService
.fetch_documents_for()
download.file_number MetricsService.record Logs that getting the document list for a claim has started and then finished with the time taken, or that there was an error
ExternalAPI::VVAService
.fetch_document_file()
document.document_id MetricsService.record Logs that getting the document list for a claim has started and then finished with the time taken, or that there was an error

Metrics and Monitoring Suggestions

Keys/Options

Services

  • Rails (Logging) - Debugging and security, uses Shoryuken
  • DataDog - Metrics
  • Sentry - Security and logging, uses Raven

Type

  • Info
  • Security
  • Errors
  • Metrics

Controllers

Api::V1::ApplicationController

Method Service(s) Type(s)
unauthorized() Sentry
Rails
Security - station_id, css_id
sensitive_record() Sentry Security - calls forbidden()
forbidden() Sentry
Rails
Security - user_id, file_number or manifest_id
record_not_found() Sentry
Rails
Info - Record ID
authenticate_or_authorize() Sentry
Rails
Info - station_id, css_id
Security - calls unauthorized()

Api::V2::ApplicationController

Method Service(s) Type(s)
veteran_not_found() Rails
Sentry
Errors - file_number
vso_denied_record() Rails
Sentry
Security - calls forbidden()
verify_veteran_file_number() Rails
DataDog
Sentry
Metrics - time taken to verify, user cache/queue info (suggestions: documents already open, attempts count, last attempt time)
Info - file_number
fetch_veteran_by_file_number() Sentry Security - user_id, file_number, can call sensitive_record() or vso_denied_record()

Api::V2::FilesDownloadsController

Method Service(s) Type(s)
start() Rails
Sentry
DataDog
Info - @files_download
Metrics - time taken to download and finish, size of manifest

Api::V2::ManifestsController

Method Service(s) Type(s)
start() Rails
Sentry
DataDog
Info - user_id, file_number
Metrics - time taken to download and finish, size of manifest
refresh() Rails
Sentry
DataDog
Info - user_id, file_number
Metrics - time taken to download and finish, size of manifest

Api::V2::RecordsController

Method Service(s) Type(s)
document_failed() Rails
Sentry
Errors - manifest_source, version_id, document_id
record() Rails
Sentry
DataDog
Info - manifest_source, version_id, document_id
Metrics - time taken to download and finish, size of document

ErrorsController

Method Service(s) Type(s)
show() Rails
Sentry
Errors - general error info, status_code

SessionsController

Method Service(s) Type(s)
build_user() Rails
Sentry
DataDog
Info - username, css_id, station_id
Metrics - time taken to finish
Security - if fails - username, css_id, station_id

Jobs

V2::DownloadManifestJob

Method Service(s) Type(s)
perform() Datadog Metrics - time taken to start and end a manifest download, manifest size, and document count

V2::PackageFilesJob

Method Service(s) Type(s)
perform() Datadog Metrics - time taken to create a zipfile, zipfile size

V2::SaveFilesInS3Job

Method Service(s) Type(s)
perform() Datadog Metrics - time taken to save all the records to s3, record count, record size

ApplicationJob

Method Service(s) Type(s)
datadog_report_runtime() Datadog Metrics - leverage this in other jobs
datadog_report_time_segment() Datadog Metrics - leverage this in other jobs

Services

POAMapper

Method Service(s) Type(s)
get_claimant_poa_from_bgs_poa() DataDog
Sentry
Metrics - time taken to create bgs_rep object
Info - bgs_record
Errors
get_claimant_poa_from_bgs_claimants_poa() DataDog
Sentry
Metrics - time taken to create object
Info - bgs_record
Errors
get_hash_of_poa_from_bgs_poas() DataDog
Sentry
Metrics - time taken to create the hash
Info - bgs_resp
Errors
get_poa_from_bgs_org_poa() DataDog
Sentry
Metrics - time taken to create object
Info - bgs_rep
Errors

ExternalApi::BGSService

Method Service(s) Type(s)
parse_veteran_info() Sentry Info - veteran_data
parse_person_info() Sentry Info - bgs_info
fetch_user_info() Sentry Security
fetch_veteran_info_cache_key() Rails Info - file_number, client

DocumentCounter

Method Service(s) Type(s)
count() DataDog Metrics - time taken to run the count, file_numbers count

DocumentCreator

Method Service(s) Type(s)
create() DataDog Metrics - time taken to create a document, external document count, external document sizes

ManifestFetcher

Method Service(s) Type(s)
fetch_documents() DataDog Metrics - time taken to fetch all the documents, file number count, document sizes

PdfService

Method Service(s) Type(s)
write() Rails
DataDog
Info - contents.size, command,
Errors
Metrics - time taken to write out the pdf, pdf size
write_attributes_file() DataDog Metrics - time taken to write the file

UserAuthorizer

Method Service(s) Type(s)
General Sentry Security

VeteranFinder

Method Service(s) Type(s)
find() Rails Info - file_number, bgs_rec_numbers
find_uniq_file_numbers() DataDog Metrics - time taken to find all the file numbers, file number count
find_duplicate_bgs_rec() DataDog Metrics - time taken to find all the duplicate file numbers, file number count

ZipfileCreator

Method Service(s) Type(s)
process() DataDog Metrics - records count, time taken to create a zipfile, total size of records, size of zipfile
write_to_tempfile() Rails Info - t.path, record
Clone this wiki locally