Quality Review

Background

A small number of appeals need to be routed to quality review before dispatching appeals each month in order to monitor decision-writing quality and consistency.

• They have a quality review team queue, with individuals on the QR team having their own queues
• QR admin will manually assign cases selected for Quality Review to individuals on the team
• Quality Review has the ability to send a case:
  • Back to their judge (if there are required corrections to the decision)
  • Straight to BVA dispatch (if there are no issues with the decision and it can go straight out)

Where cases are selected

When submitting a decision using the "Ready for Dispatch" task action on a JudgeDecisionReviewTask, the request is sent to the CaseReviewsController which uses the CompleteCaseReview workflow to process the request. For an AMA appeal, this workflow is where the QR selection occurs; the create_next_task method will use the QualityReviewCaseSelector class to randomly select an appeal using the QUALITY_REVIEW_SELECTION_PROBABILITY constant defined in the class. There also is a limit on the number of cases which can be selected for QR per month, which is checked by comparing the defined constant to the number of QualityReviewTask already created in the current month. If enough cases have already been created, the selector will return false for the rest of that month.

    def select_case_for_quality_review?
      return false if reached_monthly_limit_in_quality_reviews?

      rand < QUALITY_REVIEW_SELECTION_PROBABILITY
    end

If the appeal is selected, a QualityReviewTask will be created on the appeal and assigned to the QualityReview team. If it is not selected, a BvaDispatchTask is created and assigned to BvaDispatch instead.

    if QualityReviewCaseSelector.select_case_for_quality_review?
      QualityReviewTask.create_from_root_task(root_task)
    elsif open_qr_tasks?
      BvaDispatchTask.create_from_root_task(root_task)
    end

For Legacy cases, the selection of an appeal to go to Quality Review is done as an after_create hook on the JudgeCaseReview model. Functionally it does the same check as the QualityReviewCaseSelector class, but with constants defined in the JudgeCaseReview class itself. Like the QualityReviewCaseSelector, it also has a monthly limit of cases which can be sent to the QR team.

  def select_case_for_legacy_quality_review
    return if !legacy? || self.class.reached_monthly_limit_in_quality_reviews?

    update(location: :quality_review) if bva_dispatch? && rand < QUALITY_REVIEW_SELECTION_PROBABILITY
  end

The constants are periodically updated at the request of the Board to maintain the correct number of AMA and Legacy cases being sent to QR. There are automated tests in the quality_review_case_selector_spec.rb file which verify the function of creating the QR tasks, as well as verifying that the random selection is selecting approximately the desired number of appeals.

(OUTDATED) QR selection logic

• Github issue with history
• QR_SELECTION_PROBABILITY = 0.04
  • Probability that a given case should be selected for Quality Review (p)
  • We want the number of cases selected in a given month (X) to equal or exceed the target (x), as any excess selected cases will be ignored. The target during the pilot is 34 cases.
  • We want the probability that X is less than x (P(X<x)) to be less than 0.000833, an event that would happen once per century.
  • In December 2017, the Board wrote 5,549 decisions. Taking 1/4 of that figure, reflecting that Caseflow is piloted with 1/4 of the Board's judges, we have a number of trials (n) equal to 1,387.
  • When n = 1,387, x = 34, and p = 0.04, we find that P(X<x) = 0.000464, which is less than our maximum probability 0.000833.
  • This probability should be 0.032 when Caseflow expands to include all judges.
  • When n = 5,549, x = 136, and p = 0.032, we find that P(X<x) = 0.00043, which is less than our maximum probability 0.000833.
  • An easy way to avoid doing the math is to use this calculator: https://stattrek.com/online-calculator/binomial.aspx