multiannotator_cifar10/multiannotator_cifar10.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6f08ac68",
   "metadata": {},
   "source": [
    "# Iterative use of Cleanlab to Improve Classification Models (and Consensus Labels) from Data Labeled by Multiple Annotators"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ed1e109",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cleanlab/examples/blob/master/multiannotator_cifar10/multiannotator_cifar10.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44a0555b",
   "metadata": {},
   "source": [
    "This example shows how to repeatedly improve consensus labels established from data labeled by multiple annotators by iterating the following steps: (1) train a model on the current consensus labels, (2) leverage the model's predictions to obtain superior consensus labels that can be used to subsequently train a better model in the next round. In each round, consensus labels are established using the [CROWDLAB algorithm](https://arxiv.org/abs/2210.06812), for which a quickstart tutorial is available in the [cleanlab documentation](https://docs.cleanlab.ai/stable/tutorials/multiannotator.html). \n",
    "\n",
    "Here we demonstrate this functionality using a variant of the [CIFAR-10](https://github.com/jcpeterson/cifar-10h) dataset from Peterson et al. (2019), in which multiple human annotators were asked to suggest labels for images from this famous image classification dataset.\n",
    "Because this notebook utilizes AutoML for model training and cleanlab is compatible with any model/dataset, you should be able to run the below code with any image classification dataset where each image has been labeled by multiple annotators."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95724ad1",
   "metadata": {},
   "source": [
    "## Import required dependencies and download data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c638f760",
   "metadata": {},
   "source": [
    "Please install the dependencies specified in this [requirements.txt](https://github.com/cleanlab/examples/blob/master/multiannotator_cifar10/requirements.txt) file before running the notebook.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68b291ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib.ticker import MaxNLocator\n",
    "from datetime import datetime\n",
    "\n",
    "from utils.model_training import train_model, sum_xval_folds\n",
    "from cleanlab.multiannotator import get_majority_vote_label, get_label_quality_multiannotator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f3551b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_path = \"./experiment\"  # folder to save outputs from this notebook\n",
    "if not os.path.exists(experiment_path):\n",
    "    os.makedirs(experiment_path)\n",
    "    print(\"Directory \" , experiment_path ,  \" Created \")\n",
    "else:    \n",
    "    print(\"Directory \" , experiment_path ,  \" already exists\")\n",
    "\n",
    "print(f'Experiment saved in {experiment_path}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23c50596",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download CIFAR-10 dataset for model training (requires internet connection)\n",
    "!wget -nc 'https://cleanlab-public.s3.amazonaws.com/Multiannotator/cifar-10/cifar10_test.tar.gz'\n",
    "!tar -xzf cifar10_test.tar.gz\n",
    "\n",
    "# Here is an alternative command to download the data from the source:\n",
    "# cifar2png cifar10 ./data/cifar10_test --name-with-batch-index\n",
    "\n",
    "# Import CIFAR-10 multi-annotator labels and image paths\n",
    "!cd $experiment_path && wget -nc 'https://cleanlab-public.s3.amazonaws.com/Multiannotator/cifar-10h/cifar-10h-worst25-coin20/c10h_labels_worst25_coin20.npy'\n",
    "!cd $experiment_path && wget -nc 'https://cleanlab-public.s3.amazonaws.com/Multiannotator/cifar-10h/cifar-10h-worst25-coin20/c10h_image_paths.npy'\n",
    "!cd $experiment_path && wget -nc 'https://cleanlab-public.s3.amazonaws.com/Multiannotator/cifar-10h/cifar-10h-worst25-coin20/c10h_test_labels.npy'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d2bed92",
   "metadata": {},
   "source": [
    "## Load labels selected by each annotator\n",
    "Here `multiannotator_labels` contains labels from multiple annotators for each CIFAR-10 image. Each image has been labeled by one or more annotators, but not every annotator has labeled every image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a66d57e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load labels\n",
    "multiannotator_labels = np.load(f'{experiment_path}/c10h_labels_worst25_coin20.npy')\n",
    "\n",
    "# Load and reformat image paths to work for specific machine\n",
    "image_paths = np.load(f'{experiment_path}/c10h_image_paths.npy', allow_pickle=True)\n",
    "path = os.getcwd()\n",
    "image_paths = [f\"{path}/{image_path}\" for image_path in image_paths]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f7a322a",
   "metadata": {},
   "source": [
    "> ### Bringing Your Own Data (BYOD)?\n",
    "> \n",
    "> You can easily replace the above with your own multiannotator dataset, and continue with the rest of this notebook.\n",
    "> \n",
    "> `multiannotator_labels` should be a numpy array or pandas DataFrame with each column representing an annotator and each row representing an example. Your classes (and entries of `multiannotator_labels`) should be represented as integer indices 0, 1, ..., num_classes - 1, where examples that were not labeled by a particular annotator can be represented using `np.nan`.\n",
    ">     \n",
    "> If working with images, `image_paths` should be a list whose i-th entry is a string specifying (absolute or relative) path to the image corresponding to i-th row of `multiannotator_labels`.\n",
    "> \n",
    "> The strategy outlined in this notebook can also be used for non-image data if you replace `image_paths` with a suitable object containing training examples corresponding to the rows of `multiannotator_labels`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66bafbf8",
   "metadata": {},
   "source": [
    "To training our classification model, we must first obtain consensus labels from the multiple annotations in the dataset. An initial set of consensus labels can be established via a simple majority vote."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5a953e3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "consensus_labels = get_majority_vote_label(multiannotator_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c114734",
   "metadata": {},
   "source": [
    "## Train model and use cleanlab to get better consensus labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e31453ef",
   "metadata": {},
   "source": [
    "Next we train our model on these consensus labels and then obtain out-of-sample predicted probabilities for each example in the dataset. We will then use these `pred_probs` to generate improved `consensus_labels` using cleanlab's [CROWDLAB algorithm](https://arxiv.org/abs/2210.06812). In turn, these improved `consensus_labels` can then be used to train a better model that generates more accurate `pred_probs`. This process is iterated in the loop below to repeatedly improve both the consensus labels and our classification model.\n",
    "\n",
    "Within each round, the `train_model()` method trains a `resnet18` image classifier using cross validation to get out-of-sample predicted class probabilities for each image in the dataset. This function can be replaced with custom code for training your own model. Here we use the [AutoGluon](https://auto.gluon.ai/) AutoML library to train powerful image classifiers (like ResNet or Swin Transformer), but this requires a GPU to run in a reasonable time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "370c66ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_type = \"resnet18\"  # to use Swin Transformer model instead of ResNet, replace this with: \"swin_base_patch4_window7_224\"\n",
    "\n",
    "train_args = {  # parameters to control model training \n",
    "    \"num_cv_folds\": 5,  # decrease this value if you want code to run faster\n",
    "    \"epochs\": 50,  # decrease this value if you want code to run faster\n",
    "    \"time_limit\": 7200, # decrease this value if you want code to run faster\n",
    "    \"random_state\": 123,\n",
    "    \"verbose\": 1, \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c27da16",
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_changed = set()\n",
    "seen_consensus_labels = list()\n",
    "model_results = {}\n",
    "rounds = 0\n",
    "MAX_ROUNDS = 3  # Maximum number of times to establish new consensus labels train model\n",
    "\n",
    "# Loop to iteratively improve consensus labels and our trained classifier (this code takes a while) \n",
    "while tuple(consensus_labels) not in seen_consensus_labels and rounds < MAX_ROUNDS:\n",
    "    seen_consensus_labels.append(tuple(consensus_labels)) # add curent consensus labels into the set\n",
    "    model_results['rounds'] = rounds\n",
    "    model_xval_results_folder = f'{experiment_path}/xval_results_round{rounds}' # + [model_type]\n",
    "\n",
    "    # Zip consensus labels with their corresponding image_paths\n",
    "    consensus_data = pd.DataFrame(zip(image_paths,consensus_labels), columns=[\"image\", \"label\"])\n",
    "    \n",
    "    # Train model\n",
    "    train_model(model_type, consensus_data, model_xval_results_folder, **train_args);\n",
    "    pred_probs, labels, images = sum_xval_folds(model_type, model_xval_results_folder, **train_args)\n",
    "    \n",
    "    # Get improved consensus labels with label quality multiannotator using model pred probs\n",
    "    label_quality_multiannotator = get_label_quality_multiannotator(multiannotator_labels, pred_probs, verbose=False)\n",
    "    consensus_labels = label_quality_multiannotator[\"label_quality\"][\"consensus_label\"].tolist()\n",
    "    \n",
    "    unique_indices = len(indices_changed)\n",
    "    indices_changed.update(list(np.where(consensus_labels != np.array(seen_consensus_labels[-1]))[0]))\n",
    "    unique_indices = len(indices_changed) - unique_indices\n",
    "    label_changes_from_prior = np.sum(consensus_labels != np.array(seen_consensus_labels[-1]))\n",
    "    \n",
    "    print(\"Num changes in consensus labels from previous round: \", np.sum(consensus_labels != np.array(seen_consensus_labels[-1])))\n",
    "    print(\"Num unique indices changed: \", unique_indices)\n",
    "    # End to delete\n",
    "    \n",
    "    results = {\n",
    "        \"pred_probs\": pred_probs,\n",
    "        \"consensus_labels_in\": labels, # consensus labels used to train the model\n",
    "        \"images\": images, \n",
    "        \"consensus_labels_out\": consensus_labels, # new consensus labels generated from pred_probs\n",
    "        \"label_changes_from_prior\": label_changes_from_prior, # num changes in consensus labels from previous iterations\n",
    "        \"unique_indices_added\": unique_indices, # number of unique labels indices changed\n",
    "    }\n",
    "    \n",
    "    model_results[rounds] = results\n",
    "    rounds+=1\n",
    "    \n",
    "    if unique_indices == 0:  # no more label improvement has occurred\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aa26e05",
   "metadata": {},
   "source": [
    "## Evaluate consensus labels and model predictions\n",
    "Since our annotators annotated a dataset `CIFAR-10` for which there exist [highly accurate](https://arxiv.org/abs/2103.14749) `true_labels`, we can evaluate the accuracy of both the consensus labels and classifier predictions against ground truth gold labels. In real applications, this will not be possible since ground truth labels will not be available. Here we emphasize that all classifier accuracy estimates are calculated via cross-validation and only computed using *held-out* data that the model did not see during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3758a5fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load ground truth labels (only for evaluation purposes, you normally do not have this information)\n",
    "true_labels = np.load(f'{experiment_path}/c10h_test_labels.npy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "06915543",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final consensus label accuracy (vs ground truth):  0.9688\n",
      "Final model predictions accuracy (vs ground truth):  0.9063\n"
     ]
    }
   ],
   "source": [
    "# Calculate accuracy of consensus labels and model predictions from each round, measured against ground truth labels\n",
    "for i in range(rounds):\n",
    "    pred_probs = model_results[i]['pred_probs']\n",
    "    consensus_labels_in = model_results[i]['consensus_labels_in']\n",
    "    consensus_labels_out = model_results[i]['consensus_labels_out']\n",
    "\n",
    "    acc_model_gtruth = (pred_probs.argmax(axis=1) == true_labels).mean()\n",
    "    acc_consensus_gtruth = (consensus_labels_in == true_labels).mean()\n",
    "\n",
    "    results = {\n",
    "        \"consensus_gtruth_accuracy\": acc_consensus_gtruth, # consensus labels accuracy \n",
    "        \"model_gtruth_accuracy\": acc_model_gtruth,         # model label accuracy\n",
    "    }\n",
    "    \n",
    "    model_results[i].update(results)\n",
    "\n",
    "final_consensus_labels = model_results[rounds-1][\"consensus_labels_out\"]\n",
    "final_consensus_accuracy = (final_consensus_labels == true_labels).mean()\n",
    "\n",
    "print(\"Final consensus label accuracy (vs ground truth): \", final_consensus_accuracy)\n",
    "print(\"Final model predictions accuracy (vs ground truth): \", model_results[rounds-1][\"model_gtruth_accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03bb6b74",
   "metadata": {},
   "source": [
    "Let's plot the accuracy of our consensus labels and trained model over each round, in which we establish consensus labels (with the help of predictions from the model trained in the previous round) and then train another copy of our model using these new consensus labels. Here round 0 corresponds to the initial `consensus_labels` obtained via majority vote (without the help of a model) and the classifier produced by training a model on these majority-vote `consensus_labels`. \n",
    "\n",
    "You can see that cleanlab greatly improves the consensus labels from those obtained via basic majority vote and this also greatly improves the trained model. Iterating this process over several rounds brings further improvements in both the consensus labels and the trained classification model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0b97270d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA/IAAAGICAYAAADiRNu2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABl0klEQVR4nO3dd5gc1ZWw8fdolLNGEUlIIyGCQSABA8bGBhGMsU0wDmucdnFYHMC7DuyuvY7rvE772YADThic1wl5wQmLbBtLAokcJKE4AgmNcpp0vz+qWtMaRqOWNDM94f09Tz/dfbu6+nSqe0/VrXsjpYQkSZIkSeoe+pQ7AEmSJEmSVDoTeUmSJEmSuhETeUmSJEmSuhETeUmSJEmSuhETeUmSJEmSuhETeUmSJEmSuhETeamTRETKL59oh3V9orC+dghNQERUFX1Hl5U7nn2JiOV5jNeXMYbr8xiWt8O6usXnLvV2h/Jf7Qrbrf2JiMuK3l9VXtZu27p8fe3WDlB5tPdvoiO09lsuQwztWrd3h8+9HEzku7mIGBAR74uIv0bE5ojYGRFPRsR3I+J55Y6vu4uIOSYZ+xYRL4qIn0bEyojYHREbI+LhiPhWRFSXO772VrwDpY3L8nZ4neLf3ZxDDrx5vYXG9O3ttU5J5RERtxdtJx6PiCh6bEi+PS48fn0ZQy1Ji4Z/4bIzIp6IiC9GxJgyhLUUuBe4v9Qn7CeJuje/rG6/ELu2iBgWER+KiHvzduruvM3w54i4KiIqyh1jeyuhndBeyW1hG3D7oUe9Z53Fv9857bVedYy+5Q5ABy8iRgF/Bk7Mi7YBTwKHA28FHgQeLU906uki4mPAJ4AAElnDZBMwGbgc2AosaOP5/VNKdR0eaPtaTdYIAxgAzM5vr6G5Yba2tSd20/crqXs4Cngp8Pv8/j8CI8sWzaErbFPHA0cCVwHnRsTJKaWm1p7QEdvYlNKngE+14/pOa691dQcRMRWYB0zPi7YDjwPDgTOAs4FvkrVf97WO7lh33lt0+whgTCvl61s+qZu+V5WRR+S7t2toTuK/BFSmlE5IKY0CXgAsLiyYHzn9Q9He0Mcj4sMR0a9omcLRuhsi4r8iYm2+R/+HETGsaLnzI+Ke/LGdEfFURPw6IqYVLXNKRPxfRNTmr/dgRLylOPiiPX7/HRHXRMSGiFgXEV+NiL5Fy70pIu6LiC0RsT3fO//TiBiRP/6c7jat7RGPiKPyOJ/JY6qJiFsj4qWH8iVExMn5nuW1+Xq3R8T8iHjTPp7SP3+PtRGxKX/v/YvW1z8iPpp/R7vzz+XHETH5AOP6Zv7+H2pR/n95+Z/y+6dGxJ8i4tn89VZFxM3RxhH1iDgX+C+yJP4h4PiU0pT891cJnAT8tmj5wm/rxoj4ckQ8S16hRcSgiPhMRCyJiLr8c/ltRJxU9PzWvs/ndNtqsdzFEXFn/ht9LCIuaPEeLsl/S7si4k7g2P19piml76SUTssbY5cUPVRcflNRDP8QEQsiog54ebRySkS0OPoeWZfL24rWfVu0fkQtIuI/89/xc/6nByv/jp6MiK3597EiIr4WEcP3sfwr8893V0TcHRHHtXj8vIiYl/9/d0V2VObC/cQwJCKujeyoza78P3BvRLz/UN+f1APV59f/UlT2nvy6oeXCpWxz8+VK2kZGVrf+NLL6uy7ffvxbRBxKG7OwTZ0G/CIvmw2ckL9mYZv5xcjaAJuBn+WPDYuIr0TWNqmLrG7+ZkSMLIo5IuLjeczbIuJGYEQr76219kVExDsjYmFE7MifvzAiXphvp79ftIqnoqgrfVHcnyha35TI2l1PR0R9RKyJiOsiYlxrcUTEa/Nt7va8jju6aLkDaudExKSIaMzX/Zqi8uqiWE+PiIqI+HT+m9mZ1zmLIuLz+/kef0RzEv9hYGTeTqgCRpPtcKrLX7O4/t6r7swfP5B27PX7+Q5Lbe+OiKz9tS3/fj5K1u5pU6E9kLcJbt5H+bWx73ZRa7+TvY6+R9aOODN/+Myi51S1COd5kdXBrbaFDkZkecBd0fyf35Lff9k+njIif59b8+d8PGKvHkT7/c+2EUeb+UiPl1Ly0g0vZBVOPdmR0EVAtLHsnKJlN5LtDU355SdFyy3Py+qALcCyouU+ky8zBtidl63MX3tDfv9F+TIvLFrmGeDhovV8oOj1UtHrbSDb+14o++d8mROAprxsCfAAsDm/Pzlf5vr8/vKidV9WtK6qvGxhfr82v114vY/s57MrrOeyfSzzGqAx//zuy9dfeM4rWnm/28j2xD5VVPaFouV+m5c15u+3sL4VwKh8mU8UnttG7KcVrX9mXlaZf94JeCPZzrz1Rd/VfcDT+f03tbHunxet++QSfq+F39bu/PIgcFf+2J+K1vUo2W8vATuA2W18n1Utv5sWy9UBT+TrSfl6K/Pljidr4BbKH8u/lza/6xbvqfj1P7GP395usiNLTwIXt/a9sfdvbA7wduCRorJHgL8BH23xWe5gH//TEr6H29tYZhvwLNl/e2nRuv+3aJnr87JdeRwP07yNWQEMLPpvFP6/q/LPIeVlr2nje/xy0ed3Xx5HPXBrube9Xrx0lQtwe/4/WUhWrzeRHb0+Ny//Y/5fTsD1Rc8rZZtb0jYSmEHWrii0LxaT1V0JuLroNQvbnuvbeD/F24JPFJX/b1F5Ib7ibex2sjrlp0B/muv63Xk8hbgXAv3y57+7aB01ZO2B4vdXlS9X2NYtL4rn6qLlavPX3km27f8oe2837yfbfr+9RdyfyO+PI6sjCtvTh2muo58AhraIoz5//FGat633FMV2MO2c3+fL/KKorLANfiK//578fkP+mT6ex7ukjfUeW/R+f1vC7/myFt9rcd05hwNrxxb/3lv7DgvL7bO9my9X3NZ5gqzX4baW69vP+yq8fmpRXoihtXZRa/+D2ymqv/PfVeG/uyW//zfgsBaf5Q720RYq4XuYs49lrso/u6VkdfRWmn+fs1r5P28j+y3WFJW9O1+u1P/sXt8jJeQjveFS9gC8HOQXB6cU/Rmu3s+ydxT90AuJ4OeLnn98XlbYqGwBJpEleQvysr/ly5xctMzgotc4ARib356XL3NH0R/ww0XPKzTyC6+/jGzHxECaK7Sf5su8muYNaJ+8rA/w/MLrt/xz52XFG6KqvKywoXlx0XKTgaPb+OzmFK3nsn0scxgwvuj+QJoTlhuLygvreQwYQrZX91d52U5gMFlXs8Jy5+XPG0lzsv3hvOwTheX2890/mi/36fz+P+f3NwODyPaIF17v8KLnzQCmtrHews6ZLUVlxTsOEvBs0WPLad5In5CXVQBnFS1/VV4+geaG4S/b+D6rWn43LZb7cl52UVHZ+XnZD/L7W2neIfTp/X3XLT6D4tf/xD5+ez+i+Xdb0dr3RotEfl9lrXyW+/yfthFz4bm3t7HM7Bb3C59LPc3/3euL4js3L3tlUdlb8rJC4+hH5DsbgW+zdwOxte+xsDPro0VxDAdOOdRtpxcvPeVCc8N+AXBlfvurwNz89gW0SOQpfZtb0jYS+F5+/3FgWF72xryskbxe4cAT+dVkSclTRWWLaN6eFsrWF8VXQXaEt7C9Oi4vn0rzTok35mUr8vv3kiUSfcl6QrWsZwrbuuVFMRYS6Jto3iaOAqblty9ruZ6i97hXnUHWsy3l6zwlLzu/aLn3tIgjARfmZV8pKhuUlx1MO+dSmhO+YWRtk1V52X/myxR2Xny36HmDgBe2sd7XFsVXfBDn90XlCbiylc+tZd15oO3Y64teb6/vsMVybbV3pxet/3/ysnE0J4vL9/XeW3wOe767FuWFGPZqF7X2O2nxf7+9rbJWPst9toX2EW/xc+e08V8dWXR/FM07FT7Vyv/5NrL/WH+y/1wCVuTLlfqf3et7pIR8pDdc7FrffRV37Un7WfaU/Pr3KaWN+e0fFz3esgv1vJTSmpSdh/ZYXjY+v36YrHE+DFgXEfdHxA/J9rw+my/z/Pz6DKAu7/7z6bxsGLBX11tgbkppc0ppF1mlXfx695A1MI4EaiNiPlkiUJlS2rGf991Soav3n/MuWTeRVTRrDnA9LTUBX867sDWQJeUz8scmtrL8zSml7Snb4vxvXjaQ7Dyq5xct94f8s9tI8/lVB3p+3Q/y69fl15fm1z9LKe1MKW0A/pqXPRERD0XEz8kaezUlrL/4t7eVbAP97D6WBbgtpfQAQEqpkebfJuS/yZTS0zR3LT+UAfNuzK8fKSor/K6Oz6//klIqnNv+s0N4rX25Ov8fFd5ve2rrf3oozsl/Bzvz39+H8/K+wNgWy9amlG7Nb99E1iABmBkRY4FC97Y3AE35+t6elx0ZEaP3EUPhv/rJyLrX3wr8O62cUygJyBq5m4G3Aa8gO1J2SyvLlbrNLXUbWaizjgK25P/xH+ZlfYBTD+hdNJuUr3sCWW+8L5PtNGx5fvwvC/Hl29hCPH2Bh/J4lpMlgwCnRXaa0JT8/m9SSnUppQayHev7cwrN7a+v5O0WUkobU0pP7ftpba4PsiPb8/N1/Z6s3ofn1oGbU0qF7WNx3Vbohn8w7ZzfkB1pHkR29PtFZMl/E3BDvsz/kdX3b827Pd8BfIYsiSpFcVvhUbKdMm1pWXceaDu2VG3VozOLlvt5Hss6suS5PbVsF7W3ttpCB6s/cH3eTb6RrAdI4ZSE1tq9v0opNaTs/P/f5GVT8tMY9vuf3UcMpeQjPZ6D3XVfj5PtreoLvCgiIk8M28OmotuF8+sCIKW0KyJOBt5M9uc7lqyR/kayI9P/U/TcGrK9ui21rIjber2nIzvn9s1ke9+OJxvI760R8aqU0q9priCKRz4d0crr/iPZkYo5edznke2hnENWeR2sH9LclfERsi5Bx5JtXA5lNNa/89ydNCsPcB03klW2M/Lzos7My79ftMw5ZN/h6WRxX0JW8c8E/nUf6304X3Z4RMxKKS1OKT1M1ki6HvinfTzvmQOMv6D4cyh8pq19x8U25dfF54ju99y2dtby/e55HxFRkVfa+3sf+7Kp6PZe/5uDFRFvJBtvA7JB+1aR7UQqnON4sL/np4B1rZT3a6WMlNJ1EfEY2f/zeLL//jnAWyLiqJTS9oOMQ+qRUkrbIuL7wHvzomtTSk1Fp6F2tA1kCXdLOw9yff+VUvpECcvtq06pJ+vyW+ry3cWmotut1W0H3M7J23U/A95BtrN/Rf7QrUU7Sf4Q2RgKrwVmkY3PdAbwzxFxbEqptbbew0W3X0zWg4CU0vvy87jb2vFxqG2F/bUHCzYV3W6XevQgtPVeS30fbdmUX7dnW+hmsgNWDWSnBOwi+0305+DbCQf0nz3AfKTH8oh8N5VS2ky+h5Dsz/PZ2HuAuDMi4uz87vz8+vzIRrqH7MdesM+RxVvK92Q/D7gmpfSmlNJJZOfhQTb6aPHr1QDnpOaBPS4E/l9K6f4DeL2JZF1kvpBSel1K6Via95oWXq+QIIzLB8zoQ+sV1ouBX6eU3plSOoOsS1vxevanX0QMLL7k5YW9hd9OKc0kG5hlnyOwkg16Njgf6KMwuMwusiMo84uW+0rRZ/cCsiOS3yoxVgBSSmvIzoeErCdDBfB4SumvkA3aQzamwfUppbfmr/XdfPm2PpfiOL4XEdP3uWSLkFrcL36/b8hjmkDWIwCaf5vFSeAR+XXxYHMHqjAA4Avz3xhkDZT21vL9lvo+inubDGnXiDJ9Wv6WI5sCqPBb3krWTfT5NP+/W1NZtJ25kGwkf4CHUkrryfaqQ/Z5v7jo9/wPwOfyI4HPERGnAg+nlK5KKb2UrIswZHv6jznwtyv1CleT7SjfRtblvTWlbnNL3UYW1redrMt34T9+HvCNlFJrvQLa077qlL7Ae4vieRFZnf/DlNIWmg8yXBTZALN9Ka1OmV/0mu+NiAGwZ1C0qrz8QLbfhXhnRMQp+brOJ+uqDAfQPssdbDvn+vz6PJp78BXKiIgTgPUppQ+nlC4g27kKMJR99LpIKT0C3J3ffWVEvKd4gLP92Nf3ur92bKGOPSKPu5LmAxgHqnhHxGvz9Y0l2ynSnlo7CNfyfcxg7x4CBYXfWke0EyAbnLm4nTAg70lX6HX6sZTSbJpPz9iXSyKib2QDOxfa5ytTSlsp4T/b2gpLzEd6vvbqo++l8y9kA5fdR/M5KFvIBkcrnL/z3ny5ObTTICFkf95E1o3mAbKkurCuwoB4L6J5sJYtZIO9rCQfEK5o3fs9B4jmI93ryLpiFQ9IUhgQ77yisifzuBqLyqry5VaTbfQez2MqxHhPG5/xnKL1POeSL3NPfr+RbMO/keYB6m5v5f1uy9/PU0VlXyxa7pai8ifI9nYWzj26LF/mE8Ux7Od3cil7x/3Bosf6Fn1PD+evVfjsfrSf9X60aJ0N+We/mObfWmvnyF/fynpaDrxUGMxwJ80DG1XSfO7fBuBOms+fKv5cLmvle69qZblZRe9zS/66u1out5/3X7ze4t/wc2IoeuyYotctTGVX/Fudky83mubf5wayc0Vfs6/PklbOAdxHzMvZ+7dQfLmM5jEUEllXzGU0b0+KP9PC6xUPdleIdxXN542+rui5z5L972rIko3b2/h+fkj2O3qKbLCbwm9iG0Xn5Xnx0psvFJ0jX1Q2ChhRdL+Uwe5a2+aWtI0k61K/KS/bQVZPP5X/f1PRaz5nu9XK+yneFnxiP++91eXIdibenz/WlG+bHiXb0VC8jb2Svbd1q1q8v6p8ucK2bnnRaxQPdreBrM2xo+gzOaHo8bVk2+/TW4ubrEt8YQCwXWQ7UArb0id57mB3xXFc1kq8B9zOKVrfo0Xr20i+Hc8f+3T+ea4k2yYXxu1pAI5pY51T2LvdtpGs3bq2qKy1c+SrWqxnDqW1Yz9bVH5v/nkUfsfFn91ySqhHyWZMKKzvcbLf+q6Wy+3ncy2sN7Uof04MRY/9uOh17yD7Hxfex+1FyxWPk/AA2akH+/ptVBWVXdZGvMXPbXlZzt5jKNSRtRtraR6g7vpWXm9b/pw1RWVXHOB/dq/vhxLykd5w8Yh8N5ZSqiU7mvoBsm7YkJ+nRnZu9B/z5W4n29v+R7JeGNPIEsSPknXDOhAbyLplryX7k04lO5L83+R7flNKd5PtFf4/so18Ycqam4GPHODrLSPboG0iO09+HFn39Q8C38lf74/5emvIutM8to/X+R7ZBmc02Xn668kGVLm0lWUPxGVk5xfuIhuw7r1kG5V9+RrwE7JuUluAb7SI9xLg42TvYyrZuWrLyM4RvP0g4vsNzV2rmmg+XwqyiuGb+fonkv1+VudlV7S10pTNr3sW2XmF6/NYJ5J9P9dS+hHzi8gq32Vke5+byH47p6eUFuWvVQu8nqwiHULWs+CNJa6/tdgXkyWZS8gqkc35+jtUSukx4HKyynA0WYPk3a0st4FsKqlVZA3zwrmiHe27ZA2DZ8lODbkd+Fgbyz9NdlSkgqzy/Avw8tR83ujPgJeRDYDZn2zv+S6ysSG+1NoKczeTNV4GkHWtrwduBV6WUtp0UO9M6gVSdq725v0sVso2t6RtZErpCbLt00/JdrYeS/Zfv53mbv6dJqW0myzp+wrZdvZIsrE9HiZLRgs9Da4FPkm2rRtJlmx/mNL8C9l2+36yOn86WeLxZB7DA2Rzzz9Dtt1+Ps1H2FvGu46sJ9SNZPX00WT16XfIvo+2eve15lDaOT8ouv2zwnY8dwfZQYYgOzLcl2x7/+q8XmtVSmkl2bSBHyfbyVOoB3aStUmvIGsPtekA2rGfI9sRvImsjfojst/mwXp7/vwdZAcUvkFzb9iO9H6yenAb2Xv9b5p7NxT7ElnduI2srjyUcYVKkrIs+tVkR9IbaW6PtXVe+ofJtgnD8+U+BXw9X1+p/9mW9puP9AaFUYQlSZIkSVI34BF5SZIkSZK6ERN5SZIkSZK6ERN5SZIkSZK6ERN5SZIkSZK6ERN5SZIkSZK6kb7lDqCjjBkzJlVVVZU7DEmSDsjChQufTSmNLXccPY3tAklSd9NWm6DHJvJVVVUsWLCg3GFIknRAImJFuWPoiWwXSJK6m7baBHatlyRJkiSpGzGRlyRJkiSpGzGRlyRJkiSpGzGRlyRJkiSpGzGRlyRJkiSpGzGRlyRJkiSpGzGRlyRJkiSpGzGRlyRJkiSpGzGRlyRJkiSpGzGRlyRJkiSpGzGRlyRJkiSpG+lb7gAkSepO6hub2Lijjo3b66ndXke/iqC6qrLcYUmSpF7ERF6S1GvVNTSxaUcdtTvqqN1ex6Yd9fl1HbXb67OEfUcdG7dny2zaXs/W3Q17rWPW4SO56YrTy/QOJElSb2QiL0nqEXY3NO5JxPccMd9Rx6Y8Cd+4vY6NO4qT83q2tUjKiw3pX8GoIf2pHNKfkYP7M23MEEYN6c+owf3z635UDu7PuOEDO/FdSpIkmchLkrqgXfWNex8d35En4dvrisrq8+Q8S9K31zXuc31DB/Rl1JAs8R41uD9HjB3KyDwRb07O+1GZ3x45uB8D+lZ04juWJEkqnYm8JKlD7apv3HMEfOOOula7rhd3a9+4o44dbSTlwwb03XNEfPTQ/hw5bigjB/encki/5qS8kJgPzo6m9+/r2K6SJKnnMJGXJJVsV33jXl3X95WIFz/eZlI+sO+erutj8qS8uTv7c4+YjxxkUi5JkmQiL0m91M66xj1J+MairuvNg7vV50fOm88v31m/76R8+MC+e5LuccMGctT4YXsl4pVD+uVHzpu7r/erMCmXJEk6UCbyktQD7KxrLBrQbe/ku+UR88I557vqm/a5vhGD+jFqcNZVffzwgRwzYfhzEvFRg/vtOZpuUi5JktR5TOQlqQtJKbGz0H39OdOfFR0xz88xLxwx393QdlJemZ9TftiIgRw7cfheXdcLyXkhUR85qB99TcolSZK6LBN5SeogKSV21DU2Hw0vOmJemJe8uTt7/Z6yun0k5RF5Up4fAZ80ciAzJw7fR9f1fowa3J8RJuWSJEk9jom8JJUgpcT2usa9uq63NrjbXvd31LeZlI8c1DzK+qSRg5iZHykfVZSIF7quVw7JkvKKPtHJ71ySJEldjYm8pF4npcS23Q17EvHaHc3ToW0qHvxt+97nl9c17jspLwzeVjm4P4dXDuaEySOy0df3TIXWf88555WD+zPcpFySJEkHyUReUreWUmLr7gY2bc+7rhe6rbfanb35nPP6xtTq+voUJ+VD+jOlcjCzJo98TiI+akjzEfPhA/vRx6RckiRJncREXlKXUUjK90rE9zG4W/F0aQ1NrSflFX1iT/f1ysH9mTp6MCdOGblX1/XC0fLCeeUm5ZIkSerqTOQldYiUElt2NexzlPU955S36NbeVlJenHxPGzOEkwvnj7fSdX3U4P4MG9jXpFySJEk9jom8pAOyZtNOHlu7Za9R1vck58XTpe2op7HNpLx5lPXpY4Zy8tS95yWvHLL3EfPhA/sSYVIuSZIkmchL2q9nt+3mlgfXMndRDQtWbNzrsb59Yq+u6jPGDX1OIr5nzvJ8RPZhA0zKJUmSpIPVqYl8RJwPfBWoAL6TUvp8i8enAt8DxgK1wJtSSqvzx6YA3wEOBxLw8pTS8s6LXupdtu6q548PP8NNi2u4Z8mzNDYljho/lH976dG84IjRjBkygFFD+jHUpFzSQbBNIEnSweu0RD4iKoBrgZcAq4H5ETE3pfRI0WJfAm5IKf0gIs4GPge8OX/sBuAzKaU/RcRQoPV5oCQdtF31jdz++DrmLq7hz4+uY3dDE5NGDuIdZ0znotkTOWbC8HKHKKkHsE0gSdKh6cwj8qcCS1JKywAi4qfAxUBxpX0s8P789m3Ab/JljwX6ppT+BJBS2tZJMUs9XkNjE39dtoGbFtXwh4eeZuvuBsYM7c+lpxzORbMncdKUkR5xl9TebBNIknQIOjORnwSsKrq/Gnh+i2UWA68i62p3CTAsIkYDRwGbIuJXwDTgVuCDKaXG4idHxOXA5QBTpkzpiPcg9QgpJe5buYnfLq7h/x6o4dltdQwb0JeXzpzARbMm8sIjRtO3ok+5w5TUc3V4mwBsF0iSeq6uNtjdVcA1EXEZcCewBmgki/PFwInASuBnwGXAd4ufnFK6DrgOoLq6uvXhsqVe7PGnt3LTojX89oEaVtXupH/fPpxzzDgunj2ROUePY2C/inKHKEkFh9QmANsFkqSeqzMT+TVkg9IUTM7L9kgp1ZDtfSc/5+3VKaVNEbEaWFTUBe83wGm0UmlL2tuq2h3MXVzD3EU1PP7MVir6BKfPGMO/nnMU5x03nuED+5U7REm9j20CSZIOQWcm8vOBIyNiGlllfSnwhuIFImIMUJtSagI+RDZabeG5IyNibEppPXA2sKDTIpe6mfVbs+niblq0hvtWbgLg5Kmj+OTFx/Hy4w9jzNAB5Q1QUm9nm0CSpEPQaYl8SqkhIq4E/kA21cz3UkoPR8QngQUppbnAHOBzEZHIutFdkT+3MSKuAv4c2ahbC4Fvd1bsUnewZVc9f3joaebm08U1JThmwjD+/fyjufCEiRxeObjcIUoSYJtAkqRDFSn1zFPGqqur04IF7qBXz7arvpF5j61j7qIa5j2+jrqGJg6vHMTFsyZx0eyJHDV+WLlDlHSAImJhSqm63HH0NLYLJEndTVttgq422J2k/WhobOKepRu4adEa/vjwM2zb3cCYoQN4w6lTuHj2RGYf7nRxkiRJUk9mIi91A9l0cRu5aVENNz+wlg3b6xg2sC8vP34CF82axGnTK50uTpIkSeolTOSlLiqlxGNPb90z4vyaTTsZ0LcP5z5vPBfOmsico8c6XZwkSZLUC5nIS13Myg07mLt4DXMX1/DEM9uo6BO8+MgxfOC8o3jJseMZ5nRxkiRJUq9mIi91Aeu27uLmB9Zy06IaFq3aBMApVaP41Ctn8vKZExjtdHGSJEmScibyUpls3tk8XdxflmbTxR172HA++LJjuHDWRCaNHFTuECVJkiR1QSbyUifaVd/Inx9dx02L1nD74+upa2xi6ujBXHHWDC6aNZEjnS5OkiRJ0n6YyEsdrL6xibuXPMtvF9Xwh4efZntdI2OHDeBNp03lotkTmTV5hNPFSZIkSSqZibzUAZqaEgtXbuSmRWu45cGnqd1ex/CBfbnghIlcPHsiz58+moo+Ju+SJEmSDpyJvNROUko8snYLcxfX8NtFNdRs3sXAftl0cRfPnsQZR41hQF+ni5MkSZJ0aEzkpUO0/Nnt2Vzvi2tYsm4bffPp4v79/GN4ybHjGTLAv5kkSZKk9mOGIR2EdVt28dsH1jJ30RoWr94MwKnTKvn0K2fy8uMPo3JI/zJHKEmSJKmnMpGXSrR5Rz2/e2gtcxfX8NdlG0gJjps4nP98+TFccMJEJjpdnCRJkqROYCIvtWFnXSO3PvoMNy2q4Y4n1lHfmJg2Zgj/cvaRXDR7IkeMHVruECVJkiT1MibyUgv1jU3c9eR65i6q4Y+PPMOOukbGDx/AP72giotnT2LmpOFOFydJkiSpbEzkJbLp4uYvr+WmxTX87sG1bNxRz4hB/bh49kQumjWJU6dVOl2cJEmSpC7BRF69VkqJh2vy6eIW17B28y4G9avgJceO56JZEznjqLH079un3GFKkiRJ0l5M5NXrLFu/bc90ccvWb6dvn+DMo8bywZdl08UN7u/fQpIkSVLXZcaiXuHpzbv4vwdquGlRDQ+u2UwEPH9aJW9/0XReNnMCo5wuTpIkSVI3cdCJfETMAFanlHa1YzxSu9m0o47fPfQ0Ny1aw71P1ZISHD9pBB95xfO44ISJTBgxsNwhSpIkSdIBKymRj4jPAo+nlH4Q2XDdfwTOATZHxPkppXs7MkipVDvqGvjTI88wd1ENdz65nvrGxPSxQ/jXc47kolkTme50cZJ0yCKiD0BKqSm/PwG4AHg0pXRPOWOTJKk3KPWI/BuB1+W3XwbMBk7Lyz8PnNXukUklqmto4s4n1jN3cQ1/euQZdtY3MmH4QN5y+jQumjWR4yY6XZwktbObgd8DX42IocACYAgwNCLellK6oazRSZLUw5WayI8HVue3Xw78PKX094ioJau8pU7V1JS496la5i5ewy0PPs3mnfWMHNyPS06axMWzJnJKVSV9nC5OkjpKNfDv+e1XAVuAaWQ7+K8CTOQlSepApSbyG4CpZMn8ecAHi55vtqROkVLioTVbuGnRGn77QA3PbNnN4P4VnHfseC6aPZEXzXC6OEnqJEOBTfnt84Bfp5TqI2IecG3ZopIkqZcoNZH/JfDjiHgCqAT+kJfPBpZ0QFzSHkvXb2Puomy6uKee3U6/iuDMo8bxkVdM5JznjXO6OEnqfCuB0yPit8BLgdfm5ZXAjrJFJUlSL1FqBvR+YAUwBfj3lNL2vPww4BsdEZh6t5pNO/m/B7Lk/aE1W4iAF0wfzTvOmM7LZh7GiMH9yh2iJPVmXwFuBLaRtQ/uzMvPAB4sV1CSJPUWJSXyKaUG4MutlP9Pu0ekXqt2ex23PLiWuYtr+PtTtQDMmjyCj15wLBeccBjjhztdnCR1BSmlb0XEArId/H8qjF4PLAU+Wr7IJEnqHfaZyEfESaWuJKV0X/uEo95m++58urjFNdz5xHoamhJHjB3C+19yFBfNmkjVmCHlDlGS1IqU0kJgYYuym8sUjiRJvUpbR+QXAIn9D2aXgIp2i0g93u6GRu584lluWrSGWx99hl31TUwcMZC3vTibLu7Yw5wuTpK6uoh4PnAOMA7Ya6TRlNK/lCUoSZJ6ibYS+WmdFoV6vMamxL3LNjB3cQ23PLiWLbsaqBzSn9ecPJmLZ0/i5CmjnC5OkrqJiLgK+ALZgLc1ZDv1C1KrT5IkSe1mn4l8SmlFZwainielxAOrN3PTohr+74Ea1m3dzZD+Fbz0uAlcOHsiL5oxhn4VThcnSd3QvwL/klK6ptyBSJLUG5U8b1dEHA+8AzgCeGtKaW1EvBJYkVK6v4PiUze0ZN3WPdPFLd+wg/4VfZhz9Fgunj2Js48Zx6D+nokhSd3ccOCWcgchSVJvVVIiHxHnAXOB3wFnA4Pyh44ALgNe2QGxqRtZs2knv11cw9xFNTyydgt9Al54xBjePWcGL505gRGDnC5OknqQnwDnA18vdyCSJPVGpR6R/xTw/pTS1yNia1H57cAH2j0qdQsbtu3mloeeZu6iNcxfvhGA2YeP5OMXHssrTjiMccOcLk6SeqhVwH9FxOnAA0B98YMppa+UJSpJknqJUhP5mbTeha4WqGy/cNTVbdvdwB8ffpq5i2u468lnaWxKHDluKFeddxQXzprI1NFOFydJvcDbgW3AC/NLsQSYyEuS1IFKTeRrgUnA8hblJwGr2zMgdT27Gxq5/fH1zF1Uw62PPsPuhiYmjRzE5WdM56JZEzlmwjCni5OkXiSl5Mw2kiSVUamJ/I+BL0bEP5Dtae8bEWcCXwK+31HBqXwamxJ/XbqBuYvX8LuHnmbrrgZGD+nP6045nItnT+TEw50uTpIEETEUSCml7eWORZKk3qLURP4jwPXACiCAR/LrHwOf6ZDI1OlSSixatYmbFtVw84NrWb91N0MH9OWlx03gotkTOf2I0fR1ujhJEhARVwD/QdZjj4hYDfx3SskB8CRJ6mAlJfIppXrgjRHxMeBEoA9wf0rpyY4MTp3jiWeap4tbWbuD/n37cPbR47h49kTOOmYcA/s5XZwkqVlE/CfwIbKeeXfnxS8GPh8Rw1NKny9bcJIk9QIlzyMPkFJaGhHP5Le3dUxI6gyranfw2wey6eIee3orfQJOnzGG95ydTRc3fKDTxUmS9umdwOUppZ8Ulf05Ip4EPguYyEuS1IFKTuQj4r3A+2nuQldDNirt/0sppQ6JTu3q2W27ueXBtdy0qIaFK7Lp4k6aMpJPXHgsrzhhImOHDShzhJKkbmIcML+V8r8D4zs5FkmSep2SEvmI+AJwOfBF4K958QuAjwGHAf/eIdHpkG3dVc8fHn6GuYtruGdJNl3cMROG8W8vPZqLZk3k8MrB5Q5RktT9PAG8Afhki/I3AI93fjiSJPUupR6Rfzvw9pTSL4rK5kXE48C3KDGRj4jzga8CFcB3Wp5DFxFTge8BY8mmvHtTSml10ePDyQba+01K6coSY+91dtU3cvvj67hpUQ1/fmwddQ1NTB41iHeeOZ2LZk3i6AnDyh2iJKl7+wTw84g4A7gnLzsdOBN4bSkrsE0gSdLBO5Bz5B/YR1lJw5hHRAVwLfASsrnn50fE3JTSI0WLfQm4IaX0g4g4G/gc8Oaixz8F3HkAMfcaDY1N/GXpBuYuruEPDz3N1t0NjBnanzecOoWLZk/kxMNHOte7JKldpJR+FRHPB94HXJAXPwqcmlK6f3/Pt00gSdKhKTWRvwG4AvjXFuXvAm4scR2nAktSSssAIuKnwMVke9MLjiU7Dx/gNuA3hQci4mSy8+5+D1SX+Jo9WkqJ+1ZuYu6iNdz84Fqe3VbHsAF9eenMCVw8eyIvmO50cZKkjpFSWgi86SCfbptAkqRDsM9EPiK+1mK5N0XES4G/5WXPByYCPyrxtSYBq4rur87XUWwx8CqyrnaXAMMiYjSwEfgyWYPh3DZivpzsXH6mTJlSYljdz+NPb+WmRWuYu7iG1Rt3MqBvH8553jgumjWROUc7XZwkqf1FRGVKqbZwu61lC8u1ocPbBHmcvaJdIEnqfdo6In98i/sL8+up+fXT+eWYdoznKuCaiLiMrLvcGqAReDdwS0ppdVvdw1NK1wHXAVRXV/eokfRX1e5g7uJsurjHn9lKRZ/gRTPG8L5zj+K848YzzOniJEkda31EHJZSWgc8C7RWz0Ze3h57lA+pTQA9u10gSerd9pnIp5TOaufXWgMcXnR/cl5W/Jo1ZHvfiYihwKtTSpsi4gXAiyPi3cBQoH9EbEspfbCdY+xS1m/dzc0P1DB3cQ33rdwEQPXUUXzq4uN42fGHMWao08VJkjrN2WSDzhVuH0pibJtAkqRDcCCD3R2q+cCRETGNrLK+lGyamj0iYgxQm1JqAj5ENlotKaU3Fi1zGVDdUyvsLbvq+f1DT/PbfLq4pgTHTBjGf5x/DBfOOozJo5wuTpLU+VJKdxTdvv0QV2ebQJKkQ1ByIh8RZwGvB6YA/YsfSymdvb/np5QaIuJK4A9kXe6+l1J6OCI+CSxIKc0F5gCfi4hE1o3uilLj68521Tcy77F1zF1Uw7zHs+niplQO5t1zZnDR7IkcNd7p4iRJXUdENAKFbvbF5aOBdSmlNrvW2yaQJOnQREr77xmX7/H+JvBrsgFnbgKOAqYBP+yK87dWV1enBQsWlDuMfWpobOKepRu4adEa/vjwM2zb3cDYYQO44ITDuGjWRGY7XZwk9UoRsTCl1KVHYo+IJmBCK4n8RGBpSmlQeSLbt67eLpAkqaW22gSlHpG/CrgypfSdiNgKfCiltCwirgG2tVegPV1TU+K+lRuZu7iGmx9Yy4btdQwb2JdXHH8YF82eyGnTR1PRx+RdktQ1RURhOrgEvDMiitsAFcCLgcc6PTBJknqZUhP56cCt+e3dZIPLAFwD3A54bto+pJR4dO1W5i6u4beLa1izKZsu7txjx3PxrImcefRYBvR1ujhJUrfwnvw6gLeTjSJfUAcsB97ZyTFJktTrlJrIbwAKJ2qvAWYCDwCjgS7Xfa4rWLlhB3MXr+GmRTU8uW4bFX2CFx85hqteehQvOXYCQwd05jiDkiQdupTSNICIuA14VUppY5lDkiSpVyo1m7wLOA94EPg58LWIeAlwDvCnDoqt21m3ZRf/98Ba5i6uYdGqTQCcWlXJp145k5fPnMBop4uTJPUM55Mdld9LRAwEmlJKdZ0fkiRJvUepifyVwMD89ueABuB0sqT+Ux0QV7exeWc9f3joaW5avIa/Lt1AU4JjDxvOh152DBfMmsikkXZYkCT1OD8H7gC+0qL8nWSjzb+yk+ORJKlXKSmRTynVFt1uAv4bICKGkQ1sc0uHRNdF7axr5M+PPcPcRTXc/vh66hqbqBo9mCvPPpKLZk1kxrih+1+JJEnd1+nAh1sp/xPwn50ciyRJvc6hnqg9Hfgt2Ui1PVp9YxN3L3mWuYtq+OPDT7O9rpFxwwbw5hdM5aJZEzlh8gini5Mk9RaDyXrntdRE85g6kiSpgzjiWgm+/MfH+eHfVrBxRz3DB/blwlkTuWj2RJ4/zeniJEm90gPA64GPtyh/A/BQ54cjSVLvYiJfgrqGJl505FgumjWRM44a43RxkqTe7pPATRExA5iXl50DvBa4pGxRSZLUS5jIl+BDL39euUOQJKnLSCndEhEXAh8BvpYX3w9clFL6XfkikySpd2gzkY+IV+3n+VXtF4okSeouUkq/B35f7jgkSeqN9ndE/hclrCO1RyCSJEmSJGn/2kzkU0p9OisQSZLUdUXEFmB6SunZiNhKGzvyU0rDOy8ySZJ6H8+RlyRJpXgPsDW/fWU5A5EkqbczkZckSfuVUvpBa7clSVLns+u8JEmSJEndiEfkJUnSfkVEEyUOcJtSqujgcCRJ6tVM5CVJUin+geZEfjzwSeDXwF/zshcArwQ+3umRSZLUy5SUyEdEH4CUUlN+fwJwAfBoSumejgtPkiR1BSmlPVPSRsRc4EMppW8XLfK9iPg7WTL/9U4OT5KkXqXUc+RvJhutlogYCiwAvgjcHhH/2EGxSZKkruls4LZWym8D5nRuKJIk9T6lJvLVwLz89quALcA44J+BqzogLkmS1HU9C7ymlfLXAOs7ORZJknqdUs+RHwpsym+fB/w6pVQfEfOAazsiMEmS1GV9DPh+RJxF8znypwHnAm8rW1SSJPUSpR6RXwmcHhFDgJcCf8rLK4EdHRGYJEnqmlJKNwAvJDsyf1F+2QCc7hzzkiR1vFKPyH8FuBHYBqwA7szLzwAe7IC4JElSF5ZSuhd4Y7njkCSpNyopkU8pfSsiFgKHA38qjF4PLAU+2lHBSZKkrikixgNvBqYDH0spPRsRpwM1KaWnyhudJEk9W8nzyKeUFpCNVl9cdnO7RyRJkrq0iDgZ+DPwFHAc8CWybvYvAY4C3lC+6CRJ6vlKnUf+/W09nlL6SvuEI0mSuoEvAV9NKX08IrYWlf8BeEuZYpIkqdco9Yj8e1rc7wccBuwE1pGdQy9JknqHk2l9dPq1wPhOjkWSpF6n1HPkp7Usy8+N+z7w7fYOSpIkdWk7gVGtlB9DtoNfkiR1oFKnn3uOlNIzwIeBL7RfOJIkqRu4Cfh4RAzI76eIqAL+G/hl2aKSJKmXOOhEvuj5dqGTJKl3uQqoBNYDg4G7gSXAJuAj5QtLkqTeodTB7l7VsojsHPkrgLvaOyhJktSlNQBzgDOAk8h27N+XUrq1nEFJktRblDrY3S9a3E9ke+HnAR9o14gkSVKXFREVwGZgVkppHllbQJIkdaJSB7s71C74kiSpB0gpNUbECqB/uWORJKm3OugEPSL6tWcgkiSp2/gU8PmIGFPuQCRJ6o1KSuQj4l8i4tVF978H7IyIxyPi6A6LTpIkdUVXAS8C1kTE0oh4oPhS7uAkSerpSj1H/l+AtwJExBnAa4E3AK8Gvgxc0CHRSZKkruiXZOPlSJKkMig1kZ8EPJXfvhD435TSzyPiQRy1XpKkXiWl9IlyxyBJUm9W6jnyW4Bx+e2XAH/Ob9cDA9s7KEmS1PVExOCIuDYi1kTEuoj4sefJS5LU+Uo9Iv9H4NsRcR8wA/hdXn4czUfqJUlSz/ZfwGXAj4BdwOuBb5CdcidJkjpJqYn8FcBngCnAa1JKtXn5ScBPOiIwSZLU5bwKeFtK6acAEfFD4J6IqEgpNZY3NEmSeo9S55HfArynlfKPt3tEkiSpqzqcorFxUkp/j4gGYCKwqmxRSZLUyxzQPPIRMTEiZkfEScWXA3j++fmUdUsi4oOtPD41Iv6cT19ze0RMzstnR8RfI+Lh/LHXHUjckiSpXVQAdS3KGii9h98etgkkSTp4JVW8EXEi8EPgGCBaPJzIKvb9raMCuJZssLzVwPyImJtSeqRosS8BN6SUfhARZwOfA94M7AD+MaX0ZERMBBZGxB9SSptKiV+SJLWLAH4YEbuLygaSjaOzo1CQUrqozZXYJpAk6ZCUugf9OrIuc/8M1HBwc8eeCixJKS0DiIifAhcDxZX2scD789u3Ab8BSCk9UVggpVQTEeuAscCmg4hDkiQdnB+0UvbDg1iPbQJJkg5BqYn8scCJxZXnQZjE3ufPrQae32KZxWQD6XwVuAQYFhGjU0obCgtExKlAf2BpyxeIiMuBywGmTJlyCKFKkqSWUkpvaadVdXibIH/cdoEkqUcq9Rz5B4EJHRlI7irgzIi4HzgTWAPsGQU3Ig4DbgTeklJqavnklNJ1KaXqlFL12LFjOyFcSZLUQQ6pTQC2CyRJPVepR+T/E/hCRHyELKmvL36waDq6tqwhG+22YHJeVryeGrK970TEUODVhXPeImI4cDPw4ZTS30qMW5IkdT22CSRJOgSlJvK35td/ZO/z44MSB7sD5gNHRsQ0ssr6UuANxQtExBigNt+z/iHge3l5f+DXZIPe/KLEmCVJUtdkm0CSpENQaiJ/1qG+UEqpISKuBP5Alvh/L6X0cER8EliQUpoLzAE+FxEJuBO4In/6PwBnAKMj4rK87LKU0qJDjUuSJHUu2wSSJB2aSOlgBqDv+qqrq9OCBQvKHYYkSQckIhamlKrLHUdPY7tAktTdtNUmKPWIPBFxPPAO4AjgrSmltRHxSmBFSun+dolUkiR1SRFxUqnLppTu68hYJEnq7UpK5CPiPGAu8DvgbGBQ/tARwGXAKzsgNkmS1HUsIBsXJ/azXKlj50iSpINU6hH5TwHvTyl9PSK2FpXfDnyg3aOSJEldzbRyByBJkjKlJvIzgVtaKa8FKtsvHEmS1BWllFaUOwZJkpTpU+JytcCkVspPAla3XziSJKk7iIjjI+KaiPhdRByWl70yIk4sd2ySJPV0pSbyPwa+GBGTyc596xsRZwJfAm7oqOAkSVLXk4+dM59sJ3/LsXM+Xq64JEnqLUpN5D8CPAWsAIYCjwDzgLuBz3RMaJIkqYsqjJ1zCVBXVH47cGpZIpIkqRcp6Rz5lFI98MaI+ChZd/o+wP0ppSc7MjhJktQlOXaOJEllVPI88gAppWXAMoCImBERA1NKuzokMkmS1FUVxs5Z3qLcsXMkSeoEJXWtj4jPRsQ/5bcjIv4EPAGsjYjnd2SAkiSpy3HsHEmSyqjUc+TfCDye334ZMBs4jayy/nz7hyVJkrowx86RJKmMSu1aP57mrnIvB36eUvp7RNQCCzokMkmS1CUVjZ3zMeBEHDtHkqROVWoivwGYSpbMnwd8sOj50QFxSZKkLi6ltBRYWu44JEnqbUpN5H8J/DginiAbjfYPeflsYEkHxCVJkrqQiPheqcumlN7akbFIktTblZrIv5/sPLgpwL+nlLbn5YcB3+iIwCRJUpcytsX9M4Am4MH8/kyyLvZ3dmZQkiT1RqXOI98AfLmV8v9p94gkSVKXk1K6sHA7Ij4E7ATeUti5HxFDgO/SnNhLkqQOUvI88hExmKwr/Tj2Hu0+pZR+3c5xSZKkrutfgHOKeuiRUtoeEZ8C/owj10uS1KFKSuQj4lzgJ8DoVh5OQEV7BiVJkrq0ocBEsmnnih0GDO78cCRJ6l1KnUf+q8DNwOSUUp8WF5N4SZJ6l18C34+ISyOiKr9cSta1/ldljk2SpB6v1K71VcBFKaWaDoxFkiR1D+8iGzvneqBfXtZAlshfVaaYJEnqNUpN5O8Bjsa5YiVJ6vVSSjuBd0fEvwFH5MVLi8+ZlyRJHafURP6bwJciYiLZaLT1xQ+mlO5r78AkSVKX10g2BV3Kb0uSpE5QaiL/i/z6ulYec7A7SZJ6kYjoC3wOuBLoDwSwOyKuBj6cUqpv6/mSJOnQlJrIT+vQKCRJUnfyBeD1wDuBu/OyF5Ml933wPHlJkjpUSYl8SmlFRwciSZK6jTcAb00p3VJUtjQi1gPfwURekqQOVer0c0TECRFxQ0QsiIj5EfGDiJjZkcFJkqQuaQStD4C7FBjZuaFIktT7lJTIR8RFwH3A4cDvgN8DU4D7I+LCjgtPkiR1QYuBf2ml/F+BRZ0biiRJvU+p58h/GvhMSunjxYUR8cn8sd+2d2CSJKnL+nfglog4F/hbXnYaMBF4WdmikiSplyi1a/1RwI2tlN9INr+8JEnqJVJKd5K1DX4BDM0v/wscnVK6u63nSpLUk9Q3NrF41Sa+c9cy3nnjQp7dtrtTXrfUI/LrgJOBJS3KTwaeadeIJElSl5dSqgE+XO44JEnqTFt31XP/yk0sWLGRBctruX/lJnbWNwJweOUg1m7axZihAzo8jlIT+W8D34qIGcBf8rLTyUal/WJHBCZJkrqWiKgsZbmUUm1HxyJJUmd4evMuFqyoZcHyjcxfXsuja7fQlKBPwLETh/O6Uw7nlKpKqqtGMX74wE6L60DOkd8GfAD4VF5WA3wc+FoHxCVJkrqe9ft5PIAEVHRCLJIktaumpsSS9duYv7yWhcs3Mn9FLatqdwIwqF8FJ04ZyZVnH8kpVaM4ccoohg4oNZ1uf6XOI5+A/wH+JyKG5WVbOzIwSZLU5QSwArgeuIMsaZckqVva3dDIg6s3M3951k1+wYqNbN5ZD8CYoQM4pWoUl71wGqdUjeJ5hw2nX0XJs7d3uDYT+TxpvwT4TUppCzQn8BExArgY+GVKaXtHBypJksruWOBtwLuANwHfB67Pz5eXJKlL27yjnoUra/ck7otXb6auoQmA6WOHcP5xE6iuGsUpVZVMHT2YiChzxPu2vyPy7wTmpJRuaPlASmlzRPwDMA74UkcEJ0mSuo6U0mPAv0XEh4ALgbcCH4uIPwPfSSn9uqwBSpKUSymxeuNOFqxoTtyfeGYbAH37BMdPHsFlL6yieuooTp46itGdMEBde9pfIv864KNtPH4N8BlM5CVJ6jVSSg3Ar4FfR8Rk4AbgFxEx1oHuJEnl0NiUeHTtFhauyAalW7B8I09v2QXAsAF9OWnqKC6aNZHqqkpmTR7JoP7deziX/SXyRwIPt/H4I8AR7ReOJEnqDiJiGtkR+X8kO1f+U8DmsgYlSeo1dtQ1sGjVpj2jyd+/chPbdjcAcNiIgZw6rZJTqkZx8tRKjp4wjIo+Xbeb/MHYXyIfZF3nV+7j8XFA1znjX5IkdZiIGAi8luw8+VOBucDbgVvzgXElSeoQz27bzYK8i/z8FRt5eM1mGpoSEXD0+GG88sSJ+TRwlUwaOajc4Xa4/SXyDwHnAQv28fj5tH3EXpIk9RxPA5vIRq1/O1DoRj+qeEAgu9dLkg5FSomnnt2+52j7ghUbeerZbHz1/n37MPvwkbzjzOlUV1Vy0pRRjBjUr8wRd779JfLfA74aEQ+nlG4qfiAiXgl8CPjXDopNkiR1LcPzy8dofQwd55GXJB2w+sYmHq7Zkh1tz89v37C9DoCRg/tRPbWSS085nOqqSmZOGs6AvlYzbSbyKaXvRMQcssFsHgceyx96Htn58z9PKX2nQyOUJEldxVnlDkCS1P1t3VXPfSs37UncF63axK76bBq4qaMHM+focfk0cKOYPmYofXrY+e3tYX9H5EkpvSki5gJvAI4i29v+GPCxlNLPOzg+SZLURaSU7ih3DJKk7ufpzbvyI+3ZVHCPPb2FpgR9Ao6bOILXnzolO7996ijGDR9Y7nC7hf0m8gB5wn7ISXtEnA98lazL3XdSSp9v8fhUsu78Y8nOu3tTSml1/tg/AR/JF/10SukHhxqPJEkqD9sEktQzNTUlnly3bU/ivmDFRlZv3AnA4P4VnDRlFO85+0hOqapk9pSRDB1QUkqqFjrtU4uICuBa4CXAamB+RMxNKT1StNiXgBtSSj+IiLOBzwFvjohK4ONANdm5dwvz527srPglSVL7sE0gST3HrvpGHlyzec+57QuW17JlVzYN3NhhAzilahRvPX0ap1RV8rzDhtG3wknP2kNn7v44FViSUloGEBE/BS4mm4u+4Fjg/fnt24Df5LdfCvypMApuRPyJbMT8n3R82JIkqZ3ZJpCkbmrj9joWrtjI/BW1LFy+kQdWb6auMTu/fca4obz8+MOorsrmcJ9SOZjiWU3UfjozkZ8ErCq6vxp4fotlFgOvIutqdwkwLCJG7+O5k1q+QERcDlwOMGXKlHYLXJKk3i4ipgCr2mm++A5vE+Qx2y6QpEOQUmL1xp3Mz89tX7C8lifXbQOgX0Vw/KQRvOX0KqqrKjl56igqh/Qvc8S9xz4T+XausEt1FXBNRFwG3AmsARpLfXJK6TrgOoDq6urOjFuSpJ7uKeAwYF1EzANelVLa1IGvd0htArBdIEkHqqGxicee3rpn7vYFy2t5ZstuAIYN7MvJU0fxyhMnUT11FLMOH8nAfk4DVy5tHZFv7wp7DXB40f3JedkeKaUasr3vRMRQ4NUppU0RsQaY0+K5tx9CLJIk6cBsBcYA68jq5H6HsC7bBJLUBeyoa2DRyk3Z0fYVtdy3YiPb67J9ppNGDuK06aP3dJM/atwwp4HrQtpK5NuzwgaYDxwZEdPIKutLyaa02yMixgC1KaUm4ENko9UC/AH4bESMyu+flz8uSZI6x63AvIh4NL//64ioa23BlNLZ+1mXbQJJKoP1W3ezcEVzN/mHarbQ2JSIgKPHD+NVJ02mumoU1VWVTBo5qNzhqg1tJfLtWWGTUmqIiCvJKuAK4HsppYcj4pPAgpTSXLIdBp+LiETWje6K/Lm1EfEpsoof4JOFQW4kSVKneDPwVmAGcCbwOLDjYFZkm0CSOl5KiWXPbt8zd/uC5bUs35Bttgf07cPsw0fyrjOPoLpqFCdOGcWIQYd63FadKfZ1CnxEDKK5wv5X4Pvso8JOKb2nowI8WNXV1WnBggXlDkOSpAMSEQtTStXljqMtEXEbcEkHnyPfrmwXSOrp6hqaeKhmMwuXb9xzjnvt9uw47KjB/fZ0ka+uqmTmxBH07+s0cF1dW22CfR6RTyntJJvjlYiYDXygO1XYkiSpY6SUzirczs9fTyml7WUMSZJ6nS276rlvxUYW5In7olWb2N2QTQNXNXowZx8zjlOqRnHy1EqOGDvEaeB6mJKmn7PCliRJxSLiCuA/yKd+i4jVwH+nlL5e1sAkqYdau3nnni7y85dv5LGnt5ASVPQJjps4nDc+f2qWuFeNYtywgeUOVx2s5HnkrbAlSRJARPwn2QBzXwLuzotfDHw+IoanlD5ftuAkqQdoako8sW7rnsR9wfKNrNm0E4Ah/Ss4aeoo3nvOUZxSlU0DN2RAyWmdeoiSvnErbEmSVOSdwOUppZ8Ulf05Ip4EPgvYLpCkA7CrvpEHVm/Ozm1fXsvCFRvZsqsBgHHDBnBKVSVvf/E0Tqmq5JgJw+hb4fntvV2pu26ssCVJUsE4mkeNL/Z3YHwnxyJJ3c7G7XUsWFHoJl/LQ2u2UNeYnd9+5LihvOKEw6ieWskpVZUcXjnI89v1HKUm8lbYkiSp4Amyed8/2aL8DWTT0kmSciklVtXuzEeSz85vX7JuGwD9KoITJo/kLS+q4pSplZw8dRSjhvQvc8TqDkpN5K2wJUlSwSeAn0fEGcA9ednpZPPLv7ZcQUlSV9DQ2MSja7cyP+8iP395Leu27gZg+MC+VFdVcsmJkzilqpITJo9gYL+KMkes7qjURP4TWGFLkiQgpfSriHg+8D7ggrz4UeDUlNL95YtMkjrf9t0NLFq1KT+/fSP3rdzIjrpGACaNHMQLjxidz+FeyZHjhtKnj93kdehKnX7OCluSJO2RUloIvKnccUhSZ1u3ddeeudsXrtjIwzVbaGxKRMAxE4bzmpMnU11VSfXUUUwcOajc4aqHKnmeAitsSZIkSb1JSoml67fvmbt9wYpaVmzYAcDAfn2YffhI3j3nCKqrKjlxykiGD+xX5ojVWzjhoCRJkiQBdQ1NPFSzuTlxX17Lxh31AIwe0p+Tp47iTc+fSnXVKI6bOIL+fZ0GTuVhIi9JkiSpV9q8s577Vm7ck7gvXrWJ3Q3ZNHDTxgzh3OeN55SqSqqrRjFtzBCngVOXYSIvSZIkqVdYs2knC/JB6eYvr+XxZ7aSElT0CWZOHM6bTpvKKVWjOHlqJWOHDSh3uNI+mchLkqQDEhGvBH6bUmosdyyStC+NTYknntm6Vzf5ms27ABjSv4KTpo7i5ccfRvXUUcyeMpLB/U2N1H2U9Gu1wpYkSUV+BGyNiB8A300pPVHugCRpV30ji1dtYsGK5hHlt+5qAGD88AFUV1Vy+dRRVFdVcsyEYfSt8Px2dV+l7naywpYkSQUTgDcAbwGuioi/At8Ffp5S2l7WyCT1GrXb67Ju8nni/tCazdQ3JgCOGj+UC2dN5JSqUVRPrWTyqEGe364epdRE3gpbkiQBkFLaCnwL+FZEHAe8Ffgc8NWI+BnZTv+/lTNGST1LSomVtTv2dJGfv7yWpeuzNKR/RR9OmDyCt71oen5++yhGDu5f5oiljlVSIm+FLUmSWpNSejgi/gfYDvw78Drgsoi4D/jnlNIDZQ1QUrfU0NjEI2u3FCXuG3l2224ARgzqR/XUUbzm5MM5pWoUMyeNYGC/ijJHLHWuAx7RwQpbkiRFRD/gErKd++cA9wLvBH4GjAI+m99+XrlilNR9bN/dwP0rNzF/eS0LVtRy/8pN7KjLhueaPGoQLz5yDNVVozilqpIZY4fSp4/d5NW7lZzIW2FLkiSAiLgaeD2QgBuB96eUHilaZGdEfBCoKUd8krq+dVt2MX9586B0j6zdQmNTok/A8w4bzmtPnkx1Pn/7YSMGlTtcqcspddR6K2xJklRwLHAl8KuUUt0+lnkWOKvzQpLUVaWUWLp+257EfcHyjays3QHAwH59OPHwUVwx5wiqqyo5ccpIhg3sV+aIpa6v1CPyVtiSJAmAlNI5JSzTANzRCeFI6mJ2NzTy0JrN+fntG1m4opaNO+oBGDO0P9VTK/nHF0yluqqS4yYOp5/TwEkHrNTB7qywJUkSABHxGWBVSumbLcrfCUxKKX20PJFJKofNO+u5b0Xz0fZFqzdR19AEwPQxQ3jJseOprqrklKpKqkYPdho4qR2U2rXeCluSJBW8GXhtK+ULgQ8BtgukHm5XfSP/u3A1P/rbCh5/ZispQd8+wXGTRvCPp03dc377mKEDyh2q1COV2rXeCluSJBWMA9a3Ur4BGN/JsUjqRDvqGvjxvSu57s5lrNu6m1mHj+T95x5FdVUlsw8fyaD+TgMndYZSE3krbEmSVLASeDGwrEX5GcDqzg9HUkfbuqueG/66gu/e/RS12+t4wfTR/L/XzeYFR4y2q7xUBqUm8lbYkiSp4FvA/0REf2BeXnYO8Dngv8sWlaR2t2lHHd+7ZznX3/MUW3Y1MOfosVx51gyqqyrLHZrUq5WayFthS5IkAFJKX46IMcDXgP55cR3w1ZTSF8oXmaT28uy23Xznrqe48a/L2V7XyEuPG8+VZx3J8ZNHlDs0SZQ+ar0VtiRJ2iOl9KGI+DTZFLUAj6aUtpUzJkmH7unNu/jWnUv5yd9XUtfQxAUnTOSKs2Zw9IRh5Q5NUpFSj8hbYUuSpL2klLYD88sdh6RDt6p2B9+4Yym/WLCaxpS45MRJvHvOEUwfO7TcoUlqRcmJPFhhS5KkTEScBbwemEJzbz0AUkpnlyUoSQds6fptfP22pfxm0RoqInht9WTeeeYRHF45uNyhSWpDyYm8FbYkSQKIiMuAbwK/BuYANwFHAdOAH5YtMEkle+zpLVwzbwk3P7iWAX378E8vqOLyM6YzYcTAcocmqQQlJfJW2JIkqchVwJUppe9ExFbgQymlZRFxDeBpd1IX9sDqTVw9bwl/euQZhvSv4J1nHsHbXjSNMUMHlDs0SQeg1CPyVtiSJKlgOnBrfns3UDiJ9hrgduCDZYhJUhsWLK/l6nlLuOOJ9Qwf2Jd/PedI3nJ6FSMH99//kyV1OaUm8lbYkiSpYANQGMJ6DTATeAAYDQwqV1CS9pZS4i9LN3D1vCf527JaKof059/PP5o3nzaVYQP7lTs8SYeg1ETeCluSJBXcBZwHPAj8HPhaRLwEOAf4UzkDk5Ql8Lc9vo6r5y3h/pWbGDdsAB+94Fhef+rhDO5/QGNdS+qiSv0nW2FLkqSCK4HCiFifAxqA08naCJ8uV1BSb9fUlPjjI09z9bwlPFyzhUkjB/HpV87kNSdPZmC/inKHJ6kdlZrIW2FLkiQioi9wKfAbgJRSE/Df5YxJ6u0aGpu4+cG1XDNvCU+u28a0MUP44mtO4JUnTqJfRZ9yhyepA+w3kbfCliRJBSmlhoj4InBzuWOReru6hiZ+c/8avn77EpZv2MFR44fytdefyCuOP4yKPlHu8CR1oP0m8lbYkiSphb8BJwMryh2I1Bvtqm/kfxes4pt3LGPNpp3MnDScb77pZM47djx9TOClXqHUrvVW2JIkqeDbwJciYgqwENhe/GBK6b6yRCX1cDvqGvjxvSu57s5lrNu6m5OmjOTTl8xkzlFjiTCBl3qTUhN5K2xJklTw4/z6K608lgBH1ZLa0dZd9dzw1xV89+6nqN1exwuPGM3/u3Q2L5g+2gRe6qVKTeTbpcKOiPOBr+bLfyel9PkWj08BfgCMzJf5YErplojoB3wHOCmP+YaU0udKjF2SJLWvaYe6AtsE0v5t3F7H9/+ynOvveYotuxo46+ixXHn2DE6eWlnu0CSVWamJfHtU2BXAtcBLgNXA/IiYm1J6pGixjwA/Tyl9IyKOBW4BqoDXAgNSSsdHxGDgkYj4SUpp+aHGJUmSDkxK6ZBOtbNNILVt/dbdfOfuZfzwryvYXtfIS48bz5VnHcnxk0eUOzRJXURJifyhVti5U4ElKaVlABHxU+BioLjSTsDw/PYIoKaofEg+gv4goA7Y0g4xSZKkAxQRr2rr8ZTSr/azCtsEUivWbt7Jt+5Yxk/+vpL6xiYuOGEiV5w1g6MnDCt3aJK6mJIS+XaosAEmAauK7q8Gnt9imU8Af4yI9wBDgHPz8l+QVfBrgcHA+1JKta3EeTlwOcCUKVNKCEmSJB2EX+yjPOXX+zvlrsPbBGC7QN3Hyg07+MYdS/nFwlWkBJecOIl3zTmC6WOHljs0SV1UqV3rD7XCLtXrgetTSl+OiBcAN0bETLI9943ARGAUcFdE3FrYk78nmJSuA64DqK6uTkiSpHaXUupTfD8/On4i8EXgw+30MofUJsjjtF2gLm3p+m1ce9sSblpUQ0UErzvlcN5xxhEcXjm43KFJ6uJK7VrfHhX2GuDwovuT87JibwPOz1/zrxExEBgDvAH4fUqpHlgXEfcA1cBzKm1JktS5UkoNZOe5/yfwDWDWfp5im0C92qNrt3DtbUu4+cG1DOjbh396QRWXnzGdCSMGljs0Sd1En/0v8lwppYaU0nzgP4Gvl/i0+cCRETEtIvoDlwJzWyyzEjgHICKeBwwE1uflZ+flQ4DTgMcOJnZJktRhNgFHlLCcbQL1SotXbeKfb1jAy756F7c9to53nnkEd//H2XzswmNN4iUdkFK71u/LJkqrsEkpNUTElcAfyLrify+l9HBEfBJYkFKaC3wA+HZEvI+s2/5lKaUUEdcC34+Ih4EAvp9SeuAQY5ckSQchIk5qWQQcBvwHcP/+nm+bQL3N/OW1XD1vCXc+sZ7hA/vy3nOP5LIXVjFycP9yhyapm4qU9n/K2H4qbFJKL27/0A5NdXV1WrBgQbnDkCTpgETEwpRSdbnjaEtENJEl19Hiob8Bb0kpPd75UbXNdoE6W0qJvyzdwNf+/CT3PlXL6CH9efuLp/Om06YwbGC/cocnqRtoq01Q6hH5BbRRYR9CbJIkqfuZ1uJ+E7A+pbSrHMFIXUlKidseX8fV85Zw/8pNjB8+gI9dcCyvP3UKg/q31/jQknq7UhN5K2xJkgRASmlFuWOQupqmpsQfHn6aq+ct4ZG1W5g8ahCfuWQmrzl5MgP6msBLal+ljlpvhS1JkgCIiM8Aq1JK32xR/k5gUkrpo+WJTOp8DY1N/N8Da7n2tiU8uW4b08YM4YuvOYFXnjiJfhUHNa60JO1XSYm8FbYkSSryZuC1rZQvBD4E2C5Qj1fX0MRv7l/D129fwvINOzh6/DC+9voTecXxh1HRp+XZqJLUvkrtWm+FLUmSCsaRTQXX0gZgfCfHInWqXfWN/HzBKr51xzLWbNrJ8ZNG8K03n8xLnjeePibwkjpJqYm8FbYkSSpYCbwYWNai/AxgdeeHI3W8HXUN/PjelXzrzmWs37qbk6eO4jOXzOTMo8YSYQIvqXOVmshbYUuSpIJvAf8TEf2BeXnZOcDngP8uW1RSB9iyq54b/7qC79y1jI076nnhEaP56qWzecH00Sbwksqm1ETeCluSJAGQUvpyRIwBvgb0z4vrgK+mlL5Qvsik9rNxex3fv+cpvv+X5Wzd1cBZR4/lyrNncPLUynKHJkklj1pvhS1JkvZIKX0oIj4NHJsXPZpS2lbOmKT2sG7rLr5711Pc+LcV7Khr5PzjJnDl2TOYOWlEuUOTpD1KPSJvhS1JkgCIiAlA35TSamB+UflkoD6l9EzZgpMO0trNO/nWHcv4yd9XUt/YxIWzJvLuOTM4esKwcocmSc9R6vRzVtiSJKngh8DPgG+3KH8p8DrgvE6PSDpIKzfs4Bt3LOUXC1eRElxy4iTeNecIpo8dWu7QJGmfSj0ib4UtSZIKqoErWim/C/hiJ8ciHZQl67bx9duXcNOiGioieN0ph/OOM47g8MrB5Q5Nkvar1ETeCluSJBX0BQa0Uj5wH+VSl/Ho2i1cc9sSbnlwLQP69uGyF1Zx+RnTGT98YLlDk6SSlZrIW2FLkqSCe4F35ZdiV1B0Cp7UlSxetYmr5y3h1kefYeiAvrzrzCN464umMWaoTVlJ3U+pibwVtiRJKvgwMC8iTqB5WtqzgROBc8sWldSK+ctruXreEu58Yj0jBvXjveceyVteOI0Rg/uVOzRJOmilJvJW2JIkCYCU0t8i4gXAvwGvyovvB96dUlpcvsikTEqJe5Zs4Op5T3LvU7WMHtKf/zj/GN502hSGDTSBl9T9lTqPvBW2JEnaI6//39SyPCLOTSndWoaQJFJKzHtsHVfPW8KiVZsYP3wAH7vgWF5/6hQG9a8od3iS1G4OZB55K2xJkvQcETEJeAvwVmAqYMakTtXUlPj9w09zzbwlPLJ2C5NHDeIzl8zkNSdPZkBff46Sep6SE/liVtiSJPVuEVEBXAy8HXgJ8ADwTeB/yxmXepeGxiZ++0AN1962lCXrtjF9zBC+9NpZXDx7Iv0q+pQ7PEnqMCUn8lbYkiQpIo4mawv8I7Ad+DFZu+DNKaVHyhmbeo+6hiZ+ff9qvn77UlZs2MHR44dx9etP5OXHH0ZFnyh3eJLU4fabyFthS5IkgIi4C5gJ/BL4h5TSHXn5f5Q1MPUau+ob+fmCVXzz9qXUbN7F8ZNG8K03n8xLnjeePibwknqRNhN5K2xJklTkBcC1wHUppYfLHYx6j+27G/jxvSu57q5lrN+6m5OnjuKzrzqeM48aS4QJvKTeZ39H5K2wJUlSwSlkvfTujojlwA3AT8oakXq0LbvqueEvy/nu3U+xcUc9p88YzdcuPZHTpleawEvq1faXyFthS5IkAFJK9wNXRMQHgNeSDXr7BaAP8IqIWJtS2ljOGNUzbNxex/fveYrv/2U5W3c1cPYx47jirBmcPHVUuUOTpC6hzUTeCluSJLWUUtoF3AjcGBEzyHb6vw/4dETMSym9rKwBqttat3UX373rKW782wp21DVy/nETuPLsGcycNKLcoUlSl1LSqPVW2JIkqTUppSXAByPiw8AFZDv9pQNSs2kn1925jJ/8fSX1jU1cOGsiV5w1g6PGDyt3aJLUJR3wPPJW2JIkqaWUUiNwU36RSrJyww6+cccSfrFwNSnBq06axLvmzGDamCHlDk2SurQDTuQLrLAlSZJ0MJas28bXb1vCTYtrqIjg0lOm8I4zpzN51OByhyZJ3cJBJ/KSJEnSgXh07RauuW0Jtzy4lgF9+3DZC6u4/IzpjB8+sNyhSVK3YiIvSZKkDrVo1SaumbeEWx99hqED+vKuM4/gbS+axuihA8odmiR1SybykiRJ6hB/f6qWq+c9yV1PPsuIQf1437lHcdkLqxgxuF+5Q5Okbs1EXpIkSe0mpcTdS57l6nlL+PtTtYwZ2p8PvuwY3nTaVIYOsOkpSe3BrakkSZIOWUqJPz+6jqtvW8LiVZuYMHwgH7/wWC49ZQqD+leUOzxJ6lFM5CVJknTQmpoSv3/4aa6et4RH125h8qhBfPaS43n1yZMY0NcEXpI6gom8JEmSDlhDYxO/faCGa29bypJ125g+Zghfeu0sLp49kX4VfcodniT1aCbykiRJKlldQxO/um81X799KStrd3D0+GFc/foTefnxh1HRJ8odniT1CibykiRJ2q9d9Y38bP4qvnXHUmo27+KEySP4yCtO5tznjaePCbwkdSoTeUmSJO3T9t0N/PjelVx31zLWb91N9dRRfO7VJ3DGkWOIMIGXpHIwkZckSdJzbNlVzw1/Wc53736KjTvqOX3GaL526YmcNr3SBF6SysxEXpIkSXvUbq/j+/c8xfV/Wc7WXQ2cfcw4rjhrBidPHVXu0CRJORN5SZIksW7rLr5z11P88G8r2FHXyMtmTuCKs2Ywc9KIcocmSWrBRF6SJKkXq9m0k+vuXMZP/r6S+sYmLpo1kXefNYOjxg8rd2iSpH3o1EQ+Is4HvgpUAN9JKX2+xeNTgB8AI/NlPphSuiV/7ATgW8BwoAk4JaW0q/OilyRJ7cU2Qfmt2LCdb96xlF8sXE1K8KqTJvGuOTOYNmZIuUOTJO1HpyXyEVEBXAu8BFgNzI+IuSmlR4oW+wjw85TSNyLiWOAWoCoi+gI/BN6cUlocEaOB+s6KXZIktR/bBOW1ZN1Wvn7bUm5aXENFn+DSU6bwjjOnM3nU4HKHJkkqUWcekT8VWJJSWgYQET8FLgaKK+1EtncdYARQk98+D3ggpbQYIKW0oVMiliRJHcE2QRk8UrOFa29bwi0PrWVg3wre8sIq/vmM6YwfPrDcoUmSDlBnJvKTgFVF91cDz2+xzCeAP0bEe4AhwLl5+VFAiog/AGOBn6aUvtDyBSLicuBygClTprRr8JIkqd10eJsAbBcULFq1iWvmPcmtj65j6IC+vHvOEbz19GmMHjqg3KFJkg5SVxvs7vXA9SmlL0fEC4AbI2ImWZwvAk4BdgB/joiFKaU/Fz85pXQdcB1AdXV16tzQJUlSOzqkNgHYLrh32QauuW0Jdz35LCMG9eN95x7FZS+sYsTgfuUOTZJ0iDozkV8DHF50f3JeVuxtwPkAKaW/RsRAYAzZnvo7U0rPAkTELcBJwHMqbUmS1OXZJuggKSXuXvIsV/95CX9fXsuYof354MuO4U2nTWXogK52/EaSdLD6dOJrzQeOjIhpEdEfuBSY22KZlcA5ABHxPGAgsB74A3B8RAzOB7k5k73Po5MkSd2HbYJ2llLi1kee4ZVf/wtv/u7fWVm7g49feCx3/fvZvPPMI0ziJamH6bStekqpISKuJKuAK4DvpZQejohPAgtSSnOBDwDfjoj3kQ1yc1lKKQEbI+IrZBV/Am5JKd3cWbFLkqT2Y5ug/TQ2JX7/0NNcc9sSHl27hcmjBvHZS47n1SdPYkDfinKHJ0nqIJHViT1PdXV1WrBgQbnDkCTpgOTne1eXO46epqe1Cxoam5i7uIZrb1vC0vXbmT52CFfMmcFFsyfSr6IzO1xKkjpKW20C+1lJkiR1E3UNTfzqvtV8/falrKzdwTEThnHNG07kZTMPo6JPlDs8SVInMZGXJEnq4nbVN/Kz+av45h1LWbt5FydMHsFHXnEy5z5vPH1M4CWp1zGRlyRJ6qK2727gR/eu4Lo7n+LZbbupnjqKz7/6BM44cgwRJvCS1FuZyEuSJHUxm3fWc8NflvPde55i0456XjRjDFeefSLPn1ZpAi9JMpGXJEnqKmq31/G9u5/iB39ZztbdDZxzzDiuOHsGJ00ZVe7QJEldiIm8JElSma3bsotv37WMH/5tJTvrG3nZzAlccdYMZk4aUe7QJEldkIm8JElSmazZtJPr7ljKT+avoqGxiYtmTeTdZ83gqPHDyh2aJKkLM5GXJEnqZCs2bOcbty/ll/etJiV49UmTedecI6gaM6TcoUmSugETeUmSpE6yZN1Wrr1tKTctWkPfij68/tQpXH7GdCaPGlzu0CRJ3YiJvCRJUgd7uGYz1962hN899DQD+1bw1tOn8c9nTGf88IHlDk2S1A2ZyEuSJHWQ+1du5NrblnDro+sYOqAv755zBG89fRqjhw4od2iSpG7MRF6SJKmd3btsA9fctoS7nnyWkYP78f6XHMU/vaCKEYP7lTs0SVIPYCIvSZLUDlJK3PXks1wzbwl/X17LmKH9+dDLjuGNp01l6ACbXJKk9mOtIkmSdAhSStz66Dqumfcki1dvZsLwgXziwmO59NQpDOxXUe7wJEk9kIm8JEnSQWhsSvz+oae5et6TPPb0Vg6vHMTnXnU8rzppEgP6msBLkjqOibwkSdIBaGhsYu7iGq69bQlL129n+tghfPm1s7ho9kT6VfQpd3iSpF7ARF6SJKlEf3+qlqv+dzEra3dwzIRhXPOGE3nZzMOo6BPlDk2S1IuYyEuSJJVowvCBjBnan49ecCznHDOOPibwkqQyMJGXJEkq0ZTRg/nVu08vdxiSpF7OE7kkSZIkSepGTOQlSZIkSepGTOQlSZIkSepGTOQlSZIkSepGTOQlSZIkSepGTOQlSZIkSepGTOQlSZIkSepGTOQlSZIkSepGTOQlSZIkSepGTOQlSZIkSepGTOQlSZIkSepGTOQlSZIkSepGTOQlSZIkSepGIqVU7hg6RESsB1a04yrHAM+24/qk7sz/g9Ssvf8PU1NKY9txfaLd2wVuA6Vm/h+kZp3WJuixiXx7i4gFKaXqcschdQX+H6Rm/h96H79zqZn/B6lZZ/4f7FovSZIkSVI3YiIvSZIkSVI3YiJfuuvKHYDUhfh/kJr5f+h9/M6lZv4fpGad9n/wHHlJkiRJkroRj8hLkiRJktSNmMiXICLOj4jHI2JJRHyw3PFI5RIR34uIdRHxULljkcotIg6PiNsi4pGIeDgi/rXcManj2SaQMrYJpGblaBPYtX4/IqICeAJ4CbAamA+8PqX0SFkDk8ogIs4AtgE3pJRmljseqZwi4jDgsJTSfRExDFgIvNL6oeeyTSA1s00gNStHm8Aj8vt3KrAkpbQspVQH/BS4uMwxSWWRUroTqC13HFJXkFJam1K6L7+9FXgUmFTeqNTBbBNIOdsEUrNytAlM5PdvErCq6P5qbKhJkopERBVwInBvmUNRx7JNIElqU2e1CUzkJUk6BBExFPgl8N6U0pZyxyNJksqjM9sEJvL7twY4vOj+5LxMktTLRUQ/sgr7RymlX5U7HnU42wSSpFZ1dpvARH7/5gNHRsS0iOgPXArMLXNMkqQyi4gAvgs8mlL6SrnjUaewTSBJeo5ytAlM5PcjpdQAXAn8gWzQgp+nlB4ub1RSeUTET4C/AkdHxOqIeFu5Y5LK6HTgzcDZEbEov7y83EGp49gmkJrZJpD20ultAqefkyRJkiSpG/GIvCRJkiRJ3YiJvCRJkiRJ3YiJvCRJkiRJ3YiJvCRJkiRJ3YiJvCRJkiRJ3YiJvKROFRHVEZEioqrcsUiSpPKxTSAdPBN5qQeKiOvzijFFRENErIyIb0TEqHLHJkmSOo9tAqlnMpGXeq5bgcOAKuDtwIXA18sZkCRJKgvbBFIPYyIv9Vy7U0pPp5RWp5T+CPwMOA8gIvpExEcjYlVE7I6IByPi4sITI6Iq33NfXbzCvOw1LZZ5dUT8KSJ2RMQjEfGSFs85PyIei4hdEXEXcFSLx0dExI0RsS5fZllEvLdjPhJJknol2wRSD2MiL/UCETEdOB+oz4v+Ffg34D+A44FfA7+KiNkHsfrPAF8DZgHzgZ9GxND8dQ8HfgP8CZgNXA18ocXzP53HcAFwNPBWYM1BxCFJkvbDNoHUM/QtdwCSOsz5EbENqAAG5mXvz6+vAr6UUvpxfv9jEXFGXv6mA3yd/0kp/RYgIv4T+EeyCvpu4F3ASuBfUkoJeCwijgI+VfT8qcB9KaW/5/dXHODrS5KkttkmkHoYj8hLPdedZJXnqWR7vW8BvhYRw4GJwD0tlr8bOPYgXueBots1+fW4/Pp5wN/yCrvgry2e/w3gdRGxOCK+FBFnHkQMkiRp32wTSD2MibzUc+1IKS1JKT2YUvoXYDDw0f08p1C5NuXXUXggIvrt4zmFrnkUVc4lb1tSSr8j2wP/JWAMcHNEfL/U50uSpP2yTSD1MCbyUu/xX2Tnvw0l20t+eovHXwQ8kt9en18fVvT47IN4zUeB50dEFJWd1nKhlNKzKaUbU0qXAW8D/ikiBhzE60mSpP2zTSB1c54jL/USKaXbI+IR4CPAF4FPRsSTwEKyc+BeDJyUL7szIv4G/EdELAVGAJ87iJf9JvAB4P9FxNfJBrB5Z/ECEfFJ4D7gYbJt0quAZSml3QfxepIkaT9sE0jdn4m81Lt8Gfg+2XQvw8hGix0PPA68OqW0uGjZtwLfIRt1dinwbrJz7EqWUloZEa8CvgK8g6yB8EHgh0WL7SYb5XYasAv4G9n8tpIkqePYJpC6sdh7vAlJkiRJktSVeY68JEmSJEndiIm8JEmSJEndiIm8JEmSJEndiIm8JEmSJEndiIm8JEmSJEndiIm8JEmSJEndiIm8JEmSJEndiIm8JEmSJEndiIm8JEmSJEndyP8Hk7ZNrW511B4AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1224x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "consensus_gtruth_accuracy = [model_results[i][\"consensus_gtruth_accuracy\"] \n",
    "                             for i in range(rounds)]\n",
    "\n",
    "model_gtruth_accuracy = [model_results[i][\"model_gtruth_accuracy\"] for i in range(rounds)]\n",
    "\n",
    "\n",
    "# plot prc\n",
    "plt.rcParams[\"figure.figsize\"] = (17,6)\n",
    "plt.show()\n",
    "\n",
    "ax1 = plt.subplot(1, 2, 1)\n",
    "ax1.xaxis.set_major_locator(MaxNLocator(integer=True))\n",
    "plt.plot(range(0, rounds),consensus_gtruth_accuracy)\n",
    "plt.xlabel(\"Rounds\", fontsize=14)\n",
    "plt.ylabel(\"Accuracy of Consensus Labels\", fontsize=14)\n",
    "plt.title(\"Consensus Label vs Ground Truth Labels\", fontsize=14, fontweight=\"bold\")\n",
    "\n",
    "plt.subplot(1, 2, 2, sharex=ax1, sharey=ax1)\n",
    "plt.plot(range(0, rounds), model_gtruth_accuracy)\n",
    "plt.xlabel(\"Rounds\", fontsize=14)\n",
    "plt.ylabel(\"Accuracy of Model Predictions\", fontsize=14)\n",
    "plt.title(\"Model Predictions vs Ground Truth Labels\", fontsize=14, fontweight=\"bold\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4280ef3c",
   "metadata": {},
   "source": [
    "### References \n",
    "Peterson, J., Battleday, R., Griffiths, T., & Russakovsky, O. (2019).\n",
    "    **Human uncertainty makes classification more robust**.\n",
    "    In *Proceedings of the IEEE International Conference on Computer Vision*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aeac8b61",
   "metadata": {
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "# You can ignore this cell, it is just used during CI.\n",
    "# if consensus_gtruth_accuracy[0] >= consensus_gtruth_accuracy[-1]:  # check cleanlab has improved prediction accuracy\n",
    "#     raise Exception(\"Cleanlab failed to improve baseline consensus label accuracy.\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}