reacher-td3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DDPG for Solving Unity's \"Reacher\" Problem\n",
    "\n",
    "In this notebook we report how we used **Deep Deterministic Policy Gradient (DDPG)** algorithm to solve a modified version of Unitfy's \"Reacher\" environment, where the agent needs to control a double-jointed arm to track a moving target in a 3D environment.\n",
    "\n",
    "This notebook contains all the code for training and running the agent.\n",
    "\n",
    "A demo of a trained agent is shown in the gif below:\n",
    "\n",
    "![Demo](reacher-td3-demo.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment Setup\n",
    "\n",
    "The dependencies can be set up by following the instructions from the [DRLND repo](https://github.com/udacity/deep-reinforcement-learning#dependencies). Once it's done, the following imports should work:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections\n",
    "import copy\n",
    "import itertools\n",
    "import math\n",
    "!pip install -q dataclasses\n",
    "from dataclasses import dataclass\n",
    "from typing import Any, List, Tuple, Optional, Generator\n",
    "import random\n",
    "\n",
    "import numpy\n",
    "import torch\n",
    "from unityagents import UnityEnvironment\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that to make GPU training work on our machine, the following version of PyTorch is used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Version: 1.7.0+cu110; Cuda available: True\n"
     ]
    }
   ],
   "source": [
    "print(f\"Version: {torch.__version__}; Cuda available: {torch.cuda.is_available()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, we also need to download the pre-built Unity environment. There are 2 versions: the first version contains a single agent, while the second version contains 20 identical agents, each with its own copy of the environment. Here we'll be using **version 2** (more on this later), which can be downloaded for different platforms:\n",
    "\n",
    "- Linux: [click here](https://s3-us-west-1.amazonaws.com/udacity-drlnd/P2/Reacher/Reacher_Linux.zip)\n",
    "- Mac OSX: [click here](https://s3-us-west-1.amazonaws.com/udacity-drlnd/P2/Reacher/Reacher.app.zip)\n",
    "- Windows (32-bit): [click here](https://s3-us-west-1.amazonaws.com/udacity-drlnd/P2/Reacher/Reacher_Windows_x86.zip)\n",
    "- Windows (64-bit): [click here](https://s3-us-west-1.amazonaws.com/udacity-drlnd/P2/Reacher/Reacher_Windows_x86_64.zip)\n",
    "    \n",
    "Once downloaded and extracted, please set the path beolow according, e.g.\n",
    "\n",
    "- **Mac**: `\"path/to/Reacher.app\"`\n",
    "- **Windows** (x86): `\"path/to/Reacher_Windows_x86/Reacher.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Reacher_Windows_x86_64/Reacher.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Reacher_Linux/Reacher.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Reacher_Linux/Reacher.x86_64\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "ENV_PATH = \"../Reacher_Linux/Reacher.x86_64\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If set up correctly, we should be able to initialize the environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_speed -> 1.0\n",
      "\t\tgoal_size -> 5.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "class ParallelEnv:\n",
    "    def __init__(self):\n",
    "        self._env = UnityEnvironment(file_name=ENV_PATH)\n",
    "        \n",
    "        env_info = self._env.reset(train_mode=True)[self._brain_name]\n",
    "        self.num_agents = len(env_info.agents)\n",
    "    \n",
    "    @property\n",
    "    def _brain_name(self):\n",
    "        return self._env.brain_names[0]\n",
    "    \n",
    "    @property\n",
    "    def _brain(self):\n",
    "        return self._env.brains[self._brain_name]\n",
    "\n",
    "    @property\n",
    "    def action_dim(self) -> int:\n",
    "        return self._brain.vector_action_space_size\n",
    "\n",
    "    @property\n",
    "    def action_scale(self) -> float:\n",
    "        return 1.\n",
    "    \n",
    "    def sample_action(self) -> List[float]:\n",
    "        return numpy.random.uniform(\n",
    "            low=-self.action_scale,\n",
    "            high=self.action_scale,\n",
    "            size=(self.num_agents, self.action_dim)\n",
    "        )\n",
    "\n",
    "    @property\n",
    "    def state_dim(self) -> int:\n",
    "        return self._brain.vector_observation_space_size\n",
    "\n",
    "    def reset(self, train_mode=True):\n",
    "        env_info = self._env.reset(train_mode=train_mode)[self._brain_name]\n",
    "        return env_info.vector_observations\n",
    "\n",
    "    def step(self, actions):\n",
    "        assert len(actions) == self.num_agents\n",
    "        env_info = self._env.step(actions)[self._brain_name]\n",
    "        return (\n",
    "            numpy.array(env_info.vector_observations, dtype=numpy.float32),\n",
    "            numpy.array(env_info.rewards, dtype=numpy.float32),\n",
    "            numpy.array(env_info.local_done, dtype=bool),\n",
    "        )\n",
    "    \n",
    "    def close(self):\n",
    "        self._env.close()\n",
    "\n",
    "\n",
    "env = ParallelEnv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Environment description\n",
    "\n",
    "In this environment, a double-jointed arm can move to target locations. A reward of `+0.1` is provided for each step that the agent's hand is in the goal location. Thus, the goal of the agent is to maintain its position at the target location for as many time steps as possible.\n",
    "\n",
    "The observation space consists of `33` variables corresponding to position, rotation, velocity, and angular velocities of the arm.  Each action is a vector with four numbers, corresponding to torque applicable to two joints.  Every entry in the action vector must be a number between `-1` and `1`.\n",
    "\n",
    "The environment is considered \"solved\" when the agents get an average score of `+30` over 100 consecutive episodes, averaged over all parallel agents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 20\n",
      "Action dimension: 4\n",
      "States dimension: 33\n"
     ]
    }
   ],
   "source": [
    "print(f'Number of agents: {env.num_agents}')\n",
    "print(f'Action dimension: {env.action_dim}')\n",
    "print(f'States dimension: {env.state_dim}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Methodology & Implementation\n",
    "\n",
    "To solve this toy problem, we experimented with DDPG as described by Lillicrap et al. (2015). Additionally, we also incorporated all extensions from the **Twin Delayed DDPG (TD3)** algorithm as described by Fujimoto et al. (2018), namely:\n",
    "- **Clipped double Q-Learning for actor-critic**\n",
    "- **Delayed policy updates**\n",
    "- **Target policy smoothing regularization**\n",
    "\n",
    "TD3 also uses the first 10,000 steps for \"pure exploration\", during which actions are uniformly sampled from the action space. We find it unnecessary for this particular problem (but for the optional [Crawler](https://github.com/udacity/deep-reinforcement-learning/blob/master/p2_continuous-control/Crawler.ipynb) problem it does seem necessary).\n",
    "\n",
    "We also used **Prioritized experience replay**, which is also used by **Distributed Distributional DDPG (D4PG)** as described by Barth-Maron (2018). We also tried using the N-step returns as in D4PG, but it didn't seem to help. (For the Crawler environment, it does help with training progress initially but unfortunately slows it down later.)\n",
    "\n",
    "More details and code are presented below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyper-parameters\n",
    "\n",
    "The hyper-parameters we used is shown as the default values in the following data class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AgentConfig(replay_buffer_size=100000, critic_lr=0.0001, actor_lr=0.0001, batch_size=256, policy_noise=0.2, policy_noise_clip=0.5, policy_update_freq=2, discount_factor=0.99, soft_update_factor=0.005, pure_exploration_steps=0)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@dataclass\n",
    "class AgentConfig:\n",
    "    replay_buffer_size: int = 100_000\n",
    "    critic_lr: float = 1e-4\n",
    "    actor_lr: float = 1e-4\n",
    "    batch_size: int = 256\n",
    "    policy_noise: float = 0.2\n",
    "    policy_noise_clip: float = 0.5\n",
    "    policy_update_freq: int = 2\n",
    "    discount_factor: float = 0.99\n",
    "    soft_update_factor: float = 0.005\n",
    "    pure_exploration_steps: int = 0\n",
    "\n",
    "AgentConfig()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prioritized experience replay\n",
    "\n",
    "Here we use prioritized experience replay as described in Schaul et al. (2015). Specifically, we implemented the proportional priorization variant with the sum-tree data structure.\n",
    "\n",
    "The code for the replay buffer is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ProportionallyPrioritizedReplayBuffer:\n",
    "    \"\"\"A proportionally prioritized replay buffer implemented with sum-tree.\"\"\"\n",
    "\n",
    "    _curr_index: int\n",
    "    _size: int\n",
    "    _max_priority: float\n",
    "    _sum_tree: List[float]\n",
    "    _priorities: List[float]\n",
    "    _samples: List[Any]\n",
    "\n",
    "    def __init__(self, buffer_size: int):\n",
    "        assert buffer_size > 1\n",
    "        self._curr_index = 0\n",
    "        self._size = 0\n",
    "        self._max_priority = 1.0\n",
    "        self._sum_tree = [0] * (2 ** (math.floor(math.log2(buffer_size - 1)) + 1) - 1)\n",
    "        self._priorities = [0] * buffer_size\n",
    "        self._samples = [None] * buffer_size\n",
    "\n",
    "    def _ancestor_indices(self, sample_index: int) -> Generator[int, None, None]:\n",
    "        assert 0 <= sample_index <= len(self._samples)\n",
    "        index = sample_index + len(self._sum_tree)\n",
    "        while index > 0:\n",
    "            index = (index - 1) // 2\n",
    "            yield index\n",
    "\n",
    "    @staticmethod\n",
    "    def _children_indices(index: int) -> Tuple[int, int]:\n",
    "        # Note that it could go out-of-bounds for the sum tree array\n",
    "        left_index = index * 2 + 1\n",
    "        right_index = left_index + 1\n",
    "        return left_index, right_index\n",
    "\n",
    "    def _set_priority(self, sample_index: int, priority: float):\n",
    "        assert priority > 0, \"Weights must be non-negative\"\n",
    "        delta = priority - self._priorities[sample_index]\n",
    "        self._priorities[sample_index] = priority\n",
    "        for index in self._ancestor_indices(sample_index):\n",
    "            self._sum_tree[index] += delta\n",
    "\n",
    "        self._max_priority = max(self._max_priority, priority)\n",
    "\n",
    "    def _set_sample(self, sample_index: int, sample: Any, priority: float):\n",
    "        self._set_priority(sample_index, priority)\n",
    "        self._samples[sample_index] = sample\n",
    "\n",
    "    class _SampleHandle:\n",
    "        _parent: \"ProportionallyPrioritizedReplayBuffer\"\n",
    "        _index: int\n",
    "\n",
    "        def __init__(self, parent: \"ProportionallyPrioritizedReplayBuffer\", index: int):\n",
    "            assert 0 <= index <= len(parent._samples)\n",
    "            self._parent = parent\n",
    "            self._index = index\n",
    "\n",
    "        @property\n",
    "        def value(self) -> Any:\n",
    "            return self._parent._samples[self._index]\n",
    "\n",
    "        @property\n",
    "        def priority(self) -> float:\n",
    "            return self._parent._priorities[self._index]\n",
    "\n",
    "        @priority.setter\n",
    "        def priority(self, priority: float):\n",
    "            self._parent._set_priority(self._index, priority)\n",
    "\n",
    "        def reset(self, value: Any, priority: float):\n",
    "            self._parent._set_sample(self._index, value, priority)\n",
    "\n",
    "    def add(self, value: Any, priority: Optional[float] = None):\n",
    "        \"\"\"Add a new sample.\"\"\"\n",
    "        if priority is None:\n",
    "            priority = self._max_priority\n",
    "\n",
    "        self._SampleHandle(self, self._curr_index).reset(value, priority)\n",
    "\n",
    "        buffer_size = len(self._samples)\n",
    "        self._curr_index = (self._curr_index + 1) % buffer_size\n",
    "        self._size = min(self._size + 1, buffer_size)\n",
    "\n",
    "    @property\n",
    "    def priority_sum(self):\n",
    "        return self._sum_tree[0]\n",
    "\n",
    "    @property\n",
    "    def max_priority(self):\n",
    "        return self._max_priority\n",
    "\n",
    "    def sample_single(self, query: Optional[float] = None) -> _SampleHandle:\n",
    "        \"\"\"Draw a sample.\"\"\"\n",
    "        assert self.priority_sum > 0.0, \"Nothing has been added\"\n",
    "\n",
    "        if query is None:\n",
    "            query = random.random()\n",
    "\n",
    "        assert 0.0 <= query <= 1.0\n",
    "        target = self.priority_sum * query\n",
    "        index = 0\n",
    "        while True:\n",
    "            assert 0.0 <= target <= self._sum_tree[index]\n",
    "            index_l, index_r = self._children_indices(index)\n",
    "            assert (index_l < len(self._sum_tree)) == (index_r < len(self._sum_tree))\n",
    "            if index_l >= len(self._sum_tree):\n",
    "                index_l -= len(self._sum_tree)\n",
    "                index_r -= len(self._sum_tree)\n",
    "                break\n",
    "\n",
    "            sum_l = self._sum_tree[index_l]\n",
    "            if target <= sum_l:\n",
    "                index = index_l\n",
    "            else:\n",
    "                target -= sum_l\n",
    "                index = index_r\n",
    "\n",
    "        assert index_l < len(self._priorities)\n",
    "        if target <= self._priorities[index_l]:\n",
    "            index = index_l\n",
    "        else:\n",
    "            assert index_r < len(self._priorities)\n",
    "            index = index_r\n",
    "\n",
    "        return self._SampleHandle(self, index)\n",
    "\n",
    "    def sample_batch(self, batch_size: int) -> List[_SampleHandle]:\n",
    "        \"\"\"Draw a stratified batch of samples with the given size.\"\"\"\n",
    "        end_points = numpy.linspace(0.0, 1.0, batch_size + 1).tolist()\n",
    "        return [\n",
    "            self.sample_single(query=numpy.random.uniform(l, r))\n",
    "            for l, r in zip(end_points[:-1], end_points[1:])\n",
    "        ]\n",
    "\n",
    "    def __len__(self):\n",
    "        \"\"\"Return the current size of internal memory.\"\"\"\n",
    "        return self._size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Actor-critics\n",
    "\n",
    "Like in TD3, we also use two hidden layers of size 256, and let the critic take both state and action as the input of the first layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ActorNet(torch.nn.Module):\n",
    "    def __init__(\n",
    "        self,\n",
    "        state_dim: int,\n",
    "        action_dim: int,\n",
    "        action_scale=1.0,\n",
    "        hidden_dims: Tuple[int, ...] = (256, 256),\n",
    "    ):\n",
    "        super().__init__()\n",
    "        self.action_scale = action_scale\n",
    "        self.fcs = torch.nn.ModuleList(\n",
    "            [\n",
    "                torch.nn.Linear(in_size, out_size)\n",
    "                for in_size, out_size in zip(\n",
    "                    (state_dim,) + hidden_dims, hidden_dims + (action_dim,)\n",
    "                )\n",
    "            ]\n",
    "        )\n",
    "\n",
    "    def forward(self, state):\n",
    "        x = state\n",
    "        for i, fc in enumerate(self.fcs, start=1):\n",
    "            x = fc(x)\n",
    "            if i != len(self.fcs):\n",
    "                x = torch.relu(x)\n",
    "\n",
    "        x = torch.tanh(x) * self.action_scale\n",
    "        return x\n",
    "\n",
    "\n",
    "class CriticNet(torch.nn.Module):\n",
    "    def __init__(\n",
    "        self,\n",
    "        state_dim: int,\n",
    "        action_dim: int,\n",
    "        hidden_dims: Tuple[int, ...] = (256, 256),\n",
    "    ):\n",
    "        super().__init__()\n",
    "        self.fcs = torch.nn.ModuleList(\n",
    "            [\n",
    "                torch.nn.Linear(in_size, out_size)\n",
    "                for in_size, out_size in zip(\n",
    "                    (state_dim + action_dim,) + hidden_dims, hidden_dims + (1,)\n",
    "                )\n",
    "            ]\n",
    "        )\n",
    "\n",
    "    def forward(self, state, action):\n",
    "        x = torch.cat([state, action], dim=1)\n",
    "        for i, fc in enumerate(self.fcs, start=1):\n",
    "            x = fc(x)\n",
    "            if i != len(self.fcs):\n",
    "                x = torch.relu(x)\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Agent Implementation\n",
    "\n",
    "To put it together, the agent implemented below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Agent:\n",
    "    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "    \n",
    "    env: ParallelEnv\n",
    "    config: AgentConfig\n",
    "\n",
    "    replay_buffer: ProportionallyPrioritizedReplayBuffer\n",
    "\n",
    "    critics_local: Tuple[CriticNet, CriticNet]\n",
    "    critics_target: Tuple[CriticNet, CriticNet]\n",
    "    actor_local: ActorNet\n",
    "    actor_target: ActorNet\n",
    "\n",
    "    critic_optimizer: torch.optim.Optimizer\n",
    "    actor_optimizer: torch.optim.Optimizer\n",
    "\n",
    "    t_step: int\n",
    "        \n",
    "    def __init__(\n",
    "        self,\n",
    "        env: ParallelEnv,\n",
    "        config: Optional[AgentConfig] = None,\n",
    "    ):\n",
    "        self.env = env\n",
    "        self.config = config or AgentConfig()\n",
    "\n",
    "        self.replay_buffer = ProportionallyPrioritizedReplayBuffer(\n",
    "            self.config.replay_buffer_size\n",
    "        )\n",
    "        self.critics_local = (\n",
    "            CriticNet(env.state_dim, env.action_dim).to(self.device),\n",
    "            CriticNet(env.state_dim, env.action_dim).to(self.device),\n",
    "        )\n",
    "        self.actor_local = ActorNet(\n",
    "            env.state_dim, env.action_dim, action_scale=env.action_scale\n",
    "        ).to(self.device)\n",
    "        self.critics_target = copy.deepcopy(self.critics_local)\n",
    "        self.actor_target = copy.deepcopy(self.actor_local)\n",
    "        for m in itertools.chain(self.critics_target, [self.actor_target]):\n",
    "            m.eval()\n",
    "\n",
    "        self.critic_optimizer = torch.optim.Adam(\n",
    "            itertools.chain(\n",
    "                self.critics_local[0].parameters(),\n",
    "                self.critics_local[1].parameters(),\n",
    "            ),\n",
    "            lr=self.config.critic_lr,\n",
    "        )\n",
    "        self.actor_optimizer = torch.optim.Adam(\n",
    "            self.actor_local.parameters(),\n",
    "            lr=self.config.actor_lr,\n",
    "        )\n",
    "\n",
    "        self.t_step = 0\n",
    "\n",
    "    def act(self, state_batch):\n",
    "        state_batch = torch.from_numpy(state_batch).float().to(self.device)\n",
    "        self.actor_local.eval()\n",
    "        with torch.no_grad():\n",
    "            action_batch = self.actor_local(state_batch).cpu().numpy()\n",
    "            \n",
    "        self.actor_local.train()\n",
    "        action_batch = action_batch.clip(\n",
    "            min=-self.env.action_scale,\n",
    "            max=self.env.action_scale,\n",
    "        )\n",
    "        return action_batch\n",
    "\n",
    "    def reset(self):\n",
    "        pass\n",
    "\n",
    "    def learn(self):\n",
    "        # Only start learning after the pure exploration phase has ended\n",
    "        if self.t_step <= self.config.pure_exploration_steps:\n",
    "            return\n",
    "        \n",
    "        if len(self.replay_buffer) < self.config.batch_size:\n",
    "            return\n",
    "\n",
    "        experiences = self.replay_buffer.sample_batch(self.config.batch_size)\n",
    "        with torch.no_grad():\n",
    "            states, actions, rewards, next_states, dones = [\n",
    "                torch.from_numpy(numpy.vstack(v)).to(\n",
    "                    dtype=torch.float, device=self.device\n",
    "                )\n",
    "                for v in zip(*[e.value for e in experiences])\n",
    "            ]\n",
    "            # Calculate importance sampling weights\n",
    "            # (for simply not using alpha / beta here)\n",
    "            sample_priorities = numpy.vstack([e.priority for e in experiences])\n",
    "            sample_priorities = torch.from_numpy(sample_priorities).to(\n",
    "                device=self.device, dtype=torch.float\n",
    "            )\n",
    "            sample_probs = sample_priorities / self.replay_buffer.priority_sum\n",
    "            sample_weights = 1.0 / (sample_probs * len(self.replay_buffer))\n",
    "\n",
    "            # Calculate target Q values\n",
    "            next_actions = self.actor_target(next_states)\n",
    "            next_actions = next_actions + torch.clamp(\n",
    "                torch.randn_like(next_actions) * self.config.policy_noise,\n",
    "                -self.config.policy_noise_clip,\n",
    "                self.config.policy_noise_clip,\n",
    "            )\n",
    "            next_actions = torch.clamp(\n",
    "                next_actions,\n",
    "                -self.env.action_scale,\n",
    "                self.env.action_scale,\n",
    "            )\n",
    "            y = rewards + (\n",
    "                (1.0 - dones)\n",
    "                * self.config.discount_factor\n",
    "                * torch.min(\n",
    "                    self.critics_target[0](next_states, next_actions),\n",
    "                    self.critics_target[1](next_states, next_actions),\n",
    "                )\n",
    "            )\n",
    "\n",
    "        # Update sampling priorities\n",
    "        critic_losses = (\n",
    "            sum(\n",
    "                torch.nn.functional.mse_loss(\n",
    "                    critic(states, actions),\n",
    "                    y,\n",
    "                    reduction=\"none\"\n",
    "                )\n",
    "                for critic in self.critics_local\n",
    "            )\n",
    "        )\n",
    "        with torch.no_grad():\n",
    "            new_sample_priorities = critic_losses.sqrt().squeeze().cpu().numpy()\n",
    "            assert len(experiences) == len(new_sample_priorities)\n",
    "            for e, p in zip(experiences, new_sample_priorities):\n",
    "                e.priority = p\n",
    "\n",
    "\n",
    "        # Update local critics\n",
    "        critic_loss = torch.mean(critic_losses * sample_weights)\n",
    "        if torch.isnan(critic_loss).cpu().item():\n",
    "            raise RuntimeError(\"NaN loss\")\n",
    "\n",
    "        self.critic_optimizer.zero_grad()\n",
    "        critic_loss.backward()\n",
    "        self.critic_optimizer.step()\n",
    "\n",
    "        # Periodically update local actor as well as target networks\n",
    "        if self.t_step % self.config.policy_update_freq == 0:\n",
    "            # Update local actor\n",
    "            actor_loss = -torch.mean(\n",
    "                self.critics_local[0](states, self.actor_local(states))\n",
    "            )\n",
    "            if torch.isnan(actor_loss).cpu().item():\n",
    "                raise RuntimeError(\"NaN loss\")\n",
    "\n",
    "            self.actor_optimizer.zero_grad()\n",
    "            actor_loss.backward()\n",
    "            self.actor_optimizer.step()\n",
    "\n",
    "            # Soft-update target critics & actor\n",
    "            tau = self.config.soft_update_factor\n",
    "            for local_net, target_net in [\n",
    "                (self.actor_local, self.actor_target),\n",
    "                (self.critics_local[0], self.critics_target[0]),\n",
    "                (self.critics_local[1], self.critics_target[1]),\n",
    "            ]:\n",
    "                for local_param, target_param in zip(\n",
    "                    local_net.parameters(), target_net.parameters()\n",
    "                ):\n",
    "                    target_param.data.copy_(\n",
    "                        tau * local_param.data + (1.0 - tau) * target_param.data\n",
    "                    )\n",
    "    \n",
    "    def _train_episode(self, horizon):\n",
    "        self.reset()\n",
    "        states = self.env.reset()\n",
    "        episode_scores = numpy.zeros(self.env.num_agents, dtype=float)\n",
    "        for _ in range(horizon):\n",
    "            if numpy.any(numpy.isnan(states)):\n",
    "                print(\"\\nNaN State, episode terminated\")\n",
    "                break\n",
    "\n",
    "            # If we're in pure exploration phase, draw a random action\n",
    "            if self.t_step < self.config.pure_exploration_steps:\n",
    "                actions = self.env.sample_action()\n",
    "            else:\n",
    "                actions = self.act(states)\n",
    "                actions = (\n",
    "                    actions\n",
    "                    + numpy.random.normal(0, 0.1, size=actions.shape)\n",
    "                ).clip(-1., 1.)\n",
    "                if numpy.any(numpy.isnan(actions)):\n",
    "                    raise RuntimeError(\"NaN Action\")\n",
    "\n",
    "            next_states, rewards, dones = self.env.step(actions)\n",
    "            if numpy.any(numpy.isnan(rewards)):\n",
    "                print(\"\\nNaN Reward, episode terminated\")\n",
    "                break\n",
    "\n",
    "            self.t_step += 1\n",
    "            for experience in zip(states, actions, rewards, next_states, dones):\n",
    "                self.replay_buffer.add(experience)\n",
    "\n",
    "            self.learn()\n",
    "            states = next_states\n",
    "            episode_scores += rewards\n",
    "            if numpy.any(dones):\n",
    "                break\n",
    "\n",
    "        return numpy.mean(episode_scores)\n",
    "\n",
    "\n",
    "    def train(\n",
    "        self,\n",
    "        max_steps=1_000_000,\n",
    "        horizon=1_000,\n",
    "        solved_score=30,\n",
    "    ):\n",
    "        scores = []\n",
    "        scores_window = collections.deque(maxlen=100)\n",
    "        \n",
    "        stop_step = self.t_step + max_steps\n",
    "        while self.t_step < stop_step:\n",
    "            average_score = self._train_episode(horizon)\n",
    "            scores.append((self.t_step, average_score))\n",
    "            scores_window.append(average_score)\n",
    "            windowed_average_score = numpy.mean(scores_window)\n",
    "            print(\n",
    "                f\"\\rEpisode {len(scores)}\\tStep {self.t_step}\\tScore: {average_score:.2f}\\t\"\n",
    "                f\"Windowed average Score: {windowed_average_score:.2f}\",\n",
    "                end=\"\\n\" if len(scores) % 200 == 0 else \"\",\n",
    "            )\n",
    "            \n",
    "            if windowed_average_score >= solved_score:\n",
    "                print(f\"\\nReached average score of {windowed_average_score:.2f} between \"\n",
    "                      f\"episode {len(scores) - len(scores_window) + 1} and episode {len(scores)}\")\n",
    "                break\n",
    "\n",
    "        return scores\n",
    "\n",
    "    def run_episode(self, horizon=10_000):\n",
    "        states = self.env.reset(train_mode=False)\n",
    "        score = 0.0\n",
    "        for _ in range(horizon):\n",
    "            if numpy.any(numpy.isnan(states)):\n",
    "                print(\"\\nNaN State, episode terminated\")\n",
    "                break\n",
    "\n",
    "            actions = self.act(states)\n",
    "            if numpy.any(numpy.isnan(actions)):\n",
    "                raise RuntimeError(\"NaN Action\")\n",
    "\n",
    "            next_states, rewards, dones = self.env.step(actions)\n",
    "            if numpy.any(numpy.isnan(rewards)):\n",
    "                print(\"\\nNaN Reward, episode terminated\")\n",
    "                break\n",
    "\n",
    "            states = next_states\n",
    "            score += numpy.sum(rewards)\n",
    "            if numpy.any(dones):\n",
    "                break\n",
    "\n",
    "        return score / self.env.num_agents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train & Evaluate\n",
    "\n",
    "With everything set up, we're now ready to train the agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = Agent(env)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 103\tStep 103000\tScore: 39.15\tWindowed average Score: 30.08\n",
      "Reached average score of 30.08 between episode 4 and episode 103\n",
      "CPU times: user 31min 20s, sys: 15 s, total: 31min 35s\n",
      "Wall time: 34min 8s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "scores = agent.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(agent.actor_local.state_dict(), 'actor.pth')\n",
    "torch.save(agent.critics_local[0].state_dict(), 'critic0.pth')\n",
    "torch.save(agent.critics_local[1].state_dict(), 'critic1.pth')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl4AAAFzCAYAAADv+wfzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAA7N0lEQVR4nO3deXycZb3//9cn+540bdKmSfeWlm6UEkoLiCwCFQREUVARBI543PHnV8Xj93dQj3g8niMuR49HNgGVTXbZSkFkLYXuTZrua/at2feZ6/vH3IUASZuWzNyTyfv5eMwjc933PXN/kjt35p3rvu77NuccIiIiIhJ+cX4XICIiIjJaKHiJiIiIRIiCl4iIiEiEKHiJiIiIRIiCl4iIiEiEKHiJiIiIREiC3wUMxbhx49zUqVP9LkNERETkiNauXVvvnMsbaN6ICF5Tp05lzZo1fpchIiIickRmtm+weTrUKCIiIhIhCl4iIiIiEaLgJSIiIhIhCl4iIiIiEaLgJSIiIhIhCl4iIiIiERL24GVm8Wa23sye9NrTzGy1me00swfMLCncNYiIiIhEg0j0eH0TKOvX/g/gl865mcBB4LoI1CAiIiLiu7AGLzMrAi4EbvfaBpwNPOQtcjfw8XDWICIiIhItwt3j9Svgu0DQa48FmpxzfV67HCgc6IVmdr2ZrTGzNXV1dWEuU0RERCT8wha8zOxjQK1zbu2xvN45d6tzrtg5V5yXN+DtjkRERERGlHDeq/E04GIzuwBIAbKAXwM5Zpbg9XoVARVhrEFEREQixDmHc2AGodFFAy8TCDo6egO0dfXR1t1Ha1cfnT0B5k3MYkx6bJ9zF7bg5Zz7PvB9ADM7E/g/zrnPmdlfgcuA+4GrgcfDVYOISDRp7ujlT2/spbql633zZuRlsGRaLnMmZBEfN/AH1rHo6QuyorSaJzdVkp6UwOSxaUzOTWPK2DQm56YzLiNp0A/IcOgLBKlq7qIgO4WE+MEPugSDjoMdPWSnJh52uf5au3p5alMVD68rZ0dtGxcsKOCzSyYzvzB7uMofsZo7elm1u4GJOSksLMoZ1veuau7kH9vqeGlbHa/trKe1OzSaKM4gzoy4OAMHAecIesFsMInxxtlz8vnk4iLOnJ1PUsL7t30g6OgLBt9+H+fA4egLOgIBR28wSF8gNLMgOyWiv99DEc4er8F8D7jfzH4CrAfu8KEGEXmPyqZOVm6pYdmMsczKzwjLH6vG9h6aOnqYOjY99Md4lOjqDXDPqr387sVdtHT1Mibt3f/R9wWCtHSFPqwyUxI4eWqu9xjD/MJsUhLjB3zfvkCQfY0dpCTGk5eR/K4PqfKDHdz35n4eeOsA9W09FGSnAPDohop3ffClJ8UzyQtiU8emUzQmlbg4IxB09AYcgWCQODMuXjSR/MyUQb/HQNCxfv9BkhPiGZOeSG56EmlJCQSDju21rby2s4FVu+pZvbuR1u4+UhLjOL4gi/kTs5lfmMWk3DR217VTVtVCWVULW6tb6egJEGeQm55MfmYy+Vmhr3mZyeRlJJOflUJeZjLt3X08tr6CZ0ur6eoNMiMvndNmjuORdeXcu3o/Cwqz+ewpk7nohIlkJA/fx15fIEhTZy8H23toaO/hYHsP+VnJnDQld0ivd87R1RukrbuPjp4+Gtt7qGnppqali+qWLmpaupiRl8EXPzR9wAByuPft7gtSWtnMS9vreXl7HZvKmwh62/38eeP57vI5zMjLeN/rNhxo4uF15RRPyeXjJw44BPvtZf/w8m4eXVfBtppWIBRyLlxYQEF2KgHn3u7ZCjgXCmAG8WaYGfFxRlpSPBnJCWSkJJCRnEBifBx/31rL4xsqWFFaQ256EhcuKCAlMY7K5i4qmzqpbOqktrX7sOGtv6lj07j0xCIuPbGQyWPThvwzDCdzQ63eR8XFxW7NmjV+lyESs5xzXHnHal7b2QBAYU4qZ8/J5+w5+SybMXbQD34Iffjc9fpeVu9pZGZ+BscXZDG3IJOpY9NxwLp9B3l5Rx2v7Khnc0UzzkFGcgLzC7M4oSiHhUU5TMpNpTcQpLsvSI/3yEhJ4PgJAx92qG/r5vVdDby+s578zGS+ctbMw9ZYUtHMrro2slITyUpJJDs19BibnhTWABgIOh5dX8Etz22jsrmLDx+Xx3eXz2bexPf3wFQ0dfLWnkZW72nkzT0N7KprB0I9APMLszlp8hgWTsrhYHsPWypb2FLVwraaVnr6gm+/R256EvmZyaQlxbP+QBMGnD1nPFcuncwZs/KIizO6egOUH+xkf2M7+xo62NfQwf7GDvY1tHPgYOe73q+/jOQEvnnOLK4+deq7QoBzjn9sq+Nnz2x9+wP4kOSEOJLi497uAZk6No1lM8Yxb2IWe+rbKaloprSyhbbuvrdfk5mS4P0OZTE5N42mzl7qWruobemmtrWb2tYu6tt6CATf/dmVlZLAxYsm8snFRSyalIOZ0dzZy2PrK7h39X621bSSmhjPGceN4/x5Ezhnzniy0xKPboN69tS385Mnt/D3bbUDBoCLT5jI//+xueRlJr9v3vaaVn7+7DZW72mgvbuP4CAfwQlxxtiMJGpauplfmMWvLl/EzPzMAd/vlyu3U1LZTGdPkM6ePjp7A2+/b5zBwqIczjguj9NnjuON3Q384aVddPUF+XTxJG74yCxSEuJ5bEMF9725n63VrRz6n+u/LjuBT55U9L51Ouf40d+2cNfre1kyLZePHJ/PmbPzh+0ftr5AkJd31PHw2gpWbqnBDCbmpDIxJ4WC7FQKslPe3t/NwDDMQj+zhDgjIT6OhDij2+vtXbW7AedgydRcLl1cyAULCshOPbZtP1RmttY5VzzgPAUvEVm5pYYv3rOGb33kOPKzknmhrJbXdtbT2RsgMyWB606fxjWnTXvfH6vSymZufHgzmyuamZSbSnVzF71eF39yQuiPX3tPgPg448RJoT/+47OSKaloYVN5E2VVrfQEBv6gP2RCVgpzJ2ZxfEEmPX1BXt3ZQFlVCwCZyQm0dvcxIy+dX16+6H2HUJo6evjZM1u5/60DA773lLFpfOXMGVx6YtGAPQrOOXbXt5Odmsi4jPd/iB7O/oYOvnLvWkoqWlhYlM2Ny+dw6sxxQ359fVs36/YdZO3+g6zf18TG8ia6vVCUm57E3IIs5k7M4rjxmfQGgl4w6aK2tZuD7T0smzGWK5ZMpjAndcjrDAYd9W3dOHjXB1hVcxc3P7WFF7fVMSMvnZsumscZx+WxqbyJf396K6t2NzBlbBpfP3sWWSkJHOzoobG9l4MdPXT2BFhYlM2yGWMpGvP+Hodg0LGvsYMDjR1Mz0unMCf1iB/ehw5D1rV1U9faTV/QsWz64P8gOOdYt7+JR9eX81xpDbWt3STEGctmjOUTiwu59MT3h4uBdPT08du/7+T2V/aQlBDH506ZTOGYVMakJZGbnkROWiLPb6nldy/uJDUpnh9ccDyfKi7CzKhq7uSXK7fz0Npy0pMSuGjRRMakJZKeHOrtSU9KICctkfFZKYzPSnn7n4IVpdXc+PAmOnoC/ODC4/n80imYGRVNofd7ZF3o/c4+Pp/05ARSE+NDj6R4poxN4/SZ48h5Tw9rfVs3v/37Tv6yeh/xcYZz0N0XZGFRNlecPJnz543nG/evZ9WuBv77M4u5cGHBu36WNz9Vxu2v7uGfTp/GDy48PqyH8noDQRLi7AOto6Kpk8fWV/DIunJ21bVzy6dP4BOLh7bNj5WCl4gMqqcvyHm/fImE+Die+eaHSPTG03T1Bli9p5F7V+9jRWkNmSkJXHvaNK49fRrJCXH85oUd/OHl3YxJS+RHF8/nggUT6A04dta2sbU6dLiouy/IqTPGcerMsWSlvP8/zO6+ANuqW6lp6SbJ6x1JSogjOSGOxvaetw87balqYVddO/FmnDRlDKfPGsdpM8cxf2IWq3Y38J2/bqKurZuvnz2Tr541k4Q449H1Fdz8VBlNnb1cd/o0PnVSEa3dfTR39tLS2Utjew+PrKtgc0UzhTmp/POHp/Op4kkkJ8RRWtnCMyVVPFNSze66dhLjjYtOmMi1p00b0nih13bW89V71xEMOn5y6QIuWljwgT+cevqCbK9pJS8zdLjNj3Erf99aw4//toW9DR0cX5BFWVULuelJfPOcWXxmyeSjOhzml2DQsaG8iRWl1awoqWZvQwff/+gcvvThGYO+xjnHk5uq+OnTZVQ1d/GJxYXcuHwO+VkDH3rdWdvK9x/ZzFt7D7J0ei4nFOVw1+t7cQ4+v2wKXztr5lENIK9t6eI7D23ipe11nDk7jxl5GfzpjX0AXL1sCl858+je75B9De3870u7SIiL4/KTJ73rd7ujp4+r73yT9fub+P2VJ3Hu3PE45/jZs1v5w0u7+cKpU7nporlRN37qcJxzbK5oZmZ+BmlJ4R1ppeAlIoO67eXd3Px0GXddczJnzs4fcJnSymZ+88KOtwPYmLQk9jd2cNlJRfzfC49/33/U4dDVGwAYsFejuaOXm54o4bENlSwsyiY9KYFVuxtYNCmHn166gLkTswZ8T+cc/9hex3+/sIN1+5vIz0wmJTGe/Y0dxMcZS6fncv68Ceyua+fBNQfo6AmwZFou1542jY8cn/++Qd/OOe58bS8/fbqM6ePSue2qYqaOSx/+H4aPuvsC3PnqXh5ae4CPzi/gSx+eTuYAoXokCAQd37x/PU9uquLnly3k08WT3rdMW3cf335wAytKa5hbkMWPL5lH8dQjj+EKBh33v3WAf3+mjNauPi5ZNJH/c95sJuUe2zgj5xx/fmMfP3mqjN5AkE8uLuKGc487qh7No9Xa1cuVt6+mrKqV268u5s09jfz2xZ1cuXQy/3bJ/BEVuiJNwUtEBtTQ1s2Z//kPTpo6hruuWXLE5Q8FsMqmLr67fDYfmhVd19h7alMVP3hsM4Gg43vL5/DZJZOHNIbLOceqXQ3c+spuAD46fwLnzp1Abr9ehObOXh586wB3vb6XiqZOMpMTOGnqGJZMy+WUabkcNz6Tm54o5ZF1FZw3dzy3XL5oWAdyS3j09AW57u63eH1XA//r9ewcsre+nS/es4bd9e3cuHwO154+7ajPOD3Y3kNbd98xB673OtDYQV/QMS1Cgb6po4fP3Laa7TWtBIKOK06exE8vXTCqTo45FgpeIjKgHzy6mfvfOsCKG85gZn7GkV8wAjR39oLjmAdOH0lfIMgLW2t5aXsdb+1pZEdtGxAa5Osc3PCRWXzj7Fn6YBpB2rv7+Oxtb7C1upV7rl3CKdPH8o9ttXzjvvXExxm//exiTjuK8XmxpqGtm+vuXsP8wix+fPF8/W4PgYKXiLxPWVULF/7mFa5aNpUfXjzP73JGrIa2bt7ae5D1Bw6ydPpYzhrkcK1Et8b2Hj71v69T29LN5SdP4o7X9jB7fCa3XVU8bL1VMnooeInIuxy6fERJRQsvfefMiIzREol2FU2dXPb716lq7uLChQX852ULwz4IW2LT4YKXfqNERpmKpk7ufHUPr+1s4IcXzVXoEvEU5qTy4JeWsbG8iQsXfPAzUUUGouAlMgo453htZwP3rNrL82U1QOgij59bOsXnykSiy6TcNB1alLBS8BKJcWv3HeS7D21kV107uelJ/POHZ/DZUyYPeDFLEREJLwUvkRhWWtnMF/74JmPSkrjl0ydwwYKCw95aR0REwkvBSyRG7a5r46o73iQzOYH7r1/KxDBeaFFERIYm+u/vICJHraq5k8/f8SYAf/qnUxS6RESihIKXSIxpbO/hyttX09LZy93XLmFGXmxcGFVEJBboUKNIDGnr7uMLf3yT8oOd3HPtkiHd0FlERCJHPV4iMeRXK7dTUtHM/3xuMadMH+t3OSIi8h4KXiIxorq5iz+9sY9PLC7inOPHH/kFIiIScQpeIjHity/uIBB0fPOcWX6XIiIig1DwEokBBxo7eOCtA1x+8iRddVtEJIopeInEgN+8sAMz4+tnq7dLRCSaKXiJjHC76tp4eF05n186hQnZKX6XIyIih6HgJTLC/er5HaQkxvPlM2f4XYqIiByBgpfICFZW1cLfNlZyzWlTGZeR7Hc5IiJyBApeIiPYLSu3k5mSwPUfUm+XiMhIoCvXi4wAu+vauOmJUnoDQQAMI+gcq/c08u1zjyM7LdHnCkVEZCjU4yUyAtz+6h5W72kk6CDoIBB0OAfnzh3PNadP87s8EREZIvV4iUS5zp4Af9tQyccWFnDLpxf5XY6IiHwA6vESiXJPb66itbuPy4sn+V2KiIh8QGELXmaWYmZvmtlGMys1sx950+8ysz1mtsF7LApXDSKx4IE1B5g2Lp0l03L9LkVERD6gcB5q7AbOds61mVki8KqZPePN+45z7qEwrlskJuyua+PNPY18d/lszMzvckRE5AMKW/ByzjmgzWsmeg8XrvWJxKIH15QTH2dctrjI71JERGQYhHWMl5nFm9kGoBZY6Zxb7c262cw2mdkvzUxXfRQZQF8gyMPryjlrdj75WboVkIhILAhr8HLOBZxzi4AiYImZzQe+D8wBTgZyge8N9Fozu97M1pjZmrq6unCWKRKVXtxWR11rN5efrEH1IiKxIiJnNTrnmoAXgeXOuSoX0g38EVgyyGtudc4VO+eK8/LyIlGmSFR54K0D5GUmc9Zs/f6LiMSKcJ7VmGdmOd7zVOBcYKuZFXjTDPg4UBKuGkRGqtqWLl7cVssnFxeREK+rvoiIxIpwntVYANxtZvGEAt6DzrknzezvZpYHGLAB+Ocw1iAyIj28roJA0PHpYg2qFxGJJeE8q3ETcOIA088O1zpFYoFzjr+uOcCSqblMz8vwuxwRERlGOoYhEmVe29nA7vp2Pq1B9SIiMUfBSySK9PQF+eHfSikak8rHFhb4XY6IiAwz3SRbJIrc9spudta28ccvnExKYrzf5YiIyDBTj5dIlNjX0M5vXtjBBQsmcNacfL/LERGRMFDwEokCzjn+72MlJMbHcdNF8/wuR0REwkTBSyQKPLGxkld21POd82czXrcHEhGJWQpeIj5r7uzl354sY2FRNlcuneJ3OSIiEkYaXC/is58/u5XG9m7uuuZk4uPM73JERCSM1OMl4qOt1S38ZfV+rjltGvMLs/0uR0REwkzBS8RH6/c3AfCFU6f6WoeIiESGgpeIj/Y2tJMYb0zMSfW7FBERiQAFLxEf7avvYFJumsZ2iYiMEgpeIj7a19jB1LHpfpchIiIRouAl4hPnHPsa2pkyNs3vUkREJEIUvER8UtfWTUdPQD1eIiKjiIKXiE/2NXQAqMdLRGQUUfAS8cne+nYA9XiJiIwiCl4iPtnX0EF8nFE4RpeSEBEZLRS8RHyyt6GdojGpJMZrNxQRGS30F1/EJ/sbO5icq/FdIiKjiYKXiA+cc+ypb9f4LhGRUUbBS8QHTR29tHb16YxGEZFRRsFLxAd7G3RGo4jIaKTgJeKDQ9fwmjpOPV4iIqOJgpeID/Y2tGMGRWMUvERERhMFLxEf7G/oYGJ2KimJ8X6XIiIiEaTgJeKDvbo5tojIqKTgJeKDfQ0dTNHAehGRUUfBSyTCWrp6aWjvUY+XiMgoFLbgZWYpZvammW00s1Iz+5E3fZqZrTaznWb2gJklhasGkWi0/9AZjQpeIiKjTjh7vLqBs51zJwCLgOVmthT4D+CXzrmZwEHgujDWIBJ1Dl3DS4caRURGn7AFLxfS5jUTvYcDzgYe8qbfDXw8XDWIRKND1/DSoUYRkdEnrGO8zCzezDYAtcBKYBfQ5Jzr8xYpBwrDWcOQ/fFCWP+X0PNAb6i98YFQu6cj1C55ONTuag61tzwRarc3hNrbngm1W2tC7R3Ph9rN5aH2rhdD7cY9ofbeV0Pt+h2h9v7VoXbNllC7Ym2oXbUp1K7aFGpXrA21a7aE2vtXh9r1O0Ltva+G2o17Qu1dL4bazeWh9o7nQ+3WmlB72zOhdntDqL3liVC7qznULnk41O4JBQY2PhBqB3pD7fV/CbUPWXsX3H3xO+03b4M/f/Kd9hu/h3uveKf92m/ggSvfab9yC/z1mnfaL/0cHv7iO+2/3wyPfeWd9vM/hCe+8U57xQ/gqW+/037mxtDjkKe+HVrmkCe+EXqPQx77Smgdhzz8xVANh/z1mlCNhzxwZeh7OOTeK0Lf4yF//mToZ+A5b831fDH9ZdKSEkIT9Lun371Dwvy7x90Xh35Gh+h3T797h4ym3z2fhTV4OecCzrlFQBGwBJgz1Nea2fVmtsbM1tTV1YWrRJGI6+wNMC492e8yRETEB+aci8yKzP4V6AS+B0xwzvWZ2TLgh8658w/32uLiYrdmzZpIlCkSdqf89HnOmJXHf37qBL9LERGRMDCztc654oHmhfOsxjwzy/GepwLnAmXAi8Bl3mJXA4+HqwaRaNPR00dNSzdTx2lgvYjIaJQQxvcuAO42s3hCAe9B59yTZrYFuN/MfgKsB+4IYw0iUWV/Y2i8yORcDawXERmNwha8nHObgBMHmL6b0HgvkVFnb/2ha3ipx0tEZDTSletFImifdw2vybqUhIjIqKTgJRJB+xo7yE1PIjs10e9SRETEBwpeIhG0r6FdF04VERnFFLxEImhvfYfGd4mIjGIKXiIR0t0XoLK5Uz1eIiKjmIKXSIQcaOzEOZ3RKCIymil4iUTIrrrQPeN18VQRkdFLwUskQkoqmomPM+ZMyPS7FBER8YmCl0iEbK5oZlZ+BimJ8X6XIiIiPlHwEokA5xwlFc3ML8z2uxQREfGRgpdIBNS0dFPf1sMCBS8RkVFNwUskAjZXNAMwvzDL50pERMRPCl4iEbC5opk4g7kF6vESERnNFLxEIqC0opmZ+RmkJmlgvYjIaKbgJRIBmyuamT9RvV0iIqOdgpdImNW2dFHb2q0zGkVERMFLJNxKKkMD6xcUKXiJiIx2Cl4iYba5vAUzmFugMxpFREY7BS+RMNtc0cz0cemkJyf4XYqIiPhMwUskzEoqmnXhVBERARS8RMKqrrWb6pYuDawXERFAwUskrA4NrFfwEhERUPASCauS8lDwmjdRA+tFRETBSySsDg2sz0xJ9LsUERGJAgpeImFUWtmiw4wiIvI2BS+RMGls76GiqZP5hTrMKCIiIQpeImGyuUID60VE5N0UvETCpKTi0MB6BS8REQlR8BIJk5KKZqaMTSM7VQPrRUQkJGzBy8wmmdmLZrbFzErN7Jve9B+aWYWZbfAeF4SrBhE/ba5o1mFGERF5l3DePK4P+LZzbp2ZZQJrzWylN++Xzrn/CuO6RXx1sL2H8oOdXLl0it+liIhIFAlb8HLOVQFV3vNWMysDCsO1PpFosnbfQQAWqsdLRET6icgYLzObCpwIrPYmfc3MNpnZnWY2JhI1iETSitJqMpMTKJ6a63cpIiISRcIevMwsA3gYuME51wL8HpgBLCLUI/aLQV53vZmtMbM1dXV14S5TZNj0BYKsLKvhnOPzSUrQ+SsiIvKOsH4qmFkiodD1F+fcIwDOuRrnXMA5FwRuA5YM9Frn3K3OuWLnXHFeXl44yxQZVm/uaaSpo5fl8yf4XYqIiESZcJ7VaMAdQJlz7pZ+0wv6LXYpUBKuGkT88GxpNSmJcZxxnP5hEBGRdwvnWY2nAZ8HNpvZBm/avwCfMbNFgAP2Al8KYw0iERUMOlaUVvPh4/JISwrn7iUiIiNROM9qfBWwAWY9Ha51ivhtQ3kTNS3dOswoIiID0shfkWG0oqSahDjj7Dnj/S5FRESikIKXyDBxzvFsaTWnzhyn2wSJiMiAFLxEhsnW6lb2NXSwfJ4OM4qIyMAUvESGybMl1ZjBuXN1mFFERAam4CUyTFaUVnPylFzyMpP9LkVERKKUgpfIMNhT387W6lbO19mMIiJyGApeIsNgRWk1AOfP02FGEREZnIKXyDB4tqSaBYXZFI1J87sUERGJYgpeIh/Q2n0H2XCgSRdNFRGRI9I9TUSOkXOOP72xj397cguFOalcdlKR3yWJiEiUU/ASOQbt3X18/5HNPLGxkrPn5HPLp08gJy3J77JERCTKKXiJHKWdta3885/Xsbuuje+cP5svf3gGcXED3ZZURETk3RS8RI5CVXMnl/z2NVKT4vnzdadw6sxxfpckIiIjiIKXyFF4fksN7T0BHvnKacyekOl3OSIiMsLorEaRo/DS9nom5aZy3PgMv0sREZERSMFLZIh6+oKs2lXPGbPyMNOYLhEROXoKXiJDtG7/Qdp7ApxxXJ7fpYiIyAil4CUyRC9vryMhzjh1xli/SxERkRFKwUtkiF7eUcfiyWPITEn0uxQRERmhFLxEhqC+rZuSihbOOE6XjxARkWOn4CUyBK/uqAfQ+C4REflAhhy8zCzVzGaHsxiRaPXy9jpy05OYPzHb71JERGQEG1LwMrOLgA3As157kZk9Eca6RKJGMOh4eUc9p88cp1sDiYjIBzLUHq8fAkuAJgDn3AZgWlgqEokyZdUt1Ld16zCjiIh8YEMNXr3Oueb3THPDXYxINHp5uze+a5YG1ouIyAcz1Hs1lprZZ4F4M5sFfAN4PXxliUSPl7fXMWdCJvlZKX6XIiIiI9xQe7y+DswDuoF7gWbghjDVJBI12rv7WLOvUYcZRURkWByxx8vM4oGnnHNnAT8If0ki0eON3Q30BhxnzFLwEhGRD+6IPV7OuQAQNDOdRy+jzsvb60hJjKN46hi/SxERkRgw1DFebcBmM1sJtB+a6Jz7xmAvMLNJwD3AeEID8W91zv3azHKBB4CpwF7g0865g8dUvUiYvbyjnqXTx5KSGO93KSIiEgOGGrwe8R5How/4tnNunZllAmu94PYF4AXn3M/M7EbgRuB7R/neImFXfrCDPfXtfH7pFL9LERGRGDGk4OWcu9vMkoDjvEnbnHO9R3hNFVDlPW81szKgELgEONNb7G7gHyh4SRTaVB66gspJU3SYUUREhseQgpeZnUkoJO0FDJhkZlc7514e4uunAicCq4HxXigDqCZ0KHKg11wPXA8wefLkoaxGZFiVVjYTH2fMnpDpdykiIhIjhno5iV8A5znnPuycOwM4H/jlUF5oZhnAw8ANzrmW/vOcc45BLsTqnLvVOVfsnCvOy9MZZRJ5pZUtzMrP0PguEREZNkMNXonOuW2HGs657UDikV5kZomEQtdfnHOHxojVmFmBN78AqD26kkUio6SihbkTs/wuQ0REYshQg9caM7vdzM70HrcBaw73AjMz4A6gzDl3S79ZTwBXe8+vBh4/2qJFwq22pYv6tm7mT9RVVEREZPgM9azGLwNfJXSrIIBXgP85wmtOAz5P6DIUG7xp/wL8DHjQzK4D9gGfPpqCRSKhpDI0sH6eerxERGQYDTV4JQC/PtRz5V3NPvlwL3DOvUpoIP5AzhlyhSI+KK0IDUfUoUYRERlOQz3U+AKQ2q+dCjw//OWIRIfSyhamjk0jM+WIQxlFRESGbKjBK8U513ao4T1PC09JIv4rqWxmXqHGd4mIyPAaavBqN7PFhxpmVgx0hqckEX81d/RSfrBT47tERGTYDXWM1w3AX82s0msXAJeHpSIRn5W+PbBePV4iIjK8DtvjZWYnm9kE59xbwBxCN7fuBZ4F9kSgPpGIK60MDaxXj5eIiAy3Ix1q/APQ4z1fRuhyEL8DDgK3hrEuEd+UVjYzISuFcRmHPXFXRETkqB3pUGO8c67Re345cKtz7mHg4X7X5hKJKSWVLcwvVG+XiIgMvyP1eMWb2aFwdg7w937zhjo+TGTE6OwJsLuujbka3yUiImFwpPB0H/CSmdUTOovxFQAzmwk0h7k2kYgrq24h6GC+xneJiEgYHDZ4OeduNrMXCJ3F+Jxzznmz4oCvh7s4kUgrrfDOaNQ1vEREJAyOeLjQOffGANO2h6ccEX+VVraQk5bIxOwUv0sREZEYNNQLqIqMCiWVzcyfmI3ZYLcZFREROXYKXiKe3kCQ7dVtun6XiIiEjYKXiGdHTRs9gaDGd4mISNgoeIl4St6+VZB6vEREJDwUvEQ8WypbSE+KZ9rYdL9LERGRGKXgJeIprWzm+IIs4uI0sF5ERMJDwUsECAYdWypbdJhRRETCSsFLBHhuSw3tPQEWFuX4XYqIiMQwBS8Z9Wpbu/iXRzczb2IWF50w0e9yREQkhil4yajmnON7D22ivbuPX12+iKQE7RIiIhI++pSRUe0vq/fz4rY6bvzoHGaNz/S7HBERiXEKXjJq7a5r4+anyvjQrHFcvWyq3+WIiMgooOAlo1JvIMi3HthAcmIc//WpE3QJCRERiYgEvwsQ8cN//30nG8ub+Z/PLWZ8Vorf5YiIyCihHi8ZdSqbOvndizv5xImFXLCgwO9yRERkFFHwklHnudJqAkHH18+Z5XcpIiIyyih4yaizorSGWfkZTBunezKKiEhkKXjJqHKwvYc39zZy/rwJfpciIiKjUNiCl5ndaWa1ZlbSb9oPzazCzDZ4jwvCtX6RgbywtZZA0HHevPF+lyIiIqNQOHu87gKWDzD9l865Rd7j6TCuX+R9VpRWU5CdwoLCbL9LERGRUShswcs59zLQGK73FzlanT0BXtlRx3lzx2Om63aJiEjk+THG62tmtsk7FDlmsIXM7HozW2Nma+rq6iJZn8Sol7bX0dUb1PguERHxTaSD1++BGcAioAr4xWALOududc4VO+eK8/LyIlSexLLnSqvJTk1kybRcv0sREZFRKqLByzlX45wLOOeCwG3AkkiuX0av3kCQF7bWcs7x+STE62ReERHxR0Q/gcys/2XCLwVKBltWZDi9uaeR5s5eHWYUERFfhe1ejWZ2H3AmMM7MyoGbgDPNbBHggL3Al8K1fpH+VpRWk5IYxxmzdNhaRET8E7bg5Zz7zACT7wjX+kQG45zjudIazpiVR2pSvN/liIjIKKbBLhLzNpU3U93SxXk6zCgiIj5T8JKYt6K0mvg44yPH5/tdioiIjHIKXhLznttSwynTcslJS/K7FBERGeUUvCSm7axtY2dtG+fN1b0ZRUTEfwpeEtOe3lwFwPL5BUdYUkREJPwUvCSmPb25iuIpY5iQneJ3KSIiIgpeErt21bWxtbqVCxaot0tERKKDgpfErGe8w4wfXaDLSIiISHRQ8JKY9dTmahZPzqEgO9XvUkRERAAFL4lRe+rbKatq0WFGERGJKgpeEpOefvswo4KXiIhEDwUviUlPb65i0aQcCnN0mFFERKKHgpfEnH0N7ZRWtnChertERCTKKHhJzHl6czWgsxlFRCT6KHhJzHl6cxUnTMqhaEya36WIiIi8i4KXxJT9DR1srmjmgvnq7RIRkeij4CUx5ZmS0NmMuoyEiIhEIwUviSlPb65iYVE2k3J1mFFERKKPgpfEjF11bWwsb1Zvl4iIRC0FL4kZf1q1j8R445OLi/wuRUREZEAKXhIT2rv7eHhtORcsKCAvM9nvckRERAak4CUx4bENFbR293HVsil+lyIiIjIoBS8Z8Zxz/GnVPuYWZLF48hi/yxERERmUgpeMeGv2HWRrdStXLZuCmfldjoiIyKAUvGTEu2fVPjJTErh40US/SxERETksBS8Z0Wpbu3i2pIpPnTSJtKQEv8sRERE5LAUvGdHuf/MAvQHH5zWoXkRERgAFLxmx+gJB7l29nw/NGse0cel+lyMiInJECl4yYj1fVkN1SxdXLZvqdykiIiJDErbgZWZ3mlmtmZX0m5ZrZivNbIf3Vef+yzG7Z9U+CnNSOXtOvt+liIiIDEk4e7zuApa/Z9qNwAvOuVnAC15b5KhVNnXy+q4GPrNkEvFxuoSEiIiMDGELXs65l4HG90y+BLjbe3438PFwrV9i2/NlNQB8VDfEFhGRESTSY7zGO+eqvOfVwPjBFjSz681sjZmtqauri0x1MmKs3FLD9Lx0ZuRl+F2KiIjIkPk2uN455wB3mPm3OueKnXPFeXl5EaxMol1LVy9v7G7g3OMHze0iIiJRKdLBq8bMCgC8r7URXr/EgJe21dEbcJw7V8FLRERGlkgHryeAq73nVwOPR3j9EgNWbqlhbHoSJ+qG2CIiMsKE83IS9wGrgNlmVm5m1wE/A841sx3AR7y2yJD1BoK8uK2Ws+fk62xGEREZccJ2czvn3GcGmXVOuNYpse/NPY20dvXpMKOIiIxIunK9jCgrt9SQnBDHh2bphAsRERl5FLxkxHDOsXJLDR+aNY7UpHi/yxERETlqCl4yYpRVtVLR1KnDjCIiMmIpeMmIsXJLDWZw9hwFLxERGZkUvGTEWFlWzYmTcsjLTPa7FBERkWOi4CUjQmVTJyUVLZw7d4LfpYiIiBwzBS8ZEV7wboqt8V0iIjKSKXjJiPDclhqmjUtnRl6636WIiIgcMwUviXobDjSxalcD580dj5muVi8iIiOXgpdEtcb2Hr7y57WMz0rhy2fO8LscERGRDyRstwwS+aACQcc3719PfVsPD3/5VHLSkvwuSURE5ANR8JKo9esXdvDKjnr+/RMLWFCU7Xc5IiIiH5gONUpUenFrLb95YQeXnVTEFSdP8rscERGRYaHgJVHnQGMHNzywgeMLsvi3S+ZrQL2IiMQMBS+JKs45vnbfeoLO8fvPLdbNsEVEJKYoeElU2VLVwsYDTXz3/NlMHadrdomISGxR8JKosqKkmjiDjy4o8LsUERGRYafgJVHl2dJqTp6ay7gM3QhbRERij4KXRI3ddW1sr2lj+XzdCFtERGKTgpdEjRWloRthnz9PwUtERGKTgpdEjWdLq1lYlM3EnFS/SxEREQkLBS+JCpVNnWw80KTeLhERiWkKXhIVniutBtD4LhERiWkKXhIVVpTWMCs/gxl5GX6XIiIiEjYKXhIRgaDjm/ev54+v7XnfvMb2HlbvaVBvl4iIxLwEvwuQ0eHu1/fy+IZKHt9QSUZyAp8qfufG189vqSHodDajiIjEPgUvCbvKpk5+8dw2zjguj0AwyPcf2cz4rBTOOC4PCJ3NWJiTyryJWT5XKiIiEl461Chhd9MTpQSc4+aPz+f3V57EzPwMvvzntZRUNNPW3cerO+pZPn8CZuZ3qSIiImGl4CVhtaK0mpVbavjWR45jUm4aWSmJ3H3tErJTE7nmrrf48xv76AkENb5LRERGBV+Cl5ntNbPNZrbBzNb4UYOEX2tXLzc9XsqcCZlce/q0t6ePz0rhrmuX0N0b4GfPbGVcRhKLJ4/xsVIREZHI8LPH6yzn3CLnXLGPNUgY/eK57dS0dvHTTywgMf7dv2rHjc/k1quKSYqP48IFBcTH6TCjiIjEPg2ul7DYeKCJu1ft5cpTpgzam7V0+lhe/u5ZjElPjHB1IiIi/vCrx8sBz5nZWjO7fqAFzOx6M1tjZmvq6uoiXJ58EM45/vXxEvIykvnO8tmHXXZCdgrJCfERqkxERMRffgWv051zi4GPAl81szPeu4Bz7lbnXLFzrjgvLy/yFcox21jezMbyZr5+ziyyUtSbJSIicogvwcs5V+F9rQUeBZb4UYeEx32r95OWFM/HF030uxQREZGoEvHgZWbpZpZ56DlwHlAS6TokPFq7enliYyUXnzCRTPV2iYiIvIsfg+vHA496F8tMAO51zj3rQx0SBo9vqKSzN8Bnlkz2uxQREZGoE/Hg5ZzbDZwQ6fVK+DnnuHf1fuYWZLGwKNvvckRERKKOrlwvw2ZzRTNbqlr4zCmTdfsfERGRASh4ybC57839pCbGc4kG1YuIiAxIwUuGRVt3H49vqOSiEwp0CQkREZFBKHjJsHhiQyUdPRpULyIicji6ZZDgnONvm6rYUdPKmLQkctOTGJOeRG5aErPGZ5CSeOQry9/35n7mTMhk0aSc8BcsIiIyQil4jXJ1rd18/5FNPF9WO+D8KWPTuOPqYmbmZw76HpvLm9lc0cyPLp6nQfUiIiKHoeA1iq0oreZfHtlMa3cf//qxuVy1bApt3X00tvdwsKOHA42d/OSpLVz6u9f5zWdP5KzZ+e97j2DQ8cfX95CcEMfHTyz04bsQEREZORS8RqHWrl5+/Lct/HVtOfMmZnH/5YuYNT7Uo5WTlkROWhIAJ02Bk6fl8sW713DdXW/x/Y8ezz99aBpmRm8gyBMbKvn9S7vYWdvGlUsnk52qQfUiIiKHo+A1ytS1dvPZ295gV10bXztrJt84ZxZJCYOfY1GYk8pDX17Gtx/cyM1Pl7GtppWFRdn84aXdVDR1MmdCJr++YhEXLiiI4HchIiIyMplzzu8ajqi4uNitWbPG7zJGvPq2UOja39jBHVefzGkzxw35tcGg49cv7ODXL+wA4KQpY/jqWTM4a3a+xnWJiIj0Y2ZrnXPFA81Tj9co0dDWzeduW83+xg7u/MLJnDpj6KELIC7O+Na5x3HKtFwS4uM4eeoYBS4REZGjpOA1CjS29/C521ezt6GdPx5D6Orv1KPoJRMREZF30wVUY1xjew+fve0N9tS3c8fVJys4iYiI+Eg9XjHshbIabnqilLrWbm6/upjTZyl0iYiI+EnBKwZVNHXyoydKeW5LDbPyM7j3i6dw0pRcv8sSEREZ9RS8YkhvIMgdr+7h18+Hzjz83vI5XHf6tMNeLkJEREQiR8ErBjjneKGslp8+U8buunbOnTuemy6aS9GYNL9LExERkX4UvEa4kopmbn6qjFW7G5g+Lp07ri7mnOPH+12WiIiIDEDBa4QqP9jBLSu38+j6CnJSE/nRxfP47CmTSYzXYUUREZFopeA1gjjneG1nA/es2svzZTUkxMVx/RnT+epZM8lK0X0SRUREop2C1wjQ1t3Hw2vLuWfVXnbVtZObnsSXPjyDzy+dwsScVL/LExERkSFS8Ipyr+2s5zt/3UhlcxcnTMrhF586gQsXFpCSGO93aSIiInKUFLyiVEdPHz97Ziv3rNrH9HHpPPilZSyZpmtxiYiIjGQKXkPgnKO+rYeKpk7qWrvpDQTpDQTp6QvSF3TEGYxNTyYvM5lxmcmMy0giOSGeYNDR1RegqzdIV2+AlMR4ctOTjri+tfsa+faDG9nb0MG1p03jO+fPJjVJPVwiIiIjnYLXIF7eXsedr+3hQGMHFU2ddPUGj+r1SfFx9ATe/5rJuWksnpzD4iljWDx5DPmZyeyub2dXXRu7atvZUdvKazvrmZiTyn1fXMqyGWOH61sSERERnyl4vUcw6Pjtizv55fPbKcxJZUFhNmfPyacwJ5WiMWnkZyWTlBBHYnwcSfGhr33BIA1tPdS3dVPX2k19Wzdt3QFSEuNISYwnJSH0tbmzl/X7m3htVwOPbah837pTE+OZnpfONadN41vnHkdGsjaPiIhILNEnez/Nnb18+8ENPF9Wy6UnFvLTSxcM+RDf0Vwl3jlH+cFO1u0/SFNHL9PGpTMjP4OCrBTi4uxYyxcREZEop+Dl2Vrdwj//aS3lBzv50cXzuGrZFMzCE4LMjEm5aUzK1S19RERERhNfLnNuZsvNbJuZ7TSzG/2oob9nS6q49Hev09ET4P7rl3L1qVPDFrpERERk9Ip4j5eZxQO/A84FyoG3zOwJ59yWSNdySE5aEosm5fDrKxaRn5XiVxkiIiIS4/w41LgE2Omc2w1gZvcDlwC+Ba+l08dyyhdz1cslIiIiYeXHocZC4EC/drk3zVcKXSIiIhJuvozxGgozu97M1pjZmrq6Or/LEREREfnA/AheFcCkfu0ib9q7OOdudc4VO+eK8/LyIlaciIiISLj4EbzeAmaZ2TQzSwKuAJ7woQ4RERGRiIr44HrnXJ+ZfQ1YAcQDdzrnSiNdh4iIiEik+XIBVefc08DTfqxbRERExC9RO7heREREJNYoeImIiIhEiIKXiIiISIQoeImIiIhEiIKXiIiISIQoeImIiIhEiIKXiIiISISYc87vGo7IzOqAfcP4luOA+mF8P/lgtD2ih7ZFdNH2iB7aFtEl2rfHFOfcgPc7HBHBa7iZ2RrnXLHfdUiItkf00LaILtoe0UPbIrqM5O2hQ40iIiIiEaLgJSIiIhIhozV43ep3AfIu2h7RQ9siumh7RA9ti+gyYrfHqBzjJSIiIuKH0drjJSIiIhJxoy54mdlyM9tmZjvN7Ea/64kVZjbJzF40sy1mVmpm3/Sm55rZSjPb4X0d4003M/uNtx02mdnifu91tbf8DjO7ut/0k8xss/ea35iZRf47HTnMLN7M1pvZk157mpmt9n5+D5hZkjc92Wvv9OZP7fce3/embzOz8/tN1350FMwsx8weMrOtZlZmZsu0b/jDzL7l/Y0qMbP7zCxF+0bkmNmdZlZrZiX9poV9XxhsHb5wzo2aBxAP7AKmA0nARmCu33XFwgMoABZ7zzOB7cBc4OfAjd70G4H/8J5fADwDGLAUWO1NzwV2e1/HeM/HePPe9JY177Uf9fv7juYH8P8B9wJPeu0HgSu85/8LfNl7/hXgf73nVwAPeM/nevtIMjDN23fitR8d07a4G/gn73kSkKN9w5ftUAjsAVK99oPAF7RvRHQbnAEsBkr6TQv7vjDYOvx4jLYeryXATufcbudcD3A/cInPNcUE51yVc26d97wVKCP0R+4SQh86eF8/7j2/BLjHhbwB5JhZAXA+sNI51+icOwisBJZ787Kcc2+40J5zT7/3kvcwsyLgQuB2r23A2cBD3iLv3RaHttFDwDne8pcA9zvnup1ze4CdhPYh7UdHwcyyCX3Y3AHgnOtxzjWhfcMvCUCqmSUAaUAV2jcixjn3MtD4nsmR2BcGW0fEjbbgVQgc6Ncu96bJMPK6408EVgPjnXNV3qxqYLz3fLBtcbjp5QNMl4H9CvguEPTaY4Em51yf1+7/83v7Z+7Nb/aWP9ptJAObBtQBf/QO/d5uZulo34g451wF8F/AfkKBqxlYi/YNv0ViXxhsHRE32oKXhJmZZQAPAzc451r6z/P+A9FptGFmZh8Dap1za/2uRYBQD8ti4PfOuROBdkKHOt6mfSMyvHE9lxAKwxOBdGC5r0XJu0RiX/B7fxttwasCmNSvXeRNk2FgZomEQtdfnHOPeJNrvO5fvK+13vTBtsXhphcNMF3e7zTgYjPbS+hQx9nArwl10yd4y/T/+b39M/fmZwMNHP02koGVA+XOudVe+yFCQUz7RuR9BNjjnKtzzvUCjxDaX7Rv+CsS+8Jg64i40Ra83gJmeWewJBEaLPmEzzXFBG/cwx1AmXPuln6zngAOnXFyNfB4v+lXeWetLAWavW7gFcB5ZjbG++/0PGCFN6/FzJZ667qq33tJP8657zvnipxzUwn9jv/dOfc54EXgMm+x926LQ9voMm95502/wjuzaxowi9DAVe1HR8E5Vw0cMLPZ3qRzgC1o3/DDfmCpmaV5P6tD20L7hr8isS8Mto7I82tUv18PQmdJbCd05skP/K4nVh7A6YS6bjcBG7zHBYTGQ7wA7ACeB3K95Q34nbcdNgPF/d7rWkKDVXcC1/SbXgyUeK/5Ld4FgPU47HY5k3fOapxO6MNhJ/BXINmbnuK1d3rzp/d7/Q+8n/c2+p0pp/3oqLfDImCNt388RuhMLO0b/myLHwFbvZ/Xnwidmah9I3I///sIja/rJdQbfF0k9oXB1uHHQ1euFxEREYmQ0XaoUURERMQ3Cl4iIiIiEaLgJSIiIhIhCl4iIiIiEaLgJSIiIhIhCl4iMmKZ2Q/MrNTMNpnZBjM7xZt+g5mlhWF9q7yvjx66GKOIyNFIOPIiIiLRx8yWAR8DFjvnus1sHJDkzb4B+DPQMYzrmwns9C7MONG9c983EZEhU4+XiIxUBUC9c64bwDlX75yrNLNvELoP34tm9iKAmZ1nZqvMbJ2Z/dW7pyhmttfMfm5mm83sTS9cvYuZpZrZBuDvhC5IW0bo6uQbzGxRJL5REYkduoCqiIxIXnh6FUgjdCXqB5xzL3nz9hK6ynW91xP2CKGri7eb2fcIXZn8x95ytznnbjazq4BPO+c+Nsj6fgfcCcwHMpxzvwvztygiMUg9XiIyIjnn2oCTgOuBOuABM/vCAIsuBeYCr3k9V1cDU/rNv6/f12WHWeUCoBRYCGz8ILWLyOilMV4iMmI55wLAP4B/mNlmQqHqrvcsZsBK59xnBnubQZ6HXmz2r8AngRnAG4Tu63eemT3rnPvOB/oGRGTUUY+XiIxIZjbbzGb1m7QI2Oc9bwUyvedvAKcdGr9lZulmdly/113e7+uq967HOfdj4J+APwKnABudcwsUukTkWKjHS0RGqgzgv80sB+gDdhI67AhwK/CsmVU6587yDkHeZ2bJ3vz/C2z3no8xs01ANzBYr9iHgVeAJYSCnIjIMdHgehEZtfoPwve7FhEZHXSoUURERCRC1OMlIiIiEiHq8RIRERGJEAUvERERkQhR8BIRERGJEAUvERERkQhR8BIRERGJEAUvERERkQj5fzR6ORvaVtyBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_scores():\n",
    "    fig = plt.figure(figsize=(10, 6))\n",
    "    ax = fig.add_subplot(111)\n",
    "    \n",
    "    plt.plot(*zip(*scores))\n",
    "    plt.plot(\n",
    "        list(map(lambda pair: pair[0], scores)),\n",
    "        [30] * len(scores),\n",
    "        \":\"\n",
    "    )\n",
    "\n",
    "    plt.ylabel('Score')\n",
    "    plt.xlabel('Step #')\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "plot_scores()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the agent consistently achieves a score higher than 30 within the first 100 episodes.\n",
    "\n",
    "Finally, runing the following cell would simulate one run to demonstrate how the trained agent performs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "39.602003537118435"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agent.run_episode()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "Plappert et al. (2017) suggested using parameter space noisy for more effective exploration, which would be interesting to try. We did try using the similar technique of using NoisyNet proposed by Fortunato et al. (2017) though, but it didn't seem to help\n",
    "\n",
    "We also didn't get to try incorporating the **distributional critic** extension as in D4PG, which could provide further performance improvement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "- Lillicrap, T. P. et al. (2015) ‘Continuous control with deep reinforcement learning’. Available at: http://arxiv.org/abs/1509.02971\n",
    "- Fujimoto, S., van Hoof, H. and Meger, D. (2018) ‘Addressing Function Approximation Error in Actor-Critic Methods’. Available at: https://arxiv.org/abs/1802.09477\n",
    "- Barth-Maron, G. et al. (2018) ‘Distributed Distributional Deterministic Policy Gradients’. Available at: https://arxiv.org/abs/1804.08617\n",
    "- Schaul, T. et al. (2015) ‘Prioritized Experience Replay’. Available at: http://arxiv.org/abs/1511.05952\n",
    "- Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R. Y., Chen, X., Asfour, T., Abbeel, P., & Andrychowicz, M. (2017). Parameter Space Noise for Exploration. https://arxiv.org/abs/1706.01905\n",
    "- Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., Blundell, C., & Legg, S. (2017). Noisy Networks for Exploration. https://arxiv.org/abs/1706.10295"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "drlnd",
   "language": "python",
   "name": "drlnd"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}