public-comment-period-seg-claude-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7475a584-99a1-4ae2-bfca-f152e39a09ca",
   "metadata": {},
   "source": [
    "# Public Comment Period Segmentation\n",
    "\n",
    "## High Level Overview\n",
    "\n",
    "This notebook demonstrates how to use Large Language Models to segment out the \"public comment\" and \"public hearing\" portions of municipal meeting transcripts. There may be multiple of these periods within a single meeting.\n",
    "\n",
    "The way that this is setup is we prompt an LLM to record the \"first_sentence_text\" and the \"last_sentence_text\" for each period. In addition we specify how the model should output the processed result (a structured JSON object). You can view the whole prompt in the [./prompts/v0-public-comment-period-seg.jinja](./prompts/v0-public-comment-period-seg.jinja) file.\n",
    "\n",
    "## Dataset\n",
    "\n",
    "I manually annotated 12 meeting transcripts from the Seattle CDP dataset, marking their first and last sentence of the public comment periods. If I recall correctly, there were 3 meetings that had multiple public comment periods (or public hearings).\n",
    "\n",
    "We convert the annotated dataset into a DataFrame with the full text of the transcript, the metadata of the event, and the \"masses\" of the segmentation, the masses tell the system how long each portion of the meeting is. For example, if the masses are: `[100, 200, 100]` the portions might be something like a string 100 characters long that is unclassified, then a string of 200 characters long that we assume is the annotated public comment period, and finally another string 100 characters long to close out the meeting.\n",
    "\n",
    "In the case of multiple comment periods, the masses might look like: `[100, 200, 100, 200, 100]` (not-comment, comment, not-comment, comment, not-comment).\n",
    "\n",
    "## LLMs\n",
    "\n",
    "I have tested with a few LLMs: OpenAI's gpt-3.5-turbo, gpt-3.5-turbo-instruct-16k, gpt-3.5-turbo-16k, and gpt-4; Anthropic's claude-2-100k; and [an open-source fine-tuned version of Meta's llama-2](https://huggingface.co/Yukang/Llama-2-7b-longlora-100k-ft).\n",
    "\n",
    "The latest commit of this notebook uses Anthropic's Claude 2 for demonstration because it is the simpliest model to use. The others do not have context windows large enough to fit the whole transcript into context and need to be partitioned. There are methods for handling that that showed promising results but for the sake of demonstration, I will stick with Claude-2.\n",
    "\n",
    "## Evaluation\n",
    "\n",
    "We use `segeval` to evaluate the performance of the system. In general we want to compare the relative distances between reference public comment segmentation boundaries and the predicted public comment segmentation boundaries. The closer they are the better, however because of how much administrative \"cruft\" is in the transcript / meeting, there does need to be some flexibility here because the clerk or the chair of the meeting may talk about opening the public comment period multiple times in the adminstration portion of transitioning to public comment. For example, they might say \"Let's move on to public comment\" and then after describing how public comment will work (time allotment, rules, etc.) they will finally say \"We are now open to public comment, our first commenter is Eva.\" It isn't clear which one is \"better\" for segmentation and the system may choose either so we have to allow for tolerance but otherwise, `segeval` works great."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14b594c8-4821-404c-81bf-841350af0e75",
   "metadata": {},
   "source": [
    "## Dataset Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b36f2b86-84b4-4000-8c21-94f9a4e27f0f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>meta</th>\n",
       "      <th>true_masses</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>We are recording. Wonderful. Okay. Good aftern...</td>\n",
       "      <td>{'event_id': '84bfb428c005', 'session_id': 'c4...</td>\n",
       "      <td>[1046, 563, 41062]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Good afternoon, everybody. Today is Tuesday, M...</td>\n",
       "      <td>{'event_id': 'c86c94ed1db7', 'session_id': 'c6...</td>\n",
       "      <td>[8847, 11850, 7373]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Good afternoon, everyone, the September 14th, ...</td>\n",
       "      <td>{'event_id': 'fa3fd088de8e', 'session_id': '7c...</td>\n",
       "      <td>[4282, 12860, 21504]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Discussion. 228, council meeting will come her...</td>\n",
       "      <td>{'event_id': '7cc7c93e7a63', 'session_id': 'f7...</td>\n",
       "      <td>[141623]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thank you. Have a great day. Good morning, eve...</td>\n",
       "      <td>{'event_id': 'c511fea02999', 'session_id': '93...</td>\n",
       "      <td>[447, 3749, 72862]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Thank you very much. Thank you. The December 6...</td>\n",
       "      <td>{'event_id': '2adc154d91b0', 'session_id': '6d...</td>\n",
       "      <td>[1537, 7617, 3517, 1938, 18775]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Thank you, son. Good afternoon, everybody. It'...</td>\n",
       "      <td>{'event_id': '4477546b534d', 'session_id': '72...</td>\n",
       "      <td>[1087, 8777, 2406]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Good afternoon, the September 21st, 2020 meeti...</td>\n",
       "      <td>{'event_id': '6562c700d929', 'session_id': '97...</td>\n",
       "      <td>[12166, 34099, 59278]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Director Sawyer is ready to ready to go. After...</td>\n",
       "      <td>{'event_id': 'fc5983dfdc1f', 'session_id': 'cd...</td>\n",
       "      <td>[3744, 11233, 11469, 7982, 40536]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Budget meeting starting in a moment. I will ca...</td>\n",
       "      <td>{'event_id': '23333c839436', 'session_id': '3b...</td>\n",
       "      <td>[10222, 12327, 102269]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Good morning, everybody. The March 3, 2021 mee...</td>\n",
       "      <td>{'event_id': '749503d88894', 'session_id': 'f6...</td>\n",
       "      <td>[1671, 10938, 1491, 1725, 144754]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>The May 11th, 2022 meeting of the Seattle City...</td>\n",
       "      <td>{'event_id': 'b5e3673a68ff', 'session_id': 'c9...</td>\n",
       "      <td>[3133, 3043, 59419]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 text  \\\n",
       "0   We are recording. Wonderful. Okay. Good aftern...   \n",
       "1   Good afternoon, everybody. Today is Tuesday, M...   \n",
       "2   Good afternoon, everyone, the September 14th, ...   \n",
       "3   Discussion. 228, council meeting will come her...   \n",
       "4   Thank you. Have a great day. Good morning, eve...   \n",
       "5   Thank you very much. Thank you. The December 6...   \n",
       "6   Thank you, son. Good afternoon, everybody. It'...   \n",
       "7   Good afternoon, the September 21st, 2020 meeti...   \n",
       "8   Director Sawyer is ready to ready to go. After...   \n",
       "9   Budget meeting starting in a moment. I will ca...   \n",
       "10  Good morning, everybody. The March 3, 2021 mee...   \n",
       "11  The May 11th, 2022 meeting of the Seattle City...   \n",
       "\n",
       "                                                 meta  \\\n",
       "0   {'event_id': '84bfb428c005', 'session_id': 'c4...   \n",
       "1   {'event_id': 'c86c94ed1db7', 'session_id': 'c6...   \n",
       "2   {'event_id': 'fa3fd088de8e', 'session_id': '7c...   \n",
       "3   {'event_id': '7cc7c93e7a63', 'session_id': 'f7...   \n",
       "4   {'event_id': 'c511fea02999', 'session_id': '93...   \n",
       "5   {'event_id': '2adc154d91b0', 'session_id': '6d...   \n",
       "6   {'event_id': '4477546b534d', 'session_id': '72...   \n",
       "7   {'event_id': '6562c700d929', 'session_id': '97...   \n",
       "8   {'event_id': 'fc5983dfdc1f', 'session_id': 'cd...   \n",
       "9   {'event_id': '23333c839436', 'session_id': '3b...   \n",
       "10  {'event_id': '749503d88894', 'session_id': 'f6...   \n",
       "11  {'event_id': 'b5e3673a68ff', 'session_id': 'c9...   \n",
       "\n",
       "                          true_masses  \n",
       "0                  [1046, 563, 41062]  \n",
       "1                 [8847, 11850, 7373]  \n",
       "2                [4282, 12860, 21504]  \n",
       "3                            [141623]  \n",
       "4                  [447, 3749, 72862]  \n",
       "5     [1537, 7617, 3517, 1938, 18775]  \n",
       "6                  [1087, 8777, 2406]  \n",
       "7               [12166, 34099, 59278]  \n",
       "8   [3744, 11233, 11469, 7982, 40536]  \n",
       "9              [10222, 12327, 102269]  \n",
       "10  [1671, 10938, 1491, 1725, 144754]  \n",
       "11                [3133, 3043, 59419]  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Load annotated dataset\n",
    "df = pd.read_json(\"trial-datasets/seattle-public-comment-period-seg-v0.jsonl\", lines=True)\n",
    "\n",
    "# Prep dataset for eval\n",
    "prepped_eval_rows = []\n",
    "for _, row in df.iterrows():\n",
    "    # Get text\n",
    "    text = row[\"text\"]\n",
    "    \n",
    "    # Get meta\n",
    "    meta = row[\"meta\"]\n",
    "\n",
    "    # Construct masses\n",
    "    masses = []\n",
    "    prev_index = 0\n",
    "    if isinstance(row[\"spans\"], list):\n",
    "        for span in row[\"spans\"]:\n",
    "            # Choose what index to get based off label\n",
    "            if span[\"label\"] == \"FIRST-SENTENCE\":\n",
    "                # Get start index\n",
    "                mass_calc_index = span[\"start\"]\n",
    "            else:\n",
    "                # Get end index\n",
    "                mass_calc_index = span[\"end\"]\n",
    "\n",
    "            # Add masses to list\n",
    "            masses.append(mass_calc_index - prev_index)\n",
    "\n",
    "            # Update prev index\n",
    "            prev_index = mass_calc_index\n",
    "        \n",
    "        # Add final mass\n",
    "        masses.append(len(text) - prev_index)\n",
    "    else:\n",
    "        # Add mass for full text\n",
    "        masses.append(len(text))\n",
    "\n",
    "    # Add to list\n",
    "    prepped_eval_rows.append({\n",
    "        \"text\": row[\"text\"],\n",
    "        \"meta\": row[\"meta\"],\n",
    "        \"true_masses\": masses,\n",
    "    })\n",
    "\n",
    "# Convert to dataframe\n",
    "prepped_eval_df = pd.DataFrame(prepped_eval_rows)\n",
    "# prepped_eval_df = prepped_eval_df.sample(3)\n",
    "prepped_eval_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f991adf3-645e-463c-927a-1fd2aa3c2be1",
   "metadata": {},
   "source": [
    "## Prompt Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8e792af8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "import backoff\n",
    "from dotenv import load_dotenv\n",
    "from langchain.chat_models import ChatAnthropic, ChatOpenAI\n",
    "from langchain.chat_models.base import BaseChatModel\n",
    "from langchain.llms import HuggingFaceEndpoint\n",
    "from langchain.output_parsers import PydanticOutputParser\n",
    "from langchain import PromptTemplate\n",
    "from langchain.schema import HumanMessage\n",
    "from pydantic import BaseModel, Field\n",
    "import spacy\n",
    "\n",
    "###############################################################################\n",
    "\n",
    "load_dotenv()\n",
    "llm = ChatAnthropic(model=\"claude-2.0\", temperature=0, max_tokens_to_sample=4096)\n",
    "# llm = ChatOpenAI(model=\"gpt-3.5-turbo-16k\", temperature=0, max_tokens=4096)\n",
    "# llm = HuggingFaceEndpoint(\n",
    "#     endpoint_url=\"https://boxjj56zj0zbbjue.us-east-1.aws.endpoints.huggingface.cloud\",\n",
    "#     task=\"text2text-generation\",\n",
    "# )\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_trf\")\n",
    "\n",
    "###############################################################################\n",
    "\n",
    "class PublicCommentPeriod(BaseModel):\n",
    "    comment_period_initiated_text: str | None = Field(\n",
    "        description=(\n",
    "            \"the text of the sentence which introduces the public comment period, \"\n",
    "            \"or null if no public comment period was found\"\n",
    "        ),\n",
    "    )\n",
    "    comment_period_concluded_text: str | None = Field(\n",
    "        description=(\n",
    "            \"the text of the sentence which concludes the public comment period, \"\n",
    "            \"or if null no public comment period was found\"\n",
    "        ),\n",
    "    )\n",
    "\n",
    "class MultiPublicCommentPeriod(BaseModel):\n",
    "    periods: list[PublicCommentPeriod] = Field(\n",
    "        description=\"the list of public comment periods (sometimes also called public hearings)\",\n",
    "    )\n",
    "\n",
    "PUBLIC_COMMENT_PERIOD_SEG_PARSER = PydanticOutputParser(pydantic_object=MultiPublicCommentPeriod)\n",
    "\n",
    "###############################################################################\n",
    "\n",
    "PUBLIC_COMMENT_PERIOD_SEG_PROMPT = PromptTemplate.from_file(\n",
    "    \"prompts/v0-public-comment-period-seg.jinja\",\n",
    "    input_variables=[\"transcript\"],\n",
    "    partial_variables={\n",
    "        \"format_instructions\": PUBLIC_COMMENT_PERIOD_SEG_PARSER.get_format_instructions(),\n",
    "    },\n",
    "    template_format=\"jinja2\",\n",
    ")\n",
    "\n",
    "@backoff.on_exception(backoff.expo, json.JSONDecodeError, max_tries=3)\n",
    "def _process_transcript(text: str) -> list[int]:\n",
    "    # Convert text to sentences\n",
    "    sentences = list(nlp(text).sents)\n",
    "\n",
    "    # Convert to prompt ready string\n",
    "    transcript_str = \"\\n\\n\".join([sent.text for sent in sentences])\n",
    "\n",
    "    # Fill the prompt\n",
    "    input_ = PUBLIC_COMMENT_PERIOD_SEG_PROMPT.format_prompt(transcript=transcript_str)\n",
    "\n",
    "    # Generate\n",
    "    if isinstance(llm, BaseChatModel):\n",
    "        # Generate\n",
    "        output = llm([HumanMessage(content=input_.to_string())]).content\n",
    "    else:\n",
    "        # Generate\n",
    "        output = llm(input_.to_string())\n",
    "\n",
    "    # Parse output\n",
    "    try:\n",
    "        pc_periods = PUBLIC_COMMENT_PERIOD_SEG_PARSER.parse(output)\n",
    "\n",
    "    except:\n",
    "        print(output)\n",
    "        raise Exception(\"Failed to parse output\")\n",
    "\n",
    "    # Process all periods found\n",
    "    prev_index = 0\n",
    "    predicted_masses = []\n",
    "    for pc_period in pc_periods.periods:\n",
    "        # Process masses\n",
    "        if (\n",
    "            pc_period.comment_period_initiated_text is not None\n",
    "            and pc_period.comment_period_concluded_text is not None\n",
    "        ):\n",
    "            first_sentence_index = text.find(pc_period.comment_period_initiated_text)\n",
    "            predicted_masses.append(first_sentence_index - prev_index)\n",
    "            prev_index = first_sentence_index\n",
    "\n",
    "            last_sentence_index = text.find(pc_period.comment_period_concluded_text)\n",
    "            predicted_masses.append(last_sentence_index - prev_index)\n",
    "            prev_index = last_sentence_index\n",
    "\n",
    "    # Add final mass (or full text as mass)\n",
    "    if len(predicted_masses) == 0:\n",
    "        predicted_masses.append(len(text))\n",
    "    else:\n",
    "        predicted_masses.append(len(text) - prev_index)\n",
    "\n",
    "    # Correct for weird \"not found\" mistakes\n",
    "    if len(text) in predicted_masses:\n",
    "        predicted_masses = [len(text)]\n",
    "\n",
    "    return predicted_masses"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d728245c-f246-4b37-990c-85f5fdf313de",
   "metadata": {},
   "source": [
    "## Outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "625e7188",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [2:04:17<00:00, 621.50s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Similarity: 0.5922531512151178\n",
      "Mean Precision: 0.5860069386061516\n",
      "Mean Recall: 0.5860069386061516\n",
      "Mean F1: 0.63291926191296\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>meta</th>\n",
       "      <th>true_masses</th>\n",
       "      <th>predicted_masses</th>\n",
       "      <th>similarity</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>f1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>We are recording. Wonderful. Okay. Good aftern...</td>\n",
       "      <td>{'event_id': '84bfb428c005', 'session_id': 'c4...</td>\n",
       "      <td>[1046, 563, 41062]</td>\n",
       "      <td>[1046, 324, 41301]</td>\n",
       "      <td>0.959979906229068988613529806</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Good afternoon, everybody. Today is Tuesday, M...</td>\n",
       "      <td>{'event_id': 'c86c94ed1db7', 'session_id': 'c6...</td>\n",
       "      <td>[8847, 11850, 7373]</td>\n",
       "      <td>[-1, 20549, 7522]</td>\n",
       "      <td>0.3080448065173116089613034623</td>\n",
       "      <td>0.03654648025508952661270542065</td>\n",
       "      <td>0.03654648025508952661270542065</td>\n",
       "      <td>0.07051585423568386180785612872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Good afternoon, everyone, the September 14th, ...</td>\n",
       "      <td>{'event_id': 'fa3fd088de8e', 'session_id': '7c...</td>\n",
       "      <td>[4282, 12860, 21504]</td>\n",
       "      <td>[4282, -4283, 38647]</td>\n",
       "      <td>0.3333333333333333333333333333</td>\n",
       "      <td>0.3333333333333333333333333333</td>\n",
       "      <td>0.3333333333333333333333333333</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Discussion. 228, council meeting will come her...</td>\n",
       "      <td>{'event_id': '7cc7c93e7a63', 'session_id': 'f7...</td>\n",
       "      <td>[141623]</td>\n",
       "      <td>[-1, 0, 141624]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thank you. Have a great day. Good morning, eve...</td>\n",
       "      <td>{'event_id': 'c511fea02999', 'session_id': '93...</td>\n",
       "      <td>[447, 3749, 72862]</td>\n",
       "      <td>[447, -448, 77059]</td>\n",
       "      <td>0.3333333333333333333333333333</td>\n",
       "      <td>0.3333333333333333333333333333</td>\n",
       "      <td>0.3333333333333333333333333333</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Thank you very much. Thank you. The December 6...</td>\n",
       "      <td>{'event_id': '2adc154d91b0', 'session_id': '6d...</td>\n",
       "      <td>[1537, 7617, 3517, 1938, 18775]</td>\n",
       "      <td>[4584, 9646, 19154]</td>\n",
       "      <td>0.1675513698630136986301369864</td>\n",
       "      <td>0.03897973876375604237375295691</td>\n",
       "      <td>0.03897973876375604237375295691</td>\n",
       "      <td>0.07503464660463274599089289249</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Thank you, son. Good afternoon, everybody. It'...</td>\n",
       "      <td>{'event_id': '4477546b534d', 'session_id': '72...</td>\n",
       "      <td>[1087, 8777, 2406]</td>\n",
       "      <td>[980, 8841, 2449]</td>\n",
       "      <td>0.9125874125874125874125874125</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Good afternoon, the September 21st, 2020 meeti...</td>\n",
       "      <td>{'event_id': '6562c700d929', 'session_id': '97...</td>\n",
       "      <td>[12166, 34099, 59278]</td>\n",
       "      <td>[12166, 34051, 59326]</td>\n",
       "      <td>0.996751488900920411478072550</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Director Sawyer is ready to ready to go. After...</td>\n",
       "      <td>{'event_id': 'fc5983dfdc1f', 'session_id': 'cd...</td>\n",
       "      <td>[3744, 11233, 11469, 7982, 40536]</td>\n",
       "      <td>[-1, 0, 3745, 29505, 41715]</td>\n",
       "      <td>0.3550600343053173241852487136</td>\n",
       "      <td>0.2898903775883069427527405603</td>\n",
       "      <td>0.2898903775883069427527405603</td>\n",
       "      <td>0.4494806421152030217186024551</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Budget meeting starting in a moment. I will ca...</td>\n",
       "      <td>{'event_id': '23333c839436', 'session_id': '3b...</td>\n",
       "      <td>[10222, 12327, 102269]</td>\n",
       "      <td>[10222, 12259, 102337]</td>\n",
       "      <td>0.9961085040631795810919079775</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Good morning, everybody. The March 3, 2021 mee...</td>\n",
       "      <td>{'event_id': '749503d88894', 'session_id': 'f6...</td>\n",
       "      <td>[1671, 10938, 1491, 1725, 144754]</td>\n",
       "      <td>[1671, 10878, 1218, 1942, 144870]</td>\n",
       "      <td>0.9886788256227758007117437722</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>The May 11th, 2022 meeting of the Seattle City...</td>\n",
       "      <td>{'event_id': 'b5e3673a68ff', 'session_id': 'c9...</td>\n",
       "      <td>[3133, 3043, 59419]</td>\n",
       "      <td>[982, 5101, 59512]</td>\n",
       "      <td>0.7556087998257460248311914615</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 text  \\\n",
       "0   We are recording. Wonderful. Okay. Good aftern...   \n",
       "1   Good afternoon, everybody. Today is Tuesday, M...   \n",
       "2   Good afternoon, everyone, the September 14th, ...   \n",
       "3   Discussion. 228, council meeting will come her...   \n",
       "4   Thank you. Have a great day. Good morning, eve...   \n",
       "5   Thank you very much. Thank you. The December 6...   \n",
       "6   Thank you, son. Good afternoon, everybody. It'...   \n",
       "7   Good afternoon, the September 21st, 2020 meeti...   \n",
       "8   Director Sawyer is ready to ready to go. After...   \n",
       "9   Budget meeting starting in a moment. I will ca...   \n",
       "10  Good morning, everybody. The March 3, 2021 mee...   \n",
       "11  The May 11th, 2022 meeting of the Seattle City...   \n",
       "\n",
       "                                                 meta  \\\n",
       "0   {'event_id': '84bfb428c005', 'session_id': 'c4...   \n",
       "1   {'event_id': 'c86c94ed1db7', 'session_id': 'c6...   \n",
       "2   {'event_id': 'fa3fd088de8e', 'session_id': '7c...   \n",
       "3   {'event_id': '7cc7c93e7a63', 'session_id': 'f7...   \n",
       "4   {'event_id': 'c511fea02999', 'session_id': '93...   \n",
       "5   {'event_id': '2adc154d91b0', 'session_id': '6d...   \n",
       "6   {'event_id': '4477546b534d', 'session_id': '72...   \n",
       "7   {'event_id': '6562c700d929', 'session_id': '97...   \n",
       "8   {'event_id': 'fc5983dfdc1f', 'session_id': 'cd...   \n",
       "9   {'event_id': '23333c839436', 'session_id': '3b...   \n",
       "10  {'event_id': '749503d88894', 'session_id': 'f6...   \n",
       "11  {'event_id': 'b5e3673a68ff', 'session_id': 'c9...   \n",
       "\n",
       "                          true_masses                   predicted_masses  \\\n",
       "0                  [1046, 563, 41062]                 [1046, 324, 41301]   \n",
       "1                 [8847, 11850, 7373]                  [-1, 20549, 7522]   \n",
       "2                [4282, 12860, 21504]               [4282, -4283, 38647]   \n",
       "3                            [141623]                    [-1, 0, 141624]   \n",
       "4                  [447, 3749, 72862]                 [447, -448, 77059]   \n",
       "5     [1537, 7617, 3517, 1938, 18775]                [4584, 9646, 19154]   \n",
       "6                  [1087, 8777, 2406]                  [980, 8841, 2449]   \n",
       "7               [12166, 34099, 59278]              [12166, 34051, 59326]   \n",
       "8   [3744, 11233, 11469, 7982, 40536]        [-1, 0, 3745, 29505, 41715]   \n",
       "9              [10222, 12327, 102269]             [10222, 12259, 102337]   \n",
       "10  [1671, 10938, 1491, 1725, 144754]  [1671, 10878, 1218, 1942, 144870]   \n",
       "11                [3133, 3043, 59419]                 [982, 5101, 59512]   \n",
       "\n",
       "                        similarity                        precision  \\\n",
       "0    0.959979906229068988613529806                                1   \n",
       "1   0.3080448065173116089613034623  0.03654648025508952661270542065   \n",
       "2   0.3333333333333333333333333333   0.3333333333333333333333333333   \n",
       "3                                0                                0   \n",
       "4   0.3333333333333333333333333333   0.3333333333333333333333333333   \n",
       "5   0.1675513698630136986301369864  0.03897973876375604237375295691   \n",
       "6   0.9125874125874125874125874125                                1   \n",
       "7    0.996751488900920411478072550                                1   \n",
       "8   0.3550600343053173241852487136   0.2898903775883069427527405603   \n",
       "9   0.9961085040631795810919079775                                1   \n",
       "10  0.9886788256227758007117437722                                1   \n",
       "11  0.7556087998257460248311914615                                1   \n",
       "\n",
       "                             recall                               f1  \n",
       "0                                 1                                1  \n",
       "1   0.03654648025508952661270542065  0.07051585423568386180785612872  \n",
       "2    0.3333333333333333333333333333                              0.5  \n",
       "3                                 0                                0  \n",
       "4    0.3333333333333333333333333333                              0.5  \n",
       "5   0.03897973876375604237375295691  0.07503464660463274599089289249  \n",
       "6                                 1                                1  \n",
       "7                                 1                                1  \n",
       "8    0.2898903775883069427527405603   0.4494806421152030217186024551  \n",
       "9                                 1                                1  \n",
       "10                                1                                1  \n",
       "11                                1                                1  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import segeval\n",
    "from tqdm import tqdm\n",
    "\n",
    "results = []\n",
    "for _, row in tqdm(prepped_eval_df.iterrows(), total=len(prepped_eval_df)):\n",
    "    # Get masses\n",
    "    predicted_masses = _process_transcript(row[\"text\"])\n",
    "\n",
    "    # Get similarity\n",
    "    #\n",
    "    # The parameter `n_t` allows for greater \"translational\" edits \n",
    "    # rather than \"addition\" or \"deletion\" edits.\n",
    "    # how this is actually computed is looking at the \n",
    "    # relative distances between \"boundary marks\"\n",
    "    # i.e.\n",
    "    # a reference segementation looks like:    0000000100000001000000010001000\n",
    "    # and a predicted segmentation looks like: 0000010000000000010000001001000\n",
    "    # however depending on how far away the boundary is,\n",
    "    # it can be considered an addition, deletion, or translation\n",
    "    # additions and deletions are penelized more heavily than translations\n",
    "    # in our case, there is a lot of fuzziness around \"what is the \n",
    "    # true first start sentence and last end sentence because there is lots of admin\n",
    "    # so to allow for this fuzziness, we boost the \"n (allowed) T(ranslation)\"\n",
    "    #\n",
    "    # Ultimately what this says is that we allow translation to be\n",
    "    # within 7% of the correct portion of the transcript\n",
    "    sim = segeval.boundary_similarity(row[\"true_masses\"], predicted_masses, n_t=int(len(row[\"text\"]) * 0.07))\n",
    "\n",
    "    # Get confusion matrix\n",
    "    matrix = segeval.boundary_confusion_matrix(row[\"true_masses\"], predicted_masses, n_t=int(len(row[\"text\"]) * 0.07))\n",
    "\n",
    "    # Get precision, recall, and f1\n",
    "    precision = segeval.precision(matrix)\n",
    "    recall = segeval.recall(matrix)\n",
    "    f1 = segeval.fmeasure(matrix)\n",
    "\n",
    "    # Add to results\n",
    "    results.append({\n",
    "        \"text\": row[\"text\"],\n",
    "        \"meta\": row[\"meta\"],\n",
    "        \"true_masses\": row[\"true_masses\"],\n",
    "        \"predicted_masses\": predicted_masses,\n",
    "        \"similarity\": sim,\n",
    "        \"precision\": precision,\n",
    "        \"recall\": recall,\n",
    "        \"f1\": f1,\n",
    "    })\n",
    "\n",
    "# Convert to dataframe\n",
    "results_df = pd.DataFrame(results)\n",
    "\n",
    "print(\"Mean Similarity:\", results_df[\"similarity\"].mean())\n",
    "print(\"Mean Precision:\", results_df[\"precision\"].mean())\n",
    "print(\"Mean Recall:\", results_df[\"recall\"].mean())\n",
    "print(\"Mean F1:\", results_df[\"f1\"].mean())\n",
    "print()\n",
    "results_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4885cf7f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>true_masses</th>\n",
       "      <th>predicted_masses</th>\n",
       "      <th>similarity</th>\n",
       "      <th>f1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[1046, 563, 41062]</td>\n",
       "      <td>[1046, 324, 41301]</td>\n",
       "      <td>0.959979906229068988613529806</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[8847, 11850, 7373]</td>\n",
       "      <td>[-1, 20549, 7522]</td>\n",
       "      <td>0.3080448065173116089613034623</td>\n",
       "      <td>0.07051585423568386180785612872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[4282, 12860, 21504]</td>\n",
       "      <td>[4282, -4283, 38647]</td>\n",
       "      <td>0.3333333333333333333333333333</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[141623]</td>\n",
       "      <td>[-1, 0, 141624]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[447, 3749, 72862]</td>\n",
       "      <td>[447, -448, 77059]</td>\n",
       "      <td>0.3333333333333333333333333333</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[1537, 7617, 3517, 1938, 18775]</td>\n",
       "      <td>[4584, 9646, 19154]</td>\n",
       "      <td>0.1675513698630136986301369864</td>\n",
       "      <td>0.07503464660463274599089289249</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[1087, 8777, 2406]</td>\n",
       "      <td>[980, 8841, 2449]</td>\n",
       "      <td>0.9125874125874125874125874125</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[12166, 34099, 59278]</td>\n",
       "      <td>[12166, 34051, 59326]</td>\n",
       "      <td>0.996751488900920411478072550</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[3744, 11233, 11469, 7982, 40536]</td>\n",
       "      <td>[-1, 0, 3745, 29505, 41715]</td>\n",
       "      <td>0.3550600343053173241852487136</td>\n",
       "      <td>0.4494806421152030217186024551</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>[10222, 12327, 102269]</td>\n",
       "      <td>[10222, 12259, 102337]</td>\n",
       "      <td>0.9961085040631795810919079775</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>[1671, 10938, 1491, 1725, 144754]</td>\n",
       "      <td>[1671, 10878, 1218, 1942, 144870]</td>\n",
       "      <td>0.9886788256227758007117437722</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>[3133, 3043, 59419]</td>\n",
       "      <td>[982, 5101, 59512]</td>\n",
       "      <td>0.7556087998257460248311914615</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          true_masses                   predicted_masses  \\\n",
       "0                  [1046, 563, 41062]                 [1046, 324, 41301]   \n",
       "1                 [8847, 11850, 7373]                  [-1, 20549, 7522]   \n",
       "2                [4282, 12860, 21504]               [4282, -4283, 38647]   \n",
       "3                            [141623]                    [-1, 0, 141624]   \n",
       "4                  [447, 3749, 72862]                 [447, -448, 77059]   \n",
       "5     [1537, 7617, 3517, 1938, 18775]                [4584, 9646, 19154]   \n",
       "6                  [1087, 8777, 2406]                  [980, 8841, 2449]   \n",
       "7               [12166, 34099, 59278]              [12166, 34051, 59326]   \n",
       "8   [3744, 11233, 11469, 7982, 40536]        [-1, 0, 3745, 29505, 41715]   \n",
       "9              [10222, 12327, 102269]             [10222, 12259, 102337]   \n",
       "10  [1671, 10938, 1491, 1725, 144754]  [1671, 10878, 1218, 1942, 144870]   \n",
       "11                [3133, 3043, 59419]                 [982, 5101, 59512]   \n",
       "\n",
       "                        similarity                               f1  \n",
       "0    0.959979906229068988613529806                                1  \n",
       "1   0.3080448065173116089613034623  0.07051585423568386180785612872  \n",
       "2   0.3333333333333333333333333333                              0.5  \n",
       "3                                0                                0  \n",
       "4   0.3333333333333333333333333333                              0.5  \n",
       "5   0.1675513698630136986301369864  0.07503464660463274599089289249  \n",
       "6   0.9125874125874125874125874125                                1  \n",
       "7    0.996751488900920411478072550                                1  \n",
       "8   0.3550600343053173241852487136   0.4494806421152030217186024551  \n",
       "9   0.9961085040631795810919079775                                1  \n",
       "10  0.9886788256227758007117437722                                1  \n",
       "11  0.7556087998257460248311914615                                1  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df[[\"true_masses\", \"predicted_masses\", \"similarity\", \"f1\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09944c13",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}