[Sample] Add a sample which converts a langchain app to flex flow with custom connection

.github/workflows/samples_flexflows_langchaineval_langchaineval.yml

@@ -0,0 +1,64 @@
+# This code is autogenerated.
+# Code is generated by running custom script: python3 readme.py
+# Any manual changes to this file may cause incorrect behavior.
+# Any manual changes will be overwritten if the code is regenerated.
+
+name: samples_flexflows_langchaineval_langchaineval
+on:
+  schedule:
+    - cron: "46 21 * * *" # Every day starting at 5:46 BJT
+  pull_request:
+    branches: [ main ]
+    paths: [ examples/flex-flows/langchain-eval/**, examples/*requirements.txt, .github/workflows/samples_flexflows_langchaineval_langchaineval.yml ]
+  workflow_dispatch:
+
+env:
+  IS_IN_CI_PIPELINE: "true"
+
+jobs:
+  samples_flexflows_langchaineval_langchaineval:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+      - name: Azure Login
+        uses: azure/login@v1
+        with:
+          creds: ${{ secrets.AZURE_CREDENTIALS }}
+      - name: Setup Python 3.9 environment
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
+      - name: Prepare requirements
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r ${{ github.workspace }}/examples/requirements.txt
+          pip install -r ${{ github.workspace }}/examples/dev_requirements.txt
+      - name: setup .env file
+        working-directory: examples/flex-flows/langchain-eval
+        run: |
+          AOAI_API_KEY=${{ secrets.AOAI_API_KEY_TEST }}
+          AOAI_API_ENDPOINT=${{ secrets.AOAI_API_ENDPOINT_TEST }}
+          AOAI_API_ENDPOINT=$(echo ${AOAI_API_ENDPOINT//\//\\/})
+          if [[ -e .env.example ]]; then
+            echo "env replacement"
+            sed -i -e "s/<your_AOAI_key>/$AOAI_API_KEY/g" -e "s/<your_AOAI_endpoint>/$AOAI_API_ENDPOINT/g" .env.example
+            mv .env.example .env
+          fi
+          if [[ -e ../.env.example ]]; then
+            echo "env replacement"
+            sed -i -e "s/<your_AOAI_key>/$AOAI_API_KEY/g" -e "s/<your_AOAI_endpoint>/$AOAI_API_ENDPOINT/g" ../.env.example
+            mv ../.env.example ../.env
+          fi
+      - name: Create Aoai Connection
+        run: pf connection create -f ${{ github.workspace }}/examples/connections/azure_openai.yml --set api_key="${{ secrets.AOAI_API_KEY_TEST }}" api_base="${{ secrets.AOAI_API_ENDPOINT_TEST }}"
+      - name: Test Notebook
+        working-directory: examples/flex-flows/langchain-eval
+        run: |
+          papermill -k python langchain-eval.ipynb langchain-eval.output.ipynb
+      - name: Upload artifact
+        if: ${{ always() }}
+        uses: actions/upload-artifact@v3
+        with:
+          name: artifact
+          path: examples/flex-flows/langchain-eval

examples/README.md

@@ -146,6 +146,7 @@
 | [chat-with-class-based-flow-azure.ipynb](flex-flows/chat-basic/chat-with-class-based-flow-azure.ipynb) | [![samples_flexflows_chatbasic_chatwithclassbasedflowazure](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatbasic_chatwithclassbasedflowazure.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatbasic_chatwithclassbasedflowazure.yml) | A quickstart tutorial to run a class based flex flow and evaluate it in azure. |
 | [chat-with-class-based-flow.ipynb](flex-flows/chat-basic/chat-with-class-based-flow.ipynb) | [![samples_flexflows_chatbasic_chatwithclassbasedflow](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatbasic_chatwithclassbasedflow.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatbasic_chatwithclassbasedflow.yml) | A quickstart tutorial to run a class based flex flow and evaluate it. |
 | [chat-stream-with-flex-flow.ipynb](flex-flows/chat-stream/chat-stream-with-flex-flow.ipynb) | [![samples_flexflows_chatstream_chatstreamwithflexflow](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatstream_chatstreamwithflexflow.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatstream_chatstreamwithflexflow.yml) | A quickstart tutorial to run a class based flex flow in stream mode and evaluate it. |
+| [langchain-eval.ipynb](flex-flows/langchain-eval/langchain-eval.ipynb) | [![samples_flexflows_langchaineval_langchaineval](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_langchaineval_langchaineval.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_langchaineval_langchaineval.yml) |  |
 | [prompty-quickstart.ipynb](prompty/basic/prompty-quickstart.ipynb) | [![samples_prompty_basic_promptyquickstart](https://github.com/microsoft/promptflow/actions/workflows/samples_prompty_basic_promptyquickstart.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_prompty_basic_promptyquickstart.yml) | A quickstart tutorial to run a prompty and evaluate it. |
 | [chat-with-prompty.ipynb](prompty/chat-basic/chat-with-prompty.ipynb) | [![samples_prompty_chatbasic_chatwithprompty](https://github.com/microsoft/promptflow/actions/workflows/samples_prompty_chatbasic_chatwithprompty.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_prompty_chatbasic_chatwithprompty.yml) | A quickstart tutorial to run a chat prompty and evaluate it. |
 | [prompty-output-format.ipynb](prompty/format-output/prompty-output-format.ipynb) | [![samples_prompty_formatoutput_promptyoutputformat](https://github.com/microsoft/promptflow/actions/workflows/samples_prompty_formatoutput_promptyoutputformat.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_prompty_formatoutput_promptyoutputformat.yml) |  |

examples/flex-flows/langchain-eval/data.jsonl

@@ -0,0 +1,2 @@
+{"input": "What's 2+2?",  "prediction": "What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four."}
+{"input": "What's 2x2?",  "prediction": "What's 2x2? That's an elementary question. The answer you're looking for is that two and two is four."}

examples/flex-flows/langchain-eval/flow.flex.yaml

@@ -0,0 +1,5 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
+entry: flow:LangChainEvaluator
+environment:
+  # image: mcr.microsoft.com/azureml/promptflow/promptflow-python
+  python_requirements_txt: requirements.txt

examples/flex-flows/langchain-eval/flow.py

@@ -0,0 +1,51 @@
+from dataclasses import dataclass
+from pathlib import Path
+
+from promptflow.tracing import trace
+from promptflow.connections import CustomConnection
+
+
+from langchain.evaluation import load_evaluator
+from langchain.chat_models import AzureChatOpenAI, ChatAnthropic
+
+
+
+@dataclass
+class Result:
+    reasoning: str
+    value: str
+    score: float
+
+
+class LangChainEvaluator:
+    def __init__(self, custom_connection: CustomConnection):
+        self.custom_connection = custom_connection
+
+        # create llm according to the secrets in custom connection
+        if "anthropic_api_key" in self.custom_connection.secrets:
+            self.llm = ChatAnthropic(temperature=0, anthropic_api_key=self.custom_connection.secrets["anthropic_api_key"])
+        elif "openai_api_key" in self.custom_connection.secrets:
+            self.llm = AzureChatOpenAI(
+                deployment_name="gpt-35-turbo",
+                openai_api_key=self.custom_connection.secrets["openai_api_key"],
+                azure_endpoint=self.custom_connection.secrets["azure_endpoint"],
+                openai_api_type="azure",
+                openai_api_version="2023-07-01-preview",
+                temperature=0,
+            )
+        else:
+            raise ValueError("No valid API key found in the connection.")
+
+        self.evaluator = load_evaluator("criteria", llm=self.llm, criteria="conciseness")
+
+    @trace
+    def __call__(
+        self, input: str, prediction: str, 
+    ) -> Result:
+        """Evaluate with langchain evaluator."""
+
+        eval_result = self.evaluator.evaluate_strings(
+            prediction=prediction, input=input
+        )
+        return Result(**eval_result)
+

examples/flex-flows/langchain-eval/langchain-eval.ipynb

@@ -0,0 +1,177 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Evaluate with langchain's evaluator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
+    "\n",
+    "- Convert langchain applications to flex flow.\n",
+    "- Use CustomConnection to store secrets.\n",
+    "\n",
+    "## 0. Install dependent packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "%pip install -r ./requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Initialize a pf client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.client import PFClient\n",
+    "\n",
+    "pf = PFClient()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create a custom connection to protect your LLM's API key\n",
+    "\n",
+    "You can protect your LLM's API key in custom connection's secrets."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from dotenv import load_dotenv\n",
+    "\n",
+    "from promptflow.entities import CustomConnection\n",
+    "\n",
+    "if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
+    "    # load environment variables from .env file\n",
+    "    load_dotenv()\n",
+    "\n",
+    "# put LLM's API key in secrets\n",
+    "connection = CustomConnection(\n",
+    "    name=\"my_llm_connection\",\n",
+    "    secrets={\n",
+    "        # store API key\n",
+    "        # \"anthropic_api_key\": \"<your-api-key>\",\n",
+    "        \"openai_api_key\": os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
+    "        \"azure_endpoint\": os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
+    "    },\n",
+    ")\n",
+    "# Create the connection, note that all secret values will be scrubbed in the returned result\n",
+    "result = pf.connections.create_or_update(connection)\n",
+    "print(result)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Test the evaluator with trace"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from flow import LangChainEvaluator\n",
+    "\n",
+    "\n",
+    "evaluator = LangChainEvaluator(custom_connection=connection)\n",
+    "result = evaluator(\n",
+    "    prediction=\"What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four.\",\n",
+    "    input=\"What's 2+2?\",\n",
+    ")\n",
+    "print(result)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Batch run the function as flow with multi-line data\n",
+    "\n",
+    "Create a [flow.flex.yaml](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/langchain-eval/flow.flex.yaml) file to define a flow which entry pointing to the python function we defined.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = \"./data.jsonl\"  # path to the data file\n",
+    "# create run with the flow function and data\n",
+    "base_run = pf.run(\n",
+    "    flow=\"./flow.flex.yaml\",\n",
+    "    # reference custom connection by name\n",
+    "    init={\n",
+    "        \"custom_connection\": \"my_llm_connection\",\n",
+    "    },\n",
+    "    data=data,\n",
+    "    column_mapping={\n",
+    "        \"prediction\": \"${data.prediction}\",\n",
+    "        \"input\": \"${data.input}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(base_run)\n",
+    "details.head(10)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "flow_dev",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/flex-flows/langchain-eval/requirements.txt

@@ -0,0 +1,3 @@
+promptflow
+langchain>=0.1.5
+python-dotenv