AI-Chat-App-Hack-Vision

Tutorial on how to combine GPT-4 and Vision

This project uses the sample nature data set from Vision Studio.

Credits

Much of this content is possible because of the azure-search-openai-demo, please check it out!

Azure account requirements

IMPORTANT: In order to deploy and run this example, you'll need:

• Azure account. If you're new to Azure, get an Azure account for free and you'll get some free Azure credits to get started. See guide to deploying with the free trial.
• Azure account permissions:
  • Your Azure account must have Microsoft.Authorization/roleAssignments/write permissions, such as Role Based Access Control Administrator, User Access Administrator, or Owner. If you don't have subscription-level permissions, you must be granted RBAC for an existing resource group and deploy to that existing group.
  • Your Azure account also needs Microsoft.Resources/deployments/write permissions on the subscription level.

In order to run the notebooks in this example repository, you'll need:

• Azure subscription with access enabled for the Azure OpenAI service. You can request access with this form. If your access request to Azure OpenAI service doesn't match the acceptance criteria, you can use OpenAI public API instead.

Azure deployment

Cost estimation

Pricing varies per region and usage, so it isn't possible to predict exact costs for your usage. However, you can try the Azure pricing calculator for the resources below.

• Azure App Service: Basic Tier with 1 CPU core, 1.75 GB RAM. Pricing per hour. Pricing
• Azure OpenAI: Standard tier, ChatGPT models. Pricing per 1K tokens used, and at least 1K tokens are used per question. Pricing
• Azure AI Search: Standard tier, 1 replica. Pricing per hour. Pricing
• Azure Blob Storage: Standard tier with ZRS (Zone-redundant storage). Pricing per storage and read operations. Pricing
• Azure Monitor: Pay-as-you-go tier. Costs based on data ingested. Pricing

To reduce costs, you can switch to free SKUs for various services, but those SKUs have limitations. See this guide on deploying with minimal costs for more details.

⚠️ To avoid unnecessary costs, remember to take down your app if it's no longer in use, either by deleting the resource group in the Portal or running azd down.

Project setup

Local environment

First install the required tools:

• Azure Developer CLI
• Python 3.9, 3.10, or 3.11
  • Important: Python and the pip package manager must be in the path in Windows for the setup scripts to work.
  • Important: Ensure you can run python --version from console. On Ubuntu, you might need to run sudo apt install python-is-python3 to link python to python3.
• Node.js 14+
• Git
• Powershell 7+ (pwsh) - For Windows users only.
  • Important: Ensure you can run pwsh.exe from a PowerShell terminal. If this fails, you likely need to upgrade PowerShell.

Then bring down the project code:

1. Create a new folder and switch to it in the terminal
2. Run azd auth login
3. Run azd init -t https://github.com/mattgotteiner/AI-Chat-App-Hack-Vision
   • note that this command will initialize a git repository and you do not need to clone this repository

Deploying from scratch

Execute the following command, if you don't have any pre-existing Azure services and want to start from a fresh deployment.

1. Run azd up - This will provision Azure resources and deploy this sample to those resources, including building the search index based on the files found in the ./data folder.
   • Important: Beware that the resources created by this command will incur immediate costs, primarily from the AI Search resource. These resources may accrue costs even if you interrupt the command before it is fully executed. You can run azd down or delete the resources manually to avoid unnecessary spending.
2. After the application has been successfully deployed you will see a URL printed to the console. Click that URL to interact with the application in your browser.

NOTE: It may take 5-10 minutes for the application to be fully deployed. If you see a "Python Developer" welcome screen or an error page, then wait a bit and refresh the page.

Deploying with existing Azure resources

If you already have existing Azure resources, you can re-use those by setting azd environment values.

Existing resource group

1. Run azd env set AZURE_RESOURCE_GROUP {Name of existing resource group}
2. Run azd env set AZURE_LOCATION {Location of existing resource group}

Existing Azure AI Search resource

1. Run azd env set AZURE_SEARCH_SERVICE {Name of existing Azure AI Search service}
2. Run azd env set AZURE_SEARCH_SERVICE_RESOURCE_GROUP {Name of existing resource group with ACS service}
3. If that resource group is in a different location than the one you'll pick for the azd up step, then run azd env set AZURE_SEARCH_SERVICE_LOCATION {Location of existing service}
4. If the search service's SKU is not standard, then run azd env set AZURE_SEARCH_SERVICE_SKU {Name of SKU}. If you specify the free tier, it will use keys instead of managed identity for accessing the search service. Be advised that search SKUs cannot be changed. (See other possible SKU values)
5. If you have an existing index that is set up with all the expected fields, then run azd env set AZURE_SEARCH_INDEX {Name of existing index}. Otherwise, the azd up command will create a new index

Existing OpenAI resource

Only applies when you are running the notebooks

Azure OpenAI:

1. Run azd env set AZURE_OPENAI_SERVICE {Name of existing OpenAI service}
2. Run azd env set AZURE_OPENAI_RESOURCE_GROUP {Name of existing resource group that OpenAI service is provisioned to}
3. Run azd env set AZURE_OPENAI_RESOURCE_GROUP_LOCATION {Name of existing location}.

Openai.com OpenAI:

1. Run azd env set OPENAI_HOST openai
2. Run azd env set OPENAI_ORGANIZATION {Your OpenAI organization}
3. Run azd env set OPENAI_API_KEY {Your OpenAI API key}

You can retrieve your OpenAI key by checking your user page and your organization by navigating to your organization page. Learn more about creating an OpenAI free trial at this link. Do not check your key into source control.

Other existing Azure resources

You can also use existing Storage Accounts. See ./infra/main.parameters.json for list of environment variables to pass to azd env set to configure those existing resources.

Provision remaining resources

Now you can run azd up, following the steps in Deploying from scratch above. That will both provision resources and deploy the code.

Deploying again

If you've only changed the backend/frontend code in the app folder, then you don't need to re-provision the Azure resources. You can just run:

azd deploy

If you've changed the infrastructure files (infra folder or azure.yaml), then you'll need to re-provision the Azure resources. You can do that by running:

azd up

Deploying with minimal costs

This sample application is designed to be easily deployed using the Azure Developer CLI, which provisions the infrastructure according to the Bicep files in the infra folder. Those files describe each of the Azure resources needed, and configures their SKU (pricing tier) and other parameters. Many Azure services offer a free tier, but the infrastructure files in this project do not default to the free tier as there are often limitations in that tier.

However, if your goal is to minimize costs while prototyping your application, follow these steps below before deploying the application.

📺 Live stream: Deploying from a free account

1. Create a new azd environment for the free resource group:

   azd env new

   Enter a name that will be used for the resource group. This will create a new folder in the .azure folder, and set it as the active environment for any calls to azd going forward.

2. Use the free tier of App Service:

   azd env set AZURE_APP_SERVICE_SKU F1

   Limitation: You are only allowed a certain number of free App Service instances per region. If you have exceeded your limit in a region, you will get an error during the provisioning stage. If that happens, you can run azd down, then azd env new to create a new environment with a new region.

3. Use the free tier of Azure AI Search:

   azd env set AZURE_SEARCH_SERVICE_SKU free

   Limitations:

   1. You are only allowed one free search service across all regions. If you have one already, either delete that service or follow instructions to reuse your existing search service.
   2. The free tier does not support Managed Identity (keyless API access).

4. Use OpenAI.com instead of Azure OpenAI: This is only a necessary step for Azure free/student accounts, as they do not currently have access to Azure OpenAI.

   azd env set OPENAI_HOST openai
   azd env set OPENAI_ORGANIZATION {Your OpenAI organization}
   azd env set OPENAI_API_KEY {Your OpenAI API key}

   Both Azure OpenAI and openai.com OpenAI accounts will incur costs, based on tokens used, but the costs are fairly low for the amount of sample data (less than $10).

5. Once you've made the desired customizations, follow the steps in to run azd up. We recommend using "eastus" as the region, for availability reasons.

Running locally

You can only run locally after having successfully run the azd up command. If you haven't yet, follow the steps in Azure deployment above.

1. Run azd auth login
2. Change dir to app
3. Run ./start.ps1 to start the project locally.

Using the app

• In Azure: navigate to the Azure WebApp deployed by azd. The URL is printed out when azd completes (as "Endpoint"), or you can find it in the Azure portal.
• Running locally: navigate to 127.0.0.1:50505

Once in the web app:

• Issue a text-based search related to terms you'd find in pictures about nature (example: majestic, beach). Observe if the images returned are related to your query

Clean up

To clean up all the resources created by this sample:

1. Run azd down
2. When asked if you are sure you want to continue, enter y
3. When asked if you want to permanently delete the resources, enter y

The resource group and all the resources will be deleted.