Add service to openai integration to use openai vision

[tlanfer]

Proposed change

OpenAI provides a lot of interesting features. One feature thats quite interesting for home automation scenarios is the Vision API, which lets clients provide images and ask OpenAI questions about it.
I extended the existing openai_conversation integrations which so far only provides a service for image generation. I added an additional service, which takes a camera entity as an input and uses OpenAI to analyze it.
One usecase for this would using OpenAI to find out if the person at your front door is a delivery person, some children, or a group of dogs.
This is obviously very much inspired by AmbleGPT, which does basically the same thing, except better (using several frames and providing more context). The difference being that using my provided change lets you integrate it into arbitrary automations easily.

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Deprecation (breaking change to happen in the future)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• OpenAI vision api: https://platform.openai.com/docs/guides/vision
• AmbleGPT: https://github.com/mhaowork/amblegpt

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• I have followed the perfect PR recommendations
• The code has been formatted using Ruff (ruff format homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io: Add service to openai integration to use openai vision home-assistant.io#32683

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
• Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[home-assistant]

Hi @tlanfer

It seems you haven't yet signed a CLA. Please do so here.

Once you do that we will be able to review and accept this pull request.

Thanks!

[home-assistant]

Hey there @balloob, mind taking a look at this pull request as it has been labeled with an integration (openai_conversation) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of openai_conversation can trigger bot actions by commenting:

• @home-assistant close Closes the pull request.
• @home-assistant rename Awesome new title Renames the pull request.
• @home-assistant reopen Reopen the pull request.
• @home-assistant unassign openai_conversation Removes the current integration label and assignees on the pull request, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

[allenporter]

It seems useful to follow the conventions in https://www.home-assistant.io/integrations/google_generative_ai_conversation/#service-google_generative_ai_conversationgenerate_content in the homes that we'll be able to converge ollama and google at some point.

[balloob]

We're planning a generative_ai integration that will standardize how we describe images. It's currently waiting for config subentries to be finalized (development is in progress)

[tlanfer]

We're planning a generative_ai integration that will standardize how we describe images. It's currently waiting for config subentries to be finalized (development is in progress)

Would you say its worth waiting with this PR and possibly migrating it to whatever the result of that might be? Or should we rather close it for now, since it'll be vastly different bits of code anyway? In which case i'll maybe just dump my changes into a temporary custom component.

[balloob]

I would stick to a custom component. I don't think we'll make this release but we've moved mountains before.