allow plugins to be loaded from folders and don't crash on exceptions

[VPelt]

Background

In order to make plugin development (or just plugin installing & updating) easier it should be possible to load plugins without the code be in a zib format. Furthermore changes in the template can cause a specific plugin to stop loading. This shouldn't crash autogpt as a whole.

This way i can git checkout the plugins repository into the plugins folder and it 'just works'. See also the discussion #3654

Changes

• allow plugins to be loaded from regular folders in addition to be loaded from inside zib archives
• if a plugin fails to load don't crash autogpt but display a warning with the error and continue loading other plugins

Documentation

Test Plan

I tested loading plugins both in zip and non-zib format

PR Quality Checklist

• My pull request is atomic and focuses on a single change.
• I have thoroughly tested my changes with multiple different prompts.
• I have considered potential risks and mitigations for my changes.
• I have documented my changes clearly and comprehensively.
• I have not snuck in any "extra" small tweaks changes.
• I have run the following commands against my code to ensure it passes our linters:

  black .
  isort .
  mypy
  autoflake --remove-all-unused-imports --recursive --ignore-init-module-imports autogpt tests --in-place

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 2, 2023 11:39am

[Auto-GPT-Bot]

You changed AutoGPT's behaviour. The cassettes have been updated and will be merged to the submodule when this Pull Request gets merged.

[codecov]

Codecov Report

Patch coverage: 88.63% and project coverage change: +0.14 🎉

Comparison is base (63b79a8) 69.65% compared to head (c3a2011) 69.80%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4546      +/-   ##
==========================================
+ Coverage   69.65%   69.80%   +0.14%     
==========================================
  Files          72       72              
  Lines        3523     3553      +30     
  Branches      562      570       +8     
==========================================
+ Hits         2454     2480      +26     
- Misses        881      884       +3     
- Partials      188      189       +1     

Impacted Files	Coverage Δ
autogpt/plugins.py	77.38% <88.63%> (+2.01%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[erik-megarad]

Oh oops it looks like we both did a similar thing at the same time. Here's my solution: #4548

[Boostrix]

the two of you should probably team up - by the way, that's why it's a good idea to always communicate your plans upfront, and that is why it's possible to create "draft PRs"
please try to mutually review your PRs and then let's see how to integrate both PRs - ideally, the two of you would come up with a single PR, working together.

[erik-megarad]

There are lots of rad refactorings, log statements, and catches in here. I think they're great but can we change this PR to be just for the improvements to logging/exceptions, and then move folder-loading to my PR? #4548

My PR covers just folder-loading, but none of the improvements you've made. My PR adds requirements scanning which is necessary if we're going to enable folder-loading. LMK what you think.

[Boostrix]

Sounds good to me

[VPelt]

I'm happy to work together. I'll look at it in more detail on Monday

[Boostrix]

note that it's generally a good idea to mutually review your PRs and to ensure that the CI pipeline is happy (by adding/extending tests/coverage as needed)

[github-actions]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[Pwuts]

Closing this as resolved in #4548 :)