[ENG-5430] ENABLE_GV feature flipping for `get_addon` #10610

Johnetordoff · 2024-05-09T14:27:46Z

Purpose

Allows us to feature flip between our current system and the new GravyValet addon service. Should change files related routes to use new set up.

Changes

new get_addon functions for AbstractNode and OsfUser waffled to return mock config
creates mock config GravyValetAddonAppConfig to serve as addon proxy object
add tests and modifes some existing for new mocks
now responds to 401 and 403 from WB https://github.com/CenterForOpenScience/osf.io/pull/10610/files#diff-7b7eef8ad639532cbf555abca69f06d4dc59096b5b60402f3b6ad7b6057fed95R67

QA Notes

Please make verification statements inspired by your code and what your code touches.

Verify
Verify

What are the areas of risk?

Any concerns/considerations/questions that development raised?

Documentation

Side Effects

Ticket

https://openscience.atlassian.net/browse/ENG-5430

jwalz

Next steps:

AddonModelMixin._settings_model won't currently work. We actually need to make the request beforehand using the name as the ID for the AuthorizedStorageAccount (user.get_addon) or the ConfiguredStorageAddon (node.get_addon). If those serializers don't currently return the name of the connected service, they should. That can then be used as a lookup to get the correct SettingsModel.
MockUserSettings and MockNodeSettings should populate based on data from the AuthorizedStorageAccount and ConfiguredStorageAddon (UserReference and ResourceReference just exist to link things and contain no configuration data).
(future ticket) get_addons should request all of the authorized storage accounts for a user or connected_storage_addons for a node instead of iterating through a list of provider names (and then pack a list whatever the appropriate serializer is based on the returned values).

jwalz · 2024-05-09T17:42:58Z

website/settings/defaults.py

+GV_RESOURCE_DOMAIN = 'http://192.168.168.167:8004/v1/resource-references/?filter[resource_uri]={owner_uri}'
+GV_USER_DOMAIN = 'http://192.168.168.167:8004/v1/user-references/?filter[user_uri]={owner_uri}'


Suggested change

GV_RESOURCE_DOMAIN = 'http://192.168.168.167:8004/v1/resource-references/?filter[resource_uri]={owner_uri}'

GV_USER_DOMAIN = 'http://192.168.168.167:8004/v1/user-references/?filter[user_uri]={owner_uri}'

GV_API_ROOT = 'http://192.168.168.167:8004/v1

GV_RESOURCE_ENDPOINT = GV_DOMAIN + 'resource-references/?filter[resource_uri]={resource_uri}'

GV_USER_ENDPOINT = GV_DOMAIN + 'user-references/?filter[user_uri]={owner_uri}'

# These two are for `get_addon` vs `get_addons`

GV_USER_ADDON_ENDPOINT = GV_DOMAIN + 'v1/authorized-storage-accounts/{account_id}'

GV_NODE_ADDON_ENDPOINT = GV_DOMAIN + 'v1/configured-storage-addons/{addon_id}'

Could optionally append &include=authorized_storage_accounts|configured_storage_addons to the user/resource endpoints

addons/box/apps.py

into gv-files-provider-waffle * 'develop' of https://github.com/CenterForOpenScience/osf.io: [ENG-5140] #2 Update get_auth for GV and readability (CenterForOpenScience#10613)

api_tests/providers/test_files_with_gv.py

jwalz · 2024-05-13T15:46:55Z

api_tests/providers/test_files_with_gv.py

+        attributes = res.json['data']['attributes']
+        assert attributes['provider'] == str(provider_gv_id)
+        assert attributes['name'] == str(provider_gv_id)
+        assert res.json['data']['id'] == f'{file.target._id}:{str(provider_gv_id)}'


I think we want this to just be the provider_gv_id. @brianjgeiger?

Yes, for everything but (I believe) osfstorage, if there's a gv provider id, then that should be the whole id. Yay for unique provider ids.

jwalz · 2024-05-13T15:48:07Z

api_tests/providers/test_files_with_gv.py

+        assert attributes['provider'] == str(provider_gv_id)
+        assert attributes['name'] == str(provider_gv_id)


@brianjgeiger would we like either of these (probably name?) to be the display_name of the addon?

I think I'm using primarily gv information for this on the front end, but yeaaaahhh, let's try name to be the display_name.

api_tests/providers/test_files_with_gv.py

jwalz · 2024-05-13T23:27:14Z

osf/models/node.py

@@ -2430,6 +2433,29 @@ def _remove_from_associated_collections(self, auth=None, force=False):
                    force=True
                )

+    def get_addon(self, name, is_deleted=False):


My instinct is that, instead of having the subclasses completely override get_addon, the superclass should start with a check for the GravyValet flag and then invoke a _get_addon_from_gravyvalet or similar method that is defined on the subclass. That gets rid of the need to call super() and better delegates responsibility to the subclass

jwalz · 2024-05-13T23:29:12Z

osf/models/node.py

+    def get_addon(self, name, is_deleted=False):
+        request, user_id = get_request_and_user_id()
+
+        if hasattr(request, 'user') and not isinstance(request.user, AnonymousUser) and waffle.flag_is_active(request, features.ENABLE_GV) and name not in ['osfstorage', 'wiki']:


I don't think we need to guard this against the AnonymousUser. Unauthenticated users should still be able to see the addons for a public node.

this is still true (this is also a delta between node and user behavior)

osf/models/node.py

into gv-files-provider-waffle * 'develop' of https://github.com/CenterForOpenScience/osf.io: Fix get auth number two (CenterForOpenScience#10614)

…config

api_tests/utils.py

website/settings/defaults.py

jwalz · 2024-05-22T13:31:40Z

website/settings/defaults.py

+GV_USER_ADDON_ENDPOINT = 'http://192.168.168.167:8004/v1/authorized-storage-accounts/{account_id}'
+GV_NODE_ADDON_ENDPOINT = 'http://192.168.168.167:8004/v1/configured-storage-addons/{config_id}/base_account/'
+GV_EXTERNAL_STORAGE_ENDPOINT = 'http://192.168.168.167:8004/v1/external-storage-services/{service_id}'


Suggested change

GV_USER_ADDON_ENDPOINT = 'http://192.168.168.167:8004/v1/authorized-storage-accounts/{account_id}'

GV_NODE_ADDON_ENDPOINT = 'http://192.168.168.167:8004/v1/configured-storage-addons/{config_id}/base_account/'

GV_EXTERNAL_STORAGE_ENDPOINT = 'http://192.168.168.167:8004/v1/external-storage-services/{service_id}'

GV_USER_ADDON_ENDPOINT = GV_API_ROOT + 'authorized-storage-accounts/{account_id}'

GV_NODE_ADDON_ENDPOINT = GV_API_ROOT + 'configured-storage-addons/{config_id}/base_account/'

GV_EXTERNAL_STORAGE_ENDPOINT = GV_API_ROOT + 'external-storage-services/{service_id}'

jwalz · 2024-05-22T13:32:19Z

website/settings/defaults.py

+GV_RESOURCE_DOMAIN = 'http://192.168.168.167:8004/v1/resource-references/?filter[resource_uri]={owner_uri}'
+GV_USER_DOMAIN = 'http://192.168.168.167:8004/v1/user-references/?filter[user_uri]={owner_uri}'


Suggested change

GV_RESOURCE_DOMAIN = 'http://192.168.168.167:8004/v1/resource-references/?filter[resource_uri]={owner_uri}'

GV_USER_DOMAIN = 'http://192.168.168.167:8004/v1/user-references/?filter[user_uri]={owner_uri}'

api/nodes/utils.py

jwalz · 2024-05-22T13:36:10Z

osf/models/node.py

+    def get_addon(self, name, is_deleted=False):
+        request, user_id = get_request_and_user_id()
+
+        if hasattr(request, 'user') and not isinstance(request.user, AnonymousUser) and waffle.flag_is_active(request, features.ENABLE_GV) and name not in ['osfstorage', 'wiki']:


this is still true (this is also a delta between node and user behavior)

addons/base/utils.py

jwalz · 2024-05-22T14:17:51Z

addons/base/utils.py

+    class MockNodeSetting:
+        def __init__(self, resource, auth, legacy_config, gv_data):
+            self.gv_data = gv_data
+
+        @property
+        def configured(self):
+            return True
+
+        @property
+        def config_id(self):
+            return self.gv_data.config_id
+
+    class MockUserSetting:


What's the benefit of making these internal classes? They don't seem to be using the enclosing scope in any way. Also, the majority of the init args are going unused.

jwalz · 2024-05-22T19:30:23Z

addons/base/utils.py

+
+class GravyValetAddonAppConfig:
+    class MockNodeSetting:
+        def __init__(self, resource, auth, legacy_config, gv_data):


Something like:

def __init__(self, configured_storage_addon_id): self.gv_data = requests.get(configured-storage-addons/configured-storage-addon-id/?include=external_storage_service) self.legacy_app_config = _get_legacy_config_for_gv_service(gv_data.name) self.magically_mirror_legacy_node_settings(legacy_app_config.node_settings_module) @property def config: return self.legacy_app_config

"magically_mirror_legacy_node_settings" could come from a per-provider translator class/module that is surfaced with a standardized name on the legacy_config

into gv-files-provider-waffle * 'develop' of https://github.com/CenterForOpenScience/osf.io: add DraftRegistration file permission test (CenterForOpenScience#10626) add DraftRegistration file permission test (CenterForOpenScience#10622)

into gv-files-provider-waffle * 'develop' of https://github.com/CenterForOpenScience/osf.io: Configurable GV Mock + HMAC Auth (CenterForOpenScience#10623)

Johnetordoff force-pushed the gv-files-provider-waffle branch 3 times, most recently from 0b9d7f6 to 934ea23 Compare May 9, 2024 15:37

jwalz reviewed May 9, 2024

View reviewed changes

addons/box/apps.py Outdated Show resolved Hide resolved

Johnetordoff force-pushed the gv-files-provider-waffle branch 8 times, most recently from 293e89a to 221248b Compare May 13, 2024 13:26

waffle get_addon code for gravyvalet

d194c91

Johnetordoff force-pushed the gv-files-provider-waffle branch from 221248b to d194c91 Compare May 13, 2024 14:51

Merge branch 'develop' of https://github.com/CenterForOpenScience/osf.io

671f188

into gv-files-provider-waffle * 'develop' of https://github.com/CenterForOpenScience/osf.io: [ENG-5140] #2 Update get_auth for GV and readability (CenterForOpenScience#10613)

Johnetordoff force-pushed the gv-files-provider-waffle branch 2 times, most recently from 375673b to 074878f Compare May 13, 2024 18:07

improve gravyvalet integration

c60bd66

Johnetordoff force-pushed the gv-files-provider-waffle branch from 074878f to c60bd66 Compare May 13, 2024 18:15

jwalz reviewed May 13, 2024

View reviewed changes

John Tordoff added 2 commits May 16, 2024 10:53

debug commit

230be4d

mock WB and add mocking resolution for generic FileNode

4425024

Johnetordoff force-pushed the gv-files-provider-waffle branch 5 times, most recently from c7606c9 to dc4cda4 Compare May 16, 2024 19:35

Merge branch 'develop' of https://github.com/CenterForOpenScience/osf.io

bc3fd85

into gv-files-provider-waffle * 'develop' of https://github.com/CenterForOpenScience/osf.io: Fix get auth number two (CenterForOpenScience#10614)

Johnetordoff force-pushed the gv-files-provider-waffle branch from dc4cda4 to bc3fd85 Compare May 16, 2024 19:50

Johnetordoff force-pushed the gv-files-provider-waffle branch from ff1e20b to 65ebf16 Compare May 21, 2024 14:27

improve tests for gravyvalet feature flagging

e888e87

Johnetordoff force-pushed the gv-files-provider-waffle branch from 65ebf16 to e888e87 Compare May 21, 2024 14:30

mock user_setting and node_settings with the addon gracy valet addon …

9d1bbe5

…config

Johnetordoff force-pushed the gv-files-provider-waffle branch from a2a6176 to 9d1bbe5 Compare May 21, 2024 16:08

pass basic auth and cookie for v1 route tests

a23cf78

jwalz reviewed May 22, 2024

View reviewed changes

make CR improvements

41c0e85

Johnetordoff force-pushed the gv-files-provider-waffle branch 2 times, most recently from 7a55c02 to 9f0d6e4 Compare May 22, 2024 21:05

clean-up waffling code logic

cdaafec

Johnetordoff force-pushed the gv-files-provider-waffle branch 7 times, most recently from 647f35c to 06cf8fe Compare May 23, 2024 15:43

clean-up node_setting user_setting mocking for GraveyValet

9bd6ad5

Johnetordoff force-pushed the gv-files-provider-waffle branch from 06cf8fe to 9bd6ad5 Compare May 23, 2024 16:09

John Tordoff added 4 commits May 24, 2024 11:37

Merge branch 'develop' of https://github.com/CenterForOpenScience/osf.io

a73a069

into gv-files-provider-waffle * 'develop' of https://github.com/CenterForOpenScience/osf.io: Configurable GV Mock + HMAC Auth (CenterForOpenScience#10623)

add provider id caching

d358809

update tests to work with GV mock and refactor

3b164ff

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ENG-5430] ENABLE_GV feature flipping for `get_addon` #10610

[ENG-5430] ENABLE_GV feature flipping for `get_addon` #10610

Johnetordoff commented May 9, 2024 •

edited

jwalz left a comment

jwalz May 9, 2024

jwalz May 9, 2024

jwalz May 13, 2024

brianjgeiger May 14, 2024

jwalz May 13, 2024

brianjgeiger May 14, 2024

jwalz May 13, 2024

jwalz May 13, 2024

jwalz May 22, 2024

jwalz May 22, 2024

jwalz May 22, 2024

jwalz May 22, 2024

jwalz May 22, 2024

jwalz May 22, 2024

		GV_RESOURCE_DOMAIN = 'http://192.168.168.167:8004/v1/resource-references/?filter[resource_uri]={owner_uri}'
		GV_USER_DOMAIN = 'http://192.168.168.167:8004/v1/user-references/?filter[user_uri]={owner_uri}'

-GV_RESOURCE_DOMAIN = 'http://192.168.168.167:8004/v1/resource-references/?filter[resource_uri]={owner_uri}'
-GV_USER_DOMAIN = 'http://192.168.168.167:8004/v1/user-references/?filter[user_uri]={owner_uri}'
+GV_API_ROOT =  'http://192.168.168.167:8004/v1
+GV_RESOURCE_ENDPOINT = GV_DOMAIN + 'resource-references/?filter[resource_uri]={resource_uri}'
+GV_USER_ENDPOINT = GV_DOMAIN + 'user-references/?filter[user_uri]={owner_uri}'
+# These two are for `get_addon` vs `get_addons`
+GV_USER_ADDON_ENDPOINT = GV_DOMAIN + 'v1/authorized-storage-accounts/{account_id}'
+GV_NODE_ADDON_ENDPOINT = GV_DOMAIN + 'v1/configured-storage-addons/{addon_id}'

		assert attributes['provider'] == str(provider_gv_id)
		assert attributes['name'] == str(provider_gv_id)

[ENG-5430] ENABLE_GV feature flipping for get_addon #10610

Are you sure you want to change the base?

[ENG-5430] ENABLE_GV feature flipping for get_addon #10610

Conversation

Johnetordoff commented May 9, 2024 • edited

Purpose

Changes

QA Notes

Documentation

Side Effects

Ticket

jwalz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[ENG-5430] ENABLE_GV feature flipping for `get_addon` #10610

[ENG-5430] ENABLE_GV feature flipping for `get_addon` #10610

Johnetordoff commented May 9, 2024 •

edited