APIDataSet ease of use

[ianwhale]

Description

Howdy team!

I was working with the APIDataSet recently and had two issues out of the box.

1. Specifying the auth keyword argument in yaml

The requests library expects the auth parameter of a request to be either a HTTPBasicAuth or a tuple (lists are not allowed, see here in requests). At the moment, neither are possible to specify in my catalog.yml.

From what I hear, you're already working on this (#1011). So maybe this point is moot.

2. The auth keyword argument and credentials.yml

I would like to specify my (username, password) tuple inside credentials.yml. However, the APIDataSet's auth keyword wouldn't get filled in by the config loader.

To get this working, you'd have to extend APIDataSet to have a credentials keyword that is filled in for auth in an upcall.

It would be great to either have this by default, or even have the loader fill auth keywords in addition to credentials. Although that might have unintended consequences.

Context

Hopefully this would unify the experience a bit. Right now, the credentials keyword in a dataset and credentials.yml are the main points of access to secrets. Which is probably good.

Possible Implementation

I whipped up my own APIDataSet to solve both the problems above.

Possible Alternatives

To get this working with no changes to APIDataSet, we'd have to implement the changes in #1011 so we can specify tuples in credentials.yml and have the config loader fill in auth as well.

[antonymilne]

Hi @ianwhale, thank you for your very thoughtful write up of this. The points you raise are very fair. I'd suggest just implementing the changes you propose to APIDataSet right away as it will give an instant improvement, seems pretty uncontroversial and I like the code exactly how you wrote it 👍

For the possible alternative: #1011 is still very much up for discussion and would be a larger change. And auth is only relevant to the APIDataSet so I'd be inclined to leave the config loader alone.