Consider whether to prioritize and recommend OpenAPI and SDK integration for existing and future tap maintainers

[aaronsteers]

Background

• Original SDK issue: Leverage schema and other API details from OpenAPI spec file #308 (Not prioritized internally, but open to community PRs.)

Following from some recent conversations on this topic, I wanted to solicit contributors' and maintainers' opinions about relative cost/benefit of using the OpenAPI spec for new taps and existing taps.

For a sample in the "new taps" category, there's the tap-shortcut implementation from @edgarrmondragon which does use OpenAPI (more detail below). For a sample in the "existing taps" category, I think tap-github offers a good thought experiment - since they do publish an OpenAPI spec document, and since we already have a robust implementation built without a dependency on it.

I did some digging and found that GitHub does publish OpenAPI descriptions in this repo.

Taking a simple example, here is the native definition of MilestonesStream, followed by the definition of the milestones in OpenAPI:

Show code snippets...

Tap Stream Definition:

From https://github.com/MeltanoLabs/tap-github/blob/5a3ec399feec9253865e467b74b65a4991af7279/tap_github/repository_streams.py#L586-L617:

class MilestonesStream(GitHubRestStream):
    name = "milestones"
    path = "/repos/{org}/{repo}/milestones"
    primary_keys = ["id"]
    replication_key = "updated_at"
    parent_stream_type = RepositoryStream
    state_partitioning_keys = ["repo", "org"]
    ignore_parent_replication_key = True

    schema = th.PropertiesList(
        # Parent Keys
        th.Property("repo", th.StringType),
        th.Property("org", th.StringType),
        th.Property("repo_id", th.IntegerType),
        # Rest
        th.Property("url", th.StringType),
        th.Property("html_url", th.StringType),
        th.Property("labels_url", th.StringType),
        th.Property("id", th.IntegerType),
        th.Property("node_id", th.StringType),
        th.Property("number", th.IntegerType),
        th.Property("state", th.StringType),
        th.Property("title", th.StringType),
        th.Property("description", th.StringType),
        th.Property("creator", user_object),
        th.Property("open_issues", th.IntegerType),
        th.Property("closed_issues", th.IntegerType),
        th.Property("created_at", th.DateTimeType),
        th.Property("updated_at", th.DateTimeType),
        th.Property("closed_at", th.DateTimeType),
        th.Property("due_on", th.StringType),
    ).to_dict()

Open API Definition

From https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/api.github.com/api.github.com.yaml:

# ...
    milestone:
      title: Milestone
      description: A collection of related issues and pull requests.
      type: object
      properties:
        url:
          type: string
          format: uri
          example: https://api.github.com/repos/octocat/Hello-World/milestones/1
        html_url:
          type: string
          format: uri
          example: https://github.com/octocat/Hello-World/milestones/v1.0
        labels_url:
          type: string
          format: uri
          example: https://api.github.com/repos/octocat/Hello-World/milestones/1/labels
        id:
          type: integer
          example: 1002604
        node_id:
          type: string
          example: MDk6TWlsZXN0b25lMTAwMjYwNA==
        number:
          description: The number of the milestone.
          type: integer
          example: 42
        state:
          description: The state of the milestone.
          example: open
          type: string
          enum:
          - open
          - closed
          default: open
        title:
          description: The title of the milestone.
          example: v1.0
          type: string
        description:
          type: string
          example: Tracking milestone for version 1.0
          nullable: true
        creator:
          "$ref": "#/components/schemas/nullable-simple-user"
        open_issues:
          type: integer
          example: 4
        closed_issues:
          type: integer
          example: 8
        created_at:
          type: string
          format: date-time
          example: '2011-04-10T20:09:31Z'
        updated_at:
          type: string
          format: date-time
          example: '2014-03-03T18:58:10Z'
        closed_at:
          type: string
          format: date-time
          example: '2013-02-12T13:22:01Z'
          nullable: true
        due_on:
          type: string
          format: date-time
          example: '2012-10-09T23:39:01Z'
          nullable: true
      required:
      - closed_issues
      - creator
      - description
      - due_on
      - closed_at
      - id
      - node_id
      - labels_url
      - html_url
      - number
      - open_issues
      - state
      - title
      - url
      - created_at
      - updated_at

OpenAPI limitations

What this doesn't solve for and would still need to be separately implemented:

• Incremental key definition
• Primary key definition
• Pagination logic
• Error handling
• Rate limit handling
• Record reshaping from post_process()
• Appending properties, such as properties included from the parent stream

Request for feedback

So, the two questions I'd love feedback on:

1. Is there any benefit to migrating existing taps like tap-github to reference the OpenAPI spec and then perform all needed alterations upon that schema as a starting point?
2. For new taps (starting from scratch), would we recommend building upon the OpenAPI spec as the 'preferred' and 'recommended' approach when an OpenAPI spec exists?

If yes to either of the above, does this change the mainstream dev process? OpenAPI specs are not always easy to browse and we can't assume the developer has prior Postman experience. Are there tooling or dev patterns we'd recommend?

I think there's a possible future where we highly recommend using OpenAPI as the starting point for new taps, but there's also a possible future where we do support OpenAPI descriptors for developers who want to use them, without strongly recommending that path as the preferred path.

Example Singer implementation using OpenAPI

@edgarrmondragon has built tap-shortcut which leverages their OpenAPI spec. This can be helpful for thought experiments.

• Discovery implementation: https://github.com/edgarrmondragon/tap-shortcut/blob/main/tap_shortcut/tap.py#L99-L126
  • Could likely be made generic across a list of known OpenAPIStream classes (subclassed from RESTStream)
• A couple example stream definitions: https://github.com/edgarrmondragon/tap-shortcut/blob/main/tap_shortcut/streams.py#L16-L43
  • Note that certain things like parent-child relationships and pagination still need to be defined by the tap developer, but at least the base schema could be inferred/inherited from the OpenAPI description.
• Using only a free online account (no local installation and no paid signup), I was able to import and publish the shortcut.com OpenAPI definition to a Postman workspace here, which anyone can utilize or fork.