Recursively get data by using the 'nextCursor' attribute in Nerdgraph API

[anuppothen]

Description

Flex integration allows us to get data from an API source, and allows the use of pagination, if the page is within the URL. However, in the scenario where we use flex to get data from New Relic Nerdgraph API, pagination is done via the nextCursor and cursor attributes within the JSON body. The current flex implementation does not allow us to recursively capture data in this scenario.

Acceptance Criteria

We need the capability to use Flex integration to recursively pull data from the Nerdgraph API which uses pagination using the cursor and nextCursor attributes in the JSON body

Describe Alternatives

An alternative approach would be to write a script to pull the data using Nerdgraph API and push the data into New Relic using the insights API, which would then need to be scheduled on a regular basis. Care should also be taken to implement appropriate error handling scenarios as well. This lacks the advantage the flex provides of automatically formatting and pushing the data to New Relic

Dependencies

Do any other teams or parts of the New Relic product need to be considered?
Some common areas: UI, collector, documentation

Additional context

This use case has been request by multiple customers when they need to create custom events and custom dashboards using NRQL query when the data is data is typically only available in Nerdgraph and not NRDB

Estimates

Please provide initial t-shirt size. S = 1-3 days, M = 3-5 days (1 week), L = 1-2 weeks (1 sprint)

For Maintainers Only or Hero Triaging this bug

Suggested Priority (P1,P2,P3,P4,P5):
Suggested T-Shirt size (S, M, L, XL, Unknown):

[workato-integration]

https://new-relic.atlassian.net/browse/NR-175354

[josemore]

Hi @anuppothen , thanks for submitting the issue and the context.
Can you elaborate a bit more on how are you planning to use Nerdgraph APIs with Flex? What would be a concrete example of a Flex config?