Handle iteration/next_token automatically

[matthewkmayer]

#1031 showed a great idea for implementing an iterator for calls that can return multiple pages of results. Right now, Rusoto requires users to handle paging themselves. The S3 integration tests have examples of having to implement this as a user of Rusoto.

It'd be 💯 to have Rusoto take care of paging behind the scenes and expose an iterator interface to results that can have multiple pages.

[softprops]

I'd be willing to give this a shot. I've had experience doing this is dynomite, a dynamodb crate using a futures::Stream based model. I'd be happy to port that into rusoto code gen.

[andrewbanchich]

Hey @matthewkmayer , I actually just finished implementing this for S3 results for a personal project and came here to see if it would be okay if I made a pull request for it.

[matthewkmayer]

That'd be awesome! 👍

[andrewbanchich]

It's just for S3 results though, but it looks like you'd probably want something programmatically generated for the entire library?

[matthewkmayer]

Correct: we'll want it across all services.

Seeing an example of it working can help us figure out a solution. 😄

[softprops]

Heads up I've already got a branch going with a stream::unfold approach leveraging botocores paginator descriptors. I could push up a prelim pr with a wip of that would be helpful.

[srijs]

@softprops happy to look over that and provide feedback if you want to push that!

[matthewkmayer]

@andrewbanchich I'm booked up the next few weeks, but we could work something out. If possible, a code snippet of how you're using the iterators would be 💯 and an example implementation. I wouldn't spend too much time on the snippets since @softprops has work in progress that'll cover all services.

[andrewbanchich]

@matthewkmayer Here's a snippet of what I got working for my project. I'd love any feedback if there is a better way of doing anything.

I also added a bunch of helper impls for the Bucket type for get/put/list actions.

[softprops]

Approach I'm taking is lifting https://github.com/softprops/dynomite/blob/master/dynomite/src/ext.rs#L53 into something that's paginator definition generated

[0xjjoyy]

Is there a target milestone for pagination?

[aj-bagwell]

I have rebased @softprops work onto master and finished off the JMESPath stuff for some of the stranger pagination definitions.

[Kintaro]

Are there any updates on the progress?

[aj-bagwell]

I have updated my pull request with the latest master, but until #1651 is sorted out and there is a new maintainer who can review a change of this size I don't think this will get merged.