Recursion features in graphql compiler

[pazurururu]

Hello,

After trying out the library on some reasonably complex queries, I was immediately impressed that the @recurse directive generates a full recursive CTE. Because it seems like the complex generation work is already completed, are there any additional features planned around this area? For context, I'm using this library with sqlalchemy & mysql db.

1. Is there any reason that infinite depth recursion isn't supported? With the generated recursive CTE, the depth counter is really an additional enhancement, so it seems like it's just a matter of making it optional.
2. Is there any support for recursing on something that's more than a single, self-referential edge? Ie, can the JOINs that are on the recursive side of the CTE be a multi-vertex traversal rather than a single-edge JOIN?

Bonus Question:
After reading through the doc in https://graphql-compiler.readthedocs.io/en/latest/index.html (and I may have just missed something); is there any tooling that is provided or recommended to hydrate the SQL response back into a Graphql format?

Thanks for taking the time to look at this!
-yun

PS: I apologize if this isn't the correct place to post the questions above. I've also noticed some inconsistencies with validations that I can post on a different thread with more details (if this is the correct place to post these issues):

1. Runtime parameters used inside macros seem to require assignments on both the macro definition and the query that uses it, but then fails when merging the parameter assignments, getting a dupe key issue because the same key has to be on both sides. This seems to be somewhat similar to Colocated tags during macro expansion cause compilation errors #597
2. It seems to be impossible to use the graphql-compiler on a SqlAlchemy Date column, as sending in a datetime.date results in a SqlAlchemy "Don't know how to literal-quote value" error, while using a str(datetime.date) version results in a graphql-compiler type mismatch error.

[obi1kenobi]

Hello! Thanks for checking out GraphQL compiler; this is definitely a good place for the questions you posted.

Infinite depth recursion is a very interesting question. When we initially added the @recurse directive, we thought that it might be dangerous to have unbounded recursion: in the presence of cyclic data, it could lead to infinite result set sizes and might be a "accidentally consume all DB resources" kind of mis-feature. But as is frequently the case, there could be some elegant tweaks to the approach that might make this a safer and more powerful feature, and I'd love to chat more about this since it seems you've already spent time thinking about this in depth.

The multi-vertex recursion traversals are definitely something that would be great to support -- they'd be very useful. They are a bit tricky to implement because of some limitations of the compiler's intermediate representation, so we'd have to be careful in how we implement them. Again, I'd love to chat about what you're doing and figure out what approach would be most useful.

Overall, it sounds to me like you have an interesting use case here that is pushing the boundaries of what we've needed to build thus far. If you're up for it, I'd love to hop on a video call and talk in more depth! My Twitter DMs are open and I'd love to hear about your use case and what you've liked and not liked about GraphQL compiler so far, and talk about the different ways things could be implemented to work.

P.S.: If you're wondering why this project (and especially the SQL side) has been a bit quiet lately, it's because my focus has been on querying over non-database data sources such as Dockerfiles, Python source code, and configuration files! I recently gave a 10min conference talk on this work: https://www.hytradboi.com/2022/how-to-query-almost-everything

[blxgl]

Hi @obi1kenobi, thanks for the response; I work with @pazurururu and just wanted to follow up.

We were exploring the use of your GraphQL compiler to assist mostly in an API serving context. Although DB resource usage could still be heavy, we were planning to allow more complicated graph traversal, as well as using UNION / UNION DISTINCT rather than UNION ALL to prevent an infinite result set.

Since the API has to return GraphQL as well, we were wondering if this was planned or if the hydration step is better done outside of the compiler?

I saw your talk on trustfall; it's quite impressive to see the productivity gain for querying arbitrary data. I noticed you referred to the GraphQL compiler as being in maintenance mode; does this mean further enhancements to the SQL backend in particular are not planned?

Thanks again for the library and your help!

[obi1kenobi]

Hi @blxgl!

At Kensho, GraphQL compiler was for a long time deployed in an API-serving context, but the APIs we were serving were REST and just implemented under the hood as GraphQL compiler queries instead of GraphQL end-to-end. Since we never used Relay / React directly against GraphQL compiler, we never explored hydrating the results either inside or outside the compiler. I'm sure it's possible to do, though!

Unfortunately, the products that used GraphQL compiler directly against SQL databases are no longer in development, and the SQL backend is therefore also not being actively developed. With trustfall gaining Python bindings and additional query capabilities that GraphQL compiler doesn't have, I think over time it is likely to supersede GraphQL compiler entirely. In particular, I recently figured out how to do unlimited-depth recursion in trustfall based in part on your interest :)

I understand that you are targeting a database, and trustfall doesn't currently include any SQL compilation features. However, my trustfall roadmap includes supporting the output of proper GraphQL-structured results as well as unbounded recursion. If trustfall could be of use here, I'd be happy to prioritize them to help you get started and I would be grateful for any feedback you might have!