How to make Comunica traverse a sequence of graphs?

[Conal-Tuohy]

I understand that a hypermedia source can return a metadata graph to Comunica to cause it to read a large dataset in a paginated fashion, by following http://www.w3.org/ns/hydra/core#next predicates from one graph to the next.

I also understand that a TPF or QPF server is considered a hypermedia source, but I'm not sure if I configure Comunica with a source URL which just resolves to an RDF/XML graph, would that also be considered a hypermedia source (and hence be retrieved in a paginated manner), or is that just a file source? How should I configure Comunica so that it will traverse a linked list of graphs?

Background: I'm investigating how to provide a SPARQL query interface to a large data source (the "Trove" service of the National Library of Australia) which is accessible by the "Trove Web API". My plan is to use a transforming proxy layer to convert responses from that API into RDF graphs. The API offers pagination of query results, so I could easily convert the "next page" link in each result to use the Hydra vocabulary.

Thanks!

[Conal-Tuohy]

Another possibility I've considered, rather than configure Comunica to use a specific Trove query URL as as data source (as in my question above), was to implement a TPF or QPF server on top of the Trove API, and configure Comunica to use that service. A SPARQL query could then retrieve graphs from the TPF service with e.g. SELECT ?trove-resource WHERE ?trove-resource trove:matchesQuery "some-Trove-query-specification". This would have the advantages that Comunica's configuration would be fixed, and the Trove query URL would be a part of the SPARQL query.

But I'd also like to be able to retrieve full descriptions of each resource from the TPF server (I understand that a TPF server may return any triples it wants to, not just triples that match the query pattern), and I'm wondering if there were some way to configure Comunica so that a SPARQL query like the below would be satisfiable with a single TPF query.
e.g.

DESCRIBE ?trove-resource
WHERE ?trove-resource trove:matchesQuery "some-query-specification"

producing a single TPF query

?trove-resource trove:matchesQuery "some-query-specification"

My TPF server could return all the properties of each trove:resource, but I'd like to ensure that Comunica didn't then make a potentially very large number of follow-up TPF queries such as:

<trove:resource-1> ?predicate ?object

... for each trove:resource retrieved from the first TPF query.

[rubensworks]

Hi @Conal-Tuohy, your use case sounds very similar to a recent paper where a TPF server was setup over the ConceptNet API, and then queried with Comunica. So I'd recommend having a look at that paper, as you might be able to follow a similar approach.

[Conal-Tuohy]

Thank you very much @rubensworks I have just read the paper you cited. The situation described in the paper is indeed similar in some ways to my project, but there are some differences which are what give rise to my questions above.

• If I understand the ConceptNet paper correctly, the authors were in the position of being able to run a local instance of the ConceptNet API and to modify it to better support their TPF server. I'm not in a position to do either of those things; I am restricted to querying a remote HTTPS endpoint which I can't change. I have to live with it.
• The Trove dataset is essentially a large (n≅250M) list of records mostly describing information resources (books, newspaper articles, etc), each of which will be mapped to an RDF graph of I guess somewhere between 20 and 50 triples. These individual "records" will be linked together into clusters corresponding to common values of facets such as "genre", "date published", "newspaper title", etc. Many (most) of these records also have full text resources which I'd like to be able to search and to represent the search hits as RDF triples.
• I think it's likely that SPARQL query authors will want to retrieve data about hundreds of Trove records, but perhaps even tens or hundreds of thousands? It's hard to say. But I am trying to find an approach that could support queries of that scale. The underlying Trove API will return up to 100 records in each API call, so 100k records would require a minimum of 1000 Trove API calls.

I would also like my middle-ware (the Trove API to RDF transformation server) to be fully stateless, if possible. I'd prefer not to maintain a big cache of the retrieved data, though I can do that if it's necessary.

Ideally, for a Trove query which returns n pages of results, I'd like the following SPARQL to generate n TPF queries:

DESCRIBE ?trove-resource
WHERE ?trove-resource trove:matchesQuery "some-query-specification"

I could write a TPF server that would response to a query like http://example.com/trove-tpf?p=trove:matchesQuery&o=some-query-specification with e.g.

<trove-resource-1>  trove:matchesQuery "some-query-specification"
<trove-resource-1>  trove:title "Some book title"
<trove-resource-1>  trove:author <author-1>
<author-1> trove:name "Joe Bloggs"
<trove-resource-2>  trove:matchesQuery "some-query-specification"
<trove-resource-2>  trove:title "Some other book title"
<trove-resource-2>  trove:author <author-2>
<author-2> trove:name "Jane Doe"
etc.

i.e. the response from the TPF server would exhaustively express the data which it had received from the upstream Trove API. But it seems to me that Comunica would have no way of knowing that it had retrieved all the information, and it might still send follow-up queries for further information e.g.

• http://example.com/trove-tpf?s=<trove-resource-1>,
• http://example.com/trove-tpf?s=<author-2>
• etc.

That would make the query run at least 100x slower.

Perhaps instead I should treat the Trove query specification as a "named graph", and implement a QPF server instead of a TPF server? Could I expect that Comunica would issue QPF queries like:http://example.com/trove-tpf?g=<my-trove-query-specification> from a SPARQL query such as

DESCRIBE ?trove-resource 
FROM <my-trove-query-specification>
WHERE { ?trove-resource a trove:resource }

[rubensworks]

I would also like my middle-ware (the Trove API to RDF transformation server) to be fully stateless, if possible. I'd prefer not to maintain a big cache of the retrieved data, though I can do that if it's necessary.

Makes sense. But I would recommend putting a cache in front of the TPF server in any case.

I could write a TPF server that would response to a query like http://example.com/trove-tpf?p=trove:matchesQuery&o=some-query-specification with e.g.

TPF servers should only return triples matching the triple pattern. Even if more is returned, Comunica will only consider the triples matching the triple pattern.

(this also applies to your question on QPF)

[Conal-Tuohy]

I would recommend putting a cache in front of the TPF server in any case.

OK so I guess that means Comunica doesn't cache TPF data itself? I will do that, since it will definitely speed up repeat executions of the same or similar SPARQL queries.

TPF servers should only return triples matching the triple pattern. Even if more is returned, Comunica will only consider the triples matching the triple pattern.

Thanks; that's good to know. I read in the TPF spec that a TPF server could return additional triples, but it helps to know that they won't be used by Comunica at all.

Am I correct to think that Comunica will execute a QPF query specifying ?g=<some_named_graph> when I give it a SPARQL query like this:

SELECT ?s ?p ?o
WHERE
{
   GRAPH <some-trove-query-url> { ?s ?p ?o }
}

[rubensworks]

OK so I guess that means Comunica doesn't cache TPF data itself? I will do that, since it will definitely speed up repeat executions of the same or similar SPARQL queries.

Comunica caches as well. The server-side cache is mainly important for scalability in terms of an increasing number of parallel clients (which could be comunica, or something else).

Am I correct to think that Comunica will execute a QPF query specifying ?g=<some_named_graph> when I give it a SPARQL query like this:

Yep, that's correct.

[Conal-Tuohy]

OK in most cases I expect the QPF server to be located on the same host as Comunica and have just that Comunica instance as its only client, so I probably don't need a second cache.