Support for pagination and categories/tags

[gesposito]

I haven't dig much about it, does gatsby have built in support for pagination and categories/tags?
i.e.
http://blog.ghost.org/page/2/
http://blog.ghost.org/tag/writing/

That would sum up in two special routes that accept a wildcard * after the main path.
/page/*/
/tag/*/

While playing with gatsy I also found beneficial to leave the main route "themable", I mean
/page/_template.jsx
/tag/_template.jsx
As it would work for WordPress (use the custom page if found, or pick the default one).

Another different scenario is the 404 page that should be supported by React Router.

[KyleAMathews]

Not yet. This would be very useful. I think how it should work is that index files should be use the Router's * routes. So /page/2 would be handled by /page/index.jsx. It would be passed a prop with a 2 so it would know what to do.

I think ideally there would be a gatsby-pagination npm module that people could install that for the most part would "just work". I think for that to happen, Gatsby needs some sort of middleware concept where route components could get wrapped by plugins. Also we need a better way of representing page data, perhaps #15 so it'd be very easy for a pagination plugin or a tag module to wrap index files, look for url arguments and do logic like "filter out all markdown files matching path /blog/* that have the category of writing".

Thoughts?

[gesposito]

On the writing side (.md) page data looks fine to me, I'd leave it as it is for compatibility and porting (i.e. Jekyll).
Tags are as easy as adding a row into the .md header, then the index will do the rest (filtering). If it is for performance gains, then they could be stored/indexed in a more efficient manner (at compile/static generation time).

Pages are a little tricky, and needed (out of the box?), look at the gatsby-starter-documentation and its nested pages structure that is also suitable for organizing topics/layouts in a blog/site, all of those "sub trees" should have their own pagination too.

[KyleAMathews]

Right, pagination is tricky and it'd be nice to put that logic in its own plugin. Here's an idea for how plugins could be structured.

paginationPlugin: [{
  blog: {
    pathPattern: /^\/blog/.*/,
    outputPathPattern: "/blog/{pageIndex}/"
    process: (pages) => { // function is called with all that pages that matches the `pathPattern`
      // Order pages then return object like { 1: [// pages], 2: [// pages], 3: [// pages] }
  },
  docs: { // Do something similar here for docs }}
]

Gatsby would calculate ahead of time the pagination. Then for each possible output page (e.g. /blog/4/) Gatsby would generate a page using a template that you'd also configure with the plugin.

[KyleAMathews]

We need to start brainstorming for a proper plugin API.

[KyleAMathews]

So one thought here is to let pages use the RoutePattern from React-Router. So a blog index at /blog/index.js could declare its path as /blog/:page. It would then be called whenever some visits /blog/1/, /blog/2/ etc.

[KyleAMathews]

Category/tag pages could do something similar e.g. /tags/:tag.

[luckypoem]

hi.
@KyleAMathews
now the pagination is still not supported?
http://surmount.biz.st:2357/
how to paginate?

[KyleAMathews]

Not yet...

[funkybunky]

Hey there! :) Could you please outline the steps (files to touch, etc) in order to implement a tags feature? I'm thinking about contributing, but as I'm not familiar with the codebase, could need a few hints where to start.

[KyleAMathews]

@funkybunky this is actually something I'm working on right now :-) to make it happen will require a fairly big change so not something that's easy to describe here.

If you need tags in the short-term, the easiest thing to do is to just manually (or programmatically) create pages for each tag e.g. /pages/tag-1.js, /pages/tag-2.js, etc. and in each filter out the tagged content you want to show.

[funkybunky]

@KyleAMathews thanks for the follow-up! Cool to know that you're working on it. If you need any help, let me know :)

[KyleAMathews]

Will be releasing initial version soonish — would love feedback + help
working out kinks!

On Sat, Aug 13, 2016 at 2:14 PM Marcus Kleppe notifications@github.com
wrote:

@KyleAMathews https://github.com/KyleAMathews thanks for the follow-up!
Cool to know that you're working on it. If you need any help, let me know :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#33 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEVhzhVMoBAppoa95HMIqXWZvNarQoNks5qfjOzgaJpZM4F8oO_
.

[KyleAMathews]

Hey folks! Just wrote up ideas to make this possible. Would love to hear what you think. See #419 #420 and #421

[bvaughn]

Don't suppose any conclusion was reached regarding URL parameters / wildcard routes? I find myself needing such a feature for minified error messages. 😄

[KyleAMathews]

@bvaughn — couldn't you just create the pages normally? Or is the data not known at build time and you dynamically fetch it? If that's the case, you can create client only pages https://www.gatsbyjs.org/docs/creating-and-modifying-pages/#creating-client-only-routes

[bvaughn]

couldn't you just create the pages normally?

No, I could generate part of the routes "normally" (eg the error codes themselves) but part is also dynamic (eg the name of a component or variable, eg this).

Thanks for the client-only route link. I couldn't find that earlier when I was searching. Are there any drawbacks (in terms of performance, etc) to using this approach? There's no async fetching or anything, just parameter-injection via URL.

[KyleAMathews]

There's no async fetching or anything, just parameter-injection via URL.

Oh ok, just use the location prop from RR then. It's passed to every page/layout component and has everything parsed for ya.

[bvaughn]

Sure, sure. I just didn't know what the page.path bit so the route wasn't matching in the first place. 😄

[benjamingeorge]

I've only been evaluating Gatsby for a day by using the gatsby-source-wordpress plugin. For post index pages (list of ten posts and pagination links), do I have to get all the wordpress posts then render a static page for every 10 posts in the gatsby-node.js file?

[KyleAMathews]

@benjamingeorge basically. gatsby-source-wordpress handles efficiently pulling all Wordpress posts into Gatsby then you create the paginated pages.

[bvaughn]

Sorry to circle back after such a delay. I got side-tracked by 16 beta! 😄

I think I created a client only route as described in the docs you shared:

exports.onCreatePage = async ({ page, boundActionCreators }) => {
  const { createPage } = boundActionCreators;

  return new Promise((resolve, reject) => {
    if (page.path.includes('docs/error-decoder.html')) {
      page.matchPath = 'docs/error-decoder.html?:args';
      page.context.slug = 'docs/error-decoder.html';

      createPage(page);
    }

    resolve();
  })
};

After doing this, the route almost works but not quite:

Works?	URL
✓	localhost:8000/docs/error-decoder.html
✓	localhost:8000/docs/error-decoder.html/foo
✓	localhost:8000/docs/error-decoder.html/?foo
✖	localhost:8000/docs/error-decoder.html?foo

Unfortunately, the last one is the one I specifically need to work for backwards compatibility with the React error page.

I've tried a few variations for the matchPatch in onCreatePage:

page.matchPath = 'docs/error-decoder.html?invariant=:invariant&args=:args';
page.matchPath = 'docs/error-decoder.html?:args';
page.matchPath = 'docs/error-decoder.html:args';
page.matchPath = 'docs/error-decoder.html*';
page.matchPattern = /docs\/error-decoder\.html.+/;

Am I perhaps misunderstanding something or overlooking? 😄

Edit For what it's worth, I just updated to the latest Gatsby (and co) versions to rule out something that had already been fixed.

[KyleAMathews]

Hmm dunno. This is a react-router function under the hood so I'd check their docs on this.

[bvaughn]

Is this really a react-router thing though? To my knowledge, react-router doesn't deal with query strings. It leaves them up to the user. Route only matches location.pathname.

For example, check out this bin: https://www.webpackbin.com/bins/-KrgnTIAd88pSOiSeNQK

In it, I define a route:

<Route path="/foo.html" component={Foo}/>

And that route automatically works with query params (eg /foo.html?bar=abc). But with Gatsby, none of the following route definitions work:

page.matchPath = 'docs/error-decoder.html';
page.matchPath = 'docs/error-decoder.html?invariant=:invariant&args=:args';
page.matchPath = 'docs/error-decoder.html?:args';
page.matchPath = 'docs/error-decoder.html:args';
page.matchPath = 'docs/error-decoder.html*';
page.matchPattern = /docs\/error-decoder\.html.+/; // This one I just took a guess at

Or rather, they all almost work (as described above) but not the one format I need. 😁

[KyleAMathews]

You're right, I shouldn't have said we're vanilla react router matching. If you look in here you'll probably find where things are going bad. https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/cache-dir/find-page.js

[bvaughn]

Cool! Thanks for the pointer (and the prompt response, as always).

That function doesn't seem to be invoked though when I try to load /docs/error-decoder.html?foo. I just get an immediate 404. It is run (and matches successfully) when I use the other variants mentioned above.

[bvaughn]

I tried renaming matchPath to something totally unlike the name of the page file (template) in pages- in case this was causing troubles.

So I've just specified a matchPath of /foo:path for now.

Next I tried loading the URL, /foo?bar.

The pathname that gets passed to find-page.js (and so to matchPath) is "/foo" though. (The query string has been trimmed off.)

Stepping further, the regular expression that's generated inside of matchPath is /^\/foo\?((?:[^\/]+?))(?:\/(?=$))?(?=\/|$)/i. This regex matches "/foo?bar" but not "/foo" - and so my matchPath doesn't match.

I think maybe gatsby/cache-dir/component-renderer.js should pass location.search to getResourcesForPathname also? That way it could be considered when matching paths.

Edit: I can work around this particular issue with page.matchPath = "/foo:path?" but unfortunately /docs/error-decoder.html?invariant still 404s (before it even executes the code in find-page.js).

[KyleAMathews]

Ah this could be it. We only pass the pathname to findPage in https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/cache-dir/production-app.js

[bvaughn]

Yeah. I'm just a bit baffled still by the specific issue with this page.

I can make the foo?bar query route work by specifying page.matchPath = "/foo:path?" (eg tagging the :path bit as optional in a way react-router will understand). But for some reason, /docs/error-decoder.html?invariant... doesn't even execute find-page.js at all. It just insta-404s.

[bvaughn]

I think the other half of my issue (the instant-404) has to do with the file extension in the path.

// 404s when loading "/foo/bar.baz" and "/foo/bar.baz?qux"
// Doesn't even execute find-page.js
page.matchPath = "/foo/bar.baz:qux?";

// This on the other hand does execute find-page.js
// Successfully matches  "/foo/bar" and "/foo/bar?baz"
page.matchPath = "/foo/bar:baz?";

Breaking this down further has helped a lot. The first half of the problem was the dynamic token bit needing a "?" suffix. The second half seems to be the file-extension and I'm not yet sure how to handle that. Maybe Gatsby (or whatever is serving up the localhost env) is treating potentially real/static files (things with extensions) differently somehow?

Escaping the "." (eg "/foo/bar\.baz") also executes find-page.js.

[bvaughn]

I think the next (last?) thing to figure out is why file extensions in the path cause inconsistent routing behavior.

URL	Response
localhost:8000/foo.html	Gatsby dev 404 page
localhost:8000/foo.html?	404 (Not Found) status code
localhost:8000/foo.bar	404 (Not Found) status code

The above routing behavior block things before find-page has a chance to match-up the wildcard routes.

Edit: It looks like things are going wrong in utils/develop. See #1844

[KyleAMathews]

There's now https://github.com/pixelstew/gatsby-paginate which is pretty awesome. Closing this issue as it's old and not active.

[ciokan]

There's now https://github.com/pixelstew/gatsby-paginate which is pretty awesome. Closing this issue as it's old and not active.

That package is unmaintained and buggy. IMO pagination should be built-in somehow. It's a thing that almost every project will be using.

[MoOx]

For people interested in a static site generator that works with React and provide pagination built-in, you should take a look to Phenomic (here https://phenomic.io/en/packages/preset-react-app/docs/getting-started/05/#immutable-pagination)

The goal of my comment is to show that pagination can (and should) be taken seriously when it comes to SSG. Phenomic decided to have this built-in since day 1. I am not trying to be a dick by saying "come, my ssg is better" (cause it's probably not) but more "it can be done, you should do it".

[ciokan]

For people interested in a static site generator that works with React and provide pagination built-in, you should take a look to Phenomic (here https://phenomic.io/en/packages/preset-react-app/docs/getting-started/05/#immutable-pagination)

Stop hijacking threads please. Your comment brings no value.

Update: Why are you responding with a "confused" emoji? What you're doing here is very unprofessional. Going out to direct "competition" communities and posting your links all over the place. Especially on issues where we're trying to make the software better and collaborate on things. That's a shameless plug that is nowhere tangential with the issue being discussed. It's completely parallel and designed to suck members over to your product.
I find it worse than people scraping for phone numbers and then sending unsolicited promotional messages.
Don't act confused now because you know very well what you're doing. The right "reaction" would be to delete your useless comment because this is not how you bring awareness. Not in 2018.

[kbariotis]

I've just published https://github.com/kbariotis/gatsby-plugin-paginate which is a Gatsby plugin that does pretty much what gatsby-paginate does but without having to mess with Gatsby's Node.js API. Thank you, let me know what you think.

[clkent]

Reading through this thread, I'm not positive what I'm trying to do is possible or if there is a solution available for it that I'm just not finding...

I'm building out a blog using gatsby with content pulled from Prismic. Each blog post has an author and tag related to them via Prismic Content Relationship. My goal is to dynamically create pages via gatsby-node for the author and tag pages that also include pagination for their related blog posts. Prismic unfortunately doesn't seem to create a relationship going both ways, so I have to find related blog posts by doing a graphql query on my allPrismicBlog filtering for author uid.

example of what I'm trying to create - author-name needs to be dynamically created as well:
myblog.com/author/author-name/
myblog.com/author/author-name/2

I have the following in my gatsby-node:

exports.createPages = async ({ graphql, actions }) => {
  const { createPage } = actions;
const authors = await graphql(`
    {
      allPrismicAuthor {
        edges {
          node {
            uid
          }
        }
      }
    }
  `);
  authors.data.allPrismicAuthor.edges.forEach(edge => {
    const authorUid = edge.node.uid;
    const authorPosts = graphql(`
    {
      allPrismicBlog(filter: { data: { author: { uid: { eq: ${authorUid} } } } }) {
        edges {
          node {
            uid
        }
      }
    }
    `);
    const numAuthorPages = Math.ceil(authorPosts.length / 2);
    Array.from({ length: numAuthorPages }).forEach((_, i) =>
      createPage({
        path: i === 0 ? `/author/${authorUid}` : `/author/${authorUid}/${i + 1}`,
        component: path.resolve('./src/templates/author.jsx'),
        context: {
          limit: 2,
          skip: i * 2,
          numPages,
          currentPage: i + 1,
          uid: authorUid,
        },
      }),
    );
  });
};

I'm getting the error TypeError: Cannot read property 'page' of undefined

I'm not sure if what I'm trying to do here is the right direction or if I'm missing something important. Any help would be greatly appreciated.

[jonniebigodes]

@clkent i think the issue lies here:

const authorPosts = graphql(`
    {
      allPrismicBlog(filter: { data: { author: { uid: { eq: ${authorUid} } } } }) {
        edges {
          node {
            uid
        }
      }
    }
    `);

The graphql queries will not run synchronously, they run in a async fashion. Meaning that the code you have there, it will iterate each element in the foreach, not waiting for anything, basically taking a approach of "fire and forget" and by that that i mean it triggers a graphql query and it's not capturing either errors or results.
You'll have to make some adjustments to the code. Me personally i would prefetch all of the information in the query in something like:

const result = await graphql(`
    {
      AuthorInformation:allPrismicAuthor {
        edges {
          node {
            uid
          }
        }
      }
      BlogPosts: allPrismicBlog {
       ......
        }
      }
    }
  `);

then you have both bits of data available in one query and you paginate the data based on what you have available already. Alternatively you probably would have to use promise all to wait for it and then proceed with the pagination.

[clkent]

@jonniebigodes thanks for your response...

is there some way to use the returned uid in AuthorInformation to filter in the query BlogPosts below? In order to query the correct blog posts based on author I need to filter my query... allPrismicBlog(filter: { data: { author: { uid: { eq: UID HERE } } } }) . Alternatively I guess I could just write some JS to filter the blog posts after I query for all of them...

Regardless of the query I am still left with the issue of dynamically creating the author pages from the allPrismicAuthor query results with pagination for each based on the number of Blog Posts they have. Even if I hardcode in the number of pages, my logic above still doesn't work.

[clkent]

Figured out a solution and wanted to share here in case anyone else runs into something similar in the future.

Instead of trying to query for the blog posts with the author uid and dealing with the async nature of the two queries I am just filtering the blogList and creating pages based on that. There's probably several ways to improve this code during a refactor but wanted to share what I got working.

const blogList = await graphql(`
    {
      allPrismicBlog(sort: { fields: [data___blog_post_date], order: DESC }, limit: 1000) {
        edges {
          node {
            uid
            data {
              author {
                uid
              }
              tag {
                uid
              }
            }
          }
        }
      }
    }
  `);

 const posts = blogList.data.allPrismicBlog.edges;

const authors = await graphql(`
    {
      allPrismicAuthor {
        edges {
          node {
            uid
          }
        }
      }
    }
  `);

  authors.data.allPrismicAuthor.edges.forEach(edge => {
    const authorUid = edge.node.uid;

    const authorBlogs = posts.filter(post => post.node.data.author.uid === authorUid);
    const numAuthorPages = Math.ceil(authorBlogs.length / 1);

    for (let i = 0; i <= numAuthorPages; i++) {
      createPage({
        path: i === 0 ? `/author/${authorUid}` : `/author/${authorUid}/${i + 1}`,
        component: pageTemplates.Author,
        context: {
          limit: 1,
          skip: i * 1,
          numPages,
          currentPage: i + 1,
          uid: authorUid,
        },
      });
    }
  });

[jonniebigodes]

@clkent glad that you managed to work it out, you can even improve it and make it more efficient by merging both queries into one with aliasing, removing the need of having one extra graphql and have to await for it to resolve and get the results back.
something like the following:

 {
      allBlogPosts:allPrismicBlog(sort: { fields: [data___blog_post_date], order: DESC }, limit: 1000) {
        edges {
          node {
            uid
            data {
              author {
                uid
              }
              tag {
                uid
              }
            }
          }
        }
      }
      allAuthors:allPrismicAuthor {
        edges {
          node {
            uid
          }
        }
      }
    }

Also as a good practice you could introduce a sanity check for errors:
Something like:

const blogList=.....
if (blogLIst.errors){
   throw new Error('some error happened')
   return
}

[snikidev]

@clkent amazing, your solution worked for me, thank you! 🙌 good ol' .filter() is a nice touch! 👍