RFC: Adapters

[LekoArts]

Summary

We intend to add an additional type of plugin to Gatsby called “Adapter”. Adapters are responsible for taking the production output from Gatsby and turning it into something your deployment platform (e.g. Netlify, Vercel, Self-Hosted, etc.) understands.

We want to make it easier to deploy and host Gatsby on any platform.

As part of this work we also intend to add a headers and adapter option to gatsby-config.

You can follow the Adapters milestone to see the progress and PRs.

Try it out

Warning
This is an Alpha, we strongly recommend to try Adapters on a development branch of your site (no YOLO’s yet, please!).

Using a demo project

Clone the existing starter gatsby-adapters-alpha-demo

cd gatsby-adapters-alpha-demo
npm install --legacy-peer-deps

Follow its README to either deploy it to Netlify, run ntl serve or use the local adapter.

You can also individually install two packages and its canary versions:

• gatsby@alpha-adapters
• gatsby-adapter-netlify@alpha-adapters

You can find in-progress documentation here: #38233

Using an online IDE

• Codesandbox

Basic example

By default, Gatsby will have a manifest of popular adapters it’ll reference during a build. This enables zero-configuration deployments on those deployment platforms.

For example, on Netlify gatsby will automatically install gatsby-adapter-netlify. You can manually install it inside your dependencies if you want to skip the auto-install on each build. The adapter will then setup everything needed to host Gatsby on Netlify.

For this zero-configuration deployments no manual user interaction is needed (e.g. no installation/usage of a “auto” package).

If you want to use an adapter that is not part of Gatsby’s manifest, you can install it into your dependencies and add it to your gatsby-config like so:

import type { GatsbyConfig } from "gatsby"
import adapter from "gatsby-adapter-name"

const config: GatsbyConfig = {
  adapter: adapter(),
}

export default config

If the adapter is taking in options, you can set them:

const config: GatsbyConfig = {
  adapter: adapter({ optionA: true, optionB: false }),
}

As part of this RFC we also saw the need to allow setting HTTP headers for request paths which you’ll be able to do with the new headers option:

import type { GatsbyConfig } from "gatsby"

const config: GatsbyConfig = {
  headers: [
    {
      source: "/some-path",
      headers: [
        {
          key: "x-custom-header",
          value: "hello world"
        }
      ]
    }
  ]
}

export default config

Motivation

Right now, plugins for deployment platforms like gatsby-plugin-netlify, @vercel/gatsby-plugin-vercel-builder or gatsby-plugin-fastify are reaching into Gatsby’s internals to figure out the same things:

• List of routes and their type
• List of redirects & rewrites
• Additional information for Functions/SSR/DSG
• Often those plugins also allowed users to set HTTP headers for their request paths, making this information proprietary to that specific plugin.

These information are currently only available inside Gatsby plugins and thus the deployment platforms needs to add their plugin to gatsby-config during a build, messing with a user’s configuration. This is brittle.

We want to make it easier to write glue-code between Gatsby and a deployment platform by providing all these information (including headers) in a structured way. In the end, the hard problems should be solved by Gatsby itself and the adapter for the platform should just consume all the information and adapt it.

For popular platforms no user interaction at all should be necessary.

Detailed design

You might be familiar with our proposed APIs so, credit where credit's due, we’d like to point out:

• The Adapters pattern and API design is inspired by SvelteKit (Using Adapters, Writing Adapters)
• The headers option is inspired by Next.js (Setting headers)

Adapters

An adapter is a npm package that can be authored in CJS, ESM, or TS and for now has to be compiled to CJS. You’ll be able to publish your adapter as ESM once we implement ESM in TS support as part of ESM in Gatsby files.

The entrypoint of the adapter has to export a function as a default export with the following shape. Here’s an example adapter and what it could do:

import type { AdapterInit } from "gatsby"

interface AdapterOptions = {}

const createAdapter: AdapterInit<AdapterOptions> = (options) => {
  return {
    name: `gatsby-adapter-name`,
    cache: {
      restore({ directories, reporter }): void {
        // TODO
      },
      store({ directories, reporter }): void {
        // TODO
      },
    },
    adapt({ routesManifest, functionsManifest, reporter }): void {
      // TODO
    },
  }
}

export default createAdapter

Here’s an explanation of the code from top to bottom:

• If you’re authoring your code in TS or want to use JS type hints, you can use the AdapterInit type. It takes in a generic which should be your adapter options if you have any
• The adapter should return an object with these keys:
  • name (required)
  • cache (optional)
  • adapt (required)
• For the name, please follow the naming scheme of gatsby-adapter-X or @scope/gatsby-adapter-X
• The cache handlers receive the directories that should be cached/restored for a build. At the moment these are .cache and public. The cache restoration happens at the correct time during the build so that as little work as possible needs to be redone.
• The adapt function receives routesManifest and functionsManifest (which will be explained further below) and this is the core part of the adapter. With these information provided there you’ll hopefully be able to adapt the output to your deployment platform.
• All three utility functions receive the reporter instance you’re used to, so you can output structured logs to the user’s terminal. However, please don’t overdo it and keep the output to a minimum. The user will already get information what adapter is used and can debug things further by running in verbose mode.

routesManifest & functionsManifest

Both the routesManifest and functionsManifest are an array of objects. Here are the type definitions for routesManifest:

interface IBaseRoute {
  /**
   * Request path that should be matched for this route.
   * It can be:
   *  - static: `/about/`
   *  - dynamic:
   *    - parameterized: `/blog/:slug/`
   *    - catch-all / wildcard: `/app/*`
   */
  path: string
}

interface IStaticRoute extends IBaseRoute {
  type: `static`
  /**
   * Location of the file that should be served for this route.
   */
  filePath: string
  headers: IHeader["headers"]
}

interface IFunctionRoute extends IBaseRoute {
  type: `function`
  /**
   * Identifier of the function. Definition of that function is in function manifest.
   * Definition of function is not left directly here as some functions will be shared for multiple routes - such as DSG/SSR engine.
   */
  functionId: string
  /**
   * If `cache` is true, response of function should be cached for current deployed and served on subsequent requests for this route.
   */
  cache?: true
}

interface IRedirectRoute extends IBaseRoute {
  type: `redirect`
  toPath: string
  status: HttpStatusCode
  ignoreCase: boolean
  headers: IHeader["headers"]
  [key: string]: unknown
}

type Route = IStaticRoute | IFunctionRoute | IRedirectRoute

type RoutesManifest = Array<Route>

You’ll learn more about the headers further below.

The routesManifest array is sorted by specificity, with the highest one at the top. This way you can use the first entry that matches your URL/pattern.

And here’s the functionsManifest:

interface IFunctionDefinition {
  /**
   * Identifier of the function. Referenced in routes manifest in function routes.
   */
  functionId: string
  /**
   * Path to function entrypoint that will be used to create function.
   */
  pathToEntryPoint: string
  /**
   * List of all required files that this function needs to run
   */
  requiredFiles: Array<string>
}

type FunctionsManifest = Array<IFunctionDefinition>

The pathToEntryPoint files are being generated by Gatsby and should allow for an easier deployment. This includes some built-in guardrails for SSR/DSG functions. The functions export Express-like handlers that the platform can use (or further adapt). We’re considering to potentially change those function signatures to fetch API Response/Request format — please let us know if that would be something you’d also be interested in.

We hope that these manifests give you all the information you need to deploy all artifacts to various platforms. If you’re missing something, please let us know!

How it works under the hood

So that’s how you author an adapter and what information is available. Now, if you’re interested you can read up on how it’s all done on Gatsby’s side.

Gatsby has a manifest file in the shape of:

const adaptersManifest = [
  {
    name: `Netlify`,
    module: `gatsby-adapter-netlify`,
    test: () => !!process.env.NETLIFY,
    versions: [
      {
        gatsbyVersion: `^5.0.0`,
        moduleVersion: `^1.0.0`
      }
    ]
  },
]

This manifest is responsible for the zero-configuration deployments available on certain deployment targets. We’ll only be adding entries to this manifest if the adapter is meeting our quality standards and is a widely used platform.

When a build is triggered, Gatsby fetches this manifest (first from unpkg.com, falling back to the file inside gatsby ) and finds the adapter to use via the test function. If no adapter is found, things just continue without any changes.

Then Gatsby tries to resolve the adapter in 3 steps:

• If the user manually installed it in their project (as a dependency/devDependency), resolve it
• Check if a previous run has installed the adapter inside .cache/adapters
• If both of these checks are falsy, install the adapter into .cache/adapters and resolve it

If you define an adapter through the adapter option inside gatsby-config the manifest and zero-configuration codepaths are completely skipped and your defined adapter is directly used. Either way, once Gatsby figured out which adapter to run, its functions are run and the adapter can do its work.

Headers

The headers option inside gatsby-config accepts an array of objects in the following shape:

interface IHeader {
  source: string
  headers: Array<{
    key: string
    value: string
  }>
}

type Headers = Array<Header>

source is a request path pattern, the headers are key-value pairs. You won’t need to worry about trailing slashes for source paths as it all gets normalized.

If headers match the same path and set the same header key, the header with the highest specificity will override the other entries. Otherwise headers will be merged together. The order inside the headers array doesn’t matter. You’ll see two examples further below.

Routes inside the routesManifest will already have some default headers (which you can override if you want). If your header path pattern contains dynamic segments like : or * the routes will be scored and sorted by specificity, where the most specific route will “win” (be the last entry to override everything). Routes like /some-path are the most specific ones.

As an example, the static route wins against the dynamic and default header because it has the highest specificity:

const defaults = [
  {
    key: `x-custom-header`,
    value: `lose-1`,
  },
]

const headers = [
  {
    source: `/some-path/`,
    headers: [
      {
        key: `x-custom-header`,
        value: `win`,
      },
    ],
  },
  {
    source: `*`,
    headers: [
      {
        key: `x-custom-header`,
        value: `lose-2`,
      },
    ],
  },
]

// Matcher for path "/some-path/"
// => The `win` from `/some-path/` will be used

And for dynamic segments:

const headers = [
  {
    source: `/some-path/:slug`,
    headers: [
      {
        key: `x-custom-header`,
        value: `win`,
      },
    ],
  },
  {
    source: `*`,
    headers: [
      {
        key: `x-custom-header`,
        value: `lose-1`,
      },
    ],
  },
  {
    source: `/some-path/*`,
    headers: [
      {
        key: `x-custom-header`,
        value: `lose-2`,
      },
    ],
  },
]

// Matcher for path "/some-path/foo"
// => The `win` from `some-path/:slug` will be used

Adoption strategy

While we’d have liked to make this backwards compatible, it’s not feasible and therefore this feature set will only be available in Gatsby 5 and newer. The existing plugins for the deployment platforms will continue to work as usual and there is no need to deprecate/remove any of the APIs that they are using at this time.

As mentioned in the beginning, Gatsby will have a manifest of popular adapters for auto-installation. If you’re not deploying to those platforms and are also not manually installing & using an adapter, nothing will change in the Gatsby build.

If your deployment platform already has a Gatsby plugin, you’ll need to write an adapter first before people can migrate.
We’ll author the gatsby-adapter-netlify adapter and are happy to assist with other adapters.

If you then want your adapter to be added to the manifest for zero-configuration deployments, the adapter has to meet our quality standards (e.g. all features are supported) and is a widely used platform. Not every adapter will be added to the manifest.

Users who want to convert existing plugins or author new adapters should:

• Use the gatsby-adapter-X naming scheme (not an enforcement but encouraged)
• Read the authoring guide
• Read the source code for gatsby-adapter-netlify

How we teach this

We’ll add some guides:

• How to use an adapter for your Gatsby site (including the explanation about auto-installation of popular ones during build if needed)
• How to write an adapter
• How to use the headers option in gatsby-config

Where necessary, guides related to deployment will also be adjusted.

How can you help?

• Any information missing from the routesManifest and functionsManifest that would be critical for your use case?
• The functions inside the functionsManifest will be Express-like handlers. How important would it be for you to have these in the format of fetch API Response/Request format?
• Which platforms would you like to have adapters for? This is merely to a way to see interest, not an indication that we’d author those adapters

[moonmeister]

I'm so excited to see this happening. The initial read looks like this covers a lot of struggles I've seen in the development of gatsby-plugin-fastify.

My Questions:

1. The one thing it doesn't directly address is the future of gatsby serve and what self-hosting directly with a JS runtime(node/deno/bun/etc) will look like. Will the serve command be updated to actually be production ready? If so is a faster server like Fastify in the running or just sticking with Express?
2. What does support for image CDN look like? Is that a future aspect or is that included in dynamic function routes somehow?
3. I don't see client-only routes mentioned...again is gatsby abstracting this and I shouldn't need to specially handle these routes?
4. The headers handling, does this include default cache-control and applicable security headers? If not are we left figuring out best practices and defaults from docs or will there be APIs to help?
5. One of the hardest parts of implementing a custom server has been the interplay of features. for example, you release Functions, but how does that interact with Gatsby's other features? Does the trailingSlash config affect Functions? What about Path Prefix? A great example of this is Client-only routes. All existing documentation only suggests this works with Static routes. Because I know DSG routes are basically Static, I might guess Client-only routes would work with DSG, but how about SSR? Well, despite Gatsby's own documentation completely ignoring the interaction of these features, they do in fact work together. Is this getting abstracted away or is this something implementors will still need to figure out?
6. Has the team given any thought to implementing an adapter test suite? E.g. I build an adapter and deploy a pre-defined test-site with the adapter then give the test runner the site URL. It goes bananas on the site testing all Gatbsy functionality is working as expected with correct headers, routes, trailingSlashes, redirects, rewrites, proxies, images, etc. This would be a great requirement for your predefined list of "auto" adapters and even non-auto adapters. This gives users great data to understand if the adapter/host they're choosing supports all Gatsby features or just a subset.
7. What's the future of Gatsby Preview in this context? Is that something hosts can implement now, if so, how?

Answers to your questions

1. Re: missing route info: I think my questions above got at this...but are you abstracting me needing to understand what's DSG, SSR, Client-only, etc? If so, thank you, and then I think this information is probably sufficient.
2. Re: Express-like handlers - I think express-like handlers is a bad idea. Express was invented before standards like fetch existed. But now we have standards and should stick with them IMO. This is a new API and we have a chance to go with standards, let's do it! This should make implementation across various JS runtimes and server implementations much easier. Not every server supports express syntax, lets not force it on them.

[ZeldOcarina]

Amazing @moonmeister!!! I've actually tried it before deciding going with Express the good old way, the only issue it has with functions is I had to change the syntax to make it fastify-friendly, but it has been a while, I might revise it!

[tsdexter]

@moonmeister I'm still deciphering the code but it looks like functions/dsg/ssr all come through as IFunctionRoute where DSG routes have cache: true.

If I had to guess, I would say ImageCDN routes also come through as IFunctionRoute with a path like: /_gatsby/image/:url/:params/:filename and a functionId that resolves to the corresponding function in https://github.com/gatsbyjs/gatsby/blob/f606a35df84cfe1c615201da569997cb0bec3eae/packages/gatsby-plugin-utils/src/polyfill-remote-file/http-routes.ts#LL45C11-L45C50

[tsdexter]

@moonmeister looks like client-only routes come through as static routes with a dynamic path that resolves to a build-time generated HTML file:

    {
      path: '/app/*',
      type: 'static',
      filePath: 'public/app\\[...]\\index.html',
      headers: [
        {
          key: 'cache-control',
          value: 'public, max-age=0, must-revalidate'
        },
        { key: 'x-xss-protection', value: '1; mode=block' },
        { key: 'x-content-type-options', value: 'nosniff' },
        { key: 'referrer-policy', value: 'same-origin' },
        { key: 'x-frame-options', value: 'DENY' }
      ]
    },

Also, looking at this example return for routesManifest, the gatsby standard cache-control and default security headers are returned and specific to each route. This definitely eliminates a lot of what we're doing in gatsby-plugin-fastify (including my entire custom headers PR, so I'll guess I'll hold off on getting back to that). However, it is still unclear how exactly to serve the routes. It looks to me like we still have to spin up a server and hijack npm start.

[moonmeister]

Yup. This makes life a lot easier. @LekoArts I'll update my original post with answered questions, one new one. What role does gatsby serveplay with adapters?

My hope would be adapters could provide a function that this executes and passes host/ip/cert/verbose/etc info that we can handle if we need a node process for our hosting.

[LekoArts]

I posted the answer into a new message so it's not lost for other people :) #38231 (comment)

[LekoArts]

Hi @moonmeister 👋 Glad to hear that you’re excited, we are, too :)

I want to preface my answers with this:
When we say „We want to make it easier to deploy and host Gatsby on any platform.“, we mean that this will be a longer process and Adapters are a step towards that. Adapters will be an important step to help with this goal but afterwards there is still lots more to do. So while all of your questions are really good, I don’t have the answers to everything yet. We want to chip away on that goal bit by bit and we need to decide next steps after shipping Adapters. We certainly would want to know what y’all think we should work on afterwards.

With that out of the way, here are some answers:

1. As said in the preface, mentioning gatsby serve is a good and valid question, but not something we’ll focus on as part of this initial Adapters work. I don’t know yet how we want to tackle the problem of enabling users to more easily run Gatsby on those JS runtimes
2. Image CDN isn’t part of this initial work as it’ll also require some bigger work and thought process. Right now the functionality is e.g. only enabled on Gatsby Cloud through an ENV var. We’ll also need to investigate what a good API would be as we’ve seen that https://github.com/unjs/ipx works to a certain degree but providers like Imgix or Cloudinary are just better. Image CDN isn't strictly required to get a properly working site, but it's a worthwhile optimization.
3. Client-only routes will be visible inside the routesManifest
4. We’ll include default cache-control and security headers for all routesManifest entries
5. With the routesManifest being the source of truth, it’ll respect things like trailingSlash or pathPrefix and with that you’d know how things should behave (as they’ll be the same for everyone)
6. We’ll have a E2E test suite for Adapters (WIP: https://github.com/gatsbyjs/gatsby/tree/feat/adapters/e2e-tests/adapters) and a goal from the start was that authors of other Adapters could take this and validate their plugin. Still TBD how we’d make this really easy-to-use, but as a skateboard version people can copy/paste the test suite and start with that.
7. Similarly to 1), I see Gatsby Preview as a separate step in other overall goal. While we have https://www.gatsbyjs.com/docs/how-to/local-development/running-a-gatsby-preview-server/, it could certainly be made easier and more stable. So no, Adapters won’t help with that for now.

In regards to Fetch/Express-like handlers:

While Adapters is a new API, the functions we compile stayed the same since they were introduced in Gatsby 3.7. We unfortunately can’t just change the signature of those functions, as Gatsby Cloud and other hosts rely on those being Express-like.

Ideally, in the next Gatsby major we’d make the switch. We were just curious how much interest there would be about this :) Standardizing on fetch API (both compiled functions and how users would write functions) is definitely the way we see things going forward but we need to be mindful about scope-creep and breaking changes.

[ZeldOcarina]

Thank you very much for your hard work on this!!
Tbh the Express-like syntax is pretty much universally used and it's very ergonomic too!

[LekoArts]

A status update since the last post and since the initial release of the RFC:

• The end-to-end testing suite is in a good state for the initial release and will help other adapter authors
• The trailingSlash option is applied to routes in the routesManifest now
• The adapt function now receives trailingSlash and pathPrefix
• We've added an additional config hook that an adapter can use. This way the adapter can pass information back to Gatsby. Initially, we'll use this for the excludeDatastoreFromEngineFunction option. When this is enabled, Gatsby will place the LMDB database file into the public folder (with a randomized hash as prefix) and will download the file from that path (so CDN) when the engine is run. This way the LMDB database is not bundled with the serverless function. This can help when the LMDB file is really big and bigger than the allowed size of the serverless function
• SSR/DSG functions now look for 400.html/500.html error pages and serve those when available
• Improve docs all around

Follow the milestone for more details: https://github.com/gatsbyjs/gatsby/milestone/16

We've published a new canary version, you can delete your lock files and install the new version of alpha-adapters

[Talaxy009]

Hi, I was wondering if it is possible to deploy processed Gatsby functions to Netlify manually (with Netlify CLI)? The processed function in .netlify/functions-internal looks very different from the normal Netlify function.

[safesitefacilitiesdev]

It seems to me that at the moment gatsby-adapter-netlify does not support Image CDN, i.e. creating a function when GATSBY_CLOUD_IMAGE_CDN = true. It also seems that it does not support the config option trailingSlash: 'never'. These two issues are stopping us upgrading to Gatsby >5.11.0.