How to compile an MDX string to render later in React (server and client)?

[lunelson]

I'm looking for help understanding the usage of @mdx-js/mdx and @mdx-js/react, for rendering/hydrating a React component tree out of arbitrary MDX strings. Also, I'm looking for a separation between build-time and runtime, so this is not a case for mdx-js/runtime.

My use-case works like this:

1. in data fetching code, get a string of MDX from a CMS. This MDX may or may not reference custom elements (shortcodes). It won't contain any import statements. Compile that string, applying remark and rehype plugins if given, and return a JSON blob.
2. in React code, render said blob to HTML, applying components dynamically to support any custom elements which may have been used, or to remap standard ones. This must work in both server and client environments.

Important to note above is the components mapping: it only happens in the second (rendering) step. This is because my data-fetching step does not know about the consumer-components, which may want to use the data. Those consumers need to be able to remap the components option (both standard and custom), as they see fit. I considered a package called next-mdx-remote but it requires the components mapping at both compile and render steps.

I have made some progress on how to write the compiler function, but I cannot get my head around the render process. Virtually all packages and examples I've found seem to assume you are loading MDX content as a component from an .mdx file, and I can't figure out how to apply those patterns, particularly in the creation and provision of contexts and scope, and in what parts need to be added or removed from the compiled JSX code.

Can anyone help with this? 🙏 This is about as far as I got with the compiler...

const mdx = require('@mdx-js/mdx');
const { transformAsync } = require('@babel/core');
const presetEnv = require('@babel/preset-env');
const presetReact = require('@babel/preset-react');
const pluginRemoveExport = require('babel-plugin-remove-export-keywords');

const mdxHeader = `
import React from 'react'
import {mdx} from '@mdx-js/react'
`;

const mdxOptions = {
  remarkPlugins: [],
  rehypePlugins: [],
  skipExport: true,
};

const envOptions = { targets: { node: 'current' }, modules: false };
const babelOptions = {
  presets: [[presetEnv, envOptions], presetReact],
  plugins: [pluginRemoveExport, pluginMdxBrowser],
};

async function mdxCompile(mdxSrc) {
  const mdxJsx = await mdx(mdxSrc, mdxOptions);
  const { code } = await transformAsync(`${mdxHeader}\n${mdxJsx}`.trim(), babelOptions);
  return code;
}

[ChristianMurphy]

Taking a step back.
This doesn't really seem to be an MDX question, MDX compiles JS/JSX, which is what your example does.
The question seems to be more "How do I dynamically load React components?", or even more broadly "How do I dynamically load JavaScript?".
Both of which are outside the scope of MDX, and generally would be handled with EcmaScript Dynamic Modules, or another loader like CJS, AMD, SystemJS, etc.

---

in data fetching code, get a string of MDX from a CMS. This MDX may or may not reference custom elements (shortcodes). It won't contain any import statements. Compile that string, applying remark and rehype plugins if given, and return a JSON blob.
in React code, render said blob to HTML

I'm not sure I understand this.
What do you want the JSON blob to be?
A react component? Wouldn't that be JavaScript?

I have made some progress on how to write the compiler function, but I cannot get my head around the render process. Virtually all packages and examples I've found seem to assume you are loading MDX content as a component from an .mdx file, and I can't figure out how to apply those patterns, particularly in the creation and provision of contexts and scope, and in what parts need to be added or removed from the compiled JSX code.

The return value of

async function mdxCompile(mdxSrc) {
  const mdxJsx = await mdx(mdxSrc, mdxOptions);
  const { code } = await transformAsync(`${mdxHeader}\n${mdxJsx}`.trim(), babelOptions);
  return code;
}

is the JavaScript for a React component.
Once it is loaded/executed, it can be passed context and props as any other component would be.

[lunelson]

@ChristianMurphy thanks for answering! Let me clarify: the part that I was struggling with basically boils down to the use of a function constructor (new Function(...)—I had to read up on this) in the right combination with removing lines from the JSX (e.g. the export instructions), and adding lines to the babel-transformed code (return React.createElement(...)), in order to take transformed code and render it in React as if it were a functional component, rather than as an imported module. Most solutions in the wild assume the latter: you are importing MDX content from .mdx files. The output of the compilation pattern you show above is aligned with that by default (it exports the component). Also, the transformed code is full of scope references that you have to inject, and it's not obvious how to do this.

My solution now after looking more closely at how the function constructor works, looks like this. This is a single Next.js page component file, exporting a Page component and a getStaticProps function for data-fetching. The latter is where I would get my uncompiled MDX from my CMS and compile it; it's then received by the page component in the props, where I can dynamically determine how elements will render. Does this look right to you?

import React from 'react';
import { MDXProvider, mdx } from '@mdx-js/react';

import styles from '../styles/Page.module.css';

const mdx2jsx = require('@mdx-js/mdx');
const babel = require('@babel/core');
const presetEnv = require('@babel/preset-env');
const presetReact = require('@babel/preset-react');

const mdxOptions = {
  remarkPlugins: [],
  rehypePlugins: [],
  skipExport: true,
};

const babelOptions = {
  presets: [[presetEnv, { targets: { node: 'current' } }], presetReact],
};

async function mdxCompile(mdxSrc) {
  const jsxSrc = await mdx2jsx(mdxSrc, mdxOptions);
  const { code } = await babel.transformAsync(jsxSrc, babelOptions);
  return code;
}

function MDXRender({ code, components = {}, scope = {}, ...props }) {
  const fnBodyScope = {
    ...scope,
    React,
    mdx,
    MDXProvider,
    components,
    props,
  };
  const fnArgKeys = Object.keys(fnBodyScope);
  const fnArgValues = Object.values(fnBodyScope);
  // eslint-disable-next-line no-new-func
  const fn = new Function(
    ...fnArgKeys,
    `${code}\n\nreturn React.createElement( MDXProvider, {components}, React.createElement(MDXContent, props) )`,
  );
  return fn(...fnArgValues);
}

const testComponents = {
  h1({ children }) {
    return <h1 style={{ background: 'red' }}>{children}</h1>;
  },
  FooBar({ children }) {
    return <p style={{ background: 'green', color: 'yellow' }}>{children}</p>;
  },
};

/*
  this mocks my rendering stage
 */
export default function Page(props) {
  return (
    <MDXProvider components={testComponents}>
      <div className={styles.container}>
        <MDXRender code={props.testFn} />
      </div>
    </MDXProvider>
  );
}

/* 
  this mocks my data fetching stage
 */
export async function getStaticProps() {
  const testFn = await mdxCompile(
    `

# this is an h1

<FooBar>this is inside a custom component</FooBar>

this is a paragraph

  `,
  );
  return {
    props: {
      testFn,
    },
  };
}

[ChristianMurphy]

Also, the transformed code is full of scope references that you have to inject, and it's not obvious how to do this.

#1423 discusses this more in depth, there's also a PR in review which could simplify this #1425

The latter is where I would get my uncompiled MDX from my CMS and compile it; it's then received by the page component in the props, where I can dynamically determine how elements will render.

Since you want to set the components to render on the client side, the determination of how to dynamically render would also need to be on the client side, right?

[lunelson]

Since you want to set the components to render on the client side, the determination of how to dynamically render would also need to be on the client side, right?

Yes that's my intention. EDIT: not necessarily client-side. Server and Client should render the same thing. What I'm finding is that popular frameworks using MDX such as Gatsby and Next are following the paradigm of querying data at the "page" level (export query in Gatsby, export function getStaticProps in Next); and it's OK to fetch data at that level but I don't want to have to determine how it will be rendered yet, rather I want to pass it on to modular components (think "sections" of a given page) which an express that MDX however they want. This seems to be the power of extending a CMS with MDX-formatted content, is that it provides generic ways of representing structures which can be rendered in various specific ways, and I'd like to defer that decision to the component level.

And yes I saw that discussion in #1425 and while I can't tell if those changes impact what I'm describing, but I hope this possibility will be preserved and I think that maybe @ChristopherBiscardi is making a similar point

[ChristianMurphy]

while I can't tell if those changes impact what I'm describing, but I hope this possibility will be preserved

It is, through props and context.

I can't tell if those changes impact what I'm describing

It does, by reducing the number of variable injections, and leveraging props and context, which should work well in a fragment type situation.