Could we migrate the all newsletters page from Frontend to DCR?

[dblatcher]

Could we migrate the all newsletters page from Frontend to DCR?

The "all newsletters" page is currently rendered by Frontend as a static page. The newsletters team have a new design to implement next quarter and rendering through Frontend in the same way is unlikely to be maintainable. Our preference would be to render the new version of the page through DCR so we can:

• Keep common page elements the Nav and Footer consistent with the DCR-rendered pages
• Make use of DCR components and patterns for a common developer experience (EG Cards, islands)
• Benefit from the existing architecture and maintain performance
• Not have to build our own siloed codebase
• Eventually retire a lot of legacy frontend code used to render the page now

DCR doesn't currently have a pattern for rendering "stand alone" pages (IE pages which are neither articles or fronts) so we are looking to get some input if doing this is feasible and how to approach it.

Optimistic, over-simplified plan

Currently, DCR express App renders Article pages and Fronts pages (in development) using renderFront and renderArticle functions (which use articleToHtml and frontToHtml). Both recieve data from frontend, of the types FEArticleType and FEFrontType, which act as page models.

To follow this pattern, and re-use much of the utility code used for Articles and Fronts, we could:

• define a NewslettersPageModel type, with config, nav, footer etc data included in the FEArticleType types
• add extra route to the DCR express app, using a renderNewslettersPage function, that would use a newslettersToHtml function
• create the components to generate the main part of the page and wrap them with the existing pageTemplate using the NewslettersPageModel as a prop
• update the controller in frontend to construct the NewslettersPageModel as a json payload and post it to the new route and render the returned HTML instead of constructing the page with Scala.

Suggested page model type definition

type BasePageModel = {
  webTitle: string;
  description?: string;
  canonicalUrl?: string;
  sectionName?: string;
  format?: CAPIFormat;
  editionId: EditionId;
  tags?: TagType[];
  renderAds?: boolean;
  subscribeUrl: string;
  contributionsServiceUrl: string;
  beaconURL: string;
  twitterData?: {
    [key: string]: string;
  };
  openGraphData?: {
    [key: string]: string;
  };
  config: ConfigType;
  nav: NavType;
  footer: FooterType;
};

type NewslettersPageModel = BasePageModel & {
  newsletters: Newsletter[];
};

Current Newsletters page flow

link

sequenceDiagram
    participant Client
    participant Frontend
    participant Newsletters API

    Frontend-->>Newsletters API: request Newsletters data
    Newsletters API-->>Frontend: return Newsletters data
    note right of Frontend: updated hourly
    Frontend->>Frontend: Cache Newsletters data
    Client->>Frontend: Request[/email-newsletters]
    Frontend->>Frontend: Get Newsletters data from cache
    Frontend->>Frontend: Compile static HTML page from multiple scala files
    Frontend->>Client: Response[HTML]

Proposed Newsletters page flow

link

sequenceDiagram
    participant Client
    participant Frontend
    participant Newsletters API
    participant DCR
    Frontend-->>Newsletters API: request Newsletters data
    Newsletters API-->>Frontend: return Newsletters data
    note right of Frontend: updated hourly
    Frontend->>Frontend: Cache Newsletters data
    Client->>Frontend: Request[/email-newsletters]
    Frontend->>Frontend: Get Newsletters data from cache
    Frontend->>DCR: post Newsletters page data
    rect rgb(230, 230, 230)
    note left of DCR: [new server function]
    DCR->>DCR: Render Page Contents to string
    note right of DCR: [new Component(s)]
    note right of DCR: includes Nav and Footer
        rect rgb(200, 200, 200)
            DCR->>DCR: extract CSS and expedited islands
            DCR->>DCR: build required scripts
            DCR->>DCR: make WindowGuardian data
            DCR->>DCR: wrap page contents, scripts etc in pageTemplate
        end
    end
    DCR->>Frontend: return HTML string
    Frontend->>Client: Response[HTML]

Refactor opportunities / offshoot idea

If we keep the data and functionality that is specific to the newsletters page separate from the general page information, it could acts a template for similar 'stand alone' pages. A generic render to HTML function might have a structure like this:

pageModelToHtml<T extends BasePageModel> = (model: T, MainContentComponent:{(T):ReactNode}): string => {
	const key = 'dcr';
	const cache = createCache({ key });
	const html = renderToString(
		<CacheProvider value={cache}>
			<StandAlonePage {...model}>
				<MainContentComponent />
			</StandAlonePage>
		</CacheProvider>,
	);

	return populatePageTemplate(model, html, cache);
};

From this discussion there was a ticket to migrate the existing https://www.theguardian.com/help/accessibility-help page to DCR. If this is still required, it could potentially build off the Newsletters page work.

The populatePageTemplate function would replicate many of the steps within articleToHtml (see the steps in the dark box in the diagrams below). Potentially, articleToHtml and frontToHtml could be refactored to use this 'page type agnostic' function and reduce duplication.

DCR article page flow (simplified)

sequenceDiagram
    participant Client
    Client->>Frontend: Request[article page URL]
    participant Frontend
    participant CAPI
    participant DCR
    Frontend->>CAPI: request article data
    CAPI->>Frontend: return CAPI data
    Frontend->>Frontend: CAPI data to DCR format
    Frontend->>DCR: post DCR article data
    DCR->>DCR: enhanceCAPIType
    rect rgb(230, 230, 230)
    note left of DCR: articleToHtml
    DCR->>DCR: Render Page Contents to string
    note right of DCR: ArticlePage component
    note right of DCR: includes Nav and Footer
        rect rgb(200, 200, 200)
            DCR->>DCR: extract CSS and expedited islands
            DCR->>DCR: build required scripts
            DCR->>DCR: make WindowGuardian data 
            DCR->>DCR: wrap page contents, scripts etc in pageTemplate
        end
    end
    DCR->>Frontend: return HTML string
    Frontend->>Client: Response[HTML]

DCR article fronts page flow (simplified, in development)

sequenceDiagram
    participant Client
    Client->>Frontend: Request[article fronts URL]
    participant Frontend
    participant Facia
    participant DCR
    Frontend->>Facia: request front data
    Facia->>Frontend: return front data
    Frontend->>DCR: post DCR front data
    DCR->>DCR: enhanceFront
    rect rgb(230, 230, 230)
    note left of DCR: frontToHtml
    DCR->>DCR: Render Page Contents to string
    note right of DCR: FrontPage component
    note right of DCR: includes Nav and Footer
        rect rgb(200, 200, 200)
            DCR->>DCR: extract CSS and expedited islands
            DCR->>DCR: build required scripts
            DCR->>DCR: make WindowGuardian data 
            DCR->>DCR: wrap page contents, scripts etc in pageTemplate
        end
    end
    DCR->>Frontend: return HTML string
    Frontend->>Client: Response[HTML]

[dblatcher]

I put together a "proof of concept" branch over the last 10% day, which might illustrate what I have in mind a bit better: https://github.com/guardian/dotcom-rendering/tree/dblatcher/stand-alone-pages

[mxdvl]

A specific new route/page is proposed in #6716

[jamesgorrie]

This definitely has raised some interesting questions around adding new routes to DCR.

We'll close this discussion, but should probably open up a discussion on how we do this in a simpler way in future.

Most of the implementation can be seen in this PR and it's corresponding frontend PR.