Skip to content

Home

Jump to bottom
myfreeer edited this page Oct 3, 2023 · 8 revisions

Welcome to the mdn-local wiki!

Status

See project mdn yari for current status on supporting yari, the current platform powering mdn.

Most of mdn features excluding quick search, full-site search, multi-language switching, scroll to highlight on sidebar works with a modern web browser and double-clicking the index.html from a fully extracted release.

The packages from releases should work on 2 methods:

  1. Directly opened by a browser via file: protocol.
  2. Served in any subpath by any http or https server, accessed via http: or https: protocol.

This project is not yet published to npm registry.

Summary

mdn-local is a project focusing to download the all the accessible docs and examples on MDN Web Docs for a specified locale, and resources required for running and viewing this in local device, or local-area network, making it (mostly) work without any external network access (meaning the internet).

For locales other than en-US, en-US docs would be fetched for pages not localized (not found on mdn).

Quick start

Goto releases, download a package (not the source code), extract it to some path of your drive, double-click index.html and enjoy your docs.

Usage

mdn-local is based on website-scrap-engine, the main entry in mdn-downloader.ts. Below is an example of using it to download mdn docs:

import createDownloader from './mdn-downloader';

createDownloader({
  // The path to download docs to
  localRoot: '/home/user1000/developer.mozilla.org_20230409_zh-CN',
  meta: {
    // The locale of MDN docs to get
    locale: 'zh-CN',
    // Configure if http2 is enabled for http connection
    http2: false,
    // Configure if ipv6 is prefered on dns lookuping
    preferIpv6: true
  }
}).then(d => d.onIdle().then(() => d.dispose()))
  .catch(console.error);

History

This project starts back in early 2019, before the yari landed, when accessing mdn docs is quite slow even with the help of CDN.

We had spent a lot of time searching the internet with verious of search engines with results from stackoverflow for example, there was many existing work, including the official link to s3 with a huge tarball, which requires a lot of resources in other CDN domains, making it unable to work offline.

Some third-party solutions like DevDocs, dash, dochub existed, and kiwix added support in late 2020.

Back then, mdn docs is containing a lot of dead or broken links, and internal links ofter link across locales, it is hard to read even if using the online version, none of the solutions mentioned above resolved this in the 2019, that why the project begun.

It started firstly in pure javascript, good for quick code and debug, using callbacks for modifing the content of resources. The first release 0.0.1 came out at Sep 8, 2019, with a very limited subset of mdn docs, and became more complete at Dec 11, 2019, when release 0.0.6 came out.

It processes the link before it get requested to relative form, making it faster for the resource containing the link to be written to disk, reducing the usage of memory. Many links got redirected for many times in that time, so a placeholder is used to redirect the original page to the new page.

The project is firstly developed privately, but become open source at Oct 8, 2019.

At the beginning of this, mdn ui is based on jquery, when ui scripts are simple to be modified for local usage, but in maybe 2019 H2, it changed to react, and http requests like fetch is used, and contents were generated from window._react_data if scripting is enabled, but the whole page was still static, so the data could be generated from page so that links inside can be converted.

With the development of mdn-local, the project grows bigger, and javascript is getting harder to handle a project at that scale, so the refactoring begun at Jun 14, 2020, when project website-scrap-engine begun, which is a general-purpose rewrite of core part of this repo in typescript, based on the long-term experience of this project, where life cycles replaced the callbacks.

Life cycles are a set of functions, receving the links and resources as arguments, using the return value for processed result, making it easier to combine processes, and the new data structure, Resource, for all the info to be hold of resource (pages, css, js, sitemap, binary files) to be download, or downloaded.

The project was migrated to website-scrap-engine and typescript at Aug 10, 2020, branch v0.1 is kept for legacy code before this.

In 2020 Q4, mozilla annouced yari, the new platform of mdn, a lot has changed since then. Most of them are tracked on issue #179 and project #1.

The new frontend codebase is so hard to modify that we had to reimplement all the ui events and the rendering of the compatibility table. As a tech note, the new mdn frontend is like the history-based spa route, where wrong route lead to an error page, in which the root path / is used to detect route, but for local usage, the root path is not fixed and is hard to detect.

Why not clone yari

MDN has made yari open source, so evenyone can get a copy of its code and run it locally, but it does not fit into the offline case.

Complexity

Project yari needs a development environment set up and many external dependencies like the npm dependencies, en-US content, translated-content and possibily much more.

Also, since it's full-featured, a real backend service app would be needed, either kuma, or rumba, and databases, search engines.

So, you'll need a server, or a set of servers, and development and opeation skills to setup and run yari in your own environment, this takes many time.

External resources

MDN contains a large amount of external resources, including examples at mdn.github.io, images on *.mdn.mozit.cloud, scripts on public cdns, part of which in the yari source code, and others in pages, examples, and contents.

Clone this wiki locally