v1.8.0

Sidebar and pager improvements

Sidebar sections can now have header links, which is useful for sections with main pages. Clicking on the section will visit the section’s main page while opening the section in the sidebar to show related pages. To give a section a header link, set the path option for a given section. See the pages config option for more.

export default {
  pages: [
    {
      name: "Section 1",
      path: "/s01/",
      open: false,
      pages: [
        {name: "Page 1.1", path: "/s01/page1"}
      ]
    },
    {
      name: "Section 2",
      path: "/s02/",
      open: false,
      pages: [
        {name: "Page 2.1", path: "/s02/page1"},
        {name: "Page 2.2", path: "/s02/page2"}
      ]
    }
  ]
};

The sidebar now supports links to external pages, which automatically open in a new window. (We use this feature in the Framework docs to link to examples.) The sidebar now renders sections correctly when there are no top-level pages.

Sidebar pages and sections can now use the pager option to define separate sequences for the next & previous links in the footer, or to opt-out of the pager for specific pages or sections. (We use this feature in the Framework docs to have separate sequences for inputs, and to suppress the pager on library pages.) The pager option can also be defined in a page’s front matter. If a page sets the pager front matter option to false or null, the pager is hidden.

Other improvements

The head, header, and footer config options can now be specified as functions, allowing dynamic footers. These functions are passed an object with the page’s title, (front-matter) data, and path, and must return a string.

We’ve improved how data loaders are spawned, fixing a FATAL Error: fsync failed! error with DuckDB data loaders that output directly to stdout. We also fixed the normalization of plain HTML links such as <a href="/test.html">hello</a>; these are now correctly converted to relative links.

New examples! 🎉

We’ve been hard at work developing new examples.

• geotiff - Parsing GeoTIFF with geotiff.js, then visualizing with Observable Plot
• netcdf - Parsing NetCDF with netcdfjs, then visualizing with Observable Plot
• vega-dark - Responsive dark mode in Vega-Lite
• vega-responsive - Responsive width in Vega-Lite using ResizeObserver
• loader-arrow - Generating Apache Arrow IPC files
• loader-databricks - Loading data from Databricks
• loader-duckdb - Processing data with DuckDB
• loader-github - Loading data from GitHub
• loader-google-analytics - Loading data from Google Analytics
• loader-parquet - Generating Apache Parquet files
• loader-postgres - Loading data from PostgreSQL
• loader-snowflake - Loading data from Snowflake
• netcdf-contours - Converting NetCDF to GeoJSON with netcdfjs and d3-geo-voronoi
• custom-input-2d - A custom 2D input with bidirectional binding
• markdown-it-container - The markdown-it-container plugin
• markdown-it-footnote - The markdown-it-footnote plugin
• markdown-it-wikilinks - The markdown-it-wikilinks plugin
• custom-stylesheet - Defining a custom stylesheet (custom theme)
• intersection-observer - Scrollytelling with IntersectionObserver
• responsive-iframe - Adjust the height of an embedded iframe to fit its content

See all examples here: https://github.com/observablehq/framework/tree/main/examples

Please let us know which examples you’d like to see.

Full Changelog: v1.7.1...v1.8.0

v1.7.1

Page stats

The build command now outputs summary page size statistics, including the weight of imported JavaScript modules and referenced files such as data. This helps you keep an eye on performance, encouraging you to reduce bloat by removing unnecessary imports or by optimizing data loaders to produce smaller files.

Bug fixes

Prevent the sidebar from closing on narrow windows when the search input is focused via Command-K and then a link is clicked. Avoid a rendering glitch in Safari with animated loading indicators ↻. Downgrade JSDOM to ^23.2.0 to avoid a regression (not a valid selector error) in the selectors implementation. Automatically link H1, H2, H3, and H4 elements that have an id attribute, enabling them to also appear in the table of contents. Fix table of contents highlighting when the heading id contains non-ASCII characters.

Watch changes to static assets during preview, such as referenced stylesheets and images, so that changes to these files update automatically. Also populate FileAttachment.lastModified on update during preview.

When deploying to Observable, correctly update titles when linking to existing projects.

And more…

The resize function now implicitly awaits promises returned by the render function; the render function can also now return null to clear the container. Deprecate the scripts config option.

Full Changelog: v1.7.0...v1.7.1

v1.7.0

The default source root is now src instead of docs. For backwards compatibility, we’ll continue to default to docs if it exists, but we now recommend that you specify the root config option.

We’ve added a few new markdown-it config options. The typographer option enables simple typographic replacements, such as smart quotes (replacing straight quotes with curly quotes based on context). You can customize the quotes using the quotes option. And the linkify option lets you turn off automatic linking of URL-like text in Markdown.

We’ve improved Node import compatibility by supporting packages that require or import JSON, such as arquero. We also now better handle certain CommonJS packages (using requireReturnsDefault) and certain transitive imports. We now use the latest version of npm:parquet-wasm if no version range is specified, rather than defaulting to 0.5.0; thank you to @kylebarron for fixing parquet-wasm’s conditional exports in 0.6.0 to improve compatibility.

Deploying to Observable is now significantly faster when only some files have changed: rather than uploading every file, we now only upload files that have changed. We’ve also improved the error message when a file is too big or the file quota has been exceeded. We removed support for the deprecated deploy config option; the deploy configuration should instead live in .observablehq/deploy.json within the source root.

Lastly, we fixed the scripts config option to correctly handle resolution of local modules; we also now recommend that you use the head config option instead.

Thanks to @scresawn, @declann, and @russbiggs for contributing to the docs!

Full Changelog: v1.6.0...v1.7.0

v1.6.0

Node imports 📦

You can now import from node_modules! This lets you manage dependencies with a package manager such as npm or Yarn, import private packages from the npm registry, or import from a different package registry such as GitHub. For example, to use canvas-confetti, first install it (here using npm):

npm install canvas-confetti

Then import it:

import confetti from "canvas-confetti";

You can import named exports or import as a namespace, too. For example, here’s how you might import Apache Arrow:

import * as Arrow from "apache-arrow";

You can import specific entry points by adding the entry point subpath after the package name. For example, to import mime’s lite entry point:

import mime from "mime/lite";

Note: not all Node packages are usable in the browser; Node imports are only supported for modules that do not rely on Node-specific APIs and that can be converted to ES modules via esbuild. If you have difficulty importing a module, please ask for help by opening a discussion.

We also made some improvements to npm: imports: entry points without a file extension, such as npm:mime/lite, now implicitly get /+esm added to generate an ES module; an /+esm import no longer collides with a non-/+esm import of the same entry point; and undeclared dependencies are now correctly resolved, fixing certain imports.

More documentation 📚

We’ve overhauled and improved our documentation. Please take a look and let us know what you think. Thank you to @CAYdenberg for the thoughtful feedback, and @Stroked for helping to improve our wording. We also fixed a few links, added deep links to Observable where needed, added an interactive version of the mortgage rates dashboard, and clarified how to install markdown-it plugins.

And check out this new deck.gl example visualizing more than 100,000 reported collisions on roads in Great Britain for the year 2022:

It’s written using a data loader (dft-road-collisions.csv.sh) that fetches almost 9M records from the Department for Transport and filters them using DuckDB, reducing the data from 1.2GB to 2.1MB. Here’s what it looks like if you display the entire dataset:

This data is a strong proxy for population density (and more specifically traffic density), so the major highways are strikingly visible.

Deploy improvements 🚀

We fixed a bug where observable deploy could deploy a stale build if you recently modified your source files before deploying. Now observable deploy will always ask if you want to build. For example, it might say:

◆  You built this project 6 minutes ago. Would you like to build again before deploying?
│  ● Yes, build and then deploy / ○ No, deploy as is

If you’ve not yet built your project, you will see:

◆  No build files found. Do you want to build the project now?
│  ● Yes, build and then deploy / ○ No, cancel deploy

In non-interactive terminals, observable deploy will build automatically if the build is missing and otherwise deploy as-is. You can use the new --build flag to force a fresh build on deploy, or the --no-build flag to force a deploy of your existing build. These new flags replace the --if-stale and --if-missing flags. We also fixed a race condition that could cause deploying to Observable to fail.

Other improvements 🆕

Sidebar sections can now be marked as non-collapsible by setting collapsible: false. Use this for a cleaner sidebar if you always want the section to be open. For example:

export default {
  title: "Sample project",
  pages: [
    {name: "Page 1", path: "/page-1"},
    {name: "Page 2", path: "/page-2"},
    {
      name: "Sample section",
      collapsible: false, // always open!
      pages: [
        {name: "Item 1", path: "/section/item-1"},
        {name: "Item 2", path: "/section/item-2"}
      ]
    }
  ]
};

The new FileAttachment.dsv method allows you to load delimiter-separated files like CSV or TSV, but with an arbitrary delimiter. For example, to load semicolon-separated values:

const capitals = FileAttachment("us-state-capitals.csv").dsv({delimiter: ";", typed: true});

The “empty” project template (which is used if you answer “No” to “Include sample files to help you get started?” during observable create) is now emptier than before, with no included sample data. observable create now sets a default favicon when creating a project, and survives errors that may occur during the initial build such as a network configuration problem.

There’s a new jQuery UI example. We fixed a bug where certain runtime errors would not be cleared in code blocks that don’t otherwise display. The head, header, and footer front matter options are now correctly suppressed when set to false.

Full Changelog: v1.5.1...v1.6.0

v1.5.1

Editing the project config (observablehq.config.js) now triggers an automatic reload over the preview socket. Changing the modification time (say via touch) of a data loader now immediately re-runs the data loader during preview. The preview server now correctly handles page names containing spaces, emoji, or other non-ASCII characters.

Full Changelog: v1.5.0...v1.5.1

v1.5.0

Live config

Changes to the project config (observablehq.config.js) now take effect immediately 🎉 on reload or navigation during preview, without needing to restart the preview server: the preview server now loads the latest config on request rather than preloading it on startup. Likewise, the default list of sidebar pages is now computed on request, and hence a server restart is no longer required when adding, removing, or renaming a page. We’ve also optimized parsing and caching to improve performance during preview.

The preview socket now automatically reconnects if the preview server restarts. No more broken sockets! 😌

External link treatment

External links how have target="_blank" and rel="noopener noreferrer" applied by default. This improves privacy by not sharing the (potentially sensitive) referrer when linking to a (potentially untrusted) external site. You can override this for a given link by setting the target and/or rel attribute explicitly.

Other improvements

The new FileAttachment.href property returns the absolute URL to the associated file. Use this simpler alternative to the async FileAttachment.url() method.

Safari 16 is now (again) supported. (We were unintentionally generating code with static initializer blocks which were only added in Safari 16.4.) Invalid Markdown front matter is now treated as page content, and logs a warning, instead of crashing. The default Mermaid theme is now consistent with the chosen page theme’s color scheme, rather than exclusively reflecting the user’s preferred color scheme (consistent with the built-in dark reactive variable). Static <script type="module"> are now only preloaded with <link rel="modulepreload"> if the script element does not have an async attribute (which we assume implies a lower-priority script, such as analytics).

The observable create command now uses the default path hello-framework instead of ./hello-framework, making it less confusing on Windows (where \ is the path delimiter instead of /). In addition, observable create now builds the project, ensuring instant initial page loads during preview since npm: imports and data loaders will already be cached. The observable logout command now logs a confirmation message on successful sign-out.

Full Changelog: v1.4.0...v1.5.0

v1.4.0

Improved SQL and DuckDB 🦆

SQL code blocks and DuckDBClient now support querying remotely-hosted data. This is especially useful for live, rapidly-changing data that you do not want to “bake-in” to your built site. For example, to load recent earthquakes from the USGS real-time feed:

---
sql
  quakes: https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.csv
---

To get the maximum recent earthquake magnitude, you could then query:

SELECT MAX(mag) FROM quakes

Note: for faster page loads and to avoid a runtime dependency on external servers, we recommend using data loaders when possible to generate static snapshots of data at build time, with continuous deployment to re-build your site as needed.

Framework also now supports DuckDB’s database file format. These files conveniently store multiple tables and may offer better performance on complex or repeated queries. DuckDB database files can either be static or generated dynamically by data loaders. For example, to load a database file named chinook.db:

---
sql
  chinook: chinook.db
---

Then, to query the tracks table:

SELECT * FROM chinook.tracks

The new DuckDBClient.sql convenience method lets you redefine the sql tagged template literal used by SQL code blocks. This allows you to load or create data sources dynamically, say in response to user interaction. For example, to allow the user to choose the desired USGS real-time feed from a drop-down menu:

const feed = view(Inputs.select(new Map([["M4.5+", "4.5"], ["M2.5+", "2.5"], ["All", "all"]]), {label: "Earthquake feed"}));

const sql = DuckDBClient.sql({quakes: `https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/${feed}_day.csv`});

SELECT * FROM quakes ORDER BY updated DESC

We also fixed a bug where DuckDB was being loaded during preview even when it was not needed, and fixed the implementation of db.describeTables and db.describeColumns now that db.query returns an Arrow table instead of a materialized array-of-objects.

More local assets

You can now reference local assets such as images, stylesheets, and scripts from the head, header, and footer config options. For example, to set the favicon to a favicon.png stored in the source root:

export default {
  head: `<link rel="icon" type="image/png" href="/favicon.png" sizes="512x512">`
};

Like file attachments, referenced assets can be either static or generated dynamically by data loaders, are included in the built site, and have a content hash applied to the file name for cache breaking.

The new FileAttachment.lastModified property exposes the modification time of the given file, represented as the number of milliseconds since UNIX epoch (like File.lastModified). This is useful for showing how fresh data is, for example. For files generated by data loaders, this stores the time the data loader completed.

FileAttachment("results.csv").lastModified // 1710809036000

new Date(FileAttachment("results.csv").lastModified) // "Tue Mar 19 2024 00:43:56 UTC"

Framework also now supports script elements with the src attribute in Markdown. This is especially useful for loading older, non-module JavaScript — it “just works”. We also fixed a bug resolving file attachment paths with a leading slash.

More flexible deploys

The build and deploy commands are now decoupled, offering more flexibility: you can modify a built site prior to deploying (say to add additional assets), and you can re-deploy a site without having to re-build it (say if your deploy gets interrupted). To support this, the deploy command has two additional flags, --if-stale and --if-missing, that indicate what to do when the build is either stale (more than five minutes old) or missing. Both default to prompt in interactive terminals, meaning the deploy command will ask how you’d like to proceed, and cancel in non-interactive terminals.

Our new Deploying guide shows how to automate deploys from GitHub Actions.

Deploys to Observable are also now significantly faster thanks to concurrent file uploads. 🚀

And more…

Self-hosted npm: modules have been renamed from +esm.js to _esm.js to avoid a quirk with how Amazon S3 interprets the + character in paths. The pages config option now properly normalizes page paths, fixing a bug where the previous & next footer links wouldn’t appear. The EIA example dashboard timeline chart now has better tooltips (thanks, @pettiross!).

Full Changelog: v1.3.0...v1.4.0

v1.3.0

Mosaic

Framework now has built-in support for Mosaic vgplot, an interactive grammar of graphics that helps you build coordinated views with fast interaction for millions of data points (or more). Framework’s Mosaic integration is powered by DuckDB-Wasm and Observable Plot. Data sources are configured using the sql front matter option — meaning you can query the same data using Mosaic and Framework’s SQL code blocks. See this interactive heatmap of one million New York City taxi rides for an example:

Speaking of DuckDB, our SQL code blocks and sql tagged template literal are now much faster thanks to an optimized DuckDBClient which uses connection.query instead of connection.send for non-streaming queries, and returns Apache Arrow tables rather than materializing array-of-objects. (You can use table.toArray or Array.from to materialize an array-of-objects if needed.)

All the interpreters

The new interpreters config option allows you to register additional interpreted languages for data loaders. 🎉 For example, to use Perl (.pl) and AppleScript (.scpt):

export default {
  interpreters: {
    ".pl": ["perl"],
    ".scpt": ["osascript"]
  }
};

We also now support Java (.java), Julia (.jl), and PHP (.php) data loaders by default.

Better links

The new cleanUrls config option, if set to false, preserves the .html extension on page links. This is useful for some static site servers that don’t support “clean” URLs. We’ve also enabled automatic “linkification” of links in Markdown such as https://observablehq.com and mailto:support@observablehq.com.

And more…

The new dark built-in reactive variable lets you more easily write code that adapts to light or dark mode. Our project templates now use a JavaScript config file by default, rather than TypeScript, to improve compatibility with Node 18. Deploying a project to Observable now prompts you for the desired access level rather than silently defaulting to private.

New Contributors

• @naltmann made their first contribution in #1004

Full Changelog: v1.2.0...v1.3.0

v1.2.0

SQL code blocks, powered by DuckDB 🦆

Framework now includes built-in support for client-side SQL powered by DuckDB. You can use the new SQL code blocks and sql tagged template literal to query data from CSV, TSV, JSON, Apache Arrow, and Apache Parquet files, which can either be static or generated by data loaders.

To use SQL, first register the desired tables in the page’s front matter using the sql option. Each key is a table name, and each value is the path to the corresponding data file. For example, to register a table named gaia from a Parquet file:

---
sql:
  gaia: ./lib/gaia-sample.parquet
---

To run SQL queries, create a SQL fenced code block (```sql). For example, to query the first 10 rows from the gaia table:

```sql
SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 10
```

SQL code blocks render results using Inputs.table:

To refer to the results of a query in JavaScript, use the id directive. (When an id is specified, the table is hidden by default, but you can display it using the display directive.) For example, to refer to the results of the previous query as top10:

```sql id=top10
SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 10
```

For dynamic or interactive queries that respond to user input, you can interpolate values into SQL queries using inline expressions ${…}. For example, to show the stars around a given brightness:

```js
const mag = view(Inputs.range([6, 20], {label: "Magnitude"}));
```
```sql
SELECT * FROM gaia WHERE phot_g_mean_mag BETWEEN ${mag - 0.1} AND ${mag + 0.1};
```

SQL fenced code blocks are shorthand for the sql tagged template literal. You can invoke the sql tagged template literal directly like so:

const rows = await sql`SELECT * FROM gaia LIMIT 10`;

For more, see the new SQL guide.

Other changes

Fix FileAttachment resolution in imported modules. Fix module preloads of transitive imports. Fix a race condition in display where stale values could be displayed after invalidation. Unpin the version of Apache ECharts (from 5.4.3; currently 5.5.0). Use files rather than built-in sample datasets for the default project templates.

New Contributors

• @elevenchars made their first contribution in #991

Full Changelog: v1.1.2...v1.2.0

v1.1.2

Fix yarn dlx @observablehq/create on Yarn Berry.

Full Changelog: v1.1.1...v1.1.2