add support for dynamic vector tiles rendering #3709

[stepankuzmin]

Hey everyone! This PR adds the equivalent of GeoJSON Source setData to the Vector Sources, so a user can change sources without updating the whole map style.

Usage:

map.addSource('example', {
  type: 'vector',
  url: 'mapbox://stepankuzmin.cjtd6d7i8107f3vqtku9o5jqn-7xl1l'
});

const source = map.getSource('example');
source.setSourceProperty('url', 'mapbox://stepankuzmin.cjtd6f5jh2eyk2xqtsyuo1kbz-3w7xz');

What do you think about it?

I'm not sure how to properly reload cached tiles, maybe someone can point me a direction for this?

• briefly describe the changes in this PR
• write tests for all new functionality
• document any changes to public APIs
• manually test the debug page

[andrewharvey]

Hey @stepankuzmin I haven't looked in detail, but...

Does this just replace a source's definition, ie. it's TileJSON?

ie. you can't say, change the geometry of this feature or change this feature's attributes?

If that's the case, I'd prefer not calling it setData as it's a bit different to a GeoJSON setData.

Just thinking out loud, is it better to have a more general setSourceProperty? Similar to setLayoutProperty and setPaintProperty to change any source values.

[stepankuzmin]

Hey @andrewharvey! Yes, setData actually allows to redefine TileJSON endpoint, so I like the setSourceProperty approach more. I'll update the PR accordingly.

[stepankuzmin]

I've changed setData to a more general setSourceProperty:

source.setSourceProperty('url', 'mapbox://stepankuzmin.cjtd6f5jh2eyk2xqtsyuo1kbz-3w7xz');

source.setSourceProperty('tiles', [
  "mapbox://tiles/stepankuzmin.cjtd6f5jh2eyk2xqtsyuo1kbz-3w7xz/{z}/{x}/{y}.vector.pbf",
  "mapbox://tiles/stepankuzmin.cjtd6f5jh2eyk2xqtsyuo1kbz-3w7xz/{z}/{x}/{y}.vector.pbf"
]);

[stepankuzmin]

I've added tiles cache clearing on setSourceProperty:

[Akiyamka]

Great feature. Without this for a simple update of tile url will cause
sequential call:

map.addSource(<newSourceID>, {
  type: 'vector',
  url: <newTileURL>
});

// For each layer connected with old source
map.setLayoutProperty(<layerUsedOldSource>, 'source', <newSourceID> });

map.removeSource(<oldSourceID>);

[aMoniker]

This will be great for our app, which changes source url parameters based on which filters a user chooses.

When replacing a source and clearing the cached tiles, will this prevent the flicker that currently happens when using setStyle to make a source change? It seems like that is what's happening in the demo gif.

[stepankuzmin]

Hi, @aMoniker! Yes, this PR prevents flickering when updating sources.

[andrewharvey]

I think it would be nicer if we could avoid the duplication of code between here and the constructor.

[stepankuzmin]

I've moved options setter into _updateOptions method

[asheemmamoowala]

@stepankuzmin Thanks for putting together this concept.

The current implementation of setSourceProperty leaves a bit unclear how

• setSourceProperty can be used to override properties (like min/maxzoom) of a source after it is set using a URL. The call to load would re-fetch the tilejson and write over those properties.
• load() is asynchronous, but this.options would already have been updated with new values and referenced by SourceCache#update before the async load is complete.

This type of change is better suited to be handled via #4875

[asheemmamoowala]

The load() method should fire the metadata and content events that trigger a SourceCache#reload() and clears tiles when the new tilejson is available. See:

mapbox-gl-js/src/source/source_cache.js

Lines 69 to 83 in 4eb7d4e

	this.on('data', (e) => {
	// this._sourceLoaded signifies that the TileJSON is loaded if applicable.
	// if the source type does not come with a TileJSON, the flag signifies the
	// source data has loaded (i.e geojson has been tiled on the worker and is ready)
	if (e.dataType === 'source' && e.sourceDataType === 'metadata') this._sourceLoaded = true;
	// for sources with mutable data, this event fires when the underlying data
	// to a source is changed. (i.e. GeoJSONSource#setData and ImageSource#serCoordinates)
	if (this._sourceLoaded && !this._paused && e.dataType === "source" && e.sourceDataType === 'content') {
	this.reload();
	if (this.transform) {
	this.update(this.transform);
	}
	}
	});

Is there a reason this additional call to clearTiles is needed here?

[stepankuzmin]

SourceCache#reload() doesn't remove existing tiles. It causes that after changing the source URL, you still have cached tiles from previous source URL, because they are not in errored state (see https://github.com/mapbox/mapbox-gl-js/blob/master/src/source/source_cache.js#L225).

So I call clearTiles to force remove previously cached tiles.

[stepankuzmin]

We can force errored state thought, but it looks not so good as clearing tiles with clearTiles

[stepankuzmin]

@stepankuzmin Thanks for putting together this concept.

The current implementation of setSourceProperty leaves a bit unclear how

• setSourceProperty can be used to override properties (like min/maxzoom) of a source after it is set using a URL. The call to load would re-fetch the tilejson and write over those properties.
• load() is asynchronous, but this.options would already have been updated with new values and referenced by SourceCache#update before the async load is complete.

This type of change is better suited to be handled via #4875

I think about passing an optional callback in load() method, so we can use it to update properties after TileJSON is loaded. Something like this:

setSourceProperty(name: string, value: mixed) {
    if (this._tileJSONRequest) {
        this._tileJSONRequest.cancel();
    }

    const sourceCache = this.map.style.sourceCaches[this.id];
    sourceCache.clearTiles();

    this.load(() => {
        const options = { [name]: value};
        this._updateOptions(options);
    });
}

[robinsummerhill]

This will be an absolute game-changer for us if this gets merged. We have a platform that uses 'parameterized' vector tile URLs to apply time-based filtering etc. To be able to change source URLs without producing the flicker associated with a full map repaint will be amazing.

[arindam1993]

@stepankuzmin , thanks a lot for this! I'm closing this for now since there hasn't been any activity for a month or so. Feel free to repoen it once you have more progress on it.

[andrewharvey]

@stepankuzmin , thanks a lot for this! I'm closing this for now since there hasn't been any activity for a month or so. Feel free to repoen it once you have more progress on it.

@arindam1993 I think it's better left open (even as a draft if not completely ready yet) so it can continue to be worked on in a collective way. I'll try to help review if that's what it needs.

[ansis]

@stepankuzmin thanks for working on this! Let me know if you would like help pushing any of these last things over the finish line

I think there are a couple tasks remaining before we can merge this:

• setting any property currently reloads the tilejson which can can overwrite the just-set property
• both this method and VectorTileSource need to be documented (I'm not sure why we haven't documented VectorTileSource yet...)
• settle on naming

naming

It looks like existing methods to update source properties are one method per property. For example, ImageSource#setCoordinates(...) and GeoJSONSource#setData(...). Should we follow this convention and have a VectorTileSource#setUrl(...)? I think this convention could be broken, but if we aren't sure that breaking it would be better I'd default to keeping with the existing convetion

[stepankuzmin]

Hey @ansis!

Thanks for the clarification. I agree with one method per property and I'll get back to this issue soon.

[stevage]

I've just found this issue as it's likely to be useful for a client I'm working with. The use case there is slightly different: updating server-side data, and wanting to force a tile refresh, without a change in URL. (Although likely to be implemented with a slightly different URL [ ?update=1 etc] to defeat caching.)

It looks like existing methods to update source properties are one method per property. For example, ImageSource#setCoordinates(...) and GeoJSONSource#setData(...). Should we follow this convention and have a VectorTileSource#setUrl(...)? I think this convention could be broken, but if we aren't sure that breaking it would be better I'd default to keeping with the existing convetion

It looks like all the interest here is really just about setUrl() and setTiles(). Is there really any call for setScheme(), setTileSize(), setMinzoom() etc? If not, I'd suggest only implementing those two high-value methods, to avoid API bloat?

[stepankuzmin]

Hi everyone! Sorry for the late response. I've renamed setURL to setUrl, added unit tests and docs for this. Is there any chance for this to get merged?

/cc @ansis @arindam1993 @asheemmamoowala

[ogix]

Are you guys still alive?

[kukiel]

@stepankuzmin that's a great addition I must say. Would be nice to also support raster source with the same functionality.

[andrewharvey]

Any updates on this PR? We also need this feature because all our layers are time-enabled.

@ogix Just an aside that this shouldn't be needed for your use case (as I've understood it), GL JS will automatically re-request tiles which have expired based on the maxage cache header you set, check it out with the traffic-day-v2 style.

[andrewharvey]

I gave this a test drive, I couldn't find any issues, seems to work as intended.

[mourner]

Thanks everyone for the help landing this, and apologies for the delay!

[underbluewaters]

Having a setTiles() method available for raster sources would be very helpful. I'd be willing to put together a PR if I could get a couple pointers. My use case is loading WMS services with support for toggling layers. Completely replacing the source gets tricky when there are referencing layers. More importantly, toggling additional layers causes all tiles to disappear before loading new ones which makes for a pretty awful UI compared to other tools.

I've found that following the same steps of calling clearTiles() on the source cache causes old tiles to disappear before the new ones are loaded. What I would like is for them to only be replaced when the new tile is ready (much like updating the url on an ImageSource). Calling reload() on the source cache gives me the behavior I want. Does that seem like a reasonable approach?

[jonseppanen]

Hi @stepankuzmin, I noticed in comment #8048 (comment) you mention this new feature prevents tile flickering.

When reloading a tile source using setTiles, the tiles vector layer is still flickering for us. Is this the way we should be using it to avoid flickering? I can't seem to find any other method.

Using latest mapbox 2.x.

[stepankuzmin]

Hi @jonseppanen

Can you, please, provide some reproducible example for this?

[robinsummerhill]

Hi @stepankuzmin. I'm also seeing flickering like @jonseppanen above.

You can see a minimal example here: https://codepen.io/rsummerhill/pen/dyOGXpQ?editors=0010
(this is using a tilejson source from our own GIS system)

From looking at the source code I'm guessing that features are removed from the map when sourceCache.clearTiles() is called in setSourceProperty and are not replaced until new tiles are loaded. This leads to flicker because there is a split second between removing the old tiles and loading the new ones.

[stevage]

FWIW, I have found this to be an effective and flicker-free way of refreshing tiles:

map.style.sourceCaches['mylayer'].clearTiles();
map.style.sourceCaches['mylayer'].update(map.transform);

[mrozema]

Is there a solution for the flickering when calling setTile? The minimal example from @robinsummerhill is exactly what I'm observing. Using latest mapbox-gl (2.15.0). None of the workarounds seem to work for me.