Gridedit: mega-scale feeder visualization and editing.

[dpinney]

Goal

Find a map that runs in a browser that (1) displays 1 million vector features (lines and polygons) and (2) is editable.

Most Promising Leads

• Polymaps with image tiles. Demo here. We've tried with progressively-loading vectors but we still have problems with single feeders with 100,000 features.
• Open Street Maps editor. Demo here. They show image tiles until you get very close, at which point they load vector features. Great performance for a billion (trillion?) feature dataset.

[cshjin]

Several examples:
scaleline algorithm: http://polymaps.org/ex/tiles.html and http://bl.ocks.org/hperantunes/1100163
d3 force group: http://bl.ocks.org/GerHobbelt/3071239
d3 cluster: http://bl.ocks.org/sundar/910355

performance comments: http://stackoverflow.com/questions/15596234/improving-d3s-performance?answertab=active#tab-top

[cshjin]

For responsiveness issue, here is a discussion, force.tick() may helps. Ref: d3/d3#419 (comment) and https://groups.google.com/forum/#!topic/d3-js/t27KdiwuDFA

[cshjin]

One example implemented using KineticJS, 100,000 nodes, 100,000 links: http://jsfiddle.net/jin_hw/Kgqzg/. The performance is not what we expected, if change size=1000, zoom and drag rendering works well. Another example inplemented using pure Canvas, 100,000 nodes, 100,000 links: http://jsfiddle.net/jin_hw/dXcbB/3/. This renders much better than previous one. Right now I only add onmousewheel event. So you can zoom in and out.
One more example: http://windhistory.com/ this is a similar example as we expected, it use tiler to choose specific nodes, and zoom in to show detailed wind direction. Here is a implementation notes: http://www.somebits.com/weblog/tech/wind-history-implementation.html

[dpinney]

Just looked at the implementation notes. For next steps, I think we need to implement something like wind history with 100,000 SVG elements. It's already got 20,000ish, so we're pretty close. And we get the mapping for free! What's not to love?

[cshjin]

1. For vector tiles, one example: http://bl.ocks.org/mbostock/900050, but it doesn't work, since type of libraries is blocked. So I manually store those libs locally, and modify "d3" to "d3 v3" (for the purpose of force layout, it isn't support by version 1.8, I stuck there before), and "polymaps 2.4" to "polymaps 2.5"(for mobile device touch event listener). But I cannot ignore the "d3.geo.js" even it may already defined in d3.v3.js, may not exactly the same.
   Nelson also put the main part of its windhistory code in https://gist.github.com/NelsonMinar/3589712 in function installGeoTiler() is what we need. And the function station() is what we can apply, how to apply tile to map projection.
2. The main part of generating tile is tiler.js. But actually I can not full understand its code. Fortunately, it works for other data.
3. One suggestion: can we add a separate json file to indicate the whole feeder general info, including the name, layoutVars, and modules such as climate, GeoLocation, size, etc?
   One reason of this is no independent nodes in the feeder graph, such as climate, the pure feeder json file will only store tree, nodes, links data, nothing else.
   Another reason of thinking this is we can load feeders what we need. By zoom in to a specific level, request the select data, such that we don't need to load all feeder data at once, no giga trees load to memory. The separate json will be viewed as index, we only provide the data use need, instead of all of them. However, it may also need some modifications of study and reports.
4. If you can access to the ip address of my desktop, a server loaded there and you can access 10.0.0.4:8000/scratch/tilesTest/ to see a demo.
   The demo shows to zoom in a level of 16 to see a ACEC feeder. Since I didn't implement layout force yet, only nodes can be viewed, no lines or zoom and pan feature. I hope it is on the right track.
   One thing should be point out, different feeder has different size, while the faked latitude and longitude is hard coded in geo information import from milsoft data (milToGridlab.py). Feeder geo size is stretched to 1200*800 (Bad implementation there). The purpose of these is to present the data as precise as possible. Or we can find another way to invert the geo location data of each component to a precise lat & lon location. T.J. may helps.

[cshjin]

Mapping example comments:

1. We can use our own data structure to projection to geo location. The main part of this is in "d3.geo.js" with function mercator(coordinates) coordinates[0] and coordinates[1] to be treat as latitude and longitude.
   In tiler.tile function, it will use tiler data to projection to a normalized coordinates in [0,1]
2. Each tuple in feeders.json has the structure as follows:
   "public_Olin Barre.json": {
   "attachments": { ...
   },
   "layoutVars": {
   "theta": "0.8",
   "gravity": "0.05",
   "friction": "0.9",
   "linkStrength": "7"
   },
   "name": "public_Olin Barre",
   "coordinates": [
   -74.84071486428803,
   38.50416358275365
   ],
   "size": 1714
   }
   We can add or delete what we need or not. But the coordinates part is related to tiler.js and d3.geo.js, be cautious there.
3. We can load a specific feeder when zoom in. d3.json(dir+feederFile, function (json){ ... }
4. A separated server need to be load to see the example.

[cshjin]

smoothing out or smoothing: http://en.wikipedia.org/wiki/Homeomorphism_(graph_theory)

While in d3, it has been already detected as 'weight' of the nodes. Terminologically, it should be called as 'degree'
The processing is nothing else, but hide the nodes of weight 2 and its associated links https://github.com/mbostock/d3/wiki/Force-Layout#wiki-nodes
Probably, one third of the total nodes in each feeder can be smoothed out

[dpinney]

A test smoothing/homeomorphism function would be cool and worth the effort.

[dpinney]

Important todo item for this: get Mike Bostock's feedback on this use case. Because if anyone is qualified to solve this before us it's him.

Apparently, the way to get his attention: post on stackoverflow.com, tag it with "d3.js", and wait for "mbostock" user to appear.

[dpinney]

Closing, transitioning to different project.