Skip to content

Boundary Reviews (1)

Jump to bottom
Will Roper edited this page Jan 15, 2022 · 5 revisions

When a new boundary review is completed, boundary bot should find it and raise an issue. Once that happens, there are 3 things we need to do in order to update Every Election.

  1. Set an end date on the previous DivisionSet.
  2. Create a new DivisionSet and import the division names from the Electoral Change Order.
  3. Attach the new boundaries to the new divisions.

There are 3 corresponding management commands to help us do this:

  • manage.py update_end_dates
  • manage.py import_divisionsets_from_csv
  • manage.py import_lgbce

Run them with --help for full list of switches etc

Generally we will store all of the materials relating to a boundary review in the lgbce-mirror S3 bucket in under a key named {slug}/{ECO name} e.g: east-devon/The East Devon (Electoral Changes) Order 2017/. Slugs and ECO names can be found in https://github.com/DemocracyClub/boundary-data/blob/master/lgbce.json

TODO : get boundary bot to create this for us

Most of the data we need for this process can be found in https://github.com/DemocracyClub/boundary-data/blob/master/lgbce.json At each stage of this process, we will also need the 3-letter local auth code from the local authority register. In most cases, Boundary Bot will be able to attach a register_code for us but occasionally it will be null and we'll have to find it manually.

1. Set an end date on the previous DivisionSet.

Before creating a new DivisionSet we need to set an end date on the current one. Once an ECO is proposed, the relevant page on the LGBCE site (e.g: http://www.lgbce.org.uk/current-reviews/south-west/devon/east-devon again, grab this from https://github.com/DemocracyClub/boundary-data/blob/master/lgbce.json ) will tell us when the new boundaries will come into force. This is expressed in terms of the next election rather than a precise date so we must infer it. The end date for the old DivisionSet should be the day before the new one starts.

Create a CSV file like

org,start_date,end_date
EDE,2011-01-28,2019-05-01

Upload to our S3 bucket and import using manage.py update_end_dates -s "east-devon/The East Devon (Electoral Changes) Order 2017/end_date.csv".

By default this command will only set an end_date if the existing end_date is NULL but does have a --overwrite flag. All statements run inside a transaction.

2. Create a new DivisionSet and import the division names from the Electoral Change Order.

When the ECO is made, we need to grab the new division names from the Electoral Change Order and import them into a new DivisionSet. To extract the division names from the ECO, find the link to the ECO in https://github.com/DemocracyClub/boundary-data/blob/master/lgbce.json (in most cases) or on the LGBCE site (if Boundary Bot couldn't find it for us) and construct the XML link to the ward list. For example

Use the ECO parser to extract the ward list from the XML. Hopefully this will mostly work, but check it. There may be cases that are not handled well. This tool is still work in progress.

Use the output of the ECO parser to construct a CSV file. It should look roughly like this: https://docs.google.com/spreadsheets/d/18X4wYGGmJNbL9xwN-ALXPQYDAed4PN1M9xC0YiKZ-TM/edit#gid=1833724937 (leave the start_date blank. We'll infer it from the end_date we set in the previous step). Upload the CSV to our S3 bucket and import using manage.py import_divisionsets_from_csv -s "east-devon/The East Devon (Electoral Changes) Order 2017/eco.csv"

This command creates a new DivisionSet and infers a start_date from the end_date of the most recent DivisionSet. It will fail (by design) if the most recent DivisionSet has a NULL end_date. All statements run inside a transaction.

3. Attach the new boundaries to the new divisions.

Once we've imported the division names we can attach polygons to them. Grab the shapefiles URL (e.g: http://www.lgbce.org.uk/__data/assets/file/0006/33585/East_Devon_final_proposals.zip) from https://github.com/DemocracyClub/boundary-data/blob/master/lgbce.json and upload them to S3.

TODO : get boundary bot to do this for us

Check the file in QGIS and try the import locally using manage.py import_lgbce -s "east-devon/The East Devon (Electoral Changes) Order 2017/East_Devon_final_proposals.zip" EDE. Note that in this case we must pass the 3-letter register code in as an argument because it won't be in the shapefile. This command will only import polygons against a DivisionSet whose divisions do not already have associated geography objects. All statements run inside a transaction.

There are a few checks/possible pain points here:

  • If import_lgbce throws ValueError: Expected 1 layer, found 2, you've probably given it a zip with 2 sets of shapefiles (one for district wards and one for parishes). Make a new zip with only the district ward shapefiles and feed it that instead.
  • Sometimes Django/GDAL won't parse the shapefile. If so, use the old ogr2ogr -skipfailures fixed_shapefile.shp corrupted_shapefile.shp trick (classic).
  • If import_lgbce throws GDALException: Could not open the datasource, it might be because the shapefiles are in a subdirectory inside the zipfile. import_lgbce expects to find the shapefiles in the root. If they are in a subdir, re-create the zip with the shapefiles in the root.
  • The column with the ward names in it might not be called name. If so, pass the desired column name in with the -n param.
  • Usually the file will be srid=27700. This is the default, but do check it. If the file is srid=4326, pass that in with the --srid param.
  • Sometimes there are actual mistakes in the shapefile (e.g: 2 polygons with the same ward name). These will require follow up with LGBCE to obtain corrected data.
  • At this stage, wards don't have codes yet so we need to match the polygons to the divisions by name. Sometimes the ward names in the legislation don't exactly match the ward names in the shapefiles. If this happens, import_lgbce will throw a DiffException and output some handy information on how to fix it (see https://github.com/DemocracyClub/EveryElection/pull/165). If this happens, you'll need to upload a file name_map.json to the S3 bucket providing a lookup from the LGBCE names to the names as they appear in legislation. An example name_map.json might look like:
{
  "Conningbrook and Little Burton Farm": "Conningbrook & Little Burton Farm",
  "Kingsnorth Village and Bridgefield": "Kingsnorth Village & Bridgefield",
  "Rolvenden and Tenterden West": "Rolvenden & Tenterden West"
}

This doesn't have to be passed in with a command line argument. import_lgbce will just look for it automatically.

Clone this wiki locally