Guide to preparing GIS data for Optioneer.

A complete, step-by-step guide on how to process GIS data for Optioneer using QGIS software.

Adam Anyszewski avatar
Written by Adam Anyszewski
Updated over a week ago

Introduction

The purpose of this article is to provide a set of fail-safe steps that can be carried out in QGIS which will guarantee that you can convert any GIS dataset into a format that is acceptable by Optioneer.

This guide assumes that you understand the basic concepts around GIS data for Optioneer - please read the 'fundamentals' and 'optimal data structure' articles beforehand.

The demonstration was carried out using QGIS 3.18.2 with Grass (codename 'Zurich').

Best practices

  1. Unzip all of the raw files and upload to QGIS

  2. Create a layer with ‘project extent’ polygon - corresponding to what you have in Optioneer

  3. Create a dedicated folder for the task. Include folders listed below inside of this folder (see 'Recommended file structure').

  4. Give yourself an hour of uninterrupted time - it’s very easy to stretch this to days if you don’t remain focused on the task at hand

  5. Process all layers at once with steps outlined below.

  6. Always use batch processing (see 'How to use Batch Processing in QGIS')

  7. Find relevant functions in the ‘Processing Toolbox’

Recommended file structure

Create the following folders inside of your dedicated project folder.

  1. Reprojected

  2. Fixed

  3. Clipped

  4. DroppedMZ

  5. MultiToSingle

  6. FinalLayers

  7. ForUpload

How to use Batch Processing in QGIS

Batch processing is very handy as it allows you to apply the same function to multiple layers at once. This is why it pays off to do all layers altogether - you only need a few steps for configuration and QGIS will take care of the rest.

Follow this link to official documentation for QGIS.

Launch the Batch Processing processing panel:

Select all input layers - click on 'Use Open Layers'. See below:

Apply the same operation to all layers such as reprojecting, clipping, fixing geometries etc. In this example, reprojecting needs a 'Target CRS' for each layer. Use 'Drill Down' to fill all at once.

Specify file path and name of each file - select 'Fill with parameter value' and then 'Input layer' to get the whole column populated automatically.

Once set up, just click run. Some batch processing work can take a long time, especially if you're processing a lot of data!

Steps to carry out in QGIS

Step 1: Remove layers you don’t need

Feel free to simply remove layers that you don’t plan to use in Optioneer. Don’t delete them from your hard drive but remove them from the layer list. This will save you some processing time further down the line.

Step 2: Reproject to WGS84 using the ‘Reproject layer’ function

Use Processing Toolbox > Vector general > Reproject layer function

Use WGS 84 / EPSG: 4326 as the target projection.

Save individual result layers to the '1. Reprojected' folder.

We suggest that you put an 'r' in front of the layer names to keep record of layer stage.

When reprojection is done, remove the original layers from QGIS and drop the new layers from '1. Reprojected' folder into your workspace.

After you complete the step, your project screen should look as shown below. No visible difference, but the project should now all be in WGS84.

Step 3: Fix geometries using ‘Fix geometries’ function

Use Processing Toolbox > Vector geometry > Fix geometries

Save individual result layers to the '2. Fixed' folder.

We suggest that you put an 'f' in front of the layer names to keep record of the layer stage.

When reprojection is done, remove the original layers from QGIS and drop the new layers from '2. Fixed' folder into your workspace.

Step 4: Clip to project extent

Use Processing Toolbox > Vector geometry > Extract/clip by extent

In the batch processing tool, select extent by 'Use Layer Extent' and select your project boundary layer to set the extent for all clipping operations. It is up to you if you'd like to 'Clip features to extent' - if set to 'Yes', it will cut any features that extend beyond project boundary to project boundary. In this example, we set the 'Clip features to extent' to 'Yes'.

Step 5: Remove layers which have no features in project extent

This step is pretty simple - highlight all layers and select 'Show Feature Count'. A value in square brackets will show up after the layer name. If that value shows zero for a given layer, it should be removed (Optioneer will complain about empty layers).

You might also be able to determine that layers which contained very few features are worth removing too - especially if features are within the extent but unlikely to be part of routing considerations.

Step 6: Drop MZ values

Use Processing Toolbox > Vector geometry > Drop M/Z values

In the batch processing window, follow the pattern from before and make sure that 'Drop M' and 'Drop Z' are both set to 'Yes' for all layers.

Save the result to '4. DroppedMZ' folder.

We suggest that you put an 'd' in front of the layer names to keep record of the layer stage.

There is no visual difference after this step is carried out.

Step 7: Multipart to singlepart

Use Processing Toolbox > Vector geometry > Multipart to singleparts

Save the result to '5. MultiToSingle' folder.

We suggest that you put an 'm' in front of the layer names to keep record of the layer stage.

There is no visual difference after this step is carried out.

Optional step 8A: split layers by parameter, if necessary

If you would like to split a layer into more detailed layers, we suggest that you use

Processing Toolbox > Vector general > Split vector layer functionality.

We suggest that you create a dedicated folder for each group of split layers within

the '6. FinalLayers' folder.

Optional step 8B: merge layers if possible

If you would like to merge layers that represent similar data, we suggest that you use

Processing Toolbox > Vector general > Merge vector layers functionality.

We suggest that you output the merged layers directly to the '6. FinalLayers' folder.

Optional step 8C: buffer points if required

Set a reasonable buffer around points - this will save them as polygons of non-negligible size and will allow you to use them in Optioneer. We suggest that you use Processing Toolbox > Vector geometry > Buffer functionality.

We suggest that you output the buffered layer directly to the '6. FinalLayers' folder.

NB: Buffer value is inputted in degrees, by default. 1/3600 degrees = circa 30m.

Step 9: Rename layers appropriately

Layers in Optioneer should not contain any special characters in their names and avoid white spaces. You can either rename the layer and save in the '6. FinalLayers' folder or just edit their name directly in QGIS. Saving them to the folder will allow you to use layers in different 'packaging' in the future so it's worth doing if you can spare time to save each of them under a correct name.

Step 10: Package into datasets of choice

The final step is packaging of data to get datasets that can be uploaded to Optioneer. We suggest that you use Processing Toolbox > Database > Package layers functionality.

Package datasets in line with suggestions from this article.

We suggest you put these files in '7. ForUpload' folder and keep them in folders with version / date of upload. This is important as Optioneer doesn't allow sophisticated data management just yet so we rely on you managing your data in QGIS or ArcGIS.

NB: Please ensure that 'styles layer' is not included in the final geopackage as it will create an empty layer which will be rejected by Optioneer.

ArcGIS version of this article

We don't use ArcGIS internally - if you are a fluent user and would like to help us create an equivalent article with an ArcGIS workflow, please get in touch with adam@continuum.industries

Did this answer your question?