Category Archives: GIS

Writing one WKT file per feature in QGIS

1 Reply

Someone in #qgis just asked about this so here is a minimal pyqgis (for QGIS 3) example how you can write a separate WKT file for each feature of the currently selected layer. Careful with too many features, filesystems do not like ten thousands of files in the same directory. I am writing them to /tmp/ with $fid.wkt as filename, adjust the path to your liking.

layer = iface.activeLayer()
features = layer.getFeatures()

for feature in features:
  geometry = feature.geometry()
  wkt = geometry.asWkt()
  
  fid = feature.attribute("fid")
  filename = "/tmp/{n}.wkt".format(n=fid)
  
  with open(filename, "w") as output:
    output.write(wkt)

zstandard compression in GeoTIFF

2 Replies

I ran GDAL 2.4’s gdal_translate (GDAL 2.4.0dev-333b907 or GDAL 2.4.0dev-b19fd35e6f-dirty, I am not sure) on some GeoTIFFs to compare the new ZSTD compression support to DEFLATE in file sizes and time taken.

Hardware was a mostly idle Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz with fairly old ST33000650NS (Seagate Constellation) harddisks and lots of RAM.

A small input file was DGM1_2x2KM_XYZ_HH_2016-01-04.tif with about 40,000 x 40,000 pixels at around 700 Megabytes.
A big input file was srtmgl1.003.tif with about 1,3000,000 x 400,000 pixels at 87 Gigabytes.
Both input files had been DEFLATE compressed at the default level 6 without using a predictor (that’s what the default DEFLATE level will make them smaller here).

gdal_translate -co NUM_THREADS=ALL_CPUS -co PREDICTOR=2 -co TILED=YES -co BIGTIFF=YES --config GDAL_CACHEMAX 6144 was used all the time.
For DEFLATE -co COMPRESS=DEFLATE -co ZLEVEL=${level} was used, for ZSTD -co COMPRESS=ZSTD -co ZSTD_LEVEL=${level}

Mind the axes, sometimes I used a logarithmic scale!

Small file

DEFLATE

ZSTD

Big file

DEFLATE

ZSTD

Findings

It has been some weeks since I really looked at the numbers, so I am making the following up spontaneously. Please correct me!
Those numbers in the findings below should be percentages (between the algorithms, to their default values, etc), but my curiosity was satisfied. At the bottom is the data, maybe you can take it to present a nicer evaluation? ;)

ZSTD is powerful and weird. Sometimes subsequent levels might lead to the same result, sometimes a higher level will be fast or bigger. On low levels it is just as fast as DEFLATE or faster with similar or smaller sizes.

A <700 Megabyte version of the small file was accomplished within a minute with DEFLATE (6) or half a minute with ZSTD (5). With ZSTD (17) it got down to <600 Megabyte in ~5 Minutes, while DEFLATE never got anywhere near that.
Similarly for the big file, ZSTD (17) takes it down to 60 Gigabytes but it took almost 14 hours. DEFLATE capped at 65 Gigabytes. The sweet spot for ZSTD was at 10 with 4 hours for 65 Gigabytes (DEFLATE took 11 hours for that).

In the end, it is hard to say what default level ZSTD should take. For the small file level 5 was amazing, being even smaller than and almost twice as fast as the default (9). But for the big file the gains are much more gradual, here level 3 or level 10 stand out. I/O might be to blame?

Yes, the machine was not stressed and I did reproduce those weird ones.

Raw numbers

Small file

Algorithm Level Time [s] Size [Bytes] Size [MB] Comment
ZSTD 1 18 825420196 787
ZSTD 2 19 783437560 747
ZSTD 3 21 769517199 734
ZSTD 4 25 768127094 733
ZSTD 5 31 714610868 682
ZSTD 6 34 720153450 687
ZSTD 7 40 729787784 696
ZSTD 8 42 729787784 696
ZSTD 9 51 719396825 686 default
ZSTD 10 63 719394955 686
ZSTD 11 80 719383624 686
ZSTD 12 84 712429763 679
ZSTD 13 133 708790567 676
ZSTD 14 158 707088444 674
ZSTD 15 265 706788234 674
ZSTD 16 199 632481860 603
ZSTD 17 287 621778612 593
ZSTD 18 362 614424373 586
ZSTD 19 549 617071281 588
ZSTD 20 834 617071281 588
ZSTD 21 1422 616979884 588
DEFLATE 1 25 852656871 813
DEFLATE 2 26 829210959 791
DEFLATE 3 32 784069125 748
DEFLATE 4 31 758474345 723
DEFLATE 5 39 752578464 718
DEFLATE 6 62 719159371 686 default
DEFLATE 7 87 710755144 678
DEFLATE 8 200 705440096 673
DEFLATE 9 262 703038321 670

Big file

Algorithm Level Time [m] Size [Bytes] Size [MB] Comment
ZSTD 1 70 76132312441 72605
ZSTD 2 58 75351154492 71860
ZSTD 3 63 73369706659 69971
ZSTD 4 75 73343346296 69946
ZSTD 5 73 72032185603 68695
ZSTD 6 91 72564406429 69203
ZSTD 7 100 71138034760 67843
ZSTD 8 142 71175109524 67878
ZSTD 9 175 71175109524 67878 default
ZSTD 10 235 69999288435 66757
ZSTD 11 406 69999282203 66757
ZSTD 12 410 69123601926 65921
ZSTD 13 484 69123601926 65921
ZSTD 14 502 68477183815 65305
ZSTD 15 557 67494752082 64368
ZSTD 16 700 67494752082 64368
ZSTD 17 820 64255634015 61279
ZSTD 18 869 63595433364 60649
ZSTD 19 1224 63210562485 60282
ZSTD 20 2996 63140602703 60216
ZSTD 21 lolno
DEFLATE 1 73 87035905568 83004
DEFLATE 2 76 85131650648 81188
DEFLATE 3 73 79499430225 75817
DEFLATE 4 77 75413492394 71920
DEFLATE 5 92 76248511117 72716
DEFLATE 6 129 73901542836 70478 default
DEFLATE 7 166 73120114047 69733
DEFLATE 8 407 70446588490 67183
DEFLATE 9 643 70012677124 66769

(Not) Solving #quiztime with a spatial relationship query in Overpass-Turbo

Leave a reply

I saw a #quiztime on Twitter and it looked like a perfect geospatial puzzle so I had to give it a try: https://twitter.com/Sector035/status/987631780147679233

So we have something called “Lidl” which I know is a supermarket. And we have something else, with a potential name ending in “nden”. And that’s it.

If you have a global OSM database, this would be a simple query away. If you don’t have a global OSM database, the Overpass API let’s you query one. And overpass turbo can make it fun (not in this case though, I had a lot of trial and error until I found these nice examples in the Wiki).

I ended up looking for things within 50 meters to a Lidl named “-nden”

[out:json][timeout:180][maxsize:2000000000];

{{radius=50}}

node[name="Lidl"]->.lidls;

( 
   way(around.lidls:{{radius}})["name"~".*nden$"];
  node(around.lidls:{{radius}})["name"~".*nden$"];
);

(._;>;);  // whatever?

Too far away from the Lidl.

But driving around the place in Google StreetView I could not find the spot and it does not look very much like the photo.

So I guess my query is fine but either the "thing" or the Lidl are not in OSM yet.

Oh well, I did learn something new (about Overpass) and it was fun. :)

Specifying the read/open driver/format with GDAL/OGR

Leave a reply

For the commandline utilities you can’t. One possible workaround is using https://trac.osgeo.org/gdal/wiki/ConfigOptions#GDAL_SKIP and https://trac.osgeo.org/gdal/wiki/ConfigOptions#OGR_SKIP to blacklist drivers until the one you want is its first choice. A feature request exists at https://trac.osgeo.org/gdal/ticket/5917 but it seems that the option is not exposed as commandline option (yet?).

PS: If what you are trying to do is reading a .txt (or whatever) file as .csv and you get angry at the CSV driver only being selected if the file has a .csv extension, use CSV:yourfilename.txt

PPS: This post was motivated by not finding above information when searching the web. Hopefully this will rank high enough for *me* to find it next time. ;)

More structural detail in hillshading with this one weird trick

1 Reply

I wanted to have more structural detail in my QGIS live hillshading so I blended a duplicate layer with the sun angle set to directly overhead (90°) and the blending mode as darken.

It seems to make steep structures more prominent.

z/x/y.ext tiles to mbtiles/gpkg

1 Reply

I just had some trouble converting a set of tiles I scraped into a nice package. I tried both mb-util and tiles2gpkg_parallel.py. Both seemed to work but QGIS did not display anything. The “solution” was to convert the tiles from 512×512 pixels to 256×256. WTF?

tiles2gpkg_parallel.py also crashed at joining its intermediate files if used in Python 3.6. It needs “-tileorigin ul” for OSM-like tile names.

A hurricane map in QGIS, from geodata and hacks

2 Replies

Cartography Inspirationator John Nelson made an awesome map of hurricanes and later posted detailed how-tos for ArcMap and recently ArcGIS Pro. Right after his first post I started rebuilding it in QGIS using draw effects for adding the colored outer glow instead of using image icons, adding a vignette on-the-fly and adjusting the background raster’s saturation on the fly. All in all, less manual work, more dynamic processing in QGIS. I quickly got frustrated though and gave up.

More than a year later (triggered by the new ArcGIS Pro how-to storymap) I revisited my draft and finished it. So here is the QGIS version. As no-one is paying me to write this, I currently cannot be arsed to make it as fancy as John’s posts. Sorry! :)

This project shows QGIS’ strength in features and struggle with performance. Also some bugs. I stopped working on this once I liked the looks. It is not optimized in any way. So this is just how I ended up doing it. You could do much better. I like proof of concepts. And short sentences.

Data sources (I host a lightning-fast mirror of Natural Earth at https://www.datenatlas.de/geodata/public/sources/www.naturalearthdata.com/ if downloads are 404 again…):

That’s right, nothing but pure, unaltered geodata!

  1. Load the NE2_LR_LC_SR_W_DR.tif raster. Lower the saturation to about -50.
  2. Set your project’s projection to EPSG:3031 so that you get a nice polar viewpoint.
  3. Rotate the canvas by 150° because that’s what John did.
  4. Load the graticules. Use a rule-based style for ‘"direction" = 'W' or "direction" = 'E' or "direction" is NULL‘ so that you get all longitudes but only the equator from the latitudes. Set the layer transparency to 60 or something like that.
  5. Load the coastline.
  6. Add a vignette using this trick.
  7. Load the Allstorms.ibtracs_all_points.v03r09.shp Shapefile and then:
    1. Create a rule-based style with rules for the storm categories. I think I used this.
    2. Use white markers without outlines. Set their sizes using an expression on the wind speed or like me, manually to e.g. 0.7, 0.8, 1, 1, 1.6, 2 millimeters. This is something to play around with until it looks good.
    3. I used transparency for the markers of the lowest classes, 85% and 70%, the others are not transparent. This is something to play around with until it looks good. If I recall correctly I used transparency on the layer level here to keep the bubbly looks.
    4. By now you should be quite annoyed at how slow the rendering process is. >:) But wait, it gets much worse! :o)
    5. For each of the classes, enable draw effects on the markers.
    6. Set the source to use Addition blend mode. Set the source to be somewhat transparent, I used 50%, 80%, 80%, 50%, 40%, 0%. This is something to play around with until it looks good.
    7. Add outer glow and choose an appropriate color (Hint: Use a lot of saturation). Then play around with the spread, blur radius and transparency until it looks good. For some reason I ended up using 1mm/3/95%, 1mm/4/95%, 1mm/3/80%, 1.2mm/4/50%, 1.5mm/4/40%, 3mm/1/50%. Only now that I post this I realise how weirdly inconsistent this is and a quick test shows how irrelevant the blur radius changes are (except for the highest class). Oh well. It’s not fun to interate if you have to click so much and rendering that 40-50 seconds…
  8. That’s it! Done!

This was both fun and incredibly annoying. QGIS has the features but lacks in speed for this funky project (no wonder, blending 300,000 glowing points is not that nice). Here is a realtime video of how it rendered on my machine (take away some seconds from manually enabling the layers after another):

PS: Oh god this WordPress style sucks…

Flowers in QGIS?

Leave a reply

The other day I was working on visualisation of some intermediate research stuff and ended up with something looking like a bouquet of flowers.

Twitter liked it so here is a how-to.

  1. Have some lines that meet in a shared point. They should have a shared ID per group (in my case they were MultiLineStrings).
  2. Color them per ID.
  3. Turn the background black.
  4. Use an Arrow style for the lines. Set the Head thickness to half the Arrow width. Set the Head length to whatever you consider fancy.
  5. Remove the Outline (set No Pen).
  6. Set the Feature blending mode to Multiply (or to Screen or Dodge or Addition if you prefer fireworks to flowers).
  7. Set an appropriate color scheme. For flowers I think RdYlGn works great (that’s how I realised what my random tinkering had lead to) or PiYg or simply Spectral, for fireworks random colors.

That’s it! Now play around and have some fun!

FOSSGIS 2017

Leave a reply

Die FOSSGIS 2017 in Passau war grandios. Ich bin sooo froh, dass ich mich auf den weiten Weg gemacht hatte. Die Liste von Vortragsaufzeichnungen, die ich selbst noch anschauen will, ist lang… Ausprobieren muss ich unbedingt (mal wieder) ein aktuelles GRASS GIS, GVSIG CE (das Poster hat Lust gemacht), osmium, die ganzen Vector Tiles Tools uvm.

Selbst gewagt habe ich einen Lightning Talk über Interaktive Visualisierung von Geodaten in Jupyter Notebooks (Youtube) sowie einen Vortrag zu meinem Projekt GeoPackages der freien Hamburger Geodaten (Youtube) anzufertigen.

Der LT kam so extrem gut an, dass ich nächstes Mal wohl um einen richtigen Vortrag oder auch Workshop kaum herum komme. :o)

Für den Geopackage-Vortrag hatte ich leider die Daten und Skripte zuhause gelassen und musste daher etwas improvisieren… Trotzdem war kam auch er gut an und ich habe großartigen Input bekommen, z.B. dass es ein tolles neues QGIS-Plugin für GML Application Schema Gedöns gibt und einen GMLAS-Treiber in GDAL. Danke!

Bei spontanen QGIS-Anwender- und Vereinstreff habe ich nachgefragt, wie es eigentlich mit QGIS an den Hochschulen aussieht und wie ich meinen Arbeitgeber vielleicht mal auf den Weg von Esri/IDRISI zu QGIS bringen kann. Da war ausser Claas Leiners Lehre in Kassel wenig bekannt. Vielleicht starte ich mal eine kleine Recherche, um etwas Einblick in die Landschaft zu bekommen. Wäre doch klasse, wenn sich mehr Unis von proprietärer Software entsagen mögen!

What does it look like if you move all countries onto the same location?

2 Replies

Note from 2019-12-07: This is neither efficient nor up-to-date with modern PROJ. Better don’t copy and paste but just take non-projection/-transformation parts if you need them…

Sunday pre-lunch Python fun: What does it look like if you move all countries onto the same location?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

import fiona
from shapely.geometry import *
from fiona.crs import from_string
from fiona.transform import transform_geom
from shapely.affinity import translate
 
def get_biggest_polygon(multipolygon):
    assert isinstance(geometry, MultiPolygon)
 
    max_area = 0
    biggest_polygon = None
    for polygon in geometry:
        if polygon.area &gt; max_area:
            max_area = polygon.area
            biggest_polygon = polygon
    return biggest_polygon
 
def project_locally(geometry, from_crs):
    """centered on the centroid of the geometry"""
    # ugly because i map/shape back and forth, maybe try shapely instead
    lat = geometry.centroid.y
    lon = geometry.centroid.x
 
    to_crs = from_string("+proj=aeqd  +R=6371000 +lat_0={lat} +lon_0={lon}".format(lat=lat, lon=lon))
    reprojected_geometry = transform_geom(
        from_crs, 
        to_crs, 
        mapping(geometry)
    )
 
    return shape(reprojected_geometry)
 
with fiona.open("ne_10m_admin_0_countries.shp") as countries:
    centered_polygons = []
 
    for country in countries:
        geometry = shape(country['geometry'])
 
        # only use the biggest part of each country, otherwise everything sucks
        if isinstance(geometry, MultiPolygon):
            polygon = get_biggest_polygon(geometry)
        else:
            polygon = geometry
 
        # project nicely
        polygon = project_locally(polygon, countries.crs)
 
        # centering on 0,0 is simply moving the geometry by MINUS its x/y
        dx = -polygon.centroid.x
        dy = -polygon.centroid.y
        translated_polygon = translate(polygon, dx, dy)
 
        centered_polygons.append(translated_polygon)
 
with open("/tmp/outfile.wkt", "w") as sink:
    for polygon in centered_polygons:
        sink.write(polygon.wkt+"\n")

Use this in any way you like but please share your creations and code as well. :)

Some rough explanation: For each country I check if it is a multipolygon and if so, use only its biggest “sub”-polygon in the next steps. I then project the WGS84 coordinates to an Azimuthal Equidistant projection centered on the centroid of the polygon. That new geometry gets shifted to sit on the origin of the system. I collect all those polygons and write them as plain WKT to a file. Styling was done in QGIS.

And for the smart folk, the same but without local projection: