I tried to package a package+CLI+GUI for Python

And I gave up trying.

Here is our project: https://gitlab.com/g2lab/cogran-python

  • There is a python package in cogran/.
  • There are scripts in scripts/, one CLI interface, one QT GUI.
  • There are docs in docs/.

The GUI uses images and text from the docs for live help.
So we need to make sure that our docs directory is included when packaging this all for others to install.

I tried for two days to make sense of the jungle of Python packaging. StackOverflow is full of non-explanatory half-answers, there are about 7 APIs implemented by 35 different projects with 42 different documentations and 98 different versions. Either I massively failed to find the one, wonderful, canonical, documented approach or this really is neither explicit nor beautiful.

So I give up. Can you help me out?

The setup

For building I used

rm -r build/ dist/ cogran.egg-info/ \
python setup.py sdist \
python setup.py bdist_wheel

For looking into the resulting archives I used

watch tar tvf dist/cogran-0.1.0.tar.gz

and

watch unzip -t dist/cogran-0.1.0-py3-none-any.whl

For installing I used

pip uninstall --yes cogran \
pip install --no-cache-dir --user --upgrade dist/cogran-0.1.0.tar.gz && \
find ~/.local/ | grep cogran

and

pip uninstall --yes cogran \
pip install --no-cache-dir --user --upgrade dist/cogran-0.1.0-py3-none-any.whl && \
find ~/.local/ | grep cogran

What happens

sdist includes all kinds of stuff like the .gitignore and the examples directory. Both should not be included at the moment. Installing the resulting cogran-0.1.0.tar.gz will just install the package (cogran/__init__.py) and the scripts. Nothing else.

bdist_wheel only includes the package and the scripts. Nothing else.

Adding a MANIFEST.in

After fucking around with many iterations of fruitless attempts of writing a MANIFEST.in file that beautifully builds upon the someone-said default includes I gave in and wrote a very explicit one: https://gitlab.com/snippets/1850708

global-exclude *

graft cogran
graft docs
graft scripts
graft tests

include README.md
include LICENSE

include setup.py
include requirements.txt

global-exclude __pycache__
global-exclude *.py[co]

Now sdist includes all the stuff I want and nothing I do not want. Excellent. Installing the tar.gz however again only installed the package and the scripts.

The wheel again only had the package and the scripts inside.

include_package_data=True

You just need to add include_package_data=True to your setup.py, people said. So I did. And no, that would only work for things inside our package, not directories on the same level. So this would change nothing.

data_files

Supply the paths to the files in a data_files list, someone said. So I added an explicit list (at this stage I just wanted it to work, who cares about manual effort, future breakage and ugliness):

data_files = [
  ("docs", (
    "docs/images/Areal Weighting.png",
    "docs/images/Attribute Weighting.png",
    "docs/images/postcodes vs districts.png",
    "docs/images/Binary Dasymetric Weighting.png",
    "docs/images/N-Class Dasymetric Weighting.png",
    "docs/help.html"
  ))
]

And yes, now these files end up in the wheel. But guess what, they still do not get installed anywhere

What now?

User cannot log in to your MediaWiki?

I just spent too much time on this. A user could not log in to a wiki I maintain. We tried everything. At some point I must have fixed the main issue but sadly I had introduced a new one. Here is what happened in the hopes that if *you* read this, you can fix it as easily as me (spending much less time on it).

Looking at the browser’s network log we noticed that the wiki first logged them in but then, on the redirect to the main page, deleted the cookies again.

Looking at the debugging log of the wiki I saw:

[authentication] Primary login with MediaWiki\Auth\LocalPasswordPrimaryAuthenticationProvider succeeded
[authentication] Login for bob succeeded

...

[DBQuery] wiki BEGIN /* Wikimedia\Rdbms\Database::query (User::saveSettings) 31.16.28.131 */

[DBQuery] wiki
UPDATE /* User::saveSettings 31.16.28.131 */ user
SET user_name = 'bob',user_real_name = '',
user_email = 'bob@example.com',
user_email_authenticated = NULL,
user_touched = '20190423202751',
user_token = 'stripped',
user_email_token = NULL,
user_email_token_expires = NULL
WHERE user_id = '3' AND user_touched = '20190423202746'

[DBQuery] wiki ROLLBACK /* MWExceptionHandler::rollbackMasterChangesAndLog 31.16.28.131 */

[exception] [stripped] /w/index.php?title=Special:UserLogin&returnto=Main+Page
MWException from line 4026 of /srv/wiki/w/includes/user/User.php: CAS update failed
on user_touched for user ID '3' (read from replica); the version of the user to be
saved is older than the current version.
#0 /srv/wiki/w/includes/session/SessionBackend.php(648): User->saveSettings()
#1 /srv/wiki/w/includes/deferred/MWCallableUpdate.php(30): MediaWiki\Session\SessionBackend->MediaWiki\Session\{closure}()
#2 /srv/wiki/w/includes/deferred/DeferredUpdates.php(257): MWCallableUpdate->doUpdate()
#3 /srv/wiki/w/includes/deferred/DeferredUpdates.php(210): DeferredUpdates::runUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactorySimple, string, integer)
#4 /srv/wiki/w/includes/deferred/DeferredUpdates.php(131): DeferredUpdates::execute(array, string, integer)
#5 /srv/wiki/w/includes/MediaWiki.php(905): DeferredUpdates::doUpdates(string)
#6 /srv/wiki/w/includes/MediaWiki.php(731): MediaWiki->restInPeace(string)
#7 /srv/wiki/w/includes/MediaWiki.php(750): MediaWiki->{closure}()
#8 /srv/wiki/w/includes/MediaWiki.php(554): MediaWiki->doPostOutputShutdown(string)
#9 /srv/wiki/w/index.php(43): MediaWiki->run()
#10 {main}

So after the login succeeded there was an exception which invalidated the session and caused the wiki to delete the cookie on the next page load. The stupid reason for this was that someone (not me, ha ha ha) earlier set the user’s user_touched field to 0 because surely that was a safe value. Setting it to 20190423202746 instead let the user log in just fine. And afterwards it was happily updated by the wiki again.

Open Layers View Tracker

I built this last year for some research and then swiftly forgot about releasing it to the public. Here it is now:

https://gitlab.com/Hannes42/OpenLayersViewTracker

Try it online: https://hannes42.gitlab.io/OpenLayersViewTracker/

Some awful but working JavaScript code to track a website user’s interaction with a Open Layers map. You can use this to do awesome user studies and experiments.

  • Runs client-side
  • You will get a polygon of each “view”!
  • You can look at them in the browser!
  • There are also timestamps! Hyperaccurate in milliseconds since unix epoch!
  • And a GeoJSON export!
  • This works with rotated views!
  • Written for Open Layers 4 using some version of JSTS, see the libs/ directory. No idea if it works with the latest versions or if Open Layers changed their API again.

Please do some funky research with it and tell me about your experiences! Apart from that, you are on your own.

There is a QGIS project with example data included. Check out the Atlas setup in the Print Layout!

Screenshot from a browser session

Resulting GeoJSON in QGIS

So if Time Manager supports the timestamp format you could interactively scroll around. I did not try, that plugin is so finicky.

Replaying what I looked at in a Open Layers web map, using QGIS Atlas

Fantec QB-X8US3/H82-SU3S2 keeps resetting

Hoping that *you* found this post via Google and this helps you:

If your Fantec QB-X8US3 enclosure keeps resetting and you get lots of sd or usb errors in dmesg, check your harddisks and *remove* flaky ones. I had random, frequent resets (unmounting all your drives forcefully…) when I thought I could use a dying drive as temporary storage in the enclosure. As soon as I removed that bad drive, the device was rock-stable again. 10++ would buy again.

See also http://forum.mediasonic.ca/viewtopic.php?f=115&t=3413.

Fantec QB-X8US3 is apparently the same as a H82-SU3S2 ProBox. Here’s a review.

Sincerely yours, DenverCoder9

Replicating a media-hyped color by numbers Etsy map in 10 minutes

https://old.reddit.com/r/MapPorn/comments/ajaz42/the_arteries_of_the_world_map_shows_every_river/eeu3uj9/

Thats beautiful… how long it took?

Well, that looks like QGIS’ random colors applied to http://hydrosheds.org/

So I fired up QGIS, extracted the region from eu_riv_15s.zip, realised those rivers came without a corresponding basin, extracted the region from eu_bas_15s_beta.zip, set the map background to black, set the rivers to render in white, set the rivers’ line width to correspond to their UP_CELLS attribute, put the basins on top, colored them randomly by BASIN_ID, set the layer rendering mode to Darken and that was it.

I should open an Etsy store.

Yes, I realise that replicating things is easier than creating them. But seriously, this is just a map of features colored by category and all the credit should go to http://hydrosheds.org/

Update

But Hannes, that original has some gradients!

Ok, then set the rivers not to white but a grey and the basin layer rendering mode to Overlay instead of Darken.

This product incorporates data from the HydroSHEDS database which is © World Wildlife Fund, Inc. (2006-2013) and has been used herein under license. WWF has not evaluated the data as altered and incorporated within, and therefore gives no warranty regarding its accuracy, completeness, currency or suitability for any particular purpose. Portions of the HydroSHEDS database incorporate data which are the intellectual property rights of © USGS (2006-2008), NASA (2000-2005), ESRI (1992-1998), CIAT (2004-2006), UNEP-WCMC (1993), WWF (2004), Commonwealth of Australia (2007), and Her Royal Majesty and the British Crown and are used under license. The HydroSHEDS database and more information are available at http://www.hydrosheds.org.

Average Earth from Space 2018

Due to popular request, I now provide a high-resolution, digital download as an experiment. Any profit from this will go towards bug-fixes in GDAL, the free and open-source software I used for this project.

I collected all the daily images of the satellite-based Soumi VIIRS sensor and calculated the per-pixel average over the whole year of 2018.

You can explore it in full resolution in an interactive webmap.

WGS84 / Equirectangular
Web Mercator

You can see fascinating patterns, e.g. “downstream” of islands in the ocean or following big rivers like the Amazon. Be wary though as snow and clouds look the same here.

It’s also fun to look at the maximum to see cloudless regions (this image is highly exaggerated and does not make too much sense anyways):

Many thanks to Joshua Stevens for motivation and infos on data availability! The initial idea was of course inspired by Charlie ‘vruba’ Lyod‘s Cloudless atlas works. I explored imagery on https://worldview.earthdata.nasa.gov/, grabbed Soumi VIIRS images via https://wiki.earthdata.nasa.gov/display/GIBS/GIBS+API+for+Developers and processed it in GDAL. Poke me repeatedly to blog about the process please.

Full resolution, georeferenced imagery (12288×6144 resp. 8192×8192) in PNG available on request.

I said average but of course it is the median. The average or mean does not make much sense to use nor does it look any good. ;-)

Data-defined images in QGIS Atlas

Say you want to display a feature specific image on each page of a QGIS Atlas.

In my example I have a layer with two features:

{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"image_path": "/tmp/1.jpeg"
},
"geometry": {
"type": "Polygon",
"coordinates": [[[9,53],[11,53],[11,54],[9,54],[9,53]]]
}
},
{
"type": "Feature",
"properties": {
"image_path": "/tmp/2.jpeg"
},
"geometry": {
"type": "Polygon",
"coordinates": [[[13,52],[14,52],[14,53],[13,53],[13,52]]]
}
}
]
}

And I also have two JPEG images, named “1.jpeg” and “2.jpeg” are in my /tmp/ directory, just as the “image_path” attribute values suggest.

The goal is to have a map for each feature and its image displayed on the same page.

Create a new print layout, enable Atlas, add a map (controlled by Atlas, using the layer) and also an image.

For the “image source” enable the data-defined override and use attribute(@atlas_feature, 'image_path') as expression.

That’s it, now QGIS will try to load the image referenced in the feature’s “image_path” value as source for the image on the Atlas page. Yay kittens!

Genauigkeitsfetischismus

https://www.spiegel.de/spiegel/fergus-falls-in-minnesota-in-der-kleinen-welt-der-waehler-von-donald-trump-a-1140600.html

Fergus Falls liegt im Westen Minnesotas, zwischen den Bundesstaaten Wisconsin und North Dakota, am nördlichen Rand der USA. Die Jahresdurchschnittstemperatur nahe der Grenze zu Kanada liegt bei plus 3 Grad Celsius, im Winter bei minus 20 Grad. Von den Hochhäusern New Yorks und den Stränden San Franciscos sind es mehr als 2200 Kilometer bis nach Fergus Falls. Von El Paso, der mexikanischen Grenze, braucht ein Auto 22 Stunden. Wie viele Mexikaner zieht es in diese Gegend? Wer stellt hier so ein Schild auf?

https://www.spiegel.de/kultur/gesellschaft/fall-claas-relotius-us-kleinstadt-fergus-falls-vom-betrug-beim-spiegel-betroffen-a-1244806.html

Der SPIEGEL hat in einem ersten Schritt seine Dokumentationsabteilung das Manuskript der Relotius-Geschichte noch einmal in Stichproben prüfen lassen – und musste feststellen, dass bei der Verifikation tatsächlich nicht sauber gearbeitet wurde. (Mehr zu den Sicherheitsmechanismen des Hauses lesen Sie hier.)

Ein paar Detail-Beispiele, die auffielen:

So sind es von Fergus Falls nicht 2200 Kilometer nach New York, wie es im Text steht, sondern nur 1888.

Ich *hasse* diesen Zahlenfetischismus (passt genau in diese gefühlsmanipulative Art von Reportagen), daher war mein Interesse geweckt (während im Hintergrund gerade “for king and country” aus den Lautsprechern schallte.

Warum sollten es exakt 1888 Kilometer sein? Was hatte die Spiegel-Dokumentation denn da gemessen? Rathaus zu Rathaus? Zentroid zu Zentroid der administrativen Ortsgrenzen? Die kürzeste Distanz?

Da im nächsten Satz des Artikels eine Reisedauer mit dem Auto angegeben wurde, scheint es naheliegend, dass der Autor auch für die Distanzen eine solche Reise angenommen hatte.

Und siehe da, Google Maps gibt für die Reise mit dem Auto von Fergus Falls (Minnesota 56537) nach New York 2218 Kilometer, alternativ 2335 Kilometer an.

Graphhopper bestätigt die 2218 Kilometer (exakt!!!11).

Bing bietet Routen von 2208 bis 2372 Kilometer an.

Yandex rundet sinnvoll auf 2300 bzw. 2400 Kilometer

Vier verschiedene Routenplaner bestätigen also eine Entfernung von grob 2200 Kilometern (oder mehr) und damit die Angabe im Artikel des Autors.

Was hat die Spiegel-Dokumentation also dann wohl gemacht?

Auf Wikipedia gibt es repräsentative Koordinaten für beide Orte, 46°16′59″N 96°04′39″W für Fergus Falls (irgendwo), 40°42′46″N 74°00′21″W für New York City (Rathaus).

Betrachtet man die Erde als Spheroid läge die direkte Distanz zwischen diesen bei etwa 1878 Kilometern.

Mit einem besser passenden Ellipsoid (WGS84) wären es 1882 Kilometer.

Die Spiegel-Dokumentation hat also wohl irgendwelche Koordinaten, relativ zentral in den beiden Orten, gewählt und die direkte Distanz zwischen ihnen berechnet. Immerhin geodätisch!

Fazit: Ach, watt weiß ich, dieses Detail im Faktencheck der Spiegel-Dokumentation ist definitiv verständnisloser Aktionismus. Distanzen sind ein kompliziertes Konzept. Ist eine Reisedistanz gemeint? Diese kann durch unterschiedliche Routen oder Umleitungen stark abweichen. Ist eine direkte Distanz gemeint? Falls ja, wie genau kann das Ergebnis jetzt sein? Was sind die exakten Punkte, zwischen denen gemessen wurde und welche Art von Genauigkeit bzw. Unsicherheit haben diese? Wann ist es sinnvoll eine Stadt mit einer Koordinate zu beschreiben?

1888 Kilometer ist in jedem Fall nichts als ein blindes Vortäuschen von Exaktheit.

PS: Zahlen vergöttern die, die sie nicht verstehen. Mir schwindelte.

Fake chromatic aberration and dynamic label shadows in QGIS

Welcome to Part 43 of “Fun with Weird Hacks in QGIS”!

Fake Chromatic Aberration

Get some lines or polygons! I used German administrative polygons and a “Outline: Simple line” style.

Add a “Geometry Generator” to the same layer and set it to “LineString / MultiLineString”. If you used polygons, you will need to use “boundary($geometry)” to get the border lines of your polygons.

Translate (shift) the geometries radially away from the map center, with an amount of translation based on their distance to the map center. This will introduce ugly artifacts cool glitches for big geometries. Note the magic constant of 150 I divided the distances by, I cannot be arsed to turn this into percentages so you will have to figure out what works for you. Also, if your map is in a different CRS than the layer you do this with, you will need to transform the coordinates (I do that for the labels below).

 translate(
   boundary($geometry),
   (
    x(centroid($geometry))-x(@map_extent_center)
   )/150
   ,
   (
    y(centroid($geometry))-y(@map_extent_center)
   )/150
)

Color those lines in some fancy 80s color like #00FFFF.

We only want the color to appear in the edges of the map, so set the “Feature Blending” mode of the layer to “Lighten”. This will make sure the white lines do not get darker/colored.

I forgot to take an image for that and then it was lunch time. :x

Now do the same but for distances the other way around and color that in something else (like foofy #FF00FF).

 translate(
   boundary($geometry),
   (
    x(@map_extent_center)-x(centroid($geometry))
   )/150
   ,
   (
    y(@map_extent_center)-y(centroid($geometry))
   )/150
)

Oh, can you see it already? Move the map around! Ohhh!

For the final touch, use the layer’s “Draw Effects” to replace the “Source” with a “Blur”. Be aware that the “Blur type” can quite strongly influence the look and find a setting for the “Blur strength” that works for you. I used “Guassian blur (quality)” with a strength of 2.

Dynamic Label Shadows

Get some data to label! I used ne_10m_populated_places_simple. Label it with labels placed “Offset from Point” without any actual offset. This is just to make sure calculations on the geometry’s location make sense to affect the labeling later.

Pick a wicked cool font like Lazer 84!

Add a “Buffer” to the labels and pick an appropriate color (BIG BOLD #FF00FF works well again).

Time for magic! Add a “Shadow” to the labels and use an appropriate color (I used the other color from earlier again, #00FFFF).

We want to make the label shadows be further away from the label if the feature is further away from the map center. So OVERRIDE the offset with code that does exactly that. Note another magic constant (relating to meters in EPSG:25832) and that I needed to transform coordinates here (my map is in EPSG:25832 while this layer is EPSG:4326).

distance(  
	transform(
		$geometry, 'EPSG:4326', 'EPSG:25832'
	),
	@map_extent_center
)/10000

Cool. But that just looks weird. Time for another magical ingredient while OVERRIDING the angle at which the label is placed. You guessed it, radially away from the map center!

degrees(
	azimuth(  
		transform(
			$geometry, 'EPSG:4326', 'EPSG:25832'
		),  
		@map_extent_center)
	)
-180

By the way, it does look best if your text encoding is introducing bad character marks and missing umlauts!

That’s it, move the map around and be WOWed by the sweet effects and the time it takes to render :o)

And now it is your turn to apply this to some MURICA geodata and rake in meaningless internet points that make you feel good! Sad but true ;)

And also TODO: Make the constants percentages instead, that should make sure it works on any projection and any scale.

Making a Star Wars hologram in QGIS

I like doing silly things in QGIS.

So I wanted to make a Star Wars hologram (you know, that “You’re my only hope” Leia one) showing real geodata. What better excuse for abusing QGIS’ Inverted Polygons and Raster Fills. So, here is what I did:

  1. Find some star wars hologram leia image.
  2. Crudly remove the princess (GIMP’s Clone and Healing tools work nicely for this).
  3. In QGIS create an empty Polygon-geometry Scratchpad layer and set the renderer to Inverted Polygons to fill the whole canvas.
  4. Set the Fill to a Raster image Fill and load your image.
  5. Load some geodata, style it accordingly and rejoice.
    1. Get some geodata. I used Natural Earth’s countries, populated places and tiny countries (to have some stuff in the oceans), all in 110m.
    2. Select a nice projection, I used “+proj=ortho +lat_0=39 +lon_0=139 +x_0=0 +y_0=0 +a=6371000 +b=6371000 +units=m +no_defs”.
    3. I used a three layer style for the countries:
      • A Simple Line outline with color #4490f3, stroke width 0.3mm and the Dash Dot Line stroke style.
      • A Line Pattern Fill with a spacing of 0.8mm, color #46a8f3 and a stroke width of 0.4mm.
      • And on top of those, for some noisiness a black Line Pattern Fill rotated 45° with a spacing of 1mm and stroke width 0.1mm.
      • Then the Feature Blending Mode Dodge to the Layer Rendering and aha!
      • More special effects come from Draw Effects, I disabled Source and instead used a Blur (Gaussian, strength 2) to lose the crispness and also an Outer Glow (color #5da6ff, spread 3mm, blur radius 3) to, well, make it glow.





    4. I used a three layer style for the populated places:
      • I used a Simple Marker using the “cross” symbol and a size of 1.8mm. The Dash Line stroke style gives a nice depth effect when in Draw Effects the Source is replaced with a Drop Shadow (1 Pixel, Blur radius 2, color #d6daff).
      • A Blending Mode of Addition for the layer itself makes it blend nicely into the globe.
    5. I used a three layer style for the tiny countries:
      • I used a white Simple Marker with a size of 1.2mm and a stroke color #79c7ff, stroke width 0.4mm.
      • Feature Blending Mode Lighten makes sure that touching symbols blob nicely into each other.




  6. You can now export the image at your screen resolution (I guess) using Project -> Import/Export or just make a screenshot.
  7. Or add some more magic with random offsets and stroke widths in combination with refreshing the layers automatically at different intervals: