Category Archives: python

I tried to package a package+CLI+GUI for Python

And I gave up trying.

Here is our project: https://gitlab.com/g2lab/cogran-python

  • There is a python package in cogran/.
  • There are scripts in scripts/, one CLI interface, one QT GUI.
  • There are docs in docs/.

The GUI uses images and text from the docs for live help.
So we need to make sure that our docs directory is included when packaging this all for others to install.

I tried for two days to make sense of the jungle of Python packaging. StackOverflow is full of non-explanatory half-answers, there are about 7 APIs implemented by 35 different projects with 42 different documentations and 98 different versions. Either I massively failed to find the one, wonderful, canonical, documented approach or this really is neither explicit nor beautiful.

So I give up. Can you help me out?

Do we need to integrate the docs into the actual package? Should we instead have three separate things: The cogran module package, a cogran-cli package, a cogran-gui package?

The setup

For building I used

rm -r build/ dist/ cogran.egg-info/ ; \
python setup.py sdist && \
python setup.py bdist_wheel

For looking into the resulting archives I used

watch tar tvf dist/cogran-0.1.0.tar.gz

and

watch unzip -t dist/cogran-0.1.0-py3-none-any.whl

For installing I used

pip uninstall --yes cogran ; \
pip install --no-cache-dir --user --upgrade dist/cogran-0.1.0.tar.gz && \
find ~/.local/ | grep cogran

and

pip uninstall --yes cogran ; \
pip install --no-cache-dir --user --upgrade dist/cogran-0.1.0-py3-none-any.whl && \
find ~/.local/ | grep cogran

What happens

sdist includes all kinds of stuff like the .gitignore and the examples directory. Both should not be included at the moment. Installing the resulting cogran-0.1.0.tar.gz will just install the package (cogran/__init__.py) and the scripts. Nothing else.

bdist_wheel only includes the package and the scripts. Nothing else.

Adding a MANIFEST.in

After fucking around with many iterations of fruitless attempts of writing a MANIFEST.in file that beautifully builds upon the someone-said default includes I gave in and wrote a very explicit one: https://gitlab.com/snippets/1850708

global-exclude *

graft cogran
graft docs
graft scripts
graft tests

include README.md
include LICENSE

include setup.py
include requirements.txt

global-exclude __pycache__
global-exclude *.py[co]

Now sdist includes all the stuff I want and nothing I do not want. Excellent. Installing the tar.gz however again only installed the package and the scripts.

The wheel again only had the package and the scripts inside.

setup.py: include_package_data=True

You just need to add include_package_data=True to your setup.py, people said. So I did. And no, that would only work for things inside our package, not directories on the same level. So this would change nothing.

setup.py: data_files

Supply the paths to the files in a data_files list, someone said. So I added an explicit list (at this stage I just wanted it to work, who cares about manual effort, future breakage and ugliness):

data_files = [
  ("docs", (
    "docs/images/Areal Weighting.png",
    "docs/images/Attribute Weighting.png",
    "docs/images/postcodes vs districts.png",
    "docs/images/Binary Dasymetric Weighting.png",
    "docs/images/N-Class Dasymetric Weighting.png",
    "docs/help.html"
  ))
]

And yes, now these files end up in the wheel. But guess what, they still do not get installed anywhere

setup.py: package_data

https://github.com/pypa/sampleproject/blob/master/setup.py#L164 like include_package_data only applies to data inside the package, our’s is on the side.

What now?

Writing one WKT file per feature in QGIS

Someone in #qgis just asked about this so here is a minimal pyqgis (for QGIS 3) example how you can write a separate WKT file for each feature of the currently selected layer. Careful with too many features, filesystems do not like ten thousands of files in the same directory. I am writing them to /tmp/ with $fid.wkt as filename, adjust the path to your liking.

layer = iface.activeLayer()
features = layer.getFeatures()

for feature in features:
  geometry = feature.geometry()
  wkt = geometry.asWkt()
  
  fid = feature.attribute("fid")
  filename = "/tmp/{n}.wkt".format(n=fid)
  
  with open(filename, "w") as output:
    output.write(wkt)

z/x/y.ext tiles to mbtiles/gpkg

I just had some trouble converting a set of tiles I scraped into a nice package. I tried both mb-util and tiles2gpkg_parallel.py. Both seemed to work but QGIS did not display anything. The “solution” was to convert the tiles from 512×512 pixels to 256×256. WTF?

tiles2gpkg_parallel.py also crashed at joining its intermediate files if used in Python 3.6. It needs “-tileorigin ul” for OSM-like tile names.

What does it look like if you move all countries onto the same location?

Sunday pre-lunch Python fun: What does it look like if you move all countries onto the same location?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
import fiona
from shapely.geometry import *
from fiona.crs import from_string
from fiona.transform import transform_geom
from shapely.affinity import translate
 
def get_biggest_polygon(multipolygon):
    assert isinstance(geometry, MultiPolygon)
 
    max_area = 0
    biggest_polygon = None
    for polygon in geometry:
        if polygon.area > max_area:
            max_area = polygon.area
            biggest_polygon = polygon
    return biggest_polygon
 
def project_locally(geometry, from_crs):
    """centered on the centroid of the geometry"""
    # ugly because i map/shape back and forth, maybe try shapely instead
    lat = geometry.centroid.y
    lon = geometry.centroid.x
 
    to_crs = from_string("+proj=aeqd  +R=6371000 +lat_0={lat} +lon_0={lon}".format(lat=lat, lon=lon))
    reprojected_geometry = transform_geom(
        from_crs, 
        to_crs, 
        mapping(geometry)
    )
 
    return shape(reprojected_geometry)
 
with fiona.open("ne_10m_admin_0_countries.shp") as countries:
    centered_polygons = []
 
    for country in countries:
        geometry = shape(country['geometry'])
 
        # only use the biggest part of each country, otherwise everything sucks
        if isinstance(geometry, MultiPolygon):
            polygon = get_biggest_polygon(geometry)
        else:
            polygon = geometry
 
        # project nicely
        polygon = project_locally(polygon, countries.crs)
 
        # centering on 0,0 is simply moving the geometry by MINUS its x/y
        dx = -polygon.centroid.x
        dy = -polygon.centroid.y
        translated_polygon = translate(polygon, dx, dy)
 
        centered_polygons.append(translated_polygon)
 
with open("/tmp/outfile.wkt", "w") as sink:
    for polygon in centered_polygons:
        sink.write(polygon.wkt+"\n")

Use this in any way you like but please share your creations and code as well. :)

Some rough explanation: For each country I check if it is a multipolygon and if so, use only its biggest “sub”-polygon in the next steps. I then project the WGS84 coordinates to an Azimuthal Equidistant projection centered on the centroid of the polygon. That new geometry gets shifted to sit on the origin of the system. I collect all those polygons and write them as plain WKT to a file. Styling was done in QGIS.

And for the smart folk, the same but without local projection: