Author Archives: Hannes

Converting lots of audio files to lower quality with soundKonverter

Notes for myself but maybe useful for *you* too?

https://github.com/dfaust/soundkonverter

Add your directory or directories, let it scan.

For music select Opus with 48 kbps if that’s enough for your use case.

Create the file list, then fix special characters with https://github.com/dfaust/soundkonverter/issues/96: sed -i 's/file:\/\///; s/%20/ /g; s/%23/#/g' ~/.local/share/soundkonverter/filelist.xml

For audiobooks use Opus with a low bitrate of 16 kbps and the Additional encoder argument -ac 1 for mono.

Copy directory structure will create full path directories in the destination directory, e.g. if you convert /home/foo/music/ to /tmp/, it will create the output in /tmp/home/foo/music/.

For future updates: Select the same directories, fix the xml again, make sure that Settings -> General -> Conversion -> Conflict Handling is set to “Skip File”.

Copying files to a Fenix 6S on Archlinux

Make sure its USB mode is set to MTP mode, not Garmin mode.

Watch dmesg to see when the device is connected and ready:

[Fri Jun 28 15:27:59 2024] usb 1-2.1: new high-speed USB device number 36 using xhci_hcd
[Fri Jun 28 15:27:59 2024] usb 1-2.1: New USB device found, idVendor=091e, idProduct=CENSORED, bcdDevice= 0.01
[Fri Jun 28 15:27:59 2024] usb 1-2.1: New USB device strings: Mfr=0, Product=0, SerialNumber=5
[Fri Jun 28 15:27:59 2024] usb 1-2.1: SerialNumber: CENSORED

Use lsusb to verify:

Bus 001 Device 036: ID 091e:CENSORED Garmin International

android-file-transfer/aft-mtp-mount will not enable to you to write files on to it, instead it will show your device does not have android EditObject extension, you will not be able to write into individual files if you mount with it and creating files will lead to cannot create regular file errors.

Use simple-mtpfs instead (via).

simple-mtpfs --list-devices should list
1: GarminFenix 6S Pro/Sapphire

Mount it: simple-mtpfs /tmp/mnt

Copy something: cp -v some_file /tmp/mnt/GARMIN/

Unmount it: fusermount -u /tmp/mnt

Wait for the clock to be ready.

Done!

Waveshare Pico-RGB-LED with a Raspberry Pi via Thonny on Archlinux

https://www.waveshare.com/wiki/Pico-RGB-LED is horribly broken and this was hard to Google so hopefully this helps *you* or future-me.

I did not manage to put the display directly on top of the Pico as a hat. Also my display’s board has its pins numbered and labelled mirrored when comparing to the official pin out. FFS…!?


Install pico-sdk and thonny from AUR.

$ sudo usermod -a -G uucp $USER

Reboot.

Make sure you aren’t full of static electricity. Hold the BOOTSEL button on your PI and connect it to your PC using a USB cable.

It should appear as storage device.

Start Thonny, select “Micropython (Raspberry Pi Pico)” as interpreter and use its “Install or update MicroPython” option. This does that “copy some RPI_PICO-20231005-v1.21.0.uf2 file to your Pico in mass storage mode” step for you.

The Thonny shell should say something like:

MicroPython v1.21.0 on 2023-10-06; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>>

Enter the following code in your Thonny shell (via):

from machine import Pin
led = Pin(25, Pin.OUT)
led.on()

The Pico’s green LED should glow now. Use led.off() to turn it off again.

Unplug your Pico. Make sure you aren’t full of static electricity.

Connect your Pico and Waveshare Pico-RGB-LED like described in https://thepihut.com/blogs/raspberry-pi-tutorials/how-to-use-the-waveshare-rgb-full-colour-led-matrix-panel-for-raspberry-pi-pico-part-1:

On the display there are four pins in a length-wise direction on the board.

Connect:

  • The display’s 5V to your Pico’s VBUS (pin 40)
  • The display’s GND to your Pico’s GND (pin 38)
  • The display’s DIN to your Pico’s GP6 (pin 9)

References:

Have fun!

https://www.twilio.com/blog/programming-raspberry-pi-pico-microcontroller-micropython

https://thepihut.com/blogs/raspberry-pi-tutorials/how-to-use-the-waveshare-rgb-full-colour-led-matrix-panel-for-raspberry-pi-pico-part-1

https://docs.micropython.org/en/latest/esp8266/tutorial/neopixel.html

To make a script launch upon booting, save it with Thonny to the Pico with the name main.py.

A Klaus Schulze inspired, animated QGIS geometry generator doodle

Found this

Made that

With an empty inverted polygon layer, the canvas on 0 0,

with_variable(
  'line_star',
  collect_geometries(
    with_variable(
      'buffered_point_array',
      geometries_to_array(
        nodes_to_points(  
          buffer(make_point(0,0), distance:=100, segments:=42),
          ignore_closing_nodes:=True
        )
      ),
      array_foreach(
        @buffered_point_array,
        make_line(
          @canvas_cursor_point,
          @element
        )
      )
    )
  ),
  with_variable(
    'donut_line_star',
    difference(
      @line_star,
      buffer(
        @canvas_cursor_point,
        distance:=sin(scale_linear(epoch(now())%10000/100, 0, 100, -pi(), pi()))*30+30,  -- %10000/100 = 0-100
        segments:=42
      )
    ),
    collect_geometries(
      @donut_line_star,
      intersection(
        rotate(@donut_line_star, 1),
        translate(
          buffer(
            @canvas_cursor_point, 
            distance:=sin(scale_linear(epoch(now())%10000/100, 0, 100, -pi(), pi()))*50+50,  -- %10000/100 = 0-100
            segments:=42
          ),
          sin(scale_linear(epoch(now())%10000/100, 0, 100, -pi(), pi())),  -- %10000/100 = 0-100
          0
        )
      )
    )
  )
)

and an arrow style with feature blending.

moOde audio player 8 on a Raspberry Pi 3 Model B Rev 1.2 with Hifiberry Amp2 1.1

It’s super easy but watch out because volume levels are messed up (super loud on tiny volume values!).

Flash the image with RPI imager (enable SSH)

/boot/config.txt might need dtparam=audio=off but I don’t remember, try without that change.

Boot the device

Visit http://moode.lan (or fix your network, use the IP etc) and click “m” -> “Settings”

“Audio” -> “Audio Output” -> “I2P”: Select “HifiBerry Amp2”

Restart

Use your favorite MPD client (or the web UI) to play some music (FIRST SET A LOW VOLUME for safety, 100 is insane here, ~20 is room level). It should work already, if not, troubleshoot.

“Audio” -> “ALSA Options” -> “Max volume (%)” seems to do nothing

“Audio” -> “MPD Options” -> “Volume options” -> “Max MPD volume”: Set to 30% so at least in the Web UI of moOde you won’t be able to kill your speakers.

In your other MPD clients, make sure you never set a high volume…

Music on connected storage will be discovered automatically and added to your library.

Remotely controlling QGIS through PostgreSQL queries 🤪

I just wanted to see if it was possible, it is not a sane thing to do.

PostgreSQL has a notification event system where you NOTIFY on a notification channel and clients can LISTEN for events.

QGIS already supports this for a convenient, remotely triggered refresh function of PostGIS layers by listening on the qgis channel if a PostgreSQL connection exists:

  • Load a PostGIS layer
  • Set its symbology to something dynamically randomized so you see changes if the map is refreshed
  • In the layer’s rendering options set it to refresh on notification
  • In a PostgreSQL client trigger a notification event, e.g. NOTIFY qgis, 'hi'; or SELECT pg_notify('qgis', 'refresh plz!');
  • The layer is refreshed in QGIS
  • You can set the layer(s) to refresh on any event on the qgis channel or only for specific messages. For example you could set each PostGIS layer to refresh on a different message, e.g. their table name and use that in a PostgreSQL trigger.

This is currently bound to existing PostGIS layers and as far as I know, you cannot do more than refresh layers. Edit: You can also trigger layer actions (thanks to the great people at Oslandia)!

But…

QGIS comes with PyQt and PyQt (of course) has its own API to let you talk to database servers. And luckily its QSqlDriver class even has a notification signal that let’s you react to notification events in the most convenient way.

So I wrote something:

from qgis.PyQt.QtSql import QSqlDatabase, QSqlQuery, QSqlDriver


def exec_notification(name, source, payload):
    """ZOMG don't ever do this!"""
    try:
        exec(payload.replace('\\n', '\n'))
    except SyntaxError as e:
        iface.messageBar().pushMessage("PG NOTIFY", str(e))

db = QSqlDatabase.addDatabase("QPSQL")
db.setConnectOptions("service=foo")  # using ~/.pg_service.conf
if not db.open():
    raise Exception(db.lastError().databaseText())
db.driver().subscribeToNotification("qgis_qtsql_channel")

db.driver().notification[
    str, QSqlDriver.NotificationSource, "QVariant"
].connect(exec_notification)

Run it in QGIS’ script editor and now you can remotely execute any Python code by sending it from PostgreSQL. For example:

SELECT pg_notify(
	'qgis_qtsql_channel',
	array_to_string(
		array[
			'project = QgsProject.instance()',
			'project.read("map.qgz")'
		],
		'\n'
	)
)


SELECT pg_notify(
	'qgis_qtsql_channel',
	'iface.mapCanvas().redrawAllLayers()'
);


SELECT pg_notify(
	'qgis_qtsql_channel',
	'iface.actionExit().trigger()'
);

Of course this is a horrible idea and you should never run code that a third-party sends to you. Instead you should write specific methods on the Python side and sent well defined messages to control exactly which methods get executed. But, hey it works!

Ideas

  • Trigger a notification if a PostgreSQL-stored QGIS project gets updated and in QGIS notify the user if they have said project open.
  • Make a tiny plugin that lets the user specify a mapping of message -> Python code scripts
  • TBC

Addendum

At first I had used a simple db.driver().notification.connect(on_notification), expecting the slot to simply receive all the arguments that the notification signal would send. But this led to Qt using a different notification signal (just sending the channel name) or something because the name is overloaded. Qt super explainer eyllanesc once again saved the day by showing that one can (and in this case needs to) specify the correct signature so Qt picks the correct signal to connect to. This was a major learning for me.

For debugging I used another slot function connected to the signal which uses both the QGIS message log and the message bar to inform the user about incoming notification events, take it if you want:

def on_notification(name, source, payload):
    message = f"{name} from {source}: {payload}"
    QgsMessageLog.logMessage(message, "PG NOTIFY")
    iface.messageBar().pushMessage("PG NOTIFY", message)

A bad surprise to me was that my QGIS installation on Ubuntu 22.04 (using https://qgis.org/ubuntu) did not have a dependency graph that included PostgreSQL support in QtSql. Ouch! I would have thought this would be usable in any standard QGIS installation out of the box. I had to install libqt5sql5-psql. On Windows this seems to be no issue, both a MSI installer and an advanced OSGeo4W install had the necessary modules. Phew!

Plugin options and QGIS3.ini

In the spirit of learning in the open:

All the docs say is “It is recommended to give some structure to naming of keys.”

The [PythonPlugins] section uses the plugins’ directory names and shows if a plugin is activated or not (true/false).

Plugins can store their options anywhere. No really. Anywhere! You will find some in their own sections, e. g.:

  • Qgis2threejs (directory “Qgis2threejs“) uses a [Qgis2threejs] section, named the same as its directory
  • First Aid (firstaid) uses a [FirstAid] section, using different case than its directory, and another section [firstaid] for other options
  • Plugin Builder 3 (pluginbuilder3) uses a [PluginBuilder] section, named differently than its directory
  • QGIS GML Application Schema Toolbox (gml_application_schema_toolbox) stores options in [QGIS%20GML%20Application%20Schema%20Toolbox] (needlessly) using URL encoding
  • Räumlicher Filter (spatial_filter) uses two sections: [SpatialFilter] and [SpatialFilterSymbol] (sorry, we didn’t know better…)
  • There is a [Plugins] and a [plugins] section. I found some plugins storing some options in the [plugins] section, e.g. First Aid storing some firstaid\debugger-geometry=... there. [Plugins] seems to be used by core plugins. [plugins] stores e.g. if QGIS should automatically check for new ones (although that is also might be tracked in the [plugin-manager] section, no idea which option comes from an older QGIS version maybe).

Some things can be stored elsewhere, I am not sure who does it, maybe some objects do this automatically? E.g. in the [QgsCollapsibleGroupBox] section you might find the states of all(?) QgsCollapsibleGroupBox objects such as gml_application_schema_toolbox\gmlas_bbox_group\collapsed=false concerning a plugin’s widget. Or in [Windows] you might find FirstAidDebugDialog\geometry=....

Plugins might alter any other options of course. There is no way of knowing what plugin touched what (great power and mostly good :) ).

Uninstalling a plugin does not remove the sections from the INI. The plugin itself does not have a chance to clean that up and QGIS has no way of knowing what to remove. (Of course it might be wanted to keep settings around, but it would be nice to be able to really prune something.)

tl;dr: There is no standard, no rules nor best practices for QGIS plugins to store their options.

PS: This post might get updated as I learn. Corrections and improvements are highly appreciated.

PPS: Excuse the big bold highlighting. That’s for me.

Backing up Hetzner snapshots locally

Hetzner is a nice, cheap host for server. Unfortunately they do not let you download backups and snapshots of cloud servers locally. So you are kind of locked in with them.

Here is how I create full disk backups for e.g. standard CX11 servers with default images.

Make sure you understand everything before you attempt this. Pay special attention to your own partitions and make sure you archive what you really need. Consider imaging the whole device instead of just a partition.

Create a backup

  • Create a new snapshot of your server
  • Create a new server from it
  • Boot said server in rescue mode
  • Use good old dd and gzip to image the main disk to a local archive
  • ssh root@SERVERIP "dd if=/dev/sda1 bs=1M status=progress | gzip -" | dd of=SERVERIP.sda1.dd.gz bs=1M
    • The bs=1M is fairly random by me, worked well, didn’t care to optimize
    • gzip (with the default compression level) kind of maxed out the server CPU but still enabled me to almost max out my download bandwidth. Of course the ratio between remote CPU compression speed vs download bandwidth depends on your specific situation. You might want to use xz or zstd instead.
  • Don’t be stupid, do it twice and compare the checksums

If you do not need your server to be live during backup, you can skip the snapshotting and second server of course, just boot your server in rescue mode instead.

Build a new server from a backup

  • Create a new server using the same or a similar configuration as the backed up one
  • Boot said server in rescue mode
  • cat SERVERIP.sda1.dd.gz | ssh root@SERVERIP "gunzip -c | dd of=/dev/sda1 bs=1M status=progress conv=fsync"
  • Reboot to leave rescue mode

If you want to restore your server because of a breakage, just boot it into rescue mode and do the same as above otherwise.

I can’t read file contents from my Android SD card on Linux and it’s weird

Conclusion: The files are encrypted with Android’s file-based encryption. The SD card is broken.

The SD card is broken. The internet says it is typical for SD cards to go read-only as a symptom of fail. Trying to rsync or ddrescue the card’s contents using a separate devices led to stalls and read errors.

I might be able to salvage some of the files by using Android’s share functionality but there was nothing important on the card so I won’t bother trying. The card is broken beyond repair and all files are lost.


I want to backup my Android SD card so I inserted it to my computer (tried multiple ones by now). I can see the file tree and browse it just fine. But if I try to read a file, it fails because instead of the actual file contents, all I get is some gibberish. Weirdly enough, the file header is partially identical between different files and contains the words “whatever” and “CONSOLE”. When inserted in my phone I can read and use those files just fine. The phone is a Sony Xperia XZ1 Compact.

Here are some example hexdumps of the file headers from some JPG and OPUS files:

00000000  00 00 00 00 00 37 d5 23  d8 c3 44 86 e4 42 f3 73  |.....7.#..D..B.s|
00000010  03 00 00 02 00 00 10 00  00 02 8c 2d 04 09 03 01  |...........-....|
00000020  77 68 61 74 65 76 65 72  60 ed 60 a5 16 dd d0 08  |whatever`.`.....|
00000030  34 0d 25 2b 87 d1 df 18  94 8a f8 cf f0 fd 83 d9  |4.%+............|
00000040  06 5a 4e 48 8c a1 b9 51  98 ed 16 62 08 5f 43 4f  |.ZNH...Q...b._CO|
00000050  4e 53 4f 4c 45 00 00 00  00 60 40 2d b1 41 51 c1  |NSOLE....`@-.AQ.|
00000060  10 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002000  c7 27 3f d6 7e 99 3d 6b  d3 dc 0a a7 c9 28 37 d5  |.'?.~.=k.....(7.|
00002010  59 09 8b 4b 3e 1b 20 54  a8 87 fc 90 fd 31 05 5b  |Y..K>. T.....1.[|
...
0037ffe0  f1 9a 2d fa fd a1 4b 2a  22 dc ce 29 9d 83 3a 5a  |..-...K*"..)..:Z|
0037fff0  dc 97 e0 e9 15 d7 16 55  82 a6 57 6b 7f b0 32 7d  |.......U..Wk..2}|
00380000
00000000  00 00 00 00 00 03 5d 1d  aa ab 3d 85 96 2a 8a 70  |......]...=..*.p|
00000010  03 00 00 02 00 00 10 00  00 02 8c 2d 04 09 03 01  |...........-....|
00000020  77 68 61 74 65 76 65 72  60 90 17 d4 8b ef 31 35  |whatever`.....15|
00000030  22 b9 b3 05 59 37 3b 71  7e e1 4a 6e af a2 07 b2  |"...Y7;q~.Jn....|
00000040  4b 9a bb 7e 6a 46 18 70  29 ed 16 62 08 5f 43 4f  |K..~jF.p)..b._CO|
00000050  4e 53 4f 4c 45 00 00 00  00 60 40 2d b1 41 51 c1  |NSOLE....`@-.AQ.|
00000060  10 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002000  32 23 5d 69 cb 14 d0 75  8a 9c 2e 64 67 26 1a fc  |2#]i...u...dg&..|
00002010  12 77 ad 48 9b f4 9e d4  d5 08 48 33 cb 45 6b 90  |.w.H......H3.Ek.|
...
00037fe0  87 09 0c 80 6c 4b 6a 54  fb c2 70 13 bc 0a c9 ab  |....lKjT..p.....|
00037ff0  89 3b 52 40 f2 cc df f4  d3 65 e1 c4 0d e3 74 ea  |.;R@.....e....t.|
00038000
00000000  00 00 00 00 00 01 78 41  33 26 6e 9c 0f a7 d9 69  |......xA3&n....i|
00000010  03 00 00 02 00 00 10 00  00 02 8c 2d 04 09 03 01  |...........-....|
00000020  77 68 61 74 65 76 65 72  60 2e 2d e8 1b cd 47 11  |whatever`.-...G.|
00000030  7e b6 e7 e2 95 84 85 75  81 42 0f 5c 54 48 3e 4b  |~......u.B.\TH>K|
00000040  c7 3c f6 cf 16 ec 9f 6b  51 ed 16 62 08 5f 43 4f  |.<.....kQ..b._CO|
00000050  4e 53 4f 4c 45 00 00 00  00 60 40 2d b1 41 51 c1  |NSOLE....`@-.AQ.|
00000060  10 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002000  97 9d 44 91 25 8e b4 e0  07 31 31 bc cc c0 44 13  |..D.%....11...D.|
00002010  62 0e 6d c1 49 8d 87 fe  5d c7 ff c0 cc b5 4c 08  |b.m.I...].....L.|
...
00019fe0  ae 2b d1 04 db d2 9c 9e  e7 cd 3a 06 aa 5a 85 5f  |.+........:..Z._|
00019ff0  d3 7a 30 33 f2 3a b1 2f  73 40 50 f7 a9 a6 d7 fa  |.z03.:./s@P.....|
0001a000

fdisk shows the partition as “W95 FAT32 (LBA)“.

What’s going on?

I found some mentions of this issue on the web:

https://old.reddit.com/r/techsupport/comments/f88bd7/files_on_sd_card_taken_out_of_phone_wont_open/ posted “iu>i'Õè Œ- whatever`P‘Ã@C“Žâ=ë q¸ÌÔ9rÊ ¯Ø¶Þ¨Kíb_CONSOLE ]ŽÒ[Ó_” without a solution.

https://www.computerbase.de/forum/threads/videodatei-samsung-a-3-2016-am-pc-abspielen.1737550/ posted “@Âù©õš•Bo   Œ-  `»Üž¸½\ht1Oc.ä9á²5Ñé’Ž<žŸ9Ûßíb_CONSOLE Øþ«fQ¿É“, also without a solution. The phone involved was a Samsung A3 2016.

https://www.nikon-fotografie.de/community/threads/jpg-dateien-pl%C3%B6tztlich-nicht-mehr-lesbar.281280/ posted “#µ=Äõ¶EBC   Œ-  "3DUfw`â￾Lü4OOŸ-›·j.^aÌ?–ôaÚ ”YÜíb_CONSOLE .øŒ¤Èœ«é ‰'ñÏÆísÿä‰â¢THû´/Ò†´< bb·7iìÕ“, also without a solution.

https://superuser.com/questions/975695/reading-encrypted-files-from-android-microsd-card posted three examples, all also having at least the “b._CONSOLE” portion, Samsung Galaxy S3 device. No solution but a user commenting that “This is related to Samsung Knox and these signatures are Samsung specific” which seems unlikely as I am using a Sony device…

^ See the top of the post for my conclusion ^

Und tschüß, Forum/TPHH-Spiegel community.codeforhamburg.org

community.codeforhamburg.org war ein (auf Basis von Discourse aufgesetztes) Forum für Code for Hamburg e. V., online vom Mai 2019 bis April 2023.

Die Homepage
Jeder Eintrag im Transparenzportal Hamburg wurde gespielt
Man konnte Suchen speichern und bei neuen Ergebnissen eine Mail erhalten
Ein Beispiel für einen Eintrag
Ein Eintrag mit mehreren verknüpften Resourcen

Die Idee war ursprünglich, dass dort Kommentare und Diskussionen zu den im Transparenzportal Hamburg (TPHH) veröffentlichten Datensätzen gepostet werden könnten. Und das vielleicht sogar direkt im TPHH eingebettet.

Dazu hat ein Python-Skript (als Bot namens “Heidi Kabel” ;) ) jede Minute die API des TPHH nach neuen Datensätzen gefragt, die Metadaten abgerufen und sie schön formatiert und aufbereitet als Foren-Threads gepostet. Ein spaßiges Bastel-Projekt, irgendwann 2019.

Die Dokumentation für die Forumsinstallation und der Code für den Bot liegen in:

Naja, dann kam Corona und Treffen von Code for Hamburg gab’s eh schon kaum noch. Die Seite hat praktisch niemand außer mir genutzt, es gab keine 10 Posts, die nicht vom Bot waren. Mit der Einbindung im TPHH wurde es auch nichts (ich hatte mich aber auch nicht drum gekümmert).

“Angesagte” Posts mit vielen Views
Pageviews exklusive (erkannte) Crawler
Top Referred Topics (keine Ahnung wo sie verlinkt wurden)
Der tägliche User war unser TPHH-Spiegel-Bot, die Posts/Topics auch

Ich hatte es für mich selbst noch als einfaches Interface für die Daten und als praktischen Benachrichtigungsservice auf bestimmte Keywords genutzt, aber jetzt klappt das Update der Forensoftware nicht mehr (Bug von Discourse mit angeblich nicht mehr einzigartigen Tags und es gab keine Rückmeldung auf meine Fragen, von daher …) und ich will da nix auf veralteter Software laufen lassen. Übernehmen wollte es niemand aus dem Verein und im Fediverse hat sich auch niemand gemeldet. Von daher mach ich die Tage das Licht aus.

PS: Ich hoffe Code for Hamburg wird irgendwann mal wieder reaktiviert, mit regelmäßigen Treffen, wo gehackt und gebastelt wird. Toi toi toi!