the Geeklab

Manage Univention DNS with Terraform

2022-09-11T13:57:16+02:00

Using Terraform is a great way to manage infrastructure as code. To map all the different types of resources in a deployment, Terraform uses plugins. Plugins are executable binaries written in Go that communicate with Terraform Core via an RPC interface. Each plugin provides an implementation for a specific service.

Sometimes there is no specific plugin for a service, but if the service provides a REST API, the generic REST API provider can be helpful. This time, I was looking for a way to manage DNS records on a Univention Corporate Server (UCS) using Terraform. The Univention Directory Manager (UDM) API is well documented and can be used with the RESP API provider, but there are a few minor pitfalls to be aware of.

First of all, a basic provider configuration needs to be added to Terraform:

provider "restapi" {
  uri                   = "https://univention.example.com/univention/udm/"
  username              = "my_user"
  password              = "secure-password"
  id_attribute          = "dn"
  debug                 = true
  create_returns_object = true
  headers = {
    accept = "application/json"
  }
}

Ensure that a full URL to the UDM is used in the uri parameter. If username and password are used, the Terraform provider adds a basic authentication header. Alternatively, it is also possible to set other authentication headers manually. This is required to use e.g. token-based authentication instead of username and password. Terraform requires a unique ID for all objects under control. Because manually managing unique IDs is somewhat tedious, it is preferred to let the API handle it whenever possible. In the case of Univention, a basic understanding of object management is required. On a UCS, DNS objects are stored in the OpenLDAP objects. As a distinguished name (DN) uniquely identifies an LDAP record, this attribute can be perfectly used as id_attribute for the Terraform provider. But it is also important to set create_returns_object=true. This option tells the provider that each create operation (POST) will return an object. As a result, for creation events, the provider will parse the returned data, use the id_attribute field as a unique ID, and stores the data along with the ID in Terraforms internal data structures. To finalize the provider configuration, the accept header need to be set to application/json, otherwise the API will return HTML that can not be parsed by the Terraform provider.

With this configuration, the restapi_object resource can now be used to manage DNS records:

resource "restapi_object" "ucs_server" {
  path = "/dns/host_record/"
  data = jsonencode({
    "position" : "zoneName=example.com,cn=dns,dc=example,dc=com,
    "properties" : {
      "name" : "my_host",
      "a" : [
        192.168.0.10
      ],
    }
  })
}

Store Terraform State on Backblaze S3

2022-09-04T20:47:53+02:00

Terraform is an open source infrastructure-as-code tool for creating, modifying, and extending infrastructure in a secure and predictable way. Terraform needs to store a state about the managed infrastructure and configuration. This state is used by Terraform to map real-world resources to your configuration and track metadata. By default, this state is stored in a local file, but it can also be stored remotely.

Terraform supports multiple remote backend provider including S3. I already use Backblaze for backups and have had good experiences with it. Since Backblaze also provides an S3 Compatible API, I wanted to use it for Terraform. How to use S3 as a state backend is well documented, but as it’s focused on Amazon S3 there are a few things to take care of. A basic working configuration will look like this:

terraform {
  backend "s3" {
    bucket                      = "my-bucket"
    key                         = "state.json"
    skip_credentials_validation = true
    skip_region_validation      = true
    endpoint                    = "https://s3.us-west-004.backblazeb2.com"
    region                      = "us-west-004"
    access_key                  = "0041234567899990000000004"
    secret_key                  = "K001abcdefgklmnopqrstuvw"
  }
}

It is required to enable skip_credentials_validation and skip_region_validation because Backblaze uses a different format for these values and the validation only covers Amazon S3. For security reasons, I would recommend setting at least the access_key and secret_key parameters as environment variables instead of writing them to the Terraform file. For the credentials, it is required to create an App Key on Backblaze. After creating the Key the keyName need to be used as access_key and applicationKey as secret_key.

That’s basically all. If you want to go a step further, it is possible to save the state encrypted with Server-Side Encryption (SSE). There are two options available. The first is SSE-B2, where the key is managed and stored by Backblaze, which is quiet simple to configure. The second option is to use customer managed keys. Using this option, an AES-256 and base64 encoded key is used. To generate a proper key, the command openssl rand -base64 32 can be used.

To enable encryption in the Terraform Provider, the configuration must be extended:

terraform {
  backend "s3" {
    ...
    encrypt                     = true
    sse_customer_key            = "fsRb1SXBjiUqBM0rw/YqvDixScWnDCZsK7BhnPTc93Y="
  }
}

Use Tradfri Shortcut Button with Home Assistant

2022-08-22T21:35:25+02:00

Sometimes it can be helpful if single actions or entire automation in your Home Automation can be triggered with a physical button and not only from the Web UI. I’m using Zigbee2MQTT as a generic Zigbee to MQTT bridge as it supports a lot of different Zigbee devices and integrates flawlessly into Home Assistant. As I have a lot of different Tradfri Lamps in use already, I have bought a few of the Tradfri Shortcut Button (E1812).

Pairing the Buttons with the Zigbee2MQTT coordinator was simple as usual, but I had some trouble to figure out how to use it in Home Assistant. Zigbee2MQTT recommends to use the MQTT device triggers but the documentation wasn’t that clear on this part.

The first thing to do is to figure out the available triggers of the device. Triggers are published to the MQTT topic <discovery_prefix>/device_automation/ and you can subscribe to is to discover the messages. To subscribe to an MQTT topic mosquitto_sub CLI command can be used e.g. mosquitto_sub -h 192.168.0.1 -p 8883 -u user -P secure-password -t homeassistant/device_automation/# or the Home Assistant UI at Settings / Devices & Services / Integrations / MQTT / Configure / Listen to a topic.

After subscribing to the topic, remember that device triggers are not published until the event has been triggered at least once on the device. Afterwards the following triggers should be listed for the Tradfri Button:

{
    "automation_type": "trigger",
    "device": {
        "identifiers": [
            "zigbee2mqtt_0x84b112233445566"
        ],
        "manufacturer": "IKEA",
        "model": "TRADFRI shortcut button (E1812)",
        "name": "livingroom/switch_test",
        "sw_version": "2.3.080"
    },
    "payload": "off",
    "subtype": "off",
    "topic": "zigbee2mqtt/livingroom/switch_test/action",
    "type": "action"
}
{
    "automation_type": "trigger",
    "device": {
        "identifiers": [
            "zigbee2mqtt_0x84b112233445566"
        ],
        "manufacturer": "IKEA",
        "model": "TRADFRI shortcut button (E1812)",
        "name": "livingroom/switch_test",
        "sw_version": "2.3.080"
    },
    "payload": "brightness_stop",
    "subtype": "brightness_stop",
    "topic": "zigbee2mqtt/livingroom/switch_test/action",
    "type": "action"
}
{
    "automation_type": "trigger",
    "device": {
        "identifiers": [
            "zigbee2mqtt_0x84b112233445566"
        ],
        "manufacturer": "IKEA",
        "model": "TRADFRI shortcut button (E1812)",
        "name": "livingroom/switch_test",
        "sw_version": "2.3.080"
    },
    "payload": "brightness_move_up",
    "subtype": "brightness_move_up",
    "topic": "zigbee2mqtt/livingroom/switch_test/action",
    "type": "action"
}
{
    "automation_type": "trigger",
    "device": {
        "identifiers": [
            "zigbee2mqtt_0x84b112233445566"
        ],
        "manufacturer": "IKEA",
        "model": "TRADFRI shortcut button (E1812)",
        "name": "livingroom/switch_test",
        "sw_version": "2.3.080"
    },
    "payload": "on",
    "subtype": "on",
    "topic": "zigbee2mqtt/livingroom/switch_test/action",
    "type": "action"
}

Great, with this data determined, it’s almost done. The only missing information is the device_id. I have not found a nice way to get this ID yet, but it is part of the URL after navigating to Settings / Devices & Services / Devices / <device>.

Finally an action can be assigned to a button. The example below is starting the Vacuum cleaner on a single_click (on) trigger:

- alias: Start Vacuum
  trigger:
    - platform: device
      domain: mqtt
      device_id: 61b011e2e7d5e111111d8d804a029f61
      type: action
      subtype: "on"
      discovery_id: 0x84b112233445566 action_on
  action:
    - service: vacuum.start
      target:
        entity_id: vacuum.valetudo_rockrobo

How to (not) migrate Graylog to Opensearch

2022-07-14T14:45:51+02:00

Graylog is a centralized log management solution to capture, store and analyze log files in real-time. Starting with the latest minor release 4.3 Graylog announced to no longer support Elasticsearch (ES) due to licensing and structural changes Elastic introduced in v7.11. For this reason, the last supported ES version is 7.10, which has already reached EOL on May 11, 2022.

Fortunately, Graylog also knows this and recommends users to switch even if it is currently not enforced and ES 7.10 continues to work for now ¹ . As you usually don’t wants to operate software that no longer receives security updates, I have started to look into a migration and prepared the Container and Ansible setup. My fist mistake on this journey was to believe I can just use the latest Opensearch (OS) release. Had I read the documentation ² more carefully I would have saved myself a lot of trouble…

Anyway, the actual migration of the cluster from ES v7.10 to OS v2.1 succeeded surprisingly smoothly. Well, almost, after all I had to rewrite the complete Ansible role because OS 2.x has changed almost all configuration parameters and API calls 🎉 But as you can imagine, everything explodes while trying to start Graylog again. Dang. Just downgrading Opensearch was also not possible as the cluster and all indices were migrated successfully already. To get it back in a working state I decided to reset the entire cluster and restore the snapshots from the S3 backup repository before I start to start a next try, this time with a supported OS 1.x version :fingers_crossed: At least I have already completed the ES disaster recovery test for this year.

Lessons Learned:

Read documentations/upgrade instructions more carefully
Ensure to have a working backup
Test your recovery process frequently to stay calm and comfortable in case of an emergency
Test upgrades in a staging environment whenever possible

What annoys me a bit about the whole situation is the back and forth and the rather bad communication in the past from Graylog ³ . Furthermore, the situation with Opensearch is not really better, as it is unclear if e.g. version 1.3 is still supported or not and a general lifecycle information is still missing ⁴ .

Collect JSON metrics with Telegraf

2022-03-13T12:10:00+01:00

Telegraf is a powerful, plugin based metrics collector that also provides Prometheus compatible outputs. For various purposes, there are a number of input plugins that can collect metrics from various sources. Even more powerful are the processor plugins that allow metrics to be processed and manipulated as they pass through, and immediately output results based on the values they process. In this short blog post I’ll explain how to fetch JSON metrics from the Docker registry API to track some data of a DockerHub Repository.

Even though Prometheus has become very popular, not all applications and API’s provide native Prometheus metrics. In this example, I’ll focus on the repository metrics provided by https://hub.docker.com/v2/repositories. While the API displays the current pull numbers, storing this information to Prometheus has the advantage of displaying time-based information, e.g. the pull frequency/ratio or pulls in a certain period of time.

To fetch metrics from an HTTP endpoint, the generic HTTP input plugin can be used. This plugin allows collecting metrics from one or more HTTP(S) endpoints and also supports a lot of different data formats not only JSON.

By default, only numeric values are extracted from the collected JSON data. In the case of the Docker Registry API endpoint, the status, star_count, pull_count, and collaborator_count fields are used. This behavior can be customized by adding fields for the json_string_fields configuration option. Since Prometheus supports only numeric values for metrics, string fields are added as labels to each metric. In some cases, string fields contain useful information that can also be converted to numeric references. For example, status texts such as OK and ERROR can be converted to 0 and 1 and used as metrics in Prometheus. In the case of the Docker Registry API, I wanted to use the last_updated field as a dedicated metric in Prometheus instead of a label. Fortunately, the date time format used can be transformed to a numeric value by converting it to a Unix timestamp.

For simple type conversions, I would recommend to always check the available converter processor first. Sadly there is no such converter for date or date time values available yet, so I had to go another way and utilized the starlark processor. This processor calls a starlark function for each matched metric, allowing for custom programmatic metric processing. While starlark is a Python dialect and might look familiar, it only supports a very limited subset of the language, as explained in the specification. But it’s powerful enough for what I wanted to do: Convert the date time value in the format of 2006-01-02T15:04:05.999999Z to a valid Unix timestamp. That’s how the final configuration looks like:

#jinja2: lstrip_blocks: True
[[inputs.http]]
  name_override = "dockerhub_repository"
  urls = [
    "https://hub.docker.com/v2/repositories/library/telegraf/"
  ]

  json_string_fields = [
    "last_updated",
    "name",
    "namespace",
    "repository_type",
  ]

  data_format = "json"

[[processors.starlark]]
  namepass = ["dockerhub_repository"]

  source = '''
load("time.star", "time")
def apply(metric):

  metric.fields["last_updated"] = time.parse_time(metric.fields["last_updated"], format="2006-01-02T15:04:05.999999Z").unix

  return metric
'''

To debug and test the Telegraf configuration it’s useful to execute the binary with the --test flag:

./usr/bin/telegraf --debug --test --config etc/telegraf/telegraf.conf --config-directory etc/telegraf/telegraf.d/
2022-03-12T17:01:07Z I! Starting Telegraf 1.21.4
2022-03-12T17:01:07Z I! Loaded inputs: http
2022-03-12T17:01:07Z I! Loaded aggregators:
2022-03-12T17:01:07Z I! Loaded processors: starlark
2022-03-12T17:01:07Z W! Outputs are not used in testing mode!
2022-03-12T17:01:07Z I! Tags enabled: host=localhost project=prometheus
[...]
> dockerhub_repository,host=localhost,name=telegraf,namespace=library,project=prometheus,repository_type=image,url=https://hub.docker.com/v2/repositories/library/telegraf/ collaborator_count=0,last_updated=1646267076i,pull_count=519804664,star_count=560,status=1 1647104469000000000

The final metrics generated by the Prometheus output plugin will look like this:

dockerhub_repository_collaborator_count{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 0
dockerhub_repository_last_updated{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 1.646267076e+09
dockerhub_repository_pull_count{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 5.19802966e+08
dockerhub_repository_star_count{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 560
dockerhub_repository_status{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 1

That’s it. The Telegraf HTTP input plugin is a very flexible but generic way to collect and transform JSON metrics from various sources. If that’s still not powerful enough you can pass the raw data fetched by the HTTP input plugin to the starlark converter and write your own functions to parse the input and extract the required information into metrics.

SSL certificate monitoring pitfalls

2022-01-31T23:00:00+01:00

Certificates are a fundamental part of the Internet’s security. At least since Let’s Encrypt, a free and automated Certificate Authority, has started its service, SSL is nearly used everywhere. To avoid Certificate issues and possible service outages, it’s a good idea to monitor the SSL certificates used by your services, especially as Let’s Encrypt certificates have a short lease time of 90 days.

I’m using Prometheus to monitor my infrastructure, and for Prometheus there are multiple ways to get started. Most of the tutorials and posts of the internet will cover the case of expired certificates, and it’s pretty easy to achieve. I prefer to use Telegraf, a plugin based metrics collector that also provides Prometheus compatible outputs, instead of dedicated Prometheus exporters. To monitor SSL certificates, I’m using the x509_cert input plugin of Telegraf that provides a metric called x509_cert_expiry which can be utilized to write simple alerting rules. That’s actually pretty cool already, as Prometheus will send out alerts a few weeks before the certificates would expire in case there is a problem within the automatic renewal process.

A week ago, Let’s Encrypt has informed affected users that they need to revoke faulty certificates issued and validated with the TLS-ALPN-01 challenge. Even if I’m using the DNS-01 for almost all of my certificates, I have also received a mail and started to look into it. Sadly, the notification mail only contained a “random” ACME registration ID, and I was not able to find the matching client. As mentioned, I don’t really use TLS-ALPN-01, so I decided to stop the research and leave it to my monitoring to tell me which forgotten service is the evil one after the certificates were revoked. Nothing happened after the revocation, and the monitoring was not complaining. Good - well no, a user reported that one of the services is not reachable anymore and of course this was the one missing client that was using TLS-ALPN-01 verified certificates - dang. While the issue itself was easy to resolve by a force renew of the certificate, I was still wondering why the monitoring has not caught it.

Well, this was the first time that I had to deal with revoked certificates instead of expired certificates. To be honest, I never thought about the detection of revoked certificates in my monitoring setup before, and therefore this case wasn’t covered. But it looks like a fix is also not that straight forward as expected. The used Telegraf input x509_cert is not able to detect revoked certificates yet, and the common Prometheus blackbox_exporter also don’t want to handle this case. The only way I have found so far is to use the ssl_exporter that provides some revocation information of the certificates using OSCP. If you are already running multiple exporters, that might be the way to go for you. Personally, I prefer to handle as much as possible using Telegraf, so I might look into a fix for the x509_cert during the next weeks. However, lessons learned 📘

Toolbox 2: git-plus

2021-06-19T09:50:00+01:00

If you work with a lot of Git repositories on a regular basis, you’re bound to run into the situation where you need to make changes to multiple repositories sooner or later. While it would be possible to run your Git commands in a shell loop over everything repositories, it is often tedious to type the command or remember the correct syntax.

This is where git-plus comes into play. It is a small collection of Git utilities that tries to simplify some common tasks:

git multi execute a single Git command on multiple Git repositories
git relation show a relation between two branches/commits/tags
git old-branches find old/unused branches
git recent list branches ordered by last commit time
git semver lists and creates Git semver (semantic versioning) tags

But there is one command that I use every day now, git multi. The command works folder based, that means you have to have all target repositories in one directory. There is no grouping feature at the moment, if you want to group some repositories you would have to use multiple sub-directories. Since I also use direnv, this folder-based workflow works very well for me. Repositories can be temporarily excluded with the CLI flag -e reponame or by a .multigit_ignore file in the parent directory. Basically git multi executes normal Git commands and is therefore quite easy to use. One example where the tool helps me is when I need to adjust the CI configuration in multiple repositories, for example. Only a few steps are needed for this:

Create a new branch:
git multi checkout -b "replace-ci-step"
Use your favorite search and replace tool to add your changes to all repositories
Create a commit:
git multi add -A; git multi commit -m "ci: replace broken step in ci config"
Finally push your new branch:
git multi push origin replace-ci-step

That’s it. Simple, isn’t it? Of course it doesn’t work in all cases, especially for more complex and repository specific changes it’s getting harder, but for generic changes like CI configurations, the current year in a copyright string or a license file or globally used badges in the readme it’s really straight forward.

git-plus is written in Python and available on PyPi.

Toolbox 1: direnv

2021-05-17T21:24:00+01:00

We all use many different tools every day e.g. for our work, automation or better productivity. In the series “Toolbox” I would like to present such applications that have made my day-to-day work so much easier. All applications are free and open source software developed by big tech companies as well as lovingly handcrafted hobby projects. If you also know an awesome tool that has changed your life, I would love to hear from you on Mastodon.

direnv

The basic function of direnv is pretty simple, it manages environment variables depending on the current directory you are in. This may sound not very helpful, but it could save a lot of work, especially if you have to deal with many different project environments. I use it a lot combined with Ansible. Depending on the deployment environment I need to set a different remote user, also the roles should be loaded from a different base directory depending on whether it is a test or production environment.

As long as the required configuration can be set by an environment variable, direnv can handle it. All you have to do is to create a .envrc file in the directory where it should be loaded. A simple configuration could look like this:

$ cat .envrc
export ANSIBLE_ROLES_PATH=/home/xoxys/devel/.roles/staging
export ANSIBLE_REMOTE_USER=project-1

Each time you create or modify an .envrc file, you must approve the modification by running direnv allow. This prevents direnv from loading a file that has been modified or created by another malicious process, for example. After you have approved the file, direnv will automatically load it when you navigate to the directory where your configuration is located. To make it even better, it doesn’t have to be the root directory, the file will also be loaded if you navigate directly to a sub-directory. When you leave the directory, your environment will also be unloaded automatically.

Direnv is written in Golang and available in most Linux Distribution repositories. Binary builds can be downloaded from the GitHub Release page as well.

Run an ARM32 Docker daemon on ARM64 servers

2020-09-24T10:30:00+02:00

In the last days I worked on a suitable setup for a Drone CI server to support multi-arch builds. While the setup for common x86 Drone runners is easy, working with setups for ARM, especially ARM32, is a bit tricky. The easiest way would be to have native servers of the respective architecture available. However, it’s difficult to find hosting offers for ARM at all - for ARM32 this seems almost impossible. I decided to use Amazon EC2 ARM64 servers, they are relatively cheap and can also be used as a private customer.

But how do you turn an ARM64 server into an ARM32 server? Basic requirement is an ARMv8 CPU. This type supports (in most cases) both AArch32 and AArch64. The first step is to enable multi-arch support on your operating system. I use Ubuntu 18.10 for this setup:

dpkg --add-architecture armhf

Then add the Docker CE repository for ARM32 and install the packages:

# add GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# add repos
add-apt-repository \
   "deb [arch=armhf] https://download.docker.com/linux/ubuntu/ \
   $RELEASE \
   stable

# install
apt-get update
apt-get install docker-ce:armhf

To use the right architecture within Docker containers you still have to force the Docker daemon into ARM32 mode. This can be accomplished by two systemd overwrites:

# overwrite docker.service
# /etc/systemd/system/docker.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/setarch linux32 -B /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

# overwrite containerd.service
# /etc/systemd/system/containerd.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/setarch linux32 -B /usr/bin/containerd

That’s it. Don’t forget to reload and restart the systemd daemon:

systemctl, daemon reload
systemctl, restart, docker

What you get is a Docker daemon that runs in ARM32 mode. This can be used for example to start a Golang container and build apps without cross-compile.

root@ip-10-0-225-151:~# docker version
Client: Docker Engine - Community
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46
 Built:             Wed Sep 16 17:07:23 2020
 OS/Arch:           linux/arm
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46
  Built:            Wed Sep 16 17:01:08 2020
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
root@ip-10-0-225-151:~# docker run alpine uname -a
Linux dbad5ddeb5ea 5.3.0-1035-aws #37-Ubuntu SMP Sun Sep 6 01:17:41 UTC 2020 armv8l Linux

Just a word of warning. I’m not a hardware specialist and there might be some situations where this setup does not work. Furthermore it seems that some operating systems have problems to recognize armv8l as ARM32 architecture. For OpenSuse as an example I had to force zypper into armv7hl mode to get it working. After this step building ARM32 OpenSuse Docker images also works for me.

sed -i 's/# arch = s390/arch = armv7hl/g' /etc/zypp/zypp.conf

If you want to give it a try, I’ve prepared a minimal Cloud-Init configuration:

#cloud-config

apt_reboot_if_required: false
package_update: true
package_upgrade: true

bootcmd:
  - [ dpkg, --add-architecture, armhf  ]

apt:
  sources:
    docker.list:
      source: deb [arch=armhf] https://download.docker.com/linux/ubuntu/ $RELEASE stable
      keyid: 0EBFCD88

packages:
  - 'docker-ce:armhf'

write_files:
  - path: /etc/systemd/system/docker.service.d/override.conf
    content: |
      [Service]
      ExecStart=
      ExecStart=/usr/bin/setarch linux32 -B /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

  - path: /etc/systemd/system/containerd.service.d/override.conf
    content: |
      [Service]
      ExecStart=
      ExecStart=/usr/bin/setarch linux32 -B /usr/bin/containerd

runcmd:
  - [ systemctl, daemon-reload ]
  - [ systemctl, restart, docker ]

Color palettes on Game Boy Color and Advance

2020-09-15T21:45:00+02:00

In my last article about modernizing a Game Boy Advance, one of my complaints was the ugly coloring of old Game Boy Classic games and I thought about buying an additional Game Boy Classic. But first I wanted to check if there was a mod for the Game Boy Advance to fix this problem.

Well, what I found out after a short research was mind-blowing for me! The gorgeous engineers at Nintendo had already integrated a function into the Game Boy Color and the Game Boy Advance to choose between 12 color palettes.

Color	Shortcut
BROWN	UP
BLUE	LEFT
PASTEL	DOWN
GREEN	RIGHT
RED	UP + A
DARK BLUE	LEFT + A
ORANGE	DOWN + A
DARK GREEN	RIGHT + A
DARK BROWN	UP + B
GRAY SCALE	LEFT + B
YELLOW	DOWN + B
NEGATIVE	RIGHT + B

Comparison between default color (left) and gray scale (right).

Today I learned about a great feature that I’ve never known in all the years of my childhood. But there is still a small downside: Changing the color palette works only for GB classic games.

How to modernize a Game Boy

2020-09-13T23:45:00+02:00

Sometimes it is quite nice to revel in memories. Back to the good old days of our childhood where life was so easy. At least I have those retro feelings from time to time. This time I decided to follow these feelings and take back a part of my childhood. I wanted a Game Boy again. A real Game Boy, not a fake one or any of those emulator handhelds.

The first consideration was which model to choose: Game Boy (GB), Game Boy Color (GBC) or Game Boy Advance/SP (GBA)? From a practical point of view, the Game Boy Advance is the most worthwhile. It is backwards compatible and can play GB and GBC games as well as GBA games. I’ve never been a fan of the form factor of the SP version, so I decided to go for the classic model. Regardless of my retro feelings, the Game Boy has its weaknesses, which are simply no longer up to date. Even if you can argue that this is part of the feeling, that’s a bit too much for me.

Overview of all modding parts.

Fortunately the Game Boy is still very popular and has a strong modding community. After some research I found a sufficient IPS display mod as well as a replacement case that doesn’t need modifications to fit the new display. As most users were also very impressed by an audio mod kit I also wanted to give it a try. There are a hand full of different mod kit providers out there and a lot more if you want to look at eBay or Aliexpress. But as some users noticed a lot of quality ups and downs I decided to go with the parts from retrosix.

After a few days all parts arrived and I could finally start. The updates are all very easy to install. The display doesn’t require any soldering. Optionally brightness control cables could be soldered to the shoulder buttons. The audio mod kit includes an amplifier and a new speaker. The installation in general is simple, but since the soldering points are sometimes very small, maybe this should not be your first soldering experience.

Soldered brightness control and audio mod kit.

After about one hour all parts were installed and cleaned and the Game Boy was reassembled. Time for a first test. What can I say, the effort was definitely worth it, the case feels valuable and all buttons have a good reaction and haptic feedback. Well, the display is the real star. Everything is bright and crisp and the colors are nice and natural.

Comparison between GB games (left) and GBA games (right).

At the end I’m very happy with my “new” handheld but there are also some downsides I wanted to talk about:

For me, the audio mod kit was a bad investment and I had to remove the amplifier board after a few hours of gaming. I agree the sound of the original speaker may be a bit flat and isn’t very loud. But thanks to the amplifier, the new speaker constantly screams into my face… For me, it was impossible to find a sufficient and comfortable volume level. In the lower volume range the wheel jumped from still to loud to completely silent and could not be controlled precisely. Personally, the default maximum volume level is totally fine but that might be a personal preference.
To buy original Game Boy games could be a real challenge these days. I’m not really surprised about the prices, especially for rare games but there seems to be a lot fake games or replications out there. If you want to go with replications to save a bit money is up to you but if you just want authentic/original cartridges you have to be very careful.
While I really love playing all GBA and GBC games with the GBA, I don’t really like the look of the classic GB games because of the ugly color scheme. This has nothing to do with the IPS screen directly, as far as I remember, this was already the case with the original GBC and GBA. The color scheme of some games really hurts my eyes and makes it almost impossible to play them. ~~Maybe I still need a Game Boy classic after all.~~ There is already a solution!

Docker port publishing for localhost bindings

2020-09-08T22:15:00+02:00

While preparing a custom Docker image for a tool I wanted to use I encountered a problem that kept me busy for some time. The container could be built and started without any problems but the application in the container was simply not accessible via the published port.

Even after minutes of debugging and checking (and re-checking over and over again) that the right port was exposed and the application in the container is listening on that port I was not able to get it to work… But I had a suspicion now. For some reason I had decided to bind the application in the container to localhost. Though it may sound obvious now, I didn’t expect that with a binding like 127.0.0.0:9000 you can’t publish port 9000 to the host; or maybe I’m just too stupid to fully understand Docker networking. After I changed the binding to 0.0.0.0:9000 everything worked as expected. Anyway, lesson learned.

Ansible and the relations to the inventory

2020-08-03T22:45:00+02:00

I love Ansible and I’m pretty happy to have this configuration management solution. But some little “features” drive me crazy sometimes. My task for today was to switch from a static Ansible inventory file to multiple dynamic inventory script. In general this, should be really straight forward. Create an inventory folder and add some small Python scripts to create Ansible readable inventories from different providers.

I’ve finished the first steps really quickly. After I’ve added a script for my home lab Proxmox VE host and a static inventory for a bunch of my Raspberries, I did a check with ansible-inventory -i inventory/ --list and everything seems to be working as expected. All my hosts were listed and Ansible groups were applied as well. But here comes the clue: The test Playbook run failed, it looks like Ansible was not able to find the required variables. What the heck…

Note

Pro tip
There is one thing you should keep in mind, variables are related to the inventory file(s) or Playbooks.

Let’s assume your Ansible structure looks like this:

├── group_vars
│   └── all.yaml
├── host_vars
│   └── ...
├── inventory
└── playbooks
    └── ...

Everything should work as expected, Ansible should be able to lookup the required group_vars and host_vars. After you switched to multiple inventories the structure might look this way:

├── group_vars
│   └── all.yaml
├── host_vars
│   └── ...
├── inventory
│   ├── dynamic_script.py
│   └── static
└── playbooks
    └── ...

At this point Ansible will fail and complain about missing variables. Well, as mentioned above, variables are related to the inventory file(s)! As inventory is no longer a single file, this requirement is not met anymore. To get it working again I had to symlink the group_vars and host_vars folder into the inventory folder. Obviously, right?

Create a static site hosting platform

2020-07-30T01:05:00+02:00

There are a lot of static site generators out there and users have a lot of possibilities to automate and continuously deploy static sites these days. Solutions like GitHub pages or Netlify are free to use and easy to set up, even a cheap web space could work. If one of these services is sufficient for your use case you could stop reading at this point.

As I wanted to have more control over such a setup and because it might be fun I decided to create my own service. Before looking into the setup details, lets talk about some requirements:

deploy multiple project documentation
use git repository name as subdomain
easy CI workflow

The required software stack is quite simple:

Nginx as web server to deliver static files
Minio S3 as files backend

Of course, Minio could be removed from the stack but after a few tests, my personal impression was that Minio is a way better to handle file sync and uploads instead over SSH/SCP, at least in a CI driven workflow. The whole workflow is nearly the same as for GitHub pages. Right beside your projects source code in the Git repository you will have a docs folder which contains the required files to build the documentation site. It’s up to you what static site generator to use. Instead of using e.g. the gh-pages branch to publish the site, a CI system will build the documentation form the docs folder and publish it to a prepared Minio S3 bucket. The Nginx server will basically use the defined bucket as source directory and convert every sub-directory into something like https://<reponame>.mydocs.com.

As a first step, install Minio and Nginx on a server, I will not cover the basic setup in this guide. To simplify the setup I will use a single server for Minio and Nginx but it’s also possible to split this into a Two-Tier architecture.

After the basic setup it’s required to create a Minio bucket e.g. mydocs using the Minio client command mc mb local/mydocs. To allow Nginx to access these bucket to deliver the pages without authentication a bucket policy mc policy set download local/mydocs need to be set. This policy will allow public read access. In theory, it should also be possible to add authentication headers to Nginx to server sites from private buckets but I have not tried that on my own.

Preparing the Minio bucket was the easy part, now Nginx need to know how to rewrite the subdomains to sub-directories and properly deliver the sites. Let’s assume mydocs is still used as the base Minio bucket and mydocs.com as root domain. Here is how my current vHost configuration looks like:

upstream backend_mydocs {
    server localhost:61000;
}

server {
    listen       80;
    server_name  ~^(www\.)?(?<name>(.+\.)?mydocs\.com)$;

    return 301 https://$name$request_uri;
}

server {
    listen       443 ssl;
    server_name  ~^((?<repo>.*)\.)mydocs\.com$;

    ssl_certificate [..];
    ssl_certificate_key [..];
    client_max_body_size 100M;

    recursive_error_pages on;

    location / {
        proxy_pass http://backend_mydocs/mydocs/${repo}${request_path};
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_connect_timeout 300;
        proxy_intercept_errors on;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        rewrite ^([^.]*[^/])$ $1/ permanent;
        error_page 404 = /404_backend.html;
    }

    location /404_backend.html {
        proxy_pass http://backend_mydocs/mydocs/${repo}/404.html;
        proxy_intercept_errors on;
    }
}

We will go through this configuration to understand how it works.

Lines 1-3 defines a backend, in this case it’s the Minio server running on localhost:61000.

Lines 5-10 should also be straight forward, this block will redirect HTTP to HTTPS.

Line 14 is where the magic starts. A named regular expression is used to capture the first part of the subdomain and translate it into the bucket sub-directory. For a given URL like demo-project.mydocs.com Nginx will try to serve mydocs/demo-project from the Minio server. That’s what Line 23 does. Some of you may notice that the used variable ${request_path} is not defined in the vHost configuration.

Right, another configuration snippet needs to be added to the nginx.conf. But why is this variable required at all? For me, that was the hardest part to solve. As the setup is using proxy_pass Nginx will not try to lookup index.html automatically. That’s a problem because every folder will at least contain an index.html. In general, it’s required to tell Nginx to rewrite the request URI to /index.html if the origin is a folder and ends with /. One way would be an if condition in the vHost configuration but such conditions are evil¹ in most cases and should be avoided if possible. Luckily there is a better option:

map $request_uri $request_path {
    default $request_uri;
    ~/$ ${request_uri}index.html;
}

Nginx maps are a solid way to create conditionals. In this example set $request_uri as input and $request_path as output. Each line between the braces is a condition. The first line will simply apply $request_uri to the output variable if no other condition match. The second condition applies ${request_uri}index.html to the output variable if the input variable ends with a slash (and therefore is a directory).

Line 38-41 of the vHost configuration tries to deliver the custom error page of your site and will fallback to the default Nginx error page.

We are done! Nginx should now be able to server your static sites from a sub-directory of the Minio source bucket. I’m using it since a few weeks and I’m really happy with the current setup.

This article from the Nginx team explains it very well. ↩︎

Welcome (back)

2020-07-21T23:00:08+02:00

As some former readers may have noticed, “geeklabor.de” has been renamed to “thegeeklab.de”, welcome back nice to have you here again. If you are a first time visitor, a very warm welcome goes to you as well. This is my private blog, where I write about everything that comes to my mind but mainly about topics from the Linux and Open Source world.

For those of you who are interested in the backgrounds about the blog migration, here you go:

my old theme had to be reworked, hugo-geekblog was born
- looks pretty much the same as before but with a more up to date technical implementation
- works with recent Hugo versions
- fully Open Source
“geeklabor.de” does only work in German language, “thegeeklab.de” is bit more international
the same applies to the entire blog, all content is only available in English for two simple reasons
- English works for a lot more people
- I am too lazy to provide posts in multiple languages
the blog is running on a Hetzner Cloud machine; CI driven and using Minio S3 backend

That is a short summary right now. While I am currently still working on some remaining migration tasks, there will be more fresh posts during the next weeks - stay tuned!