alpine-buildmaster
MultiArch Alpine Linux + S6 + Python3 + Buildbot (Master)
This image serves as the base container for running a Buildbot master instance to build applications, run periodic tasks, and other automations. Checkout their docs for configurations or task definitions. Also, checkout the alpine-buildworker image for running a standalone worker as a service.
Based on Alpine Linux from the python3 image with the buildbot package(s) installed in it.
Get the Image¶
Pull the image from Docker Hub.
Image Tags
The image is tagged respectively for the following architectures,
latest tag is annotated as multiarch so pulling without any tags should fetch the correct image for your architecture. Same goes for any of the version tags.
non-x86_64 builds have embedded binfmt_misc support and contain the qemu-user-static binary that allows for running it also inside an x86_64 environment that has support for it.
Run¶
Running the container starts the service.
docker run --rm \
--name docker_buildmaster \
-c 256 -m 256m \
-p 8010:8010 \
-p 9989:9989 \
-p 9990:9990 \
-p 9991:9991 \
-v $PWD/buildmaster`#`:/home/alpine/buildbot \
woahbase/alpine-buildmaster
Multi-Arch Support
If you want to run images for other architectures on a x86_64 machine, you will need to have binfmt support configured for your machine before running the image. multiarch, has made it easy for us containing that into a docker container, just run
Now images built for other architectures will also be executable. This is optional though, without the above, you can still run the image that is made for your architecture.
Configuration¶
We can customize the runtime behaviour of the container with the following environment variables.
ENV Vars | Default | Description |
---|---|---|
BUILDBOT_HOME | /home/alpine/buildbot | Default root directory for buildbot configurations. |
BUILDBOT_PROJECTNAME | buildbot | Project name that is prepended to the master name, e.g. default is buildbot -master. |
BUILDBOT_SETUP_ARGS | --force --log-count=2 --log-size=5000 --relocatable | These arguments are passed to setup the master. |
BUILDBOT_SKIP_SETUP | unset | Skips master setup tasks, useful when your already have your configurations setup, or would like to do it manually. |
BUILDBOT_MASTERNAME | ${BUILDBOT_PROJECTNAME}-master | Name of the service, defaults to projectname-rolename . |
BUILDBOT_MASTERCFG | ${BUILDBOT_MASTERNAME}/master.cfg.sample | Path to custom configuration file that is copied into place as master.cfg before starting service. |
BUILDBOT_USE_CUSTOM_TACFILE | unset | Whether to use custom tacfile provided in the image that logs to stdout by default, set to a non-empty string (e.g 1 ) to enable, or use the one generated by package. |
BUILDBOT_CUSTOM_TACFILE | /defaults/master.tac | Customizable path to tacfile provided in the image. |
BUILDBOT_BASEDIR | unset | Used in the custom tacfile to determine where builder files are stored, defaults to "." when unset (current directory where buildbot.tac exists), or any other directory (must exist). |
BUILDBOT_LOGDEST | stdout | Used in the custom tacfile to determine where logs are sent, can be either of stdout (default), syslog , or file . |
BUILDBOT_LOGROTATE_LENGTH | 5000 | Used in the custom tacfile to determine maximum lines-in-logfile before it is rotated. |
BUILDBOT_LOGROTATE_MAXFILES | 2 | Used in the custom tacfile to determine maximum rotated logfiles that are kept in storage. |
BUILDBOT_CONFIG_URL | unset | If set, tries to fetch configuration from remote location. Can be a tar.gz file, or a git repository, or a file link to master.cfg itself. |
BUILDBOT_CONFIG_DIR | ${BUILDBOT_HOME}/config | Configurations fetched from remote location unpacked in this directory. |
BUILDBOT_CONFIG_TMP | ${BUILDBOT_HOME}/.tmp | Configurations fetched from remote location are temporarily stored in this directory. (Useful if /tmp is non-writable or limited space) |
BUILDBOT_CONFIG_CFGFILE | master.cfg | Customizable path to master.cfg as relative to downloaded configurations. Default expects at the root of directory, if it exists, it is copied into place as master.cfg before starting service. |
BUILDBOT_CONFIG_TACFILE | buildbot.tac | Customizable path to buildbot.tac as relative to downloaded configurations. Default expects at the root of directory, if it exists, it is copied into place as buildbot.tac before starting service. |
BUILDBOT_SKIP_CHECKCONFIG | unset | If true . skips checking configurations before starting service. |
BUILDBOT_UPGRADE_MASTER | unset | If true , upgrades master database before starting service. |
BUILDBOT_CLEANUP_DB | unset | If true , runs cleanup tasks on master database before starting service. |
BUILDBOT_WORKERNAME | ${BUILDBOT_PROJECTNAME}-worker | Name of the default worker updated in master.cfg . (Only when using the sample configurations) |
BUILDBOT_WORKERPASS | insecurebydefault | Password of the default worker updated in master.cfg . (Only when using the sample configurations) |
BUILDBOT_SKIP_CUSTOMIZE | unset | Skip post-setup customization tasks. |
BUILDBOT_SKIP_PERMFIX | unset | Skip ensuring files in ${BUILDBOT_HOME} are owned by ${S6_USER} , enabled by default. |
BUILDBOT_ARGS | --nodaemon --no_save | Customizable arguments passed to master service. (Runs as a twisted application instead of calling buildbot executable) |
S6_PIP_PACKAGES | empty string | Space-separated list of packages to install globally with pip . |
S6_PIP_REQUIREMENTS | empty string | Path to requirements.txt to install globally with pip . |
S6_PIP_USER_PACKAGES | empty string | Space-separated list of packages to install with pip for S6_USER . These are installed in ~/.local/ . |
S6_PIP_USER_REQUIREMENTS | empty string | Path to requirements.txt to install with pip for S6_USER . |
S6_NEEDED_PACKAGES | empty string | Space-separated list of extra APK packages to install on start. E.g. "curl git tzdata" |
PUID | 1000 | Id of S6_USER . |
PGID | 100 | Group id of S6_USER . |
S6_USER | alpine | (Preset) Default non-root user for services to drop privileges to. |
S6_USERHOME | /home/alpine | (Preset) HOME directory for S6_USER . |
Did you know?
You can check your own UID/GID by running the command id
in a terminal.
Also,
-
The env variable
BUILDBOT_ROLE
determines if you are running a master or worker. This also determines what image you'll be running when used with themakefile
. It is baked into the image so does not need to be changed unless you know what you're doing. -
Setup tasks are only run when the
buildbot.tac
file does not exist orBUILDBOT_SKIP_SETUP
is not set. Same goes for arguments / environment variables specific to setup, they are not needed anymore after setup is complete. -
However, if
BUILDBOT_MASTERCFG
is defined, it will always be copied into project before starting the service. This is useful for large projects that maintain the configurations separately from master configurations. -
Includes a placeholder script for further customizations before starting processes. Override the shellscript located at
/etc/s6-overlay/s6-rc.d/p22-buildbot-customize/run
with your custom pre-tasks as needed. -
The service does not run
buildbot
for master, instead callstwistd
directly, passBUILDBOT_ARGS
accordingly. -
For the master node, mount the configurations at the
BUILDBOT_HOME
directory inside the container, by default it is/home/alpine/buildbot
.
Stop the container with a timeout, (defaults to 2 seconds)
Restart the container with
Removes the container, (always better to stop it first and -f
only when needed most)
Shell access¶
Get a shell inside a already running container,
Optionally, login as a non-root user, (default is alpine
)
Or set user/group id e.g 1000/100,
Logs¶
To check logs of a running container in real time
As-A-Service¶
Run the container as a service with the following as reference (and modify it as needed).
With docker-compose (alpine-buildmaster.yml)
---
services:
buildmaster:
container_name: buildmaster
# depends_on:
# mysql:
# condition: service_healthy
deploy:
resources:
limits:
cpus: '2.00'
memory: 1024M
restart_policy:
condition: on-failure
delay: 10s
max_attempts: 5
window: 120s
environment:
BUILDBOT_PROJECTNAME: ${BUILDMASTER_PROJECTNAME:-buildbot}
# BUILDBOT_MASTERCFG: /custom/master.cfg
# BUILDBOT_USE_CUSTOM_TACFILE: 1
# BUILDBOT_SKIP_SETUP: 1
# # configurations from remote source
# BUILDBOT_CONFIG_URL=https://github.com/buildbot/buildbot/releases/download/v3.11.9/buildbot-3.11.9.tar.gz
# BUILDBOT_CONFIG_CFGFILE=buildbot/scripts/sample.cfg
# # or
# BUILDBOT_CONFIG_URL=https://github.com/buildbot/buildbot.git
# BUILDBOT_CONFIG_CFGFILE=master/docker-example/master.cfg
# BUILDBOT_CONFIG_TACFILE=master/contrib/docker/master/buildbot.tac
# # or
# BUILDBOT_CONFIG_URL=https://raw.githubusercontent.com/buildbot/buildbot/refs/heads/master/master/docker-example/master.cfg
BUILDBOT_SKIP_CHECKCONFIG: ${BUILDMASTER_SKIP_CHECKCONFIG:-false}
BUILDBOT_UPGRADE_MASTER: ${BUILDMASTER_UPGRADE_MASTER:-false}
BUILDBOT_CLEANUP_DB: ${BUILDMASTER_CLEANUP_DB:-false}
PUID: ${PUID}
PGID: ${PGID}
TZ: ${TZ}
# healthcheck:
# interval: 2m
# retries: 5
# start_period: 5m
# test:
# - CMD-SHELL
# - >
# wget --quiet --tries=1 --no-check-certificate --spider http://localhost:8010/ || exit 1
# timeout: 10s
hostname: buildmaster
image: woahbase/alpine-buildmaster:${BUILDMASTER_TAG:-latest}
network_mode: bridge
ports:
- protocol: tcp
host_ip: 0.0.0.0
published: ${BUILDMASTER_WEB_PORT:-8010}
target: 8010
- protocol: tcp
host_ip: 0.0.0.0
published: ${BUILDMASTER_PB_PORT:-9989}
target: 9989
- protocol: tcp
host_ip: 0.0.0.0
published: ${BUILDMASTER_CLI_PORT:-9990}
target: 9990
- protocol: tcp
host_ip: 0.0.0.0
published: ${BUILDMASTER_METRICS_PORT:-9991}
target: 9991
volumes:
- type: bind
source: ${BUILDMASTER_DIR:?err}
target: /home/alpine/buildbot
bind:
create_host_path: false
# # pass your own master.cfg
# # gets copied to proper place before service start
# - type: bind
# source: ${BUILDMASTER_DIR:?err}/custom_config/master.cfg
# target: /custom/master.cfg
# bind:
# create_host_path: false
- type: bind
source: /etc/localtime
target: /etc/localtime
read_only: true
bind:
create_host_path: false
With HashiCorp Nomad (alpine-buildmaster.hcl)
variables {
dc = "dc1" # to load the dc-local config file
pgid = 100 # gid for docker
puid = 1000 # uid for docker
version = "3.11.9"
}
# locals { var = yamldecode(file("${var.dc}.vars.yml")) } # load dc-local config file
job "buildmaster" {
datacenters = [var.dc]
# namespace = local.var.namespace
priority = 70
# region = local.var.region
type = "service"
constraint { distinct_hosts = true }
group "docker" {
count = 1
restart {
attempts = 2
interval = "2m"
delay = "15s"
mode = "fail"
}
update {
max_parallel = 1
min_healthy_time = "10s"
healthy_deadline = "3m"
auto_revert = false
}
service {
name = NOMAD_JOB_NAME
port = "http"
tags = ["ins${NOMAD_ALLOC_INDEX}", attr.unique.hostname, "urlprefix-/buildbot"]
canary_tags = ["canary${NOMAD_ALLOC_INDEX}", "urlprefix-/c/buildbot"]
check {
name = "${NOMAD_JOB_NAME}@${attr.unique.hostname}:${NOMAD_HOST_PORT_http}"
type = "http"
path = "/"
interval = "60s"
timeout = "10s"
}
check_restart {
limit = 3
grace = "10s"
}
}
service {
name = "${NOMAD_JOB_NAME}-pb"
port = "pb"
tags = ["ins${NOMAD_ALLOC_INDEX}", attr.unique.hostname, "host=build", "proto=tcp"]
canary_tags = ["canary${NOMAD_ALLOC_INDEX}"]
check {
name = "${NOMAD_JOB_NAME}@${attr.unique.hostname}:${NOMAD_HOST_PORT_pb}"
type = "tcp"
interval = "60s"
timeout = "10s"
}
check_restart {
limit = 3
grace = "10s"
}
}
ephemeral_disk { size = 128 } # MB
network {
# dns { servers = local.var.dns_servers }
port "http" { static = 8010 }
port "pb" { static = 9989 }
port "cli" { static = 9990 }
port "metrics" { static = 9991 }
}
volume "nomad-buildmaster-project" {
type = "host"
read_only = false
source = "nomad-buildmaster-project"
}
task "buildmaster" {
driver = "docker"
config {
healthchecks { disable = true }
hostname = NOMAD_JOB_NAME
image = "woahbase/alpine-buildmaster:${var.version}"
network_mode = "bridge"
ports = ["http", "pb", "cli", "metrics"]
logging {
type = "journald"
config {
mode = "non-blocking"
tag = NOMAD_JOB_NAME
}
}
# mount {
# source = "local/buildbot.tac"
# target = "/home/alpine/buildbot/buildbot-master/buildbot.tac"
# type = "bind"
# readonly = false
# }
# mount {
# source = "local/master.cfg"
# target = "/custom/master.cfg"
# type = "bind"
# readonly = false
# }
mount {
type = "bind"
target = "/etc/localtime"
source = "/etc/localtime"
readonly = true
}
}
volume_mount {
# ensure policies allow vault-generated-token to read-write to the volume
volume = "nomad-buildmaster-project"
destination = "/home/alpine/buildbot"
read_only = false
}
env {
BUILDBOT_PROJECTNAME = "buildbot"
# BUILDBOT_MASTERCFG = "/custom/master.cfg"
# BUILDBOT_USE_CUSTOM_TACFILE = "1"
# BUILDBOT_SKIP_SETUP = "1"
# # configurations from remote source
# BUILDBOT_CONFIG_URL = "https://github.com/buildbot/buildbot/releases/download/v3.11.9/buildbot-3.11.9.tar.gz"
# BUILDBOT_CONFIG_CFGFILE = "buildbot/scripts/sample.cfg"
# # or
# BUILDBOT_CONFIG_URL = "https://github.com/buildbot/buildbot.git"
# BUILDBOT_CONFIG_CFGFILE = "master/docker-example/master.cfg"
# BUILDBOT_CONFIG_TACFILE = "master/contrib/docker/master/buildbot.tac"
# # or
# BUILDBOT_CONFIG_URL = "https://raw.githubusercontent.com/buildbot/buildbot/refs/heads/master/master/docker-example/master.cfg"
BUILDBOT_SKIP_CHECKCONFIG = "true"
BUILDBOT_UPGRADE_MASTER = "false"
BUILDBOT_CLEANUP_DB = "false"
PGID = var.pgid
PUID = var.puid
# TZ = local.var.tz
}
resources {
cpu = 1000 # MHz
memory = 1024 # MB
}
# template {
# destination = "local/buildbot.tac"
# data = <<-EOC
# {{ key "nomad/${var.dc}/buildbot/master/buildbot.tac" }}
# EOC
# change_mode = "restart"
# perms = "644"
# error_on_missing_key = true
# }
# template {
# destination = "local/master.cfg"
# data = <<-EOC
# {{ key "nomad/${var.dc}/buildbot/master/master.cfg" }}
# EOC
# change_mode = "restart"
# perms = "644"
# error_on_missing_key = true
# }
}
task "await-service-mysql" {
driver = "raw_exec"
# user = local.var.exec_user
config {
command = "bash"
args = [ "-c", <<-EOS
echo -n Waiting for $SVC_MYSQL.; \
until \
drill $SVC_MYSQL 2>/dev/null | grep -q 'rcode: NOERROR'; \
do echo -n ' .'; sleep $WAIT_SEC; done; \
echo 'Available.'; \
EOS
]
}
env {
SVC_MYSQL = "mysql.service.${var.dc}.consul"
WAIT_SEC = 10
}
lifecycle {
hook = "prestart"
sidecar = false
}
}
}
}
Reverse Proxy¶
To proxy it through a web server, see below
This snippet can be used to reverse-proxy the service using NGINX.
upstream proxy_buildmaster {
server your.host.local:<buildmaster-port> fail_timeout=5;
}
## the following goes inside a server block
location /buildbot/ {
proxy_pass http://proxy_buildbot/;
}
location /buildbot/sse/ {
# proxy buffering will prevent sse to work
proxy_buffering off;
proxy_pass http://proxy_buildbot/sse/;
}
# required for websocket
location /buildbot/ws {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
# raise the proxy timeout for the websocket
proxy_read_timeout 6000s;
proxy_pass http://proxy_buildbot/ws;
}
Build Your Own¶
Feel free to clone (or fork) the repository and customize it for your own usage, build the image for yourself on your own systems, and optionally, push it to your own public (or private) repository.
Here's how...
Setting up¶
Before we clone the /repository, we must have Git, GNU make, and Docker (optionally, with buildx plugin for multi-platform images) setup on the machine. Also, for multi-platform annotations, we might require enabling experimental features of Docker.
Now, to get the code,
Clone the repository with,
To get a list of all available targets, run
Always Check Before You Make!
Did you know, we could check what any make target is going to execute before we actually run them, with
Build and Test¶
To create the image for your architecture, run the build
and test
target with
For building an image that targets another architecture, it is required to specify the ARCH
parameter when building. e.g.
Build Parameters
All images have a few common build parameters that can be customized at build time, like
ARCH
- The target architecture to build for. Defaults to host architecture, auto-detected at build-time if not specified. Also determines if binfmt support is required before build or run and runs the
regbinfmt
target automatically. Possible values areaarch64
,armhf
,armv7l
, andx86_64
.
BUILDDATE
- The date of the build. Can be used to create separate tags for images. (format:
yyyymmdd
)
DOCKERFILE
- The dockerfile to use for build. Defaults to the file Dockerfile, but if per-arch dockerfiles exist, (e.g. for x86_64 the filename would be
Dockerfile_x86_64
) that is used instead.
TESTCMD
- The command to run for testing the image after build. Runs in a bash shell.
VERSION
- The version of the app/tool, may need to be preset before starting the build (e.g. for binaries from github releases), or extracted from the image after build (e.g. for APK or pip packages).
REGISTRY
- The registry to push to, defaults to the Docker Hub Registry (
docker.io
) or any custom registry that is set via docker configurations. Does not need to be changed for local or test builds, but to override, either pass it by setting an environment variable, or with everymake
command.
ORGNAME
- The organization (or user) name under which the image repositories exist, defaults to
woahbase
. Does not need to be changed for local or test builds, but to override, either pass it by setting an environment variable, or with everymake
command.
The image may also require custom parameters (like binary architecture). Before you build, check the makefile for a complete list of parameters to see what may (or may not) need to be set.
BuildX and Self-signed certificates
If you're using a private registry (a-la docker distribution server) with self-signed certificates, that fail to validate when pulling/pushing images. You will need to configure buildx to allow insecure access to the registry. This is configured via the config.toml
file. A sample is provided in the repository, make sure to replace YOUR.PRIVATE.REGISTRY
with your own (include port if needed).
Make to Run¶
Running the image creates a container and either starts a service (for service images) or provides a shell (can be either a root-shell or usershell) to execute commands in, depending on the image. We can run the image with
But if we just need a root-shell in the container without any fance pre-tasks (e.g. for debug or to test something bespoke), we can run bash
in the container with --entrypoint /bin/bash
. This is wrapped in the makefile as
Nothing vs All vs Run vs Shell
By default, if make
is run without any arguments, it calls the target all
. In our case this is usually mapped to the target run
(which in turn may be mapped to shell
).
There may be more such targets defined as per the usage of the image. Check the makefile for more information.
Push the Image¶
If the build and test steps finish without any error, and we want to use the image on other machines, it is the next step push the image we built to a container image repository (like /hub), for that, run the push
target with
If the built image targets another architecture then it is required to specify the ARCH
parameter when pushing. e.g.
Pushing Multiple Tags
With a single make push
, we are actually pushing 3 tags of the same image, e.g. for x86_64
architecture, they're namely
alpine-buildmaster:x86_64
- The actual image that is built.
alpine-buildmaster:x86_64_(version)
- It is expected that the application is versioned when built or packaged, it can be specified in the tag, this makes pulling an image by tag possible. Usually this is obtained from the parameter
VERSION
, which by default, is set by calling a function to extract the version string from the package installed in the container, or from github releases. Can be skipped with the parameterSKIP_VERSIONTAG
to a non-empty string value like1
.
alpine-buildmaster:x86_64_(version)_(builddate)
- When building multiple versions of the same image (e.g. for providing fixes or revisions), this ensures that a more recent push does not fully replace a previously pushed image. This way, although the architecture and version tags are replaced, it is possible to roll back to the previously built image by build date (format
yyyymmdd
). This value is obtained from theBUILDDATE
parameter, and if not essential, can be skipped by setting the parameterSKIP_BUILDDATETAG
to a non-empty string value like1
.
Pushing To A Private Registry
If you want to push the image to a custom registry that is not pre-configured on your system, you can set the REGISTRY
variable either on the build environment, or as a makefile parameter, and that will be used instead of the default Docker Hub repository. Make sure to have push access set up before you actually push, and include port if needed. E.g.
or
Annotate Manifest(s)¶
For single architecture images, the above should suffice, the built image can be used in the host machine, and on other machines that have the same architecture too, i.e. after a push.
But for use-cases that need to support multiple architectures, there's a couple more things that need to be done. We need to create
(or amend
if already created beforehand) a manifest for the image(s) that we built, then annotate it to map the images to their respective architectures. And for our three tags created above we need to do it thrice.
Did you know?
We can inspect the manifest of any image by running
Tag Latest¶
Assuming we built the images for all supported architectures, to facilitate pulling the correct image for the architecture, we can create/amend the latest
manifest and annotate it to map the tags :aarch64
, :armhf
, :armv7l
, :x86_64
to the tag :latest
by running
How it works
First we create or amend the manifest with the tag latest
Then annotate the image for each architecture in the manifest with
And finally, push it to the repository using
Tag Version¶
Next, to facilitate pulling images by version, we create/amend the image-version manifest and annotate it to map the tags :aarch64_(version)
, :armhf_(version)
, :armv7l_(version)
, :x86_64_(version)
to the tag :(version)
by running
How it works
First we create or amend the manifest with the tag (version)
Then annotate the image for each architecture in the manifest with
And finally, push it to the repository using
Tag Build-Date¶
Then, (optionally) we create/amend the (version)_(builddate)
manifest and annotate it to map the tags :aarch64_(version)_(builddate)
, :armhf_(version)_(builddate)
, :armv7l_(version)_(builddate)
, :x86_64_(version)_(builddate)
to the tag :(version)_(builddate)
by running
How it works
First we create or amend the manifest with the tag (version)_(builddate)
docker manifest create \
woahbase/alpine-buildmaster:(version)_(builddate) \
woahbase/alpine-buildmaster:aarch64_(version)_(builddate) \
woahbase/alpine-buildmaster:armhf_(version)_(builddate) \
woahbase/alpine-buildmaster:armv7l_(version)_(builddate) \
woahbase/alpine-buildmaster:x86_64_(version)_(builddate) \
;
docker manifest create --amend \
woahbase/alpine-buildmaster:(version)_(builddate) \
woahbase/alpine-buildmaster:aarch64_(version)_(builddate) \
woahbase/alpine-buildmaster:armhf_(version)_(builddate) \
woahbase/alpine-buildmaster:armv7l_(version)_(builddate) \
woahbase/alpine-buildmaster:x86_64_(version)_(builddate) \
;
Then annotate the image for each architecture in the manifest with
And finally, push it to the repository using
That's all folks! Happy containerizing!
Maintenance¶
Sources at Github. Built and tested at home using Buildbot. Images at Docker Hub.
Maintained (or sometimes a lack thereof?) by WOAHBase.