Create a static site hosting platform

There are a lot of static site generators out there and users have a lot of possibilities to automate and continuously deploy static sites these days. Solutions like GitHub pages or Netlify are free to use and easy to set up, even a cheap webspace could work. If one of these services is sufficient for your use case you could stop reading at this point.

As I wanted to have more control over such a setup and because it might be fun I decided to create my own service. Before looking into the setup details, lets talk about some requirements:

• deploy multiple project documentation
• use git repository name as subdomain
• easy CI workflow

The required software stack is quite simple:

• Nginx as web server to deliver static files
• Minio S3 as files backend

Of course, Minio could be removed from the stack but after a few tests, my personal impression was that Minio is a way better to handle file sync and uploads instead over SSH/SCP, at least in a CI driven workflow. The whole workflow is nearly the same as for GitHub pages. Right beside your projects source code in the Git repository you will have a docs folder which contains the required files to build the documentation site. It’s up to you what static site generator to use. Instead of using e.g. the gh-pages branch to publish the site, a CI system will build the documentation form the docs folder and publish it to a prepared Minio S3 bucket. The Nginx server will basically use the defined bucket as source directory and convert every sub-directory into something like https://<reponame>.mydocs.com.

As a first step, install Minio and Nginx on a server, I will not cover the basic setup in this guide. To simplify the setup I will use a single server for Minio and Nginx but it’s also possible to split this into a Two-Tier architecture.

After the basic setup it’s required to create a Minio bucket e.g. mydocs using the Minio client command mc mb local/mydocs. To allow Nginx to access these bucket to deliver the pages without authentication a bucket policy mc policy set download local/mydocs need to be set. This policy will allow public read access. In theory, it should also be possible to add authentication headers to Nginx to server sites from private buckets but I have not tried that on my own.

Preparing the Minio bucket was the easy part, now Nginx need to know how to rewrite the subdomains to sub-directories and properly deliver the sites. Let’s assume mydocs is still used as the base Minio bucket and mydocs.com as root domain. Here is how my current vHost configuration looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

upstream backend_mydocs {
    server localhost:61000;
}

server {
    listen       80;
    server_name  ~^(www\.)?(?<name>(.+\.)?mydocs\.com)$;

    return 301 https://$name$request_uri;
}

server {
    listen       443 ssl;
    server_name  ~^((?<repo>.*)\.)mydocs\.com$;

    ssl_certificate [..];
    ssl_certificate_key [..];
    client_max_body_size 100M;

    recursive_error_pages on;

    location / {
        proxy_pass http://backend_mydocs/mydocs/${repo}${request_path};
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_connect_timeout 300;
        proxy_intercept_errors on;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        rewrite ^([^.]*[^/])$ $1/ permanent;
        error_page 404 = /404_backend.html;
    }

    location /404_backend.html {
        proxy_pass http://backend_mydocs/mydocs/${repo}/404.html;
        proxy_intercept_errors on;
    }
}

We will go through this configuration to understand how it works.

Lines 1-3 defines a backend, in this case it’s the Minio server running on localhost:61000.

Lines 5-10 should also be straight forward, this block will redirect HTTP to HTTPS.

Line 14 is where the magic starts. A named regular expression is used to capture the first part of the subdomain and translate it into the bucket sub-directory. For a given URL like demoproject.mydocs.com Nginx will try to serve mydocs/demoproject from the Minio server. That’s what Line 23 does. Some of you may notice that the used variable ${request_path} is not defined in the vHost configuration.

Right, another configuration snippet needs to be added to the nginx.conf. But why is this variable required at all? For me, that was the hardest part to solve. As the setup is using proxy_pass Nginx will not try to lookup index.html automatically. That’s a problem because every folder will at least contain an index.html. In general, it’s required to tell Nginx to rewrite the request URI to /index.html if the origin is a folder and ends with /. One way would be an if condition in the vHost configuration but such conditions are evil¹ in most cases and should be avoided if possible. Luckily there is a better option:

1
2
3
4

map $request_uri $request_path {
    default $request_uri;
    ~/$ ${request_uri}index.html;
}

Nginx maps are a solid way to create conditionals. In this example set $request_uri as input and $request_path as output. Each line between the braces is a condition. The first line will simply apply $request_uri to the output variable if no other condition match. The second condition applies ${request_uri}index.html to the output variable if the input variable ends with a slash (and therefor is a directory).

Line 38-41 of the vHost configuration tries to deliver the custom error page of your site and will fallback to the default Nginx error page.

We are done! Nginx should now be able to server your static sites from a sub-directory of the Minio source bucket. I’m using it since a few weeks and I’m really happy with the current setup.