NGINX Reverse Proxy LetsEncrypt Auto-Renew

Intro

I finally got round to moving all my web services off a single server and onto a new server using ESXi virtualisation. I got an older HP G7 DL380 with 2x Intel Xeon CPU’s and 64GB of RAM for around £300 off eBay. It does use more power (Averages 150W) however it is well worth it as it provides full RAID redundancy and virtualisation provides easy backup/snapshots before any modifications. I have decided to create a separate VM for each service and then use NGINX as a reverse proxy to handle all the SSL. This greatly reduces management overhead as I have only got to renew the certificates in one place, it also provides speed improvements as well as security.

I was initially put off LetsEncrypt with its short certificate lifetime and the need for automation, especially when I add a large and complex Apache configuration file however I decided to go for it with a brand new VM and I am glad I did; it is brilliant!!

Software Installation

I installed the following tools on my CentOS 7 VM.

Obviously you won’t need open-vm-tools unless it is a VMware VM.

NGINX Configuration

In this tutorial I will configure NGINX and LetsEncrypt so renewing the certificates doesn’t need any downtime however you can configure certbot to use it’s own temporary webserver. This will require NGINX to be shut down as it has to run on the standard web ports.

The first step is to create a default config, NGINX configs are stored within /etc/nginx/conf.d/ on CentOS 7.

By default this directory is blank so lets create the default.conf file.

I have defined the certificate files in the config as I know LetsEncrypt will save them there but you might have to adjust after you get the certificates. “default_server” means that server block is the default when no others match, we can use this to redirect unknown requests to a desired page such as a blog.

Next, create the config file which will proxy the requests to each internal VM using HTTP (Can use HTTPS).

I have listed it a server block at a time. I will display some examples but you can add as many or few as needed.

Note:

  1. You must add the location block for /.well-known as this allows LetsEncrypt to verify the domain, this must be added to each subdomain server block that you want the certificate valid for. Otherwise it will not verify and you will need to run certbot with its own temporary web server.
  2. GZIP has been enabled to compress the requests to wordpress as this reduces the page loading times, I did this on the proxy as it performed a lot better than on the wordpress VM with apache.
  3. The proxy set headers have been enabled in the second location block to send the real client IP to apache and wordpress. If you don’t do this then access logs are useless as it will contain the IP of the proxy only and for systems such as Nextcloud can cause the bruteforce protection to block the reverse proxy instead of a real client IP.
  4. Proxy_pass is the internal VM’s address or localhost:port if running on the same server.
  5. Server_name is the subdomain

Here are a few more examples:

Subsonic

Nextcloud

SELinux

If your running CentOS and SELinux is enabled then it can cause a few problems for NGINX if proxying to non-standard ports such as Subsonic’s. Instead of just turning it off we will add a policy to allow NGINX to connect as this is an internet facing server so security is important.

You will likely see a 502 gateway error when trying to browse to subsonic.example.org if using the above example. Firstly check for any deny statements in the audit log:

As you can see NGINX is indeed blocked. Next create a policy from the deny log, you might want to filter down to “port” so other ports are not allowed if they have been blocked.

Then apply the policy after it has created it.

LetsEncrypt

Next we need to get our certificates using a tool called certbot. This makes it very easy to grab our certificate, modify it or renew it.

Create the .well-known directory in the root nginx directory.

Note: NGINX might not start with the above SSL configuration as NGINX won’t find any SSL certificates unless like me, you already had certificates from another provider in the SSL directory. There would be two solutions to this. Run the certbot tool below with the temporary webserver on the first run just to get the certificates so NGINX will start or change the listen “443’s to 80” and remove the SSL on config line in each server block so it is only running in HTTP mode.

Run the certbot tool, note for testing it is probably best to use the “–staging” flag to prevent hitting LetsEncrypts rate limits. You can either execute the command with all the parameters like below or run “certbot certonly” to use the tool interactively.

Next enter your email address and if successfully you should see something like below.

If you receive any errors relating to unable to verify subdomain “X” then double check the location block for the .well-known directory is okay and all permissions are correct. If everything was okay then you can re-run the command without the “–staging” flag to get the proper certificates.

You can run the command again with extra sub-domains listed, press “e” to extend the certificate with the extra sub-domains.

Now we should create a stronger Diffie-Hellman than used by default.

Within the NGINX default configuration file above I have included dhparam.pem.

Renewing Certificates Automatically

Certbot can renew all your certificates by simply executing “certbot renew”, you could do this simple command every 60 days when you get an email but why not automate it instead. I will automate it using systemd instead of a cron job.

Firstly create a systemd service file for letsencrypt.

The above can be modified to stop NGINX first in case your using certbot with a temporary web server. Change the “ExecStart” line to:

Now create a systemd timer, it is recommended to run at least once a day in case of revocations.

Now enable the new systemd timer.

Check the timers status using:

References:

MyShell

Certbot User Guide

LetsEncrypt Documentation

Please share 🙂