NGINX Reverse Proxy LetsEncrypt Auto-Renew

Intro

I finally got round to moving all my web services off a single server and onto a new server using ESXi virtualisation. I got an older HP G7 DL380 with 2x Intel Xeon CPU's and 64GB of RAM for around £300 off eBay. It does use more power (Averages 150W) however it is well worth it as it provides full RAID redundancy and virtualisation provides easy backup/snapshots before any modifications. I have decided to create a separate VM for each service and then use NGINX as a reverse proxy to handle all the SSL. This greatly reduces management overhead as I have only got to renew the certificates in one place, it also provides speed improvements as well as security.

I was initially put off LetsEncrypt with its short certificate lifetime and the need for automation, especially when I add a large and complex Apache configuration file however I decided to go for it with a brand new VM and I am glad I did; it is brilliant!!

Software Installation

I installed the following tools on my CentOS 7 VM.

yum install open-vm-tools nano htop epel-release nginx certbot

Obviously you won't need open-vm-tools unless it is a VMware VM.

NGINX Configuration

In this tutorial I will configure NGINX and LetsEncrypt so renewing the certificates doesn't need any downtime however you can configure certbot to use it's own temporary webserver. This will require NGINX to be shut down as it has to run on the standard web ports.

The first step is to create a default config, NGINX configs are stored within /etc/nginx/conf.d/ on CentOS 7.

[root@reverseproxy ~]# ls /etc/nginx/conf.d/
default.conf  reverseproxy.conf

By default this directory is blank so lets create the default.conf file.

nano /etc/nginx/conf.d/default.conf

ssl_certificate  /etc/letsencrypt/live/example.org/fullchain.pem;
ssl_certificate_key  /etc/letsencrypt/live/example.org/privkey.pem;
ssl_session_timeout  5m;
ssl_prefer_server_ciphers  on;
ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES1$
ssl_dhparam ssl/dhparam.pem;

server  {

  listen  80;   # Redirect any port http/80 requests, to https/443
  server_name  *.example.org;
  return 301 https://$host$request_uri;
}

server {
        listen       443 ssl http2 default_server;
        listen       [::]:443 ssl http2 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

#       Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        error_page 404 /404.html;

        error_page 500 502 503 504 /50x.html;

        #The below will redirect any subdomain not configured in the other config file to a default page. Here we set it to say a blog.
        return 301 https://blog.example.org$request_uri;
}

I have defined the certificate files in the config as I know LetsEncrypt will save them there but you might have to adjust after you get the certificates. "default_server" means that server block is the default when no others match, we can use this to redirect unknown requests to a desired page such as a blog.

Next, create the config file which will proxy the requests to each internal VM using HTTP (Can use HTTPS).

nano /etc/nginx/conf.d/reverseproxy.conf

I have listed it a server block at a time. I will display some examples but you can add as many or few as needed.

server  {
  listen  443 ssl;   # Browseable at https://blog.example.org
  server_name blog.example.org;
  add_header Strict-Transport-Security "max-age=31536000" always;
  ssl  on;
  gzip on;
  gzip_types text/plain text/html text/xml text/css application/xml application/xhtml+xml application/rss+xml application/javascript applicat$
  gzip_proxied any;
  location /.well-known {
    root /usr/share/nginx/html/;
  }
  location  / {
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $host;
    proxy_pass  http://192.168.200.12/;
  }
}

Note:

1. You must add the location block for /.well-known as this allows LetsEncrypt to verify the domain, this must be added to each subdomain server block that you want the certificate valid for. Otherwise it will not verify and you will need to run certbot with its own temporary web server.
2. GZIP has been enabled to compress the requests to wordpress as this reduces the page loading times, I did this on the proxy as it performed a lot better than on the wordpress VM with apache.
3. The proxy set headers have been enabled in the second location block to send the real client IP to apache and wordpress. If you don't do this then access logs are useless as it will contain the IP of the proxy only and for systems such as Nextcloud can cause the bruteforce protection to block the reverse proxy instead of a real client IP.
4. Proxy_pass is the internal VM's address or localhost:port if running on the same server.
5. Server_name is the subdomain

Here are a few more examples:

Subsonic

server  {
  listen  443 ssl;   # Example config for SubSonic, browsable at https://subsonic.example.org
  server_name  subsonic.example.org;
  add_header Strict-Transport-Security "max-age=31536000" always;
  ssl  on;
  location /.well-known {
    root /usr/share/nginx/html/;
  }
  location  / {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_pass  http://192.168.200.6:4040/;
  }
}

Nextcloud

server  {
  listen  443 ssl;   # Example config for OwnCloud, browsable at https://nextcloud.example.org
  server_name  nextcloud.example.org;
  client_max_body_size  0;
  add_header Strict-Transport-Security "max-age=31536000" always;
  ssl  on;
  location /.well-known {
    root /usr/share/nginx/html/;
  }
  location  / {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $remote_addr;
    proxy_set_header X-Forwarded-Protocol $scheme;
    proxy_pass  http://192.168.200.52/;
  }
}

SELinux

If your running CentOS and SELinux is enabled then it can cause a few problems for NGINX if proxying to non-standard ports such as Subsonic's. Instead of just turning it off we will add a policy to allow NGINX to connect as this is an internet facing server so security is important.

You will likely see a 502 gateway error when trying to browse to subsonic.example.org if using the above example. Firstly check for any deny statements in the audit log:

cat /var/log/audit/audit.log | grep nginx | grep denied

t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1493307449.182:624): avc:  denied  { name_connect } for  pid=15282 comm="nginx" dest=4040 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1493307450.338:625): avc:  denied  { name_connect } for  pid=15282 comm="nginx" dest=4040 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket

As you can see NGINX is indeed blocked. Next create a policy from the deny log, you might want to filter down to "port" so other ports are not allowed if they have been blocked.

cat /var/log/audit/audit.log | grep nginx | grep denied | audit2allow -M mynginx

Then apply the policy after it has created it.

semodule -i mynginx.pp

LetsEncrypt

Next we need to get our certificates using a tool called certbot. This makes it very easy to grab our certificate, modify it or renew it.

Create the .well-known directory in the root nginx directory.

mkdir /usr/share/nginx/html/.well-known/

echo "Test NGINX" > /usr/share/nginx/html/.well-known/test.html

systemctl restart nginx
systemctl enable nginx

Note: NGINX might not start with the above SSL configuration as NGINX won't find any SSL certificates unless like me, you already had certificates from another provider in the SSL directory. There would be two solutions to this. Run the certbot tool below with the temporary webserver on the first run just to get the certificates so NGINX will start or change the listen "443's to 80" and remove the SSL on config line in each server block so it is only running in HTTP mode.

Run the certbot tool, note for testing it is probably best to use the "--staging" flag to prevent hitting LetsEncrypts rate limits. You can either execute the command with all the parameters like below or run "certbot certonly" to use the tool interactively.

certbot certonly --webroot -w /usr/share/nginx/html/ -d example.org -d www.example.org -d blog.example.org -d zabbix.example.org -d nextcloud.example.org -d subsonic.example.org -d owncloud.example.org -d tv.example.org --staging

Next enter your email address and if successfully you should see something like below.

Generating key (2048 bits): /etc/letsencrypt/keys/0002_key-certbot.pem

Creating CSR: /etc/letsencrypt/csr/0002_csr-certbot.pem

IMPORTANT NOTES:

- Congratulations! Your certificate and chain have been saved at

/etc/letsencrypt/live/example.org/fullchain.pem. Your cert will

expire on 2017-07-25. To obtain a new or tweaked version of this

certificate in the future, simply run certbot again. To

non-interactively renew *all* of your certificates, run "certbot

renew"

- Your account credentials have been saved in your Certbot

configuration directory at /etc/letsencrypt. You should make a

secure backup of this folder now. This configuration directory will

also contain certificates and private keys obtained by Certbot so

making regular backups of this folder is ideal.

- If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate

Donating to EFF: https://eff.org/donate-le

If you receive any errors relating to unable to verify subdomain "X" then double check the location block for the .well-known directory is okay and all permissions are correct. If everything was okay then you can re-run the command without the "--staging" flag to get the proper certificates.

You can run the command again with extra sub-domains listed, press "e" to extend the certificate with the extra sub-domains.

Now we should create a stronger Diffie-Hellman than used by default.

openssl dhparam -out /etc/nginx/ssl/dhparam.pem 2048

Within the NGINX default configuration file above I have included dhparam.pem.

[the_ad id="630"]

Renewing Certificates Automatically

Certbot can renew all your certificates by simply executing "certbot renew", you could do this simple command every 60 days when you get an email but why not automate it instead. I will automate it using systemd instead of a cron job.

Firstly create a systemd service file for letsencrypt.

nano /etc/systemd/system/renew-letsencrypt.service

[Unit]
Description=Renew Let's Encrypt certificates
After=network-online.target

[Service]
Type=oneshot
# check for renewal
ExecStart=/usr/bin/certbot renew --quiet --agree-tos

The above can be modified to stop NGINX first in case your using certbot with a temporary web server. Change the "ExecStart" line to:

ExecStart=/usr/bin/certbot renew --renew-hook "/bin/systemctl --no-block reload nginx" --quiet --agree-tos

Now create a systemd timer, it is recommended to run at least once a day in case of revocations.

nano /etc/systemd/system/renew-letsencrypt.timer

[Unit]
Description=Daily renewal of Let's Encrypt's certificates

[Timer]
# once a day, at 2AM
OnCalendar=*-*-* 02:00:00
# Be kind to the Let's Encrypt servers: add a random delay of 0–3600 seconds
RandomizedDelaySec=3600
Persistent=true

[Install]
WantedBy=timers.target

Now enable the new systemd timer.

systemctl daemon-reload

systemctl start renew-letsencrypt.timer

systemctl enable renew-letsencrypt.timer

Check the timers status using:

systemctl list-timers renew-letsencrypt.timer

NEXT                         LEFT          LAST                         PASSED  UNIT                    ACTIVATES
Thu 2017-05-04 02:11:00 BST  3h 20min left Wed 2017-05-03 02:14:36 BST  20h ago renew-letsencrypt.timer renew-letsencrypt.service

1 timers listed.
Pass --all to see loaded but inactive timers, too.

References:

MyShell

Certbot User Guide

LetsEncrypt Documentation