Intro
I finally got round to moving all my web services off a single server and onto a new server using ESXi virtualisation. I got an older HP G7 DL380 with 2x Intel Xeon CPU's and 64GB of RAM for around £300 off eBay. It does use more power (Averages 150W) however it is well worth it as it provides full RAID redundancy and virtualisation provides easy backup/snapshots before any modifications. I have decided to create a separate VM for each service and then use NGINX as a reverse proxy to handle all the SSL. This greatly reduces management overhead as I have only got to renew the certificates in one place, it also provides speed improvements as well as security.
I was initially put off LetsEncrypt with its short certificate lifetime and the need for automation, especially when I add a large and complex Apache configuration file however I decided to go for it with a brand new VM and I am glad I did; it is brilliant!!
Software Installation
I installed the following tools on my CentOS 7 VM.
yum install open-vm-tools nano htop epel-release nginx certbot
Obviously you won't need open-vm-tools unless it is a VMware VM.
NGINX Configuration
In this tutorial I will configure NGINX and LetsEncrypt so renewing the certificates doesn't need any downtime however you can configure certbot to use it's own temporary webserver. This will require NGINX to be shut down as it has to run on the standard web ports.
The first step is to create a default config, NGINX configs are stored within /etc/nginx/conf.d/ on CentOS 7.
[root@reverseproxy ~]# ls /etc/nginx/conf.d/ default.conf reverseproxy.conf
By default this directory is blank so lets create the default.conf file.
nano /etc/nginx/conf.d/default.conf
ssl_certificate /etc/letsencrypt/live/example.org/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/example.org/privkey.pem; ssl_session_timeout 5m; ssl_prefer_server_ciphers on; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES1$ ssl_dhparam ssl/dhparam.pem; server { listen 80; # Redirect any port http/80 requests, to https/443 server_name *.example.org; return 301 https://$host$request_uri; } server { listen 443 ssl http2 default_server; listen [::]:443 ssl http2 default_server; server_name _; root /usr/share/nginx/html; # Load configuration files for the default server block. include /etc/nginx/default.d/*.conf; error_page 404 /404.html; error_page 500 502 503 504 /50x.html; #The below will redirect any subdomain not configured in the other config file to a default page. Here we set it to say a blog. return 301 https://blog.example.org$request_uri; }
I have defined the certificate files in the config as I know LetsEncrypt will save them there but you might have to adjust after you get the certificates. "default_server" means that server block is the default when no others match, we can use this to redirect unknown requests to a desired page such as a blog.
Next, create the config file which will proxy the requests to each internal VM using HTTP (Can use HTTPS).
nano /etc/nginx/conf.d/reverseproxy.conf
I have listed it a server block at a time. I will display some examples but you can add as many or few as needed.
server { listen 443 ssl; # Browseable at https://blog.example.org server_name blog.example.org; add_header Strict-Transport-Security "max-age=31536000" always; ssl on; gzip on; gzip_types text/plain text/html text/xml text/css application/xml application/xhtml+xml application/rss+xml application/javascript applicat$ gzip_proxied any; location /.well-known { root /usr/share/nginx/html/; } location / { proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $host; proxy_pass http://192.168.200.12/; } }
Note:
- You must add the location block for /.well-known as this allows LetsEncrypt to verify the domain, this must be added to each subdomain server block that you want the certificate valid for. Otherwise it will not verify and you will need to run certbot with its own temporary web server.
- GZIP has been enabled to compress the requests to wordpress as this reduces the page loading times, I did this on the proxy as it performed a lot better than on the wordpress VM with apache.
- The proxy set headers have been enabled in the second location block to send the real client IP to apache and wordpress. If you don't do this then access logs are useless as it will contain the IP of the proxy only and for systems such as Nextcloud can cause the bruteforce protection to block the reverse proxy instead of a real client IP.
- Proxy_pass is the internal VM's address or localhost:port if running on the same server.
- Server_name is the subdomain
Here are a few more examples:
Subsonic
server { listen 443 ssl; # Example config for SubSonic, browsable at https://subsonic.example.org server_name subsonic.example.org; add_header Strict-Transport-Security "max-age=31536000" always; ssl on; location /.well-known { root /usr/share/nginx/html/; } location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://192.168.200.6:4040/; } }
Nextcloud
server { listen 443 ssl; # Example config for OwnCloud, browsable at https://nextcloud.example.org server_name nextcloud.example.org; client_max_body_size 0; add_header Strict-Transport-Security "max-age=31536000" always; ssl on; location /.well-known { root /usr/share/nginx/html/; } location / { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for; proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto $remote_addr; proxy_set_header X-Forwarded-Protocol $scheme; proxy_pass http://192.168.200.52/; } }
SELinux
If your running CentOS and SELinux is enabled then it can cause a few problems for NGINX if proxying to non-standard ports such as Subsonic's. Instead of just turning it off we will add a policy to allow NGINX to connect as this is an internet facing server so security is important.
You will likely see a 502 gateway error when trying to browse to subsonic.example.org if using the above example. Firstly check for any deny statements in the audit log:
cat /var/log/audit/audit.log | grep nginx | grep denied
t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket type=AVC msg=audit(1493307449.182:624): avc: denied { name_connect } for pid=15282 comm="nginx" dest=4040 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket type=AVC msg=audit(1493307450.338:625): avc: denied { name_connect } for pid=15282 comm="nginx" dest=4040 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
As you can see NGINX is indeed blocked. Next create a policy from the deny log, you might want to filter down to "port" so other ports are not allowed if they have been blocked.
cat /var/log/audit/audit.log | grep nginx | grep denied | audit2allow -M mynginx
Then apply the policy after it has created it.
semodule -i mynginx.pp
LetsEncrypt
Next we need to get our certificates using a tool called certbot. This makes it very easy to grab our certificate, modify it or renew it.
Create the .well-known directory in the root nginx directory.
mkdir /usr/share/nginx/html/.well-known/ echo "Test NGINX" > /usr/share/nginx/html/.well-known/test.html systemctl restart nginx systemctl enable nginx
Note: NGINX might not start with the above SSL configuration as NGINX won't find any SSL certificates unless like me, you already had certificates from another provider in the SSL directory. There would be two solutions to this. Run the certbot tool below with the temporary webserver on the first run just to get the certificates so NGINX will start or change the listen "443's to 80" and remove the SSL on config line in each server block so it is only running in HTTP mode.
Run the certbot tool, note for testing it is probably best to use the "--staging" flag to prevent hitting LetsEncrypts rate limits. You can either execute the command with all the parameters like below or run "certbot certonly" to use the tool interactively.
certbot certonly --webroot -w /usr/share/nginx/html/ -d example.org -d www.example.org -d blog.example.org -d zabbix.example.org -d nextcloud.example.org -d subsonic.example.org -d owncloud.example.org -d tv.example.org --staging
Next enter your email address and if successfully you should see something like below.
Generating key (2048 bits): /etc/letsencrypt/keys/0002_key-certbot.pem Creating CSR: /etc/letsencrypt/csr/0002_csr-certbot.pem IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at /etc/letsencrypt/live/example.org/fullchain.pem. Your cert will expire on 2017-07-25. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal. - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le
If you receive any errors relating to unable to verify subdomain "X" then double check the location block for the .well-known directory is okay and all permissions are correct. If everything was okay then you can re-run the command without the "--staging" flag to get the proper certificates.
You can run the command again with extra sub-domains listed, press "e" to extend the certificate with the extra sub-domains.
Now we should create a stronger Diffie-Hellman than used by default.
openssl dhparam -out /etc/nginx/ssl/dhparam.pem 2048
Within the NGINX default configuration file above I have included dhparam.pem.
[the_ad id="630"]
Renewing Certificates Automatically
Certbot can renew all your certificates by simply executing "certbot renew", you could do this simple command every 60 days when you get an email but why not automate it instead. I will automate it using systemd instead of a cron job.
Firstly create a systemd service file for letsencrypt.
nano /etc/systemd/system/renew-letsencrypt.service
[Unit] Description=Renew Let's Encrypt certificates After=network-online.target [Service] Type=oneshot # check for renewal ExecStart=/usr/bin/certbot renew --quiet --agree-tos
The above can be modified to stop NGINX first in case your using certbot with a temporary web server. Change the "ExecStart" line to:
ExecStart=/usr/bin/certbot renew --renew-hook "/bin/systemctl --no-block reload nginx" --quiet --agree-tos
Now create a systemd timer, it is recommended to run at least once a day in case of revocations.
nano /etc/systemd/system/renew-letsencrypt.timer
[Unit] Description=Daily renewal of Let's Encrypt's certificates [Timer] # once a day, at 2AM OnCalendar=*-*-* 02:00:00 # Be kind to the Let's Encrypt servers: add a random delay of 0–3600 seconds RandomizedDelaySec=3600 Persistent=true [Install] WantedBy=timers.target
Now enable the new systemd timer.
systemctl daemon-reload systemctl start renew-letsencrypt.timer systemctl enable renew-letsencrypt.timer
Check the timers status using:
systemctl list-timers renew-letsencrypt.timer
NEXT LEFT LAST PASSED UNIT ACTIVATES Thu 2017-05-04 02:11:00 BST 3h 20min left Wed 2017-05-03 02:14:36 BST 20h ago renew-letsencrypt.timer renew-letsencrypt.service 1 timers listed. Pass --all to see loaded but inactive timers, too.
Comments powered by Disqus.