With many services running in their own container or with their own daemon, the use of reverse proxies is getting more common, and at the same time, the use of SSL/TLS has become a standard. Let’s Encrypt is a provider of TLS certificates allowing free and fully automated certificate renewal.
The fully automated renewal process is necessary due to Let’s Encrypt’s short, 90 day certificate lifetime. To perform the (also automatic) domain validation, Let’s Encrypt provides two validation methods. The not so common DNS-challenge uses DNS entries to verify the requester of the certificate is in control of the domain. More commonly used is the HTTP-challenge where the requester provides proof of being in control by providing web-content under the requested domain name.
HTTP-challenge with container issue
A very common setup is to terminate the SSL/TLS incoming connection at the reverse proxy. As a result the TLS connection will be established between the client and the reverse proxy. The reverse proxy on the other hand connects for example to the container, depending on the setup, via HTTP or a separate HTTPS connection. With the traffic forwarded like this, the HTTP-challenge requests would be forwarded as well.
The Let’s Encrypt client expects the HTTP-challenge file to be written into the local web-server directory for verification. When the reverse proxy forwards those requests to the application in the container, the HTTP-challenge will fail.
Exclude the HTTP-challenge from reverse proxy
Let’s Encrypt HTTP-challenge requests point to a known path. The path is “/.well-known/acme-challenge/” followed by a token name. This is a fixed path which can be configured in the reverse proxy (Apache is used in this example) to be served locally instead of forwarded to the container.
To show an example, the following shows the usual reverse proxy configuration of an Apache webserver. All requests (represented by the “/”) will be sent to “http://destination.host:8001”. When Let’s Encrypt is used, requests for “/.well-known/acme-challenge/” would be forwarded too.
ProxyPass / http://destination.host:8001/ ProxyPassReverse / http://destination.host:8001/ <Proxy *> allow from all </Proxy>
To allow Let’s Encrypt requests to be answered by Apache locally, the path must be excluded from the reverse proxy. In the example below the entire “/.well-known” is excluded. It would also be possible to only exclude “/.well-known/acme-challenge”, if necessary.
ProxyPass /.well-known ! ProxyPass / http://destination.host:8001/ ProxyPassReverse / http://destination.host:8001/ <Proxy *> allow from all </Proxy>
Verify the configuration by placing a file in the DocumentRoot (RHEL or CentOS default would be /var/ww/html/) of this host (or VirtualHost) into the subdirectory “/.well-known/acme-challenge”.
$ echo "testing" >.well-known/acme-challenge/TEST
To verify the file is reachable use curl to retrieve the test file.
$ curl http://host.domain.tld/.well-known/acme-challenge/TEST testing
Read more of my posts on my blog at https://blog.tinned-software.net/.
