UPDATE: Better solution: Run Traefik 3.0.0-rc1 (at the time of this writing) and use the IPAllowList
middleware coupled with the rejectStatusCode
option. It appears the IPWhiteList
middleware is being deprecated in favor of IPAllowList
where the return code is (or will be..) user-configurable. I do not see a callout of this in the IPAllowList doc linked above yet, but you can see the commit here.
This is great in the sense that you can manipulate the response on a per-service basis rather than hacking the code and returning a 404 for all services. Nice!
Example labels for the service side look like:
-l traefik.http.middlewares.vault-ipallowlist.ipallowlist.sourcerange='192.168.2.0/24'
-l traefik.http.middlewares.vault-ipallowlist.ipallowlist.rejectStatusCode='404'
Having run Vaultwarden for a couple years now, it’s safe to say that I am confident in the tool to meet my password management needs and rely on it daily. And given that it houses the keys to the kingdom, I’ve hosted it in various ways over that time based on what I felt was the most secure yet still usable. I will skip articulating the various hosting permutations I’ve gone through and instead focus on providing the basic requirements for the ideal state I had in mind:
- Run via Podman ✔
- Run behind Traefik ✔
- Automated SSL termination via Let’s Encrypt ✔
- Allow access from defined networks ❌
Everything was pretty simple until that fourth requirement. Traefik supports defining networks for service access via the IPWhiteList middleware, but here’s the kicker: The Bitwarden clients do not play nice with the 403 (Forbidden) error code returned by the middleware when access is attempted from a non-allowed network. The client will log out of the session and until access to Vaultwarden is restored (to login again), passwords will be out of reach…which was too much risk for me.
Note: There is a feature request to change this behavior on the Bitwarden client side here, but it should be noted that this could also be approached from the Traefik side by something like allowing the IPWhiteList middleware response code to be configurable (although 403 is accurate for what the middleware is doing).
So What’s the Hack?
SEE UPDATE AT THE TOP BEFORE WASTING YOUR TIME CONTINUING.
Simple: Change the error code returned by the IPWhiteList middleware from 403 to 404! While I am not exactly proud of this solution, my initial testing proves it works. This consists of a one-line change in Traefik’s ip_whitelist.go
middleware source.
Proof-of-Concept Runthrough
Clone the Traefik github repo:
git clone https://github.com/traefik/traefik
Modify the following line in ./traefik/pkg/middlewares/ipwhitelist/ip_whitelist.go
From:
statusCode := http.StatusForbidden
To:
statusCode := http.StatusNotfound
Note: Per Go HTTP methods
Build
Use this documentation to build Traefik. Note: You will need golang 1.22.0+ to build successfully at the time of this writing.
Upon a successful build, you should have your new traefik
binary in the ./dist
directory.
Test Functionality
For a quick test, I mounted the new traefik
binary directly to /usr/local/bin/traefik
in my Traefik container via: -v /containers/traefik/traefik:/usr/local/bin/traefik:z
. For example, here’s an example Traefik start script:
podman run --name=traefik -d \
--security-opt label=type:container_runtime_t \
-v /run/podman/podman.sock:/var/run/docker.sock:z \
-v /containers/traefik/acme.json:/acme.json:z \
-v /containers/traefik/traefik:/usr/local/bin/traefik:z \
--net web \
-p 80:80 \
-p 443:443 \
docker.io/library/traefik:latest \
--entrypoints.web.address=":80" \
--entrypoints.web.http.redirections.entryPoint.to=websecure \
--entrypoints.web.http.redirections.entryPoint.scheme=https \
--entrypoints.websecure.address=":443" \
--providers.docker=true \
--certificatesresolvers.le.acme.email=admin@example.com \
--certificatesresolvers.le.acme.storage=/acme.json \
--certificatesresolvers.le.acme.tlschallenge=true \
--api.dashboard=true
Test from a Vaultwarden (or any) container using the IPWhiteList middleware:
Note: Here’s an article on using the IPWhiteList middleware if needed.
$ curl https://vault.example.com
Not Found
Success! Not Found
instead of Forbidden
and Vaultwarden clients no longer log me out! Moving forward, I will likely roll my own container and automate this based on Traefik releases. If you come across this post and know of a simple built-in way to accomplish the same result, please share!
Cheers!