Lately, I started a project for creating S3 storage from my old hard-disk. The goal is to backup footage from my cheap chinese cctv camera. When it comes to choose the s3 app solution, there are several option:
- minio: Dead, no further development, the project archived. see here
- seaweedfs: support various protocol other than s3 (e.g hdfs), handle billions of files with O(1), effortless horizontal scaling, enterprise feature available
- garage: geo distributed data storage
- rustfs: minio like architecture but written in rust, under heavy development, not yet stable
For simplicity I choose rustfs, simply because i just need the s3 api, it’s open source and with minio-like architecture it’ll be easy to find discussion when encountering a problem.
Opening to public, when the problem begin
I’m installing the rustfs in sbc with an external hdd. Then i setup a script to push cctv footage into s3. Everything goes smoothly until I need to open the s3 into the internet using cloudflare tunnel. This decision because i want to monitor the latest cctv footage from a web browser. I have plot the architecutre below:
Using presigned url user can access the resource between short period of time. with this approach, I can separate the workload for serving video directly to s3 server, instead of downloading and serving again with my backend service.
Strangely, everytime the backend checks the existence of the file using Head API, it always returns “403 Forbidden” on the first invocation and returns the answer on the second invocation.
The return code “403 Forbidden” means something is wrong with the AWS S3 SigV4 signature. AWS s3 authentication done using “signature”, it is created from a key and secret saved in the server.
Client will calculate the signature based on object_path,request method, timestamp, target host, etc. Then the result will be sent through the HTTP request header. When the server receives the request, it’ll calculate server signature based on the received request. If signature doesn’t match, 403 will be returned. In order words, there’s something between client and server that changes the parameters!
Tcpdump to the rescue
I read several articles about this strange behavior in the minio backed setup with cloudflare tunnel. Unfortunately, I tried all the community solutions and still got the same result. Then, I remembered, I can capture the request in the linux using tcpdump. It’l more easier since my rustfs setup didn’t use tls/ssl so decrypting is not needed
NOTE: this is why you MUST NOT EVER SEND SENSITIVE DATA OVER PLAIN HTTP PROTOCOL. Someone in the middle of your connection can intercept and read all of your payload requests (including your ISP). As you can see my ip is in plain sight.
Gotcha! I notice the problem. Cloudflare tunnel changed the first HEAD method into GET! It all makes sense now. The signature will always not match in the first hit because the client calculates the signature with the HEAD method and the server calculates signature with GET method. There’s a community post explaining this behaviour and the solution is to disable cache from the URL. Here’s the screenshot after implementing the solution:
Conclusion
It’s valid behaviour from cloudflare to change HEAD into GET if the file didn’t exist in the cf cache. The only downside if this solution approach is the request will not be cached and run directly to the server. It’s not a problem in my setup because i just serve a small load, will be a problem if your service serve millions of request