Nginx reverse proxy multiple locations from a single domain - nginx

Question

I'm trying to figure out how to set up Nginx as a reverse proxy from a single domain to multiple backend sites based on the location.
Nginx Config:
server {
listen 80;
underscores_in_headers on;
server_name test.example.com;
gzip on;
gzip_min_length 1100;
gzip_buffers 4 32k;
gzip_types text/plain application/x-javascript text/xml text/css application/javascript;
gzip_vary on;
gzip_proxied any;
proxy_http_version 1.1;
location /page1/ {
proxy_pass http://www.siteone.com/pageone;
proxy_set_header Host www.siteone.com;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location /page2/ {
proxy_pass http://www.sitetwo.com/pagetwo;
proxy_set_header Host www.sitetwo.com;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
The problem is that static files (images, css, etc.) are all broken. The initial request returns fine, but subsequent GET requests all go to the proxy subdomain (ex: test.example.com/css/style.css), and return 404 or 500 errors.
I tried to work around this with a static files location, or a catch all (e.g., "location /" or "location ~* ^(.*).(css|js|etc..)"), but I can't do that for both proxied sites. As a workaround I also tried catching the referer URL and setting the catch-all's proxy_pass based on that, but it didn't work for everything and seemed kind of prone to failure.
I know this isn't a common setup, but unfortunately for our use case we can't use the more common method of a subdomain & server block for each proxied request. Our requirement is for a single subdomain proxying to two or more backends based on the path (e.g., test.example.com/this-path -> backend.domain.com/can-be-anything).
We're using this proxy as a caching server, so I'd also be open to doing this with Varnish + Nginx for SSL termination if it better supports the use case.
Open to any suggestions from the community, and thanks!