Predender.io docker middleware redirect loop 301 - nginx

I am trying to setup prerender.io for my react platform on AWS ECS along with load balancer, but I keep getting 301 redirect loops, when I visit primary domain.
According to official documentation and github source code, I configured AWS Load balancer accordingly in Load balancer listener rules:
For HTTPS:443, if Host is www2.example.com Forward to ecs_website_container
For HTTPS:443, if Host is example.com OR www.example.com Forward to ecs_prerender_io_container
For HTTP:80, redirect all traffic to https://#{host}:443/#{path}?#{query}
When I check prerender docker logs, every time I visit primary domain example.com I do see some activity in logs, for example:
[13/Aug/2022:18:32:57 +0000] "GET / HTTP/1.1" 301 134 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" "12.34.56.789"
So thats the 301 redirect error. It doesnt get to the point of forwarding to www2.example.com.
The source code is exactly same as in official github repository, just the variables are changed obviously. If I go to www2.example.com directly, website works normally, so the problem must be somewhere in the middleware configuration.
I tracked down the redirects and it is as follows (from 1 to 2, from 2 to 3, etc):
1. http://example.com
2. https://example.com:443/
3. https://www.example.com:443/
4. https://www.example.com:443/
5. https://www.example.com:443/
... infinite loop ...
For the sake of clarity, here is default.conf.template nginx configuration which is used:
map $http_user_agent $prerender_ua {
default 0;
"~*Prerender" 0;
"~*googlebot" 1;
"~*yahoo!\ slurp" 1;
"~*bingbot" 1;
"~*yandex" 1;
"~*baiduspider" 1;
"~*facebookexternalhit" 1;
"~*twitterbot" 1;
"~*rogerbot" 1;
"~*linkedinbot" 1;
"~*embedly" 1;
"~*quora\ link\ preview" 1;
"~*showyoubot" 1;
"~*outbrain" 1;
"~*pinterest\/0\." 1;
"~*developers.google.com\/\+\/web\/snippet" 1;
"~*slackbot" 1;
"~*vkshare" 1;
"~*w3c_validator" 1;
"~*redditbot" 1;
"~*applebot" 1;
"~*whatsapp" 1;
"~*flipboard" 1;
"~*tumblr" 1;
"~*bitlybot" 1;
"~*skypeuripreview" 1;
"~*nuzzel" 1;
"~*discordbot" 1;
"~*google\ page\ speed" 1;
"~*qwantify" 1;
"~*pinterestbot" 1;
"~*bitrix\ link\ preview" 1;
"~*xing-contenttabreceiver" 1;
"~*chrome-lighthouse" 1;
"~*telegrambot" 1;
}
map $args $prerender_args {
default $prerender_ua;
"~(^|&)_escaped_fragment_=" 1;
}
map $http_x_prerender $x_prerender {
default $prerender_args;
"1" 0;
}
map $uri $prerender {
default $x_prerender;
"~*\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)" 0;
}
server {
listen 80;
listen [::]:80;
server_name ${PRIMARY_DOMAIN};
location / {
if ($prerender = 1) {
rewrite (.*) /prerenderio last;
}
proxy_set_header Host $SERVER_NAME;
proxy_set_header Connection "";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_hide_header Cache-Control;
add_header Cache-Control "public,max-age=31536000";
#resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
resolver 8.8.8.8 8.8.4.4;
#setting backend as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
set $backend "${SECONDARY_DOMAIN}";
rewrite .* $uri break;
proxy_pass http://$backend;
}
location /prerenderio {
if ($prerender = 0) {
return 404;
}
proxy_set_header X-Prerender-Token ${PRERENDER_API_KEY};
proxy_hide_header Cache-Control;
add_header Cache-Control "private,max-age=600,must-revalidate";
#resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
resolver 8.8.8.8 8.8.4.4;
set $prerenderio_host "${PRERENDER_HOST}";
proxy_pass http://$prerenderio_host;
rewrite .* /https://${PRIMARY_DOMAIN}$request_uri break;
}
}
Tried to contact support but no response.

Related

Why does rate limit not working when using with return directive in Nginx?

I had following nginx configuration:
worker_processes 1;
error_log logs/error.log;
events {
worker_connections 1024;
}
http {
limit_req_zone $request_uri zone=by_uri_6000:10m rate=6000r/s;
server {
listen 80;
server_name "openresty.com";
location = /limit-6000 {
limit_req zone=by_uri_6000 nodelay;
return 200 "ok"
}
}
}
But it is not working at all when testing using wrk:
wrk -c100 -t10 -d5s http://localhost/limit-6000
Running 5s test # http://localhost/
10 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 5.57ms 1.65ms 23.47ms 81.17%
Req/Sec 1.81k 495.91 9.27k 99.60%
90400 requests in 5.10s, 17.93MB read
Requests/sec: 17713.29
Transfer/sec: 3.51MB
But, if I change my configuration to something like below (I was using openresty in fact):
location = /limit-6000 {
limit_req zone=by_uri_6000 nodelay;
default_type 'application/json';
content_by_lua_block {
ngx.say('{"code": 0, "msg": "成功"}')
}
it works then, what could be the reason, I didn't see any explanation in official document.
nginx return directive works on rewrite phase, it immediately returns the result.
I believe ngx_http_limit_req_module works on access phase. So with return directive ngx_http_limit_req_module doesn't have any chances to be in game.

conditional routing with nginx based on referer

I need to route traffic based on the http request origin. I have two environments and we need to redirect every http request for "/us-en" to Environment1 and others to Environment2 using "$http_referer".
Redirection based on location works.
location ~ /us-en {
proxy_pass Environment1;
proxy_set_header Host Environment1;
}
With '$http_referer' the below option does not work. Request your suggestion on the same.
if ($http_referer ~ ^https?://dev.xyz.com/us-en){
rewrite ^/us-en(/*)$ HOME_PAGE$1 break;
proxy_pass Environment1;
}
Error: nginx: [emerg] "proxy_pass" directive is not allowed here in /opt/nginx/conf/nginx.conf.
Note: By default all the traffic goes to Environment2 as an upstream configuration is present.
# needed if your proxy destination specified with domain name instead of IP address
resolver 8.8.8.8;
location /home/ {
proxy_set_header Host HOST1;
# setup other proxied headers if needed
if ($http_referer ~ ^https?://dev.xyz.com/home) {
rewrite ^/home(/.*)$ HOME_PAGE$1 break;
proxy_pass https://HOST1:8080; # this can be specified by IP address
}
}
With such configuration requests to your_domain.com/home/path/file from dev.xyz.com/home/... (but not from dev.xyz.com/any/other/path!) will be proxied to https://HOST1:8080/HOME_PAGE/path/file. If you specify your proxy destination with domain name instead of IP address, you'll need to specify the additional parameter resolver in your server config. You can use your local name server if you have one, or use something external like Google public DNS (8.8.8.8) or DNS provided for you by your ISP. Anyway such configuration leads to additional DNS lookups, so if you can, specify your proxy destination with IP address.
Update
There is another way to do it with the valid_referers directive:
# needed if your proxy destination specified with domain name instead of IP address
resolver 8.8.8.8;
location /home/ {
proxy_set_header Host HOST1;
# setup other proxied headers if needed
valid_referers example.com/home;
if ($invalid_referer = "") {
rewrite ^/home(/.*)$ HOME_PAGE$1 break;
proxy_pass https://HOST1:8080; # this can be specified by IP address
}
}
Update # 2020.11.11
Besides this answer somehow achieved a score of 5, the given solution has an extremely bad design (it isn't a good approach to have different content handlers in the location and the nested if block; moreover, having an if block with any directive other than from the nginx rewrite module should be avoided if possible) and won't work at all on early nginx versions (I wanna cry when I look at some of my early answers). An original OP question was
The logic should be like below but has some syntax mistakes.
if ($http_origin ~ '^http?://(dev.xyz.com/home)') {
set $flag 'true';
}
if ($flag = 'true') {
location /home/ {
proxy_pass "https://HOST1:8080/HOME PAGE/";
}
}else{
Do Not proxy pass
}
It is unclear what do not proxy pass means. If it means returning some HTTP error (for example, HTTP 403 Forbidden), it can be done with the following configuration:
location /home/ {
if ($http_referer !~ ^https?://dev.xyz.com/home) {
return 403;
}
rewrite ^/home(/.*)$ HOME_PAGE$1 break;
proxy_set_header Host HOST1;
# setup other proxied headers if needed
proxy_pass https://HOST1:8080; # this can be specified by IP address
}
If do not proxy pass means to serve the request locally, the solution is more complex:
map $http_referer $loc {
~^https?://dev.xyz.com/home loc_proxy;
default loc_local;
}
server {
...
location /home/ {
try_files /dev/null #$loc;
}
location #loc_proxy {
rewrite ^/home(/.*)$ HOME_PAGE$1 break;
proxy_set_header Host HOST1;
# setup other proxied headers if needed
proxy_pass https://HOST1:8080;
}
location #loc_local {
rewrite ^/home(/.*)$ HOME_PAGE$1 break;
root /path/to/required/page;
...
}
The try_files /dev/null #the_named_location; trick is taken from this excellent answer.
However now the edited OP's question states for a different requirements, which also could be achieved with the map directive help:
map $http_referer $environment {
~^https?://dev.xyz.com/home Environment1;
default Environment2;
}
server {
...
location /home/ {
rewrite ^/home(/.*)$ HOME_PAGE$1 break;
proxy_set_header Host $environment;
# setup other proxied headers if needed
proxy_pass https://$environment;
}

Nginx locations configuration

Let's say I have two upstreams: upstream1 and upstream2.
Also I have API endpount /api/thing.
I want to configure two locations in nginx like this:
location /api/thing/? {
proxy_intercept_errors on;
proxy_pass http://upstream1;
}
location /api/thing/* {
proxy_intercept_errors on;
proxy_pass http://upstream2; }
In locations I used regular regexps (non-nginx configuration syntax):
Requests to /api/thing, /api/thing?param1=val1&param2=val2, etc should be proxied to upstream1
Requests to /api/thing/subthing1?a=b, /api/thing/subthing2/, etc should be proxied to upstream2
Is it possible in terms of nginx configuration?
I would write:
location = /api/thing {
# upstream1
}
location = /api/thing/ {
# upstream1
}
location /api/thing/ {
# upstream2
}

nginx limit_req_zone doesn't seem to work

I am trying to setup a global limit to a certain uri using nginx via openresty
with the following config, if I curl this box I get a 204 no matter how many times I request it per minute.
worker_processes 1;
error_log logs/error.log;
events {
}
http {
log_format spec_format '$request_uri $const $status';
access_log logs/access.log spec_format;#off;
resolver 10.0.0.2;
limit_req_log_level error;
limit_req_zone $const zone=one:100k rate=1r/m;
server {
set $const 1;
listen 80;
location / {
return 200 "invalid url";
}
location ~* /request/? {
limit_req zone=one burst=1 nodelay;
return 204;
}
location /health/ {
return 200 "healthy";
}
}
}
From the docs I can't find anything obvious (Ive tried switching things around a lot).
In case it helps, the box is running on AWS behind an EIP and Ubuntu 13.10. I'm using openresty-1.5.8.1 from openresty.org.
Also, the actual limit I want to work is for around 24000r/s and there are other settings I thought might be conflicting, but even stripped down it doesn't behave like I thought it should.
The problem may be that $const is not yet set when limit_req_zone processes it. Are you seeing the correct $cost value in the log?
The limit_req_module will ignore empty values, from the docs:
The key is any non-empty value of the specified variable (empty values
are not accounted).

nGINX read Header & Pass it to different Application Server

I have an nGINX server running. I want to read custom HTTP Header from the incoming Request and redirect it to different application server. I did search for similar questions but found for writing custom headers but not how to read..
if a header is set with this -> "version = Version 1.0" then it should redirect different application (say uwsgi_pass x.x.x.x:80)
if it is set as "version = Version 2.0" then it should redirect to (uwsgi_pass x.x.x.x:99)
I tried in my nginx.conf file
server{
listen 80;
server_name xyz.com;
if ($http_version ~ 'Version 1.0') {
proxy_pass http://192.168.0.116:99/calc;
}
if ($http_version ~ 'Version 2.0') {
proxy_pass http://192.168.0.116:99;
}
location /hello {
proxy_pass http://192.168.0.116:99/calc;
}
}
I'm getting error when I restart my nGINX
nginx: [emerg] "proxy_pass" directive is not allowed here in /etc/nginx/nginx.conf:19
nginx: configuration file /etc/nginx/nginx.conf test failed
Suppose you set a custom header in this form:
version: Version 1.0
Then you can configure nginx like in this way:
location / {
if ($http_version ~ 'Version 1.0') {
uwsgi_pass localhost:8888;
}
if ($http_version ~ 'Version 2.0') {
uwsgi_pass localhost:9999;
}
}
Reference: http://wiki.nginx.org/HttpCoreModule#.24http_HEADER

Resources