I would like to communicate with redis from nginx, in order to store in lists which requests have been made on images especially on images not founds that are proxied on another server.
I installed OpenResty, in order to use redis2_query and redis2_pass commands.
Here is my nginx config :
location ~* \.(jpg|jpeg|gif|png)$ {
try_files $uri #imagenotfound;
redis2_query lpush founds $uri;
redis2_pass 127.0.0.1:6379;
}
location #imagenotfound {
proxy_pass http://imgdomain.com/$uri;
proxy_set_header Host imgdomain.com;
proxy_set_header Server imgdomain.com;
redis2_query lpush notfounds $uri;
redis2_pass 127.0.0.1:6379;
}
Every request i make returns an integer, and, as i understand, redis2_pass returns the result of the query. Is there anyway to not return this result and just execute the query?
If i remove the redis2_query and redis2_pass, the images are displayed correctly.
Thanks in advance for your help!
A solution that seems to work is to use Lua scripts with access_by_lua and the resty.redis module :
location ~* \.(jpg|jpeg|gif|png)$ {
try_files $uri #imagenotfound;
access_by_lua '
local redis = require "resty.redis"
local red = redis:new()
red:set_timeout(1000)
local ok, err = red:connect("127.0.0.1", 6379)
if not ok then
ngx.say("failed to connect: ", err)
return
end
ok, err = red:lpush("founds", ngx.var.uri)
if not ok then
ngx.say("failed to set founds: ", err)
return
end
';
}
location #imagenotfound {
proxy_pass http://imgdomain.com/$uri;
proxy_set_header Host imgdomain.com;
proxy_set_header Server imgdomain.com;
access_by_lua '
local redis = require "resty.redis"
local red = redis:new()
red:set_timeout(1000)
local ok, err = red:connect("127.0.0.1", 6379)
if not ok then
ngx.say("failed to connect: ", err)
return
end
ok, err = red:lpush("notfounds", ngx.var.uri)
if not ok then
ngx.say("failed to set notfounds: ", err)
return
end
';
}
If anyone has Lua skills and can tell me if it's the right way to do it, i would be glad to have his feedback!
Anyway, thanks for your help in comments.
Related
I am trying to make nginx perform proxying based on the URI with the help of lua and redis.
So far, I am able to successfully proxy simple URI like '/hello' to desired target. Was able to achieve this by saving the mappings in a redis hashmap something like,
HGETALL "127.0.0.1:8080"
1) "/demo1/test/hello4"
2) "example.com/demo1/test/hello4"
3) "/hello"
4) "example.com/hello"
nginx.conf
worker_processes 2;
error_log logs/error.log info;
events {
worker_connections 1024;
}
http {
server {
listen 8080;
location / {
resolver 8.8.4.4; # use Google's open DNS server
set $target '';
access_by_lua '
local http_host = ngx.var.http_host
if not http_host then
ngx.log(ngx.ERR, "no http-host found")
return ngx.exit(400)
end
local redis = require "resty.redis"
local red = redis:new()
red:set_timeout(1000) -- 1 second
local ok, err = red:connect("127.0.0.1", 6379)
if not ok then
ngx.log(ngx.ERR, "failed to connect to redis: ", err)
return ngx.exit(500)
end
local fPath, err = red:hget(http_host, ngx.var.uri)
if not fPath then
ngx.log(ngx.ERR, "No fPath: ", err)
return ngx.exit(500)
end
ngx.var.target = fPath
';
proxy_pass $target;
}
}
}
However, I also want to handle dynamic URI's like example:-
user/id/1 -> "example.com/user/id/1",
user/id/2 -> "example.com/user/id/2",
user/id/3 -> "example.com/user/id/3",
and so on....
I am not sure how can I create a key value pair in redis and lua logic for this case which can handle the dynamicity of the id's.
I tried looking but haven't been able to find the right direction or some resource to aid me in figuring this out.
Any help would be really great!
If you want to achieve this in production, I would recommend using mature API gateways like Apache APISIX or Kong. To implement it yourself, maybe you could store paths with wildcard or Lua patterns in Redis to allow later matching to the original URI. Applying some simple heuristics would help reduce the range of checking.
I have a requirement to do a proxy call to url delivered via a query parameter as per example:
My nginx proxy is deployed at: https://myproxy.net
if the redirect parameter is not url encoded I can do the call with this block:
location /basepath {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
proxy_pass $arg_redirect;
proxy_intercept_errors on;
error_page 301 302 307 = #handle_redirects;
}
the error intercepts and #handle_redirects then take care of othe 30X codes that might pop up at new destination.
This works for a request:
GET: https://myproxy.net/basepath?redirect=https://destination.com/somepath/uuid
What do I need to do to make it work for:
GET: https://myproxy.net/basepath?redirect=https%3A%2F%2Fdestination.com%2Fsomepath%2Fuuid
Additionally as part of spec it has to be pure nginx, not additional modules, lua etc.
Thanks!
Actually, proxy_pass does normalisation by default, but it only affects $uri part. Thus you only need to decode the beginning of the passed string to get it working:
location / {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
if ( $arg_redirect ~ (.+)%3A%2F%2F(.+) ){ # fix :// between scheme and destination
set $arg_redirect $1://$2;
}
if ( $arg_redirect ~ (.+?)%3A(.*) ){ # fix : between destination and port
set $arg_redirect $1:$2;
}
if ( $arg_redirect ~ (.+?)%2F(.*) ){ # fix / after port, the rest will be decoded by proxy_pass
set $arg_redirect $1/$2;
}
proxy_pass $arg_redirect;
}
With the above I managed to access http://localhost/?redirect=http%3A%2F%2F127.0.0.1%3A81%2Fsfoo%20something%2Fs
The solution seems dirty and the only alternative using default modules is map (even less cleaner in my opinion). I'd rather split redirect argument into pieces: scheme (http or https), destination, port, and uri. With that you would be able to construct full address without rewriting:
proxy_pass $arg_scheme://$arg_dest:$arg_port/$arg_uri
Ok, there is very weird and curious solution
server {
listen 80;
resolver x.x.x.x;
location /basepath {
if ($arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
proxy_pass http://127.0.0.1:80/basepath/$arg_redirect;
}
location ~ ^/basepath/(?<proto>\w+):/(?<redir>.+)$ {
proxy_pass $proto://$redir;
}
}
Nginx does not encode path with variables in proxy_pass and send it as is. So, I make $arg_* part of proxy_pass uri, send request to self and nginx will receive new request which will be decoded.
But because Nginx will clean path and replace // to / I split protocol part in regexp.
And ... I would never recommend using this solution, but it works :)
try like this and let me know if it works
location /basepath {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
set_unescape_uri $decodedredirect $arg_redirect;
proxy_pass $decodedredirect;
proxy_intercept_errors on;
error_page 301 302 307 = #handle_redirects;
}
I am using Openresty as a server. I have the configuration file of the nginx as per the https://eclipsesource.com/blogs/2018/01/11/authenticating-reverse-proxy-with-keycloak/.
I am getting following error "openidc.lua:1053: authenticate(): request to the redirect_uri_path but there's no session state found, client"
Can someone throw some light and try to solve the problem.
Regards,
Allahbaksh
Your redirect URI must not be set to "/" but to some arbitrary path that is not supposed to return content (like /redirect_uri). It is a "vanity" URL that is handled by lua-resty-openidc
I had the same problem and was able to fix it by setting the $session_name variable in the server block. Example:
server {
...
server_name proxy.localhost;
#lua_code_cache off;
set $session_name nginx_session;
location / {
access_by_lua_block {
local opts = {
redirect_uri = "http://proxy.localhost/cb",
discovery = "http://127.0.0.1:9000/.well-known/openid-configuration",
client_id = "proxyclient-id",
client_secret = "secret",
ssl_verify = "no",
scope = "openid"
}
-- call authenticate for OpenID Connect user authentication
local res, err = require("resty.openidc").authenticate(opts)
if err then
ngx.status = 500
ngx.say(err)
ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
end
ngx.req.set_header("X-USER", res.id_token.sub)
}
proxy_pass http://localhost:8080/;
proxy_set_header x-forwarded-proto $scheme;
}
}
Another thing to pay attention to is the lua_code_cache off directive; It could break the session. See: https://github.com/bungle/lua-resty-session#notes-about-turning-lua-code-cache-off
I'm evaluating nginx to act as rate limiter for a multi tenancy REST API system. I need to limit API calls by tenant-id.
For example i want to allow 100 r/s for tenant1 and only 50 r/s for tenant2.
It can be easily achived when there are differant urls like: "me.com/tenant1/api" and "me.com/tenant2/api" (with the location directive).
But, in my case the urls are the same for all tenants "me.com/api" (I can't change this).
To find the tenant-id I need to extract a JSON attribute from the Body of the request, and then check the DB for the real tenant-id.
Is it possible to limit_req with my requirements?
Thank for the help!
I decided to build another service getTenant for parsing the body and extracting the Tenant from the DB. This service is called internally by Nginx.
I'm not sure if that is the best nginx (/openresty) solution, but this is what i came up with:
limit_req_zone t1Limit zone=t1Zone:10m rate=200r/s;
limit_req_zone t2Limit zone=t2Zone:10m rate=90r/s;
server {
location /api{
content_by_lua_block {
ngx.req.read_body();
local reqBody = ngx.req.get_body_data()
local res = ngx.location.capture("/getTenant", {method=ngx.HTTP_POST,body=reqBody});
local tenantId= res.body;
if tenantId== "none" then
ngx.log(ngx.ERR, "Tenant not found!");
ngx.say(tenantId);
else
ngx.req.set_header("x_myTenantId", tenantId)
local res2 = ngx.location.capture("/" .. tenantId .."/doApi", {method=ngx.HTTP_POST,body=reqBody});
if res2.status == ngx.HTTP_OK then
ngx.say(res2.body);
ngx.exit(res2.status);
else
ngx.status = res2.status
ngx.exit(res2.status)
end
end;
}
}
location /getTenant {
internal; #this is not accessible from outside.
proxy_pass http://UpStream1/getCustomer;
proxy_set_header X-Original-URI $request_uri;
}
location /tenant1/doApi {
internal; #this is not accessible from outside.
# Proxy all requests to the AReqUpStream server group
proxy_pass http://UpStream2/doApi;
limit_req zone=tenant1Zone burst=25;
limit_req_log_level notice;
}
location /tenant2/doApi {
internal; #this is not accessible from outside.
# Proxy all requests to the AReqUpStream server group
proxy_pass http://UpStream2/doApi;
limit_req zone=tenant2Zone burst=10 ;#nodelay;
limit_req_status 409;
limit_req_log_level notice;
}
}
Basically, when me.com/api is called, a new subrequest is issued to service /getTenant. The response of that call is used to build another subrequest call to the /tenant[X]/doApi service. That way i can define locations per tenant and provide different rate_limis to each.
Comments on that are more than welcome!
I am trying to serve static content from S3. I am using nginx and lua script for its configuration. I am new to both nginx and lua.
The task that I want to achieve is :
Get request URL into nginx.
Authenticate query params of url.
Serve from S3 if parameters are valid.
Send error response if parameters are not valid.
My nginx.conf file is as follows :
location ~ "^/media/(.*?)/(.*?)/(.*)$" {
set $mediaUrl "$1/$2/$3";
set $key "$2/$3"
set $target http://$1.s3.amazonaws.com
rewrite_by_lua "
local uri = '/authenticate'
local res = ngx.location.capture(uri, {args = { param = '/xmedia/'.. ngx.var.mediaUrl }})
if (res.status ~= 200) then
return ngx.exit(ngx.HTTP_GONE)
end
";
rewrite .* /$key break;
proxy_pass $target;
}
location "/authenticate" {
proxy_set_header Range "";
proxy_set_header Content-Range "";
set_by_lua $param "
local params = ngx.req.get_uri_args()
return params.param
";
set $test_url http://127.0.0.1:some_port/authenticate?url=$param;
proxy_pass $test_url;
}
I my case, if authenticate returns 200, then everything works fine. But even if authenticate returns null, nginx returns correct file and does't give error report specified in if statement : return ngx.exit(ngx.HTTP_GONE).
Am I doing something wrong? How to achieve expected behavior in efficient manner?
Thanks.
As already mentioned on your question's comments, the HttpRewriteModule is always executed before rewrite_by_lua; therefore you have to put the logic on the rewrite_by_lua section using ngx.req.set_uri, like this:
location ~ "^/media/(.*?)/(.*?)/(.*)$" {
set $mediaUrl "$1/$2/$3";
set $key "$2/$3"
set $target http://$1.s3.amazonaws.com
rewrite_by_lua "
local uri = '/authenticate'
local res = ngx.location.capture(uri, {args = { param = '/xmedia/'.. ngx.var.mediaUrl }})
if (res.status ~= 200) then
ngx.exit(ngx.HTTP_GONE)
else
ngx.req.set_uri(string.format('/%s', ngx.var.key))
end
";
proxy_pass $target;
}
...