I have below varnish command:
sudo varnishlog -c
and its output
* << Request >> 658516
- Begin req 658515 rxreq
- ReqMethod GET
- ReqURL /sample/2
- VCL_call HIT
- RespHeader X-Timestamp: 1460482977.61998
- RespHeader X-Varnish: 658516 658416
- RespHeader X-Varnish-Cache: HIT
From this command output I want to redirect the output to a file with following fomat:
Begin="req 658515 rxreq",ReqURL="/sample/2", RespHeader="X-Varnish-Cache: HIT"
I have used grep command to get the required fields:
sudo varnishlog -c | grep -E 'Begin|ReqURL|Varnish-Cache'
- Begin req 658515 rxreq
- ReqURL /sample/2
- RespHeader X-Varnish-Cache: HIT
But if I used additional commands to replace space and new lines I am facing issue.
sudo varnishlog -c | grep -E 'Begin|ReqURL|Varnish-Cache' | sed 's/ /=/g'
With this command I am not getting any output.
If I use sed or tr commands:
sudo varnishlog -c | sed 's/ /=/g'
or
sudo varnishlog -c | tr ' ' '='
Then output is :
*===<<=Request==>>=629459====
-===Begin==========req=629458=rxreq
-===ReqMethod======GET
-===ReqURL=========/sample/2
-===VCL_call=======HIT
-===RespHeader=====X-Varnish-Cache:=HIT
If I use this:
sudo varnishlog -c | sed 's/\t/=/g'
Then output is same as original :
* << Request >> 658516
- Begin req 658515 rxreq
- ReqMethod GET
- ReqURL /sample/2
- VCL_call HIT
- RespHeader X-Timestamp: 1460482977.61998
- RespHeader X-Varnish: 658516 658416
- RespHeader X-Varnish-Cache: HIT
please help me with some hints on what is the correct way to get my required output.
#Sundeep, Perl version on my machine is :
perl -version
This is perl 5, version 18, subversion 2 (v5.18.2) built for x86_64-linux-gnu-thread-multi
(with 44 registered patches, see perl -V for more detail)
#Sundeep,
Output of sudo varnishlog -c | cat -A
$
* << Request >> 363192 $
- Begin req 363191 rxreq$
- Timestamp Start: 1478514424.525802 0.000000 0.000000$
- Timestamp Req: 1478514424.525802 0.000000 0.000000$
- ReqStart 10.56.36.2 52583$
- ReqMethod GET$
- ReqURL /sample/2$
- ReqProtocol HTTP/1.1$
- ReqHeader Host: localhost:6081$
- ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu;....$
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8$
- ReqHeader Accept-Language: en-US,en;q=0.5$
- ReqHeader Accept-Encoding: gzip, deflate$
- ReqHeader Connection: keep-alive$
- ReqHeader Pragma: no-cache$
- ReqHeader Cache-Control: no-cache$
- ReqHeader X-Forwarded-For: 10.56.36.2$
- VCL_call RECV$
- VCL_return hash$
- ReqUnset Accept-Encoding: gzip, deflate$
- ReqHeader Accept-Encoding: gzip$
- VCL_call HASH$
- VCL_return lookup$
- Hit 658416$
- VCL_call HIT$
- VCL_return deliver$
- RespProtocol HTTP/1.1$
- RespStatus 200$
- RespReason OK$
- RespHeader Content-Length: 4774$
- RespHeader Last-Modified: Tue, 12 Apr 2016 17:42:58 GMT$
- RespHeader Etag: f9d34a65e8c1b30245c2f12534348ff9$
- RespHeader X-Timestamp: 1460482977.61998$
- RespHeader Content-Type: image/png$
- RespHeader X-Trans-Id: txf52283ea65004e578ddfe-0058203d84$
- RespHeader Date: Mon, 07 Nov 2016 08:38:28 GMT$
- RespHeader X-Varnish: 363192 658416$
- RespHeader Age: 6516$
- RespHeader Via: 1.1 varnish-v4$
- VCL_call DELIVER$
- RespHeader X-Varnish-Cache: HIT$
- VCL_return deliver$
- Timestamp Process: 1478514424.525880 0.000078 0.000078$
- RespHeader Accept-Ranges: bytes$
- Debug "RES_MODE 2"$
- RespHeader Connection: keep-alive$
- Timestamp Resp: 1478514424.525908 0.000106 0.000028$
- ReqAcct 415 0 415 394 4774 5168$
- End $
maybe this could help you;
sudo varnishlog -c | grep --line-buffered -E 'Begin|ReqURL|Varnish-Cache' | sed -u 's/- //;s/ */="/;s/$/"/' | tr '\n' ','
Related
I am using tileserver-gl and want to use Varnish to cache the tiles. Varnish is working with nginx, that is taking care of the SSL.
The pbf tiles return Age:0 in the header (and a MISS), while the Date and Expires headers keep their value between reloads. What am I doing wrong?
In nginx I am using the following parameters
expires 2d;
add_header Cache-Control "public, no-transform";
Here's my .vcl
sub vcl_recv {
unset req.http.cookie; #added as a test, nothing changed so don't know if I should keep it
}
sub vcl_backend_response {
}
sub vcl_deliver {
if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and the number of hits
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
}
Here's response header for one of the pbf tiles. They all give similar results, with Age: 0
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Age: 0
Cache-Control: max-age=172800, public, no-transform
Content-Encoding: gzip
Content-Length: 44891
Content-Type: application/x-protobuf
Date: Fri, 09 Dec 2022 10:30:34 GMT
ETag: W/"af5b-yQTY5UUgiSJNWC0axguA5dmojbI"
Expires: Sun, 11 Dec 2022 10:30:34 GMT
Last-Modified: Wed, 23 Nov 2022 09:25:23 GMT
Server: nginx/1.18.0 (Ubuntu)
Vary: Accept-Encoding
Via: 1.1 varnish (Varnish/6.6)
X-Cache: MISS
X-Varnish: 557061
EDIT
Tried Thijs suggestion, here's the output
* << Request >> 98318
- Begin req 98317 rxreq
- Timestamp Start: 1670854870.784403 0.000000 0.000000
- Timestamp Req: 1670854870.784403 0.000000 0.000000
- VCL_use boot
- ReqStart 127.0.0.1 38402 a0
- ReqMethod GET
- ReqURL /data/2017-07-03_europe_italy/0/0/0.pbf
- ReqProtocol HTTP/1.0
- ReqHeader X-Forwarded-Host: tiles.urbismap.com
- ReqHeader X-Real_IP: 93.41.126.66
- ReqHeader X-Forwarded-For: 93.41.126.66
- ReqHeader X-Forwarded-Proto: https
- ReqHeader Host: tiles.urbismap.com
- ReqHeader Connection: close
- ReqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36
- ReqHeader Accept: */*
- ReqHeader Sec-Fetch-Site: same-origin
- ReqHeader Sec-Fetch-Mode: cors
- ReqHeader Sec-Fetch-Dest: empty
- ReqHeader Referer: https://tiles.urbismap.com/styles/basemap/
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Language: it-IT,it;q=0.9
- ReqUnset X-Forwarded-For: 93.41.126.66
- ReqHeader X-Forwarded-For: 93.41.126.66, 127.0.0.1
- VCL_call RECV
- VCL_return hash
- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Encoding: gzip
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 98319 fetch
- Timestamp Fetch: 1670854870.792969 0.008566 0.008566
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Server: nginx/1.18.0 (Ubuntu)
- RespHeader Date: Mon, 12 Dec 2022 14:21:10 GMT
- RespHeader Content-Type: application/x-protobuf
- RespHeader Content-Length: 31578
- RespHeader Access-Control-Allow-Origin: *
- RespHeader Content-Encoding: gzip
- RespHeader Last-Modified: Wed, 23 Nov 2022 09:25:23 GMT
- RespHeader ETag: W/"7b5a-Lgi1EP0ZRDWVK1iX+VP47MXHk2M"
- RespHeader Expires: Wed, 14 Dec 2022 14:21:10 GMT
- RespHeader Cache-Control: max-age=172800, public, no-transform
- RespHeader Vary: Accept-Encoding
- RespHeader X-Varnish: 98318
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/6.6)
- VCL_call DELIVER
- RespHeader X-Cache: MISS
- VCL_return deliver
- Timestamp Process: 1670854870.793004 0.008600 0.000034
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Connection: close
- Timestamp Resp: 1670854870.794239 0.009835 0.001235
- ReqAcct 555 0 555 524 31578 32102
- End
** << BeReq >> 98319
-- Begin bereq 98318 fetch
-- VCL_use boot
-- Timestamp Start: 1670854870.784497 0.000000 0.000000
-- BereqMethod GET
-- BereqURL /data/2017-07-03_europe_italy/0/0/0.pbf
-- BereqProtocol HTTP/1.0
-- BereqHeader X-Forwarded-Host: tiles.urbismap.com
-- BereqHeader X-Real_IP: 93.41.126.66
-- BereqHeader X-Forwarded-Proto: https
-- BereqHeader Host: tiles.urbismap.com
-- BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36
-- BereqHeader Accept: */*
-- BereqHeader Sec-Fetch-Site: same-origin
-- BereqHeader Sec-Fetch-Mode: cors
-- BereqHeader Sec-Fetch-Dest: empty
-- BereqHeader Referer: https://tiles.urbismap.com/styles/basemap/
-- BereqHeader Accept-Language: it-IT,it;q=0.9
-- BereqHeader X-Forwarded-For: 93.41.126.66, 127.0.0.1
-- BereqHeader Accept-Encoding: gzip
-- BereqProtocol HTTP/1.1
-- BereqHeader X-Varnish: 98319
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- Timestamp Fetch: 1670854870.784543 0.000045 0.000045
-- Timestamp Connected: 1670854870.784548 0.000050 0.000005
-- BackendOpen 31 default 127.0.0.1 8080 127.0.0.1 41766 reuse
-- Timestamp Bereq: 1670854870.784608 0.000111 0.000060
-- Timestamp Beresp: 1670854870.792781 0.008283 0.008172
-- BerespProtocol HTTP/1.1
-- BerespStatus 200
-- BerespReason OK
-- BerespHeader Server: nginx/1.18.0 (Ubuntu)
-- BerespHeader Date: Mon, 12 Dec 2022 14:21:10 GMT
-- BerespHeader Content-Type: application/x-protobuf
-- BerespHeader Content-Length: 31578
-- BerespHeader Connection: keep-alive
-- BerespHeader Access-Control-Allow-Origin: *
-- BerespHeader Content-Encoding: gzip
-- BerespHeader Last-Modified: Wed, 23 Nov 2022 09:25:23 GMT
-- BerespHeader ETag: W/"7b5a-Lgi1EP0ZRDWVK1iX+VP47MXHk2M"
-- BerespHeader Expires: Wed, 14 Dec 2022 14:21:10 GMT
-- BerespHeader Cache-Control: max-age=172800
-- BerespHeader Cache-Control: public, no-transform
-- TTL RFC 172800 10 0 1670854871 1670854871 1670854870 1671027670 172800 cacheable
-- VCL_call BACKEND_RESPONSE
-- VCL_return deliver
-- Timestamp Process: 1670854870.792827 0.008329 0.000046
-- Filters testgunzip
-- BerespHeader Vary: Accept-Encoding
-- Storage malloc s0
-- Fetch_Body 3 length stream
-- Gzip u F - 31578 51339 80 141523 252560
-- BackendClose 31 default recycle
-- Timestamp BerespBody: 1670854870.794161 0.009664 0.001334
-- Length 31578
-- BereqAcct 552 0 552 426 31578 32004
-- End
* << Request >> 98350
- Begin req 98349 rxreq
- Timestamp Start: 1670855034.759301 0.000000 0.000000
- Timestamp Req: 1670855034.759301 0.000000 0.000000
- VCL_use boot
- ReqStart 127.0.0.1 47736 a0
- ReqMethod GET
- ReqURL /data/2017-07-03_europe_italy/0/0/0.pbf
- ReqProtocol HTTP/1.0
- ReqHeader X-Forwarded-Host: tiles.urbismap.com
- ReqHeader X-Real_IP: 93.41.126.66
- ReqHeader X-Forwarded-For: 93.41.126.66
- ReqHeader X-Forwarded-Proto: https
- ReqHeader Host: tiles.urbismap.com
- ReqHeader Connection: close
- ReqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36
- ReqHeader Accept: */*
- ReqHeader Sec-Fetch-Site: same-origin
- ReqHeader Sec-Fetch-Mode: cors
- ReqHeader Sec-Fetch-Dest: empty
- ReqHeader Referer: https://tiles.urbismap.com/styles/basemap/
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Language: it-IT,it;q=0.9
- ReqUnset X-Forwarded-For: 93.41.126.66
- ReqHeader X-Forwarded-For: 93.41.126.66, 127.0.0.1
- VCL_call RECV
- VCL_return hash
- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Encoding: gzip
- VCL_call HASH
- VCL_return lookup
- Hit 98319 172636.033479 10.000000 0.000000
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Server: nginx/1.18.0 (Ubuntu)
- RespHeader Date: Mon, 12 Dec 2022 14:21:10 GMT
- RespHeader Content-Type: application/x-protobuf
- RespHeader Content-Length: 31578
- RespHeader Access-Control-Allow-Origin: *
- RespHeader Content-Encoding: gzip
- RespHeader Last-Modified: Wed, 23 Nov 2022 09:25:23 GMT
- RespHeader ETag: W/"7b5a-Lgi1EP0ZRDWVK1iX+VP47MXHk2M"
- RespHeader Expires: Wed, 14 Dec 2022 14:21:10 GMT
- RespHeader Cache-Control: max-age=172800, public, no-transform
- RespHeader Vary: Accept-Encoding
- RespHeader X-Varnish: 98350 98319
- RespHeader Age: 163
- RespHeader Via: 1.1 varnish (Varnish/6.6)
- VCL_call DELIVER
- RespHeader X-Cache: HIT
- VCL_return deliver
- Timestamp Process: 1670855034.759391 0.000089 0.000089
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Connection: close
- Timestamp Resp: 1670855034.759479 0.000177 0.000087
- ReqAcct 555 0 555 531 31578 32109
- End
When press plus bottom for add more than 1 of product to shopping cart on WordPress, Finally product add to cart after one minute, but I see this error in console:
Request URL: https://abcds.com/cart/
Request Method: POST
Status Code: 503 Backend fetch failed
in my vcl I add this value but not work:
# The code below makes sure the AJAX "add to cart" function works
set req.url = regsub(req.url, "add-to-cart=\d+_\d+&", "");
# Do not cache AJAX requests.
if (req.http.X-Requested-With == "XMLHttpRequest") {
return(pass);
}
# Post requests will not be cached
if (req.http.Authorization || req.method == "POST") {
return (pass);
}
# also in vcl_backend_response I add this codes:
if (!(bereq.url ~ "wp-(login|admin)|login|logged-in|my-account|wc-api|resetpass|admin-ajax.php|arisn") && !(bereq.http.cookie ~ "wordpress_logged_in|woocommerce_items_in_cart|resetpass|wp_woocommerce_session_[a-zA-Z0-9]+|wordpress_logged_in_|comment_author|PHPSESSID|woocommerce_cart_hash") && !(beresp.status == 302))
{
unset beresp.http.set-cookie;
set beresp.ttl = 2h;
}
Related images links:
https://i.postimg.cc/hGPTFx0S/header.png
https://i.postimg.cc/BbQ7ztwk/cookies.png
update:
Varnish log:
<< Request >> 20751307
Begin req 20751306 rxreq
Timestamp Start: 1650725983.556894 0.000000 0.000000
Timestamp Req: 1650725983.556894 0.000000 0.000000
ReqStart 127.0.0.1 55316 a0
ReqMethod POST
ReqURL /cart/
ReqProtocol HTTP/1.0
ReqHeader X-Real-IP: 31.14.152.7
ReqHeader X-Forwarded-For: 31.14.152.7
ReqHeader X-Forwarded-Proto: https
ReqHeader X-Nginx: on
ReqHeader Host: bankketab.com
ReqHeader Connection: close
ReqHeader Content-Length: 297
ReqHeader sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Opera";v="85"
ReqHeader accept: text/html, */*; q=0.01
ReqHeader content-type: application/x-www-form-urlencoded; charset=UTF-8
ReqHeader x-requested-with: XMLHttpRequest
ReqHeader sec-ch-ua-mobile: ?0
ReqHeader user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36 OPR/85.0.4341.75
ReqHeader sec-ch-ua-platform: "Windows"
ReqHeader origin: https://bankketab.com
ReqHeader sec-fetch-site: same-origin
ReqHeader sec-fetch-mode: cors
ReqHeader sec-fetch-dest: empty
ReqHeader referer: https://bankketab.com/cart/
ReqHeader accept-encoding: gzip, deflate, br
ReqHeader accept-language: en-US,en;q=0.9
ReqHeader cookie: PHPSESSID=8beb52490b9f7ad5456f8b064a9439a5; woocommerce_recently_viewed=20077%7C20092%7C19496%7C20136; digits_countrycode=98; woocommerce_items_in_cart=1; wp_woocommerce_session_d373d61d3cf9a04133a8550c98d33197=t_df9a955c5d0c76b7f6a9ce2add86d6%7C
ReqUnset X-Forwarded-For: 31.14.152.7
ReqHeader X-Forwarded-For: 31.14.152.7, 127.0.0.1
VCL_call RECV
ReqUnset Host: bankketab.com
ReqHeader host: bankketab.com
ReqURL /cart/
VCL_return pass
VCL_call HASH
VCL_return lookup
VCL_call PASS
VCL_return fetch
Link bereq 20751308 pass
Storage malloc Transient
Timestamp ReqBody: 1650725983.557004 0.000110 0.000110
VCL_Error Uncached req.body can only be consumed once.
Timestamp Fetch: 1650726043.823773 60.266879 60.266769
RespProtocol HTTP/1.1
RespStatus 503
RespReason Backend fetch failed
RespHeader Date: Sat, 23 Apr 2022 15:00:43 GMT
RespHeader Server: Varnish
RespHeader Content-Type: text/html; charset=utf-8
RespHeader Retry-After: 5
RespHeader X-Varnish: 20751307
RespHeader Age: 0
RespHeader Via: 1.1 varnish (Varnish/6.0)
VCL_call DELIVER
RespHeader X-status: 0
RespUnset X-Varnish: 20751307
RespUnset Age: 0
RespUnset Via: 1.1 varnish (Varnish/6.0)
RespHeader X-Configured-By: ServerSetup.co
RespUnset Server: Varnish
RespHeader Server: Nitro
VCL_return deliver
Timestamp Process: 1650726043.823791 60.266897 0.000018
RespHeader Content-Length: 285
RespHeader Connection: close
Timestamp Resp: 1650726043.823839 60.266946 0.000049
ReqAcct 1166 297 1463 231 285 516
End
** << BeReq >> 20751308
-- Begin bereq 20751307 pass
-- Timestamp Start: 1650725983.556958 0.000000 0.000000
-- BereqMethod POST
-- BereqURL /cart/
-- BereqProtocol HTTP/1.0
-- BereqHeader X-Real-IP: 31.14.152.7
-- BereqHeader X-Forwarded-Proto: https
-- BereqHeader X-Nginx: on
-- BereqHeader Content-Length: 297
-- BereqHeader sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Opera";v="85"
-- BereqHeader accept: text/html, */*; q=0.01
-- BereqHeader content-type: application/x-www-form-urlencoded; charset=UTF-8
-- BereqHeader x-requested-with: XMLHttpRequest
-- BereqHeader sec-ch-ua-mobile: ?0
-- BereqHeader user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36 OPR/85.0.4341.75
-- BereqHeader sec-ch-ua-platform: "Windows"
-- BereqHeader origin: https://bankketab.com
-- BereqHeader sec-fetch-site: same-origin
-- BereqHeader sec-fetch-mode: cors
-- BereqHeader sec-fetch-dest: empty
-- BereqHeader referer: https://bankketab.com/cart/
-- BereqHeader accept-encoding: gzip, deflate, br
-- BereqHeader accept-language: en-US,en;q=0.9
-- BereqHeader cookie: PHPSESSID=8beb52490b9f7ad5456f8b064a9439a5; woocommerce_recently_viewed=20077%7C20092%7C19496%7C20136; digits_countrycode=98; woocommerce_items_in_cart=1; wp_woocommerce_session_d373d61d3cf9a04133a8550c98d33197=t_df9a955c5d0c76b7f6a9ce2add86d6%7C
-- BereqHeader X-Forwarded-For: 31.14.152.7, 127.0.0.1
-- BereqHeader host: bankketab.com
-- BereqProtocol HTTP/1.1
-- BereqHeader X-Varnish: 20751308
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- BackendOpen 50 boot.cPanelServer 144.76.9.138 8080 144.76.9.138 34358
-- BackendStart 144.76.9.138 8080
-- Timestamp Bereq: 1650725983.557008 0.000050 0.000050
-- Timestamp Beresp: 1650725983.823515 0.266558 0.266507
-- BerespProtocol HTTP/1.1
-- BerespStatus 302
-- BerespReason Found
-- BerespHeader Connection: Keep-Alive
-- BerespHeader Expires: Wed, 11 Jan 1984 05:00:00 GMT
-- BerespHeader Cache-Control: no-cache, no-store, must-revalidate, max-age=0
-- BerespHeader X-Redirect-By: WordPress
-- BerespHeader Location: https://bankketab.com/cart/
-- BerespHeader Set-Cookie: woocommerce_cart_hash=3f7d291f9814ba9163415e03a89629cf; path=/; secure
-- BerespHeader Content-Type: text/html; charset=UTF-8
-- BerespHeader Content-Length: 0
-- BerespHeader Date: Sat, 23 Apr 2022 14:59:43 GMT
-- VCL_call BACKEND_RESPONSE
-- BerespHeader Vary: User-Agent
-- VCL_return retry
-- BackendReuse 50 boot.cPanelServer
-- Timestamp Retry: 1650725983.823536 0.266578 0.000021
-- Link bereq 19813441 retry
-- End
*** << BeReq >> 19813441
--- Begin bereq 20751308 retry
--- Timestamp Start: 1650725983.823536 0.266578 0.000000
--- BereqMethod POST
--- BereqURL /cart/
--- BereqProtocol HTTP/1.1
--- BereqHeader X-Real-IP: 31.14.152.7
--- BereqHeader X-Forwarded-Proto: https
--- BereqHeader X-Nginx: on
--- BereqHeader Content-Length: 297
--- BereqHeader sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Opera";v="85"
--- BereqHeader accept: text/html, */*; q=0.01
--- BereqHeader content-type: application/x-www-form-urlencoded; charset=UTF-8
--- BereqHeader x-requested-with: XMLHttpRequest
--- BereqHeader sec-ch-ua-mobile: ?0
--- BereqHeader user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36 OPR/85.0.4341.75
--- BereqHeader sec-ch-ua-platform: "Windows"
--- BereqHeader origin: https://bankketab.com
--- BereqHeader sec-fetch-site: same-origin
--- BereqHeader sec-fetch-mode: cors
--- BereqHeader sec-fetch-dest: empty
--- BereqHeader referer: https://bankketab.com/cart/
--- BereqHeader accept-encoding: gzip, deflate, br
--- BereqHeader accept-language: en-US,en;q=0.9
--- BereqHeader cookie: PHPSESSID=8beb52490b9f7ad5456f8b064a9439a5; woocommerce_recently_viewed=20077%7C20092%7C19496%7C20136; digits_countrycode=98; woocommerce_items_in_cart=1; wp_woocommerce_session_d373d61d3cf9a04133a8550c98d33197=t_df9a955c5d0c76b7f6a9ce2add86d6%7C
--- BereqHeader X-Forwarded-For: 31.14.152.7, 127.0.0.1
--- BereqHeader host: bankketab.com
--- BereqHeader X-Varnish: 20751308
--- BereqUnset X-Varnish: 20751308
--- BereqHeader X-Varnish: 19813441
--- VCL_call BACKEND_FETCH
--- VCL_return fetch
--- BackendOpen 50 boot.cPanelServer 144.76.9.138 8080 144.76.9.138 34358
--- BackendStart 144.76.9.138 8080
--- FetchError backend write error: 0 (Success)
--- Timestamp Bereq: 1650725983.823596 0.266638 0.000060
--- FetchError Timed out reusing backend connection
--- BackendClose 50 boot.cPanelServer
--- Timestamp Beresp: 1650726043.823712 60.266755 60.000117
--- Timestamp Error: 1650726043.823717 60.266759 0.000004
--- BerespProtocol HTTP/1.1
--- BerespStatus 503
--- BerespReason Service Unavailable
--- BerespReason Backend fetch failed
--- BerespHeader Date: Sat, 23 Apr 2022 15:00:43 GMT
--- BerespHeader Server: Varnish
--- VCL_call BACKEND_ERROR
--- BerespHeader Content-Type: text/html; charset=utf-8
--- BerespHeader Retry-After: 5
--- VCL_return deliver
--- Storage malloc Transient
--- Length 285
--- BereqAcct 2358 297 2655 394 0 394
--- End
The request body error
As mentioned in the comments, the following log output is quite interesting:
Timestamp ReqBody: 1650725983.557004 0.000110 0.000110
VCL_Error Uncached req.body can only be consumed once.
Timestamp Fetch: 1650726043.823773 60.266879 60.266769
The Uncached req.body can only be consumed once error implies that your retrying a POST call and lose the request body in the process.
The following VCL code from your VCL file has logic that could cause this behavior:
if (beresp.http.Location == "https://" + bereq.http.host + bereq.url) {
if (bereq.retries > 1)
{
unset beresp.http.Location;
}
else {
return (retry);
}
}
The fact that the request is a POST request where the output is a 302 redirect with a Location: https://bankketab.com/cart/ header triggers that part of the VCL.
The problem with retrying a request that has a request body (such as a typical POST request), is that Varnish doesn't cache the request body. That information is lost when asking for a retry.
The HTTP 503 backend fetch failure
All of this explains the VCL_Error Uncached req.body can only be consumed once. error.
The actual HTTP 503 error could be something entirely different. However, they could be related.
The logs below refer to a backend write error:
--- BackendOpen 50 boot.cPanelServer 144.76.9.138 8080 144.76.9.138 34358
--- BackendStart 144.76.9.138 8080
--- FetchError backend write error: 0 (Success)
--- Timestamp Bereq: 1650725983.823596 0.266638 0.000060
--- FetchError Timed out reusing backend connection
--- BackendClose 50 boot.cPanelServer
I'm not 100% sure, but it could be that the backend is waiting for the POST request body. It doesn't get it, times out and returns the error.
Monitoring backend health
A backend returns an HTTP 503 error when it is not available or when that backend doesn't return the expected result in a timely fashion.
Luckily there's a varnishlog command that can help us check the status of a pre-defined health probe.
Your VCL code happens to feature a backend_healthcheck health probe that polls the backend every 5 seconds.
Please run the following command to check the general health of your backend:
varnishlog -g raw -i backend_health
Every 5 seconds there should be a new log line with the status of the health check. Run it for a minute or so and see what the status is. If the backend is healthy, there is no critical problem with the backend. This can also help us conclude that the problem you're experiencing is really related to the absence of the request body in the retry.
Solving the problem
My advice is to use the std.cache_req_body() function that is described in https://varnish-cache.org/docs/6.0/reference/vmod_generated.html#func-cache-req-body to ensure the request body services a return(retry) in the VCL code.
Here's an oversimplified VCL example that features this function:
sub vcl_recv {
if(std.cache_req_body(1KB)) {
std.log("Request body accessible");
} else {
std.log("Request body not accessible");
}
}
A boolean value is returned from std.cache_req_body() describing whether or not retrieving the request body from cache was successful. You could return a synthetic error in the else path if you want.
There's no guarantee that caching the request body will solve the issue, but there are indications in the log that the failure can be caused by the absence of the request body. In any case it makes sense to run varnishlog -g raw -i backend_health to figure out what the general health is of your backend.
I was just sending some data to my device in thingsboard by writing this command on command line
curl -v -X POST -d "{\"temperature\": 25}" http://demo.thingsboard.io/devices/api/v1/IG4XXXXXXXXXCQM/telemetry
--header "Content-Type:application/json"
but I get this error message at the last
{"timestamp":"2020-01-16T13:09:05.031+0000","status":405,"error":"Method Not Allowed","message":"Request method 'POST' not supported","path":"/devices/api/v1/IG4Dxxxxxxxxxxxxxxxxs6CCQM/telemetry"}* Connection #0 to host demo.thingsboard.io left intact
what's it I am not doing right?
All the efforts are appreciated
Following are the documentation and whole error message
Thingsboard
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 104.196.24.70:80...
* TCP_NODELAY set
* Connected to demo.thingsboard.io (104.196.24.70) port 80 (#0)
> POST /devices/api/v1/IGxxxxxxxs6CCQM/telemetry HTTP/1.1
> Host: demo.thingsboard.io
> User-Agent: curl/7.65.1
> Accept: */*
> Content-Type:application/json
> Content-Length: 19
>
* upload completely sent off: 19 out of 19 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 405
< Allow: GET, HEAD
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< Content-Type: application/json;charset=UTF-8
< Content-Language: en
< Transfer-Encoding: chunked
< Date: Thu, 16 Jan 2020 13:09:04 GMT
<
{"timestamp":"2020-01-16T13:09:05.031+0000","status":405,"error":"Method Not Allowed","message":"Request method 'POST' not supported","path":"/devices/api/v1/IGxxxxxxxxxxxs6CCQM/telemetry"}* Connection #0 to host demo.thingsboard.io left intact
You need to replace 'device' with 'api' in your URL,in case you are using access tokens, hence, instead of
curl -v -X POST -d "{\"temperature\": 25}" http://demo.thingsboard.io/devices/api/v1/IG4XXXXXXXXXCQM/telemetry
--header "Content-Type:application/json"
You need to use
curl -v -X POST -d "{\"temperature\": 25}" http://demo.thingsboard.io/api/v1/IG4XXXXXXXXXCQM/telemetry
--header "Content-Type:application/json"
Apache Tika should be accessible from Python program via HTTP, but I can't get it to work.
I am using this command to run the server (with and without the two options at the end):
java -jar tika-server-1.17.jar --port 5677 -enableUnsecureFeatures -enableFileUrl
And it works fine with curl:
curl -v -T /tmp/tmpsojwBN http://localhost:5677/tika
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5677 (#0)
> PUT /tika HTTP/1.1
> Host: localhost:5677
> User-Agent: curl/7.47.0
> Accept: */*
> Accept-Encoding: gzip, deflate
> Content-Length: 418074
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Date: Sat, 07 Apr 2018 12:28:41 GMT
< Transfer-Encoding: chunked
< Server: Jetty(8.y.z-SNAPSHOT)
But when I try something like (tried different combinations for headers, here I recreated same headers as python-tika client uses):
with tempfile.NamedTemporaryFile() as tmp_file:
download_file(url, tmp_file)
payload = open(tmp_file.name, 'rb')
headers = {
'Accept': 'application/json',
'Content-Disposition': 'attachment; filename={}'.format(
os.path.basename(tmp_file.name))}
response = requests.put(TIKA_ENDPOINT_URL + '/tika', payload,
headers=headers,
verify=False)
I've tried to use payload as well as fileUrl - with the same result of WARN javax.ws.rs.ClientErrorException: HTTP 406 Not Acceptable and java stack trace on the server. Full trace:
WARN javax.ws.rs.ClientErrorException: HTTP 406 Not Acceptable
at org.apache.cxf.jaxrs.utils.SpecExceptions.toHttpException(SpecExceptions.java:117)
at org.apache.cxf.jaxrs.utils.ExceptionUtils.toHttpException(ExceptionUtils.java:173)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:542)
at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:177)
at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.handleMessage(JAXRSInInterceptor.java:77)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:274)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:261)
at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:76)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1088)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1024)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:748)
I've also tried to compare ( with nc -l localhost 5677 | less) what is so different with two requests (payload abbreviated):
From curl:
PUT /tika HTTP/1.1
Host: localhost:5677
User-Agent: curl/7.47.0
Accept: */*
Content-Length: 418074
Expect: 100-continue
%PDF-1.4
%<D3><EB><E9><E1>
1 0 obj
<</Creator (Chromium)
From Python requests library:
PUT /tika HTTP/1.1
Host: localhost:5677
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.13.0
Content-type: application/pdf
Content-Length: 246176
%PDF-1.4
%<D3><EB><E9><E1>
1 0 obj
<</Creator (Chromium)
The question is, what is the correct way to call Tika server from Python?
I've also tried python tika library in client-only mode and using tika-app via jnius. With tika client, as well as using tika-app.jar with pyjnius, I only freezes (call never returns) when I use them in a celery worker. At the same, pyjnius / tika-app and tika-python script both work nicely in a script: I have not figured out what is wrong inside celery worker. I guess, something to do with threading and/or initialization in wrong place. But that is a topic for another question.
And here is what tika-python requests:
PUT /tika HTTP/1.1
Host: localhost:5677
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.13.0
Content-Disposition: attachment; filename=tmpb3YkTq
Content-Length: 183234
%PDF-1.4
%<D3><EB><E9><E1>
1 0 obj
<</Creator (Chromium)
And now it seems like this is some kind of a problem with tika server:
$ tika-python --verbose --server 'localhost' --port 5677 parse all /tmp/tmpb3YkTq
2018-04-08 09:44:11,555 [MainThread ] [INFO ] Writing ./tmpb3YkTq_meta.json
(<open file '<stderr>', mode 'w' at 0x7f0b688eb1e0>, 'Request headers: ', {'Accept': 'application/json', 'Content-Disposition': 'attachment; filename=tmpb3YkTq'})
(<open file '<stderr>', mode 'w' at 0x7f0b688eb1e0>, 'Response headers: ', {'Date': 'Sun, 08 Apr 2018 06:44:13 GMT', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/json', 'Server': 'Jetty(8.y.z-SNAPSHOT)'})
['./tmpb3YkTq_meta.json']
Cf:
$ tika-python --verbose --server 'localhost' --port 5677 parse text /tmp/tmpb3YkTq
2018-04-08 09:43:38,326 [MainThread ] [INFO ] Writing ./tmpb3YkTq_meta.json
(<open file '<stderr>', mode 'w' at 0x7fc3eee4a1e0>, 'Request headers: ', {'Accept': 'application/json', 'Content-Disposition': 'attachment; filename=tmpb3YkTq'})
(<open file '<stderr>', mode 'w' at 0x7fc3eee4a1e0>, 'Response headers: ', {'Date': 'Sun, 08 Apr 2018 06:43:38 GMT', 'Content-Length': '0', 'Server': 'Jetty(8.y.z-SNAPSHOT)'})
2018-04-08 09:43:38,409 [MainThread ] [WARNI] Tika server returned status: 406
['./tmpb3YkTq_meta.json']
I try to host some wordpress with an nginx proxy above:
This is my docker-compose.yml
proxy:
image: jwilder/nginx-proxy:0.4.0
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
# SSL Certificates
- /etc/letsencrypt/live/mysupersite.com/cert.pem:/etc/nginx/certs/site.com.crt:ro
- /etc/letsencrypt/live/mysupersite.com/privkey.pem:/etc/nginx/certs/site.com.key:ro
wordpress:
image: wordpress:4.8.1-php5.6-apache
expose:
- 80
volumes:
- /var/www/html:/var/www/html
restart: always
environment:
- WORDPRESS_DB_HOST=mysql
- WORDPRESS_DB_USER=dev
- WORDPRESS_DB_PASSWORD=passwd
- WORDPRESS_DB_NAME=db
- VIRTUAL_HOST=mysupersite.com,www.mysupersite.com
Everything seems to start well. I have 2 running containers.
But I'm not able to visit my website on https://mysupersite.com in my browser.
What part am I missing? Do I need extra configuration somewhere?
I see this error in my browser:
The page isn’t redirecting properly
output o cur -L -v https://www.mysupersite.com is the following:
* Connection #0 to host www.mysupersite.com left intact
* Issue another request to this URL: 'https://www.mysupersite.com/'
* Found bundle for host www.mysupersite.com: xx [can pipeline]
* Re-using existing connection! (#0) with host www.mysupersite.com
* Connected to www.mysupersite.com (IP) port 443 (#0)
> GET / HTTP/1.1
> Host: www.mysupersite.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.9.12
< Date: Tue, 22 Aug 2017 07:33:22 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Connection: keep-alive
< X-Powered-By: PHP/5.6.31
< Set-Cookie: PHPSESSID=xxx; path=/
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< X-Pingback: http://www.mysupersite.com/xmlrpc.php
< Location: https://www.mysupersite.com/
< Strict-Transport-Security: max-age=31536000
<
* Connection #0 to host www.mysupersite.com left intact
* Issue another request to this URL: 'https://www.mysupersite.com/'
* Found bundle for host www.mysupersite.com: xxxx [can pipeline]
* Re-using existing connection! (#0) with host www.mysupersite.com
* Connected to www.mysupersite.com (IP) port 443 (#0)
> GET / HTTP/1.1
> Host: www.mysupersite.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.9.12
< Date: Tue, 22 Aug 2017 07:33:23 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Connection: keep-alive
< X-Powered-By: PHP/5.6.31
< Set-Cookie: PHPSESSID=xxxx; path=/
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< X-Pingback: http://www.mysupersite.com/xmlrpc.php
< Location: https://www.mysupersite.com/
< Strict-Transport-Security: max-age=31536000
<
* Connection #0 to host www.mysupersite.com left intact
* Maximum (50) redirects followed
curl: (47) Maximum (50) redirects followed