How to Remove Data from WC API Endpoint Request? - wordpress

I'm shipping off order data to a 3rd party piece of fulfillment software. They integrate by default with the WooCommerce REST API. However some recent changes to my site and order data have added additional order meta. Now when grabbing the same amount of orders as it always has the request times out with a 504. The request is now unreasonably large, to fix this I've decided to optimize by reducing the irrelevant and unnecessary data produced by the request. Also I have to be able to process 100 at a time I cannot reduce the filter limit, its automatically set by the 3rd party application.
Endpoint in Question
wc-api/v2/orders?status=processing&page=1&filter%5Blimit%5D=100
This endpoint grabs the first 100 orders in processing and displays them as a piece of JSON.
Things to Remove
customer_user_agent
avatar_url
cogs_cost
cogs_total_cost
Example Response
{
"orders":[
{
"id":137314,
"order_number":"137314",
"created_at":"2019-09-18T18:37:06Z",
"updated_at":"2019-09-18T18:37:07Z",
"completed_at":"1970-01-01T00:00:00Z",
"status":"processing",
"currency":"USD",
"total":"49.50",
"subtotal":"55.00",
"total_line_items_quantity":1,
"total_tax":"0.00",
"total_shipping":"0.00",
"cart_tax":"0.00",
"shipping_tax":"0.00",
"total_discount":"0.00",
"shipping_methods":"Free shipping",
"payment_details":{
"method_id":"nmipay",
"method_title":"Pay with Credit Card",
"paid":true
},
"billing_address":{
"first_name":"XXX",
"last_name":"XXXX",
"company":"",
"address_1":"XXXX",
"address_2":"",
"city":"XXXX",
"state":"XX",
"postcode":"XXXXX",
"country":"US",
"email":"XXXXXX",
"phone":"XXXX"
},
"shipping_address":{
"first_name":"XXX",
"last_name":"XX",
"company":"",
"address_1":"XXXXX",
"address_2":"",
"city":"XXX",
"state":"XXX",
"postcode":"XXX",
"country":"XXXX"
},
"note":"",
"customer_ip":"98.216.25.236",
"customer_user_agent":"mozilla\/5.0 (iphone; cpu iphone os 12_4_1 like mac os x) applewebkit\/605.1.15 (khtml, like gecko) version\/12.1.2 mobile\/15e148 safari\/604.1",
"customer_id":127116,
"view_order_url":"XXXXX",
"line_items":[
{
"id":198261,
"subtotal":"55.00",
"subtotal_tax":"0.00",
"total":"55.00",
"total_tax":"0.00",
"price":"55.00",
"quantity":1,
"tax_class":"",
"name":"Core Hoodie - Black, Large",
"product_id":351,
"sku":"ss-hoodie-core-zip-blk-lg",
"meta":[
],
"bundled_by":"",
"bundled_item_title":"",
"bundled_items":[
],
"cogs_cost":"23.15",
"cogs_total_cost":"23.15"
}
],
"shipping_lines":[
{
"id":198263,
"method_id":"free_shipping",
"method_title":"Free shipping",
"total":"0.00"
}
],
"tax_lines":[
],
"fee_lines":[
{
"id":198262,
"title":"VIP Discount",
"tax_class":"0",
"total":"-5.50",
"total_tax":"0.00"
}
],
"coupon_lines":[
],
"cogs_total_cost":"23.15"
}
]
}
This is the furthest i've gotten
I found the following hooks but cannot get anything to trigger.
woocommerce_rest_prepare_shop_order_object
woocommerce_rest_prepare_shop_order
function remove_user_agent_from_rest_api( $response, $object, $request ) {
unset($response->data['customer_user_agent']);
return $response;
}
function test_rest_api() {
add_filter( "woocommerce_rest_pre_insert_shop_order", "remove_user_agent_from_rest_api", 10, 2 );
add_filter( "woocommerce_rest_pre_insert_shop_order_object", "remove_user_agent_from_rest_api", 10, 2 );
}
add_action( 'rest_api_init', 'test_rest_api', 0 );
Is this a performance tuning issue?
Here is a sample trace from new relic & a sample from my NGINX Error Log. What could I tune to keep the server open long enough to process this request.
2019/10/02 10:59:25 [error] 10270#10270: *5 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: XXX, server: X.net, request: "GET /?km_source=blog HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "X.net", referrer: "https://www.X.net/"
2019/10/02 11:00:42 [error] 10270#10270: *34 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XXX, server: XXX.net, request: "GET /wc-api/v2/orders?status=processing&page=10&filter%5Blimit%5D=100&consumer_key=ck_XXX&consumer_secret=cs_XXX HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock", host: "X.net"
2019/10/02 11:07:53 [error] 13021#13021: *62 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XXX, server: XXX.net, request: "GET /wc-api/v2/orders?status=processing&page=1&filter%5Blimit%5D=100&consumer_key=ck_XXX&consumer_secret=cs_XXX HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock", host: "X.net"
2019/10/02 11:13:45 [error] 15270#15270: *66 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XXX, server: XXX.net, request: "GET /wc-api/v2/orders?status=processing&page=1&filter%5Blimit%5D=100&consumer_key=ck_XXX&consumer_secret=cs_XXX HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock", host: "XXX.net"
2019/10/02 11:15:44 [error] 16010#16010: *79 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XXX, server: X.net, request: "GET /wc-api/v2/orders?status=processing&page=1&filter%5Blimit%5D=100&consumer_key=ck_XXX&consumer_secret=cs_XXX HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock", host: "X.net"

The first issue I'd notice it that your filters are only passing 2 variables, when they should be passing 3.
add_filter( "woocommerce_rest_pre_insert_shop_order", "remove_user_agent_from_rest_api", 10, 3 );
Should do it.

Related

Google Analytics API call frequently returns expected data but occasionally the same call returns error 500

we have been running Google Analytics(GA) API for several accounts .
It works well for all the accounts except for one which occasionally(not always) fails and returns the error 500.
Regarding the account with the problem, the error 500 started to happen since last year(2020/DEC).
The occurrence of this error(500) shows a tendency to happen when the API CALL is made to fetch reports for periods over 3 days (However the chance of this error for periods shorter than 3 days so far has not been zero.)
Regarding the explanation above↑ the problem seems to be in GA side.
May we ask your assistance to check up the situation and help us to solve the problem.
We deeply appreciate all your help and kindness.
Below, As an example I paste the returned response(error 500) for the API call we made in 2021/JULY/8.
Best regards,
Shahin
I, [2021-07-09T18:11:44.314945 #13000] INFO -- : get https://analytics.googleapis.com/analytics/v3/data/ga?dimensions=ga%3Adate%2Cga%3AadDistributionNetwork%2Cga%3AsourceMedium%2Cga%3AadKeywordMatchType%2Cga%3Akeyword%2Cga%3AadContent%2Cga%3Acampaign&end-date=2021-07-08&ids=ga%3A61390358&max-results=1000&metrics=ga%3Agoal11Completions%2Cga%3Agoal12Completions%2Cga%3Agoal13Completions%2Cga%3Agoal14Completions%2Cga%3Agoal15Completions%2Cga%3Agoal16Completions%2Cga%3Agoal17Completions%2Cga%3Agoal18Completions%2Cga%3Agoal19Completions%2Cga%3Agoal20Completions&samplingLevel=HIGHER_PRECISION&start-date=2021-07-04&start-index=2001
D, [2021-07-09T18:11:48.813256 #13000] DEBUG -- request: User-Agent: "analytics/v3 google-api-ruby-client/0.8.6 Linux/2.6.32-431.el6.x86_64\n (gzip)"
Accept-Encoding: "gzip"
Content-Type: ""
Authorization: "Bearer ya29.a0ARrdaM9m7nl7eZ40UtyNPMLo-quMyknO5gC6d60ucF2zhN3WgO_YcsHtldGyqpjuTsHUJAFurvcC3cryutiB2U2P_Wf-gM6DtnhkAqEjoPfCKwsWeuz8wrFgXfa7M3QknRcFDFrWO0CNrEKeDG7caf0B1R49Rws"
Cache-Control: "no-store"
D, [2021-07-09T18:11:48.813315 #13000] DEBUG -- request:
I, [2021-07-09T18:12:48.790970 #13000] INFO -- Status: 500
D, [2021-07-09T18:12:48.791044 #13000] DEBUG -- response: vary: "Origin, X-Origin, Referer"
content-type: "application/json; charset=UTF-8"
date: "Fri, 09 Jul 2021 09:12:48 GMT"
server: "ESF"
cache-control: "private"
x-xss-protection: "0"
x-frame-options: "SAMEORIGIN"
x-content-type-options: "nosniff"
alt-svc: "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""
connection: "close"
transfer-encoding: "chunked"
D, [2021-07-09T18:12:48.791065 #13000] DEBUG -- response: {
"error": {
"code": 500,
"message": "There was an internal error.",
"errors": [
{
"message": "There was an internal error.",
"domain": "global",
"reason": "internalError"
}
]
}
}

nginx blocking request till current request finishes

Boiling my question down to the simplest possible: I have a simple Flask webserver that has a GET handler like this:
#app.route('/', methods=['GET'])
def get_handler():
t = os.environ.get("SLOW_APP")
app_type = "Fast"
if t == "1":
app_type = "Slow"
time.sleep(20)
return "Hello from Flask, app type = %s" % app_type
I am running this app on two different ports: one without the SLOW_APP environment variable set on port 8000 and the other with the SLOW_APP environment variable set on port 8001.
Next I have an nginx reverse proxy that has these two appserver instances in its upstream. I am running everything using docker so my nginx conf looks like this:
upstream myproject {
server host.docker.internal:8000;
server host.docker.internal:8001;
}
server {
listen 8888;
#server_name www.domain.com;
location / {
proxy_pass http://myproject;
}
}
It works except that if I open two browser windows and type localhost, it first hits the slow server where it takes 20 seconds and during this time the second browser appears to block waiting for the first request to finish. Eventually I see that the first request was serviced by the "slow" server and the second by the "fast" server (no time.sleep()). Why does nginx appear to block the second request till the first one finishes?
No, if the first request goes to the slow server (where it takes 20 sec) and during that delay if I make a request again from the browser it goes to the second server but only after the first is finished.
I have worked with our Engineering Team on this and can share the following insights:
Our Lab environment
Lua
load_module modules/ngx_http_lua_module-debug.so;
...
upstream app {
server 127.0.0.1:1234;
server 127.0.0.1:2345;
}
server {
listen 1234;
location / {
content_by_lua_block {
ngx.log(ngx.WARN, "accepted by fast")
ngx.say("accepted by fast")
}
}
}
server {
listen 2345;
location / {
content_by_lua_block {
ngx.log(ngx.WARN, "accepted by slow")
ngx.say("accepted by slow")
ngx.sleep(5);
}
}
}
server {
listen 80;
location / {
proxy_pass http://app;
}
}
This is the same setup as it would be with another 3rd party application we are proxying traffic to. But I have tested the same with an NGINX configuration shared in your question and two NodeJS based applications as upstream.
NodeJS
Normal
const express = require('express');
const app = express();
const port = 3001;
app.get ('/', (req,res) => {
res.send('Hello World')
});
app.listen(port, () => {
console.log(`Example app listening on ${port}`)
})
Slow
const express = require('express');
const app = express();
const port = 3002;
app.get ('/', (req,res) => {
setTimeout( () => {
res.send('Hello World')
}, 5000);
});
app.listen(port, () => {
console.log(`Example app listening on ${port}`)
})
The Test
As we are using NGINX OSS the Loadbalancing protocol will be RoundRobin (RR). Our first test from another server using ap. The Result:
Concurrency Level: 10
Time taken for tests: 25.056 seconds
Complete requests: 100
Failed requests: 0
Total transferred: 17400 bytes
HTML transferred: 1700 bytes
Requests per second: 3.99 [#/sec] (mean)
Time per request: 2505.585 [ms] (mean)
Time per request: 250.559 [ms] (mean, across all concurrent requests)
Transfer rate: 0.68 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.7 0 5
Processing: 0 2505 2514.3 5001 5012
Waiting: 0 2504 2514.3 5001 5012
Total: 1 2505 2514.3 5001 5012
Percentage of the requests served within a certain time (ms)
50% 5001
66% 5005
75% 5007
80% 5007
90% 5010
95% 5011
98% 5012
99% 5012
100% 5012 (longest request)
50% of all requests are slow. Thats totally okay because we have one "slow" instance. The same test with curl. Same result. Based on the debug-log of the NGINX server we saw that the request were processed as they came in and were sent to either the slow or the fast backend (based on roundrobin).
2021/04/08 15:26:18 [debug] 8995#8995: *1 get rr peer, try: 2
2021/04/08 15:26:18 [debug] 8995#8995: *1 get rr peer, current: 000055B815BD4388 -100
2021/04/08 15:26:18 [debug] 8995#8995: *4 get rr peer, try: 2
2021/04/08 15:26:18 [debug] 8995#8995: *4 get rr peer, current: 000055B815BD4540 0
2021/04/08 15:26:18 [debug] 8995#8995: *5 get rr peer, try: 2
2021/04/08 15:26:18 [debug] 8995#8995: *5 get rr peer, current: 000055B815BD4388 -100
2021/04/08 15:26:18 [debug] 8995#8995: *7 get rr peer, try: 2
2021/04/08 15:26:18 [debug] 8995#8995: *7 get rr peer, current: 000055B815BD4540 0
2021/04/08 15:26:18 [debug] 8995#8995: *10 get rr peer, try: 2
2021/04/08 15:26:18 [debug] 8995#8995: *10 get rr peer, current: 000055B815BD4388 -100
2021/04/08 15:26:18 [debug] 8995#8995: *13 get rr peer, try: 2
2021/04/08 15:26:18 [debug] 8995#8995: *13 get rr peer, current: 000055B815BD4540 0
2021/04/08 15:26:18 [debug] 8995#8995: *16 get rr peer, try: 2
2021/04/08 15:26:18 [debug] 8995#8995: *16 get rr peer, current: 000055B815BD4388 -100
2021/04/08 15:26:18 [debug] 8995#8995: *19 get rr peer, try: 2
2021/04/08 15:26:18 [debug] 8995#8995: *19 get rr peer, current: 000055B815BD4540 0
So given that means the behaviour of "nginx blocking request till current request finishes" is not reproducible on the instance. But I was able to reproduce your issue in the Chrome Browser. Hitting the slow instance will let the other browser window waiting till the first one gets its response. After some memory analysis and debugging on the client side I came across the connection pool of the browser.
https://www.chromium.org/developers/design-documents/network-stack
The Browser makes use of the same, already established connection to the Server. In case this connection is occupied with the waiting request (Same data, same cookies...) it will not open a new connection from the pool. It will wait for the first request to finish. You can work around this by adding a cache-buster or a new header, new cookie to the request. something like:
http://10.172.1.120:8080/?ofdfu9aisdhffadf. Send this in a new browser window while you are waiting in the other one for the response. This will show an immediate response (given there was no other request to the backend because based on RR -> IF there was a request to the slow one the next one will be the fast one).
Same applies if you send request from different clients. This will work as well.

Logstash pattern for nginx error log

This is my sample error log:
2017/03/29 17:32:56 [error] 21924#21924: *212595 access forbidden by rule, client: 172.31.0.14, server: , request: "POST /app/etc/local.xml HTTP/1.1", host: "www.overcart.com"
I want a grok pattern that matches this. I don't know how to proceed. Don't know how to create one. I have tried various but none of them worked.
I am currently parsing this using the following grok pattern:
%{DATESTAMP:mydate} [%{DATA:severity}] (%{NUMBER:pid:int}#%{NUMBER}: *%{NUMBER}|*%{NUMBER}) %{GREEDYDATA:mymessage}(?:, client: (?<client_ip>%{IP}|%{HOSTNAME})) (?:, server: %{IPORHOST:server})(?:, request: %{QS:request})?(?:, host: %{QS:host})?(?:, referrer: \"%{URI:referrer})
but it's not parsing from (?:, server: onwards.
Here's the grok pattern I use. It's not perfect and can be improved, but it works and parses also an additional upstream token.
(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{GREEDYDATA:errormessage}, client: %{IP:client}, server: %{GREEDYDATA:server}, request: "(?<httprequest>%{WORD:httpcommand} %{UNIXPATH:httpfile} HTTP/(?<httpversion>[0-9.]*))"(, )?(upstream: "(?<upstream>[^,]*)")?(, )?(host: "(?<host>[^,]*)")?
this is what I use for nginx-error pattern, it can parse all fields in my error log.:
(?<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[%{DATA:err_severity}\] (%{NUMBER:pid:int}#%{NUMBER}: \*%{NUMBER}|\*%{NUMBER}) %{DATA:err_message}(?:, client: (?<clientip>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server})(?:, request: "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}")?(?:, upstream: "%{DATA:upstream}")?(?:, host: "%{IPORHOST:host}")?(?:, referrer: "%{URI:referrer}”)?
I'm using #mavlarn's grok pattern and it works well for my use case. I'm sure #dr01 will work as well. Remember to test to see what works for you!
Pro-tip: Because there are double quotes inside these grok patterns, in order to make them work properly I needed to surround the pattern with single quotes instead of double.
To highlight, here is an example.
grok {
match => { "message" => "This "double quote" pattern won't work. Backslashes to cancel/close the regex won't work either. Logstash will fail to start.."}
}
grok {
match => { "message" => 'But this "double quote" pattern will work when surrounding with single quotes instead.'}
}
Example solution.
grok {
match => { "message" => '(?<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[%{DATA:err_severity}\] (%{NUMBER:pid:int}#%{NUMBER}: \*%{NUMBER}|\*%{NUMBER}) %{DATA:err_message}(?:, client: (?<clientip>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server})(?:, request: "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}")?(?:, upstream: "%{DATA:upstream}")?(?:, host: "%{IPORHOST:host}")?(?:, referrer: "%{URI:referrer}”)?' }
}
(Tested with Logstash v6.x and v7.x)

NGINX configuration file for google-bot - rewrite

I have ajax site and static version of it in folder:
/_escaped_fragment_/
My config, doesn't working:
# nginx configuration
location / {
if ($query_string ~ "^_escaped_fragment_=$"){
rewrite ^(.*)$ /_escaped_fragment_$1/index.html? break;
}
if ($query_string ~ "^_escaped_fragment_=(.*)$"){
rewrite ^(.*)$ /_escaped_fragment_$1/%1/index.html? break;
}
}
Google bot request:
"GET /?_escaped_fragment_=/page/nerjav HTTP/1.1" 404 232 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
I just need to turn:
/#!/page/nerjav -> /_escaped_fragment_/page/nerjav/index.html
Error log:
2014/04/24 16:43:14 [error] 6087#0: *1 rewrite or internal redirection cycle while processing "/_escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment_//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html", client: 66.249.66.194, server: m-chel.ru, request: "GET /?_escaped_fragment_=/page/nerjav HTTP/1.1", host: "site.com"
2014/04/24 16:43:56 [error] 6087#0: *2 rewrite or internal redirection cycle while processing "/_escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment__escaped_fragment_//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html//page/nerjav/index.html", client: 109.191.139.106, server: m-chel.ru, request: "GET /?_escaped_fragment_=/page/nerjav HTTP/1.1", host: "site.com"
I'm not sure how your thing works, but it looks like you're missing an =.
# nginx configuration
location / {
if ($query_string ~ "^_escaped_fragment_=$"){
rewrite ^(.*)$ /_escaped_fragment_=$1/index.html? break;
}
if ($query_string ~ "^_escaped_fragment_=(.*)$"){
rewrite ^(.*)$ /_escaped_fragment_=$1/%1/index.html? break;
}
}

How can I disable nginx logging with limit_conn and limit_req?

How can I completely disable logging from HttpLimitConnModule and HttpLimitReqModule?
At least limit the damage done from extensive logging in case of a DOS-attack. I still want some error-logging but not when the request is denied.
Such messages:
2013/07/12 20:20:10 [error] 31544#0: *78 limiting requests, excess: 0.519 by zone "limit", client: *.*.*.*, server: example.com, request: "GET /static.html HTTP/1.1", host: "example.com", referrer: ""
One solution is to enable error_log just where it is needed.
server {
error_log /dev/null crit;
location ~\.php$ {
error_log /var/log/nginx_error.log;
}
}

Resources