I have defined a custom nginx log format using below template :
log_format main escape=none '"$time_local" client=$remote_addr '
'request="$request" '
'status=$status'
'Req_Header=$req_headers '
'request_body=$req_body '
'response_body=$resp_body '
'referer=$http_referer '
'user_agent="$http_user_agent" '
'upstream_addr=$upstream_addr '
'upstream_status=$upstream_status '
'request_time=$request_time '
'upstream_response_time=$upstream_response_time '
'upstream_connect_time=$upstream_connect_time ';
In return i get the request logged like below
"09/Sep/2019:13:28:39 +0530" client=59.152.52.190 request="POST /api/onboard/checkExistence HTTP/1.1"status=200Req_Header=Headers: accept: application/json
host: uat-pwa.abc.com
from: https://uat-pwa.abc.com/onboard/mf/onboard-info_v1.2.15.3
sec-fetch-site: same-origin
accept-language: en-GB,en-US;q=0.9,en;q=0.8
content-type: application/json
connection: keep-alive
content-length: 46
cookie: _ga=GA1.2.51303468.1558948708; _gid=GA1.2.1607663960.1568015582; _gat_UA-144276655-2=1
referer: https://uat-pwa.abc.com/onboard/mf/onboard-info
accept-encoding: gzip, deflate, br
ticket: aW52ZXN0aWNh
businessunit: MF
sec-fetch-mode: cors
userid: Onboarding
origin: https://uat-pwa.abc.com
investorid:
user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36
request_body={"PAN":"ABCDEGH","mobile":null,"both":"no"} response_body={"timestamp":"2019-09-09T13:28:39.132+0530","message":"Client Already Exist. ","details":"Details are in Logger database","payLoad":null,"errorCode":"0050","userId":"Onboarding","investorId":"","sessionUUID":"a2161b89-d2d7-11e9-aa73-3dba15bc0e1c"} referer=https://uat-pwa.abc.com/onboard/mf/onboard-info user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" upstream_addr=[::1]:8080 upstream_status=200 request_time=0.069 upstream_response_time=0.068 upstream_connect_time=0.000
I am facing issues in writing a parser rule in telegraf logparser section. Once this data is parsed properly, Telegraf can write it into influx DB.
I have tried various solutions online to find a parsing rule but not able to do so as I am new to it. Any assistance will be appreciated.
Thanks and do let me know if any further information is required.
Related
I am trying to scrape the following page:
https://apps.fcc.gov/oetcf/tcb/reports/Tcb731GrantForm.cfm?mode=COPY&RequestTimeout=500&tcb_code=&application_id=ll686%2BwlPnFzHQb6tru2vw%3D%3D&fcc_id=QDS-BRCM1095
headers_initial = {
'user-agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Mobile Safari/537.36',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'accept-language': 'en-US,en;q=0.9,de;q=0.8',
'cache-control': 'max-age=0',
'upgrade-insecure-requests': '1',
}
r = requests.get(url, timeout=100, headers=headers_initial)
print(r.status_code)
print(r.headers)
print(r.text)
my status code is 400
my requests.get gets hung up. I would be very appreciative of any help someone can provide.
I'm trying to do a simple get request but no matter how I'm configuring the headers I keep getting a 403 response. The page loads fine in a browser. No login is required and there are no tracked cookies either. The link I'm trying to get a response from is below, followed by my simple code.
https://i7.sportsdatabase.com/nba/query.json?sdql=50+%3C+Kobe+Bryant%3Apoints+and+site%3Daway&sport=nba
url = 'https://i7.sportsdatabase.com/nba/query.json?sdql=50+%3C+Kobe+Bryant%3Apoints+and+site%3Daway&sport=nba'
headers = {
'Host': 'i7.sportsdatabase.com',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36',
}
r = requests.get(url, headers)
I'm not seeing any other headers that need adding to the request. The full, in browser, request headers are below:
Host: i7.sportsdatabase.com
Connection: keep-alive
Cache-Control: max-age=0
sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"
sec-ch-ua-mobile: ?0
DNT: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: en,en-US;q=0.9,it;q=0.8,es;q=0.7
If-None-Match: "be833f0fb26eb81487fc09e05c85ac8c8646fc7b"
Try:
Make your URL a string
Add the accepts
This works:
url = 'https://i7.sportsdatabase.com/nba/query.json?sdql=50+%3C+Kobe+Bryant%3Apoints+and+site%3Daway&sport=nba'
headers = {
'Host': 'i7.sportsdatabase.com',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'Accept-Encoding': 'gzip, deflate, br',
}
r = requests.get(url, headers=headers)
Try using .Session()
import requests
s = requests.Session()
headers = {
'User-Agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Mobile Safari/537.36',
}
s.get('https://i7.sportsdatabase.com/nba/trends', headers=headers)
url = 'https://i7.sportsdatabase.com/nba/query.json?sdql=50+%3C+Kobe+Bryant%3Apoints+and+site%3Daway&sport=nba'
r = s.get(url, headers=headers)
print(r)
Output:
print(r)
<Response [200]>
I am trying to login on this website using the following snippet of code:
from bs4 import BeautifulSoup as bs
import requests
URL='https://app.acvauctions.com/login/'
HEADERS = { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36' ,
'origin': 'https://app.acvauctions.com'
'referer': 'https://app.acvauctions.com/'
}
s = requests.session()
login_payload = { 'email': "****", 'password': "****", 'web': 'true' }
login_req = s.post(URL, headers=HEADERS, data=login_payload, allow_redirects=True)
The header requests that I get when I login with browser is the following:
accept: application/json, text/plain, */*
accept-encoding: gzip, deflate, br
accept-language: en-GB,en;q=0.9,fa-IR;q=0.8,fa;q=0.7,en-US;q=0.6
content-length: 67
content-type: application/json
newrelic: eyJ2IjpbMCwxXSwiZCI6eyJ0eSI6IkJyb3dzZXIiLCJhYyI6IjE2NTA3NDUiLCJhcCI6IjE3ODI2MTk5MiIsImlkIjoiMGIzZjc3MmQxNGI0MWI5YSIsInRyIjoiMTc0MGVlNDA4NTE0MzA1YTBkNWU4NTJkODRlZTMxNzAiLCJ0aSI6MTYyNzUxODY1MDA0N319
origin: https://app.acvauctions.com
referer: https://app.acvauctions.com/
sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"
sec-ch-ua-mobile: ?0
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-site
traceparent: 00-592fab3e40957cc0ffa37c47c8914000-c656f42dbf8336a1-01
tracestate: 1650745#nr=0-1-1650745-178261992-0b3f772d14b41b9a----1627518650047
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36
newrelic, traceparent, and tracestate change every time I login but I do not know how to handle them?
I noticed a bug in an old version of firefox, that was shipped with my os.
These were the sympthoms:
Guile web server failed to serve the request when data was reposted.
I came up with a minimal example to show the problem.
Steps to reproduce:
start the server script
load localhost:8080
select the test.csv file for upload, and upload it
hit the refresh button in the browser
answer yes to resend post data dialog.
test.scm:
(use-modules (web server)
(rnrs bytevectors))
(define (handler request body)
(if body
(display (utf8->string body)))
(values '((content-type . (text/html)))
(string-append "<html><body>"
"<form action=\"do\" method=\"POST\" enctype=\"multipart/form-dat\
a\">"
"<input type=\"file\" name=\"x\">"
"<input type=\"submit\">")))
(run-server handler)
test.csv:
a,b
Expected result: no error displayed on the console.
Actual result:
-----------------------------18912432064747206221264673165
Content-Disposition: form-data; name="x"; filename="test.csv"
Content-Type: text/csv
In ice-9/boot-9.scm:
841:4 4 (with-throw-handler _ _ _)
In web/server/http.scm:
127:28 3 (_)
In web/request.scm:
205:31 2 (read-request #<closed: file 5559bbcb82a0> _)
In web/http.scm:
1141:6 1 (read-request-line _)
In ice-9/boot-9.scm:
752:25 0 (dispatch-exception _ _ _)
Bad request: Bad Request-Line: "a,b"
What am I doing wrong here?
Some additional information:
on a whireshark capture it turns out, that the following is sent on resend:
POST /do HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: hu-HU,hu;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://localhost:8080/do
Content-Type: multipart/form-data; boundary=---------------------------121791188820701943592108452984
Content-Length: 150
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Cache-Control: max-age=0
-----------------------------121791188820701943592108452984
Content-Disposition: form-data; name="x"; filename="test.csv"
Content-Type: text/csv
POST /do HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: hu-HU,hu;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://localhost:8080/do
Content-Type: multipart/form-data; boundary=---------------------------121791188820701943592108452984
Content-Length: 150
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Cache-Control: max-age=0
-----------------------------121791188820701943592108452984
Content-Disposition: form-data; name="x"; filename="test.csv"
Content-Type: text/csv
a,b
-----------------------------121791188820701943592108452984--
I will check the http spec if it has anything to say about this. The first http request is partial, followed by a well formed request.
UPDATE:
It turned out that guile webserver threw the error completely legitimately.
Answering my own question:
This is actually this firfox bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1434553
Fixed in firefox62.
I have a rest service and I'm trying to call it through jquery client. The backend rest service works find when I send a response like below through CURL.
curl -H "Accept: text/plain" -X POST -d 'host=<type:hostDescription xmlns:type="http://schemas.airavata.apache.org/gfac/type"><type:hostName>testHost5</type:hostName><type:hostAddress>aaaa</type:hostAddress></type:hostDescription>' http://localhost:6060/airavata-registry-rest-services/registry/api/hostdescriptor/save
This request looks like following.
POST /airavata-registry-rest-services/registry/api/hostdescriptor/save HTTP/1.1
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
Host: 127.0.0.1
Accept: text/plain
Content-Length: 189
Content-Type: application/x-www-form-urlencoded
host=<type:hostDescription xmlns:type="http://schemas.airavata.apache.org/gfac/type">
<type:hostName>testHost5</type:hostName>
<type:hostAddress>aaaa</type:hostAddress></type:hostDescription>
But the issue is that the request sent by my jquery client looks like below.
POST /airavata-registry-rest-services/registry/api/hostdescriptor/save HTTP/1.1
Host: 127.0.0.1
Connection: keep-alive
Content-Length: 276
Origin: http://localhost:7080
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.11 (KHTML, like Gecko) Ubuntu/12.04 Chromium/20.0.1132.47 Chrome/20.0.1132.47 Safari/536.11
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Accept: */*
Referer: http://localhost:7080/client-api-demo/x_host_descriptor_save.html
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
host=%3Ctype%3Ahostdescription+xmlns%3Atype%3D%22http%3A%2F%2Fschemas.airavata.apache.org%2Fgfac%2Ftype%22%3E%3Ctype%3Ahostname%3Eutav%3C%2Ftype%3Ahostname%3E%3Ctype%3Ahostaddress%3Egatekeeper2.ranger.tacc.teragrid.org%3C%2Ftype%3Ahostaddress%3E%3C%2Ftype%3Ahostdescription%3E
My jquery code snippet looks like following;
$('[name="btn2"]').click(function(){
var hostName = $("#hostName1").val();
var hostAddress = $("#hostAddress1").val();
var xml = $('<type:hostDescription xmlns:type="http://schemas.airavata.apache.org/gfac/type"><type:hostName>' + hostName + '</type:hostName><type:hostAddress>' + hostAddress + '</type:hostAddress></type:hostDescription>');
var xmlData= $(xml);
var xmlString;
if (window.ActiveXObject){
xmlString = xmlData.xml;
} else {
var oSerializer = new XMLSerializer();
xmlString = oSerializer.serializeToString(xmlData[0]);
}
console.log(xmlString);
$.post("http://localhost:6060/airavata-registry-rest-services/registry/api/hostdescriptor/save",
{ host: xmlString} ,function(data,status){
alert("Data: " + data + "\nStatus: " + status);
alert("button2 post sent !");
});
});
How should I change my client to invoke this service properly.