I want to fetch a webpage and get the result in a string, but I don't know how to do it. I search online and didn't find how to do it.
I'd simply use Lua Socket which comes with an http submodule. You can simply use http.request to get a webpage into whatever container you'd want (default is string, but you can use a table, a file, stdio, ... using ltn12 filters and sinks).
As an example:
local http=require'socket.http'
local body, statusCode, headers, statusText = http.request('http://w3.impa.br/~diego/software/luasocket/http.html')
print('statusCode ', statusCode)
print('statusText ', statusText)
print('headers ')
for index,value in pairs(headers) do
print("\t",index, value)
end
print('body',body)
If you can't find an exact http client library, you could implement on yourself, or build on someone else's work.
In that link, although it is called libhttpd, but the author clearly states that it can be used for anything. Looks like a more usable wrapper around lua sockets.
if you don't have socket (like me), but you have the http library/module, then you could try this:
http.get("https://nodemcu.readthedocs.io/en/master/en/modules/http/", nil, function(code, data)
if (code ~= 200) then
print("HTTP request failed")
else
print(code, data)
end
end)
it works for me
you can find more info in the docs https://nodemcu.readthedocs.io/en/master/en/modules/http/#httpget
Related
I'm trying to get live trading data from the Internet via HTTP, but it is updated continuously, so if I GET the data, it will keep downloading as long as there is data available. Until I stop the downloading stream, then I can access the data.
How to access the stream of data while the downloading is in progress?
I tried using Indy's TIdHTTP, so I can use SSL, but I tried the IdIOHandlerStream, but it was already used for IdSSLIOHandlerSocketOpenSSL. So I'm absolutely clueless here.
This is in response to a "multipart/form-data" request.
Please guide me...
Lrequest.Values['__RequestVerificationToken'] := RequestVerificationToken;
Lrequest.Values['acct'] := 'demo';
Lrequest.Values['pwd'] := 'demo';
try
Response.Text := Onhttp.Post('https://trading/data', Lrequest);
Form1.Memo1.Lines.Add(TimeToStr(Time) + ': ' + Response.Text);
except
on E: Exception do
Form1.Memo1.Lines.Add(TimeToStr(Time) + ': ' + E.ClassName +
' error raised, with message : ' + E.Message);
end;
UPDATE:
The data is an endless JSON string, like this:
{"id":"data","val":[{"rc":2,"tpc":"\\RealTime\\Global\\SGDIDR.FX","item":[{"val":{"F009":"10454.90","F011":"-33.1"}}]}]}
{"id":"data","val":[{"rc":2,"tpc":"\\RealTime\\Global\\SGDIDR.FX","item":[{"val":{"F009":"10458.80","F011":"-29.2"}}]}]}
and so on, and so on...
You can't use TIdIOHandlerStream to interface with a TCP connection, that is not what it is designed for. It is meant for performing I/O operations using user-provided TStream objects, ie for debugging previously captured sessions.
TIdHTTP is not really designed to handle endless HTTP responses in most cases, as you have described. What is the exact format that the server is delivering its live data as? What do the HTTP response headers look like? It is really difficult to answer your question without know the exact format being used.
However, that being said, there are some cases to consider, depending on what the server is actually sending:
if the server is using a MIME-based server-push format, like multipart/x-mixed-replace, you can enable the hoNoReadMultipartMIME flag in the TIdHTTP.HTTPOptions property, and then read the MIME data yourself from the TIdHTTP.IOHandler after TIdHTTP.Get() exits. For instance, you can use TIdMessageDecoderMIME to help you parse the MIME parts, see New TIdHTTP hoNoReadMultipartMIME flag in Indy's blog, or Delphi Indy TIdHttp and multipart/x-mixed-replace with Text and jpeg image.
Otherwise, if the server is using Transfer-Encoding: chunked, where each data update is sent as a new HTTP chunk, you can use the TIdHTTP.OnChunkReceived event. Or, you can enable the hoNoReadChunked flag in the TIdHTTP.HTTPOptions property, and then read the chunks yourself from the TIdHTTP.IOHandler after TIdHTTP.Get() exits. See New TIdHTTP flags and OnChunkReceived event in Indy's blog.
Otherwise, you could give TIdHTTP.Get() a TIdEventStream to write into, and then use that stream's OnWrite event to access the raw bytes. Or, you could write your own TStream-derived class that overrides the virtual Write() method. Either way, you would be responsible for manually parsing and buffering the raw body data as they are being written to the stream.
Otherwise, you may have to resort to using TIdTCPClient instead, implementing the HTTP protocol manually, then you would be solely responsible for reading in the HTTP response body however you want.
I am trying to get a payload from Github Webhook with lua code.
I am able to get and see all the headers in the request but when I ask for body_data(), the content is empty, although it is not.
Maybe I should call something else than body_data() to get the payload but I couldn't find anything else online.
This is my Lua code:
local method = ngx.var.request_method
local headers = ngx.req.get_headers()
if method == "POST" then
ngx.req.read_body()
local body_data = ngx.req.get_body_data()
ngx.print(body_data)
local sha = headers['X-Hub-Signature-256']
end
It prints out 'nil'. The webhook payload is about 100 lines file. So why is it happening?
If I print the 'sha' variable, I do see the result. I don't see any result only for the body..
Please consult, thanks.
Here is how I am sending the metadata from server.
def DoSomething(self, request, context):
response = detection2g_pb2.SomeResponse()
response.message = 'done'
_SERVER_TRAILING_METADATA = (
('method_status', '1010'),
('error', 'No Error')
)
context.set_trailing_metadata(_SERVER_TRAILING_METADATA)
return response
Here is what I tried:
res = _stub.DoSomething(req)
print (res.trailing_metadata())
In this case I get Attribute Error object has no attribute 'trailing_metadata'. I want to know way to access the trailing metadata in the client side.
I apologize that we don't yet have an example illustrating metadata but you can see here how getting the trailing metadata on the invocation side requires using with_call (or future, but that may change the control flow in a way that you don't want changed, so I think that with_call should be your first choice). I think your invocation-side code should look like
response, call = _stub.DoSomething.with_call(request)
print(call.trailing_metadata())
.
i', scripting mitm proxy (http://mitmproxy.org/index.html) to write HTTP and HTTPS request and responses to a file according to their IP (each client can then access it's own requests\responses) for unit tests for mobile.
As far as i can see for now i can't just use str(Flow.request) or repr(Flow.request) to get a "raw" print of the response\request like i get in fiddler, i need to reconstruct it from the internal data of the Request and Response objects.
anyone knows of a better way ? i'm using :
def response(ScriptContext, Flow):
Flow.request....
Flow.response....
To access the request or response being intercepted, i'm not changing anything, just observing.
For now the proxy is on 8080, later on it's to be transparent proxy on 80 and 443.
If anyone has done it before i'll be happy if you can share some info.
For those people who want to copy request/response data to clipboard while end up here:
## export the current request/response as curl/httpie/raw/request/response to clipboard
# press colon : and input one of commands and enter
export.clip curl #focus
export.clip httpie #focus
export.clip raw #focus
export.clip raw_request #focus
export.clip raw_response #focus
Mitmproxy: 5.0.1
Source code
couple of things.
first youcan build the raw response yourself using str(flow.request.headers) and request.httpversion and the like.
however it seems that _assemble() and _assemble_headers() do the trick just fine.
so basically:
def request(context, flow):
req = flow.request;
try:
print("Request: -----------------");
print(req._assemble());
print("--------------------------");
except Exception as ee:
print(str(ee));
def response(context, flow):
res = flow.response;
try:
print("Response: -----------------");
print(res._assemble());
if res.content:
size = len(res.content);
size = min(size, 20);
if res.content[0:size] != res.get_decoded_content()[0:size]:
print("\n\n");
print(res.get_decoded_content());
print("--------------------------");
except Exception as ee:
print(str(ee));
as you can see if the decoded body is not similar to the non decoded one (i can check for gzip content type though) i'm printing the decoded message as well.
This should be saved to files according to current dates and each file is named after the client ip taken from request\response.client_conn object. This pretty much solved my problem.
Some check with fiddler shows that the request are reproducable later on which is just what i needed.
I need to invoke a process which doesn't require any input from the user, just a trigger. I plan to use POST /uri without a body to trigger the process. I want to know if this is considered bad from both HTTP and REST perspectives?
I asked this question on the IETF HTTP working group a few months ago. The short answer is: NO, it's not a bad practice (but I suggest reading the thread for more details).
Using a POST instead of a GET is perfectly reasonable, since it also instructs the server (and gateways along the way) not to return a cached response.
POST is completely OK. In difference of GET with POST you are changing the state of the system (most likely your trigger is "doing" something and changing data).
I used POST already without payload and it "feels" OK. One thing you should do when using POST without payload: Pass header Content-Length: 0. I remember problems with some proxies when I api-client didn't pass it.
If you use POST /uri without a body it is something like using a function which does not take an argument .e.g int post (void); so it is reasonable to have function to your resource class which can change the state of an object without having an argument. If you consider to implement the Unix touch function for a URI, is not it be good choice?
Yes, it's OK to send a POST request without a body and instead use query string parameters. But be careful if your parameters contain characters that are not HTTP valid you will have to encode them.
For example if you need to POST 'hello world' to and end point you would have to make it look like this: http://api.com?param=hello%20world
Support for the answers that POST is OK in this case is that in Python's case, the OpenAPI framework "FastAPI" generates a Swagger GUI (see image) that doesn't contain a Body section when a method (see example below) doesn't have a parameter to accept a body.
the method "post_disable_db" just accepts a path parameter "db_name" and doesn't have a 2nd parameter which would imply a mandatory body.
#router.post('/{db_name}/disable',
status_code=HTTP_200_OK,
response_model=ResponseSuccess,
summary='',
description=''
)
async def post_disable_db(db_name: str):
try:
response: ResponseSuccess = Handlers.databases_handler.post_change_db_enabled_state(db_name, False)
except HTTPException as e:
raise (e)
except Exception as e:
logger.exception(f'Changing state of DB to enabled=False failed due to: {e.__repr__()}')
raise HTTPException(HTTP_500_INTERNAL_SERVER_ERROR, detail=e.__repr__())
return response