i', scripting mitm proxy (http://mitmproxy.org/index.html) to write HTTP and HTTPS request and responses to a file according to their IP (each client can then access it's own requests\responses) for unit tests for mobile.
As far as i can see for now i can't just use str(Flow.request) or repr(Flow.request) to get a "raw" print of the response\request like i get in fiddler, i need to reconstruct it from the internal data of the Request and Response objects.
anyone knows of a better way ? i'm using :
def response(ScriptContext, Flow):
Flow.request....
Flow.response....
To access the request or response being intercepted, i'm not changing anything, just observing.
For now the proxy is on 8080, later on it's to be transparent proxy on 80 and 443.
If anyone has done it before i'll be happy if you can share some info.
For those people who want to copy request/response data to clipboard while end up here:
## export the current request/response as curl/httpie/raw/request/response to clipboard
# press colon : and input one of commands and enter
export.clip curl #focus
export.clip httpie #focus
export.clip raw #focus
export.clip raw_request #focus
export.clip raw_response #focus
Mitmproxy: 5.0.1
Source code
couple of things.
first youcan build the raw response yourself using str(flow.request.headers) and request.httpversion and the like.
however it seems that _assemble() and _assemble_headers() do the trick just fine.
so basically:
def request(context, flow):
req = flow.request;
try:
print("Request: -----------------");
print(req._assemble());
print("--------------------------");
except Exception as ee:
print(str(ee));
def response(context, flow):
res = flow.response;
try:
print("Response: -----------------");
print(res._assemble());
if res.content:
size = len(res.content);
size = min(size, 20);
if res.content[0:size] != res.get_decoded_content()[0:size]:
print("\n\n");
print(res.get_decoded_content());
print("--------------------------");
except Exception as ee:
print(str(ee));
as you can see if the decoded body is not similar to the non decoded one (i can check for gzip content type though) i'm printing the decoded message as well.
This should be saved to files according to current dates and each file is named after the client ip taken from request\response.client_conn object. This pretty much solved my problem.
Some check with fiddler shows that the request are reproducable later on which is just what i needed.
Related
I'm trying to get live trading data from the Internet via HTTP, but it is updated continuously, so if I GET the data, it will keep downloading as long as there is data available. Until I stop the downloading stream, then I can access the data.
How to access the stream of data while the downloading is in progress?
I tried using Indy's TIdHTTP, so I can use SSL, but I tried the IdIOHandlerStream, but it was already used for IdSSLIOHandlerSocketOpenSSL. So I'm absolutely clueless here.
This is in response to a "multipart/form-data" request.
Please guide me...
Lrequest.Values['__RequestVerificationToken'] := RequestVerificationToken;
Lrequest.Values['acct'] := 'demo';
Lrequest.Values['pwd'] := 'demo';
try
Response.Text := Onhttp.Post('https://trading/data', Lrequest);
Form1.Memo1.Lines.Add(TimeToStr(Time) + ': ' + Response.Text);
except
on E: Exception do
Form1.Memo1.Lines.Add(TimeToStr(Time) + ': ' + E.ClassName +
' error raised, with message : ' + E.Message);
end;
UPDATE:
The data is an endless JSON string, like this:
{"id":"data","val":[{"rc":2,"tpc":"\\RealTime\\Global\\SGDIDR.FX","item":[{"val":{"F009":"10454.90","F011":"-33.1"}}]}]}
{"id":"data","val":[{"rc":2,"tpc":"\\RealTime\\Global\\SGDIDR.FX","item":[{"val":{"F009":"10458.80","F011":"-29.2"}}]}]}
and so on, and so on...
You can't use TIdIOHandlerStream to interface with a TCP connection, that is not what it is designed for. It is meant for performing I/O operations using user-provided TStream objects, ie for debugging previously captured sessions.
TIdHTTP is not really designed to handle endless HTTP responses in most cases, as you have described. What is the exact format that the server is delivering its live data as? What do the HTTP response headers look like? It is really difficult to answer your question without know the exact format being used.
However, that being said, there are some cases to consider, depending on what the server is actually sending:
if the server is using a MIME-based server-push format, like multipart/x-mixed-replace, you can enable the hoNoReadMultipartMIME flag in the TIdHTTP.HTTPOptions property, and then read the MIME data yourself from the TIdHTTP.IOHandler after TIdHTTP.Get() exits. For instance, you can use TIdMessageDecoderMIME to help you parse the MIME parts, see New TIdHTTP hoNoReadMultipartMIME flag in Indy's blog, or Delphi Indy TIdHttp and multipart/x-mixed-replace with Text and jpeg image.
Otherwise, if the server is using Transfer-Encoding: chunked, where each data update is sent as a new HTTP chunk, you can use the TIdHTTP.OnChunkReceived event. Or, you can enable the hoNoReadChunked flag in the TIdHTTP.HTTPOptions property, and then read the chunks yourself from the TIdHTTP.IOHandler after TIdHTTP.Get() exits. See New TIdHTTP flags and OnChunkReceived event in Indy's blog.
Otherwise, you could give TIdHTTP.Get() a TIdEventStream to write into, and then use that stream's OnWrite event to access the raw bytes. Or, you could write your own TStream-derived class that overrides the virtual Write() method. Either way, you would be responsible for manually parsing and buffering the raw body data as they are being written to the stream.
Otherwise, you may have to resort to using TIdTCPClient instead, implementing the HTTP protocol manually, then you would be solely responsible for reading in the HTTP response body however you want.
I have learned you can "decorate" the HTTP transport so that you can details of the request, however I can't quite figure out how you log the URL in the same line.
https://play.golang.org/p/g-ypQN9ceGa
results in
INFO[0000] Client request dns_start_ms=0 first_byte_ms=590 response_code=200 total_ms=590 url=
INFO[0000] 200
I'm perpetually confused if I should be using https://golang.org/pkg/context/#WithValue to pass around the context in a struct, especially in light where https://blog.golang.org/context-and-structs concludes with pass context.Context in as an argument.
Go through the behaviour of how the request is constructed in request.go from net/http. You see that the RequestURI field is never set there. Quoting from same reference,
Usually the URL field should be used instead.
It is an error to set this field in an HTTP client request
So, I would suggest you to use request.URL instead.
It is a parsed from the request uri. It should have the data you need.
You can construct the output as following:
f := log.Fields{
"url": fmt.Sprintf("%s %s%s", r.Method, r.URL.Host, r.URL.Path),
}
Also, in my experience, it is far more easier to use context.WithValue and pass the context as an argument.
Replace r.RequestURI by r.URL.String() in your code to log the full, valid URL (https://golang.org/pkg/net/url/#URL.String). RequestURI is empty on the client side (https://golang.org/pkg/net/http/#Request), as the output from your code is showing.
I don't see how context.Context relates to your question, but I believe https://blog.golang.org/context-and-structs is considered "best practice".
I want to return a TSV file from a web call in Hunchentoot (SBCL), but want the user to just save the raw result blatted to the page, rather than use a separate file and download link (which is hard because of local firewall complexities).
I can't figure out how to output the page without any headers at all, i.e., to make it just plain raw text. (I know that the browser would make a mess w/o headers in the DOM, but don't care; the goal is just to have the user save the page, not read it.)
I've tried various combinations of
(setf (hunchentoot:content-type*) "text/plain")
and
(cl-who:with-html-output-to-string
(*standard-output* nil :prologue nil)
and setting the content-type* inside, outside, and around the with... but I always get header junk.
Writing a string directly
I tried defining a handler as follows:
(define-easy-handler (text :uri "/text") ()
(setf (content-type*) "text/csv")
"a,b,c")
When I visit the page locally, the browser automatically downloads a text file without even displaying (this is probably a setting we can change in Chrome, I don't know).
When I enable the browser developer mode, here are the response headers I receive as part of the HTTP protocol:
HTTP/1.1 200 OK
Server: ...
Date: ...
Content-Type: text/csv; charset=utf-8
Content-Length: 5
Connection: keep-alive
But the file itself is just the string a,b,c.
If I change the content-type to "text/plain", then the browser successfully displays the text, and nothing else (the HTTP headers are the same).
Remarks
You don't need to use the cl-who macros if you do not intend to build an HTML document, in fact its better not to. In any case, you can supply your own REPLY-CLASS when initializing the acceptor (see https://edicl.github.io/hunchentoot/#replies) and have a very low-level control about what you emit as a reply, headers included. But I don't think this is necessary in your case. I don't clearly understand where your problem comes from, but sending back a plain text is something the framework is supposed to be able to do out of the box. Please add more details if you can.
Is the correct answer not to use the Content-Disposition header?
I am creating a server using Go that allows the client to upload a file and then use a server function to parse the file. Currently, I am using two separate requests:
1) First request sends the file the user has uploaded
2) Second request sends the parameters to the server that the server needs to parse the file.
However, I have realised that due to the nature of the program, there can be concurrency problem if multiple users try to use the server at the same time. My solution to that was using mutex locks. However, I am receiving the file, sending a response, and then receiving the parameters and it seems that Go cannot send a response back when the mutex is locked. I am thinking about solving this problem by sending both the file and the parameters in one single HTTP request. Is there a way to do that? Thanks
Sample code (only relevant parts):
Code to send file from client:
handleUpload() {
const data = new FormData()
for(var x = 0; x < this.state.selectedFile.length; x++) {
data.append('myFile', this.state.selectedFile[x])
}
var self = this;
let url = *the appropriate url*
axios.post(url, data, {})
.then(res => {
//other logic
self.handleParser();
})
}
Code for handleParser():
handleNessusParser(){
let parserParameter = {
SourcePath : location,
ProjectName : this.state.projectName
}
// fetch the the response from the server
let self = this;
let url = *url*
fetch(url, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(parserParameter),
}).then( (response) => {
if(response.status === 200) {
//success logic
}
}).catch (function (error) {
console.log("error: ", error);
});
}
The question is not really about Go or reactjs or any particular software library.
To solve your problem you'd first need to understand how HTTP POST works,
hence I invite you to first read this intro on MDN.
In short:
There are multiple ways to encode the data sent in a POST request.
The way the receiver should deal with this data depends on how it's encoded by the sender.
The sender has to communicate the encoding with its request — usually via the Content-Type header field.
I won't go into the details of possible encodings — the referenced introductory material covers them, and you should do your own research on them, but to maybe recap what's written there, here is some perspective.
Back in the 80s and 90s the web was "static" and the dreaded era of JavaScript-heavy "web apps" did not yet come. "Static" means you could not run any code in the client's browser, and had to encode any communication with the server in terms of plain HTML.
An HTML document could have two ways to make the client rendering it to send something back to the server: a) embed an URL which would include query parameters; this would make the client to perform a GET request with these parameters sent to the server; b) embed an HTML "form" which, when "submitted", would result in performing a rather more complex POST request with the data taken from the filled in form.
The latter approach was the way to leverage the browser's ability to perform reasonably complex data processing — such as slurpling a file selected by the user in a specific form's control, encoding it appropriately and sending it to the server along with the other form's data.
There were two ways to encode the form's data, and they are both covered by the linked article, please read about them.
The crucial thing to understand about this "static web with forms" approach is that it worked like this: the server sends an HTML document containing a web form, the browser renders the document, the user fills the form in and clicks the "submit" button rendered by the browser; the browser collects the data from the form's controls, for entries of type "file" it reads and encodes the contents of those files and finally performs an HTTP POST request with this stuff encoded to the URL specified by the form. The server would typically respond with another HTML document and so on.
OK, so here came "web 2.0", and an "XHR" (XMLHttpRequest) was invented. It has "XML" in its name because that was the time when XML was perceived by some as a holy grail which would solve any computing problem (which it, of course, failed to do). That thing was invended to be able to send almost arbitrary data payloads; XML and JSON encoding were supported at least.
The crucial thing to understand is that this way to communicate with the server is completely parallel to the original one, and the only thing they share is that they both use HTTP POST requests.
By now you should possibly see the whole picture: contemporary JS libs allow you to contruct and perform any sort of request: they allow you to create a "web form"-style request or to create a JS object, and serialise it to JSON, and send the result in an HTTP POST request.
As you can see, any approach allows you to pass structured data containing multiple distinct pieces of data to the server, and the way to handle this all is a matter of agreement between the server and the client, that is, the API convention, if you want.
The difference between various approaches is that the web-form-style approach would take care of encoding the contents of the file for you, while if you opt to send your file in a JSON object, you'll need to encode it yourself — say, using base64 encoding.
Combined approaches are possible, too.
For instance, you can directly send binary data of a file as a POST request's body, and submit a set of parameters along with the request by encoding them as query-parameters of the URL. Again, it's up to the agreement between the client and the server about how the latter encodes the data to be sent and the former decodes them.
All-in-all, I'd recommend to take a pause and educate yourself on the stuff I have outlined above, and then have another stab at solving the problem, but this time — with reasonably complete understanding about how the stuff works under the hood, and how you intend to wield it.
I need to reuse value which is generated for my previous request.
For example, at first request, I make a POST to the URL /api/products/{UUID} and get HTTP response with code 201 (Created) with an empty body.
And at second request I want to get that product by request GET /api/products/{UUID}, where UUID should be from the first request.
So, the question is how to store that UUID between requests and reuse it?
You can use the Request Sent Dynamic values https://paw.cloud/extensions?extension_type=dynamic_value&q=request+send these will get the value used last time you sent a requst for a given request.
In your case you will want to combine the URLSentValue with the RegExMatch (https://paw.cloud/extensions/RegExMatch) to first get the url as it was last sent for a request and then extract the UUID from the url.
e.g
REQUEST A)
REQUEST B)
The problem is in your first requests answer. Just dont return "[...] an empty body."
If you are talking about a REST design, you will return the UUID in the first request and the client will use it in his second call: GET /api/products/{UUID}
The basic idea behind REST is, that the server doesn't store any informations about previous requests and is "stateless".
I would also adjust your first query. In general the server should generate the UUID and return it (maybe you have reasons to break that, then please excuse me). Your server has (at least sometimes) a better random generator and you can avoid conflicts. So you would usually design it like this:
CLIENT: POST /api/products/ -> Server returns: 201 {product_id: UUID(1234...)}
Client: GET /api/products/{UUID} -> Server returns: 200 {product_detail1: ..., product_detail2: ...}
If your client "loses" the informations and you want him to be later able to get his products, you would usually implement an API endpoint like this:
Client: GET /api/products/ -> Server returns: 200 [{id:UUID(1234...), title:...}, {id:UUID(5678...),, title:...}]
Given something like this, presuming the {UUID} is your replacement "variable":
It is probably so simple it escaped you. All you need to do is create a text file, say UUID.txt:
(with sample data say "12345678U910" as text in the file)
Then all you need to do is replace the {UUID} in the URL with a dynamic token for a file. Delete the {UUID} portion, then right click in the URL line where it was and select
Add Dynamic Value -> File -> File Content :
You will get a drag-n-drop reception widget:
Either press the "Choose File..." or drop the file into the receiver widget:
Don't worry that the dynamic variable token (blue thing in URL) doesn't change yet... Then click elsewhere to let the drop receiver go away and you will have exactly what you want, a variable you can use across URLs or anywhere else for that matter (header fields, form fields, body, etc):
Paw is a great tool that goes asymptotic to awesome when you explore the dynamic value capability. The most powerful yet I have found is the regular expression parsing that can parse raw reply HTML and capture anything you want for the next request... For example, if you UUID came from some user input and was ingested into the server, then returned in a html reply, you could capture that from the reply HTML and re-inject it to the URL, or any field or even add it to the cookies using the Dynamic Value capabilities of Paw.
#chickahoona's answer touches on the more normal way of doing it, with the first request posting to an endpoint without a UUID and the server returning it. With that in place then you can use the RegExpMatch extension to extract the value from the servers's response and use it in subsequent requests.
Alternately, if you must generate the UUID on the client side, then again the RegExpMatch extension can help, simply choose the create request's url for the source and provide a regexp that will strip the UUID off the end of it, such as /([^/]+)$.
A third option I'll throw out to you, put the UUID in an environment variable and just have all of your requests reference it from there.