I'm looking for a tool to which I can feed a file of saved http requests with their respective headers and the tool executes it. I mean, is there something that does that without the need of creating a wrapper? I know I could easily achieve this in any language, but that's not the question in this case. I know Postman, Insomnia, etc, but not quite sure whether I can open a file with HTTP requests and if so what should be the delimiter per request.
Related
I have to give seminar on Requests and Requests-html. I am searching that but can't find any website. Both Requests and Requests-html has same methods but what is the difference
Requests-HTML helps you to parse contents of a webpage (aka web-scraping). You can connect to a webpage and parse its contents like links, raw data, search for specific terms. Generally, it is used for data analytical purpose and requires less technical expertise than requests.
Requests helps you to make HTTP calls programatically. You can send GET/POST et al requests just like curl commands and receive response to be processed by certain logic. Generally backend API developers use it and requires technical knowledge of how HTTP works.
I am writing a web application for an academic research group. The researchers need to be able to upload large data sets (100MB - 1GB) in CSV format. I've written the server to process the data as it comes in. This means that if there is an error in the first row of the CSV, we can return an error straight away.
However, when this happens, the browser reports that "The connection was reset" or similar. Clearly, my web server is responding in a way that doesn't make sense.
If I explicitly close the HTTP request stream (this is Kotlin on the JVM by the way) before returning the error to the browser, then the problem goes away. However, it turns out that the close implementation of the request stream first goes and reads the whole stream to its end. So at that point the user still has to wait 30mins+ to find out that there is an error in the first row of their CSV.
Is what I am trying to do possible? Does the HTTP protocol permit a web server, in any circumstances, to begin responding before the full request body has been sent? If not, can you suggest a workaround that would allow me to deliver a user experience where the user doesn't have to wait for the whole file to be uploaded before finding out if there are any problems?
The answer is yes, according to the http spec servers should be able to send responses early and the client should stop sending the request body. Most browsers however, don't implement this correctly.
In theory, your http server needs to return a 4xx error code with a response body, then reset the connection to prevent the upload continuing in the background. See the answers below for a more detailed description of the issue. There are a couple of browser versions that do support this, so if you're doing this in lab conditions where you can control the client being used the links below will help.
https://stackoverflow.com/a/14483857/2274303
https://stackoverflow.com/a/18370751/2274303
[edit]
To answer your question about using a workaround, chunking the uploads using javascript is a good way to mitigate internet connectivity issues, but if you want to parse it in real time it's not as simple as arbitrarily breaking up the file into pieces. You need to make sure you're not splitting the file in the middle of a line, otherwise it will fail even if the data is valid. That brings up the issue of parsing a 1GB file in javascript, which isn't a good idea imo.
If you want to use javascript, continue uploading the entire file at once via an ajax request, so you can get the response outside of the main dom and force a redirect or cancel the upload. Depending on which js libraries you're using there are different ways of doing this.
None of this solves the reverse scenario. What if the file is 95% uploaded before there's an error? The researcher will need to either upload the whole thing again or edit the file to only include the rows from the error going forward. That means your application needs to support partial uploads and know to pick up where it left off. All these things are possible, but you're probably not going to find a simple workaround to get this working well.
Without understanding the dataset and what kind of validation you are doing it's hard to come up with a full solution. If parsing each row doesn't depend on the previous rows being valid, you could always upload the whole file, then display the rows with errors at the end and ask them to upload a second file with just the corrections.
The normal process of a HTTP web server happens like:
Server listens for request
Client creates request
Client sends request to server
Server processes request
Server creates response
Server sends response to client
Client processes response
The client starts the connection for communication and the server is able to respond on that connection, however if you close the connection the server will need to send a response on another connection. The browser may not allow the server to start a new connection that the client didn't request.
You may be able to respond by reading the first line and creating an error quickly, but the client will not read the response until it is done sending the request.
By sending the file in chunks or asynchronously sending lines of the file, you will be able to give feedback more immediately. You will be sending many smaller requests with the ability to respond in between.
The question was about HTTP protocol. I feel like this would be allowed by the protocol if you wrote a custom app and web app, however if you are using browsers then you must use HTTP as the companies have implemented it. In a custom app you could check for interruptions however most browsers will probably fire a full request before listening for a response, which is also a reason AJAX took off 20years ago.
This probably could not possibly be a more basic HTTP question, but I am very new to web development and I do not even know the right question to ask (evidenced by the fact that googling has not helped).
What I have: an AWS server with an Elastic Beanstalk environment set up. I have successfully compiled, uploaded, and run a simple "Hello World" program to the environment using Eclipse.
What I want to do: pass the server a number via HTTP request and have the server give me back an HTTP response containing the square of that number. On the back end, I want a simple Java class to do the squaring. (Of course, the goal is to be able to pass more complicated data to the server and have more sophisticated Java code on the back end for processing.)
What I think I need to do: create a Java Servlet to listen for and process the request. I think (hope) the documentation is good enough that I can figure out the HTTPServlet API, but I can't answer a more basic question: how do you pass an HTTP request containing some elementary data, like a number?
Thanks in advance!
You need to either GET, or POST (or PUT) your data. GET provides the data in the URL of the request, and will be displayed in the browser's address bar. POST data is provided as a separate request body.
http://www.w3schools.com/tags/ref_httpmethods.asp
A simple GET would look like this:
http://example.com/server?number=4
You can make a POST using a browser extension such as PostMan:
https://chrome.google.com/webstore/detail/postman-rest-client/fdmmgilgnpjigdojojpjoooidkmcomcm?hl=en
Or you can do it from the command line using curl:
curl -X POST http://example.com/server -d'data'
Once the data is more complicated than a few variables, you probably want to use POST rather than GET. Also, you can start to think about what your requests are doing. GETs should only retrieve data from the server. If you modify or create data, then POST (or PUT) requests are the methods to use.
As your server becomes more complex, you probably want to start reading about REST.
http://en.wikipedia.org/wiki/Representational_state_transfer
This is more of an issue of trying to understand how HTTP really works and then implementing it.
I need to have a HTTP analyzer that will be able to separate between the main page requests and "background" requests from some HTTP log data. The idea is to separate HTTP requests made by the user from those that happen automatically (loosely using this term) in the background. So, from the first few impressions of the HTTP data that I've seen it seems like when I go to any normal website an text/html object is fetched followed by a lot of other objects like css, xml, javascript, images etc.
Now, the problem is how do I separate these "background" requests where the user is actively not generating the requests. This will mostly be ad fetches, redirections and some Ajax based things from what I know.
Does anyone has any idea with regards to this. Some, experience or may be resources that you could point me to get started with doing this analysis?
There's no way to distinguish which requests were generated by the browser because of specific user actions or because of other automated processes from the bare HTTP requests. The browser/client it the only one that has such knowledge, so that you have to make it part of the picture, e.g. implementing the analyzer as a browser plugin or to embed an HTTP client as part of the analyzer itself.
If you're trying to create a generic tool to analyze traffic load, it's usually not meaningful to distinguish between traffic generated by user's direct "clicks" and automated requests.
There's no direct and clean way to do this. However, you can get pretty close by filtering out requests for files that clearly are not "user" requests, like *.jpg. Furthermore, you can filter out what is not a HTTP/200 response (e.g., 301 and 302 redirects).
Try something along the lines of:
cat access.log
| grep -E -v "(.gif|.ico|.png|.jpg|.jpeg|.js|.css) HTTP"
| grep "HTTP/1.1\" 200"
(added line breaks for readability)
I would like to write a script that logs into hotmail and changes the first and last name. I am thinking of perl for this, although I imagine that any language would do.
How do I find out what requests to make? I'm guessing that I can do the change requests manually, determine what http requests are being sent to/from the server, determine what pattern is being sent, and then attempt to generate a similar pattern (substituting in the changes I want) in my script.
I took a look at ethereal, but I get a message "the installer is corrupted". I'm going to look at "Advanced http packet sniffer" next - anyone have experience with this tool?
Yes you can use Perl (or any languages with modules that support HTTP(S) ). And you can use libraries already made by people. Eg WWW::Hotmail or Hotwayd. Search CPAN for more of those modules as well.