I am logging raw http requests to kafka.
GET /api/v1/user
GET /api/v1/friends
POST /api/v2/problem/solve HTTP/1.1
Host: domain.com
Content-Length:111
Cookie:...
{
"input":"{...}"
}
Obstruction, such as sessions and JWT tokens, will modify the source of the server.
I would like to make new requests based on these data, test them, and see the results, can you recommend a suitable solution?
I looked for Jmeter, K6, and others, but it was not a satisfactory solution.
You're looking for a load testing tool which will take the stuff you put to Kafka and generate end-to-end load test out of it? Good luck.
JMeter might be used in at least 2 ways:
If you're capable of exporting messages from Kafka into files you could consider using HTTP Raw Request sampler which has more or less the same syntax as your records
If not, you could use JSR223 PreProcessor for reading the records from Kafka and constructing the HTTP Request out of them, the JMeter API class you will need is HTTPSamplerProxy
Related
I have to give seminar on Requests and Requests-html. I am searching that but can't find any website. Both Requests and Requests-html has same methods but what is the difference
Requests-HTML helps you to parse contents of a webpage (aka web-scraping). You can connect to a webpage and parse its contents like links, raw data, search for specific terms. Generally, it is used for data analytical purpose and requires less technical expertise than requests.
Requests helps you to make HTTP calls programatically. You can send GET/POST et al requests just like curl commands and receive response to be processed by certain logic. Generally backend API developers use it and requires technical knowledge of how HTTP works.
I am using a fairly expensive external API (there's a cost per request) which makes testing code which uses it impractical.
In an ideal world, I would have a proxy server I would do my requests against which would cache each request (based on URL + query string) indefinitely and only hit the actual API server when I explicitly invalidate the cache for a given request. Is such a server available off the shelf with minimal configuration?
My current stack is Node.js, Docker, Nginx, PostgreSQL & AWS S3 (for non ephemeral state). I think Varnish might accomplish what I need but I'm not sure.
Varnish can and will accomplish that, but only if you build a 'test' API that returns some similar data you can play with. Your best bet if you have to save money, is to query the API a few times to get different typical responses. Once you know the ballpark of what to expect from it, create some sort of dummy API, or even some static JSON or XML files that you can use to mimic it. At that point you can test Varnish and Cache invalidation, and I'd be more than happy to help you with the syntax for that, given some examples of the code.
I am using JBoss JMX Console to monitor my Web Application.
How can i find what http requests are being processed at any point in time?
For eg: I see 25 busy theads - i want to know what http requests are these threads processing.
I am not really sure if there is an ability to map a specific request to a thread but you can certainly see what HTTP requests are made to Tomcat using AccessLogValve. You can probably use the timestamps to map those requests if need be.
Jasper;
The arduous way to do this is to examine every instance of the MBeans that have this pattern:
jboss.web:name=HttpRequest1,type=RequestProcessor,worker=http-0.0.0.0-18080
They are MBeans that represent the web request serving threads and they have an attribute called currentQueryString which is the query string of the request currently being processed. There are also attributes for currentUri and method. You could script the collection of this data as well.
An easier way, which is enabled in JBoss servers by default, is to use the available at:
http://localhost:8080/web-console/status
It handily aggregates those same MBean's and reports them in one page.
There is also options for a fuller report
http://localhost:8080/web-console/status?full=true
and an XML formatted output
http://localhost:8080/web-console/status?XML=true
This is more of an issue of trying to understand how HTTP really works and then implementing it.
I need to have a HTTP analyzer that will be able to separate between the main page requests and "background" requests from some HTTP log data. The idea is to separate HTTP requests made by the user from those that happen automatically (loosely using this term) in the background. So, from the first few impressions of the HTTP data that I've seen it seems like when I go to any normal website an text/html object is fetched followed by a lot of other objects like css, xml, javascript, images etc.
Now, the problem is how do I separate these "background" requests where the user is actively not generating the requests. This will mostly be ad fetches, redirections and some Ajax based things from what I know.
Does anyone has any idea with regards to this. Some, experience or may be resources that you could point me to get started with doing this analysis?
There's no way to distinguish which requests were generated by the browser because of specific user actions or because of other automated processes from the bare HTTP requests. The browser/client it the only one that has such knowledge, so that you have to make it part of the picture, e.g. implementing the analyzer as a browser plugin or to embed an HTTP client as part of the analyzer itself.
If you're trying to create a generic tool to analyze traffic load, it's usually not meaningful to distinguish between traffic generated by user's direct "clicks" and automated requests.
There's no direct and clean way to do this. However, you can get pretty close by filtering out requests for files that clearly are not "user" requests, like *.jpg. Furthermore, you can filter out what is not a HTTP/200 response (e.g., 301 and 302 redirects).
Try something along the lines of:
cat access.log
| grep -E -v "(.gif|.ico|.png|.jpg|.jpeg|.js|.css) HTTP"
| grep "HTTP/1.1\" 200"
(added line breaks for readability)
I'm writing a simple HTTP server which should play nicely with most clients, but is only meant to implement a subset of HTTP 1.1.
During development it would be nice to be able to validate that the generated HTTP reponses are HTTP 1.1 compliant. Is there a tool that can do something along those lines?
Thanks /Erik
It's not a complete conformance suite, but RED does check for a number of different HTTP requirements, and finds common problems.
http://redbot.org/
You could just write unit test cases using any http client library. Make GET and POST requests to your webserver, parse the response and make assertions. As you add additional features, add more test cases.
For example, lets say you only support url-encoded POST requests. So, you write a test case which verifies your server understands url-encoded requests and responds appropriately. Tomorrow, when you add support for multi-part support - that would be another test case altogether.
Every programming language under the sun has good support for HTTP, so writing the test case is a no-brainer.