I have a proxying system that needs to understand the HTTP Cache-Control headers. The intent is not to perform any actual caching, but to derive tuning information based on the caching characteristics of sets of HTTP requests. I'm looking for a way to test the system.
I can do spot checking by pushing content from well-known websites or authored sites to make sure that the system is acting correctly. However, I'd like to expand the pool of test data.
Is there a test suite that enumerates either a set of common or complete caching headers that I can integrate with my software to make sure I'm covering all the bases I need to cover?
I know the Apache HTTPClient (HTTPCore) has a pretty extensive test suite, though I don't know how deeply it goes into the cache stuff. This is Java, don't know if you care about the language or not.
Related
TL;DR
Is there a tool can record all the network activity as I visit a website and create a mock server that responds to those requests with the same responses?
I'm investigating ways of mocking the complex backend for our React application. We're currently developing against the real backend (plus test/staging environments). I've looked around a bit and it looks there are a number of tools for mocking individual endpoints/features and sending the rest through to the real API (Mirage is leading the pack at the moment).
However, the Platonic ideal would be to mock the entire server so that a front end dev can work without an internet connection (again: Platonic ideal). It's a crazy lofty goal, I know this. And of course it would require mocking not only our backend but also requests any 3rd-party data sources. And of course the data would be thin and dumb and stale. But this is just for ultra-speedy front end development, it's just mocking. The data doesn't need to be rich, it'll be up to us to make it as useful/realistic as we need it to be.
Probably the quickest way would be to recreate the responses the backend is already sending, and then modifying as needed for new features or features under test etc.
To do this, we might go into Chrome DevTools and recreate everything on the network tab. Mock every request that was made by hardcoding response that returned. Taking it from there, do smart things like use url pattern matching to return a simple placeholder image for any request to get a user's avatar.
What I want to know is: is there any tool out there that does this automatically? That can watch as I load the site, click a bunch of stuff, take a bunch of actions, and spit out or set up a mock that recreates all the responses? And then we could edit any of them as we saw fit to simplify.
Does something like this exist? Maybe it's a browser tool. Maybe it's webpack middleware. Maybe it's a magic rooster.
PS. I imagine this may not be a specific, actionable enough question for SO. I'll understand if it's closed, but I'd really appreciate being directed somewhere where such questions/discussions would fit? I'm new enough to this world that SO is all I know!
There is a practice called service virtualization - a subset of the test double family.
Wikipedia has a list of tools you can use to do that. Here a couple of examples from that list:
Open Source Wiremock will let to record the mocks and edit the responses programmaticaly
Commercial Traffic Parrot will let you record the mocks and edit the responses via a UI and/or programatically
https://mswjs.io/ can mock all the requests for you. It intercepts all your client`s requests and returns your defined mock data.
I am playing around with making some e-learning content that is SCORM 1.2 compliant. However, it seems that you must package everything to be delivered and used on an LMS. Is it still possible to make calls to an external server? I assume it is but is this bad practice?
An example for clarification:
Say I have a service running on an external server that I cannot wrap to deliver to the LMS. However, I would still like to make requests to it. Can this be done from a SCORM course as a user is doing the course?
You can make the call, but as you point out it would be a bad practice.
Essentially you'd have no trust of who or what is making the call because you can't put a secure credential in front of it and the call itself (or an authentication request prior to the call) would be susceptible to MITM attacks.
If you were simply loading an insecure resource then you could do that, CORS would apply.
Having said that, you may be interested in the Rustici Cross Domain product https://rusticisoftware.com/products/rustici-cross-domain/ (full disclosure I work for Rustici).
I'm using HttpBuilder (a Groovy HTTP library built on top of apache's httpclient) to sent requests to the last.fm API. The docs for this API say you should set the user-agent header to "something appropriate" in order to reduce your chances of getting blocked.
Any idea what kind of values would be deemed appropriate?
The name of your application including a version number?
I work for Last.fm. "Appropriate" means something which will identify your app in a helpful way to us when we're looking at our logs. Examples of when we use this information:
investigating bugs or odd behaviour; for example if you've found an edge case we don't handle, or are accidentally causing unusual load on a system
investigating behaviour that we think is inappropriate; we might want to get in touch to help your application work better with our services
we might use this information to judge which API methods are used, how often, and by whom, in order to do capacity planning or to get general statistics on the API eco-system.
A helpful (appropriate) User-Agent:
tells us the name and version of your application (preferably something unique and easy to find on Google!)
tells us the specific version of your application
might also contain a URL at which to find out more, e.g. your application's homepage
Examples of unhelpful (inappropriate) User-Agents:
the same as any of the popular web browsers
the default user-agent for your HTTP Client library (e.g. curl/7.10.6 or PEAR HTTP_Request)
We're aware that it's not possible to change the User-Agent sent when your application is browser-based (e.g. Javascript or Flash) and don't expect you to do so. (That shouldn't be a problem in your case.)
If you're using a 3rd party Last.fm API library, such as one of the ones listed at http://www.last.fm/api/downloads , then we would prefer it if you added extra information to the User-Agent to identify your application, but left the library name and version in there as well. This is immensely useful when tracking down bugs (in either our service or in the client libraries).
Are any CDNs (Content Delivery Networks) that provide control and/or customization of all or most HTTP headers?
Specifically, I'm interested in controlling the Expires, ETag, and Cache-Control headers, although other headers interest me as well.
I understand that part of the value proposition of CDNs is that they "just work" and set these headers to somewhat optimal values (for most use cases), but I am definitely interested in controlling these headers myself.
Akamai has a full interface for allowing this type of control on a per-property, per-header basis. It is a standard XML based config file. You can set each header to be a specific value, respect the headers passed through, add if not present, have exceptions based on User Agent etc.
Essentially, within reason, it is completely configurable. I have found setting defaults when absent but allowing applications/admins to set their own values is generally the best approach but it really does depend on the quality and understanding of the developer/admin.
Like most CDN providers Akamai have some default behaviors baked in, but the values are completely configurable. It has been a couple of years since I actively managed a CDN, but at the time Limelight was working on being feature compatible with Akamai and was most of the way there, so I would expect that they have similar functionality now.
In general, most CDN vendors will strive for feature compatibility with the big player in the market and Akamai is definitely it for CDN.
OpenSocial and some of the newer Google APIs include these tokens, such as "#me" or "#self", whose values are replaced by the API server with values based on the currently authenticated user. For example "/api/people/#me/#all" is an OpenSocial REST URL.
Doesn't this break with the goal of REST APIs to support native HTTP cache servers (like Squid)?
Even if you could get around the issue using the "Vary" header, it seems like a major drawback. And only real benefit is to allow developers to hard code some URIs into their apps. Anyone know why it was designed this way?
Yes it will make the use of public caches difficult. Personally I think it is a really bad idea and does seem to be driven by the desire to make it easier for clients to construct URIs. I sometimes wonder if the extensive use of caching servers like memcached are causing developers to forget about the benefits of http caching.