Most of the question is in the title.
I am trying to test an api that generates and streams large files. I am make the request with paw.
How does it handle the response?
Where are the downloaded files located?
Thank you
You should be able to visualise the raw bytes in the "Text" tab, though maybe it won't help much:
Though, you can also enable the "Hex" tab in the Paw menu > Preferences > Response > "Show Hex Response Tab". Once you've that enabled you'll have a Hex formatted tab (note that there are known performance issues on the Hex tab for large responses, maybe > 500kB, we should work on improving it):
Also, you can export the body as a file as you pointed out. In File menu > Export Response > Response Body:
Hope it helps!
Related
I am trying to extract direct HTTP, RTSP, MP stream URLs or something that I can work with from this website:
http://kamery.ovanet.cz/
Any idea how to do it, I've tried f12 using chrome, but wasn't sucessful.
Thank you.
"Any idea how to do it, I've tried F12 using Chrome, but wasn't successful."
You have to first open the Network tab of Chrome's Developer tools then...
In the small text box that shows filter text, type in there: .m3u8
Select a camera. It should show multiple .m3u8 files being loaded
Choose a playlist.m3u8 that does not have ?auth=b64%3A etc in the URL
Try testing this link (is M3U8 playlist) :
https://stream.ovanet.cz/ovanet/f89ab1fdfeba19d51fa26dd491edcbaa/camera97/playlist.m3u8
You can try the above link with some online player : HLS Stream Tester.
Edit : Regarding comments...
"The URL link works for just a while and then it stops. Can I solve it somehow?"
Yes, their links are time-limited and will expire so you need a page refresh to get the newest version of the M3U8 link.
If you want to automate the process then you will have to write code that achieves the following:
Choose a camera ID number from this listing. Notice that the "cam_id" is followed by the related location's "title". Example is "cam_id":19.
Using the camera's ID number, make a request to this URL:
https://stream.ovanet.cz/player/api/embed-live.js?stream=camera19&_=XXXX
Note: Where XXXX is the current time as a UNIX timestamp (eg : 1659717867824).
This will return some text. Part of that text has the URL with the Base64 "auth" that you need for a correct request. You get it by finding the start position of return " and extracting (via String functions) everything between up until the first occurrence of "; going forward from the start position.
Request this newly extracted link, which looks like example:
https://stream.ovanet.cz/ovanet/camera19/playlist.m3u8?auth=b64%3AY2FtZXJhMTk6OjoxMjg0MzQzMTU3OjE2NTk3MjE1NjAwODc6MTY1OTcyNTE2MDo5Mi40MC4xOTAuMjc6MDdiZGUxN2MwMGY1Nzg5YTkwZjlkMmY3YjdmM2JmYTM%3D
There will be a re-direct to another URL of same M3U8 file name but from a different location, so you need to find out this new URL. It has the folder name you need.
Eg: the folder name is: /f3630f4527ce8ddd69bf27580737d66f/ from: https://stream.ovanet.cz/ovanet/f3630f4527ce8ddd69bf27580737d66f/camera19/playlist.m3u8
The final URL is basically (using chunklist.m3u8 seems to work better):
https://stream.ovanet.cz/ovanet/f3630f4527ce8ddd69bf27580737d66f/camera19/chunklist.m3u8
Everytime I am testing postman collection ,I need to change the authorization token under header manually followed by exporting the collection again and running through newman.
Is there any way , instead of giving here, I can give it in a CSV file which is being used as test data file. which would reduce the efforts of changing code every time.
Please suggest.
Yes you can, in your csv file, add one more column - name as "AT"
and mention that reference AT in your request as in the following picture.
I'm new to BW6 (v.6.3.1) and I'm playing around with it's REST features.
Currently I'm building a very simple echo service to figure out how it processes parameters etc.
One of the things I've noticed is that I can specify multiple Reply Client Formats, namely XML & JSON. But I can't find how I can specify what output to use in the actual reply.
Currently I've setup the following Resource Service Path:
/echo/{param}.{format}
I want to use the format parameter to drive the output I'll be getting. So
/echo/printme.xml
would result in
<messageBody>printme</messageBody> (or something to that extent)
And
/echo/printme.json
would result in
printme
I've tried several approaches, splitting the flow based on the contents of "format" but to no avail. If I have JSON checked as the Reply Client Format it will reply with JSON. Only if XML is the only Reply Client Format checked the reply will output XML. BW handles the rendering of either JSON or XML transparently.
So, how can I tell BW to output in either XML or JSON when both are selected as a Reply Client Format?
Edit:
Using the swagger UI I figured out how I can drive this behavior. By specifying the following header:
curl -X GET --header "Accept: application/xml" "http://localhost:8080/echo"
As per the documentation
Select the Invoke REST API pallette, you can choose the type(either request or response) as shown below:
If you click on it, there are three options JSON, XML and Custom. If you want to choose other than json and xml then choose Custom.
Custom(For RequestType) : Custom: to override the Content-Type value in the Input tab, select CUSTOM and provide the value in the Input tab.
Custom(For ResponseType): to override the Accept header value in the Input tab, select CUSTOM and provide the value in the Input tab.
Below is the Input tab where you can override when type is Custom:
i have written a flex component to allow the user to select an image from the local filesystem and then POST it to a CQ5 DAM.
there are 2 CQ5 instances with which i'm working. the image posts fine to one instance, but not the other. specifically, in the 2nd instance, the renditions are not getting created when using the component.
one difference i've noted is that the working images, when i look at them in crxde, have a jcr:primaryType of dam:Asset. the non-working ones are nt:File.
from Flex, I use URLLoader to POST with a multipart form. the request (in part) looks like this:
POST /content/dam/test/foo.createasset.html HTTP/1.1
Host: xxxxxxxx:4502
Content-type: multipart/form-data; boundary=doudrbitutcfasnbhlpogirdctuxem
--doudrbitutcfasnbhlpogirdctuxem
Content-Disposition: form-data; name="file"
home.png
--doudrbitutcfasnbhlpogirdctuxem
Content-Disposition: form-data; name="Filename"
home.png
--doudrbitutcfasnbhlpogirdctuxem
Content-Disposition: form-data; name="home.png"; filename="home.png"
Content-Type: application/octet-stream
*** image data ***
--doudrbitutcfasnbhlpogirdctuxem
Content-Disposition: form-data; name="Upload"
Submit Query
--doudrbitutcfasnbhlpogirdctuxem--
that does save the image at: /content/dam/test/foo/home.png
i've tried adding a variable to the form:
./jcr:contentType dam:Asset
but that didn't cause the contentType to change. instead, the file didn't show up in CQ5 at all.
i know next to nothing about CQ5. i've seen some (old) examples of code POSTing right to where they want the asset to go, instead of hitting foo.createAsset.html as i've done. i could not get the more-straightforward POST working, and instead used CQ5 DAM to upload and image and captured through Charles, then tried to replicate that.
the CQ5 version that works is 5.5.0.
the version that does not is 5.4.0.
i'm sure that there are other configuration differences as well. in addition, the client is unwilling to upgrade from 5.4.0.
am i on the right track? close?
edit to clarify server setup:
CQ 5.5.0 --> installed locally, this one is an author server. my component works when POSTing to this server. meaning, the uploaded image is marked as dam:Asset and the renditions are generated.
CQ 5.4.0 --> a dev instance used by many. this is an author and publish server. my component does not 100% work when POSTing to this server. however, if i use the DAM admin interface to upload an image, it does properly mark the image as dam:Asset and generate the renditions.
edit #2: WORKING
it turns out that the dev/5.4 instance handles file uploading differently. my multi-part POST code mostly worked, but instead of using createAsset.html, i'm uploading to /tmp/fileupload.
then i issue a 2nd POST, using application/x-www-form-urlencoded, to issue a move command.
for those wishing to do the same, the move code looks like this:
var service:HTTPService = new HTTPService();
var url:String = instanceUrl + "/tmp/fileupload";
service.url = url;
var headerData : Object = new Object();
headerData['Cache-Control'] = 'no-store';
headerData['Authorization'] = getAuthString();
service.headers = headerData;
service.contentType = "application/x-www-form-urlencoded; charset=UTF-8";
service.method = URLRequestMethod.POST;
var urlVar:URLVariables = new URLVariables();
var command:String = "/var/dam/" + destPath + "/" + filename + "#MoveFrom";
var arg:String = "/tmp/fileupload/" + filename;
urlVar[command] = arg;
urlVar["_charset_"] = "utf-8";
var token:AsyncToken = service.send(urlVar);
not knowing CQ5, i can only assume the dev server is set up to run some workflow steps when it receives the #MoveFrom; those are the steps that ensure the uploaded file is of type dam:Asset and that the desired renditions are created.
If uploading from the DAM admin page via a browser works on the 5.4.0 instance, I would suggest analysing the HTTP request that this makes, to reproduce the same request from your Flex client. There's probably a subtle difference between the 5.4.0 and 5.5.0 HTTP APIs that explains this.
as a followup, below are the broad steps i took to get this working.
my overall goal was to write a Flex component that, for a specified VO, allowed the user to upload an image from their local filesystem (i used FileReference for this) into the component, then upload that image to CQ5 and publish it. after it was published, i then read it back into the component to display it.
i won't put the full code solution here, as it's involved and belongs to my client. in addition to my component, i wrote a utility for cq5 DAM operations, and an http service with built-in retries (which ended up being necessary because even though cq will give me a 200 when i request a resource, subsequent operations on that resource may fail, because cq doesn't seem to think it's there). Note that in all retry instances, i have a max retry count. the default value is 10, and default retry interval is 250ms.
please understand i know very little of CQ; most of what i learned was reverse engineering through trying things in the tool and watching Charles. also understand the steps below may be very specific to the install of CQ5 i'm working with.
so here are my overall steps. unless indicated otherwise, all requests are on port 4502:
a destination directory is determined from data in the VO and a POST is issued to create it. this is done with Content-type=application/x-www-form-urlencoded. the url is the full path of the folder i want to create, with no trailing slash.
repeat a GET on the created directory until we get a 200. the url here does have a trailing slash.
the image is POSTed to a temp area, [instance]/tmp/fileupload, as multipart form data. To help with this, i used an MIT-licensed AS class called MultipartURLLoader (https://code.google.com/p/in-spirit/). I used Content-type=multipart/form-data; boundary=[boundary]. CQ seemed very picky about the contents of the form data. mine is set up like this:
file: [name of file]
Filename: [name of file]
[name of file]: [file data]
Upload: Submit Query
another POST is issued, with a move command, to move the image from the temp area to the directory created in step 1. the url is [instance]/tmp/fileupload, and Content-type=application/x-www-form-urlencoded; charset=UTF-8. The form data is set up like this:
/var/dam/[destination_path]/[filename]#MoveFrom: /tmp/fileupload/[filename]
charset: utf-8
repeat step 4 until we get a 200. when new destination folders are indeed created, the first POST to #MoveFrom usually results in a 500, saying the destination folder is not there. perhaps there's another way to ask CQ if the destination is ready? i don't know.
we now need to publish the file, but first we issue a series of GETs on it to ensure it's there, with this url: [instance]/content/dam/[destination]/[filename].assets.json. once it's there, CQ will respond with some JSON that we use next.
check to see if the file has already been published. it may be the case that the user has already uploaded an image with the same name to the same location. the JSON response has a results node, which i check to see if it's 1. if it is, then i look at "pages[0].replication" to see if it has a node called "action". if it does, i see if the value is "ACTIVATE". if it is, it's already published. in every other case, i try to publish it.
POST a command to activate (publish it). the url is [instance]/bin/replicate.json. Content-type=application/x-www-form-urlencoded; charset=UTF-8. The form looks like this:
path: /content/dam/[destination]/[filename]
cmd: Activate
charset: utf-8
for my purposes, i wanted to then retrieve the published image to re-display it in my component. i waited for the 200 from the publish, then tried my GET. the url i used here had no port number, and no trailing slash: [instance:80]/content/dam/[destination]/[filename]. The first call almost always gave me a 404, so i kept trying until i got the 200.
that's it. i hope this is helpful to someone.
note: just saw that "charset" is in italics in the form specification. note that i'm using (underscore)charset(underscore).
I am currently using watir to do a web scraping of a website hiding all data from the usual HTML source. If I am not wrong, they are using XML and those AJAX technology to hide it. Firefox can see it but it is displayed via "DOM Source of selection".
Everything works fine but now I am looking for an equivalent tool as watir but everything need to be done without a browser. Everything need to be done in txt file.
In fact right now, watir is using my browser to emulate the page and return me the whole html code I am looking. I would like to the same but without the browser.
Is it possible ?
Thanks
Regards
Tak
Your best guess would be to use something like webscarab and capture the URLS of the AJAX requests your browser is doing.
That way, you can just grab the "important" data yourself by simulating those calls with any HTTP library
It is possible with a little Python coding.
I wrote a simple script to fetch locations of cargo offices.
First steps
Open the ajax page with Google Chrome for example, in Turkish but you can understand it.
http://www.yurticikargo.com/bilgi-servisleri/Sayfalar/en-yakin-sube.aspx
Press F12 to show bottom developer tools and navigate to Network tab.
Navigate XHR tab on the bottom.
Make an AJAX request by selecting an item in the first combobox. And go to Headers Tab
You will GetTownByCity on left pane, click it and inspect it.
Request URL: (...)/_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-
sswservices.aspx/GetTownByCity
Request Method:POST
Status Code:200 OK
In the Request Payload tree item you will see
Request Payload :{cityId:34}
header.
This will guide us to implement a python code.
Lets do it.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests
import json
# import simplejson as json
baseUrl = 'http://www.yurticikargo.com/'
ajaxRoot = '_layouts/ArikanliHolding.YurticiKargo.WebSite/'
getTown = 'ajaxproxy-sswservices.aspx/GetTownByCity'
urlGetTown = baseUrl + ajaxRoot + getTown
headers = {'content-type': 'application/json','encoding':'utf-8'} # We are sending JSON headers, equivalent to Python dictionary
for plaka in range(1,82): # Because Turkiye has number plates from 1 to 81
payload = {'cityId':plaka}
r = requests.post(url, data=json.dumps(payload), headers=headers)
data = r.json() # Returning data is in JSON format, if you need HTML use r.content()
# ... Process the fetched data with JSON parser,
# If HTML format, Beautiful Soup, Lxml, or etc...
Note that this code is a part of my working code and it is written on the fly, the most important is I did not test it. It may require small modifications to run it.