I have a coding problem regarding Python 3.5 web clawing.
I try to use 'requests.get' to extract the real link from 'http://www.baidu.com/link?url=ePp1pCIHlDpkuhgOrvIrT3XeWQ5IRp3k0P8knV3tH0QNyeA042ZtaW6DHomhrl_aUXOaQvMBu8UmDjySGFD2qCsHHtf1pBbAq-e2jpWuUd3'. An example of the code is like below:
import requests
response = requests.get('http://www.baidu.com/link?url=ePp1pCIHlDpkuhgOrvIrT3XeWQ5IRp3k0P8knV3tH0QNyeA042ZtaW6DHomhrl_aUXOaQvMBu8UmDjySGFD2qCsHHtf1pBbAq-e2jpWuUd3')
c = response.url
I expected that 'c' should be 'caifu.cnstock.com/fortune/sft_jj/tjj_yndt/201605/3787477.htm'. (I remove http:// from the link as I can't post two links in one question.)
However, it doesn't work, and keeps return me the same link as I putted in.
Can anyone help on this. Many thanks in advance.
#
Thanks a lot to Charlie.
I have found out the solution. I first use .content.decode to read the response history, but that will be mixed up with many irrelevant info. I then use .findall to extract the redirect url from the history, which should be the first url displayed in the response history. Then, I use requests.get to retrieve the info. Below is the code:
rep1 = requests.get(url)
cont = rep1.content.decode('utf-8')
extract_cont = re.findall('"([^"]*)"', cont)
redir_url = extract_cont[0]
rep = requests.get(redir_url)
You may consider looking into the response headers for a 'location' header.
response.headers['location']
You may also consider looking at the response history, which contains a response for each response instance in a chain of redirects
response.history
Your sample URL doesn't redirect; The response is a 200 and then it uses a JavaScript window.location change. The requests library won't support this type of redirect.
<script>window.location.replace("http://caifu.cnstock.com/fortune/sft_jj/tjj_yndt/201605/3787477.htm")</script>
<noscript><META http-equiv="refresh" content="0;URL='http://caifu.cnstock.com/fortune/sft_jj/tjj_yndt/201605/3787477.htm'"></noscript>
If you know you will always be using this one service, you could parse the response, maybe using regex.
If you don't know what service will always be used and also want to handle every possible situation, you might need to instantiate a WebKit instance or something and somehow try to determine when it finally finishes. I'm sure there's a page load complete event which you could use, but you still might have pages that do a window.location change after the page is loaded using a timer. This will be very heavyweight and still not cover every conceivable type of redirect.
I recommend starting with writing a special handler for each type of edge case and fallback on a default handler that just looks at the response.url. As new edge cases come up, write new handlers. It's kind of the 'trial and error' approach.
I wrote a handler to be used within SharePoint that will generate a JSON string from a given parameter in the query-string. This JSON string will then be used by a JS function to display the relevant data in HTML to the client, however, the call is somewhat costly and would like to cache the output once.
The handler currently caters for everything regarding OUTPUT cache and tested within an application bares fruits, however, I'm baffled by the fact that, specifically in SharePoint (2007) the "If-Modified-Since" header attribute never appears, basically it always comes back as null.
I found blog-on-blog that discuss this in length in regards to images, and include files but I can't find anything specific regarding this with pages (ASPX, AXD, ASHX) and the handler self.
My only assumption here is the fact that I'm using an AXD file, which is not directly supported by OUTPUT cache by default?
The code looks something like:
bool isModifiedSinceLast = (context.Request.Headers.Get("If-Modified-Since") != null)
: true
? false;
if (!isModifiedSinceLast)
{
context.Response.Headers.AppendHeader("If-Modified-Since", Guid.NewGuid());
}
else
{
// complete the call from cache
}
Thanks,
Eric
for some reason, IE6/7 is caching the ajax call that returns a json result set back.
My page makes the call, and returns a json result which I then inject into the page.
How can I force IE6/7 to make this call and not use a cached return value?
You might want to add
Cache-Control: no-cache
to your HTML response headers when you're serving the JSON to tell the browser to not to cache the response.
In ASP.NET (or ASP.NET MVC) you can do it like this:
Response.Headers.Add("Cache-Control", "no-cache");
you can change your settings in ie, but the problem most likely lies on your server. You can't go out and change all your users' browser settings. But if you want to at least check it on your browser, go to Internet Options->General (Tab)->Browsing History(section)->Settings (button)->"Every time I visit the webpage"
Make sure you set it back, though, at some point.
To fix it on the server, have a look at http://www.mnot.net/cache_docs/
Using curl (w/ cygwin) for debugging is your great way to figure out what's actually being sent across the wire.
If cache-control doesn't work for you (see DrJokepu's answer), according to the spec the content from any URL with a query string should be non-cacheable, so you might append a pointless query parameter to your request URL. The value doesn't matter, but if you really want to be thorough you can append the epoch value, e.g.:
var url = "myrealurl?x=" + (new Date()).getTime();
But this is a hack; really this should be solved with proper caching headers at the server end.
In the controller action that returns a JsonResult, you need to specify in your headers to avoid caching:
ControllerContext.HttpContext.Response.AddHeader("Cache-Control", "no-cache");
My app generates PDFs for user consumption. The "Content-Disposition" http header is set as mentioned here. This is set to "inline; filename=foo.pdf", which should be enough for Acrobat to give "foo.pdf" as the filename when saving the pdf.
However, upon clicking the "Save" button in the browser-embedded Acrobat, the default name to save is not that filename but instead the URL with slashes changed to underscores. Huge and ugly. Is there a way to affect this default filename in Adobe?
There IS a query string in the URLs, and this is non-negotiable. This may be significant, but adding a "&foo=/title.pdf" to the end of the URL doesn't affect the default filename.
Update 2: I've tried both
content-disposition inline; filename=foo.pdf
Content-Type application/pdf; filename=foo.pdf
and
content-disposition inline; filename=foo.pdf
Content-Type application/pdf; name=foo.pdf
(as verified through Firebug) Sadly, neither worked.
A sample url is
/bar/sessions/958d8a22-0/views/1493881172/export?format=application/pdf&no-attachment=true
which translates to a default Acrobat save as filename of
http___localhost_bar_sessions_958d8a22-0_views_1493881172_export_format=application_pdf&no-attachment=true.pdf
Update 3: Julian Reschke brings actual insight and rigor to this case. Please upvote his answer.
This seems to be broken in FF (https://bugzilla.mozilla.org/show_bug.cgi?id=433613) and IE but work in Opera, Safari, and Chrome. http://greenbytes.de/tech/tc2231/#inlwithasciifilenamepdf
Part of the problem is that the relevant RFC 2183 doesn't really state what to do with a disposition type of "inline" and a filename.
Also, as far as I can tell, the only UA that actually uses the filename for type=inline is Firefox (see test case).
Finally, it's not obvious that the plugin API actually makes that information available (maybe someboy familiar with the API can elaborate).
That being said, I have sent a pointer to this question to an Adobe person; maybe the right people will have a look.
Related: see attempt to clarify Content-Disposition in HTTP in draft-reschke-rfc2183-in-http -- this is early work in progress, feedback appreciated.
Update: I have added a test case, which seems to indicate that the Acrobat reader plugin doesn't use the response headers (in Firefox), although the plugin API provides access to them.
Set the file name in ContentType as well. This should solve the problem.
context.Response.ContentType = "application/pdf; name=" + fileName;
// the usual stuff
context.Response.AddHeader("content-disposition", "inline; filename=" + fileName);
After you set content-disposition header, also add content-length header, then use binarywrite to stream the PDF.
context.Response.AddHeader("Content-Length", fileBytes.Length.ToString());
context.Response.BinaryWrite(fileBytes);
Like you, I tried and tried to get this to work. Finally I gave up on this idea, and just opted for a workaround.
I'm using ASP.NET MVC Framework, so I modified my routes for that controller/action to make sure that the served up PDF file is the last part of the location portion of the URI (before the query string), and pass everything else in the query string.
Eg:
Old URI:
http://server/app/report/showpdf?param1=foo¶m2=bar&filename=myreport.pdf
New URI:
http://server/app/report/showpdf/myreport.pdf?param1=foo¶m2=bar
The resulting header looks exactly like what you've described (content-type is application/pdf, disposition is inline, filename is uselessly part of the header). Acrobat shows it in the browser window (no save as dialog) and the filename that is auto-populated if a user clicks the Acrobat Save button is the report filename.
A few considerations:
In order for the filenames to look decent, they shouldn't have any escaped characters (ie, no spaces, etc)... which is a bit limiting. My filenames are auto-generated in this case, and before had spaces in them, which were showing up as '%20's in the resulting save dialog filename. I just replaced the spaces with underscores, and that worked out.
This is by no names the best solution, but it does work. It also means that you have to have the filename available to make it part of the original URI, which might mess with your program's workflow. If it's currently being generated or retrieved from a database during the server-side call that generates the PDF, you might need to move the code that generates the filename to javascript as part of a form submission or if it comes from a database make it a quick ajax call to get the filename when building the URL that results in the inlined PDF.
If you're taking the filename from a user input on a form, then that should be validated not to contain escaped characters, which will annoy users.
Hope that helps.
Try placing the file name at the end of the URL, before any other parameters. This worked for me.
http://www.setasign.de/support/tips-and-tricks/filename-in-browser-plugin/
In ASP.NET 2.0 change the URL from
http://www. server.com/DocServe.aspx?DocId=XXXXXXX
to
http://www. server.com/DocServe.aspx/MySaveAsFileName?DocId=XXXXXXX
This works for Acrobat 8 and the default SaveAs filename is now MySaveAsFileName.pdf.
However, you have to restrict the allowed characters in MySaveAsFileName (no periods, etc.).
Apache's mod_rewrite can solve this.
I have a web service with an endpoint at /foo/getDoc.service. Of course Acrobat will save files as getDoc.pdf. I added the following lines in apache.conf:
LoadModule RewriteModule modules/mod_rewrite.so
RewriteEngine on
RewriteRule ^/foo/getDoc/(.*)$ /foo/getDoc.service [P,NE]
Now when I request /foo/getDoc/filename.pdf?bar&qux, it gets internally rewritten to /foo/getDoc.service?bar&qux, so I'm hitting the correct endpoint of the web service, but Acrobat thinks it will save my file as filename.pdf.
If you use asp.net, you can control pdf filename through page (url) file name.
As other users wrote, Acrobat is a bit s... when it choose the pdf file name when you press "save" button: it takes the page name, removes the extension and add ".pdf".
So /foo/bar/GetMyPdf.aspx gives GetMyPdf.pdf.
The only solution I found is to manage "dynamic" page names through an asp.net handler:
create a class that implements IHttpHandler
map an handler in web.config bounded to the class
Mapping1: all pages have a common radix (MyDocument_):
<httpHandlers>
<add verb="*" path="MyDocument_*.ashx" type="ITextMiscWeb.MyDocumentHandler"/>
Mapping2: completely free file name (need a folder in path):
<add verb="*" path="/CustomName/*.ashx" type="ITextMiscWeb.MyDocumentHandler"/>
Some tips here (the pdf is dynamically created using iTextSharp):
http://fhtino.blogspot.com/2006/11/how-to-show-or-download-pdf-file-from.html
Instead of attachment you can try inline:
Response.AddHeader("content-disposition", "inline;filename=MyFile.pdf");
I used inline in a previous web application that generated Crystal Reports output into PDF and sent that in browser to the user.
File download dialog (PDF) with save and open option
Points To Remember:
Return Stream with correct array size from service
Read the byte arrary from stream with correct byte length on the basis of stream length.
set correct contenttype
Here is the code for read stream and open the File download dialog for PDF file
private void DownloadSharePointDocument()
{
Uri uriAddress = new Uri("http://hyddlf5187:900/SharePointDownloadService/FulfillmentDownload.svc/GetDocumentByID/1/drmfree/");
HttpWebRequest req = WebRequest.Create(uriAddress) as HttpWebRequest;
// Get response
using (HttpWebResponse httpWebResponse = req.GetResponse() as HttpWebResponse)
{
Stream stream = httpWebResponse.GetResponseStream();
int byteCount = Convert.ToInt32(httpWebResponse.ContentLength);
byte[] Buffer1 = new byte[byteCount];
using (BinaryReader reader = new BinaryReader(stream))
{
Buffer1 = reader.ReadBytes(byteCount);
}
Response.Clear();
Response.ClearHeaders();
// set the content type to PDF
Response.ContentType = "application/pdf";
Response.AddHeader("Content-Disposition", "attachment;filename=Filename.pdf");
Response.Buffer = true;
Response.BinaryWrite(Buffer1);
Response.Flush();
// Response.End();
}
}
I believe this has already been mentioned in one flavor or another but I'll try and state it in my own words.
Rather than this:
/bar/sessions/958d8a22-0/views/1493881172/export?format=application/pdf&no-attachment=true
I use this:
/bar/sessions/958d8a22-0/views/1493881172/NameThatIWantPDFToBe.pdf?GeneratePDF=1
Rather than having "export" process the request, when a request comes in, I look in the URL for GeneratePDF=1. If found, I run whatever code was running in "export" rather than allowing my system to attempt to search and serve a PDF in the location /bar/sessions/958d8a22-0/views/1493881172/NameThatIWantPDFToBe.pdf. If GeneratePDF is not found in the URL, I simply transmit the file requested. (note that I can't simply redirect to the file requested - or else I'd end up in an endless loop)
You could always have two links. One that opens the document inside the browser, and another to download it (using an incorrect content type). This is what Gmail does.
For anyone still looking at this, I used the solution found here and it worked wonderfully. Thanks Fabrizio!
The way I solved this (with PHP) is as follows:
Suppose your URL is SomeScript.php?id=ID&data=DATA and the file you want to use is TEST.pdf.
Change the URL to SomeScript.php/id/ID/data/DATA/EXT/TEST.pdf.
It's important that the last parameter is the file name you want Adobe to use (the 'EXT' can be about anything). Make sure there are no special chars in the above string, BTW.
Now, at the top of SomeScript.php, add:
$_REQUEST = MakeFriendlyURI( $_SERVER['PHP\_SELF'], $_SERVER['SCRIPT_FILENAME']);
Then add this function to SomeScript.php (or your function library):
function MakeFriendlyURI($URI, $ScriptName) {
/* Need to remove everything up to the script name */
$MyName = '/^.*'.preg_quote(basename($ScriptName)."/", '/').'/';
$Str = preg_replace($MyName,'',$URI);
$RequestArray = array();
/* Breaks down like this
0 1 2 3 4 5
PARAM1/VAL1/PARAM2/VAL2/PARAM3/VAL3
*/
$tmp = explode('/',$Str);
/* Ok so build an associative array with Key->value
This way it can be returned back to $_REQUEST or $_GET
*/
for ($i=0;$i < count($tmp); $i = $i+2){
$RequestArray[$tmp[$i]] = $tmp[$i+1];
}
return $RequestArray;
}//EO MakeFriendlyURI
Now $_REQUEST (or $_GET if you prefer) is accessed like normal $_REQUEST['id'], $_REQUEST['data'], etc.
And Adobe will use your desired file name as the default save as or email info when you send it inline.
I was redirected here because i have the same problem. I also tried Troy Howard's workaround but it is doesn't seem to work.
The approach I did on this one is to NO LONGER use response object to write the file on the fly. Since the PDF is already existing on the server, what i did was to redirect my page pointing to that PDF file. Works great.
http://forums.asp.net/t/143631.aspx
I hope my vague explanation gave you an idea.
Credits to Vivek.
Nginx
location /file.pdf
{
# more_set_headers "Content-Type: application/pdf; name=save_as_file.pdf";
add_header Content-Disposition "inline; filename=save_as_file.pdf";
alias /var/www/file.pdf;
}
Check with
curl -I https://example.com/file.pdf
Firefox 62.0b5 (64-bit): OK.
Chrome 67.0.3396.99 (64-Bit): OK.
IE 11: No comment.
Try this, if your executable is "get.cgi"
http://server,org/get.cgi/filename.pdf?file=filename.pdf
Yes, it's completely insane. There is no file called "filename.pdf" on the server, there is directory at all under the executable get.cgi.
But it seems to work. The server ignores the filename.pdf and the pdf reader ignores the "get.cgi"
Dan
I am doing an e-commerce solution in ASP.NET which uses PayPal's Website Payments Standard service. Together with that I use a service they offer (Payment Data Transfer) that sends you back order information after a user has completed a payment. The final thing I need to do is to parse the POST request from them and persist the info in it. The HTTP request's content is in this form :
SUCCESS
first_name=Jane+Doe
last_name=Smith
payment_status=Completed
payer_email=janedoesmith%40hotmail.com
payment_gross=3.99
mc_currency=USD
custom=For+the+purchase+of+the+rare+book+Green+Eggs+%26+Ham
Basically I want to parse this information and do something meaningful, like send it through e-mail or save it in DB. My question is what is the right approach to do parsing raw HTTP data in ASP.NET, not how the parsing itself is done.
Something like this placed in your onload event.
if (Request.RequestType == "POST")
{
using (StreamReader sr = new StreamReader(Request.InputStream))
{
if (sr.ReadLine() == "SUCCESS")
{
/* Do your parsing here */
}
}
}
Mind you that they might want some special sort of response to (ie; not your full webpage), so you might do something like this after you're done parsing.
Response.Clear();
Response.ContentType = "text/plain";
Response.Write("Thanks!");
Response.End();
Update: this should be done in a Generic Handler (.ashx) file in order to avoid a great deal of overhead from the page model. Check out this article for more information about .ashx files
Use an IHttpHandler and avoid the Page model overhead (which you don't need), but use Request.Form to get the values so you don't have to parse name value pairs yourself. Just pretend you're in PHP or Classic ASP (or ASP.NET MVC, for that matter). ;)
I'd strongly recommend saving each request to some file.
This way, you can always go back to the actual contents of it later. You can thank me later, when you find that hostile-endian, koi-8 encoded, [...], whatever it was that stumped your parser...
Well if the incoming data is in a standard form encoded POST format, then using the Request.Form array will give you all the data in a nice to handle manner.
If not then I can't see any way other than using Request.InputStream.
If I'm reading your question right, I think you're looking for the InputStream property on the Request object. Keep in mind that this is a firehose stream, so you can't reset it.