Hey guys I am pulling in a Vehicle Feed to an autodealer website for one of our clients. Every night at midnight(ish) the new XML file is uploaded to our FTP and it overwrites the current one. Currently he has two Identical websites and the file needs to be uploaded to both, I was looking into setting it up so both websites can use the same XML file so we can cut down on the risk of errors and for convince.
Pulling the file works great, both websites can read the XML file and have no issues displaying the inventory. The issue comes in when I try to display the date the file was last updated. I created a small snippet that reads the date the file was updated and displays "Last Update: and the date" but when I try and reference a non-local file I get a error that says "URI formats are not supported". Does anyone know of a way to do this or if its even possible?
what it currently is
FileInfo fileInfo = new FileInfo(Server.MapPath("~/feed/VEHICLES.XML"));
DateTime timeOfCreation = fileInfo.LastWriteTime;
what i tried
FileInfo fileInfo = new FileInfo("http://www.autodealername.com/feed/VEHICLES.XML");
DateTime timeOfCreation = fileInfo.LastWriteTime;
this was no good
This can be done via FTP, since you're using it already.
http://msdn.microsoft.com/en-us/library/system.net.ftpwebresponse.lastmodified.aspx
FileInfo uses information from the underlying file system which isn't available over HTTP. You'll need to think of some other way.
if you load the file in this way:
FileInfo fileInfo = new FileInfo("http://www.autodealername.com /feed/VEHICLES.XML");
most likely the file is retrieved to you by IIS or the webserver on that domain/site and this is not the same as opening the file from the file system directly.
I think you have two alternatives at least:
open the file from a network share like \\machinename\ShareName\FileName;
create a service endpoint on the remote server (WCF or XML web service) which gets in a file name and returns the information you need;
You can try using a WebRequest using the HEAD method and look for the Last-Modified header.
Here's the code I used...
var web = WebRequest.Create("http://cdn.sstatic.net/stackoverflow/img/sprites.png?v=4") as HttpWebRequest;
web.Method = "HEAD";
var response = web.GetResponse();
var lastModified = DateTime.Parse(response.Headers["last-modified"]);
Console.WriteLine(lastModified);
Here's what the http response looks like (from Fiddler)...
HTTP/1.1 200 OK
Server: nginx/0.8.36
Date: Wed, 23 Nov 2011 17:37:44 GMT
Content-Type: image/png
Connection: keep-alive
Cache-Control: max-age=604800
Last-Modified: Tue, 06 Sep 2011 21:44:29 GMT
ETag: "6237328de6ccc1:0"
Content-Length: 19706
X-Cache: HIT
Accept-Ranges: bytes
You could also add the updated field to the feed so you can get the last time it was updated from the feed itself.
RSS pubDate:
http://www.w3schools.com/rss/rss_tag_pubdate.asp
<?xml version="1.0" encoding="ISO-8859-1" ?>
<rss version="2.0">
<channel>
<title>W3Schools Home Page</title>
<link>http://www.w3schools.com</link>
<description>Free web building tutorials</description>
<!-- YOU COULD USE THIS -->
<pubDate>Thu, 27 Apr 2006</pubDate>
<item>
<title>RSS Tutorial</title>
<link>http://www.w3schools.com/rss</link>
<description>New RSS tutorial on W3Schools</description>
</item>
</channel>
</rss>
Atom updated:
http://www.atomenabled.org/developers/syndication/atom-format-spec.php#rfc.section.1.1
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>Example Feed</title>
<link href="http://example.org/"/>
<!-- YOU COULD USE THIS -->
<updated>2003-12-13T18:30:02Z</updated>
<author>
<name>John Doe</name>
</author>
<id>urn:uuid:60a76c80-d399-11d9-b93C-0003939e0af6</id>
<entry>
<title>Atom-Powered Robots Run Amok</title>
<link href="http://example.org/2003/12/13/atom03"/>
<id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
<updated>2003-12-13T18:30:02Z</updated>
<summary>Some text.</summary>
</entry>
</feed>
Maybe try using the FileSystemWatcher Class, which can notify you when a file was changed, modified, etc. Take a look at it.
Good luck!
Related
When I'm running my app on localhost, I get 2 warnings concerning MIME type. This is one of them:
Resource interpreted as Stylesheet but transferred with MIME type text/html: "http://localhost:3000/BootstrapEssentials/bootstrap.css".
The other warning is identical with a different file. Both files are in my working directory. So far, I have been to these similar questions but they haven't helped:
Resource interpreted as stylesheet but transferred with MIME type text/html (seems not related with web server)
Chrome says "Resource interpreted as script but transferred with MIME type text/plain.", what gives?
Resource interpreted as stylesheet but transferred with MIME type text/html
Originally I was trying to use this line:
<link rel="stylesheet" href="/BootstrapEssentials/bootstrap.css">
I have since added in the type field:
<link rel="stylesheet" href="/BootstrapEssentials/bootstrap.css" type="text/css">
but that didn't do anything. I also have used the JavaScript Console to see that in the response header it has content-type: text/html; charset=utf-8 and I believe that if I can change that to being content-type: text/css; charset=utf-8 then everything will be fine but I can't find how to do that either.
As per the Meteor docs:
All files inside a top-level directory called public/ are served as-is to the client. When referencing these assets, do not include public/ in the URL, write the URL as if they were all in the top level. For example, reference public/bg.png as /bg.png. This is the best place for favicon.ico, robots.txt, and similar files.
Moving your stylesheets to a directory in public should do the trick!
I'm trying to scrape images this site:
http://mis.historiska.se/mis/sok/bild.asp?uid=336358&g=1
The site also have the option to download different sizes, like big image here:
http://catview.historiska.se/catview/media/highres/336358
I have no problem downloading manual, scraping the image, or even scraping the url, but the image and url is missing the image extension.
I need to scrape the full url with filename and extension., NOT the actual image.
The proper way to do this would be to check the headers after making a request to the given url for the filename and extension. A simple curl request to the given url gives me the following response:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: image/jpeg
Content-Length: 569050
Date: Wed, 20 Jan 2016 15:33:49 GMT
The best way to guess the file extension would be to just check "Content-Type" header. Similarly, in order to get the filename, we'd be using the "Content-Disposition" header which need not necessarily be provided in the headers in which case we'll need to guess the filename from the URL.
A simple python snippet for guessing extension would be as follows:
import requests
import mimetypes
resp = requests.get(url)
content_type = resp.headers['content-type']
ext = mimetypes.guess_extension(content_type)
This is my html:
<!DOCTYPE html>
<html>
<head>
<meta http-equiv = "Content-Language" content = "en"/>
<meta http-equiv = "Content-Type" content="text/html; charset=utf-8">
<link rel="stylesheet" type="text/css" href="/xxx/app/www/style.css">
Now when I load the page, Safari's inspector gives me:
Resource interpreted as stylesheet but transferred with MIME type text/html
I'm working on localhost with MAMP. I'm not sure what to modify to get this to work. I read that this might be a server problem of some kind.
How do I solve it?
Create an .htaccess file into your root folder (or update the existing one) with this line inside
AddType text/css .css
this will tell apache to send the right content-type header for .css file
There is possibility that not just css resources were interpreted wrongly. It maybe a problem of your webserver configuration.
The best way to check is to go to Chrome -> Network tab and check responses section for each request.
Also you can run $ curl -I http://path_to_file_or_page with terminal and have a look at response, you should see it in following format, e.g. Content-Type: image/jpeg
So, if it will occur that webserver (apache) does it for all filesthen you can try to solve the problem in following way:
Check your /etc/apache2/apache2.conf
If there are any SetHandler application/x-httpd-php line, try to comment it and then reload your apache by $ sudo system apache2 reload
Let us know if the problem is still there.
It could be that your web server is properly configured but it really is returning HTML instead of CSS.
Check that you have the correct path, spelling, case, etc. Hit the URL directly and see what comes back. Your web server might be responding with an error page (HTML) and a 200-Ok instead of a 404-Not Found.
I'm not saying this ever happened to me...
I've spent quite a bit of time on this so here's where I'm stuck.
I'm using the debug player 10.1 to get an XMLA request from:
http://localhost/dir/blah.swf
to:
http://localhost/olapbin/msblah.dll
This worked fine in the filesystem, but now its on an IIS7 web server.
After a lot of fiddling with the crossdomain.xml file I settled on:
<?xml version="1.0"?>
<cross-domain-policy>
<site-control permitted-cross-domain-policies="master-only"/>
<allow-access-from domain="*" to-ports="*" />
<allow-http-request-headers-from domain="*" headers="*"/>
</cross-domain-policy>
which is placed in:
http://localhost/crossdomain.xml
and read with:
Security.loadPolicyFile("http://localhost:80/crossdomain.xml");
I setup Policy file logging (which helped come up with the above file)
and on IE8 its all working just fine. I get:
OK: Root-level SWF loaded: http://127.0.0.1/akts/ThinSlicerRunner.swf
OK: Policy file accepted: http://localhost/crossdomain.xml
OK: Searching for <allow-access-from> in policy files to authorize data loading from resource at http://localhost/olapbin/msmdpump.dll by requestor from http://127.0.0.1/akts/ThinSlicerRunner.swf
OK: Searching for <allow-http-request-headers-from> in policy files to authorize header sending to URL http://localhost/olapbin/msmdpump.dll by requestor from http://127.0.0.1/akts/ThinSlicerRunner.swf
OK: Request for resource at http://localhost/olapbin/msmdpump.dll by requestor from http://127.0.0.1/akts/ThinSlicerRunner.swf is permitted due to policy file at http://localhost/crossdomain.xml
On Chrome and Firefox I just get:
OK: Root-level SWF loaded: http://localhost/akts/ThinSlicerRunner.swf
OK: Policy file accepted: http://localhost/crossdomain.xml
and nothing else... no attempts to authorize the httpservice requests.
In the main flex error log I get:
*** Security Sandbox Violation ***
Connection to
http://localhost/olapbin/msmdpump.dll
halted - not permitted from http://localhost/akts/ThinSlicerRunner.swf
Which doesn't appear when I run the same thing from IE8.
Any idea what's going on ??
AS REQUESTED... MORE CODE
Main sending request:
var connection:TsConnection = this.__connection;
var token:AsyncToken = new AsyncToken(null);
connection.service.request = this.__curSoapRequest;
var actualToken:AsyncToken = connection.service.send();
__tokenArr.push(actualToken);
var responder:AsyncResponder = new AsyncResponder(resultHandler, faultHandler, actualToken);
__responderArr.push(responder);
actualToken.addResponder(responder);
Connection object highlights:
public function init():void {
//Initialize the service object needed to query the server
this.__service = new HTTPService;
this.__service.method = "POST";
this.__service.contentType = "application/xml";
this.__service.resultFormat = "e4x";
this.__service.headers = getHeaders();
this.__service.url = this.__model.xmlaUrl;
this.__initialized = true;
}
public function get service():HTTPService {
return this.__service;
}
private function getHeaders():Object {
var o:Object = {};
o["SOAPAction"] = '"urn:schemas-microsoft-com:xml-analysis:Discover"';
o["Content-Type"] = "text/xml";
return o;
}
Thanks for your help ... hope this helps others when fixed. ;-)
Shaun
http://www.vidgridz.com/
Thanks for everyone's answers. It was indeed able to be solved in the code
even if it wasn't exactly a true coding problem.
Here is the xml data file I was reading the configuration details from:
<tsConnection>
<dataSource>megan</dataSource>
<database>Adventure Works DW 2008</database>
<cube>Adventure Works</cube>
<xmlaUrl><![CDATA[
http://localhost/olapbin/msmdpump.dll
]]></xmlaUrl>
</tsConnection>
Now on the "localTrusted" or "localWithNetworking" setup, this was working just fine.
It also works on the IE8 Flash player even in "remote".
However, what was happening was that the xmlaUrl was being read as:
\n\rhttp://localhost/olapbin/msmdpump.dll
(with the newline and carriage return at the start)
This is was what was confusing the domain checking and throwing a Sandbox Violation
when run in the "remote" security sandbox.
Of course, my xml should have been better, and maybe put in some ignore white
space processing in the code, but still its quite some bizarre, inconsistent
behavior from the Flash player code in Netscape compatible browsers (10.1.x).
So the final, working solution looks like this:
<tsConnection>
<dataSource>megan</dataSource>
<database>Adventure Works DW 2008</database>
<cube>Adventure Works</cube>
<xmlaUrl><![CDATA[http://localhost/olapbin/msmdpump.dll]]></xmlaUrl>
</tsConnection>
I did become a crossdomain.xml expert in the process though. ;-)
Although, now I don't need the file at all.
Bear it in mind if you see some crazy unexplained Sandbox Violations, check for
white space in your service url.
If your DLL backend service and SWF are served from the same domain, it should be allowed. Nothing in the crossdomain.xml file should apply. You should not have to load the crossdomain file manually either. It sounds like that is what you're trying to do.
I suspect something else is going on with your code.
I am trying to get data from my server, used RemoteObject to accomplish it.
When I run the application on my localhost it works great but when iam using it on my server i get a Channel.Security.Error(Security Error accessing URL).
On the server side logs there is a mention about cross domain .
77.127.194.4 - - [23/Oct/2008 21:15:11] "GET /crossdomain.xml HTTP/1.1" 501
Any one encountered the same problem ? any idea ?
Have you tried to add to your crossdomain.xml (where your fetching the stuff from) this:
<?xml version="1.0"?>
<!DOCTYPE cross-domain-policy SYSTEM "http://www.YOUR_FRAME_WORK_CROSSDOMAIN_POLICY.com/xml/dtds/cross-domain-policy.dtd">
<cross-domain-policy>
<site-control permitted-cross-domain-policies="all"/>
<allow-access-from domain="*.YOUR_SITE_GOES_HERE.com" secure="false" />
<allow-access-from domain="*.YOUR_SITE_GOES_HERE.com" secure="false" />
</cross-domain-policy>
The stuff in capslock you'll probably have to change to fit your framework. For example i copied that from the one i use with macromedia flash. Instead of "www.YOUR_FRAME_WORK_CROSSDOMAIN_POLICY.com/..." i normaly have "www.macromedia.com/xml/dtds/...
I'm not sure but try to investigate that, it's probably your problem. For cross-domain, you normaly need to add to the server side, where your fecthing stuff from, permission for other sites to get it.
I have found the solution. You are right about crossdomain.xml file, but unfortunately, the Python SimpleXMLRPCServer library does not support the GET method by default, so we need to implement this.
from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler
class ExtendedXMLRPCRequestHandler(SimpleXMLRPCRequestHandler):
def do_GET(self):
#only allow a request for the crossdomain file
if self.path != '/crossdomain.xml':
self.send_response(403)
self.log_request(403)
return
#open the crossdomain file and read its contents
response = open('crossdomain.xml', 'r').read()
#write the data to the socket along with valid HTTP headers
self.send_response(200)
self.send_header("Content-type", "text/xml")
self.send_header("Content-length", str(len(response)))
self.end_headers()
self.wfile.write(response)
self.log_request(200)