This is both a strategy and a technical question, I'm building a web posting mechanism and I will need to store a lot of HTML posts (discussions, comments etc.)
I'm thinking about saving all my HTML posts into database as a ZIP compressed stream (instead of plain text or XML) in order to save space and increase security by encrypting those ZIP data steams, so it will be saved to the database compressed (hopefully close to 90% smaller) and secure. (it does not need to be searchable, I'm going to create the search index myself out of the content of each post)
I want to deliver the ZIP object to the web page/cache and then have the client side unzip the stream and render the HTML that it represent.
This is a Microsoft based MVC web site (c#)
I'm trying to figure out reasons not to do it... other than performance, can anyone pinpoint any other issues with doing something like that?
Also, is there any recommended libraries or built-in ones that I should use for better performance - that both server side and client side can understand (zip and unzip with encryption key/password)?
Thanks in advance.
In normal operation, http allows to send the html in a gzipped stream. The webserver compresses the data and sets the corresponding header. The client then unzips transparently.
You simply have to make sure to set the correct header and not have the webserver zip again the already zipped stream.
I see a major drawbacks :
You cannot alter the data. That means you cannot add the code for your template nor link between the pages.
I don't think this is a good approach. Store your data as you like and decompress it on the server.
Related
I am using a Azure cloud storage solution, and as such, each file has it's own URL. I need to allow users to download files such as PDFs from our ASP .Net website. Currently, we are using Response.TransmitFile to send the file to the client's browser, but this requires that we fetch the data from the cloud storage and then send it to the client (seems like an inefficient way to do it).
I'm wondering if we could not just have a direct link to the file, and if so, how would this differ from the Response.TrnasmitFile method? That is, without the TransmitFile method, we cannot set the Content-type header, etc... How does that effect anything?
Thanks
Usually I stay away from using Response.TransmitFile as it does require that you fetch the file and then stream it down to the client.
The only times I have used it was to protect files and only serve them to users that had permission to access them instead of just linking directly to the file.
If the files you are serving are public, then I would recommend just linking to them. If all you need is to set the Content-Type header, then simply make sure the .pdf extension is mapped to the correct MIME type (application/pdf).
I want to at least prevent normal users to download my flash video.
What's the best way to do it?
Create a httphandler, add a token (e.g. timeid), set the cache control to no-cache so that only the users with correct token can view the correct video. Is that feasible?
It is the requirement from client that the video should not be downloaded by users and should be watched only in the particular website.
I want to know if this works:
http://www.somesite.com/video.swf?time=1248319067
Server will generate a token(time in the above example) so that user can only have one request to this link. If the user wants to watch the video again, he needs to go to our website to get the token again. Is this okay to prevent novices from downloading?
I can't download this flash video by the downloadHelper firefox plugin:
http://news.bbc.co.uk/2/hi/americas/8164177.stm
Updated (13:49 pm 2009/07/23):
The above file can be downloaded using some video download software.
The video files of following Chinese sites are well protected (I can't download it using many video download software):
http://programme.tvb.com/drama/abrideforaride/video/
Do you know how it is done?
I dont think there is an easy way to stop people from getting your videos if they want them,
there are plenty of plugins for firefox that allow downloading from even youtube and many places. And i imagine those plugins would disable any attempt you made to hide your videos.
not too terribly different than taking an image from flicker, they put a clear gif image over the image that you want to view, so that when you right click and save you get "the shield" image, however can be defeated by the lowly print screen button.
if you want casual users from getting your file, use a flash control and buffer a minute or two of your videos and make that flash authenticate with the server to get those files. that seems reasonable to me
I don't think there really is an easy way to limit people from getting at it. Your sending them the video, that is how they are able to view it. Any user could just use FRAPS or a similar tool to copy the video from the screen as well.
If your worry is being copied and used elsewhere then you can watermark it or use a few other types of copy protection methods that will allow you to identify your work on other sites. If your worried about people copying it for personal use, then you really have no way of stopping it, you are sending it to them.
Edit: Due diligence would be to inform your customer of how easy it is to copy the work that they will be posting. Most clients have really no idea how easy it is.
This is how I like to tackle this issue.
This method works by creating a ticket to download the content over one http request...Another attempt to use the same ticket to download the content will fail, hence any extensions that attempt to download the content again or a user manually attempting to fail to do so, hence the flash player will be the only way to download the content. However there is one downfall for this approach, users will not be able to skip to a part of the video that has not been download...in some standard player implementation that may even stop the video from loading. Any ideas on this will be highly appreciated.
I begin by writing a PHP script that takes in a video_id, file_name, or a local path to your video file (Depending on the storage infrastructure of your video collection) in a GET request along with a unique hash value (a hard to guess and come up with probably generated with a secret key so it can be validated to be coming from our reciever (flash player), if the hacker send us a used hash or an invalid hash (does not satisfy our key), we will not send him the file). The PHP script then opens the video file and sends its content with the correct video mime type. for FLV the mime type is video/x-flv. It makes sure that once a unique hash has not been used before and is validly generated from your secret encryption key.
Then once the page with the flash player is loading we can give the .php file with the right get parameters as the video url to the video player. (If it is a prude player that only allows flv files you can always program your .htaccess file to parse .flv files as php script in the specific folder only, and rename your .php file as .flv and try your luck)...anyways...Also generate a hash key...perhaps you can take the servers current time and append it to a salt value such as another key known by both scripts, and encrypt this final concatenation with your secret key.
So once the video gateway php script will recieve a filename or hash key...it will decrypt the hash key and figure out if it is validly generated from teh sister script, and make sure not to send the video again to the same hash key...
For added security you can perhaps reset the secret key everyday using either a cronjob or bootstrap mechanism. To prevent duplicate use of hashkeys you can store them in a mysql database, file operations, or NOSQL (depending on your needs and infrastructure).
Make sure that the file is requested by the same user agent the hash key was generated for. In case the hacker trys to cURL or Wget your videos unused url before the flash player gets a chance to consume the hash key. In this case the hacker will have to imitate the browser's user agent or download the file using their command line tool as well...However please note that this is not your average champ.
It sounds like you need to add authorization and authentication.
You could put the flash video under a different folder in your ASP.Net application and add a web.config file in that folder to deny access to unauthorized users. For example:
Then you need to enable authentication for your website. The simplest method is forms authentication. A trivial example with hard coded username and password is provided here.
There is loads that you can do with the authentication framework in ASP.Net I suggest googling a bit.
The only way to do this is with a trusted client, DRM and an encrypted source.
Your player opens up a connection, the user has a connection to the stream, you perform some magic authentication with their token and then transmite the encrypted data to them.
If you don't do this then anyone can download your video and save it out.
However with all that aside, someone can run screen capture, then save your video and do it again. This is again where the DRM comes in as one of the key features of the DRM in windows clients is that the buffer cannot be sniffed as it's on the protected media pathway.
I guess its a question of how to protect your revenue but dealing with pirates is always going to be a problem for software devs no matter what their business is.
I have a solution that i'm gonna try for myself (as I have the same worries) but I know that it includes a lot of extra time and work...
Solution: using flash compress the video into an swf file. Before compressing add some AS code to the movie for authentication. suggestions for authentication:
1 test url
2 create a dedicated flash player that has handshake code checked by the video.swf
I like #2 better, and as an extra measure, you can overlay an id code over the video, so if someone captures the video using screen recording software, you'd at least be able to track the original source of the copied video.. and exact suitable retribution...
Simply you can't prevent it.
But..you can make it difficult.
Here some ideas come in my mind
1 First of all add your identifier to the video (always someone can download it)
2 The hard way... Add Ajax call back to server to check a random generated key that it will stored in the session every N seconds. After every post back clear the buffer of the player and start the video from were i was (using javascript).
Use again JavaScript prevent the video source from downloading by "view source".
3 Handle all your videos in urls like http://www.example.com/viewvideo/1 OR ../?id=1.
Add blank image overlay with transparent background.
Serve the original video and a blank video somewhere on the page with normal extension and style attribute "display:none". (will create problems to some download helpers)
4 Everytime you serve a video CHECK if the request is from a browser (ie check UserAgent)
5 Cookie with some random value combined with the id of the video. Check it client-side and server side and then serve the video.
6 On focusout event hide the video with javascript. put a resume button in the flash and leave the frame unchange (like pause but with no original video in buffer).
7 Combine those methods
these are random generated ideas,
not tested neither i say that guaranties no video downloading.
I have attempted two way to prevent the downloading but fails.
Using javascript to dynamically generate the object for flash.
Using the token idea proposed in the question.
What annoying me most is that a simple SAVE/AS from the firefox browser could easily bypass the tricks.
The only variable way so far is to using an empty swf file to load another swf file in. Combined with the token idea, it works.
in my answer you cant stop image/video theft but you can make harder for normal users but you can't make it harder for the programmers like us( i mean thiefs that knows little web programming) there are some tricks you can try:-
1.) Using flash as youtube and many others sites like http://www.funnenjoy.com does .
2.) Div overlaping or background pic setting (but users with little sense can easily save all resources by opening inspect element or other developer option).
3.) You can disable right click and specific keys like CTRL + S and others possibles with JAVASCRIPT but main drawback is that if user disable JAVASCRIPT our all tricks fail down.
4.)Save image in none online directories(if you have full access to web server) and read that files with server side languages like PHP every time when image / video is required and change image id time to time or create script that can automatically change ID after every access.
5.)Use .htaccess in apache to prevent linking of your images by others sites. you can use this site to automatically generate .htacess http://www.htaccesstools.com/hotlink-protection/
I am trying to design a system for something like this with ASP.net/C#.
The users pay for downloading some content (files- mp3s/PDFs,doc etc).I should be able to track the number of bytes downloaded by the user. If the number of bytes downloaded match the number of bytes on the server, I should set a flag in DB (telling that the download was successful and prevent them from downloading the file again/asking them to pay for the download again). If the download was incomplete, they should be able to download the file again without paying for it again(since the flag will not be set).
Is there any way to keep track of the number of bytes successfully downloaded by the client ?
Also when I see a file size in my WinXP machine, I see two sizes(size,size on disk). Which one should I consider ? And will it differ from one OS to another ?
You can easily measure data passed to the client in ASP.NET assuming you replace a direct IIS-controlled download with your own, which would go something like this:
while (context.Response.IsClientConnected) {
bytesRead = ReadFileChunkAsByteArrayWIthOffsetOrWhatever(buffer, offset);
context.Response.OutputStream.Write(buffer, 0, bytesRead);
context.Response.Flush();
offset += bytesRead;
if (bytesRead != bufferSize)
break;
}
It's complicated to make this 100% reliable from within ASP, but it can be done. You pretty much have to account for every possible failure point and react accordingly.
The problem though is still - as someone mentioned above - that it's impossible to know that the client received the data. If money is involved in this transaction, that can get to be a problem really quickly.
For that reason, the best approach would be to use a custom downloader client, like the one Amazon uses for MP3 file purchases. That way you're not subjecting either yourself or your customers to the vagaries of moving monetized bits over something as unreliable as HTTP.
you can create an asp.net handler that serves the file ( for asp.net mvc u can do a result action instead ... this is what I'm using). Make sure it supports resumable downloads.
from the you can track the bytes served.
Ps. this incurs a performance overhead vs. letting IIS serve it
update 1: I used something pretty similar to this http://dotnetslackers.com/articles/aspnet/Range-Specific-Requests-in-ASP-NET.aspx ... and the article has a pretty clear explanation on what's inside it. You probably can use that one as is, see the example in that post.
You could try looking into HTTP reponse codes (i.e: 200, 404 etc) - the client and server will be exchanging http headers so that they know what's going on - you should be able to monitor these to see if the reponses was successful (not sure - but you should be able to).
With regards to file size - I would try experiments on files with 'known' sizes, compare what the Http Logs tell you with what file explorer tells you.
Also, I've seen tools/wodgets that report file upload progress - so you're right you should be able to to the same in reverse, I guess. You could try looking at file upload code examples and tutorials - you might get some hints. I can't think of any off the top of my head - sorry.
To do custom byte serving like this, you will need to implement your own http handler.
This handler should do the following:
Implement some kind of authentication on the http handler, so you know who you are dealing with.
Then you will need to implement some kind of logging for files requested and files allowed to be downloaded.
Implement etags and expires headers for client side caching.
Server side caching
Deflate, gzip compression
If you want to support resumable downloads, you will need to implement 206 partial responses. This is essential for any kind of streaming and serving pdfs.
So you should be handling the following http headers:
ETag
Expires
Accept-Ranges
Range
If-Range
Last-Modified
If-Match
If-None-Match
If-Modified-Since
If-Unmodified-Since
Unless-Modified-Since
If you are looking for a sample implementations of http handlers check out:
http://code.google.com/p/talifun-web/wiki
It has a static file handler that implements all the above http headers, client side and server side caching and even compression.
There is also a log module and an authorization module that should go a long way into how to implement authentication and logging.
The size you want is the size (not the size on disk). Size on disk includes extra space that is taken up by fitting into the 4K block size of the partition. The size is the exact number of bits in the file.
I don't believe there is a good way to tell that a download has been completed. Response.TransmitFile is probably the best method for sending the file securely. But I don't believe it has anything that will tell you if the user actually recieved the file.
I don't know about the business this is supporting, but I can't think of a legitimate business where users would tolerate a single download per purchase model, and with the abiguity of the standard HTTP request/response model does not lend itself to making an accurate client side reciever. Not to mention this model could be eaisyly hacked by sending a failed response on reciept of the last packet.
I think using somthing like download windows (2hrs after purchase) and then lock it to an IP after the first request would accomplish the same result and result in alot less user issues and support calls. Also unless the file has some sort of stringent DRM, allowing the user persisten access based on their loggin is most likely the appropriate business model, because once they get the file they can copy it as many times as they like.
Look at DVD or Blu-Ray, no amount of copy protection or access controls will save your files from pirates, so make things easy for legitimate users.
I'm trying to figure out the best way to do caching for a website I'm building. It relies heavily on screen scraping the wikipedia website. Here is the process that I'm currently doing:
User requests a topic from wikipedia via my site (i.e. http://www.wikipedia.org/wiki/Kevin_Bacon would be http://www.wikipediamaze.com/wiki?topic?=Kevin_Bacon) NOTE: Because IIS can't handle requests that end in a '.' I'm forced to use the querystring parameter
Check to see if I've already stored the formatted html in my database and if it does then just display it to the user
Otherwise I perform a web request to wikipedia
Decompress the stream if needed.
Do a bunch of DOM manipulation to get rid of the stuff I don't need (and inject stuff I do need).
Store the html in my database for future requests
Return the html to the browser
Because it relies on screen scraping and DOM manipulation I am trying to keep things speedy so that I only have to do it once per topic instead of for every single request. Here are my questions:
Is there a better way of doing caching or additional things I can do to help performace?
I know asp.net has built in caching mechanism, but will it work in the way that I need it to? I don't want to have to retrieve the html (pretty heavy) from the database on every request, but I DO need to store the html so that every user get's the same page. I only ever want to get the data from Wikipedia 1 time.
Is there anything I can do with compression to get it to the browser quicker and if so can the browser handle uncmopressing and displaying the html? Or is this not even a consideration. The only reason I'm asking is that because some of the pages wikipedia sends me through the HttpWebRequest come through as a gzip stream.
Any and all suggestions, guidance, etc. are much appreciated.
Thanks!
You can try to enable the OutputCache for your page with VaryByParam=topic. That stores a copy of the page in memory if multiple clients request it. When the page is not in memory, the server can retrieve it from your database. The beauty of OutputCache is that you can even store a gzipped version of the HTML (use VaryByEncoding)
If it's a problem for you to decompress the stuff you get from Wikipedia, then don't send an Accept-Encoding header. That should force Wikipedia to send the page to you uncompressed.
Caching strategy: write the HTML to a static file and let users download from that file.
Compression strategy: check out Google's PageSpeed Best Practices.
I need to export a large amount of data (~100mb) from a sql table to a user via the web. What would be the best solution for doing so? One thought was to export the data to a folder on the db server, compress it (by some means) and then provide a download link for the user. Any other methods for doing so? Also, can we compress data from within sql server?
Any approaches are welcome.
I wouldn't tie up the database waiting for the user to download 100Mb, even for a high speed user. When the user requests the file have them specify an email address. Then call an asynch process to pull the data, write it to a temp file (don't want > 100mb in memory after all), then zip the temp file to a storage location, then send the user an email with a link to download the file.
You can respond to a page request with a file:
Response.AddHeader("Content-Disposition",
"attachment; filename=yourfile.csv");
Response.ContentType = "text/plain";
Be sure to turn buffering off, so IIS can start sending the first part of the file while you are building the second:
Response.BufferOutput = false;
After that, you can start writing the file like:
Response.Write("field1,field2,field3\r\n");
When the file is completely written, end the response, so ASP.NET doesn't append a web page to your file:
Response.End();
This way, you don't have to write files on your web servers, you just create the files in memory and send them to your users.
If compression is required, you can write a ZIP file in the same way. This is a nice free library to create ZIP files.
Your approach works fine. SSIS + 7zip might be useful for automating the process if you need to do it more than a couple times.
If XML is OK, one approach would be to select the data "FOR XML" like this:
http://www.sqljunkies.ddj.com/Article/296D1B56-8BDD-4236-808F-E62CC1908C4E.scuk
And then spit out the raw XML directly to the browser as content-type: text/xml. Also be sure to set up Gzip compression on your web server for files with XML extensions. http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/502ef631-3695-4616-b268-cbe7cf1351ce.mspx?mfr=true
This will shrink the XML file down to 1/3 or maybe 1/4 the size as it's transferred. This wouldn't be the highest performance option because of the inherent wasted space in XML files, but a lot depends on what format you're looking for in the end.
Another option would be to use the free CSharpZipLib to compress the XML (or whatever format you want) into a zip file that the user would download. Along those lines, if this is something that will be used frequently you might want to look into caching and storing the zip file on the web server with some sort of expiration so it's not regenerated for every single request.
The download link is a perfectly valid and reasonable solution. Another would be to automatically redirect the user to that file so they didn't need to click a link. It really depends on your workflow and UI experience.
I would suggest against implementing compression in the SQL Server engine. Instead look at the DotNetZip library (Or System.IO.Conpression if you think your users have the capability of uncompressing gzip archives) and implement the compression within the web application.