I am developing a web application in ASP.NET 4.5. One part of the application includes the user the option to upload images. The images are stores on Amazon S3. Right now the pathc I though about choosing is to use Amazon SDK to upload the images to the bucket on S3 and server them via CloudFront. The thing is that I think that using s3fs might be a better option.
If I mount an S3 bucket as a folder, when the user upload a photo, I can continue the application operation, knowing that the image will be transferred via the network to S3, so I don't need to wait until this process completes before continuing the code. So all I have to do is to wait until the image finished uploading to the server and continue the code.
I want to know if this a good way to do this. Waiting for images to upload can take time and I don't want the user to wait until all the images have been uploaded, which can take some time.
Any suggestions for the best implementation of image uploading?
That is a suitable approach if you will be having multiple application servers which need to interact with a single bucket. You might want to consider configuring s3fs to use a local storage directory as cache, so as to improve performance, as writing directly to your s3fs mount will typically take longer than to local storage.
Related
I have an ASP.NET core API project which lets users upload images. My first implementation was to use Base64 and save the images in the SQL server. However, I decided not to do that because of performance issues. The second implementation was to use Azure Blob storage and upload the files directly into the blob storage.
I am not sure if this is a good idea but instead of using Azure Blob, I would like to upload the images somewhere in my Linux server. Is there any special directory for saving files and would it be safe for me to do that?
As far as I know, there is no special directory for saving files in Linux server which would be more safe.
All the folder in the linux is the same, if you have enough permission to access it ,then you could read and write the image in it.
Normally, we will add a folder inside our application to store the uploaded image, so that we could use relative path in our codes.
In my opinion, use blob storage is a good opinion. This will be directly access from blob url if you have enough permission and it is safety, we could generate the SAS to allow only specific user access, it is High durability and cheap enough.
So here is the scenario:
When I access files from Firebase Storage:
I get my file from storage bucket(.html, .png, .zip etc) (Small in size btw no more than 2mb).
Store that file in my local storage so the app don't need to download it again and consume the bandwidth of the server.
Use it from local storage everytime the app needs it.
When I access files from Firebase Hosting:
I get my file from nearest CDN of Firebase(.html, .png, .zip etc) (Small in size btw no more than 2mb).
Store that file in my local storage so the app don't need to download it again and consume the bandwidth of the server.
Use it from local storage everytime the app needs it.
NOTE: I also have one file version.txt on storage bucket (Firebase Storage). According to the value in this file, I decide whether to fetch file in Step 1 again or not. It means the version.txt is fetched everytime.
Questions:
How to achieve the similar version programming part in Firebase Hosting? I know we deploy folders, can we get their version from Firebase CDN. If yes, how?
In which method I gonna first hit my LIMIT as we know Firebase is paid after a limit.
Pros of Hosting: It will be faster. Link
PS:
1. My concern is bandwidth and not security.
Currently, I am using basic Plan (free) with limits Source:
From the Firebase docs:
The Firebase Realtime Database stores JSON application data, like
game state or chat messages, and synchronizes changes instantly
across all connected devices.
Firebase Remote Config stores
developer-specified key-value pairs to change the behavior and
appearance of your app without requiring users to download an update.
Firebase Hosting hosts the HTML, CSS, and JavaScript for your website
as well as other developer-provided assets like graphics, fonts, and
icons.
Cloud Storage stores files such as images, videos, and audio as well as other user-generated content.
Storage has higher free tier limits, while Hosting might be a little faster. Note that all files on Hosting are publicly accessible, so if you need authentication or authorization, you should use Storage.
I want to deploy a php application from a git repository to AWS Opsworks service.
I've setup an App and configured chef cookbooks so it runs the database schema creation, dumping assets etc...
But my application has some user generated files in a sub folder under web root. git repository has a .gitignore file in that folder so an empty folder is there when i run deploy command.
My problem is : after generating some files (by using the site) in that folder, if I run 'deploy' command again 'Opsworks' adds a new release under 'site_name/releases/xxxx' folder and symlink to it from 'site_name/current' folder.
So it makes my previous 'user generated stuff' inaccessible. What is the best solution for this kind of situation?
Thanks in advance for your kind answers.
You have a few different options. Listed below in order of personal preference:
Use Simple Storage Service (S3) to store the files.
Add an Elastic Block Store (EBS) volume to your server and save files to the volume.
Save files to a database (This is something I would not do myself but the option is there.).
When using OpsWorks think of replicable/disposable servers.
What I mean by this is that if you can create one server (call it server A) and then switch to a different one in the same stack (call it server B), the result of using server A or server B should not impact how your application works.
While it may seem like a good idea to save your user generated files in a directory that is common between different versions of your app (every time you deploy a new release directory is generated) when you destroy your server, you run the risk of destroying your files.
Benefits and downsides of using S3?
Benefits:
S3 will give you high redundancy and availability to your files.
S3 is external to your application server so if your server dies or decide to move it to a different region, you can continue using the same s3 bucket.
Application Easy to scale. You could add multiple application servers that read and write files to S3.
Downsides:
You need extra code in you application. You will have to use the AWS API in order to store and retrieve the files. Using the S3 API is not hard but it may require an extra step to get you where you need. Take a look at the "Using an Amazon S3 Bucket" walk through for reference. This is be the code they use to upload the files to the S3 bucket in the example.
Benefits and downsides of using EBS?
Benefits:
EBS is an "external hard drive" that you can easily mount to your machine using the OpsWorks Resource Manager.
EBS volumes can be backed-up and restored.
It may be the fastest option to implement and integrate to your application.
Downsides:
You need to assign it to an instance before it is running.
It could be time consuming to move from server A to server B (downtime may be required).
You can not scale your application horizontally. While you can create copies of the EBS and assign them to different instances, the EBS will not be shared.
Downside of using a database?
Just do a google search on "storing files in database"
Take a look at Storing Images in DB - Yea or Nay?
My preferred choice would be to use S3, but ultimately this is your decision.
Good luck!
EDIT:
Take a look at this repository opsworks-chef-cookbooks it contains some recipes to deploy Symfony2 application on OpsWorks. I have been using it for over a year and works quite well.
Use Chef templates, and use them in a recipe in the opsworks deploy lifecycle event.
Using Classic ASP (stop tutting), I need to build an application that transfers high resolution photos from one server to another, around 360,000 including the thumbnails to be exact. The application will be called via a Windows schedule and will run as a background process.
What is the best way to achieve this, keeping performance in-mind? The last time I built a monster script like this was transferring and converting database tables for over one million rows, the application started really fast, but then after 25,000 records it went really, really slow! So I want to avoid this.
Obviously it will be a cross-domain transfer, so I was thinking about using an ASP/FTP component, and one-by-one, grab a file, send it, and record its success in a DB table so it knows what is has done so far.
Is it best to process one file at a time and refresh, so it doesn't abuse the server's resources, or should I process 1000 at a time, or more? I want it to be as quick as possible but without clogging up the server.
Any help/suggestions would be gratefully received.
I think is best to do one file at a time because if the connection goes down for a brief period of time you don't lost the files that you have already sent.
Even when you are using ASP Classic you can take advantage of .net for uploading the files using the FTP client classes in .net and avoid purchasing/installing a third party component. Surely .net is already installed on the server.
My process will look like this:
Upload 1 file using FTP (better performance)
If successful call an ASP page that records the action in the remote DB
Wait a second and retry up to 3 times if error uploading
Proceed to next file
If the process is clogging the server, you can put a brief pause between each upload.
i have something like that running in Classic ASP, it handles tenthousands of images without problem.
On the server that houses the images I run a (vbs)script that for each image
Makes a text-file with metadata
Makes a thumbnail and a mid-sized image copy on the second (web)server
The script runs continuously and only checks per folder and file if the files are present on the webserver and if not creates them, No need for a DB.
Between every check It sleeps a second. Like that the load on the server is only 2%. I use iPhoto in command-line modus to extract the metadata and images but you could use a library for that.
So these three files are stored on the webserver in a copy of the mapstructure from the first server but without de full-sized images.
On the webserver you only need to be able to browse the thumbnails and visualize the metadata and mid-size images.
If the user needs the full-size image he clicks the mid-sized which has as url the file on the first server.
Upload all the files via FTP
Create a CSV file with all your data
Pull it into the DB in one go
The amount of network handshake over 360,000 individual transactions would be the bottleneck.
I'm building an app in ASP.NET that will store some pictures of objects. The pictures will be uploaded by suppliers and downloaded by subscribers. In between, they will have to be edited before becoming available to subscribers.
The editing involves creating a cropping path tightly around the object in the picture, in which some advanced desktop image software will have to be used I suppose.
My problem is in exchanging pictures between my ASP.NET app and the desktop software in a manner that is easy and transparent for the user.
I've done some thinking and I've come up with:
- Manually downloading and uploading the image (Not much user friendly...)
- An image editing program that can upload to a web service (Haven't found yet...)
- Develop a plug-in for an image editing program (Too advanced...)
I'd appreciate any suggestions you may have, thank you!
It sounds like you need some automation to move files between the web server and a file share. I am assuming that the number of images that need to be processed is pretty large, because if it's not, then the overhead of downloading/re-uploading each would not be that much.
So do the following:
1) Create an API for your web app that lists files that are available, or new files since some date/time, or files that have been marked as "new". The API should probably also allow marking a status on them (so you can tell it when you've finishing pulling something down, and it won't be offered again) if you don't want to trust date/time as an indicator of it being new.
2) Write an app (non-web) that runs on a schedule and uses this API to automatically download files to a shared filesystem area in your local network, and marks them as "downloaded"
The app should also monitor these files (the ones it downloaded & saved to your local share) for changes, and if changed, upload them back to your web app. To do this you may need to keep a database of filenames and modification dates/times.
This shouldn't be too hard to write in whatever language you are using for your web (assume c# or vb). By "API" I just mean, a web page that provides a list in a standardized format (e.g. json) that you can parse with your automation application, and another page that allows posting the file back for re-upload.
I'm assuming that the web server is not your own, or generally, you can't simply have it save the file uploads directly to some area where your image editors can access them. Otherwise you could just do that.
Meanwhile I came out with another possible solution.
I'm thinking of having our own windows app on the editor's computers. This app will be associated with a custom extension. When an editor downloads a file (with this extension) for editing, it will be opened in our application which in turn will open the image in some editor program.
This app will be monitoring the files for changes, and in such case, it will upload these images.
Any thoughts on this?