In this age of *chat applications and various messaging software, I was wondering if there is already an official protocol (RFC) that would follow the following basic flow:
Client connects to Server for a new session
Client uploads an image (or video) with metadata information (size, resolution, format) to server
Server does some work (not part of the protocol)
Server replies with REJECT then client goes to 1.
Server replies with ACCEPT then client stops and gather the result as part of the reply from the server
I have a proprietary solution now that does the basic (supports basic formats) and as we know, the devil is in the details so I wonder if some existing protocol would cover the stream format and more unhappy paths I may have missed with this simple design.
I'm not aware of any protocol that can handle file probing for you ..
ffprobe is a good open source solution to do this but requires processing power and scale.
So this step must be done on server side, after the upload. You cannot trust the client for such information.
I suggest the cloud approach. Here, we're using Amazon Cloud.
Upload your file to AWS S3. You can use multi part upload for faster upload. No need to scale anything, AWS will do it for you.
Your clients just request a signed URL from your web server. The server return the URL and an ID for this new asset. Your clients upload to AWS S3 using the URL.
Once the upload is done, your client make a call to your server again to say: "I'm done with Asset ID blah". Your server knows the asset is now uploaded and can initiate transcoding, analysis, DB updates, etc.
We do this exact scenario in our project.
For transcoding at scale we use our own open source project: https://github.com/sportarchive/CloudTranscode
This is not an easy business, especially if you want to handle videos.
If you restrict yourself to pictures, then a lot can be done on the user side. You can create several versions of the image in JS, directly in the browser for example or in the mobile app, and upload them to your server. The load is much smaller and you may not need this decoupled architecture.
If you handle videos, you need a solid backend.
Hope this help
Related
Does someone knows a way of sending an HTTP post to a URL from a VG-IDMS program?
We have a mainframe database with a few programs written on it and we have so far successfully created functionality for it to work as a kinda web service provider, responding to HTTP get requests with XML data. That way we've been able to create external (and modernized) apps that requests data from it as if would be a modern server.
Now we'd like to make the opposite. We want certain VG programs to create and send HTTP post requests to an external URL (most likely JSON formatted although not important). The objective is to create some sort of notification so that when certain events happen on the mainframe (say data got updated) a VG program would notify an external web service (or web api) of it.
I've been trying to find documentation all over but failed so far.
EDIT: Replacing the old mainframe is not an option for now
My app is remotely updated with image URLs from tons of different sources, most of these on HTTP servers. Now, these HTTP URLs won't download because of App Transport Security.
Apparently I have two options.
Allow all HTTP URLs: Bad practice, possible App Store rejection
Allow specific domains: Not possible because this would mean I can't remotely add images
How can I download the images without disabling App Transport Security?
What if you programmed your own web based API that downloads and stores said images - then you can retrieve them with a call to it. Secure your API with HTTPS. This also gives you more fine grained control in that you own the server you are querying instead of accessing someone else's.
I need to implement a file upload service, which may include some larger files. I initially used a WCF service and was using non streamed (i.e. I sent a byte[]), but would get out of memory exception (reading a file into the byte[] on the client).
I then tried using Streams in the WCF. This seemed to work fine for large files, but the huge restriction is that you seem to need to only send the Stream by itself. I don't seem to be able to also send the file metadata (e.g. filename, timestamp etc)
This service may have many client apps uploading, so I can't really see how to match up file meta data sent in a separate call to just a stream with no other information.
I am wondering if Web API may be a better way. I haven't tried it yet, but in some samples I have seen, you appear to be able to send other data along with the stream.
If using Web API, it will be self hosted (not running, in IIS, it will be running in a Windows service).
So, to me it is starting to look like Web API may be a better choice here? Or is there a way to send metadata with a file Stream in WCF?
Thanks in advance for any ideas!
You can achieve it using WCF by wrapping necessary metadata in a data contract or message contract object. You can refer the implementation on: http://www.codeproject.com/Articles/166763/WCF-Streaming-Upload-Download-Files-Over-HTTP
Web API is an alternative if your clients prefer to communicate in JSON/XML and you do not want to worry about the serialization/deserialization of JSON/XML in WCF. Your client can send a HTTP POST request with multipart/form-data content type to upload the file using Web API.
We are one of the Openstack cloud user and use Swift to store lots of files in cloud. And our users (the end-users) will access these files. If the files are public, then the end-users can directly visit the object URL in web or mobile applications. However, some of files are private. Each end-user can only access his own private files.
The end-user may have two methods to access his private files:
A. The end-user always sends requests to our server. And our server will get the files from Swift, and then send them to the user. In this
method, Our server can be regarded as a file proxy. We have the
authentication of Openstack, and the user has the authentication of
our server. We should maintain our own user system.
One problem of this method is the scalability. Because each request
will be sent to our server, then the server will be the bottleneck if
many files are uploading or downloading at the same time. The high
scalability feature of Swift can't be used if we don't have enough
servers. Actually we really don't need lots of servers except file
transferring.
B. The end-user directly sends requests to Swift. It may solve the huge file bandwidth problem. But he must have the authentication
method of Openstack. So we need thousands of Openstack accounts for
our users while we are only one of the user of Openstack! It seems
strange and is not feasible.
Are there any other methods to meet the requirement?
You could use your server to get the files (scenario A), but put that server "in the cloud". Then as access needs to scale up, you spin up additional servers under a load balancer. You, therefore, are distributing the load horizontally as needed.
An idea.
(Disclaimer: I am a Rackspace employee)
A third idea is to use swift's tempurl feature, when a client needs a file, you genenerate a signed url to the file, having checked checked if they are authorised, and the url they get enables them to download directly from swift,
so two problems are solved
1. User A can not get User B's files
2. Download is straight from swift, so your server does not become the bottleneck for downloads (only for authorization).
I am working on an AIR app which should work in both online and offline model. The user can do various actions, while offline and the results would get saved in a local DB. The same needs to get synced up with the global DB once the user goes online. I googled a bit on it, and it seems that Adobe LCDS (lifecycle data services) is the only available option to do it. However, it is an enterprise solution, and way too costly.
Is there any other implementation for this? Has anyone used CouchDB for online/offline synchronization?
Thanks and Regards,
Kapil Kaushik
For doing a DB sync with your server when the Air app is only, you do not have any requirements as for which backend technology you use. LCDS makes it simpler, but it's not your only option. Heck, you could use just a normal PHP script to do the sync for you if you'd like.
The hard part of it all is that you need to figure out your syncing algorithm so that you don't lose any information. Normally what I do is that when the app is connected online again, it sends all the information that was modified/create (with timestamps on when it was modified) while offline to the server, then the server has an algorithm that checks if the offline information is newer than what was done previous (or does some other business rule depending on your situation). When the server decides which data is good, it then sends the updated data to the client and effectively syncs everything.
This can be done through a normal HTTP request, polling or pushing.