Anyone know why firebase storage would be so ridiculously slow compared to firebase hosting?
Results
Time to download image of firebase hosting: 16ms
Time to download same image from firebase storage: 2.23s (2.22s is TTFB)
Time to download same image from firebase storage (Asia Pacific Region): 1.72s (1.70s is TTFB)
(File size: 22.7kb / jpeg / firebase storage has read open to everyone)
This is repeated over and over in tests. Is there any way to speed this up to a decent time, or is firebase storage unusable for small files (images/thumbs)?
For Comparison
S3 North Cal - approximately 500ms
S3 Asia Pacific - Approximately 30ms
Cloudinary - Approximately 20ms
Extra info:
I am based in Australia.
Exact same files. Always images under 100kb.
The slow down is always in the TTFB according to dev tools.
Hosting URL: https://.firebaseapp.com/images/thumb.jpg
Storage URL: https://firebasestorage.googleapis.com/v0/b/.appspot.com/o/thumb.jpg?alt=media&token=
I found the solution.
If you have your files already uploaded to storage go to: https://console.cloud.google.com/storage/browser?project=your_project > pick your bucket > select all interesting files and click Make public (or something similar - I'm not english native).
To have all new uploaded files public by default you need to install Google cloud SDK (https://cloud.google.com/sdk/docs/) and from your command line use the following command for your bucket:
gsutil defacl set public-read gs://your_bucket
After that all my current and new images are available here storage.googleapis.com/my_project.appspot.com/img/image_name.jpg
and downloading time is definitely shorter.
Hosting = Storage + CDN, so really what you're seeing is you hitting a CDN near you, rather than going directly to the GCS or S3 bucket. Same is true with Cloudinary/Imgix. This is why performance is so much better for Hosting than Storage.
Addressing the issue of TTFB being so different between AWS and GCP: unfortunately this is a known issue of GCS vs S3 (see this great blog post w/ in depth perf analysis). I know this team is working to address this problem, but going the "stick a CDN in front of it" route will provide a faster solution (provided you don't need to restrict access, or your CDN can authorize requests).
Note: GCP has announced a Sydney region (announcement blog post) to be launched in 2017, which might help you.
In addition to #Ziwi answer.
I think it is also ok to change rules directly in Firebase
// Only a user can upload their profile picture, but anyone can view it
service firebase.storage {
match /b/<bucket>/o {
match /users/{userId}/profilePicture.png {
allow read;
allow write: if request.auth.uid == userId;
}
}
}
The source is https://firebase.googleblog.com/2016/07/5-tips-for-firebase-storage.html
Related
I am struggling to find out how to set the limit of the storage that each user can upload to my apps storage.
I found method online Storage.storageLimitInBytes method, but I don't see this method even be mentioned in Firebase docs, let alone instructions on how to set it.
In general, how do startups monitor how many times user upload images, would they have a field in users document such as amountOfImagesUploaded: and everytime user uploads image I would increment that count and this way I could see who abuse the storage that way.
Or would I have to similar document that tracks users uploads per day and when the count reaches 100 or something then take action on that user.
I would really appreciate your help regarding this issue that I am facing.
Limits in Cloud Storage for Firebase security rules apply to each file/object separately, they don't apply to an entire operation.
You can limit what a user can upload through Firebase Storage's security rules. For example, this (from the linked docs) is a way to limit the size of uploaded files:
service firebase.storage {
match /b/<your-firebase-storage-bucket>/o {
match /images/{imageId} { // Only allow uploads of any image file that's less than 5MB
allow write: if request.resource.size < 5 * 1024 * 1024 && request.resource.contentType.matches('image/.*');
} } }
But there is currently no way in these rules to limit the number of files a user can upload.
Some options to consider:
If you hardcode the names of the files that the user uploads (which
also implies you'll limit the number of files they can upload), and
create a folder for the files for each specific user, you can
determine the sum of all files in a user's folder, and thus limit on
the sum in that way.
For example : If you fix file names and limit the allowed file
names to be numbered 1..5, the user can only ever have five files in
storage:
match /public/{userId}/{imageId} {
allow write: if imageId.matches("[1-5]\.txt");
}
Alternatively, you can ZIP all files together on the client, and
then upload the resulting archive. In that case, the security rules
can enforce the maximum size of that file.
And of course you can include client-side JavaScript code to check
the maximum size of the combined files in both of these cases. A
malicious user can bypass this JavaScript easily, but most users
aren't malicious and will thank you for saving their bandwidth by
preventing the upload that will be rejected anyway.
You can also use a HTTPS Cloud Function as your upload target, and
then only pass the files onto Cloud Storage if they meet your
requirements. Alternatively you can use a Cloud Function that
triggers upon the upload from the user, and validates the files for
that user after the change. For example : You would have to
upload the files through a Cloud function/server and keep track of
the total size that a user has uploaded. For that,
Upload image to your server
Check the size and add it to total size stored in a database
If the user has exceeded 150 GB, return quota exceeded error else upload to Firebase storage user -> server -> Firebase storage
An easier alternative would be to use Cloud Storage Triggers which
will trigger a Cloud function every time a new file is uploaded. You
can check the object size using the metadata and keep adding it in
the database. In this case, you can store total storage used by a
user in custom claims in bytes.
exports.updateTotalUsage = functions.storage.object().onFinalize(async (object) => {
// check total storage currently used
// add size of new object to it
// update custom claim "size" (total storage in bytes)
})
Then you can write a security rule that checks sum of size of new
object and total storage being used does not exceed 150 GB: allow
write: if request.resource.size + request.auth.token.size < 150 *
1024 * 1024
You can also have a look at this thread too if you need a per user
storage validation. The solution is a little bit tricky, but can be
done with :
https://medium.com/#felipepastoree/per-user-storage-limit-validation-with-firebase-19ab3341492d
Google Cloud (or Firebase environment) doesn't know the users. It knows your application and your application do.
if you want to have statistic per users you have to logs those data somewhere and perform sum/aggregations to have your metrics.
A usual way is to use Firestore to store those information and to increment the number of file or the total space used.
An unusual solution is to log each action in Cloud Logging and to perform a sink from Cloud Logging to BigQuery to find your metrics in BigQuery and perform aggregation directly from there (the latency is higher, all depends on what you want to achieve, sync or async check of those metrics)
I am trying to set up real time access logs for objects stored in a google cloud storage bucket. The access logs that google specifies here are generated every hour and won't work.
I would like similar information about object access(ip, amount downloaded, client_os) but in real time. Is there a way to do this on the google cloud platform?
If it would be better to route the traffic through another point with real time logs I would appreciate advice on how one would do that and with what tools. Ultimately, I want to have the data visualized in google data studio.
Unfortunately, google storage only provide following gcp-storage metric
- api/request_count
- authz/acl_based_object_access_count
- authz/object_specific_acl_mutation_count
- network/received_bytes_count
- network/sent_bytes_count
- storage/object_count
- storage/total_byte_seconds
- storage/total_bytes
I think you want access log for your bucket objects, such as images, videos...etc.
You maybe you can try to use Google Cloud CDN in front of your bucket .
So you can get access log you need by httpRequest, and get from stackdriver
{
"requestMethod": string,
"requestUrl": string,
"requestSize": string,
"status": number,
"responseSize": string,
"userAgent": string,
"remoteIp": string,
"serverIp": string,
"referer": string,
"latency": string,
"cacheLookup": boolean,
"cacheHit": boolean,
"cacheValidatedWithOriginServer": boolean,
"cacheFillBytes": string,
"protocol": string
}
There are three types of the Logs for Cloud Storage:
Access logs - updated on hourly rate logging all the requests for the specified bucket.
Storage logs - updated daily, providing all the information about storage usage from the last day.
Audit logs - tracks access on the continuous basis, this is recommended way to log your storage activity for autenticated users.
You can see Audit logs in the Google Cloud Platform Console > Activity Activity Stream.
Resource type filter: GCS bucket.
A more detailed version of the logs can be found in the Logs Viewer.
If you need to use logs for further analytics, there are several options to do so provided in the Google Cloud documentation. There is also example query to visualize data in Data Studio. Check more custom queries for Data Studio here.
I've been trying for the past couple of weeks to get this Load Balancer + Cloud Storage + CDN combo to work. It just doesn't, at least for me.
Got some static files (2 jpg's, svg's and css's just in case) into a multi-regional US (tried on regional too) bucket to test it out, but it just seems like it doesn't wanna cache at all.
Everytime I try checking it's headers, all I get is this same old boring bucket metadata:
Cache control are set just fine, you can see the v=2 at the top because I just keep trying to make it cache in different ways and cache strings was the last attempt. Unsucessful as well. LB works because this IP resolves from it.
What the hell am I doing wrong?
You can check the links in here:
http://35.227.213.66/style.css
http://35.227.213.66/logo.svg
http://35.227.213.66/1.jpg
http://35.227.213.66/2.jpg
I can see that you are using the correct metadata
Cache-Control: public, max-age=604800
It would be interesting to check how many requests have been answered from the CDN and how many from the bucket. You can use a query with 'gcloud beta logging' to check this:
From CDN
$ gcloud beta logging read 'resource.type="http_load_balancer" AND "logo.svg" AND httpRequest.cacheHit=true AND timestamp>="2017-12-04T07:23:00.054257251Z"' | wc -l
From your bucket
$ gcloud beta logging read 'resource.type="http_load_balancer" AND "logo.svg" AND httpRequest.cacheHit=false AND timestamp>="2017-12-04T07:23:00.054257251Z"' | wc -l
Just trying to figure out something that seemed trivial in firebase, in google-cloud.
It seems as though if you're making a node.js app for HTML (i'm talking to it through Unity actually, but it's a desktop application) you can't use firebase-storage for some odd reason, you have to use google-cloud, even the firebase-admin tools use the cloud storage to do storage from here.
Nevertheless, i got it working, i am uploading the files to the firebase storage; however, the problem is in firebase, you could specify a specific file, and then do storage().ref().child(filelocation).GetDownloadURL(): this would generate a unique url for some set time that can be used publicly, without having to give out access to read to all anonymous users.
I did some research and i need to implement something called GS UTIL in order to generate my own special urls, but it's so damn complicated (im a newbie to this whole server stuff), i don't even know where to start to get this working in my node server.
Any pointers? I'm really stuck here.
-------if anyones interested, this is what im trying to do high level-----
I'm sending 3d model data to node app from Unity
the node app is publishing this model on sketchfab
then it puts the model data onto my own storage, along with some additional data specially made for my app
after it gets signed to storage, it gets saved to my Firebase DB in my global model database
to be accessed later, by users, to try to get the downloadURL of this storage file and send them all back to Unity users(s)
I would just download the files into my node app, but i wanna reduce any server load, it's supposed to be just a middleman between Unity and Firebase
(i would've done it straight from Unity, but apparently firebase isn't for desktop windows apps).
Figured it out:
var firebase_admin = require("firebase-admin");
var storage = firebase_admin.storage();
var bucket = storage.bucket();
bucket.file(childSnapshot.val().modelLink).getSignedUrl({
action: 'read',
expires: expDate
},function(err,url){
if(err){
reject(err);
}
else{
finalData.ModelDownloadLink = url;
console.log("Download model DL url: " + url);
resolve();
}
});
I'm using firebase storage to upload avatars (original.jpg).
In a node.js background process I resize them with gm and then put them back in my firebase bucket. These pictures are publicly readable, see my storage rules :
service firebase.storage {
match /b/xxx.appspot.com/o {
match /images {
match /{allImages=**} {
allow read;
}
match /{userId}/avatar/original.jpg {
allow write: if request.auth.uid == userId;
}
}
}
}
I store the url of images in firebase database ex:
https://firebasestorage.googleapis.com/v0/b/xxx.appspot.com/o/images%2Fyyy%2Favatar%2Fzzz-small.jpg?alt=media
That I retrieved right after the file upload (web sdk) var downloadURL = uploadTask.snapshot.downloadURL; ; stripped off the token parameter as it's gonna be public ; and where zzz-small.jpg has replaced original.jpg.
It works, pictures are displayed.
But it's slow even though the picture is a 40x40px jpg. And it hangs for a while before actually downloading the file, see Chrome Network record :
Chrome network record
The file is 158B, there's a ~1s waiting before the download ~3ms...
Is firebase storage supposed to be as fast as a CDN ?
Any faster way to call a public readable file ?
[EDIT]
Got some feedback from Firebase support :
We're definitely aware that many users, such as yourself, are having
issues with slow downloading of file from Europe. We're exploring
potential solutions, but I can't share any details or timelines at
this time. We'll keep your feedback in consideration moving forward
though.
Keep an eye out on our release notes for any further updates.
I forgot to mention I'm in Europe and the storage in the US.