I am using the Firebase web client and would like to upload a gzipped file to Cloud Storage. With the Nodejs client it is possible to simply specify gzip as an option like so: await bucket.upload(filepath, { destination, gzip: true });. Is it possible to gzip data upload to Cloud Storage using the Firebase web client? For example, how would I change the following to gzip the file?
const ref = firebase.storage().ref().child("test.json");
const blob = new Blob([JSON.stringify({ hello: "world" })], { type: "application/json" });
await ref.put(blob);
You can take a look at the documentation which shows you a sample to upload a picture:
// Create a root reference
var storageRef = firebase.storage().ref();
// Create a reference to 'mountains.jpg'
var mountainsRef = storageRef.child('mountains.jpg');
// Create a reference to 'images/mountains.jpg'
var mountainImagesRef = storageRef.child('images/mountains.jpg');
// While the file names are the same, the references point to different files
mountainsRef.name === mountainImagesRef.name // true
mountainsRef.fullPath === mountainImagesRef.fullPath // false
// Create file metadata including the content type
var metadata = {
contentType: 'image/jpeg',
};
// Upload the file and metadata
var uploadTask = storageRef.child('images/mountains.jpg').put(file, metadata);
But in your case you may want to upload gzip file. Just change the metadata.
Related
I'm trying to create a Cypress test that involves uploading an image directly to a google storage bucket. I'm following this stack overflow suggestion:
Upload File with Cypress.io via Request
and I have it to the point where the image file gets uploaded to the bucket without any errors during the upload. However when subsequently downloading the file from google, it's apparent that the image contents were corrupted.
The stack overflow example uses Cypress.Blob.base64StringToBlob() to create a blob to be uploaded. That method comes from this third party library which has been bundled with Cypress:
https://github.com/nolanlawson/blob-util
Here's the code that I currently am working with:
Cypress.Commands.add('fileUpload', (url, fileName) => {
const method = 'PUT';
const fileType = 'image/png';
cy.fixture(fileName).as('bin');
cy.get('#bin').then( (bin) => {
// File in binary format gets converted to blob
// so it can be sent as Form data
const blob = Cypress.Blob.base64StringToBlob(bin, fileType);
// Build up the form
const formData = new FormData();
formData.set('file', blob, fileName);
// Perform the request
cy.rawFormRequest(method, url, formData, function (response) {
cy.log(JSON.stringify(response));
});
});
});
Cypress.Commands.add('rawFormRequest', (method, url, formData, done) => {
const xhr = new XMLHttpRequest();
xhr.open(method, url);
xhr.setRequestHeader('Content-Type', 'image/png');
xhr.setRequestHeader('Access-Control-Allow-Origin', '*');
xhr.setRequestHeader('Access-Control-Allow-Headers',
'Origin, X-Requested-With, Content-Type, Accept');
xhr.setRequestHeader('Access-Control-Allow-Methods',
'GET, POST, PUT, DELETE, PATCH, OPTIONS');
xhr.onload = function () {
cy.log('success');
done(xhr);
};
xhr.onerror = function (error) {
cy.log('error');
cy.log(JSON.stringify(error));
done(xhr);
};
xhr.send(formData);
});
This is the image being uploaded:
But then when I go into the Google Storage GUI, it's apparent the image hasn't been uploaded cleanly:
I tried 2 of the other methods in the blob-util library: Cypress.Blob.arrayBufferToBlob() and Cypress.Blobl.imgSrcToBlob(), however again, although the file is uploaded without errors, when downloading or viewing in the google cloud GUI it's apparent that the file contents weren't uploaded cleanly.
I've started working with firebase storage and firebase functions recently. Right now I've been developing file upload from functions to storage .
I've got it working (upload is done and file appears on the storage section), yet, the image, stays like this forever (loading forever on the right side):
I though that it was an error from my code. Yet, if I open Google Cloud Platform - Storage, the image appears and I can open it and preview it.
In firebase storage, if I open the image (select on it and click open), it returns the following url: https://console.firebase.google.com/u/0/undefined
What may I been doing wrong? Here's the code I'm using:
function uploadImage() {
const newImageData = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAAAAgVBMVEX///8AAAAEBASAgIDr6+vw8PBYWFjU1NTGxsbz8/P29vb8/Py1tbVhYWHd3d1ra2vk5OS/v78pKSlTU1NOTk6Tk5OpqanNzc13d3dKSkplZWWbm5s5OTkfHx+NjY2GhoYcHBw9PT0TExOioqJ7e3soKCiurq5CQkI6OjoXFxcwMDAuPQWoAAAIJ0lEQVR4nO2daXuqPBCGVfYtIbKLgorLKf//B75ga2sPAdmS8F5n7m/VXgyPISGZzExWKwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOAVy6wwlCZG/YUl+vamYLoy0lAkORWbdZNN/YUUVf8ju6bomx2Kq+mEBL5K0UVD9QNCdM0Vfds9QSQMnYTWaN1sEicMCRJ9+29QSCqp28HiftiqUkoU0TLaMDLsl8PbrtGWpY8zQ7QYCoE/pe3+ZusHogW9Yln2nxnVPfljW8t4l8go2zPQV7PPkCxa3kqOvOldr52NF4nVKNuYobxPsC1OoxVilu33ZINDQf1Rw2cO+mrOWBMhMD5w0ldziLnrQyeO+mpOfOdzZspZX03KcQGC2I+gNDC3ZkQ7IQLX6x0niW7fZd83ief7Pg7tH0JcfeIlQy+kcllCyk7/O/LUWIsiDbkVv9/bcv0Rqr+MVa//BR0Ob/++AstdKLuK8n4ZZCiKK4e7cikSez2iTjxm2DPjPj8e8wf1zWviXPqpPuHyeuqXb2ZK6WxaqJCuZe7m7qTTp1da6ty7prtbMoOOVro64VktonlmyFZUqB0NybQrkna7l1ndZIhc2k0xbESjdTEf63M7yBQ9bjO2Z+enamvCXcTCphG1zZ3YNSJ9OroJmRkM6UMOZmXPoppzWE4WEX1oY7Xmv1Fs7Zn9nl9gWt+/MTImNU2dprze+6GfmmYlRrYoHZ9dF/whbJrdMTLVdMzEPDZRlOZr48DIVMMQ2wnUN5SpIiNLjaF7z74X1uiNwWbDyNI/qHCd8XBGW1nDLj+F64K9RKtomuWocMvq3fvDjbIk5ahwvc0YGXuS0dbcPBWu1zlLx4mbU23yVbi+s3srkjvdJGeF1WqGTTO6rZsH3BVWjyqae1C1EP0BFaVwvQ60OR1EshZ0GROisJqG23Pt1Gp2q49GqML1OsmK6aMOKbK3OzbCFFZsLwEZ75YySHDpE1olUmFl/XrP02jE5aM0v1972phd2ydDQkvOnjRk4aFL3pDAjiUo/LoTHOuKLMtGzet+lPn4pPpG0eMRQTnLUfi8oTreWc0j7UmUq22x0f9PhbMDCkEhKBQPKJxNYTkgEmYCXiMShZvCXEkHTUXGcPZSpbFS5KZQWq3Qu4XORGIbUTa9eCqsFqt6OGemxSvbUH8sqwUrXK0sl/gM9PnE/XKNCFdYYZozB7VvsPkzU1+Cwhrrph7mGHfOB/X226+1FIU1duztr+Nlnq97L7YbV12SworITqXdvmcw5QvlfielNtU5sDCFNYpO0iJTk2MvbcdEzYqUtIdVLVDhAwvVubIk8D3Po421j88DUufHvvEnL1XhF2Yd7+xqTR6f9wq1XbjCGQCFs/EPKsR8sq6Vxi4bvxVwzCOJBTVXLxzX+BxSrgPKm4arFwM3J1lzQk/D5eunOTjsIqN0h57Gyd0TlbBJYtVa9xFF+NpUedZSM5Ypd6UfifEm4tts5QFkdOvO4RTlLz3GoTZDVpAWxu/WJAI9wn/uRaiPj65x9bC496ixIdjnvXek7Db8HWLfMsnpWWFjAV79w9ZxqlVtzwtXq2XH2Q7I71+Aws/72FyvVz/IsrzlgnmWBX71P5vBV16IQhr3uCUYbxBLVjgPoBAUgkLxgEJQCArFAwpBISgUDzeF1+E7vWMor8IUZi5mFUrzwxa7jRRLnntP0YCqUaNwItG7a3LBUqNT1B5K0fuHboj7RSQM5YjDT7edaIV1HlbeGAwmc82/c8XEK1ytTDkcXKeuEzWUlxf1JRdzOJlq7sXv/YGlKKxwi91pQtBXHfZ12hUNn/mCFNZEaYxPY6YC5QnH9JSwhSl8qCRpFu9Pfb3Yh9M+zlLSmvAmTmGbL/uBoUfkFhY48dvDa30/wUV4I5HembMoLs7b6xGKISOE9Ce5p6qql3//XX3XY88RNVICllvbpM7Hk+WhOab8aps0nzIeRZSoZZQYWaIkyLMNNfnEbprtHAAmQCtlyL6OEq3Q34WRLcqPyb4V+RqlGVuHLMP3FEofXDPrhqsVPYIHM6xtQg86SZgZ1Kj21iWjEjVu3jL5Y3eOgEw3WK3l4vnrRVlx65qTXRFaWlGqJ/Fq1qivVUdKHMvyW3JXMJZPBs9W6BhyZ6YYZlrSW++wXHVIiaCpIg1EpO61F9syePK7fMokt8noekMWInb+rnRLzLgqu/ved3jw8tQePrq6dpp771eUDvPDA+TT25uo2Ho4v4V9n1gjvOXY6+U+P3E4OqC7K75w/DglSZymQftNyUGaxkly+ujtcOVSi7K3xF987KRXdh+jrsKn2OZK57Pl1KTkJLCaD7PejqHjcDwYsXXGyBBWs98WrBtvieWN89FkJucTkTDif9ysgTgKnDwZHMfwY5FGwucQJAqdaS2zKhR1AiIoBIWgEBSCQlAICkEhKASFi1DIokobDV/Y2ePjvG7D4eaA+gcVRnyi9zdj6p/Pg8vncOedqCX+amXxOYA85exle8Xg4XDDYrxQX2g9iyBMYM8uLqEXiHVKyZbbkfFt5GwLCZ9ZhbANIJwrfp3GnU/04xsidpM3X9yb8BcoYJQzEwjvg09M++3ZNyNIbP6bMa1YCjVIchK2IvBFT8PaDaig85bDbmHyHljhaZ7+eDyFS9RXo9jx9K3hMrY57tcPh6SX/fiWPO4vKZ8TeKdg6SRTx4g8qhnRl/p4/oWJIiKpQx7YUpVIJGCrfgqG7CIb9zjSaeNgG7kzhaZyx7JM07Ljy4XWnuXlEtv1P7B9Mv8DltyUV+hIpoIAAAAASUVORK5CYII="
var mimeTypes = require('mimetypes');
var image = newImageData,
mimeType = image.match(/data:([a-zA-Z0-9]+\/[a-zA-Z0-9-.+]+).*,.*/)![1],
fileName = 'test.' + mimeTypes.detectExtension(mimeType),
base64EncodedImageString = image.replace(/^data:image\/\w+;base64,/, ''),
imageBuffer = new Buffer(base64EncodedImageString, 'base64');
// Instantiate the GCP Storage instance
const { Storage } = require('#google-cloud/storage');
const googleCloudStorage = new Storage(firebaseSettings);
const bucket = googleCloudStorage.bucket('projectID.appspot.com');
var file = bucket.file(fileName);
return file.save(imageBuffer, {
metadata: { contentType: mimeType, cacheControl: "public, max-age=300" },
public: true,
validation: 'md5'
}, function (error: any) {
if (error) {
throw 'error';
}
return "https://storage.googleapis.com/share-expanses-dcc9f.appspot.com/" + fileName;
});
}
Thanks for the help
Haven't been able to test the solution given by Firebase, but here's the transcript of the response:
The problem that you are facing could be because of two reasons. The
first one is how you are uploading the files, via the Firebase
Console, using any Admin SDK, or via the gsutil command. If using the
Admin SDK option, the problem is a known issue where the required
metadata doesn’t exist, fortunately there is a workaround, you can try
this script to solve this issue.
Now, the second one is related to the network if you are using
comcast, please, try on a different network to see if this issue is
related to that.
When you save an image to firebase, you need to provide an access token in metadata : firebaseStorageDownloadTokens. It has to be an uuid.
More info can be found here : https://www.sentinelstand.com/article/guide-to-firebase-storage-download-urls-tokens
const { v4: uuid } = require("uuid")
function uploadImage() {
const newImageData = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAAAAgVBMVEX///8AAAAEBASAgIDr6+vw8PBYWFjU1NTGxsbz8/P29vb8/Py1tbVhYWHd3d1ra2vk5OS/v78pKSlTU1NOTk6Tk5OpqanNzc13d3dKSkplZWWbm5s5OTkfHx+NjY2GhoYcHBw9PT0TExOioqJ7e3soKCiurq5CQkI6OjoXFxcwMDAuPQWoAAAIJ0lEQVR4nO2daXuqPBCGVfYtIbKLgorLKf//B75ga2sPAdmS8F5n7m/VXgyPISGZzExWKwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOAVy6wwlCZG/YUl+vamYLoy0lAkORWbdZNN/YUUVf8ju6bomx2Kq+mEBL5K0UVD9QNCdM0Vfds9QSQMnYTWaN1sEicMCRJ9+29QSCqp28HiftiqUkoU0TLaMDLsl8PbrtGWpY8zQ7QYCoE/pe3+ZusHogW9Yln2nxnVPfljW8t4l8go2zPQV7PPkCxa3kqOvOldr52NF4nVKNuYobxPsC1OoxVilu33ZINDQf1Rw2cO+mrOWBMhMD5w0ldziLnrQyeO+mpOfOdzZspZX03KcQGC2I+gNDC3ZkQ7IQLX6x0niW7fZd83ief7Pg7tH0JcfeIlQy+kcllCyk7/O/LUWIsiDbkVv9/bcv0Rqr+MVa//BR0Ob/++AstdKLuK8n4ZZCiKK4e7cikSez2iTjxm2DPjPj8e8wf1zWviXPqpPuHyeuqXb2ZK6WxaqJCuZe7m7qTTp1da6ty7prtbMoOOVro64VktonlmyFZUqB0NybQrkna7l1ndZIhc2k0xbESjdTEf63M7yBQ9bjO2Z+enamvCXcTCphG1zZ3YNSJ9OroJmRkM6UMOZmXPoppzWE4WEX1oY7Xmv1Fs7Zn9nl9gWt+/MTImNU2dprze+6GfmmYlRrYoHZ9dF/whbJrdMTLVdMzEPDZRlOZr48DIVMMQ2wnUN5SpIiNLjaF7z74X1uiNwWbDyNI/qHCd8XBGW1nDLj+F64K9RKtomuWocMvq3fvDjbIk5ahwvc0YGXuS0dbcPBWu1zlLx4mbU23yVbi+s3srkjvdJGeF1WqGTTO6rZsH3BVWjyqae1C1EP0BFaVwvQ60OR1EshZ0GROisJqG23Pt1Gp2q49GqML1OsmK6aMOKbK3OzbCFFZsLwEZ75YySHDpE1olUmFl/XrP02jE5aM0v1972phd2ydDQkvOnjRk4aFL3pDAjiUo/LoTHOuKLMtGzet+lPn4pPpG0eMRQTnLUfi8oTreWc0j7UmUq22x0f9PhbMDCkEhKBQPKJxNYTkgEmYCXiMShZvCXEkHTUXGcPZSpbFS5KZQWq3Qu4XORGIbUTa9eCqsFqt6OGemxSvbUH8sqwUrXK0sl/gM9PnE/XKNCFdYYZozB7VvsPkzU1+Cwhrrph7mGHfOB/X226+1FIU1duztr+Nlnq97L7YbV12SworITqXdvmcw5QvlfielNtU5sDCFNYpO0iJTk2MvbcdEzYqUtIdVLVDhAwvVubIk8D3Po421j88DUufHvvEnL1XhF2Yd7+xqTR6f9wq1XbjCGQCFs/EPKsR8sq6Vxi4bvxVwzCOJBTVXLxzX+BxSrgPKm4arFwM3J1lzQk/D5eunOTjsIqN0h57Gyd0TlbBJYtVa9xFF+NpUedZSM5Ypd6UfifEm4tts5QFkdOvO4RTlLz3GoTZDVpAWxu/WJAI9wn/uRaiPj65x9bC496ixIdjnvXek7Db8HWLfMsnpWWFjAV79w9ZxqlVtzwtXq2XH2Q7I71+Aws/72FyvVz/IsrzlgnmWBX71P5vBV16IQhr3uCUYbxBLVjgPoBAUgkLxgEJQCArFAwpBISgUDzeF1+E7vWMor8IUZi5mFUrzwxa7jRRLnntP0YCqUaNwItG7a3LBUqNT1B5K0fuHboj7RSQM5YjDT7edaIV1HlbeGAwmc82/c8XEK1ytTDkcXKeuEzWUlxf1JRdzOJlq7sXv/YGlKKxwi91pQtBXHfZ12hUNn/mCFNZEaYxPY6YC5QnH9JSwhSl8qCRpFu9Pfb3Yh9M+zlLSmvAmTmGbL/uBoUfkFhY48dvDa30/wUV4I5HembMoLs7b6xGKISOE9Ce5p6qql3//XX3XY88RNVICllvbpM7Hk+WhOab8aps0nzIeRZSoZZQYWaIkyLMNNfnEbprtHAAmQCtlyL6OEq3Q34WRLcqPyb4V+RqlGVuHLMP3FEofXDPrhqsVPYIHM6xtQg86SZgZ1Kj21iWjEjVu3jL5Y3eOgEw3WK3l4vnrRVlx65qTXRFaWlGqJ/Fq1qivVUdKHMvyW3JXMJZPBs9W6BhyZ6YYZlrSW++wXHVIiaCpIg1EpO61F9syePK7fMokt8noekMWInb+rnRLzLgqu/ved3jw8tQePrq6dpp771eUDvPDA+TT25uo2Ho4v4V9n1gjvOXY6+U+P3E4OqC7K75w/DglSZymQftNyUGaxkly+ujtcOVSi7K3xF987KRXdh+jrsKn2OZK57Pl1KTkJLCaD7PejqHjcDwYsXXGyBBWs98WrBtvieWN89FkJucTkTDif9ysgTgKnDwZHMfwY5FGwucQJAqdaS2zKhR1AiIoBIWgEBSCQlAICkEhKASFi1DIokobDV/Y2ePjvG7D4eaA+gcVRnyi9zdj6p/Pg8vncOedqCX+amXxOYA85exle8Xg4XDDYrxQX2g9iyBMYM8uLqEXiHVKyZbbkfFt5GwLCZ9ZhbANIJwrfp3GnU/04xsidpM3X9yb8BcoYJQzEwjvg09M++3ZNyNIbP6bMa1YCjVIchK2IvBFT8PaDaig85bDbmHyHljhaZ7+eDyFS9RXo9jx9K3hMrY57tcPh6SX/fiWPO4vKZ8TeKdg6SRTx4g8qhnRl/p4/oWJIiKpQx7YUpVIJGCrfgqG7CIb9zjSaeNgG7kzhaZyx7JM07Ljy4XWnuXlEtv1P7B9Mv8DltyUV+hIpoIAAAAASUVORK5CYII="
var mimeTypes = require('mimetypes');
var image = newImageData,
mimeType = image.match(/data:([a-zA-Z0-9]+\/[a-zA-Z0-9-.+]+).*,.*/)![1],
fileName = 'test.' + mimeTypes.detectExtension(mimeType),
base64EncodedImageString = image.replace(/^data:image\/\w+;base64,/, ''),
imageBuffer = new Buffer(base64EncodedImageString, 'base64');
// Instantiate the GCP Storage instance
const { Storage } = require('#google-cloud/storage');
const googleCloudStorage = new Storage(firebaseSettings);
const bucket = googleCloudStorage.bucket('projectID.appspot.com');
var file = bucket.file(fileName);
return file.save(imageBuffer, {
metadata: {
contentType: mimeType,
cacheControl: "public,
max-age=300",
// THIS IS THE LINE YOU NEED TO ADD
firebaseStorageDownloadTokens: uuid(),
},
public: true,
validation: 'md5'
}, function (error: any) {
if (error) {
throw 'error';
}
return "https://storage.googleapis.com/share-expanses-dcc9f.appspot.com/" + fileName;
});
}
After that you'll need to click on "Create access token"
#jean-smaug answer is almost complete. Based on the page he linked (https://www.sentinelstand.com/article/guide-to-firebase-storage-download-urls-tokens), the only missing thing is to wrap the firebaseStorageDownloadTokens property inside a metadata object. I've just tested it and it's working fine 👌 No need to create access token afterwards.
In my case I added metadata while uploading and it loading as it showed in image but when I'm refresh page after 3 min I found that it upload correctly , so as Cafn explain if it not matter of metadata you should wait until it loaded
$uploadedObject=$bucket->upload($imageFile, [
'name' => 'Image_Name',
"metadata" => [ "contentType"=> 'image/png'],
]);
I am writing a firebase cloud function that records the download link of a recentally uploaded file to real-time database:
exports.recordImage = functions.storage.object().onFinalize((object) => {
});
"object" gives me access to two variables "selfLink" and "mediaLink" but both of them when entered in a browser they return the following:
Anonymous caller does not have storage.objects.get access to ... {filename}
So, they are not public links. How can I get the public download link within this trigger function?
You have to use the asynchronous getSignedUrl() method, see the doc of the Cloud Storage Node.js library: https://cloud.google.com/nodejs/docs/reference/storage/2.0.x/File#getSignedUrl.
So the following code should do the trick:
.....
const defaultStorage = admin.storage();
.....
exports.recordImage = functions.storage.object().onFinalize(object => {
const bucket = defaultStorage.bucket();
const file = bucket.file(object.name);
const options = {
action: 'read',
expires: '03-17-2025'
};
// Get a signed URL for the file
return file
.getSignedUrl(options)
.then(results => {
const url = results[0];
console.log(`The signed url for ${filename} is ${url}.`);
return true;
})
});
Note that, in order to use the getSignedUrl() method, you need to initialize the Admin SDK with the credentials for a dedicated service account, see this SO Question & Answer firebase function get download url after successfully save image to firebase cloud storage.
*use this function:
function mediaLinkToDownloadableUrl(object) {
var firstPartUrl = object.mediaLink.split("?")[0] // 'https://storage.googleapis.com/download/storage/v1/b/abcbucket.appspot.com/o/songs%2Fsong1.mp3.mp3'
var secondPartUrl = object.mediaLink.split("?")[1] // 'generation=123445678912345&alt=media'
firstPartUrl = firstPartUrl.replace("https://storage.googleapis.com/download/storage", "https://firebasestorage.googleapis.com")
firstPartUrl = firstPartUrl.replace("v1", "v0")
firstPartUrl += "?" + secondPartUrl.split("&")[1]; // 'alt=media'
firstPartUrl += "&token=" + object.metadata.firebaseStorageDownloadTokens
return firstPartUrl
}
this is how your code might look like:
export const onAddSong = functions.storage.object().onFinalize((object) => {
console.log("object: ", object);
var url = mediaLinkToDownloadableUrl(object);
//do anything with url, like send via email or save it in your database in playlist table
//in my case I'm saving it in mongodb database
return new playlistModel({
name: storyName,
mp3Url: url,
ownerEmail: ownerEmail
})
.save() // I'm doing nothing on save complete
.catch(e => {
console.log(e) // log if error occur in database write
})
})
*I have tested this method on mp3 files, I'm sure it will work on all type of files but incase if it doesnt work for you simply go to firebase storage dashboard open any file and copy download url, and try to generate the same url in your code, and edit this answer too if possible
Goal
We would like users to be able to upload images to Google Cloud Storage.
Problem
We could achieve this indirectly with our server as a middle man -- first, the user uploads to our server, then our privileged server can upload to Cloud Storage.
However, we think this is unnecessarily slow, and instead would like the user to upload directly to Cloud Storage.
Proposed Solution
To achieve a direct upload, we generate a Signed URL on our server. The Signed URL specifies an expiration time, and can only be used with the HTTP PUT verb. A user can request a Signed URL, and then - for a limited time only - upload an image to the path specified by the Signed URL.
Problem with the Solution
Is there any way to enforce a maximum file upload size? Obviously we would like to avoid users attempting to upload 20GB files when we expect <1MB files.
It seems like this is an obvious vulnerability, yet I don't know how to address it while still using SignedURLs.
There seems to be a way to do this using Policy Documents (Stack Overflow answer), but the question is over 2 years old now.
For all looking at the answer today be aware that link
x-goog-content-length-range:0,25000
is the way to limit the upload size between 0 and 25000 bytes Cloud Storage.
X-Upload-Content-Length will not work and you are still able to upload larger files
Policy documents are still the right answer. They are documented here: https://cloud.google.com/storage/docs/xml-api/post-object#policydocument
The important part of the policy document you'll need is:
["content-length-range", <min_range>, <max_range>].
Signing content-length should do the trick.
Google Cloud will not allow uploads with larger file size even if the content-length is set to a lower value.
This is how the signed url options should look like:
const writeOptions: GetSignedUrlConfig = {
version: 'v4',
action: 'write',
expires: Date.now() + 900000, // 15 minutes
extensionHeaders: {
"content-length": length // desired length in bytes
}
}
My worked code in NodeJS was following https://blog.koliseo.com/limit-the-size-of-uploaded-files-with-signed-urls-on-google-cloud-storage/. You must use the version v4
public async getPreSignedUrlForUpload(
fileName: string,
contentType: string,
size: number,
bucketName: string = this.configService.get('DEFAULT_BUCKET_NAME'),
): Promise<string> {
const bucket = this.storage.bucket(bucketName);
const file = bucket.file(fileName);
const response = await file.getSignedUrl({
action: 'write',
contentType,
extensionHeaders: {
'X-Upload-Content-Length': size,
},
expires: Date.now() + 60 * 1000, // 1 minute
version: 'v4',
});
const signedUrl = this.maskSignedUrl(response[0], bucketName);
return signedUrl;
}
In the Frontend, We must set the same number of the Size in the header X-Upload-Content-Length
export async function uploadFileToGCP(
signedUrl: string,
file: any
): Promise<any> {
return new Promise((resolve, reject) => {
const xhr = new XMLHttpRequest();
xhr.withCredentials = process.env.NODE_ENV === 'production';
xhr.addEventListener('readystatechange', function () {
if (this.readyState === 4) {
resolve(this.responseText);
}
});
xhr.open('PUT', signedUrl, true);
xhr.setRequestHeader('Content-Type', file.type);
xhr.setRequestHeader('X-Upload-Content-Length', file.size);
xhr.send(file);
});
}
And also don't forget to config the responseHeader in the GS CORS
gsutil cors get gs://asia-item-images
[{"maxAgeSeconds": 3600, "method": ["GET", "OPTIONS", "PUT"], "origin": ["*"], "responseHeader": ["Content-Type", "Access-Control-Allow-Origin", "X-Upload-Content-Length", "X-Goog-Resumable"]}]
You can use X-Upload-Content-Length instead of Content-Length. See blog post here.
On the server side (Java):
Map<String, String> extensionHeaders = new HashMap<>();
extensionHeaders.put("X-Upload-Content-Length", "" + contentLength);
extensionHeaders.put("Content-Type", "application/octet-stream");
var url =
storage.signUrl(
blobInfo,
15,
TimeUnit.MINUTES,
Storage.SignUrlOption.httpMethod(HttpMethod.PUT),
Storage.SignUrlOption.withExtHeaders(extensionHeaders),
Storage.SignUrlOption.withV4Signature()
);
On the client side (typescript):
const response = await fetch(url, {
method: 'PUT',
headers: {
'X-Upload-Content-Length': `${file.size}`,
'Content-Type': 'application/octet-stream',
},
body: file,
});
You will need to set up a cors policy on your bucket:
[
{
"origin": ["https://your-website.com"],
"responseHeader": [
"Content-Type",
"Access-Control-Allow-Origin",
"X-Upload-Content-Length",
"x-goog-resumable"
],
"method": ["PUT", "OPTIONS"],
"maxAgeSeconds": 3600
}
]
Is it possible to copy an existing file on firebase storage without needing to uploading it again?
I need it for a published/working version setup of my app.
There is no method in the Firebase Storage API to make a copy of a file that you've already uploaded.
But Firebase Storage is built on top of Google Cloud Storage, which means that you can use the latter's API too. It looks like gsutil cp is what you're looking for. From the docs:
The gsutil cp command allows you to copy data between your local file system and the cloud, copy data within the cloud, and copy data between cloud storage providers.
Keep in mind that gsutil has full access to your storage bucket. So it is meant to be run on devices you fully trust (such as a server or your own development machine).
Here is an approach I ended up with for my project.
While it covers a broader case and copies all files under folder fromFolder to toFolder, it can be easily adopted to the case from the question (to copy only files one can pass delimiter = "/" - refer to the docs for more details)
const {Storage} = require('#google-cloud/storage');
module.exports = class StorageManager{
constructor() {
this.storage = new Storage();
this.bucket = this.storage.bucket(<bucket-name-here>)
}
listFiles(prefix, delimiter){
return this.bucket.getFiles({prefix, delimiter});
}
deleteFiles(prefix, delimiter){
return this.bucket.deleteFiles({prefix, delimiter, force: true});
}
copyFilesInFolder(fromFolder, toFolder){
return this.listFiles(fromFolder)
.then(([files]) => {
let promiseArray = files.map(file => {
let fileName = file.name
let destination = fileName.replace(fromFolder, toFolder)
console.log("fileName = ", fileName, ", destination = ", destination)
return file.copy(destination)
})
return Promise.all(promiseArray)
})
}
}
I needed to copy folders (including all it's descendants) in Google Cloud Storage. Here is the solution proposed by #vir-us, simplified and for NodeJS.
import type { Bucket } from '#google-cloud/storage';
/** Copy Cloud Storage resources from one folder to another. */
export const copyStorageFiles = async (input: {
bucket: Bucket;
fromFolder: string;
toFolder: string;
logger?: Console['info'];
}) => {
const { bucket, fromFolder, toFolder, logger = console.info } = input;
const [files] = await bucket.getFiles({ prefix: fromFolder });
const promiseArray = files.map((file) => {
const destination = file.name.replace(fromFolder, toFolder);
logger("fileName = ", file.name, ", destination = ", destination);
return file.copy(destination);
});
return Promise.all(promiseArray);
};