I'm trying to scrape GDP data from BEA website. The following codes returned empty. Can anyone help out what should be changed in the codes? Thanks!
function test() {
const url = 'https://apps.bea.gov/iTable/iTable.cfm?reqid=19&step=3&isuri=1&nipa_table_list=5&categories=survey';
var res = UrlFetchApp.fetch(url, { muteHttpExceptions: true }).getContentText();
var $ = Cheerio.load(res);
var table = $("#tbl_wrapper").find('td').toArray().map(el => $(el).text());
console.log(table)
}
I've started working with firebase storage and firebase functions recently. Right now I've been developing file upload from functions to storage .
I've got it working (upload is done and file appears on the storage section), yet, the image, stays like this forever (loading forever on the right side):
I though that it was an error from my code. Yet, if I open Google Cloud Platform - Storage, the image appears and I can open it and preview it.
In firebase storage, if I open the image (select on it and click open), it returns the following url: https://console.firebase.google.com/u/0/undefined
What may I been doing wrong? Here's the code I'm using:
function uploadImage() {
const newImageData = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAAAAgVBMVEX///8AAAAEBASAgIDr6+vw8PBYWFjU1NTGxsbz8/P29vb8/Py1tbVhYWHd3d1ra2vk5OS/v78pKSlTU1NOTk6Tk5OpqanNzc13d3dKSkplZWWbm5s5OTkfHx+NjY2GhoYcHBw9PT0TExOioqJ7e3soKCiurq5CQkI6OjoXFxcwMDAuPQWoAAAIJ0lEQVR4nO2daXuqPBCGVfYtIbKLgorLKf//B75ga2sPAdmS8F5n7m/VXgyPISGZzExWKwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOAVy6wwlCZG/YUl+vamYLoy0lAkORWbdZNN/YUUVf8ju6bomx2Kq+mEBL5K0UVD9QNCdM0Vfds9QSQMnYTWaN1sEicMCRJ9+29QSCqp28HiftiqUkoU0TLaMDLsl8PbrtGWpY8zQ7QYCoE/pe3+ZusHogW9Yln2nxnVPfljW8t4l8go2zPQV7PPkCxa3kqOvOldr52NF4nVKNuYobxPsC1OoxVilu33ZINDQf1Rw2cO+mrOWBMhMD5w0ldziLnrQyeO+mpOfOdzZspZX03KcQGC2I+gNDC3ZkQ7IQLX6x0niW7fZd83ief7Pg7tH0JcfeIlQy+kcllCyk7/O/LUWIsiDbkVv9/bcv0Rqr+MVa//BR0Ob/++AstdKLuK8n4ZZCiKK4e7cikSez2iTjxm2DPjPj8e8wf1zWviXPqpPuHyeuqXb2ZK6WxaqJCuZe7m7qTTp1da6ty7prtbMoOOVro64VktonlmyFZUqB0NybQrkna7l1ndZIhc2k0xbESjdTEf63M7yBQ9bjO2Z+enamvCXcTCphG1zZ3YNSJ9OroJmRkM6UMOZmXPoppzWE4WEX1oY7Xmv1Fs7Zn9nl9gWt+/MTImNU2dprze+6GfmmYlRrYoHZ9dF/whbJrdMTLVdMzEPDZRlOZr48DIVMMQ2wnUN5SpIiNLjaF7z74X1uiNwWbDyNI/qHCd8XBGW1nDLj+F64K9RKtomuWocMvq3fvDjbIk5ahwvc0YGXuS0dbcPBWu1zlLx4mbU23yVbi+s3srkjvdJGeF1WqGTTO6rZsH3BVWjyqae1C1EP0BFaVwvQ60OR1EshZ0GROisJqG23Pt1Gp2q49GqML1OsmK6aMOKbK3OzbCFFZsLwEZ75YySHDpE1olUmFl/XrP02jE5aM0v1972phd2ydDQkvOnjRk4aFL3pDAjiUo/LoTHOuKLMtGzet+lPn4pPpG0eMRQTnLUfi8oTreWc0j7UmUq22x0f9PhbMDCkEhKBQPKJxNYTkgEmYCXiMShZvCXEkHTUXGcPZSpbFS5KZQWq3Qu4XORGIbUTa9eCqsFqt6OGemxSvbUH8sqwUrXK0sl/gM9PnE/XKNCFdYYZozB7VvsPkzU1+Cwhrrph7mGHfOB/X226+1FIU1duztr+Nlnq97L7YbV12SworITqXdvmcw5QvlfielNtU5sDCFNYpO0iJTk2MvbcdEzYqUtIdVLVDhAwvVubIk8D3Po421j88DUufHvvEnL1XhF2Yd7+xqTR6f9wq1XbjCGQCFs/EPKsR8sq6Vxi4bvxVwzCOJBTVXLxzX+BxSrgPKm4arFwM3J1lzQk/D5eunOTjsIqN0h57Gyd0TlbBJYtVa9xFF+NpUedZSM5Ypd6UfifEm4tts5QFkdOvO4RTlLz3GoTZDVpAWxu/WJAI9wn/uRaiPj65x9bC496ixIdjnvXek7Db8HWLfMsnpWWFjAV79w9ZxqlVtzwtXq2XH2Q7I71+Aws/72FyvVz/IsrzlgnmWBX71P5vBV16IQhr3uCUYbxBLVjgPoBAUgkLxgEJQCArFAwpBISgUDzeF1+E7vWMor8IUZi5mFUrzwxa7jRRLnntP0YCqUaNwItG7a3LBUqNT1B5K0fuHboj7RSQM5YjDT7edaIV1HlbeGAwmc82/c8XEK1ytTDkcXKeuEzWUlxf1JRdzOJlq7sXv/YGlKKxwi91pQtBXHfZ12hUNn/mCFNZEaYxPY6YC5QnH9JSwhSl8qCRpFu9Pfb3Yh9M+zlLSmvAmTmGbL/uBoUfkFhY48dvDa30/wUV4I5HembMoLs7b6xGKISOE9Ce5p6qql3//XX3XY88RNVICllvbpM7Hk+WhOab8aps0nzIeRZSoZZQYWaIkyLMNNfnEbprtHAAmQCtlyL6OEq3Q34WRLcqPyb4V+RqlGVuHLMP3FEofXDPrhqsVPYIHM6xtQg86SZgZ1Kj21iWjEjVu3jL5Y3eOgEw3WK3l4vnrRVlx65qTXRFaWlGqJ/Fq1qivVUdKHMvyW3JXMJZPBs9W6BhyZ6YYZlrSW++wXHVIiaCpIg1EpO61F9syePK7fMokt8noekMWInb+rnRLzLgqu/ved3jw8tQePrq6dpp771eUDvPDA+TT25uo2Ho4v4V9n1gjvOXY6+U+P3E4OqC7K75w/DglSZymQftNyUGaxkly+ujtcOVSi7K3xF987KRXdh+jrsKn2OZK57Pl1KTkJLCaD7PejqHjcDwYsXXGyBBWs98WrBtvieWN89FkJucTkTDif9ysgTgKnDwZHMfwY5FGwucQJAqdaS2zKhR1AiIoBIWgEBSCQlAICkEhKASFi1DIokobDV/Y2ePjvG7D4eaA+gcVRnyi9zdj6p/Pg8vncOedqCX+amXxOYA85exle8Xg4XDDYrxQX2g9iyBMYM8uLqEXiHVKyZbbkfFt5GwLCZ9ZhbANIJwrfp3GnU/04xsidpM3X9yb8BcoYJQzEwjvg09M++3ZNyNIbP6bMa1YCjVIchK2IvBFT8PaDaig85bDbmHyHljhaZ7+eDyFS9RXo9jx9K3hMrY57tcPh6SX/fiWPO4vKZ8TeKdg6SRTx4g8qhnRl/p4/oWJIiKpQx7YUpVIJGCrfgqG7CIb9zjSaeNgG7kzhaZyx7JM07Ljy4XWnuXlEtv1P7B9Mv8DltyUV+hIpoIAAAAASUVORK5CYII="
var mimeTypes = require('mimetypes');
var image = newImageData,
mimeType = image.match(/data:([a-zA-Z0-9]+\/[a-zA-Z0-9-.+]+).*,.*/)![1],
fileName = 'test.' + mimeTypes.detectExtension(mimeType),
base64EncodedImageString = image.replace(/^data:image\/\w+;base64,/, ''),
imageBuffer = new Buffer(base64EncodedImageString, 'base64');
// Instantiate the GCP Storage instance
const { Storage } = require('#google-cloud/storage');
const googleCloudStorage = new Storage(firebaseSettings);
const bucket = googleCloudStorage.bucket('projectID.appspot.com');
var file = bucket.file(fileName);
return file.save(imageBuffer, {
metadata: { contentType: mimeType, cacheControl: "public, max-age=300" },
public: true,
validation: 'md5'
}, function (error: any) {
if (error) {
throw 'error';
}
return "https://storage.googleapis.com/share-expanses-dcc9f.appspot.com/" + fileName;
});
}
Thanks for the help
Haven't been able to test the solution given by Firebase, but here's the transcript of the response:
The problem that you are facing could be because of two reasons. The
first one is how you are uploading the files, via the Firebase
Console, using any Admin SDK, or via the gsutil command. If using the
Admin SDK option, the problem is a known issue where the required
metadata doesn’t exist, fortunately there is a workaround, you can try
this script to solve this issue.
Now, the second one is related to the network if you are using
comcast, please, try on a different network to see if this issue is
related to that.
When you save an image to firebase, you need to provide an access token in metadata : firebaseStorageDownloadTokens. It has to be an uuid.
More info can be found here : https://www.sentinelstand.com/article/guide-to-firebase-storage-download-urls-tokens
const { v4: uuid } = require("uuid")
function uploadImage() {
const newImageData = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAAAAgVBMVEX///8AAAAEBASAgIDr6+vw8PBYWFjU1NTGxsbz8/P29vb8/Py1tbVhYWHd3d1ra2vk5OS/v78pKSlTU1NOTk6Tk5OpqanNzc13d3dKSkplZWWbm5s5OTkfHx+NjY2GhoYcHBw9PT0TExOioqJ7e3soKCiurq5CQkI6OjoXFxcwMDAuPQWoAAAIJ0lEQVR4nO2daXuqPBCGVfYtIbKLgorLKf//B75ga2sPAdmS8F5n7m/VXgyPISGZzExWKwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOAVy6wwlCZG/YUl+vamYLoy0lAkORWbdZNN/YUUVf8ju6bomx2Kq+mEBL5K0UVD9QNCdM0Vfds9QSQMnYTWaN1sEicMCRJ9+29QSCqp28HiftiqUkoU0TLaMDLsl8PbrtGWpY8zQ7QYCoE/pe3+ZusHogW9Yln2nxnVPfljW8t4l8go2zPQV7PPkCxa3kqOvOldr52NF4nVKNuYobxPsC1OoxVilu33ZINDQf1Rw2cO+mrOWBMhMD5w0ldziLnrQyeO+mpOfOdzZspZX03KcQGC2I+gNDC3ZkQ7IQLX6x0niW7fZd83ief7Pg7tH0JcfeIlQy+kcllCyk7/O/LUWIsiDbkVv9/bcv0Rqr+MVa//BR0Ob/++AstdKLuK8n4ZZCiKK4e7cikSez2iTjxm2DPjPj8e8wf1zWviXPqpPuHyeuqXb2ZK6WxaqJCuZe7m7qTTp1da6ty7prtbMoOOVro64VktonlmyFZUqB0NybQrkna7l1ndZIhc2k0xbESjdTEf63M7yBQ9bjO2Z+enamvCXcTCphG1zZ3YNSJ9OroJmRkM6UMOZmXPoppzWE4WEX1oY7Xmv1Fs7Zn9nl9gWt+/MTImNU2dprze+6GfmmYlRrYoHZ9dF/whbJrdMTLVdMzEPDZRlOZr48DIVMMQ2wnUN5SpIiNLjaF7z74X1uiNwWbDyNI/qHCd8XBGW1nDLj+F64K9RKtomuWocMvq3fvDjbIk5ahwvc0YGXuS0dbcPBWu1zlLx4mbU23yVbi+s3srkjvdJGeF1WqGTTO6rZsH3BVWjyqae1C1EP0BFaVwvQ60OR1EshZ0GROisJqG23Pt1Gp2q49GqML1OsmK6aMOKbK3OzbCFFZsLwEZ75YySHDpE1olUmFl/XrP02jE5aM0v1972phd2ydDQkvOnjRk4aFL3pDAjiUo/LoTHOuKLMtGzet+lPn4pPpG0eMRQTnLUfi8oTreWc0j7UmUq22x0f9PhbMDCkEhKBQPKJxNYTkgEmYCXiMShZvCXEkHTUXGcPZSpbFS5KZQWq3Qu4XORGIbUTa9eCqsFqt6OGemxSvbUH8sqwUrXK0sl/gM9PnE/XKNCFdYYZozB7VvsPkzU1+Cwhrrph7mGHfOB/X226+1FIU1duztr+Nlnq97L7YbV12SworITqXdvmcw5QvlfielNtU5sDCFNYpO0iJTk2MvbcdEzYqUtIdVLVDhAwvVubIk8D3Po421j88DUufHvvEnL1XhF2Yd7+xqTR6f9wq1XbjCGQCFs/EPKsR8sq6Vxi4bvxVwzCOJBTVXLxzX+BxSrgPKm4arFwM3J1lzQk/D5eunOTjsIqN0h57Gyd0TlbBJYtVa9xFF+NpUedZSM5Ypd6UfifEm4tts5QFkdOvO4RTlLz3GoTZDVpAWxu/WJAI9wn/uRaiPj65x9bC496ixIdjnvXek7Db8HWLfMsnpWWFjAV79w9ZxqlVtzwtXq2XH2Q7I71+Aws/72FyvVz/IsrzlgnmWBX71P5vBV16IQhr3uCUYbxBLVjgPoBAUgkLxgEJQCArFAwpBISgUDzeF1+E7vWMor8IUZi5mFUrzwxa7jRRLnntP0YCqUaNwItG7a3LBUqNT1B5K0fuHboj7RSQM5YjDT7edaIV1HlbeGAwmc82/c8XEK1ytTDkcXKeuEzWUlxf1JRdzOJlq7sXv/YGlKKxwi91pQtBXHfZ12hUNn/mCFNZEaYxPY6YC5QnH9JSwhSl8qCRpFu9Pfb3Yh9M+zlLSmvAmTmGbL/uBoUfkFhY48dvDa30/wUV4I5HembMoLs7b6xGKISOE9Ce5p6qql3//XX3XY88RNVICllvbpM7Hk+WhOab8aps0nzIeRZSoZZQYWaIkyLMNNfnEbprtHAAmQCtlyL6OEq3Q34WRLcqPyb4V+RqlGVuHLMP3FEofXDPrhqsVPYIHM6xtQg86SZgZ1Kj21iWjEjVu3jL5Y3eOgEw3WK3l4vnrRVlx65qTXRFaWlGqJ/Fq1qivVUdKHMvyW3JXMJZPBs9W6BhyZ6YYZlrSW++wXHVIiaCpIg1EpO61F9syePK7fMokt8noekMWInb+rnRLzLgqu/ved3jw8tQePrq6dpp771eUDvPDA+TT25uo2Ho4v4V9n1gjvOXY6+U+P3E4OqC7K75w/DglSZymQftNyUGaxkly+ujtcOVSi7K3xF987KRXdh+jrsKn2OZK57Pl1KTkJLCaD7PejqHjcDwYsXXGyBBWs98WrBtvieWN89FkJucTkTDif9ysgTgKnDwZHMfwY5FGwucQJAqdaS2zKhR1AiIoBIWgEBSCQlAICkEhKASFi1DIokobDV/Y2ePjvG7D4eaA+gcVRnyi9zdj6p/Pg8vncOedqCX+amXxOYA85exle8Xg4XDDYrxQX2g9iyBMYM8uLqEXiHVKyZbbkfFt5GwLCZ9ZhbANIJwrfp3GnU/04xsidpM3X9yb8BcoYJQzEwjvg09M++3ZNyNIbP6bMa1YCjVIchK2IvBFT8PaDaig85bDbmHyHljhaZ7+eDyFS9RXo9jx9K3hMrY57tcPh6SX/fiWPO4vKZ8TeKdg6SRTx4g8qhnRl/p4/oWJIiKpQx7YUpVIJGCrfgqG7CIb9zjSaeNgG7kzhaZyx7JM07Ljy4XWnuXlEtv1P7B9Mv8DltyUV+hIpoIAAAAASUVORK5CYII="
var mimeTypes = require('mimetypes');
var image = newImageData,
mimeType = image.match(/data:([a-zA-Z0-9]+\/[a-zA-Z0-9-.+]+).*,.*/)![1],
fileName = 'test.' + mimeTypes.detectExtension(mimeType),
base64EncodedImageString = image.replace(/^data:image\/\w+;base64,/, ''),
imageBuffer = new Buffer(base64EncodedImageString, 'base64');
// Instantiate the GCP Storage instance
const { Storage } = require('#google-cloud/storage');
const googleCloudStorage = new Storage(firebaseSettings);
const bucket = googleCloudStorage.bucket('projectID.appspot.com');
var file = bucket.file(fileName);
return file.save(imageBuffer, {
metadata: {
contentType: mimeType,
cacheControl: "public,
max-age=300",
// THIS IS THE LINE YOU NEED TO ADD
firebaseStorageDownloadTokens: uuid(),
},
public: true,
validation: 'md5'
}, function (error: any) {
if (error) {
throw 'error';
}
return "https://storage.googleapis.com/share-expanses-dcc9f.appspot.com/" + fileName;
});
}
After that you'll need to click on "Create access token"
#jean-smaug answer is almost complete. Based on the page he linked (https://www.sentinelstand.com/article/guide-to-firebase-storage-download-urls-tokens), the only missing thing is to wrap the firebaseStorageDownloadTokens property inside a metadata object. I've just tested it and it's working fine 👌 No need to create access token afterwards.
In my case I added metadata while uploading and it loading as it showed in image but when I'm refresh page after 3 min I found that it upload correctly , so as Cafn explain if it not matter of metadata you should wait until it loaded
$uploadedObject=$bucket->upload($imageFile, [
'name' => 'Image_Name',
"metadata" => [ "contentType"=> 'image/png'],
]);
I uploaded a raw String 'Test' to the firebase storage using the sample provided here and it went through successfully.
But when I tried to "download" the string I uploaded, using the sample below, apparently he only example on how to download data from firebase storage it returns the url of the string file.
storageRef.child('path/to/string').getDownloadURL().then(function(url) {
// I get the url of course
}).catch(function(error) {
// Handle any errors
});
How do I get the contents of the file from the callback url which is 'Test' (The string I uploaded.)
The short answer is that in the Web Storage SDK you can only get a download URL that represents that data. You'll need to "download" the file using an XMLHttpRequest (or equivalent):
storageRef.child('path/to/string').getDownloadURL().then(function(url) {
var XMLHttp = new XMLHttpRequest();
XMLHttp.onreadystatechange = function() {
if (xmlHttp.readyState == 4 && xmlHttp.status == 200)
var response = xmlHttp.responseText; // should have your text
}
XMLHttp.open("GET", url, true); // true for asynchronous
XMLHttp.send(null);
}).catch(function(error) {
// Handle any errors from Storage
});
I built a simple Instagram app using the Instagram API and Angular.
Now I would like to permanently store the users search (aka response from Instagram server for each hashtag entered) to Firebase.
I got the Instagram API call working and saved it as $scope.instadata but how do I add $scope.instadata to Firebase using $add()?
I'm getting err:
angular.js:11500 Error: Firebase.set failed: First argument contains a function..
var app = angular.module('InstagramApp', ['firebase']);
app.controller('MainController', function ($scope, $http, $firebaseArray) {
// search posts based on hashtag
function getHash() {
var base = 'https://api.instagram.com/v1/tags/'
var service = document.getElementById('hashtag').value
var apiKey = '####'
var callback = 'JSON_CALLBACK'
var url = base + service + '/media/recent?access_token=' + apiKey + '&callback=' + callback
// use jsonp method to prevent CORS error
$http.jsonp(url).then(function(data) {
$scope.instadata = data
console.log("returned json object: ")
console.log($scope.instadata)
// set up firebase reference
var ref = new Firebase('https://myapp.firebaseio.com/fbdata')
// create a synchronized array
$scope.fbdata = $firebaseArray(ref)
// add new items to the array RETURNS ERROR
$scope.fbdata.$add($scope.instadata).then(
function(p){
console.log(p.key())
console.log($scope.instadata)
},
function(err){
console.log("The expected Firebase action failed to occur this was your error: " + err)
})
}
})
}
document.getElementById('hashtag').addEventListener('click', getHash, false)
})
It's my first time asking a question here, working with API's, Angular and Firebase! I hope that's ok!
I have tried to download a file from a method of my Web API Controller (Asp.net MVC 6). So, my file is created in a api controller and I want download my file, already in excel format, with angularjs. But, I don't know how to do.
Here is my method in my api controller: (it works, I already use this methods and class in an another project Asp.net without angularj)
#region downloadFile
[Route("Download")]
public ActionResult Download()
{
string companyName = Context.Session.GetString("companyName") as string;
string fileDownloadName = Context.Session.GetString("fileDownloadName") as string;
string baseFolder = System.IO.Path.Combine("C:\\Temp");
string folder = System.IO.Path.Combine(baseFolder, companyName);
TempFile tempFile = new TempFile(folder, fileDownloadName);
tempFile.FindFileInDirectory();
return File(tempFile._pathDirectory + "\\"+tempFile._fileName, "Application/" + tempFile._fileExt, tempFile._fileName);
}
#endregion
This method returns an excel file. Now, I would download this file that is send in angularjs with a http request.
I have tried to use the method saveAs from fileSaver.js. I have added the js in my project but when I want use it, the method is always undefined.
var requestDownloadFile =
$http({
url: '/api/customers/Download'
, responseType: 'arraybuffer'
, headers: {
'contentType': 'application/json'
, 'Accept': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
}
});
requestDownloadFile.success(function (data) {
saveAs(data, 'test.xlsx');
});
I tried to use this too in my success method:
var blob = new Blob([data], { type: "application/vnd.openxmlformats- officedocument.spreadsheetml.sheet" })
var objectUrl = URL.createObjectURL(blob);
window.open(objectUrl);
It run a download file, but the file is corrupted.
So it's not realy effective because I have created a file in API and I have tried to recover an arrayBuffer but I'm blocked so I try.
All support is welcome. Thanks
So, I complicated life. I just had to do this in the success:
window.open('/api/customers/Download', '_blank', '');
window.open('link/to/method/api', '_blank', '');