trouble with filepicker script loading in meteor - meteor

I'm not having much luck so far in loading the filepicker package in my Meteor project.
What I did:
$cd ~/myMeteorProject
$mrt add filepicker
>>>Done installing smart packages
$head smart.json
>>>{
"packages": {
"router": {},
"filepicker": {}
}
}
$mrt
>>>Stand back while Meteorite does its thing
Done installing smart packages
Ok, everything's ready. Here comes Meteor!
[[[[[ ~/myMeteorProject ]]]]]
=> Meteor server running on: http://localhost:3000/
So at this point all looks like I'd expect. ( I even double checked the contents of the filepicker package and it contains all of what i'd expect, the source to load has the same URL as is found on the filepicker.io site, etc.)
However, when I try to run the following (compiled from coffeescript):
if (Meteor.isClient) {
Meteor.startup(function() {
return filepicker.setKey('A9FiXXdu5RB^GYujfDPwlz'); //not my actual key, don't worry
});}
I get: Uncaught ReferenceError: filepicker is not defined
So, that's kind of a bummer. Any ideas? I've tried removing and re-adding both coffeescript and filepicker. is there some load-order issue? I note that filepicker-load.js has an alert if the script fails to load, which I'm not seeing...

I've had the exact same problem and for me, neither wrapping it in a setTimeout() nor a setInterval() worked.
Ink themselves provide a great non-blocking script in their docs, that queues all filepicker calls until its fully loaded and then executes them in the order they were called. This is how it works:
mrt remove filepicker (if added)
mrt remove loadpicker (if added)
create new file /lib/filepicker.js
/lib/filepicker.js
if (Meteor.isClient) {
(function(a) {
if (window.filepicker) {
return
}
var b = a.createElement("script");
b.type = "text/javascript";
b.async = !0;
b.src = ("https:" === a.location.protocol ? "https:" : "http:") + "//api.filepicker.io/v1/filepicker.js";
var c = a.getElementsByTagName("script")[0];
c.parentNode.insertBefore(b, c);
var d = {};
d._queue = [];
var e = "pick,pickMultiple,pickAndStore,read,write,writeUrl,export,convert,store,storeUrl,remove,stat,setKey,constructWidget,makeDropPane".split(",");
var f = function(a, b) {
return function() {
b.push([a, arguments])
}
};
for (var g = 0; g < e.length; g++) {
d[e[g]] = f(e[g], d._queue)
}
window.filepicker = d
})(document);
filepicker.setKey(YOUR_FILEPICKER_KEY);
}
This client code is a vanilla version of the non-blocking loader taken from https://developers.inkfilepicker.com/docs/web/. Using this, you can set the key once and then use filepicker as you like without worrying about if it's already loaded or not.
Of course you could also modify loadpicker with this, just paste in this code in your loadpicker.js
loadpicker.js
loadPicker = function(key) {
(function(a) {
if (window.filepicker) {
return
}
var b = a.createElement("script");
b.type = "text/javascript";
b.async = !0;
b.src = ("https:" === a.location.protocol ? "https:" : "http:") + "//api.filepicker.io/v1/filepicker.js";
var c = a.getElementsByTagName("script")[0];
c.parentNode.insertBefore(b, c);
var d = {};
d._queue = [];
var e = "pick,pickMultiple,pickAndStore,read,write,writeUrl,export,convert,store,storeUrl,remove,stat,setKey,constructWidget,makeDropPane".split(",");
var f = function(a, b) {
return function() {
b.push([a, arguments])
}
};
for (var g = 0; g < e.length; g++) {
d[e[g]] = f(e[g], d._queue)
}
window.filepicker = d
})(document);
filepicker.setKey(key)
};
Hope this works for you!

your syntax looks good and matches what i use in my app - i'm thinking that since filepicker loads the script by injecting a script element, you have a timing issue (calling setKey before the script is loaded)
maybe hold off on setting the key until the user does something, like
filepicker.setKey("KEY");
filepicker.pickAndStore({...},function(error){...});
or set it with a timeout
Meteor.setTimeout(function(){
filepicker.setKey("KEY");
},1000);
you could also use an interval to check if desired (more tolerant of long load times) - you could adjust this to give up after a certain amount of time
var filepickerInterval = Meteor.setInterval(function(){
if(filepicker){
Meteor.clearInterval(filepickerInterval);
filepicker.setKey("KEY");
}
}, 100);

You also can use the asynchronous javascript snippet to auto-queue calls to the filepicker object until the script loads. See https://developers.inkfilepicker.com/docs/web/#javascript

Related

How to scrape any type of website

I am working on scraping websites, I have tried many technologies to scrape websites.
First of all I used PHP cURL as a scraping tool, and went up to some extent to scrape websites, but then I faced a problem, that was; the PHP cURL couldn't scrape websites that used Ajax to load the website contents/data. And that's what stopped me scraping through PHP.
After a decent research I have found another solution to scrape websites, that were beyond the limitation of Ajax loaded websites etc, and was very powerful and cool to use, they were indeed Phantom JS and Casper JS. I have scraped lot of sites with it.
The problem I faced with these tools was that, these tools works/controlled through the command line interface, for example when you want to run the Phantom/Casper JS code, you need to run it through the command line. And this is my basic problem. What I need is, to write the code in Phantom/Casper JS and I want to have a webpage with admin panel, where I can control these scripts. Currently I am scraping career/jobs listings websites, and I want to automate these tools, to scrape these sites automatically after a given time, to stay updated with the employers sites, who posts new jobs.
For instance, I have code for each website separately and I manually execute each file through the command line and then wait for it to finish scraping and then I continue with second one and so on. What I want to have is, I write a script in JavaScript (preferably in Node JS - but not compulsory) which will execute the scraper code after a specific instance, and then will start scraping all of the websites in the background.
I can do the automation, its not a problem, but the problem is, I am unable to connect the Phantom/Casper JS with the website, even I tried Spooky JS which connects Phantom/Casper JS with Node JS, but unfortunately it doesn't work for me, and its alot messy.
Is there any other tool that's powerful like these two, and I can easily interact with them through a webpage ?
Continuing my own research for scrapping sites, I was unable to find any perfect solution. But the powerful solution I came up with is to use Phantom JS module with Node JS. You can find this module here.
For installation Guide follow this documentation. Phantom JS is used asynchronously in node JS and then its alot easier to get the results, and really easy to interact with it using, express JS on server side and Ajax or Socket.io on client side to enhance the functionality.
Below is my code which I came up with :
const phantom = require('phantom');
const ev = require('events');
const event = new ev.EventEmitter();
var MAIN_URL,
TOTAL_PAGES,
TOTAL_JOBS,
PAGE_DATA_COUNTER = 0,
PAGE_COUNTER = 0,
PAGE_JOBS_DETAILS = [],
IND_JOB_DETAILS = [],
JOB_NUMBER = 1,
CURRENT_PAGE = 1,
PAGE_WEIGHT_TIME,
CLICK_NEXT_TIME,
CURRENT_WEBSITE,
CURR_WEBSITE_LINK,
CURR_WEBSITE_NAME,
CURR_WEBSITE_INDEX,
PH_INSTANCE,
PH_PAGE;
function InitScrap() {
// Initiate the Data
this.init = async function(url) {
MAIN_URL = url;
PH_INSTANCE = await phantom.create(),
PH_PAGE = await PH_INSTANCE.createPage();
console.log("Scrapper Initiated, Please wait...")
return "success";
}
// Load the Basic Page First
this.loadPage = async function(pageLoadWait) {
var status = await PH_PAGE.open(MAIN_URL),
w;
if (status == "success") {
console.log("Page Loaded . . .");
if (pageLoadWait !== undefined && pageLoadWait !== null && pageLoadWait !== false) {
let p = new Promise(function(res, rej) {
setTimeout(async function() {
console.log("Page After 5 Seconds");
PH_PAGE.render("new.png");
TOTAL_PAGES = await PH_PAGE.evaluate(function() {
return document.getElementsByClassName("flatten pagination useIconFonts")[0].textContent.match(/\d+/g)[1];
});
TOTAL_JOBS = await PH_PAGE.evaluate(function() {
return document.getElementsByClassName("jobCount")[0].textContent.match(/\d+/g)[0];
});
res({
p: TOTAL_PAGES,
j: TOTAL_JOBS,
s: true
});
}, pageLoadWait);
})
return await p;
}
}
}
function ScrapData(opts) {
var scrap = new InitScrap();
scrap.init("https://www.google.com/").then(function(init_res) {
if (init_res == "success") {
scrap.loadPage(opts.pageLoadWait).then(function(load_res) {
console.log(load_res);
if (load_res.s === true) {
scrap.evaluatePage().then(function(ev_page_res) {
console.log("Page Title : " + ev_page_res);
scrap.evaluateJobsDetails().then(function(ev_jobs_res) {
console.log(ev_jobs_res);
})
})
}
return
})
}
});
return scrap;
}
module.exports = {
ScrapData
};
}

"Dependency (AJAX)" in Application Insights

About 50% of our data points logged are "Dependency (AJAX)".
Where does this come from ?
I think these "Dependency (AJAX)" began to show up after I enabled Web sockets in the Application Settings of my web app (we are using signalr).
The Property "Command" of these data points have a value of "/signalr/ping", so this definitely has something to do with signalr.
I would like to exclude those, as it's using too much of my free plan available data points.
Microsoft.ApplicationInsights.DependencyCollector.DependencyTrackingTelemetryModule is already disabled in my ApplicationInsights.config.
EDIT
I tried to disable Web sockets, but still getting those logs.
EDIT 2016-02-24
As suggested by Alex, I set disableAjaxTracking:true, but it didn't help.
Here is the appInsights code in the <head> of my html.
<script type="text/javascript">
var appInsights = window.appInsights || function (config) {
function r(config) { t[config] = function () { var i = arguments; t.queue.push(function () { t[config].apply(t, i) }) } } var t = { config: config }, u = document, e = window, o = "script", s = u.createElement(o), i, f; for (s.src = config.url || "//az416426.vo.msecnd.net/scripts/a/ai.0.js", u.getElementsByTagName(o)[0].parentNode.appendChild(s), t.cookie = u.cookie, t.queue = [], i = ["Event", "Exception", "Metric", "PageView", "Trace"]; i.length;) r("track" + i.pop()); return r("setAuthenticatedUserContext"), r("clearAuthenticatedUserContext"), config.disableExceptionTracking || (i = "onerror", r("_" + i), f = e[i], e[i] = function (config, r, u, e, o) { var s = f && f(config, r, u, e, o); return s !== !0 && t["_" + i](config, r, u, e, o), s }), t
}({
instrumentationKey: "#Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.Active.InstrumentationKey",
disableAjaxTracking: true
});
window.appInsights = appInsights;
appInsights.trackPageView();
</script>
#if (Request.IsAuthenticated)
{
<script>
appInsights.setAuthenticatedUserContext("#User.Identity.Name".replace(/[,;=| ]+/g, "_"));
</script>
}
Starting from the end of December 2015 Application Insights JavaScript SDK is automatically collecting AJAX requests. Here's a blog post about it.
Since its enablement we also introduced a maximum cap of AJAX requests that can be logged per page view, the default is 500, but you can change this setting by adding this property to your snippet (more on this in the blog and linked documentation):
maxAjaxCallsPerView: <number>
You can also disable AJAX request collection alltogether by using this setting:
disableAjaxTracking: true
EDIT: looks like disableAjaxTracking is broken as of 2/24/2016, until it is fixed, the mitigation is to use
maxAjaxCallsPerView: 0

meteor restivus how to read multiple queryParams

I am building an API with Restivus in Meteor.
In a custom route I would like to have multiple values as queryParams like this (e.g. value1 and value2):
...domain/api/update?key=1234&value1=10
How do I get them in endpoint function?
When I try this I get undefined:
var query = this.queryParams.key // result: 1234
var value1 = this.queryParams.value1 // result: undefined
Update
This is my new fresh code with the same result.
Use a standard Meteor project. Meteor version 1.0.3.2
// Create collection
Posts = new Mongo.Collection("posts");
if (Meteor.isServer) {
Meteor.startup(function () {
// RESTIVUS
// Global configuration
Restivus.configure({
useAuth: false,
prettyJson: true
});
// Given the url: "/posts?key=1234&value1=10"
Restivus.addRoute('posts', {
get: function () {
var key = this.queryParams.key;
var value1 = this.queryParams.value1;
console.log("key: " + key); // result: 1234
console.log("value1: " + value1); // result: undefined
}
});
});
}
This is the solution to the problem. Taken from here:
https://github.com/kahmali/meteor-restivus/issues/16
You're using curl to test, right? Well apparently (and don't feel bad for not knowing this, because neither did I), the & symbol means that the previous command will be run in the background, so the query params were just being truncated once the curl command reached the & for the second query param. All you have to do is wrap the URL in quotes, and voila! Try this command instead:
curl "http://testivus.meteor.com/api/posts?key=1234&value1=10"
That should work. So if you had just punched that URL into a browser or used a mored advanced REST client, you would have seen the extra query param defined. I got the answer from this StackOverflow question.

Meteor Spiderable, errors

when i test ?_escaped_fragment_= , i get
TypeError: 'undefined' is not a function (evaluating 'document.querySelectorAll.bind(document)')
http://localhost:3000/packages/material-design.js?e252ae03c7066a6ce33a348a22662a73cee8811e:75
http://localhost:3000/packages/material-design.js?e252ae03c7066a6ce33a348a22662a73cee8811e:315
http://localhost:3000/packages/material-design.js?e252ae03c7066a6ce33a348a22662a73cee8811e:318
http://localhost:3000/packages/material-design.js?e252ae03c7066a6ce33a348a22662a73cee8811e:778
The html in the body does show up but I do not get any meta tags and there is a huge blank space in the head before the title.
i followed http://www.meteorpedia.com/read/spiderable/ and ran phantomjs phantomtest.js
❯ phantomjs phantomtest.js [17:50:01]
Loading page...
Page load status: success
not ready, Meteor undefined
i got this.
Any idea what's wrong? Thanks.
In phantomjs, which is used by spiderable, the bind method is not supported. If you're the owner of material-design I would suggest replacing bind with _.bind. Otherwise, you can add a polyfill to your project to make sure that Function.prototype.bind is properly defined.
EDIT
To make sure your browser supports bind put this code somewhere in your code base:
if (!Function.prototype.bind) {
Function.prototype.bind = function(oThis) {
if (typeof this !== 'function') {
// closest thing possible to the ECMAScript 5
// internal IsCallable function
throw new TypeError('Function.prototype.bind - what is trying to be bound is not callable');
}
var aArgs = Array.prototype.slice.call(arguments, 1),
fToBind = this,
fNOP = function() {},
fBound = function() {
return fToBind.apply(this instanceof fNOP && oThis
? this
: oThis,
aArgs.concat(Array.prototype.slice.call(arguments)));
};
fNOP.prototype = this.prototype;
fBound.prototype = new fNOP();
return fBound;
};
}
The above implementation is copy/pasted from here.

Error: Cannot apply $push modifier to non-array in Meteorjs

This is a function to add a dictionary as a subitem of a main document.
addSubItem = function(id, data) {
s = Item.findOne(id);
if(s){
Item.update({_id:id},{$push:{'subItemsList':data}});
}
};
I verify the data passed and are a valid main document id a subitem data.
data = {num:1, value: 'Subitem1'};
This works fine yesterday with the latest version of meteor (0.4.2), but today I get this message at javascript console:
Error: Cannot apply $push modifier to non-array
I created and setup a new project and I still get the same.
Thanks to Lloyd, Nice workaround and thanks for the javascript crash course (i'm a begginer on it), but I found the solution: (note the "$push")
addSubItem = function(id, data) {
s = Item.findOne(id);
if(s){
Item.update({_id:id},{"$push":{'subItemsList':data}});
}
};
try this:
addSubItem = function(id, data) {
s = Item.findOne(id);
if(s){
s.subItemsList = s.subItemsList || [];
s.subItemsList.push(data);
Item.update(id, s);
}
};

Resources