I'm trying to parse big csv-file in Meteor app and to show progress with Semantic UI Progress component. But it freezes and shows only final result.
Template.ordersImport.events({
'click button': function (e) {
e.preventDefault();
Papa.parse($('#importedFile')[0].files[0], {
delimiter: ';',
newline: '\n',
header: true,
fastMode: true,
complete: function (result) {
$('form').hide();
$('#progress').show();
var totalSize = result.data.length;
_.each(result.data, function (item, index) {
var progress = (index + 1) / totalSize * 100;
$('#progress').progress({
percent: progress
});
});
}
});
}
});
Code in the browser runs in a single thread, so even though you update a progress variable, it won't get rendered to the UI until the parsing of the file finishes.
It's not good practice to do lengthy processing in the UI - it becomes unresponsive, and may crash the browser, especially for large files.
I would recommend passing the file to the server, and doing the processing there.
https://forums.meteor.com/t/how-to-use-web-worker/17511
This answer might help too if you want to keep the processing in the browser
How to run an unblocking background task in a Meteor/JavaScript client?
Related
I am trying to make a fairly big meteor app, and I noticed that it has gotten slower over the past few days and I read on the meteor forum that publications can cause slow loading times. After I refresh the page when making a change in the application itself (code change), it usually takes between 1-2 minutes for a single change. Is there anything wrong with my publications? Although, when the page is loaded, and I reload it loads up really fast.
if(Meteor.isServer){
Meteor.publish('notes', function () {
return Notes.find()
});
Meteor.publish('users', function () {
return Meteor.users.find()
});
Meteor.publish("user", function(){
return Meteor.user()
})
Meteor.publish('notes-newest', function () {
return Notes.find({}, {sort: {createdAt: -1}, limit: 10});
});
}
document example:
let noteInfo = { title, subject, description, imageURL, userId, userEmail, createdAt }
let title = this.refs.title.value;
let subject = this.refs.subject.value;
let description = this.refs.description.value;
let allUrls = [this.refs.imageURL.value].concat(this.state.urls);
let imageURL = allUrls.filter(function(entry) { return entry.trim() != ''; });
let userId = Meteor.userId();
let userEmail = Meteor.user().emails[0].address;
let createdAt = Date.parse(new Date());
This is a very broad performance tuning question. You haven't told us how many documents are in your collections or how large these documents are. Some possible issues:
You are over-publishing either by publishing too many documents or because your documents are too big. For example when you do:
Meteor.publish('notes', function () {
return Notes.find()
});
If there are 100,000 Notes documents at 100 bytes each then that's 10 MB that needs to go over the network to the client. This is also the case if there are 1,000 notes documents at 10KB each.
Solution: Limit the number of documents with limit and/or reduce the number of fields transmitted with fields:
Meteor.publish('notes', function () {
return Notes.find({},{ limit: 100, fields: { key1: 1, key2: 1 }});
});
Your collection is missing one or more indexes. When you do:
Meteor.publish('notes-newest', function () {
return Notes.find({}, {sort: {createdAt: -1}, limit: 10});
});
if there are 1M notes documents but there is no index on the createdAt key then this will be terribly slow.
Solution: Adding an index on the createdAt key will make such a publication much faster.
You have an invalid publication.
Meteor.publish("user", function(){
return Meteor.user();
})
Is invalid because Meteor.user() is not a cursor and a publication must return either a cursor or an array of cursors. It is also redundant because Meteor.user() is automatically available on the client, albeit it doesn't include all the keys.
Solution: Remove this unnecessary publication altogether. If you want to publish some of the keys that are not available on the client for the current user you can do so as follows:
Meteor.publish("user", function(){
return Meteor.users.find(this.userId,{fields: {services: 1, emails: 1, profile: 1}})
});
After I refresh the page when making a change, it usually takes between 1-2 minutes for a single change. Is there anything wrong with my publications? Although, when the page is loaded, and I reload it loads up really fast.
What does this mean? Do you mean when you change the code of your website, it takes 1-2 minutes for your changes to appear in a newly loaded page? That's to be expected: it takes time for Meteor to rebuild your application. Watch the terminal for the progress.
Using lots of packages and external code will slow down building. The speed of compilation is unrelated to your end user's experience.
A couple things I do with my meteor projects is that I send specific fields "profile.name" etc. I've also used collection.createIndex({}) which creates an index and inside it and you could put your sort in which is created on server startup and works well with your collection.find.sort. I think limiting your notes would be quite essential as you have a limit on one and not the other.
It happened to me too. At that time I checked with kadira (formerly still opensource but now it is not). Source code at kadira on github, but service is gone kadira.io
The problem is in pubication and subscribtion. When client request subscribtions, he opens up some kind of "connection" - i say that, and it applies to others who call the same publication. imagine how many connections are formed.
where the same data accessed by many people must server side must request data to mongodb, that same to all of number of such connections. This is what makes it slow.
Finally I added redis.io.
meteor npm install --save redis
meteor npm install --save hiredis
On server side :
import { Meteor } from "meteor/meteor";
var redis = require("redis");
var clientRedis = redis.createClient({
host: "YOURREDIS_IP",
port: "YOURREDIS_PORT"
});
clientRedis.setSync = Meteor.wrapAsync(clientRedis.set);
clientRedis.getSync = Meteor.wrapAsync(clientRedis.get);
setRedis_Object = function (keyREDIS, timeRefresh, valueREDIS) {
return clientRedis.setSync(keyREDIS + moment(new Date()).format(timeRefresh), JSON.stringify(valueREDIS));
};
getRedis_Object = function (keyREDIS, timeRefresh) {
return JSON.parse(clientRedis.getSync(keyREDIS + moment(new Date()).format(timeRefresh)));
};
Meteor.methods({
getRedis_YOURCOLLECTIONS: function(timeRefresh) {
var data = getRedis_Object(getRedis_YOURCOLLECTIONS, timeRefresh);
if (data != null) {
return data;
} else {
var data = YOURCOLLECTIONS.find().fetch();
setRedis_Object(key, timeRefresh, data);
return data
}
},
});
On Client Side
Meteor.call("getRedis_YOURCOLLECTIONS", "YYYYMMDD", function (error, result) {
if (error) {
console.log(error);
} else {
Session.set("getRedis_YOURCOLLECTIONS", result);
}
});
After i am add this REDIS consumtion Memory and CPU on server low, and app more Fast. I hope its work for you. Thanks
I have angular-meteor app that needs Material md-autocomplete from a collection with 53,296 documents with angularUtils.directives.dirPagination but this amount of data make my browser hang.
I'm publishing the collection with:
Meteor.publish('city', function (options, searchString) {
var where = {
'city_name': {
'$regex': '.*' + (searchString || '') + '.*' ,
'$options': 'i'
}
};
return City.find(where, options);
});
I subscribe with:
subscriptions: function () {
Meteor.subscribe('city');
this.register('city', Meteor.subscribe('city'));
}
and have pagination on controller :
$scope.currentPage = 1;
$scope.pageSize = 100;
$scope.sort = {city_name_sort : 1};
$scope.orderProperty = '1';
$scope.helpers({
city: function(){
return City.find({});
}
});
but it takes a long time to load and its make chrome stop working.
You already have most of the server-side searching done because your search is running inside a subscription. You should make sure that the city_name field is indexed in mongo! You should only return that field to minimize data transfer. You can also simplify your regex.
Meteor.publish('city', function (searchString) {
const re = new RegExp(searchString,'i');
const where = { city_name: { $regex: re }};
return City.find(where, {sort: {city_name: 1}, fields: {city_name: 1}});
});
What I've found helps with server-side auto-complete is:
Don't start searching until the user has typed 3 or 4 characters. This drastically narrows down the search results.
Throttle the search to only run every 500ms so that you're not sending every character to the server because then it has to keep re-executing the search. If the person is typing fast the search might only run every 2 or 3 characters.
Run the same .find() on the client that you're running on the server (instead of just querying for {}). That's just good practice since the client-side collection is the union of all subscriptions on that collection, there might be documents there that you don't want to list.
Lastly I don't know why you're subscribing twice here:
subscriptions: function () {
Meteor.subscribe('city');
this.register('city', Meteor.subscribe('city'));
}
only one of those Meteor.subscribe('city') calls is necessary.
I ma using SignalR in the browser. Some request (calling function on the server) are long and I would like to show spinner/loading-bar.
Can I somehow hook for an event when this function is started and when it returns back.
I'm trying to figure out what you mean - I think basically you want some way to hook into the start of a call and the end of a call (to load and unload a spinner)?
I've done this in two different ways - firstly as a one-off (first example), and then more systematically (the second example). Hopefully one of these will be what you need.
$.connection.myHub.server.hubMethod().done(function () {
//called on success
}).fail(function (e) {
//called on failure - I don't recommend reading e
}).always(function() {
//called regardless
spinner.close();
});
spinner.open(); // must be triggerd AFTER call incase exception thrown (due to connection not being up yet)
If you don't like that - perhaps because you call hub methods in hundreds of different sections of codes, then there are other tricks which are a bit more complicated. Lets see:
function SetupSpinnerOnCallToSignalrMethod(hubServer, method, spinnerStartCallback, spinnerEndCallback) {
var prevFunc = hubServer[method];
hubServer[method] = function () {
var ret = prevFunc.apply(this, arguments);
spinnerStartCallback(); // must be triggerd AFTER call incase exception thrown (due to connection not being up yet)
ret.always(function() {
spinnerEndCallback();
});
return ret;
};
}
//then call this for each method
SetupSpinnerOnCallToSignalrMethod($.connection.myHub.server,
"hubMethod",
function() { spinner.open(); },
function() { spinner.close(); }
);
//the server call should then work exactly as before, but the spinner open and close calls are invoked each time.
There are sites whose DOM and contents are generated dynamically when the page loads. (Angularjs-based sites are notorious for this)
What approach do you use?
I tried both phantomjs and jsdom but it seems I am unable get the page to execute its javascript before I scrape.
Here's a simple jsdom example (not angularjs-based but still dynamically generated)
var env = require('jsdom').env;
exports.scrape = function(link, callback) {
var config = {
url: link,
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36'
},
done: jsdomDone
};
env(config);
}
function jsdomDone(err, window) {
var info = null;
if(err) {
console.error(err);
} else {
var $ = require('jquery')(window);
console.log($('.profilePic').attr('src'));
}
}
exports.scrape('https://www.facebook.com/elcompanies');
I tried phantomjs with moderate success.
var page = new WebPage()
var fs = require('fs');
page.onLoadFinished = function() {
console.log("page load finished");
window.setTimeout(function() {
page.render('export.png');
fs.write('1.html', page.content, 'w');
phantom.exit();
}, 10000);
};
page.open("https://www.facebook.com/elcompanies", function() {
page.evaluate(function() {
});
});
Here I wait for the onLoadFinished event and even put a 10-second timer. The interesting thing is that while my export.png image capture of the page shows a fully rendered page, my 1.html doesn't show the .profilePic class element in its rightful place. It seems to be sitting in some javascript code, surrounded by some kind of "require("TimeSlice").guard(function() {bigPipe.onPageletArrive({..." block
If you can provide me a working example that scrapes the image off this page, that'd be helpful.
I've done some scraping in Facebook by using nightmarejs.
Here is a code that I did to get some content from some posts of a Facebook page.
module.exports = function checkFacebook(callback) {
var nightmare = Nightmare();
Promise.resolve(nightmare
.viewport(1000, 1000)
.goto('https://www.facebook.com/login/')
.wait(2000)
.evaluate(function(){
document.querySelector('input[id="email"]').value = facebookEmail
document.querySelector('input[id="pass"]').value = facebookPwd
return true
})
.click('#loginbutton input')
.wait(1000)
.goto('https://www.facebook.com/groups/bierconomia')
.evaluate(function(){
var posts = document.getElementsByClassName('_1dwg')
var length = posts.length
var postsContent = []
for(var i = 0; i < length; i++){
var pTag = posts[i].getElementsByTagName('p')
postsContent.push({
content: pTag[0] ? pTag[0].innerText : '',
productLink: posts[i].querySelector('a[rel = "nofollow"]') ? posts[i].querySelector('a[rel = "nofollow"]').href : '',
photo: posts[i].getElementsByClassName('_46-i img')[0] ? posts[i].getElementsByClassName('_46-i img')[0].src : ''
})
}
return postsContent
}))
.then(function(results){
log(results)
return new Promise(function(resolve, reject) {
var leanLinks = results.map(function(result){
return {
post: {
content: result.content,
productLink: extractLinkFromFb(result.productLink),
photo: result.photo
}
}
})
resolve(leanLinks)
})
})
The thing that I find useful with nightmare is that you can use the wait function to either wait for X ms or for a specific class to render.
This is because generated web pages based on AJAX calls have asynchronous AJAX calls and you can't rely on onLoad events (because data still not available).
In my personal opinion, the most reliable way would be tracing which REST services are being called from this HTML and make direct calls to them. Sometimes you will need using values found in HTML or values taken from another calls.
I know this may sound complicated, and in fact it is. You kinda need to debug page and learn what is being called. But this will work for sure.
By the way, using chrome developer tools will help this task. Just observe which call are made in network tab. You can even observe what has been sent and received in each AJAX call.
If it is a one time thing, that is, if I just want to scrape a single page once, I just use the browser and artoo-js.
I never tried to write a page on disk using phantom, but I have two observations:
1) you are using fs.write to write things to disk, but writeFile is an async call. This means that you either need to change it to fs.writeFileSync or use a callback before closing phantom.
2) I hope you aren't expecting to write a HTML to a file and open it in a browser and get it rendered like when you saved a png, because it doesnt work this way. Some objects can be stored directly in DOM properties and certainly there are values stored in javascript variables, those things will never be persisted.
I've been scratching my head as to why this code will work some of the time, but not all (or at least most of the time). I've found that it actually does run displaying the correct content in the browser some of the time, but strangely there will be days when I'll come back to the same code, run the server (as per normal) and upon loading the page will receive an error in the console: TypeError: 'undefined' is not an object (evaluating 'Session.get('x').html')
(When I receive that error there will be times where the next line in the console will read Error - referring to the err object, and other times when it will read Object - referring the data object!?).
I'm obviously missing something about Session variables in Meteor and must be misusing them? I'm hoping someone with experience can point me in the right direction.
Thanks, in advance for any help!
Here's my dummy code:
/client/del.html
<head>
<title>del</title>
</head>
<body>
{{> hello}}
</body>
<template name="hello">
Hello World!
<div class="helloButton">{{{greeting}}}</div>
</template>
My client-side javascript file is:
/client/del.js
Meteor.call('foo', 300, function(err, data) {
err ? console.log(err) : console.log(data);
Session.set('x', data);
});
Template.hello.events = {
'click div.helloButton' : function(evt) {
if ( Session.get('x').answer.toString() === evt.target.innerHTML ) {
console.log('yay!');
}
}
};
Template.hello.greeting = function() {
return Session.get('x').html;
};
And my server-side javascript is:
/server/svr.js
Meteor.methods({
doubled: function(num) {
return num * 2;
},
foo: function(lmt) {
var count = lmt,
result = {};
for ( var i = 0; i < lmt; i++ ) {
count++;
}
count = Meteor.call('doubled', count);
result.html = "<em>" + count + "</em>";
result.answer = count;
return result;
}
});
I think it's just that the session variable won't be set yet when the client first starts up. So Session.get('x') will return undefined until your method call (foo) returns, which almost certainly won't happen before the template first draws.
However after that it will be in the session, so things will probably behave right once you refresh.
The answer is to just check if it's undefined before trying to access the variable. For example:
Template.hello.greeting = function() {
if (Session.get('x')) return Session.get('x').html;
};
One of the seven principles of Meteor is:
Latency Compensation. On the client, use prefetching and model simulation to make it look like you have a zero-latency connection to the database.
Because there is latency, your client will first attempt to draw the lay-out according to the data it has at the moment your client connects. Then it will do the call and then it will update according to the call. Sometimes the call might be able to respond fast enough to be drawn at the same time.
As now there is a chance for the variable to not be set, it would throw an exception in that occasion and thus break down execution (as the functions in the call stack will not continue to run).
There are two possible solutions to this:
Check that the variable is set when using it.
return Session.get('x') ? Session.get('x').html : '';
Make sure the variable has an initial value by setting it at the top of the script.
Session.set('x', { html = '', answer = ''});
Another approach would be to add the templates once the call responds.
Meteor.call('foo', 300, function(err, data) {
Session.set('x', data);
$('#page').html(Meteor.ui.render(function() {
return Template.someName();
}));
});