Fetch API and CasperJS - web-scraping

I am trying to do something very similar to this thread, except that instead of XHR, the web app (written in React) uses the Fetch API to do its asynchronous calls.
Is there an elegant way to intercept/overwrite fetch? Basically I want to see each AJAX call the site is doing and grab the data from there. I could just grab it as shown on the site after the fact, but we all know that JSON is easier than HTML to deal with. The site is STATEFUL, so I cannot just resend the URL.
I tried just writing a wrapper like:
function replaceFetch(){
(function(window, debug){
var _fetch = window.fetch;
window.fetch = function(u, o = {}) {
console.log("This is a test");
return _fetch(u, o);
}
})(window, false);
};
casper.on("page.initialized", function(resource){
this.evaluate(replaceFetch);
});
But I get "Untitled suite" error, and can't find anything I can require() to implement Fetch.
Any thoughts? Thanks!

Related

How do I make a database call from an Electron front end?

(Brand new learning Electron here, so I'm sure this is a basic question and I am missing something fundamental...)
How does one interact with a local database (I am using Sqlite) from an Electron application front end? I have a very basic database manager class and have no problem using it from the index.js file in my Electron application. But from the front-end (I am using Svelte, but I can probably translate solutions from other front-end frameworks), how does one interact with the database? This seems fundamental but I'm striking out trying to find a basic example.
Since everything is local it would seem it shouldn't be necessary to set up a whole API just to marshal data back and forth, but maybe it is? But if so, how does one go about telling the Electron "backend" (if that's the right term) to do something and return the results to the front end? I'm seeing something about IPC but that's not making a lot of sense right now and seems like overkill.
Here's my simple database manager class:
const sqlite3 = require("sqlite3").verbose();
class DbManager {
#db;
open() {
this.#db = new sqlite3.Database("testing.db", sqlite3.OPEN_READWRITE);
}
close() {
this.#db.close();
}
run(sql, param) {
this.#db.run(sql, param);
return this;
}
}
const manager = new DbManager();
module.exports = manager;
And I can call this and do whatever no problem from the Electron entry point index.js:
const { app, BrowserWindow, screen } = require("electron");
require("electron-reload")(__dirname);
const db = require("./src/repository/db");
const createWindow = () => {
...
};
let window = null;
app.whenReady().then(() => {
db.open();
createWindow();
});
app.on("window-all-closed", () => {
db.close();
app.quit();
});
But what to do from my component?
<script>
// this won't work, and I wouldn't expect it to, but not sure what the alternative is
const db = require("./repository/db");
let accountName;
function addAccount() {
db.run("INSERT INTO accounts (name) VALUES ($name);", { $name: accountName });
}
</script>
<main>
<form>
<label for="account_name">Account name</label>
<input id="account_name" bind:value={accountName} />
<button on:click={addAccount}>Add account</button>
</form>
</main>
If anyone is aware of a boilerplate implementation that does something similar, that would be super helpful. Obviously this is like application 101 here; I'm just not sure how to go about this yet in Electron and would appreciate someone pointing me in the right direction.
If you're absolutely 100% sure that your app won't be accessing any remote resources then you could just expose require and whatever else you may need through the preload script, just write const nodeRequire = require; window.require = nodeRequire;.
This is a fairly broad topic and requires some reading. I'll try to give you a primer and link some resources.
Electron runs on two (or more if you open multiple windows) processes - the main process and the renderer process. The main process handles things like opening new windows, starting and closing the entire app, tray icons, window visibility etc., while the renderer process is basically like your JS code in a browser. More on Electron processes.
By default, the renderer process does not have access to a Node runtime, but it is possible to let it. You can do that in two ways, with many caveats.
One way is by setting webPreferences.nodeIntegration = true when creating the BrowserWindow (note: nodeIntegration is deprecated and weird. This allows you to use all Node APIs from your frontend code, and your snippet would work. But you probably shouldn't do that because a BrowserWindow is capable of loading external URLs, and any code included on those pages would be able to execute arbitrary code on your or your users' machines.
Another way is using the preload script. The preload script runs in the renderer process but has access to a Node runtime as well as the browser's window object (the Node globals get removed from the scope before the actual front end code runs unless nodeIntegration is true). You can simply set window.require = require and essentially work with Node code in your frontend files. But you probably shouldn't do that either, even if you're careful about what you're exposing, because it's very easy to still leave a hole and allow a potential attacker to leverage some exposed API into full access, as demonstrated here. More on Electron security.
So how to do this securely? Set webPreferences.contextIsolation to true. This definitively separates the preload script context from the renderer context, as opposed to the unreliable stripping of Node APIs that nodeIntegration: false causes, so you can be almost sure that no malicious code has full access to Node.
You can then expose specific function to the frontend from the preload, through contextBridge.exposeInMainWorld. For example:
contextBridge.exposeInMainWorld('Accounts', {
addAccount: async (accountData) => {
// validate & sanitize...
const success = await db.run('...');
return success;
}
}
This securely exposes an Accounts object with the specified methods in window on the frontend. So in your component you can write:
const accountAdded = await Accounts.addAccount({id: 123, username: 'foo'});
Note that you could still expose a method like runDbCommand(command) { db.run(command) } or even evalInNode(code) { eval(code) }, which is why I said almost sure.
You don't really need to use IPC for things like working with files or databases because those APIs are available in the preload. IPC is only required if you want to manipulate windows or trigger anything else on the main process from the renderer process.

How to call method in route?

I have two forms of search
POST /search
GET /search?q=q
For POST, I can just use Meteor.call in client.
'submit form': function() {
Meteor.call('search', data......
}
For GET, How can I call Meteor.call in router?
Router.route('/search', {
template: 'ItemList',
data: function() {
return Meteor.call('search', this.params.query); // ???
}
and about the search method, it will not only search database but also search file system.
I don't know how to do the second GET search.
I would suggest not doing the calls in your router. The router is responsible for directing the flow of actions and it would make it more complex if it was also managing all of the data and external resource tracking.
In the template level, you could easily detect the routers params (Router.current()) or have them passed directly from the router in the data section.
Then on rendered you could have the template hit your call function. And then make it reactive with some dependency tracking
You can do Router.current().params.q to get the q variable from url.
You won't need data: function() in the said approach.

SignalR generate multiple executions

I'm building a search engine using SignalR to deliver partial responses in real time to the client.
The problem occurs when the first search is not over, and the client modifies the value of the textbox (txtTab) and click the button (btnSearch) again. The results of the two queries are mixed because the server keeps the two concurrent executions.
I need that when the client clicks the search button the previous execution is canceled.
Tried using hub.stop () and hub.disconnect (), but I can not.
Sorry my bad english :)
var busca = $.connection.hubBusca;
busca.client.atualizaResultados = function (arr) {
oTable.fnAddData(arr);
};
$.connection.hub.start().done(function () {
$('#btnSearch').click(function () {
if ($('#txtTab').val() != '') {
busca.server.iniciar($('#txtTab').val());
}
});
});
Thanks!
Luiz Fernando
I would suggest creating an additional server and client method to handle the final search (which happens when #btnSearch is clicked).
Instead of invoking busca.server.iniciar invoke something like busca.server.terminar.
I would also suggest setting some sort of boolean flag before calling busca.server.terminar to disable busca.client.atualizaResultados so no preliminary search results interfere with the final search results.
Then in HubBusca.Terminar you could call Clients.Caller.finalesResultados which would not be disabled unlike Clients.Caller.atualizaResultados.
In this way you can ensure that your final search results which are sent to the finalesResultados method will not be replaced by results sent to the atualizaResultados client method.
Sorry for my bad Spanish method names. I used Google Translate.

Get data from the ajax response

For school, i have to develop a Twitter client with ASP.NET.
In the app, i have a list of tweets with a delete link. This link is created with the helper Ajax.ActionLink() and i specified a callback function for OnSuccess event.
This link is okay : the action is performed, the callback is triggered BUT i can't access data sent in the Ajax response.
The callback receive only one argument. Here is the dump of this object :
>> Sys.Mvc.AjaxContext
$0: 0
$1: null
$2: Sys.Net.XMLHttpExecutor
$3: Sys.Net.WebRequest
$4: null
Where is my responseText ? I know that the response has a content (according to Chrome developer tools) and i really want to access it.
Bonus : can Ajax client automatically parse the response as JSON (the action returns JSON properly with the JSON method) ?
Thanks ! ;)
The deadline of this school project is over.
I used get_data on the response.
I'm quite disappointed by the lack of documentation for this trivial need. Even now i know the way, i can't find that on MSDN… Such a pity. :(
Back to my precious Ruby on Rails, i feel better.
Good day and thanks for your help anyway ! :)
Try calling get_object() on your AjaxContext to get a javascript object or get_data() to get the text. An easier method though is to have your OnSuccess function take an argument which will be the object returned.
public ActionResult ReturnJson()
{
return Json(new { TestMessage = "Hello, world!" }, JsonRequestBehavior.AllowGet);
}
And the view...
<script type="text/javascript">
function AjaxSuccess(obj) {
alert(obj.TestMessage);
}
</script>
#Ajax.ActionLink("Say Hi", "ReturnJson", new AjaxOptions() { OnSuccess = "AjaxSuccess" })
If you want to access the responseText within your code you do not want to use AJAX(Asynchronous JavaScript And XML), you want to use SJAX (Synchronous JavaScript And XML).
When using AJAX your code will not wait for the responseText it will just continue to execute so if you try referencing it later in your code it may not work as it might not have processed yet.
When using SJAX your code will wait for the responseText before continuing to execute.
I don't use ASP.NET so I can't help with the code.
Here is how it is done in JavaScript: JavaScript - AJAX / SJAX - Submit Form Data to a Script

Track any asp.net web services calls from javascript

I am writing a control that will warn a user that their session is about to timeout. I can easily track standard postbacks and post packs in update panels, but I'm struggling to see how I can track calls made to any / all web services.
Is there a mechanism for this, or will I have to try and override the Sys.Net.WebRequest invoke code?
There's nothing wrong in replacing the Sys.Net.WebRequest code - just keep a reference to the previous implementation and call it, after tracking the call. Something like this should do the trick:
(function() {
var originalWebRequest = Sys.Net.WebRequest;
Sys.Net.WebRequest = function() {
// track call
//...
// call original WebRequest
return originalWebRequest.apply(this, arguments);
}
})();
In a similar requirement, we wrote a HttpModule. At that point you could have a good control over your request and track your postbacks of webservices. See here for more information about how to write your own httpmodule http://msdn.microsoft.com/en-us/library/aa719858(VS.71).aspx

Resources