How to Fetch a JSON file in Webassembly - fetch

I'm currently experimenting with Webassembly, and one thing I'm trying to do here is with a Webassembly to Fetch data from a JSON file, compile that into a .wasm module, and use that module in Javascript to read the result of the fetch.
I've tried following the code on https://kripken.github.io/emscripten-site/docs/api_reference/fetch.html but the resulting .wasm code is confusing to me because I can't find how to properly load that .wasm module in Javascript.
In case I'm going about this the wrong way, I really need some help with this.
started with this fetch.c file that is supposed to fetch JSON data from a file.
#include <stdio.h>
#include <string.h>
#include <emscripten/fetch.h>
/*////////////////////////
// This file contains the code for fetching
// -> Compiled to .wasm file with emscripten <-
*////////////////////////
void downloadSucceeded(emscripten_fetch_t *fetch) {
printf("Finished downloading %llu bytes from URL %s.\n", fetch->numBytes, fetch->url);
// The data is now available at fetch->data[0] through fetch->data[fetch->numBytes-1];
emscripten_fetch_close(fetch); // Free data associated with the fetch.
}
void downloadFailed(emscripten_fetch_t *fetch) {
printf("Downloading %s failed, HTTP failure status code: %d.\n", fetch->url, fetch->status);
emscripten_fetch_close(fetch); // Also free data on failure.
}
int main() {
emscripten_fetch_attr_t attr;
emscripten_fetch_attr_init(&attr);
strcpy(attr.requestMethod, "GET");
attr.attributes = EMSCRIPTEN_FETCH_LOAD_TO_MEMORY | EMSCRIPTEN_FETCH_PERSIST_FILE;
attr.onsuccess = downloadSucceeded;
attr.onerror = downloadFailed;
emscripten_fetch(&attr, "./json/bol_list1.json");
}
I compiled this with : emcc wasm/fetch.c -Os -s WASM=1 -s FETCH=1 -s SIDE_MODULE=1 -s BINARYEN_ASYNC_COMPILATION=0 -o wasm/fetch.wasm
fetch.wasm: https://pastebin.com/cHYpgazy
So, now with that module I'm supposed to read it in Javascript and get the result, but this is where I'm stuck, because as opposed to other examples this .wasm module doesn't have an obvious export/import thing and my previous methods of trying to load it failed.
wasmbyfile.js:
Method 1:
let obj;
loadWebAssembly('./wasm/fetch.wasm') //Testing function
.then(instance => {
obj = instance.exports._main;
console.log(obj);
});
function loadWebAssembly(fileName) {
return fetch(fileName)
.then(response => response.arrayBuffer())
.then(bits => WebAssembly.compile(bits))
.then(module => { return new WebAssembly.Instance(module) });
};
error result: wasmbyfile.js:64 Uncaught (in promise) TypeError: WebAssembly Instantiation: Imports argument must be present and must be an object
at fetch.then.then.then.module (wasmbyfile.js:64)
Method 2:
(async () => {
const fetchPromise = fetch('./wasm/fetch.wasm');
const { instance } = await WebAssembly.instantiateStreaming(fetchPromise);
const result = instance.exports._main;
console.log(result);
})();
error result: Uncaught (in promise) TypeError: WebAssembly Instantiation: Imports argument must be present and must be an object
So I'm stuck at this point, and not really sure how to load the module correctly in JS. I need some help for this, or am I doing this the wrong way from the beginning and is there a better way for me to do this?

You are getting an error because your WASM has import statements, while your call to instantiateStreaming does not send an importObject.
But the basic way to use WASM from Javascript is much simpler than: Just define a function in WASM that you can call from JS, and then you do the "fetch" from JS, for instance ("add.wasm"):
(module
(type $t0 (func (param i32 i32) (result i32)))
(func $add (type $t0) (param $p0 i32) (param $p1 i32) (result i32)
get_local $p0
get_local $p1
i32.add)
(export "add" (func $add)))
And then call it from Javascript:
const wasmInstanceFromFile = await WebAssembly.instantiateStreaming(await fetch('add.wasm'));
let sum = wasmInstanceFromFile.instance.exports.add(1,2);

Related

Hyper cannot find Server module

I'm writing a "hello world" HTTP server with Hyper, however I am not able to find the Server and rt modules when trying to import them.
When invoking cargo run, then I see this error message:
26 | let server = hyper::Server::bind(&addr).serve(router);
| ^^^^^^ could not find `Server` in `hyper`
I must be missing something obvious about Rust and Hyper. What I am trying to do is writing something as dry/simple as possible with just the HTTP layer and some basic routes. I would like to include as little as possible 3rd party dependencies e.g avoiding Tokio which I think involves async behavior, but I am not sure about the context as I am new to Rust.
Looks like I must use futures, so I included this dependency and perhaps futures only work with the async reserved word (which I am not sure if it comes from Tokio or Rust itself).
What confuses me is that in the Hyper examples I do see imports like use hyper::{Body, Request, Response, Server};, so that Server thing must be there, somewhere.
These are the dependencies in Cargo.toml:
hyper = "0.14.12"
serde_json = "1.0.67"
futures = "0.3.17"
This is the code in main.rs:
use futures::future;
use hyper::service::service_fn;
use hyper::{Body, Method, Response, StatusCode};
use serde_json::json;
fn main() {
let router = || {
service_fn(|req| match (req.method(), req.uri().path()) {
(&Method::GET, "/foo") => {
let mut res = Response::new(
Body::from(json!({"message": "bar"}).to_string())
);
future::ok(res)
},
(_, _) => {
let mut res = Response::new(
Body::from(json!({"content": "route not found"}).to_string())
);
*res.status_mut() = StatusCode::NOT_FOUND;
future::ok(res)
}
})
};
let addr = "127.0.0.1:8080".parse::<std::net::SocketAddr>().unwrap();
let server = hyper::Server::bind(&addr).serve(router); // <== this line fails to compile
// hyper::rt::run(server.map_err(|e| {
// eprintln!("server error: {}", e);
// }));
}
How do I make the code above compile and run?
According to documentation, you are missing one module namespace in your call hyper::server::Server:
let server = hyper::server::Server::bind(&addr).serve(router)
In order to use server you need to activate the feature flag in cargo:
hyper = { version = "0.14.12", features = ["server"] }

"strict" mode for QML?

Does Qt's QML language provide any kind of "strict" mode? In particular, there are two features I'd like:
Application crash on reference to undefined or null (e.g. foo = bar when foo is an existing property but bar is not defined, and foo.bar when foo is null)
"hard" asserts (the console.assert feature does not crash the application).
1. Use qml lint
Run qmllint on all .qml and .js files in your build setup
find ./myproject -type f -regex ".*\.\(qml\|js\)" -exec "$QT_DIR/bin/qmllint" \{\} +
2. Crash app on QML error/warning
Write a custom QDebug message handler function static void handler(QtMsgType type, const QMessageLogContext& context, const QString &message); you register via qInstallMessageHandler(&MyQDebugMessageHandler::handler); that turns QML warnings into fatal logs:
if (type == QtWarningMsg)
{
auto fatalWarnings = std::vector<QString>{
QStringLiteral("ReferenceError:"),
QStringLiteral("is not a type"),
QStringLiteral("File not found"),
};
for (const auto &s : fatalWarnings)
{
if (message.contains(s))
{
type = QtFatalMsg;
break;
}
}
}
Then make sure that QDebug messages of type QtFatalMsg crash the app.
3. Crash on console.assert()
console.assert() creates errors but nothing specific to detect them. So adapt point 2. to crash the app on errors as well.

writeCopyToPath Realm Doesn't work

I try to follow "Bundling a Realm with an App" Documentation.I've try to use method Realm().writeCopyToPath(_:encryptionKey:)) but the problem is i can't make a file on a specific location on my OSX. It's show error that file already exits. Please give me a correct way to use writeCopyToPath
#UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate {
var window: UIWindow?
func application(application: UIApplication, didFinishLaunchingWithOptions launchOptions: [NSObject : AnyObject]?) -> Bool {
window = UIWindow(frame: UIScreen.mainScreen().bounds)
window?.rootViewController = UIViewController()
window?.makeKeyAndVisible()
NSFileManager.defaultManager().removeItemAtPath(Realm.Configuration.defaultConfiguration.path!, error: nil)
// Create a standalone object
var mydog = Dog()
// Set & read properties
mydog.name = "Rex"
mydog.age = 9
println("Name of dog: \(mydog.name)")
// Realms are used to group data together
let realm = Realm() // Create realm pointing to default file
println(realm.writeCopyToPath("/Users/taforyou/CU-TEP",encryptionKey: nil))
// Save your object
realm.beginWrite()
realm.add(mydog)
realm.commitWrite()
return true
}
}
Output Log When running
Name of dog: Rex
Optional(Error Domain=io.realm Code=4 "open() failed: File exists" UserInfo=0x7fbc704a8cd0 {NSLocalizedDescription=open() failed: File exists, Error Code=4})
Thank you
realm 0.95
The error message getting printed says it all: the file you're trying to create (/Users/taforyou/CU-TEP) already exists. Note that the path that writeCopyToPath is expecting is a full path to the resulting realm file (/path/to/file.realm, not /path/to) just in case CU-TEP is actually a directory.

Running Meteor methods that insert mongo documents in their own Fiber

I'm attempting to run ntwitter streaming API to track tweets about a certain hashtag, populating the Mongo collection Tweets with each tweet.
I've hooked it up server side like so:
t = new nTwitter({
consumer_key: credentials.consumer_key,
consumer_secret: credentials.consumer_secret,
access_token_key: credentials.access_token_key,
access_token_secret: credentials.access_token_secret
});
Meteor.methods({
trackTweets: function () {
this.unblock; // this doesn't seem to work
console.log('... ... trackTweets');
var _this = this;
t.stream(
'statuses/filter',
{ track: ['#love'] },
function(stream) {
stream.on('data', function(tweet) {
// app/packages/mongo-livedata/collection.js:247
// throw e;
// ^
// O yes I love her like money
// Error: Meteor code must always run within a Fiber
console.log(tweet.text);
Tweets.insert(tweet.text); // this call blocks
});
stream.on('error', function(error, code) {
console.log("My error: " + error + ": " + code);
});
}
);
}
});
The line: Tweets.insert(tweet.text) throws the must run inside its own Fiber error – and I've tried putting the this.unblock statement in several different places.
What should I do here?
you dont call the function unblock, you need to replace your
this.unblock;
with this:
this.unblock();
if that doesn't work i would think it has something to do with the way ntwitter is getting the data, you could try to add this
if (Meteor.isClient) return false;
so that the method doesn't run on the client, but only on the server
I believe the code you are running server-side needs to be contained within a Fiber.
Some similar examples can be found in these answers:
Meteor code must always run within a Fiber” when calling Collection.insert on server
Stream stdout to Meteor website

In Node.JS, when I do a POST request, what is the maximum size? [duplicate]

I created an upload script in node.js using express/formidable. It basically works, but I am wondering where and when to check the uploaded file e. g. for the maximum file size or if the file´s mimetype is actually allowed.
My program looks like this:
app.post('/', function(req, res, next) {
req.form.on('progress', function(bytesReceived, bytesExpected) {
// ... do stuff
});
req.form.complete(function(err, fields, files) {
console.log('\nuploaded %s to %s', files.image.filename, files.image.path);
// ... do stuff
});
});
It seems to me that the only viable place for checking the mimetype/file size is the complete event where I can reliably use the filesystem functions to get the size of the uploaded file in /tmp/ – but that seems like a not so good idea because:
the possibly malicious/too large file is already uploaded on my server
the user experience is poor – you watch the upload progress just to be told that it didnt work afterwards
Whats the best practice for implementing this? I found quite a few examples for file uploads in node.js but none seemed to do the security checks I would need.
With help from some guys at the node IRC and the node mailing list, here is what I do:
I am using formidable to handle the file upload. Using the progress event I can check the maximum filesize like this:
form.on('progress', function(bytesReceived, bytesExpected) {
if (bytesReceived > MAX_UPLOAD_SIZE) {
console.log('### ERROR: FILE TOO LARGE');
}
});
Reliably checking the mimetype is much more difficult. The basic Idea is to use the progress event, then if enough of the file is uploaded use a file --mime-type call and check the output of that external command. Simplified it looks like this:
// contains the path of the uploaded file,
// is grabbed in the fileBegin event below
var tmpPath;
form.on('progress', function validateMimetype(bytesReceived, bytesExpected) {
var percent = (bytesReceived / bytesExpected * 100) | 0;
// pretty basic check if enough bytes of the file are written to disk,
// might be too naive if the file is small!
if (tmpPath && percent > 25) {
var child = exec('file --mime-type ' + tmpPath, function (err, stdout, stderr) {
var mimetype = stdout.substring(stdout.lastIndexOf(':') + 2, stdout.lastIndexOf('\n'));
console.log('### file CALL OUTPUT', err, stdout, stderr);
if (err || stderr) {
console.log('### ERROR: MIMETYPE COULD NOT BE DETECTED');
} else if (!ALLOWED_MIME_TYPES[mimetype]) {
console.log('### ERROR: INVALID MIMETYPE', mimetype);
} else {
console.log('### MIMETYPE VALIDATION COMPLETE');
}
});
form.removeListener('progress', validateMimetype);
}
});
form.on('fileBegin', function grabTmpPath(_, fileInfo) {
if (fileInfo.path) {
tmpPath = fileInfo.path;
form.removeListener('fileBegin', grabTmpPath);
}
});
The new version of Connect (2.x.) has this already baked into the bodyParser using the limit middleware: https://github.com/senchalabs/connect/blob/master/lib/middleware/multipart.js#L44-61
I think it's much better this way as you just kill the request when it exceeds the maximum limit instead of just stopping the formidable parser (and letting the request "go on").
More about the limit middleware: http://www.senchalabs.org/connect/limit.html

Resources