As you may know, ZeroRPC documentation is sparse. I can't get Streaming between a Python server and a Node client to work.
Here is the Python method:
#zerorpc.stream
def PublishWhaterver(self, some_args):
yield "love"
yield "stack"
yield "overflow"
Here is the Node call:
export const tryStream = () => {
connectedZerorpcClient.invoke('PublishWhatever', (error, res, more) => {
console.log('STREAM', res, more);
});
};
This code will log "STREAM love", and then do nothing.
So here are my questions:
In the Python server code, am I supposed to call PublishWhatever with relevant args so that it yield additionnal values ?
In the Node client, should I call some recursive function when there is more data ?
What I am trying to implement is a Pub/Sub system but right now implementation seems to only exists for a Python server and a Python client, there are no Node example.
The example on the main page and tests are not relevant either, it shows how to stream an array that already exists when the invoke method is called.Here the messages are generated during some heavy computations, I want the server to be able to tell the client "here, some data are ready" and never disconnect.
Well, ZeroRPC actively promotes, that it is using its own python implementation code as a self-documentation how things work. In other words, no one has spent such additional efforts needed so as to publish a user-focused, the less a learning-process focused documentation.
Anyway, try to obey the few "visible" statements from the ZeroRPC description.
#zerorpc.stream
def PublishWhaterver(self, some_args):
yield ( "love", "stack", "overflow", ) # one, tuple-wrapped result-container
Related
Working with Evernote IOS SDK 3.0
I would like to retrieve a specific resource from note using
fetchResourceByHashWith
This is how I am using it. Just for this example, to be 100% sure about the hash being correct I first download the note with a single resource using fetchNote and then request this resource using its unique hash using fetchResourceByHashWith (hash looks correct when I print it)
ENSession.shared.primaryNoteStore()?.fetchNote(withGuid: guid, includingContent: true, resourceOptions: ENResourceFetchOption.includeData, completion: { note, error in
if error != nil {
print(error)
seal.reject(error!)
} else {
let hash = note?.resources[0].data.bodyHash
ENSession.shared.primaryNoteStore()?.fetchResourceByHashWith(guid: guid, contentHash: hash, options: ENResourceFetchOption.includeData, completion: { res, error in
if error != nil {
print(error)
seal.reject(error!)
} else {
print("works")
seal.fulfill(res!)
}})
}
})
Call to fetchResourceByHashWith fails with
Optional(Error Domain=ENErrorDomain Code=0 "Unknown error" UserInfo={EDAMErrorCode=0, NSLocalizedDescription=Unknown error})
The equivalent setup works on Android SDK.
Everything else works so far in IOS SDK (chunkSync, auth, getting notebooks etc.. so this is not an issue with auth tokens)
would be great to know if this is an sdk bug or I am still doing something wrong.
Thanks
This is a bug in the SDK's "EDAM" Thrift client stub code. First the analysis and then your workarounds.
Evernote's underlying API transport uses a Thrift protocol with a documented schema. The SDK framework includes a layer of autogenerated stub code that is supposed to marshal input and output params correctly for each request and response. You are invoking the underlying getResourceByHash API method on the note store, which is defined per the docs to accept a string type for the contentHash argument. But it turns out the client is sending the hash value as a purely binary field. The service is failing to parse the request, so you're seeing a generic error on the client. This could reflect evolution in the API definition, but more likely this has always been broken in the iOS SDK (getResourceByHash probably doesn't see a lot of usage). If you dig into the more recent Python version of the SDK, or indeed also the Java/Android version, you can see a different pattern for this method: it says it's going to write a string-type field, and then actually emits a binary one. Weirdly, this works. And if you hack up the iOS SDK to do the same thing, it will work, too.
Workarounds:
Best advice is to report the bug and just avoid this method on the note store. You can get resource data in different ways: First of all, you actually got all the data you needed in the response to your fetchNote call, i.e. let resourceData = note?.resources[0].data.body and you're good! You can also pull individual resources by their own guid (not their hash), using fetchResource (use note?.resources[0].guid as the param). Of course, you may really want to use the access-by-hash pattern. In that case...
You can hack in the correct protocol behavior. In the SDK files, which you'll need to build as part of your project, find the ObjC file called ENTProtocol.m. Find the method +sendMessage:toProtocol:withArguments.
It has one line like this:
[outProtocol writeFieldBeginWithName:field.name type:field.type fieldID:field.index];
Replace that line with:
[outProtocol writeFieldBeginWithName:field.name type:(field.type == TType_BINARY ? TType_STRING : field.type) fieldID:field.index];
Rebuild the project and you should find that your code snippet works as expected. This is a massive hack however and although I don't think any other note store methods will be impacted adversely by it, it's possible that other internal user store or other calls will suddenly start acting funny. Also you'd have to maintain the hack through updates. Probably better to report the bug and don't use the method until Evernote publishes a proper fix.
While trying to set up some simple end-to-end tests with Jest and Puppeteer, I've found that any test I write will inexplicably fail with a timeout.
Here's a simple example test file, which deviates only slightly from Puppeteer's own example:
import puppeteer from 'puppeteer';
describe('Load Google Puppeteer Test', () => {
test('Load Google', async () => {
const browser = await puppeteer.launch({
headless: false
});
const page = await browser.newPage();
await page.goto('https://google.co.uk');
await expect(page).toMatch("I'm Feeling Lucky");
await browser.close();
});
});
And the response it produces:
TimeoutError: Text not found "I'm Feeling Lucky"
waiting for function failed: timeout 500ms exceeded
I have tried adding in custom timeouts to the goto line, the test clause, amongst other things, all with no effect. Any ideas on what might be causing this? Thanks.
What I would say is happening here is that using toMatch expects text to be displayed. However, in your case, the text you want to verify is text associated with a button.
You should try something like this:
await expect(page).toMatchElement('input[value="I\'m Feeling Lucky"]');
Update 1:
Another possibility (and it's one you've raised yourself) is that the verification is timing out before the page has a chance to load. This is a common issue, from my experience, with executing code in headless mode. It's very fast. Sometimes too fast. Statements can be executed before everything in the UI is ready.
In this case you're better off adding some waitForSelector statements throughout your code as follows:
await page.waitForSelector('input[value="I\'m Feeling Lucky"]');
This will ensure that the selector you want is displayed before carrying on with the next step in your code. By doing this you will make your scripts much more robust while maintaining efficiency - these waits won't slow down your code. They'll simply pause until puppeteer registers the selector you want to interact with / verify as being displayed. Most of the time you won't even notice the pause as it will be so short (I'm talking milliseconds).
But this will make your scripts rock solid while also ensuring that things won't break if the web page is slower to respond for any reason during test execution.
You're probably using 'expect-puppeteer' package which does the toMatch expect. This is not a small deviation. The weird thing is that your default timeout isn't 30 seconds as the package's default, check that.
However, to fix your issue:
await expect(page).toMatch("I'm Feeling Lucky", { timeout: 6000 });
Or set the default timeout explicitly using:
page.setDefaultTimeout(timeout)
See here.
I am attempting to return an object from a AWS Lambda function instead of a simple string.
// ...
context.fail({
"email": "Email address is too short",
"firstname": "First name is too short"
});
// ...
I have already used the errorMessage for mapping error responses to status codes and that has been great:
// ...
context.fail('That "username" has already been taken.');
// ...
Am I simply trying to do something that the AWS API Gateway does not afford?
I have also already found this article which helped: Is there a way to change the http status codes returned by Amazon API Gateway?.
Update
Since time of writing, lambda has updated the invocation signature and now passes event, context, callback.
Instead of calling context.done(err, res) you should use callback(err, res). Note that what was true for context.done still applies to the callback pattern.
Should also add that with API Gateways proxy and integration implementation this entire thread is pretty much obsolete.
I recommend reading this article if you are integrating API Gateway with Lambda: http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-api-as-simple-proxy-for-lambda.html
Original response below
First things first, let's clear a few things up.
context.done() vs. context.fail()/context.success
context.done(error, result); is nothing but a wrapper around context.fail(error); and context.success(response);
The Lambda documentation clearly states that result is ignored if error is non null:
If the Lambda function was invoked using the RequestResponse (synchronous) invocation type, the method returns response body as follows:
If the error is null, set the response body to the string representation of result. This is similar to the context.succeed().
If the error is not null, set the response body to error.
If the function is called with a single argument of type error, the error value will be populated in the response body.
http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-context.html
What this means is that it won't matter whether you use a combination of fail/success or done, the behaviour is exactly the same.
API Gateway and Response Code Mapping
I have tested every thinkable combination of response handling from Lambda in combination with Response code mapping in API Gateway.
The conclusion of these tests are that the "Lambda Error RegExp" is only executed against a Lambda error, i.e: you have to call context.done(error);or context.fail(error); for the RegExp to actually trigger.
Now, this presents a problem as, has already been noted, Lambda takes your error and sticks it in an object and calls toString() on whatever you supplied:
{ errorMessage: yourError.toString() }
If you supplied an error object you'll get this:
{ errorMessage: "[object Object]" }
Not very helpful at all.
The only workaround I have found thus far is to call
context.fail(JSON.stringify(error));
and then in my client do:
var errorObject = JSON.parse(error.errorMessage);
It's not very elegant but it works.
As part of my error I have a property called "code". It could look something like this:
{
code: "BadRequest",
message: "Invalid argument: parameter name"
}
When I stringify this object I get:
"{\"code\":\"BadRequest\",\"message\":\"Invalid argument: parameter name\"}"
Lambda will stick this string in the errorMessage property of the response and I can now safely grep for .*"BadRequest".* in the API Gateway response mapping.
It's very much a hack that works around two somewhat strange quirks of Lambda and API Gateway:
Why does Lambda insist on wrapping the error instead of just giving
it back as is?
Why doesn't API Gateway allow us to grep in the
Lambda result, only the error?
I am on my way to open a support case with Amazon regarding these two rather odd behaviours.
You don't have to use context.fail, use success but send different statusCode and an errorMessage, here is an example of how i format my output:
try {
// Call the callable function with the defined array parameters
// All the function called here will be catched if they throw exceptions
result.data = callable_function.apply(this, params);
result.statusCode = 200;
result.operation = operation;
result.errorMessage = ""
} catch (e) {
result.data = [];
result.statusCode = 500;
result.errorMessage = e.toString();
result.method = method;
result.resource = resource;
}
// If everything went smooth, send back the result
// If context succeed is not called AWS Lambda will fire the function
// again because it is not successfully exited
context.succeed(result);
Use the consumer logic to handle different errors case logic, don't forget that you pay for the time your function is running...
You should replace the use of your context.fail with context.done and use context.fail only for very serious Lambda function failures since it doesn't allow more than one output parameter. Integration Response is able to match mapping template by performing regex on the first parameter passed to context.done this also maps HTTP status code to the response. You can't pass this response status code directly from Lambda since it's the role of API Gateway Integration Response to abstract the HTTP protocol.
See the following:
context.done('Not Found:', <some object you can use in the model>);
and the Integration Response panel this setting:
You can replicate similar approach for any kind of error. You should also create and map the error model to your response.
For those who tried everything put on this question and couldn't make this work (like me), check the thedevkit comment on this post (saved my day):
https://forums.aws.amazon.com/thread.jspa?threadID=192918
Reproducing it entirely below:
I've had issues with this myself, and I believe that the newline
characters are the culprit.
foo.* will match occurrences of "foo" followed by any characters
EXCEPT newline. Typically this is solved by adding the '/s' flag, i.e.
"foo.*/s", but the Lambda error regex doesn't seem to respect this.
As an alternative you can use something like: foo(.|\n)*
I'm trying to use Riak's mapreduce via http. his is what i'm sending:
{
"inputs":{
"bucket":"test",
"key_filters":[["matches", ".*"]]
},
"query":[
{
"map":{
"language":"erlang",
"source":"value(RiakObject, _KeyData, _Arg) -> Key = riak_object:key(RiakObject), Count = riak_kv_crdt:value(RiakObject, <<\"riak_kv_pncounter\">>), [ {Key, Count} ]."
}
}
]}
Riak fails with "[worker_startup_failed]", which isn't very informative. Could anyone please help me get this to actually execute the function?
WARNING
Allowing arbitrary Erlang functions via map-reduce is a security risk. Any valid Erlang can be executed, including sending your entire data set offsite or formatting the hard drive.
You have been warned.
However, if you implicitly trust any client that may connect to your cluster, you can allow Erlang source to be passed in a map-reduce request by setting {allow_strfun, true} in the riak_kv section of app.config, (or in the advanced.config if you are using riak.conf).
Once you have allowed passing an Erlang function in a map-reduce phase, you need to pass in a function of the form fun(RiakObject,KeyData,Arg) -> [result] end. Note that this must be an anonymous fun, so fun is a keyword, not a name, and it must end with end.
Your function should handle the case where {error,notfound} is passed as the first argument instead of an object. Simply adding a catch-all clause to the function could accomplish that.
Perhaps something like:
{
"inputs":{
"bucket":"test",
"key_filters":[["matches", ".*"]]
},
"query":[
{
"map":{
"language":"erlang",
"source":"fun(RiakObject, _KeyData, _Arg) ->
Key = riak_object:key(RiakObject),
Count = riak_kv_crdt:value(
RiakObject,
<<\"riak_kv_pncounter\">>),
[ {Key, Count} ];
(_,_,_) -> [{error,0}]
end."
}
}
]}
Allowing the source to be passed in the request is very useful while developing and debugging. For production, you really should put the functions in a dedicated pre-compiled module that you copy to the code path of each node so that the phase spec can specify the module and function by name instead of providing arbitrary code.
{"map":{
"language":"erlang",
"module":"yourprecompiledmodule",
"function":"functionname"}}
You need to enable allow_strfun on all nodes in your cluster. To do so in Riak 2, you will need to use the advanced.config file to add this to the riak_kv configuration:
[
{riak_kv, [
{allow_strfun, true}
]}
].
The other option is to create your own Erlang module by using the compiler shipped with Riak and placing the *.beam file in a well-known location for Riak to find. The basho-patches directory is one such place.
Please see the documentation as well:
advanced.config
Installing custom Erlang code
HTTP MapReduce
Using MapReduce
Advanced MapReduce
MapReduce / curl example
I'm having a doozie of a time trying to get a basic http test to work with vows.
I think I've followed the async example from vows http://vowsjs.org/#-writing-asynchronous-tests and substitued the appropriate calls, but I must be missing something.
The test code looks like this:
var http = require('http'),
vows = require('vows'),
assert = require('assert');
vows.describe("homepage").addBatch({
"Get the home page": {
topic: function() {
http.get({'host': "127.0.0.1", 'port': 5000, 'path': '/'}, this.callback);
},
'should respond with 200 OK': function(res) {
assert.equal(res.statusCode, 200);
}
}
}).export(module);
I get the following error when I try to run the test for this:
/Users/<home_folder>/node_modules/vows/lib/vows.js:80
rrored', { type: 'promise', error: err.stack || err.message || JSON.stringify(
^
TypeError: Converting circular structure to JSON
at Object.stringify (native)
at EventEmitter.<anonymous> (/Users/<home_folder>/node_modules/vows/lib/vows.js:80:90)
at EventEmitter.emit (events.js:64:17)
at /Users/<home_folder>/node_modules/vows/lib/vows/context.js:31:52
at ClientRequest.<anonymous> (/Users/<home_folder>/node_modules/vows/lib/vows/context.js:46:29)
at ClientRequest.g (events.js:143:14)
at ClientRequest.emit (events.js:64:17)
at HTTPParser.onIncoming (http.js:1349:9)
at HTTPParser.onHeadersComplete (http.js:108:31)
at Socket.ondata (http.js:1226:22)
I can get a simple http example to work on it's own. I can get the vows example to work on it's own but I can't combine them for whatever reason. I'd really appreciate some help here. I've been trying to get this to work for a while now (including much googling).
UPDATE:
Apparently adding an error argument to the call back solves this problem, thanks to help from Alexis Sellier (creator of vows).
But I have no idea why. When writing out the http lib example on it's own no error argument is required. I can't find any documentation in vows to indicate why it's needed so I'm at a bit of a loss.
My new question is why is the error argument required when using the http lib in vows?
After checking vow's source code, I think I know why. Vows always ensure that when you call this.callback, the resulting receiver function's first argument is always an error object. Vows interpret the callbacks by these rules:
If the first argument of your originating callback is a boolean, use that to determine whether or not to append an error object to the receiving callback (e.g. path.exists(boolean) will emit callback(error, exists) instead)
If the first argument is an object, assume it's an error object and use that to determine whether to add the originating callback to the "error" or "success" list. The reason this list exists is to support promise based tests I guess?
While I can't confirm the above is correct, my experience is that vows' async style is made to support node-styled callbacks (e.g. err as the first arg), and 3rd party npm modules that don't conform to this standard will be hard to test.
Please don't take my answer as gospel, as this is my own experience. Another gotcha is when you have async operations inside the function that you want to test - unless you provide a callback, vows won't be able to handle it properly.
Personally, I think vows still make it hard to test async code. I wish it had some waitFor() or until() flow control functions though.
My suggestion? When dealing with async code, use Step. Don't let vows control your flow.
It is actually missing in the documentations which is still a bit short. But you can get a glimpse of it here in this page :
'when peeled *asynchronously*': {
topic: function (banana) {
banana.peel(this.callback);
},
'results in a `PeeledBanana`': function (err, result) {
assert.instanceOf (result, PeeledBanana);
}
}
As it was said by Morten Siebuhr and Ruben Tan, this is how vows works and that is why it works like that.