1) What is better streamlinejs: https://github.com/Sage/streamlinejs
or narrative: http://www.neilmix.com/narrativejs/ ? any others libs?
2) How does any of those libraries even work?
(I read the docs, I am looking for a simplify explanation of what's going on behind the scene..)
As far as question #2....in general these things:
parse javascript into some abstract syntax tree (AST)
transform the AST
stringify the transformed tree back into javascript
I wrote a partial converter as a learning experience a while back. I used uglify.js to parse into an AST and then the tree walker that lib provides to do the transformations. The transformations were general purpose and produced code that looked like a state machine -- where each step started with a sequence of 0 or more sync actions and ended with an async action. E.g. this simple script:
var fs = require('fs');
console.log(fs.readFile('input.js', _).toString('utf-8'));
would get converted to this:
var fs, $v_0;
function s_0() {
fs = require("fs");
fs.readFile("input.js", function(err, res) {
if (err) s_err(err); else {
$v_0 = res;
s_1();
}
})
}
function s_1() {
console.log($v_0.toString("utf-8"));
}
s_0()
I imagine that streamline and the like do something very similar. Certain structures (loops, try/catch) need special handing but the general approach is the same -- convert into a state machine.
The issues with this approach that I found were:
1) it's not a local problem - i.e. any async behavior that needs to be handled infects everything all the way up the call stack.
2) you need function metadata so you either have to make assumptions or require people to annotate their functions in some manner.
Related
What is the sane way to go from a Module object to a path to the file in which it was declared?
To be precise, I am looking for the file where the keyword module occurs.
The indirect method is to find the location of the automatically defined eval method in each module.
moduleloc(mm::Module) = first(functionloc(mm.eval, (Symbol,)))
for example
moduleloc(mm::Module) = first(functionloc(mm.eval, (Symbol,)))
using DataStructures
moduleloc(DataStructures)
Outputs:
/home/oxinabox/.julia/v0.6/DataStructures/src/DataStructures.jl
This indirect method works, but it feels like a bit of a kludge.
Have I missed some inbuilt function to do this?
I will remind answered that Modules are not the same thing as packages.
Consider the existence of submodules, or even modules that are being loaded via includeing some abolute path that is outside the package directory or loadpath.
Modules simply do not store the file location where they were defined. You can see that for yourself in their definition in C. Your only hope is to look through the bindings they hold.
Methods, on the other hand, do store their file location. And eval is the one function that is defined in every single module (although not baremodules). Slightly more correct might be:
moduleloc(mm::Module) = first(functionloc(mm.eval, (Any,)))
as that more precisely mirrors the auto-defined eval method.
If you aren't looking for a programmatic way of doing it you can use the methods function.
using DataFrames
locations = methods(DataFrames.readtable).ms
It's for all methods but it's hardly difficult to find the right one unless you have an enormous number of methods that differ only in small ways.
There is now pathof:
using DataStructures
pathof(DataStructures)
"/home/ederag/.julia/packages/DataStructures/59MD0/src/DataStructures.jl"
See also: pkgdir.
pkgdir(DataStructures)
"/home/ederag/.julia/packages/DataStructures/59MD0"
Tested with julia-1.7.3
require obviously needs to perform that operation. Looking into loading.jl, I found that finding the module path has changed a bit recently: in v0.6.0, there is a function
load_hook(prefix::String, name::String, ::Void)
which you can call "manually":
julia> Base.load_hook(Pkg.dir(), "DataFrames", nothing)
"/home/philipp/.julia/v0.6/DataFrames/src/DataFrames.jl"
However, this has changed to the better in the current master; there's now a function find_package, which we can copy:
macro return_if_file(path)
quote
path = $(esc(path))
isfile(path) && return path
end
end
function find_package(name::String)
endswith(name, ".jl") && (name = chop(name, 0, 3))
for dir in [Pkg.dir(); LOAD_PATH]
dir = abspath(dir)
#return_if_file joinpath(dir, "$name.jl")
#return_if_file joinpath(dir, "$name.jl", "src", "$name.jl")
#return_if_file joinpath(dir, name, "src", "$name.jl")
end
return nothing
end
and add a little helper:
find_package(m::Module) = find_package(string(module_name(m)))
Basically, this takes Pkg.dir() and looks in the "usual locations".
Additionally, chop in v0.6.0 doesn't take these additional arguments, which we can fix by adding
chop(s::AbstractString, m, n) = SubString(s, m, endof(s)-n)
Also, if you're not on Unix, you might want to care about the definitions of isfile_casesensitive above the linked code.
And if you're not so concerned about corner cases, maybe this is enough or can serve as a basis:
function modulepath(m::Module)
name = string(module_name(m))
Pkg.dir(name, "src", "$name.jl")
end
julia> Pkg.dir("DataStructures")
"/home/liso/.julia/v0.7/DataStructures"
Edit: I now realized that you want to use Module object!
julia> m = DataStructures
julia> Pkg.dir(repr(m))
"/home/liso/.julia/v0.7/DataStructures"
Edit2: I am not sure if you are trying to find path to module or to object defined in module (I hope that parsing path from next result is easy):
julia> repr(which(DataStructures.eval, (String,)))
"eval(x) in DataStructures at /home/liso/.julia/v0.7/DataStructures/src/DataStructures.jl:3"
I think it is possible but I am looking for a way to map base types in R using rprotobuf package. What I want is to create a network/server very similar to Rserve but using protocol buffers to serialize the data rather than Rserve's QAP protocol. My question is how would it be possible to map something like a data.frame into a protocol buffer. Here is an example of kind of what I would like it to look like but let me know if I am going about it the wrong way.
message TextCell {
required string name = 1;
}
message NumericCell {
repeated int32 num 1;
}
message TextColumn {
repeated TextCell text 1;
}
message NumericColumn {
repeated NumericCell number 1;
}
message DataFrame {
optional NumericColumn numbericColumn = 1;
optional TextColumn textColumns = 2;
}
I mocked this up just now so it will probably have errors but this is the concept that I am looking at and it doesn't take into account things like Doubles which seems like a bad idea. Would it possibly be a better solution to use a bytes type and deserialize the column on the other side. Not sure how to attack this problem yet and feedback would be greatly appreciated from more knowledgeable people.
Note, I wish to use protocol buffers due to their storage efficiency and the possibility to use many more languages but there is nothing wrong with the QAP protocol. It is very fast and efficient.
Thanks in advance
We've integrated the catch-all Protocol Buffer schema for R objects from RHIPE into RProtoBuf and new functions serialize_pb and unserialize_pb to convert arbitrary R objects such as data.frames into protocol buffers. For example:
msg <- tempfile();
serialize_pb(iris, msg);
obj <- unserialize_pb(msg);
identical(iris, obj);
This functionality was introduced in RProtoBuf 0.4 which came out after your question was originally asked. See a preprint of our JSS paper that introduces these new features on arXiv : RProtoBuf: Efficient Cross-Language Data Serialization in R
I try to implement a list structure reading FileSystem Directory tree using node.js
DIR/file structure:
DIR1
DIR2R
file1
file2
file3
…
-> list structure:
("DIR1" ("DIR2" "file1" "file2" "file3" …))
DIR() //looks like a C function
file //looks like an Atom value
so, for instance, a given UNIX root DIR:
root bin cd
ls
lib file1
file2
opt
usr file3
file4
is equivalent to a list:
("root" ("bin" ("cd" "ls")
"lib" ("file1" "file2")
"opt" ()
"usr" ("file3")
"file4"
)
)
Having said that, I want to implement this code with some lazy async sequence (stream/ Infinite list?).
node.js fs.readdir recursive directory search
is a good reference.
node.js has many excellent libraries such as file.walk etc. , but this is a very interesting subject, in fact, many code examples exsist.
In other words, a good subject to learn a basic concept of lazy async sequence (stream/ Infinite list?), and the code can be a good library, I want to implement from a scratch.
substack/stream-handbook covers the basics of how to write node.js
programs with streams .
Truly Excellent article.
so, here is my thought on this subject - Recursive Dir tree processing:
This subject deal with a recursive search, and there are 2 ways: async and sync , for various reasons, I chose async method.
The search result is equivalent to a list structure, where Functional programming paradigm does matter (Lisp/Scheme).
For JavaScript Functional programming, there are libraries such as underscore.js, and recently, lazy.js with one important difference: lazy evaluation (also known as deferred execution) where appears to outperform underscore.js.
lazy.js wraps Stream processing in Node.js as well. I also found stream.js says: an infinite number of elements. Their power comes from being lazily evaluated, which in simple terms means that they can contain infinite items. .
Stream data(infinite list) realizes Reactive Functional Programming(FRP) paradigm. lazy.js actually demonstrates a mouse event FRP implementation. RxJS is a FRP library: RxJS or Reactive Extensions for JavaScript is a library for transforming, composing, and querying streams of data.
async method of this subject is a FRP subject.
Therefore, lazy evaluation is a core factor of faster List processing Functional Programming, and expands list to infinite list(stream) which integrates async/events to Stream data source to process in Functional Programming paradigm (FunctionalReactiveProgramming).
node-lazy Introduction states clearly: Lazy comes really handy when you need to treat a stream of events like a list. The best use case currently is returning a lazy list from an asynchronous function, and having data pumped into it via events. In asynchronous programming you can't just return a regular list because you don't yet have data for it. The usual solution so far has been to provide a callback that gets called when the data is available. But doing it this way you lose the power of chaining functions and creating pipes, which leads to not that nice interfaces. (See the 2nd example below to see how it improved the interface in one of my modules.)
At last, here is my Question on this subject - Recursive Dir tree processing:
Are there any sample code to implement this subject exclusively in lazy evaluation or FRP manner?
I know lazy.js, stream.js, or linq.js is based on lazy evaluation, but any of them can not define node.js fs.readdir as a stream data for async processing (FRP ). According to http://danieltao.com/lazy.js/docs/AsyncSequence.html , it's not yet implement, correct?
RxJS probably can, but I don't know how.
Google rx.Observable fs.readdir hits nothing.
Any ideas?
Here's a start based on https://gist.github.com/edygar/ee0945a73c79182367df
Given a directory it produces a list of subdirectories.
(coffeescript)
Rx = require "rx"
fs = require "fs"
readdir = Rx.Observable.fromNodeCallback fs.readdir
stat = Rx.Observable.fromNodeCallback (pathName, cb) ->
fs.stat pathName, (err, stats) ->
stats.pathName = pathName
cb err, stats
dirObservable = (dirPath) ->
readdir dirPath
.flatMap (items) -> Rx.Observable.from items
.flatMap (item) -> stat item
.filter (stats) -> stats.isDirectory()
.map (stats) -> stats.pathName
module.exports = dirObservable
if not module.parent
path = require "path"
dirobs = dirObservable path.resolve __dirname, ".."
dirobs.subscribe (data) -> console.log data
I just started developing a frama-c plugin that is doing some kind of alias analysis. I'm using the Dataflow.Backwards analysis and now I have to go through the different assignment statements and collect some stuff about the lvalues.
Does frama-c provide me with 3-address code? Do I have some guarantees about the shape of the lvalue (or any memory access)? I mean, sth like in soot or wala that there is at most one field access, s.t., for a->b->c, there would be a temp variable like tmp=a->b; tmp->c;? I checked the manuals, but I couldn't find anything related to this.
No, there is no such normalization in Frama-C. If you really need it, you can first use a visitor in order to normalize the code so that it suits the requirements of your plug-in. It'd go like that:
class normalize prj: Visitor.frama_c_visitor =
object
inherit Visitor.frama_c_copy prj
method vinstr i =
match i with
| Set (lv,e) -> ...
....
end
let analyze () = ...
let run () =
let my_prj = File.create_project_from_visitor "my_project" (fun prj -> new normalize prj) in
Project.on my_prj analyze ()
The following module from Cil does probably what you want:
http://www.cs.berkeley.edu/~necula/cil/ext.html#toc26. Be aware that the type of the resulting AST is the standard Cil one. You won't be getting any help from the OCaml compiler as to which constructs can be present in the simplified AST, and which ones cannot.
Note also that this module has not been ported to Frama-C so far. You will need some minor adaptation to make it work within Frama-C.
I have a problem where the Closure Compiler renames a global variable something like x.sa.xa but in all function where that global variable is referenced the compiler renames it something else like H.sa.xa
When I view the HTML page I get a JavaScript TypeError: H.sa.xa is undefined.
// Top-level namespace for all the code
var nam = nam || {};
(function($, nam) {
goog.provide('nam.jsConfig');
nam.jsConfig.cookies = {"RECENT_ITEMS": "recentitems"};
})($, nam);
(function($, nam) {
goog.provide('nam.util.cookie');
nam.util.cookie.readMyCookie = function () {
var ritems_cookie = nam.util.cookie.JSONCookie.get(nam.jsConfig.cookies['RECENT_ITEMS']);
};
})($, nam);
Closure Compiled Code:
x.sa = {};
x.sa.xa = {RECENT_ITEMS:"recentitems"};
H.a = {};
H.a.cookie = {};
H.a.Tm = function() {
var a = H.a.cookie.ja.get(H.sa.xa.RECENT_ITEMS);
};
For some reason the Closure Compiler is referencing H.sa.xa.RECENT_ITEMS instead of x.sa.xa.RECENT_ITEMS
Any reason why the compiler is doing this this?
The only way I can interpret your question is that one of two things is happening:
There is an issue with the Closure Compiler's obfuscating and minimizing code, or
The error you are seeing is from JavaScript running outside of the code compiled by the Closure Compiler that is referencing a compiled variable directly.
If it is the former, you should isolate the case that is causing variable misalignment and submit it as a bug to Google. All of us using the Closure Compiler would greatly appreciate it.
If instead, as I suspect, it is the latter, you are most likely not exporting the global variable you wish to use outside of the compiled code. The easiest way to do this is to call goog.exportSymbol() function to make the global variable available outside of your code assembled by the Closure Compiler. For example, if you wished to access the property sandwich.meat.Ham in compiled mode from non-compiled code, you could do the following:
goog.exportSymbol('sandwich.meat.Ham', sandwich.meat.Ham);
Then you could have some code that exists outside of your compiled code that references the exported variable:
function() {
var meat = new sandwich.meat.Ham();
}
Let me guess what you are doing: compiling each file independently in ADVANCED mode. If so, this isn't how ADVANCED mode works. In advanced mode if you want to share variable and properties between compilations jobs you need to export them.
There are much more significant issues in the code example you provided. For one
goog.provide('nam.util.cookie');
was turned into
H.a = {};
H.a.cookie = {};
Yet later this code:
nam.util.cookie.readMyCookie = function () {...
was turned into
H.a.Tm = function() {...
Where one would expect it should be
H.a.cookie.Tm = function() {...
Additionally, the fact that you use nam as the base namespace for both halves of the uncompiled code and that it gets turned into separate x and H namespaces, respectively, also suggests more is at play. Some suggestions:
If you wish to use the module pattern, put the provide/require statements outside of the module
Don't manually create namespaces with stuff like var nam = nam || {} because provide does this for you already
As others have mentioned, both files containing nam.jsConfig and nam.util.cookie should be included in a single compilation
Make sure you goog.require('nam.jsConfig') in the file with nam.util.cookie.readMyCookie, to ensure the dependency requirements are met
FWIW, we use closure in an extensive application with hundreds of files, containing interdependencies like this. I would be highly suspect that the issue lies not with the tools, but instead with how they are being used.