I am building a system in Scala for feature engineering where the end user API receives aggregations on list of objects/events.
For example, a client of this tool might pass into it a function that given an array of past pageviews for a specific web user, filters for the ones coming from a specify country and count them. The output of this call will then be a number.
It can be thought as a very simple reduce operation.
I am figuring out how to build the API for such system. I could write a simple custom language to perform counts and filters but I am sure that it is not the best approach, especially because it will not be expressive enough, unless designed with care.
Are you aware of something like an expression language that could be used to express simple functions without the need for me to build one from scratch?
The other option would be to allow end users to pass custom code to this library, which might be dangerous at runtime.
I am aware of apache Calcite to plug SQL into different data structures and db. It is a good option, however it forces me to think in "columnar" sql way, while here I am looking more for something row based, similar to the map-reduce way of programming.
You mention Apache Calcite, but did you know that you can use it without writing SQL? If you use Calcite's RelBuilder class you can build algebra directly, which is very similar to the algebraic approach of MapReduce. See algebra in Calcite for more details.
Related
I want to implement some game logic with a lot of relations between objects similar to those of relational databases or graph databases.
I know no language that would both allow me to do :
Strong, safe relationnal mapping with non nullable links, cascade deletion, ect.
Implement game logic
Write pure functions
Networking
If possible, a decent data access performance. (Like in-memory SQLlite is acceptable)
I want to avoid using 2 languages and map the data between both using some quite complex ORM. Instead I would like a language that is capable of all of these.
Obsiouly, there is SQL. But I do not know any implementation of SQL that :
Is capable of networking other than replying to SQL requests
Have the many features of a language like F# ? SQL is capable of functional programming but what about F# features like pipes, partial applications, pattern matching, strong typing over primitive types ?
I will accept partial alternative solutions.
Note that I do not need actual persistance storage, only objects relation like relationnal databases, or even graph databases do.
The answer is no, within the bounds as you have set them.
The purpose of The Third Manifesto is to define a language called D, which has the features of a general purpose programming language but implements a type system and relational features specifically aimed at database management. If implemented fully it might replace SQL, but not common GP languages such as C/C++, Java or C#.
There are many GP languages which can do all the things you propose, when used in conjunction with suitably chosen libraries. For the closest match to what you describe, you should stick with any language that suits your other needs, and add to it an in-memory in-process database that uses an API and not SQL. Almost by definition that means you should look for a 'NoSQL' database. There are many.
You question was mentioned here: https://forum.thethirdmanifesto.com/forum/topic/is-there-a-language-that-does-both-what-sql-does-and-general-purpose-programming/. You might find subsequent discussion enlightening.
I am interested in doing a project relying on automated proofs, in great dimension as a learning exercise. So far my online search suggests Lean is the way to go, in theory.
However, all I read about it talks about using it as a proof assistant in VS code or Emacs. But that's not what I need, I need a system I can communicate with fully programmatically. I.E string of assumptions goes in -> string specifying deductibility comes out or something like that.
To be more precise, I need to be able to call parsing functions on strings that do the heavy work of determining whether a set of results is deducible from the input assumptions.
I cant find documentation about Lean being able to do this.
I need to develop a web app that infers new rules from a given rule base. Example: A developer advised to modify a source code might reflect changes on other files, so he has to subsequently modify the affecting files also. So my engine should be capable of inferring such new rules. Can I use Windows Workflow Foundation ?
Platform: ASP.net
Short answer: No, not natively.
WF it's an API which allows you to write, through simple steps called activities, complex processes to be executed by the engine.
Those steps can be control flow activities like If, While, Switch, etc. or expressions like Add, Multiple, And, Or and many others. There's also the concept of variables, inputs and outputs. You can also write your activities to execute whatever code you want.
So basically you've all the goodies of procedural programming (including state) and you can design whatever processes you want on top of that, including a inference engine. Your workflow can be the interpreter, receiving input, and outputting your inferences.
PS: there's a native StateMachine, you may want to start from there.
I have been thinking about using JDO in my new project. However, there is always the possibility of changing the Programming language, or rewriting / adding some modules in other languages. But i fear that in that case the data may not be accessible (parseable, understandable).
Are my fears justified?
If you persist data using JDO to say RDBMS, then the data is held in RDBMS (in a logical well-defined structure) which is equally accessible using SQL via a different programming language. The datastore is what holds the data, not some mysterious "JDO" thing, which is simply an API to get the data there from a Java app and Java objects.
Getting data from a database table to an object in code has always seemed like mundane code. There are two ways I have found to do it:
have a code generator that reads a database table and creates the
class and controller to map the datafields to the class fields or
use reflection to take the database field and find it on the class.
The problems noted with the above 2 methods are as noted below
Method 1 seems to me like I'm missing something because I have to create a controller for every table.
Method 2 seems to be too labor intensive once you get into heavy data
access code.
Is there a third route that I should try to get data from a database onto my objects?
You normally use OR (Object-Relational) mappers in such situations. A good framework providing OR functionality is Hibernate. Does this answer your question?
I think the answer to this depends on the available technologies for the language you are going to use.
I for one am very successful with the use of an ORM (NHibernate) so naturally I may recommend option one.
There are other options that you may wish to take though:
If you are using .NET, you may opt to use attributes for your class properties to serve either as a mapping within a class, or as data that can be reflected
If you are using .NET, Fluent NHibernate will make it quite easy to make type-safe mappings within your code.
You can use generics so that you will not need to make a controller for every table, although I admit that it will be likely that you will do the latter anyway. However the generics can contain most of the general CRUD methods that is common to all tables, and you will only need to code specific quirks.
I use reflection to map data back and forth and it works well even under heavy data access. The "third route" is to do everything by hand, which may be faster to run but really slow to write.
I agree with lewap, an ORM (object-relational mapper) really helps in these situations. You may also want to consider the Active Record pattern (discussed in Fowler's Patterns of Enterprise Architecture book). It can really speed up creation of the DAL in simple apps.