I have a linux console application - a scientific simulation program that I use. It opens up a TCL shell that you then issue commands to. Normally what I do is pre-write all my test vectors and look at the output by manually inputting the data, but now I'd like to move on to something more complicated: incorporating external feedback.
My idea is, I'll have a external simulation running that takes the output of the simulator and then generates new test vectors on the fly to feed back into the simulation. I'm kind of hazy on the on the details of how to implement this. I am semi-familiar with C and with Python.
I guess, getting into specifics - how do I hook into the program's terminal I/O? I'd prefer to use Python if possible. Are there any references I can read to get up to speed on this?
Your idea is quite reasonable. Python supports this very well: Sub-process launching, and inter-process communication. Documentation like the following might be helpful:
http://docs.python.org/library/subprocess.html
In short, you're going to "read from" child-process stdout (and maybe stderr), and "write to" child-process stdin. You can have your interactive console like you describe, or read from/write to text files, and even "hook-together" processes to talk (like piping "mycommand | mycommand2").
For Python, there are many strong examples (like the "scons" build system written in Python that does this a lot). Further, the Qt's QProcess class makes this pretty easy, and there are a few really good Python wrappers available like "PySide", "PyQt", and "PythonQt" (probably others).
Related
How are we actually supposed to include our OpenCL code in our C projects?
We can't possibly be supposed to ship our .cl files along with our executable for the executable to find them and load them at runtime, because that's stupid, right?
We can't be supposed to use some stringify macro because a) that's apparently not portable/leads to undefined behaviour and b) it all breaks down if you use commas not enclosed in brackets like when defining many variables of the same type, I've spent an hour here looking for a solution to that and there doesn't seem to be one that actually works and c) that's kind of stupid.
Are we expected to write our code into C string literals like "int x, y;\n" "float4 p;\n"? Because I'm not doing that. Are we supposed to do a C include-style hexdump of our .cl files? That seems inconvenient. What are we actually supposed to do?
It's bad enough that all these approaches basically mean that you have to ship your program with your OpenCL code essentially open sourced when your OpenCL code is probably the last thing you want open sourced, on top of it it seems every OpenCL project I've seen uses one of the approaches listed above, it just doesn't seem right at all, it's like the people who made OpenCL forgot about something.
This thread: OpenCL bytecode running on another card mentions SPIR, a "platform-portable intermediate representation for OpenCL device programs". Other than that, you are basically restrained to the options you already mentioned.
Personally, I began to use C++11 raw string literals to get rid of my nasty stringify-macros. Don't know if C++ is an option for you, however.
Concerning your rejection of the "ship our .cl files along with our executable" approach: I don't see why this is inherently stupid -- the CL "shaders" are an application resource like all other separate files beside the executable, and thus are part of the "application bundle". It's perfectly reasonable to have such kind of files, and each operating system has its way to deal with it (in win32, the program directory is the bundle https://blogs.msdn.microsoft.com/oldnewthing/20110620-00/?p=10393 , OSX has its own bundle concept, etc...).
Now, if you are worried about other people peeking into your OpenCL code, you can still apply some obfuscation methods (e.g. encrypt your .cl-files by a key which is more or less cleverly hidden in your executable).
[edit/sidenote]: We could also investigate how other companies deal with this issue in the context of, for example, OpenGL/Direct3D shaders. In my limited experience, gaming companies tend to dump their shaders in text form somewhere in their application directory, for all to see (and even to tamper with). So in the gaming world at least, there is no great deal of secrecy in that respect... Wonder what adobe or CAD software companies do in their professional software.
Recently I am studying operating system..I just wanna know:
What’s the difference between a system call (like write()) and a standard library function (like printf())?
A system call is a call to a function that is not part of the application but is inside the kernel. The kernel is a software layer that provides you some basic functionalities to abstract the hardware to you. Roughly, the kernel is something that turns your hardware into software.
You always ultimately use write() to write anything on a peripheral whatever is the kind of device you write on. write() is designed to only write a sequence of bytes, that's all and nothing more. But as write() is considered too basic (you may want to write an integer in ten basis, or a float number in scientific notation, etc), different libraries are provided to you by different kind of programming environments to ease you.
For example, the C programming langage gives you printf() that lets you write data in many different formats. So, you can understand printf() as a function that convert your data into a formatted sequence of bytes and that calls write() to write those bytes onto the output. But C++ gives you cout; Java System.out.println, etc. Each of these functions ends to a call to write() (at least on POSIX systems).
One thing to know (important) is that such a system call is costly! It is not a simple function call because you need to call something that is outside of your own code and the system must ensure that you are not trying to do nasty things, etc. So it is very common in higher print-like function that some buffering is built-in; such that write is not always called, but your data are kept into some hidden structure and written only when it is really needed or necessary (buffer is full or you really want to see the result of your print).
This is exactly what happens when you manage your money. If many people gives you 5 bucks each, you won't go deposit each to the bank! You keep them on your wallet (this is the print) up to the point it is full or you don't want to keep them anymore. Then you go to the bank and make a big deposit (this is the write). And you know that putting 5 bucks to your wallet is much much faster than going to the bank and make the deposit. The bank is the kernel/OS.
System calls are implemented by the operating system, and run in kernel mode. Library functions are implemented in user mode, just like application code. Library functions might invoke system calls (e.g. printf eventually calls write), but that depends on what the library function is for (math functions usually don't need to use the kernel).
System Call's in OS are used in interacting with the OS. E.g. Write() could be used something into the system or into a program.
While Standard Library functions are program specific, E.g. printf() will print something out but it will only be in GUI/command line and wont effect system.
Sorry couldnt comment, because i need 50 reputation to comment.
EDIT: Barmar has good answer
I am writing a small program. At the moment it just reads each line from stdin and prints it to stdout. I can add a call to write in the loop, and it would add a few characters at the end of each line. But when I use printf instead, then all the extra characters are clustered and appear all at once, instead of appearing on each line.
It seems that using printf causes stderr to be buffered. Adding fflush(stdout); after calling printf fixes the discrepancy in output.
I'd like to mention another point that the stdio buffers are maintained in a process’s user-space memory, while system call write transfers data directly to a kernel buffer. It means that if you fork a process after write and printf calls, flushing may bring about to give output three times subject to line-buffering and block-buffering, two of them belong to printf call since stdio buffers are duplicated in the child by fork.
printf() is one of the APIs or interfaces exposed to user space to call functions from C library.
printf() actually uses write() system call. The write() system call is actually responsible for sending data to the output.
Quite straight forward question. I would like a program to read its output.
This is not a common thing to do, but it makes a whole lot of tests I have a lot easier as the program can test its output and return success or failure.
Note I can't just read the output as i'm testing the compiler and runtime.
Obviously I can use 2 programs but I would prefer each test to be independent.
There are many nice things to like about Makefiles, and many pains in the butt.
In the course of doing various project (I'm a research scientist, "data scientist", or whatever) I often find myself starting out with a few data objects on disk, generating various artifacts from those, generating artifacts from those artifacts, and so on.
It would be nice if I could just say "this object depends on these other objects", and "this object is created in the following manner from these objects", and then ask a Make-like framework to handle the details of actually building them, figuring out which objects need to be updated, farming out work to multiple processors (like Make's -j option), and so on. Makefiles can do all this - but the huge problem is that all the actions have to be written as shell commands. This is not convenient if I'm working in R or Perl or another similar environment. Furthermore, a strong assumption in Make is that all targets are files - there are some exceptions and workarounds, but if my targets are e.g. rows in a database, that would be pretty painful.
To be clear, I'm not after a software-build system. I'm interested in something that (more generally?) deals with dependency webs of artifacts.
Anyone know of a framework for these kinds of dependency webs? Seems like it could be a nice tool for doing data science, & visually showing how results were generated, etc.
One extremely interesting example I saw recently was IncPy, but it looks like it hasn't been touched in quite a while, and it's very closely coupled with Python. It's probably also much more ambitious than I'm hoping for, which is why it has to be so closely coupled with Python.
Sorry for the vague question, let me know if some clarification would be helpful.
A new system called "Drake" was announced today that targets this exact situation: http://blog.factual.com/introducing-drake-a-kind-of-make-for-data . Looks very promising, though I haven't actually tried it yet.
This question is several years old, but I thought adding a link to remake here would be relevant.
From the GitHub repository:
The idea here is to re-imagine a set of ideas from make but built for R. Rather than having a series of calls to different instances of R (as happens if you run make on R scripts), the idea is to define pieces of a pipeline within an R session. Rather than being language agnostic (like make must be), remake is unapologetically R focussed.
It is not on CRAN yet, and I haven't tried it, but it looks very interesting.
I would give Bazel a try for this. It is primarily a software build system, but with its genrule type of artifacts it can perform pretty arbitrary file generation, too.
Bazel is very extendable, using its Python-like Starlark language which should be far easier to use for complicated tasks than make. You can start by writing simple genrule steps by hand, then refactor common patterns into macros, and if things become more complicated even write your own rules. So you should be able to express your individual transformations at a high level that models how you think about them, then turn that representation into lower level constructs using something that feels like a proper programming language.
Where make depends on timestamps, Bazel checks fingerprints. So if at any one step produces the same output even though one of its inputs changed, then subsequent steps won't need to get re-computed again. If some of your data processing steps project or filter data, there might be a high probability of this kind of thing happening.
I see your question is tagged for R, even though it doesn't mention it much. Under the hood, R computations would in Bazel still boil down to R CMD invocations on the shell. But you could have complicated muliti-line commands assembled in complicated ways, to read your inputs, process them and store the outputs. If the cost of initialization of the R binary is a concern, Rserve might help although using it would make the setup depend on a locally accessible Rserve instance I believe. Even with that I see nothing that would avoid the cost of storing the data to file, and loading it back from file. If you want something that avoids that cost by keeping things in memory between steps, then you'd be looking into a very R-specific tool, not a generic tool like you requested.
In terms of “visually showing how results were generated”, bazel query --output graph can be used to generate a graphviz dot file of the dependency graph.
Disclaimer: I'm currently working at Google, which internally uses a variant of Bazel called Blaze. Actually Bazel is the open-source released version of Blaze. I'm very familiar with using Blaze, but not with setting up Bazel from scratch.
Red-R has a concept of data flow programming. I have not tried it yet.
another quick question, I want to make simple console based game, nothing too fancy, just to have some weekend project to get more familiar with C. Basically I want to make tetris, but I end up with one problem:
How to let the game engine go, and in the same time wait for input? Obviously cin or scanf is useless for me.
You're looking for a library such as ncurses.
Many Rogue-like games are written using ncurses or similar.
There's two ways to do it:
The first is to run two threads; one waits for input and updates state accordingly while the other runs the game.
The other (more common in game development) way is to write the game as one big loop that executes many times a second, updating game state, redrawing the screen, and checking for input.
But instead of blocking when you get key input, you check for the presence of pending keypresses, and if nothing has happened, you just continue through your loop. If you have multiple input sources (keyboard, network, etc.) they all get put there in the loop, checking one after another.
Yes, it's called polling. No, it's not efficient. But high-end games are usually all about pulling the maximum performance and framerates out of the computer, not running cool.
For added efficiency, you can optionally block with a timeout -- saying "wait for a keypress, but no longer than 300 milliseconds" so you can continue on with your loop.
select() comes to mind, but there are other ways of waiting or checking for input as well.
You could work out how to change stdin to non-blocking, which would enable you to write something like tetris, but the game might be more directly expressed in an event-driven paradigm. Maybe it's a good excuse to learn windows programming.
Anyway, if you want to go the console route, if you are using the microsoft compiler, then you should have kbhit() available (via conio.h) which can tell you whether a call to fgetc on stdin would block.
Actually should mention that the MinGW gcc compiler 3.4.5 also supports kbhit().