I am working using Isabelle console to load theories Thy_Info.use_thy and some of the theories fail when getting response from Isabelle server
### theory "CParser.TypHeap"
### 2.100s elapsed time, 10.446s cpu time, 0.000s GC time
*** exception Fail raised (line 66 of "System/isabelle_system.ML"): Malformed result from bash_process server
*** At command "apply" (line 1521 of "/workspace/l4v/tools/c-parser/umm_heap/TypHeap.thy")
Exception-
CONTEXT
(<context>,
EXCURSION_FAIL
(CONTEXT
(<context>, Fail "Malformed result from bash_process server"),
"At command \"apply\" (line 1521 of \"/workspace/l4v/tools/c-parser/umm_heap/TypHeap.thy\")"))
raised
Poly/ML>
When I run Isabelle build with that specific theory in the ROOT file it succeeds but use_thy still fails.
I've also seen this error from Isabelle client
*** Malformed message header: "OK {"isabelle_id":"c2a2be496f35","isabelle_name":"Isabelle2021-1"}"
*** At command "by" (line 139 of "~/workspace/l4v/lib/More_Numeral_Type.thy")
When I run isabelle build I see bash processes being completed without server running.
When does a theory require a serve and any ideas the issues with my configuration?
Is there a way to interact with Isabelle server programmatically sending ML code to be executed?
Thanks!
The server is only needed for interactive sessions, not for batch builds. The server is also not needed for the Isabelle console, which is a separate application.
The Isabelle console is the only way to interact with the server.
In this case, to process the response from the server, the Isabelle console needs to know the exact format of the response. This is not part of the Isabelle build system, but part of the Isabelle console. The Isabelle console is a separate application, not part of the Isabelle build system.
Related
I am trying to align my logging with the best practice of using STDERR.
So, I would like to understand what happens with the logs sent to STDERR.
Symfony official docs (https://symfony.com/doc/current/logging.html):
In the prod environment, logs are written to STDERR PHP stream, which
works best in modern containerized applications deployed to servers
without disk write permissions.
If you prefer to store production logs in a file, set the path of your
log handler(s) to the path of the file to use (e.g. var/log/prod.log).
This time I want to follow the STDERR stream option.
When I was writing to a specific file, I knew exactly where to look for that file, open it and check the logged messages.
But with STDERR, I don't know where to look for my logs.
So, using monolog, I have the configuration:
monolog:
handlers:
main:
type: fingers_crossed
action_level: error
handler: nested
excluded_http_codes: [404, 405]
nested:
type: stream
path: "php://stderr"
level: debug
Suppose next morning I want to check the logs. Where would I look?
Several hours of reading docs later, my understanding is as follows:
First, the usage of STDERR over STDOUT is preferred for errors because it is not buffered (gathering all output waiting for the script to end), thus errors are thrown immediately to the STDERR stream. Also, this way the normal output doesn't get mixed with errors.
Secondly, the immediate intuitive usage is when running a shell script, because in the Terminal one will directly see the STDOUT and STDERR messages (by default, both streams output to the screen).
But then, the non-intuitive usage of STDERR is when logging a website/API. We want to log the errors, and we want to be able to monitor the already occurred errors, that is to come back later and check those errors. Traditional practice stores errors in custom defined log-files. More modern practice recommends sending errors to STDERR. Regarding Symfony, Fabien Potencier (the creator of Symfony), says:
in production, stderr is a better option, especially when using
Docker, SymfonyCloud, lambdas, ... So, I now recommend to use
php://stderr
(https://symfony.com/blog/logging-in-symfony-and-the-cloud).
And he further recommends using STDERR even for development.
Now, what I believe to be missing from the picture (at least for me, as non-expert), is the guidance on HOW to access and check the error logs. Okay, we send the errors to STDERR, and then? Where am I going to check the errors next morning? I get it that containerized platforms (clouds, docker etc) have specific tools to easily and nicely monitor logs (tools that intercept STDERR and parse the messages in order to organize them in specific files/DBs), but that's not the case on a simple server, be it a local server or on a hosting.
Therefore, my understanding is that sending errors to STDERR is a good standardization when:
Resorting to using a third-party tool for log monitoring (like ELK, Grail, Sentry, Rollbar etc.)
When knowing exactly where your web-server is storing the STDERR logs. For instance, if you try (I defined a new STD_ERR constant to avoid any pre-configs):
define('STD_ERR', fopen('php://stderr', 'wb'));
fputs(STD_ERR, "ABC error message.");
you can find the "ABC error message" at:
XAMPP Apache default (Windows):
..\xampp\apache\logs\error.log
Symfony5 server (Windows):
C:\Users\your_user\.symfony5\log\ [in the most recent folder, as the logs rotate]
Symfony server (Linux):
/home/your_user/.symfony/log/ [in the most recent folder, as the logs rotate]
For Symfony server, you can actually see the logs paths when starting the server, or by command "symfony server:log".
One immediate advantage is that these STDERR logs are stored outside of the app folders, and you do not need to maintain extra writable folders or deal with the permissions etc. Of course, when developing/hosting multiple sites/apps, you need to configure the error log (the STDERR storage) location per app (in Apache that would be inside each <VirtualHost> conf ; with Symfony server, I am not sure). Personally, without a third-party tool for monitoring logs, I would stick with custom defined log files (no STDERR), but feel free to contradict me.
I'm in the (ever-going) process of diagnosing a baffling problem with an application we use at work.
First, some notes about this application:
Required to run as the root user
Runs on the Solaris 10 operating system
Compiled for C++14
Normal shutdown is conducted by receiving SIGTERM
Writes a log file (explicitly sets permissions to 660) to a data directory (with default 770 permissions)
The application runs fine and does everything it's supposed to do, up until the point it terminates. Upon termination, the application is changing the permissions on the data directory from 770 to 660.
My coworkers are as baffled as I am. Even our system administrator doesn't understand why this is happening.
Things I've tried:
Print statements: The application reports the directory permissions are 770 until the exit or return statements
Check the logging: The logging mechanism is shared with several other applications, none of which have this issue
Running as myself: The directory's permissions are not changed on termination
Change umask to 027: The directory's permissions are still changed to 660
Check system logs: The sudo and messages logs do not show any calls to chmod for the directory (except those made to change the permissions back)
Due to the nature of the application, I cannot provide any of the code here for review/inspection. Further, many of the standard diagnostics tools are unavailable on the system in question.
However, I'm hopeful the gurus here can provide insight into what might be causing this problem, where to look going forward, or (ideally) how to fix it.
You can use the following simple dTrace script to get a stack trace from any process that calls chmod() (or any of its variants such as fchmod() or fchmodat()):
#!/usr/sbin/dtrace -s
syscall::fchmodat:entry
{
printf{ "\nExecname: %s\n", execname );
ustack();
}
You can filter by execname to only print chmod call stacks from your executable with
#!/usr/sbin/dtrace -s
syscall::fchmodat:entry
/ execname == "yourExecName" /
{
ustack();
}
You can add more or less stack frames with ustack( 10 ); to print, for example, 10 stack frames. If you want longer or shorter function names in the stack trace, you can specify the string length with ustack( 10, 50 ); to print 10 stack frames with each function name printing up to 50 characters.
If your binary has been completely stripped of symbol names you may not get function names, only addresses.
As it's a C++ binary, you might have to demangle the function names.
Once you get a stack trace, you can start working on what exactly is happening.
I am attempting to add a Bash shell process type to one of my environments. This bash script updates the process tables and then executes whatever is passed into it.
The Operating system and DB type match my system. Type set to "Other." The command line and parameter fields are set.
I have added the type to "Process Types run on this Server" for my server entry with the same priority as all the other, and max occurance = 1 (no other processes are running in this dev env)
I have added a process, API aware, added a component and appended additional parameters "echo test"
"Process Output Type Settings" has web set for "other"
I start the process, it appears in the process monitor as "queued" but never progresses. But if I copy the command line in "Process Request Parameters" and run it manually on the app server, it works and shows success in the process monitor. However it doesn't post the Log/Trace. Additionally there is no change if I try to make the process not Api aware. As I understand it should at least change to successful after it starts the command line process.
Why would the actual command line process not start? What causes the process to post a Log/Trace? How can I debug this? What else can I troubleshoot?
I just started using Airflow to coordinate our ETL pipeline.
I encountered the pipe error when I run a dag.
I've seen a general stackoverflow discussion here.
My case is more on the Airflow side. According to the discussion in that post, the possible root cause is:
The broken pipe error usually occurs if your request is blocked or
takes too long and after request-side timeout, it'll close the
connection and then, when the respond-side (server) tries to write to
the socket, it will throw a pipe broken error.
This might be the real cause in my case, I have a pythonoperator that will start another job outside of Airflow, and that job could be very lengthy (i.e. 10+ hours), I wonder if what is the mechanism in place in Airflow that I can leverage to prevent this error.
Can anyone help?
UPDATE1 20190303-1:
Thanks to #y2k-shubham for the SSHOperator, I am able to use it to set up a SSH connection successfully and am able to run some simple commands on the remote site (indeed the default ssh connection has to be set to localhost because the job is on the localhost) and am able to see the correct result of hostname, pwd.
However, when I attempted to run the actual job, I received same error, again, the error is from the jpipeline ob instead of the Airflow dag/task.
UPDATE2: 20190303-2
I had a successful run (airflow test) with no error, and then followed another failed run (scheduler) with same error from pipeline.
While I'd suggest you keep looking for a more graceful way of trying to achieve what you want, I'm putting up example usage as requested
First you've got to create an SSHHook. This can be done in two ways
The conventional way where you supply all requisite settings like host, user, password (if needed) etc from the client code where you are instantiating the hook. Im hereby citing an example from test_ssh_hook.py, but you must thoroughly go through SSHHook as well as its tests to understand all possible usages
ssh_hook = SSHHook(remote_host="remote_host",
port="port",
username="username",
timeout=10,
key_file="fake.file")
The Airflow way where you put all connection details inside a Connection object that can be managed from UI and only pass it's conn_id to instantiate your hook
ssh_hook = SSHHook(ssh_conn_id="my_ssh_conn_id")
Of course, if your'e relying on SSHOperator, then you can directly pass the ssh_conn_id to operator.
ssh_operator = SSHOperator(ssh_conn_id="my_ssh_conn_id")
Now if your'e planning to have a dedicated task for running a command over SSH, you can use SSHOperator. Again I'm citing an example from test_ssh_operator.py, but go through the sources for a better picture.
task = SSHOperator(task_id="test",
command="echo -n airflow",
dag=self.dag,
timeout=10,
ssh_conn_id="ssh_default")
But then you might want to run a command over SSH as a part of your bigger task. In that case, you don't want an SSHOperator, you can still use just the SSHHook. The get_conn() method of SSHHook provides you an instance of paramiko SSHClient. With this you can run a command using exec_command() call
my_command = "echo airflow"
stdin, stdout, stderr = ssh_client.exec_command(
command=my_command,
get_pty=my_command.startswith("sudo"),
timeout=10)
If you look at SSHOperator's execute() method, it is a rather complicated (but robust) piece of code trying to achieve a very simple thing. For my own usage, I had created some snippets that you might want to look at
For using SSHHook independently of SSHOperator, have a look at ssh_utils.py
For an operator that runs multiple commands over SSH (you can achieve the same thing by using bash's && operator), see MultiCmdSSHOperator
I have a few work flows where I would like R to halt the Linux machine it's running on after completion of a script. I can think of two similar ways to do this:
run R as root and then call system("halt")
run R from a root shell script (could run the R script as any user) then have the shell script run halt after the R bit completes.
Are there other easy ways of doing this?
The use case here is for scripts running on AWS where I would like the instance to stop after script completion so that I don't get charged for machine time post job run. My instance I use for data analysis is an EBS backed instance so I don't want to terminate it, simply suspend. Issuing a halt command from inside the instance is the same effect as a stop/suspend from AWS console.
I'm impressed that works. (For anyone else surprised that an instance can stop itself, see notes 1 & 2.)
You can also try "sudo halt", as you wouldn't need to run as a root user, as long as the user account running R is capable of running sudo. This is pretty common on a lot of AMIs on EC2.
Be careful about what constitutes an assumption of R quitting - believe it or not, one can crash R. It may be better to have a separate script that watches the R pid and, once that PID is no longer active, terminates the instance. Doing this command inside of R means that if R crashes, it never reaches the call to halt. If you call it from within another script, that can be dangerous, too. If you know Linux well, what you're looking for is the PID from starting R, which you can pass to another script that checks ps, say every 1 second, and then terminates the instance once the PID is no longer running.
I think a better solution is to use the EC2 API tools (see: http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ for documentation) to terminate OR stop instances. There's a difference between the two of these, and it matters if your instance is EBS backed or S3 backed. You needn't run as root in order to terminate the instance - the fact that you have the private key and certificate shows Amazon that you're the BOSS, way above the hoi polloi who merely have root access on your instance.
Because these credentials can be used for mischief, be careful about running API tools from a given server, you'll need your certificate and private key on the server. That's a bad idea in the event that you have a security problem. It would be better to message to a master server and have it shut down the instance. If you have messaging set up in any way between instances, this can do all the work for you.
Note 1: Eric Hammond reports that the halt will only suspend an EBS instance, so you still have storage fees. If you happen to start a lot of such instances, this can clutter things up. Your original question seems unclear about whether you mean to terminate or stop an instance. He has other good advice on this page
Note 2: A short thread on the EC2 developers forum gives advice for Linux & Windows users.
Note 3: EBS instances are billed for partial hours, even when restarted. (See this thread from the developer forum.) Having an auto-suspend close to the hour mark can be useful, assuming the R process isn't working, in case one might re-task that instance (i.e. to save on not restarting). Other useful tools to consider: setTimeLimit and setSessionTimeLimit, and various checkpointing tools (I have a Q that mentions a couple). Using an auto-kill is useful if one has potentially badly behaved code.
Note 4: I recently learned of the shutdown command in package fun. This is multi-platform. See this blog post for commentary, and code is here. Dangerous stuff, but it could be useful if you want to adapt to Windows. I haven't tried it, though.
Update 1. Three more ideas:
You could use .Last() and runLast = TRUE for q() and quit(), which could shut down the instance.
If using littler or a script that invokes the script via Rscript, the same command line functions could be used.
My favorite package of today, tcltk2 has a neat timer mechanism, called tclTaskSchedule() that can be used to schedule the execution of an expression. You could then go crazy with the execution of stuff just before a hourly interval has elapsed.
system("echo 'rootpassword' | sudo halt")
However, the downside is having your root password in plain text in the script.
AFAIK those ways you mentioned are the only ones. In any case the script will have to run as root to be able to shut down the machine (if you find a way to do it without root that's possibly an exploit). You ask for an easier way but system("halt") is just an additional line at the end of your script.
sudo is an option -- it allows you to run certain commands without prompting for any password. Just put something like this in /etc/sudoers
<username> ALL=(ALL) PASSWD: ALL, NOPASSWD: /sbin/halt
(of course replacing with the name of user running R) and system('sudo halt') should just work.