I've seen the answers to nginx-how-to-run-a-shell-script-on-every-request, but my request is a bit different:
Whenever I have a problem with the back end (Mendix/Java specifically, but Python/django/plone also involved) which I'm proxying to, I would like to execute a script, but NOT return the results to the browser, ie. it is done asynchronously to the request, and perhaps issue a retry/reload/redirect page, but the important part for me is the script to be run whenever there is a gateway error.
The "need" is to either trigger a restart of the back-end, or issue an error/warning to the sysadmins.
Related
When I load my docker shiny app domain name in the browser, it crashes (greys out) and I get this "ERROR: [_parse_http_data] invalid HTTP method".
I have developed an web application that consists of a shiny app (has a login feature connected to an RMySQL database), a website and a mariadb database. I put them together in a docker-compose file and tested it on my local computer and it works fine. I then proceeded to deploy them in a Kubernetes cluster in GCE and that was also successful. I used cloudflare to install a ssl certificate for the shiny app domain (i.e. trnddaapp.com). Now when I load the shiny app domain in the browser it appends the https and loads the app successfully but after about a minute it crashes (greys out). I loaded the shiny app external ip with http and this doesn’t crash.
The closest solution I have come to is https://github.com/rstudio/shiny-server/issues/392 but there doesn't seem to be any other solution to my problem. I would be grateful if anyone help me resolve this problem.
This is the error message I get when I check with kubectl log [app pod name], I get this error:
ERROR: [_parse_http_data] invalid HTTP method
ERROR: [_parse_http_data] invalid HTTP method
ERROR: [_parse_http_data] invalid HTTP method
I expect the app not to crash when the shiny app domain (trnddaapp.com) is appended with the https.
Let's start with the analysis of the error message, it says:
[_parse_http_data]
So we know that your app is receiving something, but it doesn't understand what it is (it may be a malformed HTTP/1.0 or HTTP/1.1 or even binary data) then we have an
invalid HTTP method
Now we are sure it is not a HTTP/1.X call but a stream of (non recognized) data.
We now know is not the instance since it "deploys" and "delivers" the service, but something inside that is just breaking.
There are a few things that may be happening, since it runs in your local machine (where I am assuming it has access to more resources, especially memory) it may be an issue of resource allocation and that once ran in a container, it could be possible that it empties its allocated amount of resources and breaks (perhaps a library that is called in real time that uses a chunk of memory?) but we won't be sure unless we can debug it inside a container, so could it be possible for you to add a debug library that records your requests to see if it parses all of those and at some point in time it stops and why? I know a person from R-Studio created a httpuv that logs every request this can be done as in:
devtools::install_github('rstudio/httpuv#wch-print-req')
And after that, maybe share the output and see why the application is behaving like that and killing its own service.
I really thank you in advance, hopefully with those logs we may be able to shed more light into this matter.
Thanks once again!
-JP
I just started using Airflow to coordinate our ETL pipeline.
I encountered the pipe error when I run a dag.
I've seen a general stackoverflow discussion here.
My case is more on the Airflow side. According to the discussion in that post, the possible root cause is:
The broken pipe error usually occurs if your request is blocked or
takes too long and after request-side timeout, it'll close the
connection and then, when the respond-side (server) tries to write to
the socket, it will throw a pipe broken error.
This might be the real cause in my case, I have a pythonoperator that will start another job outside of Airflow, and that job could be very lengthy (i.e. 10+ hours), I wonder if what is the mechanism in place in Airflow that I can leverage to prevent this error.
Can anyone help?
UPDATE1 20190303-1:
Thanks to #y2k-shubham for the SSHOperator, I am able to use it to set up a SSH connection successfully and am able to run some simple commands on the remote site (indeed the default ssh connection has to be set to localhost because the job is on the localhost) and am able to see the correct result of hostname, pwd.
However, when I attempted to run the actual job, I received same error, again, the error is from the jpipeline ob instead of the Airflow dag/task.
UPDATE2: 20190303-2
I had a successful run (airflow test) with no error, and then followed another failed run (scheduler) with same error from pipeline.
While I'd suggest you keep looking for a more graceful way of trying to achieve what you want, I'm putting up example usage as requested
First you've got to create an SSHHook. This can be done in two ways
The conventional way where you supply all requisite settings like host, user, password (if needed) etc from the client code where you are instantiating the hook. Im hereby citing an example from test_ssh_hook.py, but you must thoroughly go through SSHHook as well as its tests to understand all possible usages
ssh_hook = SSHHook(remote_host="remote_host",
port="port",
username="username",
timeout=10,
key_file="fake.file")
The Airflow way where you put all connection details inside a Connection object that can be managed from UI and only pass it's conn_id to instantiate your hook
ssh_hook = SSHHook(ssh_conn_id="my_ssh_conn_id")
Of course, if your'e relying on SSHOperator, then you can directly pass the ssh_conn_id to operator.
ssh_operator = SSHOperator(ssh_conn_id="my_ssh_conn_id")
Now if your'e planning to have a dedicated task for running a command over SSH, you can use SSHOperator. Again I'm citing an example from test_ssh_operator.py, but go through the sources for a better picture.
task = SSHOperator(task_id="test",
command="echo -n airflow",
dag=self.dag,
timeout=10,
ssh_conn_id="ssh_default")
But then you might want to run a command over SSH as a part of your bigger task. In that case, you don't want an SSHOperator, you can still use just the SSHHook. The get_conn() method of SSHHook provides you an instance of paramiko SSHClient. With this you can run a command using exec_command() call
my_command = "echo airflow"
stdin, stdout, stderr = ssh_client.exec_command(
command=my_command,
get_pty=my_command.startswith("sudo"),
timeout=10)
If you look at SSHOperator's execute() method, it is a rather complicated (but robust) piece of code trying to achieve a very simple thing. For my own usage, I had created some snippets that you might want to look at
For using SSHHook independently of SSHOperator, have a look at ssh_utils.py
For an operator that runs multiple commands over SSH (you can achieve the same thing by using bash's && operator), see MultiCmdSSHOperator
My problematic seem to be simple, but I haven't find yet a way to solve it...
I have a legacy system which is working and a new system which will replace it. This is only rest webservices call, so I'm using simple bridge endpoint on http service.
To ensure the iso-functional run, I want to put them behind a camel route dispatching message to both system but returning only the response of the legacy one and log the response of both system to be sure there are running in same way...
I create this route :
from("servlet:proxy?matchOnUriPrefix=true")
.streamCaching()
.setHeader("CamelHttpMethod", header("CamelHttpMethod"))
.to("log:com.mylog?showAll=true&multiline=true&showStreams=true")
.multicast()
.to(urlServer1 + "?bridgeEndpoint=true")
.to(urlServer2 + "?bridgeEndpoint=true")
.to("log:com.mylog?showAll=true&multiline=true&showStreams=true")
;
It works to call each services and to log messages, but response are in a mess...
If the first server doesn't respond, the second is not call, if the second respond an error, only that error is send back to client...
Any Idea ?
You can check for some more details in multicast docs http://camel.apache.org/multicast.html
Default behaviour of multicast (your case) is:
parallelProcessing is false so routes are called one by one
To correctly implement your case you need probably:
add error handling for each external service call so exception will not stop correct processing
configure or implement some aggregator strategy and put it to the strategyRef so you can combine results from all calls to the single multicast result
Meteor is said to automagically (in most cases) figure out what code to run on the client and what code to run on the server so you could theoretically just write all your code in one .js file.
I would like to be able to write code in my browser console and have it executed pretty much as if I had put the code in a file on my server.
For example, in my browser console:
[20:08:19.397] Pages = new Meteor.Collection("pages");
[20:08:30.612] Pages.insert({name:"bro"});
[20:08:30.614] "sGmRrQfezZMXuPfW8"
[20:08:30.618] insert failed: Method not found
Meteor says "method not found" because I need to do new Meteor.Collection("pages"); on the server.
But is there a workaround for this, whether using the above-mentioned automagic or by explicitly saying in my browser console "run the following line of code on the server!"?
Well it doesn't "automagically" figure it out - you have to very explicitly do one of two things:
Separate the code into client and server directories.
Wrap the code in an isClient or an isServer section.
Otherwise, any code you write will execute in both environments. However, any code input by the user on the client will only be executed on the client. Meteor has been specifically designed to protect this boundary.
You can call a method on the server from the client, but again the server cannot be tricked into executing client-defined functions.
In your specific example, you can always define the collection only on the client like so:
Pages = new Meteor.Collection(null);
That will allow you do freely manipulate the collection data on the client, but it will not involve the server (nothing will be stored in the db).
I'll try provide as much information as possible:
No error message.
The instance stays in the "ready service instances".
The receive location has the same parameters (except URI, the three polling queries, user account/pw and receive pipeline) as another receive location that points to another database/table which works.
The pipeline is waiting for the correct schema.
The port surface and receive location are both waiting for the correct schema.
In my test example, there are only 10 lines being returned.
The message, which contains those 10 lines, validates against the schema.
I tried to let the instance alone to no avail - 30+ minutes - and no change in its condition.
I had also tried suspending and then resuming it which then places the instance in the "dehydrated orchestrations" list. Again, with no error message.
I'm able to get the message by looking at the body of the message that's in the "ready to run" service. (This is the message that validates versus the schema I use in Visual Studio.)
How might something like this arise?
Stupid question, but I have to ask... Is the corresponding host instance running?