I have a problem with Apache Airflow Logging. My Airflow cluster has one Webserver, one Scheduler and five Workers Celery. All Workers working usually but only four Workers can fetch the log, one worker cant fetch the record with error: "Failed to fetch log file from the worker. Client error '404 NOT FOUND' for URL". I checked /etc/hosts file, this is ok and have hostname. Please help me. Thank you very much <3 <3
Related
I am trying to deploy a Nodejs & WordPress website together on an ubuntu server and I am using apache2 for my server config. Everything works fine initially. But after some time my website start showing connection timeout in the browser and when I checked the Ubuntu machine then I found the following:
I checked the error log and increased the worker limit but then got this error again.
I also checked the tasks using htop command then I found this and as I am increasing the worker request number in the apache2 config file the tasks increase automatically.
I also checked which IP calling so many requests and then I found this:
Access logs file :
I have no idea what is going on here and don't know how to fix it. If anyonw know the solution to this problem then please reply.
Thanks
we operate CENM(1.2 and use helm template to run on k8s cluster) to construct our own private network and keep on running CENM network map server for a few week, then launching new node start failing.
with further investigation, its appeared that request timeout for http://nmap:10000/network-map causes problem.
in nmap server’s log, we found following output when access to above url with curl.
[NMServer] - Error while handling socket client message com.r3.enm.servicesapi.networkmap.handlers.LatestUnsignedNetworkParametersRetrievalMessage#760c53ea: HikariPool-1 - Connection is not available, request timed out after 30000ms.
netstat shows there is at least 3 establish connection to the database from the container which network map server runs, also I can connect database directly with using CLI.
so I don’t think it is neither database saturated nor network configuration problem.
anyone have an idea why this happens? I think restart probably solve the problem, but want to know the root cause...
regards,
Please test the following options.
Since it is the HikariCP (connection pool) component that is throwing the error it would be worth seeing if increasing the pool size in the network map configuration may help - see below)
Corda uses Hikari Pool for creating the connection pool. To configure the connection pool any custom properties can be set in the dataSourceProperties section.
dataSourceProperties = {
dataSourceClassName = "org.postgresql.ds.PGSimpleDataSource"
...
maximumPoolSize = 10
connectionTimeout = 50000
}
Has a healthcheck been conducted to verify there are sufficient resources on that postgres database i.e basic diagnostic checks ?
Another option to get more information logged from the network map service is to run with TRACE logging also:
From https://docs.corda.net/docs/cenm/1.2/troubleshooting-common-issues.html
Enabling debug/trace logging
Each service can be configured to run with a deeper log level via command line flags passed at startup:
java -DdefaultLogLevel=TRACE -DconsoleLogLevel=TRACE -jar <enm-service-jar>.jar --config-fi
I am currently setup airflow scheduler in Linux server A and airflow web server in Linux server B. Both server has no Internet access. I have start the initDB in server A and keep all the dags in server A.
However, when i refresh the webserver UI, it keep having error message:-
This DAG isn't available in the webserver DagBag object
How do i configure the dag folder for web server (server B) to read the dag from scheduler (server A)?
I am using bashoperator. Is that Celery Operator is a must?
Thanks in advance
The scheduler has found your dags_folder, and its processes, and is scheduling them accordingly. The webserver however can "see" these processes solely by their existence in the database but can't find them in its dags_folder path.
You need to ensure that the dags_folder for both servers contain the same files, and that both are kept in sync with one another. This is out of scope for Airflow and it won't handle this on your behalf.
I am deploying AWS Corda Enterprise Template. The Quick start deployed the stack as per the defined CloudFormation template. I can see 2 AWS instances, up and running as Corda nodes, in Hot-Cold setup with a load balancer.
However the Log for Corda node has following ERROR related to AMQP communication.
[ERROR] 2018-10-18T05:47:55,743Z [Thread-3
(ActiveMQ-scheduled-threads)] core.server.lambda$channelActive$0 -
AMQ224088: Timeout (10 seconds) while handshaking has occurred. {}
What can be possible reason for this error? This error keeps on occurring after a certain time interval. So it looks like some connectivity issue to me.
Note: The load balancer shows the status of this AWS Corda instances as healty (In Service). So I believe the Corda node has booted up successfully.
The ERROR message isn't necessarily tied to AMQP. Perhaps you were confused by the "AMQ" in the error ID (AMQ224088)?
In any event, this error indicates that something on the network is connecting to the ActiveMQ Artemis broker, but it's not completing any protocol handshake. This is commonly seen with, for example, load balancers that do a health check by creating a socket connection without sending any real data just to see if the port is open on the target machine.
No way to have console wired with Heroku lately:
lsoave#ubuntu:~/rails/github/gitwatcher$ heroku run console
Running console attached to terminal...
Timeout awaiting process
lsoave#ubuntu:~/rails/github/gitwatcher$
Does it happen to anyone else ?
I had the same problem on the ISP FastWeb, which is an ISP here in Italy. Based on my Googling, it appears that they block port 5000.
To get around it for now, I am running the following:
heroku run:detached rake db:migrate
That tells it to run without waiting to connect to my machine on port 5000.
Then, you can examine the logs for your process you just kicked off. (The heroku command will tell you this after you run it):
heroku logs -p run.1
That tells it to output the logs for the running process.
But I found that the logs command was exiting even though my process was still running.
To get around that, you can add "-t" if you want to "tail" the logs:
heroku logs -p run.1 -t
You'll know your process is done when you see something like this in the logs:
2012-10-14T15:36:41+00:00 heroku[run.1]: Process exited with status 0
2012-10-14T15:36:41+00:00 heroku[run.1]: State changed from up to complete
I'm sure someone could whip up a script that would
Run the given heroku command with run:detached
Examine the output to determine what its process name was (run.1, run.2, etc)
run heroku logs -p run.1 -t and show me the output
exit the logs process when it sees something like State changed from up to complete
From: http://devcenter.heroku.com/articles/oneoff-admin-ps
The heroku run process opens a connection to Heroku on port 5000. If
your local network or ISP is blocking port 5000, or you are
experiencing a connectivity issue, you will see an error similar to:
$ heroku run rails console
Running rails console attached to terminal...
Timeout awaiting process
You can test your connection to Heroku by trying to connect directly
to port 5000 by using telnet to rendezvous.heroku.com. A successful
session will look like this:
$ telnet rendezvous.heroku.com 5000
Trying 50.19.103.36...
Connected to ec2-50-19-103-36.compute-1.amazonaws.com.
Escape character is '^]'.
If you do not get this output, your computer is being blocked from
accessing our services. We recommend contacting your IT department,
ISP, or firewall manufacturer to move forward with this issue.
There's a way to bypass this restriction by executing commands you were going to execute with 'heroku run ...' directly as system calls.
For this heroku command:
heroku run rake db:migrate
One could make a call in Ruby
system("rake db:migrate")
This call could be done as a result on some HTTP GET request - you better protect by some means, so that it wouldn't be accessible for strangers.
It's not a nice way - but when you don't have a choice - it works.
I had the same problem on the ISP FastWeb, which is an ISP here in Italy.
The most simple way to overcome the problem is set a VPN when you are using heroku run (something)
Personally I use Tunnel Bear that offer you 0.5GB of internet traffic for free
I've just connected successfully to a console on Heroku
Is it all apps or just a single one? I'd suggest checking your heroku gem is up to date, that you can run other commands like heroku ps, restart your application via heroku restart - if you've tried these then get in touch with Heroku support as something is wrong with your application.