Starting from V3 the NodeDriver builds a standard output & error logfile for each node instead of printing all logs in the console. Is there still a possibility to have the logging in the console too? And are there any other possibilities regarding NodeDriver logging?
Running the NodeDriver with startNodesInProcess = true runs all the nodes in the same process, but also logs every Node output in the console window again.
Related
I'm running a DAG in Google Cloud Composer (hosted Airflow) which runs fine in Airflow locally. All it does is print "Hello World". However, when I run it through Cloud Composer I receive the error:
*** Log file does not exist: /home/airflow/gcs/logs/matts_custom_dag/main_test/2020-04-20T23:46:53.652833+00:00/2.log
*** Fetching from: http://airflow-worker-d775d7cdd-tmzj9:8793/log/matts_custom_dag/main_test/2020-04-20T23:46:53.652833+00:00/2.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='airflow-worker-d775d7cdd-tmzj9', port=8793): Max retries exceeded with url: /log/matts_custom_dag/main_test/2020-04-20T23:46:53.652833+00:00/2.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8825920160>: Failed to establish a new connection: [Errno -2] Name or service not known',))
I've also tried making the DAG add data into a database and it actually succeeds 50% of the time. However, it always returns this error message (and no other print statements or logs). Any help much appreciated on why this might be happening.
We also faced the same issue then raised a support ticket to GCP and got the following reply.
The message is related to the latency of syncing logs from Airflow workers to WebServer, it takes at least some minutes (depending on the number of objects and their size)
The total log size seems not large but it’s enough to noticeably slow down synchronization, hence, we recommend cleanup/archive the logs
Basically we recommend relying on Stackdriver logs instead, because of latency due to the design of this sync
I hope this will help you solve the problem.
I have the same problem after upgrading from 1.10.3 to 1.10.6 of Google Composer.
I can see in my logs that airflow is trying to get the logs from a bucket with a name ended with -tenant while the bucket in my account ends with -bucket
In the configuration, I can see something weird too.
## airflow.cfg
[core]
remote_base_log_folder = gs://us-east1-dada-airflow-xxxxx-bucket/logs
## also in the running configuration says
core remote_base_log_folder gs://us-east1-dada-airflow-xxxxx-tenant/logs env var
I wrote to google support and they said the team is working on a fix.
EDIT:
I've been accessing my logs with gsutil and replacing the bucket name suffix to -bucket
gsutil cat gs://us-east1-dada-airflow-xxxxx-bucket/logs/...../5.logs
I faced the same situation in multiple occasions.
As soon as when the job finished when I take a look at the log on Airflow Web UI, it used to give me the same error. Although when I check back the same logs on UI after a min or 2, I could see the logs properly.
As per the above answers, its a sync issue between the webserver and the Worker node.
In general, the issue describe here should be more like a sporadic issue.
In certain situations, what could help is setting default-task-retries to a value that allows for retrying a task at least 1.
This issue is resolved at least since Airflow version: 1.10.10+composer.
We have Websphere Message Broker v8.0.0.3 on Redhat Linux env, and we encounterd the following runtime error message:
BIP2057
Execution Group <insert_1> could not be started: broker name <insert_2>; UUID <insert_3>; label <insert_4>; Pub-Sub server <insert_5> (1=Yes, 0=No).
The situation is that one of the execution groups stopped (without any trace/log about it) and the wmb could not auto start with the BIP2057 error.
the weird part occurs when another EG restarted (unlinked reason to the first problem, because of some applicative errors) and then the first EG "took its place", started successfully, and now the second EG could not start for the same reason.
We have checked for user permissions or the logs without any success in identifying the problem.
Any help would be much appreciated.
Run the following service IBM MQSeries if it's not running already:
C:\Program Files (x86)\IBM\WebSphere MQ\bin\amqlsvc.exe
If it gives an
exception of Access Denied, your user must not be having rights to
log on.
Another possible resolution can be by restarting the
DataFlowEngine.exe process.
If none of the above scenarios fulfill the need, restart the system.
I am finding when working with larger datasets that the kernel may die, something I also experiance on my local machine. Sometimes it comes back and sometimes not. So even the Tree panel won't react to terminate a errant Kernel. EG "restart" does not work and the server itself seems to die. So the tree view won't respond or refresh. On my local machine I just kill the terminal instance and start over.
What is the "proper" way to restart everything?
FWIW the instance seems pegged at 150% cpu utilization atm
Related: is there any way to allow long running stuff to work?
I am trying to use a report generator (pandas-profiling) on a 2mm record dataset.. Works on my local..
found it here: https://cloud.google.com/datalab/getting-started
FWIW These commands can be used in the new command line shell on the Cloud console page.see https://cloud.google.com/shell/docs/ .. Without the sdk on your machine.. You need to modify the commands slightly since you will be logged into your project already,
Stopping/starting VM instances
You may want to stop a Cloud Datalab managed VM instance to avoid incurring ongoing charges. To stop a Cloud Datalab managed machine instance, go to a command prompt, and run:
$ gcloud auth login
$ gcloud config set project <YOUR PROJECT ID>
$ gcloud preview app versions stop main
After confirming that you want to continue, wait for the command to complete, and make sure that the output indicates that the version has stopped. If you used a non-default instance name when deploying, please use that name instead of "main" in the stop command, above (and in the start command, below).
For restarting a stopped instance, run:
$ gcloud auth login
$ gcloud config set project <YOUR PROJECT ID>
$ gcloud preview app versions start main
I have a Loadrunner Test Scenario, here is the snapshot for it:
after opening my Test Scenario with Loadrunner Controller, I click then the "Start Scenario" button, the Scenario must run for 2 Hours, but it stops after 1 minute, and get the following Error:
Failed to stop Service Virtualization.
Failed to start Service Virtualization.
here you can see the error snapshot:
to increase the size of the snapshot please: Ctrl++
It seems that you have unwittingly activated an integration with HP Service Virtualization (or SV for short), without having SV installed on the same machine. In order to remove it, open the SV configuration dialog and uncheck all the entries.
I solved the problem with a lot of effort.
the problem is, when you have test scenario, which worked before, and you try to remove some script (groups) of the scenario,after that the test scenario will not work, what I've done, is I created from scratch a new scenario with the same test scripts, and voila it worked, I hope everyone will pay attention at that in the future
All, According the WAS.The document says the worker process is managed by WAS .
But I found when typing w3wp /? There is a debug flag.
-debug
This option launches a worker process using the default
application host config file. By default, it will use
site id 1.
What does mean start worker process in debug mode? In what case we want to start a work process with debug option ? thanks.
Added
I didn't known why I got a exception when run w3wp.exe -debug.
ERROR: There has been an error during processing of this command.
Please check the event log and see if any errors or warnings have been
logged.
When I checked the log. It looks like:
The World Wide Web Publishing Service failed to set the application
pool for the application '/xxxx' in site '1'. The data field contains
the error number.
According to http://support.microsoft.com/kb/183480 it switches the security context of the running user which is not its normal mode of operation - I suppose this would make it easier to then attach a debugger and other utilities.