I am trying to connect to a Samba server in Airflow using the SambaHook class. The Samba server requires Kerberos authentication.
I have already defined a Samba connection in Airflow using the following parameters:
Host,Schema and Extra {"auth": "kerberos"}
airflow connections add "samba_repo" --conn-type "samba" --conn-host "myhost.mywork.com" --conn-schema "fld" --conn-extra '{"auth": "kerberos"}'
I'm trying to use the SambaHook class in Airflow to connect to a Samba server. When I run my code, I get the following error:
Failed to authenticate with server: SpnegoError (1): SpnegoError (16): Operation not supported or available, Context: Retrieving NTLM store without NTLM_USER_FILE set to a filepath, Context: Unable to negotiate common mechanism
However, when I use smbclient to connect to the same server using Kerberos authentication from the Docker terminal, it works fine with the command: smbclient //'myhost'/'fld' -c 'ls "\workpath\*" ' -k
What I tried: I set up a connection to the Samba server in Airflow using the SambaHook class and tried to use the listdirmethod to retrieve a list of files in a specific directory.
What I expected to happen: I expected the listdir method to successfully retrieve a list of files in the specified directory from the Samba server.
What actually resulted: Instead, I encountered the following error message:
Failed to authenticate with server: SpnegoError (1): SpnegoError (16): Operation not supported or available, Context: Retrieving NTLM store without NTLM_USER_FILE set to a filepath, Context: Unable to negotiate common mechanism
I am trying to build my solution and create docker image using OpenShift S2i pattern.
Using the below command to build the repository
oc new-app dotnet:3.1~http://azureserver.abc.com:8080/tfs/_git/Emp-WebServices --name employeewebservice
I am able to pull the repository but during the build it fails with below error log
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.111/NuGet.targets(123,5):
error : Unable to load the service index for source https://api.nuget.org/v3/index.json.
[/opt/app-root/src/Employee.WebServices/Employee.WebServices.csproj]
/opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.111/NuGet.targets(123,5):
error : The SSL connection could not be established, see inner exception.
[/opt/app-root/src/Employee.WebServices/Employee.WebServices.csproj] /opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.111/NuGet.targets(123,5):
error : Unable to read data from the transport connection: Connection reset by peer.
[/opt/app-root/src/Employee.WebServices/Employee.WebServices.csproj] /opt/rh/rh-dotnet31/root/usr/lib64/dotnet/sdk/3.1.111/NuGet.targets(123,5):
error : Connection reset by peer [/opt/app-root/src/employee.WebServices/Employee.WebServices.csproj] error: build error:
error building at STEP "RUN /tmp/scripts/assemble": error while running runtime: exit status 1
Below is the environment file that I created in the root of my repository
ASPNETCORE_ENVIRONMENT=Production
DOTNET_STARTUP_PROJECT=emp.WebServices/emp.WebServices.csproj
DOTNET_RESTORE_CONFIGFILE=Nuget/NuGet.Config
DOTNET_CONFIGURATION=Release
HTTP_PROXY=http://proxy-web.abc.com:80
HTTPS_PROXY=https://proxy-web.abc.com:80
NO_PROXY=.abc.com
DOTNET_SSL_DIRS=certificates
Have the certificates folder with .p7b file
※I am a beginner, so if you have any information you need, please comment.
Based on the yaml file, I am trying to publish a sample app that has already been launched in Pod via Ingress.
$kubectl apply -f ./overlays/stg/bookinfo-ingress.yaml
■Errormessage
Error from server (InternalError): error when creating "./overlays/stg/bookinfo-ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": x509: certificate signed by unknown authority
Composer is failing a task due to it not being able to read a log file, it's complaining about incorrect encoding.
Here's the log that appears in the UI:
*** Unable to read remote log from gs://bucket/logs/campaign_exceptions_0_0_1/merge_campaign_exceptions/2019-08-03T10:00:00+00:00/1.log
*** 'ascii' codec can't decode byte 0xc2 in position 6986: ordinal not in range(128)
*** Log file does not exist: /home/airflow/gcs/logs/campaign_exceptions_0_0_1/merge_campaign_exceptions/2019-08-03T10:00:00+00:00/1.log
*** Fetching from: http://airflow-worker-68dc66c9db-x945n:8793/log/campaign_exceptions_0_0_1/merge_campaign_exceptions/2019-08-03T10:00:00+00:00/1.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='airflow-worker-68dc66c9db-x945n', port=8793): Max retries exceeded with url: /log/campaign_exceptions_0_0_1/merge_campaign_exceptions/2019-08-03T10:00:00+00:00/1.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c9ff19d10>: Failed to establish a new connection: [Errno -2] Name or service not known',))
I try viewing the file in the google cloud console and it also throws an error:
Failed to load
Tracking Number: 8075820889980640204
But I am able to download the file via gsutil.
When I view the file, it seems to have text overriding other text.
I can't show the entire file but it looks like this:
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------
#-#{"task-id": "merge_campaign_exceptions", "execution-date": "2019-08-03T10:00:00+00:00", "workflow": "__campaign_exceptions_0_0_1"}
[2019-08-04 10:01:23,313] {models.py:1569} INFO - Executing <Task(BigQueryOperator): merge_campaign_exceptions> on 2019-08-03T10:00:00+00:00#-#{"task-id": "merge_campaign_exceptions", "execution-date": "2019-08-03T10:00:00+00:00", "workflow": "__campaign_exceptions_0_0_1"}
[2019-08-04 10:01:23,314] {base_task_runner.py:124} INFO - Running: ['bash', '-c', u'airflow run __campaign_exceptions_0_0_1 merge_campaign_exceptions 2019-08-03T10:00:00+00:00 --job_id 22767 --pool _bq_pool --raw -sd DAGS_FOLDER//-campaign-exceptions.py --cfg_path /tmp/tmpyBIVgT']#-#{"task-id": "merge_campaign_exceptions", "execution-date": "2019-08-03T10:00:00+00:00", "workflow": "__campaign_exceptions_0_0_1"}
[2019-08-04 10:01:24,658] {base_task_runner.py:107} INFO - Job 22767: Subtask merge_campaign_exceptions [2019-08-04 10:01:24,658] {settings.py:176} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800#-#{"task-id": "merge_campaign_exceptions", "execution-date": "2019-08-03T10:00:00+00:00", "workflow": "__campaign_exceptions_0_0_1"}
Where the #-#{} pieces seems to be "on top of" the typical log.
I faced the same problem. In my case the problem was that I removed the google_gcloud_default connection that was being used to retrieve the logs.
Check the configuration and look for the connection name.
[core]
remote_log_conn_id = google_cloud_default
Then check the credentials used for that connection name has the right permissions to access the GCS bucket.
I'm having a similar problem with viewing logs in GCP Cloud Composer. It doesn't appear to be preventing the failing DAG task from running though. What it looks like is a permissions error between the GKE and Storage Bucket where the log files are kept.
You can still view the logs by going into your cluster's storage bucket in the same directory as your /dags folder where you should also see a logs/ folder.
Your helm chart should setup global env:
- name: AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT
value: "google-cloud-platform://"
Then, you should deploy a Dockerfile with root account only (not airflow account), additionaly, you set up your helm uid, gid as:
uid: 50000 #airflow user
gid: 50000 #airflow group
Then upgrade helm chart with new config
*** Unable to read remote log from gs://bucket
1)Found the solution after assigning the roles to the service account
2)The SA key(json or txt) to be added and configured to the connection in the
remote_log_conn_id = google_cloud_default
3)restart the scheduler and webserver of the airflow
4)restart the dags on the airflow
you can find the logs on the GCS bucket where its configured
I am attempting to use after_success in a Travis CI build to deploy files to a remote server using SFTP. However, I am getting errors that prevent the upload from succeeding.
SFTP command and resulting error message:
$ sftp -b upload_sftp -i upload_key -P 2222 $sftp_user
Host key verification failed.
Couldn't read packet: Connection reset by peer
The SFTP batch file upload_sftp contains various put commands.
As the "Host key verification failed" message hints, you need to add your server's keys to the known_hosts file, as documented in the Travis CI Documentation.
Adding the following to .travis.yml uses ssh-keyscan:
addons:
ssh_known_hosts: git.example.com
Alternately, known_hosts can be appended to directly using
install:
- echo 'KEY' >> $HOME/.ssh/known_hosts