I have installed apache airflow locally on windows system on ubuntu WSL. I have created a tag consisting of two task. First task is fetching the data from data source and storing in data frame. Second task is sending data to database from data frame. My first task got executed and pulled the data to second task using xcom. Even giving correct connection credentials I am getting this error("Error while connecting to mysql"). I have used hooks too but still the same error. I am not getting what other configurations i have to do and why am I getting this error? Please suggest something.
Related
Well, for sure after trying my script sometimes, I forgot that the Airflow has a limit to load data and now I can't access the XComs to delete my data, is there a way to do that using the Ubuntu because I'm running my Airflow on LocalExecutor?
Here is the SS from the error
I am managing a Google Cloud Composer environment which runs Airflow for a data engineering team. I have recently been asked to troubleshoot one of the dags they run which is failing with this error : [12:41:18,119] {credentials_utils.py:23} WARNING - [redacted-name] connection ID not available, falling back to Google default credentials
The job is basically a data pipeline which reads from various sources and stores data into GBQ. The odd part is that they have a strictly similar Dag running for a different project and it works perfectly.
I have recreated the .json credentials for the service account behind the connection as well as the connection itself in Airflow. I have sanitized the code to see if there was any hidden spaces or so.
My knowledge of Airflow is limited and I have not been able to find any similar issue in my research, any one have encountered this before?
So the DE team came back to me saying it was actually a deployment issue where an internal module involved in service account authentication was being utilized inside another DAG running in stage environment, rendering it impossible to proceed to credential fetch from the connection ID.
I am setting up Integration Broker messaging from HCM 9.2 to FSCM 9.2 using the PERSON_BASIC_FULLSYNC service operation (the delivered process) to sync data from HCM to FSCM. I have activated the service operation, handler, queue, and routing on both sides, however when I run the Full Data Publish process, it runs to No Success with the following error:
Fetching array element 0: index is not in range 1 to 3.
(180,252) EOL_PUBLISH.PUBDTL.GBL.default.190 0-01-01.Step05.OnExecute PCPC:16088 Statement:266
I had initially run this process, and it ran to success, however it did not publish any new data in PS_PERSONAL_DATA in FSCM, so I updated the service operation version in HCM from 'INTERNAL' to 'VERSION_1', as the corresponding service operation in FSCM only had the 'VERSION_1' version available. But after I change the version so they match, and run the process it goes to No Success.
If I set the version of the service operation in HCM back to 'INTERNAL' and run the process, then it is successful but no data gets published in PS_PERSONAL_DATA. Any thoughts on what I should look at?
Sounds like a service op. routing problem. Confirm the routing directions and ensure that any alias' that are set don't cause issues. Service Ops on each side need to be the same.
We have been seeing the following 'warnings' in the event log of our BizTalk
machine since upgrading to BTS 2006. They seem to occur
randomly 6 or 8 times per day.
Does anyone know what this means and what needs to be done to clear it up?
we have only one BizTalk server which is running on only one machine.
I am new to BizTalk, so I am unable to find how many tracking host instances running for BizTalk server. Also, can you please let me know that we can configure only one instance for one server/machine?
Source: BAM EventBus Service
Event: 5
Warning Details:
Execute batch error. Exception information: TDDS failed to batch execution
of streams. SQLServer: bizprod, Database: BizTalkDTADb.Cannot insert
duplicate key row in object 'dta_MessageFieldValues' with unique index
'IX_MessageFieldValues'.
The statement has been terminated..
I see you got a partial answer in your MSDN Post
go to BizTalk Admin Console ,check in Platform Settings -> Hosts, in the list of hosts on the right, confirm that only a single Host has the Tracking column marked as Yes.
As to your other question. Yes you can run a Single Host Instance on a Single Server. Although when your server starts to come under a bit of load you may want to consider setting up some more so you can balance the workload better.
I cannot seem to figure out this issue. I have a qlikview document that pulls in a bunch of data and aggregates/joins it up. Typical qlikview stuff. At the end of my process I have an oracle stored procedure call. I am not retrieving anything back. This is a simple call to a database to trigger a process. I have setup my ODBC connection and User DSN on my local machine for the connection. When I run my qvw file on my local machine everything works just fine. The proc call is made and the script executes without any errors.
However, when I put the document on our reload server and after I setup a reload task for it the process throws a general script error when the sql proc is called. What could cause this? The user running the document has execute permissions. Do I need to setup a DSN on the reload server?
Really not sure at all here. Hopefully someone here can help me out. Thanks.
Unfortunately QlikView's SQL error messages are not that helpful for debugging purposes. In this case you can try turning on ODBC logging (http://support2.microsoft.com/kb/274551) and then reload the script to try and capture the cause of the error.
Finally, if your script refers to a "local" DSN then this also needs to be present on the machine that will perform the reload, in this case the QlikView server.