I'm looking to get some stats on the parameter retrieval from the HTTP Logs in production. We already get them when we normally execute the reports via the Report Manager or via a url. But when running the report via a schedule (we have them set up in production for the warm ups) nothing is logging....
We also tried a report via schedule in our test env and got no http entries.. :(
Scheduled reports run as SQL Server Agents jobs, so I wouldn't be at all surprised if they don't hit ReportServer, hence no HTTP logs.
Have you looked into querying the execution log? It's got quite detailed information, and will contain entries for every report execution.
Related
I would like to setup a scheduled task to run every hour and say send emails. In a Firebase project, I've setup a function with .schedule('every 60 minutes') and can see it successfully in the GCP portal as well. Recently, this error started to show up in the logs: "The request was aborted because there was no available instance. Additional troubleshooting documentation can be found at: https://cloud.google.com/functions/docs/troubleshooting#scalability" tied to this scheduled function.
There are plenty of conversations online that I saw tied to "no available instance" and mainly due to traffic spikes etc.. But for a scheduled task, what can I do to avoid this? My main goal is to not have to worry if I have 3 tasks to run the next hour or 300. I want to auto-scale and do the magic that it does. Am I missing something here?
Thanks
I see that airflow logs are stored at
base_log_folder/dag_id/task_id/date_time/1.log
i.e:
base_log_folder/dag_id={dag_id}/run_id={run_id}/task_id={task_id}/attempt={try_number}.log
Sometime my logs are huge and know its now a good idea to check them from the web ui, because the chrome cant handle so much size of logs.
I have access to the server and can check the logs.
So how can i break the longs into smaller files - v
i.e
{try_number}_1.log
{try_number}_2.log
{try_number}_3.log
...
Also noted that the log file {trynumber}.log, is only created when the task is completed.
while the task is running i can check the logs in the webui, but i dont see any file in the corresponding log folder.
So i need two things for logging from the server side:
break large log files into smaller files
see the log file live while the task is running, not only after the task is completed
In Airflow 2.4.0 there is an option to view full logs or only the first fragment thus huge logs are not loaded automatically:
Starting Airflow 2.5.0 the web UI also does auto tails for logs (PR)
Airflow does show live logs. If you will set for example a Sensor task that pokes resource you will see the poking attempts in the log when the task is running. It's important to note that there are local logs and remote logs (docs):
In the Airflow UI, remote logs take precedence over local logs when remote logging is enabled. If remote logs can not be found or accessed, local logs will be displayed. Note that logs are only sent to remote storage once a task is complete (including failure). In other words, remote logs for running tasks are unavailable (but local logs are available).
Huge logs are often a sign of not using log levels. If you have entries relevant for debugging then set DEBUG mode rather than INFO mode that way you can better control over the log size displayed in the UI using the AIRFLOW__LOGGING__LOGGING_LEVEL variable.
I am managing a Google Cloud Composer environment which runs Airflow for a data engineering team. I have recently been asked to troubleshoot one of the dags they run which is failing with this error : [12:41:18,119] {credentials_utils.py:23} WARNING - [redacted-name] connection ID not available, falling back to Google default credentials
The job is basically a data pipeline which reads from various sources and stores data into GBQ. The odd part is that they have a strictly similar Dag running for a different project and it works perfectly.
I have recreated the .json credentials for the service account behind the connection as well as the connection itself in Airflow. I have sanitized the code to see if there was any hidden spaces or so.
My knowledge of Airflow is limited and I have not been able to find any similar issue in my research, any one have encountered this before?
So the DE team came back to me saying it was actually a deployment issue where an internal module involved in service account authentication was being utilized inside another DAG running in stage environment, rendering it impossible to proceed to credential fetch from the connection ID.
I'm using the CyberSource API to generate and download reports using the Postman collection.
The issue is when sending a "GET" request to download a report the following error coming back:
<errorBean>
<code>RESOURCE_NOTFOUND</code>
<detail>The requested resource is not found. Please try again later.</detail>
<localizationKey>cybsapi.resource.notfound</localizationKey>
<message>Requested Resource Not Found</message>
</errorBean>
Error when sending a request to download a report:
Success response when asking for the available reports on site:
I used the same environment variables for both requests.
Thanks,
Udi
I experienced a similar issue and had to log in to Cybersource EBC to configure the report subscriptions which were not carried over from the legacy system. It's likely that the error is caused because there is no report for the date you are requesting.
My understanding is that once you have configured the report subscriptions, Cybersource will start to create the report files as per the subscription - usually once daily. So you might only be able to request the report later in the day, or the following day.
I'm refactoring a very old reports generating function in an ORACLE web application. It used JavaScript to construct a URL, sending to the reports server to run a reports. What I want to do is processing it in the database, in PL/SQL procedures(invoked through mod_pl/SQL). I tried to use utl_http.begin_request to do that, but sometimes when the output file is large(PDF format, about 20 pages, 1.5M), I received an error:
ORA-29259: end-of-input reached.
The test codes for sending requests are quite simple:
--------upgraded 2013/08/27----------------------------------------------------------
UTL_HTTP.set_transfer_timeout(1000);
--some params setting....
myIdent := SRW.RUN_REPORT(myPlist); --here raise the exception(ORA-29273:request_failed; ORA-29259: end-of-input reached.) and procedure stoped.
r_stat := SRW.report_status(myIdent,myPlist);
#ThinkJet, Thanks for your help.
I logon the report server, found that the report still running after I got this exception in my program, and finally finished succefully.I tested for many times, and found everytime I got the exception just 5 minutes after I started request,no matter what kind of reports I was running and what size it was (sure, all big size,running for over 5mins).I'm wondering if it's something about configuration on oracle application server?
Does anyone have idea about this? Many Thanks.
ORA-29259 while SRW.RUN_REPORT
is ias 10g , this is cause by httpd.conf timeout parameter configuration
just
1.edit $ORACLE_HOME /Apache/Apache/conf/httpd.conf
change "Timeout" great than your report running time
2.restart http server and test it.
opmnctl restartproc ias-component=HTTP_Server
It take a time on Oracle Reports side to produce a big reports so a timeouts can occur sometime. Try to increase timeout with utl_http.set_ransfer_timeout procedure before originate a request and look if it helps.