I know we can use airflow bigquery_to_gcs operator to copy a bigquery table to GCS but that will copy the column names as well. I have a use case that should have only the column values but not the names. Is it possible?
fruit | any_value |
pear | pear |
apple | pear |
banana | apple |
I want my csv file to be something like this
pear,pear
apple,pear
apple,pear
Whereas the output if I use bigquery_to_gcs is
fruit,any_value
pear,pear
apple,pear
apple,pear
You should be able to set print_header=False on the task instance.
See details here http://airflow.apache.org/docs/stable/_modules/airflow/contrib/operators/bigquery_to_gcs.html
Related
When reading 20+ columns horizontally, it gets hard on the eyes. I've been trying to find a way to display output of a single search result row as a scrollable vertical columnName:columnValue list:
+-------------------------+-+
| First_Name | John | |
| Last_Name | Doe | |
| Birthdate | 01011970 | |
| Last_login | 01012015 |X|
+-------------------------+-+
Is this possible?
This is functionality you can push to the database. For example if your DB is oracle 11g, you can use the unpivot feature.
Create a multicolumn list box with the below query.
select column_name,column_values from customers
unpivot
(column_values for column_name in ("FIRST_NAME" ,"LAST_NAME"))
I am getting data from the server in a file (in format1) everyday ,however i am getting the data for last one week.
I have to archive the data for 1.5 months exactly,because this data is being picked to make some graphical representation.
I have tried to merge the the files of 2 days and sort them uniquely (code1) ,however it didn't work because everyday name of raw file is changing.However Time-stamp is unique in this file,but I am not sure how to sort the unique data on base of a specific column also,is there any way to delete the data older than 1.5 months.
For Deletion ,The logic i thought is deleting by fetching today's date - least date of that file but again unable to fetch least date.
Format1
r01/WAS2/oss_change0_5.log:2016-03-21T11:13:36.354+0000 | (307,868,305) | OSS_CHANGE |
com.nokia.oss.configurator.rac.provisioningservices.util.Log.logAuditSuccessWithResources | RACPRS RNC 6.0 or
newer_Direct_Activation: LOCKING SUCCEEDED audit[ | Source='Server' | User identity='vpaineni' | Operation
identifier='CMNetworkMOWriterLocking' | Success code='T' | Cause code='N/A' | Identifier='SUCCESS' | Target element='PLMN-
PLMN/RNC-199/WBTS-720' | Client address='10.7.80.21' | Source session identifier='' | Target session identifier='' |
Category code='' | Effect code='' | Network Transaction identifier='' | Source user identity='' | Target user identity='' |
Timestamp='1458558816354']
Code1
cat file1 file2 |sort -u > file3
Data on Day2 ,the input file name Differ
r01/WAS2/oss_change0_11.log:2016-03-21T11:13:36.354+0000 | (307,868,305) | OSS_CHANGE |
com.nokia.oss.configurator.rac.provisioningservices.util.Log.logAuditSuccessWithResources | RACPRS RNC 6.0 or
newer_Direct_Activation: LOCKING SUCCEEDED audit[ | Source='Server' | User identity='vpaineni' | Operation
identifier='CMNetworkMOWriterLocking' | Success code='T' | Cause code='N/A' | Identifier='SUCCESS' | Target element='PLMN-
PLMN/RNC-199/WBTS-720' | Client address='10.7.80.21' | Source session identifier='' | Target session identifier='' |
Category code='' | Effect code='' | Network Transaction identifier='' | Source user identity='' | Target user identity='' |
Timestamp='1458558816354']
I have written almost similar kind of code a week back.
Awk is a good Tool ,if you want to do any operation column wise.
Also , Sort Unique will not work as file name is changing
Both unique rows and least date can be find using awk.
1 To Get Unique file content
cat file1 file2 |awk -F "\|" '!repeat[$21]++' > file3;
Here -F specifies your field separator
Repeat is taking 21st field that is time stamp
and will only print 1st occurrence of that time ,rest ignored
So,finally unique content of file1 and file2 will be available in file3
2 To Get least Date and find difference between 2 dates
Least_Date=`awk -F: '{print substr($2,1,10)}' RMCR10.log|sort|head -1`;
Today_Date=`date +%F` ;
Diff=`echo "( \`date -d $Today_Date +%s\` - \`date -d $Start_Date +%s\`) / (24*3600)" | bc -l`;
Diff1=${Diff/.*};
if [ "$Diff1" -ge "90" ]
then
Here we have used {:} as field separator, and finally substring to get exact date field then sorting and finding least
value.
Subtracting today's Date by using Binary calculator and then removing decimals.
Hope it helps .....
Due to a few company-specific features, which I need to swap in and out, I sometimes have migrated scripts which are not present in the sql directory when I run "info" or "migrate" at a later time. I just noticed an inconsistency, though, in how this displays:
+----------------+----------------------------+---------------------+---------+
| Version | Description | Installed on | State |
+----------------+----------------------------+---------------------+---------+
...
| 4.1 | Add new reports synonyms | 2013-05-31 16:38:22 | Success |
| 4.1.1 | BRNC Add new reports synon | 2013-05-31 16:38:22 | Missing |
| 4.2 | Convert old DATA to DATA2 | 2013-05-31 16:38:22 | Success |
| 4.2.1 | BRNC Convert old DATA to D | 2013-05-31 16:38:22 | Future |
+----------------+----------------------------+---------------------+---------+
So, "Success" means that scripts have been run, and "Missing" means they were run and are no longer present. But what does "Future" mean?
This is similar but not identical to a question:
state of migration scripts is "future"
which was never officially answered, but where Axel Fontaine said in a comment that this had been fixed. I checked, and my jars (3/18) are a later date than his comment (3/2).
As it currently stands this is what these mean:
missing -> executed, no longer found in configured locations, older than the newest found script
future -> executed, no longer found in configured locations, newer than the newest found script
Coming to think of it though, I feel this minor distinction might not be worth a separate state in the info results. I will revisit this in time for 2.2.
Due to a few company-specific features, which I need to swap in and out, I sometimes have migrated scripts which are not present in the sql directory when I run "info" or "migrate" at a later time. I just noticed an inconsistency, though, in how this displays:
+----------------+----------------------------+---------------------+---------+
| Version | Description | Installed on | State |
+----------------+----------------------------+---------------------+---------+
...
| 4.1 | Add new reports synonyms | 2013-05-31 16:38:22 | Success |
| 4.1.1 | BRNC Add new reports synon | 2013-05-31 16:38:22 | Missing |
| 4.2 | Convert old DATA to DATA2 | 2013-05-31 16:38:22 | Success |
| 4.2.1 | BRNC Convert old DATA to D | 2013-05-31 16:38:22 | Future |
+----------------+----------------------------+---------------------+---------+
So, "Success" means that scripts have been run, and "Missing" means they were run and are no longer present. But what does "Future" mean?
This is similar but not identical to a question:
state of migration scripts is "future"
which was never officially answered, but where Axel Fontaine said in a comment that this had been fixed. I checked, and my jars (3/18) are a later date than his comment (3/2).
As it currently stands this is what these mean:
missing -> executed, no longer found in configured locations, older than the newest found script
future -> executed, no longer found in configured locations, newer than the newest found script
Coming to think of it though, I feel this minor distinction might not be worth a separate state in the info results. I will revisit this in time for 2.2.
I know that I can access a single element from a dictionary object with this format ${dict['KEY']}. Like this:
| | Log | ${dict['KEY']} |
And I can set a regular old scalar like this:
| | ${scalar}= | RFKeyword | "Yowp"
But if I try to set a dictionary element like this
| | ${dict['KEY']}= | RFKeyword | "Yowp"
I get "RFKeyword", "Yowp" in the variable, rather than the result of what RFKeyword produces when processing "Yowp" like I do with this
| | ${scalar}= | RFKeyword | "Yowp"
Assistance please
As you probably have figured out, you can't assign to a dictionary from a keyword. You need to very specifically follow the dictionary syntax. you can only return variables to lists or scalars.
Robot framework isn't a fully fledged programming language, and it shouldn't be. By using an intermediate scalar, non-technical testers should be better able to understand what it is doing.
I added this since a google search for "robot framework dictionary" has this question high up in the list.
Just create dictionaries with:
Create dictionary | ${my_dict} | a | b
Add to dictionaries with:
set to dictionary | ${my_dict} | c | d
And retrieve from dictionaries with:
${my_dict["a"]}
Or, if you need to not fail:
${my_dict.get('non-key','default value')}
You just need to rearrange the way you call it. So for your keyword where you want the returned data to go into your dictionary you need to do the following:
${scalar}= | RFKeyword | "Yowp"
Set To Dictionary | ${dict} | KEY | ${scalar}