We're using telegraf to collect CloudWatch data from AWS and output it to InfluxDB.
We need to add dynamic tags to the input, s.t if "instancId == 12345", add tag "user = 3"
Is there a way to do this?
Take a look at the processors. If you have just a set of known values that you want to work with, I think enum would be the best option. Here is the example updated for your case:
[[processors.enum]]
[[processors.enum.mapping]]
## Name of the field to map
field = "instancId"
## Destination field to be used for the mapped value. By default the source
## field is used, overwriting the original value.
dest = "user"
## Default value to be used for all values not contained in the mapping
## table. When unset, the unmodified value for the field will be used if no
## match is found.
default = 0
## Table of mappings
[processors.enum.mapping.value_mappings]
123 = 1
1234 = 2
12345 = 3
See the CONFIGURATION.md docs:
[[inputs.cpu]]
percpu = false
totalcpu = true
[inputs.cpu.tags]
tag1 = "foo"
tag2 = "bar"
Related
I want to check and filter only if the table has value1 = 005 and value1 = 009. But it seems below query is not helping me. I dont know where I am making mistakes. Kindly help to solve this. Note - I cannot use where not as it may have many different value stored in value1 field
DEFINE TEMP-TABLE test NO-UNDO
FIELD value1 AS CHARACTER
.
EMPTY TEMP-TABLE test.
CREATE test.
ASSIGN
value1 = "005".
CREATE test.
ASSIGN
value1 = "009".
CREATE test.
ASSIGN
value1 = "001".
FOR EACH test NO-LOCK
WHERE value1 <> ""
AND (value1 = "005" AND value1 = "009")
:
MESSAGE YES.
END.
You can use can-find
if can-find(first test WHERE value1 = "005")
AND can-find(first test WHERE value1 = "009")
then message yes.
It is safest to always use can-find(first if you're looking for a non-unique value
It looks like you're looking for an OR ooperation, rather than AND.
If you want to check if both records are present you could do :
DEFINE VARIABLE isPresent005 AS LOGICAL NO-UNDO.
DEFINE VARIABLE isPresent009 AS LOGICAL NO-UNDO.
DEFINE VARIABLE bothPresents AS LOGICAL NO-UNDO.
FIND FIRST test WHERE test.value1 = "005" NO-LOCK NO-ERROR.
isPresent005 = AVAIL test.
FIND FIRST test WHERE test.value1 = "009" NO-LOCK NO-ERROR.
isPresent009 = AVAIL test.
bothPresents = isPresent005 AND isPresent009.
But, if you only want to get these 2 records, you should use OR :
FOR EACH test WHERE test.value1 = "005" OR test.value1 = "009" NO-LOCK :
/*do stuff*/
END.
Another option if you are, maybe, looking for some additional fields might look something like this:
define buffer test005 for test.
define buffer test009 for test.
for each test005 no-lock where test005.customer = 1 and test005.value1 = "005",
each test009 no-lock where test009.customer = 1 and test009.value1 = "009":
display test005.customer.
end.
Use OR instead of AND to search the records...
This will return records if value1 = 005 OR value1 = 009.
FOR EACH test NO-LOCK
WHERE value1 <> ""
AND (value1 = "005" OR value1 = "009")
:
MESSAGE YES.
END.
Is not possible to search using your way, because value1 cannot be two values at once, it's always one OR another.
I am new to progress 4gl and below is the query used to add all fields from a table to dynamic temp table except few fields but I am not sure how to add only required fields to dynamic temp table. Please help to modify the query I shared.
/* p-ttdyn2.p - a join of 2 tables */
DEFINE VARIABLE tth4 AS HANDLE.
DEFINE VARIABLE btth4 AS HANDLE.
DEFINE VARIABLE qh4 AS HANDLE.
DEFINE VARIABLE bCust AS HANDLE.
DEFINE VARIABLE bOrder AS HANDLE.
DEFINE VARIABLE i AS INTEGER.
DEFINE VARIABLE fldh AS HANDLE EXTENT 15.
bCust = BUFFER customer:HANDLE.
bOrder = BUFFER order:HANDLE.
CREATE TEMP-TABLE tth4.
tth4:ADD-FIELDS-FROM(bCust,"address,address2,phone,city,comments").
tth4:ADD-FIELDS-FROM(bOrder,"cust-num,carrier,instructions,PO,terms").
tth4:TEMP-TABLE-PREPARE("CustOrdJoinTT").
btth4 = tth4:DEFAULT-BUFFER-HANDLE.
FOR EACH customer WHERE cust.cust-num < 6, EACH order OF customer:
btth4:BUFFER-CREATE.
btth4:BUFFER-COPY(bCust).
btth4:BUFFER-COPY(bOrder).
END.
/* Create Query */
CREATE QUERY qh4.
qh4:SET-BUFFERS(btth4).
qh4:QUERY-PREPARE("for each CustOrdJoinTT").
qh4:QUERY-OPEN.
REPEAT WITH FRAME zz DOWN:
qh4:GET-NEXT.
IF qh4:QUERY-OFF-END THEN LEAVE.
REPEAT i = 1 TO 15:
fldh[i] = btth4:BUFFER-FIELD(i).
DISPLAY fldh[i]:NAME FORMAT "x(15)"
fldh[i]:BUFFER-VALUE FORMAT "x(20)".
END.
END.
btth4:BUFFER-RELEASE.
DELETE OBJECT tth4.
DELETE OBJECT qh4.
ADD-FIELDS-FROM only supports excluding fields that are not needed. Instead you can use ADD-LIKE-FIELD multiple times:
CREATE TEMP-TABLE tth4.
tth4:ADD-LIKE-FIELD("address", "customer.address").
tth4:ADD-LIKE-FIELD("address2", "customer.address2").
tth4:ADD-LIKE-FIELD("phone", customer.phone").
...
tth4:ADD-LIKE-FIELD("cust-num", "Order.cust-num").
...
tth4:TEMP-TABLE-PREPARE("CustOrdJoinTT").
btth4 = tth4:DEFAULT-BUFFER-HANDLE.
Depending on your use case, you can also invert the required field list to an except field list:
var handle ht,hb.
var longchar lcjson.
function invertFields returns character (
i_hb as handle,
i_crequired as char
):
var char cexcept,cfield.
var int ic.
do ic = 1 to i_hb:num-fields:
cfield = i_hb:buffer-field( ic ):name.
if lookup( cfield, i_crequired ) = 0 then
cexcept = cexcept + ',' + cfield.
end.
return substring( cexcept, 2 ).
end function.
create temp-table ht.
ht:add-fields-from(
buffer customer:handle,
invertFields( buffer customer:handle, "CustNum,Name" )
).
ht:temp-table-prepare( 'tt' ).
hb = ht:default-buffer-handle.
hb:buffer-create().
assign
hb::CustNum = 1
hb::Name = 'test'
.
hb:write-json( 'longchar', lcjson, true ).
message string( lcjson ).
https://abldojo.services.progress.com/?shareId=624993253fb02369b25437c4
Using query_parameters, how can I:
specify a result column name (ex: summarize ResultColumnName = count())
specify the value of a bin, when value is actually the name of a column in the table
This is easiest to summarize with an example:
let myTable = datatable (Timestamp:datetime)
[datetime(1910-06-11),
datetime(1930-01-01),
datetime(1997-06-25),
datetime(1997-06-25)];
let UntrustedUserInput_ColumnName = "MyCount"; // actually from query_parameters
let UntrustedUserInput_BinValue = "Timestamp"; // actually from query_parameters
let UntrustedUserInput_BinRoundTo = "365d"; // actually from query_parameters
// the query I really want to perform
myTable
| summarize MyCount=count() by bin(todatetime(Timestamp), totimespan(365d));
// what the query looks like if I use query_parameters
myTable
| summarize UntrustedUserInput_ColumnName=count() by bin(todatetime(UntrustedUserInput_BinValue), totimespan(UntrustedUserInput_BinRoundTo));
Results:
Timestamp MyCount
--------- -------
1909-09-26T00:00:00Z 1
1929-09-21T00:00:00Z 1
1996-09-04T00:00:00Z 2
Column1 UntrustedUserInput_ColumnName
------- -----------------------------
4
I can't find a solution to #1.
It appears #2 can almost be solved by using column_ifexists, but I don't have a "default" to fall back on, I'd rather just fail if the column doesn't exist.
Treating column names as variables is not possible since columns names are part of the result schema coming out of each operator (with the exception of the "evaluate" operator, see specifically the pivot plugin).
There actually is a way to set variable names to a column, using a hacky trick:
let VariableColumnName = "TestColumn"; // the new column name that you want
range i from 1 to 5 step 1 // this is just a sample query
| project pack(VariableColumnName, i) // this created a JSON
| evaluate bag_unpack(Column1) // unpacking the JSON creates a column with a dynamic name
This will return a column named TestColumn, which is set in VariableColumnName.
In u-sql script I must extract a variable from file to a dataset and then use it to form a name of output file. How can I get the variable from the dataset?
In details.
I have 2 input files: csv file with a set of fields and a dictionary file. The 1st file has file name like ****ClintCode*****.csv. The 2nd file-dictionary has 2 fields with mapping: ClientCode - ClintCode2. My task is extract ClientCode value from the file name, get ClientCode2 from the dictionary, insert it as a field to output file (implemented), and, moreover, form the name of output file as ****ClientCode2****.csv.
Dictionary csv file has the content:
OldCode NewCode
6HAA Alfa
CCVV Beta
CVXX gamma
? Davis
The question is how to get ClientCode2 into scalar variable to write an expression for the output file?
DECLARE #inputFile string = "D:/DFS_SSC_Automation/Tasks/FundInfo/ESP_FAD_GL_6HAA_20170930.txt"; // '6HAA' is ClientCode here that mapped to other code in ClientCode_KVP.csv
DECLARE #outputFile string = "D:/DFS_SSC_Automation/Tasks/FundInfo/ClientCode_sftp_" + // 'ClientCode' should be replaced with ClientCode from mapping in ClientCode_KVP.csv
DateTime.Now.ToString("yyyymmdd") + "_" +
DateTime.Now.ToString("HHmmss") + ".csv";
DECLARE #dictionaryFile string = "D:/DFS_SSC_Automation/ClientCode_KVP.csv";
#dict =
EXTRACT [OldCode] string,
[NewCode] string
FROM #dictionaryFile
USING Extractors.Text(skipFirstNRows : 1, delimiter : ',');
#theCode =
SELECT Path.GetFileNameWithoutExtension(#inputFile).IndexOf([OldCode]) >= 0 ? 1 : 3 AS [CodeExists],
[NewCode]
FROM #dict
UNION
SELECT *
FROM(
VALUES
(
2,
""
)) AS t([CodeExists],[NewCode]);
#code =
SELECT [NewCode]
FROM #theCode
ORDER BY [CodeExists]
FETCH 1 ROWS;
#GLdata =
EXTRACT [ASAT] string,
[ASOF] string,
[BASIS_INDICATOR] string,
[CALENDAR_DATE] string,
[CR_EOP_AMOUNT] string,
[DR_EOP_AMOUNT] string,
[FUND_ID] string,
[GL_ACCT_TYPE_IND] string,
[TRANS_CLIENT_FUND_NUM] string
FROM #inputFile
USING Extractors.Text(delimiter : '|', skipFirstNRows : 1);
// Prepare output dataset
#FundInfoGL =
SELECT "" AS [AccountPeriodEnd],
"" AS [ClientCode],
[FUND_ID] AS [FundCode],
SUM(GL_ACCT_TYPE_IND == "A"? System.Convert.ToDecimal(DR_EOP_AMOUNT) : 0) AS [NetValueOtherAssets],
SUM(GL_ACCT_TYPE_IND == "L"? System.Convert.ToDecimal(CR_EOP_AMOUNT) : 0) AS [NetValueOtherLiabilities],
0.0000 AS [NetAssetsOfSeries]
FROM #GLdata
GROUP BY FUND_ID;
// NetAssetsOfSeries calculation
#FundInfoGLOut =
SELECT [AccountPeriodEnd],
[NewCode] AS [ClientCode],
[FundCode],
Convert.ToString([NetValueOtherAssets]) AS [NetValueOtherAssets],
Convert.ToString([NetValueOtherLiabilities]) AS [NetValueOtherLiabilities],
Convert.ToString([NetValueOtherAssets] - [NetValueOtherLiabilities]) AS [NetAssetsOfSeries]
FROM #FundInfoGL
CROSS JOIN #code;
// Output
OUTPUT #FundInfoGLOut
TO #outputFile
USING Outputters.Text(outputHeader : true, delimiter : '|', quoting : false);
As David points out: You cannot assign query results to scalar variables.
However, we have a dynamic partitioned output feature in private preview right now that will give you the ability to generate file names based on column values. Please contact me if you want to try it out.
You can't. Please see Convert Rowset variables to scalar value.
You may still be able to achieve your ultimate goal in a different manner. Please consider re-writing your post with clear & concise language, small dataset, expected output, and a very minimal amount of code needed to repro - remove all details and nuances that aren't necessary to create a test case.
SELECT skill_name, character_name, cb_id, cb_id2 FROM characterbasics, characterskills WHERE characterbasics.character_name = 'Joe' & characterbasics.cb_id = characterskills.cb_id2
This, for some reason, returns 0 rows
The character name is in there (as well as 2 other dummy names).. and both cbid and cbid2 are the same.
When i try the query without the & cbid=cbid2 i get the name with the other data.. now when i check for JUST cbid=cbid2 i get 3 different dummy characters i created...
im trying to pull all "skills" associated with one character by matching the id of the character name in table 1 with the character id in table 2
Where have I erred?
cn = character name
cn cbid cbid2
Joe 2 2
This is what it SHOULD look like..
You cant use & as logical AND operator (& is binary operator), so sql should look like :
SELECT skill_name, character_name, cb_id, cb_id2
FROM characterbasics, characterskills
WHERE characterbasics.character_name = 'Joe' AND characterbasics.cb_id = characterskills.cb_id2