Kusto - Assign Custom Regex To Variable in the Parse Operator - azure-data-explorer

I am trying to use the parse operator to parse data into their respective fields. It seems that data is only parsable in between throwaway regex patterns, but I need to capture a pattern into a variable. So far I have the below query:
let Traces = datatable(EventText:string)
[
'2021-10-04T20:43:03,174 2511 INFO cd060096-c6c4-4ddf-b9f7-5795f6d04514 c2a42807-6ab3-41bb-8d72-1c48f2213c31 iTKTS Fiona (ABSDEF) () () () ITKTSUtil - <ProductFulfillmentResponse>U2028 <errorStatus>UNPROCESSED</errorStatus>U2028 <errorCode>GEN_ERR</errorCode>U2028 <errorDescription>WARNING - UNPROCESSED DUE TO OTHER ERRORS</errorDescription>U2028 <customerDocuments>U2028 <errorDescription>WARNING - UNPROCESSED DUE TO OTHER ERRORS</errorDescription>U2028 <itemFulfillmentInfos>U2028 <errorDescription>WARNING - UNPROCESSED DUE TO OTHER ERRORS</errorDescription>U2028 </itemFulfillmentInfos>U2028 </customerDocuments>U2028</ProductFulfillmentResponse>U2028'
];
Traces
| parse kind = regex EventText with _timestamp ",\\d{3} " _threadid " " _logLevel " " _clientTransactionId " " _appTransactionId " " _appService " " _bigeazy " \\(" _recordLocator "\\) \\(" _status "\\) \\(" _responseTime "\\) \\(" _serviceName "\\) " _className " - " _message
| project _className, _message
I need _className to match "ITKTSUtil". By default a variable matches the pattern (.*?). If I change it to _className:long it matches the pattern (\-\d+). But I need it to match the pattern //w* and then be captured into the variable _className. Is this possible with KQL?

Please try the following approach :
let Traces = datatable(EventText:string)
[
'2021-10-04T20:43:03,174 2511 INFO cd060096-c6c4-4ddf-b9f7-5795f6d04514 c2a42807-6ab3-41bb-8d72-1c48f2213c31 iTKTS Fiona (ABSDEF) () () () ITKTSUtil - <ProductFulfillmentResponse>U2028 <errorStatus>UNPROCESSED</errorStatus>U2028 <errorCode>GEN_ERR</errorCode>U2028 <errorDescription>WARNING - UNPROCESSED DUE TO OTHER ERRORS</errorDescription>U2028 <customerDocuments>U2028 <errorDescription>WARNING - UNPROCESSED DUE TO OTHER ERRORS</errorDescription>U2028 <itemFulfillmentInfos>U2028 <errorDescription>WARNING - UNPROCESSED DUE TO OTHER ERRORS</errorDescription>U2028 </itemFulfillmentInfos>U2028 </customerDocuments>U2028</ProductFulfillmentResponse>U2028'
];
Traces
| parse kind = regex flags=U EventText with _timestamp ",\\d{3} " _threadid " " _logLevel " " _clientTransactionId " " _appTransactionId " " _appService " " _bigeazy " \\(" _recordLocator "\\) \\(" _status "\\) \\(" _responseTime "\\) \\(" _serviceName "\\) " _className " - " _message "$"
| project _className, _message
The main idea was to use the flags of parse regex mode (using the regex flag U which means ungreedy in order to match only the required field and also add "$" to require the parse regex mode to perform a full match).
Please note that if your pattern in know in advance, it is recommended to use the parse simple mode which is much faster.

Related

R: Why does \s{2} return "\"," in regex match? And new line not matching with \\n or \\r and other variations

I'm having trouble with (1) using a dynamic variable in a regex pattern and (2) matching "\" or new line. I'd really appreciate any help!
Example: Ultimately, however possible, I'd like to match the word Administrator in the text file below. The text file's data classification is character (it was originally a list and was coerced to character using as.character(). Here's the text snippet:
[1] "c(\"Silk Road Forums\", \"\", \"*\", \"Welcome, Guest. Please login or register.\", \"[ ] [ ] [Forever] [Login]\", \"Login with username, password and session length\", \"[ ] [Search] \", \"\", \" â\\200¢ Home\", \" â\\200¢ Search\", \" â\\200¢ Login\", \" â\\200¢ Register\", \"\", \"\", \" â\\200¢ Silk Road Forums »\", \" â\\200¢ Profile of Dread Pirate Roberts »\", \" â\\200¢ Summary\", \"\", \" â\\200¢ Profile Info\", \" â–¡ Summary\", \" â–¡ Show Stats\", \" â–¡ Show Posts...\", \" â\\230† Messages\", \n\" â\\230† Topics\", \" â\\230† Attachments\", \"\", \"[profile_sm]Summary\", \"\", \"Dread Pirate Roberts Administrator\", \"\", \"[index]\", \" â–¡ SMF | SMF © 2013, Simple Machines\"\n)"
Attempts / Problems
Tried to Match New Line: In that messy text (see above), I was able to match [profile_sm]Summary\. I tried to match what comes next in that text by using:
\\n -- failed
\\n\\r -- failed
\\n|\\r -- failed
\\r\\n -- failed
\\r|\\n -- failed
It seems like there's no new line after so I tried to match the literal ""," (inside quotation marks: quotation mark and comma) that comes after characters in that text. So I also tried these two and they both failed: \\and \\"\".
Tried to Use Variable: I tried to use variable X that includes Dread Pirate Roberts from a previous regex match turned into a vector. I tried to just put X into the regex pattern but it obviously didn't work. Is there away to create a pattern using X? For example: Match one of the values found in x.
I would need to know how to solve both of these problems / methods for other parts of my current project and would really love pointers and guidance. Thank you!
Edit Note: Saw that folks had trouble understanding this post so I edited to make it more legible. Thanks and shout-out to #Wiktor Stribiżew for reading through the original post despite the difficult wording and providing the answer! :)
Your text only contains two newlines, you can easily check it using cat(text) and there are three lines:
c("Silk Road Forums", "", "*", "Welcome, Guest. Please login or register.", "[ ] [ ] [Forever] [Login]", "Login with username, password and session length", "[ ] [Search] ", "", " � Home", " � Search", " � Login", " � Register", "", "", " � Silk Road Forums »", " � Profile of Dread Pirate Roberts »", " � Summary", "", " � Profile Info", " □ Summary", " □ Show Stats", " □ Show Posts...", " � Messages",
" � Topics", " � Attachments", "", "[profile_sm]Summary", "", "Dread Pirate Roberts Administrator", "", "[index]", " □ SMF | SMF © 2013, Simple Machines"
)
So, as you see, there is no newline after [profile_sm]Summary. Note to match [ in a regex pattern you need to escape it.. There is a space, " and commas You may match these chars using [,"\s]+ pattern. The X variable will hold Dread Pirate Roberts, so, to extract Administrator you may use
\[profile_sm]Summary[",\s]*Dread Pirate Roberts\s+\K[^"]+
See the regex demo.
Details
\[profile_sm]Summary - [profile_sm]Summary string
[",\s]* - 0+ ", , or whitespace chars
Dread Pirate Roberts - a literal string
\s+ - 1+ whitespaces
\K - match reset operator that discards text matched so far in the match memory buffer
[^"]+ - 1+ chars other than ". If you need to only match letter, digits or _ you may use \w+ instead of this pattern (with \\ in the string literal).
R demo:
text <- "c(\"Silk Road Forums\", \"\", \"*\", \"Welcome, Guest. Please login or register.\", \"[ ] [ ] [Forever] [Login]\", \"Login with username, password and session length\", \"[ ] [Search] \", \"\", \" â\200¢ Home\", \" â\200¢ Search\", \" â\200¢ Login\", \" â\200¢ Register\", \"\", \"\", \" â\200¢ Silk Road Forums »\", \" â\200¢ Profile of Dread Pirate Roberts »\", \" â\200¢ Summary\", \"\", \" â\200¢ Profile Info\", \" â–¡ Summary\", \" â–¡ Show Stats\", \" â–¡ Show Posts...\", \" â\230† Messages\", \n\" â\230† Topics\", \" â\230† Attachments\", \"\", \"[profile_sm]Summary\", \"\", \"Dread Pirate Roberts Administrator\", \"\", \"[index]\", \" â–¡ SMF | SMF © 2013, Simple Machines\"\n)"
X <- "Dread Pirate Roberts"
regex <- paste0('\\[profile_sm]Summary[",\\s]*',X,'\\s+\\K[^"]+')
regmatches(text, regexpr(regex, text, perl=TRUE))
## => [1] "Administrator"

Receiving and Sending Strings over the Network in QB64

I'm working on a mini pokemon game for my brother that works over the network.
Unfortunately, when testing I find that for some reason, it gives an error about "Bad file name or number" only on the lines where it tries to send a string to another computer, but has no error when looping the receive command.
Here's my code:
SCREEN 12
CLS
PRINT ""
PRINT ""
PRINT ""
PRINT ""
PRINT ""
PRINT " POKELITE - By Mark "
PRINT ""
PRINT ""
INPUT "Join or Host a game? ", hostorjoin$
hostorjoin$ = UCASE$(hostorjoin$)
IF hostorjoin$ = "JOIN" THEN GOTO JOIN
IF hostorjoin$ = "HOST" THEN GOTO HOST
HOST:
server& = _OPENHOST("TCP/IP:300")
PRINT "Waiting for connection..."
PRINT "! Remember: If playing locally, give the other player your IPv4 Address !"
DO
HOST& = _OPENCONNECTION(server&)
LOOP UNTIL HOST& <> 0
PRINT ""
PRINT "2nd Player Joined!"
SLEEP 2
GOTO GAME
JOIN:
INPUT "Enter Server IPv4 Address (Example: 192.168.1.25): ", joinip$
handle& = _OPENCLIENT("TCP/IP:300:" + joinip$)
IF handle& = 0 THEN PRINT "Connection failed!": SLEEP 2: CLS: GOTO JOIN
GOTO GAME
GAME:
CLS
INPUT "Enter your name: ", name$
IF name$ = "" THEN GOTO GAME
PRINT "Waiting for other player..."
IF hostorjoin$ = "JOIN" THEN
PUT HOST&, , name$
DO
GET handle&, , name2$
LOOP UNTIL name2$ <> ""
END IF
IF hostorjoin$ = "HOST" THEN
PUT handle&, , name$
DO
GET HOST&, , name2$
LOOP UNTIL name2$ <> ""
END IF
PRINT name$
PRINT name2$
You need to ensure the port is available, else server& will be an invalid server handle. Choosing a port of 49152 or higher is generally safe. This probably isn't your only problem, however.
Your problem is likely that your connection variable is simply not the same, meaning HOST& and handle& should just be handle&. It's important to remember that there is never a "host handle" and a "client handle"; the only handles are the "server handle" (created using _OPENHOST to essentially reserve a port for your connections) and the "connection handle" (created using _OPENCONNECTION by the server to connect to a client or _OPENCLIENT by the client to connect to a server). This will also reduce your logic to just do a PUT, followed by a GET loop. I use the name connection& instead of handle&, but you get the idea.
SCREEN 12
CLS
PRINT ""
PRINT ""
PRINT ""
PRINT ""
PRINT ""
PRINT " POKELITE - By Mark "
PRINT ""
PRINT ""
INPUT "Join or Host a game? ", hostorjoin$
hostorjoin$ = UCASE$(hostorjoin$)
IF hostorjoin$ = "JOIN" THEN GOTO JOIN
IF hostorjoin$ = "HOST" THEN GOTO HOST
' If neither "HOST" nor "JOIN" is specified, what happens?
HOST:
server& = _OPENHOST("TCP/IP:300")
PRINT "Waiting for connection..."
PRINT "! Remember: If playing locally, give the other player your IPv4 Address !"
DO
connection& = _OPENCONNECTION(server&)
LOOP UNTIL connection& <> 0
PRINT ""
PRINT "2nd Player Joined!"
SLEEP 2
GOTO GAME
JOIN:
INPUT "Enter Server IPv4 Address (Example: 192.168.1.25): ", joinip$
connection& = _OPENCLIENT("TCP/IP:300:" + joinip$)
IF connection& = 0 THEN PRINT "Connection failed!": SLEEP 2: CLS: GOTO JOIN
GOTO GAME
GAME:
CLS
INPUT "Enter your name: ", playerName$
IF playerName$ = "" THEN GOTO GAME
PRINT "Waiting for other player..."
' Send name to opponent and wait for opponent's name.
PUT connection&, , playerName$
DO
GET connection&, , opponentName$
LOOP UNTIL opponentName$ <> ""
PRINT "You: "; playerName$
PRINT "Opponent:"; opponentName$

how to display output in the following format in unix

Hi my data looks in the below format in unix file
BENEFITS-T_FACT_DEPNDT_COVRG-230
BENEFITS-T_FACT_DEPNDT_COVRG-290
BENEFITS-T_FACT_ELECTN-0
BENEFITS-T_FACT_ELECTN-39092
HR-T_DIM_CLNT-0
HR-T_DIM_CLNT-98
HR-T_DIM_CMPNY-10
HR-T_DIM_CMPNY-45
I need to display the output like this:
domain -tablename- before load cnt: 230 after load cnt :290
ex : BENEFITS - T_FACT_DEPNDT_COVRG -before load cnt: 230 after load cnt :290
HR -T_DIM_CLNT -before load cnt: 0 after load cnt :98
Can someone please suggest an answer in unix ???
Is this enough?
awk -F "-" '{if(NR%2){printf $1 " " $2 " -before load cnt: " $3}else{printf " after load cnt: " $3 "\n"}}' filename

Create a message from a collection of collections in MULE ESB

I have a JSON like:
{
"EDITORS" : [{
"EDITOR" : "MCGRAM HILL",
"BOOKS" : [{
"NAME" : "DIFFERENTIAL CALCULUS",
"YEAR" : "1995",
"TIMES_READ" : "135"
}, {
"NAME" : "2012 THE END OF THE WORLD",
"YEAR" : "2012,",
"TIMES_READ" : "56"
}
]
}, {
"EDITOR" : "DEMIDOVICH",
"BOOKS" : [{
"NAME" : "SOME TITTLE",
"YEAR" : "1975,",
"TIMES_READ" : "154"
}, {
"NAME" : "THE LITTLE PRINCE",
"YEAR" : "1987,",
"TIMES_READ" : "57"
}
]
}, {
"EDITOR" : "ADRIAN LOPEZ ASC.",
"BOOKS" : [{
"NAME" : "SOMETHING",
"YEAR" : "2008,",
"TIMES_READ" : "10"
}
]
}
]
}
I need to transform it into an output like:
payload[0]= The editor MCGRAM HILL has:
The book DIFFERENTIAL CALCULUS published in 1995 has been readed 135 times.
The book 2012 THE END OF THE WORLD published in 2012 has been readed 56 times.
payload[1]= The editor DEMIDOVICH has:
The book SOME TITTLE published in 1975 has been readed 154 times.
The book THE LITTLE PRINCE published in 1987 has been readed 57 times.
payload[2]= The editor ADRIAN LOPEZ ASC. has:
The book SOMETHING published in 2008 has been readed 10 times.
So far I transform the JSON to an Java Object, then use a foreach scope with the collection #[payload.EDITORS]
Inside the foreach I create a variable Intro = "The editor #[payload.EDITOR] has:"
And after that I'm lost.
I have tried to set payload to #[payload.BOOKS] and then use another foreach but this only shows the last book, and have also try Collection Aggregator but don't know what to write in "Message info Mapping".
How can I accomplish the required output?
Any help could be greatly appreciated, thanks in advance.
Try following with dataweave
%dw 1.0
%output application/json
---
{(payload.EDITORS map ((data, index) -> {
payload : "The editor " ++ data.EDITOR ++ " has:\r\n" ++
{(data.BOOKS map {
pqr : "The book " ++ $.NAME ++ " published in " ++ $.YEAR ++ " has been readed " ++ $.'TIMES_READ' ++ " times."
})} pluck $ joinBy "\r\n"
}))} pluck $
Json Output for reference. You can transform to java to get desired output.
[
"The editor MCGRAM HILL has:\r\nThe book DIFFERENTIAL CALCULUS published in 1995 has been readed 135 times.\r\nThe book 2012 THE END OF THE WORLD published in 2012, has been readed 56 times.",
"The editor DEMIDOVICH has:\r\nThe book SOME TITTLE published in 1975, has been readed 154 times.\r\nThe book THE LITTLE PRINCE published in 1987, has been readed 57 times.",
"The editor ADRIAN LOPEZ ASC. has:\r\nThe book SOMETHING published in 2008, has been readed 10 times."
]
Hope this helps.
IMHO, you don't need to set payload inside the foreach scope.
I tried the following configuration and it produces a list of string which containing those message. I think you can modify the last message processor to display the expected output (e.g.: add another foreach or expression to extract the message from the final payload/ArrayList).
<json:json-to-object-transformer returnClass="java.util.Map" doc:name="JSON to Object"/>
<set-variable variableName="messageMapping" value="#[new java.util.ArrayList()]" doc:name="Variable"/>
<foreach collection="#[payload.EDITORS]" doc:name="For Each">
<set-variable variableName="messageEditor" value="The editor #[payload.EDITOR] has: " doc:name="Variable"/>
<foreach collection="#[payload.BOOKS]" doc:name="For Each">
<expression-transformer expression="#[flowVars.messageEditor += "\r\nThe book " + payload.NAME + " published in " + payload.YEAR + " has been readed " + payload.TIMES_READ + " times."]" doc:name="Expression"/>
</foreach>
<expression-transformer expression="#[flowVars.messageMapping.add(flowVars.messageEditor)]" doc:name="Expression"/>
</foreach>
<set-payload value="#[flowVars.messageMapping]" doc:name="Set Payload"/>
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
If you prefer to use DataWeave, then you can use the following code:
%dw 1.0
%output application/java
---
payload.EDITORS map (
'The editor ' ++ $.EDITOR ++ ' has:\r\n' ++
($.BOOKS map (
'The book ' ++ $.NAME ++ ' published in ' ++ $.YEAR ++ ' has been readed ' ++ $.TIMES_READ ++ ' times.'
) joinBy '\r\n')
)
Dataweave doens't feel like the right approach, specially because the output is plain txt.
Personally I'd probably use json-to-object (see below) and parse template.
<json:json-to-object-transformer returnClass="java.util.Map" />

Custom rubble in Aptana not activated automatically

I am new to Aptana, so I could be missing something obvious. My problem in that bundle I created does not get activated when Aptana is loaded. I have to update date of the rubble.rb file for it to work.
touch ~/Documents/Aptana Rubles/ioncubeEncode.ruble
After I do that, for the rest of that Aptana session things are peachy.
I do not think it is relevant, but here is bundle code, which works just fine (when it works):
require 'ruble'
#Ruble::Logger.log_level = :trace
bundle do |bundle|
bundle.author = "Sasha"
bundle.copyright = "None"
bundle.display_name = "Ioncube Encode"
bundle.description = "Encode just saved php file form base_local to base folder"
end
command "Ioncube Encode" do |cmd|
cmd.input = :document
cmd.output = :output_to_console
cmd.trigger = :execution_listener , "org.eclipse.ui.file.save"
cmd.invoke do |ctx|
source_path = ENV['TM_FILEPATH']
ext = File.extname(source_path)
if ext == '.php'
if ( source_path =~ /base_local/)
destination_path = source_path.sub('/base_local/','/base/')
#CONSOLE.puts "Ioncube Encoding: " + source_path + " to " + destination_path
exec="/usr/local/ioncube/ioncube_encoder5 -v --optimize more --without-loader-check " + source_path + " -o " +destination_path + " 2>&1"
IO.popen(exec, 'r+') do |io|
io.close_write
CONSOLE.puts io.read
end
end
end
end
end
This might not seem like a big deal but it bothers me - a lot.

Resources