I currently have formed a KQL that extracts ipv4 address from a string. similarly I would need to extract ipv6 address from the string
ipv4 extract query:
datatable (ipv4text:string)
[
'This is a random text that has IP address 128.0.0.20 that has to be extracted'
]
|extend pv4addr = extract("(([0-9]{1,3})\\.([0-9]{1,3})\\.([0-9]{1,3})\\.(([0-9]{1,3})))",1,ipv4text)
I tried the below but not sure if it covers all the edge cases
datatable (ipv6:string)
[
'IPv6 test 198.192.0.127 2345:5:2CA1:0000:0000:567:5673:256/127 in a random string'
]
|extend Ipv6Address = extract(#"(([0-9a-fA-F]{1,4}\:){7,7}[0-9a-fA-F]{1,4})|([0-9a-fA-F]{1,4}\:){1,7}\:",1,ipv6)
Can any of you one provide a complete KQL(or suggestions/hints) to extract IPV6 address?
Thanks.
The regex patterns can be simplified.
Below are the "happy paths". If it's there it will be extracted.
Theoretically you might get false positives, although less unlikely with a real-life data.
If needed, we can add some protection layers.
datatable (ipv4text:string)
[
'This is a random text that has IP address 128.0.0.20 that has to be extracted'
]
| project pv4addr = extract(#"(\d{1,3}\.){3}\d{1,3}", 0, ipv4text)
pv4addr
128.0.0.20
Fiddle
IPV6 can become a mess (see https://en.wikipedia.org/wiki/IPv6_address#Representation).
I would go with finding a full IPV6 representation (8 xdigit tokens, separated by colon) or any expression built of xdigit/colon/period that contains 2 adjacent colons.
datatable (ipv6:string)
[
'IPv6 test 198.192.0.127 2345:5:2CA1:0000:0000:567:5673:256/127 in a random string'
,'IPv6 test 198.192.0.127 2345:5:2CA1::567:5673:256/127 in a random string'
,'IPv6 test 198.192.0.127 ::ffff:198.192.0.127 in a random string'
,'IPv6 test 198.192.0.127 ::1 in a random string'
,'IPv6 test 198.192.0.127 :: in a random string'
]
| project pv6addr = extract(#"([[:xdigit:]]{1,4}:){7}[[:xdigit:]]{1,4}|[[:xdigit:]:.]*::[[:xdigit:]:.]*", 0, ipv6)
pv6addr
2345:5:2CA1:0000:0000:567:5673:256
2345:5:2CA1::567:5673:256
::ffff:198.192.0.127
::1
::
Fiddle
Related
I am getting an error when running the following code in python 3, I look all over but could not find a right way to do it. any help will be appreciated.
raise TypeError('unicode strings are not supported, please encode to bytes: {!r}'.format(seq))
TypeError: unicode strings are not supported, please encode to bytes: 'relay read 7\n\r'
I need to send the following string via serial port: relay read #of relay.
import sys
import serial
if (len(sys.argv) < 2):
print ("Usage: relayread.py <PORT> <RELAYNUM>\nEg: relayread.py COM1 0")
sys.exit(0)
else:
portName = sys.argv[1];
relayNum = sys.argv[2];
#Open port for communication
serPort = serial.Serial(portName, 19200, timeout=1)
if (int(relayNum) < 10):
relayIndex = str(relayNum)
else:
relayIndex = chr(55 + int(relayNum))
serPort.write("relay read "+ relayIndex + "\n\r")
response = serPort.read(25)
if(response.find("on") > 0):
print ("Relay " + str(relayNum) +" is ON")
elif(response.find("off") > 0):
print ("Relay " + str(relayNum) +" is OFF")
#Close the port
serPort.close()
Use the string's encode method to construct the corresponding byte sequence.
In this case all of the characters in the string are in the ASCII range so it doesn't really matter which encoding scheme you use. (Differences between encoding schemes generally only matter when you're dealing with non-ASCII characters, ones whose ord() value is greater than 127.) So in this case you don't even need to specify a particular encoding scheme, you can simply use the encode method with no argument and let Python apply the platform's default encoding.
To do that, change this:
serPort.write("relay read "+ relayIndex + "\n\r")
to this:
serPort.write(("relay read "+ relayIndex + "\n\r").encode())
You'll probably have to do the reverse operation to get a string from the byte sequence returned by serPort.read. Change this:
response = serPort.read(25)
to:
response = serPort.read(25).decode()
BTW, it's typical for line endings in transmitted data to be represented by a Carriage Return followed by a Line Feed, or "\r\n". In your serPort.write call you're using the reverse of that, "\n\r". That's unusual but if that's what your device needs then so be it.
According to what I have researched there are no illegal characters in the PWD= field of a SQL Server Connection String.
However, using SQL Server Express 2008 I changed the SA password to a GUID, specifically:
{85C86BD7-B15F-4C51-ADDA-3B6A50D89386}
So when connecting via ODBC I use this connection string:
"Driver={SQL Server};Server=.\\MyInstance;Database=Master;UID=SA;PWD={85C86BD7-B15F-4C51-ADDA-3B6A50D89386};"
But it comes back as Login failed for SA.
However, if I change the SA password to something just as long but without {}- it succeeds! Are there certain characters in PWD= that need to be escaped? I tried all different combinations with no luck.
As Microsoft's documentation states (emphasis added) --
Connection strings used by ODBC have the following syntax:
connection-string ::= empty-string[;] | attribute[;] | attribute; connection-string
empty-string ::=
attribute ::= attribute-keyword=[{]attribute-value[}]
attribute-value ::= character-string
attribute-keyword ::= identifier
Attribute values can optionally be enclosed in braces, and it is good practice to do so. This avoids problems when attribute values contain non-alphanumeric characters. The first closing brace in the value is assumed to terminate the value, so values cannot contain closing brace characters.
I would suggest you simply remove the braces when you set the password, and then the connect string you provided above should work fine.
ADDITION
I dug a bit further on Microsoft's site, and found some ABNF rules which may be relevant --
SC = %x3B ; Semicolon
LCB = %x7B ; Left curly brackets
RCB = %x7D ; Right curly brackets
EQ = %x3D ; Equal sign
ESCAPEDRCB = 2RCB ; Double right curly brackets
SpaceStr = *(SP) ; Any number (including 0) spaces
ODBCConnectionString = *(KeyValuePair SC) KeyValuePair [SC]
KeyValuePair = (Key EQ Value / SpaceStr)
Key = SpaceStr KeyName
KeyName = (nonSP-SC-EQ *nonEQ)
Value = (SpaceStr ValueFormat1 SpaceStr) / (ValueContent2)
ValueFormat1 = LCB ValueContent1 RCB
ValueContent1 = *(nonRCB / ESCAPEDRCB)
ValueContent2 = SpaceStr / SpaceStr (nonSP-LCB-SC) *nonSC
nonRCB = %x01-7C / %x7E- FFFF ; not "}"
nonSP-LCB-SC = %x01-1F / %x21-3A / %x3C-7A / %x7C- FFFF ; not space, "{" or ";"
nonSP-SC-EQ = %x01-1F / %x21-3A / %x3C / %x3E- FFFF ; not space, ";" or "="
nonEQ = %x01-3C / %x3E- FFFF ; not "="
nonSC = %x01-003A / %x3C- FFFF ; not ";"
...
ValueFormat1 is recommended to use when there is a need for Value to contain LCB, RCB, or EQ. ValueFormat1 MUST be used when the Value contains SC or starts with LCB.
ValueContent1 MUST be enclosed by LCB and RCB. Spaces before the enclosing LCB and after the enclosing RCB MUST be ignored.
ValueContent1 MUST be contained in ValueFormat1. If there is an RCB in the ValueContent1, it MUST use the two-character sequence ESCAPEDRCB to represent the one-character value RCB.
All of which comes down to... I believe the following connect string should work for you (note that there are 2 left/open braces and 3 right/close braces on the PWD value) --
"Driver={SQL Server};Server=.\\MyInstance;Database=Master;UID=SA;PWD={{85C86BD7-B15F-4C51-ADDA-3B6A50D89386}}};"
According to this page, the only legal "special character" in a name (I think they're talking about the DSN) is the UNDERSCORE:
The ODBC specification (and the SQL specification) states that names
must be in the format of " letter[digit | letter | _]...". The only
special character allowed is an underscore.
There was no reference to "the ODBC Specification". This page says it's the the ODBC 4.0 Spec.
test.data <- data.frame(summary = c("Execute commands as root via buffer overflow in Tooltalk database server (rpc.ttdbserverd)."
,"Information from SSL-encrypted sessions via PKCS #1."
,"ip_input.c in BSD-derived TCP/IP implementations allows remote attackers to cause a denial of service (crash or hang) via crafted packets."),
wascname=c(NA, NA, "Improper Input Handling"),stringsAsFactors = FALSE)
wascNames <- data.frame(wascname=c("Abuse of Functionality","Brute Force","Buffer Overflow","Content Spoofing"
,"Credential/Session Prediction","Cross-Site Scripting","Cross-Site Request Forgery","Denial of Service"
,"Fingerprinting","Format String","HTTP Response Smuggling","HTTP Response Splitting"
,"HTTP Request Smuggling","HTTP Request Splitting","Integer Overflows","LDAP Injection"
,"Mail Command Injection","Null Byte Injection","OS Commanding","Path Traversal"
,"Predictable Resource Location","Remote File Inclusion (RFI)","Routing Detour","Session Fixation"
,"SOAP Array Abuse","SSI Injection","SQL Injection","URL Redirector Abuse"
,"XPath Injection","XML Attribute Blowup","XML External Entities","XML Entity Expansion"
,"XML Injection","XQuery Injection","Cross-site Scripting","Directory Indexing"
,"Improper Filesystem Permissions","Improper Input Handling","Improper Output Handling","Information Leakage"
,"Insecure Indexing","Insufficient Anti-Automation","Insufficient Authentication","Insufficient Authorization"
,"Insufficient Password Recovery","Insufficient Process Validation","Insufficient Session Expiration","Insufficient Transport Layer Protection"
,"Remote File Inclusion","URl Redirector Abuse"),stringsAsFactors = FALSE)
Below is the code I am have been trying to fix. If test.data$summary contains string in wascNames$wascname, replace test.data$wascname only if is.na:
test.data$wascname<-sapply(test.data$summary, function(x)
ifelse(identical(wascNames$wascname[str_detect(x,regex(wascNames$wascname, ignore_case = T))&
is.na(test.data$wascname)==TRUE], character(0)),test.data$wascname,
wascNames$wascname[str_detect(x,regex(wascNames$wascname, ignore_case = T))==TRUE]))
I want the following output:
Thank you in advance. Thought of using for loop, but would be too slow for 200000 obs.
I believe this should work:
test.data$wascname2 <- sapply(1:nrow(test.data), function(x) ifelse(is.na(test.data$wascname[x]),
wascNames$wascname[str_detect(test.data$summary[x], regex(wascNames$wascname, ignore_case = TRUE))],
test.data$wascname[x]))
test.data$wascname2
#[1] "Buffer Overflow" NA "Improper Input Handling"
It still loops with sapply, but I think that's unavoidable given your data structure (i.e. for each string, you want to look it up in your wascNames$wascname table).
I'm trying to concat 2 variables Address and Payload. After that I want to send them with http to a server but I have 2 problems. When i try to concat the 2 variables with a delimiter ';' it doesn't work. Also sending the data of Payload or Address doesn't work. This is my code:
handle_rx(Gateway, #link{devaddr=DevAddr}=Link, #rxdata{port=Port, data= RxData }, RxQ)->
Data = base64:encode(RxData),
Devaddr = base64:encode(DevAddr),
TextAddr="Device address: ",
TextPayload="Payload: ",
Address = string:concat(TextAddr, Devaddr),
Payload = string:concat(TextPayload, Data),
Json=string:join([Address,Payload], "; "),
file:write_file("/tmp/foo.txt", io_lib:fwrite("~s.\n", [Json] )),
inets:start(),
ssl:start(),
httpc:request(post, {"http://192.168.0.121/apiv1/lorapacket/rx", [], "application/x-www-form-urlencoded", Address },[],[]),
ok;
handle_rx(_Gateway, _Link, RxData, _RxQ) ->
{error, {unexpected_data, RxData}}.
I have no errors that I can show you. When I write Address or Payload individually to the file it works but sending doesn't work...
Thank you for your help!
When i try to concat the 2 variables with a delimiter ';' it doesn't work.
5> string:join(["hello", <<"world">>], ";").
[104,101,108,108,111,59|<<"world">>]
6> string:join(["hello", "world"], ";").
"hello;world"
base64:encode() returns a binary, yet string:join() requires string arguments. You can do this:
7> string:join(["hello", binary_to_list(<<"world">>)], ";").
"hello;world"
Response to comment:
In erlang the string "abc" is equivalent to the list [97,98,99]. However, the binary syntax <<"abc">> is not equivalent to <<[97,98,99]>>, rather the binary syntax <<"abc">> is special short hand notation for the binary <<97, 98, 99>>.
Therefore, if you write:
Address = [97,98,99].
then the code:
Bin = <<Address>>.
after variable substitution becomes:
Bin = <<[97,98,99]>>.
and that isn't legal binary syntax.
If you need to convert a string/list contained in a variable, like Address, to a binary, you use list_to_binary(Address)--not <<Address>>.
In your code here:
Json = string:join([binary_to_list(<<Address>>),
binary_to_list(<<Payload>>)],
";").
Address and Payload were previously assigned the return value of string:concat(), which returns a string, so there is no reason to (attempt) to convert Address to a binary with <<Address>>, then immediately convert the binary back to a string with binary_to_list(). Instead, you would just write:
Json = string:join(Address, Payload, ";")
The problem with your original code is that you called string:concat() with a string as the first argument and a binary as the second argument--yet string:concat() takes two string arguments. You can use binary_to_list() to convert a binary to the string that you need for the second argument.
Sorry I'm new to Erlang
As with any language, you have to study the basics and write numerous toy examples before you can start writing code that actually does something.
You don't have to concatenate strings. It is called iolist and is one of best things in Erlang:
1> RxData = "Hello World!", DevAddr = "Earth",
1> Data = base64:encode(RxData), Devaddr = base64:encode(DevAddr),
1> TextAddr="Device address", TextPayload="Payload",
1> Json=["{'", TextAddr, "': '", Devaddr, "', '", TextPayload, "': '", Data, "'}"].
["{'","Device address","': '",<<"RWFydGg=">>,"', '",
"Payload","': '",<<"SGVsbG8gV29ybGQh">>,"'}"]
2> file:write_file("/tmp/foo.txt", Json).
ok
3> file:read_file("/tmp/foo.txt").
{ok,<<"{'Device address': 'RWFydGg=', 'Payload': 'SGVsbG8gV29ybGQh'}">>}
I am quite new pyparsing user and have missing match i don't understand:
Here is the text i would like to parse:
polraw="""
set policy id 800 from "Untrust" to "Trust" "IP_10.124.10.6" "MIP(10.0.2.175)" "TCP_1002" permit
set policy id 800
set dst-address "MIP(10.0.2.188)"
set service "TCP_1002-1005"
set log session-init
exit
set policy id 724 from "Trust" to "Untrust" "IP_10.16.14.28" "IP_10.24.10.6" "TCP_1002" permit
set policy id 724
set src-address "IP_10.162.14.38"
set dst-address "IP_10.3.28.38"
set service "TCP_1002-1005"
set log session-init
exit
set policy id 233 name "THE NAME is 527 ;" from "Untrust" to "Trust" "IP_10.24.108.6" "MIP(10.0.2.149)" "TCP_1002" permit
set policy id 233
set service "TCP_1002-1005"
set service "TCP_1006-1008"
set service "TCP_1786"
set log session-init
exit
"""
I setup grammar this way:
KPOL = Suppress(Keyword('set policy id'))
NUM = Regex(r'\d+')
KSVC = Suppress(Keyword('set service'))
KSRC = Suppress(Keyword('set src-address'))
KDST = Suppress(Keyword('set dst-address'))
SVC = dblQuotedString.setParseAction(lambda t: t[0].replace('"',''))
ADDR = dblQuotedString.setParseAction(lambda t: t[0].replace('"',''))
EXIT = Suppress(Keyword('exit'))
EOL = LineEnd().suppress()
P_SVC = KSVC + SVC + EOL
P_SRC = KSRC + ADDR + EOL
P_DST = KDST + ADDR + EOL
x = KPOL + NUM('PId') + EOL + Optional(ZeroOrMore(P_SVC)) + Optional(ZeroOrMore(P_SRC)) + Optional(ZeroOrMore(P_DST))
for z in x.searchString(polraw):
print z
Result set is such as
['800', 'MIP(10.0.2.188)']
['724', 'IP_10.162.14.38', 'IP_10.3.28.38']
['233', 'TCP_1002-1005', 'TCP_1006-1008', 'TCP_1786']
The 800 is missing service tag ???
What's wrong here.
Thanks by advance
Laurent
The problem you are seeing is that in your expression, DST's are only looked for after having skipped over optional SVC's and SRC's. You have a couple of options, I'll go through each so you can get a sense of what all is going on here.
(But first, there is no point in writing "Optional(ZeroOrMore(anything))" - ZeroOrMore already implies Optional, so I'm going to drop the Optional part in any of these choices.)
If you are going to get SVC's, SRC's, and DST's in any order, you could refactor your ZeroOrMore to accept any of the three data types, like this:
x = KPOL + NUM('PId') + EOL + ZeroOrMore(P_SVC|P_SRC|P_DST)
This will allow you to intermix different types of statements, and they will all get collected as part of the ZeroOrMore repetition.
If you want to keep these different types of statements in groups, then you can add a results name to each:
x = KPOL + NUM('PId') + EOL + ZeroOrMore(P_SVC("svc*")|
P_SRC("src*")|
P_DST("dst*"))
Note the trailing '*' on each name - this is equivalent to calling setResultsName with the listAllMatches argument equal to True. As each different expression is matched, the results for the different types will get collected into the "svc", "src", or "dst" results name. Calling z.dump() will list the tokens and the results names and their values, so you can see how this works.
set policy id 233
set service "TCP_1002-1005"
set dst-address "IP_10.3.28.38"
set service "TCP_1006-1008"
set service "TCP_1786"
set log session-init
exit
shows this for z.dump():
['233', 'TCP_1002-1005', 'IP_10.3.28.38', 'TCP_1006-1008', 'TCP_1786']
- PId: 233
- dst: [['IP_10.3.28.38']]
- svc: [['TCP_1002-1005'], ['TCP_1006-1008'], ['TCP_1786']]
If you wrap ungroup on the P_xxx expressions, maybe like this:
P_SVC,P_SRC,P_DST = (ungroup(expr) for expr in (P_SVC,P_SRC,P_DST))
then the output is even cleaner-looking:
['233', 'TCP_1002-1005', 'IP_10.3.28.38', 'TCP_1006-1008', 'TCP_1786']
- PId: 233
- dst: ['IP_10.3.28.38']
- svc: ['TCP_1002-1005', 'TCP_1006-1008', 'TCP_1786']
This is actually looking pretty good, but let me pass on one other option. There are a number of cases where parsers have to look for several sub-expressions in any order. Let's say they are A,B,C, and D. To accept these in any order, you could write something like OneOrMore(A|B|C|D), but this would accept multiple A's, or A, B, and C, but not D. The exhaustive/exhausting combinatorial explosion of (A+B+C+D) | (A+B+D+C) | etc. could be written, or you could maybe automate it with something like
from itertools import permutations
mixNmatch = MatchFirst(And(p) for p in permutations((A,B,C,D),4))
But there is a class in pyparsing called Each that allows to write the same kind of thing:
Each([A,B,C,D])
meaning "must have one each of A, B, C, and D, in any order". And like And, Or, NotAny, etc., there is an operator shortcut too:
A & B & C & D
which means the same thing.
If you want "must have A, B, and C, and optionally D", then write:
A & B & C & Optional(D)
and this will parse with the same kind of behavior, looking for A, B, C, and D, regardless of the incoming order, and whether D is last or mixed in with A, B, and C. You can also use OneOrMore and ZeroOrMore to indicate optional repetition of any of the expressions.
So you could write your expression as:
x = KPOL + NUM('PId') + EOL + (ZeroOrMore(P_SVC) &
ZeroOrMore(P_SRC) &
ZeroOrMore(P_DST))
I looked at using results names with this expression, and the ZeroOrMore's seem to be confusing things, maybe still a bug in how this is done. So you may have to reserve using Each for more basic cases like my A,B,C,D example. But I wanted to make you aware of it.
Some other notes on your parser:
dblQuotedString.setParseAction(lambda t: t[0].replace('"','')) is probably better written
dblQuotedString.setParseAction(removeQuotes). You don't have any embedded quotes in your examples, but it's good to be aware of where your assumptions might not translate to a future application. Here are a couple of ways of removing the defining quotes:
dblQuotedString.setParseAction(lambda t: t[0].replace('"',''))
print dblQuotedString.parseString(r'"This is an embedded quote \" and an ending quote \""')[0]
# prints 'This is an embedded quote \ and an ending quote \'
# removed leading and trailing "s, but also internal ones too, which are
# really part of the quoted string
dblQuotedString.setParseAction(lambda t: t[0].strip('"'))
print dblQuotedString.parseString(r'"This is an embedded quote \" and an ending quote \""')[0]
# prints 'This is an embedded quote \" and an ending quote \'
# removed leading and trailing "s, and leaves the one internal ones but strips off
# the escaped ending quote
dblQuotedString.setParseAction(removeQuotes)
print dblQuotedString.parseString(r'"This is an embedded quote \" and an ending quote \""')[0]
# prints 'This is an embedded quote \" and an ending quote \"'
# just removes leading and trailing " characters, leaves escaped "s in place
KPOL = Suppress(Keyword('set policy id')) is a bit fragile, as it will break if there are any extra spaces between 'set' and 'policy', or between 'policy' and 'id'. I usually define these kind of expressions by first defining all the keywords individually:
SET,POLICY,ID,SERVICE,SRC_ADDRESS,DST_ADDRESS,EXIT = map(Keyword,
"set policy id service src-address dst-address exit".split())
and then define the separate expressions using:
KSVC = Suppress(SET + SERVICE)
KSRC = Suppress(SET + SRC_ADDRESS)
KDST = Suppress(SET + DST_ADDRESS)
Now your parser will cleanly handle extra whitespace (or even comments!) between individual keywords in your expressions.