What's the most elegant way to extract the keywords in a sentence of string?
I have a list of keywords from a CSV, and i want to predict exact match with keywords which is present in the string.
sapply(keywords, regexpr, String, ignore.case=FALSE)
I used the above code, but it gives approximate match too.
The problem is that a regular expressions don't know what 'words' are - they look for patterns. Consider the following keywords and string.
keywords = c("bad","good","sad","mad")
string = "Some good people live in the badlands which is maddeningly close to the sad harbor."
Here "bad" matches "badlands" because the pattern of "bad" is found in the first three characters. Same with "mad" and "maddeningly".
sapply(keywords, regexpr, string, ignore.case=FALSE)
#> bad good sad mad
#> 30 6 73 48
So, we need to modify the pattern to make it detect what we really want. The problem is knowing what we really want. If we want a distinct word, then we can add boundaries around the keywords. As Andre noted in the comments, the \b in regex is a word boundary.
sapply(paste("\\b",keywords,"\\b",sep=""), regexpr, string, ignore.case=FALSE)
#> \\bbad\\b \\bgood\\b \\bsad\\b \\bmad\\b
#> -1 6 73 -1
Note, what I did was use the paste function to stick an escaped \b before and after each keyword. This returns a no-match code for 'bad' and 'mad' but finds the whole word versions of 'good' and 'sad'.
If you wanted to find hyphenated characters, you'd need to modify the boundary matching portion of the expression.
Related
This question already has an answer here:
Reference - What does this regex mean?
(1 answer)
Closed 4 years ago.
I'm trying to understand a regular expression someone has written in the gsub() function.
I've never used regular expressions before seeing this code, and i have tried to work out how it's getting the final result with some googling, but i have hit a wall so to speak.
gsub('.*(.{2}$)', '\\1',"my big fluffy cat")
This code returns the last two characters in the given string. In the above example it would return "at". This is the expected result but from my brief foray into regular expressions i don't understand why this code does what it does.
What i understand is the '.*' means look for any character 0 or more times. So it's going to look at the entire string and this is what will be replaced.
The part in brackets looks for any two characters at the end of the string. It would make more sense to me if this part in brackets was in place of the '\1'. To me it would then read look at the entire string and replace it with the last two characters of that string.
All that does though is output the actual code as the replacement e.g ".{2}$".
Finally i don't understand why '\1' is in the replace part of the function. To me this is just saying replace the entire string with a single backslash and the number one. I say a single backslash because it's my understanding the first backslash is just there to make the second backslash a none special character.
For gsub there are two ways of using the function. The most common way is probably.
gsub("-","TEST","This is a - ")
which would return
This is a TEST
What this does is simply finds the matches in the regular expression and replaces it with the replacement string.
The second way to use gsub is the method in which you described. using \\1, \\2 or \\3...
What this does is looks at the first, second or third capture group in your regular expression.
A capture group is defined by anything inside the circular brackets ex: (capture_group_1)(capture_group_2)...
Explanation
Your analysis is correct.
What i understand is the '.*' means look for any character 0 or more times. So it's going to look at the entire string and this is what will be replaced.
The part in brackets looks for any two characters at the end of the string
The last two characters are placed in a capture group and we are simply replace the whole string with this capture group. Not replacing them with anything.
if it helps, check out the result of this expression.
gsub('(.*)(.{2}$)', 'Group 1: \\1, Group 2: \\2',"my big fluffy cat")
hope the examples can help you to understand it better:
Say we have a string foobarabcabcdef
.* matches whole string.
.*abc it matches: from the beginning matches any chars till the last abc (greedy matching), thus, it matches foobarabcabc
.*(...)$ matches the whole string as well, however, the last 3 chars were groupped. Without the () , the matched string will have a default group, group0, the () will be group1, 2, 3.... think about .*(...)(...)(...)$ so we have:
group 0 : whole string
group 1 : "abc" the first "abc"
group 2 : "abc" the 2nd "abc"
group 3 : "def" the last 3 chars
So back to your example, the \\1 is a reference to group. What it does is: "replace the whole string by the matched text in group1" That is, the .{2}$ part is the replacement.
If you don't understand the backslashs, you have to reference the syntax of r, I cannot tell more. It is all about escaping.
Important part of that regular expression are brackets, that's something called "capturing group".
Regular expression .*(.{2}$) says - match anything and capture last 2 characters at the line. Replacement \\1 is referencing to that group, so it will replace whole match with captured group, which are last two characters in this case.
I'm working in R, trying to prepare text documents for analysis. Each document is stored in a column (aptly named, "document") of dataframe called "metaDataFrame." The documents are strings containing articles and their BibTex citation info. Data frame looks like this:
[1] filename document doc_number
[2] lithuania2016 Commentary highlights Estonian... 1
[3] lithuania2016 Norwegian police, immigration ... 2
[4] lithuania2016 Portugal to deply over 1,000 m... 3
I want to extract the BibTex information from each document into a new column. The citation information begins with "Credit:" but some articles contain multiple "Credit:" instances, so I need to extract all of the text after the last instance. Unfortunately, the string is only sometimes preceded by a new line.
My solution so far has been to find all of the instances of the string and save the location of the last instance of "Credit:" in each document in a list:
locate.last.credit <- lapply(gregexpr('Credit:', metaDataFrame$document), tail, 1)
This provides a list of integer locations of the last "Credit:" string in each document or a value of "-1" where no instance is found. (Those missing values pose a separate but related problem I think I can tackle after resolving this issue).
I've tried variations of strsplit, substr, stri_match_last, and rm_between...but can't figure out a way to use the character position in lieu of regular expression to extract this part of the string.
How can I use the location of characters to manipulate a string instead of regular expressions? Is there a better approach to this (perhaps with regex)?
How about like this:
test_string <- " Portugal to deply over 1,000 m Credit: mike jones Credit: this is the bibliography"
gsub(".*Credit:\\s*(.*)", "\\1", test_string, ignore.case = TRUE)
[1] "this is the bibliography"
The Regex pattern is looking for Credit, but because it's preceeded by .*, it's going to find the last instance of the word (if you wanted the first instance of Credit, you'd use .*?). \\s* matches 0 or more white space characters after credit and before the rest of the text. We then capture the remainder of each document in (.*), as capture group 1. And we return \\1. Also, I use ignore.case = TRUE so credit, CREDIT, and Credit will all be matched.
And with your object it would be:
gsub(".*Credit:\\s*(.*)", "\\1", metaDataFrame$document, ignore.case = TRUE)
I am scraping PDFs for data and am trying to search for a numeric character (1:9) that is either of length 1 or 2. Unfortunately the value I am after changes position across the PDFs so I cannot simply call the index of the value and assign it to a variable.
I have tried many regex functions and can get numbers out of the list, but cannot seem to implement the argument to only pull numbers of the specific length.
# Data comes in as a long string
Test<-("82026-424 82026-424 1 CSX10 Store Room 75.74 75.74")
# Seperate data into individual pieces with str_split
Split_Test<-str_split(Test[1],"\\s+")
# We can easily unlist it with the following code (Not sure if needed)
Test_Unlisted<-unlist(Split_Test)
> Test_Unlisted
[1] "82026-424" "82026-424" "1" "CSX10" "Store" "Room"
[8] "75.74" "75.74"
My desired outcome would be to get the "1" out of the character list, and then if the value was "20" also be able to recognize that.
The best logic I can think of in code exists below, but this does not work.:
Test_Final<-str_match(Test_Unlisted, "\\d|\\d\\d")
Using this code I can grab anything of length=1, but it is not guaranteed to be a character:
Test_Final<-which(sapply(Test_Unlisted, nchar)==1)
Thanks for all the help!
You need to use
Test<-("82026-424 82026-424 1 CSX10 Store Room 75.74 75.74, 20")
regmatches(Test, gregexpr("\\b(?<!\\d\\.)\\d{1,2}\\b(?!\\.\\d)", Test, perl=TRUE))
See the regex demo and the regex demo.
Details
\b - a word boundary
(?<!\d\.) - a negative lookbehind that fails the match if, immediately to the left of the current location, there is a digit and a dot
\d{1,2} - 1 or 2 digits
\b - a word boundary
(?!\.\d) - a negative lookahead that fails the match if, immediately to the right of the current location, there is a dot and a digit.
Note that due to the lookarounds used in the pattern, the regex should be passed to the PCRE regex engine, hence the perl=TRUE argument is required.
With stringr that is ICU regex engine powered, you may use
library(stringr)
str_extract_all(Test, "\\b(?<!\\d\\.)\\d{1,2}\\b(?!\\.\\d)")
I'm trying to extract UK postcodes from address strings in R, using the regular expression provided by the UK government here.
Here is my function:
address_to_postcode <- function(addresses) {
# 1. Convert addresses to upper case
addresses = toupper(addresses)
# 2. Regular expression for UK postcodes:
pcd_regex = "[Gg][Ii][Rr] 0[Aa]{2})|((([A-Za-z][0-9]{1,2})|(([A-Za-z][A-Ha-hJ-Yj-y][0-9]{1,2})|(([A-Za-z][0-9][A-Za-z])|([A-Za-z][A-Ha-hJ-Yj-y][0-9]?[A-Za-z])))) {0,1}[0-9][A-Za-z]{2})"
# 3. Check if a postcode is present in each address or not (return TRUE if present, else FALSE)
present <- grepl(pcd_regex, addresses)
# 4. Extract postcodes matching the regular expression for a valid UK postcode
postcodes <- regmatches(addresses, regexpr(pcd_regex, addresses))
# 5. Return NA where an address does not contain a (valid format) UK postcode
postcodes_out <- list()
postcodes_out[present] <- postcodes
postcodes_out[!present] <- NA
# 6. Return the results in a vector (should be same length as input vector)
return(do.call(c, postcodes_out))
}
According to the guidance document, the logic this regular expression looks for is as follows:
"GIR 0AA" OR One letter followed by either one or two numbers OR One letter followed by a second letter that must be one of
ABCDEFGHJ KLMNOPQRSTUVWXY (i.e..not I) and then followed by either one
or two numbers OR One letter followed by one number and then another
letter OR A two part post code where the first part must be One letter
followed by a second letter that must be one of ABCDEFGH
JKLMNOPQRSTUVWXY (i.e..not I) and then followed by one number and
optionally a further letter after that AND The second part (separated
by a space from the first part) must be One number followed by two
letters. A combination of upper and lower case characters is allowed.
Note: the length is determined by the regular expression and is
between 2 and 8 characters.
My problem is that this logic is not completely preserved when using the regular expression without the ^ and $ anchors (as I have to do in this scenario because the postcode could be anywhere within the address strings); what I'm struggling with is how to preserve the order and number of characters for each segment in a partial (as opposed to complete) string match.
Consider the following example:
> address_to_postcode("1A noplace road, random city, NR1 2PK, UK")
[1] "NR1 2PK"
According to the logic in the guideline, the second letter in the postcode cannot be 'z' (and there are some other exclusions too); however look what happens when I add a 'z':
> address_to_postcode("1A noplace road, random city, NZ1 2PK, UK")
[1] "Z1 2PK"
... whereas in this case I would expect the output to be NA.
Adding the anchors (for a different usage case) doesn't seem to help as the 'z' is still accepted even though it is in the wrong place:
> grepl("^[Gg][Ii][Rr] 0[Aa]{2})|((([A-Za-z][0-9]{1,2})|(([A-Za-z][A-Ha-hJ-Yj-y][0-9]{1,2})|(([A-Za-z][0-9][A-Za-z])|([A-Za-z][A-Ha-hJ-Yj-y][0-9]?[A-Za-z])))) {0,1}[0-9][A-Za-z]{2})$", "NZ1 2PK")
[1] TRUE
Two questions:
Have I misunderstood the logic of the regular expression and
If not, how can I correct it (i.e. why aren't the specified letter
and character ranges exclusive to their position within the regular expression)?
Edit
Since posting this answer, I dug deeper into the UK government's regex and found even more problems. I posted another answer here that describes all the issues and provides alternatives to their poorly formatted regex.
Note
Please note that I'm posting the raw regex here. You'll need to escape certain characters (like backslashes \) when porting to r.
Issues
You have many issues here, all of which are caused by whoever created the document you're retrieving your regex from or the coder that created it.
1. The space character
My guess is that when you copied the regular expression from the link you provided it converted the space character into a newline character and you removed it (that's exactly what I did at first). You need to, instead, change it to a space character.
^([Gg][Ii][Rr] 0[Aa]{2})|((([A-Za-z][0-9]{1,2})|(([A-Za-z][A-Ha-hJ-Yj-y][0-9]{1,2})|(([AZa-z][0-9][A-Za-z])|([A-Za-z][A-Ha-hJ-Yj-y][0-9]?[A-Za-z])))) [0-9][A-Za-z]{2})$
here ^
2. Boundaries
You need to remove the anchors ^ and $ as these indicate start and end of line. Instead, wrap your regex in (?:) and place a \b (word boundary) on either end as the following shows. In fact, the regex in the documentation is incorrect (see Side note for more information) as it will fail to anchor the pattern properly.
See regex in use here
\b(?:([Gg][Ii][Rr] 0[Aa]{2})|((([A-Za-z][0-9]{1,2})|(([A-Za-z][A-Ha-hJ-Yj-y][0-9]{1,2})|(([AZa-z][0-9][A-Za-z])|([A-Za-z][A-Ha-hJ-Yj-y][0-9]?[A-Za-z])))) [0-9][A-Za-z]{2}))\b
^^^^^ ^^^
3. Character class oversight
There's a missing - in the character class as pointed out by #deadcrab in his answer here.
\b(?:([Gg][Ii][Rr] 0[Aa]{2})|((([A-Za-z][0-9]{1,2})|(([A-Za-z][A-Ha-hJ-Yj-y][0-9]{1,2})|(([A-Za-z][0-9][A-Za-z])|([A-Za-z][A-Ha-hJ-Yj-y][0-9]?[A-Za-z])))) [0-9][A-Za-z]{2}))\b
^
4. They made the wrong character class optional!
In the documentation it clearly states:
A two part post code where the first part must be:
One letter followed by a second letter that must be one of ABCDEFGHJKLMNOPQRSTUVWXY (i.e..not I) and then followed by one number and optionally a further letter after that
They made the wrong character class optional!
\b(?:([Gg][Ii][Rr] 0[Aa]{2})|((([A-Za-z][0-9]{1,2})|(([A-Za-z][A-Ha-hJ-Yj-y][0-9]{1,2})|(([A-Za-z][0-9][A-Za-z])|([A-Za-z][A-Ha-hJ-Yj-y][0-9]?[A-Za-z])))) [0-9][A-Za-z]{2}))\b
^^^^^^
it should be this one ^^^^^^^^
5. The whole thing is just awful...
There are so many things wrong with this regex that I just decided to rewrite it. It can very easily be simplified to perform a fraction of the steps it currently takes to match text.
\b(?:[A-Za-z][A-HJ-Ya-hj-y]?[0-9][0-9A-Za-z]? [0-9][A-Za-z]{2}|[Gg][Ii][Rr] 0[Aa]{2})\b
Answer
As mentioned in the comments below my answer, some postcodes are missing the space character. For missing spaces in the postcodes (e.g. NR12PK), simply add a ? after the spaces as shown in the regex below:
\b(?:[A-Za-z][A-HJ-Ya-hj-y]?[0-9][0-9A-Za-z]? ?[0-9][A-Za-z]{2}|[Gg][Ii][Rr] ?0[Aa]{2})\b
^^ ^^
You may also shorten the regex above with the following and use the case-insensitive flag (ignore.case(pattern) or ignore_case = TRUE in r, depending on the method used.):
\b(?:[A-Z][A-HJ-Y]?[0-9][0-9A-Z]? ?[0-9][A-Z]{2}|GIR ?0A{2})\b
Note
Please note that regular expressions only validate the possible format(s) of a string and cannot actually identify whether or not a postcode legitimately exists. For this, you should use an API. There are also some edge-cases where this regex will not properly match valid postcodes. For a list of these postcodes, please see this Wikipedia article.
The regex below additionally matches the following (make it case-insensitive to match lowercase variants as well):
British Overseas Territories
British Forces Post Office
Although they've recently changed it to align with the British postcode system to BF, followed by a number (starting with BF1), they're considered optional alternative postcodes
Special cases outlined in that article (as well as SAN TA1 - a valid postcode for Santa!)
See this regex in use here.
\b(?:(?:[A-Z][A-HJ-Y]?[0-9][0-9A-Z]?|ASCN|STHL|TDCU|BBND|[BFS]IQ{2}|GX11|PCRN|TKCA) ?[0-9][A-Z]{2}|GIR ?0A{2}|SAN ?TA1|AI-?[0-9]{4}|BFPO[ -]?[0-9]{2,3}|MSR[ -]?1(?:1[12]|[23][135])0|VG[ -]?11[1-6]0|[A-Z]{2} ? [0-9]{2}|KY[1-3][ -]?[0-2][0-9]{3})\b
I would also recommend anyone implementing this answer to read this StackOverflow question titled UK Postcode Regex (Comprehensive).
Side note
The documentation you linked to (Bulk Data Transfer: Additional Validation for CAS Upload - Section 3. UK Postcode Regular Expression) actually has an improperly written regular expression.
As mentioned in the Issues section, they should have:
Wrapped the entire expression in (?:) and placed the anchors around the non-capturing group. Their regular expression, as it stands, will fail in for some cases as seen here.
The regular expression is also missing - in one of the character classes
It also made the wrong character class optional.
here is my regular expression
txt="0288, Bishopsgate, London Borough of Tower Hamlets, London, Greater London, England, EC2M 4QP, United Kingdom"
matches=re.findall(r'[A-Z]{1,2}[0-9][A-Z0-9]? [0-9][ABD-HJLNP-UW-Z]{2}', txt)
I want to gsub a string that contains only characters and white spaces, for example the string "to delete". I have tried this:
gsub('[^[:alpha:]$]',NA, "to delete", ignore.case=T)
But I get an NA also when the string contains digits, for example:
gsub('[^[:alpha:]$]',NA, "to 1 delete", ignore.case=T)
Anybody could tell me what I am doing wrong? Thanks.
Your regex only tests for a single unanchored bracket expression. This means that any string that has any character which matches the bracket expression will match the regex.
Your bracket expression tests for "not alphabetic not dollar". This matches many things, including spaces, digits, and all punctuation characters other than dollar.
It sounds like you want to match only strings which consist in their entirety of only alphabetic and whitespace characters. To achieve that you need an anchored multiplied bracket expression.
Also, you don't need gsub() for this; you only need sub(), since a regex that can only match the entirety of the input string cannot match multiple times within the input string.
Also, you don't need ignore.case=T, since the [:alpha:] character class already matches all alphabetics, regardless of letter-case.
regex <- '^[[:alpha:][:space:]]*$';
sub(regex,NA,'to delete');
## [1] NA
sub(regex,NA,'to 1 delete');
## [1] "to 1 delete"