Is there any effective way to remove punctuation in text but keeping hyphenated expressions, such as "accident-prone"?
I used the following function to clean my text
clean.text = function(x)
{
# remove rt
x = gsub("rt ", "", x)
# remove at
x = gsub("#\\w+", "", x)
x = gsub("[[:punct:]]", "", x)
x = gsub("[[:digit:]]", "", x)
# remove http
x = gsub("http\\w+", "", x)
x = gsub("[ |\t]{2,}", "", x)
x = gsub("^ ", "", x)
x = gsub(" $", "", x)
x = str_replace_all(x, "[^[:alnum:][:space:]'-]", " ")
#return(x)
}
and apply it on hyphenated expressions that returned
my_text <- "accident-prone"
new_text <- clean.text(text)
new_text
[1] "accidentprone"
while my desired output is
"accident-prone"
I have referenced this thread but didn't find it worked on my situation. There must be some regex things that I haven't figured out. It will be really appreciated if someone could enlighten me on this.
Putting my two cents in, you could use (*SKIP)(*FAIL) with perl = TRUE and remove any non-word characters:
data <- c("my-test of #$%^&*", "accident-prone")
(gsub("(?<![^\\w])[- ](?=\\w)(*SKIP)(*FAIL)|\\W+", "", data, perl = TRUE))
Resulting in
[1] "my-test of" "accident-prone"
See a demo on regex101.com.
Here the idea is to match what you want to keep
(?<![^\\w])[- ](?=\\w)
# a whitespace or a dash between two word characters
# or at the very beginning of the string
let these fail with (*SKIP)(*FAIL) and put what you want to be removed on the right side of the alternation, in this case
\W+
effectively removing any non-word-characters not between word characters.
You'd need to provide more examples for testing though.
The :punct: set of characters includes the dash and you are removing them. You could make an alternate character class that omits the dash. You do need to pay special attention to the square-brackets placements and escape the double quote and the backslash:
(test <- gsub("[]!\"#$%&'()*+,./:;<=>?#[\\^_`{|}~]", "", "my-test of #$%^&*") )
[1] "my-test of "
The ?regex (help page) advises against using ranges. I investigated whether there might be any simplification using my local ASCII sequence of punctuation, but it quickly became obvious that was not the way to go for other reasons. There were 5 separate ranges, and the "]" was in the middle of one of them so there would have been 7 ranges to handle in addition to the "]" which needs to come first.
Related
I have a use case where
x <- "test - hello\r\n 1...124"
and I would like to obtain "test - hello 1...124. I am aware that I can use "gsub("[\r\n]", "", x)" for this specific case. However, I am wondering how to more generally remove any backslash followed by any symbol (e.g. using something like "\." and escaping the backslash). Examples that did not work are
gsub("\.", "", x) # error
gsub("\\.", "", x) # escapes "."?
gsub("\\\.", "", x) # error
gsub("\\\\.", "", x) # ??
...
Also I would be very thankful for an explanation as to why this is not working.
With the package strings you can use str_squish to remove not only leading and trailing whitespaces but also whitespaces somewhere in the middle.
x <- "test - hello\r\n 1...124"
stringr::str_squish(x)
#> [1] "test - hello 1...124"
gsub("\\r|\\n","", x)
gives the same result.
If you're looking to remove both newlines (or other escaped characters) and other strings that begin with \, you can just include both in the expression:
\r|\n|\t|\0|\\.
I am trying to clean some garbage out of some text. While doing this, I am assuming that any word that has a letter (any letter) repeated three or more times is garbage - and I want to remove it.
I've come up with this:
gsub(pattern = "[a-zA-Z]\\1\\1", replacement = "", string)
in which string is the character vector, but this doesn't work. Everything else I've tried might find the pattern, but it just removes the pattern, leaving a mess. I'm trying to remove the whole word with the pattern in it.
Any ideas?
You need
gsub("\\s*[[:alpha:]]*([[:alpha:]])\\1{2}[[:alpha:]]*", "", string)
gsub("\\s*\\p{L}*(\\p{L})\\1{2}\\p{L}*", "", string, perl=TRUE)
stringr::str_replace_all(string, "\\s*\\p{L}*(\\p{L})\\1{2}\\p{L}*", "")
See an R demo:
string <- "This is a baaaad unnnnecessary short word"
gsub("\\s*[[:alpha:]]*([[:alpha:]])\\1{2}[[:alpha:]]*", "", string)
gsub("\\s*\\p{L}*(\\p{L})\\1{2}\\p{L}*", "", string, perl=TRUE)
library(stringr)
str_replace_all(string, "\\s*\\p{L}*(\\p{L})\\1{2}\\p{L}*", "")
All yielding [1] "This is a short word".
See the regex demo. Regex details:
\s* - zero or more whitespaces
\p{L}* / [[:alpha:]]* - zero or more letters
(\p{L}) - Capturing group 1: any single letter
\1{2} - two occurrences of the same value as in Group 1
\p{L}* / [[:alpha:]]* - zero or more letters.
You need to assign a "capture group" to the [.] class by wrapping it in parens, since the \\1 needs something to reference:
gsub("([a-zA-Z])\\1\\1", "", "aabbbccdddee")
# [1] "aaccee"
Updated on OP comment:
Try this:
gsub("([A-Z]&|[a-z])\\1{2, }", "", "AAA")
[1] "AAA"
gsub("([A-Z]&|[a-z])\\1{2, }", "", "aabbbccdddee")
[1] "aaccee"
r2evans example with different regex:
gsub("(\\w)\\1{2, }", "", "aabbbccdddee")
[1] "aaccee"
I am analysing some tweets and I have written an basic emoji to text dictionary. I use the following to convert emoji's to r-encoded unicode;
df$text <- iconv(df$text, from = "latin1", to = "ascii", sub = "byte")
After that I swap the unicode to a text string that describes the emoji, for example <c2><ae> becomes 'copyright'
Problem is I have a lot of emoji's that aren't in the dictionary and I need to remove the strings that represent them. I can remove the <> symbols with "[[:punct:]]", "", but I need to get rid of the alpha numeric characters inside the <>'s too.
I was thinking something like
gsub("^<", "")
but i'm honestly stumped on how to find the < > symbols and remove anything found between them, or how to make a regex that finds < then removes it and the next 3 characters.
Appreciate any help
example
text <- ("have a <ed><a0><bd><ed><b8><80> day")
gsub("[[:punct:]]", "", text)
gives "have a eda0bdedb880 day"
but I want "have a day"
We can use a regex to match the < followed by characters that are not space ([^ ]+), ending in > and replace with blank ("")
gsub("\\<[^ ]+\\>\\s*", "", text, perl = TRUE)
#[1] "have a day"
I want to ignore the spaces and underscores in the beginning of a string in R.
I can write something like
txt <- gsub("^\\s+", "", txt)
txt <- gsub("^\\_+", "", txt)
But I think there could be an elegant solution
txt <- " 9PM 8-Oct-2014_0.335kwh "
txt <- gsub("^[\\s+|\\_+]", "", txt)
txt
The output should be "9PM 8-Oct-2014_0.335kwh ". But my code gives " 9PM 8-Oct-2014_0.335kwh ".
How can I fix it?
You could bundle the \s and the underscore only in a character class and use quantifier to repeat that 1+ times.
^[\s_]+
Regex demo
For example:
txt <- gsub("^[\\s_]+", "", txt, perl=TRUE)
Or as #Tim Biegeleisen points out in the comment, if only the first occurrence is being replaced you could use sub instead:
txt <- sub("[\\s_]+", "", txt, perl=TRUE)
Or using a POSIX character class
txt <- sub("[[:space:]_]+", "", txt)
More info about perl=TRUE and regular expressions used in R
R demo
The stringr packages offers some task specific functions with helpful names. In your original question you say you would like to remove whitespace and underscores from the start of your string, but in a comment you imply that you also wish to remove the same characters from the end of the same string. To that end, I'll include a few different options.
Given string s <- " \t_blah_ ", which contains whitespace (spaces and tabs) and underscores:
library(stringr)
# Remove whitespace and underscores at the start.
str_remove(s, "[\\s_]+")
# [1] "blah_ "
# Remove whitespace and underscores at the start and end.
str_remove_all(s, "[\\s_]+")
# [1] "blah"
In case you're looking to remove whitespace only – there are, after all, no underscores at the start or end of your example string – there are a couple of stringr functions that will help you keep things simple:
# `str_trim` trims whitespace (\s and \t) from either or both sides.
str_trim(s, side = "left")
# [1] "_blah_ "
str_trim(s, side = "right")
# [1] " \t_blah_"
str_trim(s, side = "both") # This is the default.
# [1] "_blah_"
# `str_squish` reduces repeated whitespace anywhere in string.
s <- " \t_blah blah_ "
str_squish(s)
# "_blah blah_"
The same pattern [\\s_]+ will also work in base R's sub or gsub, with some minor modifications, if that's your jam (see Thefourthbird`s answer).
You can use stringr as:
txt <- " 9PM 8-Oct-2014_0.335kwh "
library(stringr)
str_trim(txt)
[1] "9PM 8-Oct-2014_0.335kwh"
Or the trimws in Base R
trimws(txt)
[1] "9PM 8-Oct-2014_0.335kwh"
I am trying to remove only the / from any text using R. I have tried different approaches and I got mixed results.
This is the text I am dealing with s/p Left IOLI 3/9/04.
I am trying to produce an output like this sp Left IOLI 3/9/04.
Only strip the / in text and not numbers.
I have tried these four
gsub("\", "", str, fixed=T)
gsub("/", ".", str, fixed=T)
gsub("[^A-Za-z]", ".", str, perl =T)
str_replace( str, "/", "")
So far only gsub("[^A-Za-z]", ".", str, perl =T) worked. sucker stripped the / off of everything text numbers and everything. I just need the / from text be gone. Any help is much appreciated folks.
We can use regex lookarounds to remove the forward slash that are not between numbers.
gsub('(?<![0-9])/(?![0-9])', '', str, perl=TRUE)
#[1] "sp Left IOLI 3/9/04."
If we also need to remove / when either the left or right side contain non-numeric characters,
gsub('(?<![0-9])/|/(?![0-9])', '', str1, perl=TRUE)
#[1] "sp Left IOLI 3/9/04." "s12 45p sp Left"
data
str <- 's/p Left IOLI 3/9/04.'
str1 <- c(str, 's/12 45/p s/p Left')
An alternative way is to run multiple regexes. Demonstrated here using str_replace_all of package stringr, but obviously will work using base functions as well.
#First correct for / between 2 alphabets like s/p
mystring <- str_replace_all(mystring, "([a-zA-Z])/([a-zA-Z])", "\\1\\2")
#Next, correct for / between 1 alphabet and 1 number like s/12 or 45/p
mystring <- str_replace_all(mystring, "([a-zA-Z])/([\\d])", "\\1\\2")
mystring <- str_replace_all(mystring, "([\\d])/([a-zA-Z])", "\\1\\2")