I do have two file which contains domains, after reading it with readlines(), I am getting two lists:
a = ['abc.com','cde.com','efg.com']
b = ['yabc.com','cde.com','abce.com','efg.com']
Now I need to find the common between the two.
No partial match allowed.( abc.com above has two partial match )
There is no order
Output should be : [''cde.com,'efg.com']
There is one problem I am handling manually, in some line of files we have multiple domain with "|" separator like:
abc.com|cde.com|efg.com which is treated as one string in and giving me list like:
['abc.com|cde.com|efg.com\n','xyz.com']
In this case again abc, cde and efg.com would be missed.
I tried set, intersection, two for loops, re.search, but not accurate result.
Related
How to rename file name from Pbs_d7_s2_juliam_08July2020_02_1_0_live singlets.fcs to Pbs_d7_s2.fcs
For multiple files keeping in mind that _juliam_08July2020_02_1_0_live singlets is not the same for all files?
It's a bit unclear what you're asking for, but it looks like you only want to keep the chunks before the third underscore. If so, you can tackle this with regular expressions. The regular expression
str_extract(input_string, "^(([^_])+_){3}")
will take out the first 3 blocks of characters (that aren't underscores) that end in underscores. The first ^ "anchors" the match to the beginning of the string, the "[^_]+_" matches any number of non-underscore characters before an underscore. The {3} does the preceding operation 3 times.
So for "Pbs_d7_s2_juliam_08July2020_02_1_0_live singlets.fcs" you'll end up with "Pbs_d7_s2_". Now you just replace the last underscore with ".fcs" like so
str_replace(modified string, "_$", ".fcs")
The $ "anchors" the characters that precede it to the end of the string so in this case it's replacing the last underscore. The full sequence is
string1<- "Pbs_d7_s2_juliam_08July2020_02_1_0_live singlets.fcs"
str_extract(string1, "^(([^_])+_){3}") %>%
str_replace("_$",".fcs")
[1] "Pbs_d7_s2.fcs"
Now let's assume your filenames are in a vector named stringvec.
output <- vector("character",length(stringvec))
for (i in seq_along(stringvec)) {
output[[i]] <- str_extract(stringvec[[i]],"^(([^_])+_){3}")%>%
str_replace("_$",".fcs")
}
output
I'm making some assumptions here - namely that the naming convention is the same for all of your files. If that's not true you'll need to find ways to modify the regex search pattern.
I recommend this answer How do I rename files using R? for replacing vectors of file names. If you have a vector of original file names you can use my for loop to generate a vector of new names, and then you can use the information in the link to replace one with the other. Perhaps there are other solutions not involving for loops.
It's me the newbie again with another messy file and folder situation(thanks to us biologiests): I got this directory containing a huge amount of .txt files (~900,000+), all the files have been previously handed with inconsistent naming format :(
For example, messy files in directory look like these:
ctrl_S978765_uns_dummy_00_none.txt
ctrl_S978765_3S_Cookie_00_none.txt
S59607_3S_goody_3M_V10.txt
ctrlnuc30-100_S3245678_DMSO_00_none.txt
ctrlRAP_S0846567_3S_Dex_none.txt
S6498432_2S_Fulra_30mM_V100.txt
.....
As you see the naming has no reliable consistency. What's important for me is the ID code embedded in them, such as S978765. Now I have got a list (100 ID codes) of these ID codes that I want.
The CSV file containing the list as below, mind you the list does have repetitive ID codes in the row due to different CLnumber value in the second columns:
ID code CLnumber
S978765 1
S978765 2
S306223 1
S897458 1
S514486 2
....
So I want to achieve below task: find all the messy named files using the code IDs by matching to my list. And copy them into a new directory.
I have thought of use list.files() to get all the .txt files and their names, then I got stuck at the next step at matching the code ID names, I know how to do it with one string, say "S978765", but if I do it one by one, this is almost just like manual digging the folder.
How could I feed the ID code names in column1 as a list and compare/match them with the messy file title names in the directory and then copy them into a new folder?
Many thanks,
ML
This works:
library(stringr)
# get this via list.files in your actual code
files <- c("ctrl_S978765_uns_dummy_00_none.txt",
"ctrl_S978765_3S_Cookie_00_none.txt",
"S59607_3S_goody_3M_V10.txt",
"ctrlnuc30-100_S3245678_DMSO_00_none.txt",
"ctrlRAP_S0846567_3S_Dex_none.txt",
"S6498432_2S_Fulra_30mM_V100.txt")
ids <- data.frame(`ID Code` = c("S978765", "S978765", "S306223", "S897458", "S514486"),
CLnumber = c(1, 2, 1, 1, 2),
stringsAsFactors = FALSE)
str_subset(files, paste(ids$ID.Code, collapse = "|"))
#> [1] "ctrl_S978765_uns_dummy_00_none.txt" "ctrl_S978765_3S_Cookie_00_none.txt"
str_subset takes a character vector and returns elements matching some pattern. In this case, the pattern is "S978765|S978765|S306223|S897458|S514486" (created by using paste), which is a regular expression that matches any of the ID codes separated by |. So we take files and keep only the elements that have a match in ID Code.
There are many other ways to do this, which may or may not be more clear. For example, you could pass ids$ID.Code directly to str_subset instead of constructing a regular expression via paste, but that would throw a warning about object lengths every time, which could get confusing (or cause problems if you get used to ignoring it and then ignore it in a different context where it matters). Another method would be to use purrr and keep, but while that might be a little bit more clear to write, it would be a lot more inefficient since it would mean making multiple passes over the files vector -- not relevant in this context, but possibly very relevant if you suddenly need to do this for hundreds of thousands of files and IDs.
You could use regex to extract the ID codes from the file name.
Here, I have used the pattern "S" followed by 5 or more numbers. Once we extract the ID_codes, we can compare them with the ones which we have in csv.
Assuming the csv is called df and the column name is ID_Codes we can use %in% to filter them.
We can then use file.copy to move files from one folder to another folder.
all_files <- list.files(path = '/Path/To/Folder', full.names = TRUE)
selected_files <- all_files[sub('.*(S\\d{5,}).*', '\\1', basename(all_files))
%in% unique(df$ID_Codes)]
file.copy(selected_files, 'new_path/for/files')
I have an R code that looks like this:
files <- list.files(get_directory())
files <- files[grepl("*.dat$", files)]
files
where get_directory() is a function I wrote that returns the current directory. So, I'm getting all the files with extension .dat in the directory that I want. But, my files are named as follows:
2^5-3^3-18-simul.dat
2^5-3^3-18-uniform.dat
2^7-3^4-33-simul.dat
2^7-3^4-33-uniform.dat
...
So, now I want to creates groups of 2 according to the first part, so I want 2^5-3^3-18-simul.dat and 2^5-3^3-18-uniform.dat to be one group, the other two files one group etc. While at a later stage, I need to loop through all the groups, and use the two files that are in the same group. Since, the filenames returned are already sorted, I do not think I need some fancy pattern matching here, I just need to get to group the elements of the string vector two by two as mentioned.
We can use sub to create a grouping variable to split the 'files'
split(files, sub("-[a-z].*", "", files))
#$`2^5-3^3-18`
#[1] "2^5-3^3-18-simul.dat" "2^5-3^3-18-uniform.dat"
#$`2^7-3^4-33`
#[1] "2^7-3^4-33-simul.dat" "2^7-3^4-33-uniform.dat"
data
files <- c("2^5-3^3-18-simul.dat", "2^5-3^3-18-uniform.dat",
"2^7-3^4-33-simul.dat", "2^7-3^4-33-uniform.dat")
I've got some kind of logfile I'd like to read and analyse. Unfortunately the files are saved in a pretty "ugly" way (with lots of special characters in between), so I'm not able to read in just the lines with each one being an entry. The only way to separate the different entries is using regular expressions, since the beginning of each entry follows a specified pattern.
My first approach was to identify the pattern in the character vector (I use read_file from the readr-package) and use the corresponding positions to split the vector with strsplit. Unfortunately the positions seem not always to match, since the result doesn't always correspond to the entries (I'd guess that there's a problem with the special characters).
A typical line of the file looks as follows:
16/10/2017, 21:51 - George: This is a typical entry here
The corresponding regular expressions looks as follows:
([[:digit:]]{2})/([[:digit:]]{2})/([[:digit:]]{4}), ([[:digit:]]{2}):([[:digit:]]{2}) - ([[:alpha:]]+):
The first thing I want is a data.frame with each line corresponding to a specific entry (in a next step I'd split the pattern into its different parts).
What I tried so far was the following:
regex.log = "([[:digit:]]{2})/([[:digit:]]{2})/([[:digit:]]{4}), ([[:digit:]]{2}):([[:digit:]]{2}) - ([[:alpha:]]+):"
log.regex = gregexpr(regex.log, file.log)[[1]]
log.splitted = substring(file.log, log.regex, log.regex[2:355]-1)
As can be seen this logfile has 355 entries. The first ones are separated correctly. How can I separate the character vector using a regular expression without loosing the information of the regular expression/pattern?
Use capturing and non-capturing groups to identify the parts you want to keep, and be sure to use anchors:
file.log = "16/10/2017, 21:51 - George: This is a typical entry here"
regex.log = "^((?:[[:digit:]]{2})\\/(?:[[:digit:]]{2})\\/(?:[[:digit:]]{4}), (?:[[:digit:]]{2}):(?:[[:digit:]]{2}) - (?:[[:alpha:]]+)): (.*)$"
gsub(regex.log,"\\1",file.log)
>> "16/10/2017, 21:51 - George"
gsub(regex.log,"\\2",file.log)
>> "This is a typical entry here"
I'm attempting to scrape a page that has about 10 columns using Ruby and Nokogiri, with most of the columns being pretty straightforward by having unique class names. However, some of them have class ids that seem to have long number strings appended to what would be the standard class name.
For example, gametimes are all picked up with .eventLine-time, team names with .team-name, but this particular one has, for example:
<div class="eventLine-book-value" id="eventLineOpener-118079-19-1522-1">-3 -120</div>
.eventLine-book-value is not specific to this column, so it's not useful. The 13 digits are different for every game, and trying something like:
def nodes_by_selector(filename,selector)
file = open(filename)
doc = Nokogiri::HTML(file)
doc.css(^selector)
end
Has left me with errors. I've seen ^ and ~ be used in other languages, but I'm new to this and I have tried searching for ways to pick up all data under id=eventLineOpener-XXXX to no avail.
To pick up all data under id=eventLineOpener-XXXX, you need to pass 'div[id*=eventLineOpener]' as the selector:
def nodes_by_selector(filename,selector)
file = open(filename)
doc = Nokogiri::HTML(file)
doc.css(selector) #doc.css('div[id*=eventLineOpener]')
end
The above method will return you an array of Nokogiri::XML::Element objects having id=eventLineOpener-XXXX.
Further, to extract the content of each of these Nokogiri::XML::Element objects, you need to iterate over each of these objects and use the text method on those objects. For example:
doc.css('div[id*=eventLineOpener]')[0].text