Does anyone know how to replicate the (pg_trgm) postgres trigram similarity score from the similarity(text, text) function in R? I am using the stringdist package and would rather use R to calculate these on a matrix of text strings in a .csv file than run a bunch of postgresql quires.
Running similarity(string1, string2) in postgres give me a number score between 0 and 1.
I tired using the stringdist package to get a score but I think I still need to divide the code below by something.
stringdist(string1, string2, method="qgram",q = 3 )
Is there a way to replicate the pg_trgm score with the stringdist package or another way to do this in R?
An example would be getting the similarity score between the description of a book and the description of a genre like science fiction. For example, if I have two book descriptions and the using the similarity score of
book 1 = "Area X has been cut off from the rest of the continent for decades. Nature has reclaimed the last vestiges of human civilization. The first expedition returned with reports of a pristine, Edenic landscape; the second expedition ended in mass suicide, the third expedition in a hail of gunfire as its members turned on one another. The members of the eleventh expedition returned as shadows of their former selves, and within weeks, all had died of cancer. In Annihilation, the first volume of Jeff VanderMeer's Southern Reach trilogy, we join the twelfth expedition.
The group is made up of four women: an anthropologist; a surveyor; a psychologist, the de facto leader; and our narrator, a biologist. Their mission is to map the terrain, record all observations of their surroundings and of one anotioner, and, above all, avoid being contaminated by Area X itself.
They arrive expecting the unexpected, and Area X delivers—they discover a massive topographic anomaly and life forms that surpass understanding—but it’s the surprises that came across the border with them and the secrets the expedition members are keeping from one another that change everything."
book 2= "From Wall Street to Main Street, John Brooks, longtime contributor to the New Yorker, brings to life in vivid fashion twelve classic and timeless tales of corporate and financial life in America
What do the $350 million Ford Motor Company disaster known as the Edsel, the fast and incredible rise of Xerox, and the unbelievable scandals at GE and Texas Gulf Sulphur have in common? Each is an example of how an iconic company was defined by a particular moment of fame or notoriety; these notable and fascinating accounts are as relevant today to understanding the intricacies of corporate life as they were when the events happened.
Stories about Wall Street are infused with drama and adventure and reveal the machinations and volatile nature of the world of finance. John Brooks’s insightful reportage is so full of personality and critical detail that whether he is looking at the astounding market crash of 1962, the collapse of a well-known brokerage firm, or the bold attempt by American bankers to save the British pound, one gets the sense that history repeats itself.
Five additional stories on equally fascinating subjects round out this wonderful collection that will both entertain and inform readers . . . Business Adventures is truly financial journalism at its liveliest and best."
genre 1 = "Science fiction is a genre of fiction dealing with imaginative content such as futuristic settings, futuristic science and technology, space travel, time travel, faster than light travel, parallel universes, and extraterrestrial life. It often explores the potential consequences of scientific and other innovations, and has been called a "literature of ideas".[1] Authors commonly use science fiction as a framework to explore politics, identity, desire, morality, social structure, and other literary themes."
How can I get a similarity score for the description of each book against the description of the science fiction genre like pg_trgm using an R script?
How about something like this?
library(textcat)
?textcat_xdist
# Compute cross-distances between collections of n-gram profiles.
round(textcat_xdist(
list(
text1="hello there",
text2="why hello there",
text3="totally different"
),
method="cosine"),
3)
# text1 text2 text3
#text1 0.000 0.078 0.731
#text2 0.078 0.000 0.739
#text3 0.731 0.739 0.000
Related
I have large text documents (150K lines per document; 160 documents). Then I read them in as a large VCorpus and convert them to a dataframe it runs quite quickly. Now, I want to separate each sentence in rows and remove those that do not contain a certain keyword. Then I run the code R crashes. If I try it with one document the code runs for approximately 10 minutes.
The data (text) looks like this (just with much more text):
| Company1 | Company1 hat engaged in child safety in South Africa. The Community rewards this with great recognition. Charity is a big part of the companies culture.
| Company2 | Company2 opened up several factories in the that do not pay the minimum wage. Affordable Housing is not one of thair priorities.
There is also a small example structure below.
library(readr)
library(qdap)
library(tm)
corp <- VCorpus(DirSource("test"))
text <- as.data.frame(corp)
names(text)[1] <- 'doc_id'
names(text)[2] <- 'text'
text <- dput(text)
text <- structure(list(doc_id = c("Company1.txt", "Company2.txt", "Company3.txt"
), text = c("Acucap's market cap on listing in 2002 was a meagre R372m, which would make it one of the smallest stocks in the real estate sector in today's terms. This may be a reflection of how the real estate sector of the JSE has evolved over the past 10 years. But, it is also amazing that Acucap - at a current market cap of more than R7bn - has managed to retain much of the same level of entrepreneurial spirit as at the time of its listing. Still at the helm is sector veteran Paul Theodosiou. The team has grown since listing. The core operational supporting team is Jonathan Rens, Gavin Jones and Craig Kotze, and the finance team is led by founding chief financial officer Baden Marlow. Acucap's physical portfolio is concentrated in retail, which comprises 74%, offices make up 24% and industrial takes the remaining 2%. Some of the larger retail properties include Festival Mall (the largest single asset in the portfolio) in Kempton Park, Bayside Mall in Tableview and Keywest shopping centre in Krugersdorp. The office portfolio includes assets like Mowbray Golf Park in Pinelands in the Cape, the Microsoft offices in Bryanston and 82 Grayston Drive in the Sandton CBD. Though small in Acucap's portfolio, the industrial portfolio is of a high quality and offers good scope for growth as expansion continues at the N1 Business Park in Midrand and the Montague Business Park in Cape Town. Acucap's other investments include a 17,2% shareholding in Sycom Property Fund (a fellow listed real estate property company) as well as a 100% shareholding in Sycom's management company. Sycom's other significant shareholder is Hyprop Investments. The latter's 33,9% stake in Sycom has been the subject of much speculation over the past few years as the ownership structure has created somewhat of an impasse. Acucap has, however, stated its strategic intent to retain Sycom as a separate fund. Company1 hat engaged in child safety in South Africa. The Community rewards this with great recognition. Charity is a big part of the companies culture.",
"Australian copper producer Aditya Birla Minerals Ltd. said June 16 that there is a risk that the overall reserves at its Nifty copper mine in Western Australia may be adversely affected due to the effects of an earlier ground collapse at the mine. The company also said it will be unable to provide its annual reserve update until all investigative activities are finalized. Aditya Birla is expecting site costs for the June quarter to be in the range of A$17 million to A$19 million, higher than the earlier estimate of A$12 million to A$15 million due to higher-than-expected mine activity, unplanned maintenance on critical infrastructure and additional employee costs. The company was forced to halt operations at the Nifty mine in March following suspected underground subsidence, which led to the standing down of 350 mine employees until at least July 15. Phase two of probe drilling, which is being undertaken as part of efforts to assess mine safety, is progressing ahead of schedule and is now expected to be completed by the end of June instead of mid-July. Aditya Birla said preliminary observations from this second phase are similar to those from phase one drilling, in that there is less water being intersected between levels 16 and 20 than previously expected. Areas with potential to self-propagate into new sinkholes are being reviewed and, as Aditya Birla expected, the rock mass strength has deteriorated on top of mined-out areas. Management has begun an initial review of the results to identify new gaps and/or the need for additional confirmatory drilling. The first phase of the seismic system installation has been commissioned and is fully functional, albeit limited in coverage to upper parts of the mine. The pit has been dewatered with residual mud left at the pit sump. Aditya Birla added that while surface cracking has continued to develop in the area affected by the sinkhole, this was expected and does not present a hazard. However, washouts will occur after heavy rainfall, leading to widening and deepening of the cracks.",
"In preparation for the potential completion of the merger with AGL Resources Inc., the board of directors of Nicor Inc. announced a special pro rata dividend. The dividend is contingent upon the merger being completed prior to Nicor's next scheduled dividend record date, Dec. 31, to ensure that \"shareholders continue to receive a dividend at the current rate until the closing of the merger,\" the company said. In a Nov. 1 news release, Nicor said its board of directors declared a pro rata dividend of 0.5 cent per share per day from Oct. 1 until and including the day before the merger effective date. The dividend is the daily equivalent of the current quarterly dividend rate of 46.5 cents per share. It will be paid to Nicor shareholders of record at the close of business on the day immediately before the effective merger date, which is expected in the fourth quarter. \"The dividend will be paid as soon as practical following the completion of the merger,\" Nicor said. \"This pro rata dividend is in addition to the previously announced Nov. 1 dividend that was paid to shareholders of record Sept. 30. Following the merger closing, AGL Resources is expected to pay a dividend at the rate of $0.004945055 per share, per day for the remainder of the current quarterly dividend period. These dividend payment scenarios, together with a similar plan announced today by AGL Resources, will synchronize the companies' dividends as of the merger effective date in accordance with the merger agreement.\" If the merger is not completed by Dec. 31, Nicor shareholders of record Dec. 31 will receive the regular quarterly dividend of 46.5 cents per share, payable Feb. 1, 2012, and a new pro rata dividend will be announced to ensure that shareholders receive a dividend at the current rate until the merger is completed. Company2 opened up several factories in the that do not pay the minimum wage. Affordable Housing is not one of thair priorities."
)), class = "data.frame", row.names = c(NA, 3L))
options(stringsAsFactors = FALSE)
Sys.setlocale('LC_ALL','C')
library(dplyr)
library(tidyr)
sckeywords <- c("Affordable Housing", "Benefit The Masses",
"Charitability", "Charitable", "Charitably", " Charities ", " Charity ")
pat <- paste0(sckeywords, collapse = '|')
text2 <- (text) %>%
separate_rows(text, sep = '\\.\\s*') %>%
slice({
tmp <- grep(pat, text, ignore.case = TRUE)
sort(unique(c(tmp-1, tmp, tmp + 1)))
})
Can I make it run faster somehow without needing more hardware capacity?
I have 16 GB of RAM and a 4 core CPU (i5-10210U).
There are several ways to increase the speed:
I would use data.table instead of dplyr for this amount of data
text <- structure(list(doc_id = c("Company1.txt", "Company2.txt", "Company3.txt"
), text = c("Acucap's market cap on listing in 2002 was a meagre R372m, which would make it one of the smallest stocks in the real estate sector in today's terms. This may be a reflection of how the real estate sector of the JSE has evolved over the past 10 years. But, it is also amazing that Acucap - at a current market cap of more than R7bn - has managed to retain much of the same level of entrepreneurial spirit as at the time of its listing. Still at the helm is sector veteran Paul Theodosiou. The team has grown since listing. The core operational supporting team is Jonathan Rens, Gavin Jones and Craig Kotze, and the finance team is led by founding chief financial officer Baden Marlow. Acucap's physical portfolio is concentrated in retail, which comprises 74%, offices make up 24% and industrial takes the remaining 2%. Some of the larger retail properties include Festival Mall (the largest single asset in the portfolio) in Kempton Park, Bayside Mall in Tableview and Keywest shopping centre in Krugersdorp. The office portfolio includes assets like Mowbray Golf Park in Pinelands in the Cape, the Microsoft offices in Bryanston and 82 Grayston Drive in the Sandton CBD. Though small in Acucap's portfolio, the industrial portfolio is of a high quality and offers good scope for growth as expansion continues at the N1 Business Park in Midrand and the Montague Business Park in Cape Town. Acucap's other investments include a 17,2% shareholding in Sycom Property Fund (a fellow listed real estate property company) as well as a 100% shareholding in Sycom's management company. Sycom's other significant shareholder is Hyprop Investments. The latter's 33,9% stake in Sycom has been the subject of much speculation over the past few years as the ownership structure has created somewhat of an impasse. Acucap has, however, stated its strategic intent to retain Sycom as a separate fund. Company1 hat engaged in child safety in South Africa. The Community rewards this with great recognition. Charity is a big part of the companies culture.",
"Australian copper producer Aditya Birla Minerals Ltd. said June 16 that there is a risk that the overall reserves at its Nifty copper mine in Western Australia may be adversely affected due to the effects of an earlier ground collapse at the mine. The company also said it will be unable to provide its annual reserve update until all investigative activities are finalized. Aditya Birla is expecting site costs for the June quarter to be in the range of A$17 million to A$19 million, higher than the earlier estimate of A$12 million to A$15 million due to higher-than-expected mine activity, unplanned maintenance on critical infrastructure and additional employee costs. The company was forced to halt operations at the Nifty mine in March following suspected underground subsidence, which led to the standing down of 350 mine employees until at least July 15. Phase two of probe drilling, which is being undertaken as part of efforts to assess mine safety, is progressing ahead of schedule and is now expected to be completed by the end of June instead of mid-July. Aditya Birla said preliminary observations from this second phase are similar to those from phase one drilling, in that there is less water being intersected between levels 16 and 20 than previously expected. Areas with potential to self-propagate into new sinkholes are being reviewed and, as Aditya Birla expected, the rock mass strength has deteriorated on top of mined-out areas. Management has begun an initial review of the results to identify new gaps and/or the need for additional confirmatory drilling. The first phase of the seismic system installation has been commissioned and is fully functional, albeit limited in coverage to upper parts of the mine. The pit has been dewatered with residual mud left at the pit sump. Aditya Birla added that while surface cracking has continued to develop in the area affected by the sinkhole, this was expected and does not present a hazard. However, washouts will occur after heavy rainfall, leading to widening and deepening of the cracks.",
"In preparation for the potential completion of the merger with AGL Resources Inc., the board of directors of Nicor Inc. announced a special pro rata dividend. The dividend is contingent upon the merger being completed prior to Nicor's next scheduled dividend record date, Dec. 31, to ensure that \"shareholders continue to receive a dividend at the current rate until the closing of the merger,\" the company said. In a Nov. 1 news release, Nicor said its board of directors declared a pro rata dividend of 0.5 cent per share per day from Oct. 1 until and including the day before the merger effective date. The dividend is the daily equivalent of the current quarterly dividend rate of 46.5 cents per share. It will be paid to Nicor shareholders of record at the close of business on the day immediately before the effective merger date, which is expected in the fourth quarter. \"The dividend will be paid as soon as practical following the completion of the merger,\" Nicor said. \"This pro rata dividend is in addition to the previously announced Nov. 1 dividend that was paid to shareholders of record Sept. 30. Following the merger closing, AGL Resources is expected to pay a dividend at the rate of $0.004945055 per share, per day for the remainder of the current quarterly dividend period. These dividend payment scenarios, together with a similar plan announced today by AGL Resources, will synchronize the companies' dividends as of the merger effective date in accordance with the merger agreement.\" If the merger is not completed by Dec. 31, Nicor shareholders of record Dec. 31 will receive the regular quarterly dividend of 46.5 cents per share, payable Feb. 1, 2012, and a new pro rata dividend will be announced to ensure that shareholders receive a dividend at the current rate until the merger is completed. Company2 opened up several factories in the that do not pay the minimum wage. Affordable Housing is not one of thair priorities."
)), class = "data.frame", row.names = c(NA, 3L))
library(data.table)
sckeywords <- c("Affordable Housing", "Benefit The Masses",
"Charitability", "Charitable", "Charitably", " Charities ", " Charity ")
pat <- paste0(sckeywords, collapse = '|')
setDT(text)
text <- text[, .(text = unlist(tstrsplit(text, "\\.\\s*", type.convert = TRUE))),
by = "doc_id"]
text <- text[grepl(pat, text)]
You can try to speed up the regex. One possibility is to create a trie, e.g. with trieregex. In your case, it would be:
trie_1 <- "(?:Charitab(?:ility|l[ey])|\\ Charit(?:ies\\ |y\\ )|Benefit\\ The\\ Masses|Affordable\\ Housing)"
text <- text[grepl(trie_1, text)]
# or without the white space around the last two words
trie_2 <- "(?:Charit(?:ab(?:ility|l[ey])|ies|y)|Benefit\\ The\\ Masses|Affordable\\ Housing)"
text <- text[grepl(trie_2, text)]
However, in the short example, using fixed patterns was actually the fastest (checked with microbenchmark), so you could try this:
text <- text[grepl(sckeywords[1], text, fixed = TRUE) | grepl(sckeywords[2], text, fixed = TRUE) |
grepl(sckeywords[3], text, fixed = TRUE) | grepl(sckeywords[4], text, fixed = TRUE) |
grepl(sckeywords[5], text, fixed = TRUE) | grepl(sckeywords[6], text, fixed = TRUE) |
grepl(sckeywords[7], text, fixed = TRUE)]
If this is still too slow, try to use other tools like awk or sed.
Use parallel processing. You could try to parallelise everything:
make a list of the files to read in and send parts of this file list to the different cores/workers. They then read in the files so that you don't need to send big files to different cores (you can also first try to skip this and start with your original code if the file is not too big)
make a clean data.table and filter it
return the data and join everything
When you run into memory issues, you could also try to save the cleaned data.tables separately in the workers instead of returning them.
For a parallel framework, I'd recommend foreach or the futureverse
Edit
Here is a regex that ignores the case of the first character. However, I'm not an expert on regexes, so maybe there are better ways to optimise the regex!
trie_2 <- "(?:[Cc]harit(?:ab(?:ility|l[ey])|ies|y)|[Bb]enefit\\ [Tt]he\\ [Mm]asses|[Aa]ffordable\\ [Hh]ousing)"
In advance: Sorry for all the Norwegian references, but I hope I've explained my problem good enough for them to still make sense...
So, in 2005 Norway got a new criminal law. The old one was somewhat unstructured (only three chapters), while the statutory provisions in the 2005 version have been structured into 31 chapters, depending on the area of the offense (can be seen here: https://lovdata.no/dokument/NL/lov/2005-05-20-28). I call these "areas of law". For example, in the 2005 version laws regarding sexual offenses are in chapter 26. Logically, then the statutory provisions that belong to this chapter are categorized as belonging to the area of law called "s
Some of the old laws have been structured into the new chapters, some new have been added, and some have been repealed. I have what is called a "law mirror" – a list where you can find where the old provision are in the new law, if it hasn't been repealed. The new law came into force for offenses committed from the 1st of October in 2015.
An example of a law mirror: https://no.wikipedia.org/wiki/Straffeloven_(lovspeil). I've pivoted the list longer, such that it looks like this:
Law Mirror: "Seksuallovbrud" means sexual offense, "kap_2005" says which chapter in the 2005 law that the statutory provision (norwegian: "paragraf") falls under, and "straffelov" specifies whether the provison comes from the 2005 or 1902 version of the law.
The data I have consist of two separate data frames. Df1 is the law mirror. Df2 consists of cases in the Norwegian court of appeals from between 1993 and 2019, where the criminal law was the basis of the verdict. I've made a dummy (strl1902) in Df2 for whether the verdict in the case came before or after the new law came into force. Equal to 1 if it's the old one. I've also extracted the number of the statutory provision.
On the basis of this I want to categorize the cases using statutory provisions from the old criminal law into the areas of law from the new law.
This is where I need help:
Do any of you have any idea of how I can distinguish between the provisions from the old and the new law, such that I also can make dummies for the provisions from the 1902 law, such that they are separated into the areas of the law of the 2005 law?
Hope this makes sense.
I need to count the frequency of unique words found in a column that contains descriptions in each row.
So far, I have eliminated a list of stopwords from the original column and I have extracted the unique words from the column and put them into a list called unique_description.
> description[1:5]
[1] "Come stay Vinh & Stuart (Awarded one Australia's top hosts Airbnb CEO Brian Chesky & key shareholder Ashton Kutcher. We're Sydney's #1 reviewed hosts ). Find out 've positively reviewed 500+ times. Message talk first BEFORE make reservation request - And please read listing end (hint hint). Everything need know . We're pretty relaxed hosts, fully appreciate staying someone , home home, -one. This business, hotel. We're casual Airbnb hosts, hoteliers. If 're looking alternative expensive hotel, 're . Here 'll treated same way treat family & friends stay. So... fluffy bathrobes... Please hello message *BEFORE* make reservation request... It'll help speed things up, smooth things out... Please read listing way end. It make getting confirmed reserv"
[2] "Beautifully renovated, spacious quiet, 3 Bedroom, 3 Bathroom home 10 minute walk beaches Fairlight Forty Baskets, 30 minute walk Manly via coastal promenade, Express bus runs 20 mins door. Our home thirty minute walk along seashore promenade Manly, one Sydney's beautiful beaches, village restaurants, cafes, shopping. If prefer more variety, Manly ferry take Sydney CBD 15 minutes. The residence sited sought- family-friendly street short stroll nearby North Harbour reserve Forty Baskets cafe beach. It's short walk further express CBD buses, ferries, Manly entertainment. Or bus (#131 #132) around corner drops Manly 8 minutes. Our home features stainless steel galley kitchen, including Ilve oven gas cooktop. We two separate living areas ground floor. The front lounge enjoys P&O"
[3] "Welcome sanctuary - bright, comfortable one bedroom apartment North Sydney. Free Wifi, heated pool/jacuzzi everything need make stay Sydney very comfortable. Enjoy fabulous Home away home, fantastic stay Sydney! The apartment within walking distance restaurants shops, Luna Park North Sydney business district. Access Sydney CBD easy bus, train, taxi ferry. It short bus ride famous Balmoral Beach Taronga Zoo. My apartment situated North Sydney 3 kms Sydney CBD. Here details apartment: You'll enjoy being centrally located couple blocks away train station go anywhere quickly Sydney. The apartment features several windows tons natural light. It comfortable fully stocked. Here's I here: LIVING ROOM: 50\" LCD TV DVD / blu ray player CD/Radio/Blue tooth syncing w"
[4] "Fully self-contained sunny studio apartment. 10mn walk Bondi beach. Bus city door. Private 13m swimming pool. Sunny, studio apartment . Private terrace. bus door Bondi Junction City Ground floor 1 bedroom double bed plus kitchenette & study desk. shower & toilet, share laundry, kitchen facilities Swimming pool 13m. Separate security private entrance Private entrance. Ground floor. Happy indicate best spots walking, dining, entertaining best sightseeing location Sydney. Upmarket area. Very nice quiet neighbourhood . Very safe place. Bus door city."
[5] "Sunny warehouse/loft apartment heart one Sydney's best neighbourhoods. Located corner two iconic Darlinghurst streets famous laneway bars eateries, footsteps equally amazing Surry Hills Potts Point. Walk through beautiful parks city less 10 mins, opera house 20 access Bondi Beach easily 25 via bus stop directly front building. My apartment beautiful, simple, open plan / one bedroom loft soaring high ceilings hardwood floors hint 's previous life printing factory 1940s. It huge windows flood space glorious sunshine throughout day provide refreshing breeze during summer. A few key features: * Wireless harman/kardon aura stereo system stream music wirelessly bluetooth device * Internal laundry washer dryer * The kitchen equipped gas cooking, microwave, dishwasher basics preparing m"
> unique_description[1:10]
[1] "Come" "stay" "Vinh" "&" "Stuart" "(Awarded"
[7] "one" "Australia's" "top" "hosts"
I'm not sure how to count the frequency of the words in unique_description that are found in the column 'description'. I tried using freq_terms in library(qdap), but qdap will not load for me so am trying to find another way.
You could use the stringr package.
library(stringr)
x <- "Come stay Vinh & Stuart (Awarded one Australia's top hosts Airbnb CEO Brian Chesky & key shareholder Ashton Kutcher. We're Sydney's #1 reviewed hosts ). Find out 've positively reviewed 500+ times. Message talk first BEFORE make reservation request - And please read listing end (hint hint). Everything need know . We're pretty relaxed hosts, fully appreciate staying someone , home home, -one. This business, hotel. We're casual Airbnb hosts, hoteliers. If 're looking alternative expensive hotel, 're . Here 'll treated same way treat family & friends stay. So... fluffy bathrobes... Please hello message *BEFORE* make reservation request... It'll help speed things up, smooth things out... Please read listing way end. It make getting confirmed reserv"
y <- "come stay Stuart"
unique_desc <- c("come", "stay", "Stuart")
desc <- c(x,y)
result <- lapply(desc, FUN = str_count, pattern = unique_desc)
#result holds first element is counts in first element of desc
lapply will call the str_count function to each element of desc. In this example the ith entry of result there is a vector of counts corresponding to the ith entry of desc and the vector of counts correspond to the counts of each word in unique_desc.
I have a quanteda corpus of hundreds of documents. How do I remove specific sections - like the abstract and footnotes etc. Otherwise, I am faced with doing it manually. Thanks
As requested, here is a text example. It is from a regular journal article. It shows the Meta data, then the abstract, then keywords, then introduction, then author contact details, then body of article, then Note, then Disclosure statement, then Notes on contributors, then references. I would like to remove everything apart from the introduction and body of the article. I would also like to remove the author name and Journal title - which are repeated throughout
" Behavioral Sciences of Terrorism and Political Aggression
ISSN: 1943-4472 (Print) 1943-4480 (Online) Journal homepage: http://www.tandfonline.com/loi/rirt20
Sometimes they come back: responding to
American foreign fighter returnees and other
Elusive threats
Christopher J. Wright
To cite this article: Christopher J. Wright (2018): Sometimes they come back: responding to
American foreign fighter returnees and other Elusive threats, Behavioral Sciences of Terrorism and
Political Aggression, DOI: 10.1080/19434472.2018.1464493
To link to this article: https://doi.org/10.1080/19434472.2018.1464493
Published online: 23 Apr 2018.
Submit your article to this journal
Article views: 57
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=rirt20
"
"BEHAVIORAL SCIENCES OF TERRORISM AND POLITICAL AGGRESSION, 2018
https://doi.org/10.1080/19434472.2018.1464493
Sometimes they come back: responding to American foreign
fighter returnees and other Elusive threats
Christopher J. Wright
Department of Criminal Justice, Austin Peay State University, Clarksville, TN, USA
ABSTRACT ARTICLE HISTORY
Much has been made of the threat of battle hardened jihadis from Received 8 January 2018
Islamist insurgencies, especially Syria. But do Americans who Accepted 10 April 2018
return home after gaining experience fighting abroad pose a
KEYWORDS
greater risk than homegrown jihadi militants with no such Terrorism; foreign fighters;
experience? Using updated data covering 1990–2017, this study domestic terrorism;
shows that the presence of a returnee decreases the likelihood homegrown terrorism;
that an executed plot will cause mass casualties. Plots carried out lone-wolf; homeland security
Introduction: being afraid. Being a little afraid
How great of a threat do would-be jihadis pose to their home country? And do those who
return home after gaining experience fighting abroad in Islamist insurgencies or attending
terror training camps pose a greater risk than other jihadi militants? The fear, as first outlined
by Hegghammer (2013), is two-fold. First, individuals that have gone abroad to fight might
CONTACT Christopher J. Wright wrightc#apsu.edu Department of Criminal Justice, Austin Peay State University,
Clarksville, TN 37043, USA
© 2018 Society for Terrorism Research
"
"2 C. J. WRIGHT
Many of the earliest studies on Western foreign fighters suggested that those who
returned were in fact more deadly than those with no experience fighting in Islamist insur-
gencies. Hegghammer’s (2013) analysis suggested that these foreign fighter returnees
were a greater danger than when they left. Likewise, Byman (2015), Nilson (2015),
Kenney (2015), and Vidno (2011) came to similar conclusions while offering key insights
into the various mechanisms linking foreign fighting with successful plot execution and
greater mass casualties.
Other studies came to either mixed conclusions or directly contradicted the earlier find-
ings. Adding several years of data to Hegghammer’s (2013) earlier study, Hegghammer
"
" BEHAVIORAL SCIENCES OF TERRORISM AND POLITICAL AGGRESSION 3
for them to form the types of large, local networks that would be necessary to carry out a
large-scale attack without attracting the attention of security services’ (p. 92).
"
Note
1. Charges were brought against Noor Zahi Salman, the widow of the Omar Mateen who carried
out the June, 2016 attack against the Pulse Nightclub in Orlando, Florida (US Department of
Justice., 2017a, January 17). However, in March of 2018 a jury acquitted her of the charges that
she had foreknowledge of the attack.
Disclosure statement
No potential conflict of interest was reported by the authors.
Notes on contributors
Christopher J. Wright, Ph.D., is an Assistant Professor at Austin Peay State University where he
teaches in the Homeland Security Concentration.
ORCID
Christopher J. Wright http://orcid.org/0000-0003-0043-6616
References
Byman, D. (2015). The homecomings: What happens when Arab foreign fighters in Iraq and Syria
return? Studies in Conflict & Terrorism, 38(8), 581–602.
Byman, D. (2016). The Jihadist returnee threat: Just how dangerous? Political Science Quarterly, 131(1),
69–99.
Byman, D., & Shapiro, J. (2014). Be afraid. Be a little afraid: The threat of terrorism from Western foreign
fighters in Syria and Iraq. Foreign Policy at Brookings. Washington, DC: Brookings. Retrieved from
https://www.brookings.edu/wp-content/uploads/2016/06/Be-Afraid-web.pdf
The approach
The key here is to determine the regular markers that precede each section, and then to use them as tags in a call to corpus_segment(). It's the tags that will need tweaking, based on their degree of regularity across documents.
Based on what you supplied above, I pasted that into a plain text file that I named example.txt. This code extracted the Introduction and what I think is the body of the article, but for that I had to decide a tag that marked its ending. Below, I used "Disclosure Statement". So:
library("quanteda")
crp <- readtext::readtext("~/tmp/example.txt") %>%
corpus()
pat <- c("\nIntroduction?", "\nCONTACT", "©", "\nDisclosure statement")
crpextracted <- corpus_segment(crp, pattern = pat)
summary(crpextracted)
## Corpus consisting of 4 documents:
##
## Text Types Tokens Sentences pattern
## example.txt.1 62 74 5 Introduction:
## example.txt.2 18 21 2 CONTACT
## example.txt.3 156 253 11 ©
## example.txt.4 101 180 19 Disclosure statement
##
## Source: /Users/kbenoit/Dropbox (Personal)/GitHub/quanteda/quanteda/* on x86_64 by kbenoit
## Created: Fri Jul 6 19:51:01 2018
## Notes: corpus_segment.corpus(crp, pattern = pat)
When you examine the text in the "Introduction:" tagged segment, you can see that everything from that string until the next tag was extracted to that as a new document:
corpus_subset(crpextracted, pattern == "\nIntroduction:") %>%
texts() %>% cat()
## being afraid. Being a little afraid
##
## How great of a threat do would-be jihadis pose to their home country? And do those who
##
## return home after gaining experience fighting abroad in Islamist insurgencies or attending
##
## terror training camps pose a greater risk than other jihadi militants? The fear, as first outlined
##
## by Hegghammer (2013), is two-fold. First, individuals that have gone abroad to fight might
How to remove pdf junk
All pdf conversions produce unwanted junk such as running headers, footers, etc. Here's how to remove them. (Note: You will want to do this before the step above.) How to construct the toreplace pattern? You will need to understand something about regular expressions, and use some experimentation.
library("stringr")
toreplace <- '\\n*\" \" BEHAVIORAL SCIENCES OF TERRORISM AND POLITICAL AGGRESSION,{0,1} \\d+\\n*'
texts(crp) <- str_replace_all(texts(crp), regex(toreplace), "")
cat(texts(crp))
To demonstrate this on a section from your example:
# demonstration
x <- '
" " BEHAVIORAL SCIENCES OF TERRORISM AND POLITICAL AGGRESSION 3
'
str_replace_all(x, regex(toreplace), "")
## [1] ""
I have one page story (i.e. text data), I need to use Bayesian network on that story and analyse the same. Could someone tell me whether it is possible in R? If yes, that how to proceed?
The objective of the analysis is - Extract Action Descriptions from
Narrative Text.
The data considered for analysis -
Krishna’s Dharam-shasthra to Arjuna:
The Gita is the conversation between Krishna and Arjuna leading up to the battle.
Krishna emphasised on two terms: Karma and Dharma. He told Arjun that this was a righteous war; a war of Dharma. Dharma is the way of righteousness or a set of rules and laws laid down. The Kauravas were on the side of Adharma and had broken rules and laws and hence Arjun would have to do his Karma to uphold Dharma.
Arjuna doesn't want to fight. He doesn't understand why he has to shed his family's blood for a kingdom that he doesn't even necessarily want. In his eyes, killing his evil and killing his family is the greatest sin of all. He casts down his weapons and tells Krishna he will not fight. Krishna, then, begins the systematic process of explaining why it is Arjuna's dharmic duty to fight and how he must fight in order to restore his karma.
Krishna first explains the samsaric cycle of birth and death. He says there is no true death of the soul simply a sloughing of the body at the end of each round of birth and death. The purpose of this cycle is to allow a person to work off their karma, accumulated through lifetimes of action. If a person completes action selflessly, in service to God, then they can work off their karma, eventually leading to a dissolution of the soul, the achievement of enlightenment and vijnana, and an end to the samsaric cycle. If they act selfishly, then they keep accumulating debt, putting them further and further into karmic debt.
What I want is - post tagger to separate verbs, nouns etc. and then create a meaningful network using that.
The steps that should be followed in pre-processing are:
syntactic processing (post tagger)
SRL algorithm (semantic role labelling of characters of the story)
conference resolution
Using all of the above I need to create a knowledge database and create a Bayesian network.
This is what I have tried so far to get post tagger:
txt <- c("As the years went by, they remained isolated in their city. Their numbers increased by freeing women from slavery.
Doom would come to the world in the form of Ares the god of war and the Son of Zeus. Ares was unhappy with the gods as he wanted to prove just how foul his father’s creation was. Hence, he decided to corrupt the mortal men created by Zeus. Fearing his wrath upon the world Zeus decided to create the God killer in order to stop Ares. He then commanded Hippolyta to mould a baby from the sand and clay of the island. Then the five goddesses went back into the Underworld, drawing out the last soul that remained in the Well and giving it incredible powers. The soul was merged with the clay and became flesh. Hippolyta had her daughter and named her Diana, Princess of the Amazons, the first child born on Paradise Island.
Each of the six members of the Greek Pantheon granted Diana a gift: Demeter, great strength; Athena, wisdom and courage; Artemis, a hunter's heart and a communion with animals; Aphrodite, beauty and a loving heart; Hestia, sisterhood with fire; Hermes, speed and the power of flight. Diana was also gifted with a sword, the Lasso of truth and the bracelets of penance as weapons to defeat Ares.
The time arrived when Diana, protector of the Amazons and mankind was sent to the Man's World to defeat Ares and rid the mortal men off his corruption. Diana believed that only love could truly rid the world of his influence. Diana was successfully able to complete the task she was sent out by defeating Ares and saving the world.
")
writeLines(txt, tf <- tempfile())
library(stringi)
library(cleanNLP)
cnlp_init_tokenizers()
anno <- cnlp_annotate(tf)
names(anno)
get_token(anno)
cnlp_init_spacy()
anno <- cnlp_annotate(tf)
get_token(anno)
cnlp_init_corenlp()