Related
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 3 years ago.
Improve this question
I have a somewhat tricky BED file format, which I should convert to a classic BED format so as I can properly use it for further steps:
I have this unconventional BED format:
1 12349 12398 +
1 23523 23578 -
1 23550;23570;23590 23640;23689;23652 +
1 43533 43569 +
1 56021;56078 56099;56155 +
Say that those multiple position rows are representing non-coding fragmented regions.
What I would like to get is a canonical BED file such as:
1 12349 12398 +
1 23523 23578 -
1 23550 23640 +
1 23570 23689 +
1 23590 23652 +
1 43533 43569 +
1 56021 56099 +
1 56078 56155 +
where the poliregions that were mixed in one row, are put in other rows, while maintaining chromosome number and strand.
Could you please try following.
awk '
{
num=split($2,array1,";")
num1=split($3,array2,";")
}
num>1 || num1>1{
for(i=1;i<=num;i++){
print $1,array1[i],array2[i],$NF
}
next
}
1' Input_file | column -t
Output will be as follows.
1 12349 12398 +
1 23523 23578 -
1 23550 23640 +
1 23570 23689 +
1 23590 23652 +
1 43533 43569 +
1 56021 56099 +
1 56078 56155 +
#!/usr/bin/env bash
# ^^^^-- NOT /bin/sh
while read -r a b c d; do
if [[ $b = *';'* ]]; then # if b contains any ';'s
IFS=';' read -r -a ba <<<"$b" # read string b into array ba
IFS=';' read -r -a ca <<<"$c" # read string c into array ca
for idx in "${!ba[#]}"; do # iterate over the indices of array ba
# print a and d with the values for a given index for both ba and ca
printf '%s\t%s\t%s\t%s\n' "$a" "${ba[idx]}" "${ca[idx]}" "$d"
done
else
printf '%s\t%s\t%s\t%s\n' "$a" "$b" "$c" "$d"
fi
done
This combines the answers to existing StackOverflow questions:
bash script loop through two variables in lock step
Reading a delimited string into an array in Bash
...and guidance in the BashFAQ:
How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?
See this running at https://ideone.com/wmrXPE
$ cat tst.awk
BEGIN { FS="[[:space:];]+" }
{
n = (NF - 2) / 2
for (i=1; i<=n; i++) {
print $1, $(i+1), $(i+n), $NF
}
}
$ awk -f tst.awk file
1 12349 12349 +
1 23523 23523 -
1 23550 23590 +
1 23570 23640 +
1 23590 23689 +
1 43533 43533 +
1 56021 56078 +
1 56078 56099 +
Try Perl solution
perl -lane ' if( /;/ and /(\S{2,})\s+(\S{2,})/ ) {
$i=0;#x=split(";",$1);#y=split(";",$2); while($i++<scalar(#x))
{ print join(" ",$F[0],$x[$i-1],$y[$i-1],$F[-1]) }} else { print } ' emilio.txt| column -t
with the given inputs
$ cat emilio.txt
1 12349 12398 +
1 23523 23578 -
1 23550;23570;23590 23640;23689;23652 +
1 43533 43569 +
1 56021;56078 56099;56155 +
$ perl -lane ' if( /;/ and /(\S{2,})\s+(\S{2,})/ ) {
$i=0;#x=split(";",$1);#y=split(";",$2); while($i++<scalar(#x))
{ print join(" ",$F[0],$x[$i-1],$y[$i-1],$F[-1]) }} else { print } ' emilio.txt| column -t
1 12349 12398 +
1 23523 23578 -
1 23550 23640 +
1 23570 23689 +
1 23590 23652 +
1 43533 43569 +
1 56021 56099 +
1 56078 56155 +
$
What i am trying to find out is, for example let's take as an example the following table:
| Col 1 | Col 2 |
|-------|---------|
| ab | 1 |
| ab ab | 2 |
| ac | 1 |
| ae | 1 |
| ae ae | 2 |
| af | 1 |
So basically if there are two occurrences of the same item in the cell, I want to display 2 in the next column. If there are 3, then 3 and so on. The thing is that I am looking for specific strings most of the time. Its a text and number string.
Is this doable in Power BI?
Assuming you want to count the number of occurrences of the first non-space characters that occur before the first separating space, you can do the following:
Col 2 =
VAR Trimmed = TRIM(Table2[Col 1])
VAR FirstSpace = SEARCH(" ", Trimmed, 1, LEN(Trimmed) + 1)
VAR FirstString = LEFT(Trimmed, FirstSpace - 1)
RETURN DIVIDE(
LEN(Trimmed) - LEN(SUBSTITUTE(Trimmed, FirstString, "")),
FirstSpace - 1
)
Let's go through an example to see how this works. Suppose we have a string " abc abc abc ".
The TRIM function removes any extraneous spaces at the beginning an end, so Trimmed = "abc abc abc".
The FirstSpace searches for the first space in Trimmed. In this case, FirstSpace = 4. (If there is no first space, then we define FirstSpace to be the length of Trimmed + 1 so the next part works correctly.)
The FirstString uses FirstSpace to find the first chunk. In this case, FirstString = "abc".
Finally, we use SUBSTITUTE to replace each FirstString with an empty string (leaving only the middle spaces) and look at how that changes the length of Trimmed. We know LEN(Trimmed) = 11 and LEN(" ") = 2, so the difference is the 9 characters we removed by substitution. We know that the 9 characters are n copies of FirstString, "abc" and we know the length of FirstString is FirstSpace - 1 = 3.
Thus we can solve 3n = 9 for n to get n = 9/3 = 3, the count of the "abc" substrings.
I am looking to explore the GameTheory package from CRAN, but I would appreciate help in converting my data (in the form of a data frame of unique combinations and results) in to the required coalition object. The precursor to this I believe to be an ordered list of all coalition values (https://cran.r-project.org/web/packages/GameTheory/vignettes/GameTheory.pdf).
My real data has n ~ 30 'players', and unique combinations = large (say 1000 unique combinations), for which I have 1 and 0 identifiers to describe the combinations. This data is sparsely populated in that I do not have data for all combinations, but will assume combinations not described have zero value. I plan to have one specific 'player' who will appear in all combinations, and act as a baseline.
By way of example this is the data frame I am starting with:
require(GameTheory)
games <- read.csv('C:\\Users\\me\\Desktop\\SampleGames.csv', header = TRUE, row.names = 1)
games
n1 n2 n3 n4 Stakes Wins Success_Rate
1 1 1 0 0 800 60 7.50%
2 1 0 1 0 850 45 5.29%
3 1 0 0 1 150000 10 0.01%
4 1 1 1 0 300 25 8.33%
5 1 1 0 1 1800 65 3.61%
6 1 0 1 1 1900 55 2.89%
7 1 1 1 1 700 40 5.71%
8 1 0 0 0 3000000 10 0.00333%
where n1 is my universal player, and in this instance, I have described all combinations.
To calculate my 'base' coalition value from player {1} alone, I am looking to perform the calculation: 0.00333% (success rate) * all stakes, i.e.
0.00333% * (800 + 850 + 150000 + 300 + 1800 + 1900 + 700 + 3000000) = 105
I'll then have zero values for {2}, {3} and {4} as they never "play" alone in this example.
To calculate my first pair coalition value, I am looking to perform the calculation:
7.5%(800 + 300 + 1800 + 700) + 0.00333%(850 + 150000 + 1900 + 3000000) = 375
This is calculated as players {1,2} base win rate (7.5%) by the stakes they feature in, plus player {1} base win rate (0.00333%) by the combinations he features in that player {2} does not - i.e. exclusive sets.
This logic is repeated for the other unique combinations. For example row 4 would be the combination of {1,2,3} so the calculation is:
7.5%(800+1800) + 5.29%(850+1900) + 8.33%(300+700) + 0.00333%(3000000+150000) = 529 which descriptively is set {1,2} success rate% by Stakes for the combinations it appears in that {3} does not, {1,3} by where {2} does not feature, {1,2,3} by their occurrences, and the base player {1} by examples where neither {2} nor {3} occur.
My expected outcome therefore should look like this I believe:
c(105,0,0,0, 375,304,110,0,0,0, 529,283,246,0, 400)
where the first four numbers are the single player combinations {1} {2} {3} and {4}, the next six numbers are two player combinations {1,2} {1,3} {1,4} (and the null cases {2,3} {2,4} {3,4} which don't exist), then the next four are the three player combinations {1,2,3} {1,2,4} {1,3,4} and the null case {2,3,4}, and lastly the full combination set {1,2,3,4}.
I'd then feed this in to the DefineGame function of the package to create my coalitions object.
Appreciate any help: I have tried to be as descriptive as possible. I really don't know where to start on generating the necessary sets and set exclusions.
First, we segment the email address into 2-character strings.
Then, for every segment s, we compute the following hash J:
md5(md5(s) + s + md5(s)) [where + is the string concatenation operator].
Finally, we concatenate all hash strings J to form the long hash below.
For example: for an input of helloworld#company.com, we would compute:
md5(md5('he') + 'he' + md5('he')) +
md5(md5('ll') + 'll' + md5('ll')) +
md5(md5('ow') + 'ow' + md5('ow')) +
...
Long Hash:
f894e71e1551d1833a977df952d0cc9de44a1f9669fbf97d51309a2c6574d5eaa746cdeb9ee1a5df
c771d280d33e5672bf024973657c99bf80cb242d493d5bacc771b3b0b422d5c13595cf3e73cfb1df
91caedee7a6c5f3ce2c283564a39c52d3306d60cbc0e3e33d7ed01e780acb1ccd9174cfea4704eb2
33b0f06e52f6d5aba5a5a89e6122dd55f8efcf024961c1003d116007775d60a0d5781d2e35d747b5
dece2e0e3d79d272e40c8c66555f5525
How can I recover the email address from the hash? As I understand it, a "Hash" is a One Way Function. I can only compare it to another hash to see if they match or generate a Hash of the original text.
While it may be true in general that it is impractical to extract the original message from a hash, this clearly looks like an exercise with conditions carefully crafted to make it possible to break the "encryption".
Consider that the email address is broken up into two-character segments. If you limit yourself to just lowercase letters (26 letters + 2 symbols, # and ., there are only 28 * 28 = 784 possible two-letter combinations. Even if the emails have lowercase and uppercase letters and numbers, there are only 64 * 64 = 4096 combinations -- well within computational limits.
The thing to do is to pre-compute a rainbow table, or table of all possible hash values in your search space. You could do this with a matrix:
+----------------------------------+----------------------------------+----------------------------------------+-----------------------------+
| a | b | c | ... |
+----------------------------------+----------------------------------+----------------------------------------+-----------------------------+
a| md5(md5('aa') + 'aa' + m5('aa')) | md5(md5('ba') + 'ba' + m5('ba')) | md5(md5('ca') + 'ca' + m5('ca')) | ... |
+----------------------------------+----------------------------------+----------------------------------------+-----------------------------+
b| md5(md5('ab') + 'ab' + m5('ab')) | md5(md5('bb') + 'bb' + m5('bb')) | md5(md5('cb') + 'cb' + m5('cb')) | ... |
+----------------------------------+----------------------------------+----------------------------------------+-----------------------------+
c| md5(md5('ac') + 'ac' + m5('ac')) | md5(md5('bc') + 'bc' + m5('bc')) | md5(md5('cc') + 'cc' + m5('cc')) | ... |
+----------------------------------+----------------------------------+----------------------------------------+-----------------------------+
| ... | ... | ... | ... |
+----------------------------------+----------------------------------+----------------------------------------+-----------------------------+
but then you would have to traverse the matrix each time to find a match -- slow!
An alternative is to use a dictionary with the key being the hash, and the value being the 'decoded' letters:
{
md5(md5('aa') + 'aa' + md5('aa')): 'aa',
md5(md5('ab') + 'ab' + md5('ab')): 'ab',
md5(md5('ac') + 'ac' + md5('ac')): 'ac',
...
}
Either way, you will now have the hashes for all possible two-letter combinations. Now you process the input string. Since MD5 produces 32-character long hashes, break the input up into 32-character strings, and perform lookups against your table:
'f894e71e1551d1833a977df952d0cc9d' => 'he'
'e44a1f9669fbf97d51309a2c6574d5ea' => 'll'
...
Here is implementation of your question in python.
My Code:
import hashlib, string
# lambda function for MD5
md5hashFunction = lambda data: hashlib.md5(data.encode()).hexdigest()
# lambda function for md5(md5(data) + data + md5)
finalHash = lambda data: md5hashFunction(
md5hashFunction(data) + data + md5hashFunction(data)
)
# All MD5 hashes are 32 char length size therefore we need dive 32 fixed parts
hashes = [
"f894e71e1551d1833a977df952d0cc9d",
"e44a1f9669fbf97d51309a2c6574d5ea",
"a746cdeb9ee1a5dfc771d280d33e5672",
"bf024973657c99bf80cb242d493d5bac",
"c771b3b0b422d5c13595cf3e73cfb1df",
"91caedee7a6c5f3ce2c283564a39c52d",
"3306d60cbc0e3e33d7ed01e780acb1cc",
"d9174cfea4704eb233b0f06e52f6d5ab",
"a5a5a89e6122dd55f8efcf024961c100",
"3d116007775d60a0d5781d2e35d747b5",
"dece2e0e3d79d272e40c8c66555f5525",
]
# Enumurate all alphabet and extra characters for decryption => "_+.#"
alphabet = list(
string.ascii_lowercase + string.ascii_uppercase + string.digits + "_+.#"
)
# Create python dictionary for map hashes to string
rainbowTable = {finalHash(x + y): x + y for x in alphabet for y in alphabet}
"""
rainbowTable
'31453dd786a8c6f6c7c8860d5fcea4be': 'aa',
'857dce5bcf6b6b32bec281207b2dba80': 'ab',
'e90d94b4b65ac19188fdae82acf7fbbc': 'ac',
'67299b8cedc5eafea7dda1daf9356b54': 'ad',
'40fca4e80bfc6e1faa2c4e2b7e0929f0': 'ae',
'de48fc1bd98f5508c513f9947a514ce8': 'af',
'4852089b1b43b45204907df0066c0edf': 'ag',
'e1b82a5fe4fdcf73d034a0d5063ffe3f': 'ah',
...... Continues....
"""
# Search for matched hash and join to single string
print("".join([rainbowTable[hash] for hash in hashes]))
"""
f894e71e1551d1833a977df952d0cc9de44a1f9669fbf97d51309a2c6574d5eaa746cdeb9ee1a5df
c771d280d33e5672bf024973657c99bf80cb242d493d5bacc771b3b0b422d5c13595cf3e73cfb1df
91caedee7a6c5f3ce2c283564a39c52d3306d60cbc0e3e33d7ed01e780acb1ccd9174cfea4704eb2
33b0f06e52f6d5aba5a5a89e6122dd55f8efcf024961c1003d116007775d60a0d5781d2e35d747b5
dece2e0e3d79d272e40c8c66555f5525
"""
"""
Output ==> secret_jobs#anvato.com
"""
Here is what you can do:
Step 1: Divide the hash string in 32 bit blocks
Step 2: find all possible combinations of 2 character strings from the list of strings which can be combination of alphabets, numbers and any special characters.
Step 3: generate MD5 hash code for that segment, concatenate it with plain text segment and same hash code and generate MD5 hash code again
Step 4: Compare the generates hash code with the existing hash code. If it matched save it in string buffer. Iterate this process till all the blocks are decoded. You will have your answer.
I'm trying to write some code to analyze my company's insurance plan offerings... but they're complicated! The PPO plan is straightforward, but the high deductible health plans are complicated, as they introduced a "split" deductible and out of pocket maximum (individual and total) for the family plans. It works like this:
Once the individual meets the individual deductible, he/she is covered at 90%
Once the remaining 1+ individuals on the plan meet the total deductible, the entire family is covered at 90%
The individual cannot satisfy the family deductible with only their medical expenses
I want to feed in a vector of expenses for my family members (there are four of them) and output the total cost for each plan. Below is a table of possible scenarios, with the following column codes:
ded_ind: did one individual meet the individual deductible?
ded_tot: was the total deductible reached?
oop_ind: was the individual out of pocket max reached
oop_tot: was the total out of pocket max reached?
exp_ind = the expenses of the highest spender
exp_rem = the expenses of the remaining /other/ family members (not the highest spender)
oop_max_ind = the level of expenses at which the individual has paid their out of pocket maximum (when ded_ind + 0.1 * exp_ind = out of pocket max for the individual
oop_max_fam = same as for individual, but for remaining family members
The table:
| ded_ind | oop_ind | ded_rem | oop_rem | formula
|---------+---------+---------+---------+---------------------------------------------------------------------------|
| 0 | 0 | 0 | 0 | exp_ind + exp_rem |
| 1 | 0 | 0 | 0 | ded_ind + 0.1 * (exp_ind - ded_ind) + exp_rem |
| 0 | 0 | 1 | 0 | exp_ind + ded_rem + 0.1 * (exp_rem - ded_rem) |
| 1 | 1 | 0 | 0 | oop_max_ind + exp_fam |
| 1 | 0 | 1 | 0 | ded_ind + 0.1 * (exp_ind - ded_ind) + ded_rem + 0.1 * (exp_rem - ded_rem) |
| 0 | 0 | 1 | 1 | oop_max_rem + exp_ind |
| 1 | 0 | 1 | 1 | ded_ind + 0.1 * (exp_ind - ded_ind) + oop_max_rem |
| 1 | 1 | 1 | 0 | oop_ind_max + ded_rem + 0.1 * (exp_rem - ded_rem) |
| 1 | 1 | 1 | 1 | oop_ind_max + oop_rem_max |
Omitted: 0 1 0 0, 0 0 0 1, 0 1 1 0, and 0 1 0 1 are not present, as oop_ind and oop_rem could not have been met if ded_ind and ded_rem, respectively, have not been met.
My current code is a somewhat massive ifelse loop like so (not the code, but what it does):
check if plan is ppo or hsa
if hsa plan
if exp_ind + exp_rem < ded_rem # didn't meet family deductible
if exp_ind < ded_ind # individual deductible also not met
cost = exp_ind + exp_rem
else is exp_ind > oop_ind_max # ded_ind met, is oop_ind?
ded_ind + 0.1 * (exp_ind - ded_ind) + exp_fam # didn't reach oop_max_ind
else oop_max_ind + exp_fam # reached oop_max_ind
else ...
After the else, the total is greater than the family deductible. I check to see if it was contributed by more than two people and just continue on like that.
My question, now that I've given some background to the problem: Is there a better way to manage conditional situations like this than ifelse loops to filter them down a bit at a time?
The code ends up seeming redundant, as one checks for some higher level conditions (consider the table where ded_rem is met or not met... one still has to check for ded_ind and oop_max_ind in both cases, and the code is the same... just positioned at two different places in the ifelse structure).
Could this be done with some sort of matrix operation? Are there other examples online of more clever ways to deal with filtering of conditions?
Many thanks for any suggestions.
P.S. I'm using R and will be creating an interactive with shiny so that other employees can input best and worst case scenarios for each of their family members and see which plan comes out ahead via a dot or bar chart.
The suggestion to convert to a binary value based on the result gave me an idea, which also helped me learn that one can do vectorized TRUE / FALSE checks (I guess that was probably obvious to many).
Here's my current idea:
expenses will be a vector of individual forecast medical expenses for the year (example of three people):
expenses <- c(1500, 100, 400)
We set exp_ind to the max value, and sum the rest for exp_rem
exp_ind <- max(expenses)
# [1] index of which() for cases with multiple max values
exp_rem <- sum(expenses[-which(expenses == exp_ind)[1]])
For any given plan, I can set up a vector with the cutoffs, for example:
individual deductible = 1000
individual out of pocket max = 2000 (need to incur 11k of expenses to get there)
family deductible = 2000
family out of pocket max = 4000 (need to incur 22k of expenses to get there)
Set those values:
ded_ind <- 1000
oop_max_ind <- 11000
ded_tot <- 2000
oop_max_tot <- 22000
cutoffs <- c(ded_ind, oop_max_ind, ded_tot, oop_max_tot)
Now we can check the input expense against the cutoffs:
result <- as.numeric(rep(c(exp_ind, exp_rem), each = 2) > cutoffs)
Last, convert to binary:
result_bin <- sum(2^(seq_along(result) - 1) * result)
Now I can set up functions for the possible outcomes based on the value in result_bin:
if(result_bin == 1) {cost <- ded_ind + 0.1 * (exp_ind - ded_ind) + exp_rem }
cost
[1] 1550
We can check this...
High spender would have paid his 1000 and then 10% of remaining 500 = 1050
Other members did not reach the family deductible and paid the full 400 + 100 = 500
Total: 1550
I still need to create a mapping of results_bin values to corresponding functions, but doing a vectorized check and converting a unique binary value is much, much better, in my opinion, than my ifelse nested mess.
I look at it like this: I'd have had to set the variables and write the functions anyway; this saves me 1) explicitly writing all the conditions, 2) the redundancy issue I was talking about in that one ends up writing identical "sibling" branches of parent splits in the ifelse structure, and lastly, 3) the code is far, far, far more easily followed.
Since this question is not very specific, here is a simpler example/answer:
# example data
test <- expand.grid(opt1=0:1,opt2=0:1)
# create a unique identifier to represent the binary variables
test$code <- with(allopts,paste(opt1,opt2,sep=""))
# create an input variable to be used in functions
test$var1 <- 1:4
# opt1 opt2 code var1
#1 0 0 00 1
#2 1 0 10 2
#3 0 1 01 3
#4 1 1 11 4
Respective functions to apply depending on binary conditions, along with intended results for each combo:
var1 + 10 #code 00 - intended result = 11
var1 + 100 #code 10 - intended result = 102
var1 + 1000 #code 01 - intended result = 1003
var1 + var1 #code 11 - intended result = 8
Use ifelse combinations to do the calculations:
test$result <- with(test,
ifelse(code == "00", var1 + 10,
ifelse(code == "10", var1 + 100,
ifelse(code == "01", var1 + 1000,
ifelse(code == "11", var1 + var1,
NA
)))))
Result:
opt1 opt2 code var1 result
1 0 0 00 1 11
2 1 0 10 2 102
3 0 1 01 3 1003
4 1 1 11 4 8