I trying to produce this output character string
CONTAINS(ORIG_DOC,'SECTIONS("7 - Past Medical/Surgical History")(PRECISE FORM OF "DM", PRECISE FORM OF "DM2", "diabetes")') <> 0
using the paste function as below
paste("CONTAINS(ORIG_DOC,'SECTIONS("7 - Past Medical/Surgical History")(PRECISE FORM OF "DM", PRECISE FORM OF "DM2", "diabetes")') <> 0")
I am getting an error
Error: unexpected numeric constant in "paste("CONTAINS(ORIG_DOC,'SECTIONS("7"
Not sure what I am missing here, any help is much appreciated.
Escape all double-quotes " as \" and use noquote:
> txt = noquote(paste("CONTAINS(ORIG_DOC,'SECTIONS(\"7 - Past Medical/Surgical History\") (PRECISE FORM OF \"DM\", PRECISE FORM OF \"DM2\", \"diabetes\")') <> 0"))
> txt
[1] CONTAINS(ORIG_DOC,'SECTIONS("7 - Past Medical/Surgical History") (PRECISE FORM OF "DM", PRECISE FORM OF "DM2", "diabetes")') <> 0
>
Related
Question
My question, explained below, is:
How can R be used to read a string that includes HTML emoji codes like 🤗?
I'd like to:
(1) represent the emoji symbol (e.g., as a unicode symbol: 🤗) in the parsed string, OR(2) convert it into its text equivalent (":hugging face:")
Background
I have an XML dataset of text messages (from the Android/iOS app Signal) that I am reading into R for a text mining project. The data look like this, with each text message represented in an sms node:
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!-- File Created By Signal -->
<smses count="1">
<sms protocol="0" address="+15555555555" contact_name="Jane Doe" date="1483256850399" readable_date="Sat, 31 Dec 2016 23:47:30 PST" type="1" subject="null" body="Hug emoji: 🤗" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />
</smses>
Problem
I am currently reading the data using the xml2 package for R. When I use the xml2::read_xml function, however, I get the following error message:
Error in doc_parse_raw(x, encoding = encoding, base_url = base_url, as_html = as_html, :
xmlParseCharRef: invalid xmlChar value 55358
Which, as I understand, indicates that the emoji character is not recognized as valid XML.
Using the xml2::read_html function does work, but drops the emoji character. A small example of this is here:
example_text <- "Hugging emoji: 🤗"
xml2::xml_text(xml2::read_html(paste0("<x>", example_text, "</x>")))
(Output: [1] "Hugging emoji: ")
This character is valid HTML -- Googling 🤗 actually converts it in the search bar to the "hugging face" emoji, and brings up results relating to that emoji.
Other information I've found that seems relevant to this question
I've been searching Stack Overflow, and have not found any questions relating to this particular issue. I've also not been able to find a table that straightforwardly gives HTML codes next to the emoji they represent, and so am not able to do an (albeit inefficient) conversion of these HTML codes to their textual equivalents in a big loop before parsing the dataset; for example, neither this list nor its underlying dataset seem to include the string 55358.
tl;dr: the emoji aren't valid HTML entities; UTF-16 numbers have been used to build them instead of Unicode code points. I describe an algorithm at the bottom of the answer to convert them so that they are valid XML.
Identifying the Problem
R definitely handles emoji:
In fact, a few packages exist for handling emoji in R. For example, the emojifont and emo packages both let you retrieve emoji based on Slack-style keywords. It's just a question of getting your source characters through from the HTML-escaped format so that you can convert them.
xml2::read_xml seems to do fine with other HTML entities, like an ampersand or double quotes. I looked at this SO answer to see whether there were any XML-specific constraints on HTML entities, and it seemed like they were storing emoji fine. So I tried changing the emoji codes in your reprex to the ones in that answer:
body="Hug emoji: 😀😃"
And, sure enough, they were preserved (though they're obviously not the hug emoji anymore):
> test8 = read_html('Desktop/test.xml')
> test8 %>% xml_child() %>% xml_child() %>% xml_child() %>% xml_attr('body')
[1] "Hug emoji: \U0001f600\U0001f603"
I looked up the hug emoji on this page, and the decimal HTML entity given there is not 🤗. It looks like the UTF-16 decimal codes for the emoji have been wrapped in &# and ;.
In conclusion, I think the answer is that your emoji are, in fact, not valid HTML entities. If you can't control the source, you might need to do some pre-processing to account for these errors.
So, why does the browser convert them properly? I'm wondering if the browser is a little more flexible with these things and is making some guesses about what those codes could be. I'm just speculating, though.
Converting UTF-16 to Unicode code points
After some more investigation, it looks like valid emoji HTML entities use the Unicode code point (in decimal, if it's &#...;, or hex, if it's &#x...;). The Unicode code point is different from the UTF-8 or UTF-16 code. (That link explains a lot about how emoji and other characters are variously encoded, BTW! Good read.)
So we need to convert the UTF-16 codes used in your source data to Unicode code points. Referring to this Wikipedia article on UTF-16, I've verified how it's done. Each Unicode code point (our target) is a 20-bit number, or five hex digits. When going from Unicode to UTF-16, you split it up into two 10-bit numbers (the middle hex digit gets cut in half, with two of its bits going to each block), do some maths on them and get your result).
Going backwards, as you want to, it's done like this:
Your decimal UTF-16 number (which is in two separate blocks for now) is 55358 56599
Converting those blocks to hex (separately) gives 0x0d83e 0x0dd17
You subtract 0xd800 from the first block and 0xdc00 from the second to give 0x3e 0x117
Converting them to binary, padding them out to 10 bits and concatenating them, it's 0b0000 1111 1001 0001 0111
Then we convert that back to hex, which is 0x0f917
Finally, we add 0x10000, giving 0x1f917
Therefore, our (hex) HTML entity is 🤗. Or, in decimal, 🤗
So, to preprocess this dataset, you'll need to extract the existing numbers, use the algorithm above, then put the result back in (with one &#...;, not two).
Displaying emoji in R
As far as I'm aware, there's no solution to printing emoji in the R console: they always come out as "U0001f600" (or what have you). However, the packages I described above can help you plot emoji in some circumstances (I'm hoping to expand ggflags to display arbitrary full-colour emoji at some point). They can also help you search for emoji to get their codes, but they can't get names given the codes AFAIK. But maybe you could try importing the emoji list from emojilib into R and doing a join with your data frame, if you've extracted the emoji codes into a column, to get the English names.
JavaScript Solution
I had this exact same problem, but needed the solution in JavaScript, not R. Using rensa's comment above (hugely helpful!), I created the following code to solve this issue, and I just wanted to share it in case anyone else happens across this thread as I did, but needed it in JavaScript.
str.replace(/(&#\d+;){2}/g, function(match) {
match = match.replace(/&#/g,'').split(';');
var binFirst = (parseInt('0x' + parseInt(match[0]).toString(16)) - 0xd800).toString(2);
var binSecond = (parseInt('0x' + parseInt(match[1]).toString(16)) - 0xdc00).toString(2);
binFirst = '0000000000'.substr(binFirst.length) + binFirst;
binSecond = '0000000000'.substr(binSecond.length) + binSecond;
return '&#x' + (('0x' + (parseInt(binFirst + binSecond, 2).toString(16))) - (-0x10000)).toString(16) + ';';
});
And, here's a full snippet of it working if you'd like to run it:
var str = '😊😘😀😆😂😁'
str = str.replace(/(&#\d+;){2}/g, function(match) {
match = match.replace(/&#/g,'').split(';');
var binFirst = (parseInt('0x' + parseInt(match[0]).toString(16)) - 0xd800).toString(2);
var binSecond = (parseInt('0x' + parseInt(match[1]).toString(16)) - 0xdc00).toString(2);
binFirst = '0000000000'.substr(binFirst.length) + binFirst;
binSecond = '0000000000'.substr(binSecond.length) + binSecond;
return '&#x' + (('0x' + (parseInt(binFirst + binSecond, 2).toString(16))) - (-0x10000)).toString(16) + ';';
});
document.getElementById('result').innerHTML = str;
// 😊😘😀😆😂😁
// is turned into
// 😊😘😀😆😂😁
// which is rendered by the browser as the emojis
Original:<br>😊😘😀😆😂😁<br><br>
Result:<br>
<div id='result'></div>
My SMS XML Parser application is working great now, but it stalls out on large XML files so, I'm thinking about rewriting it in PHP. If/when I do, I'll post that code as well.
I've implemented the algorithm described by rensa above in R, and am sharing it here. I am happy to release the code snippet below under a CC0 dedication (i.e., putting this implementation into the public domain for free reuse).
This is a quick and unpolished implementation of rensa's algorithm, but it works!
utf16_double_dec_code_to_utf8 <- function(utf16_decimal_code){
string_elements <- str_match_all(utf16_decimal_code, "&#(.*?);")[[1]][,2]
string3a <- string_elements[1]
string3b <- string_elements[2]
string4a <- sprintf("0x0%x", as.numeric(string3a))
string4b <- sprintf("0x0%x", as.numeric(string3b))
string5a <- paste0(
# "0x",
as.hexmode(string4a) - 0xd800
)
string5b <- paste0(
# "0x",
as.hexmode(string4b) - 0xdc00
)
string6 <- paste0(
stringi::stri_pad(
paste0(BMS::hex2bin(string5a), collapse = ""),
10,
pad = "0"
) %>%
stringr::str_trunc(10, side = "left", ellipsis = ""),
stringi::stri_pad(
paste0(BMS::hex2bin(string5b), collapse = ""),
10,
pad = "0"
) %>%
stringr::str_trunc(10, side = "left", ellipsis = "")
)
string7 <- BMS::bin2hex(as.numeric(strsplit(string6, split = "")[[1]]))
string8 <- as.hexmode(string7) + 0x10000
unicode_pattern <- string8
unicode_pattern
}
make_unicode_entity <- function(x) {
paste0("\\U000", utf16_double_dec_code_to_utf8(x))
}
make_html_entity <- function(x) {
paste0("&#x", utf16_double_dec_code_to_utf8(x), ";")
}
# An example string, using the "hug" emoji:
example_string <- "test 🤗 test"
output_string <- stringr::str_replace_all(
example_string,
"(&#[0-9]*?;){2}", # Find all two-character "&#...;&#...;" codes.
make_unicode_entity
# make_html_entity
)
cat(output_string)
# To print Unicode string (doesn't display in R console, but can be copied and
# pasted elsewhere:
# (This assumes you've used 'make_unicode_entity' above in the str_replace_all
# call):
stringi::stri_unescape_unicode(output_string)
Translated Chad's JavaScript answer to Go since I too had the same issue, but needed a solution in Go.
https://play.golang.org/p/h9JBFzqcd90
package main
import (
"fmt"
"html"
"regexp"
"strconv"
"strings"
)
func main() {
emoji := "😊😘😀😆😂😁"
regexp := regexp.MustCompile(`(&#\d+;){2}`)
matches := regexp.FindAllString(emoji, -1)
var builder strings.Builder
for _, match := range matches {
s := strings.Replace(match, "&#", "", -1)
parts := strings.Split(s, ";")
a := parts[0]
b := parts[1]
c, err := strconv.Atoi(a)
if err != nil {
panic(err)
}
d, err := strconv.Atoi(b)
if err != nil {
panic(err)
}
c = c - 0xd800
d = d - 0xdc00
e := strconv.FormatInt(int64(c), 2)
f := strconv.FormatInt(int64(d), 2)
g := "0000000000"[2:len(e)] + e
h := "0000000000"[10:len(f)] + f
j, err := strconv.ParseInt(g + h, 2, 64)
if err != nil {
panic(err)
}
k := j + 0x10000
_, err = builder.WriteString("&#x" + strconv.FormatInt(k, 16) + ";")
if err != nil {
panic(err)
}
}
fmt.Println(html.UnescapeString(emoji))
emoji = html.UnescapeString(builder.String())
fmt.Println(emoji)
}
I've a json file with plone objects and there is one field of the objects giving me an error:
UnicodeDecodeError('ascii', '{"id":"aluminio-prata", "nome":"ALUM\xc3\x8dNIO PRATA", "num_demaos":0, "rendimento": 0.0, "unidade":"litros", "url":"", "particular":[], "profissional":[], "unidades":[]},', 36, 37, 'ordinal not in range(128)') (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: 'NoneType' object has no attribute 'getMethodAliases')
I already know witch field is, is the "title" from title = obj.pretty_title_or_id(), when I remove it from here its ok:
json += '{"id":"' + str(id) + '", "nome":"' + title + '", "num_demaos":' + str(num_demaos) + ', "rendimento": ' + str(rendimento) + ', "unidade":"' + str(unidade) + '", "url":"' + link_produto + '", "particular":' + arr_area_particular + ', "profissional":' + arr_area_profissional + ', "unidades":' + json_qtd + '},
but when I leave it I've got this error.
UnicodeDecodeError('ascii', '{"id":"aluminio-prata", "nome":"ALUM\xc3\x8dNIO PRATA", "num_demaos":0, "rendimento": 0.0, "unidade":"litros", "url":"", "particular":[], "profissional":[], "unidades":[]},', 36, 37, 'ordinal not in range(128)') (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: 'NoneType' object has no attribute 'getMethodAliases')
I'm going to assume that the error occurs when you're reading the JSON file.
Internally, Plone uses Python Unicode strings for nearly everything. If you read a string from a file, it will need to be decoded into Unicode before Plone can use it. If you give no instructions otherwise, Python will assume that the string was encoded as ASCII, and will attempt its Unicode conversion on that basis. It would be similar to writing:
unicode("ALUM\xc3\x8dNIO PRATA")
which will produce the same kind of error.
In fact, the string you're using was evidently encoded with the UTF-8 character set. That's evident from the "\xc3", and it also makes sense, because that's the character set Plone uses when it sends data to the outside world.
So, how do you fix this? You have to specify the character set that you wish to use when you convert to Unicode:
"ALUM\xc3\x8dNIO PRATA".decode('UTF8')
This gives you a Python Unicode string with no error.
So, after you've read your JSON file into a string (let's call it mystring), you will need to explicitly decode it by using mystring.decode('UTF8'). unicode(mystring, 'UTF8') is another form of the same operation.
As Steve already wrote do title.decode('utf8')
An Example illustrate the facts:
>>> u"Ä" == u"\xc4"
True # the native unicode char and escaped versions are the same
>>> "Ä" == u"\xc4"
False # the native unicode char is '\xc3\x84' in latin1
>>> "Ä".decode('utf8') == u"\xc4"
True # one can decode the string to get unicode
>>> "Ä" == "\xc4"
False # the native character and the escaped string are
# of course not equal ('\xc3\x84' != '\xc4').
I find this Thread very helpfull for Problems and Understanding with Encode/Decode of UTF-8.
I am having the following issue:
I am using an application that allows users to concatenate text to build a URL that passes parameters to an ASP page via GET method, i.e. something like:
http://myhostname/process.asp?param1=value1¶m2=value2
Problem is value1 and value2 can contain the ampersand symbol, which is not interpreted as a text character.
The most popular solution to this issue is to encode the URL, which is not an option for me because I cannot modify the program that builds the URL. I can modify the process.asp page, but not the program that concatenates the text fields and builds the URL.
Things I've tried to search for are:
How to encode a URL using javascript directly in the browser
How to change IIS default behaviour when reading an &
Alternative ways to pass parameters, i.e. something like passing them as a single string of characters separated with pipes
Hope you can give me some guidance.
You can read the entire query string and parse it yourself, like this:
q = Request.QueryString
a = Split(q, "=")
i = 1
For Each s In a
If i mod 2 = 0 Then
If InStr(s, "&") <> InStrRev(s, "&") Then
Response.Write "Value: " & Left(s, InStrRev(s, "&") - 1) & "<br/>"
hidingParam = Right(s, Len(s) - InStrRev(s, "&"))
Response.Write "PAramName: " & hidingParam & "<br/>"
i = i + 1
Else
Response.Write "Value: " & s & "<br/>"
End If
Else
Response.Write "PAramName: " & s & "<br/>"
End If
i = i + 1
Next
Result:
URL: ...?Q=abc&def&P=123 produces
PAramName: Q Value: abc&def PAramName: P Value: 123
Note that this is less than robust. I am only illustrating my idea. I didn't test with no &.
It also doens't handle multiple "=" characters (if that's a possiblity as well).
If there are 2 (or more) ampersands in-between the equals, then only the last one is a parameter separator. So, using your URL above, and assuming that value1 = "abc&def", and value2 = "123", then the URL will look like:
http://myhostname/process.asp?param1=abc&def¶m2=123
Notice there's 2 ampersands in-between the 2 equals. The last one will be your parameter separator, the rest are part of the value. And any ampersands after the last equals are also part of the value.
You'll have to dissect the incoming URL and apply the appropriate logic.
I have crystal reports and in formula editor, based on date.
If {?PDATERANGE1} <> "" AND {?PDATERANGE2} <> "" Then
ToText(DateValue({?PDATERANGE1}), "MMM d, yyyy") & " - " & ToText(DateValue({?PDATERANGE2}), "MMM d, yyyy")
I am calling this from ASP and PDATERANGE1 and PDATERANGE2 are "10/10/2001" and "10/12/2001".
I am getting the following error.
Bad date format string. Details: errorKind Error in File {B6624BE5-D6DA-469B-A635-9FE86B125492}.rpt: Error in formula dt_range: 'If {?PDATERANGE1} <> "" AND {?PDATERANGE2} <> "" Then ' Bad date format string. Details: errorKind
Could someone please tell me what is wrong? I am not a crystal reports developer.
If the values of {?PDATERANGE1} and {?PDATERANGE2} include double quotes - so that they are "10/10/2001" and "10/12/2001" respectively - then these double quotes need to be removed (so that they become 10/10/2001 and 10/12/2001 respectively).
I think it's because "" is not a valid date. Are the paramaters optional and you're checking whether they have values?
In crystal you have to use function hasvalue({?parameter}) though optional parameters tend to cause real issue so i avoid them like the plague!
Having an issue with type conversion in ASP classic.
heres my code:
Set trainingCost = Server.CreateObject("ADODB.Recordset")
strSQL3 = "SELECT cost1 FROM tblMain WHERE (Booked = 'Booked') AND (Paid IS NULL) AND (PaidDate BETWEEN '01/04/" & startyear & "' AND '31/03/" & endyear & "')"
trainingCost.Open strSQL3, Connection
trainingCost.movefirst
totalTrainCost = 0
do while not trainingCost.eof
trainCost = trainingCost("cost1")
If NOT isNull(trainCost) then
trainCostStr = CStr(trainCost)
trainCostStr = Replace(trainCostStr, "£", "")
trainCostStr = Replace(trainCostStr, ",", "")
totalTrainCost = totalTrainCost + CInt(trainCostStr)
end if
trainingCost.movenext
loop
trainingCost.close
when I run this I get the following error:
Microsoft VBScript runtime (0x800A000D)
Type mismatch: 'CInt'
/systems/RFT/v1.2/Extract.asp, line 43
which is "totalTrainCost = totalTrainCost + CInt(trainCostStr)"
Im guessing that the problem is to do with the String value being uncastable to Int in which case is there any way to catch this error? I havent worked with asp classic much so any help would be usefull
cheers
-EDIT-
the type of column cost1 is String as it may contain a number or a sequence of chars eg £10.00 or TBC
You have a couple of choices. You can be proactive by checking ahead of time whether the value is numeric using the IsNumeric function:
If IsNumeric(trainCostStr) Then
totalTrainCost = totalTrainCost + CInt(trainCostStr)
Else
' Do something appropriate
End If
...or you can be reactive by using error catching; in Classic ASP probably easiest to define a function and use On Error Resume Next:
Function ConvertToInt(val)
On Error Resume Next
ConvertToInt = CInt(val)
If Err.Number <> 0 Then
ConvertToInt = Empty
Err.Clear
End If
End Function
Or return 0 or Null or whatever suits you, then use it in your trainCost code.
Note that CInt expects an integer and will stop at the first non-digit, so "123.45" comes back as 123. Look at the other conversions, CDouble, CCur, etc.
Rather than casting to a string, why not use CCur (Cast as Currency) so that your commas and any currency symbols (I think) are effectively ignored while doing arithmetic operations?
Potentially solving the wrong problem, depends on the type of Cost1 within the database but the code is looping through the records to generate a total.
strSQL3 = "SELECT sum(cost1) FROM tblMain WHERE (Booked = 'Booked') AND (Paid IS NULL) AND (PaidDate BETWEEN '01/04/" & startyear & "' AND '31/03/" & endyear & "')"
trainingCost.Open strSQL3, Connection
etc and just read off the value as a total.
I don't see why the RS is being looped to generate a sum when the database can do that work for you. All the conversion work it has generated just looks artifical.
Heh heh. Classic ASP. You have my pity :) Anyway,
On error resume next
And then on the next line, check that it worked.
Though maybe you want CDouble. Is that a function? I can't remember.