I have a tab-separated file with records like these:
no Newcastle 1 NULL NULL NULL
uk Ньюкасл 1 NULL NULL NULL
ja ニューカッスル 1 NULL NULL NULL
To import them to sqlite, I use:
create table cities (country text, name text, count integer...);
.separator ' '
.import file cities
(where ' ' above contains a tab character). The file seems to be importing fine, but some non-ASCII chars get corrupted, especially at end of strings, and replaced with question marks.
no Newcastle 1 NULL NULL NULL
uk Ньюкасл 1 NULL NULL NULL
ja ニューカッ? 1 NULL NULL NULL
The corruption occurs at import. I tried using PRAGMA encoding = "UTF-8"; before importing, but it doesn't help. Any ideas?
update per request from the user CL below
Hexdump of a portion of the input file (the same portion as shown above with some additional numeric values which are irrelevant here; in the ASCII column the corrupted chars precede the "1" which follows the "ja"):
00008180 30 30 30 30 30 30 32 30 37 33 32 33 39 09 31 34 |0000002073239.14|
00008190 38 36 33 39 36 31 33 34 35 33 33 33 36 35 38 38 |8639613453336588|
000081a0 09 75 6b 09 d0 9d d1 8c d1 8e d0 ba d0 b0 d1 81 |.uk.............|
000081b0 d0 bb 09 31 09 4e 55 4c 4c 09 4e 55 4c 4c 09 4e |...1.NULL.NULL.N|
000081c0 55 4c 4c 0a 31 32 30 30 30 30 30 30 30 30 30 30 |ULL.120000000000|
000081d0 32 30 37 33 32 34 30 09 31 34 38 36 33 39 36 31 |2073240.14863961|
000081e0 33 34 35 33 33 33 36 35 38 38 09 6a 61 09 e3 83 |3453336588.ja...|
000081f0 8b e3 83 a5 e3 83 bc e3 82 ab e3 83 83 e3 82 b9 |................|
00008200 e3 83 ab 09 31 09 4e 55 4c 4c 09 4e 55 4c 4c 09 |....1.NULL.NULL.|
00008210 4e 55 4c 4c 0a 31 32 30 30 30 30 30 30 30 30 30 |NULL.12000000000|
00008220 30 32 30 37 33 32 34 31 09 31 34 38 36 33 39 36 |02073241.1486396|
Result of select hex(name) for the offending record:
hex(name)
--------------------------------------
31323030303030303030303032303733323430
This has nothing to do with importing; the sqlite3 command-line tool does not truncate strings correctly when using .width.
Related
I'm creating a low level HTTP server and I want to validate that my responses are correct.
Some of them are fine, but others fail. I have saved some responses as bytes and viewing them in HxD, and they look correct, but they obviously aren't in some way.
Here's the start of one HTTP response formatted as hex:
48 54 54 50 2F 31 2E 31 20 32 30 30 20 4F 4B 0D 0A 43 4F 4E 54 45 4E
54 2D 54 59 50 45 3A 20 69 6D 61 67 65 2F 70 6E 67 0D 0A 4C 41 53 54
2D 4D 4F 44 49 46 49 45 44 3A 20 53 75 6E 2C 20 30 31 20 4D 61 79 20
32 30 32 32 20 32 32 3A 32 34 3A 32 34 20 47 4D 54 0D 0A 41 43 43 45
50 54 2D 52 41 4E 47 45 53 3A 20 62 79 74 65 73 0D 0A 45 54 41 47 3A
20 22 37 35 37 35 35 33 36 61 61 35 64 64 38 31 3A 30 22 0D 0A 53 45
52 56 45 52 3A 20 4D 69 63 72 6F 73 6F 66 74 2D 49 49 53 2F 31 30 2E
30 0D 0A 58 2D 50 4F 57 45 52 45 44 2D 42 59 3A 20 41 53 50 2E 4E 45
54 0D 0A 44 41 54 45 3A 20 57 65 64 2C 20 30 34 20 4D 61 79 20 32 30
32 32 20 30 39 3A 30 31 3A 34 38 20 47 4D 54 0D 0A 43 4F 4E 54 45 4E
54 2D 4C 45 4E 47 54 48 3A 20 39 38 37 35 37 0D 0A 43 6F 6E 6E 65 63
74 69 6F 6E 3A 20 63 6C 6F 73 65 0D 0A 0D 0A 0D 0A 89 50 4E 47
How can I validate that the response I'm creating is valid according to RFC 2616?
I'm working in .NET but I don't mind if it's an online tool or a different platform.
Edit:
I've solved my issue with this particular example (3 crlfs instead of 2 between head and body) but I'd still like to know how I can validate these requests automatically.
I ended up achieving this with this nuget package and the following code:
if (System.Diagnostics.Debugger.IsAttached)
{
using (var handler = new HttpParserDelegate())
using (var parser = new HttpCombinedParser(handler))
{
parser.Execute(arr);
}
if (!handler.HttpRequestResponse.IsEndOfMessage)
{
System.Diagnostics.Debugger.Break();
throw new InvalidOperationException($"An HTTP response was serialized but it is not a valid response");
}
}
I'd like to grab the file names in a directory on an ssh server using the R SSH package and save the output as a dataframe. After establishing a connection, I've tried
files <- ssh_exec_internal(ssh_connection, command = 'ls /directory_of_interest')
This gives an output that does not resemble a list of file names. Any help would be appreciated
$status
[1] 0
$stdout
[1] 32 30 32 30 30 38 32 36 2e 30 33 35 0a 32 30 32 30 30 38 32 37 2e 30 33 35 0a 32 30 32 30 30 38 32 38 2e 30 33 35 0a 32 30 32 30 30 38 32 39 2e
[49] 30 33 35 0a 32 30 32 30 30 38 33 30 2e 30 33 35 0a 32 30 32 30 30 38 33 31 2e 30 33 35 0a 32 30 32 30 30 39 30 31 2e 30 33 35 0a 32 30 32 30 30
[97] 39 30 32 2e 30 33 35 0a 32 30 32 30 30 39 30 33 2e 30 33 35 0a 32 30 32 30 30 39 30 34 2e 30 33 35 0a 32 30 32 30 30 39 30 35 2e 30 33 35 0a 32
[145] 30 32 30 30 39 30 36 2e 30 33 35 0a 32 30 32 30 30 39 30 37 2e 30 33 35 0a 32 30 32 30 30 39 30 38
[ reached getOption("max.print") -- omitted 560 entries ]
$stderr
raw(0)
This worked for me:
rawToChar(as.raw(strtoi(files$stdout, 16L))) %>% read_table(cat(.), col_names="file")
The files in a server directory can be listed as follows:
files <- ssh_exec_internal(ssh_connection, "ls /directory_of_interest") %>%
.[["stdout"]] %>%
rawToChar() %>%
strsplit("\n") %>%
unlist()
Hi i am having the following Rs485 data in that i know the value but unable to decrypt the data i am tried in taking 2's compliment but it comes to close but not get the correct data
3A 30 31 30 33 30 32 30 31 38 42 36 45 0D 0A
in above rs 485 data value is=395 that is 3842 =395 and 3645 =LRC and 0D0A=End character
3A 30 31 30 33 30 32 30 31 38 46 36 41 0D 0A
in above rs 485 data value is=399 that is 3846=399 and 3641 =LRC and 0D0A=End character
3A 30 31 30 33 30 32 30 31 39 30 36 39 0D 0A
in above rs 485 data value is=400 that is 3930 =400 and 3639 =LRC and 0D0A=End character
3A 30 31 30 33 30 32 30 31 39 32 36 37 0D 0A
in above rs 485 data value is=402 that is 3932 =402 and 3637 =LRC and 0D0A=End character
Please guide me to get LRC value and data.Lrc value is summing of data and taking 2's compliment negotion but i don't know how to take lrc value.Please help me to get data and LRC value calculation
Using the first line as an example, convert the pairs of ASCII bytes in to single hex bytes:
3A 30 31 30 33 30 32 30 31 38 42 xx xx 0D 0A ASCII
01 03 02 01 8B hex values converted to binary bytes
01 + 03 + 02 + 01 + 8B = 92 sum up the hex values
00 - 92 = 6E take the two's complement
36 45 convert the 6E to ASCII
3A 30 31 30 33 30 32 30 31 38 42 36 45 0D 0A include the 36 45 into the message
For the last line:
3A 30 31 30 33 30 32 30 31 39 32 xx xx 0D 0A
01 + 03 + 02 + 01 + 92 = 99
00 - 99 = 67
3A 30 31 30 33 30 32 30 31 39 32 36 37 0D 0A
I am trying to get readings from an ultrasonic thickness tester without using the included software, but I haven't been able to interpret the measurement readings from the tester.
Below are the hex values that are sent from the tester, and the corresponding decimal value that is displayed in the software.
3A 30 62 30 30 44 39 37 30 30 65 36 0D 0A 1.51
3A 30 62 30 30 44 66 61 30 30 34 39 0D 0A 2.50
3A 30 62 30 30 44 37 62 30 31 63 62 0D 0A 3.79
3A 30 62 30 30 44 38 32 30 31 64 32 0D 0A 3.86
3A 30 62 30 30 44 63 37 30 31 31 37 0D 0A 4.55
3A 30 62 30 30 44 39 34 30 62 65 65 0D 0A 29.64
3A 30 62 30 30 44 39 63 30 62 66 36 0D 0A 29.72
3A 30 62 30 30 44 63 61 30 62 32 34 0D 0A 30.18
3A 30 62 30 30 44 65 64 30 62 34 37 0D 0A 30.53
3A 30 62 30 30 44 38 65 33 65 31 62 0D 0A 160.14
3A 30 62 30 30 44 64 66 33 65 36 63 0D 0A 160.95
Any help on how I get from the hex values to the decimal value would be very much appreciated.
ASCII equivalent (I have omitted CR and LF which are at the end of each line) :
:0b00D9700e6 1.51
:0b00Dfa0049 2.50
:0b00D7b01cb 3.79
:0b00D8201d2 3.86
:0b00Dc70117 4.55
:0b00D940bee 29.64
:0b00D9c0bf6 29.72
:0b00Dca0b24 30.18
:0b00Ded0b47 30.53
:0b00D8e3e1b 160.14
:0b00Ddf3e6c 160.95
Finally figured it out:
The data is the two bytes after :0b00D, LSB first, multiplied by 100.
ASCII Hex Decimal
:0b00D9700e6 0097 1.51
:0b00Dfa0049 00fa 2.50
:0b00D7b01cb 017b 3.79
:0b00D8201d2 0182 3.86
:0b00Dc70117 01c7 4.55
:0b00D940bee 0b94 29.64
:0b00D9c0bf6 0b9c 29.72
:0b00Dca0b24 0bca 30.18
:0b00Ded0b47 0bed 30.53
:0b00D8e3e1b 3e8e 160.14
:0b00Ddf3e6c 3edf 160.95
I have a smart electricity meter that connects to the pc through RS485 serial cable. This hardware has a software that you can read its internal registers and view them on the software. When I tried to read the registers of the meter through the software, I ran a serial port monitor software to sniff the bytes being sent and received to the meter.
This is the sniffing result:
Port opened
Request: 9/18/2012 7:04:04 PM.28064
2F 3F 21 0D 0A /?!..
Answer: 9/18/2012 7:04:04 PM.70264 (+0.4219 seconds)
2F 45 4C 53 35 5C 40 56 38 2E 32 30 20 20 20 20 /ELS5\#V8.20
20 20 20 20 20 0D 0A ..
Request: 9/18/2012 7:04:05 PM.53064 (+0.0156 seconds)
06 30 35 30 0D 0A .050..
Answer: 9/18/2012 7:04:05 PM.06264 (+0.5313 seconds)
02 43 2E 31 2E 30 28 33 35 30 36 34 32 36 37 29 .C.1.0(35064267)
0D 0A 30 2E 30 2E 30 28 33 35 30 36 34 32 36 37 ..0.0.0(35064267
30 30 30 30 30 30 30 30 20 20 20 20 20 20 20 20 00000000
29 0D 0A 30 2E 39 2E 31 28 31 39 30 32 33 37 29 )..0.9.1(190237)
0D 0A 30 2E 39 2E 32 28 31 38 30 39 31 32 29 0D ..0.9.2(180912).
0A 31 2E 38 2E 30 28 30 30 30 30 30 32 36 2A 6B .1.8.0(0000026*k
57 68 29 0D 0A 31 2E 38 2E 30 2A 30 31 28 30 30 Wh)..1.8.0*01(00
30 30 30 32 36 2A 6B 57 68 29 0D 0A 31 2E 38 2E 00026*kWh)..1.8.
30 2A 30 32 28 30 30 30 30 30 32 36 2A 6B 57 68 0*02(0000026*kWh
29 0D 0A 31 2E 38 2E 30 2A 30 33 28 30 30 30 30 )..1.8.0*03(0000
30 32 36 2A 6B 57 68 29 0D 0A 31 2E 38 2E 30 2A 026*kWh)..1.8.0*
30 34 28 30 30 30 30 30 30 30 2A 6B 57 68 29 0D 04(0000000*kWh).
0A 32 2E 38 2E 30 28 30 30 30 30 30 30 30 2A 6B .2.8.0(0000000*k
57 68 29 0D 0A 32 2E 38 2E 30 2A 30 31 28 30 30 Wh)..2.8.0*01(00
30 30 30 30 30 2A 6B 57 68 29 0D 0A 32 2E 38 2E 00000*kWh)..2.8.
30 2A 30 32 28 30 30 30 30 30 30 30 2A 6B 57 68 0*02(0000000*kWh
29 0D 0A 32 2E 38 2E 30 2A 30 33 28 30 30 30 30 )..2.8.0*03(0000
30 30 30 2A 6B 57 68 29 0D 0A 32 2E 38 2E 30 2A 000*kWh)..2.8.0*
30 34 28 30 30 30 30 30 30 30 2A 6B 57 68 29 0D 04(0000000*kWh).
0A 35 2E 38 2E 30 28 30 30 30 30 30 32 36 2A 6B .5.8.0(0000026*k
76 61 72 68 29 0D 0A 35 2E 38 2E 30 2A 30 31 28 varh)..5.8.0*01(
30 30 30 30 30 32 36 2A 6B 76 61 72 68 29 0D 0A 0000026*kvarh)..
35 2E 38 2E 30 2A 30 32 28 30 30 30 30 30 32 36 5.8.0*02(0000026
2A 6B 76 61 72 68 29 0D 0A 35 2E 38 2E 30 2A 30 *kvarh)..5.8.0*0
33 28 30 30 30 30 30 32 36 2A 6B 76 61 72 68 29 3(0000026*kvarh)
0D 0A 35 2E 38 2E 30 2A 30 34 28 30 30 30 30 30 ..5.8.0*04(00000
30 30 2A 6B 76 61 72 68 29 0D 0A 36 2E 38 2E 30 00*kvarh)..6.8.0
28 30 30 30 30 30 30 30 2A 6B 76 61 72 68 29 0D (0000000*kvarh).
0A 36 2E 38 2E 30 2A 30 31 28 30 30 30 30 30 30 .6.8.0*01(000000
30 2A 6B 76 61 72 68 29 0D 0A 36 2E 38 2E 30 2A 0*kvarh)..6.8.0*
30 32 28 30 30 30 30 30 30 30 2A 6B 76 61 72 68 02(0000000*kvarh
29 0D 0A 36 2E 38 2E 30 2A 30 33 28 30 30 30 30 )..6.8.0*03(0000
30 30 30 2A 6B 76 61 72 68 29 0D 0A 36 2E 38 2E 000*kvarh)..6.8.
30 2A 30 34 28 30 30 30 30 30 30 30 2A 6B 76 61 0*04(0000000*kva
72 68 29 0D 0A 37 2E 38 2E 30 28 30 30 30 30 30 rh)..7.8.0(00000
30 30 2A 6B 76 61 72 68 29 0D 0A 37 2E 38 2E 30 00*kvarh)..7.8.0
2A 30 31 28 30 30 30 30 30 30 30 2A 6B 76 61 72 *01(0000000*kvar
68 29 0D 0A 37 2E 38 2E 30 2A 30 32 28 30 30 30 h)..7.8.0*02(000
30 30 30 30 2A 6B 76 61 72 68 29 0D 0A 37 2E 38 0000*kvarh)..7.8
2E 30 2A 30 33 28 30 30 30 30 30 30 30 2A 6B 76 .0*03(0000000*kv
61 72 68 29 0D 0A 37 2E 38 2E 30 2A 30 34 28 30 arh)..7.8.0*04(0
30 30 30 30 30 30 2A 6B 76 61 72 68 29 0D 0A 39 000000*kvarh)..9
2E 36 2E 31 28 30 30 2E 30 30 2A 6B 56 41 29 28 .6.1(00.00*kVA)(
30 30 30 30 30 30 30 30 30 30 29 0D 0A 39 2E 36 0000000000)..9.6
2E 31 2A 30 31 28 30 30 2E 30 30 2A 6B 56 41 29 .1*01(00.00*kVA)
28 30 30 30 30 30 30 30 30 30 30 29 0D 0A 39 2E (0000000000)..9.
36 2E 31 2A 30 32 28 30 30 2E 30 30 2A 6B 56 41 6.1*02(00.00*kVA
29 28 30 30 30 30 30 30 30 30 30 30 29 0D 0A 39 )(0000000000)..9
2E 36 2E 31 2A 30 33 28 30 30 2E 37 34 2A 6B 56 .6.1*03(00.74*kV
41 29 28 30 34 30 35 31 32 30 30 30 30 29 0D 0A A)(0405120000)..
39 2E 36 2E 31 2A 30 34 28 30 30 2E 30 34 2A 6B 9.6.1*04(00.04*k
56 41 29 28 33 30 30 34 31 32 32 33 30 30 29 0D VA)(3004122300).
0A 46 2E 46 28 30 30 30 30 30 30 30 30 29 0D 0A .F.F(00000000)..
46 2E 46 2E 31 28 30 30 30 31 30 30 30 30 29 0D F.F.1(00010000).
0A 46 2E 46 2E 32 28 30 30 30 30 30 30 30 30 29 .F.F.2(00000000)
0D 0A 46 2E 46 2E 33 28 30 30 30 30 30 30 30 30 ..F.F.3(00000000
29 0D 0A 43 2E 33 2E 30 28 30 30 30 30 30 30 30 )..C.3.0(0000000
30 29 0D 0A 43 2E 35 36 28 30 30 33 37 32 35 29 0)..C.56(003725)
0D 0A 21 0D 0A 03 6F ..!...o
Port closed
My question: Is it possible to send the request bytes (which were found above) to the meter and then get the answer bytes and saved them on pc? Basically, is it possible to communicate with the meter by sending the hex bytes and receiving the answer? or there are some other steps that is necessary to do before sending hex bytes?
Note: My aim is to develop a basic and simpler software as the software provided with the meter.