How to convert variable to RDA format in memory - r

Currently, I convert data table into rda format by using save() like so
save(data_table,file="data.rda")
But I don't want to write to disk to get .rda format of the variable.
Instead, I would like to have the output(rda bytes) be written straight into a variable.
If it were in other programming languages, It would be something like
Switching from
FileStream fs = new FileStream("data.rda","w")
Save(data_table,fs)
to
byte[] buffer = new byte[data_table.getBytes()]
Save(data_table,buffer);
Is there a way to do something like this in R?

This is possible with raw connections
rc <- rawConnection(raw(0),"wb")
save(iris,file=rc)
v1 <- rawConnectionValue(rc)
close(rc)
head(v1,80)
#> [1] 52 44 58 32 0a 58 0a 00 00 00 02 00 03 01 01 00 02 03 00 00
#> [21] 00 04 02 00 00 00 01 00 04 00 09 00 00 00 04 69 72 69 73 00
#> [41] 00 03 13 00 00 00 05 00 00 00 0e 00 00 00 96 40 14 66 66 66
#> [61] 66 66 66 40 13 99 99 99 99 99 9a 40 12 cc cc cc cc cc cd 40
Compare to writing to a temporary file and reading back
save(iris,file="temp.rda",compress=FALSE)
v2 = readBin("temp.rda", raw(), file.info("temp.rda")[1,"size"])
identical(v1,v2)
#> [1] TRUE
Note that writing to file by default uses compression and writing to a raw connection by default does not, hence the compress=FALSE argument.
See also serialize, which may be more suitable, depending on your purpose
v3 <- serialize(iris,NULL)
Note that identical(v1,v3)==FALSE. Indeed, v3 is identical instead to using saveRDS in place of save above. The encoding/content is similar
head(v3,80)
#> [1] 58 0a 00 00 00 02 00 03 01 01 00 02 03 00 00 00 03 13 00 00
#> [21] 00 05 00 00 00 0e 00 00 00 96 40 14 66 66 66 66 66 66 40 13
#> [41] 99 99 99 99 99 9a 40 12 cc cc cc cc cc cd 40 12 66 66 66 66
#> [61] 66 66 40 14 00 00 00 00 00 00 40 15 99 99 99 99 99 9a 40 12
And it is easy to recover the data
identical(unserialize(v3),iris)
#> [1] TRUE

Related

How to get Windows 10 to autoload WinUSB as driver using MS_OS_20 BOS descriptor

I am struggling to get Windows to load the default WinUSB driver for my device. Please note that I am looking for a solution that is using BOS descriptor (and not the old 0xEE string index).
The device enumerates and Windows tells me that it is installing the device, but the WinUSB driver is not loaded. I have tried everything that I can think of, but still I can't get Windows to load the driver. I even uninstall the device and delete the USB flags in the registry whenever I re-try, but to no avail. Is there anyone who can help me to get this to work?
I don't want WebUSB capabilities or anything additional. This is a non-composite device.
This is my BOS descriptor (as sent over USB):
05 0F 21 00 01 1C 10 05 00 DF 60 DD D8 89
45 C7 4C 9C D2 65 9D 9E 64 8A 9F 00 00 03
06 B2 00 01 00
And this my BOS descriptor set:
0A 00 00 00 00 00 03 06 B2 00 08 00 01 00 ..............
00 00 A8 00 08 00 02 00 00 00 A0 00 14 00 ..............
03 00 57 49 4E 55 53 42 00 00 00 00 00 00 ..WINUSB......
00 00 00 00 84 00 04 00 07 00 2A 00 44 00 ..........*.D.
65 00 76 00 69 00 63 00 65 00 49 00 6E 00 e.v.i.c.e.I.n.
74 00 65 00 72 00 66 00 61 00 63 00 65 00 t.e.r.f.a.c.e.
47 00 55 00 49 00 44 00 73 00 00 00 50 00 G.U.I.D.s...P.
7B 00 46 00 37 00 32 00 46 00 45 00 30 00 {.F.7.2.F.E.0.
44 00 34 00 2D 00 43 00 42 00 43 00 42 00 D.4.-.C.B.C.B.
2D 00 34 00 30 00 37 00 44 00 2D 00 38 00 -.4.0.7.D.-.8.
38 00 31 00 34 00 2D 00 39 00 45 00 44 00 8.1.4.-.9.E.D.
36 00 37 00 33 00 44 00 30 00 44 00 44 00 6.7.3.D.0.D.D.
36 00 42 00 7D 00 00 00 00 00 6.B.}.....
The layout is:
typedef struct _SMSOS20DescriptorSet
{
SDeviceDescSetHeader sDescriptorSetHeader;
SConfigurationSubsetHeader sConfSubsetHeader;
SFunctionSubsetHeader sFuncSubsetHeader;
SDeviceCompatibleIdDescriptor sCompIdDescriptor;
SDeviceRegDescDeviceInterfaceGUID sRegistryDescDevInterfaceGuid;
} SMSOS20DescriptorSet;
I have follewed these guides and doc:
https://learn.microsoft.com/en-us/windows-hardware/drivers/usbcon/automatic-installation-of-winusb#winusb-device-installation-by-using-the-in-box-winusbinf
MS_OS_2_0_desc.docx
https://thewindowsupdate.com/2018/10/12/how-to-install-winusb-sys-without-a-custom-inf/
https://learn.microsoft.com/en-us/windows-hardware/drivers/usbcon/winusb-installation#automatic-installation-of--winusb-without-an-inf-file
UPDATE:
when you have a non-composite device that only has a single a configuration, then you are not use any subset headers (neither 'Configuration subset header' nor 'Function subset header'). So, the correct layout in this case is:
typedef struct _SMSOS20DescriptorSet
{
SDeviceDescSetHeader sDescriptorSetHeader;
SDeviceCompatibleIdDescriptor sCompIdDescriptor;
SDeviceRegDescDeviceInterfaceGUID sRegistryDescDevInterfaceGuid;
} SMSOS20DescriptorSet;
UPDATE: when you have a non-composite device that only has a single a configuration, then you are not use any subset headers (neither 'Configuration subset header' nor 'Function subset header'). So, the correct layout in this case is:
typedef struct _SMSOS20DescriptorSet
{
SDeviceDescSetHeader sDescriptorSetHeader;
SDeviceCompatibleIdDescriptor sCompIdDescriptor;
SDeviceRegDescDeviceInterfaceGUID sRegistryDescDevInterfaceGuid;
} SMSOS20DescriptorSet;

unix - how to remove a byte from a recursive position in a ebcdic file

I have an ebcdic file and need to remove a byte that recurs every n bytes:
For example:
00 00 00 25 00 0C 25 00 00 00 00 00 0C 25 00 78 25 69 67 4C 25 00 78 90 69 67 4C 25
I need to remove "25" from position 7, 14, 21, 28, the output would be:
00 00 00 25 00 0C 00 00 00 00 00 0C 00 78 25 69 67 4C 00 78 90 69 67 4C
I tried to use cut:
l=`stat -c%s myfile`
for (( i = 1 ; i <= $l ; i += 7 )) ;
do
m=`expr $i + 5`
cat myfile | cut -c$i-$m` >> fileout
done
But the output is not what I expect:
00 00 00 25 00 0C 0A 00 00 00 00 00 0C 0A 00 78 25 69 67 4C 0A 00 78 90 69 67 4C 0A
Can somebody help me?

Float to hex conversion - reverse engineering

I'm trying to do some reverse engineering on my heating system. Monitoring the CAN BUS results in receiving hexademical strings, for example:
00 D0 68 D6 86 83 61 8F
61 C0 02 5C 12 B5 02 5C
12 78 04 39 04 03 05 02
05 C4 04 5C 12 5C 12 5C
12 5C 12 D0 68 00 00 00
00 18 08 37 D2 00 00 00
00 00 00 00 00 15 75 F2
F0 01 00 01 00 00 00 1F
I know that for example the temperature value of 22.5°C should be somewhere in there.
So far I have tried to look for the following conversions:
Possibility 1: ascii to hex
22.5 = 32 32 2E 35
Possibility 2: float to hex conversion
22.5 = 0x 41 b4 00 00
However none of these resulted in a match.
What would be other possibilities to converted a float to a hex string?
Thx
note: the given string is just a small part of my can sniffer so don't look for 22.5 in my given string here. I'm just looking for other possible conversions.

QIcon/QImage memory leak?

I'm facing some memory leaks using QIcon (Qt 4.8.1 with MSVC 2008).
This happens with simple QAction (like menu entries, even those generated automatically by Qt Designer) or container items (like QTreeWidgetItem).
For example:
QTreeWidgetItem *newItem = new QTreeWidgetItem();
newItem->setText(0, "Item");
// This causes a memory leak!
newItem->setIcon(0, QIcon("D:\\Dnl\\QtSandBoxApp\\Resources\\dataset2.png"));
treeWidget->addTopLevelItem(newItem);
After a lot of debugging I discovered that QIcon internally uses a QImage, which seems to be correctly destroyed (the reference counter drops to zero).
The simplest example I could produce is the following:
#include "stdafx.h"
#include <QtGui/QApplication>
#include <crtdbg.h>
int main(int argc, char *argv[])
{
_CrtSetDbgFlag(_CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF);
QApplication a(argc, argv);
//QPixmapCache::setCacheLimit(0);
// MEMORY LEAK!
// Internally uses QImageReader
QImage image("D:\\Dnl\\QtSandBoxApp\\Resources\\dataset2.png");
// NO MEMORY LEAK!
//QImage image(QSize(16, 16), QImage::Format_ARGB32);
return 0;
}
Am I missing something?
This is a very annoying problem, because it prevents proper memory leak tracking (i.e. caused by the application, not Qt).
EDIT: Here's the memory leak report
Detected memory leaks!
Dumping objects ->
{5637} normal block at 0x00A46020, 76 bytes long.
Data: <d ,g ^ L Bg> 64 D2 2C 67 E8 5E A4 00 00 00 00 00 4C B9 42 67
{5636} normal block at 0x00A45EE8, 12 bytes long.
Data: < z ` z > 14 7A D3 00 20 60 A4 00 00 7A D3 00
{5634} normal block at 0x00A46148, 128 bytes long.
Data: < 6 6 Za > 01 00 00 00 36 00 00 00 36 00 00 00 5A 61 A4 00
{5614} normal block at 0x00A45DC8, 76 bytes long.
Data: <d ,g x L Bg> 64 D2 2C 67 E8 78 A4 00 00 00 00 00 4C B9 42 67
{5613} normal block at 0x00A478E8, 12 bytes long.
Data: < ] > A4 96 CA 00 C8 5D A4 00 90 96 CA 00
{5611} normal block at 0x00A45C48, 128 bytes long.
Data: < 6 5 Z\ > 01 00 00 00 36 00 00 00 35 00 00 00 5A 5C A4 00
{5591} normal block at 0x00A477C8, 76 bytes long.
Data: <d ,g u L Bg> 64 D2 2C 67 10 75 A4 00 00 00 00 00 4C B9 42 67
{5590} normal block at 0x00A47510, 12 bytes long.
Data: < w > B8 96 C6 00 C8 77 A4 00 A4 96 C6 00
{5588} normal block at 0x00A45B88, 128 bytes long.
Data: < 6 5 [ > 01 00 00 00 36 00 00 00 35 00 00 00 9A 5B A4 00
{5566} normal block at 0x00A473F0, 76 bytes long.
Data: <d ,g s L Bg> 64 D2 2C 67 A8 73 A4 00 00 00 00 00 4C B9 42 67
{5565} normal block at 0x00A473A8, 12 bytes long.
Data: < s > E0 D7 C3 00 F0 73 A4 00 CC D7 C3 00
{5563} normal block at 0x00A471E0, 128 bytes long.
Data: < 6 5 q > 01 00 00 00 36 00 00 00 35 00 00 00 F2 71 A4 00
{5543} normal block at 0x00A47008, 76 bytes long.
Data: <d ,g o L Bg> 64 D2 2C 67 C0 6F A4 00 00 00 00 00 4C B9 42 67
{5542} normal block at 0x00A46FC0, 12 bytes long.
Data: <La p 8a > 4C 61 BB 00 08 70 A4 00 38 61 BB 00
{5540} normal block at 0x00A46DF8, 128 bytes long.
Data: < 6 6 n > 01 00 00 00 36 00 00 00 36 00 00 00 0A 6E A4 00
{5520} normal block at 0x003ED9A8, 76 bytes long.
Data: <d ,g` > L Bg> 64 D2 2C 67 60 D9 3E 00 00 00 00 00 4C B9 42 67
{5519} normal block at 0x003ED960, 12 bytes long.
Data: < > > A4 C6 B4 00 A8 D9 3E 00 90 C6 B4 00
{5517} normal block at 0x00A46D38, 128 bytes long.
Data: < 6 5 Jm > 01 00 00 00 36 00 00 00 35 00 00 00 4A 6D A4 00
{5496} normal block at 0x003ED8D8, 76 bytes long.
Data: <d ,g > L Bg> 64 D2 2C 67 80 D7 3E 00 00 00 00 00 4C B9 42 67
{5495} normal block at 0x003ED780, 12 bytes long.
Data: << > ( > 3C CA 00 10 D8 D8 3E 00 28 CA 00 10
{5491} normal block at 0x00A412A0, 128 bytes long.
Data: < 6 5 > 01 00 00 00 36 00 00 00 35 00 00 00 B2 12 A4 00
{5260} normal block at 0x00A48448, 64 bytes long.
Data: < Z > 01 00 00 00 16 00 00 00 13 00 00 00 5A 84 A4 00
{5233} normal block at 0x00A48380, 56 bytes long.
Data: < Ha Bg> 00 00 CD 00 C0 82 A4 00 48 61 A4 00 04 BA 42 67
{5231} normal block at 0x00A482C0, 128 bytes long.
Data: < 6 6 > 01 00 00 00 36 00 00 00 36 00 00 00 D2 82 A4 00
{4681} normal block at 0x00A48150, 64 bytes long.
Data: < b > 01 00 00 00 16 00 00 00 13 00 00 00 62 81 A4 00
{4655} normal block at 0x00A47FB8, 56 bytes long.
Data: < ~ H\ Bg> 00 00 CA 00 F8 7E A4 00 48 5C A4 00 04 BA 42 67
{4653} normal block at 0x00A47EF8, 126 bytes long.
Data: < 5 5 > 01 00 00 00 35 00 00 00 35 00 00 00 0A 7F A4 00
{4102} normal block at 0x00A47DB8, 64 bytes long.
Data: < } > 01 00 00 00 16 00 00 00 13 00 00 00 CA 7D A4 00
{4075} normal block at 0x00A47D40, 56 bytes long.
Data: < { [ Bg> 00 00 C6 00 D8 7B A4 00 88 5B A4 00 04 BA 42 67
{4073} normal block at 0x00A47BD8, 126 bytes long.
Data: < 5 5 { > 01 00 00 00 35 00 00 00 35 00 00 00 EA 7B A4 00
{3522} normal block at 0x00A479F8, 64 bytes long.
Data: < z > 01 00 00 00 16 00 00 00 13 00 00 00 0A 7A A4 00
{3496} normal block at 0x00A47AA0, 56 bytes long.
Data: < 8y q Bg> 00 00 BD 00 38 79 A4 00 E0 71 A4 00 04 BA 42 67
{3494} normal block at 0x00A47938, 126 bytes long.
Data: < 5 5 Jy > 01 00 00 00 35 00 00 00 35 00 00 00 4A 79 A4 00
{2943} normal block at 0x00A41220, 64 bytes long.
Data: < 2 > 01 00 00 00 16 00 00 00 13 00 00 00 32 12 A4 00
{2917} normal block at 0x00A40418, 56 bytes long.
Data: < X m Bg> 00 00 B6 00 58 03 A4 00 F8 6D A4 00 04 BA 42 67
{2915} normal block at 0x00A40358, 128 bytes long.
Data: < 6 6 j > 01 00 00 00 36 00 00 00 36 00 00 00 6A 03 A4 00
{2367} normal block at 0x00A40760, 64 bytes long.
Data: < r > 01 00 00 00 16 00 00 00 13 00 00 00 72 07 A4 00
{2341} normal block at 0x00A401D0, 56 bytes long.
Data: < h 8m Bg> 00 00 B4 00 68 00 A4 00 38 6D A4 00 04 BA 42 67
{2339} normal block at 0x00A40068, 126 bytes long.
Data: < 5 5 z > 01 00 00 00 35 00 00 00 35 00 00 00 7A 00 A4 00
{1788} normal block at 0x00A40820, 64 bytes long.
Data: < 2 > 01 00 00 00 16 00 00 00 13 00 00 00 32 08 A4 00
{1762} normal block at 0x00A417B0, 56 bytes long.
Data: < Bg> 00 00 00 10 B0 18 A4 00 A0 12 A4 00 04 BA 42 67
{1760} normal block at 0x00A418B0, 126 bytes long.
Data: < 5 5 > 01 00 00 00 35 00 00 00 35 00 00 00 C2 18 A4 00
{252} normal block at 0x003E8A08, 40 bytes long.
Data: < #B > 00 00 00 00 CC CD CD CD 40 42 0F 00 00 00 00 00
{251} normal block at 0x003E8950, 120 bytes long.
Data: <Lc-g > L Bg> 4C 63 2D 67 08 89 3E 00 00 00 00 00 4C B9 42 67
{250} normal block at 0x003E8908, 8 bytes long.
Data: < c-gP > > 0C 63 2D 67 50 89 3E 00
{249} normal block at 0x003E88A0, 40 bytes long.
Data: < #B > 00 00 00 00 CC CD CD CD 40 42 0F 00 00 00 00 00
{248} normal block at 0x003E8828, 60 bytes long.
Data: < > > 07 00 00 00 08 89 3E 00 80 06 00 00 00 CD CD CD
Object dump complete.
The program '[2808] QtSandBoxApp.exe: Native' has exited with code 0 (0x0).
I solved the problem.
It seems that using CrtDbg library APIs is not a good idea with Qt.
Quote from http://winfig.com/?p=154:
Qt allocates memory in it’s DLLs and this memory is released when the
DLLs are unloaded. Unfortunately this is after Visual Studio reports
the supposed leaks.
This is the case, infact when I request the QImage from disk, the following modules are loaded at runtime:
'QtSandBoxApp.exe': Loaded 'C:\Libraries\Qt\4.8.1\plugins\imageformats\qgifd4.dll'
'QtSandBoxApp.exe': Loaded 'C:\Libraries\Qt\4.8.1\plugins\imageformats\qicod4.dll'
'QtSandBoxApp.exe': Loaded 'C:\Libraries\Qt\4.8.1\plugins\imageformats\qjpegd4.dll'
'QtSandBoxApp.exe': Loaded 'C:\Libraries\Qt\4.8.1\plugins\imageformats\qmngd4.dll'
'QtSandBoxApp.exe': Loaded 'C:\Libraries\Qt\4.8.1\plugins\imageformats\qsvgd4.dll'
'QtSandBoxApp.exe': Loaded 'C:\Libraries\Qt\4.8.1\bin\QtSvgd4.dll', Symbols loaded.
'QtSandBoxApp.exe': Loaded 'C:\Libraries\Qt\4.8.1\bin\QtXmld4.dll', Symbols loaded.
'QtSandBoxApp.exe': Loaded 'C:\Libraries\Qt\4.8.1\plugins\imageformats\qtgad4.dll'
'QtSandBoxApp.exe': Loaded 'C:\Libraries\Qt\4.8.1\plugins\imageformats\qtiffd4.dll'
Visual Leak Detector correctly reports no memory leaks.
P.S: This wiki entry from QtCentre is misleading :P
An alternative way to avoid the memory leak, you can use the the call to the QIcon with a QPixmap, in your example is as follows:
QTreeWidgetItem *newItem = new QTreeWidgetItem();
newItem->setText(0, "Item");
// This causes a memory leak!
newItem->setIcon(0, QIcon(QPixmap("D:\\Dnl\\QtSandBoxApp\\Resources\\dataset2.png")));
treeWidget->addTopLevelItem(newItem);
As I explained in my blog (https://programmersexception.blogspot.com/2019/04/memory-leak-con-qicon-en-qt-594-y.html), sorry is in Spanish, there is known BUG, that I hope they will fix soon (https://bugreports.qt.io/browse/QTBUG-59621).

Web service failing with The request failed with HTTP status 401: Unauthorized

I have a very long running task I need to take from my website and put into a web service however every time I try call the web service I get this error message The request failed with HTTP status 401: Unauthorized.
I assumed this could just be an issue with the credentials as I have seen something similar when sending emails so I did this:
Dim wsCustomer As New blueprintdev.RosterEmailService()
Dim basicAuthenticationInfo As New System.Net.NetworkCredential("user", "pass")
wsCustomer.UseDefaultCredentials = False
wsCustomer.PreAuthenticate = True
wsCustomer.Credentials = basicAuthenticationInfo
txtTestResult.Text = wsCustomer.Test()
This still gives me the same issue.
Also tried this using default credentials and that still does not help.
Now I have no idea how to properly do this, and so I have mostly worked off tutorials etc, can anyone tell me what im doing wrong here?
Thanks
Additional notes: the project im working on is one I inherited when I joined the company and cant be built in visual studio so breakpointing through the code is not an option for me without massively hacking the project and removing a lot of code.
Update: Running fiddler shows me 3 entries, when I look at the auth tab i see:
No Proxy-Authenticate Header is present.
WWW-Authenticate Header (Negotiate) appears to be a Kerberos reply:
A1 15 30 13 A0 03 0A 01 03 A1 0C 06 0A 2B 06 01 ¡.0. ....¡...+..
04 01 82 37 02 02 0A ..7...
Then
No Proxy-Authenticate Header is present.
WWW-Authenticate Header (Negotiate) appears to be a Kerberos reply:
A1 81 E2 30 81 DF A0 03 0A 01 01 A2 81 D7 04 81 ¡â0ß ....¢×.
D4 4E 54 4C 4D 53 53 50 00 02 00 00 00 0E 00 0E ÔNTLMSSP........
00 38 00 00 00 15 82 89 E2 C1 20 C3 44 5E 99 21 .8....âÁ ÃD^!
A0 00 00 00 00 00 00 00 00 8E 00 8E 00 46 00 00 ..........F..
00 06 01 B1 1D 00 00 00 0F 41 00 43 00 41 00 44 ...±.....A.C.A.D
00 45 00 4D 00 59 00 02 00 0E 00 41 00 43 00 41 .E.M.Y.....A.C.A
00 44 00 45 00 4D 00 59 00 01 00 06 00 44 00 45 .D.E.M.Y.....D.E
00 56 00 04 00 1A 00 6C 00 6F 00 63 00 61 00 6C .V.....l.o.c.a.l
00 2E 00 41 00 63 00 61 00 64 00 65 00 6D 00 79 ...A.c.a.d.e.m.y
00 03 00 22 00 44 00 45 00 56 00 2E 00 6C 00 6F ...".D.E.V...l.o
00 63 00 61 00 6C 00 2E 00 41 00 63 00 61 00 64 .c.a.l...A.c.a.d
00 65 00 6D 00 79 00 05 00 1A 00 6C 00 6F 00 63 .e.m.y.....l.o.c
00 61 00 6C 00 2E 00 41 00 63 00 61 00 64 00 65 .a.l...A.c.a.d.e
00 6D 00 79 00 07 00 08 00 2E 76 48 74 53 21 CD .m.y......vHtS!Í
01 00 00 00 00
.....
And finally this one highlighted in red
No Proxy-Authenticate Header is present.
No WWW-Authenticate Header is present.
Does this mean the authentication data is being lost somewhere between client and server? or that I am not passing anything to begin with?
If you can access the service through a browser I suggest passing in your username including domain as the credentials.
The username would look something like this: DOMAIN\USER
I've had a similar problem and the username and password had to be explicitly added.
ie.
wsCustomer.username = "something"
wsCustomer.password = "something else"
I guess you're already doing that with:
Dim basicAuthenticationInfo As New System.Net.NetworkCredential("user", "pass")
but maybe worth a try

Resources