I'm trying to read an OpenCL kernel from a file "kernel.cl", but the kernel I read in ends up having unknown symbols at the end of the program once I have read it. The number of unknown symbols are the same as number of lines in the kernel file.
The code I am using to get the kernel:
FILE *fp;
char *source_str;
size_t source_size, program_size;
fp = fopen("kernel.cl", "r");
if (!fp) {
printf("Failed to load kernel\n");
return 1;
}
fseek(fp, 0, SEEK_END);
program_size = ftell(fp);
rewind(fp);
source_str = (char*)malloc(program_size + 1);
source_str[program_size] = '\0';
fread(source_str, sizeof(char), program_size, fp);
fclose(fp);
This code works on another project, so I don't know what's wrong. Also it seems to work if all the code in the kernel is on one line.
Any help would be appreciated, thanks! :)
The MSDN page for fopen() mentions that when files are opened with "r" as the mode string, some translations will happen with regards to line-endings. This means that the size of the file you query may not match the amount of data read by fread(). This explains why the number of invalid characters was equal to the number of lines in the file (and why it worked with all the code on one line).
The solution is to open the file with the "rb" flag:
fp = fopen("kernel.cl", "rb");
If using C++ is an option, take a look at the program::create_with_source_file() method provided by the Boost.Compute library. It simplifies the process of opening a file, reading the contents, and creating the OpenCL program object with the source code.
For example, you could simply do:
boost::compute::program my_program =
boost::compute::program::create_with_source_file("kernel.cl");
Related
I know Qt has a lot of good methods for reading file.But in my work , I have to open a file by using Qt and get a pointer pointing to the start address of the file.So how to do that?
Here is my codes:
char *buffer;
if(file.open(QIODevice::ReadOnly))
{
QByteArray dataArray=file.readAll();
buffer=dataArray.data();
}
char test[1024];
for(int i=0;i<1024;i++)
{
test[i]=*buffer;
buffer++;
}
I use QByteArray QIODevice::readAll()to read all available data from the device, and returns it as a byte array.
Then I use char * QByteArray::data() to set my pointer buffer.
But when I try to read the data by a for loop,my Qt debugger throw me an error:read access violation,so how would this happen?
BTW , the file to be read is very big so I can't use a buffer to read them all once. Instead , I have to do as what I did here. When I read a 7kb-sizes-file ,my codes runs well. When I read a 700kb-sizes-file , here comes the problem.
The reason you are getting access violation is that dataArray is declared inside if code block and goes out of scope. You want to declare the dataArray outside of that block, eg:
QByteArray dataArray;
if(file.open(QIODevice::ReadOnly))
{
dataArray = file.readAll();
}
else
{
// give error
}
char *buffer = dataArray.data();
char test[1024];
for(int i = 0; i < 1024; i++)
{
test[i] = *buffer;
buffer++;
}
There are other potential problems in your code though:
First, what if the size of data read from the file is less than 1024? Then you will be reading past the end of the buffer (more access violations).
Second, what if the file is really big? Reading all that data at once may cause swapping/paging.
I'm trying to modify graphic assets of the software I'm using (for aesthetic puroposess, I guess it's hard to do something harmful with graphic assets) but developer encrypted them. I'm not sure why he decided to do that since I used and modified a bunch of similar softwares and developers of those didn't bother (as I can see no reason why encrypting those assets would be necessary).
So anyway here are examples of those encrypted graphic assets:
http://www.mediafire.com/view/sx2yc0w5wkr9m2h/avatars_50-alpha.jpg
http://www.mediafire.com/download/i4fc52438hkp55l/avatars_80.png
Is there a way of decrypting those? If so how should I go about this?
The header "CF10" seems to be a privately added signature to signify the rest of the file is "encoded". This is a very simple XOR encoding: xor 8Dh was the first value I tried, and I got it right first time too. The reasoning behind trying that as the first value is that the value 8D occurs very frequently in the first 100 bytes-or-so, where they typically could be lots of zeroes.
"Decrypting" is thus very straightforward: if a file starts with the four bytes CF10, remove them and apply xor 8Dh on the rest of the file. Decoding the files show the first "JPG" is in fact a tiny PNG image (and not a very interesting one to boot), the second is indeed a PNG file:
The file extension may or may not be the original file extension; the one sample called ".jpg" is in fact also a PNG file, as can be seen by its header signature.
The following quick-and-dirty C source will decode the images. The same program can be adjusted to encode them as well, because the xor operation is exactly the same. The only thing needed is add a bit of logic flow:
read the first 4 bytes (maximum) of the input file and test if this forms the string CF10
if not, the file is not encoded:
a. write CF10 to the output file
b. encode the image by applying xor 8Dh on each byte
if so,
b. decode the image by applying xor 8Dh on each byte.
As you can see, there is no "3a" and both "b" steps are the same.
#include <stdio.h>
#include <string.h>
#ifndef MAX_PATH
#define MAX_PATH 256
#endif
#define INPUTPATH "c:\\documents"
#define OUTPUTPATH ""
int main (int argc, char **argv)
{
FILE *inp, *outp;
int i, encode_flag = 0;
char filename_buffer[MAX_PATH];
char sig[] = "CF10", *ptr;
if (argc != 3)
{
printf ("usage: decode [input] [output]\n");
return -1;
}
filename_buffer[0] = 0;
if (!strchr(argv[1], '/') && !strchr(argv[1], 92) && !strchr(argv[1], ':'))
strcpy (filename_buffer, INPUTPATH);
strcat (filename_buffer, argv[1]);
inp = fopen (filename_buffer, "rb");
if (inp == NULL)
{
printf ("bad input file '%s'\n", filename_buffer);
return -2;
}
ptr = sig;
while (*ptr)
{
i = fgetc (inp);
if (*ptr != i)
{
encode_flag = 1;
break;
}
ptr++;
}
if (encode_flag)
{
/* rewind file because we already read some bytes */
fseek (inp, 0, SEEK_SET);
printf ("encoding input file: '%s'\n", filename_buffer);
} else
printf ("decoding input file: '%s'\n", filename_buffer);
filename_buffer[0] = 0;
if (!strchr(argv[2], '/') && !strchr(argv[2], 92) && !strchr(argv[2], ':'))
strcpy (filename_buffer, OUTPUTPATH);
strcat (filename_buffer, argv[2]);
outp = fopen (filename_buffer, "wb");
if (outp == NULL)
{
printf ("bad output file '%s'\n", filename_buffer);
return -2;
}
printf ("output file: '%s'\n", filename_buffer);
if (encode_flag)
fwrite (sig, 1, 4, outp);
do
{
i = fgetc(inp);
if (i != EOF)
fputc (i ^ 0x8d, outp);
} while (i != EOF);
fclose (inp);
fclose (outp);
printf ("all done. bye bye\n");
return 0;
}
Ok so when it comes to practical usage of the code provided by #Jongware that was unclear to me - I figured it out with some help:)
I compiled the code using Visual Studio (you can find guides on how to do that, basically create new Visual C++ project and in Project -> Project Propeties choose C/C++ -> All options and Compile as C Code (/TC)).
Then I opened program in command prompt using parameter "program encrypted_file decrypted_file".
Thanks a lot for help Jongware!
I have the following lines of code which I use to first determine the file size of the .cl file I am reading from (and loading into a buffer), and subsequently building my program and kernel from the buffer. Assuming calculate.cl contains a simple vector addition kernel.
//get size of kernel source
FILE *f = fopen("calculate.cl", "r");
fseek(f, 0, SEEK_END);
size_t programSize = ftell(f);
rewind(f);
//load kernel into buffer
char *programBuffer = (char*)malloc(programSize + 1);
programBuffer[programSize] = '\0';
fread(programBuffer, sizeof(char), programSize, f);
fclose(f);
//create program from buffer
cl_program program = clCreateProgramWithSource(context, 1, (const char**) &programBuffer, &programSize, &status);
//build program for devices
status = clBuildProgram(program, numDevices, devices, NULL, NULL, NULL);
//create the kernel
cl_kernel calculate = clCreateKernel(program, "calculate", &status);
However, when I run my program, the output produced is zero instead of the intended vector addition results. I've verified that the problem is not to do with the kernel itself (I used a different method to load an external kernel which worked and gave me the intended results) however I am still curious as to why this initial method I attempted did not work.
Any help?
the problem's been solved.
following bl0z0's suggestion and looking up the error, I've found the solution here:
OpenCL: Expected identifier in kernel
thanks everyone :D I really appreciate it!
I believe this gives the programing size in terms of the number of chars:
size_t programSize = ftell(f);
and here you need to allocate in terms of bytes:
char *programBuffer = (char*)malloc(programSize + 1);
so I think that previous line should be
char *programBuffer = (char*)malloc(programSize * sizeof(char) + 1);
Double check this by just printing the programBuffer.
In RCurl a function and a class CFILE is defined to work with C-level file handles. From the manual:
The intent is to be able to pass these to libcurl as options so that it can read or write from or to the file. We can also do this with R connections and specify callback functions that manipulate these connections. But using the C-level FILE handle is likely to be significantly faster for large files.
There are no examples related to downloads so I tried:
library(RCurl)
u = "http://cran.r-project.org/web/packages/RCurl/RCurl.pdf"
f = CFILE("RCurl.pdf", mode="wb")
ret= getURL(u, write = getNativeSymbolInfo("R_curl_write_binary_data")$address,
file = f#ref)
I also tried by replacing the file option with writedata = f#ref.
The file is downloaded but it is corrupted.
Writing custom callback for the write argument works only for non-binary data.
Any idea to download a binary file straight to disk (without loading it in memory) in RCurl?
I think you want to use writedata and remember to close the file
library(RCurl)
filename <- tempfile()
f <- CFILE(filename, "wb")
url <- "http://cran.fhcrc.org/Rlogo.jpg"
curlPerform(url = url, writedata = f#ref)
close(f)
For more elaborate writing, I'm not sure if this is the best way, but Linux tells me, from
man curl_easy_setopt
that there's a curl option CURL_WRITEFUNCTION that is a pointer to a C function with prototype
size_t function(void *ptr, size_t size, size_t nmemb, void *stream);
and in R at the end of ?curlPerform there's an example of calling a C function as the 'writefunction' option. So I created a file curl_writer.c
#include <stdio.h>
size_t
writer(void *buffer, size_t size, size_t nmemb, void *stream)
{
fprintf(stderr, "<writer> size = %d, nmemb = %d\n",
(int) size, (int) nmemb);
return size * nmemb;
}
Compiled it
R CMD SHLIB curl_writer.c
which on Linux produces a file curl_writer.so, and then in R
dyn.load("curl_writer.so")
writer <- getNativeSymbolInfo("writer", PACKAGE="curl_writer")$address
curlPerform(URL=url, writefunction=writer)
and get on stderr
<writer> size = 1, nmemb = 2653
<writer> size = 1, nmemb = 520
OK
These two ideas can be integrated, i.e., writing to an arbitrary file using an arbitrary function, by modifying the C function to use the FILE * we pass in, as
#include <stdio.h>
size_t
writer(void *buffer, size_t size, size_t nmemb, void *stream)
{
FILE *fout = (FILE *) stream;
fprintf(fout, "<writer> size = %d, nmemb = %d\n",
(int) size, (int) nmemb);
fflush(fout);
return size * nmemb;
}
and then back in R after compiling
dyn.load("curl_writer.so")
writer <- getNativeSymbolInfo("writer", PACKAGE="curl_writer")$address
f <- CFILE(filename <- tempfile(), "wb")
curlPerform(URL=url, writedata=f#ref, writefunction=writer)
close(f)
getURL can be used here, too, provided writedata=f#ref, write=writer; I think the problem in the original question is that R_curl_write_binary_data is really an internal function, writing to a buffer managed by RCurl, rather than a file handle like that created by CFILE. Likewise, specifying writedata without write (which seems from the source code to getURL to be an alias for writefunction) sends a pointer to a file to a function expecting a pointer to something else; for getURL both writedata and write need to be provided.
I am working on this problem as well and don't have an answer, yet.
However, I did find this:
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTWRITEDATA
Are you working on R under Windows? I am.
This documentation for the writedata function indicates that on windows, you must use writefunction along with writedata.
Reading here: http://www.omegahat.org/RCurl/RCurlJSS.pdf I found that RCurl expects the writefunction to be an R function, so we can implement that ourselves on windows. It is going to be slower than using a C function to write the data, however I bet that the speed of the network link will be the bottleneck.
getURI(url="sftp://hostname/home/me/onegeebee", curl=con, write=function(x) writeChar(x, f, eos=NULL))
Error in curlPerform(curl = curl, .opts = opts, .encoding = .encoding) : embedded nul in string: ' <`á\017_\021
(This is after creating a 1GB file on the server to test transfer speed)
I haven't yet found an answer that doesn't choke on NUL bytes in the data. It seems that somewhere in the bowels of the RCurl package when it's passing data up into R to execute the writefunction you supply, it tries to convert the data into a character string. It must not do that if you use a C function. Notably using the recommended R_curl_write_binary_data callback along with CFILE kills rsession.exe on win32 every time for me.
I am trying to cache a program in a file so that it does not need to compile to assembly. Consequently, I am trying to dump the binaries. I am getting an issue where the binary program returned alternately has garbage data at the end.
Error checking omitted for clarity (no errors occur, though):
clGetProgramInfo(kernel->program, CL_PROGRAM_BINARY_SIZES, 0,NULL, &n);
n /= sizeof(size_t);
size_t* sizes = new size_t[n];
clGetProgramInfo(kernel->program, CL_PROGRAM_BINARY_SIZES, n*sizeof(size_t),sizes, NULL);
I have confirmed that kernel->program is identical between times. In the above code, "n" is invariably 1, but sizes[0] varies between 2296 and 2312 alternate runs.
The problem is that the 2296 number appears to be more accurate--after the final closing brace in the output, there are three newlines and then three spaces.
For the 2312 number, after the final closing brace in the output, there are the three newlines, a line of garbage data, and then the three spaces.
Naturally, the line of garbage data is problematic. I'm not sure how to get rid of it, and I'm pretty sure it's not an error on my part.
NVIDIA GeForce GTX 580M, with driver 305.60 on Windows 7.
Update: I have changed the code to the following:
//Get how many devices there are
size_t n;
clGetProgramInfo(kernel->program, CL_PROGRAM_NUM_DEVICES, 0,NULL, &n);
//Get the list of binary sizes
size_t* sizes = new size_t[n];
clGetProgramInfo(kernel->program, CL_PROGRAM_BINARY_SIZES, n*sizeof(size_t),sizes, NULL);
//Get the binaries
unsigned char** binaries = new unsigned char*[n];
for (int i=0;i<(int)n;++i) {
binaries[i] = new unsigned char[sizes[i]];
}
clGetProgramInfo(kernel->program, CL_PROGRAM_BINARIES, n*sizeof(unsigned char*),binaries, NULL);
Now, the code has n = 4, but only sizes[0] contains meaningful information (so the alloc of sizes[1] fails in the loop). Thoughts?
I get the number of devices with the following line:
clGetProgramInfo(kernel->program, CL_PROGRAM_NUM_DEVICES, sizeof(cl_uint), &n, NULL);
clGetProgramInfo(kernel->program, CL_PROGRAM_NUM_DEVICES, 0,NULL, &n);
needs to be:
clGetProgramInfo(kernel->program, CL_PROGRAM_NUM_DEVICES, sizeof(size_t), &n, NULL);
clGetProgramInfo with CL_PROGRAM_BINARY_SIZES and CL_PROGRAM_BINARIES needs a pointer to an array and not just to a single variable because it creates binaries for each device that you supplied when building the program. That is why the first line returns nothing. n for the second example should be the number of devices.
Not sure why the second example is different for each run... are you sure you are building for the same device each time?