I want to write 2 files by using aio_write.
Used 32KB buffer and repeat aio_write 2048 times for 1 file.(file size is 64MB)
However result is not 64MB but size is 64MB + 32KB, now.
Also sometimes file is written by garbage.
I want to fill 'A' to file.
Please help me.
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <string.h>
#include <errno.h>
#include <stdlib.h>
#include <aio.h>
#include <fcntl.h>
#include <siginfo.h>
#define TNAME "testAio.c"
#define BUFFER_SIZE 32 * 1024 //(32 * 1024 * 1024)
#define FILE_COUNT 2
#define FILE_PATH 256
#define FALSE 0
#define TRUE 1
int main ()
{
char sTmpFileName[FILE_COUNT][FILE_PATH];
char * sBuf;
char * sAlignedBuf;
int sFd[FILE_COUNT];
struct aiocb sAiocb[FILE_COUNT];
int sError;
int sRet;
int i;
int j;
int sWritten[FILE_COUNT];
int sWrittenSize;
int sWrittenCnt;
int sFrequence = 2048;
sBuf = (char*) malloc( BUFFER_SIZE + 512 );
sAlignedBuf = (char*)( ((long)sBuf) + (512 - ((long)sBuf) % 512));
memset( sAlignedBuf, 0x41, BUFFER_SIZE );
for( i = 0; i < FILE_COUNT; i++ )
{
memset( &sAiocb[i], 0, sizeof(struct aiocb) );
sAiocb[i].aio_buf = sAlignedBuf;
sAiocb[i].aio_nbytes = BUFFER_SIZE;
snprintf( sTmpFileName[i],
FILE_PATH,
"testAio_%d",
i);
unlink( sTmpFileName[i] );
sFd[i] = open( sTmpFileName[i],
O_CREAT | O_RDWR | O_EXCL |
O_DIRECT | O_LARGEFILE,
S_IRUSR | S_IWUSR );
sAiocb[i].aio_fildes = sFd[i];
if( sFd[i] == -1 )
{
printf( TNAME " Error at open(): %s\n", strerror( errno ) );
exit(1);
}
}
for( j = 0; j < sFrequence; j++ )
{
for( i = 0; i < FILE_COUNT; i++ )
{
if( sWrittenSize = aio_write( &sAiocb[i] ) == -1 )
{
printf( TNAME " Error at aio_write(): %s\n", strerror( errno ) );
close( sFd[i] );
exit(2);
}
sAiocb[i].aio_offset += sAiocb[i].aio_nbytes;
// printf( "offset %ld\n", sAiocb[i].aio_offset );
}
}
printf( "offset %ld %ld\n",
sAiocb[0].aio_offset,
sAiocb[1].aio_offset );
/* Wait until completion */
i = 0;
sWritten[0] = FALSE;
sWritten[1] = FALSE;
sWrittenCnt = 0;
while( 1 )
{
sError = aio_error( &sAiocb[i] );
if( sError != EINPROGRESS )
{
if( sWritten[i] == FALSE )
{
sWrittenCnt++;
sWritten[i] = TRUE;
}
}
if( sWrittenCnt == FILE_COUNT )
{
break;
}
i = (i + 1) % FILE_COUNT;
}
for( i = 0; i < FILE_COUNT; i++ )
{
sError = aio_error( &sAiocb[i] );
sRet = aio_return( &sAiocb[i] );
if( sError != 0 )
{
printf( TNAME " Error at aio_error() : %d, %s\n",
i,
strerror( sError ) );
close( sFd[i] );
exit(2);
}
if( sRet != BUFFER_SIZE )
{
printf( TNAME " Error at aio_return()\n" );
close( sFd[i] );
exit(2);
}
}
for( i = 0; i < FILE_COUNT; i++ )
{
close( sFd[i] );
}
printf( "Test PASSED\n" );
return 0;
}
Most POSIX implementations enforce severe limits on the number of concurrent asynchronous i/o operations which can be in flight in total on the system, and per process. This limit is 16 on some major implementations. You therefore cannot simply call aio_write 2048 times in sequence, you must call it only up until AIO_LISTIO_MAX which is the maximum possible, always checking error codes for system resource exhaustion before that maximum possible limit. Even on NT which has no hard limits, performance noticeably nosedives after a certain amount of concurrency when FILE_FLAG_NO_BUFFERING is on, especially on older Windows kernels.
Once you have scheduled as many aio_write as the system will take, you then need to call aio_suspend on what you've scheduled and retire out any ops which complete, trying again to refill the pending i/o queue. If you'd like to see a production example of usage, try https://github.com/ned14/boost.afio/blob/master/include/boost/afio/v2.0/detail/impl/posix/io_service.ipp.
I should emphasise that POSIX aio scales poorly, provides virtually no performance benefit, and on Linux or FreeBSD your "asynchronous i/o" is really a thread pool of workers which call the synchronous i/o APIs for you. Virtually no POSIX OS implements much asynchronicity in practice unless O_DIRECT or its equivalent is turned on, it's only really worth bothering with on NT.
As many other posts on Stackoverflow have said, async filesystem i/o is not worth the time nor hassle for 99% of users, just use a thread pool calling the synchronous APIs instead, it scales far far better and is portable across all platforms, doesn't have daft problems with signals, plus always on Linux or on FreeBSD when O_DIRECT is off it's how POSIX aio is implemented anyway.
Thanks for the comment.
Now I noticed about the code that I made mistake.
I don't know whether it is certain or not.
I assume that aio_nbytes should be handled concurrently.
After I called aio_write for certain file(ex file1), I have to wait until end of the call for next aio_write call for file1.
Is my assumption right?
First off, here is some code:
int main()
{
int days[] = {1,2,3,4,5};
int *ptr = days;
printf("%u\n", sizeof(days));
printf("%u\n", sizeof(ptr));
return 0;
}
Is there a way to find out the size of the array that ptr is pointing to (instead of just giving its size, which is four bytes on a 32-bit system)?
No, you can't. The compiler doesn't know what the pointer is pointing to. There are tricks, like ending the array with a known out-of-band value and then counting the size up until that value, but that's not using sizeof().
Another trick is the one mentioned by Zan, which is to stash the size somewhere. For example, if you're dynamically allocating the array, allocate a block one int bigger than the one you need, stash the size in the first int, and return ptr+1 as the pointer to the array. When you need the size, decrement the pointer and peek at the stashed value. Just remember to free the whole block starting from the beginning, and not just the array.
The answer is, "No."
What C programmers do is store the size of the array somewhere. It can be part of a structure, or the programmer can cheat a bit and malloc() more memory than requested in order to store a length value before the start of the array.
For dynamic arrays (malloc or C++ new) you need to store the size of the array as mentioned by others or perhaps build an array manager structure which handles add, remove, count, etc. Unfortunately C doesn't do this nearly as well as C++ since you basically have to build it for each different array type you are storing which is cumbersome if you have multiple types of arrays that you need to manage.
For static arrays, such as the one in your example, there is a common macro used to get the size, but it is not recommended as it does not check if the parameter is really a static array. The macro is used in real code though, e.g. in the Linux kernel headers although it may be slightly different than the one below:
#if !defined(ARRAY_SIZE)
#define ARRAY_SIZE(x) (sizeof((x)) / sizeof((x)[0]))
#endif
int main()
{
int days[] = {1,2,3,4,5};
int *ptr = days;
printf("%u\n", ARRAY_SIZE(days));
printf("%u\n", sizeof(ptr));
return 0;
}
You can google for reasons to be wary of macros like this. Be careful.
If possible, the C++ stdlib such as vector which is much safer and easier to use.
There is a clean solution with C++ templates, without using sizeof(). The following getSize() function returns the size of any static array:
#include <cstddef>
template<typename T, size_t SIZE>
size_t getSize(T (&)[SIZE]) {
return SIZE;
}
Here is an example with a foo_t structure:
#include <cstddef>
template<typename T, size_t SIZE>
size_t getSize(T (&)[SIZE]) {
return SIZE;
}
struct foo_t {
int ball;
};
int main()
{
foo_t foos3[] = {{1},{2},{3}};
foo_t foos5[] = {{1},{2},{3},{4},{5}};
printf("%u\n", getSize(foos3));
printf("%u\n", getSize(foos5));
return 0;
}
Output:
3
5
As all the correct answers have stated, you cannot get this information from the decayed pointer value of the array alone. If the decayed pointer is the argument received by the function, then the size of the originating array has to be provided in some other way for the function to come to know that size.
Here's a suggestion different from what has been provided thus far,that will work: Pass a pointer to the array instead. This suggestion is similar to the C++ style suggestions, except that C does not support templates or references:
#define ARRAY_SZ 10
void foo (int (*arr)[ARRAY_SZ]) {
printf("%u\n", (unsigned)sizeof(*arr)/sizeof(**arr));
}
But, this suggestion is kind of silly for your problem, since the function is defined to know exactly the size of the array that is passed in (hence, there is little need to use sizeof at all on the array). What it does do, though, is offer some type safety. It will prohibit you from passing in an array of an unwanted size.
int x[20];
int y[10];
foo(&x); /* error */
foo(&y); /* ok */
If the function is supposed to be able to operate on any size of array, then you will have to provide the size to the function as additional information.
For this specific example, yes, there is, IF you use typedefs (see below). Of course, if you do it this way, you're just as well off to use SIZEOF_DAYS, since you know what the pointer is pointing to.
If you have a (void *) pointer, as is returned by malloc() or the like, then, no, there is no way to determine what data structure the pointer is pointing to and thus, no way to determine its size.
#include <stdio.h>
#define NUM_DAYS 5
typedef int days_t[ NUM_DAYS ];
#define SIZEOF_DAYS ( sizeof( days_t ) )
int main() {
days_t days;
days_t *ptr = &days;
printf( "SIZEOF_DAYS: %u\n", SIZEOF_DAYS );
printf( "sizeof(days): %u\n", sizeof(days) );
printf( "sizeof(*ptr): %u\n", sizeof(*ptr) );
printf( "sizeof(ptr): %u\n", sizeof(ptr) );
return 0;
}
Output:
SIZEOF_DAYS: 20
sizeof(days): 20
sizeof(*ptr): 20
sizeof(ptr): 4
There is no magic solution. C is not a reflective language. Objects don't automatically know what they are.
But you have many choices:
Obviously, add a parameter
Wrap the call in a macro and automatically add a parameter
Use a more complex object. Define a structure which contains the dynamic array and also the size of the array. Then, pass the address of the structure.
You can do something like this:
int days[] = { /*length:*/5, /*values:*/ 1,2,3,4,5 };
int *ptr = days + 1;
printf("array length: %u\n", ptr[-1]);
return 0;
My solution to this problem is to save the length of the array into a struct Array as a meta-information about the array.
#include <stdio.h>
#include <stdlib.h>
struct Array
{
int length;
double *array;
};
typedef struct Array Array;
Array* NewArray(int length)
{
/* Allocate the memory for the struct Array */
Array *newArray = (Array*) malloc(sizeof(Array));
/* Insert only non-negative length's*/
newArray->length = (length > 0) ? length : 0;
newArray->array = (double*) malloc(length*sizeof(double));
return newArray;
}
void SetArray(Array *structure,int length,double* array)
{
structure->length = length;
structure->array = array;
}
void PrintArray(Array *structure)
{
if(structure->length > 0)
{
int i;
printf("length: %d\n", structure->length);
for (i = 0; i < structure->length; i++)
printf("%g\n", structure->array[i]);
}
else
printf("Empty Array. Length 0\n");
}
int main()
{
int i;
Array *negativeTest, *days = NewArray(5);
double moreDays[] = {1,2,3,4,5,6,7,8,9,10};
for (i = 0; i < days->length; i++)
days->array[i] = i+1;
PrintArray(days);
SetArray(days,10,moreDays);
PrintArray(days);
negativeTest = NewArray(-5);
PrintArray(negativeTest);
return 0;
}
But you have to care about set the right length of the array you want to store, because the is no way to check this length, like our friends massively explained.
This is how I personally do it in my code. I like to keep it as simple as possible while still able to get values that I need.
typedef struct intArr {
int size;
int* arr;
} intArr_t;
int main() {
intArr_t arr;
arr.size = 6;
arr.arr = (int*)malloc(sizeof(int) * arr.size);
for (size_t i = 0; i < arr.size; i++) {
arr.arr[i] = i * 10;
}
return 0;
}
No, you can't use sizeof(ptr) to find the size of array ptr is pointing to.
Though allocating extra memory(more than the size of array) will be helpful if you want to store the length in extra space.
int main()
{
int days[] = {1,2,3,4,5};
int *ptr = days;
printf("%u\n", sizeof(days));
printf("%u\n", sizeof(ptr));
return 0;
}
Size of days[] is 20 which is no of elements * size of it's data type.
While the size of pointer is 4 no matter what it is pointing to.
Because a pointer points to other element by storing it's address.
In strings there is a '\0' character at the end so the length of the string can be gotten using functions like strlen. The problem with an integer array, for example, is that you can't use any value as an end value so one possible solution is to address the array and use as an end value the NULL pointer.
#include <stdio.h>
/* the following function will produce the warning:
* ‘sizeof’ on array function parameter ‘a’ will
* return size of ‘int *’ [-Wsizeof-array-argument]
*/
void foo( int a[] )
{
printf( "%lu\n", sizeof a );
}
/* so we have to implement something else one possible
* idea is to use the NULL pointer as a control value
* the same way '\0' is used in strings but this way
* the pointer passed to a function should address pointers
* so the actual implementation of an array type will
* be a pointer to pointer
*/
typedef char * type_t; /* line 18 */
typedef type_t ** array_t;
int main( void )
{
array_t initialize( int, ... );
/* initialize an array with four values "foo", "bar", "baz", "foobar"
* if one wants to use integers rather than strings than in the typedef
* declaration at line 18 the char * type should be changed with int
* and in the format used for printing the array values
* at line 45 and 51 "%s" should be changed with "%i"
*/
array_t array = initialize( 4, "foo", "bar", "baz", "foobar" );
int size( array_t );
/* print array size */
printf( "size %i:\n", size( array ));
void aprint( char *, array_t );
/* print array values */
aprint( "%s\n", array ); /* line 45 */
type_t getval( array_t, int );
/* print an indexed value */
int i = 2;
type_t val = getval( array, i );
printf( "%i: %s\n", i, val ); /* line 51 */
void delete( array_t );
/* free some space */
delete( array );
return 0;
}
/* the output of the program should be:
* size 4:
* foo
* bar
* baz
* foobar
* 2: baz
*/
#include <stdarg.h>
#include <stdlib.h>
array_t initialize( int n, ... )
{
/* here we store the array values */
type_t *v = (type_t *) malloc( sizeof( type_t ) * n );
va_list ap;
va_start( ap, n );
int j;
for ( j = 0; j < n; j++ )
v[j] = va_arg( ap, type_t );
va_end( ap );
/* the actual array will hold the addresses of those
* values plus a NULL pointer
*/
array_t a = (array_t) malloc( sizeof( type_t *) * ( n + 1 ));
a[n] = NULL;
for ( j = 0; j < n; j++ )
a[j] = v + j;
return a;
}
int size( array_t a )
{
int n = 0;
while ( *a++ != NULL )
n++;
return n;
}
void aprint( char *fmt, array_t a )
{
while ( *a != NULL )
printf( fmt, **a++ );
}
type_t getval( array_t a, int i )
{
return *a[i];
}
void delete( array_t a )
{
free( *a );
free( a );
}
#include <stdio.h>
#include <string.h>
#include <stddef.h>
#include <stdlib.h>
#define array(type) struct { size_t size; type elem[0]; }
void *array_new(int esize, int ecnt)
{
size_t *a = (size_t *)malloc(esize*ecnt+sizeof(size_t));
if (a) *a = ecnt;
return a;
}
#define array_new(type, count) array_new(sizeof(type),count)
#define array_delete free
#define array_foreach(type, e, arr) \
for (type *e = (arr)->elem; e < (arr)->size + (arr)->elem; ++e)
int main(int argc, char const *argv[])
{
array(int) *iarr = array_new(int, 10);
array(float) *farr = array_new(float, 10);
array(double) *darr = array_new(double, 10);
array(char) *carr = array_new(char, 11);
for (int i = 0; i < iarr->size; ++i) {
iarr->elem[i] = i;
farr->elem[i] = i*1.0f;
darr->elem[i] = i*1.0;
carr->elem[i] = i+'0';
}
array_foreach(int, e, iarr) {
printf("%d ", *e);
}
array_foreach(float, e, farr) {
printf("%.0f ", *e);
}
array_foreach(double, e, darr) {
printf("%.0lf ", *e);
}
carr->elem[carr->size-1] = '\0';
printf("%s\n", carr->elem);
return 0;
}
#define array_size 10
struct {
int16 size;
int16 array[array_size];
int16 property1[(array_size/16)+1]
int16 property2[(array_size/16)+1]
} array1 = {array_size, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
#undef array_size
array_size is passing to the size variable:
#define array_size 30
struct {
int16 size;
int16 array[array_size];
int16 property1[(array_size/16)+1]
int16 property2[(array_size/16)+1]
} array2 = {array_size};
#undef array_size
Usage is:
void main() {
int16 size = array1.size;
for (int i=0; i!=size; i++) {
array1.array[i] *= 2;
}
}
Most implementations will have a function that tells you the reserved size for objects allocated with malloc() or calloc(), for example GNU has malloc_usable_size()
However, this will return the size of the reversed block, which can be larger than the value given to malloc()/realloc().
There is a popular macro, which you can define for finding number of elements in the array (Microsoft CRT even provides it OOB with name _countof):
#define countof(x) (sizeof(x)/sizeof((x)[0]))
Then you can write:
int my_array[] = { ... some elements ... };
printf("%zu", countof(my_array)); // 'z' is correct type specifier for size_t
I can write data to cl::BufferGL variables (on GPU) and use it to draw, but I got wrong values when reading back to host. I have ran a couple examples from AMD SDK, including GL_CL interop, and is all ok. I can read and write cl::Buffer, i.e., copy from host to device and copy from device to host.
The source code is hosted at link.
To create the cl::Buffer I do steps below (in order, and after get a cl_gl context):
glGenBuffer
glBindBuffer
glBufferData
clCreateFromGLBuffer
glBindBuffer (bind to 0, or unbind)
The program is written using:
C++
Ubuntu 12.06 64 bits, kernel 3.2.0-29-generic
OpenCL 1.2
Driver 1124.2 (Binary driver from AMD, Catalyst 13.4)
AMD APPSDK v2.8
OpenGL Version: 4.2.12217
Glew
Freeglut
EDIT: I've created a code block to reproduce the error
// Terminates OpenCL commands
cl_int status = 0;
status = clFinish( (cq)() );
if ( status != CL_SUCCESS ) { printf("File: %s:%d\n", __FILE__, __LINE__); }
// Create Vertex buffer object
GLint number = 5;
GLuint buffer;
CHk_ERR( glGenBuffers( 1, &buffer ));
CHk_ERR( glBindBuffer( GL_ARRAY_BUFFER, buffer ) );
// initialize buffer object
GLsizeiptr size = static_cast<GLsizeiptr>(sizeof(number));
const GLvoid * data = static_cast<const GLvoid *>(&number);
CHk_ERR( glBufferData( GL_ARRAY_BUFFER, size, data, GL_DYNAMIC_DRAW) );
CHk_ERR( glBindBuffer(GL_ARRAY_BUFFER, 0) );
glFinish();
if ( glGetError() != GL_NO_ERROR ) { printf("File: %s:%d\n", __FILE__, __LINE__); }
// create OpenCL buffer from GL VBO
cl_mem mem = clCreateFromGLBuffer( (context)(), CL_MEM_READ_WRITE, buffer, &status );
if ( status != CL_SUCCESS ) { printf("File: %s:%d\n", __FILE__, __LINE__); }
// Acquire OpenCL memory objects that have been created from OpenGL objects
if ( clEnqueueAcquireGLObjects( (cq)(), 1, &(mem), 0, NULL, NULL ) != CL_SUCCESS ){
std::cout << "ERROR!";
}
// Read buffer
GLint value;
cq.enqueueReadBuffer( mem, CL_TRUE, 0, size, &value );
std::cout << "Value: " << value << std::endl;
Any help is appreciated!!