why does aio_write() act wrong? - asynchronous

Question

I want to write 2 files by using aio_write.
Used 32KB buffer and repeat aio_write 2048 times for 1 file.(file size is 64MB)
However result is not 64MB but size is 64MB + 32KB, now.
Also sometimes file is written by garbage.
I want to fill 'A' to file.
Please help me.
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <string.h>
#include <errno.h>
#include <stdlib.h>
#include <aio.h>
#include <fcntl.h>
#include <siginfo.h>
#define TNAME "testAio.c"
#define BUFFER_SIZE 32 * 1024 //(32 * 1024 * 1024)
#define FILE_COUNT 2
#define FILE_PATH 256
#define FALSE 0
#define TRUE 1
int main ()
{
char sTmpFileName[FILE_COUNT][FILE_PATH];
char * sBuf;
char * sAlignedBuf;
int sFd[FILE_COUNT];
struct aiocb sAiocb[FILE_COUNT];
int sError;
int sRet;
int i;
int j;
int sWritten[FILE_COUNT];
int sWrittenSize;
int sWrittenCnt;
int sFrequence = 2048;
sBuf = (char*) malloc( BUFFER_SIZE + 512 );
sAlignedBuf = (char*)( ((long)sBuf) + (512 - ((long)sBuf) % 512));
memset( sAlignedBuf, 0x41, BUFFER_SIZE );
for( i = 0; i < FILE_COUNT; i++ )
{
memset( &sAiocb[i], 0, sizeof(struct aiocb) );
sAiocb[i].aio_buf = sAlignedBuf;
sAiocb[i].aio_nbytes = BUFFER_SIZE;
snprintf( sTmpFileName[i],
FILE_PATH,
"testAio_%d",
i);
unlink( sTmpFileName[i] );
sFd[i] = open( sTmpFileName[i],
O_CREAT | O_RDWR | O_EXCL |
O_DIRECT | O_LARGEFILE,
S_IRUSR | S_IWUSR );
sAiocb[i].aio_fildes = sFd[i];
if( sFd[i] == -1 )
{
printf( TNAME " Error at open(): %s\n", strerror( errno ) );
exit(1);
}
}
for( j = 0; j < sFrequence; j++ )
{
for( i = 0; i < FILE_COUNT; i++ )
{
if( sWrittenSize = aio_write( &sAiocb[i] ) == -1 )
{
printf( TNAME " Error at aio_write(): %s\n", strerror( errno ) );
close( sFd[i] );
exit(2);
}
sAiocb[i].aio_offset += sAiocb[i].aio_nbytes;
// printf( "offset %ld\n", sAiocb[i].aio_offset );
}
}
printf( "offset %ld %ld\n",
sAiocb[0].aio_offset,
sAiocb[1].aio_offset );
/* Wait until completion */
i = 0;
sWritten[0] = FALSE;
sWritten[1] = FALSE;
sWrittenCnt = 0;
while( 1 )
{
sError = aio_error( &sAiocb[i] );
if( sError != EINPROGRESS )
{
if( sWritten[i] == FALSE )
{
sWrittenCnt++;
sWritten[i] = TRUE;
}
}
if( sWrittenCnt == FILE_COUNT )
{
break;
}
i = (i + 1) % FILE_COUNT;
}
for( i = 0; i < FILE_COUNT; i++ )
{
sError = aio_error( &sAiocb[i] );
sRet = aio_return( &sAiocb[i] );
if( sError != 0 )
{
printf( TNAME " Error at aio_error() : %d, %s\n",
i,
strerror( sError ) );
close( sFd[i] );
exit(2);
}
if( sRet != BUFFER_SIZE )
{
printf( TNAME " Error at aio_return()\n" );
close( sFd[i] );
exit(2);
}
}
for( i = 0; i < FILE_COUNT; i++ )
{
close( sFd[i] );
}
printf( "Test PASSED\n" );
return 0;
}

Answer 1

Most POSIX implementations enforce severe limits on the number of concurrent asynchronous i/o operations which can be in flight in total on the system, and per process. This limit is 16 on some major implementations. You therefore cannot simply call aio_write 2048 times in sequence, you must call it only up until AIO_LISTIO_MAX which is the maximum possible, always checking error codes for system resource exhaustion before that maximum possible limit. Even on NT which has no hard limits, performance noticeably nosedives after a certain amount of concurrency when FILE_FLAG_NO_BUFFERING is on, especially on older Windows kernels.
Once you have scheduled as many aio_write as the system will take, you then need to call aio_suspend on what you've scheduled and retire out any ops which complete, trying again to refill the pending i/o queue. If you'd like to see a production example of usage, try https://github.com/ned14/boost.afio/blob/master/include/boost/afio/v2.0/detail/impl/posix/io_service.ipp.
I should emphasise that POSIX aio scales poorly, provides virtually no performance benefit, and on Linux or FreeBSD your "asynchronous i/o" is really a thread pool of workers which call the synchronous i/o APIs for you. Virtually no POSIX OS implements much asynchronicity in practice unless O_DIRECT or its equivalent is turned on, it's only really worth bothering with on NT.
As many other posts on Stackoverflow have said, async filesystem i/o is not worth the time nor hassle for 99% of users, just use a thread pool calling the synchronous APIs instead, it scales far far better and is portable across all platforms, doesn't have daft problems with signals, plus always on Linux or on FreeBSD when O_DIRECT is off it's how POSIX aio is implemented anyway.

Answer 2

Thanks for the comment.
Now I noticed about the code that I made mistake.
I don't know whether it is certain or not.
I assume that aio_nbytes should be handled concurrently.
After I called aio_write for certain file(ex file1), I have to wait until end of the call for next aio_write call for file1.
Is my assumption right?