I have a code that is supposed to wait for an input data file, distribute the data on a cluster, wait for the processing to finish, remove the input file and sleep until a new input file is provided by another application. Problem is the call to "usleep" of a few milliseconds makes the program go to sleep for 10-20 seconds. I see that mpirun is using high amount of cpu time during the sleep. I know that usleep requested time isn't always what it says it is, but I think there is more to it. That is something is wrong with my code (this being my first code in mpi).
Here is the relevant code:
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &myId);
MPI_Comm_size(MPI_COMM_WORLD, &numProcs);
bool newFile;
while (listeningMode) {
while (!newFile) {
inputfile.open(inputfname.c_str(),ios::binary);
if (inputfile.is_open()) {
newFile = true;
} else {
usleep(1000);
}
}
MPI_Barrier(MPI_COMM_WORLD);
// distribute data and do some processing here ....
// and then at the end
if(myId == rootProcess)
remove(inputfname.c_str());
newFile = false;
MPI_Barrier(MPI_COMM_WORLD);
} // while !newfile
} // while listeningMode
Related
I have a process which does multiple forks, generating several sub processes which have to write on stdout. So, the messages of the different sub processes might cross with themselves. How can I avoid this problem?
Say you have three processes, each trying to output an infinite series of lines composed of four characters followed by a newline:
void four(char c);
int main()
{
//insert your own error checking
pid_t p0, p1, p2;
#define PROC(pid,str) pid=fork(); if(0==pid) four(str);
PROC(p0,'a');
PROC(p1,'b');
PROC(p2,'c');
waitpid(p2, 0,0);
waitpid(p1, 0,0);
waitpid(p0, 0,0);
}
If your four function is:
void four(char c)
{
for(;;){
for(int i=0; i<4;i++)
putchar(c);
putchar('\n');
}
}
and you pipe your program into this grep invocation:
./a.out |grep -v -e aaaa -e bbbb -e cccc
You'll get matches that demonstrate your problem.
The easiest way to solve this is by relying on the Linux guarantee that it won't break write calls aimed at a pipe if the write arguments are smaller than the pipe buffer size (defaults to 4KiB on my system (you can get the size from the ulimit shell builtin).
void four(char c)
{
for(;;){
for(int i=0; i<4;i++)
putchar(c);
putchar('\n');
fflush(stdout);
//the stdout buffer is surely larger than 5
//so this is 1 `write`
}
}
If you want to be more portable and robust, you can use a lock on a shared file:
void four(char c)
{
int fd;
fd = open("/proc/self/exe", O_RDONLY);
for(;;){
if(0>flock(fd, LOCK_EX))
perror("flock");
for(int i=0; i<4;i++)
{ putchar(c); fflush(stdout); }
putchar('\n'); fflush(stdout);
//the pipe buf guarantee won't save us here
//given all these flushes
//but this lock will
if(0>flock(fd, LOCK_UN))
perror("flock");
}
}
Alternatively, can also set file locks with fcntl.
By "cross themselves", I presume you mean that you are worried about output being interleaved. That is, one process attempts to output "Hello, World!", while another prints "Goodbye, Chicago!", and the final output is "Hello, Goodby, World! Chicago!", or similar. The absolute simplest approach to solving this is to ensure that each message you write is written with a single write system call, and that the data is small. If your messages are less than 1k (or so, the exactly value is system dependent, often 4096, rarely less than 512). If you invoke write with a sufficiently small buffer, the write will be atomic and will not be interleaved with output from any other process. If your messages won't fit in the size for the system you are on, you will need to use some locking mechanism.
I want to read a String in Arduino from the keyboard outside of the loop() method.
I have the following method:
void readFromKeyboard(byte arrayAddress[])
{
int count = 0, i = 0;
while ((count = Serial.available()) == 0);
while (i<count)
{
arrayAddress[i++] = Serial.read();
}
}
In the loop() method I am calling it like:
readFromKeyboard(userInput);
where userInput is a byte[];
The problem is that when I input more than one characters it read the 1st character initially and it call the readFromKeyboard again an then reads the rest.
Example; if I input "asdf":
--the 1st time it will do ==> userInput = "a"
--the 2nd time it will do ==> userInput = "sdf"
I have tryed many things but the same happens again and again...
Any suggestions??
So that's what worked:
In the loop():
while(Serial.available() == 0);
delay(100);
readInputFlag = readFromKeyboard(userInput);`
And in the readFromKeyboard method:
void readFromKeyboard(byte arrayAddress[])
{
int i = 0;
while (Serial.available() > 0)
{
arrayAddress[i++] = Serial.read();
}
}
This delay, in the loop method, somehow makes the Serial get the whole string instead of just the first letter.
I know you got it working, but I wanted to show you something that I use to deal with this issue. This is a two-tiered delay system for catching bytes that come in a bit late for whatever reason. It's designed to minimize the delay needed to accomplish that task.
int received_length = 0;
byte serial_incoming_buffer[200];
while(Serial.available()) {
serial_incoming_buffer[received_length++] = Serial.read();
if(!Serial.available()) {
delay(3);
if(!Serial.available()) {
delay(20);
}
}
}
Sometimes the Arduino falls behind in picking up serial from the sender and sometimes it grabs serial too fast. Sometimes the sender lags a little bit. This code will wait 3 ms for more bytes, and if they come in it goes back to receiving as many as are available having only had that very brief delay. This repeats as necessary, then when 3 ms goes by without anything being available, it waits a bit longer (20 ms here) for more bytes. If nothing comes in after the long delay, then the transmission is most likely done and you can safely move on.
I recommend tweaking the delays based on your baud rate.
I am trying to implement a logical ring with MPI; where every process receives a message from process with id one less than that of current process and forwards to the next process in cyclic fashion. My aim is to traffic buffer such that it loses some messages or maybe puts them in out of order.
A cycle of communication will finish when a message dispatched by root node comes back again to root.
here is the code that I have tried:
I am just including relevant parts of it.
if(procId!=root)
{
sleep(100);
while(1)
{
tm = MPI_Wtime();
MPI_Irecv( &message, STR_LEN, MPI_CHAR,
((procId-1)>=0?(procId-1):(numProc-1)),RETURN_DATA_TAG,
MPI_COMM_WORLD,&receiveRequest);
MPI_Wait(&receiveRequest,&status);
printf("%d: Received\n",procId);
if(!strncmp(message,"STOP",4)&&(procId==(numProc-1)))
break;
MPI_Ssend( message, STR_LEN, MPI_CHAR,
(procId+1)%numProc, SEND_DATA_TAG, MPI_COMM_WORLD);
if(!strncmp(message,"STOP",4))
break;
printf("%d: Sent\n",procId);
}
}
else
{
for(iter=0;iter<benchmarkSize;iter++)
{
//Synthesize the message
message[STR_LEN-1] = '\0';
iErr = MPI_Bsend( message, STR_LEN, MPI_CHAR,
(root+1)%numProc, SEND_DATA_TAG, MPI_COMM_WORLD);
if (iErr != MPI_SUCCESS) {
char error_string[BUFSIZ];
int length_of_error_string;
MPI_Error_string(iErr, error_string, &length_of_error_string);
fprintf(stderr, "%3d: %s\n", procId, error_string);
}
tm = MPI_Wtime();
while(((MPI_Wtime()-tm)*1000)<delay);
printf("Root: Sending\n");
}
for(iter=0;iter<benchmarkSize;iter++)
{
MPI_Recv(message,STR_LEN,MPI_CHAR,
(numProc-1),RETURN_DATA_TAG,MPI_COMM_WORLD,&status);
//We should not wait for the messages to be received but wait for certain amount of time
//Extract the fields in the message
if(((prevRcvdSeqNum+1)!=atoi(seqNum))&&(prevRcvdSeqNum!=0))
outOfOrderMsgs++;
prevRcvdSeqNum = atoi(seqNum);
printf("Seq Num: %d\n",atoi(seqNum));
rcvdMsgs++;
printf("Root: Receiving\n");
}
MPI_Isend( "STOP", 4, MPI_CHAR,
(root+1)%numProc, SEND_DATA_TAG, MPI_COMM_WORLD,&sendRequest);
MPI_Wait(&sendRequest,&status);
/*This is to ask all other processes to terminate, when the work is done*/
}
Now, I have these questions:
1) Why is it that when I inject some sleep in the other processes(I mean other than root) of
the ring; NO receive is taking place?
2) Even when the buffer size is only one, how is it that root node is able to dispatch messages through MPI_Bsend without an error? for example the case when it needs to send total 10 messages at a rate of 1000 per second and with buffer size of 1. MPI_Bsend is able to dispatch all the messages without any error of "buffer full"; irrespective of the presence of sleep() in other processes of the ring!
Thanks a ton!
I am Working On a lab.
A father process will create two son processes A and B.
Son A will send some string to son B through pipe.son B will Invert the String case of the String Got from Son A and will send back the Inverted string to son A.after receiving the inverted string son A will print it to the screen.
here is the code.
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <ctype.h>
void process_A(int input_pipe[], int output_pipe[])
{
int c;
char ch;
int rc;
close(input_pipe[1]);
close(output_pipe[0]);
while ((c = getchar()) > 0) {
ch = (char)c;
rc = write(output_pipe[1], &ch, 1);
if (rc == -1) {
perror("A_TO_B: write");
close(input_pipe[0]);
close(output_pipe[1]);
exit(1);
}
rc = read(input_pipe[0], &ch, 1);
c = (int)ch;
if (rc <= 0) {
perror("A_TO_B: read");
close(input_pipe[0]);
close(output_pipe[1]);
exit(1);
}
putchar(c);
}
close(input_pipe[0]);
close(output_pipe[1]);
exit(0);
}
void process_B(int input_pipe[], int output_pipe[])
{
int c;
char ch;
int rc;
close(input_pipe[1]);
close(output_pipe[0]);
while (read(input_pipe[0], &ch, 1) > 0) {
c = (int)ch;
if (isascii(c) && isupper(c))
c = tolower(c);
else if (isascii(c) && islower(c))
c = toupper(c);
ch = (char)c;
rc = write(output_pipe[1], &ch, 1);
if (rc == -1) {
perror("B_TO_A: write");
close(input_pipe[0]);
close(output_pipe[1]);
exit(1);
}
}
close(input_pipe[0]);
close(output_pipe[1]);
exit(0);
}
int main(int argc, char* argv[])
{
/* 2 arrays to contain file descriptors, for two pipes. */
int A_TO_B[2];
int B_TO_A[2];
int pid;
int rc,i,State;
/* first, create one pipe. */
rc = pipe(A_TO_B);
if (rc == -1) {
perror("main: pipe A_TO_B");
exit(1);
}
/* create another pipe. */
rc = pipe(B_TO_A);
if (rc == -1) {
perror("main: pipe B_TO_A");
exit(1);
}
for(i=0;i<2;i++)
{
if((pid=fork()) <0){perror("fork failed\n");};
if((i==0) && (pid ==0))
{
process_A(A_TO_B, B_TO_A);
}
else if((i==1)&&(pid==0))
{
process_B(B_TO_A, A_TO_B);
}
else if(pid>0)
{
wait( &State );
}
}
return 0;
}
the problem is When i run the program the Son B gets Block.
I need u guys help.
Thanks in advance.
OK, diagram:
initially: parent process: has
B_TO_A[0] and [1] open,
has A_TO_B[0] and [1] open
fork (makes copy)
parent: child (pid==0):
B_TO_A both open, A_TO_B both open call process_A: close unwanted pipe ends, loop
call wait(), wait for one child loop reads stdin, writes one pipe, reads other pipe
if we ever get here:
fork (makes copy)
parent: child (pid==0):
B_TO_A both open, A_TO_B both open call process_B: close unwanted pipe ends, loop
parent: both ends of both pipes open
call wait(), wait for one child loop reads one pipe, writes other pipe
First, you will usually not get to "if we ever get here" because the child running process_A() runs in a loop until either EOF on stdin (if that occurs first) or one of the pipe read/write calls fails (e.g., due to EOF on input_pipe[0]). Since the parent is still waiting in a wait() call, and has both ends of both pipes open, there's no EOF on the pipe (EOF on a pipe occurs after you read all the data written by all writers, and all dups of the write end have been closed). So the only way to get there is to hit EOF on stdin, so that the while loop does not run.
Second, if you do get around to forking again and doing process_B(), that child will also wait forever, because one write end of the pipe it's reading from is still open... in the parent! The parent won't close it, because the parent will be waiting forever in wait.
In general, what you need to do here is:
create two pipes (like you do now)
fork once, and run process_A() in the child
fork again (in the parent), and run process_B() in the (new) child
close both ends of both pipes (in the parent)
wait for both children now, after both have gotten started
The error handling gets a bit messy since you have to do something (such as kill() the first child) if you can't start the second child. So you need to know how far along you have gotten. You can still loop to fork twice but you can't wait inside the loop, and with just two trips around the loop, each of which do rather different steps, you might as well just write it all out without a loop.
I am currently in the process of making a Client and Server in the Unix/Windows environment but right now I am just working on the Unix side of it. One of the function we have to create for the program is similar to the list function in Unix which shows all files within a dir but we also have to show more information about the file such as its owner and creation date. Right now I am able to get all this information and print it to the client however we have to also add that once the program has printing 40 lines it waits for the client to push any key before it continues to print.
I have gotta the program to sort of do this but it will cause my client and server to become out of sync or at least the std out to become out of sync. This means that if i enter the command 'asdad' it should print invalid command but it won't print that message until i enter another command. I have added my list functions code below. I am open to suggestions how how to complete this requirement as the method I have chosen does not seem to be working out.
Thank-you in advance.
Server - Fork Function: This is called when the list command is enter. eg
fork_request(newsockfd, "list", buf);
int fork_request(int fd, char req[], char buf[])
{
#ifndef WIN
int pid = fork();
if (pid ==-1)
{
printf("Failed To Fork...\n");
return-1;
}
if (pid !=0)
{
wait(NULL);
return 10;
}
dup2(fd,1); //redirect standard output to the clients std output.
close(fd); //close the socket
execl(req, req, buf, NULL); //run the program
exit(1);
#else
#endif
}
Here is the function used to get all the info about a file in a dir
void longOutput(char str[])
{
char cwd[1024];
DIR *dip;
struct dirent *dit;
int total;
char temp[100];
struct stat FileAttrib;
struct tm *pTm;
int fileSize;
int lineTotal;
if(strcmp(str, "") == 0)
{
getcwd(cwd, sizeof(cwd));
}
else
{
strcpy (cwd, str);
}
if (cwd != NULL)
{
printf("\n Using Dir: %s\n", cwd);
dip = opendir(cwd);
if(dip != NULL)
{
while ((dit = readdir(dip)) != NULL)
{
printf("\n%s",dit->d_name);
stat(dit->d_name, &FileAttrib);
pTm = gmtime(&FileAttrib.st_ctime);
fileSize = FileAttrib.st_size;
printf("\nFile Size: %d Bytes", fileSize);
printf("\nFile created on: %.2i/%.2i/%.2i at %.2i:%.2i:%.2i GMT \n", (pTm->tm_mon + 1), pTm->tm_mday,(pTm->tm_year % 100),pTm->tm_hour,pTm->tm_min, pTm->tm_sec);;
lineTotal = lineTotal + 4;
if(lineTotal == 40)
{
printf("40 Lines: Waiting For Input!");
fflush(stdout);
gets(&temp);
}
}
printf("\n %d \n", lineTotal);
}
else
{
perror ("");
}
}
}
At here is the section of the client where i check that a ! was not found in the returned message. If there is it means that there were more lines to print.
if(strchr(command,'!') != NULL)
{
char temp[1000];
gets(&temp);
}
Sorry for the long post but if you need anything please just ask.
Although, I didn't see any TCP/IP code, I once had a similar problem when I wrote a server-client chat program in C++. In my case, the problem was that I didn't clearly define how messages were structured in my application. Once, I defined how my protocol was suppose to work--it was a lot easier to debug communication problems.
Maybe you should check how your program determines if a message is complete. In TCP, packets are guaranteed to arrive in order with no data loss, etc. Much like a conversation over a telephone. The only thing you have to be careful of is that it's possible to receive a message partially when you read the buffer for the socket. The only way you know to stop reading is when you determine a message is complete. This could be as simple as two '\n' characters or "\n\r".
If you are using UDP, then that is a completely different beast all together (i.e. messages can arrive out of order and can be lost in transit, et cetera).
Also, it looks like you are sending across strings and no binary data. If this is the case, then you don't have to worry about endianess.