printf alternative when using "define _GNU_SOURCE" - serial-port

After reading https://www.quora.com/How-can-I-bypass-the-OS-buffering-during-I-O-in-Linux I want to try to access data on the serial port with the O_DIRECT option, but the only way I can seem to do that is by adding the GNU_SOURCE define but when I tried to execute the program, nothing at all is printed on the screen.
If I remove "#define _GNU_SOURCE" and compile, then the system gives me an error on O_DIRECT.
If I remove the define and the O_DIRECT flag, then incorrect (possibly outdated) data is always read, but the data is printed on the screen.
I still want to use the O_DIRECT flag and be able to see the data, so I feel I need an alternative command to printf and friends, but I don't know how to continue.
I attached the code below:
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <time.h>
#include <unistd.h>
#include <termios.h>
#define TIMEOUT 5
int main(){
char inb[3]; //our byte buffer
int nread=0; //number bytes read from port
int n; //counter
int iosz=128; //Lets get 128 bytes
int fd=open("/dev/ttyS0", O_NOCTTY | O_RDONLY | O_SYNC | O_DIRECT); //Open port
tcflush(fd,TCIOFLUSH);
for(n=0;n<iosz;n++){
int s=time(NULL); //Start timer for 5 seconds
while (time(NULL)-s < TIMEOUT && nread < 1){
inb[0]='A'; //Fill buffer with bad data
inb[1]='B';
inb[2]='C';
nread=read(fd,(char*)inb,1); //Read ONE byte
tcflush(fd,TCIOFLUSH);
if (nread < 0 || time(NULL)-s >= TIMEOUT){
close(fd); //Exit if read error or timeout
return -1;
}
}
printf("%x:%d ",inb[0] & 0xFF,nread); //Print byte as we receive it
}
close(fd); //program ends so close and exit
printf("\n"); //Print byte as we receive it
return 0;
}

First off, I'm no expert on this topic, just curious about it, so take this answer with a pinch of salt.
I don't know if what you're trying to do here (if I'm not looking at it the wrong way it seems to be to bypass the kernel and read directly from the port to userspace) was ever a possibility (you can find some examples, like this one but I could not find anything properly documented) but with recent kernels you should be getting an error running your code, but you're not catching it.
If you add these lines after declaring your port:
...
int fd=open("/dev/ttyS0", O_NOCTTY | O_RDONLY | O_SYNC | O_DIRECT );
if (fd == -1) {
fprintf(stderr, "Error %d opening SERIALPORT : %s\n", errno, strerror(errno));
return 1;
}
tcflush(fd,TCIOFLUSH);
....
When you try to run you'll get: Error 22 opening SERIALPORT : Invalid argument
In my humble and limited understanding, you should be able to get the same effect changing the settings on termios to raw, something like this should do:
struct termios t;
tcgetattr(fd, &t); /* get current port state */
cfmakeraw(&t); /* set port state to raw */
tcsetattr(fd, TCSAFLUSH, &t); /* set updated port state */
There are many good sources for termios, but the only place I could find taht also refers to O_DIRECT (for files) is this one.

Related

Can I link two separate executables with MPI_open_port and share port information in a text file?

I'm trying to create a shared MPI COMM between two executables which are both started independently, e.g.
mpiexec -n 1 ./exe1
mpiexec -n 1 ./exe2
I use MPI_Open_port to generate port details and write these to a file in exe1 and then read with exe2. This is followed by MPI_Comm_connect/MPI_Comm_accept and then send/recv communication (minimal example below).
My question is: can we write port information to file in this way, or is the MPI_Publish_name/MPI_Lookup_name required for MPI to work as in this, this and this? As supercomputers usually share a file system, this file based approach seems simpler and maybe avoids establishing a server.
It seems this should work according to the MPI_Open_Port documentation in the MPI 3.1 standard,
port_name is essentially a network address. It is unique within the communication universe to which it belongs (determined by the implementation), and may be used by any client within that communication universe. For instance, if it is an internet (host:port) address, it will be unique on the internet. If it is a low level switch address on an IBM SP, it will be unique to that SP
In addition, according to documentation on the MPI forum:
The following should be compatible with MPI: The server prints out an address to the terminal, the user gives this address to the client program.
MPI does not require a nameserver
A port_name is a system-supplied string that encodes a low-level network address at which a server can be contacted.
By itself, the port_name mechanism is completely portable ...
Writing the port information to file does work as expected, i.e creates a shared communicator and exchanges information using MPICH (3.2) but hangs at the MPI_Comm_connect/MPI_Comm_accept line when using OpenMPI versions 2.0.1 and 4.0.1 (on my local workstation running Ubuntu 12.04 but eventually needs to work on a tier 1 supercomputer). I have raised as an issue here but welcome a solution or workaround in the meantime.
Further Information
If I use the MPMD mode with OpenMPI,
mpiexec -n 1 ./exe1 : -n 1 ./exe2
this works correctly, so must be an issue with allowing the jobs to share ompi_global_scope as in this question. I've also tried adding,
MPI_Info info;
MPI_Info_create(&info);
MPI_Info_set(info, "ompi_global_scope", "true");
with info passed to all commands, with no success. I'm not running a server/client model as both codes run simultaneously so sharing a URL/PID from one is not ideal, although I cannot get this to work even using the suggested approach, which for OpenMPI 2.0.1,
mpirun -n 1 --report-pid + ./OpenMPI_2.0.1 0
1234
mpirun -n 1 --ompi-server pid:1234 ./OpenMPI_2.0.1 1
gives,
ORTE_ERROR_LOG: Bad parameter in file base/rml_base_contact.c at line 161
This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):
pmix server init failed
--> Returned value Bad parameter (-5) instead of ORTE_SUCCESS
and with OpenMPI 4.0.1,
mpirun -n 1 --report-pid + ./OpenMPI_4.0.1 0
1234
mpirun -n 1 --ompi-server pid:1234 ./OpenMPI_4.0.1 1
gives,
ORTE_ERROR_LOG: Bad parameter in file base/rml_base_contact.c at line 50
...
A publish/lookup server was provided, but we were unable to connect
to it - please check the connection info and ensure the server
is alive:
Using 4.0.1 means the error should not be related to this bug in OpenMPI.
Minimal code
#include "mpi.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <iostream>
#include <fstream>
using namespace std;
int main( int argc, char *argv[] )
{
int num_errors = 0;
int rank, size;
char port1[MPI_MAX_PORT_NAME];
char port2[MPI_MAX_PORT_NAME];
MPI_Status status;
MPI_Comm comm1, comm2;
int data = 0;
char *ptr;
int runno = strtol(argv[1], &ptr, 10);
for (int i = 0; i < argc; ++i)
printf("inputs %d %d %s \n", i,runno, argv[i]);
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (runno == 0)
{
printf("0: opening ports.\n");fflush(stdout);
MPI_Open_port(MPI_INFO_NULL, port1);
printf("opened port1: <%s>\n", port1);
//Write port file
ofstream myfile;
myfile.open("port");
if( !myfile )
cout << "Opening file failed" << endl;
myfile << port1 << endl;
if( !myfile )
cout << "Write failed" << endl;
myfile.close();
printf("Port %s written to file \n", port1); fflush(stdout);
printf("Attempt to accept port1.\n");fflush(stdout);
//Establish connection and send data
MPI_Comm_accept(port1, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &comm1);
printf("sending 5 \n");fflush(stdout);
data = 5;
MPI_Send(&data, 1, MPI_INT, 0, 0, comm1);
MPI_Close_port(port1);
}
else if (runno == 1)
{
//Read port file
size_t chars_read = 0;
ifstream myfile;
//Wait until file exists and is avaialble
myfile.open("port");
while(!myfile){
myfile.open("port");
cout << "Opening file failed" << myfile << endl;
usleep(30000);
}
while( myfile && chars_read < 255 ) {
myfile >> port1[ chars_read ];
if( myfile )
++chars_read;
if( port1[ chars_read - 1 ] == '\n' )
break;
}
printf("Reading port %s from file \n", port1); fflush(stdout);
remove( "port" );
//Establish connection and recieve data
MPI_Comm_connect(port1, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &comm1);
MPI_Recv(&data, 1, MPI_INT, 0, 0, comm1, &status);
printf("Received %d 1\n", data); fflush(stdout);
}
//Barrier on intercomm before disconnecting
MPI_Barrier(comm1);
MPI_Comm_disconnect(&comm1);
MPI_Finalize();
return 0;
}
The 0 and 1 simply specify if this code writes a port file or reads it in the example above. This is then run with,
mpiexec -n 1 ./a.out 0
mpiexec -n 1 ./a.out 1

How to do a non-blocking read on a non-socket fd

Is there a way to do a single read() in non-blocking mode on a pipe/terminal/etc, the way I can do it on a socket with recv(MSG_DONTWAIT)?
The reason I need that is because I cannot find any guarantee that a read() on a file-descriptor returned as ready for reading by select() or poll() will not block.
I know can make the file descriptor non-blocking with fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_NONBLOCK) but this will change the mode on that file descriptor globally, not just in the calling thread/process. For example:
% perl -MFcntl=F_SETFL,F_GETFL,O_NONBLOCK -e 'fcntl STDIN, F_SETFL, fcntl(STDIN, F_GETFL, 0) | O_NONBLOCK; select undef, undef, undef, undef'
^Z # put it in the background
% cat
cat: -: Resource temporarily unavailable
This will also make the fd non blocking for both reading and writing, which may confuse the hell out of another process doing the opposite on the same fd, as in:
non_blocking_read | filter | blocking_write
One way I think of is to save the file status flags on starting up and SIGCONT, and restore them on exiting and on SIGTSTP (just the way it's done with the termios settings), but this is very limited, race-prone, and will leave a mess behind in the case where the program exited abnormally.
Putting a save/restore with fcntl() before/after each read() also feels ugly and dumb, and may have other issues too. The same with an ioctl(FIONREAD) just before the read (which I'm not even sure it will work reliably with any fd; assurances in that direction will be welcome, though).
I would be happy even with system specific (eg. linux or bsd-only) solutions.
For reference, here is a discussion about fixing it in linux; the idea didn't seem to get anywhere, though.
A Linux only solution would be to reopen the file descriptor via
"/dev/stdin"|"/dev/tty"|"/dev/fd/$fd".
C example:
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
int main()
{
int fd;
char buf[8];
int flags;
if(0>(fd=open("/dev/stdin", O_RDONLY))) return 1;
if(0>(flags = fcntl(fd,F_GETFL))) return 1;
if(0>(flags = fcntl(fd,F_SETFL,flags|O_NONBLOCK))) return 1;
sleep(3);
puts("reading");
ssize_t nr = read(fd, buf, sizeof(buf));
printf("read=%zd\n", nr);
return 0;
}
Unlike a duplicated file descriptor, a reopened filedescriptor will have independent file status flags.

audio decoding error using libavcodec avcodec_decode_audio4

At present, I am trying audio decoding for the first time using avcodec_decode_audio4(), which is always returning error.
Note: My overall intention is to merge multiple AMR_NB encoded files into one audio file, and eventually mux this final audio file with another video file into .mp4 container.
Here's the code:
if(avformat_open_input(&sound, "/tmp/aud0", NULL, NULL)!=0) {
printf("Error in opening input file aud0\n");
exit(1);
} else {
printf("Opened aud0 into FormatContext\n");
}
if(avformat_find_stream_info(sound, NULL)<0){
printf("Error in finding stream info\n");
avformat_close_input(&sound);
exit(1);
}
int aud_stream_idx=-1;
for(int count=0; count<sound->nb_streams; count++) {
if(sound->streams[count]->codec->codec_type == AVMEDIA_TYPE_AUDIO) {
aud_stream_idx = count;
printf("Audio stream index %d found\n", count);
break;
}
}
if(aud_stream_idx==-1) {
printf("Audio stream not found in the aud0 file\n");
avformat_close_input(&sound);
exit(1);
}
AVCodecContext *audioCC = sound->streams[aud_stream_idx]->codec;
printf("Audio codec_id=%d, codec_name=%s\n", audioCC->codec_id, audioCC->codec_name);
AVCodec *audioC = avcodec_find_decoder(audioCC->codec_id);
if (audioC == NULL) { printf("Couldn't find decoder\n");
avformat_close_input(&sound);
exit(1);
} else {
printf("Found decoder name %s\n", audioCC->codec_name);
}
if(avcodec_open2(audioCC, audioC, NULL)<0) {
printf("Couldn't open decoder\n");
avformat_close_input(&sound);
exit(1);
} else {
printf("Found decoder name %s\n", audioCC->codec_name);
printf("Found bitrate %d\n", audioCC->bit_rate);
printf("Found sample_rate %d\n", audioCC->sample_rate);
printf("Found sample_fmt %d\n", audioCC->sample_fmt); }
/* decode until eof */
av_init_packet(&avpkt);
avpkt.data=NULL;
avpkt.size=0;
if(av_read_frame(sound, &avpkt)<0) {
printf("Couldn't read encoded audio packet\n");
av_free_packet(&avpkt);
avformat_close_input(&sound);
exit(1);
} else {
printf("Read encoded audio packet\n");
printf("avpkt.pts = %d\n", avpkt.pts);
printf("avpkt.dts = %d\n", avpkt.dts);
printf("avpkt.duration = %d\n", avpkt.duration);
printf("avpkt.stream_index = %d\n", avpkt.stream_index);
printf("avpkt.data = %x\n", avpkt.data);
printf("avpkt.data[0] = %02x\n", avpkt.data[0]);
printf("avpkt.data[0]>>>3&0x0f = %02x\n", avpkt.data[0]>>3 & 0x0f);
}
fprintf(stderr, "avpkt.size=%d\n", avpkt.size);
while (avpkt.size > 0) {
int got_frame = 0;
if (!decoded_frame) {
if (!(decoded_frame = avcodec_alloc_frame())) {
fprintf(stderr, "out of memory\n");
exit(1);
} } else
avcodec_get_frame_defaults(decoded_frame);
len = avcodec_decode_audio4(audioCC, decoded_frame, &got_frame, &avpkt);
if (len < 0) {
fprintf(stderr, "Error %d while decoding\n", len);
exit(1);
}
Here is the output I see (I think demuxing is working fine):
Opened aud0 into FormatContext
Audio stream index 0 found
Audio codec_id=73728, codec_name=
Found decoder name
Found decoder name
Found bitrate 5200
Found sample_rate 8000
Found sample_fmt 3
Read encoded audio packet
avpkt.pts = 0
avpkt.dts = 0
avpkt.duration = 160
avpkt.stream_index = 0
avpkt.data = 91e00680
avpkt.data[0] = 04
avpkt.data[0]>>>3&0x0f = 00
avpkt.size=13
[amrnb # 0x7fce9300bc00] Corrupt bitstream
Error -1094995529 while decoding
This audio was created by recording voice on an Android device.
I also tried a .m4a generated by QuickTime, but no luck.
I feel I am missing some crucial step (like not initializing some CodecContext field, or not reading into AVPacket properly). In any case, if someone has any input or similar example, please let me know.
To help others understand ffmpeg error codes, this may be helpful. Error codes from ffmpeg (error.h from avutil) :
http://ffmpeg.org/doxygen/trunk/error_8h_source.html
It turns out the value you specified is :
#define AVERROR_INVALIDDATA FFERRTAG( 'I','N','D','A')
The -1094995529 becomes -0x41444E49 and when you look at those letters, in ACSII, 0x41 = 'A', 0x44 = 'D', 0x4E = 'N, and 0x49 = 'I'. Due to the macro/etc things are reversed, so ADNI becomes INDA, which you can see from the #define snippet, is the AVERROR_INVALIDDATA defined FFERRTAG( 'I','N','D','A').
The rest of the error codes are in that file and I've pasted them below here :
#define AVERROR_BSF_NOT_FOUND FFERRTAG(0xF8,'B','S','F') ///< Bitstream filter not found
#define AVERROR_BUG FFERRTAG( 'B','U','G','!') ///< Internal bug, also see AVERROR_BUG2
#define AVERROR_BUFFER_TOO_SMALL FFERRTAG( 'B','U','F','S') ///< Buffer too small
#define AVERROR_DECODER_NOT_FOUND FFERRTAG(0xF8,'D','E','C') ///< Decoder not found
#define AVERROR_DEMUXER_NOT_FOUND FFERRTAG(0xF8,'D','E','M') ///< Demuxer not found
#define AVERROR_ENCODER_NOT_FOUND FFERRTAG(0xF8,'E','N','C') ///< Encoder not found
#define AVERROR_EOF FFERRTAG( 'E','O','F',' ') ///< End of file
#define AVERROR_EXIT FFERRTAG( 'E','X','I','T') ///< Immediate exit was requested; the called function should not be restarted
#define AVERROR_EXTERNAL FFERRTAG( 'E','X','T',' ') ///< Generic error in an external library
#define AVERROR_FILTER_NOT_FOUND FFERRTAG(0xF8,'F','I','L') ///< Filter not found
#define AVERROR_INVALIDDATA FFERRTAG( 'I','N','D','A') ///< Invalid data found when processing input
#define AVERROR_MUXER_NOT_FOUND FFERRTAG(0xF8,'M','U','X') ///< Muxer not found
#define AVERROR_OPTION_NOT_FOUND FFERRTAG(0xF8,'O','P','T') ///< Option not found
#define AVERROR_PATCHWELCOME FFERRTAG( 'P','A','W','E') ///< Not yet implemented in FFmpeg, patches welcome
#define AVERROR_PROTOCOL_NOT_FOUND FFERRTAG(0xF8,'P','R','O') ///< Protocol not found
#define AVERROR_STREAM_NOT_FOUND FFERRTAG(0xF8,'S','T','R') ///< Stream not found
#define AVERROR_BUG2 FFERRTAG( 'B','U','G',' ')
#define AVERROR_UNKNOWN FFERRTAG( 'U','N','K','N') ///< Unknown error, typically from an external library
#define AVERROR_EXPERIMENTAL (-0x2bb2afa8) ///< Requested feature is flagged experimental. Set strict_std_compliance if you really want to use it.
#define AVERROR_INPUT_CHANGED (-0x636e6701) ///< Input changed between calls. Reconfiguration is required. (can be OR-ed with AVERROR_OUTPUT_CHANGED)
#define AVERROR_OUTPUT_CHANGED (-0x636e6702) ///< Output changed between calls. Reconfiguration is required. (can be OR-ed with AVERROR_INPUT_CHANGED)
#define AVERROR_HTTP_BAD_REQUEST FFERRTAG(0xF8,'4','0','0')
#define AVERROR_HTTP_UNAUTHORIZED FFERRTAG(0xF8,'4','0','1')
#define AVERROR_HTTP_FORBIDDEN FFERRTAG(0xF8,'4','0','3')
#define AVERROR_HTTP_NOT_FOUND FFERRTAG(0xF8,'4','0','4')
#define AVERROR_HTTP_OTHER_4XX FFERRTAG(0xF8,'4','X','X')
#define AVERROR_HTTP_SERVER_ERROR FFERRTAG(0xF8,'5','X','X')
first you need to call av_read_frame until you get to EOF. So you need av_read_frame in the loop.
Let me explain how decoding works:
you open formatcontext, codeccontext and decoder. that is the easy part.
in a loop you read packets form one stream. so you need
while(av_read_frame(sound, &avpkt))
in this loop you want to decode the data from the packet. but care, you need to check from which stream the data in the packet is. so its always better to check if the packet contains data from the audiostream
if (avpkt.stream_index == aud_stream_idx)
now you can decode and process the audio.
exmaple:
AVFrame * decoded_frame = av_frame_alloc();
AVPacket avpkt;
av_init_packet(&avpkt);
avpkt.data = NULL;
avpkt.size = 0;
while(av_read_frame(sound, &avpkt))
{
if (avpkt.stream_index == aud_stream_idx)
{
int got_frame;
int len = avcodec_decode_audio4(audioCC, decoded_frame, &got_frame, &avpkt);
}
}
Thats all the the magic :)
PS: to find a stream just call
av_find_best_stream(sound, AVMEDIA_TYPE_AUDIO, -1, -1, NULL, 0)
PPS: did not tested code. just codec here in window :D
PPPS: read the examples. this helped me a lot :)

iptables netfilter copying the packet

I was wondering if there is a way to copy a packet using iptables/netfilter, change it and deliver both to the application.
Basically, I want to capture a packet from a flow and redirect it to some queue, then I want to copy it, issue the verdict for it(I know how to do this part in C),then I need to change something in the copied version, AND issue the verdict for that "modified" packet too.
Basically I want the app to receive both the unmodified and the modified version.
Is this possible?
Thanks in advance for any help.
Your mission can be achieved with libipq library. The tutorial in following like focus on copying & modifying a packet in userspace.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.205.2605&rep=rep1&type=pdf
You need to know C to work on it. Alternatively "Scapy" - a python based packet maipulation tool can be used.
#include <linux/netfilter.h>
#include <libipq.h>
/*
* Used to open packet ; Insert a iptables rule to get packet here
* iptables -I 1 [INPUT|OUTPUT|FORWARD] <packet header match> -j QUEUE
*/
#include <linux/netfilter.h>
#include <libipq.h>
#include <stdio.h>
#define BUFSIZE 2048
static void die(struct ipq_handle *h)
{
ipq_destroy_handle(h);
exit(1);
}
int main(int argc, char **argv)
{
int status;
unsigned char buf[BUFSIZE];
struct ipq_handle *h;
h = ipq_create_handle(0, NFPROTO_IPV4);
if (!h)
die(h);
status = ipq_set_mode(h, IPQ_COPY_PACKET, BUFSIZE);
if (status < 0)
die(h);
do{
status = ipq_read(h, buf, BUFSIZE, 0);
if (status < 0)
die(h);
if (ipq_message_type(buf) == IPQM_PACKET){
ipq_packet_msg_t *m = ipq_get_packet(buf);
status = ipq_set_verdict(h, m->packet_id, NF_ACCEPT, 0, NULL);
}
} while (1);
ipq_destroy_handle(h);
return 0;
}

MPI: Why does my MPICH program fails for large no. of processes?

/* C Example */
#include <mpi.h>
#include <stdio.h>
#include <stddef.h>
#include <stdlib.h>
int main (int argc, char* argv[])
{
int rank, size;
int buffer_length = MPI_MAX_PROCESSOR_NAME;
char hostname[buffer_length];
MPI_Init (&argc, &argv); /* starts MPI */
MPI_Comm_rank (MPI_COMM_WORLD, &rank); /* get current process id */
MPI_Comm_size (MPI_COMM_WORLD, &size); /* get number of processes */
MPI_Get_processor_name(hostname, &buffer_length); /* get hostname */
printf( "Hello world from process %d running on %s of %d\n", rank, hostname, size );
MPI_Finalize();
return 0;
}
The above program compiles and run successfully on ubuntu 12.04 for smaller no. of processes. But it fails when I try to execute with 1000s of processes. Why it is so?
I am expecting that scheduler would keep the threads in queue and can dispatch one by one (I am running this code on a single core machine)
Why the following error is coming for large no. of processes and how to resolve this issue?
root#ubuntu:/home# mpiexec -n 1000 ./hello
[proxy:0:0#ubuntu] HYDU_create_process (./utils/launch/launch.c:26): pipe error (Too many open files)
[proxy:0:0#ubuntu] launch_procs (./pm/pmiserv/pmip_cb.c:751): create process returned error
[proxy:0:0#ubuntu] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:935): launch_procs returned error
[proxy:0:0#ubuntu] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:0#ubuntu] main (./pm/pmiserv/pmip.c:226): demux engine error waiting for event
Killed
You are running into the open file limit on your system. Default in Ubuntu is 1024. You can try raising the limit in your session with the ulimit command.
ulimit -n 2048

Resources