I use the following function to read from a file descriptor...
int cread(int fd, char *buf, int n){
int nread;
if((nread=read(fd, buf, n))<0){
perror("Reading data");
exit(1);
}
return nread;
}
Following is the function that uses the above function
if(FD_ISSET(tap_fd, &rd_set)){
/* data from tun/tap: just read it and write it to the network */
nread = cread(tap_fd, buffer, BUFSIZE);
tap2net++;
do_debug("TAP2NET %lu: Read %d bytes from the tap interface\n", tap2net, nread);
/* write length + packet */
plength = htons(nread);
nwrite = cwrite(net_fd, (char *)&plength, sizeof(plength));
nwrite = cwrite(net_fd, buffer, nread);
do_debug("TAP2NET %lu: Written %d bytes to the network\n", tap2net, nwrite);
}
They both work fine with TCP siocket but not with udp socket.. Any help would be appreciated
It's not clear exactly what your problem is, but if net_fd is a UDP socket, then the two cwrite() calls will create two UDP datagrams.
There isn't a great deal of point in prepending the size with UDP - UDP maintains the message boundaries for you. So in the UDP case, just remove the plength part entirely.
Related
I'm constantly sending structs of int64 via Pyserial with:
with serial.Serial(port='COM4', baudrate=115200, timeout=.1) as arduino:
value = write_read(struct.pack(">q", int_array[1][i])) #this sends signed int.64 in bytes
print(value)
the struct.pack has this shape, for example:
b'\xff\xff\xff\xff\xff\xff\xff\xef'
and the function write_read consists of:
def write_read(x):
arduino.write((x))
data = arduino.readline()
#the idea is to receive an ACK from the Arduino after 8 bytes (the full
#number)
return data
The code I'm trying to develop in arduino is the following:
void loop() {
// send data only when you receive data:
if (Serial.available() \> 0) {
// read the incoming byte:
incomingByte = Serial.read();
//read 8 bytes and create the result
r= function_to_read_8_last_bytes // or similar
// say what you got:
Serial.print("I received: ");
Serial.printlesultn(r, DEC);
Serial.write("ACK");
}
}
I'm very curious how I could do a robust "read 8 bytes" function.
Should I add some especial character in the Python part to indentify when it ends one value?
Thanks! I'll appreciate any help :)
Given the discussion in the comments, it's hard to receive a stream of bytes and be sure that the receiver is completely synchronized. However let's make some assumptions to ease the problem:
The serial buffer is empty when you connect your laptop to Arduino. This ensures you won't receive spurious data with no meaning. I had this problem happens a lot when the serial connection was ended abruptly by any cause.
You are not constantly sending bytes, Arduino has time to process them until the start of the new sequence.
You only send this data, so there is no need to create a higher level protocol on top of it. Bare in mind that the serial communication is almost just an hardware stack, you receive bytes with no headers.
For assumption 1 you can write a simple piece of code to consume all the spurious bytes in the serial buffer as soon as your main starts from Arudino, so this will be done everytime you connect the serial (as this is also where the power supply comes from). Something like this:
void serialFlush(){
while(Serial.available() > 0) {
char t = Serial.read();
}
}
You can send a "READY" signal back to the Python interface, so that the program knows you are ready to receive data.
Going on with the solution you can implement an easy CRC in python, an additional byte which contains a XOR of all the previous bytes, and you check that in Arduino upon reception complete.
def xor_reduce_long_int(li):
res = 0;
for i in range(8):
mask = (0xFF)<<(i*8)
print(hex(mask))
masked = (li&mask)>>(i*8)
res ^= masked
return res
with serial.Serial(port='COM4', baudrate=115200, timeout=.1) as arduino:
crc=xor_reduce_long_int(int_array[1][i])
value = write_read(struct.pack(">qc", int_array[1][i],crc)) #this sends signed int.64 in bytes
print(value)
And with Arduino I would read 8 bytes when they are available and put them into an unsigned char buffer. I would then define a union that alias such buffer to interpret it as long long int.
typedef struct long_int_CRC
{
union
{
unsigned char bytes[8];
long int data;
};
unsigned char CRC;
}data_T;
// .. Later in main
data_T = received_data;
int received_bytes=0
unsigned char my_CRC = 0;
unsigned char rec_byte= 0;
while( received_bytes < 8 )
{
if(Serial.available() )
{
// Get the byte
rec_byte = Serial.read()
// Store the byte and calc CRC
received_data.bytes[received_bytes] = rec_byte;
my_CRC ^= rec_byte;
// Increment counter for next byte
received_bytes++;
}
}
// Reception complete, check CRC
unsigned char rec_CRC;
if(Serial.available() )
{
rec_CRC = Serial.read()
}
if( my_CRC != rec_CRC )
{
// Something was wrong!
}
// Now access your data as a long int
Serial.print("I received: ");
Serial.printlesultn(received_data.data, DEC);
Serial.write("ACK");
I red about netmap which allows user programmers to access packets in the user space, that means user applications can read / send network packets very quickly using this netamp.
netmap :
http://info.iet.unipi.it/~luigi/netmap/
Can any one who are very familiar with netamp, tell me should we create entire packet that we want to send out, or we using the stack features to send it out.
Edit : here is example on how to use this api
https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4
#include <net/netmap_user.h>
void sender(void)
{
struct netmap_if *nifp;
struct netmap_ring *ring;
struct nmreq nmr;
struct pollfd fds;
fd = open("/dev/netmap", O_RDWR);
bzero(&nmr, sizeof(nmr));
strcpy(nmr.nr_name, "ix0");
nmr.nm_version = NETMAP_API;
ioctl(fd, NIOCREGIF, &nmr);
p = mmap(0, nmr.nr_memsize, fd);
nifp = NETMAP_IF(p, nmr.nr_offset);
ring = NETMAP_TXRING(nifp, 0);
fds.fd = fd;
fds.events = POLLOUT;
for (;;) {
poll(&fds, 1, -1);
while (!nm_ring_empty(ring)) {
i = ring->cur;
buf = NETMAP_BUF(ring, ring->slot[i].buf_index);
// here they are saying to construct the packet
... prepare packet in buf ...
ring->slot[i].len = ... packet length ...
ring->head = ring->cur = nm_ring_next(ring, i);
}
}
}
You need to create entire packed, including ethernet, ip and tcp headers. Netmap completely bypasses kernel network stack, so you need to do all work yourself.
I'm just learning how to handle sockets and TCP connections in C. I've got an application (a long one) which basically sends and receives char arrays with the system call write from server to client and vice versa (two separate C applications of course). As long as I use it with a local connection, on the same PC, running the server on a terminal and the client on an another, everything just works fine and the data arrives at the destination. But if I try it with the server on one computer and the client on another but on the same internet line, passing to the client an address like 192.168.1.X (took from the machine on which the server is running), after the connection is established, I've got an error that tells me that the number of expected bytes (which I pass before sending the real char[]) isn't arrived. Same thing if I try the server on my PC, and the client on another one with a different line on a different provider.
There's something I'm missing, are there any limitations in sending a bunch of bytes in sequence?
The code where the error pops up.
SERVER SIDE:
r=htonl(lghstr);
w=write(myFd,&r,sizeof(int));//writes the number of incoming bytes
if(w<0) perror("writeServer4"),exit(-1);
w=write(myFd,tmp->string,lghstr);
if(w<0) perror("writeServer5"),exit(-1);
if(w!=lghstr) perror("ERROR");
CLIENT SIDE
rC=read(fdc,&cod,sizeof(int));//read incoming number of bytes
lghstr=ntohl(cod);
if(rC<0) perror("readClient3"),exit(-1);
rC=read(fdc,dest,lghstr);
if(rC<0) perror("readClient4"),exit(-1);
if(rC!=lghstr) perror("error : "), printf("didn't read the right number of bytes"),exit(-1);
Now this is basically repeated a lot of times, let's even say 300 times, and it's with big numbers that the program doesn't work.
This is the problem:
rC=read(fdc,dest,lghstr);
...
if(rC!=lghstr) perror("error : ")
The #1 fallacy with socket programming is expecting that recv() and read() will return exactly the same number of bytes corresponding to the write/send call made by the other side.
In reality, partial data is extremely likely and expected. The simple workaround is to loop on read/recv until you get the exact number of bytes expected:
size_t count = 0;
while (count < lghstr)
{
ssize_t readresult = read(fdc, dest+count, lghstr-count);
if (readresult == -1)
{
// socket error - handle appropriately (typically, just close the connection)
}
else if (readresult == 0)
{
// The other side closed the connection - handle appropriately (close the connection)
}
else
{
count += readresult;
}
}
The other alternative to looping is to the use the MSG_WAITALL flag with the socket. This means, using recv() instead of read(). You'll still need to handle the error cases.
rc = recv(fdc, dest, lghstr, MSG_WAITALL);
if (rc == -1)
{
// socket error
}
else if (rc == 0)
{
// socket closed by remote
}
else if (rc < lghstr)
{
// the other side likely closed the connection and this is residual data (next recv will return 0)
}
You do ntohl() on one side and not the other. That might be interpreting the bytes with the wrong value.
You should printf() the bytes on both sides and see what the int is being evaluated to.
Edit: I'm convinced this is a programming bug for the record.
If I had to guess, I'd say that you are not synchronous with the other side for some reason. You say this runs 'about 300 times'.
Try adding a magic integer to the protocol.
Heres an example of a client that sends in this order.
A magic integer which is always constant.
A lengh of bytes about to be sent.
The bytes to be sent.
This uses scatter gather mechanics (its nicer for serialization) but other than that it effectively is doing the same thing yours is doing, as a client, just adding a magic value.
When the receiver receives the data, it can validate that the data is coming in the right order, by checking what the magic number was that came in. If the magic is wrong it means the client or server has lost themselves positionally in the stream.
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <err.h>
#include <time.h>
#define MAGIC 0xDEADBEEFLU
#define GARBAGE_MAX 65536
const int iterations = 3000;
char * create_garbage_buf(
void)
{
int rc = -1;
int fd = -1;
char *buf = NULL;
buf = malloc(GARBAGE_MAX);
if (!buf)
err(1, "Cannot allocate buf");
fd = open("/dev/urandom", O_RDONLY);
if (fd < 0)
err(1, "Cannot open urandom");
rc = read(fd, buf, GARBAGE_MAX);
if (rc < 0)
err(1, "Cannot read from urandom");
else if (rc != GARBAGE_MAX)
errx(1, "Expected %d bytes, but got %d reading from urandom",
GARBAGE_MAX, rc);
close(fd);
return buf;
}
int main() {
int fd, offset, i, rc;
uint32_t magic = MAGIC;
uint32_t blen = 0;
char *buf = NULL;
struct iovec vecs[3];
/* Seed poor random number generator */
srand(time(NULL));
/* Use a file for demonstration, but a socket will do just fine */
fd = open("/dev/null", O_WRONLY);
/* Create some garbage to send */
buf = create_garbage_buf();
if (fd < 0)
err(1, "Cannot open file");
/* The first vector, is always the magic */
vecs[0].iov_len = sizeof(uint32_t);
vecs[0].iov_base = &magic;
for (i=0; i < iterations; i++) {
/* The second vector represents lengh of what we send
* in this demonstration it is a number between 0 and
* GARBAGE_MAX/2.
*/
blen = rand() % (GARBAGE_MAX / 2);
vecs[1].iov_len = sizeof(uint32_t);
vecs[1].iov_base = &blen;
/* The last record is the data to send. Its another random
* number between 0 and GARBAGE_MAX which represents the offset
* in our garbage data to send */
offset = rand() % (GARBAGE_MAX / 2);
vecs[2].iov_len = blen;
vecs[2].iov_base = &buf[offset];
rc = writev(fd, vecs, 3);
if (rc < 0)
err(1, "Could not write data");
if (rc != (sizeof(uint32_t)*2 + blen))
errx(1, "Did not write proper number of bytes to handle");
printf("Wrote %u bytes from offset %u in garbage\n", blen, offset);
}
free(buf);
printf("Done!\n");
return 0;
}
Closely read the documentation for read()/write() and learn that those two functions do not necessarily read()/write() as much bytes as they were told to, but few. So looping around such calls counting until all data expected had been read/written is a good idea, not to say an essential necessity.
For examples how this could be done for writing you might like to have look at this answer: https://stackoverflow.com/a/24260280/694576 and for reading on this answer: https://stackoverflow.com/a/20149925/694576
I am trying to send a small packet via tcp but i have a problem with the size of it.
My data is 255 byte buffer:
buffer[0] = 0x00;
buffer[1] = 0x04;
buffer[2] = 0x06;
buffer[3] = 0x08;
buffer[4] = 0x01;
buffer[5] = 0x01;
and i use
send(sockfd,buffer,6,MSG_NOSIGNAL|MSG_DONTWAIT);
but data can not be send when i use 6. If i use send like:
send(sockfd,buffer,sizeof(buffer),MSG_NOSIGNAL|MSG_DONTWAIT);
then data is sent but recevier has to parse extra 250 0x00 byte and i don't want it. Why 6 is no ok. I also try 10 randomly nothing changes.
Here is my receiver code Anything writes at that side:
while(1) {
ret = recv(socket,buffer,sizeof(buffer),0);
if (ret > 0)
printf("recv success");
else
{
if (ret == 0)
printf("recv failed (orderly shutdown)");
else
printf("recv failed (errno:%d)",errno);
}
}
Get rid of MSG_DONTWAIT. This is only useful when you have something else to do when the socket send buffer is full and the data can't be sent. As you aren't checking the result of send(), clearly you aren't interested in that condition. However you must check the result of send anyway, as you may have got an error, so fix your code to do that as well.
I've setup a simple TCP file transfer. Everything appears to work OK, except for the received file size is sporadically a smaller size than the file that was sent. There doesn't appear to be any pattern to the size of the received file.
(in the code below, note that the typical client/server rolls are reversed)
My client code is like:
#define kMaxBacklog (5)
// fill out the sockadd_in for the server
struct sockaddr_in servAdddress;
//memcpy() to fill in the sockaddr
//setup the socket
int sockd, returnStatus;
sockd = socket(AF_INET, SOCK_STREAM, 0);
if (sockd == -1)
NSLog(#"could not create client socket");
else
NSLog(#"created client socket");
returnStatus = connect(sockd, (struct sockaddr*)&servAdddress, sizeof(servAdddress));
if (returnStatus == -1)
NSLog(#"could not connect to server - errno:%i", errno);
else
NSLog(#"connected to server");
NSData *dataWithHeader = [self getDataToSend];
returnStatus = send(sockd, [dataWithHeader bytes], [dataWithHeader length], 0);
if (returnStatus == -1)
NSLog(#"could not send file to server");
else if( returnStatus < [dataWithHeader length])
NSLog(#"ONLY PARTIAL FILE SENT");
else
NSLog(#"file sent of size: %i", returnStatus);
shutdown(sockd, SHUT_WR);
close(sockd);
The client method ALWAYS reports that it sent the entire file.
For the server:
#define MAXBUF (10000)
int _socket;
_socket = socket(AF_INET, SOCK_STREAM, 0); // set up the socket
struct sockaddr_in addr;
bzero(&addr, sizeof(addr));
addr.sin_len = sizeof(addr);
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = INADDR_ANY;
addr.sin_port = htons(0);
int retval = bind(_socket, (struct sockaddr *)&addr, sizeof(addr));
if (retval == -1)
NSLog(#"server could not bind to socket");
else
NSLog(#"server socket bound");
socklen_t len = sizeof(addr);
retval = getsockname(_socket, (struct sockaddr *)&addr, &len);
if (retval == -1)
NSLog(#"server could not get sock name");
else
NSLog(#"server socket name got");
int socket1, socket2, clientAddrLen, returnStatus;
struct sockaddr_in servAdddress, clientAddress;
clientAddrLen = sizeof(servAdddress);
socket1 = _socket;
returnStatus = listen(socket1, kMaxBacklog);
if (returnStatus == -1)
NSLog(#"server could not listen on socket");
else
NSLog(#"server socket listening");
while(1){
FILE *fd;
int i, readCounter;
char file[MAXBUF];
NSLog(#"server blocking on accept()");
socket2 = accept(socket1, (struct sockaddr*)&clientAddress, (socklen_t*)&clientAddrLen);
if (socket2 == -1)
NSLog(#"server could not accpet the connection");
else
NSLog(#"server connection accepted");
i = 0;
readCounter = recv(socket2, file, MAXBUF, 0);
if(!readCounter)
NSLog(#"server connection cancelled, readCount = 0");
else if (readCounter == -1){
NSLog(#"server could not read filename from socket");
close(socket2);
continue;
}
else
NSLog(#"server reading file of size: %i", readCounter);
fd = fopen([myfilePathObject cStringUsingEncoding:NSASCIIStringEncoding], "wb");
if(!fd){
NSLog(#"server could not open the file for creating");
close(socket2);
continue;
}
else
NSLog(#"server file open for creating");
returnStatus = fwrite([myData bytes], 1, [myData length], fd);
if (returnStatus == -1)
NSLog(#"Error writing data to server side file: %i", errno);
else
NSLog(#"file written to disk);
readCounter = 0;
//close (fd);
returnStatus = fclose(fd);
if(returnStatus)
NSLog(#"server error closing file");
So sporadically, the readCounter variable will not contain the same size as the file that was sent, but some times it does.
If it matters the file transfer is occurring between an iPhone and an iPhone simulator, both over WIFI. This happens regardless of if the phone is the server or if the simulator is the server.
If anyone can help me understand why this is occurring I'd appreciate it. I thought the whole purpose of TCP was to avoid this kind of problem.
(to give credit where it's due, for my server and client code I borrowed heavily from the book: The Definitive Guide to Linux Network Programming, by Davis, Turner and Yocom from Apress)
The recv function can receive as little as 1 byte, you may have to call it multiple times to get your entire payload. Because of this, you need to know how much data you're expecting. Although you can signal completion by closing the connection, that's not really a good idea.
Update:
I should also mention that the send function has the same conventions as recv: you have to call it in a loop because you cannot assume that it will send all your data. While it might always work in your development environment, that's the kind of assumption that will bite you later.
Tim Sylvester and gnibbler both have very good answers, but I think the most clear and complete is a combination of the two.
The recv() function return immediately with whatever is the in the buffer. This will be somewhere between 1 byte and MAXBUF. If the buffer is being written to while recv returns, you wont have the entire data that was sent in the buffer yet.
So you need to call recv() multiple times, and concatenate the data, to get everything that was sent.
A convenient way to do this (since we are working in cocoa) is to use NSMutableData like:
NSMutableData *fileData = [[NSMutableData alloc] init]; //Don't forget to release
while ((readCounter = recv(socket2, file, MAXBUF, 0)) > 0){
if (readCounter == -1){
NSLog(#"server could not read filename from socket");
close(socket2);
continue;
}
else{
NSLog(#"server reading file of size: %i", readCounter);
[fileData appendData:[NSData dataWithBytes:file length:readCounter]];
}
bzero(file, MAXBUF);
readCounter = 0;
}
You should probably have some kind of sequence of characters to signal termination of the file transfer, and only when you read those at the end of a block do you break out of your recv loop.
Of course, you will have to find a sequence that won't occur in your files, or that can be easily escaped. If you're working with text files this is pretty easy, but if not you'll have to be clever.
Alternatively, the client could first send the file size (in a separate send call), so the server knows how many bytes to expect in the file transfer.
recv returns right away with whatever is in the buffer (upto MAXBUF). If the buffer is being written to at the same time you might not get all the data
What TCP ensures is that your message will get to the remote peer correctly. As long as it fits in the send buffer, it will be automatically split into smaller chunks and sent by the local peer, and reordered and reassembled by the remote peer. It is not uncommon for a route to dynamically change while you are sending a message, which you would have to reorder manually (the smaller chunks) before delivering to your application.
As for your actual data transfer, your application needs to agree on a custom protocol. For instance, if you are only sending one message (the file), the sender could signal the receiver that it does not intend to write anymore to the socket (with shutdown(sock, SHUT_WR)), this way recv() returns with 0 and you know the transfer is complete (this is how a HTTP/1.0 server signals the client the transfer is complete). If you intend to send more data, then this alternative is not appropriate.
Another way would be to let the receiver know how much data the sender is going to transmit by including a header, for instance. It does not need to be overly elaborate, you could simply reserve the first 8 bytes to send the length as a 64-bit unsigned integer. In this case, you still need to be careful about byte ordering (big-endian / little-endian).
There is a very useful tutorial on network programming for UNIX environments:
Beej's Guide to Network Programming
You could refer to it to get a quick start, then refer back to the book for completeness, if you need. Even though you did not ask for additional references, TCP/IP Illustrated Vol. 1 and UNIX Network Programming Vol. 1 (both by W. Richard Stevens, the latter with a recent third edition) are excellent references.