Are there any limitations to starting multiple grpc::ServerBuilders in an application? - grpc

Question

My C++ application running on Ubuntu 14.04 is having problems. I am using grpc to communicate with a go webserver application which is servering up webpages with status/configuration of the c++ application.
I have been using 1 year old version of grpc 0.14 something so before posting here, I upgraded everything (grpc 1.3.1, go version 1.8.1).
It seems my c++ application is crashing quite often with the 1.3.1(and with 1.0.0, 1.2.5, 1.2.0, etc...) grpc version.
I am getting a sigabort with a double free warning. The application will run for awhile but after a period of time in which the web application is requesting data from the c++ application, it will crash: gdb output:
[New LWP 9908]
[New LWP 9881]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./bhio'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 _int_malloc (av=0x7fb3ac000020, bytes=16) at malloc.c:3351
3351 malloc.c: No such file or directory.
(gdb) where
#0 _int_malloc (av=0x7fb3ac000020, bytes=16) at malloc.c:3351
#1 0x00007fb4205db6c0 in __GI___libc_malloc (bytes=16) at malloc.c:2891
#2 0x000000000076bb2f in gpr_malloc ()
#3 0x000000000077678d in grpc_error_create ()
#4 0x000000000078ba94 in ?? ()
#5 0x000000000078dbee in grpc_chttp2_fail_pending_writes ()
#6 0x000000000078e19f in grpc_chttp2_mark_stream_closed ()
#7 0x000000000078e2eb in grpc_chttp2_cancel_stream ()
#8 0x000000000078ef1c in ?? ()
#9 0x000000000077597e in grpc_combiner_continue_exec_ctx ()
#10 0x0000000000777678 in grpc_exec_ctx_flush ()
#11 0x000000000078095f in grpc_call_cancel_with_status ()
#12 0x0000000000780be1 in grpc_call_destroy ()
#13 0x0000000000769bd7 in grpc::ServerContext::~ServerContext() ()
#14 0x0000000000768c7c in grpc::Server::SyncRequest::CallData::~CallData() ()
#15 0x00000000007691e3 in
grpc::Server::SyncRequestThreadManager::DoWork(void*, bool) ()
#16 0x000000000076aff1 in grpc::ThreadManager::MainWorkLoop() ()
#17 0x000000000076b04c in grpc::ThreadManager::WorkerThread::Run() ()
#18 0x00007fb420eebbf0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x00007fb421146184 in start_thread (arg=0x7fb3ca686700)
at pthread_create.c:312
#20 0x00007fb42065337d in clone ()
---Type <return> to continue, or q <return> to quit---
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)
(gdb) quit
or here:
[Thread 0x7fff7d7fa700 (LWP 3521) exited]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff7e7fc700 (LWP 3524)]
__GI___libc_free (mem=0xb5) at malloc.c:2929
2929 malloc.c: No such file or directory.
(gdb)
(gdb) where
#0 __GI___libc_free (mem=0xb5) at malloc.c:2929
#1 0x000000000077b7b5 in grpc_byte_buffer_destroy ()
#2 0x0000000000773ac3 in grpc::Server::SyncRequest::CallData::~CallData() ()
#3 0x000000000077405a in grpc::Server::SyncRequestThreadManager::DoWork(void*, bool) ()
#4 0x0000000000776111 in grpc::ThreadManager::MainWorkLoop() ()
#5 0x000000000077616c in grpc::ThreadManager::WorkerThread::Run() ()
#6 0x00007ffff6c9fbf0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007ffff6efa184 in start_thread (arg=0x7fff7e7fc700)
at pthread_create.c:312
#8 0x00007ffff640737d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
So if it is ok to start multiple serverbuilder s , what else could the above error point to that I could possibly be doing wrong in using the grpc library.. I am taking this code over for someone else who wrote it, so my knowledge is lacking in the use of grpc... I don't think grpc is that unstable, so it must be something I am doing by using it incorrectly.
any ideas would be helpful..
any suggestions to debug it better would be helpful.
for grpc building, I am just doing the following:
$ git clone -b $(curl -L http://grpc.io/release) https://github.com/grpc/grpc
$ cd grpc
$ git submodule update --init
$ make
$ [sudo] make install
Is there options to compile differently which might provide more information?
thanks in advance for the help/suggestions.
Bob

Answer 1

It turns out, a small memory leak (missing close of a socket file descriptor) in the registered service function was causing the issue.