Context
I'm trying to implement a sort-of orchestrator pattern for our applications.
Basically, we have three different and independent applications developped in Qt that communicate with each other using Web Socket. We'll call them "core", "business" and "ui". This is a flexibility aim as we can simply develop a new application in a more suiting technology and connect it to the others via the same communication protocole.
Now the idea is to have a simple launcher that allows us to specify which part to start. We launch this "orchestrator-like" application and it starts all required processes from a configuration file.
Everything is done in Qt currently (QML for the UI interfaces).
Initial Issue
I've made a custom class oriented towards reading the configuration file, preparing the processes, and starting them with their respective arguments.
This uses a std::map of QProcess related to their name in the configuration file and launch them using QProcess::start(<process_path>) method.
The catch is that everything went smoothly until recently. The sub processes are started and runs perfectly ; everything goes on as normal until we reach some point were the "ui" part crashes (usually an LLVM memory error or vector:: length error).
At first we thought about a memory leak or a code error but after much debugging we found that the application had no error whatsoever when we ran each part individually (without using the custom orchestrator class).
Question / Concerns
So, our question is: could it be that the QProcess:start() method actually shares the same stack with its parent? Three processes having the same parent, it would not be surprising than a vector of ~500 elements stored in each application can exceed the stack size when returned.
Information
We use MacOS Big Sur, IDE is Qt Creator, using Qt 5.15.0 and C++11.
Tried using valgrind but as read here and here, this seems a dead-end for now. The errors below were seen in the .crash file following the application exit.
libc++abi: terminating with uncaught exception of type std::length_error: vector
ui(2503,0x108215e00) malloc: can't allocate region :*** mach_vm_map(size=140280206704640, flags: 100) failed (error code=3) ui(2503,0x108215e00) malloc: *** set a breakpoint in malloc_error_break to debug LLVM ERROR: out of memory
Also tried to redirects or completly remove the application's output. First changing the setProcessChannelMode when starting the application, then with startDetached instead of start. Then, commented my Log method dumping log info into the corresponding Qt output (info/warning/critical/fatal/debug).
As suggested by #stanislav888, we could rewrite the application manager part in bash scripts and it would probably do the trick but I'd like to understand the root issue to avoid future mistakes.
It looks like a bad design. Application running and orchestrating through bash or PowerShell script looks much more better.
But anyway.
You could try to suppress orchestrated applications output and see what happen. The programs output might flood memory and make crash.
You must check what particular trouble cause the crash. Use memory "coredump" and system error messages to understand the all problem details.
I sure the community need that details. Because "оut of memory", "stack depth exceeded" and same errors make big difference.
Try to write bash or PowerShell script which does the same workflow as the Qt application. Hope it is not hard. But it will help you to figure out the issue.
At least you can run this script from the application.
Related
We all know that Core Dumps are an essential diagnostic tools for analysing various processes in Unix . I know both jstack and gcore are both used for generating Javacore files or Core Dumps but I have a doubt that Gcore is mainly used for Processes and Jstack is used for threads .
As from an Operating System perspective Process and Threads though interrelated (Process comprises of Threads only) they are relatively different from each other w.r.t memory/speed/execution . So is that gcore will diagnose the process and jstack will analyse the threads in that process ???
GCore act at OS level and you got a dump of native code that is currently running. From a java point of view, it is not really understandable.
JStack get you the stack trace at VM level (the java stack) of all thread your application have. You can find from that what is the real java code executed at a point.
Clearly, GCore is almost never used (too low level, native code...). Only really strange issue with native library or stuff like that will perhaps need this kind of tool.
There is also jmap that can generate a hprof file which is the heap data from you VM. A tool like 'Memory Analyser Tool' can open the hprof, and you can drill down on what was going on (on memory side).
If your VM crash because of a OutOfMemory, you can also set parameter to get the hprof when the event occurs. It helps to understand why (too many users, A DB query that fetch too much data...)
Last thing is the fact that you can add a debug option when your start your VM, so that you can connect to it, and put debug on running process. It can help if you have some strange issue that you are not able to reproduce in your local environment.
I use a third-party DLL (FTD2xx) to communicate with an external device. Using Qt4, in debug mode everything works fine, but the release crashes silently after successfully completing a called function. It seems to crash at return, but if I write something to the console (with qDebug) at the end of the function, sometimes it does not crash there, but a few, or few dozen lines later.
I suspect a not properly cleaned stack, what the debug build can survive, but the release chokes on it. Did someone encounter a similar problem? The DLL itself cannot be changed, as the source is not available.
It seems the reduction of the optimization level was the only way around. The DLL itself might have problems, as a program which does nothing but calls a single function from that DLL crashes the same way if optimization is turned on.
Fortunately, the size and speed lost by the change in optimization level is negligible.
Edit: for anyone with similar problems on Qt 5.0 or higher: If you change the optimization level (for example, to QMAKE_CXXFLAGS_RELEASE = -O0), it's usually not enough to just rebuild the application. A full "clean all" is required.
Be warned - the EPANET library is not thread safe, it contains a lot of global variables.
Are you calling two methods of that library from different threads?
I have compiled my freebsd libc source with -g option, so that now I can step in into libc functions.
But I am having trouble stepping into system calls code. I have compiled the freebsd kernel source code with -g. On setting the breakpoint, gdb informs about breakpoint on .S files. On hitting the breakpoint, gdb is unable to step into the syscall source code.
Also, I have tried: gdb$catch syscall open
but this is also not working.
Can you please suggest something?
Thanks.
You appear to have fundamental lack of understanding of how UNIX systems work.
Think about it. Suppose you were able to step into the kernel function that implements a system call, say sys_open. So now you are looking at the kernel source for sys_open in the debugger. The question is: is the kernel running at that point, or is it stopped. Since you will want to do something like next in the debugger, let's assume the kernel is stopped.
So now you press the n key, and what happens?
Normally, the kernel will react to an interrupt raised by the keyboard, figure out which key was pressed, and send that key to the right process (the one that is blocked in read(2) from the terminal that has control of the keyboard).
But your kernel is stopped, so no key press for you.
Conclusion: debugging the kernel via debugger that is running on that same machine is impossible.
In fact, when people debug the kernel, they usually do it by running debugger on another machine (this is called remote debugging).
If you really want to step into kernel, the easiest way to do that is with UML.
After you've played with UML and understand how the userspace/kernel interface works and interacts, you can try kgdb, though the setup is usually a bit more complicated. You don't actually have to have a separate machine for this, you could use VMWare or VirtualPC, or VirtualBox.
As Employed Russian already stated, gdb being in userland cannot inspect anything running in the kernel.
However, nothing prevents to implement a debugger in the kernel itself. In such case, it is possible to set breakpoints and run kernel code step by step from a local debugging session (console). With FreeBSD, such a debugger is available as ddb.
Some limitations would be the lack of connection between your gdb and ddb sessions and I'm unsure source level debugging (-g) is available for kernel code under FreeBSD/ddb.
An alternate and much less intrusive way to 'debug' the kernel from userland would be to use dtrace.
I have a host application and I have written a plug-in. I compile my plug-in down to a shared object (say foo.so), and the host application will load it via dlopen. In this case, my host application is the opt tool from llvm (though I don't think that's important to the question).
I'd like to compile my plug-in with profiling enabled (i.e. g++ -pg, gprof). However, when I do this, the profile output file gmon.out is never created. Maybe gprof expects someone to call a setup routine, or something like that.
For various reasons, I would like to avoid recompiling the host app with -pg. I am curious if it is possible to profile a shared object foo.so without profiling the host application opt.
I've also looked into other profiling tools; qprof from HP should be able to handle this situation, but it is unable to resolve the names of functions in the shared object (it falls back to addr2line in a very naive way).
Thanks,
Nick
I assume the reason you want to do this is to find ways to optimize the plugin (as opposed to just getting timing information).
Can you run the host app under a debugger or IDE? Does the IDE have a pause button, or can you interrupt it with Ctrl-C or some such key?
Then you can quickly find the costly code by using this technique. Only take samples when your plugin is running, or if you can't do that, just ignore samples that don't end in your plugin.
Even if you get gprof to work, or a similar profiler, you're likely to be disappointed.
I've written a c# windows app, that performs some DB intensive operations. (MySQL connector v6).
When running the project in Debug mode, everything works fine. However, when I run the prject in release mode, it sometimes quits operation midway - with no error message, nothing in the event logs etc.
What would be the best way to debug release mode - when everything works in debug mode?
Thanks for any help,
Bob
You can create a log file and have the application write lines to it with information of your choice, similarly to how the console may be used for debug purposes in a windows form application. You can write values of certain variables to this file, or even just write distinct phrases in select places of the code that will help you detect where the program is in execution when it fails.
Bobby is correct in asking about Application Event Log. If it is bombing on a .NET error, it will likely be logged.
If that doesn't give you anything, wrap the entire app in a try/catch block. On your exception handling, log the error (application log, file, etc...). Make sure when you log it to capture the call stack.
I've got exactly the same problem - application running in debug mode and fail in release. Try the following:
Wrap everything in Program.cs in try{}..catch{} block and it will show a reason
I don't know why but my application failed on Program.SetCompatibleTextRenderingDefault() function with message that it should be called before any instances on IWin32Window or something like that
It is very strange errors for me cuz i didn't have any code before this function. But you can try - maybe it will show something useful for you