I have a Symfony website that's something inbetween an actual implementation and staging (it's used by a special client of mine). The logging is kept ON on that server because that helps when things go wrong from time to time (and they often go wrong in non-obvious, non-error/exception ways). Most of the logged lines are by Doctrine - executed queries, which is very useful to me, but I do manually disable logging for SOME of the huge, repetitive and well-tested operations that spam hundreds of queries, to keep the log easier to navigate, if needed.
My question is: How do I disable logging done from inside of the Symfony messenger component? Specifically, logging done by the doctrine transport (which I use), which spams my log with following lines every second (multiplied by the number of supervisord processes that I run):
[2020-08-24 14:19:25] doctrine.DEBUG: "START TRANSACTION" [] []
[2020-08-24 14:19:25] doctrine.DEBUG: SELECT m.* FROM messenger_messages m WHERE (m.delivered_at is null OR m.delivered_at < ?) AND (m.available_at <= ?) AND (m.queue_name = ?) ORDER BY available_at ASC LIMIT 1 FOR UPDATE ["2020-08-24 13:19:25","2020-08-24 14:19:25","default"] []
[2020-08-24 14:19:25] doctrine.DEBUG: "COMMIT" [] []
Since these messages are generated by doctrine, I can't filter them out through logging channels - because that would disable ALL doctrine log messages, and that's not what I want. I also don't want to raise the logging level to something higher than DEBUG. I want to remove ONLY these specific messages from the log.
I just wanted to tackle this issue and I found that most of the spam can be avoided by using a fingers_crossed monolog handler instead of the default one. This quick solution avoid to fully disable Doctrine logs.
In packages/dev/monolog.yaml, replace
main:
type: stream
path: "%kernel.logs_dir%/%kernel.environment%.log"
level: debug
channels: ["!event"]
by :
main:
type: fingers_crossed
action_level: info
handler: nested
channels: ["!event"]
nested:
type: stream
path: "%kernel.logs_dir%/%kernel.environment%.log"
level: debug
channels: ["!event"]
You have many options.
Exclude messenger log info:
Disable Symfony Messenger log info
Log your message in a different file:
You can log Messages to different Files. Like it's described in this part of the Symfony documentation.
Configure the logger level:
You can configure your logger entry with level info, that can help you to have a log file corresponding to what you want to see. For example, you can use the ERROR level to just have errors in your Message.
Check the documentation below Loggin component.
Create your own channel:
You can create your own channel and subscribe to it, that excludes doctrine. Creating your own channel
Just search what you want with grep:
If you want to filter more what you want to see when you read the log file, you can use grep like:
tail path/to/my/logfile.txt | grep 'what I want to see'
That permits us to avoid useless lines. grep man page
The straight solution was when running the consumer use the flag --no-debug
php bin/console messenger:consume async_email_handler --no-debug
This will stop writing the doctrine lines in the .log file
Related
I am trying to align my logging with the best practice of using STDERR.
So, I would like to understand what happens with the logs sent to STDERR.
Symfony official docs (https://symfony.com/doc/current/logging.html):
In the prod environment, logs are written to STDERR PHP stream, which
works best in modern containerized applications deployed to servers
without disk write permissions.
If you prefer to store production logs in a file, set the path of your
log handler(s) to the path of the file to use (e.g. var/log/prod.log).
This time I want to follow the STDERR stream option.
When I was writing to a specific file, I knew exactly where to look for that file, open it and check the logged messages.
But with STDERR, I don't know where to look for my logs.
So, using monolog, I have the configuration:
monolog:
handlers:
main:
type: fingers_crossed
action_level: error
handler: nested
excluded_http_codes: [404, 405]
nested:
type: stream
path: "php://stderr"
level: debug
Suppose next morning I want to check the logs. Where would I look?
Several hours of reading docs later, my understanding is as follows:
First, the usage of STDERR over STDOUT is preferred for errors because it is not buffered (gathering all output waiting for the script to end), thus errors are thrown immediately to the STDERR stream. Also, this way the normal output doesn't get mixed with errors.
Secondly, the immediate intuitive usage is when running a shell script, because in the Terminal one will directly see the STDOUT and STDERR messages (by default, both streams output to the screen).
But then, the non-intuitive usage of STDERR is when logging a website/API. We want to log the errors, and we want to be able to monitor the already occurred errors, that is to come back later and check those errors. Traditional practice stores errors in custom defined log-files. More modern practice recommends sending errors to STDERR. Regarding Symfony, Fabien Potencier (the creator of Symfony), says:
in production, stderr is a better option, especially when using
Docker, SymfonyCloud, lambdas, ... So, I now recommend to use
php://stderr
(https://symfony.com/blog/logging-in-symfony-and-the-cloud).
And he further recommends using STDERR even for development.
Now, what I believe to be missing from the picture (at least for me, as non-expert), is the guidance on HOW to access and check the error logs. Okay, we send the errors to STDERR, and then? Where am I going to check the errors next morning? I get it that containerized platforms (clouds, docker etc) have specific tools to easily and nicely monitor logs (tools that intercept STDERR and parse the messages in order to organize them in specific files/DBs), but that's not the case on a simple server, be it a local server or on a hosting.
Therefore, my understanding is that sending errors to STDERR is a good standardization when:
Resorting to using a third-party tool for log monitoring (like ELK, Grail, Sentry, Rollbar etc.)
When knowing exactly where your web-server is storing the STDERR logs. For instance, if you try (I defined a new STD_ERR constant to avoid any pre-configs):
define('STD_ERR', fopen('php://stderr', 'wb'));
fputs(STD_ERR, "ABC error message.");
you can find the "ABC error message" at:
XAMPP Apache default (Windows):
..\xampp\apache\logs\error.log
Symfony5 server (Windows):
C:\Users\your_user\.symfony5\log\ [in the most recent folder, as the logs rotate]
Symfony server (Linux):
/home/your_user/.symfony/log/ [in the most recent folder, as the logs rotate]
For Symfony server, you can actually see the logs paths when starting the server, or by command "symfony server:log".
One immediate advantage is that these STDERR logs are stored outside of the app folders, and you do not need to maintain extra writable folders or deal with the permissions etc. Of course, when developing/hosting multiple sites/apps, you need to configure the error log (the STDERR storage) location per app (in Apache that would be inside each <VirtualHost> conf ; with Symfony server, I am not sure). Personally, without a third-party tool for monitoring logs, I would stick with custom defined log files (no STDERR), but feel free to contradict me.
I have been using fluentbit 1.6.10 for quite some time now and kept on adding OUTPUT splunk plugin one after another
Now it seems limit to add number of OUTPUT plugin is crossed because after addition of any new splunk plugin is causing inconsistent execution of fluent bit
Current:
Total number of OUTPUT plugins used: 37,
Name: kafka (4),
Name: null (1),
Name: splunk (32)
But now
Whenever a new splunk plugin is added then fluentbit execution behaviour is inconsistent.
Total number of OUTPUT plugins used: 37,
Name: kafka (4),
Name: null (1),
Name: splunk (33) this seems like creating problem
After adding 33rd OUTPUT plugin, one of the existing splunk output plugin stops working
But after manually editing config at runtime --> lets say adding a simple stdout --> restart pod then all splunk plugin starts working (even though this time stdout plugin doesn't work)
Working on getting the exact behaviour and will post steps to reproduce with config but same configuration works good in version 1.7.1
(I am aware that using these many plugins is not a good design and it will be addressed.)
Is there a known limit for number of plugins to be used in fluentbit 1.6.10 ?
There is no defined upper limit as such in fluent-bit for OUTPUT plugins to be added, but depending on resource consumption, every plugin has there own defined resource utilization configurations.
The resource can be managed through proper tagging and by maintaining caches.
In short, you need to test the configs depending upon the resource limit to set the limit in your environment.
I would like to manage users settings interfaced by GSettings in salt. I have a Python code that can manage the GSettings, but it needs to do that as a specific user.
Salt execution modules (and everything actually) by default run as a user that executed salt-minion, which is root by default. I couldn't find in the documentation information how to run a specific module as someone other.
I can walkaround it by executing a shell with su -l <username>, that in turn would call my Python code, but I hope there is an elegant build-in way, like an option user: <username> in the module.
There are at least two ways to run a command as a specified user:
From a state, you can do something like this (docs):
mycommand:
cmd.run:
- name: python my_gsettings_script.py
- runas: alternate_user
There is also an execution module, that also provides a runas option.
Assuming I have a running symfony application and it encounters an exception with following configuration:
framework:
profiler:
lifetime: 604800
only_exceptions: true
Then there should be a dump with profiling information.
But what happen next?
Just copy the file to your own local profiler data folder and start the profiler?
What are the best practices to handle and debug exceptions occuring on the production system?
I think enabling profiler even with only_exceptions: true should have performance impact, because to display something on exception is should be collected first in any way.
If you want to see the profiler data from another host, you can export and import it locally
For me more correct way is to just log events or email exception with stacktrace to admin via kernel exception listeners. Within listener you can access any info you need to send or log, i.e. request stack, logged in user info etc
How can I see all available events in Symfony2?
I found a command on google
php app\console container:debug --show-private
But it does not show all available events. Like event named "security.interactive_login" is not listed in it.
Is there a way to see available events?
Console command
You can run:
app/console debug:event-dispatcher
This will show you a detailed summary of every subscriber, in order of priority per event. Unfortunately this won't show you all possible events, since it's infeasible to query the container for any events that could be registered due to the inherently dynamic nature of the events system.
To understand events you'll need to refer to docs and code of each component and bundle.
Documentation is the best place to start
Symfony standard ships with a multitude of events. Each Symfony component and bundle may or may not define events — your best bet is to look at each component or bundle's documentation for references to events.
Some very common events can be found in the docs:
HTTP Kernel Events
Console Events
Form Events
Code analysis
I used PhpStorm to look for all subclasses of Symfony's base Event class (Symfony\Component\EventDispatcher\Event).
I generated an inheritance tree each child is a subclass of it's parent.
* note: prepend Symfony\Component\ to find the FQN
EventDispatcher\Event
EventDispatcher\GenericEvent
Console\Event\ConsoleEvent
Console\Event\ConsoleCommandEvent
Console\Event\ConsoleExceptionEvent
Console\Event\ConsoleTerminateEvent
Form\FormEvent
HttpKernel\Event\KernelEvent
HttpKernel\Event\FilterResponseEvent
HttpKernel\Event\FilterControllerEvent
HttpKernel\Event\FinishRequestEvent
HttpKernel\Event\GetResponseEvent
HttpKernel\Event\GetResponseForControllerResultEvent
HttpKernel\Event\GetResponseForExceptionEvent
HttpKernel\Event\PostResponseEvent
Security\Http\Event\SwitchUserEvent
Security\Http\Event\InteractiveLoginEvent
I make no claim that these are all public events you can/should hook into — this is just one way to programmatic examine 3rd party code and get a sense for potential idioms.
For instance I noticed that both the HttpKernel, Security, and Console components use namespaced constants to expose their keys, see:
Symfony\Component\HttpKernel\KernelEvents
Symfony\Component\Security\Http\SecurityEvents
Symfony\Component\Console\ConsoleEvents
The container:debug command shows all services that are registered to the dependency injection container. With the parameter show-private it will also show services that are flagged with public=false.
So as the most events might not be services the command you are using will not give you a list of available events. But to give you a possibility to search for available events you could try the following command:
php app/console container:debug --show-private | grep -i "listener"
As the most event handlers might have the word listener in their definitions you will find many of them. If you then want to get a more detailed information about the events which are handled by those listeners just call the command with specifying the service ID. For example if you are working with the FOSUserBundle this will give you a description for the interactive login listener:
php app/console container:debug fos_user.security.interactive_login_listener