Is there a limit for "max number of OUTPUT plugin to be used in fluentbit 1.6.10" - fluent-bit

I have been using fluentbit 1.6.10 for quite some time now and kept on adding OUTPUT splunk plugin one after another
Now it seems limit to add number of OUTPUT plugin is crossed because after addition of any new splunk plugin is causing inconsistent execution of fluent bit
Current:
Total number of OUTPUT plugins used: 37,
Name: kafka (4),
Name: null (1),
Name: splunk (32)
But now
Whenever a new splunk plugin is added then fluentbit execution behaviour is inconsistent.
Total number of OUTPUT plugins used: 37,
Name: kafka (4),
Name: null (1),
Name: splunk (33) this seems like creating problem
After adding 33rd OUTPUT plugin, one of the existing splunk output plugin stops working
But after manually editing config at runtime --> lets say adding a simple stdout --> restart pod then all splunk plugin starts working (even though this time stdout plugin doesn't work)
Working on getting the exact behaviour and will post steps to reproduce with config but same configuration works good in version 1.7.1
(I am aware that using these many plugins is not a good design and it will be addressed.)
Is there a known limit for number of plugins to be used in fluentbit 1.6.10 ?

There is no defined upper limit as such in fluent-bit for OUTPUT plugins to be added, but depending on resource consumption, every plugin has there own defined resource utilization configurations.
The resource can be managed through proper tagging and by maintaining caches.
In short, you need to test the configs depending upon the resource limit to set the limit in your environment.

Related

Why isn't Carbon writing Whisper data points as per updated storage-schema retention?

My original carbon storage-schema config was set to 10s:1w, 60s:1y and was working fine for months. I've recently updated it to 1s:7d, 10s:30d, 60s,1y. I've resized all my whisper files to reflect the new retention schema using the following bit of bash:
collectd_dir="/opt/graphite/storage/whisper/collectd/"
retention="1s:7d 1m:30d 15m:1y"
find $collectd_dir -type f -name '*.wsp' | parallel whisper-resize.py \
--nobackup {} $retention \;
I've confirmed that they've been updated using whisper-info.py with the correct retention and data points. I've also confirmed that the storage-schema is valid using a storage-schema validation script.
The carbon-cache{1..8}, carbon-relay, carbon-aggregator, and collectd services have been stopped before the whisper resizing, then started once the resizing was complete.
However, when checking in on a Grafana dashboard, I'm seeing empty graphs with correct data points (per sec, but no data) on collectd plugin charts; but with the graphs that are providing data, it's showing data and data points every 10s (old retention), instead of 1s.
The /var/log/carbon/console.log is looking good, and the collectd whisper files all have carbon user access, so no permission denied issues when writing.
When running an ngrep on port 2003 on the graphite host, I'm seeing connections to the relay, along with metrics being sent. Those metrics are then getting relayed to a pool of 8 caches to their pickle port.
Has anyone else experienced similar issues, or can possibly help me diagnose the issue further? Have I missed something here?
So it took me a little while to figure this out. It had nothing to do with the local_settings.py file like some of the old responses, but it had to do with the Interval function in the collectd.conf.
A lot of the older responses mentioned that you needed to include 'Interval 1' inside each Plugin container. I think this would have been great due to the control of each metric. However, that would create config errors in my logs, and break the metric. Setting 'Interval 1' at top level of the config resolved my issues.

How do you make munin stop sending alerts for smart exit statuses?

Since I installed munin and enabled alerts, it has been intermittently sending me this for four of my hard drives:
WARNINGs: smartctl_exit_status is 4.00 (outside range [:1]).
The munin documentation says:
The ignoreexit parameter can be useful to exclude some bits in smartctl exit code, which is a bit mask described in its main page, from consideration.
So I added the following to /etc/munin/plugin-conf.d/munin-node:
[smart_sdg;smart_sdh;smart_sdi;smart_sdj]
env.ignoreexit 4
(Note that this corresponds with the four drives sending the alerts.)
Alas, the alerts keep coming. I can't make them stop and I don't understand why. Is my config location wrong? Am I doing the configuration wrong? Why isn't this working? Any help?
It turns out that I had an old version of Munin and it doesn't support the env.ignoreexit parameters. I grabbed the latest version of the plugin from Munin's repository and it seems to work.
This was a fun exploration. I wish Munin threw errors when you gave it configurations it didn't recognize, but alas.

Why Symfony3 so slow?

I installed Symfony3 framework-standard-edition. I'm trying to open the home page(app.php prod) and it is loaded 300-400ms.
This is my profiler information:
also I use php7.
Why it is so long?
You can try to optimize Zend OPCache.
Here are some recommended settings
opcache.revalidate_freq
Basically put, how often (in seconds) should the code cache expire and check if your code has changed. 0 means it checks your PHP code every single request (which adds lots of stat syscalls). Set it to 0 in your development environment. Production doesn't matter because of the next setting.
opcache.validate_timestamps
When this is enabled, PHP will check the file timestamp per your opcache.revalidate_freq value.
When it's disabled, opcache.revaliate_freq is ignored and PHP files are NEVER checked for updated code. So, if you modify your code, the changes won't actually run until you restart or reload PHP (you force a reload with kill -SIGUSR2).
Yes, this is a pain in the ass, but you should use it. Why? While you're updating or deploying code, new code files can get mixed with old ones— the results are unknown. It's unsafe as hell
opcache.max_accelerated_files
Controls how many PHP files, at most, can be held in memory at once. It's important that your project has LESS FILES than whatever you set this at. For a codebase at ~6000 files, I use the prime number 8000 for maxacceleratedfiles.
You can run find . -type f -print | grep php | wc -l to quickly calculate the number of files in your codebase.
opcache.memory_consumption
The default is 64MB. You can use the function opcachegetstatus() to tell how much memory opcache is consuming and if you need to increase the amount.
opcache.interned_strings_buffer
A pretty neat setting with like 0 documentation. PHP uses a technique called string interning to improve performance— so, for example, if you have the string "foobar" 1000 times in your code, internally PHP will store 1 immutable variable for this string and just use a pointer to it for the other 999 times you use it. Cool.
This setting takes it to the next level— instead of having a pool of these immutable string for each SINGLE php-fpm process, this setting shares it across ALL of your php-fpm processes. It saves memory and improves performance, especially in big applications.
The value is set in megabytes, so set it to "16" for 16MB. The default is low, 4MB.
opcache.fast_shutdown
Another interesting setting with no useful documentation. "Allows for faster shutdown".
Oh okay. Like that helps me. What this actually does is provide a faster mechanism for calling the destructors in your code at the end of a single request to speed up the response and recycle php workers so they're ready for the next incoming request faster.
Set it to 1 and turn it on.
opcache=1
opcache.memory_consumption=256
opcache.interned_strings_buffer=16
opcache.max_accelerated_files=8000
opcache.validate_timestamps=0
opcache.revalidate_freq=0
opcache.fast_shutdown=1
I hope it will help improve your performances
[EDIT]
You might also want to look at this answer:
Are Doctrine relations affecting application performance?
TheMrbikus, try some optimization with the following elements:
Use APC
Use Bootstrap files
Reference: http://symfony.com/doc/current/performance.html
Use the OPCache PHP7
Use Apache PHP-FPM.
E-mail sending process, and may slow down during the form rendering operations. Create a blank test Controller.

Simple todo tutorial memory increased by no activity

I have a Meteor app that is based on the Meteor simple todos tutorial. I noticed, that the second node-process (smaller one) on the server, started later when the app is started up, increases its memory usage steadily, even when there is no activity at all but at least one client is logged on to the server. I trace the RSS-Memory value with
pmap -x $PID_node | tail -1 | awk '{ print $4 }'
but it is also visible with 'top'. The RSS number is increasing steadily when doing nothing at all on the client side. I added the app here for reproducing the behavior.
When I remember correctly it is the original state as I finished the tutorial without doing any custom changes.
When the client is disconnected the memory still remains and is not reduced by any means.This goes on until the app is closed on the server when reaching the memory limit. I encounter similar behavior with the official todos-example. I start the apps with
meteor --port 61100
I found in other posts that this could be related to a public folder which I even do not have in the simple todos so there must be something different. I also updated to the latest version of meteor 1.3.4.1 which does not change the behavior.
Is this a normal meteor behavior or has to be considered a meteor bug? Or is there any bad style code in the examples?

New index in Kibana with +18000 fields causes problems

We have been running ElasticSearch v. 1.6.0 and Kibana v. 4.1.0 with NGINX as proxy in production for about two months now and the set-up handles logging of multiple applications.
We have 830.000 documents, total of 450MB and a single node. All log statements get shipped to ElasticSearch using Serilog, which creates an index per day.
A week ago Kibana started being extremely slow. I think that I've finally figured out the reason; upon creating the index in Kibana, we did not check the box "Use event times to create index names", thus I guess that even though we only query log statements which has occurred within the last 15 minutes, then Kibana actually asks ElasticSearch to look in all indices.
Thus, I created a new index in Kibana with the format [serilog-]YYYY.MM.DD and Kibana acknowledged that the pattern followed all the existing indices.
However, it found more than 18000 fields in the indices. And many of the fields looks like
fields.modelState.Value.Value.Culture.Parent.Parent.Parent.Parent.Parent.Parent.Parent.Parent.Parent.Parent.Parent.Parent.Parent.Parent.Parent.Parent.IsReadOnlye
(where IsReadOnly varies from field to field, and likewise do the number of nested Parent).
It seems like this is causing problems, because when I go to the Discover tab, then I get a JavaScript error in the Console:
Notice how indexPattern.fields is undefined. And after this error, I receive a 413:
And Kibana looks like
These fields weren't in our old index (which had the format serilog-*). I guess it's because the fields first appeared in log statements after our initial deployment to production.
Update
curl -XGET 'localhost:9200/.kibana/index-pattern/_search' yields
However, under the Settings tab I can see
I do not need all those fields with pattern
fields.modelState.Value.Value.Culture.Parent.

Resources