I was trying to fetch the resource and usage of resources of an instance using ceilometer api. I have used v2/meters/instance , v2/meters/cpu_util and v2/meters/memory. The result these api's return is too large and I'm not able to identify the paramater that needs to be taken to find the resource usage. I need to find the cpu utilization, bandwidth and memory usage of an instance using the ceilometer api. Can anyone please tell me which api I need to use to get the cpu utilization, bandwidth and memory usage of an instance and the parameter that needs to be taken to get the usage.
Thanks for any help in advance.
Regards ,
Lokesh.S
If you use the CLI, you can limit the number of samples with the -l/--limit parameter like in the example below:
`ceilometer sample-list -m cpu_util -l 10`
ceilometer --debug sample-list -m cpu_util -l 1 -q resource={your_vm_id}
Note that
--debug enables you to observe what rest API has been requested, you can learn example from it and write your own rest request, or just use CLI if you can. And this option will show rest response with full detailed sample information, CLI will format it and some information may be dropped.
-l 1 means just limit to return one result, so you will not flushed by huge mount of data
-q resource={your_vm_id} means only get cpu_util sample for that vm
you can read this official document of http://docs.openstack.org/developer/ceilometer/webapi/v2.html, or read my post http://zqfan.github.io/assets/doc/ceilometer-havana-api-v2.html (which is written in Chinese)
Related
The Problem
I have a Fluent Bit service (running in a docker container) that needs to tail log files (mounted from the host into the container) and then forward those logs to Elasticsearch. For this PoC I create a new log file every minute (eg. spring-boot-logger-2021-05-25_10_53.0.log, spring-boot-logger-2021-05-25_10_54.0.log etc)
I can see that Fluent Bit picks up all the files, but it only reads and forwards the first few lines of a file (Each log entry is a single line and formated in JSON). Only when the Fluent Bit container is restarted does it read and forward the rest of the files.
To demonstrate this issue, I have a script that generates 200 log entries over a period of 100 seconds (ie. 2 logs per second). After running this script, I get a small number of entries in Elastic as shown
in this image. Here one can see that there are only 72 entries with large gaps between the entries.
Once I restart the Fluent Bit container it processes the rest of the files and fill in all the logs as show
in this image.
Here is my Fluent Bit config file:
[SERVICE]
Flush 5
Daemon off
Log_Level debug
Parsers_File /fluent-bit/etc/parsers.conf
[INPUT]
Name tail
Parser docker
Path /var/log/serviceA/*.log
Tag service.A
DB /var/db/ServiceA
Refresh_Interval 30
[INPUT]
Name tail
Parser docker
Path /var/log/serviceB/*.log
Tag service.B
DB /var/db/ServiceB
Refresh_Interval 30
[OUTPUT]
Name stdout
Match service.*
[OUTPUT]
Name es
Host es01
Port 9200
Logstash_Format On
tls Off
Match service.*
What I've tried
I've tried the following:
Increased the flush rate to 1s
Shortened the refresh_interval to 10s
Decreased the Buffer_Chunk_Size & Buffer_Max_Size to 1k with the hope that it will force Fluent Bit to flush the logs more often.
Increased Buffer_Chunk_Size & Buffer_Max_Size to 1M as I've read an article stating that Fluent Bit's "pause" callback does not work as expected.
Explicitly configured a Mem_Buf_Limit of 5M.
Tried Fluent Bit version 1.7, 1.6 and 1.5
I've also used the debug versions of these containers to confirm that the files mounted correctly into the container and that they reflect all the logs (when Fluent Bit does not pick it up)
Fluent Bit's log level has also been set to debug, but there are no hints or errors in the logs.
Has anybody else experienced this issue?
I am not sure but I think Fluentbit (1.7 and 1.8) has bug(s) to access on shared logs in PV. It has rights, sees files but not fetches the log lines after its first fetch.
I found the solution by placed the Fluentbit as a sidecar, not a seperated pod.
I have had the same issue with fluent-bit on Openshift using glusterfs for persistent volumes.
My workaround has been to fork the official repo and build a new fluent-bit Docker image after making a small addition to the Dockerfile:
RUN cmake ... \
... \
-DFLB_INOTIFY=Off \
..
However, in the meantime, I see that there is now a configuration parameter called Inotify_Watcher in the tail input documentation, which I guess can be used for exactly this.
There's a simple game that my friends and I play both in person and and online. I developed a CLI that records our in-person games (I just type in each move), but I now want to use it to record our online games. All I need to do is pipe the HTTP response bodies being sent to my browser (Firefox) to my CLI. Unfortunately. I can't figure out how to do this.
Ideally, I'm looking for a Ubuntu package that I can run from the command line that will capture and return all HTTP response bodies from a specific endpoint. I've looked into tcpdump and some simple proxy servers, but I'm not sure they do what I want them to do.
Thanks for your help! Let me know if I need to provide any further information!
I used MITMProxy as ZachChilders recommended in the comments. I found it somewhat difficult to get set up, so I'll include what directions I followed to get it up and running:
1) Install MITMProxy.
2) Configure Firefox.
3) Create Add On to parse body.
4) Stream data via Python to CLI (TODO).
I'm currently using Xquery queries (launch via the API) to extract from Marklogic 8.0.6/
Query in my file extract_data.xqy:
xdmp:save("toto.csv",let $nl := "
"
return
document {
for $data in collection("http://book/polar")
return ($data)
})
API call :
$curl --anyauth --user ${MARKLOGIC_USERNAME}:${MARKLOGIC_PASSWORD} -X POST -i -d #extract_data.xqy \
-H "Content-type: application/x-www-form-urlencoded" \
-H "Accept: multipart/mixed; boundary=BOUNDARY" \
$node:$port/v1/eval?database=$db_name
It works fine but I'd like to schedule this extract directly in marklogic and have it running in background to avoid timeout if the request takes too much time to be executed.
Is-there a feature like that to do that ?
Regards,
Romain.
You can use the task scheduler to setup recurring script execution.
The timeout can be adjusted in the script with xdmp:set-request-time-limit
I would suggest you take a look at MLCP as well.
As suggested by Mads a tool like CORB can help pull csv data out of MarkLogic.
A schedule as suggested by Michael can trigger a periodic export, and save the output to disk, or push it elsewhere via HTTP. I'd look into running incremental exports in that case though, and I'd also suggest batching things up. In a large cluster, I'd even suggest chunking the export into batches per forest or per host on which content forests are attached. Scheduled tasks allow targeting specific hosts on which they should run.
You can also run an adhoc export, particularly if you batch up the work using a tool like taskbot. And if you combine it with its OPTIONS-SYNC-UPDATE mode, you can merge multiple batches back into one result file before emitting it as well, and get better performance out of it, compared to running it single threaded. Merging results doesn't scale endlessly, but if you have a relatively small dataset (only a few million small records maybe), that might be sufficient.
HTH!
Is there anything out there to monitor SaltStack installations besides halite? I have it installed but it's not really what we are looking for.
It would be nice if we could have a web gui or even a daily email that showed the status of all the minions. I'm pretty handy with scripting but I don't know what to script.
Anybody have any ideas?
In case by monitoring you mean operating salt, you can try one of the following:
SaltStack Enterprise GUI
Foreman
SaltPad
Molten
Halite (DEPRECATED by SaltStack)
These GUI will allow you more than just knowing whether or not minions are alive. They will allow you to operate on them in the same manner you could with the salt client.
And in case by monitoring you mean just whether the salt master and salt minions are up and running, you can use a general-purpose monitoring solutions like:
Icinga
Naemon
Nagios
Shinken
Sensu
In fact, these tools can monitor different services on the hosts they know about. The host can be any machine that has an ip address and the service can be any resource that can be queried via the underlying OS. Example of host can be a server, router, printer... And example of service can be memory, disk, a process, ...
Not an absolute answer, but we're developing saltpad, which is a replacement and improvement of halite. One of its feature is to display the status of all your minions. You can give it a try: Saltpad Project page on Github
You might look into consul while it isn't specifically for SaltStack, I use it to monitor that salt-master and salt-minion are running on the hosts they should be.
Another simple test would be to run something like:
salt --output=json '*' test.ping
And compare between different runs. It's not amazing monitoring, but at least shows your minions are up and communicating with your master.
Another option might be to use the salt.runners.manage functions, which comes with a status function.
In order to print the status of all known salt minions you can run this on your salt master:
salt-run manage.status
salt-run manage.status tgt="webservers" expr_form="nodegroup"
I had to write my own. To my knowledge, there is nothing out there which will do this, and halite didn't work for what I needed.
If you know Python, it's fairly easy to write an application to monitor salt. For example, my app had a thread which refreshed the list of hosts from the salt keys from time to time, and a few threads that ran various commands against that list to verify they were up. The monitor threads updated a dictionary with a timestamp and success/fail for each host after they ran. It had a hacked together HTML display color coded to reflect the status of each node. Took me a about half a day to write it.
If you don't want to use Python, you could, painfully, do something similar to this inefficient, quick, untested hack using command line tools in bash.
minion_list=$(salt-key --out=txt|grep '^minions_pre:.*'|tr ',' ' ') # You'
for minion in ${minion_list}; do
salt "${minion}" test.ping
if [ $? -ne 0 ]; then
echo "${minion} is down."
fi
done
It would be easy enough to modify to write file or send an alert.
halite was depreciated in favour of paid ui version, sad, but true - still saltstack does the job. I'd just guess your best monitoring will be the one you can write yourself, happily there's a salt-api project (which I believe was part of halite, not sure about this), I'd recommend you to use this one with tornado as it's better than cherry version.
So if you want nice interface you might want to work with api once you set it up... when setting up tornado make sure you're ok with authentication (i had some trouble in here), here's how you can check it:
Using Postman/Curl/whatever:
check if api is alive:
- no post data (just see if api is alive)
- get request http://masterip:8000/
login (you'll need to take token returned from here to do most operations):
- post to http://masterip:8000/login
- (x-www-form-urlencoded data in postman), raw:
username:yourUsername
password:yourPassword
eauth:pam
im using pam so I have a user with yourUsername and yourPassword added on my master server (as a regular user, that's how pam's working)
get minions, http://masterip:8000/minions (you'll need to post token from login operation),
get all jobs, http://masterip:8000/jobs (you'll n need to post token from login operation),
So basically if you want to do anything with saltstack monitoring just play with that salt-api & get what you want, saltstack has output formatters so you could get all data even as a json (if your frontend is javascript like) - it lets you run cmd's or whatever you want and the monitoring is left to you (unless you switch from the community to pro versions) or unless you want to use mentioned saltpad (which, sorry guys, have been last updated a year ago according to repo).
btw. you might need to change that 8000 port to something else depending on version of saltstack/tornado/config.
Basically if you want to have an output where you can check the status of all the minions then you can run a command like
salt '*' test.ping
salt --output=json '*' test.ping #To get output in Json Format
salt manage.up # Returns all minions status
Or else if you want to visualize the same with a Dashboard then you can see some of the available options like Foreman, SaltPad etc.
Is there a recommended/best way to generate a report for time spent on each policy for an API proxy?
Currently my approach is to use JS to collect the timestamps and calculate the delay around each policy, and then report it using the stats collection policy.
That's too invasive for performance checks and my data collection alone adds time to the overall response.
What would be the best no invasive way to report on the time taken for each step when analyzing the data across many requests (the ui, in the trace mode does show the time for each policy on an individual request basis)
Thanks,
Ricardo
There's not a public API supported to calculate this information and return a nice, clean response of aggregated policy execution time data. Best bet is to try using Analytics reports with request_processing_latency and response_processing_latency measures. (http://apigee.com/docs/content/analytics-reference). Then, if needed, utilize trace to identify policy execution times.
Alternatively, you can try downloading the trace session and parsing the timestamps between policies to build your information, but trace in UI does this already..
You consider using debug API. http://apigee.com/docs/api/debug-sessions
First you'll need to start a session for example:
curl -H "Content-type:application/octet-stream" -X POST https://api.enterprise.apigee.com/v1/organizations/{org}/environments/{env}/apis/{api_name}/revisions/{revision #}/debugsessions?"session=MySession" \
-u $ae_username:$ae_password
Get info from session:
curl -X GET -H "Accept:application/json" \
https://api.enterprise.apigee.com/v1/organizations/{org}/environments/{env}/apis/{api_name}/revisions/{revision #}/debugsessions/MySession/data \
-u $ae_username:$ae_password
The time spent for each policy can be found using the debug trace in UI.
Please see the below screenshot for the same.
Also as Diego said you can use debugsession API call to get a debug session.
For the debug session you can also define the time limit as to how much time you want to debug session to run. With this if you are running your performance test for 1 hour you can create debug session for that much amount of time.
curl -v -u jhans#apigee.com http://management:8080/v1/organizations/weatherapi/environments/prod/apis/ForeCast/revisions/6/debugsessions?session=ab\&timeout=300 -X POST
From the UI you can download the trace session which would contain an XML having the timestamp for each policy
<Point id="Condition">
<DebugInfo>
<Timestamp>05-02-14 04:38:14:088</Timestamp>
<Properties>
<Property name="ExpressionResult">true</Property>
</Point>
<Point id="StateChange">
The above is an example for getting timestamps for any policy in debug trace from UI.
Ricardo,
here is what I suggest.
Disclaimer: It is very meticulous and time consuming. I would recommend this approach only when you are really blocked on a performance issue and there is no other solution.
Let us say your proxy has few policies a service callout to external service and a backend.
So the total latency would be Sum of time taken by (p1, p2, p3...) + service callout target + time taken by your backend.
Very 1st step would be to stub out the external dependencies. You can use a null target (a stub proxy on Apigee edge without any logic) to do so.
Now disable all other policies ( enable = false on the policy schema ). Conduct a load test and benchmark your proxy performance against stubbed endpoints. Also at this time no policies are active.
Start activating the policies one by one or a few at a time, and re-run the load test each time.
Finally you can run the load test against real backends (removing the stubs)
At the end of this series of load test you will know which policy, backend is making the most significant performance impact.