As I know kibana and opentsdb both can be used in conjunction with Elastic logstash kibana (ELK) stack.
OpenTSDB offers a built-in, simple user interface for selecting one or more metrics and tags to generate a graph as an image. Alternatively an HTTP API is available to tie OpenTSDB into external systems such as monitoring frameworks, dashboards, statistics packages or automation tools. Each (Time series daemon)TSD uses the open source database HBase to store and retrieve time-series data.
Kibana can also be used for plotting metrics with access logs and custom logs, how does opentsdb helps in the system?
OpenTSDB is a time-series database. You could use OpenTSDB alongside Elasticsearch, for the metrics component. You would use Elasticsearch for text (logs perhaps) and OpenTSDB for metrics.
If you are using Logstash, there is an output that can export metrics from logs into OpenTSDB.
Kibana is a visualization tool for Elasticsearch queries. It assists you with searching and viewing the data that is stored in Elasticsearch. It is only used with Elasticsearch.
If you would like a unified front-end for Elasticsearch and OpenTSDB, you could consider Grafana, which has support for both Elasticsearch and OpenTSDB, but less functionality than Kibana in regard to Elasticsearch.
Related
tl;dr: I want to reference an external data source from a Kusto query in Application Insights.
My application is writing logs to Application Insights, and we're querying it using Kusto in the Azure portal. To give an example of what I'm trying to do:
We're currently looking at these logs to find an action that triggers when a visitor viewed a blog post on our site. This is working well on a per blog-post level, but now we want to group this data by the category these blog posts are in, or by the tags they have, but that's not information I have within the logs.
The information we log contains unique info about that blog post (unique url, our internal id, etc) that I could use to look up this information in another data source (e.g. our SQL DB where this relation is stored), but I have no idea if/how this is possible. So that's the question, is this possible? Can I query a SQL DB, or get data in JSON via a URL or something?
Alternative solutions would be to move the reporting elsewhere (e.g. PowerBI) and just use AI as a data source, or to actually log all the category/tag info, but I really don't want to go down that route.
Kusto supports accessing external data (blobs, Azure SQL, Cosmos DB), however
Application Insights / Azure Monitor and other multi-tenant services are blocking this functionality due to security and resource governance concerns.
You could try setting-up your own Azure Data Explorer (Kusto) cluster, where this functionality will be available, and then access your Application Insights data using cross-cluster query, or by exporting the data from Application Insights and hooking up EventGrid ingestion into your Kusto cluster.
Relevant links:
Kusto supporting external data:
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/schema-entities/externaltables
Querying data inside Application Insights:
https://learn.microsoft.com/en-us/azure/data-explorer/query-monitor-data
Continuous export data from Application Insights:
https://learn.microsoft.com/en-us/azure/azure-monitor/app/export-telemetry
Data ingestion into Kusto from EventGrid:
https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-event-grid
I want to fetch my website's google analytics data into cassandra using Nifi , is there any way for it?
You should be able to use InvokeHttp to query the Google Analytics data (perhaps with a POST to batchGet), or you could use ExecuteScript and include the Java API driver/JAR in your Module Directory path. Once you get the response(s), there are many processors you can use for any transformation of the data, including ReplaceText for generating a CQL statement to insert the data. Then you can send the statement(s) to PutCassandraQL for ingest into Cassandra.
I have an object delivery service on Akamai and i would like to fetch the usage data for our endpoints and send it to graphite and create some graphs on grafana.
so appreciate any suggestions or ideas to achieve this ?
We have built a Prometheus exporter for Akamai data, the cloudmonitor_exporter. If you couple this with the graphite_exporter, you should be able to get the data into Graphite (albeit a bit roundabout). If you want to go directly to Graphite, you can probably pick up some inspiration from how we built it for Prometheus.
(Note: This requires that you have the "Cloud Monitor" product in your Akamai account)
I’ve read the Kibana website so I know theoretically what Kibana can be used for, but I’m interested in real-world stories. Is Kibana solely used for log analysis? Can it be used as some kind of BI tool across your actual data set? Interested to hear what kind of applications its useful in.
Kibana is very useful for visualizing mixed types of data, not just numbers - metrics, but also text and GEO data. You can use to Kibana to visualize:
real-time data about visitors of your webpage
number of sales per region
locations from sensor data
emails sent, server load, most frequent errors
... and many other there's a plethora of use cases, you only need to feed your data into Elasticsearch (and find appropriate visualization)
Kibana is basically an analytics and visualization platform, which lets you easily visualize data from Elasticsearch and analyze it to make sense of it. You can assume Kibana as an Elasticsearch dashboard where you can create visualizations such as pie charts, line charts, and many others.
There are like the infinite number of use cases for Kibana. For example, You can plot your website’s visitors onto a map and show traffic in real time. Kibana is also where you configure change detection and forecasting. You can aggregate website traffic by the browser and find out which browsers are important to support based on your particular audience. Kibana also provides an interface to manage authentication and authorization regarding Elasticsearch. You can literally think of Kibana as a Web interface to the data stored on Elasticsearch.
As we know Elasticsearch stores, search and analyses data and then shows it on Kibana. But I have my data already stored in PostgreSQL and we have to deal with huge data, so storing it in Elasticsearch for seeing a graph on Kibana is not good. There will be duplication like we have same data in Postgres as well as in Elasticsearch and I have huge data (full traffic from a telecom company) and we want to build a reporting tool.
Kibana has all the features that we want but we don't want this duplication of data. I mean we want to use only Kibana. Is it possible? And what should I do to avoid this problem? What are the possibilities?
My opinion. If you have all this data, and it is not in a non-sql, document database, your are going about it the wrong way. Either it's elasticsearch or mongo, you should use that kind of databases.
As far as I know, there is no way of using Kibana to display information from something other than Elasticsearch.
You could check out Grafana http://grafana.org/, it has that and more.
Good luck.
For connecting to SQL databases, Tableau is one of the best options. As I worked with both Tableau and Kibana, I can tell that Tableau supports almost all operations that are supported by Kibana and also Tableau can generate graphs for complex visualizations like
sum(field1)/ sum(field2) over values of field3.
which can not be generated by using Kibana.
This is way late, but the way I would tackle this is to write an app that pulls data out of your database and publishes to elasticsearch. Sure there is duplication of data, but you can focus on only that data you care about. You also wouldn't be querying against a production database when displaying charts in kibana, which can add its own complications.