HibernateMetrics expose via spring boot actuator does not to capture the SQL statements. The SQL statements are needed to analyze individual query execution plan.
new HibernateMetrics(sessionFactory, "myapp", Collections.emptyList()).bindTo(registry);
Is there a micrometer available for this purpose?
Related
I am using redis as my data store and i am using spring boot starter data redis as the dependency and a crud repository for CRUD operation. When i do deleteById it is working. But when i do deleteByName (Name is not an id column) it is saying query method not supported. When the data source is redis and if we use spring boot starter data redis, it is possible only to deleteById column and not by other columns?
I had the same problem that after a lot of searching and also checking the following link:
Redis 2.1.2.RELEASE doc reference
I realized that since Redis is based on Key and Valve, the following solution should be used:
List<SampleClass> lst = cacheTokenRepository.findAllBySampleFields(... sampleFields);
lst.forEach(item->
SampleRepository.deleteById(item.getId())
);
In this solution we search the list of records that match our conditions and delete them all based on id
I'm using ADF pipeline for storing data from data lake into cosmos db sql api. On triggering pipeline run, I see the following error:
Currently I'm using Throughput 5000 RU/s for the cosmos db container. Please help me understand why it is showing this error?
Here is my pipeline:
For saving your cost, don't unlimited increase RUs setting of cosmos db. Based on the error message you provided in your answer,it indicates that cosmos db can't process so many rows in unit time. So i suggest you throttling the transfer data procedure. Please consider using below configuration for the sink dataset of your copy activity.
The default value is 10,000, please decrease the number here if you do not concern the speed of transmission.More details ,please refer to this document.
Additional, you could throttling the max concurrent connections:
I want to build a Kafka Connector in order to retrieve records from a database at near real time. My database is the Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 and the tables have millions of records. First of all, I would like to add the minimum load to my database using CDC. Secondly, I would like to retrieve records based on a LastUpdate field which has value after a certain date.
Searching at the site of confluent, the only open source connector that I found was the “Kafka Connect JDBC”. I think that this connector doesn’t have CDC mechanism and it isn’t possible to retrieve millions of records when the connector starts for the first time. The alternative solution that I thought is Debezium, but there is no Debezium Oracle Connector at the site of Confluent and I believe that it is at a beta version.
Which solution would you suggest? Is something wrong to my assumptions of Kafka Connect JDBC or Debezium Connector? Is there any other solution?
For query-based CDC which is less efficient, you can use the JDBC source connector.
For log-based CDC I am aware of a couple of options however, some of them require license:
1) Attunity Replicate that allows users to use a graphical interface to create real-time data pipelines from producer systems into Apache Kafka, without having to do any manual coding or scripting. I have been using Attunity Replicate for Oracle -> Kafka for a couple of years and was very satisfied.
2) Oracle GoldenGate that requires a license
3) Oracle Log Miner that does not require any license and is used by both Attunity and kafka-connect-oracle which is is a Kafka source connector for capturing all row based DML changes from an Oracle and streaming these changes to Kafka.Change data capture logic is based on Oracle LogMiner solution.
We have numerous customers using IBM's IIDR (info sphere Data Replication) product to replicate data from Oracle databases, (as well as Z mainframe, I-series, SQL Server, etc.) into Kafka.
Regardless of which of the sources used, data can be normalized into one of many formats in Kafka. An example of an included, selectable format is...
https://www.ibm.com/support/knowledgecenter/en/SSTRGZ_11.4.0/com.ibm.cdcdoc.cdckafka.doc/tasks/kcopauditavrosinglerow.html
The solution is highly scalable and has been measured to replicate changes into the 100,000's of rows per second.
We also have a proprietary ability to reconstitute data written in parallel to Kafka back into its original source order. So, despite data having been written to numerous partitions and topics , the original total order can be known. This functionality is known as the TCC (transactionally consistent consumer).
See the video and slides here...
https://kafka-summit.org/sessions/exactly-once-replication-database-kafka-cloud/
Cosmos DB FAQ says in Cassandra API section that Azure Cosmos DB provides automatic indexing of all attributes without any schema definition. https://learn.microsoft.com/en-us/azure/cosmos-db/faq#does-this-mean-i-dont-have-to-create-more-than-one-index-to-satisfy-the-queries-1
But when I try to add WHERE column1 = 'x' filter to my CQL query, I get exception from Datastax cassandra driver saying that data filtering is not supported. I tried to bypass client driver by supplying ALLOW FILTERING but this time got error from cosmos server saying this feature is not implemented.
So, if automatic indexing is implemented for Cosmos/Cassandra API, how can it be used?
I am working on the Zabbix monitoring tool.
Could any one advise whether we have any tool to generate reports.
Not at my knowledge out-of-the box.
Zabbix is tricky because MySQL backend history tables grow extremely fast and they don't have primary keys. Our current history tables have 440+ million records and we monitor 6000 servers by Zabbix. Single table scan takes 40 minutes on the active server.
So your challenge could be splitted in three smaller challenges:
History
Denormalization is the key because joins don't work on huge history tables because you have to join history, items, functions, triggers and hosts tables.
Besides you want to evaluate global and host macros, replace {ITEM.VALUE} and {HOST.NAME} in trigger and item names/descriptions.
BTW there is experimental version of Zabbix which uses Elasticsearch for keeping history and it makes possible sorting and selecting item values by intervals. Zabbix using Elasticsearch for History Tables
My approach is to generate structures like this for every Zabbix record from history tables and dump them to the document database. Make sure you don't use buffered cursors.
{'dns_name': '',
'event_clock': 1512501556,
'event_tstano': '2017-12-05 19:19:16',
'event_value': 1,
'host_id': 10084,
'host_name': 'Zabbix Server',
'ip_address': '10.13.37.82',
'item_id': 37335,
'item_key': 'nca.test.backsync.multiple',
'item_name': 'BackSync - Test - Multiple',
'trig_chg_clock': 1512502800,
'trig_chg_tstamp': '2017-12-05 19:40:00',
'trig_id': 17206,
'trig_name': 'BackSync - TEST - Multiple - Please Ignore',
'trig_prio': 'Average',
'trig_value': 'OK'
}
Current Values
Zabbix APIs are documented pretty good and JSON is handy to dump the structure like proposed for the history. Don't expect Zabbix APIs will return more than 500 metrics / second max. We currently pull 350 metrics / second.
And finally reporting ... There are many options but you have to integrate them:
Jasper
Kibana (Elasticsearch)
Tableau
Operations Bridge Reporter (Vertica)
..
JasperReports - IMHO good "framework" for reports:
connect it with SQL data connector => you have to be familiar with SQL structure of your Zabbix DB
more generic solution will be Zabbix API data connector for JasperReports, but you have to code this data connector, because it doesn't exist
You can use the Event API to export data for reporting.
Quoting from the main reference page:
Events:
Retrieve events generated by triggers, network discovery and other
Zabbix systems for more flexible situation management or third-party
tool integration
Additionally, if you've set up IT Services & SLA, you can use the Service API to extract service % availabilities