ingesting data to kakfka cluster that can send data to adx kusto db by using kafka-sink-azure-kusto .
iam successfully ingesting data to kafka cluster and its not transferring data to kusto db. how to debug this? any logs i can verify.
i have tried to check broker log no errors there
ref:https://github.com/Azure/kafka-sink-azure-kusto/blob/master/README.md
Could you please provide more information about how you are running Kafka, and how did you set up the connector?
Debugging steps would be:
Broker logs should mention that connector was picked up properly, did you see that line in the logs?
Looking at Connector logs should show more info about what is actually going on under the hood. Maybe you will see some errors there. /var/log/connect-distributed.log
Try to ingest data via any other method? like one of the SDK's
Try running the setup according to steps detailed under delpoy
Update: more info about connector setup in general can be found at this SO question: Kafka connect cluster setup or launching connect workers
Also, confluent has some helpful docs:https://docs.confluent.io/current/connect/userguide.html
Related
Is there any option in Kaa when device contacted the Kaa platform with some data, Is it able to send the same information to our external systems through message broker? For example when a temperature sensor updated the current temperature value to the Kaa, Is kaa able to send the same information to the Messaging brokers like ActiveMQ.
Maybe you can try the Kafka or Flume appender of Kaa.
I try to use the Kafka appender to send the data from some sensor to storm server like below reference and it works fine.
https://www.kaaproject.org/iot-real-time-data-processing-in-storm-using-kaa/
And you also can custom your appender by following below url:
https://kaaproject.github.io/kaa/docs/v0.10.0/Customization-guide/Log-appenders/
There are many possibilities to do that which are better or not depending on your particular use case.
But, usually the most efficient way would be to use one of the existing Log Appenders that are running on the Kaa server side and were specifically created for such messaging.
Is there a way to use flume to transmit my Oracle 11g Database data to HDFS?
I know flume is made for logs and Sqoop should be use to transmit data from Database. But is there a way to use flume instead of Sqoop? What should I do if I want to use this kind of architecture?
Please have a look in to
1) Oracle Golden gate
2) Streaming Oracle Database Logs to HDFS with Flume
The other way to do is For the present data in oracle you can run sqoop and for the subsequent changes you can use Linked in databus for change data capture(CDC) which can post messages to Kafka.
Messages from kafka can be easily consumed by Flume.
I would like to know (and retrieve via REST API) the uptime of individual VMs running in OpenStack.
I was quite surprised that OpenStack web UI has a colon called "Uptime" but it actually show time since the VM was created. If i stop the VM, the UI shows Status=Shutoff, Power State=Shutdown, but the Uptime is still being incremented...
Is there a "real" uptime (I mean for a machine that is UP)?
Can I retrieve it somehow via the OpenStack's REST API?
I saw the comment at How can I get VM instance running time in openstack via python API? but the page with the extension mentioned there does not exists and it looks to me that this extension will not be available in all OpenStack environment. I would like to have some standard way to retrieve the uptime.
Thanks.
(Version Havana)
I haven't seen any documentation saying this is the reason, but the nova-scheduler doesn't differentiate between a running and powered off instance. So your cloud can't be over-allocated or leave an instance in a position that would be unable to be powered on. I would like to see a metric of actual system runtime as well, but at the moment the only way to gather that would be through ceilometer or via Rackspaces StackTach
We are having a active-active BizTalk cluster with windows server as software load balancer. The solution includes a SAP receive adapter accepting inbound rfc calls. The goal is to make SAP adapter high availabile.
Read the documentation (), it does says 'You must always cluster the SAP receive adapter to accommodate a two-phase commit scenario.' and 'hosts running the receive handlers for FTP, MSMQ, POP3, SQL, and SAP require a clustering mechanism to provide high availability.'
What we currently did in both the active-active node for BizTalk, we have a host instance enabled. With refering to above documentation, does it mean we did it incorrectly? We should take the clustered host instance instead the active-active deployment?
thanks for all the help in advance.
You need to cluster the host that handles the SAP receive. What this means is that you will always have only one instance of the adapter running at any given time and if one of the server goes down, the other will pick up.
Compare this with your scenario where you simply have two (non-clustered) instances running concurrently: yes, this gives you high availability - but also deadlocks! The two will run independently of each other... With the cluster scenario above, they will run one at the time
To cluster the SAP receive host: open the admin console, find the host, right-click and Cluster.
Are there any patterns or practices for monitoring log4net exception logs across a cluster of web servers. I have considered several options including the following:
A central database
A log file retrieval system
A service based loggin architecture
Thanks,
Richard
Message Queuing is a great solution. It works great in a distributed enviroment where one machine or multiple can be popping log messages of the queue and persisting them somewhere (Central logging database or rolling flat files...). And if at any time a machine creating messages or popping go offline they can continue logging messages and continue as normal when they are back online.
Your could configure log4net to trace into the event log of each machine and use a hypervisor software such as SCOM to monitor each node.