I have kinesis stream from DynamoDB. And I am processing it through aws KCL sdk v1.14.0. I see below occasional errors in the logs. I also observe that startingHashKey is always 1 and endingHashKey is 0 for all shards in the dynamodb lease table. Any clue why is this happening?
Incomplete hash range found between {
"leaseKey" : "shardId-00000001601992582645-697387df",
"leaseOwner" : "eu01-stg01-vendor-service-74b4cd66b6-6hcfb:29dafc2f-c87f-4c70-b210-d38db7ebfc87",
"leaseCounter" : 4080,
"concurrencyToken" : null,
"lastCounterIncrementNanos" : null,
"checkpoint" : {
"sequenceNumber" : "47515300000000010794885359",
"subSequenceNumber" : 0,
"shardEnd" : false
},
"pendingCheckpoint" : null,
"ownerSwitchesSinceCheckpoint" : 0,
"parentShardIds" : [ ],
"childShardIds" : [ ],
"hashKeyRange" : {
"startingHashKey" : 0,
"endingHashKey" : 1
}
} and {
"leaseKey" : "shardId-00000001602005637902-6358a08a",
"leaseOwner" : "eu01-stg01-vendor-service-74b4cd66b6-6hcfb:29dafc2f-c87f-4c70-b210-d38db7ebfc87",
"leaseCounter" : 397,
"concurrencyToken" : null,
"lastCounterIncrementNanos" : null,
"checkpoint" : {
"sequenceNumber" : "TRIM_HORIZON",
"subSequenceNumber" : 0,
"shardEnd" : false
},
"pendingCheckpoint" : null,
"ownerSwitchesSinceCheckpoint" : 0,
"parentShardIds" : [ "shardId-00000001601992582736-2c3fc0ff" ],
"childShardIds" : [ ],
"hashKeyRange" : {
"startingHashKey" : 0,
"endingHashKey" : 1
}
}
I also get this error, but the stream seems to work properly. (I am using it to monitor the data change).
Then I dig the source code of this error log, find the part code is added on Jun 22, 2020,
This is the source code: https://github.com/awslabs/amazon-kinesis-client/blob/6fbfc21ad7d6a2722491806a411d168304167f7f/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/PeriodicShardSyncManager.java#L298
This is the blame: https://github.com/awslabs/amazon-kinesis-client/blame/6fbfc21ad7d6a2722491806a411d168304167f7f/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/PeriodicShardSyncManager.java#L298
This is the commit: https://github.com/awslabs/amazon-kinesis-client/commit/3a88a60a4ee7bb8f969b36cb79e7665a7395b6ec
it is only in the latest version: 1.14.0
So I go back one version to use 1.13.3 then no this error.
I have submitted an issue on the GitHub to see what the reason for the error. https://github.com/awslabs/amazon-kinesis-client/issues/758
Related
I'm trying to integrate Kibana with my OpenSearch ( is it possible ? ). Unfortunately I get a version error.
Is there any way to use opensearch in kibana?
These are the versions I get via curl:
curl -k -u "admin:PASSWORD" "https://IP:9200/"
{
"name" : "node-1",
"cluster_name" : "cluster",
"cluster_uuid" : "he6gqhl2S-6dlVv6dyPOEA",
"version" : {
"number" : "7.10.2",
"build_type" : "rpm",
"build_hash" : "e505b10357c03ae8d26d675172402f2f2144ef0f",
"build_date" : "2022-01-14T03:38:06.881862Z",
"build_snapshot" : false,
"lucene_version" : "8.10.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}
They are very tightly coupled and aren't really supposed to work together, so I doubt they will (or it would be very brittle).
If you want to have Kibana, why not Elasticsearch under the hood?
Objective
Refresh a partition using a query (override). Using these as a guide:
https://www.sqlbi.com/articles/using-process-add-in-tabular-models/
https://gist.github.com/dgosbell/a7bc9fe9ff5a99fdb4df5819b8760217#file-refresh-with-override-example-txt
Apparently the MS example is not correct: https://learn.microsoft.com/en-us/bi-reference/tmsl/refresh-command-tmsl#examples
TMSL Script
{
"refresh" : {
"type" : "add",
"objects" : [{
"database" : "dbname",
"table" : "tblname"
}
],
"overrides" : [{
"partitions" : [
{
"originalObject" : {
"database" : "dbname",
"table" : "tblname",
"partition" : "partname"
},
"source" : {
"query" :
"SELECT * FROM source.view WHERE date_field = '2014-12-06'"
}
}
]
}
]
}
}
Error Message
The JSON DDL request failed with the following error: Failed to execute XMLA. Error returned: 'The Process command for partition 'partname' in table 'tblname' cannot be executed because the pipeline rowset is missing.
'..
Technical Details:
RootActivityId: 89a6f9ac-e5d4-4eaa-b049-455190039b4b
Date (UTC): 6/28/2019 3:20:36 PM
0: PFError::SetLastError() line 2158 + 0x0 (sql\picasso\engine\src\pf\eh\pferror.cpp)
1: PFSetLastError() line 2906 + 0x0 (sql\picasso\engine\src\pf\eh\pferror.cpp)
2: ConvertExceptionsToPFResult<<lambda_764f81a97ea803a6bb1663c7971ce151> >() line 424 + 0x34 (sql\picasso\engine\src\pf\kernel\shared\pfshmacros.inl)
3: PFSetLastErrorExTag() line 3461 + 0x2e (sql\picasso\engine\src\pf\eh\pferror.cpp)
4: 0x00007FFAB599CC7E (symbolic name unavailable)
Other Info
Executed on SSMS directly and in Powershell (via Runbook) with same error message.
Question
What does this error message mean exactly? (It is very hard to find
helpful documentation.) Or, is there an alternative solution to refreshing
a partition using a query override?
You can only use either a Query or M partition sources
If you are using an M partition source the syntax is:
"source":{
"type":"m",
"expression":"…"
}
And if you are using a Query partition source the syntax is:
"source":{
"type":"query",
"query":"…",
"dataSource":"…"
}
I am using following code (MongoDB C# driver) to projects fields ,
where queryDocument = '{{ "FullName" : /myname/i }}'
BsonDocument projectionDefination = Builders.Projection.ToBsonDocument();
projectionDefination.Add(new BsonElement("FullName", "myname"));
collection.Find(queryDocument).Project(projectionDefination);
getting following error:
{"Command failed."}
{{ "_t" : "OKMongoResponse", "ok" : 0, "code" : 9, "errmsg" : "Syntax error, incorrect syntax near '9'.", "$err" : "Syntax error, incorrect syntax near '9'." }}
Can anyone update on this?
I also tried above with RoboMongo
Command: db.getCollection('Employee').find({ "FullName": /User/i}).projection({ "FullName" : "$FullName" })
Received following error:
Error: error: {
"_t" : "OKMongoResponse",
"ok" : 0,
"code" : 9,
"errmsg" : "Syntax error, incorrect syntax near '15'.",
"$err" : "Syntax error, incorrect syntax near '15'."
}
According to your description, I checked this issue and I could encounter the same issue:
As mongoDB document states about Projection in db.collection.find():
The projection parameter determines which fields are returned in the matching documents. The projection parameter takes a document of the following form:
{ field1: <value>, field2: <value> ... }
The <value> can be any of the following:
1 or true to include the field in the return documents.
0 or false to exclude the field.
My mongodb is looking like this :
"_id" : ObjectId("5751749723be8328848af30e"),
"mem_id" : 1.0,
"mem_name" : "Member1",
"home" : "8001234567",
"mobile" : "8771231234",
"email" : "jd#example.com",
"mem_create_date" : ISODate("2016-06-03T12:14:15.642Z"),
"paymethod" : {
"pm_id" : "99991",
"pm_name" : "FA11",
"pm_create_Date" : ISODate("2013-12-09T21:05:00.000Z"),
"routing_no" : "075000051"
},
"payee" : {
"pae_id" : "88881",
"pae_name" : "Payee1",
"home" : "800-123-45671",
"mobile" : "877-123-12341",
"email" : "jd#example.com",
"act_no" : "0213654781",
"pae_create_date" : ISODate("2013-12-09T21:05:00.000Z"),
"payment" : {
"pmt_id" : "66661",
"pmt_amt" : "71",
"pmt_create_date" : ISODate("2013-12-09T21:05:00.000Z"),
"pmt_type" : "check"
}
}
}
I need to get these values with the help of mongolite and show them in a plot/graph with R shiny web app.
I could not find any example with nested mongodb on net. And whenever i am trying to read the data i am getting the below err :
error in if (inherits(x j dataframe ) && ncol(xj) 1l) x j - as.matrix(x j )
Can anyone please help/guide me here. I am a java developer but i am new to R language.
Thanks.
Here is what you get in mongo shell :
db.col.find(ObjectId("5571849db1969e0a6eb32731")).pretty()
{
"_id" : ObjectId("5571849db1969e0a6eb32731"),
"name" : "Some name",
"logo" : "Some logo",
"users" : [
ObjectId("5571830031c7fc341bc2e105"),
ObjectId("5571830031c7fc341bc2e107")
],
"admins" : [ ],
"__v" : 0,
"steps" : 5782
}
Here is what I get in rmongo :
myResult <- mongo.find(Connexion, "db.col", query='ObjectId("5571849db1969e0a6eb32731")')
#Error in mongo.bson.from.JSON(arg) :
# Not a valid JSON content: ObjectId("5571849db1969e0a6eb32731")
So, how to do it right ?
Just in case : I had a look at this already. But mongolite doesn't support authentication (which is therefore a no-go) and I don't understand what to do with the second answer. If I try
result <- mongo.find(Connexion, "db.col", query=mongo.oid.from.string("5571849db1969e0a6eb32731"))
I get
# Error in mongo.bson.from.argument(query) : Can't convert to bson: argument should be one of 'list', 'mongo.bson' or 'character'(valid JSON)
This should work:
result <- mongo.find(Connexion, "db.col", query=list('_id' = mongo.oid.from.string("5571849db1969e0a6eb32731")))