I'm getting this error Could not locate method: NeptuneGraphTraversal.values() when running a query using gremlin for aws neptune. The specific query is this:
import { process } from 'gremlin';
const { statics, t } = process;
function getUsers() {
return await this.query(g =>
g
.V()
.match(
statics
.as('me')
.V(this.id)
// .has('user', t.id, this.id)
.fold()
.coalesce(
statics.unfold(),
statics.addV('user').property(t.id, this.id)
),
statics.as('me').out('followed').out('followed').as('x'),
statics.as('x').not(statics.has(t.id, this.id)),
statics
.as('x')
.out('followed')
.has(t.id, this.id)
.count()
.is(0),
statics.as('x').in_('skip').has(t.id, this.id).count().is(0)
)
.select('x')
.values(t.id)
.dedup()
.sample(3)
.toList()
);
}
The error message suggests that neptune's GraphTraversal doesn't implement the values method, but I've reviewed the implementation differences and there's no mention of that. What could be the problem? What alternatives can I consider?
This is likely due to the fact that you're using the values() step to extract the ID. values() takes a string of a property key. More likely what you need is the id() step.
Related
I am working on a basic Support Ticket System. I get the Tickets from Firebase (Either as a Stream or Future).
I want to allow some Filtering Options (e.g. sort by Status and Category).
For this, I thought about using A Future Provider to get the List and a StateNotiferProvider to update the List depending on which filter is being used.
This is the code I have so far:
final ticketListStreamProvider =
RP.FutureProvider((_) => FirestoreService.getTicketList());
class TicketListNotifier extends RP.StateNotifier<List<Ticket>> {
TicketListNotifier() : super([]);
void addTicket(Ticket ticket) {
state = List.from(state)..add(ticket);
}
void removeTicket(Ticket ticket) {
state = List.from(state)..remove(ticket);
}
}
final ticketsController =
RP.StateNotifierProvider<TicketListNotifier, List<Ticket>>(
(ref) => TicketListNotifier(),
);
There are multiple issues I have with that. Firstly it doesn't work.
The StateNotifier accepts a List and not a Future<List>. I need to convert it somehow or rewrite the StateNotifier to accept the Future.
I was trying to stay close to one of the official examples.
(https://github.com/rrousselGit/riverpod/tree/master/examples/todos)
Unfortunately, they don't use data from an outside source like firebase to do it.
What's the best approach to get resolve this issue with which combination of providers?
Thanks
You can fetch your ticketlist in your TicketListNotifier and set its state with your ticket list.
class TicketListNotifier extends RP.StateNotifier<List<Ticket>> {
TicketListNotifier() : super([]);
Future<void> fetchTicketList() async {
FirestoreService.getTicketList().when(
//success
// state = fetched_data_from_firestore
// error
// error handle
)
}
}
final ticketsController =
RP.StateNotifierProvider<TicketListNotifier, List<Ticket>>(
(ref) => TicketListNotifier(),
);
Call this method where you want to fetch it /*maybe in your widget's initState method
ref.read(ticketsController.notifier).fetchTicketList();
Now ref.read(ticketsController); will return your ticket list
Since you have the ticket list in your TicketListNotifier's state you can use your add/remove method like this:
ref.read(ticketsController.notifier).addTicket(someTicket);
I'm using both client and server from node-opcua, it's working fine apart for error management on the server side when the client tries to write a value and something goes wrong. In order to improve that, I'd like to have an asynchronous setter like in this example:
const wantedValue = namespace.addAnalogDataItem({
componentOf: node,
browseName: `${chamber}${folder}Req`,
nodeId: `s=${chamber}${folder}Req`,
dataType: "Double",
engineeringUnits: settings.unit,
engineeringUnitsRange: settings.range,
value: {
get: () => new Variant({dataType: DataType.Double, value: wrapper.readProgram(chamber, folder)}),
set: async (variant: Variant) => {
const success = await wrapper.writeProgram(chamber, folder, variant.value)
return success ? StatusCodes.Good : StatusCodes.Bad
}
}
})
This naive approach lead to a typer error:
throw new Error("Cannot find StatusCode " + statusCode);
Is there a better way to do that ?
the get and set lambdas are design to be synchronous.
you need to use the callback form with timestamped_get and timestamped_set to achieve what you need.
I have a large Firestore collection with 10,000 documents.
I want to show these documents in a table by paging and filtering the results at 25 at a time.
My idea, to limit the "reads" (and therefore the costs), was to request only 25 documents at a time (using the 'limit' method), and to load the next 25 documents at the page change.
But there's a problem. In order to show the number of pages I have to know the total number of documents and I would be forced to query all the documents to find that number.
I could opt for an infinite scroll, but even in this case I would never know the total number of results that my filter has found.
Another option would be to request all documents at the beginning and then paging and filtering using the client.
so, what is the best way to show data in this type of situation by optimizing performance and costs?
Thanks!
You will find in the Firestore documentation a page dedicated to Paginating data with query cursors.
I paste here the example which "combines query cursors with the limit() method".
var first = db.collection("cities")
.orderBy("population")
.limit(25);
return first.get().then(function (documentSnapshots) {
// Get the last visible document
var lastVisible = documentSnapshots.docs[documentSnapshots.docs.length-1];
console.log("last", lastVisible);
// Construct a new query starting at this document,
// get the next 25 cities.
var next = db.collection("cities")
.orderBy("population")
.startAfter(lastVisible)
.limit(25);
});
If you opt for an infinite scroll, you can easily know if you have reached the end of the collection by looking at the value of documentSnapshots.size. If it is under 25 (the value used in the example), you know that you have reached the end of the collection.
If you want to show the total number of documents in the collection, the best is to use a distributed counter which holds the number of documents, as explained in this answer: https://stackoverflow.com/a/61250956/3371862
Firestore does not provide a way to know how many results would be returned by a query without actually executing the query and reading each document. If you need a total count, you will have to somehow track that yourself in another document. There are plenty of suggestions on Stack Overflow about counting documents in collections.
Cloud Firestore collection count
How to get a count of number of documents in a collection with Cloud Firestore
However, the paging API itself will not help you. You need to track it on your own, which is just not very easy, especially for flexible queries that could have any number of filters.
My guess is you would be using Mat-Paginator and the next button is disabled because you cannot specify the exact length? In that case or not, a simple workaround for this is to get (pageSize +1) documents each time from the Firestore sorted by a field (such as createdAt), so that after a new page is loaded, you will always have one document in the next page which will enable the "next" button on the paginator.
What worked best for me:
Create a simple query
Create a simple pagination query
Combine both (after validating each one works separately)
Simple Pagination Query
const queryHandler = query(
db.collection('YOUR-COLLECTION-NAME'),
orderBy('ORDER-FIELD-GOES-HERE'),
startAt(0),
limit(50)
)
const result = await getDocs(queryHandler)
which will return the first 50 results (ordered by your criteria)
Simple Query
const queryHandler = query(
db.collection('YOUR-COLLECTION-NAME'),
where('FIELD-NAME', 'OPERATOR', 'VALUE')
)
const result = await getDocs(queryHandler)
Note that the result object has both query field (with relevant query) and docs field (to populate actual data)
So... combining both will result with:
const queryHandler = query(
db.collection('YOUR-COLLECTION-NAME'),
where('FIELD-NAME', 'OPERATOR', 'VALUE'),
orderBy('FIELD-NAME'),
startAt(0),
limit(50)
)
const result = await getDocs(queryHandler)
Please note that the field in the where clause and in orderBy must be the same one! Also, it is worth mentioning that you may be required to create an index (for some use cases) or that this operation will fail while using equality operators and so on.
My tip: inspect the error itself where you will find a detailed description describing why the operation failed and what should be done in order to fix it (see an example output using js client in image below)
Firebase V9 functional approach. Don't forget to enable persistence so you won't get huge bills. Don't forget to use where() function if some documents have restrictions in rules. Firestore will throw error if even one document is not allowed to read by user. In case bellow documents has to have isPublic = true.
firebase.ts
function paginatedCollection(collectionPath: string, initDocumentsLimit: number, initQueryConstraint: QueryConstraint[]) {
const data = vueRef<any[]>([]) // Vue 3 Ref<T> object You can change it to even simple array.
let snapshot: QuerySnapshot<DocumentData>
let firstDoc: QueryDocumentSnapshot<DocumentData>
let unSubSnap: Unsubscribe
let docsLimit: number = initDocumentsLimit
let queryConst: QueryConstraint[] = initQueryConstraint
const onPagination = (option?: "endBefore" | "startAfter" | "startAt") => {
if (option && !snapshot) throw new Error("Your first onPagination invoked function has to have no arguments.")
let que = queryConst
option === "endBefore" ? que = [...que, limitToLast(docsLimit), endBefore(snapshot.docs[0])] : que = [...que, limit(docsLimit)]
if (option === "startAfter") que = [...que, startAfter(snapshot.docs[snapshot.docs.length - 1])]
if (option === "startAt") que = [...que, startAt(snapshot.docs[0])]
const q = query(collection(db, collectionPath), ...que)
const unSubscribtion = onSnapshot(q, snap => {
if (!snap.empty && !option) { firstDoc = snap.docs[0] }
if (option === "endBefore") {
const firstDocInSnap = JSON.stringify(snap.docs[0])
const firstSaved = JSON.stringify(firstDoc)
if (firstDocInSnap === firstSaved || snap.empty || snap.docs.length < docsLimit) {
return onPagination()
}
}
if (option === "startAfter" && snap.empty) {
onPagination("startAt")
}
if (!snap.empty) {
snapshot = snap
data.value = []
snap.forEach(docSnap => {
const doc = docSnap.data()
doc.id = docSnap.id
data.value = [...data.value, doc]
})
}
})
if (unSubSnap) unSubSnap()
unSubSnap = unSubscribtion
}
function setLimit(documentsLimit: number) {
docsLimit = documentsLimit
}
function setQueryConstraint(queryConstraint: QueryConstraint[]) {
queryConst = queryConstraint
}
function unSub() {
if (unSubSnap) unSubSnap()
}
return { data, onPagination, unSub, setLimit, setQueryConstraint }
}
export { paginatedCollection }
How to use example in Vue 3 in TypeScript
const { data, onPagination, unSub } = paginatedCollection("posts", 8, [where("isPublic", "==", true), where("category", "==", "Blog"), orderBy("createdAt", "desc")])
onMounted(() => onPagination()) // Lifecycle function
onUnmounted(() => unSub()) // Lifecycle function
function next() {
onPagination('startAfter')
window.scrollTo({ top: 0, behavior: 'smooth' })
}
function prev() {
onPagination('endBefore')
window.scrollTo({ top: 0, behavior: 'smooth' })
}
You might have problem with knowing which document is last one for example to disable button.
I'm now implement a program to migrate large amount of data to ADX base on Ingest from Storage feature of ADX and I'm need to check that status of each ingestion request each time the request finish but I'm facing an issue
Base on MS document in here
If I set the persistDetails = true for example with the command below it must save the ingestion status but currently this setting seem not work (with or without it)
.ingest async into table MigrateTable
(
h'correct blob url link'
)
with (
jsonMappingReference = 'table_mapping',
format = 'json',
persistDetails = true
)
Above command will return an OperationId and when I using it to check export status when the ingest task finish I always get this error message :
Error An admin command cannot be executed due to an invalid state: State='Operation 'DataIngestPull' does not persist its operation results' clientRequestId: KustoWebV2;
Can someone clarify for me what is the root cause relate to this? With me it seem like a bug relate to ADX
Ingesting data directly against the Data Engine, by running .ingest commands, is usually not recommended, compared to using Queued Ingestion (motivation included in the link). Using Kusto's ingestion client library allows you to track the ingestion status.
Some tools/services already do that for you, and you can consider using them directly. e.g. LightIngest, Azure Data Factory
If you don't follow option 1, you can still look for the state/status of your command using the operation ID you get when using the async keyword, by using .show operations
You can also use the client request ID to filter the result set of .show commands to view the state/status of your command.
If you're interested in looking specifically at failures, .show ingestion failures is also available for you.
The persistDetails option you specified in your .ingest command actually has no effect - as mentioned in the docs:
Not all control commands persist their results, and those that do usually do so by default on asynchronous executions only (using the async keyword). Please search the documentation for the specific command and check if it does (see, for example data export).
============ Update sample code follow suggestion from Yoni ========
Turn out, other member in my team mess up with access right with adx, after fixing it everything work fine
I just have one concern relate to PartiallySucceeded that need clarify from #yoni or someone have better knowledge relate to that
try
{
var ingestProps = new KustoQueuedIngestionProperties(model.DatabaseName, model.IngestTableName)
{
ReportLevel = IngestionReportLevel.FailuresAndSuccesses,
ReportMethod = IngestionReportMethod.Table,
FlushImmediately = true,
JSONMappingReference = model.IngestMappingName,
AdditionalProperties = new Dictionary<string, string>
{
{"jsonMappingReference",$"{model.IngestMappingName}" },
{ "format","json"}
}
};
var sourceId = Guid.NewGuid();
var clientResult = await IngestClient.IngestFromStorageAsync(model.FileBlobUrl, ingestProps, new StorageSourceOptions
{
DeleteSourceOnSuccess = true,
SourceId = sourceId
});
var ingestionStatus = clientResult.GetIngestionStatusBySourceId(sourceId);
while (ingestionStatus.Status == Status.Pending)
{
await Task.Delay(WaitingInterval);
ingestionStatus = clientResult.GetIngestionStatusBySourceId(sourceId);
}
if (ingestionStatus.Status == Status.Succeeded)
{
return true;
}
LogUtils.TraceError(_logger, $"Error when ingest blob file events, error: {ingestionStatus.ErrorCode.FastGetDescription()}");
return false;
}
catch (Exception e)
{
return false;
}
After inserting 29447 entities of a single kind in Google Cloud DataStore I wait about 30 seconds and go and check how many entities are there for that particular kind. The surprising thing is that I notice some of them missing (getCurrentKeys returns a bit less than 29447 entities). When I check after a longer period of time (~1 hour), I can then see that all of the entities are there (getCurrentKeys returns the expected 29447 entities).
The code used to read the number of entities is the following:
const runQuery = (query) => {
return new Promise((resolve, reject) => {
datastore.runQuery(query)
.then(results => {
const entities = results[0];
resolve(entities);
})
.catch(e => reject(e));
});
};
const getCurrentKeys = () => {
const query = datastore.createQuery(KIND)
.select('__key__');
return runQuery(query);
};
async function main() {
const currentKeys = await getCurrentKeys();
console.log(`currentKeys: ${currentKeys.length}`);
}
main();
Any ideas of what could be happening?
Thanks in advance
Non ancestor queries are eventually consistent. It will take a while before all the rows will show up.
This article should explain more:
https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore/
After a bit more research I think it might be related to the indexes. I believe the indexes aren't getting updated fast enough by the time I run the query. The entities have many properties, so there are many indexes involved.