One of the data in datastore is 7766277975020011920 and similarities.
The error shown in nodejs is -
Error: We attempted to return all of the numeric values, but chain value 7766277975129421920 is out of bounds of 'Number.MAX_SAFE_INTEGER'.
It suggested "options.wrapNumbers=true" to the file
"node_modules/#google-cloud/datastore/build/src/entity.js:412:19".
But I am using google cloud RUN version and not able to edit the files. How can I pass it ?
Moving my comment into an answer, according to the Node.js client reference for Datastore, when you run queries or calls for entities, you can pass options as an additional argument, which supports passing the wrapNumbers = True option:
const [values] = await datastore.runQuery(query,{"wrapNumbers":true});
You can use that in your calls to avoid receiving out of bounds errors for large integers. The rest of the supported options are documented in this code snippet from the official repository:
const options = {
consistency: 'string',
gaxOptions: {},
wrapNumbers: true,
};
Related
I need some help regarding fetching some data.
I'm building blog with Nuxt3 and Sanity and I use https://sanity.nuxtjs.org/ module.
I need query to get all the unique tags from _type article so I can have filter by tags in my blog.
I wrote this query and It gives result back in my Sanity Vision.
array::unique(*[_type == "article" ].tag.value)
However When I run it in project I get data null. Here is how I try.
const query = groq`array::unique(*[_type == "article" ].tag.value)`
const { fetch } = useSanity()
const { data } = await useAsyncData("article tags", () => fetch(query))
I guess I'm doing something wrong with using Sanity Groq Functions.
It was problem with different versions of Sanity API.
When using #nuxtjs/sanity module the API version by default is 1.
Inside my Sanity studio vision I was testing query with newer version therefor it worked.
I updated sanity configuration inside my nuxt.config and now everything works.
I need to to trigger a Data Fusion pipeline located on a GCP project called myDataFusionProject through a Data Fusion operator (CloudDataFusionStartPipelineOperator) inside a DAG whose Cloud Composer instance is located on another project called myCloudComposerProject.
I have used the official documentation as well as the source code to write the code that roughly resembles the below snippet:
LOCATION = "someLocation"
PIPELINE_NAME = "myDataFusionPipeline"
INSTANCE_NAME = "myDataFusionInstance"
RUNTIME_ARGS = {"output.instance":"someOutputInstance", "input.dataset":"someInputDataset", "input.project":"someInputProject"}
start_pipeline = CloudDataFusionStartPipelineOperator(
location=LOCATION,
pipeline_name=PIPELINE_NAME,
instance_name=INSTANCE_NAME,
runtime_args=RUNTIME_ARGS,
task_id="start_pipeline",
)
My issue is that, every time I trigger the DAG, Cloud Composer looks for myDataFusionInstance inside myCloudComposerProject instead of myDataFusionProject, which gives an error like this one:
googleapiclient.errors.HttpError: <HttpError 404 when requesting https://datafusion.googleapis.com/v1beta1/projects/myCloudComposerProject/locations/someLocation/instances/myDataFusionInstance?alt=json returned "Resource 'projects/myCloudComposerProject/locations/someLocation/instances/myDataFusionInstance' was not found". Details: "[{'#type': 'type.googleapis.com/google.rpc.ResourceInfo', 'resourceName': 'projects/myCloudComposerProject/locations/someLocation/instances/myDataFusionInstance'}]"
So the question is: how can I force my operator to use the Data Fusion project instead of the Cloud Composer project? I suspect I may do that by adding a new runtime argument but I'm not sure how to do that.
Last piece of information: the Data Fusion pipeline simply extracts data from a BigQuery source and sends everything to a BigTable sink.
As a recommendation while developing operators on airflow, we should check the classes that are implementing the operators as documentation may lack some information due to versioning.
As commented, if you check CloudDataFusionStartPipelineOperator you will find that it makes use of a hook that gets the instance base on a project-id. This project-id its optional, so you can add your own project-id.
class CloudDataFusionStartPipelineOperator(BaseOperator):
...
def __init__(
...
project_id: Optional[str] = None, ### NOT MENTION IN THE DOCUMENTATION
...
) -> None:
...
self.project_id = project_id
...
def execute(self, context: dict) -> str:
...
instance = hook.get_instance(
instance_name=self.instance_name,
location=self.location,
project_id=self.project_id, ### defaults your project-id
)
api_url = instance["apiEndpoint"]
...
Adding the parameter to your operator call should fix your issue.
start_pipeline = CloudDataFusionStartPipelineOperator(
location=LOCATION,
pipeline_name=PIPELINE_NAME,
instance_name=INSTANCE_NAME,
runtime_args=RUNTIME_ARGS,
project_id=PROJECT_ID,
task_id="start_pipeline",
)
As a final note, besides the official documentation site you can also explore the files of apache airflow on github.
I am trying to deploy an SNS subscription through Serverless Framework to AWS using a range filter policy. My resource definition looks something like this:
MySusbscription:
Type: AWS::SNS::Subscription
Properties:
TopicArn:
Ref: MyTopic
Endpoint: !GetAtt MyQueue.Arn
Protocol: sqs
RawMessageDelivery: true
FilterPolicy:
percentage:
- numeric:
- '>='
- ${ssm:/path/to/ssm/key}
RedrivePolicy:
deadLetterTargetArn: !GetAtt MyQueueDlq.Arn
The problem is that Serverless pulls all SSM values as strings, so the compiled version of the deployment config will be:
- numeric:
- '>='
- '0.25'
This will fail to deploy as SNS requires 0.25 to be a number instead of a string. Serverless has a strToBool function, but I haven't seen documentation to indicate there is an equivalent function for converting to a number/float.
I'm sure I can get around this by using env variables, but we store all configurations in SSM and I'm hoping to not have to do a one-off just to get past this issue.
The first solution is not the cleanest one, but will work for sure.
In the directory define your custom JS file ssmToNumber.js with following script:
const AWS = require('aws-sdk');
module.exports = async (serverless) => {
# Setup AWS SDK with region and profile
const { region } = serverless.processedInput.options;
AWS.config.update({ region });
if (serverless.processedInput.options['aws-profile']) {
process.env.AWS_PROFILE = serverless.processedInput.options['aws-profile'];
}
const ssm = new AWS.SSM({ apiVersion: '2014-11-06' });
# Get SSM params details from serverless.yml
const ssmKey = serverless.service.custom.ssmKeys.percentageNumericThreshold.path;
const decryption = serverless.service.custom.ssmKeys.percentageNumericThreshold.decryption;
# Get SSM parameter value and convert it to int
const result = await ssm.getParameter({
Name: ssmKey,
WithDecryption: decryption,
}).promise();
if (result.Parameter) {
return parseInt(result.Parameter.Value, 10);
}
throw new Error(`Failed to read SSM parameter for key ${ssmKey}`);
};
Now in serverless.yml define following values:
custom:
ssmKeys:
percentageNumericThreshold:
path: '/path/to/ssm/key'
decryption: false
And in the place, where you want to have your number value from SSM, simply invoke it this way:
FilterPolicy:
percentage:
- numeric:
- '>='
- ${file(./ssmToNumber.js)}
How it works?
Serverless Framework can start any JavaScript/TypeScript file during the build and put it's output into serverless.yml file.
And this is exactly what we do here. We define our ssmToNumber.js script, that simply reads SSM parameter from SSM and then converts it to integer and returns the value.
It knows which SSM path to use, thanks to the custom.ssmKeys section in serverless.yml file.
Of course if you want to customise ssmToNumber.js file, to make it more verbose and fault tolerant, then you simply need to edit the JavaScript file.
Another, more elegant way
It requires more work. Check out the official example for Serverless with Typescript.
As you can see, it's possible to use serverless.ts or serverless.js instead of YAML file.
It would require from you some work about refactoring existing YML file, but writing helper function, that would convert it to number, is very easy and elegant way to achieve your use case.
It have some downsides, like problems with including directly other YAML templates, but still you can define your CloudFormation YAML templates in separate files and then simply import those using import call from TS/JS code.
I'm facing a problem when I'm trying to get the data shallowed from Firebase Database.
Here is the node that I'm trying to shallow:
and here is the response after I've called the Rest API:
{
"-KzPO6veQ986jxq6jlHK": true,
"-KzQeHwhWQozjb_bqRXl": true,
"-KzPLigBLTL5w8wcSsnA": true,
"-KzPOMEB7C7GawO_7o8b": true,
"-KzVcYwM3IMnaF8DbhOe": true,
"-Kz5XqskF0zLLogT_sl-": true
}
You can try yourself by open the following URL: https://librostic-fa290.firebaseio.com/books.json?&shallow=true
This is not the response that I'm specting because as you can see it is unordered in a weird way.
Am I getting the correct response?
The REST API Guide explains that query results are unordered:
The REST API Returns Unsorted Results: JSON interpreters do not
enforce any ordering on the result set. While orderBy can be used in
combination with startAt, endAt, limitToFirst, or limitToLast to
return a subset of the data, the returned results will not be sorted.
Therefore, it may be necessary to manually sort the results if
ordering is important.
I am trying to create a two column unique index on the underlying mongodb in a meteor app and having trouble. I can't find anything in the meteor docs. I have tried from the chrome console. I have tried from term and even tried to point mongod at the /db/ dir inside .meteor . I have tried
Collection.ensureIndex({first_id: 1, another_id: 1}, {unique: true}); variations.
I want to be able to prevent duplicate entries on a meteor app mongo collection.
Wondering if anyone has figured this out?
I answered my own question, what a noob.
I figured it out.
Start meteor server
Open 2nd terminal and type meteor mongo
Then create your index...for example I did these for records of thumbsup and thumbsdown type system.
db.thumbsup.ensureIndex({item_id: 1, user_id: 1}, {unique: true})
db.thumbsdown.ensureIndex({item_id: 1, user_id: 1}, {unique: true})
Now, just gotta figure out a bootstrap install setup that creates these when pushed to prod instead of manually.
Collection._ensureIndex(index, options)
Searching inside Meteor source code, I found a bind to ensureIndex called _ensureIndex.
For single-key basic indexes you can follow the example of packages/accounts-base/accounts_server.js that forces unique usernames on Meteor:
Meteor.users._ensureIndex('username', {unique: 1, sparse: 1});
For multi-key "compound" indexes:
Collection._ensureIndex({first_id:1, another_id:1}, {unique: 1});
The previous code, when placed on the server side, ensures that indexes are set.
Warning
Notice _ensureIndex implementation warning:
We'll actually design an index API later. For now, we just pass
through to Mongo's, but make it synchronous.
According to the docs "Minimongo currently doesn't have indexes. This will come soon." And looking at the methods available on a Collection, there's no ensureIndex.
You can run meteor mongo for a mongo shell and enable the indexes server-side, but the Collection object still won't know about them. So the app will let you add multiple instances to the Collection cache, while on the server-side the additional inserts will fail silently (errors get written to the output). When you do a hard page refresh, the app will re-sync with server
So your best bet for now is probably to do something like:
var count = MyCollection.find({first_id: 'foo', another_id: 'bar'}).count()
if (count === 0)
MyCollection.insert({first_id: 'foo', another_id: 'bar'});
Which is obviously not ideal, but works ok. You could also enable indexing in mongodb on the server, so even in the case of a race condition you won't actually get duplicate records.
The Smartpackage aldeed:collection2 supports unique indices, as well as schema-validation. Validation will both occure on server and client (reactivly), so you can react on errors on the client.
Actually why not use upsert on the server with a Meteor.method and you could also send also track it with a ts:
// Server Only
Meteor.methods({
add_only_once = function(id1,id2){
SomeCollection.update(
{first_id:id1,another_id:id2},{$set:{ts:Date.now()}},{upsert:True});
}
});
// Client
Meteor.call('add_only_once',doc1._id, doc2._id);
// actual code running on server
if(Meteor.is_server) {
Meteor.methods({
register_code: function (key,monitor) {
Codes.update({key:key},{$set:{ts:Date.now()}},{upsert:true});
}
...