Using the help of elastic functionBeat I'm logging the cloud-watch to kibana. This is the below example:
"message":
"2019-10-23T14:39:45.360Z\tcef55d0c-eg77-4e7a-9578-155b2260f103\tINFO\tjhon { status: 'success', "reason": "payment is not done" }\n",
The message key is holding a plain text, I want to transform this key over the fly before it reached the kibana.
Related
I'm trying to create a stack from a template using the Heat API. I'm using the API reference here as a guide. I have the following json in a file called single-server-template.json:
{
"stack_name": "api-test",
"template": {
"heat_template_version": "rocky",
"description": "Testing Heat API\n",
"resources": {
"server1": {
"type": "OS::Nova::Server",
"properties": {
"name": "Server1",
"image": "Ubuntu 22.04 (Jammy)",
"flavor": "alt.st1.small",
"key_name": "my_key",
"networks": "Internal"
}
}
}
}
}
I'm sending it with this: curl -X POST -H "X-Auth-Token:$OS_TOKEN" -d #single-server-template.json https://$OS_HOST_URL:8004/v1/$OS_PROJECT_ID/stacks
I've been at it for hours but no matter what I send I get a 400 error:
{"code": 400, "title": "Bad Request", "explanation": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "error": {"type": "HTTPBadRequest", "traceback": null, "message": "The server could not comply with the request since it is either malformed or otherwise incorrect."}}
Things I've tried:
confirmed my environment vars are filled in correctly before sending
confirmed that the template itself is valid by spinning it up from the UI console
confirmed that other endpoints that require auth work as expected
screaming and/or crying
Can anyone confirm that what I've got is correct, or test it against your own openstack instance? The only other thing I can think of is that maybe my openstack provider has an issue with their API
I am sending logs to an azure eventhub with Serilog (using WriteTo.AzureEventHub(eventHubClient)), after that I am running a filebeat process with the azure module enabled, so I send these logs to elasticsearch to be able to explore them with Kibana.
The problem I have is that all the information goes to the field "message", I would need to separate the information of my logs in different fields to be able to do good queries.
The way I found was create an ingest pipeline in Kibana and through a grok processor I separate the fields inside the "meessage" and generate multiple fields. In the filebeat.yml I set the pipeline name, but nothing happen, it seems the pipeline is not working.
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
pipeline: "filebeat-otc"
Does anybody knows what I am missing? THANKS in advance.
EDITION. I will add an example of my pipeline and my data. In the simulation is working properly:
POST _ingest/pipeline/_simulate
{
"pipeline": {
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{TIME:timestamp}\\s%{LOGLEVEL}\\s{[a-zA-Z]*:%{UUID:CorrelationID},[a-zA-Z]*:%{TEXT:OperationTittle},[a-zA-Z]*:%{TEXT:OriginSystemName},[a-zA-Z]*:%{TEXT:TargetSystemName},[a-zA-Z]*:%{TEXT:OperationProcess},[a-zA-Z]*:%{TEXT:LogMessage},[a-zA-Z]*:%{TEXT:ErrorMessage}}"
],
"pattern_definitions": {
"LOGLEVEL" : "\\[[^\\]]*\\]",
"TEXT" : "[a-zA-Z0-9- ]*"
}
}
}
]
},
"docs": [
{
"_source": {
"message": "15:13:59 [INF] {CorrelationId:83355884-a351-4c8b-af8d-b77c48462f36,OperationTittle:Operation1,OriginSystemName:Fexa,TargetSystemName:Usina,OperationProcess:Testing Log Data,LogMessage:Esto es una buena prueba,ErrorMessage:null}"
}
},
{
"_source": {
"message": "20:13:48 [INF] {CorrelationId:8451ee54-efca-40be-91c8-8c8e18e33f58,OperationTittle:null,OriginSystemName:Fexa,TargetSystemName:Donna,OperationProcess:Testing Log Data,LogMessage:null,ErrorMessage:null}"
}
}
]
}
It seems when you use a module it will create and use an ingest pipeline in elasticsearch, and the pipeline option in the output is ignored.
So my solution was modify the index.final_pipeline. For this, in Kibana I went to Stack Management / Index Management there I found my index, there I went to Edit Settings and set "index.final_pipeline": "the-name-of-my-pipeline".
I hope this helps to anybody.
This was thanks to leandrojmp
I've been trying to get this operator working for some time since switching to airflow 2.0 BigQueryInsertJobOperator.
The error I'm seeing shows there is something missing from our connection, oddly enough this connection works in another DAG where we are using google's api to access google sheets:
export AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT=
"google-cloud-platform://?extra__google_cloud_platform__project=\analytics&extra__google_cloud_platform__keyfile_dict=
{\"type\": \"service_account\", \"project_id\": \"analytics\",
\"private_key_id\": \"${GCLOUD_PRIVATE_KEY_ID}\", \"private_key\": \"${GCLOUD_PRIVATE_KEY}\",
\"client_email\": \"d#lytics.iam.gserviceaccount.com\", \"client_id\": \"12345667\",
\"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",
\"token_uri\": \"https://accounts.google.com/o/oauth2/token\",
\"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",
\"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/x509/d#lytics.iam.gserviceaccount.com\"}"
This is the error I'm seeing:
{
"error": {
"code": 401,
"message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"errors": [
{
"message": "Login Required.",
"domain": "global",
"reason": "required",
"location": "Authorization",
"locationType": "header"
}
],
"status": "UNAUTHENTICATED"
}
}
is there a way I can look up what else might be required in terms of formatting, etc, perhaps a really good example on how to get the correct connection setup for this Operator??
In my logs I'm seeing this error which makes me think perhaps it might not be a credential issue?
File "/usr/local/lib/python3.8/site-packages/google/cloud/_http.py", line 438, in api_request
raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/vice-analytics/jobs?prettyPrint=false: Required parameter is missing
Create a service account json key, which contains all the required info posted in your error message.
https://cloud.google.com/iam/docs/creating-managing-service-account-keys
Then you can paste the json key into the Airflow UI: Admin -> Connections in the json key field and reference this in your dag with: gcp_conn_id="name of connection you created"
Or add the json key as an env variable (on macos):
export GOOGLE_APPLICATION_CREDENTIALS="link to your json key file"
I have run through the Google Prediction API tutorials and documentation for "hello prediction - https://cloud.google.com/prediction/docs/hello_world
However when training the model in the developer console my request fails with the following output:
Request:
POST https://www.googleapis.com/prediction/v1.6/projects/959568262740/trainedmodels?key={YOUR_API_KEY}
{
"id": "language_id",
"storageDataLocation": "http://storage.googleapis.com/2341234/language_id.txt"
}
Response:
400 OK
- Show headers -
{
"error": {
"errors": [
{
"domain": "global",
"reason": "invalid",
"message": "Training data file is empty.",
"locationType": "other",
"location": "id"
}
],
"code": 400,
"message": "Training data file is empty."
}
}
I've implemented Authorize requests using OAuth 2.0: - is there anything else that I should be doing, or that may have changed between Google PRediction API v 1.6 and the tutorial. Any link to any additional tutorial or article on the subject would also be extremely valuable so can debug myself!
Your storageDataLocation is not correct.
When you go to the Overview tab in Google Developers Console, you'll find your Project ID: xxxxx (example) and the bucket you've stored in is called yyyy (example).
Then replace the "storageDataLocation:" "xxxxx/yyyy". Should solve this problem. You need the relative path to Google Storage, and not the absolute web path.
i need to query some data with where clause
as per the API google map engine i have request as below.
https://www.googleapis.com/mapsengine/v1/tables/14538994882799551513-11853667273131550346/features?where=gx_id%3D900
for the above URL it says BAD REQUEST where=gx_id=900
and the response is
{
"error": {
"errors": [
{
"domain": "global",
"reason": "invalid",
"message": "The value is invalid.",
"locationType": "parameter",
"location": "query"
}
],
"code": 400,
"message": "The value is invalid."
}
}
Please suggest me what is wrong in this URL
You use a Number in your query, but the particular gx_id seems to be of type String.
Enclose the Number with single-quotes:
https://www.googleapis.com/mapsengine/v1/tables/14538994882799551513-11853667273131550346/features?where=gx_id%3D%27900%27
Also, you don't have to worry about creating and keeping track of your ID's. the API says you need to include gx_id only to make sure that no Feature is sent twice.
Therefore, you can just throw in a string representing your system's current time for example.