I got a brand new empty installation of Elastic Search 6.1.1 running on port 9200 of my CentOS7 host. I have this "es.json" file with data sample ready to be inserted in the ES.
I also have a log file in which each line is a json chunk. I could use both to populate my ES database.
How do I insert this and other data to ES?
I didnt find good explanations on the documentation and on the internet. for some reason the docs does not make it very clear
Use the --data-binary flag in curl to bulk import from a JSON file.
curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/{index}/{type}/_bulk?pretty' --data-binary #es.json
Data can be posted to one of the endpoints - /_bulk, /{index}/_bulk or {index}/{type}/_bulk. When {index} or {index}/{type} is provided, they will be used by default on bulk items that don’t provide them explicitly.
Content-Type: application/x-ndjson stands for Newline delimited JSON.
Before importing the JSON file, you might want to define mappings yourself or let Elasticsearch generate mappings dynamically during import. If you don't want Elasticsearch to generate mappings dynamically during import, refer to this doc to define mappings yourself.
References:
https://www.elastic.co/guide/en/elasticsearch/reference/6.1/docs-bulk.html
https://www.elastic.co/guide/en/kibana/6.1/tutorial-load-dataset.html
You can use elasticsearch_loader for loading json files into elasticsearch (2.X, 5.X, 6.X).
You can download it with pip:
pip install elasticsearch-loader
And then you will be able to load json files into elasticsearch by issuing:
elasticsearch_loader --index incidents --type incident json file1.json file2.json
Disclaimer: I'm the author of elasticsearch_loader
Related
I am passing the following
- pipe: JfrogDev/artifactory-docker:0.2.12
variables:
ARTIFACTORY_URL: "${JFROG_URL}"
ARTIFACTORY_USER: "${JFROG_USER}"
ARTIFACTORY_PASSWORD: "${JFROG_PASSWORD}"
DOCKER_IMAGE_TAG: "xxxx/web-interface:${BITBUCKET_BUILD_NUMBER}"
DOCKER_TARGET_REPO: "local-containers"
And I am getting back the following
Status: Downloaded newer image for jfrog-int-docker-open-docker.bintray.io/artifactory-docker:0.2.12
INFO: Starting pipe execution...
jfrog rt config --url=$JFROG_URL --user=$JFROG_USER --password=$JFROG_PASSWORD --interactive=false
[Error] Wrong number of arguments. You can read the documentation at https://www.jfrog.com/confluence/display/CLI/JFrog+CLI
It can’t find anything wrong based on the documentation, it looks like the pipeline makes a cli call and that call is failing
It looks like there is an issue when trying to config the server without the mandatory server-id argument:
https://bitbucket.org/JfrogDev/artifactory-docker/src/3b32b5d01d31c07acadb2a0d29a240110f88d59d/pipe/pipe.sh#lines-65
Instead, you can use the new jfrog-setup-cli pipe. This pipe downloads and configures the JFrog CLI.
For example, in your case:
- pipe: jfrog/jfrog-setup-cli:1.0.0
- source ./jfrog-setup-cli.sh
- jfrog rt docker-push "xxxx/web-interface:${BITBUCKET_BUILD_NUMBER}" local-containers
It requires to populate one secured environment variable prefixed by JF_ARTIFACTORY_ (i.e. JF_ARTIFACTORY_MYSERVER) with a JFrog CLI server token. This token created locally in your machine using the jfrog rt config export command.
Read more about this pipe here.
Read more about the jfrog rt config export command here.
Hello I want to use the automatic deploymen on bitbucket to the galaxy server with a deployment token.
For this reason I am creating a deployment token that is comitted in the repository.
https://galaxy-guide.meteor.com/deploy-guide.html#deployment-token
To strenghten the security I would like to use Repository variables in bitbucket pipelines:
https://confluence.atlassian.com/bitbucket/environment-variables-794502608.html
And to store the deployment token of meteor in the variables instead in file.
For the deployment we use in the command:
METEOR_SESSION_FILE=deployment_token.json
And my question is - Is there any way so that I use some variable(string) where the token is used like
METEOR_SESSION_DEPLOYMENT_TOKEN=$METEOR_TOKEN
instead to call it from a file?
Some research, after having the same problem, brought me to this article, which simply solves the problem that you can't feed meteor just the json in an env var in the following simple way:
By adding the json file content as an env var and then echoes it out into a file on deploy.
echo $METEOR_TOKEN_FILE > deploy_token.json
METEOR_SESSION_FILE=deploy_token.json
Thanks to this article I figured it out.
Save json settings as env variable and then in deployment procesS:
echo $DEPLOY_SESSION_FILE > deployment_token.json
METEOR_SESSION_FILE=deployment_token.json DEPLOY_HOSTNAME=galaxy.meteor.com meteor deploy --allow-superuser myApp-staging.meteorapp.com --settings config/staging/settings.json --owner username
Is there a way to delete GAVs through a REST API in Nexus 3? From various
google searches it appears that this capability existed in Nexus 2, but not in
Nexus 3 yet. Is that true?
I tried the following with my current Nexus installation, which is OSS 3.2.1-01:
I was trying to delete GAV:
groupId = org.mycompany.myproject
artifactId = myartifact
version = 1.0.0
$ curl --request DELETE --user "USERNAME:PASSWORD" --write-out '%{http_code}\n' http://my-server:8081/service/local/repositories/my-repo/content/org.mycompany.myproject/myartifact/1.0.0
This gave me a 405.
I also looked at the release notes for 3.3 through 3.5 and nothing jumped out
that REST API support was added.
I also looked into
https://help.sonatype.com/display/NXRM3/REST+and+Integration+API. I downloaded
the nexus-book-examples and downloaded several of the Javadocs (nexus-core,
nexus-repository, nexus-common, nexus-script, nexus-commands, nexus-selector)
for version 3.2.1-01 and started to look through the code. It was not clear
where to start with a simple program to delete GAVs.
Am I correct that you cannot delete GAVs through the REST API in Nexus 3? Is
there a plan to support this in a future Nexus 3 release? Is there a way to do
what I want to do by creating a Groovy script using the code referenced by the
REST+and+Integration+API link above? Is there some sample code which will help
bootstrap me to using the above code (either 3.2.1-01, or a newer version of
Nexus).
Thanks.
You might take a look at our Beta REST API in Nexus Repository 3. Upgrade to a version greater than 3.3, preferably to 3.5 (just so you are using latest and greatest) and navigate to:
http://nexushostname:nexusport/swagger-ui/
Since the REST API is currently Beta we have yet to publish documentation or fanfare around it while we let people experiment with it and give us feedback.
You should see endpoints for deleting components and assets. You will likely want to use the component delete, so that it will clean up all associated assets.
Let me know your mileage!
According to the documentation, you can delete an ASSET (individual file) or a COMPONENT (a set of files, like jar+md5+sha1+pom.xml representing an artifact) only if you know the assetId or the componentId
https://help.sonatype.com/repomanager3/rest-and-integration-api/components-api
https://help.sonatype.com/repomanager3/rest-and-integration-api/assets-api
So you should issue a separate search call passing the GAV and finding out the componentId, then use the componentId to delete in a second call.
However I see here https://issues.sonatype.org/browse/NEXUS-11266 and here
https://issues.sonatype.org/browse/NEXUS-11881 that people can delete an asset just by specifying the entire path... I have tried with
curl -u admin:admin123 -X "DELETE" -w "%{http_code}" http://localhost:8081/repository/deployments/org/apache/commons/commons-compress/1.18/commons-compress-1.18.jar
and it gives me a HTTP 204 (no content). In my case "deployments" is a hosted repository. I have tried the same command on "central" (aproxy repo) and I get a 405.
But if I try to download the whole component (including pom, sha1 etc) with
curl -u admin:admin123 -X "DELETE" -w "%{http_code}" http://localhost:8081/repository/deployments/org/apache/commons/commons-compress/1.18/
I get a HTTP 404.
I know, it's painful and in Nexus2 it was much easier.
I want to make a simple HTTP-Request in JMeter. I'm using some variables of a CSV I just created. In this CSV are names like "Müller" or "Böhm".
So when I run the test I notice that Jmeter convert "Müller" into "Müller".
I create my CSV in Notepad++ (UTF-8 without BOM).
Furthermore I change Jmeter.properties:
sampleresult.default.encoding=UTF-8
An other idea was to use the post-Benshellsampler:
prev.setDataEncoding("UTF-8");
and
request.setCharacterEncoding("UTF-8");
None of these worked for me.
Errorreport: http://pastebin.com/EydAjfdm
JMeter: 2.13 |
Plugins: Webdriver, Standard, Extra, ExtraLibs (1.3.1)
My expectation is that you need to change encoding of JVM. Add the following line:
log.info(System.getProperty("file.encoding") + " <--------------------");
to any Beanshell Test Element and look into jmeter.log file. If you see something other than UTF-8 - you have a problem which needs to be fixed
2015/12/15 19:05:37 INFO - jmeter.util.BeanShellTestElement: UTF-8 <--------------------
Remove all Beanshell tweaks (by the way, the one with request won't work)
Add the following line to system.properties file (lives under /bin folder of your JMeter installation)
file.encoding=UTF-8
Restart JMeter and that should be it.
See:
List of Java system properties in general and file.encoding bit in particular
Apache JMeter Properties Customization Guide - to learn more about different JMeter properties types and ways of working with them
Background:
(1) I use mysql workbench to design my database.
(2) Then I use "Synchronize Model" tool in mysql workbench to apply my modification to mysql database.
(3) Finally I use symfony doctrine:build-schema command to generate schema.yml according to database(For some legacy reason, I have to use Symfony 1.4).
Here comes the problem: I wanna add actAs: { Timestampable: ~ } to schema.yml, but symfony doctrine:build-schema command cannot do that.
The solution I can imagine is to write a shell to edit schema.yml. Clearly this is not an elegant solution.
Any suggestion is greatly appreciated.
1) What errors is given out by command symfony doctrine:build-schema
2) Try such code in schema.yml
Something:
actAs:
Timestampable:
columns:
...
I use MySQL Workbench Schema Exporter (an open source PHP library project) to save time and complete all database model definition tasks in modeling process. Review this Installation Tutorial.
With MySQL Workbench Schema Exporter you can use Table Comments to define table behavior. To add a timestampable behavior to table definition put this in your comments:
{MwbExporter:actAs}actAs: [Timestampable]{/MwbExporter:actAs}
Then export your model (*.mwb file) to YAML (*.yml file) using MySQL Workbench Schema Exporter and place it in ./config/schema/ folder in order to create an sql script and then load it to a database already configured with symfony. It's easy! just like this:
Export model to YAML
php bin/mysql-workbench-schema-export your_model.mwb
Copy schema.yml resulting file to ./config/schema folder location
Create an sql script
php symfony doctrine:build-sql
Now build your model physically with:
php symfony doctrine:build-model