I've been trying to implement a artifact cleanup process based on https://www.jfrog.com/blog/advanced-cleanup-using-artifactory-query-language-aql/ ... but found that the AQL is really slow. i've seen queries still running after 24 hours with the server pinned at 100% cpu.
Using simple curl :
time curl -vv -u user:password -d "${query}" "http://server:8081/artifactory/api/search/aql"
I've measured :
items.find({"repo" : "ep-snapshots", "#milestone.keep": {"$ne" : "true"}, "#milestone.complete": {"$ne" : "true"}, "created": {"$before" : "3d"}}).limit(100)
33 minutes
items.find({"repo" : "ep-snapshots", "#milestone.keep": {"$ne" : "true"}, "#milestone.complete": {"$ne" : "true"}}).limit(100)
42 minutes
items.find({"repo" : "ep-snapshots", "#milestone.keep": {"$ne" : "true"}, "#milestone.complete": "false"}).limit(100)
574 minutes
Are there any techniques to investigate the performance of AQL ? Anything obviously broken with the above ?
Thanks,
Pete
Related
I'm trying to integrate Kibana with my OpenSearch ( is it possible ? ). Unfortunately I get a version error.
Is there any way to use opensearch in kibana?
These are the versions I get via curl:
curl -k -u "admin:PASSWORD" "https://IP:9200/"
{
"name" : "node-1",
"cluster_name" : "cluster",
"cluster_uuid" : "he6gqhl2S-6dlVv6dyPOEA",
"version" : {
"number" : "7.10.2",
"build_type" : "rpm",
"build_hash" : "e505b10357c03ae8d26d675172402f2f2144ef0f",
"build_date" : "2022-01-14T03:38:06.881862Z",
"build_snapshot" : false,
"lucene_version" : "8.10.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}
They are very tightly coupled and aren't really supposed to work together, so I doubt they will (or it would be very brittle).
If you want to have Kibana, why not Elasticsearch under the hood?
I'm trying to configure xpack for elasticsearch/kibana, I've activated the trial license for elasticsearch, configured xpack for kibana/elasticsearch and also I've generated ca.crt, node1-elk.crt, node1-elk.key and also kibana.key , kibana.crt and if I'm testing with curl towards the elasticsearch using the kibana user and password and also the ca.crt it's working like a charm, if I'm trying to access kibana from the GUI says that the "Server is not ready yet" and the logs show that " unable to verify the first certificate" :
{"type":"log","#timestamp":"2021-11-16T04:41:09-05:00","tags":["error","savedobjects-service"],"pid":13250,"message":"Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate"}
My configuration:
kibana.yml
server.name: "my-kibana"
server.host: "0.0.0.0"
elasticsearch.hosts: ["https://0.0.0.0:9200"]
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/kibana.crt
server.ssl.key: /etc/kibana/certs/kibana.key
server.ssl.certificateAuthorities: ["/etc/kibana/certs/ca.crt"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "kibana"
elasticsearch.yml
node.name: node1
network.host: 0.0.0.0
discovery.seed_hosts: [ "0.0.0.0" ]
cluster.initial_master_nodes: ["node1"]
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/certs/node1.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/node1.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/node1.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/node1.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
curl testing:
[root#localhost kibana]# curl -XGET https://0.0.0.0:9200/_cat/nodes?v -u kibana_system:kibana --cacert /etc/elasticsearch/certs/ca.crt
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.100.102 23 97 3 0.00 0.02 0.08 cdfhilmrstw * node1
I don't know what to do more here:
[root#localhost kibana]# curl -XGET https://0.0.0.0:9200/_license -u kibana_system:kibana --cacert /etc/elasticsearch/certs/ca.crt
{
"license" : {
"status" : "active",
"uid" : "872f0ad0-723e-43c8-b346-f43e2707d3de",
"type" : "trial",
"issue_date" : "2021-11-08T18:26:15.422Z",
"issue_date_in_millis" : 1636395975422,
"expiry_date" : "2021-12-08T18:26:15.422Z",
"expiry_date_in_millis" : 1638987975422,
"max_nodes" : 1000,
"issued_to" : "elasticsearch",
"issuer" : "elasticsearch",
"start_date_in_millis" : -1
}
}
Thank you for your help
When I try to use httpoison to query an elasticsearch server like
iex(1)> HTTPoison.get "http://localhost:9200"
I get
{:error, %HTTPoison.Error{id: nil, reason: :econnrefused}}.
If I do
curl -XGET "http://localhost:9200"
I get
{
"name" : "es01",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "Wik-EpMkQ8ummJE6ctNAOg",
"version" : {
"number" : "7.0.1",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "e4efcb5",
"build_date" : "2019-04-29T12:56:03.145736Z",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.7.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Does anyone know what this behavior is due to and how to fix it?
P.S.: Changing localhost to 127.0.0.1 does not solve the problem.
Here's my setup:
elasticsearch Version: 7.0.1
{:httpoison, "~> 1.5"} #=> mix.lock shows version 1.5.1 was installed
curl results:
$ curl -XGET "http://localhost:9200"
{
"name" : "My-MacBook-Pro.local",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "vEFl3B5TTYaBxPhQFuXPyQ",
"version" : {
"number" : "7.0.1",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "e4efcb5",
"build_date" : "2019-04-29T12:56:03.145736Z",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.7.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
HTTPoison results:
$ iex -S mix
Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]
===> Compiling parse_trans
===> Compiling mimerl
===> Compiling metrics
===> Compiling unicode_util_compat
===> Compiling idna
==> ssl_verify_fun
Compiling 7 files (.erl)
Generated ssl_verify_fun app
===> Compiling certifi
===> Compiling hackney
==> httpoison
Compiling 3 files (.ex)
Generated httpoison app
==> hello
Compiling 15 files (.ex)
Generated hello app
Interactive Elixir (1.6.6) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> HTTPoison.get "http://localhost:9200"
{:ok,
%HTTPoison.Response{
body: "{\n \"name\" : \"My-MacBook-Pro.local\",\n \"cluster_name\" :
\"elasticsearch\",\n \"cluster_uuid\" : \"vEFl3B5TTYaBxPhQFuXPyQ\",\n
\"version\" : {\n \"number\" : \"7.0.1\",\n \"build_flavor\" :
\"default\",\n \"build_type\" : \"tar\",\n \"build_hash\" :
\"e4efcb5\",\n \"build_date\" : \"2019-04-29T12:56:03.145736Z\",\n
\"build_snapshot\" : false,\n \"lucene_version\" : \"8.0.0\",\n
\"minimum_wire_compatibility_version\" : \"6.7.0\",\n
\"minimum_index_compatibility_version\" : \"6.0.0-beta1\"\n },\n
\"tagline\" : \"You Know, for Search\"\n}\n",
headers: [
{"content-type", "application/json; charset=UTF-8"},
{"content-length", "522"}
],
request: %HTTPoison.Request{
body: "",
headers: [],
method: :get,
options: [],
params: %{},
url: "http://localhost:9200"
},
request_url: "http://localhost:9200",
status_code: 200
}}
iex(2)>
Next, I stopped the elasticsearch server, then I ran the HTTPoison request again:
ex(2)> HTTPoison.get "http://localhost:9200"
{:error, %HTTPoison.Error{id: nil, reason: :econnrefused}}
I got similar results for the curl request:
$ curl -XGET "http://localhost:9200"
curl: (7) Failed to connect to localhost port 9200: Connection refused
What happens if you issue two curl requests in a row? Do they both succeed? Try issuing the HTTPoison request first, then the curl request. Which one fails? Try the reverse order. Same results?
I'm almost positive you have the same problem that I did. Check to make sure you are not forcing ipv6 in your /etc/hosts file.
If you have something like this:
::1 localhost
...get rid of it and Httpoison should work again
I'm working with Symfony 4, FosElasticaBundle and Kibana, and I have approximatly 600K items to index in ElasticSearch.
I'm using the command fos:elastica:populate, and when I run this I have something like :
Resetting myindex
0/546097 [>---------------------------] 0%
So, 546097 is the exact number of items I need to index, but after the indexation, when I try to get all items in ElasticSearch with curl like :
curl -XGET "http://localhost:9200/myindex/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query" : {
"match_all" : {}
}
}'
I have count=420,603
When I try to check the index with Kibana, I can see the same count 420,603.
I did the indexation several times (reset + index), and it's always the same count of 420603....
So my question is : What is this limit ? Why the indexation stop to 420603 items exactly ?
Thanks !
I was trying to print a document for one of my games but the page viewer couldn't see the printer so I checked the print spooler service
C:\WINDOWS\system32>sc qc spooler
[SC] QueryServiceConfig SUCCESS
SERVICE_NAME: spooler
TYPE : 110 WIN32_OWN_PROCESS (interactive)
START_TYPE : 2 AUTO_START
ERROR_CONTROL : 1 NORMAL
BINARY_PATH_NAME : C:\WINDOWS\System32\spoolsv.exe
LOAD_ORDER_GROUP : SpoolerGroup
TAG : 0
DISPLAY_NAME : Print Spooler
DEPENDENCIES : RPCSS
: http
SERVICE_START_NAME : LocalSystem
C:\WINDOWS\system32>sc query spooler
SERVICE_NAME: spooler
TYPE : 110 WIN32_OWN_PROCESS (interactive)
STATE : 1 STOPPED
WIN32_EXIT_CODE : 1068 (0x42c)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0
C:\WINDOWS\system32>
And tried to start it, then this happened
C:\WINDOWS\system32>net start spooler
System error 1068 has occurred.
The dependency service or group failed to start.
C:\WINDOWS\system32>
Ok so I checked the dependencies
C:\WINDOWS\system32>sc qc rpcss
[SC] QueryServiceConfig SUCCESS
SERVICE_NAME: rpcss
TYPE : 20 WIN32_SHARE_PROCESS
START_TYPE : 2 AUTO_START
ERROR_CONTROL : 1 NORMAL
BINARY_PATH_NAME : C:\WINDOWS\system32\svchost.exe -k rpcss
LOAD_ORDER_GROUP : COM Infrastructure
TAG : 0
DISPLAY_NAME : Remote Procedure Call (RPC)
DEPENDENCIES : RpcEptMapper
: DcomLaunch
SERVICE_START_NAME : NT AUTHORITY\NetworkService
C:\WINDOWS\system32>sc query rpcss
SERVICE_NAME: rpcss
TYPE : 20 WIN32_SHARE_PROCESS
STATE : 4 RUNNING
(NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0
C:\WINDOWS\system32>
Ok RPCSS is good, next one
C:\WINDOWS\system32>sc qc http && sc query http
[SC] QueryServiceConfig SUCCESS
SERVICE_NAME: http
TYPE : 1 KERNEL_DRIVER
START_TYPE : 3 DEMAND_START
ERROR_CONTROL : 1 NORMAL
BINARY_PATH_NAME : system32\drivers\HTTP.sys
LOAD_ORDER_GROUP :
TAG : 0
DISPLAY_NAME : HTTP Service
DEPENDENCIES :
SERVICE_START_NAME :
SERVICE_NAME: http
TYPE : 1 KERNEL_DRIVER
STATE : 1 STOPPED
WIN32_EXIT_CODE : 1009 (0x3f1)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0
C:\WINDOWS\system32>
OK seeing it stopped I tried to start it again
C:\WINDOWS\system32>net start http
System error 1009 has occurred.
The configuration registry database is corrupt.
C:\WINDOWS\system32>
So I run SFC to try and fix this BUT...
C:\WINDOWS\system32>sfc /scannow
Beginning system scan. This process will take some time.
Beginning verification phase of system scan.
Verification 100% complete.
Windows Resource Protection did not find any integrity violations.
C:\WINDOWS\system32>
A fat lot of help this is, it can't even fix something so inherently wrong...
So this is where I ask the community for help, I don't know what to do past this point. Help is very much appreciated.
In my case, I had a sub-key under
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters\SslBindingInfo that was missing information. i.e. all the keys such as 0.0.0.0:40015 have values like "AppId","DefaultFlags", etc.
I had one that had no values under this key. I deleted that "empty" key and HTTP was able to start up.