The airflow documentation states:
Airflow exposes an experimental Rest API. It is available through the webserver. Endpoints are available at /api/experimental/. Please note that we expect the endpoint definitions to change.
https://airflow.apache.org/api.html#experimental-rest-api
However it doesn't state in which version the API appears. We are running Airflow v1.8.0
But whenever I browse to /api/ or /api/experimental/ I get a 404 and the spinning circles.
I tried curling the same URLs but that only confirmed the same, /api/ gives me a 404:
$ curl -I -L -s http://${AIRFLOWIP}:8080/admin/ | grep HTTP
HTTP/1.1 200 OK
$ curl -I -L -s http://${AIRFLOWIP}:8080/api/ | grep HTTP
HTTP/1.1 404 NOT FOUND
We have this setting in airflow.cfg:
[api]
# How to authenticate users of the API
auth_backend = airflow.api.auth.backend.default
I don't know whether the API is only available in a later version of Airflow or if we have stood it up incorrectly.
Can someone let me know in which version of airflow we can find the experimental API?
The first experimental endpoints were added in 1.8.0, with a few more endpoints added in following releases. There is no endpoint for the root paths of /api/ and /api/experimental/ on any version, so those curls are not expected to work. However, there is a /api/experimental/test/ endpoint which you can hit to confirm the API is available.
If you're going to be using the experimental API, I think the code is the best reference at the moment.
The Airflow API is no more at the experimental phase.
Stable version here Airflow REST API.
Related
Background
We run a kubernetes cluster that handles several php/lumen microservices. We started seeing the app php-fpm/nginx reporting 499 status code in it's logs, and it seems to correspond with the client getting a blank response (curl returns curl: (52) Empty reply from server) while the applications log 499.
10.10.x.x - - [09/Mar/2020:18:26:46 +0000] "POST /some/path/ HTTP/1.1" 499 0 "-" "curl/7.65.3"
My understanding is nginx will return the 499 code when the client socket is no longer open/available to return the content to. In this situation that appears to mean something before the nginx/application layer is terminating this connection. Our configuration currently is:
ELB -> k8s nginx ingress -> application
So my thoughts are either ELB or ingress since the application is the one who has no socket left to return to. So i started hitting ingress logs...
Potential core problem?
While looking the the ingress logs i'm seeing quite a few of these:
2020/03/06 17:40:01 [crit] 11006#11006: ngx_slab_alloc() failed: no memory in vhost_traffic_status_zone "vhost_traffic_status"
Potential Solution
I imagine if i gave vhost_traffic_status_zone some more memory at least that error would go away and on to finding the next error.. but I can't seem to find any configmap value or annotation that would allow me to control this. I've checked the docs:
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/
Thanks in advance for any insight / suggestions / documentation I might be missing!
here is the standard way to look up how to modify the nginx.conf in the ingress controller. After that, I'll link in some info on suggestions on how much memory you should give the zone.
First start by getting the ingress controller version by checking the image version on the deploy
kubectl -n <namespace> get deployment <deployment-name> | grep 'image:'
From there, you can retrieve the code for your version from the following URL. In the following, I will be using version 0.10.2.
https://github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.10.2
The nginx.conf template can be found at rootfs/etc/nginx/template/nginx.tmpl in the code or /etc/nginx/template/nginx.tmpl on a pod. This can be grepped for the line of interest. I the example case, we find the following line in the nginx.tmpl
vhost_traffic_status_zone shared:vhost_traffic_status:{{ $cfg.VtsStatusZoneSize }};
This gives us the config variable to look up in the code. Our next grep for VtsStatusZoneSize leads us to the lines in internal/ingress/controller/config/config.go
// Description: Sets parameters for a shared memory zone that will keep states for various keys. The cache is shared between all worker processe
// https://github.com/vozlt/nginx-module-vts#vhost_traffic_status_zone
// Default value is 10m
VtsStatusZoneSize string `json:"vts-status-zone-size,omitempty"
This gives us the key "vts-status-zone-size" to be added to the configmap "ingress-nginx-ingress-controller". The current value can be found in the rendered nginx.conf template on a pod at /etc/nginx/nginx.conf.
When it comes to what size you may want to set the zone, there are the docs here that suggest setting it to 2*usedSize:
If the message("ngx_slab_alloc() failed: no memory in vhost_traffic_status_zone") printed in error_log, increase to more than (usedSize * 2).
https://github.com/vozlt/nginx-module-vts#vhost_traffic_status_zone
"usedSize" can be found by hitting the stats page for nginx or through the JSON endpoint. Here is the request to get the JSON version of the stats and if you have jq the path to the value: curl http://localhost:18080/nginx_status/format/json 2> /dev/null | jq .sharedZones.usedSize
Hope this helps.
My institute recently installed a new proxy server for our network. I am trying to configure my Cygwin environment to be able to run wget and download data from a remote repository.
Browsing the internet I have found two different solutions to my problem, but no one of them seem to work in my case.
The first one I tried was to follow these instructions, so in Cygwin:
cd /cygdrive/c/cygwin64/etc/
nano wgetrc
at the end of the file, I added:
use_proxy = on
http_proxy=http://username:password#my.proxy.ip:my.port/
https_proxy=https://username:password#my.proxy.ip:my.port/
ftp_proxy=http://username:password#my.proxy.ip:my.port/
(of course, using my user and password)
The second approach was what was suggested by this SO post, so in my Cygwin environment:
export http_proxy=http://username:password#my.proxy.ip:my.port/
export https_proxy=https://username:password#my.proxy.ip:my.port/
export ftp_proxy=http://username:password#my.proxy.ip:my.port/
in both cases, if I try to test my wget, I get the following:
$ wget http://www.google.com
--2020-01-30 12:12:22-- http://www.google.com/
Resolving my.proxy.ip (my.proxy.ip)... 10.1XX.XXX.XX
Connecting to my.proxy.ip (my.proxy.ip)|10.1XX.XXX.XX|:8XXX... connected.
Proxy request sent, awaiting response... 407 Proxy Authentication Required
2020-01-30 12:12:22 ERROR 407: Proxy Authentication Required.
It looks like if my user and password are not ok, but I actually checked them on my browsers and my credentials work just fine.
Any idea on what this could be due to?
This problem was solved thanks to the suggestion of a User of the community AskUbuntu.
Basically, instead of editing the global configuration file wgetrc, I should have created a new .wgetrc with my proxy configuration in my Cygwin home directory.
In summary:
Step 1 - Create a .wgetrc file;
nano ~/.wgetrc
Step 2 - record in this file the proxy info:
use_proxy=on
http_proxy=http://my.proxy.ip:my.port
https_proxy=https://my.proxy.ip:my.port
ftp_proxy=http://my.proxy.ip:my.port
proxy_user=username
proxy_password=password
My client is on a server where I can make only HTTP request to any server. The Tensorflow is hosted in an AWS machine which accepts the only gRPC requests. Looking for some leads to make this communication happen?
EDIT: 12 th June 2018
TF officially releases REST API for serving
https://www.tensorflow.org/serving/api_rest
They use this particular example: half_plus_three
Server:
$ tensorflow_model_server --rest_api_port=8501 \
--model_name=half_plus_three \
--model_base_path=$(pwd)/serving/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_three/
Client:
$ curl -d '{"instances": [1.0,2.0,5.0]}' -X POST http://localhost:8501/v1/models/half_plus_three:predict
{
"predictions": [3.5, 4.0, 5.5]
}
Which language/platform is your server running?
OTOH, TF is adding REST support now.
I am working on facebook messenger.
Facebook app only accept one url for webhook but ngrock is generating new URL every time. Now I am unable to test my app because of webhook URL changed.
UPDATE May 2020
Serveo is up and running again! No installation, no signup!
All you need to do is to run this:
ssh -R <unique subdomain>:80:<your local host>:<your local port> serveo.net
like
ssh -R youruniquesubdomain:80:localhost:8000 serveo.net
UPDATE January 2020
Since there are some issues with Serveo and localtunnel, I want to share with you another free ssh-based self-hosting service: Localhost.run
Unfortunately, it does not provide unique subdomains but it is ssh-based so you do not have to install additional applications. Still waiting for Serveo coming back.
UPDATE April 2018
I've found Serveo just now! And it is totally incredible!
UPDATE November 2017
Probably, it is not the best option for you but I started using localtunnel instead of ngrok.
An installation and run flow is very simple:
npm install -g localtunnel
lt --port <your localhost port> --subdomain youruniquesubdomain
Then I can go to my http://youruniquesubdomain.localtunnel.me
That's it!
No more free subdomain support from ngrok.....pls have a error as below
Tunnel session failed: Only paid plans may bind custom subdomains.
Failed to bind the custom subdomain 'arvindpattartestfb.ngrok.io' for the account 'arvccccc'.
This account is on the 'Free' plan.
Upgrade to a paid plan at: https://dashboard.ngrok.com/billing/plan
ERR_NGROK_313
You need to set up auth token. You can find it here https://dashboard.ngrok.com/auth. (W̶o̶r̶k̶s̶ ̶w̶i̶t̶h̶ ̶f̶r̶e̶e̶ ̶v̶e̶r̶s̶i̶o̶n̶,̶ ̶n̶o̶ ̶n̶e̶e̶d̶ ̶t̶o̶ ̶p̶a̶y̶ it's now paid feature, see ngrok pricing).
Then you can use it like this:
ngrok http 80 --subdomain yoursubdomain
Neither localtunnel.me or Serveo are working for me right now so I created a temporary solution that works for some use-cases including mine (react-native local development): using the ngrok npm package one can save the generated ngrok url into a json file, and that file can be read for any other app.
First make sure to install ngrok using npm install ngrok then use this node script:
const ngrok = require('ngrok');
const fs = require('fs').promises;
(async function() {
const url = await ngrok.connect(3000);
const api = ngrok.getApi();
let data = await api.get('api/tunnels');
data = JSON.parse(data);
let dict = {'domain': data.tunnels[0].public_url}
await fs.writeFile("config.json", JSON.stringify(dict));
console.log("saved " + data.tunnels[0].public_url);
})();
Then from your app you may read it using code similar or equal to:
const backend = require('./config.json').domain;
For DHIS2 local installation, I did this on the terminal on ubuntu server.
Make sure your web is running on the specified port. Mine was on 8080.
ssh -R dani.serveo.net:80:localhost:8080 serveo.net
The beauty of this serveo.net is you can reuse the same hostname prefix url before serveo.net as many time as you want, even if power goes off or internet diconnection.
Staqlab tunnel is providing domain for free. Its works great but need a binary to be downloaded from there website. Using this service for month without any hassle
In 2022 (almost 2023) pagekite.me works for me.
It is very similar to ngrok, and requires the installation of pagekite.py (and, obviously Python).
After installation click on pagekite.py opens pagekite shell.
Run command: 8080 subdomain.pagekite.me
I noticed that no one mentioned how to have static ngrok urls, which was the main question about.
A way to do it is to edit the ngrok.yml file, which is located at
Linux: "~/.config/ngrok/ngrok.yml"
MacOS (Darwin): "~/Library/Application Support/ngrok/ngrok.yml"
Windows: "%HOMEPATH%\AppData\Local\ngrok\ngrok.yml"
You can have content such as:
version: "2"
authtoken: valid_auth_token
tunnels:
first-app:
addr: 3000
proto: http
hostname: yourfixedngrok_id1.ngrok.io
second-app:
addr: 8000
proto: http
hostname: yourfixedngrok_id2.ngrok.io
This will help you expose multiple ports, and have a persisted url for each of which based on the value you set for hostnames.
After that, you run your ngrok using this command:
ngrok start --all
Though its not a solution but take it as workaround, I had the same problem while testing. What i did is keep the ngrok running with my http port, so my ngrok url is not changing. but I frequently changing and restarting my server for testing and debugging.
What is the proper way to run Kibana 4.5 as service on CentOS 7?
When I run it as ./kibana, I can conenct to it form another machine without any problem. When I run it with systemctl start kibana and check with ps -ef | grep '.*node/bin/node.*src/cli'it looks like running but refuses to connect. And goes down. What can be the problem? Thanks in advance.
Here is content of kibana.service file
[Unit]
Description=no description given
[Service]
Type=simple
User=kibana
Group=root
Environment=CONFIG_PATH=/opt/kibana/config/kibana.yml
ExecStart=/opt/kibana/bin/kibana
Restart=always
[Install]
WantedBy=multi-user.target
I am not that much of a linux expert but i recently installed kibana using yum (https://www.elastic.co/guide/en/kibana/4.5/setup.html#kibana-yum) on a minimal installation of CentOS 7 and did not face any issues whatsoever.
In order to have some debug logs and find out what is wrong in your case, edit the kibana configurations file
/opt/kibana/config/kibana.yml
and set a filename for the logging.dest property.
logging.dest: /var/log/kibana.log
Good luck
Igor,
I noticed a few questions you posted on Kafka so sounds like you need to set up a cluster that can ingest data and pass to Elastic. Kibana would be just user interface.
In my experience, components like ELK, Kafka, Zookeeper, etc should be managed by a watchdog process. I highly recommend looking at something like supervisord. http://supervisord.org/
You should run it as a service and the rest managed by the supervisor. It will guarantee starting components at boot but whats more important restart in case of failure and collecting logs. In case of Kibana, it is a NodeJS app that writes to stdout/stderr so to know what fails, you need to collect them.