How do I force HTTP connections to use IPv4 instead of IPv6 in Python? - python-requests

When attempting to connect to the Binance REST API from a Windows 11 computer, I got the following error:
ccxt.base.errors.ExchangeError: binanceus {"code":-71012,"msg":"IPv6 not supported"}
In this particular example I am using the ccxt Python library, but I assume the problem exists for any HTTP connection / library since the error comes from Binance. Here's sample code that replicates the issue on Windows 11 (works fine on a Linux machine):
import ccxt
exchange = ccxt.binanceus({'apiKey': '<your_key>', 'secret': '<your_secret>'})
print(exchange.fetch_balance())
I speculate that this is happening because by default in Windows 11 IPv6 has a higher priority than IPv4:
> netsh interface ipv6 show prefixpolicies
Querying active state...
Precedence Label Prefix
---------- ----- --------------------------------
50 0 ::1/128
40 1 ::/0
35 4 ::ffff:0:0/96
30 2 2002::/16
5 5 2001::/32
3 13 fc00::/7
1 11 fec0::/10
1 12 3ffe::/16
1 3 ::/96
I suppose one solution is to change this priority so IPv4 is used by default, but I don't want to do this, and I also prefer the code to work everywhere.
Under the hood, the ccxt library uses the Python requests library, and allows the user to construct a custom Session object.
I came up with the following hack/workaround which works for me
Is there a cleaner/better/more robust way to achieve the same thing?
import ccxt
import requests
from requests_toolbelt.adapters import source
import socket
def get_ipv4_session():
local_ipv4 = socket.gethostbyname(socket.gethostname())
src = source.SourceAddressAdapter(local_ipv4)
session = requests.Session()
session.mount("https://", src)
return session
exchange = ccxt.binanceus({
'apiKey': '<your key>',
'secret': '<your secret>',
'session': get_ipv4_session()
})
print(exchange.fetch_balance())
I read the answers here as well but they seem more complicated: Force requests to use IPv4 / IPv6

Related

Translate Python MySQL ssh port forwarding solution to R (dbplyr)

I'm trying to query a MySQL server through an R/Tidyverse/dbplyr workflow. My MySQL access requires configuring SSH and port forwarding.
I have this code working using python (below), but I'm struggling to get started with the SSH/port forwarding equivalent in R. Any pointers to solutions or equivalent R packages appreciated. thanks.
import pymysql
import paramiko
import pandas as pd
from paramiko import SSHClient
from sshtunnel import SSHTunnelForwarder
from os.path import expanduser
pkeyfilepath = '/.ssh/id_ed25519'
home = expanduser('~')
mypkey = paramiko.Ed25519Key.from_private_key_file(home + pkeyfilepath)
sql_hostname = 'mycompany.com'
sql_username = 'me'
sql_password = '***'
sql_main_database = 'my_db'
sql_port = 3306
ssh_host = 'jumphost.mycompany.com'
ssh_user = 'me'
ssh_port = 22
with SSHTunnelForwarder(
(ssh_host, ssh_port),
ssh_username=ssh_user,
ssh_pkey=mypkey,
remote_bind_address=(sql_hostname, sql_port)) as tunnel:
conn = pymysql.connect(host='127.0.0.1', user=sql_username,
passwd=sql_password, db=sql_main_database,
port=tunnel.local_bind_port)
query = '''SELECT VERSION();'''
data = pd.read_sql_query(query, conn)
print(data)
conn.close()
There are several ways to do ssh port forwarding for R. In no particular order:
I forward it externally to R. All of my work is remote, and for one particular client I need access to various instances of SQL Server, Redis, MongoDB, remote filesystems, and a tunnel-hop to another network only accessible from the ssh bastion host. I tend to do work in more than R, so it's important to me that I generalize this. It is not for everybody or every task.
For this, I used a mismash of autossh and my ssh-agent (in KeePass/KeeAgent).
The ssh package does have a function to Create a Tunnel. The premise is that you have already created a "session" to which you can add a forwarding rule(s). When using ssh::ssh_tunnel, it is blocking, meaning you cannot use it in the same R process and continue to work. Demo:
# R session 1
sess <- ssh::ssh_connect("user#remote")
# insert passphrase
ssh::ssh_tunnel(sess, 21433, "otherremote:1433")
# / Waiting for connection on port 21433...
# R session 2
con <- DBI::dbConnect(..., port=21433)
DBI::dbGetQuery(con, "select 1 as n")
# n
# 1 1
This connection will stay alive so long as con is not closed and the remote end does not close it (e.g., activity timeout).
Note: I cannot get the ssh package to use my ssh-agent, so all passwords must be typed in or otherwise passed in not-ideal ways. There are many ways to not have to type it, such as using the keyring package (secure) or envvars, both of which would pass the password to ssh_connect(..., passwd=<>).
The above, but using callr so that you don't need to explicit sessions active (though you will still have another R session.
bgr <- callr::r_bg(function() {
ssh <- ssh::ssh_connect("r2#remote", passwd=keyring::key_get("r2", "remote"))
ssh::ssh_tunnel(ssh, port=21433, "otherremote:1433")
}, supervise = TRUE)
DBI::dbGetQuery(con, "select 1 as n")
# n
# 1 1
### when your work is done
bgr$kill()
If you do this, I strongly recommend the use of supervise=TRUE, which ensures the background R process is killed when this (primary) R session exits. This will reduce the risk of having phantom unused R sessions hanging around; in addition to just clogging up the process tree, if one of these phantom R processes is actively forwarding a port, that means nothing else can forward that port. This allows you to continue working, but you are not longer in control of the process doing the forwarding ... and subsequent attempts to tunnel will fail.
FYI, I generally prefer using keyring::key_get("r2", "remote") for password management in situations like this: (1) it prevents me from having to set that envvar each time I start R ... which will inadvertently store the plain-string password in ~/.Rhistory, if saved; (2) it prevents me from having to set that envvar in the global environment permanently, which is prone to other stupid mistakes I make; and (3) is much better protected since it is using the native credentials of your base OS. Having said that, you can replace the above use of keyring::key_get(..) with Sys.getenv("mypass") in a pinch, or in a case where the code is running on a headless system where a credentials manager is unavailable.
And if you want this to be a little more resilient to timeout disconnects, you can instead use
bgr <- callr::r_bg(function() {
ssh <- ssh::ssh_connect("r2#remote", passwd=keyring::key_get("r2", "remote"))
while (!inherits(try(ssh::ssh_tunnel(ssh, port=21433, "otherremote:1433"), silent=TRUE), "try-error")) Sys.sleep(1)
}, supervise = TRUE)
which will repeatedly make the tunnel so long as the attempt does not error. You may need to experiment with this to get it "perfect".
callr is really just using processx under the hood to start a background R process and allow you to continue working. If you don't want the "weight" of another R process solely to forward ports, you can use processx to start an explicit call to ssh that does everything you need it to do.
proc <- processx::process$new("ssh", c("-L", "21433:otherremote:1433", "r2#remote", "-N"))
### prompts for password
DBI::dbGetQuery(con, "select 1 as n")
# n
# 1 1
### when done
proc$kill()
# [1] TRUE

EUCA 4.4.5 VPCMIDO Instances Terminate at Launch

I have achieved a small test cloud on 3 pieces of hardware. It works fine when in EDGE mode but when I try to configure it for VPCMIDO, new instances begin to launch but then timeout and move to a terminated state. I can also see the instances' initial volume and config data appear in the NC and CC data directories. Below is my system layout and network.json.
HOST 1 : CLC/UFS/WALRUS/MIDO CLUSTER/MIDO GATEWAY/MIDOLMAN AGENT:
em1 (All Services including Mido Cluster): 10.0.0.21
em3 (Target VPCMIDO Adapter): 10.0.0.22
HOST 2 : CC/SC
em1 : 10.0.0.23
HOST 3 : NC/MIDOLMAN AGENT
em1 : 10.0.0.24
{
"Mido": {
"Gateways": [
{
"Ip": "10.0.0.22",
"ExternalDevice": "em3",
"ExternalCidr": "192.168.0.0/16",
"ExternalIp": "192.168.0.2",
"ExternalRouterIp": "192.168.0.1"
}
]
},
"Mode": "VPCMIDO",
"PublicIps": [
"10.0.100.1-10.0.100.254"
]
}
I may be misunderstanding the intent of reserving an interface just for the mido gateway. All of my eucalyptus/zookeeper/cassandra/midonet configs use the 10.0.0.21 interface and seem to communicate fine. The midonet tunnel zone reports my CLC host and NC host successfully in the tunnel zone. The only part of my config that references the interface I intend to use for the midonet gateway is the network.json. No errors were returned at any time during my config so I think I may be missing something conceptual.
You may need to start eucanetd as described here:
https://docs.eucalyptus.cloud/eucalyptus/4.4.5/index.html#install-guide/starting_euca_clc.html
The eucanetd component in vpcmido mode runs on the cloud controller and is responsible for controlling midonet.
When eucanetd is not running instances will fail to start as the required network resources will not be created.
I configured a bridge on the NC and instances were able to launch and I no longer got an error in my nc.log. Docs and the eucalyptus.conf file comments tell me I shouldn't need to do this in VPCMIDO netowrking mode: https://docs.eucalyptus.cloud/eucalyptus/4.4.5/index.html#install-guide/configuring_bridge.html
Despite all that adding the bridge fixed this issue.

Openstack All-In-One Single Machine Networking

I'm having a hard time configuring an Openstack environment based on the All-In-One Single Machine installer for bridged networking in my LAN.
My objective is to SSH into the instances created in Openstack from my LAN.
The server is an Ubuntu 16.04 LTS with minimal installation and OpenSSH. The network configuration of the server is:
auto enp3s0
iface enp3s0 inet static
address 10.4.4.1
netmask 255.255.255.0
gateway 10.4.4.254
broadcast 10.4.4.255
network 10.4.4.0
dns-nameservers 10.4.1.12 10.4.1.10
Basically my network details are the following:
LAN 10.4.4.0
MASK 255.255.255.0
Gateway/DHCP Server 10.4.4.254
The local.conf file I've used for deploying the devstack is the following:
# Sample ``local.conf`` for user-configurable variables in ``stack.sh``
# NOTE: Copy this file to the root DevStack directory for it to work properly.
# ``local.conf`` is a user-maintained settings file that is sourced from ``stackrc``.
# This gives it the ability to override any variables set in ``stackrc``.
# Also, most of the settings in ``stack.sh`` are written to only be set if no
# value has already been set; this lets ``local.conf`` effectively override the
# default values.
# This is a collection of some of the settings we have found to be useful
# in our DevStack development environments. Additional settings are described
# in https://docs.openstack.org/devstack/latest/configuration.html#local-conf
# These should be considered as samples and are unsupported DevStack code.
# The ``localrc`` section replaces the old ``localrc`` configuration file.
# Note that if ``localrc`` is present it will be used in favor of this section.
[[local|localrc]]
# Minimal Contents
# ----------------
# While ``stack.sh`` is happy to run without ``localrc``, devlife is better when
# there are a few minimal variables set:
# If the ``*_PASSWORD`` variables are not set here you will be prompted to enter
# values for them by ``stack.sh``and they will be added to ``local.conf``.
FLOATING_RANGE=10.4.4.192/27
FIXED_RANGE=192.168.0.0/24
FIXED_NETWORK_SIZE=256
FLAT_INTERFACE=enp3s0
ADMIN_PASSWORD=nomoresecret
DATABASE_PASSWORD=stackdb
RABBIT_PASSWORD=stackqueue
SERVICE_PASSWORD=$ADMIN_PASSWORD
# ``HOST_IP`` and ``HOST_IPV6`` should be set manually for best results if
# the NIC configuration of the host is unusual, i.e. ``eth1`` has the default
# route but ``eth0`` is the public interface. They are auto-detected in
# ``stack.sh`` but often is indeterminate on later runs due to the IP moving
# from an Ethernet interface to a bridge on the host. Setting it here also
# makes it available for ``openrc`` to include when setting ``OS_AUTH_URL``.
# Neither is set by default.
HOST_IP=10.4.4.1
#HOST_IPV6=2001:db8::7
# Logging
# -------
# By default ``stack.sh`` output only goes to the terminal where it runs. It can
# be configured to additionally log to a file by setting ``LOGFILE`` to the full
# path of the destination log file. A timestamp will be appended to the given name.
LOGFILE=$DEST/logs/stack.sh.log
# Old log files are automatically removed after 7 days to keep things neat. Change
# the number of days by setting ``LOGDAYS``.
LOGDAYS=2
# Nova logs will be colorized if ``SYSLOG`` is not set; turn this off by setting
# ``LOG_COLOR`` false.
#LOG_COLOR=False
# Using milestone-proposed branches
# ---------------------------------
# Uncomment these to grab the milestone-proposed branches from the
# repos:
#CINDER_BRANCH=milestone-proposed
#GLANCE_BRANCH=milestone-proposed
#HORIZON_BRANCH=milestone-proposed
#KEYSTONE_BRANCH=milestone-proposed
#KEYSTONECLIENT_BRANCH=milestone-proposed
#NOVA_BRANCH=milestone-proposed
#NOVACLIENT_BRANCH=milestone-proposed
#NEUTRON_BRANCH=milestone-proposed
#SWIFT_BRANCH=milestone-proposed
# Using git versions of clients
# -----------------------------
# By default clients are installed from pip. See LIBS_FROM_GIT in
# stackrc for details on getting clients from specific branches or
# revisions. e.g.
# LIBS_FROM_GIT="python-ironicclient"
# IRONICCLIENT_BRANCH=refs/changes/44/2.../1
# Swift
# -----
# Swift is now used as the back-end for the S3-like object store. Setting the
# hash value is required and you will be prompted for it if Swift is enabled
# so just set it to something already:
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
# For development purposes the default of 3 replicas is usually not required.
# Set this to 1 to save some resources:
SWIFT_REPLICAS=1
# The data for Swift is stored by default in (``$DEST/data/swift``),
# or (``$DATA_DIR/swift``) if ``DATA_DIR`` has been set, and can be
# moved by setting ``SWIFT_DATA_DIR``. The directory will be created
# if it does not exist.
SWIFT_DATA_DIR=$DEST/data
At the end of the deployment I'm able to ping from the instance to my LAN and do nslookup on google.com for example, but I can't do it backwards, ping/ssh/telnet the instance in Openstack.
The security group permits all traffic, all ICMP ingress/egress, SSH from everywhere.
I've tried to telnet on my local computer from the Openstack instance and it's showing the IP of the Openstack host, not the host. So I'm missing something in the network topology.
netstat -ant | grep 1716
tcp6 0 0 :::1716 :::* LISTEN
tcp6 0 0 10.4.3.34:1716 10.4.4.1:42992 ESTABLISHED
Is there any type of network deployment I'm missing?
Any advice would be much appreciated!
If you are trying to access your instances from the "outside", you will need to create a floating IP pool and assign a floating IP to one of your instances.

Command died with signal 6: "/usr/libexec/dovecot/deliver

Emails are not being deliver to a particular email IDs.
We are using Sentora panel and Postfix mail server.
Error message:
Command died with signal 6: "/usr/libexec/dovecot/deliver"
Mail log:
Feb 14 09:50:27 host postfix/pipe[24913]: CBD7D2010A5: to=,
relay=dovecot, delay=13047, delays=13045/0/0/1.3, dsn=4.3.0,
status=SOFTBOUNCE (Command died with signal 6:
"/usr/libexec/dovecot/deliver")
Please help.
Signal 6 is SIGABRT, which is typically sent when there is an internal problem with the code of Dovecot's deliver binary. There are a number of reasons this could happen.
You can turn on LDA logging within your Dovecot config to get more insight on what's actually happening:
protocol lda {
...
# remember to give proper permissions for these files as well
log_path = /var/log/dovecot-lda-errors.log
info_log_path = /var/log/dovecot-lda.log
}
this can also happen when mail_temp_dir (default: /tmp) does not have enough space to extract attachments. it was fixed in https://github.com/dovecot/core/commit/43d7f354c44b358f45ddd10deb3742ec1cc94889 but is not yet available in some linux distributions (such as debian bullseye).

Can't connect to Wordpress SVN server to update repository

Okay, for some reason this morning, I am unable to connect to the Wordpress SVN repository and execute basic svn commands (e.g. checkout, update).
Here's an example of what's happening:
$ svn co http://svn.automattic.com/wordpress/tags/3.3/
# Adds a bunch of files...
svn: warning: Error handling externals definition for '3.3/wp-content/plugins/akismet':
svn: warning: PROPFIND of '/!svn/vcc/default': could not connect to server (http://plugins.svn.wordpress.org)
Checked out revision 19597.
$ cd 3.3
$ svn update
svn: OPTIONS of 'http://svn.automattic.com/wordpress/tags/3.3': could not connect to server (http://svn.automattic.com)
Yet, when I perform these same commands on a development server I have (a Linode VPS) it works fine.
I've google around about this quite a bit, and found pages like these:
http://vsingleton.blogspot.com/2008/04/svn-propfind-request-failed-on.html
http://wordpress.org/support/topic/cant-connect-to-the-pluginssvnwordpress-server
A lot of these articles say something to effect of, it's your proxy server. Well, I'm not behind a proxy server:
http://whatismyipaddress.com/proxy-check
Proxy server not detected.
IP 24.21.xxxx.xxx
rDNS FALSE
WIMIA Test FALSE
TOR Test FALSE
Loc Test FALSE
Header Test FALSE
DNSBL Test FALSE
Just a regular old Comcast home internet connection.
Also, I can browse the wordpress SVN repository just fine via my browser.
Anyhow, I'm sort of at a dead end here, and I guess I'm wondering if anyone has any suggestions as to how to either solve the issue or work around it? I tried setting up an forward proxy server on the Apache installation I have running on that dev server and then updating my ~/.subversion/server file, but that didn't work or I configured something wrong.
Well, if anyone has any brilliant ideas or explanations, I'd love to hear them...
Update
I had a co-worker test this out on his home connection -- he uses Comcast as well. He got the same error as I did. So it appears to be some Comcast-related issue specific to the Wordpress svn repository. I was able to checkout other public repositories via http (e.g. from Google Code) just fine.
I ran a series of tests and I could not find any hidden proxies or cache servers between me and the repository.
I did run traceroute per Lazy Badgers suggestion, and here's what I got:
$ traceroute svn.automattic.com
traceroute to svn.automattic.com (72.233.56.196), 64 hops max, 52 byte packets
1 192.168.1.1 (192.168.1.1) 0.659 ms 0.292 ms 0.185 ms
2 * * *
3 te-5-7-ur01.hollywood.or.bverton.comcast.net (68.85.150.225) 8.792 ms 8.309 ms 9.054 ms
4 xe-3-1-0-0-ar03.beaverton.or.bverton.comcast.net (68.87.216.33) 14.354 ms 24.859 ms 8.753 ms
5 pos-3-8-0-0-cr01.sacramento.ca.ibone.comcast.net (68.86.95.117) 21.869 ms
pos-3-1-0-0-cr01.sacramento.ca.ibone.comcast.net (68.86.95.113) 21.791 ms
pos-3-0-0-0-cr01.sacramento.ca.ibone.comcast.net (68.86.95.109) 22.983 ms
6 pos-0-7-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.85.46) 23.682 ms 25.043 ms 24.675 ms
7 xe-10-3-0.edge1.sanjose1.level3.net (4.71.118.5) 61.048 ms 23.986 ms 24.221 ms
8 vlan80.csw3.sanjose1.level3.net (4.69.152.190) 25.257 ms 25.648 ms
vlan90.csw4.sanjose1.level3.net (4.69.152.254) 24.310 ms
9 ae-82-82.ebr2.sanjose1.level3.net (4.69.153.25) 24.870 ms
ae-92-92.ebr2.sanjose1.level3.net (4.69.153.29) 25.371 ms
ae-91-91.ebr1.sanjose1.level3.net (4.69.153.13) 24.744 ms
10 ae-34-34.ebr4.sanjose1.level3.net (4.69.153.34) 36.011 ms 25.975 ms 36.053 ms
11 ae-5-5.ebr2.sanjose5.level3.net (4.69.148.141) 25.236 ms 25.307 ms 25.305 ms
12 ae-6-6.ebr2.losangeles1.level3.net (4.69.148.201) 31.299 ms 34.076 ms 33.401 ms
13 ae-3-3.ebr3.dallas1.level3.net (4.69.132.78) 59.012 ms 58.604 ms 60.576 ms
14 ae-83-83.csw3.dallas1.level3.net (4.69.151.157) 59.708 ms 65.724 ms
ae-73-73.csw2.dallas1.level3.net (4.69.151.145) 60.383 ms
15 ae-42-90.car2.dallas1.level3.net (4.69.145.196) 60.636 ms
ae-22-70.car2.dallas1.level3.net (4.69.145.68) 59.572 ms 59.758 ms
16 databank-ho.car2.dallas1.level3.net (4.71.170.2) 58.711 ms 59.994 ms 60.561 ms
I don't know if that's unusual or anything. I tried the same on my dev sever and the result looked mostly similar, save for line 2 with the * * *.
I successfully configured a forward proxy on my dev server so I've hacked together a solution for now, but I still don't quite understand what is afoot...
Update 2
In response to a question, here's how I configured things to use my dev server as a proxy for the time being.
First, I configured apache on my dev server to run as a proxy. Make sure these directives are somewhere in your Apache configuration file chain (httpd.conf, vhosts.d directory, etc.):
Listen 8080
<VirtualHost _default_:8080>
ProxyRequests On
ProxyVia On
ProxyPreserveHost On
<Proxy *>
Order deny,allow
Deny from all
Allow from xxx.xxx.xxx.xxx
</Proxy>
</VirtualHost>
This assumes you have a working Apache set up on a development server somewhere (I would definitely not use this on a production server) with mod_proxy installed. Port 8080 is arbitrary. Basically for an unmatched virtual host (i.e. any request that doesn't match your other hosts you have set up) it will turn proxy on and proxy the request through. Change "xxx.xxx.xxx.xxx" to your own IP address.
Now you have to change the server setting in your subversion config file.
In this file:
~/.subversion/servers
Find this section:
[global]
# http-proxy-exceptions = *.exception.com, www.internal-site.org
# http-proxy-host = proxy1.some-domain-name.com
# http-proxy-port = 80
# http-proxy-username = defaultusername
# http-proxy-password = defaultpassword
# http-compression = no
# http-auth-types = basic;digest;negotiate
# No http-timeout, so just use the builtin default.
# No neon-debug-mask, so neon debugging is disabled.
# ssl-authority-files = /path/to/CAcert.pem;/path/to/CAcert2.pem
Uncomment out http-proxy-host and http-proxy-port. For host use a spare domain name you have mapped to your development server or presumably you could just use your server IP. Then set the port to 8080 or whatever you used.
This should route all subversion http requests via your proxy you just set up. It doesn't affect svn or svn+ssh requests.
This was my quick hack, your mileage may vary, this might be totally insecure or broken, etc.
I have Comcast business both at my home office & the corporate office.
BOTH FAIL TO CONNECT TO THE REPO ON COMCAST.
However, I never have a problem if I go over the Windstream T1 or connect via our live server on multiple backbones.
Comcast appears to be "traffic shaping" and/or monitoring business class traffic and breaking the Internet!
Nice job Comcast!
If you don't have an alternate connection they you may need to use a proxy service and then send Comcast a nasty email about their network filtering.

Resources