Scapy_Exception: Not a supported capture file on Python3 - networking

I am trying to build a python script for wifi network monitoring.
I followed this repository, which works fine if I use Python2. But if I try to convert this code to Python3 and run I get this error:
File "/Users/ecp/PycharmProjects/pythonProject4/venv/lib/python3.9/site-packages/scapy/utils.py", line 1172, in __call__
raise Scapy_Exception("Not a supported capture file")
scapy.error.Scapy_Exception: Not a supported capture file
This is the part of the code where I get the error:
class MACCollector(object):
def __init__(self, channels, bssids, interface):
self.clients = set()
self.bssids = bssids
self.channels = cycle(channels)
self.interface = interface
self.fifo = os.path.join(os.path.dirname(__file__), "pcapfifo.cap")
if not os.path.exists(self.fifo):
os.mkfifo(self.fifo)
def start(self):
subprocess.check_call([AIRPORT, "--disassociate"])
self.hop_channel()
subprocess.Popen([TCPDUMP, "-I", "-n", "-i", self.interface, "-w", self.fifo])
print(("************",self.fifo))
pcap = scapy.PcapReader(self.fifo) #HERE SCAPY CANT READ
for packet in pcap:
self.process_packet(packet)
Scapy cant read the fifo file if I use Python3.
My operating system: macOS Monterey 12.6
My Python3 version: 3.10.8
I have tried many different things, but none of it worked. I tried different ways to create the fifo file, but no luck.

Related

GCP Composer v1.18.6 and 2.0.10 incompatible with CloudSqlProxyRunner

In my Composer Airflow DAGs, I have been using the CloudSqlProxyRunner to connect to my Cloud SQL instance.
However, after updating Google Cloud Composer from v1.18.4 to 1.18.6, my DAG started to encounter a strange error:
[2022-04-22, 23:20:18 UTC] {cloud_sql.py:462} INFO - Downloading cloud_sql_proxy from https://dl.google.com/cloudsql/cloud_sql_proxy.linux.x86_64 to /home/airflow/dXhOYoU_cloud_sql_proxy.tmp
[2022-04-22, 23:20:18 UTC] {taskinstance.py:1702} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1330, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1457, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1513, in _execute_task
result = execute_callable(context=context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/decorators/base.py", line 134, in execute
return_value = super().execute(context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/operators/python.py", line 174, in execute
return_value = self.execute_callable()
File "/opt/python3.8/lib/python3.8/site-packages/airflow/operators/python.py", line 185, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/home/airflow/gcs/dags/real_time_scoring_pipeline.py", line 99, in get_messages_db
with SQLConnection() as sql_conn:
File "/home/airflow/gcs/dags/helpers/helpers.py", line 71, in __enter__
self.proxy_runner.start_proxy()
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/cloud_sql.py", line 524, in start_proxy
self._download_sql_proxy_if_needed()
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/cloud_sql.py", line 474, in _download_sql_proxy_if_needed
raise AirflowException(
airflow.exceptions.AirflowException: The cloud-sql-proxy could not be downloaded. Status code = 404. Reason = Not Found
Checking manually, https://dl.google.com/cloudsql/cloud_sql_proxy.linux.x86_64 indeed returns a 404.
Looking at the function that raises the exception, _download_sql_proxy_if_needed, it has this code:
system = platform.system().lower()
processor = os.uname().machine
if not self.sql_proxy_version:
download_url = CLOUD_SQL_PROXY_DOWNLOAD_URL.format(system, processor)
else:
download_url = CLOUD_SQL_PROXY_VERSION_DOWNLOAD_URL.format(
self.sql_proxy_version, system, processor
)
So, for whatever reason, in both of these latest images of Composer, processor = os.uname().machine returns x86_64. Previously, it returned amd64, and https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 is in fact a valid link to the binary we need.
I replicated this error in Composer 2.0.10 as well.
I am still investigating possible workarounds, but posting this here in case someone else encounters this issue, and has figured out a workaround, and to raise this with Google engineers (who, according to Composer's docs, monitor this tag).
My current workaround is patching the CloudSqlProxyRunner to hardcode the correct URL:
class PatchedCloudSqlProxyRunner(CloudSqlProxyRunner):
"""
This is a patched version of CloudSqlProxyRunner to provide a workaround for an incorrectly
generated URL to the Cloud SQL proxy binary.
"""
def _download_sql_proxy_if_needed(self) -> None:
download_url = "https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64"
# the rest of the code is taken from the original method
proxy_path_tmp = self.sql_proxy_path + ".tmp"
self.log.info(
"Downloading cloud_sql_proxy from %s to %s", download_url, proxy_path_tmp
)
# httpx has a breaking API change (follow_redirects vs allow_redirects)
# and this should work with both versions (cf. issue #20088)
if "follow_redirects" in signature(httpx.get).parameters.keys():
response = httpx.get(download_url, follow_redirects=True)
else:
response = httpx.get(download_url, allow_redirects=True) # type: ignore[call-arg]
# Downloading to .tmp file first to avoid case where partially downloaded
# binary is used by parallel operator which uses the same fixed binary path
with open(proxy_path_tmp, "wb") as file:
file.write(response.content)
if response.status_code != 200:
raise AirflowException(
"The cloud-sql-proxy could not be downloaded. "
f"Status code = {response.status_code}. Reason = {response.reason_phrase}"
)
self.log.info(
"Moving sql_proxy binary from %s to %s", proxy_path_tmp, self.sql_proxy_path
)
shutil.move(proxy_path_tmp, self.sql_proxy_path)
os.chmod(self.sql_proxy_path, 0o744) # Set executable bit
self.sql_proxy_was_downloaded = True
And then instantiate it and use it as I would the original CloudSqlProxyRunner:
proxy_runner = PatchedCloudSqlProxyRunner(path_prefix, instance_spec)
proxy_runner.start_proxy()
But I am hoping that this is properly fixed by someone at Google soon, by fixing the os.uname().machine value,
or uploading a Cloud SQL proxy binary to the one currently generated in _download_sql_proxy_if_needed.
As mentioned by #enocom this commit to support arm64 download links actually caused a side-effect of generating broken download links. I assume the author of the commit thought that the Cloud SQL Proxy had binaries for each machine type, although in fact there are not Linux x86_64 links.
I have created an airflow PR to hopefully fix the broken links, hopefully it will get merged in soon and resolve this. Will update the thread with any updates.
Update (I've been working with Jack on this): I just merged that PR! When a new version of the providers is added to PyPI, you'll need to add it to your Composer environment. In the meantime, as a workaround, you could take the fix from Jack's PR and use it as a local dependency. (Similar to the other reply here!) If you do this, I highly recommend setting a calendar reminder (maybe a month from now?) to remove the workaround and go back to importing from the provider package, just to make sure you don't miss out on other updates to it! :)

Using Apache Airflow Tool, Implement a DAG for a batch processing pipeline to get a directory from a remote system

Using Apache airflow tool, how can I implement a DAG for the following Python code. The task accomplished in the code is to get a directory from GPU server to local system. Code is working fine in Jupyter notebook. Please help to implement in Airflow...I'm very new to this. Thanks.
import pysftp
import os
myHostname = "hostname"
myUsername = "username"
myPassword = "pwd"
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword) as sftp:
print("Connection successfully stablished ... ")
src = '/path/src/'
dst = '/home/path/path/destination'
os.mkdir(dst)
sftp.get_d(src, dst, preserve_mtime=True)
print("Fetched source images from GPU server to local directory")
# connection closed automatically at the end of the with-block```
For SFTP duties, Airflow provides SFTOperator that you can use directly.
Alternatively it's corresponding SFTPHook can be used with a simple PythonOperator
I acknowledge there aren't many examples, but this might be helpful
For SSH-connection, see this

understanding TYPE in lsof output

I opened a file through python. So, i did a lsof on the python process. output of lsof has the following line
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 15855 inaflash 3w REG 0,25 0 4150810088 /home/inaflash/he.txt
Thing is, it has 3w. which means that the file is open for writing. But, i actually opened the file as follows
a = open('he.txt','r')
I read that, w means file is open for write. Can anyone help me understand why its w instead of r
I tried the same code in Python 3 and my file is opened in read mode.
Are you sure your file is the same opened with python and same python process ?
Maybe you forgot to close the file somewhere in your code after opened it in write mode.
Edit: Also tried in Python 2, same result (read mode)

Monkeyrunner. Connect to multiple devices at the same time

I used this script to connect to multiple Android devices at the same time (i.e. to send files via adb):
from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice
import time
import sys
import time
import os
devices = os.popen('adb devices').read().strip().split('\n')[1:]
device1 = MonkeyRunner.waitForConnection( devices[0].split('\t')[0])
package = 'com.android.browser'
activity = 'com.android.browser.BrowserActivity'
runComponent = package + '/' + activity
device1.startActivity(component=runComponent)
MonkeyRunner.sleep(1)
device2 = MonkeyRunner.waitForConnection( devices[1].split('\t')[0])
package = 'com.android.browser'
activity = 'com.android.browser.BrowserActivity'
runComponent = package + '/' + activity
device2.startActivity(component=runComponent)
I used
adb 1.0.36 (Rev 1:7.0.0+r33-2) and
monkyrunner(Jython 2.5.3 (2.5:c56500f08d34+, Aug 13 2012, 14:54:35)
But all I get is:
09:02:54 E/DeviceMonitor: Adb connection Error:EOF
09:02:54 E/DeviceMonitor: Connection attempts: 1
09:02:55 E/DeviceMonitor: Connection attempts: 2
09:02:56 E/DeviceMonitor: Connection attempts: 3
Any hints what to do?
Thanks!
(Original Thread: How to run Monkeyrunner script on multiple devices at the same time )
You can use AndroidViewClient/culebra which supports multiple devices. If I recall correctly, this was one of the limitations of monkeyrunner that AndroidViewClient solved.
You can simply generate the script using
$ culebra -Uu --multi-device --start-activity='com.android.chrome/com.google.android.apps.chrome.Main' -o multi-browser.py
which generates a unit test (-U), does not verify what's in the screen dump (-u), use multiple devices, starts a specific activity as a precondition and saves the generated script to multi-device.py.
Then, run the script as
$ multi-browser.py -s all
where -s specifies the serial numbers of the devices where you want the script to run, all in this case, and the browser will start on all of them.

python requests can't find a folder with a certificate when converted to .exe

I have a program that pools ad stats from different marketing systems. Everything works fine untill i convert it to the .exe format and run it.
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python35\lib\tkinter\__init__.py", line 1549, in __call__
return self.func(*args)
File "C:\Users\user\Desktop\alg\TSK_7. Marketing\report_gui.py", line 24, in <lambda>
ok = tk.Button(root, text="DO NOT PRESS", bg="red", command=lambda: self.run())
File "C:\Users\user\Desktop\alg\TSK_7. Marketing\report_gui.py", line 43, in run
report.merge_all()
File "C:\Users\user\Desktop\alg\TSK_7. Marketing\process_data.py", line 400, in merge_all
fb_df = self.fetch_fb()
File "C:\Users\user\Desktop\alg\TSK_7. Marketing\process_data.py", line 156, in fetch_fb
fb_campaigns = from_fb.run_fb(self.start_date, self.end_date) # in JSON format
File "C:\Users\user\Desktop\alg\TSK_7. Marketing\from_fb.py", line 110, in run_fb
return s.get_stats()
File "C:\Users\user\Desktop\alg\TSK_7. Marketing\from_fb.py", line 84, in get_stats
params=params,
File "C:\Users\user\AppData\Local\Programs\Python\Python35\lib\site-packages\facebookads\adobjects\adaccount.py", line 1551, in get_insights
return request.execute()
File "C:\Users\user\AppData\Local\Programs\Python\Python35\lib\site-packages\facebookads\api.py", line 653, in execute
cursor.load_next_page()
File "C:\Users\user\AppData\Local\Programs\Python\Python35\lib\site-packages\facebookads\api.py", line 797, in load_next_page
params=self.params,
File "C:\Users\user\AppData\Local\Programs\Python\Python35\lib\site-packages\facebookads\api.py", line 305, in call
timeout=self._session.timeout
File "C:\Users\user\AppData\Local\Programs\Python\Python35\lib\site-packages\requests\sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python35\lib\site-packages\requests\sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python35\lib\site-packages\requests\adapters.py", line 407, in send
self.cert_verify(conn, request.url, verify, cert)
File "C:\Users\user\AppData\Local\Programs\Python\Python35\lib\site-packages\requests\adapters.py", line 226, in cert_verify
"invalid path: {0}".format(cert_loc))
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: C:\Users\user\AppData\Local\Temp\_MEI253762\facebookads\fb_ca_chain_bundle.crt
I tried to fix that by using this code but MEI folder keeps changing its digits everytime i run this code so it's no use.
dst = r'C:\Users\user\AppData\Local\Temp\_MEI120642\facebookads'
file = 'fb_ca_chain_bundle.crt'
try:
os.makedirs(dst); ## it creates the destination folder
except:
pass
shutil.move(file, dst)
So i went to this file
C:\Users\user\AppData\Local\Programs\Python\Python35\Lib\site-packages\requests\adapters.py
and tried to comment if statements that raise this error out but got an SSL error. I couldn't find a piece of code responsible for generating those MEI digits.
def cert_verify(self, conn, url, verify, cert):
"""Verify a SSL certificate. This method should not be called from user
code, and is only exposed for use when subclassing the
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
:param conn: The urllib3 connection object associated with the cert.
:param url: The requested URL.
:param verify: Either a boolean, in which case it controls whether we verify
the server's TLS certificate, or a string, in which case it must be a path
to a CA bundle to use
:param cert: The SSL certificate to verify.
"""
if url.lower().startswith('https') and verify:
cert_loc = None
# Allow self-specified cert location.
if verify is not True:
cert_loc = verify
if not cert_loc:
cert_loc = DEFAULT_CA_BUNDLE_PATH
if not cert_loc or not os.path.exists(cert_loc):
raise IOError("Could not find a suitable TLS CA certificate bundle, "
"invalid path: {0}".format(cert_loc))
conn.cert_reqs = 'CERT_REQUIRED'
if not os.path.isdir(cert_loc):
conn.ca_certs = cert_loc
else:
conn.ca_cert_dir = cert_loc
else:
conn.cert_reqs = 'CERT_NONE'
conn.ca_certs = None
conn.ca_cert_dir = None
if cert:
if not isinstance(cert, basestring):
conn.cert_file = cert[0]
conn.key_file = cert[1]
else:
conn.cert_file = cert
conn.key_file = None
if conn.cert_file and not os.path.exists(conn.cert_file):
raise IOError("Could not find the TLS certificate file, "
"invalid path: {0}".format(conn.cert_file))
if conn.key_file and not os.path.exists(conn.key_file):
raise IOError("Could not find the TLS key file, "
"invalid path: {0}".format(conn.key_file))
I ran into this problem as well. It looks like it comes from the certificate bundle cacert.pem not being included in the requests package directory when the program is compiled. The requests module uses the function certifi.core.where to determine the location of cacert.pem. Overriding this function and overriding the variables set by this function seems to fix the problem.
I added this code to the beginning of my program:
import sys, os
def override_where():
""" overrides certifi.core.where to return actual location of cacert.pem"""
# change this to match the location of cacert.pem
return os.path.abspath("cacert.pem")
# is the program compiled?
if hasattr(sys, "frozen"):
import certifi.core
os.environ["REQUESTS_CA_BUNDLE"] = override_where()
certifi.core.where = override_where
# delay importing until after where() has been replaced
import requests.utils
import requests.adapters
# replace these variables in case these modules were
# imported before we replaced certifi.core.where
requests.utils.DEFAULT_CA_BUNDLE_PATH = override_where()
requests.adapters.DEFAULT_CA_BUNDLE_PATH = override_where()
I solved it by input the following code:
import os
import sys
import certifi
os.environ['REQUESTS_CA_BUNDLE'] =
os.path.join(os.path.dirname(sys.argv[0]), certifi.where())
By this I detect the current path to "cacert.pem" and put it in environ
This might be an issue with requests package.
I solved this by manually copying the cacert.pem file from /lib/site-packages/certifi to /lib/site-packages/requests
If you want to fix this issue with .exe, then copy cacert.pem file from /lib/site-packages/certifi to dist/library.zip/certifi/.
I am considering you have created exe using py2exe, where py2exe will create library.zip under dist/ which contains of all script dependencies. I don't know if other exe converters create library.zip.
I encountered this same problem with the requests package when converting to .exe with PyInstaller. I was developing with conda's virtual environment. I pivoted to using Python's virtual environment (as my colleague had done this and didn't get the same error), installed my packages as usual, created the .exe with PyInstaller and didn't encounter the same error.
I'm not sure if the cacert.pem file is correctly packaged into the .exe with the Python virtual environment but is somehow ignored with the conda virtual environment. Perhaps someone else can clarify why this was the case but I hope this helps!
If you want to disable certificate verification, you can use the verify=False parameter in requests.get():
requests.get('https://example.com', verify=False)

Resources