I am trying to read and write the Mainframe Datasets data in Python3.6. I am using Anaconda's Spyder(version 3.2.4). I am using Zosftplib inorder to import mainframe features. Below is the code snippet:
import zosftplib
Myzftp = zosftplib.Zftp("ip address-mainframe","username","password")
mf_file = open("mainframe ps file-name", 'r+')
ffa = mf_file.read(16);
print ("Read record is :", ffa)
mf_file.close()
Mainframe PS-file name contains 1 record with data-0010021023457893.But the output I am getting is spaces in Spyder kernel.I also tried using ftplib but it didn't worked there too.I believe there's conversion required as its not a text file which I am reading.Does anyone has any suggestion on this.Please reply.Thanks
Thru FTPLIB and Zosftplib import
import zosftplib
Myzftp = zosftplib.Zftp("ip address-mainframe","username","password")
mf_file = open("mainframe ps file-name", 'r+')
ffa = mf_file.read(16);
print ("Read record is :", ffa)
mf_file.close()
Expected result should be 0010021023457893 after file read and print.
The zosftplib package will provide you ftp access to your dataset on z/OS, meaning you can download it, but you have to open it locally. Also, you need to be aware of the encoding differences between your local machine and the z/OS environment, so you should specify the sbdataconn() argument to provide codepage translation. I was able to do what you want with code like this:
import zosftplib
Myzftp = zosftplib.Zftp('mainframe_ip',
'mainframe_userid',
'mainframe_password',
timeout=500.0,
sbdataconn='(ibm-1147,iso8859-1)')
Myzftp.download_text('mainframe_dataset_name', '/tmp/local_filename.txt')
mf_file = open('/tmp/local_filename.txt', 'r+')
ffa = mf_file.read(16);
print ("Read record is :", ffa)
mf_file.close()
Related
I am getting my form data from frontend and reading it using fast api as shown below:
#app.post("/file_upload")
async def upload_file(pdf: UploadFile = File(...)):
print("Content = ",pdf.content_type,pdf.filename,pdf.spool_max_size)
return {"filename": "Succcess"}
Now what I need to do is extract hyperlinks from these spool Files with the help of pypdfextractor as shown below:
import pdfx
from os.path import exists
from config import availableUris
def getHrefsFromPDF(pdfPath:str)->dict:
if not(exists(pdfPath)):
raise FileNotFoundError("PDF File not Found")
pdf = pdfx.PDFx(pdfPath)
return pdf.get_references_as_dict().get('url',[])
But I am not sure how to convert spool file (Received from FAST API) to pdfx readable file format.
Additionally, I also tried to study the bytes that come out of the file. When I try to do this:
data = await pdf.read()
data type shows as : bytes when I try to convert it using str function it gives a unicoded encoded string which is totally a gibberish to me, I also tried to decode using "utf-8" which throws UnicodeDecodeError.
fastapi gives you a SpooledTemporaryFile. You may be able to use that file object directly if there is some api in pdfx which will work on a File() object rather than a str representing a path (!). Otherwise make a new temporary file on disk and work with that:
from tempfile import TemporaryDirectory
from pathlib import Path
import pdfx
#app.post("/file_upload")
async def upload_file(pdf: UploadFile = File(...)):
with TemporaryDirectory() as d: #Adding the file into a temporary storage for re-reading purposes
tmpf = Path(d) / "pdf.pdf"
with tmpf.open("wb") as f:
f.write(pdf.read())
p = pdfx.PDFX(str(tmpf))
...
It may be that pdfx.PDFX will take a Path object. I'll update this answer if so. I've kept the read-write loop synchronous for ease, but you can make it asynchronous if there is a reason to do so.
Note that it would be better to find a way of doing this with the SpooledTemporaryFile.
As to your data showing as bytes: well, pdfs are (basically) binary files: what did you expect?
I'm a python noob here just trying to learn a new skill so please be gentle :)
After executing the following script it appears that my script creates a telnet session and I can see the cisco devices banner but is not passing the username/password when prompted.
I've tried changing tn.read_until() and tn.read_very_eager() neither of which is triggering the script to move on to write the desired input. I've also tried forcing the username and password to be input as a byte as well as adding + '\n' to my write functions. I've also used sleep to pass a little extra time to wait for the banner to finish printing out.
tldr: when executing the script I see the username prompt, but can't get further than that.
any assistance here is welcomed.
'''
from time import sleep
import telnetlib
from telnetlib import Telnet
from getpass import getpass
Username = input('please provide your username ')
password = getpass()
# f is the .txt document that lists the IP's we'll be using.
f = open('devicess.txt')
for line in f:
print ('Configuring Device ' + (line))
device = (line)
#open connection to variable
tn = telnetlib.Telnet(device)
#For those devices in the above list, connect and run the below commands
for device in f:
tn.expect('Username: ')
tn.write(Username)
tn.expect('Password: ')
tn.write(password)
tn.write('show ver')
print('connection established to ' + device)
tn.write('logout')
output = tn.read_all()
print(output)
print('script complete')
'''
I have several Jupyter notebooks which perform analysis on datasets. Right now, a dataset is specified by its filename. Every time the user wants to perform analysis on a new dataset, she/he has to edit the appropriate line in the notebook and modify dataset path string. The datasets can be located in different directories. The notebooks can also be located in different directories. In each notebook I would like to provide a widget that allows the user to browse the remote file system and pick the dataset he/she wants to analyse.
Are there any open source projects that support the above functionality? I am looking for something that is still active/supported and has some basic documentation. I did quick search on Google and surprisingly I didn't find anything.
Then I realised that JupyterLab, the evolution of Jupyter, has something very similar to what I want. It already has a very capable file browser but it is a bit "isolated" from everything else.
Is it possible somehow to get the relative (to the currently opened notebook) path of the selected file in the JupyterLab file browser?
Thank you.
Here's code for a server-side file browsing widget. Only tested in regular Jypter notebook - not Jupyter Lab. Also, must use a fairly recent version. Hope this helps.
import sys
import os
import ipywidgets as ui
from IPython.display import display
class PathSelector():
def __init__(self,start_dir,select_file=True):
self.file = None
self.select_file = select_file
self.cwd = start_dir
self.select = ui.SelectMultiple(options=['init'],value=(),rows=10,description='')
self.accord = ui.Accordion(children=[self.select])
self.accord.selected_index = None # Start closed (showing path only)
self.refresh(self.cwd)
self.select.observe(self.on_update,'value')
def on_update(self,change):
if len(change['new']) > 0:
self.refresh(change['new'][0])
def refresh(self,item):
path = os.path.abspath(os.path.join(self.cwd,item))
if os.path.isfile(path):
if self.select_file:
self.accord.set_title(0,path)
self.file = path
self.accord.selected_index = None
else:
self.select.value = ()
else: # os.path.isdir(path)
self.file = None
self.cwd = path
# Build list of files and dirs
keys = ['[..]'];
for item in os.listdir(path):
if item[0] == '.':
continue
elif os.path.isdir(os.path.join(path,item)):
keys.append('['+item+']');
else:
keys.append(item);
# Sort and create list of output values
keys.sort(key=str.lower)
vals = []
for k in keys:
if k[0] == '[':
vals.append(k[1:-1]) # strip off brackets
else:
vals.append(k)
# Update widget
self.accord.set_title(0,path)
self.select.options = list(zip(keys,vals))
with self.select.hold_trait_notifications():
self.select.value = ()
f = PathSelector('/some/data')
display(f.accord)
I am automating instance creation using OpenstackSDK and passing bash script with commands as userdata. But the script does not excute even though the instance is crated. When I do this manually via GUI, the bash scripts executes fine to the newly created instance.
#Reading bash script
with open('elk.sh', 'r') as f:
init_script = f.read()
server = conn.compute.create_server(
name=name,
image_id=IMAGE_ID,
flavor_id=FLAVOUR_ID,
networks=[{"uuid": NETWORK_ID}],
user_data=init_script, # pass script to the instance
key_name=KEY_PAIR
)
Note: Also tried to encode as Base64 file butstill failed with
is not JSON serializable.
Code snippet:
with open(USER_DATA,'r') as file:
f = file.read()
bytes_content = bytes(f,encoding='utf-8')
init_script = base64.b64encode(bytes_content)
Can anyone advice on this, please?
Thanks
Python3 handles string and binary differently. Also, to pass bash/cloud-config file to --user_data via OpenstackSDK, it has to be base46 encoded.
Code snippet:
with open(USER_DATA,'r') as file:
f = encodeutils.safe_encode(file.read().encode('utf-8'))
init_script = base64.b64encode(f).decode('utf-8')
I would like to override what happens when run is pressed for certain cells in ipython notebook.
For example, I would like to be able to write an SQL query directly in a cell and define a function that processes it.
It seems it should be possible to do this as with ipython-notebook extensions. Does anyone know of a similar extension? An easy way to do this directly from ipython?
Ideally this would involve defining a custom cell type, but I would be happy to use special tags to separate the usual python code from, say, a custom SQL query cell.
I've once had the similar desire and I ended up with the following solution:
from sqlalchemy import create_engine
import pandas as pd
from IPython.core.magic import register_cell_magic
from IPython import get_ipython
con = create_engine(DB_URL)
#register_cell_magic
def sql(line, cell):
cell = cell.format(**globals())
if line.strip() != '-':
res = pd.read_sql(cell, con)
if line.strip() != '': get_ipython().user_ns[line.strip()] = res
return res
else:
con.execute(cell)
del sql
You can now write in other cells:
%sql outputvar
select * from whatever where ...
For example: