PYTHON : Saving multiple images into folder using requests - python-requests

I need to save all 6 images into a local folder.
The script I found re-writes a single files multiple times and end up producing only 1 image
import requests
img_list = ["https://ae01.alicdn.com/kf/HTB1tT70vhuTBuNkHFNRq6A9qpXa3.jpg", "https://ae01.alicdn.com/kf/HTB12HGkvwKTBuNkSne1q6yJoXXaR.jpg", "https://ae01.alicdn.com/kf/HTB1_yDic56guuRjy0Fmq6y0DXXaY.jpg", "https://ae01.alicdn.com/kf/HTB1RopgXffsK1RjSszgq6yXzpXa5.jpg", "https://ae01.alicdn.com/kf/HTB1R6sJXgHqK1RjSZFkq6x.WFXaF.jpg", "https://ae01.alicdn.com/kf/HTB1_XlhXojrK1RkHFNRq6ySvpXaR.jpg"]
for x in blob:
with open('/Users/reezalaq/PycharmProjects/wholesale/img/pic1.jpg', 'wb') as handle:
response = requests.get(x, stream=True)
if not response.ok:
print(response)
for block in response.iter_content(1024):
if not block:
break
handle.write(block)```
It needs to save all 6 images separately. No error message so far.

The script rewrites the same file each time because you're using the same file name, it never changes.
The problem is here:
with open('/Users/reezalaq/PycharmProjects/wholesale/img/pic1.jpg', 'wb')
The first argument of the open() method is the file path. The second argument is the mode, which you have set to wb, or write/binary. So in you're loop you are rewriting the file contents of pic1.jpg everytime. (See: https://docs.python.org/3.5/library/functions.html#open).
You can pre-define a list of filenames in a list and use those as the filenames or do something more dynamic like :
for img in img_list:
file_name = img.split('/')[-1]
with open(file_name, 'wb') as handle:
....
This would grab the file name of the image from the website you're downloading it from (e.g., 1HTB1tT70vhuTBuNkHFNRq6A9qpXa3.jpg for the first URL) to be used as the file name on your system. (Note: this also assumes names will be unique).
Edit:
You can define your folder path before the for-loop. Then, you can change the open() method to include that path. So:
import os # do this at the top of your file
folder_path = '/Users/reezalaq/PycharmProjects/wholesale/img/'
for img in img_list:
with open(os.path.join(folder_path, file_name), 'wb') as handle:
....

Related

Extract Hyperlink from a spool pdf file in Python

I am getting my form data from frontend and reading it using fast api as shown below:
#app.post("/file_upload")
async def upload_file(pdf: UploadFile = File(...)):
print("Content = ",pdf.content_type,pdf.filename,pdf.spool_max_size)
return {"filename": "Succcess"}
Now what I need to do is extract hyperlinks from these spool Files with the help of pypdfextractor as shown below:
import pdfx
from os.path import exists
from config import availableUris
def getHrefsFromPDF(pdfPath:str)->dict:
if not(exists(pdfPath)):
raise FileNotFoundError("PDF File not Found")
pdf = pdfx.PDFx(pdfPath)
return pdf.get_references_as_dict().get('url',[])
But I am not sure how to convert spool file (Received from FAST API) to pdfx readable file format.
Additionally, I also tried to study the bytes that come out of the file. When I try to do this:
data = await pdf.read()
data type shows as : bytes when I try to convert it using str function it gives a unicoded encoded string which is totally a gibberish to me, I also tried to decode using "utf-8" which throws UnicodeDecodeError.
fastapi gives you a SpooledTemporaryFile. You may be able to use that file object directly if there is some api in pdfx which will work on a File() object rather than a str representing a path (!). Otherwise make a new temporary file on disk and work with that:
from tempfile import TemporaryDirectory
from pathlib import Path
import pdfx
#app.post("/file_upload")
async def upload_file(pdf: UploadFile = File(...)):
with TemporaryDirectory() as d: #Adding the file into a temporary storage for re-reading purposes
tmpf = Path(d) / "pdf.pdf"
with tmpf.open("wb") as f:
f.write(pdf.read())
p = pdfx.PDFX(str(tmpf))
...
It may be that pdfx.PDFX will take a Path object. I'll update this answer if so. I've kept the read-write loop synchronous for ease, but you can make it asynchronous if there is a reason to do so.
Note that it would be better to find a way of doing this with the SpooledTemporaryFile.
As to your data showing as bytes: well, pdfs are (basically) binary files: what did you expect?

ASP File Object Issue

I am working on an ASP classic website, which the client reported suddenly exhibited an issue with a previously working function which listed only "image" file types. Upon reading the code, I found that the loop that lists the files in a folder, uses the InStr() function to identify files by their type, which should be "image." However, I found that something must have changed in the OS, as the type is not longer "image", but "JPG", or "PNG", etc. This dramatically changes the way the code works. Following is a snipped of the code:
Set oFSO = Server.CreateObject("Scripting.FileSystemObject")
Set oFolder = oFSO.GetFolder(Server.MapPath(sCurrentDirectoryPath))
Set oSubFolder = oFolder.Files
iFileCount = 0
For Each oFileName in oSubFolder
If InStr(1, LCase(oFileName.Type),"image") > 0 Then
iFileCount = iFileCount + 1
End If
Next
Because the InStr() function is trying to find a file type of "image", no files are counted up, and the function returns zero files found. Whilst debugging, I found that the value being returned by oFileName.Type, was as follows:
This is the file type:JPG File
This is the file type:JPG File
This is the file type:Text Document
This is the file type:Data Base File
Files in the folder were two "whatever.jpg" files, a "whatever.txt" file, and a "thumbs.db" file. So, it appears that the OS (Windows Server 2019) may have changed to be less generic with reporting an "image" file, and is now reporting "JPG file" or "PNG file", etc. This of course, breaks this code! Are there any suggestions from you'all on how I could go about modifying this code to work on reporting exactly how many image files are present?
On Windows 10, the Type values for .jpg and .png files are JPEG image and PNG image respectively. What OS are you using?
Also, Type doesn't actually analyze the file, you could have a virus.exe file in the folder that has been renamed to virus.jpg, and the Type value will still show it as JPEG. So if the function is indented to check user uploaded content to ensure images are actually images, the Type value will be of no use. If you have root access you could install a COM DLL that uses a program such as ExifTool to properly analyze files (https://github.com/as08/ClassicASP.ExifTool), however that would be a complete rewrite.
But assuming you're not looking to check if an image file is actually an image file, you could just split the filename extensions and use Select Case to count the image files if your OS is returning just XXX file and no longer XXX image in the Type value (alternatively you could split the Type value, but you'd still need to check for valid image file extensions):
Dim oFSO, oFolder, oSubFolder, oFileName, iFileCount, oFileNameSplit
Set oFSO = Server.CreateObject("Scripting.FileSystemObject")
Set oFolder = oFSO.GetFolder(Server.MapPath(sCurrentDirectoryPath))
Set oSubFolder = oFolder.Files
iFileCount = 0
For Each oFileName In oSubFolder
oFileNameSplit = Split(oFileName.Name,".")
If uBound(oFileNameSplit) > 0 Then
Select Case Trim(lCase(oFileNameSplit(uBound(oFileNameSplit))))
Case "jpg", "jpeg", "png", "gif" ' Maybe add some more extensions...
iFileCount = iFileCount + 1
End Select
End If
Next
Set oFSO = Nothing
Set oFolder = Nothing
Set oSubFolder = Nothing
Response.Write(iFileCount)

How to create a list and append values as in python using lua

I am trying to create a list with filename from my local directory and add that to http request body in nginx using lua. something like below
filelist = [] #a list variable to hold the filenames
for file in f:lines() do
#file i get my filename one by one
filelist.append(file)
end
request_payload = {"somedata":"its key", "files": filelist}
assuming i have the request body and i need to update that json with the new filelist variable value.
I know with lua there is nothing like list object instead it have table structure. But is there any easy hacks to get this done
I got it working with lua arrays
local filelist = {} #lua table object
for file in f:lines() do
#file i get my filename one by one
table.insert(filelist, file) #this will update the array
end
Hope that helps someone :)

With wrk, what is the most efficient way to do a HTTP PUT with a file as a body?

I want to benchmark an application by doing many HTTP PUT request that include files body. I have many files and each file need to be sent only one time.
For now I am trying to do that using WRK. One way i have find to do that is to split my data in several repo, giving each WRK thread a repo. But my big problem is how to pass the file as a PUT parameter (basically do a curl -T). For now i am doing it by reding the file in the LUA script and puting the content into wrk.body which is not very performant (too slow).
Here is the part of code i am using to do the PUT with a file parameter :
function read_file(path)
local file, errorMessage = io.open(path, "r")
if not file then
error("Could not read the file: "..path.." Error : " .. errorMessage .. "\n")
end
local content = file:read "*all"
file:close()
return content
end
request = function()
local body = read_file("/data/"..id.."/"..files[counter])
counter = counter + 1
local Boundary = "----WebKitFormBoundaryePkpFF7tjBAqx29L"
wrk.headers["Content-Type"] = "multipart/form-data; boundary=" .. Boundary
return wrk.format("PUT", url, wrk.headers,body)
end
I just want to know if there is a more efficient way to add a file as a PUT (or POST) HTTP request using WRK.

Add file at root of zip file using DotNetZip with classic ASP

I have DotNetZip installed and running fine on a Windows 2008 server.
Using a classic ASP page, I want to bundle a bunch of comma-delimited files to a user and send it over in a zip file.
The following code works fine but it stores all the path information so the files inside the zip file are located in some ridiculous directory like C:\Inetpub\wwwroot\appname\_temp\
I'm using the following code:
Set objZip = CreateObject("Ionic.Zip.ZipFile")
sFileArray = Split(sFileArray, "|")
For iCount = 0 To UBound(sFileArray)
If sFileArray(iCount) <> "" Then
objZip.AddFile sFileArray(iCount)
End If
Next
objZip.Name = sFilePath & "test.zip"
objZip.Save()
objZip.Dispose()
Set objZip = Nothing
I see that the AddFile method allows you to specify where you want the added file to reside in the zip file if you add a second parameter. According to the documentation objZip.AddFile sFileArray(iCount), "" should put the file in the root of the zip file.
However, when I add that parameter, I get the following error:
Wrong number of arguments or invalid property assignment: 'objZip.AddFile'
Anyone have any idea what I'm doing wrong?
Thanks.
I think you are misinterperting the documentation. If the second parameter is null then the directory path of the file being added is used. If the second parameter is an empty string "" then the file is added to the root level in the zip. A quick look into the Ioniz.zip.dll shows that the single parameter override of AddFile method simply calls the the double parameter override with the second parameter set to null.
Hence your add file should look like:
objZip.AddFile sFileArray(iCount), ""
to get the result you are after.

Resources