pipenv: pdfkit not found when running FastAPI app - fastapi

I've been developing an API using FastAPI and pipenv to manage my virtual environment. To start my testing setup, I use the following commands on the console:
pipenv shell
uvicorn backend:app --reload
The API starts after a couple of seconds and everything is okay. However, recently I decided to install pdfkit to generate some PDF documents that need to be sent back to a webapp. However, after I did (using this this page as reference):
pipenv install pdfkit
Now, whenever I try to run the same uvicorn command to start my API, the API returns the error module pdfkit not found. I've checked that the package is properly installed and it does appear on my Pipfile and whenever I type pip freeze once my enviroment has been activated.
Here's a snippet of the API call that returning the error:
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse, StreamingResponse
from scipy.optimize import leastsq
from openpyxl import load_workbook
from openpyxl.utils.dataframe import dataframe_to_rows
from openpyxl.styles import Font
from openpyxl import Workbook
import pandas as pd
import numpy as np
import json
import os
# ==========================
# SETUP
# ==========================
app = FastAPI()
# ==========================
# CORS
# ==========================
origins = [
"http://127.0.0.1:5500/",
"http://127.0.0.1:8000/",
"http://localhost"
]
app.add_middleware(
CORSMiddleware,
allow_origins=["*"], # List of origins that are allowed to make cross-origin requests
allow_credentials=True, # Indicate that cookies should be supported for cross-origin requests
allow_methods=["*"], # List of HTTP methods that should be allowed for cross-origin requests
allow_headers=["*"], # List of HTTP headers that should be supported for cross-origin requests
)
# ==========================
# GET: PDF RENDERING
# ==========================
#app.get("/pdf")
def render_pdf(pdf_content: str):
from jinja2 import Template, Environment, FileSystemLoader
import pdfkit as pdf
# Jinja2 template "generator" or "main" object (AKA Environment)
env = Environment(loader = FileSystemLoader('PDF Rendering'), trim_blocks=True, lstrip_blocks=True)
# Template parameters
template_params = {
"BASE_PATH": os.getcwd(),
"REPORT_CONTENT": pdf_content,
}
# Render template
template = env.get_template("pdf_template.j2")
template_text = template.render(template_params)
with open("PDF Rendering/render.html", "w") as f:
f.write(template_text)
# Turn rendered file to PDF
pdf.from_file("PDF Rendering/render.html", "PDF Rendering/render.pdf")
return {
"message": "Success"
}
(The Jinja section works mind you, but for some reason, the pdfkit one doesnt)

Related

Minimal FastAPI Static File Server Script

I want to write a minimal FastAPI static file server launched from a script that allows you to specify the directory to share on the command line. Following the example in the FastAPI documentation, I wrote this.
import uvicorn
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles
server = FastAPI()
if __name__ == "__main__":
import sys
directory = sys.argv[1]
server.mount("/static", StaticFiles(directory=directory), name="static")
uvicorn.run(app="my_package:server")
If I run this with the argument /my/directory where this directory contains file.txt I expect that I'd be able to download file.txt at the URL http://localhost:8000/static/file.txt, but this returns an HTTP 404.
How do I write this minimal static file server script?
The assumption I made about sys.argv not being available when uvicorn loads your module is wrong, so it should work as you expect by moving your static setup outside of the __main__ guard:
import uvicorn
import sys
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles
server = FastAPI()
directory = sys.argv[1]
server.mount("/static", StaticFiles(directory=directory), name="static")
if __name__ == "__main__":
uvicorn.run(app="my_package:server")
When you call uvicorn.run(app="my_package:server"), it actually starts a separate process where my_package is imported. Therefore, everything inside if __name__ == "__main__": will not be run in the uvicorn process, so your directory will never be mounted.
One possible solution would be getting the directory from an environment variable, which is set from a small bash script:
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles
server = FastAPI()
directory = os.getenv("DIRECTORY")
server.mount("/static", StaticFiles(directory=directory), name="static")
start.sh:
#!/usr/bin/env bash
DIRECTORY=$1 uvicorn mypackage:server

How to create an api with FastAPI and scrapy?

I am working on a web application project which will allow user to search from keywords and get a set of results based on the keywords entered from scraping. For this I use scrapy for scraping results from a web search engine. I wrote some code to pass the keywords to the scrapy file and display the scrapy results on a webpage. However, I'm having trouble passing the keywords to scrapy using FastAPI, because when I run my api code, I always get a set of errors from Scrapy. Here is the gist containing the runtime terminal output. I don't understand the problem, yet my scrapy code was working perfectly before I connected it to the api, I'm a beginner on creating APIs, so I ask for your help. Here is the code for my web scraper:
import scrapy
import datetime
from requests_html import HTMLSession
class PagesearchSpider(scrapy.Spider):
name = 'pageSearch'
def start_requests(self,query):
#queries = [ 'investissement']
#for query in queries:
url = f'https://www.ask.com/web?q={query}'
s = HTMLSession()
r = s.get
print(r.status_code)
qlist[""]
yield scrapy.Request(url, callback=self.parse, meta={'pos': 0})
def parse(self, response):
print('url:', response.url)
start_pos = response.meta['pos']
print('start pos:', start_pos)
dt = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
items = response.css('div.PartialSearchResults-item')
for pos, result in enumerate(items, start_pos+1):
yield {
'title': result.css('a.PartialSearchResults-item-title-link.result-link::text').get().strip(),
'snippet': result.css('p.PartialSearchResults-item-abstract::text').get().strip(),
'link': result.css('a.PartialSearchResults-item-title-link.result-link').attrib.get('href'),
'position': pos,
'date': dt,
}
qlist.append(items)
# --- after loop ---
next_page = response.css('.PartialWebPagination-next a')
if next_page:
url = next_page.attrib.get('href')
print('next_page:', url) # relative URL
# use `follow()` to add `https://www.ask.com/` to URL and create absolute URL
yield response.follow(url, callback=self.parse, meta={'pos': pos+1})
# --- run without project, and save in file ---
from scrapy.crawler import CrawlerProcess
c = CrawlerProcess({
#'USER_AGENT': 'Mozilla/5.0',
# save in file CSV, JSON or XML
'FEEDS': {'test.json': {'format': 'json'}},
#'ROBOTSTXT_OBEY': True, # this stop scraping
})
c.crawl(PagesearchSpider)
c.start()
the code allowing the functioning of my api:
from fastapi import FastAPI
from script import PagesearchSpider
app = FastAPI()
request = PagesearchSpider()
#app.get("/{cat}")
async def read_item(cat):
return request.start_requests('cat')
I changed part of my scraper code, viz:
def start_requests(self,query):
#queries = [ 'investissement']
#for query in queries:
url = f'https://www.ask.com/web?q={query}'
if __name__ == '__main__':
s = HTMLSession()
r = s.get
print(r.status_code)
qlist[""]
but I still get the same errors. Unless I got the wrong crawler code, I'm not very good at scrapy
I also rewrote the code below my function...
# --- run without project, and save in file ---
if __name__ == "__main__":
from scrapy.crawler import CrawlerProcess
c = CrawlerProcess({
#'USER_AGENT': 'Mozilla/5.0',
# save in file CSV, JSON or XML
#'FEEDS': {'test.json': {'format': 'json'}},
#'ROBOTSTXT_OBEY': True, # this stop scraping
})
c.crawl(PagesearchSpider)
c.start()
and I executed this command python -m uvicorn main:app --reload to start server and I had the following results in the command line :
←[32mINFO←[0m: Will watch for changes in these directories: ['C:\\Users\\user\\Documents\\AAprojects\\Whelpsgroups1\\searchApi\\apiFast']
←[32mINFO←[0m: Uvicorn running on ←[1mhttp://127.0.0.1:8000←[0m (Press CTRL+C to quit)
←[32mINFO←[0m: Started reloader process [←[36m←[1m10956←[0m] using ←[36m←[1mstatreload←[0m
←[33mWARNING←[0m: The --reload flag should not be used in production on Windows.
←[32mINFO←[0m: Started server process [←[36m6724←[0m]
←[32mINFO←[0m: Waiting for application startup.
←[32mINFO←[0m: Application startup complete.
←[33mWARNING←[0m: StatReload detected file change in 'main.py'. Reloading...
←[33mWARNING←[0m: The --reload flag should not be used in production on Windows.
←[32mINFO←[0m: Started server process [←[36m2720←[0m]
←[32mINFO←[0m: Waiting for application startup.
←[32mINFO←[0m: Application startup complete.
But when I click from the command line on the link of the address my server started at, it opens my file explorer on windows 10 and when I manually write that link i.e. http://127.0.0.0:8000/ in my browser search bar it says 127.0.0.0 took too long to respond. Yet I didn't change any of my files, just the command I was using in console line so I don't know why this error.
I looked at the stack overflow questions, but they weren't directly related to the problem of difficulty sharing data between an api and a web scraper and on the internet I couldn't find any relevant answers. So I hope you could help me, I look forward to your answers, thank you!

Import "requests" could not be resolved from sourcePylancereportMissingModuleSource (module) requests

I am having an error while importing the library request:
Import "requests" could not be resolved from sourcePylancereportMissingModuleSource (module) requests
In addition, flask, flask_restful and pandas are not working too, however, random works fine.
How should I fix this error?
Thanks.
Here's an example below:
import requests
x = input("write a file copy path: ")
f = open(x, "r")
lines = f.readlines()
fixed_websites = []
for website in lines:
fixed_website = website.strip("\n")
fixed_websites.append(fixed_website)
print(fixed_websites)
for site in fixed_websites:
r = requests.get(f"https://{site}")
print(f"the content of: {site} is {r.content}")
if r.status_code != 200:
print(f"Error: bad status_code for site: {site}, status_code {r.status_code}")
close(x, "r")
First try
pip uninstall requests
then
pip install requests
next time you'll import 'requests' you shouldn't have any problems

How to start Bokeh server programmatically in dev / debug mode

I'm still developing my project and I want to make a lot of changes to UI through HTML/CSS, I'm starting the Bokeh server programmatically and it looks like this:
from tornado.ioloop import IOLoop
from tornado.web import StaticFileHandler
from bokeh.server.server import Server
from bokeh.application import Application
from bokeh.application.handlers.function import FunctionHandler
from os.path import dirname, join
import sys
if not dirname(__file__) in sys.path: sys.path.append(dirname(__file__))
from models.ProjectDataLoading import modify_page1
from models.Empty import modify_page2
page1_app = Application(FunctionHandler(modify_page1))
page2_app = Application(FunctionHandler(modify_page2))
StaticFileHandler.reset()
io_loop = IOLoop.current()
server = Server(applications = {'/ProjectDataLoading': page1_app,
'/PrimaveraObjects': page2_app,
'/SpecializedReports': page2_app,
'/StatisticalReports': page2_app,
'/Predictions': page2_app},
extra_patterns=[(r'/static/css/(.*)', StaticFileHandler, {'path':join(dirname(__file__),"static","css")})],
io_loop = io_loop, port = 5006, static_hash_cache=False, debug=True, autoreload=True, compiled_template_cache=False, serve_traceback=True)
server.start()
server.show('/ProjectDataLoading')
io_loop.start()
I'm still working on page1 -> "ProjectDataLoading", and as you can see I'm trying to make the server to stop cashing the static files with the following codes:
StaticFileHandler.reset()
static_hash_cache=False
debug=True
of course, it's wrong, because it's related to tornado application, not Bokeh server, so, how can I do it Bokeh server?
Almost all the code to support dev mode, i.e. watching a list file files and auto-reloading, is in the application part of the Bokeh server, not in the library. So you would need to essentially reproduce the code here:
https://github.com/bokeh/bokeh/blob/8f9cf0a85ba51098da2d027150e900bfc073eeba/bokeh/command/subcommands/serve.py#L785-L816

Qpython import modules in script

I have installed qpython for android, the problem is that when I install a module with pip in the python console I can import it properly, but not when I try to import it to a script.
in console ... I type:
>>>import requests
>>>requests
<module 'requests' from '/data/data/com.hipipal.qpyplus/.../__init__.py>
but in a script saved in scripts' folder, when I execute:
import requests
r = requests.get("http://www.google.com")
print r.text.encode('utf-8')
I get this:
import requests
ImportError: no module named requests
Can anybody help with that?
Thank you!
Add import site.
This works on my tablet:
#-*-coding:utf8;-*-
#qpy:2
#qpy:console
import site
import requests
r = requests.get("http://www.google.com")
print r.text.encode('utf-8')

Resources