I followed https://fastapi.tiangolo.com/tutorial/bigger-applications/ resource to design my app
.....game/urls.py....
from fastapi import APIRouter
router = APIRouter()
#router.post("/", response_model=schemas.GameOut, tags=["games"])
def create_game(game: schemas.GameIn, db: Session = Depends(get_db)):
return Crud.create(db,game,model)
...main.py...
from game import urls as game_urls
app.include_router(game_urls,prefix="/games")
imported everything properly.
When i run uvicorn main:app --reload it is showing "NO attribures 'routes' " error
I am not able to find, what is the mistake i am doing here. Could any one helps me.
It seems you're injecting the entire urls module in your last line;
app.include_router(game_urls, prefix="/games")
^
I believe you should only inject the router object, e.g. (you might want to import just the router here instead)
app.include_router(game_urls.router, prefix="/games")
Also if you have issue with #router not existing make sure you define the APIRouter as router and not web_router = APIRouter()
Related
I'm trying to implement custom XCOM backend.
Those are the steps I did:
Created "include" directory at the main Airflow dir (AIRFLOW_HOME).
Created these "custom_xcom_backend.py" file inside:
from typing import Any
from airflow.models.xcom import BaseXCom
import pandas as pd
class CustomXComBackend(BaseXCom):
#staticmethod
def serialize_value(value: Any):
if isinstance(value, pd.DataFrame):
value = value.to_json(orient='records')
return BaseXCom.serialize_value(value)
#staticmethod
def deserialize_value(result) -> Any:
result = BaseXCom.deserialize_value(result)
result = df = pd.read_json(result)
return result
Set at config file:
xcom_backend = include.custom_xcom_backend.CustomXComBackend
When I restarted webserver I got:
airflow.exceptions.AirflowConfigException: The object could not be loaded. Please check "xcom_backend" key in "core" section. Current value: "include.cust...
My guess is that it not recognizing the "include" folder
But how can I fix it?
*Note: There is no docker. It is installed on a Ubuntu machine.
Thanks!
So I solved it:
Put custom_xcom_backend.py into the plugins directory
set at config file:
xcom_backend = custom_xcom_backend.CustomXComBackend
Restart all airflow related services
*Note: Do not store DataFrames that way (bad practice).
Sources I used:
https://www.youtube.com/watch?v=iI0ymwOij88
We have an airflow HTTP connection by name "service-1-con".
This is working fine when we have used airflow CLI to manage the connections.
Now the same is not working when specified using env variable AIRFLOW_CONN_SERVICE-1-CON.
There are other HTTP connections using env variables which work. Only this one is not working.
We suspect this could be due to the hyphen in the name. But it works when managed through UI and airflow CLI.
Can somebody advise what could be going wrong?
It works for me:
Here's how you can debug:
import os
from airflow.hooks.base_hook import BaseHook
conn_id = 'service-1-con'
env_var = f'AIRFLOW_CONN_{conn_id}'.upper()
os.environ[env_var] = 'ssh://hello:mypassword#'
conn = BaseHook.get_connection(conn_id)
print(conn.get_uri())
Here is the code where the connections are retrieved from env vars.
We are using Apache 1.9.0. I have written a snowflake hook plugin. I have placed the hook in the $AIRFLOW_HOME/plugins directory.
$AIRFLOW_HOME
+--plugins
+--snowflake_hook2.py
snowflake_hook2.py
# This is the base class for a plugin
from airflow.plugins_manager import AirflowPlugin
# This is necessary to expose the plugin in the Web interface
from flask import Blueprint
from flask_admin import BaseView, expose
from flask_admin.base import MenuLink
# This is the base hook for connecting to a database
from airflow.hooks.dbapi_hook import DbApiHook
# This is the snowflake provided Connector
import snowflake.connector
# This is the default python logging package
import logging
class SnowflakeHook2(DbApiHook):
"""
Airflow Hook to communicate with Snowflake
This is implemented as a Plugin
"""
def __init__(self, connname_in='snowflake_default', db_in='default', wh_in='default', schema_in='default'):
logging.info('# Connecting to {0}'.format(connname_in))
self.conn_name_attr = 'snowflake_conn_id'
self.connname = connname_in
self.superconn = super().get_connection(self.connname) #gets the values from Airflow
{SNIP - Connection stuff that works}
self.cur = self.conn.cursor()
def query(self,q,params=None):
"""From jmoney's db_wrapper allows return of a full list of rows(tuples)"""
if params == None: #no Params, so no insertion
self.cur.execute(q)
else: #make the parameter substitution
self.cur.execute(q,params)
self.results = self.cur.fetchall()
self.rowcount = self.cur.rowcount
self.columnnames = [colspec[0] for colspec in self.cur.description]
return self.results
{SNIP - Other class functions}
class SnowflakePluginClass(AirflowPlugin):
name = "SnowflakePluginModule"
hooks = [SnowflakeHook2]
operators = []
So I went ahead and put some print statements in Airflows plugin_manager to try and get a better handle on what is happening. After restarting the webserver and running airflow list_dags, these lines were showing the "new module name" (and no errors
SnowflakePluginModule [<class '__home__ubuntu__airflow__plugins_snowflake_hook2.SnowflakeHook2'>]
hook_module - airflow.hooks.snowflakepluginmodule
INTEGRATING airflow.hooks.snowflakepluginmodule
snowflakepluginmodule <module 'airflow.hooks.snowflakepluginmodule'>
As this is consistent with what the documentation says, I should be fine using this in my DAG:
from airflow import DAG
from airflow.hooks.snowflakepluginmodule import SnowflakeHook2
from airflow.operators.python_operator import PythonOperator
But the web throws this error
Broken DAG: [/home/ubuntu/airflow/dags/test_sf2.py] No module named 'airflow.hooks.snowflakepluginmodule'
So the question is, What am I doing wrong? Or have I uncovered a bug?
You need to import as below:
from airflow import DAG
from airflow.hooks import SnowflakeHook2
from airflow.operators.python_operator import PythonOperator
OR
from airflow import DAG
from airflow.hooks.SnowflakePluginModule import SnowflakeHook2
from airflow.operators.python_operator import PythonOperator
I don't think that airflow automatically goes through the folders in your plugins directory and runs everything underneath it. The way that I've set it up successfully is to have an __init__.py under the plugins directory which contains each plugin class. Have a look at the Astronomer plugins in Github, it provides some really good examples for how to set up your plugins.
In particular have a look at how they've set up the mysql plugin
https://github.com/airflow-plugins/mysql_plugin
Also someone has incorporated a snowflake hook in one of the later versions of airflow too which you might want to leverage:
https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/snowflake_hook.py
I installed Django and was able to double check that the module was in fact in Python, but when attempting to implement basic commands such as runserver or utilize manage.py; I get DJANGO_SETTEINGS_MODULE error. I already used "set DJANGO_SETTINGS_MODULE = mysite.settings" as advised and inserted mysite.settings into the PATH for Python as some documentation online directed me to.
Now instead of undefined it says no such module exists. I can't find anything else in the documentation and I used my test site name instead of "mysite" without any change. Does anyone know what am I missing? All I can find in the module library for Django in my Python is this code.
from future import unicode_literals
from django.utils.version import get_version
VERSION = (1, 11, 5, 'final', 0)
__version__ = get_version(VERSION)
def setup(set_prefix=True):
"""
Configure the settings (this happens as a side effect of accessing the first setting), configure logging and populate the app registry.
Set the thread-local urlresolvers script prefix if `set_prefix` is True.
"""
from django.apps import apps
from django.conf import settings
from django.urls import set_script_prefix
from django.utils.encoding import force_text
from django.utils.log import configure_logging
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
if set_prefix:
set_script_prefix(
'/' if settings.FORCE_SCRIPT_NAME is None else force_text(settings.FORCE_SCRIPT_NAME)
)
apps.populate(settings.INSTALLED_APPS)
Are you sure you wrote properly the environment variable? I'm asking cause I see you get the error DJANGO_SETTEINGS_MODULE (settings has a misspelling)...
I want to use Werkzeug as a local development server and cannot get the DebugApplication middle ware to work as documented - Werkzeug Debugging. Whats wrong here?
import webapp2
from system import config
from werkzeug.debug import DebuggedApplication
from werkzeug.serving import run_simple
application = webapp2.WSGIApplication(routes=config.routes, debug=False, config=config.options)
debugged_application = DebuggedApplication(application)
def main():
run_simple('localhost', 4000, debugged_application, use_reloader=True, use_debugger=True, threaded=True)
if __name__ == '__main__':
main()
I think that DebuggedApplication middleware tries to achieve the same as use_debugger=True, so no need to use both. The problem is that webapp2.WSGIApplication adds its own error handling before it goes through the debugger middleware, thus forbidding werkzeug debugger to see the actual exception.
My solution to this is to extend base WSGIApplication provided by webapp2 to re-raise the original exception. It works with python 2.7, and will pass the exception if and only if debug flag has been set to True in Application constructor.
class Application(webapp2.WSGIApplication):
def _internal_error(self, exception):
if self.debug:
raise
return super(Application, self)._internal_error(exception)
Not sure this is the cleanest possible way to do it, but it works for me.