how to do async test(pytest) in fastapi with error `There is 1 other session using the database` - asynchronous

While I was writing the Async test code for FASTAPI there is a problem that cannot be solved. this code is for test db. I'm using postgres and in order to user db as a test, I created is_testing function. It drop and create test db.
if is_testing:
db_url = self._engine.url
if db_url.host != "localhost":
raise Exception("db host must be 'localhost' in test environment")
except_schema_db_url = f"{db_url.drivername}://{db_url.username}#{db_url.host}"
schema_name = db_url.database # test
temp_engine = create_engine(except_schema_db_url, echo=echo, pool_recycle=pool_recycle, pool_pre_ping=True)
conn = temp_engine.connect()
try:
conn = conn.execution_options(autocommit=False)
conn.execute("ROLLBACK")
conn.execute(f"DROP DATABASE {schema_name}")
except ProgrammingError:
print(f"could not drop the database, probably does not exist.")
conn.execute("ROLLBACK")
except OperationalError:
print("could not drop database because it's being accessed by other users(psql prompt open?)")
conn.execute("ROLLBACK")
print(f"test db dropped! about to create {schema_name}")
conn.execute(f"CREATE DATABASE {schema_name}")
try:
conn.execute(f"create user test with encrypted password test")
except:
print("User already exist")
temp_engine.dispose()
this is conftest.py
#pytest.fixture(scope="session")
def app():
os.environ["API_ENV"] = "test"
return create_app()
#pytest.fixture(scope="session")
def client(app):
Base.metadata.create_all(db.engine)
# Create tables
client = AsyncClient(app=app, base_url="http://test")
return client
#pytest.fixture(scope="function", autouse=True)
def session():
sess = next(db.session())
yield sess
clear_all_table_data(
session=sess,
metadata=Base.metadata,
except_tables=[]
)
sess.rollback()
def clear_all_table_data(session: Session, metadata, except_tables: List[str] = None):
session.execute("SET session_replication_role = 'replica';")
for table in metadata.sorted_tables:
if table.name not in except_tables:
session.execute(table.delete())
session.execute("SET session_replication_role = 'origin';")
session.commit()
I got error sqlalchemy.exc.OperationalError: (psycopg2.errors.ObjectInUse) database "test" is being accessed by other users DETAIL: There is 1 other session using the database. in elb check test.
and I got error TypeError: 'AsyncClient' object is not callable in another api test.
I modified client function in conftest.py
#pytest.fixture(scope="session")
def client(app):
Base.metadata.create_all(db.engine)
return AsyncClient(app=app, base_url="http://test")
I passed one test, but I received the following error from the second test.
ClientState.OPENED: "Cannot open a client instance more than once.",
ClientState.CLOSED: "Cannot reopen a client instance, once it has been closed.",
how can I fix it?
thank you for reading long question!

Related

Accessing task_instance or ti via simpleHttpOperator to do an xcom push

TLDR
In the python callable for a simpleHttpOperator response function, I am trying to push an xcom that has combined information from two sources to a specificied key (a hash of the filename/path and an object lookup from a DB)
Longer Tale
I have a filesensor written which grabs all new files and passes them to MultiDagRun to parallel process the information (scientific) in the files as xcom. Works great. The simpleHttpOperator POSTs filepath info to a submission api and receives back a task_id which it must then read as a response from another (slow running) api to get the result. This I all have working fine. Files get scanned, it launches multiple dags to process, and returns objects.
But... I cannot puzzle out how to push the result to an xcom inside the python response function for the simpleHttpOperator.
My google- and SO and Reddit-fu has failed me here (and it seems overkill to use the pythonOperator tho that's my next stop.). I notice a lot of people asking similar questions though.
How do you use context or ti or task_instance or context['task_instance'] with the response function? (I cannot use "Returned Value" xcom as I need to distinguish the xcom keys as parallel processing afaik). As the default I have context set to true in the default_args.
Sure I am missing something simple here, but stumped as to what it is (note, I did try the **kwargs and ti = kwargs['ti'] below as well before hitting SO...
def _handler_object_result(response, file):
# Note: related to api I am calling not Airflow internal task ids
header_result = response.json()
task_id = header_result["task"]["id"]
api = "https://redacted.com/api/task/result/{task_id}".format(task_id=task_id)
resp = requests.get(api, verify=False).json()
data = json.loads(resp["data"])
file_object = json.dumps(data["OBJECT"])
file_hash = hash(file)
# This is the part that is not working as I am unsure how
# to access the task instance to do the xcom_push
ti.xcom_push(key=file_hash, value=file_object)
if ti.xcom_pull(key=file_hash):
return True
else:
return False
and the Operator:
object_result = SimpleHttpOperator(
task_id="object_result",
method='POST',
data=json.dumps({"file": "{{ dag_run.conf['file'] }}", "keyword": "object"}),
http_conn_id="coma_api",
endpoint="/api/v1/file/describe",
headers={"Content-Type": "application/json"},
extra_options={"verify":False},
response_check=lambda response: _handler_object_result(response, "{{ dag_run.conf['file'] }}"),
do_xcom_push=False,
dag=dag,
)
I was really expecting the task_instance object to be available in some fashion, either be default or configuration but each variation that has worked elsewhere (filesensor, pythonOperator, etc) hasn't worked, and been unable to google a solution for the magic words to make it accessible.
You can try using the get_current_context() function in your response_check function:
from airflow.operators.python import get_current_context
def _handler_object_result(response, file):
# Note: related to api I am calling not Airflow internal task ids
header_result = response.json()
task_id = header_result["task"]["id"]
api = "https://redacted.com/api/task/result/{task_id}".format(task_id=task_id)
resp = requests.get(api, verify=False).json()
data = json.loads(resp["data"])
file_object = json.dumps(data["OBJECT"])
file_hash = hash(file)
ti = get_current_context()["ti"] # <- Try this
ti.xcom_push(key=file_hash, value=file_object)
if ti.xcom_pull(key=file_hash):
return True
else:
return False
That function is a nice way of still accessing the task's execution context when context isn't explicitly handy or you don't want to pass context attrs around to access it deep in your logic stack.

Schema not being passed into Airflow MySqlHook

I'm using the MySqlHook to show results of my SQL query in Airflow logs. All is working except that I want to limit the results to a particular schema. Even though I'm passing in that schema, MySqlHook doesn't seem to be recognizing it.
Here's my function that uses the hook:
def func(mysql_conn_id, sql, schema):
"""Print results of sql query """
print("schema:", schema)
hook = MySqlHook(mysql_conn_id=mysql_conn_id, schema=schema)
df = hook.get_pandas_df(sql=sql)
print("\n" + df.to_string())
the schema that I pass in does come up in the logs from my print statement.
But when I see the results of the second print statement here (the df.to_string()), the connection that shows up in Airflow is without the schema.
INFO - Using connection to: id: dbjobs_mysql. Host: kb-qa.local, Port: None, Schema: , Login: my_user, Password: ***, extra: {}
And the query is running for more than just the given schema.
Looking in the source code it seems like what I did is what it's expecting:
class MySqlHook(DbApiHook):
conn_name_attr = 'mysql_conn_id'
default_conn_name = 'mysql_default'
supports_autocommit = True
def __init__(self, *args, **kwargs):
super(MySqlHook, self).__init__(*args, **kwargs)
self.schema = kwargs.pop("schema", None)

create a Connection to it using your credentials work with Openstack

I created connection like that
https://docs.openstack.org/openstacksdk/latest/user/guides/connect.html
def create_connection(project_id=None) -> Optional[Connection]:
"""
Create connection to Client's Project in Openstack
:param project_id:
:return: Connection
"""
os_project_id = project_id or settings.OPENSTACK.get("PROJECT_ID")
return openstack.connect(
auth_url=settings.OPENSTACK.get("AUTH_URL"),
project_id=os_project_id,
username=settings.OPENSTACK.get("USERNAME"),
password=settings.OPENSTACK.get("PASSWORD"),
user_domain_name=settings.OPENSTACK.get("USER_DOMAIN_NAME"),
project_domain_name=settings.OPENSTACK.get("PROJECT_DOMAIN_NAME"),
app_name="default",
app_version="1.0",
)
# call function create connection
conn = create_connection(project_id)
# get project
project = conn.identity.get_project(project_id)
# conn2 with project id of client
conn2 = conn.connect_as_project(project)
I succeeded get project and created conn2 in first time, but in the second, i couldn't get connection and created conn2.
It was raise exception
HttpException: No Project found for 40e090c7ee32405dae3d2814cce73094: Client Error for url: https://{BASE_URL}:5000/v3/projects/40e090c7ee32405dae3d2814cce73094, You are not authorized to perform the requested action: identity:get_project.
I thought it cached session and used conn like conn2 created

Fastapi auth with OAuth2PasswordBearer, how to check if an user is connected without raise an exception?

For an application, I have followed the fastAPI documentation for the authentification process.
By default, OAuth2PasswordBearer raise an HTTPException with status code 401. So, I can't check if an user is actually connected without return a 401 error to the client.
An example of what I want to do:
app = FastAPI()
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="api/users/token")
def get_current_user(token: str = Depends(oauth2_scheme)):
try:
settings = get_settings()
payload = jwt.decode(token, settings.secret_key,
algorithms=[settings.algorithm_hash])
email = payload.get("email")
if email is None:
raise credentials_exception
token_data = TokenData(email=email)
except jwt.JWTError:
raise credentials_exception
user = UserNode.get_node_with_email(token_data.email)
if user is None:
raise credentials_exception
return user
#app.get('/')
def is_connected(user = Depends(get_current_user)
# here, I can't do anything if the user is not connected,
# because an exception is raised in the OAuth2PasswordBearer __call__ method ...
return
I see OAuth2PasswordBearer class have an "auto_error" attribute, which controls if the function returns None or raises an error:
if not authorization or scheme.lower() != "bearer":
if self.auto_error:
raise HTTPException(
status_code=HTTP_401_UNAUTHORIZED,
detail="Not authenticated",
headers={"WWW-Authenticate": "Bearer"},
)
else:
return None
So i think about a workaround:
app = FastAPI()
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="api/users/token", auto_error=False)
def get_current_user(token: str = Depends(oauth2_scheme)):
if not token:
return None
# [ ... same token decoding logic than before ... ]
return user
#app.get('/')
def is_connected(user = Depends(get_current_user)
return user
It works, but I wonder what other ways there are to do this, is there a more "official" method?
This is a good question and as far as I know, there isn't an "official" answer that is universally agreed upon.
The approach I've seen most often in the FastAPI applications that I've reviewed involves creating multiple dependencies for each use case.
While the code works similarly to the example you've provided, the key difference is that it attempts to parse the JWT every time - and doesn't only raise the credentials exception when it does not exist. Make sure the dependency accounts for malformed JWTs, invalid JWTs, etc.
Here's an example adapted to the general structure you've specified:
# ...other code
oauth2_scheme = OAuth2PasswordBearer(
tokenUrl="api/users/token",
auto_error=False
)
auth_service = AuthService() # service responsible for JWT management
async def get_user_from_token(
token: str = Depends(oauth2_scheme),
user_node: UserNode = Depends(get_user_node),
) -> Optional[User]:
try:
email = auth_service.get_email_from_token(
token=token,
secret_key=config.SECRET_KEY
)
user = await user_node.get_node_with_email(email)
return user
except Exception:
# exceptions may include no token, expired JWT, malformed JWT,
# or database errors - either way we ignore them and return None
return None
def get_current_user_required(
user: Optional[User] = Depends(get_user_from_token)
) -> Optional[User]:
if not user:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="An authenticated user is required for that action.",
headers={"WWW-Authenticate": "Bearer"},
)
return user
def get_current_user_optional(
user: Optional[User] = Depends(get_user_from_token)
) -> Optional[User]:
return user

hiredis run Sync command from Async Context

I'm using the hiredis C client library to interact with Redis in an async context.
On some point of my workflow I have to make a Sync call to Redis but I'm not being able to get a successful response from Redis.
I'm not sure whether I can issue a sync command to Redis from an async context but...
I have something like this
redisAsyncContext * redis_ctx;
redisReply * reply;
// ...
reply = redisCommand(&(redis_ctx->c), COMMAND);
After redisCommand call, my reply is NULL what is documented as an error condition and my redis_ctx->c is like
err = 0
errstr = '\000' <repeats 127 times>
fd = 11
flags = 2
obuf = "*5\r\n$4\r\nEVAL\r\n$215\r\n\"math.randomseed(tonumber(ARGV[1])) local keys = redis.call('hkeys',KEYS[1]) if #keys == 0 then return nil end local key = keys[math.random(#keys)] local value = redis.call('hget', KEYS[1], key) return {key, value}\"\r\n$1\r\n1\r\n$0\r\n\r\n$1\r\n1\r\n"
reader = 0x943730
I can't figure out whether the command was issued or not.
Hope it's not too late. I'm not so expert about Redis, but if you need to make a Sync call to Redis, why would you use an AsyncContext?
If you just use redisCommand with a redisContext everything should be fine.
Assuming that variable ctx has been declared as
redisContext *ctx;
you can use redisCommand like this:
reply = (redisReply *)redisCommand(ctx, "HGET %s %s", hash, key);

Resources