I'd like to use a pub/sub StreamingPullFuture subscription with discordpy to receive instructions for removing users and sending updates to different servers.
Ideally, I would start this function when starting the discordpy server:
#bot.event
async def on_ready():
print(f'{bot.user} {bot.user.id}')
await pub_sub_function()
I looked at discord.ext.tasks but I don't think this use case fits since I'd like to handle irregularly spaced events dynamically.
I wrote this pub_sub_function() (based on the pub/sub python client docs) but it doesn't seem to be listening to pub/sub or return anything:
def pub_sub_function():
subscriber_client = pubsub_v1.SubscriberClient()
# existing subscription
subscription = subscriber_client.subscription_path(
'my-project-id', 'my-subscription')
def callback(message):
print(f"pubsub_message: {message}")
message.ack()
return message
future = subscriber_client.subscribe(subscription, callback)
try:
future.result()
except KeyboardInterrupt:
future.cancel() # Trigger the shutdown.
future.result() # Block until the shutdown is complete.
Has anyone done something like this? Is there a standard approach for sending data/messages from external services to a discordpy server and listening asynchronously?
Update: I got rid of pub_sub_function() and changed the code to this:
subscriber_client = pubsub_v1.SubscriberClient()
# existing subscription
subscription = subscriber_client.subscription_path('my-project-id', 'my-subscription')
def callback(message):
print(f"pubsub_message: {message}")
message.ack()
return message
#bot.event
async def on_ready():
print(f'{bot.user} {bot.user.id}')
await subscriber_client.subscribe(subscription, callback).result()
This works, sort of, but now the await subscriber_client.subscribe(subscription, callback).result() is blocking the discord bot, and returning this error:
WARNING discord.gateway Shard ID None heartbeat blocked for more than 10 seconds.
Loop thread traceback (most recent call last):
Ok, so this Github pr was very helpful.
In it, the user says that modifications are needed to make it work with asyncio because of Google's pseudo-future implementation:
Google implemented a custom, psuedo-future
need monkey patch for it to work with asyncio
But basically, to make the pub/sub future act like the concurrent.futures.Future, the discord.py implementation should be something like this:
async def pub_sub_function():
subscriber_client = pubsub_v1.SubscriberClient()
# existing subscription
subscription = subscriber_client.subscription_path('my-project-id', 'my-subscription')
def callback(message):
print(f"pubsub_message: {message}")
message.ack()
return message
future = subscriber_client.subscribe(subscription, callback)
# Fix the google pseduo future to behave like a concurrent Future:
future._asyncio_future_blocking = True
future.__class__._asyncio_future_blocking = True
real_pubsub_future = asyncio.wrap_future(future)
return real_pubsub_future
and then you need to await the function like this:
#bot.event
async def on_ready():
print(f'{bot.user} {bot.user.id}')
await pub_sub_function()
Related
I am currently programming a Discord Bot using Discord.py, aiohttp and asyncpraw to work with Reddit API requests. My problem is that every request takes a long time to respond. Do you have any solutions how to improve speed of my code / API request?
When using the /gif Command this function is getting called:
# Function for a GIF from r/gifs
async def _init_command_gif_response(interaction: Interaction):
"""A function to send a random gif using reddit api"""
# Respond in the console that the command has been ran
print(f"> {interaction.guild} : {interaction.user} used the gif command.")
# Tell Discord that Request takes some time
await interaction.response.defer()
try:
submission = await _reddit_api_request(interaction, "gifs")
await interaction.followup.send(submission.url)
except Exception:
print(f" > Exception occured processing gif: {traceback.print_exc()}")
return await interaction.followup.send(f"Exception occured processing gif. Please contact <#164129430766092289> when this happened.")
Which is calling this function to start a Reddit API request:
# Reddit API Function
async def _reddit_api_request(interaction: Interaction, subreddit_string: str):
try:
#async with aiohttp.ClientSession(trust_env=True) as session:
async with aiohttp.ClientSession() as session:
reddit = asyncpraw.Reddit(
client_id = config_data.get("reddit_client_id"),
client_secret = config_data.get("reddit_client_secret"),
redirect_uri = config_data.get("reddit_redirect_uri"),
requestor_kwargs = {"session": session},
user_agent = config_data.get("reddit_user_agent"),
check_for_async=False)
reddit.read_only = True
# Check if Subreddit exists
try:
subreddit = [sub async for sub in reddit.subreddits.search_by_name(subreddit_string, exact=True)]
except asyncprawcore.exceptions.NotFound:
print(f" > Exception: Subreddit \"{subreddit_string}\" not found")
await interaction.followup.send(f"Subreddit \"{subreddit_string}\" does not exist!")
raise
except asyncprawcore.exceptions.ServerError:
print(f" > Exception: Reddit Server not reachable")
await interaction.followup.send(f"Reddit Server not reachable!")
raise
# Respond with content from reddit
return await subreddit[0].random()
except Exception:
raise
My goal is to increase speed of the discord response. Every other function that is not using Reddit API is snappy. So it must be something with my _reddit_api_request Function.
Full Source Code can be found on Github
Let's say I am building a simple chat app using gRPC with the following .proto:
service Chat {
rpc SendChat(Message) returns (Void);
rpc SubscribeToChats(Void) returns (stream Message);
}
message Void {}
message Message {string text = 1;}
The way I often see the servicer implemented (in examples) in Python is like this:
class Servicer(ChatServicer):
def __init__(self):
self.messages = []
def SendChat(self, request, context):
self.messages.append(request.text)
return Void()
def SubscribeToChats(self, request, context):
while True:
if len(self.messages) > 0:
yield Message(text=self.messages.pop())
While this works, it seems very inefficient to spawn an infinite loop that continuously checks a condition for each connected client. It would be preferable to instead have something like this, where the send is triggered right as a message comes in and doesn't require any constant polling on a condition:
class Servicer(ChatServicer):
def __init__(self):
self.listeners = []
def SendChat(self, request, context):
for listener in self.listeners:
listener(Message(request.text))
return Void()
def SubscribeToChats(self, request, context, callback):
self.listeners.append(callback)
However, I can't seem to find a way to do something like this using gRPC.
I have the following questions:
Am I correct that an infinite loop is inefficient for a case like this? Or are there optimizations happening in the background that I'm not aware of?
Is there any efficient way to achieve something similar to my preferred solution above? It seems like a fairly common use case, so I'm sure there's something I'm missing.
Thanks in advance!
I figured out an efficient way to do it. The key is to use the AsyncIO API. Now my SubscribeToChats function can be an async generator, which makes things much easier.
Now I can use something like an asyncio Queue, which my function can await on in a while loop. Similar to this:
class Servicer(ChatServicer):
def __init__(self):
self.queue = asyncio.Queue()
async def SendChat(self, request, context):
await self.queue.put(request.text)
return Void()
async def SubscribeToChats(self, request, context):
while True:
yield Message(text=await self.queue.get())
await room.save()
showing the below error
AttributeError: 'NoneType' object has no attribute 'execute_insert'
startUp code
async def init():
# Here we create a SQLite DB using file "db.sqlite3"
# also specify the app name of "models"
# which contain models from "app.models"
await Tortoise.init(
db_url='sqlite://db1',
modules={'app1': ['app1.models']}
)
# Generate the schema
#await Tortoise.generate_schemas()
#app.on_event("startup")
async def startup_event():
nest_asyncio.apply()
run_async(init())
created a db structure that's why commented await Tortoise.generate_schemas().
Find my post method code below
#app.post("/room/{room_id}")
async def post(request: Request, room_id):
room = models.Room(id=room_id)
await room.save()
return {"message":"created successfully"}
In the #app.on_event("startup") which is an async function, you are calling run_async(init()) which according to the documentation cleans up after itself, and is meant for small scripts only.
Meaning you are creating and then destroying the DB connection. Hence the connection being a None.
Instead, just await it, and handle shutdown event like so:
#app.on_event("startup")
async def startup_event():
nest_asyncio.apply()
await init()
#app.on_event("shutdown")
async def close_orm():
await Tortoise.close_connections()
Edit: Apparently there is also issues with nest_asyncio and just leaving it out makes things work better.
I want develop a web-socket watcher in python in such a way that when I send sth then it should wait until the response is received (sort of like blocking socket programming) I know it is weird, basically I want to make a command line python 3.6 tool that can communicate with the server WHILE KEEPING THE SAME CONNECTION LIVE for all the commands coming from user.
I can see that the below snippet is pretty typical using python 3.6.
import asyncio
import websockets
import json
import traceback
async def call_api(msg):
async with websockets.connect('wss://echo.websocket.org') as websocket:
await websocket.send(msg)
while websocket.open:
response = await websocket.recv()
return (response)
print(asyncio.get_event_loop().run_until_complete(call_api("test 1")))
print(asyncio.get_event_loop().run_until_complete(call_api("test 2")))
but this will creates a new ws connection for every command which defeats the purpose. One might say, you gotta use the async handler but I don't know how to synchronize the ws response with the user input from command prompt.
I am thinking if I could make the async coroutine (call_api) work like a generator where it has yield statement instead of return then I probably could do sth like beow:
async def call_api(msg):
async with websockets.connect('wss://echo.websocket.org') as websocket:
await websocket.send(msg)
while websocket.open:
response = await websocket.recv()
msg = yield (response)
generator = call_api("cmd1")
cmd = input(">>>")
while cmd != 'exit'
result = next(generator.send(cmd))
print(result)
cmd = input(">>>")
Please let me know your valuable comments.
Thank you
This can be achieved using an asynchronous generator (PEP 525).
Here is a working example:
import random
import asyncio
async def accumulate(x=0):
while True:
x += yield x
await asyncio.sleep(1)
async def main():
# Initialize
agen = accumulate()
await agen.asend(None)
# Accumulate random values
while True:
value = random.randrange(5)
print(await agen.asend(value))
asyncio.run(main())
I am invoking multiple async calls of thrift from my code. I would like to wait
for all of them to complete before going on with my next stage.
for (...) {
TNonblockingTransport transport = new TNonblockingSocket(host, port);
TAsyncClientManager clientManager = new TAsyncClientManager();
TProtocolFactory protocolFactory = new TBinaryProtocol.Factory();
AsyncClient c = new AsyncClient(protocolFactory, clientManager, transport);
c.function(params, callback);
}
// I would like to wait for all the calls to be complete here.
I can have a countdown in the callback like wait/notify and get this done. But does the thrift system allow a way for me to wait on my async function call, preferably with a timeout ?
I didnt see any in the TAsyncClientManager or in the AsyncClient. Please help.
Given that it was not possible to do this, I used the sync api client and managed the launch and wait using executors and launchAll. I am leaving this as my answer for people to have an alternative.