I am in the process of delivering a product.
What my product needs to do is spin up and down multiple instances on google cloud.
Basically, I have a main instance which I plan to control the rest of them. A 'worker factory' if you will.
What would be a good way to communicate results from the workers back to the main instance.
The actually worker already inserts its results to a sql table also hosted by google, before spinning down.
However I am looking to additionally send off a log file to the main instance.
What would be the best way of doing this?
p.s - I am not extremely familiar with networking, however I am looking to learn
There are likely many ways to achieve your goal. One thought is to use Publish/Subscribe. In this concept, you create a resource called a topic and your main instance would register a subscription to the topic. Now, when your workers wish to transmit information they would executed a publish on the topic. There is no networking knowledge needed to make this all work. The topic becomes a centralized drop-off for information published.
Here are the docs on pub/sub: https://cloud.google.com/pubsub/docs/
Related
I am working on a small project trying to control some steps of a workflow in a web application using MS teams. My idea is to use R as an intermediate step between the application (which has a number of API endpoints I can call from R) and Microsoft Teams chats (or channels). Users would then use a set of keywords in the chat to lead to an action in the application. For example they might use "publish ABC-123" in a specific chat and this would lead to the application publishing document ABC-123 somewhere via R which would orchestrate everything.
I have a couple of ideas but there are drawbacks:
I thought originally about using microsoft365r. We have an app registered in Microsoft 365 which would allow us to monitor a specific chat for messages that trigger actions in R. The problem with this approach is that we would need to have the R code running and checking MS Teams every couple of minutes. It is certainly doable, but not very elegant.
Another option could be setting up a plumber API and an outgoing webhook in MS Teams. This seems like the ideal way to do it, but webhooks in MS Teams require https and as far as I understand this is not straightforward to implement in plumber.
I would appreciate any ideas on how to do this. I know I am not very specific, but mostly looking for high level pointers of what I could look at. Many thanks!
You actually have a bunch of options for this:
Create a bot directly in code, e.g. per https://learn.microsoft.com/en-us/microsoftteams/platform/bots/what-are-bots . There's a bit of a learning curve of course, and it depends on whether you have development skills outside of r, e.g. python, .net, whatever. The bot would then call your code as needed.
Create a no-code bot using Power Virtual Agents. This is the equivalent, for bots, of Power Apps or Power Automate, if you're familiar with those.
Create a workflow, either in Power Automate or Azure Logic Apps, that can listen for and respond to messages. This is kind of similar to a bot, but with finer scope (and therefore less capability). If you want it to call out to your app, e.g. to an endpoint, you'd need a Premium Connecter for Power Automate, or you can use an Azure Logic App directly (uses the same engine, but the pricing model is different for these and Power Automate is a little easier to work with.
Outgoing webhook - you can implement these as standalone, but actually from your use case it sounds like a bot would be better anyway, and it's kind of what you need to build to make this kind of webhook work properly anyway.
I have so far been very impressed with the firebase platform for hosting a client-side single page app and for data storage. However, I have one component that I don't know where to host...
I want to have a background process that aperiodically updates the database. The nature of when an update is needed is based on an external source and, although the general timeframe of when updates are available is known, the exact timing is not. My thinking was to have a background task running that has some smarts to determine when an update is needed, and then trigger an update at that time.
I don't know where I would host something like this. I considered running it in a loop in a firebase function, but due to pricing model being based on time, that would get very expensive, and functions are not suited for daemon-type processes. The actual "database update" would be suitable for a function, but not the triggering logic. Also, I have seen functions-cron which does offload the triggering logic, but since my updates are not truly periodic, it doesn't seem exactly appropriate. I haven't looked too much into AppEngine and how that relates to the firebase platform...so basically my question:
What are the options for "reasonably-priced" hosting an always-running background task?
Google App Engine - Standard is something you want to look at more. It is reasonably priced since what you are doing will likely fit into GAE-Std's free daily quota. In GAE-Std, you create a scheduled cron job: GAE will call you task as if it was an incoming web request.
See Firebase doc for integrating with GAE
See GAE doc for cron jobs
I'm starting to learn about google's firebase, seems really cool for real time applications. The auto-synced database seems very easy to use and I feel like diving into it.. I plan to start learning by building a simple checkers multiplayer game, but I still have an important question about it..
Firebase auto-syncs between users and devices using their 'magic' Database, which stores data and sends out to 'subscribers' of that db. Now what if I want to have some server processing of this data in between? For example, when a player makes a move, I want something that is not on client-side to make sure that is a valid move.. what would be the architecture to accomplish that?
Having a trusted process that sits between the users is a common scenario when using Firebase. Have a look at our classic blog post Where does Firebase fit in your app?, it would fit closest to pattern 2 there.
Typically you'll want to use the firebase-queue for this. Your users write their "requests" (probably moves in your case) into the queue, the server processes those and updates the actual board.
Another great thing about this is that it's easy to secure. The users can only write to the queue, while the server is the only one that can read the queue and update the board. A lot simpler to capture in security rules than many other approaches.
I have been given access to a real time data feed which provides location information, and I would like to build a website around this, but I am a little unsure on what architecture to use to achieve my needs.
Unfortunately the feed I have access to will only allow a single connection per IP address, therefore building a website that talks directly to the feed is out - as each user would generate a new request, which would be rejected. It would also be desirable to perform some pre-processing on the data, so I guess I will need some kind of back end which retrieves the data, processes it, then makes it available to a website.
From a front end connection perspective, web services sounds like it may work, but would this also create multiple connections to the feed for each user? I would also like the back end connection to be persistent, so that data is retrieved and processed even when the site is not being visited, I believe IIS will recycle web services and websites when they are idle?
I would like to keep the design fairly flexible - in future I will be adding some mobile clients, so the API needs to support remote connections.
The simple solution would have been to log all the processed data to a database, which could then be picked up by the website, but this loses the real-time aspect of the data. Ideally I would be looking to push the data to the website every time the data changes or now data is received.
What is the best way of achieving this, and what technologies are there out there that may assist here? Comet architecture sounds close to what I need, but that would require building a back end that can handle multiple web based queries at once, which seems like quite a task.
Ideally I would be looking for a C# / ASP.NET based solution with Javascript client side, although I guess this question is more based on architecture and concepts than technological implementations of these.
Thanks in advance for all advice!
Realtime Data Consumer
The simplest solution would seem to be having one component that is dedicated to reading the realtime feed. It could then publish the received data on to a queue (or multiple queues) for consumption by other components within your architecture.
This component (A) would be a standalone process, maybe a service.
Queue consumers
The queue(s) can be read by:
a component (B) dedicated to persisting data for future retrieval or querying. If the amount of data is large you could add more components that read from the persistence queue.
a component (C) that publishes the data directly to any connected subscribers. It could also do some processing, but if you are looking at doing large amounts of processing you may need multiple components that perform this task.
Realtime web technology components (D)
If you are using a .NET stack then it seems like SignalR is getting the most traction. You could also look at XSockets (there are more options in my realtime web tech guide. Just search for '.NET'.
You'll want to use signalR to manage subscriptions and then to publish messages to registered client (PubSub - this SO post seems relevant, maybe you can ask for a bit more info).
You could also look at offloading the PubSub component to a hosted service such as Pusher, who I work for. This will handle managing subscriptions and component C would just need to publish data to an appropriate channel. There are other options all listed in the realtime web tech guide.
All these components come with a JavaScript library.
Summary
Components:
A - .NET service - that publishes info to queue(s)
Queues - MSMQ, NServiceBus etc.
B - Could also be a simple .NET service that reads a queue.
C - this really depends on D since some realtime web technologies will be able to directly integrate. But it could also just be a simple .NET service that reads a queue.
D - Realtime web technology that offers a simple way of routing information to subscribers (PubSub).
If you provide any more info I'll update my answer.
A good solution to this would be something like http://rubyeventmachine.com/ or http://nodejs.org/ . It's not asp.net, but it can easily solve the issue of distributing real time data to other users. Since user connections, subscriptions and broadcasting to channels are built in to each, that will make coding the rest super simple. Your clients would just connect over standard tcp.
If you needed clients to poll for updates then you would need a que system to store info for the next request. That could be a simple array, or a more complicated que system depending on your requirements and number of users.
There may be solutions for .net that I am not aware of that do the same thing, but those are the 2 I know of.
We have a rather large document routing framework that's currently implemented in SharePoint (with a large set of cumbersome SP workflows), and it's running into the edge of what SP can do easily. It's slated for a rewrite into .NET
I've spent the past week or so reading and watching WF4 discussions and demonstrations to get an idea of WF4, because I think it's the right solution. I'm having difficulty envisioning how the system will be configured, though, so I need guidance on a few points from people with experience:
Let's say I have an approval that has to be made on a document. When the wf starts, it'll decide who should approve, and send that person an email notification. Inside the notification, the user would have an option to load an ASP.NET page to approve or reject. The workflow would then have to be resumed from the send email step. If I'm planning on running this as a WCF WF Service, how do I get back into the correct instance of the paused service? (considering I've configured AppFabric and persistence) I somewhat understand the idea of a correlation handle, but don't think it's meant for this case.
Logging and auditing will be key for this system. I see the AppFabric makes event logs of this data, but I haven't cracked the underlying database--is it simple to use for reporting, or should I create custom logging activities to put around my actions? From experience, which would you suggest?
Thanks for any guidance you can provide. I'm happy to give further examples if necessary.
To send messages to a specific workflow instance you need to set up message correlation between your different Receive activities. In order to do that you need some unique value as part of your message data.
The Appfabric logging works well but if you want to create custom a custom logging solution you don't need to add activities to your workflow. Instead you create a custom TrackingParticipant to do the work for you. How you store the data is then up to you.
Your scenario is very similar to the one I used for the Introduction to Workflow Services Hands On Lab in the Visual Studio 2010 Training Kit. I suggest you take a look at the hands on lab or the Windows Server AppFabric / Workflow Services Demo - Contoso HR sample code.