I was assigned with the re-architecture of a legacy (medical) product which is controlling several external devices. In the current architecture, we have several such stations in each customer's network, where each station is processing its own data, and they all share some of that data via a central server (that talks to the DB and BLOB storage).
I'm planning the new architecture such that it will allow more scenarios, such as monitoring the stations through a web interface, and allowing data processing to be scalable by adding additional servers.
This led me to choose NServicebus as the messaging and communication infrastructure. And I pretty much have a clear view of the new architecture.
However, another factor was recently added to the equation by my manager. He requires that the machine that communicates with the devices (hardware), will not be under the IT policies of the customer. The reason behind this, as I understand, is that we don't want the customer's IT to control OS updates, security, permissions and other settings, because we want full control over that machine in order to work properly with our hardware.
My manager thus added a requirement that this machine will be disconnected from the customer's LAN.
If I still want to deploy NServiceBus on that separated machine (because I want to pub/sub async messages to other machines - some are on the customer's LAN and some aren't), Will it require some special deployment? Will it require an NServiceBus gateway?
EDIT: I removed the other (1st) question, as it wasn't relevant to the scope of StackOverflow.
Regarding question 2, yes it would require the use of a "Gateway", however the current NServiceBus Gateway implementation does not support pub/sub so you would have to look at alternatives.
Related
I'm in the process of starting the design of the networks (VPC, subnetworks and such) as part of the process of moving a rather complex organization on-premise structure, on the cloud.
The chosen provider is GCP and I read and taken the courses to be associate engineer. However, the courses I've followed don't go into details of the technical aspects of doing something like this, just present you with the possible options.
My background is of a senior backend, then fullstack, developer. So I lack some of the very interesting and useful knowledge of a sysadmin unfortunately.
Our case is as follows:
On premise VMs on several racks, reachable only inside a VPN
Several projects on the GCP Cloud
Two of them need to connect to the on-premise VPN but there could be more
Some projects see each other resources (VMs, SQL, etc) using VPC Peering
Gradually we will abandon the on-premise, unless we find some legacy application that really is messed up
Now, I could just create a new VPN connection for every project from Hybrid Connectivity -> VPN but I'd rather create a project dedicated to having the VPN gateway set up and allow other projects to use that resources.
Is this a possible configuration? Is it a valid design? As far as I explored the VPN creation, it seems that I'll have to create a VM that will expose an IP acting as gateway, if that's the case I was thinking to be using the VPC peering to allow other projects to exit into the on premise VPN. No idea if I'm talking gibberish here. I'm still waiting for some information (IKE shared key, etc) before attempting anything, so I'm rather lost at this point.
You have to take in consideration several aspect:
Cost: if you set up a VPN in each project, and if you have to double your connectivity for HA, it will be expensive. If you have only 1 gateway project, it's cheaper
Cheaper, imply trade off. VPN have limited bandwidth: 3Gbps (Cloud Interconnect also, but higher and more expensive). If all your projects use the same VPN thanks to mutualization, take care at this bottleneck.
If you want to mutualise, at least for DEV/UAT project, I recommend you to use VPC Peering, I mean 1 VPN project, and others with VPC peering. Take care at your IP range assign for peering. If you are interested, I wrote an article on this
It's also possible to use Shared VPC, which is great! But there is less compatibility with several product (for example, serverless VPC Connector for Cloud Function and App Engine isn't yet compliant with shared VPC).
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
The Problem
My application works as follows:
Multiple (< 20) device clients (Android) are running at a single location.
Thousands of locations exist (therefore tens or hundreds of thousands of device clients exist).
A web portal client also exists that works in sync with each location's data and its device clients.
New data generated on a device is posted to the server (cloud) via a REST API (ASP.net WebAPI).
So far this application is a pretty standard application with a mobile device client and web portal client.
However, due to requirements on each device client that is out of my control (device clients need to function in offline mode, reduce network latency, etc), each device client does not use the server database as its immediate source of record. Each device client has its own local database (SQLite) that stays in sync with all data for its location. For example: when I make a data change on device client A, that change needs to be propagated to device client B and to web portal client C.
The web portal client reads directly from the server database since it does not need offline functionality.
As you can see, the problem here is that we now need a way to keep all device client databases in sync with each other in real time. Brief delays in data being in sync between two device clients is expected and considered okay.
Proposed Solution
My proposed solution is as follows:
When a new client device comes online initially, it receives a data dump for what it has missed since the last time it was online from the server via REST API.
Each new data item posted/updated/deleted from client devices via REST API is propagated through to the server database. The server database houses all data for all locations and should be considered as the permanent source of record.
The web portal works directly with the server database since it has no offline type requirements.
A connection from each client device is established to a data sync stream service via SignalR.
A worker service is "tailing" the server database for new Create/Update/Delete operations. When a CUD operation is detected, a message is dispatched to an Azure Service Bus queue/subscription (via fan-out topic) for each data sync service instance. This allows for horizontal scaling of the SignalR data sync service (with an Azure Service Bus backplane) since thousands of device client connections will exist.
The data sync service reads from its message queue/subscription and pushes a sync message (containing all needed data for the sync) to all connected client devices (for the location related to the data) via SignalR.
The following diagram illustrates this solution:
Large blocks depict servers (gray squares are HTTP web servers that can be horizontally scaled)
Arrows depict the direction of data flowing through the application.
Questions
Is SignalR the right technology for this problem/solution? Originally my solution involved each client device establishing it's own Azure Service Bus queue/subscription that collected messages from the database-tailing worker (sync river). The problem with that solution is that I would be pushing lots of wasted messages to offline device clients that may not come back online for a very long time, if ever. By dumping back the delta data when a device client comes online initially and streaming data via SignalR thereafter I can solve this.
I have not used SignalR extensively in a production environment before, so I am a bit new with it. What problems/challenges can I expect to experience with it for this solution?
The following article states that "There are some scenarios where a backplane can become a bottleneck. Here are some typical SignalR scenarios: High-frequency realtime (e.g., real-time games): A backplane is not recommended for this scenario.". Would this solution fall into this category? What problems could the backplane of Azure Service Bus messaging introduce? How else would I scale this solution if not in this way?
Your general opinions and recommendations for this solutions are also welcome and appreciated.
You have a requirement on real-time communication to devices when they are online. One of the most promising ways to do this is by using web sockets.
Using web socket itself is not practical and so there are popular
libraries for it such as SignalR, socket.io. These libraries absorb
many difficulties faced in production and also in development. These
libraries even support scaling.
Since your stack is .net based SignalR is choice here.
SignalR will work well in most of the cases. Here you don't have to
worry on backplane becoming a bottleneck as given in a real -time
games.
But maintaining a self-hosted real-time solution such as SignalR comes with a cost. The success rate of communication will be not high reliable in stock SignalR and you will have to implement various monitoring mechanisms and failover processes. Geo-distribution also not supported. So the next choice for a high reliable real-time system which addresses all mention issues is a hosted service such as pub-nub.
*If you think I should ask this question elsewhere, please let me know.
Background:
I need to build an application for converting weights into piece counts. The weights currently come from scales that are connected to PCs via serial ports. I am replacing PC based applications that connect to the scales via a serial connection. I am considering the feasibility of making the next generation of these applications into a web based solution. However, I do not want to do this if it is not a better solution than building an application that runs on the client. In addition, I do not want to use any sort of browser specific technology (ActiveX).
FYI, we currently run a Windows based environment.
What I have so far:
I am currently thinking that I will need some sort of client side “service” to allow the scale data to be retrieved by the web application. I have looked into creating a WCF service for this task and have determined that it would probably work. This would require that the scale be connected to some sort of Windows based computer that is on the network. I would then interface the WCF service (running as a Windows Service on the PC) from an ASP.NET web application running on an IIS web server. This would minimize the footprint on the client and allow us to use a web application.
I am looking for any constructive thoughts and ideas. I am open to reviewing any feasible option that would make this solution as simple and reliable as possible.
Answering my own question per request #honeycomb.
I discovered two viable options for this purpose. Following are high-level overviews of the techniques we leveraged.
Develop a scale reader to be run on a PC connected to the weigh scale device via an RS-232 connection. This reader will forward any information received from the scale into a database. Combined with technologies like change notifications and server-side push notifications, this option will allow data from a weigh scale to be pushed into a web page with little effort and no additional cost. (This option has performed well during testing but is not yet in production)
Invest in converting weigh scale devices to use ethernet connections and connect them to the network. Use an OPC server with a driver that can connect to the weigh scales you are using to read the data from these devices. Consider KEPWare's offering for this purpose. Use KEPWare's tools to forward this data to a database or wherever it is needed. Once again, you can leverage change notifications and server-side push technologies to push this data into web applications in near real-time without polling. (This option is currently working in a critical, production environment)
The second option is probably better in the long-term, but this may vary based on your specific situation. It has some up front costs and would be better suited to new implementations. For my system, I am using the first option because it will ease the transition between the new and old systems.
Note: I am not in any way associated with KEPWare. I am only suggesting their product because it is the only one I am aware of that supports this functionality. I am sure there are other OPC servers that support this type of device.
I'm researching about SDN and NFV.
In the concept of NFV on Wikipedia , it says : "Network Functions Virtualization (NFV) is a network architecture concept that proposes using IT virtualization related technologies, to virtualize entire classes of network node functions into building blocks that may be connected, or chained, together to create communication services."==> first thing to consider that it will reduce the cost of facilities.
So in real life implementation, for example, how can we virtualize a network nodes like a router?
NFV was created for the networks to be capable to extend in a dynamically way(virtualize the router) , not a static way(buy a new router), that is we must implement the router functions in the server or a computer instead of buying and then adapting the new router to the current nextwork , in this case I don't see any different in this implementation , because buying a server to implement a virtualized router is not cheaper than buying a new router.
Can anyone explain this for me , or Am i wrong understanding the NFV concept?
Thanks.
SDN is just that, software defined networking. In a Hybrid SDN model SDN decouples the logic from the physical box, rendering the physical box a simple "forwarding" box. The logic rests with the SDN controller where developers create APIs that manage these forwarding boxes (we call them network elements now) with flow tables that get pushed to them. The benefit here is that the devices can now be configured and provisioned through this controller, as opposed to having to log into each and every box.
Then you have the cloud. A small office can literally get away with porting all of their apps and services into the cloud, doing away with most of their physical boxes. Of course you still need a LAN in the office and a way to get out to the Internet and eventually the cloud. You can even ask the cloud provider to provision load-balancing on specific applications, firewalls and content delivery services. So basically your office applications and most of the supporting LAN and databases can be safely ported to cloud providers.
When you said "...because buying a server to implement a virtualized router is not cheaper than buying a new router", it depends: As it's a virtualized resource, you can use this new server to run your router and another resource from your infrastructure, if the machine has more hardware capacity than you need for a single router.
In fact, you might not even need to buy a new machine, if you have your resources in a cloud like AWS (or your own private cloud), when you have need for more routers, you can just flexibly allocate more hardware resources and spawn a new router instance (scale out) and, whenever your router demand is lower than what you have allocated, you can reduce your number of routers (scale in) and stop losing money with an infrastructure that you are not using at the moment.
Consider that a really high level explanation, if you want to know the details about how a Virtual Network Function scales in and out in a NFV implementation, I recommend you to read the ETSI specification about how it should work: http://www.etsi.org/standards-search#page=1&search=&title=1&etsiNumber=1&content=0&version=1&onApproval=1&published=1&historical=0&startDate=1988-01-15&endDate=2017-04-13&harmonized=0&keyword=&TB=789,,832,,831,,795,,796,,800,,798,,799,,797,,828&stdType=&frequency=&mandate=&collection=&sort=3
Let me continue with your example of the router. Traditionally, these routers are vendor specific. For example, the major sellers are companies like Cisco, Juniper, etc. They are implemented on proprietary hardware and therefore if you want to buy a new router you need to buy from them only. Further, when they go into some problems, you need a dedicated engineer to repair them. Therefore, the telecommunication has to take care of high Capital Expenditure (COPEX) and Operational Expenditure (OPEX).
With NFV, the entire router function is implemented as a software and deployed on a general purpose servers (GPP) or cloud. These GPPs are relatively very cheap when compared to proprietary hardware. Thanks to cloud computing, even small companies can afford servers on Amazon and Google clouds. Because of cheap availability, COPEX is now relatively cheaper. Further, you don't need a dedicated engineer when the hardware goes into a problem, the same engineer who works for GPP server maintenance is enough. This way OPEX is reduced.
Now imagine, like routers there are many networking elements present in Telecommunication. If every networking element requires a dedicated engineer, how much a Teleco operator will be spending money. Apart from this, due to software implementation, suppose, when you have very high traffic than expected, you can just roll out a new router (software network function) on GPP or Cloud instead of completely buying a new router, which is very costly. As you already know, in the cloud you pay based on usage.
There are many more uses. To know more you need to read research papers.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I'm about to start writing a web app (Asp.Net/IIS7) which will be accessible over the internet. It will be placed behind a firewall which accepts http and https.
The previous system which we are going to replace doesn't let this web server talk directly to a database, but rather have it making highly specialized web service calls (through a new firewall which only allows this kind of calls) to a separate app server which then go to the DB to operate on the data.
I have worked on many systems in my day, but this is the first one which has taken security this seriously. Is this a common setup? My first thought was to use Windows Authentication in the connectionstring on the web server and have the user be a crippled DB-user (can only view and update its own data) and then allow DB access through the inner firewall as well.
Am I Naïve? Seems like I will have to do a lot of mapping of data if we use the current setup for the new system.
Edit: The domain of this app is online ordering of goods (Business to business), Users (businesses) log in, input what they can deliver at any given time period, view previous transaction history, view projected demand for goods etc. No actual money is exchanged through this system, but this system provides the information on which goods are available for sale, which is data input to the ordering system
This type of arrangement (DMZ with web server, communicating through firewall with app server, communicating through firewall with db) is very common in certain types of environment, especially in large transactional systems (online corporate banking, for example)
There are very good security reasons for doing this, the main one being that it will slow down an attack on your systems. The traditional term for it is Defence in Depth (or Defense if you are over that side of the water)
Reasonable security assumption: your webserver will be continually under attack
So you stick it in a DMZ and limit the types of connection it can make by using a firewall. You also limit the webserver to just being a web server - this reduces the number of possible attacks (the attack surface)
2nd reasonable security assumption: at some point a zero-day exploit will be found that will get to your web server and allow it to be compromised, which could lead to to an attack on your user/customer database
So you have a firewall limiting the number of connections to the application server.
3rd reasonable security assumption: zero-days will be found for the app server, but the odds of finding zero-days for the web and app servers at the same time are reduced dramatically if you patch regularly.
So if the value of your data/transactions is high enough, adding that extra layer could be essential to protect yourself.
We have an app that is configured similarly. The interface layer lives on a web server in the DMZ, the DAL is on a server inside the firewall with a web service bridging the gap between them. In conjunction with this we have an authorization manager inside the firewall which exposes another web service that is used to control what users are allowed to see and do within the app. This app in one of our main client data tracking systems, and is accessible to our internal employees and outside contractors. It also deals with medical information so it falls under the HIPAA rules. So while I don’t think this set up is particularly common it is not unheard of, particularly with highly sensitive data or in situations where you have to deal with audits by a regulatory body.
Any reasonably scalable, reasonably secure, conventional web application is going to abstract the database away from the web machine using one or more service and caching tiers. SQL injection is one of the leading vectors for penetration/hacking/cracking, and databases often tend to be one of the more complex, expensive pieces of the overall architecture/TOC. Using services tiers allows you to move logic out of the DB, to employ out-of-process caching, to shield the DB from injection attempts, etc. etc. You get better, cheaper, more secure performance this way. It also allows for greater flexibility when it comes to upgrades, redundancy or maintenance.
Configuring the user's access rights seems like a more robust solution to me. Also your DataAccess layer should have some security built in, too. Adding this additional layer could end up being a performance hit but it really depends on what mechanism you're using to move data from "WebServer1" to "WebServer2." Without more specific information in that regard, it's not possible to give a more solid answer.