I have an existing application (WPF) that monitors OPC Servers and alarms. There is a requirement for this to be accessible via a browser so that users can view the status of alarms etc remotely. I'm feeling out of my depth (I'm not a Web developer) and I just need some advice on the best technology to accomplish this.
I've written several WCF Services, but all these have done is, via a function call, crunch some data sending back a result.
This 'service' will have to be persistent and able to be interrogated by x number of clients. For example, a client will need to be able to connect, stay connected and be informed of events as an when they happen. This has been a major problem in the past when I've developed WCF services (channel faults etc) and I've learnt to only keep a connection open for as long as it's needed. Is a WCF Service the best option in this case (as opposed to a normal Window's Service)
I need to be able to 'push' information from the service to clients. So, someone navigates to a webpage, the page shows in realtime, what is happening in the service. Do I need to use timers since this could be big problem if session state cannot be maintained.
I've read about Observer Design Pattern, but can this be implemented in ASP.net and how would ASP connect (and remain connected) to a remote windows service? Again, do I have to resort to timers?
I apologise it this appears vague, but the situation boils down to the following:
A process that's continually running (somewhere), receiving connections from remote clients (desktop/web), and then keeping the clients informed as events (alarms going off etc) occur.
Related
I want to create a program for my desktop and an app for my android. Both of them will do the same, just on those different devices. They will be something like personal assistants, so I want to put a lot of data into them ( for example contacts, notes and a huge lot of other stuff). All of this data should be saved on a server (at least for the beginning I will use my own Ubuntu server at home).
For the android app I will obviously use java and the database on the server will be a MySQL database, because that's the database I have used for everything. The Windows program will most likely be written in of these languages: Java, C#c C++, as these are the languages I am able to use quite well.
Now to the problem/question: The server should have a good backend which will be communicating with the apps/programs and read/write data in the database, manage the users and all that stuff. But I am not sure how I should approach programming the backend and the "network communication" itself. I would really like to have some relatively easy way to send secured messages between server and clients, but I have no experience in that matter. I do have programming experience in general, but not with backend and network programming.
side notes:
I would like to "scale big". At first this system will only be used by me, but it may be opened to more people or even sold.
Also I would really like to a (partly) self programmed backend on the server, because I could very well use this for a lot of other stuff, like some automation features in my house, which will be implemented.
EDIT: I would like to be able to scale big. I don't need support for hundreds of people at the beginning ;)
You need to research Socket programming. They provide relatively easy, secured network communication. Essentially, you will create some sort of connection or socket listener on your server. The clients will create Sockets, initialize them to connect to a certain IP address and port number, and then connect. Once the server receives these connections, the server creates a Socket for that specific connection, and the two sockets can communicate back and forth.
If you want your server to be able to handle multiple clients, I suggest creating a new Thread every time the server receives a connection, and that Thread will be dedicated to that specific client connection. Having a multi-threaded server where each client has its own dedicated Thread is a good starting point for an efficient server.
Here are some good C# examples of Socket clients and servers: https://msdn.microsoft.com/en-us/library/w89fhyex(v=vs.110).aspx
As a side note, you can also write Android apps in C# with Xamarin. If you did your desktop program and Android app both in C#, you'd be able to write most of the code once and share it between the two apps easily.
I suggest you start learning socket programming by creating very simple client and server applications in order to grasp how they will be communicating in your larger project. Once you can grasp the communication procedures well enough, start designing your larger project.
But I am not sure how I should approach programming the backend and
the "network communication" itself.
Traditionally, a server for your case would be a web server exposing REST API (JSON). All clients need to do http requests and render/parse JSON. REST API is mapped to database calls and exposes some data model. If it was in Java, it would be Jetty web server, Jackson Json parser.
I would really like to have some relatively easy way to send secured
messages between server and clients,
Sending HTTP requests probably the easiest way to communicate with a service. Having it secured is a matter of enabling HTTPS on the server side and implementing some user access authentication and action authorization. Enabling HTTPS with Jetty for Java will require few lines of code. Authentication is usually done via OAuth2 technique, and authorization could be based on ACL. You may go beyond of this and enable encryption of data at rest and employ other practices.
I would like to "scale big". At first this system will only be used by
me, but it may be opened to more people or even sold.
I would like to be able to scale big. I don't need support for
hundreds of people at the beginning
I anticipate scalability can become the main challenge. Depending on how far you want to scale, you may need to go to distributed (Big Data) databases and distributed serving and messaging layers.
Also I would really like to a (partly) self programmed backend on the
server, because I could very well use this for a lot of other stuff,
like some automation features in my house, which will be implemented.
I am not sure what you mean self-programmed. Usually a backend encapsulates some application specific business logic.
It could be a piece of logic between your database and http transport layer.
In more complicated scenario your logic can be put into asynchronous service behind the backend, so the service can do it's job without blocking clients' requests.
And in the most (probably) complicated scenario your backend may do machine learning (for example, if you would like you software stack to learn your home-being habits and automate house accordingly to your expectations without actually coding this automation)
but I have no experience in that matter. I do have programming
experience in general, but not with backend and network programming.
If you can code, writing a backend is not very hard problem. There are a lot of resources. However, you would need time (or money) to learn and to do it, what may distract you from the development of your applications or you may enjoy it.
The alternative to in-house developed of a backend could be a Backend-as-a-Service (BaaS) in cloud or on premises. There are number of product in this market. BaaS will allow you to eliminate the development of the backend entirely (or close to this). At minimum it should do:
REST API to data storage with configurable data model,
security,
scalability,
custom business-logic
Disclaimer: I am a member of webintrinsics.io team, which is a Backend-as-a-Service. Check our website and contact if you need to, we will be able to work with you and help you either with BaaS or with guiding you towards some useful resources.
Good luck with your work!
We have a core windows service hosting around 9 WCF service and acting as a client to another 3 WCF services. We have a front-end website that communicates with this windows service through WCF.
At somepoint, the windows services is executing some heavy operations which results in 100% CPU utilization, usually split 60-40 between the windows service and SQL server.
This is where the WCF connection/requests between the website times out, and this results in a very non responsive UI.
I am looking for a way to make sure any UI-related WCF calls gets executed anyway and takes the highest priority.
Our main problem is that we need to stick with this deployment scenario, where the windows service, the website and SQL server are all running on one machine. We are required to maintain a responsive UI even with a 100% CPU utilization. I am not sure where to start looking for a fix for that ...
It sounds like you should split your service endpoint onto two separate hosts, one for high volume, or process-intensive operations and one for low latency operations. The high-volume endpoint would process from a queue offline, and the low-latency endpoint would handle requests synchronously from the UI.
The kind of problems you are having are typical of when you try to balance the conflicting resource needs of high volume and low latency together in the same process.
If you cannot scale out in this way then I can't really suggest much you can do about it and must apologize for not answering your question directly.
Another thing you could look at is moving everything asynchronous and using a pattern such as CQRS to provide separation between your read and write requirements.
I am interested in the Pub/Sub paradigm in order to provide a notifications system (ie : like Facebook), especially in a web application which has publishers (in several web applications on the same web server IIS) and one or more subscribers, in charge to display on the web the notifications for the front user.
I found out Redis, it seems to be a great server which provides interesting features : Caching (like Memcached) , Pub/Sub, queue.
Unfortunately, I didn't find any examples in a web context (ASP.NET, with Ajax/jQuery), except WebSockets and NodeJS but I don't want to use those ones (too early). I guess I need a process (subscriber) which receives messages from the publishers but I don't see how to do that in a web application (pub/sub works fine with unit tests).
EDIT : we currently use .NET (ASP.NET Forms) and try out ServiceStack.Redis library (http://www.servicestack.net/)
Actually Redis Pub/Sub handles this scenario quite well, as Redis is an async non-blocking server it can hold many connections cheaply and it scales well.
Salvatore (aka Mr Redis :) describes the O(1) time complexity of Publish and Subscribe operations:
You can consider the work of
subscribing/unsubscribing as a
constant time operation, O(1) for both
subscribing and unsubscribing
(actually PSUBSCRIBE does more work
than this if you are subscribed
already to many patterns with the
same client).
...
About memory, it is similar or smaller
than the one used by a key, so you
should not have problems to subscribe
to millions of channels even in a
small server.
So Redis is more than capable and designed for this scenario, but the problem as Tom pointed out in order to maintain a persistent connection users will need long-running connections (aka http-push / long-poll) and each active user will take its own thread. Holding a thread isn't great for scalability and technologically you would be better off using a non-blocking http server like Manos de Mono or node.js which are both async and non-blocking and can handle this scenario. Note: WebSockets is more efficient for real-time notifications over HTTP, so ideally you would use that if the users browser supports it and fallback to regular HTTP if they don't (or fallback to use Flash for WebSockets on the client).
So it's not the Redis or its Pub/Sub that doesn't scale here, it's the number of concurrent connections that a threaded HTTP server like IIS or Apache that is the limit, with that said you can still support a fair amount of concurrent users with IIS (this post suggests 3000) and since IIS is the bottleneck and not Redis you can easily just add an extra IIS server into the mix and distribute the load.
For this application, I would strongly suggest using SignalR, which is a .Net framework that enables real-time push to connected clients.
Redis publish/subscribe is not designed for this scenario - it requires a persistent connection to redis, which you have if you are writing a worker process but not when you are working with stateless web requests.
A publish/subscribe system that works for end users over http takes a little more work, but not too much - the simplest approach is to use a sorted set for each channel and record the time a user last got notifications. You could also do it with a list recording subscribers for each channel and write to the inbox list of each of those users whenever a notification is added.
With either of those methods a user can retrieve their new notifications very quickly. It will be a form of polling rather than true push notifications, but you aren't really going to get away from that due to the nature of http.
Technically you could use redis pub/sub with long-running http connections, but if every user needs their own thread with active redis and http connections, scalability won't be very good.
First of all thanks for taking the time to read my question. Here is what I am trying to accomplish followed by what I have so far on this.
What I want to do is create a Windows application (or server of sorts) that would listen for requests from an ASP.NET application. The windows application would be installed and would listen for messages from ASP.NET application and then do some processing. The flow is like this:
A user downloads the desktop application and registers their IP address on my web site. After downloading the desktop app, the ASP.NET application can then send requests to that particular desktop client for further processing. I think further processing is independent of the resolution in this case that's why i have skipped over details on what processing would be done. But if you think it is important, please let me know and I will add those details as well.
I have looked into creating a TCP server that would listen for requests. Because the user has already registered their IP address on my web site, my web site assigns them a unique identifier and stores the ID alongwith IP address in database. Now, the ASP.NET site can send requests to that desktop application.
I have looked into creating a TCP server for this purpose. While researching I also came across PNRP and it seems something like what I am trying to do.
Can you guys recommend some solutions or where I should be looking at for this scenario? Should I create a simple TCPLISTENER or may be go with PNRP approach? Or something else?
The basic requirement is for a web application to be able to communicate with a desktop application. The web application would be servicing numerous users and each user would have a desktop application installed. Which user for which desktop client question would be addressed by the web application that would maintain a database of unique user id's and their corresponding IP Address.
Thanks in advance for your help.
You could use .NET remoting or a web-service in the desktop app. Use WCF or WSE for the latter. You can use COM to add windows firwall rules.
Whatever you do, take firewalling/NAT into account. It might be easier for the client application to poll the server (initiate the connection) otherwise you open a can of worms by trying to have a remotely-accessible server in your user's computer without having to do some very manual configurations on the user's networking equipment.
Once you have that part sorted out, what I used in your situation was .NET Remoting. At the time WCF had not come out and when it did it was to crippled for my needs. TCP IP sockets were too raw (I had to write too much code) and so Remoting solved my problem ideally (a hand full lines of code to set up the connection, and everything was automatic from there on).
EDIT: I use an excellent third party library that makes Remoting even more flexible (flexible enough that I am still waiting for WCF to catch up with the featureset so that I stop using Remoting, and no luck yet!). Check out http://www.genuinechannels.com/ to see all the features they have. It includes making calls from server to client, and that sounds exactly like what you need to do. Check it out.
I am building an ASP.NET website which will collect data from a user and submit it to a 3rd party webservice. The webservice is somewhat unreliable and for this reason there is a backup service.
If a call to the primary service fails (timeout or some other error) then I need to flip a bit in a static class which will trip the system to use the secondary service.
At this point, I need to start polling the primary service (with dummy data) to see if it is back up (at which point I will receive an OK code in return). At this point I need to flip the bit back so that the website starts using the primary service again.
I've had a read of this Should I use a Windows Service or an ASP.NET Background Thread? and I think that separating out the code into a Windows Service would be the cleanest method of performing the polling, but then how would I communicate with the web appication.
One thought I've had is to expose a webservice that the Windows Service could use to communicate into the webapp but this seems both messy and over-kill.
I'd appreciate your thoughts and experiences performing similar tasks.
Thanks
I think the Windows service is the way to go, definitely.
As for the communication between the service and your web site, the best answer depends on the size and scale of your solution. If you are building something that needs to be reliable, I'd suggest you implement some sort of queue between your ASP.NET site and your Windows service. You have a lot of options here too, depending on budget and ability: BizTalk, MSMQ, and SQL Server queues (SSIS). Alternatively if you are looking for something smaller scale, I'd recommend you just stick it into a database table somewhere.
I would avoid using files on the file system because you will encounter issues with file locks and multithreading. I would also avoid directly communicating with the service because you risk losing the in-memory queue if the service fails for any reason.
Edited to add:
If reliability isn't a concern here, you could use a WPF named-pipes hosted service for communication between your website and your Windows service. This avoids much of the overheads normally involved in classic Web Services and is surprisingly quick. The only down-side is that self-hosting a WPF service is tricky and can be difficult to keep the service up.