I have a dashboard type app, that needs to continuously fetch data from literally hundreds of Atom feeds. The current Webforms implementation is using async to build the complete page, using hundreds of threads per request (works fine, as the hardware and network are really fast and there are few users). What would be a good way to architect a SignalR replacement?
I have used SignalR with KnockoutJS for an easy push to UI solution in a web environment.
There are several full solutions over the web like this and that, which have helped me a lot.
This way you can build a page quickly and continue pushing to it while there is still data or new data appearing. using Knockout at the client side will allow you to push the received data straight into the dashboard.
I have made an example here with a progress report :
https://github.com/dragouf/SignalR.Progress
Related
I want to be able to have users on a new site be able to interact with each other. I thought searching for sample code for async controllers would help, but I can't really seem to find much.
I guess I am trying to replicate Node.js/Meteor like interactions where an action one user performs can be shown to other users. I assume this would work using async controllers and ajax, but I have found very little documentation and examples. I'm starting to think this isn't something that can happen in ASP.NET.
Sorry this is such a general question.
Watch Building Real-time Web Apps with ASP.NET SignalR, this video is by the creators of SignalR.
I think this will give you an idea of how SignalR might be the framework you want to use for your idea to build an application where multiple users can interact within a single page.
Here is the scenario:
There's a data source sitting at site A that I could communicate using a set of APIs to get data I need.
I want to build an ASP.NET web application that periodically fetch data from site A and update/store the data in my own database. And periodically process the data and store processing results in my database so that users could browse the results in my web application front-end.
I have no clue how to design the architecture? How to achieve things like periodically communicate with another data source and process data in my database periodically in a web application?
I have very little experience designing web applications. It would be really nice if you could elaborate.
In answer to:
"How to achieve things like periodically communicate with another data source and process data in my database periodically in a web application?"
I do this by creating a web service, then creating a console application. I use Windows Task Scheduler to run the console application at an interval of my choice. The task is run and the web service is called, which communicates with various data sources and processes the data.
Kind of vague to answer.
There are tools that help with the communication with each API, some services provide wrappers for communication. When they are not provided look into something like Hammock as a wrapper.
High level helps, they are not ABSOLUTEs, just tips and thoughts
Follow a mutli-tiered model where you clearly separate your layers
Model
View
Controller
Use an ORM like ServiceStack for data access
Create a small console app to do the processing
Use a schedule job in windows to run it.
DONT do this with something like Quartz ( way too much overhead )
DONT do with with SqlServerAgent, too much overhead, not enough control if you are a .net Programmer
Watch how big your objects and lists are getting you will run out of memory when working with other people's data
Use JSON it is a great format to pass data around within and external to your application
Setup logging make sure it works, other peoples data breaks
Scrub incoming data, you can't assume other people's data is clean.
Profile your application to know where the hot paths are
Write unit tests
Run your unit tests regularly.
Test on multiple browser
Thats probably good for now. Clarify your question and we may be able to give more help.
I have downloaded TheWorldsWorstStackOverflowClone. One of the project is called TheWorldWorsts.ApiWrapper, which basically is the core of accessing the API. There is a class called ApiProxy.cs, which has all the methods for the API call. This is good.
Now what I want to do is I am trying to collect data from this API interface and store it in a database. I know the limit to the API call is 10k per day. I.e: I want to be able to call the method in the ApiProxy class 10k times per day, done automatically. How can I do this?
The non-automatic way would be to create a dummy site where when every time I access the site it does all that process, but this not efficient. It seems that I have to write some kind of a scheduler by deploying a web service, but that is too complicated... as explained here. Any other simpler methods?
A Windows Service or Desktop App might be a better solution than a web application. You are not deploying a web service, you are consuming one using a proxy class, and this does not require you to have a web server or a web site.
You could use a web application to control and monitor progress as your service downloads data, but the actual work is long running and needs to be offloaded to another process or thread so you can tell the user whats going on.
Check out this one
http://stacky.codeplex.com/
This looks what you need, though I am facing some debugging issues, but hope you can figure it out.
We need the ability to send out automatic emails when certain dates occur or when some business conditions are met. We are setting up this system to work with an existing ASP.NET website. I've had a chat with one of the other devs here and had a discussion of some of the issues.
Things to note:
All the information we need is already modelled in the ASP.NET website
There is some business-logic that is required for the email generation which is also in the website already
We decided that the ideal solution was to have a separate executable that is scheduled to run overnight and do the processing and emailing. This solution has 2 main problems:
If the website was updated (business logic or model) but the executable was accidentally missed then the executable could stop sending emails, or worse, be sending them based on outdated logic.
We are hoping to use something like this to use UserControls to template the emails, which I don't believe is possible outside of an ASP.NET website
The first problem could have been avoided with build and deployment scripts (which we're looking into at the moment anyway), but I don't think we can get around the second problem.
So the solution we decided on is to have an ASP.NET page that is called regularly by SSIS and to have that do a set amount of processing (say 30 seconds) and then return. I know an ASP.NET page is not the ideal place to be doing this kind of processing but this seems to best meet our requirements. We considered spawning a new thread (not from the worker pool) to do the processing but decided that if we did that we couldn't use the page returned to signify a success or failure. By processing within the page's life-cycle we can use the page content to give an indication of how the processing went.
So the question is:
Are there any technical problems we might have with this set-up?
Obviously if you have tried something like this any reports of success/failure will be appreciated. As will suggestions of alternative set-ups.
Cheers,
Don't use the asp.net thread to do this. If the site is generating some information that you need in order to create or trigger the email-send then have the site write some information to a file or database.
Create a Windows service or scheduled process that collects the information it needs from that file or db and run the email sending process on a completely seperate process/thread.
What you want to avoid is crashing your site or crashing your emailer due to limitations within the process handler. Based on your use of the word "bulk" in the question title, the two need to be independent of each other.
I think you should be fine. We use the similar approach in our company for several years and don’t get a lot of problems. Sometimes it takes over an hour to finish the process. Recently we moved the second thread (as you said) to a separate server.
Having the emailer and the website coupled together can work, but it isn't really a good design and will be more maintenance for you in the long run. You can get around the problems you state by doing a few things.
Move the common business logic to a web service or common library. Both your website and your executable/WCF service can consume it, and it centralizes the logic. If you're copying and pasting code, you know there's something wrong ;)
If you need a template mailer, it is possible to invoke ASP.Net classes to create pages for you dynamically (see the BuildManager class, and blog posts like this one. If the mailer doesn't rely on Page events (which it doesn't seem to), there shouldn't be any problem for your executable to load a Page class from your website assembly, build it dynamically, and fill in the content.
This obviously represents a significant amount of work, but would lead to a more scalable solution for you.
Sounds like you should be creating a worker thread to do that job.
Maybe you should look at something like https://blog.stackoverflow.com/2008/07/easy-background-tasks-in-aspnet/
You can and should build your message body (templated message body) within domain logic (it means your asp.net application) when some business conditions are met and send it to external service which should only send your messages. All messages will have proper informations.
For "when certain dates occur" scenario you can use simple solution for background tasks (look at Craig answer) and do the same as above: parse template, build message and fast send to specified service.
Of course you should do this safe then app pool restarts does not breaks your tasks.
Background:
I am an intermediate web app developer working on the .Net Platform. Most of my work has been defined pretty well for me by my peers or superiors and I have no problem following instructions and getting the job done.
The task at hand:
I was recently asked by an old friend to redo his web app from scratch. His app is extremely antiquated and he is getting overwhelmed by it breaking all the time. The app in question is an inventory / CRM application and currently each customer requires a new install of the app (usually accomplished by deploying it on a different domain on the same server and pointing to a new database).
Currently if any client wants any modifications to the forms such as additional fields, new features, etc my friend goes in and manually adds those fields to the forms, scripts, database etc. As a result all installs of this application are unique. There is no one singular source repository and no one single version of this app. Generally new features are overtime rolled into the other sites, but still this is done on an individual site by site basis.
I will be approaching this on a very modular basis. Initially I will be coding a module that will query an external web service for some data, display and store it, and periodically update it automatically. The next module will likely be for storing and displaying inventory data. This way I want to over time duplicate the current feature set of his app 100% but do it incrementally.
The Million Dollar Questions
I want to make the app have user
configurable form fields. The user
should be able to go to an admin
page, create a new forms page of a
certain category, and then specify
what fields he wants in there. He
could say 'create a new text field
called Item # and make it a
requirement" and that will get
stored somewhere. All forms will be
dynamically rendered to screen based
on what the user has configured. Is
this a good way to go about the
problem of having no idea what a
customer could want in a form? and
thus be able to store and display
form data of any sort ? What sort of
design pattern should I follow here?
I am familiar with asp.net and
the .net framework in general and
have decent knowledge of javascript,
html, silverlight, jquery, c# etc
etc. I can work my way around web
apps in a good way, but I am not
sure what sort of framework or tech
I should use to accomplish this
task. Would ASP.net 3.5 webforms be
the way to go? or should I look into
ASP.NET MVC? Do I use jquery and ajax for
complete decoupling of frontend and
backend ? or will a normal asp.net
page with some spattering of ajax
thrown in working with a codebehind
be the order of the day?
Just looking for general advice before I start.
I am currently thinking of using ASP.NET 3.5 webforms, jquery for clientside animation, ui, manipulation and data validation, and sqlserver + a .net or wcf webservice for backend.
Your advice is much appreciated as always.
I've recently implemented a white-label ecommerce system for an insurance company that allowed each partner to choose their own set of input fields, screens, and order the flow of the application to suit their individual needs.
Although it wasn't rocket science, it added complexity and increased development time.
Consider the user configuration aspect very carefully In hindsight both my client and their clients in turn, would have been happy with a more rigid system.
As for the tech side of your question, I developed my project in VS2005, using asp.net webforms and webservices with a SQLserver back end, so the stack that you're looking at is definitely capable of delivering a working product. ASP.net MVC will almost certainly help as far as testability goes.
The biggest thing I would change now if I was going to start again would be to replace the intermediate webservices with message based services using nServiceBus, MassTransit or the like. While the webservices worked fine, message based communication should be quicker and more reliable.
Finally, before you start to code, make sure that you understand the current system's functionality inside and out. If the new system doesn't do something that the old system did, it will be pretty obvious to the end users straight away.