i am struggling understanding the functional role of Kubeflow (KF) compared with other (generic) workflow orchestrator.
I know KF is oriented to ML tasks, and is built on top of Argo.
Two questions:
can KF be used at a higher level as a workflow orchestrator to perform more generic tasks (i.e. ETL) whose outcome might be useful in the following ML tasks?
can use all funcionalities of Argo within KF.
what can a generic workflow orchestrator (as Airflow, argo, etc.) do that KF cannot?
Yes, you can create a python function/ general containers with code baked in which executes whatever task you like.
pre-defined component -
https://www.kubeflow.org/docs/components/pipelines/sdk-v2/component-development/
python component - https://www.kubeflow.org/docs/components/pipelines/sdk-v2/python-function-components/
KFP is an abstraction op top of Argo workflows.
it gives you the ability to create Workflows using python instead of writing YAML files. Check out this article : https://towardsdatascience.com/build-your-data-pipeline-on-kubernetes-using-kubeflow-pipelines-sdk-and-argo-eef69a80237c
since Argo Workflows development is advancing independently from KFP it's safe to assume there will be missing features in KFP (Which are the community will add according to demands).
that's a big question.
in general, airflow has sensors, SLA feature/ huge store of operators/sensors/reports/plugins and a bigger community since it's not ML oriented.
I see Rule Flow which supports action so it may be possible to build some types of workflow on top of this. In my situation I have an case management application with tasks for different roles, all working on a "document" that flows through different states and depending on state, different role will see in their queue to work on.
I'm not sure what your question is, but InRule comes with direct support for Windows Workflow Foundation, so executing any InRule RuleApplication, including those with RuleFlow definitions, is certainly possible.
If you'd like assistance setting up this integration, I would suggest utilizing the support knowledge base and forums at http://support.inrule.com
Full disclosure: I am an InRule Technology employee.
For case management scenarios, you can use decisions specifically to model a process. Create a custom table or flags in your cases that depict the transition points in your process (steps). As you transition steps, call a decision which will determine if the data state is good enough to make the transition. If it is, then set the flag for the new state. Some folks allow for multiple states at the same time. InRule is a stateless platform; however, when used with CRM it provides 95% of the process logic and relies on CRM to do the persistence. I have written about this pattern in a white paper:
https://info.inrule.com/rs/inruletechnology/images/White_Paper_InRule_Salesforce_Integration.pdf
As a novice to this realm, I am planning on building an mvc application. I had originally started a web forms application but decided the scalability and testability will benefit more with an mvc application. I chose to switch with the added benefit of being easier to add more features later on (instead of having code baked into web forms pages).
Now a little about my application, it is an application to stimulate an RPG class builder and moveset. In all simplicity, users can register for a class, and depending on other skills they can register for, they can see a custom move set based on these categories. The way I am envisioning it is I will be able to go back and add more classes and skills later in the database and have users register for this new content immediately once it has been added to the project.
Everything lives in normalized tables, so many joint tables do exist. For each new skill or class I add will mean a handful of tables will be added to the database. This speaks to the way the data will be stored, everything and all information about classes, user data, skills, etc will be stored in the database.
I have designed all the initial database tables I will need to have at the start, and functionality I need (a home page, view skills page, view move sets page, etc.). I am stuck at the next step; where do I go? Should I make my controllers first? Models? Views? Design my page layouts? I am asking for advice from people who have taken a similar organic approach to an mvc project. I am facing analysis paralysis on what to start on, knowing I have a lot of work ahead of me.
Thank you for taking your time to answer.
I've taken everyone's advise and am putting together a website to learn MVC: http://learnaspnetmvc.azurewebsites.net
The most important advice I can give you: just start. A big project can seem overwhelming, especially when you're looking at it like a big project. Instead, break it into small achievable tasks. Find something you can do right now, the ever-so-smallest subset of functionality, and do it. Then do the next one. And the next.
That said, I'll tell you my personal process. When I start on a new application or piece of an application, I first like to create my models. That way I can play with the interactions between them, flesh out the relationships, and think about the needs of my application in a somewhat low-pressure, easily disposable way. I also use code-first, whereas you've gone an created your database tables already. Some people prefer to do it that way. Personally, I find starting with my classes and letting those translate into an underlying data store much more organic. In a sense, it relegates the database to almost a non-existent layer. I don't have to think about what datatype things need to be, what should be indexed and what shouldn't, how querying will work, what kind of stored procedures I need, etc. Those questions have their time and place -- the nascent development stage is not that time and place. You want to give your brain a place to play with ideas, and classes are a cheap and low-friction medium. If an idea doesn't work out, throw the class away and create a new one.
Once I have my models, I like to hit my controllers next. This lets me start to see my models in action. I can play around with the actual flow of my application and see how my classes actually work. I can then make changes to my models where necessary, add additional functionality, etc. I can also start playing around with view models, and figuring out what data should or should not be passed to the view, how it will need to be displayed (will I need a drop down list for that? etc.), and such. This, then, naturally leads me into my views. Again, I'm testing my thinking. With each new layer, I'm hardening the previous by getting a better and better look at how it's working.
Each stage of this process is very liquid. Once I start working on my controllers, I will make changes to my models. Once I hit the views, the controllers will need to be adjusted and perhaps the models as well. You have to give yourself the freedom to screw up. Inevitably, you'll forget something, or design something in a bone-headed way, that you'll only see once you get deeper in. Again, that's the beauty of code-first. Up to this point, I don't even have a database, so any change I make is no big deal. I could completely destroy everything I have and go in a totally different way and I don't have to worry about altering tables, migrating data, etc.
Now, by this point my models are pretty static, and that's when I do my database creation and initial migration. Although, even now, really, only because it's required before I can actually fire this up in a browser to see my views in action. You can always do a migration later, but once you're working with something concrete, the friction starts to increase.
I'll tend to do some tweaking to my controllers and obviously my views, now that I'm seeing them live. Once I'm happy with everything, then I start looking at optimization and refactoring -- How can I make the code more effective? More readable? More efficient? I'll use a tool like Glimpse to look at my queries, render time, etc., and then make decisions about things like stored procedures and such.
Then, it's just a lot of rinse and repeat. Notice that it's all very piecemeal. I'm not building an application; I'm building a class, and then another class, and then some HTML, etc. You focus on just that next piece, that small chunk you need to move on to the next thing, and it's much less overwhelming. So, just as I began, I'll close the same: just start. Writers have a saying that the hardest thing is the first sentence. It's not because the first sentence is really that difficult; it's because once you get that, then you write the second sentence, and the third, and before you know it, you've got pages of writing. The hardest part is in the starting. Everything flows from there.
The other answers here have great advice and important nuggets of information, but I think they do you a disservice at this stage. I'm the first to advocate best practice, proper layering of your applications, etc. But, ultimately, a complete app that follows none of this is more valuable than an incomplete app that incorporates it all. Thankfully, we're working with a malleable medium -- digital text -- and not stone. You can always change things, improve things later. You can go back and separate your app out into the proper layers, create the repositories and services and other abstractions, add in the inversion of control and dependency injection, etc. Those of us who have been doing this awhile do that stuff from the start, but that's because we've been doing this awhile. We know how to do that stuff -- a lot of times we already have classes and libraries we drop in for that stuff. For someone just starting off, or for an app in its earliest nascent stage, it can be crippling, though. Instead of just developing your app, you end up spending days or weeks pouring through recommendations, practices, libraries, etc. trying to get a handle on it all, and by the end you have nothing really to show for it. Don't worry about doing things right and do something. Then, refactor until it's right.
As a first step in planning a MVC framework application, We should start with a strong Model (typical C# props). This process is going to take most of our time, based on the fact that we need to understand the business first and then the relationships between different workflows and entities. So times business models evolve as time passes. So spend qualitative time on building this layer, but not too much.
Once domain (Business) Models are ready, before we actually start coding for Repository classes, we should define our Repository Contracts which are typically Interfaces. Contracts help all parties(other components) to interact with each other in the exact same way. Then we implement contracts on the Repository component, which is just going to act like PUSH and PULL data from your persistent medium (say database). Remember repository component never going to have any idea on business logic.
Once backend has been established, We can concentrate on my actual business process implementation. We can define one more level of Contract which defines all business operations which are to be done using Model classes. This interface has been implemented by BusinessLogic Component which does the core business activity (specific methods for every business operation). This particular component will use Repository component to delegate business data to persistence medium.
With above step completed, We can easily go and build Controllers. We should be calling business logic component methods in controllers and get work done. Once controllers are done, we can define our views and other UI elements like partial views etc.
Pictorial representation of the flow is as follows -
A simple architecture (from high to low level)
Presentation Layer
Domain Logic Layer
Data Access Layer
Database
Presentation layer is MVC project containing Views, Controllers and optional View-Models.
Domain Logic Layer is Class Library project which Presentation layer will access (via DLL or Service reference). This layer contains business logic and rules for the application.
Data Access Layer may contain two sub-layers-
Repository. User repository is best practice for any long term application.
Entity Framework Model.
This will communicate with database.
Database you already have.
I would suggest reading through a book by Scott Millet, called Professional ASP.NET Design Patterns.
ISBN : 0-470292-78-4
Scott walks through what a good ASP.NET site should look like from an architectural standpoint - i.e. DataAccess layers, Business Logic layers, Presentation Layers, Domain events etc.
By following on from industry standards, you will gain a better knowledge of how to correctly put together an MVC web-site.
Hope this helps.
I would suggest you to make your MVC application around a ASP.NET Web API , since it will help, in case you go for a mobile application later.
Since you are a MVC newbie, you should download some open source projects on MVC shared by seniors in the community. Study two or three projects, and analyze a solution which will the best for you.
A quick googling will get you to some good projects.
e.g.
Making a simple application , Prodinner
After that you should also go through MSDN tutorial on MVC 5 app with SSO , to enable social logins.
I m working on a legacy web project, so there is no ORM(EF,Nhibernate) available here.
The problem here is I feel the structure is tedious while implementing a new function.
let's say I have biz Object Team.
Now if I want to get GetTeamDetailsByOrganisation
,following current coding style in the project,I need:
In Team's DAL, creat a method GetTeamDetailsByOrganisation
Create a method GetTeamDetailsByOrganisation in the Biz Object Team, and call the DAL method which I just created
In Team's BAL, wrap up the Biz object Team's method in another method,maybe same name, GetTeamDetailsByOrganisation
Page controller class call the BAL method.
It just feels not right. Any good practice or pattern can solve my problem here.
I know the tedium you speak of from similarily (probably worse) structured projects. Obviously there are multiple sensible answers to this problem, but it all depends upon your constraints and goals.
If the project is primarily in maintenance mode with very no new features being added I might accept that is the way things are. Although it sounds like you are adding at least some new features.
Is it possible to use a code generator? A project I worked on had a lot of tedium like this, which apparently was caused because they originally used a code generator for the code base which was lost to the sands of time. I ended up recreating the template which saved me a lot of time, sanity, and defects.
If the project is still under active development maybe it makes sense to perform some sort of large architectural change. My current project is currently in this category. We're decoupling code and adding repositories as we go. It's a slow process that takes diligence and discipline by the whole dev team. Each time a team takes on a story they tax that story with rewriting some of the legacy code in that area. To help facilitate this we gave a presentation to the rest of the team to get buy-in and understanding. We also created some documentation for our dev team that lists out the steps to take and the things to watch out for. In the past 6 months we've made a ton of progress. We don't have the duplication you speak of, but we have tight coupling issues which makes unit testing impossible without this refactor.
This is less likely to fit your scenario, but it may also be a possibility to take certain subset of features and separate those out into separate services that can be rewritten using a better platform and patterns. The old codebase can interoperate at the service layer if needed. You likely make changes in certain areas more than others, so the areas of heavy change might be top priority to move to a dedicated service. This has the benefit of allowing you to create a modern code base without having to rewrite the entire application from scratch all at once. This strategy is what Netflix has done to rewrite their their platform as they go and move it to the cloud.
I needed to do some higher-level project planning that doesn't really fit into the workflow of our day-to-day task management tools (FogBugz and whiteboards), so I figured I'd give MS Project a whirl (it being free through MSDN).
I've hit a pretty solid wall, though. What I have is about 120 tasks, a set of people (referring to them as "resources" is amazingly harsh to me, but I digress), and a rough prioritization of those tasks. Some tasks have a person assigned to them, some don't (simply because we don't know who's going to do what yet).
Fine so far. The problem is that, except in those relatively rare instances where tasks are linked (most of the work involved can be done in any order), all of the tasks are scheduled to run concurrently. What I'd like to do is have Project figure out some scheduling scenario based upon:
the defined tasks
their relative priority
any links/dependencies, if defined
the availability of the people that I've defined, while respecting the explicit "resource" assignments I've already made
Is this possible? I've fiddled with the resource leveling dialog and read more MS Project documentation than I'd care to admit, so any suggestions are welcome.
FWIW, I noticed in my searches this question on Yahoo Answers; the person there seems to be after roughly the same thing, but I figured asking here might be more fruitful.
After some further experimentation, I've found a partial solution for my own question. If you:
assign a person to each task
specify in the Advanced tab of the Task Information panel that all tasks should (select all your tasks and click the Task Info button to update these properties for all tasks):
use a calendar (called "standard" in my project file)
not ignore resource calendars when scheduling
have a constraint of As Soon As Possible (which is the default, I believe)
Choose Level Resources from the Tools menu, and specify:
Look for overallocations on a Hour by Hour basis
a leveling order of "Priority, Standard" (which rolls in the relative Priority values for each task you've defined when setting the schedule)
Click "Level Now" in that leveling resources dialog, and all of the tasks should be rescheduled so that they're not running concurrently, and that no one is "overscheduled".
You can ostensibly have Project automatically reschedule things as tasks are added, edited, etc., but I suspect that would result in chaos, as there's nothing about the resource leveling process that makes me think it's "stable" (e.g. that two levelings performed back-to-back wouldn't yield the same schedule).
It would be nice if Project would "fully allocate" whatever people you have configured, so that you don't have to assign people to tasks just to have those tasks scheduled in a way that is consistent, if not correct. Any thoughts on that front would be most welcome.
That seems (and feels!) like a lot of work, but I think the result is relatively decent -- a super-high-level view of a project that allows for a high degree of day to day flexibility, but still affords one a way to reasonably make plans around "interdisciplinary" activities (e.g. once this is done, we need to buy those four servers, make sure our legal stuff is taken care of, and pull the trigger on that marketing push one week later, etc).