I have 2 applications and each one of them uses its own application insights. Application Consumer (C) requests data from application Repository (R) via REST/HTTP. I have access on both app insights.
If I investigate C's app map, I can see that it has a dependency on R, but R is displayed the same way any other external (as if I was querying google as far as it knows)
If I investigate R's app map, I can't see any mention of C... as if it didn't exist...
How can I make one application aware of the existence of the other? Isn't this supposed to be like out-of-the-box?
Related
I'm wondering are there any best practices or recommendations out there on using Application Insights to monitor web jobs. At the moment I have all my app service and web jobs logs going to the one AI instance and there is alot of noise in there.
Specifically should I:
create a separate AI instance for all the web jobs
or create a separate AI instance per web job.
Thanks
You should generally use one instance for each system in your environment, but separate out dev, test, and prod. This is to make it simpler to track dependencies as jobs move throughout the system. So with multiple web apps, you might group an API, a separate web app that serves the front end content, and any web jobs that support those two apps. On another you may have just a single web app or web job that acts independently from the rest of your apps.
However, you should choose the number of Application Insights instances that best fit your situation. If it would work better for you to split each web job then you can certainly do that. You can query across App Insights instances, so you don't completely lose the ability to join the data from different services together if you choose to split them into separate App Insights instances.
I am building a series of interactive shiny web apps for a project that I am considering turning into a Company. My background is in data science and I don't have a lot of experience on the web app / server side of things, but these are important aspects for me to consider with my project. I currently have an Amazon Linux AMI EC2 instance with ShinyServer (free, open-source) installed, and I am currently hosting early versions of my web apps there. So far everything works fine, but I haven't made the links public yet.
My first question is whether anyone knows if there are certain limitations (scalability limitations, integration with database limitations, security / authentication limitations, etc.) that I will inevitably run into using RShiny apps and ShinyServer? I haven't heard of many successful, super-popular web apps being shiny apps hosted on ShinyServer, but rather my feeling is that ShinyServer is mainly used for hosting RShiny apps that are shared amongst only a small number of people (i.e. shared amongst team members at a company.). Per this thread - Does R-Server or Shiny Server create a new R process/instance for each user? - I am particularly concerned that my app won't be able to handle thousands of users simultaneously since only 1 R process is created for the app regardless of the # of concurrent users of the app. Having 10-20 processes through ShinyServer pro probably doesn't solve the issue either if I ever intend to scale greater than the hundreds or thousands of users. I also noticed that ShinyServer Pro would run me a not-so-negligible $10K per year.
My second question is whether RShiny apps can be deployed using other server technologies, such as Heroku. I came across this github page (https://github.com/virtualstaticvoid/heroku-buildpack-r/tree/heroku-16) but haven't dug too deep into it yet. I've been told that heroku makes it easy to update releases to apps whose code is on github (git push heroku:master), amongst other things.
My third question involves certain specific considerations of mine. In particular, I am currently working on a script that queries data from an API and writes that data to a (not-yet-setup) database of mine. This is the data my apps use, and I'd be interested in having the apps update in real time as the database updates, without requiring the user to refresh the webpage. A buddy of mine suggested AJAX for this type of asynchronous behavior, and it looks like this may be possible in R with something like this (https://github.com/daattali/advanced-shiny/tree/master/api-ajax).
Sorry that this is such a loaded question, but I hope it doesn't get closed down as I think it is fairly educational. Any suggestions / sources / pointing me in the right direction would be greatly appreciated on this.
Canovice,
I'd recommend you take a look at the following RStudio / AWS support articles. To scale a shiny server you'll need to look at using a load balancer:
RStudio
https://shiny.rstudio.com/articles/scaling-and-tuning.html
https://support.rstudio.com/hc/en-us/articles/220546267-Scaling-and-Performance-Tuning-Applications-in-Shiny-Server-Pro
https://support.rstudio.com/hc/en-us/articles/217801438-Can-I-load-balance-across-multiple-nodes-running-Shiny-Server-Pro-
AWS
https://aws.amazon.com/blogs/big-data/running-r-on-aws/
Blog Article:
http://mgritts.github.io/2016/07/08/shiny-aws/
Shiny is a great platform, their support is fabulous. I'd recommend you ring them up - they'll be sure to help answer your questions.
That said if your plan is to create a scalable website that will support thousands or hundreds of thousands of people then my sense would be to recommend you also review and consider using D3.js in conjunction with react.js or Angular.js, not forgetting to mention node.js.
My sense is that you are looking at a backend database connected to a logic engine and visualisation front end. If you are looking for a good overview of usage take a look at the following web page and git repo [A little dated but useful]:
https://anmolkoul.wordpress.com/2015/06/05/interactive-data-visualization-using-d3-js-dc-js-nodejs-and-mongodb/
https://github.com/anmolkoul/node-dc-mongo
I hope the above points you in the right direction.
I'd like to provide some notes related to your second question: Yes, you can use the mentioned buildback to deploy shiny applications on heroku.
I was in a similar situation with you (asking myself about possible ways of serving Shiny applications in a scalable manner) and decided to go the "heroku way".
You may find these hints helpful when deploying your app to heroku using the buildpack mentioned above:
Heroku tries to "guess" how to execute your application. But you can also add a special file, named Procfile, to your application to control the process commands you want to execute for your application. In my case I used web: R -f ~/run.R --gui-none --no-save, where this means that a file named run.R is being passed to the R executable for the web server process
The stack on heroku is based on Ubuntu. If you need additional deb-packages, you can create another special file named Aptfile and add the package names therein, heroku will then automatically install these for you (I needed it for RPostgreSQL)
You can add another special file named init.R and install all R packages as necessary just as you are used to, i.e. with install.packages etc. You can also add initial configuration material within this file.
As a running example, here is an example toy application that I wrote for myself to remember how a "full-stack" shiny app may look like, including compability with heroku.
For a large number of concurrent users, use a load balancer like nginx and enable the autoscaling of your app, e.g. through Kubernetes.
You can deploy your app on Heroku. On the paid tiers it includes NoOps autoscaling of your app. See this tutorial on how to deploy a Shiny app in a Docker container on Heroku: https://medium.com/analytics-vidhya/deploying-an-r-shiny-app-on-heroku-free-tier-b31003858b68
You can query the table last update timestamp in the Shiny server logic with reactivePoll() and rerun your db query if it changed. It is not "real-time" but depending on your application close enough if you set the time interval small.
I need to get hardware information from azure web roles / web worker to monitor it for critical conditionals like high memory/cpu usage.
I tried to use some addons which are provided in the azure gallery like the one from "logentries", but the gallery doesn't support my country yet...
Is there an other way to get the log information directly?
Last option would be Azure Diagnostics, but it stores everything in blob storages and I would have to pull everything out there on my own and send it to "manually" to logentries, geckoboard or whatever.
Three good options:
Windows Azure Diagnostics. Yes, it puts everything in table/blob storage which is painful, but there are tools such as Cerebrata's Azure Management Studio that can help gather and visualize the data.
Application Insights. This is still in preview, but it provides a very rich application monitoring and alerting platform.
The built in Azure monitoring. This is not quite as feature rich as Application Insights, but it is very easy to setup and use and includes monitoring and alerting.
I'm surprised that no one mentioned New Relic.
It has a comparable feature set to Application Insights but should be way more stable since it's not in preview like Insights. (although I am following the development of Insights closely, give it a while and it will be an awesome alternative)
I have downloaded TheWorldsWorstStackOverflowClone. One of the project is called TheWorldWorsts.ApiWrapper, which basically is the core of accessing the API. There is a class called ApiProxy.cs, which has all the methods for the API call. This is good.
Now what I want to do is I am trying to collect data from this API interface and store it in a database. I know the limit to the API call is 10k per day. I.e: I want to be able to call the method in the ApiProxy class 10k times per day, done automatically. How can I do this?
The non-automatic way would be to create a dummy site where when every time I access the site it does all that process, but this not efficient. It seems that I have to write some kind of a scheduler by deploying a web service, but that is too complicated... as explained here. Any other simpler methods?
A Windows Service or Desktop App might be a better solution than a web application. You are not deploying a web service, you are consuming one using a proxy class, and this does not require you to have a web server or a web site.
You could use a web application to control and monitor progress as your service downloads data, but the actual work is long running and needs to be offloaded to another process or thread so you can tell the user whats going on.
Check out this one
http://stacky.codeplex.com/
This looks what you need, though I am facing some debugging issues, but hope you can figure it out.
We have a PeopleSoft installation and I am building a separate web application that needs to pull data from the PeopleSoft database. The web application will be on a different server than PeopleSoft, but the same internal network.
What are my options?
This one's an oldie but it may still be of interest.
PeopleSoft has it's own schema within the host database (Oracle, SQL Server, DB2 etc) which are the PSxxx tables, eg: PSRECDEFN is the equivalent of Oracle's DBA_TABLES. These tables should not be touched by any external code. The application tables are stored in PS_xxx tables, eg: PS_JOB. These tables can be read and updated by any SQL code.
Many batch programs in PeopleSoft (eg: Application Engines, COBOL or SQRs) access the tables directly, and this is the fastest way to get data into or out of the database. However PeopleSoft has quite a rich application layer which is bypassed when doing direct SQL. This application layer must be replicated in direct SQL code, especially for inserts or updates. There may be updates to other tables, calculations or increments of database-stored counters.
To determine how to do this one must look through the PeopleCode (a VB6-like interpreted language), page design (via Application Designer) and use the PeopleCode and SQL trace tools. These days the application layer is huge, so this can be a lengthy task for non-trivial pages. PeopleSoft groups related pages into "Components", and all pages in the component are saved at the same time.
Component Interfaces were introduced with PeopleTools 8 as a means to avoid doing all of this. Using a generator within the PeopleSoft app designer, a Component Interface is generated based on the component. For many components these can be used to access the pages as a user would, and can be accessed via PeopleCode programs, and therefore via App Engine programs and via the Integration Broker. They can also be wrapped in Java code and access directly by code able to execute against the app server with a web service wrapper. This method is best for low-volume transactions: heavy extracts work better with native SQL.
The online development and tracing tools in PeopleSoft are pretty good, and the documentation is excellent (although quite extensive) and available on: http://download.oracle.com/docs/cd/E17566_01/epm91pbr0/eng/psbooks/psft_homepage.htm
If you are just looking at bringing out data from a given Component, the easiest way would be to turn on the SQL trace (under the utilities menu in PeopleSoft) and bring up some records for the Component. Wading through the trace file will give you a good idea of what to do, and much of the SQL could be cut and pasted. Another method would be to find an existing report that is similiar to what you are trying to do and cut out the SQL.
Have a PeopleSoft business analyst on hand to help you develop the requirements wouldn't hurt either.
Yes - Integration Broker is Peoplesoft's proprietary implementation of a publish/subscribe mechanism, speaking xml. You could of course just write code that goes against your database using JDBC or OLE/ODBC. Nothing keeps you from doing this. However, you must understand the Peoplesoft database schema, so that you are pulling from, or inserting/updating/deleting all of the proper data. Peoplesoft takes care of this for you.
Also, check out Component Interfaces - and they are exposed as an API to Java or C/C++.
I guess it depends on your requirement, and which version of PeopleSoft you're on.
Do you want real-time lookup? If that's the case then you'll want to look at Web Services/Integration Broker.
If you want a batch/bulk export then a scheduled App Engine would do the trick.
The best way is to use Integration Broker (IB) services to expose the PeopleSoft database data to external applications. The external application will be able to access the PeopleSoft IB services as XML over HTTP, thus allowing you to use any widely used XML parsers for this purpose.
The problem with component interfaces as opposed to Integration Broker is that component interfaces tend to be much slower than direct DB access from within IB service PeopleCode. Also future additions to the component attached to the component interface sometimes tend to 'break' the interface.
For more details on PeopleSoft Integration broker, you can access the online documentation at http://docs.oracle.com/cd/E26239_01/pt851h3/eng/psbooks/tibr/book.htm
Going directly to the database means you have to re-create the presentation logic... see my longer answer above. You can do this for simple pages but otherwise using a component interface is the way to go.
You can also write a sqr process for bulk data extraction. SQR will create the output file which the other application can pick. SQR would be faster than the application engine programs as it performs most of the operations in memory.