I have a .net web api application where I have used Serilog and was able to generate both structured logs with custom fields. All log files are stored in a folder in the server. Can I install Kibana/Grafana in the same server to create dashboards using info in the structured log files. Their(Kibana/Grafana) website refer to data sources like Elastisearch or some other but not directly structured logs.
Both Kibana and Grafana require some kind of database to connect to. AFAIK, so does Apache Superset.
If you don't want to provision and manage a database, you could just write something to read the files directly and render charts.
Related
I have several Symfony projects generating logs. I want to be able to view these logs in a single place. This is my plan…
Each system writes logs to separate folder in an AWS s3 bucket
A Laravel project will then access these logs probably by pulling
them down to local store. Using Arcanedev Logviewer I will then view
the currently available logs.
Has anyone been able to write Symfony logs directly to an AWS s3 bucket (I've only found reference to uploading them using a cron schedule)?
As an aside, has anyone come across a web based log service that allows you to simply view logs in a format as good as Arcanedev Logviewer. We currently use Cloudwatch which is good for what it does and certainly has its place but for me the killer feature of Arcanedev Logviewer is this screen…
Arcanedev Logviewer screenshot
Thank you
Gary
I have problem with node-red.
I have a server and i have node-red running on it, i know how to create user authentification by editing setting.js file but what i want to do is that i want to be able to create other users from a webpage that is running on the same server.
I can run a python script that edit the setting.js file and i can edit it mannualy,
but i want to be able to create and remove users from the webpage that is running on the same server.
i am using ubuntu and ngninx on my server.
The Node-RED security documentation includes a section on adding custom user authentication here. It explains how you can replace the hard coded user information with a module that will authenticate a given user. Editing the settings.js is probably not the right approach, especially as it will require you to restart Node-RED after each change.
With this approach you can build your own back end to store and manage users. There are a couple of examples of such code available, including this one that I have written that uses MongoDB to hold the user information. The management app that creates users can be found here, but you can write your own in Python if you want.
Just as a reminder, Node-RED is not a true multi-user environment, even if you declare a number of users, they will all only have shared access to one instance and set of flows. If you want a truly multi-tenant (each user has their own instance and flows) deployment you should probably look at the entire series of posts in that collection that explains how to build such a system. You can start here.
Is there anyway to open and read a SQLite database file on GAE?
I am currently uploading dbs to blobstore as admin and serving them publicly to user clients. I just can't read them in the GAE admin interface.
You can use SQLITE on Google App Engine. The problem has nothing to do with the support of certain libraries. It has to do with read-only file system. There is, however, a writable /tmp directory. If your app on startup first copies the db.sqlite3 file to /tmp/db.sqlite3 and references this path as database path, it will work.
There are, however, drawbacks.
1. This is not a "real" directory i.e. it's stored im memory. If database is too large, one will get problems.
2. Each instance has its own copy of db.sqlite3 file. Does not scale well.
Here is a django example:
Using SQLITE for local Django development for Google App Engine?
Short answer, no it is not possible to use a SQLite database on a standard Google App Engine application as it is not currently supported. However, you can give a try at implementing your own configuration with the App Engine Flexible Environment that allows to take advantage of custom libraries through Infrastructure Customization.
In case you would want to experiment, here is a sample Django application designed to be run with its default SQLite database on the App Engine Flexible Environment. Still, make sure to read the database notice providing alternative data storage options and explaining that SQLite data does not persist upon instance restart.
I want to deploy a php application from a git repository to AWS Opsworks service.
I've setup an App and configured chef cookbooks so it runs the database schema creation, dumping assets etc...
But my application has some user generated files in a sub folder under web root. git repository has a .gitignore file in that folder so an empty folder is there when i run deploy command.
My problem is : after generating some files (by using the site) in that folder, if I run 'deploy' command again 'Opsworks' adds a new release under 'site_name/releases/xxxx' folder and symlink to it from 'site_name/current' folder.
So it makes my previous 'user generated stuff' inaccessible. What is the best solution for this kind of situation?
Thanks in advance for your kind answers.
You have a few different options. Listed below in order of personal preference:
Use Simple Storage Service (S3) to store the files.
Add an Elastic Block Store (EBS) volume to your server and save files to the volume.
Save files to a database (This is something I would not do myself but the option is there.).
When using OpsWorks think of replicable/disposable servers.
What I mean by this is that if you can create one server (call it server A) and then switch to a different one in the same stack (call it server B), the result of using server A or server B should not impact how your application works.
While it may seem like a good idea to save your user generated files in a directory that is common between different versions of your app (every time you deploy a new release directory is generated) when you destroy your server, you run the risk of destroying your files.
Benefits and downsides of using S3?
Benefits:
S3 will give you high redundancy and availability to your files.
S3 is external to your application server so if your server dies or decide to move it to a different region, you can continue using the same s3 bucket.
Application Easy to scale. You could add multiple application servers that read and write files to S3.
Downsides:
You need extra code in you application. You will have to use the AWS API in order to store and retrieve the files. Using the S3 API is not hard but it may require an extra step to get you where you need. Take a look at the "Using an Amazon S3 Bucket" walk through for reference. This is be the code they use to upload the files to the S3 bucket in the example.
Benefits and downsides of using EBS?
Benefits:
EBS is an "external hard drive" that you can easily mount to your machine using the OpsWorks Resource Manager.
EBS volumes can be backed-up and restored.
It may be the fastest option to implement and integrate to your application.
Downsides:
You need to assign it to an instance before it is running.
It could be time consuming to move from server A to server B (downtime may be required).
You can not scale your application horizontally. While you can create copies of the EBS and assign them to different instances, the EBS will not be shared.
Downside of using a database?
Just do a google search on "storing files in database"
Take a look at Storing Images in DB - Yea or Nay?
My preferred choice would be to use S3, but ultimately this is your decision.
Good luck!
EDIT:
Take a look at this repository opsworks-chef-cookbooks it contains some recipes to deploy Symfony2 application on OpsWorks. I have been using it for over a year and works quite well.
Use Chef templates, and use them in a recipe in the opsworks deploy lifecycle event.
I have written a simple ASP.NET MVC 2 application that stores data and can dynamically create excel files using Microsoft's openXML for excel files.
What is the best way to push changes the user makes in excel to my database? I know it can be done via file upload, but this is rather obtrusive to the end user to navigate to my site, select upload, and then select their file.
Is there a way to do 1 click publishing from the excel file using VBA? VBA can interact with the database directly, but this seems dangerous from a data security standpoint, and duplication of logic.
Do web services work with the MVC architecture? How do I get a vba macro enabled document to send itself to the server?
For anyone out there looking for a fix, I ended up using vba's InternetExplorer.Application object and interacting with an upload form on my site.
For more info on the upload form check out:
http://haacked.com/archive/2010/07/16/uploading-files-with-aspnetmvc.aspx
For more info on VBA and the InternetExplorer.Application object check out:
www.motobit.com/tips/detpg_uploadvbaie/
You might take a look at Sql server integration services for bulk upload of data into sql server. The integration services once created can be run using a normal c# desktop program or using a windows service.
But you might
need to make sure this happens in the background and will have to be
an asynchronous task.
also need to make sure it is properly secured
by not giving direct execute access to any other users
I'm assuming that this is for a specific user. I've done something very similar to what you are describing before.
Tell the user to save the excel file in their DropBox and share the file with you.
Have the server listen for changes to this file and run a server side routine to import the data.
Disclaimer: This is not a secure solution, but it's easy and will get the job done.