I have several Symfony projects generating logs. I want to be able to view these logs in a single place. This is my plan…
Each system writes logs to separate folder in an AWS s3 bucket
A Laravel project will then access these logs probably by pulling
them down to local store. Using Arcanedev Logviewer I will then view
the currently available logs.
Has anyone been able to write Symfony logs directly to an AWS s3 bucket (I've only found reference to uploading them using a cron schedule)?
As an aside, has anyone come across a web based log service that allows you to simply view logs in a format as good as Arcanedev Logviewer. We currently use Cloudwatch which is good for what it does and certainly has its place but for me the killer feature of Arcanedev Logviewer is this screen…
Arcanedev Logviewer screenshot
Thank you
Gary
Related
I am using Firebase for a long time (since 2018) and loving it. In that time there was not Location southamerica-east1 (São Paulo). Now I would like to store the project (web app, cloud function, and database) in southamerica to reduce cost and make it near to my end-users (also based in Brazil).
I have source control, all environment parameters values stored in Custom Environment Variables. The application works fine when no data is found. No concerns with backup data. No problem about downtime. This is not a critical app.
Anyway, I can't delete the application because I already have some users logged in there and IoT devices sending data through PubSub.
How can I rebuild my Firebase/Firestore/Web application/Function from the ground up, and make sure the new location is southamerica? If possible, I need to keep user and passwords, and web
Looking forward, (I don't think moving the bucked location would be the best solution here) but based on this page Select locations for your project I can't update the location, but since it is based on bucked location, if it doesn't break the project, I will use Google Cloud Transfer Page to Moving and renaming buckets
May is it a better solution than rebuild the app (Firebase/Firestore/Web application/Function)?
May I break my Firestore database or cloud function or web app?
May I lost my project domain or any other related URL parameter like authDomain, databaseURL, storageBucket?
May I need to update some web app parameter after the change?
They cannot be moved at present and migrating data is a manual process. Difficulty varies by product.
General guidance
Do not delete the old project before fully migrated.
Hosting
This migration is nearly trivial, with the understanding that there is likely a minor service interruption while moving custom domains.
Deploy to the new site
CNAME your custom domain to the new site (myproject.firebaseapp.com)
Delete custom domain from old site
Add custom domain to new site
Cloud Functions
This migration is trivial.
Create a local directory for your new project
Run firebase init and set up project normally (enable Functions)
Copy your Functions code into the new project's functions/ directory
Deploy to the new project
Database
This migration is tricky, difficult, and highly specific to your use case and tolerance for downtime. What follows is a general template to adapt.
Reference docs for import/export: Firestore import/export, Realtime Database backups
In the old project:
Lock the database using security rules to prevent changes
Export existing database
In the new project:
Import the database backup
You probably need to migrate existing users (see account export/import) as well so user ids stored in your DB will still reference the correct accounts
Point existing apps to the new project
If downtime is not an option, or if you'll be deploying a new mobile app version and need time for changes to propagate, then you'll need to set up a dual write model:
Dual sync: Create a Cloud Function on both the new and old database that duplicate all create/update/delete operations on the respective partner endpoint.
Sync pre-existing data: Perform the export/import process as above on all data created before the dual sync was implemented, excluding the step to lock the old database
Shut down your old mobile app version (once enough accounts have migrated)
Shut down the dual sync Functions and turn down the old site
Based in your information, the main issue is Firestore because other products are globally balanced like Cloud IoT Core and Hosting (these can't be configured on a specific region)
Other products like Functions can be redeployed with the same code and name into another region.
I think that you can create another project only to move the database to the new region and configure all Cloud resources to reach the new located database.
As a caveat, you need to add another domain/subdomain and create new credentials to work with the new project; this step can´t be skipped because it is required for authentication.
On the application side you can add the access to the new database
In case you need assistance during your migration you can start a case with GCP/Firestore support.
This is a hard pill to swallow, but maybe the costs and the time to migrate to another region will be higher than keeping your application as is working today.
I have problem with node-red.
I have a server and i have node-red running on it, i know how to create user authentification by editing setting.js file but what i want to do is that i want to be able to create other users from a webpage that is running on the same server.
I can run a python script that edit the setting.js file and i can edit it mannualy,
but i want to be able to create and remove users from the webpage that is running on the same server.
i am using ubuntu and ngninx on my server.
The Node-RED security documentation includes a section on adding custom user authentication here. It explains how you can replace the hard coded user information with a module that will authenticate a given user. Editing the settings.js is probably not the right approach, especially as it will require you to restart Node-RED after each change.
With this approach you can build your own back end to store and manage users. There are a couple of examples of such code available, including this one that I have written that uses MongoDB to hold the user information. The management app that creates users can be found here, but you can write your own in Python if you want.
Just as a reminder, Node-RED is not a true multi-user environment, even if you declare a number of users, they will all only have shared access to one instance and set of flows. If you want a truly multi-tenant (each user has their own instance and flows) deployment you should probably look at the entire series of posts in that collection that explains how to build such a system. You can start here.
The Issue
I am currently in the process of integrating a pre-rendering service for SEO optimization, however we use an Azure App Service Plan to scale up or down when necessary.
One of the steps for setting up the proper configuration requires placing an applicationHost.xdt file in the /site/ directory, which is one level above the /site/wwwroot directory where the application itself gets deployed to.
What steps should I take in order to have the applicationHost.xdt file persist to new instances spawned by the scaling process?
Steps I have taken to solve the issue
So far I have been Googling a lot, but haven't succeeded in finding a lot of documentation on using an applicationHost.xdt file in combination with an Azure App Service Plan.
I am able to upload the file to an instance manually, however I have assumed that when we then scale up to more instances the manually uploaded file will not be present on the new instance(s).
Etcetera
We are using Prerender.io as pre-rendering service.
Should there be an easier to set-up & similarly priced service available, we would be open to suggestions as we are in an exploratory phase regarding pre-rendering.
Suppose this won't be a problem, cause all files under azure app are shared between all your instances. You could check it in this Kudu wiki:Persisted files. And in my test all instances will keep the file.
About upload the applicationHost.xdt, you don't have to do it manually, there is a IIS Manager Site Extension to lets you very easily create XDT files. And it will provide some sample XDT's for you.
My preview works and has data but my deployment has no data. I'm using the (Recommended) DEFAULT CLOUD SQL database configuration.
Note: This is only day 4 with Google App Maker. Finding answers to App Maker-specific questions has been super difficult, but I'm making rapid progress on my application, so overall tired but good. :{)
As written in the documentation,
App Maker deployments can use the same Cloud SQL instance, but have separate databases on that instance. Data that you had in preview mode is not available in other deployments. You have a few options for how to handle this situation:
To use data from the preview instance in your published deployment, export the deployment data from the preview instance and import it to the published deployment.
To share a database across all deployments (preview and published), use a custom Cloud SQL database.
When you deploy your app, AppMaker create a new database in your google cloud SQL instance for the deployment. All the data create in previews is in another database.
To use the same database as the preview mode you have to go in the settings of your app in the tab "DATABASE" and copy the Database Key. Then go to your cloud sql instance in google cloud platforme and on the details of the instance in the overview tab just copy the instance connexion name.
then edit your deployments and select "Use Custom Cloud SQL database" and copy with the format
"instanceConnexionName/DatabaseKey" then save and appmaker should ask you to enter you username and password of your google cloud sql insatnce.
On app settings, database page you should see
Databas key: iTIJQaCj491a4111
(Actually this is the name of the mySQL instance)
In GCP console, go to SQL, click on Instance ID, and on the Instance ID overview page is the instance connection name, e.g., MyProject-123456:us-central1:instancename
Back in app settings
Select Switch to custom database and enter the full connection string
projectname:instancename/schema as
MyProject-123456:us-central1:instancename/iTIJQaCj491a4111
Provide username and password
and follow the steps to confirm existing database
Turns out the issue is when you publish it doesn't push the data, you have to manually re-upload the data into the live version. This is actually a good thing, but I wish it'd been explicitly documented. I found it, after figuring it out on my own, in some early release notes from a few years back. I guess I wasn't the only one this stupid.
I have recently deployed my web role to Windows Azure. In the properties of my WebRole I have set Enable Diagnostics.
I can also see that it correctly maps to a storage account once deployed by viewing the configuration file of the hosted service.
I have not setup anything else for diagnostics, I am unaware that I need to do anything else.
I am now setting up AzureWatch (by paraleap) to monitor my instances however it reports that WADPerformanceCountersTable does not exist.
I am very new to Azure, don't have a clue how the diganostics work and can't find anything on Google that shows me how. Could someone please show me the way.
Ok I figured it out and will leave this here for others to follow.
Step 1
If you follow http://dunnry.com/blog/2012/02/27/SettingUpDiagnosticsMonitoringInWindowsAzure.aspx Windows Azure Diagnostics will start saving data into your attached Blob storage, full of diagnostic information.
Special Note: These count towards your storage transaction, which is why you will see them go up.
Step 2
However I needed the WADPerformanceCounterTable, which should have been located in the tables section of the storage account but it never was created. I needed this to use services like AzureWatch to monitor and spin up or down instances.
Special Note: This is performance counters, a specific subset of diagnostic information and this isn't stored in the blob section by default.
Step 3
In your project you need to add which performance counters to monitor in the WebRole.cs.
Special Note: You won't have this if you just added an existing project to an Azure deployment project. Unless you specifically started the project from scratch and chose the Azure templates, you will need to create this manually. You would also need to add: Microsoft.WindowsAzure.Diagnostics, Microsoft.WindowsAzure.ServiceRuntime and Microsoft.WindowsAzure.StorageClient as References. Best way to see how it all works is to create a blank project from an Azure template and copy over the necessary items.
Step 4
Next you need to define which performance counters to monitor. As such here is a great sample: http://code.msdn.microsoft.com/windowsazure/Windows-Azure-PerformanceCo-7d80ebf9
Extra Reference
Microsoft also has a few steps you can follow here that might help out if things still aren't working: http://msdn.microsoft.com/en-us/library/windowsazure/hh411521.aspx
Take a look at:
http://dunnry.com/blog/2012/02/27/SettingUpDiagnosticsMonitoringInWindowsAzure.aspx
There is also a lot of information on:
http://msdn.microsoft.com/en-us/library/windowsazure/gg433048.aspx