I recently created a new production cloud composer environment and migrated my existing dags to it and have been getting the error
INFO - Task exited with return code Negsignal.SIGSEGV
on some tasks that work perfectly fine in my dev environment (also on cloud composer). The production environment is loaded from a snapshot of the dev environment so they should be identical. Is there anything configuration wise that might not have come over in the snapshot that I can look into so I can try and get this resolved?
For added context, I have upgraded the scheduler, web server and worker specs in the prod environment to be more than the dev environment and this is still causing the same issue.
Are the instances the same size? This message means that some tasks run out of resources so you have either to scale up or modify your code so it is more frugal and you don't run out of resources.
Also have you checked how the dags are scheduled? Maybe you have many dags running at the same time and they use up all your resources.
Related
I was trying to follow this post on using remote branches to create a staging environment
I could not figure out the hooks part of it but I created an empty repo, created a staging remote branch:
git remote add staging https://github.com/myusername/newEmptyRepo.git
git push staging master
This did fill my staging repo with project code, but the code was several months old without many of the components I have added since then. I tried to make a small change and go through the add/commit/push process, but was then told "Everything up-to-date" when it obviously isn't in anyway, especially since I just made a change.
How do I force it to actually push the new code to my staging repo?
Alternatively, Is there a better way to make a staging env? I have seen lots of recommendations for github actions, but those seem to only work with cloud deployments like GCP/AWS. I am deploying with firebase and want to deploy staging to a subdomain and then, when ready, manually deploy to prod.
So, I'm helping someone out with a project which is developed by someone else, but he left. Well, not too relevant but I'm struggling to get a multi environment setup running. Hopefully someone can push me in the right direction.
In the beginning there was one environment. Only production. But for the changes I want an environment running next to the production env. This will be staging.
I have been searching the web on how to set this up and I got something working. It seems to deploy everything, however the changes are not visible on the staging environment.
This is what I have right now in my readme:
#### Deploy to staging
Build the application: `npm run build`
Make sure you are at the correct environment: `firebase use staging`
And apply the target (??) ; `firebase target:apply hosting staging exp-game-staging`
Now you can deploy with:
Initially, for fresh deploy: `firebase deploy`
And later for consequent deployments: `firebase deploy --only hosting`
This gives met the following output:
=== Deploying to 'exp-game-staging'...
i deploying hosting
i hosting[exp-game-staging]: beginning deploy...
i hosting[exp-game-staging]: found 739 files in dist/exp-game-staging
✔ hosting[exp-game-staging]: file upload complete
i hosting[exp-game-staging]: finalizing version...
✔ hosting[exp-game-staging]: version finalized
i hosting[exp-game-staging]: releasing new version...
✔ hosting[exp-game-staging]: release complete
✔ Deploy complete!
Project Console: https://console.firebase.google.com/project/***/overview
Hosting URL: https://***.web.app
For the files this seems to work, but it's still using the production database from the staging environment. Locally tho, this is not the case and when I'm developing locally it's using the staging database as expected.
Does anyone know what I'm missing here?
I understand that you are trying to create two environments one for Production and the other for Staging in your Firebase project, but you are having the issue that the staging stage is using the database from the production environment.
Make sure to refer to the steps mentioned in the documentation when you create the two separate environments. Also, the recommendation is to create separate databases for both the environments.
You can achieve this by taking a backup of the data to a storage bucket, and then restore it from there into the other database.
Also, you may refer to this Firestore backup/export/import info here.
If you're using Firebase RTDB, and not Firestore - this documentation might help.
I'm developing a DAG on Cloud Composer; my code is separated into a main python file and one package with subfolders, it looks like this:
my_dag1.py
package1/__init__.py
package1/functions.py
package1/package2/__init__.py
package1/package2/more_functions.py
I updated one of the functions on package1/functions.py to take an additional argument (and update the reference in my_dag1.py). The code would run correctly on my local environment and I was not getting any errors when running
gcloud beta composer environments run my-airflow-environment list_dags --location europe-west1
But the Web UI raised a python error
TypeError: my_function() got an unexpected keyword argument
'new_argument'
I have tried to rename the function and the error changed to
NameError: name 'my_function' is not defined
I tried changing the name of the DAG and to upload the files to the dag folder zipped and unzipped, but nothing worked.
The error disappeared only after I renamed the package folder.
I suspect the issue is related to the scheduler picking up my_dag1.py but not package1/functions.py. The error appeared out of nowhere as I have made similar updates on the previous weeks.
Any idea on how to fix this issue without refactoring the whole code structure?
EDIT-1
Here's the link to related discussion on Google Groups
I've run into a similar issue. the "Broken DAG" error won't dismiss in Web UI. I guess this is a cache bug in Web server of AirFlow.
Background.
I created a customized operator with Airflow Plugin features.
After I import the customized operator, the airflow Web UI keep shows the Broken DAG error says that it can't find the customized operator.
Why I think it's a bug in Web server Airflow?
I can manually run the DAG with the command airflow test, so the import should be correct.
Even if I remove the related DAG file from the /dags/ folder of airflow, the error still there.
Here are What I did to resolve this issue.
restart airflow web service. (sometimes you can resolve the issue only by this).
make sure no DAG is running, restart airflow scheduler service.
make sure no DAG is running, restart airflow worker
Hopefully, it can help someone has the same issue.
Try restarting the webserver with:
gcloud beta composer environments restart-web-server ENVIRONMENT_NAME --location=LOCATION
The Setup:
I'm setting up a Wordpress-powered application using Elastic Beanstalk from Amazon Web Services. All development is being done locally under a MAMP apache2/php5 server environment with a GIT repository controlling the entire application root.
Deployment Workflow:
After committing any code changes (edits, new plugins, etc) to the repo the application is deployed using AWS EB CLI's eb deploy command which pushes the latest version out to any running EC2 instances managed by Elastic Beanstalk.
My Issue:
Sometimes the code changes aren't exactly syncing up between my development/production environments and I'm not sure how to overcome it. Especially when trying to install and setup plugins like W3 Total Cache or WP Super Cache.
Since my local environment doesn't have things like a memcahced server installed, but my production environment does (ElastiCache) I'm unable to save the proper settings file and deploy it for use in my production environment. These plugins won't allow me to select the needed services because it sees them as not available...
It seems I can only get W3 Total Cache to work if I install it directly onto a live production environment, which seems like a bad idea.
Given the above:
Am I going about deployments the wrong way?
Should plugins like W3 Total Cache be installed and configured on
local development environments and pushed to production environments?
I cannot comment on the issues specific to Elastic Beanstalk, but based on experience I can make a suggestion about the second part of your issue statement:
You are better off running a development environment that mirrors your production environment as closely as possible. I suggest that you convert from MAMP to a VM environment like VirtualBox. You might want to check out puphpet.com for help in getting it set up. It requires some startup effort, but gives you an environment similar to or the same as your production servers. For example, you could run memcached yourself so you could actually test it with W3 Total Cache.
As for your second question, just installing a plugin in the production environment without testing it beforehand has obvious risks (but then again clients do that all the time). I would prefer to test first. To a certain extent it probably depends on how critical it is if the site experiences downtime or weirdness.
I would suggest you to create another environment on Beanstalk.
It's easy, fast and more reliable than a VM in your case because it will allow you to test your deployment process as well.
I usually have 3 environment for a every website. Each environment is on its own branch. If your configuration is different between environment (url and database access for example), just store your wp-config and other config files into S3 (you may not want production password in your git repository), and through ebextensions you can download them into your website automatically.
I use AWS Beanstalk that way for 16 websites and some are wordpress one. All with autoscaling and able to get thousands of users simultaneously.
Don't hesitate to ask me for further details.
I am attempting to setup automated tests for our applications using a virtual machine environment.
What I would like to have is something like the following scenario:
Build server is automatically triggered to start an automated test for the application
A "build" script is then run which consist of:
Copy application files and a test script to a location accessible by the VM
Start the VM
In the VM, a special application looks in the shared folder and start the test script
The tests script do its job, results are output to shared folder
Test script ends
The special application then delete the test script
The special application somehow have the VM manager close the VM and revert to the previous snapshot
When the VM has exited, process the result and send to build server.
I am using TeamCity if that matters.
For virtual machines, we use VirtualBox but we are open to any other if needed.
Is there any applications/suite that would manage this scenario?
If there are none then I would then code it myself, should be easy but the only part I am not sure is the handling of the virtual machine.
What I need to be able to do is to have the VM close itself after the test and revert to a previous snapshot since I want it to be in a known state for the next test.
Any pointers?
I have a similar setup running and I chose to use Vagrant as its the same thing our developers where using for normalizing the development environment.
The initial state of the virtualmachine was scripted using puppet, but we didn't run the deployment scripts from scratch on each test, only once a day.
You could use puppet/chef for everything, but for all other operations on the VM, we would use Fabric scripts, as they were used for the real deployment too, and somehow fitted how we worked better. In sum the script would look something like the following:
vagrant up # fire up the vm, and run the puppet provisioning tool
fab vm run_test # run tests on vm
fab local process_result # process results on local shared folder
vagrant destroy # destroy the vm
The advantage is that your developers can also use vagrant to mimic your production environment without having to take care of that themselves (i.e. changes to your database settings get synced to all your developers vm's wherever they are) and the same scripts can be used in production too.
VirtualBox does have a COM API. I have no experience with it, but it may be possible to use that. One option would be to have TeamCity fire off a script to do this. I'd suggest starting with NAnt (supported natively by TeamCity) and possibly executing PowerShell if necessary.
Though I don't have any experience with either, I happen to have heard of a couple applications in this space recently:
http://www.infoq.com/news/2011/05/virtual_machine_test_harness
http://www.automatedqa.com/techpapers/testcomplete/automated-testing-in-virtual-labs/