I was hoping to use Airflow to orchestrate and manage my AWS Glue Jobs, however the operator no longer appears in the GitHub repo. Can anyone shed any light on this?
Additionally, i'm open to alternative orchestration tool suggestions. I need to be able to Execute Glue jobs, whilst also interacting with other EC2's within my VPC.
Thanks in advance,
Paul
Why not write a custom operator in airflow, which interacts with glue using boto3
The AWSGlueJobOperator is under construction and has good progress:
https://github.com/apache/airflow/pull/6007
right now we have GlueJobOperator
Related
I'm diving into web scraping and I was looking to add an IP rotation to my python script using the request library. I also want to put this script in production thanks to AWS lambda function.
From what I've read there seem to be two solutions: either use a VPN or use proxy lists with solutions such as crawlera, proxy rotators or others. However the latest solution is too expensive for me.
I've done some research and the first solution (VPN) seems to be the best solution in my case but I didn't find how to implement it on a script, can anyone give me a hint please?
Thanks a lot :)
Is there a way to create read only users in Airflow ? I see there is some discussion about that is Airflow 2.0 https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0
Does anyone know if that has been incorporated in 1.9 ? We are currently using 1.8.
There’s work ongoing in this PR:
https://github.com/apache/incubator-airflow/pull/3015
Design (early):
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+RBAC+proposal
To my knowledge it's not available right now. Maybe at the next major version
I have a lot of tests that expose REST API using requests library.
All tests take a lot of time to finish.
And I want to make them asynchronous, but I don't such experience.
Can someone help me with giving some advice or articles where I can find information about it? Or maybe you know another approach.
P.S: using xdist pytest plugin does not work for me, it does not start running my tests in parallel due to some reason that I don't know, not only me faced with this issue.
Now that the owner of ASIHTTPRequest is no longer working on the library, is there another alternative that is as good as this library?
Or maybe will the repository from their github be updated? By, maybe someone else who is well educated about the project (At least someone knowledgeable will still continue to work on it)
Thanks
If you look at https://github.com/pokeb/asi-http-request/commits/master you'll see that it has been updated since the owner stopped working on it...
I would recommend AFNetworking as the best supported option for a general networking library
http://afnetworking.com
If you are mainly working with a RESTful API, then RestKit is a great library to use instead of writing your own glue code:
http://restkit.org
I am developing a small client application for monitoring XenServer using XenAPI provided by citrix. I am able to get all the values(cpu,n/w read,n/w write, diskread,diskwrite...) but facing the below issue.
Can anybody please help me out in getting the memory (total,free,used) usage for the VM's present in the Xenserver using XenAPI. I tried the above by using VM_guest_metrics api call of VM, but its giving me the empty results. Please help me in this regard.
I have taken SDK(XenAPI) from the below link
http://community.citrix.com/display/xs/Download+SDKs
Thanks in Advance for your help.
The recommended way to get the data is to use the XAPI Round Robin Database (RRD) that comes with XAPI.
http://wiki.xen.org/wiki/XAPI_RRDs
See also the tutorials from Xen Day:
http://wiki.xen.org/wiki/Creating_a_LVM_backed_XFS_SR
In particular, the "Nuts and Bolts" session by Steven Maresca.
See also the code in OpenXenManager:
http://sourceforge.net/projects/openxenmanager/ as it is an open source clone of Citrix XenCenter and has performance graphs using XAPI.