I am new to groovy. I have a code like this.
String flavor
HashMap config = new HashMap([ ttl: 0, url: url, appName: appName, enable: true ])
client.put("${data}.json", config)
From this client Map I need to iterate the values of appName and enable.
For that I used get method... I am not sure about this.
def values = client.get("${data}.json");
while using this get method am getting following error. Since I am new to groovy i don't know what is happening here
groovy.lang.MissingMethodException: No signature of method: com.comcast.csv.haxor.SecureFirebaseRestClient.get() is applicable for argument types: (org.codehaus.groovy.runtime.GStringImpl) values: [testJson.json]
Possible solutions: get(com.comcast.tvx.megahttp.utils.URL, java.lang.Class), get(java.lang.String, java.lang.Class), grep(), grep(java.lang.Object), getAt(java.lang.String), wait()
not sure what you are trying to do, but (without knowing other details) I'd put your code that way:
Map config = [ ttl: 0, url: url, appName: appName, enable: true ]
client[ "${data}.json" ] = config
def values = client[ "${data}.json" ]
assuming, that you wanted to use getAt() (short-cut with [] ) method instead of get()
Try this:
def config = [ ttl: 0, url: url, appName: appName, enable: true ]
def endpoint = "${data}.json" as String
client.put(endpoint, config)
def values = client.get(endpoint, HashMap)
def appName = values.appName
def enable = values.enable
I couldn't find any info on SecureFirebaseRestClient, so I'm guessing about how it works.
Related
I'm writing my second project on FastAPI. And I got this error.
For example I have this code in my routers.users.py:
#router.get('/', response_model=Page[Users])
async def get_all_users(db: Session = Depends(get_db)):
return paginate(db.query(models.User).order_by(models.User.id))
And it works. It has fields limit and page in swagger documentation.
I tried to write the same for routers.recipes.py, but in this case I have no fields for pagination(limit, page) in swagger. Ok, I googled and found out that adding dependencies could help me. And now I see pagination parameters in swagger, but error is still the same.
routers.recipes:
#router.get('/', response_model=Page[PostRecipes], dependencies=[Depends(Params)])
async def get_all_recipes(db: Session = Depends(get_db)):
return paginate(db.query(models.Recipe).order_by(models.Recipe.id))
pagination:
class Params(BaseModel, AbstractParams):
page: int = Query(1, ge=1, description="Page number")
limit: int = Query(50, ge=1, le=100, description="Page size")
def to_raw_params(self) -> RawParams:
return RawParams(
limit=self.limit,
offset=self.limit * (self.page - 1),
)
class Page(BasePage[T], Generic[T]):
page: conint(ge=1) # type: ignore
limit: conint(ge=1) # type: ignore
__params_type__ = Params
#classmethod
def create(
cls,
items: Sequence[T],
total: int,
params: AbstractParams,
) -> Page[T]:
if not isinstance(params, Params):
raise ValueError("Page should be used with Params")
return cls(
total=total,
items=items,
page=params.page,
limit=params.limit,
)
__all__ = [
"Params",
"Page",
]
So, does anyone have ideas about it?
according to doc you have to specify default parameters,
your code should look like paginate(iterable, params)
Airflow version :2.0.2
Trying to create Emr cluster, by retrying data from AWS secrets manager.
I am trying to write an airflow dag and, my task is to get data from this get_secret function and use it in Spark_steps
def get_secret():
secret_name = Variable.get("secret_name")
region_name = Variable.get(region_name)
# Create a Secrets Manager client
session = boto3.session.Session()
client = session.client(service_name='secretsmanager', region_name=region_name)
account_id = boto3.client('sts').get_caller_identity().get('Account')
try:
get_secret_value_response = client.get_secret_value(SecretId=secret_name)
if 'SecretString' in get_secret_value_response:
secret_str = get_secret_value_response['SecretString']
secret=json.loads(secret_str)
airflow_path=secret["airflow_path"]
return airflow_path
...
I need to use "airflow_path" return value in below spark_steps
Spark_steps:
SPARK_STEPS = [
{
'Name': 'Spark-Submit Command',
"ActionOnFailure": "CONTINUE",
'HadoopJarStep': {
"Jar": "command-runner.jar",
"Args": [
'spark-submit',
'--py-files',
's3://'+airflow_path+'-pyspark/pitchbook/config.zip,s3://'+airflow_path+'-pyspark/pitchbook/jobs.zip,s3://'+airflow_path+'-pyspark/pitchbook/DDL.zip',
's3://'+airflow_path+'-pyspark/pitchbook/main.py'
],
},
},
I saw on the internet I need to use Xcom, is this right ?, and do I need to run this function in python operator first and then get the value. please provide an example as I am a newbie.
Thanks for your help.
Xi
Yes if you would like to pass dynamic stuff, leveraging xcom push/pull might be easier.
Leverage PythonOperator to push data into xcom.
See reference implementation:
https://github.com/apache/airflow/blob/7fed7f31c3a895c0df08228541f955efb16fbf79/airflow/providers/amazon/aws/example_dags/example_emr.py
https://github.com/apache/airflow/blob/7fed7f31c3a895c0df08228541f955efb16fbf79/airflow/providers/amazon/aws/example_dags/example_emr.py#L108
https://www.startdataengineering.com/post/how-to-submit-spark-jobs-to-emr-cluster-from-airflow/
I am creating an api and no idea why post method not allowed on any url.
views
class MessagesView(APIView):
permission_classes = (IsAuthenticated,)
def post(self, request):
serializer = MessageSerializer(data=request.data)
if serializer.is_valid():
serializer.save()
return Response(serializer.data, status=status.HTTP_201_CREATED)
return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
chat.urls
urlpatterns = [
path("<str:pk>/", ChatDetail.as_view()),
path("messages/", MessagesView.as_view()),
]
response
{
"detail": "Method \"POST\" not allowed."
}
I am providing the token for the request, so isAuthenticated does not do anything wrong here.
Your first pattern will fire if you visit messages/. Indeed, its <str:pk> parameter matches any string (with at least one character and without slashes). But messages is thus also matched with this view.
What you can do is swap the places of the two urls, then calling messages/ will fire the correct view:
urlpatterns = [
# ↓ messages first
path('messages/', MessagesView.as_view()),
path('<str:pk>/', ChatDetail.as_view()),
]
If pk is an integer, you can further restrict the pk with the <int:…> path converter:
urlpatterns = [
path('messages/', MessagesView.as_view()),
path('<int:pk>/', ChatDetail.as_view()),
]
I have an Airflow DAG that starts an AWS EMR cluster to run steps. On the DAG we pass some variables that are set on Airflow Variables. But some of these variables are encrypted at Airflow, but when passing to EMR, we can see then clearly at EMR console. Is there any way to hide this?
Here is how we are defining the step. The airflow variable db_pass must be encrypted or hidden somehow
{
"Name": "EMR JOB",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": [
"{{var.value.job_script}}",
"--database_user={{var.value.db_user}}",
"--database_pass={{var.value.db_pass}}"
]
}
}
]
This SAMPLE_STEP_DEFINITION is then passed as to the EmrAddStepsOperator:
...
sample_task = EmrAddStepsOperator(
steps=SAMPLE_STEP_DEFINITION,
...
There are several ways to do this. First I would suggest to encrypt passwords with KMS. Here is the code how to do it:
def encryptString(plainText: String, keyArn: String): String = {
val req = new EncryptRequest().withKeyId(keyArn).withPlaintext(ByteBuffer.wrap(plainText.getBytes))
Base64.getEncoder.encodeToString(kmsClient.encrypt(req).getCiphertextBlob.array())
}
def decryptString(encryptedText: String, keyArn: String): String = {
val req = new DecryptRequest().withCiphertextBlob(ByteBuffer.wrap(Base64.getDecoder.decode(encryptedText)))
new String(kmsClient.decrypt(req).getPlaintext.array())
}
Your just need to attach decrypt permission to EMR_EC2_DefaultRole.
Another way is to pass a config file stored on S3 with password.
I've got the following specs wrapped around a controller. I've got the following Jasmine Spec:
describe 'MyApp.Controller.SomeController', ->
beforeEach module('mymodule')
beforeEach inject ($rootScope , $httpBackend, $controller, SomeService) ->
#scope = $rootScope.$new()
#httpBackend = $httpBackend
someService = SomeService
#someResource = someService.someResource
$controller 'MyApp.Controller.SomeController', $scope: #scope
describe "#fetchMethod", ->
describe "given an object", ->
beforeEach ->
#id = 17
#scope.fetchMethod(#id)
it "sets due to true", ->
#httpBackend.whenGET().respond(200, {"someStrings": ["foo", "bar"], otherStrings: ["bar", "goo"]})
expect(#scope.someStrings).toBeDefined()
expect(#scope.otherStrings).toBeDefined()
Wrapped around the following Controller:
MyApp.Controller.SomeController = (scope, someService) ->
scope.fetchMethod = (ID)->
someService.someResource.fetch
Id: artID
,(response) ->
scope.someStrings = response['someStrings']
scope.otherStrings = response['otherStrings']
scope.someStringsExist = true if scope.someStrings
MyApp.Controller.SomeController.$inject = ['$scope', 'SomeService']
Where SomeService is defined as follows:
MyApp.Service.SomeService = (resource) ->
#someResource = resource '/api/foos/:ID', {},
fetch:
method: 'GET'
#
MyApp.myModule.service 'SomeService', ['$resource', MyApp.Service.SomeService]
This setup appears to function on the site, correctly executing the request and returning values from the (rails) api endpoint.
However, when the jasmine specs are run it fails with:
Error: Unexpected request: GET /api/foos/17 No more request expected in http://localhost:3000/assets/helpers/angular-mocks.js?body=1 (line 889)
What am I missing? Why is httpBackend failing to recognize the GET request?
scope.initialize = (artifactId, artifactType)->
scope.artifactId = artifactId
scope.artifactType = artifactType
scope.currentVersionExists = false
scope.attachmentFetcher(scope.artifactId)
MyApp.Controller.SomeController.$inject = ['$scope', 'SomeService']
This line where you stub the response should go before you make the request:
#httpBackend.whenGET().respond(200, {"someStrings": ["foo", "bar"], otherStrings: ["bar", "goo"]})