How to use SimpleHttpOperator in Airflow - airflow

I get error : http_conn_id http_default isnt defined when I run following code:
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2020, 2, 27),
'email': ['ss#asd.com'],
'email_on_failure': True,
'email_on_retry': True,
'retries': 1,
'retry_delay': timedelta(seconds=5),
}
dag = DAG(
'dag1',
default_args=default_args,
description='A simple tutorial DAG',
schedule_interval='#daily',
)
t2 = SimpleHttpOperator(
task_id='get_labrador',
method='GET',
http_conn_id='http_default',
endpoint='api/breed/labrador/images',
headers={"Content-Type": "application/json"},
xcom_push=True,
dag=dag
)
What should be my http_conn_id value.
thanks


http_conn_id should contain the name of the Airflow Connection that contains the detail of the url you want to connect to. For example the Hostname to send to Slack Webhook would be https://hooks.slack.com/

Related

How to mock connection for airflow's Livy Operator using unittest.mock

#mock.patch.dict(
"os.environ",
AIRFLOW_CONN_LIVY_HOOK = "http://www.google.com",
clear= True
)
class TestLivyOperator(unittest.TestCase):
def setUp(self):
super().setUp()
self.dag = DAG(
dag_id = "test_livy",
default_args = {
"owner" : "xyz",
"start_date" : datetime(2022, 8, 16),
},
)
#requests_mock.mock()
def test_payload(self, mock_request):
task = LivyOperator(
task_id = "task_1",
class_name = "com.precious.myClass"
executor_memory = "512m",
executor_cores = 3,
arg = spark_args,
livy_conn_id = AIRFLOW_CONN_LIVY_HOOK,
dag = self.dag,
)
I get connection not defined error
airflow.exceptions.AirflowNotFoundException: The conn_id 'AIRFLOW_CONN_LIVY_HOOK' isn't defined

As per the Airflow doc
https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#storing-connections-in-environment-variables
The naming convention is AIRFLOW_CONN_{CONN_ID}, all uppercase (note the single underscores surrounding CONN). So if your connection id is my_prod_db then the variable name should be AIRFLOW_CONN_MY_PROD_DB.Hence the above code should be
#requests_mock.mock()
def test_payload(self, mock_request):
task = LivyOperator(
task_id = "task_1",
class_name = "com.precious.myClass"
executor_memory = "512m",
executor_cores = 3,
arg = spark_args,
livy_conn_id = "livy_hook",
dag = self.dag,
)

Moment.js doesn't show correct time

When running my node.js project locally on my computer, moment.js is showing correct time (2019-10-28T07:00:00.000Z because moment have adjusted for DST +02:00 in april and +01:00 in october)):
Moment {
_isAMomentObject: true,
_i: '2019-04-15T06:00:00.000Z',
_f: 'YYYY-MM-DDTHH:mm:ss.SSSSZ',
_tzm: 0,
_isUTC: false,
_pf:
{ empty: false,
unusedTokens: [],
unusedInput: [],
overflow: -1,
charsLeftOver: 0,
nullInput: false,
invalidMonth: null,
invalidFormat: false,
userInvalidated: false,
iso: true,
parsedDateParts: [ 2019, 3, 15, 6, 0, 0, 0 ],
meridiem: undefined,
rfc2822: false,
weekdayMismatch: false },
_locale:
Locale {
_calendar:
{ sameDay: '[Today at] LT',
nextDay: '[Tomorrow at] LT',
nextWeek: 'dddd [at] LT',
lastDay: '[Yesterday at] LT',
lastWeek: '[Last] dddd [at] LT',
sameElse: 'L' },
_longDateFormat:
{ LTS: 'h:mm:ss A',
LT: 'h:mm A',
L: 'MM/DD/YYYY',
LL: 'MMMM D, YYYY',
LLL: 'MMMM D, YYYY h:mm A',
LLLL: 'dddd, MMMM D, YYYY h:mm A' },
_invalidDate: 'Invalid date',
ordinal: [Function: ordinal],
_dayOfMonthOrdinalParse: /\d{1,2}(th|st|nd|rd)/,
_relativeTime:
{ future: 'in %s',
past: '%s ago',
s: 'a few seconds',
ss: '%d seconds',
m: 'a minute',
mm: '%d minutes',
h: 'an hour',
hh: '%d hours',
d: 'a day',
dd: '%d days',
M: 'a month',
MM: '%d months',
y: 'a year',
yy: '%d years' },
_months:
[ 'January',
'February',
'March',
'April',
'May',
'June',
'July',
'August',
'September',
'October',
'November',
'December' ],
_monthsShort:
[ 'Jan',
'Feb',
'Mar',
'Apr',
'May',
'Jun',
'Jul',
'Aug',
'Sep',
'Oct',
'Nov',
'Dec' ],
_week: { dow: 0, doy: 6 },
_weekdays:
[ 'Sunday',
'Monday',
'Tuesday',
'Wednesday',
'Thursday',
'Friday',
'Saturday' ],
_weekdaysMin: [ 'Su', 'Mo', 'Tu', 'We', 'Th', 'Fr', 'Sa' ],
_weekdaysShort: [ 'Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat' ],
_meridiemParse: /[ap]\.?m?\.?/i,
_abbr: 'en',
_config:
{ calendar: [Object],
longDateFormat: [Object],
invalidDate: 'Invalid date',
ordinal: [Function: ordinal],
dayOfMonthOrdinalParse: /\d{1,2}(th|st|nd|rd)/,
relativeTime: [Object],
months: [Array],
monthsShort: [Array],
week: [Object],
weekdays: [Array],
weekdaysMin: [Array],
weekdaysShort: [Array],
meridiemParse: /[ap]\.?m?\.?/i,
abbr: 'en' },
_dayOfMonthOrdinalParseLenient: /\d{1,2}(th|st|nd|rd)|\d{1,2}/ },
_a: [ 2019, 3, 15, 6, 0, 0, 0 ],
_d: 2019-10-28T07:00:00.000Z,
_isValid: true,
_z: null }
But on my server, I get this:
Moment {
_isAMomentObject: true,
_i: '2019-04-15T06:00:00.000Z',
_f: 'YYYY-MM-DDTHH:mm:ss.SSSSZ',
week: [Object],
weekdays: [Array],
weekdaysMin: [Array],
weekdaysShort: [Array],
meridiemParse: /[ap]\.?m?\.?/i,
abbr: 'en' },
_dayOfMonthOrdinalParseLenient: /\d{1,2}(th|st|nd|rd)|\d{1,2}/ },
_a: [ 2019, 3, 15, 6, 0, 0, 0 ],
_d: 2019-10-28T06:00:00.000Z,
_isValid: true,
_z: null }
The date is incorrectly set to 2019-10-28T06:00:00:000Z.
I have set the timezone on the server with sudo timedatectl set-timezone Europe/Oslo
If I am using moment.isDST() for dates on the server, it correctly gives my true in April and false in October.
I am unsure how to solve this, on the server or in my code?

I think I solved it. By answering the from Styx questions, I finally realized that the server app running in a docker container didn't have the correct timezone. By setting environment: TZ: "Europe/Oslo" in the docker-compose.yml file this fixed the problem, and I am getting correct time now.

flow error pass object w/ extra properties than parameter needs

I have the following code where defining a getByIds function that filters an array of objects, each of which has an id field and other fields.
// #flow
type X = {
id: string
};
type F = (Array<string>, Array<X>) => Array<X>;
import { curry } from "ramda";
const getByIds: F = curry((ids, xs) => xs.filter(x => ids.includes(x.id)));
export default getByIds;
when feeding a user = {id: 123, alias: 'foo'} to getByIds, flow warns that X doesn't have property alias . I thought flow allows extra properties (covariant?) to be attached to object of a less restricting type.
Here is the eslintrc.js
module.exports = {
env: {
// browser: true ensures things like window, or localStorage won't get complained
browser: true,
es6: true
},
plugins: [
"react",
"flowtype"
// "prettier",
],
extends: [
"eslint:recommended",
// "plugin:react/recommended",
"plugin:flowtype/recommended"
],
// https://github.com/feross/standard/issues/447
parser: "babel-eslint",
parserOptions: {
ecmaFeatures: {
modules: true,
classes: true,
experimentalObjectRestSpread: true,
jsx: true
},
sourceType: "module"
},
rules: {
// "indent": [
// "error",
// 4
// ],
// "prettier/prettier": "error",
"linebreak-style": ["error", "unix"],
// "quotes": [
// "error",
// "double"
// ],
// "semi": [
// "error",
// "never"
// ],
// "jsx-space-before-closing": 1,
// "no-undef": "error",
"no-unused-vars": [2, { argsIgnorePattern: "_" }], //https://eslint.org/docs/rules/no-unused-vars#argsignorepattern
"flowtype/define-flow-type": 1,
"comma-dangle": [1, "always-multiline"],
"always-multiline": 0,
"no-console": 0,
"no-constant-condition": 0,
"no-case-declarations": 0,
"react/no-danger": 0,
"react/display-name": 1,
"react/jsx-key": 1,
"react/jsx-no-comment-textnodes": 1,
"react/jsx-no-duplicate-props": 1,
"react/jsx-no-target-blank": 1,
"react/jsx-no-undef": 1,
"react/jsx-uses-react": 1,
"react/jsx-uses-vars": 1,
"react/no-children-prop": 1,
"react/no-danger-with-children": 1,
"react/no-deprecated": 1,
"react/no-direct-mutation-state": 1,
"react/no-find-dom-node": 1,
"react/no-is-mounted": 1,
"react/no-render-return-value": 1,
"react/no-string-refs": 1,
"react/no-unescaped-entities": 1,
"react/no-unknown-property": 1,
"react/prop-types": 1,
"react/react-in-jsx-scope": 1,
"react/require-render-return": 1,
"react/jsx-max-props-per-line": 1,
"react/jsx-first-prop-new-line": [1, "multiline-multiprop"],
"react/jsx-indent-props": [1, 2]
},
globals: {
module: true,
gon: true,
require: true,
__dirname: true,
_: true,
jest: true,
process: true,
it: true,
describe: true,
expect: true,
test: true,
SyntheticEvent: true,
SyntheticAnimationEvent: true,
SyntheticClipboardEvent: true,
SyntheticCompositionEvent: true,
SyntheticInputEvent: true,
SyntheticUIEvent: true,
SyntheticFocusEvent: true,
SyntheticKeyboardEvent: true,
SyntheticMouseEvent: true,
SyntheticDragEvent: true,
SyntheticWheelEvent: true,
SyntheticTouchEvent: true,
SyntheticTransitionEvent: true
},
settings: {
flowtype: {
onlyFilesWithFlowAnnotation: true
}
}
};
Any ideas?

how to hide the filter in ui by using angularjs

$scope.dataTableOpt = {
"aLengthMenu": [[10, 50, 100, -1], [10, 50, 100, 'All']],
"searching": false,
"paging": true,
"info": false,
"lengthChange": false
};
https://i.stack.imgur.com/T92Ep.png

$scope.dataTableOpt = {
//custom datatable options
// or load data through ajax call also
"aLengthMenu": [[10, 50, 100, -1], [10, 50, 100, 'All']],
"searching": false,
"paging": (listCount > 10 ? true : false),
"info": false,
"lengthChange": false
};

Add this to your options object:
"enablePaginationControls": false
If that doesn't work, you might also want to remove the "paging": true

Sendgrid Inbound Webhook Sending Empty Body

Trying to get Sendgrid's Inbound Parse Webhook following their instructions here
Using Meteor and Iron Router. I am capturing the request but the request body is empty. I'm following this tutorial https://sendgrid.com/blog/receive-inbound-email-meteorjs/ and https://sendgrid.com/blog/control-home-lighting-parse-webhook/, but when I do a console.log(this.request.body), it's returning an empty object {}.
I've also tried getting text (this.request.text), html, from, to, subject, but nothing. The only thing I've had success getting is this.request.headers.
The Sendgrid activity dashboard does show that the emails are being parsed, and I am clearly receiving them, but the body is empty. This is unexpected behaviour and I'm not sure how to troubleshoot. Can anybody point me in the right direction? Thanks. Here's a sample request.
{ _readableState:
{ highWaterMark: 16384,
buffer: [],
length: 0,
pipes: null,
pipesCount: 0,
flowing: false,
ended: false,
endEmitted: false,
reading: false,
calledRead: false,
sync: true,
needReadable: false,
emittedReadable: false,
readableListening: false,
objectMode: false,
defaultEncoding: 'utf8',
ranOut: false,
awaitDrain: 0,
readingMore: false,
decoder: null,
encoding: null },
readable: true,
domain: null,
_events: { close: [Function] },
_maxListeners: 10,
socket:
{ _connecting: false,
_handle:
{ fd: 34,
writeQueueSize: 0,
owner: [Circular],
onread: [Function: onread],
reading: true },
_readableState:
{ highWaterMark: 16384,
buffer: [],
length: 0,
pipes: null,
pipesCount: 0,
flowing: false,
ended: false,
endEmitted: false,
reading: true,
calledRead: true,
sync: false,
needReadable: true,
emittedReadable: false,
readableListening: false,
objectMode: false,
defaultEncoding: 'utf8',
ranOut: false,
awaitDrain: 0,
readingMore: false,
decoder: null,
encoding: null },
readable: true,
domain: null,
_events:
{ end: [Object],
finish: [Function: onSocketFinish],
_socketEnd: [Function: onSocketEnd],
drain: [Object],
timeout: [Function],
error: [Function],
close: [Object] },
_maxListeners: 10,
_writableState:
{ highWaterMark: 16384,
objectMode: false,
needDrain: false,
ending: false,
ended: false,
finished: false,
decodeStrings: false,
defaultEncoding: 'utf8',
length: 0,
writing: false,
sync: true,
bufferProcessing: false,
onwrite: [Function],
writecb: null,
writelen: 0,
buffer: [],
errorEmitted: false },
writable: true,
allowHalfOpen: true,
onend: [Function],
destroyed: false,
bytesRead: 8249,
_bytesDispatched: 0,
_pendingData: null,
_pendingEncoding: '',
server:
{ domain: null,
_events: [Object],
_maxListeners: 10,
_connections: 1,
connections: [Getter/Setter],
_handle: [Object],
_usingSlaves: false,
_slaves: [],
allowHalfOpen: true,
httpAllowHalfOpen: false,
timeout: 5000,
_connectionKey: '4:0.0.0.0:23683' },
_idleTimeout: 5000,
_idleNext:
{ _connecting: false,
_handle: [Object],
_readableState: [Object],
readable: true,
domain: null,
_events: [Object],
_maxListeners: 10,
_writableState: [Object],
writable: true,
allowHalfOpen: false,
onend: null,
destroyed: false,
bytesRead: 79134,
_bytesDispatched: 14036,
_pendingData: null,
_pendingEncoding: '',
_idleTimeout: 30000,
_idleNext: [Object],
_idlePrev: [Circular],
_idleStart: 1459115857090,
_monotonicStartTime: 543430569,
pipe: [Function],
addListener: [Function: addListener],
on: [Function: addListener],
pause: [Function],
resume: [Function],
read: [Function],
_consuming: true },
_idlePrev: { _idleNext: [Circular], _idlePrev: [Object] },
_idleStart: 1459115857771,
_monotonicStartTime: 543431251,
parser:
{ _headers: [],
_url: '',
onHeaders: [Function: parserOnHeaders],
onHeadersComplete: [Function: parserOnHeadersComplete],
onBody: [Function: parserOnBody],
onMessageComplete: [Function: parserOnMessageComplete],
socket: [Circular],
incoming: [Circular],
maxHeaderPairs: 2000,
onIncoming: [Function] },
ondata: [Function],
_paused: false,
_httpMessage:
{ domain: null,
_events: [Object],
_maxListeners: 10,
output: [],
outputEncodings: [],
writable: true,
_last: false,
chunkedEncoding: false,
shouldKeepAlive: true,
useChunkedEncodingByDefault: true,
sendDate: true,
_headerSent: false,
_header: '',
_hasBody: true,
_trailer: '',
finished: false,
_hangupClose: false,
socket: [Circular],
connection: [Circular],
_headers: [Object],
_headerNames: [Object],
write: [Function],
end: [Function] } },
connection:
{ _connecting: false,
_handle:
{ fd: 34,
writeQueueSize: 0,
owner: [Circular],
onread: [Function: onread],
reading: true },
_readableState:
{ highWaterMark: 16384,
buffer: [],
length: 0,
pipes: null,
pipesCount: 0,
flowing: false,
ended: false,
endEmitted: false,
reading: true,
calledRead: true,
sync: false,
needReadable: true,
emittedReadable: false,
readableListening: false,
objectMode: false,
defaultEncoding: 'utf8',
ranOut: false,
awaitDrain: 0,
readingMore: false,
decoder: null,
encoding: null },
readable: true,
domain: null,
_events:
{ end: [Object],
finish: [Function: onSocketFinish],
_socketEnd: [Function: onSocketEnd],
drain: [Object],
timeout: [Function],
error: [Function],
close: [Object] },
_maxListeners: 10,
_writableState:
{ highWaterMark: 16384,
objectMode: false,
needDrain: false,
ending: false,
ended: false,
finished: false,
decodeStrings: false,
defaultEncoding: 'utf8',
length: 0,
writing: false,
sync: true,
bufferProcessing: false,
onwrite: [Function],
writecb: null,
writelen: 0,
buffer: [],
errorEmitted: false },
writable: true,
allowHalfOpen: true,
onend: [Function],
destroyed: false,
bytesRead: 8249,
_bytesDispatched: 0,
_pendingData: null,
_pendingEncoding: '',
server:
{ domain: null,
_events: [Object],
_maxListeners: 10,
_connections: 1,
connections: [Getter/Setter],
_handle: [Object],
_usingSlaves: false,
_slaves: [],
allowHalfOpen: true,
httpAllowHalfOpen: false,
timeout: 5000,
_connectionKey: '4:0.0.0.0:23683' },
_idleTimeout: 5000,
_idleNext:
{ _connecting: false,
_handle: [Object],
_readableState: [Object],
readable: true,
domain: null,
_events: [Object],
_maxListeners: 10,
_writableState: [Object],
writable: true,
allowHalfOpen: false,
onend: null,
destroyed: false,
bytesRead: 79134,
_bytesDispatched: 14036,
_pendingData: null,
_pendingEncoding: '',
_idleTimeout: 30000,
_idleNext: [Object],
_idlePrev: [Circular],
_idleStart: 1459115857090,
_monotonicStartTime: 543430569,
pipe: [Function],
addListener: [Function: addListener],
on: [Function: addListener],
pause: [Function],
resume: [Function],
read: [Function],
_consuming: true },
_idlePrev: { _idleNext: [Circular], _idlePrev: [Object] },
_idleStart: 1459115857771,
_monotonicStartTime: 543431251,
parser:
{ _headers: [],
_url: '',
onHeaders: [Function: parserOnHeaders],
onHeadersComplete: [Function: parserOnHeadersComplete],
onBody: [Function: parserOnBody],
onMessageComplete: [Function: parserOnMessageComplete],
socket: [Circular],
incoming: [Circular],
maxHeaderPairs: 2000,
onIncoming: [Function] },
ondata: [Function],
_paused: false,
_httpMessage:
{ domain: null,
_events: [Object],
_maxListeners: 10,
output: [],
outputEncodings: [],
writable: true,
_last: false,
chunkedEncoding: false,
shouldKeepAlive: true,
useChunkedEncodingByDefault: true,
sendDate: true,
_headerSent: false,
_header: '',
_hasBody: true,
_trailer: '',
finished: false,
_hangupClose: false,
socket: [Circular],
connection: [Circular],
_headers: [Object],
_headerNames: [Object],
write: [Function],
end: [Function] } },
httpVersion: '1.1',
complete: false,
headers:
{ 'x-forwarded-proto': 'http',
'x-forwarded-port': '80',
'x-forwarded-for': '167.89.125.249,127.0.0.1',
'content-type': 'multipart/form-data; boundary=xYzZY',
'content-length': '8782',
'user-agent': 'SendGrid 1.0',
host: '6f496891.ngrok.io',
connection: 'TE, close',
te: 'deflate,gzip;q=0.3' },
trailers: {},
_pendings: [],
_pendingIndex: 0,
url: '/webhook/sendgrid',
method: 'POST',
statusCode: null,
client:
{ _connecting: false,
_handle:
{ fd: 34,
writeQueueSize: 0,
owner: [Circular],
onread: [Function: onread],
reading: true },
_readableState:
{ highWaterMark: 16384,
buffer: [],
length: 0,
pipes: null,
pipesCount: 0,
flowing: false,
ended: false,
endEmitted: false,
reading: true,
calledRead: true,
sync: false,
needReadable: true,
emittedReadable: false,
readableListening: false,
objectMode: false,
defaultEncoding: 'utf8',
ranOut: false,
awaitDrain: 0,
readingMore: false,
decoder: null,
encoding: null },
readable: true,
domain: null,
_events:
{ end: [Object],
finish: [Function: onSocketFinish],
_socketEnd: [Function: onSocketEnd],
drain: [Object],
timeout: [Function],
error: [Function],
close: [Object] },
_maxListeners: 10,
_writableState:
{ highWaterMark: 16384,
objectMode: false,
needDrain: false,
ending: false,
ended: false,
finished: false,
decodeStrings: false,
defaultEncoding: 'utf8',
length: 0,
writing: false,
sync: true,
bufferProcessing: false,
onwrite: [Function],
writecb: null,
writelen: 0,
buffer: [],
errorEmitted: false },
writable: true,
allowHalfOpen: true,
onend: [Function],
destroyed: false,
bytesRead: 8249,
_bytesDispatched: 0,
_pendingData: null,
_pendingEncoding: '',
server:
{ domain: null,
_events: [Object],
_maxListeners: 10,
_connections: 1,
connections: [Getter/Setter],
_handle: [Object],
_usingSlaves: false,
_slaves: [],
allowHalfOpen: true,
httpAllowHalfOpen: false,
timeout: 5000,
_connectionKey: '4:0.0.0.0:23683' },
_idleTimeout: 5000,
_idleNext:
{ _connecting: false,
_handle: [Object],
_readableState: [Object],
readable: true,
domain: null,
_events: [Object],
_maxListeners: 10,
_writableState: [Object],
writable: true,
allowHalfOpen: false,
onend: null,
destroyed: false,
bytesRead: 79134,
_bytesDispatched: 14036,
_pendingData: null,
_pendingEncoding: '',
_idleTimeout: 30000,
_idleNext: [Object],
_idlePrev: [Circular],
_idleStart: 1459115857090,
_monotonicStartTime: 543430569,
pipe: [Function],
addListener: [Function: addListener],
on: [Function: addListener],
pause: [Function],
resume: [Function],
read: [Function],
_consuming: true },
_idlePrev: { _idleNext: [Circular], _idlePrev: [Object] },
_idleStart: 1459115857771,
_monotonicStartTime: 543431251,
parser:
{ _headers: [],
_url: '',
onHeaders: [Function: parserOnHeaders],
onHeadersComplete: [Function: parserOnHeadersComplete],
onBody: [Function: parserOnBody],
onMessageComplete: [Function: parserOnMessageComplete],
socket: [Circular],
incoming: [Circular],
maxHeaderPairs: 2000,
onIncoming: [Function] },
ondata: [Function],
_paused: false,
_httpMessage:
{ domain: null,
_events: [Object],
_maxListeners: 10,
output: [],
outputEncodings: [],
writable: true,
_last: false,
chunkedEncoding: false,
shouldKeepAlive: true,
useChunkedEncodingByDefault: true,
sendDate: true,
_headerSent: false,
_header: '',
_hasBody: true,
_trailer: '',
finished: false,
_hangupClose: false,
socket: [Circular],
connection: [Circular],
_headers: [Object],
_headerNames: [Object],
write: [Function],
end: [Function] } },
_consuming: false,
_dumped: false,
httpVersionMajor: 1,
httpVersionMinor: 1,
upgrade: false,
originalUrl: '/webhook/sendgrid',
_parsedUrl:
{ protocol: null,
slashes: null,
auth: null,
host: null,
port: null,
hostname: null,
hash: null,
search: null,
query: null,
pathname: '/webhook/sendgrid',
path: '/webhook/sendgrid',
href: '/webhook/sendgrid' },
body: {},
query: {} }

So, the Parse API makes a POST request to an endpoint of your choosing, but the data is encoded as a multipart/form-data. I recommend using Formidable which is an express middleware to parse the data, and then extract it via a regular key-value lookup. Here's some sample code to help you out:
var form = new formidable.IncomingForm();
form.parse(req, function(err, fields, files){//do something here})

Resources