I do like this:
https://airflow.apache.org/lineage.html#
in my airflow.cfg:
[lineage]
backend = airflow.lineage.backend.atlas
[atlas]
username = my_username
password = my_password
host = host
port = 21000
I run the sample DAG in ariflow, then I meet err as follows:
{init.py:44} DEBUG - Cannot import airflow.lineage.backend.atlas due to Module "airflow.lineage.backend" does not define a "atlas" attribute/class
could someone give me a help?
I just updated this part.
https://github.com/apache/airflow/blob/master/docs/lineage.rst#apache-atlas
You can try
backend = airflow.lineage.backend.atlas.AtlasBackend
Any feedback is welcomed :)
Related
I'm trying to deploy my flask application on Ubuntu 20.04 using Nginx and Gunicorn. It's been working smoothly when I run FLASK_APP=run.py and then gunicorn -w 9 run=app. However when I setup the configuration files for gunicorn and a config.py I was only able to access the home page but every other route results in an Internal Sever Error.
I set my log file to /var/log/app_name/app_name.err.log. The error reported in the logs is raise RuntimeError( RuntimeError: The session is unavailable because no secret key was set. Set the secret_key on the application to something unique and secret. .
Here is a copy of my config.py which is inside my application module where routes.py,models.py and forms.py are.
import os
import json
with open('/etc/scheye_config.json') as config_file:
config = json.load(config_file)
class Config:
SECRET_KEY = config.get('SECRET_KEY')
SQL_ALCHEMY_DATABASE_URI = config.get('SQL_ALCHEMY_DATABASE_URI')
MAIL_SERVER = config.get('MAIL_SERVER')
MAIL_PORT = 587
MAIL_USE_TLS = True
MAIL_USERNAME = config.get('EMAIL_USER')
MAIL_PASSWORD = config.get('APPLICATION_PASS')
As you can see I'm importing the settings from my json.config file whose code is shown below. The location for the json file is /etc/app_name.json.
{
"SECRET_KEY": "mysecretkey",
"SQLALCHEMY_DATABASE_URI": "database_uri",
"EMAIL_USER": "mymail#mail.com",
"APPLICATION_PASS": "application_password",
"MAIL_SERVER": "smtp.servermail.com"
}
In my init.py also has almost the same configurations only that I use environment variables. Here's a copy of the init.py.
import os
from flask import Flask
from sqlalchemy.orm import backref
from flask_sqlalchemy import SQLAlchemy
from flask_bcrypt import Bcrypt
from flask_login import LoginManager, login_manager
from flask_mail import Mail
from sqlalchemy import create_engine
app = Flask(__name__)
app.config['SECRET_KEY'] = os.environ.get('SECRET_KEY')
# configuring the database using SQLAlchemy
ENV = 'dev'
if ENV == 'dev':
app.debug=True
app.config['SQLALCHEMY_DATABASE_URI'] = os.environ.get('SQLALCHEMY_DATABASE_URI')
else:
app.debug=False
app.config['SQLALCHEMY_DATABASE_URI'] = os.environ.get('SQLALCHEMY_DATABASE_URI')
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
db = SQLAlchemy(app)
bcrypt = Bcrypt(app)
login_manager= LoginManager(app)
login_manager.login_view='login' #
login_manager.login_message_category='info' #styling the message returned
app.config['MAIL_SERVER'] = os.environ.get('MAIL_SERVER')
app.config['MAIL_PORT'] = 587
app.config['MAIL_USE_TLS'] = True
app.config['MAIL_USERNAME']= os.environ.get('EMAIL_USER')
app.config['MAIL_PASSWORD']= os.environ.get('APPLICATION_PASS')
#initializing our extension
mail = Mail(app)
engine = create_engine('database_uri')
connection = engine.raw_connection()
cursor = connection.cursor()
from app_name import routes
I have intentionally set my SQLALCHEMY_DATABASE_URI in order to continue testing the app before creating a production database.
I've gone through Configuration Handling in the flask documentation 2.2x and tried possible solutions but to no success. Being quite new to deployment I fear the answer is quite obvious and the source of the error is in the numerous configuration files. If anyone understands what the issue is I would highly appreciate his/her help.
in init.py you do not use your config module
try loading your base config, after app = Flask(__name__)
app.config.from_object('config.Config')
but if you haven't SECRET_KEY in env vars then SECRET_KEY will None again
then remove app.config['SECRET_KEY'] = os.environ.get('SECRET_KEY')
or use it in Config class for SECRET_KEY definition as:
class Config:
SECRET_KEY = os.environ.get('SECRET_KEY', config.get('SECRET_KEY'))
I have a valid Here licence and I have been using it for long time. Suddenly I had the above response from server.
The url is: http://pde.api.here.com/2/matchroute.json?app_id=xxxxxxxxxxxxxxx&app_code=xxxxxxxxxxxx&routemode=car&file=®ions=WEU&release=LATEST
def connectPost(self, url, data, headers=None):
return self.S.post(url, data=data, headers=headers , proxies = self.proxy_dict)
As per the licensing, and delivery back-end system The account has expired that's why.
Cheers,
WaliedCheetos
Try please this python code(it works properly for me)
# Hello World program in Python
import requests
import json
data = bytes("\n".join([
'LATITUDE,LONGITUDE',
'50.875182,4.499943',
'50.875240,4.499935',
'50.875302,4.499920',
'50.875375,4.499893',
'50.875450,4.499854',
'50.875527,4.499799',
'50.875602,4.499728',
'50.875670,4.499643'
]), 'utf-8')
url = 'http://pde.api.here.com/2/matchroute.json?app_id=xxxx&app_code=xxxxx&routemode=car®ions=WEU&release=LATEST'
resp = requests.post(url, data)
parsed = json.loads(resp.text)
print(json.dumps(parsed, indent=4, sort_keys=True))
This code in on playground
I have checked the akka http docs and have learned a lot about Directives and Routes. But I have not found a simple example of how I can extract a value from a header (say user: joesmith) and then do something like set that value in a threadlocal, then finally chain off to the rest of my routes. If my goal is not clear, I'm happy to provide a little code with and 'insert header grab' here comment to illustrate.
thanks in advance !
Turns out this is insanely easy. headerValueByName is the key to success. Below is a hello world akka http that grabs the 'User' header and displays it in the response:
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.HttpResponse
import akka.http.scaladsl.model.StatusCodes._
import akka.http.scaladsl.server.Directives._
import akka.stream.ActorMaterializer
object JunkServer {
implicit val actorSystem = ActorSystem("junk-server")
implicit val materializer = ActorMaterializer()
val route =
headerValueByName("User") { userId =>
path("") {
complete(
HttpResponse(
OK,
entity = s"hello there: ${userId}")
)
}
}
def main(args: Array[String]) {
Http().bindAndHandle(route, "localhost", 9999)
}
}
When you want to test it:
curl -H "User: dracula" localhost:9999
server response should be:
hello there: dracula
I have a code in python something like this which is using dynamodbmapper
from dynamodb_mapper.model import DynamoDBModel
class Employee(DynamoDBModel):
__table__ = u"employee"
__hash_key__ = u"emp_id"
__schema__ = {
u"emp_id": int,
u"name": unicode,
u"address": set,
}
__defaults__ = {
u"address": set([u"Konami"]),
}
I have all the credentials set on aws.
Just wrote a small python client to create a table in dynamodb using dynamodbmapper in python.
from dynamodb_mapper.model import ConnectionBorg
from influencers.data_access.Employee import Employee
conn = ConnectionBorg()
conn.create_table(Employee, 10, 10, wait_for_active=True)
I am tryinng to run this locally . I am running dynamodb locally . My question is how do i say the endpoint name for this client as http://localhost:8000. I looked at the java code and there is a cleaner way of entting endpoint on dynamodbclient. But in python i dont see it. Any help would be greatly appreciated.
We have .bat Files which call an R-script and generate some data and then send an email to certain users via rJython.
If I run this script in RStudio everything goes well and the mail is sent. But if i call the .bat file we get the following error.
It has probably something to do with the firewall in our company, can somebody help us out? We need to tell our IT departement something bc they cant help us either.
Error in jython.exec(rJython,mail): (-1, 'Unmapped exception:java.net.SocketException: Permission denied: connect') Execution halted
The R-Code to send the mail is:
#Sendmail Code
rJython<-rJython()
rJython$exec("import smtplib")
rJython$exec("from email.MIMEText import MIMEText")
rJython$exec("import email.utils")
rJython$exec("COMMASPACE = ', '")
recipients_string_j = paste("toaddrs = [",recipients_string,"]",sep="")
#mail config
mail<-c( fromaddr = 'test#XXX.com",
recipients_string_j,
paste("msg = MIMEText('",data.frame,"','html')",sep="",collapse=""),
paste("msg['Subject'] = 'DUMMY SUBJECT'"),
"msg['From'] = fromaddr",
"msg['To']= COMMASPACE.join(toaddrs)",
"server = smtplib.SMTP('00.00.00.000')",
"server.sendmail(fromaddr, toaddrs, msg.as_string())",
"server.quit()")
#send mail
jython.exec(rJython,mail)