Following the documentation noted in the wiki, I'm trying to use the KeyczarTool to generate new keyset. Anyone else come across this FileNotFoundException? The KeyczarTool.jar has rwx permissions and tried running via sudo.
From docs
Command Usage:
create --location=/path/to/keys --purpose=(crypt|sign) [--name="A name"] [--asymmetric=(dsa|rsa|ec)]
Creates a new, empty key set in the given location.
This key set must have a purpose of either "crypt" or "sign"
and may optionally be given a name. The optional version
flag will generate a public key set of the given algorithm.
The "dsa" and "ec" asymmetric values are valid only for sets
with "sign" purpose.
Cmd:
$ java -jar KeyczarTool-0.71f-060112.jar create --location=/keys --purpose=crypt -name="first key" --asymmetric=rsa
output:
org.keyczar.exceptions.KeyczarException: Unable to write to: /keys/meta
at org.keyczar.KeyczarTool.create(KeyczarTool.java:366)
at org.keyczar.KeyczarTool.main(KeyczarTool.java:123)
Caused by: java.io.FileNotFoundException: /keys/meta (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
at org.keyczar.KeyczarTool.create(KeyczarTool.java:362)
... 1 more
With the current version of java keyczar the directory "keys" needs to be created first before running the program.
This is a known issue KeyczarTool should create directories automatically.
As #jbtule kindly pointed out you must create the keys dir first. But also include . before the slash.
Correct working command is:
$ java -jar KeyczarTool-0.71f-060112.jar create --location=./keys --purpose=crypt -name="first key" --asymmetric=rsa
Related
With an Oozie coordinator and workflow, I see the following in the Coord Job Log for a specific action:
JOB[0134742-190911204352052-oozie-oozi-C] ACTION[0134742-190911204352052-oozie-oozi-C#1] [0134742-190911204352052-oozie-oozi-C#1]::CoordActionInputCheck:: Missing deps: ${coord:latest(0)}#${coord:latest(0)}#${coord:latest(0)}#${coord:latest(0)}#${coord:latest(0)}#${coord:latest(0)}
It seems the full path names are missing. If the path name is not specified in the coordinator with latest(0), the paths are available as seen here:
JOB[0134742-190911204352052-oozie-oozi-C] ACTION[0134742-190911204352052-oozie-oozi-C#1] [0134742-190911204352052-oozie-oozi-C#1]::CoordActionInputCheck:: Missing deps:hdfs://labs-xxx/data/funcxx/inputs/uploads/reports-for-targeting/20190923/14
Later the paths is resolved as:
JOB[0134742-190911204352052-oozie-oozi-C] ACTION[0134742-190911204352052-oozie-oozi-C#1] [0134742-190911204352052-oozie-oozi-C#1]::ActionInputCheck:: File:hdfs://labs-xxx/data/funcxx/inputs/uploads/reports-for-targeting/20190923/14, Exists? :true
How can I see the full path name instead of the ${coord:latest(0)} strings?
You can check this vis oozie cli -
oozie job -info 0134742-190911204352052-oozie-oozi-C#1
Is there a way to encrypt the airflow config file sql_alchemy_conn string , the password shown in example is plaintext . What options are there to secure it. Also if the password has special chars how it must be escaped in the config file
Trying to install airflow using airflow role.
# See: https://www.sqlalchemy.org/
sql_alchemy_conn:
value: "postgresql+psycopg2://pgclusteradm#servername:PLAINTEXTPASSWORD#server.postgres.database.azure.com/airflow2"
Way to encrypt password, couldn't find how to encrypt this.
You can provide the database URI through environment variables instead of the config file. This doesn't encrypt it or necessarily make it more secure, but it at least isn't plainly sitting in a permanent file.
In your airflow.cfg you can put a placeholder:
[core]
...
sql_alchemy_conn = override_me
...
Then set AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://... in an environment variable when you bring up Airflow components. This way of setting and overriding configuration options through environment variables is detailed in the docs, but the basic format is AIRFLOW__{SECTION}__{KEY}=<value>.
There are 2 ways of securing this as mentioned in docs:
1) Environment Variable:
You can override the setting in airflow.cfg by setting the following environment variable:
AIRFLOW__CORE__SQL_ALCHEMY_CONN=my_conn_string
This way you can keep the setting in airflow.cfg as empty so no one can view the password.
2) Get string by running command:
You can also derive the connection string at run time by appending _cmd to the key like this:
[core]
sql_alchemy_conn_cmd = bash_command_to_run
I would like to set the configuration of my symfony2 project using environment variables.
In the server I have defined:
SYMFONY__DATABASE__USER
SYMFONY__DATABASE__PASSWORD
SYMFONY__DATABASE__NAME
SYMFONY__DATABASE__HOST
SYMFONY__DATABASE__DRIVER
My parameters.yml.dist looks like this:
#app/config/parameters.yml.dist
parameters:
database_host: "%database.host%"
database_port: ~
database_name: "%database.name%"
database_user: "%database.user%"
database_password: "%database.password%"
database_driver: "%database.driver%"
when I run composer I get an exception
composer install --dev --no-interaction --prefer-source
[Symfony\Component\DependencyInjection\Exception\ParameterNotFoundException]
You have requested a non-existent parameter "database.driver". Did you mean one of these: "database_user", "database_driver"?
These variables are defined in the server so I can modify the parameters.yml.dist to define these values. But this does not seams the right way, because wat I really want to use are the environment variables.
Note: I want to read this environment variables in travis, heroku and my vagrant machine. I only want to have in the repository the vagrant machine variables.
Which is the proper way to do this?
How should look my parameters.yml.dist?
Looks you are doing everything okay.
Here is the complete documentation for Setting Environment Variables which I believe you already read.
What is important to note is this:
Also, in order for your console to work (which does not use Apache),
you must export these as shell variables. On a Unix system, you can
run the following:
$ export SYMFONY__DATABASE__USER=user
$ export SYMFONY__DATABASE__PASSWORD=secret
I remember once I have a similar issue, I was setting everything on APACHE, but when running commands it wasn't working because I forgot to EXPORT the variables on the system.
Be aware that using export is a temp solution, if you reset your server those values will be lost, you will need to setup in a permanent way according to your OS.
I think you solved this long time ago, but the problem is actually that you have 2 _ between DATABASE and USER and the parser for this have a string replace function that replaces every __ with a . .
For your example to work you should have written like this:
SYMFONY__DATABASE_USER -> database_user
SYMFONY__DATABASE__USER -> database.user
You can try this bundle if your system version is >= 2.6.2:
This bundle provides a way to read parameters from environment
variables at runtime. The value defined in the container parameter is
used as fallback when the environment variable is not available.
I have read other posts, when searching, an answer to this question.
I am using PostgreSQL 9.1, and created extension 'citext' using CREATE EXTENSION citext, but when I try to create any columns of type 'citext', it throws this error
ERROR: type "citext" does not exist
I researched but did not find any concrete answers? Any idea why?
Ok figured it out. I have several databases and CREATE EXTENSION citext has to be run for each db to install the extension in that DB. You must do on psql prompt:
psql =# \c db_1
CREATE EXTENSION citext;
psql =# \c db_2
CREATE EXTENSION citext;
#NullException is correct that the extension needs to be created in each database. If you want to automatically have an extension created, you can create it in the template1 database which (by default, at least) is the database used as a model for "create database", so with appropriate permissions, in psql:
\c template1
create extension citext;
Then new databases will include citext by default.
To use citext, use the CITextExtension operation to setup the citext extension in PostgreSQL before the first CreateModel migration operation.
https://docs.djangoproject.com/en/2.2/ref/contrib/postgres/fields/#citext-fields
from django.contrib.postgres.operations import CITextExtension
class Migration(migrations.Migration):
...
operations = [
CITextExtension(),
...
]
similarly to HStoreField as
https://docs.djangoproject.com/en/2.2/ref/contrib/postgres/operations/#create-postgresql-extensions
If you use Docker, and want to add this extension to your database,
I have done the following,
# Dockerfile
FROM postgres:11.3
# Adds the CIText Extension to our database
COPY ./compose/production/postgres/initdb_citext.sh /docker-entrypoint-initdb.d/citext.sh
And my initdb_citext.sh:
#!/bin/sh
# Adds the citext extension to database and test database
"${psql[#]}" <<- 'EOSQL'
CREATE EXTENSION IF NOT EXISTS citext;
\c template1
CREATE EXTENSION IF NOT EXISTS citext;
EOSQL
This applies the extension to test databases that django generates too.
I need a Linux command to get current value of java.io.tmpdir
Also can you tell me the command to get all the system propeties.
JDK provides tools with which you can achieve your goal. Using the specified command, you can display the value of a system variable java.io.tmpdir.
jinfo <process id> | grep "java.io.tmpdir"
The <process id> is the number of the java process for which you want to display the value of the system variable. Id of the java process you can get by using the jps tool, which list all java processes running on your machine.
To see all system variables of the java process, please use this command
jinfo <process id>
All JDK tools are located in $JAVA_HOME/bin directory.
java.io.tmpdir is one of java system properties, so its value exists only inside jvm. To find out what is the value of java.io.tmpdir property you can also write a simple program in java. It may look something like this piece of code:
public class JSystemProperties {
public static void main(String[] args) {
System.getProperties().list(System.out);
}
}
The code above will print all java system properties, but you can modify this to print only one system property with name you pass through args array (using System.getProperty(key) method). After you compile this class you can create script (which will run compiled java class) that can be treated as Linux command to get current values of java system properties.
For black box Java applications I've deployed in the past, I have sometimes been able to find the value of java.io.tmpdir by executing the following command:
ps -ef | grep java.io.tmpdir
Particularly with tomcat apps, you may get an output like this:
-Djava.io.tmpdir=/usr/local/tomcat/temp
For temp dir: there is a good answer here.
For all environment variable, try env command.
Recommended tool now is jcmd, use it like :
jcmd $pid VM.system_properties | grep java.io.tmpdir