We're encountering some errors when starting Nifi 1.16.3 after calling encrypt-config.sh in a secure environment.
For background, Nifi is deployed and managed using an in-house deployment product, and is used as a component within our application stack. In general everything works fine when using Nifi 1.15.1, however, we are encountering issues when using Nifi 1.16.3.
In short, when deploying an instance of Nifi in our system, we first create all settings necessary, including a nifi.properties and bootstrap.conf files.
At this first stage, we set an initial password of nififtw! to sensitive.props.key, which at this moment in time is plain text.
We execute encrypt-config command:
JAVA_HOME=/tech/java/openjdk1.8.0_322
/bin/encrypt-config.sh --key AAAF0313BCC242CC9EC1DD30AD74FB2E --niFiProperties /servers/nifi/default/conf/nifi.properties --loginIdentityProviders /servers/nifi/default/conf/login-identity-providers.xml --bootstrapConf /servers/nifi/default/conf/bootstrap.conf
At this stage, we don't parse in the flow.xml as it doesn't exist yet, however, the protected values in nifi.properties, i.e sensitive.props.key, are encrypted based on the --key.
Nifi starts as expected.
In a later moment, when updating our system, we invoke encrypt-config.sh one more time, but now we attempt to change the propsKey to the same value of they hexkey used. Note that at this stage, we have a flow.xml and we parse in the path as an argument:
JAVA_HOME=/tech/java/openjdk1.8.0_322
/bin/encrypt-config.sh --key AAAF0313BCC242CC9EC1DD30AD74FB2E --niFiProperties /servers/nifi/default/conf/nifi.properties --loginIdentityProviders /servers/nifi/default/conf/login-identity-providers.xml --flowXml /databases/nifi/default/flow.xml.gz --propsKey AAAF0313BCC242CC9EC1DD30AD74FB2E --bootstrapConf ey/servers/nifi/default/conf/bootstrap.conf
No errors are encountered at this stage and the value of sensitive.props.key in nifi.properties is changed accordingly. However, when attempting to start nifi server, we encounter the error bellow:
nifi-bootstrap.log
2022-11-13 06:32:06,488 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 82873
2022-11-13 06:32:14,269 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 32908
2022-11-13 06:34:06,217 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Decryption Failed with Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]
2022-11-13 06:34:06,218 ERROR [NiFi logging handler] org.apache.nifi.StdErr Shutting down...
2022-11-13 06:34:06,945 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi
nifi-app.log
2022-11-13 06:34:06,209 INFO [main] org.eclipse.jetty.server.Server Started #119858ms
2022-11-13 06:34:06,209 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]
at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:78)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.decrypt(StandardFlowComparator.java:281)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareProperties$3(StandardFlowComparator.java:291)
at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compareProperties(StandardFlowComparator.java:289)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:267)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareComponents$1(StandardFlowComparator.java:114)
at java.util.HashMap.forEach(HashMap.java:1290)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:467)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compare$5(StandardFlowComparator.java:472)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareComponents$1(StandardFlowComparator.java:114)
at java.util.HashMap.forEach(HashMap.java:1290)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compareComponents(StandardFlowComparator.java:112)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:472)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:94)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:79)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.compareFlows(VersionedFlowSynchronizer.java:387)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:167)
at org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43)
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1524)
at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:107)
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:819)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:542)
at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1008)
at org.apache.nifi.NiFi.<init>(NiFi.java:170)
at org.apache.nifi.NiFi.<init>(NiFi.java:82)
at org.apache.nifi.NiFi.main(NiFi.java:330)
Caused by: javax.crypto.BadPaddingException: pad block corrupted
at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$BufferedGenericBlockCipher.doFinal(Unknown Source)
at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(Cipher.java:2168)
at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:74)
... 62 common frames omitted
As shown in the errors above, the decryption fails when attempting to start the nifi server.
We suspect that when we try to encrypt the flow.xml with a new sensitive.props.key, the re-encryption doesn't happen, even though the value of propsKey in nifi.properties has been replaced and encrypted accordingly to the new password we parse.
After we get the failure described, if we manually change the propsKey in nifi.properties to what it was before (the encrypted version of 'nififtw!'), we manage to start nifi successfully.
Note that the password nififtw! is just one example. The issue happens with any password we might use for the first call of encrypt-config, even if we call encrypt-config again specifying a new propsKey.
We've attempted to modify the arguments used when executing the encrypt-config.sh, i.e. by creating the modified flow.xml and nifi.properties in a /tmp/ directory and manually replacing the older versions, which hasn't worked either.
This issue only arrises in Nifi version 1.16.3.0. We were using version 1.15.1.0 and version 1.11.3.0 and the same logic works perfectly fine.
This is how my nifi.properties looks like:
# security properties #
nifi.sensitive.props.key=<encrypted-key>
nifi.sensitive.props.key.protected=aes/gcm/128
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.additional.keys=
Any idea of what might be causing this issue?
The error that you encountered in nifi 1.16 is appearing because of some changes introduced in nifi starting from 1.16 version.
Starting from nifi 1.16 , NIFI start writing on flow.xml.gz as well as JSON format ( prior to 1.16 this JSON was not there ) flow.json.gz , see the link https://www.mail-archive.com/users#nifi.apache.org/msg15332.html
When 1.16 nifi start it creates both the file, writes onto it and then encrypt it using props key from nifi.properties , when your code run encrypt-config.sh tool it only changes the flow.xml.gz with new props key and JSON flow file still encrypted with old props key,
While next time nifi going to start it try to read props key from nifi.properties , which is the new ( changed props key ) and it successfully able to decrypt the flow.xml.gz but it can not able to decrypt flow.json.gz because JSON file is still encrypted with old props key which is overwritten by new props key on nifi.properties
Solution : from 1.16 onwards "./bin/nifi.sh set-sensitive-properties-key NewSensitivePropertiesKey" tool improved to cover both xml as well as json version of flow file while changing the props key, see JIRA ticket https://issues.apache.org/jira/browse/NIFI-9711 , So while changing the props key you can try using nifi.sh in place of encrypt-config.sh
Regards
Vikas
Related
I am trying to run jupyterhub on an Ubuntu 20.04 LTS server. My idea is to run python/jupyterhub in a conda virtual environment as a system service. As I want to be able to limit the resources available to individual users I installed the systemdspawner.
After installing everything and starting the jupyterhub service I can login through my web browser. However, when trying to start the server the spawner stucks and after a while I get an error message saying "Spawn failed: Timeout"
in journalctl I can see the following messages:
User logged in: me 302 POST /hub/login?next= -> /hub/spawn (me#::ffff:[my IP address]) 59.42ms
Adding role server to token: <APIToken('93c8...', user='me', client_id='jupyterhub')
Creating oauth client jupyterhub-user-me
pam_loginuid(login:session): Error writing /proc/self/loginuid: Operation not permitted
pam_loginuid(login:session): set_loginuid failed
pam_unix(login:session): session opened for user me by (uid=0)
Failed to open PAM session for me: [PAM Error 14] Cannot make/remove an entry for the specified session
Disabling PAM sessions from now on. user:me
Unit jupyter-me-singleuser in a failed state. Resetting state.
Disclaimer: My Jupyter/Python installation is replacing an former installation that was setup by someone else and got messed up a bit during time. I tried to remove everything related and start with a clean installation from scratch. However, as I had very little documentation about the old setup there is a certain risk that there might be some left-overs of the previous installation that may cause trouble.
Any ideas?
Solved it out myself. In the end the PAM related messages seem to be non-critical and were not related to the timeout at all. Instead I found a mistake in /etc/systemd/system/jupyterhub.service, where the PATH variable was not including the bin directory of my miniconda installation.
I'm trying to set the GLOBAL time of my MariaDB database to UTC. I've followed the recommendations of their official documentation and default_time_zone="+00:00" in the my.cnf file, however it does NOT work and I get the following error when I start it in the shell: mysql: unknown variable 'default_time_zone=+00:00'.
Does anyone have an idea?
Thanks
Remove the quotes
As the error message states command line client (mysql) complains about unknown variable, since default_time_zone is a server but not a client variable. So you added it in wrong section. Move the entry to the server section:
[server]
default_time_zone=+00:00
I am trying to run the example grakn migration "phone_calls" (using python and JSON files).
Before reaching there, I need to load the schema, but I am having trouble with getting the schema loaded, as shown here: https://dev.grakn.ai/docs/examples/phone-calls-schema
System:
-Mac OS 10.15
-grakn-core 1.8.3
-python 3.7.3
The grakn server is started. I checked and the 48555 TCP port is open, so I don't think there is any firewall issue. The schema file is in the same folder (phone_calls) as where the json data files is, for the next step. I am using a virtual environment. The error is below:
(project1_env) (base) tiffanytoor1#MacBook-Pro-2 onco % grakn server start
Storage is already running
Grakn Core Server is already running
(project1_env) (base) tiffanytoor1#MacBook-Pro-2 onco % grakn console --keyspace phone_calls --file phone_calls/schema.gql
Unable to create connection to Grakn instance at localhost:48555
Cause: io.grpc.StatusRuntimeException
UNKNOWN: Could not reach any contact point, make sure you've provided valid addresses (showing first 1, use getErrors() for more: Node(endPoint=/127.0.0.1:9042, hostId=null, hashCode=5f59fd46): com.datastax.oss.driver.api.core.connection.ConnectionInitException: [JanusGraph Session|control|connecting...] init query OPTIONS: error writing ). Please check server logs for the stack trace.
I would appreciate any help! Thanks!
Nevermind -- I found the solution, in case any one else runs into a similar problem. The server configuration file needs to be edited: point the data directory to your project data files (here: the phone_calls data files) & change the server IP address to your own.
I am trying to use the migration tool utility from within PingFederate but I keep getting the following error:
List adapters... Downloading adapter index from source... ERROR:
Unable to download from source.
java.security.NoSuchAlgorithmException: E rror constructing
implementation (algorithm: Default, provider: SunJSSE, class:
sun.security.ssl.SSLContextImpl$DefaultSSLContext) Done.
From the configcopy.log:
Caused by: java.net.SocketException: java.security.NoSuchAlgorithmException
Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation
Caused by: java.io.IOException: Invalid keystore format
Windows 7 Professional SP1
java version "1.8.0_144" Java(TM) SE Runtime Environment (build
1.8.0_144-b01) Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
PingFederate: 8.4.2
I am attempting this because we want to automate a deployment process which has currently been manual. I am only trying to use the listadapters.conf template and have set the source.conf to output to a file. The command I am entering is:
configcopy.bat -Dconfigcopy.conf.file=configcopy_templates\\source.conf;configcopy_templates\
\listadapters.conf
and I am running this from the <PF_HOME>/bin directory. The contents of the two files I mentioned are:
source.conf
source.connection.management.service.url =
<my local install url on port 9999>/pf-mgmt-ws/ws/ConnectionMigrationMgr
source.connection.management.service.user = Administrator
source.connection.management.service.password =
OBF:JWE:eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2Iiwia2lkIjoibGJhaGtDZlNiSiIsInZlcnNpb24iOiI4LjQuMi4wIn0..ryNLCcpzwEx6KGzXi1FboA.34NbypXUud45R77TLwMvjg.dQFNb9NpbDY_EWIePb9hMA
configcopy.connection.trust.keystore = C:\Program Files\Ping
Identity\pingfederate-8.4.2\pingfederate\server\default\data\pf.jwk
output.file = c:\temp\pf-config.txt
The Administrator is the default one from install with all three roles added to it and the password was obfuscated using obfuscate.bat in the bin directory.
listadapters.conf
cmd=listadapters
debug=true
select.adapter.role = idp
Even though it doesn't look like it above all backslashes are escaped that just hasn't come through here.
I have tried:
removing the path to the keystore altogether
ERROR: Unable to download from source.
sun.security.validator.ValidatorException : PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target Done.
Setting the path to cacerts in jre/lib (same error as above)
I have installed the data.zip from the DotNet-Integration-Kit-2-5-2.zip and that is the only set up on this PC. (my dev box)
The integration kit puts two certificates (maybe the same one twice, not sure) that can be viewed through
Server Configuration > Trusted CAs
Server Configuration > SSL Server Certificates
And I have also added one into
Server Configuration > SSL Client Keys & Certificates
The kits certs show as RSA1024 and the one I created shows as RSA2048.
Questions:
Why does the error state algorithm:default (key store format?)
Is there a setting I am missing that would change it from default
Does anyone know of any docs other than the admin manual (almost know
it by heart now)
Why is pf.jwk the wrong format
Any other ideas at all please.
[update] Dam, I have been trying to use the migration utility but as I am on a version over 7.2 I should be using the administrative API. Back to the drawing board. Still looking for advice though!
The pf.jwk file is an encrypted Java web key. The truststore is a standard jks file that you add PingFed's SSL key to, or its signing CA's public key.
However, as you have found, you should use the admin API. Configcopy is no longer being developed.
I have a corrupted Lucene index. If I run "CheckIndex -fix" the problem is resolved, but as soon as I restart tomcat it becomes corrupted again.
The index directory is shared between two application servers running Liferay-Tomcat. I am fixing the index on 1 server and restarting that whilst the other is running. This is a production environment so I cannot bring them both down.
Any suggestions please?
Before fix, CheckIndex says:
Opening index # /usr/local/tomcat/liferay/lucene/0
Segments file=segments_5yk numSegments=1 version=FORMAT_SINGLE_NORM_FILE [Lucene 2.2]
1 of 1: name=_2vg docCount=31
compound=false
hasProx=true
numFiles=8
size (MB)=0.016
no deletions
test: open reader.........FAILED
WARNING: fixIndex() would remove reference to this segment; full exception:
java.io.IOException: read past EOF
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:151)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:78)
at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:335)
at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:119)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:652)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:605)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:491)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:903)
WARNING: 1 broken segments (containing 31 documents) detected
WARNING: would write new segments file, and 31 documents would be lost, if -fix were specified
If you access your search index with more than one application server, I would suggest integrating a Solr Server. So you don't have the problem that 2 app servers are trying to write on the same file. This could be error prone as you already found out.
To get Solr up and running you have to follow those steps:
Install a Solr Server on any machine you like. A machine running only Solr would be quite preferable.
Install the Solr search portlet in Liferay
Adjust the config files according to the setup document of Sol Search portlet.
Here are some additional links:
http://www.liferay.com/de/marketplace/-/mp/application/15193648
http://www.liferay.com/de/community/wiki/-/wiki/Main/Pluggable+Enterprise+Search+with+Solr