I get below error when tried to run the following breeze build command to build airflow docker. I clone the git airflow master branch to build this image.
Build Command:
./breeze build-image --production-image --python 3.7 --install-airflow-version 2.0.0 --additional-extras=jdbc --additional-python-deps="pandas" --additional-runtime-apt-deps="default-jre-headless"
Error:
Step 83/94 : COPY scripts/docker/install*.sh /scripts/docker/
---> 8363694670bb
Step 84/94 : RUN if [[ ${INSTALL_FROM_PYPI} == "true" ]]; then bash /scripts/docker/install_airflow.sh; fi; if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || true ; find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r || true
---> Running in 01f3dd4b7f57
+ [[ true == \t\r\u\e ]]
+ bash /scripts/docker/install_airflow.sh
Installing all packages with constraints and upgrade if needed
ERROR: Invalid requirement: 'apache-airflow[async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv,jdbc]2.0.0'
WARNING: You are using pip version 20.2.4; however, version 21.0.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
The command '/bin/bash -o pipefail -e -u -x -c if [[ ${INSTALL_FROM_PYPI} == "true" ]]; then bash /scripts/docker/install_airflow.sh; fi; if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || true ; find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r || true' returned a non-zero code: 1
ERROR: The previous step completed with error. Please take a look at output above
Edit (1):
I tried the same thing on an aws ec2 instance and get the same error. Something looks like is broken on the airflow side. Below is the screenshot.
Replace
--install-airflow-version 2.0.0
with
--install-airflow-version="2.0.0"
So the command is:
./breeze build-image --production-image --python 3.7 --install-airflow-version "2.0.0" --additional-extras=jdbc --additional-python-deps="pandas" --additional-runtime-apt-deps="default-jre-headless"
This worked for me:
./breeze build-image --production-image --python 3.7 --install-airflow-version "==2.0.0" --additional-extras=jdbc --additional-python-deps="pandas" --additional-runtime-apt-deps="default-jre-headless"
had to add the "==".
Related
Pls answer me and hope my question not so stupid but i first time put inside rpm bash skrip. May someone can sent link or tip what i must coorrect in my .spec
This rpm pakage must make just two thing - copy in linux folder /bin/ one file(inet.dbg) and next when cp make must start bash skrip for check library for this file.
This my bash skript
#!/bin/bash
ldd inet.dbg > t.t
ERR=`grep -i "not" t.t | wc -l`
if [[ $ERR -gt 0 ]]; then
grep -i "not" t.t > erg.e
echo "Die folgenden Bibliotheken wurden nicht gefunden:"
cat erg.e
rm t.t erg.e
else
ls -a ss.tar &> t.t
if [[ $? -eq 0 ]]; then
echo alles ok
else
echo no
fi
fi '
This .spec
# %_topdir and %_tmppath are defined in ~/.rpmmacros
%define name inetdbg
%define version 0.1
%define release 1
%define buildroot %{_tmppath}/%{name}-%{version}-%{release}
%define tarfile %{tarversion}.tar
%define installscript /home/adis/rpmbuild/SOURCES/skript.sh
Name: %{name}
Version: %{version}
Release: %{release}
BuildArch: noarch
Summary: adisc
License: -
Source1: %{installscript}
Source2: /home/adis/rpmbuild/SOURCES/inetdbg.tar.xz
BuildRoot: %{_builddir}/%{name}-root
%description
Tested RPM
%prep
%build
%install
mkdir -p %{buildroot}%{_bindir}
install -D -pm 755 %{SOURCE2} %{buildroot}/bin/inetdbg.tar.xz
cp -a %{installscript} %{buildroot}%{_bindir}/
chmod a+x %{buildroot}%{_bindir}/%{installscript}
%files
%{_bindir}/%{installscript}
%{SOURCE2}
%post
%{_bindir}%{installscript}
%clean
[ ${RPM_BUILD_ROOT} != "/" ] && rm -rf ${RPM_BUILD_ROOT}
%changelog
* 25 Jan 2023 <user>
- Add script.sh
- Add definit
I'm trying to deploy OpenStack using DevStack on Ubuntu 22.04, I'am facing issues on running stack.sh.
I'm not sure if it's related to the git repo code or related to the system configuration or som missing packages, please advise.
Here below the error log:
fatal: your current branch 'master' does not have any commits yet
+functions-common:git_clone:680 cd /opt/stack/devstack
+lib/neutron-legacy:install_mutnauq:469 setup_develop /opt/stack/neutron
+inc/python:setup_develop:334 local bindep
+inc/python:setup_develop:335 [[ /opt/stack/neutron == -bindep* ]]
+inc/python:setup_develop:339 local project_dir=/opt/stack/neutron
+inc/python:setup_develop:340 local extras=
+inc/python:setup_develop:341 _setup_package_with_constraints_edit /opt/stack/neutron -e
+inc/python:_setup_package_with_constraints_edit:360 local bindep
+inc/python:_setup_package_with_constraints_edit:361 [[ /opt/stack/neutron == -bindep* ]]
+inc/python:_setup_package_with_constraints_edit:365 local project_dir=/opt/stack/neutron
+inc/python:_setup_package_with_constraints_edit:366 local flags=-e
+inc/python:_setup_package_with_constraints_edit:367 local extras=
++inc/python:_setup_package_with_constraints_edit:374 cd /opt/stack/neutron
++inc/python:_setup_package_with_constraints_edit:374 pwd
+inc/python:_setup_package_with_constraints_edit:374 project_dir=/opt/stack/neutron
+inc/python:_setup_package_with_constraints_edit:376 '[' -n /opt/stack/requirements ']'
+inc/python:_setup_package_with_constraints_edit:380 local name
++inc/python:_setup_package_with_constraints_edit:381 awk '/^name.*=/ {print $3}' /opt/stack/neutron/setup.cfg
awk: fatal: cannot open file `/opt/stack/neutron/setup.cfg' for reading: No such file or directory
+inc/python:_setup_package_with_constraints_edit:381 name=
+inc/python:_setup_package_with_constraints_edit:1 exit_trap
+./stack.sh:exit_trap:516 local r=2
++./stack.sh:exit_trap:517 jobs -p
+./stack.sh:exit_trap:517 jobs=
+./stack.sh:exit_trap:520 [[ -n '' ]]
+./stack.sh:exit_trap:526 '[' -f '' ']'
+./stack.sh:exit_trap:531 kill_spinner
+./stack.sh:kill_spinner:426 '[' '!' -z '' ']'
+./stack.sh:exit_trap:533 [[ 2 -ne 0 ]]
+./stack.sh:exit_trap:534 echo 'Error on exit'
Error on exit
+./stack.sh:exit_trap:536 type -p generate-subunit
+./stack.sh:exit_trap:537 generate-subunit 1656624514 146 fail
+./stack.sh:exit_trap:539 [[ -z /opt/stack/logs ]]
+./stack.sh:exit_trap:542 /usr/bin/python3.10 /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs
/opt/stack/devstack/tools/worlddump.py:22: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import spawn
Any clue ?
I wrote a script in R that has several arguments. I want to iterate over 20 directories and execute my script on each while passing in a substring from the file path as my -n argument using sed. I ran the following:
find . -name 'xray_data' -exec sh -c 'Rscript /Users/Caitlin/Desktop/DeMMO_Pubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f {} -b "{}/SEM_images" -c "{}/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "`sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/' "{}"`"' sh {} \;
which results in this error:
ubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f {} -b "{}/SEM_images" -c "{}/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "`sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/' "{}"`"' sh {} \;
sh: command substitution: line 0: syntax error near unexpected token `('
sh: command substitution: line 0: `sed -e s/.*DeMMO.*[/](.*)_.*[/]xray_data/1/ "./DeMMO1/D1T3rep_Dec2019_Ellison/xray_data"'
When I try to use sed with my pattern on an example file path, it works:
echo "./DeMMO1/D1T1exp_Dec2019_Poorman/xray_data" | sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/'
which produces the correct substring:
D1T1exp_Dec2019
I think there's an issue with trying to use single quotes inside the interpreted string but I don't know how to deal with this. I have tried replacing the single quotes around the sed pattern with double quotes as well as removing the single quotes, both result in this error:
sed: RE error: illegal byte sequence
How should I extract the substring from the file path dynamically in this case?
To loop through the output of find.
while IFS= read -ru "$fd" -d '' files; do
echo "$files" ##: do whatever you want to do with the files here.
done {fd}< <(find . -type f -name 'xray_data' -print0)
No embedded commands in quotes.
It uses a random fd just in case something inside the loop is eating/slurping stdin
Also -print0 delimits the files with null bytes, so it should be safe enough to handle spaces tabs and newlines on the path and file names.
A good start is always put an echo in front of every commands you want to do with the files, so you have an idea what's going to be executed/happen just in case...
This is the solution that ultimately worked for me due to issues with quotes in sed:
for dir in `find . -name 'xray_data'`;
do sampleID="`basename $(dirname $dir) | cut -f1 -d'_'`";
Rscript /Users/Caitlin/Desktop/DeMMO_Pubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f "$dir" -b "$dir/SEM_images" -c "$dir/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "$sampleID";
done
I am attempting to come up with a method to remotely find a list of files on our AIX UNIX machine that meet, what seems in windows, like simple criteria. It needs to be case insensitive (sigh), use wildcards (*) and possibly contain spaces in the path.
For my tests below I was using the ksh shell. However it will need to work in an ssh shell as well.
I am attempting to implement secure FTP in Visual Basic 6 (I know) using plink, command line and a batch file.
Basically find a file like the one below but with case insensitivity:
ls -1 -d -p "/test/rick/01012017fosterYYY - Copy.txt" | grep -v '.*/$'
Thanks for any help.
ls -1 -d -p /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy] - [Cc][Oo][Pp][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'**
fails with:
ls: 0653-341 The file /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy] do
es not exist.
ls: 0653-341 The file - does not exist.
ls: 0653-341 The file [Cc][Oo][Pp][Yy].[Tt][Xx][Tt] does not exist.
ls -1 -d -p /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'**
success - as long as there are no spaces.
ls -1 -d -p "/test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy].[Tt][Xx][Tt]" | grep -v '.*\/$'**
fails with:
ls: 0653-341 The file /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy].[T
t][Xx][Tt] does not exist.
-- Assumption: We cannot use quotes with wildcard characters
ls -1 -d -p "/test/rick/01012017fosterYYY - Copy.txt" | grep -v '.*\/$'**
success. not case insensitive.
ls -1 -d -p /test/rick/[0][1][0][1][2][0][1][7][Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy] - [Cc][Oo][Pp][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'**
fails with:
ls: 0653-341 The file /test/rick/[0][1][0][1][2][0][1][7][Ff][Oo][Ss][Tt][Ee][Rr
][Yy][Yy][Yy][ does not exist.
ls: 0653-341 The file ][-][ does not exist.
ls: 0653-341 The file ][Cc][Oo][Pp][Yy].[Tt][Xx][Tt] does not exist.
ls -1 -d -p /test/rick/[0][1][0][1][2][0][1][7][Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy][ ][-][ ][Cc][Oo][Pp][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'**
fails with:
ls: 0653-341 The file /test/rick/[0][1][0][1][2][0][1][7][Ff][Oo][Ss][Tt][Ee][Rr
][Yy][Yy][Yy][ does not exist.
ls: 0653-341 The file ][-][ does not exist.
ls: 0653-341 The file ][Cc][Oo][Pp][Yy].[Tt][Xx][Tt] does not exist.
ls -1 -d -p /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy]?-?[Cc][Oo][Pp][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'**
success. not very helpful though.
ls -1 -d -p /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy][ ]-[ ][Cc][Oo][Pp][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'**
fails with:
ls: 0653-341 The file /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy][ d
oes not exist.
ls: 0653-341 The file ]-[ does not exist.
ls: 0653-341 The file ][Cc][Oo][Pp][Yy].[Tt][Xx][Tt] does not exist.
ls -1 -d -p /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy]{ }-{ }[Cc][Oo][Pp][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'**
fails with:
ls: 0653-341 The file /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy]{ d
oes not exist.
ls: 0653-341 The file }-{ does not exist.
ls: 0653-341 The file }[Cc][Oo][Pp][Yy].[Tt][Xx][Tt] does not exist.
ls -1 -d -p /test/rick/*01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy] - [Cc][Oo][Pp][Yy].[Tt][Xx][Tt]* | grep -v '.*\/$'**
fails with
ls: 0653-341 The file /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy] d
oes not exist.
ls: 0653-341 The file - does not exist.
ls: 0653-341 The file [Cc][Oo][Pp][Yy].[Tt][Xx][Tt] does not exist.
ls -1 -d -p "/test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy] - [Cc][Oo][Pp][Yy].[Tt][Xx][Tt]" | grep -v '.*\/$'**
fails with:
ls: 0653-341 The file /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy] -
[Cc][Oo][Pp][Yy].[Tt][Xx][Tt] does not exist.
ls doesn't do pattern matching, any wildcard expansion (globbing) is done by the shell. The glob pattern language is different from regular expressions. Read the ksh documentation for information about globbing ("File Name Generation" in the manpage).
So when you do:
$ touch foo flo fum
$ ls -1 f[ol]o
flo
foo
... the shell notices the globbing characters [], reads the directory contents, replaces it with the matching files, and passes those as parameters to ls. You can show this by using echo instead:
$ echo f[ol]o
flo foo
ksh has globbing options available with the ~() construct, option i is "Treat the match as case insensitive" :
ksh$ touch foo FoO FOO
ksh$ echo ~(i)foo
foo FoO FOO
bash has a nocaseglob shopt option:
bash$ shopt -s nocaseglob
bash$ touch fOo
bash$ echo FO*
foo
Although note that some globbing character needs to be present to make the magic happen:
bash$ echo FOO
FOO
bash$ echo [F]OO
foo
(to keep this option change local, see https://unix.stackexchange.com/questions/310957/how-to-undo-a-set-x/310963)
It looks as if you're using grep -v '.*/$' to remove lines that are directories. The .* is superfluous here -- grep -v '/$' is equivalent.
But find is a better tool for this kind of searching and filtering, implementing -type f (match regular files) by actually looking at the file attributes, rather than by parsing a bit of ASCII in a listing.
$ touch foo FOO FoO
$ mkdir fOo
$ find . -maxdepth 1 -type f -iname "foo"
./FOO
./foo
./FoO
You could use find's -iname option to allow for case-insensitive searching, so for the example you've provided any of the following should find your file:
find /test/rick -maxdepth 1 -iname '01012017fosterYYY - copy.txt'
# or
find /test/rick -maxdepth 1 -iname '01012017fosteryyy - copy.txt'
# or
find /test/rick -maxdepth 1 -iname '01012017FOSTERyyy - cOpY.txt'
-maxdepth 1 : don't search in sub-directories
-iname : allow for case-insensitive searching
For case insensitive wildcard searches when -maxdepth and -iname flags are not available for AIX Find , you can pass the Find results to Grep:
find /test/rick/. \( ! -name . -prune \) -type f -print | grep -i ".*foster.*\.txt"
find [InThisFolder] [ExcludeSubfolders] [FileTypes] | grep [InsensitiveWildcardName]
Though, this can still be problematic if you have a folder structure like "/test/rick/rick/".
The following code gives results with the current directory signifier ".":
find /test/rick/. \( ! -name . -prune \) -type f -print | grep -i ".*foster.*\.txt"
But you can pass the results to sed and find "/./" and replace with "/".
find /test/rick/. \( ! -name . -prune \) -type f -print | grep -i ".*foster.*\.txt" | sed 's/\/\.\//\//g'
* UPDATE *
Based on this page: http://mywiki.wooledge.org/ParsingLs
I’ve come up with the following command (for loop on file expansion or globbing) which avoids the problematic "/test/rick/rick/" folder structure from the find | grep solution above. It searches a folder from any folder, handles spaces, and handles case insensitivity without having to specify escape characters or upper/lower matching ([Aa]).
Just modify the searchfolder and searchpattern:
searchfolder="/test/rick"; searchpattern="*foster*.txt"; for file in "$searchfolder"/*.*; do [[ -e "$file" ]] || continue; if [[ "$(basename "$file" | tr '[:upper:]' '[:lower:]')" = $searchpattern ]]; then echo "$file"; fi; done
It does this:
Set the folder path to search (searchfolder="/test/rick";)
Set the search pattern (searchpattern="*foster*.txt")
Loop for every file on the search folder (for file in "$searchfolder"/*.*;)
Make sure the file exists ( [[ -e "$file" ]] || continue;)
Transform any base file name uppercase characters to lowercase (basename "$file" | tr '[:upper:]' '[:lower:]')
Test if the lowered base file name matches the search pattern and if so
then print the full path and filename (if [[ $(basename "$file" | tr
'[:upper:]' '[:lower:]') = $searchpattern ]]; then echo "$file"; fi;)
Tested on AIX (Version 6.1.0.0) in ksh (Version M-11/16/88f) and ksh93 (Version M-12/28/93e).
What I finally used (because I don't have access to -maxdepth or -iname) was just to use case insensitive wildcards together with quotes around spaces.
ls -1 -d -p /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy]' '-' '[Cc][Oo][Pp][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'
That way I don't have to install or upgrade anything and probably cause more problems just so I can get a simple list of files.
NOTE: AIX UNIX will still throw in some garbage errors if you have any sub directories under the path. I tapped out on this and just parsed these useless messages out on the client side.
Thanks everyone who responded.
I usually use the following pipeline to grep for a particular search string and yet ignore certain other patterns:
grep -Ri 64 src/install/ | grep -v \.svn | grep -v "file"| grep -v "2\.5" | grep -v "2\.6"
Can this be achieved in a succinct manner? I am using GNU grep 2.5.3.
Just pipe your unfiltered output into a single instance of grep and use an extended regexp to declare what you want to ignore:
grep -Ri 64 src/install/ | grep -v -E '(\.svn|file|2\.5|2\.6)'
Edit: To search multiple files maybe try
find ./src/install -type f -print |\
grep -v -E '(\.svn|file|2\.5|2\.6)' | xargs grep -i 64
Edit: Ooh. I forgot to add the simple trick to stop a cringeable use of multiple grep instances, namely
ps -ef | grep something | grep -v grep
Replacing that with
ps -ef | grep "[s]omething"
removes the need of the second grep.
Use the -e option to specify multiple patterns:
grep -Ri 64 src/install/ | grep -v -e '\.svn' -e file -e '2\.5' -e '2\.6'
You might also be interested in the -F flag, which indicates that patterns are fixed strings instead of regular expressions. Now you don't have to escape the dot:
grep -Ri 64 src/install/ | grep -vF -e .svn -e file -e 2.5 -e 2.6
I noticed you were grepping out ".svn". You probably want to skip any directories named ".svn" in your initial recursive grep. If I were you, I would do this instead:
grep -Ri 64 src/install/ --exclude-dir .svn | grep -vF -e file -e 2.5 -e 2.6
you can use awk instead of grep
awk '/64/&&!/(\.svn|file|2\.[56])/' file
You maybe want to use ack-grep which allow to exclude with perl regexp as well and avoid all the VC directories, great for grepping source code.
The following script will remove all files except a list of files:
echo cleanup_all $#
if [[ $# -eq 0 ]]; then
FILES=`find . -type f`
else
EXCLUDE_FILES_EXP="("
for EXCLUDED_FILE in $#
do
EXCLUDE_FILES_EXP="$EXCLUDE_FILES_EXP./$EXCLUDED_FILE|"
done
# strip last char
EXCLUDE_FILES_EXP="${EXCLUDE_FILES_EXP%?}"
EXCLUDE_FILES_EXP="$EXCLUDE_FILES_EXP)"
echo exluded files expression : $EXCLUDE_FILES_EXP
FILES=`find . -type f | egrep -v $EXCLUDE_FILES_EXP`
fi
echo removing $FILES
for FILE in $FILES
do
echo "cleanup: removing file $FILE"
rm $FILE
done