So, I am seeing some very strange behavior with Openstack Swift. If I start all the swift processes, and say do an operation (eg. upload a file), then if I kill all my Swift processes, copy all the data directories to some other temp location, delete the content of all the data directories, copy all the temp directories back to data dir and start my swift processes again, I am not able to perform a GET for the object. (I can access the container info but swift download fails).
I have spend hours debugging this very strange behavior. Anyone else has experienced this problem ?
Steps for reproducing the bug: (Assuming 4 Data Dirs say in /mnt/sdb1/ dir)
=> swift-init all start
=> swift upload some_container some_file
=> (Can be followed by any no of swift operations)
=> swift-init all kill
=> cp -R /mnt/sdb1/1 /mnt/sdb1/1_temp (copy from original to a temp dir)
=> cp -R /mnt/sdb1/2 /mnt/sdb1/2_temp (copy from original to a temp dir)
=> cp -R /mnt/sdb1/3 /mnt/sdb1/3_temp (copy from original to a temp dir)
=> cp -R /mnt/sdb1/4 /mnt/sdb1/4_temp (copy from original to a temp dir)
=> rm -rf /mnt/sdb1/1/* (delete all the contents of original data dir)
=> rm -rf /mnt/sdb1/2/* (delete all the contents of original data dir)
=> rm -rf /mnt/sdb1/3/* (delete all the contents of original data dir)
=> rm -rf /mnt/sdb1/4/* (delete all the contents of original data dir)
=> cp -R /mnt/sdb1/1_temp /mnt/sdb1/1 (copy to original from the temp dir)
=> cp -R /mnt/sdb1/2_temp /mnt/sdb1/2 (copy to original from the temp dir)
=> cp -R /mnt/sdb1/3_temp /mnt/sdb1/3 (copy to original from the temp dir)
=> cp -R /mnt/sdb1/4_temp /mnt/sdb1/4 (copy to original from the temp dir)
=> swift-init all start (start swift again)
=> swift list (this succeeds)
=> swift list some_container (this succeeds)
=> swift download some_container some_file (this fails)
This is very strange considering the fact that I just moved the content.
Any pointers ?
Openstack Swift has some background process that check replication and consistency of its files. When you move the files by yourself, Swift will probably understand that as some kind of inconsistency depending on your cluster setup and will try to fix it. So, that is not supposed to work.
However, if you still wanting to discover what happens, I recommend you list your container content after the upload, move the files, and list you container content again. Wait a few minutes and list again. Move the files back and list again. That may help you to find out if the background process are doing something while you move the files.
Related
I am having trouble symlinking dotfiles. I have a folder in my home directory ~/dotfiles which I have synced to a github repo. I am trying to take my .vimrc file in ~/dotfiles/.vimrc and create a symbolic link to put it at ~/.vimrc. To do this I type in
ln -s ~/dotfiles/.vimrc ~/.vimrc
But when I run that it says
ln: /Users/me/.vimrc: File exists
What am I doing wrong?
That error message means that you already have a file at ~/.vimrc, which ln is refusing to overwrite. Either delete the ~/.vimrc and run ln again or let ln delete it for you by passing the -f option:
ln -s -f ~/dotfiles/.vimrc ~/.vimrc
There is a better solution for managing dotfiles without using symlinks or any other tool, just a git repo initialized with --bare.
A bare repository is special in a way that they omit working directory, so you can create your repo anywhere and set the --work-tree=$HOME then you don't need to do any work to maintain it.
Approach
first thing to do is, create a bare repo
git init --bare $HOME/.dotfiles
To use this bare repo, you need to specify --git-dir=$HOME/.dotfiles/ and --work-tree=$HOME, better is to create an alias
alias dotfiles='/usr/bin/git --git-dir=$HOME/.dotfiles/ --work-tree=$HOME
At this point, all your configuration files are being tracked, and you can easily use the newly registered dotfiles command to manage the repository, ex :-
# to check the status of the tracked and untracked files
dotfiles status
# to add a file
dotfiles commit .tmux.conf -m ".tmux.conf added"
# push new files or changes to the github
dotfiles push origin main
I also use this way to sync and store my dotfiles, see my dotfiles repository and can read at Storing dotfiles with Git where I wrote about managing for multiple devices.
How to symlink all dotfiles in a directory recursively
Have a dotfiles directory that is structured as to how they should be structured at $HOME
dotfiles_home=~/dotfiles/home # for example
cp -rsf "$dotfiles_home"/. ~
-r: Recursive, create the necessary directory for each file
-s: Create symlinks instead of copying
-f: Overwrite existing files (previously created symlinks, default .bashrc, etc)
/.: Make sure cp "copy" the contents of home instead of the home directory itself.
Tips
Just like ln, if you want no headache or drama, use an absolute path for the first argument like the example above.
Note
This only works with GNU cp (preinstalled in Ubuntu), not POSIX cp. Check your man cp, you can install GNU coreutils if needed.
Thanks
To this and this.
I know this is a basic question but I'm missing something fundamental about makefiles.
Take this simple rule/action:
doc: ${SRC_DIR}/doc/dir1/file1.pdf ${SRC_DIR}/doc/dir1/file2.pdf
cp $? ${DEST_DIR}/doc/
the first time I run it, it copies file1.pdf and file2.pdf to the destination/doc directory. Perfect. I'm expecting the next time I run it, for it to do nothing. The source files haven't changed, aren't they a dependency? But when I run I get :
cp : cannot create regular file ..... :Permission denied.
so, 2 questions:
1) Why is it trying to do it again? When I run make -d I see it eventually says: No need to remake target .../file1.pdf and .../file2.pdf but then
it says : must remake target 'doc'
If it doesn't need to make either pdf file, why does it need to make doc?
2) say the pdf files had changed in the source, they are read only though, so it gets the permission denied error. How do you get around this?
A make rule:
target: preqreq0 prereq1...
command
...
says that target needs to be (re)made if it does not exist or is older than
any of the prerequisites preqreq0 prereq1..., and that target shall be
(re)made by running the recipe command ....
Your rule:
doc: ${SRC_DIR}/doc/dir1/file1.pdf ${SRC_DIR}/doc/dir1/file2.pdf
cp $? ${DEST_DIR}/doc/
never creates a file or directory doc, so doc will never exist when
the rule is evaluated (unless you create doc by other means), so the recipe
will always be run.
The kind of target that I believe you want doc to be is a phony target,
but you are going about it wrongly. A reasonable makefile for the purpose would
be:
SRC_DIR := .
DEST_DIR := .
PDFS := file1.pdf file2.pdf
PDF_TARGS := $(patsubst %,$(DEST_DIR)/doc/%,$(PDFS))
.PHONY: doc clean
doc: $(PDF_TARGS)
$(DEST_DIR)/doc/%.pdf: $(SRC_DIR)/doc/dir1/%.pdf
cp $< $#
clean:
rm -f $(PDF_TARGS)
I recommend The GNU Make documentation
As for your second problem, how to overwrite "readonly" files, it is unrelated to make.
You cannot overwrite files to which you do not have write permission, regardless
of the means by which you try to do it. You must get write permission to any files
that you need to write to. It is a system administration matter. If you do not
understand file permissions you may find help at sister-site Unix & Linux
or serverfault
I am on Mac OS.
I have a directory with round about 3000 files and several subdirectories (wordpress installation)
Now I have to find all the files in a similar directory (have to separate master and child installation) that are additional files and have to copy them away into another directory.
I use this command:
$ diff -rq dt-the7 dt-the7-master-from-Yana|grep 'Only in dt-the7'|awk {' print $3 $4 '}|sed 's/:/\//g'|xargs -J {} rsync -av {} neu/
but somehow a certain file 3d.png and a list of other that should be in a subdir of the destination dir are copied into the root dir of the destination.
Any idea why that might be?
It makes no difference whether I use cp, rsync or ditto
You need the -R relative option on your rsync command.
Without this rsync just copies the item referenced rather than the path referenced, so items at the root level are copied as you expected but items in sub-directories are also copied to the root, which is not what you wanted.
With the option rsync takes account of the relative path and recreates it at the destination.
An example with another command might help, consider:
cp A/B.txt C/
that will copy B.txt into C, it does not create a folder A in C which in turn contains the file B.txt. rsync without -R behaves like that cp command, with -R it creates the A directory in C.
HTH
When I use the command:
meteor create myfolder
It won't by default allow meteor to install itself if the folder is already existing. I can't find an option to force it. Is it really necessary that Meteor creates the directory by itself first or is this just because of 'good practice'?
I want to automate the creation of a folder first and run the meteor command afterwards, hence the question.
Maybe you want a shell script like the following:
$DIR = <some variable>
if [ ! -f "$DIR" ]; then
meteor create tempDir
mv tempDir/* "$DIR/"
rmdir tempDir
fi
which will copy the contents into your new directory, as long as it doesn't contain any Meteor files already.
I want to copy a complete directory content from /home/private_html/userx/ into the /home/private_html/usery/, the problem is that the directory userx contains few symlinks, and when using the cp it just skip them (skip occurs, if symlinks directs into a file, in case if it points into the directory, it just copy WHOLE directory instead...).
The command I was using looks following:
# cp -iprv /home/private_html/userx/ /home/private_html/usery/
Has anyone a solution to copy the directory "just as it is" into other place?
On FreeBSD, cp doesn't have an -r option. It does have -R, which should do what you want:
-R If source_file designates a directory, cp copies the directory and
the entire subtree connected at that point. If the source_file
ends in a /, the contents of the directory are copied rather than
the directory itself. This option also causes symbolic links to be
copied, rather than indirected through, and for cp to create spe‐
cial files rather than copying them as normal files. Created
directories have the same mode as the corresponding source direc‐
tory, unmodified by the process' umask.
Roland is right about the -R flag. You could also use a pair of tar-processes, which would make your command a little bit more system-independent:
tar -C /home/private_html/userx/ -cpf - . | tar -C /home/private_html/usery/ -epf -