I'm getting this error when I am trying to reinstall elk with wazuh
We need more precise information about your use case in order to help you to troubleshoot this problem.
I recommend you follow the uninstallation guide from the official documentation https://documentation.wazuh.com/current/user-manual/uninstall/elastic-stack.html, and after that, install it again https://documentation.wazuh.com/current/installation-guide/more-installation-alternatives/elastic-stack/all-in-one-deployment/unattended-installation.html.
If you want to preserve your configuration make sure to backup the following files
cp -p /var/ossec/etc/ /var/ossec_backup/etc/client.keys
cp -p /var/ossec/etc/ /var/ossec_backup/etc/ossec.conf
cp -p /var/ossec/queue/rids/sender_counter /var/ossec_backup/queue/rids/sender_counter
If you have made local changes to any of the following then also backup them:
cp -p /var/ossec/etc/local_internal_options.conf /var/ossec_backup/etc/local_internal_options.conf
cp -p /var/ossec/etc/rules/local_rules.xml /var/ossec_backup/rules/local_rules.xml
cp -p /var/ossec/etc/decoders/local_decoder.xml /var/ossec_backup/etc/local_decoder.xml
If you have the centralized configuration you must preserve:
cp -p /var/ossec/etc/shared/default/agent.conf /var/ossec_backup/etc/shared/agent.conf
Optionally the following files can be restored to preserve alert log files and syscheck/rootcheck databases:
cp -rp /var/ossec/logs/archives /var/ossec_backup/logs/archives/*
cp -rp /var/ossec/logs/alerts /var/ossec_backup/logs/alerts/*
cp -rp /var/ossec/queue/rootcheck /var/ossec_backup/queue/rootcheck/*
cp -rp /var/ossec/queue/syscheck /var/ossec_backup/queue/syscheck/*
After reinstalling, you need to place those files in their original path.
Also, in case you want to preserve your indexes after restarting consider making a backup of your indexes following this blog https://wazuh.com/blog/index-backup-management/.
I am using inotify to watch a directory and sync files between servers using rsync. Syncing works perfectly, and memory usage is mostly not an issue. However, recently a large number of files were added (350k) and this has impacted performance, specifically on CPU. Now when rsync runs, CPU usage spikes to 90%/100% and rsync takes long to complete, there are 650k files being watched/synced.
Is there any way to speed up rsync and only rsync the directory that has been changed? Or alternatively to set up multiple inotifywaits on separate directories. Script being used is below.
UPDATE: I have added the --update flag and usage seems mostly unchanged
#! /bin/bash
EVENTS="CREATE,DELETE,MODIFY,MOVED_FROM,MOVED_TO"
inotifywait -e "$EVENTS" -m -r --format '%:e %f' /var/www/ --exclude '/var/www/.*cache.*' | (
WAITING="";
while true; do
LINE="";
read -t 1 LINE;
if test -z "$LINE"; then
if test ! -z "$WAITING"; then
echo "CHANGE";
WAITING="";
rsync --update -alvzr --exclude '*cache*' --exclude '*.git*' /var/www/* root#secondwebserver:/var/www/
fi;
else
WAITING=1;
fi;
done)
I ended up removing the compression option (z) and upping the WAITING var to 10 (seconds). This seems to have helped, rsync still spikes CPU load but it is shorter lived. Credit goes to an answer on unix stackexchange
You're using rsync to synchronize the root directory of a large tree, so I'm not surprised at the performance loss.
One possible solution is to only synchronize the changed files/directories, instead of the whole root directory.
For instance, file1, file2 and file3 lay under from/dir. When changes are made to these 3 files, use
rsync --update -alvzr from/dir/file1 from/dir/file2 from/dir/file3 to/dir
rather than
rsync --update -alvzr from/dir/* to/dir
But this has a potential bug: rsync won't create directories automatically if target folders don't exist. However, you can use ssh to execute remote command and create directories by yourself.
You may need to set SSH public-key authentication as well, but according to the rsync command line you paste, I assume you've already done this.
reference:
rsync - create all missing parent directories?
rsync: how can I configure it to create target directory on server?
How to use SSH to run a shell script on a remote machine?
SSH error when executing a remote command: "stdin: is not a tty"
I've been using cp -r to duplicate some things in my file system and something bizarre keeps happening...certain directories that are permissioned as 777 have more secure permissions in the target directories I am copying to.
Is this normal behavior?
cp -a might resolve your situation, you need to tell cp to preserve the permissions so that it doesn't inherit them from your umask.
It appears that in this question, the answer was to separate statements with semicolons. However that could become cumbersome if we get into complex scripting with multiple if statements and complex quoted strings. I would think.
I imagine another alternative would be to simply issue multiple SSH commands one after the other, but again that would be cumbersome, plus I'm not set up for public/private key authentication so this would be asking for passwords a bunch of times.
What I'd ideally like is much similar to the interactive shell experience: at one point in the script you ssh into#the_remote_server and it prompts for the password, which you type in (interactively) and then from that point on until your script issues the "exit" command, all commands in the script are interpreted on the remote machine.
Of course this doesn't work:
ssh user#host.com
cd some/dir/on/remote/machine
tar -xzf my_tarball.tgz
cd some/other/dir/on/remote
cp -R some_directory somewhere_else
exit
Is there another alternative? I suppose I could take that part right out of my script and stick it into a script on the remote host. Meh. Now I'm maintaining two scripts. Plus I want a little configuration file to hold defaults and other stuff and I don't want to be maintaining that in two places either.
Is there another solution?
Use a heredoc.
ssh user#host.com << EOF
cd some/dir/on/remote/machine
tar -xzf my_tarball.tgz
cd some/other/dir/on/remote
cp -R some_directory somewhere_else
EOF
Use heredoc syntax, like
ssh user#host.com <<EOD
cd some/dir/on/remote/machine
...
EOD
or pipe, like
echo "ls -al" | ssh user#host.com
I would like to rsync from local computer to server. On a directory that does not exist, and I want rsync to create that directory on the server first.
How can I do that?
If you have more than the last leaf directory to be created, you can either run a separate ssh ... mkdir -p first, or use the --rsync-path trick as explained here :
rsync -a --rsync-path="mkdir -p /tmp/x/y/z/ && rsync" $source user#remote:/tmp/x/y/z/
Or use the --relative option as suggested by Tony. In that case, you only specify the root of the destination, which must exist, and not the directory structure of the source, which will be created:
rsync -a --relative /new/x/y/z/ user#remote:/pre_existing/dir/
This way, you will end up with /pre_existing/dir/new/x/y/z/
And if you want to have "y/z/" created, but not inside "new/x/", you can add ./ where you want --relativeto begin:
rsync -a --relative /new/x/./y/z/ user#remote:/pre_existing/dir/
would create /pre_existing/dir/y/z/.
From the rsync manual page (man rsync):
--mkpath create the destination's path component
--mkpath was added in rsync 3.2.3 (6 Aug 2020).
Assuming you are using ssh to connect rsync, what about to send a ssh command before:
ssh user#server mkdir -p existingdir/newdir
if it already exists, nothing happens
The -R, --relative option will do this.
For example: if you want to backup /var/named/chroot and create the same directory structure on the remote server then -R will do just that.
this worked for me:
rsync /dev/null node:existing-dir/new-dir/
I do get this message :
skipping non-regular file "null"
but I don't have to worry about having an empty directory hanging around.
I don't think you can do it with one rsync command, but you can 'pre-create' the extra directory first like this:
rsync --recursive emptydir/ destination/newdir
where 'emptydir' is a local empty directory (which you might have to create as a temporary directory first).
It's a bit of a hack, but it works for me.
cheers
Chris
This answer uses bits of other answers, but hopefully it'll be a bit clearer as to the circumstances. You never specified what you were rsyncing - a single directory entry or multiple files.
So let's assume you are moving a source directory entry across, and not just moving the files contained in it.
Let's say you have a directory locally called data/myappdata/ and you have a load of subdirectories underneath this.
You have data/ on your target machine but no data/myappdata/ - this is easy enough:
rsync -rvv /path/to/data/myappdata/ user#host:/remote/path/to/data/myappdata
You can even use a different name for the remote directory:
rsync -rvv --recursive /path/to/data/myappdata user#host:/remote/path/to/data/newdirname
If you're just moving some files and not moving the directory entry that contains them then you would do:
rsync -rvv /path/to/data/myappdata/*.txt user#host:/remote/path/to/data/myappdata/
and it will create the myappdata directory for you on the remote machine to place your files in. Again, the data/ directory must exist on the remote machine.
Incidentally, my use of -rvv flag is to get doubly verbose output so it is clear about what it does, as well as the necessary recursive behaviour.
Just to show you what I get when using rsync (3.0.9 on Ubuntu 12.04)
$ rsync -rvv *.txt user#remote.machine:/tmp/newdir/
opening connection using: ssh -l user remote.machine rsync --server -vvre.iLsf . /tmp/newdir/
user#remote.machine's password:
sending incremental file list
created directory /tmp/newdir
delta-transmission enabled
bar.txt
foo.txt
total: matches=0 hash_hits=0 false_alarms=0 data=0
Hope this clears this up a little bit.
eg:
from: /xxx/a/b/c/d/e/1.html
to: user#remote:/pre_existing/dir/b/c/d/e/1.html
rsync:
cd /xxx/a/ && rsync -auvR b/c/d/e/ user#remote:/pre_existing/dir/
rsync source.pdf user1#192.168.56.100:~/not-created/target.pdf
If the target file is fully specified, the directory ~/not-created is not created.
rsync source.pdf user1#192.168.56.100:~/will-be-created/
But the target is specified with only a directory, the directory ~/will-be-created is created. / must be followed to let rsync know will-be-created is a directory.
use rsync twice~
1: tranfer a temp file, make sure remote relative directories has been created.
tempfile=/Users/temp/Dir0/Dir1/Dir2/temp.txt
# Dir0/Dir1/Dir2/ is directory that wanted.
rsync -aq /Users/temp/ rsync://remote
2: then you can specify the remote directory for transfer files/directory
tempfile|dir=/Users/XX/data|/Users/XX/data/
rsync -avc /Users/XX/data rsync://remote/Dir0/Dir1/Dir2
# Tips: [SRC] with/without '/' is different
This creates the dir tree /usr/local/bin in the destination and then syncs all containing files and folders recursively:
rsync --archive --include="/usr" --include="/usr/local" --include="/usr/local/bin" --include="/usr/local/bin/**" --exclude="*" user#remote:/ /home/user
Compared to mkdir -p, the dir tree even has the same perms as the source.
If you are using a version or rsync that doesn't have 'mkpath', then --files-from can help. Suppose you need to create 'mysubdir' in the target directory
Create 'filelist.txt' to contain
mysubdir/dummy
mkdir -p source_dir/mysubdir/
touch source_dir/mysubdir/dummy
rsync --files-from='filelist.txt' source_dir target_dir
rsync will copy mysubdir/dummy to target_dir, creating mysubdir in the process. Tested with rsync 3.1.3 on Raspberry Pi OS (debian).