QEMU serial std output diverges on archlinux guest - serial-port

I'm trying to bootstrap some installation automation of a freshly downloaded ISO in QEMU. I create a clean img to install to and kick off QEMU like this:
$ qemu-img create -f qcow2 out/main.img 15G
$ qemu-system-x86_64
-m 8G \
-serial stdio \
-cdrom out/linux.iso \
-drive file=out/main.img,if=virtio \
-netdev user,id=net0 \
-device e1000,netdev=net0
and I can see Arch boot up. At first both the display and the terminal are in sync, but they soon diverge after this the GRUB boot up screen.
I'm not sure what piece I'm missing to get this to work. I've seen some people suggest adding -append "root=/dev/sda console=ttyS0" to your QEMU arguments, but (from what I can tell) while it requires you to extract the kernel and the initram from the ISO (which should be easy enough as mounting and copy/pasting the correct files) but it also expects you to already have an installed system on /dev/sda (which is what I'm trying to bootstrap).
At this point I don't know what to search for next, how do I get the full terminal session in my current terminal and not just in my display?

In this case, it was as #Peter Maydell commented; this is not a QEMU question. QEMU was doing exactly what it was supposed to do, but Arch had to be told to utilize the serial console as its primary means of communication.
Two samples of how this can be done
bash via console
pipe_dir="$(mktemp -d)"
mkfifo "${pipe_dir}/pipe.in" "${pipe_dir}/pipe.out"
function cleanup {
rm -rfv "${pipe_dir}"
}
trap cleanup EXIT
qemu-system-x86_64 \
-m 8G \
-display none \
-serial stdio \
-drive file=./out/linux.iso,index=0,media=cdrom \
-drive file=./out/main.img,if=virtio &
sleep 2s
printf "\t" > "${pipe_dir}/pipe.in"
sleep 2s
printf " console=ttyS0,115200" > "${pipe_dir}/pipe.in"
sleep 2s
echo > "${pipe_dir}/pipe.in"
# Whatever other interactions you want go here...
wait
expect via console
set timeout -1
spawn qemu-system-x86_64 \
-m 8G \
-display none \
-serial stdio \
-drive file=./out/linux.iso,index=0,media=cdrom \
-drive file=./out/main.img,if=virtio
sleep 1
send \t
sleep 1
send " console=ttyS0,115200"
sleep 1
send \n
In theory this should be fine, but in practice I still had difficulty interacting with the console and sending characters over to login correctly. I'm sure there is probably more user-error on my part than not.
A better solution (again contextual to Arch and not QEMU specifically) was to use a cloud-init script that included my SSH public key. Interactions with the VM were stable, reliable, and easily reproducible.
bash with cloud-init/ssh
$ touch ./out/meta-data
$ cat > ./out/user-data <<EOF
#cloud-config
users:
- name: root
ssh_authorized_keys:
- $(cat ${HOME}/.ssh/id_ed25519.pub)
EOF
$ xorriso -as genisoimage -output ./out/cloud-init.iso \
-volid CIDATA -joliet -rock ./out/meta-data ./out/user-data
$ qemu-system-x86_64 \
-m 8G \
-drive file=./out/linux.iso,index=0,media=cdrom \
-drive file=./out/cloud-init.iso,index=1,media=cdrom \
-drive file=./out/main.img,if=virtio \
-net user,hostfwd=tcp::10022-:22 \
-net nic &
$ function qemu-ssh {
ssh -q -o ConnectTimeout=5 -o StrictHostKeyChecking=no -o "UserKnownHostsFile /dev/null" -p 10022 root#localhost ${#}
}
$ printf 'Waiting for SSH to go live (this will take a while)...'
$ until qemu-ssh exit; do
printf '.'
done
# This convenience function starts an interactive
# session when supplied with no additional arguments
# but your automation can go here
$ qemu-ssh

Related

Cosmos DB emulator installation on Mac

ipaddr="`ifconfig | grep "inet " | grep -Fv 127.0.0.1 | awk '{print $2}' | head -n 1`"
docker pull mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator
docker run \
--publish 8081:8081 \
--publish 10250-10255:10250-10255 \
--memory 3g --cpus=2.0 \
--name=test-linux-emulator1 \
--env AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10 \
--env AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=true \
--env AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=$ipaddr \
--interactive \
--tty \
mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator
I followed these steps but getting this error
./cosmosdb-emulator: ERROR: Invalid mapping of address 0x40037db000 in reserved address space below 0x400000000000. Possible causes:
1) the process (itself, or via a wrapper) starts-up its own running environment sets the stack size limit to unlimited via syscall setrlimit(2);
2) the process (itself, or via a wrapper) adjusts its own execution domain and flag the system its legacy personality via syscall personality(2);
3) sysadmin deliberately sets the system to run on legacy VA layout mode by adjusting a sysctl knob vm.legacy_va_layout.
Mac Chip: Apple M1 Pro
From the official doc,
The emulator only supports MacBooks with Intel processors.
The Apple M1 Pro you are using is not supported by the emulator

How to install and setup WordPress using Podman

With docker I was able to run WordPress example for docker-compose on nearly every platform, without prior docker knowledge.
I look for a way to achieve the same with Podman.
In my case, to have a fast cross-platform way to setup a working WordPress installation for development.
As Podman is far younger, a valid answer in 2022 would also be: It is not possible, because... / only possible provided constraint X.
Still I would like to create an entry point for other people, who run into the same issue in the future.
I posted my own efforts below. Before I spend more hours debugging lots of small (but still solvable) issues, I wanted to find out if someone else faced the same problem and already has a solution. If you have, please clearly document its constraints.
My particular issue, as a reference
I am on Ubuntu 20.04 and podman -v gives 3.4.2.
docker/podman compose
When I use docker-compose up with Podman back-end on docker's WordPress .yml-file, I run into the "duplicate mount destination" issue.
podman-compose is part of Podman 4.1.0, which is not available on Ubuntu as I write this.
Red Hat example
The example of Red Hat gives "Error establishing a database connection ... contact with the database server at mysql could not be established".
A solution for the above does not work for me. share is likely a typo. I tried to replace with unshare.
Cent OS example
I found an example which uses pods instead of a docker-compose.yml file. But it is written for Cent OS.
I modified the Cent OS example, see the script below. I get the containers up and running. However, WordPress is unable to connect to the database.
#!/bin/bash
# Set environment variables:
DB_NAME='wordpress_db'
DB_PASS='mysupersecurepass'
DB_USER='justbeauniqueuser'
POD_NAME='wordpress_with_mariadb'
CONTAINER_NAME_DB='wordpress_db'
CONTAINER_NAME_WP='wordpress'
mkdir -P html
mkdir -P database
# Remove previous attempts
sudo podman pod rm -f $POD_NAME
# Pull before run, bc: invalid reference format eror
sudo podman pull mariadb:latest
sudo podman pull wordpress
# Create a pod instead of --link. So both containers are able to reach each others.
sudo podman pod create -n $POD_NAME -p 80:80
sudo podman run --detach --pod $POD_NAME \
-e MYSQL_ROOT_PASSWORD=$DB_PASS \
-e MYSQL_PASSWORD=$DB_PASS \
-e MYSQL_DATABASE=$DB_NAME \
-e MYSQL_USER=$DB_USER \
--name $CONTAINER_NAME_DB -v "$PWD/database":/var/lib/mysql \
docker.io/mariadb:latest
sudo podman run --detach --pod $POD_NAME \
-e WORDPRESS_DB_HOST=127.0.0.1:3306 \
-e WORDPRESS_DB_NAME=$DB_NAME \
-e WORDPRESS_DB_USER=$DB_USER \
-e WORDPRESS_DB_PASSWORD=$DB_PASS \
--name $CONTAINER_NAME_WP -v "$PWD/html":/var/www/html \
docker.io/wordpress
Also, I was a bit unsure where to post this question. If server fault or another stack exchange are a better fit, I will happily post there.
Actually, your code works with just small changes.
I removed the sudo's and changed the pods external port to 8090, instead of 80. So now everything is running as a non-root user.
#!/bin/bash
# https://stackoverflow.com/questions/74054932/how-to-install-and-setup-wordpress-using-podman
# Set environment variables:
DB_NAME='wordpress_db'
DB_PASS='mysupersecurepass'
DB_USER='justbeauniqueuser'
POD_NAME='wordpress_with_mariadb'
CONTAINER_NAME_DB='wordpress_db'
CONTAINER_NAME_WP='wordpress'
mkdir -p html
mkdir -p database
# Remove previous attempts
podman pod rm -f $POD_NAME
# Pull before run, bc: invalid reference format error
podman pull docker.io/mariadb:latest
podman pull docker.io/wordpress
# Create a pod instead of --link.
# So both containers are able to reach each others.
podman pod create -n $POD_NAME -p 8090:80
podman run --detach --pod $POD_NAME \
-e MYSQL_ROOT_PASSWORD=$DB_PASS \
-e MYSQL_PASSWORD=$DB_PASS \
-e MYSQL_DATABASE=$DB_NAME \
-e MYSQL_USER=$DB_USER \
--name $CONTAINER_NAME_DB -v "$PWD/database":/var/lib/mysql \
docker.io/mariadb:latest
podman run --detach --pod $POD_NAME \
-e WORDPRESS_DB_HOST=127.0.0.1:3306 \
-e WORDPRESS_DB_NAME=$DB_NAME \
-e WORDPRESS_DB_USER=$DB_USER \
-e WORDPRESS_DB_PASSWORD=$DB_PASS \
--name $CONTAINER_NAME_WP -v "$PWD/html":/var/www/html \
docker.io/wordpress
This is what worked for me:
#!/bin/bash
# https://stackoverflow.com/questions/74054932/how-to-install-and-setup-wordpress-using-podman
# Set environment variables:
POD_NAME='wordpress_mariadb'
DB_ROOT_PW='sup3rS3cr3t'
DB_NAME='wp'
DB_PASS='s0m3wh4tS3cr3t'
DB_USER='wordpress'
podman pod create --name $POD_NAME -p 8080:80
podman run \
-d --restart=always --pod=$POD_NAME \
-e MYSQL_ROOT_PASSWORD="$DB_ROOT_PW" \
-e MYSQL_DATABASE="$DB_NAME" \
-e MYSQL_USER="$DB_USER" \
-e MYSQL_PASSWORD="$DB_PASS" \
-v $HOME/public_html/wordpress/mysql:/var/lib/mysql:Z \
--name=wordpress-db docker.io/mariadb:latest
podman run \
-d --restart=always --pod=$POD_NAME \
-e WORDPRESS_DB_NAME="$DB_NAME" \
-e WORDPRESS_DB_USER="$DB_USER" \
-e WORDPRESS_DB_PASSWORD="$DB_PASS" \
-e WORDPRESS_DB_HOST="127.0.0.1" \
-v $HOME/public_html/wordpress/html:/var/www/html:Z \
--name wordpress docker.io/library/wordpress:latest

Varnish 6.0.8 Secret file is not created

Please we're facing some issues when installing Varnish 6.0.8 on ubutnu 18.04.6 OS, it doesn't create the secret file inside the /etc/varnish dir as shown below:
enter image description here
we use the following script to for installation :
curl -s https://packagecloud.io/install/repositories/varnishcache/varnish60lts/script.deb.sh | sudo bash
can someone please help ?
PS: we tried to install later versions (6.6 and 7.0.0) and we got the same issue.
Form a security point of view, remote CLI access is not enabled by default. You can see this when looking at /lib/systemd/system/varnish.service:
[Unit]
Description=Varnish Cache, a high-performance HTTP accelerator
After=network-online.target nss-lookup.target
[Service]
Type=forking
KillMode=process
# Maximum number of open files (for ulimit -n)
LimitNOFILE=131072
# Locked shared memory - should suffice to lock the shared memory log
# (varnishd -l argument)
# Default log size is 80MB vsl + 1M vsm + header -> 82MB
# unit is bytes
LimitMEMLOCK=85983232
# Enable this to avoid "fork failed" on reload.
TasksMax=infinity
# Maximum size of the corefile.
LimitCORE=infinity
ExecStart=/usr/sbin/varnishd \
-a :6081 \
-a localhost:8443,PROXY \
-p feature=+http2 \
-f /etc/varnish/default.vcl \
-s malloc,256m
ExecReload=/usr/sbin/varnishreload
[Install]
WantedBy=multi-user.target
There are no -T and -S parameters in the standard systemd configuration. However, you can enable this by modifying the systemd configuration yourself.
Just run sudo systemctl edit --full varnish to edit the runtime configuration and add a -T parameter to enable remote CLI access.
Be careful with this and make sure you restrict access to this endpoint via firewalling rules.
Additionally you'll add -S /etc/varnish/secret as a varnishd runtime parameter in /lib/systemd/system/varnish.service.
You can use the following command to add a random unique value to the secret file:
uuidgen | sudo tee /etc/varnish/secret
This is what your runtime parameters would look like:
ExecStart=/usr/sbin/varnishd \
-a :6081 \
-a localhost:8443,PROXY \
-p feature=+http2 \
-f /etc/varnish/default.vcl \
-s malloc,2g \
-S /etc/varnish/secret \
-T :6082
When you're done just run the following command to restart Varnish:
sudo systemctl restart varnish

rsync to local USB disk gives "rsync error 2 (Protocol incompatibility)

I am using backintime for backup which in turn uses rsync to make snapshots.
Most filesystems on the computer are XFS, including the rsync target,
system is Ubuntu 20.04 with rsync version 3.1.3 protocol version 31.
I get an exit code 2 from rsync which is Protocol incompatibility,
and some digging shows this happens if you are running rsync across some (ssh) connection
between two computers with different rsync versions, or login scripts injecting some unexpected output into the ssh connection. None of that is the case here, this is all local,
see below for the command line.
=> Anymore insights into this rsync error ? How can a local Protocol incompatibility happen
if there is just one /usr/bin/rsync ?
Yours,
Steffen
The local USB disk mounted as
type xfs (rw,nosuid,nodev,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota,uhelper=udisks2)
INFO: Call rsync to take the snapshot
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
WARNING: Command "rsync --recursive --times --devices --specials --hard-links --human-readable \
--links --acls --xattrs --perms --executability --group --owner --info=progress2 \
--no-inc-recursive --delete --delete-excluded -v -i \
--out-format=BACKINTIME: %i %n%L --link-dest=../../20210301-082432-781/backup \
--chmod=Du+wx --exclude=/media/sneumann/LinuxBackup/msbi-corei \
--exclude=/root/.local/share/backintime --exclude=.local/share/backintime/mnt \
--exclude=.gvfs --exclude=.cache* --exclude=[Cc]ache* --exclude=.thumbnails* \
--exclude=[Tt]rash* --exclude=*.backup* --exclude=*~ \
--exclude=/home/sneumann/Ubuntu One --exclude=.dropbox* --exclude=/proc/* \
--exclude=/sys/* --exclude=/dev/* --exclude=/run/* --exclude=/media \
--exclude=/root/.local/share/backintime/takesnapshot_.log \
--exclude=/root/.local/share/backintime --include=/ --include=/** \
--exclude=* / /media/sneumann/LinuxBackup/msbi-corei/backintime/msbi-corei/root/1/new_snapshot/backup"
returns 2

How are zsh autocompletions for commands with subcommands defined?

I am trying to write a tab-completion script for borg.
So far, I have managed to define completions for borg itself, as well as borg key with its subcommands and borg benchmark with its singular subcommand. However, I am now trying to define completion for borg init and I am having trouble.
The issue presents itself only when I define two arguments under the borg init command to use the same description text; i.e. both -e and --encryption should use the same description, as they are practically the same argument. This has worked fine for borg's arguments, but now it breaks.
This is my code, slightly redacted to spare you the redundancy:
compdef _borg borg
function _borg {
local line ret=1
local -a argus
local logs="--critical --error --warning --debug --info -v --verbose"
argus+=(
"(*)"{-h,--help}"[Show help and exit]"
"(*)-V[Show Borg version and exit]"
"($logs)--critical[Work on log level CRITICAL]"
"($logs)--error[Work on log level ERROR]"
"($logs)--warning[Work on log level WARNING (default)]"
"($logs)"{--info,-v,--verbose}"[Work on log level INFO]"
"($logs)--debug[Enable debug output; log level DEBUG]"
{-p,--progress}"[Show progress]"
"--log-json[Output one JSON object per log line instead of formatted text]"
"--show-version[Show/log borg version]"
"--show-rc[Show/log returncode]"
"--consider-part-files[treat part files like normal files (e.g. to list/extract them)]"
"--lock-wait[Wait at most SECONDS for acquiring a repository/cache lock (default 1)]:SECONDS:()"
"--umask[Set umask to M (local and remote; default 0077)]:M (umask value, e.g. 0077):()"
"--remote-path[Use PATH as borg executable on the remote (default: \"borg\")]:PATH:()"
"--remote-ratelimit[Set remote network upload rate limit in kiByte/s (default: 0=unlimited)]:RATE:()"
"--debug-profile[Write execution profile in Borg format into FILE.]:FILE:_files"
"--rsh[Use this command to connect to the \"borg serve\" process (default: \"ssh\")]:RSH:()"
"1: :((init\:\"Initialize a new repository\" \
create\:\"Create a new archive\" \
extract\:\"Extract the contents of an archive\" \
check\:\"Verifies consistency of a repository and its archives\" \
rename\:\"Renames an archive in a repository\" \
list\:\"Lists contents of a repository or archive\" \
diff\:\"Finds differences between archives\" \
delete\:\"Deletes an archive or an entire repository (and its cache)\" \
prune\:\"Prunes a repository\" \
info\:\"Shows info about a repository or archive\" \
mount\:\"Mounts an archive as a FUSE filesystem\" \
unmount\:\"Unmounts a FUSE filesystem mounted with \\\"borg mount\\\"\" \
key\:\"Keyword for key-related functions\" \
upgrade\:\"Upgrade a local Borg repository\" \
recreate\:\"EXPERIMENTAL: Recreates contents of existing archives\" \
export-tar\:\"Creates a tarball from an archive\" \
serve\:\"Starts repository server process. Not usually used manually.\" \
config\:\"Gets and sets options in local repository and cache config files\" \
with-lock\:\"Executes another command with the repository lock held\" \
break-lock\:\"Breaks the repository and cache locks\" \
benchmark\:\"Keyword for the benchmark function\"))" \
"*::arg:->args"
)
_arguments -w -s -S -C $argus[#] && ret=0
case $line[1] in
benchmark)
_borg_benchmark
;;
init)
_borg_init
;;
key)
_borg_key
;;
esac
return ret
}
function _borg_benchmark {
# stuff
}
function _borg_benchmark_crud {
# stuff again
}
function _borg_init {
local line ret=1
local -a argus
argus+=(
"-t[This is a test]"
"--test[This is a test]"
"(--append-only)--append-only[Create an append-only mode repository]"
"*::arg:->args"
)
_arguments -w -s -S -C $argus[#] && ret=0
return ret
}
function _borg_key {
# key stuff
}
function _borg_key_changepassphrase {
# stuff
}
function _borg_key_export {
# more stuff
}
function _borg_key_import {
# other stuff
}
If I try to tab-complete borg init - using this setup, I get the following output:
$ borg init -
Completing option
--append-only
--test
-t
-- Create an append-only mode repository
-- This is a test
--append-only
--test
-t
-- Create an append-only mode repository
-- This is a test
--append-only
--test
-t
-- Create an append-only mode repository
-- This is a test
--append-only
--test
-t
-- Create an append-only mode repository
-- This is a test
The completion appears to forget what tabs are and repeats itself four times. If I change --test[This is a test] to --test[This is another test] in _borg_init, I instead get the following completion:
$ borg init -
Completing option
--append-only -- Create an append-only mode repository
--test -- This is another test
-t -- This is a test
The above is "correct", in the sense that it's not broken, but I cannot seem to define arguments that share a description in a subcommand. How should I do that? And, more generally, how are you supposed to define completions for commands with subcommands (which may, in turn, have more arguments)?

Resources