How do you configure a swap partition using cloud-init? - swap

We have an instance that uses cloudinit for initial instantiation, and this instance and cloudinit work great.
We want to add swap to this instance, and have correctly configured a suitable disk, however we cannot figure out how to get cloudinit to initialise the swap disks, like cloudinit does with all the other disks on the machine.
Our configuration of our disks, including swap, is as follows:
fs_setup:
- label: vidi
device: /dev/xvde
filesystem: ext4
- label: swap
device: /dev/xvdg
filesystem: swap
mounts:
- [ /dev/xvde, /var/lib/vidispine, ext4, defaults, 0, 0 ]
- [ /dev/xvdg, none, swap, sw, 0, 0 ]
This results in an /etc/fstab as follows:
LABEL=cloudimg-rootfs / ext4 defaults,discard 0 0
/dev/xvde /var/lib/vidispine ext4 defaults,comment=cloudconfig 0 0
/dev/xvdg none swap sw,comment=cloudconfig 0 0
The disk /dev/xvde is formatted correctly on startup. The disk /dev/xvdg is ignored.
What addditional steps are required for cloudinit to "mkswap" and "swapon" the /dev/xvdg disk?

In response to "What addditional steps are required for cloudinit to 'mkswap' and 'swapon' the /dev/xvdg disk?", the short answer is "nothing".
The longer answer is that you need to be running a version of cloud-init with the following bugfix applied:
https://github.com/canonical/cloud-init/pull/143
Which fixes the following error when running mkswap:
mkswap: invalid block count argument: ''
Most specifically, Ubuntu Bionic images 20200131 and newer work properly.
Older versions of cloudinit require the following added to the runcmd scripts on boot to work around the bug above:
- mkswap /dev/xvdg
- swapon -a

Solution: swap is a space on the disk that is used when the amount of physical ram of memory is full, therefore it can save your system from crashing due to out of memory exception. In order to apply swap partition using cloud-init package on ubuntu, you'll need to mount dedicated disk partition ( in boot time ) on /etc/fstab ( a configuration table designed to ease the burden of mounting and unmounting file systems to a machine) create ( using mkswap ) and start swap ( using swapon ).
First create an attach an additional disk to your machine, I'm attaching example using terraform:
resource "aws_instance" "example" {
ami ="<some-ami>"
instance_type = "t3.micro"
tags = {
Name = "example"
}
// root
root_block_device {
volume_size = 50
volume_type = "gp2"
delete_on_termination = true
}
// swap partition
ebs_block_device {
device_name = "/dev/xvdb"
volume_size = 20
volume_type = "gp2"
delete_on_termination = true
}
}
Second mount an additional disk and toggle the swap functionality on template file of cloud-init:
mounts:
- [ /dev/nvme1n1, none, swap, sw, 0, 0 ]
bootcmd:
- mkswap /dev/nvme1n1
- swapon /dev/nvme1n1
For verification: type on terminal:
swapon --show
#output:
NAME TYPE SIZE USED PRIO
/dev/nvme1n1 partition 20G 0B -2

Related

how to specify "disk=[....]" setting in "xl create" config?

I'm following the wiki https://help.ubuntu.com/community/Xen#Manually_Create_a_PV_Guest_VM
(section "
Set Up Initial Guest Configuration
")
I downloaded the netboot initrd.gz from https://mirror.arizona.edu/ubuntu//ubuntu/dists/bionic/main/installer-amd64/current/images/netboot/xen/
but in the .cfg , what should I specify for the "disk = " line? ---- my host box is not using LVM, so I'll have to use "file-backed storage" for PV disk image. (https://wiki.xenproject.org/wiki/Storage_options , indeed this worked when I gave --dir= instead of --lvm= when running the xml-create command in https://wiki.xenproject.org/wiki/Xen_Project_Beginners_Guide )
here is my current config:
yy#yy-70A4000HUX:~/ub_xen$ cat ub_xen.cfg
name = "ubud1"
kernel = "/home/yy/ub_xen/vmlinuz"
ramdisk = "/home/yy/ub_xen/initrd.gz"
#bootloader = "/usr/lib/xen-4.4/bin/pygrub"
memory = 1024
vcpus = 1
# Custom option for Open vSwitch
vif = [ 'bridge=xenbr0' ]
disk = [ 'vdev=hda,target=/home/yy/ub_xen/images' ]
# You may also consider some other options
# [[http://xenbits.xen.org/docs/4.4-testing/man/xl.cfg.5.html]]
yy#yy-70A4000HUX:~/ub_xen$
I ran the command with sudo xl create -c ub_xen.cfg
this worked fine first, giving me the regular install process on console, pulling install files from remote archive, but when it comes to the step of disk paritioning, it's showing me a "SCSI" partitioning choice, with no volumes / partitions/disks to be chosen.
I guess this is because I'm not setting the right value for "disk = [ ]" option. what should I use here if I use file-backed storage for PV (just like VMware does)?
thanks a lot
Yang
found it, huge thanks to the author here: https://www.systutorials.com/create-and-manage-virtual-machines-on-xen/

Does ns2 has capability to implement store-carry-forward mechanism? If it has how to do that?

I don't know ns 2 has capability to do this but I want to implement store-carry-forward mechanism in ns 2. However, I don't know where to start. Also I don't know what its steps. What protocols this mechanism uses? Is there anybody who can help me?
ns2 + DTN
If you have other ns2 builds / installs : Rename the executable´s ns to a new name → ns-orig, 'ns-app-name', etc. And remove any ns2 *PATH text from .bashrc .
Build ns2 + DTN
tar xvf ns-allinone-2.35_gcc5.tar.gz
https://drive.google.com/file/d/0B7S255p3kFXNVVlxR0ZNRGVORjQ/view?usp=sharing
cd ns-allinone-2.35/
zcat dtn_ns235.patch.gz | patch -p0
./install
cd ns-2.35/
sudo make install
Simulation : Copy an example (from ns2dtn_campaign/) to ns-allinone-2.35/, and run ./simulate_dtn.sh. The location is important as this path is used : ../ns-allinone-2.35/dei80211mr-1.1.4/src/.libs/libdei80211mr.so
Example, simulation files (and one empty folder) to be copied : { bundle-test-large-scen.tcl, create-traffic-file.tcl, scen_n40_pt2_ms20_t5000_x2000_y2000, simulate_dtn.sh, Run1/ }.
Please note that the simulation time is an hour (or more).
Watch the trace file qtrace.tr : Will very slowly increase to ~9MB.
Result : ns-allinone-2.35/Run1/{ bundle_delays.tr, qtrace.tr, receipt_delays.tr }. The files can be used with Xgraph.

GraphicsMagick and grunt-responsive-image - engine: 'im' breaks task

Goal - I am setting up an asset pipeline using grunt to convert and compress image files for a website.
Issue - grunt-responsive-images fails to find gm/convert.
Warning: Could not execute GraphicsMagick/ImageMagick: identify "-ping" "-format" "%m:%T:%s" "src/css/images/test.jpg" this most likely means the gm/convert binaries can't be found
GraphicsMagick is installed, and I am able to call gm version from the same directory:
GraphicsMagick 1.3.25 2016-09-05 Q16 http://www.GraphicsMagick.org/
If I comment out the engine: 'im' line below, the task runs and size and compression settings are applied.
grunt.initConfig({
responsive_images: {
dev: {
options: {
// engine: 'im', <--does this invoke gm?
sizes: [{
name: 'small',
width: '30%',
suffix: '_small',
quality: 60
},{
name: 'large',
width: '50%',
suffix: '_large',
quality: 40
}]
},
Question 1 - What is engine: 'im' doing?
Question 2 - How is grunt-responsive-images resizing and compressing images files when engine: 'im' isn't specified? Is it using built-in image tools?
UPDATE
When I changed engine: 'im' to engine: 'gm', the tasks completed with no errors. Maybe im = imageMagick and gm = graphicsMagick? Also, maybe, grunt-responsive-images can find gm, but cannot find im?
Question 3 - How would I fix grunt-responsive-images to work with imageMagick (like I see in many tutorials)?
Thank you!
I found the answers in the documentation: grunt-responsive-images\options
Answer 1 -
options.engine: Chooses which graphics engine to use when resizing images. To use GraphicsMagick, set this to gm. To use ImageMagick, set this to im. You'll need the relevant engine installed.
Answer 2 -
I had gm installed via npm, and grunt-responsive-images was using that to compress and resize. I did not have imagemagick installed. brew list did not show imagemagick (graphicsmagick was listed, and probably why I thought I had installed imagemagick).
Answer 3 - The fix is simple; install imagemagick with homebrew:
brew install imagemagick then check with brew list. Now grunt runs with either 'gm' or 'im' engines selected.

Has there ever been a unix system call to create a link from an open file descriptor? [duplicate]

In Unix, it's possible to create a handle to an anonymous file by, e.g., creating and opening it with creat() and then removing the directory link with unlink() - leaving you with a file with an inode and storage but no possible way to re-open it. Such files are often used as temp files (and typically this is what tmpfile() returns to you).
My question: is there any way to re-attach a file like this back into the directory structure? If you could do this it means that you could e.g. implement file writes so that the file appears atomically and fully formed. This appeals to my compulsive neatness. ;)
When poking through the relevant system call functions I expected to find a version of link() called flink() (compare with chmod()/fchmod()) but, at least on Linux this doesn't exist.
Bonus points for telling me how to create the anonymous file without briefly exposing a filename in the disk's directory structure.
A patch for a proposed Linux flink() system call was submitted several years ago, but when Linus stated "there is no way in HELL we can do this securely without major other incursions", that pretty much ended the debate on whether to add this.
Update: As of Linux 3.11, it is now possible to create a file with no directory entry using open() with the new O_TMPFILE flag, and link it into the filesystem once it is fully formed using linkat() on /proc/self/fd/fd with the AT_SYMLINK_FOLLOW flag.
The following example is provided on the open() manual page:
char path[PATH_MAX];
fd = open("/path/to/dir", O_TMPFILE | O_RDWR, S_IRUSR | S_IWUSR);
/* File I/O on 'fd'... */
snprintf(path, PATH_MAX, "/proc/self/fd/%d", fd);
linkat(AT_FDCWD, path, AT_FDCWD, "/path/for/file", AT_SYMLINK_FOLLOW);
Note that linkat() will not allow open files to be re-attached after the last link is removed with unlink().
My question: is there any way to re-attach a file like this back into the directory structure? If you could do this it means that you could e.g. implement file writes so that the file appears atomically and fully formed. This appeals to the my compulsive neatness. ;)
If this is your only goal, you can achieve this in a much simpler and more widely used manner. If you are outputting to a.dat:
Open a.dat.part for write.
Write your data.
Rename a.dat.part to a.dat.
I can understand wanting to be neat, but unlinking a file and relinking it just to be "neat" is kind of silly.
This question on serverfault seems to indicate that this kind of re-linking is unsafe and not supported.
Thanks to #mark4o posting about linkat(2), see his answer for details.
I wanted to give it a try to see what actually happened when trying to actually link an anonymous file back into the filesystem it is stored on. (often /tmp, e.g. for video data that firefox is playing).
As of Linux 3.16, there still appears to be no way to undelete a deleted file that's still held open. Neither AT_SYMLINK_FOLLOW nor AT_EMPTY_PATH for linkat(2) do the trick for deleted files that used to have a name, even as root.
The only alternative is tail -c +1 -f /proc/19044/fd/1 > data.recov, which makes a separate copy, and you have to kill it manually when it's done.
Here's the perl wrapper I cooked up for testing. Use strace -eopen,linkat linkat.pl - </proc/.../fd/123 newname to verify that your system still can't undelete open files. (Same applies even with sudo). Obviously you should read code you find on the Internet before running it, or use a sandboxed account.
#!/usr/bin/perl -w
# 2015 Peter Cordes <peter#cordes.ca>
# public domain. If it breaks, you get to keep both pieces. Share and enjoy
# Linux-only linkat(2) wrapper (opens "." to get a directory FD for relative paths)
if ($#ARGV != 1) {
print "wrong number of args. Usage:\n";
print "linkat old new \t# will use AT_SYMLINK_FOLLOW\n";
print "linkat - <old new\t# to use the AT_EMPTY_PATH flag (requires root, and still doesn't re-link arbitrary files)\n";
exit(1);
}
# use POSIX qw(linkat AT_EMPTY_PATH AT_SYMLINK_FOLLOW); #nope, not even POSIX linkat is there
require 'syscall.ph';
use Errno;
# /usr/include/linux/fcntl.h
# #define AT_SYMLINK_NOFOLLOW 0x100 /* Do not follow symbolic links. */
# #define AT_SYMLINK_FOLLOW 0x400 /* Follow symbolic links. */
# #define AT_EMPTY_PATH 0x1000 /* Allow empty relative pathname */
unless (defined &AT_SYMLINK_NOFOLLOW) { sub AT_SYMLINK_NOFOLLOW() { 0x0100 } }
unless (defined &AT_SYMLINK_FOLLOW ) { sub AT_SYMLINK_FOLLOW () { 0x0400 } }
unless (defined &AT_EMPTY_PATH ) { sub AT_EMPTY_PATH () { 0x1000 } }
sub my_linkat ($$$$$) {
# tmp copies: perl doesn't know that the string args won't be modified.
my ($oldp, $newp, $flags) = ($_[1], $_[3], $_[4]);
return !syscall(&SYS_linkat, fileno($_[0]), $oldp, fileno($_[2]), $newp, $flags);
}
sub linkat_dotpaths ($$$) {
open(DOTFD, ".") or die "open . $!";
my $ret = my_linkat(DOTFD, $_[0], DOTFD, $_[1], $_[2]);
close DOTFD;
return $ret;
}
sub link_stdin ($) {
my ($newp, ) = #_;
open(DOTFD, ".") or die "open . $!";
my $ret = my_linkat(0, "", DOTFD, $newp, &AT_EMPTY_PATH);
close DOTFD;
return $ret;
}
sub linkat_follow_dotpaths ($$) {
return linkat_dotpaths($_[0], $_[1], &AT_SYMLINK_FOLLOW);
}
## main
my $oldp = $ARGV[0];
my $newp = $ARGV[1];
# link($oldp, $newp) or die "$!";
# my_linkat(fileno(DIRFD), $oldp, fileno(DIRFD), $newp, AT_SYMLINK_FOLLOW) or die "$!";
if ($oldp eq '-') {
print "linking stdin to '$newp'. You will get ENOENT without root (or CAP_DAC_READ_SEARCH). Even then doesn't work when links=0\n";
$ret = link_stdin( $newp );
} else {
$ret = linkat_follow_dotpaths($oldp, $newp);
}
# either way, you still can't re-link deleted files (tested Linux 3.16 and 4.2).
# print STDERR
die "error: linkat: $!.\n" . ($!{ENOENT} ? "ENOENT is the error you get when trying to re-link a deleted file\n" : '') unless $ret;
# if you want to see exactly what happened, run
# strace -eopen,linkat linkat.pl
Clearly, this is possible -- fsck does it, for example. However, fsck does it with major localized file system mojo and will clearly not be portable, nor executable as an unprivileged user. It's similar to the debugfs comment above.
Writing that flink(2) call would be an interesting exercise. As ijw points out, it would offer some advantages over current practice of temporary file renaming (rename, note, is guaranteed atomic).
Kind of late to the game but I just found http://computer-forensics.sans.org/blog/2009/01/27/recovering-open-but-unlinked-file-data which may answer the question. I haven't tested it, though, so YMMV. It looks sound.

Is there a way to display only changes and errors

I have quite extensive salt config and I want to be able to see what has changed. If I just run salt '*' state.highstate I got the whole list with things that were present and not changed - like 3 to 4 screens of log. But I'd really like to see only things that changed in the last job.
It doesn't have to work for the salt call, it can also employ salt-run jobs.lookup_jid.
You can set state_verbose: False in /etc/salt/master or /etc/salt/minion.
If you want to shorten the output to one line per state, set state_output: terse.
You can also pass these filters on command line:
salt --state-output=terse '*' state.highstate
If you only want to see changes, you can use state-output=changes or state-output=mixed. The latter one will show more information on a failure.
See the following answers fore more detail: basepi, psarossy
We've also added state_output: mixed which will give you the same output as terse, except if there's a failure, in which case it will give you the more verbose output.
To actually answer the question, yes, there is an output filter for changes only:
salt '*' state.highstate --state-output=changes
This will display one liners for things that are in the right state and the proper output for the changes. ie:
<...>
Name: /etc/sudoers - Function: file.managed - Result: Clean
Name: /etc/timezone - Function: file.managed - Result: Clean
Name: /etc/pki/tls/certs/logstash-forwarder.crt - Function: file.managed - Result: Clean
Name: /etc/init.d/logstash-forwarder - Function: file.managed - Result: Clean
----------
ID: /etc/logstash-forwarder
Function: file.managed
Result: True
Comment: File /etc/logstash-forwarder updated
Started: 14:14:28.580950
Duration: 65.664 ms
Changes:
----------
diff:
---
+++
## -1,6 +1,6 ##
{
"network": {
- "servers": [ "10.0.0.104:5000" ],
+ "servers": [ "10.0.0.72:5000" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
},
Name: deb http://packages.elasticsearch.org/logstashforwarder/debian stable main - Function: pkgrepo.managed - Result: Clean
Name: logstash-forwarder - Function: pkg.installed - Result: Clean
<...>
There are 2 options, first is to change the state_output in master's configuration file, like mentioned in the accepted answer, and it also possible to override the state output in command line, like:
salt --state-output=mixed \* test.version
As of the following PR that was merged into Salt 2015.8.0 (https://github.com/saltstack/salt/pull/26962) it is now possible to toggle the state_verbose flag from command line when running highstate. This overrides the config you can set in /etc/salt/master that was mentioned in previous answers.
The following command should now display only the changes and errors from a highstate run salt '*' state.highstate --state-verbose=False
You can use the below to shorten the output in one line and then filter that output to show only the changes:
salt -v 'minion' state.highstate test=True --state-output=terse --state-verbose=False

Resources