Image Volume Cache not working for RAW images - openstack

The image volume cache is only working for a QCOW2 image and not for a RAW image. This feature of Cinder promises to improve the volume creation performance drastically.
https://docs.openstack.org/cinder/latest/admin/blockstorage-image-volume-cache.html

Create volume from the image, first try to call clone_image interface, then try to call clone_image_volume interface, and then try and use the image cache. If all else fails, fall back to default behavior of creating volume download the image data and copy it into the volume.
If volume backend is ceph and RAW image also in ceph, create volume from a RAW image, will clone image directly in ceph, and not use the image cache.
#utils.retry(exception.SnapshotLimitReached, retries=1)
def _create_from_image(self, context, volume,
image_location, image_id, image_meta,
image_service, **kwargs):
LOG.debug("Cloning %(volume_id)s from image %(image_id)s "
" at location %(image_location)s.",
{'volume_id': volume.id,
'image_location': image_location, 'image_id': image_id})
virtual_size = image_meta.get('virtual_size')
if virtual_size:
virtual_size = image_utils.check_virtual_size(virtual_size,
volume.size,
image_id)
# Create the volume from an image.
#
# First see if the driver can clone the image directly.
#
# NOTE (singn): two params need to be returned
# dict containing provider_location for cloned volume
# and clone status.
# NOTE (lixiaoy1): Currently all images are raw data, we can't
# use clone_image to copy data if new volume is encrypted.
volume_is_encrypted = volume.encryption_key_id is not None
cloned = False
model_update = None
if not volume_is_encrypted:
model_update, cloned = self.driver.clone_image(context,
volume,
image_location,
image_meta,
image_service)
# Try and clone the image if we have it set as a glance location.
if not cloned and 'cinder' in CONF.allowed_direct_url_schemes:
model_update, cloned = self._clone_image_volume(context,
volume,
image_location,
image_meta)
# If we're going to try using the image cache then prepare the cache
# entry. Note: encrypted volume images are not cached.
if not cloned and self.image_volume_cache and not volume_is_encrypted:
# If _prepare_image_cache_entry() has to create the cache entry
# then it will also create the volume. But if the volume image
# is already in the cache then it returns (None, False), and
# _create_from_image_cache_or_download() will use the cache.
model_update, cloned = self._prepare_image_cache_entry(
context,
volume,
image_location,
image_id,
image_meta,
image_service)
# Try and use the image cache, and download if not cached.
if not cloned:
model_update = self._create_from_image_cache_or_download(
context,
volume,
image_location,
image_id,
image_meta,
image_service)
self._handle_bootable_volume_glance_meta(context, volume,
image_id=image_id,
image_meta=image_meta)
return model_update
https://github.com/openstack/cinder/blob/master/cinder/volume/flows/manager/create_volume.py

Related

how to specify "disk=[....]" setting in "xl create" config?

I'm following the wiki https://help.ubuntu.com/community/Xen#Manually_Create_a_PV_Guest_VM
(section "
Set Up Initial Guest Configuration
")
I downloaded the netboot initrd.gz from https://mirror.arizona.edu/ubuntu//ubuntu/dists/bionic/main/installer-amd64/current/images/netboot/xen/
but in the .cfg , what should I specify for the "disk = " line? ---- my host box is not using LVM, so I'll have to use "file-backed storage" for PV disk image. (https://wiki.xenproject.org/wiki/Storage_options , indeed this worked when I gave --dir= instead of --lvm= when running the xml-create command in https://wiki.xenproject.org/wiki/Xen_Project_Beginners_Guide )
here is my current config:
yy#yy-70A4000HUX:~/ub_xen$ cat ub_xen.cfg
name = "ubud1"
kernel = "/home/yy/ub_xen/vmlinuz"
ramdisk = "/home/yy/ub_xen/initrd.gz"
#bootloader = "/usr/lib/xen-4.4/bin/pygrub"
memory = 1024
vcpus = 1
# Custom option for Open vSwitch
vif = [ 'bridge=xenbr0' ]
disk = [ 'vdev=hda,target=/home/yy/ub_xen/images' ]
# You may also consider some other options
# [[http://xenbits.xen.org/docs/4.4-testing/man/xl.cfg.5.html]]
yy#yy-70A4000HUX:~/ub_xen$
I ran the command with sudo xl create -c ub_xen.cfg
this worked fine first, giving me the regular install process on console, pulling install files from remote archive, but when it comes to the step of disk paritioning, it's showing me a "SCSI" partitioning choice, with no volumes / partitions/disks to be chosen.
I guess this is because I'm not setting the right value for "disk = [ ]" option. what should I use here if I use file-backed storage for PV (just like VMware does)?
thanks a lot
Yang
found it, huge thanks to the author here: https://www.systutorials.com/create-and-manage-virtual-machines-on-xen/

changing the text in a toolbar with pywinauto

I have a file selection window from an windows application I'm trying to automate.
I can change the File name box at the bottom by using
app2.window(title_re='Select a batch file') \
.File_name_Edit.set_text(selected_filename)
I have previously entered the full filepath and filename in to the lower box, however, it will no longer accept slashes in the File name box after a windows update. Hence the problem.
So the problem is the top (directory/folder name) box.
Manually I can select the box, highlight it then overtype the directory.
I cannot find a way to do this using pywinauto.
Methods attempted are:
toolbar2 = app2.window(title_re='Select a batch file') # \
# .child_window(title_re='Address', auto_id='1001')
# .child_window(title_re='Address', control_type="ToolBar", auto_id='1001')
file_path_address = toolbar2['Address band toolbarToolbar'].click_input()
file_path_address.set_text(directory_path)
# ToolbarWrapper(toolbar2).set_text(directory_path)
# ToolbarWrapper(toolbar2).click(button='All locationsSplitButton').set_text(directory_path)
# ToolbarWrapper(toolbar2)['Address:'].set_text(directory_path)
I'm using set_text as the automation must run behind a locked screen, type_keys does not work with a locked screen as type_keys is a keyboard method and Windows blocks keyboard entry when the screen is locked.
I have no access to the internals of the program being automated in order to change the original directory path and need to change it to read in from a different location.
Any assistance would be appreciated.
See below code, It runs fine with Notepad++ on Win 10
import time
from pywinauto.application import Application
app = Application().Connect(title=u'Open', class_name='#32770')
window = app.Dialog
toolbarwindow = window.Toolbar3
toolbarwindow.Click()
time.sleep(0.30)
static = window.Edit2
directory_path = "C:\Users\Desktop"
raw_directory_path = r'{}'.format(directory_path)
static.set_edit_text(text = raw_directory_path)
static.type_keys("{ENTER}")
you can use right click then edit the field
toolbar2['Address band toolbarToolbar'].click_input(button="right")
app2.PopupMenu.child_window(title="Edit address", control_type="MenuItem").click_input()
app2.child_window(title="Address", control_type="Edit").set_edit_text(directory_path)

PYTHON : Saving multiple images into folder using requests

I need to save all 6 images into a local folder.
The script I found re-writes a single files multiple times and end up producing only 1 image
import requests
img_list = ["https://ae01.alicdn.com/kf/HTB1tT70vhuTBuNkHFNRq6A9qpXa3.jpg", "https://ae01.alicdn.com/kf/HTB12HGkvwKTBuNkSne1q6yJoXXaR.jpg", "https://ae01.alicdn.com/kf/HTB1_yDic56guuRjy0Fmq6y0DXXaY.jpg", "https://ae01.alicdn.com/kf/HTB1RopgXffsK1RjSszgq6yXzpXa5.jpg", "https://ae01.alicdn.com/kf/HTB1R6sJXgHqK1RjSZFkq6x.WFXaF.jpg", "https://ae01.alicdn.com/kf/HTB1_XlhXojrK1RkHFNRq6ySvpXaR.jpg"]
for x in blob:
with open('/Users/reezalaq/PycharmProjects/wholesale/img/pic1.jpg', 'wb') as handle:
response = requests.get(x, stream=True)
if not response.ok:
print(response)
for block in response.iter_content(1024):
if not block:
break
handle.write(block)```
It needs to save all 6 images separately. No error message so far.
The script rewrites the same file each time because you're using the same file name, it never changes.
The problem is here:
with open('/Users/reezalaq/PycharmProjects/wholesale/img/pic1.jpg', 'wb')
The first argument of the open() method is the file path. The second argument is the mode, which you have set to wb, or write/binary. So in you're loop you are rewriting the file contents of pic1.jpg everytime. (See: https://docs.python.org/3.5/library/functions.html#open).
You can pre-define a list of filenames in a list and use those as the filenames or do something more dynamic like :
for img in img_list:
file_name = img.split('/')[-1]
with open(file_name, 'wb') as handle:
....
This would grab the file name of the image from the website you're downloading it from (e.g., 1HTB1tT70vhuTBuNkHFNRq6A9qpXa3.jpg for the first URL) to be used as the file name on your system. (Note: this also assumes names will be unique).
Edit:
You can define your folder path before the for-loop. Then, you can change the open() method to include that path. So:
import os # do this at the top of your file
folder_path = '/Users/reezalaq/PycharmProjects/wholesale/img/'
for img in img_list:
with open(os.path.join(folder_path, file_name), 'wb') as handle:
....

S3: How to do a partial read / seek without downloading the complete file?

Although they resemble files, objects in Amazon S3 aren't really "files", just like S3 buckets aren't really directories. On a Unix system I can use head to preview the first few lines of a file, no matter how large it is, but I can't do this on a S3. So how do I do a partial read on S3?
S3 files can be huge, but you don't have to fetch the entire thing just to read the first few bytes. The S3 APIs support the HTTP Range: header (see RFC 2616), which take a byte range argument.
Just add a Range: bytes=0-NN header to your S3 request, where NN is the requested number of bytes to read, and you'll fetch only those bytes rather than read the whole file. Now you can preview that 900 GB CSV file you left in an S3 bucket without waiting for the entire thing to download. Read the full GET Object docs on Amazon's developer docs.
The AWS .Net SDK only shows only fixed-ended ranges are possible (RE: public ByteRange(long start, long end) ). What if I want to start in the middle and read to the end? An HTTP range of Range: bytes=1000- is perfectly acceptable for "start at 1000 and read to the end" I do not believe that they have allowed for this in the .Net library.
get_object api has arg for partial read
s3 = boto3.client('s3')
resp = s3.get_object(Bucket=bucket, Key=key, Range='bytes={}-{}'.format(start_byte, stop_byte-1))
res = resp['Body'].read()
Using Python you can preview first records of compressed file.
Connect using boto.
#Connect:
s3 = boto.connect_s3()
bname='my_bucket'
self.bucket = s3.get_bucket(bname, validate=False)
Read first 20 lines from gzip compressed file
#Read first 20 records
limit=20
k = Key(self.bucket)
k.key = 'my_file.gz'
k.open()
gzipped = GzipFile(None, 'rb', fileobj=k)
reader = csv.reader(io.TextIOWrapper(gzipped, newline="", encoding="utf-8"), delimiter='^')
for id,line in enumerate(reader):
if id>=int(limit): break
print(id, line)
So it's an equivalent of a following Unix command:
zcat my_file.gz|head -20

Phonon's VolumeSlider doesn't change the volume

I'm trying to create a VolumeSlider widget that changes the volume of my audio output.
log.debug("Starting audio player (%s)..." % url)
mediaSource = Phonon.MediaSource(url)
mediaSource.setAutoDelete(True)
self.player = Phonon.createPlayer(Phonon.VideoCategory, mediaSource)
self.player.setTickInterval(100)
self.player.tick.connect(self.updatePlayerLength)
self.mediaSlider.setMediaObject(self.player)
audioOutput = Phonon.AudioOutput(Phonon.MusicCategory)
Phonon.createPath(self.player, audioOutput)
self.mediaVolumeSlider.setAudioOutput(audioOutput)
self.player.play()
However even though I can move the volume slider, the actual volume doesn't change. What did I miss?
I have never used Phonon.createPlayer, because the API seems totally baffling. Apparently, it is supposed to be a "convenience" function that creates a path between an media object and an audio output. But it only gives a reference to the media object. There appears to be no access to the audio output object, which would seem to make it completely useless (but I may well be missing something).
Anyway, I think it is much more convenient to create the paths explicitly, so that it is clear how all the parts are connected together.
The following code works for me on Linux and WinXP:
self.media = Phonon.MediaObject(self)
self.video = Phonon.VideoWidget(self)
self.audio = Phonon.AudioOutput(Phonon.VideoCategory, self)
Phonon.createPath(self.media, self.audio)
Phonon.createPath(self.media, self.video)
self.slider = Phonon.VolumeSlider(self)
self.slider.setAudioOutput(self.audio)

Resources