I am using Octave 5.1.0 on Windows 10 (x64). I am parsing a series of directories looking for an Excel spreadsheet in each directory with "logbook" in its filename. The problem is these files are created by hand and the filenaming isn't consistent: sometimes it's "LogBook", other times it's "logbook", etc...
It looks like the string passed as input to the dir function is case-sensitive so if I don't have the correct case, dir returns an empty struct. Currently, I am using the following workaround, but I wondered if there was a better way of doing this (for a start I haven't captured all possible upper/lower case combinations):
logbook = dir('*LogBook.xls*');
if isempty(logbook)
logbook = dir('*logbook.xls*');
if isempty(logbook)
logbook = dir('*Logbook.xls*');
if isempty(logbook)
logbook = dir('*logBook.xls*');
if isempty(logbook)
error(['Could not find logbook spreadsheet in ' dir_name '.'])
end
end
end
end
You need to get the list of filenames (either via readdir, dir, ls), and then search for the string in that list. If you use readdir, it can be done like this:
[files, err, msg] = readdir ('.'); # read current directory
if (err != 0)
error ("failed to readdir (error code %d): %s", msg);
endif
logbook_indices = find (cellfun (#any, regexpi (files, 'logbook'));
logbook_filenames = files(logbook_indices);
A much less standard approach could be:
glob ('*[lL][oO][gG][bB][oO][kK]*')
Related
I'm trying to migrate a codebase from PyQt5 to PyQt6. I read in this article (see https://www.pythonguis.com/faq/pyqt5-vs-pyqt6/) that all enum members must be named using their fully qualified names. The article gives this example:
# PyQt5
widget = QCheckBox("This is a checkbox")
widget.setCheckState(Qt.Checked)
# PyQt6
widget = QCheckBox("This is a checkbox")
widget.setCheckState(Qt.CheckState.Checked)
Then the article continues:
"There are too many updated values to mention them all here. But if you're converting a codebase you can usually just search online for the short-form and the longer form will be in the results."
I get the point. This quote basically says something along the lines:
"If the Python interpreter runs into an error, and the error turns out to be a short-form enum, you'll likely find the solution online."
I get that. But this is not how I want to migrate the codebase. I want a full list of all the short-form enums and then perform a global search-and-replace for each.
Where can I find such a list?
I wrote a script to extract all the short-form and corresponding fully qualified enum names from the PyQt6 installation. It then does the conversions automatically:
# -*- coding: utf-8 -*-
# ================================================================================================ #
# ENUM CONVERTER TOOL #
# ================================================================================================ #
from typing import *
import os, argparse, inspect, re
q = "'"
help_text = '''
Copyright (c) 2022 Kristof Mulier
MIT licensed, see bottom
ENUM CONVERTER TOOL
===================
The script starts from the toplevel directory (assuming that you put this file in that directory)
and crawls through all the files and folders. In each file, it searches for old-style enums to
convert them into fully qualified names.
HOW TO USE
==========
Fill in the path to your PyQt6 installation folder. See line 57:
pyqt6_folderpath = 'C:/Python39/Lib/site-packages/PyQt6'
Place this script in the toplevel directory of your project. Open a terminal, navigate to the
directory and invoke this script:
$ python enum_converter_tool.py
WARNING
=======
This script modifies the files in your project! Make sure to backup your project before you put this
file inside. Also, you might first want to do a dry run:
$ python enum_converter_tool.py --dry_run
FEATURES
========
You can invoke this script in the following ways:
$ python enum_converter_tool.py No parameters. The script simply goes through
all the files and makes the replacements.
$ python enum_converter_tool.py --dry_run Dry run mode. The script won't do any replace-
ments, but prints out what it could replace.
$ python enum_converter_tool.py --show Print the dictionary this script creates to
convert the old-style enums into new-style.
$ python enum_converter_tool.py --help Show this help info
'''
# IMPORTANT: Point at the folder where PyQt6 stub files are located. This folder will be examined to
# fill the 'enum_dict'.
pyqt6_folderpath = 'C:/Python39/Lib/site-packages/PyQt6'
# Figure out where the toplevel directory is located. We assume that this converter tool is located
# in that directory. An os.walk() operation starts from this toplevel directory to find and process
# all files.
toplevel_directory = os.path.realpath(
os.path.dirname(
os.path.realpath(
inspect.getfile(
inspect.currentframe()
)
)
)
).replace('\\', '/')
# Figure out the name of this script. It will be used later on to exclude oneself from the replace-
# ments.
script_name = os.path.realpath(
inspect.getfile(inspect.currentframe())
).replace('\\', '/').split('/')[-1]
# Create the dictionary that will be filled with enums
enum_dict:Dict[str, str] = {}
def fill_enum_dict(filepath:str) -> None:
'''
Parse the given stub file to extract the enums and flags. Each one is inside a class, possibly a
nested one. For example:
---------------------------------------------------------------------
| class Qt(PyQt6.sip.simplewrapper): |
| class HighDpiScaleFactorRoundingPolicy(enum.Enum): |
| Round = ... # type: Qt.HighDpiScaleFactorRoundingPolicy |
---------------------------------------------------------------------
The enum 'Round' is from class 'HighDpiScaleFactorRoundingPolicy' which is in turn from class
'Qt'. The old reference style would then be:
> Qt.Round
The new style (fully qualified name) would be:
> Qt.HighDpiScaleFactorRoundingPolicy.Round
The aim of this function is to fill the 'enum_dict' with an entry like:
enum_dict = {
'Qt.Round' : 'Qt.HighDpiScaleFactorRoundingPolicy.Round'
}
'''
content:str = ''
with open(filepath, 'r', encoding='utf-8', newline='\n', errors='replace') as f:
content = f.read()
p = re.compile(r'(\w+)\s+=\s+\.\.\.\s+#\s*type:\s*([\w.]+)')
for m in p.finditer(content):
# Observe the enum's name, eg. 'Round'
enum_name = m.group(1)
# Figure out in which classes it is
class_list = m.group(2).split('.')
# If it belongs to just one class (no nesting), there is no point in continuing
if len(class_list) == 1:
continue
# Extract the old and new enum's name
old_enum = f'{class_list[0]}.{enum_name}'
new_enum = ''
for class_name in class_list:
new_enum += f'{class_name}.'
continue
new_enum += enum_name
# Add them to the 'enum_dict'
enum_dict[old_enum] = new_enum
continue
return
def show_help() -> None:
'''
Print help info and quit.
'''
print(help_text)
return
def convert_enums_in_file(filepath:str, dry_run:bool) -> None:
'''
Convert the enums in the given file.
'''
filename:str = filepath.split('/')[-1]
# Ignore the file in some cases
if any(filename == fname for fname in (script_name, )):
return
# Read the content
content:str = ''
with open(filepath, 'r', encoding='utf-8', newline='\n', errors='replace') as f:
content = f.read()
# Loop over all the keys in the 'enum_dict'. Perform a replacement in the 'content' for each of
# them.
for k, v in enum_dict.items():
if k not in content:
continue
# Compile a regex pattern that only looks for the old enum (represented by the key of the
# 'enum_dict') if it is surrounded by bounds. What we want to avoid is a situation like
# this:
# k = 'Qt.Window'
# k found in 'qt.Qt.WindowType.Window'
# In the situation above, k is found in 'qt.Qt.WindowType.Window' such that a replacement
# will take place there, messing up the code! By surrounding k with bounds in the regex pat-
# tern, this won't happen.
p = re.compile(fr'\b{k}\b')
# Substitute all occurences of k (key) in 'content' with v (value). The 'subn()' method re-
# turns a tuple (new_string, number_of_subs_made).
new_content, n = p.subn(v, content)
if n == 0:
assert new_content == content
continue
assert new_content != content
print(f'{q}{filename}{q}: Replace {q}{k}{q} => {q}{v}{q} ({n})')
content = new_content
continue
if dry_run:
return
with open(filepath, 'w', encoding='utf-8', newline='\n', errors='replace') as f:
f.write(content)
return
def convert_all(dry_run:bool) -> None:
'''
Search and replace all enums.
'''
for root, dirs, files in os.walk(toplevel_directory):
for f in files:
if not f.endswith('.py'):
continue
filepath = os.path.join(root, f).replace('\\', '/')
convert_enums_in_file(filepath, dry_run)
continue
continue
return
if __name__ == '__main__':
parser = argparse.ArgumentParser(
description = 'Convert enums to fully-qualified names',
add_help = False,
)
parser.add_argument('-h', '--help' , action='store_true')
parser.add_argument('-d', '--dry_run' , action='store_true')
parser.add_argument('-s', '--show' , action='store_true')
args = parser.parse_args()
if args.help:
show_help()
else:
#& Check if 'pyqt6_folderpath' exists
if not os.path.exists(pyqt6_folderpath):
print(
f'\nERROR:\n'
f'Folder {q}{pyqt6_folderpath}{q} could not be found. Make sure that variable '
f'{q}pyqt6_folderpath{q} from line 57 points to the PyQt6 installation folder.\n'
)
else:
#& Fill the 'enum_dict'
type_hint_files = [
os.path.join(pyqt6_folderpath, _filename)
for _filename in os.listdir(pyqt6_folderpath)
if _filename.endswith('.pyi')
]
for _filepath in type_hint_files:
fill_enum_dict(_filepath)
continue
#& Perform requested action
if args.show:
import pprint
pprint.pprint(enum_dict)
elif args.dry_run:
print('\nDRY RUN\n')
convert_all(dry_run=True)
else:
convert_all(dry_run=False)
print('\nQuit enum converter tool\n')
# MIT LICENSE
# ===========
# Copyright (c) 2022 Kristof Mulier
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
# associated documentation files (the "Software"), to deal in the Software without restriction, in-
# cluding without limitation the rights to use, copy, modify, merge, publish, distribute, sublicen-
# se, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to
# do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all copies or substan-
# tial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRIN-
# GEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Make sure you backup your Python project. Then place this file in the toplevel directory of the project. Modify line 57 (!) such that it points to your PyQt6 installation folder.
First run the script with the --dry_run flag to make sure you agree with the replacements. Then run it without any flags.
I'm running Julia code which generates a plot and a text file. There exists an "Output" folder in the same folder where the code in question is located. For the first run, I create a "Run_1" folder, with "Plots" and "Data" subfolders:
fig_path = #__DIR__
mkdir(fig_path*"/Output/Run_1/")
mkdir(fig_path*"/Output/Run_1/Plots/")
mkdir(fig_path*"/Output/Run_1/Data/")
After plotting, I save the figure to "Plots":
fig_name = "test_figure"
savefig(fig_path*"/Output/Run_1/Plots/"*fig_name*".pdf")
and the output file (contained within "output_matrix") is saved to "Data":
outfile_1 = fig_path*"/Output/Run_1/Data/test_data.txt"
open(outfile_1, "w") do f1
writedlm(f1,output_matrix)
end
However, I want to run this code multiple times. Each time it runs, it should create a new "Run" folder in the "Output" folder, i.e. on the first run its Run_1, the second run it's Run_2, and so on. All folders from previous runs are NOT deleted. In each Run folder, there's a "Plots" and a "Data" folder, and I save the plot and data to their respective folders in each run. How can I have Julia update the file name in such a manner?
The ispath function checks whether a file or directory exists in the filesystem.
If you want to keep a naming convention like Run_1...Run_N, something like this could help:
function mk_output_dir()
i = 1
while true
dir_name = joinpath(#__DIR__, "Output", "run_$i")
if !ispath(dir_name)
mkpath(dir_name)
return dir_name
end
i += 1
end
end
This produces:
# First run
julia> top_dir = mk_output_dir()
"/tmp/Output/run_1"
julia> mkdir(joinpath(top_dir, "Plots"))
"/tmp/Output/run_1/Plots"
julia> mkdir(joinpath(top_dir, "Data"))
"/tmp/Output/run_1/Data"
# Second run
julia> top_dir = mk_output_dir()
"/tmp/Output/run_2"
julia> mkdir(joinpath(top_dir, "Plots"))
"/tmp/Output/run_2/Plots"
julia> mkdir(joinpath(top_dir, "Data"))
"/tmp/Output/run_2/Data"
Be aware that race conditions could occur if you start two instances of your program at the same time.
Alternatively, I personally tend to use naming conventions involving timestamps when creating directory structures like this. Here would be a minimal example:
using Dates
function mk_output_dir()
timestamp = Dates.format(now(), "YYYYmmdd-HHMMSS")
dir_name = joinpath(#__DIR__, "Output", "run_$timestamp")
#assert !ispath(dir_name) "Somebody else already created the directory"
mkpath(dir_name)
return dir_name
end
which produces something like this:
julia> top_dir = mk_output_dir()
"/tmp/Output/run_20201229-210835"
julia> mkdir(joinpath(top_dir, "Plots"))
"/tmp/Output/run_20201229-210835/Plots"
julia> mkdir(joinpath(top_dir, "Data"))
"/tmp/Output/run_20201229-210835/Data"
Maybe something like this:
function mkresultdir(fig_path)
for i=1:1000
rundir = joinpath(fig_path, "run_$i")
if !isdir(rundir)
mkdir(rundir)
return rundir
end
end
error("too many results on disk, time for a cleanup!")
end
res_dir_1 = mkresultdir("/home/my_user/results")
res_dir_2 = mkresultdir("/home/my_user/results")
I've used below code which successfully compares two text files and logs the difference in a log file using for loop. The file names are in series, for example, File_1, File_2 etc.. but when there is a missing file in the series, the code stops the execution with error - No such file or directory.
Then I've used if condition to check the file existence but I am getting a below-mentioned error.
Please help me to skip comparison for a nonexisting file.
Code:
for(i in 1:length){
prod_file_res_name <- sprintf("path/Query_Prod_%s.txt", i)
beta_file_res_name <- sprintf("path/Query_Beta_%s.txt", i)
if (exists('prod_file_res_name' && 'beta_file_res_name')){
res <- tools::Rdiff(prod_file_res_name, beta_file_res_name, Log = TRUE)
if(res[2] != "character(0)"){
write(toString(res[2]), file = "LogFile.txt",append=TRUE)
}
else{
elsevar <- sprintf("No difference found between prod and beta responses for query %s", i)
print(elsevar)
}
}
}
Error:
Error in "prod_file_res_name" && "beta_file_res_name" :
invalid 'x' type in 'x && y'
exists checks if the given R objects (only takes R objects as input --> this caused your first error) exist in your environment. You initiate the R objects prod_file_res_name and beta_file_res_name before you check if they exist, so the exists call will always return TRUE. What you are looking for is the file.exists function which checks if the file does exist in your working directory:
file.exists(prod_file_res_name) && file.exists(beta_file_res_name)
The second error was caused by the R objects existing but not the files you want to check.
Since exists() looks up single objects (from Documentation):
Is an Object Defined?
Description
Look for an R object of the given name and possibly return it
'prod_file_res_name' && 'beta_file_res_name' doesn't work.
Rewrite
exists("prod_file_res_name" && "beta_file_res_name")
to:
exists("prod_file_res_name") && exists("beta_file_res_name")
I am unable to define empty IndexedTables, e.g.
using IndexedTables, IndexedTables.Table
t = Table(Columns(a=Int64[],b=String[]),Int64[])
t[1,"a"] = 1
t[1,"b"] = 2
t[1,"c"] = t[1,"a"] + t[1,"b"]
BoundsError: attempt to access 0-element Array{Int64,1} at index [0]
I am aware that creating the IndexedTable with already the data is more efficient that creating an empty one and then insert values, but sometimes you are obliged to go on this way.
Is this a bug ? If so, is there any workaround possible ?
(I already posted this thread on the Julia forum, but so far I had no replies there)
This is probably a bug in IndexedTables.
Inserting into an IndexedTable requires reindexing to access the data. Reindexing is done with flush!.
But flush!(t) fails in the example in the question with the empty t.
Fixing flush! which calls _merge! can be done by:
julia> function IndexedTables._merge!(dst::IndexedTable, src::IndexedTable, f)
if length(dst.index)==0 || isless(dst.index[end], src.index[1])
append!(dst.index, src.index)
append!(dst.data, src.data)
else
# merge to a new copy
new = _merge(dst, src, f)
ln = length(new)
# resize and copy data into dst
resize!(dst.index, ln)
copy!(dst.index, new.index)
resize!(dst.data, ln)
copy!(dst.data, new.data)
end
return dst
end
julia> t[1,"c"] = t[1,"a"] + t[1,"b"]
3
The change is the addition of the length(...) check in the first if.
Of course, a pull request / issue should be opened with IndexedTables.jl. Antonello, will you do this? (or shall I)
I need to isolate the file extension from a path in SQLite. I've read the post here (SQLite: How to select part of string?), which gets 99% there.
However, the solution:
select distinct replace(column_name, rtrim(column_name, replace(column_name, '.', '' ) ), '') from table_name;
fails if a file has no extension (i.e. no '.' in the filename), for which it should return an empty string. Is there any way to trap this please?
Note the filename in this context is the bit after the final '\'- it shouldn't be searching for'.'s in the full path, as it does at moment too.
I think it should be possible to do it using further nested rtrims and replaces.
Thanks. Yes, you can do it like this:
1) create a scalar function called "extension" in QtScript in SQLiteStudio
2) The code is as follows:
if ( arguments[0].substring(arguments[0].lastIndexOf('\u005C')).lastIndexOf('.') == -1 )
{
return ("");
}
else
{
return arguments[0].substring(arguments[0].lastIndexOf('.'));
}
3) Then, in the SQL query editor you can use
select distinct extension(PATH) from DATA
... to itemise the distinct file extensions from the column called PATH in the table called DATA.
Note that the PATH field must contain a backslash ('\') in this implementation - i.e. it must be a full path.