Get system memory information from julia - julia

Is there a nice way to get the current system information in julia (my use case here is memory but also interested in basically anything information that I could get from running top on linux).
This is what I have at the moment: (basically just getting the output of `free -m`)<- I can't get this to let me escape backticks and keep code highlighting...
import Base.DataFmt: readdlm_string, invalid_dlm
"""
getmeminfo()
Returns (in MB) A tuple of containing:
- Memory(total, used, buffer, available)
- Swap(total, used, free)
"""
function getmeminfo()
memstats = readdlm_string(readstring(`free -m`),invalid_dlm(Char), Int, '\n', true, Dict())
return Tuple{Array{Int,1},Array{Int,1}}((memstats[2,[2;3;6;7]], memstats[3,[2;3;4]]))
end
Is there something in Base or any better ideas?

The built-in Sys module contains functions dedicated to retrieving system information.
julia> VERSION
v"1.0.0"
julia> Sys.total_memory() / 2^20
8071.77734375
julia> Sys.free_memory() / 2^20
5437.46484375
julia> Sys.CPU_NAME
"haswell"
julia> Sys.
ARCH KERNEL WORD_SIZE eval isexecutable set_process_title
BINDIR MACHINE __init__ free_memory islinux total_memory
CPU_NAME SC_CLK_TCK _cpu_summary get_process_title isunix uptime
CPU_THREADS STDLIB _show_cpuinfo include iswindows which
CPUinfo UV_cpu_info_t cpu_info isapple loadavg windows_version
JIT WINDOWS_VISTA_VER cpu_summary isbsd maxrss
julia> # Above after pressing Tab key twice
While it does not support all of the information provided by top, it will hopefully provide the information you are looking for.

Related

How to avoid RuntimeError while call __dict__ on module?

it is appearing in some big modules like matplotlib. For example expression :
import importlib
obj = importlib.import_module('matplotlib')
obj_entries = obj.__dict__
Between runs len of obj_entries can vary. From 108 to 157 (expected) entries. Especially pyplot can be ignored like some another submodules.
it can work stable during manual debug mode with len computing statement after dict extraction. But in auto it dont work well.
such error occures:
RuntimeError: dictionary changed size during iteration
python-BaseException
using clear python 3.10 on windows. Version swap change nothing at all
during some attempts some interesting features was found.
use of repr is helpfull before dict initiation.
But if module transported between classes like variable more likely lazy-import happening? For now there is evidence that not all names showing when command line interpriter doing opposite - returning what expected. So this junk of code help bypass this bechavior...
Note: using pkgutil.iter_modules(some_path) to observe modules im internal for pkgutil ModuleInfo form.
import pkgutil, importlib
module_info : pkgutil.ModuleInfo
name = module_info.name
founder = module_info.module_finder
spec = founder.find_spec(name)
module_obj = importlib.util.module_from_spec(spec)
loader = module_obj.__loader__
loader.exec_module(module_obj)
still unfamilliar with interior of import mechanics so it will be helpfull to recive some links to more detail explanation (spot on)

How to open an SQLite database readonly in Julia?

I'd like to read my Safari history database from a Julia script (Mac OS X).
I have a command line script that works:
sqlite3 -readonly ~/Library/Safari/History.db 'SELECT v.title, i.url FROM history_items i, history_visits v WHERE i.url LIKE "%en.wikipedia.org%" AND i.id=v.history_item AND v.title LIKE "%- Wikipedia%" GROUP BY v.title ORDER BY v.visit_time'
... but trying it in Julia (in Juno / Atom) gives me a permission error
db = SQLite.DB("/Users/grimxn/Library/Safari/History.db")
sql = """
SELECT v.title, i.url, v.visit_time
FROM history_items i, history_visits v
WHERE i.url LIKE "%en.wikipedia.org%"
AND i.id=v.history_item
AND v.title LIKE "%- Wikipedia%"
GROUP BY v.title
ORDER BY v.visit_time
"""
result = DBInterface.execute(db, sql) |> DataFrame
(rows, cols) = size(result)
println("Result has $(rows) rows")
println("Earliest: $(result[1,1])")
println("Latest: $(result[rows,1])")
ERROR: LoadError: SQLite.SQLiteException("unable to open database file")
Now, when I copy the database to my home directory, and swap
db = SQLite.DB("/Users/grimxn/Library/Safari/History.db")
to
db = SQLite.DB("/Users/grimxn/History.db")
everything works, so I guess it is that the Julia / Juno process has only got read permissions, but is accessing the db read/write.
How do I attach to the database as readonly in Julia?
Theoretically, use a URI connection string: file:foo.db?mode=ro.
This is documented in the SQLite manual.
Practically, it appears the current version of the SQLite.jl package does not support URIs, and neither does it support flags that could be passed along to sqlite3_open_v2().
Leaving this answer for reference just in case the Julia package fixes this some day.
Jaen's answer was correct, and also correctly predicted that the mode=ro flag would be supported. It is now supported, and so the following will work (and does as of today):
julia> using SQLite
julia> db = SQLite.DB("file:/path/to/db.sqlite?mode=ro")
SQLite.DB("file:/path/to/db.sqlite?mode=ro")

Sqlite (within SqliteStudio): invalid command name "parray"

I am discovering writing functions in TCL for Sqlite (https://github.com/pawelsalawa/sqlitestudio/wiki/ScriptingTcl).
I wanted to play a basic exemple found in the official page of sqlite(http://sqlite.org/tclsqlite.html):
db eval {SELECT * FROM MyTable ORDER BY MyID} values {
parray values
puts ""
}
I get the following error:
Error while requesting the database « -- » : invalid command name "parray"
Help is very welcome :)
SqliteStudio does not seem to fully initialise Tcl, as you would expect it from a non-embedded installation:
Using external Tcl packages or modules is not possible, because Tcl
interpreters are not initialized with "init.tcl".
See Wiki.
Background
Standard Tcl sources init.tcl, early as part of an Tcl interpreter's initialisation. init.tcl, in turn, registers a number of Tcl procs for autoloading. parray is one of those lazily acquired procs.
Ways forward
I am not familiar with SqliteStudio. Why not stick with sqlite's standard Tcl frontend, which gives you full Tcl and comes with Tcl distributions free house? But this certainly depends on your requirements.
That said, you could attempt to force-load init.tcl in SqliteStudio's embedded Tcl, but I don't know (and can't test) whether the distribution has not pruned these scripts or whether they were effectively relocated. From the top of my head (untested):
source [file join $tcl_library init.tcl]
# ...
db eval {SELECT * FROM MyTable ORDER BY MyID} values {
parray values
puts ""
}

Openmdao 1.7.3 error with unicode variables in python2

In the file openmdao/core/problem.py on lines such as 1619 and 1638, it checks if a variable is a string by using:
isinstance(inp, str)
however, this will return false if inp is unicode in python2, and eventually cause the program to raise an exception. In python2, the correct syntax is:
isinstance(inp, basestring)
I understand that basestring is not available in python 3, but there are several ways to write python 2/3 compatible code. Can this be fixed?
feel free to submit a pull request, but please add a test that checks the new functionality

32-bit pointer overflow in 64-bit gcc code - fails in compile

I am compiling a very large legacy Fortran 90 code (screamer) with gFortran on a Mac (2.2 GHz Intel Core i7) running Yosemite. (gFortran V5.1.0) I have 16 GB of RAM. The code is memory intensive and I am trying to increase array sizes to solve larger problems. I have maintained the code for >10 years and rewriting 200,000 lines of code right now is not an option. As I carefully increase the size of the 2-D matrix (am(max_nodes, max_nodes)) and several 1-D vectors (RHS(max_nodes) and a(max_nodes*2)) by varying the integer "max_nodes" I eventually get to a 32-bit pointer limit (4 byte unsigned integer limit) during compilation. See below.
final section layout:
__TEXT/__text addr=0x100001390, size=0x0006B9CB, fileOffset=0x00001390, type=1
__TEXT/__text_startup addr=0x10006CD60, size=0x00000041, fileOffset=0x0006CD60, type=1
__TEXT/__text_exit addr=0x10006CDB0, size=0x00000031, fileOffset=0x0006CDB0, type=1
__TEXT/__stubs addr=0x10006CDE2, size=0x00000252, fileOffset=0x0006CDE2, type=28
__TEXT/__stub_helper addr=0x10006D034, size=0x000003EE, fileOffset=0x0006D034, type=32
__TEXT/__cstring addr=0x10006D428, size=0x0000CFCB, fileOffset=0x0006D428, type=13
__TEXT/__const addr=0x10007A400, size=0x00008F00, fileOffset=0x0007A400, type=0
__TEXT/__eh_frame addr=0x100083300, size=0x0000DCF8, fileOffset=0x00083300, type=19
__DATA/__got addr=0x100091000, size=0x00000060, fileOffset=0x00091000, type=29
__DATA/__nl_symbol_ptr addr=0x100091060, size=0x00000010, fileOffset=0x00091060, type=29
__DATA/__la_symbol_ptr addr=0x100091070, size=0x00000318, fileOffset=0x00091070, type=27
__DATA/__mod_init_func addr=0x100091388, size=0x00000010, fileOffset=0x00091388, type=33
__DATA/__mod_term_func addr=0x100091398, size=0x00000008, fileOffset=0x00091398, type=34
__DATA/__const addr=0x1000913A0, size=0x000007C8, fileOffset=0x000913A0, type=0
__DATA/__static_data addr=0x100091B68, size=0x00000003, fileOffset=0x00091B68, type=0
__DATA/__data addr=0x100091B80, size=0x000003E0, fileOffset=0x00091B80, type=0
__DATA/__bss4 addr=0x100091F60, size=0x00000018, fileOffset=0x00000000, type=25
__DATA/__bss5 addr=0x100091F80, size=0x00020000, fileOffset=0x00000000, type=25
__DATA/__bss3 addr=0x1000B1F80, size=0x00000028, fileOffset=0x00000000, type=25
__DATA/__pu_bss2 addr=0x1000B1FA8, size=0x00000008, fileOffset=0x00000000, type=25
__DATA/__bss2 addr=0x1000B1FB0, size=0x00000024, fileOffset=0x00000000, type=25
__DATA/__pu_bss5 addr=0x1000B1FE0, size=0x0000024C, fileOffset=0x00000000, type=25
__DATA/__pu_bss4 addr=0x1000B2230, size=0x00000018, fileOffset=0x00000000, type=25
__DATA/__common addr=0x1000B2260, size=0x000020D8, fileOffset=0x00000000, type=25
__DATA/__zo_bss3 addr=0x1000B4338, size=0x00000021, fileOffset=0x00000000, type=25
__DATA/__huge addr=0x1000B4360, size=0x984EB80C, fileOffset=0x00000000, type=25
ld: 32-bit RIP relative reference out of range (2147639505 max is +/-4GB): from _main_loop_ (0x10000E120) to _a.4206 (0x180034380) in '_main_loop_' from screamer64.a(main_loop.o) for architecture x86_64
collect2: error: ld returned 1 exit status
In this error message main_loop is the core solver subroutine in screamer that populates and solves the large matrices. In this subroutine the large real*8 matrix and real*8 vectors are defined.
This Register Instruction Pointer (RIP) error is noted many times on the web. So far this available information has not helped me solve my problem. Note: the signed 4 byte integer limit is 2,147,483,647 so the error seems to be directly related to the use of a 32-bit pointer.
The gFortran compiler options include -mcmodel=medium that should take the pointers to 64 bits. -m64 has no effect. The total memory used by the primary matrix and vectors when the pointer limit is reached is greater than 2.4 GB. The confusing thing is that the code is fully 64 bit so I was not expecting 32-bit pointers. See below for 64-bit check.
rbspielman$ file screamer64
screamer64: Mach-O 64-bit executable x86_64
The primary matrix and vector are all real*8 (64-bit). All large arrays in declared directly in this one subroutine and are not placed in common.
All other variables in common are ordered by size. real*8, real, int, char.
Simple test programs demonstrate that there is no fundamental memory limit. I can easily define static arrays to > 10 GB without a problem. Larger arrays also work but end up using virtual memory and slow down as expected.
Clearly there is some sort of memory or pointer size limit but I just cannot figure it out. The code matrix solvers are massive and more realistic test programs would be tedious.
(I also compile screamer in Ubuntu LINUX without a problem up to the same array limit as the Mac. Compilations in Windows 8 fail at the usual 2 GB memory limit NOT at the pointer limit.)
Suggestions would be appreciated.
I just ran into the same problem with GNU Fortran (GCC) 5.1.0 in a Mac running 10.11.5 but the solution offered by the OP did not work for me.
However, I did find a solution: after systematically pruning my rather pedestrian legacy code, I found that every array has to be explicitly filled with something. You can't start filling it within your code. I know it sounds silly, but once I initialized every "real" array (32 bit, it is legacy code) with 0.0 before I did any I/O or other work, it linked without complaint.
And, yes, as with the OP, my code worked until I changed the size of an array.
The reason why this worked may be in the contents of this bug report: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63793 but I am not good enough to tell you how to come up with a better workaround. My only guess is that initializing every array at the beginning favors the GOT instead of the RIP. (When will this be fixed? I just don't know how to push this up the line and the bug report is dated 2014-11-09)

Resources