I've been having trouble saving lightgbm boosters after running a model with the bonsai and the tidymodels package (as in this StackOverflow thread How to save Tidymodels Lightgbm model for reuse - however, I never managed to get it to work).
Therefore, reading about it (and as it says on the update in the abovementioned StackOverflow thread) led me to find out that the development version of the lightgbm package (version 4.0 I believe) supports using saveRDS and readRDS directly for their lightgbm models.
The thing is, I don't know how to install the development version of lightgbm. Reading this thread I can see that James, who's one of the maintainers of the lightgbm package, wrote this (https://github.com/tidymodels/bonsai/issues/34):
"If you already have a local installation of LightGBM, you can re-install from the latest development version by running the following:"
REPO_DIR="${HOME}/repos"
cd "${REPO_DIR}/lgb-dev"
git checkout master
git pull origin master
git submodule update --recursive
sh build-cran-package.sh \
--no-build-vignettes
R CMD INSTALL \
--with-keep.source \
./lightgbm_*.tar.gz
My problem is that I do not understand this code. Is it supposed to be run in R? (this does not work for me). Or in the command line? Which language is this?
If anyone can help me out regarding installation of the dev version of the lightgbm package, I would be really grateful
The code you've provided is the preferred way to install the development version of {lightgbm} (the R package for LightGBM).
That can be run from any shell that understands Unix commands, as long as you have git and R installed. If you're on a Mac, you could for example use the Terminal application. If you're on a Windows machine, you could use Git for Windows.
Unfortunately, {lightgbm} cannot be installed from its git repo using R-only tools like remotes::install_github(). That is intentional, as described in https://github.com/tidymodels/bonsai/issues/34#issuecomment-1187589193.
If you are not comfortable using git and the command line, the following R-only solution could be used to install the latest development version of {lightgbm}.
library(httr)
library(jsonlite)
TEMP_DIR <- "tmp-lightgbm"
if (!dir.exists(TEMP_DIR)) {
dir.create(TEMP_DIR)
}
builds_response <- httr::RETRY(
verb = "GET"
, url = "https://dev.azure.com/lightgbm-ci/lightgbm-ci/_apis/build/builds?branchName=refs/heads/master&resultFilter=succeeded&queryOrder=finishTimeDescending&%24top=1&api-version=7.1-preview.7"
)
build_id <- jsonlite::fromJSON(
txt = httr::content(builds_response, as = "text")
, simplifyDataFrame = FALSE
)[["value"]][[1L]][["id"]]
artifact_url <- paste0(
"https://dev.azure.com/lightgbm-ci/lightgbm-ci/_apis/build/builds/"
, build_id
, "/artifacts?artifactName=R-package&api-version=7.1-preview.5&%24format=zip"
)
download.file(
url = artifact_url
, destfile = file.path(TEMP_DIR, "lightgbm-r.zip")
)
unzip(
zipfile = file.path(TEMP_DIR, "lightgbm-r.zip")
, exdir = file.path(TEMP_DIR, "extracted")
)
artifacts_dir <- file.path(TEMP_DIR, "extracted", "R-package")
tarball_path <- list.files(
path = artifacts_dir
, pattern = "lightgbm-[0-9.]+-r-cran\\.tar\\.gz"
, full.names = TRUE
)[[1L]]
new_tarball_path <- file.path(artifacts_dir, "lightgbm.tar.gz")
file.rename(tarball_path, new_tarball_path)
install.packages(new_tarball_path, repos = NULL, type = "source")
I'm using the JuliaCall library to use Julia code in R. I'm trying to increase the number of threads but I can't seem to succeed.
I'm using Windows 10 and RStudio version 1.4.1103
Thank you so much!
Simply run
Sys.setenv(JULIA_NUM_THREADS = "8")
before calling library(JuliaCall).
Here is a full console output on my machine:
> Sys.setenv(JULIA_NUM_THREADS = "8")
> library(JuliaCall)
> julia_command("Threads.nthreads()")
Julia version 1.6.0 at location C:\JULIA-~1.0\bin will be used.
Loading setup script for JuliaCall...
Finish loading setup script for JuliaCall.
8
I tried to compute a small example with the library SimpleHypergraphs. I followed the install instructions however, I have this error :
ERROR: "HyperNetX is not installed in Python used by this Julia. Install HyperNetX and reload SimpleHypergraphs.jl"
Here's my code:
using PyCall
using Conda
Conda.runconda(`install matplotlib --yes`)
Conda.runconda(`install networkx --yes`)
run(`$(PyCall.python) -m pip install hypernetx`)
using SimpleHypergraphs
h = Hypergraph{Float64}(5,4)
h[1:3,1] .= 1.5
h[3,4] = 2.5
h[2,3] = 3.5
h[4,3:4] .= 4.5
h[5,4] = 5.5
h[5,2] = 6.5
draw(h, HyperNetX; width=5, height=5)
I don't understand why this error occurs since hypernetx is well installed.
It seems that pandas has been added to the dependencies of hypernetx and now it will not load unless pandas is available.
Restart your Julia session and run
using Conda
Conda.runconda(`install pandas --yes`)
Now it will work as expected:
draw(h, HyperNetX; width=5, height=5)
Thanks for posting that out! - I will add it to the README of SimpleHypergraphs.
Finally, note that SimpleHypergraphs uses the Python only for the last step - plotting. All processing/computations always take part in Julia so do not worry about all those Python libraries.
I am receiving this issue while trying to use notebook() in Julia (Command Line Interface REPL)
julia> using IJulia
[ Info: Precompiling IJulia [7073ff75-c697-5162-941a-fcdaad2a7d2a]
julia> notebook()
[ Info: running setenv(`/root/miniconda3/bin/jupyter notebook`,["_CE_M=", "LSCOLORS=Gxfxcxdxbxegedabagacad", "PATH=/root/miniconda3/bin:/root/miniconda3/bin:/root/miniconda3/condabin:/bin:/usr/bin:/sbin:/usr/sbin:/root/julia-1.3.1/bin", "QT_ACCESSIBILITY=1", "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus", "XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0", "XDG_SESSION_DESKTOP=GNOME", "SSH_AGENT_PID=2661", "XDG_SESSION_TYPE=x11", "QT_IM_MODULE=ibus", "USER=root", "PAM_KWALLET5_LOGIN=/run/user/0/kwallet5.socket", "LESS=-R", "CONDA_PROMPT_MODIFIER=(base) ", "XDG_DATA_DIRS=/usr/share/gnome:/root/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share", "CLUTTER_IM_MODULE=ibus", "SHELL=/usr/bin/zsh", "GJS_DEBUG_OUTPUT=stderr", "XAUTHORITY=/root/.Xauthority", "XDG_MENU_PREFIX=gnome-", "GTK_MODULES=gail:atk-bridge", "GTK_IM_MODULE=ibus", "MANAGERPID=2575", "SESSION_MANAGER=local/kali:#/tmp/.ICE-unix/2753,unix/kali:/tmp/.ICE-unix/2753", "XMODIFIERS=#im=ibus", "_CE_CONDA=", "CONDA_SHLVL=1", "CONDA_EXE=/root/miniconda3/bin/conda", "HOME=/root", "TERM=xterm-256color", "COLORTERM=truecolor", "VTE_VERSION=5802", "CONDA_PYTHON_EXE=/root/miniconda3/bin/python", "INVOCATION_ID=a29cdf2f60b140acb65b8e9d69af5b77", "IM_CONFIG_PHASE=1", "XDG_CURRENT_DESKTOP=GNOME", "LANG=C.UTF-8", "LOGNAME=root", "GNOME_DESKTOP_SESSION_ID=this-is-deprecated", "XDG_RUNTIME_DIR=/run/user/0", "SHLVL=1", "SSH_AUTH_SOCK=/run/user/0/keyring/ssh", "DESKTOP_SESSION=/usr/share/xsessions/gnome-xorg", "OLDPWD=/root", "_=/root/julia-1.3.1/bin/julia", "PAGER=less", "CONDA_DEFAULT_ENV=base", "GPG_AGENT_INFO=/run/user/0/gnupg/S.gpg-agent:0:1", "GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/73a7f463_14d7_4c04_adfb_a849e0d337b9", "ZSH=/root/.oh-my-zsh", "GNOME_TERMINAL_SERVICE=:1.261", "PWD=/root/JuliaPro-1.2.0-2", "XDG_SESSION_CLASS=user", "DISPLAY=:0", "GJS_DEBUG_TOPICS=JS ERROR;JS LOG", "XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session1", "CONDA_PREFIX=/root/miniconda3", "JOURNAL_STREAM=9:29944", "QT4_IM_MODULE=ibus", "LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:", "OPENBLAS_MAIN_FREE=1"])
Process(setenv(`/root/miniconda3/bin/jupyter notebook`,["_CE_M=", "LSCOLORS=Gxfxcxdxbxegedabagacad", "PATH=/root/miniconda3/bin:/root/miniconda3/bin:/root/miniconda3/condabin:/bin:/usr/bin:/sbin:/usr/sbin:/root/julia-1.3.1/bin", "QT_ACCESSIBILITY=1", "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus", "XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0", "XDG_SESSION_DESKTOP=GNOME", "SSH_AGENT_PID=2661", "XDG_SESSION_TYPE=x11", "QT_IM_MODULE=ibus" … "PWD=/root/JuliaPro-1.2.0-2", "XDG_SESSION_CLASS=user", "DISPLAY=:0", "GJS_DEBUG_TOPICS=JS ERROR;JS LOG", "XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session1", "CONDA_PREFIX=/root/miniconda3", "JOURNAL_STREAM=9:29944", "QT4_IM_MODULE=ibus", "LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:", "OPENBLAS_MAIN_FREE=1"]; dir="/root"), ProcessExited(1))
This is very confusing and likely a beginner issue.
As suggested by #SalchiPapa , here is my versioninfo:
Julia Version 1.3.1
Commit 2d5741174c (2019-12-30 21:36 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Core(TM) i3-6006U CPU # 2.00GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Also, the jupyter notebook starts up the localhost for me but kernel keeps having errors for julia-1.3.1 . This doesn't work for me as it shows this error : ERROR: LoadError: Blosc not installed properly, run Pkg.build("ZMQ"), restart Julia and try again and this one at the same time : ERROR: LoadError: Failed to precompile ZMQ [c2297ded-f4af-51ae-bb23-16f91089e4e1] to /root/.julia/compiled/v1.3/ZMQ/CaaO6_riTu7.ji.
Manually tried to Pkg.add and Pkg.build each of ZMQ, Blosc and CMake but to no success.
Finally found a solution. Much thanks to #SalchiPapa who took his time out to personally videocall me and solve the issue. Solution right now is to not run IJulia from root
What worked for me :
$su user-not-root
$julia
using IJulia;notebook(detached=true)
This should show ProcessRunning at the end.
run(path/to/jupyter notebook list) with pair of backticks inside run().
This gave me the url and token appended to it . That worked out well.
I suggest that the community help solve this issue.
I have linked this answer to a Github issue here
I want to use Base.compilecache.
It needs PkgId(UUID). But I don't know how to get installed Pkg.uuid in REPL.
https://docs.julialang.org/en/v1.0.0/base/base/#Base.compilecache
I tried Pkg.PackageSpec but can't get UUID.
https://docs.julialang.org/en/stable/stdlib/Pkg/#Pkg.PackageSpec
julia> using Pkg
julia> Pkg.status()
Status `~/.julia/environments/v1.0/Project.toml`
[8f4d0f93] Conda v1.0.1
[7073ff75] IJulia v1.9.3
julia> Pkg.PackageSpec("Conda")
PackageSpec(name=Conda)
julia> Pkg.PackageSpec("Conda").uuid
UUID("00000000-0000-0000-0000-000000000000")
My goal is to call Base.compilecache at Docker file.
Julia v0.6.4 code
julia -e "Base.compilecache(\"JSON\")"
You can use Base.identify_package("Conda").