I followed the documentation steps butstill encountrer the following problem when using predictdemand(). Following terminal output:
`predictdemand(gisregion="Europe13", sspscenario="ssp2-26", sspyear=2050, era_year=2018)
Building training data for Europe13...
Reading auxiliary datasets...
Get current national electricity demand from IEA statistics...
Lookup SSP R5.2 region names for each GADM level 0 region...
Calculate demand multipliers...
Calculating annual electricity demand for model regions...
Progress: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| Time: 0:00:08
Reading temperature data for Europe13...
Reading auxiliary datasets...
Reading ERA5 temperature dataset...
0.476785 seconds (203 allocations: 1.499 GiB, 38.53% gc time)
Calculating population centers at ERA5 resolution...
ERROR: No population centers found for region SWE.
Stacktrace:
[1] error(s::String)
# Base .\error.jl:33
[2] GIStemp(gisregion::String, scenarioyear::String, era_year::Int64, numcenters::Int64, mindist::Float64)
# GlobalEnergyGIS C:\Users\luka.pirskawetz.julia\packages\GlobalEnergyGIS\rQD8B\src\GIStemp.jl:24
[3] buildtrainingdata(; gisregion::String, sspscenario::String, sspyear::Int64, era_year::Int64, numcenters::Int64, mindist::Float64)
# GlobalEnergyGIS C:\Users\luka.pirskawetz.julia\packages\GlobalEnergyGIS\rQD8B\src\syntheticdemand_inputdata.jl:150
[4] predictdemand(; variables::Vector{Symbol}, gisregion::String, sspscenario::String, sspyear::Int64, era_year::Int64, numcenters::Int64, mindist::Float64, nrounds::Int64, max_depth::Int64, eta::Float64, subsample::Float64, metrics::Vector{String}, more_xgoptions::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
# GlobalEnergyGIS C:\Users\luka.pirskawetz.julia\packages\GlobalEnergyGIS\rQD8B\src\syntheticdemand_training.jl:11
[5] top-level scope
# REPL[30]:1`
Related
I have a simple simulation and I want to plot errors on 3 distinct figures. To speed things up I wanted to introduce a little bit of parallel computing.
Threads.#threads for i in 1:3
plt = plot(t, err[:, i], linecolor=:blue, label=[""], linewidth=3)
hline!(plt, [0], linestyle=:dash, linecolor=:black, label=[""])
xlabel!(L"t")
ylabel!(L"e_%$i")
savefig(plt, "fig/Staubli_Slotine_Li$i.pdf")
end
Everything stops working when I try running julia with -t3 flag
ERROR: LoadError: TaskFailedException
Stacktrace:
[1] wait
# ./task.jl:334 [inlined]
[2] threading_run(func::Function)
# Base.Threads ./threadingconstructs.jl:38
[3] top-level scope
# ./threadingconstructs.jl:97
nested task error: KeyError: key :annotations not found
Stacktrace:
[1] pop!(h::Dict{Symbol, Any}, key::Symbol)
# Base ./dict.jl:587
[2] pop_kw!(dd::RecipesPipeline.DefaultsDict, k::Symbol)
# RecipesPipeline ~/.julia/packages/RecipesPipeline/F2mWY/src/utils.jl:57
[3] _update_subplot_args(plt::Plots.Plot{Plots.PGFPlotsXBackend}, sp::Plots.Subplot{Plots.PGFPlotsXBackend}, plotattributes_in::Dict{Symbol, Any}, subplot_index::Int64, remove_pair::Bool)
# Plots ~/.julia/packages/Plots/nzdhU/src/args.jl:2058
[4] _subplot_setup(plt::Plots.Plot{Plots.PGFPlotsXBackend}, plotattributes::Dict{Symbol, Any}, kw_list::Vector{Dict{Symbol, Any}})
# Plots ~/.julia/packages/Plots/nzdhU/src/pipeline.jl:277
[5] plot_setup!(plt::Plots.Plot{Plots.PGFPlotsXBackend}, plotattributes::Dict{Symbol, Any}, kw_list::Vector{Dict{Symbol, Any}})
# Plots ~/.julia/packages/Plots/nzdhU/src/pipeline.jl:138
[6] recipe_pipeline!(plt::Any, plotattributes::Any, args::Any)
# RecipesPipeline ~/.julia/packages/RecipesPipeline/F2mWY/src/RecipesPipeline.jl:87
[7] _plot!(plt::Plots.Plot, plotattributes::Any, args::Any)
# Plots ~/.julia/packages/Plots/nzdhU/src/plot.jl:208
[8] plot!(::Plots.Plot; kw::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}})
# Plots ~/.julia/packages/Plots/nzdhU/src/plot.jl:198
[9] plot!(; kw::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}})
# Plots ~/.julia/packages/Plots/nzdhU/src/plot.jl:188
[10] #xlabel!#484
# ~/.julia/packages/Plots/nzdhU/src/shorthands.jl:416 [inlined]
[11] xlabel!
# ~/.julia/packages/Plots/nzdhU/src/shorthands.jl:416 [inlined]
[12] macro expansion
# ~/Documents/studia/master_thesis/master_thesis_code/sym_scripts/Staubli_Slotine_Li.jl:29 [inlined]
[13] (::var"#88#threadsfor_fun#1"{UnitRange{Int64}})(onethread::Bool)
# Main ./threadingconstructs.jl:85
[14] (::var"#88#threadsfor_fun#1"{UnitRange{Int64}})()
# Main ./threadingconstructs.jl:52
in expression starting at /home/jcebulsk/Documents/studia/master_thesis/master_thesis_code/sym_scripts/Staubli_Slotine_Li.jl:26
If I comment out hline! the script runs without any issue.
It looks like I can't have both hline and parallel operation.
On my machine I am getting an EXCEPTION_ACCESS_VIOLATION which is even uglier. For sure in your code xlabel!(L"t") and ylabel!(L"e_%$i") mutate the global state which is very bad for any kind parallelism and should be xlabel!(plt, L"t") and ylabel!(plt, L"e_%$i").
However, in many scenarios this might still not work because Plots.jl is also maintaining it's global state and it might be not thread safe. Hence the best way to go forward is through distributed computing:
using Distributed
addprocs(3)
#everywhere using Plots, LaTeXStrings
Distributed.#distributed for i in 1:3
plt = plot(t, err[:, i], linecolor=:blue, label=[""], linewidth=3)
hline!(plt, [0], linestyle=:dash, linecolor=:black, label=[""])
xlabel!(L"t")
ylabel!(L"e_%$i")
savefig(plt, "fig/Staubli_Slotine_Li$i.pdf")
end
Note that you will have a better performance if the plots are very complex as each of those subprocesses will experience the "Julia time to first plot" issue.
Hence before going multi-process you might want to parametrize the GR backend and see if the performance is sufficient:
using Plots
ENV["GKSwstype"]="nul" # this parameter significantly speeds up generation of GR-based plots
gr()
I am trying to figure out how to bin/histogram an array of data in Julia. I have an array of units from the Unitful.jl package and want to use Histogram from StatsBase to bin the data. The first error I got was an error saying there were no methods for log10 to use Unitful.FreeUnits so I wrote one. Now I get a dimensional error. I thought Unitful was just going to work with other stuff.. guess not. The following is where I am at.
using Unitful
using StatsBase
data = [rand()*100*1u"MHz" for x in 1:10000]
function Base.log10(x::Quantity{})
u = unit(x)
return log10(x.val)u
end
# eventually I want to define my bin width manually.. but this is a start.
fit(Histogram, data)
ERROR
ERROR: DimensionError: 0.0 and 0.8237981449864736 MHz are not dimensionally compatible.
Stacktrace:
[1] _lt at /home/mcamp/.julia/packages/Unitful/1t88N/src/quantities.jl:274 [inlined]
[2] <(::Quantity{Float64,NoDims,Unitful.FreeUnits{(),NoDims,nothing}}, ::Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}}) at /home/mcamp/.julia/packages/Unitful/1t88N/src/quantities.jl:264
[3] <(::Int64, ::Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}}) at /home/mcamp/.julia/packages/Unitful/1t88N/src/quantities.jl:266
[4] <=(::Int64, ::Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}}) at ./operators.jl:326
[5] >=(::Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}}, ::Int64) at ./operators.jl:350
[6] histrange(::Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}}, ::Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}}, ::Int64, ::Symbol) at /home/mcamp/.julia/packages/StatsBase/EA8Mh/src/hist.jl:51
[7] histrange(::Array{Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}},1}, ::Int64, ::Symbol) at /home/mcamp/.julia/packages/StatsBase/EA8Mh/src/hist.jl:39
[8] (::StatsBase.var"#127#128"{Symbol})(::Array{Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}},1}, ::Int64) at /home/mcamp/.julia/packages/StatsBase/EA8Mh/src/hist.jl:103
[9] map(::StatsBase.var"#127#128"{Symbol}, ::Tuple{Array{Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}},1}}, ::Tuple{Int64}) at ./tuple.jl:176
[10] histrange(::Tuple{Array{Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}},1}}, ::Tuple{Int64}, ::Symbol) at /home/mcamp/.julia/packages/StatsBase/EA8Mh/src/hist.jl:102
[11] fit(::Type{Histogram{Int64,N,E} where E where N}, ::Tuple{Array{Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}},1}}; closed::Symbol, nbins::Int64) at /home/mcamp/.julia/packages/StatsBase/EA8Mh/src/hist.jl:332
[12] fit(::Type{Histogram{Int64,N,E} where E where N}, ::Array{Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}},1}; closed::Symbol, nbins::Int64) at /home/mcamp/.julia/packages/StatsBase/EA8Mh/src/hist.jl:276
[13] fit(::Type{Histogram{Int64,N,E} where E where N}, ::Array{Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}},1}) at /home/mcamp/.julia/packages/StatsBase/EA8Mh/src/hist.jl:276
[14] fit(::Type{Histogram}, ::Array{Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}},1}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/mcamp/.julia/packages/StatsBase/EA8Mh/src/hist.jl:383
[15] fit(::Type{Histogram}, ::Array{Quantity{Float64,𝐓^-1,Unitful.FreeUnits{(MHz,),𝐓^-1,nothing}},1}) at /home/mcamp/.julia/packages/StatsBase/EA8Mh/src/hist.jl:383
[16] top-level scope at REPL[183]:1
[17] run_repl(::REPL.AbstractREPL, ::Any) at /build/julia/src/julia-1.5.3/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:288
[EDITED because I misunderstood the question]
Are you looking to plot a histogram of the data? If yes, instead of StatsBase.jl, I would use the histogram function of a plotting package like Plots.jl. For instance, to combine plots from Plots.jl with units from Unitful.jl, you can use the UnitfulRecipes.jl package. See this MWE that may be what you are after:
using Unitful: MHz
using Plots
using UnitfulRecipes
data = 100 * exp.(randn(10000)) * MHz
histogram(data)
will output
[BEFORE EDIT]
I don't think it makes sense mathematically or physically to take the logarithm of a non-nondimensional variable (i.e., a variable with a unit). That is, your redefinition of log10 is not a good idea IMHO. Instead, I would non-dimensionalize the data before taking the log, with something similar to
using Unitful: MHz
using StatsBase
data = 100 * rand(10000) * MHz
data_nodim = log10.(data / MHz) # <- this is valid
fit(Histogram, data_nodim) # <- this is valid too
I'm learning julia from https://docs.juliaplots.org/latest/basics/
Here is the julia code
using Plots,UnicodePlots,RDatasets
unicodeplots()
v=dataset("Ecdat","Airline")
typeof(v)
plot(v, :Cost) # this line will produce the error
plot(v) # this line will produce the error, too
I got
ERROR: Cannot convert DataFrame to series data for plotting
Stacktrace:
[1] prepareSeriesData(::DataFrame) at /home/dlin/.julia/packages/Plots/qZHsp/src/series.jl:14
[2] convertToAnyVector(::DataFrame, ::Dict{Symbol,Any}) at /home/dlin/.julia/packages/Plots/qZHsp/src/series.jl:26
[3] macro expansion at /home/dlin/.julia/packages/Plots/qZHsp/src/series.jl:130 [inlined]
[4] apply_recipe(::Dict{Symbol,Any}, ::Type{Plots.SliceIt}, ::Nothing, ::DataFrame, ::Nothing) at /home/dlin/.julia/packages/RecipesBase/zBoFG/src/RecipesBase.jl:275
[5] _process_userrecipes(::Plots.Plot{Plots.GRBackend}, ::Dict{Symbol,Any}, ::Tuple{DataFrame}) at /home/dlin/.julia/packages/Plots/qZHsp/src/pipeline.jl:83
[6] _plot!(::Plots.Plot{Plots.GRBackend}, ::Dict{Symbol,Any}, ::Tuple{DataFrame}) at /home/dlin/.julia/packages/Plots/qZHsp/src/plot.jl:178
[7] #plot#138(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(plot), ::DataFrame) at /home/dlin/.julia/packages/Plots/qZHsp/src/plot.jl:57
[8] plot(::DataFrame) at /home/dlin/.julia/packages/Plots/qZHsp/src/plot.jl:51
[9] top-level scope at REPL[4]:1
In the code of series.jl, it is
prepareSeriesData(x) = error("Cannot convert $(typeof(x)) to series data for plotting")
How to correct this?
What about simply using plot(v.Cost):
julia> plot(v.Cost)
┌────────────────────────────────────────────────────────────┐
4.88870026e6 │⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⣠⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│ y1
│⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⠀⠀⢰⢹⠀⠀⠀⠀⠀⠀⠀⠀⢀⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⠀⠀⡎⢸⠀⠀⠀⠀⠀⠀⠀⠀⡸⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⢀⠎⠀⢸⠀⠀⠀⠀⠀⠀⠀⢠⠃⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⢸⠀⠀⢸⠀⠀⠀⠀⠀⠀⠀⡇⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⡎⠀⠀⢸⠀⠀⠀⠀⠀⠀⢸⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⡇⠀⠀⢸⠀⠀⠀⠀⠀⠀⡎⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⢰⠁⠀⠀⢸⠀⠀⠀⠀⠀⠀⡇⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⡸⠀⠀⠀⢸⠀⠀⠀⠀⠀⢰⠁⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⢀⠇⠀⠀⠀⢸⠀⠀⠀⠀⠀⢸⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⡼⠀⠀⠀⠀⠈⡆⠀⠀⠀⠀⡇⠀⠀⠀⢣⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⡰⠁⠀⠀⠀⠀⠀⡇⠀⠀⠀⢰⠁⠀⠀⠀⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⡰⠁⠀⠀⠀⠀⠀⠀⡇⠀⠀⢠⠇⠀⠀⠀⠀⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⡰⠁⠀⠀⠀⠀⠀⠀⠀⡇⠀⢀⠇⠀⠀⠀⠀⠀⢸⠀⠀⠀⠀⠀⠀⠀⠀⣀⠀⠀⠀⠀⠀⠀⠀⢀⠜⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡏⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⡎⠀⠀⠀⠀⠀⠀⢸⠀⠀⠀⠀⠀⢀⠜⠊⢸⠀⠀⠀⠀⠀⠀⢀⠎⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡄⠀│
│⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⡼⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⠀⠀⢠⠋⠀⠀⢸⠀⠀⠀⠀⠀⠀⡎⠀⠀⢱⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡜⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠋⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⡠⠒⠁⠀⠀⠀⠀⡇⠀⠀⠀⢀⠜⠀⠀⠀⢸⠀⠀⠀⠀⠀⠀⡠⠔⢲⠀⠀⠀⠀⠀⠀⠀⡔⠁⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠸⠒⠊⠀⠀⠀⠀⠀⠀⠀⡇⡠⠖⠉⠁⠀⠀⠀⠀⢸⠀⠀⠀⢀⡠⠒⠁⠀⠈⡆⠀⠀⠀⠀⡠⠎⠀⠀⠀⠀│
-71402.26000000001 │⠤⡧⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠭⠤⠤⠤⠤⠤⠤⠤⠤⠼⠶⠾⠭⠥⠤⠤⠤⠤⠤⠷⠶⠾⠭⠭⠤⠤⠤⠤⠤⠤│
└────────────────────────────────────────────────────────────┘
-1.67 92.67
That part of the docs might be outdated. You have to use StatsPlots and the #df macro to get the desired behavior.
julia> using StatsPlots
julia> #df v plot(:Cost)
┌────────────────────────────────────────────────────────────┐
4.88870026e6 │⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⣠⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│ y1
│⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⠀⠀⢰⢹⠀⠀⠀⠀⠀⠀⠀⠀⢀⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⠀⠀⡎⢸⠀⠀⠀⠀⠀⠀⠀⠀⡸⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⢀⠎⠀⢸⠀⠀⠀⠀⠀⠀⠀⢠⠃⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⢸⠀⠀⢸⠀⠀⠀⠀⠀⠀⠀⡇⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⡎⠀⠀⢸⠀⠀⠀⠀⠀⠀⢸⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⡇⠀⠀⢸⠀⠀⠀⠀⠀⠀⡎⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⢰⠁⠀⠀⢸⠀⠀⠀⠀⠀⠀⡇⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⠀⡸⠀⠀⠀⢸⠀⠀⠀⠀⠀⢰⠁⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⢀⠇⠀⠀⠀⢸⠀⠀⠀⠀⠀⢸⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⠀⡼⠀⠀⠀⠀⠈⡆⠀⠀⠀⠀⡇⠀⠀⠀⢣⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⠀⡰⠁⠀⠀⠀⠀⠀⡇⠀⠀⠀⢰⠁⠀⠀⠀⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⠀⡰⠁⠀⠀⠀⠀⠀⠀⡇⠀⠀⢠⠇⠀⠀⠀⠀⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡇⡰⠁⠀⠀⠀⠀⠀⠀⠀⡇⠀⢀⠇⠀⠀⠀⠀⠀⢸⠀⠀⠀⠀⠀⠀⠀⠀⣀⠀⠀⠀⠀⠀⠀⠀⢀⠜⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⡏⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⡎⠀⠀⠀⠀⠀⠀⢸⠀⠀⠀⠀⠀⢀⠜⠊⢸⠀⠀⠀⠀⠀⠀⢀⠎⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡄⠀│
│⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⡼⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⠀⠀⢠⠋⠀⠀⢸⠀⠀⠀⠀⠀⠀⡎⠀⠀⢱⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡜⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠋⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⡠⠒⠁⠀⠀⠀⠀⡇⠀⠀⠀⢀⠜⠀⠀⠀⢸⠀⠀⠀⠀⠀⠀⡠⠔⢲⠀⠀⠀⠀⠀⠀⠀⡔⠁⠀⠀│
│⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠸⠒⠊⠀⠀⠀⠀⠀⠀⠀⡇⡠⠖⠉⠁⠀⠀⠀⠀⢸⠀⠀⠀⢀⡠⠒⠁⠀⠈⡆⠀⠀⠀⠀⡠⠎⠀⠀⠀⠀│
-71402.26000000001 │⠤⡧⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠭⠤⠤⠤⠤⠤⠤⠤⠤⠼⠶⠾⠭⠥⠤⠤⠤⠤⠤⠷⠶⠾⠭⠭⠤⠤⠤⠤⠤⠤│
└────────────────────────────────────────────────────────────┘
-1.67 92.67
This is explained in later sections of the manual: https://docs.juliaplots.org/latest/input_data/#dataframes-support
I am trying to extract automatically electricity offers from this site.Once I set the postcode (i.e: 300) , I can download(manually) the pdf files
I am using httr package :
library(httr)
qr<- POST("http://www.qenergy.com.au/What-Are-Your-Options",
query=list(postcode=3000))
res <- htmlParse(content(qr))
The problem is that the files urls are not in the query response. Any help please.
Try this
library(httr)
qr<- POST("http://www.qenergy.com.au/What-Are-Your-Options",
encode="form",
body=list(postcode=3000))
res <- content(qr)
pdfs <- as(res['//a[contains(#href, "pdf")]/#href'], "character")
head(pdfs)
# [1] "flux-content/qenergy/pdf/VIC price fact sheet jemena distribution zone business/Jemena-Freedom-Biz-5-Day-Time-of-Use-A210.pdf"
# [2] "flux-content/qenergy/pdf/VIC price fact sheet jemena distribution zone business/Jemena-Freedom-Biz-7-Day-Time-of-Use-A250.pdf"
# [3] "flux-content/qenergy/pdf/VIC price fact sheet jemena distribution zone business/Jemena-Freedom-Biz-Single-Rate-CL.pdf"
# [4] "flux-content/qenergy/pdf/VIC price fact sheet jemena distribution zone business/Jemena-Freedom-Biz-Single-Rate.pdf"
# [5] "flux-content/qenergy/pdf/VIC price fact sheet united energy distribution zone business/United-Freedom-Biz-5-Day-Time-of-Use.pdf"
# [6] "flux-content/qenergy/pdf/VIC price fact sheet united energy distribution zone business/United-Freedom-Biz-7-Day-Time-of-Use.pdf"
I am performing a subset of a large ffdf objects and I noticed that when I use subset.ff it is generating a large number of NAs. I tried an alternative way by using ffwhich and the result is much faster and no NAs are generated. Here it is my test:
library(ffbase)
# deals is the ffdf I would like to subset
unique(deals$COMMODITY)
ff (open) integer length=7 (7) levels: CASH CO2 COAL ELEC GAS GCERT OIL
[1] [2] [3] [4] [5] [6] [7]
CASH CO2 COAL ELEC GAS GCERT OIL
# Using subset.ff
started.at=proc.time()
deals0 <- subset.ff(deals,deals$COMMODITY %in% c("CASH","COAL","CO2","ELEC","GCERT"))
cat("Finished in",timetaken(started.at),"\n")
Finished in 12.640sec
# NAs are generated
unique(deals0$COMMODITY)
ff (open) integer length=8 (8) levels: CASH CO2 COAL ELEC GAS GCERT OIL <NA>
[1] [2] [3] [4] [5] [6] [7] [8]
CASH CO2 COAL ELEC GAS GCERT OIL NA
# Subset using ffwhich
started.at=proc.time()
idx <- ffwhich(deals,COMMODITY %in% c("CASH","COAL","CO2","ELEC","GCERT"))
deals1 <- deals[idx,]
cat("Finished in",timetaken(started.at),"\n")
Finished in 3.130sec
# No NAs are generated
unique(deals1$COMMODITY)
ff (open) integer length=7 (7) levels: CASH CO2 COAL ELEC GAS GCERT OIL
[1] [2] [3] [4] [5] [6] [7]
CASH CO2 COAL ELEC GAS GCERT OIL
Any idea why this is happening?
subset.ff is probably using [and your criterion but not including a !is.na(.) clause. The default for "[" is to return items that are either TRUE or NA for the criterion vector. The regular subset function adds a !is.na(.) clause, but maybe the authors of ffbase didn't get around to that.