What Julia functions can output a DataFrame so it is converted into text other than the ones shown below?
using DataFrames
A = DataFrame(randn(10, 7));
print("\n\n\n", "A = DataFrame(randn(10, 7))")
print("\n\n\n","print(A)\n")
print(A)
print("\n\n\n","show(A)\n")
show(A)
print("\n\n\n","show(A, true)\n")
show(A, true)
print("\n\n\n","show(A, false)\n")
show(A, false)
print("\n\n\n","showall(A)\n")
showall(A)
print("\n\n\n","showall(A, true)\n")
showall(A, true)
print("\n\n\n","showall(A, false)\n")
showall(A, false)
print("\n\n\n","display(A)\n")
display(A)
Most of these output something similar to the following:
10×7 DataFrames.DataFrame
│ Row │ x1 │ x2 │ x3 │ x4 │ x5 │ x6 │ x7 │
├─────┼────────────┼───────────┼───────────┼───────────┼────────────┼───────────┼───────────┤
│ 1 │ 0.377968 │ -2.23532 │ 0.560632 │ 1.00294 │ 1.32404 │ 1.30788 │ -2.09068 │
│ 2 │ -0.694824 │ -0.765572 │ -1.11163 │ 0.038083 │ -0.52553 │ -0.571156 │ 0.977219 │
│ 3 │ 0.343035 │ -1.47047 │ 0.228148 │ -1.29784 │ -1.00742 │ 0.127103 │ -0.399041 │
│ 4 │ -0.0979587 │ -0.445756 │ -0.483188 │ 0.816921 │ -1.12535 │ 0.603824 │ 0.293274 │
│ 5 │ 1.12755 │ -1.62993 │ 0.178764 │ -0.201441 │ -0.730923 │ 0.230186 │ -0.679262 │
│ 6 │ 0.481705 │ -0.716072 │ 0.747341 │ -0.310009 │ 1.4159 │ -0.175918 │ -0.079051 │
│ 7 │ 0.732061 │ -1.08842 │ -1.18988 │ 0.577758 │ -1.474 │ -1.43082 │ -0.584148 │
│ 8 │ -1.077 │ -1.41973 │ -0.330143 │ -1.12357 │ 1.01005 │ 1.06746 │ 2.09197 │
│ 9 │ -1.60122 │ -1.44661 │ 0.299586 │ 1.46604 │ -0.0200695 │ 2.62421 │ 0.396777 │
│ 10 │ -1.74101 │ -0.541589 │ 0.425117 │ 0.14669 │ 0.95779 │ 1.73954 │ -1.7994 │
This is ok and looks decent in a notebook, and is output correctly to latex/pdf with nbconvert as a plain ascii text table. However, I want more options to get text output similar to the following which looks much better in the latex/pdf generated by nbconvert.
| Column 1 | Column 2 | Column 3 | Column 4 |
|-------|-------------|-------------|--------------|
| a | b | c | d |
| A | B | C | D |
Are there any functions that output a Julia DataFrame with this formatting? What about other parameters like digits = or caption =?
Looking at the source code, there isn't an out-of-the-box way. You'll have to write your own (which you can then contribute to the github repo).
Easiest way to do this is to make use of the output writetable() to an in-memory file (using | as a separator) and then handle the column headers
Related
I understand theory that is behind tagged pointers and how it is used to save additional data in pointer.
But i dont understand this part (from wikipedia article about tagged pointers).
Most architectures are byte-addressable (the smallest addressable unit is a byte), but certain types of data will often be aligned to the size of the data, often a word or multiple thereof. This discrepancy leaves a few of the least significant bits of the pointer unused
Why is this happening ?
Does pointer have only 30 bites (on 32 bit architectures) and that 2 bites are result of aligning?
Why there are 2 bites left unused in first place ?
And does this decrease size of addresable space (from 2^32 bytes to 2^30 bytes)?
Consider an architecture that uses 16-bit alignment and also 16-bit pointers (just to avoid having too many binary digits!). A pointer will only ever be referring to memory locations that are multiples of 16, but the pointer value is still precise down to the byte. So a pointer that, in binary, is:
0000000000000100
refers to the memory location 4 (the fifth byte in memory):
┌────────────────────┬───────────────────┬─────────────┬──────────────┐
│ Address in Decimal │ Address in Binary │ 8─bit bytes │ 16─bit words │
├────────────────────┼───────────────────┼─────────────┼──────────────┤
│ 0 │ 0000000000000000 │ ┌─────────┐ │ ┌──────────┐ │
│ │ │ └─────────┘ │ │ │ │
│ 1 │ 0000000000000001 │ ┌─────────┐ │ │ │ │
│ │ │ └─────────┘ │ └──────────┘ │
│ 2 │ 0000000000000010 │ ┌─────────┐ │ ┌──────────┐ │
│ │ │ └─────────┘ │ │ │ │
│ 3 │ 0000000000000011 │ ┌─────────┐ │ │ │ │
│ │ │ └─────────┘ │ └──────────┘ │
│ 4 - this one │ 0000000000000100 │ ┌─────────┐ │ ┌──────────┐ │
│ │ │ └─────────┘ │ │ │ │
│ 5 │ 0000000000000101 │ ┌─────────┐ │ │ │ │
│ │ │ └─────────┘ │ └──────────┘ │
│ 6 │ 0000000000000110 │ ┌─────────┐ │ ┌──────────┐ │
│ │ │ └─────────┘ │ │ │ │
│ 7 │ 0000000000000111 │ ┌─────────┐ │ │ │ │
│ │ │ └─────────┘ │ └──────────┘ │
│ ... │ │ │ │
└────────────────────┴───────────────────┴─────────────┴──────────────┘
With 16-bit alignment, there will never be a pointer referring to memory location 5 because it wouldn't be aligned, the next useful value is 6:
0000000000000110
Note that the least significant bit (the one on the far right) is still 0. In fact, for all valid pointer values on that architecture, that bit will be 0. That's what they mean by leaving "...a few of the least significant bits of the pointer unused." In my example it's just one bit, but if you had 32-bit alignment, it would be two bits at the end of pointer value that would always be zero:
┌────────────────────┬───────────────────┬─────────────┬──────────────┐
│ Address in Decimal │ Address in Binary │ 8─bit bytes │ 32─bit words │
├────────────────────┼───────────────────┼─────────────┼──────────────┤
│ 0 │ 0000000000000000 │ ┌─────────┐ │ ┌──────────┐ │
│ │ │ └─────────┘ │ │ │ │
│ 1 │ 0000000000000001 │ ┌─────────┐ │ │ │ │
│ │ │ └─────────┘ │ │ │ │
│ 2 │ 0000000000000010 │ ┌─────────┐ │ │ │ │
│ │ │ └─────────┘ │ │ │ │
│ 3 │ 0000000000000011 │ ┌─────────┐ │ │ │ │
│ │ │ └─────────┘ │ └──────────┘ │
│ 4 │ 0000000000000100 │ ┌─────────┐ │ ┌──────────┐ │
│ │ │ └─────────┘ │ │ │ │
│ 5 │ 0000000000000101 │ ┌─────────┐ │ │ │ │
│ │ │ └─────────┘ │ │ │ │
│ 6 │ 0000000000000110 │ ┌─────────┐ │ │ │ │
│ │ │ └─────────┘ │ │ │ │
│ 7 │ 0000000000000111 │ ┌─────────┐ │ │ │ │
│ │ │ └─────────┘ │ └──────────┘ │
│ 8 │ 0000000000001000 │ ┌─────────┐ │ ┌──────────┐ │
│ │ │ └─────────┘ │ │ │ │
│ 9 │ 0000000000001001 │ ┌─────────┐ │ │ │ │
│ │ │ └─────────┘ │ │ │ │
│ 10 │ 0000000000001010 │ ┌─────────┐ │ │ │ │
│ │ │ └─────────┘ │ │ │ │
│ 11 │ 0000000000001011 │ ┌─────────┐ │ │ │ │
│ │ │ └─────────┘ │ └──────────┘ │
│ ... │ │ │ │
└────────────────────┴───────────────────┴─────────────┴──────────────┘
I am able to read a json file and convert into dataframe using below code.
df = open(jsontable, "normal.json") |> DataFrame
normal.json looks like below,
{"col1":["thasin", "hello", "world"],"col2":[1,2,3],"col3":["abc", "def", "ghi"]}
So final df has,
3×3 DataFrame
│ Row │ col1 │ col2 │ col3 │
│ │ String │ Int64 │ String │
├─────┼────────┼───────┼────────┤
│ 1 │ thasin │ 1 │ abc │
│ 2 │ hello │ 2 │ def │
│ 3 │ world │ 3 │ ghi │
But, the same code is not working for record formatted json file.
the format is list like {column -> value}, … , {column -> value}
My sample json
{"billing_account_id":"0139A","credits":[],"invoice":{"month":"202003"},"cost_type":"regular"}
{"billing_account_id":"0139A","credits":[1.45],"invoice":{"month":"202003"},"cost_type":"regular"}
{"billing_account_id":"0139A","credits":[2.00, 3.56],"invoice":{"month":"202003"},"cost_type":"regular"}
Expected output:
billing_account_id cost_type credits invoice
0 0139A regular [] {'month': '202003'}
1 0139A regular [1.45] {'month': '202003'}
2 0139A regular [2.0, 3.56] {'month': '202003'}
This can be done in python like below,
data = []
for line in open("sample.json", 'r'):
data.append(json.loads(line))
print(data)
df=pd.DataFrame(data)
How to do this in Julia?
Note that your file is not a valid JSON (its lines are valid JSON, not the whole file).
You can do this like this:
julia> using DataFrames, JSON3
julia> df = JSON3.read.(eachline("sample.json")) |> DataFrame;
julia> df.credits = Vector{Float64}.(df.credits);
julia> df.invoice = Dict{Symbol,String}.(df.invoice);
julia> df
3×4 DataFrame
│ Row │ billing_account_id │ credits │ invoice │ cost_type │
│ │ String │ Array{Float64,1} │ Dict{Symbol,String} │ String │
├─────┼────────────────────┼────────────────────────────┼────────────────────────┼───────────┤
│ 1 │ 0139A │ 0-element Array{Float64,1} │ Dict(:month=>"202003") │ regular │
│ 2 │ 0139A │ [1.45] │ Dict(:month=>"202003") │ regular │
│ 3 │ 0139A │ [2.0, 3.56] │ Dict(:month=>"202003") │ regular │
The transformations on :credits and :invoice columns are to make them of type that is easy to work with (otherwise they use types that are defined internally by JSON3.jl).
A more advanced option is to do it in one shot by specifying the row schema using a NamedTuple type e.g.:
julia> df = JSON3.read.(eachline("sample.json"),
NamedTuple{(:billing_account_id, :credits, :invoice, :cost_type),Tuple{String,Vector{Float64},Dict{String,String},String}}) |>
DataFrame
3×4 DataFrame
│ Row │ billing_account_id │ credits │ invoice │ cost_type │
│ │ String │ Array{Float64,1} │ Dict{String,String} │ String │
├─────┼────────────────────┼────────────────────────────┼─────────────────────────┼───────────┤
│ 1 │ 0139A │ 0-element Array{Float64,1} │ Dict("month"=>"202003") │ regular │
│ 2 │ 0139A │ [1.45] │ Dict("month"=>"202003") │ regular │
│ 3 │ 0139A │ [2.0, 3.56] │ Dict("month"=>"202003") │ regular │
Unrelated to the julia answer, but in python you can do
pd.read_json("sample.json", lines=True)
I have a csv file which looks like below,
20×2 DataFrame
│ Row │ Id │ Date │
│ │ Int64 │ String │
├─────┼───────┼────────────┤
│ 1 │ 1 │ 01-01-2010 │
│ 2 │ 2 │ 02-01-2010 │
│ 3 │ 3 │ 03-01-2010 │
│ 4 │ 4 │ 04-01-2010 │
│ 5 │ 5 │ 05-01-2010 │
│ 6 │ 6 │ 06-01-2010 │
│ 7 │ 7 │ 07-01-2010 │
│ 8 │ 8 │ 08-01-2010 │
│ 9 │ 9 │ 09-01-2010 │
│ 10 │ 10 │ 10-01-2010 │
│ 11 │ 11 │ 11-01-2010 │
│ 12 │ 12 │ 12-01-2010 │
│ 13 │ 13 │ 13-01-2010 │
│ 14 │ 14 │ 14-01-2010 │
│ 15 │ 15 │ 15-01-2010 │
│ 16 │ 16 │ 16-01-2010 │
│ 17 │ 17 │ 17-01-2010 │
│ 18 │ 18 │ 18-01-2010 │
│ 19 │ 19 │ 19-01-2010 │
│ 20 │ 20 │ 20-01-2010 │
after reading the csv file date columns is in String type. How to externally convert a string series into Datetime series. In Julia Data Frame docs doesn't talk Anything about TimeSeries.
How to externally convert a series or vector into Datetime format?
Is there anyway I can mention timeseries columns while reading a CSV File?
When reading-in a CSV file you can specify dateformat kwarg in CSV.jl:
CSV.File("your_file_name.csv", dateformat="dd-mm-yyyy") |> DataFrame
On the other hand if your data frame is called df then to convert String to Date in your case use:
using Dates
df.Date = Date.(df.Date, "dd-mm-yyyy")
Here is how I have done it:
First a helper function to convert different string formats.
parse_date(d::AbstractString) = DateTime(d, dateformat"yyyy-mm-dd HH:MM:SS")
parse_date(v::Vector{AbstractString}) = parse_date.(v)
parse_date(v::Vector{String}) = parse_date.(v)
parse_date(v::Vector{String31}) = parse_date(String.(v))
using Pipe, TimeSeries
prices = #pipe CSV.File(filename; header = 1, delim = ",") |>
TimeArray(_; timestamp = :Date, timeparser = parse_date)
Suppose I have the following DataFrame in Julia, named A:
│ Row │ x1 │ x2 │
├──────┼─────┼─────────┤
│ 1 │ 1.0 │ 5.78341 │
│ 2 │ 2.0 │ 5.05401 │
│ 3 │ 3.0 │ 4.79754 │
│ 4 │ 4.0 │ 4.4126 │
│ 5 │ 5.0 │ 4.29433 │
│ 6 │ 6.0 │ 4.14306 │
│ 7 │ 1.0 │ 5.94811 │
│ 8 │ 2.0 │ 5.0432 │
│ 9 │ 3.0 │ 4.78697 │
│ 10 │ 4.0 │ 4.40384 │
│ 11 │ 5.0 │ 4.29901 │
?
│ 3933 │ 2.0 │ 4.90528 │
│ 3934 │ 3.0 │ 4.57429 │
│ 3935 │ 4.0 │ 4.3988 │
│ 3936 │ 5.0 │ 4.19076 │
│ 3937 │ 6.0 │ 4.09517 │
│ 3938 │ 7.0 │ 3.96192 │
│ 3939 │ 1.0 │ 5.88878 │
│ 3940 │ 2.0 │ 5.87492 │
│ 3941 │ 3.0 │ 4.9453 │
│ 3942 │ 4.0 │ 4.39047 │
│ 3943 │ 5.0 │ 4.28096 │
│ 3944 │ 6.0 │ 4.13686 │
I want to calculate the mean of x2 values by x1 values only if the number of repetitions of x1 values in less or equal than 500, for example. I tried the following code, but it didn't work:
aggregate(A,length(:x1).<=500,mean)
If for example, only the values 1,2 and 3 meet the condition, the result should be:
│ Row │ x1 │ x2 │
├──────┼─────┼─────────┤
│ 1 │ 1.0 │ 5.85264 │
│ 2 │ 2.0 │ 5.15852 │
│ 3 │ 3.0 │ 4.92586 │
where the x2 values are the corresponding mean values.
Any suggestions?
I would use DataFramesMeta.jl here as it will be cleaner than using DataFrames.jl only functionality (I give two ways to obtain the desired result as examples):
using DataFramesMeta
# I generate a smaller DataFrame with cutoff of 15 for the example
df = DataFrame(x1=repeat([1,1,2,2,3], inner=10), x2=rand(50))
# first way to do it
#linq df |>
groupby(:x1) |>
where(length(:x1)>15) |>
#based_on(x2=mean(:x2))
# other way to do the same
#linq df |>
by(:x1, x2=mean(:x2), n=length(:x2)) |>
where(:n.>15) |>
select(:x1, :x2)
I have imported a DataFrame as below:
julia> df
100×3 DataFrames.DataFrame
│ Row │ ex1 │ ex2 │ admit │
├─────┼─────────┼─────────┼───────┤
│ 1 │ 34.6237 │ 78.0247 │ 0 │
│ 2 │ 30.2867 │ 43.895 │ 0 │
│ 3 │ 35.8474 │ 72.9022 │ 0 │
│ 4 │ 60.1826 │ 86.3086 │ 1 │
│ 5 │ 79.0327 │ 75.3444 │ 1 │
│ 6 │ 45.0833 │ 56.3164 │ 0 │
│ 7 │ 61.1067 │ 96.5114 │ 1 │
│ 8 │ 75.0247 │ 46.554 │ 1 │
⋮
│ 92 │ 90.4486 │ 87.5088 │ 1 │
│ 93 │ 55.4822 │ 35.5707 │ 0 │
│ 94 │ 74.4927 │ 84.8451 │ 1 │
│ 95 │ 89.8458 │ 45.3583 │ 1 │
│ 96 │ 83.4892 │ 48.3803 │ 1 │
│ 97 │ 42.2617 │ 87.1039 │ 1 │
│ 98 │ 99.315 │ 68.7754 │ 1 │
│ 99 │ 55.34 │ 64.9319 │ 1 │
│ 100 │ 74.7759 │ 89.5298 │ 1 │
I want to plot this DataFrame using ex1 as x-axis, ex2 as y-axis. In addition, the data is categorized by the third column :admit, so I want to give dots different colors based on the :admit value.
I used Scale.color_discrete_manual to set up colors, and I tried to use Guide.manual_color_key to change the color key legend. However it turns out Gadfly made two color keys.
p = plot(df, x = :ex1, y = :ex2, color=:admit,
Scale.color_discrete_manual(colorant"deep sky blue",
colorant"light pink"),
Guide.manual_color_key("Legend", ["Failure", "Success"],
["deep sky blue", "light pink"]))
My question is how to change the color key legend when using Scale.color_discrete_manual?
One related question is Remove automatically generated color key in Gadfly plot, where the best answer suggests to use two layers plus Guide.manual_color_key. Is there a better solution for using DataFrame and Scale.color_discrete_manual?
Currently, it looks like users cannot customize the color legend generated by color or Scale.color_discrete_manual based on the discussion.
From the same source, Mattriks suggested to use an extra column as "label". Although it is not "natural" for changing color key, it works pretty well.
Therefore, for the same dataset in the problem. We add one more column:
df[:admission] = map(df[:admit])do x
if x == 1
return "Success"
else
return "Failure"
end
end
julia> df
100×4 DataFrames.DataFrame
│ Row │ exam1 │ exam2 │ admit │ admission │
├─────┼─────────┼─────────┼───────┼───────────┤
│ 1 │ 34.6237 │ 78.0247 │ 0 │ "Failure" │
│ 2 │ 30.2867 │ 43.895 │ 0 │ "Failure" │
│ 3 │ 35.8474 │ 72.9022 │ 0 │ "Failure" │
│ 4 │ 60.1826 │ 86.3086 │ 1 │ "Success" │
│ 5 │ 79.0327 │ 75.3444 │ 1 │ "Success" │
│ 6 │ 45.0833 │ 56.3164 │ 0 │ "Failure" │
│ 7 │ 61.1067 │ 96.5114 │ 1 │ "Success" │
│ 8 │ 75.0247 │ 46.554 │ 1 │ "Success" │
⋮
│ 92 │ 90.4486 │ 87.5088 │ 1 │ "Success" │
│ 93 │ 55.4822 │ 35.5707 │ 0 │ "Failure" │
│ 94 │ 74.4927 │ 84.8451 │ 1 │ "Success" │
│ 95 │ 89.8458 │ 45.3583 │ 1 │ "Success" │
│ 96 │ 83.4892 │ 48.3803 │ 1 │ "Success" │
│ 97 │ 42.2617 │ 87.1039 │ 1 │ "Success" │
│ 98 │ 99.315 │ 68.7754 │ 1 │ "Success" │
│ 99 │ 55.34 │ 64.9319 │ 1 │ "Success" │
│ 100 │ 74.7759 │ 89.5298 │ 1 │ "Success" │
Then color the data using this new column Scale.color_discrete_manual:
plot(df, x = :exam1, y = :exam2, color = :admission,
Scale.color_discrete_manual(colorant"deep sky blue",
colorant"light pink"))