Rust Polars: how to change column value from string to datetime - datetime

I'm reading my CSV file with polars, but there's a column with values that I need to convert to DateTime. I've managed to manipulate the String (remove dots) now I need to strptime to convert to DateTime. How can I do this in rust polars?
My old project in Python did this, but I want to rewrite this in Rust Polars:
df["Meetdatum"] = df["Meetdatum"].str.replace(".", "", regex=False)
df["Datum"] = pd.to_datetime(df["Meetdatum"], format="%d %b %Y")

You can use something like this:
Cargo.toml:
[dependencies]
polars = { version = "*", features = [
"lazy",
"dtype-datetime",
"strings",
"lazy_regex",
] }
main.rs:
use polars::prelude::*;
fn main() {
let df = DataFrame::new(vec![Series::new(
"date_str",
&[
"5. Oct, ...2022",
"..31 ..Oct.., 2022.",
"E.R.R.O.R",
"31 Sep, 2022",
],
)])
.unwrap()
.lazy();
let df = df.with_column(
col("date_str")
.str()
.replace_all(lit("."), lit(""), true)
.str()
.strptime(StrpTimeOptions {
date_dtype: DataType::Date,
fmt: Some("%d %b, %Y".into()),
strict: false,
exact: true,
})
.alias("date"),
);
println!("{:?}", df.collect().unwrap());
}
Result:
shape: (4, 2)
┌─────────────────────┬────────────┐
│ date_str ┆ date │
│ --- ┆ --- │
│ str ┆ date │
╞═════════════════════╪════════════╡
│ 5. Oct, ...2022 ┆ 2022-10-05 │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ ..31 ..Oct.., 2022. ┆ 2022-10-31 │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ E.R.R.O.R ┆ null │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 31 Sep, 2022 ┆ null │
└─────────────────────┴────────────┘

Related

How to balance a dataset from a countmap table

I have this dataset:
text sentiment
randomstring positive
randomstring negative
randomstring netrual
random mixed
Then if I run a countmap i have:
"mixed" -> 600
"positive" -> 2000
"negative" -> 3300
"netrual" -> 780
I want to random sample from this dataset in a way that I have records of all smallest class (mixed = 600) and the same amount of each of other classes (positive=600, negative=600, neutral = 600)
I know how to do this in pandas:
df_teste = [data.loc[data.sentiment==i]\
.sample(n=int(data['sentiment']
.value_counts().nsmallest(1)[0]),random_state=SEED) for i in data.sentiment.unique()]
df_teste = pd.concat(df_teste, axis=0, ignore_index=True)
But I am having a hard time to do this in Julia.
Note: I don´t want to hardcode which of the class is the lowest one, so I am looking for a solution that infer that from the countmap or freqtable, if possible.
Why do you want a countmap or freqtable solution if you seem do want to use a data frame in the end?
This is how you would do this with DataFrames.jl (but without StatsBase.jl and FreqTables.jl as they are not needed for this):
julia> using Random
julia> using DataFrames
julia> df = DataFrame(text = [randstring() for i in 1:6680],
sentiment = shuffle!([fill("mixed", 600);
fill("positive", 2000);
fill("ngative", 3300);
fill("neutral", 780)]))
6680×2 DataFrame
Row │ text sentiment
│ String String
──────┼─────────────────────
1 │ R3W1KL5b positive
2 │ uCCpNrat ngative
3 │ fwqYTCWG ngative
⋮ │ ⋮ ⋮
6678 │ UJiNrlcw ngative
6679 │ 7aiNOQ1o neutral
6680 │ mbIOIQmQ ngative
6674 rows omitted
julia> gdf = groupby(df, :sentiment);
julia> min_len = minimum(nrow, gdf)
600
julia> df_sampled = combine(gdf) do sdf
return sdf[randperm(nrow(sdf))[1:min_len], :]
end
2400×2 DataFrame
Row │ sentiment text
│ String String
──────┼─────────────────────
1 │ positive O0QsyrJZ
2 │ positive 7Vt70PSh
3 │ positive ebFd8m4o
⋮ │ ⋮ ⋮
2398 │ neutral Kq8Wi2Vv
2399 │ neutral yygOzKuC
2400 │ neutral NemZu7R3
2394 rows omitted
julia> combine(groupby(df_sampled, :sentiment), nrow)
4×2 DataFrame
Row │ sentiment nrow
│ String Int64
─────┼──────────────────
1 │ positive 600
2 │ ngative 600
3 │ mixed 600
4 │ neutral 600
If your data is very large and you need the operation to be very fast there are more efficient ways to do it, but in most situations this should be fast enough and the solution does not require any extra packages.

Change identifier by looping through folders -- R

I have a loop related question. I have the following folder structure (excerpt):
├───Y2017
│ UDB_cSK17D.csv
│ UDB_cSK17H.csv
│ UDB_cSK17P.csv
│ UDB_cSK17R.csv
│ UDB_cUK17D.csv
│ UDB_cUK17H.csv
│ UDB_cUK17P.csv
│ UDB_cUK17R.csv
└───Y2018
│ UDB_cSK18D.csv
│ UDB_cSK18H.csv
│ UDB_cSK18P.csv
│ UDB_cSK18R.csv
│ UDB_cUK18D.csv
│ UDB_cUK18H.csv
│ UDB_cUK18P.csv
│ UDB_cUK18R.csv
All the files have the same structure. I would like to loop through them and extract data from a select number of columns. The file names also all have the same structure. All files have:
unique country identified (e.g. UK, SK in the examples above)
unique database type (D, H, P... - last character in file name)
I would like to construct a loop that iterates through the file names. For one country this would work like this:
library(data.table)
ldf<-list()
country_id<-"UK(.*)"
db_id<-"P.csv$"
listcsv<-dir(pattern = paste0(country_id,db_id), recursive = T, full.names = T)
for (k in 1:length(listcsv)){
ldf[[k]]<-fread(listcsv[k],select = c("PB010","PB020"))
}
uk_data<-bind_rows(as.data.frame(do.call(rbind,ldf[])))
This code extract all the columns I need based on the country identifier I give it (UK in this example). As I have numerous countries in my data set I would like to have a code that iterates through and updates the country identifier. I have tried the following:
ldf_new<-list()
countries <-c("SK", "UK")
for (i in 1:length(countries)) {
currcty1 <- countries[i]
listcsv<-dir(pattern = paste0(currcty1,"(.*)",db_id), recursive = T, full.names = T)
# print(listcsv)
ldf_new<-fread(listcsv[i],select = c("PB010","PB020"))
}
What happens here is that I get only the results of the last iteration in the variable ldf_new (i.e. UK in this case). Is there any way I could get the results for SK and UK.
Many thanks in advance!
Changing the last line of your loop so that a new element is added to the list should do the trick:
ldf_new<-list()
countries <-c("SK", "UK")
for (i in 1:length(countries)) {
currcty1 <- countries[i]
listcsv<-dir(pattern = paste0(currcty1,"(.*)",db_id), recursive = T, full.names = T)
# print(listcsv)
ldf_new<-c(ldf_new, fread(listcsv[i],select = c("PB010","PB020")))
}

How do i read a csv file with time in AM and PM in julia

I am using Julia CSV and I am trying to read data with DateTime in the form 10/17/2012 12:00:00 AM i tried
dfmt = dateformat"mm/dd/yyyy HH:MM:SS"
data =CSV.File("./Fremont_Bridge_Bicycle_Counter.csv", dateformat=dfmt) |> DataFrame
println(first(data,8))
but the thing is that I think the AM and PM makes the string not recognized as a date can someone help show how to pass this as a date
You can use the p specifier, which matches AM or PM. With that, your date format would look like this:
dfmt = dateformat"mm/dd/yyyy HH:MM:SS p"
You can see that the parsing is correct:
julia> DateTime("10/17/2012 12:00:00 AM", dfmt)
2012-10-17T00:00:00
To see all the possible format characters, check out the docstring of Dates.DateFormat, which is accessible in the REPL through ?DateFormat.
With the file Fremont_Bridge_Bicycle_Counter.csv
N1, N2, fecha
hola, 3, 10/03/2020 10:30:00
pepe, 5, 10/03/2020 11:40:50
juan, 5, 03/04/2020 20:10:12
And with the julia code:
using DataFrames, Dates, CSV
dfmt = dateformat"mm/dd/yyyy HH:MM:SS p"
data =CSV.File("./Fremont_Bridge_Bicycle_Counter.csv", dateformat=dfmt) |> DataFrame
println(first(data,8))
It gives the right result:
3×3 DataFrame
│ Row │ N1 │ N2 │ fecha │
│ │ String │ Int64 │ DateTime │
├─────┼────────┼───────┼─────────────────────┤
│ 1 │ hola │ 3 │ 2020-10-03T10:30:00 │
│ 2 │ pepe │ 5 │ 2020-10-03T11:40:50 │
│ 3 │ juan │ 5 │ 2020-03-04T20:10:12 │

Problem with Sqlite Query - How to convert the code from SQLite v0.9.0 to v1.0.0

I am prettry new to Julia and I am just playing around, and suddenly
the following code starts throwing errors, but it has worked in the past.
using SQLite
db = SQLite.DB("db")
data = SQLite.Query(db,"SELECT * FROM d")
throws:
ERROR: LoadError: MethodError: no method matching
SQLite.Query(::SQLite.DB, ::String)
can someone please enlighten me wha the problem is? Thank you.
I also tried with lower case: query.
Here is a short MWE of differences using SQLLite (v0.9.0 vs v1.0.0) with the current Julia version (1.3.1).
You do not have the table so you need to create it first:
using SQLite
using DataFrames
db = SQLite.DB("db")
# v0.9.0
SQLite.Query(db,"CREATE TABLE d (col1 INT, col2 varchar2(100))")
# v1.0.0
DBInterface.execute(db,"CREATE TABLE d (col1 INT, col2 varchar2(100))")
Now you can check if the table exits:
julia> SQLite.tables(db) |> DataFrame
1×1 DataFrames.DataFrame
│ Row │ name │
│ │ String⍰ │
├─────┼─────────┤
│ 1 │ d │
Let's insert some rows (note how one should sepearate data from SQL code via precompiled statements):
stmt = SQLite.Stmt(db, "INSERT INTO d (col1, col2) VALUES (?, ?)")
#v0.9.0
SQLite.execute!(stmt; values=(1, "Hello world"))
SQLite.execute!(stmt; values=(2, "Goodbye world"))
#v1.0.0
DBInterface.execute(stmt, (1, "Hello world"))
DBInterface.execute(stmt, (2, "Goodbye world"))
Now let us get the data
v0.9.0
julia> data = SQLite.Query(db,"SELECT * FROM d") |> DataFrame
3×2 DataFrame
│ Row │ col1 │ col2 │
│ │ Int64⍰ │ String⍰ │
├─────┼────────┼───────────────┤
│ 1 │ 1 │ Hello world │
│ 2 │ 2 │ Goodbye world │
v1.0.0
julia> data = DBInterface.execute(db, "select * from d") |> DataFrame
3×2 DataFrame
│ Row │ col1 │ col2 │
│ │ Int64⍰ │ String⍰ │
├─────┼────────┼───────────────┤
│ 1 │ 1 │ Hello world │
│ 2 │ 2 │ Goodbye world │

Kusto query could not be parsed

I have SecurityLog with fields like DstIP_s and want to display records matching my trojanDst table
let trojanDst = datatable (DstIP_s:string)
[ "1.1.1.1","2.2.2.2","3.3.3.3"
];
SecurityLog |
| join trojanDst on DstIP_s
I am getting query could not be parsed error ?
The query you posted has a redundant pipe (|) before the join.
From an efficiency standpoint, make sure the left side of the join is the smaller one, as suggested here: https://learn.microsoft.com/en-us/azure/kusto/query/best-practices#join-operator
This is too long for a comment. As #Yoni L pointed the problem is doubled pipe operator.
For anyone with SQL background join may be a bit counterintuitive(in reality it is kind=innerunique):
JOIN operator:
kind unspecified, kind=innerunique
Only one row from the left side is matched for each value of the on
key. The output contains a row for each match of this row with rows
from the right.
Kind=inner
There's a row in the output for every combination of matching rows
from left and right.
let t1 = datatable(key:long, value:string)
[
1, "a",
1, "b"
];
let t2 = datatable(key:long, value:string)
[
1, "c",
1, "d"
];
t1| join t2 on key;
Output:
┌─────┬───────┬──────┬────────┐
│ key │ value │ key1 │ value1 │
├─────┼───────┼──────┼────────┤
│ 1 │ a │ 1 │ c │
│ 1 │ a │ 1 │ d │
└─────┴───────┴──────┴────────┘
Demo
SQL style JOIN version:
let t1 = datatable(key:long, value:string)
[
1, "a",
1, "b"
];
let t2 = datatable(key:long, value:string)
[
1, "c",
1, "d"
];
t1| join kind=inner t2 on key;
Output:
┌─────┬───────┬──────┬────────┐
│ key │ value │ key1 │ value1 │
├─────┼───────┼──────┼────────┤
│ 1 │ b │ 1 │ c │
│ 1 │ a │ 1 │ c │
│ 1 │ b │ 1 │ d │
│ 1 │ a │ 1 │ d │
└─────┴───────┴──────┴────────┘
Demo
There are many join types in KQL such as innerunique, inner, leftouter, rightouter, fullouter, anti and more. here you can find the full list

Resources