I am trying to implement equivalent of numpy.where for dates as follows:
from datetime import date, timedelta as td, datetime
d1 = datetime.strptime('1/1/1995', "%m/%d/%Y")
d2 = datetime.strptime('12/31/2015', "%m/%d/%Y")
AllDays = []
while(d1<=d2):
AllDays.append(d1)
d1 = d1 + td(days=1)
validDate = AllDays
trainStDt = '1/1/1995'
trainEnDt = '12/31/2013'
testStDt = '1/1/2014'
testEnDt = '12/31/2015'
indTrain = (validDate >= datetime.strptime(trainStDt,'%m/%d/%Y')) & (validDate <=
datetime.strptime(trainEnDt,'%m/%d/%Y'))
indTest = (validDate >= datetime.strptime(testStDt,'%m/%d/%Y')) & (validDate <=
datetime.strptime(testEnDt,'%m/%d/%Y'))
trainDates = validDate[indTrain]
testDates = validDate[indTest]
print trainDates[0]
print trainDates[-1:]
print testDates[0]
print testDates[-1:]
However:
(1) indTrain doesn't work as it is trying to compare list to datetime
(2) my solution is to loop through each element of validDates
Is there a better way to do it?
Just turn your list into an array. Add import numpy as np to the top of your script, and after your while loop, add:
AllDays = np.array(AllDays)
Related
I'm trying to interact DatePicker with SelectionRangeSlider widgets. So far I could manage to link the slider with the DatePicker, but I'm getting trouble with the opposite way.
With this code the period slider updates the DatePicker:
from datetime import datetime
import pandas as pd
import ipywidgets as widgets
start_date = datetime(2021, 7, 1)
end_date = datetime(2021, 7, 5)
dates = pd.date_range(start_date, end_date, freq='D')
options = [(date.strftime(' %d %b %Y '), date) for date in dates]
pick_start = widgets.DatePicker(
description='',
disabled=False,
value = start_date
)
pick_end = start_day = widgets.DatePicker(
description='',
disabled=False,
value=end_date
)
selection_range_slider = widgets.SelectionRangeSlider(
options=options,
index=(0, len(options) - 1),
description='Period',
orientation='horizontal',
layout=widgets.Layout(width='100%', padding='35px')
)
def update_pick(*args):
pick_start.value = datetime.strptime(selection_range_slider.value[0].strftime("%Y-%m-%d"), "%Y-%m-%d")
pick_end.value = datetime.strptime(selection_range_slider.value[1].strftime("%Y-%m-%d"), "%Y-%m-%d")
def update_slider(*args):
selection_range_slider.value[0] = pick_start.value.strftime("%Y-%m-%d")
selection_range_slider.value[1] = pick_end.value.strftime("%Y-%m-%d")
selection_range_slider.observe(update_pick, 'value')
pick_start.observe(update_slider, 'value')
pick_end.observe(update_slider, 'value')
center_layout = widgets.Layout(display='flex',
align_items='center',
width='100%')
day = widgets.HBox(children=[pick_start, selection_range_slider, pick_end], layout=center_layout)
display(day)
Also try with this function without result:
def update_slider(*args):
start_date = datetime(pick_start.value.year, pick_start.value.month, pick_start.value.day)
end_date = datetime(pick_end.value.year, pick_end.value.month, pick_end.value.day)
dates = pd.date_range(start_date, end_date, freq='D')
options = [(date.strftime(' %d %b %Y '), date) for date in dates]
selection_range_slider.value[0] = options[0]
selection_range_slider.value[1] = options[len(options)-1]
Need help interacting DatePicker with SelectionRangeSlider
Thanks!
you have done a nice job of creating a standalone question here, thanks. A couple suggestions for your next Q to speed up an answer.
Please include your imports, I can't run your code without them:
from datetime import datetime
import pandas as pd
import ipywidgets as widgets
Please post any error messages that you see when your code runs. I got a ValueError which would have improved the question even more.
The error said there was an issue assigning to an index of a tuple, Here's my solution.
Assign the whole tuple directly, you cannot update index 0 and then index 1 of your selection_range_slider
Use raw datetimes to pass between widgets, your mixed string conversions were causing some issues. You can still change how the dates are displayed in the slider by using a dictionary for options.
from datetime import datetime
import pandas as pd
import ipywidgets as widgets
start_date = datetime(2021, 7, 1)
end_date = datetime(2021, 7, 5)
dates = pd.date_range(start_date, end_date, freq='D')
options = {date.strftime(' %d %b %Y '): date for date in dates}
pick_start = widgets.DatePicker(
description='',
disabled=False,
value = start_date
)
pick_end = start_day = widgets.DatePicker(
description='',
disabled=False,
value=end_date
)
selection_range_slider = widgets.SelectionRangeSlider(
options=options,
index=(0, len(options) - 1),
description='Period',
orientation='horizontal',
layout=widgets.Layout(width='100%', padding='35px')
)
def update_pick(*args):
pick_start.value = selection_range_slider.value[0]
pick_end.value = selection_range_slider.value[1]
def update_slider(*args):
selection_range_slider.value = (pick_start.value, pick_end.value)
selection_range_slider.observe(update_pick, 'value')
pick_start.observe(update_slider, 'value')
pick_end.observe(update_slider, 'value')
center_layout = widgets.Layout(display='flex',
align_items='center',
width='100%')
day = widgets.HBox(children=[pick_start, selection_range_slider, pick_end], layout=center_layout)
display(day)
Two simple questions:
Does Warp10 integrate into streamlit to feed visualisations?
If so, please would you specify how this can be accomplished?
Thanking you in advance.
Best wishes,
There's no direct integration of Warp 10 in streamlit.
Although streamlit can handle any kind of data, it's mainly focused on pandas DataFrame. DataFrames are tables whereas Warp 10 Geo Time Series are time series. So even if Warp 10 was integrated in streamlit, it would require some code to properly format the data for streamlit to give its full potential.
That being said, here is a small example on how to display data stored in Warp 10 with streamlit:
import json
from datetime import datetime, timedelta
import requests
import streamlit as st
from bokeh.palettes import Category10_10 as palette
from bokeh.plotting import figure
# Should be put in a configuration file.
fetch_endpoint = 'http://localhost:8080/api/v0/fetch'
token = 'READ' # Change that to your actual token
def load_data_as_json(selector, start, end):
headers = {'X-Warp10-Token': token}
params = {'selector': selector, 'start': start, 'end': end, 'format': 'json'}
r = requests.get(fetch_endpoint, params=params, headers=headers)
return r.text
st.title('Warp 10 Test')
# Input parameters
selector = st.text_input('Selector', value="~streamlit.*{}")
start_date = st.date_input('Start date', value=datetime.now() - timedelta(days=10))
start_time = st.time_input('Start time')
end_date = st.date_input('End date')
end_time = st.time_input('End time')
# Convert datetime.dates and datetime.times to microseconds (default time unit in Warp 10)
start = int(datetime.combine(start_date, start_time).timestamp()) * 1000000
end = int(datetime.combine(end_date, end_time).timestamp()) * 1000000
# Make the query to Warp 10 and get back a json.
json_data = load_data_as_json(selector, start, end)
gtss = json.loads(json_data)
# Iterate through the json and populate a Bokeh graph.
p = figure(title='GTSs', x_axis_label='time', y_axis_label='value')
for gts_index, gts in enumerate(gtss):
tss = []
vals = []
for point in gts['v']:
tss.append(point[0])
vals.append(point[-1])
p.line(x=tss, y=vals, legend_label=gts['c'] + json.dumps(gts['l']), color=palette[gts_index % len(palette)])
st.bokeh_chart(p, use_container_width=True)
# Also display the json.
st.json(json_data)
Sample code snippet tried:
for row in range(1,sheet.max_row+1):
for col in range(1, sheet.max_column+1):
temp = None
cell_obj = sheet.cell(row=row,column=col)
temp = re.search(r"requestor", str(cell_obj.value))
if temp:
if 'requestor' in cell_obj.value:
cell_obj.value.replace('requestor',
'ABC')
Trying to replace from an xlsx cell containing value "Customer name: requestor " with value "Customer name: ABC" .How can this be achieved easily ?
I found my answer in this post:https://www.edureka.co/community/42935/python-string-replace-not-working
The replace function doesn't store the result in the same variable. Hence the solution for above:
mvar = None
for row in range(1,sheet.max_row+1):
for col in range(1, sheet.max_column+1):
temp = None
cell_obj = sheet.cell(row=row,column=col)
temp = re.search(r"requestor", str(cell_obj.value))
if temp:
if 'requestor' in cell_obj.value:
mvar = cell_obj.value.replace('requestor',
'ABC')
cell_obj.value = mvar
Just keep it simple. Instead of re and replace, search for the given value and override the cell.
The example below also gives you the ability to change 'customer name' if needed:
wb = openpyxl.load_workbook("example.xlsx")
sheet = wb["Sheet1"]
customer_name = "requestor"
replace_with = "ABC"
search_string = f"Customer name: {customer_name}"
replace_string = f"Customer name: {replace_with}"
for row in range(1, sheet.max_row + 1):
for col in range(1, sheet.max_column + 1):
cell_obj = sheet.cell(row=row, column=col)
if cell_obj.value == search_string:
cell_obj.value = replace_string
wb.save("example_copy.xlsx") # remember that you need to save the results to the file
I want to select data from some pandas DataFrame in a Jupyter-notebook through a SelectionRangeSlider and plot the filtered data using holoviews bar chart.
Consider the following example:
import numpy as np
import pandas as pd
import datetime
import holoviews as hv
hv.extension('bokeh')
import ipywidgets as widgets
start = int(datetime.datetime(2017,1,1).strftime("%s"))
end = int(datetime.datetime(2017,12,31).strftime("%s"))
size = 100
rints = np.random.randint(start, end + 1, size = size)
df = pd.DataFrame(rints, columns = ['zeit'])
df["bytes"] = np.random.randint(5,20,size=size)
df['who']= np.random.choice(['John', 'Paul', 'George', 'Ringo'], len(df))
df["zeit"] = pd.to_datetime(df["zeit"], unit='s')
df.zeit = df.zeit.dt.date
df.sort_values('zeit', inplace = True)
df = df.reset_index(drop=True)
df.head(2)
This gives the test DataFrame df:
Let's group the data:
data = pd.DataFrame(df.groupby('who')['bytes'].sum())
data.reset_index(level=0, inplace=True)
data.sort_values(by="bytes", inplace=True)
data.head(2)
Now, create the SelectionRangeSlider that is to be used to filter and update the barchart.
%%opts Bars [width=800 height=400 tools=['hover']]
def view2(v):
x = df[(df.zeit > r2.value[0].date()) & (df.zeit < r2.value[1].date())]
data = pd.DataFrame(x.groupby('who')['bytes'].sum())
data.sort_values(by="bytes", inplace=True)
data.reset_index(inplace=True)
display(hv.Bars(data, kdims=['who'], vdims=['bytes']))
r2 = widgets.SelectionRangeSlider(options = options, index = index, description = 'Test')
widgets.interactive(view2, v=r2)
(I have already created an issue on github for the slider not displaying the label correctly, https://github.com/jupyter-widgets/ipywidgets/issues/1759)
Problems that persist:
the image width and size collapse to default after first update (is there a way to give %%opts as argument to hv.Bars?)
the y-Scale should remain constant (i.e. from 0 to 150 for all updates)
is there any optimization possible concerning speed of updates?
Thanks for any help.
Figured out how to do it using bokeh: https://github.com/bokeh/bokeh/issues/7082
I'm trying to convert date string to date number using dtstr2dtnummx (three time faster than datenum), but for this input
dtstr2dtnummx({'2010-12-12
12:21:13.101'},'yyyy-mm-dd
HH:MM:SS.FFF')
and this input
dtstr2dtnummx({'2010-12-12
12:21:13.121'},'yyyy-mm-dd
HH:MM:SS.FFF')
getting the same output. I used the following tutorial to build up the date format.
Ahh sorry, UPDATED
The corresponding format of 'FFF' in datenum is 'SSS' in dtstr2dtnummx, as can be seen in cnv2icudf.m line #126. The end result is:
>> d1 = dtstr2dtnummx({'2010-12-12 12:21:13.101'},'yyyy-MM-dd HH:mm:ss.SSS')
d1 =
734484.514734965
>> d2 = dtstr2dtnummx({'2010-12-12 12:21:13.121'},'yyyy-MM-dd HH:mm:ss.SSS')
d2 =
734484.514735197
>> % double check the results - difference should equal 0.02 secs:
>> secsPerDay = 24*60*60;
>> timeDiff = secsPerDay * (d2-d1)
timeDiff =
0.019996
I have now posted an article about this on http://undocumentedmatlab.com/blog/datenum-performance/