I'm getting error messages when identifying ColumnDataSource as my source, want to do it right.
Let me show the errors.
First, I generate some random data in a DataFrame and put it into the ColumnDataSource:
col_list = ['ob1','ob2','ob3','ob4','ob5']
df = pd.DataFrame(np.random.uniform(73.965,74.03,size=(25, 5)).astype(float), columns=col_list)
df.reset_index(inplace=True)
df = df.rename(columns = {'index':'order'})
df['order'] = df['order'] + 1
cds = ColumnDataSource(data=df)
So far so good.
I try to generate a graph:
p = figure(title = 'ColumnDataSource test', sizing_mode = 'stretch_both')
p.line(x=cds.data['order'], y = cds.data['ob1'], source = cds.data[['order', 'ob1']])
show(p)
and get the following error:
Traceback (most recent call last):
File "e:\Black_Belt_Six_Sigma\first_take.py", line 57, in <module>
p.line(x=cds.data['order'], y = cds.data['ob1'], source = cds.data[['order', 'ob1']])
TypeError: unhashable type: 'list'
Fair enough, I won't give the source parameter a list and try again:
p = figure(title = 'ColumnDataSource test', sizing_mode = 'stretch_both')
p.line(x=cds.data['order'], y = cds.data['ob1'], source = cds.data)
show(p)
I get no graph but only the following error:
RuntimeError:
Expected x and y to reference fields in the supplied data source.
When a 'source' argument is passed to a glyph method, values that are sequences
(like lists or arrays) must come from references to data columns in the source.
For instance, as an example:
source = ColumnDataSource(data=dict(x=a_list, y=an_array))
p.circle(x='x', y='y', source=source, ...) # pass column names and a source
Alternatively, *all* data sequences may be provided as literals as long as a
source is *not* provided:
p.circle(x=a_list, y=an_array, ...) # pass actual sequences and no source
Based on this error message I've tried the following:
cds = ColumnDataSource(data=dict(order = df['order'].to_list(), ob1 = df['ob1'].to_list()))
p = figure(title = 'ColumnDataSource test', sizing_mode = 'stretch_both')
p.line(x=cds.data['order'], y = cds.data['ob1'], source = cds)
show(p)
And
cds = ColumnDataSource(data=dict(order = df['order'], ob1 = df['ob1']))
p = figure(title = 'ColumnDataSource test', sizing_mode = 'stretch_both')
p.line(x=cds.data['order'], y = cds.data['ob1'], source = cds)
show(p)
Both keep returning the same error message.
I can get a graph/plot if I don't specify the source parameter, so maybe that's the right course of action? Seems odd, I imagine it's important if the developers made it a parameter.
You should pass your dictionary keys order and ob1 directly to the arguments x and y. And your ColumDataSource cds to the argument source (see more examples here):
import pandas as pd
import numpy as np
from bokeh.plotting import figure, show
from bokeh.models import ColumnDataSource
col_list = ['ob1','ob2','ob3','ob4','ob5']
df = pd.DataFrame(np.random.uniform(73.965,74.03,size=(25, 5)).astype(float), columns=col_list)
df.reset_index(inplace=True)
df = df.rename(columns = {'index':'order'})
df['order'] = df['order'] + 1
cds = ColumnDataSource(data=df)
p = figure(title = 'ColumnDataSource test', sizing_mode = 'stretch_both')
p.line(x='order',y='ob1',source=cds)
show(p)
Related
PyTorch-Forecasting version: 0.10.2
PyTorch version:1.12.1
Python version:3.10.4
Operating System: windows
Expected behavior
No Error
Actual behavior
The Error is
File c:\Users\josepeeterson.er\Miniconda3\envs\pytorch\lib\site-packages\pytorch_forecasting\metrics\base_metrics.py:979, in DistributionLoss.to_quantiles(self, y_pred, quantiles, n_samples)
977 except NotImplementedError: # resort to derive quantiles empirically
978 samples = torch.sort(self.sample(y_pred, n_samples), -1).values
--> 979 quantiles = torch.quantile(samples, torch.tensor(quantiles, device=samples.device), dim=2).permute(1, 2, 0)
980 return quantiles
RuntimeError: quantile() q tensor must be same dtype as the input tensor
How do I set them to be of same datatype? This is happening internally. I do not have control over this. I am not using any GPUs.
The link to the .csv file with input data is https://github.com/JosePeeterson/Demand_forecasting
The data is just sampled from a negative binomila distribution wiht parameters (9,0.5) every 4 hours. the time inbetween is all zero.
I just want to see if DeepAR can learn this pattern.
Code to reproduce the problem
from pytorch_forecasting.data.examples import generate_ar_data
import matplotlib.pyplot as plt
import pandas as pd
from pytorch_forecasting.data import TimeSeriesDataSet
from pytorch_forecasting.data import NaNLabelEncoder
from pytorch_lightning.callbacks import EarlyStopping, LearningRateMonitor
import pytorch_lightning as pl
from pytorch_forecasting import NegativeBinomialDistributionLoss, DeepAR
import torch
from pytorch_forecasting.data.encoders import TorchNormalizer
data = [pd.read_csv('1_f_nbinom_train.csv')]
data["date"] = pd.Timestamp("2021-08-24") + pd.to_timedelta(data.time_idx, "H")
data['_hour_of_day'] = str(data["date"].dt.hour)
data['_day_of_week'] = str(data["date"].dt.dayofweek)
data['_day_of_month'] = str(data["date"].dt.day)
data['_day_of_year'] = str(data["date"].dt.dayofyear)
data['_week_of_year'] = str(data["date"].dt.weekofyear)
data['_month_of_year'] = str(data["date"].dt.month)
data['_year'] = str(data["date"].dt.year)
max_encoder_length = 60
max_prediction_length = 20
training_cutoff = data["time_idx"].max() - max_prediction_length
training = TimeSeriesDataSet(
data.iloc[0:-620],
time_idx="time_idx",
target="value",
categorical_encoders={"series": NaNLabelEncoder(add_nan=True).fit(data.series), "_hour_of_day": NaNLabelEncoder(add_nan=True).fit(data._hour_of_day), \
"_day_of_week": NaNLabelEncoder(add_nan=True).fit(data._day_of_week), "_day_of_month" : NaNLabelEncoder(add_nan=True).fit(data._day_of_month), "_day_of_year" : NaNLabelEncoder(add_nan=True).fit(data._day_of_year), \
"_week_of_year": NaNLabelEncoder(add_nan=True).fit(data._week_of_year), "_year": NaNLabelEncoder(add_nan=True).fit(data._year)},
group_ids=["series"],
min_encoder_length=max_encoder_length,
max_encoder_length=max_encoder_length,
min_prediction_length=max_prediction_length,
max_prediction_length=max_prediction_length,
time_varying_unknown_reals=["value"],
time_varying_known_categoricals=["_hour_of_day","_day_of_week","_day_of_month","_day_of_year","_week_of_year","_year" ],
time_varying_known_reals=["time_idx"],
add_relative_time_idx=False,
randomize_length=None,
scalers=[],
target_normalizer=TorchNormalizer(method="identity",center=False,transformation=None )
)
validation = TimeSeriesDataSet.from_dataset(
training,
data.iloc[-620:-420],
# predict=True,
stop_randomization=True,
)
batch_size = 64
train_dataloader = training.to_dataloader(train=True, batch_size=batch_size, num_workers=8)
val_dataloader = validation.to_dataloader(train=False, batch_size=batch_size, num_workers=8)
# save datasets
training.save("training.pkl")
validation.save("validation.pkl")
early_stop_callback = EarlyStopping(monitor="val_loss", min_delta=1e-4, patience=5, verbose=False, mode="min")
lr_logger = LearningRateMonitor()
trainer = pl.Trainer(
max_epochs=10,
gpus=0,
gradient_clip_val=0.1,
limit_train_batches=30,
limit_val_batches=3,
# fast_dev_run=True,
# logger=logger,
# profiler=True,
callbacks=[lr_logger, early_stop_callback],
)
deepar = DeepAR.from_dataset(
training,
learning_rate=0.1,
hidden_size=32,
dropout=0.1,
loss=NegativeBinomialDistributionLoss(),
log_interval=10,
log_val_interval=3,
# reduce_on_plateau_patience=3,
)
print(f"Number of parameters in network: {deepar.size()/1e3:.1f}k")
torch.set_num_threads(10)
trainer.fit(
deepar,
train_dataloaders=train_dataloader,
val_dataloaders=val_dataloader,
)
Need to cast samples to torch.tensor as shown below. Then save this base_metrics.py and rerun above code.
except NotImplementedError: # resort to derive quantiles empirically
samples = torch.sort(self.sample(y_pred, n_samples), -1).values
quantiles = torch.quantile(torch.tensor(samples), torch.tensor(quantiles, device=samples.device), dim=2).permute(1, 2, 0)
return quantiles
The problem I am faced with regards to llvmlite is producing a simple hello world example.
I am unable to display the string global variable value in the function I created.
It appears to always print out the number one.
I have already tried to return the stringtype which produced a error.
from llvmlite import ir
i64 = ir.IntType(64)
i8 = ir.IntType(16)
hellostr = 'hello, world!'
stringtype = ir.ArrayType(i64, len(hellostr))
module = ir.Module( name="m_hello_example" )
hello = ir.GlobalVariable(module, stringtype, '.str4')
fn_int_to_int_type = ir.FunctionType(i64, [stringtype.as_pointer()] )
fn_hel = ir.Function( module, fn_int_to_int_type, name="fn_hel" )
fn_hel_block = fn_hel.append_basic_block( name="fn_hel_entry" )
builder = ir.IRBuilder(fn_hel_block )
# zero = builder.constant(i64, 0)
# const_1 = ir.Constant(stringtype,1);
# builder.ret(const_1)
const_1 = ir.Constant(i64,1);
# print(const_1)
builder.ret(const_1)
print( module )
I was expecting the output to print out the string 'hello, world!'.
Any help would be much appreciated.
Thanks.
It ended up that I was able to solve my problem with the following code:
import llvmlite.ir as ir
import llvmlite.binding as llvm
from ctypes import CFUNCTYPE
def main():
m = ir.Module()
func_ty = ir.FunctionType(ir.VoidType(), []) #defining printer function as type void
func = ir.Function(m, func_ty, name="printer") #define function as printer
builder = ir.IRBuilder(func.append_basic_block('entry')) #defining the entry point of the function printer
fmt = "%s\n\0" #in function printf allows for inserting arg in, next global_fmt statements allow for creating #"fstr" assignment
c_fmt = ir.Constant(ir.ArrayType(ir.IntType(8), len(fmt)),
bytearray(fmt.encode("utf8")))
global_fmt = ir.GlobalVariable(m, c_fmt.type, name="fstr")
global_fmt.linkage = 'internal'
global_fmt.global_constant = True
global_fmt.initializer = c_fmt
arg = "Hello, World!\0" #args will be passed into printf function.
c_str_val = ir.Constant(ir.ArrayType(ir.IntType(8), len(arg)),
bytearray(arg.encode("utf8"))) #creates the c_str_value as a constant
printf_ty = ir.FunctionType(ir.IntType(32), [], var_arg=True) #creation of the printf function begins here and specifies the passing of a argument
printf = ir.Function(m, printf_ty, name="printf")
c_str = builder.alloca(c_str_val.type) #creation of the allocation of the %".2" variable
builder.store(c_str_val, c_str) #store as defined on the next line below %".2"
voidptr_ty = ir.IntType(8).as_pointer()
fmt_arg = builder.bitcast(global_fmt, voidptr_ty) #creates the %".4" variable with the point pointing to the fstr
builder.call(printf, [fmt_arg, c_str]) #We are calling the prinf function with the fmt and arg and returning the value as defiend on the next line
builder.ret_void()
#Next lines are for calling llvm and returning the assembly.
llvm.initialize()
llvm.initialize_native_target()
llvm.initialize_native_asmprinter()
print(str(m)) #PRINTING OUT THE ASSEMBLY
llvm_module = llvm.parse_assembly(str(m)) #Parsing teh assembly
tm = llvm.Target.from_default_triple().create_target_machine() #creating the target machine
with llvm.create_mcjit_compiler(llvm_module, tm) as ee:
ee.finalize_object() #Making sure all modules owned by the execution engine are fully processed and usable for execution
fptr = ee.get_function_address("printer") #fptr will reference the printer function
py_func = CFUNCTYPE(None)(fptr)
py_func() #run the function printer
if __name__ == "__main__":
main()
It appears that I didn't correctly assign the variable and hence why I wasn't returning anything.
I'm using the basemap library to display spatial information from Copernicus program.
The issue is i can not figure out how to project the data on the robin projection, but I do it correctly with the orthogonal projection.
So currently, I tried this :
plt.ioff()
# adapt for location of datasources
filePath = '../data/grib/download.grib'
# load data
grbs = grb.open(filePath)
grbs.seek(0)
data, lats, lons = (None, None, None)
dataUnit = None
title = None
for g in grbs:
data, lats, lons = g.data()
name = g.name
level = g.level
pressureUnit = g.pressureUnits
date = g.validDate
dataUnit = g.units
title = name + ' at ' + str(level) + ' ' + str(pressureUnit) + ' [' + str(date) + ']'
print(title)
break
# mapPlot = Basemap(projection='ortho', lat_0=0, lon_0=0)
mapPlot = Basemap(projection='robin', lat_0=0, lon_0=0, resolution='l')
mapPlot.drawcoastlines(linewidth=0.25)
x, y = mapPlot(lons, lats)
mapPlot.contourf(x, y, data)
mapPlot.colorbar(location='bottom', format='%.1f', label=dataUnit)
plt.title(title)
plt.show()
The orthogonal projection works correctly. But for the robin projection, I have an ... interesting pattern.
What I'm doing wrong ?
So i figure out how to do. I was misled but the first examples I saw.
Here is a my code:
import matplotlib
from mpl_toolkits.basemap import Basemap, shiftgrid
import matplotlib.pyplot as plt
import numpy as np
import pygrib as grb
# Get data
data = g['values']
lats = g['distinctLatitudes'] # 1D vector
lons = g['distinctLongitudes'] # 1D vector
# Useful information for late
name = g.name
level = str(g.level) + g.pressureUnits
date = g.validDate
dataUnit = g.units
# Parse the data
# Shit the data to start à -180. This is important to mark the data to start at -180°
data, lons = shiftgrid(180., data, lons, start=False) # shiftgrid
# Choose a representation (works with both)
# mapPlot = Basemap(projection='ortho', lat_0=0, lon_0=0)
mapPlot = Basemap(projection='robin', lat_0=0, lon_0=0)
mapPlot.drawcoastlines(linewidth=0.25)
# Convert the coordinates into the map projection
x, y = mapPlot(*np.meshgrid(lons, lats))
# Display data
map = mapPlot.contourf(x, y, data, levels=boundaries, cmap=plt.get_cmap('coolwarm'))
# Add what ever you want to your map.
mapPlot.nightshade(date, alpha=0.1)
# Legend
mapPlot.colorbar(map, label=dataUnit)
# Title
plt.title(name + ' at ' + str(level) + ' [' + str(date) + ']')
plt.show()
So it returns what I'm expecting.
I have created a dictionary, and I want to add the values from each item in the dictionary. Here is what I have so far:
import sys
import re
import collections
import numpy as np
dict = {'x':['x',0], 'y':['y',0], 'z':['z',0]}
I am then using an input file to count the instances of x, y, and z and add them to the dictionary:
with open(input_file, 'r', encoding='utf-8') as f:
for line in f:
words = line.split()
count_lines += 1
num_lines += 1
num_words += len(words)
title = line
for key in dict:
title = line
if re.search(key, title):
trusted[key][1]+=1
title = re.sub(key,dict[key][0],title)
while re.search(key, title):
dict[key][1]+=1
title = re.sub(key,dict[key][0],title)
dict_values = dict.values()
output_file.write('Dict:', sum(dict_values[1:-1]))
Then my dictionary would be, for example: dict = {'x':['x',6], 'y':['y',10], 'z':['z',8]}, and I want to add 6, 10, and 8 together.
I have tried this with and without the string split, I have tried assigning the sum equation to a variable and writing the variable, etc. I am continuously getting "TypeError: unsupported operand type(s) for +: 'int' and 'list'" and "TypeError: 'dict_values' object is not subscriptable" error messages.
I have already discussed a similar type of a question in this following post
How to set a for -loop in R
each file contents as follows:
FILE_1.FASTA
>>TTBK2_Hsap ,(CK1/TTBK)
MSGGGEQLDILSVGILVKERWKVLRKIGGGGFGEIYDALDMLTRENVALKVESAQQPKQVLKMEVAVLKKLQGKDHVCRFIGCGRNDRFNYVVMQLQGRNLADLRRSQSRGTFT
FILE_2.FASTA
>>TTBK2_Hsap ,(CK1/TTBK)
MSGGGEQLDILSVGILVKERWKVLRKIGGGGFGEIYDALDMLTRENVALKVESAQQPKQVLKMEVAVLKKLQGKDHVCRFIGCGRNDRFNYVVMQLQGRNLADLRRSQSRGTFT
However, there is another package in R which works like this:
extractAPAAC(x, props = c("Hydrophobicity", "Hydrophilicity"), lambda = 30,
w = 0.05, customprops = NULL)
I tried creating a function to run it for number of file sequences and the program looks like this
read_and_extract <- function(fasta) {
seq <- readFASTA(fasta)[[1]]
return(extractAPAAC(seq, props = c("Hydrophobicity", "Hydrophilicity"), lambda = 30,
w = 0.05, customprops = NULL))
}
setwd("H:\\CC")
fasta_files <- dir(pattern = "[.]fasta$")
aa_comp <- vapply(fasta_files, read_and_extract, rep(pi, 80))
write.csv(aa_comp, file = "C:\\Users\\PAAC.csv")
This programs shows an error
Error: unexpected ',' in "w = 0.05,"
But I have given w=0.05 as of default value, could anyone tell me where is the actual problem?