Constraint issue with pyomo involving a scalar - constraints

working on an economic optimization problem with pyomo, I would like to add a constraint to prevent the product of the commodity quantity and its price to go below zero (<0), avoiding a negative revenue. It appears that all the data are in a dataframe and I can't setup a constraint like:
def positive_revenue(model, t)
return model.P * model.C >=0
model.positive_rev = Constraint(model.T, rule=positive_revenue)
The system returns the error that the price is a scalar and it cannot process it. Indeed the price is set as such in the model:
model.T = Set(doc='quarter of year', initialize=df.quarter.tolist(), ordered=True)
model.P = Param(initialize=df.price.tolist(), doc='Price for each quarter')
##while the commodity is:
model.C = Var(model.T, domain=NonNegativeReals)
I just would like to apply that for each timestep (quarter of hour here) that:
price(t) * model.C(t) >=0
Can someone help me to spot the issue ? Thanks
Here are more information:
df dataframe:
df time_stamp price Status imbalance
quarter
0 2021-01-01 00:00:00 64.84 Final 16
1 2021-01-01 00:15:00 13.96 Final 38
2 2021-01-01 00:30:00 12.40 Final 46
index = quarter from 0 till 35049, so it is ok
Here is the df.info()
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 time_stamp 35040 non-null datetime64[ns]
1 price 35040 non-null float64
2 Status 35040 non-null object
3 imbalance 35040 non-null int64
I modified the to_list() > to_dict() in model.T but still facing the same issue:
KeyError: "Cannot treat the scalar component 'P' as an indexed component" at the time model.T is defined in the model parameter, set and variables.
Here is the constraint where the system issues the error:
def revenue_positive(model,t):
for t in model.T:
return (model.C[t] * model.P[t]) >= 0
model.positive_revenue = Constraint(model.T,rule=revenue_positive)
Can't figure it out...any idea ?
UPDATE
Model works after dropping an unfortunate 'quarter' column somewhere...after I renamed the index as quarter.
It runs but i still get negative revenues, so the constraints seems not working at present, here is how it is written:
def revenue_positive(model,t):
for t in model.T:
return (model.C[t] * model.P[t]) >= 0
model.positive_revenue = Constraint(model.T,rule=revenue_positive)
What am I missing here ? Thanks for help, just beginning

Welcome to the site.
The problem you appear to be having is that you are not building your model parameter model.P as an indexed component. I believe you likely want it to be indexed by your set model.T.
When you make indexed params in pyomo you need to initialize it with some key:value pairing, like a python dictionary. You can make that from your data frame by re-indexing your data frame so that the quarter labels are the index values.
Caution: The construction you have for model.T and this assume there are no duplicates in the quarter names.
If you have duplicates (or get a warning) then you'll need to do something else. If the quarter labels are unique you can do this:
import pandas as pd
import pyomo.environ as pyo
df = pd.DataFrame({'qtr':['Q5', 'Q6', 'Q7'], 'price':[12.80, 11.50, 8.12]})
df.set_index('qtr', inplace=True)
print(df)
m = pyo.ConcreteModel()
m.T = pyo.Set(initialize=df.index.to_list())
m.price = pyo.Param(m.T, initialize=df['price'].to_dict())
m.pprint()
which should get you:
price
qtr
Q5 12.80
Q6 11.50
Q7 8.12
1 Set Declarations
T : Size=1, Index=None, Ordered=Insertion
Key : Dimen : Domain : Size : Members
None : 1 : Any : 3 : {'Q5', 'Q6', 'Q7'}
1 Param Declarations
price : Size=3, Index=T, Domain=Any, Default=None, Mutable=False
Key : Value
Q5 : 12.8
Q6 : 11.5
Q7 : 8.12
2 Declarations: T price
edit for clarity...
NOTE:
The first argument when you create a pyomo parameter is the indexing set. If this is not provided, pyomo assumes that it is a scalar. You are missing the set as shown in my example and highlighted with arrow here: :)
|
|
|
V
m.price = pyo.Param(m.T, initialize=df['price'].to_dict())
Also note, you will need to initialize model.P with a dictionary as I have in the example, not a list.

Related

Pyomo constraint issue: not returning constrained result

I setup a constraint that does not constraint the solver in pyomo.
The constraint is the following:
def revenue_positive(model,t):
for t in model.T:
return (model.D[t] * model.P[t]) >= 0
model.positive_revenue = Constraint(model.T, rule=revenue_positive)
while the model parameters are:
model = ConcreteModel()
model.T = Set(doc='quarter of year', initialize=df.index.tolist(), ordered=True)
model.P = Param(model.T, initialize=df['price'].to_dict(), within=Any, doc='Price for each quarter')
model.C = Var(model.T, domain=NonNegativeReals)
model.D = Var(model.T, domain=NonNegativeReals)
income = sum(df.loc[t, 'price'] * model.D[t] for t in model.T)
expenses = sum(df.loc[t, 'price'] * model.C[t] for t in model.T)
profit = income - expenses
model.objective = Objective(expr=profit, sense=maximize)
# Solve the model
solver = SolverFactory('cbc')
solver.solve(model)
df dataframe is:
df time_stamp price Status imbalance Difference Situation ... week month hour_of_day day_of_week day_of_year yearly_quarter
quarter ...
0 2021-01-01 00:00:00 64.84 Final 16 -3 Deficit ... 00 1 0 4 1 1
1 2021-01-01 00:15:00 13.96 Final 38 2 Surplus ... 00 1 0 4 1 1
2 2021-01-01 00:30:00 12.40 Final 46 1 Surplus ... 00 1 0 4 1 1
3 2021-01-01 00:45:00 7.70 Final 65 14 Surplus ... 00 1 0 4 1 1
4 2021-01-01 01:00:00 64.25 Final 3 -9 Deficit ... 00 1 1 4 1 1
The objective is to constraint the solver not to accept a negative revenue. As such it does not work as the solver passes 6 negative revenue values through. Looking at the indices with negative revenue, it appears the system chooses to sell at a negative price to buy later at a price even "more" negative, so from an optimization standpoint, it is ok. I would like to check the difference in results if we prohibit the solver to do that. Any input is welcome as after many searches on the web, still not the right way to write it correctly.
I did a pprint() of the constraint that returned:
positive_revenue : Size=35040, Index=T, Active=True
UPDATE following new constraint code:
def revenue_positive(model,t):
return model.D[t] * model.P[t] >= 0
model.positive_revenue = Constraint(model.T, rule=revenue_positive)
Return the following error:
ERROR: Rule failed when generating expression for constraint positive_revenue
with index 283: ValueError: Invalid constraint expression. The constraint
expression resolved to a trivial Boolean (True) instead of a Pyomo object.
Please modify your rule to return Constraint.Feasible instead of True.
Error thrown for Constraint 'positive_revenue[283]'
ERROR: Constructing component 'positive_revenue' from data=None failed:
ValueError: Invalid constraint expression. The constraint expression
resolved to a trivial Boolean (True) instead of a Pyomo object. Please
modify your rule to return Constraint.Feasible instead of True.
Error thrown for Constraint 'positive_revenue[283]'
Traceback (most recent call last):
File "/home/olivier/Desktop/Elia - BESS/run_imbalance.py", line 25, in <module>
results_df = optimize_year(df)
File "/home/olivier/Desktop/Elia - BESS/battery_model_imbalance.py", line 122, in optimize_year
model.positive_revenue = Constraint(model.T, rule=revenue_positive)
File "/home/olivier/anaconda3/lib/python3.9/site-packages/pyomo/core/base/block.py", line 542, in __setattr__
self.add_component(name, val)
File "/home/olivier/anaconda3/lib/python3.9/site-packages/pyomo/core/base/block.py", line 1087, in add_component
val.construct(data)
File "/home/olivier/anaconda3/lib/python3.9/site-packages/pyomo/core/base/constraint.py", line 781, in construct
self._setitem_when_not_present(
File "/home/olivier/anaconda3/lib/python3.9/site-packages/pyomo/core/base/indexed_component.py", line 778, in _setitem_when_not_present
obj.set_value(value)
File "/home/olivier/anaconda3/lib/python3.9/site-packages/pyomo/core/base/constraint.py", line 506, in set_value
raise ValueError(
ValueError: Invalid constraint expression. The constraint expression resolved to a trivial Boolean (True) instead of a Pyomo object. Please modify your rule to return Constraint.Feasible instead of True.
Error thrown for Constraint 'positive_revenue[283]'
So there are 2 issues w/ your constraint. It isn't clear if one is a cut & paste issue or not.
The function call to make the constraint appears to be indented and inside of your function after the return statement, making it unreachable code. Could be just the spacing in your post.
You are incorrectly adding a loop inside of your function. You are passing in the parameter t as a function argument and then you are blowing it away with the for loop, which only executes for the first value of t in T then hits the return statement. Remove the loop. When you use the rule= structure in pyomo it will call the rule for each instance of the set that you are using in the Constraint(xx, rule=) structure.
So I think you should have:
def revenue_positive(model, t):
return model.D[t] * model.P[t] >= 0
model.positive_revenue = Constraint(model.T, rule=revenue_positive)
Updated re: the error you added.
The error cites the 283rd index. My bet is that price[283] is zero, so you are multiplying by a zero and killing your variable.
You could add a check within the function that checks if the price is zero, and in that case, just return pyo.Constraint.Feasible, which is the trivial return that doesn't influence the model (or crash)

is it possible to get a new instance for namedtuple pushed into a dictionary before values are known?

It looks like things are going wrong on line 9 for me. Here I wish to push a new copy of the TagsTable into a dictionary. I'm aware that once a namedtuple field is recorded, it can not be changed. However, results baffle me as it looks like the values do change - when this code exits all entries of mp3_tags[ any of the three dictionary keys ].date are set to the last date of "1999_03_21"
So, two questions:
Is there a way to get a new TagsTable pushed into the dictionary ?
Why doesnt the code fail and not allow the second (and even third) date to be written to the TagsTable.date field (since it seems to be references to the same namedtuple) ? I thought you could not write a second value ?
from collections import namedtuple
2 TagsTable = namedtuple('TagsTable',['title','date','subtitle','artist','summary','length','duration','pub_date'])
3 mp3files = ['42-001.mp3','42-002.mp3','42-003.mp3']
4 dates = ['1999_01_07', '1999_02_14', '1999_03_21']
5
6 mp3_tags = {}
7
8 for mp3file in mp3files:
9 mp3_tags[mp3file] = TagsTable
10
11 for mp3file,date_string in zip(mp3files,dates):
12 mp3_tags[mp3file].date = date_string
13
14 for mp3file in mp3files:
15 print( mp3_tags[mp3file].date )
looks like this is the fix I was looking for:
from collections import namedtuple
mp3files = ['42-001.mp3','42-002.mp3','42-003.mp3']
dates = ['1999_01_07', '1999_02_14', '1999_03_21']
mp3_tags = {}
for mp3file in mp3files:
mp3_tags[mp3file] = namedtuple('TagsTable',['title','date','subtitle','artist','summary','length','duration','pub_date'])
for mp3file,date_string in zip(mp3files,dates):
mp3_tags[mp3file].date = date_string
for mp3file in mp3files:
print( mp3_tags[mp3file].date )

Gnuplot: data normalization

I have several time-based datasets which are of very different scale, e. g.
[set 1]
2010-01-01 10
2010-02-01 12
2010-03-01 13
2010-04-01 19
…
[set 2]
2010-01-01 920
2010-02-01 997
2010-03-01 1010
2010-04-01 1043
…
I'd like to plot the relative growth of both since 2010-01-01. To put both curves on the same graph I have to normalize them. So I basically need to pick the first Y value and use it as a weight:
plot "./set1" using 1:($2/10), "./set2" using 1:($2/920)
But I want to do it automatically instead of hard-coding 10 and 920 as dividers. I don't even need the max value of the second column, I just want to pick the first value or, better, a value for a given date.
So my question: is there a way to parametrize the value of a given column which corresponds a given value of the given X column (X is a time axis)? Something like
plot "./set1" using 1:($2/$2($1="2010-01-01")), "./set2" using 1:($2/$2($1="2010-01-01"))
where $2($1="2010-01-01") is the feature I'm looking for.
Picking the first value is quite easy. Simply remember its value and divide all data values by it:
ref = 0
plot "./set1" using 1:(ref = ($0 == 0 ? $2 : ref), $2/ref),\
"./set2" using 1:(ref = ($0 == 0 ? $2 : ref), $2/ref)
Using the value at a given date is more involved:
Using an external tool (awk)
ref1 = system('awk ''$1 == "2010-01-01" { print $2; exit; }'' set1')
ref2 = system('awk ''$1 == "2010-01-01" { print $2; exit; }'' set1')
plot "./set1" using 1:($2/ref1), "./set1" using 1:($2/ref2)
Using gnuplot
You can use gnuplot's stats command to pick the desired value, but you must pay attention to do all time settings only after that:
a) String comparison
stats "./set1" using (strcol(1) eq "2010-01-01" ? $2 : 1/0)
ref1 = STATS_max
...
set timefmt ...
set xdata time
...
plot ...
b) Compare the actual time value (works like this only since version 5.0):
reftime = strptime("%Y-%m-%d", "2010-01-01")
stats "./set1" using (timecolumn(1, "%Y-%m-%d") == reftime ? $2 : 1/0)
ref1 = STATS_max
...
set timefmt ...
set xdata time
...
plot ...

How to perform pandas drop_duplicates based on index column

I am banging my head against the wall when trying to perform a drop duplicate for time series, base on the value of a datetime index.
My function is the following:
def csv_import_merge_T(f):
dfsT = [pd.read_csv(fp, index_col=[0], parse_dates=[0], dayfirst=True, names=['datetime','temp','rh'], header=0) for fp in files]
dfT = pd.concat(dfsT)
#print dfT.head(); print dfT.index; print dfT.dtypes
dfT.drop_duplicates(subset=index, inplace=True)
dfT.resample('H').bfill()
return dfT
which is called by:
inputcsvT = ['./input_csv/A08_KI_T*.csv']
for csvnameT in inputcsvT:
files = glob.glob(csvnameT)
print ('___'); print (files)
t = csv_import_merge_T(files)
print csvT
I receive the error
NameError: global name 'index' is not defined
what is wrong?
UPDATE:
The issue appear to arise when csv input files (which are to be concatenated) are overlapped.
inputcsvT = ['./input_csv/A08_KI_T*.csv'] gets files
A08_KI_T5
28/05/2015 17:00,22.973,24.021
...
08/10/2015 13:30,24.368,45.974
A08_KI_T6
08/10/2015 14:00,24.779,41.526
...
10/02/2016 17:00,22.326,41.83
and it runs correctly, whereas:
inputcsvT = ['./input_csv/A08_LR_T*.csv'] gathers
A08_LR_T5
28/05/2015 17:00,22.493,25.62
...
08/10/2015 13:30,24.296,44.596
A08_LR_T6
28/05/2015 17:00,22.493,25.62
...
10/02/2016 17:15,21.991,38.45
which leads to an error.
IIUC you can call reset_index and then drop_duplicates and then set_index again:
In [304]:
df = pd.DataFrame(data=np.random.randn(5,3), index=list('aabcd'))
df
Out[304]:
0 1 2
a 0.918546 -0.621496 -0.210479
a -1.154838 -2.282168 -0.060182
b 2.512519 -0.771701 -0.328421
c -0.583990 -0.460282 1.294791
d -1.018002 0.826218 0.110252
In [308]:
df.reset_index().drop_duplicates('index').set_index('index')
Out[308]:
0 1 2
index
a 0.918546 -0.621496 -0.210479
b 2.512519 -0.771701 -0.328421
c -0.583990 -0.460282 1.294791
d -1.018002 0.826218 0.110252
EDIT
Actually there is a simpler method is to call duplicated on the index and invert it:
In [309]:
df[~df.index.duplicated()]
Out[308]:
0 1 2
index
a 0.918546 -0.621496 -0.210479
b 2.512519 -0.771701 -0.328421
c -0.583990 -0.460282 1.294791
d -1.018002 0.826218 0.110252

Removing duplicate records from .Xdf file

I would like to remove the duplicate records from my large .xdf file trans.xdf.
Here is the file details:
File name: /poc/revor/data/trans.xdf
Number of observations: 1000000000
Number of variables: 5
Number of blocks: 40
Compression type: zlib
Variable information:
Var 1: CARD_ID, Type: character
Var 2: SE_NO, Type: character
Var 3: r12m_cv, Type: numeric, Low/High: (-2348.7600, 40587.3900)
Var 4: r12m_roc, Type: numeric, Low/High: (0.0000, 231.0000)
Var 5: PROD_GRP_CD, Type: character
Also below is the sample data of the file:
CARD_ID SE_NO r12m_cv r12m_roc PROD_GRP_CD
900000999000000000 1045815024 110 1 1
900000999000000000 1052487253 247.52 2 1
900000999000000000 9999999999 38.72 1 1
900000999000000000 1090389768 1679.96 16 1
900000999000000000 1091226035 0 1 1
900000999000000000 1091241208 538.68 4 1
900000999000000000 9999999999 83 1 1
900000999000000000 1091468041 148.4 3 1
900000999000000000 1092640358 3.13 1 1
900000999000000000 1093468692 546.29 1 1
I have tried using rxDataStep function to use its transform parameter to call to unique() function over the .xdf file. Below is the code for the same:
uniq_dat <- function( dataList )
{
datalist <- unique(datalist)
return(datalist)
}
rxDataStepXdf(inFile = "/poc/revor/data/trans.xdf",outFile = "/poc/revor/data/trans.xdf",transformFunc = uniq_dat,overwrite = TRUE)
But was getting below error:
Error in unique(datalist) : object 'datalist' not found
Error in transformation function: Error in unique(datalist) : object 'datalist' not found
Error in rxCall("RxDataStep", params) :
So anybody could point out the mistake that I am doing here or if there is a better way to remove the duplicate records from the .Xdf file. I am avoiding loading the data into inmemory dataframe as the data is pretty huge.
I am running the above code in Revolution R Environment over HDFS.
If the same can be obtained by any other approach then the example for the same would be appreciated.
Thanks for the help in advance :)
Cheers,
Amit
you can remove the duplicate values providing removeDupKeys=TRUE parameter for rxSort() function. For example for your case:
XdfFilePath <- file.path("<your file's fully qualified path>/trans.xdf")
rxSort(inData = XdfFilePath,sortByVars=c("CARD_ID","SE_NO","r12m_cv","r12m_roc","PROD_GRP_CD"), removeDupKeys=TRUE)
if you want to remove duplicate records based on a specific key column, for example, based on SE_NO column
set the key value as sortByVars="SE_NO"

Resources