Environment:
textfsm: 1.1.2
python: 3.9.6
os: windows 10
textFSM template:
Value Filldown policy_name (\S+)
Value Required name (\S+)
Value police_rate (\d+( ms|pps)?)
Value peak_rate (\d+( pps)?)
Value police_burst (\d+( ms|packets)?)
Value peak_burst (\d+( ms|packets)?)
Value police_burst_ex (\d+( ms)?)
Value List conform_action (set|transmit|drop)
Value List conform_action_set_value (.*)
Value List exceed_action (set|transmit|drop)
Value List exceed_action_set_value (.*)
Value List violate_action (set|transmit|drop)
Start
^policy.map\s(?:${policy_name})? -> PolicyClass
PolicyClass
^ class\s${name}
^\s+police cir (percent )?${police_rate}(( bc)? ${police_burst}(( be)? ${police_burst_ex})?)?( pir (percent )?${peak_rate}( be ${peak_burst})?)?(\s+conform-action ${conform_action}(\s+exceed-action ${exceed_action}(\s+violate-action ${violate_action})?)?)?
^\s+conform-action ${conform_action}(-${conform_action_set_value})?
^\s+exceed-action ${exceed_action}(-${exceed_action_set_value})?
^\s+violate-action ${violate_action}
^ ! -> Record
config for parsing:
policy-map INDEPENDENTFIBRENETWORKSLTD348569-G0/1/1:12-Ethernet-IngressQoS-Template1-Standard
!
class CIR_BPS_CONFIG_1
police cir 1000000000 bc 12500000 pir 1800000000 be 12500000
conform-action set-mpls-exp-imposition-transmit 3
conform-action set-qos-transmit 3
conform-action set-discard-class-transmit 1
exceed-action set-mpls-exp-imposition-transmit 1
exceed-action set-discard-class-transmit 0
exceed-action set-qos-transmit 1
violate-action drop
!
!
textFSM parser result:
[['INDEPENDENTFIBRENETWORKSLTD348569-G0/1/1:12-Ethernet-IngressQoS-Template1-Standard', 'CIR_BPS_CONFIG_1', '1000000000', '1800000000', '12500000', '12500000', '', [None, 'set', 'set', 'set'], ['mpls-exp-imposition-transmit 3', 'qos-transmit 3', 'discard-class-transmit 1'], [None, 'set', 'set', 'set'], ['mpls-exp-imposition-transmit 1', 'discard-class-transmit 0', 'qos-transmit 1'], [None, 'drop']]]
As you can see, the conform_action, exceed_action and violate_action columns all have an extra None item.
How can I get rid of it (w/o post-parsing process)?
Related
I setup a constraint that does not constraint the solver in pyomo.
The constraint is the following:
def revenue_positive(model,t):
for t in model.T:
return (model.D[t] * model.P[t]) >= 0
model.positive_revenue = Constraint(model.T, rule=revenue_positive)
while the model parameters are:
model = ConcreteModel()
model.T = Set(doc='quarter of year', initialize=df.index.tolist(), ordered=True)
model.P = Param(model.T, initialize=df['price'].to_dict(), within=Any, doc='Price for each quarter')
model.C = Var(model.T, domain=NonNegativeReals)
model.D = Var(model.T, domain=NonNegativeReals)
income = sum(df.loc[t, 'price'] * model.D[t] for t in model.T)
expenses = sum(df.loc[t, 'price'] * model.C[t] for t in model.T)
profit = income - expenses
model.objective = Objective(expr=profit, sense=maximize)
# Solve the model
solver = SolverFactory('cbc')
solver.solve(model)
df dataframe is:
df time_stamp price Status imbalance Difference Situation ... week month hour_of_day day_of_week day_of_year yearly_quarter
quarter ...
0 2021-01-01 00:00:00 64.84 Final 16 -3 Deficit ... 00 1 0 4 1 1
1 2021-01-01 00:15:00 13.96 Final 38 2 Surplus ... 00 1 0 4 1 1
2 2021-01-01 00:30:00 12.40 Final 46 1 Surplus ... 00 1 0 4 1 1
3 2021-01-01 00:45:00 7.70 Final 65 14 Surplus ... 00 1 0 4 1 1
4 2021-01-01 01:00:00 64.25 Final 3 -9 Deficit ... 00 1 1 4 1 1
The objective is to constraint the solver not to accept a negative revenue. As such it does not work as the solver passes 6 negative revenue values through. Looking at the indices with negative revenue, it appears the system chooses to sell at a negative price to buy later at a price even "more" negative, so from an optimization standpoint, it is ok. I would like to check the difference in results if we prohibit the solver to do that. Any input is welcome as after many searches on the web, still not the right way to write it correctly.
I did a pprint() of the constraint that returned:
positive_revenue : Size=35040, Index=T, Active=True
UPDATE following new constraint code:
def revenue_positive(model,t):
return model.D[t] * model.P[t] >= 0
model.positive_revenue = Constraint(model.T, rule=revenue_positive)
Return the following error:
ERROR: Rule failed when generating expression for constraint positive_revenue
with index 283: ValueError: Invalid constraint expression. The constraint
expression resolved to a trivial Boolean (True) instead of a Pyomo object.
Please modify your rule to return Constraint.Feasible instead of True.
Error thrown for Constraint 'positive_revenue[283]'
ERROR: Constructing component 'positive_revenue' from data=None failed:
ValueError: Invalid constraint expression. The constraint expression
resolved to a trivial Boolean (True) instead of a Pyomo object. Please
modify your rule to return Constraint.Feasible instead of True.
Error thrown for Constraint 'positive_revenue[283]'
Traceback (most recent call last):
File "/home/olivier/Desktop/Elia - BESS/run_imbalance.py", line 25, in <module>
results_df = optimize_year(df)
File "/home/olivier/Desktop/Elia - BESS/battery_model_imbalance.py", line 122, in optimize_year
model.positive_revenue = Constraint(model.T, rule=revenue_positive)
File "/home/olivier/anaconda3/lib/python3.9/site-packages/pyomo/core/base/block.py", line 542, in __setattr__
self.add_component(name, val)
File "/home/olivier/anaconda3/lib/python3.9/site-packages/pyomo/core/base/block.py", line 1087, in add_component
val.construct(data)
File "/home/olivier/anaconda3/lib/python3.9/site-packages/pyomo/core/base/constraint.py", line 781, in construct
self._setitem_when_not_present(
File "/home/olivier/anaconda3/lib/python3.9/site-packages/pyomo/core/base/indexed_component.py", line 778, in _setitem_when_not_present
obj.set_value(value)
File "/home/olivier/anaconda3/lib/python3.9/site-packages/pyomo/core/base/constraint.py", line 506, in set_value
raise ValueError(
ValueError: Invalid constraint expression. The constraint expression resolved to a trivial Boolean (True) instead of a Pyomo object. Please modify your rule to return Constraint.Feasible instead of True.
Error thrown for Constraint 'positive_revenue[283]'
So there are 2 issues w/ your constraint. It isn't clear if one is a cut & paste issue or not.
The function call to make the constraint appears to be indented and inside of your function after the return statement, making it unreachable code. Could be just the spacing in your post.
You are incorrectly adding a loop inside of your function. You are passing in the parameter t as a function argument and then you are blowing it away with the for loop, which only executes for the first value of t in T then hits the return statement. Remove the loop. When you use the rule= structure in pyomo it will call the rule for each instance of the set that you are using in the Constraint(xx, rule=) structure.
So I think you should have:
def revenue_positive(model, t):
return model.D[t] * model.P[t] >= 0
model.positive_revenue = Constraint(model.T, rule=revenue_positive)
Updated re: the error you added.
The error cites the 283rd index. My bet is that price[283] is zero, so you are multiplying by a zero and killing your variable.
You could add a check within the function that checks if the price is zero, and in that case, just return pyo.Constraint.Feasible, which is the trivial return that doesn't influence the model (or crash)
working on an economic optimization problem with pyomo, I would like to add a constraint to prevent the product of the commodity quantity and its price to go below zero (<0), avoiding a negative revenue. It appears that all the data are in a dataframe and I can't setup a constraint like:
def positive_revenue(model, t)
return model.P * model.C >=0
model.positive_rev = Constraint(model.T, rule=positive_revenue)
The system returns the error that the price is a scalar and it cannot process it. Indeed the price is set as such in the model:
model.T = Set(doc='quarter of year', initialize=df.quarter.tolist(), ordered=True)
model.P = Param(initialize=df.price.tolist(), doc='Price for each quarter')
##while the commodity is:
model.C = Var(model.T, domain=NonNegativeReals)
I just would like to apply that for each timestep (quarter of hour here) that:
price(t) * model.C(t) >=0
Can someone help me to spot the issue ? Thanks
Here are more information:
df dataframe:
df time_stamp price Status imbalance
quarter
0 2021-01-01 00:00:00 64.84 Final 16
1 2021-01-01 00:15:00 13.96 Final 38
2 2021-01-01 00:30:00 12.40 Final 46
index = quarter from 0 till 35049, so it is ok
Here is the df.info()
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 time_stamp 35040 non-null datetime64[ns]
1 price 35040 non-null float64
2 Status 35040 non-null object
3 imbalance 35040 non-null int64
I modified the to_list() > to_dict() in model.T but still facing the same issue:
KeyError: "Cannot treat the scalar component 'P' as an indexed component" at the time model.T is defined in the model parameter, set and variables.
Here is the constraint where the system issues the error:
def revenue_positive(model,t):
for t in model.T:
return (model.C[t] * model.P[t]) >= 0
model.positive_revenue = Constraint(model.T,rule=revenue_positive)
Can't figure it out...any idea ?
UPDATE
Model works after dropping an unfortunate 'quarter' column somewhere...after I renamed the index as quarter.
It runs but i still get negative revenues, so the constraints seems not working at present, here is how it is written:
def revenue_positive(model,t):
for t in model.T:
return (model.C[t] * model.P[t]) >= 0
model.positive_revenue = Constraint(model.T,rule=revenue_positive)
What am I missing here ? Thanks for help, just beginning
Welcome to the site.
The problem you appear to be having is that you are not building your model parameter model.P as an indexed component. I believe you likely want it to be indexed by your set model.T.
When you make indexed params in pyomo you need to initialize it with some key:value pairing, like a python dictionary. You can make that from your data frame by re-indexing your data frame so that the quarter labels are the index values.
Caution: The construction you have for model.T and this assume there are no duplicates in the quarter names.
If you have duplicates (or get a warning) then you'll need to do something else. If the quarter labels are unique you can do this:
import pandas as pd
import pyomo.environ as pyo
df = pd.DataFrame({'qtr':['Q5', 'Q6', 'Q7'], 'price':[12.80, 11.50, 8.12]})
df.set_index('qtr', inplace=True)
print(df)
m = pyo.ConcreteModel()
m.T = pyo.Set(initialize=df.index.to_list())
m.price = pyo.Param(m.T, initialize=df['price'].to_dict())
m.pprint()
which should get you:
price
qtr
Q5 12.80
Q6 11.50
Q7 8.12
1 Set Declarations
T : Size=1, Index=None, Ordered=Insertion
Key : Dimen : Domain : Size : Members
None : 1 : Any : 3 : {'Q5', 'Q6', 'Q7'}
1 Param Declarations
price : Size=3, Index=T, Domain=Any, Default=None, Mutable=False
Key : Value
Q5 : 12.8
Q6 : 11.5
Q7 : 8.12
2 Declarations: T price
edit for clarity...
NOTE:
The first argument when you create a pyomo parameter is the indexing set. If this is not provided, pyomo assumes that it is a scalar. You are missing the set as shown in my example and highlighted with arrow here: :)
|
|
|
V
m.price = pyo.Param(m.T, initialize=df['price'].to_dict())
Also note, you will need to initialize model.P with a dictionary as I have in the example, not a list.
I have created a table with a collection. Inserted a record and took sstabledump of it and seeing there is range tombstone for it in the sstable. Does this tombstone ever get removed? Also when I run sstablemetadata on the only sstable, it shows "Estimated droppable tombstones" as 0.5", Similarly it shows one record with epoch time as insert time for - "Estimated tombstone drop times: 1548384720: 1". Does it mean that when I do sstablemetadata on a table having collections, the estimated droppable tombstone ratio and drop times values are not true and dependable values due to collection/list range tombstones?
CREATE TABLE ks.nmtest (
reservation_id text,
order_id text,
c1 int,
order_details map<text, text>,
PRIMARY KEY (reservation_id, order_id)
) WITH CLUSTERING ORDER BY (order_id ASC)
user#cqlsh:ks> insert into nmtest (reservation_id , order_id , c1, order_details ) values('3','3',3,{'key':'value'});
user#cqlsh:ks> select * from nmtest ;
reservation_id | order_id | c1 | order_details
----------------+----------+----+------------------
3 | 3 | 3 | {'key': 'value'}
(1 rows)
[root#localhost nmtest-e1302500201d11e983bb693c02c04c62]# sstabledump mc-5-big-Data.db
WARN 02:52:19,596 memtable_cleanup_threshold has been deprecated and should be removed from cassandra.yaml
[
{
"partition" : {
"key" : [ "3" ],
"position" : 0
},
"rows" : [
{
"type" : "row",
"position" : 41,
"clustering" : [ "3" ],
"liveness_info" : { "tstamp" : "2019-01-25T02:51:13.574409Z" },
"cells" : [
{ "name" : "c1", "value" : 3 },
{ "name" : "order_details", "deletion_info" : { "marked_deleted" : "2019-01-25T02:51:13.574408Z", "local_delete_time" : "2019-01-25T02:51:13Z" } },
{ "name" : "order_details", "path" : [ "key" ], "value" : "value" }
]
}
]
}
SSTable: /data/data/ks/nmtest-e1302500201d11e983bb693c02c04c62/mc-5-big
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.010000
Minimum timestamp: 1548384673574408
Maximum timestamp: 1548384673574409
SSTable min local deletion time: 1548384673
SSTable max local deletion time: 2147483647
Compressor: org.apache.cassandra.io.compress.LZ4Compressor
Compression ratio: 1.0714285714285714
TTL min: 0
TTL max: 0
First token: -155496620801056360 (key=3)
Last token: -155496620801056360 (key=3)
minClustringValues: [3]
maxClustringValues: [3]
Estimated droppable tombstones: 0.5
SSTable Level: 0
Repaired at: 0
Replay positions covered: {CommitLogPosition(segmentId=1548382769966, position=6243201)=CommitLogPosition(segmentId=1548382769966, position=6433666)}
totalColumnsSet: 2
totalRows: 1
Estimated tombstone drop times:
1548384720: 1
Another quuestion was on the nodetool tablestats output - what does slice refer to in cassandra?
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1
Average tombstones per slice (last five minutes): 1.0
Maximum tombstones per slice (last five minutes): 1
Dropped Mutations: 0
sstablemetadata does not have the information about your table that is not held within the sstable as it is not guaranteed to be run on system that has Cassandra running, and even if it was its very complex to be able to know how to pull the schema information from it.
Since the gc_grace_seconds is a table parameter and not in the metadata it defaults to assuming a 0 gc grace so the droppable times listed in that histogram will be more a histogram of the tombstone creation times by default. If you know your gc grace you can add it as a -g parameter to your sstablemetadata call. like:
sstablemetadata -g 864000 mc-5-big-Data.db
see http://cassandra.apache.org/doc/latest/tools/sstable/sstablemetadata.html for information on the tools output.
With collections it's just normal range tombstone with all that it entails. They are used to prevent the requirement of a read-before-write when overwriting the value of a multicell collection.
I tried with another file,While converting to EDI 837 I to XML the below errors arraised. Are these fields are mandatory fields?
I manually passed values to ISA09, ISA10 and ISA13 then I am receiving
Error encountered during parsing. The X12 interchange with id ' ', with sender id ' ', receiver id ' ' had the following errors:
Error: 4 (Field level error) SegmentID: ISA Position in TS: 1 Data
Element ID: ISA09 Position in Segment: 9 Data Value: 8:
Invalid Date
Error: 5 (Field level error) SegmentID: ISA Position in TS: 1 Data
Element ID: ISA10 Position in Segment: 10 Data Value: 9:
Invalid Time
Error: 6 (Field level error) SegmentID: ISA Position in TS: 1 Data
Element ID: ISA13 Position in Segment: 13 Data Value: 6:
Invalid character in data element
ISA09 is YYMMDD formatted.
ISA10 is HHMM formatted.
ISA13 is n8, meaning 8 digits.
ISA is fixed length so numbers must be 0 padded and text space padded.
Have you done the EDI Tutorials for BizTalk?
Trying to substract a number for a variable, which is an int in Ansible.
var:
number: 30
tasks:
- set_fact: me={{ number -1 }}
- debug: var=me
Expectation: me = 29
Result:
fatal: [node1]: FAILED! => {"failed": true, "msg": "Unexpected templating type error occurred on ({{ number - 1 }}): unsupported operand type(s) for -: 'AnsibleUnicode' and 'int'"}
It is a known issue with Ansible/Jinja that you can't preserve numeric type after templating.
Use int filter inside {{..}} expression:
- set_fact: me={{ number | int - 1 }}