Backtrader: How can I have stops execute on the same bar they were entered on? - backtrader

I'm confused about the behavior of stop orders. Consider the example below, in which the stop price is reached on the same bar on which the order is entered. However, the stop is not executed.
In this example, both the buy and the stop orders are placed on the 2022-01-02 bar, with a stop of 10.0. The buy is executed as expected, at a market price of 11.1 (the open price of the bar). However, the stop order is not executed, even though the low of the bar is 10.0.
In general in my tests it appears that stops are never executed on the same bar they are entered on, even though conceptually the orders are placed (and executed) at bar open, an presumably any low price of the bar happens after bar open.
Is this expected behavior and a limitation in Backtrader, and if so, is there a workaround?
Here's some sample output which illustrates this behavior:
Starting Portfolio Value: 10000.00
2022-01-01 OHLC: 10.1, 10.3, 10.0, 10.2
2022-01-01 placing buy order
2022-01-02 1: Buy Market: Submitted: 10 # None
2022-01-02 2: Sell Stop: Submitted: -10 # 10
2022-01-02 3: Sell Limit: Submitted: -10 # None
2022-01-02 1: Buy Market: Completed: 10 # 11.1
2022-01-02 OHLC: 11.1, 11.3, 10.0, 11.2
2022-01-03 3: Sell Limit: Completed: -10 # 12.1
2022-01-03 2: Sell Stop: Canceled
2022-01-03 OHLC: 12.1, 12.3, 12.0, 12.2
2022-01-03 placing buy order
Final Portfolio Value: 10010.00
Here's the code that generates this output:
import backtrader as bt
import pandas as pd
from io import StringIO
class TestStrategy(bt.Strategy):
def log(self, txt, dt=None):
dt = dt or self.datas[0].datetime.date(0)
print(f"{dt.isoformat()} {txt}")
def next(self):
d = self.data
self.log(f"OHLC: {d.open[0]}, {d.high[0]}, {d.low[0]}, {d.close[0]}")
if self.position.size == 0:
self.log("placing buy order")
self.buy_bracket(size=10, stopprice=10, exectype=bt.Order.Market)
def notify_order(self, order):
id = order.ref
created = bt.num2date(order.created.dt).strftime('%Y-%m-%d %H:%M')
status_name = order.Status[order.status]
ordtype_name = order.ordtypename()
exectype_name = order.ExecTypes[order.exectype]
order_header = f"{id}: {ordtype_name} {exectype_name}: {status_name}"
order_info = f"{order.size} # {order.price}"
exec_info = f"{order.executed.size} # {order.executed.price}"
if order.status in [order.Completed]:
self.log(f"{order_header}: {exec_info}")
elif order.status in [order.Submitted, order.Accepted]:
if order.status == order.Submitted:
self.log(f"{order_header}: {order_info}")
else:
self.log(f"{order_header}")
if __name__ == '__main__':
#make sure you use tabs as separators, not spaces in this input string, or change the separator in `read_csv` below
data ="""
datetime open high low close
2022-01-01 10.1 10.3 10.0 10.2
2022-01-02 11.1 11.3 10.0 11.2
2022-01-03 12.1 12.3 12.0 12.2
"""
prices = StringIO(data)
df = pd.read_csv(prices, sep = '\t', parse_dates=True, index_col=0)
pricedata = bt.feeds.PandasData(dataname = df)
cerebro = bt.Cerebro()
cerebro.addstrategy(TestStrategy)
cerebro.adddata(pricedata)
print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())
cerebro.run()
print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())

Related

Constraint issue with pyomo involving a scalar

working on an economic optimization problem with pyomo, I would like to add a constraint to prevent the product of the commodity quantity and its price to go below zero (<0), avoiding a negative revenue. It appears that all the data are in a dataframe and I can't setup a constraint like:
def positive_revenue(model, t)
return model.P * model.C >=0
model.positive_rev = Constraint(model.T, rule=positive_revenue)
The system returns the error that the price is a scalar and it cannot process it. Indeed the price is set as such in the model:
model.T = Set(doc='quarter of year', initialize=df.quarter.tolist(), ordered=True)
model.P = Param(initialize=df.price.tolist(), doc='Price for each quarter')
##while the commodity is:
model.C = Var(model.T, domain=NonNegativeReals)
I just would like to apply that for each timestep (quarter of hour here) that:
price(t) * model.C(t) >=0
Can someone help me to spot the issue ? Thanks
Here are more information:
df dataframe:
df time_stamp price Status imbalance
quarter
0 2021-01-01 00:00:00 64.84 Final 16
1 2021-01-01 00:15:00 13.96 Final 38
2 2021-01-01 00:30:00 12.40 Final 46
index = quarter from 0 till 35049, so it is ok
Here is the df.info()
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 time_stamp 35040 non-null datetime64[ns]
1 price 35040 non-null float64
2 Status 35040 non-null object
3 imbalance 35040 non-null int64
I modified the to_list() > to_dict() in model.T but still facing the same issue:
KeyError: "Cannot treat the scalar component 'P' as an indexed component" at the time model.T is defined in the model parameter, set and variables.
Here is the constraint where the system issues the error:
def revenue_positive(model,t):
for t in model.T:
return (model.C[t] * model.P[t]) >= 0
model.positive_revenue = Constraint(model.T,rule=revenue_positive)
Can't figure it out...any idea ?
UPDATE
Model works after dropping an unfortunate 'quarter' column somewhere...after I renamed the index as quarter.
It runs but i still get negative revenues, so the constraints seems not working at present, here is how it is written:
def revenue_positive(model,t):
for t in model.T:
return (model.C[t] * model.P[t]) >= 0
model.positive_revenue = Constraint(model.T,rule=revenue_positive)
What am I missing here ? Thanks for help, just beginning
Welcome to the site.
The problem you appear to be having is that you are not building your model parameter model.P as an indexed component. I believe you likely want it to be indexed by your set model.T.
When you make indexed params in pyomo you need to initialize it with some key:value pairing, like a python dictionary. You can make that from your data frame by re-indexing your data frame so that the quarter labels are the index values.
Caution: The construction you have for model.T and this assume there are no duplicates in the quarter names.
If you have duplicates (or get a warning) then you'll need to do something else. If the quarter labels are unique you can do this:
import pandas as pd
import pyomo.environ as pyo
df = pd.DataFrame({'qtr':['Q5', 'Q6', 'Q7'], 'price':[12.80, 11.50, 8.12]})
df.set_index('qtr', inplace=True)
print(df)
m = pyo.ConcreteModel()
m.T = pyo.Set(initialize=df.index.to_list())
m.price = pyo.Param(m.T, initialize=df['price'].to_dict())
m.pprint()
which should get you:
price
qtr
Q5 12.80
Q6 11.50
Q7 8.12
1 Set Declarations
T : Size=1, Index=None, Ordered=Insertion
Key : Dimen : Domain : Size : Members
None : 1 : Any : 3 : {'Q5', 'Q6', 'Q7'}
1 Param Declarations
price : Size=3, Index=T, Domain=Any, Default=None, Mutable=False
Key : Value
Q5 : 12.8
Q6 : 11.5
Q7 : 8.12
2 Declarations: T price
edit for clarity...
NOTE:
The first argument when you create a pyomo parameter is the indexing set. If this is not provided, pyomo assumes that it is a scalar. You are missing the set as shown in my example and highlighted with arrow here: :)
|
|
|
V
m.price = pyo.Param(m.T, initialize=df['price'].to_dict())
Also note, you will need to initialize model.P with a dictionary as I have in the example, not a list.

Webscraping RequestGet from Airbnb not working properly

This query is returning 0 or 20 randomly every time i run it. Yesterday when i loop through the pages i always get 20 and I am able to scrape through 20 listings and 15 pages. But now, I can't run my code properly because sometimes the listings return 0.
I tried adding headers in the request get and time sleep (5-10s random) before each request but am still facing the same issue. Tried connecting to hotspot to change my IP but am still facing the same issue. Anyone understand why?
import time
from random import randint
from bs4 import BeautifulSoup
import requests #to connect to url
airbnb_url = 'https://www.airbnb.com/s/Mayrhofen--Austria/homes?tab_id=home_tab&refinement_paths%5B%5D=%2Fhomes&date_picker_type=calendar&query=Mayrhofen%2C%20Austria&place_id=ChIJbzLYLzjdd0cRDtGuTzM_vt4&checkin=2021-02-06&checkout=2021-02-13&adults=4&source=structured_search_input_header&search_type=autocomplete_click'
soup = BeautifulSoup(requests.get(airbnb_url).content, 'html.parser')
listings = soup.find_all('div', '_8s3ctt')
print(len(listings))
It seems AirBnB returns 2 versions of the page. One "normal" HTML and other where the listings are stored inside <script>. To parse the <script> version of page you can use next example:
import json
import requests
from bs4 import BeautifulSoup
def find_listing(d):
if isinstance(d, dict):
if "__typename" in d and d["__typename"] == "DoraListingItem":
yield d["listing"]
else:
for v in d.values():
yield from find_listing(v)
elif isinstance(d, list):
for v in d:
yield from find_listing(v)
airbnb_url = "https://www.airbnb.com/s/Mayrhofen--Austria/homes?tab_id=home_tab&refinement_paths%5B%5D=%2Fhomes&date_picker_type=calendar&query=Mayrhofen%2C%20Austria&place_id=ChIJbzLYLzjdd0cRDtGuTzM_vt4&checkin=2021-02-06&checkout=2021-02-13&adults=4&source=structured_search_input_header&search_type=autocomplete_click"
soup = BeautifulSoup(requests.get(airbnb_url).content, "html.parser")
listings = soup.find_all("div", "_8s3ctt")
if len(listings):
# normal page:
print(len(listings))
else:
# page that has listings stored inside <script>:
data = json.loads(soup.select_one("#data-deferred-state").contents[0])
for i, l in enumerate(find_listing(data), 1):
print(i, l["name"])
Prints (when returned the <script> version):
1 Mariandl (MHO103) for 36 persons.
2 central and friendly! For Families and Friends
3 Sonnenheim for 5 persons.
4 MO's Apartments
5 MO's Apartments
6 Beautiful home in Mayrhofen with 3 Bedrooms
7 Quaint Apartment in Finkenberg near Ski Lift
8 Apartment 2 Villa Daringer (5 pax.)
9 Modern Apartment in Schwendau with Garden
10 Holiday flats Dornau, Mayrhofen
11 Maple View
12 Laubichl Lodge by Apart Hotel Therese
13 Haus Julia - Apartment Edelweiß Mayrhofen
14 Melcherhof,
15 Rest coke
16 Vacation home Traudl
17 Luxurious Apartment near Four Ski Lifts in Mayrhofen
18 Apartment 2 60m² for 2-4 persons "Binder"
19 Apart ZEMMGRUND, 4-9 persons in Mayrhofen/Tirol
20 Apartment Ahorn View
EDIT: To print lat, lng:
...
for i, l in enumerate(find_listing(data), 1):
print(i, l["name"], l["lat"], l["lng"])
Prints:
1 Mariandl (MHO103) for 36 persons. 47.16522 11.85723
2 central and friendly! For Families and Friends 47.16209 11.859691
3 Sonnenheim for 5 persons. 47.16809 11.86694
4 MO's Apartments 47.166969 11.863186
...

How get strings from array of tags in bs4?

After:
soup.select('tr:nth-child(1)')
I got:
[<tr>
<th bgcolor="#5ac05a" colspan="2">Date</th>
<th bgcolor="#a3c35a">T<br/>(C)</th>
<th bgcolor="#c0a35a">Td<br/>(C)</th>
<th bgcolor="#a3c35a">Tmax<br/>(C)</th>
<th bgcolor="#a3c35a">Tmin<br/>(C)</th>
...
</tr>]
How I can take list of strings (Date, T, Td) without manually select each element, like soup.select('tr:nth-child(1) > th:nth-child(5)')[0].text because this works very slow and I have different numbers of th's on different pages?
To get the table to the pandas dataframe, you can use this example:
import re
import requests
import pandas as pd
from bs4 import BeautifulSoup
url = "https://www.ogimet.com/cgi-bin/gsynres?ind=28698&lang=en&decoded=yes&ndays=31&ano=2021&mes=1&day=1"
soup = BeautifulSoup(requests.get(url).content, "html.parser")
header = [
th.get_text(strip=True) for th in soup.thead.select("tr")[0].select("th")
]
all_data = []
for row in soup.thead.select("tr")[1:]:
tds = [td.get_text(strip=True) for td in row.select("td")[:-3]]
tds.insert(0, tds.pop(0) + " " + tds.pop(0))
for td in row.select("td")[-3:]:
img = td.select_one("img[onmouseover]")
if img:
tds.append(re.search(r"'([^']+)'", img["onmouseover"]).group(1))
else:
tds.append("-")
all_data.append(tds)
df = pd.DataFrame(all_data, columns=header)
print(df)
df.to_csv("data.csv", index=False)
Prints:
Date T(C) Td(C) Tmax(C) Tmin(C) ddd ffkmh Gustkmh P0hPa P seahPa PTnd Prec(mm) Nt Nh InsoD-1 Viskm Snow(cm) WW W1 W2
0 01/01/2021 06:00 -30.6 -33.7 ----- -31.1 NNW 7.2 ---- 1027.8 1045.5 +1.5 ---- 0 - --- 20.0 ---- Diamond dust (with or without fog) Snow, or rain and snow mixed Cloud covering more than 1/2 of the sky during...
1 01/01/2021 03:00 -30.7 -33.7 ----- -30.7 NNW 7.2 ---- 1026.2 1044.0 +1.0 Tr/12h 8 8 3.7 10.0 23 Diamond dust (with or without fog) Snow, or rain and snow mixed Cloud covering more than 1/2 of the sky throug...
2 01/01/2021 00:00 -30.1 -33.1 ----- ----- NNW 7.2 ---- 1025.3 1043.0 +0.6 ---- 8 0 --- 10.0 ---- Diamond dust (with or without fog) Snow, or rain and snow mixed Cloud covering more than 1/2 of the sky during...
3 12/31/2020 21:00 -30.5 -33.5 ----- ----- NNW 3.6 ---- 1024.7 1042.4 +0.6 ---- 0 - --- 10.0 ---- Diamond dust (with or without fog) Snow, or rain and snow mixed Cloud covering 1/2 or less of the sky througho...
...and so on
And saves data.csv (screenshot from LibreOffice):

Line profiling with cython in jupyter notebook

I'm trying to use liner_profiler library in jupyter notebook with cython function. It is working only halfway. The result I get only consist of first row of the function and no profiling results.
%%cython -a
# cython: linetrace=True
# cython: binding=True
# distutils: define_macros=CYTHON_TRACE_NOGIL=1
import numpy as np
cimport numpy as np
from datetime import datetime
import math
cpdef np.int64_t get_days(np.int64_t year, np.int64_t month):
cdef np.ndarray months=np.array([31,28,31,30,31,30,31,31,30,31,30,31])
if month==2:
if (year%4==0 and year%100!=0) or (year%400==0):
return 29
return months[month-1]
For the profiling result int onlt shows one line of code
Timer unit: 1e-07 s
Total time: 0.0015096 s
File: .ipython\cython\_cython_magic_0154a9feed9bbd6e4f23e57d73acf50f.pyx
Function: get_days at line 15
Line # Hits Time Per Hit % Time Line Contents
==============================================================
15 cpdef np.int64_t get_days(np.int64_t year, np.int64_t month):
This can be seen as a bug in the line_profiler (if it is supposed to support Cython). To get the code of the profiled function, line_profiler reads the pyx-file and tries to extract the code with help of inspect.getblock:
...
# read pyx-file
all_lines = linecache.getlines(filename)
# try to extract body of the function strarting at start_lineno:
sublines = inspect.getblock(all_lines[start_lineno-1:])
...
However, getblock knows nothing about cpdef-function, as python has only def-functions and thus yields wrong function-body (i.e. only the signature).
Workaround:
A simple work around would be to introduce a dummy def-function, which would be a sentinel for the cpdef-function in such a way, that inspect.getblock would yield the whole body of the cpdef-function + body of the the sentinel function, i.e.:
%%cython
...
cpdef np.int64_t get_days(np.int64_t year, np.int64_t month):
...
def get_days_sentinel():
pass
and now the report %lprun -f get_days get_days(2019,3) looks as follows:
Timer unit: 1e-06 s
Total time: 1.7e-05 s
File: XXXX.pyx
Function: get_days at line 10
Line # Hits Time Per Hit % Time Line Contents
==============================================================
10 cpdef np.int64_t get_days(np.int64_t year, np.int64_t month):
11 1 14.0 14.0 82.4 cdef np.ndarray months=np.array([31,28,31,30,31,30,31,31,30,31,30,31])
12 1 1.0 1.0 5.9 if month==2:
13 if (year%4==0 and year%100!=0) or (year%400==0):
14 return 29
15 1 2.0 2.0 11.8 return months[month-1]
16
17 def get_days_sentinel():
18 pass
There are still somewhat ugly trailing lines from the sentinel, but it is probably better as not seeing anything at all.

Dictionary with a running total

This is for a homework I am doing.
I have a .txt file that looks like this.
11
eggs
1.17
milk
3.54
bread
1.50
coffee
3.57
sugar
1.07
flour
1.37
apple
.33
cheese
4.43
orange
.37
bananas
.53
potato
.19
What I'm trying to do is keep a running total, when you type in the word "Eggs" then the word "bread" it needs to add the cost of both and keep going until "EXIT" also I'm going to run into a 'KeyError' and need help with that also.
def main():
key = ''
infile = open('shoppinglist.txt', 'r')
total = 0
count = infile.readline()
grocery = ''
groceries = {}
print('This program keeps a running total of your shopping list.')
print('Use \'EXIT\' to exit.')
while grocery != 'EXIT':
grocery = input('Enter an item: ')
for line in infile:
line = line.strip()
if key == '':
key = line
else:
groceries[key] = line
key = ''
print ('Your current total is $'+ groceries[grocery])
main()
Does the file contain the prices of each of the different groceries?
The user input statement should have a .strip() at the end too as sometimes line ending characters can be included from user input.
You should only need to read the file once, not in the loop.
When the user enters a grocery item it should as you say check that it exists:
if grocery in groceries:
...
else:
#grocery name not recognised
I think you should have a separate dictionary to store the counts of each grocery something like this: http://docs.python.org/library/collections.html#collections.Counter
import collections
quantitiesWanted = collections.Counter()
Then any grocery can be asked for like this quantitiesWanted['eggs'] which will return 0 by default. Doing something like quantitiesWanted['eggs'] += 1 will increase it to 1 and so on.
To get the current total you can do:
total = 0
for key, value in quantitiesWanted:
total += groceries[key] * value

Resources