To anyone that can help, thank you. I am getting a very strange error when running a INSERT statement using pyodbc. The error code is:
cursor.execute(QueryInsert,params)
pyodbc.DataError: ('22008', '[22008] [Microsoft][ODBC Microsoft Access
Driver]Datetime field overflow (36) (SQLExecDirectW)')
This happens consistently with the datetime 1986-03-28 00:00:00
The code I am using is:
###Necessary Imports
from fredapi import Fred
import pyodbc
import datetime
###Connect to Access Database
conn = pyodbc.connect(r"Driver={Microsoft Access Driver (*.mdb, *.accdb)};"
+r"DBQ=G:\Financial Modelling\Lease Database v1.0.accdb;")
cursor = conn.cursor()
###3M Libor
SourceCode = 'GBP3MTD156N'
fred = Fred(api_key='insert-api-key')
data = fred.get_series_all_releases(SourceCode)
A = data.shape[0]
###Cycle Through Results
for i in range(1,A):
date1 = data.loc[i,'date']
print(date1)
###execute query at date and only upload if empty
existquery = "SELECT * FROM EconVars WHERE SourceCode = '" + SourceCode + "'
AND ValueDate = " + \
"#"+str(date1.month)+"/"+str(date1.day)+"/"+str(date1.year)+"#"
cursor.execute(existquery)
existData = cursor.fetchall()
###check if empty
if len(existData) == 0:
value1 = data.loc[i,'value']
Description = '3M Libor'
Source1 = 'Fred'
params = (date1,value1,Description,Source1,SourceCode)
QueryInsert = """INSERT into EconVars (ValueDate, ReportedValue,
Description, Source,SourceCode)
Values(?,?,?,?,?)"""
cursor.execute(QueryInsert,params)
cursor.commit()
###Commit Cursor for 3M LIBOR
cursor.commit()
cursor.close()
The table in the access file I am using has 5 columns
ValueDate defined as Date/Time (Short Date)
ReportedValue as Number (Double)
Description as Short Text
Source As Short Text
SourceCode As Short Text
Has anyone seen this error before or able to replicate it?
Python 3.7.2 64bit pyodbc 4.0.25 W10 64bit and Office 365 64bit
Thank you to anyone who has any ideas.
I figured out the problem is not with the datetime column provided by the API.
It is actually that the reported value, which should be a double, when missing is actually 'NaT' which I believe is a numpy NULL value or something equivalent.
Access is expecting a double value.
Here's an example of how to change your datetime field to what OP is saying in their answer that worked for me (in pandas):
import pandas as pd
df = pd.DataFrame(['01/01/2019',None], columns=['datetime_field'])
df['datetime_field'] = pd.to_datetime(df['datetime_field'])
df['datetime_field'] = pd.to_datetime(df['datetime_field'], errors='coerce').where(df['datetime_field'].notnull(), 0.0)
Originally the nulls in this field are NaT.
pandas where docs
Related
Hello I am new I had a question so I am trying to create a simple api using flask ..
the data I have is in CSV and I want to import it in to SQLlite file .. which I have done. and can access the data
after I data is loaded and I have conformed there is data. I try and get python to reflect the class .. as I need to confirm its there for flask ..
below is what I type.
Python:
Base = automap_base()
Base.prepare(engine, reflect=True)
Base.classes.keys()
I get nothing
know why I am getting nothing I its because there is no class set up before I load the data using pandas ..
below is the code I use to load the data to sqlite;
Base = declarative_base()
engine = create_engine("sqlite:///countrytwo.sqlite")
Base.metadata.create_all(engine)
file_name = 'us.csv'
df=pd.read_csv(file_name)
df.to_sql('us',con=engine, index_label='id', if_exists='replace')
## then to conform theirs data I do below ##
print (engine.table_names())
so do I know I need to set up the class first then load the data in to the sqlite file .. one does any one have a good webiste to do this ..
I would love a clue lead me to the answer but maybe not give me the answer ..
if this is unclear let me know I can load more code. thank you.
import sqlite3
import pandas as pd
import os
class readCSVintoDB():
def __init__(self):
'''
self.csvobj = csvOBJ
self.dbobj = dbOBJ
'''
self.importCSVintoDB()
def importCSVintoDB(self):
userInput= input("enter the path of the csv file: ")
csvfile = userInput
df = pd.read_csv(csvfile,sep=';')
#print("dataFrame Headers is {0}".format(df.columns))# display the Headers
dp = (df[['date','temperaturemin','temperaturemax']])
print(dp)
'''
check if DB file exist
if no create an empty db file
'''
if not(os.path.exists('./rduDB.db')):
open('./rduDB.db','w').close()
'''
connect to the DB and get a connection cursor
'''
myConn = sqlite3.connect('./rduDB.db')
dbCursor = myConn.cursor()
'''
Assuming i need to create a table of (Name,FamilyName,age,work)
'''
dbCreateTable = '''CREATE TABLE IF NOT EXISTS rduWeather
(id INTEGER PRIMARY KEY,
Date varchar(256),
TemperatureMin FLOAT,
TemperatureMax FLOAT)'''
dbCursor.execute(dbCreateTable)
myConn.commit()
'''
insert data into the database
'''
for i in dp:
print(i)
dbCursor.execute('''
INSERT INTO rduWeather VALUES (?,?,?,?)''', i)
myConn.commit()
mySelect=dbCursor.execute('''SELECT * from rduWeather WHERE (id = 10)''')
print(list(mySelect))
myConn.close()
test1 = readCSVintoDB()
Using :
-IronPython
-AutoDesk
-Revit (PyRevit)
-Revit API
-SQLite3
My code is as follows:
try:
conn = sqlite3.connect('SQLite_Python.db')
c = conn.cursor()
print("connected")
Insert_Volume = """INSERT INTO Column_Coordinates
(x, y, z)
VALUES
(1, 2, 3)"""
count = c.execute(Insert_Volume)
conn.commit()
print("Volume values inserted", c.rowcount)
c.close()
except sqlite3.Error as error:
print("Failed to insert data into sqlite table", error)
finally:
if (conn):
conn.close()
print("The SQLite connection is closed")'''
This code used to work within PyRevit, but now does not, with the following error:
Exception : System.IO.IOException: Could not add reference to assembly IronPython.SQLite
Please advise, this is one of the early steps of a large project and therefore is delaying my work quite a bit.
I look forward to your reply.
I am writing a program that scrapes tweets of a number of people, if the body of the tweet is unique it will get stored in the sqlite database for that person. I have two files, one to write to the databases and one to read the database for and search for tweets with a search word. Before writing to databases I printed the tweets on the terminal, the tweets are being pulled from twitter correctly. When I try a search a term all databases have zero tweets, even if there is no term. There is either a problem with the writing or reading of the database. Please help, I appreciate that I am very new to python.
the writing file:
import requests
import datetime
from bs4 import BeautifulSoup
from peewee import *
from time import sleep
databases = ["femfreq.db", "boris_johnson.db", "barack_obama.db",
"daily_mail.db", "guardian.db", "times.db", "zac_goldsmith.db",
"bernie_sanders.db", "george_osborne.db", "john_mcdonnell.db",
"donald_trump.db", "hillary_clinton.db", "nigel_farage.db"]
urls = ["https://twitter.com/femfreq", "https://twitter.com/BorisJohnson",
"https://twitter.com/BarackObama",
"https://twitter.com/MailOnline?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor",
"https://twitter.com/guardian?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor",
"https://twitter.com/thetimes",
"https://twitter.com/ZacGoldsmith?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor",
"https://twitter.com/berniesanders?lang=en-gb",
"https://twitter.com/George_Osborne?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor",
"https://twitter.com/johnmcdonnellMP?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor",
"https://twitter.com/realDonaldTrump?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor",
"https://twitter.com/HillaryClinton?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor"
"https://twitter.com/Nigel_Farage?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor"]
selection = 0
for database_chosen in databases:
r = requests.get(urls[selection])
soup = BeautifulSoup(r.content, "html.parser")
content =soup.find_all("div",
{"class":
"content"})
db = SqliteDatabase(database_chosen)
class data_input(Model):
time_position = DateTimeField(default=datetime.datetime.now)
header = CharField()
time_posted = CharField()
tweet_body = CharField(unique=True)
class Meta:
database = db
db.connect()
db.create_tables([data_input], safe=True)
for i in content:
try:
data_input.create(header = i.contents[1].text,
time_posted = i.contents[3].text,
tweet_body = i.contents[5].text)
except IntegrityError:
pass
for i in content:
print("=============")
print(i.contents[1].text)
print(i.contents[3].text)
print(i.contents[5].text)
selection += 1
print("database: {} updated".format(database_chosen))
For the reading file
from peewee import *
import datetime
databases = ["femfreq.db", "boris_johnson.db", "barack_obama.db",
"daily_mail.db", "guardian.db", "times.db", "zac_goldsmith.db",
"bernie_sanders.db", "george_osborne.db", "john_mcdonnell.db",
"donald_trump.db", "hillary_clinton.db", "nigel_farage.db"]
search_results = []
search_index = 0
print("")
print("Please enter the number for the database you want to search: ")
for i in databases:
print("{}:{}".format(i, search_index))
search_index += 1
select = int(input("please select: "))
database_chosen = databases[select]
db = SqliteDatabase(database_chosen)
class data_input(Model):
time_position = DateTimeField(default=datetime.datetime.now)
header = CharField()
time_posted = CharField()
tweet_body = CharField(unique=True)
class Meta:
database = db
db.connect()
enteries = data_input.select().order_by(data_input.time_position.desc())
print(enteries)
enteries = enteries.where(data_input.tweet_body)
print("")
print("The total number of tweets in {} is: {}".format(database_chosen,
len(enteries)))
For the reading file I haven't put in a search function yet I will move to that when I can get this problem first. Many thanks
What are you intending to accomplish by putting ".where(data_input.tweet_body)" in the query to read entries? Try removing that whole line:
entries = entries.where(data_input.tweet_body)
When you go to add your search, at that time you will want to add a where clause...something like:
entries = entries.where(data_input.tweet_body.contains(search_term))
I am doing a project in Python, django rest framework. I am using haystack SearchQuerySet. My code is here.
from haystack import indexes
from Medications.models import Salt
class Salt_Index(indexes.SearchIndex, indexes.Indexable):
text = indexes.CharField(document=True, use_template=True)
name = indexes.CharField(model_attr='name',null=True)
slug = indexes.CharField(model_attr='slug',null=True)
if_i_forget = indexes.CharField(model_attr='if_i_forget',null=True)
other_information = indexes.CharField(model_attr='other_information',null=True)
precautions = indexes.CharField(model_attr='precautions',null=True)
special_dietary = indexes.CharField(model_attr='special_dietary',null=True)
brand = indexes.CharField(model_attr='brand',null=True)
why = indexes.CharField(model_attr='why',null=True)
storage_conditions = indexes.CharField(model_attr='storage_conditions',null=True)
side_effects = indexes.CharField(model_attr='side_effects',null=True)
def get_model(self):
return Salt
def index_queryset(self, using=None):
return self.get_model().objects.all()
and my views.py file is -
from django.views.generic import View
from haystack.query import SearchQuerySet
from django.core import serializers
class Medication_Search_View(View):
def get(self,request,format=None):
try:
get_data = SearchQuerySet().all()
print get_data
serialized = ss.serialize("json", [data.object for data in get_data])
return HttpResponse(serialized)
except Exception,e:
print e
my python manage.py rebuild_index is working fine (showing 'Indexing 2959 salts') but in my 'views.py' file , SearchQuerySet() is returning an empty query set...
I am very much worried for this. Please help me friends if you know the reason behind getting empty query set while I have data in my Salt model.
you should check app name it is case sensitive.try to write app name in small letters
My problem is solved now. The problem was that i had wriiten apps name with capital letters and the database tables were made in small letters(myapp_Student). so it was creating problem on database lookup.
I know that there are simmilar questions like this on the forum, however I am still having problems to update a datetime field o the database. I dont get any problems when inserting but I get problems when updating and I am formating the same way , like this:
e.Values.Item("SelectionStartDate") = Format(startdate, "yyyy-MM-dd")
+ " " + startTime1 + ".000"
startTime is of type string.
I have tried different solution that I came across on the internet but still get this error.
Please help.
Thanks in advance
Try using DateTime.TryParse with appropriate format
if the insert work, I think the problem is on your calculation (the + " " + startTime1 + ".000")
try removing it (update the date to the same date) just for testing. If it works I suggest formatting the date on startdate and pass it. you can use #Johnny_D's method.
EDIT:
System.TimeSpan addDate = new System.TimeSpan(1, 0, 0, 0); //add one day
System.DateTime new_date = startdate.Add(addDate);
e.Values.Item("SelectionStartDate") = new_date;