I work on a project and I want to let the choice of using ammonite at each developer.
So I have to share the initialCommands to avoid dups.
Here are my current solution.
val initCommands = s""" val sc = new org.apache.spark.SparkContext("local", "shell"); val sqlContext = new org.apache.spark.sql.SQLContext(sc); import sqlContext.implicits._; import org.apache.spark.sql.functions._ """
val escaped = initCommands.replace("\"", "\\\"")
initialCommands in console := initCommands
initialCommands in (Test, console) := s"""ammonite.Main(predef = "$escaped").run()"""
First issue is that initCommands have to be on one line.
Is there a better way ?
Related
I want learn methods refineXXX of zio library.
For this purpose I write simple code
import zio.ZIOAppDefault
import zio.Unsafe.unsafe
import zio._
import java.sql.SQLException
object Bot extends ZIOAppDefault {
val codeWithExc: ZIO[Any,Throwable,Int] =
ZIO.fail(new SQLException("message of SQL error."))
val MainApp: ZIO[Any, Throwable, Int] = for {
_ <- Console.printLine("Begin")
res <- codeWithExc
_ <- Console.printLine(s" res = $res")
} yield res
def run: URIO[ZIOAppArgs, ExitCode] =
for {
res <- MainApp.exitCode
} yield res
}
unsafe{ implicit u =>
Runtime.default.unsafe.run(Bot.run.provide(ZIOAppArgs.empty))
}
When I run it in IDEA worksheet I see output:
Begin
res0: zio.Exit[Nothing,zio.ExitCode] = Success(ExitCode(1))
and expect fail in res <- codeWithExc and death of the main fiber.
The problem here is that You are mapping ZIO to .exitCode which returns URIO that cannot really fail. Failure will be mapped to success and the only reason to tell that it did actually fail is to verify that ExitCode was not 0. If You change Your code and replace that with something like below, it will look work correctly:
def run = MainApp
While I was running my code in pycharm, it is showing:
__conform__() is not a valid Streamlit command."
I am trying to store input and result in my streamlit app in sqlite3 database and and display it in tabular format in same streamlit app.
Main code page:home_task.py:
import streamlit as st
from homework_db import create_table, add_data
def main():
st.title("Streamlit Exercise")
menu = ['Insert', 'Read']
choice = st.sidebar.selectbox("Menu", menu)
create_table()
if choice == 'Insert':
st.subheader('Lets check Sentiment')
line = st.text_area("Enter the sentence")
result = st.text("Positive")
if st.button("Add Task"):
add_data(line, result)
st.success("Successfully added data in database")
elif choice == 'Read':
st.subheader('Datatable')
if __name__ == "__main__":
main()
The other file: homework_db.py:
import sqlite3
conn = sqlite3.connect("homework2_db", check_same_thread=False)
c = conn.cursor()
## database-table-field-datatype##
def create_table():
c.execute('CREATE TABLE IF NOT EXISTS database(sentence TEXT, sentiment TEXT)')
def add_data(line, result):
c.execute('INSERT INTO database(sentence,sentiment) VALUES(?,?)', (line, result))
conn.commit()
Everything seems fine, I have followed many youtube videos for sql query, it seems right but I am not sure why streamlit is not accepting the code.
I also checked python docs, (https://docs.python.org/3.6/library/sqlite3.html#letting-your-object-adapt-itself) but could not figure out how this is related to my problem.
It's because result is not a string but a Streamlit object:
>>> import streamlit
>>> result = st.text("Positive")
>>> type(result)
<class 'streamlit.delta_generator.DeltaGenerator'>
What you need is to do is remove the st.text():
if choice == 'Insert':
st.subheader('Lets check Sentiment')
line = st.text_area("Enter the sentence")
result = "Positive" # no st.text()
if st.button("Add Task"):
add_data(line, result)
st.success("Successfully added data in database")
I would like to know how I should change my code to prevent it from Injections:
import sqlite3
def search_in_database(column,searched_data):
con = sqlite3.connect('db.sqlite3')
cursor = con.cursor()
cursor.execute(f"""SELECT
id
FROM
My_library
WHERE
{column} LIKE '%{searched_data}%'
;""")
all = [i for i in cursor.fetchall()]
return all
I found code in web which gives an example of how to do it:
from sqlalchemy.sql import text
# Create a connection conn
stmt = text("""SELECT * FROM users
WHERE user = :username AND password = :password""")
conn.execute(stmt, prams={"username": "foo", "password": "bar"})
but In my HTML file I would like to give to user possibility to choose the:
Place where he wants to search in Titles, authors, published_dates,isbn, language...
and when he choose where He what to search then he types the query.
How to do it in this case, avoiding Injections?
My data base:
class My_library(db.Model):
id = db.Column(db.Integer, primary_key=True)
title = db.Column(db.String(250))
authors = db.Column(db.String(100))
published_date = db.Column(db.Integer)
isbn_or_identifier = db.Column(db.String(15), unique=True)
page_count = db.Column(db.String(10000))
language = db.Column(db.String(3))
image_links = db.Column(db.String(500))
I also added validators:
from flask_wtf import FlaskForm
from wtforms import SubmitField,StringField
from wtforms.validators import ValidationError,DataRequired,Length, URL
from wtforms.fields.html5 import DateField,IntegerField,DateField,IntegerField, URLField
class AddRecValidators(FlaskForm):
title = StringField(label=('Title:'),validators=[DataRequired(), Length(min=1,max=50)])
authors = StringField(label=('Authors:'),validators=[Length(min=1,max=100)])
published_date = IntegerField(label=('Published date:'),validators=[Length(min=1,max=4)])
isbn_or_identifier = IntegerField(label=('ISBN:'),validators=[Length(min=1,max=15)])
page_count = IntegerField(label=('Page count:'),validators=[ Length(min=1,max=10000)])
language = StringField(label=('Language:'),validators=[ Length(min=1,max=3)])
image_links = URLField(label=('Image links:'))
submit = SubmitField(label=('Add to library'))
Thanks for help in advance :D
You can use sqlalchemy to build the query. For example:
q = My_library.__table__.select().where(
My_library.__table__.c.title == "The searched title"
)
but that's not exactly what you wanted. You can also address the columns by their names like this:
q = My_library.__table__.select().where(
My_library.__table__.c["title"] == "The searched title"
)
# or
q = My_library.__table__.select().where(
My_library.__table__.c["title"].like("%The searched title%")
)
Therefore you can do this:
q = My_library.__table__.select().where(
My_library.__table__.c[column].like(f"%{searched_data}%")
)
cursor.execute(q)
In case you only want the ID, you would do this:
q = sqlalchemy.select([My_library.__table__.c.id]).where(
My_library.__table__.c[column].like(f"%{searched_data}%")
)
# you can print(q) to see what it constructed
cursor.execute(q)
That was SQLAlchemy Query Language. You are using ORM. I suggest you read-up something about a session in flask first.
It is still possible to get to the column-name related attribute and I am not sure this is the most efficient way:
q = session.query(My_library.id).filter(
getattr(My_library, column).like(f"%{searched_data}%"),
)
I've created a dictionary from an Uploaded file in Django.
This dictionary has a nested list of dictionaries:
file = {"name": "filename", "sections": [{"section_name": "string", "lines": [{line_number: 0, "line"; "data"}]}], "etc": "etc"}
The model represents the dictionaries depth too.
class Line(EmbeddedDocument):
line_number = IntField()
line = StringField()
definition = ReferenceField(Definition)
class Section(EmbeddedDocument):
section_name = StringField()
lines = EmbeddedDocumentListField(Line))
class File(Document):
name = StringField()
sections = EmbeddedDocumentListField(Section))
created_on = DateTimeField()
created_by = StringField()
modified_on = DateTimeField()
modified_by = StringField()
In the POST I have the following to chop the file up into the above Dict (the file is a simple text file):
file= {}
with open(os.path.join(path, filename + ".txt"), 'r') as temp_file:
filelines = temp_file.readlines()
sections = []
section = {}
lines = []
for i, l in enumerate(filelines):
if i == 0:
section["section_name"] = "Top"
elif '*' in l:
if l.index('*') == 0 and '*' not in lines[len(lines) - 2"line"]:
section["lines"] = lines
lines = []
sections.append(section)
section = dict()
section["section_name"] = filelines[i + 1][1:-2]
line = {"line_number": i + 1, "line": l}
lines.append(line)
section['lines'] = lines
sections.append(section)
file["name"] = filename
file["sections"] = sections
I will tidy this up eventually.
Once the dict has been made how do I serialise it using the serializer?
Is it possible to insert this into a serializer?
If not how can I get it all into the database with validation?
I've tried json.dumps() and JsonRequst() then putting them in data= for the serializer but get Unable to get repr for <class '....'>
I'm pretty new to Django and MongoDB so if you need more info I can provide :)
Thanks!
Update
Change the model's List Fields to EmbeddedDocumentListField as suggest in the answer.
Answered
Thanks to Boris' suggestion below it pointed me to an error I wasn't getting initially. I had a typo and passing the dict directly into FileSerializer(data=file) works like a charm! :)
James!
The easiest way to validate that your incoming JSONs adhere to the Mongoengine Documents schema that you've specified is to use DRF-Mongoengine's DocumentSerializer.
Basically, what you need to do is create a serializer
serializers.py
import rest_framework_mongoengine
class FileSerializer(rest_framework_mongoengine.DocumentSerializer):
class Meta:
fields = '__all__'
model = File
Then you need a view or viewset that makes use of this Serializer to respond to GET/POST/PUT/DELETE requests.
views.py
from rest_framework_mongoengine import viewsets
class FileViewSet(viewsets.ModelViewSet):
lookup_field = 'id'
serializer_class = FileSerializer
def get_queryset(self):
return File.objects.all()
and register this viewset with a router
urls.py
from rest_framework import routers
# this is DRF router for REST API viewsets
router = routers.DefaultRouter()
# register REST API endpoints with DRF router
router.register(r'file', FileViewSet, r"file")
I'd also recommend using EmbeddedDocumentListField instead of ListField(EmbeddedDocumentField(Section)) - it has additional methods.
I am just the beginner of python, so sorry about this basic problem
enter image description here
import pymongo
client = pymongo.MongoClient('localhost', 27017)
ceshi = client['ceshi']
item_info = ['item_info3']
for i in item_info:
if i.find('area'):
print(i)
I am using Jupiter to edit, but after import pymongo , I can't list the data I stored in MongoDB. What could be the reason?
from bs4 import BeautifulSoup
import requests
import time
import pymongo
client = pymongo.MongoClient('localhost',27017)
ceshi = client['ceshi']
url_list = ceshi['url_list3']
item_info = ceshi['item_info3']
def get_links_from(channel,pages,who_sells=0):
#http://bj.58.com/diannao/0/pn2/
list_view = '{}{}/pn{}/'.format(channel,str(who_sells),str(pages))
wb_date = requests.get(list_view)
time.sleep(1)
soup = BeautifulSoup(wb_date.text,'lxml')
links = soup.select('tr.zzinfo > td.t > a.t')
if soup.find('td','t'):
for link in links:
item_link = link.get('href').split('?')[0]
else:
pass
url_list.insert_one({'url':item_link})
print(url_list)
def get_item_info(url):
wb_data = requests.get(url)
soup = BeautifulSoup(wb_data.text,'lxml')
no_longer_exist = '404' in
soup.find('script',type="text/javascript").get('src').split('/')
if no_longer_exist:
pass
else:
title = soup.title.text
price = soup.select('span.price.c_f50')[0].text
date = soup.select('li.time')[0].text
area = list(soup.select('spam.c_25d a')[0].stripped_strings) if
soup.find_all('c_25d') else None
item_info.insert_one({'title':title,'price':price,'date':date,'area':area})
print(item_info)
#print({'title':title,'price':price,'date':date,'area':area})
get_item_info('http://bj.58.com/pingbandiannao/25347275157966x.shtml')
#get_links_from('http://bj.58.com/yunfuyongpin/',2)
It looks like you are looping through an array of strings and trying to call find on a string:
item_info = ['item_info3']
for i in item_info:
if i.find('area')
If item_info3 is the name of your mongo collection in the ceshi database then you should do something like:
item_info = ceshi['item_info3']
Even then, I don't believe your find query is correct. It should be something like this:
for i in item_info.find():
print(i)
More information on Mongo/python:
https://docs.mongodb.org/getting-started/python/query/