I have file, which should be updated after updating instance attributes
It looks like this:
controller.rb
def update
if UpdateNewspaperDesign.new(newspaper_design, newspaper_design_params).call
#render response
end
end
and
class UpdateNewspaperDesign
def initialize(newspaper_design, params)
#newspaper_design = newspaper_design
#params = params
end
def call
newspaper_design.assign_attributes(params)
File.open(File.join(Rails.root, 'tmp', "newspaper-design-#{Time.now.to_i}.css"), 'w') do |file|
file.write(css(newspaper_design))
newspaper_design.css = file
end
newspaper_design.save
end
private
attr_reader :newspaper_design, :params
def css(newspaper_design)
NewspaperDesignCssCompiler.new(newspaper_design).call
end
end
And then I include css in layout
= stylesheet_link_tag newspaper.design.css_url, media: 'all'
As I see in console: css url is updated, but the url, which is inserted in layout - is not update.
Replace File with Tempfile
- this is how I fixed it
Related
I am trying to write a crawler that goes to a website and searches for a list of keywords, with max_Depth of 2. But the scraper is supposed to stop once any of the keyword's appears on any page, the problem i am facing right now is that the crawler does-not stop when it first see's any of the keywords.
Even after trying, early return command, break command and CloseSpider Commands and even python exit commands.
My class of the Crawler:
class WebsiteSpider(CrawlSpider):
name = "webcrawler"
allowed_domains = ["www.roomtoread.org"]
start_urls = ["https://"+"www.roomtoread.org"]
rules = [Rule(LinkExtractor(), follow=True, callback="check_buzzwords")]
crawl_count = 0
words_found = 0
def check_buzzwords(self, response):
self.__class__.crawl_count += 1
crawl_count = self.__class__.crawl_count
wordlist = [
"sfdc",
"pardot",
"Web-to-Lead",
"salesforce"
]
url = response.url
contenttype = response.headers.get("content-type", "").decode('utf-8').lower()
data = response.body.decode('utf-8')
for word in wordlist:
substrings = find_all_substrings(data, word)
for pos in substrings:
ok = False
if not ok:
if self.__class__.words_found==0:
self.__class__.words_found += 1
print(word + "," + url + ";")
STOP!
return Item()
def _requests_to_follow(self, response):
if getattr(response, "encoding", None) != None:
return CrawlSpider._requests_to_follow(self, response)
else:
return []
I want it to stop execution when if not ok: is True.
When I want to stop a spider, I usually use the exception exception scrapy.exceptions.CloseSpider(reason='cancelled') from Scrapy-Docs.
The example there shows how you can use it:
if 'Bandwidth exceeded' in response.body:
raise CloseSpider('bandwidth_exceeded')
In your case something like
if not ok:
raise CloseSpider('keyword_found')
Or is that what you meant with
CloseSpider Commands
and already tried it?
I'm trying to parse a directory with a collection of xml files from RSS feeds.
I have a similar code for another directory working fine, so I can't figure out the problem. I want to return the items so I can write them to a CSV file. The error I'm getting is:
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column 0
Here is the site I've collected RSS feeds from: https://www.ba.no/service/rss
It worked fine for: https://www.nrk.no/toppsaker.rss and https://www.vg.no/rss/feed/?limit=10&format=rss&categories=&keywords=
Here is the function for this RSS:
import os
import xml.etree.ElementTree as ET
import csv
def baitem():
basepath = "../data_copy/bergens_avisen"
table = []
for fname in os.listdir(basepath):
if fname != "last_feed.xml":
files = ET.parse(os.path.join(basepath, fname))
root = files.getroot()
items = root.find("channel").findall("item")
#print(items)
for item in items:
date = item.find("pubDate").text
title = item.find("title").text
description = item.find("description").text
link = item.find("link").text
table.append((date, title, description, link))
return table
I tested with print(items) and it returns all the objects.
Can it be how the XML files are written?
Asked a friend and said to test with a try except statement. Found a .DS_Store file, which only applies to Mac computers. I'm providing the solution for those who might experience the same problem in the future.
def baitem():
basepath = "../data_copy/bergens_avisen"
table = []
for fname in os.listdir(basepath):
try:
if fname != "last_feed.xml" and fname != ".DS_Store":
files = ET.parse(os.path.join(basepath, fname))
root = files.getroot()
items = root.find("channel").findall("item")
for item in items:
date = item.find("pubDate").text
title = item.find("title").text
description = item.find("description").text
link = item.find("link").text
table.append((date, title, description, link))
except Exception as e:
print(fname, e)
return table
I have upgraded Refinery CMS to the newest version (2.1.0), where there is a new approach in rendering the navigation menu :
(in partial _header.html.erb)
<%= Refinery::Pages::MenuPresenter.new(refinery_menu_pages, self).to_html %>
The older version of the same partial :
<%= render(:partial => "/refinery/menu", :locals => {
:dom_id => 'menu',
:css => 'menu'
}) %>
How could I add bootstrap styles to the navbar using MenuPresenter?
It can be done, but the solution is not pretty because the Menu Presenter in Refinery 2.1 doesn't support all the right CSS options out of the box. But with a bit of perseverance, this is roughly what to do:
Firstly, create a new blank file here: config/initializers/refinery/monkey_patch_menu_presenter.rb
In this patch file, paste in the contents of this updated version of the menu presenter (published October 2013): menu_presenter.rb
Next, based on the instructions in section 5 of the menu presenter guide, in your app/helpers/application_helper.rb file, add a new method called navigation_menu:
def navigation_menu
presenter = Refinery::Pages::MenuPresenter.new(refinery_menu_pages, self)
presenter.css = "navbar-inner"
presenter.menu_tag = :div
presenter.list_tag_css = "nav"
presenter.selected_css = "active"
presenter.first_css = ""
presenter.last_css = ""
presenter.max_depth = 0 # prevents dropdown menus, which don't render correctly
presenter
end
Finally, in your app/views/refinery/_header.html.erb file (use $ bundle exec rake refinery:override view=refinery/_header if it doesn't exist), replace the call for:
<%= Refinery::Pages::MenuPresenter.new(refinery_menu_pages, self).to_html %>
with:
<div class="navbar">
<%= navigation_menu.to_html %>
</div>
Ensure that you have the loaded the Bootstrap CSS/JS files and have wrapped the whole page in a <div class="container"> element. Then restart your application for the patch to take affect and hopefully you'll see a familiar bootstrap navigation bar.
Good luck!
Martyn.
Here a version of above menu_presenter.rb that renders sub-menus as well
(This if for Bootstrap 3, RefineryCMS 2.1.1):
require 'active_support/core_ext/string'
require 'active_support/configurable'
require 'action_view/helpers/tag_helper'
require 'action_view/helpers/url_helper'
module Refinery
module Pages
class MenuPresenter
include ActionView::Helpers::TagHelper
include ActionView::Helpers::UrlHelper
include ActiveSupport::Configurable
config_accessor :roots, :menu_tag, :list_tag, :list_item_tag, :css, :dom_id,
:max_depth, :selected_css, :first_css, :last_css, :list_tag_css,
:link_tag_css
self.dom_id = 'menu'
self.css = "collapse navbar-collapse"
self.menu_tag = :div
self.list_tag = :ul
self.list_item_tag = :li
self.selected_css = 'active'
self.first_css = :first
self.last_css = :last
self.list_tag_css = "nav navbar-nav"
def roots
config.roots.presence || collection.roots
end
attr_accessor :context, :collection
delegate :output_buffer, :output_buffer=, :to => :context
def initialize(collection, context)
#collection = collection
#context = context
end
def to_html
render_menu(roots) if roots.present?
end
private
def render_menu(items)
content_tag(menu_tag, :id => dom_id, :class => css) do
render_menu_items(items)
end
end
def render_menu_items(menu_items)
if menu_items.present?
content_tag(list_tag, :class => list_tag_css) do
menu_items.each_with_index.inject(ActiveSupport::SafeBuffer.new) do |buffer, (item, index)|
buffer << render_menu_item(item, index)
end
end
end
end
def render_menu_items_children(menu_items)
if menu_items.present?
content_tag(list_tag, :class => 'dropdown-menu') do
menu_items.each_with_index.inject(ActiveSupport::SafeBuffer.new) do |buffer, (item, index)|
buffer << render_menu_item(item, index)
end
end
end
end
def render_menu_item_link_dropdown(menu_item)
link_to( menu_item.title, context.refinery.url_for(menu_item.url), class: "dropdown-toggle", data: {toggle:"dropdown", target: "#"})
end
def render_menu_item_link(menu_item)
link_to(menu_item.title, context.refinery.url_for(menu_item.url), :class => link_tag_css)
end
def render_menu_item(menu_item, index)
content_tag(list_item_tag, :class => menu_item_css(menu_item, index)) do
buffer = ActiveSupport::SafeBuffer.new
# Check for sub menu
menu_item_children(menu_item).empty? ? buffer << render_menu_item_link(menu_item) : buffer << render_menu_item_link_dropdown(menu_item)
buffer << render_menu_items_children(menu_item_children(menu_item))
buffer
end
end
# Determines whether any item underneath the supplied item is the current item according to rails.
# Just calls selected_item? for each descendant of the supplied item
# unless it first quickly determines that there are no descendants.
def descendant_item_selected?(item)
item.has_children? && item.descendants.any?(&method(:selected_item?))
end
def selected_item_or_descendant_item_selected?(item)
selected_item?(item) || descendant_item_selected?(item)
end
# Determine whether the supplied item is the currently open item according to Refinery.
def selected_item?(item)
path = context.request.path
path = path.force_encoding('utf-8') if path.respond_to?(:force_encoding)
# Ensure we match the path without the locale, if present.
if %r{^/#{::I18n.locale}/} === path
path = path.split(%r{^/#{::I18n.locale}}).last.presence || "/"
end
# First try to match against a "menu match" value, if available.
return true if item.try(:menu_match).present? && path =~ Regexp.new(item.menu_match)
# Find the first url that is a string.
url = [item.url]
url << ['', item.url[:path]].compact.flatten.join('/') if item.url.respond_to?(:keys)
url = url.last.match(%r{^/#{::I18n.locale.to_s}(/.*)}) ? $1 : url.detect{|u| u.is_a?(String)}
# Now use all possible vectors to try to find a valid match
[path, URI.decode(path)].include?(url) || path == "/#{item.original_id}"
end
def menu_item_css(menu_item, index)
css = []
css << selected_css if selected_item_or_descendant_item_selected?(menu_item)
css << "dropdown" unless menu_item_children(menu_item).empty?
css << first_css if index == 0
css << last_css if index == menu_item.shown_siblings.length
css.reject(&:blank?).presence
end
def menu_item_children(menu_item)
within_max_depth?(menu_item) ? menu_item.children : []
end
def within_max_depth?(menu_item)
!max_depth || menu_item.depth < max_depth
end
end
end
end
Using RubyMotion (for the first time!), I want to use Twitter's search API to retrieve some recent tweets for some users so have put together the class below.
The value of tweets is always an empty array. I suspect that BW::HTTP.get(url) spawns its own thread which is causing the issue.
Really, I just want twitter_search_results to return response.body.to_str but I am not sure how to do this.
How do I use RubyMotion (or BubbleWrap) to put an array of Tweet objects into my UIViewController?
class TweetsController
def initialize
#twitter_accounts = %w(dhh google)
#tweets = []
end
def tweets
twitter_search_results
puts #tweets.count
#tweets
end
def create_tweets(response)
BW::JSON.parse(response)["results"].each do |result|
#tweets << Tweet.new(result)
end
end
def twitter_search_results
query = #twitter_accounts.map{ |account| "from:#{account}" }.join(" OR ")
url = "http://search.twitter.com/search.json?q=#{query}"
BW::HTTP.get(url) do |response|
create_tweets(response.body.to_str)
end
end
end
class TwitterViewController < UIViewController
def viewDidLoad
super
self.view.backgroundColor = UIColor.blueColor
#table = UITableView.alloc.initWithFrame(self.view.bounds)
self.view.addSubview #table
#table.dataSource = self
#tweets_controller = TweetsController.new
end
def initWithNibName(name, bundle: bundle)
super
self.tabBarItem = UITabBarItem.alloc.initWithTitle(
"Twitter",
image: UIImage.imageNamed('twitter.png'),
tag: 1)
self
end
def tableView(tableView, numberOfRowsInSection: section)
#tweets_controller.tweets.length
end
def tableView(tableView, cellForRowAtIndexPath: indexPath)
#reuse_id = "Tweet"
cell = UITableViewCell.alloc.initWithStyle(UITableViewCellStyleDefault, reuseIdentifier:#reuse_id)
cell.textLabel.text = #tweets_controller.tweets[indexPath.row].text
return cell
end
end
class Tweet
attr_reader :created_at, :from_user, :text
def initialize(tweet_result)
#created_at = tweet_result["created_at"]
#from_user = tweet_result["from_user"]
#text = tweet_result["text"]
end
end
Full controller code below. I've also put the project on GitHub
class TweetsController
def initialize
#twitter_accounts = %w(dhh google)
#tweets = []
create_tweets
end
def tweets
#tweets
end
def create_tweets
json_data = twitter_search_results.dataUsingEncoding(NSUTF8StringEncoding)
e = Pointer.new(:object)
dict = NSJSONSerialization.JSONObjectWithData(json_data, options:0, error: e)
dict["results"].each do |result|
p result.class
p result
#tweets << Tweet.new(result)
end
end
def twitter_search_results
query = #twitter_accounts.map{ |account| "from:#{account}" }.join(" OR ")
url_string = "http://search.twitter.com/search.json?q=#{query}"
url_string_escaped = url_string.stringByAddingPercentEscapesUsingEncoding(NSUTF8StringEncoding)
url = NSURL.URLWithString(url_string_escaped)
request = NSURLRequest.requestWithURL(url)
response = nil
error = nil
data = NSURLConnection.sendSynchronousRequest(request, returningResponse: response, error: error)
raise "BOOM!" unless (data.length > 0 && error.nil?)
json = NSString.alloc.initWithData(data, encoding: NSUTF8StringEncoding)
end
end
the issue here is asynchronicity. you're almost there, I think, but the create_tweets method is not called before puts #tweets. In this case, I would recommend using a notification, because I think they are good ;-)
TweetsReady = 'TweetsReady' # constants are nice
NSNotificationCenter.defaultCenter.postNotificationName(TweetsReady, object:#tweets)
In your controller, register for this notification in `viewWillAppear` and unregister in `viewWillDisappear`
NSNotificationCenter.defaultCenter.addObserver(self, selector: 'tweets_ready:', name: TweetsReady, object:nil) # object:nil means 'register for all events, not just ones associated with 'object'
# ...
NSNotificationCenter.defaultCenter.removeObserver(self, name:TweetsReady, object:nil)
and you tweets_ready method should implement your UI changes.
def tweets_ready(notification)
#table.reloadData
end
I'm start using Django-MPTT app to get a tree-based approach on my Django-site pages.
For ex. I have pages with sub pages:
Trance:
Vocal Trance(sub page)
Hard Trance(sub page)
Breaks:
Atmo Breaks(sub page)
Progressive Breaks(sub page)
How can I get access to them from urls.py?
What pattern will help?
Do I need to store Full_path in model or it can be done via url pattern?
I assume you mean you want to do URLs like this:
/trance/
/trance/vocal-trance/
/trance/hard-trace/
/breaks/
/breaks/atmo-breaks/
/breaks/progressive-breaks/
If so, it's probably best to store the url fragment in your model. Something like:
from mptt.models import MPTTModel
from django.db import models
from django.template.defaultfilters import slugify
class Page(MPTTModel):
name = models.CharField(max_length=50)
slug = models.CharField(max_length=50,null=True)
url = models.CharField(max_length=255,null=True)
def save(self, *args, **kwargs)
if self.slug is None:
# create a slug that's unique to siblings
slug = slugify(self.name)
self.slug = slug
siblings = self.get_siblings()
i = 1
while siblings.filter(slug=self.slug).exists():
i += 1
self.slug = slug + '-%d' % i
# now create a URL based on parent's url + slug
if self.parent:
self.url = '%s/%s' % (self.parent.url, self.slug)
else:
self.url = self.slug
super(Page, self).save(*args, **kwargs)
Then add a URL pattern:
(r'^pages/(?P<page_url>[\w\d_/-]+)/$', 'pages.views.show_page'),
And in your view you can just fetch the right page:
def show_page(request, page_url=None):
page = get_object_or_404(Page, url=page_url)
...
Thank you for your attention to my problem.
See,How I finally do it.
models.py
class WebPage(MPTTModel):
slug=RuSlugField(max_length=20,unique=True)
title=models.CharField(max_length=50)
content=models.TextField()
parent=TreeForeignKey('self',null=True,blank=True,related_name='children')
class MPTTMeta:
order_insertion_by=['slug']
def get_absolute_url(self):#TODO:: replace with get_ancestors
url = "/%s/" % self.slug
page = self
while page.parent:
url = "/%s%s" % (page.parent.slug,url)
page = page.parent
return url
urls.py
urlpatterns = patterns('website.views',
url(r"^add/$", "add_page",name="add"),
url(r"^(?P<full_slug>.*)/add/$", "add_page",name="add"),
url(r"^(?P<full_slug>.*)/edit/$", "edit_page",name="edit"),
url(r'^$', ListView.as_view(model=WebPage,template_name='index.html',context_object_name="webpages_list",),name='index'),
url(r"^(?P<full_slug>.*)/$", "page", name="page"),
)
views.py
def page(request, full_slug):
# Make a list from full_slug.
# For ex. /trance/progressive_trance/fonarev -> ['trance','progressive_trance','fonarev']
slugs=full_slug.split('/')
page=None
# Get a page by it's slug
if len(slugs)>1:
page=get_object_or_404(WebPage,slug=slugs[-1])#slugs=['trance','vocal_trance'] -> 'vocal_trance'
elif len(slugs)==1:
page=get_object_or_404(WebPage,slug=slugs[0])#slugs=['trance'] -> 'trance'
# Check if page url matches requested full_slug
if page.get_absolute_url().strip('/') == full_slug:
return render_to_response('page.html', {'page': page},context_instance=RequestContext(request))
else:
raise Http404
def edit_page(request,full_slug):
slugs=full_slug.split('/')
page=None
if len(slugs)>1:
page=get_object_or_404(WebPage,slug=slugs[-1])
elif len(slugs)==1:
page=get_object_or_404(WebPage,slug=slugs[0])
if not page.get_absolute_url().strip('/') == full_slug:
raise Http404
# Send POST data for update an existing page.Update a page.
if request.method=='POST':
form=WebPageForm(request.POST, instance=page)
if form.is_valid():
form.save()
return HttpResponseRedirect(page.get_absolute_url())
# Render a form to edit data for existing page
else:
form=WebPageForm(instance=page)
return render_to_response('edit_page.html',{'form':form,},context_instance=RequestContext(request))
def add_page(request,full_slug=None):
parent_page=None
slug=None
if full_slug:
slug=full_slug.split('/')
# If there is a slug in REQUEST(ex.'trance')->we need to add a new_page to her parent_page.
# So get a parent page.
if slug:
if len(slug)>1:
parent_page=get_object_or_404(WebPage,slug=slug[-1])
elif len(slug)==1:
parent_page=get_object_or_404(WebPage,slug=slug[0])
# Create a new_page
if request.method=='POST':
form=WebPageForm(request.POST)
if form.is_valid():
new_page=form.save(commit=False)
if parent_page:
new_page.parent=parent_page
new_page.save()
return HttpResponseRedirect(new_page.get_absolute_url())
# Return an unbounded form
else:
form=WebPageForm()
return render_to_response('add_page.html',{'form':form,},context_instance=RequestContext(request))
The trick is in we have to check if the page really exists accessing to it via full_slug:
if not page.get_absolute_url().strip('/') == full_slug:
raise Http404
Otherwise, it could be wrong allowing to check only by slug.
There's also a django app that will do the work for you: django-mptt-urls
def get_absolute_url(self):
return '/'.join([x['slug'] for x in self.get_ancestors(include_self=True).values()])