if you can share a git that can do web scripting in rails framework.
I love to look into it and study them, so I can make my own.
I was following this https://medium.com/swlh/web-scraper-application-with-ruby-on-rails-864dfaae6270. It was not calling the process function at all.
This is my code at modal/vehicles_spider
class VehiclesSpider < Kimurai::Base
#name = 'vehicles_spider'
#engine = :mechanize
def self.process(url)
p "i am here to process"
#start_url = [url]
self.crawl!
end
def parse(response, url:, data: {})
p "i am here to parse them"
response.xpath("//div[#class='vehicle-details']").each do |vehicle|
item = {}
item[:title] = vehicle.css('h2.title').text&.squish&
Vehicle.where(item).first_or_create
end
end
end
my db struct
def change
create_table :vehicles do |t|
t.string :title
t.timestamps
end
end
my vehicle controller in controller folder
def index
#vehicles = Vehicle.all
end
def scrape
p "i am here"
url = 'https://www.cars.com/shopping/sedan/'
response = VehiclesSpider.process(url)
p "what did you have here", response
if response[:status] == :completed && response[:error].nil?
flash.now[:notice] = "Successfully scraped url"
else
flash.now[:alert] = response[:error]
end
rescue StandardError => e
flash.now[:alert] = "Error: #{e}"
end
my index view
<p style="color: green"><%= notice %></p>
<h1>Vehicles</h1>
<h1> grab that stuff </h1>
<%= button_to 'Scrape', scrape_vehicles_path %>
<div id="vehicles">
<% #vehicles.each do |vehicle| %>
<%= render vehicle %>
<p>
<%= link_to "Show this vehicle", vehicle %>
</p>
<% end %>
</div>
<%= link_to "New vehicle", new_vehicle_path %>
my scrabe.html.erb
<p id="notice"> <%= notice %> </p>
<p id="alert"> <%= alert %> </p>
<%= link_to "Go Back", root_path %>
if i run it asks me to that it does not work at all...
I tried to look for other sample but they dont really work with rails...
so I tried to use python examples... their tutorials works really fine...
if anyone has working web scraping exmaple in rails 7 frails...
please share your git or code...
I will love you so much!
Related
I´m trying to avoid to update an empty name for #post.
I´m beginner in RoR and I don´t understand why in terminal I got #post.invalid? => true but in my view edit.html.erb #post.invalid? => false
posts_controller.rb
class PostsController < ApplicationController
before_action :set_post, :only => [:edit, :show, :update, :destroy]
def index
#posts = Post.all
respond_to do |format|
format.html
format.json { render json: #posts }
end
end
def show
respond_to do |format|
format.html
format.json { render json: #post }
end
end
def edit
end
def update
if #post.update(post_params)
redirect_to posts_path, success: "Post updated"
else
puts #post.invalid? # write true
render 'edit'
end
end
def new
#post = Post.new
end
def create
post = Post.create(post_params)
redirect_to post_path(post.id), success: "Post created"
end
def destroy
#post.destroy
redirect_to posts_path, success: "Post deleted"
end
private
def post_params
params.require(:post).permit(:name, :content)
end
def set_post
#post = Post.find(params[:id])
end
end
post.rb
class Post < ApplicationRecord
validates :name, presence: true
def as_json(options = nil)
super(only: [:name, :id, :created_at] )
end
end
edit.html.erb
<h1>Editer l´article</h1>
<%= #post.invalid? %> <!-- write false -->
<% if #post.invalid? %> <!-- return false -->
<div class="alert alert-danger">
<% #post.errors.full_messages.each do |message| %>
<%= message %>
<% end %>
</div>
<% end %>
<%= form_for #post do |f| %>
<div class="form-group">
<label>Titre de l´article</label>
<%= f.text_field :name, class: 'form-control' %>
</div>
<div class="form-group">
<label>Contenu de l´article</label>
<%= f.text_area :content, class: 'form-control' %>
</div>
<div class="form-group">
<%= f.submit "Modifier l´article", class: 'btn btn-primary' %>
</div>
<% end %>
I´m confused, someone has got an idea ?
The methods valid? and invalid? all keep running the validation methods every time they are called, and therefore potentially change the state of the model.
If you want to just check for validity when validation has already been run, you should instead use #post.errors.present? or #post.errors.blank? which will never change the status, only read existing errors (that were added in your case when the call to update failed.
Additionally (even if it´s not the case here) calling valid? and invalid? without a context will clear out errors that had been added with validations like validate ..., on: :update.
The solution I found was a combination of #rewritten´s answer and this one.
So...
<h1>Editer l´article</h1>
<% if #post.errors.present? %>
<div class="alert alert-danger">
<% #post.errors.full_messages.each do |message| %>
<%= message %>
<% end %>
</div>
<% end %>
<%= form_for #post, data: { turbo: false} do |f| %>
<div class="form-group">
<label>Titre de l´article</label>
<%= f.text_field :name, class: 'form-control' %>
</div>
<div class="form-group">
<label>Contenu de l´article</label>
<%= f.text_area :content, class: 'form-control' %>
</div>
<div class="form-group">
<%= f.submit "Modifier l´article", class: 'btn btn-primary' %>
</div>
<% end %>
Specs :
Ubuntu 16.04 LTS i686
, Ruby 2.1.5 , Rails 4.2.7 , sqlite 3 , devise 4.3
when I loop through each user in users , it's rendering the users + a query output from rails console.
Devise model name : User .
controller name: users.
I tried this code:
users_controller.rb
def index
#users = User.all()
end
routes.rb
resources :users, only: [:index, :show]
views/users/index.html.erb
<%= #users.each do |user| %>
<%= link_to user do %>
<li>
<img src="images/avatar.jpg" alt="">
<div class="user_overview">
<h2><%= user.title %></h2>
<p class="posts_count">1 post</p>
</div>
</li>
<% end %>
<% end %>
In your view, just remove = from <%= #users.each do |user| %>.
It will become like:
<% #users.each do |user| %>
<%= link_to user do %>
<li>
...
...
...
...
</li>
<% end %>
<% end %>
= in erb is used for displaying the content that's why your loop is displaying the enumerator.
Read more details on Expression Printing Tags
Not the best title, but I'm honestly not sure on how to properly explain what I'm looking for help for.
So I'm using Middleman blog to well create my blog. Anyways, I'm using frontmatter to pass css that change the look of each page individually. I'm using 4 variables, link_color, text_color, bg_link. So what I want to do is reuse that same frontmatter information in the layout.html.erb file.
So the layout.html.erb is the standard
<% if paginate && num_pages > 1 %>
<p>Page <%= page_number %> of <%= num_pages %></p>
<% if prev_page %>
<p><%= link_to 'Previous page', prev_page %></p>
<% end %>
<% end %>
<% page_articles.each_with_index do |article, i| %>
<li class="article_summary">
<h1><%= link_to article.title, article, id: "#{i}" %></h1>
</li>
<% end %>
<% if paginate %>
<% if next_page %>
<p><%= link_to 'Next page', next_page %></p>
<% end %>
<% end %>
What I'm trying to do is for each article within that iterator is if the article has bg_color frontmatter then use that and change the color of the article.title if not, then do nothing. Currently if I try with something like:
<style>
<% if article.data.bg_color? %>
.article_summary a#<%= i %>{
color: rgb(<%=article.data.bg_color %>);
}
<% end %>
</style>
I'm doing it this way because my blog lives on Github.
Currently it works, but since it's just a simple iteration it gives every article that same color and not on a per article basis. So I'm trying to figure out the best way to utilize the index as some sort of id so that they're targeted individually.
Perhaps changing the li from a class to an id consisting of the index, but then I won't be able to apply a global style from the scss in the stylesheet folder no?
I've found a dirty method that works.
<% page_articles.each_with_index do |article, i| %>
<li class="article_summary" id="test_<%=i %>">
<h1><%= link_to article.title, article %></h1>
<style>
<% if article.data.bg_color? %>
#test_<%=i%> a{
color: <%=article.data.bg_color %>;
}
<% end %>
</style>
</li>
<% end %>
Pretty much added "test_" to the id (before I was just doing the index itself) and viola!
I still new with RoR and i am trying to add a simple search form to my app. This is how i proceed.
class BooksController < ApplicationController
before_action :set_book, only: [:show, :edit, :update, :destroy]
def index
if params[:search]
#users = User.search(params[:search]).order("created_at DESC")
else
#users = User.order('created_at DESC')
end
end
def new
#book= Book.new
end
def create
#book= Book.new(books_params)
if #book.save
redirect_to admin_manage_path
else
render 'new'
end
end
def list
#books= Book.all
end
def show
end
private
def set_book
#book = Book.find(params[:id])
end
def books_params
params.require(:book).permit(:title, :author, :price, :resume)
end
end
my book model
class Book < ActiveRecord::Base
def self.search(query)
where('title like ?', "%#{query}%")
end
end
my list.html.erb where the search form is shown:
<% if current_user.present? %>
Welcome <%= current_user.pseudo %> | <%= link_to 'Déconnexion' , user_logout_path %>
<% end %>
<br>
<%= form_tag(books_path, :method => "get", id: "search-form") do %>
<%= text_field_tag :search, params[:search], placeholder: "Search books" %>
<%= submit_tag "Search", :name => nil %>
<% end %>
<br>
and my index.html.erb where the search result should be shown:
<% #books.each do |b| %>
<%= b.title %>
the error diplayed is :
Couldn't find Book without an ID
and i do not understand what it refer to controller#show method
ActiveRecord::RecordNotFound in BooksController#show
I found the error's source, i forgot to specify the appropriate route
Why does this not work? I've tried multiple choices like render and puts, but it doesn't want to change properly. A couple of ideas users?
THE VIEW:
<div id = "container">
<div id = "smaller-container">
<%= user_signed_in? %>
<%= #Uchecker %>
<% else %>
<%= link_to "Sign Up", new_user_registration_path, :id => "SignUpText" %>
</div>
THE CONTROLLER:
class PostsController < ApplicationController
def Uchecker
#Uchecker = redirect_to user_session_path
end
end