I want to fetch output as string. I want to run webpage from my code like www.example.com after running i want to keep whole values in a string and again in next step i want to search all anchor href values from the string. But initially how will get domain ouput as string ASP.net.
It sounds like you're trying to scrape another website. This may help: http://www.beansoftware.com/ASP.NET-Tutorials/Screen-Scraping-Web-Fetching.aspx
Related
Is there a way to extract a table from a web page in Automation Anywhere after taking certain steps using web recorder. The table does not appear directly, it appears after clicking few controls after launching the URL.
The table that I want to extract is coming after loggin in to the website and filtering using a control for search criteria.
I used web recorder to login and putting the desired search criteria in a text field and I want to extract the table now. When I use web recorder, it launches the URL again and takes me back to the login page which I dont want. I want the bot to stay on the page. Pls help.
Also, what is the significance of session name of an extracted table?
If you clicked on Advanced View, you will find at Step 5 : to run this command using an existing IE window. Try to write the URL of the page with the table and not the one of the login page.
The extracted table is to be used using variable $Table Column(Index)$ with index being the column number or column name
you can export directly using object cloning and in the selection criteria export to csv file. But we need to click on html inner text also in search criteria
An old question, but my experience has been the Extract Data/Table commands are rather poor. Not only do they only work in IE, you cannot call them as commands, they have to be called via a web recording.
Instead, I've found it much more useful to object clone the initial element, grab the DOMXPath, and variablize that. Then throw it into a loop while command and set the condition on finding at least one element (of the elements for the table you are trying to build). You can grab all sorts of useful info in the object clone command and then right that to a variable/table.
For example
//div[#id='updatable-standings']/div[1]/div[1]/div[2]/div[1]/table[1]/tbody[1]/tr[3]/td[2]/div[1]/span[2]
//div[#id='updatable-standings']/div[1]/div[1]/div[2]/div[1]/table[1]/tbody[1]/tr[4]/td[2]/div[1]/span[2]
I can create a incremental variable for {tr[3]} and call it $vTeamLoop$ and change my DOMXPath value in the Object Clone to be
//div[#id='updatable-standings']/div[1]/div[1]/div[2]/div[1]/table[1]/tbody[1]/tr[$vTeamLoop$]/td[2]/div[1]/span[2]
Ultimately, it is more steps than the Data/Table Extract command, but it is far less limited in scope.
Hope that helps.
enter code here
I am trying to scrape a web forum using Scrapy for the href link info and when I do so, I get the href link with many letters and numbers where the question mark should be.
This is a sample of the html document that I am scraping:
I am scraping the html data for the href link using the following code:
response.xpath('.//*[contains(#id, "thread_title")]/#href').extract()
When I run this, I get the following results:
[u'showthread.php?s=f969fe6ed424b22d8fddf605a9effe90&t=2676278']
What should be returned is:
[u'showthread.php?t=2676278']
I have ran other tests scraping for href data with question marks elsewhere in the document and I also get the "s=f969fe6ed424b22d8fddf605a9effe90&" returned.
Why am I getting this data returned with the "s=f969fe6ed424b22d8fddf605a9effe90&" instead of just the question mark?
Thanks!
It seems that the site I am scraping from uses a unique identifier in order to more accurately update the number of views per the thread. I was not able to return scraped data without a unique id, it changed over time, and scraped a different HTML tag for the thread ID and then joined it to the web address (showthread.php?t=) to create the link I was looking for.
I have found a similar question earlier here:
Google Analytics Visitors Flow: grouping URLs?
However I'm confused because people suggest different way to write the Replace String, and either way I try it am not able to make it work.
So I have a ecommerce site with hundreds of different pages. The different parts of the website is:
http://example.com/sv/ (Root)
http://example.com/sv/category/1-name/
http://example.com/sv/product/1-name/
http://example.com/sv/designer-tool/1-name/
http://example.com/sv/checkout/
When I go to the visitors flow. I want to see the amount of people that go from example Root to Category, and from Category to Product, and from Product to Designer Tool, and from Designer Tool to Checkout. However now when I have so many different pages it becomes very difficult to follow the visitors flow, because the product pages are for example not grouped together.
So instead of above. I would like to remove the 1-name/ part in the end. And only see /sv/category/, /sv/product/, /sv/designer-tool/.
In the earlier post I understand you can use an advanced filter to do this. I have set the following settings:
Type: Search & Replace
Field: Request URI
Search String: ^/(category|product|designer-tool)(/\d*)(.*)
Replace String: /$A1$A3
I guess that my search string and my replace string is wrong. Any ideas?
EDIT: I updated my filter to the following:
Search String: ^/sv/(category|product|designer-tool)(/\d*)(.*)$
Replace String: /sv/\1/
Still testing and unsure if it's the correct way to set it up.
I was able to solve this by the Search String and the Replace String in my edit above.
So basically what I did was:
Create a secondary view/profile for your site. If you apply your filter to your one and only view/profile that means that you won't be able to see any detailed data about specific pages, because the filter removes/filter that.
Add an Advanced Filter with the following settings:
Type: Search & Replace
Field: Request URI
Search String: ^/sv/(category|product|designer-tool)(/\d*)(.*)$
Replace String: /sv/\1/
You need to wait 24h after creating your new profile/view before you can see any data in it.
So my confusion was regarding the Search and Replace String. The Search String is an regular expression for matching everything after your .tld. So for example http://www.example.com/sv/mypage/1-post/, the Search String will only search within /sv/mypage/1-post/.
The Replace String is what it should replace the whole Search String with. So in my case, I matched all URL's that had /sv/category/1-string/. I wanted only to keep the "category" part, so I replaced the whole string with /sv/category/ by inputting Replace String /sv/\1/
/sv/ means just what it says. \1 means that it should take the value of the first () of my Search String (In this case "category"). The ending / is just an ending slash.
All in all, it means that any URLs that looked like http://example.com/sv/category/1-string/ was changed to http://example.com/sv/category/. Meaning that I can now see data for all my categories as a group, instead of individual pages.
Basically I have a news page which stores headlines, stories, and a unique story identifier in a SQL database. I want to be able to create a hyperlink on a webpage to the pictures.
so when someone selects a news story from a drop down menu (which uses the headline) and presses submit I want to pass the storyID, which is a unique identifier, to a spot in a hyperlink. so if it was story 134 then then link would look like:
I know the SQL statement would look like:
SELECT StoryID from db.News
Where Headline = {The headline selected in the dropdown menu}
the dropdown menu is called NewsDrop
this would be an ASPX page written with a VB code base
SO I guess I need help passing the variables along to the search string and the hyperlink.
Not even sure if this is even possible.
There are a number of options available to achieve this, the most common would be to using a query string in the hyperlinks in your drop down menu to send a parameter to a SQL stored procedure which would use it in a variable in your select statement. So basically the hyperlinks you have in the drop down menu would be appended with ?storyID=<uniquestoryid> and on the far end SELECT StoryID from db.News Where StoryID = #StoryID it would be less efficient to use the headline from the link as a query string and variable in the where clause as you have shown but if that is your only option it could be done.
However you should proceed carefully when using query strings here is a link to a good basic article about query strings and another link about best practices.
I am having trouble to what i should name this in the title field...
Question is. I want to select a blogpost by entering its title, like this:
http://localhost:3254/blog/2010/10/net-programming
The title of the blogpost should be ".net programming". So i have setup a route to look for a year a month and finally a string. But how should i save this in the database to be able to search for it?
Should i add a field, like id, that contains asp-programming? or should i parse the title for every search? or should i convert the title and then convert it back before displaying it to the user?
The standard method is to create a separate column in the database (usually called 'slug' or permalink) that stores the html-friendly name that may be used by Urls. Then, if the title of the post changes you would normally keep the slug the same so that anyone who has bookmarked the link can still find the post. The value us usually defaulted from the blog title, replacing spaces with - and removing any non url friendly characters.
http://en.wikipedia.org/wiki/Slug_%28web_publishing%29
There are two possible solutions I can think of
Create your own Route handler and within its code, search for the actual record in the database
Create an action method in a Controller that accepts a string parameter and call that parameter id (or whatever the default is in your MapRoutes method).