iOpus iMacro webscraping loop - web-scraping

I am using the Opus iMacros add-on with Internet Explorer. The following Macro takes a variable from the first line of my excel CSV File and performs a search on a website. The search results from the website may be anywhere from 10 to 200 records, which are grouped 20 per page (1-10 pages). The macro then extracts each page to a text file for future reference.
SET !ERRORIGNORE YES
SET !EXTRACT_TEST_POPUP NO
TAB T=1
TAB CLOSEALLOTHERS
SET !DATASOURCE 7Digits.csv
SET !DATASOURCE_Columns 1
SET !DATASOURCE_LINE {{!LOOP}}
'Login
URL GOTO=https://SomeWebsite.com/login
TAG POS=1 TYPE=INPUT:TEXT FORM=NAME:form1 ATTR=NAME:USER_NAME CONTENT=ABC123
TAG POS=1 TYPE=INPUT:PASSWORD FORM=NAME:form1 ATTR=NAME:PASSWORD CONTENT=XXX
'set search criteria on multiple search page
TAG POS=1 TYPE=TEXTAREA FORM=NAME:form1 ATTR=NAME: NUMBER CONTENT={{!COL1}}
TAG POS=1 TYPE=INPUT:IMAGE FORM=NAME:form1 ATTR=ID:SEARCH
‘Extract results and SAVE
TAG POS=1 TYPE=A ATTR=TXT:1
TAG POS=2 TYPE=TABLE ATTR=TXT:*location* EXTRACT=TXT
SAVEAS TYPE=txt FOLDER=* FILE=mytable_{{!NOW:yymmdd_hhnnss}}
TAG POS=1 TYPE=A ATTR=TXT:2
TAG POS=2 TYPE=TABLE ATTR=TXT:*location* EXTRACT=TXT
SAVEAS TYPE=txt FOLDER=* FILE=mytable_{{!NOW:yymmdd_hhnnss}}
TAG POS=1 TYPE=A ATTR=TXT:3
TAG POS=2 TYPE=TABLE ATTR=TXT:*location* EXTRACT=TXT
SAVEAS TYPE=txt FOLDER=* FILE=mytable_{{!NOW:yymmdd_hhnnss}}
TAG POS=1 TYPE=A ATTR=TXT:4
TAG POS=2 TYPE=TABLE ATTR=TXT:*location* EXTRACT=TXT
SAVEAS TYPE=txt FOLDER=* FILE=mytable_{{!NOW:yymmdd_hhnnss}}
TAG POS=1 TYPE=A ATTR=TXT:5
TAG POS=2 TYPE=TABLE ATTR=TXT:*location* EXTRACT=TXT
SAVEAS TYPE=txt FOLDER=* FILE=mytable_{{!NOW:yymmdd_hhnnss}}
TAG POS=1 TYPE=A ATTR=TXT:6
TAG POS=2 TYPE=TABLE ATTR=TXT:*location* EXTRACT=TXT
SAVEAS TYPE=txt FOLDER=* FILE=mytable_{{!NOW:yymmdd_hhnnss}}
TAG POS=1 TYPE=A ATTR=TXT:7
TAG POS=2 TYPE=TABLE ATTR=TXT:*location* EXTRACT=TXT
SAVEAS TYPE=txt FOLDER=* FILE=mytable_{{!NOW:yymmdd_hhnnss}}
TAG POS=1 TYPE=A ATTR=TXT:8
TAG POS=2 TYPE=TABLE ATTR=TXT:*location* EXTRACT=TXT
SAVEAS TYPE=txt FOLDER=* FILE=mytable_{{!NOW:yymmdd_hhnnss}}
TAG POS=1 TYPE=A ATTR=TXT:9
TAG POS=2 TYPE=TABLE ATTR=TXT:*location* EXTRACT=TXT
SAVEAS TYPE=txt FOLDER=* FILE=mytable_{{!NOW:yymmdd_hhnnss}}
TAG POS=1 TYPE=A ATTR=TXT:10
TAG POS=2 TYPE=TABLE ATTR=TXT:*location* EXTRACT=TXT
SAVEAS TYPE=txt FOLDER=* FILE=mytable_{{!NOW:yymmdd_hhnnss}}
'END
My challenge is to figure out a way to cancel the “SAVEAS” if there are less than 10 pages of search results. For instance, if the search only returns 60 results, the Macro only needs to perform the SAVEAS command 3 times (20 x 3). How can I move to the next {{!LOOP}} without saving the same text file 7 more times?
I have tried the “!FAIL_ON_ALL_NAVIGATEERRORS” command, but the macro stops entirely. I would like to move to the next record instead of stopping the macro.

This is how you convert iMacros code to JavaScript.
How to invoke an iMacro from JavaScript?
Also your question holds an answer because you asked " if there are less then ". So you have to convert to code to JS and use if clauses.

You can do it by imacro javascript if formula.
if(search result<10) {
perform your script
}else { go to next page }

Related

Alfresco share - Tags not displayed in Tag Management

When creating a folder rule with rule criteria 'Has tag', the Select Tag panel doesn't display all tags
I have tried creating a tag by adding a tag in the file properties but it's still not showing
I checked the tag management, it's not showing there either!
I'm using activiti for managing workflows, and I tried adding tags like it shows in the code below:
<userTask id="reviewTask" name="Review Task"
activiti:formKey="wfa:activitiReviewTaskk">
<extensionElements>
<activiti:taskListener event="create" class="org.alfresco.repo.workflow.activiti.tasklistener.ScriptTaskListener">
<activiti:field name="script">
<activiti:string>
if (typeof bpm_workflowDueDate != 'undefined') task.dueDate = bpm_workflowDueDate
if (typeof bpm_workflowPriority != 'undefined') task.priority = bpm_workflowPriority;
</activiti:string>
</activiti:field>
</activiti:taskListener>
<activiti:taskListener event="complete" class="org.alfresco.repo.workflow.activiti.tasklistener.ScriptTaskListener">
<activiti:field name="script">
<activiti:string>
execution.setVariable('wfa_reviewOutcome', task.getVariable('wfa_reviewOutcome'));
if (task.getVariable('wfa_reviewOutcome') == 'Approve')
{
bpm_package.children[0].addTag("Approved");
}
else
{
if (task.getVariable('wfa_reviewOutcome') == 'Reject')
{
bpm_package.children[0].addTag("Rejected");
}
}
</activiti:string>
</activiti:field>
<activiti:field name="runAs">
<activiti:string>admin</activiti:string>
</activiti:field>
</activiti:taskListener>
</extensionElements>
<humanPerformer>
<resourceAssignmentExpression>
<formalExpression>${bpm_assignee.properties.userName}</formalExpression>
</resourceAssignmentExpression>
</humanPerformer>
</userTask>
The tags are added but still not showing in the tags management!
I fixed the problem by performing a reindex of SOLR by following the steps in this link, http://docs.alfresco.com/5.0/tasks/solr-reindex.html :
Delete the content of the directories
..\alf_data\solr4\index\workspace\SpacesStore\ and
..\alf_data\solr4\index\archive\SpacesStore\
Delete all the files in ..\alf_data\solr4\model

disabling rendering of a particular column on rendering entire datatable

In the code below there is datatable with id discoObject. This id is render on selecting a drop down which changes color of the column according to the selection hence AJAX call.
There is another field call freeText of type inputTextArea. As soon as i select a drop down value what ever text is written in freeText area it vanishes. This is because on ajax call i am rendering whole datatable. But if i don't render whole datatable and only render column id for drop down, it will not change color of drop down. I want the color of drop down to be changed retaining whatever value i have entered in freetext.
Please help me in this.
Thanks is advance.
<p:dataTable styleClass="ui-datatable-scrollable-body list" var="discoViewObject" rowIndexVar="index" id="discoObject" rowKey="#{discoViewObject.uniqueId}" style="height:500px"
rowStyleClass="#{(index % 2) eq 0? 'tablerow' : 'tablerowAlter'}" selection="#{staffMsgingHccDiscoHandler.hccDiscoBean.discoViewObjectSelectionList}"
value="#{staffMsgingHccDiscoHandler.hccDiscoBean.discoViewObjectList}" >
<p:column id="decisionColor" headerText="#{message['CNS.fltDisruption.oneFlightMessae.hccDecision']}"
styleClass="#{(discoViewObject.disruptedPaxGroupType.decision) eq 'MISSED CONNECTION'? 'decisionRed' : '' }
#{(discoViewObject.disruptedPaxGroupType.decision) eq 'WILL WAIT'? 'decisionGreen' : '' }
#{(discoViewObject.disruptedPaxGroupType.decision) eq 'PENDING'? 'decisionYellow' : '' }">
<h:selectOneMenu value="#{discoViewObject.disruptedPaxGroupType.decision}">
<f:selectItem itemLabel="CONNECTION OK" itemValue="CONNECTION OK"></f:selectItem>
<f:selectItem itemLabel="WILL WAIT" itemValue="WILL WAIT"></f:selectItem>
<f:selectItem itemLabel="PENDING" itemValue="PENDING"></f:selectItem>
<f:selectItem itemLabel="MISSED CONNECTION" itemValue="MISSED CONNECTION"></f:selectItem>
<f:selectItem itemLabel="TO BE ANNOUNCED" itemValue="TO BE ANNOUNCED"></f:selectItem>
<f:ajax event="change" render="discoObject"></f:ajax>
</h:selectOneMenu>
</p:column>
<p:column headerText="Free Text" >
<h:inputTextarea cols="6" rows="4" style="width: 192px; height: 63px;" value="#{discoViewObject.freeTextOutBoundFlt}" ></h:inputTextarea>
</p:column>

Scraping dynamic data with imacro to excell

I want to scrape dynamic data (refreshable every 4 seconds and it's a number ) with imacro and represent that number changing along the time in excell ( or any other way ).
How can i do this ? Imacro, as further as i know can get the data but can it scrap a dynamic one .
The code is from a basketball game on Flashscore ,exactly the scores table between 2 teams :
Here is an example for a game table
<table id="parts" class="parts-first horizontal">
<tbody>
<tr class="odd">
<td class="score"><span class="rb">69</span</td>
<td class="score part"><span class="p1_home">31</span></td>
</tr>
<tr class="even">
<td class="score"><span class="rb">63</span></td>
<td class="score part"><span class="p1_away">17</span></td>
</tr>
</tbody>
</table>
that gives those two lines data:
69
31
63
17
Here is the more detailed answer with a code.
TAG POS=1 TYPE=TD ATTR=CLASS:"score" EXTRACT=TXT
SET !EXTRACT EVAL("'{{!EXTRACT}}'.match(/\\d+/)[0];")
TAG POS=1 TYPE=TD ATTR=CLASS:"score part" EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=* FILE=scores.csv
TAG POS=2 TYPE=TD ATTR=CLASS:"score" EXTRACT=TXT
SET !EXTRACT EVAL("'{{!EXTRACT}}'.match(/\\d+/)[0];")
TAG POS=2 TYPE=TD ATTR=CLASS:"score part" EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=* FILE=scores.csv
WAIT SECONDS=4
Play this macro in loop mode (button ‘Play (Loop)’) with the max. number of loops equal to a very big integer (e.g. 99999).

how to add a user defined header in a rml report in openerp?

How can I add a new header/footer for a report(for example picking list report in delivery order) other than the header/ footer defined in the company?
In report tag put header='False',
eg.
<report header='False' auto="False" id="report_product_history"
model="product.product" name="stock.product.history"
string="Stock Level Forecast"/>
it will not print the default header define in the company.
then in rml file find <pageTemplate> tag, and replace it with your rml code.
eg.
<template pageSize="(595.0,842.0)" title="Test"
author="Atul Makwana" allowSplitting="20">
<pageTemplate id="first">
***Your rml header & footer***
</pageTemplate>
</template>
This way you can put new header and footer.
One way to remove the header is what Atul suggested, declare it in the report tag.
<report
header="False"
auto="False"
id="report_product_history"
model="product.product"
name="stock.product.history"
string="Stock Level Forecast"/>
In some situations there is no report tag. For example, a report might only be generated by a wizard. In that case, you can declare it as a parameter when you register the parser. See the mrp_operations module's barcode report for an example.
class code_barcode(report_sxw.rml_parse):
def __init__(self, cr, uid, name, context):
super(code_barcode, self).__init__(cr, uid, name, context=context)
self.localcontext.update({
'time': time,
})
report_sxw.report_sxw('report.mrp.code.barcode',
'mrp_operations.operation.code',
'addons/mrp_operations/report/mrp_code_barcode.rml',
parser=code_barcode,
header=False)
You can also specify a specific header using that parameter. It defaults to 'external', but it can be 'internal' or 'internal landscape' to use one of the other headers from the company configuration.
In report set header = 'False'
Now you can add your own header footer on page
<template title="Test" author="Sagar" allowSplitting="20">
<pageTemplate id="first">
<frame id="first" x1="15.0" y1="42.0" width="539" height="758"/>
<pageGraphics>
<!-- ================== Header =============== -->
<image x="14cm" y="25.6cm" height="40.0">[[ company.logo or removeParentNode('image') ]]</image>
<setFont name="Helvetica" size="10.0"/>
<drawString x="1cm" y="27.2cm">Main Header</drawString>
<!-- Order Details -->
<place x="33" y="18cm" width="530.0" height="205.0">
<blockTable colWidths="265,265" style="Table1">
<tr>
<td>Header Value 1</td>
<td><para style="normal2-center">Header Value 2</para></td>
</tr>
</blockTable>
</place>
<!-- ======================== footer =========================== -->
<place x="33" y="55cm" width="530.0" height="205.0">
<blockTable colWidths="265" style="Table1">
<tr><td><para style="normal2-center">Footer Value</para></td></tr>
</blockTable>
</place>
</pageGraphics>
</pageTemplate>
</template>
You can customize report header in your report.rml file like this,
<pageTemplate id="first">
<frame id="first" x1="57.0" y1="115.0" width="481" height="615"/>
<header>
<pageGraphics>
<image x="1.3cm" y="26.0cm" height="90.0">[[company.logo or removeParentNode('image')]]</image>
<drawString x="10.9cm" y="2.9cm">Signature:</drawString>
<drawString x="12.7cm" y="2.9cm">___________________________________</drawString>
</pageGraphics>
</header>
</pageTemplate>

Using an iframe to display google mini search results

I only want to display the iframe containing the search results when the search form has been submitted, however I'm not sure how to do this as the $_GET variables are not available to my script so I can't do
if(isset($_GET['submit'])) {
//display iframe
}
How can I capture the search results returned from the google mini (to a file) to display as in include within the webpage or how can capture the submit event for this page, which I can use as the condition to display the iframe containing the search results?
Many thanks
Return false in a form's onSubmit event, then set the src property of your iframe to what would be the URL you use for the search.
Something like:
<form onSubmit='return doSearch();'>
<input type='text' name='search' id='search' />
<!-- or whatever parameters needed for the search form -->
<input type='submit' />
</form>
<iframe id='google'></iframe>
<script>
function doSerarch() {
var aSearch = 'search=' + $('#search').val(); // or whatever
$('#google').attr('src','http://yourmini?');
return false;
}
</script>

Resources