Nutch - clone website - web-scraping

Question

I am playing with Apache Nutch and I am crawling a website successfully. I want to make a clone of a website with Nutch so that I can access the crawled webpages offline. Is there a way to do that? I'm looking for something like an endpoint that receives a url and returns the content of the webpage as if I were GETting the url with curl.
I know there are more specialized solutions like HTTrack, but I want to know if it is possible to use Nutch to do this.

Answer 1

I think you don't need this now but anyway I am putting my answer. Of course it's possible using Apache Nutch.After injecting seed urls and generating segments of urls to fetch, when you execute the fetch command -
$ bin/nutch fetch -all
The Hbase(it's very conventional to use Hbase as storage for Nutch) webpage table structure will be like this -
webpage : {
key : "com.exampe.dev:http/",
f : {
bas : {
timestamp : 1293732801833,
value : "http://dev.example.com/"
},
cnt : {
timestamp : 1293732801833,
value : "DOCTYPE html PUBLIC "-//W3C//DTD X...rest of page content"
},
fi : {
timestamp : 1293676557658,
value : "\x00'\x8D\x00"
},
prot : {
timestamp : 1293732801833,
value : "x02\x00\x00"
},
st : {
timestamp : 1293732801833,
value : "x00\x00\x00\x02"
},
ts : {
timestamp : 1293676557658,
value : "\x00\x00\x01-5!\x9D\xE5"
}
typ : {
timestamp : 1293732801833,
value : "application/xhtml+xml"
}
},
h : {
Cache-Control : {
timestamp : 1293732801833,
value : "private"
},
Content-Type : {
timestamp : 1293732801833,
value : "text/html; charset=UTF-8"
},
Date : {
timestamp : 1293732801833,
value : "Thu, 30 Dec 2010 18:13:21 GMT"
},
ETag : {
timestamp : 1293732801833,
value : 40bdf8b9-8c0a-477e-9ee4-b19995601dde"
},
Expires : {
timestamp : 1293732801833,
value : "Thu, 30 Dec 2010 18:13:21 GMT"
},
Last-Modified : {
timestamp : 1293732801833,
value : "Thu, 30 Dec 2010 15:01:20 GMT"
},
Server : {
timestamp : 1293732801833,
value : "GSE"
},
Set-Cookie : {
timestamp : 1293732801833,
value : "blogger_TID=130c0c57a66d0704;HttpOnly"
},
X-Content-Type-Options : {
timestamp : 1293732801833,
value : "nosniff"
},
X-XSS-Protection : {
timestamp : 1293732801833,
value : "1; mode=block"
}
},
mk : {
_injmrk_ : {
timestamp : 1293676557658,
value : "y"
},
_gnmrk_ : {
timestamp=1293732629430,
value : "1293732622-2092819984"
},
_ftcmrk_ : {
timestamp : 1293732801833,
value : "1293732622-2092819984"
}
},
mtdt : {
_csh_ : {
timestamp : 1293676557658,
value : "x80\x00\x00"
}
},
s : {
s : {
timestamp : 1293676557658,
value : "x80\x00\x00"
}
}
}
The column named cnt under f column family will contain the whole html content of your page. So you can use this to clone the page obviously.
Moreover, you can write custom plugins implementing Parser or ParseFilter interfaces to catch the whole content of page. The code will be very straight-forward.