Crawling/harvesting links

[zuperxtreme]

How would I go in crawling through a website(s) and harvesting URL's ?

How would I then organize them ? Say by what domain/subdomain it was taken from. Maybe even reading the title of the page?

Just curious.

[eSparq]

HTTrack/WinHTTrack (http://www.httrack.com/)?

It crawls websites and downloads them, organizing them (optionally) in one of several possible structures, and creates an index page of sites you've crawled. It can also be configured to ignore robots.txt files, BTW.

If that's not quite what you're wanting, you can modify the source code since it's GPL (assuming you know how to program in C).

[zuperxtreme]

Hah, I just finished installing that to download a web page that I need for offline browsing.

No, but what I mean is crawling a website, like, say:

http://taringa.net/posts/juegos/2993...ENG_2009).html

And gathering all the rapidshare, megaupload, etc links. And saving them in a text file(or whatever) with the title "Hunting Unlimited 2010"(gathered from the site's title)

Then crawling another website, say:
http://taringa.net/posts/juegos/1554...D%5BRS%5D.html
or
http://taringa.net/posts/juegos/2993...ited-2010.html

Which are both about the same thing, but might have different mirrors.

I would then like to merge them into the same text file(because of the same title) and sort the links.

Makes sense? One text file about the same thing, with links from the 3 sites, but all different mirrors.

EDIT: No knowledge of C, unfortunately...

[Dr Gonzo]

If you can't find one, PM me. I'm pretty sure I could modify httrack for that purpose. Interested in building something like this myself.

Should be possible, the links are in the source code of the page, as long as you specify the actual URL, then I imagine this could be accomplished with some RegEx functions.

I was going to build an app for this, but got sidetracked with other projects...

[zuperxtreme]

Hmm, so far no luck in finding anything already made.

I came across Xenu's Link Sleuth which does one hell of a job at finding and checking if the links are broken. It can export as text, which can then be used to find all the links I want. I guess.

[Dr Gonzo]

http://www.worldminer.com/webcustomer.htm

You tried this?

[zuperxtreme]

Looks promising. I'll check it out it a bit, thanks.