Comments on: PHP Crawler

By: Air Force 1

Air Force 1 — Sun, 06 Sep 2009 11:06:29 +0000

This is a wonderful script. But I used snoopy

By: Tuvok

Tuvok — Wed, 10 Jun 2009 16:01:28 +0000

Thanks. It’s nice to see examples of very useful things.

Which option is best for a 100,000+ url crawl project, with hardware not being an obstacle ?

By: David Arakelian

David Arakelian — Sun, 17 Feb 2008 18:05:51 +0000

Thanks for sharing your code on the use of the cURL multi functions. I run a lot of scripts that take a long time to execute (4 – 5 days in some cases) because I am using a single stream. Using your code and some proxies I can probably get this down to a few hours

By: Rays

Rays — Sun, 11 Nov 2007 07:40:16 +0000

I’m confuse..
Somebody help me please?
Maybe, i must create auto crawler engine.
Hemm.. i need team work for this time!

By: Mike

Mike — Mon, 15 Oct 2007 16:25:48 +0000

Rishiraj – use the ‘mycrawler_single’ function described in the post above. It should only need minimal tweaking to work in php 4.

By: Rishiraj

Rishiraj — Mon, 15 Oct 2007 16:19:15 +0000

I have freeernti hosting server with php 4.7.
Is there any alternative for me if i don’t want to use curl?

By: Mike

Mike — Wed, 13 Sep 2006 09:40:11 +0000

If you don’t know what to do with the code, then it isn’t for you

By: HM2K

HM2K — Thu, 01 Jun 2006 14:26:20 +0000

You never once actually said what this function does, “PHP Crawler” is VERY generic.

Also you have not shown any examples of output.

By: Jeff Beck

Jeff Beck — Wed, 18 Jan 2006 19:24:06 +0000

I’ve been trying to do much of the same thing on a site I have been working on. The problem I have been running up against has been fSockOpen; it waits for a connection to the server before returning. As such, when you loop through an array of URLs, it can take maybe twice as long to connect and retrieve data as with cURL multi. The fastest solution I have seen involved fSockOpen connecting to the local machine (aka PHP multithread HACK), and then letting each PHPlet file do the crawling independantly. This cuts even the cURL multi in half, but is not very pratical on the server load (1 page = 30+ httpd files running). Speed wise, have you compared your PHP function with the cURL function?