Libraries and scripts for crawling the TYPO3 page tree. Used for re-caching, re-indexing, publishing applications etc.
A simple web crawler in php to run through the links of a given URL recursively
A simple crawler (spider) writen in php just for fun, with zero dependencies
PHP library providing functionality to verify that user-agents are who they claim to be.