Yep, The latest update on my own search engine is that I have made all the crawlers into multithreaded async mode. It's like they work all together and get the best of my internet bandwidth to download websites and crawl them.
It used to be like each crawler had to wait for the other crawler to finish. now with the new quadcore 2 64bit server which I got the crawling has become easy. It used to take an hour to crawl 6 listings and now I crawl 12 listings within 10 minutes.
good good, I'm going to run it on test phase on a very good hosting. still work on the front side is not completed!
It used to be like each crawler had to wait for the other crawler to finish. now with the new quadcore 2 64bit server which I got the crawling has become easy. It used to take an hour to crawl 6 listings and now I crawl 12 listings within 10 minutes.
good good, I'm going to run it on test phase on a very good hosting. still work on the front side is not completed!
Leave a comment