I am trying to create domains list for selected .tld and then check them for some metrics.
Is there any software which I could use?

Ideal software abilities:
- multithreaded (can use as much as possible from CPU and bandwidth)
- I will set one website (or few websites) as start position for that software - for example something like DMOZ.org
- software will start to crawl them and will try to find other domain names used in a href tags and those domains will add to batch for checking - and...

Software for finding all domains (with selected tld - optional)