| gabriel rosenkoetter on 13 Oct 2006 00:07:57 -0000 |
|
On Wed, Oct 11, 2006 at 10:30:23AM -0400, Sean C. Sheridan wrote: > > So... you're trying to locate publicly-accessible hosts providing > > services other than HTTP[S] > No, I'm looking for the fastest method to locate publicly available http > servers for a particular domain. I have no interest in non-http services. [...] > Doesn't this assume there is a link from the main domain to the sub > domain? Specifically that somebody took the time to link to the sub > domain via a web page? Okay, so then you care about anything within the IP block that ARIN says the educational institution in question owns that responds on ports 80 or 443, plus maybe 8080 and 8888 for bonus points. Figuring out the IP range(s) you want for a big place like UPenn is more complicated, since their public web page is served via Akamai, but I'll figure that www.cis.upenn.edu is in at least one of their ARIN reservations (they have several, if memory serves), so: % nslookup www.cis.upenn.edu Server: localhost Address: 127.0.0.1 Non-authoritative answer: Name: C1K.cis.upenn.edu Address: 158.130.12.9 Aliases: www.cis.upenn.edu % whois -h whois.arin.net 158.130.12.9 OrgName: University of Pennsylvania OrgID: UNIVER-8 Address: 3401 Walnut Street Address: Suite 221A City: Philadelphia StateProv: PA PostalCode: 19104-6228 Country: US NetRange: 158.130.0.0 - 158.130.255.255 CIDR: 158.130.0.0/16 NetName: UPENN-SEAS2 NetHandle: NET-158-130-0-0-1 Parent: NET-158-0-0-0-0 NetType: Direct Assignment NameServer: NOC3.DCCS.UPENN.EDU NameServer: NOC2.DCCS.UPENN.EDU NameServer: DNS1.UDEL.EDU NameServer: DNS2.UDEL.EDU Comment: RegDate: 1992-03-18 Updated: 2001-04-30 [...] So then you can use a variety of tools (nmap is popular) to step through all of 158.130/16 (in the older syntax, a class B) looking for hosts that respond to a TCP SYN packet on any of the ports of interest. But I wouldn't do that, if I were you, since it'll be pretty unpopular with the UPenn network administrators, some of whom are friends of mine (and, oh hey, read this mailing list), and you'll probably get blocked unless somebody decides they really want to have fun / are bored and calls the FBI on the grounds that you're probing them for security vulnerabilities. Sure, you're not, and that wouldn't hold water, but it could certainly make your life difficult, never mind your business if it depends on it. The point that I'm trying to make by example and that others have made explicitly is that what you're trying to do here smells pretty bogus and covers trodden ground already. Even if what you're looking for a student-run web pages on student-owned servers to which Official University pages do not link, Google (and Yahoo, and MSN, and ...) has got those already. > It seems inefficient to send a spider out for this purpose and query every > page I find for links to potential sub domains. I was hoping the DNS > query could be used to quickly find the answer. Given a choice I'd think > it much more neighborly to query one dns server one time vs umpteen > million http head requests. You'd think so, especially given you're intersted in the "network" qua names rather than the "network" qua addresses (which are definitely not the same thing), but the reason that people don't let you do that is that it provides you a mostly-good hitlist for their network for you to go looking for security vulnerabilities on those hosts. Sure, you can get that by doing roughly what I describe above, but you should damn well have to spend the cycles and time to do it. Why should they help you take advantage of them? Even I don't do that (go ahead, try an AXFR of eclipsed.net... there are, um, three maybe four people who read this mailing list that can do that, and I expect that only two of them know it), and I've only got a handful of hosts for which I'm directly responsible to keep track of security vulnerabilities. > Since the answer is, apparently, that I need a spider I ask again... any > good books on spiders? Is the Spider Hacks (O'Reilly) book any good? Has > anyone seen the new O'Reilly book? I have no idea. By habit, I'd guess the ORA book is fine... but I'd consider a web spider a 30 minute hack in Perl, never bother with a book like that, and use LWP for it, referencing its man/perldoc documentation if I wanted any, so I figured I was the wrong person to answer that question and ignored it. -- gabriel rosenkoetter gr@eclipsed.net Attachment:
pgpHx523x4VSk.pgp ___________________________________________________________________________ Philadelphia Linux Users Group -- http://www.phillylinux.org Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
|
|