Re: [PLUG] cname lookups

[bergman@merctech.com]

In the message dated: Fri, 06 Oct 2006 14:19:08 EDT,
The pithy ruminations from "Sean C. Sheridan" on 
<Re: [PLUG] cname lookups> were:
=> Stephen,
=> 
=> 
=> I'm trying to identify publicly accessible subdomains on a variety of .edu

Just because a domain is publically accessible doesn't always mean that it
should be accessible, or that the content is intended to be searched.

Following CNAMEs is not going to be very comprehensive or reliable. As you've 
found, many sites prohibit arbitray domain XFERs, especially recursive 
transfers. It's not 1993 anymore.

=> domains.  We're considering developing an "education search engine".

I don't know your business plan or anything, but how's that different from 
google, with the proper parameters?

If you're looking for content that's actually meant to be searched, approaching 
the question from the point of view of the content, rather than the hosts, may 
be more appropriate. In other words, spider the public web sites in order to 
map the site and build your search engine. Ooops, I've described 
$COMMERCIAL_SEARCH_ENGINE.

Any other kind of network discovery may well be viewed as hostile, and blocked.

Mark "not speaking for Penn right now" Bergman

=> 
=> 
=> Sean C. Sheridan
=> scs@CampusClients.com
=> 
=> Campus Party, Inc.
=> 444 North Third St.
=> Philadelphia, PA 19123
=> (215) 320-1810
=> (215) 320-1814 fax
=> http://www.CampusClients.com
=> http://www.CampusParty.com
=> 
=> 
=> 
=> > On Fri, Oct 06, 2006 at 01:26:39PM -0400, Sean C. Sheridan said:
=> >> Ok, forgive the noob question...
=> >>
=> >> I'm trying to find a way to generate a list of servers on a particular
=> >> domain.  In my mind the records I need are cname records.  But I can't
=> >> seem to generate a meaningful response...
=> >>
=> >> By way of example, let's try Upenn.
=> >>
=> >> I am trying to locate domains such as:
=> >> isc.upenn.edu
=> >> www.cis.upenn.edu
=> >> www.rad.upenn.edu
=> >
=> > [snip]
=> >
=> >> Any thoughts?
=> >
=> > upenn.edu apparently does not allow axfr's to arbitrary hosts, so it's
=> > unlikely this approach is going to work.
=> >
=> > Let's take a step back - what are you trying to do?  Map out the network?
=> > Just figure out what all the subdomains are?  You can probably roughly
=> > map the network with public information (whois, plus rDNS queries for
=> > every address in the whois info), but getting a list of forward lookups
=> > that are subdomains is going to be harder.
=> > --
=> >  --------------------------------------------------------------------------
=> > |  Stephen Gran                  | Modesty:  The gentle art of enhancing
=> > |
=> > |  steve@lobefin.net             | your charm by pretending not to be
=> > |
=> > |  http://www.lobefin.net/~steve | aware of it.   -- Oliver Herford
=> > |
=> >  --------------------------------------------------------------------------

-----
Mark Bergman    Biker, Rock Climber, Unix mechanic, IATSE #1 Stagehand

http://wwwkeys.pgp.net:11371/pks/lookup?op=get&search=bergman%40merctech.com

I want a newsgroup with a infinite S/N ratio! Now taking CFV on:
rec.motorcycles.stagehands.pet-bird-owners.pinballers.unix-supporters
15+ So Far--Want to join? Check out: http://www.panix.com/~bergman 


___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug