Title: Department of Computer Science Seminar
Date January 19, 2000
Time: 4:00 pm to 5:00 pm
Venue: Room N101, CSIT Building [108]
Speaker: Dr Peter Bailey (Research Fellow, DCS at ANU)
Description: "Chart of Darkness : Mapping a Large Intranet"


                        Abstract

We introduce and define the concept of dark matter on the Web.
  Dark matter for a person or Web crawler consists of pages that 
  they cannot reach and view, but which another observer can. Dark 
  matter is important to our understanding of the Web in that the 
  portion of the Web any of us can see depends on our viewpoint. 
  Different observers see different overlapping sections of the Web. 
  However, no one can see all of the Web, even if they want to. 

  We categorise the various types of dark matter that exist and how
  they may be discovered.  Formal definitions of what constitutes
  lightness and darkness on the Web are formulated in terms of
  reachability. Our case study of dark matter within the Australian
  National University's intranet is reported. We estimate that 87% of
  the ANU's intranet is dark to our local search service, and 37% is
  dark matter kept deliberately private from almost everyone.
  Finally, we discuss some of the implications of dark matter for
  estimating the size of the Web and for general Web searching.


 Work by: Peter Bailey, Nick Craswell and David Hawking (ACSys WAR Project)


Biography: Dr Peter Bailey is a researcher in the ACSys WAR Project and
an employee of ANU. In past lives, Peter has worked with
Object Technology International, and the ANU/Fujitsu CAP 
Program. He is currently engaged in a long-term subtropical 
rainforest reafforestation project in northern NSW, and is 
learning about etching.


URL: http://cs.anu.edu.au/lib/seminars/seminars00/dept2000011