Carl Endorf - Intrusion Detection and Prevention

Carl Endorf
Dr. Eugene Schultz
Jim Mellander

McGraw-Hill/Osborne

New York Chicago San Francisco
Lisbon London Madrid Mexico City Milan
New Delhi San Juan Seoul Singapore Sydney Toronto

McGraw-Hill/Osborne
2100 Powell Street, 10th Floor
Emeryville, California 94608
U.S.A.

To arrange bulk purchase discounts for sales promotions, premiums, or fund-raisers, please contact McGraw-Hill/Osborne at the above address. For information on translations or book distributors outside the U.S.A., please see the International Contact Information page immediately following the index of this book.

Intrusion Detection & Prevention

Copyright 2004 by The McGraw-Hill Companies. All rights reserved. Printed in the United States of America. Except as permitted under the Copyright Act of 1976, no part of this publication may be reproduced or distributed in any form or by any means, or stored in a database or retrieval system, without the prior written permission of publisher, with the exception that the program listings may be entered, stored, and executed in a computer system, but they may not be reproduced for publication.

1234567890 CUS CUS 019876543

ISBN 0-07-222954-3

Publisher
Brandon A. Nordin

Vice President & Associate Publisher
Scott Rogers

Executive Editor
Jane K. Brownlow

Project Editors
Jenn Tust, Jody McKenzie,
Elizabeth Seymour

Acquisitions Coordinator
Jessica Wilson

Technical Editors
Scott Campbell
Daniel Peterson

Copy Editors
Andy Carroll, Marcia Baker,
Lisa Theobald

Proofreader
Paul Medoff

Indexer
Valerie Perry

Composition
Dick Schwartz, Lucie Ericksen

Illustrators
Kathleen Edwards, Melinda Lytle, Michael Mueller

Series Design
Dick Schwartz, Peter F. Hancik

Cover Design
Theresa Havener

This book was composed with Corel VENTURA Publisher.

Information has been obtained by McGraw-Hill/Osborne from sources believed to be reliable. However, because of the possibility of human or mechanical error by our sources, McGraw-Hill/Osborne, or others, McGraw-Hill/Osborne does not guarantee the accuracy, adequacy, or completeness of any information and is not responsible for any errors or omissions or the results obtained from the use of such information.

I would like to dedicate this book to my beautiful wife, Stashi, and to my two children.

Carl Endorf

The portion of this book that I wrote is dedicated to my three daughters, Sarah Schultz, Rachel Schultz, and Leah Schultz, all of whom I could not be prouder.

Gene Schultz

To my wife, Marilynne, who patiently kept the home fires burning during the completion of this project.

Jim Mellander

Acknowledgments

Many people contributed to the writing of this book. Thanks to Scott Campbell for his excellent technical editing. Special thanks to Chad Schieken for his contribution of on RealSecure; and to Frank Simorjay for his chapter on NFR Security. The direction, cooperation, and helpfulness of the entire McGraw-Hill staff with whom we worked, including Jane Brownlow (executive editor), Jody McKenzie (senior project editor), Jenn Tust (project editor), Elizabeth Seymour (project editor), Jessica Wilson (acquisitions coordinator), copy editors Andy Carroll, Marcia Baker, and Lisa Theobald, and DTP composition specialist Kelly Stanton-Scott, are very gratefully acknowledged. These dedicated professionals made all the difference in the world in how our writing and revision efforts went, and in the way this book turned out. Finally, the book would not have been a success without the support of our families!

About the Authors

Carl Endorf, CISSP, CISM, is a senior technical security analyst for one of the largest insurance and banking companies in the United States. He has practical experience in intrusion detection, forensics, corporate investigations, and Internet security. Carl has written many security articles for industry publications as well as three security-related books. He has a masters certificate in information security management and is currently finishing his master of science degree at the University of Illinois in management information systems.

Eugene Schultz, Ph.D., CISM, CISSP, is a principal engineer at Lawrence Berkeley National Laboratory of the University of California. He is the author/co-author of four previous books: one on Unix security, another on Internet security, a third on Windows NT/2000 security, and a fourth on incident response. He has written over 100 published papers. Gene is the editor-in-chief of Computers and Security and is an associate editor of Network Security and Information Security Bulletin. He is a member of the editorial board for the SANS NewsBites, a weekly information security-related news update and is on the technical advisory board of three companies. He was adjunct professor of computer science at Purdue University, where he taught courses and participated in research in the CERIAS (Center for Education and Research in Information Assurance and Security) program. He has received the NASA Technical Excellence Award, the Information Systems Security Association (ISSA) Professional Achievement and Honor Roll Awards, the ISACA John Kuyers Best Speaker/Best Conference Contributor Award, the National Information Systems Security Conference (NISSC) Best Paper Award, and has been elected to the ISSA Hall of Fame. While at Lawrence Livermore National Laboratory, he founded and managed the U.S. Department of Energys Computer Incident Advisory Capability (CIAC). He is also a co-founder of FIRST, the Forum of Incident Response and Security Teams. Dr. Schultz has provided expert testimony before committees within the U.S. Senate and House of Representatives on various security-related issues and has served as an expert witness in legal cases.

Jim Mellander is a principal engineer at Lawrence Berkeley National Laboratory of the University of California and holds the position of incident response manager. Jim and his team detect, investigate, and respond to cyber security incidents, using many of the techniques in this book. Jim has written several notable security software programs, including Update, a Unix-based sniffer detector, and Kazaa Obliterator, which disrupts many types of unauthorized peer-to-peer traffic in an enterprise. Jim is the author of a number of articles and was the recipient of the 2001 Best Paper Award in Information Security Bulletin. He lives in the San Francisco Bay Area with his wife, three dogs, and a cat.

About the Contributing Authors

Chad Schieken is a senior network systems consultant for International Network Services. He is responsible for the delivery of security engagements including risk assessments and infrastructure design. He has worked in the security field for eight years, with experience in some of the largest and most complex networks and organizations in the world. His background includes several years of Unix systems administration as well as studies at the University of Pittsburgh and Rutgers University. Chad speaks at seminars and conferences; most recently at the Philadelphia chapters of the Infragard and ISSA.

Patrick Swissman Ramseier, CCNA, CISSP, is a CSA systems engineer at Cisco. Patrick started out as a Unix systems administrator. Over the past 14 years, he has been involved with corporate-level security design, architecture reviews, vulnerability assessments, VPN support, physical, network and operating system security (Unix-Solaris, Linux, BSD, and Windows NT/2000), training, research, post- and pre-sales. He has a B.A. in business and is working concurrently on his masters and doctorate in psychology.