Kaner Cem - The domain testing workbook

COPYRIGHT 2013 CEM KANER & DOUGLAS HOFFMAN

ALL RIGHTS RESERVED

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, FAX (978) 646-8600 or on the web at .

Limit of Liability / Disclaimer of Warranty: While the Publisher and authors have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential or other damages.

Library of Congress Cataloging-in-Publication Data:

Kaner, Cem; Padmanabhan, Sowmya; & Hoffman, Douglas

The Domain Testing Workbook

Library of Congress Control Number: 2013947216

ISBN 978-0-9898119-1-0

The Context-Driven Press logo and The Domain Testing Workbook cover design are by Susan Handman, Handman Design, New York.

Editor: Rebecca L. Fiedler

Copy Editor: Karen Fioravanti

These materials are partially based on research that was supported by National Science Foundation research grants EIA-0113539 ITR/SY+PE: Improving the Education of Software Testers and CCLI-0717613 Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Here is a list of several tasks that people often do as part of a domain testing analysis. We organized the books chapters around this list because it puts the tasks into a logical order.

Please note that for any particular product or variable, you might skip several of these tasks or do them in a different order than we list here.

A. Identify the potentially interesting variables.

B. Identify the variable(s) you can analyze now. This is the variable(s) of interest.

C. Determine the primary dimension of the variable of interest.

D. Determine the type and scale of the variables primary dimension and what values it can take.

E. Determine whether you can order the variables values (from smallest to largest).

F. Determine whether this is an input variable or a result.

G. Determine how the program uses this variable.

H. Determine whether other variables are related to this one.

I. Partition the variable (its primary dimension).

If the dimension is ordered, determine its sub-ranges and transition points.

If the dimension is not ordered, base partitioning on similarity.

J. Lay out the analysis in a classical boundary/equivalence table. Identify best representatives.

K. Create tests for the consequences of the data entered, not just the input filter.

L. Identify secondary dimensions. Analyze them in the classical way.

M. Summarize your analysis with a risk/equivalence table.

N. Analyze independent variables that should be tested together.

O. Analyze variables that hold results.

P. Analyze non-independent variables. Deal with relationships and constraints.

Q. Identify and list unanalyzed variables. Gather information for later analysis.

R. Imagine and document risks that dont necessarily map to an obvious dimension.

This book is dedicated to my step-mother, Rosemary Kaner, and the spirit of my father, Harry, who raised me in their businesses and taught me to cherish integrity, skepticism, the applicability of mathematics in all situations and the transcendant value of the Person in each of them, of the fundamental importance of data, and the delight in well-designed tables.

Cem Kaner

To my children who are my inspiration and motivation in life: Ria, my wonderful daughter, and Simba, my awesome dog.

Sowmya Padmanabhan

Id like to dedicate my work to the ladies in my life, Connie and Jackie who have stoically endured my sometimes long absences, many of which were for collaboration on this book.

Doug Hoffman

People learn what they do. To develop skills, people need to practice. To practice, people need examples to practice on, time to work on them, and feedback.

This book is about a single software testing technique, domain testing. You might know it as equivalence class analysis or boundary testing. It is our fields most widely-taught technique.

Testing is a cognitively complex activity. Developing competence with cognitively complex skills requires mastery of routine tasks and formation of schemas (cognitive maps) that can guide you as you do tasks that require more conscious effort (van Merrienboer, 1997).

The fundamental problem underlying testings complexity is that every tester, of every nontrivial program, must choose from an impossibly large set of potential tests. Test techniques provide a cognitive toolkit for making these choices.

A test technique is both, a design tool and a selection tool:

As a design tool, it tells you what to include in the test.

As a selection tool, it provides a method for sampling a relatively small number of interesting tests from the vast set of possibilities.

Domain testing is primarily a sampling strategy:

Divide the possible values of a variable into subsets of values that are similar in some way (well call them equivalent).

Design your tests to use only one or two values from each subset. Pick extreme values (well call them boundaries) that maximize the likelihood of exposing a bug.

A critical problem with much industrial and academic training is that we teach test techniques as if there were obvious procedures to generate the correct set of tests. There are no such procedures. Instead, each technique involves its own way of thinking; one you get better at over time as you gain experience.

To learn a test technique is to learn a way of thinking about how to test well, not how to follow a procedure.

According to the domain-testing way of thinking, we focus test designs on the values of variables. We select values for those tests by partitioning variables values into equivalence classes. We pick values from within those classes that are the most extreme (such as the boundaries) because were looking for the values most likely to drive the program to failure.