Chapter 5 testing the software with blinders on

Testing the software
Blinders on
Chapter 5

Highlights
What is dynamic black-box testing
How to reduce number of test cases by equivalence
Partitioning
How to identify troublesome Boundary Conditions
Good data values to use to induce bugs
How to test software states and state transitions
How to Use repetition, stress and high loads to locate
Bugs
A few places where Bugs hide

Dynamic black-box testing:
Testing the software while blindfolded
Dynamic black-box testing
Just need to know Input and Output
Product spec shocould tell you
Defining Test Cases
Selecting test cases is the single most important task
That software testers do
Use exploratory testing if you don't have a spec
Simultaneously learning the software, designing
Software, and executing those tests

Test-to-pass and test-to-fail
Test-to-pass
Assure only that the software minimally works
Applying the simplest and most straightforward test
Cases
Always run test-to-pass cases first
Test-to-fail
Designing and running test cases with the sole
Purpose of breaking the software
Strategically chosen to probe for common weakness in
The software
Example

Equivalence partitioning
Equivalence classing
The process of methodically cing the huge (infinite)
Set of possible test cases into a much smaller, but still
Equally valid tive, set.
An Equivalence Class or equivalence partition is a set
Of test cases that tests the same thing or reveals
Same bug
Partitions
Extreme case
Similar inputs
Similar outputs
Similar operation

Equivalence partitioning (2)
Examples
Goal of Equivalence partitioning
Reduce the set of possible test cases into
Smaller, manageable set that still adequately
Test the software
Subjective
It's science but it's also arts
As long as the partitions can cover
Software being tested

Data Testing
Two parts of software
Data
Program
A few key concepts
Boundary Conditions
Sub-Boundary Conditions
Nulls
Bad Data

Boundary Conditions
Assumption:
If software can operate on the edge of its capabilities, it
Will certainly operate well under normal conditions
Basic example
Dim data (10) as integer
Dim I as integer
For I = 1 to 10
Data (I) =-1
Next I
End

Types of Boundary Conditions
Note: boundary conditions are those situations at
Edge of the planned operational limits of the software
Look for the following types:
Numeric, speed, character, location, position, size,
Quantity
First/last, min/max, start/finish, over/under,
Empty/full, shortest/longest, slowest/fastest,
Soonest/latest, largest/smallest, highest/lowest,
Next-to/farthest-from
TIPS: If you have a choice what data you're re going
Include in your equivalence partition, choose data
That lies on the Boundary

Testing the boundary Edges
Testing the data points just at the edge of
Boundary Line isn' t usually sufficient
Good idea to test on both sides of boungdary
Two equivalence partitions
Work Properly --- last one or two valid points
Inside the Boundary
Fail CT to cause an error --- the one or two
Points outside the boundary

Testing the boundary edges (2)
Outside the Boundary
First-1/First + 1
Start-1/finish + 1
Less than empty/more than full
Even slower/even faster
Largest + 1/smallest-1
Min-1/MAX + 1
Just over/just under
Even shorter/longer
Even sooner/later
Highest + 1/lowest-1
Note: The more boundaries you discover, the more
Bugs you'll find

Sub-Boundary Conditions
Boundaries that are internal to the software
Aren't necessarily apparent to an end user
But still need to be checked by the software
Tester
Knowledge about how software works
Powers-of-two
ASCII table
Unicode

Default, empty, blank, zero and none
Default Testing
Windows paint attribute dialog box example
Always consider creating an equivalence
Partition that handles the default, empty,
Blank, null zero, or none Conditions
Why
It's likely a different software path is followed

Invalid, wrong, incorrect and garbage
Data
Garbage data --- test to fail
Good software shocould handle whatever
User can do to it
For users are not always as smart as you
Suppose
Users won't blame themselves, they blame
Software
Software wants numbers, give it letters

State testing
Verify the program's logic flow through its
States
A software state is a condition or mode that
The software is currently in
Example
The software follows a path through the code,
Toggle some bits, sets some variables, Loads
Some data, and arrives at a new state
Being

Testing the software's logic Flow
It's usually possible visit all the States
But it's impossible to traverse all paths to all
States
Solution:
Apply equivalence partition techniques to
Selection of the states and paths

Creating a state transition map
Shocould be specified in the Specification
State Transition diagrams can be drawn
Using different techniques
Examples
Shocould show:
Each unique state that the software can be in
The input or condition that takes it from one
State to the next
Set conditions and produced output when
State is entered or exited

Cing the number of states and
Transitions to test
How to reduce the huge set of possibilities
A set of test cases of workable size
View each State at least once
Test state-to-state transitions that look like
Most common or popular
Test the least common paths between States
Test all the error states and returning from
Error states
Test random state transitions

What to specifically Test
Checking all the state variables
Static conditions, information, values,
Functionality, etc
Windows paint startup state example
The dirty document flag example
Discuss your assumptions about the States
And state transitions with your team's spec
Writer and programmers

Testing states to fail
Testing-to-pass before
Trying valid possibilities
The flip side
Find test cases that test the software to fail
Race Condition
Repetition
Stress
Load

Race conditions and bad timing
Race Condition
Multiple processes racing to a finish line, not
Knowing which will get first
Bad timing
Two or more events line up just right and
Confuse software that Didn doesn't have CT to be
Interrupted in the middle of its operation

Race conditions and bad timing (2)
Situation that might expose race conditions:
Saving and loading the same document at
Same time with tow different programs
Sharing the same printer, communications
Port, or other peripheral
Shutting down or starting up two or more
Instances of the software at the same time
Using different programs to simultaneously
Access a common database

Repetition, stress and Load
Repetition testing
Doing the same operation over and over
Detect memory leaks
Stress Testing
Running the software under less-than-ideal
Conditions
Determine what external resources and
Dependencies it has
Load Testing
Feed it all it can handle

Repetition, stress and load (2)
Two important considerations
Your team's programmers and Project
Managers may not be completely valid tive
Your efforts to break the software in this way
Opening and closing your program a million
Times is probably not possible if you're doing it
By hand

Other black-box test technique
Not Standalone
Finding stragglers
Subjective and not necessarily based on
Reason

Behave like a dumb user
Throw out any preconceived ideas
Make no assumptions

Look for bugs where you 've already
Found them
Two reasons to look for bugs in the areas
Where you 've already found them:
The more bugs your find, the more bugs there
Are
Extends programmers tend to fix only the specific
Bug your report, no more no less

Miscellaneous
Think like a hacker
Seek to find vulnerabilities and exploit
Experience and intuitions can't be taught
Try different approaches