I was talking with Dave Woods today and we slid into this topic.  We’ve all been in a situation where we release a new version of our software to a client’s testing environment and they immediately ask “So, what should I test?”  For most of us developers this is a tough question to answer.  We’re not experts in the black art of system test design and writing.  Heck, most of us are only a couple of years into full blown unit testing.  So how do you answer this?

I figure I’ve answered the question three different ways. 

1. I tell the client specifically what areas I changed.  I suggest that they can be confident testing only those specific things.  Usually these changes have few, if any tentacles through the rest of my codebase.  Sometimes I will suggest this method if the change is simple to test and part of a common utility.  Testing JavaScript that is used on every webpage and only changes the image of a control common to all those pages is not worthy of one or more tests per page.  Instead I would suggest testing it on one or two randomly selected pages.
2. I tell the client which modules, pages or processes have been, or could be affected by the modifications that I made.  Usually this answer would be given if I have a moderate confidence that the testers will not see side effects in places that they may not be expecting them. 
3. I tell the client to retest the entire application.  What I think rather than say is “If you guys only knew how ridiculously scared of breaking all kinds of things in the app when I made this change, you’d run for the hills.”  If I were a client and I was told by a dev team that I should retest the entire application, I would take it as a sign of some serious code and development process smells.

Underlying all three answers is the common question I have to ask myself of my code.  “How much faith do I have in the isolation of the changes that I just made?”  More confidence should mean a smaller swath of testing.  Occasionally you will get bit by this.  You’ll say “Just test that one thing there” and, sure enough, 10 minutes after it gets released to production your phone will be buzzing with bugs that the isolated testing didn’t find.

The other thing to remember is that your automated tests (assuming that you have them) can give you enormous confidence in the fact your changes didn’t ripple through the system like a mini tsunami.  The only thing is that you see them….not the client.  The boost in your confidence is not easily transmitted or felt by the people who haven’t worked with your automated tests, so they will be wary.  No matter how much you say “None of our automated tests failed after we made the changes”, clients will be justifiably skeptical.