Test Driven Development – Is it good or bad?

For short: BAD.

I’ve been thinking about this for a while and reading Michael Puleio’s blog [^] [^] has made me think even more about this.

What is often referred as Test Driven Development (TDD) is actually Test First Development (TFD), which is kind of stupid taking in account tools like Visual Studio 2005 Team Edition for Software Developers (and Professional when Orcas cames out).

Now that TFD is out of the way, my beef is with the Test Driven part (or Example Driven Development (EDD) if you like other flavors). Came on, is your software really being driven by the tests? Are you really writing software just to satisfy your tests? I’m not.

Test driven development also makes you design for testability instead of designing for funcionality, scalability and other such much nicer things. And if you use tools like TypeMock.Net, you can test almost anything without designing for testability.

My software development is driven by the need to comply with requirements. How am I doing it? Assisted by tests. So, I’m doing Test Assisted Development (TAD). (Should I copyright this?)

23 thoughts on “Test Driven Development – Is it good or bad?”

  1. I agree. I’ve worked in several strong agile-process groups that have created customer-accepted software that CUSTOMERS raved about. It’s really customer-design-driven software, all tested with quality unit tests written immediately after development. The design thinking was natural, logical, optimized, contract-based, clearly understood, and documented. And the maintainence costs were minimal. And it was great personal fun to rapidly solve customer-related design problems every day.

    Now recently I have experienced test-driven-development – the real-evangelist-pushed-thing – for about a month. I discovered that the amount of test development time and test code greatly exceeds that needed for the application code itself. And the amount of features becoming periodically available to the customer in a timely fashion needed for good adaptive feedback? Very few, just a trickle. This very slow completion of features obviously results in very high development costs. And with TDD there is also a terrific explosion of the number of classes used, creating technical confusion for developers and a big maintenance headache for new support staff. And – as is typical for TDD projects – the code documentation for important non-obvious explanations is essentially non-existant.

    Someday soon, companies will wake up and stop throwing their money – and their developers – away on TDD. And remember: XP came from just one guy writing in Smalltalk about 12 years ago whose first major XP project FAILED at Chrysler Credit corporation; the highly successful Agile process arose as a response to XP from hundreds of professionals 6 or 7 years later who wanted something much more effective and practical than XP.

  2. My beef is more on people who use these acronyms without knowing what they stand for and assuming that putting some attributes in some class with fixture appended to its name automatically gives the code a seal of high quality.

    If you do a search on TDD you’ll find contradictory definitions. So, which one is the correct one and who’s really doing TDD? Is it really useful or not?

    That being said, I think unit testing tools can be used to do usability tests on APIs (EDD – example driven design).

    Unit tests (with good unit test tools) can be very useful when developing because who can test units of code without the need of having to set up large environments and unpredictability of the behaviors of such environments. Unit tests are also good regression test tools.

    I like to write code with the help of unit tests (assisted – TAD) and end up with a TVS (Test Validated Solution) (Should I copyright this one too?). This is a pain to do if you start with a bad design, because you’ll have to rewrite code and tests.

    Bottom line, for me TDD is either a bad name or a bad practice.

  3. this is getting into a nasty habit :) again, i disagree. Sure, you can “auto-generate” your tests with visual studio tools (i recall having seen it somewhere, though i haven’t tried it) but the idea of using tdd is that you’re only writing the needed code for a test to pass. if you do that with that new methodology you’ve just invented, then you should be ok. Now, the problem is that when you write your code first, normally your tests won’t be able to test everything you’ve put in the method. to do that you’ll often (if not always) need to perform some refactoring. if you do that (I really doubt that anyone does it!), cool. if you don’t, you end up with the same low quality code you’re saying most guys end up with.

    so, to me, TDD, when used correctly, is great! the problem is that it’s too hard to find someone that is willing to spend the necessary time on it. Unfortunately, I must leave now (vacations, remember? :)), but I’ll be coming back to add more rants on this :)

  4. I still think you’re still mixing design with development.

    It’s just that words have meanings, you know.

    If we would seat side by side, you’d find out that we work pretty much the same way, and that, by some definitions, neither of us is doing TDD. At least I’m not, because I refuse to call what I do TDD.

    Let’s take this to another level. Can you show me an “official” definition of TDD. Or any definition at all?

  5. Paulo,

    I agree about people who use buzzwords without knowing what they mean. But I disagree about TDD.

    I’ve been using TDD with some success, on and off, for the past few years (pretty much since I learned about it). I’ve used it on everything from web services to web applications and console applications.

    Of course you need to do up-front design to one degree or another, and requirements must be clear enough; this can be done iteratively. But once the requirements have shown you what needs to be done,  and the design has shown you the shape of your code, I find it very worthwhile to create unit tests. These tests define what behaviors are important to me. These are the things that my code will have to do before it can be considered “done”.

    The unit tests are constructed so that they fail first. This is done so that you know that the unit test _can_ fail – it’s no good writing a unit test which cannot fail. The code is then written to cause the unit test (and all of your other related unit tests) to succeed. The test has now tested the code. You can then safely refactor both the code and the tests, because they test each other. If you screw up the unit test, it will fail, as it will if you screw up the code.

    The result is an executable statement of the correctness of the code; a statement you can test on every check-in, or every nightly build, or repeatedly during the day. The feedback is good for your self-confidence (“it works, it still works, it’s _still_ working even now! Damn but I’m good!)

    Now, I’ll admit that I don’t have any numbers as to how this affects code quality. I’m keeping an eye on this on my current project. I want to see if there’s any correlation between the numebr of defects reported vs. whether or not the code was developed via TDD. I also want to see if it’s the case that the defects are due to me not creating the unit test which should have caught the defects!

    John

  6. Hi John.

    This post started as a joke because the definition for “driven” in Encarta Dictionaries (and many others – I looked around) is:

    1. compelled by personal need: striving to achieve personal goals because of a strong need or inner compulsion
    * Driven people are often overachievers.

    2. caused by: having a particular thing as a principal cause (used in combination)
    * a demand-driven economy

    Well, I don’t develop compelled by the need to do tests. :)

    This has nothing to do with the usefulness of unit tests and unit test tools (which can be used in other tasks like integration tests)

    As to the practice, I only write the test before the code if I’m designing the APIs. If the API was already been defined there’s no need to write something that will fail to compile. (And this could be hazardous to my health because I don’t like to leave the office (to go lunch or home) when code compiles)

    I develop my unit tests in conjunction with code coverage to be sure that all code has been tested.

    I also use extensively TypeMock.NET to make the tests as unitary as possible. Even when testing on class, I mock all other methods to test one method at a time.

    As to the confidence, I remember one time when I was showing a colleague of mine my perfect code with 100% of code coverage in the unit tests. I was 100% sure it would always work as I planed 100% of the times. I was very confident until we did integration tests and my design proved to be wrong. It was the perfect code to do the wrong thing. :)

  7. When speaking to an executive, do you really tell him that the studies and case studies he’s read from reputable research firms are all lies? The studies that associate test-associated development with higher quality? There is so much evidence for the benefit of TDD (disgarding objections the exact meaning of the terms used) that this post can’t be considered well-informed.

  8. This post to me sounds more like the rant of a misguided developer that has failed to grasp onto TDD (or any variant there of). I wonder if the auther has really given testing a solid try?

  9. I’ve been doing TDD for about 2 years now, and at first, I did have the sort of class explosion you mentioned. It was only after I put it together with the GoF, Fowler’s PoEAA, Evans’ DDD and eventually behavior-driven design did good designs emerge from my software and tests. I had to completely relearn how to design, now always doing the simplest thing that works and doing a sort of JIT design. Every test I write now centers around behavior and functionality. TDD is a paradigm shift in your development practices, and takes time and effort to master.

    As for using TypeMock with unit tests, I had the same experience with it as I did with typed DataSets. Easy to use the first time, but an absolute PITA to maintain. I always felt my tests should be concerned with the public, observable behavior (or state) of a module. Any time I started tinkering with testing the internals, my tests started breaking much more often.

    I can’t speak for all TDD practitioners, but all I need to do is look at code I wrote test-after (or test-never) and code I wrote test-first to know I could never go back. I think your mileage may vary on TDD, but at least we can all agree it’s better to have code with tests than without. Now back to Michael Feathers’ book…

  10. I’ve been doing TDD for about 2 years now, and at first, I did have the sort of class explosion you mentioned. It was only after I put it together with the GoF, Fowler’s PoEAA, Evans’ DDD and eventually behavior-driven design did good designs emerge from my software and tests. I had to completely relearn how to design, now always doing the simplest thing that works and doing a sort of JIT design. Every test I write now centers around behavior and functionality. TDD is a paradigm shift in your development practices, and takes time and effort to master.

    As for using TypeMock with unit tests, I had the same experience with it as I did with typed DataSets. Easy to use the first time, but an absolute PITA to maintain. I always felt my tests should be concerned with the public, observable behavior (or state) of a module. Any time I started tinkering with testing the internals, my tests started breaking much more often.

    I can’t speak for all TDD practitioners, but all I need to do is look at code I wrote test-after (or test-never) and code I wrote test-first to know I could never go back. I think your mileage may vary on TDD, but at least we can all agree it’s better to have code with tests than without. Now back to Michael Feathers’ book…

  11. I’m still out on true TDD (writing the tests first) but I do have some numbers from one of my previous projects. The project was an ASP .NET web application with about 20k lines of code or so. I was the main developer and there were two testers executing the test plan. I utilized NMock2 for mocking and csUnit for my unit tests. Part of my Nant build script was to run unit tests and create a code coverage report in NCover. There were a total of about 300 bugs logged against the project (this is an estimate, the project had 360 bugs total but I don’t know how many of those were duplicates or pre-existing before I started working with the code, but I think 50-60 of those is a fair estimate). I had unit tests on both the database and the business layer, for a total of 107 unit tests with about 90% code coverage on those layers (according to NCover). For the complete life of this project, every single one of those 300 or so bugs was logged against bad requirements or UI issues. There were no bugs logged at any time against any code that was covered by a unit test. I did NO modifications to ANY code that was covered by unit tests as a result of a defect (there were two requirements changes in the middle of the project that required code changes but these caused me to change the unit tests also). My point is that proper use of unit tests to ensure that objects behave the way they are supposed to will have a huge impact on the quality of the software that you are creating.

  12. Jeffrey Palermo:

    Jeff, you said it all.

    “test-associated development”

    “disgarding objections the exact meaning of the terms used”

    My beef is just with the middle D.

  13. Jimmy Bogard:

    Jimmy,

    I always unit test my code. Hopefully, using automatic tools. But, when needed, the debugger in Visual Studio can do the trick.
    I’ve been playing with TypeMock.NET and I find it amazing. With it I can make really unitary tests. Even when mocking HttpApplication, HttpContext, HttpRequest and HttpResponse is needed. I love a 100% code coverage.

  14. Jeff Tucker:

    Jeff,

    I just can’t bring myself to write the tests first.

    Maybe because I can write, compile and execute code in my mind. And that would mean that I’m “writing” tests first.

    Right now, in one of the projects I’m involved, I’m literally dictating the overview of the architecture and API. Probably I should write the tests (providing I had the time) and tell the developer to come back when he could make his code pass all my tests.

    But I still maintain that Test First is not Test Driven. This Driven thing just bothers me.

  15. Thank you all for offering me your experience in TDD.

    I feel I’m beginning to shift a bit on my beliefs (and those who know me know that that can be hard to accomplish).

    Well, serious discussions can have that effect on you.

    I wonder what attracted people to this one and a half month old post.

  16. You should design to test! Not being able to test in isolation is a sign of a poor design due to high coupling and low cohesion. (IMHO) Following the principles of good design will lead to testable code.

    Apparently you have not had an “Ah, ha!” moment when a unit test catches a bug introduced by another bug fix that is so obscure you would swear no test plan was going to catch. Typically I find these when I write cross-check unit tests. I’ve had a few that would have made it to production and cost my company way more than the time it took me to design for testing and to write the tests.

    One other note, only test what is important to test! Automatically generating a unit test is worthless.

    I understand how everyone’s opinion is based on their own experiences though.

  17. I have no problem with design with testability in mind as long as it’s not testability driven.

    What I mean is that the architecture and its APIs should be primarily reliable, preformant and easy to use and not just easy to test.

    And with tools like TypeMock.NET, you can not only avoid design for testability but you can get the comfort of 100% code coverage with your tests.

    I how and why is apparent that I haven’t had and “Ah, ha!” moment like you say. I’ve had more that I would like to. Some of them were caught on integration testes. 100% code coverage with tests and all tests passed. It did exactly what I wanted it to do. It was the right code to do the wrong thing.

    I would like to have really automatic test generation. With the help of code coverage I would be able to see what tests were missing. Missing that and being a control freak, I’m pleased with Visual Studio’s test stub generation and especially with its private accessors generation.

    Thanks for your feedback, Mark.

  18. @Dwight

    Same can be said of procedural vs. OO. It depends on the people, not TDD.

Comments are closed.