That said, black-box testing has its limitations -- in particular, reality sometimes rudely intrudes. Take, for instance, the new LJ integration. One of the things I want to test is that the social network synchronizes properly: when you change your flist on LJ, the next time you synchronize the same changes should show up in CommYou. Problem is, LJ is Very Very Slow about propagating those changes, even internally: the sad fact of the matter is that it can take an hour before LJ *admits* that your flist has changed, and there's not much CommYou can do to speed that up. So it's essentially impossible to automate these tests for real, because that could take an hour per test.
This, however, is why CommYou (like almost everything I write) is based on the intensely-pluggable "Ecology" architecture. Basically, I'm a fanatic about making the whole system pluggable and indirectly-referenced, with no static references to anything. It's a mild pain upfront, but times like this are when it pays off.
So yesterday's project was writing a new test harness, at the mid-level. The core of the system is all real, so it's higher-level than an ordinary unit test, but it talks efficiently to the APIs instead of the slower web pages, and it replaces the LJ communications with a little module in the Ecology that simulates LiveJournal. So instead of it taking an hour to propagate each flist change, it's instantaneous, and I can hammer the real system with dozens of flist changes in a couple of seconds. And now I have an API-level test harness for these fiddlier tests that need to fake out the internals of the system. Quite useful: I'll do more with that.
(End result: 3 bugs found, out of 300 lines of new or rewritten code to import LJ communities as well as friends. Not too bad...)