So I posted an ad for a part time assistant on Craig’s List. And being test obsessed, I devised a test to give me a good indication of whether a given candidate had enough of the skills I needed to bring in for an in-person interview.

I asked the candidates who wrote an articulate reply to the job posting (and there were a lot of them!) to do a little research and find out what conference I’m speaking at in August, where the conference is, and what the titles of my presentations are.

It seemed to me like this was a pretty easy test that would weed out anyone who couldn’t do basic research on the web. But I made the test harder than I intended by asking about conferences in August. I forgot that there are two conferences listed on my web site for August: Agile2008 (where I am not speaking), and STANZ (where I am giving two talks).

When I realized my mistake, I thought “No problem. Good candidates will search for my name in the Agile2008 program and realize I am not speaking there.” Then I tried it myself and discovered that searching for a specific speaker is apparently a use case that the Agile conference organizers didn’t think about when they published the program this year. Even I found it difficult to determine absolutely that I am not on the program. There are too many places to look, and even the PDF program is a little difficult to search for a given speaker’s name.

Sometimes what seems like a simple test - whether of software or of a person - turns out to be much more difficult than we intended or imagined. Although I suppose that as long as none of the job candidates crash as readily as software, we’ll all be OK.

And if any of them manage the task without giving up, I will definitely know something about their ability to find information on the web!

In it, I propose standardizing on places to put different kinds of information associated with automated functional tests.

It seems to me that one of the key success factors for the xUnit family of unit testing frameworks is that they gave us just 5 places to put code related to unit tests: setup, test, teardown, suite setup, suite teardown. That simple organization has a powerful focusing effect, enabling (or, perhaps, forcing) developers to narrow their attention down to just the code needed to create one little itty bitty unit test at a time.

Functional testing frameworks have no such common, standardized structure.

FIT has given us something close with the notion of a test in natural language in a table and fixture code to hook that test to the software under test. If we extend that model a little to include the idea that we may well be testing against an external interface, like a web interface, where a driver, like Watir or SeleniumRC, would be handy, we end up with 3 big categories of things:

• Tests: scenarios describing the actions and expectations, expressed in natural language with keywords
• Fixtures: code that connects the keywords in the test to actions in the software under test
• Drivers: libraries like Watir, SeleniumRC, Perl’s Win32::GUI, etc. that know how to address external interfaces such as Web interfaces, thick client GUIs, command line interfaces, soap/XML calls, etc.

That’s the direction I think test automation tools in general are headed, and it’s an important evolution with profound implications. However, I’m still figuring out how to explain how this structure differs from what traditional tools offer, and the significance of those differences.

At the very end of the “A Place to Put Things” video, there’s a little exchange between two of the participants in the workshop. A woman’s voice says, “She just pulled that together in like a minute!” That’s Jennitta Andrea speaking. She was the co-organizer of the AA-FTT workshop.

In response, Ward Cunningham says, “Oh, no. I think she’s been pulling that together for the last 3 years.”

Ward’s right. And I’m still pulling together my ideas and figuring out how to articulate them. So while you don’t see much evidence of it on my blog, I am actually spending a fair amount of time writing, and rewriting. More soon. I hope.

I sympathize.

I recently fought my way through the process of automating a test to reproduce a bug on a legacy(*) web application that had no IDs on any of the elements that I wanted to address. And I thought it might be helpful to capture some of my lessons learned here in case they help someone else. Also, I’m putting this here so I remember what I did.

Lesson 1: SeleniumRC Rocks!

I’ve been extolling the virtues of SeleniumRC for quite a while. This project gave me the perfect opportunity to refresh my Selenium skills. And I’m delighted to report that SeleniumRC is even better than I remember it being. First, I can write my tests in my programming language of choice (Ruby). Second, it has a wide variety of ways to locate these pesky non-ID’d elements. Third, every time I ran up against a road block, I discovered that the smart folks who make Selenium had already anticipated the problem and found a solution.

Lesson 2: Selenium Server Flags can Solve Common Execution Problems

In my particular case, the app I was testing didn’t play nicely in IFrames. This is a problem: by default Selenium runs the web app in an IFrame in the same browser window where it displays its own status. Fortunately, it turns out that there is a -multiWindow flag to solve exactly this problem. I solved the IFrame problem by running the Selenium Server like so:

java -jar selenium-server.jar -multiWindow

There are a variety of other Selenium Server flags that address other common problems. See the Server Command Line Options documentation for a full list.

Lesson 3: ‘Permission Denied’ Errors Probably Mean the App Violates the ‘Same Origin Policy’

Once I’d gotten to the point where I could launch the app, I started encountering very puzzling Permission Denied errors. I vaguely recalled that such errors probably meant there was some problem with the domain names changing and browser security and cross-site scripting something-or-other.

So I checked the domains. Sure enough, the home page was at “www.example.com.” From there, when you log in, it goes to “app.example.com.” Bingo! The domain was changing in the middle of my test. I experimented a little and discovered there was no way around it: the app was going to redirect to a different domain no matter what.

It turns out I’m not the first person to have this problem. Fortunately, Selenium has a strategy for addressing the issue: experimental browsers. I tried the chrome browser for testing on FireFox and it worked perfectly.

Lesson 4: Firefox Rocks!

At this point I could launch the app and log in, but now I had another problem. After the login page, all the things I needed to click, check, or otherwise manipulate were buried deep in convoluted HTML. I realized that figuring out how to address these things was going to be non-trivial.

The most basic strategy for discovering the locator for an element is to view the HTML source. You can view the source for the whole page, but Firefox has a great feature that allows you to see just the source for just a selection. To use it: highlight a selection on the page, then right-click. One of the available menu options is View Selection Source. Choose it, and you get a window with just the relevant HTML.

However, if you’re dealing with something complex, viewing the source isn’t enough. You really need to look at the Document Object Model (DOM). The best way I know to do that is with the DOM Viewer included with the Web Developer plugin by Chris Pederick.

Web Developer also includes a feature that lets you see all the attributes for a given element. From the Information menu, choose “Display Element Information.” Now you can get the attributes for any element just by clicking on it. I love that feature.

Finally, the XPath Checker plugin by Brian Slesinsky is a very helpful tool for figuring out how to address those pesky non-ID’d elements. More on xpath in the next section.

Lesson 5: xpath Is Now My Good Friend

One of the hallmarks of legacy web apps is the annoying lack of IDs on important elements. Of course, web apps aren’t the first place where a lack of IDs is problematical. I recall struggling with Windows apps that lacked field IDs back in the 1990s.

The good news is that this is a much more tractable problem in web apps than in Windows apps. Xpath to the rescue!

Let’s take just one example. I needed to verify the text associated with a particular image. The image served as a kind of custom bullet in a bulleted list, so there were several identical ones. The text itself did not appear in the DOM immediately next to the graphic. Rather, its parent was a peer to the parent element of the image. (Got that straight? Yeah, me either. Seriously, this one took me a while.) In brief, the HTML around this thing looked kinda like this:

<div>
    <span>
        <img src='/path/to/images/check.gif'>
    <span>
    <font>
        item 1
    </font>
</div>
<div>
    <span>
        <img src='/path/to/images/check.gif'>
    <span>
    <font>
        item 2
    </font>
</div>

Mind you, the HTML didn’t look that clean. There was a lot of other random stuff in there, and every tag had a gazillion attributes, and there were hard-coded styles everywhere. But I digress. And I’m probably whining. I’ll stop that now.

So it turns out the only way I could grab the text associated with the second bullet in the list was with the following Selenium command:

get_text("xpath=(//img[contains(@src,'check.gif')])[2]/../../font")

Let’s all say it as a group: “EWWWWW!”

But let’s also appreciate that doing such a thing is actually possible. Selenium has a wide range of locator styles, and even allows you to add your own locator strategies. (I haven’t needed to do that yet, so I’m not quite sure how to use the feature, but I noticed from the documentation that it’s there.)

(For more on using xpaths with Selenium, I found the Help with XPath article on the openqa.org site useful. It’s hard not to like an article with headings like “How the $%^@$ do I locate an element?”)

Mind you, the world might be a better place if we couldn’t write such code. Every time I figure out how to automate tests against an untestable application, I feel a twinge of guilt. By automating tests against an untestable interface, I become an enabler of more untestable interfaces. For the sake of improved collaboration and more testable applications, perhaps it’s better if those of us who automate tests avoid resorting to using our xpath superpowers except in the service of wrapping legacy apps with tests so they can be refactored for testability.

But once again, I digress.

The point I really want to make is that SeleniumRC gives us the power to automate tests even against icky hard-to-test legacy apps, and to do it with real programming languages (pick your favorite: C#, VB.Net, Perl, PHP, Ruby, Java). And that means we can write maintainable automated tests using good programming practices. And *that* means we can automate regression tests for faster feedback. And ultimately, *that* means we can make changes to the legacy app to improve testability and maintainability.

So rock on, SeleniumRC. And huge thanks to everyone who’s ever worked on SeleniumRC or Selenium. Also, huge thanks to the Selenium community as a whole. When I went looking for answers to my questions I found numerous blog posts and forum messages with tips and tricks. This post would not have been possible without such a community that’s so open about sharing knowledge.

* Here I mean “Legacy” as Michael Feathers defines it: code that lacks automated unit tests. Web developers, please take note: if you write good unit tests for your web app, including JSUnit tests for the JavaScript bits, the application will be MUCH more testable and the QA people will stop whining at you so much. return to footnote reference

Usually the people asking the question, at least in this particular way, are test automation specialists. They have spent much of their career becoming experts in a particular commercial tool. They know how to make their commercial tool of choice jump through hoops, sing, and make toast on command.

Then they find themselves in a newly Agile context struggling to use the same old tool to support a whole new way of working. They’re puzzled when people like me tell them that there are better alternatives for Agile teams.

So if you are trying to make a traditional, heavyweight, record-and-playback test automation solution work in an Agile context, or if you are trying to help those other people understand why their efforts are almost certainly doomed to fail, this post is for you.

Why Traditional, Record-and-Playback, Heavyweight, Commercial Test Automation Solutions Are Not Agile

Three key reasons:

1. The test-last workflow encouraged by such tools is all wrong for Agile teams.
2. The unmaintainable scripts created with such tools become an impediment to change.
3. Such specialized tools create a need for Test Automation Specialists and thus foster silos.

Let’s look at each of these concerns in turn, then look at how Agile-friendly tools address them.

Test-Last Automation

Traditional, heavyweight, record-and-playback tools force teams to wait until after the software is done - or at least the interface is done - before automation can begin. After all, it’s hard to record scripts against an interface that doesn’t exist yet. So the usual workflow for automating tests with a traditional test automation tool looks something like this:

1. Test analysts design and document the tests
2. Test executors execute the tests and report the bugs
3. Developers fix the bugs
4. Test executors re-execute the tests and verify the fixes (repeating as needed)
5. …time passes…
6. Test automation specialists automate the regression tests using the test documents as specifications

Looking at the workflow this way, it’s surprising to me that this particular test automation strategy ever works, even in traditional environments with long release cycles and strict change management practices. By the time we get around to automating the tests, the software is done and ready to ship. So those tests are not going to uncover much information that we don’t already know.

Sure, automated regression tests are theoretically handy for the next release. But usually the changes made for the next release break those automated tests (see concern #2, maintainability, coming up next). The result for most contexts: high cost, limited benefit. In short, such a workflow is a recipe for failure on any project, not just for Agile teams. The teams that have made this workflow work well in their context have had to work very, very hard at it.

However, this workflow is particularly bad in an Agile context where it results in an intolerably high level of waste and too much feedback latency.

• Waste: the same information is duplicated in both the manual and automated regression tests. Actually, it’s duplicated elsewhere too. But for now, let’s just focus on the duplication in the manual and automated tests.
• Feedback Latency: the bulk of the testing in this workflow is manual, and that means it takes days or weeks to discover the effect of a given change. If we’re working in 4 week sprints, waiting 3 - 4 weeks for regression test results just does not work.

Agile teams need the fast feedback that automated system/acceptance tests can provide. Further, test-last tools cannot support Acceptance Test Driven Development (ATDD). Agile teams need tools that support starting the test automation effort immediately, using a test-first approach.

Unmaintainable Piles of Spaghetti Scripts

Automated scripts created with record-and-playback tools usually contain a messy combination of at least three different kinds of information:

• Expectations about the behavior of the software under test given a set of conditions.
• Implementation-specific details about the interface.
• Code to drive the application to the desired state for testing.

So a typical script will have statements to click buttons identified by hard-coded button ids followed by statements that verify the resulting window title followed by statements to verify the calculated value in a field identified by another hard-coded id, like so:

field("item_1").enter_value("12345")
button("lookup_item_1").click
field("price_1").verify_value("$7.00")
field("qty_1").enter_value("6")
button("total_next").click
active_window.verify_title("Checkout")
field("purchase_total").verify_value("$42.00")


The essence of the test was to verify that ordering 6 items at $7 each results in a shopping cart total of $42. But because the script has a mixture of expectations and UI-specific details, we end up with a whole bunch of extraneous implementation details obfuscating the real test.

(If you’re nodding along, thinking to yourself, “Yup, looks like our test scripts,” then you have my sympathies. My deep, deep sympathies. Good, maintainable, automated test scripts do not look like that.)

All that extraneous stuff doesn’t just obscure the essence of the test. It also makes such scripts hard to maintain. Every time a button id changes, or the workflow changes, say with a “Shipping Options” screen inserted before the Checkout screen, the script has to be updated. But that value $42.00? That only changes if the underlying business rules change, say during the “Buy 5, get a 6th free!” sale week.

Of course, there are teams that have poured resources, time, and effort into creating maintainable tests using traditional test automation tools. They use data-driven test strategies to pull the test data into files or databases. They create reusable libraries of functions for common action sequences like logging in. They create an abstract layer (a GUI map) between the GUI elements and the tests. They use good programming practices, have coding standards in place, and know about refactoring techniques to keep code DRY. I know about these approaches. I’ve done them all.

But I had to fight the tools the whole way. The traditional heavyweight test automation tools are optimized for record-and-playback, not for writing maintainable test code. One of the early commercial tools I used even made it impossible to create a separate reusable library of functions: you had to put any general-use functions into a library file that shipped with the tool (making tool upgrades a nightmare). That’s just EVIL.

Agile teams need tools that separate the essence of the test from the implementation details. Such a separation is a hallmark of good design and increases maintainability. Agile teams also need tools that support and encourage good programming practices for the code portion of the test automation. And that means they need to write the test automation code using real, general use languages, with real IDEs, not vendor script languages in hamstrung IDEs.

Silos of Test Automation Specialists

Traditional QA departments working in a traditional waterfall/phased context, and automating tests, usually have a dedicated team of test automation specialists. This traditional structure addresses several forces:

1. Many “black-box” testers don’t code, don’t want to code, and don’t have the necessary technical skills to do effective test automation. Yes, they can click the “Record” button in the tool. But most teams I talk to these days have figured out that having non-technical testers record their actions is not a viable test automation strategy.
2. The license fees for traditional record-and-playback test automation tools are insanely expensive. Most organizations simply do not have the budget to buy licenses for everyone. Thus only the anointed few are allowed to use the tools.
3. Many developers view the specialized QA tools with disdain. They want to write code in real programming languages, not in some wacky vendorscript language using a hamstrung IDE.

Thus, the role of the Test Automation Specialist was born. These specialists usually work in relative isolation. They don’t do day-to-day testing, and they don’t have their hands in the production code. They have limited interactions with the testers and developers. Their job is to turn manual tests into automated tests.

That isolation means that if the production code isn’t testable, these specialists have to find a workaround because testability enhancements are usually low on the priority list for the developers. I’ve been one of these specialists, and I’ve fought untestable code to get automated tests in place. It’s frustrating, but oddly addictive. When I managed to automate tests against an untestable interface, I felt like I’d slain Grendel, Grendel’s mother, all the Grendel cousins, and the horse they rode in on. I felt like a superhero.

But Agile teams increase their effectiveness and efficiency by breaking down silos, not by creating test automation superheroes. That means the test automation effort becomes a collaboration. Business stakeholders, analysts, and black box testers contribute tests expressed in an automatable form (e.g. a Fit table) while the programmers write the code to hook the tests up to the implementation.

Since the programmers write the code to hook the tests to the implementation while implementing the user stories, they naturally end up writing more testable code. They’re not going to spend 3 days trying to find a workaround to address a field that doesn’t have a unique ID when they could spend 5 minutes adding the unique ID. Collaborating means that automating tests becomes a routine part of implementing code instead of an exercise in slaying Grendels. Less fun for test automation superheroes, but much more sensible for teams that actually want to get stuff done.

So that means Agile teams need tools that foster collaboration rather than tools that encourage a whole separate silo of specialists.

Characteristics of Effective Agile Test Automation Tools

Reviewing the problems with traditional test automation tools, we find that Agile teams need test automation tools/frameworks that:

• Support starting the test automation effort immediately, using a test-first approach.
• Separate the essence of the test from the implementation details.
• Support and encourage good programming practices for the code portion of the test automation.
• Support writing test automation code using real languages, with real IDEs.
• Foster collaboration.

Fit, Fitnesse, and related tools (see the list at the end of the post for more) do just that.

Testers or business stakeholders express expectations about the business-facing, externally visible behavior in a table using keywords or a Domain Specific Language (DSL). Programmers encapsulate all the implementation details, the button-pushing or API-calling bits, in a library or fixture.

So our Shopping Cart example from above might be expressed like this:

Choose item by sku 12345
Item price should be $7.00
Set quantity to 6
Shopping cart total should be $42.00


See, no button IDs. No field IDs. Nothing except the essence of the test.

And by writing our test in that kind of stripped-down-to-the-essence way makes it no longer just a test. As Brian Marick would point out, it’s an example of how the software should behave in a particular situation. It’s something we can articulate, discuss, and explore while we’re still figuring out the requirements. The team as a whole can collaborate on creating many such examples as part of the effort to gain a shared understanding of the real requirements for a given user story.

Expressing tests this way makes them automatable, not automated. Automating the test happens later, when the user story is implemented. That’s when the programmers write the code to hook the test up to the implementation, and that’s when the test becomes an executable specification.

Before it is automated, that same artifact can serve as a manual test script. However, unlike the traditional test automation workflow where manual tests are translated into automated tests, here there is no wasteful translation of one artifact into another. Instead, the one artifact is leveraged for multiple purposes.

For that matter, because we’re omitting implementation-specific details from the test, the test can be re-used if the system were ported to a completely different technology. There is nothing specific to a Windows or Web-based interface in the test. The test would be equally valid for a green screen, a Web services interface, a command line interface, or even a punch-card interface. Leverage. It’s all about the leverage.

Traditional Tools Solve Traditional Problems in Traditional Contexts. Agile Is Not Traditional.

Traditional, heavyweight, record-and-playback tools address the challenges faced by teams operating in a traditional context with specialists and silos. They address the challenge of having non-programmers automate tests by having record-and-playback features, a simplified editing environment, and a simplified programming language.

But Agile teams don’t need tools optimized for non-programmers. Agile teams need tools to solve an entirely different set of challenges related to collaborating, communicating, reducing waste, and increasing the speed of feedback. And that’s the bottom line: Traditional test automation tools don’t work for an Agile context because they solve traditional problems, and those are different from the challenges facing Agile teams.

…

Related Links

A bunch of us are discussing the next generation of functional testing tools for Agile teams on the AA-FTT Yahoo! group. It’s a moderated list and membership is required. However, I’m one of the moderators, so I can say with some authority that we’re an open community. We welcome anyone with a personal interest in the next generation of functional tools for Agile teams. We’re also building lists of resources. In the Links section of the AA-FTT Yahoo! group, you’ll find a list of Agile-related test automation tools and frameworks. And the discussion archives are interesting.

Brian Marick wrote a lovely essay on An Alternative to Business-Facing TDD.

I discussed some of the ideas in this article in previous blog posts, most notably:

• Functional Test Tools: the Next Generation (part 1 of 2)
• Functional Test Tools: the Next Generation (part 2 of 2)
• What Problem Would Next-Generation Functional Testing Tools Solve?
• And While I’m at It, I Want a Platypus Too!

A small sampling of Agile-friendly tools and frameworks:

• Ward Cunningham’s original Fit has inspired a whole bunch of related tools/frameworks/libraries including Fitnesse, ZiBreve, Green Pepper, and StoryTestIQ.
• Concordion takes a slightly different approach to creating executable specifications where the test hooks are embedded in attributes in HTML, so the specification is in natural language rather than a table.
• SeleniumRC and Watir tests are expressed in Ruby; Ruby makes good DSLs.

Are you the author or vendor of a tool that you think should be listed here? Drop a note in the comments with a link. Please note however that comment moderation is turned on, and I will only approve the comment if I am convinced that the tool addresses the concerns of Agile teams doing functional/system/acceptance test automation.

“Want some coffee?” my host asked.

“No thanks, I’m fully caffeinated for now.” I replied.

“Even if you don’t want a coffee, you should come see how it’s prepared,” my host grinned at me expectantly.

“Um, OK.” I relented. I dutifully followed him through twisting and turning corridors until we arrived at the coffee counter.

There were three men at the counter. I watched as they made coffee for all the people in line in front of us. It was quite a production.

The first man reached up to a shelf for a ceramic cup and placed it on the counter. The cups were bigger than a demitasse, but much smaller than my typical ginormous supersized vat-o-coffee mug.

The second man then flipped the valve on the coffee maker allowing a dark, rich liquid — thicker than espresso — to flow into a small metal pitcher. He then upended the metal pitcher into the cup.

The third man had the best job of all. He was the real showman. This was what my host wanted me to see. He began by dipping a saucepan into a huge steaming pot of milk sunk into the counter. He then lifted it high and poured it back in a long stream. Dip. Pour. Dip. Pour. As he poured the milk back into the pot, it frothed.

When the third man judged the milk sufficiently foamy, he poured it into the prepared cup, careful to let just the milk out. No foam. Not yet. Once the level in the cup reached an invisible boundary, he poured the rest of the liquid back into the steaming pot, leaving just the foam in the saucepan. Then he gently tilted the saucepan over the cup, allowing just the right amount of foam to cover the center of the near-caramel-colored coffee mixture. The result was a foamy white top surrounded by a ring of darker froth around the edges. As he placed the dipper back across the pot of milk, the second man ceremoniously handed the patron their coffee mug, handle first.

Several people were in line, so I got to see the performance several times. Each time the team of three executed with precision. The resulting cups of coffee were identical in appearance: same volume in the cup, same amount of foam on top, same colors.

The milk pourer also seemed to have a quality control role. If he decided the color wasn’t quite dark enough, he would signal - almost imperceptibly - to the metal pitcher guy, who would then add a little more of the thick, dark coffee.

Of course, after such a performance, I had to have one of my own. Receiving my mug reverently, I took a sip. The drink was nothing like the coffee I usually get at home. The froth tickled a little. The drink tasted sweet and rich and just a little exotic. It was a bit like a latte, but richer and sweeter. I was hooked.

Visiting the coffee counter became a ritual for me. I drank many, many of those coffees while in India.

One day toward the end of my visit, the person in front of me requested unsweetened milk in his coffee. When it was my turn and I stepped up to the counter, the first man confirmed what I wanted: “Normal coffee, madam?” he asked.

“Yes,” I replied, smiling. “Normal coffee please.” Even if the beverage I was enjoying was not normal coffee to me, it was normal here. Sweet. Rich. Foamy. Normal. Once again, normal is in the eye of the beholder.

WordPress database error: [Can't create/write to file '/tmp/mysqltmp/#sql_11b6_0.MYI' (Errcode: 2)]


“Oh, joy,” I thought to myself as I began investigating. I knew that the problem could not have been caused by anything I did. First, my most recent update to the site was back on April 2, that was just a content posting, and I was 100% certain my site had been alive and well much more recently than that. Second, I don’t even have shell access, so I couldn’t make a file in /tmp/… disappear if I tried. That meant the problem had to be with my ISP, and was probably outside my control.

After poking around for a little while in the vain hope that if I messed with stuff the tmp file would spontaneously regenerate, I sent a missive off to tech support. The auto-responder helpfully told me that I could expect to wait 24 - 48 hours for a response. Even if this is just a blog, that’s too much downtime. So I decided a phone call was in order.

A baffled tech support rep suggested I uninstall and reinstall WordPress. When I pushed him on how this problem happened to begin with, the tech support rep backpedalled a little and said he’d escalate the issue. I now had an incident number for tracking purposes, and a promise that “someone would get back to me.”

At this point I evaluated my options.

I’ve been growing annoyed with godaddy.com anyway. Every time I log into my admin account, they try to upsell me. They don’t let me have shell access. They don’t support Ruby on Rails well - or not as well as I would like. I find their admin UI clunky. These are little annoyances; they’re not enough to push me to change hosts. But now that I had a down site and no ETA for a fix, I decided that changing hosts was actually the path of least resistance.

So my adventures began. I bought 3 months of hosting on A2hosting.com, a host I’d been contemplating for RoR hosting anyway. I managed to get a backup of my content from my godaddy.com site. (Note to self: I need to revisit my blog backup strategy. I happened to get lucky today - the catastrophic error that made my site unusable mercifully did not prevent me from exporting the data. But that was sheer luck. My previous backup was a couple months old. But I digress.)

I then put up a “pardon the mess” notification on both the old and new sites and changed my DNS entries to point to the DNS servers at the new location on A2. And I installed WordPress on the new site.

I had to wait for the DNS changes to propagate before I could do more because every attempt to log into the WordPress admin interface in the new location redirected me to the old site. Fortunately, I had to go to an appointment anyway. By the time I got back I could see the new IP address when I pinged testobsessed.com.

I then began migrating the content. That meant restoring the content from the SQL backup through a MySQL admin interface, upgrading the tables, panicking when it looked like the schema wasn’t going to upgrade cleanly, and finally breathing a sigh of relief when I saw the old content in the new site.

But wait! There was still more to do. I uploaded the theme files and other resources (including images and pdfs and such). I fixed the permalink settings. I tested. I fixed glitches. I tested again. I swore.

And finally I got the old site up on a new host.

This is, quite frankly, NOT how I expected to spend my day. I was supposed to be doing paperwork - invoices, expense reports, contracts, that sort of thing. This was a geekier, and possibly more exciting way to spend my day, but it was not at all what I had planned. And in my haste it is certainly possible that I missed some detail.

I think everything is working now. Please let me know if you find anything strange - missing content, broken links, whatever. (No, I don’t have a full set of automated regression tests to cover every link I publish. I’ve considered it, but the cost of maintaining such a suite of tests seems a little high for a non-revenue-generating blog.)

And that escalated trouble ticket filed with godaddy? Still not handled. I’m not particularly surprised. My suspicion is that the problem I encountered is somehow related to the fact that I had not migrated my WordPress to their whiz-bang new Hosting Connection application management console thingy. That first tech support rep was probably right: I could have solved my problem by uninstalling and re-installing WordPress.

But I don’t think that would have been any easier than moving hosts - the only saved step would have been the DNS change. And I’m happier having migrated. And - as an extra bonus - the site even seems to be responding a little zippier.

This little digression will probably boost another back-burner project to the foreground: I’ve been meaning to change the way I manage my main company site. Other alert readers have pointed out to me that my calendar at qualitytree.com is more than a year out of date. It’s embarrassing. But I haven’t fixed it because I publish that site using an obscure little Windows-based CMS. Since I’ve gone all-Mac, updating qualitytree.com means I have to boot up my old Windows laptop, and I hate doing that.

I’m going to see how things go with the new host for a while before migrating everything. But it looks like I’ll be migrating my other website sooner rather than later.

And now I’d better start on that paperwork I meant to take care of today.

J. is a QA manager in an organization that’s adopting Scrum. She’s
frustrated, and understandably so. From her point of view, she’s
being squeezed in all directions. The developers are producing
releasable code every month. But for her team to run a regression
test cycle - mostly manual - takes 6 weeks. That’s too long. Just
one test cycle exceeds the sprint length by 2 weeks. J. feels
tremendous pressure to reduce the time it takes to test the software.
Yet at the same time, she feels like she’s not getting any support to
do the one thing she can see that will help reduce the test cycle
time: automate the regression tests.

I’ve had some visibility into J.’s situation for some time now. J.’s
team has been trying - and failing - to automate the regression suite
for the last two years. They aren’t making any headway because as
soon as they get one script working, another one breaks. The
automation is brittle, error-prone, and incredibly expensive to create
and maintain. That’s in part because they’ve been using a cumbersome
commercial tool that doesn’t support creating maintainable tests.
It’s also because the user interface was not designed with test
automation in mind. Many UI elements don’t have IDs, and the ones
that do use automatically generated IDs that change with each build.
In short, the combination of the tool and the software under test
equals a test automation nightmare. It’s no wonder J.’s team is not
making headway.

Yet J. persists. Doing more of the same kind of test automation
that’s already failing doesn’t make much sense to me, but she
disagrees. “We just need more time!” she says.

The problem is that J. is still thinking in terms of silos. She
thinks all testing tasks must be done by QA people using specialized
QA tools. It simply would not occur to her to suggest that
development help automate tests. Nor does she suggest that developers
and testers collaborate on making the UI more testable. Instead, she
says, “QA can’t go that fast. Slow down.”

J. doesn’t want to acknowledge that test automation created by a
siloed QA team working in isolation to reverse-engineer existing
software and automate tests against an untestable UI using proprietary
tools accessible only to a few select team members is guaranteed to be
incredibly expensive both to create and to maintain, and also
ridiculously fragile. In short, her approach just isn’t going
to work.

Unfortunately, J.’s story is likely to have an unhappy ending - at
least for J. and her team. Her strategy of trying to get development
to slow down, and telling management that they can’t release monthly,
is backfiring. The development team is already bypassing QA for small
changes and getting good results. But J. is undeterred. My past
observations tell me that no matter what the reaction of the people
around her, she will keep doing the same thing and expect different
results.

But maybe, just maybe, by telling J.’s story here, I can help someone,
somewhere.

So allow me to repeat the moral of this story:

When QA works in isolation, creating automated tests after the
software is theoretically “done,” using proprietary tools that are
available only to a select few team members, the results will be a
fragile, unmaintainable mess.

For test automation to work well, it must be created in collaboration
with the whole team and the resulting test automation code must be
treated as code. That means it should be versioned with the source
code, executed with each and every build, and created and maintained
as part of the overall development cycle rather than as an afterthought.

And when a Test/QA group insists on keeping within their silo when the
rest of the organization adopts Agile practices, they will end up
bypassed and irrelevant as the rest of the organization finds ways to
move forward without their help.

Some have been really real, like the time I found a 4 inch long dragonfly (seriously, the live flying kind) hanging from the projection screen in a conference room in a hotel in Florida. I was supposed to be teaching a software testing class, projecting slides onto that screen, so I found the presence of the dragonfly more than a little distracting. (The hotel wanted to spray it dead with insecticide, but I couldn’t let them do that. The class participants worked together to capture the beast in a water pitcher and released it, unharmed, outside.)

Other bugs have been of the software variety. Like when I needed to get a receipt from a travel website and it consistently gave me an error message instead. (Frustrating, that.)

This last trip yielded more bug sightings than most.

First, there were strange limitations that inhibited my travels. Trying to arrange plane tickets revealed a constraint imposed by the flight reservation system. It seems that you can only fly through any given city 3 times on one ticket. So the fact that I needed to transfer planes 4 times, and wanted to transfer in Frankfurt each time, kept resulting in fatal error messages when I tried to book the tickets online. The helpful airline desk agent explained the limitation to me (though she couldn’t explain why it exists) and was able to book my ticket - with a stop in Warsaw instead of Frankfurt for one of the transfers.

Similarly, I was unable to book my train tickets myself online. Each time I did, I received a helpful error message: “CH.enrolled java.lang.Except.” Someone at my host site was able to book tickets for me, and I still don’t know why I couldn’t book the tickets myself.

(For the curious: I tried with different browsers and still had the same problem - not surprising, since it was a java error and thus most likely server related. I had guessed that the problem was with my data - perhaps my passport number was in an unexpected format, being a US passport rather than one issued by an EU country. But since my host was able to book the tickets, that wasn’t it either. Perhaps it had to do with my machine being in the US? Or with my keyboard being a US keyboard? Or with the way I entered the data? Who knows.)

Then, once I reached the Frankfurt airport (on transfer #1 for those who are counting), I found that one of the internet kiosks had crashed.

Based on the Blue Screen of Death displaying on the monitor, my guess is that there’s Windows under the covers of these kiosks.

And once I got to Poland, I discovered that they have seriously big bugs there. Bigger even than in Florida. The bottom picture is one I found hanging off the side of a building in Wroclaw.

After this trip I could only conclude that I am truly a bug magnet.

“That’s not a duck,” he replied.

“Yes it is,” I said.

“No, it’s not,” he replied.

It occurred to me that maybe ducks in Finland look different. “OK,” I said. “If that’s not a duck, what does a duck look like?”

“White,” he said. “And bigger. That’s not a duck, it’s a sorsa,” he gave me the Finnish name.

Certainly, the duck we were looking at was not white. It had the colorful markings of a male mallard. “Right,” I said. “There are white ducks and colored ducks. But they’re both ducks. A sorsa is a kind of duck, right?”

“No,” he insisted. “Ducks are white and have bigger beaks than this.”

I began to sense that we might be talking about two entirely different things. “Big white water bird,” I mused. “with a bigger beak. Do you mean a goose?”

As we sorted out the difference between ducks and geese and sorsa, it dawned on me that this English-Finnish discussion offered a good example of the difficulty in sorting out a common language on software projects.

I started remembering all the conversations in which someone used an overloaded term like “server.” Or “test.” Or “done.”

For the record, it turns out that we were, indeed, looking at a sorsa. And it was, indeed, what I would call a Mallard duck in English.

But just contemplating how much effort it took to establish this simple basis of understanding about something we could see and hear, I am astonished that business stakeholders and technologists on software projects - something truly intangible - are ever able to achieve any kind of shared understanding about what we’re building and how it should function.