Case Study: Metrics (feedback button data)

I once conducted a project to analyse the user feedback that my then-employer received from the feedback button on help pages. We had all been using the feedback to determine which topics needed attention and to understand doc usage. But I quickly saw that there were some pretty serious problems with the data.

Cautionary Tales: Metrics (performance measurement)

(I'm a big fan of IT people. Also, nobody likes measuring things more than me. But interpretation is everything. Or rather, a little knowledge is a dangerous thing.)

I once worked in a company where an IT team of three people provided technical support to about 1,000 employees. One of the IT guys was great and the other two were really bad. By bad, I mean that when an employee asked them to fix a problem on their PC, these two guys typically couldn't fix the problem and often also created new problems.

Then one day the good IT guy got fired. We learned later that the company had instituted metrics to measure performance of the IT department, and had determined that this fellow was too slow: metrics showed that it took him two or three times longer, on average, to close a case.

What everyone in the company knew (except, apparently, IT management), was that the good IT guy took all the complex and difficult problems, while the other two guys dealt with the sort of mundane issues they could handle. Difficult problems take longer.

I might have just passed this off as incredible stupidity, but the next company I went to also had three people in the IT department. They were extremely overworked (causing delays that affected productivity all over the company), but one day one of them got laid off. We learned later that the company had instituted metrics to measure performance of the IT department, and the metrics showed that there wasn't enough work for three. The metrics system only covered issues that were logged through a web site, but employees mostly just called IT without logging an issue.

When design and docs fail

Two Elections Canada workers were fired this week because they didn't understand the difference between encryption and compression. They were told to secure a USB flash drive that held information about voters. Instead of encrypting the data they compressed it, thinking compression was the security feature. Then the drives were stolen. (link)

I have documented encryption and compression, not for end users but for developers. When I read the article I had an immediate understanding (or at least guess) of why the Elections Canada workers were confused. The failure might have progressed something like this:

1. The internal process of encrypting and compressing data is similar (even though the end use is very different) so when the API was designed, encryption and compression were combined. For example, an API that I once documented used the encode method to both encrypt and compress data. Developers chose encryption and/or compression by setting parameters in the method.
   
2. The tech writer followed the API design and documented encryption and compression together. Yes, that's what I did. Even as I did it I realized it was wrong, and I think that's why it stuck in my head. Worse, I went on and on about all the fancy encryption options but threw in compression as a side note. (There just wasn't that much to say about compression: it was essentially a trivial add-on.).
   
3. The app developer followed the API design and the API documentation, and again implemented them in one feature in the UI, calling them Encryption and Compression.
   
4. The tech writer for the app followed the design of the UI.
   
5. The end user was faced with a USB drive that they knew had a security feature, and the UI had two side by side features, Encrypt and Compress. They guessed and chose the wrong one.

I checked out some secure USB drives, and I can see why users would be confused. The terms might be explained in the end user booklet, but if it's not clear in the UI then they probably didn't see it. Or perhaps they saw it but their eyes glazed over because of the lengthy descriptions of encryption algorithms. It would be much better for the app UI and end user docs to separate the two features and call them "Password protect your data" and "Compress data to free up file space." Important features should be completely clear in the UI, without having to resort to the docs. Way back in the API design, security features should have been separated from compression features. At the least, the developer docs should have handled them separately.

Case study: DITA topic architecture

This post is part of a series of posts that question some of the claims made about the benefits of DITA adoption.

(Note: DITA uses the word "topic" to refer to a reusable module of content. Out in the rest of the world the word topic tends to refer to an HTML page or a section of a PDF. This may be seen as an infuriating oversight, but I suspect it's actually deliberate. For a while I tried replacing DITA "topic" with "reusable module of content" but I have given up and now just use the one word to mean multiple things.)

I started with DITA several years ago when I got a job in a large doc team that had been using DITA for a while. I inherited a few deliverables and was appalled at the way the content was broken up into concept, task and code sample topics. We optimized for HTML output and there were too many brief HTML pages that users had to click through. Even for a simple idea that could have been covered in one paragraph, these docs would have three topics. For example, a description of how to stop the server would have a concept, task and sample topic, each appearing on separate HTML pages.

My audience was developers. I did quite a lot of usability testing with them and found they were furious about the documentation. They hated having to click through multiple tiny pages. They hated the minimalism and choppiness. They described the docs as unfriendly, officious, insulting and unhelpful.