Coffee and Fury

This should be self-explanatory to anybody who’s been in operations for any length of time. But for you folks who are new to this whole “production” thing, for those of you into the devops concepts but not so much the devops reality, here’s a clue.

Most organizations are not prepared for upgrades. Especially not when third party tools or software are involved. Oh, you think you are. And maybe you’ve even had the discipline to test this on some other system. Good for you! You’re one step closer to living the dream.

The awful reality, though, is this: your third party vendor’s support organization is not staffed for weekends, nights, and holidays. Oh, they’ll talk a good game about how they are a true 24*7 operation, blah, blah, blah. Every software support organization I’ve ever been on has one, maybe two people on call during the weekend. They’re typically surly, underpaid, and were awake for sixteen hours straight last night hand holding some other customer through some upgrade gone horribly wrong. It will typically take the maximum service level objective for them to get back to you, and deity help you if you haven’t paid for off-hours support for your software — you’ve never heard doors slam so hard as when you wake someone up in the middle of their night without having paid for the service.

The biggest reason you nerds get into this situation is because you don’t have redundant infrastructure, you don’t test failover routinely, and you think off-hours upgrades will minimize impact on your user base. If you want the A-Team to be there if an upgrade goes badly, you’ll need that redundant infrastructure so your users can continue to run while you and your folks faff about with the upgrade, you’ll need to do it during your vendor’s normal work hours. You’ll also need to make sure your support contract is paid-up long beforehand. It’s also a good idea to let your vendor know by submitting a ticket letting them know what you’re planning, when you’re planning, how long the window is, what you’ve done to prepare, and what kind of backups you’ve taken.

Oh, you haven’t taken any backups?

Another thing you’d better know is exactly what the vendor’s service levels, compatibility matrix, scope of support, and how to escalate your support case.

On the support side, there’s nothing more annoying and hateful than a customer who is upgrading their super-sized cluster, who’ve taken no backups, who are doing this in the middle of a holiday week, and who, in the face of a fatal error during the upgrade, bulls right ahead and continues the process, then complains via their CEO to your CEO about the horrible service they’ve received. Sure, the software may be crap (support knows this better than anyone anywhere), but blaming the vendor for your fuck-ups will earn you mad disrespect with the tech support team who have to clean up your mess in the face of inquiries every five minutes by every VP in the company asking them why they haven’t helped Important Customer Asshole.

Be smart. Stop when things get out of hand, ask for help, be patient, test beforehand, always take backups and perform test restores (even more important than backups), assume you’ll have to rebuild the installation from scratch anyway (bonus if you needn’t), be patient, and don’t push the panic button unless your vendor is truly not meeting minimum standards. Have an alternate production system in place to take over should things go pear shaped.

I knew something was horribly broken when I read this email to a friend’s support site: “This is my third email today asking for help from you. I want to delete my account so I can make a new account, and reuse the same email address I always do.”

First, why is it so hard for a user to delete their account? I looked at my friend’s site, and there was a big red “Delete my account” button on the account profile page. I found a knowledge base article on “How to delete my account”. I saw a tool tip when I hovered over the button. I found links and suggestions when I used the site search tool. That part looked OK. So I asked my friend what the problem was, and he told me that account deletion was a manual operation, and had been for, well, forever. In fact, they had a backlog of thousands of account deletion requests, some dating back more than six months (it’s a popular site, I reckon). For some reason, the developers and product managers and executives decided years ago that no one would ever delete their accounts, and even if they did, they’d keep that customer’s data in the back end databases — forever. While the user could “request” their account be deleted, and it (mostly) looked like their request was accepted, in reality, that request generated a ticket, was routed to a queue, and one part time person spent about an hour a day working on as many of them as possible. That person had to verify the account, look it up in a database, run a couple raw SQL queries they tailored to the specific account, checked the database for consistency, and then committed all the transactions. The production instance would pick up the changes in 4-24 hours, depending on circumstances. The average account deletion, once this part-timer got to it, took almost 10 minutes. Until this part-timer got to the request, the user name was marked as “to be deleted”, was unusable, and all the data retained. And, since the username and email fields were constrained to be unique (there could be only one), the username seemed available, but it wasn’t; the email address seemed like it could be reused, but it couldn’t. This was why the user wrote three times in one day, and there was a multi-month backlog of account deletions…stupid decisions by people long ago who have left the company and the code is maintained that way because reasons.

Now, the user could be smart, and use a throwaway email address to solve their own problem — that’d take minutes, actual minutes. The company could even tell users about the non-delete delete. But no, rather than own up to the non-delete delete and the impact on users, they’d rather not say they don’t actually delete user accounts. Instead, they chose not to tell their users, frustrate the hell out of them, require/engineer a system that requires manual intervention and customization for every user delete, and then wonder to themselves why there’s a six month backlog of account deletions.

Yeah, that’s going to scale.

Before you contact tech support with that gnarly problem involving a simple problem that is easily solved by searching the vendor’s knowledge base, my best advice is this:

READ.

THE.

FUCKING.

MANUAL.

I know. I know, little one. Reading is so very, very hard, what with the vast resources of the interwebs a single click away, and oh, the distractions of the ‘gram, and hey, did you hear what Celebrity X did? How could any software vendor or support person dare to suggest such a painful, heinous thing?

Grow the fuck up.

Show some semblance of trying. You’ll reap the rewards if you ever do contact tech support.

Lots of customers have done it. You didn’t get the answer you wanted from your vendor’s support organization. So you ask again, in a slightly different way. You still don’t get the answer you wanted — oh, they answered you, and said no, just like they did the first time, and maybe tried to sell some consulting hours, but you don’t want to pay that if you don’t have to (you don’t have that kind of signature authority, anyway), so you go to your vendor’s sales rep and tell them how outraged you are the product doesn’t work they way you think it should. Your sales rep texts their VP, all a-twitter the customer won’t renew, and the VP thunders both at support and development how horrible they are for not bowing and scraping to Important Customer A, and immediately calls the CEO of Important Customer Co and swears up and down how he’ll make certain they get their needs met by next Tuesday. Everyone at Important Customer Co (who, by the way, was spending a couple hundred dollars a year with you, and promising, always promising to up their purchase any day now) snickers behind their hands at how gullible VP person was, sales person was, and how they’re going to continue to sabotage the support rep over there. And Big-Co Software is now delayed in the next release, because the sales VP unwittingly committed a six month development job to be complete by next Tuesday.

Lather, rinse, repeat. I call it the a/b/c/d approach, or more colloquially, the mom/dad/aunt/uncle or parents/aunts/uncles/adults response to “No.”

Once I’ve seen a customer do it at my job, I’ll set the bozo bit on that customer contact without hesitation. Here’s how it usually runs

Someone using your software or systems is having a problem, and they want something from their software or systems vendor. They’ll send in a support case asking how to make your product do Thing X. Thing X turns out to be an extra cost option, or a new feature, or a script, or normal system administration, or just about anything that isn’t covered by the normal support agreement. Thomasina, a senior support person, knows this isn’t covered, and gives the customers their options to obtain Thing X, which typically are: a) file an enhancement request; b) pay some money for some consulting; c) wait for a future release; d) adapt the script in an application note, using the documented product API; or e) suck it up and deal, since the product doesn’t do that and the customer doesn’t pay us for that service.

The customer never responds, and leaves the case open and running. The customer files a second case, hoping the next support rep will fall for it. They make the same request. This time, Aleksandr gets the case; he’s fairly new to tech support and very new to your company. Aleksandr, in hopes of impressing the customer, and not understanding the support agreement with the customer, writes a custom script and delivers it to the customer, beaming triumphantly at how clever and helpful he is.

The customer got something free from their vendor just by asking the same question twice. They’re definitely going to continue asking for things, over and over, because you’ve trained them it works. Your company lost revenue giving something away free, and paid for the privilege. The customer is laughing all the way to the bank, and your entire organization probably doesn’t even realize it lost money.

I like to call this approach the parent 1/parent 2/aunt/uncle syndrome. Because childhood behavior works all too often when you’re grown up. Hopefully, you’ve all experienced children asking one parent if they can do something they know they shouldn’t be doing, let’s say they ask their mom. Dad says no. So, they go to the other parent and ask them; that parent says no. They will continue going through the adults in their lives until someone says yes. They will continue to do this, running craftier and more subtle campaigns, honing their skills, so long as this tactic continues to work. (Hopefully) Eventually the adults will get together, discover this con game, and put a stop to little Janie’s maneuvering once and for all by communicating amongst themselves. This con will work, though, if the adults don’t care or can’t communicate or want to be nice to little Janie for some reason or any of a thousand other reasons. And that’s why tech support teams see this sort of behavior all the time. The larger and less communicative a team is, the more frequently customers will get away with it.

Another favorite a/b/c/d tactic run by customers is to threaten to pull their account if they don’t get their way. Let’s say every time they ask for Thing X from normal channels, they are given the same answer, an answer the customer doesn’t like. This is just a slightly higher stakes version of asking the question to a succession of adults, with the added frisson created now that the customer has conflated the issue with money. Sales will definitely get involved now, since their compensation is on the line, and their inclination will be to promise the customer anything to ensure they (sales) make their numbers and reap that sweet, sweet commission at renewal time — they’ll show the customer just how powerful they are. Except now that customer has been trained to threaten revenue when they don’t get their way. In this scenario, the sales rep is the uncle, and the VP is the aunt.

In tech support life, customers will often get away with this for some time, until they establish a pattern of behavior and get a reputation. Such people need to be dealt with firmly; “No” must always mean “no”, from everybody. Make sure that the entire account team is aware of this bad behavior, including appropriate VPs — if you don’t, shame on you, because the customer will soon learn that all they have to do is email or text VP Soft-Heart and they’ll get their way, gloating at you every time they pull it off. Escalations or lateral traverses through your company must be dealt with calmly, rationally, and the first answer from someone outside customer experience should always be “Let me check on that and get back to you.” People who give other answers need to be checked before they give away the company, the fools. That person should immediately follow up with an inquiry to the rest of the organization to find out the backstory. And then it’s time to go back to the customer and say, “Amr in our customer experience team tells me your request is considered a feature request. I’ve also spoken to our development and product management teams, and they’d be delighted to speak with you in more depth regarding this. I’d like for them to give you a call next Tuesday, would 10AM your local time be good?” In other words, you’ve just told the customer no, just like Amr did, and started shoving them down the path that Amr provided, with prejudice. You did not promise them anything, you did not go on a power trip, you did not go off without the backstory, and you certainly made yourself more trusted by your customer experience team. If the customer repeats the behavior after that, or the creep-back of the behavior once the boss looks the other way, every further instance must also be dealt with firmly.

If the customer continues this bad behavior, and everyone on your side knows, then teach those assholes a lesson, and deliver updates 1 minute before they’re due, work to rule only, never give them anything more than they paid for, and explain it all quite directly if they whine in your general direction.

Of course, if you work at a major brand or tech support team with hundreds or thousands of reps, or work for a company that is losing money by the hands-full, chances are high this customer will continue to get their way. In cases like that, where you know they’re abusing your company largely with impunity, do the tech support equivalent of spitting in their food: wait until the last moment to give them an update — every time — then give them subtly incorrect advice, commands that have unknown-to-them consequences that are expensive, time consuming, and/or sub-optimal, but won’t break things or tag you as not-a-team-player nor incompetent. Use passive aggression to save your sanity and punish the assholes who won’t take no for an answer. Create a honey pot for them and string them along just long enough that they will eventually give up with you and move on to try to sucker someone else.

If it’s bad enough, fire the customer. They’re just going to continue being a money pit, and you and your entire company are better off getting rid of them now.

And if you peers and superiors can’t even deal, it’s time to move on. If your company wants to let customers get away with bad behavior because reasons, they’re likely too desperate, corrupt, or clueless to survive the long term.

Why am I on coffee? I’m in tech support, I’ve been on call for 12 of the last 14 days, and I’ve gotten a max of 4h of uninterrupted sleep for that duration. And about four pages a day, all for production outages at customer sites.

For all of you who wonder why your tech support person is surly? Sleep deprivation is a job requirement for lots of people in the operations and tech support professions. It’s probably also the reason there are very few long term professionals in support.

In those wondrous days of yore, I recall a stint at a startup where I was on call 24x7x365, and received an average of 300 hundred pages a day. Go back and read that slowly; I am not exaggerating. Between the stress, sleep deprivation, bad diet, lots of stimulants, lots of sleep enhancers, and insane responsibility, I was a very unhappy, very emotional, and a barely rational person going from crisis to crisis. And each crisis had better be resolved in less than twenty minutes, because a new one was just about to hit. While that situation was extreme, support professionals are running on scaled-down versions of that everywhere, all the time.

And everyone expects everyone in support to be the face of the company — bright, cheerful, helpful, knowledgable, courteous — you know, the face of the company. Imagine what that sort of dysphoria between person and presentation causes. It’s a wonder the phrase hasn’t ended up as “going full tech support” instead of “going full postal”. I suspect it’s because tech support staff are generally on the helpful end of the spectrum, and are more likely to self-harm than other-harm.