The best part of being an engineer at OkCupid is that — even though I spend most days hunched over the lifeless glow of C++ code — I sometimes read other people’s private messages. I don’t do this to improve my own online-dating game (not to say I couldn’t use the help). Instead, part of my job is to read messages that have been marked offensive and decide whether their authors should be kicked off the site. I do get a kick out of how cringeworthy some of our users’ rhetoric can be, but what’s more interesting are the well-meaning messages that only accidentally offend.
It’s part of my job, but sometimes it feels NSFW. A few weeks ago, I logged in and started reading a flagged conversation between Hank and Sarah (whose names and details have been altered to protect their privacy). In his profile, Hank described himself as a productivity hacker who liked to write, and who was looking to get into kink as a dom. Sarah said she was a chef at a French restaurant, and identified herself as a sub. She wrote that her account was a “kink profile.” (In this case, that meant she was very explicit about her sexual desires in her profile. For privacy, her picture only showed her torso.)
Hank messaged her first. He pointed out their high compatibility (according to the OkCupid matching algorithm), and raved that he’d never found someone so perfect on the site. Sarah said she was flattered. Hank sent Sarah long, gushing messages. Sarah replied with courteous one-liners. I read through two weeks of their lopsided back-and-forth: passionate compliments, polite excuses; excited flirtations, curt dismissals.
Finally, Sarah decided to put an end to Hank’s delusions by letting him know she didn’t return his affections. It was a gentle rejection, perhaps too subtle for Hank. He kept writing, begging to meet with her. Then he switched up his technique and sent her a story he’d authored himself. The gist of it is as follows:
A father drives his young daughter — around 18, but maybe younger — to the mall, parks them outside of Bloomingdales, and has sex with her in the passenger seat of their minivan. He calls the daughter Sarah.
Sarah said that she flagged the message because it made her so disgusted she wanted to quit online dating altogether. I was tasked with determining whether Hank should be banned.
Every social-media company, from Twitter to Tumblr to Craigslist, struggles with moderation. It’s a balancing act between freedom of expression and safe zones. Even Reddit — home to some of the internet’s most unsavory communities — began threatening “timeouts” and “permanent bans” for its trolls after they began attacking its CEO. Not to mention the upsurge in internet harassment since the elections. But at a dating site like OkCupid, moderation can be especially complicated, since risk-taking is a part of flirtation itself and what’s repulsive to one person might be a turn-on to someone else.
For example, when I look through the messages that fill up our moderation queue, some are so obviously crass that, to me, it seems like we should be able to have a computer remove them automatically. But my co-worker, whose time is dedicated to working on our moderation algorithms, assures me that the data speak less clearly. Had he allowed his program to remove “offensive” messages without human intervention, we would have quashed this budding romance:
drewcon: Wanna suck?
ugagirl: Yup
drewcon: When?
Ugagirl: Where u r
I want OkCupid to allow for the type of risquéness that — when properly applied — can be a turn-on. And what’s more, I don’t want our own idiosyncratic biases as a company to color what we allow on the site. We’re not morality police. But at the same time, I don’t want a minority of vocal, obscene users to make OkCupid feel like an internet cesspool. So where do we draw the line between risqué and obscene? Between aggressive flirting and abuse?
I made Hank’s case to OkCupid’s team of moderators, who each look through hundreds of flagged messages a day.
“Ban him, definitely ban,” one said. The rest unanimously agreed.
“He knew she didn’t want that story, and he sent it to her anyway,” one said. “It was purely selfish.”
“He didn’t escalate properly,” another added. “He didn’t make any attempt to see if she was amenable.”
“If you need a black-and-white reason,” the head of customer support told me, “we have a policy of banning any user who references an illegal act on the site. Kiddie porn is cut and dry.”
But that got me wondering how our policies were codified in the first place. I’m an OkCupid user and I’ve received some profane messages. The users I think should be unquestionably banned are the ones that flat-out try to hurt me. Like this one I got last week:
“Do you know about astronomy?” I didn’t reply. The next morning he wrote, “I’m surprised that an ugly sl*ut like you acts the way you do. Seriously bi*tch, look in the mirror. Dumb cu*nt.”
I was surprised anyone could have such a poor understanding of asterisks.
This is the same type of abuse that Instagram tried to eliminate earlier this summer, when it released a feature allowing users to create personal “blacklists” of words never to be allowed in their pictures’ comments sections. But explicit verbal attacks are the easiest kind of violations for us to tag and remove. That’s something we can do with a computer. Many more infractions that we intuitively and unambiguously believe should be grounds for ban on OkCupid are not as easily captured by a single rule.
Consider: Last month, a woman we’ll call Penny asked 15 men for drinks. Except she invited all of them to meet her on the same day at the same time at the same bar, and the only company they found when they got there was each other. She wasn’t breaking any specific term of service, but the moderators unanimously decided to ban her. As in law, the case itself set a precedent.
Or even trickier: A user’s wife wrote to OkCupid requesting we disable a “fake” account that was “posing” as her husband. Since using someone else’s photo is against site policy — the woman’s husband said the account wasn’t his — we banned it, choosing not to mention that all of the account’s network traffic was coming from their home.
It would be impossible for OkCupid to deal with its complaints in broad strokes, and those examples illustrate why the company “reserves the right to determine, at its sole discretion, what constitutes harassment or mischief, and where that has occurred.” But even so, it’s not always easy to determine what’s “mischief.” I think of all of the eclectic messages I’ve received on the site: solicitations for weird sexual favors, the opportunity to be spoiled as someone’s sugar baby, an invitation to join a world-traveling polyamorous hippie tribe. I’ve even been offered a job as a CTO and co-founder. These are not nefarious messages, and as an OkCupid user, they’re half the reason I use the site. I love meeting strangers I’d never meet in real life with wildly different jobs, tastes, and yes, vastly different standards of acceptable behavior. But it is exactly this variance in what’s acceptable that creates the gray zone of moderation.
But perhaps this is just the cost of protecting diversity in romance. To me, one of online dating’s greatest innovations is that it allows people to disclose their potentially polarizing preferences before a date ever takes place. That includes things like kink, non-monogamy, or supporting Donald Trump. As social convention stands, I can’t walk into a bar and coyly ask a cute stranger if he’d enjoy being slapped hard in the face during sex. But on OkCupid, that’s essentially what happens. So I’m existentially fulfilled by my work when I see people politely using OkCupid to express their relationship needs as a trigger warning to would-be dates. At its best, OkCupid lets daters be themselves — and find people like themselves.
But from a moderation perspective, this bluntness often backfires. A monogamous user will accidentally stumble across the profile of a polyamorous one and flag the account with a comment like, “Disgusting. User just wanted a hookup.” And users who mention kink in their profiles are disproportionately likely to be flagged. So the art of moderation is finding the distinction between self-expression and self-imposition. Which brings us back to Hank.
Unanimously, the moderation team had called for a ban on Hank, but I was unconvinced. I feared we were biased by our own gut repulsion to his sexual preferences. They thought he was selfish; I thought he was clueless. In any case, I didn’t like the idea of trying to guess what he was thinking, since mind reading is what got him into trouble in the first place.
In the end, our head of spam made an argument that convinced me: “Do we really want to expose Hank to other users on the site?” Regardless of intent, it seemed quite possible Hank would send a similarly unwelcome message in the future, and the cost of that was too high; he was now a liability. Banning Hank was a practical call, not a moral one.
As much as the engineer in me wants a cut-and-dry rule set for banning users from OkCupid — ideally, one that can be taught to and enforced by a CPU — I’m glad human moderators always make the last call. It allows us to evolve our policies as we learn the nuance of an entirely new type of dating. And while I’m sure I’m biased by my own idiosyncrasies, my goal is to optimize the site so that the most number of people go on the most enjoyable dates. What that means, for now, is that well-meaning humans with awful understandings of interpersonal interactions should be kicked off. Our users can be as alt and weird and kinky on their profiles as they like, but the moment they start messaging other users, they’re subject to all of the social-skill constraints that exist offline.
In this sense, OkCupid is like a bar with a bouncer who asks, Is this guy bothering you? Sometimes, I’m the woman at the bar. Other times, I’m the bouncer.
