January 31, 2015

#12 Back End Trouble

by Reply All

Background show artwork for Reply All

The entire internet decides to look at one famous butt at the same time. One man has to ensure that the website hosting Kardashian butt pictures doesn't crash. The sheer terror and joy of solving that problem. (Plus, a new Yes/Yes/No.) 

The Facts:
Our theme song and episode music is by 

Breakmaster Cylinder. Our ad music is by Build Buildings

Further Reading:
Paul Ford wrote a great explainer of what exactly Greg did -- it's smart and entertaining and you should 

go read it! Dril on twitter. 


PJ VOGT: Every so often, everybody on the internet decides that they want to look at one website. And that they all want to do it at the same time. Sometimes the site crashes; sometimes it survives. If it survives, it's usually because of a guy like Greg Knauss. Greg keeps the servers running for a bunch of magazine websites. One of them is Paper Magazine. It's a little art magazine, with a little over 100,000 subscribers. Recently, the higher-ups came to him with some news.

GREG KNAUSS: We have something that we think is going to generate 100 million page views. And we want to make sure, you know, will our server handle that. And, the answer was, "No! No, it will not." (laughing)

PJ: The something was four photos, and I guarantee you that by this point you've seen these photos. So has everybody.

MALE NEWSCASTER: And I don't think I need to tell you what's been the biggest story on the internet for the past 24 hours. No butts about it.

FEMALE NEWSCASTER: Kim Kardashian's impossibly tiny waist, accentuated by her most celebrated asset.

MALE NEWSCASTER: Yes, that's Kim Kardashian, baring her backside.

MALE NEWSCASTER: A photo of Kim Kardashian's boo-tay!

PJ: This is Reply All, a show about the internet. And I'm PJ Vogt. The Kardashian photos were published in November. If you have a very strange media diet, in which you basically only listen to this podcast and somehow you've not seen these photos, I'll describe them for you.

It's a series of nude and semi-nude fashion photos of Kim Kardashian. In the most famous one, she's pouring a champagne bottle over her shoulder, into a champagne glass, that's propped on her butt. These photos would not have been so ubiquitous had it not been for two people. One of them, obviously, is Kim Kardashian. The other's Greg Knauss. I'm going to go out on a limb and say that Greg's story might be the less familiar of these two.

When Paper Magazine first told Greg that they needed his help to prepare for a big traffic spike, they didn't actually tell him what was going to bring all of this traffic to the site. He had very little idea what he was in for. The only thing that they made absolutely clear to him was his deadline.

GREG: “Start now. Because the launch date is Tuesday next week.”

PJ: And so how'd you feel when they said that?

GREG: Uh ... uh. Scared shitless. It's not a lot of time in the grand scheme of things.

PJ: In theory, Greg could've just said no to this. But instead he said yes. Immediately.

GREG: It was Pavlovian.

PJ: What do you mean?

GREG: Somebody rang a bell, and I started salivating because I expected that I was going to get to design an infrastructure.

PJ: So is it joyous? Like is it like, is it like, yes! I get to solve this problem?

GREG: Joy might be too strong a word. But there is a, a, profound and nerdy satisfaction in saying, this is a big problem. I know how to solve it. I can do it in a very short amount of time, by myself. And then -- 15 minutes later you think, 'Oh, shit, what have I done?'

PJ: So Greg spent his first few days like, getting advice from friends, downloading open-source software, adding extra servers, and setting up a backup plan so that even if one of those servers failed, the Paper Magazine site could still temporarily live on Amazon's cloud computers. Those, by the way, are all the technical details we're gonna give you. Sorry, web backend development fetishists, you'll just have to get your sick thrills from some other podcast.

If you do really want to get into the technical stuff of the story, you should go read Paul Ford's Medium piece about it. It's fantastic. It's where we learned about this. Anyway, by Sunday, two days out, Greg's built his contraption, but he's got a new problem, which is: How do you test this thing?

GREG: There are a lot of tools that allow this. Not specifically to look at butts (laughing) but -- or perhaps there are. I'm not familiar with every subculture on the internet. There are a whole bunch of tools that allow you to simulate traffic against a website. The one I happen to like is called "Bees With Machine Guns." What it is is a piece of software that automatically brings up a whole bunch of tiny little machines on the Amazon network. And then -- they aim their tiny little machine guns at your server, and ... act ... like ... 10, or 50, or 100, or 250,000 people hitting the website at the same time.

PJ: So Greg tells the bees to attack, to see if the server could survive the load he expected.

GREG: I'd done some math, and calculated out what I thought the load should be. Bees wasn't able to reach it. And I finally figured out that, the requests I was making was maxing out the maximum network capacity of the bees machine. That they simply couldn't send data fast enough to simulate the load that was going to happen.

PJ: In other words, hundreds of thousands of virtual bees with virtual machine guns were no match against the very real power of Kim Kardashian's naked butt.

Greg decided he'd have to just trust his gut. That the preparations that he made were good enough. Greg told me that this week alternated being one of the most stressful workweeks in his life, and one of the most professionally satisfying. Because, it might be strange to imagine this from the outside, but Greg says there's a lot of art to being a system administrator.

GREG: There are times when you're in flow, and it's ... like being a writer in flow, or like being an artist in flow, or like being anybody that can create something. There, it just comes, and it feels awesome. It's ... a feeling that, I mean I've been chasing for years. And then ... when it goes away, it's incredibly frustrating. It's heartbreaking, even. Where ... I'm doomed. I've committed to something that I'm incapable of doing, that isn't going to work. And so ... like, the tweet that I wrote, at 6:45 a.m. on Thursday, which is, "sometimes a fun thing to do is have anxiety slowly tear you apart from the inside."

PJ: Uhhhh (sighs).

GREG: (laughing) That's how I felt then.

PJ: By Monday, the new infrastructure that Greg had built was actually up. And Greg was feeling cautiously optimistic. So much so, that on Tuesday, the very day that the pictures were supposed to post, he was actually coaching his kid's Little League team.

GREG: At the end of practice, I checked my phone and it says, "The images are now live." And I go, "Oh my god, I didn't realize it was so late," so I go tearing home. I live just a couple of minutes from the field, and I bring up, on my computer, the load ... of each of the machines that's handling the front-end requests. And it doesn't seem like there's a lot of traffic. Certainly more than normal, there's a spike, but it doesn't seem like anything like we had talked about. And what it turns out is, Kim had posted the original two pictures, which were the champagne shot, and the rear-end shot, to Instagram. And so, that's where people were going. She has, like 25 million followers on Instagram.

PJ: When you see the Instagram pictures on her site, you're just feeling relieved, you're feeling like, you know we set something up for a hoard of traffic, and instead we're getting a mini-hoard and we're fine.

GREG: I ... relief mixed with disappointment. You know, it's like you build a racecar and you wanna see it race.

PJ: Right.

GREG: And, you're glad that it doesn't burst in flame. But it's also kind of just ticking down the track. You know, you're doing 35 miles an hour, and okay, great.

PJ: Then -- Paper published the second batch of photos. They came a day later. And they took Greg by surprise.

GREG: The following two pictures were frontal nudes, which Instagram does not allow, per their terms of service. And so then, Kim points to the Paper Magazine site. And ... the ... the traffic spike is like the front of half dome. It just goes STRAIGHT up. And straight up forever. And ... it was just ... it was like, nothing -- I mean, I've been doing this for a long time, and it was like nothing I've ever seen before.

PJ: The site held. The pictures were everywhere.

GREG: My 15-year-old son is sports obsessed, and that's pretty much his entire, other than, than school, and dog-walking, he is ... is ... entirely consumed by sports. And he comes up to me like, the day afterwards, and he said "hey dad, your Kim Kardashian was on "The Crawl" on Sportscenter."

PJ: (laughing) It's like even invaded his Kardashian-less world.

GREG: Yeah, exactly. Hey, son! That was your dad. Aren't you proud?

PJ: (laughing) I mean did you feel kind of proud?

GREG: I -- I was proud that it worked.

PJ: Yeah.

GREG: That is pure technical nerd, I put together a machine that held together. There are people who do this everyday. Thousands of system administrators out there. Tens of thousands. Who handle sites like Facebook and Gmail, and, and they're, they're chuckling at this, because they see this kind of stuff on a slow day ... you know, they're the guys that keep the whole internet running. And ... sometimes it gets attention. And when it gets attention, it's like, wow! That's a lot going on. Everything is like that.

PJ: For one week, you had to be in charge of one of those invisible, complicated, necessary infrastructures. Did it change how you looked at -- did those systems seem more visible to you for awhile?

GREG: You know, I'm an old-time Twitter user. And I lived through the fail-whale era.

PJ: The fail-whale era was the time in Twitter's history where the service was just constantly crashing. When it didn't work, you'd get an error message that included a picture of a whale.

GREG: And, from the outside, it's annoying. Oh ... the small amusement I wanted to give myself at the moment is unavailable. But, from the inside? They invented new computer science to solve problems on a scale that had never been seen before. That's incredibly fascinating to me. It would bore the snot out of 99.9 percent of the population.

PJ: (laughs) You have to have pride in a job that if it works, nobody notices it. You only really get noticed if you mess something up and everything breaks.

GREG: Yeah, but that's -- I'm a nerd. I think that people that are drawn to these kind of jobs take the most satisfaction in the technical accomplishment of it working. If they were interested, in, fame, or if they were interested in large amounts of money, they would have very different personalities and pursued very different careers.

PJ: Occasionally, you can see nerds like Greg who have built something that have made them so rich, and so famous, that they can't avoid the spotlight anymore. Mark Zuckerberg comes to mind. Bill Gates, too. I mean you can't tell for sure, but when you watch them, it always seems like they don't want to be there. They'd rather be back in the basement, banging on the pipes with a wrench.

PJ: We're doing another installment of a segment that you might remember from a few weeks ago, called “Yes Yes No.” Here to explain that segment is my co-host, Alex Goldman.

ALEX GOLDMAN: As you may know, our boss, Alex Blumberg, is a very busy man. So busy, in fact, that he rarely has time to even look at the internet, much less try and understand the dark, dark corridors that me and PJ spend our days walking down. So, “Yes Yes No” is a segment where we try and explain to him one internet phenomenon, and hope, by the end, he gets it.

ALEX BLUMBERG: Alright, so, so I have this tweet. It's from this person named Libby Watson. It says "ugh i agree with almost all of this but why does he always have to say stuff like 'Weird Twitter will hoot'" And then there's a link to a website, fredrikdeboer.com, and, the ... article linked to has the headline, I don't know what to do you guys, and then it seems to be an article mocking a writer named Jonathan Chait. Do you know what this, do you understand what this tweet means?

PJ: Yes.

ALEX BLUMBERG: Alex Goldman, do you understand what this tweet means?


PJ: Uh, Alex Blumberg, I hate to ask, do you understand what this tweet means?


PJ: (laughing) Alright, let's get into it.

ALEX BLUMBERG: This one is different from the last one. The last one had a lot more words that I didn't understand, but this one has all words that understand, but they're used in a way that seems to have a significance that I don't quite get. It's referencing some sort of conversation online I'm definitely not a part of... somebody, named Fredrik deBoer, is always saying things that other people are noticing and annoyed by. So that, I don't know what that is. But then, there's this thing, Weird Twitter? And Weird Twitter is capitalized. And so I know what Twitter is, and I know what Weird is ... but I don't know what the proper noun Weird Twitter is.

ALEX GOLDMAN: Alright, well let me just set, set this up. So ... last week, Jonathan Chait, who writes for New York Magazine, wrote an article called "Not a Very P.C. Thing to Say" which was about language policing, political correctness, and the headline is, "How the Language Police Are Perverting Liberalism."

PJ: And the reaction on the internet was really bad, because they were like, oh, the old white man, says, that ... things are too hard out there for old white men.

ALEX BLUMBERG: So we have this Jonathan Chait article that seems cranky, and a little bit anachronistic, that a lot of people disagree with on the internet.


PJ: Right, yes.

ALEX BLUMBERG: Alright, I'm with you so far. But who's this guy Fredrik deBoer?

ALEX GOLDMAN: He's done, he's written opinion pieces for The New York Times. He writes a lot of articles that are about this sort of thing.

PJ: He writes this thing saying like, I get that everyone disagreed with Jonathan Chait saying this. He's' like, allow me to make the point that I think someone else could've made if he hadn't made it so poorly.

ALEX GOLDMAN: And, within that article, he said "I know writing these words, exactly how this will go down. I know Weird Twitter will hoot, and the same pack of self-absorbed media liberals will herp-de-derp about it."

ALEX BLUMBERG: There's a lot to unpack about it.

ALEX GOLDMAN: So what's Weird Twitter?

PJ: Alex ... ?

ALEX GOLDMAN: Weird Twitter is a, a loose collective that are constantly ridiculing everyone and everything. It was named by someone else, and they constantly are making fun of the fact that they're called Weird Twitter.

PJ: They don't use their real names, and they mostly make jokes. Like a lot of times they'll intentionally misspell things, or they'll be all lowercase. If you just looked at one of their tweets, it looks like it's written by a person who it's like, it's their first day on a computer, maybe. Oh! The other thing that's worth saying, is that these people have like, 20, 40, 60,000 thousand followers. And the followers are like journalists or like, comedians with big followings, within this one, stupid, narrow, media landscape, these people are actually pretty influential.

ALEX BLUMBERG: Like, so, do you, do you join Weird Twitter?

ALEX GOLDMAN: No you can't join Weird Twitter. Anyone who would try to join Weird Twitter, would, would not be had as a member.

ALEX GOLDMAN: And nobody in Weird Twitter would admit that they are a member of Weird Twitter.

ALEX BLUMBERG: Let me ask you this. Are you guys in Weird Twitter?

PJ: No. Alex wants to be.

ALEX GOLDMAN: I think I'm aspirationally in Weird Twitter, but they'd never have me (laughing).

ALEX BLUMBERG: And how do you know?

ALEX GOLDMAN: Um, you know based on I think the people who are following you. Like they would never admit to being Weird Twitter. They don't even like the name. They make fun of it constantly.

ALEX BLUMBERG: Alright. So ... how many people are in Weird Twitter?

PJ: Buzzfeed did like an oral history of Weird Twitter, and uh, they profiled maybe 10-20 people, and they had like a graph, and literally like, John Herrman who did it was like, I know I'm going to get yelled at, I know everyone's going to have a problem with this, and even, he did, his interviews with them were just like filled with crazy tolling.

ALEX BLUMBERG: God, Weird Twitter is like the internet teamsters.

ALEX GOLDMAN: Yeah! Oh that's really good analogy.

PJ: Yeah. Weird Twitter is like the internet teamsters. And one of the things I like about Weird Twitter is sometimes they will like, find a way to like, embarrass, somebody who is like, a public figure, and a jerk, in public. And this, this is not a Weird Twitter person, but here's a person playing Weird Twitter tactics.

Richard Dawkins tweeted: "'Most' doesn't mean 'All'. 'Most Muslims are NOT terrorists' is fully compatible with 'Most terrorists are Muslims.' Obvious but . . ." So I don't know, he's make some like racist academic point. And somebody said, "BOFA has supplied this evidence, have you read it?" And he said, "Sorry, who or what is BOFA?" and they said, "BOFA DESE NUTS YA DICK" and there was like a video of Wario. Like, engaging somebody who's being an asshole in a really highbrow way, in a very like, sort of smartly juvenile way, that's like a Weird Twitter thing.

ALEX BLUMBERG: I'm not sure I would put it that way.

PJ: How would you put it?

ALEX BLUMBERG: I would say ... they literally said, "BOFA has supplied the information about that," and when he wrote back saying "What's BOFA?" they said, "BOFA DESE NUTS, YA DICK."

PJ: (laughing)

ALEX BLUMBERG: And then you said that was smart.

PJ: (laughing) I think it's so smart! I think it's soooo smart.

ALEX BLUMBERG: (laughing) That's my only point.

ALEX GOLDMAN: I have to say that I pretty much only follow the Weird Twitter crew. They're all that, they to me are the meat, of, Twitter. They're the value of Twitter to me ... the absurdist component of Twitter, of Weird Twitter is super funny to me. There's like, there's a Twitter user named Kat Beltane, and he tweets, here's a couple of his tweets -- this to me is the purest distillation of Twitter as an art form. Are you ready?

PJ: Oh boy.


ALEX GOLDMAN: “hilarious prank: use millions of years of evolution to swell an ape's brain until it's capable of feeling disappointed in itself. “

ALEX BLUMBERG: (laughing)

PJ: I love you just laughing at the silence of a bombed joke.

ALEX GOLDMAN: Um, the quintessential Weird Twitter is dril, PJ Vogt hates dril.

PJ: No, I like dril now, I changed my mind. One of the dril ones I really like: "another day volunteering at the betsy ross museum. everyone keeps asking me if they can fuck the flag. buddy, they wont even let me fuck it."

ALEX BLUMBERG: (laughing) Oh. That's good. The weird thing about Weird Twitter that I'm learning is like, I mean for all it's like, irony, and like, these are just sort of silly jokes.

PJ: They have this ability to hone in one what is -- like, I feel like, a fear that I have in my heart is that I'm secretly ridiculous for reasons I don't understand, and one day someone's going to figure it out and point it out, and that's all anyone's ever going to see. Sort of like, what happened in high school or whatever, junior high. And they're really good at that.

ALEX BLUMBERG: And that's their, that's their power.

PJ: I would say so. Right Alex?


ALEX BLUMBERG: Is to find the one part of you that you haven't fully realized is full of shit.

PJ: And just stay on it forever.

ALEX BLUMBERG: That is a valuable service.

PJ: (laughing)

ALEX BLUMBERG: No, but I think I've arrived at understanding.

PJ: Really?

ALEX BLUMBERG: Yeah. Okay, so the tweet, we started with this tweet. "ugh i agree with almost all of this but why does he always have to say stuff like 'Weird Twitter will hoot'" and it linked to a fredrikdeboer.com article. And what I now know ... Fredrik deBoer is an online commentator on some kind. He was taken issue with another article that had been written online by Jonathan Chait. And in the article, in the Fredrik deBoer article, he used the line "Weird Twitter will hoot" and that brought us deep into an explanation of what exactly is Weird Twitter.

ALEX GOLDMAN: And what is Weird Twitter?

ALEX BLUMBERG: And Weird Twitter seems to be a collection of 20-40 mostly real people, many of whom use, uh... online pseudonyms, who are united by a passion for absurdist and slightly juvenile humor. It's sort of like, if Holden Caulfield had watched a bunch of Mystery Science Theater, and then ...

PJ: Yeah!

ALEX GOLDMAN: (laughing) Wow.

ALEX BLUMBERG: And then opened a fake Twitter account. That's Weird Twitter.

PJ: Yeah! Holy crap. Yeah.

ALEX GOLDMAN: We're at “Yes Yes Yes” in a big way.

PJ: Reply All is me, PJ Vogt, with Alex Goldman. We're also Chris Neary, Sruthi Pinnamaneni, and Alex Blumberg. Matt Lieber is like a memory of summer in a winter that seems to go on forever. Our show was mixed by Gimlet's technical director, The Reverend John Delore. Our theme song is by the Mysterious Breakmaster Cylinder, and our ad music is from Build Buildings. 

We're online at replyall.limo. If you like the show, you can help us by reviewing us in the iTunes store or just by listening. Thanks for listening, and we'll see you next week.

PJ: So Alex. 


PJ: You made something else besides the podcast this week. 

ALEX GOLDMAN: I did indeed. As of Friday, January 30th, I am a father! I, my wife gave birth to a baby boy named Harvey, uh, at 7:24 in the morning. I fainted, and had to be taken to the ER. But I'm alright, baby's alright, it's really really exciting. 

PJ: So you're going to take a month to, uh, raise your child, and then you'll be back making the podcast, and you'll probably be dropping in over the month, right? 

ALEX GOLDMAN: That's right. I have a microphone in my basement, so every once in awhile I'll drop in to say hey, but PJ's going to be steering the ship for awhile. Which is terrifying. 

PJ: Yeah. It should be. Well if you want, you could, you could run the podcast and I could raise your child? 

ALEX GOLDMAN: That's slightly more terrifying. 

PJ: It is crazy. I dunno. I'm proud of you.