Why I Hate Big Data

Full disclosure, I work on the Analytics team for a large Internet company in San Francisco.  Members of my team and the teams I work with have T-shirts or office signs that say things like “I <3 Big Data” and “I love it when you call me Big Data.”

biggie
Rolling. In. His. Grave.

This post is going to be … controversial… to some of them.

As you have no doubt gathered from the title, I’m not a big fan of what tends to be called “big data.”  For the uninitiated, big data refers to extremely large datasets (think hundreds of millions to billions of records, if not more) that reveal patterns, most frequently about how users are interacting with a product or metadata about users.  Examples might include click data from Facebook — essentially a list of every mouse click any user has performed on a Facebook page, including e.g. the location on the page and the link URL target and the time of day and the location of the user, among other things — or information about phone calls — the metadata the NSA is so interested in keeping tabs on, which might include things like phone numbers, locations, dates, times, durations of calls, etc.

snowden
Based all too loosely on a true story

Let me tell you right off the bat (250+ words in…) that I don’t hate big data because I’m afraid of being snooped on or I’m paranoid the gummymint or The Facebook is after me, or tracking me, or… anything me, really.  Honestly, they don’t give a flying duck about me.  No, I can break my hatred of big data down into three distinct reasons, none of which have anything to do with paranoia.

The first is that as someone who works with data a lot, I can tell you it’s messy.  This is true of “small” data provided by people paid to give you data.  You’ll get a CSV that should have 24 rows, and it has 23.  Maybe it has 25.  Maybe a row or a single cell is blank for some reason; maybe it’s ASCII-encoded and they have to report a name with an accented letter, or, God forbid, an emoji, so it just shows up as “?”.  This stuff is actually really complicated, and you usually won’t know you have a problem until suddenly you have one, which could happen at any time for any reason, no matter how stupid, esoteric, or obvious.

Now imagine you’re getting data not from a paid vendor but from users clicking on things.  Maybe they have a spotty Internet connection, so the click event doesn’t get logged.  Maybe you accidentally send the click event twice.  Maybe they clicked on the wrong thing.  Maybe the team who’s in charge of ingesting click data and mapping it back to users is at their offsite in Acapulco when their server goes down, or they changed the event code just before they left and didn’t tell anyone.  So you pull the data the next morning to find out how many links were clicked, and you see the number is way down — is that because of a new feature that was just pushed?  Is it a holiday in some large portion of the world so people aren’t using your website?  Is there a bug in the website?  Or is the bug in the mobile app, or the click event firing, or the click event ingestion, or in the query used to pull the click count?  Who can even say?

acapulco
Not me, man. I’m in Acapulco.

Related to this, typically from datasets this large, you can even find patterns that have nothing to do with what the dataset originally set out to record.  For instance, a dataset might simply tell you how long a user has your application open in a window.  From this information, you might be able to infer how “engaged” a user is with your website.  But this is a potentially terrible proxy; for instance, just now I got up and wandered into the kitchen for 5 minutes; I was not engaged with www.carscafmoo.com during that time, even though the window was open.  In this case, your most engaged user is the one who literally dies after loading your site and isn’t discovered for days.  So you get around this by developing other heuristics (scroll actions, click actions, page views, having the window open for a certain time window, etc.) that, if you’re really lucky, are based on a broad survey of user interactions, but are still messy, and probably aren’t actually based on anything you could remotely describe as “scientific”1.  Basically, without access to the user’s laptop camera to see what they’re actually doing, you’re screwed, and no matter what Zuckerberg does you can’t actually turn on the camera without turning on the indicator light on a modern Macbook unless you are able to physically reinstall the camera, so that’s pretty unlikely.

To sum up, you’re generally measuring proxies that correlate with or approximate whatever actual metric you’re searching for, and you’re constantly asking yourself, “How well are we answering this question?” The answer is almost always, “Not very well.”

The second reason I hate big data is because it is descriptive of general trends, but frequently applied to specific people.  Here’s an arbitrary example of a dataset that might shows the behaviors of two groups of hypothetical users:

rplot01
The colors, Duke! The colors!

You can see visually that group 1 tends to be higher than Group 2; indeed, group 1 has a mean of ~10.4 and Group 2’s mean is down at 8.7; unsurprisingly (by construction) the difference between the two groups is statistically significant2.  Let’s say Group 1 is users of my website with webbed fingers and Group 2 is normal people, and the value we’re recording is logins per month.  Here’s how this information would typically get reported (imagine this is part of an infographic; note the arbitrary heights of the bars):

screen-shot-2016-10-01-at-8-13-16-pm

 

hands
I object to your labeling me as a “normo”!

Except that as you can see, several of the points from Group 1 are actually pretty low — in fact, 21 of the 100 points in Group 1 are actually below the mean for Group 2.  What is actually true is that on average from our particular sampling of the dataset a user with webbed fingers will login 18% more often than a user without webbed fingers.  Any particular user will fall within a pretty large variance, so just because I have webbed hands doesn’t mean you should start targeting me as a high-login user; I may actually behave much more like a non-webbed-hand person.  That’s neither a compelling nor pithy narrative, so we distill it as much as we can in order to tell our particular story (with our particular spin — “Hey, webbies love our website!”); but from my point of view that’s not sound statistics or data science or big data analytics, it’s marketing.  Sure, that’s a role that needs to be fulfilled, but every time I see an infographic or a headline like the one above, I immediately wonder about about the actual shapes of the underlying distributions.  I would propose a WAG that 90+% of the “big data” roles at tech companies fall into support of marketing or sales and involve exactly this sort of broad-strokes “analysis.”

Of course, there are great insights to be gleaned from proper analysis of big data — in particular, recommendation models based on grouping users’ actions or preferences and then extrapolating to what other members of that group have liked or done tend to work pretty well (think Netflix recommendations), but even then there’s so much individual variance that it’s very difficult to present a great set of predictions for any individual user.

bad-recs
Or sometimes groups of users

So if big data works for generalizations but isn’t great at specific, individual insights, what is it good for?  This brings me to my third problem with big data — that it’s really a misnomer.  “Big data” datasets actually are potentially great for individual data, messiness aside.  But per our definition above, that’s not actually what big data is — it’s more than an enormous dataset, it’s about using that dataset to reveal patterns.  And if you’re looking at an individual, say, their cell phone records, you’re not really looking for patterns and thus not really performing “big data” analysis.  You’re just looking stuff up.  If you want me to look up the metadata around your phone calls between 8 AM on the 9th and 10 PM on the 10th, I can do that with just your data.  Hell, I can do that if the only data I have is your calls between those specific hours.  That’s not big data, that’s just data.

Now, if you’ve read all of this (and why would you), what you’ll hopefully see is that I don’t actually hate big data — I think data is messy and hard to work with, and I think that “big data” is a term that is frequently mis- or over-applied, or that it’s poorly defined.  The technological capability to store and process enormous amounts of information, and the resulting ability to glean both broad patterns and individual insights from that information, is a capability that can and does drive enormous value.  Even the Big Date Lite broad generalizations serve a real and necessary marketing and sales function, especially in a world of high-level (and frequently somewhat innumerate) decision making.

I just wish I didn’t know enough about the term to cringe every time I see it.

  1. With data this big, you can pretty easily tell yourself you’ll over-count some proxy actions for your metric and under-count others, and if you’re lucky they’ll offset.
  2. Welch’s 2-sample T-Test produces a P-value of 9.9e-9.

Millennial Madness

Apparently some people don’t realize that the “Millennial” generation encompasses pretty much everyone born from 1980-2000.

While I might shave off a few years on either side of that definition, one of my biggest pet peeves is when Millennials badmouth Millennials in the third person, because they don’t realize that they themselves are Millennials.  For example, I was at a career fair and getting lunch with some of the other recruiters — all of whom were under 30, including all but two under 25 — and this guy starts off the conversation with, “Man recruiting these Millennials, amiright? It’s like all they care about is what you can do for them.”  We gave him an inquisitive stare, and he responded with, “What? You remember how it was in our day, we were all ‘what can we do for you,’ y’know?”  So I asked him how old he was.  He was 28.  This was two years ago.  He was a Millennial.  He didn’t find that hilarious.  I did.

pot
Pot, meet Kettle.

Of course, being born in 19871, I am a Millennial, and that will not stop me from railing against my own kind.  We, as a generation, are really into ourselves2, and of the many criticisms that have been thrown at us, self-congratulation is probably one of the fairest.  We are the generation who invented participation trophies3, and the “like” button 4.

Like
By the way, leave a comment if you enjoyed this post!

And to be clear, I’m against all of it.  I vividly remember in fourth grade at our end of year awards ceremony, when a third of the class got something called the “Technology Award,” and I realized that it was the third of the class who didn’t get any other awards.  I was shocked. I was appalled.  I thought the recipients would want to know that they had sham awards5, but I wasn’t sure how to break it to them; my mom told me not to, so I didn’t.  And today I have friends!  Thanks, Mom!

Today, when I look around, where other people see “Generation Why?” or “The Me Generation,” I see the cumulative effects of over-recognition manifest in the Circle Jerk Generation6.  What I mean by this is the mutually reinforcing cycle of engagement with content on social media.  Those italicized words are indicative of the buzzwordification of (and associated substitution for) concepts that ought to contain actual meaning.  Where I might legitimately think that someone has done an excellent job or created tangible value, I would have to know the person in order to communicate this to them.  However, someone can produce content on social media that is highly (and measurably) engaging — even if it is of no value at all7.  In exchange for your congratulations and recognition at their grand feat, they offer their engagement, through likes and comments and shares, and if I like your post will you like mine, oooh just comment on this right here — yeah yeah now like that post like it so hard — ooh yeah your content is great let me like that right there for you yeah you like that?

OFace
Oh, oh content, oh content…

But if I think my generation is bad, man is the next one worse.  Did you know that teenagers don’t have sex anymore, they just like things on social media?  Did you know that while our generation had burn books, the mean girls of today are not liking your selfie on Instagram even if you like theirs8? Did you know that all of the problems of this generation are worse in the next one???

Which is why another, greater pet peeve of mine is when people from previous generations badmouth our generation, because remember how annoying it was when the generation before you did that to you?  There’s a (Gen-X) trainer at my gym, and he’s always ragging on Millennials — “You’re all working so hard today, and what are those Millennials out there doing?”  I shaved my beard in March, and he commented on how young I suddenly looked.  I told him I’m 28, and I swear to god he said that I “don’t act like a Millennial.”  Yeah, and you’re not high as a kite and wandering aimlessly through your cynical post-Ridgemont High life, what a goddamn miracle.  Sure, my generation lives disproportionately with its parents, but then, so did yours.  Remember when loud music and video games and recreational drug use were going to melt your brains and make you dysfunctional subhumans?  Thanks to the Internet, surviving ’80s references, and older cousins, I do!

But wait, there’s more!  Sure, the Boomers ragged on your generation really hard, but what about them?  Oh, that’s right — they were a bunch of juvenile delinquents who were being led astray by comic books and Elvis Presley, of all things.  Remember all of the crazy progressive things they did in the ’60s, like peaceful protests in favor of civil rights, women’s equality, and ending the wildly, incredibly pointless and wasteful war in Vietnam?  Remember when the Greatest Generation declared that they were a colossal disappointment for not doing their duty and maintaining the status quo?  How thoroughly selfish of them.  And don’t even get me started on jazz — what did the Lost Generation think that was doing to the Greatest?

flapper
Her hair! It’s TOO SHORT.

Yeah, so it turns out that pretty much everyone thinks that the generation after them is a total selfish wreck about to doom the world, but that’s not the case.  At least, so far — and I would wager our generation, and the generation thereafter, and the one after that, etc. etc. will turn out OK too, no matter how selfish and doomed they may appear to be right now.

You know why?  Because eventually they’ll have kids.  Everyone talks about what a total life-changing event having kids is9, how it makes you completely reorient your life around that of another human being in a way that, for example, simply being married does not.  And of course your kids are your legacy, so you’ll do anything to give them a leg up in this world.  For instance, you might tell them that they’re special, that they’re smart and wonderful and deserve praise — you might even give them the Technology Award or like their post on Instagram or give them a participation trophy so they feel good about themselves.

So while each successive generation will be fine in the end, they will start off as total self-absorbed screw-ups, largely thanks to their parents.  All I’m saying is, let’s just keep in mind that they’ll be fine while we judge them loudly for their mind-boggling iniquity in the meantime.

  1. Go Weevils!
  2. Both individually and collectively; for instance, right now you, who are probably a Millennial, are reading a post by a Millennial about Millennials.
  3. Spoiler alert — no we didn’t, our parents invented that for us, thanks Boomers
  4. JK Bret Taylor was born in ’79 — he’s a Gen-Xer
  5. That’s right, they had sham awards, but I got a real one.  I’m a winner.
  6.  “How crass!” you object, “I’m offended!”  Well isn’t that very Millennial of you.
  7. Hahahaha it’s like I lack ANY self-awareness
  8. I saw this article somewhere several months ago and cannot find it, and after enough googling of “Middle school girls Instagram” I realized I am on several watch lists and decided to stop trying before I end up in a cell with Jared from Subway.
  9. I, mercifully, have yet to have any… that I know of…

The Annotated Guide to the Official Eric Hall Bachelor Party Themed Playlist

This weekend marks the official celebration of Eric’s upcoming nuptials, and as best man (TM) it’s my official and sacred duty to plan the event.  Since I am incapable of planning an event without a themed playlist (R) (and incapable of doing any other event planning), I proudly present to you The Annotated Guide to The Official Eric Hall Bachelor Party Themed Playlist (R, TM).

Friendtimes
Well, someone said something stupid

High School

cats!
Coach Tribble can never find out we did this the night before a meet

Friendtimes

  • “Purple Stain” by the Red Hot Chili Peppers: On a trip to Cancun after freshman year, Eric introduced me to Californication; this is the song I think of when I think back to that trip and the adventures we had with Maria, Esteban, and a gaggle of 12-year-old girls.
  • “Smells like Teen Spirit” by Nirvana: Another great aspect of that trip was the amount of conspiracy theory-laden documentaries we watched about Kurt Cobain.
  • “By the Way” by the Red Hot Chili Peppers: This album was released right after that Cancun trip, which made it the soundtrack to all of sophomore year.  In college I would decide it was a “fine, but not great” album, which would lead to a heated argument and a well-timed and highly confusing out-of-the-blue text message from Tyler asking me if I remembered how great that album was.
  • “Walking on Sunshine” by Katrina & the Waves: Braves win!  Braves win!!
  • “American Girl” by Tom Petty and the Heartbreakers: There was a time in high school when the Tom Petty greatest hits album was all anyone would listen to; this is about the only song on that album I could ever stand in bulk.

Cross Country / Track

  • “My Band” by D12: The invention of the iPod allowed us to listen to this on the way back from pretty much every track meet and to sing along to it, much to everyone else’s annoyance.
  • “Gimme Some Lovin'” by the Spencer Davis Group: this song played every Friday at 5:00 on 96 Rock, which during cross country season would be after practice but before spaghetti dinners.  To this day, this song means that the weekend has started to me.
  • “American Idiot” by Green Day: I didn’t like American Idiot very much until Eric convinced me to listen to it on the way back home from the Wendy’s meet in Charlotte junior year.  After that it was one of my favorite albums.  Alba?  “Holiday” and “Letterbomb” are better songs, but apparently have been combined with their mediocre neighbors on the album in Spotify, so screw you Spotify.

Wrestling

  • “Movin’ Out (Anthony’s Song)” by Billy Joel: This song somehow ended up on the radio on our way to every single tournament junior year, becoming the “official” Westminster JV Wrestling song of 2004-2005.
  • “Boombastic” by Shaggy, “Faded” by Soul Decision: Two of the best songs on the inimitable T.H. Leet’s wrestling mix tape, which we listened to ad nauseam our entire freshman season.  Nothing gets you in the mood to wrestle like the opening lines of “Faded”: “When I get you all alone, I’m gonna take off all your clothes.”
  • “Love Shack” by the B-52s: In a brilliant move, Eric chose this as Patrick Hickey’s intro song when he neglected to choose one for himself.
  • “The Boys are Back in Town” by Thin Lizzy: Eric’s wrestling intro song junior year; “Break on Through” by the Doors was his song senior year, which doesn’t lend itself to rad partying.

Band

  • “Birdland” by Weather Report: We played this song in jazz band and later saw it performed by a jazz vocal group from Spelman College, and to this day I’d swear that Gary Coleman played in Birdland.

College

graduation
Ironically Eric just RE-graduated this past weekend.

Friendtimes

  • “One Bourbon, One Scotch, One Beer” by George Thorogood & The Destroyers: In what may have been our first experience drinking bourbon and / or scotch, Mike, Eric and I ordered the Thorogood special at Sati’s.  It wasn’t the best experience we’ve ever had.
  • “Sir Psycho Sexy” by the Red Hot Chili Peppers: Probably the dumbest song on my favorite album, we listened to this whole album on the many trips between Atlanta and Durham that we took over our four years in school.
  • “Don’t You (Forget About Me)” by Simple Minds: Eric, Grady and I used to meet every morning in the Great Hall, where I’d have a goooooood morning camper! from Alpine bagels, and we’d lament the fact that we had to be there because we taped a guy’s butt cheeks together in the locker room.
  • “Trippin’ on a Hole in a Paper Heart” by the Stone Temple Pilots: When we saw STP play the NCAA Final Four in 2010, this was their encore.  Everything about that trip was fantastic.
  • “The Greatest Man that Ever Lived” by Weezer: On my 21st birthday, while Chris was repeatedly asking what the score of the game was, Eric was testing how drunk I was by making me recite the spoken word section in the middle of this song.  I was very drunk; it didn’t go well.
  • “Superman” by Lazlo Bane: The theme song from Scrubs.  I think I watched the entirety of that show in Eric’s dorm room and apartment, and I think it’s the reason Mike became a doctor.
  • “Total Eclipse of the Heart” by the Dan Band: Classic moment from Old School, which I think Eric gave me for Christmas once.
  • “Running on Empty” by Jackson Browne: Eric and I sat in the nosebleed seats for a Jackson Browne show that devolved once he started taking requests into a competition to see who could request the most obscure B-side.  This was the only song we recognized.
  • “Rude Boy” by Rihanna: Eric loved this song.  Everyone else found that hilarious.
  • “China Grove” by The Doobie Brothers: Played every time we drove through China Grove, NC about halfway between Greensboro and Charlotte.
  • “Statesboro Blues – Live at the Fillmore East” by The Allman Brothers Band: Eric and I used to drive a shuttle for my fancy neighbors’ parties (they live on a big hill); said fancy neighbor then invited us in and told a story about being a lawyer in Georgia in the ’60s, the punchline of which was that he maybe represented Gregg and / or Duane Allman at some point.
  • “A Praise Chorus” by Jimmy Eat World ft. Davey Vonbohlen: Riding around Durham in David Hershey’s Cabrio with Christmas lights strung up along the roll bar and this song on the radio is a seminal memory of freshman year.

Running

  • “Bang a Gong” by T-Rex: Without this song stuck in our heads for the entire winter of 2006-2007, there is no way I could completed our long runs training for our first marathon.
gonnadie
One of us looks to be doing better than the other

DUMB

  • “Freeze Frame” by J. Geils Band, “Vehicle” by the Ides of March were two of my favorite songs to play in the Duke University Marching Band.  Also “Vehicle” is kinda creepy AF if you listen to the lyrics.
  • “Closing: I Can’t Turn You Loose – Live Version” by The Blues Brothers et al.: The best version of this song I can find (the Otis Redding version is too slow and doesn’t do it justice); this is the version the DUMB imitates to introduce the team.  Here comes Duke!
  • “Devil With a Blue Dress” by Mitch Ryder: This song, when performed correctly, takes exactly one minute for the band to play.
dumb
What you don’t realize about this picture is that our band director is actually choking us

LDOC

  • “Bitches Ain’t Shit” by Ben Folds: Ben Folds probably wasn’t the best LDOC performer we had, but he did play this song; in his words: “I haven’t played this song in years, but you guys look like you might be drunk enough for it.”
  • “Semi-Charmed Life” by Third Eye Blind: TEB definitely was the best LDOC performer we had, and this song is nontrivially super great.  So yeah, it’s on the list.

Gen’l

Friendtimes

  • “Take a Walk” by Passion Pit:  Passion Pit played before the 2015 NCAA Final Four, spawning a really dumb Twitter joke theme that mostly just exasperates Eric.
  • “Ninja Rap” by Vanilla Ice; “Turtle Power” by Partners in Kryme: This could go anywhere — from watching the first two live action Ninja Turtles movies together senior year during wrestling season (omg pizzaaaaa) to the flight I took to Wisconsin just to watch the first Michael Bay Turtles movie to the fact that he reads the comics, the Turtles are a running theme through pretty much everything we do.  Cowabunga, dude!
  • “Your Love” by “The Outfield”: “I’m going now, and I’m leaving the door open…”
  • “Bad Case of Loving You (Doctor, Doctor)” by Robert Palmer: A) Robert Palmer is amazing, and B) Eric is a doctor.
  • “Party Hard” by Andrew W.K.: Because that’s what we’re going to do this weekend, obviously!
  • “Ridin’ the Storm Out – Live” by REO Speedwagon: Eric once told me that when he took road trips with his dad, an REO Speedwagon album was one of the only things in the car to listen to.
  • “Should I Stay or Should I Go” by The Clash: If you stay there will be trouble, and if you go there will be double.  Seems like an obvious choice to stay, then?
image
Fancy graduation party!

Bachelor Party Participants

  • “Fireflies” by Owl City: I remember this song drifting through the air one summer morning at Diki’s parents’ old place; for some reason it always reminds me of him.
  • “Clocks” by Coldplay: You know how I know Wade’s gay?  He likes Coldplay.
  • “The Power of Equality” by the Red Hot Chili Peppers: Keegan once drove all the way from Athens to Durham and back in one day to see the Red Hot Chili Peppers play in Raleigh with Eric and me on a weeknight.  They accidentally played “Under the Bridge” for their encore, but that’s not a song that’s conducive to partying.  They also played this song at the behest of drummer Chad Smith, because “they hadn’t played it in awhile.”
  • “Roll with the Changes” by REO Speedwagon: Every time I visit Mike, he spends at least two hours just mindlessly riffing on his keyboard.  I think at least an hour and a half of it is this song.  Remember: you can tune a piano, but you can’t tuna fish.
  • “Wait (The Whisper Song)” by the Ying Yang Twins: A hard call to go with this instead of “By The Way” for the aforementioned text message, but I have fond memories of Tyler preparing us to see his dick before Diki’s wedding in Kiawah.
  • “Miss You” by The Rolling Stones: David won’t be at the bachelor party because he has better things to do, so we’ll miss him.
  • “Danny Boy” by Happy Ol’McWeasel: Grady Daniel “Boy” O’Brien actually mumbles this song in his sleep.  This is the only version I could find that maintains the party vibe without utterly ruining the song — it only ruins it a little bit.  O Danny Boy, the Danny Boys are … Danny boys.
  • “Black Skinhead” by Kanye West: Patrick loves Kanye West.  This is the first song off Yeezus I think.  OK?  Happy, Patrick?
  • “Pretty Fly (For a White Guy)” by The Offspring: I accidentally let it slip that my favorite band for along time growing up was The Offspring.  Sue me, I was 12.

Conclusion

That’s it folks! You can check out all of these songs in order on my Spotify playlist “Eric Hall Bachelor Party 2K16,” which should differentiate it from all of the other Eric Hall Bachelor Parties we’ve had and will continue to have.

just the two of us

2FA AF

On Thursday night, I lost my phone.  On Friday night, I got it back.  This is the horrifying story of the intervening 24 hours.

horror
In a world without a cell phone…

The actual story of how I lost the phone is pretty much right out of a bad movie script — I’m on the company bus from Sunnyvale to San Francisco, where normally I’d be listening to podcasts and minding my own business with my phone literally connected to my face (i.e., impossible to lose), but this time I’m talking to someone on the ride — for the sake of this movie script, let’s assume it’s a fly lady and there’s hella sexual tension and not, say, a French dude — and I walk off the bus into the late evening drizzle and I’m like, “Nice riding with you, fly lady who’s definitely not a French dude!”  Then I go to put my headphones in and realize that lost in all that sexual tension with this lady who definitely exists1 is my phone, sitting on a table on the bus, which is at a red light not 50 feet away.

No problem, I think naively, I’m sure the bus driver will, like, look over at me before the light turns and I can just get it.  Of course he does not, and the light turns before I can bang on the door to get his attention, and off he goes.  I immediately resign myself to a phoneless life and start heading back home, thinking about how sad I am not to have a phone and to be walking back in the rain, and how I have to walk so far because I got off a stop earlier than I used to before the bus switched routes and started going 5 minutes out of its way before getting to the stop I usually get off at…  And in one fluid motion I both facepalm and turn around, running the half mile to the next stop, uphill in the rain, with my backpack on and my gym bag in my hand, elbowing people off the sidewalk in my haste.

A block away from my destination, the light at the intersection is turning yellow and the little red hand has counted down to zero as I approach. I run into the street and am almost immediately run over.  I have no choice but to wait for traffic as I watch the bus pull around the corner and stop a block away.  The light turns, and I sprint up the street with my hand out waving frantically hoping to get the bus driver’s attention.  I don’t get it.  He pulls away, and with my phone, he drives right past me.  I have failed, my phone is gone, everyone who’s just gotten off the bus is staring at me, and right then it starts just pouring rain.  I walk home, sopping wet and dejected.

rain
Like this, except HE PROBABLY STILL HAS HIS PHONE

I’m an adult; I can handle a little loss of connectivity, and it’s a company bus not the Muni, so I know I’m gonna get that phone back.  This is not the end of the world.  When I get home, I’ll just call the dispatcher and tell them I left my phone on the bus.  No problem.

Except I don’t have a phone, so I can’t call them.  OK, not a big deal, I’ll email them. I still have my computer.  I’ll just check the bus app and find out what their email address is.

Except I don’t have a phone, so I can’t check the bus app.  OK, not a big deal, I’ll check the wiki, I’m sure it’s there somewhere.

Except I don’t have a phone, so I can’t generate a two-factor authentication token to access the VPN so I can access the wiki.

Balls.

I did what any normal person would do: I vented my frustration by tweeting about it:

In the end I got in touch with the dispatcher by texting a friend at work and asking them what the email address was2 and got in touch with them; they’d have it for me in Sunnyvale on Friday morning, I could have someone pick it up and bring it back to SF for me. Which I did — super huge shout out to the French dude for bringing it back and then waiting for like a half hour for me to show up at the bus stop, he is the best.

But in the meantime, I ran into about a dozen other ways in which not having a phone is inconvenient — from finding something in my apartment that I could make beep at a certain time to wake me up, to adding stuff to my grocery list, to badging in at the gym — and one way in which, it turns out, having a phone is essential.  As I mentioned earlier, at my company we use two-factor authentication to remotely access the company network.  We also use it for, basically, everything else.

Two-factor authentication (or 2FA) requires you to have a password and another piece of information, typically served from a device that the server knows and trusts.  You then authenticate with both pieces of information — “factors,” you might call them.  This is pretty obviously more secure than just having a password, but you may be wondering why having a password isn’t secure enough — after all, you could lock your door and brick it shut, but most people just opt for the locks.  Without going into too much detail, it is possible that if a bad actor has access to our network, he or she may be able to access a list of passwords, and even if they are stored in some hashed form they may be able to look up the password plaintext in a rainbow table and, long story short, they then have access to everything.

Nicholas!
Not that kind of bad actor

Two-factor authentication is more secure because even if the attacker has access to everyone’s password, and even if they’re able to get their hands on a device or two, they can only log in as a person whose device they have, and not, y’know, everyone.  Plus getting access to the devices would be, presumably, difficult.

It’s so great and wonderful and secure that, as I mentioned before, we use it for basically everything.  It’s pretty common to use 2FA to access VPN, but even inside the VPN we use it.  We have automation software running on a server that requires 2FA to access.  Didn’t get to do any work with that yesterday.  We use Okta to manage access to internal apps.  We’ve configured Okta to use 2FA.  I was lucky that I hadn’t been signed out of Okta (which seems to happen about once a week), because if I had been, I couldn’t have used the wiki even from the office.  Or viewed my pay checks.  Or seen my calendar.  Or used email.

In the end, this whole miserable ordeal wasn’t much more than an inconvenience, but that was lucky.  If I had truly lost my phone, I’d be in huge trouble — and probably in even more ways that I know now.  I guess what I’m saying is…

phone
I’ll never lose you again.

  1. You don’t know her she lives in Canada
  2.  Except I didn’t have my phone, so actually I found an old offline copy of the bus schedule that had been sent to me before I moved here in the trash folder on my personal computer

The View from the Other Side

drumpf
Ladies and gentlemen, the next president of the United States! … talking publicly about his weiner

I have nothing new or novel to say about Donald Trump; everything that I could possibly say has been said better than I could say it by someone better than I am — whether that’s Mitt Romney attacking his business acumen, John Oliver attacking his flip-flopping, Marco Rubio attacking his … hand … size, or just literally everyone in the entire world who thinks that maybe, just maybe, the American people owe it to themselves and the rest of the world to think critically for, like, a split-goddamn-second about whom they give executive authority over the second largest nuclear arsenal in the world to.  No one reading this is even thinking of voting for the man1.  I’m not going to dissuade anyone from voting for him with anything I have to say.  Therefore, I’m not going to try.

Instead, I’m going to do something I try to do from time to time to make myself into a better person2: I’m going to try to see this and understand it from the other side.  Fear not, I am no Trumpist; if he is somehow elected, I will be among the first to move to whatever the warmer, less frozen version of Canada is.  I do this in part because it is only through knowledge of our enemy that we may destroy him.

head asplode
Like this

It is possible for me to perform this monumental transformation because, somewhat improbably, I actually have a Trump supporter friend3.  And no, he’s not a hate-mongering bass-ackward rube, he’s a PhD data scientist immigrant from Singapore.

I was arguing vociferously with him the other day on this subject, when he brought up the “Make America Great Again” slogan, which just really gets my goat — like super hard goat-getting.  Because when I hear that slogan, it implies two things:

  1. America used to be great
  2. America is no longer great

When I look across pretty much all of the major categories in which a nation might be considered great, I gotta say that objectively America is as great or greater now than it’s been in the past, ever.  Militarily, we are indomitable in a way that, despite current quagmires in the Middle East, we just weren’t during the Cold War; right now we contribute over 1/3 of the world’s total military expenditure and as much as the next 9 highest countries combined.  Economically, even coming out of a recession, we have basically been single-handedly driving the world economy, and socially the last 50 years have seen an incredible expansion of civil liberties, culminating in a series of Supreme Court decisions as recently as last year4 that have brought unprecedented freedom to our people.  If I’m thinking about any time before now that America was great, it’s probably late WWII and the immediate aftermath (never mind that we saw two recessions in ’45 and ’49 and another in ’53); unequivocally the United States was the dominant force in the world after World War II.

Oh wait — by “the world” I must mean “the west,” because there was always a terrible fear that this might happen to you for just, like, no reason whatsoever if you lived here:

kaboom
Duck and cover!

Aaaaaaaand by “the United States” I really mean must “the white male United States,” because there was also this whole thing going on:

FILE - In this July 15, 1963 file photo, firefighters aim their hoses on civil rights demonstrators in Birmingham, Ala. 1963 was a year of revolution in race relations in the United States. (AP Photo/Bill Hudson, File)
Ain’t America great?

We have a long way to go and a lot of things are broken and awful, but yeah, big picture, I’d say America is greater now than it’s ever been.  So I called my friend out on this, asking him exactly what he thought it meant to make America great again.  And to my surprise, he acknowledged my critique and conceded that, generally speaking, we live in a freer and better society than existed anywhere before this time.  But he was concerned that the political process in America was completely broken5, and that to him, Trump was the one man who could fix it.

This sounds ridiculous 6, but he brought it back to something that you hear a lot from Trump supporters that, quite frankly, I think makes most people think they are insane: “he says what’s on his mind,” or “he tells it like it is.”  And we, the sane ones, hear that and think, “Man, that is a crazy reason to vote for someone7.”  And that’s in part because this is how of we all play that out in our head.

As an example, the guy I’m talking to is the kind of person who will — loudly — tell someone to suck a dick at work, in an open office, sometimes with accompanying sound effects and hand gestures.  This is a guy who’s super smart and talented, but who — for some unknown reason — feels that this is the most accurate and appropriate way to express himself in public — at work no less– and it’s easy to think that in the same way that, say, the person two desks over might be offended at hearing him tell someone to fellate an unspecified member, he might be offended by not being allowed to say it.  And he sees Donald Trump’s ascendency as a natural reaction against a recent trend toward political over-correctness in the country.

This is not an argument framed along First Amendment lines (although I’m sure that a lot of people probably think it is).  To my knowledge, there is no First Amendment protection against getting fired from your job in private industry just because you told someone to go munch a bag of dicks8, and I’m 100% definitely certain that his employment contract specifies that he can be terminated at any time and for any reason — certainly for creating a hostile work environment, for example.

In fact, this argument has nothing to do with the government at all: when I asked my friend, seriously, whether he thought that we should have someone in the White House who would say whatever was on his mind to foreign heads of state, and how mind-bogglingly disastrous that would be, he responded, “The president doesn’t do anything anyway.”  If Congress makes all the laws, and the Supreme Court decides which of those are constitutional, what does the president even do9 — especially now, since pretty much no matter whom we elect, Congress will thwart them at every turn.

So while I’m sure that, for every constitutionally literate person who feels the way my friend does about what their First Amendment rights do and don’t protect, there’s someone who thinks it’s their First Amendment right to refuse service to the gays ‘cuz they’re a goddamn ‘murcan and their great-great-great-great-granddaddy didn’t die at the Battle of First Thanksgiving ‘gainst them Injun hordes just so they’d have to bake a cake for a queer wedding, let’s assume best intentions here and assert that a good chunk of people see legitimate grievances on both sides, but don’t understand why somebody else’s rights or privileges are trumping their own (excuse the pun) purely from a societal standpoint.

So we can take this narrative and slip into the shoes of a stereotypical Trump supporter: someone who’s afraid of losing their livelihood because of the opinions they have about just how big a bag of dicks someone should munch, or someone who’s already lost their job — let’s just say potentially to low-wage immigrant labor — but who can’t say anything about it without immediately being labeled just, like, super racist.  Then along comes the richest, classiest, winningest guy on earth to run for the position of Grand Figurehead of these United States of America, and he’s up there saying that he’s going to deport all the immigrants and anyone who doesn’t like it’s a loser who can go eat the biggest, classiest, winningest bag of dicks.  This guy’s saying everything you wish you could say — he’s saying a lot of other stuff too, a lot of which contradicts the stuff he said that you agree with, but never mind that, he’s getting away with it!  There’s a mob out there and it’s just like me — I’m not racist or sexist or offensive, I’m right, because I’m part of the biggest, classiest, winningest mob!

mob
Different kinda mob…

And maybe this narrative is true for some or even all of his supporters; honestly, I don’t know.  It’s certainly easy to imagine it applying in my friend’s case.  But when I asked him why he thought having a president who was willing to say pretty much anything could possibly be a good thing, he had a surprisingly rational response.  He believes that there are a lot of issues with the country — and there are, although we’d disagree about what they are, for sure — but that they inherently cannot even be raised because raising them would be politically incorrect.  This goes beyond being upset with a societal decision that one’s right to refuse service is less important than a gay couple’s right to that same service; it goes to the critical issue that one cannot even raise the question without being labeled a bigot.  How can you bring up a question about losing jobs to immigrant labor if, in doing so, you are immediately labeled as racist?  How do you address the very real issues faced with increased competition for formerly privileged classes in the face of expanded freedoms and shifting demographics if you are immediately labeled a protectionist bigot for doing so?  Here’s a guy who’s willing to bring all of those things up, and he’s running to be the leader of the free world, and he’s winning.

So how do we destroy him?  Honestly, I don’t know.  I hope that we do, because I think a Trump presidency would be at best a disaster for the country, and at worst a disaster for the world.  But maybe, just maybe, we can learn something from him in the meantime.  And honestly, by beginning a dialogue about some of these issues — after all, free speech and the ability to tolerate offence [sic?] are the hallmarks of a free and open society — we can learn something from one another.

 

  1. In fact, probably not one’s reading this at all.
  2. Impossible, I am perfect already
  3. One of the consequences of working at a large company, I suppose.
  4. Lookin’ at you, same-sex marriage!
  5. because it is
  6. because it is
  7. because it is
  8. or telling people they can dress however they want on Halloween, apparently
  9. Like, a lot? The President still does a lot of things and it is super duper important that that person not be Donald Trump.

Forever Young

I don’t want to live forever.

I bring this up because yesterday my boss was talking about how we are at most 30 years away from curing death through some sort of regenerative DNA therapy or something; he was in particular certain that cancer has no more than a few years left to live. From there, it’s a straight shot to curing pretty much everything else — heart disease, obesity, diabetes, you name it — and presumably aging, too.

baby
I guess you’d probably want to age at least a little bit

I’m not here to disagree with him1 — he’s a smart guy2, and he has a lot of friends in the biotech industry who are apparently working on exactly this.  I’m just here to say that that’s terrifying.

Let’s just assume we manage to cure cancer — completely cure it — in the next 30 years, and while we’re at it, we wipe out heart disease and diabetes, too.  First off — awesome, way to go world!  That is a really awesome accomplishment, and I unironically applaud you for it, despite what I’m about to say. Second off, what comes next?

I attribute this argument to my brother, who once pointed out that in a world where the average adult human would not expect to live into their 60s3, you could not imagine the existential horror of unavoidable prostate cancer4 or slowly, confusingly wasting away as your brain eats itself from Alzheimers.  In a world where, if you’re lucky, you’re going to die from being old and slow and bled out by a saber-toothed cat or put out to sea on an ice floe by your immediate family, it’s hard to imagine a world where those dangers no longer exist, and — again if you’re lucky — you die painfully of gout and liver failure from the copious quantities of alcohol you’ve consumed5.  Now we’ve purified water6, and people die from 70 years of eating too much and cancer and brain autocannibalism.  Imagine what terror lies beyond treating obesity and cancer and Alzheimers?  If living to your 80s causes your organs to literally give up, what does living into your 120s cause?

Saber-Tooth-Cat
Unexpectedly, death by saber-toothed cat.

But let’s just say for the sake of argument that whatever comes next isn’t terrifying or painful.  Maybe whatever we use to cure our modern ills also — and cheaply7 — cures all future ills as well.  Now imagine you look like this for the rest of your incredibly long life.

oldest
Don’t worry though, everyone else will look like this too!

“Aha!” you say as you read this, “Your boss said some sort of DNA therapy!  Perhaps that cures aging as well!”  Then you notice that I mentioned that earlier and that this post is entitled “Forever Young,” as though I, too, had thought of this exact line of argument.  It turns out you’re pretty dumb, and you probably shouldn’t have said anything at all.

Let’s say we do also cure aging.  We don’t lose our teeth or our eyesight, and our skin stays wrinkle-free and healthy looking, and our hair stays dark, and we suffer nary even a split end.  But is any sort of DNA therapy going to actually reverse aging?  Presumably it would use our DNA at time of treatment, and since DNA degradation may contribute to aging, it’s not unreasonable to think that we may only be able to stop aging rather than reverse it.  We’re talking about something that’s 30 years away, so if you’re in, let’s say, your late 20s8, you’ll be in your late 50’s by the time this treatment comes around.  And you’ll be stuck there.  Forever.

forever

On the face of it, that’s not so bad — I’d much rather be stuck in my late 50s than, say, pretty much anything later than that.  My parents are in their early 60s and they’re not doing so bad.  But it’s not like everyone will be stuck in their late 50s — everyone born after you will stop aging whenever they feel most beautiful, which will damn sure be younger than their 50s.  And there’s just more and more of them born every day, and they all stop aging at 25.  Now you’re part of some sort of weird protected class because you were born in the Years Before Aging Stopped and are therefore ugly and gross, and an increasingly large percentage of the world is young and hot and sexy and not you.

creepy
This is you now.

If they do cure aging in the next 30 years, millennials are in one of the unluckiest generations to have been born in the history of time — and that’s a history that includes thousands of generations where making it to adulthood was an accomplishment.

But, for the sake of argument, let’s say they really cure aging — maybe you don’t get to freeze time in your 20s, but generally speaking your body doesn’t wear out, and any damage already done can be repaired.  Awesome, right?  Yeah, super great.

Except for the resulting demographic crisis.  Ignoring for a moment the obviously issues, think about your grandparents’ generation.  They were the Greatest Generation, except for all those times they fought vociferously to keep society segregated and how they still use words like “negro” and blame the occasional local mishap on a Jewish conspiracy? And it’s fine because they’re old, and back in their day, that was the norm, and hell, they’re gonna die soon anyway?  Now imagine they never die and they always vote and they’re you now.  Sure, our generation has come a long way on accepting people’s differences, but it’s almost guaranteed that in 60 years we’ll be like way sexist/racist/classist/-ist-against-something, like robots, or people who are in love with robots, or people who are in love with people instead of robots, or schizophrenics9 — and that will keep happening, as each new generation fights some new social battle against the mounting legions of preceding generations, forever.

forever

This of course ignores the really obvious demographic issue, which we’re already facing: we have enough problems funding social security now that people live to be 80, just imagine how bad it will be if no one ever dies.  Maybe we won’t need social security, though — maybe everyone can keep on working, tirelessly, forever.

forever

Great, not only are we toiling in endless perpetuity, remember how hard it was to get a job out of college, applying to job postings for “entry level” positions with like 7 years of required experience?  Now imagine being born into a world where everyone’s been doing their job for hundreds of years.  Good luck climbing the ol’ corporate ladder!

But maybe we won’t need to work at all!  Ignoring completely the population crisis that would result from a world in which no one died (not to mention in which people stayed fertile their entire, interminable lives), imagine how mind-bendingly boring it would be.  Lord knows I have enough trouble keeping myself entertained for three hours after work each night10.  While it sounds awesome to be able to do whatever you want 24 hours a day seven days a week, imagine being able to do that… forever.

forever

It’s the only thing that sounds worse to me than having to work forever.  Now imagine a world where, instead of saber-toothed cats or liver failure or breast cancer or dementia or just being, like, really old, the leading cause of death is ennui-induced suicide.  That’s why I don’t want to live forever.

  1. Rest assured, I do
  2. After all, you’d have to be pretty smart to outrank the incomparable me
  3. To say nothing of the 40% of humans who didn’t live into their 20s
  4. Much less the existential horror of having a robot crawl up your butt to cut it out
  5. In much of the past, alcohol was safer to drink than water due to poor sanitation of public water sources
  6. Everywhere but Flint
  7. Right now medicare and medicaid alone are expected to cost 15% of GDP by 2040
  8. Since I have maybe three readers, all aged 26-28, this seems like a safe assumption
  9. We think it’s awful that previous generations classified homosexuality as a disease and tried to treat it
  10. Not to mention weekends… hence the blog

Culture Shock

When I was interviewing last spring, I met a lot of people at a lot of companies.  The people and the companies varied widely in talent, competence, vision, and mission, but there were constants throughout.  One of the things that kept cropping up, especially at larger organizations, was that people spoke in buzzwordy catch-phrases that had very obviously been created somewhere at the top of the company and then forced upon the underlings as “culture” and “team branding.”    People were moving fast and breaking things, demanding excellence, working with stunning colleagues, and just generally guzzling that delicious cherry Kool-Aid (TM).

Koolaid
Don’t be evil!

When you hear that kind of cultish lingo, it can be off-putting; I know it was for me.  There I was, talking to some of the brightest engineers, project managers, and MBAs in the country, and they seemed completely incapable of thinking for themselves, just parroting back meaningless phrases.  I mean, wake up, sheeple!  You’re playing right into the hands of the your corporate masters!  When I was weighing the pros and cons of the offers I got, I actually listed LinkedIn’s well-defined culture and values as a con.  I mean, it just seemed so phony to even make that list in the first place.  What company would ever be like, “We have a strong culture of settling for mediocrity!” or, “We willfully abuse our customers, because we enjoy it, and it makes us feel good.”

gummymint
Congress notwithstanding.

But in my time at the company, I have come to view the rigorous cultural definition as a huge positive.  Not because I think the values themselves are so revolutionary; they’re obviously great things to value or whatever, but again, pretty commonsense.  What I’ve found most useful is actually what originally turned me off in the first place — specifically, that they are so well-defined and brand-y.1

What I didn’t realize about a well-defined corporate culture is that it provides a shared framework for communication, a sort of internal language you can use that everybody understands.  This is especially important in a large organization, where you’re likely to work with people you’ve never met before; it acts as a sort of common-ground handshake, like how people from the same school or city can find common ground over a shared sports team. “Members first!” becomes the new “how ’bout them Falcons?”

This shared language is also critical to the day-to-day running of the business, because it facilitates communication and provides a framework for business decisions.  When someone comes to me with a ball of code that’s held together with spit, duct-tape, and band-aids and copied to a floppy disk — and not one of those hard floppy disks from the ’90s but one of those actually floppy floppy disks from the ’80s — I don’t have to try to figure out how to tactfully but forcefully chastise this extremely hypothetical person that I’m extremely hypothetically managing.  I just tell him that I’m going to “demand excellence” on this project — in this case, excellence is cleaned up, well-tested, and preferably not floppy.  Likewise, when someone proposes a project publishing everyone’s private messages, you can pretty easily shut that down as a quick “members first” violation.

floppydisk
Not even the 5 1/2″ one. The 8″ one, from before you were even born.

Those are obviously super contrived examples, but you’d be surprised how often the core values come up in conversation at work and how effective they are for framing the decision making process.  The core values’ success in this regard betrays an interesting quality about them — namely, that they were designed.2  This is probably one of those things that seems super easy on the outside, but there’s like a secret hidden world of best practices and domain knowledge and consultants who charge hourly rates in the thousands, like compliance with the Affordable Care Act, or celebrity Twitter account management.

Not knowing any of the insider knowledge, I attribute their success to the fact that they’re pithy, they’re bulletized, they’re imperative, and they’re ubiquitous.  They’re short and easy to remember, which makes them easy to sprinkle into conversation, and they come in a single short list; the company may value other things, but these are the core values, the ones we’re going to talk about and use as a framework for internal communication.  The fact that they tell us what to do and how to do it also makes them more tangible; they are a call to action rather than a list of ethereal concepts.3   They have also been visibly adopted from the top down; they get used by execs in meetings and speeches and are displayed prominently in internal communications like emails and posters, and because of the first three factors there’s not a lot of variation in phrasing that might confuse the issue.  They are, in effect, well-managed as a brand, which has been critical to their success.

I bring this up because, as I mentioned earlier, in my interviewing I encountered a lot of different perspectives.  One of the most interesting and, frankly, valuable perspectives I got was from the CEO of a pretty successful startup, who told me that as CEO he had two jobs: Setting corporate culture and identifying talent.  After that, he just gets out of the way. Since I still dream of starting my own (actually successful) business, it’s interesting to think about how I could apply what I’ve learned about corporate culture to my own business one day.  Of course, since that business is still extremely hypothetical, I guess I’ll just blog about it.

  1. Here they are, by the way.  You can also see them on slide 27 of the deck I linked to above.
    • Our Members Come First
    • Relationships Matter
    • Be Open, Honest and Constructive
    • Demand Excellence
    • Take Intelligent Risks
    • Act Like an Owner

  2. I don’t actually know that some branding team explicitly designed them for the executive team, but I have a strong suspicion that that was the case.
  3. An interesting point here: LinkedIn also provides an overview of its culture (see slide 4 in the linked deck above), which is a list of ethereal concepts (e.g., “Collaboration”).  These are almost never brought up in conversation.

Dear Recruiters

Dear Recruiters,

As a recent job hunter, I talked to a lot of people at a lot of different companies, and words cannot express how appreciative I was when I finally talked with one of you — a professional whose entire role was to work with people in my position, looking for jobs.  You were kind, you were empathetic, you were interested in my success as a candidate, and you lent your company an air of credibility.  In many cases, you went above and beyond, scheduling on-sites at the last minute and expediting the process from start to finish during a two-week whirlwind of interviews on both coasts.  I truly cannot overstate how thankful I am for your services.

jobhunter
Jobs — the most dangerous game.

But, in the aim of being open, honest, and constructive, I have some feedback for you.  This may not apply to each of you — in fact it may apply to only one of you, or even none of you — but I urge you to examine your practices on each point and consider how you might apply each lesson going forward in your professional activities.

My first piece of feedback is simple and easy to do: get scheduling right.  Let me know when I should expect to hear back from you, and write it down.  That date is now a deadline.  If you can’t get back to me by then, it’s not a problem, but you need to let me know before that deadline passes so that I can adjust my expectations.  Just to confirm, it can be helpful to summarize all action items and deadlines in written form — a quick email — after our conversation.  In an extreme example where such action would have been helpful, I ended a phone call with one of you with a discussion about my availability for a phone interview; we discussed a particular date and time, and I took that to mean that the interview would be at that date and time.  I waited an hour and a half for the phone to ring (maybe there was a time zone issue?), and it never did.  Finally, I emailed you, and to my embarrassment, our discussion was but a preliminary one, and in fact the interview had not been scheduled.  A simple note indicating the schedule was tentative, and that I should expect confirmation from a third party, would have saved me a lot of embarrassment.

While we’re on the subject of scheduling, could you also try to make sure you’re on time for our calls?  This feedback is not just for recruiters, but is especially aimed at people conducting interviews.  Since you’re calling me (and I’m not calling you), I have no recourse in the event you just don’t show, save to email the recruiter (see above).  About three minutes after you’re supposed to call, I start to get anxious.  After five minutes, you’re late.  After 7, I assume the problem is on my end, and that I’m somehow wrong (sometimes I am; see above).  Again, it’s fine if things change or you’re running late, just communicate that to me.

Speaking of time, let’s talk about time zones and daylight saving time for a moment.  I may be on record as being vehemently against both concepts, but we live in an unfortunate world where the places we inhabit, be they San Francisco, New York, Washington D.C., or any other where in this otherwise great nation, are subject to not one, not two, but three separate time zones1.  If you don’t know which one you’re in, Google “What time is it,” and you’ll find out.

Screen Shot 2015-08-09 IRONICALLY at 4.08.15 PM
Good work Google; of the 5.4 billion results, the one in the box was the most useful.

On the one hand, I’d like to express my appreciation — especially to the west coast companies, as I was on the east coast2, for always giving me a time zone.  On the other hand, it was confusing to me to see things like “11:00 AM PST on June 14, 2015,” since there are literally no places in the U.S. that observe PST in June.  A normal person would see this as an honest mistake3 and assume that they meant Pacific Prevailing Time, but as someone who was interviewing for data-centric positions, I honestly had the thought run through my mind that this might be some sort of test, given the importance of time zone information in time data and the indescribable complexity of dealing with such information.  I was so frustrated with this phenomenon that I composed an entire tweet about it:

Hopefully that tweet gives you a solid understanding of just how passionate I am about time zone data: Honestly? Not very.  I am pretty much 140 characters (or less) passionate about it, give or take a few blog posts.  You know what else I’m not passionate about?  Messaging apps.  Nor am I passionate about scalable web services, cloud storage, social networking, dating apps for 30-to-32 -year-old Zoroastrians with kids, or whatever your company’s app does4 that you put in your job posting.  Pretty much the only thing I’m really passionate about is sex, and even for that, only when I’m actually in the middle of having it.  For certain stretches of the term, you could claim I’m passionate about automation, or maybe dinosaurs, in the way a 6 year old’s unbridled enthusiasm about a topic well beyond his comprehension will undoubtedly leave a mark on him for another 25 years.  But Hadoop? No, I am not, nor indeed is Jim Q. “Apache” Hadoop himself, passionate about Hadoop.

However, as part of my new job, I am learning about Hadoop; it seems real great or whatever.  But besides passionate, you know what else I’m not?  I’m not the best at Hadoop — in fact I haven’t actually touched it yet.  So if you tell me that you only hire the best of the best, guess whose application you’re gonna be missing out on?  Hint: it’s mine.

missing it
I wouldn’t say we’ve been missing it, Carson.

 

But wait! There’s more!  Think about how many other, actually great people aren’t going to apply either.  Somewhere in there is probably a candidate that could transform your business, and you’re missing out on that candidate.  Bringing this back to my last post, in the words of Laurie Voss in his post “You Suck at Technical Interviews,” “Don’t hire [people] for what they already know.”  Hire them for what they can learn.  Going even fuller-circle on my last post, Joel Spolsky on the Stack Exchange podcast brings up some excellent points on that practice.  You can find them at about the 31:36 mark in the embedded episode below.

To paraphrase his point, which is actually Sara Chipps’s point from a previous podcast I haven’t (yet!) listened to, you don’t get fewer bad applicants, you just get fewer non-traditional ones.  The message you’re trying to send is that applicants will work with really smart people, but you end up scaring off anyone who doesn’t think they’re already the best, which means you miss out on a lot of really smart people who will learn what they need to, enrich the overall environment by teaching others what they know, and ultimately get things done.

Also, as a side note that I can’t make into a footnote due to *technical difficulties*, I didn’t start listening to the Stack Exchange podcast until recently, so it didn’t make it into my posts about podcasts, but in my own tweetwords5,

I know that I have been throwing negative examples left and right at you, but again, my experience was overwhelmingly positive.  To round this out, I want to take a moment to call out something that was done right.  Specifically, whether or not I actually was special in any way, a number of people, from recruiters to CEOs, went out of their way to make me feel that way.  There is nothing worse than feeling like you are one of a thousand people in a pipeline — a feeling I did experience — but the personal touch of people at a number of companies really made me feel like a valued individual, even at places I had no business applying to.  A small thing, like a 30-second phone call from an executive after receiving an offer or a text to say that the schedule had changed at the last minute significantly raised my perception of the company and my likelihood to recommend it to someone in my professional network.  It just goes to show the power of even the smallest gesture; a little thing can go a long way.

So, recruiters, hopefully I have given you an idea of a few little things you can do that will go a long way to improve your performance.  I look forward to seeing how far you’ve come on our next encounter, whenever that may be.

  1. No, I didn’t forget Mountain Time — the continental U.S. actually has 12 time zones.  There are three separate time zones for each of Eastern, Central, Mountain, and Pacific Time: Standard (which runs approximately November – March), Daylight (which is an hour ahead of Standard, and runs approximately March – November), and Prevailing (which is just a shorthand of saying “whichever local time it is, pick the time zone that matches that”).  These time zones don’t disappear when they’re not in use — e.g., even if the Pacific prevailing time is 4:01 PM on August 9, that doesn’t mean that it’s not 3:01 PM PST.  3:01 PM PST is still a totally valid time designation, it’s just confusing because the prevailing time zone is PDT, so you could technically schedule something for 3:01 PM PST and mean 4:01 PDT, but if you do that you’re a jerk and everyone is going to be an hour early to your party.
  2. and actually to the east coast companies I interviewed with while on the west coast for on-site interviews
  3. Hahaha! no, I kid! I kid! A normal person wouldn’t notice at all.
  4. I was going to look up and link to the postings I saw about being passionate about stupid stuff, but it turns out I’m not passionate about doing that, either.
  5. Tons of twitter in this post, jeez, what’s up?

The Guerrilla Guide to Being Interviewed

In a famous1 blog post from the great Joel Spolsky2 entitled “The Guerrilla Guide to Interviewing,” he lays out very practical, fundamental, and altogether good rules for how a company should interview technical people.  Laurie Voss, CTO of npm — who, by the way, was just super great to apply to, because Laurie actually keeps you updated on your status, like omg what a breath of fresh air3 — referenced the post in his excellent post “You Suck at Technical Interviews,” which I also highly recommend reading. These were obviously super useful to me back in the day as someone who had to interview technical people (duh).

But what about people interviewing for a technical position?  Or, more broadly, any position? As someone who recently went through the process of interviewing for a technical position — unexpectedly successfully, I might add — I present to you the guerrilla guide to being interviewed.

Turntables
Oh how the tables have turned

The premise, for those who didn’t read the above-linked post, is that a bunch of guerrillas have hijacked a ferry and demanded something you’re good at making (Spolsky uses a Fortran compiler as an example), and you have to prove that you should be on the team that will produce this thing in a life-or-death scenario.  Ultimately, you are going to be able to do this by proving to the team that you are smart (able to produce the thing in question, or learn how to)m that you get things done (you can produce it before the guerrillas blow up the ferry), and that you’re not an asshole.

So how do you do that?  First, I’ll walk you through the interviewing process, because it’s good to know what kinds of opportunities you’ll have to demonstrate your value.  Then, I’ll give you a couple of good ways to show that you’re a non-asshole who’s smart and gets things done throughout the process.

The process

I went through the full spectrum of interviews with five companies over the course of two or three weeks, and every single one of them went something like this4:

  1. Talk to a recruiter
  2. Phone screen (this may not happen if the job is in town)
  3. On-site
  4. Decision

The recruiter is there to determine if you’re a crazy person or not; for smaller companies, it may be any sort of non-technical person (I spoke to a CEO at one company).  The phone screen may be Skype-based, and if it’s a technical position, it should include some coding — it’ll almost certainly be with a technical person.  The on-site usually consists of a tour of the office and a few face-to-face interviews, which may be technical or not, and the decision will probably be made as a team and delivered over the phone (probably not in-person that day).

There are obviously variations on this theme, so ask the recruiter what to expect.  In fact, ask everyone what to expect next, always.  It can give you a good insight into whether or not you’re going to get to the next step; if someone really liked you, they’ll probably say it.  Plus, a lot of people don’t like the awkwardness of telling you you didn’t make the cut, so they just won’t get back to you and leave you in a weird limbo. It’s good to know when you should expect to hear something, and if you don’t then send an email or follow up.  The worst case is they don’t respond.

Know your audience

It’s critical to understand the role of the person you’re talking to.  For instance, the recruiter is really interested in three things: do you know what you’re applying to, have you just completely fabricated your resume, and are you a sane person.

To the first point, it is essential that you know what you are applying to, and can reasonably say that you are interested in the subject.

To the second, remember recruiters are not engineers, and although they will ask you basic technical questions about your resume, you should not get into the weeds about how your solution to X problem was linearithmic in time compared to the previous solution, which was quadratic.  They may have a few key words they’re looking for (e.g., if your resume mentions you have SQL experience, they probably want you to say “SELECT,” “UPDATE,” etc. and maybe want to hear that you have DDL and DML experience).  They are also interested in applications — what did you do with your SQL experience? If you say that you developed a login GUI, you are lying.  But if you say that you persisted user info to a DB, great!  You passed the test — you are smart (know the stuff you claim to know) and you did a thing.  Obviously, if prompted, give additional detail.

To the final point, this is your first personality screen; you want to sound intelligent without sounding like an asshole.  You can be the smartest man in the world and be an asshole, and no one will want to work with you.  Remember that the recruiter will be your primary point of contact and, in many cases, will end up going to bat for you after the interviews are done and it’s offer time (I think some may even be paid on commission), so you want them on your side.

Similarly, if you’re talking to an executive or a high-level manager, it’s almost certainly a behavioral interview — you want to sound inquisitive and curious about the business while demonstrating that the things you have gotten done have driven value for the business.  A lower-level manager or someone in the position you’re being interviewed for wants to know that you have technical chops and aren’t a huge pain to work with.

Fail early, fail often

Even the smartest people are really dumb sometimes (or maybe it’s just me?).  If you haven’t interviewed in a while, you should apply to as many marginal places as possible, so you can use them as practice.  These can be places that don’t really have a position that matches your skills, a place that seems really cool but you’re probably not qualified for, or a place you’re just not that interested in working at, but you are qualified for.  You want to apply to as many as possible, since only a few will actually get back to you.  This way, you can get out your dumbness before you need to demonstrate that you’re smart to people you really care about.  This also lets you know what sorts of things people are going to ask about — but more than that, when you lay awake at night wondering why you bombed such an easy question, you’ll be kicking yourself for not saying the obvious, glorious answer that has formed in the last few days since the interview.  Go with that answer when you want to demonstrate that you’re smart.

For instance, I interviewed for a really cool-sounding drone company, and they asked me my experience with deployment.  I basically responded with, “I dunno, we sorta emailed folks, and then if no one objected, we pushed it?”  That is, just, like, such a bad answer — but my problem was that I hadn’t thought about that question, and I wasn’t sure of a good way to answer it that would be interesting.  Within 24 hours, I had considered every single time I had deployed something that wasn’t super vanilla, and even though it was like 1-2% of total deployments, I still had 15 examples of things that needed backward compatibility built in during a transition phase; communications and documentation that needed to support multiple versions; OS upgrades on servers; daemon upgrades for job central, firm-critical scheduling software; core parsing software that we ran two versions in parallel for a month to ensure there were no unexpected differences (hint: there were!); the very versioning system we used!  I could have gone on and on about all of it, and the next time I was asked that question, I nailed it, and I sounded super smart and I got to tell people about a lot of the stuff I did.

Prepare, prepare, prepare

It’s not hard to prepare for an interview, and it gives you a chance to look smart to anyone you’re talking to.  There are a couple of things you really want to do, and, assuming you are even slightly qualified, they just aren’t that hard and they don’t take that long.

  • Basic company research: go to the website, check out TechCrunch (or whatever); see what’s new.
  • Basic job research: what do they want? What should you be emphasizing? This information will be in the job posting.
  • Basic interview research: if they tell you who will be interviewing you, check out their LinkedIn page; they’ll see you viewed it (“Hey look, the candidate prepared!”), and you can use this to tailor your questions to your interviewer.

It takes two hours or less to do this, and if you come up with an intelligent question or insight, guess what? You’ve just demonstrated that you’re smart.

Additionally, if they send you any materials, look them over.  98% of applicants won’t do that, but it gives you a good starting point for what sets this company apart from others — more importantly, though, it gives you a starting point for what this company thinks sets it apart from others, and you can hone in on that to your advantage in the question phase.

And remember, it helps if you wear a suit

Speaking of the question phase, most interviewers will leave a little time at the end for you to ask questions.  It’s really good to have a few questions you can ask everyone, and a few questions you can ask certain people.  Your questions should reflect positively on both you and the company, or else you risk looking like an asshole.  A really good question to ask is, “What do you love about the company?”  It’s positive (never ask “what do you not like,”) and you’ll get a really good idea what the people at the company value.  Plus, you can ask the same question to everyone, and you’ll learn something new every time.  Another good one is, “why did you join?”, which is a very different question — “I joined because the hours and pay, but I stayed because I love the product.”

Otherwise, ask questions targeted to the role of the person interviewing you — ask a person doing your job what his or her day-to-day is like, but ask senior folks when they joined and what role they had (does the company promote from within?), and what the direction of the team / company is — where will we be in 5 years?  Also, use “we” — it makes it seem like you’re already seeing yourself there.  These sorts of questions make you sound like a thinker — a smart person!

You can convince them you get things done by asking questions about how they get things done — “Tell me about some production-level code you’ve shipped in the last three months?” is a great question for a peer, while, “What are the key milestones to hit in achieving that vision?” is a great follow-up to the question above for a more senior interviewer.

Be one with the interviewer

If you haven’t interviewed someone before — or maybe even if you have — maybe especially if you have? — and you’re applying for a technical position, make sure you read those two posts I mentioned earlier5.  They should give you keen insight not only into what a good company is looking for, but also into the fact that no one who is interviewing you knows what they are doing.  This is important to understand: most interviewers are not professional interviewers, they are professional something-elsers.  They are evaluating whether or not they want to work with you.  If you’ve never worked with someone who is awful to work with, congratulations on interviewing for your first job!  The rest of us know that there’s nothing worse than someone who’s defensive and protective of their knowledge, or someone who just can’t get a single thing done… or, of course, an asshole.

Another reason to read those articles is it gives you a chance to evaluate the company.  I, for one, agree with those articles, so pretty much anything that deviated from them was a knock against the company for me.  For instance, I had a technical phone screen with one company, only to be brought up to the office and have four half-hour behavioral interviews.  No whiteboards, no coding — no one gave a hoot what I could do or how I wanted to do it. Just the same questions asked over and over again on loop.  Likewise, I had a few interviews where people just asked me rote questions — they were OK questions, designed to see if I was completely lying on my resume, but they weren’t coding — and it kinda put me off, because I kept thinking, “This doesn’t matter.  Ask me to solve a problem.” The message I got was that the company interviewed to find people who were smart, not people who could get things done, which meant they probably had a relatively high proportion of people who couldn’t.

It’s also important — especially for the more junior people you interact with — not to underestimate the kindness of strangers.  They want you to get the job; it’s uncomfortable to judge people.  I can’t tell you how many people I interviewed that were marginal at best that I passed along, just because I didn’t want to be the reason they were out on the street.  That’s absurd, of course — they’ll get a job somewhere — but the only part of their life I see as an interviewer is the part where they perform their trade in front of me and I say, “Meh! Not good enough.”  So relax, they’re on your side.

Talk too much

The interviewer doesn’t know what you’re thinking.  He or she doesn’t know what you know.  You, and you alone, do.  Your job is to tell them what you’re thinking and convince them that you know what you know you know.

Consider the following completely ridiculous example: you are handed a clicky pen and told you need to write a letter.  How would you do it?

Clicky pen
OH GOD WHAT DO I DO

You can either click the pen and write the letter — congrats, you know how to operate a clicky pen — or you can walk through the absurd amount of knowledge required to get to even that stage: “This is a pen. I know that because I’ve used a number of pens before, of many types.  Pens are good for writing, so this is an excellent tool to write the letter.  I know pens come in many different colors; this appears to be blue, because the pen itself is blue and the pen color usually matches the color of the ink, but we can test that in a moment.  Pens I’ve used have largely come in three varieties — clicky, twisty, and capped.  Additionally, I know of a fourth type — fountain — but I’m not very familiar with them. From experience I can say that this appears to be clicky; there’s no twisty portion and certainly no cap, so I’m going to test that assumption by clicking the end, which should — aha! — produce the writing tip…” Not only do you know how to operate a clicky pen, you know how to operate all pens, and you’ve demonstrated deep knowledge of the subject and clearly identified that you understand the boundaries of your knowledge.  You are smart, and what’s more, your vast experience in pen applications suggests you also know how to get things done.

Admittedly, it’s not a bad idea to preface your spiel with a quick, “I’m going to explain everything I do, so if I’m going too slowly or being overly explanatory, just let me know.”

Know what you don’t know

The above example hits on something really important, that Laurie Voss also mentions: it’s important to know your limits.  As he states, “Strong candidates say ‘I don’t know’ as soon as they hit their limit, and may start asking questions.”

There’s no better way to prove that you’re not smart than to try to bluff your way through something you don’t know.6 Instead, it’s important to be up front about it — “I don’t know how I’m going to write a letter with this pen, the pen doesn’t appear to write.”

As Voss continues, “The very strongest candidates say ‘but if I had to guess’ and then attempt to extrapolate.”  It’s a great chance to show what you do know, even though you don’t know exactly what they asked.  “I don’t know how to use this pen because it doesn’t write and there’s no top to remove, but I’ve seen pens that need to be dipped in ink.  They look different than this one, though, so that might not be it either.  Can you give me a pointer here?”  Then, when they say, “Yeah, this is a clicky pen, click the thing,” extrapolate further — show that you internalized and learned something: “Ah!  The tip protrudes!  I bet there’s other mechanisms — maybe you could twist a pen and have the tip retract and protrude like that.  This retracting tip concept might offer a few advantages over the models I’m familiar with; for instance, I bet it does not dry out as quickly.”  Not only have you learned something, you’ve also shown that you’ve internalized it and can apply it to other situations.  You’ve turned a weakness into a strength!  People love that stuff.

Be honest

This is sort of in the same vein as “Know what you don’t know,” but broader — be honest about what you don’t know, and about what you do.  For one company, I had six or seven interviewers.  Three of them asked me basically the same question.  The first time, I took a broad approach — “You could do X, but I could see how Y, Z, A-W, and a few more options could really work.”  The second time, I said, “I’ll be honest, I’ve already been asked this question,” to which the interviewer replied, “Great!  Where did you end up?” and I told her, and she said, “OK — can you take that one step further?” I had the chance not only to score Honesty Points (TM), which are only awarded to people who aren’t assholes, but also to show her that I’d Learned Something (TM), which is what smart people do, and that I could Apply What I had Learned (TM), which gets stuff done.  If I had just walked through the exactly the same thing as the first time, I wouldn’t have really shown anything new, and — worst case — they’d have figured out they asked the same thing and I gave the same answer twice without saying anything.  When the third interviewer asked it, I told her I’d heard it twice before, and when she asked where I got to and I told her, she decided to use a different question, and I got to show how smart I was in another way.

This also goes for other parts of the interview — be open and up front about whether you’re looking at other companies or whether you have other offers.  Yes, you can use another offer to get a quicker response from a company you’re interested in, but don’t lie about it.  If you have an offer from a company you’re not interested in, let them know immediately that you won’t be taking the job.  Likewise, don’t make up an offer in order to get a faster / better response from a company you are interested in.  Don’t be an asshole.

Keep an open mind

My mother was a recruiter, which was a super valuable resource in my job hunt, because she’d seen it from the other side (see “Be one with the interviewer”).  The best piece of advice she gave me was, “Don’t decide on the job until you have the offer.  Until then, you’re interested.”

This has a really practical application: if the job you’re really interested in turns you down, and you stuck around with a job that gave you an offer you weren’t so interested in, guess what?  You still have a job.

But on another, less practical level, it gives you two more things.  You get something to compare other jobs to — you get additional experience with a number of different companies, and if you’ve been asking everyone your canned questions, you get a lot of answers to cross-calibrate by.  You may also not know that you’re interested in something until you go through the process.  I got an interview at an internet behemoth — something that predates Google — and I thought it would be a terrible job.  When I got there, I found out they were under new management, had a clear and high-impact way to get back on track, and were solving a really interesting problem.  Likewise, for another job in “business analytics,” I started off thinking that it wasn’t what I wanted to do.  After I went to the office and met everyone, I realized that the company was full of incredibly intelligent, bright, professional people that I wanted to work with, the position involved developing data-driven applications more than analysis, and the company had a great culture. If I had said no up front, I would probably be in Charlotte right now instead of San Francisco.  So don’t turn the job down until you get it.

Closing thoughts

I’ve spewn a lot of nonsense at you, but this is the stuff that worked for me.  In the end, you want to present yourself as someone who’s smart and gets things done, but also as someone who’s not an asshole.  The best way to do that is to show that you can learn, show that you can solve problems, and be open and honest.

And if that’s not enough?  You didn’t wanna work there anyway.

  1. Insomuch as a blog post can really be famous? More like… well-respected.  Well, insomuch as a blog post can be well-respected.  Maybe it’s more like well-respected by a guy I respect pretty well?
  2. Spolksy, a founder of StackOverflow and the Stack Exchange universe, had a blog called Joel on Software. It is phenomenal, and I highly recommend you go there and read through the reading lists.  I will reference another of his posts in an upcoming post myself — if I don’t, yell at me and remind me to.  In other news, the Stack Exchange podcast is usually hilarious, and I highly recommend listening.
  3.  This, too, will be the subject of a future post, tentatively entitled “Dear Recruiters.”  Same instructions.
  4. This post is about interviewing, not getting an interview, but let it be known: I cannot overstate the value of having a referral.  Of the five companies I made it on-site to, 4 were through referrals, and the fifth was the first place I’d heard from in a month or two of searching
  5. Here they are again, just to be sure: The Guerrilla Guide to Interviewing and You Suck at Technical Interviews
  6. Example: I am incredibly privileged to have gone to an amazing high school.  We had one of the best European history departments in the entire country — literally award-winning.  If I had to guess, 80+% of the students at my school who took the AP European history exam scored a perfect 5.  The year I took the exam, we had the choice of a question about “mass politics” at the turn of the 20th Century and another about something relatively obscure — maybe the defenestration of Prague or the importance of the 18th Century Revolution to 19th Century France.  Most of the class took the former option — “I know something about the French Revolution, but we studied the turn of the century in depth, ‘mass politics’ kinda sounds like nationalism, it’s probably something about saber-rattling diplomacy and World War I, which I know a lot about.”  Mass politics was about the expansion of the vote to women and minorities.  Most people got 3’s and 4’s because, even though they demonstrated that they knew a lot, they bluffed their way through it and ended up answering the wrong question. It’s probably one of the worst years our European History department ever had, all because people didn’t know what they didn’t know.

Disruption Disruption

As part of my afore-stated desire to get more exposure in the start-up community, I’ve been reading a lot of related news, following some movers and shakers, and of course watching Silicon Valley on HBO.

That clip is, to me, the best moment in the show (yes, even better than the whiteboard session that prompts the truly world-changing “middle-out” algorithm that ultimately saves the company).

Three jerks
Yeah — that whiteboard session.

I’ve read a lot of articles and done research on a lot of start-ups hoping they’ll hire me (hint — they will not), and I’ve seen a lot of people claiming they’re revolutionizing X or making the world a better place with Y.  I think part of this is due to the fact that we are in a huge tech bubble (this is being debated by people much smarter than me, so, in the words of The Dude…)

The Dude abides...
But it really ties this post together.

Everyone wants to be the next Google, the next Facebook, the next Twitter, the next… WhatsApp?  Remember when they sold for $19 Billion?  These guys changed the world; they completely disrupted the way we used the Internet and… chat… services… (see what I mean about the bubble?)  But the problem is that most companies that get started and succeed — and I don’t mean 51%, I mean 99.9999% — will do so by offering a useful product that is marginally better for some users than its competitors’ products, or by selling an inferior product to some people better than its competitors do.  They won’t disrupt anything, which doesn’t mean they’re not successful.  They’ll be perfectly successful and may grow to employ thousands and have six, seven, eight-figure profits, but won’t explode onto the scene and earn their founders billions in a few short years.  But no one is going to get excited about investing in / using / working for a company that has a plan for long-term, modest success — they want to change the world! They want to disrupt!  So you end up with scenes like the TechCrunch Disrupt scene above.

So what constitutes true disruption, then? Let’s look at some of the inarguably disruptive technologies of the past — I’ll focus on transportation here, because it’s easy to quantify and is something I think is poised for disruption now.  Prior to 1800, the chief means of long-distance land travel was the horse and buggy — with a generous average speed of about 8 miles per hour, on modern roads in ideal conditions.  By the 1840s, the steam locomotive had increased that speed by a factor of 10 to 75 mph.  This invention was transformative to a degree most people don’t realize — it made or broke cities; it ushered in massive demographic shifts as foreign workers came to America to work on the railroads;  it altered the way wars were conducted; it connected the two coasts so as to make a democratic republic the size of America even possible, and it literally changed the very concept of time

At the turn of the next century, the automobile would derail that industry — literally.  No longer were people subject to the schedules and routes of the railroads; they could drive themselves — at similar speeds — anywhere that roads would take them, wherever they wanted to go.  Airplanes would increase the top speed a human could travel from tens to hundreds of miles per hour; jet engines would double the speed of airplanes.  200 years ago, it would take you more than 36 hours, nonstop, with a team of horses and ideal conditions, to travel from the bottom of Tennessee to the top of Florida. Now you can go literally anywhere in the world in that time.

GEORGIA
And nothing about Georgia could be considered “ideal conditions,” especially in 1815.

Each of those technologies has in common an increase in the performance of key industry metrics by an order of magnitude — trains are ten times as fast as horses and buggies; cars cover tens to hundreds of times the ground that trains cover; planes go ten times faster than cars.  The same is true of other truly disruptive technologies: telegraphs and telephones made instant what previously took days or weeks, depending on the distance; wireless subsequently increased reliability by orders of magnitude.  Computational speed and storage space have increased by orders of magnitude over the last several decades, enabling everything from the home computer to the iPhone to the cloud and the Internet of Things.  The printing press decreased the time it took to reproduce a page of text from hours to seconds and ushered in a world where it’s actually rare to be unable to read.  Every technology you can look back on and say truly disrupted a market improved on a key metric within that market by an order of magnitude — at least a binary order of magnitude (most sane humans refer to this as “doubling”).

PTandA
Plus, they gave us this 1987 Steve Martin / John Candy vehicle.

Compare that to some other so-called “disruptive” technologies. Unless WhatsApp can read my thoughts so I don’t have to type out messages, it’s not disrupting my chat experience.  Unless eero can hold a full proxy-server’s worth of Netflix movies, I’m going to be happy with it, but it won’t disrupt my wireless internet experience.

The best example is Uber — people claim Uber disrupted the taxi market (actually, they claim it’s a ridesharing service, which I’ll get to in another post).  Uber certainly changed the market for the better; I love Uber, I think its product is superior, I think its founders are undaunted, ruthless businessmen willing to do whatever it takes to overcome challenges to their idea, and I want it to be around forever.  But it isn’t disruptive, at least not for the consumer — it’s just better.  It brought an open, transparent, and mathematically sound market to a service that was run entirely by cartels.  It marketed its app really well, to the point where I doubt most people know that there are similar, just-as-good apps for regular cabs.  But it didn’t decrease the price of a cab ride by 50%, much less 90%.  It didn’t decrease the time it takes to get from point A to point B.  In fact, I still sometimes prefer to use cabs, because I need to know they’ll show up at a certain time — if I’m going to the airport in the morning, I’ll use Curb to book a cab at 5:30  AM rather than call an Uber, because I know my cab will be there and I can schedule it in advance.

Of course, this brings up another commonality for each of the disruptive inventions above: time.  The first steam engine was created in England in 1804; it wasn’t until 1847 that it hit 75 mph (and at that, in very rare circumstances), and the first troop movement by train wasn’t until 1861 at the First Battle of Manassas.  The Wright Brothers first flight traveled at six miles per hourthe fastest plane in WWI didn’t top 150.  In a world of instant gratification, where Facebook had a million users after a year and over a billion after 10, we tend to ignore the fact that ideas and products take time to grow and mature; the railroad is useless until you have an extensive network of railroads, the jet engine is useless until airports have long enough runways for jets to land on.

With that in mind, Uber just did something that, as I’ve discussed before, could lead to something truly disruptive: they looted the ever-loving bajeezus out of Carnegie Mellon’s robotics lab to build their own self-driving cars.  In so doing, Uber may become, sooner than expected, what proponents of the self-driving car have long foreseen: a fleet of vehicles for hire, completely eliminating the need to own automobiles.

This move is disruptive — it registers an order of magnitude higher in fundamental industry metrics — in a number of ways.  Obviously, in the ride-for-hire industry, it decreases the cost by an order of magnitude, since they no longer need to pay the driver.  On the other hand, it also increases the percentage of rides that are for hire from near zero to near 100, becuse it disrupts the automobile industry itself, shifting it from a consumer-focused industry to an enterprise-focused one, as the percentage of American families who owns a vehicle drops from 95% to near-zero (this may seem far-fetched, but the economics are sound — it will be cheaper to arrange a self-guided taxi ride than to own and maintain your own vehicle, even if it is also self-guided, pretty much everywhere; there would be no reason not to get rid of your car).  The total number of cars would also plummet by an order of magnitude, since most cars are parked 95% of the time, meaning on average we have 20 times the number of cars we actually need.  Now imagine a world where you don’t have to look for parking — ever — because parking doesn’t exist, and think about how much additional land that opens up in urban and suburban settings, creating walkable communities closer to their urban centers.  This move has the power to completely change (er, disrupt) not only the ride-for-hire industry, but the personal transportation industry, and our very landscape, in the same way the automobile first did 75 years ago.

Sprawling suburbs
The automobile created a world where living in Villa Rica 30 miles outside Atlanta is considered the same thing as living in Atlanta.

With that said, I want to leave you with a few industries that I think are prime for disruption (but, you know, without giving you any ideas on how to disrupt them).  The short distance, personal transportation industry is about to change completely, but the long distance travel industry has been basically the same for 65 years; in 1952 a commercial jet could travel at 500 miles per hour, and the Boeing 737 has been able to hold 100 people since 1967.  Everything since then has been incremental improvements; I want to see either of those numbers change by an order of magnitude.  On a completely different note, software development has gotten incrementally easier and easier, but it still takes a team of experts (more on that in another post) months to create a real product; Fred Brooks argued in 1986 that there is no silver bullet to this problem, and that only incremental progress will be made in the field.  He’s just as right today as he ever was.  Finally, energy prices have been tied to commodities like oil, coal, and natural gas since the inception of the electrical grid; new technologies like solar and wind are not reliable enough to disrupt the market, and we are still tied to the same grid we’ve been tied to for over a hundred years.  Finding a cheap, clean, renewable (or at least plentiful), and scalable source of energy, or being able to broadcast that energy without maintaining an aging network of unreliable power lines, has the power to disrupt the most critical industry in our society.

Those are my thoughts on disruption — if you care to disrupt them, leave a comment! And of course, if you care to solve any of these problems, please do.  I look forward to the disrupted world of tomorrow.