Ruby

@RTR_tech

As I write this, today is beach day @RenttheRunway (July 31, 2014).

What does that mean? It means our CEO Jenn Hyman (@Jenn_RTR), decided that the whole team deserved a day of “Mandatory Fun” at an awesome beach with tennis courts and a pool and the whole nine yards, on the company's dime.

A few days before that Jenn spontaneously ordered an ice cream truck to come to the office and give everyone free ice cream. We're not talking 10 people here - we’re talking about the whole, enough-to-fill-multiple-busses, RTR office.

A few days before that there was a Hawaiian style BBQ/Luau at our Dream Fulfillment Center in Jersey (where your dresses are shipped from).

You get the idea.

But lots of startups have occasional crazy fun surprises like that (right?). What's really telling about a company is the day-to-day. This blog is about life @RTR_tech, not RTR in general. Make no mistake: this is a tech organization and it’s an awesome one.

Let me tell you a story. It's going to be a bit scattered, and probably long-winded, but I think you'll like it.

I promise I'll mention some tech along the way.

A little more than a year ago, I was living with my family in Ann Arbor, MI where I had worked for about 13 years. Ann Arbor is an awesome little college town, but my wife wanted to move back to NY where she grew up. As a prerequisite, I wanted to make sure I found a job in NYC where I could truly be happy. This was more difficult than you might imagine.

I was looking for a small-ish, startup-ish, flexible, open-minded tech company with outstanding people.

I eventually discovered a large list of NYC companies of this ilk. Cruising down the list in alphabetical order, I eventually came to Rent the Runway.

"A fashion/e-commerce company?," I thought, "meh."  I passed right over it.

Fortunately, it didn't end there. The CTO of another company (whose selfless kindness I should have repaid long ago) reconnected me with RTR_tech via CTO Camille Fournier (@skamille) I had a phone call with her and was extremely impressed. The rest is history. I also spoke to Vijay (@vjsubr), head of Analytics, Jenn Hyman and various awesome members of the tech team and I was convinced.

One year later, I can tell you that this company is far from just a "fashion/e-commerce company" and is anything but "meh."

These people are awesome and humble. They learn from their mistakes and they learn fast. They actively seek feedback. They listen deeply. They really, honestly care about their teammates. They have created a culture of openness, idea sharing, risk-taking, fun, friendliness, honesty, continuous improvement, teaching and in-the-end, success. And they pay me to hang out with them all day!

What follows is just some of the awesome stuff I’ve had the pleasure of living with @RTR_tech.

First off, we use modern technology tempered with a good bit of wisdom that the “Shiny New Thing” is not always as awesome as it seems. The grass is not always greener on the other side. We’ve adopted a lean-and-mean, modern Java web service framework called Dropwizard. Not Node. Not Scala. Not Python. They're all cool, don't get me wrong, but Java is tried and true and way cooler than you might think if you just take the time to get to know it. :) Plus: Java 8! And cool stuff like @AutoValue. We're working hard to move our front end code to Backbone. There's some Ruby in the mix for good measure. Plus, have you seen what renttherunway.com looks like? It's freaking beautiful! We work with awesome designers and we really care about making it not just work right, but also look great.

We have regular, weekly one-on-one meetings with managers, team leads and peers. These are not perfunctory tech status updates. In these meetings we talk openly about what's bugging us. We refine our goals and take steps towards achieving them. We work to carve out time for people to work on that open source contribution they've had in mind (making @AutoValue work with Hibernate perhaps). As a manager, one of my personal rules of thumb is that performance reviews should not contain any negative surprises (or only good surprises). I've learned to use this one-on-one time to make sure that's the case and that I'm guiding/teaching (and learning from!) my reports to avoid those negative surprises. Just a few days ago I was writing a performance review. When I really got into the meat of it, I realized that I had some new suggestions around goals the person could set, but that we hadn't talked about yet. Before I wrote those things into the review, I grabbed the person and talked through the suggestions. The next day, we had the Important Official Mid-Year Review meeting. I delivered my write-up, and watched the pride swell up in the individual. This is not to say there was no constructive feedback, but that that feedback had already been well established and internalized. That's how you do performance reviews. We do that here @RTR_tech. If you're on my team, you'll get a performance review like that. I really care - we really care - about our teammates as people. In the end, that's all we have. Java code is easy. People are hard. Help them refine and follow their goals. Help them thrive. Help them learn. Help us learn.

@RTR_tech we are a learning organization from top to bottom. As such we’re encouraged (by @skamille and others) to attend technical conferences. RTR_tech will send you to speak or just attend. And of course, @skamille, @markwunsch, and @OMGannaks held a panel discussion @RTR_teach - oops, @RTR_tech - about how to write successful conference presentation proposals, because… they're awesome.

Similarly, we do Drinks and Demos every Friday. This is our chance to share essentially whatever we want as engineers. Maybe it's @timx showing off the new RTR Unlimited product/feature his team just released or @ericqweinstein teaching us some new stuff about "The Rubies." It could be @CarloBarbara doing a case study of how to systematically debug a tough OutOfMemoryError, because, as @skamille says, "don't flail."

Speaking of "don't flail:" you can learn that and so much more by going to @skamille's weekly office hours and having a great discussion with her. I'm not exaggerating that every time you sit down to talk with her, you'll walk away having learned something or with a new perspective on something. She maintains these office hours as a way to remain accessible and keep the communication lines open despite the fact that our tech team is quite large and growing rapidly. How accessible? Our awesome intern Maude (@QcMaude - A Post Internship Look at RTR) has mentioned to me how great it is that she is able to openly talk to Camille as she would any other peer.

There are lots of big brains out there in the world though, and we can't send the entire tech team to the next “Edgy Tech Conference.” In order to fill that gap we bring in guest speakers from time to time. We’ve invited people like the creator of Backbone (@jashkenas) and (@mrb_bk), an engineer from Code Climate (which we use for JS/Ruby static analysis).

"What if," you say, "I get a crazy idea that I just want to try and I never have time to build it?" You'll have your chance! In the year I've been here, we've had a hack day, a (3-day) hack “week” and a full, five-day hack week the last week of August. On hack week you get to work on (more or less) whatever you want as long as it’s vaguely related to making RTR more awesome. Best of all, these hack week projects often turn into real, even huge, projects that alter the roadmap of the company! For example, on renttherunway.com we have a feature where users can upload their own photos. Unfortunately, the quality of these photos varies wildly. What if we had a way to automatically give each photo a quality score so that we could show the higher quality photos more prominently? I happen to have a background in machine vision. For my first hack day project, I built a first cut of such an image quality metric. It was simple, but actually showed visible improvements. (In a nutshell, the metric gave preference to images with 1 or 2 faces and images that are overall brighter.) Another example? Our huge new feature, a whole new subscription-based rental model called RTR Unlimited, started life as a hack week project.

But wait! There's more! A recent new hire asked me if RTR_tech does anything with open source. Indeed, we do! For starters, we are committers and some of the main promoters of the aforementioned Dropwizard. We also have more than one OSS project that we produced internally: Alchemy for A/B testing and Conduit for simplifying the use of message queues. I'm certain these projects are just the beginning. In fact, it's worth noting that our internal software development process is modeled to a degree on open source development. We use pull requests for everything, no matter how small, and naturally, have awesome unit test coverage. This is a great way to develop and combines the best of OSS development with the benefits of being in an office right next to your teammates. For one thing, I have found pervasive code reviews to be an excellent way to spin up new people on company standards or languages that are entirely new to them. It really works and has allowed me to remove one worry from my list of "oh man, new person, so much to cover."

All of this sounds fantastic, but we’re a startup so we must be working like dogs, right?

Well...

We have unlimited vacation. Now I admit to being a bit (a lot) cynical. On joining RTR_tech, I assumed that "unlimited vacation" was code for "guilt-based vacation." I was wrong. This is simply not the case. People take lots of time off, myself included, and I haven't seen any guilt-tripping at all. Remember, we're talking about real people here. Awesome people. It turns out, we respect each others’ need for time off. When @MichelleWernick goes to Paris, we all get excited for her and people step up to help fill the gap in QA. When @skamille goes to Hawaii, we don't flood her inbox; we step up and exercise our latent CTO superpowers. When she gets back, there's Hawaiian candy on the kitchen counter and funny stories about emergency room visits.

In fact, it doesn't end with unlimited vacation. We have stellar work/life balance in general. @RTR_tech we understand that the trick is to anticipate, plan, and course-correct a project early. We work hard to avoid feature creep and instead focus on quality. We work smarter, not harder. In this way, we deliver awesome software without people freaking out at the last minute, having to put out fires, and working 12-hour days. Sure, we're not perfect at it, but we've had major successes and we believe in it. We're continuously improving. (How about a Drinks and Demos presentation about what went right and what went wrong in that last big project release? OK!) We embody this in other ways too: Unlimited vacation begets unlimited maternity leave. (I've seen it more than once! It happens! It works! All companies should do it!) And after your maternity leave, maybe you want to have your newborn brought by the office every day so you can take time to feed it. Of course! Who wouldn't allow that? I have a son and a wife. I get to go home and eat dinner with them and when I return to the office the next day, renttherunway.com is still there and I get to do more cool work on it with my fantastic team! I'm more loyal for the stability it provides. I take the time to write a blog entry because of it, and I spend that much more energy recruiting the next awesome teammate because I love telling them about it! What if you don't have a family? What if you're young and unattached? RTR is in NYC! Go enjoy living in New York Effing City!

Did I mention we're encouraged to write articles for our tech blog? Perhaps you've seen said blog? These kind of things are literally listed in our quarterly tech team goals. "1) Ship feature X so we can rent some more dresses. 2) Post 6 tech blog entries."

OK, I promise I'm almost done gushing, but stick with me a bit longer. There's so many more cool things to tell you.

I had a new junior engineer start recently. I gave him a week or two to settle in before I gave him his first major task. Let me set the stage: we're nearing completion of some initial research that uses Python to do Markov Chain Monte Carlo simulation for parameter inference on a Bayesian Network. (To predict the future. NBD.) I asked New Guy to determine if it's viable for us to port that to Java for production use. (The answer, two weeks later, appears to be yes, if a bit inelegantly, via Yadas (Yada Yada Yadas). So that's a thing we do @RTR_tech. Rad, right?

Our interview process is an area we've recently been working to improve (because we're doing lots of it). So how are we doing? Yesterday, I got overwhelmingly positive feedback from a new hire. He thought that it was fantastic to have an RTR_tech engineer guiding him through the whole interview process, answering his questions, being open, honest and enthusiastic. Shortly thereafter, he took the gig. Sounds like it's working!

Did you notice that I used one or two (or forty-two) Twitter handles here? That's because I'd love for you to join our conversation! I want to hear from you! What are we missing? What more can we do? To my fellow RTR_tech-ers (is that a thing?): what is life @RTR_tech to you? Follow us on Twitter @RTR_tech! But don't stop there! Connect with these awesome people about awesome stuff: @skamille, @ericqweinstein, @markwunsch, @bhsdrew, @CarloBarbara, @timx, and many more (use transitive closure to find everyone!)….

This is life @RTR_tech.

P.S. - I really wanted to figure out how I could work in a joke about Talk Like A Pirate Day being a big deal at RTR, but, as any good engineer knows, sometimes you have to kill your little darlings.


A Rubyist's View of Modern Java

An Unpleasant Introduction

When I was in college (back in 2007), "learning Java" meant "forging Java applets in the dark heart of Mount Eclipse." As you would imagine, this involved a lot of import java.awt.* and public class SelfLoathing extends Applet and lava and blood and Uruk-hai. (Some of this is certainly my own failing; my boss, who knows much more about Java than I do, loves Eclipse.) I don't think anyone died as a direct result of writing old-school Java or attempting to use Eclipse, but I witnessed at least one grown man reduced to tears. I finished the semester and swore I would never write Java again.

The Language (Java 8, or "The Ocho")

Java has come a long way in seven years. A more comprehensive guide to those changes is available in this excellent series of posts over on the Parallel Universe blog; for the points most salient to Rubyists, Pythonistas, JavaScriveners, PHPeons, and others of more dynamically-typed persuasions, I've made a short list of the delightful bits of modern Java we're using here at Rent the Runway (TL;DR at the bottom).

To reverse each string in a list in Java 7 and earlier, you used to have to pull a stunt like this (assuming you weren't in an interview and were allowed to use Java's built-in String methods):

To a Rubyist, this is something of a cross between waiting in line at the DMV and burning a hecatomb: quite a bit of rigmarole to obtain a simple result (a driver's license, a favorable wind toward Ithaka). Now, you could write truly Javatastic Ruby:

However, you'd be much more likely to write:

Or, if you're feeling fancy:

To quote Eddie Izzard: et voilà. Truly, Ruby is an elegant language for a more civilized age. Could Java ever look like this? Surprisingly, the answer is yes! In Java 8—a.k.a. The Ocho™—you can now write:

Is this as wond'rous as the one-line Ruby version in mine eyes? No. But it's leaps and bounds above the imperative/procedural style to which Java, the quintessential object-oriented language of the 20th (and now 21st) century, has been shackled for the better part of two decades. And this is really only scratching the surface of what's available in The Ocho™ (I'm gonna make this a thing, I swear): better type inference, method parameter reflection, and fibers (sort of) are just a few of the additions that are turning Java into modern Java (and making modern Java a really pleasant language to write).

The Ecosystem (Frameworks and Tooling)

I've found that even more important than the features of the language you write is the ecosystem in which it lives; one of the reasons I quit Haskell was because its package manager, Cabal, was dreadful. (It probably still is, though I haven't checked recently.) Java's package management systems are well-known and mature (I'm a fan of Maven myself), but the same can't be said for the myriad Java web frameworks out there.

First, I'm going to straight up skip J2EE: partly because I'm not old enough to remember it, but mostly because anyone who is probably doesn't want to.

Some people will tell you about "modern" web frameworks like Spring Web MVC or Play. Spring is (near as I can tell) not much more than a wrapper around all the things nobody likes about Java EE, and I haven't been able to discern any core philosophy with regard to best practices or patterns for solving common web service problems. And if this were the worst of these frameworks—if the design were confusing but the underlying technology sound—that would be one thing. I've actually witnessed Play, via some JNI-based Unforgivable Curse, segfault when you added a route. Which is, I suppose, what you get for wanting to do something insane like that.

Enter Dropwizard, which is not only sane, but absolutely delightful. Despite my limited Java superpowers, I got the "Hello, world!" service up and running in about fifteen minutes. Everything worked as expected. It made sense. Though it's a young project (started in 2011 and recently released version 0.7), there's excellent documentation and a helpful (though still somewhat small) community. I'm really pleased it's our framework of choice at Rent the Runway, and I've actually started going out of my way to read Java pull requests to learn more about it and how we use it.

Finally, I'd be remiss in my short survey of modern Java without a brief discussion of IDEs. As mentioned, I despise Eclipse and wouldn't mind seeing it blotted out of all human memory, but I strongly believe in knowing (and, ideally, liking) your tools, so I think anyone productive with Eclipse should continue to use it. That said, I picked IntelliJ when I came back to Java earlier this year, and it's excellent: on-the-fly warnings (which I generate liberally), code completion, automatic refactoring, and (best of all) immediate and smooth integration with Java 8. IntelliJ's certainly not a new tool, but its newest version (13) is, in my humble opinion, the editor of choice for modern Java.

TL;DR

  • Java 8 is a serious improvement on the core language, and I think programmers everywhere (particularly those writing Ruby, Python, JavaScript, and PHP) owe it another look;
  • Dropwizard is a first-class web service framework and the first Java web framework I'm really excited to dig into;
  • IntelliJ makes developing in Java (particularly Java 8) a breeze, and I wholeheartedly recommend it. (This is from a guy who spends 90% of his day in Vim.)

Am I going to switch to only writing Java tomorrow? Of course not. But with Java 8, Dropwizard, and IntelliJ, I'm interested in learning more about the language, its platform, and its tools, and I'm excited to build something great with them.


Deploying Daily

Six Months Ago

“I’m not seeing the new tooltip on the grid application. Did you deploy that one yet?” “Yeah, just now.” “Did you remember to sync the submodules?” “Yes.” “Did you flush Memcached and the CDN?” “Crap. No, I forgot. Which order do we flush those in again?” “Memcached, then CDN. Then make sure to deploy to both Heroku and Engine Yard.”

These were dark times.

When I started at Rent the Runway nine months ago, our storefront was divided into seven swimlanes: independent Sinatra applications that, in theory, allowed us to handle high traffic through fine-grained control over different parts of renttherunway.com. If we suddenly got a burst of traffic from a news story or celebrity tweet, we could independently scale the home page from the rest of the application to handle it. We also gained fault isolation and tolerance, since one part of the application going down meant we both knew where the problem was and could prevent it from spreading.

We soon discovered, however, that while swimlanes were a good idea, we’d applied them at the wrong level of granularity: 99% of the time, our changes to the codebase affected multiple swimlanes. Even when we were working on just one, we ended up writing a lot of duplicate code because all the underlying data providers were identical. Fault tolerance and isolation were nice, but we were dead in the water if users could hit the home page but not transact, or vice-versa. We didn’t even have full isolation: since all the data providers were the same, a service disruption to one swimlane would likely affect several. All this meant that deploying our storefront had rapidly become an enterprise of lunar mission proportions.

Game face

The author. Or Ed Harris in

Apollo 13

Deploying all seven swimlanes entailed syncing two submodules (one for views, another for assets), possibly bumping the version on two gems (one for the base application from which each swimlane inherited, another for our API clients), updating code for all seven swimlanes (each its own Git repo), then deploying each one to both Heroku and Engine Yard (in the event of problems with one, we’d fail over to the other). The dependency graph looked like a drunk John Madden telestrator instant replay diagram.

Oh yeah: and each swimlane had its own CDN and Memcached instance, each of which had to be flushed separately and in the right order.

Oh! And this was just to deploy to stage. Once we were on stage and confirmed everything looked good, we had to do the whole thing again for production. It took two developers six hours, start to finish, to deploy renttherunway.com. We deployed once a week on Thursdays, and if for some reason we couldn’t get a release out the door (which, as you can imagine, wasn’t uncommon), site traffic and weekend on-call concerns forced us to wait until the following Tuesday for another window of opportunity. Like I said, it was like a lunar mission.

Present Day

We knew we had to do better. Camille, our head of engineering, charged us with putting together a task force to get us from our wobbly weekly deploys to a system of smooth, daily ones. Since many of Rent the Runway’s back end services are named after animals, the task force became the TuskForce™, with our unit of progress the stampede (rather than the sprint). We determined we needed to do three things:

  • Consolidate our seven swimlanes into one application, which included eliminating the submodules;
  • Develop a feature flagging system to allow us to toggle features on and off without deploying the application;
  • Move the application to a single host (rather than Heroku and Engine Yard) and automate the deploy process.

Smashing the swimlanes together was arduous and a lot of things broke, but within a couple of weeks we had a functioning, single “swimming pool” comprising the original seven Sinatra applications. At some point, “Seven [rings] for the Dwarf-lords in their halls of stone” came up. Many Lord of the Rings jokes followed.

While we were busy combining the swimlanes, I set to work forging a feature flagging service, dubbed “Flaggregator.” (Rejected names included “The Flaggro Crag” and “Spiro ‘Feature’ Flagnew”.) Flaggregator is a Sinatra application that keeps track of feature flag data through Git and GitHub, so in keeping with our LotR theme, we named Flaggregator’s GitHub user “Frodo Flaggins.”

Flaggregator

Flaggregator™

With the arrival of Flaggregator, we could modify features on the site without a deploy. Now, if we’d stopped here, we’d already have had some serious wins over the previous process: deploying one application weekly and changing features whenever we wanted was night and day compared to the stress of our old weekly, six-hour deploy marathons. But we knew we could do more, so Fellowship-style, we pressed on.

We moved the application off Heroku and Engine Yard and over to the private cloud, then automated the deploy process via Rake and Capistrano. (This is nothing groundbreaking, which is a Good Thing: you should save the experimentation for features and small, new projects, not core infrastructure.) We also spun up a Hubot instance named Fashionator to serve as our deploy concierge, telling us about pull requests, notifying us of deploys, and running integration tests.

Fashionator

Fashionator in her natural habitat (HipChat)

The result? It now takes one developer six minutes to deploy our storefront application, and we do it every day. And this wasn’t our only win: moving to a culture of continuous integration has resulted in more and better tests, better code quality and coverage, and improved code review processes (all of which I’ll cover in future posts).

TL;DR

TuskForce completed its work at the end of January of this year. In three months, we made the following changes to our storefront deploy process:

    • Went from deploying seven applications to just one;
    • Decreased our deploy time by 98%;
    • Increased our deploy velocity by 400%;
    • Added the ability to turn features on and off independently of daily deploys;
    • Made our business stakeholders, customers, and developers much, much happier.

We still have a ton of work to do: scaling, infrastructure development, configuration management, and continuous deployment (that is, multiple deploys per day) are just a few of the difficult problems we’re tackling in 2014. But with the huge cultural and technological successes we’ve seen in the last few months, I have no doubt we’ll find imaginative solutions for them. Rent the Runway’s future is very, very bright.

If you want to be part of it, come work with us.


Parallelizing Test Execution: Speed Run!

Browser tests (sometimes) take a long time to finish. If your automated tests do things such as registering for a new user, adding a dress to a shopping cart, and then renting that dress, while checking along the way that each step is good, it might take upwards of 2-3 minutes. Combine that with the fact that each test file has multiple test cases, and a file might take upwards of 15 or more minutes to finish execution. Now, with multiple test files, your entire test suite might take upwards of an hour to finish execution. This isn't conducive to agile programming. The solution: parallelize your test execution. With Ruby and RSpec, this is handled easily with the addition of two gems: parallel_tests and parallel_split_test. You can parallelize test execution of the test files and then the test cases within the files themselves.

With parallel_split_test v0.3.0, you can use regular rspec test options. This means that you can pass custom formatters and write to specific output files. This comes in handy when you want to create JUnit formatted xml files for Jenkins or other continuous integration systems.

Nat Ritmeyer has written a custom RSpec JUnit Formatter that I've been using: http://www.natontesting.com/2012/05/25/rspec-junit-formatter-for-jenkins/

After installing the parallel_split_test gem, execute the test using this command: parallel_split_test my_spec.rb -r ./junit.rb -f JUnit -o results.xml

NOTE: The one caveat is that parallel_split_test will generate an xml file that contains multiple root nodes. I've attached a function below that will normalize all .xml files in the specified folder.

[gist id=5398225 bump=1]

Using parallel_tests and parallel_split_test, we've managed to cut our execution time from 30+ minutes down to 3 minutes. Theoretically, your tests should only take as long as your slowest individual test case. Your only limitation now is how many nodes are available to your Selenium Grid. You are using Selenium Grid, right?

If you have any questions or comments about parallelization in test execution, you can leave a response in the form below or find me on Twitter at @brokenthumbs. Enjoy!