@monsky: (Somehow this got disconnected from the thread below.)
The headline of the slide you pulled that from is literally "Hard Work—Not Relevant."
Netflix doesn't care if you work 100 hours a week, they care how well you perform. If you can do an amazing job in 30 hours a week, they're one of the few places which would actually embrace you.
> I'm not sure how Netflix grades things... but in school B is "Above average".
So what? Why on Earth should Netflix be targeting "average" workers when they deliberately pay twice the average.
That's about as ridiculous as saying that Harvard should stop admitting A students because "B" is average.
Netflix has a deliberate strategy of keeping their engineering bar very high and paying accordingly. That doesn't mean they want you to sacrifice your life to the company.
The problem about the idea of "doing great things in 30 hours" is that people who are not directly involved with implementation and deployment usually think that each and every 2 day task takes at most 2 hours, because we humans are unable to expect the complexity that most tasks have on this planet. (every time you think about a college, super market employee etc "just do your job man" you are experiencing that fallacy)
And then you as the 30 hours employee are faced with not delivering 40 hours of work in 30 hours (which is possible if you are smarter, more involved, have more experience etc) but no you are expected to do 160 hours of work in 30 hours, because you don't compete with the other engineers but you compete with the expectation of your manager.
We need to accept that most people (colleagues, employees, customers, bosses) will always be disappointed by the results we achieved because they had unrealistic expectations to begin with. And we must excuse them because we do the same the other way around. That is why "40 hour week" is so relevant. Because then you can say after working hard for 40 hours, really doing all you can, maybe even achieving 30% more than your colleagues, that time is up and you need to take care of your children, wife, parents, friends and hobbies you may have.
Now let's assume for a moment that all I said was BS. Let's assume I'm super untalented and for most people it's actually possible to achieve in 30 hours enough to impress their leaders. Now you do it once. You do it twice. Maybe thrice. But at that point your leader will adjust in his head, maybe without realising, what 100% means for you. So the fourth time he doesn't think it's awesome anymore. The tenth time he thinks you are lazy. And now you have to achieve 50 hours of work in 30 hours. It still doesn't work out.
Sorry, I wasn't able to express myself clearly. If you fix a 40 hour week your boss is disappointed that you stop working at 40 hours. If you do not have a fixed amount of hours your boss will still be disappointed, but you have no good reason to ever stop working. Without the 40 hour limit you work more and more and more and still disappoint your boss. The disappointment can't be avoided. But the unlimited overtime can be avoided, by limiting the amount of hours you work.
It has nothing to do with everyone else. If you impressed your boss while sticking to a 40hr week, they'll only be impressed so long before it becomes your norm. In which case you bump up your hours to maintain your prestige. I don't buy that reasoning, just saying it goes both ways.
I think the general idea that results should be the only relevant metric is good. I'm a bit worried that the mantra could have bad long term implications though. Iirc. I read a study from the domain of education science that basically said "always praise work ethic, rarely praise results" and it seemed convincing at the time.
[I'll see if I can find it later and edit it in]
> GP is much closer to the truth, but still low
> Source: I was a senior engineer at Netflix.
OK, suppose it was $400K+. How do you describe your life (-work) balance? Also, may I ask you - was it your decision to leave Netflix? (and what you didn't like there?)
Another thing: as you can see one can have wrong expectations - e.g. on salary negotiations one can ask for 250-300K while as you say, the norm is much higher... So do Netflix propose alternative much higher salary compared to what you've asked?
Work life balance was great. Sure there were crunch times, but for the most part it was 40 to 50 hours a week. If you were at the office at 7pm, you were there pretty much alone.
When I started at Netflix, I asked for what I thought was a crazy salary, almost double what I was making previously. My boss said, "No, that's not high enough, lets give you 10K more". Every subsequent raise after that, my reaction was generally, "oh, that's more than I expected".
Netflix knows that there is no point in low balling you, because you will soon find out and then be upset/leave. They never want to lose someone over a matter of money. They only feel good about losing someone because they no longer enjoyed their work.
Thanks, really interesting. One more thing to clarify - on top of that "$400K" salary, does Netflix also pays for social/health insurance (including family)? If not, how much you normally pay for such insurance?
I mean, I've heard many times that while NET base salary e.g. in Microsoft or Adobe might be lower, when you add up all the benefits, bonuses, stocks - it becomes quite competitive to Netflix compensation. Also factor in the job security and family-friendliness and it becomes much more attractive... What do you think about this?
Basically they either cover your health insurance, or you can opt to just get paid extra and not take insurance (if say you have it from a spouse's job or something).
And yes, the other big companies total comp does come pretty close to Netflix. The difference is Netflix gives you cash to spend however you want, and those other companies force you to consume benefits that you may not want to get your "full comp".
No it's $350k+. First off, I got this information first hand from someone I know who got a job offer there. Second, go check the H1B salaries from 2015, that shows the real salaries, not glassdoor.
The problem with only having "A" talent is that you become very vulnerable of bias in your own criteria. People often primer more concrete skills like in-depth knowledge. While sometimes you just need people who are more motivated to do whatever. It's a classic problem of successful sports teams for instance. They buy a lot of high profile players that end up not working very well together, won't adapt to new situations and are sensitive to failure.
Yes, that's true but it's also part of why Netflix also has a "fire fast" culture (unlikely sports teams). So people who can't adapt their skills to performing well in a team aren't kept long.
Efficiency pay (which Netflix heavily practices) does a great job of attracting people who are "motivated to do whatever."
That doesn't make any sense and is a deliberate misreading of the slide.
It's well-accepted that when you say "our team is made of A-players" you're grading against the overall industry, not within your individual team. Grading within teams leads to perverse systems like stack ranking.
> Sustained B-level performance, despite "A for effort", generates a generous severance package, with respect.
You've got to love these startups who basically say "if you're not giving us everything beyond what we expect... go screw yourself. I'm not sure how Netflix grades things... but in school B is "Above average".
Is Netflix looking to employee and develop people or are they looking to extract life from people?
Probably the latter. In my view, Netflix (IBM, Amazon, Bloomberg etc) should consider what would happen if they were completely honest to their employees:
"You are a cog in our machine. We do not care for you. We only care for your work. We will pay you insofar as you can execute on our vision. Further, you will be paid according to how hard it is to replace you. If you can be replaced by someone who executes significantly better, relative to your pay, we will tell you to leave."
Needless to say, stating that would make many of their employees, especially the weak-minded and insecure (those who defend their terrible culture), unhappy. I imagine even those in charge of the culture, speaking those words, would feel bad.
And they should feel bad. While such overwhelmingly negative words should almost never be spoken with authority, these decision makers must at least be able to answer the question "What are we really saying to our people?" truthfully. And if the answer is something along the lines of "you're a cog in the machine", then something has gone very wrong[1]
In my non-expert opinion, if a company truly wishes to be a "good" place to work, it should know the distance between their words and the truth, and then focus on minimizing that distance.
Spoken more simply: the more dishonest a company is to its employees, the worse the culture.
[1] Or right, if the company requires joylessness to turn a profit.
>"You are a cog in our machine. We do not care for you. We only care for your work. We will pay you insofar as you can execute on our vision. Further, you will be paid according to how hard it is to replace you. If you can be replaced by someone who executes significantly better, relative to your pay, we will tell you to leave."
Isn't this the ideal of wage labor and the reality of 98% of jobs? I've never even suspected that my corporate employer cares about me as a person aside from how I produce, and have never suspected that I'm being paid by them for any other reason that it would cost them a lot more than they pay me to find somebody to replace me. I don't mean to say I haven't had great friendships and working relationships with management, but to the business, I'm a line item. When I notice a business making decisions contrary to the bottom line, I suspect corruption, not corporate affection.
They're a cog in my machine, too. I can only depend on that cog if it behaves rationally.
I agree it's the reality for a majority if not the super majority of jobs. I also think it could be the ideal of hourly wage labor: which is to say that the company wants nothing more than what they can motivate you to do with money, and that you want nothing more from the company than their money.
Looking back, I see I made my comment in the context of Silicon Valley behemoths and startups: companies that can both pretend to, and potentially change some fundamental way of doing things.
I think the difference in my mind between "creative" companies and normal ones is the "potential", achievements outside of a strictly defined expectations of contract, for both the business and the employee. Maybe a great insight, a new tool... anything that greatly benefits both the business and the employee, especially if the employee uses the product(s) and believes in the mission.
When you take away that potential (by saying employees are just cogs in a machine), you might take with it the creative spark that these hopeful companies ostensibly rely on. In my mind, that "potential" is the real joy of a job, which is my reason for the (slightly snarky) comment that "cog in the machine" is only acceptable "if the company requires joylessness to turn a profit". If I could, I would edit that post to be more clear/to take into account businesses which do not require any specialness.
At least for my central point, "the more dishonest, the worst the culture"... I believe that stands, especially in circumstances where a person is (somehow) manufacturing toilet paper and their supervisor is telling them they are (somehow) on their way to curing cancer.
> If I could, I would edit that post to be more clear/to take into account businesses which do not require any specialness.
That I'm creative (amongst other things) is why they pay me well. My successful experiments have a record of adding value well above my pay even if aggregated with my many unsuccessful experiments, although I try my best not to keep it that way for long. I'm not sure I've had any technical job with completely "strictly defined expectations of contract", I think that's a red herring. It sounds nice that the most creative and innovative projects/companies give less of a shit about the bottom line; I personally think that the bottom line is the fuel for creativity (didn't have much money when I was younger.) Anybody can sell your product with unlimited money to spill, but doing it with the best profit margin possible is creativity.
In my experience, most employers think this way, but they don't like to acknowledge it. I've had bosses that talk a lot about "win-win scenarios" and how we're all working together to this common goal, all the while creating a very worker-hostile environment where people are pushed to work much more than they feel comfortable with, with regular emails praising the workaholic who works so much he never gets to see his kids. The unspoken message is, this is what we expect of everyone.
These same people get profoundly uncomfortable if you respond to their platitudes with reality. They prefer to keep up this illusion of cooperation, because it makes them feel better.
I've said this before, but how I wish software companies based on the cooperative model[1] were more of a thing. It's not healthy for people to be treated like worker bees. Corporations should be a collaborative effort, and the wealth and decision-making should be distributed proportionally.
In a way, the structure of our corporations both reflects and influences our society. A recent study has shown that the US is now more of an oligarchy than a representative democracy. Would we be happier and better off if "working for" was replaced by "working with"?
Bit offtopic perhaps, but Netflix CEO pay is something like 15-25 times the salary of Joe Engineer.
Worth noting, but the 1000x figure that gets bandied about so often when referring to CEO pay is typically limited only to the Fortune 50 or so, and is in no way the mean pay for the top guy. S&P 500 companies lower the average to less than 400x worker pay, and most businesses have a much smaller gap than that, and across all companies, average CEO pay is something like $157k.
I guess I'm thinking of the traditional startup IPO, where the first handful of employees get "fuck you money" while everyone else only gets a small piece of the pie (despite possibly doing just as much work).
But also, more importantly, people outside the C-level rarely get a say in how the company gets run.
That's kind of a different thing altogether though, really. The "fuck you money" generally comes only in the case of a liquidity event, such as IPO, sale, etc.
To my ear, there are compelling arguments both ways. Assuming a large company, employees 1-x should probably enjoy some of that fuck you money, but clearly, at some point there's a falloff. If my company sold tomorrow, I would not have in any way been involved in its sale. Same with an IPO. Our company has been around for years, sells expensive software, has millions in revenue, and I honestly couldn't tell you whether or not we've turned a profit yet. Assuming we have, it was probably done only last year, which meant that somebody was footing the bills of a multi million dollar company for a decade or so, and that person wasn't me. If somebody was spending millions on an idea that took a decade to pay off, then absolutely they deserve more money in a liquidity event than I do, and I'd be a fool to begrudge them that.
That said, not all companies are created equally, and many get to market / revenue / profitability at different timetables with different employee counts, so YMMV. That said, the notion of 'fair' is so abstract in this scenario then I think it's safe to assume that it is universally convoluted. Hard to pin down exactly what is or isn't fair without internal knowledge I don't have.
It's not about the amount of effort that you've contributed, it's about the amount of risk you've taken on. The larger the company gets, the less likely it is in general to fail. So of course those who join sooner get more money - they took on far more risk than the guy who joined the month before the liquidity event.
I think that is economic nonsense. Risk taking does not merit any rewards otherwise it would be a sane strategy to just maximize the risk of any endeavor.
Despite what some might consider common wisdom, capital and labor are equally valuable. A company will get nowhere without its workers to move it there, and it can't get started without somebody taking a financial risk.
Which just translates, as many of these Netflix threads do, to constant terror that you're going to be out of a job. I'll take my game, A or B, somewhere else, thanks.
Good point on the terror. On the individual level, A/B/C players is your average self-justifying dichotomy: I'm better than those people because I work hard, I work smart etc. Yet on the system level, it's the perfect rationalization for companies too lazy to hire correctly and too impotent to own up to mistakes.
"Why couldn't these folks go above and beyond like these other teams? It must be that they are inherently not good enough. Fire them and get a new team in."
I would even go so far to say that it's the CEO's and senior management, not the employees, who push this myth the most. In somehow all formulations I've seen, the persons's boss, who is amazing (regardless of context), and her or his bosses who are also amazing, are the judges of all performers in the company. Because the system is perfect or filled with perfect people (read: the top part is), any point of failure must be the employee, by process of elimination.
Netflix hires very few entry level engineers. I just checked their jobs page[1] and there are almost no postings for non-senior engineers.
I'd argue that a senior engineer should be highly effective while still developing in her role. Also, expectations for different "levels" of engineer vary. Typically an "exceeds expectations" performance review for "Senior SDE" would be neutral for e.g. a "Staff SDE."
In terms of total compensation are they really that much different from other big companies? I know a fair amount of folks making Netflix level money when you factor in stock. They just give it in cash. shrug
I mention above Netflix pays Senior Software Engineers over $350k/yr. This is far above anything any other company does by a wide margin. I'm not sure how much Google or Facebook employees make, but are they really making upwards of $400k/yr total comp?
An argument I've heard against this is that stock (etc.) is a way to give delayed money; it depends on you being locked in, the company not tanking, etc. The "equivalent" money now is more valuable. I'd love to hear a refutation of that, along with what these salaries are.
Yes, principle of time value of money says you need to discount future money. But deferred compensation is also deferred tax and it can be very useful to be able to reduce/avoid a big tax bill.
Actual dollars? Sr Engineer base pay at $175k+. Mid level at 135k. There are plenty of formerly middle class engineers making over $300k a year including equity. People often discount startup equity while overlooking the very lucrative large tech company equity. (Fb, goog, etc). My source for this is both friends and hiring in this market.
To be sure not every eng in sfba is paid this amount. But they probably could be if they were willing to optimize for comp.
> "if you're not giving us everything beyond what we expect... go screw yourself.
I'm not sure how that makes sense. Netflix can't expect employees to surpass their expectations, by definition. I don't see the problem if they're clear about their expectations and send people on their way who either can't or don't wish to fulfill them. I don't know if I have the ability to fulfill their expectations, but I know I have no desire to try and thus no desire to work for them, but I'm fine with other people having different preferences.
It's fairly easy. Business asks for something. Business gets mad when what they asked for was wrong, technology failed, bad luck, or they felt like you didn't do more than what they asked.
I know a handful of people who work for Netflix, and they all love it, although some have commented it's a hire slow / fire fast environment, but once you're out of that initial trial period, it's a great place to work.
The levels are not seniority, they are an approach to learning and programming. Junior A level people are not the same as B level people. Hiring B level people to "just finish" what the A level people have done is an invitation to delays and failure.
The analogy apparently switched from "A" and "B" the grades/marks in school (hence "A for effort", where A is better than B), to something like the "A team" vs "B team", where the A-team is the elite performers and the B is not as good. Also "C-level" was not brought up until you, but it usually means executives. :)
There are really a few outcomes on that: you're going to hire people that will rip each other apart to get the A ranking, your going to get a group of people that are so afraid of screwing up that they're basically neutered, out your going to get everyone cooking the books.
I can't see this requirement on the individual actually doing anything for the team.
This reads less like a technical overview of Netflix's stack and more like an advertisement for working there.
I'll give you my two cents on the matter. Netflix spins a great narrative and ideal -- the rose-colored "meritocratic" dream of Silicon Valley.
I'd suspect the reality of working there is a lot more mundane than the beautiful narrative that they (and by extension, the author of the post) are trying to sell you.
There are probably a wide array of experiences that current employees have, ranging from pure joy to misery, which are contingent upon team, co-workers, management, compensation, etc.
faizshah - posted (and gave you credit) on the original ScaleScale post. Thanks. BinaryRage - The link is there to netflix.github.io on the original post.
Most of the cloud services are JVM-based (as is much of the NetflixOSS ecosystem [1]), and then the Node services connect to the JVM-based services via Prana. [2]
Thanks for the feedback. We heard this a little bit through twitter. Can you reference any links? Happy to make the change. It was a little tricky to put together with out interviewing someone. We want it to be accurate and useful. Please post any reference links here and we'll follow up with an edit.
Someone at Netflix did a talk perhaps it was about Falcor. They use React embedded in browser apps across all devices. So they use Javascript on the front end. They use Node.js as the proxy server / router with a framework called Falcor which is written in Javascript. Falcor queries all the multitudes of services and databases then caches that data in a graph. Falcor reaches out to all the servers and databases, scores or services, that are written in Python and Java.
Picking a nit, they don't use React in the browser across all devices. They actually use an internal rendering engine called Gibbon on some devices with client code written JavaScript with React but not rendering to HTML or a browser instead using their internal engine. Interestingly this has become easier with React breaking out React and ReactDOM.
Interesting to note that Netflix must see Gibbon as a competitive advantage, I've been told its not likely to see an open source release.
> It was a little tricky to put together with out interviewing someone.
Why didn't you just interview someone then? There are plenty of folks there who would happily grant you an interview. If I still worked there I would have.
Confluence is pretty cool, but they suck at bullet lists and indenting code blocks... so basically they suck at a couple of basic UI stuff. Apart from that it seems pretty great.
I am. It does not indent properly though - there is only one level of stable code indentation (the entire block that is; it is possible to indent the code within the block, but I'd like to indent the code frame).
- No "regularly well performing" employees are kept. You're either promoted or fired. Ensure workforce moves.
- Doing the shitty necessary work such as fixing tech-debt just gets you fired. But I bet people do it anyway, else Netflix would be in troubles. It's just that they get fired after that. Not really nice.
It also means you've to be only doing stuff that has direct visible impact, and refuse any work that does not/place yourself visually above others.
Not saying this is a bad strategy to get stuff to work, not sure if very ethical though.
Exactly--in a work environment when everyone is fearing for their livelihood, succeeding at performance theatre becomes the primary concern. You need to be a top performer, in both senses of the word--but it's human nature that over time the acting sense will take priority.
Hi Kylek - thanks for the feedback. It was a little hard on first try to find information on this. The only thing that seemed clear was that adaptive bit rate was done on the edge and transcoding was done and stored. Feel free to post links here if anyone has any.
This is a bit confusing. I don't suspect Netflix is built using primarily node.js, and I don't see how the Netflix Prize (algorithm for predicting what a person is most likely to want to watch) falls under "scaling".
hey, thanks for the feedback lappa. We've updated the original article to display node.js better and added a comment.
The idea behind mentioning the Netflix prize is to show the openness and willingness to get a lift on their existing recommendation algorithm. There was a 10+% lift in accuracy. Their ability to open up the data and implement really gives them a competitive edge IMHO. It's a shame there seems to be some regulatory reasons they don't do it any more.
Honestly, there are enough problems with the article to warrant a critique. However, someone may not want to take the time to outline every single one. It's incredibly confusing to the reader that the article states Netflix relies on frameworks for one language, and then states that all of their code is written in another language.
Hey, we got this fixed up on the original post- waiting for Todd to fix the repost on HighScalability shortly. If there are some more problems let us know. We want it to be accurate. http://www.scalescale.com/the-stack-behind-netflix-scaling/
The reason that strikes me as most important is #2 "Letting Amazon focus on data center infrastructure allows our engineers to focus on building and improving our business."
Too many companies roll their own solutions when they could take advantage of economies of scale like this and let another company dominate the problem.
Obviously there are privacy reasons to own your own data but I don't think they outweigh the performance and efficiency implications.
> Too many companies roll their own solutions when they could take advantage of economies of scale like this and let another company dominate the problem.
Too many companies build their business completely reliant on a single service provider that can/will change focus, increase costs, terminate/change service offerings, or have unreliable service availability.
Because they actually know how to implement fault tolerant scale on AWS. Most companies don't have simian army randomly wrecking infrastructure. Many companies/engineers think they can spin up a few (or few hundred) ec2 instances and think they have solved resiliency.
They are shifting capex to opex. The complexity of solution that can perform OK on aws is significantly higher then complexity of solution that runs on metal. There are use cases for which aws will never be a viable option. AWS in general is a bad option for anyone but Netflix (Netflix consumes over 30% of aws resources if they make a mistake that causes even 30-40% spike all AWS customers are f#$ed)
Why is that surprising? If you're not in the infrastructure business, why would you build infrastructure? It's like criticizing an auto manufacturer for not manufacturing their own tires.
If you mean that AWS prices are too high, I think you're thinking in terms of retail prices. If you have enough volume, AWS will negotiate a volume discount.
Netflix accounts for a huge chunk of the national bandwidth. They have specialized needs when it comes to hosting that no other company on the planet has.
I really like that Netflix is constantly open about how their systems are run. In terms of raw throughput not a lot of companies can hold a candle to them, so its cool to see how the big players do things.
I would wager because, looking at that technology stack, most of it is commercial paid services? "We use EC2" just pushes the question to someone else.
So which part of this stack is to blame for their slow clunky website? (NB their website used to work well enough on my devices the newer one is the problem.)
Over the years, I've heard a few podcasts stating that Netflix relies heavily on Grails/Groovy for infrastructure management tools (https://github.com/Netflix/asgard, etc). Wonder why that isn't mentioned?
They do say: "We are sharing the things we think are most interesting." Maybe a minor use of Groovy and Grails is only interesting to people trying to promote those products.
As the comment at the end of the article mentioned, they forgot to add Spring. I've met a few Netflix engineers, and they said their Java implementation is done using Spring.
No links, I got this from actually talking with some engineers. Have you not contacted a single Netflix engineer to verify your article? If not, I find that bizarre.
Why there's so much brag about Netflix while they can't fix even simple bugs and UX issues? After all, it's one of the simplest apps there are. Whole stack could be handled by one guy.
Curious what you mean by this, as it's overall one of the more polished products I use on a regular basis. Hasn't always been the case, and the Android app still does some weird UI things, but generally the product is solid and a pleasure to use.
"Whole stack could be handled by one guy" - comments like like this and similar others ("I could build Twitter in a weekend", etc.) strike me as unproductive.
For example, Netflix consistently forgets what I have seen and if I don't watch the ending credits it thinks I haven't watched an item and suggests it in the continue watching section. There should be visual indicator on what I have watched, and if there are unwatched episodes in series. Now I have to open it and go to the episode list, which is unreliable. Also, starting video playback is too slow. It's not possible to open items in new tabs. There are no keyboard controls.
The headline of the slide you pulled that from is literally "Hard Work—Not Relevant."
Netflix doesn't care if you work 100 hours a week, they care how well you perform. If you can do an amazing job in 30 hours a week, they're one of the few places which would actually embrace you.
> I'm not sure how Netflix grades things... but in school B is "Above average".
So what? Why on Earth should Netflix be targeting "average" workers when they deliberately pay twice the average.
That's about as ridiculous as saying that Harvard should stop admitting A students because "B" is average.
Netflix has a deliberate strategy of keeping their engineering bar very high and paying accordingly. That doesn't mean they want you to sacrifice your life to the company.
Also, Netflix is in no way a startup.