>they didn't know how to hire system administrators, but they knew how to hire software engineers
Setting aside your partial jest, I think Google's approach of hiring "programmers" to fill the role of SRE makes perfect sense for their organization.
The "sysadmin" can have a wide range of meanings and responsibilities but many will have experience managing COTS[1] software like Oracle RDBMS, Peoplesoft HR, MS Exchange email server, etc. A typical system admin can then use complimentary COTS tools like Microsoft SCOM System Center Operations Manager, HP OpenView, IBM Tivoli, CA Unicenter/Spectrum. On Linux, it would be similar tools like Nagios, Zabbix to monitor Apache webservers, Mysql, etc.
In Google's case, they didn't cluster a bunch of COTS Oracle dbs together to serve web surfers. Instead, Google wrote a proprietary platform like BigTable or Spanner[2]. As a result, they also want additional software that monitors the health of Spanner. Since none of the monitoring tools from enterprise vendors like HP/MS/CA are adequate (because they have no out-of-box software-agents for Spanner), they need another layer of programmers to write custom management/monitoring tools. Those programmers are the SREs.
All the above enterprise tools from HP/MS/CA/etc including the mother db server, the agents, the plugins are written in C/C++. If your hiring job description is "system admin", most candidates will not have the skills to develop those enterprise monitoring tools from scratch.
Yes, many sysadmins have programming skills to write Bash/Powershell scripts. E.g. a bash script might have a polling loop that checks for an error file existence and then has if/else/fi to send an email. However, that's not the same programming skill as developing a proprietary version of HP OpenView and MS SCOM. Most traditional sys admins do not write low-level C code like Zabbix.[3]
Basically, if you're a company that writes complex proprietary softare and none of the existing enterprise monitoring tools can manage it, you'll have to hire programmers instead of sysadmins as the baseline skillset for SREs.
Most sysadmins on the market primarily have worked in manners and tooling hardly different between cost and revenue centers, and for Google they needed admins that are actually knowledgeable about software engineering to make them effective in a revenue center.
It's laughable what kind of tools were written for enterprise systems back in the late 90s, so it's no surprise that absolutely none of them would meet Google's rate of growth requirements either.
Furthermore, given the licensing patterns of most enterprise software suites, it'd potentially become cost prohibitive to deploy COTS ops software at Google scale anyway even with Google scale money.
There's obviously still some cost centers to big tech companies (Facebook has contract positions open for doing service desk customizations last I checked) but custom tooling everywhere makes sense when you have justifiable reasons that nobody will be able to serve your needs besides yourself.
Makes me wonder what they pay and how they get over the usual SV prejudice against hiring people over 30. I read their SRE description a while ago and thought it sounded perfect for me, but if you end up there and for whatever reason it doesn't work out you'll end up losing a lot of money leaving SV again. It's not cheap to uproot yourself and move across country and it's not like anyone else in SV will hire folks over a certain age; at least that's what I'm left to conclude after lurking HN for a few years.
Eh...what? Don't be fooled by what you read here. Take a few interviews and find out for real if that's the case. I would be really surprised if that were the case, especially at Big Co's where their staff is (comparatively) older than at Startups.
Setting aside your partial jest, I think Google's approach of hiring "programmers" to fill the role of SRE makes perfect sense for their organization.
The "sysadmin" can have a wide range of meanings and responsibilities but many will have experience managing COTS[1] software like Oracle RDBMS, Peoplesoft HR, MS Exchange email server, etc. A typical system admin can then use complimentary COTS tools like Microsoft SCOM System Center Operations Manager, HP OpenView, IBM Tivoli, CA Unicenter/Spectrum. On Linux, it would be similar tools like Nagios, Zabbix to monitor Apache webservers, Mysql, etc.
In Google's case, they didn't cluster a bunch of COTS Oracle dbs together to serve web surfers. Instead, Google wrote a proprietary platform like BigTable or Spanner[2]. As a result, they also want additional software that monitors the health of Spanner. Since none of the monitoring tools from enterprise vendors like HP/MS/CA are adequate (because they have no out-of-box software-agents for Spanner), they need another layer of programmers to write custom management/monitoring tools. Those programmers are the SREs.
All the above enterprise tools from HP/MS/CA/etc including the mother db server, the agents, the plugins are written in C/C++. If your hiring job description is "system admin", most candidates will not have the skills to develop those enterprise monitoring tools from scratch. Yes, many sysadmins have programming skills to write Bash/Powershell scripts. E.g. a bash script might have a polling loop that checks for an error file existence and then has if/else/fi to send an email. However, that's not the same programming skill as developing a proprietary version of HP OpenView and MS SCOM. Most traditional sys admins do not write low-level C code like Zabbix.[3]
Basically, if you're a company that writes complex proprietary softare and none of the existing enterprise monitoring tools can manage it, you'll have to hire programmers instead of sysadmins as the baseline skillset for SREs.
[1] https://en.wikipedia.org/wiki/Commercial_off-the-shelf
[2] https://en.wikipedia.org/wiki/Spanner_(database)
[3] e.g. https://github.com/zabbix/zabbix/blob/trunk/src/zabbix_serve...