This is good advice -- but now that AWS has so many services, it's an extremely complex question. There are path dependencies: adopting an AWS service today may accelerate your product sufficiently to get it to the point of traction and sustainability, where it would have failed otherwise.
I think it's important to have a good framework for making these decisions. As you mentioned, EC2 and S3 are on one end of the spectrum: low switching costs and coupling, but also low in terms of differentiation at this point. Whereas other AWS services that are more bleeding edge could actually be the foundation of a temporary competitive advantage for your product, but today would introduce deep coupling of your service with AWS.
The trick is basically landing somewhere where you've made good bets: the ideal scenario is that you have a good chance of having an escape plan if/when the situation warrants it, while not missing out on the opportunity to take advantage of the services that your competitors (like, for example, those who are on the "EC2/S3 only" methodology you mention) may be forced to build themselves out of fear.
Unfortunately the best way to do this is to be able to predict the future: which services on AWS have high value and high longevity, and will have similar, cheap offerings on their competitors and/or via OSS in the coming years? S3 is an example of a good bet: if you adopted S3 when it first arrived, it may have felt like unnecessary couping with AWS, but it was a huge accelerant if your business relied upon large, reliable file storage. (In fact many startups likely today exist because they were able to bootstrap off of S3.) S3 was a good, long term bet -- now, you can drop in an OSS replacement that is API-compatible in a worst-case scenario. Other services on AWS have come-and-gone through their hayday without going through that transition, because they were either (in retrospect) transitional technology or basically "the wrong thing." So, it comes down to predicting :)
A couple of things that increase probability an AWS service is more destined for long term success + commoditzation (in my view):
- Built on open protocols/standards
- Simplicity in APIs and feature set
- Versatility of use-cases
- Deeper in the stack, vs user-facing
- No existing open source alternative with traction
- Backchannel confirmation that Amazon itself is using it :)
For example, DynamoDB when it first came out was a huge "no" for me personally -- it was a very quirky design, with complex, bespoke concepts around indexing, with lots of good open source projects offering alternatives, and I heard Amazon didn't trust it internally :) Whereas other services like Redshift ticked all the right boxes out of the gate. So there's some hope of making good bets here, but sometimes you do need to throw caution to the wind if something extremely compelling comes along that feels like it could be a shot-in-the-arm to your product despite not having most of these attributes.
Also, FWIW since I like calling my shots: I think the current iteration of AWS server-less tech (while a necessary first step towards understanding the design space, and a technical marvel imho) is probably not a good long term bet as a foundational technology for a product you would like to de-risk in this aspect, but we'll see :)
Some great points in your blog-post-length comment ;)
I'd add that there's often a danger in the middle ground. Complexity and meaningful switching costs, without the benefit of interop, is likely the worst-case scenario. IOW, might be better to go all-in than muddle through with half-measures trying to avoid lock-in.
The long-term issue with adopting Amazon is pricing power--once you are committed AWS has it and you don't.
Take your Redshift example. It's an outstanding service but once you build apps around it and load a bunch of data the switching costs become very high. AWS can raise the price quite a bit before it makes economic sense for working apps to move off it.
Consequently a key question to ask is when do my favorite AWS services become cash cows? At that point your economics will change substantially. You may be living day-to-day in your business but the stock market thinks Amazon will extract large profits out of AWS in spite of growing competition. Food for thought...
AWS is a growth business, in a competitive space, so (particularly given their overall history) it seems unlikely they will increase margins if it could risk adoption.
Of course, in the "long term" (whatever that means), the market and shareholders expect to reap vast profits, and price that, along with the company's free cash flows, into the stock.
However, my personal thesis is that Amazon specifically is a bit of an anomaly in this regard: it seems unlikely to me that we will see any strategic shift in pricing from basically any Amazon-owned entity in our lifetimes -- the culture is too ingrained towards a strategy of complete and utter dominance over all markets, margin reduction being a key tool in the toolbox. So if I were to bet I would say pricing power is pretty low on the list of concerns of AWS lock in -- more concerning are things like unexpected conflicts of interest as Amazon eats the world (see also: Netflix), operational dependency (ultimately, you rely upon AWS ops' competency for your uptime, security, etc), end-of-lifing of services you depend on, opportunity cost vs other vendors who may offer better services, and other unknown unknowns.
Certainly there are other concerns--no argument there.
That said, Amazon Prime prices just went up 20%, so Amazon will clearly raise prices if they think the market will bear it.
Moreover, "complete and utter dominance" sounds like a monopoly. It's difficult to think of a monopoly that didn't raise prices and/or decrease service once they were established. I would not start a business on Amazon without assessing the risk of tying my fate to a single large vendor, for cost as well as other reasons you cited like Amazon deciding to compete with you.
Take your Redshift example. It's an outstanding service but once you build apps around it and load a bunch of data the switching costs become very high. AWS can raise the price quite a bit before it makes economic sense for working apps to move off it.
That's a bad example. Redshift is "wire compatible" with Postgres. You use the same drivers. There are a few popular extensions they added to Sql to copy files directly from and to S3 but that's about it.
Redshift is a data warehouse. It does not require indexes and handles wide tables--e.g., fact tables with lots of columns--very efficiently since it uses column storage. For data warehouse operations you can only substitute native PostgreSQL if the dataset is quite small.
Assuming you seek wire compatibility, your choice would therefore be a PostgreSQL-compliant DW, of which there are several. However, they have greatly differing cost and operational profiles, which make switching non-trivial. They also tend to support different versions of PG depending on when they forked from PostgreSQL.
Yes, if for some reason you want to leave RedShift, you will lose the benefits that Redshift provides and you will lose the speed. But you shouldn't have to change your program.
I'm not saying it is a good idea to leave Redshift in some distant future where you might save a few pennies. Heck, I hardly ever say it's a good idea to rearchitect a core part of your infrastructure without having a very good reason -- saving a few dollars isn't one. But from the software side, you aren't stuck with Redshift and you aren't embedding a lot of proprietary AWS specific drivers in your code base.
> There are path dependencies: adopting an AWS service today may accelerate your product sufficiently to get it to the point of traction and sustainability, where it would have failed otherwise.
You can say the same thing about a loan shark or high interest loan. Just because you can doesn’t mean you should.
And you can say the same thing about mortgages and corporate financing -- sometimes you should. What is your point? Did you even read my post or just hit the "reply" button after reading the second sentence?
I think it's important to have a good framework for making these decisions. As you mentioned, EC2 and S3 are on one end of the spectrum: low switching costs and coupling, but also low in terms of differentiation at this point. Whereas other AWS services that are more bleeding edge could actually be the foundation of a temporary competitive advantage for your product, but today would introduce deep coupling of your service with AWS.
The trick is basically landing somewhere where you've made good bets: the ideal scenario is that you have a good chance of having an escape plan if/when the situation warrants it, while not missing out on the opportunity to take advantage of the services that your competitors (like, for example, those who are on the "EC2/S3 only" methodology you mention) may be forced to build themselves out of fear.
Unfortunately the best way to do this is to be able to predict the future: which services on AWS have high value and high longevity, and will have similar, cheap offerings on their competitors and/or via OSS in the coming years? S3 is an example of a good bet: if you adopted S3 when it first arrived, it may have felt like unnecessary couping with AWS, but it was a huge accelerant if your business relied upon large, reliable file storage. (In fact many startups likely today exist because they were able to bootstrap off of S3.) S3 was a good, long term bet -- now, you can drop in an OSS replacement that is API-compatible in a worst-case scenario. Other services on AWS have come-and-gone through their hayday without going through that transition, because they were either (in retrospect) transitional technology or basically "the wrong thing." So, it comes down to predicting :)
A couple of things that increase probability an AWS service is more destined for long term success + commoditzation (in my view):
- Built on open protocols/standards
- Simplicity in APIs and feature set
- Versatility of use-cases
- Deeper in the stack, vs user-facing
- No existing open source alternative with traction
- Backchannel confirmation that Amazon itself is using it :)
For example, DynamoDB when it first came out was a huge "no" for me personally -- it was a very quirky design, with complex, bespoke concepts around indexing, with lots of good open source projects offering alternatives, and I heard Amazon didn't trust it internally :) Whereas other services like Redshift ticked all the right boxes out of the gate. So there's some hope of making good bets here, but sometimes you do need to throw caution to the wind if something extremely compelling comes along that feels like it could be a shot-in-the-arm to your product despite not having most of these attributes.
Also, FWIW since I like calling my shots: I think the current iteration of AWS server-less tech (while a necessary first step towards understanding the design space, and a technical marvel imho) is probably not a good long term bet as a foundational technology for a product you would like to de-risk in this aspect, but we'll see :)