Releases with Elixir >= 1.9 are now quite good. Run mix release.init, write some configuration, then just mix release and you get a tarball ready to deploy.

We use a multistage Dockerfile to build the release and run it on a barebone alpine image, and it's never given me a single issue since I set this system up almost 2 years ago when Distillery was still the release library of choice.

And it's never crashed once, but that's the magic of OTP :-)

That's the approach that I take as well most of the time. Makes it pretty seamless to then take that container and run it in your cloud provider of choice. I put together a blog post outlining the use of Docker + Mix Releases for a Phoenix app https://akoutmos.com/post/multipart-docker-and-elixir-1.9-re... to help people get started.

The one thing that could really push it into "dream language" territory is a sound type system. The Gleam language recently popped up as an up and coming solution to that... It's ReasonML/Javascript like syntax makes it familiar and has a good interop story.

Yeah, it looks exciting.

I'd be interested in hearing more about this. I'm keen on learning Elixir, but have fallen back to Rails at least 2 times when starting something fresh, because I'm very comfortable with Rails ops. It's simple enough for my case and rails console is just a blessing to dig around when something weird is going on.

The hardest thing for me to get was that Elixir/Phoenix is not Ruby/Rails. They are very different, while syntax is very similar, Elixir favours explicitness unlike Ruby. This leads to a few more lines of code than in Rails. But in a complicated cases, Elixir's code becomes shorter and much easier to dig in.

Simple case: Adding a new column to a table Phoenix: 1. Migration 2. update schema 3. update changeset Rails: 1. Migration

Complex case: Add a new column with validation only on update Phoenix: 1. Migration 2. update schema 3. update changeset Rails: 1. Migration 2. add validation with only on update 3. pray it won't break anything in other parts of app :/

The complex case sounds like a simple one, but in a big codebase adding validations could be troublesome for a fee reasons, but let's not dig into them. In Phoenix you can have multiple changeset definitions with different validations for each of them, which makes it much easier to control when and what data is validated.

It's just one of the examples, there are many others...

I'd say, if you are content with Rails, it's great then! No reason to change a technology stack, because some random person is saying it's worth it. If you found a good enough solution for your problems, stick to it :)

It's mostly two things:

1. The stuff that never goes down (gets automatically restarted with last-known good state) and the amazing runtime in general. Not to mention it's much faster than Ruby or Python or PHP... In some cases it's faster than Javascript although JS and Erlang/Elixir are, on average, tied on performance.

2. A switch in philosophy and the way you troubleshoot your app: from magic/implicit to explicit. Many people cannot cope with this one because the force of habit is extremely hard to overcome. I for one enjoy knowing what exactly is going on in my app and exactly where but many others disagree. Again, it's a philosophical thing; you either are into it or you are not. There is also the objectively technological aspect: zero magic and implicitness mean much easier bug fighting. You no longer have to dig around in callbacks and installed on-start handlers.

Make no mistake: it's harder to start with an FP language. Nobody is disputing that. The benefits to be sown after the initial culture and brain shocks however are a lot.

> it's harder to start with an FP language. Nobody is disputing that.

I'll dispute it. For me, the functional paradigm clicked much easier than OOP and it's zillion abstrations and patterns. I'd argue that the best thing about functional programming is it's actually easier to reason about.

Eventually, yes. But if you were immersed in OOP/imperative languages for 14-15 years as I was (before learning Elixir) then it is really hard to muscle through the exercises and learn to think the FP way.

After that happened however, FP became much easier and much more pleasant to work with.

agreed. I'd been going steadily more fp in my style for years while working in c/c++, php, C#, scala,coffeescript etc. before finally going all in with elixir for the last few years. It makes many, but of course not all, things more straight forward compared to imperative programming.

> Not to mention it's much faster than Ruby or Python or PHP... In some cases it's faster than Javascript although JS and Erlang/Elixir are, on average, tied on performance.

By what metric? Phoenix on Elixir doesn't perform any better than Ruby and a similarly lightweight framework like Roda: https://www.techempower.com/benchmarks/#section=data-r18&hw=...

NodeJS is also much faster than Elixir in other circumstances thanks to the V8 JIT.

Elixir's and Erlang's strengths over other dynamic languages aren't really performance. It's mostly just that Phoenix performs much better than Rails.

By the metric of the average Phoenix request taking between 0.3ms and 10ms as opposed to 100% the same app in Rails never responding for less than 230ms -- an app that I rewrote to Elixir/Phoenix from scratch over the course of months.

You are correct in your specification though; I should have said Phoenix in particular and web development in general, not Elixir itself. The BEAM languages aren't excellent in raw performance, that much is true.

Also I am yet to be impressed by JS web apps. They are quick when you spin them off on your local machine or on a world-unknown website that only you and 3 buddies visit. But I've seen NodeJS apps crumble at 10 reqs/sec routinely. I'll grant you that this was 2 years ago though.

---

In short, I am not claiming to have made a scientific and thorough research in ideal conditions. I did work with PHP and Ruby for a long time though (and have been partially exposed to Python), and the difference in performance and reliability between them and Elixir is night and day. I've seen an Elixir app with a rudimentary 20-tables data schema endure 1000 reqs/sec on a RPi 3B.

---

Final note: I'd like to see TechEmpower's benchmarks based off of a more real-world app like https://github.com/gothinkster/realworld as well though.

I can only speak from my own experience but here is a data point (note that I do not use Elixir only, but also other technologies, in production).

For instance, I implemented a HTTP "business proxy" last year, which needs to be able to keep a large number of concurrent connections & proxy the call to a slow responding remote server.

I made multiple implementations (including Go, Node, Ruby with Roda, JRuby, Elixir, Crystal).

In this specific case I had, Elixir throughput & stability was by far the best. The gap (under load) between min & max was also the best (all this without tuning), and so was the CPU use.

Of course, YMMV, make your own experiences!

I'd be extremely interested if you open sourced all those repositories. I thought of doing something very similar and your shared experience would be invaluable.

I know (you are not the first to ask!) but honestly, sorry, I won't and I will let anyone truly interested to do their own experiments :-)

This type of topic is too prone to endless debates on the validity of this and that, and the fact that anyone proficient in whichever stack I tested could definitely improve it (including me on the subpart of those stacks which I'm most proficient with!).

I don't want to lose any energy with this, and I have made my decision for that business case (in this case, a very happy one).

That's fair and I respect your lack of desire to spend a lot of time on minutiae. Sure.

That being said, you can always release the repos and just archive them right away -- thus disallowing PRs, issues and any public commentary in general. And whoever wants to dabble further can just fork and refine them.

It's your call of course. But this is what I would do I weren't interested in further changing the repositories.

I don't have any obligation here! I'm just sharing my own little experiment, which you are free to disregard.

While providing little details, it is actually providing value to some here.

In terms of measuring success & picking stacks, I have also a good experience (after 35 years of coding - I hate to mention that and this is no "call to authority", but I've seen a fair bit of stacks), and Elixir is a good investment at the moment for my use-cases.

I'll pile on here with my experience. I took a fairly complex Python/Django app and rewrote it into Elixir/Phoenix (and ripped out nearly all the complex caching along the way).

The Elixir/Phoenix app rewrite was a little more than 10x faster, much more consistent in response times, and needed far fewer resources to run.

There's no secret sauce in Elixir which isn't in Python to make it 10x faster: https://gist.github.com/jamatthews/7b41b5ce5058f3f4ed674b6ac...

Elixir and BEAM are clearly enormously better at some tasks but general performance is actually very similar between Elixir, Ruby, and Python if you use similarly performant frameworks.

Ok, but why did my app get 10 times faster? Why do so many other people report similar results? Perhaps factorial benchmarks don't translate well to the comparison of complex real world systems.

I think, when it comes to web apps specifically, there are a few advantages that make a practical difference. These seem to be the major ones:

- Compiled templates and more efficient string interpolation

- Improved parallelism and lighter weight processes. This leads to better handling of many requests in parallel (esp. while most are just waiting on IO), better db connection pooling, less blocking, etc.

A few other ideas are mentioned here by someone who seems to know more about this than I do: https://news.ycombinator.com/item?id=12014243

> Ok, but why did my app get 10 times faster?

You rewrote it.

The language you rewrote it in probably has less to do with it than rebuilding all at once what initially was built over time/organically.

> You rewrote it. The language you rewrote it in probably has less to do with it than rebuilding all at once what initially was built over time/organically.

That's a very unfair blanket statement.

There are no ways to reduce the inherent flaws of mutable imperative languages no matter how much times you might rewrite the apps in the same language. You definitely can reduce bug surface but not by much; shared data is much more prone to bugs and human mistakes than a system with a runtime that puts immutable data and message-passing front and center.

I'll give you that rewrites do improve state of any random app. But I think you are overestimating this factor with your pretty broad and somewhat dismissive statement.

Yeah that’s probably the case for most rewrites.

But in this case the django app was a full rewrite of a php app. It’s quite similar to the eventual elixir app.

I feel it’s pretty much a fair fight, but you’ll have to take my word for it.

As another poster said, factorial calculations are hardly the dynamic languages' forte.

The measured workflows in terms of web frameworks include a lot of other elements and none of them are CPU-operations-per-watt (namely pure muscle).

Erlang's (and thus Elixir's) runtime shines particularly well in web apps' typical workflow, not on tasks where C++ likely is still king.

How else do you explain a 23-230x response time decrease? Either it’s faster or it’s not.

I’ve just shown you that it’s not faster. Benchmark Roda against Phoenix and you’ll see exactly the same is also true for the web workloads.

There is no magic.

There's no magic of course. The difference in real world apps that many people in this thread claim are in favour of Phoenix comes from the responsiveness and robustness of Erlang's OTP itself.

(I'll again reiterate that I rewrote a Rails app from scratch to Phoenix and thus accelerated it at least by a factor of 20x. I am just a working guy and have no vested interest in spreading misinformation.)

A lot of web frameworks crumble or introduce abhorrent lags after a certain threshold of pressure. Elixir/Phoenix in comparison almost don't change their average response time until much later.

Truthfully, after reviewing the TechEmpower repo for the last day and a half, I can only see their Fortunes benchmarks as being half-adequate. Most of their benchmarks are extremely unrealistic in terms of web apps usage. At this point I am not even sure what their goal is anymore.

It is definitely way too simplistic to compare two languages based on factorial (also, fwiw, the native annotation is not doing anything in your example unless the Enum module and its dependencies are natively compiled too).*

Elixir (Erlang rather) does have some "secret sauce" that makes it performant on part of the web framework space. For example, the literal pool used by the compiler+VM alongside iolists+`writev`-based operations make it so most template renderings in Phoenix allocate as little memory as possible. This means that any literal string in your template is allocated once on boot and not per-rendering, and building the output buffer of the template engine does not concatenate strings either (which would generate even more garbage).

These two blog posts have a bit more detail:

  * https://www.bignerdranch.com/blog/elixir-and-io-lists-part-1-building-output-efficiently/
  * https://www.bignerdranch.com/blog/elixir-and-io-lists-part-2-io-lists-in-phoenix/

I believe Ruby started to gain some of these benefits with the frozen-string literal annotations. So you can opt-in and no longer allocate static strings on every render. But afaik the template engine still performs a reasonable amount of string concatenation at the static/dynamic boundary. It would also likely be possible to eliminate it by implementing something akin iolists/writev. Maybe it has already been done, I haven't kept up.

So in a sense, everything is doable in any of them, but in Erlang this is the default way to do it, so it pushes towards an efficient approach to work with I/O from day 1. Case in point: I didn't invent any of this, I just learned what was there. But you won't see those differences unless you are working on an application that is rendering medium to large-sized templates (which are most apps returning HTML) and benchmarks don't tend to exercise that.

Other benefits from Erlang that may shine in the web space is the per-process garbage collection. It may reduce the variance on the latency and for short-lived requests, you may not perform garbage collection at all. All the VM does is to reclaim the space once the request is over.

At the same time, there are benefits in Ruby and Python you won't find in Elixir. For example, if you need an algorithm that relies heavily in mutability, then Ruby and Python will definitely have an edge. Discord recently had an example of where they made an algorithm much faster by moving part of it to Rust to leverage mutability. But benchmarks don't tend to exercise that either.

In fact, benchmarks may highlight non-ideal behaviour. For fully IO-based exercises, I got faster results by running an acceptor per thread that reads+writes as fast as possible, rather than multiplexing requests on all cores. But in practice, most technologies that can leverage multi-core will prefer to multiplex because that will be beneficial as soon as you do any subsequent I/O or CPU work.

TL;DR - sure Elixir can be 10x faster in some workloads, sure Python can be 10x faster in others. But of course, it would be incorrect to use any of these results to say Elixir or Python is 10x faster than the other.

*Fun fact: I am working on some VM trickery that makes things like Fib/Fac 3-4x faster but at the moment the trickery fails to show any benefit on any slightly more complex code. If they were to accept it, your factorial benchmark would show something much faster for Elixir, but in practice very misleading. :)

> Other benefits from Erlang that may shine in the web space is the per-process garbage collection. It may reduce the variance on the latency and for short-lived requests, you may not perform garbage collection at all.

This one was a big draw for me. It was a major reason I got interested in Elixir in the first place instead of going the Scala/Akka route.

Thanks for the very informative reply!

It's already the standard way in Ruby, even in Rails. Jeremy Evans did the template optimization you're describing using frozen strings 3+ years ago in Erubi and it was adopted by Rails just a few months later.

There's no long STW GC phase during requests in modern CRuby either. It's nowhere near as elegant as BEAM's heap-per-process design but the GC issues seen in CRuby < 2.2 have been gone for almost 5 years now.

GitHub retired OOBGC last year. Pause times are just 1-2ms and total time spent in GC is 0.5% on for the main application I work on.

Are you talking about generating better bytecode based on the semantics of Elixir or a bytecode optimization pass?

> It's already the standard way in Ruby, even in Rails. Jeremy Evans did the template optimization you're describing using frozen strings 3+ years ago in Erubi and it was adopted by Rails just a few months later.

Yes, I was aware of the frozen strings optimization. Thanks for confirming! The part I am not fully aware though is in regards to the output buffer, which is a separate optimization. Is the output of a template being stored in an array that is writev-ed to the socket? Or is a template still rendered to a large string? Or maybe something in the middle?

If you don't mind me asking, what is OOBC? My Google-fu failed me. In any case, 0.5% is quite good, although that is also dependent on how much garbage the app generates. And as you know some libs/frameworks tend to be more mindful of that than others. :)

> Are you talking about generating better bytecode based on the semantics of Elixir or a bytecode optimization pass?

Do you mean the VM optimizations I was working on? I was mostly AOT compiling the bytecode of a function. That removes the VM dispatch overhead which further helps with CPU speculation. I see very good results on recursive functions but not much beyond that.

It allocates a single "buffer" string. IO.write on Ruby does support using writev but the fairly performance difference hasn't motivated anyone to use it for template rendering yet.

Ahhh sorry I meant Out Of Band GC.

> Do you mean the VM optimizations I was working on? I was mostly AOT compiling the bytecode of a function. That removes the VM dispatch overhead which further helps with CPU speculation. I see very good results on recursive functions but not much beyond that.

It's extremely difficult to get a good return on this approach for two main reasons: instruction dispatch overhead is a very small part of the VM overhead for VMs like BEAM or YARV https://www.sba-research.org/wp-content/uploads/publications... and Haswell onwards have no problems branch predicting the dispatch loop https://hal.inria.fr/hal-01100647/document

Sadly this is also why the CRuby 2.6 JIT isn't very effective either.

V8 removed their baseline JIT because it wasn't worth it. Firefox is considering doing the same. JSC has a more complex but effective baseline JIT. Implementing one for BEAM would be quite a lot of work compared to a minor performance improvement.

Using a string buffer wouldn't mean that the different dynamic and static parts of the template still have to be copied? Imagine this template:

    foo
    <%= expr_that_returns_hello() %>
    bar

Imagine it builds its content like this:

    output_buffer = ""
    output_buffer << "foo"
    output_buffer << expr_that_returns_hello()
    output_buffer << "bar"

With frozen strings, it means "foo" and "bar" won't have to be instantiated on every render, but they are still copied when written to the buffer (unless the buffer uses memory references to frozen strings, akin to a rope).

Now imagine the output_buffer is an array, it means less copying, but support needs to be added upstream and other places to handle them. You would also need writev support for arrays (or at least convert them to an iovec or similar).

Does this make sense or am I missing something obvious?

Btw, sorry for all of the questions and thanks for the linked papers. Now I have something to read on the weekend. :)

TruffleRuby uses ropes to do exactly this. It can even optimize away the buffer string instance: https://chrisseaton.com/truffleruby/ropes-manlang.pdf

CRuby makes a bunch of memcpy calls but compared to everything else, they're fast enough they don't seem to matter much.

I've run perf on CRuby running a bunch of different web app type tasks (even rendering huge JSON payloads) and it's never shown up. You can see an example here: https://github.com/k0kubun/railsbench#perf

Perf output for when running Roda looks similar but vm_exec_core and gc_sweep_step are even more of a bottleneck.

To be honest, I think you are underplaying the value of the sharing optimization (be it via ropes, array buffers, etc) given it can considerably reduce the memory usage (which may put less pressure on the GC).

This is because sharing reduces the memory usage in more than half. Without sharing (but with frozen strings), the memory allocation is `1 * static_size + 2 * dynamic_size`. I.e. the cost is each static part, which is copied, and twice the dynamic size, which is created and then copied (I am assuming that most of the dynamic content is allocated on the request life-cycle).

With sharing, the cost becomes `0 * static_size + 1 * dynamic_size`. Eliminating the static allocation is particularly important in loops. If you have something like:

    <% for  ... %>
      pre
      <%= dynamic %>
      post
    <% end %>

The factor of `n` applies to both static and dynamic, so you have `n * static_size + 2 * n * dynamic_size` when not sharing but only `n * dynamic_size` when sharing. So you can completely remove a linear component from the equation.

You could say that if you use an array or rope, there is still an allocation as the rope or array grows, but that is generally much smaller than the strings being concatenated. It can be preallocated too if you compute the amount of static and dynamic parts when the template is compiled.

So while memcpy is fast enough that it wouldn't show during profiling (except maybe by increased GC times), you should see measurable benefits in terms of memory usage.

Unfortunately I cannot prove this in practice in the context of Ruby, so I will gladly accept if you think I am overplaying the value of the optimization. :)

EDIT: taking a glance at the TruffleRuby rope paper, it seems templating with erb is around 3x faster once they moved to ropes.

Thanks for the discussion! I'll try implementing it and see if it's faster. It might be the case that with mutable strings in Ruby it's only faster on really big templates? I think pretty much everything required is in place and it's just a case of connecting stuff together with an option to use an array, except for implementing writev for CRuby which also shouldn't be too hard. It can even be an extension for now.

By every real production load we know. Techempower have not been representative of anything. The only reason elixir and phoenix is not higher is because their code is buggy and they discouraged the community to try to fix it.

Can you give us an example of a performance improvement which was rejected which didn't break the rules? There were performance improvements merged just days ago with input from Jose Valim.

i did not say it was rejected. I said the community lost the momentum to work on it. Mostly because it felt nearly impossible to get return on perf. When people had momentum to do it last time, we were never able to get perf numbers and to merge fixes before the official run. This basically killed most of the interest by long time members of the community to engage with the benchmark.

I am ofc not saying it is a good thing from the elixir community. Just that it is the sad state of the momentum to change it right now.

Q: Do you guys allow DB connection pools be changed? I know for a fact from my production experience that a small enough number for a DB connection pool in a Phoenix project can cripple its performance due to a lot of requests waiting for the pool.

I don't work for TE but this is all covered in their documentation, which is great! https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Proj...

DB pool size is very subjective much dependent on how long a connection is held without making a query. A recent PR for Phoenix significantly improved performance by _decreasing_ the pool size: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5153

Oh well, 256 was too much anyway!

I meant more like, people were using pools of 10 on very beefy servers and were surprised at bad performance. I agree with the PR author that the DB pool should be anywhere from one to double the CPU cores.

Thanks for the pointer. Currently reading the docs. Wonder if I can contribute to another language (not just Elixir).

Beam thread handling is vastly different than node, I'd be deeply shocked to see node handle as well at load on the same hardware as a similar configured elixir/erlang stack.

I don't see why a functional language would be any harder to start with. It might even be easier.

Sorry, I wasn't clear. I mostly meant when you were an entrenched OOP/imperative languages veteran coming to FP.

The main thing that was frustrating for me early on, especially when digging around in the iex console, was immutability.

> The only thing that is a bit of pain is a good release support, but that has been changing for better.

The pain in Elixir releases I've seen has nearly always been self-inflicted. You can just run "mix phx.server" on the server and have basically an identical deployment experience as if you were running Rails, Django or similar.

Releases are an option that brings some power you don't get in other languages (particularly if you're using Distillery, which makes hot upgrades possible).

I wrote a guide on Elixir deployment that covers a way to use the new mix releases and deploy to Render in minutes. It also links to guides covering a number of other patterns, including the one I currently use for Alchemist Camp. It might be useful for others feeling some of the pain you did in the past!

https://alchemist.camp/articles/elixir-releases-deployment-r...

> The only thing that is a bit of pain

Or, finding developers.

Eh, this is a 50/50.

I actually completely agree that Elixir developers aren't many. What I observed however is that people who want programmers that know FP and can apply it to real world problems where teams of web devs can be reduced from 20 to 3... get treated like coding monkeys and not given much respect and creative freedom.

You can't replace a pile of clueless juniors banging their heads at the latest Node fashion with several experienced craftsmen and treat them exactly the same. It just doesn't work -- people get discouraged and leave. I've had that happen to three acquaintances of mine (and myself, although I don't dare including myself because, you know, our ego gets in the way of properly gauging our own competence).

I've been rejected with the generic "poor cultural fit" excuse enough times on interviews to realise that most managers and shareholders simply want soldiers blindly obeying orders and not captains that can successfully make a team pull through a tough challenge on their own terms as long as schedules and business priorities are respected.

So, to me, the problem is also in the attitude of the shot-callers -- not only in the scarcity or quality of the Elixir devs.

Not very helpful.

Interestingly, coming to Elixir from another dynamically typed language (Python), I tend to do much fewer runtime/typing errors than ever before. I might be more experienced, or the tooling and compiler are good enough to catch the most common bugs.

I get a few "no matching clause" with my data ingestion tool on my Sentry log because external APIs are flaky and a cosmic ray flipped a bit so that I'm reading incorrect data. A GenServer somewhere crashes, restarts and the data comes back clean, without any user disruption at all.

As you might have guess, absolutely not my experience at all. But as explained otherwise, I'm in situation where "bad" data just keep coming in (and "bad" just mean "data that exercises a path were a runtime error will happen because of a typo that the compiler did not catch.) So "let it crash" means "let it crash again", and I don't see the obvious solution...

But again, I'm completely ready to hear that it's just my setup, or how we failed to build the app.

Just wanted to confirm that it's completely possible to shoot yourself in the foot, which I'm surprised hardly anyone mention when talking about elixir.

Anyone interested should give it a try.

I think in this case having bad input is part of your business logic and should be treated as such. This is very easy in Elixir, specially since you can overload a function using pattern matching and so you can create a function (defined after all others) that matches anything. This is basically your error case.

"Let it crash" is commonly misunderstood. It doesn't mean that you shouldn't treat the known unknown cases of your business (like bad input), it means errors that are generally not known a prior (a database crash or network partition, for example) does not need to have a treatment since the Elixir/Erlang will simple crash and restart.

Things like K8s also brings a similar thing that Elixir brings to you, however having this in language is powerful since it allows ulyou to treat it better, if you want.

You can shoot yourself in the foot with every language.

Your comments read like "but I was promised magic and I found no magic" to me.

Well, of course there's no magic. What there is is a really good runtime and a good tooling that allows you to avoid some mistakes preliminarily.

I’d say it’s not really different than other dynamic languages in that you can get those runtime errors. Initially I got lots of those but after getting a bit of experience I hardly get them anymore aside from initial kinks after refactoring.

Maybe try VS Code with the Elixir LS plugin. It runs dialyzer automatically and highlights errors which also catches a ton of those runtime type issues. It’s pretty nice, often more expressive than standard typed languages.

There are OTP strategies that are a "do not recover from crash", or more to your case "do not recover more than N times per Z timeslice".

http://erlang.org/doc/design_principles/sup_princ.html#maxim...

The only part I miss about erlang is it being a little harder to shoot your self in the foot with types.

part of the Erlang Methodology is to write code that gracefully recovers from errors. it sucks for the user hitting the edge case until you patch it sure but the approach has it's merits and leaves the majority of your users running unimpaired.

Fair enough. Should have stopped at the sarcasm point. Sorry.

However, this is not my experience. When something bad happens in my situations, it's usually in response to a bad input that is going to be repeated a couple milliseconds later. So the app just crashes in loops.

I don't understand how the "let it crash" philosophy is supposed to happen when the only thing that can happen after a crash is... Another crash. Maybe my case is just too strange.

I agree with your frustrations but I don't think the blame is on "let it crash".

As far as helping prevent bugs, let it crash won't help. It is not a replacement for a type system either. A type system for Elixir would definitely help you prevent bugs (but unfortunately implementing one is challenging).

Let it crash is a mental model and a technique to handle bugs and avoid undefined behaviour when all other attempts to prevent them (be it via static types, testing, code reviews, etc) failed. Imagine some data validation functionality had a flaw and that causes bad data to reach a core component, would you rather:

1. have the core component continue functioning with said bad input, which may have unknown repercussions?

2. have the core component crash and restart itself with a known state?

So let it crash can be useful to guarantee your system is not going to venture into undefined behaviour.

I wrote a small example of this in a blog post about assertive code (http://blog.plataformatec.com.br/2014/09/writing-assertive-c...). If you compare the two get_token implementations, you are 100% correct that in the second implementation, if you receive bad input, the code will crash over and over again. But I would say that I prefer the crash, so I can think about how I want to handle said input, rather than have an implementation that works by accident (and the accident may be a happy or an unhappy one). Would I prefer to have foreseen the bad input in the first case? Of course! And do type systems help avoid undefined behaviour? Definitely! But not all of them. It would be great if we could have both.

On a larger scale, especially when building complex systems, you are going to interface with client/server, database, apis, etc and you will run into unexpected scenarios. You likely can't prevent, model, or foresee all of them, so "let it crash" works as a default behaviour to handle them. It is also a legit strategy for things you know that may happen, but the best way to handle them is to start over anyway. It provides a mental model for you to reason when things go wrong at each individual component, instead of trying to reason how an exception may affect the whole system.

I am not sure if this helps, so questions and feedback is appreciated!

The "let it crash" is mostly meant to deal with data coming from 3rd party API data IMO. So if somebody's gateway throws HTTP 503 errors for an hour, your app will eventually catch up with it when the other side fixes it.

There's no code in the world that can automagically recover from unexpected and not-coded-for user input, though.

The Zen of Erlang goes into this, look for “The reason restarting works”.

https://ferd.ca/the-zen-of-erlang.html

Very nice article, thanks. Reading with a slightly cooler head than yesterday, I still have strong disagreement with this.

I think it boils down to this:

> Here I'm classifying bohrbugs as repeatable, and heisenbugs as transient.

> If you have bohrbugs in your system's core features, they should usually be very easy to find before reaching production.

I don't know about "bohrbugs", but the vast majority of my bugs are "i-am-an-idiot-bugs", and boil down to "some combination of the handling one of the many possible inputs is going to trigger some code path where a function does not have the proper arity because I made a typo somewhere because I'm an idiot."

I claim that those are not "easy to find before reaching production" as a human being. (And I argue a large part of them are trivially found by a decent type static analyzer. Which, unfortunately, dialyzer was not last time I checked, 2 years ago.)

> By virtue of being repeatable, and often on a critical path, you should encounter them sooner or later, and fix them before shipping.

I disagree with that. My bugs happen on critical paths, but with non obvious shape of data. (I receive input from untrustworthy, aka "real-world", sources.)

I've spent enough time fighting bugs that were in the critical part of the code, were tested multiple times at the unit, integration and acceptance level, with different kind of simulators, but still broke on the first real-life usage because one function down a stack was not handling a nil or expected a string instead of a number, somewhere.

Encountering those issues "sooner" becomes a matter of "having enough imagination when writing automated tests".

I've heard property testing could be a tool to "automate" such imagination. But I'd rather have something telling me I just made a typo ;)

> So really, how efficient is restarting as a strategy?

> Well for repeatable bugs on core features, restarting is useless.

Amen.

> if the feature is a thing very important to a very small amount of users, restarting won't do much. If it's a side-feature used by everyone, but to a degree they don't care much about, then restarting or ignoring the failure altogether can work well.

It's hard to not read this as "Let it crash" being a good philosophy - provided it does not really matter, if your application works ;) (I know this is not the intended message, but still...)

I had great expectations about this section of the article:

> > I like statically typed language, and I restart my daemon after unhandled exception. What is Erlang gonna win me on fault tolerance

> This question was asked to me on a forum where I was discussing programming stuff and discussing the Erlang model. I copied it verbatim because it's a great example of a question a lot of people ask when they hear about restarting and Erlang's features.

The author follows with question (which is, basically, my opinion), with a very insightful explanation of supervision tree and OTP, but unfortunately, does not seem to talk about the "statically typed" part.

Which is a shame ! Message-passing, supervision and OTP are the great aspects of Erlang.

They're extremely powerful when used appropriately (which, in my experience, mean "sparingly, and as a plumbing-layer over purely functional code".)

My whole frustration with Elixir is that I want static typing "behind" the boundaries of GenServers.

Sure, I understand "GenServer.send" can not be typed, in general, and I'm ready to do some level of input validation at the boundaries, but beyond that point, please, If I tell you to add two apples, and I mispelled the name of one of them, don't even let me run, okay ?

I also know that there are efforts to do that (alpaca & co), but they're just too early-stages for my taste.

As said earlier, if you're solving the kind of problems where you never those kind of "i-am-an-idiot-bugs", or you're just not as idiots as I am, you may not find this problematic. I do.

But then, for dramatic purpose, the article ends with this gem:

> At some point, a team member got a very expensive credit card bill for the logging service we were using to aggregate exceptions. That's when we took a look at it and saw the horror on the leftmost side of the diagram: we were generating between 500,000 to 1,200,000 exceptions a day! Holy cow, that was a lot. But was it? If the issue was a heisenbug, and our system was seeing, say 100,000 requests a second, what were the odds of it happening? Something between 1/17000 and 1/7000. Somewhat very frequent, but because it had no impact on service, we didn't notice it until the bandwidth and storage bill came through.

Really ? Where those 500,000 to 1,200,000 exceptions not blocking anyone ? Not wasting someone's time ? Not causing improper or delayed results ?

Or, worst, just slowly making people lose faith into your software, or in software in general ? (Okay, I'm being slighly grandiloquant here, time to wrap up.)

Thanks for the link.

Depends. I have a cluster serving about half a million clients that send binary sensor data every 1-3 minutes. Sometimes the hardware has bugs and send erroneous payloads while the majority just continue running. So instead of spending a lot of time figuring out all of the possible malformed inputs I just let the processes restart in those cases and otherwise resume.

There are certainly ways you can get run away reactions that eventually take down the otp tree regardless of your settings and I probably do more input validation than strictly called for my the erlang guide books but not spending all of my time worrying about those edge cases does have it's upside with out negatively impacting other clients.

Interested: do you also do heavy business logic with the data you receive ? How easy is it to refactor this logic ?

Or is your cluster more of an agnostic broker / message pump that lets other systems do the logic ?

it calculates running and time segment based average/high/low/tally values and drives alerts off of them. Processing is done in stages it's not particularly hard to change how one stage works as long as you aren't talking about major struct type changes or message format changes as those are a bit harder to rollout if not harder to implement.