> Why would you destroy your trade relation with the U.S. for decades to come when you can just ride out a bad deal for a few years?
We aren't the ones who are destroying it, and we have a backbone strong enough to say this isn't good enough.
If what you say is true, then the next administration should understand how one sided the damage done to the relationship has been. No economic trade partner should have to tolerate attacks on their sovereignty and unreliable trade negotiators.
You have my empathy for this kind of sentiment. Personally this seems somewhat rare in practice. That being said I'm curious if anyone has anecdotes they can share about these kinds of things?
1. People add retries w/o backoff. Now we have a retry storm.
2. People don't add jitter so we get huge waves of highly correlated retries that cause self-perpetuating overload cycles and failure.
3. People add retries (even w/ backoff) at more than one layer (e.g. one in process, one in envoy), so now we have a deeply confusing multi-level of retries with super weird n^2 patterns.
4. People find ways to fetch from multiple data sources to make a composite object but don't cache/reuse data they fetched, so one data source being down causes DOS on _other_ data sources because of retries.
5. People add failover mechanisms where all failures happily pick the same failover target because, ya know, it has the most free capacity.
6. People underestimate connection setup cost so "failover" causes huge increase in overall load due to connection setup (often tls setup) causing huge influx of "new" cpu work in a loaded system. CPU spike causes unhealthy destinations, causing more failovers, causing metastable failure.
7. People realize purely local decisions aren't optimal so add a layer of global health metrics (e.g. global retry budgets) but these systems add a time delay to the metric (ie its the view of the world 5ms ago). This delay makes their naive control algorithm go into oscillations or divergence spraying stuff everywhere.
All of these things have a similar pattern that when things are almost all good, they will make the system more robust. You'll get an extra 9 of reliability on good days. But they make everything far worse when things are bad.
Configuring postgres to automatically failover instead of doing it manually. The automated system caused more downtime in a few months than manually doing it did for years before. All in the name of more automations and less downtime
For resolution of a problem you can’t prevent categorically, you have to spend a good bit of time pretending to be the computer before you can trust the computer to do it. A run book is a draft of a requirements doc for a program to do the same thing. And you can’t afford to test that program in production without loads of simulations first. And that’s running the process manually and checking aggressively for any signs of problems.
Some coworkers snaked the autoscaling work out from under me, and then ignored the advice I offered on low water marks and weeks of testing and rehearsals. All I can figure about their irrational exhuberence is something to do with claiming something for annual reviews? All I do know is we didn’t make it 40 hours from when they flipped it on until we had our worst outage in two years. Classic FAFO.
They were hoping to eventually get to a low water mark of just over 30% of our static cluster size, and they decided to use that for their initial go instead of my estimate of 40% being the low, and a recommendation that they start with 50-60% for the initial weeks and then ramp it down. When I refused to let it go in the status meeting, the team decided we should vote on it. Two guesses how that went.
A bug in the deployment logic the Ops team had for doing things like upgrading VMs caused the next update during daylight hours to spin up the new instances at the minimum cluster size, instead of the current cluster size. It was done outside peek hours but that still had us cut over with just under half of the hardware we needed at that moment. Because we were in such a fucking hurry to be something like third in line to use the new autoscaling support, nobody else had run into this problem yet (or system did a pretty good job of buffering everyone else).
A couple weeks later they’d looked at the stats and decided that we were spending less than a half hour a day running at the low water mark, and the additional shutdowns were causing churn that made it harder for us to detect problems like memory leaks. Surprise, surprise, they increased the min cluster size to exactly what I’d told them two weeks before.
Since “listen to me” isn’t a lesson that transfers to other teams, I will instead say, don’t transfer initiatives to a new team when there is less than 10% of the project left. The lack of friction you encounter may result in a lack of respect for the danger, and attention to potential problems. And if you must transfer, if your ideas about rollout timeline end up being less conservative than theirs, be patient and do it the slow way. They probably have encountered problems you haven’t seen yet.
I think the "happy path" might be a slightly wrong classification in GP, since the post is in reply to a retry-storm issue and explicitly talks about retry storms and thundering herds.
I've seen many cases where engineers optimize the sad path, but pessimize the wretched path. Or in less flowery language, they cut the occurrence rate of common non-critical failures, but by doing that they introduce code that can make rare failures much worse.
The cases I've seen generally boil down to naive retry logic or poorly tested and poorly maintained fallback paths (such as killswitches that break their environment[1], graceful degradation turned graceless, dormant feature flags that get reactivated).
The case you see with a retry storm here is the most classic one and the one that annoys me the most. I've seen engineers adding aggressive retries even into places where the impact is minor (you could show an error and let the user manually retry instead). Retries that improve user experience can be great if done correctly, but I've never seen the authors of such pull request addressing the risk and mitigation techniques for retry storm or retry amplification.
I've seen cases which had:
1. Retries on the client side (browser or mobile app).
2. Retries on the BFF.
3. Retries on Microservice A used by the BFF.
4. Retries on Microservice B used by Microservice A.
5. Retries on Critical Service C used by Microservice B.
Most of these retries had very short timeouts (e.g. 100ms), in order to keep latency SLOs during normal operations (not a good idea on retries). Every time QA saw a layer without retries, that would be a bug, and adding retries is easy, so we'd get a new retry without much thought. But the first time Critical Service C became overloaded, Microservice B started timing out a couple of times and retrying. This was too much too much for Microservice A that had a short timeout that couldn't hold the 3 retries done by Microservice B, so it making doing its own retries, all of them dropped in the middle of the way. Eventually you'll get a full-blown retry storm where every request from the client side got amplified with 3^5 retries, easily bringing down Critical Service C.
We'd usually introduce a circuit breaker for the particular path that caused the issue, but a variation of this kept happening several times because designing safe retries across a vast collection of microservices takes a lot of effort, and it's always easier to just add a quick-and-dirty retry at any point where you think you might need one and call it a day.
A proper solution (which I've never seen implemented) would be an mandating a corporate-wide inventory of retry-paths, and monitoring it for any path that is at risk of triggering a retry storm, or adding mandatory headers that cross microservices and track the amount of retries done up the chain and the time spent in total waiting for previous retries. You could have a budget for both and automatically stop performing more retries. Both solution require extra effort and a large degree of coordination.
> I've seen many cases where engineers optimize the sad path, but pessimize the wretched path.
As said GP, yes this is exactly what I meant and "happy-path" was an unclear choice of words. They optimize for the unhappy path on the good days, and make the bad days much worse.
Awesome to see this! I actually wrote PySceneDetect, was great to see it getting some use here. Would you be willing to share what parameters you were using? I'm curious why the accuracy was so low.
PySceneDetect only uses basic heuristic methods right now so it does require some degree of tuning to get things working for certain data sets. Your post inspired me to look into maybe integrating TransNetV2 as a detector in the future!
Nice to see you on here! I used the ContentDetector with a threshold of 27.0 and otherwise default parameters. Realize I could have done a grid sweep to really hone in on a good param range, but because I had only one input video labeled I wanted something that would work well enough out of the box. I imagine this dataset is rather... heterogenous.
If you happen to know a better apriori threshold I would be happy to re-run the analysis and update the chart.
If you're willing, could you try using AdaptiveDetector? It should have better defaults for handling fast camera movements a bit better.
The threshold values themselves can be tuned if you generate a statsfile and plot the result, but that can sometimes be tedious if you have a lot of files (thus the huge interest in methods like TransNetV2). Glad to see the real world applications of those in action. You can always just increase/decrease the threshold by 5-10% depending on if you find it's too sensitive or not sensitive enough as well.
Are there keyboard controls? Someone I was just playing with seemed to always lock on each "note" or "half note" exactly, wasn't sure how they were doing that.
Author of PySceneDetect here: Thank you all for the thought-provoking discussions, and the attention you've given my side project. There are some specific cases where PySceneDetect achieves great accuracy (like fast cuts or fades), and some where it's currently not so good at (like sudden flashes or large obstructions). That being said I do want to track these things and come up with solutions to improve the robustness of the content detection algorithm over time.
I'm most open to any feedback or feature requests/ideas/suggestions; feel free to checkout the issue tracker on Github, or create a new entry:
So this is likely beyond the scope of your project, but I've always thought a really good project would be a website to host scene indexes for movies and TV.
Eg. Let's say that you wanted to watch a prerecorded football game or baseball game without all the commercials, timeouts, commentators talking about the fans, etc.
Or... Let's say that you wanted to re-cut a movie in a certain way, by re-ordering the scenes, you could just generate a new scene data file and let the encoder/player use that.
This is still relevant I think :) What you mentioned is very similar to an edit decision list (EDL [1]), of which I only learned recently. I had a feature request [2] to support EDL as an ouptut format, and upon further investigation, it seems like the format is very similar to what you're talking about. The Wikipedia page also indicates that VLC supports xspf files, but I haven't done much research into that yet ("XML Shareable Playlist Format").
Author of the program here. You are correct, the program detects shots rather than scenes, but I didn't want to give the impression that this project was related to the existing ShotDetect program. I felt that the documentation explained this well enough, but I'm open to considering an alternative project name if anyone has a suggestion.
Would you be able to share a small sub-set of the episode, in particular the area where you're unable to detect the starting segment? (If not, no worries!)
There's a few issues with PySceneDetect currently that may lead to false or missed detections, but these are things that I would like to solve in the long run:
- threshold is heuristic/fixed right now, but I would like to change it to an adaptive/statistical method which can dynamically change
- single-frame events can trigger false scene changes
Thanks for your feedback, and feel free to share any other suggestions you might have.
Author of PySceneDetect here. The current implementation does exactly what you hint at, except instead of YUV, it considers deltas in the HSV domain (specifically differences in hue and color).
Other techniques being considered for future work include use of optical flow, background subtraction, and analyzing histograms.
From what I remember the Y (luma) component in a YUV video has more information than the other two components and it could also be extracted without the need to fully decompress the video (in mpeg compressed videos). Of course this info is more than 10 years old (I don't really do any video research any more) so I guess there should have been progress in that area.
This is indeed correct, I'm just using HSV instead of YUV, but the primary source of information is the luma/brightness component (although currently all 3 of the HSV components are averaged, so perhaps a better weighting may improve precision).
This is definitely a good idea, and something that I'm most open to considering for a future release of PySceneDetect. Admittedly the current version does not handle single-frame "upsets" like this, but this does seem like a logical and reasonable approach to a first attempt at filtering them out.
I would do an exponential smoothing of pixel values over some timescale, say, 0.2 seconds, before further detection of scene changes. That should do the trick.
We aren't the ones who are destroying it, and we have a backbone strong enough to say this isn't good enough.
If what you say is true, then the next administration should understand how one sided the damage done to the relationship has been. No economic trade partner should have to tolerate attacks on their sovereignty and unreliable trade negotiators.