What is the general outline of going from a model of a craft (drone, sailboat, etc) to building a sim that can do reinforcement learning over a physical object interaction with its env?
I want to start playing with models, sims and collected data for sailboat racing- I know the RL/data science stuff, and I assume a good model of your craft takes time to build, and can be improved with collected data. What are some areas to explore when chaining model -> sim -> RL for performance?
I realize this is an extremely complex topic, with several PhDs worth of potential input- if you had to explain to someone technical what it looks like and where to keep digging, what would you say?
Models for drones are actually pretty simple. The races are indoors, so we can assume 0 wind. It’s really basic physics and aerodynamics. We had some sessions in a mocap lab where contestants were able to perform some flights and get some ground truth to tweak their sim -> actual drone dev loop. The simulator itself was based on the DRL simulator, but not the same version on the Steam store.
I did not develop the sim itself but did develop the hardware-in-the-loop portion of it along with things like real-time debugging, and output to the hitl. We had the sim rendering cameras which we output from the workstation to custom HDMI bridges over MIPI that we could treat as real cameras on the NVIDIA Xavier AGX. There was a data channel over Ethernet for IMU data.
I made a custom version of Eclipse that interfaced w GDB for debugging, which also was modified to stay in sync w the sim using PTP, w rewind capability.
As for sailboat modeling, yes it’s more complicated because of the effects of both wind and fluid dynamics. If I were approaching this, I’d probably try and find a physics simulator to start with. Getting ground truth will be difficult, but I imagine you would start w the IMU and GPS data off the boat, but having time synced ground truth for the waves and wind will probably be the hardest part.
Numerical simulation of aircraft is pretty straightforward. The hard part is going from the inertial frame into the aircraft's frame of reference, and back again. Once that part is down, the main procedure is to find the gravity vector in the body frame, calculate thrust, lift and drag vectors in the body frame, calculate any torques, sum everything, and then convert back into the inertial frame to do the numerical integration step. From memory, torques are the hardest part, since you need to work with either the time derivative of the quaternion, or the time derivative of the rotation matrix. But if you stick to standard coordinate frame conventions, it's just applying a formula.
The only difference between fixed wing and multirotors is the different models for forces in the body frame. A fixed wing aircraft would use an aerodynamic model of whatever complexity you like, and a model for the prop. A multirotor would use multiple prop models, and a drag model for the body. Most models for these things are pretty well defined, but you can choose just about anything that "fits" so long as you can justify it. And you can use the same basic setup for any aircraft (and indeed any robot) with a bit of forethought. The data part is mostly specifying things like, say, damping coefficients that you would then want to measure.
The dynamics are not the difficult part of simulating aircraft for RL, it's actually the rest of the environment. E.g. navigation tasks require you to build a navigable environment around your physics model, and specify things like reward functions.
I don't have any expertise in this field, but I expect sailboat training simulations would be much harder than aerial drones because of the underlying physics involved.
I'm sure flight simulations have to deal with some amount of fluid mechanics and turbulence to get an accurate simulation, but I suspect it's fairly simple and you can mostly model it accurately enough using Newtonian physics.
But for sailboats, the entire system relies profoundly on the interaction of two separate fluids with a single body and turbulence and viscosity are deeply interwined in making the boat go. Not to mention that the sail itself is a flexible deformable surface.
As a sailor, you're giving sailors a bit too much credit. Those things all matter, but to say that the average sail boat racer is even optimizing for all those things efficiently at once is a huge stretch. A lot of sail boat racing is strategy and properly performing the basically in a hostile environment. If you could ensure great strategy based on learned data, and immaculate execution based on properly tuned controls, I think you'd win easily.
I don't think GP is saying that sailors take all of those effects into account constantly, so much as he's saying a useful simulation would be monstrously difficult due to the underlying physics.
I've made robot models in the past, but never for anything involving water (although I actually have done physics simulations with water but nothing with robots). I don't know anything about sailing and I'm definitely not an expert at building simulations, but I can grok what GP meant and I agree with his assessment.
To give a rough analogy - it would be like trying to learn how to be a racecar driver on a videogame with simplified physics. There are professional drivers that use simulators and iRacing (or Assetto Corsa or a handful of other games) but none of them are training on Need for Speed, and it's because the difference is so stark you're actually handicapping yourself instead of learning how to drive. You need the simulation to be close enough to reality before it starts to become useful.
A simulation doesn’t necessarily need to be remotely physically accurate to be useful, as long as the simulated behaviour is qualitatively similar to the real thing. For sailing a small boat, I’d imagine the important things would be the shape of the sail and the heel of the boat for a given apparent wind, presented to whatever sensors your sailbot is using. For the water’s surface, random semi-regular perturbation (sum of sine waves) would be fine, and something similar (or maybe a random walk) for wind direction and strength.
For this kind of real-world robotics, you can’t optimise in any academic sense so you just have to go for effective and robust.
Well, for the drones, we did base things off ground truth captured in a mocap lab, and this helped the models significantly. There always will be a jump from simulation to real world, but the smaller you make that jump the quicker you’ll have a functioning vehicle. Without that, you can spend a ton of time (wasted) in simulation when the real world is substantially different. I’m sure a sum of sine waves would get you somewhere, but only so far. To me this is a difficult problem and I think it might be best to do most development in a real world environment.
I was just talking about sailboats specifically, where the external forces are highly unpredictable and similar in magnitude (albeit hopefully a fair bit smaller!) than your control authority over the boat.
The work you did on drones sounds very cool and in that scenario (high powered motors, crazy nonlinear dynamics, relatively small external perturbation, at least from how you described it?) I don't doubt that ground truthing your simulation using mocap data made a huge difference.
> as long as the simulated behaviour is qualitatively similar to the real thing.
My point is that I suspect that unless you have a very sophisticated simulation, it won't be.
> I’d imagine the important things would be the shape of the sail
The shape of a sail is itself a very complex thing to model and simulate. I'm not even a real sailor and I'm already thinking about camber, twist, and luffing. When sailing upwind, the sail is functioning like a wing using lift. When sailing downwind, it's relying on drag. Points in between use a mixture. This implies that turbulence and stall must be simulated at some level of fidelity.
If you've got a mainsail and a jib, then you have to worry about the slot effect and one sail blocking the wind of the other.
> and the heel of the boat for a given apparent wind
Not just heeling, but weather helm, leeway, drag, displacement, and how those interact.
Agree, I imagine just maxing VMG on the fly as conditions change is something a computer could already outperform humans easily.
I've wanted to make a simple sailing game for a while now but found it very hard to actually get something that feels right and compared to other activities there doesn't seem to be great resources on all the different parts of the physical system (in terms of Math).
> If you could ensure great strategy based on learned data
That's my point. If your learned data is coming from a simulation that doesn't model actual sailboat fluid dynamics, you've learned little.
A robot that learned to sail by studying a hollow cube floating in a bathtub full of marbles is not going to be able to make an actual sailboat do anything useful out on real water.
Yes getting timestamped and synchronized ground truth to build/verify your model will be quite challenging for something like this. For the drones we just took them to a mocap lab, and races are inside so we had the luxury of assuming 0 wind. The only remotely tricky part is handling ground effect from the props.
That's a very interest project. As a amateur sailor I have played with some models for sailing path optimizatio, wich can be somehow related to your problem. I started with a very simple model using sailing polar graphs, these map wind speed and course to maximum speed and usually based on empirical data, I found some DBs with this data, I can check if you are interest. Nevertheless for a better simulation you will at least need a simple model that gives you the forces at least especially if you want to simulate the boat acceleration, heeling and leeway. Simulating the sail foil behaviour would be very computational intesive as the airflow around it can get very complicated especially as the sail changes it's shape with it.
Curious what recent work you are referencing? Lots of new so I don’t want to miss things. In my experimentation I basically wrote off LLMs from doing any math directly. That includes aggregations or precise calculations. I’ve been playing with sql generation and allowing the db to do its thing.
Hi holoway, I've been following your project for a couple of months now. How do you compare yourselves to upbound/crossplane?
IME crossplane has been a "much better terraform" and also borrows from some of tf's open source provider code. Seems to be one of the best IaC pattern for k8s centric shops.
Pros:
- adoption of k8s core engine, state mgmt
- model everything as a CRD, consistent definition pattern both infra and app
- open source
- great UI when layered with argoCD
- declarative
Cons:
- steep abstraction learning curve (for me anyway)
- docs lacked key context for newbs (also getting way better, big efforts here)
I think you hit the nail on the head with crossplane being a 'much better terraform' by design. Our goal isn't so much a better IaC / Declarative infrastructure tool - it's a better overall workflow for doing collaborative DevOps work. We think that by having an active model of your component, and tracking the resources along side, we can fix the feedback loops in a way that things like crossplane, terraform, or pulumi really can't.
Of course today it's early - so you have to look at what we're building as a foundation for the future. But it's a solid foundation to build on!
Wait, I thought k8s and open shift had this entire service catalog concept that was basically sort of a CRD->IaaS bridge.
Did that die? Is that not even what crossplane is? Once again, I'm 6 clicks deep and havent seen a single actionable example. I wish I could get inside some people's heads and wonder wtf theyre thinking when marketing this stuff.
Edit. Wow OSB and Service Catalog are dead. Is there a obituary?
> Once again, I'm 6 clicks deep and havent seen a single actionable example
Based on the terse reply to my question, it seems they really want you to create an account and go through their onboarding tutorial after agreeing to an absolute telephone book sized ToS. Hell, it's possible I'm violating one of the paragraphs by even talking about the process. I appreciate that's likely not what you had in mind, but it seems to be what they have in mind
@cyberax said, "My problem with the current IAS systems is state storage. It should not be needed! Instead, the IAS tool should introspect the systems it's managing and build the necessary state on the fly.
@firesteelrain said, "you can do that through abstraction. You "include" your Terraform Azure Provider or Terraform AWS Provider. At the end of the day, your module needs to know what it’s interacting with but not the higher level of abstraction. We have done it at my work to make it cloud agnostic just in case we need to go to another CSP"
Single ops eng in a 3 person startup here. Ops eng is only one of my hats right now :) I found crossplane to be a solid tool for managing cloud inf. My assertion is that "the only multi-cloud is k8s" and crossplane's solution is "everything is a CRD". They have an extensive abstraction hierarchy over the base providers (GCP, TF, Azure, AWS, etc) so it's feasible to do what firesteelrain did. My client requirements span from- you must deploy into our tenant (could be any provider) to host this for us.
I can setup my particular pile of yaml and say - "deploy a k8s cluster, loadbalancers, ingress, deployments, service accounts (both provider and k8s), managed certs, backend configs, workload identity mgmt, IAP" in one shot. I use kustomize to stitch any new, isolated environment together. So far, it's been a help to have a single API style (k8s, yaml) to interact with and declaratively define everything. ArgoCD manages my deployments and provides great visibility to active yaml state and event logs.
I have not fully tested this across providers yet, but that's what crossplane promises with composite resource definitions, claims and compositions. I'm curious if any other crossplane users have feedback on what to expect when I go to abstract the next cloud provider.
cyberax's note on state management is what led me away from TF. You still have to manage state somewhere, and crossplane's idea was- k8s is already really good at knowing what exists and what should exist. Let k8s do it. I thought that was clever enough to go with it and I haven't been dissapointed so far.
The model extends the k8s ecosystem, and allows you to keep going even into things like db schema mgmt. Check out Atlas k8s operator for schema migrations- testing that next...
I also like that I can start very simple, everything about my app defined in one repo- then as systems scale I can easily pull out things like "networking" or "data pipeline" and have them operating in their own deployment repo. Everything has a common pattern for IAC. Witchcraft.
Back in my early days of data eng for ML pipelines, I stumbled onto Drake- and it opened my eyes to managing pipelines as DAGs. This pattern is supremely effective, and I try to teach anyone who might benefit.
Drake looks interesting, the visualization looks fun.
I don't see any evidence that it handles refunding of steps when transitive depended on code is changed, though. E.g., if script BBB.py includes CC, and CC is changed, then all steps transitively depending on BBB should be rerun. My make-booster specifically deals with that case.
I also expect drake to have a slow start, which slows development.
Don't overlook the comment about form. Form is everything, weight is secondary. Get a professional trainer to help you setup your form once you move to barbell work and then you can go from there on your own.
r/homegym will have some good recs for gear to get started. Plenty of bodyweight training you can do too- great for turning on your main muscle groups (youtube).
A barbell and some plates w/ rack will cover the main lifts. Squats, deadlifts, presses (chest and shoulder) are the first ones to build. I recommend Starting Strength for an on-ramp to get going. Adjustable dumbbells are great too- plenty of options out there. Some of this equipment is expensive, look for used and consider it an investment in health.
I want to start playing with models, sims and collected data for sailboat racing- I know the RL/data science stuff, and I assume a good model of your craft takes time to build, and can be improved with collected data. What are some areas to explore when chaining model -> sim -> RL for performance?
I realize this is an extremely complex topic, with several PhDs worth of potential input- if you had to explain to someone technical what it looks like and where to keep digging, what would you say?