Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

    Why not just use a time decaying ring buffer 
Because you don't want to through the data away. RRDTool and Whisper implement ring buffers but you lose data and resolution with them. If that's acceptable then absolutely use those tools. If you don't want to lose data though then you need something else.


You don't have to decay, I just assumed eventually you would want to decay the data but in a separate ring buffer, thus no resolution loss.


This is how things work at the industrial plant where I work. We dump a heap of data out of PLC's at basically whatever the native frequency the instrument can log it hits the first datastore (which is essentially a flushed ring buffer) where it might get thrown at a HMI display or something like that after that it decays into the slower historical archive where it can get some metadata added to it and get rationalized aggregated, batched together or whatever.

It varies but datastores tend to be 1 hour worth of data, then 3 days, then 3 months, then permanent. As you move between datastores latency to access a timestamp increases.

We tend to call Time Series databases "Data Historians" its a big industry and from what I can tell most commercial products are built around ring buffers.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: