Repository navigation
implement multi resolution lightcurves #462
Description
At the moment we just create keep up adding new extracted sources to the database without a retention policy. This will cause slowdowns and eventually disk fill up.
I think we should implement (optional) multi resolution ligtcurves where a lightcurve is 'compressed' or averaged into a lower resolution after a configurable amount of time. Maybe multiple scales of resolution.
For example RRDtool and graphite apply similar techniques to their metrics:
Activity
timstaley commentedon Sep 4, 2015
So this is a cool idea, but I think it's important to realise that it's basically an entirely new sub-project, and will likely require splitting TKP into sub-packages, then forking / rewriting the database sub-package.
Here's my reasoning:
Ideally, we split out PySE, so you can use that, and probably something like a tkp.utils package that includes DataAccessors, quality-checks and the like. But for the database code you will effectively be starting from scratch schema-wise - you'll definitely be able to re-use some ideas (dataset / runningcatalog / image etc) but all of the extractedsource and variability metrics will have to be rethought because they won't apply to a multi-resolution lightcurve. Likewise you'll need a re-implemented visualization tool.
Accordingly, I'd estimate this as a work-package of maybe 3 to 9 developer-months, or maybe just
¯_(ツ)_/¯