config system - #1080
config system#1080
Conversation
|
@clewis7 r4r This looks large but the main contribution is The rest is is just adding the decorators and stuff to register each class relevant method. |
|
|
||
|
|
||
| class Graphic: | ||
| config = global_config.descriptor |
There was a problem hiding this comment.
@clewis7 you might be wondering we have this whole descriptor stuff and it's because we can't do:
class A:
config = global_config._registry[A]Because A doesn't exist yet when the python interpreter is creating the config variable! By using the descriptor you don't need A to exist yet, it only needs to exist when the user calls A.config 😆
The interpreter parses everything in the class and creates all declared objects in the class (methods as well) and the class is created only after everything in the class has been created! It's like filling a cup of water but the cup exists after all the water has been poured into it 😂
| class ConfigDescriptor: | ||
| """Descriptor pattern so classes can access their configuration for users to set/get config options""" | ||
|
|
||
| def __init__(self, classes): | ||
| self.__classes = classes | ||
|
|
||
| def __get__(self, instance, cls: type = None): | ||
| if instance is not None: | ||
| raise AttributeError("set config options on the class, not an instance") | ||
|
|
||
| if cls not in self.__classes.keys(): | ||
| raise AttributeError("Class is not registered") | ||
|
|
||
| return self.__classes[cls] | ||
|
|
||
| def __set__(self, obj, value): | ||
| raise AttributeError("Cannot set") |
There was a problem hiding this comment.
@clewis7 See comment on base Graphic if you're wondering why this exists
|
I think I can just get rid of ConfigValue and replace using this https://stackoverflow.com/questions/17625695/is-it-possible-to-change-a-functions-default-parameters-in-python Then I can just specify which arguments are configurable on the method decorator, and it populates the defaults in the wrapper, and inject the config values into |
| def declare(self, *configurable): | ||
| """ | ||
| Declare configurable arguments for a method. | ||
| """ | ||
| if not configurable: | ||
| raise IndexError( | ||
| "No configurable arguments declared, this cannot be left empty. " | ||
| "Either declare configurable arguments or don't decorate this method." | ||
| ) | ||
|
|
||
| def append_to_config(method): | ||
| new_pending = Pending(method, configurable) | ||
| if self._pending and not new_pending.is_sibling(self._pending[-1]): | ||
| raise TypeError( | ||
| f"{self._pending[-1].cls_qual} is not registered with the global config" | ||
| ) | ||
|
|
||
| self._pending.append(new_pending) | ||
|
|
||
| # keep these to use them in the injector | ||
| method_name = new_pending.name | ||
| # create signature object just once when the method is decorated instead of every time the method is called | ||
| sig = inspect.signature(method) | ||
|
|
||
| # NOTE: variables within here are available in the injector because they exist in its __closure__ | ||
| # any variables from the outer function that are used in the inner function are always in the __closure__ | ||
| # source: https://stackoverflow.com/questions/14413946/what-exactly-is-contained-within-a-obj-closure | ||
| # official docs: https://docs.python.org/3/reference/datamodel.html#function.__closure__ | ||
|
|
||
| @wraps(method) | ||
| def injector(instance, *args, **kwargs): | ||
| # get the method config dataclass | ||
| method_config = getattr(type(instance).config, method_name) | ||
|
|
||
| # create a binding | ||
| try: | ||
| binding = sig.bind(instance, *args, **kwargs) | ||
| except TypeError as e: | ||
| # if *args and **kwargs don't match the signature raises a TypeError | ||
| # useful if the user passed wrong things, we need to catch and tell them what method it was | ||
| # since binding has no idea of the full namespace when we're handling it here | ||
| raise TypeError(f"{method.__qualname__}: {e}") from None | ||
|
|
||
| config_dict = method_config.to_dict() | ||
| # merge config values with the binding | ||
| # any values that the user explicitly provided will be in binding.arguments | ||
| # therefore an explicit user provided value will override the config value | ||
| binding.arguments = {**config_dict, **binding.arguments} | ||
|
|
||
| # apply any missing default vals from the method signature | ||
| # this isn't actually necessary but is just a robust failsafe | ||
| # I think it should account for any weirdness with methods that have positional-only arguments | ||
| binding.apply_defaults() | ||
|
|
||
| # finally call method with updated binding from config | ||
| return method(*binding.args, **binding.kwargs) | ||
|
|
||
| return injector | ||
|
|
||
| return append_to_config |
There was a problem hiding this comment.
@clewis7 fanciest decorator I've written 😂 (updated version which is even more elegant)
|
Merging, @clewis7 has agreed to fast-track this PR. |
* start ndprocessors * basic timeseries * add __init__ * heatmap for timeseries works! * NDPositions, basics work, reorganize, increase default scatter size * black * NDPositions working with multi-dim stack of lines, need to test window funcs * scatter collection * progress, need to change to other branch so committing * better * interpolation for heatmap * better place for check * window functions working on n_datapoints dim * p dim window funcs working for single and multiple dims I think * black * index_mappings is working I think, lightly tested on p dim * remove nd_timeseries since nd_positions is sufficient * auto-replace buffers (#974) * remove isolated_buffer * remove isolated_buffer from mixin * basics works for positions data * replaceable buffers for all positions related features * image data buffer can change * resizeable buffers for volume * black * buffer resize condition checked only if new value is an array * gc for buffer managers * uniform colors WIP * switching color modes works! * typo * balck * update tests for color_mode * update examples * backend tests passing * default for all uniforms is True * update examples * forgot * update test * example tests passing * dereferencing test and fixes * simplify texture array tests a bit * image replace buffer tests pass yay * forgot a file * comments, check image graphic * add image reshaping example * add buffer replace imgui thing for manual testing * black * dont call wgpu_obj.destroy(), seems to work and clear VRAM with normal dereferencing * slower changes * update * update example * fixes and tweaks for test * remove unecessary stuff * update * docstrings * fix example * update example * update example * update docs * some basic OOC working * max num of dipslay datapoints * scatter stack, not tested * progress * scatter collection updates * tootip handlers for ndpositions * refactoring, general NDPP_Pandas processor for any dataframe data * separate array logic and graphic logic in `ImageWidget` (#868) * start separating iw plotting and array logic * some more basics down * comment * collapse into just having a window function, no frame_function * progress * placeholder for computing histogram * formatting * remove spaghetti * more progress * basics working :D * black * most of the basics work in iw * fix * progress * progress but still broken * flippin display dims works * camera scale must be positive for MIP rendering * a very difficult to encounter iterator bug! * patch iterator caveats * mostly worksgit status * add ArrayProtocol * rename * fixes * set camera orthogonal to xy plane when going from 3d -> 2d * naming, cleaning * cleanup, correct way to push and pop dims * quality of life improvements * new histogram lut tool * new hlut tool * imagewidget rgb toggle works * more progress * support rgb(a) image volumes * ImageGraphic cleanup * cleanup, docs * fix * updates * new per-data array properties work * black formatting * fixes and other things * typing tweaks * better iterator, fix bugs * fixes * show tooltips in right clck menu * ignore nans and inf for histogram * histogram of zeros * docstrings * fix imgui pixels * iw indices event handlers only get a tuple of the indices * bugfix * fix cmap setter * spatial_func better name * bugfix * hist specify quantile * fix typos (#991) * fix typos * add rendercanvas to intersphinx_mapping * nd-iw backup * correct ImageGraphic w.r.t. ndw * last fixes in ndi * update * basic minimal ndw orchestration working * implement auto-x for timeseries * bugfix update worldobject -> graphic map for image tiles * bugfix linear selector set limits * linear selector for timeseries * return full data if display_window is Noen * arrow key to step indices * imgui separator * fix and ui stuff * both auto x range modes working * progress * moving stuff * much much better organization of things * GlobalIndexVector working with ndpostions and ndimage * examples * progress * do not reset vmin vmax when replacing Image buffer * WIP migrate to xarray * window funcs in NDProcessor class using xarray, WIP, not tested * typo * basic single index slicing working with xarray * window funcs working for NDPositions and NDPP_Pands * display_window window funcs working for NDPositions * imgui stuff * finish migrate NDImage to xarray, basics work * NDImage working mostly, behavior viz is back * better flipping logic * update examples * more progress * update example * histogram working for images * NDProcessor property aliases * more aliasing * fix * update example * fix * ui * cleanup old iw-array, imports, add deprecation warning on old iw * add ndwidget section to deps with xarray * nice repr for NDProcessor * imgui right click menu for ndgraphics * better * controller options separate window * update imgui * fix * fix compute histogram * other features WIP * basics of other features works with ScatterStack for colors, markers, sizes, need to keep testing * require min pygfx v0.16.0 due to gc hash fix necessary for NDWidget * fix PlotArea.y_range * fix to create isolated buffer for colors when buffer replaced * np.empty * cmap_transform_each WIP * progress * fix * replace graphic when data changed, tweak index_mappings * Update installation docs (#1013) * add simplejpeg to notebook deps * Update guide.rst * Update guide.rst * Update README.md * cmap lib handles image colormaps now * multi-windows ndwidget, maintain features like cmap when switching graphics * progress * lighting objects only when a mesh is added * fix * update axes only when camera or view changes * clean heatmap func * stupid print * docstrings, comments * remove unused attr, comments * remove print * add NDGraphic.pause, expose histogram widget * ndg pause in imgui * add helper function to convert heatmap timeseries to postional data shape * index wans't calling handlers * remove * allow image types other than float32 (#1027) * Includes code for doing batched transforms in both directions (#1025) * Includes code for doing batched transforms in both directions * Streamlined parsing logic and updated docstrings for model to world code * Fixes inconsistent error messages * More streamlined type checking * Gets rid of checks and uses np asarray * Some more aesthetic updates to the syntax * Update fastplotlib/graphics/_base.py --------- Co-authored-by: Kushal Kolar <kushalkolar@gmail.com> * async NDProcessor (#1026) * async NDProcessor established, NOT TESTED * ASYNC NDPROC IS WORKING :D :D CELEBRATE * comments * type annot * fix * fix * polish async integration, cuda also integrated * no longer using xarray, allow simpler ArrayProtocol * comments * comments * docs * forgot to set back to False * remove xarray from pyproject.toml * throttling * Adds nd vector graphic (#1034) * Includes nd vector code that works * Faster position assignment, no more for loop * Batched computations for vector set function * Formatting updates * Includes improved annotations and changes ordering of the data slice from the vectors graphic * Some further improvements to the pylinalg code * Fixes remaining formatting and naming issues * Apply suggestions from code review Co-authored-by: Kushal Kolar <kushalkolar@gmail.com> --------- Co-authored-by: Kushal Kolar <kushalkolar@gmail.com> * Adds kwargs so user can customize vector field estimates (#1036) * Adds kwargs so user can customize vector field estimates * Does the kwarg organization the way ndpositions does * fixes graphic constructor * implement yuv and other colorspaces and "bufferless" `TextureArray` (#1033) * implement yuv and 'bufferless' TextureArraY * unbuffered and yuv420 works * warning on tooltip * NDImage always uses unbuffered, support colorspaces in NDIMage * update docstrings * by default disable AA and set pixel_scale=1.0 for performance * unpacked yuv support * independent graphics and texture features for rgb and yuv * docstrings * docstrings * import order * yuv graphic working nicely * add enum to top level namespace * update script to produce add graphics mixin * add yuv example * update ndimage with yuv stuff * fixes * yuv video working well with NDWidget * cleanup * `SelectionVector`, `HighlightSelector`, `VisibilitySelector`, `SelectorCollection` (#1038) * start selection vector stuff, protocol * highlight selector WIP * fixes * progress * lut repeat * highlight and visibility selectors working, selection vector working * PlotArea checks for SelectorProtocol instead, move some stuff around * selector collection, not yet tested * cleanup * bugfix for linear selector limits * add example * better append, remove * cleanup * basically rewrote entire ImageHighlightSelector * black * happy with iamge selectors * toy multi session example * correct mapping stuff * better scalar/vector handling * comments * compute capabilities (#1040) * allow setting texture usage directly * ImageHighlightSelector, VisibilitySelector and ImageVisibilitySelector can handle 'None' as placeholders in the selection * fix example * comments * better axes padding for tick labels, settable axis label (#1049) * better axes padding for tick labels, settable axis label * cleanup * move axes to tools * update * move stuff around * update docs * update * update * use fully fledged async, `NDPositions` improvements (#1050) * use fully fledged async * better throttling * torch.Tensor.tranpose() doesn't like tuples * we need time-based throttling, but it can be gentler * improvements * improvements * fixes * x_range fix * fix NDPandasProcessor * fix * HighlightSelector fix to append None * cleanup better * know ndg current dipslayed indices * improve example * docstrings, cleanup * remove __array_ufunc__ from required attrs * do not require __array__() to be implemented for ndwidget * auto ref ranges * fix imgui * tear out NDTimeseries from NDPositions * imgui refactor (#1066) * start imgui window refactor * update w.r.t. imgui changes * imgui hlut colorbar stuff * better focus logic * imgui popup stuff * update docs * update examples, remove HistogramLUTTool * update examples * imgui menubar example * update docs, tweaks * docs * doc fixes * docs * fix * right click anywhere in colorbar/hlut window * inf line, dashing, thin bool (#1064) * Create partial_camera_linking.py (#1020) * inf line, dashing, thin bool * tests * new screenshots * Selection vector improve (#1061) * Full selection vector implementation * Includes some documentation at top of SelectionVector * Minor typing fix in linear selector selection setter * First working version with selection vector * Reworks the logic for adding selectors, improves some documentation, adds partial instead of lambda functions, improves typing in highlight selector * Fixes casting bug in the integer version of the code * Update fastplotlib/graphics/selectors/_selection_vector.py * Update fastplotlib/graphics/selectors/_selection_vector.py * Removes unused init time parameter and also updates the typing on the default inverse mapping function to always return an integer * Adds a nonneg check in selection vector selection setter * Adds nonneg test for inverse handler --------- Co-authored-by: Kushal Kolar <kushalkolar@gmail.com> * Includes basic np dlpack function that is tested to work across jax a… (#1067) * Includes basic np dlpack function that is tested to work across jax and torch * delays dimension permutation for all code that uses ndprocessor * Eliminates old comment * Updates the numpy dep to be the min version that allows dlpack conversion across devices * Includes code to update spatial dims indices whenever spatial dims is changed * Moves the spatial dims indices computation to the read only property in the base class, updates docs, uses public property in the ndprocessor subclasses * Filters for None values in the inv handler (#1073) * bugfix after last PR * alpha is material.opacity (#1074) * clamp vmin, vmax in colorbar histogram LUT tool (#1076) * Flat graphic features 2nd edition (#1072) * refactor cmap and cmap_transform for positional graphics * color mode stuff * WIP * more WIP * better add_graphic autogen * more progress * mostly done refactoring positions graphics cmap stuff * forgot to update mixin * cmap_transform tweaks * wip, JaggedArray * mostly works, writing tests * remove a print * image collection examples * fix * feature inheritance in PositionsGraphic and cmap_range * cmap handling * cmap tweaks * remove parse_cmap_values, rely on cmap lib * fix example, remove unused line of code * Image just uses cmap lib now too * delete_ndgraphic() * ndtimeseries cmap example * steps for stacks * ndtimeseries sets stack steps * docstrings * docstrings * comments * renames * docstring * fixes * update tests/test_colors_buffer_manager.py * update test_markers_buffer_manager.py * update plot_helpers test * update test_point_rotations_buffer_manager.py * update test_positions_graphics.py * update another test * more tests * update yet more tests * remove unused * infline tests * collections tests * updating examples, wip * more examples updates * more examples updates * more example updates * more example updates * example and docstring * docstring * remove complex line collection slicing example, stick with real usecases * better line stack example * update example * update example * update link * update more examples * examples and fixes * fix * more fixes * fix * updates * Fix * update docs * docs * add image collection to docs conf.py * update docs * Ndw rename consistency (#1078) * rename for consistency * more consistency * more * final renames * fix * cursor fix * missed renames * pandas fix * restore ImageWidget, auto-collapse NDWidget sliders UI if no slider dims and no appended UIs * imagewidget stuff * ImageVolumeGraphic args fix * NDWSubplot dims check only for array-like * collapsed property on ImaguiWindoW * update iw examples * colorbar fix * lingering rename * fix * type * clim_quantile for NDImage, other fixes * public 'max_display_datapoints' property * dw setter * imgui right click for NDGraphics * axes label tweaks * remove unused zarr placeholder * fix docstring * config system (#1080) * basic scaffold done * inheritance * done * config works! * config on graphics * axes config * full config implementation basically works * fix * mixins call Graphic construtors with kwargs nothing is positional * print * config presets * comments, docstrings * docstrings * remove ConfigValue * comments * much better add graphics mixin using descriptors, examples, fix a test * anotehr example * GlobalConfig.to_dict() * docs * reset to default config after each screenshot test * add_<graphics>() stub generator, fix maintain_aspect logic w.r.t. config stuff * change so maintain_aspect can be tested better * docstring * reset config after running each docs gallery examle * better example * docs * remove randon shit * ndw stuff is all top level importable, move types to utils * fix black * fix import * matplotlib migration docs * wording * uncommnet iw for docs conf.py * update example * update * update * temp skip nds_extras in docs * temp comment out vis selector example * fix usage args * just make usage a kwarg to ImageGraphic * update api docs stuff * update example * fix docstrings etc. * type annots * ndw import stuff * docs for nds extras * docs reqs * nds extras * ReferenceIndices.pop_dims * graphics stubs * fix for None data, remove delete_ndgraphic() for now * collections stubs * ndw docs, WIP * ndw guide * types fix * better axes ticks text outline, WIP on ndwidget playback in docs, stubs * ndw animate in docs works * fix display_dims bug * remove tranpose from ArrayProtocol * bugfix, spatial func stuff * ndw tests * black * more balck * add example * fixes * fix * update * other CI fiel * skip ndw and imgui test if not present * garbage collection of graphics in imgui popup * temp imgui ping * fix * update CI * Update screenshots.yml * update screenshots * fix docstrings * update test thresholds, remove line dashing from screenshot tests * update api docs * another example * skip testing an example * higher RMSE threshold on mac because I am so done with this shit * Update ci.yml * Update ci-pygfx-release.yml --------- Co-authored-by: Amol Pasarkar <amolpasarkar@gmail.com> Co-authored-by: Flynn <75346097+FlynnOConnell@users.noreply.github.com> Co-authored-by: Caitlin Lewis <caitlin9165@gmail.com>

An actually maintainable config system.
add_<graphic>methods by using fancie descriptor logicUser API
Some presets:
Presets can be stacked, they "merge" into each other, so
fpl.presets.light()followed byfpl.presets.compact()will apply both. This of course has to be done sensibly,fpl.presets.light()followed byfpl.presets.dark()just reverts things. Each preset just sets config values, that's all.Registering a class to the config system
To summarize
Rules:
@global_config.registerconfigset toglobal_config.descriptor@global_config.seton them, or its parent classes must have them.@global_config.register, however if this subclass is overriding methods in the parent class which have@global_config.setI don't think it has to be registered but we don't have a test case for this in fastplotlib AFAIK.add_line(data=...)will inject the colors from theglobal_config, if we doadd_line(data=..., colors="r")the explicitly providedcolors="r"will be used and the config value will be ignored.All rules above are necessary for a class to be in the config system.
Other notes:
@global_config.setuses its parent's config values. For exampleImguiFigureis registered, but theImguiFigureconstructor has no@global_config.setso it'll use values from the parent, i.e. the config object is shared.How this works
This is some fancie python. Relies on the fact that the class decorator executes after the class has been created, and therefore after all the method decorators have also been executed. So we populate a "_pending" list with method defaults, and then the class
@config.registerdecorator triggers that class to be registered with those defaults for its methods.The rest is really just some parsing, and the actual "config objects", i.e.
LineGraphic.configis a dataclass so that you can tab complete, and thenLineGraphic.config.inititself is also a dataclass with each argument-value pair for that method, so users can just doLineGraphic.config.init.colors = "r". This is much better than matplotlib rcParams where everything is just a string you have to remember: https://matplotlib.org/stable/users/explain/customizing.html#runtime-rc-settingsAlso relies on proper namespacing of the methods using the qualifying name and making sure everything matches up at all levels of this process, relies on the same system that
pickleuses for knowing the identity of classes it serializes (i.e.__qualname__), straight from the horse's mouth:https://github.com/python/cpython/blob/978fba58aef347de4a1376e525df2dacc7b2fff3/Lib/pickle.py#L1050-L1062
Better Mixin with descriptors!
The script to generate the mixin class with
add_<graphic>()methods is gone! 🥳 . It's replaced with elegant descriptors 😄https://github.com/fastplotlib/fastplotlib/blob/09dfc8d7d51ad97fe700c91f543aa9bf5cc326e9/fastplotlib/layouts/_graphic_methods_mixin.py
Video
config-2026-09-14_01.50.36.mp4
Didn't show in the video but obviously if a value is provided for an argument it'll use that and not the config value:
This PR is fully organic written by a human. Claude was producing completely useless garbage for this task.