make (partial) disaggregation easier #199

Activity

1. gdementen

   added

   enhancement

   on Apr 4, 2017
2. gdementen

   changed the title [-]make reverse aggregation easier[/-] [+]make disaggregation easier[/+] on Apr 4, 2017
3. gdementen commented on Apr 4, 2017

   

   gdementen

   on Apr 4, 2017

   ContributorAuthor

   More actions

   see #200, #202 and #203 for linked issues.
4. gdementen commented on Apr 5, 2017

   

   gdementen

   on Apr 5, 2017

   ContributorAuthor

   More actions

   The loop can be rewritten more generically as:

   >>> for g in group:
   ...     start, stop = g.eval().split(':')
   ...     target = a[start:stop]
   ...     expand[g, target] = 1 / len(target)
5. gdementen commented on Apr 5, 2017

   

   gdementen

   on Apr 5, 2017

   ContributorAuthor

   More actions

   and (agg * expand).sum(group) is equivalent to simply agg @ expand in this case
6. gdementen commented on Apr 6, 2017

   

   gdementen

   on Apr 6, 2017

   Last edited by gdementen

   ContributorAuthor

   More actions

   A first proof of concept:

   def disag_array(array, source_axis, target_axis, mapping=None, fixoverlap=True):
       source_axis = array.axes[source_axis]
       disag_array = zeros((source_axis, target_axis))
       for source_group in source_axis:
           source_label = source_group.eval()
           target_labels = source_label if mapping is None else mapping[source_label]
           target_group = target_axis[target_labels]
           disag_array[source_group, target_group] = 1 / len(target_group)
       if fixoverlap:
           disag_array /= (disag_array > 0).sum(source_axis)
       return disag_array
   
   # make a method out of this
   def disag(array, source_axis, target_axis, mapping=None):
       return array @ disag_array(array, source_axis, target_axis, mapping)

   then

   >>> arr = ndtest(3)
   >>> arr
   a | a0 | a1 | a2
     |  0 |  1 |  2
   >>> agg = arr.sum('a1:a2;a0:a1').rename('a', 'group')
   >>> agg
   group | a1:a2 | a0:a1
         |     3 |     1
   >>> disag(agg, x.group, arr.a)
   a |  a0 |  a1 |  a2
     | 0.5 | 1.0 | 1.5
7. gdementen commented on Apr 6, 2017

   

   gdementen

   on Apr 6, 2017

   ContributorAuthor

   More actions

   see also:
   http://stackoverflow.com/questions/12223689/how-can-i-efficiently-disaggregate-data-in-a-dataframe-given-a-set-of-weights
8. gdementen commented on Apr 25, 2017

   

   gdementen

   on Apr 25, 2017

   Last edited by gdementen

   ContributorAuthor

   More actions

   Here is a new version:

   def disag_array(array, source_axis, groups=None, target_axis=None, fixoverlap=True):
       source_axis = array.axes[source_axis]
       if isinstance(groups, collections.Sequence):
           assert len(groups) == len(source_axis)
           if target_axis is None:
               target_axis = Axis(groups[0].axis, np.unique(np.concatenate([g.eval() for g in groups])))
           groups = dict(zip(source_axis, groups))
       if isinstance(groups, collections.Mapping):    
           if target_axis is None:
               groups = [groups[source_group.eval()] for source_group in source_axis]
               groups_labels = [g.eval() if isinstance(g, Group) else g for g in groups]
               target_name = groups[0].axis if isinstance(groups[0], Group) else None
               target_axis = Axis(target_name, np.unique(np.concatenate(groups_labels)))
       if target_axis is None:
           raise ValueError('must specify groups, target_axis or both')
   
       disag_array = zeros((source_axis, target_axis))
       for source_group in source_axis:
           target_labels = source_group.eval() if groups is None else groups[source_group]
           # make sure we have a group in case mapping returned raw labels
           target_group = target_axis[target_labels]
           disag_array[source_group, target_group] = 1 / len(target_group)
       if fixoverlap:
           disag_array /= (disag_array > 0).sum(source_axis)
       return disag_array
   
   # make a method out of this
   def disag(array, source_axis, groups=None, target_axis=None):
       return (array * disag_array(array, source_axis, groups, target_axis)).sum(source_axis)

   then:

   >>> disag(agg2, x.group, (arr.a['a1:a2'], arr.a['a0:a1']))
   a |  a0 |  a1 |  a2
     | 0.5 | 1.0 | 1.5
   
   >>> disag(agg2, x.group, target_axis=arr.a)
   a |  a0 |  a1 |  a2
     | 0.5 | 1.0 | 1.5
   
   >>> disag(agg, agegr, age.by(5))

   points to improve:

   • I would like to support passing a single {agg_label: group} mapping. e.g.

   >>> disag(agg, dict(zip(agegr, age.by(5))))
   >>> disag(agg, {'a1:a2': ['a1', 'a2'], 'a0:a1': ['a0', 'a1']})

   • np.unique sorts the labels which we should not (at least by default)
   • the mapping is a bit fragile vs Group or raw labels as keys or as values. We should support the 4 different combinations.
9. gdementen commented on May 8, 2017

   

   gdementen

   on May 8, 2017

   Last edited by gdementen

   ContributorAuthor

   More actions

   Note that the @ optimization currently only works when the disag array is 2D, mostly due to the fact that @ uses the axes positions. It should be possible to transpose to make it work in all cases. Not sure it is worth it though.
10. alixdamman

    added this to the nice_to_have milestone on Mar 7, 2018
11. alixdamman

    added

    difficulty: high

    on Mar 7, 2018
12. gdementen

    modified the milestones: nice_to_haveThis milestone has been deleted, 0.31 on Jan 9, 2019
13. gdementen

    removed this from the 0.31 milestone on Aug 1, 2019
14. alixdamman

    added this to the nice_to_have milestone on Oct 10, 2019
15. gdementen

    changed the title [-]make disaggregation easier[/-] [+]make (partial) disaggregation easier[/+] on Mar 20, 2023
16. gdementen commented on Mar 20, 2023

    

    gdementen

    on Mar 20, 2023

    Last edited by gdementen

    ContributorAuthor

    More actions

    Katia needs this too but to disaggregate only a few labels. I don't know if the above code already works for that case or not. We should make sure it works though.
17. gdementen

    added

    priority: high

    on Mar 20, 2023
18. gdementen commented on Mar 20, 2023

    

    gdementen

    on Mar 20, 2023

    Last edited by gdementen

    ContributorAuthor

    More actions

    The implementation so far assumes the aggregated cells correspond to sums. We should support not dividing by the length of the group (if the aggregation was a mean).

    Are there other ways/methods to automatically infer the values (possibly user-defined function???).

    If users want an uneven split, I guess we should redirect them (probably worth mentioning in the disag function documentation) to use an explicit disag array.

    We could also (probably in addition to mentioning the explicit disag array) provide some special syntax to create it (for disaggregating only a few labels, providing a disag array would seem overkill).

    >>> disag(agg, {'a1:a2': {'a1': 0.4, 'a2': 0.6}, 'a0:a1': {'a0': 0.7, 'a1': 0.3}})
19. gdementen commented on Oct 2, 2023

    

    gdementen

    on Oct 2, 2023

    Last edited by gdementen

    ContributorAuthor

    More actions

    Partial disaggregation is done quite often in user models:

    >>> arr = ndtest('axis_v1=a0,a12,a3')
    >>> arr
    axis_v1  a0  a12  a3
              0    1   2
    >>> arr.rename('axis_v1', 'axis_v2').set_labels('axis_v2', {'a12': 'a1'}).insert(0, after='a1', label = 'a2')
    axis_v2  a0  a1  a2  a3
              0   1   0   2

    It would be nice if we had something nicer for this. The special syntax above would help:

    >>> disag(arr, {'a12': {'a1': 1, 'a2': 0}})
    axis_v2  a0  a1  a2  a3
              0   1   0   2