implement a way to deduplicate labels easily #444

Activity

1. gdementen

   added

   enhancement

   on Sep 22, 2017
2. gdementen commented on Sep 25, 2017

   

   gdementen

   on Sep 25, 2017

   ContributorAuthor

   More actions

   Here is the code Bernard is currently using to aggregate all duplicate labels. We need to make this easier (and more generic obviously).

   def agg_dupl(mat):
       l_c = []
       for i, p in enumerate(mat.prods):
           l_c.append((p + '_' + str(i)) if p in l_c else str(p))
       mat = mat.set_labels(x.prods, l_c)
       agg = tuple(mat.prods.startswith(code) >> code for code in mat.prods.matches('^.....$'))
       return mat.sum(agg)

   Here is a first step.

   class Axis(object):
       def deduplicate(self):
           # TODO: handle non-string labels (unsure how!? add a user-defined constant?)
           seen = set()
           labels = []
           for i, label in enumerate(self.labels):
               if label in seen:
                   label = '{}_{}'.format(label, i)
               else:
                   seen.add(label)
               labels.append(label)
           return Axis(labels, self.name)
   
   def agg_dupl(mat):    
       mat = mat.set_axes('prods', mat.prods.deduplicate())
       agg = tuple(mat.prods.startswith(code) >> code for code in mat.prods.matches('^.....$'))
       return mat.sum(agg)

   Remaining problems:

   • the aggregation relies on the fact his codes are always exactly the same length
   • we should provide a way to "deduplicate()" all axes at once

   We should be able to write something like:

   def agg_dupl(mat):
       mat = mat.deduplicate()
       agg = tuple(mat.prods.startingwith(code) >> code for code in mat.prods[~mat.prods.endswith('/DUPE/')])
       return mat.sum(agg)

   or even

   def agg_dupl(mat):
       mat = mat.deduplicate()
       agg = tuple(mat.prods.startingwith(code) >> code for code in ~mat.prods.endingwith('/DUPE/'))
       return mat.sum(agg)

   Remaining problems:

   • ideally the .deduplicate() step should not be necessary. It should be possible to easily create a positional-based agg directly.
3. alixdamman

   added this to the 0.30 milestone on Jan 25, 2018
4. gdementen commented on Mar 14, 2018

   

   gdementen

   on Mar 14, 2018

   ContributorAuthor

   More actions

   This particular problem (aggregating all duplicates) can be solved much more elegantly by a groupby.

   agg = mat.groupby(mat.prods).sum()
   # and/or
   agg = mat.sum(mat.prods.groupunique())

   The question is then whether de-duplication has value outside of aggregating the duplicates. I can see some cases where this is true, but it is lower priority I think.
5. gdementen

   self-assigned this

   on Mar 23, 2018
6. gdementen

   added 2 commits that reference this issue on Jul 20, 2018

   implemented Axis.group_unique and Axis.groupby(column) (closes larray…

   ...

   This issue will close once commit 0cc272f is merged into the 'master' branch.

   0cc272f

   implemented Axis.group_unique and Axis.groupby(column) (closes larray…

   ...

   This issue will close once commit 66ecfb3 is merged into the 'master' branch.

   66ecfb3
7. gdementen

   removed their assignment

   on Aug 23, 2018
8. alixdamman

   modified the milestones: 0.30, 0.32This milestone has been deleted on Oct 16, 2018
9. alixdamman

   removed this from the 0.32This milestone has been deleted milestone on Aug 1, 2019
10. alixdamman

    added this to the nice_to_have milestone on Oct 10, 2019