Apply, reduce and group

Running functions over rows and columns, and grouping.

Apply

apply

apply(func, by='row', keylist=None, **kwargs)

Apply a function to each row and collect the results in a new Daf.

Use it to convert one table into another, with a plain Python function that turns one row into the new row. This Daf is not changed. Only the new Daf gets rows.

The function gets a row and returns what to add to the new Daf:

• A dict adds one row.
• A list of dicts adds a row for each dict, so one row can become several.
• None, an empty dict or an empty list adds nothing, so the row is left out.

The new Daf takes its columns from the first row returned. A later dict is placed by column name, and a column it lacks is NULL. A key that the first row did not have is dropped without an error. So when the columns are known, have the function return all of them in every row. The new Daf has no keyfield.

With by='table' the function gets the whole Daf, and its result is returned as it is. Use that to run any function on the table. by='col' is not supported.

The function gets the rows as dicts, or as KeyedList objects if itermode is keyedlist. The extra keyword arguments are passed on to it. To work on only some rows, give the keylist, or select them first.

Parameters:

Name	Type	Description	Default
func	Callable[..., Any]	The function to apply. It takes a row, or the Daf, and the keyword arguments.	required
by	str	row to apply to each row, or table to apply to the whole Daf.	'row'
keylist	T_la | T_lota | None	Keys of the rows to include. All rows if None. This may be removed.	None
**kwargs	Any	Keyword arguments passed on to the function.	{}

Returns:

Type	Description
'Daf'	The new Daf, or the result of the function for by='table'.

Raises:

Type	Description
NotImplementedError	by is col, or is not recognized.

Examples:

>>> d = Daf(lol=[['a', 1, 10], ['b', 2, 20], ['a', 3, 30]], cols=['g', 'x', 'y'])
>>> d.apply(lambda row: {'g': row['g'].upper(), 'z': row['y'] * 2})

| g  | z  |
| -: | -: |
|  A | 20 |
|  B | 40 |
|  A | 60 |

%% daf rows=3; cols=2; keyfield=''; name=''

>>> d.apply(lambda row: row if row['y'] > 15 else None)

| g  | x  | y  |
| -: | -: | -: |
|  b |  2 | 20 |
|  a |  3 | 30 |

%% daf rows=2; cols=3; keyfield=''; name=''

>>> d.apply(lambda row: [{'g': row['g'], 'part': 1}, {'g': row['g'], 'part': 2}] if row['x'] == 2 else None)

| g  | part |
| -: | ---: |
|  b |    1 |
|  b |    2 |

%% daf rows=2; cols=2; keyfield=''; name=''

apply_in_place

apply_in_place(func, by='row', rowkeys=None, **kwargs)

Apply a function to each row and store the results in this Daf.

With by='row' the function gets each row as a dict. It must return a dict. Its values are written back by column name. A column that the dict lacks keeps its value, and a key that is not a column is ignored. The order of the keys does not matter, and the row keeps its length.

With by='row_klist' the function gets each row as a KeyedList. It changes the row and returns nothing, so no dict is built for each row. The cost of row grows with the number of columns, because each row is turned into a dict of all its columns. The cost of row_klist does not. In a test that changed one column, the two took the same time at about 6 columns. At 1,000 columns and 2,000 rows, row_klist took 0.005 s and row took 0.245 s. With 3 columns and 200,000 rows, row was faster, 0.25 s against 0.30 s. Add or delete no keys in the KeyedList, because that changes the length of the row, not the columns.

The key index is rebuilt when it is next needed. Use apply() to get a new Daf.

Parameters:

Name	Type	Description	Default
func	Callable[..., T_ma | None]	The function to apply to each row. It takes a row and the keyword arguments.	required
by	str	row or row_klist.	'row'
rowkeys	T_la | T_lota | None	Keys of the rows to include. All rows if None. The Daf needs a keyfield.	None
**kwargs	Any	Keyword arguments passed on to the function.	{}

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
ValueError	With by='row' the function returned None.
NotImplementedError	by is not row or row_klist.

Examples:

>>> d = Daf(lol=[['a', 1, 10], ['b', 2, 20], ['a', 3, 30]], cols=['g', 'x', 'y'])
>>> _ = d.apply_in_place(lambda row: {**row, 'y': row['y'] + 1})
>>> d.col('y')
[11, 21, 31]
>>> _ = d.apply_in_place(lambda row: row.__setitem__('y', 0), by='row_klist')
>>> d.col('y')
[0, 0, 0]

update_row staticmethod

update_row(row, da)

Update a row with the items of a dict, and return the row.

This is a small helper to use inside apply(), as in d.apply(lambda row: Daf.update_row(row, {'z': 0})). It is a static method.

Parameters:

Name	Type	Description	Default
row	T_ma	The row, as a dict. It is changed.	required
da	T_da	The items to put in the row.	required

Returns:

Type	Description
T_ma	The same row.

Examples:

>>> Daf.update_row({'a': 1}, {'b': 2})
{'a': 1, 'b': 2}

apply_colwise

apply_colwise(target_col, func, *, default=0)

Compute one column from the other columns of each row, in place.

The function gets each row, as a dict, and returns the value for target_col. If the function raises an error for a row, that row gets default. If target_col is not a column, it is added at the right, and the rows are filled with default first. Use it for a ratio or a total of other columns.

Parameters:

Name	Type	Description	Default
target_col	str	The column to store the result in.	required
func	Callable[[T_da], Any]	A function that takes a row and returns the value.	required
default	Any	The value for a row where the function raises an error.	0

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 2], [3, 0]], cols=['a', 'b'])
>>> d.apply_colwise('ratio', lambda row: row['a'] / row['b'], default=-1.0)

| a  | b  | ratio |
| -: | -: | ----: |
|  1 |  2 |   0.5 |
|  3 |  0 |  -1.0 |

%% daf rows=2; cols=3; keyfield=''; name=''

apply_to_col

apply_to_col(col, func, **kwargs)

Replace each value of a column by the result of a function, in place.

Parameters:

Name	Type	Description	Default
col	str	The column name.	required
func	Callable	A function that takes a value and returns the new value.	required
**kwargs	Any	Keyword arguments passed on to the function with each value.	{}

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 5], [2, 6]], cols=['a', 'b'])
>>> _ = d.apply_to_col('b', lambda value: value * 2)
>>> d.col('b')
[10, 12]
>>> _ = d.apply_to_col('b', lambda value, factor: value * factor, factor=10)
>>> d.col('b')
[100, 120]

apply_formulas

apply_formulas(formulas_daf)

Fill cells from spreadsheet like formulas, in place.

formulas_daf is a Daf of the same shape. Each cell holds a Python expression as text. An empty cell is skipped. The result of each expression is stored in the same cell of this Daf. The formulas are evaluated again and again until no cell changes, so a cell may use the result of another. If the cells still change after 100 passes, it raises RuntimeError. This is a limit on the passes, not a check for circular formulas. A circle of formulas that settles, such as two cells that copy each other, raises nothing.

In a formula, $d is this Daf, $r is the row number of the cell and $c is its column number. References are absolute unless you build them from $r and $c. So sum($d[$r, :$c]) is the sum of the cells to the left in the same row.

Other examples:

$d[14,20]+$d[15,25]       the sum of two cells
max(0,$d[($r-1),$c])      the cell above, but not below 0
$d[($r-1),$c] * 0.15      15 percent of the cell above

Warning: the formulas are run with eval(). Never use formulas from a source you do not trust.

An error in a formula prints the cell and the formula, and is raised again. The retmode of this Daf is restored, and the key index is rebuilt when it is next needed. The cells that were already changed stay changed.

Parameters:

Name	Type	Description	Default
formulas_daf	'Daf'	The formulas, with the same shape as this Daf.	required

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
RuntimeError	The shapes differ, or the formulas never settle.

Examples:

>>> d = Daf(cols=['A', 'B', 'C'], lol=[[1, 2, 0], [4, 5, 0], [7, 8, 0], [0, 0, 0]])
>>> f = Daf(cols=['A', 'B', 'C'], lol=[
...     ['', '', 'sum($d[$r,:$c])'],
...     ['', '', 'sum($d[$r,:$c])'],
...     ['', '', 'sum($d[$r,:$c])'],
...     ['sum($d[:$r,$c])', 'sum($d[:$r,$c])', 'sum($d[:$r,$c])']])
>>> _ = d.apply_formulas(f)
>>> d

| A  | B  | C  |
| -: | -: | -: |
|  1 |  2 |  3 |
|  4 |  5 |  9 |
|  7 |  8 | 15 |
| 12 | 15 | 27 |

%% daf rows=4; cols=3; keyfield=''; name=''

Reduce

reduce

reduce(
    func,
    by="row",
    cols=None,
    initial_da=None,
    indirect_col="",
    silent_error=False,
    **kwargs,
)

Combine all the rows into one result, using a function.

The function gets each row, and the result so far, and returns the new result. A sum or a count is a reduction. With by='row', the function is called as func(row, result, cols=cols, **kwargs). The result starts as 0 for each of the columns, or as initial_da, which is changed in place. It returns a dict with every column. Columns that are not in cols are empty. A key of the result that is not a column of the Daf is dropped, so a reduction by row gives one value for each column. For a result of another shape, use by='table', or a loop.

A plain sum, with sum_da() or sum_kl() and no other options, adds down each column instead of across each row. That is about 5 times faster, and gives the same totals. The function is then not called for each row.

With by='row', each row is a new KeyedList that points at the row of the table. It reads like a dict, as in row['x'] or row.get('x', 0). Writing to it changes the table. A function that needs a real dict, for example to pass to json.dumps(), can call dict(row). A row that is shorter or longer than the columns gets a KeyedList of a copy, padded with NULL or cut to fit.

The standard functions are sum_kl() and count_values_kl(). They read the KeyedList directly. Daf.sum_da and Daf.count_values_da are replaced by them here, since they give the same results.

With by='col' the function gets each column as a list and the result so far as a list. With by='table' it gets this Daf and cols, and its result is returned as it is. With by='sparse_row', rows are read from the dict in indirect_col, and the result starts as initial_da or an empty dict.

An error in the function stops the reduction. With silent_error=True the row is skipped. The result may then be missing rows, with no sign of it. An empty Daf gives an empty dict for by='row'.

To reduce part of the table, select the rows or columns first, or give cols.

Parameters:

Name	Type	Description	Default
func	Callable[..., Any]	The reduction function. For example Daf.sum_da.	required
by	str	row, col, table or sparse_row.	'row'
cols	T_cs | None	The columns included in the reduction. All columns if None.	None
initial_da	T_ma | None	The result to start from, instead of zeros.	None
indirect_col	str	A column that holds a dict. Required for sparse_row.	''
silent_error	bool	If True, a row for which the function raises is skipped.	False
**kwargs	Any	Keyword arguments passed on to the function.	{}

Returns:

Type	Description
T_ma | T_la	A dict for row, table and sparse_row. A list for col.

Raises:

Type	Description
ValueError	sparse_row is used with no indirect_col.
NotImplementedError	by is not recognized.

Examples:

>>> d = Daf(lol=[['a', 1, 10], ['b', 2, 20], ['a', 3, 30]], cols=['g', 'x', 'y'])
>>> d.reduce(Daf.sum_da, cols=['x', 'y'])
{'g': '', 'x': 6, 'y': 60}

daf_sum

daf_sum(by='row', cols=None, indirect_col='', **kwargs)

Add up the columns, using reduce() and sum_kl().

Cells that cannot be added, such as text, are skipped with no error. See sum_kl for what that means for a bad number. A column that is not in cols is empty in the result.

Parameters:

Name	Type	Description	Default
by	str	How the function is applied. See reduce().	'row'
cols	T_cs | None	The columns to sum. All if None.	None
indirect_col	str	A column that holds a dict. Required for sparse_row.	''
**kwargs	Any	Keyword arguments passed on to reduce().	{}

Returns:

Type	Description
T_ma	A dict with a total for each column.

Examples:

>>> d = Daf(lol=[['a', 1, 10], ['b', 2, 20], ['a', 3, 30]], cols=['g', 'x', 'y'])
>>> d.daf_sum(cols=['y'])
{'g': '', 'x': '', 'y': 60}

sum

sum(colnames_ls=None, numeric_only=False)

Total the columns, and return a dict of the totals.

Each total starts as a float. Empty cells are skipped. Text that is a number, such as '2', is converted and added. Other text raises ValueError, so give colnames_ls to leave those columns out.

reduce(Daf.sum_da) adds differently. It skips every value that is not a number, including the text '2', and never raises. So the two can give different totals for a column that holds text. See the example. With numeric_only, and dtypes of int or float, only those columns are totaled, and a cell that is not a number counts as 0. The totals are converted to the dtypes, if the Daf has them.

Parameters:

Name	Type	Description	Default
colnames_ls	T_cs | None	The columns to total. All if None.	None
numeric_only	bool	If True, total only the columns with an int or float dtype.	False

Returns:

Type	Description
dict	A dict that maps each column name to its total.

Raises:

Type	Description
ValueError	A cell cannot be converted to a number.

Examples:

>>> Daf(lol=[[1, 10], [2, 20]], cols=['x', 'y']).sum()
{'x': 3.0, 'y': 30.0}
>>> mixed = Daf(cols=['n'], lol=[[1], ['2'], [''], [3.5]])
>>> mixed.sum(['n']), mixed.reduce(Daf.sum_da, cols=['n'])
({'n': 6.5}, {'n': 4.5})

sum_np

sum_np(colnames_ls=None)

Total the columns with NumPy, and return a dict of the totals.

This needs NumPy. A blank, None or NaN cell counts as 0, as it does in sum(). Use colnames_ls to pass only the columns that hold numbers. A column that holds text raises TypeError and names the column.

Parameters:

Name	Type	Description	Default
colnames_ls	T_cs | None	The columns to total. All if None.	None

Returns:

Type	Description
dict	A dict that maps each column name to its total. An empty Daf gives an empty dict.

Raises:

Type	Description
KeyError	A name in colnames_ls is not a column.
TypeError	A column holds text or other values that cannot be added.

Examples:

>>> Daf(lol=[[1, 10], [2, 20]], cols=['x', 'y']).sum_np()
{'x': 3, 'y': 30}
>>> Daf(lol=[[1, 10], [2, '']], cols=['x', 'y']).sum_np()
{'x': 3, 'y': 10}

sum_kl staticmethod

sum_kl(
    row_kl,
    reduction_da,
    *,
    cols=None,
    astype=None,
    is_sparse=False,
    diagnose=False,
)

Add the values of a row to a running total. This is the standard sum for reduce().

It gives the same result as sum_da(), and is faster on the rows that reduce() gives it, which are KeyedList objects. It reads the values of the row directly, instead of looking up each one through the row. reduce() uses it in place of sum_da(), so a caller that passes Daf.sum_da gets this speed too.

A value that cannot be added, such as text or an empty cell, is skipped, with no error. That keeps the sum fast, and lets a sum of all columns pass over the text columns. But a bad number, such as abc in an int column, is then left out of the total with no sign. Call check_schema() first to find such cells.

A column that the row does not have adds 0. With astype or is_sparse, or a row that is not a KeyedList, the work is passed to sum_da().

Parameters:

Name	Type	Description	Default
row_kl	Any	The current row.	required
reduction_da	T_ma	The running total. It is changed.	required
cols	T_cs | None	The columns to add. All if None.	None
astype	Type | None	A type to convert each value to before adding.	None
is_sparse	bool	If True, the row may hold only some of the columns.	False
diagnose	bool	Not used.	False

Returns:

Type	Description
T_ma	The running total.

Examples:

>>> from daffodil.keyedlist import KeyedList
>>> Daf.sum_kl(KeyedList(['x', 'y'], [1, 'abc']), {'x': 10, 'y': 0}, cols=['x', 'y'])
{'x': 11, 'y': 0}

sum_da staticmethod

sum_da(
    row_da,
    reduction_da,
    *,
    cols=None,
    astype=None,
    is_sparse=False,
    diagnose=False,
)

Add the values of a row to a running total. Use it with reduce().

This is a static method. A value that cannot be added, such as text or an empty cell, is skipped with no error. See sum_kl for what that means for a bad number. The total is changed and returned. With astype, each value is first converted to that type. reduce() uses sum_kl() in its place.

Parameters:

Name	Type	Description	Default
row_da	T_ma	The current row.	required
reduction_da	T_ma	The running total. It is changed.	required
cols	T_cs | None	The columns to add. All if None.	None
astype	Type | None	A type to convert each value to before adding.	None
is_sparse	bool	If True, the row may hold only some of the columns.	False
diagnose	bool	Not used.	False

Returns:

Type	Description
T_ma	The running total.

Examples:

>>> Daf.sum_da({'x': 1, 'y': 2}, {'x': 10, 'y': 0}, cols=['x', 'y'])
{'x': 11, 'y': 2}

diff_da staticmethod

diff_da(d1_da, d2_da, keys=None)

Subtract one dict from another, for the keys you name.

This is a static method. A key that is missing, or an empty value, counts as 0. Keys that you do not name are left out. The values must be numbers.

Parameters:

Name	Type	Description	Default
d1_da	T_ma	The first dict.	required
d2_da	T_ma	The dict to subtract.	required
keys	T_ls | str | None	The key or keys to include. If None, the result is empty.	None

Returns:

Type	Description
T_da	A dict of the differences.

Examples:

>>> Daf.diff_da({'a': 5, 't': 'x'}, {'a': 2, 't': 'y'}, keys=['a'])
{'a': 3}

count_values_kl staticmethod

count_values_kl(
    row_kl, reduction_da, cols, *, omit_nulls=False
)

Add one row to running counts of the values in each column. This is the standard count for reduce().

It gives the same result as count_values_da(), and is faster on the rows that reduce() gives it, which are KeyedList objects. reduce() uses it in place of count_values_da().

A cell that holds a dict is passed to count_values_da(), one column at a time, as is a row that is not a KeyedList. A value that cannot be a dict key, such as a list, raises TypeError. Leave a column of lists out with cols.

Parameters:

Name	Type	Description	Default
row_kl	Any	The current row.	required
reduction_da	T_da	The running counts. They are changed.	required
cols	T_cs	The columns to count.	required
omit_nulls	bool	If True, empty cells are not counted.	False

Returns:

Type	Description
T_da	The running counts.

Raises:

Type	Description
TypeError	A cell holds a value that cannot be counted, such as a list.

Examples:

>>> from daffodil.keyedlist import KeyedList
>>> Daf.count_values_kl(KeyedList(['g'], ['a']), {'g': {'a': 2}}, ['g'])
{'g': {'a': 3}}

count_values_da staticmethod

count_values_da(
    row_da, reduction_da, cols, *, omit_nulls=False
)

Add one row to running counts of the values in each column. Use it with reduce().

This is a static method. The counts are a dict that maps each column to a dict of value and count. The counts are changed and returned. A cell that holds a dict is added to the counts for that column with sum_da(). Use omit_nulls to skip empty cells.

A value that cannot be a dict key, such as a list, raises TypeError. There is no check for it beforehand, to keep the count fast. Leave a column of lists out with cols. reduce() uses count_values_kl() in its place.

Parameters:

Name	Type	Description	Default
row_da	T_ma	The current row.	required
reduction_da	T_da	The running counts. They are changed.	required
cols	T_cs	The columns to count.	required
omit_nulls	bool	If True, empty cells are not counted.	False

Returns:

Type	Description
T_da	The running counts.

Raises:

Type	Description
TypeError	A cell holds a value that cannot be counted, such as a list.

Examples:

>>> Daf.count_values_da({'g': 'a'}, {}, ['g'])
{'g': {'a': 1}}

sum_dodis staticmethod

sum_dodis(this_dodi, accum_dodi)

Add one dict of dicts of numbers into another, in place.

This is a static method. For each key, the numbers of the inner dicts are added with sum_da(). A key that is new is stored as it is, not copied.

Parameters:

Name	Type	Description	Default
this_dodi	T_dodi	The counts to add.	required
accum_dodi	T_dodi	The running totals. They are changed.	required

Examples:

>>> total = {'c': {'x': 2, 'y': 1}}
>>> Daf.sum_dodis({'c': {'x': 1}}, total)
>>> total
{'c': {'x': 3, 'y': 1}}

Group

groupby

groupby(
    colname="", colnames=None, omit_nulls=False, cols=None
)

Split the Daf into several Daf instances, one for each value of a column.

The result is a KeyedList of Daf, a klodaf. It works as a dict: groups[key], keys(), items(), len() and in. Its values also have positions, so groups.values()[0] is the first group. Each key is a value found in the column, in the order first seen. Each value is a Daf of the rows that have it, with all columns, or only the columns in cols.

With all columns, a group holds the same row lists as this Daf. Nothing is copied, so it is fast and uses little memory. But a cell changed in a group is also changed here. To keep this Daf as it is, group a copy made with copy(level='editable'). Adding a column to a group makes new rows, and this Daf does not get the column.

Use cols when you need only a few of many columns. The rows are then new lists of the columns kept. The other columns are never copied, and this Daf is not changed. It is much faster for a wide table.

With several columns, as a list, the keys are tuples of their values. See groupby_cols(). With omit_nulls, rows that have an empty value in the column are left out.

Parameters:

Name	Type	Description	Default
colname	str	The column to group by.	''
colnames	T_ls | None	Several columns to group by. Use this or colname.	None
omit_nulls	bool	If True, leave out rows that have an empty value.	False
cols	T_cs | None	The columns that each group keeps, in this order. If None, all columns. A keyfield is kept only if all its columns are kept.	None

Returns:

Type	Description
T_klodaf	A KeyedList that maps each value, or tuple of values, to a Daf.

Raises:

Type	Description
KeyError	The group column, or a name in cols, is not a column.

Examples:

>>> d = Daf(lol=[['a', 1, 10], ['b', 2, 20], ['a', 3, 30]], cols=['g', 'x', 'y'])
>>> groups = d.groupby('g')
>>> list(groups)
['a', 'b']
>>> groups['a']

| g  | x  | y  |
| -: | -: | -: |
|  a |  1 | 10 |
|  a |  3 | 30 |

%% daf rows=2; cols=3; keyfield=''; name=''

>>> d.groupby('g', cols=['y'])['a']

| y  |
| -: |
| 10 |
| 30 |

%% daf rows=2; cols=1; keyfield=''; name=''

groupby_cols

groupby_cols(colnames, cols=None)

Split the Daf by the values of several columns.

The result is a KeyedList of Daf, as groupby() gives. Each key is a tuple of the values in the columns, even for one column. Each value is a Daf of the rows that have them, with all columns, or only the columns in cols. With all columns, the rows are not copied. With cols, they are new lists of the columns kept, and the other columns are never copied.

Parameters:

Name	Type	Description	Default
colnames	T_ls	The columns to group by.	required
cols	T_cs | None	The columns that each group keeps, in this order. If None, all columns. A keyfield is kept only if all its columns are kept.	None

Returns:

Type	Description
T_klodaf	A KeyedList that maps each tuple of values to a Daf.

Raises:

Type	Description
KeyError	A name in colnames or cols is not a column.

Examples:

>>> d = Daf(lol=[['a', 1, 10], ['b', 2, 20], ['a', 3, 30]], cols=['g', 'x', 'y'])
>>> list(d.groupby_cols(['g']))
[('a',), ('b',)]
>>> d.groupby_cols(['g'], cols=['y'])[('a',)]

| y  |
| -: |
| 10 |
| 30 |

%% daf rows=2; cols=1; keyfield=''; name=''

group_where

group_where(where, *, indirect_col=None)

Group the rows by the result of a function.

The function gets each row and returns a key, or a list of keys, or None. A row goes into the group of each key. A list of keys puts the same row in several groups. None leaves the row out. The groups are Daf instances in a KeyedList, as groupby() gives. The rows are not copied, so changing a cell in a group changes it here too.

To split the rows in two, have the function return True or False. A side with no rows then has no group, so read each side with .get().

Parameters:

Name	Type	Description	Default
where	Callable[[Any], Any]	The function. It gets a row and returns None, a key, or an iterable of keys.	required
indirect_col	str | None	A column that holds a dict, to read names that are not columns from.	None

Returns:

Type	Description
T_klodaf	A KeyedList that maps each key to a Daf.

Examples:

>>> d = Daf(lol=[['a', 1, 10], ['b', 2, 20], ['a', 3, 30]], cols=['g', 'x', 'y'])
>>> groups = d.group_where(lambda row: [row['g'], 'all'])
>>> {key: len(group) for key, group in groups.items()}
{'a': 2, 'all': 3, 'b': 1}

groupby_cols_reduce

groupby_cols_reduce(
    groupby_colnames,
    func,
    by="row",
    reduce_cols=None,
    diagnose=False,
    **kwargs,
)

Group the rows by several columns and reduce each group to one row.

Use it to total the numbers for each combination of a few identifying columns. The result has one row for each combination. It has the group columns first and then the reduce_cols. It has no keyfield. For each group, func is used as in reduce().

With reduce_cols, and by of row or sparse_row, each group holds only those columns, and the indirect_col if there is one. A function that reads another column of the row will not find it.

For example, group by gender, religion and zip code, and sum the counts of several causes in each group. The number of rows is the number of combinations that occur.

Parameters:

Name	Type	Description	Default
groupby_colnames	T_ls	The columns that identify a group.	required
func	Callable[..., Any]	The reduction function. See reduce().	required
by	str	How the function is applied. See reduce().	'row'
reduce_cols	T_la | None	The columns to reduce.	None
diagnose	bool	If True, print progress messages.	False
**kwargs	Any	Keyword arguments passed on to the function.	{}

Returns:

Type	Description
'Daf'	The Daf with one row for each group.

Examples:

>>> d = Daf(lol=[['a', 1, 10], ['b', 2, 20], ['a', 3, 30]], cols=['g', 'x', 'y'])
>>> d.groupby_cols_reduce(['g'], Daf.sum_da, reduce_cols=['y'])

| g  | y  |
| -: | -: |
|  a | 40 |
|  b | 20 |

%% daf rows=2; cols=2; keyfield=''; name=''

groupby_reduce

groupby_reduce(
    colname,
    func,
    by="row",
    reduce_cols=None,
    diagnose=False,
    **kwargs,
)

Group the rows by one column and reduce each group to one row.

The result has one row for each value in the column, and its keyfield is that column. The columns in reduce_cols hold the reduced values. Other columns are empty. For each group, func is used as in reduce().

With reduce_cols, and by of row or sparse_row, each group holds only those columns, and the indirect_col if there is one. The other columns are never copied, which is much faster for a wide table. A function that reads another column of the row will not find it. The result still has all the columns of this Daf, and the columns that are not reduced are empty.

Parameters:

Name	Type	Description	Default
colname	str	The column to group by.	required
func	Callable[..., Any]	The reduction function. See reduce().	required
by	str	How the function is applied. See reduce().	'row'
reduce_cols	T_cs | None	The columns to reduce. All except colname if None.	None
diagnose	bool	If True, print progress messages.	False
**kwargs	Any	Keyword arguments passed on to the function.	{}

Returns:

Type	Description
'Daf'	The Daf with one row for each group.

Examples:

>>> d = Daf(lol=[['a', 1, 10], ['b', 2, 20], ['a', 3, 30]], cols=['g', 'x', 'y'])
>>> d.groupby_reduce('g', Daf.sum_da, reduce_cols=['y'])

| g  | x  | y  |
| -: | -: | -: |
|  a |    | 40 |
|  b |    | 20 |

%% daf rows=2; cols=3; keyfield='g'; name=''

multi_groupby

multi_groupby(
    groupby_colnames, colnames=None, omit_nulls=False
)

Group the rows by each of several columns, one column at a time.

The result is a dict of dicts. The first key is the column. The second key is a value in that column. Each innermost value is a Daf of the rows that have it, with all columns, or only the columns in colnames. This is not a grouping by combinations. Use groupby_cols() for that. The groups are not reduced.

With all columns, the groups hold the same row lists as this Daf, as in groupby(). A cell changed in a group is also changed here, and in the groups of the other columns. To keep this Daf as it is, group a copy made with copy(level='editable').

Use colnames when you need only a few of many columns. The rows are then new lists of the columns kept. The other columns are never copied, and this Daf is not changed. It is much faster for a wide table.

Parameters:

Name	Type	Description	Default
groupby_colnames	T_cs	The columns to group by, one at a time.	required
colnames	T_cs | None	The columns that each group keeps, in this order. If None, all columns. A keyfield is kept only if all its columns are kept.	None
omit_nulls	bool	If True, leave out rows that have an empty value.	False

Returns:

Type	Description
Dict[str, Dict[str, 'Daf']]	A dict that maps each column to a dict of value and Daf.

Raises:

Type	Description
KeyError	A group column, or a name in colnames, is not a column.

Examples:

>>> d = Daf(lol=[['a', 1, 10], ['b', 2, 20], ['a', 3, 30]], cols=['g', 'x', 'y'])
>>> groups = d.multi_groupby(['g', 'x'])
>>> list(groups), list(groups['g'])
(['g', 'x'], ['a', 'b'])
>>> d.multi_groupby(['g'], colnames=['y'])['g']['a']

| y  |
| -: |
| 10 |
| 30 |

%% daf rows=2; cols=1; keyfield=''; name=''

multi_groupby_reduce

multi_groupby_reduce(
    colnames,
    func,
    by="row",
    reduce_cols=None,
    diagnose=False,
    **kwargs,
)

Group by each of several columns and reduce each group to one row.

This is multi_groupby() followed by groupby_reduce() for each column. The result is a dict. Each key is a column. Each value is a Daf with one row for each value of that column, and that column as its keyfield.

With reduce_cols, and by of row or sparse_row, each group holds only those columns, and the indirect_col if there is one. A function that reads another column of the row will not find it. The result still has all the columns of this Daf, and the columns that are not reduced are empty.

Parameters:

Name	Type	Description	Default
colnames	T_cs	The columns to group by, one at a time.	required
func	Callable[..., Any]	The reduction function. See reduce().	required
by	str	How the function is applied. See reduce().	'row'
reduce_cols	T_cs | None	The columns to reduce.	None
diagnose	bool	If True, print progress messages.	False
**kwargs	Any	Keyword arguments passed on to the function.	{}

Returns:

Type	Description
Dict[str, 'Daf']	A dict that maps each column to its Daf.

Examples:

>>> d = Daf(lol=[['a', 1, 10], ['b', 2, 20], ['a', 3, 30]], cols=['g', 'x', 'y'])
>>> d.multi_groupby_reduce(['g'], Daf.sum_da, reduce_cols=['y'])['g']

| g  | x  | y  |
| -: | -: | -: |
|  a |    | 40 |
|  b |    | 20 |

%% daf rows=2; cols=3; keyfield='g'; name=''

reduce_dodaf_to_daf staticmethod

reduce_dodaf_to_daf(
    colname,
    func,
    grouped_dodaf,
    reduce_cols=None,
    diagnose=False,
    all_cols=None,
    **kwargs,
)

Reduce each Daf in a dict to one row, and join the rows in a Daf.

This is the second half of groupby_reduce(). The dict maps the values of a column to Daf instances. Each Daf is reduced with reduce(). The value is stored in the column colname of the row. The result has colname as its keyfield.

Parameters:

Name	Type	Description	Default
colname	str	The column that holds the value of each group.	required
func	Callable[..., Any]	The reduction function. See reduce().	required
grouped_dodaf	T_dodaf	A dict that maps each value to a Daf.	required
reduce_cols	T_la | None	The columns to reduce. All columns except colname if None.	None
diagnose	bool	If True, print progress messages.	False
all_cols	T_ls | None	The columns of the result, in order. Use it when each group holds only some columns, so that the rows have the same columns as for whole groups. The columns that are not reduced are then empty.	None
**kwargs	Any	Keyword arguments passed on to the function.	{}

Returns:

Type	Description
'Daf'	The Daf with one row for each group.

Examples:

>>> groups = {'a': Daf(lol=[[1], [2]], cols=['n']), 'b': Daf(lol=[[3]], cols=['n'])}
>>> result = Daf.reduce_dodaf_to_daf('g', Daf.sum_da, groups)
>>> result

| n  | g  |
| -: | -: |
|  3 |  a |
|  3 |  b |

%% daf rows=2; cols=2; keyfield='g'; name=''

groupsum_daf

groupsum_daf(colname, by='row', reduce_cols=None)

Group by a column and add up the other columns of each group.

This is groupby_reduce() with sum_da().

Parameters:

Name	Type	Description	Default
colname	str	The column to group by.	required
by	str	How the function is applied. See reduce().	'row'
reduce_cols	T_cs | None	The columns to add.	None

Returns:

Type	Description
'Daf'	The Daf with one row for each group.

Examples:

>>> d = Daf(lol=[['a', 1, 10], ['b', 2, 20], ['a', 3, 30]], cols=['g', 'x', 'y'])
>>> d.groupsum_daf('g', reduce_cols=['y'])

| g  | x  | y  |
| -: | -: | -: |
|  a |    | 40 |
|  b |    | 20 |

%% daf rows=2; cols=3; keyfield='g'; name=''

multi_groupsum

multi_groupsum(colnames=None, by='row', reduce_cols=None)

Group by each of several columns, one at a time, and add up the columns.

This is multi_groupby_reduce() with sum_da().

Parameters:

Name	Type	Description	Default
colnames	T_cs | None	The columns to group by. These are required.	None
by	str	How the function is applied. See reduce().	'row'
reduce_cols	T_cs | None	The columns to add.	None

Returns:

Type	Description
Dict[str, 'Daf']	A dict that maps each column to a Daf with one row for each value.

Raises:

Type	Description
ValueError	No colnames are given.

Examples:

>>> d = Daf(lol=[['a', 'x', 1], ['a', 'y', 2], ['b', 'x', 3]], cols=['g', 'k', 'n'])
>>> sums = d.multi_groupsum(colnames=['g', 'k'], reduce_cols=['n'])
>>> sums['g']

| g  | k  | n  |
| -: | -: | -: |
|  a |    |  3 |
|  b |    |  3 |

%% daf rows=2; cols=3; keyfield='g'; name=''

>>> sums['k']

| g  | k  | n  |
| -: | -: | -: |
|    |  x |  4 |
|    |  y |  2 |

%% daf rows=2; cols=3; keyfield='k'; name=''

>>> d.multi_groupsum()
Traceback (most recent call last):
    ...
ValueError: multi_groupsum: colnames is required

Manifest processing

manifest_apply

manifest_apply(
    func,
    load_func,
    save_func,
    by="row",
    cols=None,
    **kwargs,
)

Run a function on each chunk that a manifest lists, and save the results.

A manifest is a Daf in which each row describes one chunk of data. For each row, load_func loads the chunk as a Daf. func is applied to it with by='table', so it gets the loaded Daf and the keyword cols. It returns a tuple of a dict that describes the result chunk and the new Daf. save_func saves the new Daf. The result manifest has one row for each dict.

Parameters:

Name	Type	Description	Default
func	Callable[[T_da, T_la | None], Tuple[T_da, 'Daf']]	Gets a loaded Daf. Returns a dict that describes the result and the new Daf.	required
load_func	Callable[[T_ma], 'Daf']	Loads the chunk that a manifest row describes.	required
save_func	Callable[[T_da, 'Daf'], str]	Saves a new Daf, given the dict that describes it.	required
by	str	Must be table. The default is row, which applies func to each row, so always pass by='table'.	'row'
cols	T_la | None	Passed to func as the keyword cols.	None
**kwargs	Any	Keyword arguments passed on to func.	{}

Returns:

Type	Description
'Daf'	The manifest of the result chunks.

Examples:

>>> chunks = {'a': Daf(cols=['n'], lol=[[1], [2]]), 'b': Daf(cols=['n'], lol=[[10]])}
>>> saved = []
>>> def load(spec):
...     return chunks[spec['chunk']]
>>> def double(daf, cols=None):
...     new = Daf(cols=['n'], lol=[[row['n'] * 2] for row in daf])
...     return {'rows': len(new)}, new
>>> def save(spec, daf):
...     saved.append(daf.to_lod())
...     return 'saved'
>>> manifest = Daf(cols=['chunk'], lol=[['a'], ['b']])
>>> result = manifest.manifest_apply(double, load, save, by='table')
>>> result

| rows |
| ---: |
|    2 |
|    1 |

%% daf rows=2; cols=1; keyfield=''; name=''

>>> saved
[[{'n': 2}, {'n': 4}], [{'n': 20}]]

manifest_reduce

manifest_reduce(
    func, load_func=None, by="row", cols=None, **kwargs
)

Reduce the chunks that a manifest lists into one row.

Each chunk is loaded with load_func and reduced with reduce(). The reductions are put in a Daf, and that is reduced again with the same function. This works for functions such as sum_da() that can be combined in this way.

Parameters:

Name	Type	Description	Default
func	Callable[..., Any]	The reduction function. See reduce().	required
load_func	Callable[[T_ma], 'Daf'] | None	Loads the chunk that a manifest row describes. It is required.	None
by	str	How the function is applied. See reduce().	'row'
cols	T_la | None	The columns to reduce. All columns if None.	None
**kwargs	Any	Keyword arguments passed on to the function.	{}

Returns:

Type	Description
T_da	The reduced row, as a dict.

Raises:

Type	Description
ValueError	No load_func is given.

Examples:

>>> store = {'c1': Daf(lol=[[1, 2], [3, 4]], cols=['a', 'b']), 'c2': Daf(lol=[[10, 20]], cols=['a', 'b'])}
>>> manifest = Daf(lol=[['c1'], ['c2']], cols=['chunk'])
>>> manifest.manifest_reduce(Daf.sum_da, load_func=lambda spec: store[spec['chunk']])
{'a': 14, 'b': 26}

manifest_process

manifest_process(func, **kwargs)

Call a function once for each chunk that a manifest lists.

The function gets the manifest row as a dict, and does its own loading and saving. It returns a dict of information about what it did. These dicts are collected as the rows of the result.

Parameters:

Name	Type	Description	Default
func	Callable[..., T_da]	Gets a manifest row and returns a dict of results.	required
**kwargs	Any	Keyword arguments passed on to the function.	{}

Returns:

Type	Description
'Daf'	A Daf with one row for each returned dict.

Examples:

>>> manifest = Daf(lol=[['c1'], ['c2']], cols=['chunk'])
>>> manifest.manifest_process(lambda spec: {'chunk': spec['chunk'], 'seen': True})

| chunk | seen |
| ----: | ---: |
|    c1 | True |
|    c2 | True |

%% daf rows=2; cols=2; keyfield=''; name=''

Regex and settings

set_col2_from_col1_using_regex_select

set_col2_from_col1_using_regex_select(
    col1, col2="", regex=""
)

Fill a column with the part of another column that a regex selects, in place.

regex must have one pair of parentheses around the part to keep. A cell that does not match gives an empty cell. Give regex as a keyword. Without col2, col1 is changed. If col2 is not a column, it is added at the right.

Parameters:

Name	Type	Description	Default
col1	str	The column to read.	required
col2	str	The column to write. Defaults to col1.	''
regex	str	A regular expression with one group.	''

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 'ab12'], [2, 'cd34']], cols=['id', 's'])
>>> _ = d.set_col2_from_col1_using_regex_select('s', regex=r'(\d+)')
>>> d.col('s')
['12', '34']

apply_replace_regex

apply_replace_regex(col, col2='', replace_regex='')

Change a column with a pattern of the form /find/replace/, in place.

The pattern has three parts that are separated by /. The first is a regular expression. The second is what replaces the part it finds. The groups it finds can be used as \1. A pattern with an empty second part removes the text.

/find//                       remove
/find/replace/                replace
/pre(select)post/\1/          keep only the selected part
/pre(select)post/a\1b/        keep it, with new text around it

The result goes to col2, or to col if there is no col2. A column that is not found does nothing. If col2 is not a column, it is added at the right.

Parameters:

Name	Type	Description	Default
col	str	The column to read.	required
col2	str	The column to write. Defaults to col.	''
replace_regex	str	The pattern.	''

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 'ab12']], cols=['id', 's'])
>>> d.apply_replace_regex('s', replace_regex='/ab//').col('s')
['12']

alter_daf_per_setting

alter_daf_per_setting(
    settingsdict,
    setting_name,
    setting_select_dict,
    silent_error=False,
)

Change this Daf with the replace patterns found in a settings dict.

The setting is a dict, or a list of dicts. Each has a spec_name, a colname and a replace_regex. The ones whose fields match setting_select_dict are used. For each, apply_replace_regex() changes the column. Use it to give different edits to different files, such as fixing ids in one source.

A setting that is None or empty changes nothing. A name that is not in the settings dict raises KeyError, unless silent_error is True.

Parameters:

Name	Type	Description	Default
settingsdict	T_da	A dict of settings.	required
setting_name	str	The key of the setting in the dict.	required
setting_select_dict	dict	Selects which specs apply, such as {'spec_name': 'file1.csv'}.	required
silent_error	bool	If True, a missing setting changes nothing.	False

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
KeyError	The setting is missing and silent_error is False.

Examples:

>>> d = Daf(lol=[['04_1'], ['05_2']], cols=['ballot_id'])
>>> spec = {'spec_name': 'a.zip', 'colname': 'ballot_id', 'replace_regex': r'/^04_/14_/'}
>>> d.alter_daf_per_setting({'fix': [spec]}, 'fix', {'spec_name': 'a.zip'}).col('ballot_id')
['14_1', '05_2']

alter_daf_per_alter_specs_daf

alter_daf_per_alter_specs_daf(alter_specs_daf)

Change this Daf with the replace patterns listed in a Daf.

Each row of alter_specs_daf has a colname and a replace_regex. The pattern is applied to the column with apply_replace_regex(). Give it only the rows that apply, for example by selecting on spec_name first.

Parameters:

Name	Type	Description	Default
alter_specs_daf	'Daf'	The specs, with the columns colname and replace_regex.	required

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[['04_1'], ['05_2']], cols=['ballot_id'])
>>> specs = Daf.from_lod([{'colname': 'ballot_id', 'replace_regex': r'/^04_/14_/'}])
>>> d.alter_daf_per_alter_specs_daf(specs).col('ballot_id')
['14_1', '05_2']

Other

cols_to_dol

cols_to_dol(colname1, colname2)

Make a lookup from the values of one column to the values of another.

For each value in colname1, the result lists the different values that appear with it in colname2, in the order first seen. Use it to see how two columns relate. The values must be hashable. If a name is not a column, or the Daf is empty, the result is empty.

Parameters:

Name	Type	Description	Default
colname1	str	The column of keys.	required
colname2	str	The column of values.	required

Returns:

Type	Description
T_dola	A dict that maps each value of the first column to a list of values of the second.

Examples:

>>> d = Daf(lol=[['a', 'b'], ['b', 'd'], ['a', 'f'], ['b', 'd']], cols=['c1', 'c2'])
>>> d.cols_to_dol('c1', 'c2')
{'a': ['b', 'f'], 'b': ['d']}

insert_dif_row

insert_dif_row(
    irow1, irow2=None, irow_insert=None, cols=None
)

Insert a row that holds the difference of two rows.

The difference is the first row minus the second row, for the numeric columns you name. Other columns of the new row are empty. A cell that is empty counts as 0. By default the second row is the one after the first, and the new row goes between them. Columns that hold text must not be in cols.

Parameters:

Name	Type	Description	Default
irow1	int	The position of the first row.	required
irow2	int | None	The position of the second row. If None, the row after the first.	None
irow_insert	int | None	Where to insert the new row. If None, at irow2.	None
cols	T_ls | None	The columns to subtract. If None, all columns.	None

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 10], [3, 14], [6, 20]], cols=['a', 'n'])
>>> d.insert_dif_row(0)

| a  | n  |
| -: | -: |
|  1 | 10 |
| -2 | -4 |
|  3 | 14 |
|  6 | 20 |

%% daf rows=4; cols=2; keyfield=''; name=''

insert_dif_rows

insert_dif_rows(irows_rli=None, cols=None, offset=0)

Insert a difference row after each of several rows.

For each position, a row is inserted that holds that row minus the next row, as in insert_dif_row(). Do not list the last row. By default all rows are used. With offset=1 the new row goes after the next row, not between them.

Parameters:

Name	Type	Description	Default
irows_rli	T_rli | None	The positions of the first rows. If None, every row but the last.	None
cols	T_ls | None	The columns to subtract. If None, all columns.	None
offset	int	0 inserts between the two rows. 1 inserts after the second.	0

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 10], [3, 14], [6, 20]], cols=['a', 'n'])
>>> d.insert_dif_rows()

| a  | n  |
| -: | -: |
|  1 | 10 |
| -2 | -4 |
|  3 | 14 |
| -3 | -6 |
|  6 | 20 |

%% daf rows=5; cols=2; keyfield=''; name=''