Modifying

Adding, changing, removing and reordering records and columns.

Appending

append

append(
    data_item=None,
    respect_kd=False,
    *,
    lol=None,
    la=None,
    fast=False,
)

Add one row, or several, to the end of the Daf.

This is the general way to add data. What it does depends on what you give it. Give exactly one of data_item, lol and la.

A dict or a KeyedList is one row. It is placed by column name, so its keys may be in any order. A missing key gets NULL. A key that is not a column is dropped.

A list of values is one row, in column order. A short list is padded with NULL. A list with more values than there are columns raises ValueError. With no columns defined, the list is added as it is. A list of lists given as data_item is therefore one row whose cells are lists, which fits only if there are enough columns.

To say what you mean, use the keywords. lol is several rows, each a list in column order, as in append(lol=[[2, 'b'], [3, 'c']]). la is one row, even if its items are lists, as in append(la=[[2, 'b'], [3, 'c']]). Each row of lol follows the length rule above.

A list of dicts is several rows. See extend().

A Daf is several rows. See concat(). Its columns must match.

An empty dict, list or Daf adds nothing.

By default the keyfield is not checked, so a key that is already present is added again. This keeps appending fast. Pass respect_kd=True to replace the row that has the same key instead. That looks the key up on every call, so it costs more when you add many rows one at a time.

With columns defined, a list is added as a copy, so you can reuse it. With no columns, the list itself is added.

A KeyedList made by this Daf, as by default_record(astype=KeyedList), iloc() or iter_klist(), shares its hd. Its keys are then the columns, so its columns are not checked. Its values are still copied.

With fast=True nothing is copied. A KeyedList must share the hd of this Daf, and then nothing else is checked. A dict or a list is checked fully if it is the first row added to an empty Daf: a dict must have the columns as its keys, in order, and a list one value for each column. Every later dict or list is checked only for its number of values. A row that fails raises ValueError. Use it when you build each row yourself in column order, as in a loop, best with default_record(). A list, or the value list of a KeyedList, becomes the row itself, so do not change it afterwards. Giving a variable a new list each time is fine. Filling one list in place and appending it again is not: every row is then that same list. A dict gives its values in its own order. A later row with the right number of values in another order is added anyway, and its values land in the wrong columns. If you want every row checked, leave out fast. fast makes no difference to the first row of a Daf with no columns, to a Daf or a list of dicts, or with respect_kd=True and a keyfield.

None, an empty dict, an empty list and an empty Daf add nothing.

Parameters:

Name	Type	Description	Default
data_item	'Daf' | T_loda | T_da | T_la | KeyedList | None	The row or rows to add.	None
respect_kd	bool	If True, replace the row that has the same key. If False, the default, add it.	False
lol	T_lola | None	Several rows, each a list of values in column order.	None
la	T_la | None	One row, as a list of values in column order. Its items are not read as rows.	None
fast	bool	If True, add the row with no copy. Check the first row of an empty Daf fully, and every row's length. See above.	False

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
TypeError	More than one of data_item, lol and la is given, or data_item is not a supported type.
ValueError	A list has more values than there are columns.

Examples:

>>> d = Daf(lol=[[1, 'a']], cols=['id', 'v'], keyfield='id')
>>> d.append({'v': 'b', 'id': 2})

| id | v  |
| -: | -: |
|  1 |  a |
|  2 |  b |

%% daf rows=2; cols=2; keyfield='id'; name=''

>>> d.append([3, 'c'])

| id | v  |
| -: | -: |
|  1 |  a |
|  2 |  b |
|  3 |  c |

%% daf rows=3; cols=2; keyfield='id'; name=''

>>> d.append(lol=[[4, 'd'], [5, 'e']])

| id | v  |
| -: | -: |
|  1 |  a |
|  2 |  b |
|  3 |  c |
|  4 |  d |
|  5 |  e |

%% daf rows=5; cols=2; keyfield='id'; name=''

>>> d.append({'id': 2, 'v': 'new'}, respect_kd=True)

| id |  v  |
| -: | --: |
|  1 |   a |
|  2 | new |
|  3 |   c |
|  4 |   d |
|  5 |   e |

%% daf rows=5; cols=2; keyfield='id'; name=''

>>> e = Daf(lol=[[1, 'a']], cols=['id', 'v'])
>>> e.append(None).append({}).append([])

| id | v  |
| -: | -: |
|  1 |  a |

%% daf rows=1; cols=2; keyfield=''; name=''

concat

concat(other_instance, respect_kd=False)

Add the rows of another Daf to the end of this one.

Use this to combine two tables with the same columns. By default every row is appended, so a key that exists in both tables appears twice afterwards. Pass respect_kd=True to get an upsert instead.

Parameters:

Name	Type	Description	Default
other_instance	'Daf'	Daf whose rows are added. It is not changed.	required
respect_kd	bool	If False, the default, append every row without looking at the keys. If True, replace the row that has the same key and append the rows with new keys.	False

Returns:

Type	Description
'Daf'	This Daf, which has been changed in place. The rows are deep copies.

Raises:

Type	Description
KeyError	The column names of the two Daf instances differ.
ValueError	With respect_kd=True, both Daf instances have a keyfield and the keyfields differ.

Notes

The columns must be equal, including their order. With respect_kd=True, this Daf's keyfield decides the key. If it has no keyfield, the rows are simply appended. A key repeated in the other Daf replaces the earlier row, so the last one wins. If the other Daf is empty, nothing happens.

Examples:

>>> a = Daf(lol=[[1, 'a1'], [2, 'a2']], cols=['id', 'v'], keyfield='id')
>>> b = Daf(lol=[[2, 'b2'], [3, 'b3']], cols=['id', 'v'])
>>> a.concat(b)

| id | v  |
| -: | -: |
|  1 | a1 |
|  2 | a2 |
|  2 | b2 |
|  3 | b3 |

%% daf rows=4; cols=2; keyfield='id'; name=''

>>> a = Daf(lol=[[1, 'a1'], [2, 'a2']], cols=['id', 'v'], keyfield='id')
>>> a.concat(b, respect_kd=True)

| id | v  |
| -: | -: |
|  1 | a1 |
|  2 | b2 |
|  3 | b3 |

%% daf rows=3; cols=2; keyfield='id'; name=''

extend

extend(records_lod=None, respect_kd=False, *, lol=None)

Append several records, given as a list of dicts or as a list of lists.

Give a list of dicts as records_lod. Each dict is a row, placed by column name, as in append(). A Daf with no columns takes them from the first dict. An empty list, or a list that holds one empty dict, adds nothing.

Give a list of lists as lol, as in extend(lol=[[2, 'b'], [3, 'c']]). Each list is a row, in column order. A short list is padded with NULL. A list with more values than there are columns raises ValueError, and then no row is added. A Daf with no columns takes the lists as they are. The keyword says that these are several rows, and not one row whose cells are lists. For that, see append(la=...).

Without respect_kd, a key that already exists is added again. Use respect_kd=True when the keyfield must stay unique.

Parameters:

Name	Type	Description	Default
records_lod	T_loda | None	The records, as dicts.	None
respect_kd	bool	If True, replace the row that has the same key. If False, the default, add it.	False
lol	T_lola | None	The rows, as lists of values in column order.	None

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
TypeError	Both records_lod and lol are given, or a row of lol is not a list.
ValueError	A row of lol has more values than there are columns.

Examples:

>>> d = Daf(lol=[[1, 'a']], cols=['id', 'v'], keyfield='id')
>>> d.extend([{'id': 2, 'v': 'b'}, {'id': 1, 'v': 'new'}], respect_kd=True)

| id |  v  |
| -: | --: |
|  1 | new |
|  2 |   b |

%% daf rows=2; cols=2; keyfield='id'; name=''

>>> d.extend(lol=[[3, 'c'], [4]])

| id |  v  |
| -: | --: |
|  1 | new |
|  2 |   b |
|  3 |   c |
|  4 |     |

%% daf rows=4; cols=2; keyfield='id'; name=''

record_append

record_append(record, respect_kd=True)

Add one row that is given as a dict or a KeyedList.

This is the single row case of append(). The row is placed by column name, a missing key gets NULL, and a key that is not a column is dropped. An empty Daf takes its columns from the first row. An empty record adds nothing.

With a keyfield, the row that has the same key is replaced, and a new key is added at the end. This is the default here, unlike append(). Pass respect_kd=False to always add. The key index is kept up to date, so adding many rows one at a time stays fast.

A plain list is not accepted. Use append() for that.

Parameters:

Name	Type	Description	Default
record	T_da | KeyedList	The row, as a dict or a KeyedList.	required
respect_kd	bool	If True, the default, replace the row that has the same key. Otherwise add it.	True

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 'a'], [2, 'b']], cols=['id', 'v'], keyfield='id')
>>> d.record_append({'id': 2, 'v': 'new'})

| id |  v  |
| -: | --: |
|  1 |   a |
|  2 | new |

%% daf rows=2; cols=2; keyfield='id'; name=''

>>> d.record_append({'id': 2, 'v': 'dup'}, respect_kd=False)

| id |  v  |
| -: | --: |
|  1 |   a |
|  2 | new |
|  2 | dup |

%% daf rows=3; cols=2; keyfield='id'; name=''

Records

assign_record

assign_record(record)

Put one row in the Daf by its key, replacing a row that has the same key.

The row is a dict. If a row with that key exists, the whole row is replaced. The cells of columns that the dict lacks become NULL. If the key is new, the row is added at the end. This is an upsert for one row. For that, append() with respect_kd=True also works. Use update_by_keylist() to change only some cells.

Parameters:

Name	Type	Description	Default
record	T_da	The row, as a dict. It must have the keyfield.	required

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
KeysDisabledError	The Daf has no keyfield.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> _ = d.assign_record({'id': 2, 'v': 'new'})
>>> d

| id |  v  | n  |
| -: | --: | -: |
|  1 |   a | 10 |
|  2 | new |    |
|  3 |   c | 30 |

%% daf rows=3; cols=3; keyfield='id'; name=''

assign_record_irow

assign_record_irow(irow=None, record=None)

Put one row in the Daf by position, replacing the row there.

The row is a dict. The whole row is replaced, and the cells of columns that the dict lacks become NULL. Use update_record_irow() to change only some cells.

With the default position, None, the row is added at the end. So it is with a position beyond the last row, and with any position when the Daf has no rows. A negative position counts from the end, as in a list, so -1 replaces the last row. This is also what my_daf[-1] = {...} does.

This does not update the key index (kd), so don't change a key with it. See __setitem__.

Parameters:

Name	Type	Description	Default
irow	int | None	The row position. None, the default, adds the row at the end.	None
record	T_da | None	The row, as a dict. If None, nothing happens.	None

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
IndexError	A negative position is before the first row, for a Daf that has rows.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> _ = d.assign_record_irow(1, {'v': 'q'})
>>> d.iloc(1)
{'id': '', 'v': 'q', 'n': ''}
>>> _ = d.assign_record_irow(-1, {'id': 9})
>>> d.iloc(-1)
{'id': 9, 'v': '', 'n': ''}
>>> _ = d.assign_record_irow(record={'id': 10})
>>> d.iloc(-1), len(d)
({'id': 10, 'v': '', 'n': ''}, 4)

update_by_keylist

update_by_keylist(keylist=None, record=None)

Change some cells in the rows that have the given keys.

Only the columns that are keys of the dict are changed. Other cells keep their values. A key that is not found is skipped. This is a bulk form of update_record_krow(), for rows that all take the same values. The row positions do not change, so the key index stays valid.

Parameters:

Name	Type	Description	Default
keylist	T_ls | None	The keys of the rows to change.	None
record	T_da | None	The new values, as a dict of column name and value. Names that are not columns are ignored.	None

Returns:

Type	Description
'Daf'	This Daf, which has been changed. With no keyfield, rows, keys or record, nothing happens.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.update_by_keylist([1, 3, 9], {'v': 'q'})

| id | v  | n  |
| -: | -: | -: |
|  1 |  q | 10 |
|  2 |  b | 20 |
|  3 |  q | 30 |

%% daf rows=3; cols=3; keyfield='id'; name=''

update_record_irow

update_record_irow(irow=-1, record=None)

Change some cells in the row at a position.

Only the columns that are keys of the dict are changed. Other cells keep their values. A negative position counts from the end, as in a list, so the default, -1, is the last row. A position that is out of range raises IndexError, and nothing is changed. A Daf with no rows or no columns, and a record of None, change nothing.

This does not update the key index (kd), so don't change a key with it. See __setitem__.

Parameters:

Name	Type	Description	Default
irow	int	The row position. -1, the default, is the last row.	-1
record	T_da | None	The new values, as a dict of column name and value. Names that are not columns are ignored.	None

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
IndexError	The position is out of range, for a Daf that has rows and columns.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> _ = d.update_record_irow(1, {'v': 'q'})
>>> d.iloc(1)
{'id': 2, 'v': 'q', 'n': 20}
>>> _ = d.update_record_irow(record={'n': 99})
>>> d.iloc(-1)
{'id': 3, 'v': 'c', 'n': 99}

update_record_krow

update_record_krow(key, record=None)

Change some cells in the row that has this key.

Only the columns that are keys of the dict are changed. Other cells keep their values. It is the form of update_record_irow() that finds the row by its key. An int is always a key here, never a position, so it suits a keyfield of ints. For many rows that take the same values, use update_by_keylist().

As with update_record_irow(), this does not update the key index (kd), so don't change a key with it. See __setitem__. A record of None changes nothing.

Parameters:

Name	Type	Description	Default
key	Any	The key of the row to change.	required
record	T_da | None	The new values, as a dict of column name and value. Names that are not columns are ignored.	None

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
KeyError	No row has this key.
ValueError	The Daf has no keyfield.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> _ = d.update_record_krow(2, {'v': 'q'})
>>> d.select_record(2)
{'id': 2, 'v': 'q', 'n': 20}

insert_irow

insert_irow(irow=None, row=None, default='')

Insert a row at a position and move the later rows down.

The row is a list of values, or a dict that is placed by column name. A short list is filled out with default. With the default position, None, or with a position beyond the last row, the row is added at the end. A negative position, such as -1, also adds it at the end. This differs from list.insert(), which puts the item before the last one. Prefer None. The key index is rebuilt when it is next needed.

Parameters:

Name	Type	Description	Default
irow	int | None	The row position. None, the default, adds the row at the end.	None
row	T_la | T_da | None	The row, as a list or a dict.	None
default	Any	The value for cells that a short list does not reach.	''

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
TypeError	The row is not a list or a dict. This includes the default, None.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.insert_irow(1, {'id': 9, 'v': 'z'})

| id | v  | n  |
| -: | -: | -: |
|  1 |  a | 10 |
|  9 |  z |    |
|  2 |  b | 20 |
|  3 |  c | 30 |

%% daf rows=4; cols=3; keyfield='id'; name=''

Columns

assign_col

assign_col(colname, la=None, default='')

Fill a column by name with the values of a list, or add it if it is new.

This is my_daf[:, colname] = values, and it also adds a column. A list that is too short is filled out with default. With no list, every cell gets default. If the column is the keyfield, the key index is rebuilt when it is next needed.

Parameters:

Name	Type	Description	Default
colname	str	The column name.	required
la	T_la | None	The values, one for each row.	None
default	Any	The value for rows that the list does not reach.	''

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.assign_col('w', default=0).columns()
['id', 'v', 'n', 'w']
>>> d.col('w')
[0, 0, 0]

assign_icol

assign_icol(icol=-1, col_la=None, default='')

Fill a column by position with the values of a list.

A list that is too short is filled out with default. With no list, every cell gets default. With icol=-1 a new column is added at the right. If the Daf has column names, the new column gets the next spreadsheet name, such as C, made unique. Use insert_col() to add a column with a name of your own.

Parameters:

Name	Type	Description	Default
icol	int	The column position. -1 adds a column at the right.	-1
col_la	T_la | None	The values, one for each row.	None
default	Any	The value for rows that the list does not reach.	''

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> _ = d.assign_icol(1, ['x', 'y'], default='D')
>>> d.col('v')
['x', 'y', 'D']

insert_col

insert_col(colname, col_la=None, icol=-1, default='')

Add a named column at a position, or overwrite it if the name exists.

A list that is too short is filled out with default. With no list, every cell gets default, so this can add a constant column. If the name already exists the column is overwritten, and icol is ignored. An empty name does nothing. Use set_keyfield() if the column is to be the keyfield.

When a column is added and the rows are shared with another Daf, as after a selection, this Daf first takes its own copies of the rows. The other Daf is not changed.

Parameters:

Name	Type	Description	Default
colname	str	The name of the column.	required
col_la	T_la | None	The values, one for each row.	None
icol	int	The column position. -1 adds the column at the right.	-1
default	Any	The value for rows that the list does not reach.	''

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.insert_col('w', ['x', 'y', 'z'], icol=1).columns()
['id', 'w', 'v', 'n']
>>> d.insert_col('k', default=5).col('k')
[5, 5, 5]

insert_icol

insert_icol(icol=-1, col_la=None, colname='', default='')

Insert a column at a position and move the later columns right.

A list that is too short is filled out with default. With icol=-1, or a position beyond the last column, the column is added at the right. Give colname to name it. Without a name, if the Daf has column names, the column gets the next spreadsheet name, such as C, made unique. The dtypes are not changed. Use set_keyfield() if the column is to be the keyfield.

If the rows are shared with another Daf, as after a selection, this Daf first takes its own copies of the rows. The other Daf is not changed.

Parameters:

Name	Type	Description	Default
icol	int	The column position. -1 adds the column at the right.	-1
col_la	T_la | None	The values, one for each row.	None
colname	str	The name of the new column.	''
default	Any	The value for rows that the list does not reach.	''

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.insert_icol(1, ['x', 'y', 'z'], colname='new').columns()
['id', 'new', 'v', 'n']

insert_idx_col

insert_idx_col(colname='idx', icol=0, startat=0)

Insert a column of row numbers.

This is insert_col(), so rows shared with another Daf are copied first and the other Daf is not changed.

Parameters:

Name	Type	Description	Default
colname	str	The name of the new column.	'idx'
icol	int	The column position.	0
startat	int	The number of the first row.	0

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.insert_idx_col().col('idx')
[0, 1, 2]

set_col_irows

set_col_irows(colname, irows, val)

Set one value in the given rows of a named column.

This is deprecated. Use my_daf[irows, colname] = value, which does the same. A column name that is not found raises KeyError. Row positions that are out of range are skipped.

Parameters:

Name	Type	Description	Default
colname	str	The column name.	required
irows	T_li	The row positions.	required
val	Any	The value to set.	required

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
KeyError	The column name is not found.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.set_col_irows('v', [0, 2], 'Z').col('v')
['Z', 'b', 'Z']

set_icol

set_icol(icol, val)

Set one value in every row of a column, by position.

This is my_daf[:, icol] = value.

Parameters:

Name	Type	Description	Default
icol	int	The column position.	required
val	Any	The value to set.	required

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
IndexError	The position is beyond the end of a row.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.set_icol(1, 'Z').col('v')
['Z', 'Z', 'Z']

set_icol_irows

set_icol_irows(icol, irows, val)

Set one value in the given rows of a column, by position.

This is my_daf[irows, icol] = value. Row positions that are out of range are skipped.

Parameters:

Name	Type	Description	Default
icol	int	The column position.	required
irows	T_li	The row positions.	required
val	Any	The value to set.	required

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> _ = d.set_icol_irows(1, [0, 9, -1], 'Z')
>>> d.col('v')
['Z', 'b', 'c']

Removing

delete_records

delete_records(keys_ls, inplace=False, silent_error=False)

Remove the rows with these keys.

By default, this Daf is left as it is, and a new Daf without the rows is returned, as select_records_daf(keys, inverse=True) gives. That is the usual way in daffodil, and the new Daf shares the rows that remain.

With inplace=True, the rows are deleted from this Daf itself, and it is returned. The row list is rebuilt once, in place, so it costs one pass however many rows go. The remaining rows keep their order, and the key index is rebuilt when next needed. That is safe for a Daf in memory: a selection made from it earlier holds its own row list, so it still has the deleted rows. On a SqlDaf parent table it is not safe in the same way. See SqlDaf.delete_records() in sqldaf.py.

The keys may be a list, any iterable, or another Daf, whose keys are then used through a view of its key index, as in select_records_daf().

Parameters:

Name	Type	Description	Default
keys_ls	'T_ls | Iterable | Daf'	The keys of the rows to remove, or a Daf whose keys are used.	required
inplace	bool	If True, delete from this Daf and return it. If False, return a new Daf.	False
silent_error	bool	If True, keys that are not found are ignored.	False

Returns:

Type	Description
The Daf without the rows	this one if inplace, otherwise a new one.

Raises:

Type	Description
KeysDisabledError	This Daf, or a Daf given as the keys, has no keyfield and no key index.
KeyError	A key is not found and silent_error is False. Nothing is removed.

Examples:

>>> d = Daf(lol=[['r1', 'a'], ['r2', 'b'], ['r3', 'c']], cols=['id', 'v'], keyfield='id')
>>> d.delete_records(['r2']).to_lod()
[{'id': 'r1', 'v': 'a'}, {'id': 'r3', 'v': 'c'}]
>>> len(d)
3
>>> _ = d.delete_records(['r2', 'r9'], inplace=True, silent_error=True)
>>> d.to_lod()
[{'id': 'r1', 'v': 'a'}, {'id': 'r3', 'v': 'c'}]

remove_key

remove_key(keyval, silent_error=False)

Make a new Daf without the row that has the given key.

This is deprecated. Use select_krows(key, inverse=True), which does the same.

This does not remove the row from this Daf. It returns a new Daf that leaves the row out, and this Daf is unchanged. Keep the result, as in d = d.remove_key(2).

The new Daf shares the surviving rows with this Daf. Changing a cell in one changes it in the other. Call copy('editable') if you need row lists of your own.

A tuple is a range of keys, as in remove_key((2, 3)). With a composite keyfield, a tuple as long as the keyfield, with no tuples inside it, is one key, as in remove_key(('a', 1)). A tuple of key tuples is a range, and a list of key tuples is a list of keys.

Parameters:

Name	Type	Description	Default
keyval	str | int | T_la | T_ta | None	The key of the row to leave out.	required
silent_error	bool	If True, a key that is not found is ignored.	False

Returns:

Type	Description
'Daf'	The new Daf. It has the same keyfield.

Raises:

Type	Description
KeysDisabledError	The Daf has no keyfield and no key index.
KeyError	The key is not found and silent_error is False.

Examples:

>>> d = Daf(lol=[[1, 'a'], [2, 'b']], cols=['id', 'v'], keyfield='id')
>>> d.remove_key(1)

| id | v  |
| -: | -: |
|  2 |  b |

%% daf rows=1; cols=2; keyfield='id'; name=''

>>> d.num_rows()
2
>>> c = Daf(lol=[['a', 1], ['a', 2]], cols=['g', 'n'], keyfield=('g', 'n'))
>>> c.remove_key(('a', 1))

| g  | n  |
| -: | -: |
|  a |  2 |

%% daf rows=1; cols=2; keyfield='('g', 'n')'; name=''

remove_keylist

remove_keylist(keylist, silent_error=False)

Make a new Daf without the rows that have the given keys.

This is deprecated. Use select_krows(keys, inverse=True), which does the same.

This does not remove the rows from this Daf. It returns a new Daf that leaves them out, and this Daf is unchanged. See remove_key() for how the rows are shared.

Parameters:

Name	Type	Description	Default
keylist	T_ls	The keys of the rows to leave out.	required
silent_error	bool	If True, keys that are not found are ignored.	False

Returns:

Type	Description
'Daf'	The new Daf. It has the same keyfield.

Raises:

Type	Description
KeysDisabledError	The Daf has no keyfield and no key index.
KeyError	A key is not found and silent_error is False.

Examples:

>>> d = Daf(lol=[[1, 'a'], [2, 'b'], [3, 'c']], cols=['id', 'v'], keyfield='id')
>>> d.remove_keylist([1, 3])

| id | v  |
| -: | -: |
|  2 |  b |

%% daf rows=1; cols=2; keyfield='id'; name=''

>>> d2 = Daf(lol=[[1, 'a', 0], [2, 'b', 1]], cols=['p', 'q', 'r'], keyfield=['p', 'q'])
>>> d2.remove_keylist([(1, 'a')])

| p  | q  | r  |
| -: | -: | -: |
|  2 |  b |  1 |

%% daf rows=1; cols=3; keyfield='['p', 'q']'; name=''

remove_dups

remove_dups(keyfield='')

Split the rows into those with a unique key and those with a repeated key.

Only the key columns are compared, not the whole row. For each key, the last row is kept as the unique one. The earlier rows with that key are the duplicates. The unique rows are in the order in which their keys first appear.

Without keyfield, the keyfield of this Daf is used. If there is none, a KeysDisabledError is raised. This Daf is not changed. Both results share their rows with it, and both have the key columns as their keyfield.

Parameters:

Name	Type	Description	Default
keyfield	str | T_ta | T_la	The column, or tuple or list of columns, that identifies a row. If empty, the keyfield of this Daf.	''

Returns:

Type	Description
Tuple['Daf', 'Daf']	A tuple of the Daf of unique rows and the Daf of duplicate rows.

Raises:

Type	Description
KeysDisabledError	There is no keyfield and this Daf has none.
KeyError	A key column is not found.

Examples:

>>> d = Daf(lol=[[1, 'a'], [2, 'b'], [1, 'c']], cols=['id', 'v'])
>>> unique, dups = d.remove_dups('id')
>>> unique

| id | v  |
| -: | -: |
|  1 |  c |
|  2 |  b |

%% daf rows=2; cols=2; keyfield='id'; name=''

>>> dups

| id | v  |
| -: | -: |
|  1 |  a |

%% daf rows=1; cols=2; keyfield='id'; name=''

drop_cols

drop_cols(exclude_cols=None)

Remove columns from this Daf, in place.

The rows are rebuilt without those columns, so this copies all the data. Avoid it for large tables. Instead, name only the columns you need: with cols= in reduce(), by reading only those fields in an apply() function, or with select_kcols() for a new Daf.

The column names and dtypes are updated. A name that is not a column is ignored. If a column of the keyfield is dropped, the keyfield is cleared, as in the other methods that remove a key column. Set it again with set_keyfield(). With no names, nothing happens.

Parameters:

Name	Type	Description	Default
exclude_cols	T_ls | None	The names of the columns to remove.	None

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.drop_cols(['v']).columns()
['id', 'n']

Find and replace

find_replace

find_replace(find_pat, replace_val)

Replace every cell that matches a pattern, in place.

Each cell is converted to text and searched with the regular expression find_pat. If it matches anywhere in the text, the whole cell is replaced by replace_val. It is not a substitution inside the text. Every column is searched, including numbers, and the key index is rebuilt when it is next needed.

Parameters:

Name	Type	Description	Default
find_pat	str	A regular expression.	required
replace_val	Any	The value that replaces a matching cell.	required

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> _ = d.find_replace(r'^[ab]$', 'HIT')
>>> d.col('v')
['HIT', 'HIT', 'c']

replace_in_columns

replace_in_columns(
    cols, find_values=None, replacement=_MISSING
)

Replace listed values with another value, in some columns, in place.

A cell is replaced when it equals any value in find_values. A typical use is ['', None] to fill empty cells. Columns may be given by name or by position, or all columns if cols is None. With no find_values nothing happens. The key index is rebuilt if the keyfield may have changed.

Parameters:

Name	Type	Description	Default
cols	T_lsi | None	Column names or positions. If None, all columns.	required
find_values	List[Any] | None	The values to look for.	None
replacement	Any	The value to put in their place. This is required.	_MISSING

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
ValueError	No replacement is given.
KeyError	A column name is not found.
TypeError	A column is neither a name nor a position.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.replace_in_columns(['v'], ['a', 'c'], '-').col('v')
['-', 'b', '-']

Sorting

sort_by_colname

sort_by_colname(
    colname,
    *,
    reverse=False,
    length_priority=False,
    as_str=False,
)

Sort the rows by one column, in place.

Make a copy first if you need the original order. An empty cell sorts before a cell with content, unlike a spreadsheet. Values in the column must be comparable with each other. A column that mixes None or text with numbers raises TypeError, which names the column. Use as_str=True to sort by the text of each value instead.

With length_priority, a shorter text sorts before a longer one, so numbers that are stored as text sort as numbers. Without it, '10', '100', '8' are in text order. length_priority needs text, so for real numbers use it with as_str=True. Then whole numbers sort in numeric order, and negative and decimal numbers sort in text order.

Parameters:

Name	Type	Description	Default
colname	str	The column to sort by.	required
reverse	bool	If True, sort from high to low.	False
length_priority	bool	If True, sort by length first, then by value.	False
as_str	bool	If True, sort by the text of each value, and None sorts as an empty cell.	False

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
KeyError	The column name is not found.
TypeError	The column holds values that cannot be compared, and as_str is False.

Examples:

>>> d = Daf(lol=[['10'], ['99'], ['8'], ['100']], cols=['a'])
>>> d.sort_by_colname('a', length_priority=True).col('a')
['8', '10', '99', '100']
>>> d = Daf(lol=[[10], [None], [2], ['']], cols=['n'])
>>> d.sort_by_colname('n', as_str=True, length_priority=True).col('n')
[None, '', 2, 10]

sort_by_colnames

sort_by_colnames(
    colnames,
    reverse=False,
    length_priority=False,
    as_str=False,
)

Sort the rows by several columns, in place.

The first column is the main sort key. The others break ties. See sort_by_colname() for the rules of ordering, for length_priority and for as_str. Calling sort_by_colname() for each column, last column first, gives the same order.

Parameters:

Name	Type	Description	Default
colnames	T_ls	The columns to sort by, main key first.	required
reverse	bool	If True, sort from high to low.	False
length_priority	bool	If True, sort by length first, then by value.	False
as_str	bool	If True, sort by the text of each value, and None sorts as an empty cell.	False

Returns:

Type	Description
'Daf'	This Daf, which has been changed.

Raises:

Type	Description
KeyError	A column name is not found.
TypeError	A column holds values that cannot be compared, and as_str is False.

Examples:

>>> d = Daf(lol=[[2, 'b'], [1, 'z'], [1, 'a']], cols=['p', 'q'])
>>> d.sort_by_colnames(['p', 'q'])

| p  | q  |
| -: | -: |
|  1 |  a |
|  1 |  z |
|  2 |  b |

%% daf rows=3; cols=2; keyfield=''; name=''

Reshaping and splitting

transpose

transpose(
    new_keyfield="", new_cols=None, include_header=False
)

Turn rows into columns and columns into rows.

The result has one row for each column of this Daf. With include_header=True, the first column of the result holds the column names of this Daf, and the names given in new_cols or the default names must then include that column. The default cols are A, B and so on, one for each row of this Daf. With include_header=True they start with key. If you pass new_cols, give one name for each row of this Daf, plus one with include_header. The data is copied.

Parameters:

Name	Type	Description	Default
new_keyfield	str	The keyfield of the result.	''
new_cols	T_la | None	The cols of the result.	None
include_header	bool	If True, the column names become the first column.	False

Every row must be as long as the columns. Check with is_rectangular(), and pad short rows with force_rectangular().

Returns:

Type	Description
'Daf'	The new Daf.

Raises:

Type	Description
ValueError	A row is not as long as the columns.

Examples:

>>> d = Daf(lol=[[1, 'a'], [2, 'b']], cols=['id', 'v'])
>>> d.transpose(include_header=True)

| key | A  | B  |
| --: | -: | -: |
|  id |  1 |  2 |
|   v |  a |  b |

%% daf rows=2; cols=3; keyfield=''; name=''

>>> d.transpose().columns()
['A', 'B']
>>> d.transpose(include_header=True).columns()
['key', 'A', 'B']

wide_to_narrow

wide_to_narrow(
    id_cols,
    varname_colname="variable",
    value_colname="value",
)

Turn columns into rows. This is called melt or unpivot.

Each column that is not an id column gives one row for each row of this Daf. The row holds the id values, the name of the column, and its value.

The default names, variable and value, are those that narrow_to_wide() reads, and those that pandas melt() uses. So the two methods chain with their defaults.

Parameters:

Name	Type	Description	Default
id_cols	T_ls	The columns that identify a row. They are kept as they are.	required
varname_colname	str	The name of the new column that holds the old column names.	'variable'
value_colname	str	The name of the new column that holds the values.	'value'

Returns:

Type	Description
'Daf'	The new Daf.

Raises:

Type	Description
TypeError	id_cols is not a list.

Examples:

>>> d = Daf(lol=[['x', 1, 2], ['y', 3, 4]], cols=['id', 'a', 'b'])
>>> d.wide_to_narrow(['id'])

| id | variable | value |
| -: | -------: | ----: |
|  x |        a |     1 |
|  x |        b |     2 |
|  y |        a |     3 |
|  y |        b |     4 |

%% daf rows=4; cols=3; keyfield=''; name=''

A round trip:

>>> d.wide_to_narrow(['id']).narrow_to_wide(['id']).lol
[['x', 1, 2], ['y', 3, 4]]

narrow_to_wide

narrow_to_wide(
    id_cols,
    varname_col="variable",
    value_col="value",
    wide_cols=None,
)

Turn rows into columns. This is called pivot or spread.

The rows of one id may be anywhere in the table. The result has one row for each id, in the order in which the ids first appear. The columns are the id columns and then the names found in varname_col, in the order first seen. An id that has no value for a name gets NULL. If an id has the same name more than once, the last value is kept.

With wide_cols, those are the columns after the ids, in that order. A name that is not listed is left out.

The default names, variable and value, are those that wide_to_narrow() writes.

An empty Daf gives an empty Daf.

Parameters:

Name	Type	Description	Default
id_cols	T_cs	The columns that identify a row.	required
varname_col	str	The column whose values become the new column names.	'variable'
value_col	str	The column whose values fill the new columns.	'value'
wide_cols	T_cs | None	The names to use as columns, in order. If None, all names found.	None

Returns:

Type	Description
'Daf'	The new Daf.

Examples:

>>> d = Daf(lol=[['x', 'a', 1], ['y', 'a', 3], ['x', 'b', 2], ['y', 'b', 4]], cols=['id', 'variable', 'value'])
>>> d.narrow_to_wide(['id'])

| id | a  | b  |
| -: | -: | -: |
|  x |  1 |  2 |
|  y |  3 |  4 |

%% daf rows=2; cols=3; keyfield=''; name=''

>>> d.narrow_to_wide(['id'], wide_cols=['b', 'a'])

| id | b  | a  |
| -: | -: | -: |
|  x |  2 |  1 |
|  y |  4 |  3 |

%% daf rows=2; cols=3; keyfield=''; name=''

split_daf_into_ranges

split_daf_into_ranges(chunk_ranges)

Split the rows into several Daf instances, by position ranges.

Each range is (start, end), and end is not included. The new Daf instances share their rows with this one.

Parameters:

Name	Type	Description	Default
chunk_ranges	List[Tuple[int, int]]	The ranges of row positions.	required

Returns:

Type	Description
List['Daf']	A list of Daf instances, one for each range.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> parts = d.split_daf_into_ranges([(0, 2), (2, 3)])
>>> parts[0]

| id | v  | n  |
| -: | -: | -: |
|  1 |  a | 10 |
|  2 |  b | 20 |

%% daf rows=2; cols=3; keyfield='id'; name=''

>>> parts[1]

| id | v  | n  |
| -: | -: | -: |
|  3 |  c | 30 |

%% daf rows=1; cols=3; keyfield='id'; name=''

split_daf_into_chunks_lodaf

split_daf_into_chunks_lodaf(max_chunk_size)

Split the rows evenly into Daf instances of at most a given size.

The sizes are as equal as possible, so some chunks are smaller than the maximum. None is larger. The chunks share their rows with this Daf.

Parameters:

Name	Type	Description	Default
max_chunk_size	int	The most rows in one chunk.	required

Returns:

Type	Description
List['Daf']	A list of Daf instances.

Examples:

>>> d = Daf(lol=[[i] for i in range(7)], cols=['a'])
>>> [len(part) for part in d.split_daf_into_chunks_lodaf(3)]
[3, 2, 2]