Selecting and indexing¶

Reading rows, columns and cells by index, key, or condition.

Indexing with brackets¶

getitem ¶

__getitem__(slice_spec)

Select rows, columns or cells, as in my_daf[rows, cols].

The selector is [rows] or [rows, cols]. With one selector, all columns are returned. Use : for all. A selector is one of these.

integer       A position. Negative counts from the end.
slice         `2:5`, `:3`, `::2`, as for a Python list.
list          A list of positions, in the order given, or a list of ranges.
range         A range of positions.
string        A key of the keyfield for rows, or a column name for columns.
list of str   Several keys, or several column names, in the order given.
tuple         An inclusive range of keys, or of column names. Use None to
              start at the first or to end at the last, as in `(None, 'r3')`.

A tuple of two items is read as [rows, cols] when it stands alone. To give a range of row keys, add the column selector, as in my_daf[('r1', 'r3'), :].

Integers are always positions. A keyfield or column names that are integers cannot be used in brackets. Use select_krows() and select_kcols() instead.

The result is a new Daf. Its rows are shared with this Daf when you select rows, so changing a cell in the result changes it here too. Selecting columns makes new row lists, so replacing a cell in the result does not change this Daf. A list or dict held in a cell is still the same object in both. Use copy('deep') when those must be separate too. The keyfield and dtypes carry over if their columns are still there. See retmode for getting a bare value or list when the result is one cell, row or column.

A column slice works as it does for a Python list.

Parameters:

Name	Type	Description	Default
`slice_spec`	`slice \| int \| str \| T_li \| T_ls \| range \| T_lor \| Tuple[slice \| int \| str \| T_li \| T_ls \| range \| T_lor \| Tuple[str, str], slice \| int \| str \| T_li \| T_ls \| range \| T_lor \| Tuple[str, str]]`	A row selector, or a tuple of a row selector and a column selector.	required

Returns:

Type	Description
`Any`	A new Daf, or a value or list if `retmode` is `val`.

Raises:

Type	Description
`IndexError`	A row or column position is out of range.
`KeyError`	A key or column name is not found.
`KeysDisabledError`	Rows are selected by key and there is no keyfield.
`TypeError`	A selector is None, or has a type that is not accepted.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d[1]

| id | v  | n  |
| -: | -: | -: |
|  2 |  b | 20 |

%% daf rows=1; cols=3; keyfield='id'; name=''

>>> d[1:]

| id | v  | n  |
| -: | -: | -: |
|  2 |  b | 20 |
|  3 |  c | 30 |

%% daf rows=2; cols=3; keyfield='id'; name=''

>>> d[:, 'v']

| v  |
| -: |
|  a |
|  b |
|  c |

%% daf rows=3; cols=1; keyfield=''; name=''

>>> d[[2, 0], ['n', 'id']]

| n  | id |
| -: | -: |
| 30 |  3 |
| 10 |  1 |

%% daf rows=2; cols=2; keyfield='id'; name=''

>>> d[1, 'n'].to_value()
20
>>> d[(1, 2), :]

| id | v  | n  |
| -: | -: | -: |
|  1 |  a | 10 |
|  2 |  b | 20 |

%% daf rows=2; cols=3; keyfield='id'; name=''

setitem ¶

__setitem__(slice_spec, value)

Assign values to a selection, as in my_daf[rows, cols] = value.

The selector is the same as for []. The Daf is changed in place.

A single value fills every cell of the selection. A list fills the selection in order. A dict given for a row sets that row from the dict. The cells whose columns are not in the dict become NULL. See set_irows_icols() for what happens when the source and the selection differ in size.

A str or bytes is one value, not a list of characters, so my_daf[:, 'v'] = 'xyz' sets every row of v to xyz. A list assigned to several whole rows is copied, so each row has its own list.

A write does not update the key index (kd). Writing into a keyfield cell, or writing a whole row, which includes the key, leaves the kd as it was: a lookup still finds the row by its old key, and not by the new one. So don't change keys this way. If you must, call set_keyfield() with the same keyfield afterwards, which invalidates the kd, so it is rebuilt when it is next needed. Writing other columns is safe.

A selection that shares the row has a key index of its own. Resetting one Daf does not reset the other. Call set_keyfield() on each Daf that will look up keys.

A single name, as in my_daf['x'] = value, selects the row with that key. It is not a column, as it would be in pandas. Use my_daf[:, 'x'] for a column, or assign_col() to add one. A name that is not a row key raises an error that says this.

Parameters:

Name	Type	Description	Default
`slice_spec`	`slice \| int \| str \| range \| T_li \| T_ls \| T_lb \| T_lor \| Tuple[slice \| int \| str \| range \| T_lor \| T_li \| T_ls \| T_lb \| Tuple[Any, Any], slice \| int \| str \| range \| T_lor \| T_li \| T_ls \| T_lb \| Tuple[Any, Any]]`	A row selector, or a tuple of a row selector and a column selector.	required
`value`	`Any`	The value, list, dict or Daf to assign.	required

Raises:

Type	Description
`KeyError`	A row key or a column name is not found.
`KeysDisabledError`	A row is selected by key, and the Daf has no keyfield.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d[1, 'v'] = 'z'
>>> d[:, 'n'] = [1, 2, 3]
>>> d

| id | v  | n  |
| -: | -: | -: |
|  1 |  a |  1 |
|  2 |  z |  2 |
|  3 |  c |  3 |

%% daf rows=3; cols=3; keyfield='id'; name=''

>>> d[0] = {'v': 'q'}
>>> d.iloc(0)
{'id': '', 'v': 'q', 'n': ''}

A key changed after the index is built is found only after set_keyfield():

>>> k = Daf(cols=['id', 'q'], lol=[['a', 1], ['b', 2]], keyfield='id')
>>> k.select_record('a')
{'id': 'a', 'q': 1}
>>> k[0, 'id'] = 'z'
>>> k.select_record('z')
{}
>>> _ = k.set_keyfield(k.keyfield)
>>> k.select_record('z')
{'id': 'z', 'q': 1}

set_irows_icols ¶

set_irows_icols(irows, icols, value)

Set values at row positions and column positions, in place.

This is what my_daf[rows, cols] = value calls, after the selectors are turned into positions. irows and icols are an integer, a slice, a range or a list. If icols is None, the whole row is set. If irows is None, nothing is set. Use slice(None) for all rows.

What is set depends on the value.

a single value    fills every selected cell. A str or bytes is a single value.
a list            fills a selection in order. For several whole rows, each
                  row becomes a copy of the list.
a dict            sets a row. The cells of columns that the dict lacks become NULL.
a Daf             is copied as a block, from its top left corner.

Nothing is checked, and no error is raised for a size mismatch. A smaller source fills the top left of the selection and leaves the rest. A larger one fills the selection and the extra values are ignored. Check the sizes first if it matters.

Parameters:

Name	Type	Description	Default
`irows`	`slice \| int \| range \| T_li \| Iterable \| None`	The row positions.	required
`icols`	`slice \| int \| range \| T_li \| None`	The column positions.	required
`value`	`Any`	The value, list, dict or Daf to set.	required

Returns:

Type	Description
`'Daf'`	This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'])
>>> d.set_irows_icols([0, 1], [1, 2], 'Z')

| id | v  | n  |
| -: | -: | -: |
|  1 |  Z |  Z |
|  2 |  Z |  Z |
|  3 |  c | 30 |

%% daf rows=3; cols=3; keyfield=''; name=''

>>> d.set_irows_icols(2, None, ['x', 'y', 'z']).iloc(2)
{'id': 'x', 'v': 'y', 'n': 'z'}

Rows¶

select_krows ¶

select_krows(krows, inverse=False, silent_error=False)

Select rows by key. Rows with those keys are kept, or dropped if inverse is True.

This is the same as my_daf[keys], with a choice to drop rows and to ignore keys that are not found. It works with integer keys, which brackets cannot. The result is a new Daf with its own outer row list. Each row is the same list as in this Daf. Use copy('editable') before changing cells in one without the other.

A bare tuple means an inclusive range of keys, so (1, 2) is the rows from key 1 through key 2. For a composite keyfield, give a list of tuples.

Parameters:

Name	Type	Description	Default
`krows`	`slice \| str \| T_la \| int \| T_lota \| Tuple[Any, Any] \| Iterable \| None`	A key, a list of keys, or a tuple that gives a range of keys.	required
`inverse`	`bool`	If True, drop the selected rows and keep the others.	`False`
`silent_error`	`bool`	If True, keys that are not found are ignored.	`False`

Returns:

Type	Description
`'Daf'`	The new Daf.

Raises:

Type	Description
`KeysDisabledError`	The Daf has no keyfield and no key index.
`KeyError`	A key is not found and `silent_error` is False.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.select_krows([3, 1])

| id | v  | n  |
| -: | -: | -: |
|  3 |  c | 30 |
|  1 |  a | 10 |

%% daf rows=2; cols=3; keyfield='id'; name=''

>>> d.select_krows([1, 2], inverse=True)

| id | v  | n  |
| -: | -: | -: |
|  3 |  c | 30 |

%% daf rows=1; cols=3; keyfield='id'; name=''

>>> d.select_krows([1, 9], silent_error=True)

| id | v  | n  |
| -: | -: | -: |
|  1 |  a | 10 |

%% daf rows=1; cols=3; keyfield='id'; name=''

select_irows ¶

select_irows(irows, inverse=False, invert=False)

Select rows by position. Those rows are kept, or dropped if inverse is True.

This is the same as my_daf[rows], with a choice to drop rows. It is cheap. The result is a new Daf with its own outer row list. Each row is the same list as in this Daf, so changing a cell in the result changes it here too. This is also so when the selection is empty and inverse is True, which keeps all the rows. The keyfield, dtypes and column names carry over.

Parameters:

Name	Type	Description	Default
`irows`	`slice \| int \| T_li \| range \| T_lor \| Iterable \| None`	A position, a slice, a range, a list of positions, or a list of ranges.	required
`inverse`	`bool`	If True, drop the selected rows and keep the others.	`False`
`invert`	`bool`	The old name of `inverse`, still accepted.	`False`

Returns:

Type	Description
`'Daf'`	The new Daf.

Raises:

Type	Description
`IndexError`	A single position is out of range.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.select_irows(1)

| id | v  | n  |
| -: | -: | -: |
|  2 |  b | 20 |

%% daf rows=1; cols=3; keyfield='id'; name=''

>>> d.select_irows(1, inverse=True)

| id | v  | n  |
| -: | -: | -: |
|  1 |  a | 10 |
|  3 |  c | 30 |

%% daf rows=2; cols=3; keyfield='id'; name=''

>>> d.select_irows(1, invert=True)

| id | v  | n  |
| -: | -: | -: |
|  1 |  a | 10 |
|  3 |  c | 30 |

%% daf rows=2; cols=3; keyfield='id'; name=''

>>> d.select_irows(slice(1, None))

| id | v  | n  |
| -: | -: | -: |
|  2 |  b | 20 |
|  3 |  c | 30 |

%% daf rows=2; cols=3; keyfield='id'; name=''

irow ¶

irow(irow=0, include_cols=None)

Get one row as a dict, by position.

This is iloc() with the default rtype.

Parameters:

Name	Type	Description	Default
`irow`	`int`	The row position.	`0`
`include_cols`	`T_ls \| None`	Only these columns are included.	`None`

Returns:

Type	Description
`T_da`	The row as a dict. A negative position counts from the end. A Daf with no rows
`T_da`	gives an empty dict.

Raises:

Type	Description
`IndexError`	The position is out of range, for a Daf that has rows.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.irow(1, include_cols=['n'])
{'n': 20}

irow_la ¶

irow_la(irow)

Get one row as a list, by position.

The list is the row itself, not a copy. Changing it changes the Daf. Use iloc() with rtype='list' for a copy.

Parameters:

Name	Type	Description	Default
`irow`	`int`	The row position.	required

Returns:

Type	Description
`T_la`	The row.

Raises:

Type	Description
`IndexError`	The position is out of range.

Examples:

>>> d = Daf(lol=[['r1', 'a', 10], ['r2', 'b', ''], ['r3', 'a', 30]], cols=['id', 'v', 'n'], keyfield='id')

>>> d.irow_la(1)
['r2', 'b', '']
>>> row = d.irow_la(0)
>>> row[1] = 'Z'
>>> d.iloc(0)
{'id': 'r1', 'v': 'Z', 'n': 10}
>>> d.irow_la(9)
Traceback (most recent call last):
    ...
IndexError: list index out of range

iloc ¶

iloc(irow=0, include_cols=None, rtype='dict')

Get one row by position, as a dict, a KeyedList or a list.

A negative position counts from the end, so -1 is the last row. A position that is out of range raises IndexError. A Daf with no rows, and a row with no cells, give an empty dict, or an empty KeyedList. A dict or a KeyedList needs column names, so a Daf that has rows and no column names raises KeysDisabledError. Name them with set_cols(), or ask for rtype='list'.

Parameters:

Name	Type	Description	Default
`irow`	`int`	The row position.	`0`
`include_cols`	`T_ls \| None`	Only these columns are included. This applies to a dict.	`None`
`rtype`	`str`	`dict` for a new dict, `klist` for a KeyedList that shares the row, or `list` for a copy of the row.	`'dict'`

Returns:

Type	Description
`T_ma \| T_la`	The row.

Raises:

Type	Description
`IndexError`	The position is out of range, for a Daf that has rows.
`KeysDisabledError`	`rtype` is `dict` or `klist`, and the Daf has rows and no column names.
`ValueError`	`rtype` is not one of the three names.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.iloc(1)
{'id': 2, 'v': 'b', 'n': 20}
>>> d.iloc(1, rtype='list')
[2, 'b', 20]
>>> d.iloc(-1)
{'id': 3, 'v': 'c', 'n': 30}

select_record ¶

select_record(key, silent_error=True)

Get one row as a dict, by its key.

If the key is not found, the answer is an empty dict, unless silent_error is False. Then a KeyError is raised. A typo in a key gives an empty dict, so check it, or pass silent_error=False. An empty Daf gives an empty dict. For a composite keyfield the key is a tuple.

Parameters:

Name	Type	Description	Default
`key`	`str \| int \| T_ta`	The key of the row.	required
`silent_error`	`bool`	If False, raise an error when the key is not found.	`True`

Returns:

Type	Description
`T_da`	The row as a dict, or an empty dict.

Raises:

Type	Description
`KeysDisabledError`	The Daf has rows but no keyfield and no key index.
`KeyError`	The key is not found and `silent_error` is False.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.select_record(2)
{'id': 2, 'v': 'b', 'n': 20}
>>> d.select_record(9)
{}

select_records_daf ¶

select_records_daf(
    keys_ls, inverse=False, silent_error=False
)

Select several rows by key and return them as a Daf.

This is select_krows() with a friendlier answer for an empty list of keys. No keys gives an empty Daf, or all the rows if inverse is True.

The keys may be another Daf. Its keys are then used, through a view of its key index, so no list of them is made. That other Daf must have a keyfield. Its rows are not changed. The keys come in the order of its key index, so the rows selected come in that order too. A key that repeats in this Daf selects only its last row. See keys().

The rows are shared with this Daf, as in select_krows(). The new Daf has its own row list, also with no keys and inverse True, so adding a row to one does not add it to the other.

Parameters:

Name	Type	Description	Default
`keys_ls`	`'T_ls \| Iterable \| Daf'`	The keys of the rows, or a Daf whose keys are used.	required
`inverse`	`bool`	If True, drop the selected rows and keep the others.	`False`
`silent_error`	`bool`	If True, keys that are not found are ignored.	`False`

Returns:

Type	Description
`'Daf'`	The new Daf.

Raises:

Type	Description
`KeysDisabledError`	This Daf, or a Daf given as the keys, has no keyfield and no key index.
`KeyError`	A key is not found and `silent_error` is False.

Examples:

>>> d = Daf(lol=[['r1', 'a', 10], ['r2', 'b', ''], ['r3', 'a', 30]], cols=['id', 'v', 'n'], keyfield='id')

>>> d.select_records_daf(['r3', 'r1'])

| id | v  | n  |
| -: | -: | -: |
| r3 |  a | 30 |
| r1 |  a | 10 |

%% daf rows=2; cols=3; keyfield='id'; name=''

>>> d.select_records_daf([])


%% daf rows=0; cols=0; keyfield='id'; name=''

>>> kept = d.select_records_daf([], inverse=True)
>>> kept == d
True
>>> _ = kept.append(['r4', 'd', 40])
>>> d.num_rows(), kept.num_rows()
(3, 4)
>>> kept[0, 'v'] = 'Q'
>>> d.iloc(0)['v']
'Q'

The keys of another Daf, without making a list of them:

>>> other = Daf(lol=[['r3', 1], ['r9', 2]], cols=['id', 'w'], keyfield='id')
>>> d.select_records_daf(other, silent_error=True).to_lod()
[{'id': 'r3', 'v': 'a', 'n': 30}]

Columns¶

select_kcols ¶

select_kcols(
    kcols, inverse=False, flip=False, silent_error=False
)

Select columns by name. Columns with those names are kept, or dropped if inverse is True.

This is the same as my_daf[:, names], with more choices. The result is a new Daf with new rows, in the order of the names you give. With flip=True the selected columns become rows, and the result has no column names and no keyfield. This costs no more than selecting the columns.

Selecting columns copies data, so it is not cheap. For apply and reduce, use their cols argument instead.

Parameters:

Name	Type	Description	Default
`kcols`	`slice \| str \| T_la \| int \| Tuple[Any, Any] \| None`	A name, a list of names, or a tuple that gives a range of names.	required
`inverse`	`bool`	If True, drop the named columns and keep the others.	`False`
`flip`	`bool`	If True, turn the selected columns into rows.	`False`
`silent_error`	`bool`	If True, names that are not found are ignored.	`False`

Returns:

Type	Description
`'Daf'`	The new Daf.

Raises:

Type	Description
`KeysDisabledError`	The Daf has no column names.
`KeyError`	A name is not found and `silent_error` is False.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.select_kcols(['n', 'id'])

| n  | id |
| -: | -: |
| 10 |  1 |
| 20 |  2 |
| 30 |  3 |

%% daf rows=3; cols=2; keyfield='id'; name=''

>>> d.select_kcols('v', inverse=True).columns()
['id', 'n']
>>> d.select_kcols('v', flip=True)

| A  | B  | C  |
| -: | -: | -: |
|  a |  b |  c |

%% daf rows=1; cols=3; keyfield=''; name=''

select_icols ¶

select_icols(icols, flip=False)

Select columns by position. Those columns are kept, in the order given.

This is the same as my_daf[:, cols]. It makes new rows, so it is not cheap. For apply and reduce, use their cols argument instead. A slice works as it does for a Python list. The keyfield and dtypes carry over if their columns are kept. With flip=True the columns become rows, and the result has no column names, no dtypes and no keyfield.

With flip=True the columns are turned into rows as they are selected. That costs less than selecting them and then calling transpose(). In one test with 5 of 50 columns and 20,000 rows it took 0.003 s, against 0.034 s.

Parameters:

Name	Type	Description	Default
`icols`	`slice \| int \| T_li \| range \| T_lor \| None`	A position, a slice, a range, a list of positions, or a list of ranges.	required
`flip`	`bool`	If True, turn the selected columns into rows.	`False`

Returns:

Type	Description
`'Daf'`	The new Daf.

Raises:

Type	Description
`IndexError`	A position is beyond the end of some row.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.select_icols([2, 0])

| n  | id |
| -: | -: |
| 10 |  1 |
| 20 |  2 |
| 30 |  3 |

%% daf rows=3; cols=2; keyfield='id'; name=''

>>> d.select_icols(slice(-2, None)).columns()
['v', 'n']
>>> d.select_icols([0, 1], flip=True)

| A  | B  | C  |
| -: | -: | -: |
|  1 |  2 |  3 |
|  a |  b |  c |

%% daf rows=2; cols=3; keyfield=''; name=''

select_cols ¶

select_cols(cols=None, exclude_cols=None)

Make a new Daf with only some columns, chosen by name.

The columns come in the order of cols, as with select_kcols() and my_daf[:, cols]. A name given twice is used once. A name that is not a column raises KeyError. With no cols, all columns are kept, in the order of this Daf. Then exclude_cols leaves out the columns you name. A name in exclude_cols that is not a column is ignored.

This copies data, so it is not cheap. For apply and reduce, use their cols argument instead. The keyfield carries over if its column is kept. A Daf with no column names gives rows with no columns.

Parameters:

Name	Type	Description	Default
`cols`	`T_ls \| None`	The names of the columns to keep, in the order you want. If empty, all columns.	`None`
`exclude_cols`	`T_ls \| None`	The names of the columns to leave out.	`None`

Returns:

Type	Description
`'Daf'`	The new Daf.

Raises:

Type	Description
`KeyError`	A name in `cols` is not a column.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.select_cols(['n', 'id']).columns()
['n', 'id']
>>> d.select_cols(exclude_cols=['v']).columns()
['id', 'n']

col ¶

col(
    colname,
    *,
    unique=False,
    omit_nulls=False,
    silent_error=False,
    astype=None,
    indirect_col=None,
    default="",
)

Get one column as a list, by name.

This does not make a Daf first, as my_daf[:, 'v'].to_list() does.

With indirect_col, a name that is not a column is read from the dict held in that column, row by row. A row that lacks it gets default.

Parameters:

Name	Type	Description	Default
`colname`	`str`	The column name.	required
`unique`	`bool`	If True, leave out repeated values and keep the order.	`False`
`omit_nulls`	`bool`	If True, leave out the empty values.	`False`
`silent_error`	`bool`	If True, a column that is not found gives an empty list.	`False`
`astype`	`Callable \| str \| type \| None`	A type or function to convert each value with.	`None`
`indirect_col`	`str \| None`	A column that holds a dict, to read the name from.	`None`
`default`	`Any \| None`	The value for a row that lacks the name. It is used only with `indirect_col`.	`''`

Returns:

Type	Description
`list`	The values of the column.

Raises:

Type	Description
`ColumnNotFoundError`	The column is not found and `silent_error` is False. This is a `KeyError`, and also a `RuntimeError`, which this method raised in earlier versions.
`RuntimeError`	The name is empty.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.col('v')
['a', 'b', 'c']
>>> d.col('n', astype=str)
['10', '20', '30']

col_to_la ¶

col_to_la(
    colname,
    *,
    unique=False,
    omit_nulls=False,
    silent_error=False,
    astype=None,
    indirect_col=None,
    default="",
)

Get one column as a list, by name.

This does the same as col(). See that method.

Parameters:

Name	Type	Description	Default
`colname`	`str`	The column name.	required
`unique`	`bool`	If True, leave out repeated values and keep the order.	`False`
`omit_nulls`	`bool`	If True, leave out the empty values.	`False`
`silent_error`	`bool`	If True, a column that is not found gives an empty list.	`False`
`astype`	`Callable \| str \| type \| None`	A type or function to convert each value with.	`None`
`indirect_col`	`str \| None`	A column that holds a dict, to read the name from.	`None`
`default`	`Any \| None`	The value for a row that lacks the name. It is used only with `indirect_col`.	`''`

Returns:

Type	Description
`list`	The values of the column.

Raises:

Type	Description
`ColumnNotFoundError`	The column is not found and `silent_error` is False. This is a `KeyError`, and also a `RuntimeError`, which this method raised in earlier versions.
`RuntimeError`	The name is empty.

Examples:

>>> d = Daf(lol=[['r1', 'a', 10], ['r2', 'b', ''], ['r3', 'a', 30]], cols=['id', 'v', 'n'], keyfield='id')

>>> d.col_to_la('v')
['a', 'b', 'a']
>>> d.col_to_la('v', unique=True)
['a', 'b']
>>> d.col_to_la('n', omit_nulls=True)
[10, 30]
>>> d.col_to_la('n', astype=str)
['10', '', '30']
>>> d.col_to_la('zz', silent_error=True)
[]

icol ¶

icol(icol)

Get one column as a list, by position.

A negative position counts from the end, so -1 is the last column. A position that is out of range raises IndexError. A Daf with no rows gives an empty list.

Parameters:

Name	Type	Description	Default
`icol`	`int`	The column position.	required

Returns:

Type	Description
`list`	The values of the column.

Raises:

Type	Description
`IndexError`	The position is out of range, for a Daf that has rows.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.icol(1)
['a', 'b', 'c']
>>> d.icol(-1)
[10, 20, 30]

icol_to_la ¶

icol_to_la(icol, unique=False, omit_nulls=False)

Get one column as a list, by position, with options.

A negative position counts from the end, so -1 is the last column. A position that is out of range raises IndexError. A Daf with no rows gives an empty list.

Parameters:

Name	Type	Description	Default
`icol`	`int`	The column position.	required
`unique`	`bool`	If True, leave out repeated values and keep the order.	`False`
`omit_nulls`	`bool`	If True, leave out the empty values.	`False`

Returns:

Type	Description
`list`	The values of the column.

Raises:

Type	Description
`IndexError`	The position is out of range, for a Daf that has rows.

Examples:

>>> d = Daf(lol=[['r1', 'a', 10], ['r2', 'b', ''], ['r3', 'a', 30]], cols=['id', 'v', 'n'], keyfield='id')

>>> d.icol_to_la(1)
['a', 'b', 'a']
>>> d.icol_to_la(-1)
[10, '', 30]
>>> d.icol_to_la(1, unique=True)
['a', 'b']
>>> d.icol_to_la(5)
Traceback (most recent call last):
    ...
IndexError: icol: column position 5 is out of range for 3 columns.

By condition¶

select_where ¶

select_where(where, indirect_col=None)

Select the rows for which a function is true.

The function gets each row, as a KeyedList. Read cells by column name, as in row['n']. Values are used as stored, so convert text first if the Daf was read from a CSV.

With indirect_col, a name that is not a column is looked up in the dict held in that column.

If the test only compares columns to values, use select_by_dict(). It compares the cells by position and does not call a function for each row. For 200,000 rows by 50 columns it took 0.012 s, against 0.15 s for this method.

The new Daf shares the selected rows with this one. The keyfield and dtypes carry over.

To test a value against a list, a set or another table, write the test in the function. There is no need to build a list of bools first. Build a set of the values before the call, so that each lookup is fast and the set is built once. The function can use and, or, not and any other Python. It is called once for each row, so for a test on one column of a large Daf it is not the fastest way. For a test of equality, or of "any of these values", use select_by_dict() with a list of dicts, as in select_by_dict([{'name': n} for n in names]). It took 0.009 s where select_where() took 0.15 s, for 200,000 rows and 100 names. A comprehension over col(), followed by select_irows(), is also faster than a function.

Parameters:

Name	Type	Description	Default
`where`	`Callable`	A function that takes a row and returns True to keep it.	required
`indirect_col`	`str \| None`	A column that holds a dict, to read names that are not columns from.	`None`

Returns:

Type	Description
`'Daf'`	The new Daf.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.select_where(lambda row: row['n'] > 10)

| id | v  | n  |
| -: | -: | -: |
|  2 |  b | 20 |
|  3 |  c | 30 |

%% daf rows=2; cols=3; keyfield='id'; name=''

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30], [4, 'a', 40]], cols=['id', 'v', 'n'])
>>> keep = {'a', 'c'}
>>> d.select_where(lambda row: row['v'] in keep)

| id | v  | n  |
| -: | -: | -: |
|  1 |  a | 10 |
|  3 |  c | 30 |
|  4 |  a | 40 |

%% daf rows=3; cols=3; keyfield=''; name=''

>>> d.select_where(lambda row: row['v'] not in keep)

| id | v  | n  |
| -: | -: | -: |
|  2 |  b | 20 |

%% daf rows=1; cols=3; keyfield=''; name=''

>>> d.select_where(lambda row: row['v'] in keep and row['n'] > 15)

| id | v  | n  |
| -: | -: | -: |
|  3 |  c | 30 |
|  4 |  a | 40 |

%% daf rows=2; cols=3; keyfield=''; name=''

>>> d.select_where(lambda row: row['id'] % 2 == 0 or row['v'] == 'c')

| id | v  | n  |
| -: | -: | -: |
|  2 |  b | 20 |
|  3 |  c | 30 |
|  4 |  a | 40 |

%% daf rows=3; cols=3; keyfield=''; name=''

The values can come from another Daf. Make the set once, outside the function:

>>> other = Daf(lol=[['a'], ['c']], cols=['v'], keyfield='v')
>>> other_keys = set(other.keys())
>>> d.select_where(lambda row: row['v'] in other_keys)

| id | v  | n  |
| -: | -: | -: |
|  1 |  a | 10 |
|  3 |  c | 30 |
|  4 |  a | 40 |

%% daf rows=3; cols=3; keyfield=''; name=''

For a large Daf, the faster form picks the positions from the column:

>>> d.select_irows([irow for irow, v in enumerate(d.col('v')) if v in keep])

| id | v  | n  |
| -: | -: | -: |
|  1 |  a | 10 |
|  3 |  c | 30 |
|  4 |  a | 40 |

%% daf rows=3; cols=3; keyfield=''; name=''

select_where_idxs ¶

select_where_idxs(where)

Get the positions of the rows for which a function is true.

The function gets each row as a KeyedList, whatever the itermode is, as in select_where(). Read cells by column name, as in row['n']. For a test that only compares columns to values, select_by_dict() is faster.

The positions are a snapshot. Inserting, removing or sorting rows makes them point to other rows. Use them right away. To keep a reference to a row for longer, keep its key.

Parameters:

Name	Type	Description	Default
`where`	`Callable`	A function that takes a row and returns True to keep it.	required

Returns:

Type	Description
`T_li`	The row positions.

Raises:

Type	Description
`KeysDisabledError`	The Daf has rows and no column names.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.select_where_idxs(lambda row: row['n'] > 10)
[1, 2]

split_where ¶

split_where(where, *, indirect_col=None)

Split the rows in two, by a function that is true or false for each row.

This is deprecated. Use group_where() with a function that returns True or False. It gives the two groups under the keys True and False. A side with no rows has no group, so read each side with .get().

The function gets each row, as in select_where(). Both new Daf instances share their rows with this one, so changing a cell in one changes it here too. The keyfield and dtypes carry over to both.

Parameters:

Name	Type	Description	Default
`where`	`Callable`	A function that takes a row and returns True or False.	required
`indirect_col`	`str \| None`	A column that holds a dict, to read names that are not columns from.	`None`

Returns:

Type	Description
`Tuple['Daf', 'Daf']`	A tuple of the Daf of the rows where the function is true and the Daf of the others.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> big, small = d.split_where(lambda row: row['n'] > 10)
>>> big

| id | v  | n  |
| -: | -: | -: |
|  2 |  b | 20 |
|  3 |  c | 30 |

%% daf rows=2; cols=3; keyfield='id'; name=''

>>> small

| id | v  | n  |
| -: | -: | -: |
|  1 |  a | 10 |

%% daf rows=1; cols=3; keyfield='id'; name=''

select_by_dict ¶

select_by_dict(
    selector_da, expectmax=-1, inverse=False, keyfield=""
)

Select the rows that match every field of a dict, or any one of a list of dicts.

With a dict, a row matches if each key is a column whose value in that row equals the value given. With a list of dicts, a row matches if it matches at least one of the dicts. So a dict is an AND of equalities, and a list is an OR of those. With inverse=True the rows that do not match are returned. The cells are compared by position, so no row is turned into a dict.

To select the rows whose cell is any of several values, give one small dict for each value, as in [{'name': n} for n in names]. To select by several columns together, give a dict for each combination, as in [{'a': 1, 'b': 2}, {'a': 3, 'b': 4}]. The dicts are grouped by their keys, and the values of each group are put in a set, so a row costs about one lookup however many dicts there are. This is much faster than select_where() with a function. The result is the same as comparing with ==. A value or a cell that cannot be hashed, such as a list or a dict held in a cell, is compared with == as well. It never raises an error, and it takes a slower path for that call.

A value in a dict is only ever compared for equality. A list or a set given as a value matches a cell that holds an equal list or set, and not a cell that is one of its items. To say "any of these", use a list of dicts.

An empty list matches no row. An empty dict matches every row. A dict that names a column that is not there matches no row, and the other dicts still apply.

A Daf with rows and no column names raises KeysDisabledError. The new Daf is a live view of this one. It has its own outer row list, and its rows are the same lists as the rows of this Daf. Changing a value in it changes this Daf. Adding a column with insert_col() or insert_idx_col() does not, because those copy shared rows first. Use copy('editable') for rows of your own.

Parameters:

Name	Type	Description	Default
`selector_da`	`T_da \| T_loda`	A dict of column names and the values they must have, or a list or tuple of such dicts.	required
`expectmax`	`int`	If this is not -1 and more rows match, raise `LookupError`.	`-1`
`inverse`	`bool`	If True, return the rows that do not match.	`False`
`keyfield`	`str \| int \| T_ta`	The keyfield of the new Daf. If empty, the keyfield of this Daf.	`''`

Returns:

Type	Description
`'Daf'`	The new Daf.

Raises:

Type	Description
`LookupError`	More than `expectmax` rows match.
`KeysDisabledError`	The Daf has rows and no column names.
`TypeError`	`selector_da` is not a dict, or a list or tuple of dicts.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.select_by_dict({'v': 'b'})

| id | v  | n  |
| -: | -: | -: |
|  2 |  b | 20 |

%% daf rows=1; cols=3; keyfield='id'; name=''

>>> d.select_by_dict({'v': 'b'}, inverse=True)

| id | v  | n  |
| -: | -: | -: |
|  1 |  a | 10 |
|  3 |  c | 30 |

%% daf rows=2; cols=3; keyfield='id'; name=''

>>> d.select_by_dict([{'v': 'a'}, {'v': 'c'}])

| id | v  | n  |
| -: | -: | -: |
|  1 |  a | 10 |
|  3 |  c | 30 |

%% daf rows=2; cols=3; keyfield='id'; name=''

>>> d.select_by_dict([{'v': 'a', 'n': 10}, {'v': 'c', 'n': 99}])

| id | v  | n  |
| -: | -: | -: |
|  1 |  a | 10 |

%% daf rows=1; cols=3; keyfield='id'; name=''

>>> names = ['a', 'b']
>>> d.select_by_dict([{'v': name} for name in names], inverse=True)

| id | v  | n  |
| -: | -: | -: |
|  3 |  c | 30 |

%% daf rows=1; cols=3; keyfield='id'; name=''

select_first_row_by_dict ¶

select_first_row_by_dict(selector_da, inverse=False)

Get the first row that matches every field of a dict.

The matching rule is that of select_by_dict(). With inverse=True it is the first row that does not match.

Parameters:

Name	Type	Description	Default
`selector_da`	`T_da`	The column names and the values they must have.	required
`inverse`	`bool`	If True, find the first row that does not match.	`False`

Returns:

Type	Description
`T_ma`	The row, as a dict or a KeyedList according to `itermode`. An empty dict if none matches.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.select_first_row_by_dict({'v': 'b'})
{'id': 2, 'v': 'b', 'n': 20}
>>> d.select_first_row_by_dict({'v': 'zz'})
{}

Index translation¶

krows_to_irows ¶

krows_to_irows(krows, inverse=False, silent_error=False)

Turn row keys into row positions.

This is the step that lets my_daf['r2'] work. Call it when you need the positions themselves. The Daf must have a keyfield, or a key index passed in as kd.

The keys may be a key, a list of keys, or a tuple that gives an inclusive range of keys. A range is made from positions, so the keys must be in the same order as the rows.

The first lookup builds the key index, by reading the keyfield column. In one test with 200,000 rows that took 0.05 s, and a later lookup took about 2 microseconds. Adding or removing rows clears the index, and the next lookup builds it again.

Parameters:

Name	Type	Description	Default
`krows`	`slice \| str \| T_la \| int \| Tuple[Any, Any] \| T_lota \| Iterable \| None`	A key, a list of keys, or a tuple that gives a range of keys.	required
`inverse`	`bool`	If True, return the positions of the rows that are not selected.	`False`
`silent_error`	`bool`	If True, keys that are not found are ignored.	`False`

Returns:

Type	Description
`slice \| int \| T_li \| range`	The row positions.

Raises:

Type	Description
`KeysDisabledError`	There is no keyfield and no key index, or the keyfield is not a column.
`KeyError`	A key is not found and `silent_error` is False.

Examples:

>>> d = Daf(lol=[['r1', 'a', 10], ['r2', 'b', ''], ['r3', 'a', 30]], cols=['id', 'v', 'n'], keyfield='id')

>>> d.krows_to_irows('r2')
[1]
>>> d.krows_to_irows(['r3', 'r1'])
[2, 0]
>>> d.krows_to_irows(('r1', 'r2'))
slice(0, 2, 1)
>>> d.krows_to_irows(['r1'], inverse=True)
[1, 2]
>>> d.krows_to_irows(['zz'], silent_error=True)
[]
>>> d.krows_to_irows(['zz'])
Traceback (most recent call last):
    ...
KeyError: 'zz'

kcols_to_icols ¶

kcols_to_icols(
    kcols=None, inverse=False, silent_error=False
)

Turn column names into column positions.

This is the step that lets my_daf[:, 'v'] work. A name that looks like an integer is still read as a name.

Parameters:

Name	Type	Description	Default
`kcols`	`str \| T_ls \| slice \| int \| T_li \| Tuple[Any, Any] \| Iterable \| None`	A name, a list of names, or a tuple that gives an inclusive range of names.	`None`
`inverse`	`bool`	If True, return the positions of the columns that are not selected.	`False`
`silent_error`	`bool`	If True, names that are not found are ignored.	`False`

Returns:

Type	Description
`slice \| int \| T_li \| range \| None`	The column positions. With no column names the result is empty, or all
`slice \| int \| T_li \| range \| None`	positions if `inverse` is True.

Raises:

Type	Description
`KeyError`	A name is not found and `silent_error` is False.

Examples:

>>> d = Daf(lol=[['r1', 'a', 10], ['r2', 'b', ''], ['r3', 'a', 30]], cols=['id', 'v', 'n'], keyfield='id')

>>> d.kcols_to_icols('v')
[1]
>>> d.kcols_to_icols(['n', 'id'])
[2, 0]
>>> d.kcols_to_icols(('id', 'v'))
slice(0, 2, 1)
>>> d.kcols_to_icols('v', inverse=True)
[0, 2]
>>> d.kcols_to_icols('zz')
Traceback (most recent call last):
    ...
KeyError: 'zz'

gkeys_to_idxs `staticmethod` ¶

gkeys_to_idxs(
    keydict,
    gkeys=None,
    inverse=False,
    silent_error=False,
    axis="rowkeys",
    name="unspecified",
)

Turn keys into positions, using a dict of key to position.

This is the step under krows_to_irows() and kcols_to_icols(). It is a static method. It is for internal use.

A key gives a list with one position. A list of keys gives their positions in that order. A tuple of two keys is an inclusive range, and None in it means the start or the end. A tuple of one key means that key to the end. A slice is taken as positions. With inverse, the positions that are not selected are returned.

Parameters:

Name	Type	Description	Default
`keydict`	`Dict[str \| int, int]`	Maps each key to its position.	required
`gkeys`	`str \| T_ls \| slice \| int \| T_li \| Tuple[Any, Any] \| T_lota \| Iterable \| None`	The keys.	`None`
`inverse`	`bool`	If True, return the positions that are not selected.	`False`
`silent_error`	`bool`	If True, keys that are not found are ignored.	`False`
`axis`	`str`	A label for error messages.	`'rowkeys'`
`name`	`str`	A label for error messages.	`'unspecified'`

Returns:

Type	Description
`slice \| int \| T_li \| range`	The positions, as a list or a slice.

Raises:

Type	Description
`KeysDisabledError`	The dict is empty, as it is for a Daf with no rows.
`KeyError`	A key is not found and `silent_error` is False.
`TypeError`	The keys are None, or a tuple of a length other than 1 or 2.

Examples:

>>> Daf.gkeys_to_idxs({'a': 0, 'b': 1, 'c': 2}, ['c', 'a'])
[2, 0]
>>> Daf.gkeys_to_idxs({'a': 0, 'b': 1, 'c': 2}, ('a', 'b'))
slice(0, 2, 1)

Selecting and indexing¶

Indexing with brackets¶

__getitem__ ¶

__setitem__ ¶

set_irows_icols ¶

Rows¶

select_krows ¶

select_irows ¶

irow ¶

irow_la ¶

iloc ¶

select_record ¶

select_records_daf ¶

Columns¶

select_kcols ¶

select_icols ¶

select_cols ¶

col ¶

col_to_la ¶

icol ¶

icol_to_la ¶

By condition¶

select_where ¶

select_where_idxs ¶

split_where ¶

select_by_dict ¶

select_first_row_by_dict ¶

Index translation¶

krows_to_irows ¶

kcols_to_icols ¶

gkeys_to_idxs staticmethod ¶

getitem ¶

setitem ¶

gkeys_to_idxs `staticmethod` ¶