Selecting and indexing¶

The two coordinates¶

daf[rows, cols]     rows and columns
daf[rows]           rows, all columns
daf[:, cols]        columns, all rows

You write	As rows (first)	As columns (second)
`:`	all rows	all columns
`2`	row 2, by position	column 2, by position
`-1`	the last row	the last column
`2:5`, `2:`, `:5`, `::2`	a slice of rows	a slice of columns
`range(2, 5)`	rows 2, 3, 4	columns 2, 3, 4
`[0, 4, 2]`	these rows, in this order	these columns, in this order
`[range(0, 2), range(4, 6)]`	rows 0, 1, 4, 5	columns 0, 1, 4, 5
`'a1'`	the row with key `'a1'`	the column named `'a1'`
`['c3', 'a1']`	the rows with these keys, in this order	the columns with these names, in this order
`('a1', 'c3')`	the rows from key a1 through key c3, in row order	the columns from a1 through c3, in column order
`('a1', None)`, `('a1',)`	key a1 to the end	name a1 to the end
`(None, 'c3')`	the start through key c3	the start through name c3
`[]`	no rows	no columns

Selecting whole rows, as daf[rows] or daf[rows, :], copies no data: the new Daf shares the rows with the original, so a change to a cell in one shows in the other. Selecting some of the columns makes new row lists. Use copy('editable') for row lists of your own.
A list or dict held in a cell is the same object in every selection and in copy('editable'). Only copy('deep') copies it.
An integer alone, or in a list, slice or range, is a position. A key or a column name that is an integer needs select_krows() or select_kcols().
The two ends of a tuple are always keys or names, of any type, as in daf[(10, 20), :] for int keys. The tuple takes the rows between them in the table's order. It does not compare the values. If the second end comes first in the table, the result is empty.
Rows by key need a keyfield.
A tuple of row keys needs the column part, even if it is :, as in daf[('a1', 'c3'), :]. A tuple of two standing alone is read as [rows, cols].
A list holds integers, or ranges, or strings, not a mix. None alone is not a selector.
The result is a Daf. .to_value(), .to_list() or .to_dict() give plain values, or set daf.retmode = 'val'.
The same selectors set values: daf[rows, cols] = value.

Square brackets¶

daf[rows, cols] selects rows and columns. daf[rows] selects rows with all their columns. Each part can be a position, a slice, a range, a list, a key or a column name. The result is a new Daf.

>>> from daffodil.daf import Daf
>>> daf = Daf(cols=['id', 'a', 'b', 'c'], keyfield='id',
...          lol=[['r1', 1, 2, 3], ['r2', 4, 5, 6], ['r3', 7, 8, 9], ['r4', 10, 11, 12]])
>>> daf[1].lol
[['r2', 4, 5, 6]]
>>> daf[-1].lol
[['r4', 10, 11, 12]]
>>> daf[:2].lol
[['r1', 1, 2, 3], ['r2', 4, 5, 6]]
>>> daf[-2:].lol
[['r3', 7, 8, 9], ['r4', 10, 11, 12]]
>>> daf[[0, 3], 'a'].to_list()
[1, 10]
>>> daf[:, ['c', 'a']].lol[:2]
[[3, 1], [6, 4]]

Keys and names:

>>> daf['r3'].lol
[['r3', 7, 8, 9]]
>>> daf[['r4', 'r1'], 'b'].to_list()
[11, 2]

A tuple of two keys, or of two names, is an inclusive range. None stands for the start or the end. A range of row keys needs the column part, even if it is :, because a tuple of two items standing alone is read as [rows, cols].

The range follows the order of the rows, not the order of the keys. It starts at the row of the first key and ends at the row of the second. The ends of a tuple are always keys, so a tuple of ints works for int keys.

>>> daf[('r2', 'r3'), :].lol
[['r2', 4, 5, 6], ['r3', 7, 8, 9]]
>>> daf[(None, 'r2'), 'id'].to_list()
['r1', 'r2']
>>> daf[('r3', None), 'id'].to_list()
['r3', 'r4']
>>> daf[0, ('a', 'b')].to_list()
[1, 2]

Here the keys are not in sorted order. The range from b to a takes b, z and a, the rows between them in the table. The range from a to b is empty, as a comes after b in the table:

>>> unsorted = Daf(cols=['id', 'v'], keyfield='id', lol=[['m', 1], ['b', 2], ['z', 3], ['a', 4]])
>>> unsorted[('b', 'a'), :].col('id')
['b', 'z', 'a']
>>> unsorted[('a', 'b'), :].col('id')
[]
>>> by_number = Daf(cols=['n', 'v'], keyfield='n', lol=[[10, 'x'], [20, 'y'], [30, 'z']])
>>> by_number[(20, 30), :].col('n')
[20, 30]

Outside a tuple, an integer in [] is always a position. To reach a key or a column name that is an integer, use select_krows() and select_kcols().

Getting values out¶

A selection is a Daf. Turn it into values with to_value() for one cell, to_list() for one row or column, and to_dict() for one row. Or read the values directly, with the methods below, which build no table and are faster.

>>> daf[2, 'b'].to_value()
8
>>> daf.col('b')
[2, 5, 8, 11]
>>> daf.select_record('r2')
{'id': 'r2', 'a': 4, 'b': 5, 'c': 6}
>>> daf.iloc(0)
{'id': 'r1', 'a': 1, 'b': 2, 'c': 3}

With retmode = 'val', a selection of one cell, one row or one column returns the value or a list instead of a Daf:

>>> daf.retmode = 'val'
>>> daf[2, 'b'], daf['r1'], daf[:, 'a']
(8, ['r1', 1, 2, 3], [1, 4, 7, 10])
>>> daf.retmode = 'obj'

Selecting by condition¶

select_where() keeps the rows for which a function is true. The function gets each row as a KeyedList, which reads like a dict:

>>> daf.select_where(lambda row: row['a'] > 5).col('id')
['r3', 'r4']

For a test of equality, select_by_dict() is much faster, as it compares the cells without calling a function for each row. A list of dicts means any of them:

>>> daf.select_by_dict({'b': 5}).col('id')
['r2']
>>> daf.select_by_dict([{'b': 5}, {'b': 11}]).col('id')
['r2', 'r4']

inverse=True keeps the rows that do not match, in select_by_dict(), select_irows() and select_krows():

>>> daf.select_krows(['r1', 'r4'], inverse=True).col('id')
['r2', 'r3']

Setting values¶

The same selectors set values. A single value fills the selection. A list fills it in order. A dict given for a row is placed by column name, and a column it lacks becomes NULL.

>>> daf[0, 'a'] = 100
>>> daf[1] = ['r2', 40, 50, 60]
>>> daf[:, 'c'] = 0
>>> daf.lol[:2]
[['r1', 100, 2, 0], ['r2', 40, 50, 0]]

Do selections copy the rows?¶

A selection of rows is a new Daf with a new list of rows, but each row in it is the same list as in the original. That is why it is fast. It also means that changing a cell in a selection changes the original:

>>> sel = daf.select_where(lambda row: row['id'] == 'r1')
>>> sel[0, 'b'] = 99
>>> daf.select_record('r1')['b']
99

Selection	The rows of the result
`daf[rows]`, `select_irows`, `select_krows`, `select_records_daf`	shared
`select_where`, `split_where`, `select_by_dict`	shared
`groupby`, `multi_groupby`, `groupby_cols`, `group_where`, `copy()`	shared
a selection of columns: `daf[:, cols]`, `select_cols`, `select_kcols`, `select_icols`	new
`groupby`, `multi_groupby` and `groupby_cols` with only some columns kept	new

To change a selection without changing the original, copy it first with copy('editable'), which gives each row its own list. Adding a column to a selection, as with insert_col(), copies the shared rows first, so the original keeps its shape.

New row lists still hold the same cell values. A list or dict in a cell is one object, seen from the original, from any selection, and from copy('editable'). Changing it in place shows everywhere. Only copy('deep') copies it:

>>> tagged = Daf(cols=['id', 'tags'], lol=[['a', ['x']]])
>>> cols_only = tagged[:, ['tags']]
>>> cols_only[0, 'tags'].to_value().append('y')
>>> tagged.lol
[['a', ['x', 'y']]]
>>> deep = tagged.copy('deep')
>>> deep[0, 'tags'].to_value().append('z')
>>> tagged.lol
[['a', ['x', 'y']]]

The same holds for to_lod(), to_dict(), the rows of a loop, and a KeyedList: each is new, but its values are the cells of the table.

Changing a key¶

The key index maps each key to its row. It is built at the first lookup, and a write with [] does not update it. So after you change a key cell, or replace a row, the old key still finds the row, and the new key finds nothing. Call set_keyfield() with the same keyfield to have the index rebuilt:

>>> stock = Daf(cols=['id', 'qty'], keyfield='id', lol=[['a1', 3], ['b2', 10]])
>>> stock.select_record('a1')
{'id': 'a1', 'qty': 3}
>>> stock[0, 'id'] = 'z9'
>>> stock.select_record('z9')
{}
>>> _ = stock.set_keyfield(stock.keyfield)
>>> stock.select_record('z9')
{'id': 'z9', 'qty': 3}

A selection that shares the row has its own index, so do the same on each Daf that looks up keys after the change.