Skip to content

Structure, columns and keys

The shape of a Daf, its column names, its keyfield, copying, and iteration.

Size and shape

num_rows

num_rows()

Return the number of rows.

Returns:

Type Description
int

The number of rows.

Examples:

>>> Daf(cols=['x'], lol=[[1], [2], [3]]).num_rows()
3
>>> Daf().num_rows()
0

num_cols

num_cols()

Count the columns by looking at the rows.

This does not use the column names. It returns the length of the longest of the first 10 rows, and 0 if there are no rows. For a table with equal row lengths that is the real width. Use len(d.columns()) to count the names, and is_rectangular() to check the rows.

Returns:

Type Description
int

The number of columns.

Examples:

>>> Daf(lol=[[1, 2, 3]], cols=['a', 'b']).num_cols()
3
>>> Daf(cols=['a', 'b']).num_cols()
0

len

len()

Return the number of rows.

This does the same as num_rows() and len(d).

Returns:

Type Description
int

The number of rows.

Examples:

>>> d = Daf(cols=['x'], lol=[[1], [2], [3]])
>>> d.len(), len(d)
(3, 3)

__len__

__len__()

Return the number of rows, so len(d) works.

Returns:

Type Description
int

The number of rows.

shape

shape()

Return the number of rows and columns as a tuple.

This is a method, not a property, because the column count is worked out from the rows. It is the same as (num_rows(), num_cols()). A Daf with column names but no rows has shape (0, 0).

Returns:

Type Description
Tuple[int, int]

A tuple of the number of rows and the number of columns.

Examples:

>>> Daf(lol=[[1, 2], [3, 4], [5, 6]], cols=['a', 'b']).shape()
(3, 2)

is_rectangular

is_rectangular()

Check whether every row has the same length.

With column names, every row must be as long as the number of names. Without names, every row must be as long as the first row. An empty Daf is rectangular.

This looks at every row. Use it when you do not trust the source of the data. num_cols() only samples the first rows.

Returns:

Type Description
bool

True if all rows have the expected length.

Examples:

>>> Daf(lol=[[1, 2], [3]], cols=['a', 'b']).is_rectangular()
False

force_rectangular

force_rectangular()

Pad short rows with empty strings, in place.

Use this when a source leaves off trailing empty cells. An xlsx file read by xlsx_to_csv() does that. The target width is the number of column names. With no names, it is the length of the longest row.

Rows that are too long are not cut. That would hide damage, such as an unquoted comma that split a value. They raise an error instead.

Each short row is padded in place, so a selection that shares it sees the new cells. No row moves, so the key index stays valid.

Returns:

Type Description
'Daf'

This Daf, which has been changed.

Raises:

Type Description
ValueError

A row is longer than the number of columns. Nothing is changed.

Examples:

>>> Daf(lol=[[1, 2], [3]], cols=['a', 'b']).force_rectangular()

| a  | b  |
| -: | -: |
|  1 |  2 |
|  3 |    |

%% daf rows=2; cols=2; keyfield=''; name=''

__bool__

__bool__()

Say whether the Daf holds data.

bool(d) and if d: are true when the Daf has at least one row with a value. A Daf that has column names but no rows is false. Daffodil uses this as its test for an empty table.

Returns:

Type Description
bool

True if there is data.

Examples:

>>> bool(Daf(cols=['a']))
False
>>> bool(Daf(lol=[[1]], cols=['a']))
True

Column names

columns

columns()

Return the column names.

The list is a new copy, so changing it does not change the Daf. Use set_cols() or rename_cols() to change the names.

Returns:

Type Description
T_ls

The column names, in order.

Examples:

>>> Daf(cols=['a', 'b']).columns()
['a', 'b']

calc_cols

calc_cols(
    include_cols=None,
    exclude_cols=None,
    include_types=None,
    exclude_types=None,
)

Work out a list of column names from rules.

Use it to choose the columns for apply or reduce. The rules are applied in this order: include by name, exclude by name, include by type and exclude by type. The result keeps the order of the Daf. A single name may be given as a string.

With a group by operation, leave the group by columns out of the list.

Parameters:

Name Type Description Default
include_cols Iterable | None

Keep only these column names.

None
exclude_cols Iterable | None

Drop these column names.

None
include_types List[Type] | None

Keep only columns whose dtype is in this list.

None
exclude_types List[Type] | None

Drop columns whose dtype is in this list.

None

Returns:

Type Description
Iterable

The selected column names.

Raises:

Type Description
RuntimeError

A type rule is given and no dtypes are set.

Examples:

>>> d = Daf(lol=[[1, 2, 3]], cols=['a', 'b', 'c'])
>>> d.calc_cols(exclude_cols='b')
['a', 'c']

rename_cols

rename_cols(from_to_dict)

Rename columns in place.

Names that are not in the mapping stay as they are. Names in the mapping that are not in the Daf are ignored. The dtypes are renamed too.

The keyfield follows the rename. If its column is renamed, the keyfield has the new name, and key lookups keep working.

Parameters:

Name Type Description Default
from_to_dict T_ds

Maps old names to new names.

required

Returns:

Type Description
'Daf'

This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 2]], cols=['a', 'b'])
>>> d.rename_cols({'b': 'c'}).columns()
['a', 'c']
>>> k = Daf(lol=[[1, 'x']], cols=['id', 'v'], keyfield='id')
>>> k.rename_cols({'id': 'ident'}).keyfield
'ident'

set_cols

set_cols(
    new_cols=None, sanitize_cols=True, unnamed_prefix="col"
)

Set the column names, in place.

The new names are given by position, so the first name goes to the first column. Without a list, the names are A, B, C and so on, like a spreadsheet.

With sanitize_cols on, a repeated name gets a suffix, so ['a', 'a'] becomes ['a', 'a_1']. An empty name becomes the prefix and its position, like col2. The prefix is short, because these names are printed. Names that come from parsing a header, in the constructor and in from_md(), use Unnamed instead, which says that there is no name. The dtypes are renamed by position as well.

The keyfield follows the names, as the dtypes do. If its column gets a new name, the keyfield has that name. If the Daf had no column names yet, the keyfield is kept when it is one of the new names, so it can be given before the names are known. Otherwise it is cleared.

Parameters:

Name Type Description Default
new_cols T_ls | None

The names, in order. If None, spreadsheet names are made.

None
sanitize_cols bool

If True, make the names valid and unique.

True
unnamed_prefix str

The start of a name made for an empty one.

'col'

Returns:

Type Description
'Daf'

This Daf, which has been changed.

Raises:

Type Description
AttributeError

The number of names is not the number of columns.

Examples:

>>> Daf(lol=[[1, 2, 3]]).set_cols().columns()
['A', 'B', 'C']
>>> Daf(lol=[[1, 2, 3]]).set_cols(['a', 'a', '']).columns()
['a', 'a_1', 'col2']
>>> d = Daf(lol=[[1, 'x']], keyfield='id').set_cols(['id', 'v'])
>>> d.keyfield
'id'
>>> d = Daf(lol=[[1, 'x']], cols=['a', 'b'], keyfield='a').set_cols(['id', 'v'])
>>> d.keyfield
'id'

Keyfield

keys

keys(*, silent_error=True, astype='list')

Return the row keys.

The keys come from the key index. There is one for each distinct value of the keyfield, in the order of the first row that has it. With a composite keyfield each key is a tuple. The key index is built here if it is needed.

A repeated key appears once. The index then points to the last row that has it, so a lookup of that key finds the last row. So len(keys()) is less than len(daf) when keys repeat.

Without a keyfield the answer is an empty list, unless silent_error is False. Then it raises an error. An index passed in as kd is not used by this method.

With astype='view' you get a view of the key index, not a copy. It keeps the old keys after the Daf changes, so use it right away.

Parameters:

Name Type Description Default
silent_error bool

If False, raise an error when there is no keyfield.

True
astype str

list for a new list, or view for a view of the index.

'list'

Returns:

Type Description
T_la | T_kva

The keys.

Raises:

Type Description
KeysDisabledError

There is no keyfield and no key index, and silent_error is False.

ValueError

astype is neither list nor view.

Examples:

>>> d = Daf(lol=[[1, 'a'], [2, 'b']], cols=['id', 'v'], keyfield='id')
>>> d.keys()
[1, 2]
>>> dup = Daf(cols=['id', 'q'], lol=[['a', 1], ['b', 2], ['a', 3]], keyfield='id')
>>> dup.keys(), dup.select_record('a')
(['a', 'b'], {'id': 'a', 'q': 3})

set_keyfield

set_keyfield(
    keyfield="",
    *,
    silent_error=False,
    force_kd_rebuild=False,
)

Choose the column, or columns, that identify the rows.

Give a column name, or a tuple or list of names for a composite key. A composite key is a tuple of those values. An empty keyfield turns key lookups off. The key index is built later, when it is first needed.

The Daf must have column names first. With none, nothing happens.

A name that is not a column raises KeyError. Pass silent_error=True to store it anyway, as older versions did. Then key lookups find nothing. A key that is not unique is not checked. A lookup finds the last row that has it.

A Daf that has column names and no rows can have a keyfield. It applies to the rows that are added later.

Parameters:

Name Type Description Default
keyfield str | T_ta | T_la

Column name, or a tuple or list of names. Empty to turn off.

''
silent_error bool

If True, store a name that is not a column, and do not raise.

False
force_kd_rebuild bool

If True, build the key index now.

False

Returns:

Type Description
'Daf'

This Daf, which has been changed.

Raises:

Type Description
KeyError

The keyfield is not a column, and silent_error is not True.

Examples:

>>> d = Daf(lol=[[1, 'a'], [2, 'b']], cols=['id', 'v'])
>>> d.set_keyfield('id').keys()
[1, 2]
>>> d.set_keyfield(['id', 'v']).keys()
[(1, 'a'), (2, 'b')]
>>> e = Daf(cols=['id', 'v']).set_keyfield('id')
>>> e.keyfield
'id'
>>> e.append({'id': 7, 'v': 'x'}).keys()
[7]

get_existing_keys

get_existing_keys(keylist)

Keep the keys that are in the Daf.

Use it to find out which of a list of keys have a row. The order of the list is kept. The result is empty if the Daf has no keyfield.

Parameters:

Name Type Description Default
keylist T_ls

The keys to check.

required

Returns:

Type Description
T_ls

The keys from the list that have a row.

Examples:

>>> d = Daf(lol=[[1, 'a'], [2, 'b']], cols=['id', 'v'], keyfield='id')
>>> d.get_existing_keys([1, 5, 2])
[1, 2]

__contains__

__contains__(key)

Test whether a key is in the keyfield column.

Use it as key in my_daf. For a composite keyfield, the key is a tuple. An empty Daf contains nothing.

Parameters:

Name Type Description Default
key Any

The key to look for.

required

Returns:

Type Description
bool

True if a row has this key.

Raises:

Type Description
KeyError

The Daf has rows but no keyfield, and no key index was adopted.

Examples:

>>> d = Daf(lol=[[1, 'a'], [2, 'b']], cols=['id', 'v'], keyfield='id')
>>> 2 in d, 5 in d
(True, False)

Copying and initializing

copy

copy(
    level=None,
    name="",
    deep=False,
    for_sorting=False,
    inplace=False,
)

Make a copy of the Daf. You choose what the copy shares with the original.

Unless inplace=True, the copy is a new object of the same class. What it holds is shared with the original unless you ask for its own. Sharing is what makes daffodil fast, because nothing is copied that does not have to be. Pick the cheapest level that is safe for what you do next.

The level is one of four names, or a sum of the COPY_ constants of Daf. Without a level, the class setting copy_level_default is used. It is sortable. For copies with their own row lists by default, set it on a subclass, as in class SafeDaf(Daf): copy_level_default = 'editable'. The table gives the cost for 200,000 rows of 50 columns.

Level Parts copied Cost
shallow attrs and disp_cols 0.00002 s
sortable shallow, the row list, hd, dtypes, the key index 0.002 s, 2 MB
editable sortable, plus a new list for every row 0.5 s, 93 MB
deep everything that is a container 4 s, 107 MB

To choose your own mix, add these constants of Daf with |. A part without its bit is shared. A copy is always a new object, so there is no bit for that. The two bits COPY_ROWS and COPY_OUTER imply others: COPY_ROWS adds COPY_OUTER, and COPY_OUTER adds COPY_KD. They are added for you, because a row list that is copied while the key index is shared would leave the index wrong for one of the two. The name deep cannot be combined with any bit.

Constant Value Parts copied
COPY_NONE 0 nothing. Every part is shared.
COPY_ATTRS 1 the attrs dict, deep copied, and the disp_cols list
COPY_OUTER 2 the list that holds the rows. The rows are still shared
COPY_HD 4 the header dict
COPY_DTYPES 8 the dtypes dict
COPY_KD 16 the key index, and a keyfield that is a list
COPY_ROWS 32 a new list for every row

With shallow, the copy shares the row list, the rows, hd and dtypes with the original. Reading is safe. Sorting with sort_by_colname() is safe too, because it builds a new row list. Appending a row, sorting the row list in place or dropping a column is seen by the original, and can leave it with a key index that is wrong.

With sortable, the copy has its own row list, so you can append, extend, insert, remove, sort or reverse rows, rename columns, change the keyfield or dtypes, and use drop_cols(). The key index is cleared on the copy and is rebuilt on first use. The rows are still shared. Changing values is seen by the original, whichever way you do it. That makes a selection a live view for editing. Adding a column is not seen by the original. insert_col(), insert_icol(), insert_idx_col() and assign_col() with a new name copy shared rows first.

With editable, each row is a new list. You can also add columns and change cells. The cells themselves are shared. That is safe for text and numbers, and not for a list or dict held in a cell.

With deep, nothing that can be changed is shared. Text and numbers are not copied, because they cannot change. A list or dict held in a cell is copied.

The copy has no name unless you give one with name, so that two tables do not claim to be the same one. The other settings, such as md_max_rows, the schema and the retmode and itermode, are taken over. A schema is shared.

With inplace=True, the Daf copies into itself and returns itself, instead of making a new one. Use it on a selection, to make it a table of its own: its rows then no longer share anything with the Daf they came from. The default level is then deep. editable may be asked for, to copy only the rows. A lower level would leave the rows shared, so it raises ValueError. The name stays, unless name gives a new one.

The arguments deep and for_sorting are deprecated. Use level='deep' and level='sortable' instead. They still work: deep=True gives the level deep, and for_sorting=True gives sortable. A higher level wins if you give both. A call like copy(True) still means deep. copy(False) uses the class setting.

Parameters:

Name Type Description Default
level str | T_copybits | None

A name, a sum of the COPY_ constants, or None for the class setting copy_level_default.

None
name str

The name of the copy. It is empty if not given.

''
deep bool

Deprecated. Use level='deep'. True gives the level deep.

False
for_sorting bool

Deprecated. Use level='sortable'. True gives the level sortable.

False
inplace bool

If True, copy into this Daf and return it. The level must copy the rows.

False

Returns:

Type Description
'Daf'

The new Daf, or this one if inplace.

Raises:

Type Description
ValueError

level is a name that is not one of the four, or inplace with a level that does not copy the rows.

Examples:

>>> d = Daf(lol=[[2, 'b'], [1, 'a']], cols=['id', 'v'], name='orig')
>>> c = d.copy(name='work')
>>> c.name, d.name
('work', 'orig')
>>> _ = c.append([3, 'c'])
>>> d.num_rows()
2
>>> s = d.copy('shallow')
>>> _ = s.append([3, 'c'])
>>> d.num_rows()
3
>>> d = Daf(lol=[[2, 'b'], [1, 'a']], cols=['id', 'v'])
>>> e = d.copy('editable')
>>> e[0, 'v'] = 'X'
>>> d.iloc(0)
{'id': 2, 'v': 'b'}
>>> m = d.copy(Daf.COPY_OUTER | Daf.COPY_HD)
>>> m[0, 'v'] = 'Y'
>>> d.iloc(0)
{'id': 2, 'v': 'Y'}
>>> sel = d.select_where(lambda row: row['id'] == 1)
>>> sel.copy(inplace=True) is sel
True
>>> d[1, 'v'] = 'Z'
>>> sel.lol
[[1, 'a']]

clone_empty

clone_empty(lol=None, cols=None, name='')

Make a new Daf with the same layout and no rows.

This is copy() with the layout bits and then no rows. The new Daf is an object of the same class. It has the same column names, keyfield, dtypes, attrs, schema and display settings. It has its own hd, dtypes, attrs and key index. The name is not carried over. Give name to set it.

Give lol to fill the new Daf with rows. They are adopted, not copied.

If you give cols that differ from the columns of the original, the columns are new. The dtypes keep the entries for columns that are still there. The keyfield is cleared if any of its columns is gone. The schema and the disp_cols describe the old columns, so they are not carried over. If cols are the same as the columns of the original, everything is carried over.

Parameters:

Name Type Description Default
lol T_lola | None

Rows for the new Daf. If None, it has no rows.

None
cols T_ls | None

Column names to use instead of the existing ones.

None
name str

The name of the new Daf.

''

Returns:

Type Description
'Daf'

The new Daf.

Examples:

>>> d = Daf(lol=[[1, 'a']], cols=['id', 'v'], keyfield='id')
>>> c = d.clone_empty()
>>> c.columns(), c.keyfield, c.num_rows()
(['id', 'v'], 'id', 0)

set_lol

set_lol(new_lol)

Replace the rows with a new list of lists.

The list is adopted, not copied. The column names, the keyfield and the other settings stay. The key index is rebuilt when it is next needed.

Parameters:

Name Type Description Default
new_lol T_lola

The new rows.

required

Returns:

Type Description
'Daf'

This Daf, which has been changed.

Examples:

>>> d = Daf(lol=[[1, 'a']], cols=['id', 'v'], keyfield='id')
>>> d.set_lol([[5, 'q'], [6, 'r']]).keys()
[5, 6]

Iteration and comparison

retmode property writable

retmode

What a selection like daf[row, col] returns: a Daf, or plain values.

'obj' is the default. Every selection returns a new Daf, so you can keep calling Daf methods on the result.

'val' returns plain Python values when the selection is one cell, one row or one column. You get a single value, or a list. A selection of several rows and several columns is still a Daf.

A one-row result is the row itself, not a copy. Changing it changes this Daf.

In 'val' mode, one row or one cell is read straight from the rows, with no Daf built on the way. This is when the row is an int position or a str key, and the column, if given, is an int position or a str name. On a 10-row table it took about 0.6 µs by position and 3.3 µs by key, against 12 to 29 µs through a Daf.

Use 'val' when you read values for your own code. Use 'obj' when you want to keep working with Daf methods.

The mode belongs to the Daf you set it on. A Daf made by a selection starts again at 'obj'. A copy keeps the mode.

Set it with daf.retmode = 'val', or with the retmode argument when you create the Daf. Any other value raises ValueError.

Returns:

Type Description
str

'obj' or 'val'.

Examples:

>>> daf = Daf(cols=['x', 'y'], lol=[[1, 2], [3, 4]])
>>> type(daf[0, 'x']).__name__
'Daf'
>>> daf.retmode = 'val'
>>> daf[0, 'x']
1
>>> daf[1, :]
[3, 4]
>>> daf[:, 'y']
[2, 4]

itermode property writable

itermode

What each row is when you loop over a Daf: a dict, or a KeyedList.

Your code reads a row the same way in both modes, by column name, as in row['qty']. The difference is what the row is.

'dict' is the default. Each row is a new dict of column name to value. The values are copied into it, so changing it does not change the Daf.

'keyedlist' gives each row as a KeyedList. It points into the table instead of copying it. See there for how it works, what changes the Daf, and when it is faster.

The mode is used by for row in daf, and by methods that loop over the rows, such as reduce(). The methods iter_dict(), iter_klist() and iter_list() ignore it and always give their own kind of row. A Daf made by a selection starts again at 'dict'.

Set it with daf.itermode = 'keyedlist', or with the itermode argument when you create the Daf. Any other value raises ValueError.

Returns:

Type Description
str

'dict' or 'keyedlist'.

Examples:

>>> daf = Daf(cols=['x', 'y'], lol=[[1, 2], [3, 4]])
>>> for row in daf:
...     row['y'] = 0
>>> daf

| x  | y  |
| -: | -: |
|  1 |  2 |
|  3 |  4 |

%% daf rows=2; cols=2; keyfield=''; name=''

>>> daf.itermode = 'keyedlist'
>>> for row in daf:
...     row['y'] = 0
>>> daf

| x  | y  |
| -: | -: |
|  1 |  0 |
|  3 |  0 |

%% daf rows=2; cols=2; keyfield=''; name=''

__iter__

__iter__()

Loop over the rows, as in for row in daf.

Each row is a dict or a KeyedList, depending on itermode. For plain lists, use iter_list().

In the default itermode, each row is a new dict. Assigning a new value to it does not change the Daf. To change values in a loop, use iter_klist(), whose rows write through, or apply_in_place(). A list or dict held in a cell is the same object in the dict and the Daf, so changing it in place does show.

Each loop gets a new iterator, which goes through the rows once. It goes through the list of rows it started with: rows appended during the loop are seen, but a new list put in place, as by a sort, is not.

Returns:

Type Description
Iterator

The rows, as dicts or KeyedList objects.

iter_dict

iter_dict()

Loop over the rows as dicts, whatever the itermode is.

Each dict is a copy, so changing it does not change the Daf.

Returns:

Type Description
Iterator[Dict[str, Any]]

Each row as a dict of column name to value.

Examples:

>>> d = Daf(cols=['x', 'y'], lol=[[1, 2], [3, 4]])
>>> list(d.iter_dict())
[{'x': 1, 'y': 2}, {'x': 3, 'y': 4}]
>>> for row in d.iter_dict():
...     row['y'] = 0
>>> d

| x  | y  |
| -: | -: |
|  1 |  2 |
|  3 |  4 |

%% daf rows=2; cols=2; keyfield=''; name=''

iter_klist

iter_klist()

Loop over the rows as KeyedList objects, whatever the itermode is.

See KeyedList for how these rows work.

Returns:

Type Description
Iterator[KeyedList]

Each row as a KeyedList.

Examples:

>>> d = Daf(cols=['x', 'y'], lol=[[1, 2], [3, 4]])
>>> [row['x'] for row in d.iter_klist()]
[1, 3]
>>> for row in d.iter_klist():
...     row['y'] = 0
>>> d

| x  | y  |
| -: | -: |
|  1 |  0 |
|  3 |  0 |

%% daf rows=2; cols=2; keyfield=''; name=''

iter_list

iter_list()

Loop over the rows as the plain lists that the Daf stores.

Values are read by position, not by column name. These are the Daf's own row lists, not copies, so assigning to one changes the Daf.

Returns:

Type Description
Iterator[list]

Each row as a list.

Examples:

>>> d = Daf(cols=['x', 'y'], lol=[[1, 2], [3, 4]])
>>> [row[0] for row in d.iter_list()]
[1, 3]
>>> for row in d.iter_list():
...     row[1] = 9
>>> d

| x  | y  |
| -: | -: |
|  1 |  9 |
|  3 |  9 |

%% daf rows=2; cols=2; keyfield=''; name=''

__eq__

__eq__(other)

Compare two Daf instances.

Two Daf instances are equal when their rows, their column names and their keyfield are equal. The order of the columns matters. The name, dtypes, attrs and modes are not compared. Anything that is not a Daf is not equal.

Parameters:

Name Type Description Default
other object

The object to compare with.

required

Returns:

Type Description
bool

True if equal.

Examples:

>>> a = Daf(lol=[[1]], cols=['x'])
>>> a == Daf(lol=[[1]], cols=['x'])
True
>>> a == Daf(lol=[[1]], cols=['x'], keyfield='x')
False