Types and schema¶

Column dtypes and schema support.

Dtypes¶

set_dtypes ¶

set_dtypes(default_type=str, typ_to_cols_dict=None)

Set the dtype of every column, from a default and a few exceptions.

Use it when most columns have one type and a few differ. The result is stored in dtypes. This does not convert any data. Call apply_dtypes() for that.

If most columns differ, assign a dict to dtypes yourself.

Parameters:

Name	Type	Description	Default
`default_type`	`Type`	The type of every column that is not listed.	`str`
`typ_to_cols_dict`	`Dict[Type, T_ls] \| None`	Maps a type to the names of the columns of that type.	`None`

Returns:

Type	Description
`'Daf'`	This Daf, which has been changed.

Raises:

Type	Description
`NotImplementedError`	The Daf has no column names.

Examples:

>>> d = Daf(lol=[['1', '2', 'x']], cols=['a', 'b', 'c'])
>>> d.set_dtypes(str, {int: ['a', 'b']}).dtypes
{'a': <class 'int'>, 'b': <class 'int'>, 'c': <class 'str'>}

apply_dtypes ¶

apply_dtypes(
    *,
    dtypes=None,
    cols=None,
    unflatten=True,
    from_str=True,
    default_type=str,
    silent_error=False,
)

Convert the columns to their dtypes, in place.

A CSV file is read as text, because that is the fastest way to load it. Call this method to turn the columns you need into numbers, lists and so on. It changes the cells where they are and does not make a new table.

Three steps are separate in daffodil:

Step	What it does
set `dtypes`, or `set_dtypes()`	declares the types of the columns. Nothing is converted
`apply_dtypes()`	tries to convert the cells
`check_schema()`	finds the cells that do not fit their dtypes

Conversion is permissive. Text that is not a number stays as it is, and no error is raised. So a column with an int dtype can still hold some text after this call.

Conversion costs time, about 0.04 s for each column of 200,000 rows, so convert only the columns that you use. Pass cols to name them. A column that is not in cols is not converted, is not checked, and keeps its entry in the dtypes of the Daf. Without cols, every column of the Daf is converted, and each one needs a dtype.

The dtypes argument is a dict that maps a column name to a type, or one type for all columns. It may hold more columns than the Daf. Without cols, it replaces the dtypes of the Daf. With cols, it only supplies the types of those columns, and the other entries of the dtypes of the Daf stay as they are. If neither the argument nor the Daf has dtypes, nothing happens. With no columns defined, the names are taken from the dtypes.

Without cols, a dtypes that leaves out a column of the Daf raises ValueError, unless silent_error is True. Then the column gets default_type, which is str and is not converted, and the dtypes of the Daf are replaced by the dict with that column added. That replaces the types of the columns that you meant to convert later. To convert some columns and not others, use cols.

Types must be plain types such as int, float, bool, str, list, dict, tuple or set. Annotations such as List[str] do not work.

By default, columns of type str are skipped. The cells are assumed to be text already. Pass from_str=False when they may hold other values. Then each cell is converted with str(). Columns of type list or dict are read from their text. Pass unflatten=False to leave them as text. That text is read about eight times faster when pyjson5 is installed, as with pip install daffodil[pyjson5].

An empty cell, such as two commas next to each other in a CSV file, is NULL, which is the empty string. It stays NULL when its column is converted to int or float, so a number column can hold NULL. A bool column is the exception: an empty cell becomes 0 there.

A bool column gets 0 or 1. The text 0, false, no, n, f and off give 0, and 1, true, yes, y, t and on give 1, in lower case, capitalized or upper case. Other text stays as it is. So 'False' gives 0, where Python's own bool('False') is True. None gives 0, and any other value gives 0 or 1 by its truth. The sum methods, sum(), sum_np() and Daf.sum_da(), skip a NULL cell. Python's own sum() of such a column raises TypeError, so use the methods of the Daf.

A cell that cannot be converted to an int or a float keeps its text, so a bad value is still there to be found, as with list and dict. No error is raised for text. Call check_schema() to find such cells. A later step, such as a sum or a sort, may raise one. A cell that is not text and not a number, such as a list in an int column, cannot be given to int(), and raises TypeError. The dtypes do not limit what a cell can hold, so leave such a column out of the conversion. Whole number text of any size is converted exactly. Text with a decimal point or an exponent is converted to an int by way of a float, which cuts the decimal part.

This method does the common conversions and keeps them simple. For your own rules, convert the columns yourself and then say what the types are. Use apply_to_col() for one column, or apply_in_place() for several. Your function can raise, collect the bad values, or use any default. Then set dtypes. This does not convert anything. See the last example.

Parameters:

Name	Type	Description	Default
`dtypes`	`T_dtype_dict \| None`	Maps column names to types, or a single type for all columns.	`None`
`cols`	`str \| T_ls \| None`	The columns to convert, or one column name. Others are left alone.	`None`
`unflatten`	`bool`	If True, read list and dict columns from their text.	`True`
`from_str`	`bool`	If True, the cells are text, so `str` columns are not converted.	`True`
`default_type`	`Type`	The type to use for a column that has no dtype.	`str`
`silent_error`	`bool`	If False, raise an error when a column has no dtype.	`False`

Returns:

Type	Description
`'Daf'`	This Daf, which has been changed.

Raises:

Type	Description
`ValueError`	A column to convert has no dtype and `silent_error` is False.
`KeyError`	A name in `cols` is not a column of the Daf.
`TypeError`	A cell in an `int` or `float` column is not text and not a number.

Examples:

>>> d = Daf(lol=[['1', '2.5', 'x']], cols=['a', 'b', 'c'])
>>> d.apply_dtypes(dtypes={'a': int, 'b': float, 'c': str})

| a  |  b  | c  |
| -: | --: | -: |
|  1 | 2.5 |  x |

%% daf rows=1; cols=3; keyfield=''; name=''

An empty cell stays empty, and the sum methods skip it:

>>> d = Daf.from_csv_buff('a,b\n1,\n,3\n4,5\n')
>>> d.apply_dtypes(dtypes={'a': int, 'b': int})

| a  | b  |
| -: | -: |
|  1 |    |
|    |  3 |
|  4 |  5 |

%% daf rows=3; cols=2; keyfield=''; name=''

>>> d.sum()
{'a': 5, 'b': 8}

Convert only some columns with cols. The others stay text, and the dtypes of the Daf are not changed:

>>> d = Daf(lol=[['1', '2.5', '7']], cols=['a', 'b', 'n'],
...         dtypes={'a': int, 'b': float, 'n': int})
>>> _ = d.apply_dtypes(cols=['a'])
>>> d.iloc(0)
{'a': 1, 'b': '2.5', 'n': '7'}
>>> d.dtypes == {'a': int, 'b': float, 'n': int}
True
>>> _ = d.apply_dtypes(cols=['n'])
>>> d.iloc(0)
{'a': 1, 'b': '2.5', 'n': 7}

A bool column, and text that does not convert:

>>> b = Daf(cols=['ok', 'n'], lol=[['', '1'], ['0', 'x'], ['True', '3'], ['maybe', '']])
>>> _ = b.apply_dtypes(dtypes={'ok': bool, 'n': int})
>>> b.lol
[[0, 1], [0, 'x'], [1, 3], ['maybe', '']]

Your own conversion, here one that records the values that fail:

>>> bad = []
>>> def to_int(value):
...     try:
...         return int(value)
...     except ValueError:
...         bad.append(value)
...         return ''
>>> d = Daf(lol=[['1'], ['x'], ['3']], cols=['n'])
>>> _ = d.apply_to_col('n', to_int)
>>> d.dtypes = {'n': int}
>>> d

| n  |
| -: |
|  1 |
|    |
|  3 |

%% daf rows=3; cols=1; keyfield=''; name=''

>>> bad
['x']

flatten ¶

flatten(convert_bool_to_int=True, use_pyon=True)

Turn list and dict cells into text, in place.

You rarely need this. to_csv_buff() and to_csv_file() write every cell as text already, so a separate pass is not needed.

Only the columns whose dtype is list or dict are changed. Each cell becomes its str() text. A column of dtype bool becomes 0 and 1. Without dtypes nothing happens.

Parameters:

Name	Type	Description	Default
`convert_bool_to_int`	`bool`	If True, write bool columns as 0 and 1.	`True`
`use_pyon`	`bool`	Kept for compatibility. Only True is supported.	`True`

Returns:

Type	Description
`'Daf'`	This Daf, which has been changed.

Raises:

Type	Description
`ValueError`	`use_pyon` is False.

Examples:

>>> d = Daf(lol=[[[1, 2], True]], cols=['a', 'b'], dtypes={'a': list, 'b': bool})
>>> d.flatten()

|   a    | b  |
| -----: | -: |
| [1, 2] |  1 |

%% daf rows=1; cols=2; keyfield=''; name=''

strip ¶

strip(chrs=' ')

Remove characters from both ends of every text cell, in place.

Each character in chrs is removed on its own, so '()"' removes any mix of parentheses and quotes. Cells that are not text, and empty cells, are skipped.

Parameters:

Name	Type	Description	Default
`chrs`	`str`	The characters to remove.	`' '`

Returns:

Type	Description
`'Daf'`	This Daf, which has been changed.

Examples:

>>> Daf(lol=[[' a ', 3, '("x")']], cols=['p', 'q', 'r']).strip(' ()"')

| p  | q  | r  |
| -: | -: | -: |
|  a |  3 |  x |

%% daf rows=1; cols=3; keyfield=''; name=''

Schema¶

apply_schema ¶

_apply_schema(self, schema=None)

Attach a schema to this Daf and fill in what the Daf is missing.

A schema describes columns: their names, types and default values. Use one when many tables share a layout, or to make new records with the defaults. The constructor calls this method, so Daf(schema=...) applies the schema.

Two kinds of schema are accepted.

A class decorated with @schemaclass. The annotated attributes give the column names and types. Their values are the defaults. A __keyfield__ attribute gives the keyfield.

A schema Daf, with one row for each column of the table. It must have a Name column. It may have a dtype column, which holds one of str, int, float, bool, list or dict. A Default column gives the defaults. Its attrs may hold a keyfield.

Only gaps are filled. The column names, the dtypes and the keyfield are taken from the schema only if the Daf has none. The rows are never changed and nothing is validated. Call check_schema() to check the cells.

The other columns of a schema Daf are for building input forms. They are not used by daffodil. The Type column says what kind of form control to use.

checkbox   One or more checkboxes. Value lists the labels.
               checkbox+buttons adds Set and Clear buttons.
               checkbox+values allows values that differ from the labels.
date       A text box with a calendar button.
label      Read only text.
radio      Like checkbox, but only one can be chosen.
select     A dropdown or list box. Value lists the options.
               select+multi allows several choices.
               select+values allows values that differ from the labels.
text       A one line text box. Value is the initial text.
textarea   A multi line text box. Size is columns x rows, such as 80x6.

Parameters:

Name	Type	Description	Default
`schema`	`type \| Daf \| None`	The schema to apply. If None, the schema already attached is used.	`None`

Returns:

Type	Description
`Daf`	This Daf, which has been changed.

Raises:

Type	Description
`TypeError`	The schema is neither a `@schemaclass` nor a schema Daf. Nothing is stored.
`RuntimeError`	A `dtype` in a schema Daf is not one of the supported names.

Examples:

>>> from daffodil.daf import Daf
>>> from daffodil.lib.schemaclass import schemaclass
>>> @schemaclass
... class Person:
...     name: str = ''
...     age: int = 0
>>> d = Daf(schema=Person)
>>> d.columns()
['name', 'age']

On a Daf with rows and no names, it fills the names and dtypes, and converts nothing:

>>> raw = Daf(lol=[['ann', '7']])
>>> _ = raw.apply_schema(Person)
>>> raw.columns(), raw.lol
(['name', 'age'], [['ann', '7']])
>>> d.dtypes
{'name': <class 'str'>, 'age': <class 'int'>}

attach_schema ¶

_attach_schema(self, schema)

Attach a @schemaclass to this Daf.

This is the strict form of apply_schema(), for a schema class only. It stores the schema, then fills in the column names, the dtypes and the keyfield, each only if the Daf has none. The rows are never changed and nothing is validated.

Parameters:

Name	Type	Description	Default
`schema`	`type`	A class decorated with `@schemaclass`.	required

Returns:

Type	Description
`Daf`	This Daf, which has been changed.

Raises:

Type	Description
`TypeError`	The class is not a `@schemaclass`.

Examples:

>>> from daffodil.daf import Daf
>>> from daffodil.lib.schemaclass import schemaclass
>>> @schemaclass
... class Person:
...     name: str = ''
...     age: int = 0
>>> Daf().attach_schema(Person).columns()
['name', 'age']

check_schema ¶

_check_schema(self, max_report=10)

Check that every cell fits the dtypes and the required columns. Raise if one does not.

Loading does not check the data, so that it stays fast. apply_dtypes() leaves a cell that does not convert as text, and an empty cell is NULL. Call this method when you want to find such cells, for example after a load while developing, or in a test. It is never called by daffodil itself.

The dtypes are those of the Daf, which apply_schema() takes from the schema. A cell must be of its dtype, or NULL. NULL is allowed in a column of any dtype, unless the column is required, and it is false when tested. A float column may also hold an int. A bool column holds 0 or 1. apply_dtypes() keeps an empty cell as NULL, except in a bool column, where it becomes 0. A column of a dtype not listed here is not checked.

The required columns come from the Required column of an attached schema Daf. A value such as yes, true or 1 marks the column as required. A required column must exist, and none of its cells may be NULL. A @schemaclass has no required columns.

Parameters:

Name	Type	Description	Default
`max_report`	`int`	How many bad cells the error lists. The error also gives the total.	`10`

Returns:

Type	Description
`Daf`	This Daf, unchanged, when every cell fits.

Raises:

Type	Description
`ValueError`	A cell does not fit its dtype, a required cell is NULL, or a required column is missing.

Examples:

>>> from daffodil.daf import Daf
>>> schema = Daf.from_md('''
... | Name | dtype | Required |
... |:-----|:------|:---------|
... | id   | str   | yes      |
... | n    | int   |          |
... ''')
>>> d = Daf.from_csv_buff('id,n\na,1\n,x\n')
>>> d = d.apply_schema(schema).apply_dtypes(dtypes=d.dtypes)
>>> d.check_schema()
Traceback (most recent call last):
...
ValueError: check_schema(): 2 cells do not fit the schema:
  row 1, column 'id': required, but empty
  row 1, column 'n': 'x' is not int

default_record ¶

_default_record(self, astype=dict)

Make a new record for this Daf: its columns, with the defaults of the attached schema.

Use it to start a record that you fill in, then append. Each call returns a new record, so changing it does not change the next one. Nothing is converted or validated.

The defaults are worked out the first time, and kept until the columns or the schema of the Daf are replaced. A default that is changed on the schema class after that is not seen.

The record follows the columns of the Daf, in their order, however they were set. A column gets the schema's default if the schema has one, and NULL otherwise. A field of the schema that is not a column is left out. So a Daf that keeps only some of the schema's columns gets records that fit it. A Daf with no schema gets NULL in every column. A Daf with no columns takes its columns from the schema.

With astype=KeyedList the record is a KeyedList that shares the hd of this Daf, so record.hd is daf.hd. Fill it by name, in any order: assigning to a key writes to that column's position. append() then knows the record fits, and skips the check of its columns.

For a schema Daf, the Name column gives the fields and the Default column gives the values. A missing Default column gives empty strings.

Parameters:

Name	Type	Description	Default
`astype`	`type`	`dict`, the default, or `KeyedList`.	`dict`

Returns:

Type	Description
`Any`	A dict or a KeyedList of each column and its default.

Raises:

Type	Description
`AttributeError`	The Daf has no columns and no schema.
`RuntimeError`	A schema Daf has no `Name` column.
`TypeError`	The attached schema is of an unsupported kind, or `astype` is not dict or KeyedList.

Examples:

>>> from daffodil.daf import Daf
>>> from daffodil.keyedlist import KeyedList
>>> from daffodil.lib.schemaclass import schemaclass
>>> @schemaclass
... class Person:
...     name: str = ''
...     age: int = 0
>>> Daf(schema=Person).default_record()
{'name': '', 'age': 0}
>>> Daf(cols=['age', 'city'], schema=Person).default_record()
{'age': 0, 'city': ''}
>>> d = Daf(schema=Person)
>>> row = d.default_record(astype=KeyedList)
>>> row['age'] = 41
>>> row.hd is d.hd, row.to_dict()
(True, {'name': '', 'age': 41})