Import and export

Creating a Daf from other data and files, and converting it back out.

Python structures

from_lod classmethod

from_lod(
    records_lod,
    *,
    keyfield="",
    dtypes=None,
    name="",
    cols=None,
    ignore_extra_keys=False,
    fast=False,
)

Make a Daf from a list of dicts, one dict for each row.

Without cols or dtypes, the columns are all the keys of all the dicts, in the order in which they first appear. The keys of the first dict come first. A dict that lacks a key gets NULL there. A key that first appears in a later dict adds a column at the right, and the rows before it get NULL there. You do not need to look through the data first. Empty dicts and items that are not dicts are skipped.

If cols is given, those are the columns. Otherwise the keys of dtypes are the columns. You chose the columns, so a dict with a key that is not one of them raises ValueError, which names the keys, because that value would be lost. Pass ignore_extra_keys=True to leave such keys out on purpose, as when you pick a few columns of wide records.

With fast=True the first dict is checked fully, and every dict for its number of keys. Use it when you built every dict yourself, with the same keys in the same order, as in a loop. Each row is the values of a dict, in the dict's own order. The columns are cols, or the keys of dtypes, or the keys of the first dict. With cols or dtypes, the first dict must have the columns as its keys, in order, or it raises ValueError. A later dict with the wrong number of keys raises ValueError. A later dict with its keys in another order is added anyway, and its values land in the wrong columns.

Parameters:

Name	Type	Description	Default
records_lod	T_loda	The rows, as dicts.	required
keyfield	str | int | T_ta | T_la	Column, or tuple or list of columns, whose values identify rows.	''
dtypes	T_dtype_dict | None	Type for each column. When given, it also selects the columns.	None
name	str	Name of the new Daf.	''
cols	T_ls | None	Column names to use, in order.	None
ignore_extra_keys	bool	With cols or dtypes, leave out keys that are not columns, and do not raise.	False
fast	bool	If True, take each dict's values as a row. Check the first dict fully, and every dict's length. See above.	False

Returns:

Type	Description
'Daf'	The new Daf.

Raises:

Type	Description
ValueError	cols or dtypes is given, a dict has a key that is not one of those columns, and ignore_extra_keys is not True.

Examples:

>>> Daf.from_lod([{'a': 1, 'b': 2}, {'a': 3}])

| a  | b  |
| -: | -: |
|  1 |  2 |
|  3 |    |

%% daf rows=2; cols=2; keyfield=''; name=''

>>> Daf.from_lod([{'a': 1}, {'a': 2, 'b': 5}, {'c': 9}])

| a  | b  | c  |
| -: | -: | -: |
|  1 |    |    |
|  2 |  5 |    |
|    |    |  9 |

%% daf rows=3; cols=3; keyfield=''; name=''

>>> Daf.from_lod([{'a': 1, 'b': 2}], cols=['b', 'a'])

| b  | a  |
| -: | -: |
|  2 |  1 |

%% daf rows=1; cols=2; keyfield=''; name=''

>>> Daf.from_lod([{'a': 1, 'b': 2, 'c': 3}], cols=['a', 'b'], ignore_extra_keys=True)

| a  | b  |
| -: | -: |
|  1 |  2 |

%% daf rows=1; cols=2; keyfield=''; name=''

to_lod

to_lod()

Make a list of dicts, one dict for each row.

The dicts are new, but the values in them are the same objects as in the Daf. An empty Daf gives an empty list. See iter_dict() to go through rows one at a time without building the whole list.

Returns:

Type	Description
T_loda	The rows as dicts.

Examples:

>>> d = Daf(lol=[[1, 'a'], [2, ['x']]], cols=['id', 'v'])
>>> lod = d.to_lod()
>>> lod
[{'id': 1, 'v': 'a'}, {'id': 2, 'v': ['x']}]
>>> lod[1]['v'].append('y')
>>> d.iloc(1)
{'id': 2, 'v': ['x', 'y']}

from_lot classmethod

from_lot(
    records_lot,
    cols=None,
    dtypes=None,
    keyfield="",
    name="",
)

Make a Daf from a list of tuples, one tuple for each row.

Without cols the Daf has no column names, as with Daf(lol=...). Call set_cols() to give it spreadsheet names such as A and B, or to name them. A Daf with no names cannot give its rows as dicts or KeyedList objects.

Parameters:

Name	Type	Description	Default
records_lot	T_lota	The rows, as tuples.	required
cols	T_ls | None	Column names. They must be as many as the items in each tuple.	None
dtypes	T_dtype_dict | None	Type for each column.	None
keyfield	str | int | T_ta | T_la	Column, or tuple or list of columns, whose values identify rows.	''
name	str	Name of the new Daf.	''

Returns:

Type	Description
'Daf'	The new Daf.

Raises:

Type	Description
ValueError	A tuple does not have as many items as there are columns.

Examples:

>>> Daf.from_lot([(1, 'a'), (2, 'b')], cols=['id', 'v'])

| id | v  |
| -: | -: |
|  1 |  a |
|  2 |  b |

%% daf rows=2; cols=2; keyfield=''; name=''

>>> Daf.from_lot([(1, 'a')]).columns()
[]
>>> Daf.from_lot([(1, 'a')]).set_cols().columns()
['A', 'B']

from_dod classmethod

from_dod(
    dod,
    keyfield="rowkey",
    dtypes=None,
    ignore_extra_keys=False,
)

Make a Daf from a dict of dicts, where the outer key names the row.

A dict of dicts usually does not repeat the row key inside each row. A Daf table always has it as a column. If the inner dicts lack the keyfield column, it is added from the outer keys. The new Daf has that keyfield.

If dtypes is given, its keys are the columns, and it must name the keyfield column too. An inner dict with a key that dtypes does not name raises ValueError, which names the keys, because that value would be lost. Pass ignore_extra_keys=True to leave such keys out on purpose.

Use to_dod() to go back.

Parameters:

Name	Type	Description	Default
dod	T_doda	A dict that maps a row key to a dict of that row.	required
keyfield	str	The column that holds the outer key.	'rowkey'
dtypes	T_dtype_dict | None	Type for each column. When given, it also selects the columns.	None
ignore_extra_keys	bool	With dtypes, leave out keys that it does not name, and do not raise.	False

Returns:

Type	Description
'Daf'	The new Daf.

Raises:

Type	Description
ValueError	dtypes is given, an inner dict has a key that it does not name, and ignore_extra_keys is not True.
KeyError	dtypes does not name the keyfield column.

Examples:

>>> d = Daf.from_dod({'r0': {'x': 1}, 'r1': {'x': 2}})
>>> d

| rowkey | x  |
| -----: | -: |
|     r0 |  1 |
|     r1 |  2 |

%% daf rows=2; cols=2; keyfield='rowkey'; name=''

>>> dod = {'r0': {'x': 1, 'id': 7}, 'r1': {'x': 2, 'id': 8}}
>>> Daf.from_dod(dod, dtypes={'rowkey': str, 'x': int}, ignore_extra_keys=True)

| rowkey | x  |
| -----: | -: |
|     r0 |  1 |
|     r1 |  2 |

%% daf rows=2; cols=2; keyfield='rowkey'; name=''

to_dod

to_dod(remove_keyfield=True)

Make a dict of dicts, where the keyfield value names each row.

The keyfield column is left out of the inner dicts by default. Pass remove_keyfield=False to keep it there too. The Daf must have a keyfield. Without one, a KeysDisabledError is raised, unless the Daf has no rows.

Parameters:

Name	Type	Description	Default
remove_keyfield	bool	If True, do not repeat the key inside each inner dict.	True

Returns:

Type	Description
T_doda	A dict that maps each key to a dict of that row.

Raises:

Type	Description
KeysDisabledError	The Daf has no keyfield.

Examples:

>>> d = Daf(lol=[[1, 'a']], cols=['id', 'v'], keyfield='id')
>>> d.to_dod()
{1: {'v': 'a'}}
>>> d.to_dod(remove_keyfield=False)
{1: {'id': 1, 'v': 'a'}}

from_cols_dol classmethod

from_cols_dol(cols_dol, keyfield='', dtypes=None)

Make a Daf from a dict of lists, where each list is a column.

All the lists must be as long as each other. If one differs in length from the first, a ValueError is raised, because the rows could not be made without losing values or guessing them.

Parameters:

Name	Type	Description	Default
cols_dol	T_dola	Maps a column name to the list of its values.	required
keyfield	str	Column, or tuple or list of columns, whose values identify rows.	''
dtypes	T_dtype_dict | None	Type for each column.	None

Returns:

Type	Description
'Daf'	The new Daf.

Raises:

Type	Description
ValueError	A list has a different length from the first.

Examples:

>>> d = Daf.from_cols_dol({'A': [1, 2, 3], 'B': [4, 5, 6]})
>>> d

| A  | B  |
| -: | -: |
|  1 |  4 |
|  2 |  5 |
|  3 |  6 |

%% daf rows=3; cols=2; keyfield=''; name=''

to_cols_dol

to_cols_dol()

Make a dict of lists, one list of values for each column.

Returns:

Type	Description
dict	A dict that maps each column name to the list of its values.

Examples:

>>> Daf(lol=[[1, 'a'], [2, 'b']], cols=['id', 'v']).to_cols_dol()
{'id': [1, 2], 'v': ['a', 'b']}

from_lod_to_cols classmethod

from_lod_to_cols(lod, cols=None, keyfield='', dtypes=None)

Make a Daf in which each dict becomes a column, not a row.

The keys of the dicts become the first column, and each dict adds one more column after it. Use this to compare several results side by side, such as the same features measured in several tries. It is a transpose of from_lod().

Without cols, the first column is named key and the others A, B and so on. The names in cols include the first column. The keyfield is not set unless keyfield is given.

Parameters:

Name	Type	Description	Default
lod	T_loda	The dicts. They should all have the same keys.	required
cols	List | None	Column names, starting with the name of the column of keys.	None
keyfield	str	Column whose values identify rows.	''
dtypes	T_dtype_dict | None	Type for each column of the dicts, before the change. It must name every key of the dicts.	None

Returns:

Type	Description
'Daf'	The new Daf.

Raises:

Type	Description
ValueError	dtypes is given and a dict has a key that it does not name. The keys would be matched to the wrong values. The message mentions ignore_extra_keys, which this method does not take. Name all the keys.

Examples:

>>> d = Daf.from_lod_to_cols([{'A': 1, 'B': 2}, {'A': 4, 'B': 5}], cols=['Feature', 'T1', 'T2'])
>>> d

| Feature | T1 | T2 |
| ------: | -: | -: |
|       A |  1 |  4 |
|       B |  2 |  5 |

%% daf rows=2; cols=3; keyfield=''; name=''

to_attrib_dict

to_attrib_dict()

Make a dict with the column names and the rows.

This is deprecated. It keeps the rows as they are, not copied. Use to_lod() or to_cols_dol() instead.

Returns:

Type	Description
dict	A dict with the keys cols and lol.

Examples:

>>> Daf(lol=[[1, 'a']], cols=['id', 'v']).to_attrib_dict()
{'cols': ['id', 'v'], 'lol': [[1, 'a']]}

Values and lists

to_value

to_value(default=_MISSING, astype=None)

Get the one value of a Daf that has one row and one column.

Use it on the result of a selection, as in my_daf[1, 'n'].to_value().

Parameters:

Name	Type	Description	Default
default	Any	Returned if the Daf is not one cell. Without it, an error is raised.	_MISSING
astype	Callable | str | None	A type or function to convert the value with.	None

Returns:

Type	Description
Any	The value.

Raises:

Type	Description
ValueError	The Daf is not one cell and no default is given.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d[1, 'n'].to_value()
20

to_list

to_list(
    unique=False,
    flatten=False,
    omit_nulls=False,
    default=_MISSING,
    astype=None,
)

Get the values of a Daf that has one row or one column, as a list.

Use it on the result of a selection, as in my_daf[:, 'v'].to_list(). A column is read from the Daf, so use col() if you only need the list. An empty Daf gives an empty list. A Daf with more than one row and more than one column is not accepted.

Parameters:

Name	Type	Description	Default
unique	bool	If True, leave out repeated values and keep the order.	False
flatten	bool	If True, join items that are lists into one list.	False
omit_nulls	bool	If True, leave out the empty values.	False
default	Any	Replaces NULL, None and NaN values. This may itself be None.	_MISSING
astype	Callable | str | type | None	A type or function to convert each value with.	None

Returns:

Type	Description
list	The list.

Raises:

Type	Description
ValueError	The Daf has more than one row and more than one column.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d[:, 'v'].to_list()
['a', 'b', 'c']
>>> d[1].to_list()
[2, 'b', 20]

to_lota

to_lota()

Make a list of tuples, one tuple for each row.

This is handy for making composite keys. The tuples are new, but the values in them are the same objects as in the Daf.

Returns:

Type	Description
T_lota	The rows as tuples.

Examples:

>>> Daf(lol=[[1, 'a'], [2, 'b']], cols=['id', 'v']).to_lota()
[(1, 'a'), (2, 'b')]

to_dict

to_dict()

Get the one row of a Daf as a dict.

Use it on the result of a selection, as in my_daf[1].to_dict(). A column is not turned into a dict. Use to_list() for that. An empty Daf gives an empty dict. A Daf with no column names raises KeysDisabledError.

Returns:

Type	Description
T_da	The row, as a dict that maps column names to values.

Raises:

Type	Description
KeysDisabledError	The Daf has a row and no column names.
ValueError	The Daf has more than one row.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d[1].to_dict()
{'id': 2, 'v': 'b', 'n': 20}

to_klist

to_klist(irow=0)

Get one row as a KeyedList.

The KeyedList shares the row and the column names with the Daf, so it costs little. A negative position counts from the end. A position that is out of range raises IndexError. A Daf with no rows gives an empty KeyedList.

Parameters:

Name	Type	Description	Default
irow	int	The row position.	0

Returns:

Type	Description
KeyedList	The row.

Raises:

Type	Description
IndexError	The position is out of range, for a Daf that has rows.

Examples:

>>> d = Daf(lol=[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]], cols=['id', 'v', 'n'], keyfield='id')
>>> d.to_klist(1)['v']
'b'

CSV, Excel and OpenDocument

from_csv classmethod

from_csv(source, **kwargs)

Read a CSV file, a web address or an S3 object into a Daf.

The source may be a path, a Path, an http or https address, or an s3://bucket/key name. The requests and boto3 packages are imported only when they are needed.

Every value is read as text. Call apply_dtypes() on the result to convert them. Giving dtypes here also converts them, but that is deprecated. The other keyword arguments, such as keyfield, are those of from_csv_buff(). The length of each row is not checked. See from_csv_buff().

Parameters:

Name	Type	Description	Default
source	str | Path	A file path, an http address or an s3:// name.	required
**kwargs	Any	Passed on to from_csv_buff().	{}

Returns:

Type	Description
'Daf'	The new Daf.

Raises:

Type	Description
RuntimeError	The download fails, a needed package is missing, or the local file cannot be read or parsed. The message says which.

Examples:

>>> import os, tempfile
>>> path = os.path.join(tempfile.mkdtemp(), 'x.csv')
>>> with open(path, 'w') as f:
...     n = f.write('id,v\n1,a\n')
>>> Daf.from_csv(path)

| id | v  |
| -: | -: |
|  1 |  a |

%% daf rows=1; cols=2; keyfield=''; name=''

from_csv_buff classmethod

from_csv_buff(
    csv_buff,
    keyfield="",
    dtypes=None,
    noheader=False,
    user_format=False,
    sep=",",
    unflatten=True,
    include_cols=None,
    name="",
)

Make a Daf from CSV text, bytes or an iterator of lines.

The first row holds the column names, unless noheader is True. Quoted fields may hold commas. Empty rows at the end are dropped. Cells are text, unless dtypes is given. Then the cells are converted, and list and dict columns are read from their text unless unflatten is False.

Converting here is deprecated. It does no more than calling apply_dtypes() on the result, which is clearer: a reader only reads, and the types are applied where they are needed. Pass no dtypes, and call apply_dtypes() afterwards. The conversion still works, for now.

Empty text, or a source with no rows, gives an empty Daf with no columns, as from_md() does. Check len() of the result if an empty source would be an error.

A keyfield that is not a column of the file raises KeyError.

The length of each row is not checked, so that a read costs no more than it must. A row with a missing cell, or with an extra one from an unquoted comma, is kept as it is, and a later call may fail with an error that does not mention it. If you do not trust the source, call is_rectangular(). It looks at every row. To pad short rows, call force_rectangular(). It raises ValueError for a row that is too long.

Parameters:

Name	Type	Description	Default
csv_buff	bytes | str | Iterator[str]	The CSV, as text, bytes or an iterator of lines.	required
keyfield	str	Column, or tuple or list of columns, whose values identify rows.	''
dtypes	T_dtype_dict | None	Type for each column. Deprecated: call apply_dtypes() on the result instead.	None
noheader	bool	If True, the first row is data, and the Daf has no column names.	False
user_format	bool	If True, skip comment lines and blank lines.	False
sep	str	The character that separates fields.	','
unflatten	bool	If True, read list and dict columns from their text.	True
include_cols	T_ls | None	Keep only these columns, in this order. The cells of the other columns are never kept, so a wide file costs little memory.	None
name	str	Name of the new Daf.	''

Returns:

Type	Description
'Daf'	The new Daf.

Raises:

Type	Description
KeyError	A name in include_cols, or the keyfield, is not in the header of the file.
ValueError	include_cols is given with noheader=True, because the names of the columns come from the header.

Examples:

>>> d = Daf.from_csv_buff('id,v\n1,a\n2,"b,c"\n')
>>> d

| id |  v  |
| -: | --: |
|  1 |   a |
|  2 | b,c |

%% daf rows=2; cols=2; keyfield=''; name=''

>>> Daf.from_csv_buff('id,v\n1,a\n').apply_dtypes(dtypes={'id': int, 'v': str})

| id | v  |
| -: | -: |
|  1 |  a |

%% daf rows=1; cols=2; keyfield=''; name=''

>>> d = Daf.from_csv_buff('a,b,c\n1,2,3\n4,5,6\n', include_cols=['c', 'a'])
>>> d

| c  | a  |
| -: | -: |
|  3 |  1 |
|  6 |  4 |

%% daf rows=2; cols=2; keyfield=''; name=''

>>> d = Daf.from_csv_buff('id,v\n1,a\n2\n')
>>> d.is_rectangular()
False
>>> d.force_rectangular()

| id | v  |
| -: | -: |
|  1 |  a |
|  2 |    |

%% daf rows=2; cols=2; keyfield=''; name=''

from_csv_file classmethod

from_csv_file(
    filepath,
    keyfield="",
    dtypes=None,
    noheader=False,
    user_format=False,
    sep=",",
    unflatten=True,
    include_cols=None,
    name="",
)

Read a CSV file into a Daf. Deprecated, use from_csv().

This now calls from_csv(), so it reads the file as UTF-8, and a file that cannot be read raises RuntimeError. It used to print a message and return None, and to read the file with the encoding of the machine.

Parameters:

Name	Type	Description	Default
filepath	str	Path of the file.	required
keyfield	str	Column, or tuple or list of columns, whose values identify rows.	''
dtypes	T_dtype_dict | None	Type for each column. Deprecated: call apply_dtypes() on the result instead.	None
noheader	bool	If True, the first row is data, and the Daf has no column names.	False
user_format	bool	If True, skip comment lines and blank lines.	False
sep	str	The character that separates fields.	','
unflatten	bool	If True, read list and dict columns from their text.	True
include_cols	T_ls | None	Keep only these columns, in this order. The cells of the other columns are never kept, so a wide file costs little memory.	None
name	str	Name of the new Daf.	''

Returns:

Type	Description
'Daf'	The new Daf.

Raises:

Type	Description
RuntimeError	The file cannot be read or parsed.

Examples:

>>> import os, tempfile
>>> path = os.path.join(tempfile.mkdtemp(), 'x.csv')
>>> with open(path, 'w') as f:
...     n = f.write('id,v\n1,a\n')
>>> Daf.from_csv_file(path)

| id | v  |
| -: | -: |
|  1 |  a |

%% daf rows=1; cols=2; keyfield=''; name=''

to_csv_file

to_csv_file(
    file_path="", line_terminator=None, include_header=True
)

Write the Daf to a CSV file.

The file has a header row of column names unless include_header is False. Each cell is written as text, so lists and dicts are written in their str() form. A NULL cell is written as nothing. The default line ending is \r\n.

Parameters:

Name	Type	Description	Default
file_path	str | Path	Where to write the file.	''
line_terminator	str | None	The line ending. If None, \r\n is used.	None
include_header	bool	If True, write the column names first.	True

Returns:

Type	Description
str	The path that was written.

Examples:

>>> import os, tempfile
>>> path = os.path.join(tempfile.mkdtemp(), 'x.csv')
>>> d = Daf(lol=[[1, 'a']], cols=['id', 'v'])
>>> d.to_csv_file(path) == path
True

to_csv_buff

to_csv_buff(line_terminator=None, include_header=True)

Make CSV text from the Daf.

The text can be saved to a file or uploaded. There is no need to call flatten() first. Each cell is written with str(), so a list or dict is written as its Python text, with single quotes. The bool True is written as True. A NULL cell is written as nothing. The text is not JSON.

A column converted by apply_dtypes() to bool holds 0 and 1, so it is written as 0 and 1. A column of Python bools is written as True and False. Both read back as 0 and 1 when the column has dtype bool. The cells are written as they are held, with no conversion, since any pass over them would slow every save.

A dict whose keys are not text, such as {1: 'a'}, is written with its keys as they are.

Parameters:

Name	Type	Description	Default
line_terminator	str | None	The line ending. If None, \r\n is used.	None
include_header	bool	If True, write the column names first.	True

Returns:

Type	Description
T_buff	The CSV text.

Examples:

>>> d = Daf(lol=[[1, 'a']], cols=['id', 'v'])
>>> d.to_csv_buff(line_terminator='\n')
'id,v\n1,a\n'
>>> Daf(lol=[[{1: 'a'}]], cols=['x']).to_csv_buff(line_terminator='\n')
"x\n{1: 'a'}\n"

buff_to_file staticmethod

buff_to_file(buff, file_path, fmt='.csv')

Write text or bytes to a file.

This is a static method, so call it as Daf.buff_to_file(buff, path). It is the helper that to_csv_file() uses.

Parameters:

Name	Type	Description	Default
buff	T_buff	The text or bytes to write.	required
file_path	str | Path	Where to write.	required
fmt	str	The file format, such as .csv.	'.csv'

Returns:

Type	Description
str	The path that was written.

Examples:

>>> import os, tempfile
>>> path = os.path.join(tempfile.mkdtemp(), 'x.csv')
>>> Daf.buff_to_file('id,v\n1,a\n', path) == path
True
>>> open(path).read()
'id,v\n1,a\n'

from_excel_buff classmethod

from_excel_buff(
    excel_buff,
    keyfield="",
    dtypes=None,
    noheader=False,
    user_format=False,
    unflatten=True,
)

Make a Daf from the bytes of an xlsx file.

The first sheet is turned into CSV by the xlsx2csv package. The CSV is then read like any CSV text. All values start as text, so give dtypes or call apply_dtypes() to convert them.

Short rows are not padded here. xlsx2csv pads them. See xlsx_to_csv().

Parameters:

Name	Type	Description	Default
excel_buff	bytes	The bytes of the xlsx file.	required
keyfield	str	Column whose values identify rows.	''
dtypes	T_dtype_dict | None	Type for each column. Deprecated: call apply_dtypes() on the result instead.	None
noheader	bool	If True, the first row is data, not column names.	False
user_format	bool	If True, skip comment lines and blank lines.	False
unflatten	bool	If True, read list and dict columns from their text.	True

Returns:

Type	Description
'Daf'	The new Daf.

Examples:

>>> import io, xlsxwriter
>>> buff = io.BytesIO()
>>> workbook = xlsxwriter.Workbook(buff, {'in_memory': True})
>>> sheet = workbook.add_worksheet()
>>> for irow, row in enumerate([['id', 'v'], [1, 'a'], [2, 'b']]):
...     for icol, value in enumerate(row):
...         _ = sheet.write(irow, icol, value)
>>> workbook.close()
>>> d = Daf.from_excel_buff(buff.getvalue(), keyfield='id')
>>> d

| id | v  |
| -: | -: |
|  1 |  a |
|  2 |  b |

%% daf rows=2; cols=2; keyfield='id'; name=''

>>> Daf.from_excel_buff(buff.getvalue(), dtypes={'id': int, 'v': str})

| id | v  |
| -: | -: |
|  1 |  a |
|  2 |  b |

%% daf rows=2; cols=2; keyfield=''; name=''

from_ods classmethod

from_ods(
    source,
    sheet=0,
    keyfield="",
    dtypes=None,
    noheader=False,
    unflatten=True,
    name="",
)

Make a Daf from one sheet of an OpenDocument spreadsheet, an .ods file.

Use it for a table kept in LibreOffice Calc. Only the first sheet is read, unless you give sheet. Only the content of the cells is read: styles, comments, links and formulas are left out. No package is needed, since the file is read with the standard library. See lib/daf_ods.py.

All values start as text, as from a CSV file, so give dtypes or call apply_dtypes() to convert them. A number, date or boolean cell gives its stored value, such as 168.5 or 2026-09-01, not the text shown, such as $168.50. A cell with several lines gives them joined with a newline.

Rows are padded with NULL to the width of the widest row. A blank row inside the table is kept as a row of NULL, so the rows line up with the spreadsheet. Drop such rows yourself if they must not be used. The blank rows at the end are left out.

Parameters:

Name	Type	Description	Default
source	str | Path | bytes	The path of the file, or its bytes.	required
sheet	int | str	The position of the sheet, from 0, or its name.	0
keyfield	str	Column whose values identify rows.	''
dtypes	T_dtype_dict | None	Type for each column. Deprecated: call apply_dtypes() on the result instead.	None
noheader	bool	If True, the first row is data, not column names.	False
unflatten	bool	If True, read list and dict columns from their text.	True
name	str	The name of the new Daf.	''

Returns:

Type	Description
'Daf'	The new Daf.

Raises:

Type	Description
IndexError	There is no sheet at that position.
KeyError	There is no sheet with that name.

JSON

to_json

to_json(concise=True)

Make JSON text that holds the whole Daf.

The text holds the rows, the column names, the dtypes, the keyfield, the name, the attrs and the display columns. With concise=True, the parts that are empty are left out. Use from_json() to read it back.

Types are written by name. Only int, float, str, bool, list and dict are read back as types. Other names, such as date, come back as text.

Cells must be JSON values. A tuple comes back as a list, and a set raises a TypeError. A NaN is written as NaN, which some JSON readers reject.

Parameters:

Name	Type	Description	Default
concise	bool	If True, leave out the parts that are empty.	True

Returns:

Type	Description
str	The JSON text.

Examples:

>>> Daf(lol=[[1]], cols=['a']).to_json()
'{"lol": [[1]], "hd": {"a": 0}}'

from_json classmethod

from_json(json_str)

Make a Daf from JSON text made by to_json().

Parameters:

Name	Type	Description	Default
json_str	str	The JSON text.	required

Returns:

Type	Description
'Daf'	The new Daf.

Examples:

>>> d = Daf(lol=[[1, 'a']], cols=['id', 'v'], keyfield='id')
>>> Daf.from_json(d.to_json()) == d
True

Markdown

from_md classmethod

_from_md(cls, md_str)

Make a Daf from a Markdown table. This reads what to_md() writes.

Only the first table in the text is read. Text before it, such as headings and prose, is skipped. A table row starts and ends with |. The table must have a header row and then a separator row such as | -: | -: |. The columns are the header cells. A table with no header row is not read, and raises RuntimeError. The table ends at the first line that is not a row, such as a blank line. A row with more or fewer cells than the header raises RuntimeError, which names the line, so that no row is lost without notice.

A Daf with rows and no column names is written by to_md() with the spreadsheet names A, B and so on in the header. Reading that text gives a Daf with those names, so the round trip adds column names. The text of a table cannot tell these names from names that you chose.

A footer line that starts with %% daf and follows the table may give the keyfield and name, as to_md(include_summary=True) writes. Every cell is read as text. Convert them with apply_dtypes(). Text with no table, or empty text, gives an empty Daf.

Markdown is a report format, so a round trip through to_md() is not exact:

• Text that to_md() shortened, rows and columns it left out, and numbers that smart_fmt rounded stay that way.
• A newline in a cell is written as <br>, and is read back as <br>.

A | in a cell is written as \|, and is read back as |. A composite keyfield in the footer is read back as the tuple or list it was.

Parameters:

Name	Type	Description	Default
md_str	str	The Markdown text.	required

Returns:

Type	Description
'Daf'	The new Daf.

Raises:

Type	Description
RuntimeError	A table is found but it has no header row and separator row, or a row has a different number of cells than the header.

Examples:

>>> from daffodil.daf import Daf
>>> text = "| id | v |\n| -: | -: |\n|  1 | a |\n|  2 | b |\n\n%% daf rows=2; cols=2; keyfield='id'; name='nm'\n"
>>> d = Daf.from_md(text)
>>> d

| id | v  |
| -: | -: |
|  1 |  a |
|  2 |  b |

%% daf rows=2; cols=2; keyfield='id'; name='nm'

>>> Daf.from_md(Daf(lol=[[1, 'a']]).to_md()).columns()
['A', 'B']

dodaf_to_md staticmethod

dodaf_to_md(dodaf, report_header='', **to_md_kwargs)

Make one Markdown report from a dict of Daf instances.

This is a static method. The report has an optional first level heading, then a second level section for each key, in the order of the dict. Each section holds that Daf as a table, from its own to_md(). A Daf with no rows still gets its section, with a line *(no rows)*, so you can see which sections were empty. Use dodaf_from_md() to read the report back.

Parameters:

Name	Type	Description	Default
dodaf	Dict[str, Any]	A dict that maps a section name to a Daf.	required
report_header	str	The title of the report. Left out if empty.	''
**to_md_kwargs	Any	Passed on to the to_md() of each Daf, such as max_rows.	{}

Returns:

Type	Description
str	The Markdown text.

Examples:

>>> from daffodil.daf import Daf
>>> print(Daf.dodaf_to_md({'first': Daf(lol=[[1]], cols=['a'])}, report_header='Report'))
# Report


## first


| a  |
| -: |
|  1 |

dodaf_from_md classmethod

_dodaf_from_md(cls, md_str, header_level=2)

Read a report made by dodaf_to_md() back into a dict of Daf instances.

A section starts at a line that begins with the marker of header_level, which is ## by default. Text before the first section, such as the report title, is skipped. Each section becomes a Daf, read with from_md(), so the cells are text. The section title is the dict key and the name of the Daf, unless the table's footer gives a name. The footer wins for the name, never for the key. Empty text gives an empty dict.

Parameters:

Name	Type	Description	Default
md_str	str	The Markdown text.	required
header_level	int	The number of # that start a section heading.	2

Returns:

Type	Description
Dict[str, Any]	A dict that maps each section title to a Daf.

Examples:

>>> from daffodil.daf import Daf
>>> report = Daf.dodaf_to_md({'first': Daf(lol=[[1]], cols=['a'])})
>>> back = Daf.dodaf_from_md(report)
>>> back['first']

| a  |
| -: |
|  1 |

%% daf rows=1; cols=1; keyfield=''; name='first'

Pandas and NumPy

from_pandas_df classmethod

_from_pandas_df(
    cls,
    df,
    keyfield="",
    name="",
    use_csv=False,
    dtypes=None,
)

Make a Daf from a Pandas DataFrame or Series.

The values become plain Python values. The column names come from the DataFrame. The index is not kept. The dtypes are worked out from the Pandas dtypes, but the values are not converted to them. A Series becomes one row, with the index labels as column names. Its dtypes are the Python types of its values, one for each label.

With use_csv=True the DataFrame is turned into CSV text first, and that is read back. This can be faster for some frames.

Parameters:

Name	Type	Description	Default
df	T_df	The DataFrame or Series.	required
keyfield	str	Column, or tuple or list of columns, whose values identify rows.	''
name	str	Name of the new Daf.	''
use_csv	bool	If True, convert by way of CSV text.	False
dtypes	T_dtype_dict | None	Deprecated, and it is not used. The dtypes come from df. It will be removed. A DeprecationWarning is given if it is passed.	None

Returns:

Type	Description
Daf	The new Daf.

Examples:

>>> import pandas as pd
>>> from daffodil.daf import Daf
>>> df = pd.DataFrame({'id': [1, 2], 'v': ['a', 'b']})
>>> d = Daf.from_pandas_df(df, keyfield='id')
>>> d

| id | v  |
| -: | -: |
|  1 |  a |
|  2 |  b |

%% daf rows=2; cols=2; keyfield='id'; name=''

to_pandas_df

_to_pandas_df(
    self,
    cols=None,
    *,
    use_csv=False,
    use_donpa=False,
    default=_MISSING,
    defaulting_cols=None,
)

Make a Pandas DataFrame from the Daf.

Daffodil stores a missing value as an empty string. Pandas treats that as text, so a column with empty cells is of type object, even if the rest are numbers. Text that looks like a number stays text. Convert columns afterwards if you want other types.

Giving default replaces empty and None cells in the Daf itself, before the conversion. That changes the Daf you called it on. Copy it first if you need it unchanged. default cannot be used with use_csv. With use_donpa, the default replaces the cells in the arrays only. The Daf is not changed, and it goes to every included column, as defaulting_cols is not used on that route.

Parameters:

Name	Type	Description	Default
cols	T_ls | None	Names, or positions, of the columns to include. If None, all columns.	None
use_csv	bool	If True, convert by way of CSV text.	False
use_donpa	bool	If True, convert by way of to_donpa(). This suits numeric columns.	False
default	Any	Value that replaces empty and None cells. This changes the Daf, except with use_donpa.	_MISSING
defaulting_cols	T_ls | None	The columns that default applies to. If None, all included columns.	None

Returns:

Type	Description
Any	The DataFrame.

Raises:

Type	Description
NotImplementedError	default is given with use_csv=True.

Examples:

>>> from daffodil.daf import Daf
>>> d = Daf(lol=[[1, 'a'], [2, 'b']], cols=['id', 'v'])
>>> d.to_pandas_df().values.tolist()
[[1, 'a'], [2, 'b']]

from_numpy classmethod

from_numpy(npa, keyfield='', cols=None, name='')

Make a Daf from a NumPy array.

The values become plain Python values. A two dimensional array gives one row for each row of the array. A one dimensional array gives a single row.

An array has one dtype. np.array([[1, 'x']]) makes every value text, before this method sees it, so the Daf gets text too. An array with dtype=object keeps each value as the Python object it was, such as an int, a str or None.

Parameters:

Name	Type	Description	Default
npa	Any	The NumPy array.	required
keyfield	str	Column, or tuple or list of columns, whose values identify rows.	''
cols	T_la | None	Column names.	None
name	str	Name of the new Daf.	''

Returns:

Type	Description
'Daf'	The new Daf.

Examples:

>>> import numpy as np
>>> Daf.from_numpy(np.array([[1, 2], [3, 4]]), cols=['a', 'b'])

| a  | b  |
| -: | -: |
|  1 |  2 |
|  3 |  4 |

%% daf rows=2; cols=2; keyfield=''; name=''

to_numpy

to_numpy()

Make a NumPy array of the rows.

Column names, the keyfield and any dtypes are not kept. NumPy picks one type for the whole array. If the Daf mixes numbers and text, every value becomes text.

Returns:

Type	Description
T_npa	The NumPy array.

Examples:

>>> Daf(lol=[[1, 2.5]], cols=['a', 'b']).to_numpy().tolist()
[[1.0, 2.5]]

to_donpa

to_donpa(colnames=None, default=_MISSING)

Make a dict of NumPy arrays, one array for each column.

This is a light form of a DataFrame. The arrays can be used in vector operations, such as donpa['D_pct'] = donpa['D_votes'] / donpa['RV_total']. Each column gets its own type, so numbers and text can be mixed.

Parameters:

Name	Type	Description	Default
colnames	T_ls | None	The columns to include. If None, all columns.	None
default	Any	A value that replaces each NULL, None and NaN cell, in the arrays only. The Daf is not changed. It applies to every column in colnames, so give only the numeric columns, or a text column gets it too.	_MISSING

Returns:

Type	Description
T_donpa	A dict that maps each column name to an array.

Notes

A column that mixes numbers and NULL becomes a text array, because NumPy has one type for an array. Give a numeric default to keep it numeric. A column with a default is also faster to make than one with blanks and no default, because the array is built from numbers.

Raises:

Type	Description
ColumnNotFoundError	A name in colnames is not a column. This is a KeyError, and also a RuntimeError.

Examples:

>>> Daf(lol=[[1, 'a'], [2, 'b']], cols=['id', 'v']).to_donpa(['id'])['id'].tolist()
[1, 2]
>>> d = Daf(lol=[[1, 5], [2, ''], [None, 7]], cols=['n', 'm'])
>>> d.to_donpa(['m'])['m'].tolist()
['5', '', '7']
>>> d.to_donpa(['n', 'm'], default=0)['m'].tolist()
[5, 0, 7]
>>> d

|  n   | m  |
| ---: | -: |
|    1 |  5 |
|    2 |    |
| None |  7 |

%% daf rows=3; cols=2; keyfield=''; name=''

Other sources

from_directory classmethod

from_directory(
    source,
    schema=None,
    recursive=True,
    file_pat=None,
    include_dirs=False,
)

Make a Daf that lists the files in a folder.

Each row describes one file. The columns are the path, the folder, the name, the name without its extension, the extension, the size in bytes, and the modified and changed times. On Linux and macOS the ctime is the time of the last change to the file's metadata, such as its permissions, not the time it was created. Paths use / on every system.

With include_dirs=True, the folders are listed too, each before the files in the same folder. A folder has an is_dir of 1, a size of 0 and no extension. Without it, no folder is listed and is_dir is always 0.

The rows are in the order in which the file system lists the files. That order is not sorted, and it differs between machines. Sort the result if the order matters.

The schema chooses the columns of the result. A @schemaclass that lists only some of the fields above keeps only those. Other columns of the schema get their defaults. The new Daf has no keyfield. Files that cannot be read are skipped.

Only local folders are supported.

Parameters:

Name	Type	Description	Default
source	str | Path	The folder to list.	required
schema	type | None	A @schemaclass that chooses the columns. If None, the built in one is used.	None
recursive	bool	If True, list files in all sub folders. Otherwise only the folder itself.	True
file_pat	str | None	A regular expression. Only names that match it are listed. Case is ignored.	None
include_dirs	bool	If True, list the folders as well as the files.	False

Returns:

Type	Description
'Daf'	The new Daf.

Examples:

>>> import os, tempfile
>>> folder = tempfile.mkdtemp()
>>> os.mkdir(os.path.join(folder, 'sub'))
>>> for name in ('b.txt', 'a.csv', os.path.join('sub', 'c.csv')):
...     with open(os.path.join(folder, name), 'w') as f:
...         n = f.write('hello')
>>> d = Daf.from_directory(folder)
>>> sorted(d.col('basename')), d.col('size')
(['a.csv', 'b.txt', 'c.csv'], [5, 5, 5])
>>> sorted(Daf.from_directory(folder, recursive=False).col('basename'))
['a.csv', 'b.txt']
>>> sorted(Daf.from_directory(folder, file_pat=r'\.csv$').col('basename'))
['a.csv', 'c.csv']
>>> sorted(Daf.from_directory(folder, include_dirs=True).col('basename'))
['a.csv', 'b.txt', 'c.csv', 'sub']

from_pdf class-attribute instance-attribute

from_pdf = from_pdf

from_pdf() reads one kind of county report, of election participation by precinct, from a PDF file, with pdfplumber. It is experimental and written for that report. Its interface may change, so it is not documented here.

from_googlesheet classmethod

from_googlesheet(
    spreadsheet_id,
    sheetname="Sheet1",
    *,
    service_account_file,
)

Read a Google Sheet into a Daf. This is not implemented.

It always raises NotImplementedError. It holds the place of the interface, which takes the path of a service account file.

Parameters:

Name	Type	Description	Default
spreadsheet_id	str	The ID of the Google Sheet.	required
sheetname	str	The name of the sheet.	'Sheet1'
service_account_file	str	The path of the Google service account file with the credentials.	required

Raises:

Type	Description
NotImplementedError	Always.

Examples:

>>> Daf.from_googlesheet('some-id', service_account_file='key.json')
Traceback (most recent call last):
    ...
NotImplementedError: from_googlesheet() is not implemented yet.

to_googlesheet

to_googlesheet(
    spreadsheet_id,
    sheetname="Sheet1",
    *,
    service_account_file,
)

Write the Daf to a Google Sheet. This is not implemented.

It always raises NotImplementedError. It holds the place of the interface, which takes the path of a service account file.

Parameters:

Name	Type	Description	Default
spreadsheet_id	str	The ID of the Google Sheet.	required
sheetname	str	The name of the sheet.	'Sheet1'
service_account_file	str	The path of the Google service account file with the credentials.	required

Raises:

Type	Description
NotImplementedError	Always.

Examples:

>>> Daf(cols=['x'], lol=[[1]]).to_googlesheet('some-id', service_account_file='key.json')
Traceback (most recent call last):
    ...
NotImplementedError: to_googlesheet() is not implemented yet.