Statistics and reporting

Value counts, statistics, and Markdown output.

Value counts

valuecounts_for_colname

valuecounts_for_colname(
    colname, sort=False, reverse=True, omit_nulls=False
)

Count how often each value occurs in one column.

With sort=True the dict is ordered from the most common value to the least. Use reverse=False for the other way. A column that does not exist gives an empty dict. An empty cell is counted as the empty string, unless omit_nulls is True.

Parameters:

Name	Type	Description	Default
colname	str	The column to count.	required
sort	bool	If True, order by count.	False
reverse	bool	With sort, True puts the most common first.	True
omit_nulls	bool	If True, leave out the count of empty cells.	False

Returns:

Type	Description
T_di	A dict that maps each value to its count.

Examples:

>>> d = Daf(lol=[['a'], ['b'], ['a'], ['']], cols=['g'])
>>> d.valuecounts_for_colname('g', sort=True, omit_nulls=True)
{'a': 2, 'b': 1}

valuecounts_for_colnames_ls

valuecounts_for_colnames_ls(
    colnames_ls=None,
    sort=False,
    reverse=True,
    omit_nulls=False,
)

Count how often each value occurs, in each of several columns.

Parameters:

Name	Type	Description	Default
colnames_ls	T_cs | None	The columns to count. All if None.	None
sort	bool	If True, order each count by size.	False
reverse	bool	With sort, True puts the most common first.	True
omit_nulls	bool	If True, leave out the count of empty cells.	False

Returns:

Type	Description
T_dodi	A dict that maps each column to a dict of value and count.

Examples:

>>> d = Daf(lol=[['a', 'x'], ['b', 'x']], cols=['g', 'h'])
>>> d.valuecounts_for_colnames_ls()
{'g': {'a': 1, 'b': 1}, 'h': {'x': 2}}

valuecounts_for_colname_selectedby_colname

valuecounts_for_colname_selectedby_colname(
    colname,
    selectedby_colname,
    selectedby_colvalue,
    sort=False,
    reverse=True,
)

Count the values of a column, in the rows where another column has a value.

Parameters:

Name	Type	Description	Default
colname	str	The column to count.	required
selectedby_colname	str	The column to test.	required
selectedby_colvalue	str	Only rows where that column equals this are counted.	required
sort	bool	If True, order the counts by size.	False
reverse	bool	With sort, True puts the most common first.	True

Returns:

Type	Description
T_di	A dict that maps each value to its count. It is empty if a column does not exist.

Examples:

>>> d = Daf(lol=[['a', 'x'], ['b', 'x'], ['a', 'y']], cols=['g', 'h'])
>>> d.valuecounts_for_colname_selectedby_colname('g', 'h', 'x')
{'a': 1, 'b': 1}

valuecounts_for_colnames_ls_selectedby_colname

valuecounts_for_colnames_ls_selectedby_colname(
    colnames_ls=None,
    selectedby_colname="",
    selectedby_colvalue="",
    sort=False,
    reverse=True,
)

Count the values of several columns, in the rows where another column has a value.

Parameters:

Name	Type	Description	Default
colnames_ls	T_cs | None	The columns to count. All if None.	None
selectedby_colname	str	The column to test.	''
selectedby_colvalue	str	Only rows where that column equals this are counted.	''
sort	bool	If True, order the counts by size.	False
reverse	bool	With sort, True puts the most common first.	True

Returns:

Type	Description
T_dodi	A dict that maps each column to a dict of value and count.

Examples:

>>> d = Daf(lol=[['a', 'x', 1], ['a', 'y', 2], ['b', 'x', 3]], cols=['g', 'k', 'n'])
>>> d.valuecounts_for_colnames_ls_selectedby_colname(['k'], 'g', 'a')
{'k': {'x': 1, 'y': 1}}
>>> d.valuecounts_for_colnames_ls_selectedby_colname(['g', 'k'], 'g', 'a')
{'g': {'a': 2}, 'k': {'x': 1, 'y': 1}}

valuecounts_for_colname1_groupedby_colname2

valuecounts_for_colname1_groupedby_colname2(
    colname1, groupedby_colname2, sort=False, reverse=True
)

Count the values of one column, for each value of another.

The data is read once. Use it to see whether two columns relate one to one: each group should then hold a single value.

Parameters:

Name	Type	Description	Default
colname1	str	The column whose values are counted.	required
groupedby_colname2	str	The column whose values form the groups.	required
sort	bool	If True, order each count by size.	False
reverse	bool	With sort, True puts the most common first.	True

Returns:

Type	Description
T_dodi	A dict that maps each value of the second column to a dict of value and count.

Examples:

>>> d = Daf(lol=[['a', 'x'], ['b', 'x'], ['a', 'y']], cols=['g', 'h'])
>>> d.valuecounts_for_colname1_groupedby_colname2('g', 'h')
{'x': {'a': 1, 'b': 1}, 'y': {'a': 1}}

value_counts_daf

value_counts_daf(
    colname,
    sort=False,
    reverse=True,
    include_total=False,
    omit_nulls=False,
)

Make a Daf that lists each value of a column and its count.

The columns are the name of the column, and counts. There is no keyfield. With include_total a last row holds the total.

Parameters:

Name	Type	Description	Default
colname	str	The column to count.	required
sort	bool	If True, order by count.	False
reverse	bool	With sort, True puts the most common first.	True
include_total	bool	If True, add a row with the total.	False
omit_nulls	bool	If True, leave out the count of empty cells.	False

Returns:

Type	Description
'Daf'	The new Daf.

Examples:

>>> d = Daf(lol=[['a'], ['b'], ['a']], cols=['g'])
>>> d.value_counts_daf('g', sort=True)

| g  | counts |
| -: | -----: |
|  a |      2 |
|  b |      1 |

%% daf rows=2; cols=2; keyfield=''; name=''

daf_valuecount

daf_valuecount(by='row', cols=None)

Count how often each value occurs, in each column, using reduce() and count_values_kl().

Parameters:

Name	Type	Description	Default
by	str	How the function is applied. See reduce().	'row'
cols	T_cs | None	The columns to count. All if None.	None

Returns:

Type	Description
T_ma	A dict that maps each column to a dict of value and count. A column that is
T_ma	not in cols is empty.

Examples:

>>> d = Daf(lol=[['a', 1, 10], ['b', 2, 20], ['a', 3, 30]], cols=['g', 'x', 'y'])
>>> d.daf_valuecount(cols=['g'])['g']
{'a': 2, 'b': 1}

Statistics

gen_stats_daf

gen_stats_daf(col_def_lot)

Work out statistics for columns, given a profile for each.

col_def_lot has a tuple for each column of interest. The tuple is the column name, a type, a format, and a profile. The profile is one of index, attrib, file_paths, scalar or localidx, and chooses what is measured. An index looks for repeats, an attribute counts the values, and a scalar gives the minimum, maximum, mean and standard deviation. The type and format are not used.

Parameters:

Name	Type	Description	Default
col_def_lot	T_lota	A list of tuples of column name, type, format and profile.	required

Returns:

Type	Description
T_doda	A dict that maps each column name to its statistics, as a dict.

Raises:

Type	Description
NotImplementedError	A profile is not one of the five.

Examples:

>>> d = Daf(lol=[[1], [3]], cols=['n'])
>>> d.gen_stats_daf([('n', int, '', 'scalar')])['n']['mean']
2

Markdown and text

to_md

to_md(
    max_rows=0,
    max_cols=0,
    just="",
    shorten_text=True,
    max_text_len=80,
    smart_fmt=False,
    include_summary=False,
    disp_cols=None,
    header=None,
)

Make a Markdown table of the Daf.

Without limits the whole table is written. With max_rows or max_cols, the first and last are kept, and the middle is replaced by .... Text longer than max_text_len is shortened by cutting out its middle. just has one character for each column: < left, ^ center, > right. The default is right. With no column names, A, B and so on are used. With include_summary, the Markdown can be read back with from_md(), as text.

Use max_rows and max_cols together, or neither. With only max_cols, a row of ... is added under the header by mistake.

Parameters:

Name	Type	Description	Default
max_rows	int	The most rows to show. 0 for all.	0
max_cols	int	The most columns to show. 0 for all.	0
just	str	The justification of each column.	''
shorten_text	bool	If True, shorten text that is longer than max_text_len.	True
max_text_len	int	The longest text to show in full.	80
smart_fmt	bool	If True, show numbers with fewer decimal places.	False
include_summary	bool	If True, add a line with the size, keyfield and name.	False
disp_cols	T_cs | None	Column names to show instead of the real ones.	None
header	T_cs | None	A header to use instead.	None

Returns:

Type	Description
str	The Markdown text.

Examples:

>>> d = Daf(lol=[[1, 'a'], [2, 'b']], cols=['id', 'v'])
>>> print(d.to_md())
| id | v  |
| -: | -: |
|  1 |  a |
|  2 |  b |

to_md_cols

to_md_cols(
    max_rows=0,
    max_cols=0,
    just="",
    shorten_text=True,
    max_text_len=80,
    smart_fmt=False,
    include_summary=False,
    disp_cols=None,
)

Make a Markdown table in which each row of the Daf is a column.

There is no header row and no separator row. The first column holds the column names of the Daf. Use it for a Daf with few rows and many columns. Some Markdown renderers, such as Python-Markdown, show this text as plain text, because they need a header and a separator row to make a table.

Parameters:

Name	Type	Description	Default
max_rows	int	The most rows to show. 0 for all.	0
max_cols	int	The most columns to show. 0 for all.	0
just	str	The justification of each column.	''
shorten_text	bool	If True, shorten text that is longer than max_text_len.	True
max_text_len	int	The longest text to show in full.	80
smart_fmt	bool	If True, show numbers with fewer decimal places.	False
include_summary	bool	Not used.	False
disp_cols	T_ls | None	Column names to show instead of the real ones.	None

Returns:

Type	Description
str	The Markdown text.

Examples:

>>> d = Daf(lol=[[1, 'a'], [2, 'b']], cols=['x', 'y'])
>>> print(d.to_md_cols(), end='')
| x  | 1  | 2  |
| y  | a  | b  |

md_daf_table_snippet

md_daf_table_snippet()

Make a short Markdown table of the Daf, with a summary line.

This is what str() shows. It keeps at most md_max_rows rows and md_max_cols columns, 10 by default. Longer text is shortened to 80 characters.

If the Daf has no rows, it is empty, and it shows no header. The summary line says rows=0; cols=0, because the number of columns is counted from the rows. columns() still gives the names.

Returns:

Type	Description
str	The Markdown text.

Examples:

>>> d = Daf(lol=[[1, 'a'], [2, 'b']], cols=['x', 'y'])
>>> print(d.md_daf_table_snippet(), end='')
| x  | y  |
| -: | -: |
|  1 |  a |
|  2 |  b |

%% daf rows=2; cols=2; keyfield=''; name=''

An empty Daf has no rows and no columns, so only the summary line shows. This is so even if it has column names:

>>> Daf(cols=['x', 'y'])


%% daf rows=0; cols=0; keyfield=''; name=''

daf_to_lol_summary

daf_to_lol_summary(
    max_rows=10, max_cols=10, disp_cols=None
)

Make a list of lists for display, with the column names first.

If there are more rows or columns than the limits, the first and last are kept and the middle is replaced by .... The limit is the number of data rows kept, and an odd limit keeps the extra row at the start. A limit of 0 means no limit, so with only max_cols all the rows are kept. The rows are not copied.

Parameters:

Name	Type	Description	Default
max_rows	int	The most rows to keep. 0 for no limit.	10
max_cols	int	The most columns to keep. 0 for no limit.	10
disp_cols	T_cs | None	Column names to use instead of the real ones.	None

Returns:

Type	Description
T_lola	The rows, with a header row first if there are column names.

Examples:

>>> d = Daf(lol=[[1, 'a'], [2, 'b']], cols=['x', 'y'])
>>> d.daf_to_lol_summary()
[['x', 'y'], [1, 'a'], [2, 'b']]
>>> big = Daf(lol=[[i, i] for i in range(20)], cols=['x', 'y'])
>>> big.daf_to_lol_summary(max_rows=4)
[['x', 'y'], [0, 0], [1, 1], ['...', '...'], [18, 18], [19, 19]]

dict_to_md staticmethod

dict_to_md(da, cols=None, just='<<')

Show a dict as a two column Markdown table, for looking at it.

This is a static method. Use it as print(Daf.dict_to_md(my_da)). The keys are in the first column and the values in the second.

Parameters:

Name	Type	Description	Default
da	T_da	The dict.	required
cols	T_ls | None	The two column names. Default key and value.	None
just	str	The justification of the two columns.	'<<'

Returns:

Type	Description
str	The Markdown text.

Examples:

>>> print(Daf.dict_to_md({'a': 1, 'b': 'two'}))
| key | value |
| :-- | :---- |
| a   | 1     |
| b   | two   |

__str__

__str__()

Show the Daf as a Markdown table.

The table shows at most md_max_rows rows and md_max_cols columns, 10 by default. Larger tables show the first five and the last five, with ... between them. A summary line with the size and keyfield follows the table. Use md_daf_table_snippet() for control over the output.

Returns:

Type	Description
str	The Markdown text.

__repr__

__repr__()

Show the Daf as a Markdown table, for the Python prompt.

This is the same text as str(), with a newline in front, so the table starts at the left margin when it is echoed at the prompt.

Returns:

Type	Description
str	The Markdown text.

__format__

__format__(format_spec)

Format a Daf that holds one cell.

With no format spec this is the same as str(). With a spec, the Daf must have exactly one cell. A number is formatted with the spec. Anything else is converted with str().

Parameters:

Name	Type	Description	Default
format_spec	str	A format spec such as .2f.	required

Returns:

Type	Description
str	The formatted text.

Raises:

Type	Description
ValueError	A spec is given and the Daf does not have exactly one cell.

Examples:

>>> format(Daf(lol=[[3.14159]], cols=['a']), '.2f')
'3.14'