simpletable.dictdataframe package

simpletable.dictdataframe package#

Submodules#

simpletable.dictdataframe.convert module#

Module for converting dictdataframes to other formats.

convert_dict_to_structured_ndarray(data, keys=None)[source]#
Parameters:

  • data (dictionary like object) – data structure which provides iteritems and itervalues

  • keys (sequence, optional) – ordered subset of columns to export

Returns:

tab – structured numpy array

Return type:

structured ndarray

pickle_dump(self, fname)[source]#

create a pickle dump of the dataset

to_astropy_table(self, **kwargs)[source]#

A class to represent tables of heterogeneous data.

astropy.table.Table provides a class for heterogeneous tabular data, making use of a numpy structured array internally to store the data values. A key enhancement provided by the Table class is the ability to easily modify the structure of the table by adding or removing columns, or adding new rows of data. In addition table and column metadata are fully supported.

Parameters:

  • masked (bool, optional) – Specify whether the table is masked.

  • names (list, optional) – Specify column names

  • dtype (list, optional) – Specify column data types

  • meta (dict, optional) – Metadata associated with the table.

  • copy (bool, optional) – Copy the input data (default=True).

  • rows (numpy ndarray, list of lists, optional) – Row-oriented data for table instead of data argument

  • copy_indices (bool, optional) – Copy any indices in the input data (default=True)

  • **kwargs (dict, optional) – Additional keyword args when converting table-like object

Returns:

df – dataframe

Return type:

astropy.table.Table

to_dict(self, keys=None, contiguous=False)[source]#

Construct a dictionary from this dataframe with contiguous arrays

Parameters:

  • keys (sequence, optional) – ordered subset of columns to export

  • contiguous (boolean) – make sure each value is a contiguous numpy array object (C-aligned)

Returns:

data – converted data

Return type:

dict

to_pandas(self, **kwargs)[source]#

Construct a pandas dataframe

Parameters:

  • data (ndarray) – (structured dtype), list of tuples, dict, or DataFrame

  • keys (sequence, optional) – ordered subset of columns to export

  • index (string, list of fields, array-like) – Field of array to use as the index, alternately a specific set of input labels to use

  • exclude (sequence, default None) – Columns or fields to exclude

  • columns (sequence, default None) – Column names to use. If the passed data do not have names associated with them, this argument provides names for the columns. Otherwise this argument indicates the order of the columns in the result (any names not found in the data will become all-NA columns)

  • coerce_float (boolean, default False) – Attempt to convert values to non-string, non-numeric objects (like decimal.Decimal) to floating point, useful for SQL result sets

Returns:

df

Return type:

DataFrame

unpickle(cls, fname)[source]#

restore a previously pickled object

simpletable.dictdataframe.dictdataframe module#

DictDataFrame, a simplistic column based dataframe

The DataFrame container allows easier manipulations of the data but is basically deriving dictionary objects. In particular it allows easy conversions to many common dataframe containers: numpy.recarray, pandas.DataFrame, dask.DataFrame, astropy.Table, xarray.Dataset, vaex.DataSetArrays.

author:

Morgan Fouesneau

class DictDataFrame[source]#

Bases: dict

A simple-ish dictionary like structure allowing usage as array on non constant multi-dimensional column data.

It initializes like a normal dictionary and can be used as such. A few divergence points though: some default methods such as len() may refer to the lines and not the columns as a normal dictionary would.

This data object implements also array slicing, shape, dtypes and some data functions (sortby, groupby, where, select, etc)

property Plotter#

Plotter instance related to this dataset. Requires plotter add-on to work

__init__(*args, **kwargs)[source]#

A dictionary constructor and attributes declaration

aggregate(func, keys, args=(), kwargs={})[source]#

Apply func on groups within the data

Parameters:

  • func (callable) – function to apply

  • keys (sequence(str)) – sequence of keys defining the groups

  • args (tuple) – optional arguments to func (will be given at the end)

  • kwargs (dict) – optional keywords to func

Returns:

seq – flattened sequence of keys and value (key1, key2, … keyn, {})

Return type:

sequence

arg_groupby(key)[source]#

create an iterator which returns (key, index) grouped by each value of key(value)

Parameters:

key (str | Hashable)

Return type:

Iterator[tuple[str | Hashable, list[int]]]

property columns: ItemsView#

Iterator on the columns refers to dict.items()

property dtype: dict[str | Hashable, dtype]#

the dtypes of each column of the dataset

evalexpr(expr, exprvars=None, dtype=<class 'float'>)[source]#
evaluate expression based on the data and external variables

all np function can be used (log, exp, pi…)

Parameters:

  • data (dict or dict-like structure) – data frame / dict-like structure containing named columns

  • expr (str) – expression to evaluate on the table includes mathematical operations and attribute names

  • exprvars (dictionary, optional) – A dictionary that replaces the local operands in current frame.

  • dtype (dtype definition) – dtype of the output array

Returns:

out – array of the result

Return type:

np.array

evaluate(expr, exprvars=None, dtype=<class 'float'>)#
evaluate expression based on the data and external variables

all np function can be used (log, exp, pi…)

Parameters:

  • data (dict or dict-like structure) – data frame / dict-like structure containing named columns

  • expr (str) – expression to evaluate on the table includes mathematical operations and attribute names

  • exprvars (dictionary, optional) – A dictionary that replaces the local operands in current frame.

  • dtype (dtype definition) – dtype of the output array

Returns:

out – array of the result

Return type:

np.array

classmethod from_lines(iterable)[source]#

Create a DataFrame object from its lines instead of columns

Parameters:

iterable (iterable) – sequence of lines with the same keys (expecting dict like structure)

Returns:

df – a new object

Return type:

DataFrame

groupby(key)[source]#

create an iterator which returns (key, DataFrame) grouped by each value of key(value)

Parameters:

key (str | Hashable)

Return type:

Iterator[tuple[str | Hashable, DictDataFrame]]

iterlines()[source]#

Iterator on the lines of the dataframe

Return type:

Iterator[Any]

join(key, other, key_other=None, columns_other=None, prefix=None)[source]#

Experimental joining of structures, (equivalent to SQL left outer join) Example: >>> x = np.arange(10) >>> y = x**2 >>> z = x**3 >>> df = DictDataFrame(x=x, y=y) >>> df2 = DictDataFrame(x=x[:4], z=z[:4]) >>> df.join(‘x’, df2, ‘x’, column_other=[‘z’])

Parameters:

  • key (str) – key for the left table (self)

  • other (DictDataFrame) – Other dataset to join with (the right side of the outer join)

  • key_other (str, optional) – key on which to join (default identical to key)

  • columns_other (tuple, optional) – column names to add to the dataframe (default: all fields)

  • prefix (str, optional) – add a prefix to the new column

Returns:

self – itself

Return type:

DictDataFrame

property lines: Iterator[Any]#

Iterator on the lines of the dataframe

multigroupby(key, *args)[source]#

Returns nested grouped DataFrames given the multiple keys

Parameters:

  • key1 (sequence) – keys over which indexing the data

  • key2 (sequence) – keys over which indexing the data

  • ... (sequence) – keys over which indexing the data

  • key (Sequence[str | Hashable])

Returns:

it – (key1, (key2, (… keyn, {})))

Return type:

generator

property nbytes: int#

number of bytes of the object

property ncols: int#

Number of columns in the dataset

property nrows: int#

Number of rows in the dataset

pickle_dump(fname)#

create a pickle dump of the dataset

classmethod pickle_load(fname)#

restore a previously pickled object

property rows: Iterator[Any]#

Iterator on the lines of the dataframe

select(keys, caseless=False)[source]#

Read table data but returns only selected fields.

Parameters:

  • keys (str, sequence of str) – field names to select. Can be a single field names as follow: ‘RA’, or [‘RA’, ‘DEC’], or ‘RA,DEC’, or ‘RA DEC’

  • caseless (bool) – if set, do not pay attention to case.

Returns:

  • df (DataFrame) – reduced dataframe

  • ..note – there is no prior check on the variables and names

Return type:

DictDataFrame

property shape: dict[str | Hashable, tuple[int, ...]]#

dict of shapes

sortby(key, reverse=False, copy=False)[source]#
Parameters:

  • key (str) – key to sort on. Must be in the data

  • reverse (bool) – if set sort by decreasing order

  • copy (bool) – if set returns a new dataframe

Returns:

it – new dataframe only if copy is False

Return type:

DataFrame or None

to_astropy_table(**kwargs)#

A class to represent tables of heterogeneous data.

astropy.table.Table provides a class for heterogeneous tabular data, making use of a numpy structured array internally to store the data values. A key enhancement provided by the Table class is the ability to easily modify the structure of the table by adding or removing columns, or adding new rows of data. In addition table and column metadata are fully supported.

Parameters:

  • masked (bool, optional) – Specify whether the table is masked.

  • names (list, optional) – Specify column names

  • dtype (list, optional) – Specify column data types

  • meta (dict, optional) – Metadata associated with the table.

  • copy (bool, optional) – Copy the input data (default=True).

  • rows (numpy ndarray, list of lists, optional) – Row-oriented data for table instead of data argument

  • copy_indices (bool, optional) – Copy any indices in the input data (default=True)

  • **kwargs (dict, optional) – Additional keyword args when converting table-like object

Returns:

df – dataframe

Return type:

astropy.table.Table

to_dict(keys=None, contiguous=False)#

Construct a dictionary from this dataframe with contiguous arrays

Parameters:

  • keys (sequence, optional) – ordered subset of columns to export

  • contiguous (boolean) – make sure each value is a contiguous numpy array object (C-aligned)

Returns:

data – converted data

Return type:

dict

to_pandas(**kwargs)#

Construct a pandas dataframe

Parameters:

  • data (ndarray) – (structured dtype), list of tuples, dict, or DataFrame

  • keys (sequence, optional) – ordered subset of columns to export

  • index (string, list of fields, array-like) – Field of array to use as the index, alternately a specific set of input labels to use

  • exclude (sequence, default None) – Columns or fields to exclude

  • columns (sequence, default None) – Column names to use. If the passed data do not have names associated with them, this argument provides names for the columns. Otherwise this argument indicates the order of the columns in the result (any names not found in the data will become all-NA columns)

  • coerce_float (boolean, default False) – Attempt to convert values to non-string, non-numeric objects (like decimal.Decimal) to floating point, useful for SQL result sets

Returns:

df

Return type:

DataFrame

to_records(keys=None)#

convert_dict_to_structured_ndarray

Parameters:

  • data (dictionary like object) – data structure which provides iteritems and itervalues

  • keys (sequence, optional) – ordered subset of columns to export

Returns:

tab – structured numpy array

Return type:

structured ndarray

where(condition, condvars=None)[source]#

Read table data fulfilling the given condition. Only the rows fulfilling the condition are included in the result.

Parameters:

  • condition (str) – The evaluation of this condition should return True or False the condition can use field names and variables defined in condvars

  • condvars (dict) – dictionary of variables necessary to evaluate the condition.

Returns:

  • it (generator) – iterator on the query content. Each iteration contains one selected entry.

  • ..note – there is no prior check on the variables and names

Return type:

Iterator[Any]

write(fname, keys=None, header=None, **kwargs)[source]#

Write DictDataFrame into file

Parameters:

  • fname (string) – file to write into

  • keys (sequence, optional) – ordered sequence of columns

  • header (string) – header to add to the file

  • extension (str) – csv or txt/dat file auto guess from fname sets the delimiter and comments accordingly

  • delimiter (str) – force the column delimiter

  • comments (str) – force the char defining comment line

  • kwargs (dict) – forwarded to np.savetxt

simpletable.dictdataframe.helpers module#

pretty_size_print(num_bytes)[source]#

Output number of bytes in a human readable format

Parameters:

num_bytes (int) – number of bytes to convert

Returns:

output – string representation of the size with appropriate unit scale

Return type:

str

Module contents#

class DictDataFrame[source]#

Bases: dict

A simple-ish dictionary like structure allowing usage as array on non constant multi-dimensional column data.

It initializes like a normal dictionary and can be used as such. A few divergence points though: some default methods such as len() may refer to the lines and not the columns as a normal dictionary would.

This data object implements also array slicing, shape, dtypes and some data functions (sortby, groupby, where, select, etc)

property Plotter#

Plotter instance related to this dataset. Requires plotter add-on to work

__init__(*args, **kwargs)[source]#

A dictionary constructor and attributes declaration

aggregate(func, keys, args=(), kwargs={})[source]#

Apply func on groups within the data

Parameters:

  • func (callable) – function to apply

  • keys (sequence(str)) – sequence of keys defining the groups

  • args (tuple) – optional arguments to func (will be given at the end)

  • kwargs (dict) – optional keywords to func

Returns:

seq – flattened sequence of keys and value (key1, key2, … keyn, {})

Return type:

sequence

arg_groupby(key)[source]#

create an iterator which returns (key, index) grouped by each value of key(value)

Parameters:

key (str | Hashable)

Return type:

Iterator[tuple[str | Hashable, list[int]]]

property columns: ItemsView#

Iterator on the columns refers to dict.items()

property dtype: dict[str | Hashable, dtype]#

the dtypes of each column of the dataset

evalexpr(expr, exprvars=None, dtype=<class 'float'>)[source]#
evaluate expression based on the data and external variables

all np function can be used (log, exp, pi…)

Parameters:

  • data (dict or dict-like structure) – data frame / dict-like structure containing named columns

  • expr (str) – expression to evaluate on the table includes mathematical operations and attribute names

  • exprvars (dictionary, optional) – A dictionary that replaces the local operands in current frame.

  • dtype (dtype definition) – dtype of the output array

Returns:

out – array of the result

Return type:

np.array

evaluate(expr, exprvars=None, dtype=<class 'float'>)#
evaluate expression based on the data and external variables

all np function can be used (log, exp, pi…)

Parameters:

  • data (dict or dict-like structure) – data frame / dict-like structure containing named columns

  • expr (str) – expression to evaluate on the table includes mathematical operations and attribute names

  • exprvars (dictionary, optional) – A dictionary that replaces the local operands in current frame.

  • dtype (dtype definition) – dtype of the output array

Returns:

out – array of the result

Return type:

np.array

classmethod from_lines(iterable)[source]#

Create a DataFrame object from its lines instead of columns

Parameters:

iterable (iterable) – sequence of lines with the same keys (expecting dict like structure)

Returns:

df – a new object

Return type:

DataFrame

groupby(key)[source]#

create an iterator which returns (key, DataFrame) grouped by each value of key(value)

Parameters:

key (str | Hashable)

Return type:

Iterator[tuple[str | Hashable, DictDataFrame]]

iterlines()[source]#

Iterator on the lines of the dataframe

Return type:

Iterator[Any]

join(key, other, key_other=None, columns_other=None, prefix=None)[source]#

Experimental joining of structures, (equivalent to SQL left outer join) Example: >>> x = np.arange(10) >>> y = x**2 >>> z = x**3 >>> df = DictDataFrame(x=x, y=y) >>> df2 = DictDataFrame(x=x[:4], z=z[:4]) >>> df.join(‘x’, df2, ‘x’, column_other=[‘z’])

Parameters:

  • key (str) – key for the left table (self)

  • other (DictDataFrame) – Other dataset to join with (the right side of the outer join)

  • key_other (str, optional) – key on which to join (default identical to key)

  • columns_other (tuple, optional) – column names to add to the dataframe (default: all fields)

  • prefix (str, optional) – add a prefix to the new column

Returns:

self – itself

Return type:

DictDataFrame

property lines: Iterator[Any]#

Iterator on the lines of the dataframe

multigroupby(key, *args)[source]#

Returns nested grouped DataFrames given the multiple keys

Parameters:

  • key1 (sequence) – keys over which indexing the data

  • key2 (sequence) – keys over which indexing the data

  • ... (sequence) – keys over which indexing the data

  • key (Sequence[str | Hashable])

Returns:

it – (key1, (key2, (… keyn, {})))

Return type:

generator

property nbytes: int#

number of bytes of the object

property ncols: int#

Number of columns in the dataset

property nrows: int#

Number of rows in the dataset

pickle_dump(fname)#

create a pickle dump of the dataset

classmethod pickle_load(fname)#

restore a previously pickled object

property rows: Iterator[Any]#

Iterator on the lines of the dataframe

select(keys, caseless=False)[source]#

Read table data but returns only selected fields.

Parameters:

  • keys (str, sequence of str) – field names to select. Can be a single field names as follow: ‘RA’, or [‘RA’, ‘DEC’], or ‘RA,DEC’, or ‘RA DEC’

  • caseless (bool) – if set, do not pay attention to case.

Returns:

  • df (DataFrame) – reduced dataframe

  • ..note – there is no prior check on the variables and names

Return type:

DictDataFrame

property shape: dict[str | Hashable, tuple[int, ...]]#

dict of shapes

sortby(key, reverse=False, copy=False)[source]#
Parameters:

  • key (str) – key to sort on. Must be in the data

  • reverse (bool) – if set sort by decreasing order

  • copy (bool) – if set returns a new dataframe

Returns:

it – new dataframe only if copy is False

Return type:

DataFrame or None

to_astropy_table(**kwargs)#

A class to represent tables of heterogeneous data.

astropy.table.Table provides a class for heterogeneous tabular data, making use of a numpy structured array internally to store the data values. A key enhancement provided by the Table class is the ability to easily modify the structure of the table by adding or removing columns, or adding new rows of data. In addition table and column metadata are fully supported.

Parameters:

  • masked (bool, optional) – Specify whether the table is masked.

  • names (list, optional) – Specify column names

  • dtype (list, optional) – Specify column data types

  • meta (dict, optional) – Metadata associated with the table.

  • copy (bool, optional) – Copy the input data (default=True).

  • rows (numpy ndarray, list of lists, optional) – Row-oriented data for table instead of data argument

  • copy_indices (bool, optional) – Copy any indices in the input data (default=True)

  • **kwargs (dict, optional) – Additional keyword args when converting table-like object

Returns:

df – dataframe

Return type:

astropy.table.Table

to_dict(keys=None, contiguous=False)#

Construct a dictionary from this dataframe with contiguous arrays

Parameters:

  • keys (sequence, optional) – ordered subset of columns to export

  • contiguous (boolean) – make sure each value is a contiguous numpy array object (C-aligned)

Returns:

data – converted data

Return type:

dict

to_pandas(**kwargs)#

Construct a pandas dataframe

Parameters:

  • data (ndarray) – (structured dtype), list of tuples, dict, or DataFrame

  • keys (sequence, optional) – ordered subset of columns to export

  • index (string, list of fields, array-like) – Field of array to use as the index, alternately a specific set of input labels to use

  • exclude (sequence, default None) – Columns or fields to exclude

  • columns (sequence, default None) – Column names to use. If the passed data do not have names associated with them, this argument provides names for the columns. Otherwise this argument indicates the order of the columns in the result (any names not found in the data will become all-NA columns)

  • coerce_float (boolean, default False) – Attempt to convert values to non-string, non-numeric objects (like decimal.Decimal) to floating point, useful for SQL result sets

Returns:

df

Return type:

DataFrame

to_records(keys=None)#

convert_dict_to_structured_ndarray

Parameters:

  • data (dictionary like object) – data structure which provides iteritems and itervalues

  • keys (sequence, optional) – ordered subset of columns to export

Returns:

tab – structured numpy array

Return type:

structured ndarray

where(condition, condvars=None)[source]#

Read table data fulfilling the given condition. Only the rows fulfilling the condition are included in the result.

Parameters:

  • condition (str) – The evaluation of this condition should return True or False the condition can use field names and variables defined in condvars

  • condvars (dict) – dictionary of variables necessary to evaluate the condition.

Returns:

  • it (generator) – iterator on the query content. Each iteration contains one selected entry.

  • ..note – there is no prior check on the variables and names

Return type:

Iterator[Any]

write(fname, keys=None, header=None, **kwargs)[source]#

Write DictDataFrame into file

Parameters:

  • fname (string) – file to write into

  • keys (sequence, optional) – ordered sequence of columns

  • header (string) – header to add to the file

  • extension (str) – csv or txt/dat file auto guess from fname sets the delimiter and comments accordingly

  • delimiter (str) – force the column delimiter

  • comments (str) – force the char defining comment line

  • kwargs (dict) – forwarded to np.savetxt

Contents