simpletable package

simpletable package#

Subpackages#

simpletable.dataframe package
simpletable.dictdataframe package
simpletable.legacy package
- Submodules
- simpletable.legacy.simpletable module
  - Requirements
  - AstroHelpers
  - AstroTable
  - SimpleTable
  - stats
    - has_nan()
    - max()
    - mean()
    - min()
    - p16()
    - p50()
    - p84()
    - std()
    - var()
- Module contents

Module contents#

class DataFrame[source]#

Bases: DataFrame

DataFrame class that is basically a wrapper around pandas.DataFrame which supports: - Column aliases - Units - Case sensitivity - search in the columns - preserve attrs information at initialization from DataFrame - inputs and outputs into various additional formats: FITS, CSV, ECSV

__init__(data, info=None, caseless=False)[source]#

Parameters:

data (ndarray[tuple[Any, ...], dtype[_ScalarT]] | Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | bytes | str | _NestedSequence[complex | bytes | str] | Iterable | dict | DataFrame)
info (HeaderInfo | None)
caseless (bool)

property colnames: list[str]#

classmethod from_ascii(filepath_or_buffer, *, commented_header=False, **kwargs)[source]#

Read an ASCII file into a DataFrame.

from_csv with delimiter set to “ “ by default

Parameters:

filepath_or_buffer (str, path object or file-like object) – Any valid string path is acceptable. The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected. A local file could be: file://localhost/path/to/table.csv.
commented_header (bool, default False) – Whether the header is commented or not.
**kwargs (dict) – Additional keyword arguments passed to pd.read_csv.

Returns:

DataFrame (pd.DataFrame) – The parsed data as a pd.DataFrame.
header (HeaderInfo) – The header information extracted from the file.

Return type:

DataFrame

See also

from_csv: Read a CSV file into a DataFrame

classmethod from_csv(filepath_or_buffer, *, commented_header=False, **kwargs)[source]#

Read a CSV file into a DataFrame while preserving header information

Equivalent to pd.read_csv with preserved header information.

Also supports optionally iterating or breaking of the file into chunks.

Additional help can be found in the online docs for IO Tools.

Parameters:

filepath_or_buffer (str, path object or file-like object) – Any valid string path is acceptable.
commented_header (bool, default False) – Whether the column definition header line starts with a comment character.
commentchar (str, default '#') – Character to treat as a comment character.
sep (str, default ',') – Character or regex pattern to treat as the delimiter. If sep=None, the C engine cannot automatically detect the separator, but the Python parsing engine can, meaning the latter will be used and automatically detect the separator from only the first valid row of the file by Python’s builtin sniffer tool, csv.Sniffer. In addition, separators longer than 1 character and different from '\s+' will be interpreted as regular expressions and will also force the use of the Python parsing engine. Note that regex delimiters are prone to ignoring quoted data. Regex example: '\r\t'.

Returns:

DataFrame (pd.DataFrame) – The parsed data as a pd.DataFrame.
header (HeaderInfo) – The header information extracted from the file.

Return type:

DataFrame

See also

from_ascii: Read a ASCII file into a DataFrame

classmethod from_ecsv(fname, **kwargs)[source]#

Read the content of an Enhanced Character Separated Values

Parameters:: fname (str) – The name of the file to read.
Returns:: data – The data read from the file. The chunked data read from the file.
Return type:: pd.DataFrame | Generator[pd.DataFrame, Any, None]

classmethod from_fits(filename, extension_number=1)[source]#

Load a DataFrame from a FITS file.

Parameters:

filename (str) – The path to the FITS file.
extension_number (int, optional) – The extension number to load, by default 1.

Returns:

The loaded DataFrame.

Return type:

DataFrame

classmethod from_votable(fname, *, table_index=0, is_url=False)[source]#

Read a VOTable file and return a pandas DataFrame and header information.

Parameters:

fname (str, bytes, IOBase, PathLike) – The filename or file-like object to read.
table_index (int, optional) – The index of the table to read, by default 0.
is_url (bool, optional) – Whether the file is a URL, by default False.

Returns:

df – Dataframe from VOTable file.

Return type:

DataFrame

property header: HeaderInfo#

info(header=False)[source]#

prints information on the table

Parameters:: header (bool, default False) – If True, print the header information.

keys(regexp=None, full_match=False, skip_aliases=False)[source]#

Return the data column names or a subset of it

Parameters:

regexp (str) – pattern to filter the keys with
full_match (bool) – if set, use re.fullmatch() instead of re.findall()
skip_aliases (bool) – include aliases if unset
string (Try to apply the pattern at the start of the)
returning
object (a match)
found. (or None if no match was)

Returns:

seq – sequence of keys

Return type:

sequence

property nbytes#: number of bytes of the object

property ncols: int#

property nrows: int#

pandas_info(verbose=None, buf=None, max_cols=None, memory_usage=None, show_counts=None)[source]#

Print a concise summary of a DataFrame.

This method prints information about a DataFrame including the index dtype and columns, non-null values and memory usage.

Parameters:

verbose (bool, optional) – Whether to print the full summary. By default, the setting in pandas.options.display.max_info_columns is followed.
buf (writable buffer, defaults to sys.stdout) – Where to send the output. By default, the output is printed to sys.stdout. Pass a writable buffer if you need to further process the output.
max_cols (int, optional) – When to switch from the verbose to the truncated output. If the DataFrame has more than max_cols columns, the truncated output is used. By default, the setting in pandas.options.display.max_info_columns is used.
memory_usage (bool, str, optional) –
Specifies whether total memory usage of the DataFrame elements (including the index) should be displayed. By default, this follows the pandas.options.display.memory_usage setting.

True always show memory usage. False never shows memory usage. A value of ‘deep’ is equivalent to “True with deep introspection”. Memory usage is shown in human-readable units (base-2 representation). Without deep introspection a memory estimation is made based in column dtype and number of rows assuming values consume the same memory amount for corresponding dtypes. With deep memory introspection, a real memory usage calculation is performed at the cost of computational resources. See the Frequently Asked Questions for more details.
show_counts (bool, optional) – Whether to show the non-null counts. By default, this is shown only if the DataFrame is smaller than pandas.options.display.max_info_rows and pandas.options.display.max_info_columns. A value of True always shows the counts, and False never shows the counts.

Returns:

This method prints a summary of a DataFrame and returns None.

Return type:

None

See also

DataFrame.describe: Generate descriptive statistics of DataFrame columns.
DataFrame.memory_usage: Memory usage of DataFrame columns.

resolve_alias(colname)[source]#

Resolve an alias to the actual column name

Parameters:: colname (str | bytes | Hashable | Any) – Name or alias of the column
Returns:: Actual column name or initial value if not found
Return type:: str | bytes | Hashable | Any

reverse_alias(colname)[source]#

Returns aliases of a given column

Parameters:: colname (str | bytes | Hashable | Any) – Name or alias of the column
Returns:: Actual column name or initial value if not found
Return type:: str | bytes | Hashable | Any

set_alias(alias, colname)[source]#

Set an alias for a column referenced by name or alias

Parameters:

alias (str)
colname (str)

Return type:

None

set_column_comment(colname, comment)[source]#

Set the comment for a column referenced by name or alias

Parameters:

colname (str) – Name or alias of the column
comment (str) – Comment to set for the column

set_column_unit(colname, unit)[source]#

Set the unit for a column referenced by name or alias

Parameters:

colname (str) – Name or alias of the column
unit (str) – Unit to set for the column

property shape: tuple[int, int]#

Return a tuple representing the dimensionality of the DataFrame.

See also

ndarray.shape: Tuple of array dimensions.

Examples

>>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
>>> df.shape
(2, 2)

>>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4],
...                    'col3': [5, 6]})
>>> df.shape
(2, 3)

to_ascii(filepath_or_buffer, *, sep=' ', commentchar='#', **kwargs)#

Write object to an ASCII values file while preserving attrs

Equivalent to to_csv with default sep set to a space.

See also

to_csv: Write object to a CSV file while preserving attrs

Parameters:

self (DataFrame)
filepath_or_buffer (str | PathLike[str] | BaseBuffer)
sep (str)
commentchar (str)

Return type:

str | None

to_csv(filepath_or_buffer, *, sep=',', commentchar='#', na_rep='', float_format=None, columns=None, header=True, index=True, index_label=None, mode='w', encoding=None, compression='infer', quoting=None, quotechar='"', lineterminator=None, chunksize=None, date_format=None, doublequote=True, escapechar=None, decimal='.', errors='strict', storage_options=None)#

Write object to a comma-separated values (csv) file while preserving attrs

Fallsback to pd.DataFrame.to_csv if no attrs content

Parameters:

path_or_buf (str, path object, file-like object, or None, default None) – String, path object (implementing os.PathLike[str]), or file-like object implementing a write() function. If None, the result is returned as a string. If a non-binary file object is passed, it should be opened with newline=’’, disabling universal newlines. If a binary file object is passed, mode might need to contain a ‘b’.
sep (str, default ',') – String of length 1. Field delimiter for the output file.
commentchar (str, default '#') – Character starting a comment line for the output file.
na_rep (str, default '') – Missing data representation.
float_format (str, Callable, default None) – Format string for floating point numbers. If a Callable is given, it takes precedence over other numeric formatting parameters, like decimal.
columns (sequence, optional) – Columns to write.
header (bool or list of str, default True) – Write out the column names. If a list of strings is given it is assumed to be aliases for the column names.
index (bool, default True) – Write row names (index).
index_label (str or sequence, or False, default None) – Column label for index column(s) if desired. If None is given, and header and index are True, then the index names are used. A sequence should be given if the object uses MultiIndex. If False do not print fields for index names. Use index_label=False for easier importing in R.
mode ({{'w', 'x', 'a'}}, default 'w') –
Forwarded to either open(mode=) or fsspec.open(mode=) to control the file opening. Typical values include:
- ’w’, truncate the file first.
- ’x’, exclusive creation, failing if the file already exists.
- ’a’, append to the end of file if it exists.
encoding (str, optional) –
A string representing the encoding to use in the output file, defaults to ‘utf-8’. encoding is not supported if path_or_buf is a non-binary file object. compression : str or dict, default ‘infer’

For on-the-fly compression of the output data. If ‘infer’ and ‘path_or_buf’ is path-like, then detect compression from the following extensions: ‘.gz’, ‘.bz2’, ‘.zip’, ‘.xz’, ‘.zst’, ‘.tar’, ‘.tar.gz’, ‘.tar.xz’ or ‘.tar.bz2’ (otherwise no compression). Set to None for no compression. Can also be a dict with key 'method' set to one of {'zip', 'gzip', 'bz2', 'zstd', 'xz', 'tar'} and other key-value pairs are forwarded to zipfile.ZipFile, gzip.GzipFile, bz2.BZ2File, zstandard.ZstdCompressor, lzma.LZMAFile or tarfile.TarFile, respectively. As an example, the following could be passed for faster compression and to create a reproducible gzip archive: compression={'method': 'gzip', 'compresslevel': 1, 'mtime': 1}.

May be a dict with key ‘method’ as compression mode and other entries as additional compression options if compression mode is ‘zip’.

Passing compression options as keys in dict is supported for compression modes ‘gzip’, ‘bz2’, ‘zstd’, and ‘zip’.
quoting (optional constant from csv module) – Defaults to csv.QUOTE_MINIMAL. If you have set a float_format then floats are converted to strings and thus csv.QUOTE_NONNUMERIC will treat them as non-numeric.
quotechar (str, default '"') – String of length 1. Character used to quote fields.
lineterminator (str, optional) – The newline character or character sequence to use in the output file. Defaults to os.linesep, which depends on the OS in which this method is called (’\n’ for linux, ‘\r\n’ for Windows, i.e.).
chunksize (int or None) – Rows to write at a time.
date_format (str, default None) – Format string for datetime objects.
doublequote (bool, default True) – Control quoting of quotechar inside a field.
escapechar (str, default None) – String of length 1. Character used to escape sep and quotechar when appropriate.
decimal (str, default '.') – Character recognized as decimal separator. E.g. use ‘,’ for European data.
errors (str, default 'strict') – Specifies how encoding and decoding errors are to be handled. See the errors argument for open() for a full list of options.
storage_options (dict, optional) – Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc. For HTTP(S) URLs the key-value pairs are forwarded to urllib.request.Request as header options. For other URLs (e.g. starting with “s3://”, and “gcs://”) the key-value pairs are forwarded to fsspec.open. Please see fsspec and urllib for more details, and for more examples on storage options refer here.
self (DataFrame)
filepath_or_buffer (str | PathLike[str] | BaseBuffer)
compression (Literal['infer', 'gzip', 'bz2', 'zip', 'xz', 'zstd', 'tar'] | dict[str, ~typing.Any] | None)

Returns:

If path_or_buf is None, returns the resulting csv format as a string. Otherwise returns None.

Return type:

None or str

to_ecsv(df, fname, mode='w', **meta)[source]#

output data into ecsv file

Parameters:

df (pd.DataFrame) – data to be written to the file.
fname (str) – the name of the file to write.
mode (str) – the mode to open the file.
meta (dict) – meta data to be written to the header.

Return type:

None

to_fits(filename, header_info=None, output_verify='exception', checksum=False, index=True, overwrite=False, append=False, **kwargs)[source]#

Save a DataFrame to a FITS file.

Parameters:

filename (str) – The path to the FITS file.
header_info (HeaderInfo | None, optional) – Header information to save with the FITS file by default None and taken from data.attrs override data.attrs if provided
output_verify (str) – Output verification option. Must be one of "fix", "silentfix", "ignore", "warn", or "exception". May also be any combination of "fix" or "silentfix" with "+ignore", +warn, or +exception" (e.g. ``"fix+warn"). See verify for more info.
checksum (bool, optional) – If True, adds both DATASUM and CHECKSUM cards to the headers of all HDU’s written to the file
index (bool, optional) – If True, includes the index in the FITS file. Default is True.
append (bool, optional) – If True, appends the DataFrame to the FITS file. Default is False.
overwrite (bool, optional) – If True, overwrites the DataFrame in the FITS file. Default is False.
**kwargs (dict) – Additional keyword arguments to pass to the FITS writer.

Return type:

None

class DictDataFrame[source]#

Bases: dict

A simple-ish dictionary like structure allowing usage as array on non constant multi-dimensional column data.

It initializes like a normal dictionary and can be used as such. A few divergence points though: some default methods such as len() may refer to the lines and not the columns as a normal dictionary would.

This data object implements also array slicing, shape, dtypes and some data functions (sortby, groupby, where, select, etc)

property Plotter#: Plotter instance related to this dataset. Requires plotter add-on to work

__init__(*args, **kwargs)[source]#: A dictionary constructor and attributes declaration

aggregate(func, keys, args=(), kwargs={})[source]#

Apply func on groups within the data

Parameters:

func (callable) – function to apply
keys (sequence(str)) – sequence of keys defining the groups
args (tuple) – optional arguments to func (will be given at the end)
kwargs (dict) – optional keywords to func

Returns:

seq – flattened sequence of keys and value (key1, key2, … keyn, {})

Return type:

sequence

arg_groupby(key)[source]#

create an iterator which returns (key, index) grouped by each value of key(value)

Parameters:: key (str | Hashable)
Return type:: Iterator[tuple[str | Hashable, list[int]]]

property columns: ItemsView#: Iterator on the columns refers to dict.items()

property dtype: dict[str | Hashable, dtype]#: the dtypes of each column of the dataset

evalexpr(expr, exprvars=None, dtype=<class 'float'>)[source]#

evaluate expression based on the data and external variables: all np function can be used (log, exp, pi…)

Parameters:

data (dict or dict-like structure) – data frame / dict-like structure containing named columns
expr (str) – expression to evaluate on the table includes mathematical operations and attribute names
exprvars (dictionary, optional) – A dictionary that replaces the local operands in current frame.
dtype (dtype definition) – dtype of the output array

Returns:

out – array of the result

Return type:

np.array

evaluate(expr, exprvars=None, dtype=<class 'float'>)#

evaluate expression based on the data and external variables: all np function can be used (log, exp, pi…)

Parameters:

data (dict or dict-like structure) – data frame / dict-like structure containing named columns
expr (str) – expression to evaluate on the table includes mathematical operations and attribute names
exprvars (dictionary, optional) – A dictionary that replaces the local operands in current frame.
dtype (dtype definition) – dtype of the output array

Returns:

out – array of the result

Return type:

np.array

classmethod from_lines(iterable)[source]#

Create a DataFrame object from its lines instead of columns

Parameters:: iterable (iterable) – sequence of lines with the same keys (expecting dict like structure)
Returns:: df – a new object
Return type:: DataFrame

groupby(key)[source]#

create an iterator which returns (key, DataFrame) grouped by each value of key(value)

Parameters:: key (str | Hashable)
Return type:: Iterator[tuple[str | Hashable, DictDataFrame]]

iterlines()[source]#

Iterator on the lines of the dataframe

Return type:: Iterator[Any]

join(key, other, key_other=None, columns_other=None, prefix=None)[source]#

Experimental joining of structures, (equivalent to SQL left outer join) Example: >>> x = np.arange(10) >>> y = x**2 >>> z = x**3 >>> df = DictDataFrame(x=x, y=y) >>> df2 = DictDataFrame(x=x[:4], z=z[:4]) >>> df.join(‘x’, df2, ‘x’, column_other=[‘z’])

Parameters:

key (str) – key for the left table (self)
other (DictDataFrame) – Other dataset to join with (the right side of the outer join)
key_other (str, optional) – key on which to join (default identical to key)
columns_other (tuple, optional) – column names to add to the dataframe (default: all fields)
prefix (str, optional) – add a prefix to the new column

Returns:

self – itself

Return type:

DictDataFrame

property lines: Iterator[Any]#: Iterator on the lines of the dataframe

multigroupby(key, *args)[source]#

Returns nested grouped DataFrames given the multiple keys

Parameters:

key1 (sequence) – keys over which indexing the data
key2 (sequence) – keys over which indexing the data
... (sequence) – keys over which indexing the data
key (Sequence[str | Hashable])

Returns:

it – (key1, (key2, (… keyn, {})))

Return type:

generator

property nbytes: int#: number of bytes of the object

property ncols: int#: Number of columns in the dataset

property nrows: int#: Number of rows in the dataset

pickle_dump(fname)#: create a pickle dump of the dataset

classmethod pickle_load(fname)#: restore a previously pickled object

property rows: Iterator[Any]#: Iterator on the lines of the dataframe

select(keys, caseless=False)[source]#

Read table data but returns only selected fields.

Parameters:

keys (str, sequence of str) – field names to select. Can be a single field names as follow: ‘RA’, or [‘RA’, ‘DEC’], or ‘RA,DEC’, or ‘RA DEC’
caseless (bool) – if set, do not pay attention to case.

Returns:

df (DataFrame) – reduced dataframe
..note – there is no prior check on the variables and names

Return type:

DictDataFrame

property shape: dict[str | Hashable, tuple[int, ...]]#: dict of shapes

sortby(key, reverse=False, copy=False)[source]#

Parameters:

key (str) – key to sort on. Must be in the data
reverse (bool) – if set sort by decreasing order
copy (bool) – if set returns a new dataframe

Returns:

it – new dataframe only if copy is False

Return type:

DataFrame or None

to_astropy_table(**kwargs)#

A class to represent tables of heterogeneous data.

astropy.table.Table provides a class for heterogeneous tabular data, making use of a numpy structured array internally to store the data values. A key enhancement provided by the Table class is the ability to easily modify the structure of the table by adding or removing columns, or adding new rows of data. In addition table and column metadata are fully supported.

Parameters:

masked (bool, optional) – Specify whether the table is masked.
names (list, optional) – Specify column names
dtype (list, optional) – Specify column data types
meta (dict, optional) – Metadata associated with the table.
copy (bool, optional) – Copy the input data (default=True).
rows (numpy ndarray, list of lists, optional) – Row-oriented data for table instead of data argument
copy_indices (bool, optional) – Copy any indices in the input data (default=True)
**kwargs (dict, optional) – Additional keyword args when converting table-like object

Returns:

df – dataframe

Return type:

astropy.table.Table

to_dict(keys=None, contiguous=False)#

Construct a dictionary from this dataframe with contiguous arrays

Parameters:

keys (sequence, optional) – ordered subset of columns to export
contiguous (boolean) – make sure each value is a contiguous numpy array object (C-aligned)

Returns:

data – converted data

Return type:

dict

to_pandas(**kwargs)#

Construct a pandas dataframe

Parameters:

data (ndarray) – (structured dtype), list of tuples, dict, or DataFrame
keys (sequence, optional) – ordered subset of columns to export
index (string, list of fields, array-like) – Field of array to use as the index, alternately a specific set of input labels to use
exclude (sequence, default None) – Columns or fields to exclude
columns (sequence, default None) – Column names to use. If the passed data do not have names associated with them, this argument provides names for the columns. Otherwise this argument indicates the order of the columns in the result (any names not found in the data will become all-NA columns)
coerce_float (boolean, default False) – Attempt to convert values to non-string, non-numeric objects (like decimal.Decimal) to floating point, useful for SQL result sets

Returns:

Return type:

DataFrame

to_records(keys=None)#

convert_dict_to_structured_ndarray

Parameters:

data (dictionary like object) – data structure which provides iteritems and itervalues
keys (sequence, optional) – ordered subset of columns to export

Returns:

tab – structured numpy array

Return type:

structured ndarray

where(condition, condvars=None)[source]#

Read table data fulfilling the given condition. Only the rows fulfilling the condition are included in the result.

Parameters:

condition (str) – The evaluation of this condition should return True or False the condition can use field names and variables defined in condvars
condvars (dict) – dictionary of variables necessary to evaluate the condition.

Returns:

it (generator) – iterator on the query content. Each iteration contains one selected entry.
..note – there is no prior check on the variables and names

Return type:

Iterator[Any]

write(fname, keys=None, header=None, **kwargs)[source]#

Write DictDataFrame into file

Parameters:

fname (string) – file to write into
keys (sequence, optional) – ordered sequence of columns
header (string) – header to add to the file
extension (str) – csv or txt/dat file auto guess from fname sets the delimiter and comments accordingly
delimiter (str) – force the column delimiter
comments (str) – force the char defining comment line
kwargs (dict) – forwarded to np.savetxt

simpletable package

Contents

simpletable package#

Subpackages#

Module contents#