simpletable.dataframe.convert package

Submodules

simpletable.dataframe.convert.ascii module

Export dataframe to ASCII format while preserving attrs

ascii_generate_header(df, comments='#', delimiter=' ', commented_header=True)

Generate the corresponding ascii Header that contains all necessary info

Parameters:

• df (pd.DataFrame) – table to export

• comments (str) – string to prepend header lines

• delimiter (str, optional) – The string used to separate values. By default, this is any whitespace.

• commented_header (bool, optional) – if set, the last line of the header is expected to be the column titles

Returns:

hdr – string that will be be written at the beginning of the file

Return type:

str

ascii_read_header(fname, *, commentchar='#', delimiter=',', commented_header=True, **kwargs)

Read ASCII/CSV header

Parameters:

• fname (str, FilePath, BaseBuffer) – File, filename, or generator to read. Note that generators should return byte strings for Python >=3.

• comments (str, optional) – The character used to indicate the start of a comment; default: ‘#’. (”” is equivalent to None)

• delimiter (str, optional) – The string used to separate values. By default, this is any whitespace.

• commented_header (bool, optional) – if set, the last line of the header is expected to be the column titles (with comment character) otherwise, the first line of the data will be the column titles

• commentchar (str)

Returns:

• nlines (int) – number of lines from the header

• info (HeaderInfo) – header information (header, alias, units, comments)

• names (List[str]) – sequence or str, first data line after header, expected to be the column names.

Return type:

tuple[int, HeaderInfo, list[str]]

from_ascii(filepath_or_buffer, *, commented_header=False, **kwargs)

Read an ASCII file into a DataFrame.

from_csv with delimiter set to “ “ by default

Parameters:

• filepath_or_buffer (str, path object or file-like object) – Any valid string path is acceptable. The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected. A local file could be: file://localhost/path/to/table.csv.

• commented_header (bool, default False) – Whether the header is commented or not.

• **kwargs (dict) – Additional keyword arguments passed to pd.read_csv.

Returns:

• DataFrame (pd.DataFrame) – The parsed data as a pd.DataFrame.

• header (HeaderInfo) – The header information extracted from the file.

See also

from_csv

Read a CSV file into a DataFrame

from_csv(filepath_or_buffer, *, commented_header=False, **kwargs)

Read a CSV file into a DataFrame while preserving header information

Equivalent to pd.read_csv with preserved header information.

Also supports optionally iterating or breaking of the file into chunks.

Additional help can be found in the online docs for IO Tools.

Parameters:

• filepath_or_buffer (str, path object or file-like object) – Any valid string path is acceptable.

• commented_header (bool, default False) – Whether the column definition header line starts with a comment character.

• commentchar (str, default '#') – Character to treat as a comment character.

• sep (str, default ',') – Character or regex pattern to treat as the delimiter. If sep=None, the C engine cannot automatically detect the separator, but the Python parsing engine can, meaning the latter will be used and automatically detect the separator from only the first valid row of the file by Python’s builtin sniffer tool, csv.Sniffer. In addition, separators longer than 1 character and different from '\s+' will be interpreted as regular expressions and will also force the use of the Python parsing engine. Note that regex delimiters are prone to ignoring quoted data. Regex example: '\r\t'.

Returns:

• DataFrame (pd.DataFrame) – The parsed data as a pd.DataFrame.

• header (HeaderInfo) – The header information extracted from the file.

See also

pd.read_csv

Read a CSV file into a DataFrame

to_ascii(self, filepath_or_buffer, *, sep=' ', commentchar='#', **kwargs)

Write object to an ASCII values file while preserving attrs

Equivalent to to_csv with default sep set to a space.

See also

to_csv

Write object to a CSV file while preserving attrs

Parameters:

• self (DataFrame)

• filepath_or_buffer (str | PathLike[str] | BaseBuffer)

• sep (str)

• commentchar (str)

Return type:

str | None

to_csv(self, filepath_or_buffer, *, sep=',', commentchar='#', na_rep='', float_format=None, columns=None, header=True, index=True, index_label=None, mode='w', encoding=None, compression='infer', quoting=None, quotechar='"', lineterminator=None, chunksize=None, date_format=None, doublequote=True, escapechar=None, decimal='.', errors='strict', storage_options=None)

Write object to a comma-separated values (csv) file while preserving attrs

Fallsback to pd.DataFrame.to_csv if no attrs content

Parameters:

• path_or_buf (str, path object, file-like object, or None, default None) – String, path object (implementing os.PathLike[str]), or file-like object implementing a write() function. If None, the result is returned as a string. If a non-binary file object is passed, it should be opened with newline=’’, disabling universal newlines. If a binary file object is passed, mode might need to contain a ‘b’.

• sep (str, default ',') – String of length 1. Field delimiter for the output file.

• commentchar (str, default '#') – Character starting a comment line for the output file.

• na_rep (str, default '') – Missing data representation.

• float_format (str, Callable, default None) – Format string for floating point numbers. If a Callable is given, it takes precedence over other numeric formatting parameters, like decimal.

• columns (sequence, optional) – Columns to write.

• header (bool or list of str, default True) – Write out the column names. If a list of strings is given it is assumed to be aliases for the column names.

• index (bool, default True) – Write row names (index).

• index_label (str or sequence, or False, default None) – Column label for index column(s) if desired. If None is given, and header and index are True, then the index names are used. A sequence should be given if the object uses MultiIndex. If False do not print fields for index names. Use index_label=False for easier importing in R.

• mode ({{'w', 'x', 'a'}}, default 'w') –

  Forwarded to either open(mode=) or fsspec.open(mode=) to control the file opening. Typical values include:

  • ’w’, truncate the file first.

  • ’x’, exclusive creation, failing if the file already exists.

  • ’a’, append to the end of file if it exists.

• encoding (str, optional) –

  A string representing the encoding to use in the output file, defaults to ‘utf-8’. encoding is not supported if path_or_buf is a non-binary file object. compression : str or dict, default ‘infer’

  For on-the-fly compression of the output data. If ‘infer’ and ‘path_or_buf’ is path-like, then detect compression from the following extensions: ‘.gz’, ‘.bz2’, ‘.zip’, ‘.xz’, ‘.zst’, ‘.tar’, ‘.tar.gz’, ‘.tar.xz’ or ‘.tar.bz2’ (otherwise no compression). Set to None for no compression. Can also be a dict with key 'method' set to one of {'zip', 'gzip', 'bz2', 'zstd', 'xz', 'tar'} and other key-value pairs are forwarded to zipfile.ZipFile, gzip.GzipFile, bz2.BZ2File, zstandard.ZstdCompressor, lzma.LZMAFile or tarfile.TarFile, respectively. As an example, the following could be passed for faster compression and to create a reproducible gzip archive: compression={'method': 'gzip', 'compresslevel': 1, 'mtime': 1}.

  May be a dict with key ‘method’ as compression mode and other entries as additional compression options if compression mode is ‘zip’.

  Passing compression options as keys in dict is supported for compression modes ‘gzip’, ‘bz2’, ‘zstd’, and ‘zip’.

• quoting (optional constant from csv module) – Defaults to csv.QUOTE_MINIMAL. If you have set a float_format then floats are converted to strings and thus csv.QUOTE_NONNUMERIC will treat them as non-numeric.

• quotechar (str, default '"') – String of length 1. Character used to quote fields.

• lineterminator (str, optional) – The newline character or character sequence to use in the output file. Defaults to os.linesep, which depends on the OS in which this method is called (’\n’ for linux, ‘\r\n’ for Windows, i.e.).

• chunksize (int or None) – Rows to write at a time.

• date_format (str, default None) – Format string for datetime objects.

• doublequote (bool, default True) – Control quoting of quotechar inside a field.

• escapechar (str, default None) – String of length 1. Character used to escape sep and quotechar when appropriate.

• decimal (str, default '.') – Character recognized as decimal separator. E.g. use ‘,’ for European data.

• errors (str, default 'strict') – Specifies how encoding and decoding errors are to be handled. See the errors argument for open() for a full list of options.

• storage_options (dict, optional) – Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc. For HTTP(S) URLs the key-value pairs are forwarded to urllib.request.Request as header options. For other URLs (e.g. starting with “s3://”, and “gcs://”) the key-value pairs are forwarded to fsspec.open. Please see fsspec and urllib for more details, and for more examples on storage options refer here.

• self (DataFrame)

• filepath_or_buffer (str | PathLike[str] | BaseBuffer)

• compression (Literal['infer', 'gzip', 'bz2', 'zip', 'xz', 'zstd', 'tar'] | dict[str, ~typing.Any] | None)

Returns:

If path_or_buf is None, returns the resulting csv format as a string. Otherwise returns None.

Return type:

None or str

simpletable.dataframe.convert.ecsv module

The ECSV format allows for specification of key table and column meta-data, in particular the data type and unit.

The “Comma-Separated-Values” format is probably the most common text-based method for representing data tables. The proposed standard in APE6 leverages this universality by using this format for serializing the actual data values.

For science applications, pure CSV has a serious shortcoming in the complete lack of support for table metadata. This is frequently not a showstopper but represents a serious limitation which has motivated alternate standards in the astronomical community.

The proposed Enhanced CSV (ECSV) format has the following overall structure:

A header section which consists of lines that start with the # character and provide the table definition and data format via a YAML-encoded data structure. An initial line in the header section which identifies the file as ECSV and provides a version number. A CSV-formatted data section in which the first line contains the column names and subsequent lines contains the data values. Version 1.0 of the ECSV format specification and the reference Python implementation assumes ASCII-encoded header and data sections. Support for unicode (in particular UTF-8) may be added in subsequent versions.

Note

“Comma-Separated-Values” (CSV) is a misleading name since tab-separated or whitespace-separated tabular data generally fall in this category. We should use “Character-Separated-Values”, which keep the same acronym.

example of file:

# %ECSV 1.0
# ---
# datatype:
# - {name: a, unit: m / s, datatype: int64, format: '%03d'}
# - {name: b, unit: km, datatype: int64, description: This is column b}
a b
1 2
4 3

See also

Enhanced Character Separated Values table format description from the Astropy project.

generate_header(df, **meta)

Generates the yaml equivalent string for the ECSV header

Parameters:

• df (pd.DataFrame) – data to be written to the file.

• meta (dict) – meta data to be written to the header. Typically keywords, comments, history and so forth should be part of meta. df.attrs will be automatically added to the meta data.

Returns:

header – the header corresponding to the data.

Return type:

str

read(fname, **kwargs)

Read the content of an Enhanced Character Separated Values

Parameters:

fname (str) – The name of the file to read.

Returns:

data – The data read from the file.

Return type:

pd.DataFrame

read_header(fname)

read the header of ECSV file as a dictionary

Parameters:

fname (str) – The name of the file to read.

Returns:

data – the header of the file.

Return type:

dict

write(df, fname, mode='w', **meta)

output data into ecsv file

Parameters:

• df (pd.DataFrame) – data to be written to the file.

• fname (str) – the name of the file to write.

• mode (str) – the mode to open the file.

• meta (dict) – meta data to be written to the header.

simpletable.dataframe.convert.fits module

fits_generate_hdu(df, index=True, header_info=None)

Generate a FITS BinTableHDU from a DataFrame.

Parameters:

• df (pd.DataFrame) – The DataFrame to convert.

• index (bool, optional) – Whether to include the index in the table, by default True.

• header_info (HeaderInfo | None)

Returns:

The generated HDU.

Return type:

BinTableHDU

fits_generate_header(df, header_info=None)

Generate the corresponding fits Header that contains all necessary info

Parameters:

• df (pd.DataFrame) – DataFrame or HeaderInfo instance

• header_info (HeaderInfo | None)

Returns:

hdr – header instance

Return type:

fits.Header

fits_read_header(hdr)

Convert pyfits header into dictionary with relevant values

Parameters:

hdr (pyftis.Header) – fits unit

Returns:

headerinfo – extracted information from header

Return type:

HeaderInfo

fix_endian_issue(arr)

Fix endian issue in array which happens often when reading FITS files

Parameters:

arr (ndarray[tuple[Any, ...], dtype[_ScalarT]] | Any)

Return type:

ndarray[tuple[Any, …], dtype[_ScalarT]]

from_fits(filename, extension_number=1)

Load a DataFrame from a FITS file.

Parameters:

• filename (str) – The path to the FITS file.

• extension_number (int, optional) – The extension number to load, by default 1.

Returns:

The loaded DataFrame and its header information.

Return type:

Tuple[npt.NDArray, HeaderInfo]

to_fits(df, filename, header_info=None, output_verify='exception', checksum=False, index=True, overwrite=False, append=False, **kwargs)

Save a DataFrame to a FITS file.

Parameters:

• df (pd.DataFrame) – The DataFrame to save. header information taken from data.attrs or header_info if provided

• filename (str) – The path to the FITS file.

• header_info (HeaderInfo | None, optional) – Header information to save with the FITS file by default None and taken from data.attrs override data.attrs if provided

• output_verify (str) – Output verification option. Must be one of "fix", "silentfix", "ignore", "warn", or "exception". May also be any combination of "fix" or "silentfix" with "+ignore", +warn, or +exception" (e.g. ``"fix+warn"). See verify for more info.

• checksum (bool, optional) – If True, adds both DATASUM and CHECKSUM cards to the headers of all HDU’s written to the file

• index (bool, optional) – If True, includes the index in the FITS file. Default is True.

• append (bool, optional) – If True, appends the DataFrame to the FITS file. Default is False.

• overwrite (bool, optional) – If True, overwrites the DataFrame in the FITS file. Default is False.

• **kwargs (dict) – Additional keyword arguments to pass to the FITS writer.

Return type:

None

simpletable.dataframe.convert.hdf module

Read and write HDF5 files with pytables preserving metadata (tables, https://www.pytables.org/)

Important

This module relies on pytables

simpletable.dataframe.convert.votable module

VOTable parser for astronomical tabular data.

VOTable is the standard XML format for astronomical tabular data. This module implements a custom VOTableParser that uses XML parsing and not other dependencies. from_votable provides the standard interface of io operations (pandas.DataFrame, HeaderInfo)

class VOTableParser

Bases: object

A custom VOTable parser using XML parsing.

VOTable is the standard XML format for astronomical tabular data. This example shows how to parse the structure and extract data.

__init__(source, is_url=False)

Initialize VOTable parser

Parameters:

• source (str or bytes) – Either a file path, URL, or XML string/bytes

• is_url (bool) – If True, treat source as URL to fetch

get_table_as_dict(table_index=0)

Convert table data to dictionary with column names as keys

Parameters:

table_index (int) – Index of table to convert (default: 0 for first table)

Returns:

Dictionary with column names as keys and data as lists

Return type:

dict

print_table_info(table_index=0)

Print information about a table

Parameters:

table_index (int)

from_votable(fname, *, table_index=0, is_url=False)

Read a VOTable file and return a pandas DataFrame and header information.

Parameters:

• fname (str, bytes, IOBase, PathLike) – The filename or file-like object to read.

• table_index (int, optional) – The index of the table to read, by default 0.

• is_url (bool, optional) – Whether the file is a URL, by default False.

Returns:

A tuple containing the pandas DataFrame and header information.

Return type:

Tuple[pd.DataFrame, HeaderInfo]

to_votable(*args, **kwargs)

Module contents