U
|
¬ý°d§@ ã @ sÊ d Z ddlmZ ddlmZmZ ddlZddlmZm Z m
|
Z
|
m Z G dd dej Z G dd d ej ZG d
|
d d ej ZG d d d e ZG dd de ZG dd deZG dd deZG dd deZdS )zX
|
A verbatim copy (vendored) of the spec from https://github.com/data-apis/dataframe-api
|
é )Ú annotations)ÚABCÚabstractmethodN)ÚAnyÚIterableÚSequenceÚ TypedDictc @ s0 e Zd ZdZdZdZdZdZdZdZ dZ
|
d Z d
|
S ) ÚDlpackDeviceTypez3Integer enum for device type codes matching DLPack.é é é é é é é é
|
N) Ú__name__Ú
|
__module__Ú __qualname__Ú__doc__ZCPUZCUDAZ
|
CPU_PINNEDZOPENCLZVULKANZMETALZVPIZROCM© r r úad:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\pandas/core/interchange/dataframe_protocol.pyr s r c @ s, e Zd ZdZdZdZdZdZdZdZ dZ
|
d S )
|
Ú DtypeKindað
|
Integer enum for data types.
|
|
Attributes
|
----------
|
INT : int
|
Matches to signed integer data type.
|
UINT : int
|
Matches to unsigned integer data type.
|
FLOAT : int
|
Matches to floating point data type.
|
BOOL : int
|
Matches to boolean data type.
|
STRING : int
|
Matches to string data type (UTF-8 encoded).
|
DATETIME : int
|
Matches to datetime data type.
|
CATEGORICAL : int
|
Matches to categorical data type.
|
r r
|
r é é é é N) r r r r ÚINTÚUINTÚFLOATÚBOOLÚSTRINGZDATETIMEZ CATEGORICALr r r r r ! s r c @ s$ e Zd ZdZdZdZdZdZdZdS )ÚColumnNullTypea¼
|
Integer enum for null type representation.
|
|
Attributes
|
----------
|
NON_NULLABLE : int
|
Non-nullable column.
|
USE_NAN : int
|
Use explicit float NaN value.
|
USE_SENTINEL : int
|
Sentinel value besides NaN/NaT.
|
USE_BITMASK : int
|
The bit is set/unset representing a null on a certain position.
|
USE_BYTEMASK : int
|
The byte is set/unset representing a null on a certain position.
|
r r
|
r r r N) r r r r Z NON_NULLABLEZUSE_NANZ USE_SENTINELZ USE_BITMASKZ USE_BYTEMASKr r r r r" @ s r" c @ s& e Zd ZU ded< ded< ded< dS )Ú ColumnBuffersztuple[Buffer, Any]Údataztuple[Buffer, Any] | NoneZvalidityÚoffsetsN©r r r Ú__annotations__r r r r r# Y s
|
r# c @ s& e Zd ZU ded< ded< ded< dS )ÚCategoricalDescriptionÚboolZ
|
is_orderedZ is_dictionaryz Column | NoneÚ
|
categoriesNr& r r r r r( j s
|
r( c @ sZ e Zd ZdZeeddddZeeddddZedd Zed
|
dd d Z d S )ÚBufferaY
|
Data in the buffer is guaranteed to be contiguous in memory.
|
|
Note that there is no dtype attribute present, a buffer can be thought of
|
as simply a block of memory. However, if the column that the buffer is
|
attached to has a dtype that's supported by DLPack and ``__dlpack__`` is
|
implemented, then that dtype information will be contained in the return
|
value from ``__dlpack__``.
|
|
This distinction is useful to support both data exchange via DLPack on a
|
buffer and (b) dtypes like variable-length strings which do not have a
|
fixed number of bytes per element.
|
Úint©Úreturnc C s dS )z'
|
Buffer size in bytes.
|
Nr ©Úselfr r r Úbufsize s zBuffer.bufsizec C s dS )z?
|
Pointer to start of the buffer as an integer.
|
Nr r/ r r r Úptr s z
|
Buffer.ptrc C s t ddS )a~
|
Produce DLPack capsule (see array API standard).
|
|
Raises:
|
|
- TypeError : if the buffer contains unsupported dtypes.
|
- NotImplementedError : if DLPack support is not implemented
|
|
Useful to have to connect to array libraries. Support optional because
|
it's not completely trivial to implement for a Python-only library.
|
Ú
|
__dlpack__N)ÚNotImplementedErrorr/ r r r r3 s zBuffer.__dlpack__z#tuple[DlpackDeviceType, int | None]c C s dS )zÆ
|
Device type and device ID for where the data in the buffer resides.
|
Uses device type codes matching DLPack.
|
Note: must be implemented even if ``__dlpack__`` is not.
|
Nr r/ r r r Ú__dlpack_device__ s zBuffer.__dlpack_device__N)
|
r r r r Úpropertyr r1 r2 r3 r5 r r r r r+ t s
|
r+ c @ sà e Zd ZdZeddddZeeddddZeeddd d
|
Zeed dd d Z eeddddZ
|
eeddddZ eeddddZ eddddZ ed!dddd d Zed dd d ZdS )"ÚColumna¦
|
A column object, with only the methods and properties required by the
|
interchange protocol defined.
|
|
A column can contain one or more chunks. Each chunk can contain up to three
|
buffers - a data buffer, a mask buffer (depending on null representation),
|
and an offsets buffer (if variable-size binary; e.g., variable-length
|
strings).
|
|
TBD: Arrow has a separate "null" dtype, and has no separate mask concept.
|
Instead, it seems to use "children" for both columns with a bit mask,
|
and for nested dtypes. Unclear whether this is elegant or confusing.
|
This design requires checking the null representation explicitly.
|
|
The Arrow design requires checking:
|
1. the ARROW_FLAG_NULLABLE (for sentinel values)
|
2. if a column has two children, combined with one of those children
|
having a null dtype.
|
|
Making the mask concept explicit seems useful. One null dtype would
|
not be enough to cover both bit and byte masks, so that would mean
|
even more checking if we did it the Arrow way.
|
|
TBD: there's also the "chunk" concept here, which is implicit in Arrow as
|
multiple buffers per array (= column here). Semantically it may make
|
sense to have both: chunks were meant for example for lazy evaluation
|
of data which doesn't fit in memory, while multiple buffers per column
|
could also come from doing a selection operation on a single
|
contiguous buffer.
|
|
Given these concepts, one would expect chunks to be all of the same
|
size (say a 10,000 row dataframe could have 10 chunks of 1,000 rows),
|
while multiple buffers could have data-dependent lengths. Not an issue
|
in pandas if one column is backed by a single NumPy array, but in
|
Arrow it seems possible.
|
Are multiple chunks *and* multiple buffers per column necessary for
|
the purposes of this interchange protocol, or must producers either
|
reuse the chunk concept for this or copy the data?
|
|
Note: this Column object can only be produced by ``__dataframe__``, so
|
doesn't need its own version or ``__column__`` protocol.
|
r, r- c C s dS )z³
|
Size of the column, in elements.
|
|
Corresponds to DataFrame.num_rows() if column is a single chunk;
|
equal to size of this current chunk otherwise.
|
Nr r/ r r r ÚsizeÕ s z Column.sizec C s dS )zâ
|
Offset of first element.
|
|
May be > 0 if using chunks; for example for a column with N chunks of
|
equal size M (only the last chunk may be shorter),
|
``offset = n * M``, ``n = 0 .. N-1``.
|
Nr r/ r r r ÚoffsetÞ s z Column.offsetz tuple[DtypeKind, int, str, str]c C s dS )aú
|
Dtype description as a tuple ``(kind, bit-width, format string, endianness)``.
|
|
Bit-width : the number of bits as an integer
|
Format string : data type description format string in Apache Arrow C
|
Data Interface format.
|
Endianness : current only native endianness (``=``) is supported
|
|
Notes:
|
- Kind specifiers are aligned with DLPack where possible (hence the
|
jump to 20, leave enough room for future extension)
|
- Masks must be specified as boolean with either bit width 1 (for bit
|
masks) or 8 (for byte masks).
|
- Dtype width in bits was preferred over bytes
|
- Endianness isn't too useful, but included now in case in the future
|
we need to support non-native endianness
|
- Went with Apache Arrow format strings over NumPy format strings
|
because they're more complete from a dataframe perspective
|
- Format strings are mostly useful for datetime specification, and
|
for categoricals.
|
- For categoricals, the format string describes the type of the
|
categorical in the data buffer. In case of a separate encoding of
|
the categorical (e.g. an integer to string mapping), this can
|
be derived from ``self.describe_categorical``.
|
- Data types not included: complex, Arrow-style null, binary, decimal,
|
and nested (list, struct, map, union) dtypes.
|
Nr r/ r r r Údtypeé s z Column.dtyper( c C s dS )a©
|
If the dtype is categorical, there are two options:
|
- There are only values in the data buffer.
|
- There is a separate non-categorical Column encoding for categorical values.
|
|
Raises TypeError if the dtype is not categorical
|
|
Returns the dictionary with description on how to interpret the data buffer:
|
- "is_ordered" : bool, whether the ordering of dictionary indices is
|
semantically meaningful.
|
- "is_dictionary" : bool, whether a mapping of
|
categorical values to other objects exists
|
- "categories" : Column representing the (implicit) mapping of indices to
|
category values (e.g. an array of cat1, cat2, ...).
|
None if not a dictionary-style categorical.
|
|
TBD: are there any other in-memory representations that are needed?
|
Nr r/ r r r Údescribe_categorical s zColumn.describe_categoricalztuple[ColumnNullType, Any]c C s dS )a7
|
Return the missing value (or "null") representation the column dtype
|
uses, as a tuple ``(kind, value)``.
|
|
Value : if kind is "sentinel value", the actual value. If kind is a bit
|
mask or a byte mask, the value (0 or 1) indicating a missing value. None
|
otherwise.
|
Nr r/ r r r Ú describe_null s zColumn.describe_nullú
|
int | Nonec C s dS )z
|
Number of null elements, if known.
|
|
Note: Arrow uses -1 to indicate "unknown", but None seems cleaner.
|
Nr r/ r r r Ú
|
null_count* s zColumn.null_countúdict[str, Any]c C s dS )zY
|
The metadata for the column. See `DataFrame.metadata` for more details.
|
Nr r/ r r r Úmetadata3 s zColumn.metadatac C s dS )zE
|
Return the number of chunks the column consists of.
|
Nr r/ r r r Ú
|
num_chunks: s zColumn.num_chunksNúIterable[Column]©Ún_chunksr. c C s dS )zz
|
Return an iterator yielding the chunks.
|
|
See `DataFrame.get_chunks` for details on ``n_chunks``.
|
Nr ©r0 rD r r r Ú
|
get_chunks@ s zColumn.get_chunksr# c C s dS )ab
|
Return a dictionary containing the underlying buffers.
|
|
The returned dictionary has the following contents:
|
|
- "data": a two-element tuple whose first element is a buffer
|
containing the data and whose second element is the data
|
buffer's associated dtype.
|
- "validity": a two-element tuple whose first element is a buffer
|
containing mask values indicating missing data and
|
whose second element is the mask value buffer's
|
associated dtype. None if the null representation is
|
not a bit or byte mask.
|
- "offsets": a two-element tuple whose first element is a buffer
|
containing the offset values for variable-size binary
|
data (e.g., variable-length strings) and whose second
|
element is the offsets buffer's associated dtype. None
|
if the data buffer does not have an associated offsets
|
buffer.
|
Nr r/ r r r Ú get_buffersH s zColumn.get_buffers)N)r r r r r r8 r6 r9 r: r; r< r> r@ rA rF rG r r r r r7 © s6 +
|
r7 c @ s e Zd ZdZdZed0dddddZeed d
|
d d Zed d
|
ddZ edd
|
ddZ
|
ed d
|
ddZ edd
|
ddZ ed ddddZ ed dd d d Zed d
|
d!d"Zed#d d$d%d&Zed'd d(d)d*Zed1dd,d-d.d/Zd+S )2Ú DataFramea2
|
A data frame class, with only the methods required by the interchange
|
protocol defined.
|
|
A "data frame" represents an ordered collection of named columns.
|
A column's "name" must be a unique string.
|
Columns may be accessed by name or by position.
|
|
This could be a public data frame class, or an object with the methods and
|
attributes defined on this DataFrame class could be returned from the
|
``__dataframe__`` method of a public data frame class in a library adhering
|
to the dataframe interchange protocol specification.
|
r FTr) )Ú nan_as_nullÚ
|
allow_copyc C s dS )zHConstruct a new interchange object, potentially changing the parameters.Nr )r0 rI rJ r r r Ú __dataframe__y s zDataFrame.__dataframe__r? r- c C s dS )a!
|
The metadata for the data frame, as a dictionary with string keys. The
|
contents of `metadata` may be anything, they are meant for a library
|
to store information that it needs to, e.g., roundtrip losslessly or
|
for two implementations to share data that is not (yet) part of the
|
interchange protocol specification. For avoiding collisions with other
|
entries, please add name the keys with the name of the library
|
followed by a period and the desired name, e.g, ``pandas.indexcol``.
|
Nr r/ r r r r@ } s zDataFrame.metadatar, c C s dS )z@
|
Return the number of columns in the DataFrame.
|
Nr r/ r r r Ú num_columns s zDataFrame.num_columnsr= c C s dS )zK
|
Return the number of rows in the DataFrame, if available.
|
Nr r/ r r r Únum_rows s zDataFrame.num_rowsc C s dS )zH
|
Return the number of chunks the DataFrame consists of.
|
Nr r/ r r r rA s zDataFrame.num_chunksz Iterable[str]c C s dS )z?
|
Return an iterator yielding the column names.
|
Nr r/ r r r Ú column_names s zDataFrame.column_namesr7 )Úir. c C s dS )z>
|
Return the column at the indicated position.
|
Nr )r0 rO r r r Ú
|
get_column¥ s zDataFrame.get_columnÚstr)Únamer. c C s dS )zE
|
Return the column whose name is the indicated name.
|
Nr )r0 rR r r r Úget_column_by_name« s z DataFrame.get_column_by_namerB c C s dS )z:
|
Return an iterator yielding the columns.
|
Nr r/ r r r Ú get_columns± s zDataFrame.get_columnsz Sequence[int])Úindicesr. c C s dS )zS
|
Create a new DataFrame by selecting a subset of columns by index.
|
Nr )r0 rU r r r Úselect_columns· s zDataFrame.select_columnsz Sequence[str])Únamesr. c C s dS )zR
|
Create a new DataFrame by selecting a subset of columns by name.
|
Nr )r0 rW r r r Úselect_columns_by_name½ s z DataFrame.select_columns_by_nameNzIterable[DataFrame]rC c C s dS )a2
|
Return an iterator yielding the chunks.
|
|
By default (None), yields the chunks that the data is stored as by the
|
producer. If given, ``n_chunks`` must be a multiple of
|
``self.num_chunks()``, meaning the producer must subdivide each chunk
|
before yielding it.
|
Nr rE r r r rF Ã s zDataFrame.get_chunks)FT)N)r r r r Úversionr rK r6 r@ rL rM rA rN rP rS rT rV rX rF r r r r rH h s6 rH )r Ú
|
__future__r Úabcr r ÚenumÚtypingr r r r ÚIntEnumr r r" r# r( r+ r7 rH r r r r Ú<module> s
|
5 @
|
