U
|
¬ý°d9Y ã @ s\ d Z ddlmZ ddlmZ ddlmZmZmZm Z m
|
Z
|
m Z m Z ddl ZddlmZmZmZ ddlmZ ddlmZmZmZmZmZmZm Z dd l m Z m Z m Z dd
|
l!m"Z"m#Z# dd l$m%Z% dd l&m'Z' erðdd l(m)Z) ddl*m+Z+ ddl,m-Z- dddddddddddZ.dddd d d d Z/dd d!d"d#Z0ddd$d%d&Z1d'dd(d)d*d+Z2d'd'dd,dd(d-d.d/Z3djddd'd1d2d3Z4dkd5d6d'd7d8d9Z5dld5dd5d6d;d'd<d=d>Z6dmd?d5d@dAdBdCZ7dDdEdFdGdHdIZ8dndJdKdJdLdMdNZ9dod6dOdPdQZ:d'dFdRdSdTdUdVdWZ;dXdYdZd[d\d]Z<dpd'd^d'd_d`daZ=dqd ddbdcdddeZ>d d'dfdgdhdiZ?dS )rz+ miscellaneous sorting / groupby utilities é )Ú annotations)Ú defaultdict)Ú TYPE_CHECKINGÚCallableÚ DefaultDictÚHashableÚIterableÚSequenceÚcastN)ÚalgosÚ hashtableÚlib)Úunique_label_indices)ÚAxisIntÚ IndexKeyFuncÚLevelÚ
|
NaPositionÚShapeÚSortKindÚnpt)Ú ensure_int64Úensure_platform_intÚis_extension_array_dtype)Ú ABCMultiIndexÚ ABCRangeIndex)Úisna)Ú extract_array)Ú
|
MultiIndex)ÚExtensionArray©ÚIndexr zLevel | list[Level] | Nonezlist[bool] | boolr r Úboolr znpt.NDArray[np.intp] | None)ÚtargetÚlevelÚ ascendingÚkindÚ na_positionÚsort_remainingÚkeyÚreturnc C s t | ||d} | ¡ } |dk r4| j|||d\}}nLt| trRt| ¡ ||d}n.|r\| jsf|sj| jrjdS t | |t
|
t ||d}|S )aU
|
Helper method that return the indexer according to input parameters for
|
the sort_index method of DataFrame and Series.
|
|
Parameters
|
----------
|
target : Index
|
level : int or level name or list of ints or list of level names
|
ascending : bool or list of bools, default True
|
kind : {'quicksort', 'mergesort', 'heapsort', 'stable'}, default 'quicksort'
|
na_position : {'first', 'last'}, default 'last'
|
sort_remaining : bool, default True
|
key : callable, optional
|
|
Returns
|
-------
|
Optional[ndarray[intp]]
|
The indexer for the new index.
|
)ÚlevelsN)r$ r' )Úordersr& )r% r$ r& ) Úensure_key_mappedZ_sort_levels_monotonicZ sortlevelÚ
|
isinstancer Úlexsort_indexerZ_get_codes_for_sortingZis_monotonic_increasingZis_monotonic_decreasingÚnargsortr
|
r! ) r" r# r$ r% r& r' r( Ú_Úindexer© r2 úJd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\pandas/core/sorting.pyÚget_indexer_indexer4 s6 ÿ
|
ÿ
|
ÿÿür4 r znpt.NDArray[np.int64])ÚshapeÚsortÚxnullr) c C s| dddd}dddd}dd | D } t |}|sptt| |D ]*\}\}} ||| \}} || |< | ||< qDt | } ||}
|
tj|d
|
|
|
d d } | | d jd ddd } td
|
|
|
D ]8}|| d krÔt d ¡} n | || } | | | | 7 } q¸|r,| d dk} | d
|
|
|
D ]}| |dkO } qd| | < |
|
t|kr>qxt | |d\}}|g| |
|
d
} t|g||
|
d
}qx| S )a{
|
For the particular label_list, gets the offsets into the hypothetical list
|
representing the totally ordered cartesian product of all possible label
|
combinations, *as long as* this space fits within int64 bounds;
|
otherwise, though group indices identify unique combinations of
|
labels, they cannot be deconstructed.
|
- If `sort`, rank of returned ids preserve lexical ranks of labels.
|
i.e. returned id's can be used to do lexical sort on labels;
|
- If `xnull` nulls (-1 labels) are passed through.
|
|
Parameters
|
----------
|
labels : sequence of arrays
|
Integers identifying levels at each location
|
shape : tuple[int, ...]
|
Number of unique levels at each location
|
sort : bool
|
If the ranks of returned ids should match lexical ranks of labels
|
xnull : bool
|
If true nulls are excluded. i.e. -1 values in the labels are
|
passed through.
|
|
Returns
|
-------
|
An array of type int64 where two elements are equal if their corresponding
|
labels are equal at all location.
|
|
Notes
|
-----
|
The length of `labels` and `shape` must be identical.
|
Úint)r) c S s< d}t | D ]&\}}|t|9 }|tjk s | S q t| S ©Né )Ú enumerater8 r Úi8maxÚlen)r5 ÚaccÚiÚmulr2 r2 r3 Ú_int64_cut_off s
|
|
z'get_group_index.<locals>._int64_cut_offztuple[np.ndarray, int]c S s$ | dk ¡ r | d |d fS | |fS )Néÿÿÿÿr: ©Úany)ÚlabÚsizer2 r2 r3 Ú
|
maybe_lift s z#get_group_index.<locals>.maybe_liftc S s g | ] }t |qS r2 )r ©Ú.0Úxr2 r2 r3 Ú
|
<listcomp> s z#get_group_index.<locals>.<listcomp>r: Úi8©Zdtyper F©ZsubokÚcopyrB ©r6 N)
|
Úlistr; ÚzipÚnpÚprodÚastypeÚrangeÚint64r= Úcompress_group_index)Úlabelsr5 r6 r7 rA rG Zlshaper? rE rF ZnlevZstrideÚoutÚmaskÚcomp_idsÚobs_idsr2 r2 r3 Úget_group_indexm s: #
|
r^ z2tuple[npt.NDArray[np.intp], npt.NDArray[np.int64]])Úsizesr) c C s t | |ddd}t|ddS )a¾
|
Group_index is offsets into cartesian product of all possible labels. This
|
space can be huge, so this function compresses it, by computing offsets
|
(comp_ids) into the list of unique labels (obs_group_ids).
|
|
Parameters
|
----------
|
labels : list of label arrays
|
sizes : tuple[int] of size of the levels
|
|
Returns
|
-------
|
np.ndarray[np.intp]
|
comp_ids
|
np.ndarray[np.int64]
|
obs_group_ids
|
TF©r6 r7 rP )r^ rX )rY r_ Úidsr2 r2 r3 Úget_compressed_idsË s rb )r5 r) c C s$ d}| D ]}|t |9 }q|tjkS r9 )r8 r r< )r5 Zthe_prodrJ r2 r2 r3 Úis_int64_overflow_possibleã s rc znpt.NDArray[np.intp]zlist[npt.NDArray[np.intp]])Ú comp_labelsr5 r) c C s t |rtdg }d}t d¡}| }ttt|D ]L}|| ||| | }t || dk d¡ | |¡ || }||| 9 }q6|d d d
S )Nz,cannot deconstruct factorized group indices!r: r rB ) rc Ú
|
ValueErrorrS ÚarrayÚreversedrV r= ÚputmaskÚappend)rd r5 Ú
|
label_listZfactorÚyrJ r? rY r2 r2 r3 Ú_decons_group_indexë s
|
|
rl z'Sequence[npt.NDArray[np.signedinteger]])r\ r] r5 rY r7 r) c s |s:t jdd |D t jd}t j|t jd| }t|}t|spt||}|sX| ¡ s\|S dd t||D S t | fdd|D S )a6
|
Reconstruct labels from observed group ids.
|
|
Parameters
|
----------
|
comp_ids : np.ndarray[np.intp]
|
obs_ids: np.ndarray[np.intp]
|
shape : tuple[int]
|
labels : Sequence[np.ndarray[np.signedinteger]]
|
xnull : bool
|
If nulls are excluded; i.e. -1 labels are passed through.
|
c s s | ]}|d k ¡ V qdS )rB NrC )rI Úar2 r2 r3 Ú <genexpr> s z'decons_obs_group_ids.<locals>.<genexpr>rM c S s g | ]\}}|| qS r2 r2 )rI rJ rk r2 r2 r3 rK s z(decons_obs_group_ids.<locals>.<listcomp>c s" g | ]}| j tjd ddqS )FTrN )rU rS Úintp©rI rE ©r1 r2 r3 rK s )
|
rS Zfromiterro ÚasarrayÚtuplerc rl rD rR r )r\ r] r5 rY r7 ZliftZ arr_shaperZ r2 rq r3 Údecons_obs_group_ids s
|
$rt T)r5 Úcompressr) c C sJ t | |ddd}|s(|jo | ¡ d }nt|dd\}}t|}t||S )NTFr` r: rP )r^ rF ÚmaxrX r= Úget_group_index_sorter)rY r5 ru ra ÚngroupsZobsr2 r2 r3 Úindexer_from_factorized" s ry ÚlastÚstrzCallable | None)r& r( r) c sH ddl m} g }g }t|tr.|gt| }n|dkrDdgt| } fdd| D } t| |D ]Ø\}}||dd} |dkrtd | t| j}
|
| j ¡ } | jd
|
k} |rÖ|d krÄt
|
| |
|
| ¡} n|d krÔ| d 7 } n<|d kröt
|
| |
|
|
|
| d ¡} n |d krt
|
| d|
|
| ¡} | ¡ r$|
|
d 7 }
|
| |
|
¡ | | ¡ q`t|t|S )aL
|
Performs lexical sorting on a set of keys
|
|
Parameters
|
----------
|
keys : sequence of arrays
|
Sequence of ndarrays to be sorted by the indexer
|
orders : bool or list of booleans, optional
|
Determines the sorting order for each element in keys. If a list,
|
it must be the same length as keys. This determines whether the
|
corresponding element in keys should be sorted in ascending
|
(True) or descending (False) order. if bool, applied to all
|
elements as above. if None, defaults to True.
|
na_position : {'first', 'last'}, default 'last'
|
Determines placement of NA elements in the sorted list ("last" or "first")
|
key : Callable, optional
|
Callable key function applied to every element in keys before sorting
|
|
Returns
|
-------
|
np.ndarray[np.intp]
|
r )Ú CategoricalNTc s g | ]}t | qS r2 )r, )rI Úk©r( r2 r3 rK R s z#lexsort_indexer.<locals>.<listcomp>)Zordered)rz Úfirstúinvalid na_position: rB rz r r: )Úpandas.core.arraysr| r- r! r= rR re Ú
|
categoriesÚcodesrO rS ÚwhererD ri ry rs ) Úkeysr+ r& r( r| rY r5 r} ÚorderÚcatÚnr r[ r2 r~ r3 r. 0 s:
|
|
|
|
|
|
|
|
r. Ú quicksortz npt.NDArray[np.bool_] | None)r% r$ r& r( r[ r) c C sB |dk r&t | |} t| |||d|dS t| tr<| j|dS t| tsNt| } |dkrdt t | ¡}t
|
| r|| j|||dS t | ¡} t t | ¡}| | }|| }t |¡d } |sÖ|ddd
}|ddd
}||j|d }
|
|sø|
|
ddd
}
|
|dkrt |
|
| g¡}
|
n(|d kr,t | |
|
g¡}
|
ntd
|
| t|
|
S ) a
|
Intended to be a drop-in replacement for np.argsort which handles NaNs.
|
|
Adds ascending, na_position, and key parameters.
|
|
(GH #6399, #5231, #27237)
|
|
Parameters
|
----------
|
kind : str, default 'quicksort'
|
ascending : bool, default True
|
na_position : {'first', 'last'}, default 'last'
|
key : Optional[Callable], default None
|
mask : Optional[np.ndarray[bool]], default None
|
Passed when called by ExtensionArray.argsort.
|
|
Returns
|
-------
|
np.ndarray[np.intp]
|
N)r% r$ r& r( r[ )r$ )r$ r% r& r rB ©r% rz r r )r, r/ r- r Úargsortr r rS rr r r Z
|
asanyarrayÚaranger= ZnonzeroZ concatenatere r ) Úitemsr% r$ r& r( r[ ÚidxÚnon_nansÚ non_nan_idxZnan_idxr1 r2 r2 r3 r/ q sF
|
ú
|
|
|
|
|
|
|
r/ r r )ÚvaluesÚmethodÚaxisc s |dks t |dkrtjntj t t| ¡}| ¡ }|jdkr| ¡ r|dkr\t ||}nt |j
|
|j
|
}t fdd|D ¡S ||dS t || S )a
|
Implementation of np.argmin/argmax but for ExtensionArray and which
|
handles missing values.
|
|
Parameters
|
----------
|
values : ExtensionArray
|
method : {"argmax", "argmin"}
|
axis : int, default 0
|
|
Returns
|
-------
|
int
|
> ÚargminÚargmaxr r: c s g | ]\}}t || qS r2 )Ú _nanargminmax)rI ÚvÚm©Úfuncr2 r3 rK Ö s z nargminmax.<locals>.<listcomp>)r ) ÚAssertionErrorrS r r rr r Z_values_for_argsortÚndimrD rR ÚTrf r )r r r r[ Z
|
arr_valuesZzippedr2 r r3 Ú
|
nargminmax» s
|
r z
|
np.ndarrayznpt.NDArray[np.bool_]r8 )r r[ r) c C s0 t | jd ¡}| | }|| }||| S )z#
|
See nanargminmax.__doc__.
|
r )rS r r5 )r r[ r r r r r2 r2 r3 r Ü s
|
|
r r r )Úindexr( r) c sn |dk r6t |ttfr |gn| fddD ntt j fddt jD }t |¡S )ad
|
Returns a new MultiIndex in which key has been applied
|
to all levels specified in level (or all levels if level
|
is None). Used for key sorting for MultiIndex.
|
|
Parameters
|
----------
|
index : MultiIndex
|
Index to which to apply the key function on the
|
specified levels.
|
key : Callable
|
Function that takes an Index and returns an Index of
|
the same shape. This key is applied to each level
|
separately. The name of the level can be used to
|
distinguish different levels for application.
|
level : list-like, int or str, default None
|
Level or list of levels to apply the key function to.
|
If None, key function is applied to all levels. Other
|
levels are left unchanged.
|
|
Returns
|
-------
|
labels : MultiIndex
|
Resulting MultiIndex with modified levels.
|
Nc s g | ]} |¡qS r2 )Z_get_level_number)rI Zlev)r r2 r3 rK
|
s z1_ensure_key_mapped_multiindex.<locals>.<listcomp>c s. g | ]&}|kr t |¡n |¡qS r2 )r, Z_get_level_values)rI r# ©r r( Z sort_levelsr2 r3 rK s þÿ)r- r{ r8 rQ rV ZnlevelsÚtypeZ from_arrays)r r( r# Zmappedr2 r r3 Ú _ensure_key_mapped_multiindexç s ür¢ r~ c C s° ddl m} |s| S t| tr,t| ||dS || ¡ }t|t| krPtdz(t| |rf||}nt| }||}W n2 t k
|
rª t dt| dt| dY nX |S )aÁ
|
Applies a callable key function to the values function and checks
|
that the resulting value has the same shape. Can be called on Index
|
subclasses, Series, DataFrames, or ndarrays.
|
|
Parameters
|
----------
|
values : Series, DataFrame, Index subclass, or ndarray
|
key : Optional[Callable], key to be called on the values array
|
levels : Optional[List], if values is a MultiIndex, list of levels to
|
apply the key to.
|
r r )r# zDUser-provided `key` function must not change the shape of the array.z6User-provided `key` function returned an invalid type z- which could not be converted to Ú.)
|
Zpandas.core.indexes.apir r- r r¢ rO r= re r¡ Ú TypeError)r r( r* r ÚresultZtype_of_valuesr2 r2 r3 r, s2
|
ÿ ÿ
|
ÿÿ
|
r, zIterable[Index]zIterable[np.ndarray]z list[tuple])r\ rx r* rY r) c C s | j tjdd} tt}t||D ]P\}}t |¡}| | |j tjdd¡ t |D ] }||
|
|| |¡ ¡ qTq"dd | ¡ D S )z%Map compressed group id -> key tuple.F)rO c S s g | ] }t |qS r2 )rs )rI rf r2 r2 r3 rK R s z&get_flattened_list.<locals>.<listcomp>) rU rS rW r rQ rR r ÚInt64HashTableZmap_keys_to_valuesrV ri Zget_itemr ) r\ rx r* rY ZarraysZlabsr# Útabler? r2 r2 r3 Úget_flattened_listD s
|
r¨ zlist[np.ndarray]z list[Index]z$dict[Hashable, npt.NDArray[np.intp]])rj r
r) c s t dd |D }t| |ddd}t |dk¡r4i S t|rN|joH| ¡ d n tj|dd}t|| fd d
|
| D }| ¡}t
|
|||¡S ) zJ
|
Returns
|
-------
|
dict:
|
Labels mapped to indexers.
|
c s s | ]}t |V qd S )N)r= rH r2 r2 r3 rn ^ s z#get_indexer_dict.<locals>.<genexpr>Tr` rB r: rL rM c s g | ]}| ¡qS r2 )Útakerp ©Úsorterr2 r3 rK l s z$get_indexer_dict.<locals>.<listcomp>) rs r^ rS Úallrc rF rv rT rw r© r Z indices_fast)rj r
r5 Ú group_indexrx Z sorted_labelsr2 rª r3 Úget_indexer_dictU s ÿ ý
|
|
r® z
|
int | None)r rx r) c C st |dkrd| ¡ }t| }d}d}|dkoD||| |t |¡ k }|r`t t| |¡\}}n | jdd}t|S )a
|
algos.groupsort_indexer implements `counting sort` and it is at least
|
O(ngroups), where
|
ngroups = prod(shape)
|
shape = map(len, keys)
|
that is, linear in the number of combinations (cartesian product) of unique
|
values of groupby keys. This can be huge when doing multi-key groupby.
|
np.argsort(kind='mergesort') is O(count x log(count)) where count is the
|
length of the data-frame;
|
Both algorithms are `stable` sort and that is necessary for correctness of
|
groupby operations. e.g. consider:
|
df.groupby(key)[col].transform('first')
|
|
Parameters
|
----------
|
group_index : np.ndarray[np.intp]
|
signed integer dtype
|
ngroups : int or None, default None
|
|
Returns
|
-------
|
np.ndarray[np.intp]
|
Nr: g g ð?r Z mergesortr )rv r= rS Úlogr Zgroupsort_indexerr r )r rx ÚcountÚalphaÚbetaZ do_groupsortr« r0 r2 r2 r3 rw v s "þ
|
rw z3tuple[npt.NDArray[np.int64], npt.NDArray[np.int64]])r r6 r) c C sV t | }t |¡}t| } | | ¡\}}|rFt |dkrFt||\}}t|t|fS )zß
|
Group_index is offsets into cartesian product of all possible labels. This
|
space can be huge, so this function compresses it, by computing offsets
|
(comp_ids) into the list of unique labels (obs_group_ids).
|
r )r= r r¦ r Zget_labels_groupbyÚ_reorder_by_uniques)r r6 Z size_hintr§ r\ Z obs_group_idsr2 r2 r3 rX ¡ s
|
rX z2tuple[npt.NDArray[np.int64], npt.NDArray[np.intp]])ÚuniquesrY r) c C sd | ¡ }tjt|tjd}| |t t|¡¡ |dk }| |¡}t ||d¡ | |¡} | |fS )z±
|
Parameters
|
----------
|
uniques : np.ndarray[np.int64]
|
labels : np.ndarray[np.intp]
|
|
Returns
|
-------
|
np.ndarray[np.int64]
|
np.ndarray[np.intp]
|
rM r rB ) r rS Úemptyr= ro Úputr r© rh )r´ rY r« Zreverse_indexerr[ r2 r2 r3 r³ · s
|
|
r³ )T)Nrz N)r Trz NN)r )N)N)N)T)@Ú__doc__Ú
|
__future__r Ú collectionsr Útypingr r r r r r r
|
ÚnumpyrS Z pandas._libsr r r Zpandas._libs.hashtabler Zpandas._typingr r r r r r r Zpandas.core.dtypes.commonr r r Zpandas.core.dtypes.genericr r Zpandas.core.dtypes.missingr Zpandas.core.constructionr Zpandasr r r Zpandas.core.indexes.baser r4 r^ rb rc rl rt ry r. r/ r r r¢ r, r¨ r® rw rX r³ r2 r2 r2 r3 Ú<module> sZ $
|
$
|
9^" ÿ ÿC úJ! ÿ1," ÿ, ÿ
|
