U
|
¬ý°dêj ã @ sL d dl mZ d dlZd dlZd dlmZmZmZ d dl Z
|
d dl m Z m Z d dlmZ d dlmZmZmZmZmZ d dlmZ d dlmZ d d lm Z m Z d d
|
l m Z m Z m!Z!m"Z" d d l#m$Z$ d d l%m&Z&m'Z' d d l(m)Z)m*Z*m+Z+ d dl,m- m.Z/ d dl0m1Z1m2Z2 d dl3m4Z4 d dl5m6Z6 d dl7m8Z8m9Z9 d dl:m;Z;m<Z< d dl=m>Z> erdd dl?m@Z@ d dl:mAZA ddddddd ZBd d d d d!ZCdddddd"d#ZDddd$d%d&d'ZEdd(d(d)d*d+ZFd$d,d-d.d/ZGG d0d1 d1ZHd2dd d3d4d5ZId6dd7d8d9ZJd:d6d;d<d=ZKd2dd;d>d?ZLdd@dAdBZMd1dCd1dDdEdFZNdGdH ZOdS )Ié )Ú annotationsN)Ú TYPE_CHECKINGÚSequenceÚcast)ÚNaTÚ internals)ÚNA)Ú ArrayLikeÚAxisIntÚDtypeObjÚManagerÚShape)Úcache_readonly)Ú astype_array)Úensure_dtype_can_hold_naÚfind_common_type)Úis_1d_only_ea_dtypeÚis_dtype_equalÚ is_scalarÚneeds_i8_conversion)Ú concat_compat)ÚDatetimeTZDtypeÚExtensionDtype)Úis_valid_na_for_dtypeÚisnaÚisna_all)Ú DatetimeArrayÚExtensionArray)Ú SparseDtype)Ú ensure_wrapped_if_datetimelike)Ú ArrayManagerÚNullArrayProxy)Úensure_block_shapeÚ new_block_2d)Ú BlockManager)ÚIndex)ÚBlockz list[Index]r
|
Úboolr )ÚaxesÚ concat_axisÚcopyÚreturnc sä g | D ]l\}}d}| ¡ D ]4\}}|j|| ||ddd}|dkr |dk r d}q |rj|dkrj|sj| ¡ } |¡ q|dkr fddtt d jD } n&|dks¬ttt j
|
d d D ¡} t | |d |d gdd
|
}
|
|
|
S ) zý
|
Concatenate array managers into one.
|
|
Parameters
|
----------
|
mgrs_indexers : list of (ArrayManager, {axis: indexer,...}) tuples
|
axes : list of Index
|
concat_axis : int
|
copy : bool
|
|
Returns
|
-------
|
ArrayManager
|
FT)ÚaxisÚ
|
allow_dupsÚ use_na_proxyé Nr c s, g | ]$ t fd dttD qS )c s g | ]}| j qS © ©Úarrays©Ú.0Úi)ÚjÚmgrsr0 úSd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\pandas/core/internals/concat.pyÚ
|
<listcomp>k s z:_concatenate_array_managers.<locals>.<listcomp>.<listcomp>)Ú concat_arraysÚrangeÚlen)r4 ©r7 )r6 r8 r9 j s ÿz/_concatenate_array_managers.<locals>.<listcomp>c S s g | ]
|
}|j qS r0 r1 )r4 Úmgrr0 r0 r8 r9 q s )Zverify_integrity) ÚitemsÚreindex_indexerr* Úappendr; r< r2 ÚAssertionErrorÚlistÚ itertoolsÚchainÚ from_iterabler ) Ú mgrs_indexersr( r) r* r> ÚindexersZaxis1_made_copyÚaxÚindexerr2 Znew_mgrr0 r= r8 Ú_concatenate_array_managersG s0 ÿ
|
þ rK rC r )Ú to_concatr+ c sê dd | D }dd |D }t |dk}|r8|d j n6tdd |D r\t t|g ¡ ntd d |D fd
|
d| D } t| d tr¤t| d }| | ¡S t
|
| ¡}t |dkræd d |D }t |dkræd |kræ| t ¡}|S ) aÈ
|
Alternative for concat_compat but specialized for use in the ArrayManager.
|
|
Differences: only deals with 1D arrays (no axis keyword), assumes
|
ensure_wrapped_if_datetimelike and does not skip empty arrays to determine
|
the dtype.
|
In addition ensures that all NullArrayProxies get replaced with actual
|
arrays.
|
|
Parameters
|
----------
|
to_concat : list of arrays
|
|
Returns
|
-------
|
np.ndarray or ExtensionArray
|
c S s g | ]}t |ts|qS r0 )Ú
|
isinstancer! ©r4 Úxr0 r0 r8 r9 s
|
z!concat_arrays.<locals>.<listcomp>c S s h | ]
|
}|j qS r0 ©ÚdtyperN r0 r0 r8 Ú <setcomp> s z concat_arrays.<locals>.<setcomp>r/ r c s s$ | ] }|j d kot|tjV qdS ))r5 ÚuÚbN)ÚkindrM ÚnprQ rN r0 r0 r8 Ú <genexpr> s z concat_arrays.<locals>.<genexpr>c S s g | ]
|
}|j qS r0 rP ©r4 Zarrr0 r0 r8 r9 s c s. g | ]&}t |tr | ¡n t| d dqS )F©r* )rM r! Zto_arrayr rX ©Z target_dtyper0 r8 r9 s þÿ c S s h | ] }|j jqS r0 )rQ rU )r4 Úobjr0 r0 r8 rR ¨ s rT ) r< rQ ÚallrV r rC rM r ÚtypeZ_concat_same_typeÚ concatenateÚastypeÚobject)rL Zto_concat_no_proxyÚdtypesZ single_dtypeÚclsÚresultÚkindsr0 rZ r8 r: w s*
|
ü
|
|
|
r: c C sT t | d d tr t| |||S |dkr4t| ||S t|| } dd | D }t|}g }|D ]è\}}|d } | j}
|
t|dkrª|d jsª|
|
j } |r|
|
¡ } n| ¡ } d} nnt |rdd |D } |
|
j sØtj| dd} nt| dd} t| dd } t| } |
|
j j| jk} nt||d
|
} d } | r.|
|
j| |d }n t| |d }| |¡ q\tt||S ) zý
|
Concatenate block managers into one.
|
|
Parameters
|
----------
|
mgrs_indexers : list of (BlockManager, {axis: indexer,...}) tuples
|
axes : list of Index
|
concat_axis : int
|
copy : bool
|
|
Returns
|
-------
|
BlockManager
|
r c S s g | ]\}}t ||qS r0 )Ú_get_mgr_concatenation_plan)r4 r> rH r0 r0 r8 r9 Ï s z(concatenate_managers.<locals>.<listcomp>r/ Tc S s g | ] }|j jqS r0 )ÚblockÚvalues©r4 Zjur0 r0 r8 r9 á s ©r, é )ÚndimrY F)Ú placement)rM r rK Ú_concat_managers_axis0Ú _maybe_reindex_columns_na_proxyÚ_combine_concat_plansrf r< rH rg r* ÚviewÚ_is_uniform_join_unitsÚ is_extensionrV r^ r r" r rQ Ú_concatenate_join_unitsZmake_block_same_classr# rA r$ Útuple)rG r( r) r* Z concat_plansZ concat_planÚblocksrl Ú
|
join_unitsÚunitÚblkrg ZfastpathÚvalsrT r0 r0 r8 Úconcatenate_managers¯ sD
|
ÿ
|
|
rz r$ )r( r* r+ c sÆ fddt t D }t| dd D }d}g }t|D ]p\}}|| } |jD ]J}
|
| rn|
|
jdd} n |r||
|
¡ } n|
|
td¡} | j |¡| _|
|
| ¡ qX|t|j 7 }qBt t ||} | S ) z
|
concat_managers specialized to concat_axis=0, with reindexing already
|
having been done in _maybe_reindex_columns_na_proxy.
|
c s" i | ]}|t | d dkqS )r/ r )r< r3 ©rG r0 r8 Ú
|
<dictcomp> s z*_concat_managers_axis0.<locals>.<dictcomp>c S s g | ] }|d qS )r r0 rN r0 r0 r8 r9
|
s z*_concat_managers_axis0.<locals>.<listcomp>r F)ÚdeepN)r; r< rn Ú enumerateru r* Ú getitem_blockÚsliceZ _mgr_locsÚaddrA r? r$ rt ) rG r( r* Zhad_reindexersr7 Úoffsetru r5 r> Z made_copyrx Únbrc r0 r{ r8 rm þ s(
|
|
ÿ
|
|
|
rm z0list[tuple[BlockManager, dict[int, np.ndarray]]])r( rG r+ c C sV g }|D ]H\}}| ¡ D ](\}}|j| | || |ddddd}q| |i f¡ q|S )zô
|
Reindex along columns so that all of the BlockManagers being concatenated
|
have matching columns.
|
|
Columns added in this reindexing have dtype=np.void, indicating they
|
should be ignored when choosing a column's final dtype.
|
FT)r, r* Z
|
only_slicer- r. )r? r@ rA )r( rG Znew_mgrs_indexersr> rH r5 rJ r0 r0 r8 rn $ s
|
ù rn zdict[int, np.ndarray])r> rH c C s` t |dkstt| j}| ¡ D ]\}}t |||< q"t|}d|ksLt| jrp| jd }|jt |||fgS | j
|
}| j }g } t j |ddD ]Ì\}
|
} | js t|
|
dks¬t| ¡ } t|} t | | d< t| }| j|
|
}|| j }t | t |jko"|jjr|jjjdkp"t |¡dk ¡ }|r8| dd¡ n|| d< t ||| }| | |f¡ q| S )zø
|
Construct concatenation plan for given block manager and indexers.
|
|
Parameters
|
----------
|
mgr : BlockManager
|
indexers : dict of {axis: indexer}
|
|
Returns
|
-------
|
plan : list of (BlockPlacement, JoinUnit) tuples
|
|
r F)Úgroupéÿÿÿÿr/ N)r< rB rC Úshaper? rt Zis_single_blockru Zmgr_locsÚJoinUnitÚblknosÚblklocsÚ libinternalsZget_blkno_placementsZ is_slice_liker* rJ Zas_sliceÚsteprV Zdiffr\ ÚpoprA )r> rH Zmgr_shape_listrI rJ Z mgr_shaperx r r ZplanZblknoÚ
|
placementsZjoin_unit_indexersZ
|
shape_listr Zax0_blk_indexerZunit_no_ax0_reindexingrw r0 r0 r8 re A s>
|
|
|
|
|
ó re c @ s e Zd ZdddddddZdd d
|
d Zed d d dZedd ddZdd dddZed d ddZ dddddZ
|
dS ) r Nr& r ÚNone)rf r r+ c C s" |d kr i }|| _ || _|| _d S ©N©rf rH r )Úselfrf r rH r0 r0 r8 Ú__init__ s
|
zJoinUnit.__init__Ústr©r+ c C s$ t | j dt| j d| j dS )Nú(z, ú))r] Ú__name__Úreprrf rH )r r0 r0 r8 Ú__repr__ s zJoinUnit.__repr__r' c C s& | j ¡ D ]}|dk ¡ r
|
dS q
|
dS )Nr
TF)rH rg Úany)r rJ r0 r0 r8 Ú needs_filling s zJoinUnit.needs_fillingr c C s2 | j }|jjjdkr td| js(|jS t|jS )NÚVzBlock is None, no dtype)rf rg rQ rU rB r r )r rx r0 r0 r8 rQ £ s zJoinUnit.dtype)rQ r+ c s | j s
|
dS | jjjdkr dS | jtkrL| jj}t fdd|jddD S | jj}|t krlt
|
| j sldS |t krt rdS t | S )z¯
|
Check that we are all-NA of a type/dtype that is compatible with this dtype.
|
Augments `self.is_na` with an additional check of the type of NA values.
|
Fr Tc 3 s | ]}t | V qd S r )r rN rP r0 r8 rW ¹ s z,JoinUnit._is_valid_na_for.<locals>.<genexpr>ÚK©Úorder)Úis_narf rQ rU r` rg r\ ÚravelÚ
|
fill_valuer r r r r )r rQ rg Úna_valuer0 rP r8 Ú_is_valid_na_for s
|
zJoinUnit._is_valid_na_forc C s¨ | j }|jjdkrdS |js dS |j}|jdkr4dS t|jtrDdS |jdkrr|d }t |rft
|
|sjdS t |S |d d }t |rt
|
|sdS t dd |D S d S )Nr TFr r/ c s s | ]}t |V qd S r )r )r4 Úrowr0 r0 r8 rW æ s z!JoinUnit.is_na.<locals>.<genexpr>) rf rQ rU Z _can_hold_narg ÚsizerM r rk r r r r\ )r rx rg Úvalr0 r0 r8 r Ë s&
|
|
zJoinUnit.is_nar )Ú empty_dtyper+ c C sì |d kr*| j jjdkr*| j j}| j j}n|}| |¡rn| j j}|t d¡krx| j jjdd}t|rx|d d krxd }t |t
|
rt | j |j ¡}t||dS t|r t||rº| jrºn`tt|}| ¡ }|jg |d}| j \} }
|
| dksôt| dtj|
|
ftjd } |j| d |d
|
S nRt |trP| ¡ }|j| j |d }||d d
< |S tj| j |d}| |¡ |S | js| j j s| j jS | j j r¨| j t d¡¡j}n| j j}| jsÂ| ¡ }n&| j !¡ D ]\} } t"j#|| | d }qÌ|S ) Nr r` r r r rP r/ r
T)Z
|
allow_fillr¢ )r rQ ri )$rf rQ rU r¢ rg r¤ rV r¡ r< rM r Úfullr Ú_valuer r r rH r r Zconstruct_array_typeZ_from_sequencerB ZonesZintpZtakeÚ_emptyÚemptyÚfillZ_can_consolidateZis_boolr_ rp r? ÚalgosZtake_nd)r r¨ Ú upcasted_nar¢ rg Z blk_dtypeZi8valuesrb Z missing_arrZncolsZnrowsZ empty_arrrI rJ r0 r0 r8 Úget_reindexed_valuesè sZ
|
|
|
|
ÿ
|
|
|
z JoinUnit.get_reindexed_values)N) r Ú
|
__module__Ú __qualname__r r r r rQ r¤ r r° r0 r0 r0 r8 r s r zlist[JoinUnit])rv r* r+ c sÄ t | tdd | D }t | fdd| D }t|dkrz|d }|rÀt|tjrp|jdk rx| ¡ }qÀ| ¡ }nFtdd |D r´d d |D }t |dd
|
d }t
|
|d }n t |dd }|S )zB
|
Concatenate values from several join units along axis=1.
|
c s s | ]}|j jjd kV qdS ©r N©rf rQ rU ©r4 rw r0 r0 r8 rW A s z*_concatenate_join_units.<locals>.<genexpr>c s g | ]}|j d qS )©r¨ r¯ )r° rh r¶ r0 r8 r9 D s ÿz+_concatenate_join_units.<locals>.<listcomp>r/ r Nc s s | ]}t |jV qd S r ©r rQ ©r4 Útr0 r0 r8 rW U s c S s* g | ]"}t |jr|n|d dd
f qS )r Nr· r¸ r0 r0 r8 r9 Z s þÿT)r, Zea_compat_axisrj ri ) Ú_get_empty_dtyper Ú_dtype_to_na_valuer< rM rV ZndarrayÚbaser* r r" )rv r* Úhas_none_blocksrL Z concat_valuesr0 r¶ r8 rs ; s*
|
þ
|
|
|
ü rs r ©rQ r½ c C sv t | tr| jS | jdkr$| d¡S | jdkr8| d¡S | jdkrFdS | jdkr^|sXdS tjS | jdkrntjS tdS ) z2
|
Find the NA value to go with this dtype.
|
)ÚmÚMr )ÚfÚcÚNaNrT N)r5 rS ÚO)rM r r£ rU r] rV ÚnanÚNotImplementedErrorr¾ r0 r0 r8 r» i s
|
|
|
|
|
|
|
|
r» zSequence[JoinUnit])rv r+ c C s t | dkr | d j}|jS t| r4| d jj}|S tdd | D }dd | D }t |sjdd | D }t|}|r~t|}|S )zÂ
|
Return dtype and N/A values to use when concatenating specified units.
|
|
Returned N/A value may be None which means there was no casting involved.
|
|
Returns
|
-------
|
dtype
|
r/ r c s s | ]}|j jjd kV qdS r³ r´ rµ r0 r0 r8 rW s z#_get_empty_dtype.<locals>.<genexpr>c S s g | ]}|j s|jqS r0 )r rQ rµ r0 r0 r8 r9 s z$_get_empty_dtype.<locals>.<listcomp>c S s g | ]}|j jjd kr|jqS )r r´ rµ r0 r0 r8 r9 s )r< rf rQ Ú_is_uniform_reindexr r r )rv rx r¨ r½ ra rQ r0 r0 r8 rº s
|
|
rº c sv | d j jjdkrdS t fdd| D ott fdd| D ottdd | D ottdd | D ott| d kS )
|
zÏ
|
Check if the join units consist of blocks of uniform type that can
|
be concatenated using Block.concat_same_type instead of the generic
|
_concatenate_join_units (which uses `concat_compat`).
|
|
r r Fc 3 s | ]}t |jt kV qd S r )r] rf rh ©Úfirstr0 r8 rW ª s z)_is_uniform_join_units.<locals>.<genexpr>c 3 s, | ]$}t |jj jp"|jjjd kV qdS ))rT r5 rS N)r rf rQ rU rh rÈ r0 r8 rW s üc s s | ]}|j p|jjV qd S r )r rf rr rh r0 r0 r8 rW · s c s s | ]}|j V qd S r )rH rh r0 r0 r8 rW º s r/ )rf rQ rU r\ r< ©rv r0 rÈ r8 rq s
|
ûý óð
|
ërq r c C s( t dd | D o&tdd | D dkS )Nc s s | ]}|j jV qd S r )rf rr rh r0 r0 r8 rW Ä s z&_is_uniform_reindex.<locals>.<genexpr>c S s h | ]}|j jjqS r0 )rf rQ Únamerh r0 r0 r8 rR Å s z&_is_uniform_reindex.<locals>.<setcomp>r/ )r\ r< rÊ r0 r0 r8 rÇ Á s ýrÇ Úint)Ú join_unitÚlengthr+ c C sÆ d| j krF| j }| jdkr d}q| j t|d¡}| j t|¡| _n>| j}t | j ¡}|d |d
|d< | j d d|
| j d< | jd | f| jdd
}|f| jdd
| _t|||dS )z
|
Reduce join_unit's shape along item axis to length.
|
|
Extra items that didn't fit are returned as a separate block.
|
r Nr/ r )rH rf r r Úcpr* r r )rÍ rÎ Zextra_indexersZ extra_blockZ extra_shaper0 r0 r8 Ú_trim_join_unitÉ s
|
|
rÐ c # sZ t | dkr2| d D ]}|d |d gfV qn$dg fdd}ttt| } tt|| } d t |krV d dkrtdt| \}}ttt |}t|t| }}||krÜ|d |fV t|| |dd
< q`d} dgt | }
|
t|D ]R\} \} } | |
|
| < t | |kr4| |d
t | |f|| < qö| } || | || < qö| |
|
fV q`dS )z
|
Combine multiple concatenation plans into one.
|
|
existing_plan is updated in-place.
|
|
We only get here with concat_axis == 1.
|
r/ r c s& t | d }|d kr" d d7 < |S )Nr r/ )Únext)ÚseqÚretval©Z num_endedr0 r8 Ú _next_or_noneô s
|
z,_combine_concat_plans.<locals>._next_or_nonezPlan shapes are not alignedN)
|
r< rC ÚmapÚiterÚ
|
ValueErrorÚzipÚminÚmaxr~ rÐ )ZplansÚprÕ Z
|
next_itemsr ZunitsÚlengthsZmin_lenÚmax_lenZyielded_placementZ yielded_unitsr5 Zplcrw r0 rÔ r8 ro ä s2 ro )PÚ
|
__future__r r* rÏ rD Útypingr r r ÚnumpyrV Z pandas._libsr r r Zpandas._libs.missingr Zpandas._typingr r
|
r r r Zpandas.util._decoratorsr Zpandas.core.dtypes.astyper Zpandas.core.dtypes.castr r Zpandas.core.dtypes.commonr r r r Zpandas.core.dtypes.concatr Zpandas.core.dtypes.dtypesr r Zpandas.core.dtypes.missingr r r Zpandas.core.algorithmsÚcoreZ
|
algorithmsr® Zpandas.core.arraysr r Zpandas.core.arrays.sparser Zpandas.core.constructionr Z#pandas.core.internals.array_managerr r! Z pandas.core.internals.blocksr" r# Z pandas.core.internals.managersr$ Zpandasr% r& rK r: rz rm rn re r rs r» rº rq rÇ rÐ ro r0 r0 r0 r8 Ú<module> sN 08O& L /. #
|
