U
|
ý°dZ ã @ s d Z ddlmZ ddlmZ ddlmZ ddlZddl Z ddl
|
Z ddl
|
m Z ddl mZ e j d¡Zdd Ze jjd
|
d Zd d Ze jjdd ZdS )zd
|
Tests multithreading behaviour for reading and
|
parsing files for each parser defined in parsers.py
|
é )Ú ExitStack)ÚBytesIO)Ú
|
ThreadPoolN)Ú DataFrameZ pyarrow_skipc C s\ t tj | d¡tdd}d|d< d|d< d|d< tjd| dd |d
|
< tj| d d |d < |S )z¸
|
Construct a DataFrame for testing.
|
|
Parameters
|
----------
|
num_rows : int
|
The number of rows for our DataFrame.
|
|
Returns
|
-------
|
df : DataFrame
|
é Zabcde)ÚcolumnsZfooÚbarZbazz20000101 09:00:00Ús)ZperiodsÚfreqÚdateÚint64)ZdtypeÚint)r ÚnpÚrandomZrandÚlistÚpdZ
|
date_rangeZarange)Únum_rowsÚdf© r ú_d:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\pandas/tests/io/parser/test_multi_thread.pyÚ_construct_dataframe s r c s | }d d} fddt |D }t Rfdd|D } td¡}| |j|¡}|d }|D ]}t ||¡ qdW 5 Q R X d S )Ni' éd c s( g | ] }d dd t D ¡ ¡ qS )Ú
|
c S s& g | ] }|d d|d d|d qS )Údú,r ©Ú.0Úir r r Ú
|
<listcomp>3 s zCtest_multi_thread_string_io_read_csv.<locals>.<listcomp>.<listcomp>)ÚjoinÚrangeÚencode)r Ú_)Ú max_row_ranger r r 2 s ÿz8test_multi_thread_string_io_read_csv.<locals>.<listcomp>c s g | ]} t|¡qS r )Ú enter_contextr )r Úb)Ústackr r r 9 s é r )r r r$ r ÚmapÚread_csvÚtmÚassert_frame_equal) Ú all_parsersÚparserZ num_filesZ bytes_to_dfÚfilesÚpoolÚresultsZ first_resultÚresultr )r# r& r Ú$test_multi_thread_string_io_read_csv+ s
|
þr2 c sx fdd} fddt D }td}| ||¡}W 5 Q R X |d j}|dd
D ]
|
} || _q^t |¡}
|
|
|
S ) a
|
Generate a DataFrame via multi-thread.
|
|
Parameters
|
----------
|
parser : BaseParser
|
The parser object to use for reading the data.
|
path : str
|
The location of the CSV file to read.
|
num_rows : int
|
The number of rows to read per task.
|
num_tasks : int
|
The number of tasks to use for reading this DataFrame.
|
|
Returns
|
-------
|
df : DataFrame
|
c sB | \}}|s" j dd|dgdS j ddt|d |dgdS )aj
|
Create a reader for part of the CSV.
|
|
Parameters
|
----------
|
arg : tuple
|
A tuple of the following:
|
|
* start : int
|
The starting row to start for parsing CSV
|
* nrows : int
|
The number of rows to read.
|
|
Returns
|
-------
|
df : DataFrame
|
r r )Ú index_colÚheaderÚnrowsÚ parse_datesNé é )r3 r4 Zskiprowsr5 r6 )r) r )ÚargÚstartr5 )r- Úpathr r ÚreaderX s" ÿ
|
úz0_generate_multi_thread_dataframe.<locals>.readerc s g | ]} | fqS r r r )r Ú num_tasksr r r z s z4_generate_multi_thread_dataframe.<locals>.<listcomp>)Z processesr r7 N)r r r( r r Úconcat) r- r; r r= r< Ztasksr/ r0 r4 ÚrÚfinal_dataframer )r r= r- r; r Ú _generate_multi_thread_dataframeD s " ÿ
|
|
rA c C sV d}d}| }d}t |}t |¡*}| |¡ t||||}t ||¡ W 5 Q R X d S )Né i z__thread_pool_reader__.csv)r r* Z ensure_cleanZto_csvrA r+ )r, r= r r- Ú file_namer r; r@ r r r Ú)test_multi_thread_path_multipart_read_csv s
|
ÿrD )Ú__doc__Ú
|
contextlibr Úior Zmultiprocessing.poolr Únumpyr ZpytestZpandasr r Zpandas._testingZ_testingr* ÚmarkZ usefixturesZ
|
pytestmarkr Zslowr2 rA rD r r r r Ú<module> s
|
F
|
