U
|
ý°dj/ ã @ s4 d Z ddlmZmZ ddlZddlZddlZddlm Z ddl
|
m m Z ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ejjejd d
|
d ej d d dg¡ej dddg¡dd Z ejjejdd
|
d dd Z ej dd Z ejj ej !d¡ejj"de#dd e $¡ G d d d Z%dS ) zl
|
Tests parsers ability to read and parse non-local files
|
and hence require a network connection to be read.
|
é )ÚBytesIOÚStringION)Úis_ci_environment)Ú DataFrame)Ú_compression_to_extension)Ú read_feather©Úread_csvúVhttps://github.com/pandas-dev/pandas/raw/main/pandas/tests/io/parser/data/salaries.csvT)ÚurlZcheck_before_testÚmodeÚexplicitÚinferÚengineÚpythonÚcc C s@ t | }d}|| }|dkr |}t|d||d}t || ¡ d S )Nr
|
r ú )ÚsepÚ compressionr )r r ÚtmÚassert_frame_equal)Zsalaries_tabler r Zcompression_onlyÚ extensionÚbase_urlr Z url_table© r úZd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\pandas/tests/io/parser/test_network.pyÚtest_compressed_urls s ÿr úghttps://raw.githubusercontent.com/pandas-dev/pandas/main/pandas/tests/io/parser/data/unicode_series.csvc C s( d} t | ddd}|jd dks$tdS )zN
|
read_csv should honor the requested encoding for URLs.
|
|
GH 10424
|
r zlatin-1N)ÚencodingÚheader)é é u$ à köldum klaka (Cold Fever) (1994))r ÚlocÚAssertionError)ÚpathÚdfr r r Útest_url_encoding_csv5 s ÿr% c C s t | ddddS )z DataFrame with the tips dataset.ÚioÚdataÚcsvútips.csvr )Údatapathr r r Útips_dfK s r+ Ú s3_resourcez$CI race condition GH 45433, GH 44584F)ÚreasonÚraisesÚstrictc @ sì e Zd Ze d¡dd Zdd Zdd Zdd Zd
|
d Z d d Z
|
dd Z dd Z dd Z dd Zejjddddd Zejjddde d¡d d Zejjd d Zejjejje d d!d"d# Zd$d% Ze d¡d&d' Zd(S ))ÚTestS3Ús3fsc C sx dD ]>\}}t d| ||d}t|ts,t|jr6tt ||¡ qt d|d}t|ts^t|jrhtt ||¡ d S )N©)Ú N)ú.gzÚgzip)ú.bz2Úbz2ús3://pandas-test/tips.csv)r Ústorage_optionszs3://cant_get_it/tips.csv©r9 ©r Ú
|
isinstancer r" Úemptyr r ©Úselfr+ Ús3soÚextÚcompr$ r r r Útest_parse_public_s3_bucketZ s ý
|
|
z"TestS3.test_parse_public_s3_bucketc C s@ t dd|d}t|ts t|jr&tt |jd d
|¡ d S )Nzs3n://pandas-test/tips.csvé
|
©Únrowsr9 ©r r< r r" r= r r Úiloc©r? r+ r@ r$ r r r Ú test_parse_public_s3n_bucketn s
|
z#TestS3.test_parse_public_s3n_bucketc C s@ t dd|d}t|ts t|jr&tt |jd d
|¡ d S )Nzs3a://pandas-test/tips.csvrD rE rG rI r r r Ú test_parse_public_s3a_bucketu s
|
z#TestS3.test_parse_public_s3a_bucketc C sT dD ]J\}}t d| d||d}t|ts.t|jr8tt |jd d
|¡ qd S )Nr2 r8 rD )rF r r9 rG r> r r r Ú!test_parse_public_s3_bucket_nrows| s ü
|
z(TestS3.test_parse_public_s3_bucket_nrowsc
|
|
C s d}dD ]\}}t d| |||dd}|j|ks4tdD ]J}| ¡ }t|tsRt|jr\t|j|| ||d
} t | |¡ q8W 5 Q R X qd S )Né r2 r8 )Ú chunksizer r9 ©r r é r ©
|
r rN r" Z get_chunkr< r r= rH r r ©
|
r? r+ r@ rN rA rB Z df_readerZi_chunkr$ Ztrue_dfr r r Ú#test_parse_public_s3_bucket_chunked s& ü
|
|
ÿz*TestS3.test_parse_public_s3_bucket_chunkedc
|
|
C s d}dD ]\}}t d| ||d|dd}|j|ks6tdD ]J}| ¡ }t|tsTt|jr^t|j|| ||d
} t | |¡ q:W 5 Q R X qd S )NrM r2 r8 r )rN r r r9 rO r rQ rR r r r Ú*test_parse_public_s3_bucket_chunked_python s( û
|
|
ÿz1TestS3.test_parse_public_s3_bucket_chunked_pythonc C sJ dD ]@\}}t d| d||d}t|ts.t|jr8tt ||¡ qd S )Nr2 r8 r ©r r r9 r; r> r r r Ú"test_parse_public_s3_bucket_python´ s ü
|
z)TestS3.test_parse_public_s3_bucket_pythonc C sF dD ]<}t d| dd|d}t|ts*t|jr4tt ||¡ qd S )N)r3 r4 r6 r8 r r rU r; )r? r+ r@ rA r$ r r r Útest_infer_s3_compressionÀ s ü
|
z TestS3.test_infer_s3_compressionc C sV dD ]L\}}t d| dd||d}t|ts0t|jr:tt |jd d
|¡ qd S )Nr2 r8 r rD )r rF r r9 rG r> r r r Ú(test_parse_public_s3_bucket_nrows_pythonÌ s û
|
z/TestS3.test_parse_public_s3_bucket_nrows_pythonc C sP d}t jt|d td|d W 5 Q R X t jt|d td W 5 Q R X d S )Nú#The specified bucket does not exist©Úmatchzs3://nyqpug/asdf.csvr: zs3://cant_get_it/file.csv)Úpytestr. ÚOSErrorr )r? r@ Úmsgr r r Útest_read_s3_failsÙ s
|
zTestS3.test_read_s3_failszGH#39155 s3fs upgradeF)r- r/ c C s@ dd l }t|jjf}tj|dd |jd|d W 5 Q R X d S )Nr rY rZ z/s3://an_s3_bucket_data_doesnt_exit/not_real.csvr: )ÚbotocoreÚFileNotFoundErrorÚ
|
exceptionsÚ ClientErrorr\ r. Úto_csv©r? r+ r@ r` Úerrorr r r Útest_write_s3_csv_failsã s ÿz TestS3.test_write_s3_csv_failsZpyarrowc C s@ dd l }t|jjf}tj|dd |jd|d W 5 Q R X d S )Nr rY rZ z3s3://an_s3_bucket_data_doesnt_exit/not_real.parquetr: )r` ra rb rc r\ r. Z
|
to_parquetre r r r Útest_write_s3_parquet_failsó s þz"TestS3.test_write_s3_parquet_failsc C sj |j jjddd}t|d ¡ }t|dd}W 5 Q R X t|tsHt|j rRtt|}t
|
||¡ d S )Nú pandas-testr) )ÚBucketÚKeyÚBodyÚutf8)r ) ÚmetaÚclientZ
|
get_objectr Úreadr r< r r" r= r r )r? r, Z tips_fileZ s3_objectÚbufferÚresultÚexpectedr r r Ú$test_read_csv_handles_boto_s3_object s ÿ
|
z+TestS3.test_read_csv_handles_boto_s3_objectzThis test can hang in our CI min_versions build and leads to '##[error]The runner has received a shutdown signal...' in GHA. GH: 45651)r- c C s¨ dd l }ttj dd¡tdd}t }| |¡ t| ¡
|
d¡}| d¡j d|d |j ¡ |jtjd
|
d , td d |d ddd |jD kstW 5 Q R X d S )Nr i é Zabcd)Úcolumnszutf-8ri zlarge-file.csv)rk rl r1 )Úloggerz s3://pandas-test/large-file.csvrM rE )r i T c s s | ]}|j d d
V qdS )éþÿÿÿN)Úargs)Ú.0Úxr r r Ú <genexpr>2 s z8TestS3.test_read_csv_chunked_download.<locals>.<genexpr>)r1 r ÚnpÚrandomZrandnÚlistr rd r ÚgetvalueÚencoderj Z
|
put_objectZ S3FileSystemZclear_instance_cacheZat_levelÚloggingÚDEBUGr Úrecordsr" )r? r, Zcaplogr@ r1 r$ Zstr_bufÚbufr r r Ú test_read_csv_chunked_download s
|
|
z%TestS3.test_read_csv_chunked_downloadc C s t d|d}t ||¡ d S )Nzs3://pandas-test/tips#1.csvr: )r r r )r? r+ r@ rr r r r Ú test_read_s3_with_hash_in_key4 s z$TestS3.test_read_s3_with_hash_in_keyc C s$ t |}t d|d}t ||¡ d S )Nz's3://pandas-test/simple_dataset.featherr: )r r r )r? Z feather_filer@ rs Úresr r r Ú test_read_feather_s3_file_path9 s ÿz%TestS3.test_read_feather_s3_file_pathN)Ú__name__Ú
|
__module__Ú __qualname__ÚtdZ
|
skip_if_norC rJ rK rL rS rT rV rW rX r_ r\ ÚmarkÚxfailrg rh Ú
|
single_cpurt Zskipifr r r r r r r r r0 Q s6
|
|
|
|
þ r0 )&Ú__doc__r& r r r Únumpyr} r\ Z pandas.compatr Z pandas.util._test_decoratorsÚutilZ_test_decoratorsr Zpandasr Zpandas._testingZ_testingr Z pandas.tests.io.test_compressionr Zpandas.io.feather_formatr Zpandas.io.parsersr r ÚnetworkZ parametrizer r% Zfixturer+ r Z usefixturesr ra Zskip_if_not_us_localer0 r r r r Ú<module> sH ûû
|
|
ý
|
