U
|
ý°d ã @ s d dl mZ d dlZd dlZd dlZd dlZd dlZd dlm Z m
|
Z
|
m Z m Z m Z mZ d dlmZ d dlmZ d dlmZ ejdd Ze d¡ej d d
|
d d d dg¡dd Z e e e dddZ e d¡ej dddg¡dd Z e d¡e d¡dd Z!e "d¡d d Z#dS ) é )ÚBytesION)Ú DataFrameÚ
|
date_rangeÚread_csvÚ
|
read_excelÚ read_jsonÚ read_parquet)Ú_compression_to_extension)Ú_test_decoratorsc sB ddl }t dd _G fddd|j}|jd|dd S )
|
z"Emulate GCS using a binary buffer.r Nc S s dS )NT© r r r úOd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\pandas/tests/io/test_gcs.pyÚ<lambda> ó z gcs_buffer.<locals>.<lambda>c s$ e Zd Ze fddZdd ZdS )z%gcs_buffer.<locals>.MockGCSFileSystemc s d¡ S )Nr )Úseek)ÚargsÚkwargs©Ú
|
gcs_bufferr r Úopen s
|
z*gcs_buffer.<locals>.MockGCSFileSystem.openc [ s |ddgS )NÚfile)ÚnameÚtyper )ÚselfÚpathr r r r Úls$ s z(gcs_buffer.<locals>.MockGCSFileSystem.lsN)Ú__name__Ú
|
__module__Ú __qualname__Ú staticmethodr r r r r r ÚMockGCSFileSystem s r ZgsT)Úclobber)Úfsspecr ÚcloseÚAbstractFileSystemZregister_implementation)Ú monkeypatchr! r r r r r s
|
r ZgcsfsÚformatÚcsvÚjsonÚparquetÚexcelÚmarkdownc C sú t ddgdtjgddgtdddd }d
|
| }|d krZ|j|d d t|dgdd}n|dkrd}| |¡ t|dgdd}nh|dkr¤| |¡ t |dgd}nF|dkrÊt
|
d¡ | |¡ t |}n |dkrêt
|
d¡ | |¡ |}t ||¡ dS )zE
|
Test that many to/read functions support GCS.
|
|
GH 33987
|
é é ç @ÚtÚsú
|
2018-06-18é ©Zperiods©ÚintÚfloatÚstrÚdtzgs://test/test.r& T)Úindexr7 r )Z parse_datesÚ index_colr) zgs://test/test.xlsxr' )Z convert_datesr( Zpyarrowr* ÚtabulateN)r ÚnpÚnanr Úto_csvr Zto_excelr Úto_jsonr ÚpytestZ importorskipÚ
|
to_parquetr Z to_markdownÚtmÚassert_frame_equal)r r% Údf1r Zdf2r r r Útest_to_read_gcs. s6
|
üÿ
|
|
|
|
|
|
|
|
rD )ÚresultÚexpectedÚ compressionc
|
C s( |dkrjt t| ¡L}t t|¡2}t| ¡ | ¡ D ]\}}|j|jks:tq:W 5 Q R X W 5 Q R X nº|dkrtjt| d}tjt|dn}t| ¡ | ¡ D ]T\} }
|
|
|
| ¡} |
|
|
|
¡} | dk| dkksÚt| dk rª| dk rª| ¡ | ¡ ksªtqªW 5 Q R X W 5 Q R X n| |ks$tdS )a
|
For zip compression, only compare the CRC-32 checksum of the file contents
|
to avoid checking the time-dependent last-modified timestamp which
|
in some CI builds is off-by-one
|
|
See https://en.wikipedia.org/wiki/ZIP_(file_format)#File_headers
|
ÚzipÚtar)ÚfileobjN) ÚzipfileÚZipFiler rH ÚinfolistÚCRCÚAssertionErrorÚtarfiler Ú
|
getmembersÚ extractfileÚread) rE rF rG ÚexpÚresZres_infoZexp_infoZtar_expZtar_resZ tar_res_infoZ tar_exp_infoZ actual_fileZ expected_filer r r Úassert_equal_zip_safeX s, ÿ(
|
ÿ ÿ
|
|
,rV Úencodingzutf-8Úcp1251c C sî t ¡ }d|i}|dkr d|d< t }|j|||dd d}|j|||d | ¡ }| ¡ }t||| t|d ||d
|
} t || ¡ t| }
|
d |d< |d |
|
7 }|j|||d | ¡ }| ¡ }t||| t|d d |d
|
} t || ¡ d S )z
|
Compression and encoding should with GCS.
|
|
GH 35677 (to_csv, compression), GH 26124 (to_csv, encoding), and
|
GH 32392 (read_csv, encoding)
|
ÚmethodÚgzipr+ ÚmtimeÚwb)rG rW Úmodeúgs://test/test.csv)rG rW r )r9 rG rW ZinferÚ.N) rA Z makeDataFramer r= ÚgetvaluerV r rB r ) r Zcompression_onlyrW ZdfrG ÚbufferZpath_gcsrU rF Zread_dfZfile_extr r r Ú$test_to_csv_compression_encoding_gcsw s6 ÿ rb Ú fastparquetc sj ddl m} tddgdtjgddgtdd d
|
d }G fd d d |}| d|¡ |jddddd dS )zCRegression test for writing to a not-yet-existent GCS Parquet file.r )r# r+ r, r- r. r/ r0 r1 r2 r3 c s e Zd Zd fdd ZdS )z7test_to_parquet_gcs_new_file.<locals>.MockGCSFileSystemÚrc s d|kr t ttj d¡|S )NÚwz test.parquet)ÚFileNotFoundErrorr Úosr Újoin)r r r] r ©Útmpdirr r r ³ s z<test_to_parquet_gcs_new_file.<locals>.MockGCSFileSystem.openN)rd )r r r r r ri r r r ² s r zgcsfs.GCSFileSystemr^ Trc N)r8 ZenginerG )r! r# r r; r< r Úsetattrr@ )r$ rj r# rC r r ri r Ú test_to_parquet_gcs_new_file£ s
|
üÿ ÿrl c C s" t t¡ td W 5 Q R X d S )Nr^ )rA Zexternal_error_raisedÚ ImportErrorr r r r r Ú test_gcs_not_present_exception¾ s rn )$Úior rg rP rK Únumpyr; r? Zpandasr r r r r r Zpandas._testingZ_testingrA Z pandas.tests.io.test_compressionr Z pandas.utilr
|
ÚtdZfixturer Z
|
skip_if_noÚmarkZ parametrizerD Úbytesr6 rV rb rl Zskip_if_installedrn r r r r Ú<module> s.
|
( *
|
