U
|
ý°d / ã @ s d Z ddlZddlZddlmZ ddlZddlZddlZddl Z
|
ddl Z ddl m Z ddlm mZ ddlZddlmZ e jje ddZG dd dejjZ G d d
|
d
|
e Z G d d d e Z G d d de Z G dd de Z G dd de Z!G dd de Z"G dd de Z#G dd de Z$G dd dejjZ%dd Z&d d Z'e j(d d Z)e jj*d!e ej+dfe ej,dfe!ej-d"fe j.e"ej-d#ej/gd$e#ej0dfe$ej1dfe ej+dfe ej,dfgd gd%d&d' Z2e jj*d!e ej+dfe ej,dfe!ej-d"fe j.e"ej-d#ej/gd$e#ej0dfe$ej1dfe ej+dfe ej,dfgd gd%d(d) Z3e jj*d*e%ej+fgd gd%d+d, Z4e j *d-d"d#g¡d.d/ Z5dS )0z9
|
Tests for the pandas custom headers in http(s) requests
|
é N)ÚBytesIO)Úis_ci_environmentz
This test can hang in our CI min_versions build and leads to '##[error]The runner has received a shutdown signal...' in GHA. GH 45651)Úreasonc @ s( e Zd ZdZdd Zdd Zdd ZdS ) ÚBaseUserAgentResponderzü
|
Base class for setting up a server that can be set up to respond
|
with a particular file format with accompanying content-type headers.
|
The interfaces on the different io methods are different enough
|
that this seemed logical to do.
|
c C s, | d¡ | jd | _t d| jgi¡}|S )z<
|
shared logic at the start of a GET request
|
éÈ ú
|
User-AgentÚheader)Ú send_responseÚheadersZrequested_from_user_agentÚpdÚ DataFrame)ÚselfÚ response_df© r úVd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\pandas/tests/io/test_user_agent.pyÚstart_processing_headers$ s
|
ÿÿz/BaseUserAgentResponder.start_processing_headersc
|
C sB t 2}tj|dd}| |¡ W 5 Q R X | ¡ }W 5 Q R X |S )zQ
|
some web servers will send back gzipped files to save bandwidth
|
Úw)ÚfileobjÚmode)r ÚgzipÚGzipFileÚwriteÚgetvalue)r Úresponse_bytesÚbioZzipperr r r Ú
|
gzip_bytes1 s
|
z!BaseUserAgentResponder.gzip_bytesc C s | j |¡ dS )z:
|
shared logic at the end of a GET request
|
N)Úwfiler )r r r r r Úwrite_back_bytes; s z'BaseUserAgentResponder.write_back_bytesN)Ú__name__Ú
|
__module__Ú __qualname__Ú__doc__r r r r r r r r s
|
r c @ s e Zd Zdd ZdS )ÚCSVUserAgentResponderc C s< | ¡ }| dd¡ | ¡ |jdd d¡}| |¡ d S )Nú Content-Typeútext/csvF©Úindexúutf-8)r Ú send_headerÚ end_headersÚto_csvÚencoder ©r r r r r r Údo_GETC s
|
z CSVUserAgentResponder.do_GETN©r r r r- r r r r r" B s r" c @ s e Zd Zdd ZdS )Ú GzippedCSVUserAgentResponderc C sR | ¡ }| dd¡ | dd¡ | ¡ |jdd d¡}| |¡}| |¡ d S )Nr# r$ úContent-Encodingr Fr% r' )r r( r) r* r+ r r r, r r r r- N s
|
z#GzippedCSVUserAgentResponder.do_GETNr. r r r r r/ M s r/ c @ s e Zd Zdd ZdS )ÚJSONUserAgentResponderc C s8 | ¡ }| dd¡ | ¡ | ¡ d¡}| |¡ d S )Nr# úapplication/jsonr' )r r( r) Úto_jsonr+ r r, r r r r- [ s
|
z JSONUserAgentResponder.do_GETNr. r r r r r1 Z s r1 c @ s e Zd Zdd ZdS )Ú GzippedJSONUserAgentResponderc C sN | ¡ }| dd¡ | dd¡ | ¡ | ¡ d¡}| |¡}| |¡ d S )Nr# r2 r0 r r' )r r( r) r3 r+ r r r, r r r r- f s
|
z$GzippedJSONUserAgentResponder.do_GETNr. r r r r r4 e s r4 c @ s e Zd Zdd ZdS )Ú ParquetPyArrowUserAgentResponderc C s8 | ¡ }| dd¡ | ¡ |jddd}| |¡ d S )Nr# úapplication/octet-streamFÚpyarrow)r& Úengine)r r( r) Ú
|
to_parquetr r, r r r r- s s
|
z'ParquetPyArrowUserAgentResponder.do_GETNr. r r r r r5 r s r5 c @ s e Zd Zdd ZdS )Ú$ParquetFastParquetUserAgentResponderc C sd | ¡ }| dd¡ | ¡ dd l}|jdddd d | dd¡}| ¡ }W 5 Q R X | |¡ d S ) Nr# r6 r z'memory://fastparquet_user_agent.parquetFÚ fastparquet)r& r8 Ú compressionÚrb)r r( r) Úfsspecr9 ÚopenÚreadr )r r r> Úfr r r r r- ~ s üz+ParquetFastParquetUserAgentResponder.do_GETNr. r r r r r: } s r: c @ s e Zd Zdd ZdS )ÚPickleUserAgentResponderc C sB | ¡ }| dd¡ | ¡ t }| |¡ | ¡ }| |¡ d S )Nr# r6 )r r( r) r Z to_pickler r ©r r r r r r r r- s
|
z PickleUserAgentResponder.do_GETNr. r r r r rB s rB c @ s e Zd Zdd ZdS )ÚStataUserAgentResponderc C sF | ¡ }| dd¡ | ¡ t }|j|dd | ¡ }| |¡ d S )Nr# r6 F)Z write_index)r r( r) r Zto_statar r rC r r r r- ¥ s z StataUserAgentResponder.do_GETNr. r r r r rD ¤ s rD c @ s e Zd ZdZdd ZdS )ÚAllHeaderCSVResponderz?
|
Send all request headers back for checking round trip
|
c C sP t | j ¡ ¡}| d¡ | dd¡ | ¡ |jdd d¡}| j
|
|¡ d S )Nr r# r$ Fr% r' ) r r r
|
Úitemsr r( r) r* r+ r r r, r r r r- ¶ s
|
z AllHeaderCSVResponder.do_GETN)r r r r! r- r r r r rE ± s rE c s fdd}|S )Nc s6 z | |W S t jjk
|
r. t d¡ Y q X q d S )Nç¹?)ÚurllibÚerrorÚURLErrorÚtimeÚsleep)ÚargsÚkwargs©Úfuncr r ÚinnerÀ s z wait_until_ready.<locals>.innerr )rP rM rN rQ r rO r Úwait_until_ready¿ s rR c C s2 t j d|f| ¡}| ¡ W 5 Q R X | ¡ d S )NÚ localhost)ÚhttpÚserverZ
|
HTTPServerÚhandle_requestÚ server_close)Ú responderÚportrU r r r Úprocess_serverË s rZ c c s t ¡ }| d¡ | ¡ d }W 5 Q R X tjt| j|fd}| ¡ |V | d¡ | ¡ d}d}|
|
¡ r||kr| ¡ q|d7 }t d¡ qf| ¡ dS ) zã
|
Fixture that starts a local http server in a separate process on localhost
|
and returns the port.
|
|
Running in a separate process instead of a thread to allow termination/killing
|
of http server upon cleanup.
|
)rS r é )ÚtargetrM é
|
é r rG N)ÚsocketÚbindÚ getsocknameÚmultiprocessingÚProcessrZ ÚparamÚstartÚjoinÚ terminateÚis_aliveÚkillrK rL Úclose)ÚrequestÚsockrY Zserver_processZ kill_timeZ wait_timer r r rX Ñ s(
|
|
|
ÿ
|
rX z&responder, read_method, parquet_enginer7 r; )Zmarks)Zindirectc C sd |d k r$t |¡ |dkr$t d¡ t|}|d krD|d| }n|d| |d}|jr`td S )Nr; r> úhttp://localhost:)r8 )ÚpytestÚ importorskiprR ÚemptyÚAssertionError)rX Ú read_methodÚparquet_engineÚdf_httpr r r Ú test_server_and_default_headersñ s
|
|
ru c C s |d k r$t |¡ |dkr$t d¡ d}t d|gi¡}t|}|d kr`|d| d|id}n|d| d|i|d}t ||¡ d S ) Nr; r> úSuper Cool Oner rm r ©Ústorage_options©rx r8 )rn ro r r rR ÚtmÚassert_frame_equal)rX rr rs Úcustom_user_agentÚdf_truert r r r Ú test_server_and_custom_headers s$
|
|
þýr~ zresponder, read_methodc s¸ d}d}||d t |}|d| d}||d ¡ ¡ }| dg¡ ¡ }|ddg }t ¡ }t | fdd |D d
|
¡}| dg¡}| ¡ jd gd d }t
|
||¡ d S )Nrv zSuper Secret One©r ZAuthrm rw Ú0Ú1c s g | ] } | qS r r )Ú.0Úkrw r r Ú
|
<listcomp>^ s z6test_server_and_all_custom_headers.<locals>.<listcomp>)r r r& r[ )Zaxis) rR ÚisinÚkeysZ sort_valuesZ reset_indexÚlistr r Zdroprz r{ )rX rr r| Zcustom_auth_tokenrt r r} r rw r Ú"test_server_and_all_custom_headersE s$ þþ r r8 c C sV ddd}t | ¡ t ddgi¡}d}t jt|d |jd|| d W 5 Q R X d S )
|
NZcustomZ other_customr Z column_nameZ column_valuezxstorage_options passed with file object or non-fsspec file path|storage_options passed with buffer, or non-supported URL)Úmatchz/tmp/junk.parquetry )rn ro r r ZraisesÚ
|
ValueErrorr9 )r8 r
|
Ztrue_dfÚmsgr r r Ú,test_to_parquet_to_disk_with_storage_optionse s þ
|
ÿr )6r! r Z http.serverrT Úior rb r_ rK Ú urllib.errorrH rn Z pandas.compatr Z pandas.util._test_decoratorsÚutilZ_test_decoratorsÚtdZpandasr Zpandas._testingZ_testingrz ÚmarkZskipifZ
|
pytestmarkrU ZBaseHTTPRequestHandlerr r" r/ r1 r4 r5 r: rB rD rE rR rZ ZfixturerX Z parametrizeZread_csvÚ read_jsonZ read_parquetrd Z&skip_array_manager_not_yet_implementedZ read_pickleZ
|
read_stataru r~ r r r r r r Ú<module> s¨ þ&
|
|
|
|
ÿû
|
|
|
|
ðì
|
|
|
|
ÿû
|
|
|
|
ðì
|
ÿû
|
þþ
|
