U
|
ý°d·H ã @ sÞ d dl mZ d dlmZ d dlmZ d dlZd dlZd dlm Z m
|
Z
|
d dl m Z d dlmZ ejdd Zd d
|
Zd d Zd d Zdd Zdd Zdd Zej dddg¡dd Zdd Zd d Z d d Z d d! Z ej dddg¡d"d# Z ej dd d$d%d&g¡d'd( Z ej dddd)g¡d*d+ Z!d,d- Z"ej d.dd)g¡d/d0 Z#ej d1d2d3g¡d4d5 Z$d6d7 Z%d8d9 Z&d:d; Z'ej d<d=d>d?g¡d@dA Z(dBdC Z)ej dDdEdFg¡dGdH Z*dIdJ Z+dKdL Z,dMdN Z-dOdP Z.dS )Qé )ÚStringIO)ÚPath)ÚIteratorN)Ú DataFrameÚ read_json)Ú
|
JsonReaderc C s( t dddgdddgd} | jdd d
|
S ) Né é é é é é ©ÚAÚBTÚrecords©ÚlinesÚorient)r Úto_json)Údf© r úZd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\pandas/tests/io/json/test_readlines.pyÚ lines_json_df s r c C s8 t ddd} tddgddggddgd}t | |¡ d S ) Nz!{"a": 1, "b": 2}
|
{"b":2, "a" :1}
|
T©r r r ÚaÚb©Úcolumns©r r ÚtmÚassert_frame_equal)ÚresultÚexpectedr r r Útest_read_jsonl s r$ c C sB t | ddddd|d}tddd gd
|
d d gd }t ||¡ d S )NÚioÚjsonÚdataúline_delimited.jsonT©r Úenginer r
|
r r r r ©r r r )Údatapathr* r" r# r r r Ú test_read_jsonl_engine_pyarrow s ýr- c C s |dkr$d}| j tjj|td¡ tddgddgdfgd d
|
d gd }|jd dd}t||d}tdddgdddggd d
|
d gd }t
|
||¡ d S )NÚpyarrowzEPyarrow only supports a file path as an input and line delimited json©ÚreasonÚraisesr r z
|
2020-03-05z2020-04-08T09:58:49+00:00ZhectorZaccountsÚdateÚnamer Tr r ©r* ) ÚnodeÚ
|
add_markerÚpytestÚmarkÚxfailÚ
|
ValueErrorr r r r r! )Úrequestr* r0 r Z json_liner" r# r r r Útest_read_datetime( s þ þr< c C s| d} t | } t| dd}tddgddggddgd }t ||¡ d} t| dd}tddgddggddgd }t ||¡ d S )
|
Nu5 {"a": "fooâ", "b": "bar"}
|
{"a": "foo", "b": "bar"}
|
Tr u fooâÚbarÚfoor r r )r r r r r! )r& r" r# r r r Ú test_read_jsonl_unicode_chars< s r? c C sÚ t ddgddggddgd} | jddd}d }||ks:tt d
|
d gd d ggddgd} | jddd}d }||ksttt t|dd| ¡ t dd gd d ggddgd} | jddd}d}||ksÂtt t|dd| ¡ d S )Nr r r r r r T©r r z {"a":1,"b":2}
|
{"a":1,"b":2}
|
zfoo}r= zfoo"z/{"a":"foo}","b":"bar"}
|
{"a":"foo\"","b":"bar"}
|
r zfoo\za\z4{"a\\":"foo\\","b":"bar"}
|
{"a\\":"foo\"","b":"bar"}
|
)r r ÚAssertionErrorr r! r )r r" r# r r r Ú test_to_jsonlN s rB c C sD t ddgddggddgd} | jddd d ¡}d}||ks@td S )
|
Nr r r r r r Tr@ Ú
|
)r r ÚcountrA )r Zactual_new_lines_countZexpected_new_lines_countr r r Ú test_to_jsonl_count_new_linesc s rE Ú chunksizer g ð?c C sn |dkr$d}| j tjj|td¡ tt|dd}tt|d||d}t |¡}W 5 Q R X t
|
||¡ d S )Nr. úmPyarrow only supports a file path as an input and line delimited jsonand doesn't support chunksize parameter.r/ Tr ©r rF r* ) r5 r6 r7 r8 r9 r: r r ÚpdÚconcatr r! )r; r rF r* r0 Ú unchunkedÚreaderÚchunkedr r r Útest_readjson_chunksk s ÿ ÿrN c
|
C sB d}t jt|d& tt| dd|d}W 5 Q R X W 5 Q R X d S )Nz*chunksize can only be passed if lines=True©ÚmatchFr rH ©r7 r1 r: r r )r r* ÚmsgÚ_r r r Ú&test_readjson_chunksize_requires_lines s ÿrT c C s |dkr"d}| j tjj|d¡ t ddd¡}t|jddd }t |dd
|
|d }t|jddd }t |dd
|
d|d }t
|
|¡}W 5 Q R X t ||¡ d S ) Nr. rG )r0 r r r Tr r ÚSeries)r Útypr* )r rV rF r* ) r5 r6 r7 r8 r9 rI rU r r r rJ r Zassert_series_equal)r; r* r0 ÚsZstriorK rL rM r r r Útest_readjson_chunks_series s$ ÿ ÿrX c C st |dkr$d}| j tjj|td¡ tt|dd|d}t|}W 5 Q R X |d j dks^t
|
|d j d
|
kspt
|
d S ) Nr. rG r/ Tr rH r ©r r r )r r ) r5 r6 r7 r8 r9 r: r r ÚlistÚshaperA )r; r r* r0 rL Úchunksr r r Útest_readjson_each_chunk£ s ÿ ÿr] c
|
C s¨ |dkr$d}| j tjj|td¡ t d¡p}tdddgdd d
|
gd }|j |d d d t
|
|d d|d}t |¡}W 5 Q R X t
|
|d |d}t ||¡ W 5 Q R X d S )Nr. rG r/ ú test.jsonr r r
|
r r r r Tr r rH r) )r5 r6 r7 r8 r9 r: r Ú ensure_cleanr r r rI rJ r! )r; r* r0 Úpathr rL rM rK r r r Ú test_readjson_chunks_from_file¶ s ÿ ra c C s t d¡}tdddgdddgd}|j|d d
|
d t|d d d d d d d d d d | d d d}| | ¡ W 5 Q R X |jjjst d| W 5 Q R X d S )Nr^ r r r
|
r r r r Tr r ÚframeF) r rV ZdtypeZ convert_axesZ convert_datesZkeep_default_datesZ precise_floatZ date_unitÚencodingr rF Ú compressionÚnrowsz%didn't close stream with chunksize = )
|
r r_ r r r ÚreadZhandlesÚhandleÚclosedrA )rF r` r rL r r r Útest_readjson_chunks_closesÈ s2 òÿþri éÿÿÿÿg@r> c
|
C sB d}t jt|d& tt| d||d}W 5 Q R X W 5 Q R X d S )Nz"'chunksize' must be an integer >=1rO TrH rQ )r rF r* rR rS r r r Ú test_readjson_invalid_chunksizeä s ÿrk r c C sh d}t dddgdddgd}t|d | d
|
}| d k rN| t |¡}W 5 Q R X tj||d | d d S ) NzF
|
|
{"A":1,"B":4}
|
|
|
|
{"A":2,"B":5}
|
|
|
|
|
|
|
|
{"A":3,"B":6}
|
r r r
|
r r r r T©r rF z chunksize: )Úobj)r r rI rJ r r! )rF ÚjÚorigÚtestr r r Ú)test_readjson_chunks_multiple_empty_linesï s rq c
|
C s |dkr$d}| j tjj|td¡ t d¡`}| ddd ¡ t |dd d
|
}|
|
d ¡ W 5 Q R X t ||d }t d dgi}t ||¡ W 5 Q R X d S )Nr. rG r/ r^ zlocale.getpreferredencodingc S s dS )NÚcp949r )Ú do_setlocaler r r Ú<lambda> ó z'test_readjson_unicode.<locals>.<lambda>Úwzutf-8)rc u6 {"£©µÃÃÃÃÃéöÿ":["ÐÐÐÐÐабвгдê°"]}r4 u £©µÃÃÃÃÃéöÿu ÐÐÐÐÐабвгдê°)r5 r6 r7 r8 r9 r: r r_ ÚsetattrÚopenÚwriter r r! )r; Z monkeypatchr* r0 r` Úfr" r# r r r Útest_readjson_unicode s ÿ r{ re c C sJ d}t |d| d}tddddgdd d
|
d gd jd |
}t ||¡ d S ) Nú[{"a": 1, "b": 2}
|
{"a": 3, "b": 4}
|
{"a": 5, "b": 6}
|
{"a": 7, "b": 8}T)r re r r
|
r é r r r é r+ )r r Úilocr r! )re r* Újsonlr" r# r r r Útest_readjson_nrows s (r znrows,chunksizerY )r r c C s |dkr$d}| j tjj|td¡ d}t|d|||d}t |¡}W 5 Q R X t ddd d
|
gd d d dgdj
|
d |
}t ||¡ d S )Nr. rG r/ r| T)r re rF r* r r
|
r r} r r r r~ r+ ) r5 r6 r7 r8 r9 r: r rI rJ r r r r! ) r; re rF r* r0 r rL rM r# r r r Útest_readjson_nrows_chunks) s ÿ ÿ(r c C s6 d}d}t jt|d t|dd| d W 5 Q R X d S )Nr| z&nrows can only be passed if lines=TruerO Fr )r re r* )r7 r1 r: r )r* r rR r r r Ú"test_readjson_nrows_requires_linesA s r c
|
C sÌ |dkr$d}| j tjj|td¡ tddggddgdgd td
|
d ggddgdgd td d ggddgdgd g}|dddd}t| ¡ }t |dd|d(}t
|
|D ]\}} t | || ¡ q¤W 5 Q R X d S )Nr. rG r/ r r r r r )r Úindexr
|
r r r r% r& r' r( TrH ) r5 r6 r7 r8 r9 r: r r Úas_urir Ú enumerater r! )
|
r; r, r* r0 Zdf_list_expectedZos_pathZfile_urlZ
|
url_readerr Zchuckr r r Ú"test_readjson_lines_chunks_fileurlM s ÿý r c C sN dd } G dd d}|| }t tt|ddddks<t|jd ksJtd S )
|
Nz\{"a": 1, "b": 2}
|
{"a": 3, "b": 4}
|
{"a": 5, "b": 6}
|
{"a": 7, "b": 8}
|
iè c @ s0 e Zd ZddddZdd ZedddZdS ) z/test_chunksize_is_incremental.<locals>.MyReaderN)Úreturnc S s d| _ t|| _d S )Nr )Ú
|
read_countr Ústringio)ÚselfÚcontentsr r r Ú__init__o s z8test_chunksize_is_incremental.<locals>.MyReader.__init__c W s | j d7 _ | jj| S ©Nr )r r rf )r Úargsr r r rf s s z4test_chunksize_is_incremental.<locals>.MyReader.readc S s | j d7 _ t| jS r )r Úiterr )r r r r Ú__iter__w s z8test_chunksize_is_incremental.<locals>.MyReader.__iter__)Ú__name__Ú
|
__module__Ú __qualname__r rf r r r r r r ÚMyReadern s r Téd rl r é
|
)ÚlenrZ r rA r )r r rL r r r Ú test_chunksize_is_incrementald s üÿ r Úorient_Úsplitr Útablec C sF t ddgddgd}d}tjt|d |jd| d W 5 Q R X d S ) Nr r r r ©Úcol1Úcol2úOmode='a' \(append\) is only supported whenlines is True and orient is 'records'rO )Úmoder ©r r7 r1 r: r )r r rR r r r Útest_to_json_append_orient s
|
ÿr£ c C sH t ddgddgd} d}tjt|d | jddd d
|
W 5 Q R X d S ) Nr r r r r r rO Fr ©r¡ r r r¢ )r rR r r r Útest_to_json_append_lines s
|
ÿr¥ Úmode_ÚrÚxc C sP t ddgddgd}d| d}tjt|d |j| d d
|
d W 5 Q R X d S ) Nr r r r r zmode=z@ is not a valid option.Only 'w' and 'a' are currently supported.rO Fr r¤ r¢ )r¦ r rR r r r Útest_to_json_append_mode s
|
|
ÿr© c C s t ddgddgd} t ddgdd gd}t ddddgdddd gd}t d
|
¡@}| j|d d d |j|dd d d t|d d}t ||¡ W 5 Q R X d S )Nr r r r r r
|
r ÚcÚdr^ Tr r r¤ r ©r r r_ r r r! )Údf1Údf2r# r` r" r r r Ú-test_to_json_append_output_consistent_columns¦ s r¯ c C sª t ddgddgd} t ddgdd gd
|
}t ddd d gddddgd d dd gd }t d ¡B}| j|dd dd |j|dd dd t|d d}t ||¡ W 5 Q R X d S )Nr r r r r Úerz ú!ú#©r Úcol3)r r r´ r^ Tr r¤ r r¬ )r Údf3r# r` r" r r r Ú/test_to_json_append_output_inconsistent_columns¸ s
|
|
|
ýÿ r¶ c C s( t ddgddgd} t ddgdd gd}t d
|
d gd d gd}t dddgi}t ddddd d d d gdddd d
|
d d d gd d d d d d d d gd d d d d d ddgd ddi¡}t d¡f}| j|dddd |j|dddd |j|dddd |j|dddd t|dd}t ||¡ W 5 Q R X d S )Nr r r r r r
|
r rª r« r° rz r± r² r³ Úcol4TF)r r r´ r· Úfloatr^ r r¤ r ©r Zastyper r_ r r r! ©r r® rµ Zdf4r# r` r" r r r Ú,test_to_json_append_output_different_columnsÐ s( üÿù r» c C s( t ddgddgd} t ddgdd gd}t d
|
d gd d gd}t dddgi}t ddd d d d d d gd d d
|
d dd ddgd d d d d d d d gd d d d ddddgd ddi¡}t d¡f}|j|dddd |j|dddd |j|dddd | j|dddd t|dd}t ||¡ W 5 Q R X d S )Nr r r r r r
|
r rª r« r° rz r± r² r³ r· TF)r· r r´ r r¸ r^ r r¤ r r¹ rº r r r Ú6test_to_json_append_output_different_columns_reorderedí s( üÿù r¼ )/r% r Úpathlibr Útypingr r7 ZpandasrI r r Zpandas._testingZ_testingr Zpandas.io.json._jsonr Zfixturer r$ r- r< r? rB rE r8 Z parametrizerN rT rX r] ra ri rk rq r{ r r r r r r£ r¥ r© r¯ r¶ r» r¼ r r r r Ú<module> sX
|
|
|
|
|
|
|
|
|
|
|
|
