U
|
¬ý°d©< ã @ sN d Z ddlmZ ddlmZ ddlmZ ddlZddlZddlZ ddl
|
m Z m Z m Z mZ ddlmZ ddlmZ ddlZdd lmZ dd
|
lmZ d Zd Zd ZdZ dddddddddddddd d d gZ d Z d Z d!Z d"Z!d#e d$e d$e d$e! d% Z"d&e d$e d'Z#d(Z$d)d*d+d,d-Z%d)d.d/d0Z&d1d2 Z'd3d4 Z(G d5d6 d6eej)Z*dS )7a-
|
Read a SAS XPort format file into a Pandas DataFrame.
|
|
Based on code from Jack Cushman (github.com/jcushman/xport).
|
|
The file format is defined here:
|
|
https://support.sas.com/content/dam/SAS/support/en/technical-papers/record-layout-of-a-sas-version-5-or-6-data-set-in-sas-transport-xport-format.pdf
|
é )Ú annotations)Úabc)ÚdatetimeN)ÚCompressionOptionsÚDatetimeNaTTypeÚFilePathÚ
|
ReadBuffer)ÚAppender)Úfind_stack_level)Ú
|
get_handle)Ú
|
ReaderBasezPHEADER RECORD*******LIBRARY HEADER RECORD!!!!!!!000000000000000000000000000000 zKHEADER RECORD*******MEMBER HEADER RECORD!!!!!!!000000000000000001600000000zPHEADER RECORD*******DSCRPTR HEADER RECORD!!!!!!!000000000000000000000000000000 zPHEADER RECORD*******OBS HEADER RECORD!!!!!!!000000000000000000000000000000 ÚntypeZnhfunÚ field_lengthZnvar0ÚnameÚlabelZnformZnflZ num_decimalsZnfjZnfillZniformZniflZnifdZnposÚ_zParameters
|
----------
|
filepath_or_buffer : str or file-like object
|
Path to SAS file or object implementing binary read method.zâindex : identifier of index column
|
Identifier of column that should be used as index of the DataFrame.
|
encoding : str
|
Encoding for text data.
|
chunksize : int
|
Read file `chunksize` lines at a time, returns iterator.zBformat : str
|
File format, only `xport` is currently supported.z\iterator : bool, default False
|
Return XportReader object for reading file incrementally.z#Read a SAS file into a DataFrame.
|
|
Ú
|
a
|
|
Returns
|
-------
|
DataFrame or XportReader
|
|
Examples
|
--------
|
Read a SAS Xport file:
|
|
>>> df = pd.read_sas('filename.XPT')
|
|
Read a Xport file in 10,000 line chunks:
|
|
>>> itr = pd.read_sas('filename.XPT', chunksize=10000)
|
>>> for chunk in itr:
|
>>> do_something(chunk)
|
|
z$Class for reading SAS Xport files.
|
|
z
|
|
Attributes
|
----------
|
member_info : list
|
Contains information about the file
|
fields : list
|
Contains information about the variables in the file
|
zÊRead observations from SAS Xport file, returning as data frame.
|
|
Parameters
|
----------
|
nrows : int
|
Number of rows to read from data file; if None, read whole
|
file.
|
|
Returns
|
-------
|
A DataFrame.
|
Ústrr )ÚdatestrÚreturnc C s. zt | d¡W S tk
|
r( tj Y S X dS )z1Given a date in xport format, return Python date.z%d%b%y:%H:%M:%SN)r ÚstrptimeÚ
|
ValueErrorÚpdZNaT)r © r úNd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\pandas/io/sas/sas_xport.pyÚ _parse_date s r )Úsc C s@ i }d}|D ](\}}| |||
¡ ||< ||7 }q |d= |S )a
|
Parameters
|
----------
|
s: str
|
Fixed-length string to split
|
parts: list of (name, length) pairs
|
Used to break up string, name '_' will be filtered from output.
|
|
Returns
|
-------
|
Dict of name:contents of string at given location.
|
r r )Ústrip)r ÚpartsÚoutÚstartr Úlengthr r r Ú _split_line s
|
r" c C sT |dkrPt t| t d¡¡}t d| dd| ¡}|j|d}| |d< |S | S )Né ZS8ÚSz,S©ÚdtypeÚf0)ÚnpÚzerosÚlenr& Úview)ÚvecÚnbytesÚvec1r& Zvec2r r r Ú_handle_truncated_float_vec° s r/ c C s t d¡}| j|d}|d }|d }|d@ }t jt| t jd}d|t |d@ ¡< d|t |d @ ¡< d
|
|t |d @ ¡< ||L }||? |d @ d d
|
| > B }|dM }||d? d@ d d> | d d> |d@ B O }t jt|fdd}||d< ||d< |jdd}| d¡}|S )zf
|
Parse a vector of float values representing IBM 8 byte floats into
|
native 8 byte floats.
|
z>u4,>u4r% r' Úf1iÿÿÿ é i é i @ é i é é l ÿß é é éA iÿ é l z>f8Zf8) r( r& r+ r) r* Zuint8ÚwhereÚemptyZastype) r, r& r. Zxport1Zxport2Zieee1ÚshiftZieee2Zieeer r r Ú_parse_float_vecà s*
|
ÿ
|
r= c @ s e Zd ZeZd dddddd d
|
Zdd d d Zdd Zdd Zdd ddZ dd ddZ
|
d!dd ddZ dd Z e ed"d dd d d ZdS )#Ú XportReaderNú
|
ISO-8859-1Úinferz FilePath | ReadBuffer[bytes]z
|
str | Noner ÚNone)Úfilepath_or_bufferÚencodingÚ compressionr c C sf || _ d| _|| _|| _t|d|d|d| _| jj| _z | ¡ W n t k
|
r` |
|
¡ Y nX d S )Nr ÚrbF)rC Zis_textrD ) Ú _encodingÚ _lines_readÚ_indexÚ
|
_chunksizer ÚhandlesÚhandlerB Ú _read_headerÚ ExceptionÚclose)ÚselfrB ÚindexrC Ú chunksizerD r r r Ú__init__ÿ s" û
|
zXportReader.__init__)r c C s | j ¡ d S )N)rJ rN ©rO r r r rN s zXportReader.closec C s | j d¡ ¡ S )NéP )rB ÚreadÚdecoderS r r r Ú_get_row s zXportReader._get_rowc C sB | j d¡ | ¡ }|tkr4d|kr,tdtd| ¡ }ddgddgd dgd
|
dgd d gg}t||}|d d kr|tdt|d |d < || _| ¡ }t|d d
|d< | ¡ }| ¡ }| t ¡}|t
|
k} |rØ| sàtdt |dd
}
|
ddgddgddgddgd dgd
|
dgd d gg} t| ¡ | } dd gd
|
d gddgddgg} | t| ¡ | ¡ t| d | d< t| d | d < | | _ ddd} t | ¡ dd
}|
|
| }|d rÂ|d |d 7 }| j |¡}g }d}t||
|
krÌ|d |
|
||
|
d
}}| d ¡}t d |¡}ttt|}|d
|
= | |d |d < |d! }|d dkrv|d"k sb|dkrvd#| d$}t|| ¡ D ]2\}}z| ¡ ||< W n tk
|
r¬ Y nX q~||d! 7 }||g7 }qÖ| ¡ }|tksætd%|| _|| _ | j ¡ | _ | ¡ | _ d&d' | jD | _!d(d' t"| jD }t# $|¡} | | _%d S ))Nr z**COMPRESSED**z<Header record indicates a CPORT file, which is not readable.z#Header record is not an XPORT file.Úprefixr6 Úversionr# ZOSr Úcreatedé zSAS SAS SASLIBz!Header record has invalid prefix.ÚmodifiedzMember header not foundéûÿÿÿéþÿÿÿÚset_nameZsasdatar é( ÚtypeÚnumericÚchar)r1 r2 é6 é: rT é z>hhhh8s40s8shhh2s8shhl52sr r r2 zFloating field width z is not between 2 and 8.z Observation header not found.c S s g | ]}|d ¡ qS )r )rV )Ú.0Úxr r r Ú
|
<listcomp>
s z,XportReader._read_header.<locals>.<listcomp>c S s, g | ]$\}}d t | dt |d fqS )r r$ r )r )rg ÚiÚfieldr r r ri s ÿ)&rB ÚseekrW Ú_correct_line1r r" r Ú file_infoÚ
|
startswithÚ_correct_header1Ú_correct_header2ÚintÚupdateÚ member_inforU r* ÚljustÚstructÚunpackÚdictÚzipÚ
|
_fieldkeysÚ TypeErrorÚitemsr ÚAttributeErrorÚ_correct_obs_headerÚfieldsÚ record_lengthÚtellÚ record_startÚ _record_countÚnobsÚcolumnsÚ enumerater( r& Ú_dtype) rO Zline1Zline2Zfifrn Zline3Zheader1Zheader2Z headflag1Z headflag2ZfieldnamelengthZmemrt ÚtypesZ
|
fieldcountZ
|
datalengthZ fielddatar Z
|
obs_lengthZ
|
fieldbytesZ fieldstructrk ÚflÚmsgÚkÚvÚheaderZdtypelr& r r r rL ! s ÿ"
|
|
ù
|
|
|
|
þ
|
"
|
|
|
þ
|
zXportReader._read_headerz pd.DataFramec C s | j | jp ddS )Nr1 ©Únrows)rU rI rS r r r Ú__next__ s zXportReader.__next__rr c C sÎ | j dd¡ | j ¡ | j }|d dkr:tjdt d | jdkr\| j | j¡ || j S | j dd¡ | j d¡}t j
|
|t j d}t |dk¡}t |dkr¦d}n d t | }| j | j¡ || | j S )
|
zÇ
|
Get number of records in file.
|
|
This is maybe suboptimal because we have to seek to the end of
|
the file.
|
|
Side effect: returns file position to record_start.
|
r r2 rT z xport file may be corrupted.)Ú
|
stackleveli°ÿÿÿr% l @@ r# )rB rl r r ÚwarningsÚwarnr
|
r rU r( Ú
|
frombufferZuint64Z flatnonzeror* )rO Ztotal_records_lengthZlast_card_bytesZ last_cardÚixZtail_padr r r r s& þ
|
|
zXportReader._record_countc C s |dkr| j }| j|dS )a
|
Reads lines from Xport file and returns as dataframe
|
|
Parameters
|
----------
|
size : int, defaults to None
|
Number of lines to read. If None, reads whole file.
|
|
Returns
|
-------
|
DataFrame
|
Nr )rI rU )rO Úsizer r r Ú get_chunk¸ s zXportReader.get_chunkc C sl |j dd}|d dk|d dk@ |d dk@ }|d dk|d d k@ |d d
|
kB |d d kB }||M }|S ) Nz u1,u1,u2,u4r% r0 r Úf2Zf3r' r8 éZ é_ é. )r+ )rO r, r ÚmissZmiss1r r r Ú_missing_doubleÉ s $
|
ÿ
|
þÿzXportReader._missing_doublez
|
int | None)r r c sj |d kr j }t| j j }| j }|dkr> ¡ t j |¡}tj | j
|
|d}i }t j D ]®\}}|dt | } j| d }
|
|
|
dkrÊt| j| d } | ¡} t| } tj| | < n@ j| d dkr
|
dd | D } jd k r
|
fd
|
d | D } | || i¡ qjt |¡} jd krLt t j j| ¡| _n | j¡} j|7 _| S ) Nr )r& Úcountr r rb r rc c S s g | ] }| ¡ qS r )Úrstrip©rg Úyr r r ri ë s z$XportReader.read.<locals>.<listcomp>c s g | ]}| j¡qS r )rV rF r rS r r ri î s ) r ÚminrG r rN Ú StopIterationrB rU r( r r r r
r r r/ r r= ÚnanrF rs r Z DataFramerH ZIndexÚrangerP Z set_index)rO r Ú
|
read_linesÚread_lenÚrawÚdataZdf_dataÚjrh r, r r r Zdfr rS r rU Ô s:
|
|
|
zXportReader.read)Nr? Nr@ )N)N)Ú__name__Ú
|
__module__Ú __qualname__Ú_xport_reader_docÚ__doc__rR rN rW rL r r r r r Ú_read_method_docrU r r r r r> ü s ú n& r> )+r¯ Ú
|
__future__r Ú collectionsr r rv r Únumpyr( Zpandas._typingr r r r Zpandas.util._decoratorsr Zpandas.util._exceptionsr
|
Zpandasr Zpandas.io.commonr Zpandas.io.sas.sasreaderr rm rp rq r~ rz Z_base_params_docZ _params2_docZ_format_params_docZ _iterator_docZ _read_sas_docr® r° r r" r/ r= ÚIteratorr> r r r r Ú<module> sz ÿÿÿÿðþýüûýü 9
|
