U
|
L ±d9! ã @ s< d dl mZmZ ddlmZ ddlmZ G dd deZdS )é )ÚListÚUnioné )Ú CharSetProber)Ú ProbingStatec s e Zd ZdZdZdZdd fdd Zdd fdd Zee dd
|
d Z
|
ee dd d Z e dddZ e dddZedddZedddZedddZedddZee dddd Zee dd d d Zeeef ed d!d"Z eedd#d$Z e dd%d&Z Z S )'Ú UTF1632Proberad
|
This class simply looks for occurrences of zero bytes, and infers
|
whether the file is UTF16 or UTF32 (low-endian or big-endian)
|
For instance, files looking like ( [nonzero] )+
|
have a good probability to be UTF32BE. Files looking like ( [nonzero] )+
|
may be guessed to be UTF16BE, and inversely for little-endian varieties.
|
é g®Gázî?N)Úreturnc sn t ¡ d| _dgd | _dgd | _tj| _ddddg| _d| _ d| _
|
d| _ d| _ d| _ d| _| ¡ d S ©Nr é F)ÚsuperÚ__init__ÚpositionÚ zeros_at_modÚnonzeros_at_modr Ú DETECTINGÚ_stateÚquadÚinvalid_utf16beÚinvalid_utf16leÚinvalid_utf32beÚinvalid_utf32leÚ'first_half_surrogate_pair_detected_16beÚ'first_half_surrogate_pair_detected_16leÚreset©Úself©Ú __class__© úLd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\chardet/utf1632prober.pyr ) s
|
zUTF1632Prober.__init__c sf t ¡ d| _dgd | _dgd | _tj| _d| _d| _ d| _
|
d| _ d| _ d| _ ddddg| _d S r
|
)r r r r r r r r r r r r r r r r r r r r 8 s
|
zUTF1632Prober.resetc C s4 | ¡ r dS | ¡ rdS | ¡ r$dS | ¡ r0dS dS )Nzutf-32bezutf-32lezutf-16bezutf-16lezutf-16)Úis_likely_utf32beÚis_likely_utf32leÚis_likely_utf16beÚis_likely_utf16ler r r r Ú charset_nameF s zUTF1632Prober.charset_namec C s dS )NÚ r r r r r ÚlanguageS s zUTF1632Prober.languagec C s t d| jd S )Nç ð?g @©Úmaxr r r r r Úapprox_32bit_charsW s z UTF1632Prober.approx_32bit_charsc C s t d| jd S )Nr( g @r) r r r r Úapprox_16bit_charsZ s z UTF1632Prober.approx_16bit_charsc C sj | ¡ }|| jkoh| jd | | jkoh| jd | | jkoh| jd | | jkoh| jd | | jkoh| j S ©Nr r é é )r+ ÚMIN_CHARS_FOR_DETECTIONr ÚEXPECTED_RATIOr r ©r Z approx_charsr r r r! ] s
|
ÿþýûz UTF1632Prober.is_likely_utf32bec C sj | ¡ }|| jkoh| jd | | jkoh| jd | | jkoh| jd | | jkoh| jd | | jkoh| j S r- )r+ r0 r r1 r r r2 r r r r" g s
|
ÿþýûz UTF1632Prober.is_likely_utf32lec C sV | ¡ }|| jkoT| jd | jd | | jkoT| jd | jd | | jkoT| j S )Nr r/ r r. )r, r0 r r1 r r r2 r r r r# q s
|
ÿÿþûz UTF1632Prober.is_likely_utf16bec C sV | ¡ }|| jkoT| jd | jd | | jkoT| jd | jd | | jkoT| j S )Nr r. r r/ )r, r0 r r1 r r r2 r r r r$ { s
|
ÿÿþûz UTF1632Prober.is_likely_utf16le)r r c C s¨ |d dksL|d dksL|d dkrR|d dkrRd|d krHdkrRn nd| _ |d dks|d dks|d dkr¤|d dkr¤d|d krdkr¤n nd| _d S )
|
zÖ
|
Validate if the quad of bytes is valid UTF-32.
|
|
UTF-32 is valid in the range 0x00000000 - 0x0010FFFF
|
excluding 0x0000D800 - 0x0000DFFF
|
|
https://en.wikipedia.org/wiki/UTF-32
|
r r é éØ r. éß Tr/ N)r r )r r r r r Úvalidate_utf32_characters
s8
|
|
ÿ
|
þ
|
ý
|
ý ýý
|
|
ÿ
|
þ
|
ý
|
ý ýý
|
z'UTF1632Prober.validate_utf32_characters)Úpairr c C sô | j sNd|d kr dkr*n nd| _ qxd|d krBdkrxn qxd| _n*d|d krfdkrrn nd| _ nd| _| jsÆd|d krdkr¢n nd| _qðd|d krºdkrðn qðd| _n*d|d krÞdkrên nd| _nd| _d S )
|
a9
|
Validate if the pair of bytes is valid UTF-16.
|
|
UTF-16 is valid in the range 0x0000 - 0xFFFF excluding 0xD800 - 0xFFFF
|
with an exception for surrogate pairs, which must be in the range
|
0xD800-0xDBFF followed by 0xDC00-0xDFFF
|
|
https://en.wikipedia.org/wiki/UTF-16
|
r4 r éÛ TéÜ r5 Fr N)r r r r )r r7 r r r Úvalidate_utf16_characters s
|
z'UTF1632Prober.validate_utf16_characters)Úbyte_strr c C s |D ]}| j d }|| j|< |dkrX| | j¡ | | jdd
¡ | | jdd
¡ |dkrt| j| d7 < n| j| d7 < | j d7 _ q| jS )Nr r/ r r. r )r r r6 r: r r Ústate)r r; ÚcZmod4r r r Úfeed» s
|
|
zUTF1632Prober.feedc C sF | j tjtjhkr| j S | ¡ dkr.tj| _ n| jdkr@tj| _ | j S )Ngé?i )r r ÚNOT_MEÚFOUND_ITÚget_confidencer r r r r r< Ê s
|
|
zUTF1632Prober.statec C s( | ¡ s | ¡ s | ¡ s | ¡ r$dS dS )Ng333333ë?g )r$ r# r" r! r r r r rA × s þýüûøz UTF1632Prober.get_confidence) Ú__name__Ú
|
__module__Ú __qualname__Ú__doc__r0 r1 r r ÚpropertyÚstrr% r' Úfloatr+ r, Úboolr! r" r# r$ r Úintr6 r: r ÚbytesÚ bytearrayr r> r< rA Ú __classcell__r r r r r s*
|
|
|
|
r N)Útypingr r Ú charsetproberr Úenumsr r r r r r Ú<module> s
|
