U
|
M ±dêJ ã @ s\ d dl Z d dlmZ d dlmZmZmZmZmZ ddl m
|
Z
|
m Z m Z m Z ddlmZmZmZmZ ddlmZ ddlmZmZ dd lmZmZmZm Z m Z m Z m Z e d
|
¡Z!e "¡ Z#e# $e %d ¡¡ d e&e'e'e(eee) eee) e*e*e(ed
|
ddZ+d ee'e'e(eee) eee) e*e*e(ed
|
ddZ,d de'e'e(eee) eee) e*e*e(ed
|
ddZ-dS ) é N)ÚPathLike)ÚAnyÚBinaryIOÚListÚOptionalÚSeté )Úcoherence_ratioÚencoding_languagesÚmb_encoding_languagesÚmerge_coherence_ratios)ÚIANA_SUPPORTEDÚTOO_BIG_SEQUENCEÚTOO_SMALL_SEQUENCEÚTRACE)Ú
|
mess_ratio)Ú CharsetMatchÚCharsetMatches)Úany_specified_encodingÚcut_sequence_chunksÚ iana_nameÚidentify_sig_or_bomÚ is_cp_similarÚis_multi_byte_encodingÚshould_strip_sig_or_bomZcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)sé é çÉ?TFç¹?)
|
Ú sequencesÚstepsÚ
|
chunk_sizeÚ thresholdÚ cp_isolationÚ cp_exclusionÚpreemptive_behaviourÚexplainÚlanguage_thresholdÚreturnc . C s t | ttfs td t| ¡|r>tj} t t ¡ t
|
t ¡ t | }
|
|
|
dkrt d¡ |rvt t ¡ t
|
| prtj¡ tt| dddg dgS |dk rºt t d d
|
|¡¡ d d |D }ng }|dk rêt t d d
|
|¡¡ dd |D }ng }|
|
|| krt t d|||
|
¡ d}|
|
}|dkr:|
|
| |k r:t|
|
| }t | tk } t | tk} | rlt t d |
|
¡¡ n| rt t d |
|
¡¡ g } |rt| nd}|dk r¼| |¡ t t d|¡ t }g }g }d}d}d}t }t| \}}|dk r| |¡ t t dt ||¡ | d¡ d| kr.| d¡ | t D ]Ì}|rP||krPq6|rd||krdq6||krrq6| |¡ d}||k}|ot |}|dkr¸|s¸t t d|¡ q6|dkrÚ|sÚt t d|¡ q6z t |} W n. t t!fk
|
r t t d|¡ Y q6Y nX zr| r`| dkr`t"|dkrD| dtd
n| t |td
|d n&t"|dkrp| n| t |d
|d }W n\ t#t$fk
|
rä } z8t | t$sÀt t d |t"| ¡ | |¡ W Y ¢q6W 5 d} ~ X Y nX d} |D ] } t%|| rîd } q qî| r&t t d || ¡ q6t&|s2dnt ||
|
t|
|
| } | ob|dk obt ||
|
k }!|!rxt t d |¡ tt | d! }"t'|"d"}"d}#d}$g }%g }&zt(| || ||||| | D ]|}'|% |'¡ |& t)|'||d koúdt | koöd"kn ¡ |&d# |kr|#d7 }#|#|"ks0|r¼|dkr¼ q:q¼W n@ t#k
|
r| } z t t d$|t"| ¡ |"}#d }$W 5 d} ~ X Y nX |$sü| rü| süz | td%d
j*|d&d' W nL t#k
|
rú } z,t t d(|t"| ¡ | |¡ W Y ¢q6W 5 d} ~ X Y nX |&rt+|&t |& nd}(|(|ks*|#|"kr¢| |¡ t t d)||#t,|(d* d+d,¡ |dd|fkr6|$s6t| ||dg |})||kr|)}n|dkr|)}n|)}q6t t d-|t,|(d* d+d,¡ | sÎt-|}*nt.|}*|*rôt t d. |t"|*¡¡ g }+|dkr4|%D ],}'t/|'||*r d/ |*¡nd},|+ |,¡ qt0|+}-|-rVt t d0 |-|¡¡ | t| ||(||-|¡ ||ddfkrÀ|(d1k rÀt d2|¡ |r®t t ¡ t
|
| ¡ t|| g S ||kr6t d3|¡ |rðt t ¡ t
|
| ¡ t|| g S q6t |dkr¶|s&|s&|r2t t d4¡ |rRt d5|j1¡ | |¡ nd|rb|dks|r||r||j2|j2ks|dk rt d6¡ | |¡ n|r¶t d7¡ | |¡ |rÚt d8| 3¡ j1t |d ¡ n
|
t d9¡ |rþt t ¡ t
|
| ¡ |S ):af
|
Given a raw bytes sequence, return the best possibles charset usable to render str objects.
|
If there is no results, it is a strong indicator that the source is binary/not text.
|
By default, the process will extract 5 blocks of 512o each to assess the mess and coherence of a given sequence.
|
And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will.
|
|
The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page
|
but never take it for granted. Can improve the performance.
|
|
You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that
|
purpose.
|
|
This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32.
|
By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain'
|
toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging.
|
Custom logging format and handler can be set manually.
|
z4Expected object of type bytes or bytearray, got: {0}r z<Encoding detection on empty bytes, assuming utf_8 intention.Úutf_8g FÚ Nz`cp_isolation is set. use this flag for debugging purpose. limited list of encoding allowed : %s.z, c S s g | ]}t |d qS ©F©r ©Ú.0Úcp© r0 úMd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\charset_normalizer/api.pyÚ
|
<listcomp>Z s z from_bytes.<locals>.<listcomp>zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.c S s g | ]}t |d qS r+ r, r- r0 r0 r1 r2 e s z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.r z>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.Úascii> Úutf_16Úutf_32z\Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.> Úutf_7zREncoding %s won't be tested as-is because detection is unreliable without BOM/SIG.z2Encoding %s does not provide an IncrementalDecoderg A)Úencodingz9Code page %s does not fit given bytes sequence at ALL. %sTzW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.é é éÿÿÿÿzaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sg jè@Ústrict)Úerrorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.éd é )Úndigitsz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}ú,z We detected language {} using {}r z.Encoding detection: %s is most likely the one.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)4Ú
|
isinstanceÚ bytearrayÚbytesÚ TypeErrorÚformatÚtypeÚloggerÚlevelÚ
|
addHandlerÚexplain_handlerÚsetLevelr ÚlenÚdebugÚ removeHandlerÚloggingÚWARNINGr r ÚlogÚjoinÚintr r r ÚappendÚsetr r Úaddr r ÚModuleNotFoundErrorÚ ImportErrorÚstrÚUnicodeDecodeErrorÚ LookupErrorr ÚrangeÚmaxr r ÚdecodeÚsumÚroundr
|
r r r r7 Ú fingerprintÚbest).r r r! r" r# r$ r% r& r' Zprevious_logger_levelÚlengthZis_too_small_sequenceZis_too_large_sequenceZprioritized_encodingsZspecified_encodingZtestedZtested_but_hard_failureZtested_but_soft_failureZfallback_asciiZ fallback_u8Zfallback_specifiedÚresultsZ sig_encodingZ sig_payloadZ encoding_ianaZdecoded_payloadZbom_or_sig_availableZstrip_sig_or_bomZis_multi_byte_decoderÚeZsimilar_soft_failure_testZencoding_soft_failedZr_Zmulti_byte_bonusZmax_chunk_gave_upZearly_stop_countZlazy_str_hard_failureZ md_chunksZ md_ratiosÚchunkZmean_mess_ratioZfallback_entryZtarget_languagesZ cd_ratiosZchunk_languagesZcd_ratios_mergedr0 r0 r1 Ú
|
from_bytes! sî ÿÿ
|
|
|
|
üüû ÿþÿþÿ
|
|
ý
|
|
ü
|
|
|
|
|
ÿýý ý ÿüÿü
|
ü
|
|
ü
|
ýÿ
|
ýü
|
÷
|
&ýÿ
|
ÿÿÿüÿþý ü
|
|
ú
|
ÿþ ÿ
|
|
ü
|
ÿþ
|
ý ÿþúÿ ÿþ ÿ
|
|
|
ý
|
|
þþ ÿÿýü
|
ûù
|
|
|
|
ý
|
|
|
rg )
|
Úfpr r! r" r# r$ r% r& r' r( c
|
C s t | ¡ |||||||| S )z
|
Same thing than the function from_bytes but using a file pointer that is already ready.
|
Will not close the file pointer.
|
)rg Úread) rh r r! r" r# r$ r% r& r' r0 r0 r1 Úfrom_fpô s ÷rj z PathLike[Any])
|
Úpathr r! r" r# r$ r% r& r' r( c
|
C s: t | d&} t| |||||||| W 5 Q R £ S Q R X dS )z
|
Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode.
|
Can raise IOError.
|
ÚrbN)Úopenrj )
|
rk r r! r" r# r$ r% r& r' rh r0 r0 r1 Ú from_path s ÷rn )r r r NNTFr )r r r NNTFr )r r r NNTFr ).rO Úosr Útypingr r r r r Zcdr r
|
r r Zconstantr r r r Zmdr Úmodelsr r Úutilsr r r r r r r Ú getLoggerrG Ú StreamHandlerrJ Ú setFormatterÚ FormatterrC rS ÚfloatrY Úboolrg rj rn r0 r0 r0 r1 Ú<module> s $
|
ÿ ÷
|
|
ö X ÷
|
|
ö ÷
|
|
ö
|
