U
|
M ±d¶. ã @ sb d dl Z d dlZd dlZd dlmZ d dlmZ d dlmZ d dl m
|
Z
|
d dl m Z m Z mZmZmZmZ d dlmZ dd lmZmZmZmZmZmZ eed
|
ee d d d Z eed
|
eed ddZ eed
|
eee d ddZ eed
|
ee d ddZ eed
|
ee d ddZ!eed
|
ee d ddZ"eed
|
ee d ddZ#eed
|
ee d ddZ$eed
|
ee d d d Z%eed
|
ee d d d Z&ee d d d!Z'eed
|
ee d d"d#Z(eed
|
ee d d$d%Z)eed
|
ee d d&d'Z*eed
|
ee d d(d)Z+eed
|
ee d d*d+Z,ee-ed
|
ee d,d-d.Z.eed
|
ee d d/d0Z/dSe0e1ee d2d3d4Z2ed5d
|
ee d6d7d8Z3e0eee e0f d9d:d;Z4ee d<d=d>Z5dTee ed@dAdBZ6ee e dCdDdEZ7eee8dFdGdHZ9eee dFdIdJZ:dKej;dLfee1eddMdNdOZ<dUe0ee=e1e e e0e ee e eddf dP
|
dQdRZ>dS )Vé N)ÚIncrementalDecoder)Úaliases)Ú lru_cache)Úfindall)Ú GeneratorÚListÚOptionalÚSetÚTupleÚUnion)ÚMultibyteIncrementalDecoderé )ÚENCODING_MARKSÚIANA_SUPPORTED_SIMILARÚ RE_POSSIBLE_ENCODING_INDICATIONÚUNICODE_RANGES_COMBINEDÚ UNICODE_SECONDARY_RANGE_KEYWORDÚUTF8_MAXIMAL_ALLOCATION)Úmaxsize)Ú characterÚreturnc C sV zt | ¡}W n tk
|
r$ Y dS X d|kpTd|kpTd|kpTd|kpTd|kpTd|kS )NFz
|
WITH GRAVEz
|
WITH ACUTEz WITH CEDILLAzWITH DIAERESISzWITH CIRCUMFLEXz
|
WITH TILDE©Ú unicodedataÚnameÚ
|
ValueError©r Ú description© r úOd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\charset_normalizer/utils.pyÚis_accentuated s ÿþýüúr c C s. t | ¡}|s| S | d¡}tt|d dS )Nú r é )r Ú decompositionÚsplitÚchrÚint)r Z
|
decomposedÚcodesr r r Ú remove_accent& s
|
|
|
r' c C s. t | }t ¡ D ]\}}||kr| S qdS )zK
|
Retrieve the Unicode range official name from a single character.
|
N)Úordr Úitems)r Z character_ordÚ
|
range_nameZ ord_ranger r r Ú unicode_range1 s
|
|
r+ c C s. zt | ¡}W n tk
|
r$ Y dS X d|kS )NFZLATINr r r r r Úis_latin? s
|
r, c C s* z| d¡ W n tk
|
r$ Y dS X dS )NÚasciiFT)ÚencodeÚUnicodeEncodeError©r r r r Úis_asciiH s
|
r1 c C s2 t | ¡}d|krdS t| }|d kr*dS d|kS )NÚPTFZ Punctuation©r Úcategoryr+ ©r Úcharacter_categoryÚcharacter_ranger r r Úis_punctuationQ s
|
r8 c C s: t | ¡}d|ksd|kr dS t| }|d kr2dS d|kS )NÚSÚNTFZFormsr3 r5 r r r Ú is_symbol` s
|
r; c C s t | }|d krdS d|kS )NFZ Emoticons)r+ )r r7 r r r Ú is_emoticono s r< c C s& | ¡ s| dkrdS t | ¡}d|kS )N> ú+ú>ú<ú;ú,õ ï½TÚZ)Úisspacer r4 ©r r6 r r r Ú is_separatory s
|
rF c C s | ¡ | ¡ kS ©N)ÚislowerÚisupperr0 r r r Úis_case_variable s rJ c C s t | ¡}|dkS )NÚCo)r r4 rE r r r Úis_private_use_only s
|
rL c C s. zt | ¡}W n tk
|
r$ Y dS X d|kS )NFÚCJKr ©r Zcharacter_namer r r Úis_cjk s
|
rO c C s. zt | ¡}W n tk
|
r$ Y dS X d|kS )NFZHIRAGANAr rN r r r Ú is_hiragana s
|
rP c C s. zt | ¡}W n tk
|
r$ Y dS X d|kS )NFZKATAKANAr rN r r r Ú is_katakana¢ s
|
rQ c C s. zt | ¡}W n tk
|
r$ Y dS X d|kS )NFZHANGULr rN r r r Ú is_hangul¬ s
|
rR c C s. zt | ¡}W n tk
|
r$ Y dS X d|kS )NFZTHAIr rN r r r Úis_thai¶ s
|
rS )r* r c s t fddtD S )Nc 3 s | ]}| kV qd S rG r )Ú.0Úkeyword©r* r r Ú <genexpr> s z-is_unicode_range_secondary.<locals>.<genexpr>)Úanyr rV r rV r Úis_unicode_range_secondaryÀ s rY c C s( | ¡ dko&| ¡ dko&| dko&| dkS )NFúu )rD Ú isprintabler0 r r r Úis_unprintableÅ s
|
ÿþür\ é )ÚsequenceÚ search_zoner c C s t | tstt| }tt| dt||
jddd}t|dkrHdS |D ]N}| ¡ dd¡}t
|
¡ D ]0\}}||kr| S ||krh| S qhqLdS )zW
|
Extract using ASCII-only decoder any specified encoding in the first n-bytes.
|
Nr- Úignore©Úerrorsr ú-Ú_) Ú
|
isinstanceÚbytesÚ TypeErrorÚlenr r ÚminÚdecodeÚlowerÚreplacer r) )r^ r_ Zseq_lenÚresultsZspecified_encodingÚencoding_aliasÚ encoding_ianar r r Úany_specified_encodingÐ s"
|
þ rp é )r r c C s | dkp t t d | ¡¡jtS )zQ
|
Verify is a specific encoding is a multi byte one based on it IANA name
|
> Ú utf_16_leÚutf_7Úutf_16Ú utf_32_leÚutf_32Úutf_8Ú utf_8_sigÚ utf_16_beÚ utf_32_beú encodings.{})Ú
|
issubclassÚ importlibÚ import_moduleÚformatr r )r r r r Úis_multi_byte_encodingð s
|
þör )r^ r c C sJ t D ]@}t | }t|tr |g}|D ] }| |¡r$||f S q$qdS )z9
|
Identify and extract SIG/BOM in given sequence.
|
)Nó )r re rf Ú
|
startswith)r^ Ú iana_encodingZmarksÚmarkr r r Úidentify_sig_or_bom s
|
|
r
)r r c C s | dkS )N> rt rv r )r r r r Úshould_strip_sig_or_bom s r T)Úcp_nameÚstrictr c C sL | ¡ dd¡} t ¡ D ] \}}| ||fkr| S q|rHtd | ¡| S )Nrc rd z Unable to retrieve IANA for '{}')rk rl r r) r r )r r rn ro r r r Ú iana_name s
|
r )Údecoded_sequencer c C s4 t }| D ] }t|}|d kr q
|
| |¡ q
|
t|S rG )Úsetr+ ÚaddÚlist)r Úrangesr r7 r r r Ú
|
range_scan+ s r )Ú iana_name_aÚ iana_name_br c C s t | st |rdS t d | ¡¡j}t d |¡¡j}|dd}|dd}d}tdD ]*}t|g}| |¡| |¡krX|d7 }qX|d S ) Ng r{ r` ra r éÿ r éþ )r r} r~ r r Úrangerf rj ) r r Z decoder_aZ decoder_bZid_aZid_bZcharacter_match_countÚiZ to_be_decodedr r r Ú cp_similarity9 s ÿÿ
|
|
|
|
r c C s | t ko|t | kS )z
|
Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using
|
the function cp_similarity.
|
)r )r r r r r Ú is_cp_similarQ s
|
þr Zcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s)r ÚlevelÚ format_stringr c C s: t | ¡}| |¡ t ¡ }| t |¡¡ | |¡ d S rG )ÚloggingÚ getLoggerÚsetLevelÚ StreamHandlerÚ setFormatterÚ FormatterÚ
|
addHandler)r r r ÚloggerÚhandlerr r r Úset_logging_handler\ s
|
|
|
r£ )
|
Ú sequencesro ÚoffsetsÚ
|
chunk_sizeÚbom_or_sig_availableÚstrip_sig_or_bomÚ sig_payloadÚis_multi_byte_decoderÚdecoded_payloadr c c s* |r6|dkr6|D ]"} || | |
}
|
|
|
s, q4|
|
V qnð|D ]ê} | | } | t | d krXq:| | | |
} |r||dkr||| } | j||rdndd}
|
|r | dkr t|d} |r |
|
d |
|kr t| | d d D ]H}| ||
} |rø|dkrø|| } | j|dd}
|
|
|
d |
|krÔ q qÔ|
|
V q:d S )
|
NFé r` r ra r r! é éÿÿÿÿ)rh rj ri r )r¤ ro r¥ r¦ r§ r¨ r© rª r« r ÚchunkZ chunk_endZ cut_sequenceZchunk_partial_size_chkÚjr r r Úcut_sequence_chunksi s>
|
|
þ
|
ÿþ r± )r] )T)N)?r} r r Úcodecsr Zencodings.aliasesr Ú functoolsr Úrer Útypingr r r r r
|
r Ú_multibytecodecr Zconstantr r r r r r ÚstrÚboolr r' r+ r, r1 r8 r; r< rF rJ rL rO rP rQ rR rS rh rY r\ rf r% rp r r
r r r Úfloatr r ÚINFOr£ r r± r r r r Ú<module> s
|
|
|
ýü ÷ ö
|
