U
|
M ±d2 ã @ s° d dl Z d dlmZ d dlmZ d dlmZ d dlmZm Z m
|
Z
|
m Z m Z ddl mZ ddlmZmZmZmZ dd lmZ dd
|
lmZ dd lmZmZmZm Z m Z e e
|
e d d dZ e e
|
e dddZ e e e
|
e d ddZ!e e e
|
e d ddZ"eede e e#e#f dddZ$d.e
|
e e#e
|
e dd d Z%e e
|
e e&d d d Z'e e
|
e d!d"d#Z(e
|
e ed$d%d&Z)eed$d'd(Z*ed)dd/e e&e e ed+d,d-Z+dS )0é N)ÚIncrementalDecoder)ÚCounter)Ú lru_cache)r ÚDictÚListÚOptionalÚTupleé )Ú FREQUENCIES)ÚKO_NAMESÚLANGUAGE_SUPPORTED_COUNTÚTOO_SMALL_SEQUENCEÚZH_NAMES)Ú is_suspiciously_successive_range)ÚCoherenceMatches)Úis_accentuatedÚis_latinÚis_multi_byte_encodingÚis_unicode_range_secondaryÚ unicode_range)Ú iana_nameÚreturnc s¶ t | rtdt d | ¡¡j}|dd}i d tddD ]^}| t|g¡}|r>t |}|dkrhq>t
|
|d kr>|krd|< | d
|
7 < d
|
7 q>t fd d D S ) zF
|
Return associated unicode ranges in a single byte code page.
|
z.Function not supported on multi-byte code pagez encodings.{}Úignore)Úerrorsr é@ éÿ NFr c s g | ]}| d kr|qS )g333333Ã?© )Ú.0Úcharacter_range©Úcharacter_countZ seen_rangesr úLd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\charset_normalizer/cd.pyÚ
|
<listcomp>3 s þz*encoding_unicode_range.<locals>.<listcomp>) r ÚIOErrorÚ importlibÚ import_moduleÚformatr ÚrangeÚdecodeÚbytesr r Úsorted)r ÚdecoderÚpÚiÚchunkr r r r! Úencoding_unicode_range s0 ÿ
|
|
þÿr/ )Ú primary_ranger c C s> g }t ¡ D ],\}}|D ] }t|| kr| |¡ q qq |S )z>
|
Return inferred languages used with a unicode range.
|
)r
|
Úitemsr Úappend)r0 Ú languagesÚlanguageÚ
|
charactersÚ characterr r r! Úunicode_range_languages; s
|
r7 c C s< t | }d}|D ]}d|kr|} q&q|dkr4dgS t|S )z
|
Single-byte encoding language association. Some code page are heavily linked to particular language(s).
|
This function does the correspondence.
|
NZLatinú Latin Based)r/ r7 )r Zunicode_rangesr0 Zspecified_ranger r r! Úencoding_languagesJ s r9 c C s` | d¡s&| d¡s&| d¡s&| dkr,dgS | d¡s>| tkrDdgS | d¡sV| tkr\d gS g S )
|
z
|
Multi-byte encoding language association. Some code page are heavily linked to particular language(s).
|
This function does the correspondence.
|
Zshift_Ú
|
iso2022_jpZeuc_jÚcp932ÚJapaneseÚgbÚChineseÚ
|
iso2022_krÚKorean)Ú
|
startswithr r )r r r r! Úmb_encoding_languages^ s ÿþýürB )Úmaxsize)r4 r c C sB d}d}t | D ](}|s$t|r$d}|rt|dkrd}q||fS )zg
|
Determine main aspects from a supported language if it contains accents and if is pure Latin.
|
FT)r
|
r r )r4 Útarget_have_accentsÚtarget_pure_latinr6 r r r! Úget_target_featuress s rF F)r5 Úignore_non_latinr c s¬ g }t dd D }t ¡ D ]l\}}t|\}}|r@|dkr@q |dkrN|rNq t|}t fdd|D } | | }
|
|
|
dkr | ||
|
f¡ q t|dd d d
|
}d d |D S ) zE
|
Return associated languages associated to given characters.
|
c s s | ]}t |V qd S ©N)r )r r6 r r r! Ú <genexpr> s z%alphabet_languages.<locals>.<genexpr>Fc s g | ]}| kr|qS r r )r Úc©r5 r r! r" s z&alphabet_languages.<locals>.<listcomp>gÉ?c S s | d S ©Nr r ©Úxr r r! Ú<lambda>¢ ó z$alphabet_languages.<locals>.<lambda>T©ÚkeyÚreversec S s g | ] }|d qS )r r )r Zcompatible_languager r r! r" ¤ s )Úanyr
|
r1 rF Úlenr2 r* ) r5 rG r3 Zsource_have_accentsr4 Zlanguage_charactersrD rE r Zcharacter_match_countÚratior rK r! Úalphabet_languages s" ÿrW )r4 Úordered_charactersr c C s¦ | t krtd | ¡d}tt | }t|}tt | }|dk}t|td|D ]D\}}||krfqRt | |¡} || }
|
t||
|
} |dkr¢t | | dkr¢qR|dkrÈt | | |d k rÈ|d7 }qRt | d|
} t | | d
} |d|
}||d
}tt|t| @ }tt|t| @ }t| dkrJ|dkrJ|d7 }qRt| dkrl|dkrl|d7 }qR|t| d
|
ks|t| d
|
krR|d7 }qRqR|t| S ) aN
|
Determine if a ordered characters list (by occurrence from most appearance to rarest) match a particular language.
|
The result is a ratio between 0. (absolutely no correspondence) and 1. (near perfect fit).
|
Beware that is function is not strict on the match in order to ease the detection. (Meaning close match is 1.)
|
z{} not availabler é Fé Té r NgÙ?)
|
r
|
Ú
|
ValueErrorr& ÚsetrU Úzipr' ÚindexÚintÚabs)r4 rX Zcharacter_approved_countZFREQUENCIES_language_setZordered_characters_countZ target_language_characters_countZlarge_alphabetr6 Zcharacter_rankZcharacter_rank_in_languageZexpected_projection_ratioZcharacter_rank_projectionZcharacters_before_sourceZcharacters_after_sourceZcharacters_beforeZcharacters_afterZbefore_match_countZafter_match_countr r r! Ú characters_popularity_compare§ st ÿÿ ÿþÿ
|
ÿþ ÿÿ ÿÿÿþrb )Údecoded_sequencer c C s i }| D ]~}| ¡ dkrqt|}|dkr,qd}|D ]}t||dkr4|} qPq4|dkr\|}||krr| ¡ ||< q|| | ¡ 7 < qt| ¡ S )a
|
Given a decoded text sequence, return a list of str. Unicode range / alphabet separation.
|
Ex. a text containing English/Latin with a bit a Hebrew will return two items in the resulting list;
|
One containing the latin letters and the other hebrew.
|
FN)Úisalphar r ÚlowerÚlistÚvalues)rc Zlayersr6 r Zlayer_target_rangeZdiscovered_ranger r r! Úalpha_unicode_split÷ s, ÿÿ rh )Úresultsr c sf i | D ]8}|D ].}|\}}| kr0|g |< q | |¡ qq fdd D }t|dd ddS )z
|
This function merge results previously given by the function coherence_ratio.
|
The return type is the same as coherence_ratio.
|
c s. g | ]&}|t t | t | d fqS )rZ )ÚroundÚsumrU )r r4 ©Zper_language_ratiosr r! r" , s úþþz*merge_coherence_ratios.<locals>.<listcomp>c S s | d S rL r rM r r r! rO 7 rP z(merge_coherence_ratios.<locals>.<lambda>TrQ )r2 r* )ri ÚresultZ
|
sub_resultr4 rV Úmerger rl r! Úmerge_coherence_ratios s
|
|
ø ro c s t | D ]6}|\}}| dd¡}| kr2g |< | |¡ q
|
t fdd D rg } D ]}| |t | f¡ q`|S | S )u³
|
We shall NOT return "Englishâ" in CoherenceMatches because it is an alternative
|
of "English". This function only keeps the best match and remove the em-dash in it.
|
u âÚ c 3 s | ]}t | d kV qdS )r N)rU )r Úe©Z index_resultsr r! rI J s z/filter_alt_coherence_matches.<locals>.<genexpr>)ÚdictÚreplacer2 rT Úmax)ri rm r4 rV Z
|
no_em_nameZfiltered_resultsr rr r! Ú filter_alt_coherence_matches: s rv i ç¹?)rc Ú thresholdÚ lg_inclusionr c C sð g }d}d}|dk r | d¡ng }d|kr8d}| d¡ t| D ]}t|}| ¡ } tdd | D }
|
|
|
tkrpq@d d
|
| D } |pt| |D ]J} t| | } | |k r¦qn| d kr¶|d 7 }| | t
|
| d f¡ |dkr q@qq@t t |dd ddS )z¨
|
Detect ANY language that can be identified in given sequence. The sequence will be analysed by layers.
|
A layer = Character extraction by alphabets/ranges.
|
Fr Nú,r8 Tc s s | ]\}}|V qd S rH r ©r rJ Úor r r! rI l s z"coherence_ratio.<locals>.<genexpr>c S s g | ] \}}|qS r r r{ r r r! r" q s z#coherence_ratio.<locals>.<listcomp>gé?r rZ r[ c S s | d S rL r rM r r r! rO
rP z!coherence_ratio.<locals>.<lambda>rQ ) ÚsplitÚremoverh r Ú most_commonrk r rW rb r2 rj r* rv )rc rx ry ri rG Zsufficient_match_countZlg_inclusion_listZlayerZsequence_frequenciesr r Zpopular_character_orderedr4 rV r r r! Úcoherence_ratioU sD
|
ÿ ÿ ÿr )F)rw N),r$ Úcodecsr Ú collectionsr Ú functoolsr ÚtypingZ TypeCounterr r r r Zassetsr
|
Zconstantr r r r Zmdr Úmodelsr Úutilsr r r r r Ústrr/ r7 r9 rB ÚboolrF rW Úfloatrb rh ro rv r r r r r! Ú<module> sN ' ÿ þ $ þ P' ÿ þ
|
