U
|
L ±d, ã @ sL d dl Z d dlZd dlmZmZ ddlmZmZ e d¡Z G dd dZ
|
dS )é N)ÚOptionalÚUnioné )ÚLanguageFilterÚ ProbingStates% [a-zA-Z]*[-ÿ]+[a-zA-Z]*[^a-zA-Z-ÿ]?c @ sì e Zd ZdZejfeddddZddddZee e
|
dd d
|
Z ee e
|
dd d Z e eef ed ddZeedddZedddZee eef edddZee eef edddZee eef edddZdS )Ú CharSetProbergffffffî?N)Ú lang_filterÚreturnc C s$ t j| _d| _|| _t t¡| _d S )NT) r Ú DETECTINGÚ_stateÚactiver ÚloggingÚ getLoggerÚ__name__Úlogger)Úselfr © r úLd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\chardet/charsetprober.pyÚ__init__, s zCharSetProber.__init__)r c C s t j| _d S ©N)r r
|
r ©r r r r Úreset2 s zCharSetProber.resetc C s d S r r r r r r Ú charset_name5 s zCharSetProber.charset_namec C s t d S r ©ÚNotImplementedErrorr r r r Úlanguage9 s zCharSetProber.language)Úbyte_strr c C s t d S r r )r r r r r Úfeed= s zCharSetProber.feedc C s | j S r )r r r r r Ústate@ s zCharSetProber.statec C s dS )Ng r r r r r Úget_confidenceD s z CharSetProber.get_confidence)Úbufr c C s t dd| ¡} | S )Ns ([ -])+ó )ÚreÚsub)r r r r Úfilter_high_byte_onlyG s z#CharSetProber.filter_high_byte_onlyc C sZ t }t | ¡}|D ]@}| |dd
¡ |dd
}| ¡ sJ|dk rJd}| |¡ q|S )u7
|
We define three types of bytes:
|
alphabet: english alphabets [a-zA-Z]
|
international: international characters [Â-ÿ]
|
marker: everything else [^a-zA-ZÂ-ÿ]
|
The input buffer can be thought to contain a series of words delimited
|
by markers. This function works to filter all words that contain at
|
least one international character. All contiguous sequences of markers
|
are replaced by a single space ascii character.
|
This filter applies to all scripts which do not use English characters.
|
Néÿÿÿÿó r! )Ú bytearrayÚINTERNATIONAL_WORDS_PATTERNÚfindallÚextendÚisalpha)r ÚfilteredÚwordsÚwordÚ last_charr r r Úfilter_international_wordsL s
|
z(CharSetProber.filter_international_wordsc C s t }d}d}t| d¡} t| D ]R\}}|dkrB|d }d}q$|dkr$||krr|sr| | ||
¡ | d¡ d}q$|s| | |d
¡ |S )
|
a[
|
Returns a copy of ``buf`` that retains only the sequences of English
|
alphabet and high byte characters that are not between <> characters.
|
This filter can be applied to all scripts which contain both English
|
characters and extended ASCII characters, but is currently only used by
|
``Latin1Prober``.
|
Fr Úcó >r ó <r! TN)r' Ú
|
memoryviewÚcastÚ enumerater* )r r, Úin_tagÚprevÚcurrÚbuf_charr r r Úremove_xml_tagsn s
|
z CharSetProber.remove_xml_tags)r Ú
|
__module__Ú __qualname__ÚSHORTCUT_THRESHOLDr ÚNONEr r Úpropertyr Ústrr r r Úbytesr' r r r Úfloatr Ú staticmethodr$ r0 r; r r r r r ( s" !r ) r r" Útypingr r Úenumsr r Úcompiler( r r r r r Ú<module> s ÿ
|
