U
|
¡ý°dÆ ã @ s^ d Z dZddlmZ ddlmZ ddlZddlZddlZddl Z dZ
|
z ddl Z
|
W n\ e k
|
r¬ z ddl Z
|
W n: e k
|
r¦ z ddlZ
|
W n e k
|
r dZ
|
Y nX Y nX Y nX e
|
r¼dd Zndd Zd Zd
|
Ze Ze e d ¡ej¡e e d ¡ej¡d ee< e eej¡e eej¡d ee< dd lmZ G dd deZG dd dZ G dd dZ dS )aB Beautiful Soup bonus library: Unicode, Dammit
|
|
This library converts a bytestream to Unicode through any means
|
necessary. It is heavily based on code from Mark Pilgrim's Universal
|
Feed Parser. It works best on XML and HTML, but it does not rewrite the
|
XML or HTML to reflect a new encoding; that's the tree builder's job.
|
ÚMITé )Úcodepoint2name)Ú defaultdictNc C s t | trd S t | ¡d S )NÚencoding)Ú
|
isinstanceÚstrÚchardet_moduleÚdetect©Ús© r úAd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\bs4/dammit.pyÚchardet_dammit+ s
|
r c C s d S ©Nr r
|
r r r r 0 s z$^\s*<\?.*encoding=['"](.*?)['"].*\?>z0<\s*meta[^>]+charset\s*=\s*["']?([^>]*?)[ /;'">]Úascii)ÚhtmlÚxml)Úhtml5c @ s e Zd ZdZdd Ze \ZZZdddddd Ze
|
d
|
¡Z e
|
d ¡Z e d d Ze dd Ze dd Ze dddZe dddZe dd ZdS ) ÚEntitySubstitutionzFThe ability to substitute XML or HTML entities for certain characters.c C s i } i }t }tt }tt ¡ D ]¨\}}| d¡rB|dd
}n|}||krV|||< || |< t|dkrt|dk r|dkrq"t|dkr tdd |D r q"t|dkr¸| |¡ q"||d |¡ q"t }|D ]B}|| } | sò| |¡ qÖd
|
|
d d | D ¡}
|
| d ||
|
f ¡ qÖt | ¡ D ] } | D ]} | | ¡ q.q&dd
|
|¡ } t t ¡ D ]\}}t|}|| |< q^| |t | ¡fS )u¤ Initialize variables used by this class to manage the plethora of
|
HTML5 named entities.
|
|
This function returns a 3-tuple containing two dictionaries
|
and a regular expression:
|
|
unicode_to_name - A mapping of Unicode strings like "⦨" to
|
entity names like "angmsdaa". When a single Unicode string has
|
multiple entity names, we try to choose the most commonly-used
|
name.
|
|
name_to_unicode: A mapping of entity names like "angmsdaa" to
|
Unicode strings like "⦨".
|
|
named_entity_re: A regular expression matching (almost) any
|
Unicode string that corresponds to an HTML5 named entity.
|
ú;Néÿÿÿÿé é z<>&c s s | ]}t |d k V qdS )r N)Úord©Ú.0Úxr r r Ú <genexpr> s z?EntitySubstitution._populate_class_variables.<locals>.<genexpr>r Ú c S s g | ] }|d qS )r r r r r r Ú
|
<listcomp>¦ s z@EntitySubstitution._populate_class_variables.<locals>.<listcomp>z
|
%s(?![%s])z(%s)ú|)Úsetr Úsortedr ÚitemsÚendswithÚlenr ÚallÚaddÚjoinÚlistÚvaluesr ÚchrÚreÚcompile)Zunicode_to_nameZname_to_unicodeZshort_entitiesZ long_entities_by_first_characterZname_with_semicolonÚ characterÚnameZ particlesÚshortZ long_versionsÚignoreZ long_entitiesZ long_entityZ re_definitionÚ codepointr r r Ú_populate_class_variablesF sF
|
|
ÿ z,EntitySubstitution._populate_class_variablesZaposÚquotÚampÚltÚgt)ú'ú"ú&ú<ú>z&([<>]|&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;))z([<>&])c C s | j | d¡¡}d| S )zpUsed with a regular expression to substitute the
|
appropriate HTML entity for a special character string.r ú&%s;)ÚCHARACTER_TO_HTML_ENTITYÚgetÚgroup©ÚclsZmatchobjÚentityr r r Ú_substitute_html_entityÍ s z*EntitySubstitution._substitute_html_entityc C s | j | d¡ }d| S )zoUsed with a regular expression to substitute the
|
appropriate XML entity for a special character string.r r= )ÚCHARACTER_TO_XML_ENTITYr@ rA r r r Ú_substitute_xml_entityÔ s z)EntitySubstitution._substitute_xml_entityc C s6 d}d|kr*d|kr&d}| d|¡}nd}|| | S )a* Make a value into a quoted XML attribute, possibly escaping it.
|
|
Most strings will be quoted using double quotes.
|
|
Bob's Bar -> "Bob's Bar"
|
|
If a string contains double quotes, it will be quoted using
|
single quotes.
|
|
Welcome to "my bar" -> 'Welcome to "my bar"'
|
|
If a string contains both single and double quotes, the
|
double quotes will be escaped, and the string will be quoted
|
using double quotes.
|
|
Welcome to "Bob's Bar" -> "Welcome to "Bob's bar"
|
r9 r8 z")Úreplace)ÚselfÚvalueZ
|
quote_withZ replace_withr r r Úquoted_attribute_valueÛ s z)EntitySubstitution.quoted_attribute_valueFc C s" | j | j|¡}|r | |¡}|S )a Substitute XML entities for special XML characters.
|
|
:param value: A string to be substituted. The less-than sign
|
will become <, the greater-than sign will become >,
|
and any ampersands will become &. If you want ampersands
|
that appear to be part of an entity definition to be left
|
alone, use substitute_xml_containing_entities() instead.
|
|
:param make_quoted_attribute: If True, then the string will be
|
quoted, as befits an attribute value.
|
)ÚAMPERSAND_OR_BRACKETÚsubrF rJ ©rB rI Zmake_quoted_attributer r r Úsubstitute_xml s ÿ
|
z!EntitySubstitution.substitute_xmlc C s" | j | j|¡}|r | |¡}|S )a Substitute XML entities for special XML characters.
|
|
:param value: A string to be substituted. The less-than sign will
|
become <, the greater-than sign will become >, and any
|
ampersands that are not part of an entity defition will
|
become &.
|
|
:param make_quoted_attribute: If True, then the string will be
|
quoted, as befits an attribute value.
|
)ÚBARE_AMPERSAND_OR_BRACKETrL rF rJ rM r r r Ú"substitute_xml_containing_entities s ÿ
|
z5EntitySubstitution.substitute_xml_containing_entitiesc C s | j | j|¡S )a Replace certain Unicode characters with named HTML entities.
|
|
This differs from data.encode(encoding, 'xmlcharrefreplace')
|
in that the goal is to make the result more readable (to those
|
with ASCII displays) rather than to recover from
|
errors. There's absolutely nothing wrong with a UTF-8 string
|
containg a LATIN SMALL LETTER E WITH ACUTE, but replacing that
|
character with "é" will make it more readable to some
|
people.
|
|
:param s: A Unicode string.
|
)ÚCHARACTER_TO_HTML_ENTITY_RErL rD )rB r r r r Úsubstitute_html+ s ÿz"EntitySubstitution.substitute_htmlN)F)F)Ú__name__Ú
|
__module__Ú __qualname__Ú__doc__r3 r> ZHTML_ENTITY_TO_CHARACTERrQ rE r, r- rO rK Ú classmethodrD rF rJ rN rP rR r r r r r C s4 wÿû
|
|
|
|
|
$ ÿ r c @ sH e Zd ZdZdddZdd Zedd Zed
|
d Z edd d Z
|
dS )ÚEncodingDetectoraL Suggests a number of possible encodings for a bytestring.
|
|
Order of precedence:
|
|
1. Encodings you specifically tell EncodingDetector to try first
|
(the known_definite_encodings argument to the constructor).
|
|
2. An encoding determined by sniffing the document's byte-order mark.
|
|
3. Encodings you specifically tell EncodingDetector to try if
|
byte-order mark sniffing fails (the user_encodings argument to the
|
constructor).
|
|
4. An encoding declared within the bytestring itself, either in an
|
XML declaration (if the bytestring is to be interpreted as an XML
|
document), or in a <meta> tag (if the bytestring is to be
|
interpreted as an HTML document.)
|
|
5. An encoding detected through textual analysis by chardet,
|
cchardet, or a similar external library.
|
|
4. UTF-8.
|
|
5. Windows-1252.
|
|
NFc C sn t |pg | _|r | j|7 _|p&g | _|p0g }tdd |D | _d| _|| _d| _| |¡\| _ | _
|
dS )a Constructor.
|
|
:param markup: Some markup in an unknown encoding.
|
|
:param known_definite_encodings: When determining the encoding
|
of `markup`, these encodings will be tried first, in
|
order. In HTML terms, this corresponds to the "known
|
definite encoding" step defined here:
|
https://html.spec.whatwg.org/multipage/parsing.html#parsing-with-a-known-character-encoding
|
|
:param user_encodings: These encodings will be tried after the
|
`known_definite_encodings` have been tried and failed, and
|
after an attempt to sniff the encoding by looking at a
|
byte order mark has failed. In HTML terms, this
|
corresponds to the step "user has explicitly instructed
|
the user agent to override the document's character
|
encoding", defined here:
|
https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encoding
|
|
:param override_encodings: A deprecated alias for
|
known_definite_encodings. Any encodings here will be tried
|
immediately after the encodings in
|
known_definite_encodings.
|
|
:param is_html: If True, this markup is considered to be
|
HTML. Otherwise it's assumed to be XML.
|
|
:param exclude_encodings: These encodings will not be tried,
|
even if they otherwise would be.
|
|
c S s g | ] }| ¡ qS r )Úlowerr r r r r s z-EncodingDetector.__init__.<locals>.<listcomp>N) r) Úknown_definite_encodingsÚuser_encodingsr! Úexclude_encodingsÚchardet_encodingÚis_htmlÚdeclared_encodingÚstrip_byte_order_markÚmarkupÚsniffed_encoding)rH ra rZ r^ r\ r[ Úoverride_encodingsr r r Ú__init__X s "
|
zEncodingDetector.__init__c C s8 |dk r4| ¡ }|| jkr dS ||kr4| |¡ dS dS )zÕShould we even bother to try this encoding?
|
|
:param encoding: Name of an encoding.
|
:param tried: Encodings that have already been tried. This will be modified
|
as a side effect.
|
NFT)rY r\ r' )rH r Útriedr r r Ú_usable s
|
|
zEncodingDetector._usablec c sÖ t }| jD ]}| ||¡r |V q | | j|¡r:| jV | jD ]}| ||¡r@|V q@| jdkrt| | j| j¡| _| | j|¡r| jV | j dkr t
|
| j| _ | | j |¡r¶| j V dD ]}| ||¡rº|V qºdS )zmYield a number of encodings that might work for this markup.
|
|
:yield: A sequence of strings.
|
N)úutf-8ú windows-1252) r! rZ rf rb r[ r_ Úfind_declared_encodingra r^ r] r )rH re Úer r r Ú encodings s.
|
|
|
ÿ
|
zEncodingDetector.encodingsc C sþ d}t |tr||fS t|dkrT|dd
dkrT|dd
dkrTd}|dd
}n¢t|dkr|dd
dkr|dd
dkrd}|dd
}nd|dd
d
|
kr´d }|d d
}nB|dd
d krÖd }|dd
}n |dd
dkröd}|dd
}||fS )z¶If a byte-order mark is present, strip it and return the encoding it implies.
|
|
:param data: Some markup.
|
:return: A 2-tuple (modified data, implied encoding)
|
Né é s þÿz zutf-16bes ÿþzutf-16leé s rg s þÿzutf-32bes ÿþ zutf-32le)r r r% )rB Údatar r r r r` à s.
|
ÿ ÿ z&EncodingDetector.strip_byte_order_markc C sº |rt | }}nd}tdtt |d }t|tr@tt }ntt }|d }|d }d} |j||d}
|
|
|
s|r|j||d}
|
|
|
dk r|
|
¡ d } | r¶t| tr®| d d
|
¡} |
|
¡ S dS ) a§ Given a document, tries to find its declared encoding.
|
|
An XML encoding is declared at the beginning of the document.
|
|
An HTML encoding is declared in a <meta> tag, hopefully near the
|
beginning of the document.
|
|
:param markup: Some markup.
|
:param is_html: If True, this markup is considered to be HTML. Otherwise
|
it's assumed to be XML.
|
:param search_entire_document: Since an encoding is supposed to declared near the beginning
|
of the document, most of the time it's only necessary to search a few kilobytes of data.
|
Set this to True to force this method to search the entire document.
|
i i g©?r r N)Úendposr r rG ) r% ÚmaxÚintr ÚbytesÚ encoding_resr ÚsearchÚgroupsÚdecoderY ) rB ra r^ Zsearch_entire_documentZ
|
xml_endposZ html_endposÚresÚxml_reZhtml_rer_ Zdeclared_encoding_matchr r r ri á s(
|
|
|
z'EncodingDetector.find_declared_encoding)NFNNN)FF) rS rT rU rV rd rf Úpropertyrk rW r` ri r r r r rX = s þ
|
/
|
+
|
rX c @ sê e Zd ZdZdddZdddgZg dd g ddfd
|
d Zd d ZdþddZdÿddZ e
|
dd Z dd Z dd Z dddd d d d d d!d"d#d$d%d&d'd&d&d(d)d*d+d,d-d.d/d0d1d2d3d&d4d5d6 Zd7dd8d9d:d;d<d=d>d?d@dAdBd&dCd&d&dDdDdEdEdFdGdHdIdJdKdLdMd&dNdOddPdQdRdSdTdUd@dVdWdXdYdPddZdGd[d\d]d^d_d`dadFd8dbdXdcdddedfd&dgdgdgdgdgdgdhdidjdjdjdjdkdkdkdkdldmdndndndndndFdndododododOdpdqdrdrdrdrdrdrdsdQdtdtdtdtdudududud[dvd[d[d[d[d[dwd[d`d`d`d`dxdpdxdyZdzd{d|d}d~ddddddd
ddddddddddddddddddddddddddd d¡d¢d£d¤d¥d¦d§d¨d©dªd«d¬dd®d¯d°d±d²d³d´dµd¶d·d¸d¹dºd»d¼d½d¾d¿dÀdÁdÂdÃdÄdÅdÆdÇdÈdÉdÊdËdÌdÍdÎdÏdÐdÑdÒdÓdÔdÕdÖd×dØdÙdÚdÛdÜdÝdÞdßdàdádâdãdädådædçdèdédêdëdìdídîdïdðdñdòdódôzZdõdöd÷gZedø dø Zedù dú Zed düdýZdS ( Ú UnicodeDammitzÏA class for detecting the encoding of a *ML document and
|
converting it to a Unicode string. If the source encoding is
|
windows-1252, can replace MS smart quotes with their HTML or XML
|
equivalents.z mac-romanz shift-jis)Ú macintoshzx-sjisrh z
|
iso-8859-1z
|
iso-8859-2NFc
|
C sö || _ g | _d| _|| _t t¡| _t||||||| _ t
|
|t sJ|dkrd|| _ t || _ d| _dS | j j | _ d}| j jD ]"} | j j }| | ¡}|dk rz qqz|sâ| j jD ]6} | dkrÂ| | d¡}|dk rª| j d¡ d| _ qâqª|| _ |sòd| _dS )a2 Constructor.
|
|
:param markup: A bytestring representing markup in an unknown encoding.
|
|
:param known_definite_encodings: When determining the encoding
|
of `markup`, these encodings will be tried first, in
|
order. In HTML terms, this corresponds to the "known
|
definite encoding" step defined here:
|
https://html.spec.whatwg.org/multipage/parsing.html#parsing-with-a-known-character-encoding
|
|
:param user_encodings: These encodings will be tried after the
|
`known_definite_encodings` have been tried and failed, and
|
after an attempt to sniff the encoding by looking at a
|
byte order mark has failed. In HTML terms, this
|
corresponds to the step "user has explicitly instructed
|
the user agent to override the document's character
|
encoding", defined here:
|
https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encoding
|
|
:param override_encodings: A deprecated alias for
|
known_definite_encodings. Any encodings here will be tried
|
immediately after the encodings in
|
known_definite_encodings.
|
|
:param smart_quotes_to: By default, Microsoft smart quotes will, like all other characters, be converted
|
to Unicode characters. Setting this to 'ascii' will convert them to ASCII quotes instead.
|
Setting it to 'xml' will convert them to XML entity references, and setting it to 'html'
|
will convert them to HTML entity references.
|
:param is_html: If True, this markup is considered to be HTML. Otherwise
|
it's assumed to be XML.
|
:param exclude_encodings: These encodings will not be considered, even
|
if the sniffing code thinks they might make sense.
|
|
Fr Nr rG zSSome characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.T)Úsmart_quotes_toÚtried_encodingsZ contains_replacement_charactersr^ ÚloggingÚ getLoggerrS ÚlogrX Údetectorr r ra Zunicode_markupÚoriginal_encodingrk Ú _convert_fromÚwarning)
|
rH ra rZ r} r^ r\ r[ rc Úur r r r rd sL & þ
|
|
|
ÿzUnicodeDammit.__init__c C s | d¡}| jdkr&| j |¡ ¡ }nf| j |¡}t|tkr| jdkrfd ¡ |d ¡ d ¡ }qd ¡ |d ¡ d ¡ }n| ¡ }|S )z[Changes a MS smart quote character to an XML or HTML
|
entity, or an ASCII character.r r r z&#xr r: r )r@ r} ÚMS_CHARS_TO_ASCIIr? ÚencodeÚMS_CHARSÚtypeÚtuple)rH ÚmatchÚorigrL r r r Ú _sub_ms_charu s
|
|
|
zUnicodeDammit._sub_ms_charÚstrictc
|
C s´ | |¡}|r ||f| jkr dS | j ||f¡ | j}| jdk rf|| jkrfd}t |¡}| | j |¡}z |
|
|||¡}|| _|| _ W n( t k
|
r¬ } z
|
W Y ¢dS d}~X Y nX | jS )z|Attempt to convert the markup to the proposed encoding.
|
|
:param proposed: The name of a character encoding.
|
Ns ([-])) Ú
|
find_codecr~ Úappendra r} ÚENCODINGS_WITH_SMART_QUOTESr, r- rL r Ú _to_unicoder Ú Exception)rH ZproposedÚerrorsra Zsmart_quotes_reZsmart_quotes_compiledr rj r r r r s$
|
|
ÿ
|
|
zUnicodeDammit._convert_fromc C s t |||S )z}Given a string and its encoding, decodes the string into Unicode.
|
|
:param encoding: The name of an encoding.
|
)r )rH ro r r r r r r ¥ s zUnicodeDammit._to_unicodec C s | j s
|
dS | jjS )zhIf the markup is an HTML document, returns the encoding declared _within_
|
the document.
|
N)r^ r r_ )rH r r r Údeclared_html_encoding¬ s z$UnicodeDammit.declared_html_encodingc C s` | | j ||¡¡pN|r*| | dd¡¡pN|r@| | dd¡¡pN|rL| ¡ pN|}|r\| ¡ S dS )zConvert the name of a character set to a codec name.
|
|
:param charset: The name of a character set.
|
:return: The name of a codec.
|
ú-r Ú_N)Ú_codecÚCHARSET_ALIASESr? rG rY )rH ÚcharsetrI r r r r µ s ÿþ
|
ýüzUnicodeDammit.find_codecc C s< |s|S d }zt |¡ |}W n ttfk
|
r6 Y nX |S r )ÚcodecsÚlookupÚ LookupErrorÚ
|
ValueError)rH r Úcodecr r r r Å s
|
zUnicodeDammit._codec)ÚeuroZ20ACú )ÚsbquoZ201A)ÚfnofZ192)ÚbdquoZ201E)ÚhellipZ2026)ÚdaggerZ2020)ÚDaggerZ2021)ÚcircZ2C6)ÚpermilZ2030)ÚScaronZ160)ÚlsaquoZ2039)ÚOEligZ152ú?)z#x17DZ17D)ÚlsquoZ2018)ÚrsquoZ2019)ÚldquoZ201C)ÚrdquoZ201D)ÚbullZ2022)ÚndashZ2013)ÚmdashZ2014)ÚtildeZ2DC)ÚtradeZ2122)ÚscaronZ161)ÚrsaquoZ203A)ÚoeligZ153)z#x17EZ17E)ÚYumlr ) ó ó ó ó ó ó
ó ó ó ó ó ó ó ó ó ó ó ó ó ó ó ó ó ó ó ó ó ó ó ó ó ó ZEURú,Úfz,,z...ú+z++ú^ú%ÚSr; ZOEÚZr8 r9 Ú*r z--ú~z(TM)r r< ZoeÚzÚYú!ÚcZGBPú$ZYENr z..r z(th)z<<z(R)Úoz+-Ú2Ú3)r8 Úacuter ÚPÚ1z>>z1/4z1/2z3/4ÚAZAEÚCÚEÚIÚDÚNÚOÚUÚbÚBÚaZaerj ÚiÚnú/Úy)r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ ó ó ¡ó ¢ó £ó ¤ó ¥ó ¦ó §ó ¨ó ©ó ªó «ó ¬ó ó ®ó ¯ó °ó ±ó ²ó ³ó ´ó µó ¶ó ·ó ¸ó ¹ó ºó »ó ¼ó ½ó ¾ó ¿ó Àó Áó Âó Ãó Äó Åó Æó Çó Èó Éó Êó Ëó Ìó Íó Îó Ïó Ðó Ñó Òó Óó Ôó Õó Öó ×ó Øó Ùó Úó Ûó Üó Ýó Þó ßó àó áó âó ãó äó åó æó çó èó éó êó ëó ìó íó îó ïó ðó ñó òó óó ôó õó öó ÷ó øó ùó úó ûó üó ýó þó ÿs â¬s âs Æs âs â¦s â s â¡s Ës â°s Å s â¹s Ås Žs âs âs âs âs â¢s âs âs Ës â¢s Å¡s âºs Ås žs Ÿs  s ¡s ¢s £s ¤s Â¥s ¦s §s ¨s ©s ªs «s ¬s Âs ®s ¯s °s ±s ²s ³s ´s µs ¶s ·s ¸s ¹s ºs »s ¼s ½s ¾s ¿s Ãs Ãs Ãs Ãs Ãs Ã
s Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs Ãs à r s âs ãs äs Ã¥s æs çs ès és ês ës ìs Ãs îs ïs ðs ñs òs ós ôs õs ös ÷s øs ùs ús ûs üs ýs þ)zr é é é é
é é é é é é é é é é é é é é é é é é é é é é é é¡ é¢ é£ é¤ é¥ é¦ é§ é¨ é© éª é« é¬ é é® é¯ é° é± é² é³ é´ éµ é¶ é· é¸ é¹ éº é» é¼ é½ é¾ é¿ éÀ éÁ é éà éÄ éÅ éÆ éÇ éÈ éÉ éÊ éË éÌ éÍ éÎ éÏ éÐ éÑ éÒ éÓ éÔ éÕ éÖ é× éØ éÙ éÚ éÛ éÜ éÝ éÞ éß éà éá éâ éã éä éå éæ éç éè éé éê éë éì éí éî éï éð éñ éò éó éô éõ éö é÷ éø éù éú éû éü éý éþ )r r¸ rm )r¹ rÈ rn )rÉ rÍ rl r r r Úutf8c C s | dd¡ ¡ dkr td| ¡ dkr0tdg }d}d}|t|k rò|| }t|tsbt|}|| jkr¦|| jkr¦| j D ]&\}} }
|
||kr||| kr|||
|
7 } qðq|q<|dkrè|| j
|
krè| |||
¡ | | j
|
| ¡ |d 7 }|}q<|d 7 }q<|dkr |S | ||d
|
¡ d |¡S ) a Fix characters from one encoding embedded in some other encoding.
|
|
Currently the only situation supported is Windows-1252 (or its
|
subset ISO-8859-1), embedded in UTF-8.
|
|
:param in_bytes: A bytestring that you suspect contains
|
characters from multiple encodings. Note that this _must_
|
be a bytestring. If you've already converted the document
|
to Unicode, you're too late.
|
:param main_encoding: The primary encoding of `in_bytes`.
|
:param embedded_encoding: The encoding that was used to embed characters
|
in the main document.
|
:return: A bytestring in which `embedded_encoding`
|
characters have been converted to their `main_encoding`
|
equivalents.
|
r r )rh Ú windows_1252zPWindows-1252 and ISO-8859-1 are the only currently supported embedded encodings.)rØ rg z4UTF-8 is the only currently supported main encoding.r r r Nó ) rG rY ÚNotImplementedErrorr% r rr r ÚFIRST_MULTIBYTE_MARKERÚLAST_MULTIBYTE_MARKERÚMULTIBYTE_MARKERS_AND_SIZESÚWINDOWS_1252_TO_UTF8r r( ) rB Zin_bytesZ main_encodingZembedded_encodingZ byte_chunksZ chunk_startÚposÚbyteÚstartÚendÚsizer r r Ú detwingle s@ ÿ ÿ
|
|
ÿ
|
|
zUnicodeDammit.detwingle)r )r )rØ rh )rS rT rU rV r r rd r r r rz r r r r r rß rÞ rÜ rÝ rW rå r r r r r{
|
s~ ÿý þ
|
X
|
|
|
á% ÿ ~ý ÿr{ ) rV Ú __license__Ú html.entitiesr Ú collectionsr r r, r Ústringr ZcchardetÚ ImportErrorÚchardetZcharset_normalizerr Z xml_encodingZ html_metaÚdictrt r- r ró rs r r Úobjectr rX r{ r r r r Ú<module> sH
|
þ
|
þ
|
{ N
|
