U
|
¡ý°dI_ ã @ sr d Z ddlmZ ddlZddlZddlZddlZddlmZm Z m
|
Z
|
m Z m Z m Z mZmZ ddddgZd Zd
|
Zd Zd Zd ZdZG dd deZG dd deZe ZG dd deZ G dd de Z G dd de Z G dd deZ dd Z G dd de!Z"dd l#m$Z$ e e$ zdd l#m%Z% e e% W n e&k
|
r< Y nX zdd l#m'Z' e e' W n e&k
|
rl Y nX dS ) ÚMITé )Ú defaultdictN)ÚCharsetMetaAttributeValueÚContentMetaAttributeValueÚRubyParenthesisStringÚRubyTextStringÚ
|
StylesheetÚScriptÚTemplateStringÚnonwhitespace_reÚHTMLTreeBuilderÚSAXTreeBuilderÚ TreeBuilderÚTreeBuilderRegistryÚfastZ
|
permissiveÚstrictÚxmlÚhtmlÚhtml5c @ s e Zd ZdZdZdS )ÚXMLParsedAsHTMLWarningzWThe warning issued when an HTML parser is used to parse
|
XML that is not XHTML.
|
a It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument `features="xml"` into the BeautifulSoup constructor.N)Ú__name__Ú
|
__module__Ú __qualname__Ú__doc__ÚMESSAGE© r r úKd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\bs4/builder/__init__.pyr # s r c @ s( e Zd ZdZdd Zdd Zdd ZdS ) r zYA way of looking up TreeBuilder subclasses by their name or by desired
|
features.
|
c C s t t| _g | _d S ©N)r ÚlistÚbuilders_for_featureÚbuilders©Úselfr r r Ú__init__/ s
|
z TreeBuilderRegistry.__init__c C s0 |j D ]}| j| d|¡ q| j d|¡ dS )z¼Register a treebuilder based on its advertised features.
|
|
:param treebuilder_class: A subclass of Treebuilder. its .features
|
attribute should list its features.
|
r N)Úfeaturesr Úinsertr )r" Ztreebuilder_classÚfeaturer r r Úregister3 s
|
z TreeBuilderRegistry.registerc G s¾ t | jdkrdS t |dkr(| jd S t|}| ¡ d}d}t |dkr| ¡ }| j |g ¡}t |dkr@|dkr|}t|}q@| t|¡}q@|dkr dS |D ]}||kr¤| S q¤dS )aj Look up a TreeBuilder subclass with the desired features.
|
|
:param features: A list of features to look for. If none are
|
provided, the most recently registered TreeBuilder subclass
|
will be used.
|
:return: A TreeBuilder subclass, or None if there's no
|
registered subclass with all the requested features.
|
r N) Úlenr r ÚreverseÚpopr ÚgetÚsetÚ intersection)r" r$ Ú
|
candidatesZ candidate_setr& Zwe_have_the_featureÚ candidater r r Úlookup= s0
|
|
ÿ
|
zTreeBuilderRegistry.lookupN)r r r r r# r' r0 r r r r r * s
|
c @ s e Zd ZdZdZg Zg ZdZdZdZ e
|
e Z e Zi Ze ZdZeeeefddZdd Zd d
|
Zd d Zd d ZdddZdd Zdd Zdd ZdS )r z:Turn a textual document into a Beautiful Soup object tree.z[Unknown tree builder]FNc C sb d| _ || jkr| j}|| _|| jkr,| j}|| _|| jkrB| j}|| _|| jkrX| j}|| _ dS )aÜ Constructor.
|
|
:param multi_valued_attributes: If this is set to None, the
|
TreeBuilder will not turn any values for attributes like
|
'class' into lists. Setting this to a dictionary will
|
customize this behavior; look at DEFAULT_CDATA_LIST_ATTRIBUTES
|
for an example.
|
|
Internally, these are called "CDATA list attributes", but that
|
probably doesn't make sense to an end-user, so the argument name
|
is `multi_valued_attributes`.
|
|
:param preserve_whitespace_tags: A list of tags to treat
|
the way <pre> tags are treated in HTML. Tags in this list
|
are immune from pretty-printing; their contents will always be
|
output as-is.
|
|
:param string_containers: A dictionary mapping tag names to
|
the classes that should be instantiated to contain the textual
|
contents of those tags. The default is to use NavigableString
|
for every tag, no matter what the name. You can override the
|
default by changing DEFAULT_STRING_CONTAINERS.
|
|
:param store_line_numbers: If the parser keeps track of the
|
line numbers and positions of the original markup, that
|
information will, by default, be stored in each corresponding
|
`Tag` object. You can turn this off by passing
|
store_line_numbers=False. If the parser you're using doesn't
|
keep track of this information, then setting store_line_numbers=True
|
will do nothing.
|
N)
|
ÚsoupÚ USE_DEFAULTÚ DEFAULT_CDATA_LIST_ATTRIBUTESÚcdata_list_attributesÚ DEFAULT_PRESERVE_WHITESPACE_TAGSÚpreserve_whitespace_tagsÚTRACKS_LINE_NUMBERSÚstore_line_numbersÚDEFAULT_STRING_CONTAINERSÚstring_containers)r" Zmulti_valued_attributesr6 r8 r: r r r r# s $
|
|
|
|
zTreeBuilder.__init__c C s
|
|| _ dS )zThe BeautifulSoup object has been initialized and is now
|
being associated with the TreeBuilder.
|
|
:param soup: A BeautifulSoup object.
|
N)r1 )r" r1 r r r Úinitialize_soup½ s zTreeBuilder.initialize_soupc C s dS )zDo any work necessary to reset the underlying parser
|
for a new document.
|
|
By default, this does nothing.
|
Nr r! r r r ÚresetÅ s zTreeBuilder.resetc C s | j dkrdS || j kS )aß Might a tag with this name be an empty-element tag?
|
|
The final markup may or may not actually present this tag as
|
self-closing.
|
|
For instance: an HTMLBuilder does not consider a <p> tag to be
|
an empty-element tag (it's not in
|
HTMLBuilder.empty_element_tags). This means an empty <p> tag
|
will be presented as "<p></p>", not "<p/>" or "<p>".
|
|
The default implementation has no opinion about which tags are
|
empty-element tags, so a tag will be presented as an
|
empty-element tag if and only if it has no children.
|
"<foo></foo>" will become "<foo/>", and "<foo>bar</foo>" will
|
be left alone.
|
|
:param tag_name: The name of a markup tag.
|
NT)Úempty_element_tags)r" Útag_namer r r Úcan_be_empty_elementÍ s
|
z TreeBuilder.can_be_empty_elementc C s
|
t dS )zùRun some incoming markup through some parsing process,
|
populating the `BeautifulSoup` object in self.soup.
|
|
This method is not implemented in TreeBuilder; it must be
|
implemented in subclasses.
|
|
:return: None.
|
N©ÚNotImplementedError©r" Úmarkupr r r Úfeedä s zTreeBuilder.feedc c s |dddfV dS )a Run any preliminary steps necessary to make incoming markup
|
acceptable to the parser.
|
|
:param markup: Some markup -- probably a bytestring.
|
:param user_specified_encoding: The user asked to try this encoding.
|
:param document_declared_encoding: The markup itself claims to be
|
in this encoding. NOTE: This argument is not used by the
|
calling code and can probably be removed.
|
:param exclude_encodings: The user asked _not_ to try any of
|
these encodings.
|
|
:yield: A series of 4-tuples:
|
(markup, encoding, declared encoding,
|
has undergone character replacement)
|
|
Each 4-tuple represents a strategy for converting the
|
document to Unicode and parsing it. Each strategy will be tried
|
in turn.
|
|
By default, the only strategy is to parse the markup
|
as-is. See `LXMLTreeBuilderForXML` and
|
`HTMLParserTreeBuilder` for implementations that take into
|
account the quirks of particular parsers.
|
NFr )r" rC Zuser_specified_encodingZdocument_declared_encodingZexclude_encodingsr r r Úprepare_markupï s zTreeBuilder.prepare_markupc C s |S )a Wrap an HTML fragment to make it look like a document.
|
|
Different parsers do this differently. For instance, lxml
|
introduces an empty <head> tag, and html5lib
|
doesn't. Abstracting this away lets us write simple tests
|
which run HTML fragments through the parser and compare the
|
results against other HTML fragments.
|
|
This method should not be used outside of tests.
|
|
:param fragment: A string -- fragment of HTML.
|
:return: A string -- a full HTML document.
|
r )r" Úfragmentr r r Útest_fragment_to_document s z%TreeBuilder.test_fragment_to_documentc C s dS )a2 Set up any substitutions that will need to be performed on
|
a `Tag` when it's output as a string.
|
|
By default, this does nothing. See `HTMLTreeBuilder` for a
|
case where this is used.
|
|
:param tag: A `Tag`
|
:return: Whether or not a substitution was performed.
|
Fr )r" Útagr r r Úset_up_substitutions s
|
z TreeBuilder.set_up_substitutionsc C s |s|S | j r~| j dg ¡}| j | ¡ d¡}t| ¡ D ]B}||ksR|r:||kr:|| }t|trpt |¡}n|}|||< q:|S )aá When an attribute value is associated with a tag that can
|
have multiple values for that attribute, convert the string
|
value to a list of strings.
|
|
Basically, replaces class="foo bar" with class=["foo", "bar"]
|
|
NOTE: This method modifies its input in place.
|
|
:param tag_name: The name of a tag.
|
:param attrs: A dictionary containing the tag's attributes.
|
Any appropriate attribute values will be modified in place.
|
Ú*N) r4 r+ Úlowerr ÚkeysÚ
|
isinstanceÚstrr Úfindall)r" r> ÚattrsÚ universalZ tag_specificÚattrÚvalueÚvaluesr r r Ú$_replace_cdata_list_attribute_values' s ÿ
|
|
z0TreeBuilder._replace_cdata_list_attribute_values)NNN) r r r r ÚNAMEZALTERNATE_NAMESr$ Zis_xmlZ picklabler= r r r3 r, r5 r9 Úobjectr2 r7 r# r; r< r? rD rE rG rI rU r r r r r o s8 ý
|
2 ÿ
|
c @ sh e Zd ZdZdd Zdd Zdd Zdd Zd
|
d Zd d Z dd Z
|
dd Z dd Z dd Z dd ZdS )r z©A Beautiful Soup treebuilder that listens for SAX events.
|
|
This is not currently used for anything, but it demonstrates
|
how a simple TreeBuilder would work.
|
c C s
|
t d S r r@ rB r r r rD T s zSAXTreeBuilder.feedc C s d S r r r! r r r ÚcloseW s zSAXTreeBuilder.closec C s, t dd t| ¡ D }| j ||¡ d S )Nc s s | ]\}}|d |fV qdS )é Nr )Ú.0ÚkeyrS r r r Ú <genexpr>[ s z.SAXTreeBuilder.startElement.<locals>.<genexpr>)Údictr Úitemsr1 Úhandle_starttag)r" ÚnamerP r r r Ú startElementZ s zSAXTreeBuilder.startElementc C s | j |¡ d S r )r1 Ú handle_endtag©r" r` r r r Ú
|
endElement_ s zSAXTreeBuilder.endElementc C s | ||¡ d S r )ra )r" ÚnsTupleÚnodeNamerP r r r ÚstartElementNSc s z SAXTreeBuilder.startElementNSc C s | |¡ d S r )rd )r" re rf r r r Ú endElementNSg s zSAXTreeBuilder.endElementNSc C s d S r r )r" ÚprefixZ nodeValuer r r ÚstartPrefixMappingl s z!SAXTreeBuilder.startPrefixMappingc C s d S r r )r" ri r r r ÚendPrefixMappingp s z SAXTreeBuilder.endPrefixMappingc C s | j |¡ d S r )r1 Ú handle_data)r" Úcontentr r r Ú
|
charactersu s zSAXTreeBuilder.charactersc C s d S r r r! r r r Ú startDocumentx s z SAXTreeBuilder.startDocumentc C s d S r r r! r r r Ú endDocument{ s zSAXTreeBuilder.endDocumentN)r r r r rD rX ra rd rg rh rj rk rn ro rp r r r r r M s c $ @ sø e Zd ZdZedddddddd d
|
d d d ddddddddddddgZeddd d d d d d!d"d#d$d%d&d'd(d)d*d+d,d-d.dd/d0d1d2d3d4d5d6d7d8d9d:d;g#Zeee e
|
e d<Z d=d>d?gd@dAgd@dAgdBgdBgdBgdCgdDgd@gdEgdFgdGgdH Z ed6dIgZdJdK ZdLS )Mr z]This TreeBuilder knows facts about HTML.
|
|
Such as which tags are empty-element tags.
|
ÚareaÚbaseÚbrÚcolÚembedÚhrÚimgÚinputÚkeygenÚlinkZmenuitemÚmetaÚparamÚsourceÚtrackÚwbrÚbasefontÚbgsoundÚcommandÚframeÚimageÚisindexZnextidZspacerÚaddressÚarticleÚasideÚ
|
blockquoteZcanvasÚddÚdivÚdlÚdtÚfieldsetÚ
|
figcaptionÚfigureÚfooterÚformÚh1Úh2Úh3Úh4Úh5Úh6ÚheaderÚliÚmainÚnavÚnoscriptÚolÚoutputÚpÚpreÚsectionÚtableÚtfootÚulÚvideo)ÚrtÚrpÚstyleÚscriptÚtemplateÚclassZ accesskeyZdropzoneÚrelÚrevÚheaderszaccept-charsetÚarchiveÚsizesZsandboxÚfor) rJ Úarz ÚtdÚthr´ r rW rq ZiconÚiframer Útextareac C sz |j dkrdS | d¡}| d¡}| d¡}d}|dk rJ|}t||d< n(|dk rr|dk rr| ¡ dkrrt||d< |dk S )a Replace the declared encoding in a <meta> tag with a placeholder,
|
to be substituted when the tag is output to a string.
|
|
An HTML document may come in to Beautiful Soup as one
|
encoding, but exit in a different encoding, and the <meta> tag
|
needs to be changed to reflect this.
|
|
:param tag: A `Tag`
|
:return: Whether or not a substitution was performed.
|
r{ Fz
|
http-equivrm ÚcharsetNz content-type)r` r+ r rK r )r" rH Z
|
http_equivrm r¸ Z meta_encodingr r r rI Å s
|
|
|
|
|
ÿ z$HTMLTreeBuilder.set_up_substitutionsN)r r r r r, r= Zblock_elementsr r r r r
|
r9 r3 r5 rI r r r r r sb û Nûò c @ sd e Zd ZdZe dej¡Ze dej¡ZdZ dZ
|
e dd Z e dd Z d
|
d Zd d Zdd ZdS )ÚDetectsXMLParsedAsHTMLa A mixin class for any class (a TreeBuilder, or some class used by a
|
TreeBuilder) that's in a position to detect whether an XML
|
document is being incorrectly parsed as HTML, and issue an
|
appropriate warning.
|
|
This requires being able to observe an incoming processing
|
instruction that might be an XML declaration, and also able to
|
observe tags as they're opened. If you can't do that for a given
|
TreeBuilder, there's a less reliable implementation based on
|
examining the raw markup.
|
z
|
<[^ +]htmls
|
<[^ +]htmlz<?xmls <?xmlc C sX t |tr| j}| j}n | j}| j}|dk rT| |¡rT| |dd
¡sT| ¡ dS dS )aJ Perform a check on some markup to see if it looks like XML
|
that's not XHTML. If so, issue a warning.
|
|
This is much less reliable than doing the check while parsing,
|
but some of the tree builders can't do that.
|
|
:return: True if the markup looks like non-XHTML XML, False
|
otherwise.
|
Niô TF) rM ÚbytesÚ XML_PREFIX_BÚLOOKS_LIKE_HTML_BÚ
|
XML_PREFIXÚLOOKS_LIKE_HTMLÚ
|
startswithÚsearchÚ_warn)ÚclsrC ri Zlooks_like_htmlr r r Ú warn_if_markup_looks_like_xml s
|
ÿþz4DetectsXMLParsedAsHTML.warn_if_markup_looks_like_xmlc C s t tjt¡ dS )z/Issue a warning about XML being parsed as HTML.N)ÚwarningsÚwarnr r )r r r r rÁ s ÿz DetectsXMLParsedAsHTML._warnc C s d| _ d| _dS )z+Call this method before parsing a document.N©Ú _first_processing_instructionÚ _root_tagr! r r r Ú_initialize_xml_detector% s z/DetectsXMLParsedAsHTML._initialize_xml_detectorc C s" | j dk s| jdk rdS || _ dS )zCall this method when encountering an XML declaration, or a
|
"processing instruction" that might be an XML declaration.
|
NrÆ )r" Zprocessing_instructionr r r Ú_document_might_be_xml* s
|
|
ÿz-DetectsXMLParsedAsHTML._document_might_be_xmlc C sB | j dk rdS || _ |dkr>| jdk r>| j ¡ d¡r>| ¡ dS )zÃCall this when you encounter the document's root tag.
|
|
This is where we actually check whether an XML document is
|
being incorrectly parsed as HTML, and issue the warning.
|
Nr zxml )rÈ rÇ rK r¿ rÁ rc r r r Ú_root_tag_encountered9 s
|
ÿz,DetectsXMLParsedAsHTML._root_tag_encounteredN)r r r r ÚreÚcompileÚIr¾ r¼ r½ r» Ú classmethodrà rÁ rÉ rÊ rË r r r r r¹ ð s
|
|
r¹ c C sR t jt }| jD ]<}t| |}t|trt||| |j |¡ |j
|
|¡ qdS )z9Copy TreeBuilders from the given module into this module.N) ÚsysÚmodulesr Ú__all__ÚgetattrÚ
|
issubclassr ÚsetattrÚappendÚbuilder_registryr' )ÚmoduleZ this_moduler` Úobjr r r Úregister_treebuilders_fromM s
|
|
|
|
rÚ c s e Zd ZdZ fddZ ZS )ÚParserRejectedMarkupzgAn Exception to be raised when the underlying parser simply
|
refuses to parse the given markup.
|
c s6 t |tr"|}d|jjt|f }tt| |¡ dS )zzExplain why the parser rejected the given markup, either
|
with a textual explanation or another exception.
|
z%s: %sN)rM Ú ExceptionÚ __class__r rN ÚsuperrÛ r# )r" Zmessage_or_exceptionÚe©rÝ r r r# ] s
|
z ParserRejectedMarkup.__init__)r r r r r# Ú __classcell__r r rà r rÛ Y s rÛ rY )Ú _htmlparser)Ú _html5lib)Ú_lxml)(Ú __license__Ú collectionsr Ú itertoolsrÌ rÄ rÐ Z bs4.elementr r r r r r r
|
r rÒ ZFASTZ
|
PERMISSIVEZSTRICTÚXMLZHTMLZHTML_5Ú UserWarningr rW r r× r r r r¹ rÚ rÜ rÛ Ú râ rã Ú ImportErrorrä r r r r Ú<module> sN ( üC _2q]
|
