U
|
¡ý°d ã @ s d Z dZddlZddlmZ ddlmZ ddlZddlmZm Z ddl
|
m Z ddl Z ddl Z ddlZddlZddlZddlZddlZddlZdd Zd#d d ZG d d deZdd ZdZdZd$ddZd%ddZd&ddZd'd d Z d(d d!Z e d"kreej ¡ dS ))z=Diagnostic functions, mainly for use when doing tech support.ÚMITé N)ÚBytesIO)Ú
|
HTMLParser)Ú BeautifulSoupÚ __version__)Úbuilder_registryc
|
C sÀ t dt t dtj dddg}|D ]4}tjD ]}||jkr2 q(q2| |¡ t d| q(d|krÆ| d¡ z*dd l m
|
} t d
|
d t t |j¡ W n* tk
|
rÄ } z t d W 5 d }~X Y nX d|krzdd l}t d|j W n, tk
|
r } z t d W 5 d }~X Y nX t| dr,| ¡ } |D ]}t d| d}zt| |d} d}W n8 tk
|
r } zt d| t ¡ W 5 d }~X Y nX |r°t d| t | ¡ t d q0d S )z¼Diagnostic suite for isolating common problems.
|
|
:param data: A string containing markup that needs to be explained.
|
:return: None; diagnostics are printed to standard output.
|
z'Diagnostic running on Beautiful Soup %szPython version %sú html.parserÚhtml5libÚlxmlz;I noticed that %s is not installed. Installing it may help.zlxml-xmlr ©ÚetreezFound lxml version %sÚ.z.lxml is not installed or couldn't be imported.NzFound html5lib version %sz2html5lib is not installed or couldn't be imported.Úreadz#Trying to parse your markup with %sF)ÚfeaturesTú %s could not parse the markup.z#Here's what %s did with the markup:zP--------------------------------------------------------------------------------)Úprintr ÚsysÚversionr Zbuildersr ÚremoveÚappendr
|
r ÚjoinÚmapÚstrZ LXML_VERSIONÚ ImportErrorr Úhasattrr r Ú ExceptionÚ tracebackÚ print_excZprettify)
|
ÚdataZ basic_parsersÚnameZbuilderr Úer ÚparserÚsuccessÚsoup© r$ úCd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\bs4/diagnose.pyÚdiagnose sZ
|
|
|
|
ÿÿ
|
ÿ
|
ÿ r& Tc K sp ddl m} | dd¡}t| tr,| d¡} t| }|j|f||d|D ] \}}td||j |j
|
f qLdS ) a´ Print out the lxml events that occur during parsing.
|
|
This lets you see how lxml parses a document when no Beautiful
|
Soup code is running. You can use this to determine whether
|
an lxml-specific problem is in Beautiful Soup's lxml tree builders
|
or in lxml itself.
|
|
:param data: Some markup.
|
:param html: If True, markup will be parsed with lxml's HTML parser.
|
if False, lxml's XML parser will be used.
|
r r ÚrecoverTÚutf8)Úhtmlr' z %s, %4s, %sN) r
|
r ÚpopÚ
|
isinstancer Úencoder Ú iterparser ÚtagÚtext)r r) Úkwargsr r' ÚreaderÚeventÚelementr$ r$ r% Ú
|
lxml_traceN s
|
|
ÿ ÿÿr4 c @ s` e Zd ZdZdd Zdd Zdd Zdd Zd
|
d Zd d Z dd Z
|
dd Z dd Z dd Z dS )ÚAnnouncingParserzèSubclass of HTMLParser that announces parse events, without doing
|
anything else.
|
|
You can use this to get a picture of how html.parser sees a given
|
document. The easiest way to do this is to call `htmlparser_trace`.
|
c C s t | d S )N)r )ÚselfÚsr$ r$ r% Ú_pl s zAnnouncingParser._pc C s | d| ¡ d S )Nz%s START©r8 )r6 r Úattrsr$ r$ r% Úhandle_starttago s z AnnouncingParser.handle_starttagc C s | d| ¡ d S )Nz%s ENDr9 ©r6 r r$ r$ r% Ú handle_endtagr s z AnnouncingParser.handle_endtagc C s | d| ¡ d S )Nz%s DATAr9 ©r6 r r$ r$ r% Ú handle_datau s z AnnouncingParser.handle_datac C s | d| ¡ d S )Nz
|
%s CHARREFr9 r< r$ r$ r% Úhandle_charrefx s z AnnouncingParser.handle_charrefc C s | d| ¡ d S )Nz %s ENTITYREFr9 r< r$ r$ r% Úhandle_entityref{ s z!AnnouncingParser.handle_entityrefc C s | d| ¡ d S )Nz
|
%s COMMENTr9 r> r$ r$ r% Úhandle_comment~ s z AnnouncingParser.handle_commentc C s | d| ¡ d S )Nz%s DECLr9 r> r$ r$ r% Ú handle_decl s z AnnouncingParser.handle_declc C s | d| ¡ d S )Nz%s UNKNOWN-DECLr9 r> r$ r$ r% Ú unknown_decl s z AnnouncingParser.unknown_declc C s | d| ¡ d S )Nz%s PIr9 r> r$ r$ r% Ú handle_pi s zAnnouncingParser.handle_piN)Ú__name__Ú
|
__module__Ú __qualname__Ú__doc__r8 r; r= r? r@ rA rB rC rD rE r$ r$ r$ r% r5 d s r5 c C s t }| | ¡ dS )zÂPrint out the HTMLParser events that occur during parsing.
|
|
This lets you see how HTMLParser parses a document when no
|
Beautiful Soup code is running.
|
|
:param data: Some markup.
|
N)r5 Úfeed)r r! r$ r$ r% Úhtmlparser_trace s rK ZaeiouZbcdfghjklmnpqrstvwxyzé c C s: d}t | D ](}|d dkr"t}nt}|t |¡7 }q |S )z#Generate a random word-like string.Ú é r )ÚrangeÚ _consonantsÚ_vowelsÚrandomÚchoice)Úlengthr7 ÚiÚtr$ r$ r% Úrword s rW é c C s d dd t| D ¡S )z'Generate a random sentence-like string.ú c s s | ]}t t d d¡V qdS )rX é N)rW rR Úrandint)Ú.0rU r$ r$ r% Ú <genexpr>¥ s z rsentence.<locals>.<genexpr>)r rO )rT r$ r$ r% Ú rsentence£ s r^ éè c C s¤ dddddddg}g }t | D ]r}t dd ¡}|dkrPt |¡}| d
|
| ¡ q |d krp| tt d d ¡¡ q |d kr t |¡}| d| ¡ q dd |¡ d S )z+Randomly generate an invalid HTML document.ÚpÚdivÚspanrU ÚbÚscriptÚtabler é z<%s>é rX rN z</%s>z<html>Ú
|
z</html>)rO rR r[ rS r r^ r )Ú num_elementsZ tag_namesÚelementsrU rS Ztag_namer$ r$ r% Úrdoc§ s
|
|
rk é c
|
C s$ t dt t| }t dt| dddgddfD ]z}d}z"t ¡ }t||}t ¡ }d}W n6 tk
|
r } zt d | t ¡ W 5 d
|
}~X Y nX |r4t d ||| f q4d d l m
|
} t ¡ }| |¡ t ¡ }t d|| d d
|
l } | ¡ }t ¡ }| |¡ t ¡ }t d|| d
|
S )z.Very basic head-to-head performance benchmark.z1Comparative parser benchmark on Beautiful Soup %sz3Generated a large invalid HTML document (%d bytes).r
|
r) r r FTr Nz"BS4+%s parsed the markup in %.2fs.r r z$Raw lxml parsed the markup in %.2fs.z(Raw html5lib parsed the markup in %.2fs.)r r rk ÚlenÚtimer r r r r
|
r ZHTMLr r Úparse)
|
ri r r! r" Úar# rc r r r r$ r$ r% Úbenchmark_parsers¹ s4
|
|
|
rq r
|
c C sX t ¡ }|j}t| }tt||d}t d|||¡ t |¡}|
|
d¡ | dd¡ dS )z7Use Python's profiler on a randomly generated document.)Úbs4r r! z bs4.BeautifulSoup(data, parser)Z
|
cumulativez _html5lib|bs4é2 N) ÚtempfileÚNamedTemporaryFiler rk Údictrr ÚcProfileZrunctxÚpstatsZStatsZ
|
sort_statsZ print_stats)ri r! Z
|
filehandleÚfilenamer ÚvarsÚstatsr$ r$ r% ÚprofileÙ s
|
|
r| Ú__main__)T)rL )rX )r_ )rl )rl r
|
)!rI Ú __license__rw Úior Ú html.parserr rr r r Z bs4.builderr Úosrx rR rt rn r r r& r4 r5 rK rQ rP rW r^ rk rq r| rF Ústdinr r$ r$ r$ r% Ú<module> s8 8
|
&
|
|
|
|
|
|
|
