U
|
¬ý°d¶ ã @ sn d Z ddlmZ ddlmZ ddlZddlZddlmZm Z m
|
Z
|
m Z m Z m Z ddlmZ ddlmZmZmZmZ ddlmZ dd lmZmZ dd
|
lmZ dd l m Z dd l m Z dd l m!Z! ddl"m#Z# ddl$m%Z% ddl&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z, ddl-m.Z. ddl/m0Z0 er ddl m1Z1 da2da3da4da5ddddZ6e 7d¡Z8e8fdddd d d Z9d d d!d"d#Z:d$d%d&d'd(d)Z;G d*d+ d+Z<G d,d- d-e<Z=ddd.d/Z>d0d1iZ?G d2d3 d3e<Z@ddd4d5ZAd6d7 ZBe@e@e=e=d8ZCd%d9d:d;d<ZDddd=d>ZEd?d@ ZFdAdB ZGdCdddddddDddEdddFdFdejHdGdHdId%dJdJd dKdLd%d%ddMdNdLdLdOdPdQdRdSdTZIdS )UzW
|
:mod:`pandas.io.html` is a module containing functionality for dealing with
|
HTML IO.
|
|
é )Ú annotations)ÚabcN)Ú TYPE_CHECKINGÚIterableÚLiteralÚPatternÚSequenceÚcast)Úlib)Ú
|
BaseBufferÚ DtypeBackendÚFilePathÚ
|
ReadBuffer)Úimport_optional_dependency)ÚAbstractMethodErrorÚEmptyDataError)Úcheck_dtype_backend)Ú is_list_like©Úisna)ÚIndex)Ú
|
MultiIndex)ÚSeries)Ú file_existsÚ
|
get_handleÚis_urlÚstringify_pathÚurlopenÚvalidate_header_arg©Ú pprint_thing)Ú
|
TextParser)Ú DataFrameFÚNone©Úreturnc C sL t rd S tddd} | d k atddd}|d k atddd}|d k ada d S )NÚbs4Úignore)Úerrorsz
|
lxml.etreeÚhtml5libT)Ú_IMPORTSr Ú_HAS_BS4Ú _HAS_LXMLÚ _HAS_HTML5LIB)r& Úlxmlr) © r/ úEd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\pandas/io/html.pyÚ
|
_importers> s r1 z[\r\n]+|\s{2,}Ústrr )ÚsÚregexr% c C s | d| ¡ ¡S )a
|
Replace extra whitespace inside of a string with a single space.
|
|
Parameters
|
----------
|
s : str or unicode
|
The string from which to remove extra whitespace.
|
regex : re.Pattern
|
The regular expression to use to remove extra whitespace.
|
|
Returns
|
-------
|
subd : str or unicode
|
`s` with all extra whitespace replaced with a single space.
|
ú )ÚsubÚstrip)r3 r4 r/ r/ r0 Ú_remove_whitespaceY s r8 z"int | Sequence[int] | slice | Noneúint | Sequence[int])Úskiprowsr% c C st t | tr2| jpd| jpd }}tt|| j|S t | tjsFt | rPt
|
d| S | dkr\dS t t | j ddS )a
|
Get an iterator given an integer, slice or container.
|
|
Parameters
|
----------
|
skiprows : int, slice, container
|
The iterator to use to skip rows; can also be a slice.
|
|
Raises
|
------
|
TypeError
|
* If `skiprows` is not a slice, integer, or Container
|
|
Returns
|
-------
|
it : iterable
|
A proper iterator to use to skip rows of a DataFrame.
|
r é r9 Nz& is not a valid type for skipping rows)Ú
|
isinstanceÚsliceÚstartÚstepÚlistÚrangeÚstopÚnumbersÚIntegralr r Ú TypeErrorÚtypeÚ__name__)r: r> r? r/ r/ r0 Ú _get_skiprowsl s
|
|
rH zFilePath | BaseBufferú
|
str | Nonez str | bytes)ÚobjÚencodingr% c C sx t | s$t| ds$t| trJt| rJt| d|d}|j ¡ }W 5 Q R X n*t| ttfr^| }nt dt
|
| j d|S )zµ
|
Try to read from a url, file or string.
|
|
Parameters
|
----------
|
obj : str, unicode, path object, or file-like object
|
|
Returns
|
-------
|
raw_text : str
|
ÚreadÚr)rK z Cannot read object of type 'ú') r Úhasattrr< r2 r r ÚhandlerL ÚbytesrE rF rG )rJ rK ZhandlesÚtextr/ r/ r0 Ú_read s ÿþýýrS c @ s e Zd ZdZdddddddd d
|
d Zd d Zdd Zdd Zdd Zdd Z dd Z
|
dd Z dd Z d d Z d d Zd d! Zd"d# Zd$d%d&d'Zd(d) Zd*S )+Ú_HtmlFrameParseraZ
|
Base class for parsers that parse HTML into DataFrames.
|
|
Parameters
|
----------
|
io : str or file-like
|
This can be either a string of raw HTML, a valid URL using the HTTP,
|
FTP, or FILE protocols or a file-like object.
|
|
match : str or regex
|
The text to match in the document.
|
|
attrs : dict
|
List of HTML <table> element attributes to match.
|
|
encoding : str
|
Encoding to be used by parser
|
|
displayed_only : bool
|
Whether or not items with "display:none" should be ignored
|
|
extract_links : {None, "all", "header", "body", "footer"}
|
Table elements in the specified section(s) with <a> tags will have their
|
href extracted.
|
|
.. versionadded:: 1.5.0
|
|
Attributes
|
----------
|
io : str or file-like
|
raw HTML, URL, or file-like object
|
|
match : regex
|
The text to match in the raw HTML
|
|
attrs : dict-like
|
A dictionary of valid table attributes to use to search for table
|
elements.
|
|
encoding : str
|
Encoding to be used by parser
|
|
displayed_only : bool
|
Whether or not items with "display:none" should be ignored
|
|
extract_links : {None, "all", "header", "body", "footer"}
|
Table elements in the specified section(s) with <a> tags will have their
|
href extracted.
|
|
.. versionadded:: 1.5.0
|
|
Notes
|
-----
|
To subclass this class effectively you must override the following methods:
|
* :func:`_build_doc`
|
* :func:`_attr_getter`
|
* :func:`_href_getter`
|
* :func:`_text_getter`
|
* :func:`_parse_td`
|
* :func:`_parse_thead_tr`
|
* :func:`_parse_tbody_tr`
|
* :func:`_parse_tfoot_tr`
|
* :func:`_parse_tables`
|
* :func:`_equals_tag`
|
See each method's respective documentation for details on their
|
functionality.
|
z.FilePath | ReadBuffer[str] | ReadBuffer[bytes]ú str | Patternúdict[str, str] | Noner2 Úboolú2Literal[(None, 'header', 'footer', 'body', 'all')]r# )ÚioÚmatchÚattrsrK Údisplayed_onlyÚ extract_linksr% c C s( || _ || _|| _|| _|| _|| _d S ©N)rY rZ r[ rK r\ r] )ÚselfrY rZ r[ rK r\ r] r/ r/ r0 Ú__init__é s z_HtmlFrameParser.__init__c s( ¡ j j¡} fdd|D S )z
|
Parse and return all tables from the DOM.
|
|
Returns
|
-------
|
list of parsed (header, body, footer) tuples from tables.
|
c 3 s | ]} |¡V qd S r^ )Ú_parse_thead_tbody_tfoot)Ú.0Útable©r_ r/ r0 Ú <genexpr> s z0_HtmlFrameParser.parse_tables.<locals>.<genexpr>)Ú _parse_tablesÚ
|
_build_docrZ r[ )r_ Útablesr/ rd r0 Ú parse_tablesù s z _HtmlFrameParser.parse_tablesc C s
|
| |¡S )aC
|
Return the attribute value of an individual DOM node.
|
|
Parameters
|
----------
|
obj : node-like
|
A DOM node.
|
|
attr : str or unicode
|
The attribute, such as "colspan"
|
|
Returns
|
-------
|
str or unicode
|
The attribute value.
|
)Úget)r_ rJ Úattrr/ r/ r0 Ú _attr_getter s z _HtmlFrameParser._attr_getterc C s t | dS )a
|
Return a href if the DOM node contains a child <a> or None.
|
|
Parameters
|
----------
|
obj : node-like
|
A DOM node.
|
|
Returns
|
-------
|
href : str or unicode
|
The href from the <a> child of the DOM node.
|
N©r ©r_ rJ r/ r/ r0 Ú _href_getter s z _HtmlFrameParser._href_getterc C s t | dS )a
|
Return the text of an individual DOM node.
|
|
Parameters
|
----------
|
obj : node-like
|
A DOM node.
|
|
Returns
|
-------
|
text : str or unicode
|
The text from an individual DOM node.
|
Nrm rn r/ r/ r0 Ú _text_getter( s z _HtmlFrameParser._text_getterc C s t | dS )a
|
Return the td elements from a row element.
|
|
Parameters
|
----------
|
obj : node-like
|
A DOM <tr> node.
|
|
Returns
|
-------
|
list of node-like
|
These are the elements of each row, i.e., the columns.
|
Nrm rn r/ r/ r0 Ú _parse_td8 s z_HtmlFrameParser._parse_tdc C s t | dS )a;
|
Return the list of thead row elements from the parsed table element.
|
|
Parameters
|
----------
|
table : a table element that contains zero or more thead elements.
|
|
Returns
|
-------
|
list of node-like
|
These are the <tr> row elements of a table.
|
Nrm ©r_ rc r/ r/ r0 Ú_parse_thead_trH s z _HtmlFrameParser._parse_thead_trc C s t | dS )aì
|
Return the list of tbody row elements from the parsed table element.
|
|
HTML5 table bodies consist of either 0 or more <tbody> elements (which
|
only contain <tr> elements) or 0 or more <tr> elements. This method
|
checks for both structures.
|
|
Parameters
|
----------
|
table : a table element that contains row elements.
|
|
Returns
|
-------
|
list of node-like
|
These are the <tr> row elements of a table.
|
Nrm rr r/ r/ r0 Ú_parse_tbody_trW s z _HtmlFrameParser._parse_tbody_trc C s t | dS )a,
|
Return the list of tfoot row elements from the parsed table element.
|
|
Parameters
|
----------
|
table : a table element that contains row elements.
|
|
Returns
|
-------
|
list of node-like
|
These are the <tr> row elements of a table.
|
Nrm rr r/ r/ r0 Ú_parse_tfoot_trj s z _HtmlFrameParser._parse_tfoot_trc C s t | dS )ad
|
Return all tables from the parsed DOM.
|
|
Parameters
|
----------
|
doc : the DOM from which to parse the table element.
|
|
match : str or regular expression
|
The text to search for in the DOM tree.
|
|
attrs : dict
|
A dictionary of table attributes that can be used to disambiguate
|
multiple tables on a page.
|
|
Raises
|
------
|
ValueError : `match` does not match any text in the document.
|
|
Returns
|
-------
|
list of node-like
|
HTML <table> elements to be parsed into raw data.
|
Nrm )r_ ÚdocrZ r[ r/ r/ r0 rf y s z _HtmlFrameParser._parse_tablesc C s t | dS )a?
|
Return whether an individual DOM node matches a tag
|
|
Parameters
|
----------
|
obj : node-like
|
A DOM node.
|
|
tag : str
|
Tag name to be checked for equality.
|
|
Returns
|
-------
|
boolean
|
Whether `obj`'s tag name is `tag`
|
Nrm ©r_ rJ Útagr/ r/ r0 Ú _equals_tag s z _HtmlFrameParser._equals_tagc C s t | dS )zÃ
|
Return a tree-like object that can be used to iterate over the DOM.
|
|
Returns
|
-------
|
node-like
|
The DOM from which to parse the table element.
|
Nrm rd r/ r/ r0 rg ¦ s z_HtmlFrameParser._build_docc s |¡} |¡} |¡} fdd}|sP|rP||d rP| | d¡¡ q. j|dd} j|dd} j|dd}|||fS )a
|
Given a table, return parsed header, body, and foot.
|
|
Parameters
|
----------
|
table_html : node-like
|
|
Returns
|
-------
|
tuple of (header, body, footer), each a list of list-of-text rows.
|
|
Notes
|
-----
|
Header and body are lists-of-lists. Top level list is a list of
|
rows. Each row is a list of str text.
|
|
Logic: Use <thead>, <tbody>, <tfoot> elements to identify
|
header, body, and footer, otherwise:
|
- Put all rows into body
|
- Move rows from top of body to header only if
|
all elements inside row are <th>
|
- Move rows from bottom of body to footer only if
|
all elements inside row are <th>
|
c s t fdd | ¡D S )Nc 3 s | ]} |d ¡V qdS )ÚthN)ry )rb Útrd r/ r0 re Ï s zS_HtmlFrameParser._parse_thead_tbody_tfoot.<locals>.row_is_all_th.<locals>.<genexpr>)Úallrq )Úrowrd r/ r0 Ú row_is_all_thÎ s z@_HtmlFrameParser._parse_thead_tbody_tfoot.<locals>.row_is_all_thr Úheader©ÚsectionÚbodyÚfooter)rs rt ru ÚappendÚpopÚ_expand_colspan_rowspan) r_ Z
|
table_htmlZ header_rowsZ body_rowsZ footer_rowsr~ r r r r/ rd r0 ra ± s
|
|
|
z)_HtmlFrameParser._parse_thead_tbody_tfootz%Literal[('header', 'footer', 'body')]r c C s g }g }|D ]T}g }g }d}| |¡} | D ]ì}
|
|r|d d |kr| d¡\} } } | | ¡ | dkrz| | | | d f¡ |d7 }q0t| |
|
¡}| jd|fkr²| |
|
¡}||f}t| |
|
d¡pÂd}t| |
|
d¡pÖd}t |D ]4}| |¡ |dkr| |||d f¡ |d7 }qâq,|D ]4\} } } | | ¡ | dkr | | | | d f¡ q | |¡ |}q |r¾g }g }|D ]4\} } } | | ¡ | dkrv| | | | d f¡ qv| |¡ |}qd|S )a*
|
Given a list of <tr>s, return a list of text rows.
|
|
Parameters
|
----------
|
rows : list of node-like
|
List of <tr>s
|
section : the section that the rows belong to (header, body or footer).
|
|
Returns
|
-------
|
list of list
|
Each returned row is a list of str text, or tuple (text, link)
|
if extract_links is not None.
|
|
Notes
|
-----
|
Any cell with ``rowspan`` or ``colspan`` will have its contents copied
|
to subsequent cells.
|
r r; r| ÚrowspanÚcolspan)
|
rq r
r r8 rp r] ro Úintrl rA )r_ Úrowsr Z all_textsÚ remainderÚtrZtextsZnext_remainderÚindexZtdsÚtdZprev_iZ prev_textZ prev_rowspanrR Úhrefr r Ú_r/ r/ r0 r Þ sT þ
|
|
|
|
|
|
|
|
|
|
|
|
|
z(_HtmlFrameParser._expand_colspan_rowspanc s | j s
|
|S fdd|D S )a
|
Return list of tables, potentially removing hidden elements
|
|
Parameters
|
----------
|
tbl_list : list of node-like
|
Type of list elements will vary depending upon parser used
|
attr_name : str
|
Name of the accessor for retrieving HTML attributes
|
|
Returns
|
-------
|
list of node-like
|
Return type matches `tbl_list`
|
c s. g | ]&}d t | dd¡ dd¡kr|qS )ú display:noneÚstyleÚ r5 )Úgetattrrj Úreplace)rb Úx©Ú attr_namer/ r0 Ú
|
<listcomp>C s
|
ÿþz:_HtmlFrameParser._handle_hidden_tables.<locals>.<listcomp>)r\ )r_ Ztbl_listr r/ r r0 Ú_handle_hidden_tables0 s
|
|
þz&_HtmlFrameParser._handle_hidden_tablesN)rG Ú
|
__module__Ú __qualname__Ú__doc__r` ri rl ro rp rq rs rt ru rf ry rg ra r r r/ r/ r/ r0 rT ¤ s D -RrT c s| e Zd ZdZdd fdd Zdd Zddd d
|
Zd d Zd d Zdd Z dd Z
|
dd Z dd Z dd Z dd Z ZS )Ú!_BeautifulSoupHtml5LibFrameParsera/
|
HTML to DataFrame parser that uses BeautifulSoup under the hood.
|
|
See Also
|
--------
|
pandas.io.html._HtmlFrameParser
|
pandas.io.html._LxmlFrameParser
|
|
Notes
|
-----
|
Documentation strings for this class are in the base class
|
:class:`pandas.io.html._HtmlFrameParser`.
|
r# r$ c s( t j|| ddlm} |d| _d S )Nr )Ú SoupStrainerrc )Úsuperr` r& r Ú _strainer)r_ ÚargsÚkwargsr ©Ú __class__r/ r0 r` Z s z*_BeautifulSoupHtml5LibFrameParser.__init__c
|
C s° | j j}|j||d}|s"tdg }t }| |d¡}|D ]V}| jrf|jt d¡dD ] } | ¡ qX||kr|j
|
|dd k r| |¡ | |¡ q<|s¬tdt |j |S )N)r[ zNo tables foundr[ zdisplay:\s*none)r )Ústringz!No tables found matching pattern )r¡ ÚnameÚfind_allÚ
|
ValueErrorÚsetr r\ ÚreÚcompileZ decomposeÚfindr ÚaddÚreprÚpattern)
|
r_ rv rZ r[ Z element_namerh ÚresultZ unique_tablesrc Úelemr/ r/ r0 rf ` s"
|
|
z/_BeautifulSoupHtml5LibFrameParser._parse_tablesrI c C s |j ddd}|sd S |d S )NÚaT)r r )r )r_ rJ r³ r/ r/ r0 ro x s z._BeautifulSoupHtml5LibFrameParser._href_getterc C s |j S r^ )rR rn r/ r/ r0 rp | s z._BeautifulSoupHtml5LibFrameParser._text_getterc C s
|
|j |kS r^ )r§ rw r/ r/ r0 ry s z-_BeautifulSoupHtml5LibFrameParser._equals_tagc C s |j dddS )N)r rz F©Ú recursive)r¨ ©r_ r} r/ r/ r0 rq s z+_BeautifulSoupHtml5LibFrameParser._parse_tdc C s
|
| d¡S )Nzthead tr©Úselectrr r/ r/ r0 rs
s z1_BeautifulSoupHtml5LibFrameParser._parse_thead_trc C s | d¡}|jddd}|| S )Nztbody trr Fr´ )r¸ r¨ ©r_ rc Z
|
from_tbodyZ from_rootr/ r/ r0 rt s
|
z1_BeautifulSoupHtml5LibFrameParser._parse_tbody_trc C s
|
| d¡S )Nztfoot trr· rr r/ r/ r0 ru s z1_BeautifulSoupHtml5LibFrameParser._parse_tfoot_trc C s& t | j| j}|s"td| j |S )Nz No text parsed from document: )rS rY rK r© )r_ Zraw_textr/ r/ r0 Ú_setup_build_doc s z2_BeautifulSoupHtml5LibFrameParser._setup_build_docc C sv ddl m} | ¡ }t|tr:| jd k r:| | j¡}d }n
|
|}| j}||d|d}| d¡D ]}| d|j ¡ q\|S )Nr )Ú BeautifulSoupr) )ÚfeaturesÚ from_encodingÚbrÚ
|
)
|
r& r» rº r< rQ rK Údecoder¨ Z replace_withrR )r_ r» ZbdocZudocr½ Zsoupr¾ r/ r/ r0 rg s z,_BeautifulSoupHtml5LibFrameParser._build_doc)rG r r r r` rf ro rp ry rq rs rt ru rº rg Ú __classcell__r/ r/ r¤ r0 r K s r c C s: d| kr| d¡| d< d dd | ¡ D ¡}d| dS )ar
|
Build an xpath expression to simulate bs4's ability to pass in kwargs to
|
search for attributes when using the lxml parser.
|
|
Parameters
|
----------
|
attrs : dict
|
A dict of HTML attributes. These are NOT checked for validity.
|
|
Returns
|
-------
|
expr : unicode
|
An XPath expression that checks for the given HTML attributes.
|
Úclass_Úclassz and c S s$ g | ] \}}d | dt | qS )ú@ú=)r¯ )rb ÚkÚvr/ r/ r0 r ½ s z%_build_xpath_expr.<locals>.<listcomp>ú[ú])r
ÚjoinÚitems)r[ r3 r/ r/ r0 Ú_build_xpath_exprª s rÌ r« z$http://exslt.org/regular-expressionsc @ s^ e Zd ZdZddddZdd Zdd Zd
|
d Zd d Zdd Z dd Z
|
dd Z dd Z dS )Ú_LxmlFrameParseraY
|
HTML to DataFrame parser that uses lxml under the hood.
|
|
Warning
|
-------
|
This parser can only handle HTTP, FTP, and FILE urls.
|
|
See Also
|
--------
|
_HtmlFrameParser
|
_BeautifulSoupLxmlFrameParser
|
|
Notes
|
-----
|
Documentation strings for this class are in the base class
|
:class:`_HtmlFrameParser`.
|
rI r$ c C s | d¡}|sd S |d S )Nz
|
.//a/@hrefr ©Úxpath)r_ rJ r r/ r/ r0 ro × s
|
z _LxmlFrameParser._href_getterc C s | ¡ S r^ )Ú text_contentrn r/ r/ r0 rp Û s z _LxmlFrameParser._text_getterc C s
|
| d¡S )Nú ./td|./thrÎ r¶ r/ r/ r0 rq Þ s z_LxmlFrameParser._parse_tdc C s¢ |j }dt| d}|r&|t|7 }|j|td}| |d¡}| jr|D ]<}| d¡D ],}d|j dd¡ d d¡krX|
|
¡ |¡ qXqJ|st d
|
t| |S ) Nz //table[.//text()[re:test(., z)]])Ú
|
namespacesÚattribz .//*[@style]r r r r5 z No tables found matching regex ) r° r¯ rÌ rÏ Ú _re_namespacer r\ rÓ rj r Z getparentÚremover© ) r_ rv rZ r£ r° Z
|
xpath_exprrh rc r² r/ r/ r0 rf ã s z _LxmlFrameParser._parse_tablesc C s
|
|j |kS r^ )rx rw r/ r/ r0 ry ÿ s z _LxmlFrameParser._equals_tagc
|
C s: ddl m} ddlm}m}m} |d| jd}zbt| jr^t | j}|||d}W 5 Q R X n|| j|d}z |
|
¡ }W n t k
|
r Y nX W nf t t fk
|
rö } zDt| jsâ|| j|d}z |
|
¡ }W qæ t k
|
rÞ Y qæX n|W 5 d}~X Y n X t|ds|d ddd| d
|
¡D ]} d | jp,d | _q |S ) a
|
Raises
|
------
|
ValueError
|
* If a URL that lxml cannot parse is passed.
|
|
Exception
|
* Any other ``Exception`` thrown. For example, trying to parse a
|
URL that is syntactically correct on a machine with no internet
|
connection will fail.
|
|
See Also
|
--------
|
pandas.io.html._HtmlFrameParser._build_doc
|
r )ÚXMLSyntaxError)Ú
|
HTMLParserÚ
|
fromstringÚparseT)ZrecoverrK )ÚparserNrÐ z no text parsed from documentz*//brr¿ r )Z
|
lxml.etreerÖ Z lxml.htmlr× rØ rÙ rK r rY r ÚgetrootÚAttributeErrorÚUnicodeDecodeErrorÚOSErrorrO rÏ Útail)
|
r_ rÖ r× rØ rÙ rÚ ÚfrM Úer¾ r/ r/ r0 rg s2
|
|
|
z_LxmlFrameParser._build_docc C s@ g }| d¡D ],}| | d¡¡ | d¡}|r| |¡ q|S )Nz.//theadú./trrÑ )rÏ Úextendr )r_ rc r ÚtheadZelements_at_rootr/ r/ r0 rs : s
|
z _LxmlFrameParser._parse_thead_trc C s | d¡}| d¡}|| S )Nz .//tbody//trrâ rÎ r¹ r/ r/ r0 rt L s
|
|
z _LxmlFrameParser._parse_tbody_trc C s
|
| d¡S )Nz .//tfoot//trrÎ rr r/ r/ r0 ru R s z _LxmlFrameParser._parse_tfoot_trN) rG r r r ro rp rq rf ry rg rs rt ru r/ r/ r/ r0 rÍ Ä s 8rÍ c C s^ dd | D }t |}| ¡ }|||k }dg}| ¡ D ] \}}| | ||| 7 < q8d S )Nc S s g | ] }t |qS r/ )Úlen)rb r² r/ r/ r0 r W s z$_expand_elements.<locals>.<listcomp>r )r ÚmaxrË )r ÚdataZlensZlens_maxZnot_maxÚemptyÚindÚlengthr/ r/ r0 Ú_expand_elementsV s rë c
|
K sª | d¡\}}}| d¡}t| d | d< |rb|| }|d krbt|dkrPd}ndd t|D }|rn||7 }t| t|fd|i| }| ¡ W 5 Q R £ S Q R X d S )Nrç r r: r; r c S s& g | ] \}}t d d |D r|qS )c s s | ]
|
}|V qd S r^ r/ )rb rR r/ r/ r0 re n s z,_data_to_frame.<locals>.<listcomp>.<genexpr>)Úany)rb Úir} r/ r/ r0 r n s z"_data_to_frame.<locals>.<listcomp>)r
rH rå Ú enumeraterë r! rL )r£ Úheadr Zfootr Útpr/ r/ r0 Ú_data_to_framea s
|
rñ )r. Nr) r& ztype[_HtmlFrameParser])Úflavorr% c C sh t t ¡ }| |kr*tt| d| | dkrTts>tdtsJtdtd}n t s`tdt| S )aº
|
Choose the parser based on the input flavor.
|
|
Parameters
|
----------
|
flavor : str
|
The type of parser to use. This must be a valid backend.
|
|
Returns
|
-------
|
cls : _HtmlFrameParser subclass
|
The parser class based on the requested input flavor.
|
|
Raises
|
------
|
ValueError
|
* If `flavor` is not a valid backend.
|
ImportError
|
* If you do not have the requested `flavor`
|
z* is not a valid flavor, valid flavors are )r& r) z%html5lib not found, please install itz1BeautifulSoup4 (bs4) not found, please install itr& z!lxml not found, please install it)
|
r@ Ú_valid_parsersÚkeysr© r¯ r- Ú ImportErrorr+ r r, )rò Z valid_parsersr& r/ r/ r0 Ú_parser_dispatch s ÿ
|
rö c C s d dd | D ¡}d| dS )Nz, c S s g | ] }t |qS r/ r )rb Úelr/ r/ r0 r « s z!_print_as_set.<locals>.<listcomp>Ú{Ú})rÊ )r3 Úargr/ r/ r0 Ú _print_as_setª s rû c C s | d krd} nvt | tr | f} ndt | tjrZtdd | D stdtt| j dn*t | trlt| nt| }|d7 }t |t
|
| } t t }t | }||@ s¾t t | dt | | S )N)r. r& c s s | ]}t |tV qd S r^ )r< r2 )rb Úflavr/ r/ r0 re µ s z#_validate_flavor.<locals>.<genexpr>zObject of type z is not an iterable of stringsz is not a valid flavorz2 is not a valid set of flavors, valid flavors are )r< r2 r r r| rE r¯ rF rG r© Útuplerª ró rû )rò ÚmsgZ valid_flavorsZ
|
flavor_setr/ r/ r0 Ú_validate_flavor¯ s(
|
ÿÿrÿ c K sD t | } t |¡}d } | D ]}
|
t|
|
} | ||||||} z | ¡ } W nh tk
|
r¬ } zJt|drt| ¡ rt| d¡ n$t|dr| ¡ std|
|
d||} W 5 d }~X Y qX qÄq| d k sÀt | g }| D ]r}zRt
|
f d|i|}|dkrt |j t rtdd |j D d d
|
|_ | |¡ W qÌ tk
|
r< Y qÌY qÌX qÌ|S ) NÚseekabler z The flavor z failed to parse your input. Since you passed a non-rewindable file object, we can't rewind it to try another parser. Try read_html() with a different flavor.rç )r| r c s s. | ]&}|d t |d r dn|d fV qdS )r r; Nr )rb Úcolr/ r/ r0 re ö s z_parse.<locals>.<genexpr>F)Z tupleize_cols)rÿ r« r¬ rö ri r© rO r ÚseekÚAssertionErrorrñ r< Úcolumnsr r r r )rò rY rZ r[ rK r\ r] r£ Zcompiled_matchZretainedrü rÚ Úprh ZcaughtÚretrc Zdfr/ r/ r0 Ú_parseË sJ
|
|
ÿû ÿþ r z.+ú,Ú.T)rZ rò r Ú index_colr: r[ Ú parse_datesÚ thousandsrK ÚdecimalÚ
|
convertersÚ na_valuesÚkeep_default_nar\ r] Ú dtype_backendzFilePath | ReadBuffer[str]rU zint | Sequence[int] | NonerV rW z dict | NonezIterable[object] | NonerX z DtypeBackend | lib.NoDefaultzlist[DataFrame])rY rZ rò r r
|
r: r[ r r rK r r r r r\ r] r r% c C s| t t|tjr"|dk r"td|dkr:td| dt| t| t| } t|| |||||||| |
|
| | | |||dS )a
|
Read HTML tables into a ``list`` of ``DataFrame`` objects.
|
|
Parameters
|
----------
|
io : str, path object, or file-like object
|
String, path object (implementing ``os.PathLike[str]``), or file-like
|
object implementing a string ``read()`` function.
|
The string can represent a URL or the HTML itself. Note that
|
lxml only accepts the http, ftp and file url protocols. If you have a
|
URL that starts with ``'https'`` you might try removing the ``'s'``.
|
|
match : str or compiled regular expression, optional
|
The set of tables containing text matching this regex or string will be
|
returned. Unless the HTML is extremely simple you will probably need to
|
pass a non-empty string here. Defaults to '.+' (match any non-empty
|
string). The default value will return all tables contained on a page.
|
This value is converted to a regular expression so that there is
|
consistent behavior between Beautiful Soup and lxml.
|
|
flavor : str, optional
|
The parsing engine to use. 'bs4' and 'html5lib' are synonymous with
|
each other, they are both there for backwards compatibility. The
|
default of ``None`` tries to use ``lxml`` to parse and if that fails it
|
falls back on ``bs4`` + ``html5lib``.
|
|
header : int or list-like, optional
|
The row (or list of rows for a :class:`~pandas.MultiIndex`) to use to
|
make the columns headers.
|
|
index_col : int or list-like, optional
|
The column (or list of columns) to use to create the index.
|
|
skiprows : int, list-like or slice, optional
|
Number of rows to skip after parsing the column integer. 0-based. If a
|
sequence of integers or a slice is given, will skip the rows indexed by
|
that sequence. Note that a single element sequence means 'skip the nth
|
row' whereas an integer means 'skip n rows'.
|
|
attrs : dict, optional
|
This is a dictionary of attributes that you can pass to use to identify
|
the table in the HTML. These are not checked for validity before being
|
passed to lxml or Beautiful Soup. However, these attributes must be
|
valid HTML table attributes to work correctly. For example, ::
|
|
attrs = {'id': 'table'}
|
|
is a valid attribute dictionary because the 'id' HTML tag attribute is
|
a valid HTML attribute for *any* HTML tag as per `this document
|
<https://html.spec.whatwg.org/multipage/dom.html#global-attributes>`__. ::
|
|
attrs = {'asdf': 'table'}
|
|
is *not* a valid attribute dictionary because 'asdf' is not a valid
|
HTML attribute even if it is a valid XML attribute. Valid HTML 4.01
|
table attributes can be found `here
|
<http://www.w3.org/TR/REC-html40/struct/tables.html#h-11.2>`__. A
|
working draft of the HTML 5 spec can be found `here
|
<https://html.spec.whatwg.org/multipage/tables.html>`__. It contains the
|
latest information on table attributes for the modern web.
|
|
parse_dates : bool, optional
|
See :func:`~read_csv` for more details.
|
|
thousands : str, optional
|
Separator to use to parse thousands. Defaults to ``','``.
|
|
encoding : str, optional
|
The encoding used to decode the web page. Defaults to ``None``.``None``
|
preserves the previous encoding behavior, which depends on the
|
underlying parser library (e.g., the parser library will try to use
|
the encoding provided by the document).
|
|
decimal : str, default '.'
|
Character to recognize as decimal point (e.g. use ',' for European
|
data).
|
|
converters : dict, default None
|
Dict of functions for converting values in certain columns. Keys can
|
either be integers or column labels, values are functions that take one
|
input argument, the cell (not column) content, and return the
|
transformed content.
|
|
na_values : iterable, default None
|
Custom NA values.
|
|
keep_default_na : bool, default True
|
If na_values are specified and keep_default_na is False the default NaN
|
values are overridden, otherwise they're appended to.
|
|
displayed_only : bool, default True
|
Whether elements with "display: none" should be parsed.
|
|
extract_links : {None, "all", "header", "body", "footer"}
|
Table elements in the specified section(s) with <a> tags will have their
|
href extracted.
|
|
.. versionadded:: 1.5.0
|
|
dtype_backend : {"numpy_nullable", "pyarrow"}, defaults to NumPy backed DataFrames
|
Which dtype_backend to use, e.g. whether a DataFrame should have NumPy
|
arrays, nullable dtypes are used for all dtypes that have a nullable
|
implementation when "numpy_nullable" is set, pyarrow is used for all
|
dtypes if "pyarrow" is set.
|
|
The dtype_backends are still experimential.
|
|
.. versionadded:: 2.0
|
|
Returns
|
-------
|
dfs
|
A list of DataFrames.
|
|
See Also
|
--------
|
read_csv : Read a comma-separated values (csv) file into DataFrame.
|
|
Notes
|
-----
|
Before using this function you should read the :ref:`gotchas about the
|
HTML parsing libraries <io.html.gotchas>`.
|
|
Expect to do some cleanup after you call this function. For example, you
|
might need to manually assign column names if the column names are
|
converted to NaN when you pass the `header=0` argument. We try to assume as
|
little as possible about the structure of the table and push the
|
idiosyncrasies of the HTML contained in the table to the user.
|
|
This function searches for ``<table>`` elements and only for ``<tr>``
|
and ``<th>`` rows and ``<td>`` elements within each ``<tr>`` or ``<th>``
|
element in the table. ``<td>`` stands for "table data". This function
|
attempts to properly handle ``colspan`` and ``rowspan`` attributes.
|
If the function has a ``<thead>`` argument, it is used to construct
|
the header, otherwise the function attempts to find the header within
|
the body (by putting rows with only ``<th>`` elements into the header).
|
|
Similar to :func:`~read_csv` the `header` argument is applied
|
**after** `skiprows` is applied.
|
|
This function will *always* return a list of :class:`DataFrame` *or*
|
it will fail, e.g., it will *not* return an empty list.
|
|
Examples
|
--------
|
See the :ref:`read_html documentation in the IO section of the docs
|
<io.read_html>` for some examples of reading in HTML tables.
|
r zPcannot skip rows starting from the end of the data (you passed a negative value))Nr r r r| zO`extract_links` must be one of {None, "header", "footer", "body", "all"}, got "ú")rò rY rZ r r
|
r: r r r[ rK r r r r r\ r] r ) r1 r< rC rD r© r r r r )rY rZ rò r r
|
r: r[ r r rK r r r r r\ r] r r/ r/ r0 Ú read_html s@ )ÿ
|
ÿïr )Jr Ú
|
__future__r Ú collectionsr rC r« Útypingr r r r r r Z pandas._libsr
|
Zpandas._typingr r r r Zpandas.compat._optionalr Z pandas.errorsr r Zpandas.util._validatorsr Zpandas.core.dtypes.commonr Zpandasr Zpandas.core.indexes.baser Zpandas.core.indexes.multir Zpandas.core.seriesr Zpandas.io.commonr r r r r r Zpandas.io.formats.printingr Zpandas.io.parsersr! r" r* r+ r, r- r1 r¬ Z_RE_WHITESPACEr8 rH rS rT r rÌ rÔ rÍ rë rñ ró rö rû rÿ r Z
|
no_defaultr r/ r/ r/ r0 Ú<module> s
|
*_ ü) 8î
|
