U
|
H=®dË ã @ sD d dl Z d dlmZ d dlZd dlZd dlZd dlZd dlZz d dlZW n e k
|
rd d dl
|
ZY nX d dl Z ddl m Z ddlmZmZmZmZmZmZmZmZmZmZmZmZm Z m Z m Z ddl m Z m!Z!m"Z" ddl#m$Z$m%Z% ddl&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/ dd l0m1Z1m2Z2 dd
|
l3m4Z4m5Z5 e 6e7¡Z8e 9d ¡Z:e 9d ej;¡Z<e 9d ¡Z=dZ>d-ddZ?G dd deZ@G dd deAZBG dd deBZCG dd deBZDG dd deAZEG dd d eBZFG d d d eBZGG d d d eBZHG d!d" d"eBZIG d#d$ d$eBZJeJeH eFd%d&d'd(d)ZKeKjLZLe 9d*¡ZMG d+d, d,eAZNdS ).é N)ÚBytesIOé )ÚDistlibException)ÚurljoinÚurlparseÚ
|
urlunparseÚ url2pathnameÚ pathname2urlÚqueueÚquoteÚunescapeÚ string_typesÚ build_openerÚHTTPRedirectHandlerÚ text_typeÚRequestÚ HTTPErrorÚURLError)Ú DistributionÚDistributionPathÚ make_dist)ÚMetadataÚMetadataInvalidError) Úcached_propertyÚparse_credentialsÚ ensure_slashÚsplit_filenameÚget_project_dataÚparse_requirementÚparse_name_and_versionÚ ServerProxyÚnormalize_name)Ú
|
get_schemeÚUnsupportedVersionError)ÚWheelÚ is_compatiblez^(\w+)=([a-f0-9]+)z;\s*charset\s*=\s*(.*)\s*$z text/html|application/x(ht)?mlzhttps://pypi.org/pypic C s6 | dkr t } t| dd}z | ¡ W ¢S |d X dS )z
|
Return all distribution names known by an index.
|
:param url: The URL of the index.
|
:return: A list of all known distribution names.
|
Nç @©ÚtimeoutÚclose)Ú DEFAULT_INDEXr Ú list_packages)ÚurlÚclient© r. úSD:\z\workplace\VsCode\pyvenv\venv\Lib\site-packages\pip/_vendor/distlib/locators.pyÚget_all_distribution_names) s r0 c @ s$ e Zd ZdZdd Ze Z ZZdS )ÚRedirectHandlerzE
|
A class to work around a bug in some Python 3.2.x releases.
|
c C s d }dD ]}||kr|| } q"q|d kr.d S t |}|jdkrnt| ¡ |}t|drf| ||¡ n|||< t | |||||¡S )N)ÚlocationÚuriÚ Úreplace_header)r Úschemer Ú get_full_urlÚhasattrr5 ÚBaseRedirectHandlerÚhttp_error_302) ÚselfÚreqÚfpÚcodeÚmsgÚheadersÚnewurlÚkeyÚurlpartsr. r. r/ r: @ s
|
|
ÿz RedirectHandler.http_error_302N)Ú__name__Ú
|
__module__Ú __qualname__Ú__doc__r: Úhttp_error_301Úhttp_error_303Úhttp_error_307r. r. r. r/ r1 7 s r1 c @ s¶ e Zd ZdZdZdZdZdZed Zd)dd Z d
|
d Z
|
d d Z dd Z dd Z dd Zee eZdd Zdd Zdd Zdd Zd d Zd d Zd d! Zd"d# Zd$d% Zd*d'd(ZdS )+ÚLocatorzG
|
A base class for locators - things that locate distributions.
|
)z.tar.gzz.tar.bz2z.tarz.zipz.tgzz.tbz)z.eggz.exeú.whl)z.pdfN)rL Údefaultc C s, i | _ || _tt | _d| _t ¡ | _dS )a^
|
Initialise an instance.
|
:param scheme: Because locators look for most recent versions, they
|
need to know the version scheme to use. This specifies
|
the current PEP-recommended scheme - use ``'legacy'``
|
if you need to support existing distributions on PyPI.
|
N) Ú_cacher6 r r1 ÚopenerÚmatcherr
|
ÚQueueÚerrors)r; r6 r. r. r/ Ú__init__f s
|
zLocator.__init__c C sV g }| j ¡ sRz| j d¡}| |¡ W n | j jk
|
rD Y qY nX | j ¡ q|S )z8
|
Return any errors which have occurred.
|
F)rR ÚemptyÚgetÚappendÚEmptyÚ task_done)r; ÚresultÚer. r. r/ Ú
|
get_errorsy s
|
|
zLocator.get_errorsc C s | ¡ dS )z>
|
Clear any errors which may have been logged.
|
N)r[ ©r; r. r. r/ Ú clear_errors s zLocator.clear_errorsc C s | j ¡ d S ©N)rN Úclearr\ r. r. r/ Ú clear_cache s zLocator.clear_cachec C s | j S r^ ©Ú_schemer\ r. r. r/ Ú _get_scheme s zLocator._get_schemec C s
|
|| _ d S r^ ra )r; Úvaluer. r. r/ Ú _set_scheme s zLocator._set_schemec C s t ddS )a=
|
For a given project, get a dictionary mapping available versions to Distribution
|
instances.
|
|
This should be implemented in subclasses.
|
|
If called from a locate() request, self.matcher will be set to a
|
matcher for the requirement to satisfy, otherwise it will be None.
|
ú Please implement in the subclassN©ÚNotImplementedError)r; Únamer. r. r/ Ú _get_project s
|
zLocator._get_projectc C s t ddS )úJ
|
Return all the distribution names known to this locator.
|
rf Nrg r\ r. r. r/ Úget_distribution_names¥ s z Locator.get_distribution_namesc C sL | j dkr| |¡}n2|| j kr,| j | }n | ¡ | |¡}|| j |< |S )zÖ
|
For a given project, get a dictionary mapping available versions to Distribution
|
instances.
|
|
This calls _get_project to do all the work, and just implements a caching layer on top.
|
N)rN rj r] )r; ri rY r. r. r/ Ú get_project« s
|
|
|
|
zLocator.get_projectc C s^ t |}t |j¡}d}| d¡}| | j¡}|rBtt|| j}|j dkd|j
|
k||||fS )zu
|
Give an url a score which can be used to choose preferred URLs
|
for a given project release.
|
TrL Úhttpszpypi.org) r Ú posixpathÚbasenameÚpathÚendswithÚdownloadable_extensionsr% r$ Ú
|
wheel_tagsr6 Únetloc)r; r, Útrp Ú
|
compatibleÚis_wheelZis_downloadabler. r. r/ Ú score_url¼ s
|
ÿzLocator.score_urlc C sR |}|rN| |¡}| |¡}||kr(|}||kr@t d||¡ nt d||¡ |S )a{
|
Choose one of two URLs where both are candidates for distribution
|
archives for the same version of a distribution (for example,
|
.tar.gz vs. zip).
|
|
The current implementation favours https:// URLs over http://, archives
|
from PyPI over those from other locations, wheel compatibility (if a
|
wheel) and then the archive name.
|
zNot replacing %r with %rzReplacing %r with %r)ry ÚloggerÚdebug)r; Úurl1Úurl2rY Ús1Ús2r. r. r/ Ú
|
prefer_urlË s
|
|
|
zLocator.prefer_urlc C s
|
t ||S )zZ
|
Attempt to split a filename in project name, version and Python version.
|
)r )r; ÚfilenameÚ project_namer. r. r/ r á s zLocator.split_filenamec C s dd }d}t |\}}}}} }
|
|
|
¡ d¡r<t d||
|
¡ t |
|
¡} | rX| ¡ \} } nd\} } |}|r|d dkr|dd
}| d ¡r>z~t |}t
|
|| j s°t d
|
|¡ nX|dkr¾d }n ||j |}|r|j |j |jt||||| d fd dd |jD ¡d}W n0 tk
|
r: } zt d|¡ W 5 d}~X Y nX nÄ| | j¡sZt d|¡ n¨t |¡ }}| jD ]}| |¡rn|dt|
}| ||¡}|s°t d|¡ nH|\}}}|rÌ|||rø|||t||||| d fd}|rø||d< qqn|r| r| |d| < |S )a
|
See if a URL is a candidate for a download URL for a project (the URL
|
has typically been scraped from an HTML page).
|
|
If it is, a dictionary is returned with keys "name", "version",
|
"filename" and "url"; otherwise, None is returned.
|
c S s t | t |kS r^ )r! )Zname1Zname2r. r. r/ Ú same_projectï s z:Locator.convert_url_to_download_info.<locals>.same_projectNzegg=z %s: version hint in fragment: %r)NNéÿÿÿÿú/rL zWheel not compatible: %sTr4 z, c S s" g | ]}d t|dd
¡qS )Ú.é N)ÚjoinÚlist)Ú.0Úvr. r. r/ Ú
|
<listcomp> s z8Locator.convert_url_to_download_info.<locals>.<listcomp>)ri Úversionr r, úpython-versionzinvalid path for wheel: %szNot downloadable: %sz No match for project/version: %s)ri r r r, r ú %s_digest)r ÚlowerÚ
|
startswithrz r{ Ú HASHER_HASHÚmatchÚgroupsrr r$ r% rt ri r r r r ÚpyverÚ ExceptionÚwarningrs ro rp Úlenr )r; r, r r rY r6 ru rq ÚparamsÚqueryÚfragÚmÚalgoÚdigestZorigpathÚwheelÚincluderZ r Úextrv ri r r r. r. r/ Ú convert_url_to_download_infoç s ÿ
|
ÿÿú
|
|
|
ÿü
|
z$Locator.convert_url_to_download_infoc C sh d}d|kr6|d }dD ] }||kr||| f} q6q|sddD ]$}d| }||kr>||| f} qdq>|S )zò
|
Get a digest from a dictionary by looking at a "digests" dictionary
|
or keys of the form 'algo_digest'.
|
|
Returns a 2-tuple (algo, digest) if found, else None. Currently
|
looks only for SHA256, then MD5.
|
NÚdigests)Úsha256Úmd5r r. )r; ÚinforY r£ r rB r. r. r/ Ú _get_digest1 s zLocator._get_digestc C s® | d¡}| d¡}||kr,|| }|j}nt||| jd}|j}| |¡ |_}|d }||d |< |j|d kr| |j|¡|_|d |t ¡
|
|¡ | |_ |||< dS )zñ
|
Update a result dictionary (the final result from _get_project) with a
|
dictionary for a specific version, which typically holds information
|
gleaned from a filename or URL for an archive for the distribution.
|
ri r ©r6 r, r£ ÚurlsN) ÚpopÚmetadatar r6 r§ r Ú
|
source_urlr Ú
|
setdefaultÚsetÚaddÚlocator) r; rY r¦ ri r ÚdistÚmdr r, r. r. r/ Ú_update_version_dataH s
|
|
z Locator._update_version_dataFc C s¤ d}t |}|dkr td| t| j}| |j¡ | _}t d|t|j ¡ |
|
|j ¡}t |dkr2g }|j } |D ]z}
|
|
|
dkrqxzH| |
|
¡s¢t d||
|
¡ n*|s°| |
|
js¼| |
|
¡ nt d|
|
|j ¡ W qx tk
|
rð t d||
|
¡ Y qxX qxt |d krt||jd
|
}|r2t d |¡ |d } || }|r|jrH|j|_| d i ¡ | t ¡|_i } | di ¡} |jD ] }|| krv| | | |< qv| |_d| _|S )a
|
Find the most recent distribution which matches the given
|
requirement.
|
|
:param requirement: A requirement of the form 'foo (1.0)' or perhaps
|
'foo (>= 1.0, < 2.0, != 1.3)'
|
:param prereleases: If ``True``, allow pre-release versions
|
to be located. Otherwise, pre-release versions
|
are not returned.
|
:return: A :class:`Distribution` instance, or ``None`` if no such
|
distribution could be located.
|
NzNot a valid requirement: %rzmatcher: %s (%s)r ©r© r£ z%s did not match %rz%skipping pre-release version %s of %szerror matching %s with %rr )rB zsorted list: %sr r© r£ )r r r" r6 rP Ú requirementrz r{ ÚtyperD rm ri r Z version_classr Ú is_prereleaserV r r ÚsortedrB ÚextrasrU r® Ú download_urlsr£ )r; rµ Ú prereleasesrY Úrr6 rP ÚversionsÚslistZvclsÚkr ÚdÚsdr, r. r. r/ Úlocate_ sX
|
|
ÿ
|
|
zLocator.locate)rM )F)rD rE rF rG Úsource_extensionsÚbinary_extensionsÚexcluded_extensionsrt rs rS r[ r] r` rc re Úpropertyr6 rj rl rm ry r r r¢ r§ r³ r r. r. r. r/ rK V s.
|
|
JrK c s0 e Zd ZdZ fddZdd Zdd Z ZS )ÚPyPIRPCLocatorz
|
This locator uses XML-RPC to locate distributions. It therefore
|
cannot be used with simple mirrors (that only mirror file content).
|
c s* t t| jf | || _t|dd| _dS )z
|
Initialise an instance.
|
|
:param url: The URL to use for XML-RPC.
|
:param kwargs: Passed to the superclass constructor.
|
r& r' N)ÚsuperrÇ rS Úbase_urlr r- ©r; r, Úkwargs©Ú __class__r. r/ rS s zPyPIRPCLocator.__init__c C s t | j ¡ S ©rk )r® r- r+ r\ r. r. r/ rl « s z%PyPIRPCLocator.get_distribution_namesc C s i i d}| j |d¡}|D ]Þ}| j ||¡}| j ||¡}t| jd}|d |_|d |_| d¡|_ | dg ¡|_
|
| d¡|_ t |}|r |d } | d
|
|_ | | ¡|_| |_|||< |D ]:} | d
|
}
|
| | ¡} |d |t ¡ |
|
¡ | |d |
|
< q¾q |S ) Nr´ Tr¨ ri r ÚlicenseÚkeywordsÚsummaryr r, r© r£ )r- Zpackage_releasesZ release_urlsZ release_datar r6 ri r rU rÏ rÐ rÑ r r¬ r§ r r° r r® r¯ ) r; ri rY r½ r r© Údatar« r± r¦ r, r r. r. r/ rj ± s0
|
|
|
|
|
zPyPIRPCLocator._get_project©rD rE rF rG rS rl rj Ú __classcell__r. r. rÌ r/ rÇ s rÇ c s0 e Zd ZdZ fddZdd Zdd Z ZS )ÚPyPIJSONLocatorzw
|
This locator uses PyPI's JSON interface. It's very limited in functionality
|
and probably not worth using.
|
c s t t| jf | t|| _d S r^ )rÈ rÕ rS r rÉ rÊ rÌ r. r/ rS Ð s zPyPIJSONLocator.__init__c C s t ddS ©rk z Not available from this locatorNrg r\ r. r. r/ rl Ô s z&PyPIJSONLocator.get_distribution_namesc
|
C s
|
i i d}t | jdt| }z¤| j |¡}| ¡ ¡ }t |¡}t | j
|
d}|d }|d |_ |d |_ | d¡|_| dg ¡|_| d ¡|_t|}| |_|d
|
} |||j < |d
|
D ]T}
|
|
|
d }|j |¡ | |
|
¡|j|< |d
|
|j t ¡ |¡ | |
|
¡|d |< qº|d ¡ D ]¤\} } | |j kr4q t | j
|
d} |j | _ | | _ t| }| |_||| < | D ]T}
|
|
|
d }|j |¡ | |
|
¡|j|< |d
|
| t ¡ |¡ | |
|
¡|d |< qhq W n@ tk
|
r } z | j t |¡ t d|¡ W 5 d }~X Y nX |S )Nr´ z%s/jsonr¨ r¦ ri r rÏ rÐ rÑ r© r, r£ ZreleaseszJSON fetch failed: %s) r rÉ r rO ÚopenÚreadÚdecodeÚjsonÚloadsr r6 ri r rU rÏ rÐ rÑ r r° rº r¯ r§ r£ r r® Úitemsr rR Úputr rz Ú exception)r; ri rY r, ÚresprÒ rÀ r² r± r© r¦ r ÚinfosZomdÚodistrZ r. r. r/ rj Ú sT
|
|
|
|
|
z PyPIJSONLocator._get_projectrÓ r. r. rÌ r/ rÕ Ë s rÕ c @ s` e Zd ZdZe dejejB ejB ¡Z e dejejB ¡Z
|
dd Z e dej¡Z e dd Zd S )
|
ÚPagez4
|
This class represents a scraped HTML page.
|
zÝ
|
(rel\s*=\s*(?:"(?P<rel1>[^"]*)"|'(?P<rel2>[^']*)'|(?P<rel3>[^>\s
|
]*))\s+)?
|
href\s*=\s*(?:"(?P<url1>[^"]*)"|'(?P<url2>[^']*)'|(?P<url3>[^>\s
|
]*))
|
(\s+rel\s*=\s*(?:"(?P<rel4>[^"]*)"|'(?P<rel5>[^']*)'|(?P<rel6>[^>\s
|
]*)))?
|
z!<base\s+href\s*=\s*['"]?([^'">]+)c C s4 || _ | | _| _| j | j ¡}|r0| d¡| _dS )zk
|
Initialise an instance with the Unicode page contents and the URL they
|
came from.
|
r N)rÒ rÉ r, Ú_baseÚsearchÚgroup)r; rÒ r, r r. r. r/ rS s
|
z Page.__init__z[^a-z0-9$&+,/:;=?@.#%_\\|-]c C s¾ dd }t }| j | j¡D ]}| d¡}|d pX|d pX|d pX|d pX|d pX|d }|d
|
pp|d pp|d }t| j|}t|}| j d d |¡}|
|
||f¡ q t |dd dd}|S )zâ
|
Return the URLs of all the links on a page together with information
|
about their "rel" attribute, for determining which ones to treat as
|
downloads and which ones to queue for further scraping.
|
c S s, t | \}}}}}}t||t||||fS )zTidy up an URL.)r r r )r, r6 ru rq r r r r. r. r/ Úclean4 s ÿzPage.links.<locals>.cleanr4 Zrel1Zrel2Zrel3Zrel4Zrel5Zrel6r| r} Zurl3c S s dt | d¡ S )Nz%%%2xr )Úordrå )r r. r. r/ Ú<lambda>B ó z Page.links.<locals>.<lambda>c S s | d S )Nr r. )rv r. r. r/ rè F ré T)rB Úreverse) r® Ú_hrefÚfinditerrÒ Ú groupdictr rÉ r Ú _clean_reÚsubr¯ r¸ )r; ræ rY r rÀ Úrelr, r. r. r/ Úlinks- s$
|
ÿÿÿ z
|
Page.linksN)rD rE rF rG ÚreÚcompileÚIÚSÚXrë rã rS rî r rñ r. r. r. r/ râ s ü râ c s e Zd ZdZejdd dd dZd fdd Zd
|
d Zd d Z dd Z
|
e de j ¡Zdd Zdd Zdd Zdd Zdd Ze d¡Zd d Z ZS ) ÚSimpleScrapingLocatorzã
|
A locator which scrapes HTML pages to locate downloads for a distribution.
|
This runs multiple threads to do the I/O; performance is at least as good
|
as pip's PackageFinder, which works in an analogous fashion.
|
c C s t jttd ¡ S )N)Úfileobj)ÚgzipÚGzipFiler rÀ rØ ©Úbr. r. r/ rè T ré z SimpleScrapingLocator.<lambda>c C s | S r^ r. rû r. r. r/ rè U ré )Údeflaterù ÚnoneNé
|
c sl t t| jf | t|| _|| _i | _t | _t
|
¡ | _ t | _ d| _ || _t ¡ | _t ¡ | _d| _dS )a¤
|
Initialise an instance.
|
:param url: The root URL to use for scraping.
|
:param timeout: The timeout, in seconds, to be applied to requests.
|
This defaults to ``None`` (no timeout specified).
|
:param num_workers: The number of worker threads you want to do I/O,
|
This defaults to 10.
|
:param kwargs: Passed to the superclass.
|
FN)rÈ r÷ rS r rÉ r( Ú _page_cacher® Ú_seenr
|
rQ Ú _to_fetchÚ
|
_bad_hostsÚskip_externalsÚ num_workersÚ threadingÚRLockÚ_lockÚ_gplockÚplatform_check)r; r, r( r rË rÌ r. r/ rS X s
|
|
|
|
|
z SimpleScrapingLocator.__init__c C sF g | _ t| jD ]0}tj| jd}| d¡ | ¡ | j |¡ qdS )z¾
|
Threads are created only when get_project is called, and terminate
|
before it returns. They are there primarily to parallelise I/O (i.e.
|
fetching web pages).
|
)ÚtargetTN) Ú_threadsÚranger r ÚThreadÚ_fetchÚ setDaemonÚstartrV )r; Úirv r. r. r/ Ú_prepare_threadss s
|
z&SimpleScrapingLocator._prepare_threadsc C s6 | j D ]}| j d¡ q| j D ] }| ¡ q g | _ dS )zu
|
Tell all the threads to terminate (by sending a sentinel value) and
|
wait for them to do so.
|
N)r r rÝ r )r; rv r. r. r/ Ú _wait_threads s
|
|
|
|
z#SimpleScrapingLocator._wait_threadsc C s i i d}| j x || _|| _t| jdt| }| j ¡ | j ¡ | ¡ z&t d|¡ | j |¡ | j ¡ W 5 |
|
¡ X | `W 5 Q R X |S )Nr´ z%s/z Queueing %s)r rY r r rÉ r r r_ r r r rz r{ r rÝ r )r; ri rY r, r. r. r/ rj s
|
|
|
|
z"SimpleScrapingLocator._get_projectz<\b(linux_(i\d86|x86_64|arm\w+)|win(32|_amd64)|macosx_?\d+)\bc C s | j |¡S )zD
|
Does an URL refer to a platform-specific download?
|
)Úplatform_dependenträ )r; r, r. r. r/ Ú_is_platform_dependent¢ s z,SimpleScrapingLocator._is_platform_dependentc C sZ | j r| |¡rd}n| || j¡}t d||¡ |rV| j | | j|¡ W 5 Q R X |S )a%
|
See if an URL is a suitable download for a project.
|
|
If it is, register information in the result dictionary (for
|
_get_project) about the specific version it's for.
|
|
Note that the return value isn't actually used other than as a boolean
|
value.
|
Nzprocess_download: %s -> %s) r
|
r r¢ r rz r{ r r³ rY )r; r, r¦ r. r. r/ Ú_process_download¨ s
|
z'SimpleScrapingLocator._process_downloadc
|
C sÄ t |\}}}}}}| | j| j | j ¡r2d}n|| jrJ| | j¡sJd}nd| | j¡s\d}nR|dkrjd}nD|dkrxd}n6| |¡rd}n&| dd¡d } |
|
¡ dkrªd}nd}t d ||||¡ |S )
|
z
|
Determine whether a link URL from a referring page and with a
|
particular "rel" attribute should be queued for scraping.
|
F)ZhomepageÚdownload)Úhttprn Úftpú:r r Ú localhostTz#should_queue: %s (%s) from %s -> %s) r rr rà rÄ rÅ r r rÉ r Úsplitr rz r{ )
|
r; ÚlinkZreferrerrð r6 ru rq Ú_rY Úhostr. r. r/ Ú _should_queue¼ s0 ÿ
|
|
ÿz#SimpleScrapingLocator._should_queuec
|
C sð | j ¡ }zÌz|r| |¡}|dkr,W W ¢®q |jD ]j\}}|| jkr2zB| j |¡ | |¡s| |||¡rt
|
d||¡ | j |¡ W q2 t k
|
r Y q2X q2W n2 t k
|
rÒ } z| j t|¡ W 5 d}~X Y nX W 5 | j ¡ X |s qìq dS )z×
|
Get a URL to fetch from the work queue, get the HTML page, examine its
|
links for download candidates and candidates for further scraping.
|
|
This is a handy method to run in a thread.
|
NzQueueing %s from %s)r rU rX Úget_pagerñ r r¯ r r! rz r{ rÝ r r rR r )r; r, Úpager rð rZ r. r. r/ r Ù s,
|
|
|
|
ÿ & z SimpleScrapingLocator._fetchc C sX t |\}}}}}}|dkr:tj t|¡r:tt|d}|| jkr`| j| }t d||¡ nô|
|
dd¡d }d}|| j krt d||¡ nÄt |d d
|
id }z¤zât d |¡ | j j|| jd } t d|¡ | ¡ }
|
|
|
dd¡} t | ¡r| ¡ } | ¡ } |
|
d¡}|r"| j| }|| } d}t | ¡}|r@| d¡}z| |¡} W n tk
|
rn | d¡} Y nX t | | }|| j| < W n¼ t k
|
rÄ } z |j dkr´t d||¡ W 5 d}~X Y n t k
|
r } z0t d||¡ | j! | j "|¡ W 5 Q R X W 5 d}~X Y n2 t#k
|
rB } zt d||¡ W 5 d}~X Y nX W 5 || j|< X |S )a
|
Get the HTML for an URL, possibly from an in-memory cache.
|
|
XXX TODO Note: this cache is never actually cleared. It's assumed that
|
the data won't get stale over the lifetime of a locator instance (not
|
necessarily true for the default_locator).
|
Úfilez
|
index.htmlzReturning %s from cache: %sr r r Nz Skipping %s due to bad host %szAccept-encodingÚidentity)r@ z Fetching %sr' z
|
Fetched %sz Content-Typer4 zContent-Encodingzutf-8zlatin-1i zFetch failed: %s: %s)$r Úosrq Úisdirr r r r rz r{ r r r rO r× r( r¦ rU ÚHTML_CONTENT_TYPEr ÚgeturlrØ ÚdecodersÚCHARSETrä rå rÙ Ú UnicodeErrorrâ r r> rÞ r r r¯ r )r; r, r6 ru rq r rY r r< rß r@ Ú content_typeZ final_urlrÒ ÚencodingÚdecoderr rZ r. r. r/ r" ú sZ
|
|
|
|
|
|
|
|
&$ z SimpleScrapingLocator.get_pagez<a href=[^>]*>([^<]+)<c C sL t }| | j¡}|s$td| j | j |j¡D ]}| | d¡¡ q2|S )rk zUnable to get %sr ) r® r" rÉ r Ú _distname_rerì rÒ r¯ rå )r; rY r# r r. r. r/ rl 7 s z,SimpleScrapingLocator.get_distribution_names)Nrÿ )rD rE rF rG ÚzlibÚ
|
decompressr* rS r r rj rò ró rô r r r r! r r" r0 rl rÔ r. r. rÌ r/ r÷ J s& ý ÿ !;
|
r÷ c s8 e Zd ZdZ fddZdd Zdd Zdd Z ZS )
|
ÚDirectoryLocatorz?
|
This class locates distributions in a directory tree.
|
c sN | dd¡| _tt| jf | tj |¡}tj |¡sDt d| || _
|
dS )a
|
Initialise an instance.
|
:param path: The root of the directory tree to search.
|
:param kwargs: Passed to the superclass constructor,
|
except for:
|
* recursive - if True (the default), subdirectories are
|
recursed into. If False, only the top-level directory
|
is searched,
|
Ú recursiveTzNot a directory: %rN) rª r4 rÈ r3 rS r& rq Úabspathr' r Úbase_dir)r; rq rË rÌ r. r/ rS H s
|
zDirectoryLocator.__init__c C s | | j¡S )zá
|
Should a filename be considered as a candidate for a distribution
|
archive? As well as the filename, the directory which contains it
|
is provided, though not used by the current implementation.
|
)rr rs )r; r Úparentr. r. r/ Úshould_includeY s z DirectoryLocator.should_includec C s i i d}t | j¡D ]t\}}}|D ]Z}| ||¡r$t j ||¡}tddtt j |¡dddf}| ||¡}|r$|
|
||¡ q$| j s qq|S )Nr´ r$ r4 ) r& Úwalkr6 r8 rq r r r r5 r¢ r³ r4 ) r; ri rY ÚrootÚdirsÚfilesÚfnr, r¦ r. r. r/ rj a s"
|
þ z DirectoryLocator._get_projectc C s t }t | j¡D ]v\}}}|D ]\}| ||¡r tj ||¡}tddttj |¡dddf}|
|
|d¡}|r | |d ¡ q | j s qq|S )rk r$ r4 Nri ) r® r& r9 r6 r8 rq r r r r5 r¢ r¯ r4 )r; rY r: r; r< r= r, r¦ r. r. r/ rl q s" þ z'DirectoryLocator.get_distribution_names) rD rE rF rG rS r8 rj rl rÔ r. r. rÌ r/ r3 C s
|
r3 c @ s e Zd ZdZdd Zdd ZdS )Ú JSONLocatora
|
This locator uses special extended metadata (not available on PyPI) and is
|
the basis of performant dependency resolution in distlib. Other locators
|
require archive downloads before dependencies can be determined! As you
|
might imagine, that can be slow.
|
c C s t ddS rÖ rg r\ r. r. r/ rl s z"JSONLocator.get_distribution_namesc C sÚ i i d}t |}|rÖ| dg ¡D ]²}|d dks"|d dkr@q"t|d |d | d d
|
¡| jd }|j}|d |_d |kr|d rd|d f|_| di ¡|_| di ¡|_|||j < |d
|
|j t ¡ |d ¡ q"|S )Nr´ r< ÚptypeÚsdistZ pyversionÚsourceri r rÑ zPlaceholder for summary)rÑ r6 r, r r¥ Ú requirementsÚexportsr© ) r rU r r6 r« r¬ r Z dependenciesrC r r r® r¯ )r; ri rY rÒ r¦ r± r² r. r. r/ rj s*
|
ÿý
|
|
zJSONLocator._get_projectN)rD rE rF rG rl rj r. r. r. r/ r> s r> c s( e Zd ZdZ fddZdd Z ZS )ÚDistPathLocatorz
|
This locator finds installed distributions in a path. It can be useful for
|
adding to an :class:`AggregatingLocator`.
|
c s* t t| jf | t|ts t|| _dS )zs
|
Initialise an instance.
|
|
:param distpath: A :class:`DistributionPath` instance to search.
|
N)rÈ rD rS Ú
|
isinstancer ÚAssertionErrorÚdistpath)r; rG rË rÌ r. r/ rS ¯ s zDistPathLocator.__init__c C sP | j |¡}|d kr i i d}n,|j|d|jt|jgid|jtd gii}|S )Nr´ r© r£ )rG Úget_distributionr r® r¬ )r; ri r± rY r. r. r/ rj ¹ s ýz DistPathLocator._get_project)rD rE rF rG rS rj rÔ r. r. rÌ r/ rD ª s
|
rD c sR e Zd ZdZ fddZ fddZdd Zeej j
|
eZ dd Z d
|
d Z Z S ) ÚAggregatingLocatorzI
|
This class allows you to chain and/or merge a list of locators.
|
c s* | dd¡| _|| _tt| jf | dS )aÏ
|
Initialise an instance.
|
|
:param locators: The list of locators to search.
|
:param kwargs: Passed to the superclass constructor,
|
except for:
|
* merge - if False (the default), the first successful
|
search from any of the locators is returned. If True,
|
the results from all locators are merged (this can be
|
slow).
|
ÚmergeFN)rª rJ ÚlocatorsrÈ rI rS )r; rK rË rÌ r. r/ rS Ê s zAggregatingLocator.__init__c s& t t| ¡ | jD ] }| ¡ qd S r^ )rÈ rI r` rK ©r; r° rÌ r. r/ r` Ú s
|
z AggregatingLocator.clear_cachec C s || _ | jD ]
|
}||_q d S r^ )rb rK r6 )r; rd r° r. r. r/ re ß s
|
z AggregatingLocator._set_schemec C sì i }| j D ]Ü}| |¡}|r
|
| jr¨| di ¡}| di ¡}| |¡ | d¡}|r|r| ¡ D ]*\}} ||kr|| | O < q^| ||< q^| d¡}
|
|ræ|
|
ræ|
|
|¡ q
|
| jd kr¸d} n"d} |D ]}| j |¡rÀd} qÚqÀ| r
|
|} qèq
|
|S )Nr© r£ TF)rK rm rJ rU ÚupdaterÜ rP r ) r; ri rY r° rÀ r< r£ Zdfr¿ r ÚddÚfoundr. r. r/ rj æ s8
|
|
|
|
|
|
|
z AggregatingLocator._get_projectc C s< t }| jD ]*}z|| ¡ O }W q tk
|
r4 Y q X q |S rÎ )r® rK rl rh )r; rY r° r. r. r/ rl s
|
z)AggregatingLocator.get_distribution_names)rD rE rF rG rS r` re rÆ rK r6 Úfgetrj rl rÔ r. r. rÌ r/ rI Æ s ,rI zhttps://pypi.org/simple/r& r' Úlegacyr¨ z1(?P<name>[\w-]+)\s*\(\s*(==\s*)?(?P<ver>[^)]+)\)$c @ sL e Zd ZdZdddZdd Zdd Zd d
|
Zd d Zd d Z dddZ
|
dS )ÚDependencyFinderz0
|
Locate dependencies for distributions.
|
Nc C s |pt | _t| jj| _dS )zf
|
Initialise an instance, using the specified locator
|
to locate distributions.
|
N)Údefault_locatorr° r" r6 rL r. r. r/ rS 1 s
|
zDependencyFinder.__init__c C sr t d|¡ |j}|| j|< || j||jf< |jD ]:}t|\}}t d|||¡ | j |t
|
¡ ||f¡ q2dS )z¨
|
Add a distribution to the finder. This will update internal information
|
about who provides what.
|
:param dist: The distribution to add.
|
zadding distribution %szAdd to provided: %s, %s, %sN) rz r{ rB Ú dists_by_nameÚdistsr Úprovidesr Úprovidedr r® r¯ )r; r± ri Úpr r. r. r/ Úadd_distribution9 s
|
|
z!DependencyFinder.add_distributionc C sx t d|¡ |j}| j|= | j||jf= |jD ]D}t|\}}t d|||¡ | j| }| ||f¡ |s.| j|= q.dS )z°
|
Remove a distribution from the finder. This will update internal
|
information about who provides what.
|
:param dist: The distribution to remove.
|
zremoving distribution %sz Remove from provided: %s, %s, %sN)
|
rz r{ rB rT rU r rV r rW Úremove)r; r± ri rX r Úsr. r. r/ Úremove_distributionH s
|
|
z$DependencyFinder.remove_distributionc C sB z| j |¡}W n, tk
|
r< | ¡ d }| j |¡}Y nX |S )zÞ
|
Get a version matcher for a requirement.
|
:param reqt: The requirement
|
:type reqt: str
|
:return: A version matcher (an instance of
|
:class:`distlib.version.Matcher`).
|
r )r6 rP r# r )r; ÚreqtrP ri r. r. r/ Ú get_matcherZ s z DependencyFinder.get_matcherc C st | |¡}|j}t }| j}||krp|| D ]B\}}z| |¡}W n tk
|
rZ d}Y nX |r,| |¡ qpq,|S )zÓ
|
Find the distributions which can fulfill a requirement.
|
|
:param reqt: The requirement.
|
:type reqt: str
|
:return: A set of distribution which can fulfill the requirement.
|
F)r^ rB r® rW r r# r¯ ) r; r] rP ri rY rW r Úproviderr r. r. r/ Úfind_providersj s
|
|
|
z DependencyFinder.find_providersc C s | j | }t }|D ]$}| |¡}| |j¡s| |¡ q|rZ| d||t|f¡ d}n@| |¡ | j |= |D ]}| j |t ¡ |¡ qp| |¡ d}|S )a
|
Attempt to replace one provider with another. This is typically used
|
when resolving dependencies from multiple sources, e.g. A requires
|
(B >= 1.0) while C requires (B >= 1.1).
|
|
For successful replacement, ``provider`` must meet all the requirements
|
which ``other`` fulfills.
|
|
:param provider: The provider we are trying to replace with.
|
:param other: The provider we're trying to replace.
|
:param problems: If False is returned, this will contain what
|
problems prevented replacement. This is currently
|
a tuple of the literal string 'cantreplace',
|
``provider``, ``other`` and the set of requirements
|
that ``provider`` couldn't fulfill.
|
:return: True if we can replace ``other`` with ``provider``, else
|
False.
|
Z cantreplaceFT)
|
Úreqtsr® r^ r r r¯ Ú frozensetr\ r rY ) r; r_ ÚotherÚproblemsZrlistÚ unmatchedr[ rP rY r. r. r/ Útry_to_replace s$
|
|
|
ÿ
|
|
z DependencyFinder.try_to_replaceFc C sò i | _ i | _i | _i | _t|p g }d|krH| d¡ |tdddgO }t|trh| }}t d|¡ n4| j
|
j ||d }}|dkrt d| t d |¡ d
|
|_ t }t|g}t|g}|r¤| ¡ }|j} | | jkræ| |¡ n"| j| }
|
|
|
|kr| ||
|
|¡ |j|jB } |j} t } |r`||kr`d D ]*}d | }||kr4| t|d | O } q4| | B | B }|D ].}| |¡}|sFt d|¡ | j
|
j ||d}|dkrÂ|sÂ| j
|
j |d
|
d}|dkrèt d|¡ | d|f¡ n^|j|j }}||f| jkr| |¡ | |¡ || krF||krF| |¡ t d|j¡ |D ]R}|j} | | jkrx| j |t ¡ |¡ n"| j| }
|
|
|
|krJ| ||
|
|¡ qJqpq¼t| j ¡ }|D ]&}||k|_ |j r¶t d|j¡ q¶t d|¡ ||fS )a
|
Find a distribution and all distributions it depends on.
|
|
:param requirement: The requirement specifying the distribution to
|
find, or a Distribution instance.
|
:param meta_extras: A list of meta extras such as :test:, :build: and
|
so on.
|
:param prereleases: If ``True``, allow pre-release versions to be
|
returned - otherwise, don't return prereleases
|
unless they're all that's available.
|
|
Return a set of :class:`Distribution` instances and a set of
|
problems.
|
|
The distributions returned should be such that they have the
|
:attr:`required` attribute set to ``True`` if they were
|
from the ``requirement`` passed to ``find()``, and they have the
|
:attr:`build_time_dependency` attribute set to ``True`` unless they
|
are post-installation dependencies of the ``requirement``.
|
|
The problems should be a tuple consisting of the string
|
``'unsatisfied'`` and the requirement which couldn't be satisfied
|
by any distribution known to the locator.
|
z:*:z:test:z:build:z:dev:zpassed %s as requirement)r» NzUnable to locate %rz
|
located %sT)ÚtestÚbuildÚdevz:%s:z %s_requireszNo providers found for %rzCannot satisfy %rZ unsatisfiedzAdding %s to install_distsz#%s is a build-time dependency only.zfind done for %s) rW rU rT ra r® rZ rE r rz r{ r° r r Ú requestedrª rB rY rf Z run_requiresZ meta_requiresZbuild_requiresÚgetattrr` r¯ r Zname_and_versionr ÚvaluesZbuild_time_dependency)r; rµ Z meta_extrasr» r± rá rd ÚtodoZ install_distsri rc ZireqtsZsreqtsZereqtsrB rZ Z all_reqtsr¼ Z providersr_ Únr rX rU r. r. r/ Úfindª s
|
|
ÿ
|
|
|
|
|
|
|
|
|
|
|
|
|
ÿ
|
|
|
ÿ zDependencyFinder.find)N)NF) rD rE rF rG rS rY r\ r^ r` rf ro r. r. r. r/ rR , s
|
(rR )N)Orù Úior rÚ Úloggingr& ro rò r Ú ImportErrorZdummy_threadingr1 r4 r Úcompatr r r r r r
|
r r r r r r9 r r r r Zdatabaser r r r« r r Úutilr r r r r r r r r! r r" r# r r$ r% Ú getLoggerrD rz ró r rô r+ r( r* r0 r1 ÚobjectrK rÇ rÕ râ r÷ r3 r> rD rI rS r ÚNAME_VERSION_RErR r. r. r. r/ Ú<module> s^ D,
|
|
|
|
G0E: zA& [ÿü
|
|
