U
|
P ±d3[ ã @ s~ d Z ddlZddlZddlmZ eZdd ZG dd dZe Z dej
|
ddfd d
|
Zed G d d d Z G dd de Z dS )a" A file interface for handling local and remote data files.
|
|
The goal of datasource is to abstract some of the file system operations
|
when dealing with data files so the researcher doesn't have to know all the
|
low-level details. Through datasource, a researcher can obtain and use a
|
file with one function call, regardless of location of the file.
|
|
DataSource is meant to augment standard python libraries, not replace them.
|
It should work seamlessly with standard file IO operations and the os
|
module.
|
|
DataSource files can originate locally or remotely:
|
|
- local files : '/home/guido/src/local/data.txt'
|
- URLs (http, ftp, ...) : 'http://www.scipy.org/not/real/data.txt'
|
|
DataSource files can also be compressed or uncompressed. Currently only
|
gzip, bz2 and xz are supported.
|
|
Example::
|
|
>>> # Create a DataSource, use os.curdir (default) for local storage.
|
>>> from numpy import DataSource
|
>>> ds = DataSource()
|
>>>
|
>>> # Open a remote file.
|
>>> # DataSource downloads the file, stores it locally in:
|
>>> # './www.google.com/index.html'
|
>>> # opens the file and returns a file object.
|
>>> fp = ds.open('http://www.google.com/') # doctest: +SKIP
|
>>>
|
>>> # Use the file as you normally would
|
>>> fp.read() # doctest: +SKIP
|
>>> fp.close() # doctest: +SKIP
|
|
é N)Ú
|
set_modulec C sD d| kr d| kr@t d| f n |dk r0t d|dk r@t ddS )zàCheck mode and that encoding and newline are compatible.
|
|
Parameters
|
----------
|
mode : str
|
File open mode.
|
encoding : str
|
File encoding.
|
newline : str
|
Newline for text files.
|
|
ÚtÚbzInvalid mode: %rNz0Argument 'encoding' not supported in binary modez/Argument 'newline' not supported in binary mode)Ú
|
ValueError©ÚmodeÚencodingÚnewline© r
|
úLd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\numpy/lib/_datasource.pyÚ _check_mode. s r c @ s0 e Zd ZdZdd Zdd Zdd Zdd Zd
|
S ) Ú _FileOpenersa
|
|
Container for different methods to open (un-)compressed files.
|
|
`_FileOpeners` contains a dictionary that holds one method for each
|
supported file format. Attribute lookup is implemented in such a way
|
that an instance of `_FileOpeners` itself can be indexed with the keys
|
of that dictionary. Currently uncompressed files as well as files
|
compressed with ``gzip``, ``bz2`` or ``xz`` compression are supported.
|
|
Notes
|
-----
|
`_file_openers`, an instance of `_FileOpeners`, is made available for
|
use in the `_datasource` module.
|
|
Examples
|
--------
|
>>> import gzip
|
>>> np.lib._datasource._file_openers.keys()
|
[None, '.bz2', '.gz', '.xz', '.lzma']
|
>>> np.lib._datasource._file_openers['.gz'] is gzip.open
|
True
|
|
c C s d| _ d tji| _d S )NF)Ú_loadedÚioÚopenÚ _file_openers©Úselfr
|
r
|
r Ú__init__c s z_FileOpeners.__init__c C s® | j r
|
d S zdd l}|j| jd< W n tk
|
r6 Y nX zdd l}|j| jd< W n tk
|
rd Y nX z$dd l}|j| jd< |j| jd< W n ttfk
|
r¢ Y nX d| _ d S )Nr z.bz2z.gzz.xzz.lzmaT)r Úbz2r r Ú ImportErrorÚgzipÚlzmaÚAttributeError)r r r r r
|
r
|
r Ú_loadg s& z_FileOpeners._loadc C s | ¡ t| j ¡ S )a[
|
Return the keys of currently supported file openers.
|
|
Parameters
|
----------
|
None
|
|
Returns
|
-------
|
keys : list
|
The keys are None for uncompressed files and the file extension
|
strings (i.e. ``'.gz'``, ``'.xz'``) for supported compression
|
methods.
|
|
)r Úlistr Úkeysr r
|
r
|
r r s z_FileOpeners.keysc C s | ¡ | j| S ©N)r r )r Úkeyr
|
r
|
r Ú __getitem__ s z_FileOpeners.__getitem__N)Ú__name__Ú
|
__module__Ú __qualname__Ú__doc__r r r r r
|
r
|
r
|
r r J s
|
r Úrc C s t |}|j| |||dS )a
|
Open `path` with `mode` and return the file object.
|
|
If ``path`` is an URL, it will be downloaded, stored in the
|
`DataSource` `destpath` directory and opened from there.
|
|
Parameters
|
----------
|
path : str
|
Local file path or URL to open.
|
mode : str, optional
|
Mode to open `path`. Mode 'r' for reading, 'w' for writing, 'a' to
|
append. Available modes depend on the type of object specified by
|
path. Default is 'r'.
|
destpath : str, optional
|
Path to the directory where the source file gets downloaded to for
|
use. If `destpath` is None, a temporary directory will be created.
|
The default path is the current directory.
|
encoding : {None, str}, optional
|
Open text file with given encoding. The default encoding will be
|
what `io.open` uses.
|
newline : {None, str}, optional
|
Newline to use when reading text file.
|
|
Returns
|
-------
|
out : file object
|
The opened file.
|
|
Notes
|
-----
|
This is a convenience function that instantiates a `DataSource` and
|
returns the file object from ``DataSource.open(path)``.
|
|
©r r )Ú
|
DataSourcer )Úpathr Údestpathr r Zdsr
|
r
|
r r s %r Únumpyc @ s e Zd ZdZejfddZdd Zdd Zdd Z d
|
d Z
|
d d Z dd Z dd Z dd Zdd Zdd Zdd Zd d d ZdS ) r& aû
|
DataSource(destpath='.')
|
|
A generic data source file (file, http, ftp, ...).
|
|
DataSources can be local files or remote files/URLs. The files may
|
also be compressed or uncompressed. DataSource hides some of the
|
low-level details of downloading the file, allowing you to simply pass
|
in a valid file path (or URL) and obtain a file object.
|
|
Parameters
|
----------
|
destpath : str or None, optional
|
Path to the directory where the source file gets downloaded to for
|
use. If `destpath` is None, a temporary directory will be created.
|
The default path is the current directory.
|
|
Notes
|
-----
|
URLs require a scheme string (``http://``) to be used, without it they
|
will fail::
|
|
>>> repos = np.DataSource()
|
>>> repos.exists('www.google.com/index.html')
|
False
|
>>> repos.exists('http://www.google.com/index.html')
|
True
|
|
Temporary directories are deleted when the DataSource is deleted.
|
|
Examples
|
--------
|
::
|
|
>>> ds = np.DataSource('/home/guido')
|
>>> urlname = 'http://www.google.com/'
|
>>> gfile = ds.open('http://www.google.com/')
|
>>> ds.abspath(urlname)
|
'/home/guido/www.google.com/index.html'
|
|
>>> ds = np.DataSource(None) # use with temporary file
|
>>> ds.open('/home/guido/foobar.txt')
|
<open file '/home/guido.foobar.txt', mode 'r' at 0x91d4430>
|
>>> ds.abspath('/home/guido/foobar.txt')
|
'/tmp/.../home/guido/foobar.txt'
|
|
c C s6 |rt j |¡| _d| _nddl}| ¡ | _d| _dS )z2Create a DataSource with a local path at destpath.Fr NT)Úosr' ÚabspathÚ _destpathÚ
|
_istmpdestÚtempfileÚmkdtemp)r r( r. r
|
r
|
r r ö s
|
zDataSource.__init__c C s( t | dr$| jr$dd l}| | j¡ d S )Nr- r )Úhasattrr- ÚshutilÚrmtreer, )r r1 r
|
r
|
r Ú__del__ s zDataSource.__del__c C s t j |¡\}}|t ¡ kS )zNTest if the filename is a zip file by looking at the file extension.
|
|
)r* r' Úsplitextr r )r ÚfilenameÚfnameÚextr
|
r
|
r Ú_iszip s zDataSource._iszipc C s d}|D ]}||kr dS qdS )z4Test if the given mode will open a file for writing.)Úwú+TFr
|
)r r Z _writemodesÚcr
|
r
|
r Ú _iswritemode s
|
zDataSource._iswritemodec C s" | |¡rtj |¡S |dfS dS )zSplit zip extension from filename and return filename.
|
|
Returns
|
-------
|
base, zip_ext : {tuple}
|
|
N)r8 r* r' r4 )r r5 r
|
r
|
r Ú _splitzipext s
|
zDataSource._splitzipextc C s4 |g}| |¡s0t ¡ D ]}|r| || ¡ q|S )z9Return a tuple containing compressed filename variations.)r8 r r Úappend)r r5 ÚnamesZzipextr
|
r
|
r Ú_possible_names& s
|
zDataSource._possible_namesc C s, ddl m} ||\}}}}}}t|o(|S )z=Test if path is a net location. Tests the scheme and netloc.r ©Úurlparse)Ú urllib.parserB Úbool) r r' rB ÚschemeÚnetlocÚupathÚuparamsÚuqueryÚufragr
|
r
|
r Ú_isurl/ s zDataSource._isurlc
|
C s ddl }ddlm} | |¡}tj tj |¡¡sDt tj |¡¡ | |¡r||(}t
|
|d}| ||¡ W 5 Q R X W 5 Q R X n | ||¡ |S )zhCache the file specified by path.
|
|
Creates a copy of the file in the datasource cache.
|
|
r N©ÚurlopenÚwb) r1 Úurllib.requestrM r+ r* r' ÚexistsÚdirnameÚmakedirsrK Ú_openÚ copyfileobjÚcopyfile)r r' r1 rM rG Z openedurlÚfr
|
r
|
r Ú_cache> s
|
|
|
" zDataSource._cachec C s| | |¡s*| |¡}|| | |¡¡7 }n | | |¡¡}|| |¡ }|D ]*}| |¡rL| |¡rn| |¡}| S qLdS )ay Searches for ``path`` and returns full path if found.
|
|
If path is an URL, _findfile will cache a local copy and return the
|
path to the cached file. If path is a local file, _findfile will
|
return a path to that local file.
|
|
The search will include possible compressed versions of the file
|
and return the first occurrence found.
|
|
N)rK r@ r+ rP rW )r r' ÚfilelistÚnamer
|
r
|
r Ú _findfileX s
|
|
|
|
|
|
zDataSource._findfilec
|
C sh ddl m} | | jd¡}t|dkr.|d }||\}}}}}} | |¡}| |¡}tj | j||¡S )aF
|
Return absolute path of file in the DataSource directory.
|
|
If `path` is an URL, then `abspath` will return either the location
|
the file exists locally or the location it would exist when opened
|
using the `open` method.
|
|
Parameters
|
----------
|
path : str
|
Can be a local file or a remote URL.
|
|
Returns
|
-------
|
out : str
|
Complete path, including the `DataSource` destination directory.
|
|
Notes
|
-----
|
The functionality is based on `os.path.abspath`.
|
|
r rA é é ) rC rB Úsplitr, ÚlenÚ_sanitize_relative_pathr* r' Újoin)
|
r r' rB Ú splitpathrE rF rG rH rI rJ r
|
r
|
r r+ w s
|
|
|
zDataSource.abspathc C sV d}t j |¡}||krR|}| t j¡ d¡}| t j¡ d¡}t j |¡\}}q|S )zvReturn a sanitised relative path for which
|
os.path.abspath(os.path.join(base, path)).startswith(base)
|
Nú/z..)r* r' ÚnormpathÚlstripÚsepÚpardirÚ
|
splitdrive)r r' ÚlastÚdriver
|
r
|
r r_ ¡ s z"DataSource._sanitize_relative_pathc C s t j |¡rdS ddlm} ddlm} | |¡}t j |¡rBdS | |¡r|z||}| ¡ ~W dS |k
|
rz Y dS X dS )a3
|
Test if path exists.
|
|
Test if `path` exists as (and in this order):
|
|
- a local file.
|
- a remote URL that has been downloaded and stored locally in the
|
`DataSource` directory.
|
- a remote URL that has not been downloaded, but is valid and
|
accessible.
|
|
Parameters
|
----------
|
path : str
|
Can be a local file or a remote URL.
|
|
Returns
|
-------
|
out : bool
|
True if `path` exists.
|
|
Notes
|
-----
|
When `path` is an URL, `exists` will return True if it's either
|
stored locally in the `DataSource` directory, or is a valid remote
|
URL. `DataSource` does not discriminate between the two, the file
|
is accessible if it exists in either location.
|
|
Tr rL )ÚURLErrorF)
|
r* r' rP rO rM Ú urllib.errorrj r+ rK Úclose)r r' rM rj rG Znetfiler
|
r
|
r rP ¯ s
|
|
zDataSource.existsr$ Nc C sr | |¡r | |¡r td| |¡}|r`| |¡\}}|dkrL| dd¡ t| ||||dS t| ddS )aD
|
Open and return file-like object.
|
|
If `path` is an URL, it will be downloaded, stored in the
|
`DataSource` directory and opened from there.
|
|
Parameters
|
----------
|
path : str
|
Local file path or URL to open.
|
mode : {'r', 'w', 'a'}, optional
|
Mode to open `path`. Mode 'r' for reading, 'w' for writing,
|
'a' to append. Available modes depend on the type of object
|
specified by `path`. Default is 'r'.
|
encoding : {None, str}, optional
|
Open text file with given encoding. The default encoding will be
|
what `io.open` uses.
|
newline : {None, str}, optional
|
Newline to use when reading text file.
|
|
Returns
|
-------
|
out : file object
|
File object.
|
|
zURLs are not writeabler r: Ú r z not found.N)rK r< r rZ r= Úreplacer ÚFileNotFoundError)r r' r r r ÚfoundZ_fnamer7 r
|
r
|
r r ç s "
|
|
ÿzDataSource.open)r$ NN)r r! r" r# r* Úcurdirr r3 r8 r< r= r@ rK rW rZ r+ r_ rP r r
|
r
|
r
|
r r& Ä s 0
|
|
*8r& c @ sX e Zd ZdZejfddZdd Zdd Zdd Z d
|
d Z
|
d d Z dddZ dd Z dS )Ú
|
Repositorya
|
Repository(baseurl, destpath='.')
|
|
A data repository where multiple DataSource's share a base
|
URL/directory.
|
|
`Repository` extends `DataSource` by prepending a base URL (or
|
directory) to all the files it handles. Use `Repository` when you will
|
be working with multiple files from one base URL. Initialize
|
`Repository` with the base URL, then refer to each file by its filename
|
only.
|
|
Parameters
|
----------
|
baseurl : str
|
Path to the local directory or remote location that contains the
|
data files.
|
destpath : str or None, optional
|
Path to the directory where the source file gets downloaded to for
|
use. If `destpath` is None, a temporary directory will be created.
|
The default path is the current directory.
|
|
Examples
|
--------
|
To analyze all files in the repository, do something like this
|
(note: this is not self-contained code)::
|
|
>>> repos = np.lib._datasource.Repository('/home/user/data/dir/')
|
>>> for filename in filelist:
|
... fp = repos.open(filename)
|
... fp.analyze()
|
... fp.close()
|
|
Similarly you could use a URL for a repository::
|
|
>>> repos = np.lib._datasource.Repository('http://www.xyz.edu/data')
|
|
c C s t j| |d || _dS )z>Create a Repository with a shared url or directory of baseurl.)r( N)r& r Ú_baseurl)r Zbaseurlr( r
|
r
|
r r @ s zRepository.__init__c C s t | ¡ d S r )r& r3 r r
|
r
|
r r3 E s zRepository.__del__c C s4 | | jd¡}t|dkr,tj | j|¡}n|}|S )z>Return complete path for path. Prepends baseurl if necessary.r[ r\ )r] rs r^ r* r' r` )r r' ra Úresultr
|
r
|
r Ú _fullpathH s
|
zRepository._fullpathc C s t | | |¡¡S )z8Extend DataSource method to prepend baseurl to ``path``.)r& rZ ru ©r r' r
|
r
|
r rZ Q s zRepository._findfilec C s t | | |¡¡S )ak
|
Return absolute path of file in the Repository directory.
|
|
If `path` is an URL, then `abspath` will return either the location
|
the file exists locally or the location it would exist when opened
|
using the `open` method.
|
|
Parameters
|
----------
|
path : str
|
Can be a local file or a remote URL. This may, but does not
|
have to, include the `baseurl` with which the `Repository` was
|
initialized.
|
|
Returns
|
-------
|
out : str
|
Complete path, including the `DataSource` destination directory.
|
|
)r& r+ ru rv r
|
r
|
r r+ U s zRepository.abspathc C s t | | |¡¡S )aÕ
|
Test if path exists prepending Repository base URL to path.
|
|
Test if `path` exists as (and in this order):
|
|
- a local file.
|
- a remote URL that has been downloaded and stored locally in the
|
`DataSource` directory.
|
- a remote URL that has not been downloaded, but is valid and
|
accessible.
|
|
Parameters
|
----------
|
path : str
|
Can be a local file or a remote URL. This may, but does not
|
have to, include the `baseurl` with which the `Repository` was
|
initialized.
|
|
Returns
|
-------
|
out : bool
|
True if `path` exists.
|
|
Notes
|
-----
|
When `path` is an URL, `exists` will return True if it's either
|
stored locally in the `DataSource` directory, or is a valid remote
|
URL. `DataSource` does not discriminate between the two, the file
|
is accessible if it exists in either location.
|
|
)r& rP ru rv r
|
r
|
r rP l s zRepository.existsr$ Nc C s t j| | |¡|||dS )aÜ
|
Open and return file-like object prepending Repository base URL.
|
|
If `path` is an URL, it will be downloaded, stored in the
|
DataSource directory and opened from there.
|
|
Parameters
|
----------
|
path : str
|
Local file path or URL to open. This may, but does not have to,
|
include the `baseurl` with which the `Repository` was
|
initialized.
|
mode : {'r', 'w', 'a'}, optional
|
Mode to open `path`. Mode 'r' for reading, 'w' for writing,
|
'a' to append. Available modes depend on the type of object
|
specified by `path`. Default is 'r'.
|
encoding : {None, str}, optional
|
Open text file with given encoding. The default encoding will be
|
what `io.open` uses.
|
newline : {None, str}, optional
|
Newline to use when reading text file.
|
|
Returns
|
-------
|
out : file object
|
File object.
|
|
r% )r& r ru )r r' r r r r
|
r
|
r r s ÿzRepository.openc C s& | | j¡rtdn t | j¡S dS )a
|
List files in the source Repository.
|
|
Returns
|
-------
|
files : list of str
|
List of file names (not containing a directory part).
|
|
Notes
|
-----
|
Does not currently work for remote repositories.
|
|
z-Directory listing of URLs, not supported yet.N)rK rs ÚNotImplementedErrorr* Úlistdirr r
|
r
|
r rx ® s
|
ÿzRepository.listdir)r$ NN)r r! r" r# r* rq r r3 ru rZ r+ rP r rx r
|
r
|
r
|
r rr s ' "
|
rr ) r# r* r Znumpy.core.overridesr r rS r r r rq r& rr r
|
r
|
r
|
r Ú<module> s $ O) U
|
