U
|
¸ý°dcC ã @ sv d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlm Z dd lm
|
Z
|
dd
|
lm Z dd lm Z dd lm Z d dlmZ d dlmZ d dlmZ d dlmZ d dlmZ d dlmZ d dlmZ d dlmZ d dlm Z d dl m Z d dl m Z d dl m Z d dl!m"Z" d dl!m#Z# e röd d l$m%Z% d d l$m&Z& d d l$m'Z' d d l(m)Z) d d l(m*Z* d d!lm+Z+ d d"lm,Z, d d#l-m.Z. d d$l/m0Z0 d d%l m1Z1 d d&l m2Z2 d d'l m3Z3 d d(l4m5Z5 d d)l6m7Z7 d d*l8m9Z9 d d+l:m;Z; d,d-gZ<e d.ed/Z=e>Z?G d0d1 d1e"Z@G d2d3 d3e"ZAG d4d- d-e e= ZBG d5d, d,e ZCG d6d7 d7eZDd8d9d:d;d<ZEd=S )>aË Horizontal sharding support.
|
|
Defines a rudimental 'horizontal sharding' system which allows a Session to
|
distribute queries and persistence operations across multiple databases.
|
|
For a usage example, see the :ref:`examples_sharding` example included in
|
the source distribution.
|
|
.. deepalchemy:: The horizontal sharding extension is an advanced feature,
|
involving a complex statement -> database interaction as well as
|
use of semi-public APIs for non-trivial cases. Simpler approaches to
|
refering to multiple database "shards", most commonly using a distinct
|
:class:`_orm.Session` per "shard", should always be considered first
|
before using this more complex and less-production-tested system.
|
|
|
|
é )Ú annotations)ÚAny)ÚCallable)ÚDict)ÚIterable)ÚOptional)ÚTuple)ÚType)Ú TYPE_CHECKING)ÚTypeVar)ÚUnioné )Úevent)Úexc)Úinspect)Úutil)Ú PassiveFlag)ÚOrmExecuteOptionsParameter)Ú ORMOption)ÚMapper)ÚQuery)Ú_BindArguments)Ú_PKIdentityArgument)ÚSession)ÚProtocol)ÚSelf)Ú
|
Connection)ÚEngine)Ú OptionEngine)ÚIteratorResult)ÚResult)ÚLoaderCallableStatus)Ú_O)ÚBulkUDCompileState)Ú QueryContext)Ú_EntityBindKey)Ú _SessionBind)ÚORMExecuteState)Ú InstanceState)Ú
|
Executable)Ú_TP)Ú ClauseElementÚShardedSessionÚ ShardedQueryÚ_T)Úboundc @ s e Zd ZdddddddZdS )Ú ShardChooserúOptional[Mapper[_T]]r úOptional[ClauseElement])ÚmapperÚinstanceÚclauseÚreturnc C s d S ©N© )Úselfr3 r4 r5 r8 r8 úVd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\sqlalchemy/ext/horizontal_shard.pyÚ__call__Q s zShardChooser.__call__N©Ú__name__Ú
|
__module__Ú __qualname__r; r8 r8 r8 r: r0 P s r0 c @ s& e Zd Zdddddddddd Zd
|
S ) ÚIdentityChooserú
|
Mapper[_T]r ú Optional[InstanceState[Any]]r r r ©r3 Ú primary_keyÚlazy_loaded_fromÚexecution_optionsÚbind_argumentsÚkwr6 c K s d S r7 r8 )r9 r3 rD rE rF rG rH r8 r8 r: r; [ s
|
zIdentityChooser.__call__Nr< r8 r8 r8 r: r@ Z s r@ c s: e Zd ZdZdddd fdd Zddd d
|
d Z ZS ) r- a Query class used with :class:`.ShardedSession`.
|
|
.. legacy:: The :class:`.ShardedQuery` is a subclass of the legacy
|
:class:`.Query` class. The :class:`.ShardedSession` now supports
|
2.0 style execution via the :meth:`.ShardedSession.execute` method.
|
|
r ÚNone)ÚargsÚkwargsr6 c s< t j|| t| jts t| jj| _| jj| _d | _d S r7 ) ÚsuperÚ__init__Ú
|
isinstanceÚsessionr, ÚAssertionErrorÚidentity_chooserÚexecute_chooserZ _shard_id)r9 rJ rK ©Ú __class__r8 r: rM q s
|
|
|
zShardedQuery.__init__ÚShardIdentifierr ©Úshard_idr6 c C s | j |dS )aà Return a new query, limited to a single shard ID.
|
|
All subsequent operations with the returned query will
|
be against the single shard regardless of other state.
|
|
The shard_id can be passed for a 2.0 style execution to the
|
bind_arguments dictionary of :meth:`.Session.execute`::
|
|
results = session.execute(
|
stmt,
|
bind_arguments={"shard_id": "my_shard"}
|
)
|
|
)Ú _sa_shard_id)rF )r9 rW r8 r8 r: Ú set_shardy s zShardedQuery.set_shard)r= r> r? Ú__doc__rM rY Ú __classcell__r8 r8 rS r: r- h s c sò e Zd ZU ded< ded< ded< dddefddddd d
|
d d d dddd fddZdejdej dfdddddddddd fd d Z
|
d dddd d!d"Z d4d#dd$dd%d&d'd(Z d5dddd)d d$dd*dd+d,d-d.Z d/d0dd1d2d3Z ZS )6r, r0 Ú shard_chooserr@ rQ z*Callable[[ORMExecuteState], Iterable[Any]]rR N)Ú
|
id_chooserÚ query_chooserzOptional[IdentityChooser]z4Optional[Callable[[ORMExecuteState], Iterable[Any]]]zOptional[Dict[str, Any]]zType[Query[_T]]z<Optional[Callable[[Query[_T], Iterable[_T]], Iterable[Any]]]z/Optional[Callable[[Executable], Iterable[Any]]]r rI ) r\ rQ rR ÚshardsÚ query_clsr] r^ rK r6 c s t jf d|i| tjdtdd |_|rj| t dd¡ ddd d
|
d d d d fdd } | _n|rv|_n
|
t
|
d¡|rÂ|t dd¡ |r¢t
|
d¡dddfdd }
|
|dkrÂ|
|
}|dkrÔt
|
d¡|_ i _ |dk r|D ]} | || ¡ qîdS )aî Construct a ShardedSession.
|
|
:param shard_chooser: A callable which, passed a Mapper, a mapped
|
instance, and possibly a SQL clause, returns a shard ID. This id
|
may be based off of the attributes present within the object, or on
|
some round-robin scheme. If the scheme is based on a selection, it
|
should set whatever state on the instance to mark it in the future as
|
participating in that shard.
|
|
:param identity_chooser: A callable, passed a Mapper and primary key
|
argument, which should return a list of shard ids where this
|
primary key might reside.
|
|
.. versionchanged:: 2.0 The ``identity_chooser`` parameter
|
supersedes the ``id_chooser`` parameter.
|
|
:param execute_chooser: For a given :class:`.ORMExecuteState`,
|
returns the list of shard_ids
|
where the query should be issued. Results from all shards returned
|
will be combined together into a single listing.
|
|
.. versionchanged:: 1.4 The ``execute_chooser`` parameter
|
supersedes the ``query_chooser`` parameter.
|
|
:param shards: A dictionary of string shard names
|
to :class:`~sqlalchemy.engine.Engine` objects.
|
|
r` Zdo_orm_executeT)ÚretvalzLThe ``id_chooser`` parameter is deprecated; please use ``identity_chooser``.z2.0rA r rB r r r rC c s" | ¡}|r| |¡} ||S r7 )ÚqueryZ_set_lazyload_from)r3 rD rE rF rG rH Úq)Ú _id_chooserr9 r8 r: Ú_legacy_identity_chooserË s
|
|
z9ShardedSession.__init__.<locals>._legacy_identity_chooserz*identity_chooser or id_chooser is requiredzNThe ``query_chooser`` parameter is deprecated; please use ``execute_chooser``.z1.4z>Can't pass query_chooser and execute_chooser at the same time.r' z Iterable[Any]©Ú orm_contextr6 c s
|
| j S r7 )Z statement©rg )Ú_query_chooserr8 r: Ú_default_execute_chooserî s z9ShardedSession.__init__.<locals>._default_execute_chooserNz,execute_chooser or query_chooser is required)rL rM r ÚlistenÚexecute_and_instancesr\ r Zwarn_deprecatedrQ r Ú ArgumentErrorrR Ú_ShardedSession__shardsÚ
|
bind_shard) r9 r\ rQ rR r_ r` r] r^ rK re rj ÚkrS )rd ri r9 r: rM sV , ÿý ÿýÿÿ
|
zShardedSession.__init__z
|
Mapper[_O]zUnion[Any, Tuple[Any, ...]]z Optional[Any]r rB r zOptional[_BindArguments]z)Union[Optional[_O], LoaderCallableStatus]) r3 Úprimary_key_identityÚidentity_tokenÚpassiverE rF rG rH r6 c s |dk r&t j||fd|i|} | S | j|||||r>t|ni dD ]0}
|
t j||f|
|
|d|} | dk rF| S qFdS dS )a_ override the default :meth:`.Session._identity_lookup` method so
|
that we search for a given non-token primary key identity across all
|
possible identity tokens (e.g. shard ids).
|
|
.. versionchanged:: 1.4 Moved :meth:`.Session._identity_lookup` from
|
the :class:`_query.Query` object to the :class:`.Session`.
|
|
Nrr )rE rF rG )rr rE )rL Ú_identity_lookuprQ Údict) r9 r3 rq rr rs rE rF rG rH ÚobjrW Úobj2rS r8 r: rt s: þýüû
|
þüû
|
z ShardedSession._identity_lookupz Optional[_EntityBindKey[_O]])r3 r4 rH r6 c K sl |d k r<t |}|jr0|jd }|d k s,t|S |jr<|jS t|tsJt| j||f|}|d k rh||_|S )Nr )r ÚkeyrP rr rN r r\ )r9 r3 r4 rH ÚstateÚtokenrW r8 r8 r: Ú_choose_shard_and_assign1 s
|
z'ShardedSession._choose_shard_and_assignr1 zOptional[ShardIdentifier]r )r3 r4 rW rH r6 c K sz |dkr| ||¡}| ¡ r>| ¡ }|dk s0t|j||dS | j|||d}t|trd|jf |S t|t srt|S dS )zaProvide a :class:`_engine.Connection` to use in the unit of work
|
flush process.
|
|
N)rW )r3 rW r4 )
|
r{ Zin_transactionZget_transactionrP Ú
|
connectionÚget_bindrN r Úconnectr )r9 r3 r4 rW rH ZtransÚbindr8 r8 r: Úconnection_callableF s ÿ
|
z"ShardedSession.connection_callable)rW r4 r5 r2 r& )r3 rW r4 r5 rH r6 c K s. |d kr$| j |||d}|d k s$t| j| S )N)r4 r5 )r{ rP rn )r9 r3 rW r4 r5 rH r8 r8 r: r} d s ÿ zShardedSession.get_bindrU zUnion[Engine, OptionEngine])rW r r6 c C s || j |< d S r7 )rn )r9 rW r r8 r8 r: ro t s zShardedSession.bind_shard)NNN)N)r= r> r? Ú__annotations__r- rM r Z PASSIVE_OFFr Ú
|
EMPTY_DICTrt r{ r r} ro r[ r8 r8 rS r: r, s:
|
ø ó&tø$1 ü þú c @ s& e Zd ZdZdZd dddddZdS )
|
Ú set_shard_ida a loader option for statements to apply a specific shard id to the
|
primary query as well as for additional relationship and column
|
loaders.
|
|
The :class:`_horizontal.set_shard_id` option may be applied using
|
the :meth:`_sql.Executable.options` method of any executable statement::
|
|
stmt = (
|
select(MyObject).
|
where(MyObject.name == 'some name').
|
options(set_shard_id("shard1"))
|
)
|
|
Above, the statement when invoked will limit to the "shard1" shard
|
identifier for the primary query as well as for all relationship and
|
column loading strategies, including eager loaders such as
|
:func:`_orm.selectinload`, deferred column loaders like :func:`_orm.defer`,
|
and the lazy relationship loader :func:`_orm.lazyload`.
|
|
In this way, the :class:`_horizontal.set_shard_id` option has much wider
|
scope than using the "shard_id" argument within the
|
:paramref:`_orm.Session.execute.bind_arguments` dictionary.
|
|
|
.. versionadded:: 2.0.0
|
|
©rW Úpropagate_to_loadersTrU Úboolc C s || _ || _dS )aH Construct a :class:`_horizontal.set_shard_id` option.
|
|
:param shard_id: shard identifier
|
:param propagate_to_loaders: if left at its default of ``True``, the
|
shard option will take place for lazy loaders such as
|
:func:`_orm.lazyload` and :func:`_orm.defer`; if False, the option
|
will not be propagated to loaded objects. Note that :func:`_orm.defer`
|
always limits to the shard_id of the parent row in any case, so the
|
parameter only has a net effect on the behavior of the
|
:func:`_orm.lazyload` strategy.
|
|
Nr )r9 rW r
r8 r8 r: rM s zset_shard_id.__init__N)T)r= r> r? rZ Ú __slots__rM r8 r8 r8 r: r z s ÿr r' ú&Union[Result[_T], IteratorResult[_TP]]rf c s j r j}n js jr" j}nd } j}t|ts:tddd fdd } j D ]}t|t
|
rT|j } q´qT|r|j d k r|j }n0d j kr j d }nd jkr° jd }nd }|d k rÄ||S g }| ¡D ]}||}| |¡ qÒ|d j|d d
S d S )
|
NrU r rV c s* t j}| |d< j| d j|dS )NrW )rr )rG )ru rG Zupdate_execution_optionsZinvoke_statement)rW rG rh r8 r: Úiter_for_shard s
|
z-execute_and_instances.<locals>.iter_for_shardrX rW r é )Z is_selectZ load_optionsZ is_updateZ is_deleteZupdate_delete_optionsrO rN r, rP Z_non_compile_orm_optionsr rW Z_identity_tokenrF rG rR ÚappendÚmerge)rg Zactive_optionsrO r Zorm_optrW ÚpartialZresult_r8 rh r: rl ¬ s4
|
|
|
|
|
rl N)FrZ Ú
|
__future__r Útypingr r r r r r r r
|
r r Ú r r r r Zormr Z orm._typingr Zorm.interfacesr Z
|
orm.mapperr Z orm.queryr Z orm.sessionr r r Z util.typingr r Z engine.baser r r Z engine.resultr r r! r" Zorm.bulk_persistencer# Z orm.contextr$ r% r& r' Z orm.stater( Zsqlr) Z sql._typingr* Z sql.elementsr+ Ú__all__r. ÚstrrU r0 r@ r- r, r rl r8 r8 r8 r: Ú<module> sh
|
# p2
|
