U vId@sXddlmZddlmZddlmZddlZddlZ ddl Z ddl m Z m Z mZmZmZmZddlmZddlmZmZmZdd lmZdd lmZmZdd lmZdd lm Z d ddgZ!ddZ"d"ddZ#ddZ$GdddZ%GdddZ&GdddZ'Gddde%e&Z(e e)efZ*e ee*e e)e*ffZ+Gddde%e&Z,Gd d!d!e&Z-dS)#)Mapping)reduce)DatasetN)DictUnionOptionalSequenceCallableLiteral)AnnData)_normalize_indices_normalize_indexIndex) _resolve_idx) concat_arraysinner_concat_aligned_mapping) SparseDataset) AxisArraysobsobsmlayerscCsddgt|}t||S)NcSs|SN)xZ fill_valueaxisrrh/home/sam/Atlas/atlas_env/lib/python3.8/site-packages/anndata/experimental/multi_files/_anncollection.pyz_merge..)lenr)arrsZrxersrrr_mergesr!cCsPd}t|r|}nt|tr,||kr,||}|dk rH|dk rD||S|S|SdSr)callable isinstancedict)keyconvertarrZ key_convertrrr_select_convertsr(csfi}dfdd }|D]:\}}t|dkr0qi||<|D]}||||||<q3sz8_harmonize_types..check_type..cSs g|]}|ddddfqSr)rr+rrrr-5s)getattrappendr!dtype ValueError)attrr%r aattr_arrr0adatasrr check_type*s  z$_harmonize_types..check_typerX)N)itemsr) attrs_keysr6Zattrs_keys_typesr7r2keysr%rr5r_harmonize_types's  r<c@seZdZddZdS)_ConcatViewMixincCsg}d}t|dd}|dk r.t|||jd}t|tr^||jd\}}}t|||}nt|trtt |gn|}|} t |j dkr| g|||fSt t dg|j|j} d} | D]&\} } t| | k| | k@r| d7} q|jo| dko| jdkot| dd| ddk}|r4tj| dd\} }| D]:\} } | | k| | k@}||rj| || ndq8t|dd}|dk rt|||j dj}t|trt |g}||||fS)Noidxr*rTZreturn_inversevidx)r.rlimitsr#sliceindicesnparangeintarrayrr6listzipanyindices_strictsizeuniquer/n_vars)selfr>rA adatas_oidxreverseZold_oidxstartstopstepZu_oidxZ iter_limitsZ n_adatas_usedlowerupperZ need_reversemaskZold_vidxrrrrEsF     $    z_ConcatViewMixin._resolve_idxN)__name__ __module__ __qualname__rrrrrr=Dsr=c@s&eZdZdeedeedddZdS) _IterateViewMixinrFrr*) batch_sizershuffle drop_lastc cs|dkrtd|j|}|r0tj|}n tt|}td||D]X}||t|||}|dkr||dd|f} n||} t | |kr|rqH| |fVqHdS)aIterate the lazy object over an axis. Parameters ---------- batch_size How many samples to put into a batch when iterating. axis The axis to iterate over. shuffle Set to `True` to have the indices reshuffled before iterating. drop_last Set to `True` to drop a batch with the length lower than `batch_size`. r]zAxis should be either 0 or 1.rr*N) r1shaperErandomZ permutationtolistrIrangeminr) rPr^rr_r`nrDiidxbatchrrr iterate_axisus  z_IterateViewMixin.iterate_axisN)rFF)rYrZr[rGr boolrjrrrrr\tsr\c@sFeZdZdddZdddZddZdd d Zed d Zd dZ dS) MapObsViewNc Cs:||_||_||_||_||_||_||_||_| |_dSr) r6_keysrQ adatas_vidxr2r&rRdtypes obs_names) rPr2r6r;rQrnr&rRrorprrr__init__s zMapObsView.__init__Tc Csb|jdk r,||jkr,td|d|jdg}t|jD]\}}|dkrLq:t|j||j|}|jdk rx|j|}nd}|dk r||f}n|}t|t j r| |j |q:|dk rt|dt stj|n|}| ||q:t|dkrt|} |jdkr | n| |j} n(|d} |jdk r>| j|j|dd} |jdk r^|r^t||j| } | S)NzNo z in z viewr*rFcopy)rmKeyErrorr2 enumeraterQr.r6rnr#pd DataFramer/ZilocrCrEix_rr!rRroastyper&r() rPr% use_convertr rgr>r'rArhZ_arrrrr __getitem__s6     zMapObsView.__getitem__cCs.|jdk r|jStt|jd|jSdSNr)rmrIr.r6r2r;rPrrrr;s zMapObsView.keyscCs6i}|dkr|n|}|D]}|||||<q|Sr)r;r{)rPr;rzdctr%rrrto_dicts zMapObsView.to_dictcCs&|jdkrdStj|jdd|jdS)NrFrzindex)r2rvrwrrpr}rrrdfs z MapObsView.dfcCs&d|jdt|dd}|S)NzView of z with keys: r*r?)r2strr;)rPdescrrrr__repr__s"zMapObsView.__repr__)NNNNN)T)NT) rYrZr[rqr{r;rpropertyrrrrrrrls  *  rlc@seZdZdZddZd-ddZddZed d Zed d Z ed dZ eddZ eddZ eddZ eddZeddZeddZeddZejddZddZed d!d"Zed#d$Zd%d&Zd.eed'd(d)Zed*d+Zd,S)/AnnCollectionViewa An object to access the observation attributes of `adatas` in AnnCollection. Created as a result of subsetting an :class:`~anndata.experimental.AnnCollection` object. An object of this class can have `.obs`, `.obsm`, `.layers`, `.X` depending on the results of joins in the reference AnnCollection object. Notes ----- Nothing is copied until keys of the attributes or `.X` are accessed. cCs||_|jj|_|jj|_|jj|_|\|_|_|_|_g|_t |jjD]B\}}|dkrj|j |jqJt ||j|j|j }|j |qJ|jj |_ |jj|_|jj|_d\|_|_|_d|_d|_d|_||_dS)N)NNN) referencerLr6rBrQr>rArRrnrur/rrO_view_attrs_keys_attrs_dtypes _layers_view _obsm_view _obs_view_X_convert _convert_Xr&)rPrr& resolved_idxrgrAZnew_vidxrrrrqs&      zAnnCollectionView.__init__Fc Cst|d|ddk rdSd}d}||jkrn|j}|j|}t|dkrLdS|j}|j}|jdk r|j|}nd}|jg}|jg}|r|j nd}d} |j dk rt ||j } |dkr|j } nd} t |d|dt|||||| ||| dS)N__viewrr)r.rrRrr6rQrrr>rnr&r(rpsetattrrl) rPr2set_vidxr;Z attr_dtypesrRr6rQrnZ attr_convertrprrr_lazy_init_attr%sL        z!AnnCollectionView._lazy_init_attrc Cs|jdk r|jSg}t|jD]B\}}|dkr2q|j|}|j}|j|}t|trd}|jdkrt |dd|ddkrt j |dd\}}t|t r|||f}n||dd|f}| |dkr|n||qt|tr2t|t st|dkr| |||fn| ||dd|fq||f} t|t sPt j| n| } | || qt|dkrt|} |jdkr| n| |j} n(|d} |jdk r| j|jddd } | |_| S) Nr*r?Tr@irr8Frr)rrurQr6r8rnr#rrMrErKrNrCr/rrrxr!rRrry) rPZXsrgr>adatar8rArRr'rhrrrr _gather_XRs@    (   zAnnCollectionView._gather_XcCs |}|jdk r||S|S)a!Lazy subset of data matrix. The data matrix formed from the `.X` attributes of the underlying `adatas`, properly reindexed and lazily merged. Nothing is copied until `.X` is accessed, no real concatenation of the unerlying `.X` attributes is done. N)rr)rPrrrrr8s zAnnCollectionView.XcCs|jddd|jS)a Lazy subset of layers. The layers attribute formed from lazy inner join and subsetting of the `.layers` of the underlying `adatas`. No copy is made until you access a key from `.layers`, only the subset of the accessed key is copied. To get `.layers` as a dictionary, use `.layers.to_dict()`. You can also specify keys to include in the dict `.layers.to_dict(keys=['key1', 'key2'])` and if you want converters to be turned off when copying to dict `.layers.to_dict(use_convert=False)`. rT)r)rrr}rrrrs zAnnCollectionView.layerscCs|d|jS)aLazy subset of multi-dimensional annotation of observations. Points to the `.obsm` attributes of the underlying adatas ot to `.obsm` of the parent AnnCollection object depending on the `join_obsm` option of the AnnCollection object. See the docs of :class:`~anndata.experimental.AnnCollection` for details. Copy rules are the same as for `.layers`, i.e. everything is lazy. To get `.obsm` as a dictionary, use `.obsm.to_dict()`. You can also specify keys to include in the dict `.obsm.to_dict(keys=['key1', 'key2'])` and if you want converters to be turned off when copying to dict `.obsm.to_dict(use_convert=False)`. r)rrr}rrrrs zAnnCollectionView.obsmcCs|d|jS)aLazy suset of one-dimensional annotation of observations. Points to the `.obs` attributes of the underlying adatas ot to `.obs` of the parent AnnCollection object depending on the `join_obs` option of the AnnCollection object. See the docs of `~anndata.experimental.AnnCollection` for details. Copy rules are the same as for `.layers`, i.e. everything is lazy. To get `.obs` as a DataFrame, use `.obs.df`. To get `.obs` as a dictionary, use `.obs.to_dict()`. You can also specify keys to include in the dict `.obs.to_dict(keys=['key1', 'key2'])` and if you want converters to be truned off when copying to dict `.obs.to_dict(use_convert=False)`. r)rrr}rrrrs zAnnCollectionView.obscCs|jj|jS)z,Names of observations of this subset object.)rrpr>r}rrrrpszAnnCollectionView.obs_namescCs|jj|jS)z)Names of variables of this subset object.)r var_namesrAr}rrrrszAnnCollectionView.var_namescCst|jt|jfS)z;Shape of the lazily concatenated subset of the data matrix.)rrprr}rrrraszAnnCollectionView.shapecCs |jdSzNumber of observations.rrar}rrrn_obsszAnnCollectionView.n_obscCs |jdSzNumber of variables/features.r*rr}rrrrOszAnnCollectionView.n_varscCs|jS)aOn the fly converters for keys of attributes and data matrix. A function or a Mapping of functions which will be applied to the values of attributes (`.X`) or to specific keys of these attributes (`.obs`, `.obsm`, `.layers`). The keys of the Mapping should correspond to the attributes or keys of the attributes (hierarchically) and the values should be functions used for conversion. Examples ---------- :: { 'X': lambda a: a.toarray() if issparse(a) else a, # densify .X 'obsm': lambda a: np.asarray(a, dtype='float32'), # change dtype for all keys of .obsm 'obs': dict(key1 = lambda c: c.astype(str)) # change type only for one key of .obs } rr}rrrr&szAnnCollectionView.convertcCs6||_td|j|_tD]}t|d|ddqdS)Nr8rr)rr(rATTRSr)rPvaluer2rrrr&scCs t|jSr)rrpr}rrr__len__szAnnCollectionView.__len__rcCs0t||j|j\}}|||}t|j|j|Sr)r rprrrrr&rPrr>rArrrrr{s zAnnCollectionView.__getitem__cCs2t|jD]"\}}|jr |j|dk r dSq dS)zO`True` if the current subset of `adatas` has backed objects, `False` otherwise.NTF)rur6isbackedrQ)rPrgrrrr has_backedszAnnCollectionView.has_backedcCs|j\}}d|d|}|j}|jD]}tt|j|||<q*|D]4\}}t |dkrP|d|dt |dd7}qP|S)Nu0AnnCollectionView object with n_obs × n_vars =  × r : r*r?) rarrsrrIr.rr;r9rr)rPrrOrZall_attrs_keysr2r;rrrr s    "zAnnCollectionView.__repr__)ignore_X ignore_layersc Cs|s|jdkrd}n|jjdd}|jdkr0dn |jjdd}|jdkrLdnt|jjdd}|rpd}|j}n |}d}t|||||d}|j |_ |j |_ |S)a Convert this AnnCollectionView object to an AnnData object. Parameters ---------- ignore_X if `True`, adds `.X` to the AnnData object. ignore_layers if `True`, copies `.layers` to the AnnData object. NFr)rrrra) rrrrrvrwrarr rpr) rPrrrrrr8rarrrrto_adatas" zAnnCollectionView.to_adatacCs|jjSz1Dict of all accessible attributes and their keys.)rr:r}rrrr:8szAnnCollectionView.attrs_keysN)F)FF)rYrZr[__doc__rqrrrr8rrrrprrarrOr&setterrrr{rrrkrr:rrrrrsD  -2             !rc @seZdZdZd%eeeeeeffe e de e de e de ee eee ee e e e d ddZ ed d d Zed d Zejdd ZeddZeddZeddZeddZeddZddZddZd&ddZedd Zed!d"Zd#d$ZdS)' AnnCollectionu Lazily concatenate AnnData objects along the `obs` axis. This class doesn't copy data from underlying AnnData objects, but lazily subsets using a joint index of observations and variables. It also allows on-the-fly application of prespecified converters to `.obs` attributes of the AnnData objects. Subsetting of this object returns an `AnnCollectionView`, which provides views of `.obs`, `.obsm`, `.layers`, `.X` from the underlying AnnData objects. Parameters ---------- adatas The objects to be lazily concatenated. If a Mapping is passed, keys are used for the `keys` argument and values are concatenated. join_obs If "inner" specified all `.obs` attributes from `adatas` will be inner joined and copied to this object. If "outer" specified all `.obsm` attributes from `adatas` will be outer joined and copied to this object. For "inner" and "outer" subset objects will access `.obs` of this object, not the original `.obs` attributes of `adatas`. If `None`, nothing is copied to this object's `.obs`, a subset object will directly access `.obs` attributes of `adatas` (with proper reindexing and dtype conversions). For `None`the inner join rule is used to select columns of `.obs` of `adatas`. join_obsm If "inner" specified all `.obsm` attributes from `adatas` will be inner joined and copied to this object. Subset objects will access `.obsm` of this object, not the original `.obsm` attributes of `adatas`. If `None`, nothing is copied to this object's `.obsm`, a subset object will directly access `.obsm` attributes of `adatas` (with proper reindexing and dtype conversions). For both options the inner join rule for the underlying `.obsm` attributes is used. join_vars Specify how to join `adatas` along the var axis. If `None`, assumes all `adatas` have the same variables. If "inner", the intersection of all variables in `adatas` will be used. label Column in `.obs` to place batch information in. If it's None, no column is added. keys Names for each object being added. These values are used for column values for `label` or appended to the index if `index_unique` is not `None`. Defaults to incrementing integer labels. index_unique Whether to make the index unique by using the keys. If provided, this is the delimeter between "{orig_idx}{index_unique}{key}". When `None`, the original indices are kept. convert You can pass a function or a Mapping of functions which will be applied to the values of attributes (`.obs`, `.obsm`, `.layers`, `.X`) or to specific keys of these attributes in the subset object. Specify an attribute and a key (if needed) as keys of the passed Mapping and a function to be applied as a value. harmonize_dtypes If `True`, all retrieved arrays from subset objects will have the same dtype. indices_strict If `True`, arrays from the subset objects will always have the same order of indices as in selection used to subset. This parameter can be set to `False` if the order in the returned arrays is not important, for example, when using them for stochastic gradient descent. In this case the performance of subsetting can be a bit better. Examples ---------- >>> from scanpy.datasets import pbmc68k_reduced, pbmc3k_processed >>> adata1, adata2 = pbmc68k_reduced(), pbmc3k_processed() >>> adata1.shape (700, 765) >>> adata2.shape (2638, 1838) >>> dc = AnnCollection([adata1, adata2], join_vars='inner') >>> dc AnnCollection object with n_obs × n_vars = 3338 × 208 constructed from 2 AnnData objects view of obsm: 'X_pca', 'X_umap' obs: 'n_genes', 'percent_mito', 'n_counts', 'louvain' >>> batch = dc[100:200] # AnnCollectionView >>> batch AnnCollectionView object with n_obs × n_vars = 100 × 208 obsm: 'X_pca', 'X_umap' obs: 'n_genes', 'percent_mito', 'n_counts', 'louvain' >>> batch.X.shape (100, 208) >>> len(batch.obs['louvain']) 100 innerNT)router) r6join_obs join_obsm join_varslabelr; index_uniquer&harmonize_dtypesrLc sttr6|dk rtdtt}ntddD|_ddD} tfdd| ddD} | rdj|_nh|dkrt t j j | } g|_D]6}| |jr|jdqt| |j}|j|q| |_ntd t jd dDd d }|dkr*ttt}t jjtttd dD|d}|dk rv|jj|t|d}t ||_|jjstdt t!"}g|_#|dk r|$d|j#dt jddD|d d}|j|_%||_&nt j'|jd|_&|dk r||j&|<d|_(|dkrt|$d|j#dt)ddD|jd|_(|j(ikrlt*|ddn|j(|_(i|_+|D]"}tt,d||j+|<q~ddD]}|j+-D]\}}t,||}t,d|}g}|D]\}||kr||j.}||j.}t|dks4|d|dks4|dkr||q||j+|<qq|_/dj0g|_1t2tdD]&}|j1|j1||dj0qx||_3d|_4tdkr| rt5|j+|j/|_4| |_6dS)NzXCannot specify categories in both mapping keys and using `keys`. Only specify this once.cSsg|]}dqSrrr,rrrrr-sz*AnnCollection.__init__..cSsg|] }|jqSr)rrrrrr-scsg|]}dj|qSr)requals)r,Zvrsr5rrr-sr*rrzSAdatas have different variables. Please specify join_vars='inner' for intersection.cSsg|]}t|jqSr)rvZSeriesrpr,r3rrrr-sT) ignore_indexcSsg|]}|jdqSrrrrrrr-s) categories)sepz!Observation names are not unique.rcSsg|] }|jqSr)rrrrrr-s)joinrrrcSsg|] }|jqSr)rrrrrr-s)rr)7r#r TypeErrorrIr;valuesrnallrrrvr intersectionrr/rr1concatrErFrryrZ CategoricalZ from_codesrepeatcatmaprpZ is_uniquewarningswarn UserWarningrrsrremover_obsrw_obsmrrrr.r9rar6rrBrdrrr<rL)rPr6rrrrr;rr&rrLZvars_names_listZvars_eqrrZ adata_vidxZconcat_indicesZ label_colZ view_attrsZ concat_annotr2r3Zai_attrZa0_attrZnew_keysr%Z a0_ashapeZ ai_ashapergrr5rrqs                         $zAnnCollection.__init__rcCs.t||j|j\}}|||}t||j|Sr)r rprrrr&rrrrr{s zAnnCollection.__getitem__cCs|jS)aOn the fly converters for keys of attributes and data matrix. A function or a Mapping of functions which will be applied to the values of attributes (`.X`) or to specific keys of these attributes (`.obs`, `.obsm`, `.layers`) of subset objects. The converters are not applied to `.obs` and `.obsm` (if present) of this object, only to the attributes of subset objects. The keys of the Mapping should correspond to the attributes or keys of the attributes (hierarchically) and the values should be functions used for conversion. Examples -------- :: { 'X': lambda a: a.toarray() if issparse(a) else a, # densify .X 'obsm': lambda a: np.asarray(a, dtype='float32'), # change dtype for all keys of .obsm 'obs': dict(key1 = lambda c: c.astype(str)) # change type only for one key of .obs } rr}rrrr& szAnnCollection.convertcCs ||_dSrr)rPrrrrr&8scCs|jS)zOne-dimensional annotation of observations. If `join_obs` was set to "inner" and "outer", subset objects' `.obs` will point to this `.obs`; otherwise, to `.obs` of the underlying objects (`adatas`). )rr}rrrr<szAnnCollection.obscCs|jS)aMulti-dimensional annotation of observations. If `join_obsm` was set to "inner", subset objects' `.obsm` will point to this `.obsm`; otherwise, to `.obsm` of the underlying objects (`adatas`). In the latter case, `.obsm` of this object will be `None`. )rr}rrrrEszAnnCollection.obsmcCs|jdt|jfS)z,Shape of the lazily concatenated data matrixr?)rBrrr}rrrraOszAnnCollection.shapecCs |jdSrrr}rrrrTszAnnCollection.n_obscCs |jdSrrr}rrrrOYszAnnCollection.n_varscCs |jdS)Nr?)rBr}rrrr^szAnnCollection.__len__cCsd|jksd|jkr||j}d|jkrF|jdk r@|jjddnd}n |j}|j}d|jkr|jdk r|jjddD]\}}|||<qtd|||jd}|j|_|j |_ |S)zConvert this AnnCollection object to an AnnData object. The AnnData object won't have `.X`, only `.obs` and `.obsm`. rrNFr)r8rrra) rrprrrsrr9r rar)rPZ concat_viewrrr%rrrrrras      zAnnCollection.to_adatacCs t|||S)a Get a subsettable key from an attribute (array-like) or an attribute. Returns a LazyAttrData object which provides subsetting over the specified attribute (`.obs` or `.obsm`) or over a key from this attribute. In the latter case, it acts as a lazy array. ) LazyAttrData)rPr2r%rrr lazy_attr|szAnnCollection.lazy_attrcCstdd|jDS)zB`True` if `adatas` have backed AnnData objects, `False` otherwise.cSsg|] }|jqSr)rrrrrr-sz,AnnCollection.has_backed..)rKr6r}rrrrszAnnCollection.has_backedcCs:i}|jD]}tt||}|||<q ||j|Sr)rrIr.r;updater)rPZ _attrs_keysr2r;rrrr:s    zAnnCollection.attrs_keyscCs|j\}}d|d|}|dt|jd7}|jD]4\}}t|dkr:|d|dt|dd 7}q:|jD]B}tt|| }t|dkrv|d |dt|dd 7}qvd |jkrt|j }t|dkr|d t|dd 7}|S) Nu,AnnCollection object with n_obs × n_vars = rz constructed from z AnnData objectsrz view of rr*r?rrz own obs: ) rarr6rr9rrrIr.r;r)rPrrOrr2r;rrrrs  "  "  zAnnCollection.__repr__) rNNNNNNTT)N)rYrZr[rrrr rrrr ConvertTyperkrqrr{rr&rrrrarrOrrrrr:rrrrrrBs\Z               rc@sPeZdZd eeeedddZddZeddZ ed d Z ed d Z dS)rNadsetr2r%cCs||_||_||_dSrr)rPrr2r%rrrrqszLazyAttrData.__init__cCsd}d}t|tr8|jdkr8|d}t|dkr8|d}|dkrL|j|}n |j|}t||j}|jdk rv||j}|dkr|S|dd|fS)N)rrrr*)r#tupler2rrr.r%)rPrr>rAviewr4rrrr{s      zLazyAttrData.__getitem__cCs`|jj}|jdkr|S|jdkr*|dfS|jdkrX|jdk rX|d|ddjdfSdSdS)N)r8rrrrr*)rrar2r%)rPrarrrras   zLazyAttrData.shapecCs|jdk rt|jSdSr|)rarr}rrrndimszLazyAttrData.ndimcCsN|jj}|dk r*|j|kr*||j|jS|dd}t|drF|jSdSdS)Nr*r0)rrr2r%hasattrr0)rPrr2rrrr0s  zLazyAttrData.dtype)N) rYrZr[rrrrqr{rrarr0rrrrrs  r)N).collections.abcr functoolsrZh5pyrnumpyrEZpandasrvrtypingrrrrr r Z _core.anndatar Z _core.indexr rrZ _core.viewsrZ _core.mergerrZ_core.sparse_datasetrZ_core.aligned_mappingrrr!r(r<r=r\rlrrZ DictCallablerrrrrrrs8          0,YG g