U mdɎ@sdZddlmZddlmZddlZddlmZmZm Z m Z m Z m Z ddl Z ddlZddlZddlmZmZmZmZddlmZmZddlmZdd lmZd d lmZ d d l!m"Z#d d l$m%Z%m&Z&m'Z'm(Z(m)Z)d dl*m+Z+d dl,m-Z-m.Z.ddl/m0Z0ddl$m1Z1zddl2m3Z4Wne5k r.dZ4YnXddl6m7Z7dVeee8ee8ee8ee8e9e9ee ej:ej:fdddZ;dWeee8ee8ee8ee8e9e9eede ej:ej:ffdddZe>?eddd eee9d d!d"Z@e>?ej:ddd eee9d d#d$ZAe>?edddddddeee9e9ee8ee=ee=eed%d&d'ZBdXee9e9ee8eed(d)d*ZCdYeeej:efeeDeej:e=e9ee+d-e e=fee+d.e8eed/ d0d1ZEdZeee=e e=fee8e9eed2d3d4ZFd5d6ZGed[eeeej:fe9eeDe9ee=ee=d7d8d9ZHeH?ej:ddddd:e9eeDe9e9d:d;d<ZIeH?eddddd:e9eeDe9e9d:d=d>ZJeH?edddddd?ee9eeDe9ee=ee=eed@dAdBZKd\eeej:efeeDee8e(e9eedCdDdEZLe&dFdGid]ddddHeeee8e e8fee8e(e9e9eedIdJdKZMdLdMZNdNdOZOe jPddPd^ej:e8e(e9e9dQdRdSZQd_dTdUZRdS)`zdSimple Preprocessing Functions Compositions of these functions are found in sc.preprocess.recipes. )singledispatch)NumberN)UnionOptionalTuple CollectionSequenceIterable)issparseisspmatrix_csr csr_matrixspmatrix) sparsefuncs check_array)is_categorical_dtype)AnnData)logging)settings)sanitize_anndatadeprecated_arg_namesview_to_actual AnyRandom_check_array_function_arguments)Literal) _get_obs_rep _set_obs_rep)materialize_as_ndarray) _get_mean_var)filter_genes_dispersionTF)data min_counts min_genes max_counts max_genesinplacecopyreturncCs|rtdtdd||||fD}|dkr8tdt|tr|rN|n|}tt|j ||||\} } |sx| | fS|dkr|dkr| |j d<n | |j d<| | |r|SdS|} |dkr|n|} |dkr|n|} t j|dkr|dkr| n| d kdd }t | r|j}| dk r || k} | dk r2|| k} t | }|d krd |d }|dk sh|dk r|d 7}||dkr|dn|d7}|dk s|dk r|d7}||dkr|dn|d7}t|| |fS)u Filter cell outliers based on counts and numbers of genes expressed. For instance, only keep cells with at least `min_counts` counts or `min_genes` genes expressed. This is to filter measurement outliers, i.e. “unreliable” observations. Only provide one of the optional parameters `min_counts`, `min_genes`, `max_counts`, `max_genes` per call. Parameters ---------- data The (annotated) data matrix of shape `n_obs` × `n_vars`. Rows correspond to cells and columns to genes. min_counts Minimum number of counts required for a cell to pass filtering. min_genes Minimum number of genes expressed required for a cell to pass filtering. max_counts Maximum number of counts required for a cell to pass filtering. max_genes Maximum number of genes expressed required for a cell to pass filtering. inplace Perform computation inplace or return result. Returns ------- Depending on `inplace`, returns the following arrays or directly subsets and annotates the data matrix: cells_subset Boolean index mask that does filtering. `True` means that the cell is kept. `False` means the cell is removed. number_per_cell Depending on what was tresholded (`counts` or `genes`), the array stores `n_counts` or `n_cells` per gene. Examples -------- >>> import scanpy as sc >>> adata = sc.datasets.krumsiek11() >>> adata.n_obs 640 >>> adata.var_names ['Gata2' 'Gata1' 'Fog1' 'EKLF' 'Fli1' 'SCL' 'Cebpa' 'Pu.1' 'cJun' 'EgrNab' 'Gfi1'] >>> # add some true zeros >>> adata.X[adata.X < 0.3] = 0 >>> # simply compute the number of genes per cell >>> sc.pp.filter_cells(adata, min_genes=0) >>> adata.n_obs 640 >>> adata.obs['n_genes'].min() 1 >>> # filter manually >>> adata_copy = adata[adata.obs['n_genes'] >= 3] >>> adata_copy.obs['n_genes'].min() >>> adata.n_obs 554 >>> adata.obs['n_genes'].min() 3 >>> # actually do some filtering >>> sc.pp.filter_cells(adata, min_genes=3) >>> adata.n_obs 554 >>> adata.obs['n_genes'].min() 3 ,`copy` is deprecated, use `inplace` instead.css|]}|dk VqdSN.0optionr+r+U/home/sam/Atlas/atlas_env/lib/python3.8/site-packages/scanpy/preprocessing/_simple.py zszfilter_cells..rzjOnly provide one of the optional parameters `min_counts`, `min_genes`, `max_counts`, `max_genes` per call.Nn_countsZn_genesrZaxis filtered out z cells that have z less than z genes expressed countsz more than )loggwarningsum ValueError isinstancerr'r filter_cellsXobs_inplace_subset_obsnpr A1info)r!r"r#r$r%r&r'n_given_optionsadata cell_subsetnumberr; min_number max_numberZnumber_per_cellsmsgr+r+r/r:*sjN              r:)r!r" min_cellsr$ max_cellsr&r'r(cCs|rtdtdd||||fD}|dkr8tdt|tr|rN|n|}tt|j ||||d\} } |sz| | fS|dkr|dkr| |j d<n | |j d <| | |r|SdS|} |dkr|n|} |dkr|n|} t j|dkr|dkr| n| d kd d }t | r|j}| dk r"|| k} | dk r4|| k} t | }|d krd |d }|dk sj|dk r|d7}||dkr|dn|d7}|dk s|dk r|d7}||dkr|dn|d7}t|| |fS)ua Filter genes based on number of cells or counts. Keep genes that have at least `min_counts` counts or are expressed in at least `min_cells` cells or have at most `max_counts` counts or are expressed in at most `max_cells` cells. Only provide one of the optional parameters `min_counts`, `min_cells`, `max_counts`, `max_cells` per call. Parameters ---------- data An annotated data matrix of shape `n_obs` × `n_vars`. Rows correspond to cells and columns to genes. min_counts Minimum number of counts required for a gene to pass filtering. min_cells Minimum number of cells expressed required for a gene to pass filtering. max_counts Maximum number of counts required for a gene to pass filtering. max_cells Maximum number of cells expressed required for a gene to pass filtering. inplace Perform computation inplace or return result. Returns ------- Depending on `inplace`, returns the following arrays or directly subsets and annotates the data matrix gene_subset Boolean index mask that does filtering. `True` means that the gene is kept. `False` means the gene is removed. number_per_gene Depending on what was tresholded (`counts` or `cells`), the array stores `n_counts` or `n_cells` per gene. r)css|]}|dk VqdSr*r+r,r+r+r/r0szfilter_genes..rzjOnly provide one of the optional parameters `min_counts`, `min_cells`, `max_counts`, `max_cells` per call.)rIr"rJr$Nr1Zn_cellsrr2r3z genes that are detected z in less than z cellsr4z in more than )r5r6r7r8r9rr'r filter_genesr;varZ_inplace_subset_varr>r r?r@)r!r"rIr$rJr&r'rArBZ gene_subsetrDr;rErFZnumber_per_generGrHr+r+r/rKsn/               rK)baser'chunked chunk_sizelayerobsmr;rMr'rNrOrPrQcCst||||dt|||dS)uj Logarithmize the data matrix. Computes :math:`X = \log(X + 1)`, where :math:`log` denotes the natural logarithm unless a different base is given. Parameters ---------- X The (annotated) data matrix of shape `n_obs` × `n_vars`. Rows correspond to cells and columns to genes. base Base of the logarithm. Natural logarithm is used by default. copy If an :class:`~anndata.AnnData` is passed, determines whether a copy is returned. chunked Process the data matrix in chunks, which will save memory. Applies only to :class:`~anndata.AnnData`. chunk_size `n_obs` of the chunks to process the data in. layer Entry of layers to tranform. obsm Entry of obsm to transform. Returns ------- Returns or updates `data`, depending on `copy`. )rNrOrPrQr'rM)r log1p_arrayrRr+r+r/log1ps)rUrMr'cCs.t|dtjtjf|d}t|jd|d|_|S)N)ZcsrZcsc)Z accept_sparsedtyper'FrS)rr>Zfloat64float32rUr!r;rMr'r+r+r/ log1p_sparseLs rZcCs|r*t|jtjs |t}qR|}n(t|jtjsRt|jtsR|t}tj||d|dk r~tj |t ||d|S)N)out) r> issubdtyperWZfloatingastypefloatr'complexrUdividelogrYr+r+r/rTUs   rT)rMr'rNrOrPrQr(c Csd|krtd|r"|n|}t||rz|dk sB|dk rJtd||D]"\}}} t||dd|j|| <qTn,t |||d} t| d|d} t || ||dd|i|j d<|r|SdS) NrUz,adata.X seems to be already log-transformed.zFCurrently cannot perform chunked operations on arrays not stored in X.FrVrPrQrSrM) Zuns_keysr5r6r'rNotImplementedError chunked_XrUr;rrZuns) rBrMr'rNrOrPrQchunkstartendr;r+r+r/ log1p_anndatafs"  rh)r!r'rNrOr(c Cst|tr`|r|n|}|rH||D]\}}}t||j||<q(n t|j|_|r\|SdS|}t|svt|S|SdS)up Square root the data matrix. Computes :math:`X = \sqrt(X)`. Parameters ---------- data The (annotated) data matrix of shape `n_obs` × `n_vars`. Rows correspond to cells and columns to genes. copy If an :class:`~anndata.AnnData` object is passed, determines whether a copy is returned. chunked Process the data matrix in chunks, which will save memory. Applies only to :class:`~anndata.AnnData`. chunk_size `n_obs` of the chunks to process the data in. Returns ------- Returns or updates `data`, depending on `copy`. N)r9rr'rdsqrtr;r r>) r!r'rNrOrBrerfrgr;r+r+r/ris    rir1r+all)afterr;) r!counts_per_cell_aftercounts_per_cell key_n_countsr'layersuse_repr"r(c Cst|tr$td}|r"|n|} |dkr`tt| j|d\} }|| j|<| | || }t | j|||dkr| j n|}|dkr|} n.|dkrt || } n|dkrd} ntd|D]:} t| j | |d\} }t | j | | |dd }|| j | <qtjd |d |d |r | SdS|r2|n|}|dkrn|sNtd t||d\} }|| }|| }|dkrt |}tZtd||dk7}||}t|s|t|ddt jf}nt|d|W5QRX|r|SdS)u Normalize total counts per cell. .. warning:: .. deprecated:: 1.3.7 Use :func:`~scanpy.pp.normalize_total` instead. The new function is equivalent to the present function, except that * the new function doesn't filter cells based on `min_counts`, use :func:`~scanpy.pp.filter_cells` if filtering is needed. * some arguments were renamed * `copy` is replaced by `inplace` Normalize each cell by total counts over all genes, so that every cell has the same total count after normalization. Similar functions are used, for example, by Seurat [Satija15]_, Cell Ranger [Zheng17]_ or SPRING [Weinreb17]_. Parameters ---------- data The (annotated) data matrix of shape `n_obs` × `n_vars`. Rows correspond to cells and columns to genes. counts_per_cell_after If `None`, after normalization, each cell has a total count equal to the median of the *counts_per_cell* before normalization. counts_per_cell Precomputed counts per cell. key_n_counts Name of the field in `adata.obs` where the total counts per cell are stored. copy If an :class:`~anndata.AnnData` is passed, determines whether a copy is returned. min_counts Cells with counts less than `min_counts` are filtered out during normalization. Returns ------- Returns or updates `adata` with normalized version of the original `adata.X`, depending on `copy`. Examples -------- >>> import scanpy as sc >>> adata = AnnData(np.array([[1, 0], [3, 0], [5, 6]])) >>> print(adata.X.sum(axis=1)) [ 1. 3. 11.] >>> sc.pp.normalize_per_cell(adata) >>> print(adata.obs) >>> print(adata.X.sum(axis=1)) n_counts 0 1.0 1 3.0 2 11.0 [ 3. 3. 3.] >>> sc.pp.normalize_per_cell( >>> adata, counts_per_cell_after=1, >>> key_n_counts='n_counts2', >>> ) >>> print(adata.obs) >>> print(adata.X.sum(axis=1)) n_counts n_counts2 0 1.0 3.0 1 3.0 3.0 2 11.0 3.0 [ 1. 1. 1.] z#normalizing by total count per cellN)r"rjrkr;z&use_rep should be "after", "X" or NoneTr'z> finished ({time_passed}): normalized adata.X and added z2, counts per cell before normalization (adata.obs)timezCan only be run with copy=Trueignorerr)r9rr5r@r'rr:r;r<r=normalize_per_cellrokeysr>Zmedianr8warningscatch_warnings simplefilterr ZnewaxisrZinplace_row_scale)r!rlrmrnr'rorpr"rfrBrCrkrPZsubsetcountstempr;r+r+r/rusZQ              ru)rBrvn_jobsr'r(csptd|}t|jr$td|r0|n|}t||jrP||t|t r`|g}t|jrv|j |_|dkrt j n|}d}|d| kr@t|j|dr@t|dkrtdtdtj|jjd d }|j|djjD]D}||j|dkj}t|jjD]\} } | |||| f<qqd }n*|rR|j|}n |j}|dd d ttd|jjd|t } t|jjd| t } g} tj!|j| dd}|rtj!|| dd}t|D]2\}}|r||}n|}| "t#|||fqddl$m%}m&||dfdd| D}t'|j|jj(|_tjd|d|rl|SdS)a Regress out (mostly) unwanted sources of variation. Uses simple linear regression. This is inspired by Seurat's `regressOut` function in R [Satija15]. Note that this function tends to overcorrect in certain circumstances as described in :issue:`526`. Parameters ---------- adata The annotated data matrix. keys Keys for observation annotation on which to regress on. n_jobs Number of jobs for parallel computation. `None` means using :attr:`scanpy._settings.ScanpyConfig.n_jobs`. copy Determines whether a copy of `adata` is returned. Returns ------- Depending on `copy` returns or updates `adata` with the corrected data matrix. zregressing out z= sparse input is densified and may lead to high memory useNFrrzwIf providing categorical variable, only a single one is allowed. For this one we regress on the mean for each category.z2... regressing on per-gene means within categoriesrX)rWTonesg?ir2)Paralleldelayed)r|c3s|]}t|VqdSr*)_regress_out_chunk)r-taskrr+r/r0szregress_out..z finishedrr))r5r@r r;r'rZis_viewZ_init_as_actualr9strtoarraysettr|Zobs_keysrr<lenr8debugr>zerosshapecat categoriesvalues enumerateTmeaninsertceilminr]intZ array_splitappendtupleZjoblibr~rvstackrW)rBrvr|r'rfvariable_is_categorical regressorscategorymaskZixxZ len_chunkZn_chunkstasksZ chunk_listZregressors_chunkidx data_chunkregresr~resr+rr/ regress_out:sZ     &    " rc Cs&|d}|d}|d}g}ddlm}ddlm}t|jdD]}|dd|f|d|fks~||dd|fqB|rtj t |jd|dd|ff}n|}z0|j |dd|f||j d} | j} Wn0|k rtdt|jd} YnX|| qBt|S)Nrrr)PerfectSeparationError)familyz9Encountered PerfectSeparationError, setting to 0 as in R.)Zstatsmodels.apiapiZstatsmodels.tools.sm_exceptionsrrangeranyrr>Zc_r}ZGLMZfamiliesZGaussianfitZresid_responser5r6rr) r!rrrZresponses_chunk_listsmrZ col_indexrresultZ new_columnr+r+r/rs2   (    rr; zero_center max_valuer'rPrQcCsPt||d|dk r&tdt||dk r@tdt|t||||dS)uQ Scale data to unit variance and zero mean. .. note:: Variables (genes) that do not display any variation (are constant across all observations) are retained and (for zero_center==True) set to 0 during this operation. In the future, they might be set to NaNs. Parameters ---------- X The (annotated) data matrix of shape `n_obs` × `n_vars`. Rows correspond to cells and columns to genes. zero_center If `False`, omit zero-centering variables, which allows to handle sparse input efficiently. max_value Clip (truncate) to this value after scaling. If `None`, do not clip. copy Whether this function should be performed inplace. If an AnnData object is passed, this also determines if a copy is returned. layer If provided, which element of layers to scale. obsm If provided, which element of obsm to scale. Returns ------- Depending on `copy` returns or updates `adata` with a scaled `adata.X`, annotated with `'mean'` and `'std'` in `adata.var`. rbNz1`layer` argument inappropriate for value of type z0`obsm` argument inappropriate for value of type )rrr')rr8type scale_arrayrr+r+r/scales ( rrrr'return_mean_stdcCs|r |}|s"|dk r"tdt|jtjrFtd|t}t |\}}t |}d||dk<t |r|r|t dt |d|n|r||8}||}|dk rtd|||||k<|r|||fS|SdS)Nz<... be careful when using `max_value` without `zero_center`.zV... as scaling leads to float results, integer input is cast to float, returning copy.rrz!Cannot zero-center sparse matrix.z... clipping at max_value )r'r5r@r>r\rWintegerr]r^rrir r8rZinplace_column_scaler)r;rrr'rrrLstdr+r+r/rs6        rcCs,|rtd|}d}t|||||dS)Nz]... as `zero_center=True`, sparse input is densified and may lead to large memory consumptionF)rr'rr)r5r@rr)r;rrr'rr+r+r/ scale_sparse&s r)rrr'rPrQ)rBrrr'rPrQr(cCsf|r |n|}t|t|||d}t|||ddd\}|jd<|jd<t||||d|rb|SdS)NrbFTrrr)r'rrrrLr)rBrrr'rPrQr;r+r+r/ scale_anndata@s r)r!fractionn_obs random_stater'r(c Cstj|t|tr|jn|jd}|dk r4|}nN|dk rz|dksL|dkrZtd|t||}t d|dntdtjj ||dd }t|tr|r|| S| |n|}|||fSdS) u Subsample to a fraction of the number of observations. Parameters ---------- data The (annotated) data matrix of shape `n_obs` × `n_vars`. Rows correspond to cells and columns to genes. fraction Subsample to this `fraction` of the number of observations. n_obs Subsample to this number of observations. random_state Random seed to change subsampling. copy If an :class:`~anndata.AnnData` is passed, determines whether a copy is returned. Returns ------- Returns `X[obs_indices], obs_indices` if data is array-like, otherwise subsamples the passed :class:`~anndata.AnnData` (`copy == False`) or returns a subsampled copy of it (`copy == True`). rNrz*`fraction` needs to be within [0, 1], not z... subsampled to z data pointsz"Either pass `n_obs` or `fraction`.F)sizereplace)r>randomseedr9rrrr8rr5rchoicer'r=) r!rrrr'Z old_n_obsZ new_n_obsZ obs_indicesr;r+r+r/ subsampleYs"     rZ target_countsrm)rrr')rBrm total_countsrrr'r(cCsf|dk }|dk }||kr td|r,|}|rDt|j||||_n|rZt|j||||_|rb|SdS)a Downsample counts from count matrix. If `counts_per_cell` is specified, each cell will downsampled. If `total_counts` is specified, expression matrix will be downsampled to contain at most `total_counts`. Parameters ---------- adata Annotated data matrix. counts_per_cell Target total counts per cell. If a cell has more than 'counts_per_cell', it will be downsampled to this number. Resulting counts can be specified on a per cell basis by passing an array.Should be an integer or integer ndarray with same length as number of obs. total_counts Target total counts. If the count matrix has more than `total_counts` it will be downsampled to have this number. random_state Random seed for subsampling. replace Whether to sample the counts with replacement. copy Determines whether a copy of `adata` is returned. Returns ------- Depending on `copy` returns or updates an `adata` with downsampled `.X`. Nz@Must specify exactly one of `total_counts` or `counts_per_cell`.)r8r'_downsample_total_countsr;_downsample_per_cell)rBrmrrrr'Ztotal_counts_callZcounts_per_cell_callr+r+r/downsample_countss)rc CsT|jd}t|tr"t||}n t|}|jtjdd}t|tjrTt ||kr\t dt |rt |}t |s|t|}t|jdd}t||kd}t|j|jdd}|D]"} || } t| || ||dd q||tk r||}nVt|jdd}t||kd}|D],} || ddf} t| || ||dd q"|S) NrFrqzIf provided, 'counts_per_cell' must be either an integer, or coercible to an `np.ndarray` of length as number of observations by `np.asarray(counts_per_cell)`.rr2Trrr&)rr9rr>fullZasarrayr]int_ndarrayrr8r rr r Zravelr7Znonzerosplitr!Zindptr_downsample_arrayeliminate_zeros) r;rmrrr original_typeZtotalsZ under_targetrowsZrowidxrowr+r+r/rsP     rcCst|}|}||kr|St|rjt|}t|smultiplyr)r;rrrtotalrvr+r+r/rs* r)cache)coltargetrrr&c Cstj||}|r&d|dd<n t|}t|d}tjj|||d}|d}|D]*} | ||krz|d7}qd||d7<q`|S)z Evenly reduce counts in cell to target amount. This is an internal function and has some restrictions: * total counts in cell must be less than target rNr)rr)r>rrZcumsumZ zeros_likerrsort) rrrrr&Z cumcountsrsampleZgeneptrcountr+r+r/rs    rcCs||jdd8}tj|dd}tjjj||d\}}t|ddd}|dd|f}||}|ddd|f}t|j |j j S)Nrr2F)Zrowvar)kr) rr>ZcovspsparseZlinalgZeigshZargsortdotr)r!Zn_compsCZevalsZevecsZidcsr+r+r/ _pca_fallback5sr)NNNNTF)NNNNTF)FFN)NNr1Fr+Nr)NF)TNFNN)NNrF)NN)rTF)r)S__doc__ functoolsrnumbersrrwtypingrrrrrr Znumbanumpyr>ZscipyrZ scipy.sparser r r r Z sklearn.utilsrrZpandas.api.typesrZanndatarrr5Z _settingsrr_utilsrrrrrZ_compatrgetrrZ _distributedrrZ dask.arrayarrayda ImportErrorZ!_deprecated.highly_variable_genesr rboolrr:rKrrUregisterrZrTrhrir^rurrrrrrrrrrZnjitrrr+r+r+r/s               l. # .    i%/ - 5 80 $