U mdU@sddlmZddlmZddlZddlZddlm Z ddl m Z ddl m ZddlmZdeeeeeeeeeeeeeeeeeeeeeeeedddZdeeeeeeeeeeeeeeeeedddZdeeeeedddZdS))AnnData)OptionalN)sparse)logging) preprocessing) _get_obs_rep@皙?{Gz?? euclideanTF)adata adata_sim batch_keysim_doublet_ratioexpected_doublet_ratestdev_doublet_rate!synthetic_doublet_umi_subsamplingknn_dist_metricnormalize_variance log_transform mean_center n_prin_compsuse_approx_neighborsget_doublet_neighbor_parents n_neighbors thresholdverbosecopy random_statereturncsvz ddl}Wntk r(tdYnX|r6|}td}|d fdd dk r|jkrtdt |j}fdd |D}t d d |D}|j |j j|_i|jd <tt|d d |D|jd d <|jd d<n<}|dd|jd<|dd|jd<|d|jd <tjd|d|rn|SdSdS)u Predict doublets using Scrublet [Wolock19]_. Predict cell doublets using a nearest-neighbor classifier of observed transcriptomes and simulated doublets. Works best if the input is a raw (unnormalized) counts matrix from a single sample or a collection of similar samples from the same experiment. This function is a wrapper around functions that pre-process using Scanpy and directly call functions of Scrublet(). You may also undertake your own preprocessing, simulate doublets with scanpy.external.pp.scrublet_simulate_doublets(), and run the core scrublet function scanpy.external.pp.scrublet.scrublet(). .. note:: More information and bug reports `here `__. Parameters ---------- adata The annotated data matrix of shape ``n_obs`` × ``n_vars``. Rows correspond to cells and columns to genes. Expected to be un-normalised where adata_sim is not supplied, in which case doublets will be simulated and pre-processing applied to both objects. If adata_sim is supplied, this should be the observed transcriptomes processed consistently (filtering, transform, normalisaton, hvg) with adata_sim. adata_sim (Advanced use case) Optional annData object generated by sc.external.pp.scrublet_simulate_doublets(), with same number of vars as adata. This should have been built from adata_obs after filtering genes and cells and selcting highly-variable genes. batch_key Optional `adata.obs` column name discriminating between batches. sim_doublet_ratio Number of doublets to simulate relative to the number of observed transcriptomes. expected_doublet_rate Where adata_sim not suplied, the estimated doublet rate for the experiment. stdev_doublet_rate Where adata_sim not suplied, uncertainty in the expected doublet rate. synthetic_doublet_umi_subsampling Where adata_sim not suplied, rate for sampling UMIs when creating synthetic doublets. If 1.0, each doublet is created by simply adding the UMI counts from two randomly sampled observed transcriptomes. For values less than 1, the UMI counts are added and then randomly sampled at the specified rate. knn_dist_metric Distance metric used when finding nearest neighbors. For list of valid values, see the documentation for annoy (if `use_approx_neighbors` is True) or sklearn.neighbors.NearestNeighbors (if `use_approx_neighbors` is False). normalize_variance If True, normalize the data such that each gene has a variance of 1. `sklearn.decomposition.TruncatedSVD` will be used for dimensionality reduction, unless `mean_center` is True. log_transform Whether to use :func:``~scanpy.pp.log1p`` to log-transform the data prior to PCA. mean_center If True, center the data such that each gene has a mean of 0. `sklearn.decomposition.PCA` will be used for dimensionality reduction. n_prin_comps Number of principal components used to embed the transcriptomes prior to k-nearest-neighbor graph construction. use_approx_neighbors Use approximate nearest neighbor method (annoy) for the KNN classifier. get_doublet_neighbor_parents If True, return (in .uns) the parent transcriptomes that generated the doublet neighbors of each observed transcriptome. This information can be used to infer the cell states that generated a given doublet state. n_neighbors Number of neighbors used to construct the KNN graph of observed transcriptomes and simulated doublets. If ``None``, this is automatically set to ``np.round(0.5 * np.sqrt(n_obs))``. threshold Doublet score threshold for calling a transcriptome a doublet. If `None`, this is set automatically by looking for the minimum between the two modes of the `doublet_scores_sim_` histogram. It is best practice to check the threshold visually using the `doublet_scores_sim_` histogram and/or based on co-localization of predicted doublets in a 2-D embedding. verbose If True, print progress updates. copy If ``True``, return a copy of the input ``adata`` with Scrublet results added. Otherwise, Scrublet results are added in place. random_state Initial state for doublet simulation and nearest neighbors. Returns ------- adata : anndata.AnnData if ``copy=True`` it returns or else adds fields to ``adata``. Those fields: ``.obs['doublet_score']`` Doublet scores for each observed transcriptome ``.obs['predicted_doublets']`` Boolean indicating predicted doublet status ``.uns['scrublet']['doublet_scores_sim']`` Doublet scores for each simulated doublet transcriptome ``.uns['scrublet']['doublet_parents']`` Pairs of ``.obs_names`` used to generate each simulated doublet transcriptome ``.uns['scrublet']['parameters']`` Dictionary of Scrublet parameters See also -------- :func:`~scanpy.external.pp.scrublet_simulate_doublets`: Run Scrublet's doublet simulation separately for advanced usage. :func:`~scanpy.external.pl.scrublet_score_distribution`: Plot histogram of doublet scores for observed transcriptomes and simulated doublets. rNLPlease install scrublet: `pip install scrublet` or `conda install scrublet`.zRunning Scrubletcs|dkrtj|ddtj|dd|j|jd<t|tj|dd}t||dd|j df}t |d d}rt|t|tj|d d tj|d d t ||   d }|j |j d d S)Nr)Z min_cells)Z min_genesrawT)r Zhighly_variable)layerrrg.A)Z target_sum) adata_obsrrrrrrrrrrrr!rscrublet)obsuns)ppZ filter_genesZ filter_cellsXr ZlayersZnormalize_totallog1pZhighly_variable_genesvarscrublet_simulate_doublets_scrublet_call_doubletsr(r))Zad_obsZad_simZlogged)rrrrrrrrr!rrrrrrU/home/sam/Atlas/atlas_env/lib/python3.8/site-packages/scanpy/external/pp/_scrublet.py _run_scrubletsH    zscrublet.._run_scrubletzA`batch_key` must be a column of .obs in the input annData object.cs&g|]}j|kfqSr0r().0batch)r2r&rrr0r1 szscrublet..cSsg|] }|dqSr3r0r4scrubr0r0r1r6sr'cSsg|] }|dqS)r)r0r7r0r0r1r6sbatchesZ batched_byr( doublet_scorepredicted_doubletr)z Scrublet finished)time)N)r' ImportErrorr logginfor(keys ValueErrornpuniquepdconcatlocZ obs_namesvaluesr)dictzip)rrrrrrrrrrrrrrrrrr r!slstartr9ZscrubbedZ scrubbed_obsr0)r2r&rrrrrrrrrrr!rrrrrrr1r' sH   *=    r')r&rrrrrrrrrrrr!rr"cCsz ddl}Wntk r(tdYnX|dkrNttdt|jd}|j|j|||| d}t |j|_ t |j|_ |j d|_|r|r||n|r||n|r|||rtdt|j tjrt|j |_ t|j tjrt|j |_ |j|||jdntd |j|||jd|j|| | d |j| | d |j|jd <|j|j d||j d i di dd|| dd|jd <t!|dr|j"|jd d<|j#|jd<n d|jd<| r|j$|jd d<|S)uQ Core function for predicting doublets using Scrublet [Wolock19]_. Predict cell doublets using a nearest-neighbor classifier of observed transcriptomes and simulated doublets. This is a wrapper around the core functions of `Scrublet `__ to allow for flexibility in applying Scanpy filtering operations upstream. Unless you know what you're doing you should use the main scrublet() function. .. note:: More information and bug reports `here `__. Parameters ---------- adata_obs The annotated data matrix of shape ``n_obs`` × ``n_vars``. Rows correspond to cells and columns to genes. Should be normalised with scanpy.pp.normalize_total() and filtered to include only highly variable genes. adata_sim Anndata object generated by sc.external.pp.scrublet_simulate_doublets(), with same number of vars as adata_obs. This should have been built from adata_obs after filtering genes and cells and selcting highly-variable genes. n_neighbors Number of neighbors used to construct the KNN graph of observed transcriptomes and simulated doublets. If ``None``, this is automatically set to ``np.round(0.5 * np.sqrt(n_obs))``. expected_doublet_rate The estimated doublet rate for the experiment. stdev_doublet_rate Uncertainty in the expected doublet rate. mean_center If True, center the data such that each gene has a mean of 0. `sklearn.decomposition.PCA` will be used for dimensionality reduction. normalize_variance If True, normalize the data such that each gene has a variance of 1. `sklearn.decomposition.TruncatedSVD` will be used for dimensionality reduction, unless `mean_center` is True. n_prin_comps Number of principal components used to embed the transcriptomes prior to k-nearest-neighbor graph construction. use_approx_neighbors Use approximate nearest neighbor method (annoy) for the KNN classifier. knn_dist_metric Distance metric used when finding nearest neighbors. For list of valid values, see the documentation for annoy (if `use_approx_neighbors` is True) or sklearn.neighbors.NearestNeighbors (if `use_approx_neighbors` is False). get_doublet_neighbor_parents If True, return the parent transcriptomes that generated the doublet neighbors of each observed transcriptome. This information can be used to infer the cell states that generated a given doublet state. threshold Doublet score threshold for calling a transcriptome a doublet. If `None`, this is set automatically by looking for the minimum between the two modes of the `doublet_scores_sim_` histogram. It is best practice to check the threshold visually using the `doublet_scores_sim_` histogram and/or based on co-localization of predicted doublets in a 2-D embedding. random_state Initial state for doublet simulation and nearest neighbors. verbose If True, print progress updates. Returns ------- adata : anndata.AnnData if ``copy=True`` it returns or else adds fields to ``adata``: ``.obs['doublet_score']`` Doublet scores for each observed transcriptome ``.obs['predicted_doublets']`` Boolean indicating predicted doublet status ``.uns['scrublet']['doublet_scores_sim']`` Doublet scores for each simulated doublet transcriptome ``.uns['scrublet']['doublet_parents']`` Pairs of ``.obs_names`` used to generate each simulated doublet transcriptome ``.uns['scrublet']['parameters']`` Dictionary of Scrublet parameters rNr#g?)rrrr!doublet_parentsz%Embedding transcriptomes using PCA...)rr!z/Embedding transcriptomes using Truncated SVD...)rZdistance_metricr)rrr:r' parametersr)rrrr!)Zdoublet_scores_simrLrM threshold_rr;FZdoublet_neighbor_parents)%r'r=introundrBsqrtshapeScrubletr+rZ csc_matrixZ _E_obs_normZ _E_sim_normobsmdoublet_parents_Zpipeline_zscoreZpipeline_mean_centerZpipeline_normalize_variancer>r? isinstancematrixZasarrayZ pipeline_pcar!Zpipeline_truncated_svdZcalculate_doublet_scoresZ call_doubletsZdoublet_scores_obs_r(Zdoublet_scores_sim_r)gethasattrrNZpredicted_doublets_Zdoublet_neighbor_parents_)r&rrrrrrrrrrrr!rrJr8r0r0r1r/si             r/)rrr random_seedr"c Csz ddl}Wntk r(tdYnXt||d}||}|j||dt|j}|j|jd<|j |j d<dd |ii|j d <|S) u Simulate doublets by adding the counts of random observed transcriptome pairs. Parameters ---------- adata The annotated data matrix of shape ``n_obs`` × ``n_vars``. Rows correspond to cells and columns to genes. Genes should have been filtered for expression and variability, and the object should contain raw expression of the same dimensions. layer Layer of adata where raw values are stored, or 'X' if values are in .X. sim_doublet_ratio Number of doublets to simulate relative to the number of observed transcriptomes. If `None`, self.sim_doublet_ratio is used. synthetic_doublet_umi_subsampling Rate for sampling UMIs when creating synthetic doublets. If 1.0, each doublet is created by simply adding the UMIs from two randomly sampled observed transcriptomes. For values less than 1, the UMI counts are added and then randomly sampled at the specified rate. Returns ------- adata : anndata.AnnData with simulated doublets in .X Adds fields to ``adata``: ``.obsm['scrublet']['doublet_parents']`` Pairs of ``.obs_names`` used to generate each simulated doublet transcriptome ``.uns['scrublet']['parameters']`` Dictionary of Scrublet parameters See also -------- :func:`~scanpy.external.pp.scrublet`: Main way of running Scrublet, runs preprocessing, doublet simulation (this function) and calling. :func:`~scanpy.external.pl.scrublet_score_distribution`: Plot histogram of doublet scores for observed transcriptomes and simulated doublets. rNr#)r%)rrZn_countsrLrMrr') r'r=rrSZsimulate_doubletsrZ_E_simZ_total_counts_simr(rUrTr)) rr%rrrZrJr+r8rr0r0r1r.s"/       r.)NNr r r r r TFTrTFNNTFr) Nr r TTrTr FNrT)Nr r r)ZanndatartypingrnumpyrBZpandasrDZscipyrrr>rr*rXrstrfloatboolrOr'r/r.r0r0r0r1s        ^