U md]- @sddlmZmZmZmZddlZddlZddlm Z ddl m Z ddl mZddlmZddlmZejeeeejd d d Zejejeeejejejejfd d dZdeeeeeeeeejdfdddZdejejejeeeeeeejejfd ddZddZddZdS)) CollectionTupleOptionalUnionN)linalg)issparse)AnnData)logging)sanitize_anndata)model batch_key batch_levelsreturnc s0ddl}|jd||dd}|j|gdd}|djjtd |j dd fd d |jjD}|rd dd|D}|jd|||dd}t j ||fdd}td t |dtdd |ddk r,td t dtdd dD]}||||<q|S)a( Computes a simple design matrix. Parameters -------- model Contains the batch annotation batch_key Name of the batch column batch_levels Levels of the batch annotation Returns -------- The design matrix for the regression problem rNz%~ 0 + C(Q('{}'), levels=batch_levels)Z dataframe) return_typeZaxisnumberFound z batches csg|]}|kr|qSr).0cZnumerical_covariatesrU/home/sam/Atlas/atlas_env/lib/python3.8/site-packages/scanpy/preprocessing/_combat.py +sz"_design_matrix..z + css|]}d|VqdS)zQ('{}')N)format)rxrrr .sz!_design_matrix..z~ 0 + {}z categorical variables: z,  z numerical variables:)patsyZdmatrixrZdropZ select_dtypescolumnsvalueslogginfoshapejoinpdconcatlen) r r rr designZ other_colsZcol_reprZ factor_matrixZnCrrr_design_matrix s4 r+)r datar rc Cs||j}t|\}}t|}tdd|D}tt|}t |||} t t t t | j | | j |j } t ||j | d|ddf} |t | | j d} t | tt|dft|} t| dkdkrtdt| dkdt | j t| dftdt|f} t| }d|ddd|f<| t || j 7} t| dkd|| t t| tdt|f}tj||j|jd }|| | | fS) a Standardizes the data per gene. The aim here is to make mean and variance be comparable across batches. Parameters -------- model Contains the batch annotation data Contains the Data batch_key Name of the batch column in the model matrix Returns -------- s_data Standardized Data design Batch assignment as one-hot encodings var_pooled Pooled variance per gene stand_mean Gene-wise mean cSsg|] }t|qSrr)rvrrrrbsz%_standardize_data..Nr rrrz genes with zero variance.)indexr!)groupbygroupsitemszipr)nparrayfloatsumr+dotlainvTonesintprintreshapecopywheresqrtr' DataFramer0r!)r r,r Z batch_itemsr batch_infon_batch n_batchesn_arrayr*ZB_hatZ grand_mean var_pooled stand_meantmps_datarrr_standardize_dataAs4   (""&rMbatchT)adatakey covariatesinplacerc$ Csn||krtd||dk rt||}t|r`t||}td|||krptdt|tt |krtdt |j r|j j j }n|j j }tj||j|jd}t||j|g|r|ng}||j} t| } tdd| D} tt| } td t|||\} }}}td ||jd| }t|j ||j | j j}g}t| D]*\}}| | j!dd|fj"d d qz|j#d d }|j"d d }t$t%t&|}t$t%t'|}td gg}}t| D]`\}}t(| j!dd|fj||||j||||||||\}}| || |qtd| }t|}t|}t| D]\}}t)||ddf} | *t| d f} t+| t,d | |f}!t|j!dd|ft+|j!||j }"|"|!|j!dd|f<qt)|*t|d f}#|t+|#t,d t-| f|}|r`|j.|_ n |j.SdS)a ComBat function for batch effect correction [Johnson07]_ [Leek12]_ [Pedersen12]_. Corrects for batch effects by fitting linear models, gains statistical power via an EB framework where information is borrowed across genes. This uses the implementation `combat.py`_ [Pedersen12]_. .. _combat.py: https://github.com/brentp/combat.py Parameters ---------- adata Annotated data matrix key Key to a categorical annotation from :attr:`~anndata.AnnData.obs` that will be used for batch effect removal. covariates Additional covariates besides the batch variable such as adjustment variables or biological condition. This parameter refers to the design matrix `X` in Equation 2.1 in [Johnson07]_ and to the `mod` argument in the original combat function in the sva R package. Note that not including covariates may introduce bias or lead to the removal of biological signal in unbalanced designs. inplace Whether to replace adata.X or to return the corrected data Returns ------- Depending on the value of `inplace`, either returns the corrected matrix or or modifies `adata.X`. z(Could not find the key {!r} in adata.obsNz1Could not find the covariate(s) {!r} in adata.obsz'Batch key and covariates cannot overlapzCovariates must be unique)r,r0r!cSsg|] }t|qSrr-r.rrrrszcombat..z!Standardizing Data across genes. z%Fitting L/S model and finding priors rrzFinding parametric adjustments zAdjusting data )/Zobs_keys ValueErrorrr5isinanyr6tolistr)setrXAr<r'rDZ var_namesZ obs_namesr Zobsr1indicesr"r7r8r#r$rMr!r:r; enumerateappendZilocvarmeanlistmap_aprior_bprior_it_solrCr@r9r=r>Z transpose)$rOrPrQrRZ cov_existZ missing_covrXr,r rErFrGrHrLr*rIrJZ batch_designZ gamma_hat delta_hatiZ batch_idxsZ gamma_bart2Za_priorZb_priorZ gamma_starZ delta_stargammadeltaZ bayesdatajZdsqdenomZnumerZvpsqrrrcombats(       $        "rk-C6?) rLg_hatd_hatg_barrfabconvrcCsdt|jdd}|} |} d} d} | } | }| |kr|||| |||| } || | jddftd|jdf}|d}|jdd}d|||d|d}tt| | | t|| | } | } |} | d} q6| |fS)a= Iteratively compute the conditional posterior means for gamma and delta. gamma is an estimator for the additive batch effect, deltat is an estimator for the multiplicative batch effect. We use an EB framework to estimate these two. Analytical expressions exist for both parameters, which however depend on each other. We therefore iteratively evalutate these two expressions until convergence is reached. Parameters -------- s_data Contains the standardized Data g_hat Initial guess for gamma d_hat Initial guess for delta g_bar, t_2, a, b Hyperparameters conv: float, optional (default: `0.0001`) convergence criterium Returns: -------- gamma estimated value for gamma delta estimated value for delta rrrr g?g@g?) r5isnanr8rAr@r%r=maxabs)rLrmrnrorfrprqrrnZg_oldZd_oldZchangecountZg_newZd_newZsum2rrrrcs.'    rccCs$|}|}d||d|S)Nr r^r]rdms2rrrra]sracCs$|}|}|||d|S)Nrxryrrrrbcsrb)rNNT)rl)typingrrrrZpandasr'numpyr5rr:Z scipy.sparserZanndatarr r#_utilsr rDstrr+ZndarrayrMboolrkr7rcrarbrrrrsT      5 D   F