U md @sddlmZmZmZmZddlZddlZddl Z ddl m Z ddl mZddlmZddlmZmZdd d Zde eeeeeeeee dddZGdddeZdS))TupleOptionalSequenceListN)AnnData) natsorted)logging) NeighborsOnFlySymMatrixcCs`td|d}t||d}||j||d|j|jd<|j|jd<tjd|dd dS) Nz'computing Diffusion Maps using n_comps=z(=n_dcs) neighbors_key)n_comps random_state X_diffmapZ diffmap_evals finishedzyadded 'X_diffmap', diffmap coordinates (adata.obsm) 'diffmap_evals', eigenvalues of transition matrix (adata.uns)timedeep) logginfoDPTZcompute_transitionsZ compute_eigenZ eigen_basisobsmZ eigen_valuesuns)adatarrrstartdptrJ/home/sam/Atlas/atlas_env/lib/python3.8/site-packages/scanpy/tools/_dpt.py_diffmap s   r {Gz?TF)rn_dcs n_branchingsmin_group_sizeallow_kendall_tau_shiftrcopyreturnc Cs|r |n|}|dkrd}||jkr.tdd|jkrLd|jkrLtdd|jkrptdt||d t ||||||d }t d |}|d krt d |j dk r| |j |jd<|j |jd<|dkrl|tj|jdtt|jdd|jd<|j|jd<|j|jd<tj|jtd} t|jD]\} } | | | <qB| |jd<|j|jd<tj d|d|j dk rdnd|dkrdndd|r|SdS)a Infer progression of cells through geodesic distance along the graph [Haghverdi16]_ [Wolf19]_. Reconstruct the progression of a biological process from snapshot data. `Diffusion Pseudotime` has been introduced by [Haghverdi16]_ and implemented within Scanpy [Wolf18]_. Here, we use a further developed version, which is able to deal with disconnected graphs [Wolf19]_ and can be run in a `hierarchical` mode by setting the parameter `n_branchings>1`. We recommend, however, to only use :func:`~scanpy.tl.dpt` for computing pseudotime (`n_branchings=0`) and to detect branchings via :func:`~scanpy.tl.paga`. For pseudotime, you need to annotate your data with a root cell. For instance:: adata.uns['iroot'] = np.flatnonzero(adata.obs['cell_types'] == 'Stem')[0] This requires to run :func:`~scanpy.pp.neighbors`, first. In order to reproduce the original implementation of DPT, use `method=='gauss'` in this. Using the default `method=='umap'` only leads to minor quantitative differences, though. .. versionadded:: 1.1 :func:`~scanpy.tl.dpt` also requires to run :func:`~scanpy.tl.diffmap` first. As previously, :func:`~scanpy.tl.dpt` came with a default parameter of ``n_dcs=10`` but :func:`~scanpy.tl.diffmap` has a default parameter of ``n_comps=15``, you need to pass ``n_comps=10`` in :func:`~scanpy.tl.diffmap` in order to exactly reproduce previous :func:`~scanpy.tl.dpt` results. Parameters ---------- adata Annotated data matrix. n_dcs The number of diffusion components to use. n_branchings Number of branchings to detect. min_group_size During recursive splitting of branches ('dpt groups') for `n_branchings` > 1, do not consider groups that contain less than `min_group_size` data points. If a float, `min_group_size` refers to a fraction of the total number of data points. allow_kendall_tau_shift If a very small branch is detected upon splitting, shift away from maximum correlation in Kendall tau criterion of [Haghverdi16]_ to stabilize the splitting. neighbors_key If not specified, dpt looks .uns['neighbors'] for neighbors settings and .obsp['connectivities'], .obsp['distances'] for connectivities and distances respectively (default storage places for pp.neighbors). If specified, dpt looks .uns[neighbors_key] for neighbors settings and .obsp[.uns[neighbors_key]['connectivities_key']], .obsp[.uns[neighbors_key]['distances_key']] for connectivities and distances respectively. copy Copy instance before computation and return a copy. Otherwise, perform computation inplace and return `None`. Returns ------- Depending on `copy`, returns or updates `adata` with the following fields. If `n_branchings==0`, no field `dpt_groups` will be written. `dpt_pseudotime` : :class:`pandas.Series` (`adata.obs`, dtype `float`) Array of dim (number of samples) that stores the pseudotime of each cell, that is, the DPT distance with respect to the root cell. `dpt_groups` : :class:`pandas.Series` (`adata.obs`, dtype `category`) Array of dim (number of samples) that stores the subgroup id ('0', '1', ...) for each cell. The groups typically correspond to 'progenitor cells', 'undecided cells' or 'branches' of a process. Notes ----- The tool is similar to the R package `destiny` of [Angerer16]_. N neighborsz6You need to run `pp.neighbors` and `tl.diffmap` first.irootZxrootzNo root cell found. To compute pseudotime, pass the index or expression vector of a root cell, one of: adata.uns['iroot'] = root_cell_index adata.var['xroot'] = adata[root_cell_name, :].XrzpTrying to run `tl.dpt` without prior call of `tl.diffmap`. Falling back to `tl.diffmap` with default parameters.r )r#r%r$r&rz+computing Diffusion Pseudotime using n_dcs=z+ this uses a hierarchical implementationZdpt_pseudotimerU)values categoriesZ dpt_groupsZdpt_changepointsZ dpt_grouptipsZdtypeZ dpt_orderZdpt_order_indicesrzadded z0 'dpt_pseudotime', the pseudotime (adata.obs)ze 'dpt_groups', the branching subgroups of dpt (adata.obs) 'dpt_order', cell order (adata.obs)r)r'r ValueErrorvarrwarningrkeysr rrr*Z_set_pseudotime pseudotimeZobsbranchings_segmentspdZ Categorical segs_namesastypernparraysegs_names_unique changepoints segs_tipszerosZn_obsint enumerateindices) rr#r$r%r&rr'rrZ ordering_idcountidxrrrrsvW              rc sLeZdZdZd0fdd Zdd Zd d Zd d Zee e fdddZ ddZ ddZ ddZ eejeeje ejdddZd1ejejeeejeejeee eee e fdddZddZdd Zd!d"Zejejejdd#d$Ze dd%d&Ze e ed'd(d)Ze e ed*d+d,Zejejee e fd-d.d/ZZS)2rz+ Hierarchical Diffusion Pseudotime. Nr"rFcsVtj|||dd|_||_|dkr*|nt||jjd|_||_d|_ ||_ dS)N)r#r haghverdi16r+rF) super__init__flavorr$r@_adatashaper%Z passed_adatachoose_largest_segmentr&)selfrr#r%r$r&r __class__rrrGs z DPT.__init__cCs$||||dS)u Detect branchings and partition the data into corresponding segments. Detect all branchings up to `n_branchings`. Writes ------ segs : np.ndarray Array of dimension (number of segments) × (number of data points). Each row stores a mask array that defines a segment. segs_tips : np.ndarray Array of dimension (number of segments) × 2. Each row stores the indices of the two tip points of each segment. segs_names : np.ndarray Array of dimension (number of data points). Stores an integer label for each segment. N)detect_branchingspostprocess_segmentsset_segs_namesorder_pseudotime)rLrrrr6szDPT.branchings_segmentsc Cstd|jd|jdkrdndtj|jjdtd}|g}|jdk r`t |j |j}nt |j d}t |t |j |g}|g}gg}d g}gg}td |j d t |jD]^} ||||\} } | d krtd q td| dd| ||||||| | q||_||_||_tjjt|t|ftd|_tjjt|t|ftd|_t|D]\} } || |j| | f<qvt t|D]F} t t|D]2}|j |j| |f|j|| ff|j| |f<qq|j|_|j|_dS)a  Detect all branchings up to `n_branchings`. Writes Attributes ----------------- segs : np.ndarray List of integer index arrays. segs_tips : np.ndarray List of indices of the tips of segments. z detect z branchingr+r0srr/NTz* do not consider groups with less than z points for splittingz partitioning convergedz branching z: split group )rdebugr$r:arangerIrJr@r*argmax distances_dptr;r%rangeselect_segmentdetect_branchingsegsr>segs_undecidedspsparse lil_matrixlenfloatsegs_adjacency segs_connectsrAZtocsr)rLZ indices_allr\tip_0Ztips_allr>rdr]rcibranchisegtips3i seg_adjacencyjrrrrOs^      zDPT.detect_branchingscstjjdkddj}tdt|dD]}tjjdD]||kr>jj}fddjDjfddj D}fddt |D}t |d|d}t ||q>q*dS)Nrr+Zaxiscs.g|]&}|tjjd|fqSr)r:argminrXr>).0seg)rgrLrr \sz'DPT.check_adjacency..cs(g|] }tj|dfqSrm)r:rnrX)rotips)rprLrrrqascs"g|]\}}j||fqSr)rX)roZipointpoint)closest_points_other_segsrLrrrqes) r:sumrcA1rYmaxrJZtodenser\r>rAargsortprint)rLZn_edges_per_segZn_edgesZ neighbor_segsZclosest_points_in_segsZ distance_segsZ closest_segsr)rtrgrprLrcheck_adjacencyTs$     zDPT.check_adjacency)r(c stt|df}tj|jjdtdt|D]\}||ddkrLq0t|j t sl|j t }n |j }d}||rt t|D]`}||krt dD]J} |j ||d||| fd|j ||| ||| fkr| }qqfd d ||D} || d|| d} t| s4q0t| } |dk r| || 7} t| } | | dkr| | kr| | d<| || d8} n | || 8} t| | }| |d||d|df}|jrtn|}tt|jkrd |d |d tddndt|jkr*d}|||df<|||ddf<q0t|dddf}||dfdkrtdS||ddft}||fS)a Out of a list of line segments, choose segment that has the most distant second data point. Assume the distance matrix Ddiff is sorted according to seg_idcs. Compute all the distances. Returns ------- iseg Index identifying the position within the list of line segments. tips3 Positions of tips within chosen segment. rr/rTNrr+g?cs&g|]}t|kddqSrm)r:where)roZtipZ allindicesrprrrqsz&DPT.select_segment.. group z score z n_points  z (too small)r0)rTN)r:r?rarVrIrJr@rA isinstancerXr ix_restrictrYisfiniteanyrWappendrKrrUr%r9)rLr\r>r]Z scores_tipsrgDsegZthird_maximizerjsegZitiprrZdsegZ third_tipZ fourth_tiprhZscorerr}rrZusl           $ zDPT.select_segmentcCs\t|jD]0\}}tj|jjdtd}d||<||j|<q t|j|_t|j|_dS)z0Convert the format of the segment class members.rr/TN) rAr\r:r?rIrJboolr;r>)rLrgrpmaskrrrrPs  zDPT.postprocess_segmentscCsPtj|jjdtjd}g|_t|jD]\}}|||<|j|q(||_ dS)z9Return a single array that stores integer segment labels.rr/N) r:r?rIrJZint8r<rAr\rr8)rLr8rgrprrrrQs zDPT.set_segs_namesc Cs|jdk rbt|jD]L\}}|ddkrNt|j|}|j|||j|<qtd|dqt|j}|j|}t |j dt |dkd}|jdk r|j|}t|j D].\}}||} t|| } || | || <q||_ ||_dS)a Define indices that reflect segment and pseudotime order. Writes ------ indices : np.ndarray Index array of shape n, which stores an ordering of the data points with respect to increasing segment index and increasing pseudotime. changepoints : np.ndarray Index array of shape len(ssegs)-1, which stores the indices of points where the segment index changes, with respect to the ordering of indices. NrrTr~z is very smallr+)r*rAr>r:rxr5rrUr8rVsizediffr\rBr=) rLZitipsrrrBr8r=r5rgrpZ seg_sortedZ seg_indicesrrrrRs"    "  zDPT.order_pseudotime)r\r>rgrhc* sP|}t|jts(|jt||} n |j|} || ||} | \} } } }t| D]8\}}||| |<|| || |<t|||||<qX| | | | | | |fddt| D7}|fddt| D7}t | dkr| | dt | d}| |j dkr|fddt|D7}|fd dt|D7}|}tD]\}}d }t| D]\}}|kr||}|||}||kr8t ||||||<||}| || |}||||q|d7}qq|ttt ||t |7<||7<nd d l}|d dt|D7}|d dt|D7}gttt ||t |}D] }||}g}g}g}|D]|d }|||t|j|||f|d} ||t|j| |f||j|d|dfqt|}||}!|!|||<|||||<||}| || |||!|||!||qd}"|D](g}g}g}fddtt |D}#|#D]}|d }|||t|j|||f|d} ||t|j| |f||j|d|dfq.csg|]\}}|kr|qSrr)rorgZseg_tipsrrrrqEsr{Tr+rEcsg|] }gqSrrrori)rgrrrqPscsg|]\}}|kr|qSrr)rorgZ seg_connectsrrrrqQsrNcSsg|]}gqSrrrrrrrqpscSsg|]}gqSrrrrrrrqqsrTFcs g|]}|kr|kr|qSrr)ror)ksegprev_connecting_segmentsrrrqsr/)sourcez attaching new segment z at z cannot attach new segment z (would produce cycle)z continuez do not add another linkcSsg|]}dqS)Frrrrrrqs)rrXr r:rr_detect_branchingrAlistpopinsertrar'rHrYindexrZnetworkxrnr^r_r`rbZGraphZsingle_source_dijkstra_pathrrU)*rLr\r>rdr]rcrgrhrprresultssegs ssegs_tipsssegs_adjacencyssegs_connectsZiseg_newZseg_newZn_addZprev_connecting_pointsZjseg_cntrZiseg_cntposZconnection_to_isegZpos_2rDnxZ kseg_listZ distancesZclosest_points_in_jsegZclosest_points_in_ksegZreference_point_in_kZreference_point_in_jZkseg_minZdo_not_attach_ksegZ jseg_listZjseg_minZsegs_adjacency_sparserirjGZ paths_allr)rgrrrrr[s<                                 zDPT.detect_branching)rrrr(cs|jdkr|||}nB|jdkr0|||}n*|jdksD|jdkrR|||}ntdtjt||jdft d}t |D]\}}d|||<q~tj |dd d k}g}t |D].\}} d | |<| tj |jdtd| qg} t |D]`\} } tt| d kr"td t| d | t||| | } | || | gqtj |jdtd|}t|dkrV| |ggggg}t | D]b\} }|d}|t|||}||  ||| t|||| }|d |q|t||d|}|t|||}| ||gdgdgdgdd dgg}dnnt|dkr\tjdtd}| dd|d<| d d|d <| dd|d<tjdtd|d t||d|d d<|dt||d |dd<|dt||d|dd<|dt||d|dd<|dt||d |dd<|d t||d|d d<td}|ddf|ddf|d<|ddf|ddf|d <|ddf|ddf|d<t|fddtdD}fddtdD}nhdd gdgg}| dd}|d t|||d }|}|dt|||d}|g|gg}|| ||fS)as Detect branching on given segment. Call function __detect_branching three times for all three orderings of tips. Points that do not belong to the same segment in all three orderings are assigned to a fourth segment. The latter is, by Haghverdi et al. (2016) referred to as 'undecided cells'. Parameters ---------- Dseg Dchosen distance matrix restricted to segment. tips The three tip points. They form a 'triangle' that contains the data. Returns ------- ssegs List of segments obtained from splitting the single segment defined via the first two tip cells. ssegs_tips List of tips of segments in ssegs. ssegs_adjacency ? ssegs_connects ? trunk ? rEZ wolf17_triZ wolf17_biZ wolf17_bi_unzC`flavor` needs to be in {"haghverdi16", "wolf17_tri", "wolf17_bi"}.rr/Trlr+Fzdetected group with only z cellsrTr)rr)rr+)r+r)rr)rr)r+r)rr+cs0g|](}|krgnfddtdDqS)csg|]}|kr|qSrrrorkrrrrqns4DPT._detect_branching...rrYrrrrrqmsz)DPT._detect_branching..cs:g|]2}|kr|fgnfddtdDqS)cs g|]}|kr|fqSrrrZclosest_pointsrrrrqtsrrrrrrrrqqs)rH$_detect_branching_single_haghverdi16#_detect_branching_single_wolf17_tri"_detect_branching_single_wolf17_bir1r:r?rarJrrArurrVr@Z flatnonzerorr3rWrnrY)rLrrrZ seg_referencermasksrgrpZ nonuniquerrZinewsegZnewsegZ secondtipZundecided_cellsrZ newseg_tipsZreference_pointZ closest_cellreZtip_1rZ added_distZreference_point_in_0Zclosest_point_in_1Zreference_point_in_1Zclosest_point_in_0rrrrs)            zDPT._detect_branchingcCsLg}dddgdddgdddgg}t|D]\}}|||||q(|S)z"Detect branching on given segment.rr+r)rAr"_DPT__detect_branching_haghverdi16)rLrrrrZpsriprrrrsz(DPT._detect_branching_single_haghverdi16cCs||d}||d}||d}||k}||k}||k}tjd|jdftd} || d<|| d<tj| dddk} tjd|jdftd} || d<|| d<tj| dddk} tjd|jdftd} || d<|| d<tj| dddk} | | | g} | S)Nrr+rr/rl)r:r?rJrru)rLrrr dist_from_0 dist_from_1Z dist_from_2closer_to_0_than_to_1Zcloser_to_0_than_to_2Zcloser_to_1_than_to_2rZ segment_0Z segment_1Z segment_2rrrrrs(       z'DPT._detect_branching_single_wolf17_tricCs.||d}||d}||k}||g}|S)Nrr+r)rLrrrrrrrrrrrs    z&DPT._detect_branching_single_wolf17_bicCs|t||d}|||d|||d|}g}|dt|krh|jrhtdtd|}n|d}|d|S)a Detect branching on given segment. Compute point that maximizes kendall tau correlation of the sequences of distances to the second and the third tip, respectively, when 'moving away' from the first tip: tips[0]. 'Moving away' means moving in the direction of increasing distance from the first tip. Parameters ---------- Dseg Dchosen distance matrix restricted to segment. tips The three tip points. They form a 'triangle' that contains the data. Returns ------- Segments obtained from "splitting away the first tip cell". rr+rgffffff?zwshifting branching point away from maximal kendall-tau correlation (suppress this with `allow_kendall_tau_shift=False`)N)r:rxkendall_tau_splitrar&rr3r@)rLrrrZidcsimaxrrfrrrZ__detect_branching_haghverdi16sz"DPT.__detect_branching_haghverdi16cCsF|j|jkrtd|j|jkr,dkr8nntdddl}d}|j}tj||j|dtd}t|j}|j |d||d|d}|j ||d||dd} t |D]V\} } | ||| \} } || | | |}| | || | | }|}|} |||| <qt|}||}||}|dkrBtd |S) akReturn splitting index that maximizes correlation in the sequences. Compute difference in Kendall tau for all splitted sequences. For each splitting index i, compute the difference of the two correlation measures kendalltau(a[:i], b[:i]) and kendalltau(a[i:], b[i:]). Returns the splitting index that maximizes kendalltau(a[:i], b[:i]) - kendalltau(a[i:], b[i:]) Parameters ---------- a, b : np.ndarray One dimensional sequences. Returns ------- Splitting index according to above description. z"a and b need to have the same sizer+z)a and b need to be one-dimensional arraysrNr/g333333?z8 is root itself, never obtain significant correlation)rr1ndimscipyr:rVr@r?statsZ kendalltaurA_kendall_tau_diff_kendall_tau_add_kendall_tau_subtractrWrrU)rLabr^Z min_lengthnZ idx_rangeZ corr_coeffZpos_oldZneg_oldiiridiff_posdiff_negrnegZiimaxrZcorr_coeff_maxrrrrs0  ""   zDPT.kendall_tau_split)len_oldrtau_oldcCsd|dt|||S)apCompute Kendall tau delta. The new sequence has length len_old + 1. Parameters ---------- len_old The length of the old sequence, used to compute tau_old. diff_pos Difference between concordant and non-concordant pairs. tau_old Kendall rank correlation of the old sequence. @r+rb)rLrrrrrrr1szDPT._kendall_tau_add)rrrcCs"d|dt| |d|S)apCompute Kendall tau delta. The new sequence has length len_old - 1. Parameters ---------- len_old The length of the old sequence, used to compute tau_old. diff_neg Difference between concordant and non-concordant pairs. tau_old Kendall rank correlation of the old sequence. rrr+r)rLrrrrrrrAszDPT._kendall_tau_subtract)rrr(c CsFtj|d|jtd}d||d|||k<d||d|||k<tj|d|jtd}d||d|||k<d||d|||k<t||t}tj||djtd}d|||d||k<d|||d||k<tj||djtd}d|||d||k<d|||d||k<t||} || fS)aCompute difference in concordance of pairs in split sequences. Consider splitting a and b at index i. Parameters ---------- a ? b ? Returns ------- diff_pos Difference between concordant pairs for both subsequences. diff_neg Difference between non-concordant pairs for both subsequences. Nr/r+rT)r:r?rr@dotr9rb) rLrrriZa_posZb_posrZa_negZb_negrrrrrQs zDPT._kendall_tau_diff)Nr"rFN)N)__name__ __module__ __qualname____doc__rGr6rOrzrr@rZrPrQrRrr:Zndarrayr[rrrrrrrrbrrr __classcell__rrrMrrsZb!^  * ]    >@r)r Nr)r!rr"TNF)typingrrrrnumpyr:Zpandasr7rr^ZanndatarZnatsortrr0r rr)r r r r@rbrstrrrrrrrs4     #