U md @sdZddlmZmZddlmZmZmZmZm Z ddl Z ddl Z ddl Z ddlZddlmZddlZddlmZmZmZmZmZmZmZddlmZdd lmZdd lmZdd l m!Z!m"Z"dd l#m$Z%d ddddhZ&ddddddddhe&BZ'ddddddde"feee(feedee(ee(ee(e)ee(e)eedde!fed ddZdfee(efee(e)ee(ed!d"d#Z*ddd$d%d&Z+e,e j-d'd(d)Z.dd*d+d,Z/dgd-dd dd.ee(efee(e(e(ee)eee(efed/d0d1Z0d2d de"d fdd3eee(fed4e)e)eedde!fe)e(ed5d6d7Z1d2d de"fd8d3d9d:Z2d2d de"fd8d3d;d<Z3dhee(efeeed>eedee4d?d@dAZ5dieee(fe)ee(ee4e6e)e(dffdBdCdDZ7eee(fdEdFdGZ8djedHdIdJZ9ee(efe(dKdLdMZ:ee(e;ee fedNdOdPZe(ee)e)fdQdVdWZ?e(ee4e6e)e(dfdQdXdYZ@dZd[ZAdked\d]d^ZBe(ed_d`daZCdldbdcZDdmedHdddeZEdS)nzReading and Writing )PathPurePath)UnionDictOptionalTupleBinaryION)imread)AnnDataread_csv read_text read_excelread_mtx read_loomread_hdf)read)settings)Literal)Empty_empty)loggingcsvtsvtabdatatxtanndataxlsxh5h5admtxmtx.gzsoft.gzloomF)rzr+)gzipZlzf) filenamebackedsheetext delimiterfirst_column_names backup_urlcachecache_compressionreturnc Ks~t|}t|r2t|f||||||||d| St|} tj| dtj}|srtd| d|dt dt ||dS)a Read file and return :class:`~anndata.AnnData` object. To speed up reading, consider passing ``cache=True``, which creates an hdf5 cache file. Parameters ---------- filename If the filename has no file extension, it is interpreted as a key for generating a filename via ``sc.settings.writedir / (filename + sc.settings.file_format_data)``. This is the same behavior as in ``sc.read(filename, ...)``. backed If ``'r'``, load :class:`~anndata.AnnData` in ``backed`` mode instead of fully loading it into memory (`memory` mode). If you want to modify backed attributes of the AnnData object, you need to choose ``'r+'``. sheet Name of sheet/table in hdf5 or Excel file. ext Extension that indicates the file type. If ``None``, uses extension of filename. delimiter Delimiter that separates data within text file. If ``None``, will split at arbitrary number of white spaces, which is different from enforcing splitting at any single white space ``' '``. first_column_names Assume the first column stores row names. This is only necessary if these are not strings: strings in the first column are automatically assumed to be row names. backup_url Retrieve the file from an URL if not present on disk. cache If `False`, read from source, if `True`, read from fast 'h5ad' cache. cache_compression See the h5py :ref:`dataset_compression`. (Default: `settings.cache_compression`) kwargs Parameters passed to :func:`~anndata.read_loom`. Returns ------- An :class:`~anndata.AnnData` object )r(r)r*r+r,r-r.r/.zReading with filekey z failed, the inferred filename zx does not exist. If you intended to provide a filename, either use a filename ending on one of the available extensions z or pass the parameter `ext`.r() ris_valid_filename_readstrrwritedirfile_format_dataexists ValueError avail_exts read_h5ad) r'r(r)r*r+r,r-r.r/kwargsZfilekeyr=I/home/sam/Atlas/atlas_env/lib/python3.8/site-packages/scanpy/readwrite.pyr6s.8  rT)r'genomegex_onlyr-r0c Cstd|}t||d}|s0td|tt|d}d|k}W5QRX|rt||d}|r||jdj krt d|d |d t |jd d |d d |jd|kf}|r|d d |jd dkf}|j r|}nt|||d}|S)a Read 10x-Genomics-formatted hdf5 file. Parameters ---------- filename Path to a 10x hdf5 file. genome Filter expression to genes within this genome. For legacy 10x h5 files, this must be provided if the data contains more than one genome. gex_only Only keep 'Gene Expression' data and ignore other feature types, e.g. 'Antibody Capture', 'CRISPR Guide Capture', or 'Custom' backup_url Retrieve the file from an URL if not present on disk. Returns ------- Annotated data matrix, where observations/cells are named by their barcode and variables/genes by gene name. Stores the following information: :attr:`~anndata.AnnData.X` The data matrix is stored :attr:`~anndata.AnnData.obs_names` Cell names :attr:`~anndata.AnnData.var_names` Gene names :attr:`~anndata.AnnData.var`\ `['gene_ids']` Gene IDs :attr:`~anndata.AnnData.var`\ `['feature_types']` Feature types reading r-... did not find original file r%z/matrixstartr?z-Could not find data corresponding to genome '' in ''. Available genomes are: r1N feature_typesGene Expressionr?rE)logginfo$_check_datafile_present_and_downloaddebugh5pyFiler5_read_v3_10x_h5varvaluesr9listuniqueZis_viewcopy_read_legacy_10x_h5) r'r?r@r-rE is_presentfZv3adatar=r=r> read_10x_h5s(&  $ r[rJc Cs|tt|d`}z8t|}|sRt|dkrHtd|d||d}n"||krttd|d|d|i}t|||dd lm }|d \}}|d } |d j t d kr|d  d } |d | dd<|| |d|df||fd} t | t|dtdt|dt|dtdd} tjd|d| WW5QRStk rltdYnXW5QRXdS)zA Read hdf5 file from Cell Ranger v2 or earlier versions. r%r'z' contains more than one genome. For legacy 10x h5 files you must specify the genome if more than one is present. Available genomes are: rzCould not find genome 'rFrG csr_matrixshaperint32float32Nindicesindptrr_barcodes obs_names gene_namesgenes) var_namesgene_idsobsrRtime.File is missing one or more required datasets.)rOrPr5rTkeyslenr9_collect_datasets scipy.sparser^dtypenpviewr dictastyperKrLKeyError Exception) r'r?rErYchildrendsetsr^MNrmatrixrZr=r=r>rWsJ       rW)r~groupcCs:|D],\}}t|tjr*|d||<qt||qdS)Nr=)items isinstancerOZDatasetrt)r~rkvr=r=r>rts rtrDc Cs6tt|d}zi}t||dddlm}|d\}}|d}|djtdkrz|dd}|d|d d <|||d |d f||fd }t |t |d  tdt |d t|d t|d t|d tdd} t j d|d| WW5QRStk r&tdYnXW5QRXd S)z? Read hdf5 file from Cell Ranger v3 or later versions. r%rrr]r_rr`raNrbrcrdrerfnameidZ feature_typer?)rjrkrHr?rlrnrorq)rOrPr5rtrur^rvrwrxr ryrzrKrLr{r|) r'rErYr~r^rrrrrZr=r=r>rQs8       rQzfiltered_feature_bc_matrix.h5) count_file library_id load_imagessource_image_path)pathr?rrrrr0c sRt|}t|||d}t|jd<ddlm}|||ddtjW5QRX|dkrrtddd }t|jd|<|rNt|d |d |d |d d}| D]F st fdddDrt ddqtddqt|jd|d<dD]Z} z,tt|| d|jd|d| <Wn&tk rftd| dYnXqt|d|jd|d<fdddD|jd|d<tj|d dd!} d"d#d$d%d&d'g| _| d"| _|jj| d(d)|_|jd'd&g|jd<|jjd"d'd&gd*d+|dk rNtt|}t||jd|dd,<|S)-a Read 10x-Genomics-formatted visum dataset. In addition to reading regular 10x output, this looks for the `spatial` folder and loads images, coordinates and scale factors. Based on the `Space Ranger output docs`_. See :func:`~scanpy.pl.spatial` for a compatible plotting function. .. _Space Ranger output docs: https://support.10xgenomics.com/spatial-gene-expression/software/pipelines/latest/output/overview Parameters ---------- path Path to directory for visium datafiles. genome Filter expression to genes within this genome. count_file Which file in the passed directory to use as the count file. Typically would be one of: 'filtered_feature_bc_matrix.h5' or 'raw_feature_bc_matrix.h5'. library_id Identifier for the visium library. Can be modified when concatenating multiple adata objects. source_image_path Path to the high-resolution tissue image. Path will be included in `.uns["spatial"][library_id]["metadata"]["source_image_path"]`. Returns ------- Annotated data matrix, where observations/cells are named by their barcode and variables/genes by gene name. Stores the following information: :attr:`~anndata.AnnData.X` The data matrix is stored :attr:`~anndata.AnnData.obs_names` Cell names :attr:`~anndata.AnnData.var_names` Gene names :attr:`~anndata.AnnData.var`\ `['gene_ids']` Gene IDs :attr:`~anndata.AnnData.var`\ `['feature_types']` Feature types :attr:`~anndata.AnnData.uns`\ `['spatial']` Dict of spaceranger output files with 'library_id' as key :attr:`~anndata.AnnData.uns`\ `['spatial'][library_id]['images']` Dict of images (`'hires'` and `'lowres'`) :attr:`~anndata.AnnData.uns`\ `['spatial'][library_id]['scalefactors']` Scale factors for the spots :attr:`~anndata.AnnData.uns`\ `['spatial'][library_id]['metadata']` Files metadata: 'chemistry_description', 'software_version', 'source_image_path' :attr:`~anndata.AnnData.obsm`\ `['spatial']` Spatial spot coordinates, usable as `basis` by :func:`~scanpy.pl.embedding`. )r?Zspatialr)rPr%modeNZ library_idsutf-8z!spatial/tissue_positions_list.csvzspatial/scalefactors_json.jsonzspatial/tissue_hires_image.pngzspatial/tissue_lowres_image.png)tissue_positions_filescalefactors_json_file hires_image lowres_imagec3s|]}|tkVqdS)N)r5.0x)rYr=r> szread_visium..)rrz6You seem to be missing an image file. Could not find 'z'.zCould not find 'r\Zimages)ZhiresZlowresZ_imagez_image'rZ scalefactorscs:i|]2}|kr|t|tr.t|dn|qS)r)rbytesr5rr)attrsr=r> szread_visium..)Zchemistry_descriptionZsoftware_versionmetadatarheaderZbarcodeZ in_tissueZ array_rowZ array_colZpxl_col_in_fullresZpxl_row_in_fullresleft)howT)columnsZinplacer)rr[ryZunsrOrPrr5poprSr8anyrKwarningOSErrorr r|jsonloads read_bytespdr rindexrmjoinZto_numpyZobsmZdropresolve) rr?rrrrrZrPfilesresZ positionsr=)rrYr> read_visium)sx>        r gene_symbols)prefix)rrk)rrj make_uniquer.r/r@rr0c Cst|}|dkrdn|}||d}|r2tnt}|t||||||d} |sV|sZ| Sttdd| jd} | dd| fSdS)a Read 10x-Genomics-formatted mtx directory. Parameters ---------- path Path to directory for `.mtx` and `.tsv` files, e.g. './filtered_gene_bc_matrices/hg19/'. var_names The variables index. make_unique Whether to make the variables index unique by appending '-1', '-2' etc. or not. cache If `False`, read from source, if `True`, read from fast 'h5ad' cache. cache_compression See the h5py :ref:`dataset_compression`. (Default: `settings.cache_compression`) gex_only Only keep 'Gene Expression' data and ignore other feature types, e.g. 'Antibody Capture', 'CRISPR Guide Capture', or 'Custom' prefix Any prefix before `matrix.mtx`, `genes.tsv` and `barcodes.tsv`. For instance, if the files are named `patientA_matrix.mtx`, `patientA_genes.tsv` and `patientA_barcodes.tsv` the prefix is `patientA_`. (Default: no prefix) Returns ------- An :class:`~anndata.AnnData` object Nrn genes.tsv)rjrr.r/rcSs|dkS)NrIr=)rr=r=r>zread_10x_mtx..rH) ris_file_read_legacy_10x_mtx_read_v3_10x_mtxr5rTmaprRrV) rrjrr.r/r@rZgenefile_existsrrZZgex_rowsr=r=r> read_10x_mtxs$) rrncCst|}t||d||dj}tj||dddd}|dkr||dj}|rdtjt |}||_ |d j|j d <n.|d kr|d j|_ |dj|j d<nt d tj||d dd d j|_ |S)zF Read mex from output from Cell Ranger v2 or earlier versions z matrix.mtxr.r/rN rseprrrrk4`var_names` needs to be 'gene_symbols' or 'gene_ids'z barcodes.tsvrrrTrr rSrutilsZmake_index_uniqueIndexrjrRr9rgrrjrr.r/rrZrir=r=r>rs&     rcCst|}t||d||dj}tj||dddd}|dkr||dj}|rdtjt |}||_ |d j|j d <n.|d kr|d j|_ |dj|j d<nt d |d j|j d <tj||dddd j|_ |S)zD Read mtx from output from Cell Ranger v3 or later versions z matrix.mtx.gzrzfeatures.tsv.gzNrrrrrrkrrHzbarcodes.tsv.gzrrrr=r=r>rs,    rr&)rrrZnpz)r'rZr* compressioncompression_optscCst|}t|r@|}t|dd}|dkr.|}q`||kr`tdn |}|dkrRtjn|}t||}|dkrt||n|j|||ddS)a Write :class:`~anndata.AnnData` objects to file. Parameters ---------- filename If the filename has no file extension, it is interpreted as a key for generating a filename via `sc.settings.writedir / (filename + sc.settings.file_format_data)`. This is the same behavior as in :func:`~scanpy.read`. adata Annotated data matrix. ext File extension from wich to infer file format. If `None`, defaults to `sc.settings.file_format_data`. compression See http://docs.h5py.org/en/latest/high/dataset.html. compression_opts See http://docs.h5py.org/en/latest/high/dataset.html. T return_extNzxIt suffices to provide the file type by providing a proper extension to the filename.One of "txt", "csv", "h5" or "npz".r)rr)rr3r9rr7_get_filename_from_keyZ write_csvswrite)r'rZr*rrZext_keyr=r=r>rBs(   r)r'asheaderr0cCst|}ddlm}|g}t|D]^}d|kr$|r>|dr$|drT|ddn|}|d\}}|}|}t|||<q$|S)a Read parameter dictionary from text file. Assumes that parameters are specified in the format:: par1 = value1 par2 = value2 Comments that start with '#' are allowed. Parameters ---------- filename Filename of data file. asheader Read the dictionary from the header (comment section) of a file. Returns ------- Dictionary that stores parameters. r) OrderedDict=#rN)r5 collectionsropen startswithsplitstripconvert_string)r'rrparamslinervalr=r=r> read_paramszs  r)rc Ost|}|js |jjddt|dkr8|d|d<|dZ}|D]J\}}|dk rn|d|d|D]\}}||d |d qvqLW5QRXdS) zc Write parameters to file, so that it's readable by read_params. Uses INI file format. TparentsrrNw[z] z =  )rparentis_dirmkdirrsrrr)rargsmapsrYrrrrr=r=r> write_paramss    r)r'c KsB|dk r |tkr tdtn t|dd}t||d} | sLtd||dkr|dkrht||dStd|d |t||Stj t | d |d } | j d kr| d } |r| rtd| t| S| std|d td||s| std|dks(|dkrH|dkrtd tjd!|tkrtj}| js0| jjdd"| j| |d#| S)$Nz0Please provide one of the available extensions. TrrBrC>r rr2zreading sheet z from file r1z.h5ad>.gz.bz2rnz... reading from cache file zDid not find file rAzlThis might be very slow. Consider passing `cache=True`, which enables much faster reading from a cache file.rZxlsz5Provide `sheet` parameter when reading '.xlsx' files.>r"r!r)r,>rrrrrzA... assuming '.data' means tab or white-space separated text filez'change this by passing `ext` to sc.readr#r$r'zUnknown extension z... writing an z( cache file to speedup reading next timer)r)r:r9r3rMrKrNr;rrZcachedir_slugifyreplacesuffix with_suffixrrLFileNotFoundErrorhintr rr r _read_softgzrr7rr/rrrr)r'r(r)r*r+r,r-r.r/Zsuppress_cache_warningr<rXZ path_cacherZr=r=r>r4sv                         r4)rr0cCst|tst|}t|j}|ddkr4|dn4t|ddkrh|ddddkrh|dd|d<d|}d|kst||dddrt||S) zMake a path into a filename.r/rNz:\-:) rrrTpartsrrsrAssertionErrorr)rrr'r=r=r>r s    $ r)r'r0c s|ddl}|j|dd"}i|D]p}|dr6qq"|drT|dd}q"|d r"|ddd }d d |D}|D] }||<qq"|d dd tD}fdd |Dfdd D}gg} } |D]J}|drq6|d fdd |D} | | | dqW5QRXt | j } t j d|id} t j | d} t | | | | jdS)a- Read a SOFT format data file. The SOFT format is documented here http://www.ncbi.nlm.nih.gov/geo/info/soft2.html. Notes ----- The function is based on a script by Kerby Shedden. http://dept.stat.lsa.umich.edu/~kshedden/Python-Workshop/gene_expression_comparison.html rNrtrz!dataset_table_beginz!subset_descriptionrrz!subset_sample_id,cSsg|] }|qSr=)rrr=r=r> 5sz _read_softgz..rcSsg|]\}}|dr|qS)ZGSM)r)rirr=r=r>r;s csg|] }|qSr=r=rr) sample_namesr=r>r=scsg|] }|qSr=r=r) samples_infor=r>r?sz!dataset_table_endcsg|]}t|qSr=)floatr)Vr=r>rKsgroups)r)XrmrRrv)r&rrrrreadline enumerateappendrwarrayrrZ DataFramer rv)r'r&filerZsubset_descriptionZ subset_idsrrbrrhrrrmrRr=)rrrr>rs:           r)stringr0cCs*zt|WdStk r$YdSXdS)zCheck whether string is float. See also -------- http://stackoverflow.com/questions/736043/checking-if-a-string-can-be-converted-to-float-in-python TFN)rr9rr=r=r>is_float[s rcCs*zt|WdStk r$YdSXdS)z Check whether string is integer.TFN)intr9rr=r=r>is_intis r cCs |dkr dS|dkrdSdSdS)z Check whether string is boolean.True)TTFalse)TF)FFNr=rr=r=r> convert_boolrs r cCsLt|rt|St|r t|St|dr8t|dS|dkrDdS|SdS)z%Convert string to int, float or bool.rrNoneN)r r rrr rr=r=r>r|s  rc Csjddl}dd|D}g}|D]>}z"|}|D]}||jq4Wq"|jk r^Yq"Xq"t|S)z-Get files used by processes with name scanpy.rNcss|]}|dkr|VqdS)ZscanpyN)r)rprocr=r=r>rs z!get_used_files..)psutilZ process_iterZ open_filesrrZ NoSuchProcessset)rZloop_over_scanpy_processes filenamesrflistntr=r=r>get_used_filessr)r0cCs&|dkrtjn|}tj|d|S)Nr1)rr7r6)rr*r=r=r>rsr)urlrc Cszddl}ddlm}Wn tk r8ddlm}YnXddlm}m}ddlm}d}d}z||ddid} z || } WnJ|k rt d dd l m } dd l m} || | | d d } YnX| } | dd}|dddd|dkr|nt|dZ}|dD}| |}|rN|||d7}|t|| |}qW5QRXW5QRXW5QRXWn.ttfk r|r|YnXdS)Nr)tqdm)urlopenRequest)URLErrori z User-agentz scanpy-user)headerszFFailed to open the url with default certificates, trying with certifi.)where)create_default_context)cafile)contextzcontent-lengthBTri)unitZ unit_scaleZminitersZ unit_divisortotalwb) ipywidgetsZ tqdm.autor ImportErrorurllib.requestrr urllib.errorrrKrcertifirsslrrLgetr rrrupdatersKeyboardInterruptr|runlink)rrr$rrrr blocksizeblocknumreqZopen_urlrrrespr"trYblockr=r=r> _downloadsV      0 r4cCslt|}|rdS|dkr dStd|d|js^td|jd|jjddt||dS) z6Check whether the file is present, otherwise download.TNFztry downloading from url z0 ... this may take a while but only happens oncezcreating directory z/ for saving datar)rrrKrLrrrr4)rr-r=r=r>rMs   rMcCs|j}t|dkr>td|d|ddd|dd}t|dkr|dddtkr|dddd kr|r|dddSd S|r|d ddtkr|r|d ddSd Sd |d kr|rdSd Sd |dkr|rdSd S|sdSt|dtddS)z)Check whether the argument is a filename.rz,Your filename has more than two extensions: z!. Only considering the two last: Nr1rr)gzbz2Trnz.soft.gzr#z.mtx.gzr"FzU does not end on a valid extension. Please, provide one of the available extensions. z< Text files with .gz and .bz2 extensions are also supported.)suffixesrsrKr text_extsr:rr9)r'rr*r=r=r>r3s*  4  r3)NTN)N)Nr&N)F) NNNNNNFNF)N)N)F)F__doc__pathlibrrtypingrrrrrrOrnumpyrwZpandasrZmatplotlib.imager rr r r r rrrrr;Z _settingsrZ_compatr_utilsrrrnrrKr:r:r5boolr[rWryGrouprtrQrrrrr rrrrr4rrrrr r rrrr4rMr3r=r=r=r>s. $        V  >5&    @ $ (    9  ' T?  3