> For the complete documentation index, see [llms.txt](https://academy.dnanexus.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://academy.dnanexus.com/public-datasets-on-the-dnanexus-platform/target-validation/the-lincs-l1000-dataset.md).
# The LINCS L1000 dataset
## Necessary Disclaimers and Legal
The user is responsible for reviewing and complying with the license requirements of the software, notebooks, and data referenced in this documentation.
Users are responsible for the costs associated with analyzing the LINCS L1000 dataset and its storage in their project spaces.
Instance type availability and pricing are subject to the contract between the user or the user’s organization and DNAnexus.
## Citations for LINCS L1000 dataset
The primary publication describing the L1000 platform and the Connectivity Map resource is: [The Library of Integrated Network-Based Cellular Signatures NIH Program: System-Level Cataloging of Human Cells Response to Perturbations](https://www.cell.com/cell-systems/fulltext/S2405-4712%2817%2930490-8) and [A Next Generation Connectivity Map: L1000 Platform and the First 1,000,000 Profiles](https://www.cell.com/cell/fulltext/S0092-8674\(17\)31309-0?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS0092867417313090%3Fshowall%3Dtrue)
More publications related to LINCS L1000 are listed in the[ LINCS Data Portal](https://lincsportal.ccs.miami.edu/signatures) and[ CLUE.io](https://clue.io).
More information about the LINCS L1000 project can be found in the[ LINCS Project Website](https://lincsproject.org) and the[ CLUE Connectopedia](https://clue.io/connectopedia).
## Overview of the LINCS L1000 Dataset
### What is LINCS L1000?
The LINCS program was established to develop a network-based understanding of human biology by cataloging cellular changes in response to a wide variety of perturbing agents. To achieve this at scale, researchers developed the L1000 assay as a cost-effective, high-throughput alternative to traditional methods. By measuring a core set of landmark genes and computationally inferring the remaining transcriptome, the program expanded the Connectivity Map (CMap) resource to encompass over one million expression profiles.
The LINCS program has been implemented in two phases:
1. The pilot phase of the program was completed in 2013 and was focused on initial production of perturbation-induced molecular and cellular signatures, assay development, development of data standards, as well as tools and databases for accessing the data.
2. Phase 2, which began in 2014, and is the focus of this Perspective, supports six LINCS Data and Signature Generation Centers (DSGCs) and one Data Coordination and Integration Center (DCIC)
### Perturbation Types
L1000 covers two major categories of perturbations:
* Chemical perturbagens (Small Molecules): Bioactive small molecules, including FDA-approved drugs and experimental compounds.
* Genetic perturbagens (Nucleic Acid Reagents): Reagents that alter gene expression through genetic manipulation, such as shRNA and siRNA (gene silencing), sgRNA for CRISPR-based gene knockout, and cDNA for gene overexpression.
### LINCS L1000 Data Levels
The LINCS L1000 dataset is organized into five hierarchical levels, each representing a distinct stage of data processing in the analysis pipeline. These levels range from raw measurement outputs to fully processed differential expression signatures, enabling users to select data appropriate for their analysis needs.
| Level | Name / Format | Description | Key Purpose |
| ------- | ----------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------ |
| Level 1 | Raw (LXB) | Unprocessed flow cytometry data from the Luminex platform | Captures original signal measurements |
| Level 2 | GEX | Gene expression values for 978 landmark genes after deconvolution | Converts raw signals into gene-level expression |
| Level 3 | Q2NORM / INF | Quantile-normalized expression profiles (landmark and imputed genes) | Standardizes data across samples and infers the full transcriptome |
| Level 4 | ZSPCINF / ZSVCINF | Gene signatures represented as z-scores relative to controls | Normalizes against plate populations or vehicle controls |
| Level 5 | Differential signatures | Differential gene expression profiles | Final processed data for downstream biological interpretation |
For more information about data levels, please refer to [lincsproject.org](https://lincsproject.org/LINCS/tools/workflows/find-the-best-place-to-obtain-the-lincs-l1000-data) and [clue.io/data\_levels](https://clue.io/connectopedia/data_levels).
According to the [GEO CMAP LINCS User Guide v2.1](https://docs.google.com/document/d/1q2gciWRhVCAAnlvF2iRLuJ7whrGP6QjpsCMq1yWz7dU/edit?tab=t.0), users are generally advised to use Level 5 data for analysis. This level provides fully processed differential gene expression signatures most suitable for biological discovery, whereas lower levels are primarily useful for investigating intermediate processing steps.
## Where to Access LINCS L1000 data
The following datasets are available on this platform, sourced from two origins: the [LINCS Data Portal ](https://lincsportal.ccs.miami.edu/datasets/)and [NCBI GEO](https://www.ncbi.nlm.nih.gov/geo/).
From LINCS Data Portal (8 datasets):
| No | Dataset (LINCS portal) | Data Level | Dataset ID |
| -- | ----------------------------------------------------------------------------------------- | -------------------------------- | ------------------------------------------------------------ |
| 1 | L1000 Dataset - small molecule, nucleic acid perturbagens - LINCS Phase 2 (December 2021) |
|
Note: Level 5 data (fully processed differential expression signatures) is available for three datasets: December 2021, Phase 1, and March 2017.
From NCBI GEO (4 datasets):
| Dataset / Series | Description | Data Type / Content | Update / Notes |
| ------------------------------------------------------------------------- | ------------------------------------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------ |
| [GSE92742](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE92742) | LINCS Phase 1 dataset (early release) | L1000 gene expression profiles | Not updated (except bug fixes) |
| [GSE70138](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE70138) | LINCS Phase 2 dataset (main production phase) | Expanded L1000 data | Updated every 6 months (2016–2020); also available via LINCS DCIC portal |
| [GSE106127](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE106127) | Genetic perturbation dataset | Subset of L1000 data (shRNA & CRISPR signatures) | Not new data; derived from GSE92742 & GSE70138 |
| [GSE92743](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE92743) | Contest/benchmark datasets for computational challenges | Curated data bundles | Static archive; supports analytics benchmarking & community engagement |
## Data Structure
All datasets on this platform are organized into two groups based on their source: LINCS Data Portal and NCBI GEO. Each source has a different file structure and requires different steps to access.
### LINCS Data Portal Datasets
All datasets downloaded from the LINCS Data Portal are packaged as tar archives and must be extracted before use. To extract a dataset, run the following command:
\# example
`tar -xvzf LDS-1482_1.0.tar.gz`
Once extracted, each dataset follows a consistent folder structure. Below is an example using LDS-1613 (L1000 Phase 2 — December 2021, Level 5):
```
LDS-1613/
├── Data/
│ ├── DataFieldDefinitions.xlsx — definitions of all data fields
│ ├── instinfo.txt — per-instance experiment information
│ ├── level5_beta_all_n1102210x12328.gctx — main data matrix
│ ├── pertinfo.txt — perturbagen information
│ └── siginfo.txt — signature information
│
├── Metadata/
│ ├── Antibody_Metadata.txt
│ ├── Cell_Line_Metadata.txt
│ ├── DatasetMetadata.txt
│ ├── Differentiated_Cell_Metadata.txt
│ ├── ESC_Metadata.txt
│ ├── ExperimentalMetadata.txt
│ ├── Nucleic_Acid_Metadata.txt
│ ├── Primary_Cell_Metadata.txt
│ ├── Protein_Metadata.txt
│ ├── Small_Molecule_Metadata.txt
│ ├── Unclassified_Perturbagens_Metadata.txt
│ └── iPSC_Metadata.txt
│
├── MetadataMapping/
│ ├── CellMapping.txt
│ ├── MetadataPropertyMapping.txt
│ └── PerturbagenMapping.txt
│
├── ProcessingPipeline/
│ └── ProcessingPipelineSpecifications.txt
│
└── ReadMe.txt
```
Each dataset contains four main components:
* Data/: the main expression data matrix (.gctx format) along with key metadata files describing each instance, perturbagen, and signature.
* Metadata/:detailed annotations for each perturbagen type (small molecules, nucleic acids, proteins, antibodies) and each cell model system (cell lines, primary cells, stem cells).
* MetadataMapping/: files mapping perturbagens and cell lines to their corresponding metadata records.
* ProcessingPipeline/: specifications describing how raw data was processed into the final signatures.
### NCBI GEO Datasets
Datasets from NCBI GEO do not require tar extraction. Files are available individually in .gctx.gz format and can be decompressed directly:
gunzip GSE92743\_Broad\_GTEx\_L1000\_Level3\_Q2NORM\_n3176x12320.gctx.gz
Each GEO dataset contains files in a single folder, including the main data matrix (.gctx), instance metadata (inst\_info.txt), and gene metadata (gene\_info.txt).
## Running analyses on DNAnexus
### Copying LINCS L1000 data and demo Notebooks into a Project
To try our demo analysis, please copy your data of interest and notebooks from this project into your own project space. Follow these steps to copy the notebooks:
1. Create a project for your LINCS L1000 data and analysis, billed to your own organization. Tutorials on project setup can be [found here](https://academy.dnanexus.com/overview-of-the-platform/setting-up-a-project).
2. Navigate to the Resources tab, locate the project titled "Public Datasets AWS US (East)," and select the "LINCS\_L1000\_data" folder.
3. Click "Copy" in the top-right menu and select the project created in Step 1.
4. Return to your project space to begin exploring the LINCS L1000 dataset and notebooks.
5. To run the notebooks, refer to the [JupyterLab](https://academy.dnanexus.com/interactivecloudcomputing/jupyterlab/introduction) section of the Academy documentation, which includes instructions for [standard](https://academy.dnanexus.com/interactivecloudcomputing/jupyterlab/running_notebook) environments.
{% hint style="info" %}
Note: Since the LINCS L1000 dataset exceeds 600 GB, we suggest copying only the specific datasets required for your analysis.
{% endhint %}
### Example LINCS L1000 notebooks
We prepared three Jupyter notebooks (.ipynb) for demo analyses using LINCS L1000 data. They are optimized for the [JupyterLab with Python, R, Stata, ML, Image Processing](https://academy.dnanexus.com/interactivecloudcomputing/jupyterlab) (version 2.11.0).
* Notebook 1: cmapPy\_pandasGEXpress\_tutorial.ipynb
* A tutorial on efficient handling of large-scale gene expression data in .gct and .gctx formats using cmapPy, including metadata loading, subset filtering, and selective parsing. Example dataset: GSE70138.
* Notebook 2: Drug\_treatment\_finding\_tutorial.ipynb
* A treatment-finding workflow for identifying drugs that reverse a disease gene expression signature, a core strategy in transcriptomic drug discovery. Example dataset: LDS-1613, December 2021, Level 5.
* Notebook 3: Drug\_repurposing\_tutorial.ipynb
* A drug repurposing workflow for identifying drugs that mimic a query gene expression signature, supporting mechanism-of-action analysis. Example dataset: LDS-1613, December 2021, Level 5.
Use snapshot: snapshot-target\_validation-jupyterlab.tar.gz when starting the job. The snapshot is available in the “Notebook\_snapshot” folder in “Public Datasets AWS US (East)”.
To use the snapshot, choose this snapshot in the Snapshot box when you start the JupyterLab.
We recommend running this notebook in a clean conda environment with \`cmap\`. Please select kernel: \`cmap\` to run this notebook. If you don't see the kernel, please activate the environment by using below command
```
conda activate cmap
python -m ipykernel install --name cmap --display-name "cmap"
```
Then, refresh the page and select registered kernel before running this notebook
Then select the appropriate kernel and instance type listed in the table below to run the notebooks:
| Notebook | AWS Instance Type | Azure Instance Type | Kernel |
| ---------------------------------------- | ------------------- | --------------------- | ------ |
| Drug\_treatment\_finding\_tutorial.ipynb | mem3\_ssd1\_v2\_x8 | azure:mem3\_ssd1\_x8 | cmap |
| Drug\_repurposing\_tutorial.ipynb | mem1\_ssd1\_v2\_x16 | azure:mem1\_ssd1\_x16 | cmap |
| cmapPy\_pandasGEXpress\_tutorial.ipynb | mem1\_ssd1\_v2\_x16 | azure:mem1\_ssd1\_x16 | cmap |