> For the complete documentation index, see [llms.txt](https://academy.dnanexus.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://academy.dnanexus.com/public-datasets-on-the-dnanexus-platform/csv-builders-for-nextflow-pipelines/technical-considerations.md).
# Technical Considerations
This section applies to all nf-core input CSV builder applets.
### Validation does not block export
The applet validates all inputs and logs warnings or errors, but does not prevent users from exporting the output samplesheet even if it contains invalid values. This means a job can complete successfully with warnings while the output samplesheet is still invalid for pipeline use. Therefore, please always review the log file for any warnings or errors before using the output samplesheet to run the pipeline.
### Input Source Priority
When both a prefilled samplesheet and file inputs (FASTA files for proteinfold, FASTQ files or folder for scrnaseq) are provided simultaneously, the applet prioritizes the prefilled samplesheet and ignores the file inputs. To avoid unexpected behavior, users should choose only one input source.
For example, if a prefilled samplesheet with 5 rows is uploaded alongside 2 selected FASTA files, the output samplesheet will contain 5 rows from the prefilled samplesheet only. The 2 FASTA files will be ignored and the applet will log a warning.
### Consistent input length
When providing manual input, all columns must have the same number of values. The only exception is a single constant value, which will be broadcast across all rows. For example, if 3 sample names are provided, all other columns must also have 3 values or a single constant. If not, the applet will log the following error
Manual input value of column '{col}' must be a constant or an array with the length of {expected\_length} instead of {len(values)}.
### CSV field count mismatch
When preparing a prefilled samplesheet, ensure that each row has the same number of fields as the header. A common mistake is having extra or missing commas in a row, which causes the applet to log the following error:
Prefilled samplesheet row (0): Header (3) and Data (4) lengths do not match.
Missing required columns: sample
Missing required columns: fastq\_1
Missing required columns: fastq\_2
Note that the Missing required columns errors are a consequence of the mismatch and do not indicate that the columns are actually missing. It is recommended to open the CSV file to verify the column alignment.
### Empty cell in required columns
Leaving a cell empty in a required column (e.g. fastq\_1) is treated differently from omitting the column entirely, but both result in similar error messages. When an empty cell is detected, the applet logs the following errors:
(0,fastq\_1): File not found: nan
(0,fastq\_1): FastQ file for reads 1 must be provided, cannot contain spaces and must have extension '.fq.gz' or '.fastq.gz'
Missing required columns: fastq\_1
The output samplesheet will be generated but the affected column will be dropped. Ensure all required columns are fully populated before running the pipeline.
### Sample name auto-detection
When no sample name is provided, the applet automatically extracts the sample name from the fastq\_1 filename by taking the substring from the beginning of the filename to the first underscore. For example:
| File name | Extracted sample name |
| ------------------------------------ | --------------------- |
| AEG588A1\_S1\_L002\_R1\_001.fastq.gz | AEG588A1 |
| SRR14575500\_1.fastq.gz | SRR14575500 |
| pbmc8k\_S1\_L007\_R1\_001.fastq.gz | pbmc8k |
The only requirement is that the filename must contain at least one underscore \_. If the filename does not contain an underscore, the applet will not be able to extract the sample name and will log a warning. In this case, it is recommended to provide the sample name manually via the sample field or use a prefilled samplesheet.
### DNAnexus path resolution
The applet automatically resolves all file references into valid dx:// URIs in the output samplesheet. If a file path is provided in project-xxx:/path/to/file format (without dx:// prefix), the applet will automatically prepend dx://. For example:
| Input format | Output in samplesheet |
| --------------------------------------- | ------------------------------------------- |
| file-xxxxxxxxx | dx://project-xxx:/path/to/file.fastq.gz |
| project-xxx:/path/to/file.fastq.gz | dx://project-xxx:/path/to/file.fastq.gz |
| dx://project-xxx:/path/to/file.fastq.gz | dx://project-xxx:/path/to/file.fastq.gz |
| filename.fastq.gz | dx://project-xxx:/path/to/filename.fastq.gz |
### Using files from another project
The applet resolves file references within the project it is launched from. If a filename or relative path is provided, the applet will search for the file within the current project only. Files from other projects will not be found and the applet will log a File not found error.
To use files from another project, you must explicitly specify the project ID in one of the following formats:
`project-xxxxxxxxx:file-xxxxxxxxx`
`project-xxxxxxxxx:/path/to/file.fastq.gz`
`dx://project-xxxxxxxxx:/path/to/file.fastq.gz`
For example, if you are launching the applet from project-A but your FASTQ files are stored in project-B, providing only the filename will result in a File not found error even if the file exists in project-B. You must provide the full path with project-B ID explicitly.
Note: Make sure you have read access to the other project.
You can also select files in another project from UI.
* Click on All projects to view all projects you have access to.
* Then search a project by using project name or project ID.
### Use files from public project
If you cannot find public projects in All projects, you need to prepare a prefilled sample sheet file with full path including project ID of public project.
For example, we want to use FASTA files in Public Project AWS East (project-J3JyY6j030gzQypGpk273241).
```
sequence,fasta,comment
T1024,project-J3JyY6j030gzQypGpk273241:/ProteinFold/samplesheet_input/Fasta/T1024.fasta,need to have absolute path for files in public project
T1026,project-J3JyY6j030gzQypGpk273241:/ProteinFold/samplesheet_input/Fasta/T1026.fasta,need to have absolute path for files in public project
```
### Output file naming
If output\_samplesheet\_name is not provided, the applet defaults to sample\_sheet.csv. If a name is provided without the .csv extension, the applet automatically appends it. For example:
| Input | Output file name |
| ------------------- | ------------------- |
| my\_samplesheet | my\_samplesheet.csv |
| my\_samplesheet.csv | my\_samplesheet.csv |
| empty | sample\_sheet.csv |
### Run applet with dx run
Here is an example of how to use dx run:
dx run $APPLET\_ID --priority high -y \\
-iversion="1.1.1" \\
-isequence="T1024\_2" \\
-ifasta="file-J81Fz6j09QzKyF7xyv04vkKf" \\
-isequence="T1026\_2" \\
-ifasta="file-J81Fz6j09Qz55XVJ9fgQyx0f" \\
-ioutput\_samplesheet\_name="samplesheet\_proteinfold.csv" \\
\--destination "project-J5Xqzy009Qz80FqQpy7QGKfV:/users/piyapong/test\_applet/output/" \\
\--name "Test proteinfold v 1.1.1 with manual input sequence and fasta"
samplesheet\_proteinfold.csv:
\| | sequence | fasta |
\|---:|:-----------|:-----------------------------------------------------------------------------------------------|
\| 0 | T1024\_2 | dx://project-J5Xqzy009Qz80FqQpy7QGKfV:/users/piyapong/test\_applet/fasta/dataset2/T1024\_2.fasta |
\| 1 | T1026\_2 | dx://project-J5Xqzy009Qz80FqQpy7QGKfV:/users/piyapong/test\_applet/fasta/dataset2/T1026\_2.fasta |
## Limitations
The applet is designed as a helper tool for straightforward use cases. For the following scenarios, it is recommended to use the more capable JupyterLab notebook interface instead:
* Large studies with many samples.
* Complex multi-lane experiments.
* Cross-project file references requiring extensive path manipulation.
* Cases requiring custom metadata inference from filenames.
## Schema drift
Here, the CSV Builder for nf-core/scrnaseq is built for version 4.1.0 and the CSV Builder for nf-core/proteinfold is built for version 1.1.1 and 2.0.0. If the pipeline is updated to a newer version with changes to required columns, validation rules, or accepted values, the applet may produce unexpected validation errors or incorrect output because it is still validating against the older schema.
## Troubleshooting
| Error message | Explanation | How to fix |
| ------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Manual input value of column '{col}' must be a constant or an array with the length of {expected\_length} instead of {len(values)} | The number of values provided for a column does not match the number of values in other columns. | Ensure all columns have the same number of values, or provide a single constant value to broadcast across all rows. See \[Consistent Input Length] for more details. |
| Prefilled samplesheet row ({index}): Header ({len(header)}) and Data ({len(row)}) lengths do not match | The number of fields in a CSV row does not match the number of columns in the header, usually caused by extra or missing commas. | Open the CSV file to verify column alignment. See \[CSV Field Count Mismatch] for more details |
| Missing required columns: {col} | A required column is completely missing from the prefilled samplesheet, or a required column contains an empty cell. | Add the missing column with valid values. If the column exists but has empty cells, ensure all cells are populated. See \[Empty Cell in Required Column] for more details |
| ({row\_index},{col}): File not found: nan | A required cell is empty in the prefilled samplesheet. | Ensure all required columns are fully populated before uploading. |
| ({row\_index},{col}): FastQ file for reads 1 must be provided, cannot contain spaces and must have extension '.fq.gz' or '.fastq.gz' | A required FASTQ file is missing or has an invalid extension. | Provide a valid FASTQ file with extension .fq.gz or .fastq.gz. |
| ({row\_index},{col}): File name must contain 'R2' | The file assigned to fastq\_2 does not contain R2 in its filename. | Check that the correct R2 file is assigned to fastq\_2. |
| ({row\_index},{col}): Expect fastq\_2 file name to be: {filename} | The applet detected a mismatch between the R1 and R2 filenames. | Check that the R1 and R2 files are correctly paired. |
| ({row\_index},{col}): Found {str(len(items))} files: {items} | Multiple files with the same name exist in the project, applet cannot determine which file to use. | Use a relative or absolute path, or file ID to uniquely identify the file. See \[Ambiguous File Resolution] for more details. |
| ({row\_index},{col}): The entity {file\_id} could not be found, code 404 | The file ID provided does not exist in the project. | Check that the file ID is correct and the file exists in the project. |
| ({row\_index},{col}): Fasta file must be provided, cannot contain spaces and must have extension '.fa' or '.fasta' | A required FASTA file is missing or has an invalid extension. | Provide a valid FASTA file with extension .fa or .fasta. |
| Required column fasta cannot contain null | One or more cells in the fasta column are empty. | Ensure all rows in the fasta column have valid file references. |
| Require at least one of these columns: \['sequence', 'id'] | Required columns are not provided. | Provide required columns for Proteinfold v1.1.1 (sequence) or v2.0.0 (id). |
| Found more than one additional required columns. Choose one of these columns as an input: Index(\['sequence', 'id'], dtype='str') | Provide both id and sequence column while Proteinfold v1.1.1 or 2.0.0 requires only one. | Provide required columns for Proteinfold v1.1.1 (sequence) or v2.0.0 (id). |