Milab 4784 fix deduplication and update #39
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| --- | ||
| '@platforma-open/milaboratories.dimensionality-reduction.workflow': minor | ||
| '@platforma-open/milaboratories.dimensionality-reduction': minor | ||
| --- | ||
|
|
||
| Enable block deduplication and improve trace label |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,19 @@ | ||
| # Overview | ||
|
|
||
| Takes high-dimensional gene expression data from scRNA-seq preprocessing blocks (e.g., Cell Ranger) and transforms it into a lower-dimensional space while preserving biological variation using three complementary methods. Principal Component Analysis (PCA) reduces the data to a configurable number of principal components capturing major sources of variation. t-distributed Stochastic Neighbor Embedding (t-SNE) creates a two-dimensional embedding optimized for local structure preservation, while Uniform Manifold Approximation and Projection (UMAP) generates a three-dimensional embedding balancing local and global structure. Both t-SNE and UMAP use adaptive parameters that adjust based on dataset size, and all methods operate on the PCA space for computational efficiency. | ||
|
|
||
| When metadata covariates are provided, the block optionally performs batch correction using Harmony, which integrates cells across batches while preserving biological variation. Harmony correction is applied to the PCA space, and both UMAP and t-SNE embeddings are recomputed using the corrected principal components. | ||
|
|
||
| The resulting dimension values are used by downstream blocks: PCA components for clustering and pseudotime inference, and UMAP/t-SNE embeddings for visualization. The block provides both standard and batch-corrected embeddings for comparison. | ||
|
|
||
| The block uses scanpy v1.10.1 for dimensionality reduction algorithms and preprocessing. When using this block in your research, cite the scanpy publication (Wolf et al. 2018) listed below. | ||
|
|
||
| The following publications describe the methodologies used: | ||
|
|
||
| > Wolf, F. A., Angerer, P., & Theis, F. J. (2018). SCANPY: large-scale single-cell gene expression data analysis. _Genome Biology_ **19**, 15 (2018). [https://doi.org/10.1186/s13059-017-1382-0](https://doi.org/10.1186/s13059-017-1382-0) | ||
|
|
||
| > McInnes, L., Healy, J., & Melville, J. (2018). UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. _Journal of Open Source Software_ **3**(29), 861. [https://doi.org/10.21105/joss.00861](https://doi.org/10.21105/joss.00861) | ||
|
|
||
| > van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. _Journal of Machine Learning Research_ **9**, 2579-2605. [http://www.jmlr.org/papers/v9/vandermaaten08a.html](http://www.jmlr.org/papers/v9/vandermaaten08a.html) | ||
|
|
||
| > Korsunsky, I., Millard, N., Fan, J. et al. (2019). Fast, sensitive and accurate integration of single-cell data with Harmony. _Nature Methods_ **16**, 1289–1296 (2019). [https://doi.org/10.1038/s41592-019-0619-0](https://doi.org/10.1038/s41592-019-0619-0) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| self := import("@platforma-sdk/workflow-tengo:tpl") | ||
| exec := import("@platforma-sdk/workflow-tengo:exec") | ||
| assets := import("@platforma-sdk/workflow-tengo:assets") | ||
|
|
||
| self.defineOutputs("umapDimensionsCsv", "tsneDimensionsCsv", "harmonyResultsCsv") | ||
|
|
||
| self.body(func(args) { | ||
| // Input parameters | ||
| csvCounts := args.csvCounts | ||
| csvCovariates := args.csvCovariates | ||
| mem := args.mem | ||
| cpu := args.cpu | ||
|
|
||
| // Batch correction software execution | ||
| batchCorrection := exec.builder(). | ||
| software(assets.importSoftware("@platforma-open/milaboratories.dimensionality-reduction.software:calculate-batchCorrection")). | ||
| mem(mem). | ||
| cpu(cpu). | ||
| addFile("rawCounts.csv", csvCounts). | ||
| addFile("metadata.csv", csvCovariates). | ||
| arg("--counts").arg("rawCounts.csv"). | ||
| arg("--metadata").arg("metadata.csv"). | ||
| arg("--output").arg("."). | ||
| saveFile("umap_dimensions.csv"). | ||
| saveFile("tsne_dimensions.csv"). | ||
| saveFile("harmony_results.csv"). | ||
| cache(24 * 60 * 60 * 1000). | ||
| run() | ||
|
|
||
| // Get result files | ||
| umapDimensionsCsv := batchCorrection.getFile("umap_dimensions.csv") | ||
| tsneDimensionsCsv := batchCorrection.getFile("tsne_dimensions.csv") | ||
| harmonyResultsCsv := batchCorrection.getFile("harmony_results.csv") | ||
|
|
||
| return { | ||
| umapDimensionsCsv: umapDimensionsCsv, | ||
| tsneDimensionsCsv: tsneDimensionsCsv, | ||
| harmonyResultsCsv: harmonyResultsCsv | ||
| } | ||
| }) | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| self := import("@platforma-sdk/workflow-tengo:tpl") | ||
| exec := import("@platforma-sdk/workflow-tengo:exec") | ||
| assets := import("@platforma-sdk/workflow-tengo:assets") | ||
|
|
||
| self.defineOutputs("umapResultsCsv", "tsneResultsCsv", "pcaResultsCsv") | ||
|
|
||
| self.body(func(args) { | ||
| // Input parameters | ||
| csvCounts := args.csvCounts | ||
| nPCs := args.nPCs | ||
| nNeighbors := args.nNeighbors | ||
| mem := args.mem | ||
| cpu := args.cpu | ||
|
|
||
| // Dimensionality reduction software execution | ||
| dimReduction := exec.builder(). | ||
| software(assets.importSoftware("@platforma-open/milaboratories.dimensionality-reduction.software:calculate-dimRed")). | ||
| mem(mem). | ||
| cpu(cpu). | ||
| addFile("rawCounts.csv", csvCounts). | ||
| arg("--file_path").arg("rawCounts.csv"). | ||
| arg("--output_dir").arg("."). | ||
| arg("--n_pcs").arg(string(nPCs)). | ||
| arg("--n_neighbors").arg(string(nNeighbors)). | ||
| saveFile("umap_results.csv"). | ||
| saveFile("tsne_results.csv"). | ||
| saveFile("pca_results.csv"). | ||
| cache(24 * 60 * 60 * 1000). | ||
|
There was a problem hiding this comment. The cache duration is hardcoded as a magic number. To improve readability and maintainability, it's better to define this value as a named constant at the top of the file. For example: Then, you can use this constant here: |
||
| run() | ||
|
|
||
| // Get result files | ||
| umapResultsCsv := dimReduction.getFile("umap_results.csv") | ||
| tsneResultsCsv := dimReduction.getFile("tsne_results.csv") | ||
| pcaResultsCsv := dimReduction.getFile("pca_results.csv") | ||
|
|
||
| return { | ||
| umapResultsCsv: umapResultsCsv, | ||
| tsneResultsCsv: tsneResultsCsv, | ||
| pcaResultsCsv: pcaResultsCsv | ||
| } | ||
| }) | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The cache duration is hardcoded as a magic number. To improve readability and maintainability, it's better to define this value as a named constant at the top of the file.
For example:
Then, you can use this constant here:
cache(cacheDuration).