diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/11Duplicate.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/11Duplicate.png new file mode 100644 index 00000000..e2e9df68 Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/11Duplicate.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/12Ontologies.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/12Ontologies.png new file mode 100644 index 00000000..a3d487e2 Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/12Ontologies.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/13Validation.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/13Validation.png new file mode 100644 index 00000000..c31bdd6d Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/13Validation.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/14Invalid.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/14Invalid.png new file mode 100644 index 00000000..be208c64 Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/14Invalid.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/15Customize.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/15Customize.png new file mode 100644 index 00000000..8ac7bed3 Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/15Customize.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/16ChangeOntologies.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/16ChangeOntologies.png new file mode 100644 index 00000000..853d82e6 Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/16ChangeOntologies.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/1TemplateEditor.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/1TemplateEditor.png new file mode 100644 index 00000000..99fe6322 Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/1TemplateEditor.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/2Overview.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/2Overview.png new file mode 100644 index 00000000..e1faf9d6 Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/2Overview.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/3Edit.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/3Edit.png new file mode 100644 index 00000000..36a9649c Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/3Edit.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/4Explore.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/4Explore.png new file mode 100644 index 00000000..a929c1d4 Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/4Explore.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/5Explore.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/5Explore.png new file mode 100644 index 00000000..e4470d8c Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/5Explore.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/6EditorView.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/6EditorView.png new file mode 100644 index 00000000..592e2831 Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/6EditorView.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/7Format.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/7Format.png new file mode 100644 index 00000000..a2c55c4d Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/7Format.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/8Export.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/8Export.png new file mode 100644 index 00000000..67372d15 Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/8Export.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/9Change.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/9Change.png new file mode 100644 index 00000000..388a2e2a Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/9Change.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/Template_dropdown.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/Template_dropdown.png deleted file mode 100644 index 8c349dd6..00000000 Binary files a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/Template_dropdown.png and /dev/null differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/change-template-me.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/change-template-me.png deleted file mode 100644 index 7b498ea0..00000000 Binary files a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/change-template-me.png and /dev/null differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/edit-template.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/edit-template.png deleted file mode 100644 index b8bc2d68..00000000 Binary files a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/edit-template.png and /dev/null differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/shortcut_1_37.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/shortcut_1_37.png deleted file mode 100644 index 5e9e914a..00000000 Binary files a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/shortcut_1_37.png and /dev/null differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/template-example.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/template-example.png deleted file mode 100644 index 05271d7d..00000000 Binary files a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/template-example.png and /dev/null differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/template_grouping.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/template_grouping.png deleted file mode 100644 index da365a64..00000000 Binary files a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/template_grouping.png and /dev/null differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/templates-list.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/templates-list.png deleted file mode 100644 index 6fe4f0c0..00000000 Binary files a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/templates-list.png and /dev/null differ diff --git a/docs/user-guide/doc-odm-user-guide/template-editor.md b/docs/user-guide/doc-odm-user-guide/template-editor.md index caf5059a..0556e40a 100644 --- a/docs/user-guide/doc-odm-user-guide/template-editor.md +++ b/docs/user-guide/doc-odm-user-guide/template-editor.md @@ -1,99 +1,209 @@ # Template Editor -The Template Editor application allows you to create, customize, and update templates for data import/validation. +The Template Editor in ODM enables users to create, customize, and update templates for data import and validation. These templates define metadata attributes, ensuring terms and values are harmonized and validated according to predefined rules. + +## What is a template? + +A **Template** in ODM is a structured framework that defines metadata attributes for data objects, ensuring that terms and values are harmonized and validated according to predefined rules. Templates specify the required fields, data types, validation dictionaries, and descriptions, providing consistency in how metadata is managed across studies, samples, and experimental data. + +Each data object (e.g., Study, Sample, Tabular Data Group, Variant, Flow Cytometry) uses a separate template. + +Templates include the following metadata options: + +* **Name**: The name of the metadata field that should be included (e.g., “Accession,” “Organism”). +* **Required**: Indicates whether the metadata field is mandatory. If a required field is left blank or incorrectly filled, it will be highlighted in red. +* **Metainfo Type**: Specifies the type of metadata, such as text, integer, decimal, date, yes/no, or external link. +* **Read-only**: Defines whether the metadata field can be edited (“yes” means it cannot be edited). +* **Dictionary**: Enables the use of a predefined dictionary to provide standardized terms, helping with data curation and validation. +* **Description**: A brief description of the attribute, shown as a hint during data curation. ## Accessing the Template Editor -To get started, click **Set up templates** on the Dashboard: +To access the Template Editor: -Or use the shortcut dock and select **Template Editor**: -![image](doc-odm-user-guide/images/shortcut_1_37.png) +1. Click **Set up templates** on the Dashboard. +2. Alternatively, click on the menu in the top-left corner and select **Template Editor**. -On the application page you will see the available templates. Also, you can find out by whom and when it was -created, updated. Click on the template name to explore it in more detail. +![1TemplateEditor.png](doc-odm-user-guide/images/1TemplateEditor.png) +
Access the Template Editor by clicking Set up templates on the dashboard (A) or by clicking the three lines at the top left corner of the dashboard and selecting Template Editor (B)
-![image](doc-odm-user-guide/images/templates-list.png) -## What is a template? +## Template Editor Overview -Templates control metadata attributes in Genestack, which allow terms and values to be harmonized and validated. -A separate template is used for each data object, e.g. “Study”, “Sample”, “Expression”, “Variant” -and “Flow cytometry”. +When you open the Template Editor, you can see the list of available templates in your ODM instance, including the dates of creation, the author, and the latest update. There is no limit on the number of templates you can create. -Templates include the following metadata options: +![2Overview.png](doc-odm-user-guide/images/2Overview.png) +
View of the Template Editor. Explore the full list of templates, creation dates, authors, and latest updates
-- *Name* — name of metadata field that should be included in the metainfo (e.g. “Accession”, “Organism”); -- *Required* — determines whether the metadata field should be mandatory or not; if a field is marked as required - and is left blank or it is incorrectly filled in, it is highlighted in red; -- *Metainfo type* — indicates metadata type for the filed, namely: text, integer, decimal, date, yes/no and external link; -- *Read-only* — represents permissions for editing, where “yes” means that you cannot edit; the metadata -- *Dictionary* — allows you to specify a dictionary providing standardized and unified terms that can be used in data - curation and to help validate and harmonise metadata. -- *Description* - description for the attribute that will be shown during curation as a hint. +## Editing Templates -![image](doc-odm-user-guide/images/template-example.png) +Click on the three dots next to each template to access options: -!!! abstract "Mandatory Attributes" - All templates contain mandatory technical fields that are **grayed out** in the user interface and cannot be edited or removed. +* **Rename**: Change the name of the template. +* **Duplicate**: Create a copy of the template, useful for creating similar templates with slight modifications. +* **Set as default**: Assign a new default template for any new studies. The default template can also be set when creating a new study (see the section [Create a new study](create-study.md) for more details). +* **Export**: Export the selected template in JSON format, allowing for modification and integration with downstream applications. +* **Copy accession**: Each template has a unique accession number, which is used for API endpoints. +* **Add to bookmarks**: Mark the template as a favorite. +* **More info**: View detailed information about the template, including sharing settings and included ontologies. - **Common field:** +![3Edit.png](doc-odm-user-guide/images/3Edit.png) +
Edit templates. Click on the three dots next to the template’s name to access editing options such as rename, duplicate, export, etc. You can select the default template (A) or a new template from the list (B)
- - `genestack:accession` +## Exploring Templates - **Tabular data fields:** +Click on any template to open it in a new window and explore its features. This window displays the components of the template for each part of your study: - - `Features (string)` +* Study +* Sample +* Library +* Preparation +* Tabular Data Group +* Variant +* Flow Cytometry - - `Features (numeric)` +![4Explore.png](doc-odm-user-guide/images/4Explore.png) +
View of the templates and their features. Explore templates, including sections such as samples metadata, library, preparation, etc.
- - `Values (numeric)` +Alternatively, access the Template Editor from a study by opening the study, clicking on the study title in the top bar, then clicking on the template title and selecting **Explore**. - - `Data Class` (for Tabular, Variants, and Flow Cytometry data) +![5Explore.png](doc-odm-user-guide/images/5Explore.png) +
Access the Template Editor from an open study by clicking on the study title, then the template name, and Explore
+ +## Edit Template Metadata + +Within the Template Editor, specify metadata fields and their properties: + +* **Name**: Name of the metadata field (e.g., "Accession," "Organism," "Sample name"). +* **Required**: Indicates if the field is mandatory. Required fields left blank or incorrectly filled will be highlighted in red. +* **Metainfo type**: Type of metadata (text, integer, decimal, date, yes/no, external link). +* **Read-only**: Permission setting for editing the metadata (Yes/No). +* **Dictionary**: Standardized terms for data curation and validation. +* **Description**: A hint or description for the attribute shown during curation. + +![6EditorView.png](doc-odm-user-guide/images/6EditorView.png) +
Template Editor view: rows represent metadata attributes, and columns show classification features like Required and Metainfo Type
+ +## Actions on Templates + +* **Grouping Metadata Fields**: Group fields under a common header using the **"/"** character (e.g., Signature/Initial and Signature/Quintile). + +![7Format.png](doc-odm-user-guide/images/7Format.png) +
Use the "/" symbol to create subgroups within metadata fields. For example, The category Signature contains two values: Initial and Quintile. Use the / to create subgroups within the Signature attribute. The changes will be visible in the study where the template is being applied
+ +* **Export Template**: Export templates in JSON format via the **Export** option. + +![8Export.png](doc-odm-user-guide/images/8Export.png) +
Export the template in JSON format. In the template editor window, click on the name of the template, whether the Default template (A) or another template from the list (B), and click on Export. This action will automatically start the download of the template in JSON format
+ +* **Change Template**: Change the template associated with a study by selecting the study name and **Apply another**, then choosing the desired template. + +![9Change.png](doc-odm-user-guide/images/9Change.png) +
Change the template applied to the study. On the study view, select the name of the study (top bar) and click on the name of the template. Then select Apply another. This action will show a list of available templates. Select the preferred one and click on Apply. You can also explore the template before applying by clicking on Explore
+ +## Creating a New Template + +Users with appropriate permissions can create new templates. Refer to the section [Users and Groups](../doc-odm-user-guide/setup.md) for more on user roles and permissions. Templates can be created via: +* **Python Script**: Import a new template using JSON files. Follow the instructions in the [Create a new template](/tools/odm-sdk/terminal/templates/create-or-update-template/) +* **Graphical User Interface**: Duplicate an existing template, rename it, and customize the metadata fields as needed. To create a new template, navigate to the template editor to see the list of templates. Then select any template and click on the option **Duplicate**, and select a new name for this template. -## Create and edit template +!!! note + It is recommended that you assign a new name to identify your templates easily. This action will create a new template with the same metadata fields. Customize the metadata fields and actions by opening the template editor for the recently created template. -If you have the permission to edit/manage templates you can click on the template name and select **Duplicate** to create your own editable copy of a template and modify it if needed. +![11Duplicate.png](doc-odm-user-guide/images/11Duplicate.png) +
Create a new template from a previously created template, whether the Default template (A) or a template from the list (B). Simply select the template (from the template editor), click on the three dots next to the template, and click on Duplicate. Select a new name for the template. This action will create a new template with the same metadata fields. You can open it to customize the metadata fields
-![image](doc-odm-user-guide/images/Template_dropdown.png) +## Dictionaries and Ontologies -Autocomplete can help you to specify an appropriate dictionary term to be used in metadata validation in the Metadata Editor. +Dictionaries and ontologies are essential in metadata curation and validation for ensuring accuracy, consistency, and organization. -![image](doc-odm-user-guide/images/edit-template.png) +### What Are Dictionaries? -## Grouping/compounding metadata fields +Dictionaries in ODM provide controlled vocabularies that standardize the names and definitions of data elements, crucial for: -Metadata fields can be grouped together under a common header using the **/** character. Use the common header before -the **/** character. For example: +* **Standardization**: Consistent use of terms and values across datasets. +* **Clarity**: Clear definitions that reduce misinterpretation. +* **Validation**: Ensuring data entries match predefined acceptable values. -![image](doc-odm-user-guide/images/template_grouping.png) +For example, a dictionary for the **Species** attribute ensures correct and uniform scientific names. + +### What Are Ontologies? + +Ontologies extend the functionality of dictionaries by organizing data into structured, relational frameworks. They offer: + +* **Hierarchical Structure**: Categories and subcategories that clarify data relationships. +* **Conceptual Clarity**: Defined concepts and their relationships support better data analysis. +* **Enhanced Metadata**: Additional context, such as taxonomic lineage, enriches the dataset. + +For example, the **NCBI ontology** organizes species names hierarchically, aiding in accurate classification and analysis. + +![12Ontologies.png](doc-odm-user-guide/images/12Ontologies.png) +
Dictionaries and ontologies. Selecting the NCBI Taxonomy for the Microbial species attribute ensures Data Consistency: All species names conform to the standardized taxonomy provided by the NCBI, avoiding discrepancies such as misspellings or synonyms. Accurate Classification: The ontology's hierarchical structure allows for accurate classification and grouping of species, facilitating comprehensive data analysis. Enriched Context: The additional information provided by the ontology (e.g., genetic data, ecological information) enriches the dataset, providing valuable insights for researchers.
+ +### Why Use Dictionaries and Ontologies in ODM? + +Incorporating dictionaries and ontologies into your metadata curation process offers several benefits: + +* **Improved Data Quality**: By standardizing and validating data entries, these tools significantly reduce errors and inconsistencies. +* **Enhanced Data Interoperability**: The use of standardized vocabularies and structures enables seamless data integration across different systems and datasets. +* **Facilitated Data Analysis**: The additional context and relationships defined by ontologies support more advanced data queries and insights. + +## Validation + +**Validation** in the context of data curation refers to ensuring that metadata fields conform to predefined rules and standards set by a specific template. This process is essential for maintaining data integrity, consistency, and reliability. + +### Steps in validation + + +* **Template-Based Validation**: Metadata fields are checked against a designated template. Templates are designed to include rules for each type of data object, such as Study, Sample, and experimental data. They specify required fields and acceptable values. + +!!! abstract "Mandatory Attributes" + All templates contain mandatory technical fields that are **grayed out** in the user interface and cannot be edited or removed. + + **Common field:** + + - `genestack:accession` + + **Tabular data fields:** + + - `Features (string)` + + - `Features (numeric)` + + - `Values (numeric)` + + - `Data Class` (for Tabular, Variants, and Flow Cytometry data) -## Export template -You can export a JSON file which contains the template information by clicking the template name and selecting export from the drop down menu: +![13Validation.png](doc-odm-user-guide/images/13Validation.png) +
Metadata fields are validated against dictionaries and ontologies set in templates
-![image](doc-odm-user-guide/images/Template_dropdown.png) +* **Error Indication**: If any required fields are missing or contain incorrect values according to the template, the system flags the metadata as **Invalid**. The specific field that is invalid is highlighted with a red background, providing a clear cue for correction. -## Change template +![14Invalid.png](doc-odm-user-guide/images/14Invalid.png) +
Invalid metadata is highlighted in red, with details accessible by clicking Invalid metadata
-By default a **Default Template** is associated with data, which includes metadata -columns along with associated controlled dictionaries and ontologies. The associated template can be changed in the **Metadata Editor** -when editing metadata. +* **Customizable Templates**: Modify metadata fields as needed, with caution against changing the Default template to avoid organization-wide impacts. -From the **Metadata Editor**, click the study name, then **Apply another** and select -the template of interest from the list of suggested files. Click **Explore** to open it with the Template Editor. +![15Customize.png](doc-odm-user-guide/images/15Customize.png) +
Customize templates. Add, remove, and update values in the templates. You can change the settings of each attribute depending on the needs of your studies
-![image](doc-odm-user-guide/images/change-template-me.png) +* **Customizable dictionaries and ontologies**. Users have the flexibility to choose or load their dictionaries within the templates, which allows for specialized metadata curation and harmonization. This feature supports the use of standardized and unified terms, making the data more consistent and interoperable. -## Template validity checks +!!! note "Important" + Uploading dictionaries is an advanced feature and it is done via a Python script. [Follow the instructions](/tools/odm-sdk/terminal/dictionaries-and-ontologies/creating-new-dictionary/) -When you apply or modify templates, a **validity check** is automatically triggered to ensure consistency between the template and associated studies. +![16ChangeOntologies.png](doc-odm-user-guide/images/16ChangeOntologies.png) +
Change ontologies in the template editor by selecting a new option. To change the current ontologies, open the template editor, select the metadata field to change, click on the triangle, and select the new ontology
-- **When changing a template assigned to a study:** - The validity check is run **only for that specific study**. +### **Importance of Validation** -- **When editing a field in a template:** - The validity check is triggered for **all studies that use this template**. +Validation is crucial for: +* **Ensuring Data Quality**: By enforcing metadata standards, validation helps maintain high data quality. +* **Facilitating Data Harmonization**: Custom dictionaries and standardized terms within templates aid in harmonizing metadata, making it easier to integrate and compare data from different sources. +* **Streamlining Data Management**: Automated error detection and clear indications of invalid fields streamline the data curation process, saving time and reducing manual errors. +By adhering to these validation processes, users can ensure that their metadata is accurate, complete, and standardized, which is fundamental for effective data management and analysis within the ODM.