Scoping doc

[yulric]

No description provided.

[DougManuel]

Re: Import/Export
I wonder if being more opinionated about the metadata structure would orient this section, but I appreciate that this may be the next step.

• We'll store the metadata using a "sidecar" approach with three main tables: catalog, variable, variable_details.
• There is support for variable labels, value labels and tagged_NA and established use. i.e. haven and labelled packages. For import, much metadata for key users comes from SAS or Stata files that are imported using haven. We should allow users to import this metadata and the store it in variables and variable_details.
• There should be future provisions to be interoperable with other common metadata formats. I've added a conceptual diagram.

[DougManuel]

1. Wide versus long may be better situated as a future feature.
2. Do we want to include concepts such as logging and testing? Having clear expectations for the user about what will happen during transformations and the ability to debug, examine transformation effects, etc. Again, this could go in the next document.

[DougManuel]

I've made a few comments and suggestions.

We can close the PR whenever you want and address any outstanding issues with the specifications.

However, feel free to ask for more clarity.

[yulric]

1. Wide versus long may be better situated as a future feature.
2. Do we want to include concepts such as logging and testing? Having clear expectations for the user about what will happen during transformations and the ability to debug, examine transformation effects, etc. Again, this could go in the next document.

1. Agreed
2. Makes sense for those to go into the next document since its hard to provide a high level overview of what should go into logging and testing without getting into the specifications.

[yulric]

@DougManuel The scoping document should now be ready for you to review. The document is in the scope-docs folder and can be rendered using the command quarto preview ./scope-docs. You will need to delete and re-pull your version of this branch since I changed the entire history. I added a commit with everything you wrote to ensure that its all preserved. I also pulled in the changed from your catalog branch into the scope. Finally, I created a v2 branch into which this branch will be merged.

[DougManuel]

Small point: the current version is 0.1.0. Let's call this update v1.0.0 (not v2).

[DougManuel]

@yulric I didn't see the catalog.qmd in the commit list, but we can access the document from this closed PR #65.

[yulric]

@DougManuel Let me know once you're done with your review

[DougManuel]

I've finished with my reviews.