From 5903ce5133e1af805feb8467a66be8454e50708d Mon Sep 17 00:00:00 2001 From: pavelkomarov Date: Wed, 25 Jun 2025 09:47:22 -0700 Subject: [PATCH 1/2] trying to be more specific about what's what so others don't have to dig down in every file like I did --- .gitignore | 1 + README.md | 64 +++++++++++++------------------------------- linting.py | 15 +++-------- pynumdiff/.gitignore | 1 - pynumdiff/README.md | 11 ++++++++ 5 files changed, 34 insertions(+), 58 deletions(-) delete mode 100644 pynumdiff/.gitignore create mode 100644 pynumdiff/README.md diff --git a/.gitignore b/.gitignore index 7ebc547..9a634df 100644 --- a/.gitignore +++ b/.gitignore @@ -117,3 +117,4 @@ dist *.whl .vscode docs/build +pynumdiff/_version.py diff --git a/README.md b/README.md index 8f1b481..d8d192f 100644 --- a/README.md +++ b/README.md @@ -49,43 +49,19 @@ For more details, refer to [this paper](https://doi.org/10.1109/ACCESS.2020.3034 ## Structure - PyNumDiff/ - |- README.md - |- pynumdiff/ - |- __init__.py - |- __version__.py - |- finite_difference/ - |- kalman_smooth/ - |- linear_model/ - |- smooth_finite_difference/ - |- total_variation_regularization/ - |- utils/ - |- optimize/ - |- __init__.py - |- __optimize__.py - |- finite_difference/ - |- kalman_smooth/ - |- linear_model/ - |- smooth_finite_difference/ - |- total_variation_regularization/ - |- tests/ - |- examples - |- 1_basic_tutorial.ipynb - |- 2a_optimizing_parameters_with_dxdt_known.ipynb - |- 2b_optimizing_parameters_with_dxdt_unknown.ipynb - |- docs/ - |- Makefile - |- make.bat - |- build/ - |- source/ - |- _static - |- _summaries - |- conf.py - |- index.rst - |- ... - |- .gitignore - |- LICENSE.txt - |- pyproject.toml +- `.github/workflows` contains `.yaml` that configures our GitHub Actions continuous integration (CI) runs. +- `docs/` contains `make` files and `.rst` files to govern the way `sphinx` builds documentation, either locally by navigating to this folder and calling `make html` or in the cloud by `readthedocs.io`. +- `examples/` contains Jupyter notebooks that demonstrate some usage of the library. +- `pynumdiff/` contains the source code. For a full list of modules and further navigation help, see the readme in this subfolder. +- `.editorconfig` ensures tabs are displayed as 4 characters wide. +- `.gitignore` ensures files generated by local `pip install`s, Jupyter notebook runs, caches from code runs, virtual environments, and more are not picked up by `git` and accidentally added to the repo. +- `.pylintrc` configures `pylint`, a tool for autochecking code quality. +- `.readthedocs.yaml` configures `readthedocs` and is necessary for documentation to get auto-rebuilt. +- `CITATION.cff` is citation information for the Journal of Open-Source Software (JOSS) paper associated with this project. +- `LICENSE.txt` allows free usage of this project. +- `README.md` is the text you're reading, hello. +- `linting.py` is a script to run `pylint`. +- `pyproject.toml` governs how this package is set up and installed, including dependencies. ## Citation @@ -121,11 +97,12 @@ See CITATION.cff file as well as the following references. ### Prerequisite -PyNumDiff requires common packages like `numpy`, `scipy`, `matplotlib`, `pytest` (for unittests), `pylint` -(for PEP8 style check). For a full list, you can check the file [pyproject.toml](pyproject.toml) +PyNumDiff requires common packages like `numpy`, `scipy`, and `matplotlib`. For a full list, you can check the file [pyproject.toml](pyproject.toml) In addition, it also requires certain additional packages for select functions, though these are not required for a successful install of PyNumDiff: -* Total Variation Regularization methods: [`cvxpy`](http://www.cvxpy.org/install/index.html) +- Total Variation Regularization methods: [`cvxpy`](http://www.cvxpy.org/install/index.html) +- `pytest` for unittests +- `pylint` for PEP8 style check When using `cvxpy`, our default solver is set to be `MOSEK` (highly recommended), you would need to download their free academic license from their [website](https://www.mosek.com/products/academic-licenses/). Otherwise, you can also @@ -147,7 +124,6 @@ again. Note: If using the optional MOSEK solver for cvxpy you will also need a [MOSEK license](https://www.mosek.com/products/academic-licenses/), free academic license. - ## Usage **PyNumDiff** uses [Sphinx](http://www.sphinx-doc.org/en/stable/) for code documentation. @@ -189,7 +165,6 @@ We will frequently update simple examples for demo purposes, and here are curren * Parameter Optimization with known ground truth (only for demonstration purpose): [2a_optimizing_parameters_with_dxdt_known.ipynb](examples/2a_optimizing_parameters_with_dxdt_known.ipynb) * Parameter Optimization with unknown ground truth: [2b_optimizing_parameters_with_dxdt_unknown.ipynb](./examples/2b_optimizing_parameters_with_dxdt_unknown.ipynb) - ### Important notes * Larger values of `tvgamma` produce smoother derivatives @@ -197,18 +172,15 @@ We will frequently update simple examples for demo purposes, and here are curren * The optimization is not fast. Run it on subsets of your data if you have a lot of data. It will also be much faster with faster differentiation methods, like savgoldiff and butterdiff, and probably too slow for sliding methods like sliding DMD and sliding LTI fit. * The following heuristic works well for choosing `tvgamma`, where `cutoff_frequency` is the highest frequency content of the signal in your data, and `dt` is the timestep: `tvgamma=np.exp(-1.6*np.log(cutoff_frequency)-0.71*np.log(dt)-5.1)` - ### Running the tests -We are using Travis CI for continuous intergration testing. You can check out the current status -[here](https://travis-ci.com/github/florisvb/PyNumDiff). +We are using GitHub Actions for continuous intergration testing. To run tests locally, type: ```bash > pytest pynumdiff ``` - ## License This project utilizes the [MIT LICENSE](LICENSE.txt). diff --git a/linting.py b/linting.py index 7065d35..fc71cfe 100644 --- a/linting.py +++ b/linting.py @@ -1,20 +1,13 @@ # -*- coding: utf-8 -*- -import re -import sys -import argparse +import re, sys, argparse from pylint import lint +# Call `echo $?` to see the exit code after a run. If the score is over this, the exit +# code will be 0 (success), and if not will be nonzero (failure). THRESHOLD = 8.5 if len(sys.argv) < 2: raise argparse.ArgumentError("Module to evaluate needs to be the first argument") sys.argv[1] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[1]) -run = lint.Run([sys.argv[1]], do_exit=False) -score = run.linter.stats['global_note'] - -if score < THRESHOLD: - print("Your code doesn't pass the PEP8 style score threshold: %f!" % THRESHOLD) - sys.exit(1) - -print("Congratulations! Your code has passed the PEP8 style score threshold: %f!" % THRESHOLD) \ No newline at end of file +run = lint.Run([sys.argv[1], f"--fail-under={THRESHOLD}"]) diff --git a/pynumdiff/.gitignore b/pynumdiff/.gitignore deleted file mode 100644 index f528fc4..0000000 --- a/pynumdiff/.gitignore +++ /dev/null @@ -1 +0,0 @@ -_version.py \ No newline at end of file diff --git a/pynumdiff/README.md b/pynumdiff/README.md new file mode 100644 index 0000000..d93ad59 --- /dev/null +++ b/pynumdiff/README.md @@ -0,0 +1,11 @@ +- `finite_difference` contains basic first and second order finite differencing methods. The first order method supports iterative application. +- `kalman_smooth` contains Kalman filtering and smoothing methods, currently constant-derivative methods up to 3rd order (jerk) and a classic linear Kalman Filter based on known dynamics. +- `linear_model` is a bit of a miscellaneous module, containing methods which work linearly: `lineardiff`, `polydiff`, `savgoldiff`, and `spectraldiff`. +- `optimize` contains code to find best parameter settings for methods, tuned using Nelder-Mead according to the paper "Numerical differentiation of noisy data: A unifying multi-objective optimization framework" +- `smooth_finite_difference` contains methods which do a smoothing step followed by simple finite difference. +- `tests` contains `pytest` unit tests of + 1. all the differentiation methods, checking their results against a suite of known analytic functions (including an ability to plot if the `--plot` command is passed to `pytest`, see `conftest.py`) + 2. the optimizer + 3. utilities, auxiliary functions used throughout the code +- `total_variation_regularization` contains code to take the derivative based on a finite differencing scheme which is regularized by shrinking changes of value in some derivative (1st, 2nd, or 3rd order) +- `utils` contains `utility` functions used throughout differentation methods, `evaluate` functions used by the parameter optimizer, and `simulate` examples for demonstrating and testing the methods. From 416136d4b50827fc9b3e1c9be4157176febe8f68 Mon Sep 17 00:00:00 2001 From: pavelkomarov Date: Wed, 25 Jun 2025 13:34:10 -0700 Subject: [PATCH 2/2] added a message about the --plot flag I've worked so hard on --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index d8d192f..2d7c99d 100644 --- a/README.md +++ b/README.md @@ -181,6 +181,8 @@ To run tests locally, type: > pytest pynumdiff ``` +Add the flag `--plot` to see plots of the methods against test functions. + ## License This project utilizes the [MIT LICENSE](LICENSE.txt).