If `setup(config_name)` is called with a `config_name` that already exists, remove old config_name folder.