Error restoring full config

[AJGilDiaz]

Hi, I am getting an error when trying to restore the whole config for a domain or all domains:

[alex@localhost python3.9]$ incap site restore --domain=DOMAIN_NAME /home/alex/imperva/backup/CONF_FILE.json
Traceback (most recent call last):
File "/home/alex/.local/bin/incap", line 8, in
sys.exit(main())
File "/home/alex/.local/lib/python3.9/site-packages/cwafcli/main.py", line 9, in main
sys.exit(clidriver.main(sys.argv[1:]))
File "/home/alex/.local/lib/python3.9/site-packages/cwafcli/Utils/clidriver.py", line 896, in main
args.func(args)
File "/home/alex/.local/lib/python3.9/site-packages/cwafcli/Utils/cSite_restore.py", line 22, in c_site_restore
config = IncapConfigurations()
File "/home/alex/.local/lib/python3.9/site-packages/cwafcli/Config/configuration.py", line 10, in init
config[profile] = {'id': api_id,
File "/usr/local/lib/python3.9/configparser.py", line 974, in setitem
self.read_dict({key: value})
File "/usr/local/lib/python3.9/configparser.py", line 754, in read_dict
self.set(section, key, value)
File "/usr/local/lib/python3.9/configparser.py", line 1200, in set
self._validate_value_types(option=option, value=value)
File "/usr/local/lib/python3.9/configparser.py", line 1185, in _validate_value_types
raise TypeError("option values must be strings")
TypeError: option values must be strings

Any idea?

[joeymoore]

Thank you, I think I recall this and will check to see if there is a fix. Stay tune

[AJGilDiaz]

Hi Joe,

I've been testing and I changed the configuration.py variables from:

    config[profile] = {'id': api_id,
                       'key': api_key,
                       'account': account,
                       'baseUrl': baseurl,
                       'repo': repo}

to this:

    config[profile] = {'id': str(api_id),
                       'key': str(api_key),
                       'account': str(account),
                       'baseUrl': str(baseurl),
                       'repo': str(repo)}

But now I have another error, the script is not reading properly the variables:

export IMPV_API_ID=xxx
export IMPV_API_KEY=xxx
export IMPV_ACCOUNT_ID=xxx
export IMPV_BASEURL=https://my.incapsula.com/api/prov/v1/
export IMPV_REPO=/backup

Is not getting variables from config.ini neither local env:

[root@imperva-cli Config]# incap site restore --domain=DOMAIN /backup/Domain.json
Creating DOMAIN.com from DOMAIN.com template
Traceback (most recent call last):
File "/usr/local/bin/incap", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/cwafcli/main.py", line 9, in main
sys.exit(clidriver.main(sys.argv[1:]))
File "/usr/local/lib/python3.6/site-packages/cwafcli/Utils/clidriver.py", line 896, in main
args.func(args)
File "/usr/local/lib/python3.6/site-packages/cwafcli/Utils/cSite_restore.py", line 32, in c_site_restore
recover_site(file, param)
File "/usr/local/lib/python3.6/site-packages/cwafcli/Utils/cSite_restore.py", line 49, in recover_site
result = create(params)
File "/usr/local/lib/python3.6/site-packages/cwafcli/Sites/cSite.py", line 42, in create
return execute(resturl, params)
File "/usr/local/lib/python3.6/site-packages/cwafcli/Utils/executeRest.py", line 22, in execute
param["api_id"] = os.getenv("IMPV_API_ID", IncapConfigurations.get_config(param['profile'], 'id'))
KeyError: 'profile'

After I run this the config.ini changes to:

[root@imperva-cli .incap]# cat config.ini
[api]
id = xxx
key = xxx
account = xxx
baseurl = https://my.incapsula.com/api/prov/v1/
repo = /backup
[None]
id = None
key = None
account = None
baseurl = None
repo = None

a new entry is added in blank:

[None]
id = None
key = None
account = None
baseurl = None
repo = None

I also tried to pass the variables in the command but I can't pass the base URL:

[root@imperva-cli Config]# incap site restore --api_id=xxx --api_key=xxx --account_id=xxx --domain=DOMAIN /backup/DOMAIN.json
Creating domain.com from domain.com template
Traceback (most recent call last):
File "/usr/local/bin/incap", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/cwafcli/main.py", line 9, in main
sys.exit(clidriver.main(sys.argv[1:]))
File "/usr/local/lib/python3.6/site-packages/cwafcli/Utils/clidriver.py", line 896, in main
args.func(args)
File "/usr/local/lib/python3.6/site-packages/cwafcli/Utils/cSite_restore.py", line 32, in c_site_restore
recover_site(file, param)
File "/usr/local/lib/python3.6/site-packages/cwafcli/Utils/cSite_restore.py", line 49, in recover_site
result = create(params)
File "/usr/local/lib/python3.6/site-packages/cwafcli/Sites/cSite.py", line 42, in create
return execute(resturl, params)
File "/usr/local/lib/python3.6/site-packages/cwafcli/Utils/executeRest.py", line 29, in execute
baseurl = os.getenv("IMPV_BASEURL", IncapConfigurations.get_config(param['profile'], 'baseurl')) or "https://my.imperva.com"
KeyError: 'profile'

Not sure why.. any thoughts?

Regards,

Alex

[joeymoore]

Hello @AJGilDiaz , can you clone the dev branch, there are a lot of changes and this should be fixed in there. Also, there are output changes to the print statements. Please let me know if you need help with anything.
To run this, cd in to the root dir (incapsula-cli) and run "python3 -m cwafcli -h"

[AJGilDiaz]

Hi @joeymoore,

I cloned the dev branch and replaced the cwafcli folder.

I created a new docker to test it, so here are my findings:

When trying to create a new config I get this output --->

[root@imperva-cli-dev site-packages]# python3 -m cwafcli config API_ID API_KEY ACCOUNT_ID
Traceback (most recent call last):
File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.6/site-packages/cwafcli/main.py", line 13, in
main()
File "/usr/local/lib/python3.6/site-packages/cwafcli/main.py", line 9, in main
sys.exit(clidriver.main(sys.argv[1:]))
File "/usr/local/lib/python3.6/site-packages/cwafcli/Utils/clidriver.py", line 48, in main
if "domain" in response:
TypeError: argument of type 'IncapConfigurations' is not iterable

However the config is added correctly to config.ini.

Now I see new options and site status shows more info and list correctly the rules and rewrites, but I am missing the export option now.

It seems that "export" is not an option anymore:

[root@imperva-cli-dev /]# incap site export
usage: incap [options] site [options]
incap [options] site: error: invalid choice: 'export' (choose from 'add', 'list', 'status', 'delete', 'csr', 'upcert', 'delcert', 'add-dc', 'list-dc', 'edit-dc', 'del-dc', 'add-server', 'edit-server', 'del-server', 'configure', 'security', 'acl', 'whitelist')

export and restore are missing from new menu.

Do you happen to know if export option has been replaced with a new command? I couldn't find it in the help menu.

Regards,

Alex

[joeymoore]

@AJGilDiaz You're right, it was removed, can do another pull on the dev?
Also, the additional ADR rules are now in the "Rule" aka IncapRule list function.

[AJGilDiaz]

Hi @joeymoore,

Now I can see export and restore, export works fine, but I am getting this error from restore:

[root@imperva-cli-dev .incap]# incap site restore --domain=DOMAIN.com /backup/DOMAIN.json
Traceback (most recent call last):
File "/usr/local/bin/incap", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/cwafcli/main.py", line 9, in main
sys.exit(clidriver.main(sys.argv[1:]))
File "/usr/local/lib/python3.6/site-packages/cwafcli/Utils/clidriver.py", line 40, in main
response = args.func(args)
File "/usr/local/lib/python3.6/site-packages/cwafcli/Utils/export.py", line 15, in export
if args.site_id is None:
AttributeError: 'Namespace' object has no attribute 'site_id'

usage: incap [options] site restore [options] path
positional arguments:
path The file or directory with multiple files.
optional arguments:
-h, --help show this help message and exit
--account_id ACCOUNT_ID
Numeric identifier of the account to operate on. If
not specified, operation will be performed on the
account identified by the authentication parameters.
--domain DOMAIN The new domain/site you would like to add.

Where should I define site_id ?

Regards,

Alex

[tamsky]

Any updates?
At a minimum, the dev branch needs the following to even start invoking the restore function.
It currently invokes the export function.

diff -r 009f8dfbe297 cwafcli/Sites/__init__.py
--- a/cwafcli/Sites/__init__.py Fri Jul 02 08:37:50 2021 -0700
+++ b/cwafcli/Sites/__init__.py Wed Aug 18 04:00:28 2021 +0000
@@ -6,6 +6,7 @@
 from cwafcli.Sites.uCertificate import u_certificate
 from cwafcli.Sites.uWaf import u_security
 from cwafcli.Sites.uWhitelist import u_whitelist
+from cwafcli.Utils.cSite_restore import c_site_restore
 from cwafcli.Utils.export import export
 
 
@@ -323,5 +324,6 @@
              'If not specified, operation will be performed on the account '
              'identified by the authentication parameters.')
     site_restore_parser.add_argument('--domain', default=None, help='The new domain/site you would like to add.')
+    site_restore_parser.add_argument('--site_id', help='Numeric identifier of the site to operate on.')
     site_restore_parser.add_argument('path', help='The file or directory with multiple files.')
-    site_restore_parser.set_defaults(func=export)
+    site_restore_parser.set_defaults(func=c_site_restore)

[tamsky]

c50c034#r55055409

[joeymoore]

Hello @tamsky, can you and I set up some time to discuss this feature.
The export function is useless without the import however we found that the import was too difficult with site that contained complex configuration. Additionally, Imperva CWAF has change to the policy feature which nullified a lot of the CLI function around ACLs and other.

[tamsky]

Hello @tamsky, can you and I set up some time to discuss this feature.

Certainly. Apologies for not replying sooner. Just seeing it now.
Please look for a private repo invite and an issue we can use to sync up.

The export function is useless without the import however we found that the import was too difficult with site that contained complex configuration.

I found the export function useful by itself, even without a working import.
If it accurately enumerates all resources and cleanly prints their values, it can help drive automation and other actions, such as terraform import. Improving the output to the point where it's clean enough to either parse or massage would be great.

We would not need import/export if proper support for the priority attribute on incapsula_incap_rule resources existed in the terraform provider.

cf. imperva/terraform-provider-incapsula#69

[hashishrajan]

@joeymoore - We are having the same issue, can I confirm if this was looked issue has been investigated since the last message here? Thank you.

[joeymoore]

@hashishrajan @tamsky The import/export in the CLI has not been review and the reason being; PD has developed a new API end point for an account level all covering export to TF formats. So export will create the TF file structure to recreate/restore everything via a simple TF apply. I am asking when this will be released and will update.