Import error when turning on multithreading

[jonathan-lord-compsci]

When I leave out the --xule-multi option, the DQC runs without error. However, when I provide the --xule-multi option, the DQC fails with the following output:

Rule Set Loaded 0:00:00.174394
Traceback (most recent call last):
  File "/tmp/Arelle/arelleCmdLine.py", line 17, in <module>
    CntlrCmdLine.main()
  File "/tmp/Arelle/arelle/CntlrCmdLine.py", line 47, in main
    parseAndRun(args)
  File "/tmp/Arelle/arelle/CntlrCmdLine.py", line 428, in parseAndRun
    cntlr.run(options)
  File "/tmp/Arelle/arelle/CntlrCmdLine.py", line 870, in run
    pluginXbrlMethod(self, options, modelXbrl, _entrypoint)
  File "/tmp/Arelle/arelle/plugin/xule/__init__.py", line 398, in xuleCmdXbrlLoaded
    runXule(cntlr, options, modelXbrl)
  File "/tmp/Arelle/arelle/plugin/xule/__init__.py", line 421, in runXule
    from .XuleMultiProcessing import start_process
ImportError: No module named 'xule.XuleMultiProcessing'

I cannot find any definition for this module within the DQC codebase.

Without multithreaded support, my tests indicate that version 5 of the DQC is significantly slower than previous versions of the DQC on multiprocessor machines.

I am running Arelle and DQC from source (master as of 2018-01-30) on CentOS 6. Please let me know if you require any additional information.

[campbellpryde]

Jonathan Are you running this from the webservice? Yes you are correct it will be slower with the configuration you have, the reason for this is that running time includes the loading of the US GAAP taxonomy. This can take up to 20 seconds to load in Arelle. In addition time is lost in loading other files and processing calculation relationships. These process contribute to the time increase. The previous version of the rules in python did not have the ability to read the US GAAP taxonomy and loaded straight JSON files which was faster. However as time goes on these files become far more numerous and varied for each taxonomy version and increase the processing time as well. To get around these time constraints the US GAAP taxonomy and other constants should be preloaded into memory. This eliminates much of the time delays. In addition, to speed arelle up we added multi processing to a server version that allows the rules to be split up over multiple cores. This is not supported in the base version of arelle but the option was left in the xule plugin, which you can see in the code. We are working with Herm Fisher to see if he can add changes to Arelle to support processing over multiple cores. Note this is limited and should improve once Arelle is updated to C. However we did not provide the server version of Arelle with the DQC release as this would fork from the current Arelle code base, and wanted to wait until this is added to the base Arelle code. When the rules run with the taxonomy preloaded the time to process becomes faster than the previous DQC releases. This can be seen by running the rules on the XBRL US website which preloads the taxonomy. Regards Campbell Pryde President and CEO

[jonathan-lord-compsci]

I have the DQC running both from the command line and as a web service. Both methods are slower when running version 5 of the DQC versus version 4 of the DQC. I just ran a test against the XBRL US-hosted service and it indeed is significantly faster than what I am experiencing. If I understand correctly, configuring multithreading is not an option with the GitHub-hosted Arelle and DQC codebases. However, preloading taxonomies and constants is an option. Could you please point me to documentation for preloading taxonomies and constants?