Skip to content

Conversation

@JackByrne
Copy link

Description

This PR improves rendering performance for large docxtpl templates by removing a major XML manipulation bottleneck and reducing repeated per-render overhead in XML parsing, regex compilation, and Jinja2 environment setup.

The primary change avoids replacing the entire <w:body> element during rendering and instead mutates its children in place. On very large documents, this reduces render time from hours to minutes.


Problem

Rendering large templates (especially those with many tables and repeated data blocks) is extremely slow. Profiling showed that the dominant cost came from replacing the <w:body> element via root.replace(body, new_body) during render.

Replacing a large XML subtree triggers expensive libxml2 operations, including:

  • Detaching the existing element
  • Validating the incoming subtree
  • Re-parenting and namespace reconciliation
  • Fixing sibling links
  • Re-homing or deep-copying nodes

For documents with millions of nodes, this results in effectively O(size of subtree) work and becomes the main performance bottleneck.

Additional overhead came from:

  • Repeated compilation of the same regex patterns
  • Recreating Jinja2 Environment objects on every render path
  • Parsing XML into raw lxml elements that later need reconciliation with python-docx OXML elements

Current behavior

  • Rendering replaces the <w:body> element entirely using root.replace(...)
  • XML parsing in fix_tables() uses etree.fromstring(), producing raw lxml elements
  • Regex patterns in patch_xml() and resolve_listing() are compiled on every render
  • Jinja2 Environment objects are repeatedly constructed

On large real-world templates, this can result in render times of multiple hours.


Expected behavior

  • Rendering should mutate the existing <w:body> element rather than replacing it
  • XML should be parsed into python-docx OXML element types where possible
  • Regex compilation and Jinja2 environment creation should be reused across renders
  • Rendering output should remain unchanged

Fix

1. Mutate <w:body> children instead of replacing the element

  • map_tree() now removes existing <w:body> children and appends rendered children in order
  • This preserves the original body element identity used by python-docx
  • Avoids expensive subtree replacement and reconciliation

2. Use parse_xml() in fix_tables() with safe fallback

  • Primary parse uses docx.opc.oxml.parse_xml() to ensure OXML element subclasses
  • Falls back to etree.fromstring(..., recover=True) to preserve robustness for malformed XML

3. Ensure new table grid columns use OXML elements

  • New <w:gridCol> elements are created using OxmlElement and qn(...)
  • Avoids mixing raw lxml elements with python-docx OXML elements

4. Precompile commonly used regex patterns

  • Regexes in patch_xml() and resolve_listing() are compiled once and reused

5. Cache Jinja2 environments

  • Reuses cached Environment instances (autoescape and non-autoescape)
  • Avoids repeated environment creation across render paths

6. Header/footer fast path

  • Skips rendering/parsing for headers and footers that contain no Jinja tags

Performance impact

On a large real-world template:

  • Before: ~3 hours render time
  • After: ~23 minutes render time

The majority of the improvement comes from avoiding <w:body> replacement.


Backward compatibility

  • Output is intended to be equivalent to the previous implementation
  • Behavior for malformed XML is preserved via fallback parsing
  • Caller-provided Jinja2 environments are still respected

Notes

  • All existing tests have been run and pass successfully.
  • Manual verification was performed on large real-world templates.
  • Rendering behavior and output are intended to remain unchanged; this PR focuses strictly on performance.

Bonggoprasetyanto and others added 4 commits December 24, 2025 13:57
- Add try/except fallback with recover=True for malformed XML in fix_tables()
- Use OxmlElement with qn() instead of etree.SubElement for new grid columns
- Remove unused _cached_jinja_env variables
- Clean up redundant comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants