10 24 memory explosion

[jneums]

Problem

The JSON parser in Parser.mo uses Array.append() in tight loops within parseObject() and parseArray() functions. Since arrays are immutable in Motoko, each append operation creates a full copy of both arrays, resulting in O(n²) memory growth when parsing large JSON documents.

This causes severe memory bloat - applications parsing moderately-sized JSON responses (e.g., API responses with dozens of objects) can see memory usage spike to hundreds of megabytes. The dip in the graph is from reinstalling erasing non-stable memory being held by arrays in the json library, and the flatline is after replacing array with buffer.

Solution

Replace array concatenation with Buffer for O(1) append operations:

In parseObject():

• Changed var fields : [(Text, Types.Json)] = [] to let fields = Buffer.Buffer<(Text, Types.Json)>(8)
• Replaced fields := Array.append(fields, [(next.0, next.1)]) with fields.add(next)
• Convert to array once at the end: #ok(#object_(Buffer.toArray(fields)))

In parseArray():

• Changed var elements : [Types.Json] = [] to let elements = Buffer.Buffer<Types.Json>(16)
• Replaced elements := Array.append(elements, [next]) with elements.add(next)
• Convert to array once at the end: #ok(#array(Buffer.toArray(elements)))

Impact

• Memory complexity: Reduced from O(n²) to O(n)
• Real-world testing: Memory usage reduced from 247MB → 20.8MB (92% reduction) when parsing API responses with ~130 objects
• Performance: Faster parsing due to elimination of repeated array copies

Files Changed

• src/Parser.mo - Modified parseObject() and parseArray() methods

[jneums]

Converted to a draft because the issue popped up again when I added Json.parse back in another place. Will look into it more and try to find out what is causing it.