Memory leak with many async requests on a single client #41
Description
I believe I have isolated a memory leak in net-http2 (0.18.2) or http-2 (0.10.1). It appears that the client is hanging onto the requests unless client.close is called.
I followed the format of #7 - run the below code and then use the script at https://blog.skylight.io/hunting-for-leaks-in-ruby/ to generate a diff
# frozen_string_literal: true
require 'net-http2'
require 'objspace'
ObjectSpace.trace_object_allocations_start
STREAMS = 20
COUNT = 1000 / STREAMS
DUMPS = 3
def make_requests(client, j)
COUNT.times do |i|
STREAMS.times do |h|
puts "#{j}:#{i}:#{h}"
request = client.prepare_request(:get, "/?#{rand * 1_000_000}")
request.on(:headers) { |_headers| puts "headers - #{request.object_id}" }
request.on(:body_chunk) { |_chunk| puts "chunk - #{request.object_id}" }
request.on(:close) { puts "close - #{request.object_id}" }
client.call_async(request)
end
client.join
sleep 0.5
end
#client.close - uncomment to clear memory leak
end
client = NetHttp2::Client.new("https://http2.akamai.com/")
client.on(:error) { |e| puts 'error' }
DUMPS.times do |j|
make_requests(client, j)
file_name = "heap-#{j}.dump"
file = File.open(file_name, 'w')
GC.start
ObjectSpace.dump_all(output: file)
file.close
end
$ ruby make_requests.rb # without client.close
...
$ ./diff.rb heap-0.dump heap-1.dump heap-2.dump
Leaked 16988 ARRAY objects of size 0/679520 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:41
Leaked 8989 DATA objects of size 0/719120 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:12
Leaked 6993 IMEMO objects of size 0/279720 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:12
Leaked 3998 DATA objects of size 0/303848 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:696
Leaked 3998 IMEMO objects of size 0/159920 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:696
Leaked 3998 DATA objects of size 0/319840 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/callbacks.rb:10
Leaked 2995 ARRAY objects of size 0/119800 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/callbacks.rb:9
Leaked 2050 IMEMO objects of size 0/82000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/callbacks.rb:10
Leaked 2000 DATA objects of size 0/172000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:687
Leaked 1999 IMEMO objects of size 0/79960 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:41
Leaked 1999 HASH objects of size 0/1159072 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:41
Leaked 1998 OBJECT objects of size 0/287712 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:674
Leaked 1998 ARRAY objects of size 0/79920 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/stream.rb:88
Leaked 1997 DATA objects of size 0/159760 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:41
Leaked 1000 OBJECT objects of size 0/112000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/client.rb:45
Leaked 1000 DATA objects of size 0/64000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/stream.rb:13
Leaked 1000 STRING objects of size 0/40000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/stream.rb:8
Leaked 1000 OBJECT objects of size 0/104000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/client.rb:86
Leaked 1000 HASH objects of size 0/232000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:315
Leaked 1000 STRING objects of size 3000/40000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:28
Leaked 1000 STRING objects of size 20000/40000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:32
Leaked 1000 HASH objects of size 0/232000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:17
Leaked 1000 HASH objects of size 0/232000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/callbacks.rb:8
Leaked 999 HASH objects of size 0/231768 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/stream.rb:7
Leaked 999 STRING objects of size 19134/39960 at: make_requests.rb:17
Leaked 997 DATA objects of size 0/71784 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/stream.rb:12
Leaked 997 HASH objects of size 0/231304 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:19
Leaked 996 HASH objects of size 0/231072 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:22
Leaked 23 ARRAY objects of size 0/920 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/compressor.rb:181
Leaked 1 STRING objects of size 27/68 at: make_requests.rb:37
Total Size: 42161/6505068
$ ruby make_requests.rb # without client.close
...
$ ./diff.rb heap-0.dump heap-1.dump heap-2.dump
Leaked 2 STRING objects of size 54/136 at: make_requests.rb:37
Total Size: 54/136