Dedupe option in feed processing

[pierre427]

If we look at one feed:

http://botscout.com/last_caught_cache.htm

You can see that the one field we care about 'Bot IP' shows a lot of duplicate entries. It would be nice to have an option to do something like this:

parse_eachline(:separator => "\n") do |event_generator, record|
  m = feed_re.match(record.data)
  next if m.nil?
  next if dedupe(m[:ip])

  event_generator.call() do |event|
    event.type = :scanning
    event.add_ipv4(m[:ip]) do |ipv4_event|
    end
  end
end

And just have the dedupe function keep a hash table for fhe current feed for any fields which is receives, and return true if there's a previous match.

[justfalter]

I imagine being able specify which attributes of an event to use for identity:

dedupe_events_using :ipv4s, :fqdns

Everything else would be business as usual within the feed.
I would handle the 'de-duplication' at the output step, keeping track of seen events as they go by, and not outputting any events that have already been 'seen'.

This way would:

• Avoid adding more logic to the feed implementations. Keeps things focused on parsing.
• Ensure that feed record data is parsed and normalized prior to any sort of de-duplication checking.

[pierre427]

That works, and you maybe could also do field verification before kicking it out.