Skip to content

Reading 1,000,000 lines (or more) CSVs using Parallelism

Notifications You must be signed in to change notification settings

rodrigocitadin/reports

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

24 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

How to run

First, clone the repo:

git clone https://github.com/rodrigocitadin/reports && cd reports

Then, install deps

mix deps.get

So, run iex

iex -S mix

Normal version

Reports.build "report_complete.csv"

Parallel version

reports = Enum.map(1..10, fn _ -> "splitted_report.csv" end)

Reports.build_many reports

Time comparison

# Normal
:timer.tc fn -> Reports.build("report_complete.csv") end

# Parallel
reports = Enum.map(1..10, fn _ -> "splitted_report.csv" end)

:timer.tc fn -> Reports.build_many(reports) end

Time is in microseconds

Tests

mix test

About

Reading 1,000,000 lines (or more) CSVs using Parallelism

Resources

Stars

Watchers

Forks

Languages