nn.GPU

[nicholas-leonard]

This PR adds nn.GPU which can be used to distribute modules across multiple devices.

OMG why is it in nn and not in cunn? As discussed with @soumith, putting in nn means that it can be used in CPU-only environments, which is common for production.

The unit tests are located in cunn to avoid a pcall(function() require cunn end) in the nn unit tests (see PR torch/cunn#282).

[iamalbert]

I am wondering if this line can run in CPU-only environments

[nicholas-leonard]

the cutorch.getDevice will not. But it isn't mandatory to use cutorch.getDevice(), this is just an example.

[nicholas-leonard]

I have tested the nn.GPU implementation on a language model task using 4 GPUs and it works.

[iamalbert]

This is cool but the name and the order of parameters is .. a little confusing

:add(nn.OnGPU(1, nn.Linear(10000,10000))
:add(nn.OnGPU(2, nn.Linear(10000,10000))
:add(nn.OnGPU(3, nn.Linear(10000,10000))
:add(nn.OnGPU(4, cutorch.getDevice(), nn.Linear(10000,10000)))

may be clearer than

:add(nn.GPU(nn.Linear(10000,10000), 1))
:add(nn.GPU(nn.Linear(10000,10000), 2))
:add(nn.GPU(nn.Linear(10000,10000), 3))
:add(nn.GPU(nn.Linear(10000,10000), 4, cutorch.getDevice()))

how do you think?

[nicholas-leonard]

@iamalbert I prefer nn.GPU, but I like your order. Except for the optional outdevice argument which I think should still be last (as it is optional):

:add(nn.GPU(1, nn.Linear(10000,10000))
:add(nn.GPU(2, nn.Linear(10000,10000))
:add(nn.GPU(3, nn.Linear(10000,10000))
:add(nn.GPU(4, nn.Linear(10000,10000)), cutorch.getDevice())

[szagoruyko]

The order is module, device in ModelParallel already, it would be confusing if we introduce another order

[nicholas-leonard]

@szagoruyko Good point. The argument of backwards compatibility always wins (plus I am lazy). I will leave it as it is.

[szagoruyko]

@nicholas-leonard there is a couple of places like self.output = output that break tensor sharing and optnet, can we avoid it?

[nicholas-leonard]

@szagoruyko how does it break tensor sharing (I don't know anyone that shares outputs). As for optnet, what does it do that cannot support self.output = output? because I am pretty sure I do this in many other places.

[fmassa]

Hi,

Sorry for the delay in replying.

The current underlying requirement in optnet is that the tensor/storage objects corresponding to the output and gradInput of each module doesn't change across runs of forward/backward.
There are two cases actually:

• when one wants to create graph visualizations, the tensor shouldn't change.
  Thus doing something like the following will currently not work (although there is a pending PR which tries to fix it, but I'm not 100% sure it doesn't have bad side effects):

function MySelectModule:updateOutput(input)
  self.output = input[1] -- creates a new tensor which shares the storage
  return self.output
end

• when we care only about optimizing for memory, optnet only looks for the storages. So a module which allocates new memory for the output/gradInput during each forward/backward won't be able to reuse buffers, but I think that it shouldn't cause any problems wrt the correctness of the entire network forward/backward.

About this PR, as it's a generic module that supports both tensors and tables, using set is not an option. But from a quick look I have the impression that this module could potentially work as is with optnet, but I haven't tried it to check.

[nicholas-leonard]

So then I guess this PR is ready to merge then :)

[nicholas-leonard]

[soumith]

this :cuda() is no executing in the context of "device". needs a fix

[nicholas-leonard]

It is : https://github.com/nicholas-leonard/nn/blob/db7b97209e3a4b540d51447be6ebfaf3b8fc28a9/GPU.lua#L158-L162

[soumith]

Thanks for your patience Nicholas!

[szagoruyko]

just realized, we need to add support for other cuda datatypes than float

[soumith]

he seems to be checking for Cuda*Tensor, is that not sufficient?