title: How is OpenCV's template match so fast?

body:
The cross-correlation coefficient can easily be parallelised. I gave it a shot in Pytorch. However, OpenCV's code running in series is faster than my parallelised Pytorch code. I'm trying to find out what OpenCV is doing differently. If I can parallelise their algorithm it would probably lead to real-time cross-correlation coefficient calculation that could be useful in flow applications.
I'm using the following code:
import torch
import numpy as np
import cv2
from scipy.signal import fftconvolve, convolve2d
import time

def opencv(original, template):
    return cv2.matchTemplate(original, template, cv2.TM_CCOEFF_NORMED)

def complex_multiplication(t1, t2):
    real1 = t1[:,:,0]
    real2 = t2[:,:,0]
    imag1 = t1[:,:,1]
    imag2 = t2[:,:,1]

    return torch.stack([real1 * real2 - imag1 * imag2, real1 * imag2 + imag1 * real2], dim = -1)

def fft_convolve_pytorch(original, template, original_pre_fft=False, template_pre_fft=False, signal_sizes=None, template_size=None):
    if signal_sizes is None:
        signal_sizes = original.size()

    if template_size is None:
        template_size = template.size()

    if not original_pre_fft:
        original_fft = torch.rfft(original, 2)
    else:
        original_fft = original

    if not template_pre_fft:
        padded_template = torch.zeros_like(original)
        padded_template[:-(original.shape[0]-template.shape[0]),:-(original.shape[1]-template.shape[1])] = template
        template_fft = torch.rfft(padded_template, 2)
    else:
        template_fft = template

    return torch.irfft(complex_multiplication(original_fft, template_fft), 2, signal_sizes=signal_sizes)[template_size[0]-1:,template_size[1]-1:], original_fft, template_fft

def pytorch_fft_optimised(original, template):
    original = (original - original.mean())
    template = (template - template.mean())

    a1 = torch.ones_like(template)

    template_flip = torch.flip(template, [0,1])

    out, original_fft, _ = fft_convolve_pytorch(original, template_flip)

    sq_term, _, a1_fft = fft_convolve_pytorch(original**2, a1)
    normal_term, _, _ = fft_convolve_pytorch(original_fft, a1_fft, True, True, signal_sizes=original.size(), template_size=template.size())
    normal_term = (normal_term**2)/(template.size(0)*template.size(1))

    image = sq_term - normal_term

    image[image<0] = 0

    template = torch.mul(template,template).sum()
    out = out/torch.sqrt(image*template)

    out[torch.logical_not(torch.isfinite(out))] = 0

    return out

class Correlation(torch.nn.Module):
    def __init__(self, template_size):
        super(Correlation, self).__init__()
        self.a1 = torch.ones(template_size)

    def forward(self, original, template):
        normalised_original = original - original.mean()

        normalised_template = template - template.mean()

        normalised_template_flip = torch.flip(normalised_template, [1,2])

        out, original_fft, _ = self.fft_convolve_pytorch(normalised_original, normalised_template_flip)

        sq_term, _, a1_fft = self.fft_convolve_pytorch(normalised_original**2, self.a1)

        normal_term, _, _ = self.fft_convolve_pytorch(original_fft, a1_fft, True, True, signal_sizes=normalised_original.size(), template_size=normalised_template.size())
        normal_term = (normal_term**2)/(template.size(1)*template.size(2))

        image = sq_term - normal_term

        image[image<0] = 0

        template = torch.mul(normalised_template,normalised_template).sum()
        out = out/torch.sqrt(image*template)

        out[torch.logical_not(torch.isfinite(out))] = 0

        return out

    def fft_convolve_pytorch(self, original, template, original_pre_fft=False, template_pre_fft=False, signal_sizes=None, template_size=None):
        if signal_sizes is None:
            signal_sizes = original.size()

        signal_sizes = signal_sizes[1:]

        if template_size is None:
            template_size = template.size()

        template_size = template_size[1:]

        if not original_pre_fft:
            original_fft = torch.rfft(original, 2)
        else:
            original_fft = original

        if not template_pre_fft:
            padded_template = torch.zeros_like(original)
            padded_template[:, :-(original.shape[1]-template.shape[1]),:-(original.shape[2]-template.shape[2])] = template
            template_fft = torch.rfft(padded_template, 2)
        else:
            template_fft = template

        return torch.irfft(self.complex_multiplication(original_fft, template_fft), 2, signal_sizes=signal_sizes)[:,template_size[0]-1:,template_size[1]-1:], original_fft, template_fft

    def complex_multiplication(self, t1, t2):
        real1 = t1[:,:,:,0]
        real2 = t2[:,:,:,0]
        imag1 = t1[:,:,:,1]
        imag2 = t2[:,:,:,1]

        return torch.stack([real1 * real2 - imag1 * imag2, real1 * imag2 + imag1 * real2], dim = -1)

def pytorch_fft_cuda(original, template):
    correlation_net = Correlation(template.size()).to("cuda")
    return correlation_net(original.to("cuda"), template.to("cuda"))

and the following to benchmark:
benchmark_value = 5000

original_crop = np.random.rand(benchmark_value, 128,128)*256
current_crop = np.random.rand(benchmark_value, 32,32)*256

original_crop_tensor = torch.from_numpy(original_crop).float()
current_crop_tensor = torch.from_numpy(current_crop).float()

t = time.time()
for i in range(benchmark_value):
    opencv(original_crop[i].astype(np.uint8), current_crop[i].astype(np.uint8))
print("opencv", time.time() - t)

t = time.time()
pytorch_fft_cuda(original_crop_tensor, current_crop_tensor)
print("pytorch fft cuda", time.time() - t)

For good measure, I have also implemented a few other techniques. For the sake of brevity, I won't post the code. The techniques are slower than the 'optimised pytorch' technique. The technique is based on the code found here.
Here's the outcome:
opencv 2.048783540725708 (fast opencv method)
matlab 11.689873218536377 (method from [here](https://github.com/Sabrewarrior/normxcorr2-python/blob/master/normxcorr2.py))
pytorch conv2d 17.52115273475647 (Without fft multiplication)
pytorch fft 9.19994854927063 (Does not reuse recalculated fft matrices)
pytorch fft optimised 8.534855604171753 (second method)
pytorch fft cuda 1.308340311050415 (third gpu method)

Pytorch fft optimised is 4 times as slow as the opencv method. Pytorch fft cuda uses the Pytorch fft optimised method and is faster than the opencv method, but could be made a lot faster if it used the same method as the opencv implementation.
The code for the OpenCV method can be found here. I suspect that it might have something to do with the fact that they are using uint8 matrices instead of floats. But it may very well be just a faster algorithm. In that case I would like to know how. I've tried deciphering their code to no avail. I hope someone can help me!


title: Pytorch: why logging fails in DDP?

body:
I would like to use logging in one of the processes managed by Distributed Data Parallel. However, logging print nothing in the following codes (the codes are derived from this tutorial):
#!/usr/bin/python

import os, logging
# logging.basicConfig(level=logging.DEBUG)

def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # Initialize the process group.
    dist.init_process_group('NCCL', rank=rank, world_size=world_size)

def cleanup():
    dist.destroy_process_group()

class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = nn.Linear(10, 10)
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(10, 5)

    def forward(self, x):
        return self.net2(self.relu(self.net1(x)))

def demo_basic(rank, world_size):
    setup(rank, world_size)

    if rank == 0:
        logger = logging.getLogger('train')
        logger.setLevel(logging.DEBUG)
        logger.info(f'Running DPP on rank={rank}.')

    # Create model and move it to GPU.
    model = ToyModel().to(rank)
    ddp_model = DDP(model, device_ids=[rank])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)  # optimizer takes DDP model.

    optimizer.zero_grad()
    inputs = torch.randn(20, 10)  # .to(rank)

    outputs = ddp_model(inputs)

    labels = torch.randn(20, 5).to(rank)
    loss_fn(outputs, labels).backward()

    optimizer.step()

    cleanup()

def run_demo(demo_func, world_size):
    mp.spawn(
        demo_func,
        args=(world_size,),
        nprocs=world_size,
        join=True
    )

def main():
    run_demo(demo_basic, 4)

if __name__ == "__main__":
    main()

However, when I uncomment the 4th line, the logging works. May I know the reason and how to fix the bug please?


title: Loss to minimize the overlap of two collections/sets

body:
I wonder if there is a loss function can measure the overlap of two collections/sets (order doesn't matter).
E.g. the groud truth is a set [a, b, c] and my model prediction is a set [b, e, f], the overlap is [b]. My goal is to maximize the overlap of my prediction.
Do we have a loss function that can measure the size of overlap that I can minimize (negative of) the metric as a result I can maximize the overlap.
(I know one solution may follow the REIFORCE learning that treat the overlap as a reward for each data sample and use the reward to weight the loss, but do we have another solution)
Thank you.


title: How can I give both images and labels to DataLoader in pytorch?

body: I have a 2D tensor which has my frames information and a 1D tensor with the output or label information related to each frame. I make a function and return (frames, labels). Then I give these two to "torch.utils.data.DataLoader()". But the DataLoader is not working for both(frames, labels) and just can iterate for one of them.


title: How to create Affix(Prefix + Suffix) embeddings in NLP

body:
I am working on a named entity recognition task. Traditional method is to concatenate word embeddings and character level embeddings for creating a word representation first. I want to also use affix embeddings to better understand the relation between the tags and the words.

For example the words "Afghanistan" and "Kajikistan" are clear examples of Location. Here the suffix "istan" or "tan" will be useful to identify future "location" tags. So I want to extract the suffixes and prefixes of all the words and create embeddings for them, and then concatenate with the initial word representation. How to achieve this?


title: Expected dimension sizes for pytorch models

body:
I'm struggling with understanding what sort of dimensions my pytorch model needs as input. My model setup is:
import torch
from torch import nn, tensor

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        self.rnn_b = nn.RNN(input_size=input_size,
                           hidden_size=hidden_size,
                           num_layers=1,
                           bias=False)
        self.hidden_size = hidden_size
        self.linearout = nn.Linear(hidden_size, output_size)

    def forward(self, input, hidden):
        out, hidden = self.rnn_b(input, hidden)
        output = self.linearout(out)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

model = RNN(1, 2, 1)

If I call model(tensor(25.), tensor([[0., 0.]])) then I get the error:
IndexError: dimension specified as 1 but tensor has no dimensions

If I call model(tensor([25.]), tensor([[0., 0.]])) then I get the error
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Can someone please explain what's going on here? What is the correct way to format the data for the input to my model?


title: How pytorch calculates the gradient of Conv2d, respect to its input?

body:
Given the following code, I want to understand how conv1. x.register_hook: is being calculated?

try_grad = nn.Conv2d(1,4,kernel_size=1)

def try_grad_assign(x):
  try_grad.data = x

class try_conv_model(nn.Module):
  def __init__(self):
    super(try_conv_model, self).__init__()
    self.conv1 = nn.Conv2d(1, 4, kernel_size=1, bias=False)
    #self.conv2 = nn.Conv2d(1, 4, kernel_size=1, bias=False)
    self.gap = nn.AdaptiveAvgPool2d((1,1))

  def forward(self, x):
    print("-"*150)
    print("before conv1: ",x)
    x = self.conv1(x)
    x.register_hook(lambda x: print("conv1. x.register_hook: ",x))
    #print("before conv2: ",x)
    #x = self.conv2(x)
    print("before AdaptiveAvgPool2D: ",x)
    x = self.gap(x)
    x.register_hook(try_grad_assign)
    print("after AdaptiveAvgPool2D: ",x)
    x = x.view(-1, self.num_flat_features(x))
    return(x)

  def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

conv_model = try_conv_model()

def my_func(z):
  print("z[0]: ",z[0])
  z = z[0] + torch.tensor([[0,0,1,0]])
  z = z[0] - torch.tensor([[0,0,-1,0]])
  print("new_grad: ",z)
  return z

conv_model.conv1.register_forward_hook(lambda x,y,z: print(" \n conv_model.conv1.register_forward_hook: ",x,y,z))
conv_model.conv1.register_backward_hook(lambda x,y,z: print(" \n conv_model.conv1.register_backward_hook: ",x,y,z))
#conv_model.conv1.register_hook(lambda x: print("conv_model.conv1.register_hook: ",conv_model.conv1.register_hook))
#conv_model.conv2.register_forward_hook(lambda x,y,z: print(" \n conv_model.conv2.register_forward_hook: ",x,y,z))
#conv_model.conv2.register_backward_hook(lambda x,y,z: print(" \n conv_model.conv2.register_backward_hook: ",x,y,z))

conv_model.gap.register_forward_hook(lambda x,y,z: print(" \n conv_model.gap.register_forward_hook: ",x,y,z))
conv_model.gap.register_backward_hook(lambda x,y,z: print(" \n conv_model.gap.register_backward_hook: ",x,y,z))
#conv_model.gap.register_hook(lambda x: print("conv_model.gap.register_hook: ",x))

conv_model.conv1.weight.data = torch. tensor([[[[1]]], [[[1]]], [[[1]]], [[[1]]]], dtype = torch.float)
img = torch.ones(1,1,5,5, dtype = torch.float)
print(img)

labels = torch.tensor([2], dtype = torch.long)
print(labels)

criterion = F.cross_entropy
#criterion.register_hook(lambda x: print("babasjbdkajskjs"))
optimizer = optim.SGD(conv_model.parameters(), lr=0.01, momentum=0.9)
epoch = 1

def train(epoch):
  conv_model.train()
  for i in range(epoch):
    print("&"*300)
    print("*"*25)
    for param in conv_model.parameters():
      print(param)
    print("^"*25)
    conv_model.train()
    outputs = conv_model(img)
    outputs.require_grad = True
    #outputs.register_hook(lambda x: print("outputs.register_hook: ", x))
    outputs.register_hook(my_func)
    print("outputs: ",outputs)
    loss = criterion(outputs, labels)
    loss.register_hook(lambda x: print(" \n before backward loss hook: ",x))

    print(" \n before backward () conv_model.conv1.weight.grad: ",conv_model.conv1.weight.grad)
    #print(" \n before backward () conv_model.conv2.weight.grad: ",conv_model.conv2.weight.grad)
    #print(" \n before backward () linear_model.fc1.bias.grad: ",linear_model.fc1.bias.grad)
    loss.backward()
    loss.register_hook(lambda x: print(" \n after backward loss hook: ",x))

    print("loss.grad: ",loss.grad)

    #print(" \n after backward () conv_model.conv2.weight.grad: ",conv_model.conv2.weight.grad)
    print(" \n after backward () conv_model.conv1.weight.grad: ",conv_model.conv1.weight.grad)
    #print(" \n after backward () linear_model.fc1.bias.grad: ",linear_model.fc1.bias.grad)

    print("outputs.grad: ", outputs.grad)

    optimizer.step()
    print("*"*25)
    for param in conv_model.parameters():
      print(param)
    print("^"*25)

train(epoch)

print("model prediction ",conv_model(img).max(1, keepdim=True)[1])

result:

tensor([[[[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]]]])
tensor([2])
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
*************************
Parameter containing:
tensor([[[[1.]]],

        [[[1.]]],

        [[[1.]]],

        [[[1.]]]], requires_grad=True)
^^^^^^^^^^^^^^^^^^^^^^^^^
------------------------------------------------------------------------------------------------------------------------------------------------------
before conv1:  tensor([[[[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]]]])

 conv_model.conv1.register_forward_hook:  Conv2d(1, 4, kernel_size=(1, 1), stride=(1, 1), bias=False) (tensor([[[[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]]]]),) tensor([[[[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]],

         [[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]],

         [[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]],

         [[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]]]], grad_fn=<MkldnnConvolutionBackward>)
before AdaptiveAvgPool2D:  tensor([[[[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]],

         [[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]],

         [[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]],

         [[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]]]], grad_fn=<MkldnnConvolutionBackward>)

 conv_model.gap.register_forward_hook:  AdaptiveAvgPool2d(output_size=(1, 1)) (tensor([[[[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]],

         [[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]],

         [[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]],

         [[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]]]], grad_fn=<MkldnnConvolutionBackward>),) tensor([[[[1.]],

         [[1.]],

         [[1.]],

         [[1.]]]], grad_fn=<ViewBackward>)
after AdaptiveAvgPool2D:  tensor([[[[1.]],

         [[1.]],

         [[1.]],

         [[1.]]]], grad_fn=<ViewBackward>)
outputs:  tensor([[1., 1., 1., 1.]], grad_fn=<ViewBackward>)

 before backward () conv_model.conv1.weight.grad:  None

 before backward loss hook:  tensor(1.)
z[0]:  tensor([ 0.2500,  0.2500, -0.7500,  0.2500])
new_grad:  tensor([[0.2500, 0.2500, 1.2500, 0.2500]])

 conv_model.gap.register_backward_hook:  AdaptiveAvgPool2d(output_size=(1, 1)) (tensor([0.2500, 0.2500, 1.2500, 0.2500]),) (tensor([[[[0.2500]],

         [[0.2500]],

         [[1.2500]],

         [[0.2500]]]]),)
conv1. x.register_hook:  tensor([[[[0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100]],

         [[0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100]],

         [[0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
          [0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
          [0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
          [0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
          [0.0500, 0.0500, 0.0500, 0.0500, 0.0500]],

         [[0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100]]]])

 conv_model.conv1.register_backward_hook:  Conv2d(1, 4, kernel_size=(1, 1), stride=(1, 1), bias=False) (None, tensor([[[[0.2500]]],

        [[[0.2500]]],

        [[[1.2500]]],

        [[[0.2500]]]]), None) (tensor([[[[0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100]],

         [[0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100]],

         [[0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
          [0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
          [0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
          [0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
          [0.0500, 0.0500, 0.0500, 0.0500, 0.0500]],

         [[0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100]]]]),)
loss.grad:  None

 after backward () conv_model.conv1.weight.grad:  tensor([[[[0.2500]]],

        [[[0.2500]]],

        [[[1.2500]]],

        [[[0.2500]]]])
outputs.grad:  None
*************************
Parameter containing:
tensor([[[[0.9975]]],

        [[[0.9975]]],

        [[[0.9875]]],

        [[[0.9975]]]], requires_grad=True)
^^^^^^^^^^^^^^^^^^^^^^^^^
------------------------------------------------------------------------------------------------------------------------------------------------------
before conv1:  tensor([[[[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]]]])

 conv_model.conv1.register_forward_hook:  Conv2d(1, 4, kernel_size=(1, 1), stride=(1, 1), bias=False) (tensor([[[[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]]]]),) tensor([[[[0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975]],

         [[0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975]],

         [[0.9875, 0.9875, 0.9875, 0.9875, 0.9875],
          [0.9875, 0.9875, 0.9875, 0.9875, 0.9875],
          [0.9875, 0.9875, 0.9875, 0.9875, 0.9875],
          [0.9875, 0.9875, 0.9875, 0.9875, 0.9875],
          [0.9875, 0.9875, 0.9875, 0.9875, 0.9875]],

         [[0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975]]]],
       grad_fn=<MkldnnConvolutionBackward>)
before AdaptiveAvgPool2D:  tensor([[[[0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975]],

         [[0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975]],

         [[0.9875, 0.9875, 0.9875, 0.9875, 0.9875],
          [0.9875, 0.9875, 0.9875, 0.9875, 0.9875],
          [0.9875, 0.9875, 0.9875, 0.9875, 0.9875],
          [0.9875, 0.9875, 0.9875, 0.9875, 0.9875],
          [0.9875, 0.9875, 0.9875, 0.9875, 0.9875]],

         [[0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975]]]],
       grad_fn=<MkldnnConvolutionBackward>)

 conv_model.gap.register_forward_hook:  AdaptiveAvgPool2d(output_size=(1, 1)) (tensor([[[[0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975]],

         [[0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975]],

         [[0.9875, 0.9875, 0.9875, 0.9875, 0.9875],
          [0.9875, 0.9875, 0.9875, 0.9875, 0.9875],
          [0.9875, 0.9875, 0.9875, 0.9875, 0.9875],
          [0.9875, 0.9875, 0.9875, 0.9875, 0.9875],
          [0.9875, 0.9875, 0.9875, 0.9875, 0.9875]],

         [[0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975],
          [0.9975, 0.9975, 0.9975, 0.9975, 0.9975]]]],
       grad_fn=<MkldnnConvolutionBackward>),) tensor([[[[0.9975]],

         [[0.9975]],

         [[0.9875]],

         [[0.9975]]]], grad_fn=<ViewBackward>)
after AdaptiveAvgPool2D:  tensor([[[[0.9975]],

         [[0.9975]],

         [[0.9875]],

         [[0.9975]]]], grad_fn=<ViewBackward>)
model prediction  tensor([[3]])

If the gradient being back-propagated by the avg2d layer is

(tensor([[[[0.2500]],

         [[0.2500]],

         [[1.2500]],

         [[0.2500]]]]),)

what is it being convoluted with so that it is giving:

tensor([[[[0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100]],

         [[0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100]],

         [[0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
          [0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
          [0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
          [0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
          [0.0500, 0.0500, 0.0500, 0.0500, 0.0500]],

         [[0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100],
          [0.0100, 0.0100, 0.0100, 0.0100, 0.0100]]]])

If convoluted with img using print(try_grad(img)), it gives the following result:

tensor([[[[-0.1372, -0.1372, -0.1372, -0.1372, -0.1372],
          [-0.1372, -0.1372, -0.1372, -0.1372, -0.1372],
          [-0.1372, -0.1372, -0.1372, -0.1372, -0.1372],
          [-0.1372, -0.1372, -0.1372, -0.1372, -0.1372],
          [-0.1372, -0.1372, -0.1372, -0.1372, -0.1372]],

         [[ 0.3811,  0.3811,  0.3811,  0.3811,  0.3811],
          [ 0.3811,  0.3811,  0.3811,  0.3811,  0.3811],
          [ 0.3811,  0.3811,  0.3811,  0.3811,  0.3811],
          [ 0.3811,  0.3811,  0.3811,  0.3811,  0.3811],
          [ 0.3811,  0.3811,  0.3811,  0.3811,  0.3811]],

         [[-1.4801, -1.4801, -1.4801, -1.4801, -1.4801],
          [-1.4801, -1.4801, -1.4801, -1.4801, -1.4801],
          [-1.4801, -1.4801, -1.4801, -1.4801, -1.4801],
          [-1.4801, -1.4801, -1.4801, -1.4801, -1.4801],
          [-1.4801, -1.4801, -1.4801, -1.4801, -1.4801]],

         [[-0.2196, -0.2196, -0.2196, -0.2196, -0.2196],
          [-0.2196, -0.2196, -0.2196, -0.2196, -0.2196],
          [-0.2196, -0.2196, -0.2196, -0.2196, -0.2196],
          [-0.2196, -0.2196, -0.2196, -0.2196, -0.2196],
          [-0.2196, -0.2196, -0.2196, -0.2196, -0.2196]]]],
       grad_fn=<MkldnnConvolutionBackward>)

So my question is, how is conv2d calculating the gradient respect to the input?
and why does it differ from mine? The way I am manually calculating the gradient, is that wrong or am i missing something?


title: Using BatchNorm in PyTorch gives a significantly low accuracy

body:
I am working on an image classification problem on a Custom Dataset consisting of 1888training images and 800 validation images(8 classes). I have tried applying transfer learning using various models from the torchvision.models library. For each model, I am using pre-trained weights and am only training the final Linear layer which performs classification. I have the following results so far on my validation set(using batch size 32 and using SGD with momentum as the optimizer with lr 0.001)

1. Alexnet - 93
2. VGG16 - 93
3. VGG16_bn - 57
4. Resnet50 - 26
5. VGG19 - 91
6. VGG19_bn - 59

I have repeated the experiments with both model.train() and model.eval() but the results do not seem to change much. So, from the results, I am guessing that the models having BatchNorm layers are performing very poorly compared to other models that don't have them. Any ideas why this might be happening? Any help would be appreciated. Thanks!

I have looked at this post on the forums about a possible solution which involves increasing the momentum value in the BatchNorm constructor.
How exactly am I supposed to make this change? Do I have to manually change the batchnorm.py code or is there any better way to make this happen?


title: What does "*" operator do before the tensor size in PyTorch?

body:
I am now learning building neural networks in PyTorch. Here are the codes cut from the .py file:

x = torch.unsqueeze(torch.linspace(-1, 1, 1000), dim=1)
y = x.pow(2) + 0.1*torch.normal(torch.zeros(*x.size()))

I am quite comfused about the utility of the * operator before  x.size(). I tried to delete it and plot the scatter graph, which was proved the same as the one with * not removed.

I also checked the official documentation of size in https://pytorch.org/docs/stable/tensors.html but I couldn't figure it out.

Image of torch.size item in documentation

I'd appreciate it very much if you may help me.


title: Is there a package called document_base?

body:
I tried to implement YOLOv3 using PyTorch and I found that models had changed to doqu and I changed it accordingly. What I got next was this

ModuleNotFoundError: No module named 'document_base'

When I tried to install it with pip however I got this

ERROR: Could not find a version that satisfies the requirement document_base (from versions: none)
ERROR: No matching distribution found for document_base

Is there a way to install this or are there any workarounds to this??


title: Updating learning rate with Libtorch 1.5 and optimiser options in C++

body:
With the release of the 1.5 stable version of the C++ API for PyTorch, there are some changes in some of the object interfaces. For instance, now

optimizer.options.learning_rate();

won't work (here the optimiser being used is Adam) since learning_rate has changed to lr (see https://github.com/pytorch/pytorch/releases) but moreover the optimiser no longer has options (no member named 'options' in 'torch::optim::Adam'). So my question is: how would one run

optimizer.options.learning_rate();

or update the learning rate

optimizer.options.learning_rate(updatedlearningrate);

with the new release? Any help will be appreciated! Thank you


title: Problems loading model; Pytorch

body:
I am trying to load a model I saved with the following and it gives me this error:
import torch

model = torch.load('./grunet.pkl')
model.eval()

This is the error:
  Traceback (most recent call last):
  File ".\eval.py", line 4, in <module>
    model.eval()
  AttributeError: 'NoneType' object has no attribute 'eval'

Please help!


title: Torchvision's make_grid has a problem with grayscale images?

body:
I am trying to see a batch of grayscale images in PyTorch DataLoader. Noticed that there is a problem with calculation of grid size. However it works for rgb images. I used following snippet code:
images, labels = next(iter(train_loader)) # a batch of images
print(images.size())
# torch.Size([4, 28, 28, 1])   [B-H-W-C] B-number of images in a batch = 4

image_grid = torchvision.utils.make_grid(images)  # making a grid of 4 images
print(image_grid.size())
# torch.Size([28, 122, 26])

image_grid = image_grid.permute(1, 2, 0) # [C-H-W] -> [H-W-C]
print(image_grid.size())
# torch.Size([122, 26, 28])

However, the output should be:
torch.Size([1, 32, 112])

What is wrong here? Does torchvision's make_grid have a problem with grayscale images or am I doing wrong here?
Any help kindly appreciated.


title: pytorch linear regression given wrong results

body:
I implemented a simple linear regression and I’m getting some poor results. Just wondering if these results are normal or I’m making some mistake.

I tried different optimizers and learning rates, I always get bad/poor results

Here is my code:

import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt
from torch.autograd import Variable

class LinearRegressionPytorch(nn.Module):

   def __init__(self, input_dim=1, output_dim=1):

       super(LinearRegressionPytorch, self).__init__()
       self.linear = nn.Linear(input_dim, output_dim)

   def forward(self,x):
       x = x.view(x.size(0),-1)
       y = self.linear(x)
       return y

input_dim=1
output_dim = 1
if torch.cuda.is_available():
   model = LinearRegressionPytorch(input_dim, output_dim).cuda()
else:
   model = LinearRegressionPytorch(input_dim, output_dim)

criterium = nn.MSELoss()
l_rate =0.00001
optimizer = torch.optim.SGD(model.parameters(), lr=l_rate)
#optimizer = torch.optim.Adam(model.parameters(),lr=l_rate)
epochs = 100

#create data
x = np.random.uniform(0,10,size = 100) #np.linspace(0,10,100);
y = 6*x+5
mu = 0
sigma = 5
noise =  np.random.normal(mu, sigma, len(y))
y_noise = y+noise

#pass it to pytorch
x_data = torch.from_numpy(x).float()
y_data = torch.from_numpy(y_noise).float()
if torch.cuda.is_available():
   inputs = Variable(x_data).cuda()
   target = Variable(y_data).cuda()
else:
   inputs = Variable(x_data)
   target = Variable(y_data)

for epoch in range(epochs):

   #predict data
   pred_y= model(inputs)

   #compute loss
   loss = criterium(pred_y, target)

   #zero grad and optimization
   optimizer.zero_grad()
   loss.backward()
   optimizer.step()

   #if epoch % 50 == 0:
   #   print(f'epoch = {epoch}, loss =  {loss.item()}')

#print params
for name, param in model.named_parameters():
   if param.requires_grad:
       print(name, param.data)

There are the poor results :

linear.weight tensor([[1.7374]], device='cuda:0')
linear.bias tensor([0.1815], device='cuda:0')

The results should be weight = 6 , bias = 5


title: Custom LSTM model in Pytorch showing input size mismatch

body:
I have a custom bidirectional LSTM model where the custom part is

- extract the forward and backward last hidden state
- concat those states
- create a fully connected layer and pass it through softmax layer.

The code looks like below:

class customModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(customModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.bilstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=False, bidirectional=True)
        self.fcl = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        # Set initial hidden and cell states
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)

        # Forward propagate LSTM
        out, hidden = self.bilstm(x, (h0, c0))  # out: tensor of shape (batch_size, seq_length, hidden_size)
        #concat hidden state of forward and backword
        fw_bilstm = out[-1, :, :self.hidden_size]
        bk_bilstm = out[0, :, :self.hidden_size]
        concat_fw_bw = torch.cat((fw_bilstm, bk_bilstm), dim = 1)
        fc = nn.Linear(concat_fw_bw, num_classes)
        x = F.relu(fc(x))
        return F.softmax(x)

I use below parameters and input

input_size = 2
hidden_size = 32
num_layers = 1
num_classes = 2

input_embedding = [
    torch.FloatTensor([[-0.8264],  [0.2524]]),
    torch.FloatTensor([[-0.3259],  [0.3564]])
]

Then I create a model object

model = customModel(input_size, hidden_size, num_layers, num_classes)

Which then I use like below:

for item in input_embedding:
    print(item.size())
    for epoch in range(1):
        pred = model(item)
        print (pred)

When I run it, I see for this line out, hidden = self.bilstm(x, (h0, c0)), it shows error

RuntimeError: input must have 3 dimensions, got 2

I am not sure why the model is thinking that input must have 3 dimensions when I explicitly specified input_size=2

What am I missing?


I want to save each individual image in grid image. How should I modify this code? Or is there an answer that I would like to refer to?

class Generator(nn.Module):
def __init__(self):
    super(Generator, self).__init__()

    self.label_emb = nn.Embedding(opt.n_classes, opt.latent_dim)

    self.init_size = opt.img_size // 4  # Initial size before upsampling
    self.l1 = nn.Sequential(nn.Linear(opt.latent_dim, 128 * self.init_size ** 2))

    self.conv_blocks = nn.Sequential(
        nn.BatchNorm2d(128),
        nn.Upsample(scale_factor=2),
        nn.Conv2d(128, 128, 3, stride=1, padding=1),
        nn.BatchNorm2d(128, 0.8),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Upsample(scale_factor=2),
        nn.Conv2d(128, 64, 3, stride=1, padding=1),
        nn.BatchNorm2d(64, 0.8),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Conv2d(64, opt.channels, 3, stride=1, padding=1),
        nn.Tanh(),
    )

..

generator = Generator()

..

def sample_image(n_row, batches_done):
    """Saves a grid of generated digits ranging from 0 to n_classes"""

    # Sample noise
    z = Variable(FloatTensor(np.random.normal(0, 1, (n_row ** 2, opt.latent_dim))))

    # Get labels ranging from 0 to n_classes for n rows
    labels = np.array([num for _ in range(n_row) for num in range(n_row)])
    labels = Variable(LongTensor(labels))
    gen_imgs = generator(z, labels)
    save_image(gen_imgs.data[:i], "images/%d.png" % batches_done, nrow=1, normalize=True)


title: Pytorch nn.Conv1d outputs different results than Tensorflow 2.1 tensorflow.keras.layers.Conv1D

body:
I am trying to preproduce a seq-to-seq model that i have created in PyTorch to Tensorflow 2.1.
The Pytorch model works perfectly, but the output of the same model implemented in tf 2.1 are very bad. After a lot of debugging i found out that the bad results may occur because of the difference in implementation between nn.Conv1d and tf.keras.layers.Conv1D.

In my Pytorch model, i use:

  self.conv = nn.Conv1d(in_channels=ic, out_channels=oc, kernel_size=k, stride=1, padding=0, bias=False)

In my Tensorflow model, i use:

  self.conv = tf.keras.layers.Conv1D(filters=filters, kernel_size=kernel_size, padding='same', use_bias=use_bias)

Does anyone know how can i produce the same results in tf, as in my Pytorch code? Is there any other function that can help me? Any ideas?

Thank you in advance!!


title: How to understand tensors with multiple dimensions?

body:
I've been recently a bit confused about tensors. Say if we have a tensor with shape (3,2,3,4), we mean that in the first dimension there are 3 groups of numbers? OR it just means that there are exactly just 3 numbers in the first dimension?
Then here comes the second question, with the tensor A that has the shape (3,2), why is that the output of torch.max(A,0) returns a group of max values that contains 2 max values instead of 3,considering the fact that there are 3 numbers in the first dimension.
>>>a = torch.randn(3,2)
a tensor([[-1.1254, -0.1549],
        [-0.5308,  1.0427],
        [-0.1268,  1.0866]])
>>>torch.max(a,0)
torch.return_types.max(
values=tensor([-0.1268,  1.0866]),
indices=tensor([2, 2]))

I mean why doesn't it return a list of 3 max values?
Then the third question, if we have two tensors with shape(3,3,10,2) and (2,4,10,1), can we just concatenate these two tensors on the third dimension considering they have the same size on that dimension? If it is feasible, what is the reason behind it?
I'll be much appreciated if you help me understand this!


title: "RuntimeError: expected scalar type Long but found Float" in the forward process

body:
I'm trying to create a model with PyTorch but during the forward, I encounter this issue "RuntimeError: expected scalar type Long but found Float".
Source code:

dataset_loader.py:

import h5py
import torch
import numpy as np

from PIL import Image
from os import listdir
from sklearn.utils import shuffle

WIDTH = 64
HEIGHT = 64
CREATE_DATASET = False

def load_set(path: str):
    dataset = []
    for f in listdir(path):
        dataset.append(np.asarray(Image.open(path + f).resize((WIDTH, HEIGHT)).convert('RGB'), dtype=np.int32).reshape(3, WIDTH, HEIGHT))

    return np.array(dataset, dtype=np.int32)

def create_batch(dataset, labels, batch_size):
    dataset_result = []
    labels_result = []
    data_batch = []
    label_batch = []

    for index in range(len(dataset)):
        if len(data_batch) == batch_size:
            # dataset_result.append(np.array(data_batch, dtype=np.int32))
            # labels_result.append(np.array(label_batch, dtype=np.int32))
            dataset_result.append(data_batch)
            labels_result.append(label_batch)
            data_batch = []
            label_batch = []
        else:
            # data_batch.append(torch.tensor(dataset[index]))
            # label_batch.append(torch.tensor(labels[index]))
            data_batch.append(np.array(dataset[index], dtype=np.int32))
            label_batch.append(np.array(labels[index], dtype=np.int32))

    dataset_result = np.array(dataset_result, dtype=np.int32)
    labels_result = np.array(labels_result, dtype=np.int32)
    return torch.from_numpy(dataset_result), torch.from_numpy(labels_result)
    # return dataset_result, labels_result

def create_dataset():
    dataset_file = h5py.File("dataset.hdf5", "w")

    train_normal = load_set("chest_xray/train/NORMAL/")
    dataset_file.create_dataset("train_normal", train_normal.shape, dtype=np.int32, data=train_normal)

    train_pneumonia = load_set("chest_xray/train/PNEUMONIA/")
    dataset_file.create_dataset("train_pneumonia", train_pneumonia.shape, dtype=np.int32, data=train_pneumonia)

    test_normal = load_set("chest_xray/test/NORMAL/")
    dataset_file.create_dataset("test_normal", test_normal.shape, dtype=np.int32, data=test_normal)

    test_pneumonia = load_set("chest_xray/test/PNEUMONIA/")
    dataset_file.create_dataset("test_pneumonia", test_pneumonia.shape, dtype=np.int32, data=test_pneumonia)

    val_normal = load_set("chest_xray/val/NORMAL/")
    dataset_file.create_dataset("val_normal", val_normal.shape, dtype=np.int32, data=val_normal)

    val_pneumonia = load_set("chest_xray/val/PNEUMONIA/")
    dataset_file.create_dataset("val_pneumonia", val_pneumonia.shape, dtype=np.int32, data=val_pneumonia)

def load_dataset():
    dataset = h5py.File('dataset.hdf5', 'r')

    train_set = np.array(list(dataset["train_normal"]) + list(dataset["train_pneumonia"]), dtype=np.int32)
    test_set = np.array(list(dataset["test_normal"]) + list(dataset["test_pneumonia"]), dtype=np.int32)
    val_set = np.array(list(dataset["val_normal"]) + list(dataset["val_pneumonia"]), dtype=np.int32)

    train_labels = [0] * len(dataset["train_normal"]) + [1] * len(dataset["train_pneumonia"])
    test_labels = [0] * len(dataset["test_normal"]) + [1] * len(dataset["test_pneumonia"])
    val_labels = [0] * len(dataset["val_normal"]) + [1] * len(dataset["val_pneumonia"])

    BATCH_SIZE = 32

    train_set, train_labels = shuffle(np.array(train_set, dtype=np.int32), np.array(train_labels, dtype=np.int32))
    train_set, train_labels = create_batch(train_set, train_labels, BATCH_SIZE)

    test_set, test_labels = shuffle(np.array(test_set, dtype=np.int32), np.array(test_labels, dtype=np.int32))
    test_set, test_labels = create_batch(test_set, test_labels, BATCH_SIZE)

    val_set, val_labels = shuffle(np.array(val_set, dtype=np.int32), np.array(val_labels, dtype=np.int32))
    val_set, val_labels = create_batch(val_set, val_labels, BATCH_SIZE)

    return train_set, train_labels, test_set, test_labels, val_set, val_labels, BATCH_SIZE

network.py:

import torch.nn as nn
import torch.nn.functional as F

class Network(nn.Module):
  def __init__(self):
    super().__init__()
    self.conv1 = nn.Conv2d(in_channels=3, out_channels=6, kernel_size=5)
    self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)

    self.fc1 = nn.Linear(in_features=12*5*5, out_features=120)
    self.fc2 = nn.Linear(in_features=120, out_features=60)
    self.out = nn.Linear(in_features=60, out_features=2)

  def forward(self, t):
    t = self.conv1(t)
    t = F.relu()
    t = F.max_pool2d(t, kernel_size=2, stride=2)

    t = F.relu(self.conv2(t))
    t = F.max_pool2d(t, kernel_size=2, stride=2)

    t = t.reshape(-1, 12 * 4 * 4)
    t = F.relu(self.fc1(t))
    t = F.relu(self.fc2(t))
    t = self.out(t)

    return t

main.py:

import numpy as np
import torch
import torch.optim as optim
import torch.nn.functional as F

import network
import dataset_loader

import matplotlib.pyplot as plt

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

WIDTH = 64
HEIGHT = 64
NEED_TO_CREATE_DATASET = False

if NEED_TO_CREATE_DATASET:
    dataset_loader.create_dataset()

train_set, train_labels, test_set, test_labels, val_set, val_labels, BATCH_SIZE = dataset_loader.load_dataset()

TRAINING_SIZE = len(train_set) * BATCH_SIZE
TESTING_SIZE = len(test_set) * BATCH_SIZE

EPOCHS = 5
LEARNING_RATE = 0.01

network = network.Network().to(device)
optimizer = optim.Adam(network.parameters(), lr=LEARNING_RATE)

training_losses = []
training_accuracies = []

testing_losses = []
testing_accuracies = []

def get_num_correct(preds, labels):
    return preds.argmax(dim=1).eq(labels).sum().item()

def train():
    network.train()
    correct_in_episode = 0
    episode_loss = 0

    for index, images in enumerate(train_set):
        labels = train_labels[index]

        print(images.shape)
        # exit()
        predictions = network(images.type(torch.LongTensor)) # TODO: fix crash "RuntimeError: expected scalar type Long but found Float"
        # predictions = network(images)
        loss = F.cross_entropy(predictions, labels)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        episode_loss += loss.item()
        correct_in_episode += get_num_correct(predictions, labels)

    training_losses.append(episode_loss)
    training_accuracies.append(correct_in_episode * 100 / TRAINING_SIZE)
    print(f"Epoch: {epoch + 1} accuracy: {correct_in_episode * 100 / TRAINING_SIZE:.2f} loss: {episode_loss:.3f}", end="\t")

def test():
    network.eval()
    episode_loss = 0
    correct_in_episode = 0

    with torch.no_grad():
        for index, images in enumerate(test_set):
            labels = test_labels[index]

            predictions = network(images)
            loss = F.cross_entropy(predictions, labels)

            episode_loss = loss.item()
            correct_in_episode += get_num_correct(predictions, labels)

    testing_losses.append(episode_loss)
    testing_accuracies.append(correct_in_episode * 100 / TESTING_SIZE)
    print(f'Validation: Accuracy: {correct_in_episode * 100 / TESTING_SIZE:.2f} loss: {episode_loss:.3f}')

for epoch in range(EPOCHS):
    train()
    test()

fig = plt.figure()

plt.plot(list(range(1, len(training_losses)+1)), training_losses, color='blue')
plt.plot(list(range(1, len(testing_losses)+1)), testing_losses, color='red')

plt.legend(['Train Loss', 'Test Loss'], loc='upper right')
plt.xlabel('number of training examples seen')
plt.ylabel('Loss')

fig = plt.figure()

plt.plot(list(range(1, len(training_accuracies)+1)), training_accuracies, color='blue')
plt.plot(list(range(1, len(testing_accuracies)+1)), testing_accuracies, color='red')

plt.legend(['Train Accuracy', 'Test Accuracy'], loc='upper right')
plt.xlabel('number of training examples seen')
plt.ylabel('Accuracy')

Error:
torch.Size([32, 3, 64, 64])
Traceback (most recent call last):
  File "Soluce.py", line 86, in <module>
    train()
  File "Soluce.py", line 50, in train
    predictions = network(images.type(torch.LongTensor)) # TODO: fix crash "RuntimeError: expected scalar type Long but found Float"
  File "/home/thytu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/thytu/Prog/PoC/pool_2021/Day3/Admin-XRAI/Admin/network.py", line 15, in forward
    t = self.conv1(t)
  File "/home/thytu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/thytu/.local/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "/home/thytu/.local/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: expected scalar type Long but found Float

Do you have any ideas chat the issue is?
Thanks for your time and your help.
(PS: It's not about Cross-Entropy, it happens before)


title: RuntimeError: The size of tensor a (32) must match the size of tensor b (7) at non-singleton dimension 1

body:
enter image description here
enter image description here
What's wrong with this?? How to fix it?


title: Initialize optim and saving model when using DataParallel

body:
Are the two marked lines 1) and 2) equivalent when using DataParallel?

model = torch.nn.DataParallel(net).cuda()
optim = torch.optim.Adam(net.parameters(), lr)        # 1)
optim = torch.optim.Adam(net.module.parameters(), lr) # 2)

Or does one of them create weird synchronization issues?

Also, while saving params to file, which one is preferred when using a DataParallel module?

model = torch.nn.DataParallel(net).cuda()
...
torch.save({
  'epoch': epoch,
  'state_dict': model.state_dict(),        # 1)
  'state_dict': model.module.state_dict(), # 2)
  'best_loss': best_loss,
}, filename)


title: How to compute Pairwise L1 Distance matrix on very large images in neighborhood only?

body:
I am working on Deep learning approach for my project. And I need to calculate Distance Matrix on 4D Tensor which will be of size N x 128 x 64 x 64 (Batch Size x Channels x Height x Width). The distance matrix for this type of tensor will of size N x 128 x 4096 x 4096 and it will be impossible to fit this type of tensor in GPU, even on CPU it will require lot of memory. So, I would like to calculate the distance matrix only in some neighborhood pixels (say within radius of 5) and consider this rectangular matrix for further processing in neural network. With this approach my distance matrix will be of size N x 128 x 4096 x 61. It will take less memory in comparison to full distance matrix.

Precisely, I am trying to implement the Convolution Random Walk Networks for Semantic Segmentation. This network needs to calculate the Pairwise L1 Distance for features.

Architecture

Just to add this type of Distance Matrix is usually calculated for image segmentation via spectral clustering.

For Example

X = [[a,b],[c,d]]

L1_dist = [ [0, |a-b|, |a-c|, 0],
            [|a-b|, 0, 0, |b-d|],
            [|a-c|, 0, 0, |c-d| ],
            [0,    |b-d|, |c-d|, 0 ]
          ]

Final_L1_dist = [ [0, |a-b|, |a-c|], // "a" is near to b and c. Including self element i.e. a
            [|a-b|, 0, |b-d|], // "b" is near to a and d.
            [|a-c|, 0, |c-d| ], // "c" is near to a and d.
            [|b-d|, |c-d|, 0 ] // "d" is near to b and c.
          ]

I would appreciate, if some one can help me to find an efficient way to compute such a matrix.

Thanks


title: How to create a list of modulelists

body:
Is it ok to create a python-list of PyTorch modulelists?
If for example, I want to have a few Conv1d in a layer and then another layer with different Conv1d. In each layer I need to do a different manipulation on the output depending on the layer number.  What is the correct way to build this "python-list" of modulelists?

This way:

    class test(nn.Module):
        def __init__(...):
            self.modulelists = []
            for i in range(4):
                self.modulelists.append(nn.ModuleList([nn.Conv1d(10, 10, kernel_size=5) for _ in range(5)]))

or this way:

    class test(nn.Module):
        def __init__(...):
            self.modulelists = nn.ModuleList()
            for i in range(4):
                self.modulelists.append(nn.ModuleList([nn.Conv1d(10, 10, kernel_size=5) for _ in range(5)]))

Thanks


title: how to keep pytorch model in redis cache to access model faster for video streaming?

body:
I have this code belonging to feature_extractor.py which is a part of this folder in here:

import torch
import torchvision.transforms as transforms
import numpy as np
import cv2
from .model import Net

class Extractor(object):
    def __init__(self, model_path, use_cuda=True):
        self.net = Net(reid=True)
        self.device = "cuda" if torch.cuda.is_available() and use_cuda else "cpu"
        state_dict = torch.load(model_path, map_location=lambda storage, loc: storage)['net_dict']
        self.net.load_state_dict(state_dict)
        print("Loading weights from {}... Done!".format(model_path))
        self.net.to(self.device)
        self.size = (64, 128)
        self.norm = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
        ])

    def _preprocess(self, im_crops):
        def _resize(im, size):
            return cv2.resize(im.astype(np.float32) / 255., size)

        im_batch = torch.cat([self.norm(_resize(im, self.size)).unsqueeze(0) for im in im_crops], dim=0).float()
        return im_batch

    def __call__(self, im_crops):
        im_batch = self._preprocess(im_crops)
        with torch.no_grad():
            im_batch = im_batch.to(self.device)
            features = self.net(im_batch)
        return features.cpu().numpy()

if __name__ == '__main__':
    img = cv2.imread("demo.jpg")[:, :, (2, 1, 0)]
    extr = Extractor("checkpoint/ckpt.t7")
    feature = extr(img)
    print(feature.shape)

Now Imagine 200 requests are in row to proceed. The process of loading model for each request makes the code run slowly.

So I thought it might be a good idea to keep the pytorch model in cache. I modified it like this:

from redis import Redis
import msgpack as msg

r = Redis('111.222.333.444')

class Extractor(object):
    def __init__(self, model_path, use_cuda=True):
        try:
            self.net = msg.unpackb(r.get('REID_CKPT'))
        finally:
            self.net = Net(reid=True)
            self.device = "cuda" if torch.cuda.is_available() and use_cuda else "cpu"
            state_dict = torch.load(model_path, map_location=lambda storage, loc: storage)['net_dict']
            self.net.load_state_dict(state_dict)
            print("Loading weights from {}... Done!".format(model_path))
            self.net.to(self.device)
            packed_net = msg.packb(self.net)
            r.set('REID_CKPT', packed_net)

        self.size = (64, 128)
        self.norm = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
        ])

Unfortunately this error comes up:

 File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
 File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
 File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
 File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
 TypeError: can not serialize 'Net' object

The reason obviously is because that it cannot convert Net object (pytorch nn.Module class) to bytes.

How can I efficiently save pytorch model in cache (or somehow keep it in RAM) and call for it for each request?

Thanks everyone.


title: Is there a way to monitor optimizer's step in Pytorch?

body:
Consider that you are using a Pytorch optimizer such as torch.optim.Adam(model_parameters).

So in your training loop you will have something like:

optimizer = torch.optim.Adam(model_parameters)
# put the training loop here

loss.backward()
optimizer.step()
optimizer.zero()

Is there a way to monitor what steps are taking your optimizer ? To make sure that you are not on a flat area and thus taking no steps since the gradient are null. Maybe checking the learning rate would be a solution ?


title: Pytorch weighted Tensor

body:
I'm porting a little bit complex TF2 code to Pytorch. Since TF2 does not distinguish Tensor and numpy array, it was straightforward on it. However, I feel like I came back to the TF1 era when I encountered several errors saying 'you cannot mix Tensor and numpy array here in Pytorch!'. Here is the original TF2 code:
def get_weighted_imgs(points, centers, imgs):
  weights = np.array([[tf.norm(p - c) for c in centers] for p in points], dtype=np.float32)
  weighted_imgs = np.array([[w * img for w, img in zip(weight, imgs)] for weight in weights])

  weights = tf.expand_dims(1 / tf.reduce_sum(weights, axis=1), axis=-1)
  weighted_imgs = tf.reshape(tf.reduce_sum(weighted_imgs, axis=1), [len(weights), 64*64*3])

  return weights * weighted_imgs

And my problematic Pytorch code:
def get_weighted_imgs(points, centers, imgs):
  weights = torch.Tensor([[torch.norm(p - c) for c in centers] for p in points])
  weighted_imgs = torch.Tensor([[w * img for w, img in zip(weight, imgs)] for weight in weights])

  weights = torch.unsqueeze(1 / torch.sum(weights, dim=1), dim=-1)
  weighted_imgs = torch.sum(weighted_imgs, dim=1).view([len(weights), 64*64*3])

  return weights * weighted_imgs

def reproducible():
  points = torch.Tensor(np.random.random((128, 5)))
  centers = torch.Tensor(np.random.random((10, 5)))
  imgs = torch.Tensor(np.random.random((10, 64, 64, 3)))

  weighted_imgs = get_weighted_imgs(points, centers, imgs)

I can guarantee that there is no issue with the dimension order or shape of the tensors/arrays. The error message I got is
ValueError: only one element tensors can be converted to Python scalars

which comes from
weighted_imgs = torch.Tensor([[w * img for w, img in zip(weight, imgs)] for weight in weights])

Could someone help me to solve this problem? That would be greatly appreciated.


title: Is there a `Split` equivalent to torch.nn.Sequential?

body:
A sample code for a Sequential block is
self._encoder = nn.Sequential(
        # 1, 28, 28
        nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3, stride=3, padding=1),
        # 32, 10, 10 = 16, (1//3)(28 + 2 * 1 - 3) + 1, (1//3)(28 + 2*1 - 3) + 1
        nn.ReLU(True),
        nn.MaxPool2d(kernel_size=2, stride=2),
        # 32, 5, 5
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, stride=2, padding=1),
        # 64, 3, 3
        nn.ReLU(True),
        nn.MaxPool2d(kernel_size=2, stride=1),
        # 64, 2, 2
)

Is there some construct like nn.Sequential that puts modules in it in parallel?
I would like to now define something like
self._mean_logvar_layers = nn.Parallel(
    nn.Conv2d(in_channels=64, out_channels=64, kernel_size=2, stride=1, padding=0),
    nn.Conv2d(in_channels=64, out_channels=64, kernel_size=2, stride=1, padding=0),
)

Whose output should be two pipes of data - one for each element in self._mean_logvar_layers which are then feedable to the rest of the network. Kind of like a multi-headed network.

My current implementation:
self._mean_layer = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=2, stride=1, padding=0)
self._logvar_layer = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=2, stride=1, padding=0)

and
def _encode(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
    for i, layer in enumerate(self._encoder):
        x = layer(x)

    mean_output = self._mean_layer(x)
    logvar_output = self._logvar_layer(x)

    return mean_output, logvar_output

I would like to treat the parallel construct as a layer.
Is that doable in PyTorch?


title: Moving PyTorch model prediction results from GPU to CPU seemingly "destroys" the data?

body:
I have a PyTorch model (UNet for semantic segmentation) that I am training and evaluating using this training/prediction implmentation
class Accuracy(nn.Module):
    def __init__(self):
        super(Accuracy, self).__init__()

    def forward(self, outputs, labels, dim=1, train=False):
        predicted = torch.argmax(outputs.data, dim=dim)
        total = labels.nelement()
        correct = (predicted == labels).sum().item()
        return correct / total

The training/validation implementation
import time
import numpy as np

import torch.optim as optim

from util import progress
from hsnet.losses import *

from torch.utils.data import DataLoader

def train(model, train_dataset, val_dataset, batch_size, epochs, lrate, loss, weights):
    print('> Training...')

    print('  Preparing.')
    opt = optim.Adam(model.parameters(), lr=lrate)
    #opt = optim.SGD(model.parameters(), lr=0.000001, momentum=0.9)
    metric_loss = get_loss(loss, torch.Tensor(weights).cuda())

    metrics = [('acc', get_loss('acc'))]

    val_data = DataLoader(val_dataset, batch_size=batch_size)
    train_data = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, pin_memory=True)

    print('  Begin training...')
    history = {
        'train': {name: [] for name in ['loss'] + [m for m, _ in metrics]},
        'val': {name: [] for name in ['loss'] + [m for m, _ in metrics]}
    }
    for e in range(epochs):
        start_time = time.perf_counter()

        # -------------------------
        # Training pass
        # -------------------------
        model.train(mode=True)

        train_batch_history = {name: [] for name in ['loss'] + [m for m, _ in metrics]}
        num_train_batch = len(train_data)
        for b_ind, b_sample in enumerate(train_data):
            # Get the data for this batch and move it to the GPU
            x, y = b_sample['x'], b_sample['y']
            x = torch.FloatTensor(x).cuda()
            y = torch.LongTensor(y).cuda()

            # Perform the actual training
            model.zero_grad()
            pred = model(x)
            loss_value = metric_loss(pred, y)
            loss_value.backward()
            opt.step()
            torch.cuda.synchronize('cuda')

            # Calculate and save the batch loss and metrics
            train_batch_history['loss'].append(loss_value.item())
            for i, (name, metric) in enumerate(metrics):
                train_batch_history[name].append(metric(pred, y, train=True))

            # Report training progress
            suffix = 'train - loss: ' + '{0:.4f}'.format(np.average(train_batch_history['loss']))
            for i, (name, metric) in enumerate(metrics):
                suffix += ' ' + name + ': ' + '{0:.4f}'.format(np.average(train_batch_history[name]))

            progress(b_ind, num_train_batch,
                     prefix='Batch (' + str(b_ind + 1) + '/' + str(num_train_batch) + ') ',
                     suffix=suffix, decimals=1, length=50, fill='#')

        # Save the training history loss and metrics
        history['train']['loss'].append(sum(train_batch_history['loss']) / num_train_batch)
        for (name, metric) in metrics:
            history['train'][name].append(sum(train_batch_history[name]) / num_train_batch)

        # -------------------------
        # Validation pass
        # -------------------------
        model.eval()
        model.train(mode=False)

        val_batch_history = {name: [] for name in ['loss'] + [m for m, _ in metrics]}
        num_val_batch = len(val_data)
        with torch.no_grad():
            for b_ind, b_sample in enumerate(val_data):
                # Get the data for this batch and move it to the GPU
                x, y = b_sample['x'], b_sample['y']
                x = torch.FloatTensor(x).cuda()
                y = torch.LongTensor(y).cuda()

                # Run the validation data through the model
                pred = model(x)
                loss_value = metric_loss(pred, y)

                # Calculate and save the batch loss and metrics
                val_batch_history['loss'].append(loss_value.item())
                for i, (name, metric) in enumerate(metrics):
                    val_batch_history[name].append(metric(pred, y, train=False))

        # Save the training history loss and metrics
        history['val']['loss'].append(sum(val_batch_history['loss']) / num_val_batch)
        for (name, metric) in metrics:
            history['val'][name].append(sum(val_batch_history[name]) / num_val_batch)

        # Report the epoch results
        suffix = 'train - loss: ' + '{0:.4f}'.format(history['train']['loss'][-1])
        for (name, metric) in metrics:
            suffix += ' ' + name + ': ' + '{0:.4f}'.format(history['train'][name][-1])
        suffix += ', val - loss: ' + '{0:.4f}'.format(history['val']['loss'][-1])
        for (name, metric) in metrics:
            suffix += ' ' + name + ': ' + '{0:.4f}'.format(history['val'][name][-1])
        progress(1, 1, prefix='Epoch (' + str(e + 1) + '/' + str(epochs) + ') ({0:.0f}s)'.format(time.perf_counter() - start_time),
                 suffix=suffix, decimals=1, length=50, fill='#')

    return model, history

And the prediction/testing/evaluation implementation
import time
import numpy as np

from util import progress
from hsnet.losses import *

from torch.utils.data import DataLoader

def test(model, test_dataset, batch_size, loss, weights):
    print('> Testing...')

    print('  Preparing.')
    metric_loss = get_loss(loss, torch.Tensor(weights).cuda())
    metrics = [('acc', get_loss('acc'))]

    test_data = DataLoader(test_dataset, batch_size=batch_size)

    print('  Begin testing...')
    history = {'test': {name: [] for name in ['loss'] + [m for m, _ in metrics]}}

    start_time = time.perf_counter()

    # -------------------------
    # Test pass
    # -------------------------
    model.eval()
    model.train(mode=False)

    y_pred = None
    test_batch_history = {name: [] for name in ['loss'] + [m for m, _ in metrics]}
    num_test_batch = len(test_data)
    with torch.no_grad():
        for b_ind, b_sample in enumerate(test_data):
            # Get the data for this batch and move it to the GPU
            x, y = b_sample['x'], b_sample['y']
            x = torch.FloatTensor(x).cuda()
            y = torch.LongTensor(y).cuda()

            # Run the validation data through the model
            pred = model(x)
            loss_value = metric_loss(pred, y)

            # Calculate and save the batch loss and metrics
            test_batch_history['loss'].append(loss_value.item())
            for i, (name, metric) in enumerate(metrics):
                test_batch_history[name].append(metric(pred, y))
                print('AverageAcc: ', metric(pred, y, train=False))

            # Collect the model predictions
            if y_pred is not None:
                y_pred = np.concatenate([y_pred, pred.cpu().detach().numpy()], axis=0)
            else:
                y_pred = pred.cpu().detach().numpy()

            print('y_pred.shape: ', y_pred.shape)

    # Save the training history loss and metrics
    history['test']['loss'].append(sum(test_batch_history['loss']) / num_test_batch)
    for (name, metric) in metrics:
        history['test'][name].append(sum(test_batch_history[name]) / num_test_batch)

    # Report the epoch results
    suffix = 'test - loss: ' + '{0:.4f}'.format(history['test']['loss'][-1])
    for (name, metric) in metrics:
        suffix += ' ' + name + ': ' + '{0:.4f}'.format(history['test'][name][-1])
    progress(1, 1, prefix='Test ({0:.0f}s)'.format(time.perf_counter() - start_time),
             suffix=suffix, decimals=1, length=50, fill='#')

    return model, y_pred, test_batch_history

While training the model if I begin plotting the actual and predicted labels (within AccuracyLoss.forward) from the training set I receive "logical" (relative) predictions

If I also plot actual and predicted labels from the validation set I still receive some logical output

Further, during calls to test I see the model is producing some logical output.

Though, if I take the first sample from y_pred x and y during test I find that the y_pred data (which is what is being used to plot in AccuracyLoss.forward) seems to be "destoryed" (The exact same sample as above)

The specific area where I think the issue is within test where y_pred is accumulated to return to the caller
if y_pred is not None:
    y_pred = np.concatenate([y_pred, pred.cpu().detach().numpy()], axis=0)
else:
    y_pred = pred.cpu().detach().numpy()

This combination of cpu().detach().numpy() seems to necessarily destroy the data that is within pred whilst on the GPU. Though again, x and y which have also been moved to the GPU at the start of the batch
x, y = b_sample['x'], b_sample['y']
x = torch.FloatTensor(x).cuda()
y = torch.LongTensor(y).cuda()

and taken back to CPU land (reshaped for plotting)
hs = x[0].clone().cpu().detach().numpy()
hs = hs.reshape((128, 128, 48))

lb = y[0].clone().cpu().detach().numpy()
lb = lb.reshape((128, 128))

do not seem to be "destroyed".

What is the proper/correct way to accumulate the model predictions from the GPU during validation/testing to CPU memory for further evaluation/plotting under PyTorch?
Maybe the issue is that the model is "doing something" to pred/y_pred even after .clone and moving it to CPU land? I understand that calls to nn.Tensor.numpy() are memory backed (ie the data between a Tensor.numpy() and the resulting numpy object effect each other). Is there a better way to .detach the tensor from the computational graph..?

EDIT: After more tinkering I have found that if I collect y_pred as such
if y_pred is not None:
    y_pred_cpu = torch.argmax(pred.detach().clone(), dim=1).cpu().numpy()
    y_pred = np.concatenate([y_pred, y_pred_cpu], axis=0)
else:
    y_pred = torch.argmax(pred.detach().clone(), dim=1).cpu().numpy()

There is no issue. The data in y_pred is "correct" (though maybe not a great prediction).

So I guess the new, but very related question is, what is the difference between the two proposed methods of collection y_pred?


title: Does the fastai library automatically infer text and label from header?

body:
I have dataframe object which has two fields i.e. text and label and I want to do text classification similar to ULMFit

Here are codes for that with the specific purpose below

data_lm = TextLMDataBunch.from_df(train_df = df_trn, valid_df = df_val, path = "")

# Language model learner object

data_clas = TextClasDataBunch.from_df(path = "", train_df = df_trn, valid_df = df_val, vocab=data_lm.train_ds.vocab, bs=32)

classification object

learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.5)

# learn language model object

learn.fit_one_cycle(10, 1e-2)

# Fit a language model

learn.save_encoder('good_ft_enc')

# save the encoder of the language model

learn = text_classifier_learner(data_clas,AWD_LSTM,drop_mult=0.7)

learn.load_encoder('good_ft_enc')

# classifier object with the loaded encoder

learn.fit_one_cycle(3, 1e-2)

# fit the classifier

These are the series of steps followed , but I am not able to understand if the dataloader has loaded my text and label field properly because I have not given them anywhere in the function arguments. Any help will be greatly appreciated


title: How to do element wise multiplication for two 4D unequal size tensors in pytorch?

body:
I have got a tensor A and Tensor B.

Size of A = [2,64,56,56]

Size of B = [2,64,29,29]

How can I perform torch.mul(A,B)? The tensors are of unequal size.

  RuntimeError: shape [2, 64, 56, 56] is invalid for input of size 107648


title: installing CuPy with Pytorch

body:
I want to pass tensors from Pytorch to CuPy and back to do some ops not available in Pytorch. What's the recommended way to install them together?

I already installed Pytorch using conda, and it has installed cudatoolkit package. I'd prefer to use conda to install cupy as well. However CuPy installation instructions recommend uninstalling cudatoolkit (at the very bottom).

So, how do I make sure both Pytorch and CuPy are using the same cuda version, as well as CuDNN, NVCC, etc? Or should I not use conda to install cupy?


title: How to see the size of a computation graph in memory in Pytorch?

body:
How can one trace the memory allocation for the autograd graph created by the forward pass on cpu? For instance trying to use tracemalloc on cpu:

rnn=nn.RNNCell(100,100).to('cuda')
x=torch.ones((1000,100),device='cuda')
tracemalloc.start(25)
while True:
   print(tracemalloc.get_traced_memory())
   x=rnn(x)

The printed memory should continually increase as the graph is increasing in each loop step, but the printed memory from

tracemalloc.get_traced_memory()

remains constant after the 3rd loop. What is going on?


when I try to finetune the pytorcfaster-rcnn following the official tutorial:
https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html
with the Wider-Face dataset.
I got:
RuntimeError: input and output sizes should be greater than 0, but got input (H: 1024, W: 1) output (H: 800, W: 0)
here is the Dataset getitem function I used:
def __getitem__(self,idx):
        img_path = self.imgs[idx]
        ann_path = self.anns[idx]
        img_id = self.images_id[idx]

        img = Image.open(img_path).convert('RGB')
        img = np.asarray(img)
        img = img/255.0
        img = np.moveaxis(img,2,0)
        img = torch.from_numpy(img).float()

        boxes ,w,h= self.__extract_boxes__(ann_path)
        masks = np.zeros([h,w,len(boxes)],dtype='uint8')
        labels=list()
        for i in range (len(boxes)):
            box = boxes[i]

            row_s,row_e = box[1],box[3]
            col_s,col_e = box[0],box[2]

            masks[row_s:row_e,col_s:col_e,i]=1
            labels.append(1)

        boxes    = torch.as_tensor(boxes,dtype=torch.float32)
        masks    = torch.as_tensor(masks,dtype=torch.uint8)
        labels   = torch.ones((len(labels),),dtype=torch.int64)
        image_id = torch.tensor([idx])
        area     = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])

        target = {}
        target['boxes']=boxes
        target['labels'] = labels
        target['masks']=masks
        target['image_id']=image_id
        target['area']=area

        return img,target


title: pytorch repeat 3rd dimension

body:
I'm following this example on doc
In [42]: x = torch.tensor([1,2,3])

In [45]: x.repeat(4,2)
Out[45]: tensor([[1, 2, 3, 1, 2, 3],
        [1, 2, 3, 1, 2, 3],
        [1, 2, 3, 1, 2, 3],
        [1, 2, 3, 1, 2, 3]])

In [46]: x.repeat(4,2).shape
Out[46]: torch.Size([4, 6])

So far, so good.
But why does repeating just 1 time on 3rd dimension expand 3rd dim to 3 (not 1)?
[On the doc]

>>> x.repeat(4, 2, 1).size()
torch.Size([4, 2, 3])

Double checking.
In [43]: x.repeat(4,2,1)
Out[43]:
tensor([[[1, 2, 3],
         [1, 2, 3]],

        [[1, 2, 3],
         [1, 2, 3]],

        [[1, 2, 3],
         [1, 2, 3]],

        [[1, 2, 3],
         [1, 2, 3]]])

Why does it behave this way?


title: tqdm keeps printing progress bar in the same line

body:
i am trying to train a network, but the progress bar for "tqdm" is not working properly, it keeps printing a new bar one after the other in the same line, i don't know why this is happening, but to my understanding i should i just call tqdm in the for-loop initialization and that's it, also it seems like the estimated time on the progress bar might be off, could this also have to do with how the model is implemented, when i run the code it doesn't allocate the entire VRAM of the GPU.
my code:
import torch
import torch.nn as nn
from tqdm import tqdm

def loss_fn(outputs, targets):
    return nn.BCEWithLogitsLoss()(outputs, targets.view(-1, 1))

def train_fn(data_loader, model, optimizer, device, scheduler):
    model.train()

    for bi, d in tqdm(enumerate(data_loader), total=len(data_loader) , leave = False):
        ids = d["ids"]
        token_type_ids = d["token_type_ids"]
        mask = d["mask"]
        targets = d["targets"]

        ids = ids.to(device, dtype=torch.long)
        token_type_ids = token_type_ids.to(device, dtype=torch.long)
        mask = mask.to(device, dtype=torch.long)
        targets = targets.to(device, dtype=torch.float)

        optimizer.zero_grad()
        outputs = model(ids=ids, mask=mask, token_type_ids=token_type_ids)

        loss = loss_fn(outputs, targets)
        loss.backward()
        optimizer.step()
        scheduler.step()

def eval_fn(data_loader, model, device):
    model.eval()
    fin_targets = []
    fin_outputs = []
    with torch.no_grad():
        for bi, d in tqdm(enumerate(data_loader), total=len(data_loader)):
            ids = d["ids"]
            token_type_ids = d["token_type_ids"]
            mask = d["mask"]
            targets = d["targets"]

            ids = ids.to(device, dtype=torch.long)
            token_type_ids = token_type_ids.to(device, dtype=torch.long)
            mask = mask.to(device, dtype=torch.long)
            targets = targets.to(device, dtype=torch.float)

            outputs = model(ids=ids, mask=mask, token_type_ids=token_type_ids)
            fin_targets.extend(targets.cpu().detach().numpy().tolist())
            fin_outputs.extend(torch.sigmoid(outputs).cpu().detach().numpy().tolist())
    return fin_outputs, fin_targets

the output:


title: Why does torchvision.utils.make_grid() return copies of the wanted grid?

body:
In the below coding example I can not understand why the output tensor , grid has a shape of
3,28,280. I understand why its 28 in height and 280 in width, but not the 3. It seems from running plt.imshow() on all 3 28x280 arrays along axis 0 that they are identical copies since printing any 1 of these gives me the image I want.
Also I do not understand why I can pass grid as an argument to plt.imshow() given that it is supposed to take in a 2D array, not a 3D one as grid clearly is.
import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np

train_set = torchvision.datasets.FashionMNIST(
    root = './pytorch_obj_classifier/data/FashionMNIST',
    train = True,
    download = True,
    transform = transforms.Compose([
            transforms.ToTensor()
    ])
)
sample = next(iter(train_loader))
image,label = sample
print(image.shape)

grid = torchvision.utils.make_grid(image,padding=0, nrow=10)
print(grid.shape)

plt.figure(figsize=(15,15))
grid = np.transpose(grid,(1,2,0))
grid1 = grid[:,:,0]
grid2 = grid[:,:,1]
grid3 = grid[:,:,2]
plt.imshow(grid1,cmap = 'gray')
plt.imshow(grid2,cmap = 'gray')
plt.imshow(grid3,cmap = 'gray')
plt.imshow(grid,cmap = 'gray')


title: Pytorch Error: mat1 dim 1 must match mat2 dim 0 in CNN

body:
I had a couple problems with a NN I took from (https://gist.github.com/johnolafenwa/96b3322aabb61d4d36fd870a77f02aa3)
First I had a problem with output sizes and input batch sizes as described here (How do I modify this PyTorch convolutional neural network to accept a 64 x 64 image and properly output predictions?)
I think now the input batch_size matches the target batch_size, but a new problem accured and I do not have the answer to why this error occurs. Any help would be helpful.
def get_dls(bs, size):
    dblock = DataBlock(blocks=(ImageBlock, CategoryBlock),
                   get_items=get_image_files,
                   get_y=parent_label,
                   splitter = GrandparentSplitter(train_name='train', valid_name='test'),
                   item_tfms=Resize(460),
                   batch_tfms=[*aug_transforms(size=size, min_scale=0.75, do_flip=False, mult=2, pad_mode='zeros'),
                               Normalize.from_stats(*own_stats)])
    return dblock.dataloaders(path, bs=bs)

dls = get_dls(32, 64)

class Unit(nn.Module):
    def __init__(self,in_channels,out_channels):
        super(Unit,self).__init__()

        self.conv = nn.Conv2d(in_channels=in_channels,kernel_size=3,out_channels=out_channels,stride=1,padding=1)
        self.bn = nn.BatchNorm2d(num_features=out_channels)
        self.relu = nn.ReLU()

    def forward(self,input):
        output = self.conv(input)
        output = self.bn(output)
        output = self.relu(output)

        return output

class SimpleNet(nn.Module):
    def __init__(self,num_classes=7):
        super(SimpleNet,self).__init__()

        #Create 14 layers of the unit with max pooling in between
        self.unit1 = Unit(in_channels=3,out_channels=32)
        self.unit2 = Unit(in_channels=32, out_channels=32)
        self.unit3 = Unit(in_channels=32, out_channels=32)

        self.pool1 = nn.MaxPool2d(kernel_size=2)

        self.unit4 = Unit(in_channels=32, out_channels=64)
        self.unit5 = Unit(in_channels=64, out_channels=64)
        self.unit6 = Unit(in_channels=64, out_channels=64)
        self.unit7 = Unit(in_channels=64, out_channels=64)

        self.pool2 = nn.MaxPool2d(kernel_size=2)

        self.unit8 = Unit(in_channels=64, out_channels=128)
        self.unit9 = Unit(in_channels=128, out_channels=128)
        self.unit10 = Unit(in_channels=128, out_channels=128)
        self.unit11 = Unit(in_channels=128, out_channels=128)

        self.pool3 = nn.MaxPool2d(kernel_size=2)

        self.unit12 = Unit(in_channels=128, out_channels=128)
        self.unit13 = Unit(in_channels=128, out_channels=128)
        self.unit14 = Unit(in_channels=128, out_channels=128)

        self.avgpool = nn.AvgPool2d(kernel_size=4)

        #Add all the units into the Sequential layer in exact order
        self.net = nn.Sequential(self.unit1, self.unit2, self.unit3, self.pool1, self.unit4, self.unit5, self.unit6
                                 ,self.unit7, self.pool2, self.unit8, self.unit9, self.unit10, self.unit11, self.pool3,
                                 self.unit12, self.unit13, self.unit14, self.avgpool)

        self.fc = nn.Linear(in_features=128,out_features=num_classes)

    def forward(self, input):
        output = self.net(input)
        print(output.shape)
        output = output.view(-1,128*2*2)
        print(output.shape)
        #output = output.view(output.size(0), -1)
        print(output.shape)
        output = self.fc(output)
        return output

model = SimpleNet(num_classes=7)
learn = Learner(dls, model, loss_func=CrossEntropyLossFlat(),
                metrics=accuracy)

learn.fit_one_cycle(1, 3e-3)

And this is the Error-Message:
RuntimeError                              Traceback (most recent call last)
<ipython-input-50-db5d2a1287cb> in <module>()
----> 1 learn.fit_one_cycle(1, 3e-3)

18 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1688     if input.dim() == 2 and bias is not None:
   1689         # fused op is marginally faster
-> 1690         ret = torch.addmm(bias, input, weight.t())
   1691     else:
   1692         output = input.matmul(weight.t())

RuntimeError: mat1 dim 1 must match mat2 dim 0

Any kind of help, hint or reference would be helpful.
Thanks in advance.


title: Coreml model float input for a pytorch model

body:
I have a pytorch model that takes 3 x width x height image as input with the pixel values normalized between 0-1

E.g., input in pytorch

img = io.imread(img_path)
input_img =  torch.from_numpy( np.transpose(img, (2,0,1)) ).contiguous().float()/255.0

I converted this model to coreml and exported an mlmodel which takes the input with correct dimensions

Image (Color width x height)

However, my predictions are incorrect as the model is expecting a float value between 0-1 and cvpixelbuffer is a int bwetween 0-255

I tried to normalize the values inside the model like so,

z = x.mul(1.0/255.0) # div op is not supported for export yet

However, when this op is done inside model at coreml level,  int * float is casted as int and all values are essentially 0

Cast op is not supported for export e.g., x = x.float()

How can I make sure my input is properly shaped for prediction? Essentially, I want to take the pixel rgb and float divide 255.0 and pass it to the model for inference?


Hello to anyone who may sees this. I'm trying convert a c3d model to onnx for general use purpose.

The C3D train code is took from :https://github.com/jfzhang95/pytorch-video-recognition

So the original env for c3d model is torch=0.4.0 python=3.5.2 as requested README.MD.

from network import C3D_model
import torch
import torch.onnx

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print("Device being used:", device)

def test():

    model = C3D_model.C3D(num_classes=2)
    checkpoint = torch.load('C3D-ucf101_epoch-3.pth.tar',
                        map_location=lambda storage, loc: storage)
    model.load_state_dict(checkpoint['state_dict'])
    torch.save(model, 'c3d_thunder.pkl')
    dummy_input1 = torch.randn(1, 3, 16, 112, 112)
    torch.onnx.export(model, dummy_input1, "C3D.onnx", verbose=True)

if __name__ == "__main__":
    test()

At first I tried to convert this model under original env, but met this error:

Traceback (most recent call last):
  File "/workspace/pytorch-video-recognition/model2onnx.py", line 31, in <module>
    test()
  File "/workspace/pytorch-video-recognition/model2onnx.py", line 27, in test
    torch.onnx.export(model, dummy_input1, "C3D.onnx", verbose=True)
  File "/usr/local/lib/python3.5/dist-packages/torch/onnx/__init__.py", line 25, in export
    return utils.export(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/onnx/utils.py", line 84, in export
    _export(model, args, f, export_params, verbose, training, input_names, output_names)
  File "/usr/local/lib/python3.5/dist-packages/torch/onnx/utils.py", line 154, in _export
    _onnx_opset_version, defer_weight_export)
RuntimeError: ONNX export failed: Couldn't export operator aten::max_pool3d

After some efforts on searching for solution, I decided to save the whole model to .pkl and reload it under torch=1.1.0. It works for loading model, but still has some problems in transformation.

import torch
import torch.onnx

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

def test():
    PATH = 'c3d_thunder.pkl'
    model = torch.load(PATH)
    dummy_input1 = torch.randn(1, 3, 16, 112, 112)
    torch.onnx.export(model, dummy_input1, "C3D.onnx", verbose=True)

if __name__ == "__main__":
    test()

still got error:

Traceback (most recent call last):
  File "C:/Users/google/Desktop/onnx_test/t1.py", line 33, in <module>
    test()
  File "C:/Users/google/Desktop/onnx_test/t1.py", line 29, in test
    torch.onnx.export(model, dummy_input1, "C3D.onnx", verbose=True)
  File "D:\Anaconda3\envs\torch_learn\lib\site-packages\torch\onnx\__init__.py", line 25, in export
    return utils.export(*args, **kwargs)
  File "D:\Anaconda3\envs\torch_learn\lib\site-packages\torch\onnx\utils.py", line 131, in export
    strip_doc_string=strip_doc_string)
  File "D:\Anaconda3\envs\torch_learn\lib\site-packages\torch\onnx\utils.py", line 363, in _export
    _retain_param_name, do_constant_folding)
  File "D:\Anaconda3\envs\torch_learn\lib\site-packages\torch\onnx\utils.py", line 266, in _model_to_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args, training)
  File "D:\Anaconda3\envs\torch_learn\lib\site-packages\torch\onnx\utils.py", line 225, in _trace_and_get_graph_from_model
    trace, torch_out = torch.jit.get_trace_graph(model, args, _force_outplace=True)
  File "D:\Anaconda3\envs\torch_learn\lib\site-packages\torch\jit\__init__.py", line 231, in get_trace_graph
    return LegacyTracedModule(f, _force_outplace, return_inputs)(*args, **kwargs)
  File "D:\Anaconda3\envs\torch_learn\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "D:\Anaconda3\envs\torch_learn\lib\site-packages\torch\jit\__init__.py", line 294, in forward
    out = self.inner(*trace_inputs)
  File "D:\Anaconda3\envs\torch_learn\lib\site-packages\torch\nn\modules\module.py", line 491, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "D:\Anaconda3\envs\torch_learn\lib\site-packages\torch\nn\modules\module.py", line 481, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "C:\Users\google\Desktop\onnx_test\network\C3D_model.py", line 46, in forward
    x = self.relu(self.conv1(x))
  File "D:\Anaconda3\envs\torch_learn\lib\site-packages\torch\nn\modules\module.py", line 491, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "D:\Anaconda3\envs\torch_learn\lib\site-packages\torch\nn\modules\module.py", line 481, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "D:\Anaconda3\envs\torch_learn\lib\site-packages\torch\nn\modules\conv.py", line 468, in forward
    if self.padding_mode == 'circular':
  File "D:\Anaconda3\envs\torch_learn\lib\site-packages\torch\nn\modules\module.py", line 539, in __getattr__
    type(self).__name__, name))
AttributeError: 'Conv3d' object has no attribute 'padding_mode'

Can anyone tell me how to fix this problem? Thanks for your time!~


title: torch.nn.DataParallel and to(device) does not support nested modules

body:
I have a torch.nn.module class defined in the following way:

class MyModule(torch.nn.Module):
    def __init__(self):
       super(MyModule, self).__init__()
       self.sub_module_a = ....  # nn.module
       self.sub_module_b_dict = {
          'B': .... # nn.module
       }

However after I call torch.nn.DataParallel(MyModule) and MyModule.to(device) only sub_module_a is put on cuda. The 'B' inside self.sub_module_b_dict is still on CPU.

Looks like DataParallel and to(device) only support first level variables inside a torch.nn.Module class. The modules nested inside a customized structure (in this case, a dictionary) seem to be ignored.

Am I missing some caveats here?


title: Pytorch ResNet model did not use GPU memory size as expected

body:
I’m using torchvision.models.resnet50 to do some experiment. But I found that the memory size used by by the model on GPU is less than it is expected to be.
The following is my code:

import torch
import torchvision
from torchsummary import summary
import numpy as np

model = torchvision.models.resnet50(pretrained=True)
model = model.cuda()
summary(model, (3, 448, 448), 24)
input = torch.tensor(np.random.random((24, 3, 448, 448)), dtype=torch.float32)
input = input.cuda()
while True:
    model(input)

When I run this code, the summary function tell me that, total memory to be used is about 27GB, including parameters and forward/backward pass size. I think this is nearly to the expected size according to resnet50’s network.
But, when the model was transformed to GPU and run training, I found the GPU memory usage was only kept about 9GB.
Is this normal? I’m worried about the GPU is not doing whole computing for my training job.
Could someone give me some help? Thanks!


title: LSTM error: AttributeError: 'tuple' object has no attribute 'dim'

body:
I have the following code:
import torch
import torch.nn as nn

model = nn.Sequential(
          nn.LSTM(300, 300),
          nn.Linear(300, 100),
          nn.ReLU(),
          nn.Linear(300, 7),
          )

s = torch.ones(1, 50, 300)
a = model(s)

And I get:
My-MBP:Desktop myname$ python3 testmodel.py
Traceback (most recent call last):
  File "testmodel.py", line 12, in <module>
    a = model(s)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/functional.py", line 1688, in linear
    if input.dim() == 2 and bias is not None:
AttributeError: 'tuple' object has no attribute 'dim'

Why? The dimensions should be fine. I saw related fixes to this issue when *input is defined in model.forward, but I don't even have anything implemented yet.
/edit: WAIT, there IS a *input!? How can I override this?


title: How to use Dataparallel when there is ‘for’ cycle in the network?

body:
I have a server with two GPUs, if I use one GPU I need more than 10 days to finish 1000 epoch. But when I tried to use Dataparallel, the programm didn’t work. It seems because there is a ‘for’ cycle in my network. So how can I use Dataparallel in this case. Or is there any other solution to speed up training?

class WaveNet( nn.Module ):
    def __init__(self, mu, n_residue, n_skip, dilation_depth, n_repeat):
        # mu: audio quantization size
        # n_residue: residue channels
        # n_skip: skip channels
        # dilation_depth & n_repeat: dilation layer setup
        self.mu = mu
        super( WaveNet, self ).__init__()
        self.dilation_depth = dilation_depth
        dilations = self.dilations = [2 ** i for i in range( dilation_depth )] * n_repeat
        self.one_hot = One_Hot( mu )
        self.from_input = nn.Conv1d( in_channels=mu, out_channels=n_residue, kernel_size=1 )
        self.from_input = nn.DataParallel(self.from_input)
        self.conv_sigmoid = nn.ModuleList(
            [nn.Conv1d( in_channels=n_residue, out_channels=n_residue, kernel_size=2, dilation=d )
             for d in dilations] )
        self.conv_sigmoid = nn.DataParallel(self.conv_sigmoid)
        self.conv_tanh = nn.ModuleList(
            [nn.Conv1d( in_channels=n_residue, out_channels=n_residue, kernel_size=2, dilation=d )
             for d in dilations] )
        self.conv_tanh = nn.DataParallel(self.conv_tanh)
        self.skip_scale = nn.ModuleList( [nn.Conv1d( in_channels=n_residue, out_channels=n_skip, kernel_size=1 )
                                          for d in dilations] )
        self.skip_scale = nn.DataParallel(self.skip_scale)
        self.residue_scale = nn.ModuleList( [nn.Conv1d( in_channels=n_residue, out_channels=n_residue, kernel_size=1 )
                                             for d in dilations] )
        self.residue_scale = nn.DataParallel(self.residue_scale)
        self.conv_post_1 = nn.Conv1d( in_channels=n_skip, out_channels=n_skip, kernel_size=1 )
        self.conv_post_1 = nn.DataParallel(self.conv_post_1)
        self.conv_post_2 = nn.Conv1d( in_channels=n_skip, out_channels=mu, kernel_size=1 )
        self.conv_post_2 = nn.DataParallel(self.conv_post_2)

    def forward(self, input, train=True):
        output = self.preprocess( input, train )
        skip_connections = []  # save for generation purposes
        for s, t, skip_scale, residue_scale in zip( self.conv_sigmoid, self.conv_tanh, self.skip_scale,
                                                    self.residue_scale ):
            output, skip = self.residue_forward( output, s, t, skip_scale, residue_scale )
            skip_connections.append( skip )
        # sum up skip connections
        output = sum( [s[:, :, -output.size( 2 ):] for s in skip_connections] )
        output = self.postprocess( output, train )
        return output

TypeError: zip argument #1 must support iteration


title: Training a machine learning model on multiple CSV files?

body: I want to train a machine learning model on multiple csv files that are all unique. Each file is a collection of time series data from basketball games. I want to train a model to look at each game and be able to predict outcomes. Should I simply tell sci kit learn or another package to iterate through the files in the folder of interest and run regressions on each? Thank you in advance.


title: What's different between convolution layer in `Torch`(i.e `nn.SpatialConvolution`) and convolution layer in `Pytorch`(i.e `torch.nn.Conv2d`)

body:
I would like to know the difference between convolution layer in Torch(i.e nn.SpatialConvolution) and convolution layer in Pytorch(i.e torch.nn.Conv2d)

In Torch's Docs, I found the output shape of SpatialConvolution

It says "If the input image is a 3D tensor nInputPlane x height x width, the output image size will be nOutputPlane x oheight x owidth where

owidth  = floor((width  + 2*padW - kW) / dW + 1)
oheight = floor((height + 2*padH - kH) / dH + 1)

"

which is different from torch.nn.Conv2d's in Pytorch Docs.

Does it mean they are different operation?