title: TypeError: string indices must be integers - PyTorch

body:
I'm trying to loop through my pre-trained CNN using the following code, it's slightly modified from PyTorch's example:
def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for i, batch in loaders[phase]:
                inputs = batch["image"].float().to(device)   # <---- error happens here
                labels = batch["label"].float().to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
            if phase == 'train':
                scheduler.step()

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

However I get the error:
Epoch 0/24
----------
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-53-79684c739f29> in <module>()
----> 1 model_ft = train_model(resnet_cnn, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=25)

<ipython-input-49-55bb790e99a0> in train_model(model, criterion, optimizer, scheduler, num_epochs)
     21             # Iterate over data.
     22             for i, batch in loaders[phase]:
---> 23                 inputs = batch["image"].float().to(device)
     24                 labels = batch["label"].float().to(device)
     25

TypeError: string indices must be integers

The loaders variable is:
loaders = {"train":train_loader, "val":valid_loader}

The Dataset class I'm using for this for my train_loader and valid_loader is, and explains why I'm using the string in my initial model function:
class GetDataLabel(Dataset):

  def __init__(self, df, root, transform = None):
    self.df = df
    self.root = root
    self.transform = transform

  def __len__(self):
    return len(self.df)

  def __getitem__(self, idx):
    if torch.is_tensor(idx):
      idx = idx.tolist()

    img_path = os.path.join(self.root, self.df.iloc[idx, 0])
    img = Image.open(img_path)
    label = self.df.iloc[idx, 1]

    if self.transform:
      img = self.transform(img)

    img_lab = {"image": img,
               "label": label}
    return (img_lab)

Thank you in advance.


I a trying to install PyTorch-segmentation-models in a Kaggle notebook. However, I'm facing an error .. Please help me resolve it...
Thanks!!
Command used:
!pip install -q -U segmentation-models-pytorch albumentations > /dev/null
import segmentation_models_pytorch as smp

Error:
ERROR: Could not find a version that satisfies the requirement segmentation-models-pytorch (from
versions: none)
ERROR: No matching distribution found for segmentation-models-pytorch

Screenshot:


title: Pytorch semantic segmentation loss function

body:
I’m new to segmentation model.
I would like to use the deeplabv3_resnet50 model.
My image has shape (256, 256, 3) and my label has shape (256, 256). Each pixel in my label has a class value(0-4). And the batch size set in the DataLoader is 32.
Therefore, the shape of my input batch is [32, 3, 256, 256] and the shape of corresponding target is [32, 256, 256]. I believe this is correct.
I was trying to use nn.BCEWithLogitsLoss().

Is this the correct loss function for my case? Or should I use
CrossEntropy instead?
If this is the right one, the output of my model is [32, 5, 256, 256]. Each image prediction has the shape [5,256, 256], does layer 0 means the unnomarlized probabilities of class 0? In order to make a [32, 256, 256] tensor to match the target to feed into the BCEWithLogitsLoss, do I need to transform the unnomarlized probabilities to classes?
If I should use CrossEntropy, what the size of my output and label should be?

Thank you everyone.


title: stack expects each tensor to be equal size pytorch

body:
I'm trying to train a cnn network for image classification. all the images are from the same size. But a mismatch error  of tensor size happned. Basically the error says some images are rotated.
The exact error message is here:
RuntimeError: stack expects each tensor to be equal size, but got [3, 200, 266] at entry 0 and [3, 266, 200] at entry 4

I wrote a simple program to check if any of the images are rotated using opencv.
for folder in folders:
    for f in glob(folder+"/*.jpg"):
        img = cv2.imread(f)
        if img.shape[1] != 200:
            print(f)

If there's a mismatch in size it should print the file name. But unfortunately nothing was printed means all the image are exactly the same size. Please Take note that I don't want to resize the image maybe can rotate it but not resize.
The code I use to read the data for training is below
train_transforms = transforms.Compose([
                                 transforms.ToTensor()])
    test_transforms = transforms.Compose([
                                      transforms.ToTensor()])
    train_data = datasets.ImageFolder(datadir,
                    transform=train_transforms)
    test_data = datasets.ImageFolder(datadir,
                    transform=test_transforms)
    num_train = len(train_data)
    indices = list(range(num_train))
    split = int(np.floor(valid_size * num_train))
    np.random.shuffle(indices)
    from torch.utils.data.sampler import SubsetRandomSampler
    train_idx, test_idx = indices[split:], indices[:split]
    train_sampler = SubsetRandomSampler(train_idx)
    test_sampler = SubsetRandomSampler(test_idx)
    trainloader = torch.utils.data.DataLoader(train_data,
                   sampler=train_sampler, batch_size=5)
    testloader = torch.utils.data.DataLoader(test_data,
                   sampler=test_sampler, batch_size=1)


title: TensorFlow equivalent of PyTorch's transforms.Normalize()

body:
I'm trying to inference a TFLite model that was originally built in PyTorch. I have been following along the lines of the PyTorch implementation and have to preprocess images along the RGB channels. I found the closest TensorFlow equivalent of transforms.Normalize() to be tf.image.per_image_standardization() (documentation). Although this is a pretty good match, tf.image.per_image_standardization() does this by taking mean and std across the channels and applies it to them. Here's their full implementation from here
def per_image_standardization(image):
  """Linearly scales `image` to have zero mean and unit norm.
  This op computes `(x - mean) / adjusted_stddev`, where `mean` is the average
  of all values in image, and
  `adjusted_stddev = max(stddev, 1.0/sqrt(image.NumElements()))`.
  `stddev` is the standard deviation of all values in `image`. It is capped
  away from zero to protect against division by 0 when handling uniform images.
  Args:
    image: 3-D tensor of shape `[height, width, channels]`.
  Returns:
    The standardized image with same shape as `image`.
  Raises:
    ValueError: if the shape of 'image' is incompatible with this function.
  """
  image = ops.convert_to_tensor(image, name='image')
  _Check3DImage(image, require_static=False)
  num_pixels = math_ops.reduce_prod(array_ops.shape(image))

  image = math_ops.cast(image, dtype=dtypes.float32)
  image_mean = math_ops.reduce_mean(image)

  variance = (math_ops.reduce_mean(math_ops.square(image)) -
              math_ops.square(image_mean))
  variance = gen_nn_ops.relu(variance)
  stddev = math_ops.sqrt(variance)

  # Apply a minimum normalization that protects us against uniform images.
  min_stddev = math_ops.rsqrt(math_ops.cast(num_pixels, dtypes.float32))
  pixel_value_scale = math_ops.maximum(stddev, min_stddev)
  pixel_value_offset = image_mean

  image = math_ops.subtract(image, pixel_value_offset)
  image = math_ops.div(image, pixel_value_scale)
  return image

whereas PyTorch's transforms.Normalize() allows us to mention the mean and std to be applied across each channel like below.
# transformation
    pose_transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225]),
    ])

What would be a way to get this functionality in TensorFlow 2.x?
Edit:
I created a quick botch that seems to solve this by defining a function as such:
def normalize_image(image, mean, std):
    for channel in range(3):
        image[:,:,channel] = (image[:,:,channel] - mean[channel])/std[channel]

    return image

I'm not sure how efficient this is but seems to get the job done. I still have to convert the output to a tensor before inputing to the model.


title: Training PyTorch models on different machines leads to different results

body:
I am training the same model on two different machines, but the trained models are not identical. I have taken the following measures to ensure reproducibility:
# set random number
random.seed(0)
torch.cuda.manual_seed(0)
np.random.seed(0)

# set the cudnn
torch.backends.cudnn.benchmark=False
torch.backends.cudnn.deterministic=True

# set data loader work threads to be 0
DataLoader(dataset, num_works=0)

When I train the same model multiple times on the same machine, the trained model is always the same. However, the trained models on two different machines are not the same. Is this normal? Are there any other tricks I can employ?


title: Python - How to output a numpy array of probabilities with certain precision and remain its sum

body:
I have a numpy array of 6 probabilities which come from pytorch softmax function.
[0.055709425,0.04365404,0.008613999,0.0022386343,0.0037478858,0.88603604]

I want to convert all 6 float numbers to string to represent a score output,
and all of them need to be rounded to a certain precision, say 4.
I used the following code to get the output text:
','.join(f'{x:.4f}' for x in scores)  # scores is the array above

and the output is
0.0557,0.0437,0.0086,0.0022,0.0037,0.8860

which sums up to 0.9999 instead of 1.0. And I have a bunch of arrays like this one but sums up to either 0.9999 or 1.0001.
So my question is, how do I get the output that sums up exactly to 1.0?
I know it's a floating point computation problem. What am I missing, some rounding operation or some adjustment?
Thank you very much.


title: Why isn't .data attribute obsolete/deprecated in favor of .detach() in PyTorch?

body:
I know that .detach() is safer in that it catches in-place modifications that will cause autograd to give wrong results when backpropagating through the graph.

Does, it mean that in the newer code we longer need to use .data, and we can use .detach() everywhere, whenever we need it?


title: translate from pyrotch to tensorflow

body:
What does this line of code in PyTorch do?

normA = A.mul(A).sum(dim=1).sum(dim=1).sqrt()
Y = A.div(normA.view(batchSize, 1, 1).expand_as(A))

Normally it should be a second term like this:

torch.div(input, value, out=None) → Tensor


title: Sample each image from dataset N times in single batch

body:
I'm currently working on task of learning representation (deep embeddings). The dataset I use have only one example image per object. I also use augmentation.

During training, each batch must contain N different augmented versions of single image in dataset (dataset[index] always returns new random transformation).

Is there some standart solution or library with DataLoader for this purpose, that will work with torch.utils.data.distributed.DistributedSampler?
If not, will any DataLoader, inherited from torch.utils.data.DataLoader (and calling super().__init__(...)), will work in distributed training?


title: How to freeze one filter in a layer while keeping other filters trainable?

body:
Suppose that the weight matrix for one layer is [32,64,4,2]. Is it possible to freeze its first filter while keeping the other 31 filters trainable?

I've tried to set requires_grad, however this parameter is for the whole layer.


title: Getting the gradients of a model trained in OpenNMT-py

body:
When training a model using OpenNMT-py, we get a dict as output, containing the weights and biases of the network. However, these tensors have requires_grad = False, and so, do not have a gradient. For example. with one layer, we might have the following tensors, denoting embeddings as well as weights and biases in the encoder and decoder. None of them have a gradient attribute.

encoder.embeddings.emb_luts.0.weight

decoder.embeddings.emb_luts.0.weight

encoder.rnn.weight_ih_l0

encoder.rnn.weight_hh_l0

encoder.rnn.bias_ih_l0

encoder.rnn.bias_hh_l0

decoder.rnn.layers.0.weight_ih

decoder.rnn.layers.0.weight_hh

decoder.rnn.layers.0.bias_ih

decoder.rnn.layers.0.bias_hh

Can OpenNMT-py be made to set requires_gradient = True with some option I have not found or is there some other way to obtain the gradient of these tensors?


title: Weird reults in training AlexNet

body:
I’m trying to train a dataset with AlexNet model. The task is multiclass classification (15 classes). I am wondering why I am getting very low accuracy.
I tried different learning rate but has not been improved.

Here is the snippet for the training.

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=1e-3, momentum=0.9)
#optimizer = optim.Adam(model.parameters(), lr=1e-2)  # 1e-3, 1e-8

def train_valid_model():

  num_epochs=5

since = time.time()
out_loss = open("history_loss_AlexNet_exp1.txt", "w")
out_acc = open("history_acc_AlexNet_exp1.txt", "w")

losses=[]
ACCes =[]
#losses = {}

for epoch in range(num_epochs):  # loop over the dataset multiple times
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 50)

    if epoch % 10 == 9:
       torch.save({
        'epoch': epoch + 1,
        'model_state_dict': model.state_dict(),
        'optimizer_state_dict': optimizer.state_dict(),
         'loss': loss
        }, 'AlexNet_exp1_epoch{}.pth'.format(epoch+1))

    for phase in ['train', 'valid', 'test']:
        if phase == 'train':

            model.train()
        else:
            model.eval()

        train_loss = 0.0
        total_train = 0
        correct_train = 0

        for t_image, target, image_path in dataLoaders[phase]:
            #print(t_image.size())
            #print(target)

            t_image = t_image.to(device)
            target = target.to(device)

            optimizer.zero_grad()

            with torch.set_grad_enabled(phase == 'train'):
                outputs = model(t_image)
                outputs = F.softmax(outputs, dim=1)

                loss = criterion(outputs,target)
                if phase == 'train':
                    loss.backward()
                    optimizer.step()

            _, predicted = torch.max(outputs.data, 1)
            train_loss += loss.item()* t_image.size(0)
            correct_train += (predicted == target).sum().item()

        epoch_loss = train_loss / len(dataLoaders[phase].dataset)
        #losses[phase] = epoch_loss
        losses.append(epoch_loss)

        epoch_acc = 100 * correct_train / len(dataLoaders[phase].dataset)
        ACCes.append(epoch_acc)

        print('{} Loss: {:.4f} {} Acc: {:.4f}'.format(phase, epoch_loss, phase, epoch_acc))

This is the output for two epochs

   Epoch 0/4

train Loss: 2.7026 train Acc: 17.2509
valid Loss: 2.6936 valid Acc: 28.7632
test Loss: 2.6936 test Acc: 28.7632

Epoch 1/4

train Loss: 2.6425 train Acc: 17.8019
valid Loss: 2.6357 valid Acc: 28.7632
test Loss: 2.6355 test Acc: 28.7632


title: How to set training weight on softmax during training

body:
I'm reproducing Auto-DeepLab with PyTorch and I got a problem, that is, I can't set the architecture weight(both cell and layer) on softmax. It either leads to twice backward or weights get no upgrade with gradients but only softmax.

class Architect () :
    def __init__(self, model, args):
        self.network_momentum = args.momentum
        self.network_weight_decay = args.weight_decay
        self.model = model
        self.optimizer =
            torch.optim.Adam(self.model.arch_parameters(),
                lr=args.arch_lr, betas=(0.5, 0.999),
                weight_decay=args.arch_weight_decay)

    def step (self, input_valid, target_valid) :
        self.model.soft_parameters()
        self.optimizer.zero_grad ()
        self._backward_step(input_valid, target_valid)
        self.optimizer.step()

    def _backward_step (self, input_valid, target_valid) :
        _, loss = self.model._loss (input_valid, target_valid)
        loss.backward ()

under this code, it leads softmax works but the weights are not optimized with gradients.

[[0.3333, 0.3333, 0.3333],
         [0.3333, 0.3333, 0.3333],
         [0.3333, 0.3333, 0.3333],
         [0.3333, 0.3333, 0.3333]]*12], device='cuda:0',
       grad_fn=<SoftmaxBackward>)


title: Model stops at loss.backward() with "Illegal instruction prompt"

body:
Similar to this but OP states his programme ran successfully on linux, while my problem is on Ubuntu. I've tried downgrading both torch and torchvision and the problem persists. The part I suspect that causes the problem is shown below:

for epoch in range(max_epoch):

    running_loss = 0.0
    for image, label in train_loader:

        optimiser.zero_grad()

        output = model(image)

        print("Label:{}\n".format(label))
        print("Output is {}\n".format(output))

        loss = criterion(output,label)
        print("Loss is {}".format(loss.item()))
        loss.backward()
        print("Backward loss and now optimiser")
        optimiser.step()

        running_loss += loss.item()
        print("Loss: {:.4f}".format(running_loss))

The full code is available here
Here's the prompt I get:
Terminal


title: Python array copy causes memory (re)allocation

body:
def Scramble_acton(net, ort, inputs, dpinda, hocmat):

   # copies net in ort
   for lr in range(0,6):
      for bi in range(0,BATCHSIZE):
         for no in range(0,int(lrsize[lr][0])):
            if lrsize[lr][1] == 1:
               ort.nx[lr][bi][no] = net.nx[lr][bi][no]
            if lrsize[lr][1] != 1:
               for xi in range(0,int(lrsize[lr][1])):
                  for yi in range(0,int(lrsize[lr][1])):
                     ort.nx[lr][bi][no][xi][yi] = \
                     net.nx[lr][bi][no][xi][yi]

Hi guys,
I pass to this function two pytorch neural network models (net and ort), then copy all output values from one (net) to the other (ort). In the process, the memory allocation increases (which I would not expect to happen) and, after some cycle, the program crashes. I tried to use copy.deepcopy to make the copy, but I got the message "Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol".
Thank you,
Alex

This is the model definition:

class Net(nn.Module):

   def __init__(self, x):

      super(Net, self).__init__()
      self.pool =  nn.MaxPool2d (2,2)
      self.conv1 = nn.Conv2d ( 3,10,5)
      self.conv2 = nn.Conv2d (10,16,5)
      self.fc3 =   nn.Linear (16*5*5,120)
      self.fc4 =   nn.Linear (120,84)
      self.fc5 =   nn.Linear (84,10)

      lrsize[0][0] =  3;      lrsize[0][1] =32
      lrsize[1][0] = 10;      lrsize[1][1] =14
      lrsize[2][0] = 16*5*5;  lrsize[2][1] = 1
      lrsize[3][0] = 120;     lrsize[3][1] = 1
      lrsize[4][0] =  84;     lrsize[4][1] = 1
      lrsize[5][0] =  10;     lrsize[5][1] = 1

      self.nx = np.empty(6, dtype=np.object)
      self.nl = np.empty(6, dtype=np.object)
      self.nl[1] = self.conv1
      self.nl[2] = self.conv2
      self.nl[3] = self.fc3
      self.nl[4] = self.fc4
      self.nl[5] = self.fc5
      self.forward(x,0,0)

   def forward(self, x0, opt, sitnx):

      #Tanhact # torch.sigmoid
      xx = self.nx; xx[0] = x0
      if opt == 1: xx = sitnx
      self.nx[1] = self.pool(Tanhact(self.conv1(xx[0])))
      self.nx[2] = self.pool(Tanhact(self.conv2(xx[1])))
      self.nx[2] = self.nx[2].view(-1, 16*5*5)
      xt = self.fc3(xx[2]); self.nx[3] = Tanhact(SLOPETP*xt)
      xt = self.fc4(xx[3]); self.nx[4] = Tanhact(SLOPETP*xt)
      xt = self.fc5(xx[4]); self.nx[5] = Tanhact(SLOPETP*xt)

nx[] contains the layer values,
BATCHSIZE = 25

I come from C, I just wanted to make a copy of the tensor /array, but something strange is happening here.


title: Why is CNN convolution output size in PyTorch DQN tutorial computed with `kernel_size -1`?

body:
Based on my understanding, CNN output size for 1D is

output_size = (input_size - kernel_size + 2*padding)//stride + 1

Refer to PyTorch DQN Tutorial. In the tutorial, it uses 0 padding, which is fine. However, it computes the output size as follows:

        def conv2d_size_out(size, kernel_size = 5, stride = 2):
            return (size - (kernel_size - 1) - 1) // stride  + 1

It the above a mistake or is there something I missed?


title: Create a specific tensor from another tensor

body:
q_pred = self.Q.forward(states) gives me the following output :

tensor([[-4.4713e-02,  4.2878e-03],
        [-2.2801e-01,  2.2295e-01],
        [-9.8098e-03, -1.0766e-01],
        [-1.4654e-01,  1.2742e-01],
        [-1.6224e-01,  1.6565e-01],
        [-3.6515e-02,  3.1022e-02],
        [-4.5094e-02,  1.4848e-02],
        [-1.4157e-01,  1.3974e-01],
        [-3.0593e-03, -4.2342e-02],
        [-4.1689e-02,  2.9376e-02],
        [-9.3629e-02,  1.0297e-01],
        [-5.2163e-04, -7.4799e-02],
        [-2.8944e-02, -1.2417e-01]], grad_fn=<AddmmBackward>)

and actions gives me the the following output

tensor([1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1], dtype=torch.int32)

From the two above output, how can I can the following result :

tensor([4.2878e-03, -2.2801e-01, -1.0766e-01, 1.2742e-01, -1.6224e-01,  3.1022e-02, 1.4848e-02,
        -1.4157e-01, -4.2342e-02, -4.1689e-02, -9.3629e-02, -7.4799e-02,, -1.2417e-01], grad_fn=<AddmmBackward>)

?

UPDATE

Here is how self.Q is implemented :

class LinearDeepQNetwork(nn.Module):
    def __init__(self, lr, n_actions, input_dims):
        super(LinearDeepQNetwork, self).__init__()

        self.fc1 = nn.Linear(*input_dims, 128)
        self.fc2 = nn.Linear(128, n_actions)

        self.optimizer = optim.Adam(self.parameters(), lr=lr)
        self.loss  = nn.MSELoss()
        self.device = T.device('cuda:0' if T.cuda.is_available() else 'cpu')
        self.to(self.device)

    def forward(self, state):
        layer1 = F.relu(self.fc1(state))
        actions = self.fc2(layer1)

        return actions


title: How do I get pytorch working for my 765m?

body:
Yes I know people have tried this again and again. There are so many forums about this issue but this is what I've been dealing with.

Specs:
Python=3.7
Pip=18.1
Cuda=10.0
cuDNN= 10.1
Kubuntu=19.10
GPU=gtx765m

I just want to mention I am pretty new to python and linux for that matter but I'm not an idiot. So please be thorough in your explanation.

With that out of the way let's get into what I've done. I've started looking into AI and the "real time voice cloning" in particular. https://github.com/CorentinJ/Real-Time-Voice-Cloning

What i've done is install pip, install all the components and then found out pytorch didn't support older GPUs (after playing around with CUDA for a few hours).

So I tried to "build/compile" it from the source to get it to work but now I'm stuck here:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
CUDA_cublas_device_LIBRARY (ADVANCED)
    linked by target "THC" in directory /home/aydenrobert/pytorch/torch/lib/THC

-- Configuring incomplete, errors occurred!

(The full log is at: https://pastebin.com/B5nYNKjL)

I've searched through the whole of the internet for answers and I can't find any. This has taken me 2 days already and I am heading into the third. If someone could help me I would be forever grateful.


title: Getting the following error while using scikit-image to read images "AttributeError: 'PngImageFile' object has no attribute '_PngImageFile__frame' "

body:
I am using scikit-image to load a random image from a folder. OpenCV is being used for operations later on..

Code is as follows (only relevant parts included)

import imageio
import cv2 as cv
import fileinput
from collections import Counter

from data.apple_dataset import AppleDataset
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor
from torchvision.transforms import functional as F

import utility.utils as utils
import utility.transforms as T

from PIL import Image
import skimage.io
from skimage.viewer import ImageViewer
from matplotlib import pyplot as plt
%matplotlib inline

 APPLE_IMAGE_PATH = r"__mypath__\samples\apples\images"

# Load a random image from the images folder
FILE_NAMES = next(os.walk(APPLE_IMAGE_PATH))[2]
random_apple_in_folder = os.path.join(APPLE_IMAGE_PATH, random.choice(FILE_NAMES))
apple_image = skimage.io.imread(random_apple_in_folder)
apple_image_cv = cv.imread(random_apple_in_folder)
apple_image_cv = cv.cvtColor(apple_image_cv, cv.COLOR_BGR2RGB)

Error is as follows

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-9575eed18f18> in <module>
     11 FILE_NAMES = next(os.walk(APPLE_IMAGE_PATH))[2]
     12 random_apple_in_folder = os.path.join(APPLE_IMAGE_PATH, random.choice(FILE_NAMES))
---> 13 apple_image = skimage.io.imread(random_apple_in_folder)
     14 apple_image_cv = cv.imread(random_apple_in_folder)

    AttributeError: 'PngImageFile' object has no attribute '_PngImageFile__frame'

How do i proceed from here? What should i change???


title: how to solve size mismatch for this model?

body:
data_transforms = {
'train': transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
]),
'val': transforms.Compose([
    transforms.Resize(224),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
]),

batchsize=4

class Net(nn.Module):
def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(3, 64, kernel_size=5, padding=1)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(64, 128, kernel_size=5, padding=1)
    self.fc1 = nn.Linear(256*6*6, 4096)
    self.fc2 = nn.Linear(4096, 4096)
    self.fc3 = nn.Linear(4096, 2)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.reshape(x.size(0), -1)
    print(x.shape)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

RuntimeError: size mismatch, m1: [4 x 373248], m2: [9216 x 4096] at C:/w/1/s/tmp_conda_3.8_075429/conda/conda-bld/pytorch_1579852542185/work/aten/src\THC/generic/THCTensorMathBlas.cu:290


title: I am trying to build a neural network with one neuron using the pytorch library. It keeps giving me an error

body:
I am trying to build a neural network with one neuron using the pytorch library. This is my code (the error is at the bottom)

import numpy as np
import random
import matplotlib.pyplot as plt

x_train = np.array([random.randint(1,1000) for x in range(1000)], dtype = np.float32)
y_train = np.array([int(num*3+1) for num in x_train], dtype = np.float32)

x_test = np.array([random.randint(1,1000) for x in range(1000)], dtype = np.float32)
y_test = np.array([int(num*3+1) for num in x_train], dtype = np.float32)

X_train = torch.from_numpy(x_train)
Y_train = torch.from_numpy(y_train)

plt.figure(figsize = (8,8))
plt.scatter(X_train, Y_train)
plt.show()

X_test = torch.from_numpy(x_test)
Y_test = torch.from_numpy(y_test)

input_size = 1
hidden_size = 1
output_size = 1
learning_rate = 0.1

w1 = torch.rand(input_size, hidden_size, requires_grad = True)
b1 = torch.rand(hidden_size, output_size, requires_grad = True)

for i in range(100):
    y_pred = X_train.mm(w1).clamp(min = 0).add(b1)
    loss = (Y_train-y_pred).pow(2).sum()

    loss.backward()

    with torch.no_grad():
        w1-=w1.grad*learning_rate
        b1 -= b1.grad*learning_rate
        w1.grad.zero_()
        b1.grad.zero_()

When I run this code, it gives me a runtime error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-84-5142b17ecfff> in <module>
     32
     33 for i in range(100):
---> 34     y_pred = X_train.mm(w1).clamp(min = 0).add(b1)
     35     loss = (Y_train-y_pred).pow(2).sum()
     36

RuntimeError: matrices expected, got 1D, 2D tensors at C:\w\1\s\windows\pytorch\aten\src\TH/generic/THTensorMath.cpp:192

What is wrong with that line of code and how to make it work as planned.


title: What is given module "pymss"?

body:
I am trying to get a deep learning project running on my GPU for the first time and am consulting the linked repo.

A venv environment has been set up and I should have all the tooling to start training off of the given data.

The issue lies in the fact that to run training with the given lines:

# Starting in MASS directory
cd python
source ~/usr/bin/virtualenv/bin/activate
python3 main.py -d ../data/metadata.txt

The error is returned of --

  File "main.py", line 20, in <module>
    from pymss import EnvManager
ModuleNotFoundError: No module named 'pymss'

This repo (MASS) contains starter training data as well as the scaffolding for simulation and control of the PPO(Proximal Policy Optimization) Deep Reinforcement Learning algorithm.
https://github.com/lsw9021/MASS

# Within MASS/python/main.py
from pymss import EnvManager

class PPO(object):
    def __init__(self,meta_file):
        np.random.seed(seed = int(time.time()))
        self.num_slaves = 16
        self.env = EnvManager(meta_file,self.num_slaves)

        self.use_muscle = self.env.UseMuscle()
        self.num_state = self.env.GetNumState()
        self.num_action = self.env.GetNumAction()
        self.num_muscles = self.env.GetNumMuscles()

It seems like an incredibly well put together project. May someone please help me resolve what the given EnvManager function in "pymss"?? is from?

I thought it to be pymysql but my search lends me to say no.

This EnvManager function seems essential for rendering the test environment.. but it has been almost impossible to discover a missing/mispelled py module.

I am using Ubuntu 18.04, Python 3.6.9, DART 3.6.0 http://dartsim.github.io/

Finally could it matter where my DART (animation toolkit) is compiled/built in reference to MASS? Mine is in users/downloads while MASS is in a separate folder.


title: Network Training is too slow for Custom Network even though network is not too large(Resnet + LSTM type network)

body:
I have made a custom network for the image data. But the training on this network is very slow though the network is not too huge.
When I am training on resnet150 with lstm, the training is quite fast. Not sure which operation is lagging my training speed?
I have tried reducing Batch_size, seq_dim and network parameters. My guess is some operation is reducing the speed
The dataset I am using are images and they are also very small size(96*96) gray scale images.
***** CODE *****
class Residual(nn.Module):
    def __init__(self, input_channels, num_channels,
                 use_1x1conv=True, strides=1, dilation=2, padding=1, kernel_size=5):
        super(Residual, self).__init__()
        self.conv1 = nn.Conv2d(input_channels, num_channels,
                               kernel_size=kernel_size, padding=padding, stride=strides, dilation=dilation)
        self.conv2 = nn.Conv2d(num_channels, num_channels,
                               kernel_size=kernel_size, padding=2 * padding, dilation=(2 * dilation))
        if use_1x1conv:
            self.conv3 = nn.Conv2d(input_channels, num_channels,
                                   kernel_size=1, stride=strides)
        else:
            self.conv3 = None
        self.bn1 = nn.BatchNorm2d(num_channels)
        self.bn2 = nn.BatchNorm2d(num_channels)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, X):
        Y = F.relu(self.bn1(self.conv1(X)))
        Y = self.bn2(self.conv2(Y))
        if self.conv3:
            X = self.conv3(X)
        Y += X
        return F.relu(Y)

def weight_init(m):
    '''
    Usage:
        model = Model()
        model.apply(weight_init)
    '''
    if isinstance(m, nn.Conv1d):
        init.normal_(m.weight.data)
        if m.bias is not None:
            init.normal_(m.bias.data)
    elif isinstance(m, nn.Conv2d):
        init.xavier_normal_(m.weight.data)
        if m.bias is not None:
            init.normal_(m.bias.data)
    elif isinstance(m, nn.BatchNorm1d):
        init.normal_(m.weight.data, mean=1, std=0.02)
        init.constant_(m.bias.data, 0)
    elif isinstance(m, nn.BatchNorm2d):
        init.normal_(m.weight.data, mean=1, std=0.02)
        init.constant_(m.bias.data, 0)
    elif isinstance(m, nn.Linear):
        init.xavier_normal_(m.weight.data)
        init.normal_(m.bias.data)
    elif isinstance(m, nn.LSTM):
        for param in m.parameters():
            if len(param.shape) >= 2:
                init.orthogonal_(param.data)
            else:
                init.normal_(param.data)

class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        num_channels = 64
        strides = 1
        dilation = 2
        padding = 4
        kernel_size = 5
        input_channel = 3
        self.fc = nn.Linear(config['hidden_dim'], config['output_dim'])
        self.lstm = None

        b1 = Residual(input_channels=input_channel, num_channels=num_channels,
                      use_1x1conv=True, strides=strides, dilation=dilation, padding=padding, kernel_size=kernel_size)

        b2 = Residual(input_channels=num_channels, num_channels=2 * num_channels,
                      use_1x1conv=True, strides=strides, dilation=2 * dilation, padding=2 * padding,
                      kernel_size=kernel_size)

        b3 = Residual(input_channels=2 * num_channels, num_channels=4 * num_channels,
                      use_1x1conv=True, strides=strides, dilation=4 * dilation, padding=4 * padding,
                      kernel_size=kernel_size)

        self.net = nn.Sequential(b1, b2, b3, nn.AdaptiveMaxPool2d((2, 2)))
        self.apply(weight_init)

    def forward(self, x):
        x = self.net(x)
        x = x.view(config['batch_size'], config['seq_dim'], -1)
        if self.lstm is None:
            self.lstm = nn.LSTM(x.size(2), config['hidden_dim'], 1, batch_first=True).to(self.device)
            for param in self.lstm.parameters():
                if len(param.shape) >= 2:
                    init.orthogonal_(param.data)
                else:
                    init.normal_(param.data)

        h0 = torch.zeros(config['layer_dim'], x.size(0), config['hidden_dim']).to(self.device)
        # Initialize cell state
        c0 = torch.zeros(config['layer_dim'], x.size(0), config['hidden_dim']).to(self.device)

        output, (hn, cn) = self.lstm(x, (h0,c0))
        output = output[:, :, :]
        output = self.fc(output)

        return output, (hn, cn)

 for epoch in range(config['num_epochs']):
            print('epoch', epoch)
            running_loss = 0
            nb_classes = config['output_dim']
            confusion_matrix = torch.zeros(nb_classes, nb_classes)
            for i, (image, label) in enumerate(trainLoader):
                print('batch: ',i)
                image = image.float().to(device)
                label = label.to(device)
                optimizer.zero_grad()

                batch_size, timesteps, H, W, C = image.size()
                # Change Image shape
                image = image.view(batch_size * timesteps, H, W, C)
                image = image.permute(0, 3, 1, 2)  # from NHWC to NCHW

                output, (hn,cn) = model(image)
                label = label.view(-1)
                output = output.view(-1, output.size(2))
                loss = criterion(output, label)
                loss *= config['seq_dim']
                loss.backward()  # Backward pass
                optimizer.step()  # Now we can do an optimizer step
                running_loss += loss.item()


title: Is it the right way to process a PIL image in pytorch?

body:
I want to train a deep model from .png images by using Pytorch. I'm using a pre-trained model on ImageNet so I need to normalize images before feeding them to the network, but when I look at the result of the transform I see there are some values more than 1 and some less than -1. I'm wondering shouldn't they all be in [-1, 1] range? Am I doing it in the right way?
There is my code:
normalize = transforms.Normalize(
mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225])
preprocess = transforms.Compose([
transforms.ToTensor(),
normalize])

x = Image.open("path/to/the/file").convert('RGB')
x = self.transform(x) # I feed the network by x


title: How to represent those types of values in a tensor?

body:
I have a dataset like this:
(0, 1), UpDownUpUp
(2, 3), UpUpUpDownDownDown
(0, 2), DownUp
(0, Undefined), DownUp

How to represent this type of data in a PyTorch tensor? So I can then train a neural network with it?


title: Training not speeding up even after using GPU

body:
transform = transforms.Compose([transforms.Resize(IMG_SIZE),
                                transforms.CenterCrop(CROP_SIZE),
                                transforms.ToTensor()])

class LandmarksDatasetTrain(Dataset):
    """Landmarks dataset."""

    def __init__(self, landmarks_frame, root_dir, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.landmarks_frame = landmarks_frame
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.landmarks_frame)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        img_name = os.path.join(self.root_dir,self.landmarks_frame.loc[idx, 'id'][0],self.landmarks_frame.loc[idx, 'id'][1], self.landmarks_frame.loc[idx, 'id'][2], self.landmarks_frame.loc[idx, 'id'])
        img_name += ".jpg"
        image = Image.open(img_name)
        landmarks = self.landmarks_frame.loc[idx, 'landmark_id']
        sample = {'image': image, 'landmarks': landmarks}

        if self.transform:
            sample['image'] = self.transform(sample['image'])
            sample['landmarks'] = torch.tensor(sample['landmarks'])

        return sample

dataset_train = LandmarksDatasetTrain(landmarks_frame = frame,
                                      root_dir='/kaggle/input/landmark-recognition-2020/train',
                                      transform=transform)

train_loader = DataLoader(dataset_train, batch_size=4, shuffle=True, num_workers=4, drop_last=False)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(CROP_SIZE*CROP_SIZE*3, 64)
        self.fc2 = nn.Linear(64, 64)
        self.fc3 = nn.Linear(64, 64)
        self.fc4 = nn.Linear(64, frame['landmark_id'].nunique())

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return F.log_softmax(x, dim=1)

net = Net()
net.to(device)

for epoch in range(3):
    optimizer = optim.Adam(net.parameters(), lr=0.001)
    for data in tqdm(train_loader):
        X = data['image'].to(device)
        y = data['landmarks'].to(device)
        net.zero_grad()
        output = net(X.view(-1,CROP_SIZE*CROP_SIZE*3))
        loss = F.nll_loss(output, y)
        loss.backward()
        optimizer.step()
    print(loss)

batch size is 4
data['image'] and data['landmarks'] are tensors device = torch.device("cuda:0") and deep learning library I am using is Pytorch but GPU is still not working for me. Its usage shows 5% and total time for 1 epoch 3.5 to 4 hours
Will be really helpful if someone points out my mistake.
Attaching an image of resource usage and GPU config.

Attaching image for showing that GPU is on

Here is the link to my notebook
https://www.kaggle.com/hiteshsom/google-landmark-recognition


title: Is my batch accumulation implementation correct?

body:
I’d like to know if my code for training a model with batch accumulation is correct. Especially the part regarding the loss calculation because I’m not so sure this is the right way to do this.
Here’s my code:
def train (start_epochs, n_epochs, best_acc, train_generator, val_generator, model, optimizer, criterion, checkpoint_path, best_model_path):

#num_epochs = 25
  since = time.time()

  #best_model_wts = copy.deepcopy(model.state_dict())
  #best_acc = 0.0
  train_loss = []
  val_loss = []
  train_acc = []
  val_acc = []

  batch_accumulation = 8

  for epoch in tqdm(range(start_epochs, n_epochs+1)):

    running_train_loss = 0.0
    running_val_loss = 0.0

    running_train_corrects = 0
    running_val_corrects = 0

    optimizer.zero_grad
    #Training
    model.train()
    for i, (faces, labels) in tqdm(enumerate(train_generator)):

      faces = faces.to(device)
      labels = labels.to(device)

      #forward
      outputs = model(faces)

      #predictions of the model determined using the torch.max() function, which returns the index of the maximum value in a tensor.
      _, preds = torch.max(outputs[1], 1)

      #pass the model outputs and the true image labels to the loss function
      loss = criterion(outputs[1], labels)
      #loss = loss / batch_accumulation
      running_train_loss += loss.item()
      # Backprop and Adam optimisation
      loss.backward()
      # Track the accuracy and loss
      running_train_corrects += torch.sum(preds == labels.data)

      if (i+1)% batch_accumulation == 0:
        optimizer.step()
        optimizer.zero_grad # zero the gradient buffers

    # calculate average losses and accuracy
    epoch_train_loss = running_train_loss / len(train_generator.dataset)
    epoch_train_acc = ((running_train_corrects.double() / len(train_generator.dataset)) * 100)
    train_loss.append(epoch_train_loss)
    train_acc.append(epoch_train_acc)

    print('Train Loss: {:.4f} Train Acc: {:.2f}%'.format(epoch_train_loss, epoch_train_acc))

    #Validation
    with torch.set_grad_enabled(False):
      model.eval()
      for i , (faces_val, labels_val) in tqdm(enumerate(val_generator)):

        faces_val = faces_val.to(device)
        labels_val = labels_val.to(device)

        if (i+1)% batch_accumulation == 0:

          outputs_val = model(faces_val)

          _, preds_val = torch.max(outputs_val[1], 1)
          loss_val = criterion(outputs_val[1], labels_val)

          running_val_loss += loss_val.item()
          #running_val_loss = running_val_loss +((1 /(i+1)) * (loss.item() - running_val_loss))
          running_val_corrects += torch.sum(preds_val == labels_val.data)

    # calculate average losses and accuracy
    epoch_val_loss = running_val_loss / len(validation_generator.dataset)
    epoch_val_acc = (running_val_corrects.double() / len(validation_generator.dataset)) * 100
    val_loss.append(epoch_val_loss)
    val_acc.append(epoch_val_acc)

    print('Validation Loss: {:.4f} Validation Acc: {:.2f}%'.format(epoch_val_loss, epoch_val_acc))

I got strange epoch train results (like 456.890) and I’m note sure about the if statement in the validation part.


title: Add cpp file to setuptools but don't compile

body:
I have a library that I want to publish to PyPi that includes python code and C++ code. The special thing is that the C++ code is loaded via ninja, using
from torch.utils.cpp_extension import load

lib_dir = os.path.dirname(os.path.realpath(__file__))
backend = load(
  name="backend",
  sources=[os.path.join(lib_dir, "backend.cpp")],
  verbose=True)

My layout:
package_name/
  package_name/
    __init__.py
    package.py
    backend.cpp

The code above is at the top of package.py. This works flawlessly and compiles on the fly, if tested locally.
BUT, when packaging this with pypi, the .cpp file does not get added. My setup.py:
setup(
  name='package_name',
  packages=['package_name'],
  ...)

I tried:

adding the .cpp as an Extension. This causes setuptools to try to compile backend.cpp, which I don't want, as it won't work (missing headers)
adding the .cpp via scripts=. Doesn't seem to do anything.

EDIT The file shows up in the .tar.gz that is produced (in dist/). Also if I download it from pypi. But it won't show up on disk if I do a pip install:
> tar tvf dist/package_name-0.1.0.tar.gz
...
drwxr-xr-x  0 fabian staff       0 Nov 14 10:17 package_name-0.1.0/package_name/
-rw-r--r--  0 fabian staff     203 Nov 14 09:36 package_name-0.1.0/package_name/__init__.py
-rw-r--r--  0 fabian staff    5569 Nov 14 09:36 package_name-0.1.0/package_name/package.py
-rw-r--r--  0 fabian staff   10101 Nov 12 14:43 package_name-0.1.0/package_name/backend.cpp
...

> ls  /Users/.../miniconda3/envs/testing/lib/python3.8/site-packages/package_name/
__init__.py
__pycache__
package.py

EDIT 2: It's getting wilder. I tried to add the following to setup.py:
setup(
  name='package_name',
  packages=['package_name'],
  ext_modules=[CppExtension(
    name='backend',
    sources=['package_name/backend.cpp']),
  ],
  cmdclass={'build_ext': BuildExtension.with_options(use_ninja=True)},
  ...

And now it's getting built. But it is still not adding the .cpp file to the output.


title: Pytorch DCGAN example for kernel 3

body:
I tried out the pytorch dcgan example and it worked fine but when I tried to change the kernel from 4x4 to 3x3. I only changed the kernel and It gave the following error. Why this error is occurring? To solve this what changes are needed?
ValueError: Using a target size (torch.Size([64])) that is different to the input size (torch.Size([256])) is deprecated. Please ensure they have the same size.

Here is the generator:
class Generator(nn.Module):
def __init__(self, ngpu):
    super(Generator, self).__init__()
    self.ngpu = ngpu
    self.main = nn.Sequential(
        # input is Z, going into a convolution
        nn.ConvTranspose2d( nz, ngf * 8, kernel_size, 1, 0, bias=False),
        nn.BatchNorm2d(ngf * 8),
        nn.ReLU(True),
        # state size. (ngf*8) x 4 x 4
        nn.ConvTranspose2d(ngf * 8, ngf * 4, kernel_size, 2, 1, bias=False),
        nn.BatchNorm2d(ngf * 4),
        nn.ReLU(True),
        # state size. (ngf*4) x 8 x 8
        nn.ConvTranspose2d( ngf * 4, ngf * 2, kernel_size, 2, 1, bias=False),
        nn.BatchNorm2d(ngf * 2),
        nn.ReLU(True),
        # state size. (ngf*2) x 16 x 16
        nn.ConvTranspose2d( ngf * 2, ngf, kernel_size, 2, 1, bias=False),
        nn.BatchNorm2d(ngf),
        nn.ReLU(True),
        # state size. (ngf) x 32 x 32
        nn.ConvTranspose2d( ngf, nc, kernel_size, 2, 1, bias=False),
        nn.Tanh()
        # state size. (nc) x 64 x 64
    )

def forward(self, input):
    return self.main(input)

discriminator code:
class Discriminator(nn.Module):
def __init__(self, ngpu):
    super(Discriminator, self).__init__()
    self.ngpu = ngpu
    self.main = nn.Sequential(
        # input is (nc) x 64 x 64
        nn.Conv2d(nc, ndf, kernel_size, 2, 1, bias=False),
        nn.LeakyReLU(0.2, inplace=True),
        # state size. (ndf) x 32 x 32
        nn.Conv2d(ndf, ndf * 2, kernel_size, 2, 1, bias=False),
        nn.BatchNorm2d(ndf * 2),
        nn.LeakyReLU(0.2, inplace=True),
        # state size. (ndf*2) x 16 x 16
        nn.Conv2d(ndf * 2, ndf * 4, kernel_size, 2, 1, bias=False),
        nn.BatchNorm2d(ndf * 4),
        nn.LeakyReLU(0.2, inplace=True),
        # state size. (ndf*4) x 8 x 8
        nn.Conv2d(ndf * 4, ndf * 8, kernel_size, 2, 1, bias=False),
        nn.BatchNorm2d(ndf * 8),
        nn.LeakyReLU(0.2, inplace=True),
        # state size. (ndf*8) x 4 x 4
        nn.Conv2d(ndf * 8, 1, kernel_size, 1, 0, bias=False),

        # 1x1x1
        nn.Sigmoid()
    )

def forward(self, input):
    return self.main(input)

The error showing line is:
In the training loop
        # Calculate loss on all-real batch
    errD_real = criterion(output, label)

Here criterion = nn.BCELoss()
Whole training loop:
img_list = []
G_losses = []
D_losses = []
iters = 0
# For each epoch
for epoch in range(num_epochs):
    # For each batch in the dataloader
    for i, data in enumerate(dataloader, 0):

        ############################
        # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
        ###########################
        ## Train with all-real batch
        netD.zero_grad()
        # Format batch
        real_cpu = data[0].to(device)
        #real_cpu = (data.unsqueeze(dim=1).type(torch.FloatTensor)).to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label,dtype=torch.float, device=device, )
        # Forward pass real batch through D
        output = netD(real_cpu).view(-1)
        # Calculate loss on all-real batch
        errD_real = criterion(output, label)#ERROR
        # Calculate gradients for D in backward pass
        errD_real.backward()
        D_x = output.mean().item()

        ## Train with all-fake batch
        # Generate batch of latent vectors
        noise = torch.randn(b_size, nz, 1, 1, device=device)
        # Generate fake image batch with G
        fake = netG(noise)
        label.fill_(fake_label)
        # Classify all fake batch with D
        output = netD(fake.detach()).view(-1)
        # Calculate D's loss on the all-fake batch
        errD_fake = criterion(output, label)
        # Calculate the gradients for this batch
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        # Add the gradients from the all-real and all-fake batches
        errD = errD_real + errD_fake
        # Update D
        optimizerD.step()

        ############################
        # (2) Update G network: maximize log(D(G(z))) using 'log D' trick
        ###########################
        netG.zero_grad()
        label.fill_(real_label)  # fake labels are real for generator cost
        # Since we just updated D, perform another forward pass of all-fake batch through D
        output = netD(fake).view(-1)
        # Calculate G's loss based on this output
        errG = criterion(output, label)
        # Calculate gradients for G
        errG.backward()
        D_G_z2 = output.mean().item()
        # Update G
        optimizerG.step()

        # Output training stats
        if i % 50 == 0:
            print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f'
                  % (epoch, num_epochs, i, len(dataloader),
                     errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))

        # Save Losses for plotting later
        G_losses.append(errG.item())
        D_losses.append(errD.item())

        # Check how the generator is doing by saving G's output on fixed_noise
        if (iters % 500 == 0) or ((epoch == num_epochs-1) and (i == len(dataloader)-1)):
            with torch.no_grad():
                fake = netG(fixed_noise).detach().cpu()
            img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

        iters += 1


title: Pandas Dataframe to tensor

body:
I have a dataframe with 3 columns (a date index, a price and a string symbol).
It looks like that:

Date
Price
Symbol

2019-01-02
39.480000
AAPL

2019-01-02
101.120003
MSFT

2019-01-02
62.023998
TSLA

2019-01-03
35.547501
AAPL

2019-01-03
97.400002
MSFT

2019-01-03
60.071999
TSLA

I'm looking for some panda/pytorch/python syntactic sugar to turn that into a tensor/matrix that will be:
[ [ 39.480000, 101.120003, 62.023998], [35.547501, 97.400002, 60.071999]]

With the number length of the first dimension will be the number of unique dates, and the length of the second will be the number of unique symbols.
I'm guaranteed to have exactly 3 symbols per date and I want that each row of my matrix follow the same order for its columns (e.g always AAPL, MSFT, TSLA).
Now, that is very easy with some for loops, but I'm looking for something more "pythonic"


title: Implementing BandRNN with pytorch and tensorflow

body:
So I am trying to figure out how to train my matrix in a way that I will get a BandRNN.
BandRnn is a diagonalRNN model with a different number of connections per neuron.
For example:

C is the number of connections per neuron.
I found out that there is a way to turn off some of the gradients in a for loop, in a way that prevents them from being trained as follows:
for p in model.input.parameters():
         p.requires_grad = False

But I can't find a proper way to do so, in a way that will make my matrix become a BandRNN.
Hopefully, someone will be able to help me with this issue.


title: Is the pytorch_transformers from huggingface maintained?

body: Huggingface has two libraries Pytorch_transformers 1.2.0 and transformers 4. and others? There are some papers using the code from pytorch_transformers and I am trying to implement some production based solutions on pytorch_transformers? Are the huggingface maintaining the "pytorch_transformers" library?


title: Time Series Classification Problem: Training EEG dataset with Sliding Window and Majority Vote

body:
I'm working to a classification problem (Emotion Recognition throuhg EEG data) with a custom dataset (not publically available, for now) in pytorch.
EEG data are samples of shape (C, T), with C being the number of channels (electrodes) and T the number of time-points (variable-lenth samples, say T in range [9000-120000] time points). Each sample has a emotion label for emotion classification.
I'm trying to train a dataset with 400 samples. For each epoch, from each sample a random trial of shape (C, 1280) is selected with the given label.
Since the used time-points are only 1/10 of the whole sample, I was thinking to create a trainer that, for each epoch, for each sample:

gets N trials of size (C, 1280), sliding a window of 1280 time-points over the whole sample;
gets N labels, one for each trial;
assigns to the sample, the label that got the majority among the N labels.

The solution I'm thinking about is that of:

create N datasets, each one samples the eeg data, fetching a fixed window;
Train simoultaneisy a model using the N given datasets (using SequentialSampler, simultaneously batches will be formed based on the same sample);
calculate the cross-entropy using only the label that gave the majority among the N labels.

I'm not very happy about this solution: is it necessary to create N Datasets?
Does, for example, DataLoader in pytorch could fetch the sample N times, with a sliding window instead?


title: How to index a tensor using a mask and scalar

body:
I have a tensor y
import torch
y = torch.rand(1, 4, 50, 100, 2)

And I have a mask (notice, it lacks the last dimension in comparison with y)
y_mask = torch.rand(1, 4, 50, 100) > 0.5

And I want to modify another tensor with the same shape as y using the mask but the last dimension of y should be 0.
new_y[other_mask, 0] = y[mask, 0]

I can't do it. I have an error:
IndexError: The shape of the mask [1, 4, 50, 100] at index 1 does not match the shape of the indexed tensor [1, 50, 100, 2] at index 1


title: How to remove inplace operation error in Pytorch?

body:
I get this error from the following Pytorch code:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [3]] is at version 10; expected version 9 instead.

As it is seen the code does not have inplace operations.
import torch
device = torch.device('cpu')
class MesNet(torch.nn.Module):
        def __init__(self):
            super(MesNet, self).__init__()
            self.cov_lin = torch.nn.Sequential(torch.nn.Linear(6, 5)).double()
        def forward(self, u):
            z_cov = self.cov_lin(u.transpose(0, 2).squeeze(-1))
            return z_cov
class UpdateModel(torch.nn.Module):
    def __init__(self):
        torch.nn.Module.__init__(self)
        self.P_dim = 18
        self.Id3 = torch.eye(3).double()
    def run_KF(self):
        N = 10
        u = torch.randn(N, 6).double()
        v = torch.zeros(N, 3).double()
        model = MesNet()
        measurements_covs_l = model(u.t().unsqueeze(0))
        # remember to remove this afterwards
        torch.autograd.set_detect_anomaly(True)
        for i in range(1, N):
            v[i] = self.update_pos(v[i].detach(), measurements_covs_l[i-1])

        criterion = torch.nn.MSELoss(reduction="sum")
        targ = torch.rand(10, 3).double()
        loss = criterion(v, targ)
        loss = torch.mean(loss)
        loss.backward()
        return v, p

    def update_pos(self, v, measurement_cov):
        Omega = torch.eye(3).double()
        H = torch.ones((5, self.P_dim)).double()
        R = torch.diag(measurement_cov)
        Kt = H.t().mm(torch.inverse(R))
        # it is indicating inplace error even with this:
        # Kt = H.t().mm(R)
        dx = Kt.mv(torch.ones(5).double())
        dR = self.trans(dx[:9].clone())
        v_up = dR.mv(v)
        return v_up

    def trans(self, xi):
        phi = xi[:3].clone()
        angle = torch.norm(phi.clone())

        if angle.abs().lt(1e-10):

            skew_phi = torch.eye(3).double()
            J = self.Id3 + 0.5 * skew_phi
            Rot = self.Id3 + skew_phi
        else:
            axis = phi / angle
            skew_axis = torch.eye(3).double()
            s = torch.sin(angle)
            c = torch.cos(angle)

            Rot = c * self.Id3
        return Rot
net =  UpdateModel()
net.run_KF()


title: R: Optimizing with Torch - optimizer$step() throws up this error

body:
Using Torch in R to create a simple neural network, with optim_adam as the optimizer.  While trying to update the model parameters, I get the following error on running optimizer$step() - "Error in param$state[["exp_avg"]] : object of type 'closure' is not subsettable".
What could be the reason?
I get the error on running this code (https://blogs.rstudio.com/ai/posts/2020-10-09-torch-optim/) as well.
d_in <- 3
d_out <- 1
n <- 100

# create random data
x <- torch_randn(n, d_in)
y <- x[, 1, NULL] * 0.2 - x[, 2, NULL] * 1.3 - x[, 3, NULL] * 0.5 + torch_randn(n, 1)

d_hidden <- 32

model <- nn_sequential(
  nn_linear(d_in, d_hidden),
  nn_relu(),
  nn_linear(d_hidden, d_out)
)

learning_rate <- 0.08

optimizer <- optim_adam(model$parameters, lr = learning_rate)

### training loop --------------------------------------------------------------

for (t in 1:200) {

  y_pred <- model(x)

  loss <- nnf_mse_loss(y_pred, y, reduction = "sum")

  optimizer$zero_grad()
  loss$backward()

  optimizer$step()
}

Thanks!


title: Pytorch model not learning on multiple batches or large batch sizes

body:
Background Info
My data is in the shape of (11423, 2, 10, 34). I am dealing with sports data, where each instance is a matchup of two teams, with 10 players each, and 34 stats per player. I have data from 11423 matchups.
I want to use a 1D convolutional filter that is 10x1, to slide to the right across the 34 statistics for all 10 players on the two channels.
My target variable is the point differential for the game.
I am attempting to roughly recreate this architecture:
CNN Architecture
My Problem
I can train and get near zero loss on a single batch of up to 39 training examples.
However as soon as I increase to 40 training examples, my model simply does not learn, as the loss stays about the same on every epoch, and the outputs are all the same number (which I found is about the mean of all the ground truth targets).
Moreover, if I try to split up a set of say 32 training examples into two batches of 16, my model does not learn. I get the same problem of the loss not decreasing and predicting the same output for every training example. What could be causing this behavior?
Example
torch.manual_seed(0)
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

batch_size = 39

X_train_tensor = torch.from_numpy(X_train).float()[:batch_size]
y_train_tensor = torch.from_numpy(y_train).float()[:batch_size]

print("X_train_tensor shape:", X_train_tensor.shape,
      "\nX_test_tensor shape:", y_test_tensor.shape,
     "\n")

# Model Architecture
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.conv1 = nn.Conv1d(2, 2, kernel_size=(10,1), stride=1)
        self.fc1 = nn.Linear(72, 32)
        self.fc2 = nn.Linear(32, 1)

    def forward(self, x):

        x = F.tanh(self.conv1(x))
        x = x.reshape(-1, 72)
        x = F.tanh(self.fc1(x))
        x = self.fc2(x)
        return x

# Instantiating Model
net = Net()
loss_function = nn.MSELoss()
optimizer = optim.Adam(net.parameters(), lr=0.1)

epochs = 300

# Training Loop
for e in range(epochs):
    net.train()

    for i in range(0, len(X_train_tensor), batch_size):
        batch_X = X_train_tensor[i:i+batch_size]
        batch_y = y_train_tensor[i:i+batch_size]

        net.zero_grad()

        output = net(batch_X)

        loss= loss_function(output, batch_y)

        loss.backward()
        optimizer.step()
    if e %10 == 0:
        print("loss:", loss)

print("model predictions:\n", output.detach().numpy().flatten(), "\nGround Truth:\n", batch_y.detach().numpy().flatten())

Loss and Outputs when batch_size = 39
Changing batch_size to 40 Breaks It
Loss and Outputs when batch_size = 40
Notice the model predicts and output of 2.1 for all the examples, which is the mean of the ground truths.
I have tried changing the learning rate, using batch normalization, and double checking that my inputs are actually different (they are!).
Any ideas of what is causing this and how to fix it? Thank you for any input you have!


title: ERROR:optimizer got an empty parameter list

body:
Here is code:
import torch
import matplotlib.pyplot as plt
from sklearn.manifold import TSNE
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import Node2Vec
from collections import namedtuple

dataset = Planetoid(root='tmp/', name='Cora')
data = dataset[0]

device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = Node2Vec(data.edge_index, embedding_dim=128, walk_length=20,
                 context_size=10, walks_per_node=10, num_negative_samples=1,
                 sparse=True).to(device)
loader = model.loader(batch_size=128, shuffle=True, num_workers=4)

optimizer = torch.optim.SparseAdam(model.parameters(), lr=0.01)

when running these codes
optimizer = torch.optim.SparseAdam(model.parameters(), lr=0.01)

I got:
ValueError: optimizer got an empty parameter list

I am so confused and I can't solve the problem. Thanks for your help!


title: TypeError: an integer is required (got type tuple) , when calling next( iter( ) )

body:
I was trying to load some data using pytorch, the code is like the followings:
test_ds = ImageFolder(root="./test", transform=data_transform)

test_dl = DataLoader(test_ds,batch_size=12)

x , y= next(iter(test_dl))

when next(iter(test_dl)) is called, it throws TypeError: an integer is required (got type tuple) , I could not figure out why, since earlier when I did the same task using MAC, the result is OK, does this has something to do with OS
The full traceback:
TypeError                                 Traceback (most recent call last)
<ipython-input-89-cecf634332ce> in <module>
----> 1 next(iter(test_dl))

D:\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in __next__(self)
    312         if self.num_workers == 0:  # same-process loading
    313             indices = next(self.sample_iter)  # may raise StopIteration
--> 314             batch = self.collate_fn([self.dataset[i] for i in indices])
    315             if self.pin_memory:
    316                 batch = pin_memory_batch(batch)

D:\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in <listcomp>(.0)
    312         if self.num_workers == 0:  # same-process loading
    313             indices = next(self.sample_iter)  # may raise StopIteration
--> 314             batch = self.collate_fn([self.dataset[i] for i in indices])
    315             if self.pin_memory:
    316                 batch = pin_memory_batch(batch)

D:\Anaconda3\lib\site-packages\torchvision\datasets\folder.py in __getitem__(self, index)
    101         sample = self.loader(path)
    102         if self.transform is not None:
--> 103             sample = self.transform(sample)
    104         if self.target_transform is not None:
    105             target = self.target_transform(target)

D:\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py in __call__(self, img)
     47     def __call__(self, img):
     48         for t in self.transforms:
---> 49             img = t(img)
     50         return img
     51

D:\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py in __call__(self, img)
    544         """
    545         i, j, h, w = self.get_params(img, self.scale, self.ratio)
--> 546         return F.resized_crop(img, i, j, h, w, self.size, self.interpolation)
    547
    548     def __repr__(self):

D:\Anaconda3\lib\site-packages\torchvision\transforms\functional.py in resized_crop(img, i, j, h, w, size, interpolation)
    329     assert _is_pil_image(img), 'img should be PIL Image'
    330     img = crop(img, i, j, h, w)
--> 331     img = resize(img, size, interpolation)
    332     return img
    333

D:\Anaconda3\lib\site-packages\torchvision\transforms\functional.py in resize(img, size, interpolation)
    204             return img.resize((ow, oh), interpolation)
    205     else:
--> 206         return img.resize(size[::-1], interpolation)
    207
    208

D:\Anaconda3\lib\site-packages\PIL\Image.py in resize(self, size, resample, box)
   1890         self.load()
   1891
-> 1892         return self._new(self.im.resize(size, resample, box))
   1893
   1894     def rotate(

TypeError: an integer is required (got type tuple)


title: Linear Regression with Pytorch : constant loss

body:
I'm working on a linear regression problem with Pytorch (y=A*x, where the dimensions of A are 2x2). I wrote the following code. I don't know why the loss doesn't change... Can someone help me ?
Thanks,
Thomas
import torch
import numpy as np
from scipy.integrate import odeint
from matplotlib import pyplot as plt
from torch.autograd import Variable

def EDP(X,t):
    X_0=-2*X[0]
    X_1=-2*X[1]
    grad=np.array([X_0,X_1])
    return grad

T=np.arange(0,10,0.1)
X_train=odeint(EDP,[10,20],T)

Y_train=np.zeros_like(X_train)
for i in range(Y_train.shape[0]):
    Y_train[i,:]=np.dot(np.array([[2,0],[0,2]]),X_train[i,:])
print(X_train,Y_train)

X_train=torch.Tensor(X_train)
torch.transpose(X_train,0,1)
Y_train=torch.Tensor(Y_train)
print(X_train.shape)

import torch.nn as nn

class LinearRegression(torch.nn.Module):

    def __init__(self):
        super(LinearRegression, self).__init__()
        self.linear = torch.nn.Linear(2,2,bias = False) # bias is default True

    def forward(self, x):
        y_pred = self.linear(x)
        return y_pred

criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(our_model.parameters(), lr = 0.0001)

our_model = LinearRegression()

x_train = X_train
y_train = Y_train
#x_train.requires_grad=True
print(x_train.shape)
print(y_train.shape)
ntrain=10

for t in range(ntrain):

    y_pred=our_model(x_train)
    loss=criterion(y_train,y_pred)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    print(t,loss)

print(our_model.linear.weight)


title: Pytorch: how to change requires_grad to be true in an OrderedDict

body:
Suppose I have a neural network object from torch.nn, by default the requires_grad is False for its parameters. I want to change it to be True. But the following naive approach fails:
From torch import nn
a = nn.Linear(1, 1)
a.state_dict()[‘weight’].requires_grad = True
print(a.state_dict()[‘weight’].requires_grad)

The result is False. Could anyone explain what the problem is and how to fix it? Thank you! My torch version is 1.7.1.


title: Load a saved NN model in different Python file

body:
I am trying to implement the code from a Pytorch beginner's tutorial. But I have written the code for loading the saved model in another Python file.
The FashionClassify file contains the code exactly as its in the tutorial.
Below is the code:
from FashionClassify import NeuralNetwork
from FashionClassify import test_data
import torch

model = NeuralNetwork()
model.load_state_dict(torch.load("model.pth"))
classes = [
    "T-shirt/top", "Trouser","Pullover","Dress","Coat","Sandal","Shirt","Sneaker","Bag","Ankle boot",
]

model.eval()
x, y = test_data[0][0], test_data[0][1]
with torch.no_grad():
    pred = model(x)
    predicted, actual = classes[pred[0].argmax(0)],classes[y]
    print(f'Predicted: "{predicted}", Actual: "{actual}"')

However, when I run this, the entire training process starts again. Why is that so ?
OR
Is it an expected behavior ?
(I have gone through a couple of webpages and StackOverflow answers but couldn't find my problem)
FashionClassify file code:
import torch
from torch import nn
from torch.utils.data import DataLoader  # wraps an iterable around dataset
from torchvision import datasets  # stores samples and their label
from torchvision.transforms import ToTensor, Lambda, Compose
import matplotlib as plt

training_data = datasets.FashionMNIST(root='data', train=True, download=True, transform=ToTensor(), )
test_data = datasets.FashionMNIST(root='data', train=False, download=True, transform=ToTensor(), )

batch_size = 64
train_dataLoader = DataLoader(training_data, batch_size=batch_size)
test_dataLoader = DataLoader(test_data, batch_size=batch_size)

for X, y in test_dataLoader:
    print('Shape of X [N,C,H,W]:', X.size())
    print('Shape of y:', y.shape, y.dtype)
    break

device = 'cuda' if torch.cuda.is_available() else 'cpu'
print('Using {} device'.format(device))

# to define a NN, we inherit a class from nn.Module
class NeuralNetwork(nn.Module):
    def __init__(self):
        # will specify how data will proceed in the forward pass
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(28 * 28, 512),
            nn.ReLU(),
            nn.Linear(512, 512),
            nn.ReLU(),
            nn.Linear(512, 10),
            nn.ReLU()
        )

    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits

model = NeuralNetwork().to(device)
print(model)

loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

def train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X,y) in enumerate(dataloader):
        X,y = X.to(device), y.to(device)

        #compute prediction error
        pred = model(X)
        loss = loss_fn(pred, y)

        #backprop
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch%100 ==0:
            loss,current = loss.item(), batch * len(X)
            print(f"loss: {loss:>7f} [{current:>5d}/{size:>5d}]")

def test(dataloader, model):
    size = len(dataloader.dataset)
    model.eval()
    test_loss, correct = 0,0
    with torch.no_grad():
        for X, y in dataloader:
            X,y = X.to(device), y.to(device)
            pred = model(X)
            test_loss += loss_fn(pred, y).item()
            correct +=  (pred.argmax(1) == y).type(torch.float).sum().item()
    test_loss /= size
    correct /= size
    print(f"Test Error: \n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \n")

epochs = 5
for t in range(epochs):
    print(f"Epoch {t+1}\n-------------------------------")
    train(train_dataLoader, model, loss_fn, optimizer)
    test(test_dataLoader, model)
print("Done!")

torch.save(model.state_dict(), "model.pth")
print("Saved PyTorch Model State to model.pth")


title: Wav2Vec pytorch element 0 of tensors does not require grad and does not have a grad_fn

body:
I am retraining a wav2vec model from hugging face for classification problem. I have 5 classes and the input is a list of tensors [1,400].
Here is how I am getting the model
num_labels = 5
model_name = "Zaid/wav2vec2-large-xlsr-53-arabic-egyptian"
model_config = AutoConfig.from_pretrained(model_name, num_labels=num_labels)  ##needed for the visualizations
tokenizer = Wav2Vec2CTCTokenizer.from_pretrained(model_name)
model = Wav2Vec2ForCTC.from_pretrained(model_name, config=model_config)

Here is the model updated settings
# Freeze the pre trained parameters
for param in model.parameters():
    param.requires_grad = False
criterion = nn.MSELoss().to(device)
optimizer = AdamW(model.parameters(), lr=2e-5, eps=1e-6)

# Add three new layers at the end of the network
model.classifier = nn.Sequential(
    nn.Linear(768, 256),
    nn.Dropout(0.25),
    nn.ReLU(),
    nn.Linear(256, 64),
    nn.Dropout(0.25),
    nn.ReLU(),
    nn.Linear(64, 2),
    nn.Dropout(0.25),
    nn.Softmax(dim=1)
)

Then the training loop
print_every = 300

total_loss = 0
all_losses = []
model.train()
for epoch in range(2):
    print("Epoch number: ", epoch)
    for row in range(16918):
        Input = torch.tensor(trn_ivectors[row]).double()
        label = torch.tensor(trn_labels[row]).long().to(device)
        label = torch.unsqueeze(label,0).to(device)
        #print("Label", label.shape)
        Input = torch.unsqueeze(Input,1).to(device)
        #print(Input.shape)
        optimizer.zero_grad()

        #Input.requires_grad = True
        Input = F.softmax(Input[0], dim=-1)

        if label == 0:
            label = torch.tensor([1.0, 0.0]).float().to(device)
        elif label == 1:
            label = torch.tensor([0.0, 1.0]).float().to(device)

        # print(overall_output, label)

        loss = criterion(Input, label)
        total_loss += loss.item()

        loss.backward()
        optimizer.step()

        if idx % print_every == 0 and idx > 0:
            average_loss = total_loss / print_every
            print("{}/{}. Average loss: {}".format(idx, len(train_data), average_loss))
            all_losses.append(average_loss)
            total_loss = 0

torch.save(model.state_dict(), "model_after_train.pt")

Unfortunately when I try to train the program it gives me the following error
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Please I would appreciate if you could tell me how to fix this error. I have been searching a lot on a way fixing it but didn't fix it
Thanks


title: Training a model on GPU is very slow

body:
I am using A100-SXM4-40GB Gpu but training is terribly slow. I tried two models, a simple classification on cifar and a Unet on Cityscapes. I tried my code on other GPUs and it worked totally fine, but I do not know why training on this high capacity GPU is super slow.
I would appreciate any help.
Here are some other properties of GPUs.
GPU 0: A100-SXM4-40GB
GPU 1: A100-SXM4-40GB
GPU 2: A100-SXM4-40GB
GPU 3: A100-SXM4-40GB

Nvidia driver version: 460.32.03
cuDNN version: Could not collect


I have a 3 file. In the datamodule file, I have created data and used the basic format of the PyTorch Lightning. In the linear_model I made a linear regression model based on this page. Finally, I have a train file, I am calling the model and trying to fit the data. But I am getting this error
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/mostafiz/Dropbox/MSc/Thesis/regreesion_EC/src/test_train.py", line 10, in <module>
    train_dataloader=datamodule.DataModuleClass().setup().train_dataloader(),
AttributeError: 'tuple' object has no attribute 'train_dataloader'

Sample datamodule file
class DataModuleClass(pl.LightningDataModule):
    def __init__(self):
        super().__init__()
        self.sigma = 5
        self.batch_size = 10
        self.prepare_data()

def prepare_data(self):
    x = np.random.uniform(0, 10, 10)
    e = np.random.normal(0, self.sigma, len(x))

    y = x + e

    X = np.transpose(np.array([x, e]))

    self.x_train_tensor = torch.from_numpy(X).float().to(device)
    self.y_train_tensor = torch.from_numpy(y).float().to(device)

    training_dataset = TensorDataset(self.x_train_tensor, self.y_train_tensor)
    self.training_dataset = training_dataset

def setup(self):
    data = self.training_dataset
    self.train_data, self.val_data = random_split(data, [8, 2])
    return self.train_data, self.val_data

def train_dataloader(self):
    return DataLoader(self.train_data)

def val_dataloader(self):
    return DataLoader(self.val_data)

Sample training file
from . import datamodule, linear_model

model = linear_model.LinearRegression(input_dim=2, l1_strength=1, l2_strength=1)

trainer = pl.Trainer()
trainer.fit(model,
            train_dataloader=datamodule.DataModuleClass().setup().train_dataloader(),
            val_dataloaders=datamodule.DataModuleClass().setup().val_dataloaders())

Let me know if you need more code or explanation.
Update (Based on the comment)
Now, I am getting the following error after removing self.prepare_data() from the __init__() of the DataModuleClass(), removed return self.train_data, self.val_data from setup(), and changed the test file to
data_module = datamodule.DataModuleClass()

trainer = pl.Trainer()
trainer.fit(model,data_module)

Error:
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/mostafiz/Dropbox/MSc/Thesis/regreesion_EC/src/test_train.py", line 10, in <module>
    train_dataloader=datamodule.DataModuleClass().train_dataloader(),
  File "/home/mostafiz/Dropbox/MSc/Thesis/regreesion_EC/src/datamodule.py", line 54, in train_dataloader
    return DataLoader(self.train_data)
AttributeError: 'DataModuleClass' object has no attribute 'train_data'


title: Calculate Flops in Pytorch and Tensorflow are not equal?

body:
Given the same model, I found that the calculated flops in pytorch and tensorflow are different.
I used the keras_flops (https://pypi.org/project/keras-flops/) in tensorflow, and ptflops (https://pypi.org/project/ptflops/) in pytorch to calculate flops. It seems that flops in pytorch are closed to my own calculation by hands.
Is that tensorflow has some tricks to speed up the computation so that few flops are measured?
My model in tensorflow
d=56
s=12

inp = Input((750 ,750, 1))
x = Conv2D(d, (5,5), padding='same')(inp)
x = PReLU()(x)

x = Conv2D(s, (1,1), padding='valid')(x)
x = PReLU()(x)

x = Conv2D(s, (3,3), padding='same')(x)
x = PReLU()(x)
x = Conv2D(s, (3,3), padding='same')(x)
x = PReLU()(x)
x = Conv2D(s, (3,3), padding='same')(x)
x = PReLU()(x)
x = Conv2D(s, (3,3), padding='same')(x)
x = PReLU()(x)

x = Conv2D(d, (1,1), padding='same')(x)
x = PReLU()(x)
out = Conv2DTranspose(1 ,(9,9), strides=(4, 4),padding='same',output_padding = 3)(x)

The Flops output in tensorflow is:
Profile:
node name | # float_ops
Conv2D                   8.92b float_ops (100.00%, 61.95%)
Conv2DBackpropInput      5.10b float_ops (38.05%, 35.44%)
Neg                      180.00m float_ops (2.61%, 1.25%)
BiasAdd                  105.75m float_ops (1.36%, 0.73%)
Mul                      90.00m float_ops (0.63%, 0.63%)

======================End of Report==========================
The FLOPs is:14.3 GFlops

However, the FLops in pytorch is
Model_1(
  0.013 M, 100.000% Params, 45.486 GMac, 100.000% MACs,
  (begin): Sequential(
    0.002 M, 11.804% Params, 0.851 GMac, 1.870% MACs,
    (0): Conv2d(0.001 M, 11.367% Params, 0.819 GMac, 1.801% MACs, 1, 56, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (1): PReLU(0.0 M, 0.437% Params, 0.032 GMac, 0.069% MACs, num_parameters=56)
  )
  (middle): Sequential(
    0.007 M, 52.775% Params, 3.803 GMac, 8.360% MACs,
    (0): Conv2d(0.001 M, 5.340% Params, 0.385 GMac, 0.846% MACs, 56, 12, kernel_size=(1, 1), stride=(1, 1))
    (1): PReLU(0.0 M, 0.094% Params, 0.007 GMac, 0.015% MACs, num_parameters=12)
    (2): Conv2d(0.001 M, 10.212% Params, 0.736 GMac, 1.618% MACs, 12, 12, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): PReLU(0.0 M, 0.094% Params, 0.007 GMac, 0.015% MACs, num_parameters=12)
    (4): Conv2d(0.001 M, 10.212% Params, 0.736 GMac, 1.618% MACs, 12, 12, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (5): PReLU(0.0 M, 0.094% Params, 0.007 GMac, 0.015% MACs, num_parameters=12)
    (6): Conv2d(0.001 M, 10.212% Params, 0.736 GMac, 1.618% MACs, 12, 12, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): PReLU(0.0 M, 0.094% Params, 0.007 GMac, 0.015% MACs, num_parameters=12)
    (8): Conv2d(0.001 M, 10.212% Params, 0.736 GMac, 1.618% MACs, 12, 12, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): PReLU(0.0 M, 0.094% Params, 0.007 GMac, 0.015% MACs, num_parameters=12)
    (10): Conv2d(0.001 M, 5.684% Params, 0.409 GMac, 0.900% MACs, 12, 56, kernel_size=(1, 1), stride=(1, 1))
    (11): PReLU(0.0 M, 0.437% Params, 0.032 GMac, 0.069% MACs, num_parameters=56)
  )
  (final): ConvTranspose2d(0.005 M, 35.420% Params, 40.833 GMac, 89.770% MACs, 56, 1, kernel_size=(9, 9), stride=(4, 4), padding=(4, 4), output_padding=(3, 3))
)
Computational complexity:       45.49 GMac


title: Why does Pytorch claim my input has 4 channels?

body:
I was trying to train an SRGAN-model on my dataset of text images. Those are 100 160x30 .png files.

Here is the model:
https://github.com/leftthomas/SRGAN/blob/master/model.py
https://github.com/leftthomas/SRGAN/blob/master/train.py
(I didn't change anything apart from the path to my dataset)

I set the crop_size to 24, the upscale to 2 and start the training, but then the following error occurs:

RuntimeError: Given groups=1, weight of size [64, 3, 9, 9], expected input[64, 4, 12, 12] to have 3 channels, but got 4 channels instead

I am sure that all of my images have 3 channels, which is confirmed by img.shape.

Nevertheless, I am stuck and, honestly, I have no idea where exactly this fourth channel might be.


title: A strange behavior about pytorch tensor. Any one can explain it clear?

body:
When I create a PyTorch tensor and tried to explore its type, I found this：

>>> a = torch.rand(3,5)
>>> a
tensor([[0.3356, 0.0968, 0.2185, 0.9843, 0.7846],
        [0.8523, 0.3300, 0.7181, 0.2692, 0.6523],
        [0.0523, 0.9344, 0.3505, 0.8901, 0.6464]])
>>> type(a)
<class 'torch.Tensor'>
>>> a = a.cuda()
>>> a.is_cuda
True
>>> type(a)
<class 'torch.Tensor'>
>>> a.dtype
torch.float32
>>>

Why is type(a) still torch.Tensor rather than torch.cuda.Tensor, even though this tensor is already on GPU?


title: Running Caffe2 Model on android - my model running very slow on the android device(dosen't finish the init phase)

body:
I'm trying to run a Caffe2 model on an android device, i ran the AICameraDemo App successfully but my model init net is 7 times larger than sqeeznet model, and the load of the init net to the workspace with the predictor or with RunNetOnce takes forever.

I saw posts about this, most of them from 2018 so i'm wondering if anything change since then.

If it is not possible, is there another framework which i could run a model this big on an android device ?

Thanks for any answer.


title: Pytorch Question from 'Deep Reinforcement Learning: Hands-On'

body:
I'm reading Maxim Lapan's Deep Learning Hands On. I came across this code in chapter 2 and I don't understand a few things. Could anybody explain why the output of print(out) gives three parameters instead of the single float tensor we put in. Also, why is the super function necessary here? Finally, what is the x parameter that forward is accepting? Thank you.

class OurModule(nn.Module):
    def __init__(self, num_inputs, num_classes, dropout_prob=0.3):  #init
        super(OurModule, self).__init__() #Call OurModule and pass the net instance (Why is this necessary?)
        self.pipe = nn.Sequential( #net.pipe is the nn object now
            nn.Linear(num_inputs, 5),
            nn.ReLU(),
            nn.Linear(5, 20),
            nn.ReLU(),
            nn.Linear(20, num_classes),
            nn.Dropout(p=dropout_prob),
            nn.Softmax(dim=1)
        )

    def forward(self, x): #override the default forward method by passing it our net instance and (return the nn object?). x is the tensor? This is called when 'net' receives a param?
        return self.pipe(x)

if __name__ == "__main__":
    net = OurModule(num_inputs=2, num_classes=3)
    print(net)
    v = torch.FloatTensor([[2, 3]])
    out = net(v)
    print(out) #[2,3] put through the forward method of the nn? Why did we get a third param for the output?
    print("Cuda's availability is %s" % torch.cuda.is_available()) #find if gpu is available
    if torch.cuda.is_available():
        print("Data from cuda: %s" % out.to('cuda'))

OurModule.__mro__


title: Why I am getting this error when i am trying to run this and getting the loss training?

body:
Why I am facing this error when I am trying to run this code below?
the error is in this section of the code (predictions.append(prd.item())) still not understand why its error?

train_accuracies = []
train_loss = []
predictions = []
for epoch in range(10):
    iterations = 0
    running_loss = 0
    for i,(inputs,labels) in enumerate(train_loader):

        iterations+=1

        inputs = inputs.float()
        labels = labels.long()

        # Feed Forward
        output = net(inputs)
        # Loss Calculation
        loss = criterion(output, labels)

        running_loss = running_loss + loss.item()
        _, prd = torch.max(output, dim = 1)
        predictions.append(prd.item())
        accuracy = (prd == labels).float().mean()
        train_accuracies.append(accuracy.item())
        train_loss.append(running_loss / iterations)

        #i = i.view(i.shape[0], -1)

        # Clear the gradient buffer (we don't want to accumulate gradients)
        optimizer.zero_grad()
        # Backpropagation
        loss.backward()
        # Weight Update: w <-- w - lr * gradient
        optimizer.step()

        #print("Epoch [{}][{}/{}], Loss: {:.3f}".format(epoch, i, len(train_loader), running_loss / iterations))
        print("Epoch [{}][{}/{}], Loss: {:.3f}".format(epoch ,i , len(train_loader), running_loss))

The error is:

ValueError                                Traceback (most recent call last)

<ipython-input-41-8bf6c94dc5dd> in <module>()
     19         running_loss = running_loss + loss.item()
     20         _, prd = torch.max(output, dim = 1)
---> 21         predictions.append(prd.item())
     22         accuracy = (prd == labels).float().mean()
     23         train_accuracies.append(accuracy.item())

ValueError: only one element tensors can be converted to Python scalars

How to fix this?


title: Pytorch Geometric dataset and torchvision

body:
I've met some trouble when building the gnn to classify images. I want to use CIFAR-10 as my dataset from the torchvision. However, I find it difficult converting this dataset into the class of pytorch_geometric.data.Dataset.

I wonder is there any way to solve this problem. At first I try to modify all images in CIFAR-10 to graph structure, which is too slow and space-wasted.

Thanks!


title: Pytorch nn.CrossEntropyLoss stays near 5 without convergence

body:
I have build a simpel DL net with the Pytorch ResNet API to train the Stanford Cars Dataset. My task is image classification. I use the CrossEntropyLoss as my loss function.

transform = T.Compose([
    T.Resize(224),
    T.CenterCrop(224),
    T.ToTensor(),
    T.Normalize(mean=[.5, .5, .5], std=[.5, .5, .5])
])

class data_dataset(Dataset):

    def __init__(self, img_path, train_labelspath, img_transform=None):
        with open(train_labelspath, 'r') as f:
            lines = f.readlines()
            self.img_list = [os.path.join(img_path, i.split()[1]) for i in lines]
            self.labels_list = [i.split()[2] for i in lines]
            self.img_transform = img_transform
        f.close()

    def __getitem__(self, index):
        img = self.img_list[index]
        label = self.labels_list[index]
        img = Image.open(img).convert('RGB').resize((224, 224))
        if self.img_transform is not None:
            img = self.img_transform(img)
        return img, label

    def __len__(self):
        return len(self.labels_list)

train_data = data_dataset(data_path, train_labelpath, img_transform=transform)
train_loader = DataLoader(dataset=train_data, batch_size=4, shuffle=True, num_workers=0)

net = models.resnet50(pretrained=True)
net.fc = nn.Linear(2048, 196)
#net.add_module = nn.Softmax(dim=1)

optimizer = optim.Adam(net.parameters(), lr=0.0001)
epoch = 1000
criterion = nn.CrossEntropyLoss()
keras.initializers.glorot_uniform(net.parameters)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

for ep in range(epoch):
    for i, (img, label) in enumerate(train_loader):
        net.train()
        label = np.array(label, int)
        label = torch.LongTensor(label)
        img = torch.FloatTensor(img)
        label.to(device)
        img.to(device)
        optimizer.zero_grad()
        outputs = net(img)

        loss = criterion(outputs, label - 1)
        loss.backward()
        if i % 10 == 0:
            print("epoch:{}, batch:{}, loss is:{}".format(ep, i, loss))

        optimizer.step()

However the Loss function has not changed：

epoch:0, batch:0, loss is:5.312277793884277
epoch:0, batch:10, loss is:5.654966354370117
epoch:0, batch:20, loss is:5.4962687492370605
epoch:0, batch:30, loss is:5.62110710144043
...
epoch:0, batch:590, loss is:5.2842936515808105
epoch:0, batch:600, loss is:5.304643630981445
epoch:0, batch:610, loss is:5.143134117126465
epoch:0, batch:620, loss is:5.408026218414307

Why? This problem has troubled me for four days. I don't konw what should I do. Thanks for your help.