title: numpy.empty() data type not understood error on Google colab

body:
numpy.empty() returns data type not understood exception on Google colab's default numpy library. I checked all questions in stackoverflow but didnt see related to this problem since i am using google colab. Here is full code and exception output:
Full code:
def diffMask(img1=None, img2=None, opt=None, dataset=None, args=None):
    netG = args[0]
    netB = args[1]
    netD = args[2]
    f = args[3]
    res_path = opt.results_Stage3
    res_folders = ['temp_masks',
                   'temp_Stage2',
                   'temp_ref',
                   'temp_diff',
                   'temp_Stage3',
                   'temp_skel',
                   'temp_res',
                   'temp_Stage1',
                   'temp_src']
    for x in res_folders:
        if os.path.isdir("{}{}".format(res_path, x)) == False:
            os.mkdir("{}{}".format(res_path, x))
    save_masks = "{}{}".format(res_path, "temp_masks")
    save_Stage2 = "{}{}".format(res_path, "temp_Stage2")
    save_ref = "{}{}".format(res_path, "temp_ref")
    save_diff = "{}{}".format(res_path, "temp_diff")
    save_Stage3 = "{}{}".format(res_path, "temp_Stage3")
    save_skel = "{}{}".format(res_path, "temp_skel")
    save_res = "{}{}".format(res_path, "temp_res")
    save_Stage1 = "{}{}".format(res_path, "temp_Stage1")
    save_src = "{}{}".format(res_path, "temp_src")

    resize2 = transforms.Resize(size=(128, 128))
    src, mask, style_img, target, gt_cloth, skel, cloth = dataset.get_img("{}_0.jpg".format(img1[:-6]),
                                                                          "{}_1.jpg".format(img1[:-6]))
    src, mask, style_img, target, gt_cloth, skel, cloth = src.unsqueeze(0), mask.unsqueeze(0), style_img.unsqueeze(
        0), target.unsqueeze(0), gt_cloth.unsqueeze(0), skel.unsqueeze(0), cloth.unsqueeze(0)  # , face.unsqueeze(0)
    src1, mask1, style_img1, target1, gt_cloth1, skel1, cloth1 = Variable(src.cuda()), Variable(mask.cuda()), Variable(
        style_img.cuda()), Variable(target.cuda()), Variable(gt_cloth.cuda()), Variable(skel.cuda()), Variable(
        cloth.cuda())  # , Variable(face.cuda())
    src, mask, style_img, target, gt_cloth, skel, cloth = dataset.get_img("{}_0.jpg".format(img2[:-6]),
                                                                          "{}_1.jpg".format(img2[:-6]))
    src, mask, style_img, target, gt_cloth, skel, cloth = src.unsqueeze(0), mask.unsqueeze(0), style_img.unsqueeze(
        0), target.unsqueeze(0), gt_cloth.unsqueeze(0), skel.unsqueeze(0), cloth.unsqueeze(0)  # , face.unsqueeze(0)
    src2, mask2, style_img2, target2, gt_cloth2, skel2, cloth2 = Variable(src.cuda()), Variable(mask.cuda()), Variable(
        style_img.cuda()), Variable(target.cuda()), Variable(gt_cloth.cuda()), Variable(skel.cuda()), Variable(
        cloth.cuda())

    gen_targ_Stage1, s_128, s_64, s_32, s_16, s_8, s_4 = netG(skel1, cloth2)  # gen_targ11 is structural change cloth
    gen_targ_Stage2, s_128, s_64, s_32, s_16, s_8, s_4 = netB(src1, gen_targ_Stage1,
                                                              skel1)  # gen_targ12 is Stage2 image

    # saving structural
    pic_Stage2 = (torch.cat([gen_targ_Stage2], dim=0).data + 1) / 2.0
    #     save_dir = "/home/np9207/PolyGan_res/temp_Stage2/"
    save_image(pic_Stage2, '%s/%d_%s_%d.jpg' % (save_Stage2, f, img1[:-6], 0), nrow=1)

    msk1 = mask1[0, :, :, :].detach().cpu().permute(1, 2, 0)
    plt.imsave("{}/{}_{}_mask.jpg".format(save_masks, f, img1[:-6]), msk1, cmap="gray")

I got exception on this line:
plt.imsave("{}/{}_{}_mask.jpg".format(save_masks, f, img1[:-6]), msk1, cmap="gray")

Exception Output:
Traceback (most recent call last):
  File "test.py", line 208, in test
    diffMask(image1, image2, opt, test_loader, args)
  File "test.py", line 96, in diffMask
    plt.imsave("{}/{}_{}_mask.jpg".format(save_masks, f, img1[:-6]), msk1, cmap="gray")
  File "/usr/local/lib/python3.6/dist-packages/matplotlib/pyplot.py", line 2066, in imsave
    return matplotlib.image.imsave(fname, arr, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/matplotlib/image.py", line 1550, in imsave
    rgba = sm.to_rgba(arr, bytes=True)
  File "/usr/local/lib/python3.6/dist-packages/matplotlib/cm.py", line 217, in to_rgba
    xx = np.empty(shape=(m, n, 4), dtype=x.dtype)
TypeError: data type not understood

Numpy Version : 1.18.5
Thank you in advance.


title: PyTorch Multiprocessing validation freezes

body:
I am trying to go through many images for testing the model. So I create N instances of the network and send it to function with data and wait for the losses.
from torch.multiprocessing import Process, Queue, cpu_count

def _multiprocFunc(aQ, _procID,
                   inData, labelsImg, _model_valid):
    _tmp = _model_valid(inData) ############### ALL PROCESSES HANG HERE ###############
    crit = LossFunc.diffraLossSq_perImage()
    result = crit(_tmp, labelsImg).detach().numpy()
    aQ.put(list([_procID, result]))

def validation(X_valid, Y_valid, model_valid, valid_device = 'cpu'):

    #... some code here

    modelSet = []
    while(numProcessed<numProc):
        if (not aQ.empty()):
            aQElement = aQ.get()
            #... some code here

        if(not done):
            procID += 1
            modelSet.append(copy.deepcopy(model_valid))
            Process(target = _multiprocFunc, args = (aQ, procID,
                                       data, Labels, modelSet[-1])).start()

What happens is that all processes hang at _model_valid(inData)....I tested the function with a single call and it works fine and fast.
Any comment is appreciated.


title: Gradient Exploding Using CIFAR-10 Dataset from the Official Website

body:
I want to train a VGG16 model with Horovod PyTorch on 4 GPUs. Instead of using the CIFAR10 dataset of torch vision.datasets.CIFAR10, I would like to split the dataset on my own. So I downloaded the dataset from the official website and split the dataset. This is how I split the data:
if __name__ == '__main__':
    import pickle

    train_data, train_label = [], []
    test_data, test_label = [], []
    for i in range(1, 6):
        with open('/Users/wangqipeng/Downloads/cifar-10-batches-py/data_batch_{}'.format(i), 'rb') as f:
            b = pickle.load(f, encoding='bytes')
        train_data.extend(b[b'data'].tolist()[:8000])
        train_label.extend(b[b'labels'][:8000])
        test_data.extend(b[b'data'].tolist()[8000:])
        test_label.extend(b[b'labels'][8000:])
    num_train = len(train_data)
    num_test = len(test_data)
    print(num_train, num_test)
    train_data = np.array(train_data)
    test_data = np.array(test_data)
    for i in range(4):
        with open('/Users/wangqipeng/Downloads/train_{}'.format(i), 'wb') as f:
            d = {b'data': train_data[int(0.25 * i * num_train): int(0.25 * (i + 1) * num_train)],
                 b'labels': train_label[int(0.25 * i * num_train): int(0.25 * (i + 1) * num_train)]}
            pickle.dump(d, f)
    with open('/Users/wangqipeng/Downloads/test'.format(i), 'wb') as f:
        d = {b'data': test_data,
             b'labels': test_label}
        pickle.dump(d, f)

However, I found that if I use the dataset that I downloaded from the official website, there will be an exploding gradient problem. I found that the loss will increase and be "nan" after several iterations. This is how I read the dataset:
class DataSet(torch.utils.data.Dataset):
    def __init__(self, path):
        self.dataset = unpickle(path)

    def __getitem__(self, index):
        data = torch.tensor(
            self.dataset[b'data'][index], dtype=torch.float32).resize(3, 32, 32)
        return data, torch.tensor(self.dataset[b'labels'][index])

    def __len__(self):
        return len(self.dataset[b'data'])

train_dataset = DataSet("./cifar10/train_" + str(hvd.rank()))
train_loader = torch.utils.data.DataLoader(
    train_dataset, batch_size=args.batch_size, sampler=None, **kwargs)

If I print the loss of every iteration, I see something like this:
Mon Nov  9 11:28:29 2020[0]<stdout>:epoch 0 iter[ 0 / 313 ] loss 7.725658416748047 accuracy 5.46875
Mon Nov  9 11:28:29 2020[0]<stdout>:epoch 0 iter[ 1 / 313 ] loss 15.312677383422852 accuracy 8.59375
Mon Nov  9 11:28:29 2020[0]<stdout>:epoch 0 iter[ 2 / 313 ] loss 16.333066940307617 accuracy 9.375
Mon Nov  9 11:28:30 2020[0]<stdout>:epoch 0 iter[ 3 / 313 ] loss 15.549728393554688 accuracy 9.9609375
Mon Nov  9 11:28:30 2020[0]<stdout>:epoch 0 iter[ 4 / 313 ] loss 14.090616226196289 accuracy 9.843750298023224
Mon Nov  9 11:28:31 2020[0]<stdout>:epoch 0 iter[ 5 / 313 ] loss 12.310989379882812 accuracy 9.63541641831398
Mon Nov  9 11:28:31 2020[0]<stdout>:epoch 0 iter[ 6 / 313 ] loss 11.578919410705566 accuracy 9.15178582072258
Mon Nov  9 11:28:31 2020[0]<stdout>:epoch 0 iter[ 7 / 313 ] loss 13.210229873657227 accuracy 8.7890625
Mon Nov  9 11:28:32 2020[0]<stdout>:epoch 0 iter[ 8 / 313 ] loss 764.713623046875 accuracy 9.28819477558136
Mon Nov  9 11:28:32 2020[0]<stdout>:epoch 0 iter[ 9 / 313 ] loss 4.590414250749922e+20 accuracy 8.984375
Mon Nov  9 11:28:32 2020[0]<stdout>:epoch 0 iter[ 10 / 313 ] loss nan accuracy 9.446022659540176
Mon Nov  9 11:28:33 2020[0]<stdout>:epoch 0 iter[ 11 / 313 ] loss nan accuracy 10.09114608168602
Mon Nov  9 11:28:33 2020[0]<stdout>:epoch 0 iter[ 12 / 313 ] loss nan accuracy 10.39663478732109

However, If I use the dataset from torchvision, everything will be fine:
train_dataset = \
    datasets.CIFAR10(args.train_dir, download=True,
                         transform=transforms.Compose([transforms.ToTensor(),
                           transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))]))
train_sampler = torch.utils.data.distributed.DistributedSampler(
    train_dataset, num_replicas=hvd.size(), rank=hvd.rank())
train_loader = torch.utils.data.DataLoader(
    train_dataset, batch_size=args.batch_size, sampler=train_sampler, **kwargs)

There can also be something wrong with the DistributedSampler. But I think the DistributedSampler only serves to split the data. I don't know whether the DistributedSampler can be a reason to this problem.
Is there something wrong with the way I read the CIFAR10 dataset? Or is there something wrong with the way I "reshape" it? Thanks for your help!


title: Why doesn't setting random seed give same performance across runs?

body:
I am training some deep learning models using pytorch, which also includes the usage of numpy. Since the randomisation is not truly random and it is pseudo-random, why aren't the numbers i.e., accuracy etc same across different runs?
I mean, even if I do not set some random seed, there should be some default random seed according to which my code must run and give same results across different runs. Is there something more to it?


title: Pytorch CrossEntropyLoss Tensorflow Equivalent

body:
I am currently translating a pytorch code into tensorflow.
There is a point that i am aggregating 3 losses in a tensorflow custom loop and i get an error that i am passing a two dimensional array vs a 1 dimensional into CategoricalCrossEntropy of tensorflow which is very legit and i understand why this happens... but in the pytorch code i am passing the same shapes and it's perfectly working with CrossEntropyLoss. Does anybody know what i have to do to transfer this into TF?
the shapes that are passing in are (17000,100) vs (17000)


title: What does this code in PyTorch do? How can I express it with tensorflow

body:
I found a code that would solve my problem that looks like this:
(self.conv_diag(input_tensor.diagonal(dim1=2, dim2=3))).diag_embed(dim1=2, dim2=3)

While self.conv_diag is a layer I have defined before.
As far as I understood it extracts the diagonal of a subtensor in the second and third dimension puts it into the layer and constructs a new tensor filled with zeros and replaces its second and third dimension with the new values calculated by my layer.
What I have found to extract the diagonal is
tf.math.reduce_diag(input_tensor)

but I cannot choose the axis and I have not yet found an equivalent function to replace torch.diag_embed()
How can I express it in Tensorflow?


title: What is the opacity graph in tensorboard?

body:
I am new to Tensorboard.
This is an output I have by this line:
self.logger.experiment.add_scalars("losses", {"train_loss": loss}, global_step=self.current_epoch)

I can only touch the dark blue line, which is probably the loss I logged, but not the light blue line.

What is the light blue line?
Why is the dark blue line not strictly continuous?


title: Translate TensorFlow to PyTorch

body:
I'm trying to translate the following code from tensorflow to pytorch:
decoder = GLOGenerator()
g_optimizer = tf.keras.optimizers.Adam(0.0001)
z_optimizer = tf.keras.optimizers.Adam(0.01)

def train_step(images, z_train):
    with tf.GradientTape() as g_tape, tf.GradientTape() as z_tape:
        result_images = decoder(z_train)
        g_loss = tf.keras.losses.mean_squared_error(images, tf.squeeze(result_images))
    gradients_z = z_tape.gradient(g_loss, z_train)
    gradients_g = g_tape.gradient(g_loss, decoder.trainable_variables)

    z_optimizer.apply_gradients(zip([gradients_z], [z_train]))
    g_optimizer.apply_gradients(zip(gradients_g, decoder.trainable_variables))

If it necessary, the calling function is here (if you need to understand what is the arguments of the train_step function):
z_vectors = np.random.normal(0, 1, (number_of_images, 10))
BATCH_SIZE = 32
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
images = x_train
number_of_images = images.shape[0]

def train(number_of_epoches):
    image_indexs = np.arange(images.shape[0])
    number_of_steps = images.shape[0] // BATCH_SIZE
    np.random.shuffle(image_indexs)
    for epoch in range(number_of_epoches):
        print("epoch: ", epoch)
        np.random.shuffle(image_indexs)
        for index in range(number_of_steps):
            z_for_train = tf.Variable(z_vectors[image_indexs[BATCH_SIZE * index: index * BATCH_SIZE + BATCH_SIZE]], dtype='float32')
            train_step(images[image_indexs[BATCH_SIZE * index: index * BATCH_SIZE + BATCH_SIZE]], z_for_train)
            z_train = z_for_train.numpy()
            z_train /= np.max(np.linalg.norm(z_vectors, axis=0, keepdims=True), 1)
            z_vectors[image_indexs[BATCH_SIZE * index: index * BATCH_SIZE + BATCH_SIZE]] = z_train


title: Tensor manipulation - creating a positional tensor from a given tensor

body:
I have a input tensor which has zero padding at the start and then a sequence of values. So something like:
x = torch.tensor([[0, 2, 8, 12],
                  [0, 0, 6, 3]])

What I need is another tensor having same shape and retaining 0's for the padding and an increasing sequence for the rest of the numbers. So my output tensor should be:
y = ([[0, 1, 2, 3],
      [0, 0, 1, 2]])

I tried something like:
MAX_SEQ=4
seq_start = np.nonzero(x)
start = seq_start[0][0]
pos_id = torch.cat((torch.from_numpy(np.zeros(start, dtype=int)).to(device), torch.arange(1, MAX_SEQ-start+1).to(device)), 0)
print(pos_id)

This works if the tensor is 1 dimensional but needs additional logic to handle it for 2-D shape. This can be done as np.nonzeros returns a tuple and we could probably loop thru' those tuples updating a counter or something. However I am sure there must be a simple tensor operation which should do this in 1-2 lines of code and also perhaps more effectively.
Help appreciated


title: Pytorch CNN Loss is not changing

body:
I am making a CNN using Pytorch for an image classification problem between people who are wearing face masks and who aren't. The images are converted to a 256x256 with 3 channels.
When the code is run, whatever the initial loss value is will stay the same. I have tried changing the learning rates, however there is no change. I have also tried using a simple single linear layer to check if it was the model that was incorrect. My guess is that the training loop is incorrect, but when comparing to other sample training loops, my code looks correct.
Model
class NetConv(nn.Module):
    def __init__(self):
        super(NetConv, self).__init__()
        # self.conv1 = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=1, padding=1)
        # self.conv2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, stride=1, padding=1)
        # self.conv3 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, stride=1, padding=1)
        self.conv1 = nn.Conv2d(in_channels= 3, out_channels = 8, kernel_size = 3)
        self.conv2 = nn.Conv2d(in_channels= 8, out_channels = 16, kernel_size = 3)
        self.conv3 = nn.Conv2d(in_channels= 16, out_channels = 32, kernel_size = 3)

        self.fc1 = nn.Linear(30*30*32, 1000)
        self.fc2 = nn.Linear(1000, 200)
        self.fc3 = nn.Linear(200, 3)

    def forward(self, x):
        # print(x.shape)
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2)
        x = F.relu(self.conv3(x))
        x = F.max_pool2d(x, 2)

        x = x.reshape(-1, 30*30*32)

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        x = F.softmax(x, dim=1)

        # print(x.shape)

        return x

Dataloader
class faceMaskDataset(Dataset):
    def __init__(self, img_folder, annot_folder, transform=None):
        # Extracting image name and class from xml file
        desc = []
        for dirname, _, filenames in os.walk(annot_folder):
            for filename in filenames:
                desc.append(os.path.join(dirname, filename))

        img_name,label = [],[]

        for d in desc:
            content = []
            n = []

            with open(d, "r") as file:
                content = file.readlines()
            content = "".join(content)
            soup = BeautifulSoup(content,"html.parser")
            file_name = soup.filename.string
            name_tags = soup.find_all("name")

            for t in name_tags:
                n.append(t.get_text())

            # selecting tag with maximum occurence in an image (If it has multiple tags)
            name = max(set(n), key = n.count)

            img_name.append(file_name)
            label.append(name)

        labels = pd.get_dummies(label)
        print(labels.head())

        # Our target classes
        classes = list(labels.columns)
        print(classes)

        data, target = [],[]
        img_h, img_w = 256, 256

        # Loading images and converting them to pixel array
        for i in range(len(img_name)):
            name = os.path.join("./images/1", img_name[i])
            image = cv2.imread(name)
            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
            image = cv2.resize(image, (img_w, img_h), cv2.INTER_AREA)

            data.append(image)
            target.append(tuple(labels.iloc[i,:]))

        print(type(data))
        # data = np.array(data)
        data = np.array(data) / 255 # Normalise pixel data to between 0 and 1
        target = np.array(target)

        data = np.swapaxes(data, 1, 3)
        data = np.swapaxes(data, 2, 3)

        self.data = data

        self.target = target
        self.transform = transform

    def __len__(self):
        return len(self.target)

    def __getitem__(self, index):
        image = self.data[index,:]
        label = (self.target[index])
        return (image, label)

Training loop
def train(model, optimizer, loss_fn, train_dl, val_dl, epochs=20, device='cuda'):

    for epoch in range(epochs):

        model.train()
        train_loss         = 0.0
        num_train_correct  = 0
        num_train_examples = 0

        for i, (image, target) in enumerate(train_dl):
            image = image.to(device).float()
            target = target.to(device)

            optimizer.zero_grad()

            output = model(image)

            loss = loss_fn(output, torch.max(target, 1)[1])

            loss.backward()
            optimizer.step()

            train_loss         += loss.data.item() * image.size(0)
            num_train_correct  += (torch.max(output, 1)[1] == torch.max(target, 1)[1]).sum().item()

            num_train_examples += image.shape[0]

        train_acc   = num_train_correct / num_train_examples
        train_loss  = train_loss / len(train_dl.dataset)
        print("Epoch: %2d, Loss: %.3f, Acc: %.3f" % (epoch, train_loss, train_acc))


title: Pytorch Geometric split dataset by mask tensors

body: If I have a data object that also contains the .train_mask and .test_mask tensors, how can I use these to split the dataset into two data objects, a trainset and a testset?


title: Can't convert Pillow Image to numpy array or to RGB mode

body:
I'm trying to load a Dataset of PNGs. The PNGs are stored in "L" mode (8bit per pixel monochrome).

When trying to convert them with  image.convert('RGB'), I get the following error:

Traceback (most recent call last):
  File "C:\Users\Saman\AppData\Local\Programs\Python\Python38\lib\site-packages\PIL\ImageFile.py", line 180, in load
    seek = self.load_seek
AttributeError: 'PngImageFile' object has no attribute 'load_seek'

If I leave that step out and convert them to a numpy array directly with np.array(image), I get a numpy array with  dtype=object and length 1, with the only Pillow Image object as its content.


title: Add a layer to model state dictionary in PyTorch

body:
I have a neural network in PyTorch like,
class Net1(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.conv_g = ResBlock(2,81)
        self.conv_low1 = ResBlock(2,90)
        self.conv_low2 = ResBlock(2,90)
        self.conv_low3 = ResBlock(2,90)
        self.conv_low4 = ResBlock(2,90)
        self.conv_low5 = ResBlock(2,90)
        self.conv_low6 = ResBlock(2,90)
        self.conv_low7 = ResBlock(2,90)
        self.conv_low8 = ResBlock(2,90)
        self.conv_low9 = ResBlock(2,90)

        self.stagIII = stageIII()

    def forward(self,x):
        ...

        return out

To this network I have added a new layer m whose weights I know.
class Net2(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.m = nn.Linear(5,1)

        self.conv_g = ResBlock(2,81)
        self.conv_low1 = ResBlock(2,90)
        self.conv_low2 = ResBlock(2,90)
        self.conv_low3 = ResBlock(2,90)
        self.conv_low4 = ResBlock(2,90)
        self.conv_low5 = ResBlock(2,90)
        self.conv_low6 = ResBlock(2,90)
        self.conv_low7 = ResBlock(2,90)
        self.conv_low8 = ResBlock(2,90)
        self.conv_low9 = ResBlock(2,90)

        self.stagIII = stageIII()

    def forward(self,x):
        ...

        return out

The state dictionary of m.state_dict() is
OrderedDict([('weight', tensor([[ 1.0000,  2.0000,  3.0000,  4.0000, 5.0000]])), ('bias', tensor([0.]))])

Now given that I have Net1.state_dict() and m.state_dict(), how can I append m dictionary to Net1 dictionary to obtain state dictionary of Net2?


title: PyTorch nn.Conv2d output comptation

body:
I am using Python 3.8 and PyTorch 1.7.1. I saw a code which defines a Conv2d layer as follows:
Conv2d(3, 6, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)

The input 'X' being passed to it is a 4D tensor-
X.shape
# torch.Size([4, 3, 6, 6])

The output volume for this conv layer is:
c1(X).shape
# torch.Size([4, 6, 3, 3])

I am trying to use the formula to compute output spatial dimensions for any conv layer: O = ((W - K + 2P)/S) + 1, where W = spatial dimension of image, K = filter/kernel size, P = zero padding & S = stride.
For 'c1' conv layer, we get, W = 6, K = 3, S = 2 & P = 1. Using the formula, you get O = ((6 - 3 + (2 x 1)) / 2) + 1 = 5/2 + 1 = 3.5.
The output volume: (4, 6, 3, 3) since number of filters used = 6.
How is the spatial output from 'c1' then (3, 3)? What am I not getting?
Thanks!


title: Number of learnable parameters of MultiheadAttention

body:
While testing (using PyTorch's MultiheadAttention), I noticed that increasing or decreasing the number of heads of the multi-head attention does not change the total number of learnable parameters of my model.
Is this behavior correct? And if so, why?
Shouldn't the number of heads affect the number of parameters the model can learn?


title: How to create a 1 to 1 feed forward layer?

body:
I'm familiar with a fully connected layer, but how can I create a custom layer in PyTorch that is just 1 to 1? That is, each neuron is only connected to 1 other neuron.
Example: Layer 1 neurons: a,b,c
Layer 2 neurons d,e,f
Connections:
a-d
b-e
c-f


title: creating a progress bar for a validation test Python

body:
The following code is used to find p-values for a significant test on Cifar-10 database.
Because we need a min of 1000 permutations, it is a very slow process, and I want to inlude a progress bar to show how time for each permutation.
I was thinking of using the tqdm library and sleep function, but am stuck on where to go from here. any help would be greatly appreciated!
'''
def validate_significance(val_loader, model, criterion, args):
model.eval()

vec_acc1 = []
vec_acc1_chance = []
vec_acc5 = []
vec_acc5_chance = []

for ss in range(0, args.num_permutations):

    val_loader = get_rand_sample_loader(val_loader.dataset, args)

    acc1_over_one_permutaion = 0
    acc1_chance_over_one_permutation = 0
    acc5_over_one_permutaion = 0
    acc5_chance_over_one_permutation = 0
    cnt = 0
    with torch.no_grad():
        for i, (images, target) in enumerate(val_loader):

            if args.gpu is not None:
                images = images.cuda(args.gpu, non_blocking=True)
            if torch.cuda.is_available():
                target = target.cuda(args.gpu, non_blocking=True)

            # compute output
            output = model(images)

            # measure accuracy on true labels
            acc1, acc5 = accuracy(output, target, topk=(1, 5))
            acc1_over_one_permutaion += acc1.item()
            acc5_over_one_permutaion += acc5.item()

            # now, measure accuracy on permuted labels
            target = target[torch.randperm(len(target))]  # scrambling the labels
            acc1_perm, acc5_perm = accuracy(output, target, topk=(1, 5))
            acc1_chance_over_one_permutation += acc1_perm.item()
            acc5_chance_over_one_permutation += acc5_perm.item()
            cnt += 1

    vec_acc1.append(acc1_over_one_permutaion / cnt)
    vec_acc1_chance.append(acc1_chance_over_one_permutation / cnt)
    vec_acc5.append(acc5_over_one_permutaion / cnt)
    vec_acc5_chance.append(acc5_chance_over_one_permutation / cnt)

p_acc1 = stats.ttest_ind(vec_acc1, vec_acc1_chance, equal_var=False)
p_acc5 = stats.ttest_ind(vec_acc5, vec_acc5_chance, equal_var=False)

return p_acc1, p_acc5

'''


title: Floating point operations results are different in Android, TensorFlow and Pytorch

body:
I am trying to compare the floating point numbers on Android, Tensorflow and Pytorch. What I have observed is I am getting same result for Tensorflow and Android but different on Pytorch as Android and Tensorflow are performing round-down operation. Please see the following result:
TensorFlow
import tensorflow as tf
a=tf.convert_to_tensor(np.array([0.9764764, 0.79078835, 0.93181187]), dtype=tf.float32)

session = tf.Session()
result = session.run(a*a*a*a)

print(result)

PyTorch
import torch as th

th.set_printoptions(precision=8)

a=th.from_numpy(np.array([0.9764764, 0.79078835, 0.93181187])).type(th.FloatTensor)

result = a*a*a*a

print(result)

Android:
for (index in 0 until a.size) {
   var res = a[index] * a[index] * a[index] * a[index]
   result.add(res)
}

print("r=$result")

The result is as follows:
Android:           [0.9091739,  0.3910579,  0.7538986]
TensorFlow:        [0.9091739,  0.3910579,  0.7538986]
PyTorch:           [0.90917391, 0.39105791, 0.75389862]

You can see that PyTorch value is different. I know that this effect is minimal in this example but when we are performing training, and we are running for 1000 rounds with different batches and epochs, this different can be accumulated and can show un-desirable results. Can anyone point out how can we fix to have same number on three platforms.
Thanks.


title: Type error using pytorch2keras - Input 'y' of 'Mul' Op has type float32 that does not match type int32 of argument 'x'

body:
I am trying to convert a Pytorch model to Keras using the Pytorch2Keras library. I'm doing this on a pre-trained colorization model here.
Just to see if my model and the PyTorch weights are fine, I tried converting it into onnx format first. I am able to export it to onnx but not able to convert the onnx model to keras or tensorflow. I get the same error that I get while using the Pytorch2Keras library
The traceback is
ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/math_ops.py in binary_op_wrapper(x, y)
   1174             r_op = getattr(y, "__r%s__" % op_name)
-> 1175             out = r_op(x)
   1176             if out is NotImplemented:

19 frames
ValueError: Tensor conversion requested dtype int32 for Tensor with dtype float32: <tf.Tensor 'Cast:0' shape=() dtype=float32>

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/op_def_library.py in _apply_op_helper(op_type_name, name, **keywords)
    556                 "%s type %s of argument '%s'." %
    557                 (prefix, dtypes.as_dtype(attrs[input_arg.type_attr]).name,
--> 558                  inferred_from[input_arg.type_attr]))
    559
    560         types = [values.dtype]

TypeError: Input 'y' of 'Mul' Op has type int32 that does not match type float32 of argument 'x'

I looked into this post and from what I could understand, it has something to do with the newer version of tensorflow.
Is there any way I could find out what layer is causing the issue? I am really not sure how to go about this
The full code on Google Colab can be found here
Any help would be appreciated. Thanks!


title: Pythorch Error - UnboundLocalError: local variable 'out' referenced before assignment

body:
I'm trying to write an artificial neural network with Pytorch.I'm using the MNIST dataset.
I'm new to Pytorch. I could not solve the error.
I guess the fault is here "outfc1lrelu = self.lrelu(out)"
Codes:
import torch
import torch.nn as nn
from sklearn.datasets import load_breast_cancer

device = torch.device("cpu")

#Hyper Parameter
input_size = 30
hidden_size = 500
num_classes = 2
num_epoch = 100
learning_rate = 1e-8

girdi, cikti = load_breast_cancer(return_X_y=True)

train_input = torch.from_numpy(girdi).float()
train_output = torch.from_numpy(cikti)

class NeuralNet(nn.Module): #sinir agı torch u ımport etmıs oldu
    def __init__(self,input_size,hidden_size,num_classes):
        super(NeuralNet,self).__init__()
        self.fc1 = nn.Linear(input_size,hidden_size)
        self.lrelu = nn.LeakyReLU(negative_slope=0.02)
        self.fc2 = nn.Linear(hidden_size,num_classes)
    def forward(self,input):
        outfc1 = self.fc1(input)
        outfc1lrelu = self.lrelu(out)
        out = self.fc2(outfc1lrelu)
        return out

model = NeuralNet(input_size,hidden_size,num_classes)

lossf = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for epoch in range(num_epoch):
    outputs=model(train_input)
    loss=lossf(outputs,train_output)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch + 1,num_epoch,loss.item()))

Error part:
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-105-70765d74f98b> in <module>
      1 for epoch in range(num_epoch): #100 tekrar yapcaz
----> 2     outputs = model(train_input) #çıktı olarak il cıktıyı versın
      3     loss = lossf(outputs,train_output) #kaybı hesapladıkçoutputsu ardından traın outputsu alsın
      4
      5     optimizer.zero_grad() #0 a esıtledık

~\anaconda3\envs\pytorchTutorial\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    870         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
    871                 or _global_forward_hooks or _global_forward_pre_hooks):
--> 872             return forward_call(*input, **kwargs)
    873         # Do not call functions when jit is used
    874         full_backward_hooks, non_full_backward_hooks = [], []

<ipython-input-101-58455cd48d9f> in forward(self, input)
      8     def forward(self,input):
      9         outfc1 = self.fc1(input) #girdi degerımızı alıcaz
---> 10         outfc1lrelu = self.lrelu(out) #ilk katman fc1 girdi ardından lrelu gırdı cıktı olarak(out) belli bir deger geldı
     11         out = self.fc2(outfc1lrelu) #1 mı sıfır mı tahmın etsın
     12         return out

UnboundLocalError: local variable 'out' referenced before assignment

Why am I making this mistake? Where did it go wrong? I would be glad if you could help.


title: How to save a list of pytorch models

body:
This is a newbie question. I have trained 8 pytorch convolutional models and put them in a list called models. I can use them for prediction so they are working. I would like to save them. I can't even work out how to save one however. I tried:

torch.save(models[0].save_dict(), "test.pth")

but this gives me:

ModuleAttributeError: 'Net' object has no attribute 'save_dict'

type(models[0]) gives me __main__.Net
How can I save this list of models?


title: Confusion about Pytorch `torch.split` documentation

body:
When I view the explanation of the function torch.split in PyTorch, I find it difficult for me to read as a non-English-speaker:

torch.split(tensor, split_size_or_sections, dim=0)
[...]
If split_size_or_sections is a list, then tensor will be split
into len(split_size_or_sections) chunks with sizes in dim according
to split_size_or_sections.

Does "with sizes in dim" mean "with sizes in split_size_or_sections along the dimension dim"?


title: Loop optimization in Pytorch

body:
I'm trying to find a way to prevent a painfully slow for loop in Pytorch. Basically, I have a tensor, and I want to split it up into pieces and feed those pieces into my model, similar in spirit to a grouped convolution of sorts.
        self.C = C
        self.block = Block(C, 3, 64)
    def forward(self, x):
        x_shape = x.shape
        x = torch.flatten(x, start_dim=1, end_dim=-1).unsqueeze(1)
        x = torch.split(x, self.C, -1)
        attention = []
        for i in x:
            attended = self.block(i)
            attention.append(attended)
        attention = torch.stack(attention, 1)

Small values of C, alongside a large tensor, I think, makes this operation surprisingly much slower, due to the Python for-loop the above code runs through. However, when I exchange the batch dimension for a 'C' dimension and loop through the batch dimension instead, this causes significant speedups, however still feels hacky to me, and might still prove to be slow with a large enough batch size. I'd like a method to fix this while still keeping the batch-dim intact and avoiding the for loop. What I guess I'm looking for is a method to add a second batch dimension to my model, or something equivalent.
Is there any way to fix this issue other than the slightly-hacky method described above?
EDIT:
MWE: (Pretend like the single linear layer is something like a split Attention layer...)
import torch

class Net(torch.nn.Module):
    def __init__(self, split_size):
        super().__init__()
        self.split_size = split_size
        self.linear = torch.nn.Linear(split_size, split_size)
    def forward(self, x):
        #Slow implementation:

        #Input is B,C,H and is flattened to B,C*H.
        y = x.flatten(start_dim=1, end_dim=-1)

        y_split = torch.split(y, self.split_size, 1) #Tensor is split and each piece is fed into the model...

        outs = []

        for i in y_split:
            i_out = self.linear(i)
            outs.append(i_out)
        y = torch.cat(outs, 1)
        print(y.shape)

        #Fast implementation using batch dims, but possibly slower for large batches...
        y = x.flatten(start_dim=1, end_dim=-1)
        y_split = torch.split(y, self.split_size, 1)

        y = torch.stack(y_split, 0)
        outs = []
        for i in torch.split(y, 1, 1):
            i_out = self.linear(i.squeeze(1)).unsqueeze(0)
            outs.append(i_out)
        y = torch.cat(outs, 0)
        y = y.flatten(start_dim=1, end_dim=-1)
        print(y.shape)
        return y

if __name__ == "__main__":
    net = Net(32)
    net(torch.randn(256,3,32,32))
    net(torch.randn(32,3,32,32))


title: PyTorch Tensor Operation for adding the maximum of the previous row to the next

body:
Follow-Up question to PyTorch: Dynamic Programming as Tensor Operation.
Could the following be written as a tensor operation instead of a loop?
a = torch.Tensor([[1, 2, 3, 4],
                  [5, 6, 7, 8],
                  [9, 10, 11, 12]])

print(a.shape)
# (3, 4)

for i in range(1, a.shape[0]):
    a[i] = a[i-1].max(dim=0)[0] + a[i]

print(a)
# tensor([[ 1,  2,  3,  4],
#         [ 9, 10, 11, 12],
#         [21, 22, 23, 24]])

Basically adding the maximum of the previous row to all elements of the next.
The interesting part is that you can't compute the maximum for each row beforehand and then add that to the respective row, because adding the first maximum influences what the maximum of the next row is.


title: huggingface transformer CTRL model can not work for generation text

body:
torch ver: 1.8.1+cu102
transformers ver: 4.4.2
I adopt the example codes from https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py
to generate text by using ctrl.
here is the head part of my codes:
import torch
from transformers import CTRLTokenizer, CTRLLMHeadModel
tokenizer = CTRLTokenizer.from_pretrained('ctrl')
model = CTRLLMHeadModel.from_pretrained('ctrl')

encoded_prompt = tokenizer.encode("Links Hello, my dog is cute", add_special_tokens=False)

ERROR:
ValueError Traceback (most recent call last) in
----> 1 encoded_prompt = tokenizer.encode("Links Hello, my dog is cute", add_special_tokens=False)

~/yanan/env/lib/python3.6/site-packages/transformers/tokenization_utils_base.py in encode(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, return_tensors, **kwargs) 2030 stride=stride, 2031 return_tensors=return_tensors,
-> 2032 **kwargs, 2033 ) 2034

~/yanan/env/lib/python3.6/site-packages/transformers/tokenization_utils_base.py in encode_plus(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs) 2355 return_length=return_length, 2356 verbose=verbose,
-> 2357 **kwargs, 2358 ) 2359

~/yanan/env/lib/python3.6/site-packages/transformers/tokenization_utils.py in _encode_plus(self, text, text_pair, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs) 458 return_special_tokens_mask=return_special_tokens_mask, 459 return_length=return_length,
--> 460 verbose=verbose, 461 ) 462

~/yanan/env/lib/python3.6/site-packages/transformers/tokenization_utils_base.py in prepare_for_model(self, ids, pair_ids, add_special_tokens, padding, truncation, max_length, stride, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, prepend_batch_axis,
**kwargs) 2792 padding=padding_strategy.value, 2793 pad_to_multiple_of=pad_to_multiple_of,
-> 2794 return_attention_mask=return_attention_mask, 2795 ) 2796

~/yanan/env/lib/python3.6/site-packages/transformers/tokenization_utils_base.py in pad(self, encoded_inputs, padding, max_length, pad_to_multiple_of, return_attention_mask, return_tensors, verbose) 2591 else: 2592 raise ValueError(
-> 2593 f"type of {first_element} unknown: {type(first_element)}. " 2594 f"Should be one of a python, numpy, pytorch or tensorflow object." 2595 )

ValueError: type of None unknown: <class 'NoneType'>. Should be one of a python, numpy, pytorch or tensorflow object.


title: How to call method without method name in python

body:
class LinearRegression(nn.Module):
def __init__(self,input_size,output_size):
    # super function. It inherits from nn.Module and we can access everythink in nn.Module
    super(LinearRegression,self).__init__()
    # Linear function.
    self.linear = nn.Linear(input_dim,output_dim)

def forward(self,x):
    return self.linear(x)

input_dim = 1
output_dim = 1
model = LinearRegression(input_dim,output_dim)

In this code block when I want to call forward method in model object I can call 2 different ways

The first way
results = model.forward(car_price_tensor)

Second way
results = model(car_price_tensor)

And I try the second way on a different class and object it doesnt work. How it possible?


title: Is this a right way to train and test the model using Pytorch?

body:
I am trying to implement a binary classification with Neural Networks with Pytorch to train a model as shown below:

model = nn.Sequential(
    bnn.BayesLinear(prior_mu=0, prior_sigma=0.1, in_features=196, out_features=300),
    nn.ReLU(),
    bnn.BayesLinear(prior_mu=0, prior_sigma=0.1, in_features=300, out_features=196),
)
cross_entropy_loss = nn.CrossEntropyLoss()
klloss = bnn.BKLLoss(reduction='mean', last_layer_only=False)
klweight = 0.01
optimizer = optim.Adam(model.parameters(), lr=0.01)

Traing part:
 #training
    for step in range(200):
        models = model(data_tensor)
        cross_entropy = cross_entropy_loss(models, target_tensor)
        #cross_entropy = 0
        kl = klloss(model)
        total_cost = cross_entropy + klweight*kl

        optimizer.zero_grad()
        total_cost.backward()
        optimizer.step()

    _, predicted = torch.max(models.data, 1)
    final = target_tensor.size(0)
    correct = (predicted == target_tensor).sum()
    print('- Accuracy: %f %%' % (100 * float(correct) / final))
    print('- CE : %2.2f, KL : %2.2f' % (cross_entropy.item(), kl.item()))

Question 1: Is it a right way to train a model? In many articles I found that there is a section to iterate over the DataLoader for training data, such as :
for i, data in enumerate(trainloader, 0):
 # Get inputs
 inputs, targets = data

Question 2: What is the use of this loop if I can directly give the data features(inputs) as data_tensor and data labels(target) as target_tensor? Because when iterating through data loader, it takes more time.
I am not sure how to test my model. I did as given below:
correct = 0
total = 0
with torch.no_grad():
    for step in range(data_tensor_test.size(0)):
        models = model(data_tensor_test)
        _, predicted = torch.max(models.data, 1)
        total += target_tensor_test.size(0)
        correct += (predicted == target_tensor_test).sum().item()

print('Accuracy of the test data: %d %%' % (
    100 * correct / total))

Question 3: Is it right way to test the model that I trained above?


title: How to manually obtain the minus log-likelihood in Pytorch?

body:
I'm implementing a VAE and I want to obtain the negativev log-likelihood manually (not using an existing function). The given equation is equation1, and I have also found it can as well be expressed as equation2. I have been stuck on this for a couple of days now, and don't know where my code is wrong.
def loss_loglik(y_mean, y_logvar, x):
    out_1 = (x.size()[2]*x.size()[3] / 2) * np.log(2 * np.pi)
    out_2 = (x.size()[2]*x.size()[3] / 2) * torch.log(y_logvar.exp())
    x_diff = x - y_mean
    out_3 = torch.sum(x_diff.pow(2)) / (2 * y_logvar.exp())
    loss = out_1 + out_2 + out_3

The shape of the three arguments is (batch_size, 1, 28, 28).


title: Unable to replicate Keras model to Pytorch

body:
I am currently working on a DL project where I would like to reproduce the work from an existing research paper. It is about detecting sleep episodes from EEG data using CNN model. The source code is using Keras and I have hard time to reproduce the model in Pytorch.
The existing Keras model runs very fast and after 2 epochs it has the following stats ( This is 4 classes classification problem):
Training loss = 0.431
Training accuracy = 0.842
On validation set:
Kappa score per class = [0.373 0.5571 0.033 0.129]
Overall kappa = 0.319
accuracy = 0.687
precision = [0.993 0.514 0.029 0.115 ]
recall = [0.696 0.719 0.180 0.573]
f1 = [0.818 0.600 0.05 0.192]
I created the Pytorch version of the model, and run on the same data. It runs much slower and after 6 epochs this is what I got:
Training loss = 0.379899
Training accuracy = 0.869
On validation set:
Kappa score per class = [ 0.111 0.079 -0.016 0.040]
Overall kappa = 0.078
accuracy = 0.559
precision = [0.682 0.432 0.012 0.105]
recall = [0.797 0.106 0.035 0.155 ]
f1 = [0.735 0.170 0.018 0.125]
The only difference in these 2 models are in the initialization,optimizers, and loss function:

The Keras model uses Nadam optimizer, the Pytorch model uses SGD(I am not aware that Nadam is available in Pytorch)
The Keras model uses glorot-normal for kernel initialization, Pytorch model uses xavier_uniform.
The Keras model uses “Categorical Cross Entropy” loss function with softmax as the last output layer, Pytorch model uses “CrossEntrropyLoss”. I do not use Softtmax in the last layer since CrossEntropyLoss internally uses LogSoftmax.

I have been spending a lot of time trying to understand why my Pytorch model performs much worse with no avail. I would appreciate if gurus and experts could help advise here :slight_smile:
Here is the Keras model:
def build_model(data_dim, n_channels, n_cl):
eeg_channels = 1
act_conv = 'relu'
init_conv = 'glorot_normal'
dp_conv = 0.3
def cnn_block(input_shape):
    input = Input(shape=input_shape)
    x = GaussianNoise(0.0005)(input)
    x = Conv2D(32, (3, 1), strides=(1, 1), padding='same', kernel_initializer=init_conv)(x)
    x = BatchNormalization()(x)
    x = Activation(act_conv)(x)
    x = MaxPooling2D(pool_size=(2, 1), padding='same')(x)

    x = Conv2D(64, (3, 1), strides=(1, 1), padding='same', kernel_initializer=init_conv)(x)
    x = BatchNormalization()(x)
    x = Activation(act_conv)(x)
    x = MaxPooling2D(pool_size=(2, 1), padding='same')(x)
    for i in range(4):
        x = Conv2D(128, (3, 1), strides=(1, 1), padding='same', kernel_initializer=init_conv)(x)
        x = BatchNormalization()(x)
        x = Activation(act_conv)(x)
        x = MaxPooling2D(pool_size=(2, 1), padding='same')(x)
    for i in range(6):
        x = Conv2D(256, (3, 1), strides=(1, 1), padding='same', kernel_initializer=init_conv)(x)
        x = BatchNormalization()(x)
        x = Activation(act_conv)(x)
        x = MaxPooling2D(pool_size=(2, 1), padding='same')(x)
    flatten1 = Flatten()(x)
    cnn_eeg = Model(inputs=input, outputs=flatten1)
    return cnn_eeg

hidden_units1  = 256
dp_dense = 0.5

eeg_channels = 1
eog_channels = 2

input_eeg = Input(shape=( data_dim, 1,  3))
cnn_eeg = cnn_block(( data_dim, 1, 3))
x_eeg = cnn_eeg(input_eeg)
x = BatchNormalization()(x_eeg)
x = Dropout(dp_dense)(x)
x =  Dense(units=hidden_units1, activation=act_conv, kernel_initializer=init_conv)(x)
x = BatchNormalization()(x)
x = Dropout(dp_dense)(x)

predictions = Dense(units=n_cl, activation='softmax', kernel_initializer=init_conv)(x)

model = Model(inputs=[input_eeg] , outputs=[predictions])
return [cnn_eeg, model]

The model is used as follows:
[cnn_eeg, model] = build_model(data_dim, n_channels, n_cl)
Nadam = optimizers.Nadam( )
model.compile(optimizer='Nadam',  loss='categorical_crossentropy', metrics=['accuracy'], sample_weight_mode=None)
print(cnn_eeg.summary())
print(model.summary())

model.fit_generator(generator_train, steps_per_epoch = steps_per_epoch, class_weight = weight, epochs = 1, verbose=1,  callbacks=[history], initial_epoch=0 )

Printout of the model:
Layer (type)                 Output Shape              Param #

input_2 (InputLayer)         [(None, 3200, 1, 3)]      0

gaussian_noise (GaussianNois (None, 3200, 1, 3)        0

conv2d (Conv2D)              (None, 3200, 1, 32)       320

batch_normalization (BatchNo (None, 3200, 1, 32)       128

activation (Activation)      (None, 3200, 1, 32)       0

max_pooling2d (MaxPooling2D) (None, 1600, 1, 32)       0

conv2d_1 (Conv2D)            (None, 1600, 1, 64)       6208

batch_normalization_1 (Batch (None, 1600, 1, 64)       256

activation_1 (Activation)    (None, 1600, 1, 64)       0

max_pooling2d_1 (MaxPooling2 (None, 800, 1, 64)        0

conv2d_2 (Conv2D)            (None, 800, 1, 128)       24704

batch_normalization_2 (Batch (None, 800, 1, 128)       512

activation_2 (Activation)    (None, 800, 1, 128)       0

max_pooling2d_2 (MaxPooling2 (None, 400, 1, 128)       0

conv2d_3 (Conv2D)            (None, 400, 1, 128)       49280

batch_normalization_3 (Batch (None, 400, 1, 128)       512

activation_3 (Activation)    (None, 400, 1, 128)       0

max_pooling2d_3 (MaxPooling2 (None, 200, 1, 128)       0

conv2d_4 (Conv2D)            (None, 200, 1, 128)       49280

batch_normalization_4 (Batch (None, 200, 1, 128)       512

activation_4 (Activation)    (None, 200, 1, 128)       0

max_pooling2d_4 (MaxPooling2 (None, 100, 1, 128)       0

conv2d_5 (Conv2D)            (None, 100, 1, 128)       49280

batch_normalization_5 (Batch (None, 100, 1, 128)       512

activation_5 (Activation)    (None, 100, 1, 128)       0

max_pooling2d_5 (MaxPooling2 (None, 50, 1, 128)        0

conv2d_6 (Conv2D)            (None, 50, 1, 256)        98560

batch_normalization_6 (Batch (None, 50, 1, 256)        1024

activation_6 (Activation)    (None, 50, 1, 256)        0

max_pooling2d_6 (MaxPooling2 (None, 25, 1, 256)        0

conv2d_7 (Conv2D)            (None, 25, 1, 256)        196864

batch_normalization_7 (Batch (None, 25, 1, 256)        1024

activation_7 (Activation)    (None, 25, 1, 256)        0

max_pooling2d_7 (MaxPooling2 (None, 13, 1, 256)        0

conv2d_8 (Conv2D)            (None, 13, 1, 256)        196864

batch_normalization_8 (Batch (None, 13, 1, 256)        1024

activation_8 (Activation)    (None, 13, 1, 256)        0

max_pooling2d_8 (MaxPooling2 (None, 7, 1, 256)         0

conv2d_9 (Conv2D)            (None, 7, 1, 256)         196864

batch_normalization_9 (Batch (None, 7, 1, 256)         1024

activation_9 (Activation)    (None, 7, 1, 256)         0

max_pooling2d_9 (MaxPooling2 (None, 4, 1, 256)         0

conv2d_10 (Conv2D)           (None, 4, 1, 256)         196864

batch_normalization_10 (Batc (None, 4, 1, 256)         1024

activation_10 (Activation)   (None, 4, 1, 256)         0

max_pooling2d_10 (MaxPooling (None, 2, 1, 256)         0

conv2d_11 (Conv2D)           (None, 2, 1, 256)         196864

batch_normalization_11 (Batc (None, 2, 1, 256)         1024

activation_11 (Activation)   (None, 2, 1, 256)         0

max_pooling2d_11 (MaxPooling (None, 1, 1, 256)         0

flatten (Flatten)            (None, 256)               0

Total params: 1,270,528
Trainable params: 1,266,240
Non-trainable params: 4,288

None
Model: "model_1"

Layer (type)                 Output Shape              Param #

input_1 (InputLayer)         [(None, 3200, 1, 3)]      0

model (Functional)           (None, 256)               1270528

batch_normalization_12 (Batc (None, 256)               1024

dropout (Dropout)            (None, 256)               0

dense (Dense)                (None, 256)               65792

batch_normalization_13 (Batc (None, 256)               1024

dropout_1 (Dropout)          (None, 256)               0

dense_1 (Dense)              (None, 4)                 1028

Total params: 1,339,396
Trainable params: 1,334,084
Non-trainable params: 5,312

Below is my Pytorch model:
class MSECNN16s(nn.Module):

def __init__(self):
# input shape = (batchsize, 3, 1, windowsize=16*200=3200)

    super(MSECNN16s, self).__init__()
    self.conv1 = nn.Conv2d(3, 32, kernel_size=(1,3), padding=(0,1))
    self.conv2 = nn.Conv2d(32, 64, kernel_size=(1,3), padding=(0,1))
    self.conv3 = nn.Conv2d(64, 128, kernel_size=(1,3), padding=(0,1))

    self.conv4 = nn.ModuleList()
    for x in range(3):
        conv = nn.Conv2d(128, 128, kernel_size=(1,3), padding=(0,1))
        nn.init.xavier_uniform_(conv.weight)
        nn.init.zeros_(conv.bias)
        self.conv4.append(conv)

    self.conv5 = nn.Conv2d(128, 256, kernel_size=(1,3), padding=(0,1))

    self.conv6 = nn.ModuleList()
    for x in range(5):
        conv = nn.Conv2d(256, 256, kernel_size=(1,3), padding=(0,1))
        nn.init.xavier_uniform_(conv.weight)
        nn.init.zeros_(conv.bias)
        self.conv6.append(conv)

    self.fc1 = nn.Linear(256, 256)
    self.fc2 = nn.Linear(256, 4)

    nn.init.xavier_uniform_(self.conv1.weight)
    nn.init.zeros_(self.conv1.bias)

    nn.init.xavier_uniform_(self.conv2.weight)
    nn.init.zeros_(self.conv2.bias)

    nn.init.xavier_uniform_(self.conv3.weight)
    nn.init.zeros_(self.conv3.bias)

    nn.init.xavier_uniform_(self.conv5.weight)
    nn.init.zeros_(self.conv5.bias)

    nn.init.xavier_uniform_(self.fc1.weight)
    nn.init.zeros_(self.fc1.bias)

    nn.init.xavier_uniform_(self.fc2.weight)
    nn.init.zeros_(self.fc2.bias)

def forward(self, x):
    # x = (batchsize, 1, 3, windowsize=16*200)

    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

    x = x.permute(0, 2, 1, 3) #convert to (batchsize, 3, 1, 3200)

    std = 0.0005
    x = (torch.randn_like(x) * std) + x

    x = self.conv1(x)  #output: (batchsize, 32, 1, 3200)
    x = nn.BatchNorm2d(32).to(device)(x)
    x = F.relu(x)
    x = nn.MaxPool2d(kernel_size=(1,2))(x) #output: (batchsize, 32, 1, 1600)

    x = self.conv2(x) #output: (batchsize, 64, 1, 1600)
    x = nn.BatchNorm2d(64).to(device)(x)
    x = F.relu(x)
    x = nn.MaxPool2d(kernel_size=(1,2))(x) #output: (batchsize, 64, 1, 800)

    x = self.conv3(x) #output: (batchsize, 128, 1, 800)
    x = nn.BatchNorm2d(128).to(device)(x)
    x = F.relu(x)
    x = nn.MaxPool2d(kernel_size=(1,2))(x) #output: (batchsize, 128, 1, 400)

    for conv in self.conv4:
        x = conv(x)
        x = nn.BatchNorm2d(128).to(device)(x)
        x = F.relu(x)

        padding = (0,0)
        if x.shape[-1] % 2 > 0: padding = (0,1)
        x = nn.MaxPool2d(kernel_size=(1,2), padding = padding)(x)  # output channels will be 200->100->50

    x = self.conv5(x) #output: (batchsize, 256, 1, 50)
    x = nn.BatchNorm2d(256).to(device)(x)
    x = F.relu(x)
    x = nn.MaxPool2d(kernel_size=(1,2))(x) #output: (batchsize, 256, 1, 25)

    for conv in self.conv6:
        x = conv(x)
        x = nn.BatchNorm2d(256).to(device)(x)
        x = F.relu(x)

        padding = (0,0)
        if x.shape[-1] % 2 > 0: padding = (0,1)
        x = nn.MaxPool2d(kernel_size=(1,2), padding = padding)(x)  # output channels will be 13->7->4->2->1

    # x is (batchsize, 256, 1, 1)
    x = x.squeeze() #x is (batchsize, 256)
    x = nn.BatchNorm1d(256).to(device)(x)
    x = nn.Dropout(p=0.5)(x)
    x = self.fc1(x)
    x = F.relu(x)
    x = nn.BatchNorm1d(256).to(device)(x)
    x = nn.Dropout(p=0.5)(x)
    x = self.fc2(x)

    return x

with the following optimizer and criterion:
    model  = MSECNN16s()
model.to(device)

criterion = nn.CrossEntropyLoss(weight=torch.Tensor([1.0, 11.1, 102.9, 38.1]).to(device))

# load the optimizer
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9, weight_decay=0.0001  )

Any advice is appreciated! many thanks!
Regards
Edwin


title: PyTorch optimizer.step() doesn't update weights when I use "if statement" like function

body:
My model needs to learn certain parameters to solve this function:
self.a * (r > self.b) * (self.c)

Where
self.a, self.b, and self.c

are learnable parameters.
My issue is that b does not change during the gradient updates. I assume it's because the function is discontinuous. Though, it is a step function so I'm not sure how to modify it.
Any ideas/tips will be appreciated
My code is
import torch
import torch.nn as nn
import torch.optim as optim

class model(nn.Module):
        def __init__(self):
            super(model, self).__init__()
            self.a = torch.nn.Parameter(torch.rand(1, requires_grad=True))
            self.b = torch.nn.Parameter(torch.rand(1, requires_grad=True))
            self.c = torch.nn.Parameter(torch.rand(1, requires_grad=True))

model_net = model()
#function to learn = 5 * (r > 2) * (3)
optimizer = optim.Adam(model_net.parameters(), lr = 0.1)

for epoch in range(10):
  for r in range(10):
    optimizer.zero_grad()
    loss = 5 * (r > 2) * (3) - model_net.a * (r > model_net.b) * (model_net.c)
    loss.backward()
    optimizer.step()
  print(model_net.a)
  print(model_net.b)
  print(model_net.c)
  print()

Update 1:
I found this question to be quite similar. The person had a discontinuous function as well that seemed to be solved using tanh instead. Though, I haven't found a way to use the same method.
I also went through the tutorial that @seraph suggested, but I haven't found a discontinuous function there.


title: torchvision.transforms.RandomRotation fill argument not working

body:
I am working on tensors and want to rotate them with torchvision.transforms.RandomRotation and use the fill option.
import torch
import torchvision

img1 = torch.rand((1, 16, 16))
img2 = torchvision.transforms.RandomRotation(45, fill=1)(img1)

However, I always get:

Argument fill/fillcolor is not supported for Tensor input. Fill value is zero

and it gets not filled with ones. I have the same problem with torchvision.transforms.RandomPerspective.
I am using Python 3.8 and PyTorch 1.7.1. I tried using fill=(1,), which seemed to be a workaround, but it did not worked for me. Do you know what could be wrong?


title: How to understand a periodicity in the training loss using a pre-trained model of PyTorch?

body:
I'm using a pre-trained model from Pytorch ( Resnet 18,34,50) in order to classify images. During the training, a weird periodicity appears in the training as you can see in the image below. Did somebody already have a similar issue?In order to deal with the overfitting, I'm using Data augmentation in the preprocessing.
When using SGD as an optimizer with the following parameters, we obtain this sort of graph:

criterion: NLLLoss()
learning rate: 0.0001
epoch: 40
print every 40 iteration

We also try adam and Adam bound as optimizers but the same periodicity was observed.
Thank's in advance for your answer!
Here is the code :
def train_classifier():
    start=0
    stop=0
    start = timeit.default_timer()
    epochs = 40
    steps = 0
    print_every = 40

    model.to('cuda')
    epo=[]
    train=[]
    valid=[]
    acc_valid=[]
    for e in range(epochs):
        print('Currently running epoch',e,':')
        model.train()

        running_loss = 0

        for images, labels in iter(train_loader):

            steps += 1

            images, labels = images.to('cuda'), labels.to('cuda')

            optimizer.zero_grad()

            output = model.forward(images)
            loss = criterion(output, labels)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()

            if steps % print_every == 0:

                model.eval()

                # Turn off gradients for validation, saves memory and computations
                with torch.no_grad():
                    validation_loss, accuracy = validation(model, val_loader, criterion)

                print("Epoch: {}/{}.. ".format(e+1, epochs),
                      "Training Loss: {:.3f}.. ".format(running_loss/print_every),
                      "Validation Loss: {:.3f}.. ".format(validation_loss/len(val_loader)),
                      "Validation Accuracy: {:.3f}".format(accuracy/len(val_loader)))
                stop = timeit.default_timer()
                print('Time: ', stop - start)
                acc_valid.append(accuracy/len(val_loader))
                train.append(running_loss/print_every)
                valid.append(validation_loss/len(val_loader))
                epo.append(e+1)
                running_loss = 0
                model.train()
    return train,epo,valid,acc_valid


title: Parallelized simple optimizations in modern Pytorch or Tensorflow

body:
I have a task where have to quickly run N parallel simple optimizations with Adam optimizers. I have been doing this with tensorflow 1.x for a while but trying to update everything to 2.x or alternatively modern pytorch has resulted in much slower behavior.
I constructed minimal(at least to best of my ability) examples demonstrating my problem in tensorflow 1.x 2.x and pytorch, all on cpu.
https://gist.github.com/gftabor/abeb108fc9aa8b1c799bfc63287c2e5f
As you can see tensorflow 2.x takes roughly 8 times longer than tensorflow 1.x. Pytorch is similar to tf 2.x.
I assume the problem is dynamic graph executions so that is the rabbit hole I have dug down the most, but all I care about is performance.
Rabbit hole
I have been trying to match Tensorflow 1.x performance by statically building graphs using the tf.function and torch.jit.script functionality. Neither of them seem to give me nearly the expressive power of Tensorflow 1.x and neither of them seem match the performance of Tensorflow 1.x either, on my problem the parts I can build into static graph are just the forward + loss functions not a full adam optimization step. While with Tensorflow 1.x I can build N of these full adam optimization steps in parallel as a single static graph and pass data into it quickly.
Claim
It doesn't seem possible to get even half as fast as Tensorflow 1.x with either of the modern Tensorflow or pytorch libraries making updating very unattractive.
Is there a method to get large speedup in either Tensorflow 2.x or pytorch that I am missing?


title: Pytorch transfer learning error: The size of tensor a (16) must match the size of tensor b (128) at non-singleton dimension 2

body:
Currently, I'm working on an image motion deblurring problem with PyTorch. I have two kinds of images: Blurry images (variable = blur_image) that are the input image and the sharp version of the same images (variable = shar_image), which should be the output. Now I wanted to try out transfer learning, but I can't get it to work.
Here is the code for my dataloaders:
train_loader = torch.utils.data.DataLoader(train_dataset,
                                           batch_size=batch_size,
                                           shuffle = True)
validation_loader = torch.utils.data.DataLoader(valid_dataset,
                                                batch_size=batch_size,
                                                shuffle = False)
test_loader = torch.utils.data.DataLoader(test_dataset,
                                          batch_size=batch_size,
                                          shuffle = False)

Their shape:
Trainloader - Shape of blur_image [N, C, H, W]:  torch.Size([16, 3, 128, 128])
Trainloader - Shape of sharp_image [N, C, H, W]:  torch.Size([16, 3, 128, 128]) torch.float32
Validationloader - Shape of blur_image [N, C, H, W]:  torch.Size([16, 3, 128, 128])
Validationloader - Shape of sharp_image [N, C, H, W]:  torch.Size([16, 3, 128, 128]) torch.float32
Testloader- Shape of blur_image [N, C, H, W]:  torch.Size([16, 3, 128, 128])
Testloader- Shape of sharp_image [N, C, H, W]:  torch.Size([16, 3, 128, 128]) torch.float32

The way I use transfer learning (I thought that for the 'in_features' I have to put in the amount of pixels):
model = models.alexnet(pretrained=True)
model.classifier[6] = torch.nn.Linear(model.classifier[6].in_features, 128)
device_string = "cuda" if torch.cuda.is_available() else "cpu"
device = torch.device(device_string)
model = model.to(device)

The way I define my training process:
# Define the loss function (MSE was chosen due to the comparsion of pixels
# between blurred and sharp images
criterion = nn.MSELoss()

# Define the optimizer and learning rate
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Learning rate schedule - If the loss value does not improve after 5 epochs
# back-to-back then the new learning rate will be:  previous_rate*0.5

#scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)

scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
        optimizer,
        mode='min',
        patience=5,
        factor=0.5,
        verbose=True
    )

def training(model, trainDataloader, epoch):
  """ Function to define the model training

    Args:
        model (Model object): The model that is going to be trained.
        trainDataloader (Dataloader object): Dataloader object of the trainset.
        epoch (Integer): Number of training epochs.

  """
  # Changing model into trainings mode
  model.train()
  # Supporting variable to display the loss for each epoch
  running_loss = 0.0
  running_psnr = 0.0
  for i, data in tqdm(enumerate(trainDataloader),
                      total=int(len(train_dataset)/trainDataloader.batch_size)):
    blur_image = data[0]
    sharp_image = data[1]

    # Transfer the blurred and sharp image instance to the device
    blur_image = blur_image.to(device)
    sharp_image = sharp_image.to(device)

    # Sets the gradient of tensors to zero
    optimizer.zero_grad()
    outputs = model(blur_image)
    loss = criterion(outputs, sharp_image)

    # Perform backpropagation
    loss.backward()
    # Update the weights
    optimizer.step()

    # Add the loss that was calculated during the trainigs run
    running_loss += loss.item()

    # calculate batch psnr (once every `batch_size` iterations)
    batch_psnr =  psnr(sharp_image, blur_image)
    running_psnr += batch_psnr

  # Display trainings loss
  trainings_loss = running_loss/len(trainDataloader.dataset)
  final_psnr = running_psnr/int(len(train_dataset)/trainDataloader.batch_size)
  final_ssim = ssim(sharp_image, blur_image, data_range=1, size_average=True)
  print(f"Trainings loss: {trainings_loss:.5f}")
  print(f"Train PSNR: {final_psnr:.5f}")
  print(f"Train SSIM: {final_ssim:.5f}")

  return trainings_loss, final_psnr, final_ssim

And here is my way to start the training:
train_loss  = []
val_loss = []
train_PSNR_score  = []
train_SSIM_score  = []
val_PSNR_score  = []
val_SSIM_score  = []

start = time.time()
for epoch in range(nb_epochs):
    print(f"Epoch {epoch+1}\n-------------------------------")
    train_epoch_loss = training(model, train_loader, nb_epochs)
    val_epoch_loss = validation(model, validation_loader, nb_epochs)
    train_loss.append(train_epoch_loss[0])
    val_loss.append(val_epoch_loss[0])

    train_PSNR_score.append(train_epoch_loss[1])
    train_SSIM_score.append(train_epoch_loss[2])

    val_PSNR_score.append(val_epoch_loss[1])
    val_SSIM_score.append(val_epoch_loss[2])

    scheduler.step(train_epoch_loss[0])
    scheduler.step(val_epoch_loss[0])
end = time.time()
print(f"Took {((end-start)/60):.3f} minutes to train")

But every time when I want to perform the training I receive the following error:
 0%|          | 0/249 [00:00<?, ?it/s]Epoch 1
-------------------------------
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py:528: UserWarning: Using a target size (torch.Size([16, 3, 128, 128])) that is different to the input size (torch.Size([16, 128])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-195-ff0214e227cd> in <module>()
      9 for epoch in range(nb_epochs):
     10     print(f"Epoch {epoch+1}\n-------------------------------")
---> 11     train_epoch_loss = training(model, train_loader, nb_epochs)
     12     val_epoch_loss = validation(model, validation_loader, nb_epochs)
     13     train_loss.append(train_epoch_loss[0])

<ipython-input-170-dfa2c212ad23> in training(model, trainDataloader, epoch)
     25     optimizer.zero_grad()
     26     outputs = model(blur_image)
---> 27     loss = criterion(outputs, sharp_image)
     28
     29     # Perform backpropagation

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    526
    527     def forward(self, input: Tensor, target: Tensor) -> Tensor:
--> 528         return F.mse_loss(input, target, reduction=self.reduction)
    529
    530

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in mse_loss(input, target, size_average, reduce, reduction)
   2926         reduction = _Reduction.legacy_get_string(size_average, reduce)
   2927
-> 2928     expanded_input, expanded_target = torch.broadcast_tensors(input, target)
   2929     return torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
   2930

/usr/local/lib/python3.7/dist-packages/torch/functional.py in broadcast_tensors(*tensors)
     72     if has_torch_function(tensors):
     73         return handle_torch_function(broadcast_tensors, tensors, *tensors)
---> 74     return _VF.broadcast_tensors(tensors)  # type: ignore
     75
     76

RuntimeError: The size of tensor a (16) must match the size of tensor b (128) at non-singleton dimension 2

model structure:
AlexNet(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
    (1): ReLU(inplace=True)
    (2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (4): ReLU(inplace=True)
    (5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): ReLU(inplace=True)
    (8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): ReLU(inplace=True)
    (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(6, 6))
  (classifier): Sequential(
    (0): Dropout(p=0.5, inplace=False)
    (1): Linear(in_features=9216, out_features=4096, bias=True)
    (2): ReLU(inplace=True)
    (3): Dropout(p=0.5, inplace=False)
    (4): Linear(in_features=4096, out_features=4096, bias=True)
    (5): ReLU(inplace=True)
    (6): Linear(in_features=4096, out_features=128, bias=True)
  )
)

I'm a newbie in terms of using Pytorch (and image deblurring in general) and so I rather confused about the meaning of the error message and how to fix it. I tried to change my parameters and nothing worked. Does anyone have any advice for me on how to solve this problem?
I would appreciate every input :)


title: Pytorch : Gradient of parameters remains 0

body:
During a homework where I have to implement Variational Inference for a Bayesian Logistic Regression, I have trouble making the optimization step because the gradients keeps being 0
Just to sum up what is Variational Inference:

We want to approximate the unknown posterior law using a Gaussian.
We do that by defining a Gaussian and by optimizing its mean and covariance matrix relative to the KullBack Divergence between itself and the posterior.
We can show that minimizing the Kullback divergence is equivalent to minimize de NELBO, which is computable
The parameters are the mean and covariance matrix (simplified into a diagonal matrix with a single value in the diagonal) of the gaussian we use to approximate the true posterior. So it's a vector and a scalar.
The loss is the NELBO

However, it seems that the gradient of my parameters keeps being 0 and thus never update. I do not know the cause of it...
Here is my Normal Distribution class that I use to define the gaussian approximation. The parameters to optimize are self.mean (vector) and self.logvar (scalar)
class NormalDiagonal(Distribution):
    @property
    def var(self):
        return self.logvar.exp()

    def extra_repr(self):
        return 'train=%s' % self.train

    def __init__(self, d, train=True):
        super(NormalDiagonal, self).__init__()
        self.train = train
        self.d = d
        self.mean = nn.Parameter(torch.zeros(d), requires_grad=train)
        self.logvar = nn.Parameter(torch.zeros(d), requires_grad=train) # v = 1 so log(v) = 0

    def sample(self, n=1):
        eps = MultivariateNormal(self.mean, torch.diag(self.var))
        samples = [eps.sample().tolist() for _ in range(n)] # shape: [n x self.d]
        return torch.tensor(samples)

Here is the class of my Logistic Regression model where I define self.posterior_w , the gaussian approximation , with the previous class.
class LogisticRegression(nn.Module):
    def __init__(self, input_dim):
        super(LogisticRegression, self).__init__()

        self.prior_w = NormalDiagonal(d=input_dim)
        self.posterior_w = NormalDiagonal(d=input_dim)

    @args_as_tensors(1)
    def predict_y(self, X, mc_samples=1):
        w_samples = self.posterior_w.sample(mc_samples)
        y_samples = logistic(torch.mm(X, w_samples.T)).mean(axis=1)
        return y_samples

Here is the class that compute the loss NELBO :
class VariationalObjective(nn.Module):
    def __init__(self, model, likelihood, N, mc_samples=1):
        super(VariationalObjective, self).__init__()
        self.N = N
        self.model = model
        self.likelihood = likelihood
        self.mc_samples = mc_samples

    def expected_loglikelihood(self, Xbatch, ybatch):
        ypred = model.predict_y(Xbatch, self.mc_samples)
        ybatch = torch.tensor(ybatch.reshape((ybatch.shape[0],))) # Makes it cleaner for later
        logliks = likelihood.logdensity(ybatch, ypred)
        return (1/self.mc_samples)*(self.N / len(Xbatch) ) * torch.sum(logliks)

    def kl(self):
        return kl_divergence(model.posterior_w, model.prior_w)

    def compute_objective(self, Xbatch, ybatch):
        return - self.expected_loglikelihood(Xbatch, ybatch) + self.kl()

Here is my optimization step :
data = genfromtxt('data/binaryclass2.csv', delimiter=',')
X = data[...,:-1]
y = data[...,-1].reshape(-1,1)
dataset = Dataset(X, y, minibatch_size=1000)

likelihood = Bernoulli()
model = LogisticRegression(X.shape[1])

nelbo = VariationalObjective(model, likelihood, X.shape[1], mc_samples= 10)

optim = torch.optim.SGD([model.posterior_w.mean, model.posterior_w.logvar], lr=0.001)
num_iterations = 50

for step in range(num_iterations):
    optim.zero_grad()
    Xbatch, ybatch = dataset.next_batch()
    loss = nelbo.compute_objective(Xbatch, ybatch)

    loss.backward()

    print(model.posterior_w.mean.grad) # PRINT 0
    print(model.posterior_w.logvar.grad) # PRINT 0
    optim.step()

I don't know what I'm doing wrong, maybe pytorch can't follow with all these nested instantiations ?
Thank you very much !


title: Attribute Error when running a Pytorch neural network in Spyder

body:
I tried to run a neural network to learn more about categorical embedding (the exaplanation of the neural network code is here https://yashuseth.blog/2018/07/22/pytorch-neural-network-for-tabular-data-with-categorical-embeddings/) but Spyder gives AttributeError after trying to run the loop in the end.

Traceback (most recent call last):
File "", line 1, in
File "C:\Workspace\Python_Runtime\Python\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Workspace\Python_Runtime\Python\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
AttributeError: Can't get attribute 'TabularDataset' on <module 'main' (built-in)>

My understanding is that this comes from Spyder having issue with multiprocessing functionality.
I have tried, as some answers suggested, to wrap everything that is not in a class or def in
if __name__ == '__main__':

but that did not seem to help, the error still comes up.
I also tried to import multiprocess package instead of multiprocessing but that did not help. I guess I would need to go and change the line in spawn.py file, but not sure how exactly.
The issue is that on my current PC I only have Spyder. I tried to run the same code on another dataset on my personal PC at home with Pycharm and it worked alright, with no errors at all.
Does anyone know how can I resolve the issue in Spyder?
The code for the neural network that I used is here:
from torch.utils.data import Dataset, DataLoader

class TabularDataset(Dataset):
  def __init__(self, data, cat_cols=None, output_col=None):
    """
    Characterizes a Dataset for PyTorch

    Parameters
    ----------

    data: pandas data frame
      The data frame object for the input data. It must
      contain all the continuous, categorical and the
      output columns to be used.

    cat_cols: List of strings
      The names of the categorical columns in the data.
      These columns will be passed through the embedding
      layers in the model. These columns must be
      label encoded beforehand.

    output_col: string
      The name of the output variable column in the data
      provided.
    """

    self.n = data.shape[0]

    if output_col:
      self.y = data[output_col].astype(np.float32).values.reshape(-1, 1)
    else:
      self.y =  np.zeros((self.n, 1))

    self.cat_cols = cat_cols if cat_cols else []
    self.cont_cols = [col for col in data.columns
                      if col not in self.cat_cols + [output_col]]

    if self.cont_cols:
      self.cont_X = data[self.cont_cols].astype(np.float32).values
    else:
      self.cont_X = np.zeros((self.n, 1))

    if self.cat_cols:
      self.cat_X = data[cat_cols].astype(np.int64).values
    else:
      self.cat_X =  np.zeros((self.n, 1))

  def __len__(self):
    """
    Denotes the total number of samples.
    """
    return self.n

  def __getitem__(self, idx):
    """
    Generates one sample of data.
    """
    return [self.y[idx], self.cont_X[idx], self.cat_X[idx]]

import torch
import torch.nn as nn
import torch.nn.functional as F

class FeedForwardNN(nn.Module):

  def __init__(self, emb_dims, no_of_cont, lin_layer_sizes,
               output_size, emb_dropout, lin_layer_dropouts):

    """
    Parameters
    ----------

    emb_dims: List of two element tuples
      This list will contain a two element tuple for each
      categorical feature. The first element of a tuple will
      denote the number of unique values of the categorical
      feature. The second element will denote the embedding
      dimension to be used for that feature.

    no_of_cont: Integer
      The number of continuous features in the data.

    lin_layer_sizes: List of integers.
      The size of each linear layer. The length will be equal
      to the total number
      of linear layers in the network.

    output_size: Integer
      The size of the final output.

    emb_dropout: Float
      The dropout to be used after the embedding layers.

    lin_layer_dropouts: List of floats
      The dropouts to be used after each linear layer.
    """

    super().__init__()

    # Embedding layers
    self.emb_layers = nn.ModuleList([nn.Embedding(x, y)
                                     for x, y in emb_dims])

    no_of_embs = sum([y for x, y in emb_dims])
    self.no_of_embs = no_of_embs
    self.no_of_cont = no_of_cont

    # Linear Layers
    first_lin_layer = nn.Linear(self.no_of_embs + self.no_of_cont,
                                lin_layer_sizes[0])

    self.lin_layers = nn.ModuleList([first_lin_layer] + [nn.Linear(lin_layer_sizes[i], lin_layer_sizes[i + 1]) for i in range(len(lin_layer_sizes) - 1)])

    for lin_layer in self.lin_layers:
      nn.init.kaiming_normal_(lin_layer.weight.data)

    # Output Layer
    self.output_layer = nn.Linear(lin_layer_sizes[-1],
                                  output_size)
    nn.init.kaiming_normal_(self.output_layer.weight.data)

    # Batch Norm Layers
    self.first_bn_layer = nn.BatchNorm1d(self.no_of_cont)
    self.bn_layers = nn.ModuleList([nn.BatchNorm1d(size)
                                    for size in lin_layer_sizes])

    # Dropout Layers
    self.emb_dropout_layer = nn.Dropout(emb_dropout)
    self.droput_layers = nn.ModuleList([nn.Dropout(size)
                                  for size in lin_layer_dropouts])

  def forward(self, cont_data, cat_data):

    if self.no_of_embs != 0:
      x = [emb_layer(cat_data[:, i])
           for i,emb_layer in enumerate(self.emb_layers)]
      x = torch.cat(x, 1)
      x = self.emb_dropout_layer(x)

    if self.no_of_cont != 0:
      normalized_cont_data = self.first_bn_layer(cont_data)

      if self.no_of_embs != 0:
        x = torch.cat([x, normalized_cont_data], 1)
      else:
        x = normalized_cont_data

    for lin_layer, dropout_layer, bn_layer in\
        zip(self.lin_layers, self.droput_layers, self.bn_layers):

      x = F.relu(lin_layer(x))
      x = bn_layer(x)
      x = dropout_layer(x)

    x = self.output_layer(x)

    return x

categorical_features = ["cat1", "cat2", "cat3"]
output_feature = ["output"]
data = data[output_feature + categorical_features + ["cont1", "cont2"]].copy().dropna()

from sklearn.preprocessing import LabelEncoder
label_encoders = {}
for cat_col in categorical_features:
    label_encoders[cat_col] = LabelEncoder()
    data[cat_col] = label_encoders[cat_col].fit_transform(data[cat_col])

dataset = TabularDataset(data=data, cat_cols=categorical_features,output_col=output_feature)

batchsize = 256
dataloader = DataLoader(dataset, batchsize, shuffle=True, num_workers=1)

cat_dims = [int(data[col].nunique()) for col in categorical_features]

emb_dims = [(x, min(50, (x + 1) // 2)) for x in cat_dims]

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = FeedForwardNN(emb_dims, no_of_cont=2, lin_layer_sizes=[50, 100],
                          output_size=1, emb_dropout=0.04,
                          lin_layer_dropouts=[0.001,0.01]).to(device)

import tqdm
no_of_epochs = 5
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.1)
for epoch in tqdm.tqdm(range(no_of_epochs)):
      for y, cont_x, cat_x in dataloader:

        cat_x = cat_x.to(device)
        cont_x = cont_x.to(device)
        y  = y.to(device)

        # Forward Pass
        preds = model(cont_x, cat_x)
        loss = criterion(preds, y)

        # Backward Pass and Optimization
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()


title: Can Theano / Pytorch / Tensorflow compute the following gradient automatically?

body:
I am trying to run a recurrent neural network where the state update function for each neuron is the following

z = g*y

given that

g = (x<x_max & x>x_max-e) | (x>-x_max & x<-x_max+e)

Note that all the variables here are just scalars.

The variable x is defined in a way that it will always  update continually so that g will always be a pulse as shown in the this picture. That is, g won't be 1 for a single update but it will be 1 for several consecutive updates.

Can any of these packages implement an automatic gradient computation given this transfer function?


title: Result of auto-encoder dimensions are incorrect

body:
Using below code I'm attempting to encode image from mnist into a lower dimension representation :

import warnings
warnings.filterwarnings('ignore')
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from matplotlib import pyplot as plt
from sklearn import metrics
import datetime
from sklearn.preprocessing import MultiLabelBinarizer
import seaborn as sns
sns.set_style("darkgrid")
from ast import literal_eval
import numpy as np
from sklearn.preprocessing import scale
import seaborn as sns
sns.set_style("darkgrid")
import torch
import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable

%matplotlib inline

low_dim_rep = 32
epochs = 2

cuda = torch.cuda.is_available() # True if cuda is available, False otherwise
FloatTensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor
print('Training on %s' % ('GPU' if cuda else 'CPU'))

# Loading the MNIST data set
transform = torchvision.transforms.Compose([torchvision.transforms.ToTensor(),
                torchvision.transforms.Normalize((0.1307,), (0.3081,))])
mnist = torchvision.datasets.MNIST(root='../data/', train=True, transform=transform, download=True)

# Loader to feed the data batch by batch during training.
batch = 100
data_loader = torch.utils.data.DataLoader(mnist, batch_size=batch, shuffle=True)

encoder = nn.Sequential(
                # Encoder
                nn.Linear(28 * 28, 64),
                nn.PReLU(64),
                nn.BatchNorm1d(64),

                # Low-dimensional representation
                nn.Linear(64, low_dim_rep),
                nn.PReLU(low_dim_rep),
                nn.BatchNorm1d(low_dim_rep))

decoder = nn.Sequential(
                # Decoder
                nn.Linear(low_dim_rep, 64),
                nn.PReLU(64),
                nn.BatchNorm1d(64),
                nn.Linear(64, 28 * 28))

autoencoder = nn.Sequential(encoder, decoder)

encoder = encoder.type(FloatTensor)
decoder = decoder.type(FloatTensor)
autoencoder = autoencoder.type(FloatTensor)

optimizer = torch.optim.Adam(params=autoencoder.parameters(), lr=0.00001)

data_size = int(mnist.train_labels.size()[0])

print('data_size' , data_size)
for i in range(epochs):
    for j, (images, _) in enumerate(data_loader):
        images = images.view(images.size(0), -1) # from (batch 1, 28, 28) to (batch, 28, 28)
        images = Variable(images).type(FloatTensor)

        autoencoder.zero_grad()
        reconstructions = autoencoder(images)
        loss = torch.dist(images, reconstructions)
        loss.backward()
        optimizer.step()
    print('Epoch %i/%i loss %.2f' % (i + 1, epochs, loss.data[0]))

print('Optimization finished.')

# Get the encoded images here
encoded_images = []
for j, (images, _) in enumerate(data_loader):
    images = images.view(images.size(0), -1)
    images = Variable(images).type(FloatTensor)

    encoded_images.append(encoder(images))

Upon completion of this code

len(encoded_images) is 600 when I expect the length to match the number of images in mnist : len(mnist) - 60'000.

How to encode the images to a lower dimension representation of 32 ( low_dim_rep = 32 ) ? I've defined the network parameters incorrectly ?


title: Model weights are not being updatesd, but loss is decreasing

body:
The following code is to train an MLP with images of size 64*64, while using the loss ||output - input||^2.

For some reason, my weights per epoch are not being updated as shown at the end.

class MLP(nn.Module):
    def __init__(self, size_list):
        super(MLP, self).__init__()
        layers = []
        self.size_list = size_list
        for i in range(len(size_list) - 2):
            layers.append(nn.Linear(size_list[i],size_list[i+1]))
            layers.append(nn.ReLU())
        layers.append(nn.Linear(size_list[-2], size_list[-1]))
        self.net = nn.Sequential(*layers)

    def forward(self, x):
        return self.net(x)

model_1 = MLP([4096, 64, 4096])

And for training each epoch:

def train_epoch(model, train_loader, criterion, optimizer):
    model.train()
    model.to(device)

running_loss = 0.0

    start_time = time.time()
    # train batch
    for batch_idx, (data) in enumerate(train_loader):
        optimizer.zero_grad()

        data = data.to(device)

        outputs = model(data)
        loss = criterion(outputs, data)
        running_loss += loss.item()

        loss.backward()
        optimizer.step()

    end_time = time.time()

    weight_ll = model.net[0].weight
    running_loss /= len(train_loader)

    print('Training Loss: ', running_loss, 'Time: ',end_time - start_time, 's')
    return running_loss, outputs, weight_ll

for training the data:

n_epochs = 20
Train_loss = []
weights=[]

criterion = nn.MSELoss()

optimizer = optim.SGD(model_1.parameters(), lr = 0.1)

for i in range(n_epochs):
    train_loss, output, weights_ll = train_epoch(model_1, trainloader, criterion, optimizer)
    Train_loss.append(train_loss)
    weights.append(weights_ll)
    print('='*20)

Now, when I print the weights of the first fully connected layer per epoch they aren't being updated.

print(weights[0][0])
print(weights[19][0])

The output for the above is (showing the weight in epoch 0 and in epoch 19):

tensor([ 0.0086,  0.0069, -0.0048,  ..., -0.0082, -0.0115, -0.0133],
       grad_fn=<SelectBackward>)
tensor([ 0.0086,  0.0069, -0.0048,  ..., -0.0082, -0.0115, -0.0133],
       grad_fn=<SelectBackward>)

What may be going wrong? Looking at my loss, it's decreasing at a steady rate but there is no change in the weights.


title: Pytorch equivalent of Google Seedbank

body: Does Pytorch have an equivalent of Google Seedbank ? Everything in Seedbank is (unsurprisingly) Tensorflow based, and I want to learn Pytorch.


title: Fastest way to convert video to frames wihout saving the frames locally

body:
I want to apply batch prediction on Videos using Pytorch.

The Problem is I am having many videos to process and don't want to save additional frames.
Is there a way I can save 'n frames' in 1FPS and apply deep learning at the same time?

In other words, I am having VIDEODATASET is there any VIDEODATALOADER?


title: how to load the data and how to do data augmentation using pytorch

body:
I am newbie to Pytorch, I am doing Image classification problem , But I didn't understand how to load the image from load directory, please help me how to load the data image data and how to augmentation.

here my data looks like this:

train=pd.read_csv('dataset/train.csv')
test=pd.read_csv('dataset/test.csv')
train.head()
Image   Class
0   image7042.jpg   Food
1   image3327.jpg   misc
2   image10335.jpg  Attire
3   image8019.jpg   Food
4   image2128.jpg   Attire

here my Images folder:

file_path='dataset/Train Images'


title: BERT with 256 hidden embeddings

body:
I am trying to use BERT to get word embeddings from different data sets for my NLP task. I have used the 'bert_base_uncased' with 768 word embeddings but it runs out of memory. The versions with 256 word embeddings has been released? Or is there any way that I can compress the 768 hidden embeddins?
Thank you!


title: How to make a 3D tensor into one_hot encoding

body: for example ,labels is a groundtruth, labels size is [4,224,224] where 4 means batchsize ,224 means h and w,  labels dtype is torch.int64, in my training code ,label's pixel from 0 to 255 .I use my network to Semantic segmentation


title: How to feed ranked multiple labels for a single image to a supervised training pipeline

body:
In supervised learning for image classification, we feed the training images with its labels. Is it possible for each image, can I provide a series of label ranked in the order of relatedness instead of just one label?

I am using PyTorch for this supervised training. Right now, under each label, I have my training images for that label. Can anyone plz help me saying how can I implement ranked multi-label image training in PyTorch?


title: Python : convert a float encoded as a byte string (from PyTorch) to an int

body:
I have converted an output from PyTorch using .detach().numpy() which produces this kind of data :

b'0.06722715'

which is a byte type according to type() from Python. How can I convert this to an integer ?


title: Issue training pytorch model on gpu

body:
I am trying to implement the discriminator of a basic MNIST GAN in PyTorch. When I run training on the CPU, it works without any issue giving the desired output. However, when I run it on the GPU it shows a runtime error. I am pasting the code for my model as well as my training below along with the modifications I made to try and run training on the GPU.

dev = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

def preprocess(x):
    return x.view(-1,1,28,28).to(dev)

discriminator = nn.Sequential(
    Lambda(preprocess),
    nn.Conv2d(1,64,3,stride=2,padding=1),
    nn.LeakyReLU(negative_slope=0.2),
    nn.Dropout(0.4),
    nn.Conv2d(64,64,3,stride=2,padding=1),
    nn.LeakyReLU(negative_slope=0.2),
    nn.Dropout(0.4),
    Lambda(lambda x:x.view(x.size(0),-1)),
    nn.Linear(3136,1),
    nn.Sigmoid()
)
loss = nn.BCELoss()
opt = optim.Adam(discriminator.parameters(),lr = 0.002)

discriminator.to(dev)

def train_discriminator(model, dataset,opt, n_iter=100, n_batch=256):
    half_batch = int(n_batch / 2)
    for i in range(n_iter):
        X_real, y_real = generate_real_samples(dataset, half_batch)
        error = loss(model(X_real),y_real)
        error.backward()
        X_fake, y_fake = generate_fake_samples(half_batch)
        error = loss(model(X_fake),y_fake)
        error.backward()
        opt.step()

Now on running train_discriminator(discriminator,dataset,opt)
I get the following error which I cannot make sense of.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-ee20eb2a8e55> in <module>
----> 1 train_discriminator(discriminator,dataset,opt)

<ipython-input-13-9e6f9b4874c8> in train_discriminator(model, dataset, opt, n_iter, n_batch)
      3     for i in range(n_iter):
      4         X_real, y_real = generate_real_samples(dataset, half_batch)
----> 5         error = loss(model(X_real),y_real)
      6         error.backward()
      7         X_fake, y_fake = generate_fake_samples(half_batch)

~/environments/workspace/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/environments/workspace/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    496
    497     def forward(self, input, target):
--> 498         return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
    499
    500

~/environments/workspace/lib/python3.7/site-packages/torch/nn/functional.py in binary_cross_entropy(input, target, weight, size_average, reduce, reduction)
   2075
   2076     return torch._C._nn.binary_cross_entropy(
-> 2077         input, target, weight, reduction_enum)
   2078
   2079

RuntimeError: Expected object of device type cuda but got device type cpu for argument #2 'target' in call to _thnn_binary_cross_entropy_forward

Would really appreciate it if anyone could suggest any changes that are needed to be made that will solve this issue.


title: Why I am getting this error "OSError: [WinError 126] The specified module could not be found", while installing pytorch in anaconda prompt?

body:
I am trying to install torch module in anacondo prompt but i'm getting this error;

import torch Traceback (most recent call last):
File "", line 1, in File "C:\Users\Shivansh Computers\anaconda3\lib\site-packages\torch__init__.py", line 81, in ctypes.CDLL(dll) File "C:\Users\Shivansh Computers\anaconda3\lib\ctypes__init__.py", line 364, in init self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found


title: Can't use with import in Pytorch

body:
When I import torch, there's a problem like this:

C: lUsers / ruiha Desktopl flappy_DQL>python flappy.py
Traceback (most recent call last):
File "flappy.py", line 2, in <module>
import torch
File "C: IUsers I rui ha /AppDat a / Local Programs I Python/ Python37 libIsite-packagesI torchl__init__.py", line 81, in <mod
ctypes.CDLL(dll)
File "C: LUsersI ruiha lAppDat a l Local | Programs I Python/ Python37 liblctypesl__init_.py", line 364, in _init_
self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found


title: What's the usage for convolutional layer that output is the same as the input applied with MaxPool

body:
what's the idea behind when using the following convolutional layers?

especially for nn.Conv2d(16, 16, 3, padding = 1)

self.conv1 = nn.Conv2d(3, 16, 3, padding = 1 )
self.conv2 = nn.Conv2d(16, 16, 3, padding = 1)
self.conv3 = nn.Conv2d(16, 32, 3, padding = 1)
self.pool = nn.MaxPool2d(2, 2)

 x = F.relu(self.conv1(x))
 x = self.pool(F.relu(self.conv2(x)))
 x = F.relu(self.conv3(x))

I thought Conv2d always uses a bigger size like
from (16,32) to (32,64) for example.

Is nn.Conv2d(16, 16, 3, padding = 1) merely for reducing the size?