title: Pytorch linear regression loss increase

body:
I tried to implement a simple demo that gets a polynomial regression, but the linear model's loss fails to decrease.
I am confused about where I went wrong.
If I trained the model one sample(batch size = 1) each time, it works fine. but when I feed the model with many samples a time, the loss increase and get inf.
import numpy as np

import torch
import math
from matplotlib import pyplot as plt

def rand_series(size):
    x = np.linspace(-100, 100, size)
    np.random.shuffle(x)
    base_y = 20 * np.sin(2 * math.pi / 200 * x)
    y = base_y + 10 * np.random.rand(size)
    return x, y

def rescale_vec(vector):
    vec_as_tensor = torch.tensor(vector, dtype=torch.float32)
    max_in_vec = torch.max(vec_as_tensor)
    min_in_vec = torch.min(vec_as_tensor)
    if max_in_vec - min_in_vec == 0:
        return torch.ones(vec_as_tensor.size(), dtype=torch.float32)
    else:
        return (vec_as_tensor - min_in_vec) / (max_in_vec - min_in_vec)

def rescale(vectors):
    if len(vectors.shape) == 1:
        return rescale_vec(vectors)
    nor_vecs = torch.empty(vectors.shape)
    for i in range(vectors.shape[0]):
        nor_vecs[i] = rescale_vec(vectors[i])
    return nor_vecs

class LinearRegression (torch.nn.Module):
    def __init__ (self, power=4):
        super().__init__()
        self.layer = torch.nn.Linear(power, 1)

    def forward(self, x):
        return self.layer(x)

def regression(x_, y_, learning_rate):
    x = torch.t(torch.tensor(x_, dtype=torch.float32))
    y = torch.tensor(y_, dtype=torch.float32)
    dim_size = x.size()[1]
    print(dim_size, x.size())

    model = LinearRegression(dim_size)
    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
    loss_func = torch.nn.MSELoss(reduction='sum')
    batch_size = 400
    for round in range(50):
        sample_indices = torch.randint(0, len(x), (batch_size, ))
        x_samples = torch.index_select(x, 0, sample_indices)
        y_samples = torch.index_select(y, 0, sample_indices)
        optimizer.zero_grad()

        y_hat = model(x_samples.view(-1, dim_size))
        loss = loss_func(y_hat, y_samples)
        print(loss.item())
        loss.backward()

        optimizer.step()

    return model

x_one, y = rand_series(1000)
b = np.ones(len(x_one))
x = np.array([b, x_one, x_one ** 2, x_one ** 3, x_one ** 4, x_one ** 5])

model = regression(rescale(x), torch.tensor(y, dtype=torch.float32), 0.002)

nor_x = rescale(x)

y_hat = model(torch.t(torch.tensor(x, dtype=torch.float32)))
plt.scatter(x_one, y)
plt.scatter(x_one, y_hat.data, c='red')
plt.show()

the loss:
4.7375866968775066e+19
1.6979300048622735e+26
6.0214270068868396e+32
inf
inf
inf


title: How to build a reinforcement learning with certain constraints on the actions

body:
I'm trying to build a reinforcement learning algorithm for the following scheme:
There is a 2D N X N grid where each vertex of the grid can store a single particle. The particles are initially randomly distributed in the 2D grid. I have a 'picker' which grabs the particles and can move them to certain place.The goal of the algorithm is that given a target sub-area of the grid I need it to be completely filled with particles with fewest number of moves. There is a constraint for the movement of the picker, which is that the picker can grab multiple particles but it can only do so and move in one dimension at a time (one row or one column).
For example:
For a 3 x 3 grid with initially filling [[1,0,1,1],[0,1,0,0],[0,0,0,1]] where 1 means a filled vertex and 0 is empty vertex
Then the following senarios are allowed:
1.I can pick a row and move particles into any configuration in 1D. For example, the first row can be moved as [1,1,1,0] or [0,1,1,1]...
2. I can pick a column and perform same operation as step 1 except this operation is confined in column(1D) only
3.I can pick a row and shift the row column-wise. For example I can pick the entire 2nd row and shift it down to the third row. The final configuration would look like
[[1,0,1,1],[0,0,0,0],[0,1,0,1]]
4. I can also pick a column and shift the column row-wise. For example I can pick the entire 1st column and shift it right to the second column. The final configuration would look like
[[0,1,1,1],[0,1,0,0],[0,0,0,1]]
I think for 1 particle this will just be the simple version of 'snake find the food' game. For N particles, I have couple thoughts:

at each move, I assign 4 possible actions to each particle so there will be 4N action in total. And I made a giant penalty for the moves that violate the allowed moves. But the biggest drawback would be that even if the network learns this rule after a long train it's still possible that it might run into this constraint during the test.
somehow I hardcode this constraint into actions but not sure how that can be done.

I think the first option is not what I want. For the second option, the question is how would I add the constraint to the action in this senario?


title: Gradients vanishing despite using Kaiming initialization

body:
I was implementing a conv block in pytorch with activation function(prelu). I used Kaiming initilization to initialize all my weights and set all the bias to zero. However as I tested these blocks (by stacking 100 such conv and activation blocks on top of each other), I noticed that the output I am getting values of the order of  10^(-10). Is this normal, considering I am stacking upto 100 layers. Adding a small bias to each layer fixes the problem. But in Kaiming initialization the biases are supposed to be zero.
Here is the conv block code
from collections import Iterable

def convBlock(
    input_channels, output_channels, kernel_size=3, padding=None, activation="prelu"
):
    """
    Initializes a conv block using Kaiming Initialization
    """
    padding_par = 0
    if padding == "same":
        padding_par = same_padding(kernel_size)
    conv = nn.Conv2d(input_channels, output_channels, kernel_size, padding=padding_par)
    relu_negative_slope = 0.25
    act = None
    if activation == "prelu" or activation == "leaky_relu":
        nn.init.kaiming_normal_(conv.weight, a=relu_negative_slope, mode="fan_in")
        if activation == "prelu":
            act = nn.PReLU(init=relu_negative_slope)
        else:
            act = nn.LeakyReLU(negative_slope=relu_negative_slope)
    if activation == "relu":
        nn.init.kaiming_normal_(conv.weight, nonlinearity="relu")
        act = nn.ReLU()
    nn.init.constant_(conv.bias.data, 0)
    block = nn.Sequential(conv, act)
    return block

def flatten(lis):
    for item in lis:
        if isinstance(item, Iterable) and not isinstance(item, str):
            for x in flatten(item):
                yield x
        else:
            yield item

def Sequential(args):
    flattened_args = list(flatten(args))
    return nn.Sequential(*flattened_args)

This is the test Code
ls=[]
for i in range(100):
    ls.append(convBlock(3,3,3,"same"))
model=Sequential(ls)

test=np.ones((1,3,5,5))
model(torch.Tensor(test))

And the output I am getting is
tensor([[[[-1.7771e-10, -3.5088e-10,  5.9369e-09,  4.2668e-09,  9.8803e-10],
          [ 1.8657e-09, -4.0271e-10,  3.1189e-09,  1.5117e-09,  6.6546e-09],
          [ 2.4237e-09, -6.2249e-10, -5.7327e-10,  4.2867e-09,  6.0034e-09],
          [-1.8757e-10,  5.5446e-09,  1.7641e-09,  5.7018e-09,  6.4347e-09],
          [ 1.2352e-09, -3.4732e-10,  4.1553e-10, -1.2996e-09,  3.8971e-09]],

         [[ 2.6607e-09,  1.7756e-09, -1.0923e-09, -1.4272e-09, -1.1840e-09],
          [ 2.0668e-10, -1.8130e-09, -2.3864e-09, -1.7061e-09, -1.7147e-10],
          [-6.7161e-10, -1.3440e-09, -6.3196e-10, -8.7677e-10, -1.4851e-09],
          [ 3.1475e-09, -1.6574e-09, -3.4180e-09, -3.5224e-09, -2.6642e-09],
          [-1.9703e-09, -3.2277e-09, -2.4733e-09, -2.3707e-09, -8.7598e-10]],

         [[ 3.5573e-09,  7.8113e-09,  6.8232e-09,  1.2285e-09, -9.3973e-10],
          [ 6.6368e-09,  8.2877e-09,  9.2108e-10,  9.7531e-10,  7.0011e-10],
          [ 6.6954e-09,  9.1019e-09,  1.5128e-08,  3.3151e-09,  2.1899e-10],
          [ 1.2152e-08,  7.7002e-09,  1.6406e-08,  1.4948e-08, -6.0882e-10],
          [ 6.9930e-09,  7.3222e-09, -7.4308e-10,  5.2505e-09,  3.4365e-09]]]],
       grad_fn=<PreluBackward>)


I get this weird error when I start my script I really don't know why this error is caused or by what it's caused. It would be very very nice if some of you guy's could help me out with this error:)
Error message:
  File "c:/Users/supre/Documents/Python Programs/neuralnet/optimize_graph.py", line 11, in trace
    traced = torch.trace(model.forward)(xample_inputs=(x, hidden))
TypeError: trace(): argument 'input' (position 1) must be Tensor, not method

This is my code:
import argparse
import torch
from model import SpeechRecognition
from collections import OrderedDict

def trace(model):
    model.eval()
    x = torch.rand(1, 81, 300)
    hidden = model._init_hidden(1)
    traced = torch.trace(model.forward)(xample_inputs=(x, hidden))
    return traced

def main(args):
    print("loading model from", args.model_checkpoint)
    checkpoint = torch.load(args.model_checkpoint, map_location=torch.device('cpu'))
    h_params = SpeechRecognition.hyper_parameters
    model = SpeechRecognition(**h_params)
    model_state_dict = checkpoint['state_dict']
    new_state_dict = OrderedDict()
    for k, v in model_state_dict.items():
        name = k.replace("model.", "") # remove `model.`
        new_state_dict[name] = v

    model.load_state_dict(new_state_dict)

    print("tracing model...")
    traced_model = trace(model)
    #print(type(traced_model))
    print("saving to", args.save_path)
    traced_model.save(args.save_path)
    print("Done!")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="testing the wakeword engine")
    parser.add_argument('--model_checkpoint', type=str, default='NameOfYourCheckpoint', required=False,
                        help='Checkpoint of model to optimize')
    parser.add_argument('--save_path', type=str, default='path/where/you/want/to/save/the/optimized/model', required=False,
                        help='path to save optmized model')

    args = parser.parse_args()
    main(args)

Thank's for every help in advance


title: The differences of BatchNorm layer backpropagation at mode of model.train() and model.eval() in Pytorch?

body:
I test the gradient of BatchNorm layer for two mode: model.train() and model.eval().
I bulid a simple CNN network NetWork and input the same input X to the network at model.train() mode and model.eval() mode.
I know the differences of model.train() and model.eval() of BatchNorm layer. I have replaced the mean and the var of Batchnorm layer in mode of model.eval() as the values in mode of model.train(). Therefore both outputs and parameters of two mode are the same.
output of two mode
parameters of two mode
However, when I calculate the gradients of each parameters, I found the gradients of the layer before BatchNorm layer is different, although their parameters and the loss is same.
different gradients of the layer before BatchNorm
I think it's because the difference of BatchNorm layer backpropagation at model.train() and model.eval(), but I don't understrand the detail of it. Does anyone know? Thank you so much.


title: Understanding Loss Functions

body: I was following this pytorch tutorial on how to setup a neural network, but I don't understand the loss function, loss = (y_pred - y).pow(2).sum().item(). Why is this being used rather than the derivative of the function used to calculate the predicted y value? I also don't understand what that function returns.


title: Transformer model to generate XML output

body:
I have a strangely specific problem but hopefully someone can at least give an idea for where to go.
I am attempting to generate an XML file from a PNG using a transformer (encoder+decoder) model. In certain video games you can generate in game images using layers of primitive shapes. For those that are having a hard time imagining it you can follow this link and when the page loads click "Try Sample" at the top:
https://malulleybovo.github.io/SymbolArtEditorOnline/.
Additionally you can see each layer it is comprised of by clicking the top left icon (each layer is an entry in the list).
My model uses a vision transformer encoder with a fairly standard decoder. I say fairly standard because I am decoding each line to an XML line (file type is called SAML) of the form:
<layer name="Symbol 0" visible="true" type="346" color="#ebebeb" alpha="1" ltx="-83" lty="-19" lbx="-112" lby="-37" rtx="-50" rty="-21" rbx="-79" rby="-39"/>
where type is the primitive shape used for that layer, color is a hex representation of the color, alpha is transparency and the rest is describing how the shape is transformed. In order to not just one hot encode the layer type I create embeddings for each unique layer shape which is trained along with the rest of the model (in an sequential, autoregressive manner since this is a sequence of XML lines). My output of the model is a tensor of size (batch, layer_count, embedding_size+12), where 12 is the number of parameters that aren't the layer type. This creates a strange dilemma and where my question comes from.
Because each line is a combination of embedding values and values directly from the XML line I have to use 2 different loss functions (cross entropy and L1/L2 loss for the embeddings and remaining 12 values respectively). In order to facilitate this I have to do the following manipulation to tensors to calculate loss (code sort of abridged for brevity):
x = decoder(input)
emb_out, loc_out = torch.split(x, [unique_layer_cnt, 12], dim=-1)
batch_emb_loss = nn.functional.cross_entropy(emb_out.transpose(1,2), emb_label)
batch_loc_loss = nn.functional.l1_loss(loc_out, loc_label)
print(batch_emb_loss + batch_loc_loss)
batch_emb_loss.backward(retain_graph=True)
batch_loc_loss.backward()

I am doing experiments but the loss is ridiculously large (nearly 10000 but that is also summed over 225 lines) albeit it is going down so it is converging to something. Honestly the whole thing feels like a hack and doesn't sit well with me but I can't seem to think of any other way to attack this problem.
Is this a problem and if it is then how else can I generate a loss for each XML line? I avoided using L1/L2 loss for the entire output but it would at least make things slightly more uniform. Am I backpropagating the error correctly if this is okay to do? I left out the line by line iteration for autoregressive, generative training and the optimizer parts but that is somewhat standard (I used an approach similar to the Pytorch documentation). Is this even feasible? I saw it as a pseudo-NLP problem so I thought I could maybe do it.
Any input would be appreciated and if this isn't the place or the format to appropriately address this question I am sorry, just let me know where to move it/delete it. If you need more information let me know.
Thanks in advance!


title: PyTorch input/output data sizes

body:
I'm trying out PyTorch for the first time, and running into a couple problems. I've shared some of my code below and have two questions.
Q1: What should my output size be? Each input should lead to one output, equal to one of 6 possible output labels (1-6). Should output size be 1 or 6?
Q2: Something is wrong with my accuracy calculation, which I think is tied into Q1. predicted ends up being of size 4 x 4411, where 4 is my batch size (and so I think that's correct) but 4411 is my feature/input size, which is almost certainly wrong. I would expect it to be 6 (number of possible output labels). labels is 4x6, which I think is correct. If I change the dim I'm taking the max over from 1 to 2, then it gets the correct size of 4x6, but logically it makes no sense as it's returning the max index across all feature values for an input.
I think I'm missing something crucial about what pytorch is doing with my data. I feel so close.... Any ideas on how I can fix this? Thanks!
    def __init__(self, input_size, output_size, hidden_dim, n_layers):
        super(Net, self).__init__()
        self.input_size  = input_size
        self.output_size = output_size
        self.hidden_dim  = hidden_dim
        self.n_layers    = n_layers

        self.rnn = torch.nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)
        # Q1: What should my output_size be?
        self.fc = torch.nn.Linear(hidden_dim, output_size)

    def forward(self, x):
        batch_size = x.size(0)
        hidden = self.init_hidden(batch_size)
        out, hidden = self.rnn(x, hidden)
        out = self.fc(out)
        return out, hidden

    def init_hidden(self, batch_size):
        return torch.zeros(self.n_layers, batch_size, self.hidden_dim)
if __name__ == '__main__':
   # ... code removed that just creates the Dataloaders, and initialises some size variables

    net = Net(input_size=dataset.input_size, output_size=6, hidden_dim=24, n_layers=1)
    net.to(device)
    criterion = torch.nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(net.parameters(), lr=lr)

    for epoch in range(1, epochs+1):
        total = 0
        correct = 0
        for inputs, labels in dataloaders['train']:
            optimizer.zero_grad()
            inputs, labels = inputs.to(device), labels.to(device)
            output, hidden = net(inputs)
            loss = criterion(output, labels) # error
            loss.backward()
            optimizer.step()

            # Q2: Something is wrong here
            _, predicted = torch.max(output.data, 1)
            total += inputs.size(0)
            correct += (predicted == labels).sum().item()
            print(predicted == labels)
        accuracy = 100 * correct / total```


title: how to exclude particular dependency package version from setup file in python

body:
I've a small package that does not work on torch v1.8.0 but it works fine on the new v1.8.1 version and other older versions v1.7.1 so want to exclude the v1.8.0 version.

I could just set
install_requires=[
    "torch>=1.8.1",
      ...

but the torch package size is huge and also want mypackage to work on the older versions of the torch.
I've tried
install_requires=[
    "torch>=1.8.1,!=1.8.0,<=1.7.1",
       ...

but when installing the package with pip install mypackage getting the following error:
ERROR: Could not find a version that satisfies the requirement torch!=1.8.0,<=1.7.1,>=1.8.1 (from mypackage) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.4.1, 0.4.1.post2, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1)
ERROR: No matching distribution found for torch!=1.8.0,<=1.7.1,>=1.8.1 (from mypackage)

how can exclude the v1.8.0 version? thank you.


title: ResolvePackageNotFound with old pytorch

body:
I am trying to run some code from GitHub from a paper but stuck on the very first step. I am using ssh to access a DM server, where I would like to run this code. The instruction was as simple as:
The codebase is built on PyTorch 0.3.1 and tested on Ubuntu 16.04 environment (Python3.6, CUDA9.0, cuDNN7.5).
For installing, follow these instructions
conda env create -f environment.yml
conda activate pytorch0.3.1

When I ran the first command, I got the following message:
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:
  - pytorch==0.3.1=py36_cuda9.0.176_cudnn7.0.5_2

I currently have python 3.8.3. Would that be the problem?


title: where I can deploy ml algorithm without time out?

body:
I used Heroku to deploy ml algorithm with Pytorch but when I sent request to app to train model with 100 epochs then I got this
error

at=error code=H12 desc="Request Timeout" method=POST path="/train"

I searched about request time out and I find that the limit to request time
from Heroku is 30 sec
What's the solution of time out problem and is there an platform for training model without time out?


title: Variational Autoencoder (VAE) returns consistent output

body:
I'm working on the signal compression and reconstruction with VAE. I've trained 1600 fragments but the values of 1600 reconstructed signals are very similar. Moreover, results from same batch are almost consistent. As using the VAE, loss function of the model contains binary cross entropy (BCE) and the output of the train model should be located between 0 to 1 (The input data also normalized to 0~1).
VAE model(LSTM) :
class LSTM_VAE(nn.Module):
def __init__(self,
             input_size=3000,
             hidden=[1024, 512, 256, 128, 64],
             latent_size=64,
             num_layers=8,
             bidirectional=True):
    super().__init__()

    self.input_size = input_size
    self.hidden = hidden
    self.latent_size = latent_size
    self.num_layers = num_layers
    self.bidirectional = bidirectional

    self.actv = nn.LeakyReLU()

    self.encode = nn.LSTM(input_size=self.input_size,
                          hidden_size=self.hidden[0],
                          num_layers=self.num_layers,
                          batch_first=True,
                          bidirectional=True)
    self.bn_encode = nn.BatchNorm1d(1)

    self.decode = nn.LSTM(input_size=self.latent_size,
                          hidden_size=self.hidden[2],
                          num_layers=self.num_layers,
                          batch_first=True,
                          bidirectional=True)
    self.bn_decode = nn.BatchNorm1d(1)

    self.fc1 = nn.Linear(self.hidden[0]*2, self.hidden[1])
    self.fc2 = nn.Linear(self.hidden[1], self.hidden[2])
    self.fc31 = nn.Linear(self.hidden[2], self.latent_size)
    self.fc32 = nn.Linear(self.hidden[2], self.latent_size)
    self.bn1 = nn.BatchNorm1d(1)
    self.bn2 = nn.BatchNorm1d(1)
    self.bn3 = nn.BatchNorm1d(1)

    self.fc4 = nn.Linear(self.hidden[2]*2, self.hidden[1])
    self.fc5 = nn.Linear(self.hidden[1], self.hidden[0])
    self.fc6 = nn.Linear(self.hidden[0], self.input_size)
    self.bn4 = nn.BatchNorm1d(1)
    self.bn5 = nn.BatchNorm1d(1)
    self.bn6 = nn.BatchNorm1d(1)

def encoder(self, x):
    x = torch.unsqueeze(x, 1)
    x, _ = self.encode(x)
    x = self.actv(x)
    x = self.fc1(x)
    x = self.actv(x)
    x = self.fc2(x)
    x = self.actv(x)

    mu = self.fc31(x)
    log_var = self.fc32(x)

    return mu, log_var

def decoder(self, z):
    z, _ = self.decode(z)
    z = self.bn_decode(z)
    z = self.actv(z)
    z = self.fc4(z)
    z = self.bn4(z)
    z = self.fc5(z)
    z = self.bn5(z)
    z = self.fc6(z)
    z = self.bn6(z)
    z = torch.sigmoid(z)

    return torch.squeeze(z)

def sampling(self, mu, log_var):
    std = torch.exp(0.5 * log_var)
    eps = torch.randn_like(std)

    return mu + eps * std

def forward(self, x):
    mu, log_var = self.encoder(x.view(-1, self.input_size))
    z = self.sampling(mu, log_var)
    z = self.decoder(z)

    return z, mu, log_var

Loss function and Train code :
def lossF(recon_x, x, mu, logvar, input_size):
BCE = F.binary_cross_entropy(recon_x, x.view(-1, input_size), reduction='sum')
KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())

return BCE + KLD

optim = torch.optim.Adam(model.parameters(), lr=opt.lr)

for epoch in range(opt.epoch):
    for batch_idx, data in enumerate(train_set):
        data = data.to(device)
        optim.zero_grad()
        recon_x, mu, logvar = model(data)
        loss = lossF(recon_x, data, mu, logvar, opt.input_size)
        loss.backward()
        train_loss += loss.item()
        optim.step()

I built the code by refer the example codes of others and only changed very few parameters. I rebuilt the code, change the dataset, update parameters but nothing worked. If you have any suggestion to solve this problem, PLEASE let me know.


title: [python]Colab crashes for unknown reason while training gan(pytorch)

body:
I tried to train gan on some monkey pics but it crashes colab for unknown reason if try to train it.
I am using 1370 128*128 monkey images.
I have no idea where the issue might be, please respond
btw the runtime is gpu, so the problem doesn't linked to that
from torch import optim
import torchvision
from torchvision import transforms
import torch, torch.nn as nn

batch_size = 4

generic_transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.ToPILImage(),
    transforms.Resize((128,128)),
    transforms.ToTensor(),
    transforms.Normalize((0., 0., 0.), (6, 6, 6)),
    transforms.Grayscale(),
])

trainset=torchvision.datasets.ImageFolder(root='drive/My Drive/monkeys', transform=generic_transform)

trainloader = torch.utils.data.DataLoader(trainset,batch_size=batch_size, shuffle=True)

def _init_weights(m):
    if isinstance(m, nn.Conv2d):
        nn.init.normal_(m.weight, 0.0, 0.02)

def gen_noise(noise_shape, n_samples, device='cuda:0'):
    return torch.randn(noise_shape, n_samples).to(device)

class Discriminator(nn.Module):
 #convolutional discriminator
  def __init__(self) -> None:
    super(Discriminator, self).__init__()

    self.hidden_dim = 64

    self.relu = nn.ReLU(inplace=False)
    self.sigmoid = nn.Sigmoid()

    self.conv_1 = nn.Conv2d(in_channels=1, out_channels=32, kernel_size=32, stride = 2)
    self.maxPooling_1 = nn.MaxPool2d(kernel_size=3)
    self.conv_2 = nn.Conv2d(in_channels=32, out_channels=16, kernel_size=8, stride = 2)
    self.maxPooling_2 = nn.MaxPool2d(kernel_size=2)
    self.linear_layer = nn.Linear(in_features=self.hidden_dim, out_features=1)

  def forward(self, x) -> float:
    self.x = x
    self.x = self.relu(self.conv_1(self.x))
    self.x = self.maxPooling_1(self.x)
    self.x = self.relu(self.conv_2(self.x))
    self.x = self.maxPooling_2(self.x)
    print(self.x.shape)
    self.x = self.x.view(self.x.shape[0],
                         self.x.shape[1]*self.x.shape[2]*self.x.shape[3])
    self.x = self.sigmoid(self.linear_layer(self.x))

    return self.x

 class Generator(nn.Module):
    #fully connected generator
    def __init__(self, hidden_dim, output_dim, z_dim) -> None:
    super(Generator, self).__init__()

    self.relu = nn.ReLU(inplace=False)
    self.hidden_dim = hidden_dim
    self.output_dim = output_dim
    self.z_dim = z_dim

    self.linear_layer_1 = nn.Linear(in_features=self.z_dim, out_features=self.hidden_dim)
    self.linear_layer_2 = nn.Linear(in_features=self.hidden_dim, out_features=self.hidden_dim*2)
    self.linear_layer_3 = nn.Linear(in_features=self.hidden_dim*2, out_features=self.output_dim)

    def forward(self, x) -> torch.tensor:
      self.x = x
      self.x = self.relu(self.linear_layer_1(self.x))
      self.x = self.relu(self.linear_layer_2(self.x))
      self.x = self.relu(self.linear_layer_3(self.x))
      return self.x

class GAN():
  def __init__(self, hidden_dim, output_dim, z_dim, criterion, device="cuda:0") -> None:
    if device == "cuda:0":
      assert torch.cuda.is_available(), "apply gpu"

    self.hidden_dim = hidden_dim
    self.output_dim = output_dim
    self.device = device
    self.criterion = criterion
    self.z_dim = torch.tensor(z_dim).long()

    self.discriminator = Discriminator().to(self.device)
    self.d_opt = optim.Adam(self.discriminator.parameters(), lr=0.0001)

    self.generator = Generator(hidden_dim=self.hidden_dim, output_dim=self.output_dim, z_dim=self.z_dim).to(self.device)
    self.g_opt = optim.Adam(self.generator.parameters(), lr=0.0001)

    self.generator = self.generator.apply(_init_weights)
    self.discriminator = self.discriminator.apply(_init_weights)

class GAN_Trainer():
  def __init__(self, z_dim, model, device="cuda:0") -> None:
    self.device = device
    self.gan = model
    self.z_dim = z_dim

    self._d_mean_train_loss = None
    self._g_mean_train_loss = None

  def train(self, batch) -> None:
    print(1)
    self.batch = batch.to(self.device)

    self.noise = gen_noise(self.batch.shape[0],self.z_dim).to(self.device)
    self.gan.g_opt.zero_grad()
    self._g_output = self.gan.generator.forward(self.noise.to(self.device))

    self._g_output = self._g_output.view(self.batch.shape[0],
                                        1,
                                        torch.sqrt(torch.tensor(self.gan.output_dim)).int(),
                                        torch.sqrt(torch.tensor(self.gan.output_dim)).int())

    print(self._g_output.shape)

    self._d_for_g_pred = self.gan.discriminator.forward(self._g_output)
    self._g_loss = self.gan.criterion(self._d_for_g_pred, torch.zeros_like(self._d_for_g_pred))
    self._g_loss.backward()
    self.gan.g_opt.step()

    self.gan.d_opt.zero_grad()
    self._d_fake_pred = self.gan.discriminator.forward(self._g_output)
    self._d_fake_loss = self.gan.criterion(self._g_output, torch.zeros_like(self._g_output))

    self._d_real_pred = self.gan.discriminator.forward(self.batch)
    self._d_real_loss = self.gan.criterion(self.batch, torch.ones_like(self.batch))

    self._d_mean_loss = torch.mean(torch.cat((self._d_fake_loss, self._d_real_loss),0))
    self._d_mean_loss.backward(retain_graph=True)
    self.gan.d_opt.step()

    self._d_mean_train_loss = self._d_mean_train_loss + self._d_mean_loss.detach()
    self._g_mean_train_loss = self._g_mean_train_loss + self._g_loss.detach()

torch.cuda.empty_cache()

gan = GAN(hidden_dim=1200,
          output_dim=16384,
          z_dim = 1000,
          criterion=nn.BCEWithLogitsLoss())

trainer = GAN_Trainer(model=gan, z_dim=1000)

#here is where it crashes
from tqdm import trange
torch.cuda.empty_cache()

image = trainset[0][0].to("cuda:0").view(1,
                                         trainset[0][0].shape[0],
                                         trainset[0][0].shape[1],
                                         trainset[0][0].shape[2])
trainer.train(batch=image)

please help! I am starting to lose my sanity already, thanks! <3


title: Average Precision (AP) from scratch for bounding boxes

body:
I am trying to evaluate average precision for a face detection algorithm. I have created a CSV file with all the predicted bounding boxes. I have another CSV in a similar format with all the ground-truth bounding boxes. I found a lot of simple codes online to calculate bounding box overlap and calculating AP but they don't work for a variable number of predictions given ground-truth i.e. if an image has 8 bounding box predictions whereas ground truths are 10 then how do I calculate the average precision?
I tried to mimic one of the algorithms and create a CSV file in the format below:
# Filename-1.jpg
[ x1, y1, x2, y2 ]
[ x1, y1, x2, y2 ]
[ x1, y1, x2, y2 ]
[ x1, y1, x2, y2 ]
# Filename-2.jpg
[ x1, y1, x2, y2 ]
[ x1, y1, x2, y2 ]
[ x1, y1, x2, y2 ]
# Filename-3.jpg
[ x1, y1, x2, y2 ]
[ x1, y1, x2, y2 ]
...
...
...

I am also sharing what I tried so far:
def get_max_iou(pred_boxes, gt_box):
    """
    calculate the iou multiple pred_boxes and 1 gt_box (the same one)
    pred_boxes: multiple predict  boxes coordinate
    gt_box: ground truth bounding  box coordinate
    return: the max overlaps about pred_boxes and gt_box
    """
    # 1. calculate the inters coordinate
    if pred_boxes.shape[0] > 0:
        ixmin = np.maximum(pred_boxes[:, 0], gt_box[:, 0])
        ixmax = np.minimum(pred_boxes[:, 2], gt_box[:, 2])
        iymin = np.maximum(pred_boxes[:, 1], gt_box[:, 1])
        iymax = np.minimum(pred_boxes[:, 3], gt_box[:, 3])

        iw = np.maximum(ixmax - ixmin + 1., 0.)
        ih = np.maximum(iymax - iymin + 1., 0.)

    # 2.calculate the area of inters
        inters = iw * ih

    # 3.calculate the area of union
        uni = ((pred_boxes[:, 2] - pred_boxes[:, 0] + 1.) * (pred_boxes[:, 3] - pred_boxes[:, 1] + 1.) +
               (gt_box[2] - gt_box[0] + 1.) * (gt_box[3] - gt_box[1] + 1.) -
               inters)

    # 4.calculate the overlaps and find the max overlap ,the max overlaps index for pred_box
        iou = inters / uni
        iou_max = np.max(iou)
        nmax = np.argmax(iou)
        return iou, iou_max, nmax


title: While running pytorch i got an error 'TypeError: object of type'CatsAndDogsDataset' has no len()' and I want to know how to fix it

body:
I got an error while running pytorch
I train artificial intelligence with ResNet, and I wrote my own custom dataset for the dataset. After loading the data set from ResnNet, training data and test data were set separately by learning with artificial intelligence. But even though I ran it, an error occurred, but I don't know what kind of problem occurred.
Next, I will attach my ResNet.py code and my own data set CustomDataset.py code.
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torchvision import transforms, datasets, models
    from customDataset import CatsAndDogsDataset

    USE_CUDA = torch.cuda.is_available()
    DEVICE = torch.device("cuda" if USE_CUDA else "cpu")

    EPOCHS = 3
    BATCH_SIZE = 10

    dataset = CatsAndDogsDataset(csv_file = 'cats_dogs.csv', root_dir = 'cats_dogs_resized',transform = transforms.ToTensor())

    train_set, test_set = torch.utils.data.random_split(dataset, [28,4])
    train_loader = DataLoader(dataset=train_set, batch_size=BATCH_SIZE, shuffle=True)
    test_loader = DataLoader(dataset=test_set, batch_size=BATCH_SIZE, shuffle=True)

    class BasicBlock(nn.Module):
    def __init__(self, in_planes, planes, stride=1):
    super(BasicBlock, self).__init__()
    self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=3,stride=stride, padding=1, bias=False)
    self.bn1 = nn.BatchNorm2d(planes)
    self.conv2 = nn.Conv2d(planes, planes, kernel_size=3,stride=1, padding=1, bias=False)
    self.bn2 = nn.BatchNorm2d(planes)

    self.shortcut = nn.Sequential()
    if stride != 1 or in_planes != planes:
    self.shortcut = nn.Sequential(nn.Conv2d(in_planes, planes,kernel_size=1, stride=stride, bias=False),
    nn.BatchNorm2d(planes))

    def forward(self, x):
    out = F.relu(self.bn1(self.conv1(x)))
    out = self.bn2(self.conv2(out))
    out += self.shortcut(x)
    out = F.relu(out)
    return out

    class ResNet(nn.Module):
    def __init__(self, num_classes=10):
    super(ResNet, self).__init__()
    self.in_planes = 16

    self.conv1 = nn.Conv2d(3, 16, kernel_size=3,stride=1, padding=1, bias=False)
    self.bn1 = nn.BatchNorm2d(16)
    self.layer1 = self._make_layer(16, 2, stride=1)
    self.layer2 = self._make_layer(32, 2, stride=2)
    self.layer3 = self._make_layer(64, 2, stride=2)
    self.linear = nn.Linear(64, num_classes)

    def _make_layer(self, planes, num_blocks, stride):
    strides = [stride] + [1] * (num_blocks - 1)
    layers = []
    for stride in strides:
    layers.append(BasicBlock(self.in_planes, planes, stride))
    self.in_planes = planes
    return nn.Sequential(*layers)

    def forward(self, x):
    out = F.relu(self.bn1(self.conv1(x)))
    out = self.layer1(out)
    out = self.layer2(out)
    out = self.layer3(out)
    out = F.avg_pool2d(out, 8)
    out = out.view(out.size(0), -1)
    out = self.linear(out)
    return out

    model = ResNet().to(DEVICE)
    optimizer = optim.SGD(model.parameters(), lr=0.1,momentum=0.9, weight_decay=0.0005)
    scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=50, gamma=0.1)

    print(model)

    def train(model, train_loader, optimizer, epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
    data, target = data.to(DEVICE), target.to(DEVICE)
    optimizer.zero_grad()
    output = model(data)
    loss = F.cross_entropy(output, target)
    loss.backward()
    optimizer.step()

    def evaluate(model, test_loader):
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
    for data, target in test_loader:
    data, target = data.to(DEVICE), target.to(DEVICE)
    output = model(data)

    test_loss += F.cross_entropy(output, target,reduction='sum').item()

    pred = output.max(1, keepdim=True)[1]
    correct += pred.eq(target.view_as(pred)).sum().item()

    test_loss /= length(test_loader.dataset)
    test_accuracy = 100. * correct / length(test_loader.dataset)
    return test_loss, test_accuracy

    for epoch in range(1, EPOCHS + 1):
    scheduler.step()
    train(model, train_loader, optimizer, epoch)
    test_loss, test_accuracy = evaluate(model, test_loader)

    print('[{}] Test Loss: {:.4f}, Accuracy: {:.2f}%'.format(
    epoch, test_loss, test_accuracy))

import os
import pandas as pd
import torch
from torch.utils.data import Dataset
from skimage import io

    class CatsAndDogsDataset(Dataset):
    def __init__(self, csv_file, root_dir, transform=None):
    self.annotations = pd.read_csv(csv_file)
    self.root_dir = root_dir
    self.transform = transform

    def __length__(self):
    return length(self.annotations)

    def __getitem__(self, index):
    img_path = os.path.join(self.root_dir, self.annotations.iloc[index, 0])
    image = io.imread(img_path)
    y_label = torch.tensor(int(self.annotations.iloc[index, 1]))

    if self.transform:
    image = self.transform(image)

    return (image, ylabel)

In this way, if I write the code and then run it
TypeError: object of type'CatsAndDogsDataset' has no len()
I wonder why I can't have len(). In addition, an error occurred in the result of running Backend.ai instead of pycharm, but the error content is
Cannot verify that dataset is Sized
if sum(lengths) != len(dataset):
It is raise ValueError("sum of input lengths does not equal the length of the input dataset!").
Error appears. Is there a workaround? help me plz


title: PyTorch tensor declared as torch.long becomes torch.int64

body:
I am new to PyTorch so I haven't worked a lot with PyTorch Tensors. Something I am puzzled about is if I declare the dytpe of a tensor as torch.long, and then check the dtype it is int64. For example:
In [62]: a = torch.tensor([[0, 1, 1, 2],
                       [1, 0, 2, 1]], dtype=torch.long)
         a.dtype

Out[62]: torch.int64

I am probably making some silly mistake.
Why is this happening?
Edit:
     89         if isinstance(edge_index, Tensor):
---> 90             assert edge_index.dtype == torch.long
     91             assert edge_index.dim() == 2
     92             assert edge_index.size(0) == 2

In my case a is edge_index.


title: Training a model on an entire dataset by dividing the dataset into chunks & loading the model back again untill all chunks of the dataset are trained

body:
This is my first question on StackOverflow. I am working on the CUAD(Contract Understanding Atticus Dataset) which is a Q&A based dataset. But training 80% of the dataset in one go is impossible due to resource constraints. I am using the boilerplate code provided by HuggingFace Transformer docs for Q&A task here. My hands are tied with Google Colab Pro. So, it's not possible for me to use multiple GPU's in training the dataset. Inspite of using the hyperparameters below, I'm unable to avoid errors due to memory constraints like "CUDA out of Memory" etc.
args = TrainingArguments(
    'cuad-roberta',
    evaluation_strategy = "epoch",
    learning_rate=3e-5,
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    num_train_epochs=2,
    weight_decay=0.01,
    save_steps=5000,
    logging_steps=5000,
    save_total_limit=100,
    gradient_accumulation_steps = 12,
    eval_accumulation_steps = 4,
)

Under these circumstances, I have divided my training set(80%) into 4 parts with each part holding 25% data. So, using any Q&A supported pretrained model from Transformers, I've trained the first 25% of the training data and then saved the model in a directory of my drive. Then, I have loaded that tokenizer and model from the saved directory and trained the next 25% of my training data on the same model as shown below.
tokenizer = AutoTokenizer.from_pretrained('/content/drive/MyDrive/models/cuad-25%-roberta-base')
model = AutoModelForQuestionAnswering.from_pretrained('/content/drive/MyDrive/models/cuad-25%-roberta-base')

I repeated the step two more times to complete training the model on the entire training data.
Now, my question is that, Is this approach correct in terms of training a model when I have resource constraints? If it is correct, will this approach hurt the performance of my model? I'm relatively new to ML and NLP so please kindly consider any silly mistakes.
Also, any sources for understanding, visualising or implementing the Q&A task through HuggingFace Transformers would be really helpful.


title: RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same even after put model and input to cuda

body:
I met "RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same" even after I put my input and model to device. I would be appreciated if someone could help me out.
point_net.to(device)
for epoch in range(100):
    running_loss = 0.0
    correct = 0.0
    total = 0

    for i in range(0, points_tensor.shape[0], 2048):
        X = points_tensor[i:i + 2048]
        y = labels_tensor[i:i + 2048]

        X = X.to(device)
        y = y.to(device)
        optimizer.zero_grad()

        y_pred = point_net(X)
        loss = criterion(y_pred, y)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()


title: CNN Pytorch only batches of spatial targets supported error

body:
So I've architected the following model which I will use to classify the MNIST Fashion data.
class CNN(nn.Module):
    def __init__(self, **kwargs):
      super().__init__()

      self.conv1    = nn.Conv2d(784, 64, 2, 1, padding=5)
      self.maxpool  = nn.MaxPool2d(kernel_size=2, stride=2)
      self.conv2    = nn.Conv2d(64, 128, 2, 2, padding = 0)
      self.conv2_bn = nn.BatchNorm2d(128)
      self.relu     = nn.ReLU()
      self.dense    = nn.Linear(1, 128)
      self.softmax  = nn.Softmax()

    def forward(self, x): # you can add any additional parameters you want

        x = self.conv1(x)
        x = F.max_pool2d(F.relu(x), kernel_size=2)
        x = self.conv2(x)
        x = self.conv2_bn(x)
        x = F.max_pool2d(F.relu(x), kernel_size=2)
        print(x.shape)
        x = self.dense(x)
        x = F.relu(x)

        return F.log_softmax(x)

And this is where I run my code:
for epoch in range(max_epoch):

    print('EPOCH='+str(epoch))

    correct = 0
    total = 0

    running_loss = 0

    for data, label in tzip(TRAX, TRAY):

        #train = data.view(64,1,2,2)

        DAAA = data.view(1,784,1,1)
        #zeroing the parameter
        optimizer.zero_grad()
        label = torch.tensor([label]).type(torch.LongTensor)
        #forwards prop
        outputs = model2(DAAA)
        loss = criterion(outputs, label)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

        '========================================'
        _, predicted = torch.max(outputs.data, 1)
        total += label.size(0)
        correct += (predicted == label).sum().item()

        '========================================'

    print('\n')
    print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))
    print('\n')
    print(str(epoch)+'loss= '+str(running_loss))

    lossjournal.append(running_loss)
    accjournal.append(100 * correct / total)

print('Finished Training')

<ipython-input-378-27ce013b2c10> in <module>
     55         #forwards prop
     56         outputs = model2(DAAA)
---> 57         loss = criterion(outputs, label)
     58         loss.backward()
     59         optimizer.step()

/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
   1045     def forward(self, input: Tensor, target: Tensor) -> Tensor:
   1046         assert self.weight is None or isinstance(self.weight, Tensor)
-> 1047         return F.cross_entropy(input, target, weight=self.weight,
   1048                                ignore_index=self.ignore_index, reduction=self.reduction)
   1049

/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2691     if size_average is not None or reduce is not None:
   2692         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2693     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2694
   2695

/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2388         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2389     elif dim == 4:
-> 2390         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2391     else:
   2392         # dim == 3 or dim > 4

RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 1

When I run my model I got this error but I don't know what to do from now on? What adjustments should I make for this model to work? I know the problem is with the criterion but is it because of the output of the model, which is of shape [1, 128, 1, 128]?


title: How to propagate neural network gradients through a Networkx graph?

body:
I'm trying to train my neural network to make every edge's weight in the graph to be 10.
I'm starting out by generating random points (inp), and make each 2 adjacent points (using idx) have an edge with weight = 1. And then if the 2 adjacent points already have an edge, the edge's weight is being sent to the NN that outputs what additional weight to add to it.
import warnings
warnings.filterwarnings("ignore", category=UserWarning)

import torch
import torch.nn as nn
import networkx as nx
import matplotlib.pyplot as plt
import torch.optim as optim

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1,3)
        self.fc2 = nn.Linear(3,1)

    def forward(self, x):
        x = self.fc1(x)
        x = self.fc2(x)
        return x

g = nx.DiGraph()
model = Net()
optimizer = optim.Adam(model.parameters(), lr = 1e-3)

def training(n_iter):
    for epoch in range(n_iter):
        print(epoch)
        inp = torch.randint(0,10,(20,))
        idx = 0
        while idx < len (inp) - 1:
            if  g.has_edge(inp[idx].item(), inp[idx+1].item()): #edge exist

                edge_weight = g[inp[idx].item()][inp[idx+1].item()]["weight"]
                edge_weight_tensor = torch.tensor([edge_weight]).float() #to tensor

                added_edge_weight = model(edge_weight_tensor) #value from network
                g[inp[idx].item()][inp[idx+1].item()]["weight"] += added_edge_weight
                idx +=1
            else:
                g.add_edge(inp[idx].item(),inp[idx+1].item(), weight = 1)
                idx +=1

        edges = g.edges()
        weights = [g[u][v]['weight'] for u,v in edges]
        optimizer.zero_grad()

        loss_list = [w for w in weights if not isinstance(w, int)] #only take tensors
        try:
            loss_tensors = (torch.stack(loss_list, dim=0)-10)
            loss_square = torch.square(loss_tensors)
            loss = torch.sum(loss_square)
            print(loss)
        except RuntimeError: #no tensors - hence create a 0 loss
            loss = torch.tensor(0.0, requires_grad = True)

        loss.backward(retain_graph=True)
        optimizer.step()
    return weights

weights = training(5)

#plot
plt.figure(figsize=(6,6))
pos = nx.spring_layout(g, k = 0.5)

nx.draw(g, with_labels=True, node_color='skyblue', font_weight='bold',  width=weights, pos=pos)

My issue is that I'm not sure that the gradients can propagate this way, and also that it doesn't seem like I can add the NN weight addition to the edge's weight-- getting the following error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).


title: How does one run PyTorch on a A40 GPU without errors (with DDP too)?

body:
I tried running my pytorch code but got this error:
A40 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.
If you want to use the A40 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
Using backend: pytorch
/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/cuda/__init__.py:104: UserWarning:
A40 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.
If you want to use the A40 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
Traceback (most recent call last):
  File "/home/miranda9/ML4Coq/ml4coq-proj-src/embeddings_zoo/tree_nns/main_brando.py", line 305, in <module>
    main_distributed()
  File "/home/miranda9/ML4Coq/ml4coq-proj-src/embeddings_zoo/tree_nns/main_brando.py", line 201, in main_distributed
    mp.spawn(fn=train, args=(opts,), nprocs=opts.world_size)
  File "/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
    while not context.join():
  File "/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 118, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/miranda9/ML4Coq/ml4coq-proj-src/embeddings_zoo/tree_nns/main_brando.py", line 210, in train
    setup_process(opts, rank, master_port=opts.master_port, world_size=opts.world_size)
  File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch/distributed.py", line 165, in setup_process
    dist.init_process_group(backend, rank=rank, world_size=world_size)
  File "/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 455, in init_process_group
    barrier()
  File "/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1607369981906/work/torch/lib/c10d/ProcessGroupNCCL.cpp:31, unhandled cuda error, NCCL version 2.7.8

but then it sends me to download it for my mac...? which is weird. What version of
pytorch, cuda, cudnn, nccl and other things do I need for a GPU A40?
to see the code I ran and conda env info see this: https://github.com/pytorch/pytorch/issues/58794

related links

https://github.com/pytorch/pytorch/issues/45021
https://github.com/pytorch/pytorch/issues/45028
https://github.com/pytorch/pytorch/issues/58794


title: PyTorch does not make initial weights random

body:
I created a Neural Network that takes two greyscale images 14x14 pixels portraying a digit (from MNIST database) and returns 1 if the first digit is less or equal to the second digit, returns 0 otherwise. The code runs, but every time the initial weights are the same. They should be random
Forcing the initial weights to be random, by using the following line of code in the Net class, does not help.
torch.nn.init.normal_(self.layer1.weight, mean=0.0, std=0.01)

Here is the code of the "main.py" file:
import os; os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"
import torch
import torch.nn as nn
from dlc_practical_prologue import *

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(2*14*14, 32)
        #torch.nn.init.normal_(self.layer1.weight, mean=0.0, std=0.01)
        #self.layer2 = nn.Linear(100, 100)
        #self.layer3 = nn.Linear(100, 100)
        self.layer2 = nn.Linear(32, 1)

    def forward(self, x):
        x = torch.relu(self.layer1(x))
        #x = torch.relu(self.layer2(x))
        #x = torch.relu(self.layer3(x))
        x = torch.sigmoid(self.layer2(x))
        return x

if __name__ == '__main__':
    # Data initialization
    N = 1000
    train_input, train_target, train_classes, _, _, _, = generate_pair_sets(N)
    _, _, _, test_input, test_target, test_classes = generate_pair_sets(N)

    train_input = train_input.view(-1, 2*14*14)
    test_input = test_input.view(-1, 2*14*14)

    train_target = train_target.view(-1, 1)
    test_target = test_target.view(-1, 1)

    # I convert the type to torch.float32
    train_input, train_target, train_classes, test_input, test_target, test_classes = \
        train_input.type(torch.float32), train_target.type(torch.float32), train_classes.type(torch.long), \
        test_input.type(torch.float32), test_target.type(torch.float32), test_classes.type(torch.long)

    # Create the neural network
    net = Net()

    # Training
    learning_rate = 0.01
    # Use MSELoss
    loss = nn.MSELoss()
    # Use Adam optimizer
    optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)

    EPOCHS = 50

    for param in net.parameters():
        print(param)

    for epoch in range(EPOCHS):
        target_predicted = net(train_input)
        l = loss(train_target, target_predicted)  #loss = nn.MSELoss()
        #l = loss(target_predicted, train_target)
        l.backward()
        optimizer.step()
        optimizer.zero_grad()

        #print(l)

    # Testing
    total = 1000
    correct = 0
    with torch.no_grad():
        correct = ( test_target == net(test_input).round() ).sum()

    print("Accuracy %.2f%%" % (correct / total * 100))

Here is the code for "dlc_practical_monologue.py":
import os; os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"
import torch
from torchvision import datasets

import argparse
import os

import urllib

######################################################################

parser = argparse.ArgumentParser(description='DLC prologue file for practical sessions.')

parser.add_argument('--full',
                    action='store_true', default=False,
                    help = 'Use the full set, can take ages (default False)')

parser.add_argument('--tiny',
                    action='store_true', default=False,
                    help = 'Use a very small set for quick checks (default False)')

parser.add_argument('--seed',
                    type = int, default = 0,
                    help = 'Random seed (default 0, < 0 is no seeding)')

parser.add_argument('--cifar',
                    action='store_true', default=False,
                    help = 'Use the CIFAR data-set and not MNIST (default False)')

parser.add_argument('--data_dir',
                    type = str, default = None,
                    help = 'Where are the PyTorch data located (default $PYTORCH_DATA_DIR or \'./data\')')

# Timur's fix
parser.add_argument('-f', '--file',
                    help = 'quick hack for jupyter')

args = parser.parse_args()

if args.seed >= 0:
    torch.manual_seed(args.seed)

######################################################################
# The data

def convert_to_one_hot_labels(input, target):
    tmp = input.new_zeros(target.size(0), target.max() + 1)
    tmp.scatter_(1, target.view(-1, 1), 1.0)
    return tmp

def load_data(cifar = None, one_hot_labels = False, normalize = False, flatten = True):

    if args.data_dir is not None:
        data_dir = args.data_dir
    else:
        data_dir = os.environ.get('PYTORCH_DATA_DIR')
        if data_dir is None:
            data_dir = './data'

    if args.cifar or (cifar is not None and cifar):
        print('* Using CIFAR')
        cifar_train_set = datasets.CIFAR10(data_dir + '/cifar10/', train = True, download = True)
        cifar_test_set = datasets.CIFAR10(data_dir + '/cifar10/', train = False, download = True)

        train_input = torch.from_numpy(cifar_train_set.data)
        train_input = train_input.transpose(3, 1).transpose(2, 3).float()
        train_target = torch.tensor(cifar_train_set.targets, dtype = torch.int64)

        test_input = torch.from_numpy(cifar_test_set.data).float()
        test_input = test_input.transpose(3, 1).transpose(2, 3).float()
        test_target = torch.tensor(cifar_test_set.targets, dtype = torch.int64)

    else:
        print('* Using MNIST')

        ######################################################################
        # import torchvision

        # raw_folder = data_dir + '/mnist/raw/'
        # resources = [
            # ("https://fleuret.org/dlc/data/train-images-idx3-ubyte.gz", "f68b3c2dcbeaaa9fbdd348bbdeb94873"),
            # ("https://fleuret.org/dlc/data/train-labels-idx1-ubyte.gz", "d53e105ee54ea40749a09fcbcd1e9432"),
            # ("https://fleuret.org/dlc/data/t10k-images-idx3-ubyte.gz", "9fb629c4189551a2d022fa330f9573f3"),
            # ("https://fleuret.org/dlc/data/t10k-labels-idx1-ubyte.gz", "ec29112dd5afa0611ce80d1b7f02629c")
        # ]

        # os.makedirs(raw_folder, exist_ok=True)

        # # download files
        # for url, md5 in resources:
            # filename = url.rpartition('/')[2]
            # torchvision.datasets.utils.download_and_extract_archive(url, download_root=raw_folder, filename=filename, md5=md5)
        ######################################################################

        mnist_train_set = datasets.MNIST(data_dir + '/mnist/', train = True, download = True)
        mnist_test_set = datasets.MNIST(data_dir + '/mnist/', train = False, download = True)

        train_input = mnist_train_set.data.view(-1, 1, 28, 28).float()
        train_target = mnist_train_set.targets
        test_input = mnist_test_set.data.view(-1, 1, 28, 28).float()
        test_target = mnist_test_set.targets

    if flatten:
        train_input = train_input.clone().reshape(train_input.size(0), -1)
        test_input = test_input.clone().reshape(test_input.size(0), -1)

    if args.full:
        if args.tiny:
            raise ValueError('Cannot have both --full and --tiny')
    else:
        if args.tiny:
            print('** Reduce the data-set to the tiny setup')
            train_input = train_input.narrow(0, 0, 500)
            train_target = train_target.narrow(0, 0, 500)
            test_input = test_input.narrow(0, 0, 100)
            test_target = test_target.narrow(0, 0, 100)
        else:
            print('** Reduce the data-set (use --full for the full thing)')
            train_input = train_input.narrow(0, 0, 1000)
            train_target = train_target.narrow(0, 0, 1000)
            test_input = test_input.narrow(0, 0, 1000)
            test_target = test_target.narrow(0, 0, 1000)

    print('** Use {:d} train and {:d} test samples'.format(train_input.size(0), test_input.size(0)))

    if one_hot_labels:
        train_target = convert_to_one_hot_labels(train_input, train_target)
        test_target = convert_to_one_hot_labels(test_input, test_target)

    if normalize:
        mu, std = train_input.mean(), train_input.std()
        train_input.sub_(mu).div_(std)
        test_input.sub_(mu).div_(std)

    return train_input, train_target, test_input, test_target

######################################################################

def mnist_to_pairs(nb, input, target):
    input = torch.functional.F.avg_pool2d(input, kernel_size = 2)
    a = torch.randperm(input.size(0))
    a = a[:2 * nb].view(nb, 2)
    input = torch.cat((input[a[:, 0]], input[a[:, 1]]), 1)
    classes = target[a]
    target = (classes[:, 0] <= classes[:, 1]).long()
    return input, target, classes

######################################################################

def generate_pair_sets(nb):
    if args.data_dir is not None:
        data_dir = args.data_dir
    else:
        data_dir = os.environ.get('PYTORCH_DATA_DIR')
        if data_dir is None:
            data_dir = './data'

    train_set = datasets.MNIST(data_dir + '/mnist/', train = True, download = True)
    train_input = train_set.data.view(-1, 1, 28, 28).float()
    train_target = train_set.targets

    test_set = datasets.MNIST(data_dir + '/mnist/', train = False, download = True)
    test_input = test_set.data.view(-1, 1, 28, 28).float()
    test_target = test_set.targets

    return mnist_to_pairs(nb, train_input, train_target) + \
           mnist_to_pairs(nb, test_input, test_target)

######################################################################

Note that I have to add the following line of code to run the code on Windows 10, while it is not necessary to run it on Linux.
import os; os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"

Also on Linux I always get the same initial weights.
Please, can you help me?


I want to test a particular code, and I wrote the following data loader as an experiment (for a larger project)
import pandas as pd
import datetime
from torch.utils.data import Dataset, DataLoader
from random import randint
import random

class TestLoader(Dataset):
    def __init__(self, num_profiles):
        self.num_profiles = num_profiles
        self.profiles = {}

    def __len__(self):
        return self.num_profiles

    def __getitem__(self, idx):
        symbol = random.sample(["A","B","C","D","E"], 1)[0]
        if symbol not in self.profiles.keys():
            country = random.sample(['China', 'UK', 'Germany', 'Fiji', 'Lichtenstein'], 1)[0]
            gender = random.sample(['Male', 'Female', 'Other'], 1)[0]
            self.profiles[symbol] = {"country": country, "gender": gender}

        timeSince = random.uniform(10,80)
        dt=datetime.date(randint(2005,2025), randint(1,12),randint(1,28))
        date = np.datetime64(dt)

        sample = {'value': float(idx), 'country': self.profiles[symbol]["country"],
                  'gender': self.profiles[symbol]["gender"],
                  "timeSince": timeSince, "symbol": symbol, "date": date}
        return sample

And I have a code like the following to try visualize the code:
user_profiles = TestLoader(2000)
batch_size = 40
dataloader = DataLoader(user_profiles, batch_size=batch_size, shuffle=True)

visualize = pd.DataFrame()

for i,data in enumerate(dataloader):
    if i < 10:
       visualize = pd.concat([visualize, pd.DataFrame(data)], axis = 0).copy()
visualize

But then, on the line of for i,data in enumerate(dataloader):, I get the following error:
TypeError                                 Traceback (most recent call last)
<ipython-input-91-584c34ba614b> in <module>
      2
      3
----> 4 for i,data in enumerate(dataloader):
      5     pass
      6 #     if i < 10:

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in __next__(self)
    515             if self._sampler_iter is None:
    516                 self._reset()
--> 517             data = self._next_data()
    518             self._num_yielded += 1
    519             if self._dataset_kind == _DatasetKind.Iterable and \

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)
    555     def _next_data(self):
    556         index = self._next_index()  # may raise StopIteration
--> 557         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    558         if self._pin_memory:
    559             data = _utils.pin_memory.pin_memory(data)

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py in fetch(self, possibly_batched_index)
     45         else:
     46             data = self.dataset[possibly_batched_index]
---> 47         return self.collate_fn(data)

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py in default_collate(batch)
     71         return batch
     72     elif isinstance(elem, container_abcs.Mapping):
---> 73         return {key: default_collate([d[key] for d in batch]) for key in elem}
     74     elif isinstance(elem, tuple) and hasattr(elem, '_fields'):  # namedtuple
     75         return elem_type(*(default_collate(samples) for samples in zip(*batch)))

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py in <dictcomp>(.0)
     71         return batch
     72     elif isinstance(elem, container_abcs.Mapping):
---> 73         return {key: default_collate([d[key] for d in batch]) for key in elem}
     74     elif isinstance(elem, tuple) and hasattr(elem, '_fields'):  # namedtuple
     75         return elem_type(*(default_collate(samples) for samples in zip(*batch)))

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py in default_collate(batch)
     63             return default_collate([torch.as_tensor(b) for b in batch])
     64         elif elem.shape == ():  # scalars
---> 65             return torch.as_tensor(batch)
     66     elif isinstance(elem, float):
     67         return torch.tensor(batch, dtype=torch.float64)

TypeError: can't convert np.ndarray of type numpy.datetime64. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

I have some parts later in the code, which really would benefit from using numpy.datetime64, so I prefer to not change the type I am using in the definition of the loader. How can I fix this?


title: 2D Poisson equation with ANNs

body:
I am quite a novice in PyTorch and Python also. The reason I try to familiarize myself with PyTorch is the application of ANNs to PDEs which seems an extremely interesting topic. So I assigned myself to extend this implementation https://github.com/dbgannon/NNets-and-Diffeqns/blob/master/non-linear-pde-final.ipynb . More specifically this code approximates the solution of a 1D Poisson equation with a feedforward neural network. I would like to ivestigate, along the same lines, the following 2D BVP
Delta u(x,y)=-2pi^2sin(pi x)sin(pi y)
u(0,y)=u(1,y)=u(x,0)=u(x,1)=0

where Delta is the Laplace operator. An analytic solution is u_exact=sin(pi x) sin(pi y). The extension of some points is straightforward, but I got stuck in the imposition of the 2D boundary condition i.e. the appropriate modification of In [67] and the modification of In [70] where, frankly, I don’t really understand what it does (Creates mini batches?). Could someone provide some guidelines for my implementation or/and other related implementations and projects using PyTorch?


title: What kind of tricks we can play with to further refine the trained neural network model so that it has lower objective function value?

body:
I ask this question because many deep learning frameworks, such as Caffe, supports model refining function. For example, in Caffe, we can use snapshot to initialling the neural network parameters and then continue performing training as the following command shows:

./caffe train -solver solver_file.prototxt -snapshot snap_file.solverstate

In order to further train the model, the following tricks I can play with:

use smaller learning rate
change optimisation method. For example, change stochastic gradient descent to ADAM algorithm

Any other tricks I can play with?

ps: I understand that reducing the loss function value of the training samples does not mean that we can get a better model.


title: What is the best clean way to handle three datasets with different size for a training neural net?

body:
I have three datasets,each has different lengths. I have a model with three input pipeline (similar to Siamese, but three input pipeline) and therefore, I have to separately feed the data. I cannot concatenate them. The first one has around 600 ,second one has 3000 and last one has 40000 samples. In each training loop, I want to go through them and feed them into my model.

I defined three datasets and three dataloaders in pytorch; however, I am not sure how to iterate over them?

In the case of two, I have tried cycle over the smaller dataset and worked. But it is not working for three.

for step, ((images_Di, labels_Di),(images_Dj, labels_Dj)) in enumerate(zip(loader_D2, cycle(loader_D1))):

I appreciate if any one help me how is best way to handle them.


title: why is this piece wrong in jupyter-notebook while it's correct in terminal?

body:
When I ran this piece of code in jupyter, it didn't run correctly. The error is :

TypeError                                 Traceback (most recent call
last) <ipython-input-16-76d9b57527d4> in <module>
     14 embed_layer = emb()
     15
---> 16 inp = torch.tensor([ 1, 2, 3, 4, 5, 6 ])
     17
     18 out = embed_layer( inp )

TypeError: 'module' object is not callable

BUT, when I ran it in terminal, I can run it correctly.
I am trying to train a model to generate words embedding.

import torch
import torch.nn as nn

class emb( nn.Module ):

    def __init__( self ):
        super( emb, self ).__init__()
        self.embed_layer = nn.Embedding(8,3)

    def forward( self, inp ):
        out = self.embed_layer( inp )
        return out

embed_layer = emb()

inp = torch.tensor([ 1, 2, 3, 4, 5, 6 ])

out = embed_layer( inp )
print(out)


title: How to change input type(image) to list or array when using PyTorch tutorial code

body:
I have searched the code that uses list or array input data for training DQN code. But I have could not find any code.

Currently, I reference the reinforcement learning tutorial(DQN) of Pytorch.
However, this code uses image input data.

I want to know how to change the image input data to list or array input data.

(I need help to resolve my research that uses list input data. List input data shape is 1 by 9. )


title: pytorch make customized gradient update with label

body:
I want to make customized gradient update like below, modifying backprop value of midlayer (dL/dx_midlayer = dL/dx_midlayer + some_custom_value_using_'label')

for epoch in range(epochs):
    x, y = some_inputs
    output = model(x)
    loss = criterion(output, label)

    backprop_grad_until_last_conv = loss.backward_until_last_conv...()
    x_hat = last_conv_input_features
    backprop_grad_until_last_conv += customfunc(x_hat, label, epoch)

    loss.backward_after_last_conv...()
    optimizer.step()

I made a layer for custom backward function, but cannot get label and epoch argument because torch.nn.Module 's backward argument is (self, grad_output), so I just make a backward function that can use 'label' and 'epoch'

are there any way that can do that..?


title: Obtaining the shape (4, 1, 84, 84) with pytorch

body:
Suppose I have four pytorch tensors (tensor1, tensor2, tensor3, tensor4). Each tensor is of shape (1, 1, 84, 84). The first dimension is the number of tensors, the second dimension is the number of colors (e.g. grayscale in our example) and the last two dimensions represent the height and the width of the image.

I want to stack them so that I get the shape (4, 1, 84, 84).

I tried torch.stack((tensor1, tensor2, tensor3, tensor4), dim=0), but I got a shape (4, 1, 1, 84, 84).

How can I stack those tensors so that the shape will be (4, 1, 84, 84)


title: Combining feature matrixes of different shapes into a single feature

body:
A library that I am using only supports 1 feature matrix as an input.

Therefore, I would like to merge my two features into a single feature.

Feature #1: a simple float e.g. tensor([1.9])
Feature #2: categorical that I would like to one-hot encode tensor([0., 1., 0])

tensor([
    [1.9, 0., 0.],     # row 1 for float
    [0., 1., 0.]       # row 2 for OHE
])

My plan would be to take the 1x1 feature and the 3x1 feature merge them into a 3x2.

For the float row, I would always have the 2nd and 3rd entries zeroed out. <-- is there a better approach? e.g. should i use three 1.9's?

Would this method give the affect of training on both features simultaneously?


title: Increasing outputs of Resnet?

body:
I'm trying to do a CAPTCHA recognition based on this code https://github.com/shishishu/pytorch-captcha-recognition. Instead of recognizing 4 characters, I decided to increase to 6 and modified the model.py as follow (here are some parts of it):

class ResidualBlock(nn.Module):
    def __init__(self, inchannel, outchannel, stride=1):
        super().__init__()
        self.left = nn.Sequential(
            nn.Conv2d(inchannel, outchannel, kernel_size=3, stride=stride, padding=1, bias=False),
            nn.BatchNorm2d(outchannel, track_running_stats=True),
            nn.ReLU(inplace=True),
            nn.Conv2d(outchannel, outchannel, kernel_size=3, stride=1, padding=1, bias=False),
            nn.BatchNorm2d(outchannel, track_running_stats=True)
        )
        self.shortcut = nn.Sequential()
        if stride != 1 or inchannel != outchannel:
            self.shortcut = nn.Sequential(
                nn.Conv2d(inchannel, outchannel, kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(outchannel, track_running_stats=True)
            )

    def forward(self, x):
        out = self.left(x)
        out += self.shortcut(x)
        out = F.relu(out)
        return out

class ResNet(nn.Module):
    def __init__(self, ResidualBlock, num_classes=charLength):
        super().__init__()
        self.inchannel = 64
        self.conv1 = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False),
            nn.BatchNorm2d(64, track_running_stats=True),
            nn.ReLU(),
        )
        self.layer1 = self.make_layer(ResidualBlock, 64, 2, stride=1)
        self.layer2 = self.make_layer(ResidualBlock, 128, 2, stride=2)
        self.layer3 = self.make_layer(ResidualBlock, 256, 2, stride=2)
        self.layer4 = self.make_layer(ResidualBlock, 512, 2, stride=2)
        self.fc1 = nn.Linear(512, num_classes)
        self.fc2 = nn.Linear(512, num_classes)
        self.fc3 = nn.Linear(512, num_classes)
        self.fc4 = nn.Linear(512, num_classes)
        self.fc5 = nn.Linear(512, num_classes)
        self.fc6 = nn.Linear(512, num_classes)

    def make_layer(self, block, channels, num_blocks, stride):
        strides = [stride] + [1] * (num_blocks - 1)
        layers = []
        for stride in strides:
            layers.append(block(self.inchannel, channels, stride))
            self.inchannel = channels
        return nn.Sequential(*layers)

    def forward(self, x):
        x = self.conv1(x)  # [3, 32, 32] -> [64, 32, 32]
        x = self.layer1(x)  # -> [64, 32, 32] -> [64, 32, 32]
        x = self.layer2(x)  # -> [128, 16, 16] -> [128, 16, 16], floor in conv2d
        x = self.layer3(x)  # -> [256, 8, 8] -> [256, 8, 8]
        x = self.layer4(x)  # -> [512, 4, 4] -> [512, 4, 4]
        x = F.avg_pool2d(x, 4)  # -> [512, 1, 1]
        x = x.view(-1, 512)  # -> [1, 512]
        y1 = self.fc1(x)  # -> [1, 62]
        y2 = self.fc2(x)
        y3 = self.fc3(x)
        y4 = self.fc4(x)
        y5 = self.fc5(x)
        y6 = self.fc6(x)
        return y1, y2, y3, y4, y5, y6

Other files were adjusted as well to get 6 outputs. However, when training, I noticed that the test accuracy is very low when compared to the 4 characters' code. I generated captcha dataset with 6 characters (100K trainset, 10K testset). Example of accuracy of first 5 epochs for 4 characters CAPTCHA (using original code, same amount of dataset):

epoch 1, the training loss is 2054.17636
the accuracy of test set is 0.675
epoch 2, the training loss is 312.97247
the accuracy of test set is 0.77109375
epoch 3, the training loss is 201.88016
the accuracy of test set is 0.7998697916666667
epoch 4, the training loss is 154.38759
the accuracy of test set is 0.816015625
epoch 5, the training loss is 123.43352
the accuracy of test set is 0.838671875

And this is first 5 epochs when training 6 characters:

epoch 1, the training loss is 5583.41545
the accuracy of test set is 0.00068359375
epoch 2, the training loss is 2519.27621
the accuracy of test set is 0.00078125
epoch 3, the training loss is 2051.24451
the accuracy of test set is 0.00068359375
epoch 4, the training loss is 1840.40299
the accuracy of test set is 0.00087890625
epoch 5, the training loss is 1721.26417
the accuracy of test set is 0.00087890625

Is this normal behaviour or the code I edited has issue? If so, how can I properly increase the outputs of Resnet?

Edit: turns out some typos caused this. Thanks to @Rika for spotting them.


title: Practical Uncertainties in "Reverse" NLLLoss for Generative Adversarial Network

body:
I've been attempting to implement my own GAN with some limited successes. I tweaked a bit of how they trained it in the tutorial and was wondering if this more drastic change is viable. The first change I made was to make the discriminator classify n+1 classes where n could be, for example 10 in MNIST, and the n+1th class is the fake class. The discriminator was an imported architecture I made from scratch for a really good classifier. Then my GAN will have the "opposite" of the traditional NLLLoss.

This is the tricky non-traditional calculation part. So because I have a softmax in the last layer of my discriminator, my output from the discriminator will always be from 0 to 1. So I can create a custom loss function for the generator to be a horizontal flip of the NLLLoss which tries to make sure the discriminator does not classify the fakes as the n+1th class.  The idea is that I don't care what class the fakes are classified as long as they're not the n+1th class. This behavior of misclassification is what I want to maximize for the generator.
Here is the function I plotted on desmos to give some visualization:

https://www.desmos.com/calculator/6gdqs28ihk

my actual code for the generator loss function is the code below while my discriminator loss function is the traditional NLLLoss

loss_G = torch.mean(-torch.log(1 - outputG.float()[:,classes]))

Please let me know if this is completely wrong or there is an easier way.


title: Stacking/concatenating tensors increases memory usage

body:
In PyTorch, I found stack-ing or cat-ing multiple tensors would increase memory usage by the sum of the sizes of all these arrays. An example is as follows:

import torch as tc
import torch.autograd as tag
import sys
import psutil
import os
import resource

def get_ru_maxrss():
    """ Return max RSS usage (in kilobytes) """
    size = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    if sys.platform == 'darwin':
        # on Mac OS X ru_maxrss is in bytes, on Linux it is in KB
        size //= 1024
    return size / 1024

def cpuStats():
        print(sys.version)
        print(psutil.cpu_percent())
        print(psutil.virtual_memory())  # physical memory usage
        pid = os.getpid()
        py = psutil.Process(pid)
        memoryUse = py.memory_info()[0] / 2. ** 30  # memory use in GB...I think
        print('memory GB:', memoryUse)

m0 = get_ru_maxrss()

x1 = tc.ones([8192, 8192], requires_grad=True)
print(x1.dtype)
print(get_ru_maxrss() - m0)
print('=======')

y = x1 * 1.1
print(y.dtype)
print(get_ru_maxrss() - m0)
print('=======')

for i in range(10):
    y = tc.cat([y, x1])
    print(y.dtype)
    print(get_ru_maxrss() - m0)
    print('=======')

loss = tc.mean(y)

print(get_ru_maxrss() - m0)
print('=======')

loss.backward()

print(get_ru_maxrss() - m0)
print('=======')

And we can see that each occurence of cat in the for loop increases memory usage by 512 MB, which is the sum of y and x1 (256 MB each). This isn't a major issue for me right now but I'm just curious about it. If I understand it correctly, the vector-Jacobian product of stack or concatenation is just to perform the reversed operation to the gradient vector, splitting it back into multiple arrays whose shapes match the original inputs to the stack or cat function. This process doesn't need the intermediate value of the stacked tensor computed in the forward pass, which is similar to a linear operation, but the latter doesn't incur additional memory usage (e.g., if I change the cat in the for loop to a linear op such as y = y * 1.1, it won't increase memory consumption). So I'm wondering if the increased memory usage is essentially just an empty space allocated to hold the ``splitted'' gradient arrays in the backward pass, which has to be contiguous, so the splitting operation doesn't have to stride in memory? It follows that a linear operation doesn't need this additional memory because the memory space of the input gradient can be directly overwritten without worrying about memory contiguity. Is that right?


title: Worse performance on Windows 10 than Ubuntu 18.04

body:
I have a face detector model retina face implemented in pytorch running at 20 FPS on webcam in windows 10.
But the same model on webcam runs at 60 Fps on ubuntu 18.04.
 cap = cv2.VideoCapture(conf.webcam)
 while (cap.isOpened()):
    #read frame
    ret, frame = cap.read()
    if ret == True:

        start = time.time()
        res = DetectFace(frame)
        end = time.time()
        fps  = ( fps + (1./(end - start)) ) / 2
        print("Run Time ------", end - start)
        cv2.putText(res, "FPS: %f"%(fps),(int(20), int(40)),0, 5e-3 * 200, (0,255,0),3)

        cv2.namedWindow("Demo", 0)
        cv2.resizeWindow('Demo', 1024, 768)
        cv2.imshow('Demo',res)
        if cv2.waitKey(1) & 0xFF ==ord('q'):
            break
    else:
        break

cap.release()
cv2.destroyAllWindows()

The same code was used in both operating systems with CUDA 10.1 (cudnn enabled) and pytorch 1.4.0
Not sure why there's significant drop in performance.


title: Deep Markov model example

body:
I've been going through the tutorial of deep Markov model:

https://pyro.ai/examples/dmm.html

In this model, emissions and transitions are paramaterized by neural networks. However, my question is pretty elementary. Here they say that in the introduction section that 'Each time slice in a sequence spans a quarter note and is represented by an 88-dimensional binary vector that encodes the notes at that time step.' I need to understand what they mean. Does 88 means the # of features they have is 88. So essentially they'll have something like n x 88 where n is number of examples.
Insights would be appreciated.


title: How to add an Image to a table in python?

body:
I'm working on Image Captioning using Pytorch and need to visualize the Image and the corresponding captions side-by-side in a tabular format. Is there any way to achieve this?

table = PrettyTable()
table.field_names = ["Image", "Caption"]

for image in test_captions_dict:
    # img = Image.open(os.path.join(image_dir, image))
    all_captions = test_captions_dict[image]
    captions = '\n'.join(all_captions)
    table.add_row([image, captions])
print(table)

This produces a table with image name in the first column and the corresponding captions in the next column. I need to display the images in the first column.


title: Tensorflow FFJORD Implementation: Solving for multiple time-steps

body:
So tensorflow has an inbuilt bijector function for FFJORD: See here

However, as I can see in its argument final_time, it only supports a single time-step (i.e. scalar).

Now, what I want is to have it calculate for multiple time-steps (i.e. say given an initial_state=x0 and initial_time = 0.0 and final_times=[1.0,2.0,3.0], then it should return calculated values for these 3 time steps). For reference, the original implementation by the authors of FFJORD paper in Pytorch supports calculating for multiple time steps: Original Pytorch Implementation of FFJORD

Is there a way to implement the same in Tensorflow?

TIA


title: Is "the number of 1's in a generated ternary sequence" really a suitable loss function?

body:
I'm building a model that outputs ternary sequences. A constraint I have is that the number of 1s I have in the generated sequence should be small (while other constraints might encourage the number of 1s to be large). Instinctively, I want a term in my loss function to be the number of 1s in the sequence I generate.
I implemented this in Pytorch and I don't have any errors. While this loss function makes sense conceptually, how on earth can my model learn from it? What's the gradient of "the number of 1s in the generated sequence"? Is there something I'm missing, or is autograd more capable than I thought?


title: How to set Pytorch seed for all files?

body: I have different files each importing random and pytorch. I want one seed to be able to apply everywhere (for all the imported files) for both random and pytorch. How can we do this? Just setting seed for random and pytorch in main.py doesn't do anything, resulting in differently trained models, every time.


title: If I employ pack_sequence in a Pytorch RNN, will I have backprop issues?

body:
I have a model that produces logits that I sample from. When I sample, I get a sequence of one-hots.
Part of my loss function involves passing this sequence into an RNN which will output a scalar. This RNN takes a batch of lists of indices, which get packed (they're of variable length) via nn.utils.rnn.pack_sequence and sent through the network, which has an embedding layer.
If the output of my RNN is part of my loss, does my batch of lists of indices (the input to the RNN) need to be differentiable? If so, I think it needs to be of type float to store gradients, but if I do this I get the error RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.FloatTensor instead (while checking arguments for embedding). (In other words, PyTorch requires integer indices).
Am I correct in thinking that this poses a differentiability/backprop problem?


title: pytorch batch_size not returning correct batch?

body:
No matter what I put for batch_size, the batch_size defaults to 1.  Here is my code
train_dataset = DataLoader(dataset=dataset,
                       batch_size=4,
                       shuffle=True,
                       num_workers=0)

and dataset is a custom dataset as follows
class ImageDataset(data.Dataset):

def __init__(self, root_dir, num_augments=2, transform=None):

    self.root_dir = root_dir
    self.img_names = os.listdir(root_dir)[::600]
    self.num_augments = num_augments
    self.transform = transform

def __getitem__(self, index):

    output = []
    img = Image.open(self.root_dir + '/' + self.img_names[index]).convert('RGB')

    for i in range(self.num_augments):
        if self.transform is not None:
            img_transform = self.transform(img)

        output.append(img_transform)

    output = torch.stack(output, axis=0)

    return output

def __len__(self):

    return len(self.img_names)

I am expecting each batch to be of size [batch, num_augments, 3, height, width], but I am getting [1, num_augments, 3, height, width] regardless of my batch size.


title: PyTorch mobile GAN generator occasionally outputs binary noise

body:
I have a PyTorch GAN generator, running on iOS via LibTorch, that's periodically outputting random binary noise. Some of the generated images are correct, but many images are just binary noise -- sample "good" and noise images are shown below. The model is a ClusterGAN, and I've double-checked the latent (gaussian) and class (one-hot encoded) vectors at runtime, which look fine to me. I also tried using a fixed latent and class (normally these vary based on context), and even with the exact same inputs it will still flip between generating good images and generating binary noise. Very strange. I also tried retraining and re-exporting the model, but I get the same issue. Generation is consistently correct in Python.
Has anybody seen this kind of behaviour before?


title: Where are those numbers coming from in pytorch neural networks?

body:
I'm newbie in Pytorch (python), i was just scrolling through their official tutorial and i found this simple neural network architecture. Everything is clear, but those numbers, last three fully connected layers, where are they coming from?
self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension and 16 from last conv layer
self.fc2 = nn.Linear(120, 84) # but here, 120? 84? why? is this just random or there is some logic behind it?
self.fc3 = nn.Linear(84, 10)

full code - https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py


title: Why in tensorflow there is a dense layer in the beginning of the net and not in pytorch?

body:
As said in the title, usually in the pytorch implementations i've found of a gan generator, the first layer usually is a nn.ConvTranspose2d even for 2d noise as input. In the same case for tf models usually i'm finding a dense layer. I'm wrong? How can i convert 'nn.ConvTranspose2d' in tensorflow? thank you all :)
For the code: https://discuss.pytorch.org/t/convert-a-pytorch-generator-to-tensorflow/89952


title: Anaconda error while installing pytorch on windpws

body:
I am running anaconda on windows; When i try to install pytorch in the anaconda prompt I get the following error
ModuleNotFoundError: No module named 'torchvision'
i checked the openssl version via conda list and it is 1.1.g3
I also moved the lib crypto and libssl from the anaconda/library/bin to anaconda3/DLLs (as per one the prior solutions.
What could be the issue? Please help


title: how to represent a game's sequential actions in a PyTorch tensor?

body:
I have dataset that looks like this (extracted from recording an agent play a game):
INDEX        VALUES                        LABELS
0            (0, 1), UpDownUpUp            Up
1            (2, 3), UpUpUpDownDownDown    Up
2            (0, 2), DownUp                Down
3            (0, Undefined), DownUp        Up

where

the values represent the initial random chance events (in the tuple) and the previous actions at this point in the game.
the labels represent the next optimal action.

How can I represent the values into a PyTorch tensor? My issues come mainly from the fact that the values are of different sizes (DownUp is two actions, UpDownUp is three, etc.), and the second element in the tuple is sometimes undefined.


title: How to declare per neuron connections in a pytorch model?

body:
I want to create a brain-alike mess:

We have input tensor I of length n and output tensor O of length p
In between, we have K "intersection" layers
At intersection layers, neurons share their values at that time with a random subset of "close" neurons (+- range c). In the form of w_i*current_neuron_val
and J "creation" layers new neurons are created from a set or ReLued  "close" neurons (+- range c). closed neurons do not go into deeper layers.

Can we do such a thing with PyTorch so that such model will be trainable?


title: Massive drop in training error after the first epoch

body:
I am training an LSTM autoencoder to recreate the input consisting of eight features(floating-point numbers between 0 and 1). Currently, am utilizing a window size of two and am training the model for 50 epochs. However, while training the network I observed that the training error (Mean Square Error) drops significantly after the first epoch. For example, during the first epoch the training error was 17.25. It dropped to 1.8 at the very next and stagnates after the seventh epoch. I was wondering if random initialization of weights might be causing this therefore I retrained one more network and the same phenomenon repeated.
I am not able to deduce the reason for this significant drop in training error after the first epoch and would appreciate any help. I have attached the training error graph and model information for reference.
Model info:
LSTM_AutoencoderModel(
(encoder): Encoder(
(lstm1): LSTM(16, 64)
(lstm2): LSTM(64, 16)
)
(decoder): Decoder(
(lstm1): LSTM(16, 64)
(lin1): Linear(in_features=64, out_features=16, bias=True)
)
)
Training error graph


title: How to reset MaxBytesInUse in Tensorflow 1.13

body:
I want to measure peak memory usage in TensorFlow 1.13 per epoch.
In PyTorch there is a very simple solution by using combination of two functions:
torch.cuda.max_memory_allocated() and torch.cuda.reset_max_memory_allocated()
In TF 1.13 I found only tf.contrib.memory_stats.MaxBytesInUse() which, if I get it right, max memory allocated from the start of the script. Is there anything that allows me to reset that value? I want to measure the necessary GPU memory per epoch in my use case, and some of my epochs require more, some of them less.


title: How can I get the gradient of each weight matrix after passing an image

body:
I'm using a pre-trained resnet101 model from Pytorch:
import torch
import matplotlib.pyplot as plt
import numpy as np
import torch.nn as nn
import cv2 as cv
from torchvision import models, transforms

resnet = models.resnet101(pretrained=True)

I'm trying to pass a single image through the network and get the gradient of each conv2D weight matrix (not each pixel in the filter, but each filter matrix as a whole)
I can read the image like this:
# read and visualize an image
img = cv.imread('dog.jpeg')
img = cv.cvtColor(img, cv.COLOR_BGR2RGB)
plt.imshow(img)
plt.show()

# define the transforms
transform = transforms.Compose([
    transforms.ToPILImage(),
    transforms.Resize((512, 512)),
    transforms.ToTensor(),
])

img = np.array(img)
# apply the transforms
img = transform(img)
print(img.size())
# unsqueeze to add a batch dimension
img = img.unsqueeze(0)
print(img.size())

But I'm not sure how to pass it to the network and access the specific gradients


title: Same Prediction Result for all image in Custom Model with Pytorch

body:
I am trying to create Custom Model for Image Classification to get prediction with use of library Pytorch.
But its's giving something not per expectation like same prediction (showing same index of class no matter what image is chosen) for different images

I am following below References

https://towardsdatascience.com/a-beginners-tutorial-on-building-an-ai-image-classifier-using-pytorch-6f85cb69cba7
https://www.pluralsight.com/guides/image-classification-with-pytorch
https://towardsdatascience.com/create-an-image-classification-web-app-using-pytorch-and-streamlit-f043ddf00c24
https://github.com/terryz1/Image_Classification_App

please find the Code of Model Creation and Prediction  https://github.com/ankiku/CustomVRAppRepo
I also tried to check below references where same issues are posted and some fixtures, but still no luck yet.

https://discuss.pytorch.org/t/get-same-output-with-different-image-on-pre-trained-resnet18/23918
pytorch doesn't give expected output

Please help in finding the root cause of the issue.
Regards
Ankit


title: Getting wrong shape of logits from TFRobertaForSequenceClassification

body:
I am using the the TFRobertaForSequenceClassification class of the huggingface library to create a classifier. According to the documentation, the logits output should have a shape of (batch_size, num_labels). I however get (batch_size, seq_length, num_labels) and I dont understand why.
To reproduce this:
from transformers import TFRobertaForSequenceClassification, RobertaConfig
import numpy as np

seq_len = 512

classifier = TFRobertaForSequenceClassification(RobertaConfig())

#create random inputs for demo
input_ids = np.random.randint(0,10000, size=(seq_len,))
attention_mask = np.random.randint(0,2, size=(seq_len,))
token_type_ids = np.random.randint(0,2, size=(seq_len,))

#make a prediction with batch_size of 1
output = classifier.predict([input_ids, attention_mask, token_type_ids])

print(output.logits.shape)

This outputs logits in the shape of (512,2) but I am expecting (1,2) or (batch_size, num_labels). Can anyone shed any light on why it behaves like this?


title: How can I run multiple pre-trained pytorch models in parallel?

body:
I have four pre-trained models that I want to get predictions from in parallel. The models are running on the GPU. Are there any resources for running models on the GPU in parallel?.
I am sorry I don't have code to show, I just don't know how to start.


title: why using log-sum-exp to calculator forward features

body:
In the pytorch implementation of BLSTM-CRF tutorial (https://pytorch.org/tutorials/beginner/nlp/advanced_tutorial.html#advanced-making-dynamic-decisions-and-the-bi-lstm-crf), log-sum-exp operation is used to calcualted forward features, which is lised below
    def _forward_alg(self, feats):
        # Do the forward algorithm to compute the partition function
        init_alphas = torch.full((1, self.tagset_size), -10000.)
        # START_TAG has all of the score.
        init_alphas[0][self.tag_to_ix[START_TAG]] = 0.

        # Wrap in a variable so that we will get automatic backprop
        forward_var = init_alphas

        # Iterate through the sentence
        for feat in feats:
            alphas_t = []  # The forward tensors at this timestep
            for next_tag in range(self.tagset_size):
                # broadcast the emission score: it is the same regardless of
                # the previous tag
                emit_score = feat[next_tag].view(
                    1, -1).expand(1, self.tagset_size)
                # the ith entry of trans_score is the score of transitioning to
                # next_tag from i
                trans_score = self.transitions[next_tag].view(1, -1)
                # The ith entry of next_tag_var is the value for the
                # edge (i -> next_tag) before we do log-sum-exp
                next_tag_var = forward_var + trans_score + emit_score
                # The forward variable for this tag is log-sum-exp of all the
                # scores.
                alphas_t.append(log_sum_exp(next_tag_var).view(1))
            forward_var = torch.cat(alphas_t).view(1, -1)
        terminal_var = forward_var + self.transitions[self.tag_to_ix[STOP_TAG]]
        alpha = log_sum_exp(terminal_var)
        return alpha

The log_sum_exp operation on the calculated features is in the line
                alphas_t.append(log_sum_exp(next_tag_var).view(1))

My question is that why log_sum_exp operation is needed? Thanks.