title: Will installation from source in different conda env lead to conflict?

body:
I want to install a nightly-version(v1.2) of pytorch from source in a new conda env, and I got v1.1 outside conda env installed from pip.

Will that lead to conflict?


title: PyTorch model expects objects on CPU despite being on GPU

body:
I have a PyTorch Model that I've sent to my device (a GPU on GCS):

dnn_regressor = DNNRegressor(n_feature=dataset.X.shape[1],
                         n_hidden=512,
                         n_hidden_layers=5,
                         batch_norm=True,
                         n_output=1).to(device)

similarly, before training I have sent my (batched) data to the GPU as well. However, when I optimize the model, it appears not to be associated with the GPU:

     17         X_batch, y_batch = X_batch.to(device), y_batch.to(device)
     18
---> 19         prediction = dnn_regressor(X_batch)     # input x and predict based on x

RuntimeError: Expected object of backend CPU but got backend CUDA for argument #4 'mat1'

I'm sure I've done something obviously wrong, but I can't see what it might be.


title: Object parts detection

body: The situation is that I need to generate bounding box proposals for object parts, however I don't have any hard labels for the same. Selective Search was the prevalent method for region proposals. So, is selective search suitable for my purpose (generate bounding box proposals for OBJECT PARTS) or does it go against the idea of object parts?


title: Understanding type of gradient descent and how its calculated in Pytorch framework and making it more dynamic

body:
I recently started implementing Neural network using pytorch framework and here is my code:

      class NN(nn.Module):

         def __init__(self):
            super(NN,self).__init__()
            self.f1 = nn.Linear(1,h)#,bias = False)
            self.f2 = nn.Linear(h,1)#,bias = False)
         def forward(self,x):
            x = self.f1(x)
            x = f.relu(x)
            x = self.f2(x)
            return x

     model = NN()
     loss_type = nn.MSELoss()

     optimizer = torch.optim.Adam(model.parameters(), lr=0.1)

     l = []
     for i in range(epochs):
        y_pred = model.forward(X_train)

        loss = loss_type(y_pred,Y_train)
        l.append(loss.item())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

I have few questions related to this:

(1) Is it updating weights at every data point/training example that is stoachastic gradient decsent (SGD) or its averaging weight and then updating at every epoc.?

(2) Why this super(NN,self).__init__() is used?

(3) Instead of passing x = f.relu(x), can I make it more dynamic like for an example can I dynamically pass following parameters keeping the same architecture and running the model for every combination and storing the result or loss for every combination.

  { hiddenLayer=[10,20,30],activation_function=['sigmoid','relu'],learning rate=[0.1,0.2], epoc=[10,20,30] }

   How can I modify the above code to implement this. So that everytime I do not have to change my hyperparemeters.

(4) What does the NN.parameters contain. How to get dictionary having every layer weights and biases using this?

Thanks!


title: numpy select values based on list of indices. Process batch at once

body:
The general task seems rather straightforwad.
I have given two arrays:
Array A of shape (5,20) i.e. five lists of 100 values each.
and another array B of shape (5, 10) containing the indices I want to select. [Thus array B contains values between 0 and 19]

A minimal example would be solved by

# saqmple initialization
def rnd():
    return np.random.choice(20, 10, replace=False)
A = np.random.random_sample((5, 20))
B = np.array([rnd(),  rnd(),  rnd(),  rnd(),  rnd()])

# solution
selected_values = [A[i,B[i]] for i in range(A.shape[0])]

as even even simpler example take

A = [[4,5,6],[7,8,9]]
B = [[0,1],[1,2]]
out = [[4,5],[8,9]]

Is there a way to reshape, reformat or... to be then able to do this operation without a for loop? I.e. implicitly letting numpy realize that the second entry in B are indices corresponding to the second entry in A and so forth?


title: cant see android app after ./gradlew installDebug on device

body:
I have recently started learning to build android apps, and I am trying to follow the instructions from the Pytorch website, which says:

git clone https://github.com/pytorch/android-demo-app.git
cd HelloWorldApp
./gradlew installDebug

I have my device plugged in and I ran the above and got:

BUILD SUCCESSFUL in 8s
26 actionable tasks: 1 executed, 25 up-to-date

But.. I cannot find the application on the phone..
I am baffled as to why this is.

I would REALLY appreciate any directions to troubleshoot this - spent the last two hours on this!


title: pytorch summary RuntimeError, expected channels mismatch

body:
I am new to AI, when i try to get the summary info from my network, i got RuntimeError:

what parameter should i pass, according to the error? thanks.


title: Does the file get changed in squeue if I modify after being sent into queue?

body:
I have a question: a have neural net file model.py with some parameters set. I have sent it to the slurm queue. When doing squeue I can see that it is still waiting because there are other jobs running. Now, I want to send another variation of model.py but changing the params. Do you know if I change it right now, will it change the params of the first instance sent to the queue?

I mean: suppose that the first instance has param = 20. I send it to the queue. Still not running. Now, I change it to param = 50 and send the second instance to the machine. Will the first have param = 20 or param = 50 ?

Thank you in advance!


title: How to efficiently find correspondences between two point sets without nested for loop in Pytorch?

body:
I now have two point sets (tensor) A and B that shape like

A.size() >>(50, 3) , example: [ [0, 0, 0], [0, 1, 2], ..., [1, 1, 1]]

B.size() >>(10, 3)

where the first dimension stands for number of points and the second dim stands for coordinates (x,y,z)

To some extent, the question could also be simplified into " Finding common elements between two tensors ". Is there a quick way to do this without nested loop ?


title: NLP model giving results only for RNN but not for any of GRU, LSTM, Bidirectional, Stacked, GloVe (PyTorch)

body:
I am building a NLP model for Jigsaw Toxic Comments classification using PyTorch. My model with RNN cell gives me good accuracy results as just over 90% accuracy but when I am using ANY OF THE GRU,LSTM (UniDirectional,Bi,Stacked) and even Stacked/Bi RNN, it is showing accuracy of a max of 30%. I trued using GloVe too but the model did not improve. Just the Plain Vanilla RNN gives me 90% nothing else. Could something please explain what am I doing wrong here?? Full code is in this kaggle notbook.

I have use BCEWithLogitLoss() to the code model below.

class Network(torch.nn.Module):

    def __init__(self,in_neuron,embedding_dim=128,hidden_size=256,out_neuron=1,m_type='rnn',drop=0.53,**kwargs):

        super(Network,self).__init__()

        self.m_type = m_type

        self.embedding = torch.nn.Embedding(in_neuron,embedding_dim)
        if self.m_type == 'lstm':

            self.lstm = torch.nn.LSTM(embedding_dim,hidden_size,**kwargs)
        else:
            self.rnn = torch.nn.RNN(embedding_dim,hidden_size,**kwargs)

        self.dropout = torch.nn.Dropout(drop)

        self.dense = torch.nn.Linear(hidden_size,out_neuron)

    def forward(self,t):

        drop_emb = self.dropout(self.embedding(t)) # apply dropout to embeddings

        if self.m_type == 'lstm':
            out, (hidden_state,_) = self.lstm(drop_emb)
        else:
            out, hidden_state = self.rnn(drop_emb)
            #  shape of rnn_out = (seq_len, batch, num_directions * hidden_size)

        hidden_squeezed = hidden_state.squeeze(0)
        # remove extra dim as (num_layers * num_directions, batch, hidden_size) = (1*1,b,h)

        # Pass the "last" hidden state only because we only want 1 output based on the last hidden state
        return self.dense(hidden_squeezed)

Deep/Stacked Model

class DeepNetwork(torch.nn.Module):
    def __init__(self,in_neuron,embedding_dim=100,hidden_size=256,out_neuron=1,m_type='rnn',drop=0.53,**kwargs):

        super(DeepNetwork,self).__init__()

        self.m_type = m_type

        self.embedding = torch.nn.Embedding(in_neuron,embedding_dim)

        if self.m_type == 'lstm':
            self.lstm = torch.nn.LSTM(embedding_dim,hidden_size,**kwargs) # respective kwargs
        elif self.m_type == 'gru':
            self.gru = torch.nn.GRU(embedding_dim,hidden_size,**kwargs) # respective kwargs
        else:
            self.rnn = torch.nn.RNN(embedding_dim,hidden_size,**kwargs)

        self.dropout = torch.nn.Dropout(drop)

        self.dense = torch.nn.Linear(hidden_size*2,out_neuron)
        # Last output Linear Layer will have the two Hidden States from both the directions to have the result

    def forward(self,t):

        t = self.dropout(self.embedding(t))

        if self.m_type == 'lstm':
            out, (hidden,_) = self.lstm(t)
        elif self.m_type == 'gru':
            out, hidden = self.gru(t)
        else:
            out, hidden = self.rnn(t)
        # shape of rnn = (seq_len, batch, num_directions * hidden_size)

        # Concatenate the last and second last hidden. One is from backward and one is from forward
        t = self.dropout(torch.cat((hidden[-2,:,:],hidden[-1,:,:]),dim=1))

        return self.dense(t)

Can someone please tell me if this is the right implementation or am I doing something wrong here??


title: list the operators in the forward method of module in Pytorch

body:
Is there a way to list the operators used in the forward method of a module, something similar to children() for module.
I need to know if the method is applying softmax in forward and in case remove that.


title: Using pytorch after Training an nn.module with batches can one predict single instances

body: I trained a recurrent net using batches of 10, now I want to continue the predicted time series, but of course to predict t+1 one needs t therefore predicting in batches is not very effective. Is there a way to train using batches but still predict singles instances of the time series.


title: Neural network with pytorch to output the percentage of damage of a given structure/building (regressor, not classifier)

body:
I am writing a NN in Pytorch that takes as input some characteristics of a building (e.g., inertia of each element) and gives you output the % of damage in a given element (e.g, if element #1 is damaged by 10%, then the output is 0.1)
In a nutshell, the output should be between 0 and 1, referring to the percentage of damage (regression) and not the probability of damage (classifier)
For the hidden layers of my NN, I am using the Relu activation function, but I am confused on what to used for the output layer:
1- I tried no activation function (just linear_transform) but I am getting negative number which is not logical (since it should be between 0 and 1)
2- I tried the Relu function, but I am getting zeros every time (not sure why is this happening)
Could you please advise?
NB: I am used to build classifiers, but this is my first regression NN, so if you have any recommendation or any tutorial you can share with me, I would highly appreciate your help.
Best,
Kev


title: Number of outputs in final linear layer for binary classification

body: For binary classification, we could either go for a final linear layer with 1 output, and use a sigmoid with a threshold, or a final linear layer with 2 outputs, and use a softmax. Is there any advantage to one vs the other?


title: How can I resume training from same epoch (int)?

body:
I'm using Pytorch. When loading the model, I want my it to start epoch count from where it was last saved. Weights are loaded fine, but the epoch is reverted back to 0.
checkpoint = {'epoch': epoch + 1, 'state_dict': model.state_dict(), 'optimizer': optimizer.state_dict()}

This is how I resume from checkpoint:
def resume_from_checkpoint(fpath='xyz.pth', model, optimizer=None, scheduler=None):

    print('Loading checkpoint from "{}"'.format(fpath))
    checkpoint = load_checkpoint(fpath)
    model.load_state_dict(checkpoint['state_dict'])
    print('Loaded model weights')

    if optimizer is not None and 'optimizer' in checkpoint.keys():
        optimizer.load_state_dict(checkpoint['optimizer'])
        print('Loaded optimizer')
    if scheduler is not None and 'scheduler' in checkpoint.keys():
        scheduler.load_state_dict(checkpoint['scheduler'])
        print('Loaded scheduler')

    epoch = checkpoint['epoch']
    print('Last epoch = {}'.format(epoch))

    if 'rank1' in checkpoint.keys():
        print('Last rank1 = {:.1%}'.format(checkpoint['rank1']))
    return epoch

And this is how I do training:
for epoch in range(num_epochs):
    total_correct = 0
    total_loss = 0
    if epoch % 1 == 0:
        checkpoint = {'epoch': epoch + 1, 'state_dict': model.state_dict(), 'optimizer': optimizer.state_dict()}
        print("saving checkpoint")
        save_checkpoint(checkpoint)
    for i, traindata in enumerate(trainloader):
        images =
        labels =
        optimizer.zero_grad()


title: Is it possible to do the cnn method without optimizer? Because i'll already tried and try to delete optimization process then i get 0 accuration?

body: my teacher ask me to make a comparison CNN + optimizer and CNN without optimizer. Is it possible to do the cnn method without optimizer? Because i'll already tried to delete optimization process and i get 0 accuration?. Maybe anyone can help me, i am really appreciate it.


title: How to convert output of each hidden layer to local binary pattern (LBP) and feed it to next layer’s input in pytorch

body:
I want to convert output of each hidden layer to local binary pattern (LBP) and feed it to next layer’s input in PyTorch,
In the other word I want to classify my input images (RGB or LBP) via extract the features in each layer in ResNet18!
Thanks.


title: Calculating accuracy gives wonky results

body:
I have a neural network that uses PyTorch. When I run it the accuracy seems a bit wonky (constant 1.00, see below).
epoch=20, loss=0.00002, val_loss=33.33334, acc=1.00000, val_acc=0.66667
.
.
epoch=70, loss=0.00000, val_loss=2.69771, acc=1.00000, val_acc=0.66667
Also val_acc doesn’t seem to change. Here’s my code I use to calculate accuracy:
for epoch in range(epochs):
    loss = n_correct = 0
    model3.train()
    for batch, target in train_loader:
        batch,target = batch.view(-1, n_metabolites).to(device),target.to(device)
        batch = batch.view(-1, n_metabolites).to(device)
        optimizer3.zero_grad()
        outputs = model3(batch)
        train_loss = objective3(outputs, target)
        loss += train_loss.item()
        n_correct += (target == (outputs.reshape(-1) > 0.5).float()).sum()
        train_loss.backward()
        optimizer3.step()

    loss = loss / len(train_loader)
    acc = (n_correct.float() / len(train)).cpu().numpy()
    epoch += 1

    model3.eval();
    val_loss = val_n_correct = 0
    with torch.no_grad():
        for batch, target in val_loader:
            batch,target = batch.view(-1, n_metabolites).to(device),target.to(device)
            batch = batch.view(-1, n_metabolites).to(device)
            outputs = model3(batch)
            val_loss += objective3(outputs, target)
            val_n_correct += (target == (outputs.reshape(-1) > 0.5).float()).sum()
        val_loss = (val_loss / len(val_loader)).cpu().numpy()
        val_acc = (val_n_correct.float() / len(val)).cpu().numpy()

    if (epoch % print_stats_interval) == 0 or epoch == epochs:
        print(f'epoch={epoch:.0f}, loss={loss:.5f}, val_loss={val_loss:.5f}, acc={acc:.5f}, val_acc={val_acc:.5f}')
    log3.append((epoch, loss, val_loss, acc, val_acc))
log3 = pd.DataFrame(log3, columns=['epoch', 'loss', 'val_loss', 'acc', 'val_acc'])

Where am I going wrong in calculating the accuracy?


title: Tensorboard did't open in my Anaconda Prompt

body:
I heard pytorch also can use tensorboard.
after installing tensorboard I tried to open tensorboard in my Anaconda Prompt.
I found this,  but I don't know how to solve this problem. and what is this problem
File "D:\anaconda\lib\site-packages\tensorboard\program.py", line 507, in server_bind
super(WerkzeugServer, self).server_bind()
File "D:\anaconda\lib\http\server.py", line 139, in server_bind
self.server_name = socket.getfqdn(host)
File "D:\anaconda\lib\socket.py", line 676, in getfqdn
hostname, aliases, ipaddrs = gethostbyaddr(name)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte


title: Word count Log file processing

body:
I have a data sets as follows
**Dataset 1:**
Date | Server | Database | Table Name | Query

**Dataset 2**
Server | Database | Table Name | Column name

Problem statement: to count the number of columns from dataset 2  that occur in the query column of dataset 1
expected output as
*Date | Server | Database | Table Name   | Column name  COUNT of occurrence.*

I am able to count the columns but am not able to keep the count to the particular server or database : for eg
if column 1 is in both Sever 1 and server 2 and count is 10 and 15 respectively then the code outputs as
Observed
server 1  column 1 25
server 2  column 1 25

Expected
server 1  column 1 10
server 2  column 1 15


title: On Pytorch, is there implementation difference on Loss function or Optimizer over operating systems?

body:
I'm using Pytorch with following tool and versions:

conda 4.8.3
Python 3.6.10
torch==1.5.1
torchvision==0.6.1

And, trained the same script and dataset with two different OS.
My machine learning task is classification of some Japanese sentences with CNN, RNN.

OS1: macOS Mojave version 10.14.6
OS2: Debian GNU/Linux 10 (buster)

I expected two scripts on different OS will have the same loss convergence. But the result is different, apparently the script on Debian will not converge like on macOS one.

macOS: loss changed from 4.0 to 0.001.
Debian: loss changed from 4.0 to 2.9.

on debian, script will not return more better value. So I stopped training.

My training script is probably common thing. Like following:
while True:
    # retrive dataset with batch size
    fw_args, bw_args = next(train_batcher())
    # call ml models, and get results
    net_out = net(*fw_args)

    # get loss with loss function
    # I'm using `cross_entropy`
    loss = ... some loss function ...
    loss.backward()

    # optimizer = torch.optim.Adam
    optimizer.step()

    # reset grad
    net.zero_grad()
    ...

I guess there are some implementation difference , for example the module to randomize next step ( This is the optimizer ? ).
Is there any suggestion ?
EDIT:
I changed python platform anaconda to pyenv, then now loss changed from 4.0 to 0.02 on Debian. Anaconda causes this ?


title: YCB models in LCP physics engine

body: I am trying to use LCP physics engine(https://github.com/locuslab/lcp-physics) for moving YCB objects. Has anyone done something similar before?


title: Validation accuracy when training and testing within same loop

body:
Since I am training and testing within same loop (for each epoch on training set, network is applied on entire validation set).
Now does it make sense that the highest validation accuracy I get at some instant (nth epoch) be my network's highest accuracy or should I only use the validation accuracy once the graph has settled and weights don't change?


title: How to create a normalize distribution 2-d array

body: I want to create a 2D feature map that have a normalized distribution. As in the figure2d normalized feature map. How can I do that? Pytorch distribution seems doesn't support such function.


title: Model predicting robot offsets not learning

body:
I am currently trying to predict a serial robot's offsets by using a neural network on Pytorch. The method is the following :

Sampling random a vector of joint_angles.
Feeding neural network with it in order to predict corresponding offsets.
Computing FK(joint_angles+offsets) to get estimated_effector_position corresponding to the estimated joints states.
Moving the robot with joint_angles and measure real_effector_position which contains informations about the real offsets
Compute MSE(real_effector_position,estimated_effector_position) and then backprop error.
The network should output offsets that minimize the error with training.

Formally, I'm trying to solve the following optimization problem :
arg min(offsets) ||FK(joint_angles+offsets) - real_effector_position||
I'm using pytroch to implement this but the network doesn't learn and the loss stay the same. Here is my code
robot = BraccioRobot()
loss = 0
model = OffsetMLP(3,64)
opti = optim.Adam(net.parameters(), lr=3e-4)
for i in range(10000):
    desired_joints_state = torch.FloatTensor(np.random.randint(0,180,(10,3))) # Get joint_angles
    offset = model(desired_joints_state/180)  * 15                            # Predict offset
    angles = desired_joints_state + offset                                    # Effective angles
    x,y = robot.forward_kinematic(((angles-90)*np.pi)/180)                    # Compute FK with offsets
    x_t,y_t = robot.offset_forward_kinematic(((desired_joints_state-90)*np.pi)/180) # Real joints_states
    loss = F.mse_loss(x,x_t) + F.mse_loss(y,y_t)                              # Compute loss
    print(loss)
    opti.zero_grad()
    loss.backward()
    opti.step()

the robot class :
class BraccioRobot():
    def __init__(self,) :
        self.proportions = np.array([0.17,0.29,0.285,0.25])
        self.joints_offsets = np.array([10,9,8]) * (np.pi/180.0)

    def forward_kinematic(self,joint_states):
        # Compute standard FK
        parent_angle = 0
        x = torch.zeros(joint_states.shape[0])
        y = torch.zeros(joint_states.shape[0])
        for i in range(joint_states.shape[1]):
            x += self.proportions[i+1] * torch.sin(joint_states[:,i] + parent_angle)
            y += self.proportions[i+1] * torch.cos(joint_states[:,i] + parent_angle)
            parent_angle += joint_states[:,i]
        y += self.proportions[0]
        return x,y # Effector position

    def offset_forward_kinematic(self,joint_states):
        # Compute FK with added offsets
        parent_angle = 0
        x = torch.zeros(joint_states.shape[0])
        y = torch.zeros(joint_states.shape[0])
        for i in range(joint_states.shape[1]):
            x += self.proportions[i+1] * torch.sin(joint_states[:,i] + parent_angle + self.joints_offsets[i])
            y += self.proportions[i+1] * torch.cos(joint_states[:,i] + parent_angle + self.joints_offsets[i])
            parent_angle += joint_states[:,i]
        y += self.proportions[0]
        return x,y # Effector position

and the model :
class OffsetMLP(nn.Module):

    def __init__(self,n_joints,n_hidden):
        super(OffsetMLP,self).__init__()
        self.fc1 = nn.Linear(n_joints,n_hidden)
        self.fc2 = nn.Linear(n_hidden,n_joints)
        self.outputAct = nn.Tanh()

    def forward(self,desired_joints_state):
        offset = self.fc1(desired_joints_state)
        offset = F.relu(offset)
        offset = self.fc2(offset)
        offset = self.outputAct(offset)
        return offset

the offset_forward_kinematic function is used to emulate real robot movement. It just add the offsets that the network must learn. In this case it is 3 simple constant. I'm guessing the error is somewhere in the forward_kinematic function with the gradient maybe not propagating as it should but I can't seem to find a solution. Thanks in advance for any fix.


title: How to stop importing line arguments with imported file

body:
I have two programs train.py and predict.py and I am importing a trained model from train to predict.
Both programs accept line arguments and train runs fine, but when I run predict with its line arguments, an error occurs that I haven't typed in the arguments required by train.py.
How I can solve this?


title: Is it fine if the generator loss in GANs increases while the quality of images increases?

body:
I looked up some implementations where generator loss increases and discriminator loss decreases but images are created successfully and no further workaround is done. Right now, even if i train for 50 epochs the generator loss increases and reaches up to a value of 5. Is this normal?
I tried adding labels( using CGANs) but no noticeable improvement.
Note: Using pytorch and about 10,000 images for training. Is that the problem?


title: What is the best way to represent spatio-temporal image data in a vanilla LSTM

body:
I'm working on video frame segmentation prediction and I want to start using a vanilla LSTM as baseline, I know it won't get good results.
In my current approach is I have the original inputs image frame in the first channel and segmentation frame is the second channel like this Data representation, then a flatten out the input into a 1-D array. How should I represent my spatio-temporal such that it works for a vanilla LSTM?
Here is the snippet to the vanilla LSTM that i'm using in Pytorch
class ImageLSTM(nn.Module):
def __init__(self,  n_inputs:int=49,
                    n_outputs:int=4096,
                    n_hidden:int=256,
                    n_layers:int=1,
                    bidirectional:bool=False):
    """
    Takes a 1D flatten images.
    """
    super(ImageLSTM, self).__init__()
    self.n_inputs   = n_inputs
    self.n_hidden   = n_hidden
    self.n_outputs  = n_outputs
    self.n_layers   = n_layers
    self.bidirectional = bidirectional
    self.lstm       = nn.LSTM(  input_size=self.n_inputs,
                                hidden_size=self.n_hidden,
                                batch_first=False,
                                num_layers=self.n_layers,
                                bidirectional=self.bidirectional)

    if (self.bidirectional):
        self.FC         = nn.Sequential(
                                        nn.Linear(self.n_hidden*2, self.n_outputs),
                                        nn.Dropout(p=0.5),
                                        nn.Sigmoid()
                                            )

    else:
        self.FC         = nn.Sequential(

                                        nn.Linear(self.n_hidden, self.n_outputs),
                                        nn.Dropout(p=0.5),
                                        nn.Sigmoid()
                                        )

def init_hidden(self, x, device=None): # input 4D tensor: (batch size, channels, width, height)
    # initialize the hidden and cell state to zero
    # vectors:(number of layer, sequence length, number of hidden nodes)
    if (self.bidirectional):
        h0 = torch.zeros(2*self.n_layers,  1, self.n_hidden)
        c0 = torch.zeros(2*self.n_layers,  1, self.n_hidden)
    else:
        h0 = torch.zeros(self.n_layers,  1, self.n_hidden)
        c0 = torch.zeros(self.n_layers,  1, self.n_hidden)

    if device is not None:
        h0 = h0.to(device)
        c0 = c0.to(device)
    self.hidden = (h0,c0)

def forward(self, X): # X: tensor of shape (batch_size, channels, width, height)
    # forward propagate LSTM
    lstm_out, self.hidden = self.lstm(X, self.hidden) # lstm_out: tensor of shape (seq_length, batch_size, hidden_size)
    out = self.FC(lstm_out[:, -1, :])
    return out


title: Compute loss between binary matrix and list of 2D coordinates

body:
I have a PyTorch model that produces a list of coordinates, e.g.
>>> coords = model(inputs)
tensor([[1.1000, 3.8000, 0.2000, 4.4000, 2.5000],
        [3.4000, 3.2000, 1.3000, 0.9000, 1.7000]])

where coords[0] are x coordinates and coords[1] are y coordinates, representing pixel values. These pixel values form a binary "image":
>>> indexes = coords.round().long()
>>> torch.sparse.FloatTensor(indexes, torch.ones(indexes.shape[1]), torch.Size([5, 5])).to_dense().T
tensor([[0., 0., 0., 0., 0.],
        [1., 0., 0., 0., 1.],
        [0., 0., 1., 0., 0.],
        [0., 1., 0., 0., 1.],
        [0., 0., 0., 0., 0.]])

I want to optimize my model so that the output matrixes match binary target images.
However, the indexing step is not differentiable. The .long() operation results in a
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

in the backwards step.
Is it possible to do a "2D one-hot encoding" or some other operation that turns floating point coordinate lists into a binary matrix, in a differentiable fashion?


title: Where should i put the input image dimesions in the following architecture in Pytorch?

body:
class Discriminator(nn.Module):
def __init__(self, channels=3):
    super(Discriminator, self).__init__()

    self.channels = channels

    def convlayer(n_input, n_output, k_size=4, stride=2, padding=0, bn=False):
        block = [nn.Conv2d(n_input, n_output, kernel_size=k_size, stride=stride, padding=padding, bias=False)]
        if bn:
            block.append(nn.BatchNorm2d(n_output))
        block.append(nn.LeakyReLU(0.2, inplace=True))
        return block

    self.model = nn.Sequential(
        *convlayer(self.channels, 32, 4, 2, 1),
        *convlayer(32, 64, 4, 2, 1),
        *convlayer(64, 128, 4, 2, 1, bn=True),
        *convlayer(128, 256, 4, 2, 1, bn=True),
        nn.Conv2d(256, 1, 4, 1, 0, bias=False),  # FC with Conv.
    )

def forward(self, imgs):
    logits = self.model(imgs)
    out = torch.sigmoid(logits)

    return out.view(-1,1)

The above architecture is of Discriminator of GAN model, i am little confused as in the first layer
*convlayer(self.channels, 32, 4, 2, 1)

self.channels ,which is 3 (colored image), is passed , I have an input image of 64 * 64 * 3. My first question is where the dimensions of input image are taken care in the above architecture?
I have got this confusion because when i saw the generator architecture ,
class Generator(nn.Module):
def __init__(self, nz=128, channels=3):
    super(Generator, self).__init__()

    self.nz = nz
    self.channels = channels

    def convlayer(n_input, n_output, k_size=4, stride=2, padding=0):
        block = [
            nn.ConvTranspose2d(n_input, n_output, kernel_size=k_size, stride=stride, padding=padding, bias=False),
            nn.BatchNorm2d(n_output),
            nn.ReLU(inplace=True),
        ]
        return block

    self.model = nn.Sequential(
        *convlayer(self.nz, 1024, 4, 1, 0), # Fully connected layer via convolution.
        *convlayer(1024, 512, 4, 2, 1),
        *convlayer(512, 256, 4, 2, 1),
        *convlayer(256, 128, 4, 2, 1),
        *convlayer(128, 64, 4, 2, 1),
        nn.ConvTranspose2d(64, self.channels, 3, 1, 1),

        nn.Tanh()
    )

def forward(self, z):
    z = z.view(-1, self.nz, 1, 1)
    img = self.model(z)
    return img

In the first layer
*convlayer(self.nz, 1024, 4, 1, 0)

they are passing self.nz ,which is 128 random latent points required to generate image of 64 * 64 * 3, as opposed to the above discriminator model where the channels are passed.
My second question is, if i have an image of 300 * 300 * 3, what should i change in my architecture of Discriminator to process the image?
P.S. I am new to Pytorch.


title: Apply a different dense layer for each timestep in Keras

body:
I want to have different dense layer for each of the time steps of  a sequential 3d input: (N, timesteps, features).
In my mind came TimeDistributed, but then recognized that it just train 1 matrix for all timesteps. (I think the pytorch nn.Linear does a similiar job like the keras Dense layer because it trains just a single matrix)
The only thing I found is this here:
How to apply a different dense layer for each timestep in Keras
So, this leads to a question, why is this so "tricky" to achieve above thing? This means my use case is used very rarely?
I found it very normal and natural (?) and interesting because you can learn more complex embeddings of your input?


title: ML model not loading full batch

body:
I tried to build a machine learning model using CIFAR 10 dataset, but I am encountering a bug that my model stops training past i = 78 (looped 78 times, see code for more).
import torch
import torchvision.transforms as transforms
from torchvision.datasets import CIFAR10
from torchvision.transforms import ToTensor
from torch.utils.data.dataloader import DataLoader
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5,
0.5, 0.5))])
classes = ('plane', 'car', 'bird', 'cat','deer', 'dog', 'frog', 'horse', 'ship', 'truck')

train_dataset = CIFAR10(root = './data', train = True, download = True, transform = transform)
train_loader = DataLoader(train_dataset, batch_size = 4, shuffle = True, num_workers = 2)

test_dataset = CIFAR10(root = './data', train = False, download = True, transform = transform)
test_loader = DataLoader(test_dataset, batch_size = 128, shuffle = False, num_workers = 2)
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

optimiser = torch.optim.SGD(model.parameters(), lr = 0.001, momentum=0.9)
loss_fn = nn.CrossEntropyLoss()
for epoch in range(2):
  running_loss = 0
  for i, data in enumerate(test_loader, 0):
    images, labels = data
    outputs = model(images)

    loss = loss_fn(outputs, labels)

    optimiser.zero_grad()
    loss.backward()
    optimiser.step()

    running_loss += loss.item()
    print(i)
    if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0

Sorry, I had to post the entire code because I cannot spot the mistake I made.
Moreover, since I could not make it work, I tried copying the tutorial's exact code, and it works as intended! I am posting that code too below,
import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose(
     [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                      shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                     shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.view(-1, 16 * 5 * 5)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

net = Net()

import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(2):  # loop over the dataset multiple times

running_loss = 0.0
for i, data in enumerate(trainloader, 0):
    # get the inputs; data is a list of [inputs, labels]
    inputs, labels = data

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    # print statistics
    running_loss += loss.item()
    if i % 2000 == 1999:    # print every 2000 mini-batches
        print('[%d, %5d] loss: %.3f' %
              (epoch + 1, i + 1, running_loss / 2000))
        running_loss = 0.0

print('Finished Training')

Please help me find the bug!


title: Deep learning performance degradation according to pytorch version in google colab?

body:
As in the title, if I uploaded the deep learning semantic segmentation network that was currently performing well on the local PC to Google Colab and run the experiment, I experienced a significant decrease in performance.
To solve this problem, I tried to match the PyTorch version on the Local PC. The PyTorch version on my local PC is 1.5+cu101, so I reinstalled it with the following command in Google Colab.
!pip install torch==1.5.1+cu101 torchvision==0.6.1+cu101 -f https://download.pytorch.org/whl/torch_stable.html

However, the problem is still the same, the performance of the network is greatly degraded, and the cause cannot be determined.
Besides the PyTorch version, are there any other factors that affect network performance..?
Network structure and code are the same as Local PC, and random.seed is also the same.


title: Artificial neural network performance degradation in other PCs

body:
I am experimenting with designing a semantic segmentation network using Pytorch. It performs well on my computer. For better performance, we experimented by moving the network to a computer with a lot of GPU capacity. Basically, if you set the same environment for only the versions of Pytorch and torchvision and proceed with the experiment, performance degradation occurs even on the moved PC or Google Colab. I just copied and pasted the code and ran it as a test before doing other experiments.
The network structure is the same, but are there other external factors that will degrade performance? (ex: gpu, ram, etc...)...


title: Best way to get joint probability matrix from categorical data

body:
My goal is to get joint probability (here we use count for example) matrix from data samples. Now I can get the expected result, but I'm wondering how to optimize it. Here is my implementation:
def Fill2DCountTable(arraysList):
    '''
    :param arraysList: List of arrays, length=2
    each array is of shape (k, sampleSize),
        k == 1 (or None. numpy will align it) if it's single variable
        else k for a set of variables of size k
    :return: xyJointCounts, xMarginalCounts, yMarginalCounts
    '''
    jointUniques, jointCounts = np.unique(np.vstack(arraysList), axis=1, return_counts=True)
    _, xReverseIndexs = np.unique(jointUniques[[0]], axis=1, return_inverse=True) ###HIGHLIGHT###
    _, yReverseIndexs = np.unique(jointUniques[[1]], axis=1, return_inverse=True)
    xyJointCounts = np.zeros((xReverseIndexs.max() + 1, yReverseIndexs.max() + 1), dtype=np.int32)
    xyJointCounts[tuple(np.vstack([xReverseIndexs, yReverseIndexs]))] = jointCounts
    xMarginalCounts = np.sum(xyJointCounts, axis=1) ###HIGHLIGHT###
    yMarginalCounts = np.sum(xyJointCounts, axis=0)
    return xyJointCounts, xMarginalCounts, yMarginalCounts

def Fill3DCountTable(arraysList):
    # :param arraysList: List of arrays, length=3
    jointUniques, jointCounts = np.unique(np.vstack(arraysList), axis=1, return_counts=True)
    _, xReverseIndexs = np.unique(jointUniques[[0]], axis=1, return_inverse=True)
    _, yReverseIndexs = np.unique(jointUniques[[1]], axis=1, return_inverse=True)
    _, SReverseIndexs = np.unique(jointUniques[2:], axis=1, return_inverse=True)
    SxyJointCounts = np.zeros((SReverseIndexs.max() + 1, xReverseIndexs.max() + 1, yReverseIndexs.max() + 1), dtype=np.int32)
    SxyJointCounts[tuple(np.vstack([SReverseIndexs, xReverseIndexs, yReverseIndexs]))] = jointCounts
    SMarginalCounts = np.sum(SxyJointCounts, axis=(1, 2))
    SxJointCounts = np.sum(SxyJointCounts, axis=2)
    SyJointCounts = np.sum(SxyJointCounts, axis=1)
    return SxyJointCounts, SMarginalCounts, SxJointCounts, SyJointCounts

My use scenario is to do conditional independence test over variables. SampleSize is usually quite big (~10k) and each variable's categorical cardinality is relatively small (~10). I still find the speed not satisfying.
How to best optimize this code, or even logic outside the code? I may have some thoughts:

The ###HIGHLIGHT### lines. On a single X I may calculate (X;Y1), (Y2;X), (X;Y3|S1)... for many times, so what if I save cache variable's (and conditional set's) {uniqueValue: reversedIndex} dictionary and its marginal count, and then directly get marginalCounts (no need to sum) and replace to get reverseIndexs (no need to unique).
How to further use matrix parallelization to do CITest in batch, i.e. calculate (X;Y|S1), (X;Y|S2), (X;Y|S3)... simultaneously?
Will torch be faster than numpy, on same CPU? Or on GPU?

It's an open question. Thank you for any possible ideas. Big thanks for your help :)
================== A test example is as follows ==================
xs = np.array(  [2, 4, 2, 3, 3, 1, 3, 1, 2, 1]  )
ys = np.array(  [5, 5, 5, 4, 4, 4, 4, 4, 6, 5]  )
Ss = np.array([ [1, 0, 0, 0, 1, 0, 0, 0, 1, 1],
                [1, 1, 1, 0, 1, 0, 1, 0, 1, 0] ])

xyJointCounts, xMarginalCounts, yMarginalCounts = Fill2DCountTable([xs, ys])
SxyJointCounts, SMarginalCounts, SxJointCounts, SyJointCounts = Fill3DCountTable([xs, ys, Ss])

get 2D from (X;Y): xMarginalCounts=[3 3 3 1], yMarginalCounts=[5 4 1], and xyJointCounts (added axes name FYI):
xy| 4 5 6
--|-------
1 | 2 1 1
2 | 0 2 1
3 | 3 0 0
4 | 0 1 0

get 3D from (X;Y|{Z1,Z2}): SxyJointCounts is of shape 4x4x3, where the first 4 means the cardinality of {Z1,Z2} (00, 01, 10, 11 with respective SMarginalCounts=[3 3 1 3]). SxJointCounts is of shape 4x4 and SyJointCounts is of shape 4x3.


title: Text dataset for RNN maxing out memory

body:
I'm still learning how to handle text datasets while also working on learning RNNs and other models. I'm using PyTorch to build an RNN, which as I understand needs a tensor shape of (Batch_size x sequence_size x input_size).
My dataset has total chars of 73461 and a vocab size (input_size) of 52. I'm using a sequence_length of 100 to predict the next char. (Size of [73361 x 100 x 52]).
My attempt was to build a list of the batches and convert that to a numpy array which I can then build to a tensor.
Unfortunately that creates some massive data chunk that eats up all my machine memory and eventually quits with an "unable to allocate X memory" error.
I tried to use memory_profiler to see what was happening and this is the output:
Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
    33  379.828 MiB  379.828 MiB           1   @profile
    34                                         def encode(seq_size):
    35  642.719 MiB    0.000 MiB       73362       for i in range(0, n_chars - seq_size, 1):
    36  642.719 MiB    0.000 MiB       73361           seq_in = text[i:i + seq_size]
    37  642.719 MiB    0.000 MiB       73361           seq_out = text[i + seq_size]
    38                                                 # Make a list item of each batch
    39  642.719 MiB  235.672 MiB     7556183           input_data.append([char_to_int[char] for char in seq_in])
    40  642.719 MiB    2.188 MiB       73361           target_data.append(char_to_int[seq_out])
    41  642.719 MiB    0.000 MiB           1       n_patterns = len(input_data)
    42  866.625 MiB  223.906 MiB           1       X = np.reshape(input_data, (n_patterns, seq_size, 1))
    43
    44  866.625 MiB    0.000 MiB           1       return X, target_data

Here is the code:
with open(file, 'r', encoding='utf-8') as fh:
    text = fh.read().lower()

# Create the mapping
chars = sorted(list(set(text)))
char_to_int = {ch: i for i, ch in enumerate(chars)}

n_chars, n_vocab = len(text), len(chars)  # chars is the input size

print(f'Total Characters: {n_chars}')
print(f'Total Vocab: {n_vocab}')
# Prepare the training data
# Data will consist of len(text) - seq_size batches
# Which in this case will create an input shape of 3231615x100x81
seq_size = 100
input_data = []
target_data = []

# @profile
def encode(seq_size):
    for i in range(0, n_chars - seq_size, 1):
        seq_in = text[i:i + seq_size]
        seq_out = text[i + seq_size]
        # Make a list item of each batch
        input_data.append([char_to_int[char] for char in seq_in])
        target_data.append(char_to_int[seq_out])
    n_patterns = len(input_data)
    X = np.reshape(input_data, (n_patterns, seq_size, 1))

    return X, target_data

# batches
# n_patterns = len(input_data)

# reshape X to be [samples, time steps, num_features]
# X = np.reshape(input_data, (n_patterns, seq_size, 1))
X, target_data = encode(seq_size)

I have a one_hot_encode function later but I can never get to it since this kills it.
Any help is greatly appreciated.


title: sklearn's TfidfVectorizer has unknown type annotation for TorchScript

body:
I am trying to export my Pytorch network using TorchScript, since that seemed like the most straight forward method to deploy a trained network (only for inference, no more training). However, the network uses TfidfVectorizer which generates the following error.
ValueError: Unknown type annotation: 'TfidfVectorizer(analyzer='word', binary=False, decode_error='strict',
                dtype=<class 'numpy.float64'>, encoding='utf-8',
                input='content', lowercase=True, max_df=1.0, max_features=None,
                min_df=1, ngram_range=(1, 1), norm='l2', preprocessor=None,
                smooth_idf=True, stop_words=None, strip_accents=None,
                sublinear_tf=False, token_pattern='(?u)\\b\\w\\w+\\b',
                tokenizer=None, use_idf=True, vocabulary=None)'

The code:
# Before there are some data pre-processing which creates an TfidfVectorizer
# variable called tfidf_vec, labels and some responses.

class Chatbot(nn.Module):
  tfidf_vec: TfidfVectorizer()

  def __init__(self, tfidf_vec, labels, responses):
    super(Chatbot, self).__init__()
    self.tfidf_vec = tfidf_vec
    self.labels = labels
    self.responses = responses

    self.lin11 = nn.Linear(len(tfidf_vec.vocabulary_), 50)
    self.lin12 = nn.Linear(50, len(labels))
    self.sigmoid = nn.Sigmoid()
    self.softmax = nn.Softmax(dim=1)

  def forward(self, input):
    input = self.lin11(input)
    input = self.sigmoid(input)
    input = self.lin12(input)
    input = self.softmax(input)
    return input

  #@torch.jit.export
  def TFIDF_input(self, sentence): #TF-IDF embedd the input based on the amazon corpus/document
    sentence = [sentence]
    sentence=self.tfidf_vec.transform(sentence)
    sentence=sentence.todense()
    sentence=torch.from_numpy(np.array(sentence)).type(torch.FloatTensor)
    return sentence

  @torch.jit.export
  def chat(self, input):
    input = str(input)
    input = input.replace('!', '.')
    input = input.replace('?', '.')
    sentence_list = input.split('. ')
    return_response = ""
    for i in range(len(sentence_list)):
        tfidf_sentence = self.TFIDF_input(sentence_list[i])
        pred = self.forward(tfidf_sentence)
        item = torch.argmax(pred).item()
        label = self.labels[item]
        response = self.responses[label]
        response = response[random.randint(0, len(response)-1)]
        return_response = return_response + response + ". "
    return return_response

network = Chatbot(tfidf_vec, labels, respons)
# Train the network
response = network.chat("Hello. can you help me")
print(response)

script_module = torch.jit.script(network)
script_module.save("model.pt")

Have I overlooked something? Is this even a viable strategy for exporting the network? Should I use a completely different strategy? I haven't found much good resources on how to deploy this kind of NLP networks.


title: Type error while training a model from pre-trained Bert model

body:
I am getting following error while running script of training my Vqa model from pretrained Bert model , my dataset is DocVqa.  following is error, why it is showing? environment is pytorch , transformers
File "C:\Users\HP\anaconda3\envs\vqa\lib\site-packages\pytorch_pretrained_bert\modeling.py", line 600, in from_pretrained
model = cls(config, *inputs, **kwargs)
TypeError: init() got multiple values for argument 'config'


title: Loading pandas df to pytorch dataset loader results in "nan" training loss if df is shuffled

body:
I am trying to train an autoencoder in pytorch.
When I am loading the data into a dataloader object like this, the autoencoder training works:
total_set = Dataset(df.iloc[:15000,1:], df.iloc[:15000,0])

However, when I load the data after shuffling or sampling from the dataset the autoencoder train loss is always nan.
df = df.sample(frac = 1).reset_index(drop=True)
df = df[(df != 0).all(1)].dropna().reset_index(drop=True)
total_set = Dataset(df.iloc[:15000,1:], df.iloc[:15000,0])

I made sure I removed the 0 or nan values from the training set. Am I missing something here?

df.info() before and after shuffling:
before:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 513408 entries, 0 to 513407
Columns: 673 entries, 0 to 672
dtypes: float64(672), object(1)
memory usage: 2.6+ GB
None
after:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 513408 entries, 0 to 513407
Columns: 673 entries, 0 to 672
dtypes: float64(672), object(1)
memory usage: 2.6+ GB
None


title: Torch is installed but I'm unable to import it in a computer vision python project in Jupyter notebook

body: I'm working on a computer vision project using detectron2. I'm having trouble installing torch or importing it into my jupyter notebook. I'm working on a mac that is running MACOS Catalina, Python3 version in 3.8.2 and I'm using Anaconda for my Development environment. In the screenshot, it says torch is installed but it throws error while importing it


title: How to impose a Loss to prevent Outliers (Unsupervised Landmark Detection)

body:
I am trying to predict full body landmarks in an unsupervised fashion. I am currently training on the Human3.6M Dataset and the results look as follows:

As can be seen, the results are quite good - however, the Model also learns two Landmarks in order to predict the background. Currently the reconstruction Loss is only focused around the predicted landmarks. I am searching for an idea, to prevent the model to place predict "outlier landmarks". Maybe there is an algorithmic approach, that restricts landmarks, if they are too far away from the currently detected landmarks? On the other hand, this is a very strong restriction and will easily hinder the model from learning useful landmarks as well. Has anyone worked on a similar problem and can give a smart advice? :) Thanks a lot!


title: Compatibility between cudatoolkit and cuda

body:
Background
I am trying to run some old PyTorch code. Back to that time, the authors were still using libraries of following versions (written in requirements.txt).
torch==1.0.1
torchtext==0.3.1
...

Question
I am trying to create a conda environment to install these libraries. By invoking the following command
/usr/local/cuda/bin/nvcc --version

it turns out that my machine is currently installed with CUDA 11.0. However, according to doc (which provides the following command), it seems that the highest compatible version is CUDA 10.
conda install pytorch==1.0.1 torchvision==0.2.2 cudatoolkit=10.0 -c pytorch

Does this mean I need to downgrade my system-wide CUDA installation to CUDA 10?
What I have Done
I have already read this post but still confused. More specifically, I don't understand what is the relation between cudatoolkit and system-wide CUDA installation. Some guesses

cudatoolkit does nothing except some precompiling, so eventually PyTorch version still has to be compatible with corresponding CUDA version. In my example, I will have to downgrade my CUDA anyway.

cudatoolkit provides a interface that could help PyTorch to use CUDA that might be incompatible. For example, "officially" PyTorch needs to use CUDA 10.0, but system is only installed with CUDA 11.0, then cudatoolkit could somehow utilize the system-wide CUDA 11.0 to run the PyTorch code that actually needs CUDA 10.0. In my example, I do not have to downgrade my CUDA thanks to the assistance provided by cudatoolkit.


title: Webcam and PyTorch mage classifier integration

body:
I am using following codes to classify defected items from real time webcame, But I am confused about it. Sometime it is performing good, sometime bad. Classification result is not consistant.  Could you please take a look on the code and help me to figure out any error:
#This is the Label
Labels = ['Perfect','Defected']

# Let's preprocess the inputted frame

data_transforms = torchvision.transforms.Compose([
    torchvision.transforms.Resize(size=(352, 288)),
    torchvision.transforms.RandomHorizontalFlip(),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])
])

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
# Load the model and set in eval
resnet18 = torchvision.models.resnet18(pretrained=True)
resnet18.fc = torch.nn.Sequential(nn.Linear(resnet18.fc.in_features,512),
                                  nn.ReLU(),
                                  nn.Dropout(),
                                  nn.Linear(512, 2))

resnet18.load_state_dict(torch.load('defect_classifier_demo.pt', map_location=torch.device('cpu')))
resnet18.eval()

#Set the Webcam
def Webcam_720p():
    cap.set(3,1280)
    cap.set(4,720)

def preprocess(image):
    image = PIL.Image.fromarray(image) #Webcam frames are numpy array format
                                       #Therefore transform back to PIL image
    print(image)
    image = data_transforms(image)
    image = image.float()
    #image = Variable(image, requires_autograd=True)
    image = image.cpu()
    image = image.unsqueeze(0) #I don't know for sure but Resnet-50 model seems to only
                               #accpets 4-D Vector Tensor so we need to squeeze another
    return image                            #dimension out of our 3-D vector Tensor

#Let's start the real-time classification process!

cap = cv2.VideoCapture(0) #Set the webcam
Webcam_720p()

fps = 0
show_score = 0
show_res = 'Nothing'
sequence = 0

while True:
    ret, frame = cap.read() #Capture each frame

    if fps == 4:
        image        = frame[100:450,150:570]
        image_data   = preprocess(image)
        #print(image_data)
        prediction   = resnet18(image_data)[0]
        torch.nn.functional.softmax(prediction, dim=0)
        #result,score = argmax(prediction)
        prediction = prediction.cpu().detach().numpy()
        predicted_class_index = np.argmax(prediction)
        predicted_class_name = Labels[predicted_class_index]
        print(prediction)
        print(predicted_class_index)
        print(predicted_class_name)
        #print(result)
        #print(score)
        #prediction = np.vectorize(prediction)
        fps = 0
        if prediction.any():
            show_res  = predicted_class_name
            show_score= prediction
        else:
            show_res   = "Nothing"
            show_score = prediction

    fps += 1
    cv2.putText(frame, '%s' %(show_res),(950,250), cv2.FONT_HERSHEY_SIMPLEX, 2, (255,255,255), 3)
    #cv2.putText(frame, '(score = %.5f)' %(show_score), (950,300), cv2.FONT_HERSHEY_SIMPLEX, 1,(255,255,255),2)
    cv2.rectangle(frame,(400,150),(900,550), (250,0,0), 2)
    cv2.imshow("ASL SIGN DETECTER", frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyWindow("ASL SIGN DETECTER") ```

I will really appreciate your help.


title: Need help passing in 2D Matrix to Conv1D layer and outputting a softmax probability

body:
Hi how's it going? I'm trying to build a model that takes in a 2D Matrix as a single sample and outputs the row index that's the best action by using softmax.
This is what I have so far:

names = ['Bob','Henry','Mike','Phil']

max_squat = [300,400,200,100]

max_bench = [200,100,225,100]

max_deadlift = [600,400,300,225]

strongest_worker_df = pd.DataFrame({'Name':names,'Max_Squat':max_squat,'Max_Bench':max_bench,'Max_Deadlift':max_deadlift})

class Policy(nn.Module):

   def __init__(self):
       super(Policy, self).__init__()
       self.layer1 = torch.nn.Conv1d(in_channels=4, out_channels=4, kernel_size=3, stride=1)

   def forward(self, x):

       x = self.layer1(x)
       x = F.softmax(x,dim=1)

       return x

   def act(self, state):

       state = state.float()
       value = self.forward(state)

       return value

policy = Policy()
result = policy.act(input_torch)

The result shape is torch.Size([1,4,1]). How do I get this to output a column vector instead?
Also, is Conv1D with kernel size equal to number of features per row the most logical approach to this input state representation?


title: How to use PyTorch DataParallel with non-tensor input

body:
I'm building an image segmentation model using the PyTorch Mask-RCNN, and I'd like to parallelize it across GPUs. However, the PyTorch MaskRCNN requires that the target is a dictionary of {key: Tensor} with specific keys rather than a single Tensor.
On a single GPU it runs fine, but when wrapped with torch.nn.DataParallel it returns this error:
chunk expects at least a 1-dimensional tensor, which I expect happens because it's given a list of dictionaries instead of a Tensor.
The code relevant to the error is given here, where the error occurs on the last line.
    for images, targets in data_loader:

        images = list(image.to(device) for image in images)
        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

        loss_dict = model(images, targets)

Is there an established way of dealing with this problem?


title: Pytorch multi task autoencoder

body:
I have a question, let’s say I have an input image which I would then create many versions of it using autoencoder and multi task learning
Now for testing, I want the user to give let’s say a task number so the output would be only that version of image. Is it possible to do it with pytorch ?
Thanks in advance


title: Using PyTorch DataLoader makes me have the same losses, loops without any progress

body:
So my issue is that when not using DataLoader, just creating 1000 epochs and doing the learning, the results are ok, and the losses drop to ~0.2. However, when trying to use DataLoader, the output is:
|batch|index|loss|
8 0 0.6232748031616211
..
8 23 0.6030591726303101
9 0 0.5626393556594849
9 1 0.6434788703918457
..
9 20 0.6232720017433167

I think it means that it doesn't learn on anything new. I do not know how to fix it, though. Doesn't matter if the no. epochs is 10 or 100. Loss is always ~0.6. Here is a code fragment without the model:
xy = np.loadtxt('diabetes.csv', delimiter=',', dtype=np.float32)
x_data = (torch.from_numpy(xy[:, 0:-1]))
y_data = (torch.from_numpy(xy[:, [-1]]))

class DiabetesDataset(Dataset):

    def __init__(self):
        xy = np.loadtxt('diabetes.csv', delimiter=',', dtype=np.float32)
        self.len = (xy.shape[0])
        self.x_data = (torch.from_numpy(xy[:, 0:-1]))
        self.y_data = (torch.from_numpy(xy[:, [-1]]))

    def __getitem__(self, index):
        return self.x_data[index], self.y_data[index]

    def __len__(self):
        return self.len

dataset = DiabetesDataset()
train_loader = DataLoader(dataset=dataset,
                          batch_size=32,
                          shuffle=True,
                          num_workers=0)

model = Model()

criterion = torch.nn.BCELoss(reduction='mean')
optimizer = torch.optim.Rprop(model.parameters(), lr=0.01)

for epoch in range(10):
    for i, data in enumerate(train_loader, 0):
        inputs, labels = data

        inputs, labels = Variable(inputs), Variable(labels)

        y_pred = model(inputs)

        loss = criterion(y_pred, labels)
        print(epoch, i, float(loss.data))
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

What might be the cause? Thanks!


title: Layer data types mismatch in keras operation

body:
Primarily I'm trying to convert a PyTorch model to Tensorflow.
It's this model taken from this repo and the model architecture you can find it here.
Following this blog and this Readme, I am trying to convert the model.
 This is the colab notebook I'm working on to convert my model. I'm using Pytorch2Keras to convert my model.
Error
I keep getting this error while converting the model. I think it's the issue based on the datatype inside the onnx2keras layer. It's highly difficult to understand that library to find where that layer is and fixing it.

TypeError: Input 'y' of 'Mul' Op has type int32 that does not match type float32 of argument 'x'.

`  HELP: Is there any way to bypass this issue apart from manually converting the layer dtype. Or if you know which layer and how to convert the dtype. let me know.


title: How to calculate torch.mean with default value for empty slice?

body:
What's the correct way to calculate torch.mean such that in case of an empty slice, some default value would be returned?
import torch

a = torch.arange(5, dtype=torch.float)
mask = torch.ones(5, dtype=torch.bool)
mask[2] = 0
mask_empty = torch.zeros(5, dtype=torch.bool)

a_masked = a[mask]
a_empty = a[mask_empty]
m = torch.mean(a_masked)
m_empty = torch.mean(a_empty)

print(m)
print(m_empty)

current output:
tensor(2.)
tensor(nan)

required output:
tensor(2.)
tensor(0.)

I know I can do
m_empty = torch.zeros_like(m_empty) if torch.isnan(m_empty) else m_empty

but this seems like the wrong way to go, as it forces more cpu-gpu communication that I would like to avoid if possible.
Also, this makes the code less clean than if something like
m_empty = torch.mean(a_empty, default=0) were available.

Is there a clean, Pythonic way of achieving default values for empty slices of torch.mean [or other standard functions]?


title: Initialise pytorch convolution layer with my own values

body:
I want to know if there's a way to initialize pytorch convolution filter with my own values.
eg, I have a tuple [-0.8423,  0.3778][-3.1070, -2.6518] and I want to initialize a 2X2 filter with these values, how do I do that? I looked up some answers but they were mostly using torch normal distribution and others. Nothing that I can't specify.
Let me know.


I am getting this error after running my code:
"RuntimeError: mean is not implemented for type torch.ByteTensor"?
Do anyone know what I am doing wrong here?
accuracy = torch.mean(output)


title: Parallel learning of several networks simultaneously

body:
I am working on testing diffrent methods of hyperparameter optimalization. First i want to test grid search, this method requries to train same type of network, using same dataset, same optimzer but setting diffrent parameters of this optimazer. Is it possible to train many models of networks at once. Currently I training single network, then I set diffrent paramaters and I am traning again, but this tooks a lot of time.
This is code of my current training process, where "myparam" is an array of diffrent parameters. There is 81 different combination so, the main loop is done 81 times.
for x in range(81):
    torch.cuda.empty_cache()
    model = models.alexnet(pretrained=True)
    model.classifier[6] = torch.nn.Linear(model.classifier[6].in_features, 9)
    device = torch.device('cuda')
    model = model.to(device)
    optimizer = optim.SGD(model.parameters(), lr = myparam[(x,0)], momentum=myparam[(x,1)])
    print("Attempt: ",x)

    train_loader = torch.utils.data.DataLoader(
        train_dataset,
        batch_size=10,
        shuffle=True,
        num_workers=2
    )

    test_loader = torch.utils.data.DataLoader(
        test_dataset,
        batch_size=10,
        shuffle=True,
        num_workers=2
    )
    for epoch in range(NUM_EPOCHS):

        for images, labels in iter(train_loader):
            images = images.to(device)
            labels = labels.to(device)
            optimizer.zero_grad()
            outputs = model(images)
            loss = F.cross_entropy(outputs, labels)
            loss.backward()
            optimizer.step()

        test_error_count = 0.0
        for images, labels in iter(test_loader):
            images = images.to(device)
            labels = labels.to(device)
            outputs = model(images)
            test_error_count += float(torch.sum(torch.abs(labels - outputs.argmax(1))))

        test_accuracy = 1.0 - float(test_error_count) / float(len(test_dataset))


title: How to take gradient in Python Pytorch autograd returns Nonetype

body:
Please do ctrl-f main , that's where the script runs & calls other members
Calling a sample function in my algorithm main class which uses a gradient descent type algorithm (unadjusted Langevin). I have put the function in another class as then I can pass it without calling and then differentiate?
But, I am not completely sure whats the best way to do the gradient loss step. the current code is telling me that torch.autograd.grad(input_tensor,output_tensor) is returning Nonetype.
I have taken the norm of the two tensors & used the function I passed in to call it. I am not sure entirely how to do
import torch
from precondSGLD import pSGLD
import copy
class UnadjustedLangevin(object):
def init(self, x, func, step_size, iteration_limit):
super(UnadjustedLangevin, self).init()
        self.func = func
        self.step_size = step_size
        self.counter = 0.0
        self.x = x
        self.x.requires_grad = True
        self.iteration_limit = iteration_limit
        self.samples = []

    #pull one sample x_k1 from initial sample x_k
    def sample(self,alpha):
        #output = self.R.matmul(self.normal.sample()) + self.mu
        #output = 0

        #need to take -log of fx_output as that's f
        fx_output = torch.Tensor(self.func(self.x))
        fx_output.requires_grad = True

        x_flat = torch.norm(self.x)
        fx_flat = torch.norm(fx_output)

        print(x_flat)
        print(fx_flat)

        grad_f_at_x = torch.autograd.grad(x_flat, fx_flat, allow_unused=True)

        loss_term = torch.Tensor(self.step_size*grad_f_at_x)
        z_k = torch.normal(torch.zeros_like(self.x), torch.eye(2))
        x_k1 = self.x - loss_term + torch.sqrt(2*self.step_size)*z_k

        self.samples.append(x_k1)

        self.counter += 1
        return self.samples
'''

'''
import torch
from sampling_algs import *
import numpy as np
from scipy.linalg import logm
import matplotlib.pyplot as plt
import copy
np.random.seed(14)
torch.manual_seed(14)

##This is the main script

torch.set_default_tensor_type('torch.FloatTensor') #cuda not available
#torch.set_default_tensor_type('torch.cuda.FloatTensor') #cuda available

class GaussianDistribution(object):
    def __init__(self, mean, cov_matrix):
        super(GaussianDistribution, self).__init__()

        self.mu = mean
        self.cov = cov_matrix
        self.precision = torch.inverse(cov_matrix)

        self.R = torch.cholesky(self.cov)
        self.normal = torch.distributions.normal.Normal(torch.zeros_like(mean),
            torch.ones_like(mean))

    def pdf_func1(self, alpha):
        #mixed Gaussian
        alpha = torch.Tensor(alpha)
        #print(torch.zeros_like(alpha))
        loss_val = (torch.normal(-torch.zeros_like(alpha), torch.eye(2)) +
        torch.normal(torch.zeros_like(alpha), torch.eye(2)))
        return 0.5*loss_val
        #return 0.5*(((x - mean).T).matmul(self.precision)).matmul(x - self.mean)

    def sample(self):
        #print(self.normal.sample())
        return self.normal.sample() + self.mu

#MAIN
if __name__ == '__main__':
    #working in n=2
    dim = 2

    #############################
    #Log Concave Test Cases
    #alpha=torch.Tensor([0.1,0.1])
    #alpha=torch.Tensor([0.3,0.3])
    alpha=torch.Tensor([0.5,0.5]) #log concave value of a mag<1

    #Non-Log Concave Test Cases
    #alpha=torch.Tensor([1,1]) #non log concave value of a mag>1
    #alpha=torch.Tensor([2,2])
    #alpha=torch.Tensor([3,3])
    ##############################
    gaussian_dist = GaussianDistribution(alpha, torch.eye(dim))
    step_size = 1
    p0 = torch.normal(torch.zeros_like(alpha), torch.eye(2))

    ##RUN ULA algorithm. should output an array of values to plot
    unadjusted_langevin = UnadjustedLangevin(p0, gaussian_dist.pdf_func1, step_size, iteration_limit=int(1e4))

    hist_samples = []
    loss_log = []

    #Collect 10,000 samples
    for j in range(int(1e4)):
        est= unadjusted_langevin.sample(alpha)
        if j%3 == 0:
            hist_samples.append(est.cpu().numpy())

    est_samples = np.array(hist_samples)[200:]
    num_samples = est_samples.shape[0]

    #TODO: change gaussian_dist here to the mixture Gaussian in problem 9
    num_samples = 10000
    true_samples = np.zeros([num_samples, 2])
    for j in range(num_samples):
        true_samples[j, :] = gaussian_dist.sample().cpu().numpy()

'''


title: When to use `on_fit_start` vs `setup` in pytorch lightning?

body:
Pytorch lightning models have two distinct hooks:
From the doc:

setup - Called at the beginning of fit and test.
on_fit_start - Called at the very beginning of fit. If on DDP it is called on every process.

I fail to understand the differences, and when to use either.
I also noticed some class variables are still not defined on setup, such as self.datamodule.

What are the intended use cases for setup vs on_fit_start?


title: Is it possible to combine multiple .stl files to form 3d object?

body: I am currently working on creation of 3d models with picture and I succeed in creating one face my question is obviously there are multiple faces of for example CPU so it is possible to combine them and create a proper 3d object?Output of .stl file


title: CUDNN_STATUS_NOT_INITIALIZED when installing pytorch with pip but not with conda

body:
I am using a T4 GPU from AWS (g4dn.xlarge) and I have NVIDIA-SMI 440.33.01 as verified by running nvidia-smi.
I am using conda. If I create a new environment and install pytorch with pip install pytorch=1.8, I get a CUDNN error:
$ CUDNN_LOGINFO_DBG=1 CUDNN_LOGDEST_DBG=stdout python test.py
cuda:0

I! CuDNN (v7605) function cudnnCreate() called:
i! Time: 2021-03-25T11:45:59.463353 (0d+0h+0m+4s since start)
i! Process=14769; Thread=14769; GPU=NULL; Handle=NULL; StreamId=NULL.

Traceback (most recent call last):
  File "test.py", line 44, in <module>
    output = net(t)
  File "/home/ubuntu/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "test.py", line 20, in forward
    x = self.conv1(x)
  File "/home/ubuntu/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/miniconda3/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/ubuntu/miniconda3/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 395, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

However, if I create another environment but instead of using pip I use conda install pytorch=1.8.0, then it works without problems.
Any clue on what might be the cause?
Thanks
P.S. This is what I ran to setup to install cuda and cudnn on the instance:
# Install cudatoolkit 10.2
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo apt-key add /var/cuda-repo-10-2-local-10.2.89-440.33.01/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

echo "export PATH=/usr/local/cuda-10.2/bin${PATH:+:${PATH}}" >> .bashrc
echo "export LD_LIBRARY_PATH=/usr/local/cuda-10.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" >> .bashrc

# Install CuDNN 7 and NCCL 2
wget https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
sudo dpkg -i nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb

sudo apt update
sudo apt install -y libcudnn7=7.6.5.32-1+cuda10.2 libcudnn7-dev=7.6.5.32-1+cuda10.2

sudo apt autoremove
sudo apt upgrade

sudo ln -s /usr/lib/x86_64-linux-gnu/libcudnn.so.7 /usr/local/cuda-10.2/lib64/


title: How to get the partial derivative of probability to input in pyTorch?

body:
I want to generate attack samples via the following steps:

Find a pre-trained CNN classification model, whose input is X and output is P(y|X), and the most possible result of X is y.

I want to input X' and get y_fool, where X' is not far away from X and y_fool is not equal to y

The steps for getting X' is:enter image description here

How can I get the partial derivative described in the image?

Here is my code but I got None: (The model is Vgg16)
x = torch.autograd.Variable(image, requires_grad=True)
output = model(image)
prob = nn.functional.softmax(output[0], dim=0)

prob.backward(torch.ones(prob.size()))
print(x.grad)

How should I modify my codes? Could someone help me? I would be absolutely grateful.


title: backward failed when i use a for circulate in rnn model

body:
I tried to build a RNN model using Pytorch.
class Rnn(nn.Module):
    def __init__(self,input_size,output_size,hidden_size):
        super().__init__()
        self.hidden_layer=torch.Tensor(1,hidden_size))
        self.U=nn.Linear(input_size,hidden_size)
        self.V=nn.Linear(hidden_size,output_size)
        self.W=nn.Linear(hidden_size,hidden_size)
        self.Softmax=nn.Softmax()
        self.ReLU=nn.ReLU()
    def forward(self,x):
        for i in range(len(x)):
            self.hidden_layer=self.ReLU(self.U(x[i])+self.W(self.hidden_layer))
            output=self.Softmax(self.ReLU(self.V(self.hidden_layer)))
        return output

the train_step code is
        output=model(input)
        loss=criterion(output,label)
        print(loss)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

and the error is
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed.

That is very confused. After thinking and try, i change the hidden_layer's definition position.
class Rnn(nn.Module):
def __init__(self,input_size,output_size,hidden_size):
    super().__init__()
    self.U=nn.Linear(input_size,hidden_size)
    self.V=nn.Linear(hidden_size,output_size)
    self.W=nn.Linear(hidden_size,hidden_size)
    self.Softmax=nn.Softmax()
    self.ReLU=nn.ReLU()
def forward(self,x,hidden_layer):
    for i in range(len(x)):
        hidden_layer=self.ReLU(self.U(x[i])+self.W(hidden_layer))
        output=self.Softmax(self.ReLU(self.V(hidden_layer)))
    return output,hidden_layer

and the train code define a hidden_layer
        hidden_layer=torch.zeros(1,500)
        for i in range(len(sentence)):
            sen=torch.zeros(1,10000)
            sen[0][sentence[i]]=1
            output,hidden_layer=model(sen,hidden_layer)
        loss=criterion(output,label)
        print(loss)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

it works!
So, why treat the hidden_layer as model input but not class build in variable can solve the problem?


title: Convolution with different kernel for each pixel

body:
For example, I have a tensor
import torch
x = torch.tensor([0, 1, 2, 3, 4])

and I have some kernels
k = torch.tensor([[0, 0.4, 0.6], [0.1, 0.3, 0.6], [0.9, 0.05, 0]])

I want to do the convolution of x but kernels (k) are different for each element of x. The answer should be:
y = torch.tensor([1.6, 2.5, 1.95])

I get this answer in such a way:
1.6 = 0 * 0 + 1 * 0.4 + 2 * 0.6
2.5 = 1 * 0.1 + 2 * 0.3 + 3 * 0.6

I can do the simple for-loop but it's very inefficient.


title: Pytorch Computational efficiency with zeros

body:
Recently I realized that you can implement multihead attention without doing any concatenation if you combined all the matrices in a particular way. It involves some linear algebra tricks like stacking marices A,B,C like
[A 0 0
 0 B 0
 0 0 C]

Where those 0s are matrices of the appropriate dimension.
I know pytorch's implementation of multi-head attention is probably better than what I'll write but assuming I have the option to either 1) use a for loop to compute all the head attentions before concatenation vs 2) getting rid of concatenation and the for loop by making some large block matrices with a number of zeros, which one would be expected to run faster?
I think it comes down to if pytorch doesn't worry about computing gradients after there's multiplication by zero...


title: Need help in understanding feature extraction using transfer learning with Pytorch

body:
I am looking for help in understanding the following behavior of feature extraction from the pretrained model.
Output1:
def get_net():
   model = bninception(pretrained="imagenet")
   model.gp = nn.AdaptiveAvgPool2d(1)
   model.conv = nn.Conv2d(config.channels, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3))
   model.last = nn.Sequential(nn.BatchNorm1d(1024),nn.Dropout(0.5),nn.Linear(1024, config.num_classes))
   return model

model = get_net()
model.cuda()
output1 =model(images)

Output2:
I am after features obtained at model.conv, therefore i removed the later section as follows.
def get_net():
    model = bninception(pretrained="imagenet")
    model.gp = nn.AdaptiveAvgPool2d(1)
    model.conv = nn.Conv2d(config.channels, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3))
    return model

model = get_net()
model.cuda()
features = model(images)

and got the features for N images as : [N, 1000]. I fed these features into the following as follows
class ManualLinearRegression(nn.Module):
    def __init__(self):
       super(ManualLinearRegression,self).__init__()
            #self.linear = nn.Linear(1000, 28)
            self.linear = nn.Sequential(
                    nn.BatchNorm1d(1000),#1024
                    nn.Dropout(0.5),
                    nn.Linear(1000, config.num_classes),
                )

     def forward(self, x):
            return self.linear(x)

model = ManualLinearRegression().cuda()
output2 = model(images)

I was expecting Output1 almost similar to Output2. But it is a lot different. On computation of F1 score, output1 is 0.40 while Output2 is 0.09.
I think i am falling short in understanding some major aspect here. May I know what am I doing wrong here.
Just to add on, ultimately I want to extract features from convolutional layer, apply oversampling and perform classification.
Thank you.


title: Simple Pytorch model for histogram matrching does not update its parameters

body:
I am currently trying to fit an extremely simple model which basically should find an optimal histogram for histogram matching. I wrote a super simple model with only a Parameter object which I use directly:
import torch.nn.functional as F

class AutoHist(pl.LightningModule):
    def __init__(self, channel=1, bins=255):
        super().__init__()
        self.hist = torch.nn.Parameter(torch.rand((1, channel, bins), requires_grad=True))
        self.eps = 1e-5

    def b_distance(self, h1, h2):
        distance = 1
        distance -= 1/(torch.sqrt(torch.mean(h1, axis=2)*torch.mean(h2, axis=2)*h1.size(2)**2))
        distance *= torch.sum(torch.sqrt(h1*h2 + self.eps),axis=2)
        return torch.sqrt(distance + self.eps)

    def training_step(self, batch, batch_idx):
        # training_step defined the train loop.
        # It is independent of forward
        x, y = batch
        hist = self.hist / self.hist.sum()
        distances = self.b_distance(hist,x)
        loss = F.binary_cross_entropy(distances[:,0], y)
        return loss

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
        return optimizer

But for some reason the backprop does not go through and the parameter does not get updated. Does somebody have an idea where the problem may lie? The gradients actually exist and do change from batch to batch. I use pytorch lightning to remove boilerplate code, but the crux should lie in the code I wrote.


title: AssertionError: size of input tensor and input format are different. - Tensorboard add image

body:
I have tried to add an image to the tensorboard.
img = Image.open(image_path)

img = (np.asarray(img) / 255.0)
img = torch.from_numpy(img).float()
img = img.cuda().permute(2, 0, 1)
img = img.cuda().unsqueeze(0)

writer.add_image('Original', img)

but there is an error.

tensor shape: {}, input_format: {}".format(tensor.shape, input_format)
AssertionError: size of input tensor and input format are different.         tensor shape: (1, 3, 420, 629), input_format: CHW


title: Save model from running training - Pytorch

body: Is there a way to save a model from a running script, when not mentioned in the code? I am running a model since 5 days on GPU and save the model whenever a new highscore in the metric is achieved. Unfortunately, the dataset contains many mislabeled samples. Thus, the score is declining at the moment although from a qualitative perspective the model is still making progress. As a result, the model is not saved anymore. Is there a way to save the current model or do I have to run it again?


title: Repeated Branches in TorchScript Model Export

body:
I'm trying to export a PyTorch model to TorchScript via scripting and I am stuck. I've created a toy class to showcase the issue:
import torch
from torch import nn

class SadModule(nn.Module):
    """Takes a (*, 2) input and runs it thorugh a linear layer. Can optionally
    use a skip connection. The usage of the skip connection or not is an
    architectural choice.

    """
    def __init__(self, use_skip: bool):
        nn.Module.__init__(self)
        self.use_skip = use_skip
        self.layer = nn.Linear(2, 2)

    def forward(self, x):
        if self.use_skip:
            x_input = x
        x = self.layer(x)
        if self.use_skip:
            x = x + x_input
        return x

It basically consists of only a linear layer and an optional skip connection. If I try to script the model using
mod1 = SadModule(False)
scripted_mod1 = torch.jit.script(mod)

I get the following error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-10-a7ebc7af32c7> in <module>
----> 1 scripted_mod1 = torch.jit.script(mod)

~/Software/miniconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/jit/_script.py in script(obj, optimize, _frames_up, _rcb)
    895
    896     if isinstance(obj, torch.nn.Module):
--> 897         return torch.jit._recursive.create_script_module(
    898             obj, torch.jit._recursive.infer_methods_to_compile
    899         )

~/Software/miniconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/jit/_recursive.py in create_script_module(nn_module, stubs_fn, share_types)
    350     check_module_initialized(nn_module)
    351     concrete_type = get_module_concrete_type(nn_module, share_types)
--> 352     return create_script_module_impl(nn_module, concrete_type, stubs_fn)
    353
    354 def create_script_module_impl(nn_module, concrete_type, stubs_fn):

~/Software/miniconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
    408     # Compile methods if necessary
    409     if concrete_type not in concrete_type_store.methods_compiled:
--> 410         create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
    411         torch._C._run_emit_module_hook(cpp_module)
    412         concrete_type_store.methods_compiled.add(concrete_type)

~/Software/miniconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/jit/_recursive.py in create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
    302     property_rcbs = [p.resolution_callback for p in property_stubs]
    303
--> 304     concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
    305
    306

RuntimeError:

x_input is not defined in the false branch:
  File "<ipython-input-7-d08ed7ff42ec>", line 12
    def forward(self, x):
        if self.use_skip:
        ~~~~~~~~~~~~~~~~~
            x_input = x
            ~~~~~~~~~~~ <--- HERE
        x = self.layer(x)
        if self.use_skip:
and was used here:
  File "<ipython-input-7-d08ed7ff42ec>", line 16
        x = self.layer(x)
        if self.use_skip:
            x = x + x_input
                    ~~~~~~~ <--- HERE
        return x

So, basically TorchScript isn't able to recognise that for mod1 the True branch of either if statement won't ever be used. Moreover, if we create an instance that actually uses the skip connection,
mod2 = SadModule(True)
scripted_mod2 = torch.jit.script(mod2)

we will get another error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-21-b5ca61d8aa73> in <module>
----> 1 scripted_mod2 = torch.jit.script(mod2)

~/Software/miniconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/jit/_script.py in script(obj, optimize, _frames_up, _rcb)
    895
    896     if isinstance(obj, torch.nn.Module):
--> 897         return torch.jit._recursive.create_script_module(
    898             obj, torch.jit._recursive.infer_methods_to_compile
    899         )

~/Software/miniconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/jit/_recursive.py in create_script_module(nn_module, stubs_fn, share_types)
    350     check_module_initialized(nn_module)
    351     concrete_type = get_module_concrete_type(nn_module, share_types)
--> 352     return create_script_module_impl(nn_module, concrete_type, stubs_fn)
    353
    354 def create_script_module_impl(nn_module, concrete_type, stubs_fn):

~/Software/miniconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
    408     # Compile methods if necessary
    409     if concrete_type not in concrete_type_store.methods_compiled:
--> 410         create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
    411         torch._C._run_emit_module_hook(cpp_module)
    412         concrete_type_store.methods_compiled.add(concrete_type)

~/Software/miniconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/jit/_recursive.py in create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
    302     property_rcbs = [p.resolution_callback for p in property_stubs]
    303
--> 304     concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
    305
    306

RuntimeError:

x_input is not defined in the false branch:
  File "<ipython-input-18-ac8b9713c789>", line 17
    def forward(self, x):
        if self.use_skip:
        ~~~~~~~~~~~~~~~~~
            x_input = x
            ~~~~~~~~~~~ <--- HERE
        x = self.layer(x)
        if self.use_skip:
and was used here:
  File "<ipython-input-18-ac8b9713c789>", line 21
        x = self.layer(x)
        if self.use_skip:
            x = x + x_input
                    ~~~~~~~ <--- HERE
        return x

So in this case TorchScript doesn't understand that both ifs will always be true and that in fact x_input is well defined.
To avoid the issue, I could split the class into two subclasses, as in:
class SadModuleNoSkip(nn.Module):
    """Takes a (*, 2) input and runs it thorugh a linear layer. Can optionally
    use a skip connection. The usage of the skip connection or not is an
    architectural choice.

    """
    def __init__(self):
        nn.Module.__init__(self)
        self.layer = nn.Linear(2, 2)

    def forward(self, x):
        x = self.layer(x)
        return x

class SadModuleSkip(nn.Module):
    """Takes a (*, 2) input and runs it thorugh a linear layer. Can optionally
    use a skip connection. The usage of the skip connection or not is an
    architectural choice.

    """
    def __init__(self):
        nn.Module.__init__(self)
        self.layer = nn.Linear(2, 2)

    def forward(self, x):
        x_input = x
        x = self.layer(x)
        x = x + x_input
        return x

However, I am working on a huge code base and I would have to repeat the process for many classes, which is time consuming and could introduce bugs. Moreover, often the modules I'm working on are huge convolutional nets and the ifs just control the presence of an additional batch normalization. It seems to me undesirable to have to classes that are identical in 99% of the blocks, save for a single batch norm layer.
Is there a way in which I can help TorchScript with its handling of branches?
Edit: added a minimum viable example.
Update: doesn't work even if I type hint use_skip as constant
from typing import Final

class SadModule(nn.Module):
    use_skip: Final[bool]
    ...


title: Accumulating and plotting regression results in PyTorch Lightning

body:
I'm using PyTorch lightning to develop a convolutional NN for a regression problem. I've partitioned my data into train, validation and test dataloaders, and written the base code for the LightningModule training and validation steps. Here is what I have so far:
class Model(pl.LightningModule):
    def __init__(self,lr=3e-5):
        super(Model, self).__init__()
        self.save_hyperparameters('lr')

        self.cnn_layers = nn.Sequential(
            nn.Conv3d(4, 8, 5, stride=1, padding=2),
            nn.ReLU(inplace=True),
            nn.Conv3d(8, 2, 5, stride=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv3d(2, 2, 3, stride=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv3d(2,1,3,stride=3, padding=0)
        )

    def forward(self, x):
        x = self.cnn_layers(x).squeeze()
        return x

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), self.hparams.lr)
        return optimizer

    def L1_loss(self,pred,actual):
        return F.l1_loss(pred, actual)

    def L2_loss(self,pred,actual):
        return F.mse_loss(pred,actual)

    def training_step(self,batch,batch_idx):
        x,y = batch
        pred = self.forward(x)
        loss = self.L2_loss(pred,y)
        err = self.L1_loss(pred,y)
        res = {'loss':loss,'train_err':err}
        self.log_dict(res, on_step=True, on_epoch=True, prog_bar=True, logger=True)
        return res

    def validation_step(self,batch,batch_idx):
        x,y = batch
        pred = self.forward(x)
        loss = self.L2_loss(pred,y)
        err = self.L1_loss(pred,y)
        res = {'val_loss':loss,'val_err':err}
        self.log_dict(res, on_step=True, on_epoch=True, prog_bar=True, logger=True)
        return res

    def training_epoch_end(self,outputs):
        avg_loss = torch.stack([x['loss'] for x in outputs]).mean()
        avg_err = torch.stack([x['train_err'] for x in outputs]).mean()
        self.logger.experiment.add_scalar("Train/loss",avg_loss,self.current_epoch)
        self.logger.experiment.add_scalar("Train/err",avg_err,self.current_epoch)

    def validation_epoch_end(self,outputs):
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        avg_err = torch.stack([x['val_err'] for x in outputs]).mean()
        self.logger.experiment.add_scalar("Validation/loss",avg_loss,self.current_epoch)
        self.logger.experiment.add_scalar("Validation/err",avg_err,self.current_epoch)

After the training phase, I would like to freeze the model, and collect test predictions over all three datasets. In the end, I would like to make a parity scatter plot of predicted vs actual values for all points (distinguished with color by the set they belong to). One of the approaches I investigated was to modify init to also declare 6 separate lists, as shown here:
class Model(pl.LightningModule):
    def __init__(self,lr=3e-5):
        super(Model, self).__init__()
        self.save_hyperparameters('lr')

        self.cnn_layers = nn.Sequential(
            nn.Conv3d(4, 8, 5, stride=1, padding=2),
            nn.ReLU(inplace=True),
            nn.Conv3d(8, 2, 5, stride=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv3d(2, 2, 3, stride=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv3d(2,1,3,stride=3, padding=0)
        )
        self.train_actual,self.train_pred = [],[]
        self.val_actual,self.val_pred = [],[]
        self.test_actual,self.test_pred = [],[]

I then attempted to run trainer.test() on the three dataloaders separately, but I'm not sure how to specify which pair of lists to populate with each call to test(). I'm also not sure if this is the optimal way to log final regression results. Any insights on this would be appreciated.


title: When I using jupyter notebook to train a model, the "ipykernel_launcher.py: error: unrecognized arguments" existed

body:
I'm learning to use jupyter notebook to train a model with Pytorch. But there exited a problem here when I run the cell. What should I do?
This is the code I think may existing problem.
parser = argparse.ArgumentParser(description='PyTorch Training')
parser.add_argument('--batch_size', default=1, type=int, metavar='BT',
                    help='batch size')
# =============== optimizer
parser.add_argument('--lr', '--learning_rate', default=1e-6, type=float,
                    metavar='LR', help='initial learning rate')
parser.add_argument('--momentum', default=0.9, type=float, metavar='M',
                    help='momentum')
parser.add_argument('--weight_decay', '--wd', default=2e-4, type=float,
                    metavar='W', help='default weight decay')
parser.add_argument('--stepsize', default=3, type=int,
                    metavar='SS', help='learning rate step size')
parser.add_argument('--gamma', '--gm', default=0.1, type=float,
                    help='learning rate decay parameter: Gamma')
parser.add_argument('--maxepoch', default=30, type=int, metavar='N',
                    help='number of total epochs to run')
parser.add_argument('--itersize', default=10, type=int,
                    metavar='IS', help='iter size')
# =============== misc
parser.add_argument('--start_epoch', default=0, type=int, metavar='N',
                    help='manual epoch number (useful on restarts)')
parser.add_argument('--print_freq', '-p', default=1000, type=int,
                    metavar='N', help='print frequency (default: 50)')
parser.add_argument('--gpu', default='0', type=str,
                    help='GPU ID')
parser.add_argument('--resume', default='', type=str, metavar='PATH',
                    help='path to latest checkpoint (default: none)')
parser.add_argument('--tmp', help='tmp folder', default='tmp/RCF')
# ================ dataset
parser.add_argument('--dataset', help='root folder of dataset', default='data/HED-BSDS_PASCAL')
args = parser.parse_args()

The error is :
"usage: ipykernel_launcher.py [-h] [--batch_size BT] [--lr LR] [--momentum M]
                             [--weight_decay W] [--stepsize SS]
                             [--gamma GAMMA] [--maxepoch N] [--itersize IS]
                             [--start_epoch N] [--print_freq N] [--gpu GPU]
                             [--resume PATH] [--tmp TMP] [--dataset DATASET]
ipykernel_launcher.py: error: unrecognized arguments: -f C:\Users\59863\AppData\Roaming\jupyter\runtime\kernel-4a7fc5df-ea52-4c9e-93a9-f466074bfc90.json"


title: I don't understand dimentions transformation in counting loss function in Pytorch

body:
I'm new to neural networks, so I'm sorry if I describe something incorrectly. Now I'm trying to build a model to generate text by character using Pytorch. The model is 5 character fixedwindow/ It seems like my model is OK, but after I'm computing loss function and training my model I get a result with just the same letter repeating over and over
    èJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
    %333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333
    őJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ

And I'm expecting something sentence-alike.
I'm not yet very good with tensor dimensions, so I guess I turned my tensor incorrectly somewhere.
So, my model is:
    self.pad = nn.ZeroPad2d((5, 0, 0, 0))
    self.emb = nn.Embedding(n_tokens, emb_size) #max_norm=True
    self.conv_1 = nn.Conv1d(emb_size, hid_size, kernel_size=6, stride=1, padding=0)
    self.fc = nn.Linear(hid_size, n_tokens)

And here I'm turning it twice, because it doesn't work otherwise:
    input = self.pad(input)
    input = self.emb(input)
    input = input.permute(0, 2, 1)
    input = self.conv_1(input)
    input = input.permute(0, 2, 1)
    input = self.fc(input)

As the output I get the result in the shape of [batch_size, sequence_size, number_of_tokens]
When I computing loss I do not really understand how my tensor dimensions must look like before entering the loss function. So right now it goes like this:
    input = torch.as_tensor(input_ix, dtype=torch.int64)
    logits = model(input)
    reference_answers = input

    mask = torch.from_numpy(compute_mask(input).to(torch.int32).cpu().numpy()) #mask is needed because I'm making same sequence length for each example.

    criterion = nn.CrossEntropyLoss()
    probs = nn.Softmax(dim=1)
    softmax_output = probs(logits)
    mask_ = mask.unsqueeze(-1).expand(softmax_output.size())
    softmax_masked = softmax_output * mask_
    softmax_masked = softmax_masked.permute(0,2,1)
    loss = criterion(softmax_masked, reference_answers)
    return loss

I feel like I'm doing everything incorrectly with loss. I'm just trying to understand, how it must work in the real life, because I have no example.


title: MLP in pytorch for persistent modeling

body:
I am using pytorch to implement MLP for a simple task. Let's say I get a time series like x_1,...,x_10, and I want to predict x_11. The problem is that my model quickly stops learning and the performance can not even beat use x_10 as the prediction for x_11. I'm using Adam.
I guess it might relate to initialization, learning rate, batch normalization... Could someone give me some guidance about how to improve it.