title: pytorch, How can i make same size of tensor model(x) and answer(x)?

body:
I'm try to make a simple linear model to predict parameters of formula.

y = 3*x1 + x2 - 2*x3

Unfortunately, there are some problem when i try to compute loss.

def answer(x):
return 3 * x[:,0] + x[:,1] - 2 * x[:,2]

def loss_f(x):

y = answer(x)
y_hat = model(x)
loss = ((y - y_hat).pow(2)).sum() / x.size(0)

return loss

When i set batch_size = 3, the size of each result is different

x = torch.randn(3,3)
answer(x)
tensor([ 2.0201, -3.8354,  2.0059])

model(x)
tensor([[ 0.2085],
    [-0.0670],
    [-1.3635]], grad_fn=<ThAddmmBackward>)

answer(x.data).size()
torch.Size([3])
model(x.data).size()
torch.Size([3, 1])

I think the broadcast applied automatically.

loss = ((y - y_hat).pow(2)).sum() / x.size(0)

How can i make same size of two tensors? Thanks
This is my code

import torch
import torch.nn as nn
import torch.optim as optim

class model(nn.Module):

    def __init__(self, input_size, output_size):
        super(model, self).__init__()

        self.linear = nn.Linear(input_size, output_size)

        def forward(self, x):

        y = self.linear(x)

        return y

model = model(3,1)
optimizer = optim.SGD(model.parameters(), lr = 0.001, momentum=0.1)

print('Parameters : ')
for p in model.parameters():
    print(p)

print('')
print('Optimizer : ')
print(optimizer)

def generate_data(batch_size):
    x = torch.randn(batch_size, 3)

    return x

def answer(x):

    return 3 * x[:,0] + x[:,1] - 2 * x[:,2]

def loss_f(x):

    y = answer(x)
    y_hat = model(x)

    loss = ((y - y_hat).pow(2)).sum() / x.size(0)

    return loss

x = torch.randn(3,3)
print(x)
x = torch.FloatTensor(x)

batch_size = 3
epoch_n = 1000
iter_n = 100

for epoch in range(epoch_n):
    avg_loss = 0

    for i in range(iter_n):
        x = torch.randn(batch_size, 3)

        optimizer.zero_grad()
        loss = loss_f(x.data)
        loss.backward()
        optimizer.step()

        avg_loss += loss

    avg_loss = avg_loss / iter_n

    x_valid = torch.FloatTensor([[1,2,3]])
    y_valid = answer(x_valid)

    model.eval()
    y_hat = model(x_valid)
    model.train()

    print(avg_loss, y_valid.data[0], y_hat.data[0])

    if avg_loss < 0.001:
        break


title: Problems passing tensor to linear layer - Pytorch

body:
I'm trying to build a neural net however I can't figure out where I'm going wrong with the max pooling layer.

    self.embed1 = nn.Embedding(256, 8)
    self.conv_1 = nn.Conv2d(1, 64, (7,8), padding = (0,0))
    self.fc1 = nn.Linear(64, 2)

def forward(self,x):

    import pdb; pdb.set_trace()
    x = self.embed1(x) #input a tensor of ([1,217]) output size: ([1, 217, 8])
    x = x.unsqueeze(0) #conv lay needs a tensor of size (B x C x W x H) so unsqueeze here to make ([1, 1, 217, 8])
    x = self.conv_1(x) #creates 64 filter of size (7, 8).Outputs ([1, 64, 211, 1]) as 6 values lost due to not padding.

    x = torch.max(x,0) #returning max over the 64 columns. This returns a tuple of length 2 with 64 values in each att, the max val and indices.
    x = x[0] #I only need the max values. This returns a tensor of size ([64, 211, 1])
    x = x.squeeze(2) #linear layer only wants the number of inputs and number of outputs so I squeeze the tensor to ([64, 211])
    x = self.fc1(x) #Error Size mismatch (M1: [64 x 211] M2: [64 x 2])

I understand why the linear layer isn't accepting 211 however I don't understand why my tensor after maxing over the columns isn't 64 x 2.


title: Pytorch: Using Adam Optimizer for a custom variable/Model with requires_grad?

body:
Have the following model as follows:
import torch.nn as nn
class DefaultModel(nn.Module):
    def __init__(self, guess, K):
        super(DefaultModel, self).__init__()

        #guess = [-0.7541, -0.044,  0.0916,  1.5914, -0.0017,  1.4991]
        self.T = torch.tensor(guess).unsqueeze(0)
        self.T.requires_grad = True

        self.K = K

    def forward(self, datapoints, draw):
        Mat = pose_vec2mat(self.T).squeeze(0)
        loss = stuff(datapoints)
        return loss

Normally with hand coded gradient descent I would do something like this:
model = DefaultModel(guess, K)
gamma = torch.tensor([5e-5, 5e-5, 5e-5, 2e-5, 2e-5, 2e-5])
for i in range(0, 10000):

    loss = model(datapoints, draw=[1, 0,5,6])
    loss.backward()

    with torch.no_grad():
        model.T = model.T - gamma * model.T.grad
    model.T.requires_grad = True

However, if I want to do this using Adam Optimizer:
model = DefaultModel(guess, K)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)

It crashes with ValueError: optimizer got an empty parameter list How can I get my variable T which requires grad in the parameter list?


I was making a CNN with Pytorch but encountered this error
Here is a model that I used, it is a very simple CNN. I also tried to use a pretrained model and that was also throwing the same error.
class ResNet18(nn.Module):
    def __init__(self):
        super(ResNet18, self).__init__()
        self.network = nn.Sequential(
            nn.Conv2d(3, 16, kernel_size=(3,3), stride=1),
            nn.MaxPool2d(kernel_size=(2,2)),
            nn.BatchNorm2d(16),
            nn.ReLU(),

            nn.Conv2d(16, 32, kernel_size=(3,3), stride=1),
            nn.MaxPool2d(kernel_size=(2,2)),
            nn.BatchNorm2d(32),
            nn.ReLU(),

            nn.Flatten(),
            nn.Linear(16*16*32, 64),
            nn.ReLU(),
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Linear(32, 7),
            )

    def forward(self, xb):
        return self.network(xb)

And here is my Dataset class:
class CurrencyDataset(Dataset):
    def __init__(self, DATA_DIR, transforms=None):
        self.DATA_DIR = DATA_DIR
        self.transforms = transforms

    def __len__(self):
        return len(self.DATA_DIR)

    def __getitem__(self, idx):
        img, label = ImageFolder(self.DATA_DIR, transform=self.transforms)[idx]
        return img, label

And training Loop:
def train(model, optimizer, dataloader, EPOCHS):

    model.train()
    final_loss = 0
    for _ in range(EPOCHS):
        for img, label in tqdm(enumerate(dataloader)):
            optimizer.zero_grad()
            out = model(img)
            loss = F.cross_entropy(out, label)

            loss.backward()
            optimizer.step()
            final_loss+=loss.item()
    return final_loss/len(dataloader)

Here is the complete error, I have also used a ToTensor and resize transforms but nothing helped:
0it [00:00, ?it/s]
Traceback (most recent call last):
  File "c:/Users/LENOVO/Desktop/Dev/Currency-Classification/Using Pytorch/engine.py", line 52, in <module>
    train(model, optimizer, train_dl, 20)
  File "c:/Users/LENOVO/Desktop/Dev/Currency-Classification/Using Pytorch/engine.py", line 44, in train
    out = model(img)
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "c:\Users\LENOVO\Desktop\Dev\Currency-Classification\Using Pytorch\model.py", line 29, in forward
    return self.network(xb)
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\container.py", line 100, in forward
    input = module(input)
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\conv.py", line 349, in forward
    return self._conv_forward(input, self.weight)
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\conv.py", line 346, in _conv_forward
    self.padding, self.dilation, self.groups)
TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not int


title: How reshape 3D tensor of shape (3, 1, 2) to (1, 2, 3)

body:
I intended

(Pdb) aa = torch.tensor([[[1,2]], [[3,4]], [[5,6]]])
(Pdb) aa.shape
torch.Size([3, 1, 2])
(Pdb) aa
tensor([[[ 1,  2]],

        [[ 3,  4]],

        [[ 5,  6]]])
(Pdb) aa.view(1, 2, 3)
tensor([[[ 1,  2,  3],
         [ 4,  5,  6]]])

But what I really want is

tensor([[[ 1,  3,  5],
         [ 2,  4,  6]]])

How?

In my application, I am trying to transform my input data of shape (L, N, C_in) to (N, C_in, L) in order to use Conv1d, where

L: sequence length
N: batch size
C_in: number of channels in the input, I also understand it as the dimensionality of the input at each position of a sequence.

I am also wondering the input of Conv1d doesn't have the same input shape as GRU?


I built this acoustic model with features dim = [1124823,13] and labels dim = [1124823,1] and I split both to train, test, and dev. The problem that when I try to run my model I get this error

RuntimeError: expected scalar type Long but found Int in
  loss = criterion(outputs, y_train)

import torch
import torch.nn as nn
from fela import feat, labels
from Dataloader import train_loader, test_loader, X_train, X_test, X_val, y_train, y_test, y_val

################################################################################################
input_size = 13
hidden1_size = 13
hidden2_size = 128
hidden3_size = 64
output_size = 50

################################################################################################

class DNN(nn.Module):
   def __init__(self, input_size, hidden2_size, hidden3_size, output_size):
       super(DNN, self).__init__()
       self.fc1 = nn.Linear(input_size, hidden1_size)
       self.relu1 = nn.ReLU()
       self.fc2 = nn.Linear(hidden1_size, hidden2_size)
       self.relu2 = nn.ReLU()
       self.fc3 = nn.Linear(hidden2_size, hidden3_size)
       self.relu3 = nn.ReLU()
       self.fc4 = nn.Linear(hidden3_size, output_size)
       self.relu4 = nn.ReLU()

   def forward(self, x):
       out = self.fc1(x)
       out = self.relu1(out)
       out = self.fc2(out)
       out = self.relu2(out)
       out = self.fc3(out)
       out = self.relu3(out)
       out = self.fc4(out)
       out = self.relu4(out)
       return out
################################################################################################
# Instantiate the model
batch_size = 50
n_iterations = 50
no_epochs = 80
model = DNN(input_size, hidden2_size, hidden3_size, output_size)

################################################################################################
# Define the loss criterion and optimizer
criterion = nn.CrossEntropyLoss()
learning_rate = 0.01
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
print(model)
########################################################################################################################
# train the network
iter = 0
for epoch in range(no_epochs):
   for i, (X_train, y_train) in enumerate(train_loader):
       optimizer.zero_grad()
       outputs = model(X_train)
       loss = criterion(outputs, torch.max(labels, 1)[1])
       loss.backward()
       optimizer.step()
       iter += 1
       if iter % 500 == 0:
           correct = 0
           total = 0
           for X_test, y_test in test_loader:
               outputs = model(X_test)
               _, predicted = torch.max(outputs.data, 1)
               total += labels.size(0)
               correct += (predicted == labels).sum()
           accuracy = 100 * correct / total
           print(iter, loss.data[0], accuracy)


title: How can I assign a list to a torch.tensor?

body:
Assume that I have a list [(0,0),(1,0),(1,1)] and another list [4,5,6] and a matrix X with size is (3,2). I am trying to assign the list to the matrix like X[0,0] = 4, X[1,0] = 5 and X[1,1] = 6. But it seems like I have a problem of assigning a list to tensor
x = torch.zeros(3,2)
indices = [(0,0),(1,0),(1,1)]
value = [4,5,6]
x[indices] = values

Error:
TypeError                                 Traceback (most recent call last)
<ipython-input-7-dec4e6a479a5> in <module>

   4 indices = [(0, 0), (1, 0), (1, 1)]

   5 values = [4, 5, 6]

----> 6 x[indices] = values

TypeError: can't assign a list to a torch.FloatTensor


title: PyTorch Total CUDA time

body:
Autograd profiler is a handy tool to measure the execution time in PyTorch as it is shown in what follows:

import torch
import torchvision.models as models

model = models.densenet121(pretrained=True)
x = torch.randn((1, 3, 224, 224), requires_grad=True)

with torch.autograd.profiler.profile(use_cuda=True) as prof:
    model(x)
print(prof)

The output looks like this:

-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------
Name                                        CPU time        CUDA time            Calls        CPU total       CUDA total
-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------
conv2d                                    9976.544us       9972.736us                1       9976.544us       9972.736us
convolution                               9958.778us       9958.400us                1       9958.778us       9958.400us
_convolution                              9946.712us       9947.136us                1       9946.712us       9947.136us
contiguous                                   6.692us          6.976us                1          6.692us          6.976us
empty                                       11.927us         12.032us                1         11.927us         12.032us

Which will include many lines. My questions are:

1) How can I use autograd profiler to get the entire CUDA time? (i.e., sum of CUDA time column)

2) Is there any solution to use it pragmatically? For example, prof[0].CUDA_Time?


What is the difference between mm and spmm in Pytorch? I know that spmm does sparse matrix multiplication, but what exactly does that mean? Why would you ever choose mm as opposed to spmm if spmm has the potential to save space?


title: How to prune a Detectron2 model?

body:
I'm a teacher who is studying computer vision for months. I was very excited when I was able to train my first object detection model using Detectron2's Faster R-CNN model. And it works like a charm! Super cool!
But the problem is that, in order to increase the accuracy, I used the largest model in the model zoo.
Now I want to deploy this as something people can use to ease their job. But, the model is so large that it takes ~10 seconds to infer a single image on my CPU which is Intel i7-8750h.
Therefore, it's really difficult to deploy this model even on a regular cloud server. I need to use either GPU servers or latest model CPU servers which are really expensive and I'm not sure if I can even compensate for server expenses for months.
I need to make it smaller and faster for deployment.
So, yesterday I found that there's something like pruning the model!! I was very excited (since I'm not a computer or data scientists, don't blame me (((: )
I read official pruning documentation of PyTorch, but it's really difficult for me to understand.
I found global pruning is of the easiest one to do.
But the problem is, I have no idea what parameters should I write to prune.
Like I said, I used Faster R-CNN X-101 model. I have it as "model_final.pth". And it uses Base RCNN FPN.yaml and its meta architecture is "GeneralizedRCNN".
It seems like an easy configuration to do. But like I said, since it's not my field it's very hard for a person like me.
I'd be more than happy if you could help me on this step by step.
I'm leaving my cfg.yaml which I used training the model and I saved it using "dump" method in Detectron2 config class just in case. Here's the Drive link.
Thank you very much in advance.


title: Huggingface BERT SequenceClassification - ValueError: too many values to unpack (expected 2)

body:
The class labels for the two class model is 0, 1, 0, 0, etc. There is only one
label per input sequence. The labels are set in a python list and converted to
torch.Tensor. (reading from a csv file - straight forward).
its value: tensor(0)
its shape is shown as: torch.Size([])
when i create the batch, etc. and set the parameters in the model (huggingface sequenceclassification model)
for each batch iteration,
   b_input_ids = batch[0].to(device)
   b_input_mask = batch[1].to(device)
   b_token_type_ids = batch[2].to(device)
   b_labels = batch[3].to(device)

and then set to the model (as part of training)
loss, logits = model(b_input_ids,
                     attention_mask=b_input_mask,
                     token_type_ids=b_token_type_ids,
                     labels=b_labels)

The error message is as follows:
    ValueError

        Traceback (most recent call last)

        <ipython-input-17-b3a36df2f659> in <module>()
             89                              attention_mask=b_input_mask,
             90                              token_type_ids=b_token_type_ids,
        ---> 91                              labels=b_labels)
             92
             93         # Accumulate the training loss over all of the batches so that we can

ValueError: too many values to unpack (expected 2)

This was code I was able to use and train a few days ago. (I have a saved model).
The error message (I assume ) probably has something to do with the size etc.
Has anyone hit a similar problem? thanks.


title: Celery: make worker and child processes ( or threads ) share GPU-allocated data

body:
In the example below, I allocate a model and tensor on GPU s.t. it is globally visible. I dont' want the child processes to have to copy the data and I don't see why it needs to, since passing the address in the device memory space would be sufficient. ( throughput is extremely important, and if each child copies the data whenever a request is received, it takes too much time ). However, when I send a request, I can see ( using nvidia-smi ) that the child allocates its own d0_model and d0_in_tensor.

app = Celery('tasks', backend='rpc://', broker='pyamqp://guest@localhost//')

# allocate model and weight on the GPU
d0_model = nn.Conv1d(in_channel, 10, kernel_size).to("cuda:0")
d0_in_tensor = torch.randn(batch, in_channel, out_channel).to("cuda:0")

# task using the model and weight
@app.task
def torch_conv1d(gpu):
    if gpu == 0:
        return d0_in_tensor

I don't want my child to have to copy the worker's GPU data, as it puts unnecessary overhead and allocation time.

Is there to make the worker process ( or threads, if this is not possible using processes ) and child processes share GPU allocated data?
If answer to my first question is "no", then is there a way to pre-allocate each child s.t. they do not have to spend time allocating the model and tensor on GPU ( each child would have it's unique copy of model and tensor, but they would have it allocated when they first spawn and would just hold onto it until a request is received) ?


title: Get probability from predicted class pytorch

body:
I have a network like this for image classification, for now i have 2 classes:

class ActionNet(Module):

    def __init__(self, num_class=4):

        super(ActionNet, self).__init__()

        self.cnn_layer = Sequential(

            #conv1

            Conv2d(in_channels=1, out_channels=32, kernel_size=1, bias=False),

            BatchNorm2d(32),

            PReLU(num_parameters=32),

            MaxPool2d(kernel_size=3),

            #conv2

            Conv2d(in_channels=32, out_channels=64, kernel_size=1, bias=False),

            BatchNorm2d(64),

            PReLU(num_parameters=64),

            MaxPool2d(kernel_size=3),

            #flatten

            Flatten(),

            Linear(576, 128),

            BatchNorm1d(128),

            ReLU(inplace=True),

            Dropout(0.5),

            Linear(128, num_class)

        )

    def forward(self, x):

        x = self.cnn_layer(x)

        return x

then after i training my network, i predict the image using this code:

def predict_image(image):

    input = torch.from_numpy(image)

    input = input.unsqueeze(1)

    input = input.to(device)

    output = model(input)

    index = output.data.cpu().numpy().argmax()

    return index

how do i get all class probability of prediction image? so the result would be array of index with probaility like 0=0.1, 1=0.7


title: How to convert PyTorch tensor to image and send it with flask?

body:
I have a neural network in PyTorch which gives an image as a Tensor. I have converted it to a numpy array and then followed the explanation here to send that image to the html. The problem is that it's always black.
This is my code in the PyTorch:
def getLinearImage():
  with torch.no_grad():
      test_z = torch.randn(1, 100).to(device)
      generated = G(test_z)
      generated = generated.cpu()
      numpy = generated[0].view(64, 64).numpy()
      numpu = (numpy + 1) * 128
      return numpy

This is the code in the flask where arr is the returned value from getLinearImage()
def getImage(arr):
    img = Image.fromarray(arr.astype("uint8"))
    file_object = io.BytesIO()

    img.save(file_object, "PNG")
    file_object.seek(0)

    return send_file(file_object, mimetype="image/PNG")

If I open a static image and I send it to getImage() it works but won't work with the generated one. In the html I call it like:
<img src="/getLinearImage" alt="User Image" height="100px" width="100px">


title: Pytorch linear/affine layer parameters confusing

body:
I'm on the Pytorch documentation (https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html) and I'm not really understanding why they are making the the affine layer (16 * 6 * 6, 120).  I understand that the last outputs from the convolution layer were 16 and the output here is 120, but even with their annotation, I'm not understanding where the 6 * 6 comes from.  Can someone please explain?

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()
print(net)


title: `RuntimeError: DataLoader worker (pid (s) 11343, 11344) exited unexpectedly`

body:
https://github.com/python-engineer/pytorch-chatbot#usage
import numpy as np
import random
import json

import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader

from nltk_utils import bag_of_words, tokenize, stem
from model import NeuralNet

with open('intents.json', 'r') as f:
    intents = json.load(f)

all_words = []
tags = []
xy = []
# loop through each sentence in our intents patterns
for intent in intents['intents']:
    tag = intent['tag']
    # add to tag list
    tags.append(tag)
    for pattern in intent['patterns']:
        # tokenize each word in the sentence
        w = tokenize(pattern)
        # add to our words list
        all_words.extend(w)
        # add to xy pair
        xy.append((w, tag))

# stem and lower each word
ignore_words = ['?', '.', '!']
all_words = [stem(w) for w in all_words if w not in ignore_words]
# remove duplicates and sort
all_words = sorted(set(all_words))
tags = sorted(set(tags))

print(len(xy), "patterns")
print(len(tags), "tags:", tags)
print(len(all_words), "unique stemmed words:", all_words)

# create training data
X_train = []
y_train = []
for (pattern_sentence, tag) in xy:
    # X: bag of words for each pattern_sentence
    bag = bag_of_words(pattern_sentence, all_words)
    X_train.append(bag)
    # y: PyTorch CrossEntropyLoss needs only class labels, not one-hot
    label = tags.index(tag)
    y_train.append(label)

X_train = np.array(X_train)
y_train = np.array(y_train)

# Hyper-parameters
num_epochs = 1000
batch_size = 8
learning_rate = 0.001
input_size = len(X_train[0])
hidden_size = 8
output_size = len(tags)
print(input_size, output_size)

class ChatDataset(Dataset):

    def __init__(self):
        self.n_samples = len(X_train)
        self.x_data = X_train
        self.y_data = y_train

    # support indexing such that dataset[i] can be used to get i-th sample
    def __getitem__(self, index):
        return self.x_data[index], self.y_data[index]

    # we can call len(dataset) to return the size
    def __len__(self):
        return self.n_samples

dataset = ChatDataset()
train_loader = DataLoader(dataset=dataset,
                          batch_size=batch_size,
                          shuffle=True,
                          num_workers=2)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = NeuralNet(input_size, hidden_size, output_size).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
for epoch in range(num_epochs):
    for (words, labels) in train_loader:
        words = words.to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(words)
        # if y would be one-hot, we must apply
        # labels = torch.max(labels, 1)[1]
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    if (epoch+1) % 100 == 0:
        print (f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')

print(f'final loss: {loss.item():.4f}')

data = {
"model_state": model.state_dict(),
"input_size": input_size,
"hidden_size": hidden_size,
"output_size": output_size,
"all_words": all_words,
"tags": tags
}

FILE = "data.pth"
torch.save(data, FILE)

print(f'training complete. file saved to {FILE}')

when i run the train.py file (up)
RuntimeError: DataLoader worker (pid (s) 11343, 11344) exited unexpectedly
I am getting error.
After train.py, I need to run chat.py file. However, when I run the chat.py file, I get the following error.
FileNotFoundError: [Errno 2] No such file or directory: 'data.pth'
First of all, does anyone know the reason for this error I got on train.py? I would be glad if you help.


title: Why bottleneck structure is slower and costs more memory when training a network comparing to standard convolution?

body:
I'm using vnet to train a model. I want to train the model faster with less memory. So I replaced the standard 3x3 convolution to a combination of [1x1, 3x3 , 1x1] convolutions. The first 1x1 conv will reduce the channel to 1/N to reduce memory cost. The code is as follows.

The first two classes is bottleneck structure and standard convolution. When I replace the standard convolution to bottleneck structure, although the model size and flops decrease, the real GPU memory cost and training time increase.

For example, I got :

Using standard convolution..........
Total parameters : 10,052,609 float, model size : 39,268.00390625M
191.78 GFLOPs
end : 10.62517523765564s
Max memory allocated : 3818.25341796875M

Using bottleneck...........
Total parameters : 1,145,061 float, model size : 4,472.89453125M
16.05 GFLOPs
end : 16.890745162963867s
Max memory allocated : 4408.35107421875 M

However, in inference stage, the bottleneck structure can accelerate the network to some extend.

Does anyone know why this happens and how to accelerate the network both in training and inference stage?

Code :

import torch
import torch.nn as nn
import torch.nn.functional as F

def groupNorm(channel, num_groups=16):
    return nn.GroupNorm(num_groups=num_groups, num_channels=channel)

Norm = nn.BatchNorm3d

BottleNeck_Ratio = 4

class BottleNeck(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, drop, stride=1, padding=1, N=BottleNeck_Ratio):
        super(BottleNeck, self).__init__()

        self.conv_1 = nn.Conv3d(in_channels=in_channels, out_channels=out_channels // N, kernel_size=1, stride=1)
        self.conv_2 = nn.Conv3d(in_channels=out_channels // N, out_channels=out_channels // N, kernel_size=kernel_size,
                                stride=stride, padding=padding)
        self.conv_3 = nn.Conv3d(in_channels=out_channels // N, out_channels=out_channels, kernel_size=1, stride=1)

        self.norm = Norm(out_channels)
        self.relu = nn.ReLU()
        self.drop = nn.Dropout3d(drop)

    def forward(self, input):
        x = self.conv_1(input)
        x = self.conv_2(x)
        x = self.conv_3(x)
        return self.drop(self.relu(self.norm(x)))

class CBR(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, drop, stride=1, padding=1):
        super(CBR, self).__init__()

        self.conv = nn.Conv3d(in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size,
                              stride=stride, padding=padding)
        self.norm = Norm(out_channels)
        self.relu = nn.ReLU()
        self.drop = nn.Dropout3d(drop)

    def forward(self, input):
        return self.drop(self.relu(self.norm(self.conv(input))))

ConvBnReluDrop = BottleNeck

class ResidualDown(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, drop, conv_nums, down=True):
        super(ResidualDown, self).__init__()

        if down:
            self.down = ConvBnReluDrop(in_channels, out_channels, kernel_size=2, stride=2, padding=0, drop=drop)
        else:
            self.down = ConvBnReluDrop(in_channels, out_channels, kernel_size=3, stride=1, padding=1, drop=drop)

        self.convs = nn.ModuleList()
        for i in range(conv_nums):
            self.convs.append(ConvBnReluDrop(out_channels, out_channels, kernel_size, drop))

        self.has_down = down

    def forward(self, x):
        # downsample
        res = self.down(x)
        # convolution
        out = res
        for conv in self.convs:
            out = conv(out)
        # residual
        return out + res

class ResidualUp(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, drop, conv_nums, up=True):
        super(ResidualUp, self).__init__()
        if up:
            self.deconv = nn.ConvTranspose3d(in_channels, out_channels, kernel_size=2, stride=2)
        else:
            self.deconv = ConvBnReluDrop(in_channels, out_channels, kernel_size=3, stride=1, padding=1, drop=drop)

        self.convs = nn.ModuleList()
        self.convs.append(ConvBnReluDrop(2 * out_channels, out_channels, kernel_size, drop))
        for i in range(conv_nums - 1):
            self.convs.append(ConvBnReluDrop(out_channels, out_channels, kernel_size, drop))

    def forward(self, big, small):
        x = self.deconv(small)
        # interpolate to prevent size not match
        x = F.interpolate(x, big.size()[-3:], mode='trilinear', align_corners=False)
        # save x as residual, [out_ch]
        res = x

        # skip connection, concat and conv to small's channel
        # [2*out_ch] => [out_ch]
        x = torch.cat([big, x], 1)
        for conv in self.convs:
            x = conv(x)
        return x + res

class VBNet(nn.Module):
    def __init__(self, in_ch=1, nclass=1, drop=0.01, level=5, bn='batch', bottleneck=False):
        super(VBNet, self).__init__()
        # levels
        self.level = level
        # Normalization layer
        global Norm
        if bn == 'batch':
            Norm = nn.BatchNorm3d
        elif bn == 'group':
            Norm = groupNorm
        # elif bn == 'syncbn':
        #     Norm = SyncBN3d
        else:
            raise Exception("Error for bn")

        global ConvBnReluDrop
        if bottleneck:
            ConvBnReluDrop = BottleNeck
        else:
            ConvBnReluDrop = CBR

        # down 2
        self.downs = nn.ModuleList()
        self.downs.append(ResidualDown(in_ch, 16, 3, drop, 1, False))
        self.downs.append(ResidualDown(16, 32, 3, drop, 2))

        # down layers
        channels = 32
        for i in range(level - 2):
            self.downs.append(ResidualDown(channels, channels * 2, 3, drop, 3))
            channels *= 2

        # up layers
        self.ups = nn.ModuleList()
        for i in range(level - 3):
            self.ups.append(ResidualUp(channels, channels // 2, 3, drop, 3))
            channels = channels // 2

        # up 2
        self.ups.append(ResidualUp(channels, channels // 2, 3, drop, 2))
        channels = channels // 2
        self.ups.append(ResidualUp(channels, channels // 2, 3, drop, 1, False))
        channels = channels // 2

        # classifier
        self.classifier = nn.Conv3d(channels, nclass, kernel_size=1)

    def forward(self, x): # 4,472.89453125M
        outs = []
        for layer in self.downs:
            x = layer(x)
            outs.append(x)

        small = outs[-1]
        for i in range(len(self.ups)):
            layer = self.ups[i]
            big = outs[self.level - i - 2]
            small = layer(big, small)

        out = self.classifier(small)
        return out

def get_net_size(net):
    params = list(net.parameters())
    k = 0
    for i in params:
        l = 1
        for j in i.size():
            l *= j
        k = k + l
    s = ("Total parameters : {:,} float, model size : {:,}M".format(k, k * 4 / 1024))
    return s

if __name__ == '__main__':
    # count_ops is taken from : https://github.com/1adrianb/pytorch-estimate-flops/blob/master/pthflops/ops.py
    import count_ops
    import os
    import time

    os.environ['CUDA_VISIBLE_DEVICES'] = '1'

    # 4003728896
    print("Using standard convolution..........")
    a = torch.randn(6, 1, 32, 128, 128)
    net = VBNet(bn='batch', bottleneck=False)
    print(get_net_size(net))
    print(count_ops(net, a))

    net = net.cuda()
    start = time.time()
    for i in range(10):
        a = torch.randn(6, 1, 32, 128, 128).cuda()
        b = net(a)
        b.sum().backward()
    print('end : {}s'.format(time.time() - start))
    print("Max memory allocated : {}M".format(torch.cuda.max_memory_allocated(0) / (1024.**2)))

    # 4543840768 4622491136
    print("\nUsing bottleneck...........")
    # torch.cuda.reset_max_memory_allocated(0)
    a = torch.randn(6, 1, 32, 128, 128)
    net = VBNet(bn='batch', bottleneck=True)
    print(get_net_size(net))
    print(count_ops(net, a))

    net = net.cuda()
    start = time.time()
    for i in range(10):
        a = torch.randn(6, 1, 32, 128, 128).cuda()
        b = net(a)
        b.sum().backward()
    print('end : {}s'.format(time.time() - start))
    print("Max memory allocated : {} M".format(torch.cuda.max_memory_allocated(0) / (1024.**2)))


title: Assign a tensor to multiple slices

body:
Let

a = tensor([[0, 0, 0, 0],
            [0, 0, 0, 0],
            [0, 0, 0, 0]])
b = torch.tensor([1, 2])
c = tensor([[1, 2, 0, 0],
            [0, 1, 2, 0],
            [0, 0, 1, 2]])

Is there a way to obtain c by assigning b to slices of a without any loops? That is, a[indices] = b for some indices or something similar?


title: OSError: Can't read data (wrong B-tree signature). PyTorch shows this error when I read data from 2 .hdf5 files. What to do?

body:
I m trying to run the following code in Pytorch:

import numpy as np
import torch
import torchvision
import torch.nn as nn
import torch.nn.functional as F
import torch.utils.data as data

class H5Dataset(data.Dataset):

    def __init__(self, trainx_path, trainy_path):

        super(H5Dataset, self).__init__()
        x_file = h5py.File(trainx_path)
        y_file = h5py.File(trainy_path)

        self.data = x_file.get('X')
        self.target = y_file.get('y')

    def __getitem__(self, size):

      permutation1 = list(np.random.permutation(249000))
      permutation2 = list(np.random.permutation(np.arange(249000,498000)))

      size1 = int(size/2)
      index1=list(permutation1[0:size1])
      index2=list(permutation2[0:size1])
      index = index1+index2
      labels=np.array(self.target).reshape(498000,-1)
      train_labels=labels[index]
      train_batch=[]

      for i in range(size):
          img=(self.data)[index[i]]
          train_batch.append(img)
      train_batch=np.array(train_batch)

      return (torch.from_numpy(train_batch).float(), torch.from_numpy(train_labels).float())

    def __len__(self):
        return len(self.data)

dataset = H5Dataset('//content//drive//My Drive//E2E_1//train_x.hdf5','//content//drive//My Drive//E2E_1//train_y.hdf5')
train_loader = torch.utils.data.DataLoader(dataset, batch_size=64, shuffle=True, num_workers=2, pin_memory=False)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(2, 64, kernel_size=(2,2), padding=(8,8), stride=(2,2),padding_mode='zeros')
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(64, 128, kernel_size=(2,2), padding=(8,8), stride=(2,2),padding_mode='zeros')
        self.fc1 = nn.Linear(32768, 500)
        self.drop = nn.Dropout(p=0.5)
        self.fc2 = nn.Linear(500, 1)
        self.fc3 = nn.Linear(1,1)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = F.relu(self.conv2(x))
        x = x.view(-1, 16 * 16 * 128)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
print(net)

import torch.optim as optim

criterion = nn.BCELoss()
optimizer = optim.Adam(net.parameters(), lr=0.0001)

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(train_loader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 32 == 31:    # print every 32 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 32))
            running_loss = 0.0

print('Finished Training')

But i am getting the following error. My train_x and train_y data are stored in 2 separate .hdf5 files and when i try to read them while training, this error pops up. Please could anyone tell what changes must be made.

OSError                                   Traceback (most recent call last)
<ipython-input-8-a8f66fac8b5c> in <module>()
      2
      3     running_loss = 0.0
----> 4     for i, data in enumerate(train_loader, 0):
      5         # get the inputs; data is a list of [inputs, labels]
      6         inputs, labels = data

3 frames
/usr/local/lib/python3.6/dist-packages/torch/_utils.py in reraise(self)
    392             # (https://bugs.python.org/issue2651), so we work around it.
    393             msg = KeyErrorMessage(msg)
--> 394         raise self.exc_type(msg)

OSError: Caught OSError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "<ipython-input-5-0df8cfc88081>", line 37, in __getitem__
    img=(self.data)[index[i]]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/dataset.py", line 573, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 182, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw
  File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread
OSError: Can't read data (wrong B-tree signature)

I am getting the error shown above. I am new to PyTorch so please could you suggest what could be done?


title: what is the torch's torch.cat equivalence with tensorflow?

body:
def cxcy_to_xy(cxcy):
    """
    Convert bounding boxes from center-size coordinates (c_x, c_y, w, h) to boundary coordinates (x_min, y_min, x_max, y_max).

    :param cxcy: bounding boxes in center-size coordinates, a tensor of size (n_boxes, 4)
    :return: bounding boxes in boundary coordinates, a tensor of size (n_boxes, 4)
    """
    return torch.cat([cxcy[:, :2] - (cxcy[:, 2:] / 2),  # x_min, y_min
                      cxcy[:, :2] + (cxcy[:, 2:] / 2)], 1)  # x_max, y_max

I want to change
this torch.cat with tensorflow 2.0


title: How to solve dist.init_process_group from hanging (or deadlocks)?

body:
I was to set up DDP (distributed data parallel) on a DGX A100 but it doesn't work. Whenever I try to run it simply hangs. My code is super simple just spawning 4 processes for 4 gpus (for the sake of debugging I simply destroy the group immediately but it doesn't even reach there):
def find_free_port():
    """ https://stackoverflow.com/questions/1365265/on-localhost-how-do-i-pick-a-free-port-number """
    import socket
    from contextlib import closing

    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s:
        s.bind(('', 0))
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        return str(s.getsockname()[1])

def setup_process(rank, world_size, backend='gloo'):
    """
    Initialize the distributed environment (for each process).

    gloo: is a collective communications library (https://github.com/facebookincubator/gloo). My understanding is that
    it's a library/API for process to communicate/coordinate with each other/master. It's a backend library.

    export NCCL_SOCKET_IFNAME=eth0
    export NCCL_IB_DISABLE=1

    https://stackoverflow.com/questions/61075390/about-pytorch-nccl-error-unhandled-system-error-nccl-version-2-4-8

    https://pytorch.org/docs/stable/distributed.html#common-environment-variables
    """
    if rank != -1:  # -1 rank indicates serial code
        print(f'setting up rank={rank} (with world_size={world_size})')
        # MASTER_ADDR = 'localhost'
        MASTER_ADDR = '127.0.0.1'
        MASTER_PORT = find_free_port()
        # set up the master's ip address so this child process can coordinate
        os.environ['MASTER_ADDR'] = MASTER_ADDR
        print(f"{MASTER_ADDR=}")
        os.environ['MASTER_PORT'] = MASTER_PORT
        print(f"{MASTER_PORT}")

        # - use NCCL if you are using gpus: https://pytorch.org/tutorials/intermediate/dist_tuto.html#communication-backends
        if torch.cuda.is_available():
            # unsure if this is really needed
            # os.environ['NCCL_SOCKET_IFNAME'] = 'eth0'
            # os.environ['NCCL_IB_DISABLE'] = '1'
            backend = 'nccl'
        print(f'{backend=}')
        # Initializes the default distributed process group, and this will also initialize the distributed package.
        dist.init_process_group(backend, rank=rank, world_size=world_size)
        # dist.init_process_group(backend, rank=rank, world_size=world_size)
        # dist.init_process_group(backend='nccl', init_method='env://', world_size=world_size, rank=rank)
        print(f'--> done setting up rank={rank}')
        dist.destroy_process_group()

mp.spawn(setup_process, args=(4,), world_size=4)

why is this hanging?
nvidia-smi output:
$ nvidia-smi
Fri Mar  5 12:47:17 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.102.04   Driver Version: 450.102.04   CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
| N/A   26C    P0    51W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  A100-SXM4-40GB      On   | 00000000:0F:00.0 Off |                    0 |
| N/A   25C    P0    52W / 400W |      3MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  A100-SXM4-40GB      On   | 00000000:47:00.0 Off |                    0 |
| N/A   25C    P0    51W / 400W |      3MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  A100-SXM4-40GB      On   | 00000000:4E:00.0 Off |                    0 |
| N/A   25C    P0    51W / 400W |      3MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   4  A100-SXM4-40GB      On   | 00000000:87:00.0 Off |                    0 |
| N/A   30C    P0    52W / 400W |      3MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   5  A100-SXM4-40GB      On   | 00000000:90:00.0 Off |                    0 |
| N/A   29C    P0    53W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   6  A100-SXM4-40GB      On   | 00000000:B7:00.0 Off |                    0 |
| N/A   29C    P0    52W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   7  A100-SXM4-40GB      On   | 00000000:BD:00.0 Off |                    0 |
| N/A   48C    P0   231W / 400W |   7500MiB / 40537MiB |     99%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    7   N/A  N/A    147243      C   python                           7497MiB |
+-----------------------------------------------------------------------------+

How do I set up ddp in this new machine?

Update
btw I've successfully installed APEX because some other links say to do that but it still fails. For I did:
went to: https://github.com/NVIDIA/apex follwed their instructions
git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

but before the above I had to update gcc:
conda install -c psi4 gcc-5

it did install it as I successfully imported it but it didn't help.

Now it actually prints an error msg:
Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
KeyboardInterrupt
Process SpawnProcess-3:
Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 252, in train
    setup_process(rank, world_size=opts.world_size)
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/distributed.py", line 85, in setup_process
    dist.init_process_group(backend, rank=rank, world_size=world_size)
  File "/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 436, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
  File "/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/site-packages/torch/distributed/rendezvous.py", line 179, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)
RuntimeError: connect() timed out.

During handling of the above exception, another exception occurred:

related:

https://github.com/pytorch/pytorch/issues/9696
https://discuss.pytorch.org/t/dist-init-process-group-hangs-silently/55347/2
https://forums.developer.nvidia.com/t/imagenet-hang-on-dgx-1-when-using-multiple-gpus/61919
apex suggestion: https://discourse.mozilla.org/t/hangs-on-dist-init-process-group-in-distribute-py/44686
https://github.com/pytorch/pytorch/issues/15638
https://github.com/pytorch/pytorch/issues/53395


title: is it possible make cuda deterministic?

body:
I am trying to refactor an application and testing new against old version by comparing outputs. At the first pytorch conv layer using identical input I get identical output when using CPU. However using cuda on GPU the output tensors differ in the 5th decimal place on some cells. Is there any way to eliminate this difference? I am already setting:

random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.backends.cudnn.deterministic = True


title: categorical labels using cross entropy loss, accuracy does not change | deep learning pytorch

body:
I have a question concerning my recent project.

I have been trying using PyTorch to train my multiclass-classification work. I have 3 labels (namely, 0 -> none, 1 -> left, 2 -> right) in my image dataset. I have used nn.CrossEntropyLoss() as my loss function and Adam as optimizer. However, the training result looks like this, the accuracy does not change at all.

==> Building new CNN model ...
==> Initialize CUDA support for CNN model ...
==> Preparing RcCar Image dataset ...
==> Start training ...
Iteration: 1 | Loss: 1.3453235626220703 | Training accuracy: 70% | Test accuracy: 43%
==> Saving model ...
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:251: UserWarning: Couldn't retrieve source code for container of type SimpleCNN. It won't be checked for correctness upon loading.
  "type " + obj.__name__ + ". It won't be checked "
Iteration: 2 | Loss: 0.9048898816108704 | Training accuracy: 70% | Test accuracy: 43%
Iteration: 3 | Loss: 0.873579740524292 | Training accuracy: 70% | Test accuracy: 43%
Iteration: 4 | Loss: 0.8702362179756165 | Training accuracy: 70% | Test accuracy: 43%
Iteration: 5 | Loss: 0.8713874220848083 | Training accuracy: 70% | Test accuracy: 43%
Iteration: 6 | Loss: 0.8639134168624878 | Training accuracy: 70% | Test accuracy: 43%
Iteration: 7 | Loss: 0.8590883612632751 | Training accuracy: 70% | Test accuracy: 43%
Iteration: 8 | Loss: 0.8576076626777649 | Training accuracy: 70% | Test accuracy: 43%
Iteration: 9 | Loss: 0.8523686528205872 | Training accuracy: 70% | Test accuracy: 43%
Iteration: 10 | Loss: 0.8462777137756348 | Training accuracy: 70% | Test accuracy: 43%

I am thinking whether this is because the loss function I chose is not suitable or I have to one-hot encode the labels into

[
[0,0,1],
[0,1,0],
...
]

like this.

I have enclosed my custom Dataset part. Please, please, please help me with this. Thank you!

def RcCarImageLoader(root, batch_size_train, batch_size_test):
    """
    RC Car Image Loader.
    Args:
        train_root:
        test_root:
        batch_size_train:
        batch_size_test:
    Return:
        train_loader:
        test_loader:
    """

    # Normalize training set together with augmentation
    transform_train = transforms.Compose([
        transforms.RandomResizedCrop(64),
        transforms.RandomRotation(10),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225])
    ])

    # Normalize test set same as training set without augmentation
    transform_test = transforms.Compose([
        transforms.Resize(64),
        transforms.CenterCrop(64),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225])
    ])

    # Loading Tiny ImageNet dataset
    print("==> Preparing RcCar Image dataset ...")

    train_set = ImageLoader(csv_filename="./train.csv", transform=transform_train)
    train_loader = torch.utils.data.DataLoader(
        train_set, batch_size=batch_size_train, num_workers=2)

    test_set = ImageLoader(csv_filename="./test.csv", transform=transform_test, train=False)
    test_loader = torch.utils.data.DataLoader(
        test_set, batch_size=batch_size_test, num_workers=2)

    return train_loader, test_loader

def image_loader(path):
    """Image Loader helper function."""
    return Image.open(path.rstrip("\n")).convert('RGB')

class ImageLoader(Dataset):
    """Image Loader for Tiny ImageNet."""

    def __init__(self, csv_filename, transform=None, train=True, loader=image_loader):
        """
        Image Loader Builder.
        Args:
            base_path: path to triplets.txt
            filenames_filename: text file with each line containing the path to an image e.g., `images/class1/sample.JPEG`
            triplets_filename: A text file with each line containing three images
            transform: torchvision.transforms
            loader: loader for each image
        """
        self.transform = transform
        self.loader = loader

        self.train_flag = train

        # load training data
        if self.train_flag:
            train_data = []

            csv_file = pd.read_csv(csv_filename)
            self.train_label = np.asarray(csv_file.iloc[:, 1])
            train_img_names = np.asarray(csv_file.iloc[:, 0])

            for train_img_name in train_img_names:
                train_img = self.loader(os.path.join("./train/", train_img_name))
                train_data.append(train_img)
            self.train_data = train_data

            # train_label_one_hot = [[0 for _ in range(3)] for _ in range(len(train_label))]
            # for i, row in enumerate(train_label_one_hot):
            #     row[train_label[i]] = 1
            #
            # self.train_label = np.asarray(train_label_one_hot)

        # load test data
        else:
            test_data = []

            csv_file = pd.read_csv(csv_filename)
            self.test_label = np.asarray(csv_file.iloc[:, 1])
            test_img_names = np.asarray(csv_file.iloc[:, 0])

            for test_img_name in test_img_names:
                test_img = self.loader(os.path.join("./test/", test_img_name))
                test_data.append(test_img)
            self.test_data = test_data

            # test_label_one_hot = [[0 for _ in range(3)] for _ in range(len(test_label))]
            # for i, row in enumerate(test_label_one_hot):
            #     row[test_label[i]] = 1
            #
            # self.test_label = np.asarray(test_label_one_hot)

    def __getitem__(self, index):
        """Get image and label in dataset."""
        # get training images

        if self.train_flag:
            img = self.train_data[index]
            label = self.train_label[index]
            if self.transform is not None:
                img = self.transform(img)

            return (img, label)

        else:
            img = self.test_data[index]
            label = self.test_label[index]
            if self.transform is not None:
                img = self.transform(img)

            return (img, label)

    def __len__(self):
        if self.train_flag:
            return len(self.train_label)
        else:
            return len(self.test_label)


title: PyTorch: Batch Outer-Addition

body: I have two PyTorch tensors: A and B, both of shape (b, c, 3). I want to make outer product C of A and B so that the resulting shape is (b, c, 3, 3), and replace the multiplication operation with addition. How should I do it?


title: Visualize a tensor (video) in pytorch

body:
While training my nets, I want to visualize video of each epoch.
Until now I made a function that gets my tensor (size: [3, 32, 64,64] - 3 batch size, 32 frames, 64X64 resolution).
I want to visualize some videos as a grid.
I tried to use

import torch
import torchvision.utils as vutils
import numpy as np
import matplotlib.pyplot as plt
import torchvision.transforms as transforms
import scipy.misc

def show(img):
   npimg = img.numpy()
   plt.imshow(np.transpose(npimg, (1,2,0))interpolation='nearest')

video = torch.randn(3,32,64,64)
grid = vutils.make_grid(video)
show(grid) # here to use save_img func

But it failed. Does anyone know how to do it? Thanks!


title: NameError: name 'utils' is not defined in Pytorch

body:
I have pytorch 1.7. The following code is same as from Pytorch's tutorial page for Object detection and finetuning.
But I have error for the following line
data_loader = torch.utils.data.DataLoader(dataset, batch_size=2, shuffle=True, num_workers=4, collate_fn=utils.collate_fn)

as NameError: name 'utils' is not defined
What could be wrong?
The whole code is as follows.
import os
import numpy as np
import torch
from PIL import Image

class PrepareDataset(object):
    def __init__(self, root, transforms):
        self.root = root
        self.transforms = transforms
        # load all image files, sorting them to
        # ensure that they are aligned
        self.imgs = list(sorted(os.listdir(os.path.join(root, "images"))))
        self.masks = list(sorted(os.listdir(os.path.join(root, "masks"))))
        self.annotations = list(sorted(os.listdir(os.path.join(root, "annotations"))))

    def __getitem__(self, idx):
        # load images ad masks
        img_path = os.path.join(self.root, "images", self.imgs[idx])
        mask_path = os.path.join(self.root, "masks", self.masks[idx])
        annotation_path = os.path.join(self.root, "annotations", self.annotations[idx])
        img = Image.open(img_path).convert("RGB")
        # note that we haven't converted the mask to RGB,
        # because each color corresponds to a different instance
        # with 0 being background
        mask = Image.open(mask_path)
        # convert the PIL Image into a numpy array
        mask = np.array(mask)
        # instances are encoded as different colors
        obj_ids = np.unique(mask)
        # first id is the background, so remove it
        obj_ids = obj_ids[1:]

        # split the color-encoded mask into a set
        # of binary masks
        masks = mask == obj_ids[:, None, None]

        # get bounding box coordinates for each mask
        num_objs = len(obj_ids)
        boxes = []
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])

        # convert everything into a torch.Tensor
        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        # there is only one class
        labels = torch.ones((num_objs,), dtype=torch.int64)
        masks = torch.as_tensor(masks, dtype=torch.uint8)

        image_id = torch.tensor([idx])
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        # suppose all instances are not crowd
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["masks"] = masks
        target["image_id"] = image_id
        target["area"] = area
        target["iscrowd"] = iscrowd

        if self.transforms is not None:
            img, target = self.transforms(img, target)

        return img, target

    def __len__(self):
        return len(self.imgs)

import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor

# load a model pre-trained pre-trained on COCO
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)

# replace the classifier with a new one, that has
# num_classes which is user-defined
num_classes = 2  # 1 class (person) + background
# get number of input features for the classifier
in_features = model.roi_heads.box_predictor.cls_score.in_features
# replace the pre-trained head with a new one
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

import torchvision
from torchvision.models.detection import FasterRCNN
from torchvision.models.detection.rpn import AnchorGenerator

# load a pre-trained model for classification and return
# only the features
backbone = torchvision.models.mobilenet_v2(pretrained=True).features
# FasterRCNN needs to know the number of
# output channels in a backbone. For mobilenet_v2, it's 1280
# so we need to add it here
backbone.out_channels = 1280

# let's make the RPN generate 5 x 3 anchors per spatial
# location, with 5 different sizes and 3 different aspect
# ratios. We have a Tuple[Tuple[int]] because each feature
# map could potentially have different sizes and
# aspect ratios
anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),), aspect_ratios=((0.5, 1.0, 2.0),))

# let's define what are the feature maps that we will
# use to perform the region of interest cropping, as well as
# the size of the crop after rescaling.
# if your backbone returns a Tensor, featmap_names is expected to
# be [0]. More generally, the backbone should return an
# OrderedDict[Tensor], and in featmap_names you can choose which
# feature maps to use.
roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0], output_size=7, sampling_ratio=2)

# put the pieces together inside a FasterRCNN model
model = FasterRCNN(backbone, num_classes=5, rpn_anchor_generator=anchor_generator, box_roi_pool=roi_pooler)

import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor

def get_model_instance_segmentation(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)

    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    # now get the number of input features for the mask classifier
    in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
    hidden_layer = 256
    # and replace the mask predictor with a new one
    model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask,
                                                       hidden_layer,
                                                       num_classes)

    return model

from torchvision import transforms as T

def get_transform(train):
    transforms = []
    transforms.append(T.ToTensor())
    if train:
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
dataset = PrepareDataset('/home/centos/atic-nyan/Traffic', get_transform(train=True))
data_loader = torch.utils.data.DataLoader(dataset, batch_size=2, shuffle=True, num_workers=4, collate_fn=utils.collate_fn)
# For Training
images,targets = next(iter(data_loader))
images = list(image for image in images)
targets = [{k: v for k, v in t.items()} for t in targets]
output = model(images,targets)   # Returns losses and detections
# For inference
model.eval()
x = [torch.rand(3, 300, 400), torch.rand(3, 500, 400)]
predictions = model(x)           # Returns predictions

from engine import train_one_epoch, evaluate
import utils

def main():
    # train on the GPU or on the CPU, if a GPU is not available
    device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

    # our dataset has two classes only - background and person
    num_classes = 2
    # use our dataset and defined transformations
    dataset = PrepareDataset('/home/centos/atic-nyan/Traffic', get_transform(train=True))
    dataset_test = PrepareDataset('/home/centos/atic-nyan/Traffic', get_transform(train=False))

    # split the dataset in train and test set
    indices = torch.randperm(len(dataset)).tolist()
    dataset = torch.utils.data.Subset(dataset, indices[:-50])
    dataset_test = torch.utils.data.Subset(dataset_test, indices[-50:])

    # define training and validation data loaders
    data_loader = torch.utils.data.DataLoader(
        dataset, batch_size=2, shuffle=True, num_workers=4,
        collate_fn=utils.collate_fn)

    data_loader_test = torch.utils.data.DataLoader(
        dataset_test, batch_size=1, shuffle=False, num_workers=4,
        collate_fn=utils.collate_fn)

    # get the model using our helper function
    model = get_model_instance_segmentation(num_classes)

    # move model to the right device
    model.to(device)

    # construct an optimizer
    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.SGD(params, lr=0.005,
                                momentum=0.9, weight_decay=0.0005)
    # and a learning rate scheduler
    lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                                   step_size=3,
                                                   gamma=0.1)

    # let's train it for 10 epochs
    num_epochs = 10

    for epoch in range(num_epochs):
        # train for one epoch, printing every 10 iterations
        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        evaluate(model, data_loader_test, device=device)

    print("That's it!")


title: How to bring debug library "python37_d.lib" to Anaconda?

body:
I try to build pytorch from source on Windows. It failed with fatal error LNK1104 "can't open python37_d.lib".

Then I installed python3.7 from www.python.org and select "Download debug binaries". Then I copied the whole newly installed python folder to Anaconda3\envs\MyEnv and replace existing old files. When I built pytorch again, it shows LNK1104 with "can't open python37.lib". I checked and made sure both "python37_d.lib" and "python37.lib" are under Anaconda3\envs\MyEnv\libs.

Is my method correct(to use self-installed python files to replace the ones under Anaconda)? Is there any package from which I can install the python37_d.lib by executing "conda install "?


title: How did Pytorch process images in ImageNet when training resnet pretrained models in torchvision.models.resnet34?

body:
I downloaded the pretrained parameters of resnet34 in torchvision.models and put them to a tensorflow1.X network, but just get 58% accurary testing on the ImageNet2015 Validation set (50,000 picture).

I guess it may be caused by the different precessing method to the data set. The Validation I am using is in TFRecord format processed by my friend.

So I am wondering how Pytorch process images of ImageNet when training resnet34 pretrained models? Turn RGB to BGR? Scale pictures values to 0--1?


title: Reshaping order in PyTorch - Fortran-like index ordering

body:
In numpy, there is an ordering feature for reshaping arrays, by default it is C, but you can specify other ordering like F:
a = np.arange(6).reshape((3, 2))
f = np.reshape(a, (2, 3), order='F') # Fortran-like index ordering
c = np.reshape(a, (2, 3))
print('a= \n', a)
print('f= \n', b)
print('c= \n', c)

the result:
a=
 [[0 1]
 [2 3]
 [4 5]]
f=
 [[0 4 3]
 [2 1 5]]
c=
 [[0 1 2]
 [3 4 5]]

There is no such option in torch.reshape or tensor.view for reshaping in F order.
Is there any way to do that F order reshape in PyTorch? I need everything to be in PyTorch.


title: Best practices for generating a random seeds to seed Pytorch?

body:
What I really want is to seed the dataset and dataloader. I am adapting code from:

https://gist.github.com/kevinzakka/d33bf8d6c7f06a9d8c76d97a7879f5cb

Anyone know how to seed this properly? What are the best practices for seeding things in Pytorch.

Honestly, I have no idea if there is an algorithm specific way for GPU vs CPU. I care mostly about general pytorch and make sure my code is "truly random". Specially when it uses GPU I guess...

related:

https://discuss.pytorch.org/t/best-practices-for-seeding-random-numbers-on-gpu/18751
https://discuss.pytorch.org/t/the-random-seed/19516/4
https://discuss.pytorch.org/t/best-practices-for-generating-a-random-seed-to-seed-pytorch/52894/2

My answer was deleted and here is its content:

I don't know if this is the best for pytorch but this is what seems the best for any programming language:

Usually the best random sample you could get in any programming language is generated through the operating system. In Python, you can use the os module:

random_data = os.urandom(4)

In this way you get a cryptographic safe random byte sequence which you may convert in a numeric data type for using as a seed.

seed = int.from_bytes(random_data, byteorder="big")

EDIT: the snippets of code works only on Python 3

'''
Greater than 4 I get this error:

ValueError: Seed must be between 0 and 2**32 - 1
'''

RAND_SIZE = 4


title: Pytorch model prediction in production with uwsgi

body:
I have a problem deploying a pytorch model in production. For a demonstration, I build a simple model and a flask app. I put everything in a docker container (pytorch+flask+uwsgi) plus another container for nginx. Everything is running well, my app is rendered and I can navigate inside. However, well I navigate into the URL that launches a prediction of the model, the server hangs and does not seem to compute anything.

The uWSGI is run like this:

/opt/conda/bin/uwsgi --ini  /usr/src/web/uwsgi.ini

with uwsgi.ini

[uwsgi]
#application's base folder
chdir = /usr/src/web/
#python module to import
wsgi-file = /usr/src/web/wsgi.py
callable = app
#socket file's location
socket = /usr/src/web/uwsgi.sock
#permissions for the socket file
chmod-socket = 666
# Port to expose
http = :5000
# Cleanup the socket when process stops
vacuum = true
#Log directory
logto = /usr/src/web/app.log
# minimum number of workers to keep at all times
cheaper = 2
processes = 16

As said, the server hangs and I finally got a timeout. What is strange is when I run the flask application directly (also in the container) with

python /usr/src/web/manage.py runserver --host 0.0.0.0

I get my prediction in no time


I read this question but it doesnt seem to answer my question :(.

So basically I'm trying to vectorize the game snake so it can run faster.
Here is my code till now:

import torch
import torch.nn.functional as F
device = torch.device("cpu")
class SnakeBoard:
  def __init__(self, board=None):
    if board != None:
      self.channels = board
    else:
      # 0 - Food, 1 - Head, 2 - Body
      self.channels = torch.zeros(1, 3, 15, 17,
                                  device=device)

      # Initialize game channels
      self.channels[:, 0, 7, 12] = 1
      self.channels[:, 1, 7, 5] = 1
      self.channels[:, 2, 7, 2:6] = torch.arange(1, 5)
    self.move()

  def move(self):
    self.channels[:, 2] -= 1
    F.relu(self.channels[:, 2], inplace=True)
    # Up movement test
    F.conv2d(self.channels[:, 1], torch.tensor([[[0,1,0],[0,0,0],[0,0,0]]]), padding=1)

SnakeBoard()

The first dimension in channels represents batch size, second dimension represent the 3 channels of the snake game: food, head, and body, and finally the third and fourth dimensions represent the height and width of the board.

Unfortunately when running the code I get error: Expected stride to be a single integer value or a list of 1 values to match the convolution dimensions, but got stride=[1, 1]

How can I fix that?


title: problem on training(first epoch take a long time more than 4 hours )using colab and fastai

body:
I have a problem on training (first epoch takes a long time more than 4 hours) using colab and fastai. It takes me about 5 hours only for the first epoch which is not useful using google colab because the limitation using of GPU
I'm using  'efficientnet-b4' and about 52k training photos after data augmentation and 11k valid photos.

The problem is shown on the photo:


title: PyTorch - GPU is not used by tensors despite CUDA support is detected

body:
As the title of the question clearly describes, even though torch.cuda.is_available() returns True, CPU is used instead of GPU by tensors. I have set the device of the tensor to GPU through the images.to(device) function call after defining the device. When I debug my code, I am able to see that the device is set to cuda:0; but the tensor's device is still set to cpu.

Defining the device:

use_cuda = torch.cuda.is_available()  # returns True
device = torch.device('cuda:0' if use_cuda else 'cpu')

Determining the device of the tensors:

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        images.to(device)
        labels.to(device)
        # both of images and labels' devices are set to cpu

The software stack:

Python 3.7.1
torch 1.0.1
Windows 10 64-bit

p.s. PyTorch is installed with the option of Cuda 9.0 support.


title: GPU Drivers not working on AWS Ubuntu Deep Learning AMI

body:
It seems as though GPU drivers do not seem to work on the Deep Learning AMI (Ubuntu 180.04) g4dn.xlarge instances I have set up for the past few hours. nvidia-smi gives output:

  NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

and torch.cuda.is_available() returns false.

Does anyone know why this is happening?


title: Cuda version issue while using Detectron2 in Google Colab

body:
I am trying to run the Detectron2 module on Colab using CUDA version 10.0 but since today there have been some issues regarding the versions of Cuda Compiler.

The output I get after running !nvidia-smi is :

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.36.06    Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   36C    P0    26W / 250W |      0MiB / 16280MiB |      0%      Default |
|                               |                      |                 ERR! |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

And what I get after running !nvcc --version is :

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

I am not able to understand the reason for the mismatch. Also the output from detectron after running !python -m detectron2.utils.collect_env is :

----------------------  ----------------------------------------------------------------------------
sys.platform            linux
Python                  3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]
numpy                   1.18.5
detectron2              0.1.3 @/content/gdrive/My Drive/Data/Table_Struct/detectron2_repo/detectron2
Compiler                GCC 7.5
CUDA compiler           CUDA 10.1
detectron2 arch flags   sm_60
DETECTRON2_ENV_MODULE   <not set>
PyTorch                 1.4.0+cu100 @/usr/local/lib/python3.6/dist-packages/torch
PyTorch debug build     False
GPU available           True
GPU 0                   Tesla K80
CUDA_HOME               /usr/local/cuda
Pillow                  7.0.0
torchvision             0.5.0+cu100 @/usr/local/lib/python3.6/dist-packages/torchvision
torchvision arch flags  sm_35, sm_50, sm_60, sm_70, sm_75
fvcore                  0.1.1
cv2                     4.1.2
----------------------  ----------------------------------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CUDA Runtime 10.0
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  - CuDNN 7.6.3
  - Magma 2.5.1
  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

My guess is that the version of CUDA on Colab doesn't match the Detectron2 I am using. IF so how can I change something to make this work on Google Colab.


title: Distrubuted PyTorch code halts on multiple nodes when using MPI backend

body:
I am trying to run Pytorch code on three nodes using openMPI but the code just halts without any errors or output. Eventually my purpose is to distribute a Pytorch graph on these nodes.

Three of my nodes are connected in same LAN and have SSH access to each other without password and have similar specifications:

Ubuntu 18.04
Cuda 10.0
OpenMPI built and installed from source
PyTorch built and installed from source

The code shown below works on single node - multiple processes, as:

> mpirun -np 3 -H 192.168.100.101:3 python3 run.py

With following output:

INIT 0 of 3 Init env://
INIT 1 of 3 Init env://
INIT 2 of 3 Init env://
RUN 0 of 3 with tensor([0., 0., 0.])
RUN 1 of 3 with tensor([0., 0., 0.])
RUN 2 of 3 with tensor([0., 0., 0.])
Rank  1  has data  tensor(1.)
Rank  0  has data  tensor(1.)
Rank  2  has data  tensor(1.)

But when I placed the code on three nodes and run following command on each node separately, it does nothing:

> mpirun -np 3 -H 192.168.100.101:1,192.168.100.102:1,192.168.100.103:1 python3 run.py

Please give some idea about any modifications in code or configurations for MPI to run given Pytorch code on multiple nodes?

#!/usr/bin/env python
import os
import torch
import torch.distributed as dist
from torch.multiprocessing import Process

def run(rank, size):
    tensor = torch.zeros(size)
    print(f"RUN {rank} of {size} with {tensor}")

    # incrementing the old tensor
    tensor += 1

    # sending tensor to next rank
    if rank == size-1:
       dist.send(tensor=tensor, dst=0)
    else:
       dist.send(tensor=tensor, dst=rank+1)

    # receiving tensor from previous rank
    if rank == 0:
        dist.recv(tensor=tensor, src=size-1)
    else:
        dist.recv(tensor=tensor, src=rank-1)

    print('Rank ', rank, ' has data ', tensor[0])

def init_processes(rank, size, fn, backend, init):
    print(f"INIT {rank} of {size} Init {init}")
    dist.init_process_group(backend, init, rank=rank, world_size=size)
    fn(rank, size)

if __name__ == "__main__":
    os.environ['MASTER_ADDR'] = '192.168.100.101'
    os.environ['BACKEND'] = 'mpi'
    os.environ['INIT_METHOD'] = 'env://'
    world_size = int(os.environ['OMPI_COMM_WORLD_SIZE'])
    world_rank = int(os.environ['OMPI_COMM_WORLD_RANK'])

    init_processes(world_rank, world_size, run, os.environ['BACKEND'], os.environ['INIT_METHOD'])

N.B. NCCL is not an option for me due to arm64-based hardware.


title: Tokenizer doesn't work properly in torchtext

body:
I got a problem in torchtext, and was struggling with it for a long time. I was trying to tokenize and numericalize the text using torchtext and spacy. I defined my tokenizer as this:

def Sp_Tokenizer(text):
    return [tok.text for tok in spacy_en.tokenizer(text)]

It worked good:

Sp_Tokenizer('How are you today')

['How', 'are', 'you', 'today']

Then I passed this tokenizer into torchtext:

TEXT = data.Field(sequential=True, tokenize=Sp_Tokenizer, lower=False)

and built the vocab:

corps = ['How are you', 'I am good today', 'He is not well']
TEXT.build_vocab(corps, vectors="glove.6B.100d")

Then I tried

TEXT.numericalize('How are you today')

I assumed I should get a tensor with 4 numbers (word level), however, what I got was like char level:

tensor([[ 6,  3, 10,  2,  4, 17,  5,  2, 11,  3, 19,  2,  9,  3,  7,  4, 11]])

What's wrong with that? Is there anythin I can do to fix it? Thanks!


title: AttributeError : 'tuple' has no attribute 'to'

body:
I am writing this Image Classifier and I have defined the loaders but getting this mistake and I have no clue about it.
I have defined the train loader, for a better explanation I tried this
for ina,lab in train_loader:
    print(type(ina))
    print(type(lab))

and I got
<class 'torch.Tensor'>
<class 'tuple'>

Now, For training of the model, I did
def train_model(model,optimizer,n_epochs,criterion):
    start_time = time.time()
    for epoch in range(1,n_epochs-1):
        epoch_time = time.time()
        epoch_loss = 0
        correct = 0
        total = 0
        print( "Epoch {}/{}".format(epoch,n_epochs))

        model.train()

        for inputs,labels in train_loader:
            inputs = inputs.to(device)
            labels  = labels.to(device)
            optimizer.zero_grad()
            output = model(inputs)
            loss = criterion(output,labels)
            loss.backward()
            optimizer.step()
            epoch_loss +=loss.item()
            _,pred =torch.max(output,1)
            correct += (pred.cpu()==label.cpu()).sum().item()
            total +=labels.shape[0]

        acc = correct/total

and I got the error:
Epoch 1/15
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-36-fea243b3636a> in <module>
----> 1 train_model(model=arch, optimizer=optim, n_epochs=15, criterion=criterion)

<ipython-input-34-b53149a4bac0> in train_model(model, optimizer, n_epochs, criterion)
     12         for inputs,labels in train_loader:
     13             inputs = inputs.to(device)
---> 14             labels  = labels.to(device)
     15             optimizer.zero_grad()
     16             output = model(inputs)

AttributeError: 'tuple' object has no attribute 'to'

If you want anything more, please tell me!
Thanks
Edit: The label looks like this.
This was an Image Classification between Bee and Wasp. It also contains insects and non insects
('wasp', 'wasp', 'insect', 'insect', 'wasp', 'insect', 'insect', 'wasp', 'wasp', 'bee', 'insect', 'insect', 'other', 'bee', 'other', 'wasp', 'other', 'wasp', 'bee', 'bee', 'wasp', 'wasp', 'wasp', 'wasp', 'bee', 'wasp', 'wasp', 'other', 'bee', 'wasp', 'bee', 'bee')
('wasp', 'wasp', 'insect', 'bee', 'other', 'wasp', 'insect', 'wasp', 'insect', 'insect', 'insect', 'wasp', 'wasp', 'insect', 'wasp', 'wasp', 'wasp', 'bee', 'wasp', 'wasp', 'insect', 'insect', 'wasp', 'wasp', 'bee', 'wasp', 'insect', 'bee', 'bee', 'insect', 'insect', 'other')


title: Under macOS using pip install pytorch failure

body:
When I'm using pip to install pytorch, some exception appeared.

Env:

Sys: MaxOS High Sierra
python version : 3.6
pip version : 19.0.2

input: pip install pytorch

output:

Collecting pytorch
  Using cached https://files.pythonhosted.org/packages/a9/41/4487bc23e3ac4d674943176f5aa309427b011e00607eb98899e9d951f67b/pytorch-0.1.2.tar.gz
Building wheels for collected packages: pytorch
  Building wheel for pytorch (setup.py) ... error
  Complete output from command /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-install-ygd1rucx/pytorch/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-wheel-p0npaj5r --python-tag cp36:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-install-ygd1rucx/pytorch/setup.py", line 17, in <module>
      raise Exception(message)
  Exception: You should install pytorch from http://pytorch.org

  ----------------------------------------
  Failed building wheel for pytorch
  Running setup.py clean for pytorch
Failed to build pytorch
Installing collected packages: pytorch
  Running setup.py install for pytorch ... error
    Complete output from command /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-install-ygd1rucx/pytorch/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-record-us45ly0z/install-record.txt --single-version-externally-managed --compile:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-install-ygd1rucx/pytorch/setup.py", line 13, in <module>
        raise Exception(message)
    Exception: You should install pytorch from http://pytorch.org

    ----------------------------------------
Command "/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-install-ygd1rucx/pytorch/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-record-us45ly0z/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/lw/s7b4_22d1v30nfm0wkys878w0000gn/T/pip-install-ygd1rucx/pytorch/


title: Confused about torch.nn.Sequential

body:
Supposing we want to add a new layer, say a linear layer, to the end of the classifier of another model, such as VGG16, why exactly do these two implementations lead to different results? More specifically, I don't understand why the first implementation produces 2 classfiers:

vgg = torchvision.models.vgg16(pretrained=True)
vgg.classifer=nn.Sequential(vgg.classifier, nn.Linear(4096,300))
print(vgg)

output:

    VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace)
    (2): Dropout(p=0.5)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace)
    (5): Dropout(p=0.5)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )
  (classifer): Sequential(
    (0): Sequential(
      (0): Linear(in_features=25088, out_features=4096, bias=True)
      (1): ReLU(inplace)
      (2): Dropout(p=0.5)
      (3): Linear(in_features=4096, out_features=4096, bias=True)
      (4): ReLU(inplace)
      (5): Dropout(p=0.5)
      (6): Linear(in_features=4096, out_features=1000, bias=True)
    )
    (1): Linear(in_features=4096, out_features=300, bias=True)
  )
)

implementation2:

vgg = models.vgg16(pretrained=True)
vgg=nn.Sequential(vgg, nn.Linear(4096,300))
print(vgg)

output:

Sequential(
  (0): VGG(
    (features): Sequential(
      (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (1): ReLU(inplace)
      (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (3): ReLU(inplace)
      (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (6): ReLU(inplace)
      (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (8): ReLU(inplace)
      (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (11): ReLU(inplace)
      (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (13): ReLU(inplace)
      (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (15): ReLU(inplace)
      (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (18): ReLU(inplace)
      (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (20): ReLU(inplace)
      (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (22): ReLU(inplace)
      (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (25): ReLU(inplace)
      (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (27): ReLU(inplace)
      (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (29): ReLU(inplace)
      (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    )
    (classifier): Sequential(
      (0): Linear(in_features=25088, out_features=4096, bias=True)
      (1): ReLU(inplace)
      (2): Dropout(p=0.5)
      (3): Linear(in_features=4096, out_features=4096, bias=True)
      (4): ReLU(inplace)
      (5): Dropout(p=0.5)
      (6): Linear(in_features=4096, out_features=1000, bias=True)
    )
  )
  (1): Linear(in_features=4096, out_features=300, bias=True)
)


title: PyTorch normalize two sets of gradients during training

body:
In this GAN tutorial, if you scroll down to the training loop you can see they combine the gradients
 errD = errD_real + errD_fake like this. Where errD_real = criterion(output, label) and errD_fake = criterion(output, label) and criterion = nn.BCELoss(). I want to do the same thing but before doing a backward pass I want to normalize both gradients to the lower Euclidean norm of the two. How would I do that?

I know I can access the gradients of each weight individually on netD by printing out netD.weight.grad, but is there some way to batchnorm them to the lower Euclidean norm of the two?

Here's the part of the training loop I'm talking about:

for epoch in range(num_epochs):
    # For each batch in the dataloader
    for i, data in enumerate(dataloader, 0):

        ############################
        # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
        ###########################
        ## Train with all-real batch
        netD.zero_grad()
        # Format batch
        real_cpu = data[0].to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, device=device)
        # Forward pass real batch through D
        output = netD(real_cpu).view(-1)
        # Calculate loss on all-real batch
        errD_real = criterion(output, label)
        # Calculate gradients for D in backward pass
        errD_real.backward()
        D_x = output.mean().item()

        ## Train with all-fake batch
        # Generate batch of latent vectors
        noise = torch.randn(b_size, nz, 1, 1, device=device)
        # Generate fake image batch with G
        fake = netG(noise)
        label.fill_(fake_label)
        # Classify all fake batch with D
        output = netD(fake.detach()).view(-1)
        # Calculate D's loss on the all-fake batch
        errD_fake = criterion(output, label)
        # Calculate the gradients for this batch
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        # Add the gradients from the all-real and all-fake batches
        errD = errD_real + errD_fake
        # Update D
        optimizerD.step()
        ...


title: PyTorch method for returning element with highest count in a 1D tensor?

body:
Is there a built-in PyTorch method that takes a 1D tensor, and returns the element in the tensor with the highest count?
For example, if we input torch.tensor([2,2,2,3,4,5]), the method should return 2 as it occurs the most. In case of a tie in frequency, the element with the lower value should be returned; inputting torch.tensor([1,1,2,2,4,5]) should return 1.
Just to be clear, I only wish to know if there's an existing built-in PyTorch method that does exactly this. If there's no such method, please refrain from posting the solution, as I'd like to try solving it on my own.


title: Load pytorch model from 0.4.1 to 0.4.0?

body:
I trained a DENSENET161 model using pytorch 0.4.1 (GPU) and on testing environment I have to load it in pytorch version 0.4.0 (CPU). I am already using model.cpu()
but when I am loading static dictionary model.load_state_dict(checkpoint['state_dict'])

I am getting following error:

  RuntimeError: Error(s) in loading state_dict for DenseNet:    Unexpected
  key(s) in state_dict: "features.norm0.num_batches_tracked",
  "features.denseblock1.denselayer1.norm1.num_batches_tracked",
  "features.denseblock1.denselayer1.norm2.num_batches_tracked",
  "features.denseblock1.denselayer2.norm1.num_batches_tracked",...


title: Learning rate & gradient descent difference?

body:
What is the difference between the two?, the two serve to reach the minimum point (lower loss) of a function for example.

I understand (I think)  that the learning rate is multiplied by the gradient ( slope ) to make the  gradient descent , but is that so ? Do I miss something?

What is the difference between lr and gradient?

Thanks