title: Pytorch - Not able to achieve reproducibility

body:
I am a training a image classifier model using Pytorch. While training it I am not able to set the seed. I have exploited all my options but still not getting any consistent results. Please help me with the same.

I was using this but my model is still not consistent.

    torch.manual_seed(1)
    torch.cuda.manual_seed(1)
    np.random.seed(1)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

    model = models.resnet50(pretrained=True)
    num_ftrs = model.fc.in_features
    model.fc = nn.Linear(num_ftrs, 10)

    #Define loss function & optimizer

    loss_function = nn.CrossEntropyLoss()
    optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
    lrscheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='max', patience=3, threshold = 0.9)
    model = model.to(device)

    #Train model

    model.train()
    for epoch in range(num_epochs):
        for i, (images, labels) in enumerate(train_loader):
            images, labels = images.to(device), labels.to(device)

            optimizer.zero_grad()
            outputs = model(images)

            _, predicted = torch.max(outputs.data, 1)
            loss = loss_function(outputs, labels)

            loss.backward()
            optimizer.step()

            train_acc = (labels==predicted).sum().item() / images.size(0)

            if (i+1) % 2 == 0:
                print('Epoch [%d/%d], Step [%d/%d], Loss: %.4f, Acc: %.4f'
                      % (epoch+1, num_epochs, i+1, len(train_dset)//batch_size,
                         loss.item(), train_acc))

            if (i+1) % 5 == 0:
                model.eval()

                with torch.no_grad():

                    num_correct, num_total = 0, 0

                    for (images, labels) in val_loader:
                        images, labels = images.to(device), labels.to(device)

                        outputs = model(images)
                        _, predicted = torch.max(outputs.data, 1)

                        num_correct += (labels==predicted).sum().item()
                        num_total += labels.size(0)

                    val_acc = 1. * num_correct / num_total

                    print('Epoch [%d/%d], Step [%d/%d], Val Acc: %.4f'
                          %(epoch+1, num_epochs, i+1, len(train_dset)//batch_size,
                            val_acc))

                model.train()


title: I am trying to use pytorch's implementation of XLNet and got 'Trying to create tensor with negative dimension -1: [-1, 768]' when loading XLNet

body:
I started working on this about two months ago on Google Colab for a midterm project and everything worked perfectly. Now I am modifying it for a final project and keep getting the error 'RuntimeError: Trying to create tensor with negative dimension -1: [-1, 768]'. It looks like pytorch recently pushed a new version 1.5, so I downgraded to version 1.4 and still got the same error. Same with 1.3, and I know I wasn't using anything lower since that came out last year. I checked it with my midterm code and still got the same error, so I don't know what's going on. Here is the chunk of code related to downloading and using the model.

    train_inputs, validation_inputs, train_labels, validation_labels = train_test_split(inputIds,
                                                                                labels,
                                                                                random_state=2020,
                                                                                test_size=0.2)
    train_masks, validation_masks, _, _ = train_test_split(attention_masks, inputIds, random_state=2020,
                                                           test_size=0.2)
    # Turn data into torch tensors
    train_inputs = torch.tensor(train_inputs)
    validation_inputs = torch.tensor(validation_inputs)
    train_labels = torch.tensor(train_labels)
    validation_labels = torch.tensor(validation_labels)
    train_masks = torch.tensor(train_masks)
    validation_masks = torch.tensor(validation_masks)

    # Create Iterators of the datasets
    train_data = TensorDataset(train_inputs, train_masks, train_labels)
    train_sampler = RandomSampler(train_data)
    train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)
    validation_data = TensorDataset(validation_inputs, validation_masks, validation_labels)
    validation_sampler = SequentialSampler(validation_data)
    validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)

    model = XLNetForSequenceClassification.from_pretrained('xlnet-base-cased', num_labels=2)
    # Loads model into GPU memory
    model.cuda()

    param_optimizer = list(model.named_parameters())
    no_decay = ['bias','gamma','beta']
    optimizer_grouped_parameters = [
        {'params':[p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
         'weight_decay_rate':0.01},
        {'params':[p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
         'weight_decay_rate':0.0}
    ]
    optimizer = AdamW(optimizer_grouped_parameters, lr=2e-5)

The error happens on the line model = XLNetForSequenceClassification.from_pretrained('xlnet-base-cased', num_labels=2). The packages I am using:

from pandas import to_datetime
import torch
from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler
from keras.preprocessing.sequence import pad_sequences
from sklearn.model_selection import train_test_split
# MUST INSTALL PYTORCH-TRANSFORMERS
from pytorch_transformers import XLNetTokenizer, XLNetForSequenceClassification, AdamW
from tqdm import trange
from numpy import argmax, sum
import nltk
nltk.download('punkt')

Thank you to anyone who tries to help.


title: How Can I Solve : magic_number = pickle_module.load(f, **pickle_load_args) _pickle.UnpicklingError: invalid load key, 'H'

body:
Recently I Studying Tensorflow. And Nowadays I progress detecting Mask Project.
But there is Error..
First I show My code
# import the necessary packages
from tensorflow.keras.applications.mobilenet_v2 import preprocess_input
from tensorflow.keras.preprocessing.image import img_to_array
#from tensorflow.keras.models import model.load
from imutils.video import VideoStream
from keras.models import load_model
from pickle import *
import numpy as np
import torch
import imutils
import time
import cv2
import os

def detect_and_predict_mask(frame, faceNet, maskNet):
        # grab the dimensions of the frame and then construct a blob
        # from it
        (h, w) = frame.shape[:2]
        blob = cv2.dnn.blobFromImage(frame, 1.0, (224, 224),
                (104.0, 177.0, 123.0))

        # pass the blob through the network and obtain the face detections
        faceNet.setInput(blob)
        detections = faceNet.forward()
        print(detections.shape)

        # initialize our list of faces, their corresponding locations,
        # and the list of predictions from our face mask network
        faces = []
        locs = []
        preds = []

        # loop over the detections
        for i in range(0, detections.shape[2]):
                # extract the confidence (i.e., probability) associated with
                # the detection
                confidence = detections[0, 0, i, 2]

                # filter out weak detections by ensuring the confidence is
                 # greater than the minimum confidence
                if confidence > 0.5:
                        # compute the (x, y)-coordinates of the bounding box for
                        # the object
                        box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
                        (startX, startY, endX, endY) = box.astype("int")

                        # ensure the bounding boxes fall within the dimensions of
                        # the frame
                        (startX, startY) = (max(0, startX), max(0, startY))
                        (endX, endY) = (min(w - 1, endX), min(h - 1, endY))

                        # extract the face ROI, convert it from BGR to RGB channel
                        # ordering, resize it to 224x224, and preprocess it
                        face = frame[startY:endY, startX:endX]
                        face = cv2.cvtColor(face, cv2.COLOR_BGR2RGB)
                        face = cv2.resize(face, (224, 224))
                        face = img_to_array(face)
                        face = preprocess_input(face)

                        # add the face and bounding boxes to their respective
                        # lists
                        faces.append(face)
                        locs.append((startX, startY, endX, endY))

        # only make a predictions if at least one face was detected
        if len(faces) > 0:
                # for faster inference we'll make batch predictions on *all*
                # faces at the same time rather than one-by-one predictions
                # in the above `for` loop
                faces = np.array(faces, dtype="float32")
                preds = maskNet.predict(faces, batch_size=32)

        # return a 2-tuple of the face locations and their corresponding
        # locations
        return (locs, preds)
# model from disk
prototxtPath = r"/home/pi/Desktop/Face-Mask-Detection-master//face_detector/deploy.prototxt"
weightsPath = r"/home/pi/Desktop/Face-Mask-Detection-master/face_detector/res10_300x300_ssd_iter_140000.caffemodel"
faceNet = cv2.dnn.readNet(prototxtPath, weightsPath)

# load the face mask detector model from disk
maskNet = torch.load("/home/pi/Desktop/Face-Mask-Detection-master/mask_detector.model")

# initialize the video stream
print("[INFO] starting video stream...")
vs = VideoStream(src=0).start()

# loop over the frames from the video stream
while True:
        # grab the frame from the threaded video stream and resize it
        # to have a maximum width of 400 pixels
        frame = vs.read()
        frame = imutils.resize(frame, width=400)

        # detect faces in the frame and determine if they are wearing a
        # face mask or not
        (locs, preds) = detect_and_predict_mask(frame, faceNet, maskNet)

        # loop over the detected face locations and their corresponding
        # locations
        for (box, pred) in zip(locs, preds):
                # unpack the bounding box and predictions
                (startX, startY, endX, endY) = box
                (mask, withoutMask) = pred

                # determine the class label and color we'll use to draw
                # the bounding box and text
                label = "Mask" if mask > withoutMask else "No Mask"
                color = (0, 255, 0) if label == "Mask" else (0, 0, 255)

                # include the probability in the label
                label = "{}: {:.2f}%".format(label, max(mask, withoutMask) * 100)

                # display the label and bounding box rectangle on the output
                # frame
                cv2.putText(frame, label, (startX, startY - 10),
                        cv2.FONT_HERSHEY_SIMPLEX, 0.45, color, 2)
                cv2.rectangle(frame, (startX, startY), (endX, endY), color, 2)

        # show the output frame
        cv2.imshow("Frame", frame)
        key = cv2.waitKey(1) & 0xFF

        # if the `q` key was pressed, break from the loop
        if key == ord("q"):
                break

# do a bit of cleanup
cv2.destroyAllWindows()
vs.stop()

I Run This Code In RaspberryPi 3B But This Error is occurs
  File "detect_mask_video.py", line 82, in <module>
    maskNet = torch.load("/home/pi/Desktop/Face-Mask-Detection-master/mask_detector.model")
  File "/home/pi/.local/lib/python3.7/site-packages/torch/serialization.py", line 427, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "/home/pi/.local/lib/python3.7/site-packages/torch/serialization.py", line 621, in _load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'H'.

I Search this error about in google. But There was no solution for me.
I think this error occurs when I load model file by Pytorch... How I Can Solve This Error...


title: Binary Cross Entropy in PyTorch vs Keras

body:
I am trying to recreate a model from Keras in Pytorch. Both use mobilenetV2 and they are multi-class multi-label problems. So I am optimizing the model using binary cross entropy. In Keras this is implemented with model.compile(..., loss='binary_crossentropy',...) and in PyTorch I have implemented the same thing with torch.nn.BCEWithLogitsLoss(). And I sending logits instead of sigmoid activated outputs to the PyTorch model. Although both models converge very similar in terms of loss (both losses goes down to 0.05 after 10 epochs), the output of PyTorch's model is not giving good predictions. Meaning that PyTorch's prediction are not as confident as the Keras model. Investigating this, I realized that the Keras model has a very stron logit at the index of a positive label, however the logits of the PyTorch model is very small at the index of the positive label; hence the sigmoid is not as strong. For example, for a prticualar sample that can be classified in 54 classes, the output is:
output =
tensor([[-1.2380, -2.3283, -2.3025, -2.1275, -2.1020, -2.3684, -3.4669, -3.4503,
         -2.1905, -1.8565, -3.4215, -3.5318, -3.5715, -4.3836, -4.5215, -6.2270,
         -3.8660, -3.7280, -4.6043, -4.7601, -9.5219, -9.4969, -9.4392, -8.0596,
         -6.0773, -5.7972, -4.2495, -4.4533, -4.2641, -4.1068, -4.9987, -4.9321,
         -7.9726, -7.4475, -4.8016, -5.6634, -6.3762, -6.0103, -6.7561, -3.3259,
         -3.8778, -6.7682, -6.5663, -4.0945, -3.0747, -5.5408, -5.6429, -5.9659,
         -5.8574, -7.6435, -7.8895, -6.6514, -6.5506, -5.0583]],
       device='cuda:0')

for which the target is:
target =
tensor([[0., 0., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.,
             0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
             0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 0., 1., 0., 0., 0., 0., 0.]])

You can see why the loss is still very low. Because most of the model outputs are negative and the reduction=mean for PyTorch so it produces a very small loss while the predictions are not great.
These are the things that I have tried with no success at improving the PyTorch model:

changing reduction to reduction='sum'
I was also suggested to use pos_weight for torch.nn.BCEWithLogitsLoss(), however the same model in Keras does not use any pos_weight and it still generates good predictions.
I have tested the outcome of 'binary_crossentropy' and torch.nn.BCEWithLogitsLoss() for the same inputs and targets, but they result in the same loss value. So I don't think there are implementation difference between the two.

Does anyone has any other suggestion or things to consider while moving from keras model to PyTorch?


title: Google Ngram Viewer - English One Million

body:
I'm training a language model in PyTorch and I'd need the most common one million words in English to serve as dictionary.

From what I've understood, the Google Ngram English One Million (1-grams) might suit to this task, but after downloading every part (0-9) of this dataset and using tail on them to check if they were what I supposed, I found out that no part of this dataset contains words beyond the F letter.

As far as I understood, any Version 1 file has its ngrams alphabetically and cronologically sorted and I'm concerned if it might be possible that the most common one million words do not go beyond the F?

Or am I missing the point of this dataset and it isn't the most commond one million words?


title: Unity - Running a Neural Network

body:
I trained a model using PyTorch. In Unity, I am using a WebCamTexture to display a live video. How can I feed the webcam frames into the PyTorch model, then perform actions in Unity with the output of the model?

I found Unity ML-agents, but it doesn't seem like it would help with this situation.


title: How to create mini-batches for non-image data in PyTorch?

body:
I want to load my training and testing data

import torch
from torchvision import datasets, transforms

transform = transforms.Compose([transforms.ToTensor(),
                                transforms.Normalize((0.5, ), (0.5, ))])

trainset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)

testset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/', download=True, train=False, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=True)

I see the implementation for image data is there a way to load non-image data in a similar manner?


title: Torch.nn.Transformer Example Code Throwing Tensor Shape Errors

body:
I was trying to implement a Transformer model with Pytorch and was experimenting with the example from this GitHub repo, which was linked from here in the documentation, and ran into a problem within the PositionalEncoding class, found within model.py.

The code for the class's __init__() function is as follows:

def __init__(self, d_model, dropout=0.1, max_len=5000):
    super(PositionalEncoding, self).__init__()
    self.dropout = nn.Dropout(p=dropout)

    pe = torch.zeros(max_len, d_model)
    position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
    div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
    pe[:, 0::2] = torch.sin(position * div_term)
    pe[:, 1::2] = torch.cos(position * div_term)
    pe = pe.unsqueeze(0).transpose(0, 1)
    self.register_buffer('pe', pe)

This code, ran with d_model = 103, threw the following error on the forth last line (pe[:, 0::2] =...):

RuntimeError: The expanded size of the tensor (52) must match the existing size (51) at non-singleton dimension 1.  Target sizes: [5000, 52].  Tensor sizes: [5000, 51]

I've found this error fairly impenetrable and haven't had much success writing my own similarly effective implementation.
My first guess would be that this is a problem with version changes within Python/PyTorch, but it could of course be something else I'm missing.


title: What's the difference between dim in Pytorch and axis in Tensorflow?

body:
I have two lines and I want to understand whether they will produce the same output or not?
In tensorflow: tf.norm(my_tensor, ord=2, axis=1)
In pytorch: torch.norm(my_tensor, p=2, dim=1)
Say the shape of my_tensor is [100,2]

Will the above two lines give the same result? Or is the axis attribute different from dim?


title: Custom data loader is returning list in pytorch

body:
I want to get 3 batches of images from 3 different folders. I have written custom data loader in pytorch. but it is returning list that has all the batches instead of single batch at a time.(running in google colab)

#custom data loader
class set(Dataset):
    def __init__(self, dataset_input, dataset_expertA, dataset_expertB):
        self.dataset1 = dataset_input
        self.dataset2 = dataset_expertA
        self.dataset3 = dataset_expertB

    def __getitem__(self, index):
        x1 = self.dataset1[index]
        x2 = self.dataset2[index]
        x3 = self.dataset3[index]

        return x1, x2, x3

    def __len__(self):
        return len(self.dataset1)

input_path = "/content/gdrive/My Drive/project/input/"

dataset = datasets.ImageFolder(root= input_path, transform=transforms.Compose([
                               transforms.Resize([64,64]),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
                               ]))

expertA_path = "/content/gdrive/My Drive/project/expertA/"

datasetA = datasets.ImageFolder(root= expertA_path, transform=transforms.Compose([
                               transforms.Resize([64,64]),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
                               ]))

expertB_path = "/content/gdrive/My Drive/project/expertB/"

datasetB = datasets.ImageFolder(root= expertB_path, transform=transforms.Compose([
                               transforms.Resize([64,64]),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
                               ]))

data = set(dataset, datasetA, datasetB)
dataloader = torch.utils.data.DataLoader(data, batch_size=64,
                                         shuffle=True, num_workers=2)

for i, (inp, expA, expB) in enumerate(dataloader):

  print(inp.shape)
  break

this prints error that inp is list and when i print(inp[0].shape) i get proper shape i think inp contains all batches ie inp[0], inp[1]...

what mistake am i doing in data loader code?


I recently have access to a GPU and have tried to run my code but there is an error. I've read about adding .cuda() to the layers could help but I have ran similar code without the "qml" part and it worked just fine without changing the layer (with cuda I mean). For the sake of brevity, I only added the class for the neural network and the entire loop of the code. The missing part of the code is for other aspects. But if there is a need for the rest of the code, I would be happy to include everything.
Code:
class DQN(nn.Module):

    def __init__(self, img_height, img_width):
        super().__init__()
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(in_features=img_height * img_width * 3, out_features=12)
        self.fc2 = nn.Linear(in_features=12, out_features=8)
       # self.fc3 = nn.Linear(in_features=10, out_features=8)
        self.clayer_in = torch.nn.Linear(in_features=8, out_features=wires)
        self.clayer_out = torch.nn.Linear(wires, out_dim)

        dev = qml.device('strawberryfields.fock', wires=wires, cutoff_dim=3)
        self.layer_qnode = qml.QNode(layer, dev)

        weights = qml.init.cvqnn_layers_all(n_quantum_layers, wires)
        weight_shapes = {"w{}".format(i): w.shape for i, w in enumerate(weights)}

        self.qlayer = qml.qnn.TorchLayer(self.layer_qnode, weight_shapes)

    def forward(self, t):
        t = self.flatten(t)
        t = self.fc1(t)
        t = self.fc2(t)
       # t = self.fc3(t)
        t = self.clayer_in(t)
        t = self.qlayer(t)
        t = self.clayer_out(t)
        t = t.sigmoid()
        return t

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
#print(device)
em = CartPoleEnvManager(device)
strategy = EpsilonGreedyStrategy(eps_start, eps_end, eps_decay)
agent = Agent(strategy, em.num_actions_available(), device)
memory = ReplayMemory(memory_size)
#learning_rate = LearningRate(lr_start,lr_end,lr_decay)
#learn = lr(learning_rate)

policy_net = DQN(em.get_screen_height(), em.get_screen_width()).to(device)
target_net = DQN(em.get_screen_height(), em.get_screen_width()).to(device)
target_net.load_state_dict(policy_net.state_dict())
target_net.eval() #tells pytorch that target_net is only used for inference, not training
optimizer = optim.Adam(params=policy_net.parameters(), lr=0.01)

i = 0
episode_durations = []
for episode in range(num_episodes): #iterate over each episode
    program_starts = time.time()
    em.reset()
    state = em.get_state()

    for timestep in count():
        action = agent.select_action(state, policy_net)
        reward = em.take_action(action)
        next_state = em.get_state()
        memory.push(Experience(state, action, next_state, reward))
        state = next_state
        #i+=1
        #print(i)
        if memory.can_provide_sample(batch_size):
            scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.9)
            experiences = memory.sample(batch_size)
            states, actions, rewards, next_states = extract_tensors(experiences)

            current_q_values = QValues.get_current(policy_net, states, actions)
            next_q_values = QValues.get_next(target_net, next_states) #will get the max qvalues of the next state, q values of next state are used via next state
            target_q_values = (next_q_values * gamma) + rewards

            loss = F.mse_loss(current_q_values, target_q_values.unsqueeze(1))
            optimizer.zero_grad() # sets the gradiesnt of all weights n biases in policy_net to zero
            loss.backward() #computes gradient of loss with respect to all weights n biases in the policy net
            optimizer.step() # updates the weights n biases with the gradients that were computed form loss.backwards
            scheduler.step()
        if em.done:
            episode_durations.append(timestep)
            plot(episode_durations, 100)
            break
    if episode % target_update == 0:
        target_net.load_state_dict(policy_net.state_dict())
    now = time.time()
    print("Episode hat {0} Sekunden gedauert".format(now - program_starts))

em.close()

And the error:
Traceback (most recent call last):
  File "qdqn.py", line 328, in <module>
    loss.backward() #computes gradient of loss with respect to all weights n biases in the policy net
  File "/home/ubuntu/anaconda3/envs/gymm/lib/python3.8/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/ubuntu/anaconda3/envs/gymm/lib/python3.8/site-packages/torch/autograd/__init__.py", line 98, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Expected object of device type cuda but got device type cpu for argument #2 'mat2' in call to _th_mm


title: Getting different eigenvalues between using numpy.linalg.eigh() and torch.symeig()

body:
I am trying to understand why am I getting different eigenvalues between using numpy.linalg.eigh() and torch.symeig().
An example is as below:
Code:
import numpy as np
import torch

arr_symmetric = np.array([[1.,2,3], [2,5,6], [3,6,9]])
arr_symmetric, arr_symmetric.dtype

Output:
(array([[1., 2., 3.],
        [2., 5., 6.],
        [3., 6., 9.]]), dtype('float64'))

Code:
tsr_symmetric = torch.tensor(arr_symmetric)
tsr_symmetric

Output:
tensor([[1., 2., 3.],
        [2., 5., 6.],
        [3., 6., 9.]], dtype=torch.float64)

Code:
w, v = np.linalg.eigh(arr_symmetric)
w, v

Output:
(array([4.05517871e-16, 6.99264746e-01, 1.43007353e+01]),
 array([[-9.48683298e-01,  1.77819106e-01, -2.61496397e-01],
        [ 2.22044605e-16, -8.26924214e-01, -5.62313386e-01],
        [ 3.16227766e-01,  5.33457318e-01, -7.84489190e-01]]))

Code:
e, v = torch.symeig(tsr_symmetric, eigenvectors=True)
e, v

Output:
(tensor([-2.6047e-16,  6.9926e-01,  1.4301e+01], dtype=torch.float64),
 tensor([[ 9.4868e-01, -1.7782e-01,  2.6150e-01],
         [ 8.6389e-16,  8.2692e-01,  5.6231e-01],
         [-3.1623e-01, -5.3346e-01,  7.8449e-01]], dtype=torch.float64))

As you can see one of the eigenvalues is different, ie. 4.05517871e-16 vs. -2.6047e-16
Why is this happening?


title: Pytorch CNN:RuntimeError: Given groups=1, weight of size [16, 16, 3], expected input[500, 1, 19357] to have 16 channels, but got 1 channels instead

body:
class ConvolutionalNetwork(nn.Module):
    def __init__(self, in_features, trial):
        super().__init__()
        self.in_features = in_features
        self.trial = trial
        # this computes num features outputted from the two conv layers
        c1 = int(((self.in_features - 2)) / 64)  # this is to account for the loss due to conversion to int type
        c2 = int((c1 - 2) / 64)
        self.n_conv = int(c2 * 16)
        # self.n_conv = int((( ( (self.in_features - 2)/4 ) - 2 )/4 ) * 16)
        self.conv1 = nn.Conv1d(16, 16, 3, 1)
        self.conv1_bn = nn.BatchNorm1d(16)
        self.conv2 = nn.Conv1d(16, 16, 3, 1)
        self.conv2_bn = nn.BatchNorm1d(16)
        # self.dp = nn.Dropout(trial.suggest_uniform('dropout_rate',0,1.0))
        self.dp = nn.Dropout(0.5)
        self.fc3 = nn.Linear(self.n_conv, 2)

    def forward(self, x):
        # shape x for conv 1d op
        x = x.view(-1, 1, self.in_features)
        x = self.conv1(x)
        x = F.tanh(x)
        x = F.max_pool1d(x, 64, 64)
        x = self.conv2(x)
        x = F.tanh(x)
        x = F.max_pool1d(x, 64, 64)
        x = x.view(-1, self.n_conv)

        x = self.dp(x)
        x = self.fc3(x)
        x = F.log_softmax(x, dim=1)

        return x

Ran the code above and this error popped up :
RuntimeError: Given groups=1, weight of size [16, 16, 3], expected input[500, 1, 19357] to have 16 channels, but got 1 channels instead.

Anyone able to advise on this? It says discrepancies in input , but the above codes works well earlier , unsure what happened after I re-arranged the code.


title: Python PyTorch executing in multiple threads, runs out of memory and get killed by operating system

body:
I am trying to execute a retrained PyTorch FasterRCNN in multiple threads on an Nvidia Jetson Xavier.
The main threads add the image path to a Queue. Four worker threads are doing the following things:

loading the image with PIL img = Image.open(imgPath)
transform it into a tensor by img = to_tensor(img) from tourchvision.transforms
put it to GPU img = img.to(device)
execute the RCNN Network pred = model([img])
save the results in a normal list resultList.append(pred)
delete the variable holding the image with del img

However, the process runs out of memory after around 10.000 images and get killed by the operating system.
I tried to do the following steps after 1000 images:

stop all threads
do garbage collection by gc.collect()
clear GPU Memory by torch.cuda.empty_cache()
restart threads

However, as expected, it does not solve the problem.
I know there is the DataLoader of PyTorch to do multithreading. Since I use the RCNN in a larger project I tried it without the DataLoader within the execution task.
I'm pretty sure there is no list that stores images, since then, the memory would run out faster. The results of the network are just bounding boxes. Therefore, they also should not consume so much memory. Additionally, the memory consumption is not growing slow, instead its jumps sometimes by around 1GB.
I hope someone have an idea for solving the problem or how to better debug.
Thanks,
Peter


title: Can I specify kernel-weight specific learning rates in PyTorch?

body:
I would like to set specific learning rates for each parameter on their lowest level. I.e. each value in a kernels weight and biases should have their own learning rate.

I can specify filter-wise learning rates like that:

optim = torch.optim.SGD([{'params': model.conv1.weight, 'lr': 0.1},], lr=0.01)

But when I want to get a level lower, like that:

optim = torch.optim.SGD([{'params': model.conv1.weight[0, 0, 0, 0], 'lr': 0.1},], lr=0.01)

I receive an error: ValueError: can't optimize a non-leaf Tensor
I also tried specifying a learning rate that has the same shape as the filter such as 'lr': torch.ones_like(model.conv1.weight), but that also didn't work out.

Is there even a way to do this using torch.optim?


title: Does Pytorch 1.4 support cuda10.1?

body:
I try to install pytorch 1.5+cu101, but it can't find caffe2_nvrtc.dll(it indeed exists in lib\caffe2_nvrtc.dll). So I try to install pytorch 1.4+cu101, but I can not find such wheel(only pytorch1.4+cu100/cu92) from https://download.pytorch.org/whl/torch_stable.html.
Any help?


title: Training Generator of Generative Adversarial Network (GAN) in PyTorch

body:
I am working on implementing a Generative Adversarial Network (GAN) in PyTorch 1.5.0.

For computing the loss of the generator, I compute both the negative probabilities that the discriminator mis-classifies an all-real minibatch and an all-(generator-generated-)fake minibatch. Then, I back-propagate both parts sequentially and finally apply the step function.

Calculating and back-propagating the part of the loss which is a function of the mis-classifications of the generated fake data seems straight forward, since during back-propagation of that loss term, the backward path leads through the generator who has produced the fake data in the first place.

However, classification of all-real-data minibatches does not involve passing data through the generator. Therefore, I was wondering whether the following code snipped would still calculate gradients for the generator or whether it would not calculate any gradients at all (since the backward path does not lead through the generator and the discriminator is in eval-mode while updating the generator)?

# Update generator #
net.generator.train()
net.discriminator.eval()
net.generator.zero_grad()

# All-real minibatch
x_real = get_all_real_minibatch()
y_true = torch.full((batch_size,), label_fake).long()  # Pretend true targets were fake
y_pred = net.discriminator(x_real)  # Produces softmax probability distribution over (0=label_fake,1=label_real)

loss_real = NLLLoss(torch.log(y_pred), y_true)
loss_real.backward()
optimizer_generator.step()

If this doesn’t work as intended, how could I make it work? Thanks in advance!


title: ValueError: Expected target size (128, 44), got torch.Size([128, 100]), LSTM Pytorch

body:
I want to build a model, that predicts next character based on the previous characters.
I have spliced text into sequences of integers with length = 100(using dataset and dataloader).
Dimensions of my input and target variables are:
inputs dimension: (batch_size,sequence length). In my case (128,100)
targets dimension: (batch_size,sequence length). In my case (128,100)

After forward pass I get dimension of my predictions: (batch_size, sequence_length, vocabulary_size) which is in my case (128,100,44)
but when I calculate my loss using nn.CrossEntropyLoss() function:
batch_size = 128
sequence_length   = 100
number_of_classes = 44
# creates random tensor of your output shape
output = torch.rand(batch_size,sequence_length, number_of_classes)
# creates tensor with random targets
target = torch.randint(number_of_classes, (batch_size,sequence_length)).long()

# define loss function and calculate loss
criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
print(loss)

I get an error:
ValueError: Expected target size (128, 44), got torch.Size([128, 100])

Question is: how should I handle calculation of the loss function for many-to-many LSTM prediction? Especially sequence dimension? According to nn.CrossEntropyLoss Dimension must be(N,C,d1,d2...dN), where N is batch_size,C - number of classes. But what is D? Is it related to sequence length?


title: LSTM/RNN in pytorch The relation between forward method and training model

body:
I'm still fairly new to neural networks, so sorry on beforehand for any ambiguities to the following.
In a "standard" LSTM implementation for language task, we have the following (sorry for the very rough sketches):
class LSTM(nn.Module):
    def __init__(*args):
    ...

    def forward(self, input, states):

        lstn_in = self.model['embed'](input)
        lstm_out, hidden = self.model['lstm'](lstm_in,states)

        return lstm_out, hidden

Later on, we call upon this model in the training step:
def train(*args):

    for epoch in range(epochs):
        ....
        *init_zero_states
        ...
        out, states = model(input, states)
        ...
    return model

Let's just say, that I have 3 sentences as input:
sents = [[The, sun, is, shiny],
 [The, beach, was, very, windy],
 [Computer, broke, down, today]]

model = train(LSTM, sents)

All words in all sentences gets converted to embeddings and loaded into the model.
Now the question:

Does the self.model['lstm'] iterate though all words from all articles and makes one output after every word? or every sentence?

How does the model make distinction between the 3 sentences, such as after getting "The", "sun", "is", "shiny", does something (such as the states) in the 'lstm' reset and begin anew?

The "out" in the train step after out, states = model(input, states) is the output after running all 3 sentences and hence the combined "information" from all 3 sentences?

Thanks!


title: Test set accuracy is very high after very few epochs on mnist dataset

body:
With very few epochs this model learns to classify beween 1 and 0 extremely quickly which leads me to consider something is wrong.

Below code downloads mnist dataset, extracts the mnist images that contain 1 or 0 only. A random sample of size 200 is selected from this subset of mnist images. This random sample is the dataset the model is trained on. With just 2 epochs the model achieves 90%+ test set accuracy, is this expected behaviour ? I expected many more epochs would be required in order to train the model to achieve this level of test set accuracy.

Model code :

%reset -f

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch.utils.data as data_utils
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from matplotlib import pyplot
from pandas import DataFrame
import torchvision.datasets as dset
import os
import torch.nn.functional as F
import time
import random
import pickle
from sklearn.metrics import confusion_matrix
import pandas as pd
import sklearn

trans = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])

root = './data'
if not os.path.exists(root):
    os.mkdir(root)
train_set = dset.MNIST(root=root, train=True, transform=trans, download=True)
test_set = dset.MNIST(root=root, train=False, transform=trans, download=True)

batch_size = 64

train_loader = torch.utils.data.DataLoader(
                 dataset=train_set,
                 batch_size=batch_size,
                 shuffle=True)
test_loader = torch.utils.data.DataLoader(
                dataset=test_set,
                batch_size=batch_size,
shuffle=True)

class NeuralNet(nn.Module):
    def __init__(self):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(28*28, 500)
        self.fc2 = nn.Linear(500, 256)
        self.fc3 = nn.Linear(256, 2)
    def forward(self, x):
        x = x.view(-1, 28*28)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

num_epochs = 2
random_sample_size = 200

values_0_or_1 = [t for t in train_set if (int(t[1]) == 0 or int(t[1]) == 1)]
values_0_or_1_testset = [t for t in test_set if (int(t[1]) == 0 or int(t[1]) == 1)]

print(len(values_0_or_1))
print(len(values_0_or_1_testset))

train_loader_subset = torch.utils.data.DataLoader(
                 dataset=values_0_or_1,
                 batch_size=batch_size,
                 shuffle=True)

test_loader_subset = torch.utils.data.DataLoader(
                 dataset=values_0_or_1_testset,
                 batch_size=batch_size,
                 shuffle=False)

train_loader = train_loader_subset

# Hyper-parameters
input_size = 100
hidden_size = 100
num_classes = 2
# learning_rate = 0.00001
learning_rate = .0001
# Device configuration
device = 'cpu'
print_progress_every_n_epochs = 1

model = NeuralNet().to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

N = len(train_loader)
# Train the model
total_step = len(train_loader)

most_recent_prediction = []
test_actual_predicted_dict = {}

rm = random.sample(list(values_0_or_1), random_sample_size)
train_loader_subset = data_utils.DataLoader(rm, batch_size=4)

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader_subset):
        # Move tensors to the configured device
        images = images.reshape(-1, 2).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    if (epoch) % print_progress_every_n_epochs == 0:
        print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))

predicted_test = []
model.eval()  # eval mode (batchnorm uses moving mean/variance instead of mini-batch mean/variance)
probs_l = []

predicted_values = []
actual_values = []
labels_l = []

with torch.no_grad():
    for images, labels in test_loader_subset:
        images = images.to(device)
        labels = labels.to(device)
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        predicted_test.append(predicted.cpu().numpy())

        sm = torch.nn.Softmax()
        probabilities = sm(outputs)
        probs_l.append(probabilities)
        labels_l.append(labels.cpu().numpy())

    predicted_values.append(np.concatenate(predicted_test).ravel())
    actual_values.append(np.concatenate(labels_l).ravel())

if (epoch) % 1 == 0:
    print('test accuracy : ', 100 * len((np.where(np.array(predicted_values[0])==(np.array(actual_values[0])))[0])) / len(actual_values[0]))

Output of model (12665 & 2115 represents the training and test set sizes) :

12665
2115
Epoch [1/2], Step [50/198], Loss: 0.1256
Epoch [2/2], Step [50/198], Loss: 0.0151
test accuracy :  99.76359338061465

/anaconda3/envs/pytorch/lib/python3.7/site-packages/ipykernel_launcher.py:143: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.


title: GKE - GPU nvidia - cuda drivers dont work

body:
I have setup a kubernetes node with a nvidia tesla k80 and followed this tutorial to try to run a pytorch docker image with nvidia drivers and cuda drivers working.

I have managed to install the nvidia daemonsets and i can now see the following pods:

nvidia-driver-installer-gmvgt
nvidia-gpu-device-plugin-lmj84

The problem is that even while using the recommendend image nvidia/cuda:10.0-runtime-ubuntu18.04 i still can't find the nvidia drivers inside my pod:

root@pod-name-5f6f776c77-87qgq:/app# ls /usr/local/
bin  cuda  cuda-10.0  etc  games  include  lib  man  sbin  share  src

But the tutorial mention:

  CUDA libraries and debug utilities are made available inside the container at /usr/local/nvidia/lib64 and /usr/local/nvidia/bin, respectively.

I have also tried to test if cuda was working through torch.cuda.is_available() but i get False as a return value.

Many help in advance for your help


title: Should I set model.eval for getting the current training loss in Pytorch?

body: We set model.train() during training, but during my training iterations, I also want to do a forward pass of the training dataset to see what my new loss is. When doing this, should I temporarily set model.eval()?


title: pytorch fix gpu mem leak after exactly 10 minibatches

body:
So I had this problem for a few days that is driving me crazy. I have a seq2seq model (specifically a Listen, Attend and Spell model from google) that Im training and always goes crazy wild in gpu memory usage when reaching mini batch number 10. I have tried shuffling the dataloader to make sure the problem wasnt related to the specific data being retrieved in the 10th time but the error persisted to happen in this 10th iteration.

My train loop looks as follows:

for epoch in range(start_epoch, epochs):
    epoch_step = 0
    train_loss = []
    train_ler = []
    batch_loss = 0
    for i, (data) in enumerate(train_loader):
        print(
            f"Current Epoch: {epoch} Loss {np.round(batch_loss, 3)} | Epoch step: {epoch_step}/{len(train_loader)}",
            end="\r",
            flush=True,
        )
        # Adjust LR
        tf_rate = tf_rate_upperbound - (tf_rate_upperbound - tf_rate_lowerbound) * min(
            (float(global_step) / tf_decay_step), 1
        )
        with torch.no_grad():
            inputs = data[1]["inputs"].cuda()
            labels = data[2]["targets"].cuda()

        # minibatch execution
        batch_loss, batch_ler = batch_iterator(
            batch_data=inputs,
            batch_label=labels,
            las_model=las,
            optimizer=optimizer,
            tf_rate=tf_rate,
            is_training=True,
            max_label_len=params["data"]["vocab_size"],
            label_smoothing=params["training"]["label_smoothing"],
        )
        del inputs
        del labels
        torch.cuda.empty_cache()

        train_loss.append(batch_loss)
        train_ler.extend(batch_ler)

        global_step += 1
        epoch_step += 1

Where batch_iterator trains the network from the minibatch. Batch_iterator looks as follows:

def batch_iterator(
    batch_data,
    batch_label,
    las_model,
    optimizer,
    tf_rate,
    is_training,
    max_label_len,
    label_smoothing,
    use_gpu=True,
):
    label_smoothing = label_smoothing
    max_label_len = min([batch_label.size()[1], max_label_len])
    criterion = nn.NLLLoss(ignore_index=0).cuda()
    optimizer.zero_grad()
    raw_pred_seq, _ = las_model(
        batch_data=batch_data,
        batch_label=batch_label,
        teacher_force_rate=tf_rate,
        is_training=is_training,
    )
    pred_y = (
        torch.cat([torch.unsqueeze(each_y, 1) for each_y in raw_pred_seq], 1)[:, :max_label_len, :]
    ).contiguous()

    if label_smoothing == 0.0 or not (is_training):
        pred_y = pred_y.permute(0, 2, 1)  # pred_y.contiguous().view(-1,output_class_dim)
        true_y = torch.max(batch_label, dim=2)[1][:, :max_label_len].contiguous()  # .view(-1)

        loss = criterion(pred_y, true_y)
        # variable -> numpy before sending into LER calculator
        batch_ler = LetterErrorRate(
            torch.max(pred_y.permute(0, 2, 1), dim=2)[1]
            .cpu()
            .numpy(),  # .reshape(current_batch_size,max_label_len),
            true_y.cpu().data.numpy(),
        )  # .reshape(current_batch_size,max_label_len), data)

    else:
        true_y = batch_label[:, :max_label_len, :].contiguous()
        true_y = true_y.type(torch.cuda.FloatTensor) if use_gpu else true_y.type(torch.FloatTensor)
        loss = label_smoothing_loss(pred_y, true_y, label_smoothing=label_smoothing)
        batch_ler = LetterErrorRate(
            torch.max(pred_y, dim=2)[1].cpu().numpy(),  # .reshape(current_batch_size,max_label_len),
            torch.max(true_y, dim=2)[1].cpu().data.numpy(),
        )  # .reshape(current_batch_size,max_label_len), data)

    if is_training:
        loss.backward()
        optimizer.step()

    batch_loss = loss.cpu().data.numpy()

    return batch_loss, batch_ler

The full code of the project is inside this github repository https://github.com/jiwidi/las-pytorch if want to check it:

While running in a v100 (16gb of gpu ram) the usage of memory is constant around 4gb and spikes to 14gb in 10th iteration. As the error persists to happen in this exact iteration regardless that I shuffle the dataloader I believe the error is in my code.

I also tried to put print statmentes before each batch_iterator where I would print the size of both inputs and labels, the size of them were equal to the ones in the previous minibatches. Output here:

Current Epoch: 0 Loss 0 | Epoch step: 0/28539
For epoch 0 inputs has size torch.Size([1, 1536, 40])mb and labels has size torch.Size([1, 177, 42])mb
Current Epoch: 0 Loss 3.7290000915527344 | Epoch step: 1/28539
For epoch 0 inputs has size torch.Size([1, 1536, 40])mb and labels has size torch.Size([1, 177, 42])mb
Current Epoch: 0 Loss 4.315999984741211 | Epoch step: 2/28539
For epoch 0 inputs has size torch.Size([1, 1568, 40])mb and labels has size torch.Size([1, 177, 42])mb
Current Epoch: 0 Loss 3.928999900817871 | Epoch step: 3/28539
For epoch 0 inputs has size torch.Size([1, 1088, 40])mb and labels has size torch.Size([1, 177, 42])mb
Current Epoch: 0 Loss 3.822999954223633 | Epoch step: 4/28539
For epoch 0 inputs has size torch.Size([1, 1504, 40])mb and labels has size torch.Size([1, 177, 42])mb
Current Epoch: 0 Loss 4.47599983215332 | Epoch step: 5/28539
For epoch 0 inputs has size torch.Size([1, 1376, 40])mb and labels has size torch.Size([1, 177, 42])mb
Current Epoch: 0 Loss 3.9019999504089355 | Epoch step: 6/28539
For epoch 0 inputs has size torch.Size([1, 1248, 40])mb and labels has size torch.Size([1, 177, 42])mb
Current Epoch: 0 Loss 4.1620001792907715 | Epoch step: 7/28539
For epoch 0 inputs has size torch.Size([1, 1216, 40])mb and labels has size torch.Size([1, 177, 42])mb
Current Epoch: 0 Loss 3.5490000247955322 | Epoch step: 8/28539
For epoch 0 inputs has size torch.Size([1, 1600, 40])mb and labels has size torch.Size([1, 177, 42])mb
Current Epoch: 0 Loss 5.377999782562256 | Epoch step: 9/28539
For epoch 0 inputs has size torch.Size([1, 1504, 40])mb and labels has size torch.Size([1, 177, 42])mb
Current Epoch: 0 Loss 4.515999794006348 | Epoch step: 10/28539
For epoch 0 inputs has size torch.Size([1, 1248, 40])mb and labels has size torch.Size([1, 177, 42])mb

The the actual error pytorch gives me:

raceback (most recent call last):
  File "train.py", line 138, in <module>
    label_smoothing=params["training"]["label_smoothing"],
  File "/home/fhjaime966/las-pytorch/solver/solver.py", line 63, in batch_iterator
    is_training=is_training,
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/fhjaime966/las-pytorch/model/las_model.py", line 34, in forward
    listener_feature, ground_truth=batch_label, teacher_force_rate=teacher_force_rate
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/fhjaime966/las-pytorch/model/las_model.py", line 212, in forward
    rnn_input, hidden_state, listener_feature
  File "/home/fhjaime966/las-pytorch/model/las_model.py", line 180, in forward_step
    attention_score, context = self.attention(rnn_output, listener_feature)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/fhjaime966/las-pytorch/model/las_model.py", line 296, in forward
    * attention_score[0].unsqueeze(2).repeat(1, 1, listener_feature.size(2)),
RuntimeError: CUDA out of memory. Tried to allocate 12.00 MiB (GPU 0; 15.75 GiB total capacity; 14.69 GiB already allocated; 6.88 MiB free; 14.69 GiB reserved in total by PyTorch)

I know this is quite a big problem to debug as you would need to check the code at the repository https://github.com/jiwidi/las-pytorch to get an understanding of what is running so any help or tips with debugging gpu mem usage in pytorch are greatly appreciated.


title: How to find c++ source code of torch.bmm of pytorch

body:
I have trouble finding the source code of torch.bmm(), which is defined in https://pytorch.org/cppdocs/api/function_namespaceat_1aac51f71f807ca70fd210814114520c34.html#exhale-function-namespaceat-1aac51f71f807ca70fd210814114520c34.

I have confidence it is located in namespace at, since it is referenced as at::bmm in other places.
What I have searched through is :

The directory Aten https://github.com/pytorch/pytorch/tree/34877448216149024f44cbcab830169fdb2fa7fb/aten/src/ATen
The directory of caffe2 https://github.com/pytorch/pytorch/tree/74b65c32be68b15dc7c9e8bb62459efbfbde33d8/caffe2/core
direct search in github with keyword bmm of c++ file

but have found nothing. Is there any systematic way to locate a function(in this case, bmm) in such a large project ?


title: Why we need a decoder_start_token_id during generation in HuggingFace BART?

body:
During the generation phase in HuggingFace's code:
https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L88-L100
They pass in a decoder_start_token_id, I'm not sure why they need this. And in the BART config, the decoder_start_token_id is actually 2 (https://huggingface.co/facebook/bart-base/blob/main/config.json), which is the end of sentence token </s>.
And I tried a simple example:
from transformers import *

import torch
model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')
tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')
input_ids = torch.LongTensor([[0, 894, 213, 7, 334, 479, 2]])
res = model.generate(input_ids, num_beams=1, max_length=100)

print(res)

preds = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True).strip() for g in res]
print(preds)

The results I obtained:
tensor([[  2,   0, 894, 213,   7, 334, 479,   2]])
['He go to school.']

Though it does not affect the final "tokenization decoding" results. But it seems weird to me that the first token we generate is actually 2(</s>).


title: What is the most efficient way to broadcast an operation on slices of PyTorch Tensors?

body:
I have a tensor T of shape (b, r)
I want to do an operation for each (r), in a way that it gets parallelized by the GPU
The naive implementation, in numpy for simplicity, would look something like:
T_dash = np.array([(T[i] - np.max(T[i]) for i in range(T.size[0])])

What would be the best way to do this?


title: Pytorch Chatbot Tutorial problem: How can I solve List Index Out of Range

body:
I’m new to pytorch and have been following the many tutorials available.
But, When I did The CHATBOT TUTORIAL is not work.

Like the figure below

What should I do and what is causing this?


title: Not able to load the saved graph using torch.utils.tensorboard.SummaryWriter.add_graph method

body:
I am saving the scalar summary along with model graph using  add_scalar and add_graph methods from torch.utils.tensorboard.SummaryWriter.

While running tensorboard on the summary file, it doesnt show the model graph. Just 2 small rectangle at the bottom right, However, it is able to show the scalar variable and images.

Sample code from the pytorch documentation attached

import torch
import torchvision
from torch.utils.tensorboard import SummaryWriter
from torchvision import datasets, transforms

# Writer will output to ./runs/ directory by default
writer = SummaryWriter()

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
trainset = datasets.MNIST('mnist_train', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)
model = torchvision.models.resnet50(False)

print(model)

# Have ResNet model take in grayscale rather than RGB
model.conv1 = torch.nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)
images, labels = next(iter(trainloader))

grid = torchvision.utils.make_grid(images)
writer.add_image('images', grid, 0)
writer.add_graph(model, images)
writer.close()

Any pointer to the solution would be appreciated

  PyTorch version - 1.2.0 Tensorboard version - 1.15.0a20190828


title: How to properly implement data reorganization using PyTorch?

body:
It's going to be a long post, sorry in advance...

I'm working on a denoising algorithm and my goal is to:

Use PyTorch to design / train the model
Convert the PyTorch model into a CoreML model

The denoising algorithm consists in the following 3 parts:

A "down-sampling" + noise level map

A regular convnet

An "up-sampling"

The first part is quite simple in its idea, but not so easy to explain. Given for instance an input color image and a input value "sigma" that represents the standard deviation of the image noise.
The "down-sampling" part is in fact a space-to-depth. In short, for a given channel and for a subset of 2x2 pixels, the space-to-depth creates a single pixel composed of 4 channels. The number of channels is multiplied by 4 while the height and width are divided by 2. The data is simply reorganized.
The noise level map consists in creating 3 channels containing the standard deviation value so that the convnet knows how to properly denoise the input image.
This will be maybe more clear with some code:

def downsample_and_noise_map(input, sigma):

    # Input tensor size (batch, channels, height, width)
    in_n, in_c, in_h, in_w = input.size()

    # Output tensor size
    out_h = in_h // 2
    out_w = in_w // 2
    sigma_c = in_c      # nb of channels of the standard deviation tensor
    image_c = in_c * 4  # nb of channels of the image tensor

    # Standard deviation tensor
    output_sigma = sigma.view(1, 1, 1, 1).repeat(in_n, sigma_c, out_h, out_w)

    # Image tensor
    output_image = torch.zeros((in_n, image_c, out_h, out_w))
    output_image[:, 0::4, :, :] = input[:, :, 0::2, 0::2]
    output_image[:, 1::4, :, :] = input[:, :, 0::2, 1::2]
    output_image[:, 2::4, :, :] = input[:, :, 1::2, 0::2]
    output_image[:, 3::4, :, :] = input[:, :, 1::2, 1::2]

    # Concatenate standard deviation and image tensors
    return torch.cat((output_sigma, output_image), dim=1)

This function is then called as the first step in the model's forward function:

def forward(self, x, sigma):
    x = downsample_and_noise_map(x, sigma)
    x = self.convnet(x)
    x = upsample(x)
    return x

Let's consider an input tensor of size 1x3x100x100 (PyTorch standard: batch, channels, height, width) and a sigma value of 0.1. The output tensor has the following properties:

Tensor's shape is 1x15x50x50
Tensor's values for channels 0, 1 and 2 are all equal to sigma = 0.1
Tensor's values for channels 3, 4, 5, 6 are composed of the input image values of channel 0
Tensor's values for channels 7, 8, 9, 10 are composed of the input image values of channel 1
Tensor's values for channels 11, 12, 13, 14 are composed of the input image values of channel 2

If this code is not clear enough, I can post an even more naive version.

The up-sampling part is the reciprocal function of the downsampling one.

I was able to use this function for training and testing in PyTorch.

Then, I tried to convert the model to CoreML with ONNX as an intermediate step.
The conversion to ONNX generated "TracerWarning". Conversion from ONNX to CoreML failed (TypeError: 1.0 has type numpy.float64, but expected one of: int, long). The problem came from the down-sampling + noise level map (and from up-sampling too).

When I removed the down-sampling + noise level map and up-sampling layers, I was able to convert to ONNX and to CoreML very easily since only a simple convnet remained. This means I have a solution to my problem: implement these 2 layers using 2 shaders on the mobile side. But I'm not satisfied with this solution as I want my model to contain all layers ^^

Before considering writing a post here, I crawled Internet to find an answer and I was able to write a better version of the previous function using reshape and permute. This version removed all ONNX warning, but the CoreML conversion still failed...

def downsample_and_noise_map(input, sigma):

    # Input image size
    in_n, in_c, in_h, in_w = input.size()

    # Output tensor size
    out_n = in_n
    out_h = in_h // 2
    out_w = in_w // 2

    # Create standard deviation tensor
    output_sigma = sigma.view(out_n, 1, 1, 1).repeat(out_n, in_c, out_h, out_w)

    # Split RGB channels
    channels_rgb = torch.split(input, 1, dim=1)

    # Reshape (space-to-depth) each image channel
    channels_reshaped = []
    for channel in channels_rgb:
        channel = channel.reshape(1, out_h, 2, out_w, 2)
        channel = channel.permute(2, 4, 0, 1, 3)
        channel = channel.reshape(1, 4, out_h, out_w)
        channels_reshaped.append(channel)

    # Concatenate all reshaped image channels together
    output_image = torch.cat(channels_reshaped, dim=1)

    # Concatenate standard deviation and image tensors
    output = torch.cat([output_sigma, output_image], dim=1)

    return output

So here are (some of) my questions:

What is the preferred PyTorch way to implement a function such as downsample_and_noise_map function within a model?
Same question but when the conversion to ONNX and then to CoreML is part of the equation?
Is the PyTorch -> ONNX -> CoreML still best path to deploy the model for iOS production?

Thanks for your help (and your patience) ^^


title: Changing back to CPU with Google Colab

body:
I am running a model on Google Colab.  The final step I would like to do is print an image, and show the top 5 classification predictions of the model.  Here is the code:

image = process_image(imgpath)

index = 17
plot = imshow(image, ax = plt)
plot.axis('off')
plot.title(cat_to_name[str(index)])
plot.show()

axes = predict(image, model)

yaxis = [cat_to_name[str(i)] for i in np.array(axes[1][0])]
y_pos = np.arange(len(yaxis))
xaxis = np.array(axes[0][0])

plt.barh(y_pos, xaxis)
plt.xlabel('probability')
plt.yticks(y_pos, yaxis)
plt.title('probability of flower classification')

plt.show()

I am getting this error when I run this cell:

TypeError                                 Traceback (most recent call last)
<ipython-input-19-d0bb6f461eec> in <module>()
     11 axes = predict(image, model)
     12
---> 13 yaxis = [cat_to_name[str(i)] for i in np.array(axes[1][0])]
     14 y_pos = np.arange(len(yaxis))
     15 xaxis = np.array(axes[0][0])

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in __array__(self, dtype)
    447     def __array__(self, dtype=None):
    448         if dtype is None:
--> 449             return self.numpy()
    450         else:
    451             return self.numpy().astype(dtype, copy=False)

TypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Is there a way to temporarily use CPU on Google Colab and in this particular step?  I don't really need to switch back to GPU because this is the final step in my code.


title: how to solve the values changed when converting a pytorch Variable to numpy?

body:
I am trying to convert a parameter of resnet34 to numpy , but I found that the values would change after converting, as shown in the figure.
Why would this happen? What can I do to get precise values in numpy format?
enter image description here

( I am trying to get parameter in torch pretrained models and put them in tensorflow 1.x model , since I can’t find a pretrained resnet34 models in tensorflow1 afteing searching for several days. I am afraid this changed of values would affect the accuracy of the model. )

(BTW, is there someway to download tensorflow1.x resnet34 pretrained modles with basic block instead of bottleneck block?
I have searched in github for several days but failed to find one. I hate tensorflow.)


title: Stuck on PyTorch - RuntimeError: size mismatch

body:
I am trying to build a recommender system as a demo project work for self practice, almost on the verge but getting a runtime error of size mismatch which just doesn't seem to get over. This is the exact error code ...

  RuntimeError: size mismatch, m1: [3951 x 1], m2: [1682 x 20] at C:\w\1\s\tmp_conda_3.7_105232\conda\conda-bld\pytorch_1579085620499\work\aten\src\TH/generic/THTensorMath.cpp:136

Below is the code and output that attached. Tried changing the unsqueeze(0) to 1 & -1 but didn't help not able to find a way out in solving this error can someone please help out. Thanks :)

#CODE

import os.path
import csv
import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.optim as optim
import torch.utils.data
from torch.autograd import Variable
from pathlib import Path

# Importing the dataset
movies = pd.read_csv('ml-1m/movies.dat', sep = '::', header = None, engine = 'python', encoding = 'latin-1')
users = pd.read_csv('ml-1m/users.dat', sep = '::', header = None, engine = 'python', encoding = 'latin-1')
ratings = pd.read_csv('ml-1m/ratings.dat', sep = '::', header = None, engine = 'python', encoding = 'latin-1')

# Preparing the training set and the test set
training_set = pd.read_csv('ml-100k/u2.base', delimiter = '\t')
training_set = np.array(training_set, dtype = 'int')
test_set = pd.read_csv('ml-100k/u2.test', delimiter = '\t')
test_set = np.array(test_set, dtype = 'int')

# Getting the number of users and movies
nb_users = int(max(max(training_set[:,0]), max(test_set[:,0])))
nb_movies = int(max(max(training_set[:,1]), max(test_set[:,1])))

# Converting the data into an array with users in lines and movies in columns
def convert(data):
    new_data = []
    for id_users in range(1, nb_users + 1):
        id_movies = data[:,1][data[:,0] == id_users]
        id_ratings = data[:,2][data[:,0] == id_users]
        ratings = np.zeros(nb_movies)
        ratings[id_movies - 1] = id_ratings
        new_data.append(list(ratings))
    return new_data
training_set = convert(training_set)
test_set = convert(test_set)

# Converting the data into Torch tensors
training_set = torch.FloatTensor(training_set)
test_set = torch.FloatTensor(test_set)

class SAE(nn.Module):
    def __init__(self, ):
        super(SAE, self).__init__()
        self.fc1 = nn.Linear(nb_movies, 20)
        self.fc2 = nn.Linear(20, 10)
        self.fc3 = nn.Linear(10, 20)
        self.fc4 = nn.Linear(20, nb_movies)
        self.activation = nn.Sigmoid()
    def forward(self, x):
        x = self.activation(self.fc1(x))
        x = self.activation(self.fc2(x))
        x = self.activation(self.fc3(x))
        x = self.fc4(x)
        return x
sae = SAE()
criterion = nn.MSELoss()
optimizer = optim.RMSprop(sae.parameters(), lr = 0.01, weight_decay = 0.5)

from pathlib import Path, PureWindowsPath
filename = Path("C:/Users/MrWhoztheBoss/Documents/Movie-Recommendation-System-using-AutoEncoders-master/checkpoint.pth")
# Convert path to Windows format
path_on_windows = PureWindowsPath(filename)
print(path_on_windows)

def load_checkpoint(path_on_windows):
    checkpoint = torch.load(path_on_windows)
    model = checkpoint['model']
    model.load_state_dict(checkpoint['state_dict'])
    for parameter in model.parameters():
        parameter.requires_grad = False
    model.eval()
    return model

model = load_checkpoint('checkpoint.pth')
print(model)

movie_dict = {3709 : 'Sleepwalkers',
              2846 : 'Adventures of Milo and Otis',
              3880 : 'Ballad of Ramblin Jack',
              2971 : 'All That Jazz',
              3951 : 'Two Family House',
              3681 : 'For a Few Dollars More',
              3921 : 'Beach Party',
              3541 : 'Third World Cop',
              2189 : 'I Married A Strange Person',
              3687 : 'Light Years',
              3390 : 'Shanghai Surprise',
              2940 : 'Gilda',
              3857 : 'Bless the Child',
              1464 : 'Lost Highway',
              3376 : 'Fantastic Night-The (La Nuit Fantastique)',
              3670 : 'Story of G.I. Joe',
              3906 : 'Under Suspicion',
              792 : 'Hungarian Fairy Tale'}
count = nb_users
sid = count

with open('ml-100k/u2.base', mode='a') as file:
    for iD in movie_dict:
        print(movie_dict[iD])
        a = int(input('Enter rating:'))
        writer = csv.writer(file, delimiter='\t', lineterminator='\n')
        writer.writerow([sid,iD,a,'888692464'])

def check():
    test_loss = 0
    s = 0.
    for id_user in range(nb_users):
        input = Variable(training_set[id_user]).unsqueeze(0)
        output = model(input)
        lol = output.detach().numpy()
        loll = lol[0]
        for i in loll:
            if(i>0):
                print(id_user,':',round(i))

check()

#OUTPUT

C:\Users\MrWhoztheBoss\Documents\Movie-Recommendation-System-using-AutoEncoders-master\checkpoint.pth
SAE(
  (fc1): Linear(in_features=1682, out_features=20, bias=True)
  (fc2): Linear(in_features=20, out_features=10, bias=True)
  (fc3): Linear(in_features=10, out_features=20, bias=True)
  (fc4): Linear(in_features=20, out_features=1682, bias=True)
  (activation): Sigmoid()
)
Sleepwalkers
Enter rating:3
Adventures of Milo and Otis
Enter rating:3
Ballad of Ramblin Jack
Enter rating:3
All That Jazz
Enter rating:3
Two Family House
Enter rating:3
For a Few Dollars More
Enter rating:3
Beach Party
Enter rating:3
Third World Cop
Enter rating:3
I Married A Strange Person
Enter rating:3
Light Years
Enter rating:3
Shanghai Surprise
Enter rating:3
Gilda
Enter rating:3
Bless the Child
Enter rating:3
Lost Highway
Enter rating:3
Fantastic Night-The (La Nuit Fantastique)
Enter rating:3
Story of G.I. Joe
Enter rating:3
Under Suspicion
Enter rating:3
Hungarian Fairy Tale
Enter rating:3
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-25c653a68ff6> in <module>
    124
    125
--> 126 check()

<ipython-input-1-25c653a68ff6> in check()
    116     for id_user in range(nb_users):
    117         input = Variable(training_set[id_user]).unsqueeze(dim=-1)
--> 118         output = model(input)
    119         lol = output.detach().numpy()
    120         loll = lol[0]

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

<ipython-input-1-25c653a68ff6> in forward(self, x)
     53         self.activation = nn.Sigmoid()
     54     def forward(self, x):
---> 55         x = self.activation(self.fc1(x))
     56         x = self.activation(self.fc2(x))
     57         x = self.activation(self.fc3(x))

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~\Anaconda3\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
     85
     86     def forward(self, input):
---> 87         return F.linear(input, self.weight, self.bias)
     88
     89     def extra_repr(self):

~\Anaconda3\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1368     if input.dim() == 2 and bias is not None:
   1369         # fused op is marginally faster
-> 1370         ret = torch.addmm(bias, input, weight.t())
   1371     else:
   1372         output = input.matmul(weight.t())

RuntimeError: size mismatch, m1: [1682 x 1], m2: [1682 x 20] at C:\w\1\s\tmp_conda_3.7_105232\conda\conda-bld\pytorch_1579085620499\work\aten\src\TH/generic/THTensorMath.cpp:136


title: pytorch training loop ends with ''int' object has no attribute 'size' exception

body:
The code I am posting below is just a small part of the application:
def train(self, training_reviews, training_labels):

        # make sure out we have a matching number of reviews and labels
        assert(len(training_reviews) == len(training_labels))

        # Keep track of correct predictions to display accuracy during training
        correct_so_far = 0

        # Remember when we started for printing time statistics
        start = time.time()

        criterion = nn.CrossEntropyLoss()
        optimizer =  torch.optim.SGD(self.parameters(), lr=self.learning_rate)

        # loop through all the given reviews and run a forward and backward pass,
        # updating weights for every item
        for i in range(len(training_reviews)):

            # TODO: Get the next review and its correct label
            review = training_reviews[i]
            label = training_labels[i]
            print('processing item ',i)
            self.update_input_layer(review)
            output = self.forward(torch.from_numpy(self.layer_0).float())
            target = self.get_target_for_label(label)
            print('output ',output)
            print('target ',target)
            loss = criterion(output, target)

...
mlp = SentimentNetwork(reviews[:-1000],labels[:-1000], learning_rate=0.1)
mlp.train(reviews[:-1000],labels[:-1000])

and it ends with the exception in the title line when evaluating:
loss = criterion(output, target)

prior to that, the variables are as follows:
output  tensor([[0.5803]], grad_fn=<SigmoidBackward>)
target  1


title: values in torch.nn.conv2d and torch.nn.Linear

body:
I am confiused how to get the out_channels in torch.nn.Conv2d and the in_features, out_features in torch.nn.Linear.
For example I have a non-color 28*28 image input.
the in_channels = 1,kernel_size=5,padding=2 how can I figure the out_channels.
After the convolutional, I want to make a linear layer.
How do I figure the values of in_features, out_features ?


title: Jupyter Kernel crash/dies when use large Neural Network layer, any idea pls?

body:
I am experimenting Autoencoder with Pytorch. It seems when I use relatively larger neural network for instance nn.Linear(250*250, 40*40) as the first layer, the Jupyter kernel keep crashing. when I use smaller layer size e.g. nn.Linear(250*250, 20*20). the Jupyter kernel is ok. Any idea how to fix this? So I can run larger network.Thank you. The entire network is as below.

# model:
class AutoEncoder(nn.Module):
    def __init__(self):
        super().__init__()

        self.encoder = nn.Sequential(
            nn.Linear(250*250, 20*20),
            nn.BatchNorm1d(20*20,momentum=0.5),
            nn.Dropout(0.5),
            nn.LeakyReLU(),
            nn.Linear(20*20, 20*20),
            nn.BatchNorm1d(20*20,momentum=0.5),
            nn.Dropout(0.5),
            nn.LeakyReLU(),
            nn.Linear(20*20, 20*20),
            nn.BatchNorm1d(20*20,momentum=0.5),
            nn.Dropout(0.5),
            nn.LeakyReLU(),
            nn.Linear(20*20, 15*15),
            nn.BatchNorm1d(15*15,momentum=0.5),
            nn.Dropout(0.5),
            nn.LeakyReLU(),
            nn.Linear(15*15, 3),
            nn.BatchNorm1d(3,momentum=0.5),
            #nn.Dropout(0.5),
            #nn.Tanh(),
            #nn.Linear(5*5,5),
        )
        self.decoder = nn.Sequential(
            #nn.Linear(5, 5*5),
            #nn.BatchNorm1d(5*5,momentum=0.5),
            #nn.Dropout(0.5),
            #nn.Tanh(),
            nn.Linear(3, 15*15),
            nn.BatchNorm1d(15*15,momentum=0.5),
            nn.Dropout(0.5),
            nn.LeakyReLU(),
            nn.Linear(15*15, 20*20),
            nn.BatchNorm1d(20*20,momentum=0.5),
            nn.Dropout(0.5),
            nn.LeakyReLU(),
            nn.Linear(20*20, 20*20),
            nn.BatchNorm1d(20*20,momentum=0.5),
            nn.Dropout(0.5),
            nn.LeakyReLU(),
            nn.Linear(20*20, 250*250),
            nn.BatchNorm1d(250*250,momentum=0.5),
            nn.Dropout(0.5),
            nn.Sigmoid(),
        )
    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return encoded, decoded


title: Pytorch method for conditional use of intermediate layer instead of final cnn layer output. ie: allow nn to learn to use more layers or less

body:
I'm implementing a residual cnn(modified smaller version of xception) in a low latency environment. I've done a lot of manual tuning to minimize the run time speed of my network (reducing number of filters, removing layers, etc).

But now I want to try allowing my network to make its classification prediction(final fcnn layer) on the residual connection after each residual block.

basic logic-

attempt final prediction with residual connection as input

if this fcnn layer predicts a certain class with a probability > a set threshold:

return fcnn output as if it was normal final layer

else:

do next residual block like normal and try the previous conditional again unless we are already at final block

My hope is this will allow my network to learn to solve easier problems with less computation while allowing it to still do the additional layers if it is still unsure of the classification.

So my basic question is: In pytorch, whats the best way to implement this conditional in a way that allows my nn at run time to decide whether to do more processing or not

Currently Ive tested returning the intermediate x's after the blocks in the forward function, but I dont know how best to setup the conditional to chose which x to return

Also side note: I believe I may end up needing another cnn layer between the residual and fcnn to serve as a function to convert the internal representation for processing to a representation the fcnn understands for classification.


title: What is the meaning of keep_vars in state_dict?

body:
state_dict(destination=None, prefix='', keep_vars=False)

what does changing keep_vars to True do?


title: Validation loss lower than training loss but worse accuracy

body:
I am training a FFNN for classification and wonder why my validation loss always seems to be to low compared to training loss, as accuracy is also worse for validation than training.

I found some similar questions leading to the point that it's per se possible to have better performance in validation data set than in the training set, but not in the scenario of lower loss AND lower accuracy.

here's the code i use for training my pytorch NN including the loss calculation:

optimizer = optim.Adam(model_pyt.parameters(), lr=learning_rate, betas=(0.9,0.999))
criterion = nn.CrossEntropyLoss()

for epoch in range(epochs):
    start_time = time.time()
    train_running_loss = 0
    train_acc = 0
    with torch.set_grad_enabled(True):
        for i, data_pack in enumerate(training_generator):
            x_data, y_data = data_pack
            optimizer.zero_grad()
            outputs = model_pyt(x_data)
            loss = criterion(outputs, y_data)
            loss.backward()
            optimizer.step()
            train_running_loss += loss.detach().item()
            train_acc += get_accuracy(outputs, y_data, batch_size)
    test_labels = torch.tensor(labels_test).long()
    test_inputs = torch.tensor(np.array(data_bal_test)).float()
    test_outputs = model_pyt(test_inputs)
    test_loss = criterion(test_outputs, test_labels).detach().item()
    test_acc = get_accuracy(test_outputs, test_labels, len(test_labels))
    print('Epoch:  %d | Loss: %.4f | Acc %.4f | Test-Loss: %.4f | Test-Acc %.4f | Time Elapsed: %s'
          %(epoch+1, train_running_loss/(i+1), train_acc/(i+1), loss, test_acc, time_since(start_time)))
    print('=====================================================================================================')


title: How do I separate the input and targets from Pytorch Fashion MNIST?

body:
The Fashion MNIST dataset is implemented pretty weirdly in Pytorch. I want to do something like:

X, y = FashionMNIST

But in reality, it's a little more complicated. This is what I have:

from torchvision.datasets import FashionMNIST
train = FashionMNIST(root='.', download=True, train=True)
print(train)

The output:

Dataset FashionMNIST
    Number of datapoints: 60000
    Root location: c:/users/nicolas/documents/data/fashionmnist
    Split: Train

What one observation looks like:

print(train[0])

(<PIL.Image.Image image mode=L size=28x28 at 0x20868074780>, 9)

I could only do it for one observation.

X, y = train[0]

So how do I separate the input and targets?


title: How to read numerical data from CSV in PyTorch?

body:
I'm new to PyTorch; trying to implement a model I developed in TF and compare the results. The model is an Autoencoder model. The input data is a csv file including n samples each with m features (a n*m numerical matrix in a csv file). The targets (the labels) are in another csv file with the same format as the input file. I've been looking online but couldn't find a good documentation for reading non-image data from csv file with multiple labels. Any idea how can I read my data and iterate over it during training?

Thank you


title: How to pass identifier of torch layers to ONNX graph

body:
For TensorFlow, the name of the layer is also the name of the nodes in the ONNX graph:
import onnx
from tensorflow import keras
import keras2onnx

model = keras.Sequential([
    keras.layers.Conv2D(4, (3, 3), input_shape=(5, 5, 2), name='rainbow'),
    keras.layers.Conv2D(8, (3, 3), name='flower')])

model_onnx = keras2onnx.convert_keras(model)
nodes = model_onnx.graph.node

for node in nodes:
    print('type:', node.op_type, '- name:', node.name)

with output:
type: Conv - name: rainbow
type: Conv - name: flower

However, for PyTorch the name field is an empty string:
import torch
from torch import nn
import onnx

class NetConv(nn.Module):
    def __init__(self):
        super(NetConv, self).__init__()
        self.conv_banana = nn.Conv2d(2, 4, 3)
        self.conv_apple = nn.Conv2d(4, 8, 3)

    def forward(self, x):
        x = self.conv_banana(x)
        return self.conv_apple(x)

model = NetConv()
image = torch.randn(1, 2, 5, 5)

filename_model = 'model.onnx'
torch.onnx.export(model, image, filename_model, export_params=False)
model_onnx = onnx.load(filename_model)
nodes = model_onnx.graph.node

for node in nodes:
    print('type:', node.op_type, '- name:', node.name)
    print('inputs:', node.input)

with output:
type: Conv - name:
inputs: ['input.1', 'conv_banana.weight', 'conv_banana.bias']
type: Conv - name:
inputs: ['5', 'conv_apple.weight', 'conv_apple.bias']

I would have expected the layer names to be assigned to the name attribute of the graph nodes, but this attribute is ''.
Is there any clean way to get an identifier such that layers in PyTorch could be assigned to nodes in the ONNX graph?
The only workaround I could think of is extracting the name of the layer from the input of the node in the graph, but this kind of hacky.
Update:
The above output was generated with torch==1.4.0.
For torch==1.6.0, the name attribute is not a empty string anymore, but also not more useful than an empty string:
type: Conv - name: Conv_0
inputs: ['input.1', 'conv_banana.weight', 'conv_banana.bias']
type: Conv - name: Conv_1
inputs: ['5', 'conv_apple.weight', 'conv_apple.bias']


title: Torchtext BucketIterator wrapper from tutorial produces SyntaxError

body:
I am following and implementing code from this short tutorial on Torchtext, which is surprisingly clear given the poor documentation of Torchtext.

When the Iterator has been created (the batch generator) he proposes to create a wrapper to produce more reusable code. (See step 5 in the tutorial).

The code contains a surprisingly long and weird line, which I don't understand and which raises a SyntaxError: invalid syntax. Does anyone have a clue of what is going on?

(The problematic line is the one that starts with: if self.y_vars is &amp;amp;amp;lt;g [...])

class BatchWrapper:
  def __init__(self, dl, x_var, y_vars):
        self.dl, self.x_var, self.y_vars = dl, x_var, y_vars # we pass in the list of attributes for x &amp;amp;amp;amp;lt;g class="gr_ gr_3178 gr-alert gr_spell gr_inline_cards gr_disable_anim_appear ContextualSpelling ins-del" id="3178" data-gr-id="3178"&amp;amp;amp;amp;gt;and y&amp;amp;amp;amp;lt;/g&amp;amp;amp;amp;gt;

  def __iter__(self):
        for batch in self.dl:
              x = getattr(batch, self.x_var) # we assume only one input in this wrapper

              if self.y_vars is &amp;amp;amp;amp;lt;g class="gr_ gr_3177 gr-alert gr_gramm gr_inline_cards gr_disable_anim_appear Grammar replaceWithoutSep" id="3177" data-gr-id="3177"&amp;amp;amp;amp;gt;not&amp;amp;amp;amp;lt;/g&amp;amp;amp;amp;gt; None: # we will concatenate y into a single tensor
                    y = torch.cat([getattr(batch, feat).unsqueeze(1) for feat in self.y_vars], dim=1).float()
              else:
                    y = torch.zeros((1))

              yield (x, y)

  def __len__(self):
        return len(self.dl)


title: Pytorch: How to assign a default value for look-up table using torch tensor

body:
Say that I have two tensors as follows:

a = torch.tensor([[1, 2, 3], [1, 2, 3]])
b = torch.tensor([0, 2, 3, 4])

where b is the lookup value for a such as:

b[a]

will return the value of:

tensor([[2, 3, 4], [2, 3, 4]])

My problem is, what if I only have a look-up table of:

c = torch.tensor([0, 2, 3])

In which, for every out-of-index, I would like it to be assigned to index 0, such as c[a] will return

tensor([[2, 3, 0], [2, 3, 0]])

If I run c[a], of course, I will get this result:

RuntimeError: index 3 is out of bounds for dim with size 3

Thanks for your help.


title: Implementing self attention

body:
I am trying to implement self attention in Pytorch.
I need to calculate the following expressions.

Similarity function S (2 dimensional), P(2 dimensional), C'

S[i][j] = W1 * inp[i] + W2 * inp[j] + W3 * x1[i] * inp[j]

P[i][j] = e^(S[i][j]) / Sum for all j( e ^ (S[i]))

basically, P is a softmax function

C'[i] = Sum (for all j)  P[i][j] * x1[j]

I tried the following code using for loops

        for i in range(self.dim):
            for j in range(self.dim):
                S[i][j] = self.W1 * x1[i] + self.W2 * x1[j] + self.W3 * x1[i] * x1[j]

        for i in range(self.dim):
            for j in range(self.dim):
                P[i][j] = torch.exp(S[i][j]) / torch.sum( torch.exp(S[i]))

        # attend

        for i in range(self.dim):
            out[i] = 0
            for j in range(self.dim):
                out[i] += P[i][j] * x1[j]

Is there any faster way to implement this in Pytorch?


title: Memory efficient batch-pairwise KL divergence in pytorch

body:
Given a Batch tensor S of dimensions BxD where each row is a probability distribution over D dimensions, I want to calculate the batch-pairwise KL divergence matrix KL (of dimensions BxB) such that KL[i,j] = KL-DIVERGENCE(S[i],S[j]).

This works:

softmax = torch.nn.Softmax(dim = 1)
B,D = 100, 1000
S = softmax(torch.rand(B,D))
KL = torch.mean(S[:,None,:] * (S[:,None,:]/S).log(),dim = 2)

but is very memory inefficient. Is there a way to calculate this matrix more efficiently?

Thanks.


title: pysyft torrch.jit. script RuntimeError: undefined value _Reduction

body:
I was trying to reproduce Pysyft Asynchronous-federated-learning-on-MNIST from its advanced example. where @torch.jit.script is used before loss function. I am getting this error and have no clue what this is about

  RuntimeError: undefined value _Reduction: at
  /home/ab/.virtualenvs/aic/lib/python3.6/site-packages/syft/generic/frameworks/hook/hook.py:1829:20

  reduction = _Reduction.legacy_get_string(size_average, reduce)

It is actually caused by these lines

@torch.jit.script
def loss_fn(pred, target):
    return F.nll_loss(input=pred, target=target)

train_config = sy.TrainConfig(
        model=traced_model,
        loss_fn=loss_fn,
        batch_size=batch_size,
        shuffle=True,
        max_nr_batches=max_nr_batches,
        epochs=1,
        optimizer="SGD",
        optimizer_args={"lr": lr},
    )


title: L-BFGS-B code, Scipy (sciopt.fmin_l_bfgs_b(func, init_guess, maxiter=10, bounds=list(bounds), disp=1, iprint=101))

body:
I'm using the L-BFGS-B optimizer to find the minima of a function. This will help me calculate sharpness for the function. However, I'm not sure if this following message is considered a normal message (i.e. Is there something wrong with my program or is this message typical?) See below:

RUNNING THE L-BFGS-B CODE

           * * *

Machine precision = 2.220D-16
 N =     28149514     M =           10

At X0         0 variables are exactly at the bounds
^[[C
At iterate    0    f= -3.59325D+00    |proj g|=  2.10249D-03

At iterate    1    f= -2.47853D+01    |proj g|=  4.20499D-03

 Bad direction in the line search;
   refresh the lbfgs memory and restart the iteration.

At iterate    2    f= -2.53202D+01    |proj g|=  4.17686D-03

At iterate    3    f= -2.53202D+01    |proj g|=  4.17686D-03

           * * *

Tit   = total number of iterations
Tnf   = total number of function evaluations
Tnint = total number of segments explored during Cauchy searches
Skip  = number of BFGS updates skipped
Nact  = number of active bounds at final generalized Cauchy point
Projg = norm of the final projected gradient
F     = final function value

           * * *

   N    Tit     Tnf  Tnint  Skip  Nact     Projg        F
*****      3     43 ******     0 *****   4.177D-03  -2.532D+01
  F =  -25.320247650146484

CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH

 Warning:  more than 10 function and gradient
   evaluations in the last line search.  Termination
   may possibly be caused by a bad search direction.

I got the following sharpness anyway which is relatively consistent with the paper I'm trying to reproduce: It's just that I'm a bit concerned with the above message.
tensor(473.0201)

Here is my code for computing sharpness:

def get_sharpness(data_loader, model, criterion, epsilon, manifolds=0):

  # extract current x0
  x0 = None
  for p in model.parameters():
    if x0 is None:
      x0 = p.data.view(-1)
    else:
      x0 = torch.cat((x0, p.data.view(-1)))
  x0 = x0.cpu().numpy()

  # get current f_x
  f_x0, _ = get_minus_cross_entropy(x0, data_loader, model, criterion)
  f_x0 = -f_x0
  logging.info('min loss f_x0 = {loss:.4f}'.format(loss=f_x0))

  # find the minimum
  if 0==manifolds:
    x_min = np.reshape(x0 - epsilon * (np.abs(x0) + 1), (x0.shape[0], 1))
    x_max = np.reshape(x0 + epsilon * (np.abs(x0) + 1), (x0.shape[0], 1))
    bounds = np.concatenate([x_min, x_max], 1)
    func = lambda x: get_minus_cross_entropy(x, data_loader, model, criterion, training=True)
    init_guess = x0
  else:
    warnings.warn("Small manifolds may not be able to explore the space.")
    assert(manifolds<=x0.shape[0])
    #transformer = rp.GaussianRandomProjection(n_components=manifolds)
    #transformer.fit(np.random.rand(manifolds, x0.shape[0]))
    #A_plus = transformer.components_
    #A = np.linalg.pinv(A_plus)
    A_plus = np.random.rand(manifolds, x0.shape[0])*2.-1.
    # normalize each column to unit length
    A_plus_norm = np.linalg.norm(A_plus, axis=1)
    A_plus = A_plus / np.reshape(A_plus_norm, (manifolds,1))
    A = np.linalg.pinv(A_plus)
    abs_bound = epsilon * (np.abs(np.dot(A_plus, x0))+1)
    abs_bound = np.reshape(abs_bound, (abs_bound.shape[0], 1))
    bounds = np.concatenate([-abs_bound, abs_bound], 1)
    def func(y):
      floss, fg = get_minus_cross_entropy(x0 + np.dot(A, y), data_loader, model, criterion, training=True)
      return floss, np.dot(np.transpose(A), fg)
    #func = lambda y: get_minus_cross_entropy(x0+np.dot(A, y), data_loader, model, criterion, training=True)
    init_guess = np.zeros(manifolds)

  #rand_selections = (np.random.rand(bounds.shape[0])+1e-6)*0.99
  #init_guess = np.multiply(1.-rand_selections, bounds[:,0])+np.multiply(rand_selections, bounds[:,1])

  minimum_x, f_x, d = sciopt.fmin_l_bfgs_b(func, init_guess, maxiter=10, bounds=list(bounds), disp=1, iprint=101)
    #factr=10.,
    #pgtol=1.e-12,

  f_x = -f_x
  logging.info('max loss f_x = {loss:.4f}'.format(loss=f_x))
  sharpness = (f_x - f_x0)/(1+f_x0)*100
  print(sharpness)

  # recover the model
  x0 = torch.from_numpy(x0).float()
  x0 = x0.cuda()
  x_start = 0
  for p in model.parameters():
      psize = p.data.size()
      peltnum = 1
      for s in psize:
          peltnum *= s
      x_part = x0[x_start:x_start + peltnum]
      p.data = x_part.view(psize)
      x_start += peltnum

  return sharpness

Which was taken from this repository:
https://github.com/wenwei202/smoothout/blob/master/measure_sharpness.py

I'm concerned about exact accuracy.


title: Is A PyTorch Dataset Accessed by Multiple DataLoader Workers?

body:
When using more than 1 DataLoader workers in PyTorch, does every worker access the same Dataset instance? Or does each DataLoader worker have their own instance of Dataset?

from torch.utils.data import DataLoader, Dataset

class NumbersDataset(Dataset):
    def __init__(self):
        self.samples = list(range(1, 1001))

    def __len__(self):
        return len(self.samples)

    def __getitem__(self, idx):
        return self.samples[idx]

dataset = NumbersDataset()
train_loader = DataLoader(dataset, num_workers=4)


title: Object detection in detectron2 using pytorch on google colab. Reuse already trained model or import existing trained model and predict the objects

body:
First I downloaded the output folder of the trained model and imported it in a new project on the google colab server.
In a new project without training the model, I have given the path of model_final.pth of the existing output folder to cfg.MODEL.WEIGHTS =/content/output/model_final.pth. but goes in an infinite loop.
3.I change the model weights cfg.MODEL.WEIGHTS = "detectron2://COCO-Detection/faster_rcnn_R_101_FPN_3x/137851257/model_final_f6e8b1.pkl". but still it doesn't predict objects.
I change the model weights path and gave the previously trained model metrics JSON file still it not working
cfg.MODEL.WEIGHTS=/content/output/metrics.json 5.By using DetectionCheckpointer(model).load("/content/output/model_final.pth") DetectionCheckpointer(model).load("detectron2://COCO-Detection/faster_rcnn_R_101_FPN_3x/137851257/model_final_f6e8b1.pkl")
it gives an error model is not defined.
what is this model_final.pkl file? and where did we get it?
what should we do to import the existing train model and predict the objects in the new project?

cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5
cfg.DATASETS.TEST = ("microcontroller_test", )
predictor = DefaultPredictor(cfg)

Above code goes in an infinite loop
DetectionCheckpointer(model).load("/content/output/model_final.pth")
DetectionCheckpointer(model).load("detectron2://COCO-Detection/faster_rcnn_R_101_FPN_3x/137851257/model_final_f6e8b1.pkl")

Error:
NameError                                 Traceback (most recent call last)
<ipython-input-12-69f2a7846756> in <module>()
----> 1 DetectionCheckpointer(model).load("/content/output/model_final.pth")
      2
      3 DetectionCheckpointer(model).load("detectron2://COCO-Detection/faster_rcnn_R_101_FPN_3x/137851257/model_final_f6e8b1.pkl")

NameError: name 'model' is not defined


title: How to add GPU computation for the CIFAR 10 pytorch Tutorial?

body:
The cifar 10 tutorial for pytorch can be found here:
https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py

The tutorial goes through how to use pytorch to make a simple convolutional neural network for the CIFAR 10 dataset. Near the end, it slightly goes through how to implement the above code for GPU.

Essentially the difference is that we create the device =torch.device('gpu') and send all the tensors to the gpu device before computing with them. So essentially, we want the net object and the input and labels tensors to be sent to the gpu device before doing the computations each time.

So I went back and added:

net.to(device)

And in the loop of computation:

with torch.no_grad():
    for data in testloader:
        images, labels = data
        #I added this line of change:
        inputs, labels = inputs.to(device), labels.to(device)
        outputs = net(images)
        _, predicted = torch.max(outputs, 1)
        c = (predicted == labels).squeeze()
        for i in range(4):
            label = labels[i]
            class_correct[label] += c[i].item()
            class_total[label] += 1

However, this caused an error: OSError: [Errno 12] Cannot allocate memory

What am i misunderstanding? Do I need to make any other changes so the whole tutorial can work for GPU?


title: Why pytorch transformer src_mask doesn't block positions from attending?

body:
I am trying to train word embedding with transformer encoder by masking the word itself with diagonal src_mask:
def _generate_square_subsequent_mask(self, sz):
    mask = torch.diag(torch.full((sz,),float('-inf')))
    return mask

def forward(self, src):

    if self.src_mask is None or self.src_mask.size(0) != len(src):
        device = src.device
        mask = self._generate_square_subsequent_mask(len(src)).to(device)
        self.src_mask = mask

    src = self.embedding(src) * math.sqrt(self.ninp)
    src = self.dropout(src)
    src = self.pos_encoder(src)
    src = self.transformer_encoder(src, self.src_mask)
    output = self.decoder(src) # Linear layer
    return output

After training the model predicts exactly the same sentence from the input. If I change any word in the input - it predict the new word. So the model doesn't block according to the mask.
Why is it ?
I understand that there is a mistake in my logic because BERT would probably be much simpler if it worked. But where am I wrong ?
Edit:
I am using the a sequence of word indices as input. Output is the same sequence as input.