title: Pytorch torchvision FasionMNIST dataset won't load

body:
I'm on a Windows 10 machine running vanilla Python 3.9.5 using IDLE. I'm following along with some Pytorch tutorials found here. Everything is working fine until Datasets & Dataloaders when I try to download the FashionMNIST data set.
I first installed the torchvision package with pip.
pip install torchvision===0.9.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html

Below are the packages I have installed.
Package           Version
----------------- -----------
cycler            0.10.0
keyboard          0.13.5
kiwisolver        1.3.1
matplotlib        3.4.1
MouseInfo         0.1.3
mss               6.1.0
numpy             1.20.2
opencv-python     4.5.2.52
Pillow            8.2.0
pip               21.1.1
PyAutoGUI         0.9.52
PyGetWindow       0.0.9
PyMsgBox          1.0.9
pyparsing         2.4.7
pyperclip         1.8.2
PyRect            0.1.4
PyScreeze         0.1.27
python-dateutil   2.8.1
PyTweening        1.0.3
setuptools        56.0.0
six               1.16.0
torch             1.8.1+cu111
torchaudio        0.8.1
torchvision       0.9.1+cu111
typing-extensions 3.10.0.0

I try to run the following script.
import torch
from torch.utils.data import Dataset
from torchvision import datasets
from torchvision.transforms import ToTensor
import matplotlib.pyplot as plt

training_data = datasets.FashionMNIST(
    root="C:/../../../Python/LearningPytorch/data",
    train=True,
    download=True,
    transform=ToTensor()

)

test_data = datasets.FashionMNIST(
    root="C:/../../../Python/LearningPytorch/data",
    train=False,
    download=True,
    transform=ToTensor()
)

Everything seems to start out fine with expected output.
Python 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.
>>>
= RESTART: C:/../../../Python/LearningPytorch/DatasetsAndDataLoaders.py
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz to C:/../../../Python/LearningPytorch/data\FashionMNIST\raw\train-images-idx3-ubyte.gz

= RESTART: C:/../../../Python/LearningPytorch/DatasetsAndDataLoaders.py
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz to C:/../../../Python/LearningPytorch/data\FashionMNIST\raw\train-images-idx3-ubyte.gz

0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.1%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.2%0.3%0.3%0.3%0.3%0.3%0.3%0.3%0.3%0.3%0.3%0.3%0.3%0.3%0.3%0.3%0.3%...

Around a half hour, at about the fifty percent mark of the download, it fails. I have great internet, 500 Mbps down throughout the process.
I am getting the following error message.
Traceback (most recent call last):
  File "C:\..\..\..\DatasetsAndDataLoaders.py", line 8, in <module>
    training_data = datasets.FashionMNIST(
  File "C:\..\..\..\Python\Python39\lib\site-packages\torchvision\datasets\mnist.py", line 85, in __init__
    self.download()
  File "C:\..\..\..\Python\Python39\lib\site-packages\torchvision\datasets\mnist.py", line 155, in download
    download_and_extract_archive(
  File "C:\..\..\..\Python\Python39\lib\site-packages\torchvision\datasets\utils.py", line 316, in download_and_extract_archive
    download_url(url, download_root, filename, md5)
  File "C:\..\..\..\Python\Python39\lib\site-packages\torchvision\datasets\utils.py", line 142, in download_url
    raise e
  File "C:\..\..\..\Python\Python39\lib\site-packages\torchvision\datasets\utils.py", line 134, in download_url
    _urlretrieve(url, fpath)
  File "C:\..\..\..\Python\Python39\lib\site-packages\torchvision\datasets\utils.py", line 31, in _urlretrieve
    for chunk in iter(lambda: response.read(chunk_size), ""):
  File "C:\..\..\..\Python\Python39\lib\site-packages\torchvision\datasets\utils.py", line 31, in <lambda>
    for chunk in iter(lambda: response.read(chunk_size), ""):
  File "C:\..\..\..\Python\Python39\lib\http\client.py", line 455, in read
    n = self.readinto(b)
  File "C:\..\..\..\Python\Python39\lib\http\client.py", line 499, in readinto
    n = self.fp.readinto(b)
  File "C:\..\..\..\Python\Python39\lib\socket.py", line 704, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

A portion of a single downloaded gzip file can be found at C:......\Python\LearningPytorch\data\FashionMNIST\raw
data/FashionMNIST/raw/train-images-idx3-ubyte.gz
I try to download from other sources, replacing just the data in the folder structure that was created by the download process. This doesn't work either.
What is going wrong here?


title: Running inference using a custom PyTorch model

body:
I am pretty new to using PyTorch and using custom models in general. But recently, I started working on training a custom-built model for detecting skin lesions. I found some code online, and I ran it all on Google Colab, and it works ( accuracy = 92%).  I attached the model training part below.
 But now I would like to run inference using this model to see how it classifies images. I tried using some tutorials on PyTorch to convert this model to onnx format. But I could not understand how to do it. How should I convert this PyTorch model to onxx? Or is there another way I can run inference using this?

Please let me know if you need any additional information. Thank you so much in advance.
Class AverageMeter(object):
    def __init__(self):
        self.reset()

    def reset(self):
        self.val = 0
        self.avg = 0
        self.sum = 0
        self.count = 0

    def update(self, val, n=1):
        self.val = val
        self.sum += val * n
        self.count += n
        self.avg = self.sum / self.count

total_loss_train, total_acc_train = [],[]
def train(train_loader, model, criterion, optimizer, epoch):
    model.train()
    train_loss = AverageMeter()
    train_acc = AverageMeter()
    curr_iter = (epoch - 1) * len(train_loader)
    for i, data in enumerate(train_loader):
        images, labels = data
        N = images.size(0)
        # print('image shape:',images.size(0), 'label shape',labels.size(0))
        images = Variable(images).to(device)
        labels = Variable(labels).to(device)

        optimizer.zero_grad()
        outputs = model(images)

        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        prediction = outputs.max(1, keepdim=True)[1]
        train_acc.update(prediction.eq(labels.view_as(prediction)).sum().item()/N)
        train_loss.update(loss.item())
        curr_iter += 1
        if (i + 1) % 100 == 0:
            print('[epoch %d], [iter %d / %d], [train loss %.5f], [train acc %.5f]' % (
                epoch, i + 1, len(train_loader), train_loss.avg, train_acc.avg))
            total_loss_train.append(train_loss.avg)
            total_acc_train.append(train_acc.avg)
    return train_loss.avg, train_acc.avg

def validate(val_loader, model, criterion, optimizer, epoch):
    model.eval()
    val_loss = AverageMeter()
    val_acc = AverageMeter()
    with torch.no_grad():
        for i, data in enumerate(val_loader):
            images, labels = data
            N = images.size(0)
            images = Variable(images).to(device)
            labels = Variable(labels).to(device)

            outputs = model(images)
            prediction = outputs.max(1, keepdim=True)[1]

            val_acc.update(prediction.eq(labels.view_as(prediction)).sum().item()/N)

            val_loss.update(criterion(outputs, labels).item())

    print('------------------------------------------------------------')
    print('[epoch %d], [val loss %.5f], [val acc %.5f]' % (epoch, val_loss.avg, val_acc.avg))
    print('------------------------------------------------------------')
    return val_loss.avg, val_acc.avg
epoch_num = 10
best_val_acc = 0
total_loss_val, total_acc_val = [],[]

for epoch in range(1, epoch_num+1):
    loss_train, acc_train = train(train_loader, model, criterion, optimizer, epoch)
    loss_val, acc_val = validate(val_loader, model, criterion, optimizer, epoch)
    total_loss_val.append(loss_val)
    total_acc_val.append(acc_val)
    if acc_val > best_val_acc:
        best_val_acc = acc_val
        print('*****************************************************')
        print('best record: [epoch %d], [val loss %.5f], [val acc %.5f]' % (epoch, loss_val, acc_val))
        print('*****************************************************')


title: in collate_batch(self, features)->ValueError: only one element tensors can be converted to Python scalars

body:
I am trying to use transformers on commonsense_qa dataset. I want to use data as

question, option1
question, option2, etc.
for each option I will pass question also for every input. For input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, I do the following.
max_length = 128

def convert_to_commonsense_qa_features(example_batch):
    num_examples = len(example_batch["question"])
    num_choices = len(example_batch["choices"][0]["text"])
    features, features2 = {}, {}
    labels2id = {char: i for i, char in enumerate("ABCDE")}
    for example_i in range(num_examples):
        choices_inputs = tokenizer.batch_encode_plus(
            list(zip(
                [example_batch["question"][example_i]] * num_choices,
                example_batch["choices"][example_i]["text"],
            )),
            max_length=max_length, pad_to_max_length=True,
        )
        for k, v in choices_inputs.items():
            if k not in features:
                features[k] = []
            features[k].append(v)
        choices_outputs = tokenizer.batch_encode_plus(
                      str(labels2id[example_batch["answerKey"][example_i]]),
                      max_length=max_length, pad_to_max_length=True,
        )
        for k, v in choices_outputs.items():
            if k not in features2:
                features2[k] = []
            features2[k].append(v)
    labels2id = {char: i for i, char in enumerate("ABCDE")}

    features["decoder_input_ids"] = features2["input_ids"]
    features["decoder_attention_mask"] = features2["attention_mask"]
    # Dummy answers for test
    """
    if example_batch["answerKey"][0]:
        features["labels"] = [labels2id[ans] for ans in example_batch["answerKey"]]
    else:
        features["labels"] = [0] * num_examples
    """
    features["labels"] = features2["input_ids"]
    #print(features)
    return features

convert_func_dict = {
    "commonsense_qa": convert_to_commonsense_qa_features,
}

columns_dict = {
    "commonsense_qa": ['input_ids', 'attention_mask', 'decoder_input_ids', 'decoder_attention_mask', 'labels'],
}

features_dict = {}
for task_name, dataset in dataset_dict.items():
    features_dict[task_name] = {}
    for phase, phase_dataset in dataset.items():
        print(phase, " ",phase_dataset)
        if phase!="test":
          features_dict[task_name][phase] = phase_dataset.map(
              convert_func_dict[task_name],
              batched=True,
              load_from_cache_file=False,
          )
          print(task_name, phase, len(phase_dataset), len(features_dict[task_name][phase]))
          features_dict[task_name][phase].set_format(
              type="torch",
              columns=columns_dict[task_name],
          )
          print(task_name, phase, len(phase_dataset), len(features_dict[task_name][phase]))

and dataloader is
class NLPDataCollator(DataCollator):
    """
    Extending the existing DataCollator to work with NLP dataset batches
    """
    def collate_batch(self, features: List[Union[InputDataClass, Dict]]) -> Dict[str, torch.Tensor]:
        first = features[0]
        if isinstance(first, dict):
          # NLP data sets current works presents features as lists of dictionary
          # (one per example), so we  will adapt the collate_batch logic for that
          if "labels" in first and first["labels"] is not None:
              if first["labels"].dtype == torch.int64:
                  labels = torch.tensor([f["labels"] for f in features], dtype=torch.long)
              else:
                  labels = torch.tensor([f["labels"] for f in features], dtype=torch.float)
              batch = {"labels": labels}
          for k, v in first.items():
              if k != "labels" and v is not None and not isinstance(v, str):
                  batch[k] = torch.stack([f[k] for f in features])
          return batch
        else:
          # otherwise, revert to using the default collate_batch
          return DefaultDataCollator().collate_batch(features)

trainer is as
trainer = MultitaskTrainer(
    model=model,
    args=transformers.TrainingArguments(
        output_dir="./models/",
        overwrite_output_dir=True,
        learning_rate=1e-5,
        do_train=True,
        do_eval=True,
        num_train_epochs=3,
        # Adjust batch size if this doesn't fit on the Colab GPU
        per_device_train_batch_size=8,
        save_steps=3000,
    ),
    data_collator=NLPDataCollator(),
    train_dataset=train_dataset,
    eval_dataset = valid_dataset,

)
trainer.train()

I get the following error:
ValueError                                Traceback (most recent call last)

<ipython-input-49-68181c1063c6> in <module>()
     26
     27 )
---> 28 trainer.train()

8 frames

<ipython-input-39-4359c3725689> in collate_batch(self, features)
     20           if "labels" in first and first["labels"] is not None:
     21               if first["labels"].dtype == torch.int64:
---> 22                   labels = torch.tensor([f["labels"] for f in features], dtype=torch.long)
     23               else:
     24                   labels = torch.tensor([f["labels"] for f in features], dtype=torch.float)

ValueError: only one element tensors can be converted to Python scalars


title: pytorch neural network regression model not learning

body:
I want to create a regression model using a neural network that predicts the 60th data using data from 1 to 59.
Each data has 5 features. The 59x5 data was solved and used in 295 one-dimensional in the same way as the classic image recognition neural network.
But I don't know why the learning is not working properly. Is regression learning impossible instead of classification in neural networks?
This is full code
import numpy as np
import random
import torch
import torch.nn as nn
import torch.nn.functional as F

data = np.loadtxt('Apr-May_mod.csv',delimiter=",")

X = []
Y = []
for i in range(0,100):
    start_point = random.randrange(1,len(data)-60)
    X.append(data[start_point:start_point+59,:])
    Y.append(data[start_point+60,:])

X = torch.tensor(X, dtype= torch.float)
Y = torch.tensor(Y, dtype= torch.float)

X_Pmax = torch.max(X[:,:,0:4])
X_Vmax = torch.max(X[:,:,4:5])
X = torch.cat([torch.div(X[:,:,0:4], X_Pmax), torch.div(X[:,:,4:5], X_Vmax)], dim=2)

Y_Pmax = torch.max(Y[:,0:4])
Y_Vmax = torch.max(Y[:,4:5])
Y = torch.cat([torch.div(Y[:,0:4], Y_Pmax), torch.div(Y[:,4:5], Y_Vmax)], dim=1)

X = X.view([100,-1])

class NeuralNetwork(nn.Module):
    def __init__(self):
        self.inputSize = 295
        self.outputSize = 5
        self.hiddenSize = 500

        self.w1 = torch.rand(self.inputSize, self.hiddenSize, requires_grad=True)
        self.w2 = torch.rand(self.hiddenSize, self.outputSize, requires_grad=True)

        self.learning_rate = 0.001

    def forward(self,X):
        self.z1 = torch.matmul(X, self.w1)
        self.z2 = torch.sigmoid(self.z1)
        self.z3 = torch.matmul(self.z2, self.w2)
        out = torch.sigmoid(self.z3)
        return out

    def train(self, X, Y):
        out = self.forward(X)
        error = ((Y-out)**2).mean()
        error.backward()
        self.w1.data += self.learning_rate * (self.w1).grad
        self.w2.data += self.learning_rate * (self.w2).grad

    def saveWeights(self, model):
        torch.save(model, "NeuralNetwork")

    def predict(self):
        print("Input Data: X[3]")
        print("Predicted: ", str(self.forward(X[3])))

NN = NeuralNetwork()
trainIdx = 1000
for idx in range(trainIdx):
    if idx%100 == 0:
        print("#" + str(idx) + " Loss: " + str(torch.mean((Y-NN.forward(X))**2).detach().item()))

    NN.train(X,Y)

NN.saveWeights(NN)
NN.predict()

The result is like this.
#0 Loss: 0.14510075747966766
#100 Loss: 0.14510075747966766
#200 Loss: 0.14510075747966766
#300 Loss: 0.14510075747966766
#400 Loss: 0.14510075747966766
#500 Loss: 0.14510075747966766
#600 Loss: 0.14510075747966766
#700 Loss: 0.14510075747966766
#800 Loss: 0.14510075747966766
#900 Loss: 0.14510075747966766
Input Data: X[3]
Predicted:  tensor([1., 1., 1., 1., 1.], grad_fn=<SigmoidBackward>)

This is the file I used, Apr-May_mod.csv
58739.46,58796.25,58720.82,58795.74,56.541494
58795.74,58858.21,58787.48,58815.42,45.844834
58815.42,58963.91,58808.99,58963.41,80.257514
58953.83,59080.26,58952.34,59063.13,107.880493
59063.13,59093,58937.3,59061.14,68.599352
59071.84,59083,58977.78,58993.04,38.179872
58993.04,59080,58985.46,59043.88,29.342092
59043.88,59088.95,59007,59007.79,26.401287
59007.79,59071.02,58997.24,59047.38,44.14176
59047.38,59070.88,59030,59030.01,29.801803
59030,59065.61,59030,59060.37,19.928143
59060.38,59068.05,59037.86,59043.86,17.594566
59045.86,59052.55,59029.95,59049.48,21.902506
59049.49,59049.49,58967.44,58994.47,32.587782
58995.12,59007.75,58965,58965,25.056395
58965.01,58966.02,58907.76,58956.39,25.896267
58956.39,58957.68,58895.01,58900,25.493498
58900,58958.94,58877.55,58955.48,27.695467
58955.48,59027.09,58951.55,58994.19,19.519458
58994.2,59022.43,58852.94,58883.57,79.41166
58883.57,58929.89,58862.21,58926.21,17.17355
58926.22,58960.93,58926.21,58959.07,18.647385
...
about 40000 lines


title: Runtime Error: Found an in-place operation that changes one the variables needed by gradient computation

body:
I'm training vanilla RNN in PyTorch to learn the changes in the hidden dynamics.
There's no problem with forward pass and bk prop for the initial batch but when it comes to the part where I
use prev. hidden state as initial state it's somehow considered as in-place operation.
I don't really understand why this makes a problem and how to solve it.
I tried to follow the given hint by setting retain_graph=True in backward()but no progress.
Code edited
#define a class for the neural DE func
class ODE_RNN(nn.Module):
    def __init__(self, x, hidden_size):
        super().__init__()

        self.x= x
        #self.h_0= h_0
        self.rnn= nn.RNN(1, hidden_size, nonlinearity='tanh', batch_first=True)
        self.dense= nn.Linear(hidden_size, 1)

    def forward(self,t,k):
        y0, h_0= k
        print(k)
        h_t, h_n= self.rnn(self.x, h_0)
        y= self.dense(h_t)

        return y, h_t

    #main function

    ii = 0
    #training loop -> using standard bk prop

    for itr in range(1, niters + 1):

        for batch in range(nbatchs):

            #get data batches
            t, x0, y0, true_y= get_batch(batch)

            #make instance of ODEFunc
            func = ODE_RNN(x0, hidden_size).to(device)

            #declare optimizer
            optimizer = optim.Adam(func.parameters(), lr=1e-3)

            print("batch <{0}>\n".format(batch)+
                  "-----------------------------------------------------------\n"+
                  "t: {0}, {1}\n".format(t,t.size())+
                  "x0: {0}, {1}\n".format(x0,x0.size())+
                  "y0: {0}, {1}\n".format(y0,y0.size())
                )

            if batch == 0:

               #clear grads
               optimizer.zero_grad()

               #forward pass
               pred_y, pred_h = odeint(func, tuple([y0, h_0]), t) #note: pred_xx.size()= eval_pts x batch x seq x feature  4D

               #print prediction vs true value
               print(
                     "pred_y: {0}, {1}\n".format(pred_y,pred_y.size())+
                     "true_y: {0}, {1}\n".format(true_y,true_y.size())
               )

            else:

               prev_pred_h= pred_h[-1]
               print(prev_pred_h, pred_h)

               #clear grads
               optimizer.zero_grad()

               #forward pass
               pred_y, pred_h = odeint(func, tuple([y0, prev_pred_h]), t)

               print(
                     "pred_y: {0}, {1}\n".format(pred_y,pred_y.size())+
                     "true_y: {0}, {1}\n".format(true_y,true_y.size())
               )

            #evaluate the loss
            loss = nn.MSELoss()(pred_y, true_y)

            print("loss: {0}\n".format(loss)+
                  "-----------------------------------------------------------"
            )

            #with torch.autograd.set_detect_anomaly(True):

            #bkprop error
            loss.backward(retain_graph=True)

            #update parameters
            optimizer.step()

-----------------------------------------------------------
/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py:147: UserWarning: Error detected in MmBackward. No forward pass information available. Enable detect anomaly during forward pass for more information. (Triggered internally at  /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:85.)
  allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-68-a34fbe7cdca8> in <module>()
     60
     61         #bkprop error
---> 62              loss.backward(retain_graph=True)
     63
     64         #update parameters

1 frames
/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    145     Variable._execution_engine.run_backward(
    146         tensors, grad_tensors_, retain_graph, create_graph, inputs,
--> 147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
    148
    149

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [5, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!


title: Data Loading in Pytorch for a dataset having all the classes in same folder

body:
I am new to deep learning and Pytorch. I have data set of 6000 images that have all four classes in a single folder. I used the following snippet to upload my data.

torchvision.datasets.ImageFolder(root='/content/drive/My Drive/DFU/base_dir/train_dir', transform=None)

I read that for ImageFolder, the images should be organized into sub-folders based on class labels. However, my dataset has all four class images in a single folder. I have a .csv file that contains the one-hot-encoded class label for each image. How to load my dataset to Pytorch?


title: How to access pytorch embeddings lookup table as a tensor

body:
I want to show my embeddings with the tensorboard projector. I would like to access the embeddings matrix (lookup table) of one of my layers so I can write it to the logs.
I instantiate my layer as this:
self.embeddings_user = torch.nn.Embedding(30,300)
And I'm looking for the tensor with shape (30,300) of 30 users with embedding on 300 to dimensions to replace it with the vectors variable in this sample code:
import numpy as np
import tensorflow as tf
import tensorboard as tb
tf.io.gfile = tb.compat.tensorflow_stub.io.gfile
from torch.utils.tensorboard import SummaryWriter

vectors = np.array([[0,0,1], [0,1,0], [1,0,0], [1,1,1]])
metadata = ['001', '010', '100', '111']  # labels
writer = SummaryWriter()
writer.add_embedding(vectors, metadata)
writer.close()


title: RNN hidden value contains almost only 1 and -1

body:
I'm using encoder-decoder model with RNN to generate the abstract for a document. After testing, I found that the outputs are totally useless, and they are all same. I printed the hidden value and found that it contains almost only 1 and -1.

Output of the decoder:

['<SOS>', 'the', 'new', 'york', 'city', "'s", 'mother', 'was', 'found',      'in', 'the', 'world', 'cup', '.', 'the', 'former', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'the', 'yoselyn', 'is', 'the', 'the', 'much-loved', 'condition', '.', '<EOS>']

part of hidden value:

( 0 ,.,.) =

Columns 0 to 8
   1.0000  0.9846 -1.0000 -1.0000  1.0000 -1.0000  0.0000  1.0000  1.0000

Columns 9 to 17
  -1.0000  0.0000  1.0000 -1.0000 -0.9019  1.0000 -1.0000  1.0000 -1.0000

Columns 18 to 26
   1.0000  1.0000  1.0000  1.0000 -1.0000  0.0000  1.0000  1.0000 -1.0000

Columns 27 to 35
  -1.0000 -1.0000  1.0000  0.0000 -1.0000 -1.0000 -1.0000  0.0000 -1.0000

Columns 36 to 44
   1.0000 -1.0000 -1.0000  1.0000  0.0000 -1.0000  1.0000 -1.0000 -1.0000

I'm a beginner in deep learning and I cannot understand why this happened. I checked my dataset and I think they are fine.

Encoder and decoder:

class EncoderRNN(nn.Module):
    def __init__(self, dict_size, hidden_size):
        super(EncoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.embedding = nn.Embedding(dict_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True)

    def forward(self, input_batch, input_batch_length, hidden):
        embedded = self.embedding(input_batch)
        packed_input = nn.utils.rnn.pack_padded_sequence(embedded, input_batch_length.cpu().numpy(), batch_first=True)
        output, hidden = self.gru(packed_input, hidden)
        return output, hidden

class DecoderRNN(nn.Module):
    def __init__(self, hidden_size, dict_size):
        super(DecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.embedding = nn.Embedding(dict_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True)
        self.out = nn.Linear(hidden_size, dict_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, target_batch, target_batch_length, hidden, train=False):
        embedded = self.embedding(target_batch)
        output = F.relu(embedded)

        if train:
            # minus 1 to eliminate <EOS>
            output = nn.utils.rnn.pack_padded_sequence(output, (target_batch_length - 1).cpu().numpy(),
                                                       batch_first=True)

        output, hidden = self.gru(output, hidden)
        output = self.softmax(self.out(output[0]))
        return output, hidden


title: need help understanding pytorch blitz math notation

body:
I just came across this notation in the pytorch blitz tutorial and I dont know what the vertical line is

Does anyone have any suggestions on the notation?


title: Number of layers of the model

body:
I'm training a CNN. For reporting purposes, I want to find out the number of layers my model has.

From what I see in the code below, I have a total of 6 layers, layer1, layer2, conv2_drop, fc1, fc2, fc3. Am I right?

Net(
  (layer1): Sequential(
    (0): Conv2d(3, 10, kernel_size=(5, 5), stride=(1, 1))
    (1): BatchNorm2d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU()
    (3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (4): Dropout(p=0.2, inplace=False)
  )
  (layer2): Sequential(
    (0): Conv2d(10, 20, kernel_size=(5, 5), stride=(1, 1))
    (1): BatchNorm2d(20, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU()
    (3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (4): Dropout(p=0.2, inplace=False)
  )
  (conv2_drop): Dropout2d(p=0.5, inplace=False)
  (fc1): Linear(in_features=500, out_features=250, bias=True)
  (fc2): Linear(in_features=250, out_features=50, bias=True)
  (fc3): Linear(in_features=50, out_features=10, bias=True)
)

Does my NN have 6 layers, or layer1 and layer2 themselves add 4 more each? That would be 14 layers total?


title: test accuracy fluctuate even train and test are always same

body:
Even though my train, validation and test set are always the same, test accuracy fluctuate. The only reason I can think of is weight initialization. I am using PyTorch and I guess they use advance initialization technique (kaiming initialization).

What might be the reason for accuracy fluctuation even though train, validation and test data are the same?


title: PyTorch: one dimension in input data

body:
I don't understand why this code works:

# Hyperparameters for our network
input_size = 784
hidden_sizes = [128, 64]
output_size = 10

# Build a feed-forward network
model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[0], hidden_sizes[1]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[1], output_size),
                      nn.Softmax(dim=1))
print(model)

# Forward pass through the network and display output
images, labels = next(iter(trainloader))
images.resize_(images.shape[0], 1, 784)
ps = model.forward(images[0,:])

The size of an image is (images.shape[0], 1, 784) but our networks has input_size = 784. How does the network handle 1 dimension in an input image? I tried to change images.resize_(images.shape[0], 1, 784) to images = images.view(images.shape[0], -1) but I got an error:

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

For reference, data loader is created the next way:

# Define a transform to normalize the data
transform = transforms.Compose([transforms.ToTensor(),
                              transforms.Normalize((0.5,), (0.5,)),
                              ])
# Download and load the training data
trainset = datasets.MNIST('~/.pytorch/MNIST_data/', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)


title: Does the reduction of the dimensions over multiple layers allow more details to be stored within the final representation?

body:
From : https://debuggercafe.com/implementing-deep-autoencoder-in-pytorch/
the following autoencoder is defined

class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder, self).__init__()

        # encoder
        self.enc1 = nn.Linear(in_features=784, out_features=256)
        self.enc2 = nn.Linear(in_features=256, out_features=128)
        self.enc3 = nn.Linear(in_features=128, out_features=64)
        self.enc4 = nn.Linear(in_features=64, out_features=32)
        self.enc5 = nn.Linear(in_features=32, out_features=16)

        # decoder
        self.dec1 = nn.Linear(in_features=16, out_features=32)
        self.dec2 = nn.Linear(in_features=32, out_features=64)
        self.dec3 = nn.Linear(in_features=64, out_features=128)
        self.dec4 = nn.Linear(in_features=128, out_features=256)
        self.dec5 = nn.Linear(in_features=256, out_features=784)

    def forward(self, x):
        x = F.relu(self.enc1(x))
        x = F.relu(self.enc2(x))
        x = F.relu(self.enc3(x))
        x = F.relu(self.enc4(x))
        x = F.relu(self.enc5(x))

        x = F.relu(self.dec1(x))
        x = F.relu(self.dec2(x))
        x = F.relu(self.dec3(x))
        x = F.relu(self.dec4(x))
        x = F.relu(self.dec5(x))
        return x

net = Autoencoder()

From the Autoencoder class, we can see that 784 features are passed through a series of transformations and are converted to 16 features.

The transformations (in_features to out_features) for each layer are:

784 to 256
256 to 128
128 to 64
64 to 32
32 to 16

Why do we perform this sequence of operations? For example, why don't we perform the following sequence of operations instead?

784 to 256
256 to 128

Or maybe

784 to 512
512 to 256
256 to 128

Or maybe just encode in two layers:

784 to 16

Does the reduction of the dimensions over multiple layers (instead of a single layer) allow more details to be stored within the final representation? For example, if we used only the transformation $784 \rightarrow 16$, may this cause some detail not to be encoded? If so, why is this the case?


title: How to compare the dis-/ similarity between images?

body:
I have a data set which looks like following:

Image_1    Image_2    Image_3
  A           B         C
 ...         ...       ...

If image A is similar to B, it will be assigned with label 1, otherwise with label 0.

I first use pre-trained resnet18 to extract features for each RGB image and get a 1000 dimensional vector. Then I build a deep network and use triplet as loss function to train model, there is a part of my code:

class Network(torch.nn.Module):
    def __init__(self, n_feature = 1000, n_hidden_1,n_output = 10):
        super(Network, self).__init__()
        self.net = torch.nn.Sequential(
                        torch.nn.Linear(n_feature, n_hidden_1),
                        torch.nn.BatchNorm1d(n_hidden_1),
                        torch.nn.ReLU(),
                        torch.nn.Linear(n_hidden_1, n_output)
                        )
    def forward(self, x):
        x = self.net(x)
        return x

Training step:

for step, (batch_anchor, batch_positive, batch_negative )in enumerate(train_loader):
    optimizer.zero_grad()
    anchor_out = model(batch_anchor)
    positive_out = model(batch_positive)
    negative_out = model(batch_negative)
    loss = loss_func(anchor_out, positive_out, negative_out)
    loss.backward()
    optimizer.step()

where I define loss function and optimiser with:

optimizer = optim.Adam(model.parameters(), lr=0.002)
loss_func = torch.nn.TripletMarginLoss()

After the training process is done, I test this network with validation set:

with torch.no_grad():
    anchor_out_val = model(val_data_anchor).numpy()
    positive_out_val = model(val_data_positive).numpy()
    negative_out_val = model(val_data_negative).numpy()

Now I use L2 Norm to measure similarity and assign labels, this works very well in validation set and I got accuracy 80% measured by accuracy_score from sklearn. But when I try it with test set, I only get 50% accuracy. Maybe someone could tell me why? Is the metric to measure similarity not good? or maybe it is the problem from network?


title: Gradient becomes 0 for 1D convolutions

body:
I'm trying to implement the attention module with 1D convolution.
Here's my custom attention module for that.
class convSelfAttention(nn.Module):
    def __init__(self, in_dim):
        super(convSelfAttention, self).__init__()
        self.channel_in = in_dim
        self.query_conv = nn.Conv1d(in_channels=in_dim, out_channels=in_dim//2, kernel_size=1)
        self.key_conv = nn.Conv1d(in_channels=in_dim, out_channels=in_dim//2, kernel_size=1)
        self.value_conv = nn.Conv1d(in_channels=in_dim, out_channels=in_dim, kernel_size=1)
        self.gamma = nn.Parameter(torch.zeros(1))
        self.activ = nn.Softmax(dim=-1)

    def forward(self, x):
        p_query = self.query_conv(x).permute(0, 2, 1)
        p_key = self.key_conv(x)
        energy = torch.bmm(p_query, p_key)
        attention = self.activ(energy)
        p_value = self.value_conv(x)

        out = torch.bmm(p_value, attention.permute(0, 2, 1))
        out = self.gamma * out + x
        out = out.permute(0, 2, 1)
        return out

However, the gradients are 0 except 'self.gamma' and it does not change even with deleting softmax function.
For your convenience, I plot the gradient flow of my entire architecture.(The last two layers are fc and all layers from '~~Encoder' are convolutions.)


title: pytorch tensor duplication with keeping gradient

body:
Good morning,
I'm searching the methods to duplicate the tensor.
For example,
a = torch.Tensor(1, 4) ## tensor([[0.1, 0.2, 0.3, 0.4]])

and I'm gonna make it into 4x4 tensor like this
tensor([[0.1, 0.1, 0.1, 0.1],
        [0.2, 0.2, 0.2, 0.2],
        [0.3, 0.3, 0.3, 0.3],
        [0.4, 0.4, 0.4, 0.4]])

to do the multiplication with a tensor size of 4xN
Any idea to do it with preserving gradient? (I don't fully understand about how to keep the gradient of each tensor.)
Thank you


title: Denoisner returns choppy results

body:
I am building a denoiser for cyclic functions (cos(x), cos(2x) etc..) and I am getting bad results to say the least.
Up until a few hours ago the model didnt work at all (just returned 0 for everything), then I added nn.BatchNorm1d and it started to "learn".
The problem is that the results are really bad and look even worse than the noisy example:

The model is really straight forward (in_channels = 1, K = 128 number of samples in each interval, B = 2 number of intervals, z_dim = 64 hidden dimension size, out_channels = 1)
class LinAutoencoder(nn.Module):
    def __init__(self, in_channels, K, B, z_dim, out_channels):
        super(LinAutoencoder, self).__init__()

        self.in_channels = in_channels
        self.K = K
        self.B = B
        self.out_channels = out_channels

        encoder_layers = []
        decoder_layers = []

        encoder_layers += [
            nn.Linear(in_channels * K * B, in_channels * K * B, bias=True),
            nn.BatchNorm1d(in_channels * K * B),
            nn.ReLU(),
            nn.Linear(in_channels * K * B, z_dim, bias=True),
            nn.BatchNorm1d(z_dim),
            nn.ReLU()
        ]

        decoder_layers += [
            nn.Linear(z_dim, out_channels * K * B, bias=True),
            nn.BatchNorm1d(out_channels * K * B),
            nn.ReLU(),
            nn.Linear(out_channels * K * B, out_channels * K * B, bias=True),
            nn.BatchNorm1d(out_channels * K * B),
            nn.ReLU()
        ]

        self.encoder = nn.Sequential(*encoder_layers)
        self.decoder = nn.Sequential(*decoder_layers)

        print(self.decoder)

    def forward(self, x):
        batch_size = x.shape[0]
        x_flat = torch.flatten(x, start_dim=1)
        enc = self.encoder(x_flat)
        dec = self.decoder(enc)
        res = dec.view((batch_size, self.out_channels, self.K * self.B))
        return res

I tried to make the model deeper, use tanh instead of relu with no success.
I trained it for 900 epochs with lr = 0.005 and momentum = 0.9 using the SGD optimizer.
I have also tried different optimizers.
cheers


title: Correct way to go from non-sequential input to sequence output using GRU/LSTM in pytorch

body:
I've read many PyTorch tutorials, but for some reason I'm still fundamentally stumped on one things.
My input is a 400-dimensional tensor, and my desired output is a variable-length sequence vector.
nn.LSTM and nn.GRU both take in and output a sequence.  So how do I go from a 400-dimensional tensor (or of size (batch_size, 400)) to something of size (batch_size, max_seq_length, output_dim) in an idiomatic way?


title: Trying to run the facebook Ax Bayesian optimisation platform for tuning hyperparameters, but the optimizer loop from Ax does not read my parameters?

body:
I basically disected the code from the MNIST CNN tutorial from Ax and put my network into it, the Optimisation process was able to run when I explicitly define the hyperparameter (which means nothing because it is not iterating through the range), but when I put my model into the optimise loop with library definition, train_evaluate says it got an empty parameter list. The facebook Ax website didn't help much with the trouble-shooting, have anyone experienced similar issue?
made train a function to be called later with dictionary inside
the optimize Loop from Ax that returns empty parameter list


title: Can't understand about pytorch tensor broadcast

body:
I have the following code:
import torch
d = 2
n = 50
X = torch.randn(n,d)
z = torch.tensor([[-1.0], [2.0]])
y = X @ z
X.size()
z.size()
y.size()

The output is:
torch.Size([50, 2])
torch.Size([2, 1])
torch.Size([50, 1])

My question is, why after broadcasting, the size of the result y is [50,1] rather than [50,2], I think it should be [50,2], am I correct?


title: What's the difference between these two code for inference with toy MNIST example

body:
I found that there're different accuracy values between those two codes.
I think that those are the same codes but give different values.
Why are the results of the two codes different?
[ code 1 ]
def mytest(model, data_loader):
    with torch.no_grad():
        corrected_sample = 0
        for x_test, y_test in data_loader:
            x_test, y_test = x_test.to(device), y_test.to(device)
            x_test = x_test.view(-1,28*28).float()
            prediction = model(x_test)
            #predicted_class = prediction.max(1)[1]
            predicted_class = torch.argmax(prediction, 1)
            corrected_sample += (predicted_class==y_test).float().sum().item()
    accuracy = corrected_sample / len(data_loader.dataset)
    print("Accuracy : {}".format(accuracy))

data_loader_test = torch.utils.data.DataLoader(mnist_test, batch_size=1, shuffle=False)
mytest(model, data_loader_test)

Accuracy : 0.92232

[ code 2 ]
with torch.no_grad():
    x_test = mnist_test.test_data.view(-1,28*28).float().to(device)
    y_test = mnist_test.test_labels.to(device)

    prediction = model(x_test)
    correct_prediction = torch.argmax(prediction,1) == y_test
    accuracy = correct_prediction.float().mean()
    print("Accuracy ", accuracy.item())

Accuracy  0.8826999664306641


title: Weights of network stays the same after optimizer step

body:
My network just refuses to train. To make code reading less of a hassle, I abbreviate some complicated logic. Would update more if needed.
model = DistMultNN()
optimizer = optim.SGD(model.parameters(), lr=0.0001)
for t in range(500):
    e1_neg = sampling_logic()
    e2_neg = sampling_logic()

    e1_pos = sampling_logic()
    r = sampling_logic()
    e2_pos = sampling_logic()
    optimizer.zero_grad()
    y_pred = model(tuple(zip(e1_pos, r, e2_pos)), e1_neg, e2_neg)
    loss = model.loss(y_pred)
    loss.backward()
    optimizer.step()

I define my network as follow
class DistMultNN(nn.Module):

    def __init__(self):
        super().__init__()
        self.seed = 42
        self.entities_embedding = nn.init.xavier_uniform_(
            torch.zeros((self.NO_ENTITIES, self.ENCODING_DIM), requires_grad=True))
        self.relation_embedding = nn.init.xavier_uniform_(
            torch.zeros((self.NO_RELATIONSHIPS, self.ENCODING_DIM), requires_grad=True))
        self.W = torch.rand(self.ENCODING_DIM, self.ENCODING_DIM, requires_grad=True)  # W is symmetric, todo: requireGrad?
        self.W = (self.W + self.W.t()) / 2
        self.b = torch.rand(self.ENCODING_DIM, 1, requires_grad=True)
        self.lambda_ = 1.
        self.rnn = torch.nn.RNN(input_size=encoding_dim, hidden_size=1, num_layers=1, nonlinearity='relu')
        self.loss_func = torch.nn.LogSoftmax(dim=1)

    def loss(self, y_pred):
        softmax = -1 * self.loss_func(y_pred)
        result = torch.mean(softmax[:, 0])
        result.requires_grad = True
        return result

    def forward(self, samples, e1neg, e2neg):
        batch_size = len(samples)
        batch_result = np.zeros((batch_size, len(e1neg[0]) + 1))
        for datapoint_id in range(batch_size):
            entity_1 = entities_embed_lookup(datapoint_id[0])
            entity_2 = entities_embed_lookup(datapoint_id[2])
            r = relation_embed_lookup(datapoint_id[1])
            x = self.some_fourier_transform(entity_1, r, entity_2)
            batch_result[datapoint_id][0] = self.some_matmul(x)
            for negative_example_id in range(len(e1neg[0])):
                same_thing_with_negative_examples()
                batch_result[datapoint_id][negative_example_id + 1] = self.some_matmul(x)
        batch_result_tensor = torch.tensor(data=batch_result)
        return batch_result_tensor

I tried checking weights using e.g. print(model.rnn.all_weights) in the training loop but they do not change. What did I do wrong?


title: Error: index 7 is out of range while adding Layer in CNN(vgg16) using pytorch

body:
I was implementing transfer learning using pytorch. This was my initial code: (just change in last layer)
from torchvision import datasets, models, transforms
model_ft = models.vgg16_bn(pretrained=True)
model_ft.classifier[6] = nn.Linear(in_features=4096,
                               out_features=2,
                               bias=True)

The problem is that the result was not satisfactory. That's why I wanted to add more layers so that I don't have to jump from 4096 to 2.
from torchvision import datasets, models, transforms
model_ft = models.vgg16_bn(pretrained=True)
model_ft.classifier[6] = nn.Linear(in_features=4096,
                                   out_features=1024,
                                   bias=True)
model_ft.classifier[7] = nn.ReLU()
model_ft.classifier[8] = nn.Dropout(p=0.5)
model_ft.classifier[9] = nn.Linear(in_features=1024,
                                   out_features=10,
                                   bias=True)
Traceback (most recent call last):

  File "<ipython-input-14-519d30b6f8d1>", line 5, in <module>
    model_ft.classifier[7] = nn.ReLU()

  File "C:\Users\frmas\anaconda3\envs\tfgpu\lib\site-packages\torch\nn\modules\container.py", line
86, in __setitem__
    key = self._get_item_by_idx(self._modules.keys(), idx)

  File "C:\Users\frmas\anaconda3\envs\tfgpu\lib\site-packages\torch\nn\modules\container.py", line
74, in _get_item_by_idx
    raise IndexError('index {} is out of range'.format(idx))

IndexError: index 7 is out of range

Why can't I add more layers to the vgg16 classification. How to solve this?


title: Is that normal to train properly with 2 sets and/or 2 diff losses function?

body:
I have 2 train sets: one with label and one with no label.
When training, i’m simultaneously loading one batch from a labelled set, then calculating using the first loss function; and then, one batch from unlabeled set, calculating using the using the other function. Finally I sum them (2 losses) and loss.backward() .
Does this way work ? it’s quite uncommon in my mind so just ask if the engine know how to back-propagate properly (not wrong)?
Thank you.


title: 3D plotting of Skeleton data is not working for later frames

body:
Hi I am trying to visualize NTU RGB+D skeleton data. I am trying this code I found here https://www.programmersought.com/article/47095170137/
The first frame is being plotted but it fails to plot the later ones. From 2nd frame to last figure opens but empty.  Data size is (xyz_coordinates * no_of_frames * no_of_joints * no_of_body)
def visual_skeleton(self, data):
    fig = plt.figure()
    ax = Axes3D(fig)

    ax.view_init(self.init_vertical, self.init_horizon)
    plt.ion()

    data = np.transpose(data, (3, 1, 2, 0))

    # data rotation
    if (self.x_rotation is not None) or (self.y_rotation is not None):

        if self.x_rotation > 180 or self.y_rotation > 180:
            raise Exception("rotation angle should be less than 180.")

        else:
            data = self._rotation(data, self.x_rotation, self.y_rotation)

    # data normalization
    data = self._normal_skeleton(data)
    #print(data.shape)

    # show every frame 3d skeleton
    for frame_idx in range(data.shape[1]-298):
        #print(frame_idx)
        plt.cla()
        plt.title("Frame: {}".format(frame_idx))

        ax.set_xlim3d([-1, 1])
        ax.set_ylim3d([-1, 1])
        ax.set_zlim3d([-0.8, 0.8])

        x = data[0, frame_idx, :, 0]
        y = data[0, frame_idx, :, 1]
        z = data[0, frame_idx, :, 2]

        for part in body:
            x_plot = x[part]
            y_plot = y[part]
            z_plot = z[part]
            ax.plot(x_plot, z_plot, y_plot, color='b', marker='o', markerfacecolor='r')

        ax.set_xlabel('X')
        ax.set_ylabel('Z')
        ax.set_zlabel('Y')
        save_path = "skl_plots"
        if self.save_path is not None:
            save_pth = os.path.join(self.save_path, '{}.png'.format(frame_idx))
            plt.savefig(save_pth)
        print("The {} frame 3d skeleton......".format(frame_idx))

        ax.set_facecolor('none')
        plt.pause(self._pause_step)

    plt.ioff()
    ax.axis('off')
    plt.show()


title: How does one resolve the Miniimagenet data integrity issue with torchmeta?

body:
I was trying to test torchmeta:
import torch

import torchvision.transforms as transforms

from torchmeta.datasets.helpers import miniimagenet
from torchmeta.utils.data import BatchMetaDataLoader

from tqdm import tqdm

from pathlib import Path

meta_split = 'train'
data_path = Path('~/data/').expanduser()
dataset = miniimagenet(data_path, ways=5, shots=5, test_shots=15, meta_split=meta_split, download=True)
dataloader = BatchMetaDataLoader(dataset, batch_size=16, num_workers=4)
print(f'len normal = {len(dataloader)}')

num_batches = 10
with tqdm(dataloader, total=num_batches) as pbar:
    for batch_idx, batch in enumerate(pbar):
        train_inputs, train_targets = batch["train"]
        print(train_inputs.size())
        # print(batch_idx)
        if batch_idx >= num_batches:
            break

print('success\a'

but it fails:
Using downloaded and verified file: /home/miranda9/data/miniimagenet/mini-imagenet.tar.gz
Traceback (most recent call last):
  File "pro/experiments/meta_learning/meta_learning_experiments_submission.py", line 421, in <module>
    main(args)
  File "proj/experiments/meta_learning/meta_learning_experiments_submission.py", line 328, in main
    meta_split='train', download=True)
  File "/home/miranda9/miniconda/envs/meta-learning/lib/python3.7/site-packages/torchmeta/datasets/helpers.py", line 126, in miniimagenet
    seed=seed, defaults=defaults, **kwargs)
  File "/home/miranda9/miniconda/envs/meta-learning/lib/python3.7/site-packages/torchmeta/datasets/helpers.py", line 34, in helper_with_default
    dataset = klass(folder, num_classes_per_task=ways, **kwargs)
  File "/home/miranda9/miniconda/envs/meta-learning/lib/python3.7/site-packages/torchmeta/datasets/miniimagenet.py", line 89, in __init__
    download=download)
  File "/home/miranda9/miniconda/envs/meta-learning/lib/python3.7/site-packages/torchmeta/datasets/miniimagenet.py", line 127, in __init__
    raise RuntimeError('MiniImagenet integrity check failed')

others have reported it here but their solutions don't work for me:
https://github.com/sungyubkim/GBML/issues/3
I tried setting the download=True flag. Now I will try to send my local copy up to the cluster and see if that works...


title: Is there a way to figure out if an NDArray is trainable?

body:
Is there a way to figure out if an NDArray variable has a gradient/requires backprop, similar to the requires_grad attribute in pytorch?
I've tried checking whether x.grad is None, but that doesn't work for intermediate variables:
import mxnet as mx

x = mx.nd.array([1, 2])
x.attach_grad()
with mx.autograd.record():
  y = x**2
  z = 2 * y
z.backward()

>>> print(x.grad)
[4. 8.]
<NDArray 2 @cpu(0)>
>>> print(y.grad)
None

In this example, x has a gradient but the intermediate variable y does not, but it should also be trainable.
I've tried looking through the source code, but can't figure it out.


title: Torch tensor with i-th element as product of all previous

body:
I have a torch tensor like this one:
A = torch.tensor([1,2,3,4,5,6])

there's in torch a simple way to generate a tensor like this?
[1, 1*2, 1*2*3, 1*2*3*4, 1*2*3*4*5, 1*2*3*4*5*6]

Thanks a lot


title: How do you rearrange different dateloaders in the same order?

body:
traindir = os.path.join(args.data, "train")
valdir = os.path.join(args.data, "val")
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])

train_dataset = datasets.ImageFolder(root=traindir, transform=transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    normalize,
]))
val_dataset = datasets.ImageFolder(root=valdir, transform=transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    normalize,
]))

def worker_init_fn(worker_id):
    # seed=10
    # seed+=worker_id
    # np.random.seed(seed)
    # print(worker_id)
    print(np.random.get_state()[1][0])
    print(np.random.get_state()[1][0]+worker_id)
    np.random.seed(np.random.get_state()[1][0]+worker_id)

def _init_fn(worker_id):
    np.random.seed(42)
train_loader = torch.utils.data.DataLoader(
    dataset=train_dataset,
    batch_size=args.batch_size, shuffle=True,
    num_workers=args.workers, pin_memory=True)
train_loader2 = torch.utils.data.DataLoader(
    dataset=train_dataset,
    batch_size=args.batch_size, shuffle=True,
    num_workers=args.workers, pin_memory=True)
train_loader3 = torch.utils.data.DataLoader(
    dataset=train_dataset,
    batch_size=args.batch_size, shuffle=True,
    num_workers=args.workers, pin_memory=True)

val_loader = torch.utils.data.DataLoader(
    dataset=val_dataset,
    batch_size=args.batch_size, shuffle=False,
    # batch_size=len(val_dataset), shuffle=False,
    num_workers=args.workers, pin_memory=True)
val_loader2 = torch.utils.data.DataLoader(
    dataset=val_dataset,
    batch_size=args.batch_size, shuffle=False,
    # batch_size=len(val_dataset), shuffle=False,
    num_workers=args.workers, pin_memory=True)
val_loader3 = torch.utils.data.DataLoader(
    dataset=val_dataset,
    batch_size=args.batch_size, shuffle=False,
    # batch_size=len(val_dataset), shuffle=False,
    num_workers=args.workers, pin_memory=True)

for i, dd in enumerate(zip(train_loader,train_loader2,train_loader3)):

    input1,target1=dd[0]
    input2,target2=dd[1]
    input3,target3=dd[2]

    print("1",target1)
    print("2",target2)
    print("3",target3)

There is a requirement that three copies of the same data label should be shuffled in parallel and read into three different networks for joint training. However, if shuffle is set to True, three copies of Dateloader will produce different labels for each batch. Therefore, we cannot train together


title: Deep Learning - Gradient Aggregation in Parameter Servers

body:
I have some questions regarding parameter servers and the gradient aggregation performed. My main source is the Dive into Deep Learning book [1]. I assume the BSP model, i.e., we synchronize after each mini-batch. I wasn't sure whether this belongs to the data science or AI StackExchange community, so I defaulted here...
Figure 12.7.1 on [1] suggests the following approach:
Assume batch size 32. If we have one GPU and 128 training datapoints, each epoch has 4 mini batches. After every mini batch, we update our model (i.e. there are 4 updates). Hence, we have to calculate four gradients (one per minibatch).
For the multiple GPUs case, assume two GPUs and 128 training datapoints. We feed each GPU a mini-batch, let them calculate a gradient, sum them and update our model with the sum. Hence, there are two steps instead of four involved (from a BSP point of view).
My questions are the following:

Is my described understanding on how parameter servers work correct? Especially I am not sure if we keep the same batch size 32 per GPU or if we have to divide the batch size by the number of GPUs.

Why do we sum the gradients instead of averaging them? This is even more confusing to me, as in the DistributedDataParallel documentation of PyTorch [2], there is the following statement:

When a model is trained on M nodes with batch=N, the gradient will be M times smaller when >compared to the same model trained on a single node with batch=M*N (because the gradients >between different nodes are averaged). You should take this into consideration when you want to >obtain a mathematically equivalent training process compared to the local training counterpart.

There are two things confusing here:
2.1. It states that the gradients are averaged, which does not comply to the D2L book which states that we sum the gradients.
2.2. Until now, I always thought, in mini-batch gradient descent, the loss function (optimization goal) averages the error of the mini-batch data points. Hence, if M nodes run mini-batch gradient descent with batch size N, and we take the average of their gradients, we should receive a number that is in the same order of magnitude as if 1 node runs mini-batch gradient-descent with batch size NM, as the single node averages the error functions of nm data points for the loss function, while with M nodes we just take the average of averages.
I am not sure whether the DistributedDataParallel class of PyTorch can be seen as a parameter server (especially because they even have a guide on how to build a parameter server in PyTorch [3]), but it maps to what is described in the book as a parameter server.
Any help on resolving my confusion is much appreciated. Thank you very much!
Kind regards,
Maximilian
[1] https://d2l.ai/chapter_computational-performance/parameterserver.html
[2] https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html
[3] https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html


title: Pytorch exported Model working on Windows but not Linux

body:
I am trying to deploy my flask application using Docker, but before running it in a container, I would like to verify that it works in a linux environment.
After installing all the packages in the right versions (pytorch 1.6, torchtext 0.7.0) as I did in windows, I get the following error for my exported model:
RuntimeError: Error(s) in loading state_dict for RNN:
        size mismatch for embedding.weight: copying a param with shape torch.Size([101616, 100]) from checkpoint, the shape in current model is torch.Size([100944, 100]).

How can i fix this?


title: Repeat matrix operation for several batches without using for loop

body:
I tried this operation for one batch and worked.
x = torch.FloatTensor([[ax,bx],[cx,dx],[ex,fx],[gx,hx]])
y = torch.FloatTensor([[ay,by],[cy,dy],[ey,fy],[gy,hy]])
z = x[:,None,0] * y[:,1]

print(z)

tensor([[ ax*by,  ax*dy,  ax*fy,  ax*hy],
        [ cx*by,  cx*dy,  cx*fy,  cx*hy],
        [ ex*by,  ex*dy,  ex*fy,  ex*hy],
        [ gx*by,  gx*dy,  gx*fy,  gx*hy]])

I want to repeat above operation for several batches like below example without using for loop.
x = torch.FloatTensor([[[ax1,bx1],[cx1,dx1],[ex1,fx1],[gx1,hx1]],
                       [[ax2,bx2],[cx2,dx2],[ex2,fx2],[gx2,hx2]]])
y = torch.FloatTensor([[[ay1,by1],[cy1,dy1],[ey1,fy1],[gy1,hy1]],
                       [[ay2,by2],[cy2,dy2],[ey2,fy2],[gy2,hy2]]])

This is what I want to get for result.
tensor([[[ ax1*by1,  ax1*dy1,  ax1*fy1,  ax1*hy1],
         [ cx1*by1,  cx1*dy1,  cx1*fy1,  cx1*hy1],
         [ ex1*by1,  ex1*dy1,  ex1*fy1,  ex1*hy1],
         [ gx1*by1,  gx1*dy1,  gx1*fy1,  gx1*hy1]],

        [[ ax2*by2,  ax2*dy2,  ax2*fy2,  ax2*hy2],
         [ cx2*by2,  cx2*dy2,  cx2*fy2,  cx2*hy2],
         [ ex2*by2,  ex2*dy2,  ex2*fy2,  ex2*hy2],
         [ gx2*by2,  gx2*dy2,  gx2*fy2,  gx2*hy2]]])

I tried as below, but gave me errors
z = x[...,None,0] * y[...,1]
RuntimeError: The size of tensor a (4) must match the size of tensor b (2) at non-singleton dimension 1

Is there any way doing this without for loop?


title: Shuffling parameters of a pretrained model in pytroch?

body:
I imported a pretrained resnet50 in Pytorch and I want to shuffle the weights, how can it be done?
I have done this-
    model_conv = torchvision.models.resnet50(pretrained=True)

    for param in model_conv.parameters():
       torch.utils.data.DataLoader(dataset=param, shuffle=True)


title: Torch self implementation of Neural Network

body:
I made this Neural Network but something is wrong with my backward derivatives which cause the wights to be 'nan' and the prediction to be very low.
I used Softmax and Cross entropy loss.
class Neural_Network:
    def __init__(self, input_size, output_size, hidden_size):
        # parameters
        self.inputSize = input_size
        self.outputSize = output_size
        self.hiddenSize = hidden_size

        # weights
        self.W1 = torch.randn(self.inputSize, self.hiddenSize)
        self.b1 = torch.zeros(self.hiddenSize)

        self.W2 = torch.randn(self.hiddenSize, int(self.hiddenSize/2))
        self.b2 = torch.zeros(int(self.hiddenSize/2))

        self.W3 = torch.randn(int(self.hiddenSize/2), self.outputSize)
        self.b3 = torch.zeros(self.outputSize)

    def forward(self, X):
        self.z1 = torch.matmul(X, self.W1) + self.b1
        self.h1 = tanh(self.z1)
        self.z2 = torch.matmul(self.h1, self.W2) + self.b2
        self.h2 = tanh(self.z2)
        self.z3 = torch.matmul(self.h2, self.W3) + self.b3

        return softmax(self.z3)

    def backward(self, X, y, y_hat, lr=.1):
        batch_size=y.size(0)

        y_hat1 = tnnfunc.one_hot(y_hat)
        y1 = tnnfunc.one_hot(y)

        if y1.shape[1]<10:
          print('enter y1 reshape')
          diff = 10-y1.shape[1]
          add = torch.zeros(y1.shape[0],diff)
          y1 = torch.cat((y1,add),1)
          #y1=y1_T.T

        if y_hat1.shape[1]<10:
          print('enter y1_hat reshape')
          diff = 10-y_hat1.shape[1]
          add = torch.zeros(y_hat1.shape[0],diff)
          y_hat1 = torch.cat((y_hat1,add),1)

        dl_dz3=(1/batch_size)*(y_hat1-y1)
        dl_dh2=torch.matmul(dl_dz3,torch.t(self.W3))
        dl_dz2=dl_dh2*(self.h2)
        dl_dh1=torch.matmul(dl_dz2,torch.t(self.W2))
        dl_dz1=dl_dh1*self.h1

        self.W1 -= lr*torch.matmul(torch.t(X), dl_dz1)
        self.b1 -= lr*torch.matmul(torch.t(dl_dz1), torch.ones(batch_size))
        self.W2 -= lr*torch.matmul(torch.t(self.h1), dl_dz2)
        self.b2 -= lr*torch.matmul(torch.t(dl_dz2), torch.ones(batch_size))
        self.W3 -= lr*torch.matmul(torch.t(self.h2), dl_dz3)
        self.b3 -= lr*torch.matmul(torch.t(dl_dz3), torch.ones(batch_size))

    def train(self, X, y):
        print('enter training')
        # forward + backward pass for training
        o = self.forward(X)
        _, predicted = torch.max(o.data, 1)
        self.backward(X, y, predicted)


title: Log_softmax function returns "nan" in raspbian OS running torch wheels, but works fine in anaconda (windows OS)

body:
I have been working on a Pytorch model for one of my testing purposes. So far I trained the model using anaconda in windows and the model is working fine. I wanted to train the same model in raspberry pi 4 (aarch64). So I did the following,
Deployed Raspbian Lite OS
Used torch wheels to install torch in Raspberry PI as it has ARM architecture ( got the wheels for torch(v1.7) and Torchvision(v.0.8.1) from - https://mathinf.eu/pytorch/arm64/ )
Torch was imported successfully and I ran the model in raspberry pi without errors.
But the loss got a value of nan. So I debugged and found out that there was no issue with the data or data transformation (no nan inputs). But found out that
F.log_softmax(r_out2, dim=1)

returns a nan value from the beginning of first batch of data itself.
Anaconda environment versions:

numpy - 1.18.5
torch - 1.7.0
torchvision - 0.8.1

Raspberry Pi versions:

torch - v1.7
torchvision - v.0.8.1
numpy - v1.18.5 (tried upgrading to latest version as well but no luck)

I have no clue why the same model and same data returns nan, when it runs perfectly well in the anaconda environment. The only issue is with the log_softmax as of yet.
Does it have something to do with the torch wheels I installed ?
Or does it have to do something with the processing power on raspberry ?


title: how to set the starting point for nvprof in python code

body:
I am trying to use nvprof to profile my python code. But I wanted to start from a specific
code point. I know how to do this in C/C++ using cudaProfilerStart. But how to set for python code or pytorch ?


title: Why does my function get good values for LSTM but not for GRU?

body:
I'm trying to implement a program that compares LSTM's performance vs GRU's performance for word prediction. I am using the same parameters for both of them, however while I am getting good perplexity values for the LSTM, the GRU values I'm getting are absolutely terrible.
I recently attempted to debug the training function since it originally only ranfor the LSTM model but not for the GRU model. As I already said, both models should get similar values, however for now the LSTM models starts with around ~150 perplexity and converges to a normal value, when the GRU model starts with some random value that's in the 1000s that does not converge at all.
I am quite new for all the RNN, LSTM, and GRU stuff, so forgive me if there's something obvious that I am missing.
Any help would be appriciated!
I use the following two models:
class LSTM_Model(nn.Module):
    def __init__(self, vocab_size, embed_size, hidden_size, num_layers, dropout=0):
        super(LSTM_Model, self).__init__()
        self.embed = nn.Embedding(vocab_size, embed_size)
        self.lstm = nn.LSTM(embed_size, hidden_size, num_layers, batch_first=True, dropout = dropout)
        self.fc = nn.Linear(hidden_size, vocab_size)

    def forward(self, x, hidden_state):
        x = self.embed(x)
        out, (hidden_state, cell_state) = self.lstm(x, hidden_state)
        out = out.reshape(out.size(0)*out.size(1), out.size(2)) # Reshape output to (batch_size*sequence_length, hidden_size)
        out = self.fc(out)
        return out, (hidden_state, cell_state)

class GRU_Model(nn.Module):
    def __init__(self, vocab_size, embed_size, hidden_size, num_layers, dropout=0):
        super(GRU_Model, self).__init__()
        self.embed = nn.Embedding(vocab_size, embed_size)
        self.gru = nn.GRU(embed_size, hidden_size, num_layers, batch_first=True, dropout = dropout)
        self.fc = nn.Linear(hidden_size, vocab_size)

    def forward(self, x, hidden_state):
        x = self.embed(x)
        out, hidden_state = self.gru(x, hidden_state)
        out = out.reshape(out.size(0)*out.size(1), out.size(2)) # Reshape output to (batch_size*sequence_length, hidden_size)
        out = self.fc(out)
        return out, hidden_state

Training function:
def run_model(model, epochs=epochs, learning_rate=learning_rate, clip=clip, momentum=momentum, LSTM=True, GRU=False, Dropout=False):
  # Define loss criterion and optimizer
  criterion = nn.CrossEntropyLoss()
  optimizer = optim.SGD(model.parameters(), lr=learning_rate, momentum=momentum)
  lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer=optimizer, step_size=step_size, gamma=decay_rate)

  train_perplexity, test_perplexity, valid_perplexity = [], [], []

  # Train the model
  for e in range(epochs):
      # Set all initial hidden and cell states to zeroes
      train_states=init_states(LSTM, GRU, num_layers, batch_size, hidden_size)
      test_states=init_states(LSTM, GRU, num_layers, batch_size, hidden_size)
      valid_states=init_states(LSTM, GRU, num_layers, batch_size, hidden_size)

      # RUN TRAINING SET #
      model.train()
      for i in range(0, ids.size(1) - seq_length, seq_length):
          # Set train_inputs and train_targets
          train_inputs = ids[:, i:i+seq_length].to(device)
          train_targets = ids[:, (i+1):(i+1)+seq_length].to(device)

          # Forward pass
          model.zero_grad()
          if(LSTM==True):
              train_states = [state.detach() for state in train_states] # Detach the hidden state from how it was previously produced
          if(GRU==True):
              train_states = train_states.data #detach?
          train_outputs, train_states = model(train_inputs, train_states)
          train_loss = criterion(train_outputs, train_targets.reshape(-1))

          # Backward and optimize
          train_loss.backward()
          clip_grad_norm_(model.parameters(), clip)
          optimizer.step()

      lr_scheduler.step()
      model.eval()
      with torch.no_grad():
       #test and validation, removed to reduce length

      model.train() # reset to train mode after iterating through validation data
      train_perplexity.append(math.exp(train_loss.item()))
      test_perplexity.append(np.exp(np.mean(test_losses)))
      valid_perplexity.append(np.exp(np.mean(valid_losses)))

      print('Epoch ' + str(e+1) + '/' + str(epochs) + ': ')
      print('Train Perplexity - ' + str(train_perplexity[e]))
      print('Test Perplexity - ' + str(test_perplexity[e]))
      print('Validation Perplexity - ' + str(valid_perplexity[e]))
      print("----------------------------------------------------")
  return train_perplexity, test_perplexity, valid_perplexity

Hidden state initialization:
def init_states(LSTM, GRU, num_layers=num_layers, batch_size=batch_size, hidden_size=hidden_size):
    if (LSTM==True):
        return (torch.FloatTensor(num_layers, batch_size, hidden_size).uniform_(r1, r2).to(device),
                    torch.FloatTensor(num_layers, batch_size, hidden_size).uniform_(r1, r2).to(device))
    if (GRU==True):
        return torch.FloatTensor(num_layers, batch_size, hidden_size).uniform_(r1, r2).to(device)


title: Is the differentiation correct in pytorch tutorial?

body:
I'm going through https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#sphx-glr-beginner-blitz-autograd-tutorial-py and noticed an equation I couldn't understand. For the below equation for the tutorial,

Isn't do/dx = 6 (xi+2). How is it 3/2(xi+2)?


title: Adam costum implementation by PyTorch

body:
guys, could you help me.
I'm trying to implement my own realization of Adam optimization alghoritm, but when I try to find of optim for function f(x,y) = xx + yy, I find that method goes very strange.
Here is the code and graph for each point on the Adam's path and more simple alghoritm - SGD's path.
Thanks!
class optimizer:
    def __init__(self, params):
        self.parameters = list(params)

    def zero_grad(self):
        for param in self.parameters: # Have to be an iter object.
            try:
                param.grad.zero_()
            except:
                pass

    def step(self):
        pass

class Adam(optimizer):
    def __init__(self, params, lr, beta1=0.9, beta2=0.999):
        self.parameters = list(params)
        self.lr = lr
        self.beta1 = beta1
        self.beta2 = beta2
        self.EMA1 = [torch.zeros_like(param) for param in self.parameters]
        self.EMA2 = [torch.zeros_like(param) for param in self.parameters]
        self.iter_num = 0
        self.eps = 1e-9

    def step(self):
        self.iter_num += 1

        correct1 = 1 - self.beta1**self.iter_num # EMA1 bias correction.
        correct2 = 1 - self.beta2**self.iter_num # EMA2 bias correction.

        with torch.no_grad():

            for param, EMA1, EMA2 in zip(self.parameters, self.EMA1, self.EMA2):

                EMA1.set_((1 - self.beta1) * param.grad + self.beta1 * EMA1)
                EMA2.set_((1 - self.beta2) * (param.grad**2) + self.beta2 * EMA2)

                numenator = EMA1 / correct1
                denominator = (EMA2 / correct2).sqrt() + self.eps

                param -= self.lr * numenator / denominator


title: Why the video memory takes too much when I replace FC layers with FCN in vgg11 as a classifier?

body:
I just replaced FC layers with FCN in vgg11 as a classifier. But I don't know why the video memory takes too much using FCN than original FC layers. When I use original FC layers as a classifier, I can adjust batchsize to 32, it works well. Now, pycharm will give an error "cuda out of memory", I just change batchsize to 8 in using FCN.
Say, nn.Linear(512 * 7 * 7, 4096) converts to nn.Conv2d(512, 4096, (7,7)), I queried the video memory using the command nvidia-smi respectivtely when defining a rand tensor 10 * 512 * 7 * 7.
Here is my codes.
import torch
import torch.nn as nn

flag1 = False
if flag1:
    tensor1 = torch.randn(10, 512, 7, 7)
    tensor1 = tensor1.cuda()
    conv2d = nn.Conv2d(512, 4096, (7, 7))
    conv2d = conv2d.cuda()
    output1 = conv2d(tensor1)

flag2 = True
if flag2:
    tensor2 = torch.randn(10, 512, 7, 7)
    tensor2 = tensor2.cuda()
    linear = nn.Linear(512 * 7 * 7, 4096)
    linear = linear.cuda()
    output2 = linear(tensor2.view(10, -1))

print("end")

Here is the output of video merrory.
orginal video memory
video memory using FCN
video memory using FC layers
Since they have same prameters, why the video memory usage is so different? Looks like FC layers is better than FCN. I don't understand.


title: Using textual and non textual features in torchtext

body:
I have a dataset consisting of text and other numerical features which I want to my multi input neural network.
An example of a dataset:

index
label
text
feature1
feature2
feature3
feature4

1
1
This is text :)
0.2
0.4
0.5
0.7

Is there a proper way to  build a torchtext field that will take the whole subset of columns as a tensor:
text_field = Field(sequential=True, use_vocab=True, init_token='<SOS>', eos_token='<EOS>', fix_length=1000)
label_field = Field(sequential=False, use_vocab=False)
meta_features_field = Field(sequential=False, use_vocab=False)

Maybe somehow to pass a list of columns:
features_columns_list = [feature1, feature2, feature3, feature4]
dataset = Dataset(examples='path/to/dataset', fields={'label': label_field, 'text': text_field, features_columns_list : meta_features_field })

If this cant be done, would be glad to hear other suggestions to work with multimodal datasets.
Thanks in advance :)


title: ValueError using enumerate in Pytorch (HDF5 data)

body:
whenever I'm loading my custom dataset (reduced CIFAR-10 to 512 features) and try to train it using the common methods, I run into a ValueError.
Since I've been handed a HDF5 file, I defined my custom Pytorch-Dataset as follows:
class HDF5Dataset(Dataset):
    # load the dataset
    def __init__(self, path):
        # load the csv file as a dataframe
        features = pd.DataFrame(np.array(h5py.File(path)['features']))
        labels = pd.DataFrame(np.array(h5py.File(path)['labels']))

        # store the inputs and outputs
        self.X = features
        self.y = labels
        # ensure input data is floats
        self.X = self.X.astype('float32')
        # label encode target and ensure the values are floats
        self.y = LabelEncoder().fit_transform(self.y)
        self.y = self.y.astype('float32')
        self.y = self.y.reshape((len(self.y), 1))

    # number of rows in the dataset
    def __len__(self):
        return len(self.X)

    # get a row at an index
    def __getitem__(self, idx):
        return self.X[idx], self.y[idx]

    # get indexes for train and test rows
    def get_splits(self, n_test=0.33):
        # determine sizes
        test_size = round(n_test * len(self.X))
        train_size = len(self.X) - test_size
        print(test_size)
        # calculate the split
        return random_split(self, [train_size, test_size])

When I'm now trying to execute my custom defined (toy) neural network via
path = "resnet50_dim512.hdf5"
train_dl, test_dl = prepare_data(path)
print(len(train_dl.dataset), len(test_dl.dataset))
# define the network
model = MLP(34)
# train the model
train_model(train_dl, model)

I run into the following error message:
ValueError                                Traceback (most recent call last)
~/anaconda3/lib/python3.8/site-packages/pandas/core/indexes/range.py in get_loc(self, key, method, tolerance)
    349             try:
--> 350                 return self._range.index(new_key)
    351             except ValueError:

ValueError: 901 is not in range

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-37-fd90001f6a16> in <module>
    156 # train the model
    157 print(train_dl)
--> 158 train_model(train_dl, model)
    159 # evaluate the model
    160 acc = evaluate_model(test_dl, model)

<ipython-input-37-fd90001f6a16> in train_model(train_dl, model)
    106     for epoch in range(100):
    107         # enumerate mini batches
--> 108         for i, (inputs, targets) in enumerate(train_dl):
    109             # clear the gradients
    110             optimizer.zero_grad()

The other functions that I'm using follow the standard approach, but maybe there are issues in that matter, so I'm including them below.
def prepare_data(path):
    # load the dataset
    dataset = HDF5Dataset(path)
    # calculate split
    train, test = dataset.get_splits()
    # prepare data loaders
    train_dl = DataLoader(train, batch_size=32, shuffle=True)
    test_dl = DataLoader(test, batch_size=1024, shuffle=False)
    return train_dl, test_dl

And the training function where the error occurs:
def train_model(train_dl, model):
    # define the optimization
    criterion = BCELoss()
    optimizer = SGD(model.parameters(), lr=0.01, momentum=0.9)

    # enumerate epochs
    for epoch in range(100):
        # enumerate mini batches
        for i, (inputs, targets) in enumerate(train_dl):
            # clear the gradients
            optimizer.zero_grad()
            # compute the model output
            yhat = model(inputs)
            # calculate loss
            loss = criterion(yhat, targets)
            # credit assignment
            loss.backward()
            # update model weights
            optimizer.step()

It would be great if someone can tell me why this error happens, how to fix it or even circumvent working on the HDF5 file in this manner, or if it would be a better approach to first transform it into another file format in order to ease the training process.
All the best.


title: How to compute stride mean in Pytorch?

body: For example, we have x = [1, 2, 3, 4, 5] with stride = 3, step = 1, then the stride mean would be result = [2, 3, 4]. More specifically, I have a vector with shape = (batch_size, sequence_len, embed_dim), and I want to perform the above operation on dim = 1 (sequence_len). How can I implement that efficiently in PyTorch?


title: osError: [Errno 22] Invalid argument due to batch size

body: I get this error in Google colab. If I use GPU and I set to batch size 16 I get this error but I set to batch size 32 I didnt get error, Also If I didnt use GPU, I can set batch size to 16. What is happening ?


title: Running CTRL model, getting "could not find Pytorch checkpoint" on colab

body:
I'm stumped as I'm just running a template of the Salesforce CTRL on Colab.
My error is happening in this last section of the code:
!python2 generation.py --model seqlen256_v1.ckpt/model.ckpt-413000.data-00000-of-00001

I'm getting an error like this:
   /usr/local/lib/python2.7/dist-packages/OpenSSL/crypto.py:14: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography import utils, x509
246534 unique words
tcmalloc: large alloc 1262256128 bytes == 0x5643035d4000 @  0x7ff4fb8d1b6b 0x7ff4fb8f1379 0x7ff4a67c9b4a 0x7ff4a67cb5fa 0x7ff4a8afb78a 0x7ff4a8d4430b 0x7ff4a8d8bb37 0x7ff4a8afc0b0 0x7ff4a8b05d95 0x7ff4a8e3f973 0x7ff4a8e83709 0x7ff4f18a9c43 0x7ff4f1680adb 0x5642f48487ca 0x5642f4845e0a 0x5642f48616a9 0x5642f4879e1e 0x5642f48799ca 0x5642f4836a6b 0x5642f484d750 0x5642f4845e0a 0x5642f4845879 0x5642f4875f5f 0x5642f4871222 0x5642f4870c4d 0x5642f481fa86 0x7ff4fb4ecbf7 0x5642f481f37a
Loading vocabulary from vocab ...
Read 6086453827 words (246531 unique) from vocabulary file.
Loading codes from codes ...
Read 200000 codes from the codes file.
Could not find PyTorch checkpoint
Converting weights and will store the PyTorch checkpoint as  5d11fdadc5aa3b7e75036e3df8c58c2b
Traceback (most recent call last):
  File "pytorch_generation.py", line 114, in <module>
    reader = pywrap_tensorflow.NewCheckpointReader(chkpt_for_reader)
AttributeError: 'module' object has no attribute 'NewCheckpointReader'


title: What distribution is used when you make a tensor with torch.Tensor constructor in Pytorch?

body:
I typed and ran torch.Tensor(2, 3) in Google Colab. It did work but it returned an weird-valued 2x3 tensor which includes even nan.
tensor([[3.8202e-36, 0.0000e+00, 3.9236e-44],
        [0.0000e+00,        nan, 1.8750e+00]])

I searched Pytorch(1.7.1)'s tensor.Tensor Doc to find out what distribution the default constructor has but the case when you create a tensor with Tensor class constructor was not written.
what happens when you use Tensor class constructor and what are the parameters for it?


title: Train simple RNN from a map of T/F values and bitstring keys in pytorch

body:
I want to train rnn netwwork that will be able to identify the folowing:
I have a Language L that made of bitstrings (can be in lengh n>20) which every bitstring that in the language satisfy the pattern of the language (which is not rellevant for now).
I created a dataset which is like a map with bitstrings as keys and True/False as values:
001101000010010, False
111001001000000, False
111011101001100, False
011111000101101, False
10000110000110001100, True
011100100001010, False
....

I tried to create rnn network in pytorch:
class myDataset(T.utils.data.Dataset):

def __init__(self, src_file, m_rows=None):
    tmp_x = np.loadtxt(src_file, max_rows=m_rows,
                       usecols=[0], delimiter=",", skiprows=0, dtype=np.int64)

    tmp_y = np.genfromtxt(src_file, max_rows=m_rows,
                       usecols=[1], delimiter=",", dtype=bool)

    tmp_y = tmp_y.reshape(-1, 1)  # 2-D required

    self.x_data = T.from_numpy(tmp_x).to(device)
    self.y_data = T.from_numpy(tmp_y).to(device)

def __len__(self):
    return len(self.x_data)

def __getitem__(self, idx):
    preds = self.x_data[idx, :]  # or just [idx]
    val = self.y_data[idx, :]
    return (preds, val)  # tuple of two matrices

and train it:
    net.train()  # set mode
for epoch in range(0, max_epochs):
    T.manual_seed(1 + epoch)  # recovery reproducibility
    epoch_loss = 0  # for one full epoch

    for (batch_idx, batch) in enumerate(train_ldr):
        (X, Y) = batch  # (predictors, targets)
        optimizer.zero_grad()  # prepare gradients
        oupt = net(X)  # predicted prices
        loss_val = loss_func(oupt, Y)  # avg per item in batch
        epoch_loss += loss_val.item()  # accumulate avgs
        loss_val.backward()  # compute gradients
        optimizer.step()  # update wts

but I get error when loading the data:

OverflowError: Python int too large to convert to C long


title: Second order derivative information in GPytorch

body:
I was wondering whether there is a straightforward way to construct a GPModelWithDerivatives class (similar to this tutorial), where I could use second derivative information (e.g. d^2f/d[x^2] = d/dx(df/dx) ).
Any thoughts?


title: checkpoint file not detected in colab after reconnecting

body:
!pwd gives /content/gdrive/My Drive/slt/sign_sample_model
!ls gives best.ckpt    gls   references.dev.gls  train.log
config.yaml  gls.vocab  tensorboard     validations.txt
checkpoint = torch.load('best.ckpt')``` gives error  <br/>
 **FileNotFoundError: [Errno 2] No such file or directory: 'best.ckpt'**


title: Shadow Generation using pix2pix

body:
I'm working on projects where I have to generate close to real shadows on the car. I thought pix2pix model would work great. so I downloaded images of car from the internet with shadows isolated cars from image by segmentation model. In this way created the dataset of paired images. first I tried overfitting on small 30-40 images just to see if the model can handle it and it did, model worked Amazing. So I Switched to large dataset there are some things I'm noticing the model is doing fine on images from internet but doing worse on images of car taken by camera. Real images of Cars do exists in dataset there are 30% of images are of real car. Also the generator loss isn't decreasing beyond 35. I'm taking a RGB image as input with 3 channel and producing a RGBA channel as output with 4 channels
Can anyone point out mistake?
#update
It seems like discriminator's loss is 0.09 that is almost negligible
and Generator loss is increasing. I am thinking due to this discriminator isn't providing enough feedback to the generator any idea how to mitigate This ?


title: Memory issues for creating predictions using a pytorch model

body: I have made a simple neural network in pytorch and already trained it. Now, I want to pass a relatively big data set and save predictions of the model. The dataset is about 40million rows and 5 columns all containing float values. I run this model on an instance with 12 CPUs and 128GB memory available. I use dataloader in pytorch to break the data into different batch sizes and after around 10% of data is passed for predictions, the instance runs out of memory. I used htop command to see how the process goes and all CPUa are almost fully utilized and memory quickly gets used to it's max. How can I handle this situation?


title: Multi task LSTM using Pytorch for Non-NLP tasks

body:
I am trying to implement a multi task LSTM which obtains sequence of inputs and returns task specific outputs, but couldn't find proper method to do it. Existing Multitask LSTM works only for NLP tasks like text chunking, summarization. Can it be done to obtain two task specific output with common input sequence using single LSTM network. Initially tried adding two Fully connected layer after LSTM unit to obtain two task specific outputs, but how will the LSTM know which sequence of input represent which task specific class.
For example. Consider task A has 9 classes, and task B has 3 classes. Both these task have a common input sequence I. The LSTM should give two outputs for task A out of 9 classes and task B out of 3 classes. I have two question.

Is it possible to create such a LSTM network. If so how to create such a network(should we add fully connected layers) or is there any method?
If possible, how will the LSTM know which Sequence belong to which task and which task specific class

Thanks in advance


title: add datatable in pytorch tensorboard

body:
I want to add training and validation loss to tensorboard as well as the result of test set.
I can't find add datatable in tensorboard.
Is the effort to add a data table to the tensor board meaningless?
Then let me know the alternative.


title: When does one divide by the meta_batch_size for MAML during meta-learning?

body:
I was going through the maml example in the higher library but it seems to me they are forgetting to divide by the meta_batch_size, they have:
qry_loss.backward()

where they should have
meta_batch_size = qry_loss.size(0)
(qry_loss/meta_batch_size).backward()

is this correct? should they be dividing by the meta_batch_size?
references:

full code example working on omniglot: https://github.com/facebookresearch/higher/blob/master/examples/maml-omniglot.py
crossposted: https://github.com/facebookresearch/higher/issues/104
cross: https://discuss.pytorch.org/t/when-does-one-divide-by-the-meta-batch-size-for-maml-during-meta-learning/114625


title: How to solve gradient explosion/loss issue in custom loss function in PyTorch?

body:
I've written a custom loss function in accordance with the directions of this paper: https://epubs.siam.org/doi/pdf/10.1137/1.9781611976236.18 However, when I try to train a model using this loss function, all the output values become NaN after the first loss.backwards() operation. My guess is that something in this code is causing the gradient to explode or be lost but I have no idea where/what it is. Could someone help? A minimum reproducible example is below:
import torch
from torch import optim
import torchvision
from torchvision import datasets
import timm # you may need to pip install this!!

### LOSS FUNCTION ###

def class_means(Z, Y):
    '''Returns unique classes in batch and per class means'''
    classes = torch.unique(Y)
    means = None
    for c in classes:
        if means == None:
            means = torch.mean(Z[Y == c], axis = 0)
        else:
            means = torch.vstack([means, torch.mean(Z[Y == c], axis = 0)])
    return classes, means

def intra_spread(Z, Y, classes, means):
    ''' Takes the L2 norm of all outputs (Z) from their respective class means
    and averages them'''
    N = Z.shape[0]
    intraSpread = 0
    for i in range(classes.shape[0]):
        intraSpread += torch.sqrt(torch.sum((means[i] - Z[Y == classes[i]]) ** 2))
    return intraSpread / N

def similarity_matrix(mat):
    '''Return the distances between all rows of the input matrix'''
    r = torch.mm(mat, mat.t())
    diag = r.diag().unsqueeze(0)
    diag = diag.expand_as(r)
    D = diag + diag.t() - 2*r
    return D.sqrt()

def inter_separation(means):
    '''Returns the distance between the two closest means in input means'''
    return torch.min(similarity_matrix(means)[similarity_matrix(means) > 0])

def ii_loss(Z, Y):
    '''Returns intraSpread - interSep'''
    classes, means = class_means(Z, Y)
    intraSpread = intra_spread(Z, Y, classes, means)
    interSep = inter_separation(means)
    return intraSpread - interSep

criterion = ii_loss # use ii_loss as the criterion

# Loading in CIFAR 10. Make a files folder in your working directory
batch_size_train = 32
train_loader = torch.utils.data.DataLoader(
torchvision.datasets.CIFAR10('./files/', train=True, download=True,
                        transform=torchvision.transforms.Compose([
                           torchvision.transforms.Resize((300, 300)),
                           torchvision.transforms.ToTensor(),
                           torchvision.transforms.Normalize(
                             (0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
                         ])),
  batch_size=batch_size_train, shuffle=True)

### Fitting Model ###
## Load model
device = 0 # or 0 for gpu
m = timm.create_model('efficientnet_b3', pretrained = True)
m.classifier = torch.nn.Linear(m.classifier.in_features, 10) # 10 classes in cifar10
m = m.to(device)

## Establish hyperparameters
learning_rate = 1e-4
momentum = 0.9
reg = 1e-4
epochs = 18
decay_epochs = 15
decay = 0.1
optimizer = optim.Adam(m.parameters(), lr = learning_rate)

## Train model
for i in range(epochs):
    print("Epoch: " + str(i + 1))
    for batch_idx, (inputs, targets) in enumerate(train_loader):
        print("  " + str(batch_idx + 1) + "...", end = "")
        inputs = inputs.to(device) # X_batch
        targets = targets.to(device) # Y_batch
        optimizer.zero_grad()
        outputs = m(inputs) # Z_batch
        loss = criterion(outputs, targets) # loss_batch
        loss.backward()
        optimizer.step()
        print("DONE")


title: Basic question about CNN : relationship between 'number of parameter' and 'learning speed'

body:
I'm newbie in CNN.
My model classify 3D image. When I modify my model more complex by making more layer and more parameter, learning speed become quite slow. Learning speed means 'the rate of decrease in loss according to training epoch'. In some case, model seems to be trapped in local region and loss doesn't decrease at all from early stage of training.
Is this natural?


title: Loss from external data

body:
I'm trying to implement reinforcement learning algorithms in which a python neural network interacts with an external program: a simulator (via socket). The network outputs actions, applied in the simulator which is able to return a score function that I want to use. However when I tried to use it to compute a loss function, I'm always having errors telling me that no gradient has been computed. I tried in TensorFlow (in eager mode with gradient tape) and in PyTorch but in both cases it fail. I can only compute a loss which is directly function of the network output.
My understanding is that these framework are declarative and thus efficiently compute gradient thanks to the formal description which results of the declaration of the network and of the loss. But that's only suppositions (that I thought incorrect for PyTorch, and TensorFlow in eager mode).
I would like to understand what is the issue. How does it works exactly? And how can I use computations based on my external score to get a loss I can use? Currently I had to reimplement an optimization function, but I would like to use the already implemented (and more efficient) ones
Thank you for your help.


title: Pytorch CNN for Chess position evaluation not learning

body:
I am trying to create a network that will provide a value [-1, 1] of a chess position.
I am using a Kaggle dataset of board positions and evaluations by stockfish that can be found here: https://www.kaggle.com/ronakbadhe/chess-evaluations
However, my net is not learning (loss starts at around 0.2 then decreases to 0.18 then starts to increase again).
I am following the structure of L0Zero that can be found here https://lczero.org/dev/backend/nn/
Here is the structure of my network:
class SqueezeExcitation(nn.Module):
  def __init__(self, filters, se_channels):
    super().__init__()
    self.filters, self.se_channels = filters, se_channels

    self.pool1 = nn.AvgPool2d(8)

    self.fc1 = nn.Linear(self.filters, self.se_channels)
    self.relu = nn.ReLU()

    self.fc2 = nn.Linear(self.se_channels, 2 * self.filters)
    self.sig = nn.Sigmoid()

  def forward(self, x):
    a = self.pool1(x)
    a = a.view(-1, 128)

    a = self.fc1(a)
    a = self.relu(a)
    a = self.fc2(a)

    W, B = torch.split(a, self.filters, dim=1)
    W = W.view(-1, self.filters, 1, 1)
    B = B.view(-1, self.filters, 1, 1)

    Z = self.sig(W)
    return (Z * x) + B

class Residual(nn.Module):
    def __init__(self, filters, se_channels):
      super().__init__()
      self.filters, self.se_channels = filters, se_channels

      self.conv1 = nn.Conv2d(self.filters, self.filters, 3, padding=1)
      self.norm1 = nn.BatchNorm2d(self.filters)
      self.relu1 = nn.ReLU()
      self.conv2 = nn.Conv2d(self.filters, self.filters, 3, padding=1)
      self.norm2 = nn.BatchNorm2d(self.filters)
      self.relu2 = nn.ReLU()

      self.sqex = SqueezeExcitation(self.filters, self.se_channels)
      self.relu3 = nn.ReLU()

    def forward(self, x):
      x = self.conv1(x)
      x = self.norm1(x)
      x = self.relu1(x)

      x = self.conv2(x)
      x = self.norm2(x)
      x = self.relu2(x)

      sqex = self.sqex(x)
      x = x + sqex

      return self.relu3(x)

class ChessNet(nn.Module):
  def __init__(self, blocks, filters, se_channels):
    super().__init__()
    self.blocks, self.filters, self.se_channels = blocks, filters, se_channels

    #Initial conolutions
    self.conv1 = nn.Conv2d(6, 112, 3, padding=1)
    self.conv2 = nn.Conv2d(112, self.filters, 3, padding=1)

    BLOCKS = OrderedDict([(f'block{i + 1}', Residual(self.filters, self.se_channels)) for i in range(blocks)])
    #Residual Blocks
    self.sequence = nn.Sequential(BLOCKS)

    #Value Head
    self.conv3 = nn.Conv2d(self.filters, 32, 1)
    self.norm1 = nn.BatchNorm2d(32)
    self.relu1 = nn.ReLU()
    self.conv4 = nn.Conv2d(32, 128, 8)
    self.relu2 = nn.ReLU()

    self.fc1 = nn.Linear(128, 1)
    self.tanh = nn.Tanh()

  def forward(self, x):
    x = self.conv1(x)
    x = self.conv2(x)
    x = self.sequence(x)

    x = self.conv3(x)
    x = self.norm1(x)
    x = self.relu1(x)
    x = self.conv4(x)
    x = x.view(x.size(0), -1)

    x = self.relu2(x)
    x = self.fc1(x)

    return self.tanh(x)

You can find the Colab notebooks I used to both generate the data and to train the model here: https://colab.research.google.com/drive/1-nG12yqr0QNDOW_Ur63JSaRq6jeBv1wV?usp=sharing, https://colab.research.google.com/drive/1AYAkhkbT0WkYW1lQEmI3F74B8f-V5vY3?usp=sharing.
Any help would be much appreciated.


title: How gradients are accumulated in real

body:
‘Gradient will not be updated but be accumulated, and updated every N rounds.’ I have a question that how the gradients are accumulated in the below code snippet: in every round of the below loop I can see a new gradient is computed by loss.backward() and should be stored internally, but would this internally stored gradient be refreshed in the next round? How the gradient is summed up, and later be applied every N rounds?
for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)                     # Forward pass
    loss = loss_function(predictions, labels)       # Compute loss function
    loss = loss / accumulation_steps                # Normalize our loss (if averaged)
    loss.backward()                                 # Backward pass
    if (i+1) % accumulation_steps == 0:             # Wait for several backward steps
        optimizer.step()                            # Now we can do an optimizer step
        model.zero_grad()


title: How to find newly created objects via gc.get_objects() which are causing a memory leak

body:
I've been trying to solve a memory leak within my code via monitoring the number of objects created. To give a short overview, I have a PyTorch module which trains to solve a problem by iteratively make a prediction and minimising an error over a given number of epochs.
I've been using the package gc to monitor the number of objects created within my script and it slowly rises (as does the RAM usage). Every epoch there's around 200mb-400mb of RAM increase. Is there a way to see which objects are new/differ between epochs? I.e. grab two different gc.get_objects() and see how they differ in memory size? And, more importantly, see which python lines are resulting in these objects increasing in memory usage which is causing my the memory leak!