title: Does BertForSequenceClassification classify on the CLS vector?

body:
I'm using the Huggingface Transformer package and BERT with PyTorch. I'm trying to do 4-way sentiment classification and am using BertForSequenceClassification to build a model that eventually leads to a 4-way softmax at the end.

My understanding from reading the BERT paper is that the final dense vector for the input CLS token serves as a representation of the whole text string:

  The first token of every sequence is always a special classification token ([CLS]). The final hidden state corresponding to this token is used as the aggregate sequence representation for classification tasks.

So, does BertForSequenceClassification actually train and use this vector to perform the final classification?

The reason I ask is because when I print(model), it is not obvious to me that the CLS vector is being used.

model = BertForSequenceClassification.from_pretrained(
    model_config,
    num_labels=num_labels,
    output_attentions=False,
    output_hidden_states=False
)

print(model)

Here is the bottom of the output:

        (11): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
      )
    )
    (pooler): BertPooler(
      (dense): Linear(in_features=768, out_features=768, bias=True)
      (activation): Tanh()
    )
  )
  (dropout): Dropout(p=0.1, inplace=False)
  (classifier): Linear(in_features=768, out_features=4, bias=True)

I see that there is a pooling layer BertPooler leading to a Dropout leading to a Linear which presumably performs the final 4-way softmax. However, the use of the BertPooler is not clear to me. Is it operating on only the hidden state of CLS, or is it doing some kind of pooling over hidden states of all the input tokens?

Thanks for any help.


title: Is there any way to copy all parameters of one Pytorch model to another specially Batch Normalization mean and std?

body:
I have found many correct ways online to copy one pytorch model parameters to another but somehow the copy-paste operation always misses the batch normalization parameters. Everything works fine as long as  I only use modules such as conv2d, linear, drop out, max pool etc in my model. But as soon as I add Batch normalization in pytorch model, the below-given script stop working and accuracy at test time is different :

net = model()
copy_net = model()

for param in net.module.parameters():
    copy_param.append(param.clone().detach())

count = 0
for param in copy_net.module.parameters():
    param.data =  copy_param[count]
    param.requires_grad = False
    count = count +1

Can anybody give me a possible solution to copying batch normalization also ?


title: Torch imports in python in terminal, but not in spyder

body:
Python 3.7.0, Spyder 3.7, macOS High Sierra

In the terminal, I activate the environment in which torch is installed, and then when I start python within the terminal (just by $python), I can import torch and work with it. However, when I open Spyder in the same environment ($spyder), it won't import torch, because it cannot find it for some reason.

I've tried out most things on the first couple of pages of google when you search this problem, including creating a new environment, tried both pip and conda installs, updated conda and spyder.

The errormessage I get is simply:

ModuleNotFoundError: No module named 'torch'

Does anyone know how I can fix this?


title: How to split and load huge dataset that doesn't fit into memory into pytorch Dataloader?

body:
I'm training a deep learning model to do multi label classification of diseases in NIH's Chest Xray-14 dataset using Google's Colab. I can't load all images into Dataloader at once, given around 112k training examples and limited RAM.

Is there a way to just store path of images in pytorch's DataLoader, reading only those images needed for current iteration during training, and once iteration is complete, the images are unloaded from memory, so on so forth until one epoch is complete.


title: addition of 2 pytorch tensors with diffrent size

body:
I have 2 tensors with dimension, A = [64,155,300] and B =[64,155,100]
when I add this 2 tensors ie. C= A+B,
I get this error ==> " RuntimeError: The size of tensor a (300) must match the size of tensor b (100) at non-singleton dimension 2 "

could anyone please help how should I add above tensors? any help will be appreciated!


title: PyTorch - AttributeError: 'bool' object has no attribute 'sum'

body:
I am running a deep learning model using PyTorch and getting the following error.

'correct+=(yhat==y_test).sum().int()'

AttributeError: 'bool' object has no attribute 'sum'

Below is a larger snippet of the code.

'''

for x_test, y_test in validation_loader:
            model.eval()
            z = model(x_test)
            yhat = torch.max(z.data,1)
            correct+=(yhat==y_test).sum().int()
            accuracy = correct / n_test
            accuracy_list.append(accuracy)

'''


title: ImportError:cannot import name 'Optional' from 'torch.jit.annotaions'

body:
I've installed pytorch and torchvision(not a cpuonly version) in anaconda. But when I run my code I get the following error:

Traceback (most recent call last):
  File "train.py", line 15, in <module>
    from model import Model
  File "/home/liangjing/code/DLAC/base_db/model.py", line 5, in <module>
    from ModelZoo import *
  File "/home/liangjing/code/DLAC/base_db/ModelZoo/__init__.py", line 1, in <module>
    from .TSN.tsn import TSN
  File "/home/liangjing/code/DLAC/base_db/ModelZoo/TSN/tsn.py", line 19, in <module>
    import Backbone
  File "/home/liangjing/code/DLAC/base_db/Backbone/__init__.py", line 4, in <module>
    from .inceptionv3.inception_v3 import inception_v3
  File "/home/liangjing/code/DLAC/base_db/Backbone/inceptionv3/inception_v3.py", line 6, in <module>
    from torch.jit.annotations import Optional
ImportError: cannot import name 'Optional' from 'torch.jit.annotations' (/home/liangjing/anaconda3/lib/python3.7/site-packages/torch/jit/annotations.py)

I've searched this problem for a long time, and someone tell me that it happened because the version of Pytorch and torchvision are not correspond to each other.But I check it and re-install them, the version of torch is 1.1.0 and torch vision is 0.3.0, it fit the requirement, so I sure it is not a version problem. How can I fix this problem?


title: Why is the super constructor necessary in PyTorch custom modules?

body:
Why does super(LR, self).__init__() need to be called in the code below? I get the error "AttributeError: cannot assign module before Module.init() call" otherwise. That error is caused by self.linear = nn.Linear(input_size, output_size).
I don't understand what the connection is between calling super(LR, self).__init__() and being able to assign the nn.Linear object to self.linear. nn.Linear is a separate object which can be assigned to a variable outside of any class, so why does super(LR, self).__init__() need to be called to assign a Linear object to self.linear within the class?
class LR(nn.Module):

    # Constructor
    def __init__(self, input_size, output_size):

        # Inherit from parent
        super(LR, self).__init__()
        self.test = 1
        self.linear = nn.Linear(input_size, output_size)

    # Prediction function
    def forward(self, x):
        out = self.linear(x)
        return out


title: Error while installing PyTorch using pip - cannot build wheel

body:
I get the following output when I try to run pip3 install pytorch or pip install pytorch
Collecting pytorch
  Using cached pytorch-1.0.2.tar.gz (689 bytes)
Building wheels for collected packages: pytorch
  Building wheel for pytorch (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/chaitanya/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3v4wd97t/pytorch/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3v4wd97t/pytorch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-8rsdyb8e
       cwd: /tmp/pip-install-3v4wd97t/pytorch/
  Complete output (5 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-3v4wd97t/pytorch/setup.py", line 15, in <module>
      raise Exception(message)
  Exception: You tried to install "pytorch". The package named for PyTorch is "torch"
  ----------------------------------------
  ERROR: Failed building wheel for pytorch
  Running setup.py clean for pytorch
Failed to build pytorch
Installing collected packages: pytorch
    Running setup.py install for pytorch ... error
    ERROR: Command errored out with exit status 1:
     command: /home/chaitanya/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3v4wd97t/pytorch/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3v4wd97t/pytorch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-eld9j0g4/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/chaitanya/.local/include/python3.8/pytorch
         cwd: /tmp/pip-install-3v4wd97t/pytorch/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-3v4wd97t/pytorch/setup.py", line 11, in <module>
        raise Exception(message)
    Exception: You tried to install "pytorch". The package named for PyTorch is "torch"
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/chaitanya/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3v4wd97t/pytorch/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3v4wd97t/pytorch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-eld9j0g4/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/chaitanya/.local/include/python3.8/pytorch Check the logs for full command output.

I downloaded the matching wheel from here, but am couldn't figure out what to do with it. My Python installation is using anaconda3, if that's needed. What should I do from here? Tips on how I could have resolved this on my own would also be appreciated.


title: Cannot Uninstall Pytorch

body:
I've installed pytorch 0.4.1 with both pip and conda in an effort to get a specific package working. Somehow version 1.5.0 is installed somewhere on my filesystem and despite my best efforts (following all these instructions) I cannot find and uninstall this version.
Running python3.6 -c "import torch; print(torch.__version__)" results in 1.5.0 no matter how many uninstalls I attempt.
I was using some virtual environments to help in my development, using venv.
I am running macOS 10.15.4, Intel Core i9 8-core, 32 GB 2667 DDR4, AMD Radeon Pro 5500M 8 GB.


title: How to deploy pytorch in docker image?

body: I want to deploy pytorch on a docker image. Is there a way that I can do that? I faced so much problems installing pytorch, their official installation links does not seem to be working; neither pip/conda works. Does anyone have any alternative solution in installing pytorch?


title: Unsupported GNU version! gcc versions later than 6 are not supported! - causes importing cpp_extension

body:
Then I import it says before error this:

/home/dex/anaconda3/lib/python3.6/site-packages/torch/utils/cpp_extension.py:118: UserWarning:

                               !! WARNING !!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (c++) may be ABI-incompatible with PyTorch!
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.

See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
for instructions on how to install GCC 4.9 or higher.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

                              !! WARNING !!

  warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))

But following gist  trying to install 4.9 and 5.9 same errors
Cant get installed, also on 18.04 but with other error:

  dex@dexpc:~$ sudo apt-get install software-properties-common
[sudo] password for dex:
Reading package lists... Done
Building dependency tree
Reading state information... Done
software-properties-common is already the newest version (0.96.24.32.4).
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
dex@dexpc:~$ sudo apt-get update
Ign:1 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Hit:4 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease
Hit:5 http://dl.google.com/linux/chrome/deb stable Release
Hit:6 http://ppa.launchpad.net/noobslab/apps/ubuntu bionic InRelease
Hit:7 https://download.docker.com/linux/ubuntu bionic InRelease
Hit:8 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Hit:9 http://ppa.launchpad.net/teejee2008/ppa/ubuntu bionic InRelease
Get:10 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Hit:11 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu bionic InRelease
Ign:13 http://ppa.launchpad.net/upubuntu-com/ppa/ubuntu bionic InRelease
Err:14 http://ppa.launchpad.net/upubuntu-com/ppa/ubuntu bionic Release
  404  Not Found [IP: 91.189.95.83 80]
Reading package lists... Done
E: The repository 'http://ppa.launchpad.net/upubuntu-com/ppa/ubuntu bionic Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
dex@dexpc:~$ sudo apt-get install gcc-5.9 g++-5.9
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package gcc-5.9
E: Couldn't find any package by glob 'gcc-5.9'
E: Couldn't find any package by regex 'gcc-5.9'
E: Unable to locate package g++-5.9
E: Couldn't find any package by glob 'g++-5.9'
E: Couldn't find any package by regex 'g++-5.9'

dex@dexpc:~$ gcc --version
gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0


title: How to properly Forward the dropout layer

body:
I created the following deep network with dropout layers like below:

class QNet_dropout(nn.Module):

    """
        A MLP with 2 hidden layer and dropout

        observation_dim (int): number of observation features
        action_dim (int): Dimension of each action
        seed (int): Random seed
    """

    def __init__(self, observation_dim, action_dim, seed):
        super(QNet_dropout, self).__init__()
        self.seed = torch.manual_seed(seed)
        self.fc1 = nn.Linear(observation_dim, 128)
        self.fc2 = nn.Dropout(0.5)
        self.fc3 = nn.Linear(128, 64)
        self.fc4 = nn.Dropout(0.5)
        self.fc5 = nn.Linear(64, action_dim)

    def forward(self, observations):
        """
           Forward propagation of neural network

        """

        x = F.relu(self.fc1(observations))
        x = F.linear(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.linear(self.fc4(x))
        x = self.fc5(x)
        return x

However, when I tried to run the code, I got the following errors:

/home/workspace/QNetworks.py in forward(self, observations)
     90
     91         x = F.relu(self.fc1(observations))
---> 92         x = F.linear(self.fc2(x))
     93         x = F.relu(self.fc3(x))
     94         x = F.linear(self.fc4(x))

TypeError: linear() missing 1 required positional argument: 'weight'

It seems like I didn't properly use/forward the dropout layer. What should be the correct way to do the Forward for the dropout layer? Thanks!


title: Why does my tensorboard say: 'No scalar data was found' when using SummaryWriter.add_scalar?

body:
I am attempting to visualize a loss vs. epoch graph using tensorboard (by calling .addscalar()) so that I can appropriately tune my neural network model. However, after training the model, I check my tensorboard in the web browser and it reads: 'No scalar data was found.'

I've tried checking my events statistics and it seems like there are scalars:

Event statistics for runs/tf_druggability/classification_ffnn_1:
audio -
graph -
histograms -
images -
scalars
   first_step           0
   last_step            90
   max_step             90
   min_step             0
   num_steps            10
   outoforder_steps     [(90L, 0L), (90L, 0L)]
sessionlog:checkpoint -
sessionlog:start -
sessionlog:stop -
tensor -

Here's my code:

from torch.utils.tensorboard import SummaryWriter

sess = tensorflow.Session()
writer = SummaryWriter('runs/tf_druggability/classification_ffnn_1', sess.graph)

for epoch in range(100):

    # Wrap input data and labels in Variable to can gradient descent (in place of DataLoader)
    inputs = Variable(traindf_to_tensor)
    labels = Variable(trainlabels_to_tensor)

    optimizer.zero_grad()
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    if(epoch%10 == 0):

        # HERE I WRITE to the LOG FILE for TENSORBOARD:
        writer.add_scalar('training loss', loss.item(), epoch)
        print('epoch {}, loss {}'.format(epoch, loss.item()))

writer.close()
print('Finished training! :)')

When I try to open the events file in Jupyter Notebooks it has this in it (... means I've omitted the middle part):

1800 0000 0000 0000 a37f 4b22 09f2 3d6b
a51e 58d7 411a 0d62 7261 696e 2e45 7665
6e74 3a32 ac51 0a16 2100 0000 0000 0000
...
f723 0000 0000 0000 00b0 9f77 4309 d00a
0ea7 1e58 d741 105a 2a16 0a14 0a0d 7472
6169 6e69 6e67 5f6c 6f73 7315 c193 403f


title: DataLoader crashes when shuffling

body:
I'm using DataLoader to read from a custom Dataset object based on numpy memmap.
As long as I read the data without shuffling everything works fine but, as I set shuffle=True, the runtime crash.

I tried implementing the shuffling mechanism in the Dataset class by using a permutation vector and setting shuffle=False in the DataLoader but the issue persists.
I also noticed that, when shuffling, the __getitem__() function of the Dataset object is called n times, where n is the batch_size.

Here's the Dataset code:

class CustomDataset(Dataset):
  num_pattern = 60112
  base_folder = 'dataset'

  def __init__(self, root):
    self.root = os.path.expanduser(root)

    self.output_ = np.memmap('{0}/output'.format(root), 'int64', 'r', shape=(60112, 62))
    self.out_len = np.memmap('{0}/output-lengths'.format(root), 'int32', 'r', shape=(60112))
    self.input_ = np.memmap('{0}/input'.format(root), 'float32', 'r', shape=(60112, 512, 1024))
    self.in_len = np.memmap('{0}/input-lengths'.format(root), 'int32', 'r', shape=(60112))

  def __len__(self):
    return self.num_pattern

  def __getitem__(self, index):
    return (self.in_len[index], torch.from_numpy(self.input_[index])), (self.out_len[index], torch.from_numpy(self.output_[index]))

if __name__ == '__main__':
  dataset = CustomDataset(root='/content/')
  data_loader = data.DataLoader(dataset, batch_size=32, shuffle=False, num_workers=1)
  for i, data in enumerate(data_loader, 0):
    # training

The error stack is the following:

RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _try_get_batch(self, timeout)
    510         try:
--> 511             data = self.data_queue.get(timeout=timeout)
    512             return (True, data)

9 frames
/usr/lib/python3.6/multiprocessing/queues.py in get(self, block, timeout)
    103                     timeout = deadline - time.monotonic()
--> 104                     if not self._poll(timeout):
    105                         raise Empty

/usr/lib/python3.6/multiprocessing/connection.py in poll(self, timeout)
    256         self._check_readable()
--> 257         return self._poll(timeout)
    258

/usr/lib/python3.6/multiprocessing/connection.py in _poll(self, timeout)
    413     def _poll(self, timeout):
--> 414         r = wait([self], timeout)
    415         return bool(r)

/usr/lib/python3.6/multiprocessing/connection.py in wait(object_list, timeout)
    910             while True:
--> 911                 ready = selector.select(timeout)
    912                 if ready:

/usr/lib/python3.6/selectors.py in select(self, timeout)
    375             try:
--> 376                 fd_event_list = self._poll.poll(timeout)
    377             except InterruptedError:

/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/signal_handling.py in handler(signum, frame)
     62         # Python can still get and update the process status successfully.
---> 63         _error_if_any_worker_fails()
     64         if previous_handler is not None:

RuntimeError: DataLoader worker (pid 3978) is killed by signal: Bus error.

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-8-b407a8532808> in <module>()
      5   data_loader = data.DataLoader(dataset, batch_size=4, shuffle=True, num_workers=1)
      6
----> 7   for i, data in enumerate(data_loader, 0):
      8     print(i)

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    574         while True:
    575             assert (not self.shutdown and self.batches_outstanding > 0)
--> 576             idx, batch = self._get_batch()
    577             self.batches_outstanding -= 1
    578             if idx != self.rcvd_idx:

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _get_batch(self)
    551         else:
    552             while True:
--> 553                 success, data = self._try_get_batch()
    554                 if success:
    555                     return data

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _try_get_batch(self, timeout)
    517             if not all(w.is_alive() for w in self.workers):
    518                 pids_str = ', '.join(str(w.pid) for w in self.workers if not w.is_alive())
--> 519                 raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
    520             if isinstance(e, queue.Empty):
    521                 return (False, None)

RuntimeError: DataLoader worker (pid(s) 3978) exited unexpectedly


title: How to extract value from multi dimensional tensor without losing backward info - PyTorch

body:
I am trying to extract some specific columns of a multi-dimensional tensor. I can take these values in the concept of multi-dimensional access; however I am losing the "grad_fn" option of the tensor when I create new values.

My actual multi-dimensional tensor, x, is:

tensor([[[-8.5780, -7.1091, -8.9204,  ..., -8.0616, -8.4115, -7.6345],
     [-7.9776, -7.3767, -8.2914,  ..., -7.9634, -9.1003, -7.5687],
     [-7.7192, -7.4307, -8.4294,  ..., -7.8605, -8.1345, -7.0781],
     ...,
     [-8.3652, -7.5910, -8.6671,  ..., -8.0487, -8.5826, -7.8624],
     [-8.1572, -7.1679, -8.8977,  ..., -8.1059, -8.1500, -7.7310],
     [-8.1821, -7.5455, -9.2328,  ..., -8.7435, -8.4430, -7.2503]]],
   grad_fn=<LogSoftmaxBackward>)

with shape (1,10,2000).

For example, I want to extract its specific 10 columns and finally create an array with a shape (1,10,10) or (10,10).

Currently, of course, I can extract these values into an array:

    for i in range(x.size(0)):

        values = np.array([x[i][1:, 1].detach().numpy()])

        idx_to_get = [1,5,7,25,37,44,720,11,25,46]

        for idx in idx_to_get:
            values = np.append(sentence_scores, np.array([np.array(x[0][1:,idx].detach().numpy())]), axis=0)

        values = np.delete(values, 0, axis=0)

        print(torch.from_numpy(values))

Running the code above gave me this output for the values:

tensor([[-7.5589, -6.7990, -7.2068, -7.4451, -7.6688, -7.2991, -7.1398, -7.4362,
     -7.4959, -8.0101, -7.5106, -8.0425, -7.6203, -7.7266, -7.9249, -7.6479,
     -7.6684],
    [-7.2831, -7.7666, -7.8302, -7.3651, -7.2184, -6.7932, -7.1968, -7.6590,
     -7.4033, -6.9504, -7.0767, -7.5366, -7.8364, -7.5935, -8.1235, -7.3222,
     -7.8096],
    [-7.5589, -6.7990, -7.2068, -7.4451, -7.6688, -7.2991, -7.1398, -7.4362,
     -7.4959, -8.0101, -7.5106, -8.0425, -7.6203, -7.7266, -7.9249, -7.6479,
     -7.6684],
    [-7.5650, -7.6627, -7.4230, -7.4726, -7.5621, -7.4489, -7.8344, -7.6130,
     -7.9440, -7.6158, -7.1895, -7.8070, -7.2306, -7.6364, -7.7390, -7.6832,
     -7.5931],
    [-7.5589, -6.7990, -7.2068, -7.4451, -7.6688, -7.2991, -7.1398, -7.4362,
     -7.4959, -8.0101, -7.5106, -8.0425, -7.6203, -7.7266, -7.9249, -7.6479,
     -7.6684],
    [-7.5589, -6.7990, -7.2068, -7.4451, -7.6688, -7.2991, -7.1398, -7.4362,
     -7.4959, -8.0101, -7.5106, -8.0425, -7.6203, -7.7266, -7.9249, -7.6479,
     -7.6684],
    [-7.2831, -7.7666, -7.8302, -7.3651, -7.2184, -6.7932, -7.1968, -7.6590,
     -7.4033, -6.9504, -7.0767, -7.5366, -7.8364, -7.5935, -8.1235, -7.3222,
     -7.8096],
    [-8.3559, -8.3751, -8.2082, -8.6825, -8.4860, -8.4156, -8.4683, -8.8760,
     -8.7354, -8.6155, -8.7544, -8.4527, -8.3690, -8.5795, -8.6023, -8.2267,
     -8.4736],
    [-7.4392, -7.4694, -7.4094, -7.5062, -7.7691, -7.9009, -7.7664, -7.1755,
     -8.0641, -7.6327, -7.6413, -7.9604, -7.9520, -7.8893, -7.8119, -7.8718,
     -8.0961],
    [-8.2182, -8.0280, -8.1398, -8.0258, -7.9951, -8.0664, -8.1976, -7.6182,
     -8.0356, -8.0293, -7.7228, -7.7847, -7.4966, -7.6925, -7.5268, -7.0476,
     -7.2920]])

But the values also should have grad_fn.
What should I do?

I know, using values.requires_grad_(True) worked, but I believe that with using that function, I loss the LogSoftmaxBackward on x.


title: Pytorch equivalent of tf.Variable

body:
I am trying to implement this code in pytorch:

self.scale_var = tf.Variable(
          0.1, name='scale_var',
          trainable=True,
          dtype=tf.float32,
          constraint=lambda x: tf.clip_by_value(x, 0, np.infty))

I want to have a scalar value that is trainable and would like to scale a constant with this value in the loss function. Is the below mentioned code  appropriate?

class pytorch_variable(nn.Module):
    def __init__(self):
        super(pytorch_variable,self).__init__()
        self.var = nn.Parameter(torch.tensor(0.1))

    def forward(self):
        return self.var

What is happening right now is that the gradients flow through this, but the trainable scalar value slowly reduces to zero, decreasing by 0.001 from initial value of 0.1 (till zero because I clip the data after loss.backward() call).


I tried to use tensorboard in torch.utils, but it says "module 'torch.utils' has no attribute 'tensorboard'".
My torch version is "1.6.0+cu101"
PS C:\Users\kelekelekle> python
Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 01:54:44) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.__version__)
1.6.0+cu101
>>> writer = torch.utils.tensorboard.SummaryWriter()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'torch.utils' has no attribute 'tensorboard'
>>>


title: Your session crashed after using all available RAM in Google Colab

body:
I am trying to get this to run in google colab https://github.com/oawiles/X2Face/blob/master/UnwrapMosaic/Face2Face_UnwrapMosaic.ipynb,

I was able to get it to run and display results once but since then lately I have been getting your session crashed after using all available RAM at this cell

    BASE_MODEL = '/scratch/shared/slow/ow/eccv/2018/release_models/' # Change to your path
state_dict = torch.load(BASE_MODEL + 'x2face_model.pth')

model = UnwrappedFaceWeightedAverage(output_num_channels=2, input_num_channels=3, inner_nc=128)
model.load_state_dict(state_dict['state_dict'])
model = model.cuda()

model = model.eval()

im not sure if it is a colab issue or not

here is the logs

warn("IPython.utils.traitlets has moved to a top-level traitlets package.")
/usr/local/lib/python2.7/dist-packages/IPython/utils/traitlets.py:5: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package.


title: Getting nan loss in Adam Model using PyTorch

body:
I am new to training neural nets. Please forgive me if this is a very stupid question or violates any of the unsaid rules of stack overflow. I started working on the titanic data set recently. I cleaned the data. I have a features tensor that I made by concatenating the normalized continuous data and the one hot tensors of the categorical data. I am passing this data into a simple linear model and I am getting nan loss for all epochs.

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from tqdm import tqdm
import pickle
import pathlib

path = pathlib.Path('./drive/My Drive/Kaggle/Titanic')

with open(path/'feature_tensor.pickle', 'rb') as f:
    features = pickle.load(f)

with open(path/'label_tensor.pickle', 'rb') as f:
    labels = pickle.load(f)

features = features.float()
labels = labels.float()

import math
valid_size = -1 * math.floor(0.2*len(features))

train_features = features[:valid_size]
valid_features = features[valid_size:]

train_labels = labels[:valid_size]
valid_labels = labels[valid_size:]

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.h_l1 = nn.Linear(18, 64)
        self.h_l2 = nn.Linear(64, 32)
        self.o_l = nn.Linear(32, 2)

    def forward(self, x):
        x = F.relu(self.h_l1(x))
        x = F.relu(self.h_l2(x))
        return self.o_l(x)

model = Model()
model.to('cuda')

optimizer = optim.Adam(model.parameters())
loss_fn = nn.MSELoss()

EPOCHS = 5
BATCH_SIZE = 20

for EPOCH in range(0, EPOCHS):
    for i in tqdm(range(0, len(features), BATCH_SIZE)):
        train_feature_batch = train_features[i:i+BATCH_SIZE,:].to('cuda')
        train_label_batch = train_labels[i:i+BATCH_SIZE,:].to('cuda')
        valid_feature_batch = valid_features[i:i+BATCH_SIZE,:].to('cuda')
        valid_label_batch = valid_labels[i:i+BATCH_SIZE,:].to('cuda')
        train_loss = loss_fn(model(train_feature_batch), train_label_batch)
        with torch.no_grad():
            valid_loss = loss_fn(model(valid_feature_batch), valid_label_batch)
        optimizer.zero_grad()
        train_loss.backward()
        optimizer.step()
    print(f"Epoch : {EPOCH}\tTrain Loss : {train_loss}\tValid_loss : {valid_loss}\n")

I am getting the following output:

100%|██████████| 45/45 [00:00<00:00, 511.50it/s]
100%|██████████| 45/45 [00:00<00:00, 604.10it/s]
100%|██████████| 45/45 [00:00<00:00, 586.21it/s]
  0%|          | 0/45 [00:00<?, ?it/s]Epoch : 0 Train Loss : nan    Valid_loss : nan

Epoch : 1   Train Loss : nan    Valid_loss : nan

Epoch : 2   Train Loss : nan    Valid_loss : nan

100%|██████████| 45/45 [00:00<00:00, 555.55it/s]
100%|██████████| 45/45 [00:00<00:00, 607.65it/s]Epoch : 3   Train Loss : nan    Valid_loss : nan

Epoch : 4   Train Loss : nan    Valid_loss : nan

Yes the output is scattered like this. Please help.


title: NumPy/PyTorch extract subsets of images

body:
In Numpy, given a stack of large images A of size(N,hl,wl), and coordinates x of size(N) and y of size(N) I want to get smaller images of size (N,16,16)

In a for loop it would look like this:

B=numpy.zeros((N,16,16))
for i in range(0,N):
  B[i,:,:]=A[i,y[i]:y[i]+16,x[i]:x[i]+16]

But can I do this just with indexing?
Bonus question: Will this indexing also work in pytorch? If not how can I implement this there?


title: Does tensorflow have the function similar to pytorch's "masked_fill_"

body:
I want to set INF value to matrix by mask matrix, just like pytorch code:

scores.data.masked_fill_(y_mask.data, -float('inf'))

I try to use tf.map_fn to implement that, but the performance is too slow. So does tensorflow have any efficient function to implement that?


title: Mask specific elements in a final layer in PyTorch

body:
I am now reproducing the following model which outputs an action and uses filter for filtering inappropriate candidates.
https://arxiv.org/abs/1702.03274

In this model, output is filtered after last softmax layer. Let's assume action_size==3. So the output after dense & asoftmax layer is like below.

output: [0.1, 0.7, 0.2]
filter: [0, 1, 1]
output*filter: [0, 0.7, 0.2]

But in pytorch, logsoftmax is preferred with NLLLoss. So my output is like below. This doesn't make sense.

output: [-5.4, -0.2, -4.9]
filter: [0, 1, 1]
output*filter: [0, -0.2, -4.9]

So pytoroch doesn't recommend vanilla Softmax. How should I apply mask to eliminate specific actions?
Or is there any categorical cross entropy loss functions with vanilla Softmax?

  This module doesn’t work directly with NLLLoss, which expects the Log to be computed between the Softmax and itself. Use Logsoftmax instead (it’s faster and has better numerical properties).
  http://pytorch.org/docs/master/nn.html#torch.nn.Softmax


title: Can't see graph using torch.utils.tensorboard

body:
I'm trying to get used to tensorboard, and I code my models using pytorch.
However when I try to see my model using the add_graph() function, I've got this:

With this as the test code:

import numpy as np
import torch
import torchvision.transforms as transforms
import torch.nn as nn
import torch.optim as optim
from torch.utils.tensorboard import SummaryWriter

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.linear = nn.Linear(2, 1)

    def forward(self, x):
        x = self.linear(x)
        return x

writer = SummaryWriter('runs_pytorch/test')
net = Net()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
writer.add_graph(net, torch.zeros([4, 2], dtype=torch.float))
writer.close()

On the other hand, if I try to see a graph using TensorFlow, everything seems fine:

with this as the test code this time:

import tensorflow as tf
tf.Variable(42, name='foo')
w = tf.summary.FileWriter('runs_tensorflow/test')
w.add_graph(tf.get_default_graph())
w.flush()
w.close()

In case you are wondering, I'm using this command to start tensorboard:

tensorboard --logdir runs_pytorch

Something I noticed is that when I use it on the directory allocated for my tensorflow test, I've got the usual message with the address, but if I do the same thing with  --logdir runs_pytorch I've got something more:

W1010 15:19:24.225109 15308 plugin_event_accumulator.py:294] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.
W1010 15:19:24.226075 15308 plugin_event_accumulator.py:322] Found more than one "run metadata" event with tag step1. Overwriting it with the newest event.

I'm on windows, I tried on different browsers (chrome, firefox...).
I have tensorflow 1.14.0, torch 1.2.0, Python 3.7.3

Thank you very much for your help, it's driving me crazy!


title: I am getting a ValueError: All bounding boxes should have positive height and width

body:
Hey I am getting the error
ValueError: All bounding boxes should have positive height and width. Found invaid box [264.0, 632.0, 264.0, 633.3333740234375] for target at index 2.
Epoch 1/1
Mini-batch: 1/1220 Loss: 0.1509
Mini-batch: 101/1220 Loss: 0.1201
Mini-batch: 201/1220 Loss: 0.1103
Mini-batch: 301/1220 Loss: 0.1098
Mini-batch: 401/1220 Loss: 0.1076
Mini-batch: 501/1220 Loss: 0.1056
Mini-batch: 601/1220 Loss: 0.1044
Mini-batch: 701/1220 Loss: 0.1035
ValueError Traceback (most recent call last)
in ()
13
14 # Calculate losses
—> 15 loss_dict = model(images, targets)
16 batch_loss = sum(loss for loss in loss_dict.values()) / len(loss_dict)
17

1 frames
/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/generalized_rcnn.py in forward(self, images, targets)
91 raise ValueError(“All bounding boxes should have positive height and width.”
92 " Found invaid box {} for target at index {}."
—> 93 .format(degen_bb, target_idx))
94
95 features = self.backbone(images.tensors)

ValueError: All bounding boxes should have positive height and width. Found invaid box [264.0, 632.0, 264.0, 633.3333740234375] for target at index 2.

I cant find a bounding box that has these coordinates in label csv file. Can anybody please help me out with this.
here is my dataset class
from torch.utils.data import Dataset, DataLoader

Inherit from pytorch Dataset for convenience
class DamageDataset(Dataset):
def __init__(self, dataframe):

    super().__init__()

    self.filename = dataframe['filename'].unique()

    self.df = dataframe

def __len__(self) -> int:

    return len(self.filename)

def __getitem__(self, index: int):

    filename = self.filename[index]

    image = read_image_from_train_folder(filename).astype(np.float32)

    # Scale to [0,1] range expected by the pre-trained model

    image /= 255.0

    # Convert the shape from [h,w,c] to [c,h,w] as expected by pytorch

    image = torch.from_numpy(image).permute(2,0,1)

    records = self.df[self.df['filename'] == filename]

    boxes = records[['xmin', 'ymin', 'xmax', 'ymax']].values

    classes= records['class'].values

    damage_labels=[]

    damage_dict={

        'D00': 1,

        'D10': 2,

        'D20': 3,

        'D40': 4,

    }

    for label in classes:

      damage_labels.append(damage_dict[label])

    boxes = torch.as_tensor(boxes, dtype=torch.float32)

    n_boxes = boxes.shape[0]

    # there is only one foreground class, WHEAT

    labels = torch.as_tensor(damage_labels, dtype=torch.float32)

    target = {}

    target['boxes'] = boxes

    target['labels'] = labels

    return image, target

and here is my train code:
num_epochs = 1

Prepare the model for training
model = model.to(device)

model.train()

for epoch in range(num_epochs):

print("Epoch %i/%i " % (epoch + 1, num_epochs) )

average_loss = 0

for batch_id, (images, targets) in enumerate(train_data_loader):

    # Prepare the batch data

    images, targets = move_batch_to_device(images, targets)

    # Calculate losses

    loss_dict = model(images, targets)

    batch_loss = sum(loss for loss in loss_dict.values()) / len(loss_dict)

    # Refresh accumulated optimiser state and minimise losses

    optimizer.zero_grad()

    batch_loss.backward()

can someone help me find out the index of this bounding box so that I can delete it, I have iterated through my dataframe using the code:
for idx, row in merge_labels.iterrows():
if(row[‘xmin’]==264 and row[‘ymin’]== 632 and row[‘xmax’]== 264 and row[‘ymax’]== 633.3333740234375 ):

print(idx)

but it doesnt print any index.
thank you


title: what is the meaning of a 'mini-batch' in deep learning?

body:
Im taking the fast-ai course and in lesson 2 SGD it says .

Mini-batch : 'a random bunch of points that you use to update your weights'

also it says that a gradient descent uses Mini-batches ...

What is a mini-batch, and what difference a mini-batch from a batch?

thanks for the help


title: How to train deeplabv3 on custom dataset on pytorch?

body:
import torch
model = torch.hub.load('pytorch/vision:v0.6.0', 'deeplabv3_resnet101', pretrained=True)
model.eval()
Pytorch provides pre-trained deeplabv3 on Pascal dataset, I want to train it on cityscapes. What's the efficient way to do it?


title: How to delete maxpooling layer from pytorch network

body:
I have "simple" Unet with resnet encoder on pytorch v3.1, which works pretty fine:

class UNetResNet(nn.Module):
    """PyTorch U-Net model using ResNet(34, 101 or 152) encoder.

    UNet: https://arxiv.org/abs/1505.04597
    ResNet: https://arxiv.org/abs/1512.03385

    Args:
            encoder_depth (int): Depth of a ResNet encoder (34, 101 or 152).
            num_classes (int): Number of output classes.
            num_filters (int, optional): Number of filters in the last layer of decoder. Defaults to 32.
            dropout_2d (float, optional): Probability factor of dropout layer before output layer. Defaults to 0.2.
            pretrained (bool, optional):
                False - no pre-trained weights are being used.
                True  - ResNet encoder is pre-trained on ImageNet.
                Defaults to False.
            is_deconv (bool, optional):
                False: bilinear interpolation is used in decoder.
                True: deconvolution is used in decoder.
                Defaults to False.

    """

    def __init__(self, encoder_depth, num_classes, num_filters=32, dropout_2d=0.2,
                 pretrained=False, is_deconv=False):
        super().__init__()
        self.num_classes = num_classes
        self.dropout_2d = dropout_2d

        if encoder_depth == 34:
            self.encoder = torchvision.models.resnet34(pretrained=pretrained)
            bottom_channel_nr = 512
        elif encoder_depth == 101:
            self.encoder = torchvision.models.resnet101(pretrained=pretrained)
            bottom_channel_nr = 2048
        elif encoder_depth == 152:
            self.encoder = torchvision.models.resnet152(pretrained=pretrained)
            bottom_channel_nr = 2048
        else:
            raise NotImplementedError('only 34, 101, 152 version of Resnet are implemented')

        self.pool = nn.MaxPool2d(2, 2)

        self.relu = nn.ReLU(inplace=True)

        self.conv1 = nn.Sequential(self.encoder.conv1,
                                   self.encoder.bn1,
                                   self.encoder.relu,
                                   self.pool)

        self.conv2 = self.encoder.layer1

        self.conv3 = self.encoder.layer2

        self.conv4 = self.encoder.layer3

        self.conv5 = self.encoder.layer4

        self.center = DecoderBlockV2(bottom_channel_nr, num_filters * 8 * 2, num_filters * 8, is_deconv)
        self.dec5 = DecoderBlockV2(bottom_channel_nr + num_filters * 8, num_filters * 8 * 2, num_filters * 8, is_deconv)
        self.dec4 = DecoderBlockV2(bottom_channel_nr // 2 + num_filters * 8, num_filters * 8 * 2, num_filters * 8,
                                   is_deconv)
        self.dec3 = DecoderBlockV2(bottom_channel_nr // 4 + num_filters * 8, num_filters * 4 * 2, num_filters * 2,
                                   is_deconv)
        self.dec2 = DecoderBlockV2(bottom_channel_nr // 8 + num_filters * 2, num_filters * 2 * 2, num_filters * 2 * 2,
                                   is_deconv)
        self.dec1 = DecoderBlockV2(num_filters * 2 * 2, num_filters * 2 * 2, num_filters, is_deconv)
        self.dec0 = ConvRelu(num_filters, num_filters)
        self.final = nn.Conv2d(num_filters, num_classes, kernel_size=1)

    def forward(self, x):
        conv1 = self.conv1(x)
        conv2 = self.conv2(conv1)
        conv3 = self.conv3(conv2)
        conv4 = self.conv4(conv3)
        conv5 = self.conv5(conv4)

        pool = self.pool(conv5) # that pool layer I would like to delete
        center = self.center(pool)

        dec5 = self.dec5(torch.cat([center, conv5], 1))

        dec4 = self.dec4(torch.cat([dec5, conv4], 1))
        dec3 = self.dec3(torch.cat([dec4, conv3], 1))
        dec2 = self.dec2(torch.cat([dec3, conv2], 1))
        dec1 = self.dec1(dec2)
        dec0 = self.dec0(dec1)

        return self.final(F.dropout2d(dec0, p=self.dropout_2d))

By deleting that pool layer from forward we come to next forward function:

def forward(self, x):
        conv1 = self.conv1(x)
        conv2 = self.conv2(conv1)
        conv3 = self.conv3(conv2)
        conv4 = self.conv4(conv3)
        conv5 = self.conv5(conv4)

        center = self.center(conv5) #now center connects with conv5

        dec5 = self.dec5(torch.cat([center, conv5], 1))

        dec4 = self.dec4(torch.cat([dec5, conv4], 1))
        dec3 = self.dec3(torch.cat([dec4, conv3], 1))
        dec2 = self.dec2(torch.cat([dec3, conv2], 1))
        dec1 = self.dec1(dec2)
        dec0 = self.dec0(dec1)

        return self.final(F.dropout2d(dec0, p=self.dropout_2d))

Applying only such changes and we will face such error:

~/Desktop/ml/salt/unet_models.py in forward(self, x)
    397         center = self.center(conv5)
    398
--> 399         dec5 = self.dec5(torch.cat([center, conv5], 1))
    400
    401         dec4 = self.dec4(torch.cat([dec5, conv4], 1))

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 4 and 8 in dimension 2 at /pytorch/torch/lib/THC/generic/THCTensorMath.cu:111

So we have to somehow change dec4, but how?

More info:

    class DecoderBlockV2(nn.Module):
        def __init__(self, in_channels, middle_channels, out_channels, is_deconv=True):
            super(DecoderBlockV2, self).__init__()
            self.in_channels = in_channels

            if is_deconv:
                """
                    Paramaters for Deconvolution were chosen to avoid artifacts, following
                    link https://distill.pub/2016/deconv-checkerboard/
                """

                self.block = nn.Sequential(
                    ConvRelu(in_channels, middle_channels),
                    nn.ConvTranspose2d(middle_channels, out_channels, kernel_size=4, stride=2,
                                       padding=1),
            nn.BatchNorm2d(out_channels),
                    nn.ReLU(inplace=True)
                )
            else:
                self.block = nn.Sequential(
                    nn.Upsample(scale_factor=2, mode='bilinear'),
                    ConvRelu(in_channels, middle_channels),
                    ConvRelu(middle_channels, out_channels),
                )

        def forward(self, x):
            return self.block(x)

class ConvRelu(nn.Module):
    def __init__(self, in_, out):
        super().__init__()
        self.conv = conv3x3(in_, out)
        self.activation = nn.ReLU(inplace=True)

    def forward(self, x):
        x = self.conv(x)
        x = self.activation(x)
        return x


title: How to to drop a specific labeled pixels in semantic segmentation

body:
I am new to semantic segmentation. I used the FCN to train my dataset. In the data set there are some pixels for the unknown class. I would like to exclude this class from my loss. So I defined a weight based on the class distribution of whole dataset and set the weight for the unknown class to zero as following. But I am still getting prediction for this class. Do you have any idea how to properly exclude one specific class?

loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits
                      (logits=logits, labels=tf.squeeze(annotation,
                                                        squeeze_dims=[3]),name="entropy"))
weighted_losses = (loss * weights)
train_op = optimizer.minimize(weighted_losses,
                                  var_list=tf.trainable_variables(),
                                  global_step=tf.train.get_global_step())

I do not know pytorch, but I heard that there is some thing for this purpose "ignore_index" in loss function and you can ignore a specific class. If this is right approach to my problem, do you know if there is some thing equivalent in tensorflow?


title: How to change (assign) new value in FloatTensor, Pytorch?

body:
I am changing/assigning the value on the array(torch.cuda.floatTensor). I tried some way but it does not work. Please help me!

#1

#dis is [torch.cuda.FloatTensor of size 3185x1 (GPU 0)]

s = dis.size(0) #3185
for i in range (0,s,1):
    if (dis[i,0] < 0):
        dis[i,0]== 0
#There is no error but It does not work.

#2

#dis is [torch.cuda.FloatTensor of size 3185x1 (GPU 0)]

s = dis.size(0)
a = torch.zeros(s, 1).cuda()
idx = (dis > a)
dis[idx] = a[idx]

AssertionError: can't compare Variable and tensor

#3

#dis is [torch.cuda.FloatTensor of size 3185x1 (GPU 0)]

s = dis.size(0)
a = torch.zeros(s, 1).cuda()
for i in range (0,s,1):
    if (dis[i,0] < a[i, 0]):
        dis[i,0]==a[i, 0]

#RuntimeError: bool value of Variable objects containing non-empty      torch.cuda.ByteTensor is ambiguous


title: Defining named parameters for a customized NN module in Pytorch

body:
This question is about how to appropriately define the parameters of a customized layer in Pytorch.  I am wondering how one can make the parameter to end up being named parameter?
class My_layer(torch.nn.Module):
    def __init__(self):
        self.mu = torch.nn.Parameter(torch.tensor([[0.0],[1.0]]))

So that when I print the parameters as below, the p.name is not empty.
my_layer = My_Layer()
for p in my_layer.parameters():
     print(p.name, p.data, p.requires_grad)


title: Best way to handle OOV words when using pretrained embeddings in PyTorch

body: I am using word2vec pretrained embedding in PyTorch (following code here). However, it does not seem to handle unseen words. Is there any good way to solve it?


I am trying to use Alexnet over the CIFAR-10 dataset. I have resized my image to 224x224 which I'm guessing is the issue. Nevertheless, I get the following error:

<ipython-input-11-34884668038d> in forward(self, x)
     37     def forward(self, x):
     38         x = self.features(x)
---> 39         x = x.view(x.size(0), 256 * 2 * 2)
     40         x = self.classifier(x)
     41         return x

RuntimeError: shape '[1, 1024]' is invalid for input of size 50176

My Alexnet model code is as follows:

NUM_CLASSES = 10

class AlexNet(nn.Module):
    def __init__(self, num_classes=NUM_CLASSES):
        super(AlexNet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, stride=2, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2),
            nn.Conv2d(64, 192, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2),
            nn.Conv2d(192, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2),
        )
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(256 * 2 * 2, 4096),
            nn.ReLU(inplace=True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),
            nn.Linear(4096, num_classes),
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), 256 * 2 * 2)
        x = self.classifier(x)
        return x

Any help would be appreciated :)


title: What is the idea behind using nn.Identity for residual learning?

body:
So, I've read about half the original ResNet paper, and am trying to figure out how to make my version for tabular data.
I've read a few blog posts on how it works in PyTorch, and I see heavy use of nn.Identity(). Now, the paper also frequently uses the term identity mapping. However, it just refers to adding the input for a stack of layers the output of that same stack in an element-wise fashion. If the in and out dimensions are different, then the paper talks about padding the input with zeros or using a matrix W_s to project the input to a different dimension.
Here is an abstraction of a residual block I found in a blog post:

class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels, activation='relu'):
        super().__init__()
        self.in_channels, self.out_channels, self.activation = in_channels, out_channels, activation
        self.blocks = nn.Identity()
        self.shortcut = nn.Identity()

    def forward(self, x):
        residual = x
        if self.should_apply_shortcut: residual = self.shortcut(x)
        x = self.blocks(x)
        x += residual
        return x

    @property
    def should_apply_shortcut(self):
        return self.in_channels != self.out_channels

block1 = ResidualBlock(4, 4)

And my own application to a dummy tensor:
x = tensor([1, 1, 2, 2])
block1 = ResidualBlock(4, 4)
block2 = ResidualBlock(4, 6)
x = block1(x)
print(x)
x = block2(x)
print(x)

>>> tensor([2, 2, 4, 4])
>>> tensor([4, 4, 8, 8])

So at the end of it, x = nn.Identity(x) and I'm not sure the point of its use except to mimic math lingo found in the original paper. I'm sure that's not the case though, and that it has some hidden use that I'm just not seeing yet. What could it be?
EDIT Here is another example of implementing residual learning, this time in Keras. It does just what I suggested above and just keeps a copy of the input for adding to the output:
def residual_block(x: Tensor, downsample: bool, filters: int,                                        kernel_size: int = 3) -> Tensor:
    y = Conv2D(kernel_size=kernel_size,
               strides= (1 if not downsample else 2),
               filters=filters,
               padding="same")(x)
    y = relu_bn(y)
    y = Conv2D(kernel_size=kernel_size,
               strides=1,
               filters=filters,
               padding="same")(y)

    if downsample:
        x = Conv2D(kernel_size=1,
                   strides=2,
                   filters=filters,
                   padding="same")(x)
    out = Add()([x, y])
    out = relu_bn(out)
    return out


Shall we lower case input data for (pre) training a BERT uncased model using huggingface? I looked into this response from Thomas Wolf (https://github.com/huggingface/transformers/issues/92#issuecomment-444677920) but not entirely sure if he meant that.

What happens if we lowercase the text ?


title: Pytorch - (Categorical) Cross Entropy Loss using one hot encoding and softmax

body:
I'm looking for a cross entropy loss function in Pytorch that is like the CategoricalCrossEntropyLoss in Tensorflow.
My labels are one hot encoded and the predictions are the outputs of a softmax layer. For example (every sample belongs to one class):
targets = [0, 0, 1]
predictions = [0.1, 0.2, 0.7]

I want to compute the (categorical) cross entropy on the softmax values and do not take the max values of the predictions as a label and then calculate the cross entropy. Unfortunately, I did not find an appropriate solution since Pytorch's CrossEntropyLoss is not what I want and its BCELoss is also not exactly what I need (isn't it?).
Does anyone know which loss function to use in Pytorch or how to deal with it?
Many thanks in advance!


title: How does Normalizing in Pytorch operate?

body:
This must be a really basic question, but even after trying out stuff I just don't understand how
the normalization in Pytorch works.
I want to set the mean to 0 and the standard deviation to 1 across all columns in a tensor x of size (2,2,3).
A simple example:
norm = transforms.Normalize((0,0),(1,1))
x = torch.tensor([[[1.0,2,3],[4,5,6]],[[7,8,9],[10,11,12]]])
out = norm(x)
print(x)
print(out)

gives
tensor([[[ 1.,  2.,  3.],
         [ 4.,  5.,  6.]],

        [[ 7.,  8.,  9.],
         [10., 11., 12.]]])

tensor([[[ 1.,  2.,  3.],
         [ 4.,  5.,  6.]],

        [[ 7.,  8.,  9.],
         [10., 11., 12.]]])

so nothing has changed from normalization. Why?


title: A better way to make pytorch code agnostic to running on a CPU or GPU?

body:
The Migration guide recommends the following to make code CPU/GPU agnostic:

> # at beginning of the script
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
...
# then whenever you get a new Tensor or Module
# this won't copy if they are already on the desired device
input = data.to(device)
model = MyModule(...).to(device)

I did this and ran my code on a CPU-only device, but my model crashed when fed an input array, as saying it was expecting a CPU tensor not a GPU one.  Somehow my model was automatically converting the a CPU-input array to a GPU array.  Finally I traced it down to this command in my code:

model = torch.nn.DataParallel(model).to(device)

Even though I convert the model to 'cpu', the nn.DataParallel overrides this.  The best solution I came up with was a conditional:

if device.type=='cpu':
    model = model.to(device)
else:
    model = torch.nn.DataParallel(model).to(device)

This does not seem elegant.  Is there a better way?


title: Why would we use to() method in pytorch?

body: I've seen this method multiple times. What are the purposes and advantages of doing this?


title: Pytorch minibatching keeps model from training

body:
I am trying to classify sequences by a binary feature. I have a dataset of sequence/label pairs and am using a simple one-layer LSTM to classify each sequence. Before I implemented minibatching, I was getting reasonable accuracy on a test set (80%), and the training loss would go from 0.6 to 0.3 (averaged).

I implemented minibatching, using parts of this tutorial: https://pytorch.org/tutorials/beginner/chatbot_tutorial.html

However, now my model won’t do better than 70-72% (70% of the data has one label) with batch size set to 1 and all other parameters exactly the same. Additionally, the loss starts out at 0.0106 and quickly gets really really small, with no significant change in results. I feel like the results between no batching and batching with size 1 should be the same, so I probably have a bug, but for the life of me I can’t find it. My code is below.

Training code (one epoch):

for i in t:
    model.zero_grad()

    # prep inputs
    last = i+self.params['batch_size']
    last = last if last < len(train_data) else len(train_data)
    batch_in, lengths, batch_targets = self.batch2TrainData(train_data[shuffled][i:last], word_to_ix, label_to_ix)

    iters += 1

    # forward pass.
    tag_scores = model(batch_in, lengths)

    # compute loss, then do backward pass, then update gradients
    loss = loss_function(tag_scores, batch_targets)
    loss.backward()

    # Clip gradients: gradients are modified in place
    nn.utils.clip_grad_norm_(model.parameters(), 50.0)

    optimizer.step()

Functions:

def prep_sequence(self, seq, to_ix):
    idxs = [to_ix[w] for w in seq]
    return torch.tensor(idxs, dtype=torch.long)

# transposes batch_in
def zeroPadding(self, l, fillvalue=0):
    return list(itertools.zip_longest(*l, fillvalue=fillvalue))

# Returns padded input sequence tensor and lengths
def inputVar(self, batch_in, word_to_ix):
    idx_batch = [self.prep_sequence(seq, word_to_ix) for seq in batch_in]
    lengths = torch.tensor([len(idxs) for idxs in idx_batch])
    padList = self.zeroPadding(idx_batch)
    padVar = torch.LongTensor(padList)
    return padVar, lengths

# Returns all items for a given batch of pairs
def batch2TrainData(self, batch, word_to_ix, label_to_ix):
    # sort by dec length
    batch = batch[np.argsort([len(x['turn']) for x in batch])[::-1]]
    input_batch, output_batch = [], []
    for pair in batch:
        input_batch.append(pair['turn'])
        output_batch.append(pair['label'])
    inp, lengths = self.inputVar(input_batch, word_to_ix)
    output = self.prep_sequence(output_batch, label_to_ix)
    return inp, lengths, output

Model:

class LSTMClassifier(nn.Module):

    def __init__(self, params, vocab_size, tagset_size, weights_matrix=None):
        super(LSTMClassifier, self).__init__()
        self.hidden_dim = params['hidden_dim']

        if weights_matrix is not None:
            self.word_embeddings = nn.Embedding.from_pretrained(weights_matrix)
        else:
            self.word_embeddings = nn.Embedding(vocab_size, params['embedding_dim'])

        self.lstm = nn.LSTM(params['embedding_dim'], self.hidden_dim, bidirectional=False)

        # The linear layer that maps from hidden state space to tag space
        self.hidden2tag = nn.Linear(self.hidden_dim, tagset_size)

    def forward(self, batch_in, lengths):
        embeds = self.word_embeddings(batch_in)
        packed = nn.utils.rnn.pack_padded_sequence(embeds, lengths)
        lstm_out, _ = self.lstm(packed)
        outputs, _ = nn.utils.rnn.pad_packed_sequence(lstm_out)
        tag_space = self.hidden2tag(outputs)
        tag_scores = F.log_softmax(tag_space, dim=0)
        return tag_scores[-1]


I got this error when running training a deep learning model and although looking at many solutions over the Internet, they did not help me. The log is as follows:

Traceback (most recent call last):
  File "main.py", line 208, in <module>
    main()
  File "main.py", line 100, in main
    model = nn.DataParallel(model).cuda()
  File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 105, in __init__
    self.module.cuda(device_ids[0])
  File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/modules/module.py", line 249, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/modules/rnn.py", line 112, in _apply
    self.flatten_parameters()
  File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/modules/rnn.py", line 78, in flatten_parameters
    if not any_param.is_cuda or not torch.backends.cudnn.is_acceptable(any_param):
  File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/backends/cudnn/__init__.py", line 87, in is_acceptable
    if _libcudnn() is None:
  File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/backends/cudnn/__init__.py", line 58, in _libcudnn
    'but linked against {}'.format(compile_version, __cudnn_version))
RuntimeError: cuDNN version mismatch: PyTorch was compiled against 7102 but linked against 7604


title: Pytorch 1.7.0 | DataLoader Error - TypeError: 'module' object is not callable

body:
This is my code, I am using pycharm!
Imports
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import torch.utils.data as DataLoader
import torchvision.datasets as Datasets
import torchvision.transforms as transforms

create Fully Connected Network
class NN(nn.Module):
    def __init__(self, input_size, num_classes): #(28x28 = 784)
        super(NN, self).__init__()
        self.fc1 = nn.Linear(input_size, 50)
        self.fc2 = nn.Linear(50, input_size) #hidden layer

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

#Set Device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

Hyperparameters
input_size = 784
num_classes = 10
learning_rate = 0.001
batch_size = 2
num_epochs = 1

Load Data
train_dataset = Datasets.MNIST(root='dataset/', train=True, transform=transforms.ToTensor(), download=True)
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_dataset = Datasets.MNIST(root='dataset/', train=False, transform=transforms.ToTensor(), download=True)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=True)

Initialize network
model = NN(input_size=input_size, num_classes=num_classes).to(device)

Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

Train network
for epoch in range(num_epochs):
    for batch_idx, (data, targets) in enumerate(train_loader):
        data = data.to(device=device)
        targets = targets.to(device=device)

        print(data.shape)

I am getting the error on this line
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
Error is
in <module>
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
TypeError: 'module' object is not callable


title: How to get two scalars on same chart with tensorboardX?

body:
The docs seem to indicate that add_custom_scalars_multilinechart does it but it is not working.

Have something like this:

from tensorboardX import SummaryWriter
writer = SummaryWriter(comment='test')

writer.add_custom_scalars_multilinechart(['loss/train', 'loss/test'], title='losses')

for blahblah:
    ...
    writer.add_scalar('loss/train', loss.item(), epoch)
    writer.add_scalar('loss/test', loss_test.item(), epoch)


title: Extract elements from .npy file, convert them to PyTorch tensors

body:
I read a .npy file that contains just the labels for images. The labels are stored in dictionary format. I need to convert this to an array of Tensors. But I'm unable to extract elements one of the other from the object the file returns, which is numpy.ndarray type.

import numpy as np
data = np.load('/content/drive/My Drive/targets.npy')
print(data.item())

{0: array(5), 1: array(0), 2: array(4), 3: array(1), 4: array(9), 5: array(2), 6: array(1), 7: array(3)}

print(data[()].values())

dict_values([array(5), array(0), array(4), array(1), array(9), array(2), array(1), array(3)])

I would like to create an array of tensors instead.

Thanks in advance.


title: Consecutive calls to stack throws RuntimeError: stack expects each tensor to be equal size

body:
I think both torch.cat and torch.stack cannot fully satisfy my requirement.
Initially, I define an empty tensor. Then I want to append a 1d-tensor to it for multiple times.
x = torch.tensor([]).type(torch.DoubleTensor)
y = torch.tensor([ 0.3981,  0.6952, -1.2320]).type(torch.DoubleTensor)
x = torch.stack([x,y])

This will throw an error:

RuntimeError: stack expects each tensor to be equal size, but got [0] at entry 0 and [3] at entry 1

So I have to initialise x as torch.tensor([0,0,0]) (but can this be avoided?)
x = torch.tensor([0,0,0]).type(torch.DoubleTensor)
y = torch.tensor([ 0.3981,  0.6952, -1.2320]).type(torch.DoubleTensor)
x = torch.stack([x,y])  # this is okay
x = torch.stack([x,y])  # <--- this got error again

But when I run x = torch.stack([x,y]) the second time, I got this error:

RuntimeError: stack expects each tensor to be equal size, but got [2, 3] at entry 0 and [3] at entry 1

What I want to achieve is being able to append a 1d-tensor multiple times (the added 1d-tensor is different at each time, here I use the same one for simplicity)**:
tensor([[ 0.3981,  0.6952, -1.2320],
        [ 0.3981,  0.6952, -1.2320],
        [ 0.3981,  0.6952, -1.2320],
        [ 0.3981,  0.6952, -1.2320],
        ...
        [ 0.3981,  0.6952, -1.2320]], dtype=torch.float64)

How to achieve this?


title: Speeding up Beam Search with seq2seq

body:
I have a fully working seq2seq attention model with beam search and it does give improved results. But it takes > 1min for inferencing (batch-size 1024) with k=5 (k is my hypotheses) because none of it is parallelised. Everything happens 1 sample at a time.

Task (simplified)

Goal is sentence translation, 15 words Lang A to 15 words Lang B.

Encoder is a RNN  that takes in 15 word sentence and encodes a representation of it, gives out a [timestep, 512] matrix along with final hidden state.
Decoder is another RNN, takes encoder hidden state as initial state, uses [timestep, 512] matrix for attention and outputs translated words[batches] one timestep at a time. Naturally, there is some form of parallelization till this point.
During inference stage, beam search is used. At each timestep of the decoder, rather than taking the predicted word with highest prob, I take k best words. And provide k words as input to the next timestep so that it can predict the next word in the sentence (rest of the algorithm is given below). Algorithm makes decoding less greedy anticipating results with higher total probability in succeeding timesteps.

for each element in the test-set
    calculate initial k (decoder-encoder step)
    for range(timesteps-1)
        for each prev k
            get hidden state
            obtain its best k
            save hidden state
        find new k from k*k possible ones
        ##update hypotheses based on new found k
        for element in k
            copy hidden state
            change hypotheses if necessary
            append new k to hypotheses

There are 6 tensors and 2 lists to keep track and handle state changes. Is there any room for speedup or parallelisation here? perhaps each k can go through enncode-decode simultaneously? Any help is much appreciated.


title: Matrix multiplication (element-wise) from numpy to Pytorch

body:
I got two numpy arrays (image and and environment map),

MatA
MatB

Both with shapes (256, 512, 3)

When I did the multiplication (element-wise) with numpy:

prod = np.multiply(MatA,MatB)

I got the wanted result (visualize via Pillow when turning back to Image)

But when I did it using pytorch, I got a really strange result(not even close to the aforementioned).

I did it with the following code:

MatATensor = transforms.ToTensor()(MatA)
MatBTensor = transforms.ToTensor()(MatB)

prodTensor = MatATensor * MatBTensor

For some reasons, the shape for both MatATensor and MatBtensor is

torch.Size([3, 256, 512])

Same for the prodTensor too.
When I tried to reshape to (256,512,3), I got an error.

Is there a way to get the same result?

I am new to pytorch, so any help would be appreciated.


title: Optimizing docker image size

body:
I am building a docker image for my python app using the base image python:3.5-slim-buster.

I am running the below command inside the Dockerfile:

RUN pip install --no-cache-dir -r requirements.txt which has torch library in the requirements file.

After building the image it is having a size of 2.29 GB. But if I build the image without torch inside the requirements file, it has only ~900 MB.

when I manually run the image and check inside the container:

The torch (/usr/local/lib/python3.5/site-packages/torch) directory is 1.3GB.

So even if I do a multistage build and try to copy the contents from
/usr/local/lib/python3.5/site-packages to a new image, I suppose it is not going to help me.

Is there any other standard optimization practice which can help me to reduce the image size?


title: PyTorch RuntimeError Invalid argument 2 of size

body:
I am experimenting with a neural network (PyTorch) and I get this error.

  RuntimeError: invalid argument 2: size '[32 x 9216]' is invalid for input with 8192 elements at /pytorch/aten/src/TH/THStorage.cpp:84

My task is about image classification with AlexNet and I have backtracked the error to be the size of the images supplied to the neural network. My question is, given the network architecture with its parameters, how does one determine the correct image size required by the network?

As per my code below, I first transform the training images before feeding into the neural network. But I noticed the neural network can only accept the size of 224 and or else it gives the error above. For instance, my instinct was to apply transforms.RandomResizedCrop of size 64 but apparently this is wrong. Is there a formula to determine the size required?

Code

# transformation to be done on images
transform_train = transforms.Compose([
    transforms.RandomResizedCrop(64),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

class AlexNet(nn.Module):

    def __init__(self, num_classes=1000):
        super(AlexNet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(64, 192, kernel_size=5, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(192, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(256 * 6 * 6, 4096),
            nn.ReLU(inplace=True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),
            nn.Linear(4096, num_classes),
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), 256 * 6 * 6)
        x = self.classifier(x)
        return x


title: An error occured while installing flair and pytorch with pipenv in windows with Pycharm

body:
I am having problems installing the python packages flair and pytorch via pipenv and have not been able to resolve this issue yet. Since I am trying to version my python git repository with Pipfile + Pipfile.lock instead of requirements.txt this is currently not possible:
pipenv install flair
 ERROR: Could not find a version that satisfies the requirement torch>=1.1.0 (from flair->-r c:\users\user.name\appdata\local\temp\pipenv-plyx3uwp-requirements\pipenv-xh_afa_r-requirement.txt (line 1)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2) ERROR: No matching distribution found for torch>=1.1.0 (from flair->-r c:\users\user.name\appdata\local\temp\pipenv-plyx3uwp-requirements\pipenv-xh_afa_r-requirement.txt (line 1)) Installation Failed
I tried these variants of installing torchvision:
pipenv install torchvision
to install torchvision which should pick up the latest torch version
pipenv install torch==1.3
to install torch
pipenv install https://download.pytorch.org/whl/cu92/torch-0.4.1-cp37-cp37m-win_amd64.whl
alternative way to install torch (here is more binaries:  https://pytorch.org/get-started/previous-versions/#windows-binaries)
pipenv install git+https://github.com/pytorch/vision#egg=torchvision
Another alternative way,
Error text: Collecting torchvision
  Downloading torchvision-0.5.0-cp37-cp37m-win_amd64.whl (1.2 MB)
Collecting numpy
  Using cached numpy-1.18.5-cp37-cp37m-win_amd64.whl (12.7 MB)
Collecting pillow>=4.1.1
  Downloading Pillow-7.1.2-cp37-cp37m-win_amd64.whl (2.0 MB)
Collecting six
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)

pipenv install torchvision
users\user.name\appdata\local\temp\pipenv-hf2be0xq-requirements\pipenv-57akhz4j-requirement.txt (line
1)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.4.0 (from torchvision->-r c:\users\user.name\appdat
a\local\temp\pipenv-hf2be0xq-requirements\pipenv-57akhz4j-requirement.txt (line 1))
Installation Failed

The only way it was possible to install torchvision was without its dependent packages:
 pipenv run pip install --no-deps torchvision
But this did not resolve the problem of installing flair via pipenv since the dependencies are needed.


title: Why is my DataLoader so much slower than a for loop?

body:
I am writing a neural-net based classifier for the MNIST dataset. I first tried loading the data manually using loops and indexes for the epochs and batches. In a tutorial I saw someone using the torch.utils.data.DataLoader for this exact task, so i changed my code to use a DataLoader instead. This resulted in major differences in the duration of the learning process.

I've tried to troubleshoot this issue by trying to narrow it down using benchmarks. I always benchmarked on both CPU(i7 8700k) and GPU(1080ti) and the data is stored on my ssd(970 evo).

I first tried to compare Batch Gradient Descent with and without DataLoader and then Mini-Batch Gradient Descent with and without DataLoader. The results were rather confusing to me.

|                 | BGD         | BGD with DL | MB-GD       | MB-GD with DL |
|-----------------|-------------|-------------|-------------|---------------|
| Time on CPU     | 00:00:56.70 | 00:05:59.31 | 00:01:31.29 | 00:07:46.56   |
| Accuracy on CPU | 82.47       | 33.44       | 94.84       | 87.67         |
| Time on GPU     | 00:00:15.89 | 00:05:41.79 | 00:00:17.48 | 00:05:37.33   |
| Accuracy on GPU | 82.3        | 30.66       | 94.88       | 87.74         |
| Batch Size      | M           | M           | 500         | 500           |
| Epoch           | 100         | 100         | 100         | 100           |

This is the code using DataLoader, stripped down to essentials.

num_epoch = 100
train_loader = DataLoader(batch_size=500, shuffle=False, dataset=dataset_train)

for epoch in range(num_epoch):
    for i, (images, labels) in enumerate(train_loader):
        images = images.view(-1, 28 * 28)
        optimizer.zero_grad()
        outputs = model(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

vs the code using the loop

num_epoch = 100
batch_size = 500
num_batch = int(len(dataset_train) / batch_size)

for epoch in range(num_epoch):
    for batch_idx in range(num_batch):
        images = dataset_train.data[batch_idx*batch_size:(batch_idx+1)*batch_size].view(-1, 28 * 28)
        labels = dataset_train.targets[batch_idx*batch_size:(batch_idx+1)*batch_size]
        optimizer.zero_grad()
        outputs = model(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

I would expect the DataLoader to atleast perform somewhere close to the loop in terms of time and performance, but not 10 times slower. I am also confused why the DataLoader affects the model accuracy.

Am I using the DataLoader wrong, or is this just the wrong use case for it and a loop is better suited for what I am doing ?

EDIT: here are two fiddles containing the full code of the loop and the dataloader variant

EDIT: I believe I might have figured out how to fix my main problem, the performance difference between dataloader and loop. By setting the num_workers parameter of the loader to 8, i managed to drive down the time for mini-batch with DL on GPU to around 1 minute. While this is definitely better than 5 minutes, it's still bad, considering that minibatch with DL on GPU is on par with the performance of minibatch with loop on CPU.


title: Target size different to input size - LSTM

body:
I am relatively new to Pytorch and have been training an LSTM model. Any feedback on code in general would be appreciated.

When I train the model I receive the following warning

  UserWarning: Using a target size (torch.Size([4050, 1, 1])) that is
  different to the input size (torch.Size([1])). This will likely lead
  to incorrect results due to broadcasting. Please ensure they have the
  same size.

Can anyone help me with this problem?

My goal is to train a model on a time series containing multiple features. Please see the code below:

LSTM:

import torch
import torch.nn as nn
import pandas as pd
import numpy as np
from Tools.data_loader import CreateDataset

class LSTMModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim):
        super(LSTMModel, self).__init__()
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        self.output_dim = output_dim
        self.num_layers = num_layers
        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers)
        self.forecast = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):

        batch_size = 1
        h0 = torch.zeros(self.num_layers, batch_size, self.hidden_dim).requires_grad_()
        c0 = torch.zeros(self.num_layers, batch_size, self.hidden_dim).requires_grad_()

        out, (hn,cn) = self.lstm(x, (h0.detach(), c0.detach()))

        y_pred = self.forecast(out[-1].view(batch_size,-1))
        return y_pred.view(-1)

dataset = CreateDataset('Data/Bloomberg_PV_weather.csv', 'Datetime', 0.8, 'Power') #PLACEHOLDER
input_dim = dataset.num_features
hidden_dim = 50
num_layers = 1
output_dim = 1
num_epochs = 50

X_train = dataset.X_train
y_train = dataset.y_train
X_test = dataset.X_test
y_test = dataset.y_test

model = LSTMModel(input_dim, hidden_dim, num_layers, output_dim)
criterion = nn.MSELoss()
optimizer=torch.optim.Adam(model.parameters())

hist = np.zeros(num_epochs)

for epoch in range(num_epochs):
    optimizer.zero_grad()
    output=model(X_train)
    loss = criterion(output,y_train)
    if epoch % 100 == 0:
        print('Epoch ', epoch, 'Loss: ',loss.item())
    hist[epoch] = loss.item()
    loss.backward()
    optimizer.step()

Data Loader:

'''
Loads data and creates train and test sets
'''
import torch
import math
import pandas as pd

class CreateDataset():
    def __init__(self, file, datetime_col, train_proportion, target):
        self.file = file
        self.datetime_col = datetime_col
        self.train_proportion = train_proportion
        self.target = target
        self.data = self.load_data()
        self.train_set, self.test_set = self.split_data()
        self.X_train, self.y_train,self.X_test, self.y_test, self.num_features = self.reshape_data()

    def load_data(self):
        '''
        Reads in data
        '''
        data = pd.read_csv(self.file, header=0)
        data.drop(columns=self.datetime_col, inplace=True)
        return data

    def split_data(self):
        '''
        Creates test and train sets
        '''
        train_length = math.ceil(len(self.data)*self.train_proportion)
        train_set = self.data[0:train_length]
        test_set = self.data[train_length:]

        return train_set, test_set

    def reshape_data(self):
        '''
        Splits datasets into X and y sets and reshapes into 3D tensor
        '''
        num_features = (len(self.test_set.columns)-1)
        y_train = torch.tensor(self.train_set[self.target].values).float()
        X_train = torch.tensor(self.train_set.drop(columns=self.target).values).float()
        y_test = torch.tensor(self.test_set[self.target].values).float()
        X_test = torch.tensor(self.test_set.drop(columns=self.target).values).float()

        X_train = X_train.view(-1,1,num_features)
        y_train = y_train.view(-1,1,1)
        X_test = X_test.view(-1,1,num_features)
        y_test = y_test.view(-1,1,1)

        return X_train, y_train, X_test, y_test, num_features


title: How is data augmentation done in each epoch?

body:
I'm new to PyTorch and want to apply data augmentation to the datasets on each epoch. I
train_transform = Compose([
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.RandomCrop(32, padding=4),
    transforms.ToTensor(),
    transforms.Normalize([0, 0, 0], [1, 1, 1])
])

test_transform = Compose([
    transforms.ToTensor(),
    transforms.Normalize([0, 0, 0], [1, 1, 1])
])

cifar10_train = CIFAR10(root = "/data", train=True, download = True, transform=train_transform)
train_loader = torch.utils.data.DataLoader(cifar10_train, batch_size=128, shuffle=True)

cifar10_test = CIFAR10(root = "/data", train=False, download = True, transform=test_transform)
test_loader = torch.utils.data.DataLoader(cifar10_test, batch_size=128, shuffle=True)

I got the code from an online tutorial. So from what I understand train_transform and test_transform is the augmentation code while cifar10_train and cifar10_test are where data is loaded and augmentation is done at the same time. Does this mean data augmentation is only done once before training? What if I want to do data augmentation for each epoch.


title: How to calculate kernel dimensions from original image dimensions?

body:
https://github.com/kuangliu/pytorch-cifar/blob/master/models/resnet.py

From reading https://www.cs.toronto.edu/~kriz/cifar.html the cifar dataset consists of images each with 32x32 dimension.

My understanding of code :

self.conv1 = nn.Conv2d(3, 6, 5)
self.conv2 = nn.Conv2d(6, 16, 5)
self.fc1   = nn.Linear(16*5*5, 120)

Is :

self.conv1 = nn.Conv2d(3, 6, 5) # 3 channels in, 6 channels out ,  kernel size of 5
self.conv2 = nn.Conv2d(6, 16, 5) # 6 channels in, 16 channels out ,  kernel size of 5
self.fc1   = nn.Linear(16*5*5, 120) # 16*5*5 in features , 120 ouot feature

From resnet.py the following :

self.fc1   = nn.Linear(16*5*5, 120)

From http://cs231n.github.io/convolutional-networks/ the following is stated :

  Summary. To summarize, the Conv Layer:

  Accepts a volume of size W1×H1×D1 Requires four hyperparameters:
  Number of filters K, their spatial extent F, the stride S, the amount
  of zero padding P. Produces a volume of size W2×H2×D2 where:
  W2=(W1−F+2P)/S+1 H2=(H1−F+2P)/S+1 (i.e. width and height are computed
  equally by symmetry) D2=K With parameter sharing, it introduces F⋅F⋅D1
  weights per filter, for a total of (F⋅F⋅D1)⋅K weights and K biases. In
  the output volume, the d-th depth slice (of size W2×H2) is the result
  of performing a valid convolution of the d-th filter over the input
  volume with a stride of S, and then offset by d-th bias.

From this I'm attempting to understand how the training image dimension 32x32 (1024 pixels) is transformed to feature map (16*5*5 -> 400) as aprt of nn.Linear(16*5*5, 120)

From https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d can see default stride is 1 and padding is 0.

What are steps to arrive at 16*5*5 from image dimension of 32*32 and can 16*5*5 be derived from above steps ?

From above steps how to calculate spatial extent ?

Update :

Source code :

'''LeNet in PyTorch.'''
import torch.nn as nn
import torch.nn.functional as F

class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1   = nn.Linear(16*5*5, 120)
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, 10)

    def forward(self, x):
        out = F.relu(self.conv1(x))
        out = F.max_pool2d(out, 2)
        out = F.relu(self.conv2(out))
        out = F.max_pool2d(out, 2)
        out = out.view(out.size(0), -1)
        out = F.relu(self.fc1(out))
        out = F.relu(self.fc2(out))
        out = self.fc3(out)
        return out

Taken from https://github.com/kuangliu/pytorch-cifar/blob/master/models/lenet.py

My understanding is that the convolution operation is applied to the image data per kernel. So if 5 kernels are set then 5 convolutions are applied to the data which generates a 5 dimensional image representation.


title: Pytorch: how to mask flexible size of input for average pooling?

body:
Backgroud: I would like to apply convolutions and average pooling to sequences for sequence classification task.

Problem: without mask for inputs, the calculation of mini-batch and one-by-one will be different.
For example:

# Two sequences.
s1 = torch.range(start=1, end=6).view(-1, 1)
s2 = torch.range(start=1, end=3).view(-1, 1)

##########################################
# one-by-one

# Convolutions.
kernels = torch.ones(1, 1, 2)
h1 = F.conv1d(s1.view(1, 1, -1), kernels) #h1=[[[3, 5, 7, 9, 11]]]
h2 = F.conv1d(s2.view(1, 1, -1), kernels) #h2=[[[3, 5]]]

# Average pooling.
h1 = h1.mean(-1) #h1=[[7]]
h2 = h2.mean(-1) #h2=[[4]]

##########################################
# mini-batch
s = torch.nn.utils.rnn.pad_sequence([s1, s2], batch_first=True)
s = s.permute(0, 2, 1)
h = torch.mean(F.conv1d(s, kernels), dim=-1) #h=[[7], [2.2]]

As you can see, the h is different from [h1, h2].

Questions:

In order to make mini-batch the same as one-by-one, what is the normal way to employ?
Is there a given pytorch api?
If not, how to implement that?


title: How to change the structure of a model in PyTorch

body:
I want to build a stacked auto encoder or recursive network. These are necessary to build a dynamic neural network, which can change its structure in each iteration.
For example, I first train
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(784,500)
        self.fc2 = nn.Linear(500,784)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return x

next, I want to train using previous fc1 and fc2
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(784,500)
        self.fc3 = nn.Linear(500,10)
        self.fc4 = nn.Linear(10,500)
        self.fc2 = nn.Linear(500,784)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = F.relu(self.fc2(x))
        return x

How can I build these networks in single model?


title: Is it possible to check if elements of a tensor are out of boundaries?

body:
Is it possible to check if elements of a tensor are out of boundaries using torch.cuda.FloatTensor on PyTorch, GPU approach?

Example (check limits):

for i in range(pop):
    if (x[i]>xmax):
        x[i]=xmax
    elif (x[i]<xmin):
        x[i]=xmin

I tried the following, but did not speed up:

idxmax    = (x > xmax) # elements that are bigger that upper limit
idxmim    = (x < min)  # elements that are smaller that upper limit
x[idxmax] = xmax
x[idxmin] = xmin

If not, is it possible to do this check limits part, using only the CPU? How?


title: How to get top k accuracy in semantic segmentation using PyTorch?

body:
How do you compute the top k accuracy in semantic segmentation? In classification, we might compute the topk accuracy as:
correct = output.eq(gt.view(1, -1).expand_as(output))


title: PyTorch C++ convert data to tensor fails

body:
I am trying / using the torch_script tool to run my PyTorch trained NN in C++.
The below code fails with linker errors

cv::Mat amplitudePatch = Assigned somewhere above in code;

auto options = torch::TensorOptions().dtype(torch::kF32);
torch::from_blob((void*)amplitudePatch.data,
            {1, amplitudePatch.size().height, amplitudePatch.size().width, amplitudePatch.channels()},
            options);

When I uncomment the from_blob line my code compiles well, enable it and it fails with below error.
I'm following examples on the web that claim this works. Where shall I look?

In function `c10::tensorTypeIdToBackend(c10::TensorTypeId)':
/usr/local/lib/python3.6/dist-packages/torch/include/c10/core/Backend.h:107: undefined reference to `c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
SunlightNoiseClassifier/libSunlightNoiseClassifier.a(SunlightNoiseClassifier.cpp.o): In function `c10::backendToDeviceType(c10::Backend)':
/usr/local/lib/python3.6/dist-packages/torch/include/c10/core/Backend.h:169: undefined reference to `c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/local/lib/python3.6/dist-packages/torch/include/c10/core/Backend.h:171: undefined reference to `c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
SunlightNoiseClassifier/libSunlightNoiseClassifier.a(SunlightNoiseClassifier.cpp.o): In function `c10::scalarTypeToTypeMeta(c10::ScalarType)':
/usr/local/lib/python3.6/dist-packages/torch/include/c10/core/ScalarType.h:125: undefined reference to `c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
SunlightNoiseClassifier/libSunlightNoiseClassifier.a(SunlightNoiseClassifier.cpp.o): In function `c10::typeMetaToScalarType(caffe2::TypeMeta)':
/usr/local/lib/python3.6/dist-packages/torch/include/c10/core/ScalarType.h:151: undefined reference to `c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
SunlightNoiseClassifier/libSunlightNoiseClassifier.a(SunlightNoiseClassifier.cpp.o):/usr/local/lib/python3.6/dist-packages/torch/include/c10/core/Device.h:93: more undefined references to `c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' follow
SunlightNoiseClassifier/libSunlightNoiseClassifier.a(SunlightNoiseClassifier.cpp.o): In function `at::Context::getDeviceFromPtr(void*, c10::DeviceType)':
/usr/local/lib/python3.6/dist-packages/torch/include/ATen/Context.h:47: undefined reference to `c10::DeviceTypeName[abi:cxx11](c10::DeviceType, bool)'
/usr/local/lib/python3.6/dist-packages/torch/include/ATen/Context.h:47: undefined reference to `c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
SunlightNoiseClassifier/libSunlightNoiseClassifier.a(SunlightNoiseClassifier.cpp.o): In function `at::from_blob(void*, c10::ArrayRef<long>, c10::ArrayRef<long>, std::function<void (void*)> const&, c10::TensorOptions const&)':
/usr/local/lib/python3.6/dist-packages/torch/include/ATen/Functions.h:1099: undefined reference to `c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
SunlightNoiseClassifier/libSunlightNoiseClassifier.a(SunlightNoiseClassifier.cpp.o): In function `torch::autograd::make_variable(at::Tensor, bool, bool)':
/usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/autograd/variable.h:524: undefined reference to `c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
SunlightNoiseClassifier/libSunlightNoiseClassifier.a(SunlightNoiseClassifier.cpp.o): In function `c10::intrusive_ptr<c10::StorageImpl, c10::detail::intrusive_target_default_null_type<c10::StorageImpl> >::retain_()':
/usr/local/lib/python3.6/dist-packages/torch/include/c10/util/intrusive_ptr.h:183: undefined reference to `c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
SunlightNoiseClassifier/libSunlightNoiseClassifier.a(SunlightNoiseClassifier.cpp.o): In function `c10::intrusive_ptr<c10::TensorImpl, c10::UndefinedTensorImpl>::retain_()':
/usr/local/lib/python3.6/dist-packages/torch/include/c10/util/intrusive_ptr.h:183: undefined reference to `c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
SunlightNoiseClassifier/libSunlightNoiseClassifier.a(SunlightNoiseClassifier.cpp.o):/usr/local/lib/python3.6/dist-packages/torch/include/c10/util/intrusive_ptr.h:183: more undefined references to `c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' follow
collect2: error: ld returned 1 exit status


title: How can I only update some specific tensors in network with pytorch?

body:
For instance, I want to only update all cnn weights in Resnet in the first 10 epochs and freeze the others.
And from 11th epoch, I wanna change to update the whole model.
How can I achieve the goal?


title: How can I use the LBFGS optimizer with pytorch ignite?

body:
I started using Ignite recently and i found it very interesting.
I would like to train a model using as an optimizer the LBFGS algorithm from the torch.optim module.
This is my code:
from ignite.engine import Events, Engine, create_supervised_trainer, create_supervised_evaluator
from ignite.metrics import RootMeanSquaredError, Loss
from ignite.handlers import EarlyStopping

 D_in, H, D_out = 5, 10, 1
 model = simpleNN(D_in, H, D_out) # a simple MLP with 1 Hidden Layer
 model.double()
 train_loader, val_loader = get_data_loaders(i)

 optimizer = torch.optim.LBFGS(model.parameters(), lr=1)
 loss_func = torch.nn.MSELoss()

 #Ignite
 trainer = create_supervised_trainer(model, optimizer, loss_func)
 evaluator = create_supervised_evaluator(model, metrics={'RMSE': RootMeanSquaredError(),'LOSS': Loss(loss_func)})

 @trainer.on(Events.ITERATION_COMPLETED)
 def log_training_loss(engine):
     print("Epoch[{}] Loss: {:.5f}".format(engine.state.epoch, len(train_loader), engine.state.output))

def score_function(engine):
    val_loss = engine.state.metrics['RMSE']
    print("VAL_LOSS: {:.5f}".format(val_loss))
    return -val_loss

handler = EarlyStopping(patience=10, score_function=score_function, trainer=trainer)
evaluator.add_event_handler(Events.COMPLETED, handler)

trainer.run(train_loader, max_epochs=100)

And the error that raises is:
TypeError: step() missing 1 required positional argument: 'closure'
I know that is required to define a closure for the implementation of LBFGS, so my question is how can I do it using ignite? or is there another approach for doing this?


title: huggingface transformers gpt2 generate multiple GPUs

body:
I'm using huggingface transformer gpt-xl model to generate multiple responses. I'm trying to run it on multiple gpus because gpu memory maxes out with multiple larger responses. I've tried using dataparallel to do this but, looking at nvidia-smi it does not appear that the 2nd gpu is ever used. Here's my code:

import numpy as np
from transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch
n_gpu=torch.cuda.device_count()
#device = xm.xla_device()
device=torch.device("cuda:0")
tokenizer = GPT2Tokenizer.from_pretrained('/spell/GPT2Model/GPT2Model/') #downloaded pre-trained model and tokenizer earlier
model = GPT2LMHeadModel.from_pretrained('/spell/GPT2Model/GPT2Model/')
model.to(device)
model = torch.nn.DataParallel(model, device_ids=[0,1])
encoded_prompt=tokenizer.encode(prompt_text, add_special_tokens=True,return_tensors="pt")
encoded_prompt = encoded_prompt.to(device)
outputs = model.module.generate(encoded_prompt,response_length,temperature=.8,num_return_sequences=num_of_responses,repetition_penalty=85,do_sample=True,top_k=80,top_p=.85 )

program gets oom on dual T4, memory of 2nd gpu never goes above 11M.