title: Building RNN from scratch in pytorch

body:
I am trying to build RNN from scratch using pytorch and I am following this tutorial to build it.

import torch
import torch.nn as nn
import torch.nn.functional as F

class BasicRNN(nn.Module):
    def __init__(self, n_inputs, n_neurons):
        super(BasicRNN, self).__init__()

        self.Wx = torch.randn(n_inputs, n_neurons) # n_inputs X n_neurons
        self.Wy = torch.randn(n_neurons, n_neurons) # n_neurons X n_neurons

        self.b = torch.zeros(1, n_neurons) # 1 X n_neurons

    def forward(self, X0, X1):
        self.Y0 = torch.tanh(torch.mm(X0, self.Wx) + self.b) # batch_size X n_neurons

        self.Y1 = torch.tanh(torch.mm(self.Y0, self.Wy) +
                            torch.mm(X1, self.Wx) + self.b) # batch_size X n_neurons

        return self.Y0, self.Y1

class CleanBasicRNN(nn.Module):

    def __init__(self, batch_size, n_inputs, n_neurons):
        super(CleanBasicRNN, self).__init__()

        self.rnn = BasicRNN(n_inputs, n_neurons)
        self.hx = torch.randn(batch_size, n_neurons) # initialize hidden state

    def forward(self, X):
        output = []

        # for each time step
        for i in range(2):
            self.hx = self.rnn(X[i], self.hx)
            output.append(self.hx)

        return output, self.hx

FIXED_BATCH_SIZE = 4 # our batch size is fixed for now
N_INPUT = 3
N_NEURONS = 5

X_batch = torch.tensor([[[0,1,2], [3,4,5],
                         [6,7,8], [9,0,1]],
                        [[9,8,7], [0,0,0],
                         [6,5,4], [3,2,1]]
                       ], dtype = torch.float) # X0 and X1

model = CleanBasicRNN(FIXED_BATCH_SIZE,N_INPUT,N_NEURONS)

a1,a2 = model(X_batch)

Running this code returns this error

  RuntimeError: size mismatch, m1: [4 x 5], m2: [3 x 5] at /pytorch/..

After some digging I found this error happens when passing the hidden states to the BasicRNN model

N_INPUT = 3 # number of features in input
N_NEURONS = 5 # number of units in layer
X0_batch = torch.tensor([[0,1,2], [3,4,5],
                         [6,7,8], [9,0,1]],
                        dtype = torch.float) #t=0 => 4 X 3

X1_batch = torch.tensor([[9,8,7], [0,0,0],
                         [6,5,4], [3,2,1]],
                        dtype = torch.float) #t=1 => 4 X 3

test_model = BasicRNN(N_INPUT,N_NEURONS)
a1,a2 = test_model(X0_batch,X1_batch)
a1,a2 = test_model(X0_batch,torch.randn(1,N_NEURONS)) # THIS LINE GIVES ERROR

What is happening in the hidden states and How can I solve this problem?


title: Pytorch Softmax giving nans and negative values as output

body:
I am using softmax at the end of my model.
However after some training softmax is giving negative probability.In some situations I have encountered nans as probability as well.
one solution i found on searching is to use normalized softmax…however I can not find any pytorch imlpementaion for this.
Can someone please help to let know if there is a normalized softmax available or how to achieve this so that forward and backward propagations are smooth.
Please note that I am already using torch.nn.utils.clip_grad_norm_(model.parameters(), 40) to avoid exploding gradients
I am using pytorch 1.6.0


I am running code that UI downloaded from github. It is supposed to be working (I saw that other people managed to activate it). When I try to run it I get the following error message:
RuntimeError: CuDNN error: CUDNN_STATUS_SUCCESS

The code uses pytorch 0.4.1. I have cuda installed.
When I run the command cat /usr/local/cuda/version.txt
I get the answer:
CUDA Version 10.0.130

When I run the command conda list -n <my env name>
I see:
cudatoolkit ver 9.0
cudnn       ver 7.6.5

And now, my question:
What should I do to avoid this error?
Do I need to use pip install for a more recent version of cudnn? If so, which one?


title: Weight Initialization from pretrained BERT error in pytorch

body:
I am trying to train the model using pretrained model(BERT) using pytorch. The pretrained model weights still arent accepted.

I see this error:

Weights of BertForMultiLable not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
Weights from pretrained model not used in BertForMultiLable: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']

Here is the complete traceback:

Training/evaluation parameters Namespace(adam_epsilon=1e-08, arch='bert', data_name='ICD9', do_data=False, do_lower_case=True, do_test=False, do_train=True, epochs=6, eval_batch_size=8, eval_max_seq_len=256, fp16=False, fp16_opt_level='O1', grad_clip=1.0, gradient_accumulation_steps=1, learning_rate=2e-05, local_rank=-1, loss_scale=0, mode='min', monitor='valid_loss', n_gpu='0', resume_path='', save_best=True, seed=42, sorted=1, train_batch_size=8, train_max_seq_len=256, valid_size=0.2, warmup_proportion=0.1, weight_decay=0.01)
Loading examples from cached file pybert/dataset/cached_train_examples_bert
Loading features from cached file pybert/dataset/cached_train_features_256_bert
sorted data by th length of input
Loading examples from cached file pybert/dataset/cached_valid_examples_bert
Loading features from cached file pybert/dataset/cached_valid_features_256_bert
initializing model
loading configuration file pybert/pretrain/bert/base-uncased/config.json
Model config {
  "attention_probs_dropout_prob": 0.1,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_labels": 19,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 28996
}

loading weights file pybert/pretrain/bert/base-uncased/pytorch_model.bin
Weights of BertForMultiLable not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
Weights from pretrained model not used in BertForMultiLable: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']
initializing callbacks
***** Running training *****
  Num examples = 21479
  Num Epochs = 6
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 16110
Warning: There's no GPU available on this machine, training will be performed on CPU.
Warning: The number of GPU's configured to use is 0, but only 0 are available on this machine.
Traceback (most recent call last):
  File "run_bert.py", line 227, in <module>
    main()
  File "run_bert.py", line 220, in main
    run_train(args)
  File "run_bert.py", line 125, in run_train
    trainer.train(train_data=train_dataloader, valid_data=valid_dataloader, seed=args.seed)
  File "/home/aditya_vartak/bert_pytorch/pybert/train/trainer.py", line 168, in train
    summary(self.model,*(input_ids, segment_ids,input_mask),show_input=True)
  File "/home/aditya_vartak/bert_pytorch/pybert/common/tools.py", line 307, in summary
    model(*inputs)
  File "/home/aditya_vartak/virtualenvs/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/aditya_vartak/bert_pytorch/pybert/model/nn/bert_for_multi_label.py", line 14, in forward
    outputs = self.bert(input_ids, token_type_ids=token_type_ids,attention_mask=attention_mask, head_mask=head_mask)
  File "/home/aditya_vartak/virtualenvs/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/aditya_vartak/virtualenvs/anaconda3/envs/pytorch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 722, in forward
    embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)
  File "/home/aditya_vartak/virtualenvs/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 533, in __call__
    result = hook(self, input)
  File "/home/aditya_vartak/bert_pytorch/pybert/common/tools.py", line 269, in hook
    summary[m_key]["input_shape"] = list(input[0].size())
IndexError: tuple index out of range

Any help would be great. Thanks in advance


title: No module named 'torch.autograd'

body:
Working with torch package:

import torch

from torch.autograd import Variable

x_data = [1.0,2.0,3.0]

y_data = [2.0,4.0,6.0]

w = Variable(torch.Tensor([1.0]), requires_grad = True)

def forward(x):

return x*w

def loss(x,y):

y_pred = forward(x)

return (y_pred-y)*(y_pred-y)

print("my prediction before training",4,forward(4))

for epoch in range(10):

for x_val, y_val in zip(x_data,y_data):

l= loss(x_val, y_val)

l.backward()

print("\tgrad: ", x_val, y_val, w.grad.data[0])

w.data=w.data-0.01*w.grad.data

w.grad.data.zero_()

print("progress:", epoch, l.data[0] )

print("my new prediction after training ", forward(4))

Got error:

runfile('C:/gdrive/python/temp2.py', wdir='C:/gdrive/python')

Traceback (most recent call last):

File "C:\gdrive\python\temp2.py", line 11, in <module>

from torch.autograd import Variable

ModuleNotFoundError: No module named 'torch.autograd'

Command conda list pytorch brings:

# packages in environment at C:\Users\g\.conda\envs\test:

#

# Name Version Build Channel

(test) PS C:\gdrive\python>

How to fix this problem?


title: PyTorch: Multi GPU error: RuntimeError: binary_op(): expected both inputs to be on same device, but input a is on cuda:0 and input b is on cuda:7

body:
When I use multiple GPUs
and also when I use .cuda() for the tensors
in the middle of training, I got following error

RuntimeError: binary_op(): expected both inputs to be on same device,
but input a is on cuda:0 and input b is on cuda:7

which means following R2 is on cuda:7 (not sure R2 in on cuda:0 or cuda:7) and
R1 is on cuda:0
so, operation is impossible because they’re on difference GPU

R2=torch.where(R2<1e-4,torch.Tensor([1e-4]).squeeze().cuda(),R2)
div_R1_R2=torch.div(R1,R2)

I get exactly same error in the following code
complaining O_img_tc and R_gt_img_tc can’t be calculated
because they’re on different GPU

R_gt_img_tc=torch.where(
  torch.abs(R_gt_img_tc)<1e-4,
  torch.Tensor([1e-4]).squeeze().cuda(),R_gt_img_tc)
sha=torch.clamp(torch.div(O_img_tc,R_gt_img_tc),0.0,1.3)[:,0,:,:].unsqueeze(1)

How to solve this, and what am I doing wrong?

What I've tried:
- Use horovod: got same error.
- Check the GPU number by using dense_O_img_tc.get_device() and dense_S_gt_img_tc.get_device()
When they were like dense_O_img_tc.get_device() returned 0, dense_S_gt_img_tc.get_device() returned 7

And I tried this

same_cuda=torch.device('cuda:'+str(dense_O_img_tc.get_device()))
dense_S_gt_img_tc=torch.where(
  torch.abs(dense_S_gt_img_tc)<1e-4,
  # Note here that I'm using cuda(same_cuda)
  torch.Tensor([1e-4]).squeeze().cuda(same_cuda),dense_S_gt_img_tc)

ref=torch.div(dense_O_img_tc,dense_S_gt_img_tc)

That actually could resolved "different GPU issue"
but only GPU:0 was used, resulting in full of GPU:0 error.

So, I tried to move dense_O_img_tc which is on the GPU:0 towards GPU:7 (or anywhere dense_S_gt_img_tc is on by using dense_S_gt_img_tc.get_device()), I might have got following illegal memory access error

RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /opt/conda/conda-bld/pytorch_1544199946412/work/aten/src/THC/generic/THCTensorMath.cu:238


title: PyTorch transformations change dimensions

body:
I do not understand why pytorch transformations from 100x100 pic make 3x100 pic.

print("Original shape ", x.shape)

x = transforms.Compose([
    transforms.ToPILImage(),
    transforms.ToTensor()
])(x)

print("After transformation shape ", x.shape)

outputs

Original shape  torch.Size([100, 100, 3])
After transformation shape  torch.Size([3, 100, 3])

What is happening?


title: Runtime Error - element 0 of tensors does not require grad and does not have a grad_fn

body:
I am using a Unet model for semantic segmentation - I have a custom dataset of images and their masks both in .png format. I have looked in the online forums and tried stuff, but not much works?
Any suggestions in how to resolve the error or improve the code would be helpful.
model.eval()
with torch.no_grad():
    for xb, yb in val_dl:
        yb_pred = model(xb.to(device))
        # yb_pred = yb_pred["out"].cpu()
        print(yb_pred.shape)
        yb_pred = torch.argmax(yb_pred,axis = 1)
        break

    print(yb_pred.shape)

criteron = nn.CrossEntropyLoss(reduction = 'sum')
opt = optim.Adam(model.parameters(), lr = 3e-4)

def loss_batch(loss_func, output, target, opt = None):
    loss = loss_func(output, target)

    if opt is not None:
        opt.zero_grad()
        loss.backward()
        opt.step()
    return loss.item(), None

lr_scheduler = ReduceLROnPlateau(opt, mode = 'min', factor = 0.5, patience= 20, verbose = 1)

def get_lr(opt):
    for param_group in opt.param_groups:
        return param_group['lr']

current_lr = get_lr(opt)
print('current_lr = {}'.format(current_lr))

def loss_epoch(model, loss_func, dataset_dl, sanity_check = False, opt = None):
    running_loss = 0.0
    len_data = len(dataset_dl.dataset)

    for xb, yb in dataset_dl:
        xb = xb.to(device)
        yb = yb.to(device)

        # xb = torch.tensor(xbh, requires_grad=True)

        output = model(xb)

        loss_b, metric_b = loss_batch(loss_func, output, yb, opt)
        running_loss += loss_b
        if sanity_check is True:
            break
    loss = running_loss/float(len_data)
    return loss, None

def train_val(model, params):
    num_epochs = params["num_epochs"]
    loss_func = params["loss_func"]
    opt = params["optimizer"]
    train_dl = params["train_dl"]
    val_dl = params["val_dl"]
    sanity_check = params["sanity_check"]
    lr_scheduler = params["lr_scheduler"]
    path2weights = params["path2weights"]

    loss_history = {"train": [],
                    "val": []}
    best_model_wts = copy.deepcopy(model.state_dict())
    best_loss = float('inf')

    for epoch in range(num_epochs):
        current_lr = get_lr(opt)
        print('Epoch {}/{}, current_lr = {}'.format(epoch, num_epochs - 1, current_lr))

        with torch.enable_grad():
            model.train()
            train_loss, _ = loss_epoch(model, loss_func, train_dl, sanity_check, opt)
        loss_history["train"].append(train_loss)
        model.eval()

        with torch.no_grad():
            val_loss, _ = loss_epoch(model, loss_func, val_dl, sanity_check, opt)
        loss_history["val"].append(val_loss)

        if val_loss < best_loss:
            best_loss = val_loss
            best_model_wts = copy.deepcopy(model.state_dict())
            torch.save(model.state_dict(), path2weights)
            print("copied best model weights!!")

        lr_scheduler.step(val_loss)
        if current_lr != get_lr(opt):
            print("Loading best model weights!!")
            model.load_state_dict(best_model_wts)
            print("train Loss: %.6f" %(train_loss))
            print("val_loss: %.6f" %(val_loss))
            print("-"*20)

        model.load_state_dict(best_model_wts)
        return model, loss_history, metric_history

path2models = "./models/"
if not os.path.exists(path2models):
    os.mkdir(path2models)

param_train = {
    "num_epochs": 10,
    "loss_func": criteron,
    "optimizer": opt,
    "train_dl": train_dl,
    "val_dl": val_dl,
    "sanity_check": False,
    "lr_scheduler": lr_scheduler,
    "path2weights": path2models + "weights.pt"
model, loss_hist, _ = train_val(model, param_train)

The error message looks like -
File "", line 10, in
model, loss_hist, _ = train_val(model, param_train)
File "", line 27, in train_val
val_loss, _ = loss_epoch(model, loss_func, val_dl, sanity_check, opt)
File "", line 13, in loss_epoch
loss_b, metric_b = loss_batch(loss_func, output, yb, opt)
File "", line 6, in loss_batch
loss.backward()
File "C:\Users\W540\anaconda3\lib\site-packages\torch\tensor.py", line 198, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "C:\Users\W540\anaconda3\lib\site-packages\torch\autograd_init_.py", line 100, in backward
allow_unreachable=True)  # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
I am not sure which variable to set as require_grad = True or where I should enable grad...


title: how to install pytorch and fastai with conda

body:
I created conda environment and install pytorch and fastai (Mac OS Mojave) as below:

conda create -n fai_course python=3.7
source activate fai_course
conda install -c pytorch pytorch-nightly-cpu
conda install -c fastai torchvision-nightly-cpu
jupyter notebook

When I import a package from jupyter notebook, I get OSError as shown below:

from fastai.imports import *

-

--------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matplotlib/font_manager.py in <module>
   1352         try:
-> 1353             fontManager = json_load(_fmcache)
   1354             if (not hasattr(fontManager, '_version') or

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matplotlib/font_manager.py in json_load(filename)
    887     """
--> 888     with open(filename, 'r') as fh:
    889         return json.load(fh, object_hook=_json_decode)

FileNotFoundError: [Errno 2] No such file or directory: '/Users/user/.matplotlib/fontlist-v300.json'

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-5-9f9378ae0f2a> in <module>
----> 1 from fastai.imports import *

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/fastai/__init__.py in <module>
----> 1 from .basic_train import *
      2 from .callback import *
      3 from .callbacks import *
      4 from .core import *
      5 from .data import *

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/fastai/basic_train.py in <module>
      1 "Provides basic training and validation with `Learner`"
----> 2 from .torch_core import *
      3 from .data import *
      4 from .callback import *
      5

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/fastai/torch_core.py in <module>
      1 "Utility functions to help deal with tensors"
----> 2 from .imports.torch import *
      3 from .core import *
      4
      5 AffineMatrix = Tensor

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/fastai/imports/__init__.py in <module>
----> 1 from .core import *
      2 from .torch import *

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/fastai/imports/core.py in <module>
      1 import csv, gc, gzip, os, pickle, shutil, sys, warnings
----> 2 import math, matplotlib.pyplot as plt, numpy as np, pandas as pd, random
      3 import scipy.stats, scipy.special
      4 import abc, collections, hashlib, itertools, json, operator
      5 import mimetypes, inspect, typing, functools

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matplotlib/pyplot.py in <module>
     30 from cycler import cycler
     31 import matplotlib
---> 32 import matplotlib.colorbar
     33 import matplotlib.image
     34 from matplotlib import rcsetup, style

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matplotlib/colorbar.py in <module>
     30 import matplotlib.collections as collections
     31 import matplotlib.colors as colors
---> 32 import matplotlib.contour as contour
     33 import matplotlib.cm as cm
     34 import matplotlib.gridspec as gridspec

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matplotlib/contour.py in <module>
     16 import matplotlib.colors as mcolors
     17 import matplotlib.collections as mcoll
---> 18 import matplotlib.font_manager as font_manager
     19 import matplotlib.text as text
     20 import matplotlib.cbook as cbook

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matplotlib/font_manager.py in <module>
   1361             raise
   1362         except Exception:
-> 1363             _rebuild()
   1364     else:
   1365         _rebuild()

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matplotlib/font_manager.py in _rebuild()
   1342         global fontManager
   1343
-> 1344         fontManager = FontManager()
   1345
   1346         if _fmcache:

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matplotlib/font_manager.py in __init__(self, size, weight)
    976         self.defaultFont = {}
    977
--> 978         ttffiles = findSystemFonts(paths) + findSystemFonts()
    979         self.defaultFont['ttf'] = next(
    980             (fname for fname in ttffiles

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matplotlib/font_manager.py in findSystemFonts(fontpaths, fontext)
    268             # check for OS X & load its fonts if present
    269             if sys.platform == 'darwin':
--> 270                 fontfiles.update(OSXInstalledFonts(fontext=fontext))
    271
    272     elif isinstance(fontpaths, str):

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matplotlib/font_manager.py in OSXInstalledFonts(directories, fontext)
    216         directories = OSXFontDirectories
    217     return [path
--> 218             for directory in directories
    219             for ext in get_fontext_synonyms(fontext)
    220             for path in list_fonts(directory, ext)]

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matplotlib/font_manager.py in <listcomp>(.0)
    218             for directory in directories
    219             for ext in get_fontext_synonyms(fontext)
--> 220             for path in list_fonts(directory, ext)]
    221
    222

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matplotlib/font_manager.py in list_fonts(directory, extensions)
    155     extensions = ["." + ext for ext in extensions]
    156     return [str(path)
--> 157             for path in filter(Path.is_file, Path(directory).glob("**/*.*"))
    158             if path.suffix in extensions]
    159

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matplotlib/font_manager.py in <listcomp>(.0)
    154     """
    155     extensions = ["." + ext for ext in extensions]
--> 156     return [str(path)
    157             for path in filter(Path.is_file, Path(directory).glob("**/*.*"))
    158             if path.suffix in extensions]

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py in glob(self, pattern)
   1080             raise NotImplementedError("Non-relative patterns are unsupported")
   1081         selector = _make_selector(tuple(pattern_parts))
-> 1082         for p in selector.select_from(self):
   1083             yield p
   1084

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py in _select_from(self, parent_path, is_dir, exists, scandir)
    541             try:
    542                 successor_select = self.successor._select_from
--> 543                 for starting_point in self._iterate_directories(parent_path, is_dir, scandir):
    544                     for p in successor_select(starting_point, is_dir, exists, scandir):
    545                         if p not in yielded:

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py in _iterate_directories(self, parent_path, is_dir, scandir)
    531                 if entry.is_dir() and not entry.is_symlink():
    532                     path = parent_path._make_child_relpath(entry.name)
--> 533                     for p in self._iterate_directories(path, is_dir, scandir):
    534                         yield p
    535         except PermissionError:

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py in _iterate_directories(self, parent_path, is_dir, scandir)
    531                 if entry.is_dir() and not entry.is_symlink():
    532                     path = parent_path._make_child_relpath(entry.name)
--> 533                     for p in self._iterate_directories(path, is_dir, scandir):
    534                         yield p
    535         except PermissionError:

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py in _iterate_directories(self, parent_path, is_dir, scandir)
    531                 if entry.is_dir() and not entry.is_symlink():
    532                     path = parent_path._make_child_relpath(entry.name)
--> 533                     for p in self._iterate_directories(path, is_dir, scandir):
    534                         yield p
    535         except PermissionError:

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py in _iterate_directories(self, parent_path, is_dir, scandir)
    531                 if entry.is_dir() and not entry.is_symlink():
    532                     path = parent_path._make_child_relpath(entry.name)
--> 533                     for p in self._iterate_directories(path, is_dir, scandir):
    534                         yield p
    535         except PermissionError:

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py in _iterate_directories(self, parent_path, is_dir, scandir)
    531                 if entry.is_dir() and not entry.is_symlink():
    532                     path = parent_path._make_child_relpath(entry.name)
--> 533                     for p in self._iterate_directories(path, is_dir, scandir):
    534                         yield p
    535         except PermissionError:

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py in _iterate_directories(self, parent_path, is_dir, scandir)
    531                 if entry.is_dir() and not entry.is_symlink():
    532                     path = parent_path._make_child_relpath(entry.name)
--> 533                     for p in self._iterate_directories(path, is_dir, scandir):
    534                         yield p
    535         except PermissionError:

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py in _iterate_directories(self, parent_path, is_dir, scandir)
    531                 if entry.is_dir() and not entry.is_symlink():
    532                     path = parent_path._make_child_relpath(entry.name)
--> 533                     for p in self._iterate_directories(path, is_dir, scandir):
    534                         yield p
    535         except PermissionError:

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py in _iterate_directories(self, parent_path, is_dir, scandir)
    529             entries = list(scandir(parent_path))
    530             for entry in entries:
--> 531                 if entry.is_dir() and not entry.is_symlink():
    532                     path = parent_path._make_child_relpath(entry.name)
    533                     for p in self._iterate_directories(path, is_dir, scandir):

OSError: [Errno 62] Too many levels of symbolic links: '.Trash/NETGEARGenie.app/Contents/Frameworks/QtPrintSupport.framework/Versions/5/5'

Can you please let me know what am I missing?
Thanks


title: PyTorch: Trying to backward through the graph a second time, but the buffers have already been freed

body:
My model is:

class BaselineModel(nn.Module):
    def __init__(self, feature_dim=5, hidden_size=5, num_layers=2, batch_size=32):
        super(BaselineModel, self).__init__()
        self.num_layers = num_layers
        self.hidden_size = hidden_size

        self.lstm = nn.LSTM(input_size=feature_dim,
                            hidden_size=hidden_size, num_layers=num_layers)

    def forward(self, x, hidden):
        lstm_out, hidden = self.lstm(x, hidden)
        return lstm_out, hidden

    def init_hidden(self, batch_size):
        hidden = Variable(next(self.parameters()).data.new(
            self.num_layers, batch_size, self.hidden_size))
        cell = Variable(next(self.parameters()).data.new(
            self.num_layers, batch_size, self.hidden_size))
        return (hidden, cell)

My training loop looks like:

    for epoch in range(250):
        hidden = model.init_hidden(13)
        # hidden = (torch.zeros(2, 13, 5),
        #           torch.zeros(2, 13, 5))
        # model.hidden = hidden
        for i, data in enumerate(train_loader):
            inputs = data[0]
            outputs = data[1]

            print('inputs',  inputs.size())
            # print('outputs', outputs.size())

            optimizer.zero_grad()
            model.zero_grad()

            # print('inputs', inputs)
            pred, hidden = model(inputs, hidden)

            loss = loss_fn(pred[0], outputs)

            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
            optimizer.step()

I appear to get through the first epoch, then see this error: RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

I've seen postings about it, but in my case, I am reinitializing my hidden for each batch.


title: Encounter Error after installing Pytorch using pip in python3.5

body:
I am using windows 10 OS with Python 3.5.
I am trying to run a program related to torch. So I installed Pytorch with the help of the instructions specified on their website: Pytorch installation
See the following image for selection of the downloadable:

I installed it using :
pip install http://download.pytorch.org/whl/cpu/torch-0.4.0-cp35-cp35m-win_amd64.whl
pip install torchvision

I have encountered the following error:
Error
Traceback (most recent call last):
  File "C:\Python35\lib\site-packages\twisted\trial\runner.py", line 701, in findByName
    __import__(name)
  File "C:\Users\aims\Desktop\CNN\source\cnn4matrix.py", line 4, in <module>
    import torch
  File "C:\Python35\lib\site-packages\torch\__init__.py", line 78, in <module>
    from torch._C import *
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python35\lib\site-packages\twisted\trial\runner.py", line 803, in loadByName
    return self.suiteFactory([self.findByName(name, recurse=recurse)])
  File "C:\Python35\lib\site-packages\twisted\trial\runner.py", line 704, in findByName
    return self.loadFile(_name, recurse=recurse)
  File "C:\Python35\lib\site-packages\twisted\trial\runner.py", line 674, in loadFile
    module = SourceFileLoader(name, fileName).load_module()
  File "<frozen importlib._bootstrap_external>", line 385, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 806, in load_module
  File "<frozen importlib._bootstrap_external>", line 665, in load_module
  File "<frozen importlib._bootstrap>", line 268, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 693, in _load
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 662, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "C:\Users\aims\Desktop\CNN\source\cnn4matrix.py", line 4, in <module>
    import torch
  File "C:\Python35\lib\site-packages\torch\__init__.py", line 78, in <module>
    from torch._C import *
ImportError: DLL load failed: The specified module could not be found.

After searching a lot I couldn't found anything related. Instead I found that I have tried: Reinstall with upgrade mode, the numpy library.
Please let me know what I can try.
EDITED:
This is the scenario for me:  I have install torch from the website instructions and then opened python terminal. just written import torch and the issue is what I got. Have a look at the image: https://ibb.co/fL53HT


title: Pytorch - why does preallocating memory cause "trying to backward through the graph a second time"

body:
Suppose I have a simple one-hidden-layer network that I'm training in the typical way:

    for x,y in trainData:
        optimizer.zero_grad()
        out = self(x)
        loss = self.lossfn(out, y)
        loss.backward()
        optimizer.step()

This works as expected, but if I instead pre-allocate and update the output array, I get an error:

    out = torch.empty_like(trainData.tensors[1])
    for i,(x,y) in enumerate(trainData):
        optimizer.zero_grad()
        out[i] = self(x)
        loss = self.lossfn(out[i], y)
        loss.backward()
        optimizer.step()

  RuntimeError: Trying to backward through the graph a second time, but
  the buffers have already been freed. Specify retain_graph=True when
  calling backward the first time.

What's happening here that in the second version Pytorch attempts to backward through the graph again? Why is this not an issue in the first version? (Note that this error occurs even if I don't zero_grad())


title: Moving Dataloader data to GPU PyTorch

body:
I am just getting started with PyTorch...

Currently I am following this method of getting training data to the GPU. Source: https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel

# Loop over epochs
for epoch in range(max_epochs):
    # Training
    for local_batch, local_labels in training_generator:
        # Transfer to GPU
        local_batch, local_labels = local_batch.to(device), local_labels.to(device)

        # Model computations
        [...]

Question: This seems less efficient than just loading the entire data set onto the GPU. Does loading data on each batch not cause latency? Am I missing something?


title: Image data cannot be converted to float

body:
I have a code for predicting dog breed after training on CNN model, I get the class index from the below function. I want to display an random image from the class idx folder obtained from the function.
    class_name = [item for item in loaders['train'].dataset.classes]

      def predict_dog_breed(img,model,class_names):
           image = Image.open(img).convert('RGB')
           transform = transforms.Compose([
                          transforms. RandomResizedCrop(224),
                          transforms.ToTensor(),
                         transforms.Normalize(mean=[0.485,0.456,0.406],
                         std=[0.229, 0.224, 0.225])])
           image = transform(image)
           test_image = image.unsqueeze(0)
           net.eval()
           output = net(test_image)
          idx = torch.argmax(output)
          a = random.choice(os.listdir("./dogImages/train/{}/".format (class_name[idx])))
          imshow(a)
          return class_name[idx]

When I tried to display the random image, I am getting the below error:

TypeError Traceback (most recent call last) in 1 for img_file in os.listdir('./images'): 2 image = os.path.join('./images', img_file) ----> 3 dog_or_human(image)
in dog_or_human(img) 5 plt.show() 6 if dog_detector(img) == True: ----> 7 predict_dog = predict_dog_breed(img, net, class_name) 8 print("Dog Detected!The breed is {}".format(predict_dog)) 9 elif face_detector(img) > 0:
in predict_dog_breed(img, model, class_name) 18 a = random.choice(os.listdir("./dogImages/train/{}/".format(class_name[idx]))) 19 print(a) ---> 20 imshow(a) 21 #subdir = ''.join(["/dogImages/train/", class_name[idx]]) 22 #print(file)
~/Library/Python/3.7/lib/python/site-packages/matplotlib/pyplot.py in imshow(X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, data, **kwargs) 2697 filternorm=filternorm, filterrad=filterrad, imlim=imlim, 2698 resample=resample, url=url, **({"data": data} if data is not -> 2699 None else {}), **kwargs) 2700 sci(__ret) 2701 return __ret
~/Library/Python/3.7/lib/python/site-packages/matplotlib/init.py in inner(ax, data, *args, **kwargs) 1808 "the Matplotlib list!)" % (label_namer, func.name), 1809 RuntimeWarning, stacklevel=2) -> 1810 return func(ax, *args, **kwargs) 1811 1812 inner.doc = _add_data_doc(inner.doc,
~/Library/Python/3.7/lib/python/site-packages/matplotlib/axes/_axes.py in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs) 5492 resample=resample, **kwargs) 5493 -> 5494 im.set_data(X) 5495 im.set_alpha(alpha) 5496 if im.get_clip_path() is None:
~/Library/Python/3.7/lib/python/site-packages/matplotlib/image.py in set_data(self, A) 632 if (self._A.dtype != np.uint8 and 633 not np.can_cast(self._A.dtype, float, "same_kind")): --> 634 raise TypeError("Image data cannot be converted to float") 635 636 if not (self._A.ndim == 2
TypeError: Image data cannot be converted to float

Any help on this would be appreciated!


title: Restarting with adam

body:
I am training my network with early stopping strategy. I start with a higher learning rate, and based on validation loss, I need to restart training from an earlier snapshot.

I am able to save/load snapshot with model and optimizer state_dicts. No problem with that.

My question is, once I restart training, how do I set the learning rate of adam again? Should I restart adam fresh instead of using a state_dict or should I use
optimizer.param_groups[0][‘lr’] = lr to adjust learning rate with loaded optimizer state_dict?

For example,
I train my network with lr = 1e-6 for 5 epochs, saved model and optimizer state_dict.
I am now restarting from epoch 6, but I need lr = 1e-7 instead. What is the best approach for this?

Thanks!


title: Extract sub tensor in PyTorch

body:
For this tensor is PyTorch,

tensor([[ 0.7646,  0.5573,  0.4000,  0.2188,  0.7646,  0.5052,  0.2042,  0.0896,
          0.7667,  0.5938,  0.3167,  0.0917],
        [ 0.4271,  0.1354,  0.5000,  0.1292,  0.4260,  0.1354,  0.4646,  0.0917,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [ 0.7208,  0.5656,  0.3000,  0.1688,  0.7177,  0.5271,  0.1521,  0.0667,
          0.7198,  0.5948,  0.2438,  0.0729],
        [ 0.6292,  0.8250,  0.4000,  0.2292,  0.6271,  0.7698,  0.2083,  0.0812,
          0.6281,  0.8604,  0.3604,  0.0917]], device='cuda:0')

How can I extract to new Tensor for those values

0.7646,  0.5573,  0.4000,  0.2188
0.4271,  0.1354,  0.5000,  0.1292

How to get the first 4 of two rows into a new tensor?


title: MNIST dataset is failed to transform as tensor object

body:
How to properly transform the MNIST dataset to tensor type? I tried below but not work. The error message AttributeError: 'int' object has no attribute 'type' indicates it is not tensor type.
The codes below can be tested in Google Colab.
It appears that PyTorch Version 1.3.1 can run this, but not for 1.5.1.
>>> import torch
>>> import torch.nn as nn
>>> import torchvision.transforms as transforms
>>> import torchvision.datasets as dsets
>>> import numpy as np
>>> torch.__version__
1.5.1+cu101

>>> train_dataset = dsets.MNIST(root='./data', train=True, download=True, transform=transforms.ToTensor())
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ./data/MNIST/raw/train-images-idx3-ubyte.gz
100.1%Extracting ./data/MNIST/raw/train-images-idx3-ubyte.gz to ./data/MNIST/raw
Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to ./data/MNIST/raw/train-labels-idx1-ubyte.gz
113.5%Extracting ./data/MNIST/raw/train-labels-idx1-ubyte.gz to ./data/MNIST/raw
Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw/t10k-images-idx3-ubyte.gz
100.4%Extracting ./data/MNIST/raw/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw
Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz
180.4%Extracting ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw
Processing...
/pytorch/torch/csrc/utils/tensor_numpy.cpp:141: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program.
Done!

>>> print("Print the training dataset:\n ", train_dataset)
Print the training dataset:
  Dataset MNIST
    Number of datapoints: 60000
    Root location: ./data
    Split: Train
    StandardTransform
Transform: ToTensor()

>>> print("Type of data element: ", train_dataset[0][1].type())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute 'type'


title: System hangs after first epoch training in pytorch

body:
So, I was trying to train on ResNet model in PyTorch using the ImageNet example in the GitHub repository.

Here's what my train method looks like (it is almost similar to that in example)

def train(train_loader, model, criterion, optimizer, epoch):
    batch_time = AverageMeter()
    data_time = AverageMeter()
    losses = AverageMeter()
    top1 = AverageMeter()
    top5 = AverageMeter()

    args = get_args()

    # switch to train mode
    model.train()

    end = time.time()

    for i, (input, target) in enumerate(train_loader):
        print(i)
        # data loading time
        data_time.update(time.time() - end)

        if cuda:
            target = target.cuda(async = True)
            input_var = torch.autograd.Variable(input).cuda()
        else:
            input_var = torch.autograd.Variable(input)

        target_var = torch.autograd.Variable(target)

        # compute output
        output = model(input_var)
        loss = criterion(output, target_var)

        # measure accuracy and record loss
        prec1, prec5 = accuracy(output.data, target, topk=(1, 5))
        losses.update(loss.item(), input.size(0))
        top1.update(prec1.item(), input.size(0))
        # top5.update(prec5.item(), input.size(0))

        # compute gradient and do optimizer step
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        #measure elapsed time
        batch_time.update(time.time() - end)
        end = time.time()

        # print to console and write logs to tensorboard
        if i % args.print_freq == 0:
            print('Epoch: [{0}][{1}/{2}]\t'
                  'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
                  'Data {data_time.val:.3f} ({data_time.avg:.3f})\t'
                  'Loss {loss.val:.4f} ({loss.avg:.4f})\t'
                  'Prec@1 {top1.val:.3f} ({top1.avg:.3f})\t'.format(
                epoch, i, len(train_loader), batch_time=batch_time,
                data_time=data_time, loss=losses, top1=top1, top5=top5))
            niter = epoch * len(train_loader) + i
            # writer.add_scalar('Train/Loss', losses.val, niter)
            # writer.add_scalar('Train/Prec@1', top1.val, niter)
            # writer.add_scalar('Train/Prec@5', top5.val, niter)

System Information:
GPU: Nvidia Titan XP
Memory: 32 Gb

PyTorch: 0.4.0

When I run this code, training starts with epoch 0

Epoch: [0][0/108]   Time 5.644 (5.644)  Data 1.929 (1.929)  Loss 6.9052 (6.9052)    Prec@1 0.000 (0.000)

And then the remote server automatically disconnects. It happened five times.

And this is the data loader:

#Load the Data --> TRAIN
    traindir = 'train'
    normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                     std=[0.229, 0.224, 0.225])
    train_dataset = datasets.ImageFolder(traindir, transforms.Compose([
            transforms.RandomResizedCrop(224),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            normalize,
        ]))
    train_loader = torch.utils.data.DataLoader(
        train_dataset,
        batch_size=args.batch_size, shuffle=True, num_workers=args.num_workers,
        pin_memory=cuda
    )

    # Load the data --> Validation
    valdir = 'valid'
    valid_loader = torch.utils.data.DataLoader(
        datasets.ImageFolder(valdir, transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            normalize,
        ])),
        batch_size=args.batch_size, shuffle=False, num_workers=args.num_workers,
        pin_memory=cuda
    )

    if args.evaluate:
        validate(valid_loader, model, criterion, epoch=0)
        return

    # Start
    for epoch in range(args.start_epoch, args.epochs):
        adjust_learning_rate(optimizer, epoch)

        # train for epoch
        train(train_loader, model, criterion, optimizer, epoch)

        # evaluate on valid
        prec1 = validate(valid_loader, model, criterion, epoch)

        # remember best prec1 and save checkpoint
        is_best = prec1 > best_prec1
        best_prec1 = max(prec1, best_prec1)
        save_checkpoint({
            'epoch': epoch + 1,
            'arch': args.arch,
            'state_dict': model.state_dict(),
            'best_prec1': best_prec1,
            'optimizer': optimizer.state_dict()
        }, is_best)

With this params for the loader:

args.num_workers = 4
args.batch_size = 32
pin_memory = torch.cuda.is_available()

Is there something wrong in my approach?


I am trying to export a pytorch text detection model to onnx format. The model uses grid_sample in the code of one module . I am unable to convert it to the onnx format because of the following error
/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py:738: UserWarning: ONNX export failed on ATen operator grid_sampler because torch.onnx.symbolic_opset9.grid_sampler does not exist**
  .format(op_name, opset_version, op_name))
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-134-0350cc891688> in <module>()
----> 1 torch.onnx.export(models_module, (dummy_input,dummy_text), "test1.onnx", verbose=True)

7 frames
/usr/local/lib/python3.6/dist-packages/torch/onnx/symbolic_registry.py in get_registered_op(opname, domain, version)
    107         else:
    108             msg += "Please open a bug to request ONNX export support for the missing operator."
--> 109         raise RuntimeError(msg)
    110     return _registry[(domain, version)][opname]

RuntimeError: Exporting the operator grid_sampler to ONNX opset version 9 is not supported. Please open a bug to request ONNX export support for the missing operator.

I am not able to resolve this error . Also the code I am writing to convert to onnx format is shown below
import torch.onnx
dummy_input = torch.autograd.Variable(torch.rand(10,1,32,100,dtype=torch.float32))
dummy_text = torch.autograd.Variable(torch.LongTensor(batch_size, batch_max_length + 1).fill_(0).to(device))     # shape is (10,26)
models_module = model.module.to(device)
torch.onnx.export(models_module, (dummy_input,dummy_text), "test1.onnx", verbose=True)


title: how does the pytorch autograd work?

body:
I submitted this as an issue to cycleGAN pytorch implementation, but since nobody replied me there, i will ask again here.

I'm mainly puzzled by the fact that multiple forward passes was called before one single backward pass, see the following in code cycle_gan_model

# GAN loss
# D_A(G_A(A))
self.fake_B = self.netG_A.forward(self.real_A)
pred_fake = self.netD_A.forward(self.fake_B)
self.loss_G_A = self.criterionGAN(pred_fake, True)
# D_B(G_B(B))
self.fake_A = self.netG_B.forward(self.real_B)
pred_fake = self.netD_B.forward(self.fake_A)
self.loss_G_B = self.criterionGAN(pred_fake, True)
# Forward cycle loss G_B(G_A(A))
self.rec_A = self.netG_B.forward(self.fake_B)
self.loss_cycle_A = self.criterionCycle(self.rec_A, self.real_A) * lambda_A
# Backward cycle loss G_A(G_B(B))
self.rec_B = self.netG_A.forward(self.fake_A)
self.loss_cycle_B = self.criterionCycle(self.rec_B, self.real_B) * lambda_B
# combined loss
self.loss_G = self.loss_G_A + self.loss_G_B + self.loss_cycle_A + self.loss_cycle_B + self.loss_idt_A + self.loss_idt_B
self.loss_G.backward()

The way I see it, G_A and G_B each has three forward passes, twice accepting the real data (real_A or real_B) and twice the fake data (fake_B or fake_A).

In tensorflow (I think) the backward pass is always computed w.r.t the last input data. In this case, the backpropagation of loss_G would be wrong. One should instead do backward pass thrice, each immediately following their involving forward pass.

Specifically, netG_A's gradients from loss_G_A is w.r.t real_A but its gradients from loss_cycle_B is w.r.t fake_A.

I assume this is somehow taken care of in pytorch. But how does the model know w.r.t which input data it should compute the gradients?


title: pytorch runs in anaconda prompt but not in python idle

body:
I know this question might be stupid, but I couldn't find any help on the internet. Recently I installed anaconda in my computer, it runs Windows 10 x64. Then I used anaconda prompt to download and install pytorch for 3.6 python:

conda install pytorch torchvision cuda100 -c pytorch

After the installation I verified in anaconda's prompt that pytorch is installed:

>>> Python
>>> Import torch
>>> torch.cuda.is_available()
True

I also checked conda list and indeed pytorch is installed in my machine.
However, I write Python code in python 3.6.7 IDLE, not in anaconda prompt, so, whenever I try to import pytorch I get the message:

Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
import pytorch
ModuleNotFoundError: No module named 'pytorch'

For some reason, Anacoda prompt recognizes pytorch, but not IDLE. Is there any solution for this? Is there any way to import pytorch module to IDLE?
Thanks in advance.


title: How to create a custom layer for Sampling in Keras Tensorflow?

body:
I'm building a CNN in Keras with a Tensorflow backend, and I'd like to introduce a custom layer that should perform the following:

Output a tensor of same shape and dtype as the input tensor.
The output is made of few samples of the input tensor, let's say 25%. The rest of the output tensor should be zeros.
The samples must be picked at random, such that the highest values pixel are sampled with higher probability. In other words, the probability distribution should be the input tensor itself (normalized).

For now, I've managed to build a mockup where I pick the top 25% pixels of the input tensor and create an output tensor of same size only from them. But it is not a random sampling.

Ideally I'd like to use a tensorflow equivalent of : np.random.choice(input_tensor, num_samples, input_tensor_normalized) where the third argument is the probability distribution to follow. Note that this only works on 1D np.array.

I've heard of tf.random.multinomial but it's depreciated and tf.random.categorical takes logits as inputs (I don't think it's my case) and doesn't propose a probability distribution.

A possibility is to reshape the input tensor as a vector, perform 1D sampling in Tensorflow if there is a way, construct a similar vector with the sampled values at the corresponding index and zeros elsewhere, and then reshape as a tensor afterwards.

Any other idea?

Should I move to PyTorch?

Thank you very Much


title: Google Colab - downloads some files, TypeError: Failed to fetch on others

body:
I have a Google Colab notebook with PyTorch code running in it.
At the beginning of the train function, I create, save and download word_to_ix and tag_to_ix dictionaries without a problem, using the following code:

from google.colab import files

torch.save(tag_to_ix, pos_dict_path)
files.download(pos_dict_path)
torch.save(word_to_ix, word_dict_path)
files.download(word_dict_path)

I train the model, and then try to download it with the code:

torch.save(model.state_dict(), model_path)
files.download(model_path)

Then I get a MessageError: TypeError: Failed to fetch.
Obviously, the problem is not with the third party cookies (as suggested here), because the first files are downloaded without a problem. (I actually also tried adding the link in my Allow section, but, surprise surprise, it made no difference.)

I was originally trying to save the model as is (which, to my understanding, saves it as a Pickle), and I thought maybe Colab files doesn't handle downloading Pickles well, but as you can see above, I'm now trying to save a dict object (which is also what word_to_ix and tag_to_ix) are, and it's still not working.

Downloading the file manually with right-click isn't a solution, because sometimes I leave the code running while I do other things, and by the time I get back to it, the runtime has disconnected, and the files are gone.

Any suggestions?


title: Upsample ONNX gives INVALID_GRAPH when trying to convert bilinear layer to onnx

body:
when i convert network with bilinear layer trained on Pytorch to ONNX, i get following error

  RuntimeError: [ONNXRuntimeError] : 10 : INVALID_GRAPH : Load model
  from test.onnx failed:Type Error: Type 'tensor(int64)' of input
  parameter (11) of operator (Floor) in node () is invalid.

I m not sure on why this error occurs, I tried building ONNX from source but still the issue dosen't seems to go way.

Any ideas on what might cause this error? or how to tackle the issue?

Way to Reproduce-

from torch import nn

import torch
import torch.nn.functional as F
import onnxruntime as rt

class Upsample(torch.nn.Module):
    def forward(self, x):
        #l = nn.Conv2d(3, 3, kernel_size=1, stride=1, padding=1, bias=True)
        return F.interpolate(x, scale_factor=2, mode="bilinear", align_corners=False)

m = Upsample()
v = torch.randn(1,3,128,128, dtype=torch.float32, requires_grad=False)

torch.onnx.export(m, v, "test.onnx")
sess = rt.InferenceSession("test.onnx")


title: PyTorch & Google Colab: ImageFolder doesn't load all images

body:
I'm using PyTorch for a 'Cats vs Dogs' image classification task (9999 images of Cats and 9999 images of Dogs) on Google colab with the following piece of code to create a data loader:

import torchvision
from torchvision import transforms
data = torchvision.datasets.ImageFolder(r'/content/drive/My Drive/kaggle/trainset/trainset',transform = transforms.ToTensor())

and 'data' has a length of 6594 (vs 19998 expected). I try restarting the runtime and I get a different number of images every time. Then I tried running the same piece of code in a jupyter notebook and it worked as expected, loading all 19998 images. What am I doing wrong in colab?


title: NVIDIA driver is not recognized when using Pytorch and Matplotlib

body:
I have been trying to use Pytorch 0.4.0 in Python 3.5. I have an NVIDIA K40c GPU and CUDA driver version V9.0.176. However when I execute "torch.cuda.is_available()" it returns False. So, when I tried to define a Tensor it gives the following error:

AssertionError:
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from

I reduced the code to only these three lines:

import torch
import matplotlib.pyplot as plt
print("Is CUDA available? ",torch.cuda.is_available())

And the result is still False. However, if I comment the line:

#import matplotlib.pyplot as plt

it works fine. So it seems like there are some compatibility problems with matplotlib, any ideas?


title: Config change for a pre-trained transformer model

body:
I am trying to implement a classification head for the reformer transformer. The classification head works fine, but when I try to change one of the config parameters- config.axial_pos_shape i.e sequence length parameter for the model it throws an error;

size mismatch for reformer.embeddings.position_embeddings.weights.0: copying a param with shape torch.Size([512, 1, 64]) from checkpoint, the shape in current model is torch.Size([64, 1, 64]).
size mismatch for reformer.embeddings.position_embeddings.weights.1: copying a param with shape torch.Size([1, 1024, 192]) from checkpoint, the shape in current model is torch.Size([1, 128, 192]).

The config:
{
  "architectures": [
    "ReformerForSequenceClassification"
  ],
  "attention_head_size": 64,
  "attention_probs_dropout_prob": 0.1,
  "attn_layers": [
    "local",
    "lsh",
    "local",
    "lsh",
    "local",
    "lsh"
  ],
  "axial_norm_std": 1.0,
  "axial_pos_embds": true,
  "axial_pos_embds_dim": [
    64,
    192
  ],
  "axial_pos_shape": [
    64,
    256
  ],
  "chunk_size_feed_forward": 0,
  "chunk_size_lm_head": 0,
  "eos_token_id": 2,
  "feed_forward_size": 512,
  "hash_seed": null,
  "hidden_act": "relu",
  "hidden_dropout_prob": 0.05,
  "hidden_size": 256,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": true,
  "layer_norm_eps": 1e-12,
  "local_attention_probs_dropout_prob": 0.05,
  "local_attn_chunk_length": 64,
  "local_num_chunks_after": 0,
  "local_num_chunks_before": 1,
  "lsh_attention_probs_dropout_prob": 0.0,
  "lsh_attn_chunk_length": 64,
  "lsh_num_chunks_after": 0,
  "lsh_num_chunks_before": 1,
  "max_position_embeddings": 8192,
  "model_type": "reformer",
  "num_attention_heads": 2,
  "num_buckets": [
    64,
    128
  ],
  "num_chunks_after": 0,
  "num_chunks_before": 1,
  "num_hashes": 1,
  "num_hidden_layers": 6,
  "output_past": true,
  "pad_token_id": 0,
  "task_specific_params": {
    "text-generation": {
      "do_sample": true,
      "max_length": 100
    }
  },
  "vocab_size": 320
}

Python Code:
config = ReformerConfig()
config.max_position_embeddings = 8192
config.axial_pos_shape=[64, 128]

#config = ReformerConfig.from_pretrained('./cnp/config.json', output_attention=True)

model = ReformerForSequenceClassification(config)
model.load_state_dict(torch.load("./cnp/pytorch_model.bin"))


title: Training works but prediction produces constant values (cnn with pytorch)

body:
I have a model trying to predict the class of image: cat or dog. I receive 95% accuracy in training. However when I try to predict a single image, I am stuck with almost constant output every time I run the model. There are some non-constant values, but they mostly look like catastrophic failure.
I read similar topics from forums but that hasn't helped, as it appears there is no particular solution for this problem...
I have tried the following:

Changing epochs 5 to 15,20,30...
Changing lr = 0.001 to 0.01, 0.0001...
I implemented with both dropout regularization model and batch
normalization model...
I changed testing pictures...
Changing last activation layer from softmax to torch.sigmoid...
Reducing batch size from 100 to 30, 75...
Trying with a batch, which results with normal acc, loss and
predictions.
My dataset is scaled which is mentioned in forums as solution.
My optim is Adam which is mentioned in forums as solution.
Loading dataset with torch.data.DataLoader...
Sampling randomly...
I saved and load the model, in case there are problems with that.
However, I already checked that state_dict's are different...
I re-prepared data which resulted the constant value to change
otherwise (dog to cat), somehow? Idk if that's a coincidence though.

Infos:

Dataset :
https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip

Here is all my code with predictions in Jupyter Notebook, feel free to investigate. I am really tired of this solution. Any help is  highly appreciated!

https://github.com/yusuftengriverdi/neural_networks/blob/master/CNN_Last.ipynb

Similar topics around the web:

https://discuss.pytorch.org/t/rnn-predicting-a-constant-output/40397/5
https://discuss.pytorch.org/t/cnn-does-not-predict-properly-does-not-converge-as-expected/43567
https://discuss.pytorch.org/t/making-a-prediction-with-a-trained-model/2193
https://datascience.stackexchange.com/questions/46779/predict-gives-the-same-output-value-for-every-image-keras
https://github.com/keras-team/keras/issues/6447
PyTorch model prediction fail for single item
Having trouble with CNN prediction


title: How is the print and view functions works in pytorch?

body:
This is a convolutional neural network which I found in the web

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(500, 50)
        self.fc2 = nn.Linear(50, 64)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 500)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x)

and its summary

print(net)
Net(
  (conv1): Conv2d(3, 10, kernel_size=(5, 5), stride=(1, 1))
  (conv2): Conv2d(10, 20, kernel_size=(5, 5), stride=(1, 1))
  (conv2_drop): Dropout2d(p=0.5)
  (fc1): Linear(in_features=500, out_features=50, bias=True)
  (fc2): Linear(in_features=50, out_features=64, bias=True)
)

What is x.view does? Is it similar to the Flatten function in keras. The other query is reagarding how pytorch prints summary of a model. Eventhough the model uses two dropouts nn.Dropout2d() and F.dropout. When printing the model we can see only one (conv2_drop): Dropout2d(p=0.5), why?. The last question is why pytorch  dosen't print F.max_pool2d layer?


I was trying to train a DCGAN model using MNIST datasets, but I can't load the gen.state_dict() after I finished training.
import torch
import torch.nn as nn
import torchvision.datasets as datasets
import torchvision.transforms as transforms
from torch.utils.tensorboard import SummaryWriter
import torchvision
import os
from torch.autograd import Variable

workspace_dir = '/content/drive/My Drive/practice'
device=torch.device('cuda' if torch.cuda.is_available else 'cpu')
print(device)

img_size=64
channel_img=1
lr=2e-4
batch_size=128
z_dim=100
epochs=10
features_gen=64
features_disc=64
save_dir = os.path.join(workspace_dir, 'logs')
os.makedirs(save_dir, exist_ok=True)
import matplotlib.pyplot as plt
transforms=transforms.Compose([transforms.Resize(img_size),transforms.ToTensor(),transforms.Normalize(mean=(0.5,),std=(0.5,))])
train_data=datasets.MNIST(root='dataset/',train=True,transform=transforms,download=True)
train_loader=torch.utils.data.DataLoader(train_data,batch_size=batch_size,shuffle=True)
count=0
for x,y in train_loader:
  if count==5:
    break
  print(x.shape,y.shape)
  count+=1

class Discriminator(nn.Module):
  def __init__(self,channels_img,features_d):
    super(Discriminator,self).__init__()

    self.disc=nn.Sequential(
        #input:N * channels_img * 64 *64
        nn.Conv2d(channels_img,features_d,4,2,1),#paper didn't use batchnorm in the early layers in the discriminator features_d* 32 *32
        nn.LeakyReLU(0.2),
        self._block(features_d,features_d*2,4,2,1),#features_d*2 *16 *16
        self._block(features_d*2,features_d*4,4,2,1),#features_d*4 *8 *8
        self._block(features_d*4,features_d*8,4,2,1), #features_d*8 *4 *4
        nn.Conv2d(features_d*8,1,4,2,0),#1 * 1 *1
        nn.Sigmoid()

    )
  def _block(self,in_channels,out_channels,kernel_size,stride,padding):
    return nn.Sequential(
        nn.Conv2d(in_channels,out_channels,kernel_size,stride,padding,bias=False),
        nn.BatchNorm2d(out_channels),
        nn.LeakyReLU(0.2)
    )

  def forward(self,x):
    return self.disc(x)

class Generator(nn.Module):
  def __init__(self,Z_dim,channels_img,features_g):
    super(Generator,self).__init__()

    self.gen=nn.Sequential(
        #input :n * z_dim * 1 *1
        self._block(Z_dim,features_g*16,4,1,0),#features_g*16 * 4 * 4
        self._block(features_g*16,features_g*8,4,2,1),#features_g*8 * 8 * 8
        self._block(features_g*8,features_g*4,4,2,1),#features_g*4 * 16 * 16
        self._block(features_g*4,features_g*2,4,2,1),#features_g*2 * 32 * 32
        nn.ConvTranspose2d(features_g*2,channels_img,4,2,1), #
        nn.Tanh()# [-1 to 1] normalize the image
    )
  def _block(self,in_channels,out_channels,kernel_size,stride,padding):
    return nn.Sequential(
          nn.ConvTranspose2d(in_channels,out_channels,kernel_size,stride,padding,bias=False),#w'=(w-1)*s-2p+k
          nn.BatchNorm2d(out_channels),
          nn.ReLU()
      )

  def forward(self,x):
      return self.gen(x)

def initialize_weights(model):
  for m in model.modules():
    if isinstance(m,(nn.Conv2d,nn.ConvTranspose2d,nn.BatchNorm2d)):
      nn.init.normal_(m.weight.data,0.0,0.02)

gen=Generator(z_dim,channel_img,features_gen).to(device)
disc=Discriminator(channel_img,features_disc).to(device)
initialize_weights(gen)
initialize_weights(disc)
opt_gen=torch.optim.Adam(gen.parameters(),lr=lr,betas=(0.5,0.999))
opt_disc=torch.optim.Adam(disc.parameters(),lr=lr,betas=(0.5,0.999))
criterion=nn.BCELoss()
#fixed_noise=torch.randn(32,z_dim,1,1).to(device)
#writer_real=SummaryWriter(f"logs/real")
#writer_fake=SummaryWriter(f"logs/fake")
step=0
gen.train()
disc.train()

z_sample = Variable(torch.randn(100, z_dim,1,1)).cuda()
for epoch in range(2):
  for batch_idx,(real,_) in enumerate(train_loader):
    real=real.to(device)
    noise=torch.randn((batch_size,z_dim,1,1)).to(device)
    fake=gen(noise)

    #Train Discriminator max log(D(x)) + log(1-D(G(z)))
    disc_real=disc(real).reshape(-1)
    loss_disc_real=criterion(disc_real,torch.ones_like(disc_real))
    disc_fake=disc(fake).reshape(-1)
    loss_disc_fake=criterion(disc_fake,torch.zeros_like(disc_fake))
    loss_disc=(loss_disc_fake+loss_disc_real)/2
    disc.zero_grad()
    loss_disc.backward(retain_graph=True)
    opt_disc.step()

    #Train Generator  min log(1-D(G(z))) <--> max log(D(G(z)))
    output=disc(fake).reshape(-1)
    loss_gen=criterion(output,torch.ones_like(output))
    gen.zero_grad()
    loss_gen.backward()
    opt_gen.step()

    print(f'\rEpoch [{epoch+1}/{3}] {batch_idx+1}/{len(train_loader)} Loss_D: {loss_disc.item():.4f} Loss_G: {loss_gen.item():.4f}', end='')
  gen.eval()
  f_imgs_sample = (gen(z_sample).data + 1) / 2.0
  filename = os.path.join(save_dir, f'Epoch_{epoch+1:03d}.jpg')
  torchvision.utils.save_image(f_imgs_sample, filename, nrow=10)
  print(f' | Save some samples to {filename}.')
  # show generated image
  grid_img = torchvision.utils.make_grid(f_imgs_sample.cpu(), nrow=10)
  plt.figure(figsize=(10,10))
  plt.imshow(grid_img.permute(1, 2, 0))
  plt.show()
  gen.train()

  torch.save(gen.state_dict(), os.path.join(workspace_dir, f'dcgan_d.pth'))
  torch.save(disc.state_dict(), os.path.join(workspace_dir, f'dcgan_g.pth'))

I can't load the gen state_dict in this step:
# load pretrained model
#gen = Generator(z_dim,1,64)
gen=Generator(z_dim,channel_img,features_gen).to(device)
gen.load_state_dict(torch.load('/content/drive/My Drive/practice/dcgan_g.pth'))
gen.eval()
gen.cuda()

Here's the error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-4bda27faa444> in <module>()
      5
      6 #gen.load_state_dict(torch.load('/content/drive/My Drive/practice/dcgan_g.pth'))
----> 7 gen.load_state_dict(torch.load(os.path.join(workspace_dir, 'dcgan_g.pth')))
      8 #/content/drive/My Drive/practice/dcgan_g.pth
      9 gen.eval()

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
   1050         if len(error_msgs) > 0:
   1051             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
-> 1052                                self.__class__.__name__, "\n\t".join(error_msgs)))
   1053         return _IncompatibleKeys(missing_keys, unexpected_keys)
   1054

***RuntimeError: Error(s) in loading state_dict for Generator:
    Missing key(s) in state_dict***: "gen.0.0.weight", "gen.0.1.weight", "gen.0.1.bias", "gen.0.1.running_mean", "gen.0.1.running_var", "gen.1.0.weight", "gen.1.1.weight", "gen.1.1.bias", "gen.1.1.running_mean", "gen.1.1.running_var", "gen.2.0.weight", "gen.2.1.weight", "gen.2.1.bias", "gen.2.1.running_mean", "gen.2.1.running_var", "gen.3.0.weight", "gen.3.1.weight", "gen.3.1.bias", "gen.3.1.running_mean", "gen.3.1.running_var", "gen.4.weight", "gen.4.bias".
    Unexpected key(s) in state_dict: "disc.0.weight", "disc.0.bias", "disc.2.0.weight", "disc.2.1.weight", "disc.2.1.bias", "disc.2.1.running_mean", "disc.2.1.running_var", "disc.2.1.num_batches_tracked", "disc.3.0.weight", "disc.3.1.weight", "disc.3.1.bias", "disc.3.1.running_mean", "disc.3.1.running_var", "disc.3.1.num_batches_tracked", "disc.4.0.weight", "disc.4.1.weight", "disc.4.1.bias", "disc.4.1.running_mean", "disc.4.1.running_var", "disc.4.1.num_batches_tracked", "disc.5.weight", "disc.5.bias".


How can I fix this? I am using the code from this GitHub repo: https://github.com/benjiebob/SMALViewer
(base) mona@mona:~/research/3danimals/SMALViewer$ python smal_viewer.py
dict_keys(['f', 'J_regressor', 'kintree_table', 'weights', 'posedirs', 'v_template', 'shapedirs'])
torch.Size([4, 3889])
 Tensor J_regressor shape:  torch.Size([37, 3889])
 Tensor weights shape:  torch.Size([3889, 33])
 Tensor posedirs shape:  torch.Size([3889, 3, 288])
 Tensor v_template shape:  torch.Size([3889, 3])
 Tensor shapedirs shape:  torch.Size([3889, 3, 41])
 Tensor faces shape:  torch.Size([7774, 3])
Traceback (most recent call last):
  File "smal_viewer.py", line 13, in <module>
    main()
  File "smal_viewer.py", line 6, in main
    main_window = pyqt_viewer.MainWindow()
  File "/home/mona/research/3danimals/SMALViewer/pyqt_viewer.py", line 50, in __init__
    self.setup_ui()
  File "/home/mona/research/3danimals/SMALViewer/pyqt_viewer.py", line 219, in setup_ui
    self.update_render()
  File "/home/mona/research/3danimals/SMALViewer/pyqt_viewer.py", line 241, in update_render
    rendered_images, rendered_silhouettes, rendered_joints, verts, joints_3d = self.model_renderer(self.smal_params)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/mona/research/3danimals/SMALViewer/smal/smal3d_renderer.py", line 45, in forward
    rendered_joints = self.renderer.render_points(joints_3d[:, self.smal_info.include_classes])
  File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 772, in __getattr__
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'Renderer' object has no attribute 'render_points'

$ python
Python 3.7.6 (default, Jan  8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.6.0'
>>> torch.version.cuda
'10.1'
>>> torch.cuda.is_available()
True
>>> torch.backends.cudnn.version()
7603
>>> torch.backends.cudnn.enabled
True

$ lsb_release -a
LSB Version:    core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:    20.04
Codename:   focal

https://github.com/benjiebob/SMALViewer/issues/4
also:
(base) mona@mona:~/research/3danimals/SMALViewer$ cat smal/smal3d_renderer.py
import sys, os
sys.path.append(os.path.dirname(sys.path[0]))

import torch
import torch.nn as nn
import neural_renderer as nr
import torch.nn.functional as F
from SMPL.smal_torch_batch import SMALModel
from smal.joint_catalog import SMALJointInfo

import pickle as pkl

import numpy as np
import cv2
import matplotlib.pyplot as plt

class SMAL3DRenderer(nn.Module):
    def __init__(self, image_size, z_distance = 2.5, elevation = 89.9, azimuth = 0.0):
        super(SMAL3DRenderer, self).__init__()

        self.smal_model = SMALModel()
        self.image_size = image_size
        self.smal_info = SMALJointInfo()

        self.renderer = nr.Renderer(camera_mode='look_at')
        self.renderer.eye = nr.get_points_from_angles(z_distance, elevation, azimuth)

        self.renderer.image_size = image_size
        self.renderer.light_intensity_ambient = 1.0

        with open("smal/dog_texture.pkl", 'rb') as f:
            self.textures = pkl.load(f).cuda()

    def forward(self, batch_params):
        batch_size = batch_params['betas'].shape[0]

        verts, joints_3d = self.smal_model(
            batch_params['betas'],
            torch.cat((batch_params['global_rotation'], batch_params['joint_rotations']), dim = 1),
            batch_params['trans'])

        faces = self.smal_model.faces.unsqueeze(0).expand(batch_size, -1, -1)
        textures = self.textures.unsqueeze(0).expand(batch_size, -1, -1, -1, -1, -1)

        rendered_joints = self.renderer.render_points(joints_3d[:, self.smal_info.include_classes])
        rendered_silhouettes = self.renderer.render_silhouettes(verts, faces)
        rendered_silhouettes = rendered_silhouettes.unsqueeze(1)

        rendered_images = self.renderer.render(verts, faces, textures)
        rendered_images = torch.clamp(rendered_images[0], 0.0, 1.0)

        return rendered_images, rendered_silhouettes, rendered_joints, verts, joints_3d(base)


title: Using CNN for pixel classification,how?

body:
I have already learn about some classification using CNN like for Mnist. But recently I received a dataset which is consist of a vector set. The normal image dataset(mnist) is like nxcxwxh. The one I received is (w*h)x1xc. The goal is to train a network to classify these pixels(as I understand,is classification for pixels). The label length is groundtruth picture.

I am a little confused about this work. As I understand, for Image processing, we use the CNN with different receiption feld to make the convolution operations so that a feature can be obtained to represent the image. But in this case, the image is already expand to pixel-set. Why is the convolutional neural network still suitable?

Still I am not sure about the work but I started to try.I used the 1d convolution instead of 2d in the network. After 4-Conv1d,the output is connected to a softmax layer then fed to crossentropy loss function. It seems, I have some problems with the output dimensions so the network is not able to train.

I use pytorch to implement the work. Below is the network form I try to build. The dimensions do not match those need for crossentropyloss function. 122500 was set to be the sample numbers. So I think the convolution was performed along the 1-200 directions.

First I want to know,is that right to implement like this using conv1d when I want to classify the pixels?

If this thought was right, how can I continue to feed the features to the loss function?

If this is wrong,can I have some similar examples for this kind of work? I am new to python, so if there were some stupid mistakes, pls point out.

Thanks all.

class network(nn.Module):
"""
    Building network

"""
def __init__(self):
    super(network, self).__init__()
    self.conv1 = nn.Conv1d(in_channels = 1,out_channels = 32,stride = 1,kernel_size = 3)
    self.conv2 = nn.Conv1d(in_channels = 32,out_channels = 64,stride = 1,kernel_size = 3)
    self.conv3 = nn.Conv1d(in_channels = 64,out_channels = 128,stride = 1,kernel_size = 3)
    self.conv4 = nn.Conv1d(in_channels = 128,out_channels = 256,stride = 1,kernel_size = 3)
    self.fc = nn.Linear(13, 2)

def forward(self,s):
    s = self.conv1(s)
    s = F.relu(F.max_pool1d(s, 2))
    s = self.conv2(s)
    s = F.relu(F.max_pool1d(s, 2))
    s = self.conv3(s)
    s = F.relu(F.max_pool1d(s, 2))
    s = self.conv4(s)
    s = F.relu(F.max_pool1d(s, 2))

    s = self.fc(s)

    s = F.softmax(s,1)

output = model(input)
loss = loss_fn(output, labels)


title: do I need to initialize lstm hidden state when in validation and test set? or just reset it to zero

body: On training, it is good to initialize hidden state instead of setting it  to 0. But I wonder whether initialize hidden state on validation and testing is good or bad. Thanks


title: Python get pytorch tensor size

body:
I wanna know how to get the shape of this tensor in Python ? I have tried this :

> len(x)

But this prints 1, why ? I want to print the number of tuples here which is 3. Using len(x) prints only 1.

What's the problem ?

Here's the tensor :

 (x=array([[[[ 0.07499999,  0.        ],
         [ 0.0703125 ,  0.        ],
         [ 0.0703125 ,  0.        ],
         [ 0.09218752,  0.        ],
         [ 0.1953125 ,  0.        ],
         [ 0.05312502,  0.        ],
         [ 0.2890625 ,  0.        ],
         [ 0.015625  ,  0.        ],
         [ 0.32656252,  0.        ],
         [ 0.09218752,  0.        ],
         [ 0.23906249,  0.        ],
         [ 0.09218752,  0.        ],
         [ 0.22812498,  0.        ],
         [ 0.06406248,  0.        ],
         [ 0.19062501,  0.        ],
         [ 0.02031249,  0.        ],
         [ 0.17343748,  0.        ]],

        [[ 0.06875002,  0.        ],
         [ 0.06875002,  0.        ],
         [ 0.06875002,  0.        ],
         [ 0.09062499,  0.        ],
         [ 0.19375002,  0.        ],
         [ 0.05781251,  0.        ],
         [ 0.2921875 ,  0.        ],
         [ 0.01406252,  0.        ],
         [ 0.325     ,  0.        ],
         [ 0.08437502,  0.        ],
         [ 0.23124999,  0.        ],
         [ 0.09531248,  0.        ],
         [ 0.22031248,  0.        ],
         [ 0.06406248,  0.        ],
         [ 0.18906248,  0.        ],
         [ 0.02031249,  0.        ],
         [ 0.171875  ,  0.        ]],

        [[ 0.06718749,  0.        ],
         [ 0.06093752,  0.        ],
         [ 0.07187498,  0.        ],
         [ 0.078125  ,  0.        ],
         [ 0.18593752,  0.        ],
         [ 0.03437501,  0.        ],
         [ 0.2765625 ,  0.        ],
         [-0.00312501,  0.        ],
         [ 0.29843748,  0.        ],
         [ 0.078125  ,  0.        ],
         [ 0.21718752,  0.        ],
         [ 0.078125  ,  0.        ],
         [ 0.21249998,  0.        ],
         [ 0.07187498,  0.        ],
         [ 0.19062501,  0.        ],
         [ 0.13749999,  0.        ],
         [ 0.1796875 ,  0.        ]]]], dtype=float32), 0)


Help me understand few things, im new with python and all these depended libraries.
I m trying to run a project which is written in python. Git repository can be found here: https://github.com/ifzhang/FairMOT
On anaconda prompt I have been running these commands:
conda create -n FairMOT
conda activate FairMOT
conda install pytorch==1.2.0 torchvision==0.4.0 cudatoolkit=10.0 -c pytorch
cd ${FAIRMOT_ROOT}
pip install -r requirements.txt

All being successfully installed
I have performed all the steps that were listed in the Readme file
Now there is training step which requires to run shell script.
Assuming the fact that shell script would not be called on anaconda prompt, I switched to git bash, and I run the script (keeping in mind of path)
 sh experiments/crowdhuman_dla34.sh

it throws error
Sanam@LAPTOP-NPVR76P7 MINGW64 /f/NTNU/Deep learning/Repositories/FairMOT (master)
$ sh experiments/crowdhuman_dla34.sh
Traceback (most recent call last):
File "train.py", line 10, in <module>
import torch
ModuleNotFoundError: No module named 'torch'

Question is: is there any other console where I need to run this command other than git bash? it would definitely not work on anaconda prompt. what im doing wrong?
coming back to anaconda prompt, when I test import there, it worked
  (FairMOT) F:\NTNU\Deep learning\Repositories\FairMOT>python
  Python 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on
  win32
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import torch
  >>>

where & how do I run this command? so that it would work?is running on git bash not good?
thanks in advance!!
PS:
I have already tried reinstalling, activation/ deactivation of environment but it does not work


title: Row-wise Element Indexing in PyTorch for C++

body:
I am using the C++ frontend for PyTorch and am struggling with a relatively basic indexing problem.

I have an 8 by 6 Tensor such as the one below:

[ Variable[CUDAFloatType]{8,6} ]
                 0           1           2           3           4           5
0       1.7107e-14  4.0448e-17  4.9708e-06  1.1664e-08  9.9999e-01  2.1857e-20
1       1.8288e-14  5.9356e-17  5.3042e-06  1.2369e-08  9.9999e-01  2.4799e-20
2       2.6828e-04  9.0390e-18  1.7517e-02  1.0529e-03  9.8116e-01  6.7854e-26
3       5.7521e-10  3.1037e-11  1.5021e-03  1.2304e-06  9.9850e-01  1.4888e-17
4       1.7811e-13  1.8383e-15  1.6733e-05  3.8466e-08  9.9998e-01  5.2815e-20
5       9.6191e-06  2.6217e-23  3.1345e-02  2.3024e-04  9.6842e-01  2.9435e-34
6       2.2653e-04  8.4642e-18  1.6085e-02  9.7405e-04  9.8271e-01  6.3059e-26
7       3.8951e-14  2.9903e-16  8.3518e-06  1.7974e-08  9.9999e-01  3.6993e-20

I have another Tensor with just 8 elements in it such as:

[ Variable[CUDALongType]{8} ]
 0
 3
 4
 4
 4
 4
 4
 4

I would like to index the rows of my first tensor using the second to produce:

        0
0       1.7107e-14
1       1.2369e-08
2       9.8116e-01
3       9.9850e-01
4       9.9998e-01
5       9.6842e-01
6       9.8271e-01
7       9.9999e-01

I have tried a few different approaches including index_select but it seems to produce an output that has the same dimensions as the input (8x6).

In Python I think I could index with Python's built-in indexing as discussed here: https://github.com/pytorch/pytorch/issues/1080

Unfortunately, in C++ I can only index a Tensor with a scalar (zero-dimensional Tensor) so I don't think that approach works for me here.

How can I achieve my desired result without resorting to loops?


title: Load multiple .npy files (size > 10GB) in pytorch

body:
Im looking for a optimized solution to load multiple huge .npy files using pytorch data loader.
I'm currently using the following method which creates a new dataloader for each file in each epoch.

My data loader is something like:

class GetData(torch.utils.data.Dataset):

    def __init__(self, data_path, target_path, transform=None):
        with open(data_path, 'rb') as train_pkl_file:
            data = pickle.load(train_pkl_file)
            self.data = torch.from_numpy(data).float()
        with open(target_path, 'rb') as target_pkl_file:
            targets = pickle.load(target_pkl_file)
            self.targets = torch.from_numpy(targets).float()

    def __getitem__(self, index):
        x = self.data[index]
        y = self.targets[index]
        return index, x, y

    def __len__(self):
        num_images = self.data.shape[0]
        return num_images

I have a npy list of files:

list1 = ['d1.npy', 'd2.npy','d3.npy']
list1 = ['s1.npy', 's2.npy','s3.npy']

I have created a dataloader which gives the filenames

class MyDataset(torch.utils.data.Dataset):
    def __init__(self,flist):
        self.npy_list1 = flist1
        self.npy_list2 = flist2

    def __getitem__(self, idx):
        filename1 = self.npy_list1[idx]
        filename2 = self.npy_list2[idx]
        return filename1,filename2

    def __len__(self):
        return len(self.npy_list1)

And I itreate through them as follows:

for epoch in range(500):
    print('Epoch #%s' % epoch)
    model.train()
    loss_, elbo_, recon_ = [[] for _ in range(3)]
    running_loss = 0

    # FOR EVERY SMALL FILE
    print("Training: ")

    # TRAIN HERE
    my_dataset = MyDataset(npyList)
    for idx, (dynamic_file, static_file) in tqdm(enumerate(my_dataset)):
         ...Do stuff ....

The above method works but i'm looking for more memory efficient solution. Note: I have huge amount of data > 200 GB so concatenating the numpy arrays into 1 file may not be the solution (due to RAM limitations).
Thanks in advance


title: Linear regression with pytorch

body:
I tried to run linear regression on ForestFires dataset.
Dataset is available on Kaggle and gist of my attempt is here:
https://gist.github.com/Chandrak1907/747b1a6045bb64898d5f9140f4cf9a37

I am facing two problems:

Output from prediction is of shape 32x1 and target data shape is 32.

  input and target shapes do not match: input [32 x 1], target [32]¶

Using view I reshaped predictions tensor.

  y_pred = y_pred.view(inputs.shape[0])

Why there is a mismatch in shapes of predicted tensor and actual tensor?

SGD in pytorch never converges. I tried to compute MSE manually using

  print(torch.mean((y_pred - labels)**2))

This value does not match

  loss = criterion(y_pred,labels)

Can someone highlight where is the mistake in my code?

Thank you.


title: PyTorch Linear Algebra Gradients

body:
I'm looking to back-propagate gradients through a singular value decomposition for regularisation purposes. PyTorch currently does not support backpropagation through a singular value decomposition.
I know that I could write my own custom function that operates on a Variable; takes its .data tensor, applies the torch.svd to it, wraps a Variable around its singular values and returns it in the forward pass, and in the backward pass applies the appropriate Jacobian matrix to the incoming gradients.
However, I was wondering whether there was a more elegant (and potentially faster) solution, where I could overwrite the "Type Variable doesn't implement stateless method svd" Error directly, call Lapack, etc. ?
If someone could guide me through the appropriate steps and source files I need to look at, I'd be very grateful. I suppose these steps would similarly apply to other linear algebra operations which have no associated backward method currently.


title: Why does pytorch F.mse_loss behave differently w.r.t. Tensor and Parameter?

body:
Below is my code:

import torch as pt
from torch.nn import functional as F

a = pt.Tensor([[0, 1], [2, 3]])
b = pt.Tensor([[1, 0], [5, 4]])
print(F.mse_loss(a, b), F.mse_loss(a, b, reduction='elementwise_mean'))

a = pt.nn.Parameter(a)
b = pt.nn.Parameter(b)
print(F.mse_loss(a, b), F.mse_loss(a, b, reduction='elementwise_mean'))

The output was:

tensor(3.) tensor(3.)
tensor(12., grad_fn=<SumBackward0>) tensor(12., grad_fn=<SumBackward0>)

I wonder why they gave two different results?

Environment setting:
python 3.6
pytorch 0.4.1


title: Padding in Pytorch

body:
In PyTorch tensor, I would like to obtain the output from the input as follows:

How can I achieve this padding in Pytroch?


title: Training a model with multiple learning rate in PyTorch

body:
I am new to PyTorch and getting used to some concepts.

I need to train a Neural Network. For optimization, I need to use Adam optimizer with 4 different learning rates = [2e-5, 3e-5, 4e-5, 5e-5]

The optimizer function is defined as below

def optimizer(no_decay = ['bias', 'gamma', 'beta'], lr=2e-5):
    param_optimizer = list(model.named_parameters())
    optimizer_grouped_parameters = [
    {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
     'weight_decay_rate': 0.01},
    {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
     'weight_decay_rate': 0.0}
     ]
     # This variable contains all of the hyperparemeter information our training loop needs
     optimizer = BertAdam(optimizer_grouped_parameters, lr, warmup=.1)
     return optimizer

How do I make sure the optimizer uses my specified set of learning rate and returns the best model?

During training, we use the optimizer like below where I don't see a way to tell it to try different learning rate

def model_train():
    #other code
    # clear out the gradient
    optimizer.zero_grad()
    # Forward pass
    loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
    train_loss_set.append(loss.item())
    # Backward pass
    loss.backward()
    # Update parameters and take a step using the computed gradient
    optimizer.step()

I know that the optimizer.step() internally steps through to optimize the gradient. But how do I make sure the optimizer tries my specified set of learning rate and returns the best model to me?

Please suggest.


title: How to change max sequence length for transformers.bert?

body:
I download bert-base pretrained model. I edit the config.json (from 512 to 256)
 "max_position_embeddings": 256,

Then I want to use bert model,
from transformers import BertForSequenceClassification
model = BertForSequenceClassification.from_pretrained(
    MODEL_PATH,
    num_labels = 2, # The number of output labels--2 for binary classification.
    output_attentions = False,
    output_hidden_states = False,
)
# Tell pytorch to run this model on the GPU.
model.cuda()

But it raise an error
Error(s) in loading state_dict for BertForSequenceClassification:
    size mismatch for bert.embeddings.position_embeddings.weight: copying a param with shape torch.Size([512, 768]) from checkpoint, the shape in current model is torch.Size([256, 768]).

I know the reason is because I change the max sequence length.  What is the right way, if I want to change the max seq lenght?


title: RuntimeError with copy_() and expand() - Unsupported operation: more than one element of the written-to tensor refers to a single memory location

body:
I'm migrating a repository from Pytorch Nightly 1.0.0 to 1.3.1. Stripping the unnecessary details, it is basically performing the following sequence of operations:

mu = torch.tensor(0.005)
bar = torch.eye(5, 5)
foo = torch.eye(5).expand(5, 5, 5)

# update
bar.copy_(mu * bar)  # ok!
foo.copy_(mu * foo)  # error

bar.copy_(mu * bar) works, while when I try to foo.copy_() the result, it gives the following error:

  RuntimeError: unsupported operation: more than one element of the written-to tensor refers to a single memory location. Please clone() the tensor before performing the operation.


title: cmake - linking static library pytorch cannot find its internal functions during build

body:
I'm trying to build a program using cmake. For several reasons, the program must be built using static libraries rather than dynamic libraries, and I need to use PyTorch so this is what I've done:

Downloaded and installed PyTorch static library (I've found libtorch.a in the proper path, in /home/me/pytorch/torch/lib)
Made CMakeLists.txt with the following contents:

cmake_minimum_required(VERSION 3.5.1 FATAL_ERROR)
project(example-app LANGUAGES CXX)
find_package(Torch REQUIRED)
add_executable(example-app example-app.cpp argparse/argparse.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}" -static -fopenmp)
set_property(TARGET example-app PROPERTY CXX_STANDARD 14)

FYI, example-app.cpp is the file with the main function, and argparse/ is a directory with some source code for functions called in example-app.cpp

It works until cmake -DCMAKE_PREFIX_PATH=/home/me/pytorch/torch .., but the following build incurs some errors, saying it could not find the reference to some functions, namely functions starting with fbgemm::. fbgemm is (as long as I know) some sort of GEMM library used in implementing PyTorch.

It seems to me that while linking the static PyTorch library, its internal libraries like fbgemm stuff have not been linked properly, but I'm not an expert on cmake and honestly not entirely sure.

Am I doing something wrong, or is there a workaround for this problem? Any help or push in the right direction would be greatly appreciated.

P.S.

The exact error has not been posted because it is way too long, but it consists of mostly undefined reference to ~ errors. If looking at the error message might be helpful for some people, I'd be happy to edit the question and post it.
building and running the file works fine if I remove the parts that require the library's functions from the code without commenting out #include <torch/torch.h> from example-app.cpp.


title: Initialising weights and bias with PyTorch - how to correct dimensions?

body:
Using this model I'm attempting to initialise my network with my predefined weights and bias :

dimensions_input = 10
hidden_layer_nodes = 5
output_dimension = 10

class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear = torch.nn.Linear(dimensions_input,hidden_layer_nodes)
        self.linear2 = torch.nn.Linear(hidden_layer_nodes,output_dimension)

        self.linear.weight = torch.nn.Parameter(torch.zeros(dimensions_input,hidden_layer_nodes))
        self.linear.bias = torch.nn.Parameter(torch.ones(hidden_layer_nodes))

        self.linear2.weight = torch.nn.Parameter(torch.zeros(dimensions_input,hidden_layer_nodes))
        self.linear2.bias = torch.nn.Parameter(torch.ones(hidden_layer_nodes))

    def forward(self, x):
        l_out1 = self.linear(x)
        y_pred = self.linear2(l_out1)
        return y_pred

model = Model()

criterion = torch.nn.MSELoss(size_average = False)
optim = torch.optim.SGD(model.parameters(), lr = 0.00001)

def train_model():
    y_data = x_data.clone()
    for i in range(10000):
        y_pred = model(x_data)
        loss = criterion(y_pred, y_data)

        if i % 5000 == 0:
            print(loss)
        optim.zero_grad()

        loss.backward()
        optim.step()

RuntimeError:

  The expanded size of the tensor (10) must match the existing size (5)
  at non-singleton dimension 1

My dimensions appear correct as they match the corresponding linear layers ?


title: In PyTorch, how to I make certain module `Parameters` static during training?

body:
Context:

In pytorch, any Parameter is a special kind of Tensor. A Parameter is automatically registered with a module's parameters() method when it is assigned as an attribute.

During training, I will pass m.parameters() to the Optimizer instance so they can be updated.

Question: For a built-in pytorch module, how to I prevent certain parameters from being modified by the optimizer?

s = Sequential(
        nn.Linear(2,2),
        nn.Linear(2,3),   # I want this one's .weight and .bias to be constant
        nn.Linear(3,1)
    )

Can I make it so they don't appear in s.parameters()?
Can I make the parameters read-only so any attempted changes are ignored?