title: Object Detection Model (PyTorch) to ONNX:empty output by ONNX inference

body:
I try to convert my PyTorch object detection model (Faster R-CNN) to ONNX. I have two setups. The first one is working correctly but I want to use the second one for deployment reasons. The difference lies in the example image which I use for the export of the function torch.onnx.export().

In the first setup I use a real image as input for the ONNX export. But in a official tutorial they say that I can use a dummy input, which should have the same size as the model expects the input. So I created a tensor with the same shape but with random values. The export in both setups is working correctly. But the second setup does not deliver the desired results after inference with the ONNX runtime. The code and the exemplary output can be found below.

Setup 1

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained = True)
...
checkpoint = torch.load(model_state_dict_path)
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

to_tensor = transforms.ToTensor()
img_rgb = Image.open(image_path_model).convert('RGB')
img_rgb = to_tensor(img_rgb)
img_rgb.unsqueeze_(0)

torch.onnx.export(model, img_rgb, "detection.onnx", opset_version=11)

I get no error and the export works. Afterwards I run the model with the ONNX runtime and I get the following output:

[array([[704.0696  , 535.19556 , 944.8986  , 786.1619  ],
         ...], dtype=float32),
array([2, 2, 2, 2, 2, 1, 1], dtype=int64),
array([0.9994363 , 0.9984769 , 0.99816966, ...], dtype=float32)]

The output is as I expect it to be (Bounding boxes, object classes and probabilities).

Setup 2

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained = True)
...
checkpoint = torch.load(model_state_dict_path)
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

img_rgb = torch.randn(1, 3, 1024, 1024)

torch.onnx.export(model, img_rgb, "detection.onnx", opset_version=11)

Like in setup 1 I get no error and the export works. Afterwards I run the model with the ONNX runtime and with the same image as in setup 1 and I get the following output:

[array([], shape=(0, 4), dtype=float32),
array([], dtype=int64),
array([], dtype=float32)]

It is just an empty array.

What is wrong with the second setup? I am new to ONNX. The export runs the model. Do I have to provide an input on which the model also recognizes objects and therefore the dummy input with random values does not work? Is the statement "The values in this can be random as long as it is the right type and size." only valid for the provided tutorial?


I am following this online tutorial for coding a DQN,https://github.com/philtabor/Youtube-Code-Repository/blob/master/ReinforcementLearning/DeepQLearning/torch_deep_q_model.py
, however I am running into this Runtime Error that I am unsure of how to debug or modify to prevent this error. Thanks!

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-196-00975d66fd2d> in <module>
     28         agent.storeTransition(preprocess(obs),action,reward,preprocess(obs_))
     29         obs= obs_
---> 30         agent.learn(batch_size)
     31         lastAction = action
     32     scores.append(score)

<ipython-input-191-f6b163cc3a8a> in learn(self, batch_size)
     72         Qtarget = Qpred.clone()
     73         print(Qnext[1])
---> 74         Qtarget[:,maxA] = rewards + self.GAMMA*torch.max(Qnext[1])
     75         # epsilon decay action
     76         if self.steps > 2000:

RuntimeError: the derivative for 'indices' is not implemented

These are my code blocks in my jupyter notebook

class DeepQNetwork(nn.Module):
    def __init__(self,Alpha):
        super(DeepQNetwork,self).__init__()
        self.conv1 = nn.Conv2d(1,32,8,stride=4, padding=1)
        self.conv2 = nn.Conv2d(32,64,4,stride=2)
        self.conv3 = nn.Conv2d(64,128,3)
        self.fc1 = nn.Linear(128* 21* 12,512)
        self.fc2 = nn.Linear(512,6)

        self.optimizer = optim.RMSprop(self.parameters(), lr = Alpha)
        self.loss = nn.MSELoss()
        self.device =  torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
        self.to(self.device)

    def forward(self,obs):
        '''Passing in a sequence of arrays'''
        obs = torch.Tensor(obs).to(self.device) # send to the GPU
        ''' Feed forward the Network Parameters'''
        obs = obs.view(-1, 1,200,125)
        #print(obs.shape)
        obs = F.relu(self.conv1(obs))
        #print(obs.shape)
        obs = F.relu(self.conv2(obs))
        #print(obs.shape)
        obs = F.relu(self.conv3(obs))
        #print(obs.shape)
        obs = obs.view(-1,128* 21* 12)
        obs = F.relu(self.fc1(obs))
        # 4 Rows and 6 columns
        actions = self.fc2(obs)
        return actions

This is the Agent Code, and it contains the error causing line of code

class DQNAgent(object):
    def __init__(self, gamma, epsilon, alpha, maxMemory,
                 epsEnd = 0.05, replace =10000, actionSpace = [0,1,2,3,4,5]):
        '''
        Gamma -> discount factor of valuing current reward over future reward
        Epsilon -> for trade off between exploration-exploitation
        alpha -> learn rate
        maxMemory -> max size of Memory buffer
        epsEnd -> smallest value of Exploration
        repace -> how often to replace target network
        '''
        self.GAMMA = gamma
        self.EPSILON = epsilon
        self.EPS_END = epsEnd
        self.actionSpace = actionSpace
        self.maxMemory = maxMemory
        self.steps = 0
        self.learn_step_counter = 0
        self.memory = []
        self.memCount = 0
        self.replace_tgt_count = replace
        self.Q_eval = DeepQNetwork(alpha)
        self.Q_next = DeepQNetwork(alpha)

    def storeTransition(self, state, action, reward, state_):
        '''Stores Transition states'''
        if self.memCount < self.maxMemory:
            self.memory.append([state,action,reward,state_])
        else:
            self.memory[self.memCount%self.maxMemory] = [state,action,reward,state_]
        self.memCount +=1

    def chooseAction(self,obs):
        '''
        Exploration if np.random > epsilon
        else take epsilon greedy action
        '''
        rand = np.random.random()
        # Get the value for all actions for the current set of states
        # Forward pass the stack of frames to get value of each action given subset of staes in obs
        actions = self.Q_eval.forward(obs)
        if rand<1-self.EPSILON:
            action = torch.argmax(actions[1]).item()
        else:
            action = np.random.choice(self.actionSpace)
        self.steps += 1
        return action

    def learn(self, batch_size):
        self.Q_eval.optimizer.zero_grad()
        #0 gradient to do batch optimisation
        if self.replace_tgt_count is not None and self.learn_step_counter % self.replace_tgt_count==0:
            self.Q_next.load_state_dict(self.Q_eval.state_dict())

        # memory subsampling
        if self.memCount + batch_size < self.maxMemory:
            memStart = int(np.random.choice(range(self.memCount)))
        else:
            memStart = int(np.random.choice(range(self.maxMemory-batch_size-1)))

        miniBatch = self.memory[memStart:memStart+batch_size]
        memory = np.array(miniBatch)

        #feed forward current state and successor state conv to list as memory is array of numpy objects
        Qpred = self.Q_eval.forward(list(memory[:,0][:])).to(self.Q_eval.device)
        Qnext = self.Q_next.forward(list(memory[:,3][:])).to(self.Q_eval.device)

        maxA = torch.argmax(Qnext,dim = 1).to(self.Q_eval.device)
        #calculate rewards
        rewards = torch.Tensor(list(memory[:,2])).to(self.Q_eval.device)
        # loss for every action except max action to be 0
        Qtarget = Qpred.clone()
        print(Qnext.shape)
        Qtarget[:,maxA] = rewards + self.GAMMA*torch.max(Qnext[1])# PROBLEMATIC LINE
        # epsilon decay action
        if self.steps > 2000:
            if self.EPSILON-1e-4 >self.EPS_END:
                self.EPSILON-= 1e-4
            else:
                self.EPSILON = self.EPS_END
        loss = self.Q_eval.loss(Qtarget,Qpred).to(self.Q_eval.device)
        loss.backward()
        self.Q_eval.optimizer.step()
        self.learn_step_counter +=1

env = gym.make("Invader-v0")
agent = DQNAgent(gamma=0.95,epsilon = 1.0,alpha = 0.003, maxMemory = 5000,replace = None)
while agent.memCount < agent.maxMemory:
        obs = env.reset()
        done = False
        lives = 3
        while not done:
            action = env.action_space.sample()
            obs_ , reward, done, info = env.step(action)
            if done and info['lives']<lives:
                lives = info['lives']
                reward -= 200
            agent.storeTransition(preprocess(obs),action,reward,preprocess(obs_))
            obs= obs_
initialised = True

scores = []
epsHistory = []
numGames = 50
batch_size = 16

for i in range(numGames):

    print(f'starting game {i+1}, epsilon = {agent.EPSILON}')
    epsHistory.append(agent.EPSILON)
    done = False
    obs = env.reset()

    frames = [np.sum(obs)]
    score = 0
    lastAction = 0
    lives = 3
    while not done:
        if len(frames) == 4:
            action = agent.chooseAction(frames)
            frames = []
        else:
            action = lastAction
        obs_, reward, done, info = env.step(action)
        score += score-reward
        frames.append(preprocess(obs_))
        if done and info['lives'] < lives:
            reward -=200
        agent.storeTransition(preprocess(obs),action,reward,preprocess(obs_))
        obs= obs_
        agent.learn(batch_size)
        lastAction = action
    scores.append(score)
    print('score: ', score)
    x = [i+1 for i in range(numGames)]


title: How to transform labels in pytorch to onehot

body:
How to give target_transform a function for changing the labels to onehot encoding?
For example, the MNIST dataset in torchvision:
train_dataset = torchvision.datasets.MNIST(root='./mnist_data/',
                                           train=True,
                                           download=True,
                                           transform=train_transform,
                                           target_transform=<????>)

Tried F.onehot() but it didn't work.


title: Is location-dependent convolution filter possible in PyTorch or TensorFlow?

body:
Let's pretend that in plus of having an image, I also have a gradient from left to right on the X axis of an image, and another gradient from top to bottom on the Y axis. Those two gradients are of the same size of the image, and could both range from -0.5 to 0.5.

Now, I'd like to make the convolution kernel (a.k.a. convolution filter, or convolution weights) depend on the (x, y) location in the gradient. So the kernel is a function of the gradient as if the kernel was the output of a nested mini-neural net. This would make the weights of the filter to be different in every position, but slightly similar to their neighbors. How do I do that within PyTorch or TensorFlow?

Sure, I could compute a Toeplitz matrix (a.k.a. diagonal-constant matrix)  by myself, but the matrix multiplication would take O(n^3) operations if pretending x==y==n, whereas convolutions can be implemented in O(n^2) normally. Or I could maybe iterate on every element myself and do the multiplications in an unvectorized fashion.

Any better ideas? I'd like to see creativity here, thinking about how could this be implemented neatly. I believe coding that would be an interesting way to build a network layer capable of doing things similar to a simplified version of a Spatial Transformer Networks, but which's spatial transformation would be independent of the image.


title: How to customize number of multiple hidden layer units in pytorch LSTM?

body:
In pytorch LSTM, RNN or GRU models, there is a parameter called "num_layers", which controls the number of hidden layers in an LSTM. I wonder that since there are multiple layers in an LSTM, why the parameter "hidden_size" is only one number instead of a list containing the number of hidden states in multiple layers, like [10, 20, 30].

I came across when I worked on a regression project, in which I feed sequence data of (seq_len, batch, feature) to LSTM, and I want to get the scalar output of every time step.

A helpful link to understand the pytorch LSTM framework, here.
I'd really appreciate it if anyone can answer this.


title: How to create a pipeline in sagemaker with pytorch

body:
I am dealing with a classification problem with text data in sagemaker. Where, i first fit and transform it into structured format(say by using TFIDF in sklearn) then i kept the result in S3 bucket and i used it for training my pytorch model for which i have written the code in my entry point.

if we notice, by the end of the above process, i have two models

sklearn TFIDF model
actual PyTorch model

So, when every time i need to predict on a new text data, i need to separately process(transform) the text data with TFIDF model which i created during my training.

How can i create a pipeline in sagemaker with sklearn's TFIDF and pytorch models.

if i fit and transform text data using TFIDF in my main method in entrypoint then if i train my pytorch model in my main method, i can return only one model which will be used in model_fn()


title: How the gradient is calculated in pytorch

body:
I have an example code. When I calculate dloss/dw manually I get the result 8, but the following code gives me a 16. Please tell me how the gradient is 16.
import torch
x = torch.tensor(2.0)
y = torch.tensor(2.0)
w = torch.tensor(3.0, requires_grad=True)
# forward
y_hat = w * x
s = y_hat - y
loss = s**2
#backward
loss.backward()
print(w.grad)


title: what versions of Fastai, torch & torchvision are compatible for cuda 10.1, python 3.7?

body:
I have created a new environment for installing fastai, which has dependency on torch & torchvision packages.
After installation, I get: RuntimeError: Detected that PyTorch and torchvision were compiled with different CUDA versions. PyTorch has CUDA Version=9.2 and torchvision has CUDA Version=10.1. Please reinstall the torchvision that matches your PyTorch install.
So recommend the steps for installing compatible versions of fastai and it's depencies torch, torchvision on Cuda10.1 & python 3.7


title: How to write a RNN with RNNCell in pytorch?

body:
I am trying to rewrite a code from this simple Vanilla RNN to RNNCell format in pytorch. This is the full code
import torch
import torch.nn as nn
from torch.autograd import Variable

torch.manual_seed(777)

class SimpleRNN(nn.Module):
    def __init__(self,inputs,hiddens,n_class):
        super().__init__()
        self.rnn = nn.RNNCell(inputs,hiddens)
        self.linear = nn.Linear(hiddens,n_class)
        self.hiddens = hiddens
    def forward(self,x):
        hx = torch.zeros((x.shape[1],hiddens))
        rnn_out = []
        for i in x:
            hx = self.rnn(i,hx)
            rnn_out.append(hx)
        linear_out =  self.linear(rnn_out.view(-1, hiddens))
        return linear_out

# hyperparameters
seq_len = 6      # |hihell| == 6, equivalent to time step
input_size = 5   # one-hot size
batch_size = 1   # one sentence per batch
num_layers = 1   # one-layer rnn
num_classes = 5  # predicting 5 distinct character
hidden_size = 4  # output from the RNN

# create an index to character mapping
idx2char = ['h', 'i', 'e', 'l', 'o']

# Teach hihell -> ihello
x_data = [[0, 1, 0, 2, 3, 3]]    # hihell
x_one_hot = [[[1, 0, 0, 0, 0],   # h 0
              [0, 1, 0, 0, 0],   # i 1
              [1, 0, 0, 0, 0],   # h 0
              [0, 0, 1, 0, 0],   # e 2
              [0, 0, 0, 1, 0],   # l 3
              [0, 0, 0, 1, 0]]]  # l 3

x_one_hot = torch.Tensor(x_one_hot)
y_data = torch.Tensor([1, 0, 2, 3, 3, 4])  # ihello

rnn = SimpleRNN(input_size,hidden_size,num_classes)

# train the model
num_epochs = 15
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(rnn.parameters(), lr = 0.1)
for epoch in range(1, num_epochs + 1):
    optimizer.zero_grad()
    outputs = rnn(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    # check the current predicted string
    # max gives the maximum value and its
    # corresponding index, we will only
    # be needing the index
    _, idx = outputs.max(dim = 1)
    idx = idx.data.numpy()
    result_str = [idx2char[c] for c in idx]
    print('epoch: {}, loss: {:1.3f}'.format(epoch, loss.item()))
    print('Predicted string: ', ''.join(result_str))

I am trying to emulate the original RNN + fully connected class object from the tutorial and reusing many of its code. I computed the rnn_out and appended its value in a python list
        rnn_out = []
        for i in x:
            hx = rnn(i,hx)
            rnn_out.append(hx)

because its a python list I cant further execute the code and it will result in this error

AttributeError: 'list' object has no attribute 'view'

How should I write a RNN using RNNCell?


title: Can I use BERT as a feature extractor without any finetuning on my specific data set?

body:
I'm trying to solve a multilabel classification task of 10 classes with a relatively balanced training set consists of ~25K samples and an evaluation set consists of ~5K samples.
I'm using the huggingface:
model = transformers.BertForSequenceClassification.from_pretrained(...

and obtain quite nice results (ROC AUC = 0.98).
However, I'm witnessing some odd behavior which I don't seem to make sense of -
I add the following lines of code:
for param in model.bert.parameters():
    param.requires_grad = False

while making sure that the other layers of the model are learned, that is:
[param[0] for param in model.named_parameters() if param[1].requires_grad == True]
gives
['classifier.weight', 'classifier.bias']

Training the model when configured like so, yields some embarrassingly poor results (ROC AUC = 0.59).
I was working under the assumption that an out-of-the-box pre-trained BERT model (without any fine-tuning) should serve as a relatively good feature extractor for the classification layers. So, where do I got it wrong?


title: Specifying cpu-only for pytorch in conda YAML file

body:
I can set up a conda environment successfully as follows:
conda create --name temp python=3.8.5
conda install pytorch==1.6.0 torchvision==0.7.0 cpuonly -c pytorch

I then save the environment to a YAML config file. The looks like this:
name: temp
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - blas=1.0=mkl
  - ca-certificates=2020.10.14=0
  - certifi=2020.6.20=pyhd3eb1b0_3
  - cpuonly=1.0=0
  - freetype=2.10.4=h5ab3b9f_0
  - intel-openmp=2020.2=254
  - jpeg=9b=h024ee3a_2
  - lcms2=2.11=h396b838_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20191231=h14c3975_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libpng=1.6.37=hbc83047_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtiff=4.1.0=h2733197_1
  - lz4-c=1.9.2=heb0550a_3
  - mkl=2020.2=256
  - mkl-service=2.3.0=py38he904b0f_0
  - mkl_fft=1.2.0=py38h23d657b_0
  - mkl_random=1.1.1=py38h0573a6f_0
  - ncurses=6.2=he6710b0_1
  - ninja=1.10.1=py38hfd86e86_0
  - numpy=1.19.2=py38h54aff64_0
  - numpy-base=1.19.2=py38hfa32c7d_0
  - olefile=0.46=py_0
  - openssl=1.1.1h=h7b6447c_0
  - pillow=8.0.1=py38he98fc37_0
  - pip=20.2.4=py38h06a4308_0
  - python=3.8.5=h7579374_1
  - pytorch=1.6.0=py3.8_cpu_0
  - readline=8.0=h7b6447c_0
  - setuptools=50.3.0=py38h06a4308_1
  - six=1.15.0=py_0
  - sqlite=3.33.0=h62c20be_0
  - tk=8.6.10=hbc83047_0
  - torchvision=0.7.0=py38_cpu
  - wheel=0.35.1=py_0
  - xz=5.2.5=h7b6447c_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.5=h9ceee32_0
prefix: /data/anaconda/envs/temp

But if I try making a conda environment from the following file:
name: temp
channels:
  - defaults
  - pytorch
dependencies:
  - python==3.8.5
  - pytorch==1.6.0=py3.8_cpu_0

it fails, with the following incompatibilities:
UnsatisfiableError: The following specifications were found to be incompatible with each other:

Package ld_impl_linux-64 conflicts for:
python==3.8.5 -> ld_impl_linux-64
Package sqlite conflicts for:
python==3.8.5 -> sqlite[version='>=3.32.3,<4.0a0|>=3.33.0,<4.0a0']
Package * conflicts for:
pytorch==1.6.0=py3.8_cpu_0 -> *[track_features=cpuonly]
Package ncurses conflicts for:
python==3.8.5 -> ncurses[version='>=6.2,<7.0a0']
Package mkl conflicts for:
pytorch==1.6.0=py3.8_cpu_0 -> mkl[version='>=2018']
Package blas conflicts for:
pytorch==1.6.0=py3.8_cpu_0 -> blas=[build=mkl]
Package zlib conflicts for:
python==3.8.5 -> zlib[version='>=1.2.11,<1.3.0a0']
Package openssl conflicts for:
python==3.8.5 -> openssl[version='>=1.1.1g,<1.1.2a']
Package python conflicts for:
pytorch==1.6.0=py3.8_cpu_0 -> python[version='>=3.8,<3.9.0a0']
Package xz conflicts for:
python==3.8.5 -> xz[version='>=5.2.5,<6.0a0']
Package libffi conflicts for:
python==3.8.5 -> libffi[version='>=3.3,<3.4.0a0']
Package libgcc-ng conflicts for:
python==3.8.5 -> libgcc-ng[version='>=7.3.0']
Package tk conflicts for:
python==3.8.5 -> tk[version='>=8.6.10,<8.7.0a0']
Package pip conflicts for:
python==3.8.5 -> pip
Package ninja conflicts for:
pytorch==1.6.0=py3.8_cpu_0 -> ninja
Package readline conflicts for:
python==3.8.5 -> readline[version='>=8.0,<9.0a0']
Package numpy conflicts for:
pytorch==1.6.0=py3.8_cpu_0 -> numpy[version='>=1.11']

Why? How could such a simple configuration, which was cut down from a successful environment, fail? How should I specify the CPU-only version of pytorch 1.6.0 in a YAML config file for a conda environment?


title: nn.CrossEntropyLoss() function results in torch.FloatTensor has no 'requires_gradient' attribute error

body:
I am using Resnet18 pre-trained model. So, basically, I take the "output" of the model and then call CrossEntropyLoss(). Let the output of the model be "output" and "labels" is the class label. So, my CrossEntropyLoss(output,labels) is called. I checked the type of "output", it is <tensor.autograde.variable.Variable>. I tried different combinations with "labels" also. First, made it as a numpy array and then a variable. But nothing seems to work. I am using pytorch 0.3.1. Please refrain yourself from suggesting to upgrade the pytorch as it can't possibly be done in my current situation. I have also attached the error stack. However, it seems to work in version 0.4.0.

Criterion function is the CrossEntropyLoss function.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-76-6d5f48373efd> in <module>()
      5
      6 # Train and evaluate
----> 7 model_ft, hist = train_model(model_ft, data, criterion, optimizer_ft, num_epochs=num_epochs, is_inception=(model_name=="inception"))

<ipython-input-70-bfd03f976e97> in train_model(model, dataloaders, criterion, optimizer, num_epochs, is_inception)
     47                 labels=(torch.from_numpy(np.array([labels])))
     48                 #print(((outputs.requires_gradient)))
---> 49                 loss = criterion(outputs, labels)  ##calculate entropy loss
     50
     51                 _, preds = torch.max(outputs, 1)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    355             result = self._slow_forward(*input, **kwargs)
    356         else:
--> 357             result = self.forward(*input, **kwargs)
    358         for hook in self._forward_hooks.values():
    359             hook_result = hook(self, input, result)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    675
    676     def forward(self, input, target):
--> 677         _assert_no_grad(target)
    678         return F.cross_entropy(input, target, self.weight, self.size_average,
    679                                self.ignore_index, self.reduce)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/loss.py in _assert_no_grad(variable)
      8
      9 def _assert_no_grad(variable):
---> 10     assert not variable.requires_grad, \
     11         "nn criterions don't compute the gradient w.r.t. targets - please " \
     12         "mark these variables as volatile or not requiring gradients"

AttributeError: 'torch.LongTensor' object has no attribute 'requires_grad'

My code:

val_acc_history = []

best_model_wts = copy.deepcopy(model.state_dict())
best_acc = 0.0

for epoch in range(num_epochs):
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 10)

    # Each epoch has a training and validation phase
    for phase in ['train', 'val']:
        if phase == 'train':
            model.train()  # Set model to training mode
        else:
            model.eval()   # Set model to evaluate mode

        running_loss = 0.0
        running_corrects = 0

        # Iterate over data.
        count=0
        for inputs, labels in dataloaders[phase]:

            # zero the parameter gradients
            optimizer.zero_grad()

            outputs = model(inputs.unsqueeze(0))  ###input to the model and output porduced
            labels=(torch.from_numpy(np.array([labels])))
            loss = criterion(outputs, labels)  ##calculate entropy loss

            _, preds = torch.max(outputs, 1)

            # backward + optimize only if in training phase
            if phase == 'train':
                loss.backward()    ### loss gradient going backward
                optimizer.step()    ### Optimizer performs parameter update based on current gradient

            # statistics
            running_loss += loss.item() * inputs.size(0)
            running_corrects += torch.sum(preds == labels.data)
            count=count+1

        epoch_loss = running_loss / count
        epoch_acc = running_corrects.double() / count

        print('{} Loss: {:.4f} Acc: {:.4f}'.format(phase, epoch_loss, epoch_acc))

        # deep copy the model
        if phase == 'val' and epoch_acc > best_acc:
            best_acc = epoch_acc
            best_model_wts = copy.deepcopy(model.state_dict())
        if phase == 'val':
            val_acc_history.append(epoch_acc)

    #print()

time_elapsed = time.time() - since
print('Training complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))
print('Best val Acc: {:4f}'.format(best_acc))

# load best model weights
model.load_state_dict(best_model_wts)
return model, val_acc_history

'''


title: pytorch with CIFAR-100

body:
import torch
import torchvision
import torchvision.transforms as transforms

transform=transforms.Compose(
[transforms.ToTensor(),
 transforms.Normalize((0.5),(0.5))])

trainset=torchvision.datasets.CIFAR100(root='./dataset',train=True,
                                  download=True,transform=transform)

trainloader=torch.utils.data.CIFAR100(trainset,batch_size=4,shuffle=True)

testset=torchvision.datasets.CIFAR100(root='./dataset',train=False,
                                 download=True,transform=transform)

testloader=torch.utils.data.DataLoader(testset,batch_size=4,shuffle=False)

classes=('0','1','2','3','4','5','6','7','8','9')

the error message written as below
module 'torch.utils.data' has no attribute 'CIFAR100'
when I use torch.utils.data with cifar-10, it worked
but it doesn't work with cifar-100
could u plz let me know why this happen?


title: Modify layers in Tensorflow, Keras or PyTorch

body:
I want to modify output of every layer in tensorFlow before passing to next layer in this format for standard deep neural network (Not CNN necessarily):

Say this is Pseudocode of layers during forward propagation prior to any modification, and X is input feature matrix:

Prior Modification:
Step 0: l0 = X
Step 1: l1 = nonlin(np.dot(l0,syn0)) and
Step 2: l2 = nonlin(np.dot(l1,syn1))
Step 3: get error, back propagate and modify weights syn0 and syn1.

So outputs of each layer are in l1 and l2, where nonlin is the sigmoid activation function (or someother like ReLU). Syn0 and Syn1 are the corresponding weights which are initially randomized and then modified later by back-propagation using loss function like cross-entropy or l2 error etc. So far so good- all looks standard. Ofcourse, tensorflow allows for bias term etc in the standard sense along with epochs, batches, learning rate, variations of drop-out/regularization and other fancy features etc.

What i instead want to do is the following modification to that pseudo-code in forward propagation steps:

 After Modification:
    l0 = M0 %*% X
    l1 = nonlin(np.dot(M1 %*% l0, syn0)) and
    l2 = nonlin(np.dot(M2 %*% l1, syn1))
    get error, back propagate and modify weights syn0 and syn1.

If you notice I updated l0, l1 and l2 with corresponding matrix multiplication (denoted by %*%) with matrices M0, M1, M2 respectively, which we can refer to in this pseudo-code as YoYoMats. All YoYoMats are square and symmetric and provided before hand along with dataset X and response or label column.

how can this be implemented in Keras or TensorFlow? Ideally in RKeras or RtensorFlow. If not Python version or Pytorch etc is okay too! I can implement this change by implementing vanilla NN from scratch in Python, but I want to do it in TensorFlow or Keras to use this change with all of the other supercool features these matured tools give and I am not skilled to create TensorFlow of my own kind from scratch :). hence, this question of implementing this within TensorFlow or Keras code etc.

Note as you see the modification is easy till layer l1 because the modified version is same as:

After Modification:
        l0 = M1 %*% M0 %*% X
        l1 = nonlin(np.dot(l0, syn0)) and
        l2 = nonlin(np.dot(M2 %*% l1, syn1))
        get error, back propagate and modify weights syn0 and syn1.

So, its easy to just modify l0 as follows and use it in l1 the usual way. Nothing changes till here. So simple. The tricky part is only from layer l2 and onwards if we had more layers in implementing the multiplication with M2 or say M3, M4 etc. Do I need to modify output of tf.contrib.layers.fully_connected to get this done? If so how? If not, alternate way? Caveat-if the above pseudocode was modified for training. I need prediction function modified with corresponding YoYoMats as well for testing/out-of-sample prediction purposes as well.


I am trying to use a pre-trained model. Here's where the problem occurs

Isn't the model supposed to take in a simple colored image? Why is it expecting a 4-dimensional input?

The eval() method is:

def eval(file):
    image = io.imread(file)
    plt.imshow(image)
    image = cv2.resize(image, (160,40)).transpose((2,1,0))
    output = model(torch.tensor(image[np.newaxis,...]).float())[0].squeeze().detach().numpy()
    return decode_prob(output)

And the output of eval('image.png') :

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-66-0b951c2596f8> in <module>()
----> 1 eval('/content/image.png')

5 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    344                             _pair(0), self.dilation, self.groups)
    345         return F.conv2d(input, weight, self.bias, self.stride,
--> 346                         self.padding, self.dilation, self.groups)
    347
    348     def forward(self, input):

RuntimeError: Given groups=1, weight of size [32, 3, 3, 3], expected input[1, 4, 160, 40] to have 3 channels, but got 4 channels instead


title: How to install PyTorch with pipenv and save it to Pipfile and Pipfile.lock?

body:
I’m currently using Pipenv to maintain the Python packages used in a specific project. Most of the downloads I’ve tried so far have worked as intended; that is, I enter pipenv install [package] and it installs the package into the virtual environment, then records the package information into both the Pipfile and Pipfile.lock.
However, I’m running into some problems installing PyTorch.
I’ve tried running pipenv install torch, but every time the locking step fails. Instead, I’ve tried forcing a download directly from the PyTorch website using
pipenv run pip install torch===1.6.0 torchvision===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

And it actually installs! If I run pipenv graph it displays both torch and torchvision with their dependencies. But one problem remains: neither torch nor torchvision are being saved into Pipfile and Pipfile.lock.
Any idea on how I can make this happen?


title: Is there a mistake in pytorch tutorial?

body:
The official pytorch tutorial (https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#gradients) indicates that out.backward() and out.backward(torch.tensor(1)) are equivalent. But this does not seem to be the case.

import torch

x = torch.ones(2, 2, requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()

# option 1
out.backward()

# option 2. Replace! do not leave one after the other
# out.backward(torch.tensor(1))

print(x.grad)

Using option 2 (commented out) results in an error.

Note: do not leave two backward calls. Replace option 1 with 2.

Is the tutorial out of date? What is the purpose of the argument?

Update
If I use out.backward(torch.tensor(1)) as the tutorial says, I get:

E       RuntimeError: invalid gradient at index 0 - expected type torch.FloatTensor but got torch.LongTensor

../../../anaconda3/envs/phd/lib/python3.6/site-packages/torch/autograd/__init__.py:90: RuntimeError

I tried also using out.backward(torch.Tensor(1)) and I get instead:

E       RuntimeError: invalid gradient at index 0 - expected shape [] but got [1]

../../../anaconda3/envs/phd/lib/python3.6/site-packages/torch/autograd/__init__.py:90: RuntimeError


title: Caching for a custom pytorch dataset

body:
I am creating a custom pytorch dataset to train an audio classification system.
Since some of the transformations applied during training as data augmentation are computationally demanding (e.g. pitch shift audio data), I would like to add a cache to my dataset to speed up the training. HDF5 seems to be the recommended option for this task.

Basically, in each training loop, I need to search the cache. If a sample with certain augmentation parameters has already been calculated the sample will be loaded from the cache, otherwise it will be calculated and cached. How should I structure the HDF5 file to do that?

Thanks in advance for any suggestion.


title: How do I make a mask of diagonal matrix, but starting from the 2nd column?

body:
So here is what I can get with torch.eye(3,4) now

The matrix I get:

[[1, 0, 0, 0],
 [0, 1, 0, 0],
 [0, 0, 1, 0]]

Is there any (easy)way to transform it, or make such a mask in this format:

The matrix I want:

[[0, 1, 0, 0],
 [0, 0, 1, 0],
 [0, 0, 0, 1]]


title: Unable to import torch.distributed.rpc

body:
I was trying to run the RPC rnn example from the following link - https://github.com/pytorch/examples/tree/master/distributed/rpc/rnn

but I am unable to import RPC module of the torch.distributed and getting the following error.

Traceback (most recent call last):
  File ".\main.py", line 6, in <module>
    import torch.distributed.rpc as rpc
  File "C:\Users\Public\Anaconda\lib\site-packages\torch\distributed\rpc\__init__.py", line 9, in <module>
    from . import backend_registry
  File "C:\Users\Public\Anaconda\lib\site-packages\torch\distributed\rpc\backend_registry.py", line 8, in <module>
    import torch.distributed.distributed_c10d as dc10d
  File "C:\Users\Public\Anaconda\lib\site-packages\torch\distributed\distributed_c10d.py", line 9, in <module>
    from .rendezvous import rendezvous, register_rendezvous_handler  # noqa: F401
  File "C:\Users\Public\Anaconda\lib\site-packages\torch\distributed\rendezvous.py", line 9, in <module>
    from . import FileStore, TCPStore
ImportError: cannot import name 'FileStore' from 'torch.distributed' (C:\Users\Public\Anaconda\lib\site-packages\torch\distributed\__init__.py)

Torch Version:
torch                              1.4.0+cpu
torchvision                        0.5.0+cpu


I have executed the following code

   import matplotlib.pyplot as plt
   import torch
   import torch.nn as nn
   import torch.optim as optim
   from torch.autograd import Variable
   from torch.utils import data as t_data
   import torchvision.datasets as datasets
   from torchvision import transforms
   data_transforms = transforms.Compose([transforms.ToTensor()])
  mnist_trainset = datasets.MNIST(root='./data', train=True,
                           download=True, transform=data_transforms)
batch_size=4
dataloader_mnist_train = t_data.DataLoader(mnist_trainset,
                                           batch_size=batch_size,
                                           shuffle=True
                                           )

def make_some_noise():
    return torch.rand(batch_size,100)

class generator(nn.Module):

    def __init__(self, inp, out):

        super(generator, self).__init__()

        self.net = nn.Sequential(
                                 nn.Linear(inp,784),
                                 nn.ReLU(inplace=True),
                                 nn.Linear(784,1000),
                                 nn.ReLU(inplace=True),
                                 nn.Linear(1000,800),
                                 nn.ReLU(inplace=True),
                                 nn.Linear(800,out)
                                    )

    def forward(self, x):
        x = self.net(x)
        return x

class discriminator(nn.Module):

    def __init__(self, inp, out):

        super(discriminator, self).__init__()

        self.net = nn.Sequential(
                                 nn.Linear(inp,784),
                                 nn.ReLU(inplace=True),
                                 nn.Linear(784,784),
                                 nn.ReLU(inplace=True),
                                 nn.Linear(784,200),
                                 nn.ReLU(inplace=True),
                                 nn.Linear(200,out),
                                 nn.Sigmoid()
                                    )

    def forward(self, x):
        x = self.net(x)
        return x

def plot_img(array,number=None):
    array = array.detach()
    array = array.reshape(28,28)

    plt.imshow(array,cmap='binary')
    plt.xticks([])
    plt.yticks([])
    if number:
        plt.xlabel(number,fontsize='x-large')
    plt.show()

d_steps = 100
g_steps = 100

gen=generator(4,4)
dis=discriminator(4,4)

criteriond1 = nn.BCELoss()
optimizerd1 = optim.SGD(dis.parameters(), lr=0.001, momentum=0.9)

criteriond2 = nn.BCELoss()
optimizerd2 = optim.SGD(gen.parameters(), lr=0.001, momentum=0.9)

printing_steps = 20

epochs = 5

for epoch in range(epochs):

    print (epoch)

    # training discriminator
    for d_step in range(d_steps):
        dis.zero_grad()

        # training discriminator on real data
        for inp_real,_ in dataloader_mnist_train:
            inp_real_x = inp_real
            break

        inp_real_x = inp_real_x.reshape(batch_size,784)
        dis_real_out = dis(inp_real_x)
        dis_real_loss = criteriond1(dis_real_out,
                              Variable(torch.ones(batch_size,1)))
        dis_real_loss.backward()

        # training discriminator on data produced by generator
        inp_fake_x_gen = make_some_noise()
        #output from generator is generated
        dis_inp_fake_x = gen(inp_fake_x_gen).detach()
        dis_fake_out = dis(dis_inp_fake_x)
        dis_fake_loss = criteriond1(dis_fake_out,
                                Variable(torch.zeros(batch_size,1)))
        dis_fake_loss.backward()

        optimizerd1.step()

    # training generator
    for g_step in range(g_steps):
        gen.zero_grad()

        #generating data for input for generator
        gen_inp = make_some_noise()

        gen_out = gen(gen_inp)
        dis_out_gen_training = dis(gen_out)
        gen_loss = criteriond2(dis_out_gen_training,
                               Variable(torch.ones(batch_size,1)))
        gen_loss.backward()

        optimizerd2.step()

    if epoch%printing_steps==0:
        plot_img(gen_out[0])
        plot_img(gen_out[1])
        plot_img(gen_out[2])
        plot_img(gen_out[3])
        print("\n\n")

On running the code,following error is shown

 File "mygan.py", line 105, in <module>
    dis_real_out = dis(inp_real_x)
    RuntimeError: size mismatch, m1: [4 x 784], m2: [4 x 784] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:136

How can I resolve this?

I got the code from https://blog.usejournal.com/train-your-first-gan-model-from-scratch-using-pytorch-9b72987fd2c0


title: How to deserialize a PyTorch saved model with private methods inside a class?

body:
I used the PyTorch saving method to serialize a bunch of essential objects. Among those, there was one class referencing a private method inside the __init__ of that same class. Now, after the serialization, I can't deserialize (unpickle) files because the private method is not accessible outside the class. Any idea how to solve or bypass it? I need to recover the data saved into the attributes of that class.

  File ".conda/envs/py37/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-a5666d77c70f>", line 1, in <module>
    torch.load("snapshots/model.pth", map_location='cpu')
  File ".conda/envs/py37/lib/python3.7/site-packages/torch/serialization.py", line 529, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File ".conda/envs/py37/lib/python3.7/site-packages/torch/serialization.py", line 702, in _legacy_load
    result = unpickler.load()
AttributeError: 'Trainer' object has no attribute '__iterator'

EDIT-1:

Here there is a piece of code that will generate the problem I’m facing right now.

import torch

class Test:
    def __init__(self):
        self.a = min
        self.b = max
        self.c = self.__private  # buggy

    def __private(self):
        return None

test = Test()

torch.save({"test": test}, "file.pkl")
torch.load("file.pkl")

However, if you remove the private attribute from the method, you won’t get any error.

import torch

class Test:
    def __init__(self):
        self.a = min
        self.b = max
        self.c = self.private  # not buggy

    def private(self):
        return None

test = Test()

torch.save({"test": test}, "file.pkl")
torch.load("file.pkl")


title: TensorBoard: Tutorial Pytorch: module 'tensorflow._api.v2.io.gfile' has no attribute 'get_filesystem'

body:
I am having issues running this tutorial about Pytorch and TensorBoard with Embeddings https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html

I am having this message,

AttributeError                            Traceback (most recent call last)
<ipython-input-10-e0404d94b4cd> in <module>()
     22 writer.add_embedding(features,
     23                     metadata=class_labels,
---> 24                     label_img=images.unsqueeze(1))
     25 writer.close()

/usr/local/lib/python3.6/dist-packages/torch/utils/tensorboard/writer.py in add_embedding(self, mat, metadata, label_img, global_step, tag, metadata_header)
    779         save_path = os.path.join(self._get_file_writer().get_logdir(), subdir)
    780
--> 781         fs = tf.io.gfile.get_filesystem(save_path)
    782         if fs.exists(save_path):
    783             if fs.isdir(save_path):

and I can visualize the projector


title: Why doesn't pytorch allow inplace operations on leaf variables?

body:
So if I run this code in Pytorch:

x = torch.ones(2,2, requires_grad=True)
x.add_(1)

I will get the error:

  RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

I understand that Pytorch does not allow inplace operations on leaf variables and I also know that there are ways to get around this restrictions. What I don't understand is the philosophy behind this rule. Why is it wrong to change a leaf variable with inplace operations?


title: torch.save(model, PATH) error : NotADirectoryError: [Errno 20] Not a directory: '~/KoGPT2_checkpoint.tar'

body:
I'm using pytorch, and I want to use pytorch checkpoint
this is my code
import os

save_path = 'drive/My Drive/Colab Notebooks/KoGPT2_checkpoint/'
torch.save(model, os.path.join(save_path+'KoGPT2_checkpoint.tar'))

and the error says
/usr/local/lib/python3.6/dist-packages/torch/serialization.py in __init__(self, name, mode)
    213 class _open_file(_opener):
    214     def __init__(self, name, mode):
--> 215         super(_open_file, self).__init__(open(name, mode))
    216
    217     def __exit__(self, *args):

NotADirectoryError: [Errno 20] Not a directory: 'drive/My Drive/Colab Notebooks/KoGPT2_checkpoint/KoGPT2_checkpoint.tar'

I also tested this kind of code to solve this
import os

save_path = 'drive/My Drive/Colab Notebooks/KoGPT2_checkpoint'
torch.save(model, 'drive/My Drive/Colab Notebooks/KoGPT2_checkpoint')

but the problem is, I couldn't make .tar file in my colab.
I could only make "KoGPT2_checkpoint" file, which didn't have any extension
I don't know why I'm getting this error
Could you guys tell me how to make checkpoints using pytorch?


title: Problem importing torch in command prompt, but it works directly in VS code / Jupyter

body:
I am trying to run a flask app in my command prompt on Windows 10. Whenever I try to run it, it errors out when trying to import torch. I have created a fresh environment and installed the latest pytorch from this page https://pytorch.org/get-started/locally/ but it still doesn't work in command prompt. Oddly, if I just execute code that says "import torch" in jupyter notebook or VS code, it doesn't error out. It only gives me the error in Command Prompt.

Does anyone know what the issue could be?


title: Will using pre trained word embeddings speed up model training in Pytorch? Also, what to do if some words of the dataset are not in the embeddings?

body:
I have a text classification model that passes word embeddings through a GRU and the output of that GRU is passed fed into an ANN that gives one class as an output.

The dataset is huge(1.4 million lines) and I am training it on Google Colab. It takes 15 mins to just go through 500 lines of the dataset.

So to speed up training, I decided to use pre-trained Glove vectors instead of the random embeddings that Pytorch provides. How do I implement that?

And any idea abou what should I do about missing words that are not in the Glove vectors? How will they be embedded?

Any other ideas about speeding it up will be appreciated.

EDIT:
Based on @scarecrow 's request for the code.
The code I used to convert embeddings to a float tensor:

print(f'First embedding: {embeddings[0]}')
print(f'The dimensions of the embeddings are: {len(embeddings[0])}')
embed = torch.FloatTensor([embeddings[0], embeddings[1]])

This is an example only. I thought it will be better to show what block of code is giving the error.

The error I get is:

First embedding: ['0.418', '0.24968', '-0.41242', '0.1217', '0.34527', '-0.044457', '-0.49688', '-0.17862', '-0.00066023', '-0.6566', '0.27843', '-0.14767', '-0.55677', '0.14658', '-0.0095095', '0.011658', '0.10204', '-0.12792', '-0.8443', '-0.12181', '-0.016801', '-0.33279', '-0.1552', '-0.23131', '-0.19181', '-1.8823', '-0.76746', '0.099051', '-0.42125', '-0.19526', '4.0071', '-0.18594', '-0.52287', '-0.31681', '0.00059213', '0.0074449', '0.17778', '-0.15897', '0.012041', '-0.054223', '-0.29871', '-0.15749', '-0.34758', '-0.045637', '-0.44251', '0.18785', '0.0027849', '-0.18411', '-0.11514', '-0.78581']
The dimensions of the embeddings are: 50
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-59-f8951bd3f739> in <module>()
      1 print(f'First embedding: {embeddings[0]}')
      2 print(f'The dimensions of the embeddings are: {len(embeddings[0])}')
----> 3 embed = torch.FloatTensor([embeddings[0], embeddings[1]])

ValueError: too many dimensions 'str'


title: `BCEWithLogitsLoss` and training class dataset imbalances in Pytorch

body:
A bit of clarification on pytorch's BCEWithLogitsLoss: I am using : pos_weights = torch.tensor([len_n/(len_n + len_y), len_y/(len_n + len_y)]) to initialize the loss, with [1.0, 0.0] being the negative class and [0.0, 1.0] being the positive class, and len_n, len_y being respectively the length of negative and positive samples.

The reason to use BCEWithLogitsLoss in the first place is precisely because I assume that it is compensating an imbalance between the quantity of positive and negative samples by avoiding the network from simply "defaulting" to the most abundant class type in the training set. I want to control the priorization of the loss on detecting the less abundant class correctly. In my case, negative train samples exceed positive samples by a factor of 25 to 1, so it is very important that the network predicts a high fraction of positive samples correctly, rather than having a high overall prediction rate (even by defaulting always to negative, that would lead to 96% prediction if I only cared about that).

Question Is it correct my assumption about BCEWithLogitsLoss using the pos_weights parameter to control training class imbalances? Any insight into how the imbalance is being addressed in the loss evaluation?


title: Implementing STFT with Pytorch gives a slightly different result than the STFT with Librose

body:
I am trying to implement STFT with Pytorch. But the output from the Pytorch implementation is slightly off, when compared with the implementation from Librosa.

Librosa version

import numpy as np
from librosa.core import stft
import matplotlib.pyplot as plt

np.random.seed(3)
y = np.sin(2*np.pi*50*np.linspace(0,10,2048))+np.sin(2*np.pi*20*np.linspace(0,10,2048)) + np.random.normal(scale=1,size=2048)

S_stft = np.abs(stft(y, hop_length=512, n_fft=2048,center=False))

plt.plot(S_stft)

Pytorch version

import torch
from torch.autograd import Variable
from torch.nn.functional import conv1d

from scipy.signal.windows import hann

stride = 512

def create_filters(d,k,low=50,high=6000):
    x = np.arange(0, d, 1)
    wsin = np.empty((k,1,d), dtype=np.float32)
    wcos = np.empty((k,1,d), dtype=np.float32)
    start_freq = low
    end_freq = high
    # num_cycles = start_freq*d/44000.
    # scaling_ind = np.log(end_freq/start_freq)/k

    window_mask = hann(2048, sym=False) # same as 0.5-0.5*np.cos(2*np.pi*x/(k))
    for ind in range(k):
        wsin[ind,0,:] = window_mask*np.sin(2*np.pi*ind/k*x)
        wcos[ind,0,:] = window_mask*np.cos(2*np.pi*ind/k*x)

    return wsin,wcos

wsin, wcos = create_filters(2048,2048)

wsin_var = Variable(torch.from_numpy(wsin), requires_grad=False)
wcos_var = Variable(torch.from_numpy(wcos),requires_grad=False)

network_input = torch.from_numpy(y).float()
network_input = network_input.reshape(1,-1)

zx = np.sqrt(conv1d(network_input[:,None,:], wsin_var, stride=stride).pow(2)+conv1d(network_input[:,None,:], wcos_var, stride=stride).pow(2))
pytorch_Xs = zx.cpu().numpy()
plt.plot(pytorch_Xs[0,:1025,0])

My Question

The two graphs might look the same, but if I check the two outputs with np.allclose, we can see that they are slightly different.

np.allclose(S_stft, pytorch_Xs[0,:1025,0].reshape(1025,1))
output >>> False

Only when I tune up the tolerance to 1e-5, it gives me True result

np.allclose(S_stft, pytorch_Xs[0,:1025,0].reshape(1025,1),atol=1e-5)
output >>> True

What causes the difference in values? Is it because of the data conversion by using torch.from_numpy(y).float()?

I would like to have a difference in value less than 1e-7, 1e-8 is even better.


title: How to sample from normal distribution?

body:
If I have n sized mean vector, n sized variance vector, then how do I do this?

z ∼ N (μ, σ)

import torch
x = torch.randn(3, 3)
mu = x.mean()
sigma = x.var()

What do I do to get z?


title: Parameter-specific learning rate in PyTorch

body:
How can I set a learning rate for each specific parameter (weights and biases) in a network?

On PyTorch's docs I found this:

optim.SGD([{'params': model.base.parameters()},
           {'params': model.classifier.parameters(), 'lr': 1e-3}],
           lr=1e-2, momentum=0.9)

where model.classifier.parameters(), which defines a group of parameters obtains a specific learning rate of 1e-3.

But how can I translate this into parameter level?


title: Pytorch lightning, tensorboard: TypeError: can't pickle _thread.lock objects

body:
this is my error. I don't know what's my error is. And I really do not know what to do right now. I am using PyTorch-lightning with tensorboard. I do not where this error comes from, I can provide more details if you ask. I delete several error information because StackOverflow doesn't allow me to post so much code.

  File "/home/jq/PycharmProjects/Unet/Code/Lit_train.py", line 49, in <module>
    trainer.fit(model)
  File "/home/jq/.local/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 859, in fit
    self.single_gpu_train(model)
  File "/home/jq/.local/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 503, in single_gpu_train
    self.run_pretrain_routine(model)
  File "/home/jq/.local/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1015, in run_pretrain_routine
    self.train()
  File "/home/jq/.local/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 347, in train
    self.run_training_epoch()
  File "/home/jq/.local/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 451, in run_training_epoch
    self.run_evaluation(test_mode=self.testing)
  File "/home/jq/.local/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 391, in run_evaluation
    self.log_metrics(log_metrics, {})
  File "/home/jq/.local/lib/python3.6/site-packages/pytorch_lightning/trainer/logging.py", line 74, in log_metrics
    self.logger.save()
  File "/home/jq/.local/lib/python3.6/site-packages/pytorch_lightning/utilities/distributed.py", line 10, in wrapped_fn
    return fn(*args, **kwargs)
  File "/home/jq/.local/lib/python3.6/site-packages/pytorch_lightning/loggers/tensorboard.py", line 161, in save
    save_hparams_to_yaml(hparams_file, self.hparams)
  File "/home/jq/.local/lib/python3.6/site-packages/pytorch_lightning/core/saving.py", line 151, in save_hparams_to_yaml
    yaml.dump(hparams, fp)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/__init__.py", line 290, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all
    dumper.represent(data)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent
    node = self.represent_data(data)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/representer.py", line 343, in represent_object
    'tag:yaml.org,2002:python/object:'+function_name, state)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/representer.py", line 346, in represent_object
    return self.represent_sequence(tag+function_name, args)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/representer.py", line 92, in represent_sequence
    node_item = self.represent_data(item)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/representer.py", line 343, in represent_object
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/home/jq/.local/lib/python3.6/site-packages/yaml/representer.py", line 317, in represent_object
    reduce = data.__reduce_ex__(2)
TypeError: can't pickle _thread.lock objects
Exception ignored in: <object repr() failed>
Traceback (most recent call last):
  File "/home/jq/.local/lib/python3.6/site-packages/tqdm/std.py", line 1086, in __del__
  File "/home/jq/.local/lib/python3.6/site-packages/tqdm/std.py", line 1293, in close
  File "/home/jq/.local/lib/python3.6/site-packages/tqdm/std.py", line 1471, in display
  File "/home/jq/.local/lib/python3.6/site-packages/tqdm/std.py", line 1089, in __repr__
  File "/home/jq/.local/lib/python3.6/site-packages/tqdm/std.py", line 1433, in format_dict
TypeError: 'NoneType' object is not iterable


title: How to debug(monitoring value of object in other class' function) in Pycharm

body:
I'm running seq2seq code in pycharm in order to study pytorch.

The code has many classes and these classes have many function.

I'd like to monitor value of objects in other function, so I'm running code in console one by one.

Is there any good way to this using debug?

I haven't done debug before.

Please help me..


title: Custom conv2d operation Pytorch

body:
I have tried a custom Conv2d function which has to work similar to nn.Conv2d but the multiplication and addition used inside nn.Conv2d are replaced with mymult(num1,num2) and myadd(num1,num2).

As per insight from very helpful forums 1,2 what i can do is try unfolding it and then do matrix multiplication. That @ part given in the code below can be done using loops with mymult() and myadd() as i believe this @ is doing matmul.

def convcheck():
    torch.manual_seed(123)
    batch_size = 2
    channels = 2

    h, w = 2, 2
    image = torch.randn(batch_size, channels, h, w) # input image
    out_channels = 3
    kh, kw = 1, 1# kernel size
    dh, dw = 1, 1 # stride
    size = int((h-kh+2*0)/dh+1)    #include padding in place of zero

    conv = nn.Conv2d(in_channels=channels, out_channels=out_channels, kernel_size=kw, padding=0,stride=dh ,bias=False)

    out = conv (image)
    #print('out', out)
    #print('out.size()', out.size())
    #print('')
    filt = conv.weight.data

    imageunfold = F.unfold(image,kernel_size=kh,padding=0,stride=dh)

    print("Unfolded image","\n",imageunfold,"\n",imageunfold.shape)
    kernels_flat = filt.view(out_channels,-1)
    print("Kernel Flat=","\n",kernels_flat,"\n",kernels_flat.shape)
    res = kernels_flat @ imageunfold        # I have to replace this operation with mymult() and myadd()
    print(res,"\n",res.shape)
    #print(res.size(2),"\n",res.shape)
    res = res.view(-1, out_channels, size, size)
    #print("Same answer as buitlin function",res)

res = kernels_flat @ imageunfold can be replaced with this. although there can be some other efficient implementation which i am looking to get help for.

     for m_batch in range(len(imageunfold)):
        #iterate through rows of X
        # iterate through columns of Y
        for j in range(imageunfold.size(2)):
            # iterate through rows of Y
            for k in range(imageunfold.size(1)):
                #print(result[m_batch][i][j]," +=",   kernels_flat[i][k], "*", imageunfold[m_batch][k][j])
                result[m_batch][i][j] +=   kernels_flat[i][k] * imageunfold[m_batch][k][j]

Can someone please help me vectorize these three loops for faster execution.


title: Train test split Fastai

body:
I am using numpy.random.seed() to split the dataset in same order every time I am going to train using keras.

I have a doubt when I am trying fastai, is that numpy.random.seed() function will work same as in keras, where I am using ImageDataBunch.from_folder() function to load the dataset...

If not, what to do to split the dataset in same order every time I load the dataset in fastai

Thanks


I'm trying to implement ResNet18 on pyTorch but I'm having some troubles with it. My code is this:
device = torch.device("cuda:0")

class ResnetBlock(nn.Module):
    def __init__(self, strides, nf, nf0, reps, bn):
        super(ResnetBlock, self).__init__()
        self.adapt = strides == 2
        self.layers = []
        self.relus = []
        self.adapt_layer = nn.Conv2d(nf0, nf, kernel_size=1, stride=strides, padding=0) if self.adapt else None
        for i in range(reps):
            self.layers.append(nn.Sequential(
                nn.Conv2d(nf0, nf, kernel_size=3, stride=strides, padding=1),
                nn.BatchNorm2d(nf, eps=0.001, momentum=0.99),
                nn.ReLU(),
                nn.Conv2d(nf, nf, kernel_size=3, stride=1, padding=1),
                nn.BatchNorm2d(nf, eps=0.001, momentum=0.99)))
            self.relus.append(nn.ReLU())

            strides = 1
            nf0 = nf

    def forward(self, x):
        for i, (layer, relu) in enumerate(zip(self.layers, self.relus)):
            rama = layer(x)
            if self.adapt and i == 0:
                x = self.adapt_layer(x)
            x = x + rama
            x = relu(x)
        return x

class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3),
            nn.MaxPool2d(kernel_size=2, stride=2))

        self.blocks = nn.Sequential(
            ResnetBlock(1, 64, 64, 2, bn),
            ResnetBlock(2, 128, 64, 2, bn),
            ResnetBlock(2, 256, 128, 2, bn),
            ResnetBlock(2, 512, 256, 2, bn))

        self.fcout = nn.Linear(512, 10)

    def forward(self, x):
        out = self.layer1(x)
        out = self.blocks(out)
        out = out.reshape(out.size(0), -1)
        out = self.fcout(out)
        return out

num_epochs = 50
num_classes = 10
batch_size = 50
learning_rate = 0.00001

trans = transforms.ToTensor()
train_dataset = torchvision.datasets.CIFAR10(root="./dataset_pytorch", train=True, download=True, transform=trans)
test_dataset = torchvision.datasets.CIFAR10(root="./dataset_pytorch", train=False, download=True, transform=trans)

train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False)

def weights_init(m):
    if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):
        nn.init.xavier_uniform_(m.weight.data)
        nn.init.zeros_(m.bias.data)

model = ConvNet()
model.apply(weights_init)

model.to(device)

summary(model, (3,32,32))

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, eps=1e-6)

# Train the model
total_step = len(train_loader)
loss_list = []
acc_list = []
acc_list_test = []
for epoch in range(num_epochs):
    total = 0
    correct = 0
    for i, (images, labels) in enumerate(train_loader):
        images = images.to(device)
        labels = labels.to(device)

        optimizer.zero_grad()

        # Run the forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)
        loss_list.append(loss.item())

        # Backprop and perform Adam optimisation
        loss.backward()
        optimizer.step()

        # Track the accuracy
        total += labels.size(0)
        _, predicted = torch.max(outputs.data, 1)
        correct += (predicted == labels).sum().item()
    acc_list.append(correct / total)

    print("Train")
    print('Epoch [{}/{}], Accuracy: {:.2f}%'
          .format(epoch + 1, num_epochs, (correct / total) * 100))

    total_test = 0
    correct_test = 0
    for i, (images, labels) in enumerate(test_loader):
        images = images.to(device)
        labels = labels.to(device)

        # Run the forward pass
        outputs = model(images)

        # Track the accuracy
        total_test += labels.size(0)
        _, predicted = torch.max(outputs.data, 1)
        correct_test += (predicted == labels).sum().item()
    acc_list_test.append(correct_test / total_test)
    print("Test")
    print('Epoch [{}/{}], Accuracy: {:.2f}%'
          .format(epoch + 1, num_epochs, (correct_test / total_test) * 100))

It's weird because it's throwing me that error Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same even though I've moved both the model and the data to cuda.
I guess it's related with how I defined or used "ResnetBlock", because if I remove from ConvNet those blocks (removing the line out = self.blocks(out)), the code works. But I don't know what I'm doing wrong.


title: how to convert HuggingFace's Seq2seq models to onnx format

body:
I am trying to convert the Pegasus newsroom in HuggingFace's transformers model to the ONNX format. I followed this guide published by Huggingface. After installing the prereqs, I ran this code:
!rm -rf onnx/
from pathlib import Path
from transformers.convert_graph_to_onnx import convert

convert(framework="pt", model="google/pegasus-newsroom", output=Path("onnx/google/pegasus-newsroom.onnx"), opset=11)

and got these errors:
ValueError                                Traceback (most recent call last)
<ipython-input-9-3b37ed1ceda5> in <module>()
      3 from transformers.convert_graph_to_onnx import convert
      4
----> 5 convert(framework="pt", model="google/pegasus-newsroom", output=Path("onnx/google/pegasus-newsroom.onnx"), opset=11)
      6
      7

6 frames
/usr/local/lib/python3.6/dist-packages/transformers/models/pegasus/modeling_pegasus.py in forward(self, input_ids, attention_mask, encoder_hidden_states, encoder_attention_mask, head_mask, encoder_head_mask, past_key_values, inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
    938             input_shape = inputs_embeds.size()[:-1]
    939         else:
--> 940             raise ValueError("You have to specify either decoder_input_ids or decoder_inputs_embeds")
    941
    942         # past_key_values_length

ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds

I have never seen this error before. Any ideas?


title: RuntimeError: The size of tensor a (256) must match the size of tensor b (32) at non-singleton dimension 3

body:
I just trained a model and now performing iteration on custom images with different sizes and types (.tif,,bmp,and .PNG).
I just load a model and want to generate three images, compressed, original and final images. The below code works fine when i put into training file because where i i gave CIFAR dataset for testing. Now i am giving custom images and make separate file for testing.
import torch
from model import End_to_end
from torch.autograd import Variable
from loss import loss_function
from grid import save_image
from torchvision import datasets, transforms

CUDA = torch.cuda.is_available()
if CUDA:
    model = End_to_end().cuda()
else:
    model = End_to_end()

EPOCHS = 20

testset = datasets.ImageFolder(root="/home/khawar/Desktop/End-to-End_IEEE-TVSCT/test/", transform=transforms.ToTensor())
model.load_state_dict(torch.load('./checkpoint/model.pth'))
print(testset.imgs)

def test(epoch):
    model.eval()
    test_loss = 0
    for i, (data, _) in enumerate(testset):
        data = Variable(data)
        final, residual_img, upscaled_image, com_img, orig_im = model(data.cuda())
        test_loss += loss_function(final, residual_img, upscaled_image, com_img, orig_im).data
        if epoch == EPOCHS and i == 0:
            #             save_image(final.data[0],'reconstruction_final',nrow=8)
            #             save_image(com_img.data[0],'com_img',nrow=8)
            n = min(data.size(0), 6)
            print("saving the image " + str(n))
            comparison = torch.cat([data[:n], final[:n].cpu()])
            comparison = comparison.cpu()
            #             print(comparison.data)
            save_image(com_img[:n].data, 'compressed_' + str(epoch) + '.png', nrow=n)
            save_image(comparison.data, 'reconstruction_' + str(epoch) + '.png', nrow=n)

    test_loss /= len(testset.dataset)
    print('====> Test set loss: {:.4f}'.format(test_loss))

def save_images():
    epoch = EPOCHS
    model.eval()
    test_loss = 0
    for i, (data, _) in enumerate(testset):
        data = Variable(data).unsqueeze(0)
        final, residual_img, upscaled_image, com_img, orig_im = model(data.cuda())
        test_loss += loss_function(final, residual_img, upscaled_image, com_img, orig_im).data
        if i == 3:
            #             save_image(final.data[0],'reconstruction_final',nrow=8)
            #             save_image(com_img.data[0],'com_img',nrow=8)
            n = min(data.size(0), 6)
            print("saving the image " + str(n))
            comparison = torch.cat([data[:n], final[:n].cpu()])
            comparison = comparison.cpu()
            #             print(comparison.data)
            save_image(com_img[:1].data, './compressed_image/compressed_' + str(i) + '.png', nrow=n)
            save_image(final[:1].data, './final_image/final_' + str(epoch) + '.png', nrow=n)
            save_image(orig_im[:1].data, './orginal_image/original_' + str(epoch) + '.png', nrow=n)

    test_loss /= len(testset.dataset)
    print('====> Test set loss: {:.4f}'.format(test_loss))

save_images()

Error
Traceback (most recent call last):
  File "/home/khawar/Desktop/End-to-End_IEEE-TVSCT/test.py", line 74, in <module>
    save_images()
  File "/home/khawar/Desktop/End-to-End_IEEE-TVSCT/test.py", line 51, in save_images
    for i, (data, _) in enumerate(test_loader):
  File "/home/khawar/anaconda3/envs/End-to-End_IEEE-TVSCT/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/khawar/anaconda3/envs/End-to-End_IEEE-TVSCT/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/khawar/anaconda3/envs/End-to-End_IEEE-TVSCT/lib/python3.5/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/khawar/anaconda3/envs/End-to-End_IEEE-TVSCT/lib/python3.5/site-packages/torch/utils/data/_utils/collate.py", line 79, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/home/khawar/anaconda3/envs/End-to-End_IEEE-TVSCT/lib/python3.5/site-packages/torch/utils/data/_utils/collate.py", line 79, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "/home/khawar/anaconda3/envs/End-to-End_IEEE-TVSCT/lib/python3.5/site-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [3, 288, 352] at entry 0 and [3, 256, 256] at entry 1


title: Python: BrokenPipeError when calling iter on a torch.utils.data.DataLoader object

body:
I'm currently following a PyTorch tutorial, and I'm trying to get this example to work. However, upon running the line dataiter = iter(trainloader) my program first does something I don't understand how it's possible – it seems to jump backwards in the program and re-run lines of codes which it has already run even though there is no loop. Then it runs into a BrokenPipeError and crashes.

Here is a minimal working example that triggers the problem:

import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose([transforms.ToTensor(),
                                transforms.Normalize((0.5, 0.5, 0.5),
                                                     (0.5, 0.5, 0.5))])
trainset = torchvision.datasets.CIFAR10(root='./data',
                                        train=True,
                                        download=True,
                                        transform=transform)
trainloader = torch.utils.data.DataLoader(trainset,
                                          batch_size=4,
                                          shuffle=True,
                                          num_workers=2)
dataiter = iter(trainloader)

and this is the output I'm getting:

"D:\Program Files\Python35\python.exe" D:/work/programs/pytorch/60_minute_blitz_tutorial/3_training_a_classifier/MWE.py
Files already downloaded and verified
Files already downloaded and verified
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\Program Files\Python35\lib\multiprocessing\spawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "D:\Program Files\Python35\lib\multiprocessing\spawn.py", line 115, in _main
    prepare(preparation_data)
  File "D:\Program Files\Python35\lib\multiprocessing\spawn.py", line 226, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "D:\Program Files\Python35\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
    run_name="__mp_main__")
  File "D:\Program Files\Python35\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "D:\Program Files\Python35\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "D:\Program Files\Python35\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\work\programs\pytorch\60_minute_blitz_tutorial\3_training_a_classifier\MWE.py", line 16, in <module>
    dataiter = iter(trainloader)
  File "D:\Program Files\Python35\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
    return _DataLoaderIter(self)
  File "D:\Program Files\Python35\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
    w.start()
  File "D:\Program Files\Python35\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "D:\Program Files\Python35\lib\multiprocessing\context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "D:\Program Files\Python35\lib\multiprocessing\context.py", line 313, in _Popen
    return Popen(process_obj)
  File "D:\Program Files\Python35\lib\multiprocessing\popen_spawn_win32.py", line 34, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "D:\Program Files\Python35\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
    _check_not_importing_main()
  File "D:\Program Files\Python35\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "D:/work/programs/pytorch/60_minute_blitz_tutorial/3_training_a_classifier/MWE.py", line 16, in <module>
    dataiter = iter(trainloader)
  File "D:\Program Files\Python35\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
    return _DataLoaderIter(self)
  File "D:\Program Files\Python35\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
    w.start()
  File "D:\Program Files\Python35\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "D:\Program Files\Python35\lib\multiprocessing\context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "D:\Program Files\Python35\lib\multiprocessing\context.py", line 313, in _Popen
    return Popen(process_obj)
  File "D:\Program Files\Python35\lib\multiprocessing\popen_spawn_win32.py", line 66, in __init__
    reduction.dump(process_obj, to_child)
  File "D:\Program Files\Python35\lib\multiprocessing\reduction.py", line 59, in dump
    ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe

Process finished with exit code 1

The output "Files already downloaded and verified" should only be printed once – when creating trainset – but it's printed twice.

Why is this? And why am I getting a BrokenPipeError? And most importantly, how do I fix this?

I'm running Windows 10, Python 3.5.3, CUDA 8.0.60 and a PyTorch 0.4.0 that I have installed by running

pip install http://download.pytorch.org/whl/cu80/torch-0.4.0-cp35-cp35m-win_amd64.whl
pip install torchvision

as instructed by the PyTorch webpage (well, they use pip3 instead of pip, but that shouldn't make any difference since Python 3.5 is the only Python version I have installed).


title: Getting Bad Images After Data Augmentation in PyTorch

body:
I'm working on a nuclear segmentation problem where I'm trying to identify the positions of nuclei in images of stained tissues. The given training dataset has a picture of the stained tissue and a mask with the nuclei positions. Since the dataset was small, I wanted to try data augmentation in PyTorch, but after doing that, for some reason, when I output my mask image, it looks fine, but the corresponding tissue image is incorrect.

All my training images are in X_train with shape (128, 128, 3), the corresponding masks in Y_train with shape (128, 128, 1) and similarly the cross-validation images and masks in X_val and Y_val respectively.

Y_train and Y_val have dtype = np.bool, X_train and X_val have dtype = np.uint8.

Before data augmentation, I check my images like this:

fig, axis = plt.subplots(2, 2)
axis[0][0].imshow(X_train[0].astype(np.uint8))
axis[0][1].imshow(np.squeeze(Y_train[0]).astype(np.uint8))
axis[1][0].imshow(X_val[0].astype(np.uint8))
axis[1][1].imshow(np.squeeze(Y_val[0]).astype(np.uint8))

The output is as follows:
Before Data Augmentation

For the data augmentation, I define a custom class as follows:

Here I have imported torchvision.transforms.functional as TF and torchvision.transforms as transforms. images_np and masks_np are the inputs which are numpy arrays.

class Nuc_Seg(Dataset):
def __init__(self, images_np, masks_np):
    self.images_np = images_np
    self.masks_np = masks_np

def transform(self, image_np, mask_np):
    ToPILImage = transforms.ToPILImage()
    image = ToPILImage(image_np)
    mask = ToPILImage(mask_np.astype(np.int32))

    angle = random.uniform(-10, 10)
    width, height = image.size
    max_dx = 0.2 * width
    max_dy = 0.2 * height
    translations = (np.round(random.uniform(-max_dx, max_dx)), np.round(random.uniform(-max_dy, max_dy)))
    scale = random.uniform(0.8, 1.2)
    shear = random.uniform(-0.5, 0.5)
    image = TF.affine(image, angle = angle, translate = translations, scale = scale, shear = shear)
    mask = TF.affine(mask, angle = angle, translate = translations, scale = scale, shear = shear)

    image = TF.to_tensor(image)
    mask = TF.to_tensor(mask)
    return image, mask

def __len__(self):
    return len(self.images_np)

def __getitem__(self, idx):
    image_np = self.images_np[idx]
    mask_np = self.masks_np[idx]
    image, mask = self.transform(image_np, mask_np)

    return image, mask

This is followed by:

I have used from torch.utils.data import DataLoader

train_dataset = Nuc_Seg(X_train, Y_train)
train_loader = DataLoader(train_dataset, batch_size = 16, shuffle = True)
val_dataset = Nuc_Seg(X_val, Y_val)
val_loader = DataLoader(val_dataset, batch_size = 16, shuffle = True)

After this step, I try to check on my first set training image and mask using this:

%matplotlib inline

for ex_img, ex_mask in train_loader:

    img = ex_img[0]
    img = img.reshape(128, 128, 3)
    mask = ex_mask[0]
    mask = mask.reshape(128, 128)

    img = img.numpy()
    mask = mask.numpy()

    fig, (axis_1, axis_2) = plt.subplots(1, 2)
    axis_1.imshow(img.astype(np.uint8))
    axis_2.imshow(mask.astype(np.uint8))

    break

I get this as my output:
After Data Augmentation 1

When I change the line axis_1.imshow(img.astype(np.uint8)) to axis_1.imshow(img),

I get this image:
After Data Augmentation 2

The images of the mask are correct, but for some reason, the images of nuclei are wrong. With the .astype(np.uint8), the tissue image is completely black.

Without the .astype(np.uint8), the positions of the nuclei are correct, but the color scheme is all messed up (I expect images like those seen before data augmentation, either gray-ish or pink-ish), plus 9 copies of the same image in a grid are displayed for some reason. Can you please help me get the correct output of the tissue images?


I am implementing nn.DataParallel class in code and i encapsulate all my models in this class. It gives an error ambiguous error.
Class Code
import time
import os
import argparse

import numpy as np

import torch
import torch.optim as optim
import torch.optim.lr_scheduler as LS
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.utils.data as data
from torchvision import transforms

parser = argparse.ArgumentParser()
parser.add_argument(
    '--batch-size', '-N', type=int, default=16, help='batch size')
parser.add_argument(
    '--train', '-f', required=True, type=str, help='folder of training images')
parser.add_argument(
    '--max-epochs', '-e', type=int, default=4, help='max epochs')
parser.add_argument('--lr', type=float, default=0.0005, help='learning rate')
parser.add_argument('--cuda', '-g', action='store_true', help='enables cuda')
parser.add_argument(
    '--iterations', type=int, default=16, help='unroll iterations')
parser.add_argument('--checkpoint', type=int, help='unroll iterations')
args = parser.parse_args()

## load 32x32 patches from images
import dataset

train_transform = transforms.Compose([
    transforms.ToTensor(),
])

train_set = dataset.ImageFolder(root=args.train, transform=train_transform)

train_loader = data.DataLoader(
    dataset=train_set, batch_size=args.batch_size, shuffle=True, num_workers=1)

print('total images: {}; total batches: {}'.format(
    len(train_set), len(train_loader)))

## load networks on GPU
import network

print("Devices are ", torch.cuda.device_count())

torch.cuda.set_device(1)

encoder = nn.DataParallel(network.EncoderCell(), device_ids = [1, 3])
binarizer = nn.DataParallel(network.Binarizer(), device_ids = [1, 3])
decoder = nn.DataParallel(network.DecoderCell(), device_ids = [1, 3])

solver = optim.Adam(
    [
        {
            'params': encoder.parameters()
        },
        {
            'params': binarizer.parameters()
        },
        {
            'params': decoder.parameters()
        },
    ],
    lr=args.lr)

def resume(epoch=None):
    if epoch is None:
        s = 'iter'
        epoch = 0
    else:
        s = 'epoch'

    encoder.load_state_dict(
        torch.load('checkpoint/encoder_{}_{:08d}.pth'.format(s, epoch)))
    binarizer.load_state_dict(
        torch.load('checkpoint/binarizer_{}_{:08d}.pth'.format(s, epoch)))
    decoder.load_state_dict(
        torch.load('checkpoint/decoder_{}_{:08d}.pth'.format(s, epoch)))

def save(index, epoch=True):
    if not os.path.exists('checkpoint'):
        os.mkdir('checkpoint')

    if epoch:
        s = 'epoch'
    else:
        s = 'iter'

    torch.save(encoder.state_dict(), 'checkpoint/encoder_{}_{:08d}.pth'.format(
        s, index))

    torch.save(binarizer.state_dict(),
               'checkpoint/binarizer_{}_{:08d}.pth'.format(s, index))

    torch.save(decoder.state_dict(), 'checkpoint/decoder_{}_{:08d}.pth'.format(
        s, index))

# resume()

scheduler = LS.MultiStepLR(solver, milestones=[3, 10, 20, 50, 100], gamma=0.5)

last_epoch = 0
if args.checkpoint:
    resume(args.checkpoint)
    last_epoch = args.checkpoint
    scheduler.last_epoch = last_epoch - 1

for epoch in range(last_epoch + 1, args.max_epochs + 1):

    scheduler.step()

    for batch, data in enumerate(train_loader):
        batch_t0 = time.time()

        ## init lstm state
        encoder_h_1 = (Variable(torch.zeros(data.size(0), 256, 8, 8).cuda()),
                       Variable(torch.zeros(data.size(0), 256, 8, 8).cuda()))
        # print(encoder_h_1)
        encoder_h_2 = (Variable(torch.zeros(data.size(0), 512, 4, 4).cuda()),
                       Variable(torch.zeros(data.size(0), 512, 4, 4).cuda()))
        encoder_h_3 = (Variable(torch.zeros(data.size(0), 512, 2, 2).cuda()),
                       Variable(torch.zeros(data.size(0), 512, 2, 2).cuda()))

        decoder_h_1 = (Variable(torch.zeros(data.size(0), 512, 2, 2).cuda()),
                       Variable(torch.zeros(data.size(0), 512, 2, 2).cuda()))
        decoder_h_2 = (Variable(torch.zeros(data.size(0), 512, 4, 4).cuda()),
                       Variable(torch.zeros(data.size(0), 512, 4, 4).cuda()))
        decoder_h_3 = (Variable(torch.zeros(data.size(0), 256, 8, 8).cuda()),
                       Variable(torch.zeros(data.size(0), 256, 8, 8).cuda()))
        decoder_h_4 = (Variable(torch.zeros(data.size(0), 128, 16, 16).cuda()),
                       Variable(torch.zeros(data.size(0), 128, 16, 16).cuda()))

        patches = Variable(data.cuda())

        solver.zero_grad()

        losses = []

        res = patches - 0.5

        bp_t0 = time.time()

        for _ in range(args.iterations):
            encoded, encoder_h_1, encoder_h_2, encoder_h_3 = encoder(
                res, encoder_h_1, encoder_h_2, encoder_h_3)

            codes = binarizer(encoded)

            output, decoder_h_1, decoder_h_2, decoder_h_3, decoder_h_4 = decoder(
                codes, decoder_h_1, decoder_h_2, decoder_h_3, decoder_h_4)

            res = res - output
            losses.append(res.abs().mean())

        bp_t1 = time.time()

        loss = sum(losses) / args.iterations
        loss.backward()

        solver.step()

        batch_t1 = time.time()

        print(
            '[TRAIN] Epoch[{}]({}/{}); Loss: {:.6f}; Backpropagation: {:.4f} sec; Batch: {:.4f} sec'.
                format(epoch, batch + 1,
                       len(train_loader), loss.data, bp_t1 - bp_t0, batch_t1 -
                       batch_t0))
        print(('{:.4f} ' * args.iterations +
               '\n').format(*[l.data for l in losses]))

        index = (epoch - 1) * len(train_loader) + batch

        ## save checkpoint every 500 training steps
        if index % 500 == 0:
            save(0, False)

    save(epoch)

TraceBack.
total images: 9271670; total batches: 579480
Devices are  4
/data1/khawar/khawar/Conference/CVPR/lib/python3.5/site-packages/torch/optim/lr_scheduler.py:82: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule.See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
Traceback (most recent call last):
  File "train.py", line 152, in <module>
    res, encoder_h_1, encoder_h_2, encoder_h_3)
  File "/data1/khawar/khawar/Conference/CVPR/lib/python3.5/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/data1/khawar/khawar/Conference/CVPR/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 146, in forward
    "them on device: {}".format(self.src_device_obj, t.device))
RuntimeError: module must have its parameters and buffers on device cuda:1 (device_ids[0]) but found one of them on device: cpu


title: How can I fix "ValueError: Target size (torch.Size([128])) must be the same as input size (torch.Size([128, 1]))" error?

body:
I am trying to train GAN with below code:
def run_a_gan(D, G, D_solver, G_solver, discriminator_loss, generator_loss, show_every=250,
             batch_size=128, noise_size=96, num_epochs=10):
 """
 Train a GAN!

 Inputs:
 - D, G: PyTorch models for the discriminator and generator
 - D_solver, G_solver: torch.optim Optimizers to use for training the
   discriminator and generator.
 - discriminator_loss, generator_loss: Functions to use for computing the generator and
   discriminator loss, respectively.
 - show_every: Show samples after every show_every iterations.
 - batch_size: Batch size to use for training.
 - noise_size: Dimension of the noise to use as input to the generator.
 - num_epochs: Number of epochs over the training dataset to use for training.
 """
 iter_count = 0
 for epoch in range(num_epochs):
   for x, _ in loader_train:
     if len(x) != batch_size:
       continue
     D_solver.zero_grad()
     real_data = x.to(device)
     logits_real = D(2* (real_data - 0.5))

     g_fake_seed = sample_noise(batch_size, noise_size, dtype=real_data.dtype, device=real_data.device)
     fake_images = G(g_fake_seed).detach()
     logits_fake = D(fake_images.view(batch_size, 1, 28, 28))

     d_total_error = discriminator_loss(logits_real, logits_fake)
     d_total_error.backward()
     D_solver.step()

     G_solver.zero_grad()
     g_fake_seed = sample_noise(batch_size, noise_size, dtype=real_data.dtype, device=real_data.device)
     fake_images = G(g_fake_seed)

     gen_logits_fake = D(fake_images.view(batch_size, 1, 28, 28))
     g_error = generator_loss(gen_logits_fake)
     g_error.backward()
     G_solver.step()

     if (iter_count % show_every == 0):
       print('Iter: {}, D: {:.4}, G:{:.4}'.format(iter_count,d_total_error.item(),g_error.item()))
       imgs_numpy = fake_images.data.cpu()#.numpy()
       show_images(imgs_numpy[0:16])
       plt.show()
       print()
     iter_count += 1

with
fix_random_seed(0)

# Make the discriminator
D = discriminator().to(device)

# Make the generator
G = generator().to(device)

# Use the function you wrote earlier to get optimizers for the Discriminator and the Generator
D_solver = get_optimizer(D)
G_solver = get_optimizer(G)
# Run it!
run_a_gan(D, G, D_solver, G_solver, discriminator_loss, generator_loss)

But I got an error that given in this issue:

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-19-bc85cbbdf398> in <module>()
     11 G_solver = get_optimizer(G)
     12 # Run it!
---> 13 run_a_gan(D, G, D_solver, G_solver, discriminator_loss, generator_loss)

2 frames

<ipython-input-18-701b71b7afd2> in run_a_gan(D, G, D_solver, G_solver, discriminator_loss, generator_loss, show_every, batch_size, noise_size, num_epochs)
     28       logits_fake = D(fake_images.view(batch_size, 1, 28, 28))
     29
---> 30       d_total_error = discriminator_loss(logits_real, logits_fake)
     31       d_total_error.backward()
     32       D_solver.step()

<ipython-input-14-e1f56d791cff> in discriminator_loss(logits_real, logits_fake)
     17   N = logits_real.shape[0]
     18   device = logits_real.device
---> 19   loss = (bce_loss(logits_real, torch.ones(N).to(dtype).to(device)))+(bce_loss(logits_fake, torch.zeros(N).to(dtype).to(device)))
     20
     21   ##############################################################################

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction, pos_weight)
   2578
   2579     if not (target.size() == input.size()):
-> 2580         raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
   2581
   2582     return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)

ValueError: Target size (torch.Size([128])) must be the same as input size (torch.Size([128, 1]))

Actually, before this 2 code snippet, I did not get any error related to my implementation. All expected results return true values but when I'm trying to do these steps I failed, how can I solve this problem? I think the real problem consist of array size, but I do not know where I need to look.
After Ivan's answer I updated such code snippets like this:
  logits_real_loss = bce_loss(logits_real, torch.ones(N, 1).to(dtype).to(device))
  logits_fake_loss = bce_loss(logits_fake, torch.zeros(N, 1).to(dtype).to(device))
  loss = logits_real_loss + logits_fake_loss

and
  N = logits_fake.shape[0]
  device = logits_fake.device
  loss = (bce_loss(logits_fake, torch.ones(N,1).to(dtype).to(device)))

But I am getting other value error:
---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-15-b089f365cd5e> in <module>()
      2   d_loss = discriminator_loss(logits_real, logits_fake)
      3   print("Maximum error in d_loss: %g"%rel_error(d_loss_true, d_loss))
----> 4 test_discriminator_loss(answers['logits_real'], answers['logits_fake'],answers['d_loss_true'])

2 frames

<ipython-input-15-b089f365cd5e> in test_discriminator_loss(logits_real, logits_fake, d_loss_true)
      1 def test_discriminator_loss(logits_real, logits_fake, d_loss_true):
----> 2   d_loss = discriminator_loss(logits_real, logits_fake)
      3   print("Maximum error in d_loss: %g"%rel_error(d_loss_true, d_loss))
      4 test_discriminator_loss(answers['logits_real'], answers['logits_fake'],answers['d_loss_true'])

<ipython-input-14-15d437ae3425> in discriminator_loss(logits_real, logits_fake)
     17   N = logits_real.shape[0]
     18   device = logits_real.device
---> 19   logits_real_loss = bce_loss(logits_real, torch.ones(N, 1).to(dtype).to(device))
     20   logits_fake_loss = bce_loss(logits_fake, torch.zeros(N, 1).to(dtype).to(device))
     21   loss = logits_real_loss + logits_fake_loss

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction, pos_weight)
   2578
   2579     if not (target.size() == input.size()):
-> 2580         raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
   2581
   2582     return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)

ValueError: Target size (torch.Size([10, 1])) must be the same as input size (torch.Size([10]))

how can I proceed?


title: Long Sequence In a seq2seq model with attention?

body:
I am following along this pytorch tutorial and trying to apply this principle to summarization, where the encoding sequence would be around 1000 words and decoder target 200 words.
How do I apply seq2seq to this? I know it would be very expensive and almost infeasible to run through the whole sequence of 1000 words at once. So dividing the seq into say 20 seq and running in parallel could be an answer. But I'm not sure how to implement it; I also want to incorporate attention into it.


title: Issue while reshaping a torch tensor from [10,200,1] to [2000,1,1]

body:
I am having a problem when trying to reshape a torch tensor Yp of dimensions [10,200,1] to [2000,1,1]. The tensor is obtained from a numpy array y of dimension [2000,1]. I am doing the following:

Yp = reshape(Yp, (-1,1,1))

I try to subtract the result to a torch tensor version of y by doing:

Yp[0:2000,0] - torch.from_numpy(y[0:2000,0])

I expect the result to be an array of zeros, but that is not the case. Calling different orders when reshaping (order = 'F' or 'C') does not solve the problem, and strangely outputs the same result when doing the subtraction. I only manage to get an array of zeros by calling on the tensor Yp the ravel method with order = 'F'.

What am I doing wrong? I would like to solve this using reshape!