title: How to implement some trainable parameters in the model of Keras like nn.Parameters() in Pytorch?

body:
I just wanna to implement some trainable parameters in my model with Keras. In Pytorch, we can do it by using torch.nn.Parameter() like below:

self.a = nn.Parameter(torch.ones(8))
self.b = nn.Parameter(torch.zeros(16,8))

I think by doing this in pytorch it can add some trainable parameters into the model. And now I wanna to know, how to achieve similar operations in keras？
Any suggestions or advice are welcomed!

THX! :)

p.s. I just write a custom layer in Keras as below:

class Mylayer(Layer):

    def __init__(self,input_dim,output_dim,**kwargs):
        self.input_dim = input_dim
        self.output_dim = output_dim
        super(Mylayer,self).__init__(**kwargs)

    def build(self):

        self.kernel = self.add_weight(name='pi',
                                      shape=(self.input_dim,self.output_dim),
                                      initializer='zeros',
                                      trainable=True)
        self.kernel_2 = self.add_weight(name='mean',
                                        shape=(self.input_dim,self.output_dim),
                                        initializer='ones',
                                        trainable=True)

        super(Mylayer,self).build()

    def call(self,x):
        return x,self.kernel,self.kernel_2

and I wanna to know if I haven't change the tensor which pass through the layer, should I write the function def compute_output_shape() for necessary?


title: How to simplify DataLoader for Autoencoder in Pytorch

body:
Is there any easier way to set up the dataloader, because input and target data is the same in case of an autoencoder and to load the data during training? The DataLoader always requires two inputs.

Currently I define my dataloader like this:

X_train     = rnd.random((300,100))
X_val       = rnd.random((75,100))
train       = data_utils.TensorDataset(torch.from_numpy(X_train).float(), torch.from_numpy(X_train).float())
val         = data_utils.TensorDataset(torch.from_numpy(X_val).float(), torch.from_numpy(X_val).float())
train_loader= data_utils.DataLoader(train, batch_size=1)
val_loader  = data_utils.DataLoader(val, batch_size=1)

and train like this:

for epoch in range(50):
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = Variable(data), Variable(target).detach()
        optimizer.zero_grad()
        output = model(data, x)
        loss = criterion(output, target)


title: Parallel Cholesky decomposition in PyTorch GPU

body:
To get a differentiable term containing the determinant of a D-dimensional positive-definite matrix C (differential entropy of a multivariate Gaussian in my case), I can use:
torch.log2(torch.potrf(C).diag()).sum() + D / 2.0 * (np.log2(2 * np.pi * np.e))

potrf(C) performs Cholesky decomposition, whose diagonal elements' log values sum to the log determinant divided by 2.
I want to call potrf on a mini-batch of matrices, so that calling potrf on a Tensor of shape (N, D, D) produces N different Cholesky decompositions.
At the moment I can only call potrf() repeatedly in a Python loop, which is a poor use of the GPU's parallel compute capability and as a result runs about 3 times slower than CPU.
Is it possible to launch Cholesky decomposition on GPU in parallel with PyTorch?


title: Have I implemented implemenation of learning rate finder correctly?

body:
Using implementation of lr_finder from https://github.com/davidtvs/pytorch-lr-finder based on paper https://arxiv.org/abs/1506.01186

Without the learning rate finder :

from __future__ import print_function, with_statement, division
import torch
from tqdm.autonotebook import tqdm
from torch.optim.lr_scheduler import _LRScheduler
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch.utils.data as data_utils
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from matplotlib import pyplot
from pandas import DataFrame
import torchvision.datasets as dset
import os
import torch.nn.functional as F
import time
import random
import pickle
from sklearn.metrics import confusion_matrix
import pandas as pd
import sklearn

class LRFinder(object):
    """Learning rate range test.

    The learning rate range test increases the learning rate in a pre-training run
    between two boundaries in a linear or exponential manner. It provides valuable
    information on how well the network can be trained over a range of learning rates
    and what is the optimal learning rate.

    Arguments:
        model (torch.nn.Module): wrapped model.
        optimizer (torch.optim.Optimizer): wrapped optimizer where the defined learning
            is assumed to be the lower boundary of the range test.
        criterion (torch.nn.Module): wrapped loss function.
        device (str or torch.device, optional): a string ("cpu" or "cuda") with an
            optional ordinal for the device type (e.g. "cuda:X", where is the ordinal).
            Alternatively, can be an object representing the device on which the
            computation will take place. Default: None, uses the same device as `model`.

    Example:
        >>> lr_finder = LRFinder(net, optimizer, criterion, device="cuda")
        >>> lr_finder.range_test(dataloader, end_lr=100, num_iter=100)

    Cyclical Learning Rates for Training Neural Networks: https://arxiv.org/abs/1506.01186
    fastai/lr_find: https://github.com/fastai/fastai

    """

    def __init__(self, model, optimizer, criterion, device=None):
        self.model = model
        self.optimizer = optimizer
        self.criterion = criterion
        self.history = {"lr": [], "loss": []}
        self.best_loss = None

        # Save the original state of the model and optimizer so they can be restored if
        # needed
        self.model_state = model.state_dict()
        self.model_device = next(self.model.parameters()).device
        self.optimizer_state = optimizer.state_dict()

        # If device is None, use the same as the model
        if device:
            self.device = device
        else:
            self.device = self.model_device

    def reset(self):
        """Restores the model and optimizer to their initial states."""
        self.model.load_state_dict(self.model_state)
        self.model.to(self.model_device)
        self.optimizer.load_state_dict(self.optimizer_state)

    def range_test(
        self,
        train_loader,
        val_loader=None,
        end_lr=10,
        num_iter=100,
        step_mode="exp",
        smooth_f=0.05,
        diverge_th=5,
    ):
        """Performs the learning rate range test.

        Arguments:
            train_loader (torch.utils.data.DataLoader): the training set data laoder.
            val_loader (torch.utils.data.DataLoader, optional): if `None` the range test
                will only use the training loss. When given a data loader, the model is
                evaluated after each iteration on that dataset and the evaluation loss
                is used. Note that in this mode the test takes significantly longer but
                generally produces more precise results. Default: None.
            end_lr (float, optional): the maximum learning rate to test. Default: 10.
            num_iter (int, optional): the number of iterations over which the test
                occurs. Default: 100.
            step_mode (str, optional): one of the available learning rate policies,
                linear or exponential ("linear", "exp"). Default: "exp".
            smooth_f (float, optional): the loss smoothing factor within the [0, 1[
                interval. Disabled if set to 0, otherwise the loss is smoothed using
                exponential smoothing. Default: 0.05.
            diverge_th (int, optional): the test is stopped when the loss surpasses the
                threshold:  diverge_th * best_loss. Default: 5.

        """
        # Reset test results
        self.history = {"lr": [], "loss": []}
        self.best_loss = None

        # Move the model to the proper device
        self.model.to(self.device)

        # Initialize the proper learning rate policy
        if step_mode.lower() == "exp":
            lr_schedule = ExponentialLR(self.optimizer, end_lr, num_iter)
        elif step_mode.lower() == "linear":
            lr_schedule = LinearLR(self.optimizer, end_lr, num_iter)
        else:
            raise ValueError("expected one of (exp, linear), got {}".format(step_mode))

        if smooth_f < 0 or smooth_f >= 1:
            raise ValueError("smooth_f is outside the range [0, 1[")

        # Create an iterator to get data batch by batch
        iterator = iter(train_loader)
        for iteration in tqdm(range(num_iter)):
            # Get a new set of inputs and labels
            try:
                inputs, labels = next(iterator)
            except StopIteration:
                iterator = iter(train_loader)
                inputs, labels = next(iterator)

            # Train on batch and retrieve loss
            loss = self._train_batch(inputs, labels)
            if val_loader:
                loss = self._validate(val_loader)

            # Update the learning rate
            lr_schedule.step()
            self.history["lr"].append(lr_schedule.get_lr()[0])

            # Track the best loss and smooth it if smooth_f is specified
            if iteration == 0:
                self.best_loss = loss
            else:
                if smooth_f > 0:
                    loss = smooth_f * loss + (1 - smooth_f) * self.history["loss"][-1]
                if loss < self.best_loss:
                    self.best_loss = loss

            # Check if the loss has diverged; if it has, stop the test
            self.history["loss"].append(loss)
            if loss > diverge_th * self.best_loss:
                print("Stopping early, the loss has diverged")
                break

        print("Learning rate search finished. See the graph with {finder_name}.plot()")

    def _train_batch(self, inputs, labels):
        # Set model to training mode
#         self.model.train()

        # Move data to the correct device
        inputs = inputs.to(self.device)
        labels = labels.to(self.device)

        # Forward pass
        self.optimizer.zero_grad()
        outputs = self.model(inputs)
        loss = self.criterion(outputs, labels)

        # Backward pass
        loss.backward()
        self.optimizer.step()

        return loss.item()

    def _validate(self, dataloader):
        # Set model to evaluation mode and disable gradient computation
        running_loss = 0
        self.model.eval()
        with torch.no_grad():
            for inputs, labels in dataloader:
                # Move data to the correct device
                inputs = inputs.to(self.device)
                labels = labels.to(self.device)

                # Forward pass and loss computation
                outputs = self.model(inputs)
                loss = self.criterion(outputs, labels)
                running_loss += loss.item() * inputs.size(0)

        return running_loss / len(dataloader.dataset)

    def plot(self, skip_start=10, skip_end=5, log_lr=True):
        """Plots the learning rate range test.

        Arguments:
            skip_start (int, optional): number of batches to trim from the start.
                Default: 10.
            skip_end (int, optional): number of batches to trim from the start.
                Default: 5.
            log_lr (bool, optional): True to plot the learning rate in a logarithmic
                scale; otherwise, plotted in a linear scale. Default: True.

        """

        if skip_start < 0:
            raise ValueError("skip_start cannot be negative")
        if skip_end < 0:
            raise ValueError("skip_end cannot be negative")

        # Get the data to plot from the history dictionary. Also, handle skip_end=0
        # properly so the behaviour is the expected
        lrs = self.history["lr"]
        losses = self.history["loss"]
        if skip_end == 0:
            lrs = lrs[skip_start:]
            losses = losses[skip_start:]
        else:
            lrs = lrs[skip_start:-skip_end]
            losses = losses[skip_start:-skip_end]

        # Plot loss as a function of the learning rate
        plt.plot(lrs, losses)
        if log_lr:
            plt.xscale("log")
        plt.xlabel("Learning rate")
        plt.ylabel("Loss")
        plt.show()

class LinearLR(_LRScheduler):
    """Linearly increases the learning rate between two boundaries over a number of
    iterations.

    Arguments:
        optimizer (torch.optim.Optimizer): wrapped optimizer.
        end_lr (float, optional): the initial learning rate which is the lower
            boundary of the test. Default: 10.
        num_iter (int, optional): the number of iterations over which the test
            occurs. Default: 100.
        last_epoch (int): the index of last epoch. Default: -1.

    """

    def __init__(self, optimizer, end_lr, num_iter, last_epoch=-1):
        self.end_lr = end_lr
        self.num_iter = num_iter
        super(LinearLR, self).__init__(optimizer, last_epoch)

    def get_lr(self):
        curr_iter = self.last_epoch + 1
        r = curr_iter / self.num_iter
        return [base_lr + r * (self.end_lr - base_lr) for base_lr in self.base_lrs]

class ExponentialLR(_LRScheduler):
    """Exponentially increases the learning rate between two boundaries over a number of
    iterations.

    Arguments:
        optimizer (torch.optim.Optimizer): wrapped optimizer.
        end_lr (float, optional): the initial learning rate which is the lower
            boundary of the test. Default: 10.
        num_iter (int, optional): the number of iterations over which the test
            occurs. Default: 100.
        last_epoch (int): the index of last epoch. Default: -1.

    """

    def __init__(self, optimizer, end_lr, num_iter, last_epoch=-1):
        self.end_lr = end_lr
        self.num_iter = num_iter
        super(ExponentialLR, self).__init__(optimizer, last_epoch)

    def get_lr(self):
        curr_iter = self.last_epoch + 1
        r = curr_iter / self.num_iter
        return [base_lr * (self.end_lr / base_lr) ** r for base_lr in self.base_lrs]

trans = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])

root = './data'
if not os.path.exists(root):
    os.mkdir(root)
train_set = dset.MNIST(root=root, train=True, transform=trans, download=True)
test_set = dset.MNIST(root=root, train=False, transform=trans, download=True)

batch_size = 64

train_loader = torch.utils.data.DataLoader(
                 dataset=train_set,
                 batch_size=batch_size,
                 shuffle=True)

test_loader = torch.utils.data.DataLoader(
                dataset=test_set,
                batch_size=batch_size,
shuffle=True)

class NeuralNet(nn.Module):
    def __init__(self):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(28*28, 500)
        self.fc2 = nn.Linear(500, 256)
        self.fc3 = nn.Linear(256, 10)
    def forward(self, x):
        x = x.view(-1, 28*28)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

num_epochs = 2
random_sample_size = 200

# Hyper-parameters
input_size = 100
hidden_size = 100
num_classes = 10
learning_rate = .0001
# Device configuration
device = 'cpu'

model = NeuralNet().to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
# lr_finder = LRFinder(model, optimizer, criterion, device="cpu")
# lr_finder.range_test(train_loader, end_lr=100, num_iter=100)
# lr_finder.plot()
# optimizer = torch.optim.Adam(model.parameters(), lr=lr_finder.history['lr'][0])
# print(lr_finder.history['lr'])

predicted_test = []
labels_l = []
actual_values = []
predicted_values = []

N = len(train_loader)
# Train the model
total_step = len(train_loader)

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Move tensors to the configured device
#         images = images.reshape(-1, 50176).to(device)
        images = images.to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        predicted = outputs.data.max(1)[1]
        predicted_test.append(predicted.cpu().numpy())
        labels_l.append(labels.cpu().numpy())

        loss = criterion(outputs, labels)
        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    predicted_values.append(np.concatenate(predicted_test).ravel())
    actual_values.append(np.concatenate(labels_l).ravel())

    print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))
    print('training accuracy : ', 100 * len((np.where(np.array(predicted_values[0])==(np.array(actual_values[0])))[0])) / len(actual_values[0]))

Results :

Epoch [1/2], Step [938/938], Loss: 0.5374
training accuracy :  84.09833333333333
Epoch [2/2], Step [938/938], Loss: 0.2055
training accuracy :  84.09833333333333

With the learning rate finder code being uncommented :

Below commented out code is now not un-commented :

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
lr_finder = LRFinder(model, optimizer, criterion, device="cpu")
lr_finder.range_test(train_loader, end_lr=100, num_iter=100)
lr_finder.plot()
optimizer = torch.optim.Adam(model.parameters(), lr=lr_finder.history['lr'][0])
print(lr_finder.history['lr'])

the model achieves results after two epochs :

Epoch [1/2], Step [938/938], Loss: 3.7311
training accuracy :  9.93
Epoch [2/2], Step [938/938], Loss: 3.5106
training accuracy :  9.93

Can see the training accuracy is much lower 84.09833333333333 versus 9.93 . Should the learning rate finder find a learning rate that allows to achieve greater training set accuracy ?


title: Expected input to torch Embedding layer with pre_trained vectors from gensim

body:
I would like to use pre-trained embeddings in my neural network architecture. The pre-trained embeddings are trained by gensim. I found this informative answer which indicates that we can load pre_trained models like so:

import gensim
from torch import nn

model = gensim.models.KeyedVectors.load_word2vec_format('path/to/file')
weights = torch.FloatTensor(model.vectors)
emb = nn.Embedding.from_pretrained(torch.FloatTensor(weights.vectors))

This seems to work correctly, also on 1.0.1. My question is, that I don't quite understand what I have to feed into such a layer to utilise it. Can I just feed the tokens (segmented sentence)? Do I need a mapping, for instance token-to-index?

I found that you can access a token's vector simply by something like

print(weights['the'])
# [-1.1206588e+00  1.1578362e+00  2.8765252e-01 -1.1759659e+00 ... ]

What does that mean for an RNN architecture? Can we simply load in the tokens of the batch sequences? For instance:

for seq_batch, y in batch_loader():
    # seq_batch is a batch of sequences (tokenized sentences)
    # e.g. [['i', 'like', 'cookies'],['it', 'is', 'raining'],['who', 'are', 'you']]
    output, hidden = model(seq_batch, hidden)

This does not seem to work so I am assuming you need to convert the tokens to its index in the final word2vec model. Is that true? I found that you can get the indices of words by using the word2vec model's vocab:

weights.vocab['world'].index
# 147

So as an input to an Embedding layer, should I provide a tensor of int for a sequence of sentences that consist of a sequence of words? Example use with dummy dataloader (cf. example above) and dummy RNN welcome.


title: How to get count number of equal elements in two torch tensors that also equal a specific value

body:
I'm working in pytorch and trying to count the number of equal elements in 2 torch tensors, that also equal a specific value.
That is, if tensor a=[0,1,2,0,1,2] and tensor b = [0,2,1,0,2,1]
I want it to return:

3 when I check how many element are equal in the a,b and also equals 0. sum(a == b and a == 0 and b == 0) = 2,
0 when I check how many element are equal in the a,b and also equals 1 or 2. sum(a == b and a == 1,2 and b == 1,2) = 0.

Thanks!


title: 'iterator' object has no attribute 'next' in python 3.7

body:
i'm trying to iterate through my data set and get the first element

transform = transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5),(0.5)),])
trainloader = datasets.MNIST('~/.pytorch/MNIST_data' , download=True,train=True , transform=transform)
ds = iter(trainloader)
img, labels = ds.next()

but it return this error

AttributeError: 'iterator' object has no attribute 'next'

i also tried this

img , labels = next(ds)

returned this error

StopIteration:

does i have missed something !?


title: Can't use GPU with Pytorch

body:
I keep getting this error when trying to use Pytorch.

  RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

I installed Pytorch using conda install pytorch torchvision cudatoolkit=10.1 -c pytorch.

With tensorflow my GPU runs just fine.


title: OSError: cannot identify image file

body:
I am trying impelement code in pytorch but I get bellow error. my python version is 3.6 and my os is linux ubuntu 16.04 lts. I installed my linux alongside of mac os. We will use torchvision and torch.utils.data packages for loading the data.There are 75 validation images for each class.

OSError                                   Traceback (most recent call    last)
<ipython-input-4-e0e3a841f698> in <module>()
     62
 63 # Get a batch of training data
---> 64 inputs, classes = next(iter(dset_loaders['train']))
 65
 66 # Make a grid from batch

/home/zeinab/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
172                 self.reorder_dict[idx] = batch
173                 continue
--> 174             return self._process_next_batch(batch)
175
176     next = __next__  # Python 2 compatibility

/home/zeinab/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _process_next_batch(self, batch)
196         self._put_indices()
197         if isinstance(batch, ExceptionWrapper):
--> 198             raise batch.exc_type(batch.exc_msg)
199         return batch
200

OSError: Traceback (most recent call last):
  File "/home/zeinab/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 32, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/zeinab/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 32, in <listcomp>
samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/zeinab/anaconda3/lib/python3.6/site-packages/torchvision-0.1.7-py3.6.egg/torchvision/datasets/folder.py", line 57, in __getitem__
img = self.loader(os.path.join(self.root, path))
  File "/home/zeinab/anaconda3/lib/python3.6/site-packages/torchvision-0.1.7-py3.6.egg/torchvision/datasets/folder.py", line 38, in default_loader
return Image.open(path).convert('RGB')
  File "/home/zeinab/anaconda3/lib/python3.6/site-packages/PIL/Image.py", line 2349, in open
% (filename if filename else fp))
OSError: cannot identify image file 'hymenoptera_data/train/ants/._154124431_65460430f2.jpg'

my code is bellow:

%matplotlib inline
# License: BSD
# Author: Sasank Chilamkurthy

from __future__ import print_function, division

import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable
import numpy as np
import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import time
import copy
import os
from PIL import Image
import os, sys

plt.ion()   # interactive mode
# Data augmentation and normalization for training
# Just normalization for validation
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomSizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
]),
    'val': transforms.Compose([
    transforms.Scale(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
]),
}

 data_dir = 'hymenoptera_data'
dsets = {x: datasets.ImageFolder(os.path.join(data_dir, x),         data_transforms[x])
     for x in ['train', 'val']}
dset_loaders = {x: torch.utils.data.DataLoader(dsets[x], batch_size=4,
                                           shuffle=True, num_workers=4)
            for x in ['train', 'val']}
dset_sizes = {x: len(dsets[x]) for x in ['train', 'val']}
dset_classes = dsets['train'].classes

use_gpu = torch.cuda.is_available()
def imshow(inp, title=None):
"""Imshow for Tensor."""
inp = inp.numpy().transpose((1, 2, 0))

mean = np.array([0.485, 0.456, 0.406])
std = np.array([0.229, 0.224, 0.225])
inp = std * inp + mean
plt.imshow(inp)
if title is not None:
    plt.title(title)
plt.pause(0.001)  # pause a bit so that plots are updated

# Get a batch of training data
inputs, classes = next(iter(dset_loaders['train']))

# Make a grid from batch
out = torchvision.utils.make_grid(inputs)

imshow(out, title=[dset_classes[x] for x in classes])


title: groupby aggregate mean in pytorch

body:
I have a 2D tensor:

samples = torch.Tensor([
                 [0.1, 0.1],    #-> group / class 1
                 [0.2, 0.2],    #-> group / class 2
                 [0.4, 0.4],    #-> group / class 2
                 [0.0, 0.0]     #-> group / class 0
          ])

and a label for each sample corresponding to a class:

labels = torch.LongTensor([1, 2, 2, 0])

so len(samples) == len(labels). Now I want to calculate the mean for each class / label. Because there are 3 classes (0, 1 and 2) the final vector should have dimension [n_classes, samples.shape[1]] So the expected solution should be:

result == torch.Tensor([
                 [0.1, 0.1],
                 [0.3, 0.3], # -> mean of [0.2, 0.2] and [0.4, 0.4]
                 [0.0, 0.0]
          ])

Question: How can this be done in pure pytorch (i.e. no numpy so that I can autograd) and ideally without for loops?


title: I am getting error when importing torch and torch vision

body:
I installed torch and torchvision using pip3 on MAC. When I imported the same, getting the below error.

Environment:

OS : macOS High Sierra
Python : 3.7
pip : 3
Pytorch 1.1

code:

import torch
import torchvision

Error:

  ImportError                               Traceback (most recent call
  last)  in
        1 import numpy as np
        2 from glob import glob
  ----> 3 import torch
        4 import torchvision
        5 from torchvision import datasets

  /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/init.py
  in
       77 del _dl_flags
       78
  ---> 79 from torch._C import *
       80
       81 all += [name for name in dir(_C)

  ImportError:
  dlopen(/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/_C.cpython-37m-darwin.so,
  9): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from:
  /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/lib/libshm.dylib
  Reason: image not found

Any help on solving this error would be appreciated.


title: Unable to install pytorch in conda due to setuptools is a dependency of conda and cannot be removed

body:
conda install pytorch torchvision cudatoolkit=9.0 -c pytorch

Here is the error I encountered

 RemoveError: 'setuptools' is a dependency of conda and cannot be removed from
conda's operating environment.

However, setuptools is not found in
conda list. What should I do?

  Name                    Version                   Build  Channel

  _ipyw_jlab_nb_ext_conf    0.1.0            py36he6757f0_0
  _r-mutex                  1.0.0                     mro_2    r
  _tflow_select             2.1.0                       gpu    anaconda absl-py                   0.4.0                    pypi_0    pypi
  alabaster                 0.7.10           py36hcd07829_0 anaconda
  custom           py36h363777c_0 anaconda-client           1.6.5
  py36hd36550c_0 anaconda-navigator        1.6.9
  py36hc720852_0 anaconda-project          0.8.0
  py36h8b3bf89_0 asn1crypto                0.22.0
  py36h8e79faa_1 astor                     0.7.1
  pypi_0    pypi astroid                   1.5.3
  py36h9d85297_0 astropy                   2.0.2
  py36h06391c4_4 babel                     2.5.0
  pypi_0    pypi backports                 1.0
  py36h81696a8_1 backports.shutil_get_terminal_size 1.0.0
  py36h79ab834_2 beautifulsoup4            4.6.0
  py36hd4cc5e8_1 bitarray                  0.8.1
  py36h6af124b_0 bkcharts                  0.2
  py36h7e685f7_0 blas                      1.0
  mkl blaze                     0.11.3           py36h8a29ca5_0 bleach
  2.0.0            py36h0a7e3d6_0 bokeh                     0.12.10          py36h0be3b39_0 boto                      2.48.0
  py36h1a776d2_1 bottleneck                1.2.1
  py36hd119dfa_0 bzip2                     1.0.6
  vc14hdec8e7a_1  [vc14] ca-certificates           2018.11.29
  ha4d7672_0    conda-forge cachecontrol              0.12.3
  py36hfe50d7b_0 certifi                   2018.11.29
  py36_1000    conda-forge cffi                      1.10.0
  py36hae3d1b5_1 chardet                   3.0.4
  py36h420ce6e_1 click                     6.7
  py36hec8c647_0 cloudpickle               0.4.0
  py36h639d8dc_0 clyent                    1.2.2
  py36hb10d595_1 colorama                  0.3.9
  py36h029ae33_0 comtypes                  1.1.2
  py36heb9b3d1_0 conda                     4.6.8
  py36_0    conda-forge conda-build               3.0.27
  py36h309a530_0 conda-env                 2.6.0
  h36134e3_1 conda-verify              2.0.0            py36h065de53_0
  console_shortcut          0.1.1                h6bb2dd7_3 contextlib2
  0.5.5            py36he5d52c0_0 crowdposetools            2.0                      pypi_0    pypi cryptography              2.5
  py36h74b6da3_1    conda-forge cudatoolkit               9.0
  1    anaconda cudnn                     7.1.4
  cuda9.0_0    anaconda curl                      7.55.1
  vc14hdaba4a4_3  [vc14] cycler                    0.10.0
  py36h009560c_0 cython                    0.26.1
  py36h18049ac_0 cytoolz                   0.8.2
  py36h547e66e_0 dask                      0.17.5
  py36_0 dask-core                 0.17.5                   py36_0
  datashape                 0.5.4            py36h5770b85_0 decorator
  4.1.2            py36he63a57b_0 distlib                   0.2.5            py36h51371be_0 distributed               1.21.8
  py36_0 docker-pycreds            0.3.0                    pypi_0
  pypi docutils                  0.14             py36h6012d8f_0
  entrypoints               0.2.3            py36hfd66bb0_2 et_xmlfile
  1.0.1            py36h3d2d736_0 fastcache                 1.0.2            py36hffdae1b_0 filelock                  2.0.12
  py36hd7ddd41_0 flask                     0.12.2
  py36h98b5e8f_0 flask-cors                3.0.3
  py36h8a3855d_0 font-ttf-dejavu-sans-mono 2.37
  h6964260_0 font-ttf-inconsolata      2.001                hcb22688_0
  font-ttf-source-code-pro  2.030                h7457263_0
  font-ttf-ubuntu           0.83                 h8b1ccd4_0
  fonts-anaconda            1                    h8fa9717_0 freetype
  2.9.1             h5db478b_1005    conda-forge gast                      0.2.0                    pypi_0    pypi geos                      3.6.2                he025d50_3    conda-forge get_terminal_size         1.0.0                h38e98db_0 gevent                    1.2.2            py36h342a76c_0 glob2                     0.5
  py36h11cc1bd_1 google-images-download    2.4.2
  pypi_0    pypi greenlet                  0.4.12
  py36ha00ad21_0 grpcio                    1.14.1
  pypi_0    pypi h5py                      2.8.0
  py36hfa6e2cd_0    conda-forge hdf5                      1.10.1
  vc14hb361328_0  [vc14] heapdict                  1.0.0
  py36h21fa5f4_0 html5lib                  0.999999999
  py36ha09b1f3_0 icc_rt                    2017.0.4
  h97af966_0 icu                       58.2             vc14hc45fdbb_0
  [vc14] idna                      2.6              py36h148d497_1
  imageio                   2.2.0            py36had6c2d2_0 imagesize
  0.7.1            py36he29f638_0 imutils                   0.5.1                    pypi_0    pypi intel-openmp              2018.0.0
  hcd89f80_7 ipykernel                 4.6.1            py36hbb77b34_0
  ipython                   6.1.0            py36h236ecc8_1
  ipython_genutils          0.2.0            py36h3c5d0ee_0 ipywidgets
  7.0.0            py36h2e74ada_0 isort                     4.2.15           py36h6198cc5_0 itsdangerous              0.24
  py36hb6c5a24_1 jdcal                     1.3
  py36h64a5255_0 jedi                      0.10.2
  py36hed927a0_0 jinja2                    2.9.6
  py36h10aa3a0_1 jpeg                      9b
  vc14h4d7706e_1  [vc14] jsonschema                2.6.0
  py36h7636477_0 jupyter                   1.0.0
  py36h422fd7e_2 jupyter_client            5.1.0
  py36h9902a9a_0 jupyter_console           5.2.0
  py36h6d89b47_1 jupyter_core              4.3.0
  py36h511e818_0 jupyterlab                0.27.0
  py36h34cc53b_2 jupyterlab_launcher       0.4.0
  py36h22c3ccf_0 keras                     2.2.4
  0    anaconda keras-applications        1.0.6
  py36_0    anaconda keras-base                2.2.4
  py36_0    anaconda keras-preprocessing       1.0.5
  py36_0    anaconda kiwisolver                1.0.1
  py36he980bc4_1002    conda-forge kmodes                    0.9
  pypi_0    pypi krb5                      1.16.3
  h038dc86_1000    conda-forge lazy-object-proxy         1.3.1
  py36hd1c21d2_0 libcurl                   7.64.0
  h4496350_2    conda-forge libgpuarray               0.7.6
  hfa6e2cd_1003    conda-forge libiconv                  1.15
  vc14h29686d3_5  [vc14] libpng                    1.6.36
  h7602738_1000    conda-forge libprotobuf               3.6.1
  h1a1b453_1000    conda-forge libssh2                   1.8.0
  vc14hcf584a9_2  [vc14] libtiff                   4.0.9
  hf1753bf_1    conda-forge libxml2                   2.9.4
  vc14h8fd0f11_5  [vc14] libxslt                   1.1.29
  vc14hf85b8d4_5  [vc14] llvmlite                  0.20.0
  py36_0 locket                    0.2.0            py36hfed976d_1
  lockfile                  0.12.2           py36h0468280_0 lxml
  4.1.0            py36h0dcd83c_0 lzo                       2.10             vc14h0a64fa6_1  [vc14] m2-bash                   4.3.042
  5 m2-gcc-libs               5.3.0                         4 m2-heimdal
  1.5.3                        10 m2-heimdal-libs           1.5.3                        10 m2-icu                    56.1                          2
  m2-libcrypt               1.3                           2 m2-libdb
  5.3.28                        3 m2-libedit                3.1                    20150326 m2-libopenssl             1.0.2.g                       2
  m2-libreadline            6.3.008                       8 m2-libsqlite
  3.10.0.0                      2 m2-msys2-runtime          2.5.0.17080.65c939c               3 m2-ncurses                6.0.20160220                  2 m2-openssh                7.1p2                         2 m2-openssl                1.0.2.g                       2 m2-zlib
  1.2.8                         4 m2w64-bzip2               1.0.6                         6 m2w64-diffutils           3.3.2                         2
  m2w64-gcc-libgfortran     5.3.0                         6
  m2w64-gcc-libs            5.3.0                         7
  m2w64-gcc-libs-core       5.3.0                         7 m2w64-gmp
  6.1.0                         2 m2w64-grep                2.24                          2 m2w64-libwinpthread-git   5.0.0.4634.697f757               2
  m2w64-pcre                8.38                          2
  m2w64-wineditline         2.101                         5 m2w64-zlib
  1.2.8                        10 mako                      1.0.7                      py_1    conda-forge markdown                  2.6.11
  py_0    conda-forge markupsafe                1.0
  py36h0e26971_1 matplotlib                3.0.2
  py36_1002    conda-forge matplotlib-base           3.0.2
  py36h3e3dc42_1002    conda-forge mccabe                    0.6.1
  py36hb41005a_1 menuinst                  1.4.14
  py36_1000    conda-forge mistune                   0.7.4
  py36h4874169_0 mkl                       2018.0.0
  h36b65af_4 mkl-service               1.1.2            py36h57e144c_4
  mpmath                    0.19             py36he326802_2 mro-base
  3.5.1                         3    r mro-base_impl             3.5.1                         0    r msgpack                   0.5.6                    pypi_0
  pypi msgpack-python            0.4.8            py36h58b1e9d_0
  msys2-conda-epoch         20160418                      1
  multipledispatch          0.4.9            py36he44c36e_0
  navigator-updater         0.1.0            py36h8a7b86b_0 nbconvert
  5.3.1            py36h8dc0fde_0 nbformat                  4.4.0            py36h3a5bc1b_0 networkx                  2.0
  py36hff991e3_0 nltk                      3.2.4
  py36hd0e0a39_0 nose                      1.3.7
  py36h1c3779e_2 notebook                  5.0.0
  py36hd9fbf6f_2 npm                       0.1.1
  pypi_0    pypi numba                     0.35.0
  np112py36_0 numexpr                   2.6.2            py36h7ca04dc_1
  numpy                     1.14.5                   pypi_0    pypi
  numpydoc                  0.7.0            py36ha25429e_0 odo
  0.5.1            py36h7560279_0 olefile                   0.44             py36h0a7bdd2_0 open3d                    0+unknown
  pypi_0    pypi opencv-contrib-python     3.4.1.15
  pypi_0    pypi opencv-python             4.0.0.21
  pypi_0    pypi openpyxl                  2.4.8
  py36hf3b77f6_1 openssl                   1.0.2r
  hfa6e2cd_0    conda-forge optional-django           0.1.0
  pypi_0    pypi packaging                 16.8
  py36ha0986f6_1 pandas                    0.23.0
  py36h830ac7b_0 pandoc                    1.19.2.1
  hb2460c7_1 pandocfilters             1.4.2            py36h3ef6317_1
  partd                     0.3.8            py36hc8e763b_0 path.py
  10.3.1           py36h3dd8b46_0 pathlib2                  2.3.0            py36h7bfb78b_0 patsy                     0.4.1
  py36h42cefec_0 pep8                      1.7.0
  py36h0f3d67a_0 pickleshare               0.7.4
  py36h9de030f_0 pillow                    5.2.0
  py36h08bbbbd_0 pip                       19.0.3
  pypi_0    pypi pkginfo                   1.4.1
  py36hb0f9cfa_1 plotly                    2.7.0
  pypi_0    pypi ply                       3.10
  py36h1211beb_0 progress                  1.3
  py36hbeca8d3_0 prompt_toolkit            1.0.15
  py36h60b8f86_0 protobuf                  3.6.1
  pypi_0    pypi psutil                    5.4.0
  py36h4e662fb_0 py                        1.4.34
  py36ha4aca3a_1 pyclustering              0.8.0
  pypi_0    pypi pycocotools               2.0
  pypi_0    pypi pycodestyle               2.3.1
  py36h7cc55cd_0 pycosat                   0.6.3
  py36h413d8a4_0 pycparser                 2.18
  py36hd053e01_1 pycrypto                  2.6.1
  py36he68e6e2_1 pycurl                    7.43.0.2
  py36h74b6da3_0 pydotplus                 2.0.2
  pypi_0    pypi pyflakes                  1.6.0
  py36h0b975d6_0 pygments                  2.2.0
  py36hb010967_0 pygpu                     0.7.6
  py36h452e1ab_1000    conda-forge pylint                    1.7.4
  py36ha4e6ded_0 pyodbc                    4.0.17
  py36h0006bc2_0 pyopenssl                 17.2.0
  py36h15ca2fc_0 pyparsing                 2.2.0
  py36h785a196_1 pyqt                      5.6.0
  py36_2 pyreadline                2.1                   py36_1000
  conda-forge pysocks                   1.6.7            py36h698d350_1
  pytables                  3.4.2            py36h71138e3_2 pytesseract
  0.2.0                    pypi_0    pypi pytest                    3.2.1            py36h753b05e_1 python                    3.6.3                h9e2ca53_1 python-dateutil           2.6.1            py36h509ddcb_1
  python-graphviz           0.8.3                    pypi_0    pypi pytz
  2017.2           py36h05d413f_1 pywavelets                0.5.2            py36hc649158_0 pywin32                   221
  py36h9c10281_0 pyyaml                    3.12
  py36h1d1928f_1 pyzmq                     16.0.2
  py36h38c27d9_2 qt                        5.6.2
  vc14_1    conda-forge qtawesome                 0.4.4
  py36h5aa48f6_0 qtconsole                 4.3.1
  py36h99a29a9_0 qtpy                      1.3.1
  py36hb8717c5_0 r-assertthat              0.2.0
  mro351_0    r r-backports               1.1.2
  mro351_0    r r-base64enc               0.1_3
  mro351_0    r r-bh                      1.66.0_1
  mro351hf348343_0    r r-bindr                   0.1.1
  mro351_0    r r-bindrcpp                0.2.2
  mro351_0    r r-bit                     1.1_14
  mro351_0    r r-bit64                   0.9_7
  mro351_0    r r-bitops                  1.0_6
  mro351_0    r r-blob                    1.1.1
  mro351_0    r r-broom                   0.5.0
  mro351_0    r r-catools                 1.17.1.1
  mro351hf348343_0    r r-cellranger              1.1.0
  mro351_0    r r-cli                     1.0.0
  mro351_0    r r-config                  0.3
  mro351_0    r r-crayon                  1.3.4
  mro351_0    r r-curl                    3.2
  mro351_0    r r-dbi                     1.0.0
  mro351hf348343_0    r r-dbplyr                  1.2.2
  mro351_0    r r-digest                  0.6.15
  mro351_0    r r-dplyr                   0.7.6
  mro351_0    r r-evaluate                0.11
  mro351_0    r r-fansi                   0.2.3
  mro351_0    r r-forcats                 0.3.0
  mro351_0    r r-glue                    1.3.0
  mro351_0    r r-haven                   1.1.2
  mro351_0    r r-highr                   0.7
  mro351_0    r r-hms                     0.4.2
  mro351_0    r r-htmltools               0.3.6
  mro351_0    r r-htmlwidgets             1.2
  mro351_0    r r-httpuv                  1.4.5
  mro351_0    r r-httr                    1.3.1
  mro351_0    r r-jsonlite                1.5
  mro351_0    r r-knitr                   1.20
  mro351_0    r r-later                   0.7.3
  mro351_0    r r-lattice                 0.20_35
  mro351_0    r r-lazyeval                0.2.1
  mro351_0    r r-magrittr                1.5
  mro351_0    r r-markdown                0.8
  mro351_0    r r-mime                    0.5
  mro351_0    r r-miniui                  0.1.1.1
  mro351hf348343_0    r r-mongolite               1.6
  mro351_0    r r-nlme                    3.1_137
  mro351_0    r r-odbc                    1.1.6
  mro351_0    r r-openssl                 1.0.2
  mro351_0    r r-packrat                 0.4.9_3
  mro351_0    r r-pillar                  1.3.0
  mro351_0    r r-pkgconfig               2.0.1
  mro351_0    r r-pki                     0.1_5.1
  mro351hf348343_0    r r-plogr                   0.2.0
  mro351_0    r r-plyr                    1.8.4
  mro351_0    r r-prettyunits             1.0.2
  mro351_0    r r-profvis                 0.3.5
  mro351_0    r r-promises                1.0.1
  mro351_0    r r-purrr                   0.2.5
  mro351_0    r r-r6                      2.2.2
  mro351hf348343_0    r r-rappdirs                0.3.1
  mro351_0    r r-rcpp                    0.12.18
  mro351hf348343_0    r r-rcurl                   1.95_4.11
  mro351hf348343_0    r r-readr                   1.1.1
  mro351_0    r r-readxl                  1.1.0
  mro351_0    r r-rematch                 1.0.1
  mro351_0    r r-reshape2                1.4.3
  mro351_0    r r-revoutils               11.0.0
  mro351_0    r r-revoutilsmath           11.0.0
  mro351_0    r r-rjava                   0.9_9
  mro343h889e2dd_0    r r-rjdbc                   0.2_7.1
  mro351hf348343_0    r r-rjsonio                 1.3_0
  mro351hf348343_0    r r-rlang                   0.2.1
  mro351_0    r r-rmarkdown               1.10
  mro351_0    r r-rprojroot               1.3_2
  mro351_0    r r-rsconnect               0.8.8
  mro351_0    r r-rstudioapi              0.7
  mro351_0    r r-shiny                   1.1.0
  mro351_0    r r-sourcetools             0.1.7
  mro351_0    r r-sparklyr                0.8.4
  mro351_0    r r-stringi                 1.1.7
  mro351_0    r r-stringr                 1.3.1
  mro351_0    r r-tibble                  1.4.2
  mro351_0    r r-tidyr                   0.8.1
  mro351_0    r r-tidyselect              0.2.4
  mro351_0    r r-tinytex                 0.6
  mro351_0    r r-utf8                    1.1.4
  mro351_0    r r-withr                   2.1.2
  mro351_0    r r-xfun                    0.3
  mro351_0    r r-xml2                    1.2.0
  mro351_0    r r-xtable                  1.8_2
  mro351_0    r r-yaml                    2.2.0
  mro351_0    r requests                  2.18.4
  py36h4371aae_1 rope                      0.10.5
  py36hcaf5641_0 rstudio                   1.1.456
  vc14h1ad3211_1  [vc14]  r ruamel_yaml               0.11.14
  py36h9b16331_2 scikit-image              0.13.0
  py36h6dffa3f_1 scikit-learn              0.19.1
  py36h53aea1b_0 scipy                     1.0.0
  py36h1260518_0 seaborn                   0.8.0
  py36h62cb67c_0 selenium                  3.14.1
  pypi_0    pypi shapely                   1.6.4
  py36hc90234e_1000    conda-forge simplegeneric             0.8.1
  py36heab741f_0 singledispatch            3.4.0.3
  py36h17d0c80_0 sip                       4.19.8
  py36h6538335_1000    conda-forge six                       1.11.0
  py36h4db2310_1 slidingwindow             0.0.13
  pypi_0    pypi snowballstemmer           1.2.1
  py36h763602f_0 sortedcollections         0.5.3
  py36hbefa0ab_0 sortedcontainers          1.5.7
  py36ha90ac20_0 sphinx                    1.6.3
  py36h9bb690b_0 sphinxcontrib             1.0
  py36hbbac3d2_1 sphinxcontrib-websupport  1.0.1
  py36hb5e5916_1 spyder                    3.2.4
  py36h8845eaa_0 sqlalchemy                1.1.13
  py36h5948d12_0 sqlite                    3.26.0
  hfa6e2cd_1001    conda-forge statsmodels               0.8.0
  py36h6189b4c_0 sympy                     1.1.1
  py36h96708e0_0 tblib                     1.3.2
  py36h30f5020_0 tensorboard               1.11.0
  py36he025d50_0    anaconda tensorflow                1.11.0
  gpu_py36h5dc63e2_0    anaconda tensorflow-base           1.11.0
  gpu_py36h6e53903_0    anaconda tensorflow-gpu            1.11.0
  h0d30ee6_0    anaconda termcolor                 1.1.0
  pypi_0    pypi testpath                  0.3.1
  py36h2698cfe_0 theano                    1.0.3
  py36_0    conda-forge tk                        8.6.9
  hfa6e2cd_1000    conda-forge toolz                     0.8.2
  py36he152a52_0 tornado                   4.5.2
  py36h57f6048_0 traitlets                 4.3.2
  py36h096827d_0 typing                    3.6.2
  py36hb035bda_0 unicodecsv                0.14.1
  py36h6450c06_0 urllib3                   1.22
  py36h276f60a_0 vc                        14
  0    conda-forge vs2015_runtime            15.5.2
  3 vs2015_win-64             14.0.25123           h17c34da_3 wcwidth
  0.1.7            py36h3d5aa90_0 webencodings              0.5.1            py36h67c50ae_1 websocket-client          0.53.0
  pypi_0    pypi werkzeug                  0.12.2
  py36h866a736_0 wheel                     0.29.0
  py36h6ce6cde_1 widgetsnbextension        3.0.2
  py36h364476f_1 win_inet_pton             1.0.1
  py36he67d7fd_1 win_unicode_console       0.5
  py36hcdbd4b5_0 wincertstore              0.2
  py36h7fe50ca_0 wrapt                     1.10.11
  py36he5f5981_0 xlrd                      1.1.0
  py36h1cb58dc_1 xlsxwriter                1.0.2
  py36hf723b7d_0 xlwings                   0.11.4
  py36hd3cf94d_0 xlwt                      1.3.0
  py36h1a4751e_0 yaml                      0.1.7
  vc14hb31d195_1  [vc14] zict                      0.1.3
  py36h2d8e73e_0 zlib                      1.2.11
  vc14h1cdd9ab_1  [vc14]


title: net.zero_grad() vs optim.zero_grad() pytorch

body: Here they mention the need to include optim.zero_grad() when training to zero the parameter gradients. My question is: Could I do as well net.zero_grad() and would that have the same effect? Or is it necessary to do  optim.zero_grad(). Moreover, what happens if I do both? If I do none, then the gradients get accumulated, but what does that exactly mean? do they get added? In other words, what's the difference between doing optim.zero_grad() and net.zero_grad(). I am asking because here, line 115 they use net.zero_grad() and it is the first time I see that, that is an implementation of a reinforcement learning algorithm, where one has to be especially careful with the gradients because there are multiple networks and gradients, so I suppose there is a reason for them to do net.zero_grad() as opposed to optim.zero_grad().


title: Issues installing pytorch for OS X with conda

body:
I used to have pytorch working for python 3 on OS X but now I can't get it to install automatically for some reason (I don't want to do from source).

I did:

conda install pytorch torchvision -c pytorch

as the website suggested... then I got a mkl error so I installed it but it still complains about it:

(FTIR_py3) brandomiranda~/home_simulation_research/FTIR/FTIR_proj $ conda install pytorch torchvision -c pytorch
Fetching package metadata ...........
Solving package specifications:

PackageNotFoundError: Package not found: '' Dependencies missing in current osx-64 channels:
  - pytorch -> mkl >=2018
  - torchvision -> pytorch >=0.3 -> mkl >=2018

You can search for packages on anaconda.org with

    anaconda search -t conda mkl

You may need to install the anaconda-client command line client with

    conda install anaconda-client

but I do have mkl:

(FTIR_py3) brandomiranda~/home_simulation_research/FTIR/FTIR_proj $ conda install mkl
Fetching package metadata .........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /Users/brandomiranda/miniconda3/envs/FTIR_py3:
#
mkl                       2017.0.3                      0

anyone know whats going on? it used to work a few days ago...

longer thread on pytorch forum:

https://discuss.pytorch.org/t/issues-installing-pytorch-for-os-x-with-conda/11496


I'm doing an image processing task and I want to concat two sites of pictures. For concatting, I first converted the image to tensor, then converted the tensor to a PIL image to display it, but it was reported incorrectly.Could someone please help me?
Here is my code:

import skimage.io as io
import torch
from torchvision import  models, transforms
from PIL import Image
import matplotlib.pyplot as plt
from torchvision.transforms import ToPILImage
import numpy as np
from skimage import data_dir,io,color

coll1 = io.ImageCollection('F:\\code1/*.jpg')
coll2 = io.ImageCollection('F:\\code2/*.jpg')
a = torch.tensor(coll1)
print(a)
print(a.shape)
b = torch.tensor(coll2)
print(b)
print(b.shape)
c=torch.cat((a,b),1)
print(c.shape)
print(c)
img= transforms.ToPILImage()
img.show()

and here is the error code:

  Traceback (most recent call last):   File "F:/filelist.py", line 39,
  in
      img.show() AttributeError: 'ToPILImage' object has no attribute 'show'


I am new to PyTorch and neural networks in general. I was trying to implement the resnet-50 model from torchvision on the CIFAR-10 dataset.

import torchvision
import torch
import torch.nn as nn
from torch import optim
import os
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
import numpy as np
from collections import OrderedDict
import matplotlib.pyplot as plt

transformations=transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))])
trainset=torchvision.datasets.CIFAR10(root='./CIFAR10',download=True,transform=transformations,train=True)

testset=torchvision.datasets.CIFAR10(root='./CIFAR10',download=True,transform=transformations,train=False)

trainloader=DataLoader(dataset=trainset,batch_size=4)
testloader=DataLoader(dataset=testset,batch_size=4)

inputs,labels=next(iter(trainset))
inputs.size()
resnet=torchvision.models.resnet50(pretrained=True)

if torch.cuda.is_available():
  resnet=resnet.cuda()
  inputs,labels=inputs.cuda(),torch.Tensor(labels).cuda()

outputs=resnet(inputs)

OUTPUT

--------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-6-904acb410fe4> in <module>()
      6   inputs,labels=inputs.cuda(),torch.Tensor(labels).cuda()
      7
----> 8 outputs=resnet(inputs)

5 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    344                             _pair(0), self.dilation, self.groups)
    345         return F.conv2d(input, weight, self.bias, self.stride,
--> 346                         self.padding, self.dilation, self.groups)
    347
    348     def forward(self, input):

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 3, 7, 7], but got 3-dimensional input of size [3, 32, 32] instead

Is there a problen with the dataset for some reason and if not, how do i give a 4 dimensional input? Is the torchvision implementation of ResNet-50 not usable for CIFAR-10?


title: Pytorch 'Tensor' object is not callable

body:
I am trying to apply sobel filter to my network. I am getting this error : "'Tensor' object is not callable"
Here is my code:

class SobelFilter(nn.Module):
    def __init__(self):
        super(SobelFilter, self).__init__()

        kernel1=torch.Tensor([[1, 0, -1],[2,0,-2],[1,0,-1]])
        kernela=kernel1.expand((1,1,3,3))

        kernel2=torch.Tensor([[1, 2, 1],[0,0,0],[-1,-2,-1]])
        kernelb=kernel2.expand((1,1,3,3))

        inputs = torch.randn(1,1,64,128)

        self.conv1 = F.conv2d(inputs,kernela,stride=1,padding=1)
        self.conv2 = F.conv2d(inputs,kernelb,stride=1,padding=1)

    def forward(self, x ):

        print(x.shape)
        G_x = self.conv1(x)    %GIVES ERROR AT THIS LINE
        G_y = self.conv2(x)
        out = torch.sqrt(torch.pow(G_x,2)+ torch.pow(G_y,2))

        return out

class EXP(nn.Module):
    def __init__(self, maxdisp):
        super(EXP, self).__init__()
        self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        self.SobelFilter = SobelFilter()

    def forward(self, im):
        % just think "im" to be [1,32,64,128]

        for i in range(im.shape[1]):
            out1= im[:,i,:,:].unsqueeze(1)
            im[:,i,:,:] = self.SobelFilter(out1)

what can cause the problem?
Thank you!


title: Error during training on ResNet50: Expected 4-dimensional input for 4-dimensional weight [64, 3, 7, 7], but got input of size [8, 196608] instead

body:
python v 3.7.3, pytorch v 0.4.1, using jupyter

I am working on creating an image classifier using transfer learning with ResNet50 as the base model.  I've run across an error that I don't understand how to debug, even after searching for solutions online.  I'm not sure why my dimensions are so off.  Any help on what to do would be appreciated.
Also, any book recs on Deep Learning using Pytorch would be great :)

Code for final layer of ResNet:

fc_inputs = model.fc.in_features

model.fc = nn.Sequential(OrderedDict([
            ('fc1', nn.Linear(fc_inputs, 256)),
            ('relu', nn.ReLU()),
            ('dropout', nn.Dropout(0.2)),
            ('fc2', nn.Linear(256, 25)),
            ('output', nn.LogSoftmax(dim=1))
]))

model.classifier = classifier

print(model)
print(model.classifier[0])
print(model.classifier[3])

optimizer = torch.optim.Adam(model.classifier.parameters(), lr = 0.001)
criterion = nn.CrossEntropyLoss()
scheduler = lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)

code for training:

epochs = 1
steps = 0
running_loss = 0
print_every = 10

for epoch in range(epochs):

    for images, labels in dataloaders['train']:

        images = images.view(images.shape[0], -1)  #this flattens it?
        steps += 1

        images, labels = images.to(device), labels.to(device)

        optimizer.zero_grad()

        logps = model(images)

        loss = criterion(logps, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

        if step % print_every == 0:
            model.eval()
            test_loss = 0
            accuracy = 0

            for images, labels in dataloader:
                images, labels = images.to(device), labels.to(device)

                logps = model(images)
                loss = criterion(logps, labels)
                test_loss += loss.item()

                ps = torch.exp(logps)

                top_ps, top_class = ps.topk(1, dim=1)
                equality = top_class == labels.view(*top_class.shape)
                accuracy += torch.mean(equality.type(torch.FloatTensor)).item()

            train_losses.append(running_loss / len(dataloader['train']))
            test_losses.append(test_loss / len(dataloader['test']))

            print(f"Epoch {epoch + 1}/{epochs}.."
                  f"Train loss: {running_loss / print_every:.3f}.."
                  f"Test loss: {test_loss/len(dataloader['test']):.3f}.."
                  f"Test accuracy: {accuracy/len(dataloader['test']):.3f}")

            running_loss = 0
            model.train()

error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-47-8f88694d7909> in <module>
     16         optimizer.zero_grad()
     17
---> 18         logps = model(images)
     19
     20         loss = criterion(logps, labels)

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--> 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

~\Anaconda3\lib\site-packages\torchvision\models\resnet.py in forward(self, x)
    137
    138     def forward(self, x):
--> 139         x = self.conv1(x)
    140         x = self.bn1(x)
    141         x = self.relu(x)

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--> 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

~\Anaconda3\lib\site-packages\torch\nn\modules\conv.py in forward(self, input)
    299     def forward(self, input):
    300         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 301                         self.padding, self.dilation, self.groups)
    302
    303

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 3, 7, 7], but got input of size [8, 196608] instead

EDIT: I removed images = images.view.shape[0], -1, which was converting my tensor to [8, 196608].  But now I am receiving a new error:

RuntimeError: size mismatch, m1: [8 x 8192], m2: [2048 x 256] at c:\programdata\miniconda3\conda-bld\pytorch_1532509700152\work\aten\src\th\generic/THTensorMath.cpp:2070


title: Pytorch - is this the right way to load a big dataset using pytorch.data.Dataset and train it for Linear Regression

body:
I'm new to PyTorch and currently working with a historical weather dataset and have built PyTorch DataLoader iterators and split the data to a train and test set successfully.

I want to run a simple Linear Regression model on this data to predict the column ["Temperature (C)"]. I'm not doing any feature engineering and instead am just using all float point dtypes as features.

On running the model (just a fully connected layer with bias True - Linear Regression) , I see a decrease in loss in the first 2 batches or so and then the loss just fluctuates around (100 +- 25). When I normalize the features, the loss decreases in the first batch but the fluctuation is around 200 +- 25.

From what I've learnt, I know that once features are normalized SGD has a higher chance to converge faster. But there is a difference in the base loss ie. without feature normalization ~ 100, with feature normalization/scaling ~ 200.

I feel that I've implemented something wrong somewhere but have not got any lines to figure out what I've messed up or whether this is even normal behavior.

[Update]
This is how my loss function plot looks through the descent. But I'm noticing that the variation of loss is too much. Am I wrong ?
Loss vs number of iterations plot
x-axis - number of iterations across 100 batch sizes; y-axis - Loss

Here's my code

import pandas as pd
import numpy as np
import torch
from torch.utils.data import DataLoader, Dataset, random_split
from sklearn.preprocessing import StandardScaler

import torch
import torch.nn as nn
from torch.autograd import Variable
from torch.optim import SGD

class LinearRegression(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LinearRegression, self).__init__()
        self.fc1 = nn.Linear(input_dim, output_dim)

    def forward(self, x):
        output = self.fc1(x)
        return output

class LoaderWeather(Dataset):
    def __init__(self, file_path, transform=None):
        self.file_path = file_path
        self.transform = transform
        # self.scaler = StandardScaler()

        self.data = pd.read_csv(self.file_path)
        self.data = self.data.loc[:, self.data.dtypes == np.float64]

        self.outputs = self.data["Temperature (C)"]
        self.inputs = self.data.drop(["Apparent Temperature (C)", "Temperature (C)"], axis=1)

        self.outputs = np.array(self.outputs).reshape(-1,1)
        self.inputs = np.array(self.inputs)

        # self.inputs = self.scaler.fit_transform(np.array(self.inputs))

        #Normalize data
        # self.means = self.inputs.mean(axis=1).reshape(-1,1)
        # self.stds = self.inputs.std(axis=1).reshape(-1,1)
        # self.inputs = (self.inputs - self.means)/self.stds

    def __getitem__(self, idx):
        return self.inputs[idx], self.outputs[idx]

    def __len__(self):
        return len(self.inputs)

def train_epoch(epoch, train_loader, regression_model, criterion, optimizer):

        print(epoch)
        train_loader = iter(train_loader)
        #Convert to tensors
        for iteration, (inputs, outputs) in enumerate(train_loader):
            # inputs,outputs = train_loader.next()

            inputs = Variable(torch.tensor(inputs).type(torch.FloatTensor))
            outputs = Variable(torch.tensor(outputs).type(torch.FloatTensor))

            #Clear the gradients w.r.t the parameters
            optimizer.zero_grad()

            #Forward pass
            predicted = regression_model(inputs)

            #Calculate loss
            loss = criterion(predicted, outputs)

            #Backpropogate gradient of loss
            loss.backward()

            #Perform one step of gradient descent
            optimizer.step()

            #Verbosity
            if iteration % 100 == 0:
                print("epochs :{}, loss :{} , iteration: {},".format(epoch, loss, iteration))

        return regression_model, loss

def main():

 ############################################
    #Hyper parameters
    batch_size=100
    n_iters = 20000
    learning_rate=0.0000001
#############################################

    weather_data = LoaderWeather("data/weatherHistory.csv")

    n = len(weather_data)
    n_train = int(0.8 * len(weather_data))
    n_test = n - n_train

    train, test = random_split(weather_data, lengths=[n_train, n_test])

    train_loader = DataLoader(train, batch_size=batch_size, shuffle=True, num_workers=4)
    test_loader = DataLoader(test, batch_size=batch_size, num_workers=4)

    input_dim = 6
    output_dim = 1

    regression_model = LinearRegression(input_dim, output_dim)
    criterion = nn.MSELoss()

    optimizer = SGD(regression_model.parameters(), lr=learning_rate)

    no_of_batches = np.floor(n/batch_size)

    print(no_of_batches)
    epochs = int(n_iters/no_of_batches)

    print("Total epochs : {}".format(epochs))

    for epoch in range(epochs):
        regression_model, loss = train_epoch(epoch, train_loader, regression_model, criterion, optimizer)

    print(regression_model, loss)
    print(regression_model.parameters())
if __name__ == '__main__':
    main()

Any help would be greatly appreciated. Also, please let me know if the way I'm doing my training is fine or something is fundamentally wrong here.

Thank you


title: Subprocessing Data Loading in pytroch into Google Colab

body: I'm working on training a deep neural network using pytorch and I use DataLoader for preprocessing data and multi-processing purpose over dataset. I set num_workers attribute to positive number like 4 and my batch_size is 8. I train network on Google Colab Environment but when training keep on after few minutes, stop training and get error in reading .PNG files. I think it's memory error and I want to know what is relation between number of GPU and batch_size and num_workers to set up a reasonable relation between them specially in Google Colab .


title: Pytorch differences between two tensors

body:
I have two tensors like this:
1st tensor
[[0,0],[0,1],[0,2],[1,3],[1,4],[2,1],[2,4]]

2nd tensor
[[0,1],[0,2],[1,4],[2,4]]

I want the result tensor to be like this:
[[0,0],[1,3],[2,1]] # differences between 1st tensor and 2nd tensor

I have tried to use set, list, torch.where,.. and couldn't find any good way to achieve this. Is there any way to get the different rows between two different sizes of tensors? (need to be efficient)


title: Using PyTorch on AWS Lambda

body: Has anyone had any luck being able to use PyTorch on AWS Lambda for feature extraction from images or just using the framework at all? I finally got PyTorch, numpy, and pillow zipped in a folder under the uncompressed size limit (which is actually around 262 MB) but I had to build PyTorch from source to do this. The problem I am having now is that Lambda has a very old version of gcc running on it (4.8.3) which is very buggy and missing whole header files altogether. I believe the Pytorch docs state you should be using at least gcc 7 or later but I'm hoping someone may have found a way around this? I built the source using gcc 7.5 but then when I tried to import torch Lambda obviously used it's installed version of 4.8.3 causing an error on import: Floating point exception (core dumped) which stems from the old version of gcc. Is there a possible solution around this? I've been at this for a day and a half now so any help would be great. I think the bottom line is I am facing this similar issue. Better yet does anyone have a Pytorch lambda layer I could use?


title: Select specific rows of 2D PyTorch tensor

body:
Suppose I have a 2D tensor looking something like this:

[[44, 50, 1, 32],
.
.
.
[7, 13, 90, 83]]

and a list of  row indices that I want to select that looks something like this [0, 34, 100, ..., 745]. How can I go through and create a new tensor that contains only the rows whose indices are contained in the array?


title: Pytorch modified DQN algorithm error "the derivative for ' ' is not implemented"

body:
I'm trying to construct new observation and applying it into DQN.
I use pytorch DQN algorithm with my observation code.
its not finished yet, so data is not clear.

I changed some line in whole code as follows.

import sys, math
import random as rd
import numpy as np
import matplotlib
import matplotlib.pyplot as plt

######################################
from collections import namedtuple
from itertools import count
from PIL import Image

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.autograd import Variable
import torchvision.transforms as T

Transition = namedtuple('Transition',
                    ('state', 'action', 'next_state', 'reward'))

class ReplayMemory(object):

def __init__(self, capacity):
    self.capacity = capacity
    self.memory = []
    self.position = 0

def push(self, *args):
    """Saves a transition."""
    if len(self.memory) < self.capacity:
        self.memory.append(None)
    self.memory[self.position] = Transition(*args)
    self.position = (self.position + 1) % self.capacity

def sample(self, batch_size):
    return rd.sample(self.memory, batch_size)

def __len__(self):
    return len(self.memory)

class DQN(nn.Module):

def __init__(self):
    super(DQN, self).__init__()
    self.l1 = nn.Linear(5, 16)
    self.l2 = nn.Linear(16, 12)
    self.l3 = nn.Linear(12, 20)
    self.head = nn.Linear(20, 1)
def forward(self, x):
    x = F.relu(self.l1(x))
    x = F.relu(self.l2(x))
    x = F.relu(self.l3(x))
    return self.head(x.view(x.size(0), -1))

BATCH_SIZE = 5
GAMMA = 0.999
EPS_START = 0.9
EPS_END = 0.05
EPS_DECAY = 200
TARGET_UPDATE = 5
policy_net = DQN()
target_net = DQN()
target_net.load_state_dict(policy_net.state_dict())
target_net.eval()

optimizer = optim.RMSprop(policy_net.parameters())
memory = ReplayMemory(10000)

def optimize_model():
if len(memory) < BATCH_SIZE:
    return
transitions = memory.sample(BATCH_SIZE)
# Transpose the batch (see http://stackoverflow.com/a/19343/3343043 for
# detailed explanation).
batch = Transition(*zip(*transitions))
print("batch = ", batch, "\n")
# Compute a mask of non-final states and concatenate the batch elements

state_batch = Variable(torch.cat(batch.state))
print("state_batch = ", state_batch)
action_batch = Variable(torch.cat(batch.action))
print("action_batch = ", action_batch)
reward_batch = Variable(torch.cat(batch.reward), requires_grad = False)
print("reward_batch = ", reward_batch)
next_state_batch = Variable(torch.cat(batch.next_state), requires_grad = False)

# Compute Q(s_t, a) - the model computes Q(s_t), then we select the
# columns of actions taken
state_action_values = policy_net(state_batch)
print("state_action_values = ", state_action_values)

# Compute V(s_{t+1}) for all next states.
next_state_values = target_net(next_state_batch)
print("next_state_values = ", next_state_values)

# Compute the expected Q values
expected_state_action_values = (next_state_values * GAMMA) + reward_batch
print("expected next state values = ", expected_state_action_values)

# Compute Huber loss
loss = F.smooth_l1_loss(state_action_values, expected_state_action_values, reduce = False)

# Optimize the model
optimizer.zero_grad()
loss.backward()
for param in policy_net.parameters():
    param.grad.data.clamp_(-1, 1)
optimizer.step()

num_episodes = 5
for i_episode in range(num_episodes):
    # Initialize the environment and state
    drive = AutoDrive(20, 20, 0, 16, 0) # x/y/yaw/velocity/heading
    drive._make_observation(0, -1, -1, -1, -1, -1) random other vehicle     location, parameters

    stand = 3

    # exploit 1
    e = 1. / ((i_episode // 100) + 1) # conditioon for choosing action
    optimizer.zero_grad()

    for t in range(stand):
        # Select and perform an action

#         exploit 2
        if np.random.rand(1) > e:
            action = rd.randint(1, 4)
        else:
            action = np.argmax(drive._select_action(0.5, 0.5)) + 1 #index + 1
        print("state = ", drive.state, ", action = ", action, ", yaw = ",     drive.yaw, ", mag = ", drive.mag)
    state = drive.state
    drive._step(action)
    drive._calc_reward(0.5, 0.5)
    print(drive.reward)
    if (drive.reward == -10):
        break

    # Store the transition in memory
    state1 = torch.FloatTensor(state).view(1, 5)
    state2 = torch.FloatTensor(drive.state).view(1, 5)
    action = torch.FloatTensor([float(action)]).view(1, 1)
    reward = torch.FloatTensor([drive.reward]).view(1, 1)
    memory.push(state1, action\
                , state2, reward)

    # Perform one step of the optimization (on the target network)
    optimize_model()
    if done:
        episode_durations.append(t + 1)
        plot_durations()
        break
# Update the target network
if i_episode % TARGET_UPDATE == 0:
    target_net.load_state_dict(policy_net.state_dict())

the error occurs in the loss function

  File "<ipython-input-190-29dcdbbf0383>", line 1, in <module>
    runfile('C:/Users/desktop/.spyder-py3/temp.py',     wdir='C:/Users/desktop/.spyder-py3')

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
execfile(filename, namespace)

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/desktop/.spyder-py3/temp.py", line 441, in <module>
optimize_model()

  File "C:/Users/desktop/.spyder-py3/temp.py", line 362, in optimize_model
loss = F.smooth_l1_loss(state_action_values, expected_state_action_values, reduce = False)

RuntimeError: the derivative for 'target' is not implemented

the inputs in the function are as follows.

expected next state values =  Variable containing:
  8.9615
 12.0198
 12.0488
 12.2920
 13.9062
[torch.FloatTensor of size 5x1]

state_action_values =  Variable containing:
 0.3765
 0.5196
 0.4587
 0.3765
 0.5636
[torch.FloatTensor of size 5x1]

what i have to do? im really beginner of it, so helpful advice are welcome


title: How to vectorize this pytorch code over (at least) the batch dimension?

body:
I want to implement a code to build an adjacency matrix such that (for example):

If X[0] : [0, 1, 2, 0, 1, 0], then,

A[0, 1] = 1
A[1, 2] = 1
A[2, 0] = 1
A[0, 1] = 1
A[1, 0] = 1

The following code works fine, however, it's too slow! So, please help me to vectorize this code on the batch (first) dimension at least:

A = torch.zeros((3, 3, 3), dtype = torch.float)
X = torch.tensor([[0, 1, 2, 0, 1, 0], [1, 0, 0, 2, 1, 1], [0, 0, 2, 2, 1, 1]])
for a, x in zip(A, X):
    for i, j in zip(x, x[1:]):
        a[i, j] = 1

Thanks! :)


title: how to use torchaudio with torch xla on google colab tpu

body:
I'm trying to run a pytorch script which is using torchaudio on a google TPU. To do this I'm using pytorch xla following this notebook, more specifically I'm using this code cell to load the xla:

!pip install torchaudio
import os
assert os.environ['COLAB_TPU_ADDR'], 'Make sure to select TPU from Edit > Notebook settings > Hardware accelerator'

VERSION = "20200220"  #@param ["20200220","nightly", "xrt==1.15.0"]
!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!python pytorch-xla-env-setup.py --version $VERSION

import torch

import torchaudio

import torch_xla

however this is incompatible with the version of torchaudio that I need as: ERROR: torchaudio 0.4.0 has requirement torch==1.4.0, but you'll have torch 1.5.0a0+e95282a which is incompatible.

I couldn't find anywhere how to load torch 1.4.0 using pytorch xla.

I tried to use the nightly version of torch audio but that gives the error as follows:

!pip install torchaudio_nightly -f https://download.pytorch.org/whl/nightly/torch_nightly.html

import os
assert os.environ['COLAB_TPU_ADDR'], 'Make sure to select TPU from Edit > Notebook settings > Hardware accelerator'

VERSION = "20200220"  #@param ["20200220","nightly", "xrt==1.15.0"]
!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!python pytorch-xla-env-setup.py --version $VERSION

import torch
import torchaudio

import torch_xla

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-968e9d93c06f> in <module>()
      9
     10 import torch
---> 11 import torchaudio
     12
     13 import torch_xla

/usr/local/lib/python3.6/dist-packages/torchaudio/__init__.py in <module>()
      3
      4 import torch
----> 5 import _torch_sox
      6
      7 from .version import __version__, git_version

ImportError: /usr/local/lib/python3.6/dist-packages/_torch_sox.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_29E

---------------------------------------------------------------------------

So how would I go to load the stable version or 1.4.0 version of pytorch using xla or is there any other workaround for this situation?

Thanks a lot for your help!


title: How to initialize the weights in pytorch with my own function? (cs231n PyTorch notebook)

body:
In the cs231n pytorch notebook, they define random_weight function to return a tensor object that initialized by Kaiming normalization, and zero_weight function to return a tensor contains zero bias.

The random_weight implementation:

def random_weight(shape):
    if len(shape) == 2:  # FC weight
        fan_in = shape[0]
    else:
        fan_in = np.prod(shape[1:])
    w = torch.randn(shape, device=device, dtype=dtype) * np.sqrt(2. / fan_in)
    w.requires_grad = True
    return w

The zero_weight implementation:

def zero_weight(shape):
    return torch.zeros(shape, device=device, dtype=dtype, requires_grad=True)

Now the goal is to define a three-layer ConvNet using the pytorch sequential api, and we have to use the two above functions to initialize all the weights in the ConvNet.

I found some implementation like this:

model = nn.Sequential(....)

def init_weights(layer):
    if type(layer) in [nn.Conv2d, nn.Linear]:
        random_weight(layer.weight.size())
        zero_weight(layer.bias.size())

model.apply(init_weights)
optimizer = optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9, nesterov=True)

But I suspect that the weights are not initialized by the two functions.
Because I print the layer.weight.data before and after calling the random_weight, I got same data.

So I try to modify the init_weights code like this:

def init_weights(layer):
    if type(layer) in [nn.Conv2d, nn.Linear]:
        layer.weight.data = random_weight(layer.weight.size())
        layer.bias.data = zero_weight(layer.bias.size())

But the performance really sucks, I get about 10% accuracy on the validation set.
So, does anyone know the right way to using this two functions to initialize the weights?

All suggestions are welcome!
Thanks a lot!


title: Anoconda does not allow Torchvision to be installed

body:
I tried the following commands
conda create -n torch_env -c pytorch pytorch torchvision
conda install -c soumith/label/pytorch torchvision
conda install -c soumith torchvision

Provided by Anaconda but none of them works. Please help me I am stack !

Error message:
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

torchvision

Current channels:

https://conda.anaconda.org/pytorch/win-64
https://conda.anaconda.org/pytorch/noarch
https://repo.anaconda.com/pkgs/main/win-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/win-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/win-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/win-64
https://repo.anaconda.com/pkgs/pro/noarch
https://repo.anaconda.com/pkgs/msys2/win-64
https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.


title: DataLoader issue : RNN module weights are not part of single contiguous chunk of memory

body:
I am trying to build an LSTM model for time series data. The details of dataset:
Input data is time series with 800 subjects, each having a 2D array data of 60 rows and 200 columns. I loaded the entire data as a Tensor of shape [800,60,200] and labels for the classification problem is of shape [800,1].
I made a dictionary of the data using the following code :

class DataCurate(Dataset):
    def __init__(self, l1,l2, transform=None):
        self.l1=l1
        self.l2=l2
        self.transform=transform
    def __len__(self):
        return len(self.l1)
    def __getitem__(self, index):
        array=self.l1[index]
        label=self.l2[index]
        sample = {'time_data': array, 'labels': label}
        return sample

the data and labels are in variables x & y. I call data=Datacurate(x,y)

Later on I build and LSTM model for classification problem using the code:

class RNNModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
    super(RNNModel, self).__init__()
    self.hidden_size = hidden_size
    self.num_layers = num_layers
    self.lstm = nn.LSTM(input_size, hidden_size, num_layers,batch_first=True)
    self.linear = nn.Linear(hidden_size, num_layers, bias=True)
    self.softmax = nn.LogSoftmax()

    def forward(self, x)
        self.lstm.flatten_parameters()
        out_packed, state = self.lstm(x)  # RNN
        print("lstm output size: {out.size()}"+str(out_packed.size()))
        out = self.linear(out_packed[-1])  # linear transform
    print("linear output size {out.size()} "+str(out.size()))
    log_probs = F.log_softmax(out,dim=1)
    print("softmax output size {log_probs.size()}"+str(log_probs.size()))
        return log_probs

This gives me an error when I run the training script:

UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().
  out_packed, state = self.lstm(x)  # RNN
Traceback (most recent call last):
  File "main_2.py", line 100, in <module>
    output = model(train_inputs.transpose(0,1))
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/iab/disk_a/meghal/test/quickdraw_tutorial_dataset_v1/pytorch_RNN_examples/model.py", line 26, in forward
    out_packed, state = self.lstm(x)  # RNN
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/rnn.py", line 192, in forward
    output, hidden = func(input, self.all_weights, hx, batch_sizes)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/_functions/rnn.py", line 324, in forward
    return func(input, *fargs, **fkwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/_functions/rnn.py", line 288, in forward
    dropout_ts)
RuntimeError: param_from.type() == param_to.type() ASSERT FAILED at /pytorch/aten/src/ATen/native/cudnn/RNN.cpp:491, please report a bug to PyTorch. parameter types mismatch

I don't know what this means and how do I resolve it. I am completely new to LSTM.


title: Why doesn't the Discriminator's and Generators' loss change?

body:
I'm trying to implement a Generative Adversarial Network (GAN) for the MNIST-Dataset.
I use Pytorch for this. My problem is, that after one epoch the Discriminator's and the Generator's loss doesn't change.

I already tried two other methods to build the network, but they cause all the same problem :/

import os
import torch
import matplotlib.pyplot as plt
import matplotlib.gridspec as grd
import numpy as np
import torch.optim as optim
import torch.nn as nn
import torch.nn.functional as F
import torchvision #Datasets
from torchvision.utils import save_image
import torchvision.transforms as transforms
from torch.autograd import Variable
import pylab

#Parameter
batch_size = 64
epochs = 50000
image_size = 784
hidden_size = 392
sample_dir = 'samples'
save_dir = 'save'
noise_size = 100
lr = 0.001

# Image processing
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5,),(0.5,))])

# Discriminator
D = nn.Sequential(
    nn.Linear(image_size, hidden_size),
    nn.ReLU(),
    nn.Linear(hidden_size, 1),
    nn.Sigmoid()
)

# Generator
G = nn.Sequential(
    nn.Linear(noise_size, hidden_size),
    nn.ReLU(),
    nn.Linear(hidden_size, image_size),
    nn.Sigmoid()
)

# Lossfunction and optimizer (sigmoid cross entropy with logits and Adam)
criterion = nn.BCEWithLogitsLoss()
d_optimizer = torch.optim.Adam(D.parameters(), lr = lr)
g_optimizer = torch.optim.Adam(G.parameters(), lr = lr)

def reset_grad():
    d_optimizer.zero_grad()
    g_optimizer.zero_grad()

# Statistics to be saved
d_losses = np.zeros(epochs)
g_losses = np.zeros(epochs)
real_scores = np.zeros(epochs)
fake_scores = np.zeros(epochs)

# Start training
total_step = len(data_loader)
for epoch in range(epochs):
    for i, (images, _) in enumerate(data_loader):
        if images.shape[0] != 64:
            continue
        images = images.view(batch_size, -1).cuda()
        images = Variable(images)
        # Create the labels which are later used as input for the BCE loss
        real_labels = torch.ones(batch_size, 1).cuda()
        real_labels = Variable(real_labels)
        fake_labels = torch.zeros(batch_size, 1).cuda()
        fake_labels = Variable(fake_labels)

        # Train discriminator

        # Compute BCE_WithLogitsLoss using real images
        outputs = D(images)
        d_loss_real = criterion(outputs, real_labels)
        real_score = outputs

        # Compute BCE_WithLogitsLoss using fake images
        # First term of the loss is always zero since fake_labels == 0
        z = torch.randn(batch_size, noise_size).cuda()
        z = Variable(z)
        fake_images = G(z)
        outputs = D(fake_images)
        d_loss_fake = criterion(outputs, fake_labels)
        fake_score = outputs

        # Backprop and optimize
        # If D is trained so well, then don't update
        d_loss = d_loss_real + d_loss_fake
        reset_grad()
        d_loss.backward()
        d_optimizer.step()

        # Train generator

        # Compute loss with fake images
        z = torch.randn(batch_size, noise_size).cuda()
        z = Variable(z)
        fake_images = G(z)
        outputs = D(fake_images)

        # We train G to maximize log(D(G(z)) instead of minimizing log(1 -D(G(z)))
        # For the reason, see the last paragraph of section 3. https://arxiv.org/pdf/1406.2661.pdf
        g_loss = criterion(outputs, real_labels)

        # Backprop and optimize
        # if G is trained so well, then don't update
        reset_grad()
        g_loss.backward()
        g_optimizer.step()

        # Update statistics

        d_losses[epoch] = d_losses[epoch]*(i/(i+1.)) + d_loss.item()*(1./(i+1.))
        g_losses[epoch] = g_losses[epoch]*(i/(i+1.)) + g_loss.item()*(1./(i+1.))
        real_scores[epoch] = real_scores[epoch]*(i/(i+1.)) + real_score.mean().item()*(1./(i+1.))
        fake_scores[epoch] = fake_scores[epoch]*(i/(i+1.)) + fake_score.mean().item()*(1./(i+1.))

    # print results
    print('Epoch [{}/{}], d_loss: {:.4f}, g_loss: {:.4f}, D(x): {:.2f}, D(G(z)): {:.2f}'
            .format(epoch, epochs, d_loss.item(), g_loss.item(),
                    real_score.mean().item(), fake_score.mean().item()))

The Generator's and Discriminator's loss should change from epoch to epoch, but they don't.

Epoch [0/50000], d_loss: 1.0069, g_loss: 0.6927, D(x): 1.00, D(G(z)): 0.00
Epoch [1/50000], d_loss: 1.0065, g_loss: 0.6931, D(x): 1.00, D(G(z)): 0.00
Epoch [2/50000], d_loss: 1.0064, g_loss: 0.6931, D(x): 1.00, D(G(z)): 0.00
Epoch [3/50000], d_loss: 1.0064, g_loss: 0.6931, D(x): 1.00, D(G(z)): 0.00
Epoch [4/50000], d_loss: 1.0064, g_loss: 0.6931, D(x): 1.00, D(G(z)): 0.00
Epoch [5/50000], d_loss: 1.0064, g_loss: 0.6931, D(x): 1.00, D(G(z)): 0.00

Thank's for your help.


title: Why do I get RuntimeError: CUDA error: invalid argument in pytorch?

body:
Recently I've frequently been getting RuntimeError: CUDA error: invalid argument when calling functions like torch.cholesky e.g.:

import torch
a = torch.randn(3, 3, device="cuda:0")
a = torch.mm(a, a.t()) # make symmetric positive-definite
torch.cholesky(a)

This works fine if I use device="cpu" instead. This error isn't very descriptive, so I'm not sure what's wrong here.


title: How to open pretrained models in python

body:
Hi I'm trying to load some pretrained models from .sav files and so far nothing is working. The models were originally made in pytorch and when I open the raw file in vs-code I can see that all the appropiate information was stored correctly.

I've tried the following libraries:

sklearn.externals.joblib

pickle

scipy.io

pyreadstat

Each library either gave me an error (such as wrong timestamp or signature mismatch) or just return an int instead of a python object.

The models can be downloaded from this link.


I am building a Single Shot Multi-Box Detection project for object detection of a running dog in an mp4 video.
I am following a tutorial for this & have messaged the instructor.
This is the complete error when I run my code in Spyder:
RuntimeError: Legacy autograd function with non-static forward method is deprecated.
Please use new-style autograd function with static forward method.
(Example: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)

This is what my code looks like:

I have also, checked the Pytorch documentation as with the previous questioner to this issue on Jul 8.
https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html
I am a beginner in Deep Learning & am certain I am making a rube error.
I would be extremely grateful for the help.
Cheers,
Ryan


title: How to train a Pytorch net

body:
I'm using this Pytorch implementation of Segnet with pretrained values I found for object segmentation, and it works fine.
Now I want to resume the training from the values I have, using a new dataset with similar images.
How can I do that?

I guess I have to use the "train.py" file found in the repository, but I don't know what to write in order to replace the "fill the batch" comment.
Here is that portion of the code:

def train(epoch):
    model.train()

    # update learning rate
    lr = args.lr * (0.1 ** (epoch // 30))
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr

    # define a weighted loss (0 weight for 0 label)
    weights_list = [0]+[1 for i in range(17)]
    weights = np.asarray(weights_list)
    weigthtorch = torch.Tensor(weights_list)
    if(USE_CUDA):
        loss = nn.CrossEntropyLoss(weight=weigthtorch).cuda()
    else:
        loss = nn.CrossEntropyLoss(weight=weigthtorch)

    total_loss = 0

    # iteration over the batches
    batches = []
    for batch_idx,batch_files in enumerate(tqdm(batches)):

        # containers
        batch = np.zeros((args.batch_size,input_nbr, imsize, imsize), dtype=float)
        batch_labels = np.zeros((args.batch_size,imsize, imsize), dtype=int)

        # fill the batch
        # ...
        # What should I write here?

        batch_th = Variable(torch.Tensor(batch))
        target_th = Variable(torch.LongTensor(batch_labels))

        if USE_CUDA:
            batch_th =batch_th.cuda()
            target_th = target_th.cuda()

        # initilize gradients
        optimizer.zero_grad()

        # predictions
        output = model(batch_th)

        # Loss
        output = output.view(output.size(0),output.size(1), -1)
        output = torch.transpose(output,1,2).contiguous()
        output = output.view(-1,output.size(2))
        target = target.view(-1)

        l_ = loss(output.cuda(), target)
        total_loss += l_.cpu().data.numpy()
        l_.cuda()
        l_.backward()
        optimizer.step()

return total_loss/len(files)


title: How to install CUDA in Google Colab - Cannot initialize CUDA without ATen_cuda library

body:
I am trying to use cuda in Goolge Colab but while running my program I get the following error.

  RuntimeError: Cannot initialize CUDA without ATen_cuda library. PyTorch splits its backend into two shared libraries: a CPU library and a CUDA library; this error has occurred because you are trying to use some CUDA functionality, but the CUDA library has not been loaded by the dynamic linker for some reason.  The CUDA library MUST be loaded, EVEN IF you don't directly use any symbols from the CUDA library! One common culprit is a lack of -Wl,--no-as-needed in your link arguments; many dynamic linkers will delete dynamic library dependencies if you don't depend on any of their symbols.  You can check if this has occurred by using ldd on your binary to see if there is a dependency on *_cuda.so library.

I have the following libraries installed.

from os.path import exists
from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())
cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\.\([0-9]*\)\.\([0-9]*\)$/cu\1\2/'
accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'

!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.4.1-

{platform}-linux_x86_64.whl torchvision
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.pyplot as plt
import time
import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
from torchvision import datasets, transforms, models
!pip install Pillow==5.3.0
# import the new one
import PIL

And I am trying to run the following code.

for device in ['cpu', 'cuda']:

    criterion = nn.NLLLoss()
    # Only train the classifier parameters, feature parameters are frozen
    optimizer = optim.Adam(model.classifier.parameters(), lr=0.001)

    model.to(device)

    for ii, (inputs, labels) in enumerate(trainloader):

        # Move input and label tensors to the GPU
        inputs, labels = inputs.to(device), labels.to(device)

        start = time.time()

        outputs = model.forward(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        if ii==3:
            break

    print(f"Device = {device}; Time per batch: {(time.time() - start)/3:.3f} seconds")


title: what is the equivalent of theano.tensor.clip in pytorch?

body: I want to clip my tensor (not gradient) values to some range. Is there any function in pytorch like there is a function theano.tensor.clip() in theano?


title: Pytorch RNN always gives the same output for multivariate time series

body:
I have a time series data looking something like this:

I am trying to model this with a sequence to sequence RNN in pytorch. It trains well and I can see the loss going down. But on testing it gives the same out put irrespective of the input.

My Model:

class RNNModel(nn.Module):

def __init__(self, predictor_size, hidden_size, num_layers, dropout = 0.3, output_size=83):
    super(RNNModel, self).__init__()
    self.drop = nn.Dropout(dropout)
    self.rnn = nn.GRU(predictor_size, hidden_size, num_layers=num_layers, dropout = dropout)
    self.decoder = nn.Linear(hidden_size, output_size)
    self.init_weights()
    self.hidden_size = hidden_size
    self.num_layers = num_layers

def init_weights(self):
    initrange = 0.1
    self.decoder.bias.data.fill_(0)
    self.decoder.weight.data.uniform_(-initrange, initrange)

def forward(self, input, hidden):
    output, hidden = self.rnn(input, hidden)
    output = self.drop(output)
    decoded = self.decoder(output.view(output.size(0) * output.size(1), output.size(2)))
    return decoded.view(output.size(0), output.size(1), decoded.size(1)), hidden

def init_hidden(self, batch_size):
    weight = next(self.parameters()).data
    return Variable(weight.new(self.num_layers, batch_size, self.hidden_size).zero_())

Train Method:

def train(data_source, lr):
    # turn on training mode that enables dropout

    model.train()
    total_loss = 0
    hidden = model.init_hidden(bs_train)
    optimizer = optim.Adam(model.parameters(), lr = lr)

    for batch, i in enumerate(range(0, data_source.size(0) - 1, bptt_size)):

        data, targets = get_batch(data_source, i)

        # Starting each batch, we detach the hidden state from how it was previously produced
        # so that model doesen't ry to backprop to all the way start of the dataset
        # unrolling of the graph will go from the last iteration to the first iteration
        hidden = Variable(hidden.data)
        if cuda.is_available():
            hidden = hidden.cuda()
        optimizer.zero_grad()

        output, hidden = model(data, hidden)
        loss = criterion(output, targets)
        loss.backward()

        # clip_grad_norm to prevent gradient explosion
        torch.nn.utils.clip_grad_norm(model.parameters(), clip)

        optimizer.step()
        total_loss += len(data) * loss.data
        # return accumulated loss for all the iterations
        return total_loss[0] / len(data_source)

Evaluation Method:

def evaluate(data_source):
    # turn on evaluation to disable dropout
    model.eval()
    model.train(False)
    total_loss = 0
    hidden = model.init_hidden(bs_valid)

    for i in range(0, data_source.size(0) - 1, bptt_size):
        data, targets = get_batch(data_source, i, evaluation = True)

        if cuda.is_available():
            hidden = hidden.cuda()

        output, hidden = model(data, hidden)
        total_loss += len(data) * criterion(output, targets).data
        hidden = Variable(hidden.data)

    return total_loss[0]/len(data_source)

Training Loop:

best_val_loss = None
best_epoch = 0
def run(epochs, lr):
    val_losses = []
    num_epochs = []
    global best_val_loss
    global best_epoch
    for epoch in range(0, epochs):
        train_loss = train(train_set, lr)
        val_loss = evaluate(test_set)
        num_epochs.append(epoch)
        val_losses.append(val_loss)
        print("Train Loss: ", train_loss, " Validation Loss: ", val_loss)

        if not best_val_loss or val_loss < best_val_loss:
            best_val_loss = val_loss
            torch.save(model.state_dict(), "./4.model.pth")
            best_epoch = epoch
    return num_epochs, val_losses

Loss with epochs:

Getting the output:

model = RNNModel(predictor_size, hidden_size, num_layers, dropout_pct, output_size)
model.load_state_dict(torch.load("./4.model.pth"))

if cuda.is_available():
    model.cuda()

model.eval()
model.train(False)
hidden = model.init_hidden(1)
inp = torch.Tensor(var[105])
input = Variable(inp.contiguous().view(1,1,predictor_size), volatile=True)
if cuda.is_available():
    input.data = input.data.cuda()
output, hidden = model(input, hidden)
op = output.squeeze().data.cpu()
print(op)

Here I always get the same output irrespective of datapoint I give as input. Can somebody please tell me what I am doing wrong.