title: How to get the filename of a sample from a DataLoader?

body:
I need to write a file with the result of the data test of a Convolutional Neural Network that I trained. The data include speech data collection. The file format needs to be "file name, prediction", but I am having a hard time to extract the file name. I load the data like this:

import torchvision
from torchvision import transforms
from torch.utils.data import DataLoader

TEST_DATA_PATH = ...

trans = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))
])

test_dataset = torchvision.datasets.MNIST(
    root=TEST_DATA_PATH,
    train=False,
    transform=trans,
    download=True
)

test_loader = DataLoader(dataset=test_dataset, batch_size=1, shuffle=False)

and I am trying to write to the file as follows:

f = open("test_y", "w")
with torch.no_grad():
    for i, (images, labels) in enumerate(test_loader, 0):
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        file = os.listdir(TEST_DATA_PATH + "/all")[i]
        format = file + ", " + str(predicted.item()) + '\n'
        f.write(format)
f.close()

The problem with os.listdir(TESTH_DATA_PATH + "/all")[i] is that it is not synchronized with the loaded files order of test_loader. What can I do?


title: Pytorch TypeError: eq() received an invalid combination of arguments

body:
num_samples = 10
def predict(x):
    sampled_models = [guide(None, None) for _ in range(num_samples)]
    yhats = [model(x).data for model in sampled_models]
    mean = torch.mean(torch.stack(yhats), 0)
    return np.argmax(mean.numpy(), axis=1)

print('Prediction when network is forced to predict')
correct = 0
total = 0
for j, data in enumerate(test_loader):
    images, labels = data
    predicted = predict(images.view(-1,28*28))
    total += labels.size(0)
    correct += (predicted == labels).sum().item()
print("accuracy: %d %%" % (100 * correct / total))

Error:

correct += (predicted == labels).sum().item() TypeError:
eq() received an invalid combination of arguments - got (numpy.ndarray), but expected one of:
* (Tensor other)
  didn't match because some of the arguments have invalid types: (!numpy.ndarray!)
* (Number other)
  didn't match because some of the arguments have invalid types: (!numpy.ndarray!)

*


title: "no module named torch". But installed pytorch 1.3.0 with conda in Ubuntu 18.04.02 Server Edition

body:
installed pytorch with conda :

(base) (3.8.0/envs/my_virtual_env-3.8.0) marco@pc:~/facenet_pytorch/examples$ conda install
pytorch torchvision cpuonly -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

I updated conda:

(base) (3.8.0/envs/my_virtual_env-3.8.0) marco@pc:~/facenet_pytorch/examples$ conda update
conda
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

Installed mkl=2019 :

(base) (3.8.0/envs/my_virtual_env-3.8.0) marco@pc:~/facenet_pytorch/examples$ conda install
mkl=2019
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

(base) (3.8.0/envs/my_virtual_env-3.8.0) marco@pc:~/facenet_pytorch/examples$ conda list | grep
torch
cpuonly                   1.0                           0    pytorch
facenet-pytorch           0.1.0                    pypi_0    pypi
pytorch                   1.3.0               py3.7_cpu_0  [cpuonly]  pytorch
torchfile                 0.1.0                    pypi_0    pypi
torchvision               0.4.1                  py37_cpu  [cpuonly]  pytorch

But it still says "no module torch" :

(base) (3.8.0/envs/my_virtual_env-3.8.0) marco@pc:~/facenet_pytorch/examples$ python3
Python 3.8.0 (default, Oct 30 2019, 16:20:23)
[GCC 7.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
>>>

I discovered that the problem appears only with python 3.8.0 version

(base) marco@pc:~/facenet_pytorch$ python3
Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>>

Ubuntu 18.04.02 Server Edition

Or, may be, it's just a matter of python environments, as you said.
But I do not understand why just activating conda environment, with "conda activate", it doesn't work

Marco


title: Impact of using data shuffling in Pytorch dataloader

body:
I implemented an image classification network to classify a dataset of 100 classes by using Alexnet as a pretrained model and changing the final output layers.
I noticed when I was loading my data like

trainloader = torch.utils.data.DataLoader(train_data, batch_size=32, shuffle=False)

, I was getting accuracy on validation dataset around 2-3 % for around 10 epochs but when I just changed shuffle=True and retrained the network, the accuracy jumped to 70% in the first epoch itself.

I was wondering if it happened because in the first case the network was being shown one example after the other continuously for just one class for few instances resulting in network making poor generalizations during training or is there some other reason behind it?

But, I did not expect that to have such a drastic impact.

P.S: All the code and parameters were exactly the same for both the cases except changing the shuffle option.


title: NameError: name 'nn' is not defined

body:
Several times when I copy PyTorch code I get this error:

NameError: name 'nn' is not defined

What is missing? What is nn?

To reproduce:

class SLL(nn.Module):
    "single linear layer"
    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(10,100)

    def forward(self)->None:
        print("SLL:forward")

m1 = SLL()


title: How to construct a network with two inputs in PyTorch

body:
Suppose I want to have the general neural network architecture:

Input1 --> CNNLayer
                    \
                     ---> FCLayer ---> Output
                    /
Input2 --> FCLayer

Input1 is image data, input2 is non-image data. I have implemented this architecture in Tensorflow.

All pytorch examples I have found are one input go through each layer. How can I define forward func to process 2 inputs separately then combine them in a middle layer?


title: Pytorch: RuntimeError: reduce failed to synchronize: cudaErrorAssert: device-side assert triggered

body:
I am running into the following error when trying to train this on this dataset.

Since this is the configuration published in the paper, I am assuming I am doing something incredibly wrong.

This error arrives on a different image every time I try to run training.

C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [6,0,0] Assertion `t >= 0 && t < n_classes` failed.
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.1\helpers\pydev\pydevd.py", line 1741, in <module>
    main()
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.1\helpers\pydev\pydevd.py", line 1735, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.1\helpers\pydev\pydevd.py", line 1135, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.1\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Noam/Code/vision_course/hopenet/deep-head-pose/code/original_code_augmented/train_hopenet_with_validation_holdout.py", line 187, in <module>
    loss_reg_yaw = reg_criterion(yaw_predicted, label_yaw_cont)
  File "C:\Noam\Code\vision_course\hopenet\venv\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Noam\Code\vision_course\hopenet\venv\lib\site-packages\torch\nn\modules\loss.py", line 431, in forward
    return F.mse_loss(input, target, reduction=self.reduction)
  File "C:\Noam\Code\vision_course\hopenet\venv\lib\site-packages\torch\nn\functional.py", line 2204, in mse_loss
    ret = torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
RuntimeError: reduce failed to synchronize: cudaErrorAssert: device-side assert triggered

Any ideas?


title: is there any difference between matmul and usual multiplication of tensors

body:
I am confused between the multiplication between two tensors using * and matmul.
Below is my code

import torch
torch.manual_seed(7)
features = torch.randn((2, 5))
weights = torch.randn_like(features)

here, i want to multiply weights and features. so, one way to do it is as follows

print(torch.sum(features * weights))

Output:

tensor(-2.6123)

Another way to do is using matmul

print(torch.mm(features,weights.view((5,2))))

but, here output is

tensor([[ 2.8089,  4.6439],
        [-2.3988, -1.9238]])

What i don't understand here is that why matmul and usual multiplication are giving different outputs, when both are same. Am i doing anything wrong here?

Edit: When, i am using feature of shape (1,5) both * and matmul outputs are same.
but, its not the same when the shape is (2,5).


title: CUDA kernel failed : no kernel image is available for execution on the device, Error when running PyTorch model inside Google Compute VM

body:
I have a docker image of a PyTorch model that returns this error when run inside a google compute engine VM running on debian/Tesla P4 GPU/google deep learning image:

CUDA kernel failed : no kernel image is available for execution on the device

This occurs on the line where my model is called. The PyTorch model includes custom c++ extensions, I'm using this model https://github.com/daveredrum/Pointnet2.ScanNet

My image installs these at runtime

The image runs fine on my local system. Both VM and my system have these versions:

Cuda compilation tools    10.1, V10.1.243

torch                     1.4.0

torchvision               0.5.0

The main difference is the GPU as far as I'm aware

Local:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.21       Driver Version: 435.21       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960M    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   36C    P8    N/A /  N/A |    361MiB /  2004MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

VM:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   42C    P0    23W /  75W |      0MiB /  7611MiB |      3%      Default |

If I ssh into the VM torch.cuda.is_available() returns true

Therefore I suspect it must be something to do with the compilation of the extensions

This is the relevant part of my docker file:

ENV CUDA_HOME "/usr/local/cuda-10.1"
ENV PATH /usr/local/nvidia/bin:/usr/local/cuda-10.1/bin:${PATH}
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
ENV NVIDIA_REQUIRE_CUDA "cuda>=10.1 brand=tesla,driver>=384,driver<385 brand=tesla,driver>=396,driver<397 brand=tesla,driver>=410,driver<411 brand=tesla,driver>=418,driver<419"
ENV FORCE_CUDA=1

# CUDA 10.1-specific steps
RUN conda install -c open3d-admin open3d
RUN conda install -y -c pytorch \
    cudatoolkit=10.1 \
    "pytorch=1.4.0=py3.6_cuda10.1.243_cudnn7.6.3_0" \
    "torchvision=0.5.0=py36_cu101" \
 && conda clean -ya
RUN pip install -r requirements.txt
RUN pip install flask
RUN pip install plyfile
RUN pip install scipy

# Install OpenCV3 Python bindings
RUN sudo apt-get update && sudo apt-get install -y --no-install-recommends \
    libgtk2.0-0 \
    libcanberra-gtk-module \
    libgl1-mesa-glx \
 && sudo rm -rf /var/lib/apt/lists/*

RUN dir
RUN cd pointnet2 && python setup.py install
RUN cd ..

I have already re-running this line from ssh in the VM:

TORCH_CUDA_ARCH_LIST="6.0 6.1 7.0" python setup.py install

Which I think targets the installation to the Tesla P4 compute capability?

Is there some other setting or troubleshooting step I can try?

I didn't know anything about docker/VMs/pytorch extensions until a couple of days ago, so somewhat shooting in the dark. Also this is my first stackoverflow post, apologies if I'm not following some etiquette, feel free to point out.


title: Pytorch RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select

body:
I am training a model that takes tokenized strings which are then passed through an embedding layer and an LSTM thereafter. However, there seems to be an error in the input, as it does not pass through the embedding layer.

class DrugModel(nn.Module):
    def __init__(self, input_dim, output_dim, hidden_dim, drug_embed_dim,
            lstm_layer, lstm_dropout, bi_lstm, linear_dropout, char_vocab_size,
            char_embed_dim, char_dropout, dist_fn, learning_rate,
            binary, is_mlp, weight_decay, is_graph, g_layer,
            g_hidden_dim, g_out_dim, g_dropout):

        super(DrugModel, self).__init__()

        # Save model configs
        self.drug_embed_dim = drug_embed_dim
        self.lstm_layer = lstm_layer
        self.char_dropout = char_dropout
        self.dist_fn = dist_fn
        self.binary = binary
        self.is_mlp = is_mlp
        self.is_graph = is_graph
        self.g_layer = g_layer
        self.g_dropout = g_dropout
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

        # For one-hot encoded SMILES
        if not is_mlp:
            self.char_embed = nn.Embedding(char_vocab_size, char_embed_dim,
                                           padding_idx=0)
            self.lstm = nn.LSTM(char_embed_dim, drug_embed_dim, lstm_layer,
                                bidirectional=False,
                                batch_first=True, dropout=lstm_dropout)
        # Distance function
        self.dist_fc = nn.Linear(drug_embed_dim, 1)

        if binary:
            # Binary Cross Entropy
            self.criterion = lambda x, y: y*torch.log(x) + (1-y)*torch.log(1-x)

    def init_lstm_h(self, batch_size):
        return (Variable(torch.zeros(
                self.lstm_layer*1, batch_size, self.drug_embed_dim)).cuda(),
                Variable(torch.zeros(
                self.lstm_layer*1, batch_size, self.drug_embed_dim)).cuda())

    # Set Siamese network as basic LSTM
    def siamese_sequence(self, inputs, length):
        # Character embedding
        inputs = inputs.long()
        inputs = inputs.cuda()

        self.char_embed = self.char_embed(inputs.to(self.device))
        c_embed = self.char_embed(inputs)
        # c_embed = F.dropout(c_embed, self.char_dropout)
        maxlen = inputs.size(1)

        if not self.training:
            # Sort c_embed
            _, sort_idx = torch.sort(length, dim=0, descending=True)
            _, unsort_idx = torch.sort(sort_idx, dim=0)
            maxlen = torch.max(length)

            # Pack padded sequence
            c_embed = c_embed.index_select(0, Variable(sort_idx).cuda())
            sorted_len = length.index_select(0, sort_idx).tolist()
            c_packed = pack_padded_sequence(c_embed, sorted_len, batch_first=True)

        else:
            c_packed = c_embed

        # Run LSTM
        init_lstm_h = self.init_lstm_h(inputs.size(0))
        lstm_out, states = self.lstm(c_packed, init_lstm_h)

        hidden = torch.transpose(states[0], 0, 1).contiguous().view(
                                 -1, 1 * self.drug_embed_dim)
        if not self.training:
            # Unsort hidden states
            outputs = hidden.index_select(0, Variable(unsort_idx).cuda())
        else:
            outputs = hidden

        return outputs

    def forward(self, key1, key2, targets, key1_len, key2_len, status, predict = False):
        if not self.is_mlp:
            output1 = self.siamese_sequence(key1, key1_len)
            output2 = self.siamese_sequence(key2, key2_len)

After instantiating the class I get the following error when passing the input through the embedding layer:

<ipython-input-128-432fcc7a1e39> in forward(self, key1, key2, targets, key1_len, key2_len, status, predict)
    129     def forward(self, key1, key2, targets, key1_len, key2_len, status, predict = False):
    130         if not self.is_mlp:
--> 131             output1 = self.siamese_sequence(key1, key1_len)
    132             output2 = self.siamese_sequence(key2, key2_len)
    133             set_trace()

<ipython-input-128-432fcc7a1e39> in siamese_sequence(self, inputs, length)
     74         inputs = inputs.cuda()
     75
---> 76         self.char_embed = self.char_embed(inputs.to(self.device))
     77         set_trace()
     78         c_embed = self.char_embed(inputs)

~/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/miniconda3/lib/python3.7/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115
    116     def extra_repr(self):

~/miniconda3/lib/python3.7/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485
   1486

RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select

despite the fact that the input (e.g. key1) has already been passed to cuda and has been transformed into long format:

tensor([[25, 33, 30,  ...,  0,  0,  0],
        [25,  7,  7,  ...,  0,  0,  0],
        [25,  7, 30,  ...,  0,  0,  0],
        ...,
        [25,  7, 33,  ...,  0,  0,  0],
        [25, 33, 41,  ...,  0,  0,  0],
        [25, 33, 41,  ...,  0,  0,  0]], device='cuda:0')


title: Why can GPU do matrix multiplication faster than CPU?

body:
I've been using GPU for a while without questioning it but now I'm curious.

Why can GPU do matrix multiplication much faster than CPU? Is it because of parallel processing? But I didn't write any parallel processing code. Does it do it automatically by itself?

Any intuition / high-level explanation will be appreciated!


title: pytorch: can't load CNN model and do prediction TypeError: 'collections.OrderedDict' object is not callable

body:
I trained a CNN model using MNIST dataset and now want to predict a classification of the image, which contains a number 3.

But when I tried to use this CNN to predict, pytorch gives me this error:

TypeError: 'collections.OrderedDict' object is not callable

And here's what I write:

cnn = torch.load("/usr/prakt/w153/Desktop/score_detector.pkl")
img = scipy.ndimage.imread("/usr/prakt/w153/Desktop/resize_num_three.png")
test_x = Variable(torch.unsqueeze(torch.FloatTensor(img), dim=1), volatile=True).type(torch.FloatTensor).cuda()
test_output, last_layer = cnn(test_x)
pred = torch.max(test_output, 1)[1].cuda().data.squeeze()
print(pred)

here's some explaination:
img is the to be predicted image with size 28*28 score_detector.pkl is the trained CNN model

any help will be appreciated!


title: Custom weight initialization in PyTorch

body:
What would be the right way to implement a custom weight initialization method in PyTorch?

I believe I can't directly add any method to 'torch.nn.init` but wish to initialize my model's weights with my own proprietary method.


title: PyTorch "Caught IndexError in DataLoader worker process 0", "IndexError: too many indices for array"

body:
I am trying to implement a detection model based on "finetuning object detection" official tutorial of PyTorch.
It seemed to have worked with minimal data, (for 10 of images). However I uploaded my whole dataset to Drive and checked the index-data-label correspondences. There are not unmatching items in my setup, I have all the errors in that part solved. (I deleted extra items from the labels on GDrive)

class SomeDataset(torch.utils.data.Dataset):
def __init__(self, root_path, transforms):
    self.root_path = root_path
    self.transforms = transforms
    # load all image files, sorting them to
    # ensure that they are aligned
    self.imgs = list(sorted(os.listdir(os.path.join(root_path, "images"))))
    self.labels = list(sorted(os.listdir(os.path.join(root_path, "labels"))))

def __getitem__(self, idx):
    # load images ad masks
    img_path = os.path.join(self.root_path, "images", self.imgs[idx])
    label_path = os.path.join(self.root_path, "labels", self.labels[idx])

    img = Image.open(img_path).convert("RGB")

    # get labels and boxes
    label_data = np.loadtxt(label_path, dtype=str, delimiter=' ');
    print(f"{len(label_data)} is the length of label data")
    num_objs = label_data.shape[0];
    if num_objs != 0:
        print(f"number of objects {num_objs}")
        # label values should start from 1
        for i,label_name in enumerate(classnames):
            label_data[np.where(label_name==label_data)] = i;

        label_data = label_data.astype(np.float);
        print(f"label data {label_data}")
        xs = label_data[:,0:8:2];
        ys = label_data[:,1:8:2];

        x_min = np.min(xs, axis=1)[...,np.newaxis];
        x_max = np.max(xs, axis=1)[...,np.newaxis];
        y_min = np.min(ys, axis=1)[...,np.newaxis];
        y_max = np.max(ys, axis=1)[...,np.newaxis];

        boxes = np.hstack((x_min,y_min,x_max,y_max));

        labels = label_data[:,8];
    else:
        # if there is no label add background whose label is 0
        boxes = [[0,0,1,1]];
        labels = [0];
        num_objs = 1;

    boxes = torch.as_tensor(boxes, dtype=torch.float32)
    labels = torch.as_tensor(labels, dtype=torch.int64)

    image_id = torch.tensor([idx])
    area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
    # suppose all instances are not crowd
    iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

    target = {}
    target["boxes"] = boxes
    target["labels"] = labels
    target["image_id"] = image_id
    target["area"] = area
    target["iscrowd"] = iscrowd

    if self.transforms is not None:
          img, target = self.transforms(img, target)

    return img, target

def __len__(self):
    return len(self.imgs)

My main method is like the following,

def main():
# train on the GPU or on the CPU, if a GPU is not available
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

# our dataset has 16 classes - background and others
num_classes = 16
# use our dataset and defined transformations
dataset = SomeDataset('trainImages', get_transform(train=True))
print(f"{len(dataset)} number of images in training dataset")
dataset_validation = SomeDataset('valImages', get_transform(train=True))
print(f"{len(dataset_validation)} number of images in validation dataset")

# define training and validation data loaders
data_loader = torch.utils.data.DataLoader(
    dataset, batch_size=20, shuffle=True, num_workers=4,
    collate_fn=utils.collate_fn)

data_loader_val = torch.utils.data.DataLoader(
    dataset_validation, batch_size=10, shuffle=False, num_workers=4,
    collate_fn=utils.collate_fn)

# get the model using our helper function
#model = get_model_instance_segmentation(num_classes)
model = get_rcnn(num_classes);

# move model to the right device
model.to(device)

# construct an optimizer
params = [p for p in model.parameters() if p.requires_grad]
#optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005);
optimizer = torch.optim.Adam(params, lr=0.0005);
# and a learning rate scheduler
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                               step_size=3,
                                               gamma=0.1)

# let's train it for 10 epochs
num_epochs = 5

for epoch in range(num_epochs):
    # train for one epoch, printing every 10 iterations
    train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=100)
    # update the learning rate
    lr_scheduler.step()
    # evaluate on the test dataset
    #evaluate(model, data_loader_test, device=device)

print("That's it!")
return model;

When I run my code, it runs for a few number of data (for example 10 of them) and then stops and gives out this error.

IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "<ipython-input-114-e0ccd94603fd>", line 31, in __getitem__
    xs = label_data[:,0:8:2];
IndexError: too many indices for array

The error goes from model = main() to > train_one_epoch() and goes on.

I do not understand why this is happening.

Also, this is an example from one instance of dataset,

(<PIL.Image.Image image mode=RGB size=1024x1024 at 0x7F46FC0A94A8>, {'boxes': tensor([[ 628.,    6.,  644.,   26.],
    [ 633.,   50.,  650.,   65.],
    [ 620.,   27.,  637.,   44.],
    [ 424.,  193.,  442.,  207.],
    [ 474.,  188.,  496.,  204.],
    [ 383.,  226.,  398.,  236.],
    [ 399.,  218.,  418.,  231.],
    [  42.,  189.,   63.,  203.],
    [ 106.,  159.,  129.,  169.],
    [ 273.,   17.,  287.,   34.],
    [ 225.,  961.,  234.,  980.],
    [ 220., 1004.,  230., 1024.]]), 'labels': tensor([5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5]), 'image_id': tensor([0]), 'area': tensor([320., 255., 289., 252., 352., 150., 247., 294., 230., 238., 171., 200.]), 'iscrowd': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])})


title: Difference between 'ctx' and 'self' in python?

body:
While using the deep learning library PyTorch I came across definitions like this. Does ctx have the same behavior as self?

class LinearFunction(Function):

    @staticmethod
    def forward(ctx, input, weight, bias=None):
        ctx.save_for_backward(input, weight, bias)
        output = input.mm(weight.t())
        if bias is not None:
            output += bias.unsqueeze(0).expand_as(output)
        return output

    @staticmethod
    def backward(ctx, grad_output):
        input, weight, bias = ctx.saved_variables
        grad_input = grad_weight = grad_bias = None
        if ctx.needs_input_grad[0]:
            grad_input = grad_output.mm(weight)
        if ctx.needs_input_grad[1]:
            grad_weight = grad_output.t().mm(input)
        if bias is not None and ctx.needs_input_grad[2]:
            grad_bias = grad_output.sum(0).squeeze(0)

        return grad_input, grad_weight, grad_bias


title: Pytorch equivalent of Numpy's logical_and and kin?

body:
Does Pytorch have an equivalent of Numpy's element-wise logical operators (logical_and, logical_or, logical_not, and logical_xor)? Calling the Numpy functions on Pytorch tensors seems to work well enough when using the CPU, even producing a Pytorch tensor as output. I mainly ask because I assume this would not work so well if the pytorch calculation were running in the GPU.

I've looked through Pytorch's documentation index at all functions containing the string "and" and none seem relevant.


title: "expected CPU tensor(got CUDA tensor)" error for PyTorch

body:
In my inference code with Trained PyTorch model, What's wrong ?

There is an runtime error message: "expected CPU tensor(got CUDA tensor)"

import torch
import torch.nn as nn
#from __future__ import print_function
import argparse
from PIL import Image
import torchvision.models as models
import skimage.io
from torch.autograd import Variable as V
from torch.nn import functional as f
from torchvision import transforms as trn

# define image transformation
centre_crop = trn.Compose([
trn.ToPILImage(),
trn.Scale(256),
trn.CenterCrop(224),
trn.ToTensor(),
trn.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

filename=r'ed91.png'
img = skimage.io.imread(filename)

x = V(centre_crop(img).unsqueeze(0), volatile=True)

model = models.__dict__['resnet18']()
model = torch.nn.DataParallel(model).cuda()

model = torch.load('mw_model0831.pth')
#model.load_state_dict(checkpoint['state_dict'])

#best_prec1 = checkpoint['best_prec1']
logit = model(x)

print(logit)
print(len(logit))
h_x = f.softmax(logit).data.squeeze()

How can I fix this ?


title: IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number

body:
def nms(bboxes,scores,threshold=0.5):
    '''
    bboxes(tensor) [N,4]
    scores(tensor) [N,]
    '''
    x1 = bboxes[:,0]
    y1 = bboxes[:,1]
    x2 = bboxes[:,2]
    y2 = bboxes[:,3]
    areas = (x2-x1) * (y2-y1)

    _,order = scores.sort(0,descending=True)
    keep = []
    while order.numel() > 0:
        i = order[0]
        keep.append(i)

        if order.numel() == 1:
            break

        xx1 = x1[order[1:]].clamp(min=x1[i])
        yy1 = y1[order[1:]].clamp(min=y1[i])
        xx2 = x2[order[1:]].clamp(max=x2[i])
        yy2 = y2[order[1:]].clamp(max=y2[i])

        w = (xx2-xx1).clamp(min=0)
        h = (yy2-yy1).clamp(min=0)
        inter = w*h

        ovr = inter / (areas[i] + areas[order[1:]] - inter)
        ids = (ovr<=threshold).nonzero().squeeze()
        if ids.numel() == 0:
            break
        order = order[ids+1]
    return torch.LongTensor(keep)

I tried

i=order.item()

But it does not work


title: Pytorch: Trying to apply the transform to a numpy array... fails with an error

body:
Any help will be much appreciated. The code in transforms.py says that the transformation should/would apply to PIL images as well as ndarrays.
Given the transforms:

data_transforms = {
    'train': transforms.Compose([
        transforms.Scale(256),
        transforms.Pad(4,0),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Scale(256),
        transforms.Pad(4,0),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

I wish to apply the transform on ndarrays that I obtained from some other code. Let's say it is x_data, whose shape is (1000,120,160,3) where the dimensions are (total rows, width, height, channels)

doing the following fails (All I'm trying to do is apply a transformation) :

foo = data_transforms['train']
bar = foo(x_data[0])

with the following message:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-93-a703e3b9c76d> in <module>()
----> 1 foo(x_data[1])

~/anaconda3/envs/pytorch/lib/python3.5/site-packages/torchvision-0.1.9-py3.5.egg/torchvision/transforms.py in __call__(self, img)
     32     def __call__(self, img):
     33         for t in self.transforms:
---> 34             img = t(img)
     35         return img
     36

~/anaconda3/envs/pytorch/lib/python3.5/site-packages/torchvision-0.1.9-py3.5.egg/torchvision/transforms.py in __call__(self, img)
    185         """
    186         if isinstance(self.size, int):
--> 187             w, h = img.size
    188             if (w <= h and w == self.size) or (h <= w and h == self.size):
    189                 return img

TypeError: 'int' object is not iterable


title: How can I add an element to a PyTorch tensor along a certain dimension?

body: I have a tensor inps, which has a size of [64, 161, 1] and I have some new data d which has a size of [64, 161]. How can I add d to inps such that the new size is [64, 161, 2]?


title: How to resize a PyTorch tensor?

body:
I have a PyTorch tensor of size (5, 1, 44, 44) (batch, channel, height, width), and I want to 'resize' it to (5, 1, 224, 224)
How can I do that? What functions should I use?


title: How to use torchvision.transforms for data augmentation of segmentation task in Pytorch?

body:
I am a little bit confused about the data augmentation performed in PyTorch.

Because we are dealing with segmentation tasks, we need data and mask for the same data augmentation, but some of them are random, such as random rotation.

Keras provides a random seed guarantee that data and mask do the same operation, as shown in the following code:

    data_gen_args = dict(featurewise_center=True,
                         featurewise_std_normalization=True,
                         rotation_range=25,
                         horizontal_flip=True,
                         vertical_flip=True)

    image_datagen = ImageDataGenerator(**data_gen_args)
    mask_datagen = ImageDataGenerator(**data_gen_args)

    seed = 1
    image_generator = image_datagen.flow(train_data, seed=seed, batch_size=1)
    mask_generator = mask_datagen.flow(train_label, seed=seed, batch_size=1)

    train_generator = zip(image_generator, mask_generator)

I didn't find a similar description in the official Pytorch documentation, so I don't know how to ensure that data and mask can be processed synchronously.

Pytorch does provide such a function, but I want to apply it to a custom Dataloader.

For example：

def __getitem__(self, index):
    img = np.zeros((self.im_ht, self.im_wd, channel_size))
    mask = np.zeros((self.im_ht, self.im_wd, channel_size))

    temp_img = np.load(Image_path + '{:0>4}'.format(self.patient_index[index]) + '.npy')
    temp_label = np.load(Label_path + '{:0>4}'.format(self.patient_index[index]) + '.npy')

    for i in range(channel_size):
        img[:,:,i] = temp_img[self.count[index] + i]
        mask[:,:,i] = temp_label[self.count[index] + i]

    if self.transforms:
        img = np.uint8(img)
        mask = np.uint8(mask)
        img = self.transforms(img)
        mask = self.transforms(mask)

    return img, mask

In this case, img and mask will be transformed separately, because some operations such as random rotation are random, so the correspondence between mask and image may be changed. In other words, the image may have rotated but the mask did not do this.

EDIT 1

I used the method in augmentations.py, but I got an error:：

Traceback (most recent call last):
  File "test_transform.py", line 87, in <module>
    for batch_idx, image, mask in enumerate(train_loader):
  File "/home/dirk/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 314, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/dirk/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 314, in <listcomp>
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/dirk/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataset.py", line 103, in __getitem__
    return self.dataset[self.indices[idx]]
  File "/home/dirk/home/data/dirk/segmentation_unet_pytorch/data.py", line 164, in __getitem__
    img, mask = self.transforms(img, mask)
  File "/home/dirk/home/data/dirk/segmentation_unet_pytorch/augmentations.py", line 17, in __call__
    img, mask = a(img, mask)
TypeError: __call__() takes 2 positional arguments but 3 were given

This is my code for __getitem__()：

data_transforms = {
    'train': Compose([
        RandomHorizontallyFlip(),
        RandomRotate(degree=25),
        transforms.ToTensor()
    ]),
}

train_set = DatasetUnetForTestTransform(fold=args.fold, random_index=args.random_index,transforms=data_transforms['train'])

# __getitem__ in class DatasetUnetForTestTransform
def __getitem__(self, index):
    img = np.zeros((self.im_ht, self.im_wd, channel_size))
    mask = np.zeros((self.im_ht, self.im_wd, channel_size))
    temp_img = np.load(Label_path + '{:0>4}'.format(self.patient_index[index]) + '.npy')
    temp_label = np.load(Label_path + '{:0>4}'.format(self.patient_index[index]) + '.npy')
    temp_img, temp_label = crop_data_label_from_0(temp_img, temp_label)
    for i in range(channel_size):
        img[:,:,i] = temp_img[self.count[index] + i]
        mask[:,:,i] = temp_label[self.count[index] + i]

    if self.transforms:
        img = T.ToPILImage()(np.uint8(img))
        mask = T.ToPILImage()(np.uint8(mask))
        img, mask = self.transforms(img, mask)

    img = T.ToTensor()(img).copy()
    mask = T.ToTensor()(mask).copy()
    return img, mask

EDIT 2

I found that after ToTensor, the dice between the same labels becomes 255 instead of 1, how to fix it?

# Dice computation
def DSC_computation(label, pred):
    pred_sum = pred.sum()
    label_sum = label.sum()
    inter_sum = np.logical_and(pred, label).sum()
    return 2 * float(inter_sum) / (pred_sum + label_sum)

Feel free to ask if more code is needed to explain the problem.


title: Multi dimensional inputs in pytorch Linear method?

body:
When building a simple perceptron neural network we usuall passes a 2D matrix of input of format (batch_size,features) to a 2D weight matrix, similar to this simple neural network in numpy. I always assumed a Perceptron/Dense/Linear layer of a neural network only accepts an input of 2D format and outputs another 2D output. But recently I came across this pytorch model in which a Linear layer accepts a 3D input tensor and output another 3D tensor (o1 = self.a1(x)).

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.a1 = nn.Linear(4,4)
        self.a2 = nn.Linear(4,4)
        self.a3 = nn.Linear(9,1)
    def forward(self,x):
        o1 = self.a1(x)
        o2 = self.a2(x).transpose(1,2)
        output = torch.bmm(o1,o2)
        output = output.view(len(x),9)
        output = self.a3(output)
        return output

x = torch.randn(10,3,4)
y = torch.ones(10,1)

net = Net()

criterion = nn.MSELoss()
optimizer = optim.Adam(net.parameters())

for i in range(10):
    net.zero_grad()
    output = net(x)
    loss = criterion(output,y)
    loss.backward()
    optimizer.step()
    print(loss.item())

These are the question I have,

Is the above neural network a valid one? that is whether the model will train correctly?
Even after passing a 3D input x = torch.randn(10,3,4), why is the pytorch nn.Linear doesn't shows any error and gives a 3D output?


title: RuntimeError: Attempting to deserialize object on CUDA device 2 but torch.cuda.device_count() is 1

body:
I've got a snippet of python code for training a model. The problem is that after running:

loaded_state = torch.load(model_path+seq_to_seq_test_model_fname)

to load a pretrained model,I'm getting:

  Traceback (most recent call last):
  File "img_to_text.py", line 480, in <module>
    main()
  File "img_to_text.py", line 475, in main
    r = setup_test()
  File "img_to_text.py", line 259, in setup_test
    s2s_data = s2s.setup_test()
  File "/media/ahrzb/datasets/notebooks/mzh/SemStyle/semstyle/code/seq2seq_pytorch.py", line 220, in setup_test
    loaded_state= torch.load(model_path+seq_to_seq_test_model_fname)
  File "/home/ahrzb/.pyenv/versions/2.7.15/envs/mzh2.7/lib/python2.7/site-packages/torch/serialization.py", line 358, in load
    return _load(f, map_location, pickle_module)
  File "/home/ahrzb/.pyenv/versions/2.7.15/envs/mzh2.7/lib/python2.7/site-packages/torch/serialization.py", line 542, in _load
    result = unpickler.load()
  File "/home/ahrzb/.pyenv/versions/2.7.15/envs/mzh2.7/lib/python2.7/site-packages/torch/serialization.py", line 508, in persistent_load
    data_type(size), location)
  File "/home/ahrzb/.pyenv/versions/2.7.15/envs/mzh2.7/lib/python2.7/site-packages/torch/serialization.py", line 372, in restore_location
    return default_restore_location(storage, location)
  File "/home/ahrzb/.pyenv/versions/2.7.15/envs/mzh2.7/lib/python2.7/site-packages/torch/serialization.py", line 104, in default_restore_location
    result = fn(storage, location)
  File "/home/ahrzb/.pyenv/versions/2.7.15/envs/mzh2.7/lib/python2.7/site-packages/torch/serialization.py", line 85, in _cuda_deserialize
    device, torch.cuda.device_count()))

I think this is because they have trained the model on two GPUs and I need to load it in one GPU. I changed this line:

loaded_state = torch.load(model_path+seq_to_seq_test_model_fname)

to

loaded_state = torch.load(model_path+seq_to_seq_test_model_fname, map_location={'cuda:1': 'cuda:0'} )

in order to map data of cuda 1 to cuda 0 but it did not work.


import asyncio
import torch
import os
import pandas as pd
from flair.data import Sentence
from flair.embeddings import FlairEmbeddings, DocumentPoolEmbeddings, WordEmbeddings

device = torch.device("cpu")
print(device)
# first, declare how you want to embed
embeddings = DocumentPoolEmbeddings(
    [WordEmbeddings('glove'), FlairEmbeddings('news-forward'), FlairEmbeddings('news-backward')])

path = os.getcwd()

df=pd.read_pickle(path+'/embedding_all_courses_2.pkl')

query_emd=[]
cos = torch.nn.CosineSimilarity(dim=0, eps=1e-6)

query= Sentence("some text")
embeddings.embed([query])
query_emd.append(query.embedding)

async def count(index,row):
    for i in query_emd:
        print(words,row['course_name'],cos(i, row['embedding']))

    print(index)

async def main():
    await asyncio.gather(*(count(index,row) for index,row in df.iterrows()))

if __name__ == "__main__":
    import time
    s = time.perf_counter()
    asyncio.run(main())

    elapsed = time.perf_counter() - s
    print(f"{__file__} executed in {elapsed:0.2f} seconds.")

Trying to run pytorch cosine similarity in asyncio package to get the parallel result. Using a flair model for embedding text. Need to compare a text with a huge dataframe and get the most similar text as a result and response should be pretty fast. Can you please suggest an alternative way also? I also need to run this code on the CPU memory only not on the GPU Cuda system
Error:
raceback (most recent call last):
  File "asyn_emd.py", line 74, in <module>
    asyncio.run(main())
  File "/home/linuxbrew/.linuxbrew/Cellar/python/3.7.5/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/home/linuxbrew/.linuxbrew/Cellar/python/3.7.5/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/asyn_emd.py", line 68, in main
    await asyncio.gather(*(count(index,row) for index,row in df.iterrows()))
  File "/asyn_emd.py", line 61, in count
    print(words,row['course_name'],cos(i, row['embedding']))
  File "/home/karthickaravindan/.virtualenvs/test/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/karthickaravindan/.virtualenvs/test/lib/python3.7/site-packages/torch/nn/modules/distance.py", line 75, in forward
    return F.cosine_similarity(x1, x2, self.dim, self.eps)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!


title: Pytorch ImageNet dataset

body:
I am unable to download the original ImageNet dataset from their official website. However, I found out that pytorch has ImageNet as one of it’s torch vision datasets.

Q1. Is that the original ImageNet dataset?

Q2. How do I get the classes for the dataset like it’s being done in Cifar-10

classes = [‘airplane’, ‘automobile’, ‘bird’, ‘cat’, ‘deer’, ‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’]


I am trying to import torch in my windows machine using python 3.5. (CPU only)(pip)
I have followed the steps given in the official website.

When I try to import torch it gives me the error:

from torch._C import *
ImportError: DLL load failed: The specified module could not be found.

I have looked up from torch._C import * (ImportError: DLL load failed: The specified module could not be found. and ModuleNotFoundError: No module named 'torch._C' and followed their instructions, still the error persists.

The total traceback:

Traceback (most recent call last):
File "D:\CFPS.py", line 1, in <module>
import torch
File "C:\Program Files (x86)\Python35\lib\site-packages\torch\__init__.py",
line 78, in <module>
from torch._C import *
ImportError: DLL load failed: The specified module could not be found.
[Finished in 0.6s]

Is there a way to get past through this?


title: How to load a checkpoint file in a pytorch model?

body:
In my pytorch model, I'm initializing my model and optimizer like this.

model = MyModelClass(config, shape, x_tr_mean, x_tr,std)
optimizer = optim.SGD(model.parameters(), lr=config.learning_rate)

And here is the path to my checkpoint file.

checkpoint_file = os.path.join(config.save_dir, "checkpoint.pth")

To load this checkpoint file, I check and see if the checkpoint file exists and then I load it as well as the model and optimizer.

if os.path.exists(checkpoint_file):
    if config.resume:
        torch.load(checkpoint_file)
        model.load_state_dict(torch.load(checkpoint_file))
        optimizer.load_state_dict(torch.load(checkpoint_file))

Also, here's how I'm saving my model and optimizer.

 torch.save({'model': model.state_dict(), 'optimizer': optimizer.state_dict(), 'iter_idx': iter_idx, 'best_va_acc': best_va_acc}, checkpoint_file)

For some reason I keep getting a strange error whenever I run this code.

model.load_state_dict(torch.load(checkpoint_file))
File "/home/Josh/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 769, in load_state_dict
self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for MyModelClass:
        Missing key(s) in state_dict: "mean", "std", "attribute.weight", "attribute.bias".
        Unexpected key(s) in state_dict: "model", "optimizer", "iter_idx", "best_va_acc"

Does anyone know why I'm getting this error?


title: How to reset Colab after the following CUDA error 'Cuda assert fails: device-side assert triggered'?

body:
I'm running my Jupyter Notebook using Pytorch on Google Colab. After I received the 'Cuda assert fails: device-side assert triggered' I am unable to run any other code that uses my pytorch module. Does anyone know how to reset my code so that my Pytorch functions that were working before can still run?

I've already tried implementing CUDA_LAUNCH_BLOCKING=1but my code still doesn't work as the Assert is still triggered!


title: Taking the last state from BiLSTM (BiGRU) in PyTorch

body:
After reading several articles, I am still quite confused about correctness of my implementation of getting last hidden states from BiLSTM.

Understanding Bidirectional RNN in PyTorch (TowardsDataScience)
PackedSequence for seq2seq model (PyTorch forums)
What's the difference between “hidden” and “output” in PyTorch LSTM? (StackOverflow)
 Select tensor in a batch of sequences (Pytorch formums)

The approach from the last source (4) seems to be the cleanest for me, but I am still uncertain if I understood the thread correctly. Am I using the right final hidden states from LSTM and reversed LSTM? This is my implementation

# pos contains indices of words in embedding matrix
# seqlengths contains info about sequence lengths
# so for instance, if batch_size is 2 and pos=[4,6,9,3,1] and
# seqlengths contains [3,2], we have batch with samples
# of variable length [4,6,9] and [3,1]

all_in_embs = self.in_embeddings(pos)
in_emb_seqs = pack_sequence(torch.split(all_in_embs, seqlengths, dim=0))
output,lasthidden = self.rnn(in_emb_seqs)
if not self.data_processor.use_gru:
    lasthidden = lasthidden[0]
# u_emb_batch has shape batch_size x embedding_dimension
# sum last state from forward and backward  direction
u_emb_batch = lasthidden[-1,:,:] + lasthidden[-2,:,:]

Is it correct?


I have executed the following code and getting the error shown at extreme bottom. I would like to know how to resolve this. thanks

import torch.nn as nn
import torch.nn.functional as F
from torch import optim

from torchvision import transforms
_tasks = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

from torchvision.datasets import MNIST
mnist = MNIST("data", download=True, train=True, transform=_tasks)

from torch.utils.data import DataLoader
from torch.utils.data.sampler import SubsetRandomSampler

create training and validation split
split = int(0.8 * len(mnist))

index_list = list(range(len(mnist)))
train_idx, valid_idx = index_list[:split], index_list[split:]

create sampler objects using SubsetRandomSampler
tr_sampler = SubsetRandomSampler(train_idx)
val_sampler = SubsetRandomSampler(valid_idx)

create iterator objects for train and valid datasets
trainloader = DataLoader(mnist, batch_size=256, sampler=tr_sampler)
validloader = DataLoader(mnist, batch_size=256, sampler=val_sampler)

Creating model for execution

class Model(nn.Module):
  def init(self):
    super().init()
    self.hidden = nn.Linear(784, 128)
    self.output = nn.Linear(128, 10)

  def forward(self, x):
    x = self.hidden(x)
    x = F.sigmoid(x)
    x = self.output(x)
    return x

model = Model()

loss_function = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.01, weight_decay= 1e-6, momentum = 0.9, nesterov = True)

for epoch in range(1, 11): ## run the model for 10 epochs
  train_loss, valid_loss = [], []

  #training part
  model.train()
  for data, target in trainloader:
    optimizer.zero_grad()

    #1. forward propagation
    output = model(data)

    #2. loss calculation
    loss = loss_function(output, target)

    #3. backward propagation
    loss.backward()

    #4. weight optimization
    optimizer.step()

    train_loss.append(loss.item())

  # evaluation part
  model.eval()
  for data, target in validloader:
     output = model(data)
     loss = loss_function(output, target)
     valid_loss.append(loss.item())

Executing this I am getting the following error :

  RuntimeError Traceback (most recent call last) in ()
  ----> 1 output = model(data) 2 3 ## 2. loss calculation 4 loss = loss_function(output, target) 5

  /usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in
  call(self, *input, **kwargs) 487 result = self._slow_forward(*input,
  **kwargs)

  /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in
  linear(input, weight, bias) 1352 ret =
  torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t()) 1353
  else:
  -> 1354 output = input.matmul(weight.t()) 1355 if bias is not None: 1356 output += torch.jit._unwrap_optional(bias)

  RuntimeError: size mismatch, m1: [3584 x 28], m2: [784 x 128] at
  /pytorch/aten/src/TH/generic/THTensorMath.cpp:940


title: Optimizer and scheduler for BERT fine-tuning

body:
I'm trying to fine-tune a model with BERT (using transformers library), and I'm a bit unsure about the optimizer and scheduler.

First, I understand that I should use transformers.AdamW instead of Pytorch's version of it. Also, we should use a warmup scheduler as suggested in the paper, so the scheduler is created using get_linear_scheduler_with_warmup function from transformers package.

The main questions I have are:

get_linear_scheduler_with_warmup should be called with the warm up. Is it ok to use 2 for warmup out of 10 epochs?
When should I call scheduler.step()? If I do after train, the learning rate is zero for the first epoch. Should I call it for each batch?

Am I doing something wrong with this?

from transformers import AdamW
from transformers.optimization import get_linear_scheduler_with_warmup

N_EPOCHS = 10

model = BertGRUModel(finetune_bert=True,...)
num_training_steps = N_EPOCHS+1
num_warmup_steps = 2
warmup_proportion = float(num_warmup_steps) / float(num_training_steps)  # 0.1

optimizer = AdamW(model.parameters())
criterion = nn.BCEWithLogitsLoss(pos_weight=torch.Tensor([class_weights[1]]))

scheduler = get_linear_schedule_with_warmup(
    optimizer, num_warmup_steps=num_warmup_steps,
    num_training_steps=num_training_steps
)

for epoch in range(N_EPOCHS):
    scheduler.step() #If I do after train, LR = 0 for the first epoch
    print(optimizer.param_groups[0]["lr"])

    train(...) # here we call optimizer.step()
    evaluate(...)

My model and train routine(quite similar to this notebook)

class BERTGRUSentiment(nn.Module):
    def __init__(self,
                 bert,
                 hidden_dim,
                 output_dim,
                 n_layers=1,
                 bidirectional=False,
                 finetune_bert=False,
                 dropout=0.2):

        super().__init__()

        self.bert = bert

        embedding_dim = bert.config.to_dict()['hidden_size']

        self.finetune_bert = finetune_bert

        self.rnn = nn.GRU(embedding_dim,
                          hidden_dim,
                          num_layers = n_layers,
                          bidirectional = bidirectional,
                          batch_first = True,
                          dropout = 0 if n_layers < 2 else dropout)

        self.out = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim, output_dim)
        self.dropout = nn.Dropout(dropout)

    def forward(self, text):
        #text = [batch size, sent len]

        if not self.finetune_bert:
            with torch.no_grad():
                embedded = self.bert(text)[0]
        else:
            embedded = self.bert(text)[0]
        #embedded = [batch size, sent len, emb dim]
        _, hidden = self.rnn(embedded)

        #hidden = [n layers * n directions, batch size, emb dim]

        if self.rnn.bidirectional:
            hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1))
        else:
            hidden = self.dropout(hidden[-1,:,:])

        #hidden = [batch size, hid dim]

        output = self.out(hidden)

        #output = [batch size, out dim]

        return output

import torch
from sklearn.metrics import accuracy_score, f1_score

def train(model, iterator, optimizer, criterion, max_grad_norm=None):
    """
    Trains the model for one full epoch
    """
    epoch_loss = 0
    epoch_acc = 0

    model.train()

    for i, batch in enumerate(iterator):
        optimizer.zero_grad()
        text, lens = batch.text

        predictions = model(text)

        target = batch.target

        loss = criterion(predictions.squeeze(1), target)

        prob_predictions = torch.sigmoid(predictions)

        preds = torch.round(prob_predictions).detach().cpu()
        acc = accuracy_score(preds, target.cpu())

        loss.backward()
        # Gradient clipping
        if max_grad_norm:
            torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)

        optimizer.step()

        epoch_loss += loss.item()
        epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)


title: How to implement dropout in Pytorch, and where to apply it

body:
I am quite unsure whether this is correct. It is really sad I can't find many good examples on how to parametrize a NN.

What do you think of this way of dropping out in those two classes. First I'm writing the original class :

class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.fc3(out)
      return out

and then here, I found two different ways to write things, which I don't know how to distinguish. The first one uses :

self.drop_layer = nn.Dropout(p=p)

whereas the second :

self.dropout = nn.Dropout(p)

and here is my result :

class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)
      self.drop_layer = nn.Dropout(p=p)

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.fc3(out)
      return out

 class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)
      self.dropout = nn.Dropout(p)

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.fc3(out)
      return out

could this work, if not how to improve that, and is it giving me the result I m expecting, meaning creating a neural network where I can dropout some neurons. Important detail, I only want to dropout the second layer of neural network, no touch to the rest!


title: How to solve ' CUDA out of memory. Tried to allocate xxx MiB' in pytorch?

body:
I am trying to train a CNN in pytorch,but I meet some problems.
The RuntimeError:

  RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0;
  2.00 GiB total capacity; 584.97 MiB already allocated; 13.81 MiB free; 590.00 MiB reserved in total by PyTorch)

This is my code:

import os
import numpy as np
import cv2
import torch as t
import torch.nn as nn
import torchvision.transforms as transforms
from torch.utils.data import DataLoader,Dataset
import time
import matplotlib.pyplot as plt
%matplotlib inline
root_path='C:/Users/60960/Desktop/recet-task/course_LeeML20/course_LeeML20-datasets/hw3/food-11'
training_path=root_path+'/training'
testing_path=root_path+'/testing'
validation_path=root_path+'/validation'
def readfile(path,has_label):
    img_paths=sorted(os.listdir(path))
    x=np.zeros((len(img_paths),128,128,3),dtype=np.uint8)
    y=np.zeros((len(img_paths)),dtype=np.uint8)
    for i,file in enumerate(img_paths):
        img=cv2.imread(path+'/'+file)
        x[i,:,:]=cv2.resize(img,(128,128))
        if has_label:
            y[i]=int(file.split('_')[0])
    if has_label:
        return x,y
    else:
        return x
def show_img(img_from_cv2):
    b,g,r=cv2.split(img_from_cv2)
    img=cv2.merge([r,g,b])
    plt.imshow(img)
    plt.show()
x_train,y_train=readfile(training_path,True)
x_val,y_val=readfile(validation_path,True)
x_test=readfile(testing_path,False)
train_transform=transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(15),
    transforms.ToTensor()
])
test_transform=transforms.Compose([
    transforms.ToPILImage(),
    transforms.ToTensor()
])
class ImgDataset(Dataset):
    def __init__(self,x,y=None,transform=None):
        self.x=x
        self.y=y
        if y is not None:
            self.y=t.LongTensor(y)
        self.transform=transform
    def __len__(self):
        return len(self.x)
    def __getitem__(self,idx):
        X=self.x[idx]
        if self.transform is not None:
            X=self.transform(X)
        if self.y is not None:
            Y=self.y[idx]
            return X,Y
        return X
batch_size=128
train_set=ImgDataset(x_train,y_train,transform=train_transform)
val_set=ImgDataset(x_val,y_val,transform=test_transform)
train_loader=DataLoader(train_set,batch_size=batch_size,shuffle=True)
val_loader=DataLoader(val_set,batch_size=batch_size,shuffle=False)
class Classifier(nn.Module):
    def __init__(self):
        super(Classifier,self).__init__()
        self.cnn=nn.Sequential(
            nn.Conv2d(3,64,3,1,1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(2,2,0),

            nn.Conv2d(64,128,3,1,1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.MaxPool2d(2,2,0),

            nn.Conv2d(128,256,3,1,1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.MaxPool2d(2,2,0),

            nn.Conv2d(256,512,3,1,1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.MaxPool2d(2,2,0),

            nn.Conv2d(512,512,3,1,1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.MaxPool2d(2,2,0)
        )
        self.fc=nn.Sequential(
            nn.Linear(512*4*4,1024),
            nn.ReLU(),
            nn.Linear(1024,512),
            nn.ReLU(),
            nn.Linear(512,11)
        )
    def forward(self,x):
        out=self.cnn(x)
        out=out.view(out.size()[0],-1)
        return self.fc(out)
model=Classifier().cuda()
loss_fn=nn.CrossEntropyLoss()
optim=t.optim.Adam(model.parameters(),lr=0.001)
epochs=30
for epoch in range(epochs):
    epoch_start_time=time.time()
    train_acc=0.0
    train_loss=0.0
    val_acc=0.0
    val_loss=0.0
    model.train()
    for i,data in enumerate(train_loader):
        optim.zero_grad()
        train_pred=model(data[0].cuda())
        batch_loss=loss_fn(train_pred,data[1].cuda())
        batch_loss.backward()
        optim.step()
        train_acc+=np.sum(np.argmax(train_pred.cpu().data.numpy(),axis=1)==data[1].numpy())
        train_loss+=batch_loss.item()
    model.eval()
    with t.no_grad():
        for i,data in enumerate(val_loader):
            val_pred=model(data[0].cuda())
            batch_loss=loss_fn(val_pred,data[1].cuda())
            val_acc+=np.sum(np.argmax(val_pred.cpu().data.numpy(),axis=1)==data[1].numpy())
            val_loss+=batch_loss.item()
        print('[%03d/%03d] %2.2f sec(s) Train Acc: %3.6f Loss: %3.6f | Val Acc: %3.6f loss: %3.6f' % (epoch + 1, epochs, time.time()-epoch_start_time,train_acc/train_set.__len__(), train_loss/train_set.__len__(), val_acc/val_set.__len__(), val_loss/val_set.__len__()))
x_train_val=np.concatenate((x_train,x_val),axis=0)
y_train_val=np.concatenate((y_train,y_val),axis=0)
train_val_set=ImgDataset(x_train_val,x_train_val,train_transform)
train_val_loader=DataLoader(train_val_set,batch_size=batch_size,shuffle=True)
model_final=Classifier().cuda()
loss_fn=nn.CrossEntropy()
optim=t.optim.Adam(model_final.parameters(),lr=0.001)
epochs=30
for epoch in range(epochs):
    epoch_start_time=time.time()
    train_acc=0.0
    train_loss=0.0
    model_final.train()
    for i,data in enumerate(train_val_loader):
        optim.zero_grad()
        train_pred=model_final(data[0].cuda())
        batch_loss=loss_fn(train_pred,data[1].cuda())
        batch_loss.backward()
        optim.step()
        train_acc+=np.sum(np.argmax(train_pred.cpu().data.numpy(),axis=1)==data[1].numpy())
        train_loss+=batch_loss.item()
    print('[%03d/%03d] %2.2f sec(s) Train Acc: %3.6f Loss: %3.6f' % (epoch + 1, epochs, time.time()-epoch_start_time,train_acc/train_val_set.__len__(), train_loss/train_val_set.__len__()))
test_set=ImgDataset(x_test,transform=test_transform)
test_loader=DataLoader(test_set,batch_size=batch_size,shuffle=False)
model_final.eval()
prediction=[]
with t.no_grad():
    for i,data in enumerate(test_loader):
        test_pred=model_final(data.cuda())
        test_label=np.argmax(test_pred.cpu().data.numpy(),axis=1)
        for y in test_label:
            prediction.append(y)
with open('predict.csv','w') as f:
    f.write('Id,Category\n')
    for i,y in enumerate(prediction):
        f.write('{},{}\n,'.format(i,y))

Pytorch version is 1.4.0, opencv2 version is 4.2.0.
The training dataset are pictures like these:training set

The error happens at this line:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-770be67177f4> in <module>
    119     for i,data in enumerate(train_loader):
    120         optim.zero_grad()
--> 121         train_pred=model(data[0].cuda())
    122         batch_loss=loss_fn(train_pred,data[1].cuda())
    123         batch_loss.backward()

I have already installed:
some information.
GPU utilization is low,close to zero:
GPU utilization.
Error message says:

  RuntimeError: CUDA out of memory.  Tried to allocate 512.00 MiB.

So I want to know how to allocate more memory.
What's more, I have tried to reduce the batch size to 1, but this doesn't work.
HELP!!!


title: Variable size input for LSTM in Pytorch

body:
I am using features of variable length videos to train one layer LSTM. Video sizes are changing from 10 to 35 frames. I am using batch size of 1.
I have the following code:

lstm_model = LSTMModel(4096, 4096, 1, 64)
for step, (video_features, label) in enumerate(data_loader):
    bx = Variable(score.view(-1, len(video_features), len(video_features[0]))) #examples = 1x12x4096, 1x5x4096
    output = lstm_model(bx)

Lstm model is;

class LSTMModel(nn.Module):
def __init__(self, input_size, hidden_size, num_layers, num_classes):
    super(LSTMModel, self).__init__()
    self.l1 = nn.LSTM(input_size=input_size, hidden_size=hidden_size, num_layers=num_layers, batch_first=True)
    self.out = nn.Linear(hidden_size, num_classes)
def forward(self, x):
    r_out, (h_n, h_c) = self.l1(x, None) #None represents zero initial hidden state
    out = self.out(r_out[:, -1, :])
    return out

I just want to ask; am I doing the right for training LSTM with variable size input. The code works okay and loss decreases but I am not sure if I am doing the right thing. Because I haven't used LSTMs in Pytorch before.


title: How to convert torch int64 to torch LongTensor?

body:
I am going through a course which uses a deprecated version of PyTorch which does not change torch.int64 to torch.LongTensor as needed. The current section of code which is throwing the error is:

loss = loss_fn(Ypred, Ytrain_) # calc loss on the prediction

I believe the dtype should be changed in thhis section though:

Ytrain_ = torch.from_numpy(y_train.values).view(1, -1)[0].

When testing the data-type by using Ytrain_.dtype it returns torch.int64. I have tried to convert it by applying the long() function as such: Ytrain_ = Ytrain_.long() to no avail.

I have also tried looking for it in the documentation but it seems that it says torch.int64 OR torch.long which I assume means torch.int64 should work.

RuntimeError                              Traceback (most recent call last)
----> 9     loss = loss_fn(Ypred, Ytrain_) # calc loss on the prediction
RuntimeError: Expected object of scalar type Long but got scalar type Int for argument #2 'target'


title: PyTorch - Torchvision - BrokenPipeError: [Errno 32] Broken pipe

body:
I'm trying to carry out the tutorial named "Training a classifier" with PyTorch.
WHen trying to debug this part of the code :

import matplotlib.pyplot as plt
import numpy as np

# functions to show an image

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))

# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()

# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))

I get this error message :

Files already downloaded and verified Files already downloaded and verified
Files already downloaded and verified Files already downloaded and verified Traceback (most recent call last):
File "<string>", line 1, in <module>
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 277, in
_fixup_main_from_path
    run_name="__mp_main__")
File "D:\Anaconda\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
File "D:\Anaconda\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
File "D:\Anaconda\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
File "d:\Yggdrasil\Programmation\PyTorch\TutorialCIFAR10.py", line 36, in <module>
    dataiter = iter(trainloader)
File "D:\Anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
    return _DataLoaderIter(self)
File "D:\Anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
    w.start()
File "D:\Anaconda\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
File "D:\Anaconda\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
File "D:\Anaconda\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
File "D:\Anaconda\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 136, in
_check_not_importing_main
    is not going to be frozen to produce an executable.)
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
 Traceback (most recent call last):
 File "d:\Yggdrasil\Programmation\PyTorch\TutorialCIFAR10.py", line 36, in <module>
    dataiter = iter(trainloader)
 File "D:\Anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
    return _DataLoaderIter(self)
 File "D:\Anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
    w.start()
 File "D:\Anaconda\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
 File "D:\Anaconda\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)   File "D:\Anaconda\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
 File "D:\Anaconda\lib\multiprocessing\popen_spawn_win32.py", line 65, in
__init__
    reduction.dump(process_obj, to_child)
 File "D:\Anaconda\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
 BrokenPipeError: [Errno 32] Broken pipe

All the previous lines in the tutorial are working perfectly.

Does someone know how to solve this, please ?
Thanks a lot in advance


title: What does log_prob do?

body:
In some (e.g. machine learning) libraries, we can find log_prob function. What does it do and how is it different from taking just regular log?

For example, what is the purpose of this code:

dist = Normal(mean, std)
sample = dist.sample()
logprob = dist.log_prob(sample)

And subsequently, why would we first take a log and then exponentiate the resulting value instead of just evaluating it directly:

prob = torch.exp(dist.log_prob(sample))


title: Pytorch DataLoader fails when the number of examples are not exactly divided by the batch size

body:
I have coded a custom data loader class in the pytorch. But it fails when iterating through all the number of batches inside an epoch. For example, think I have 100 data examples and my batch size is 9. It will fail in the 10th iteration saying batch size is different which will give a batch size 1 instead of 10. I have put my custom data loader in below.  Also I have put how I extract the data from the loader inside the for-loop.

class FlatDirectoryAudioDataset(tdata.Dataset): #customized dataloader

    def __init__(self, data_dir, transform=None):
        self.data_dir = data_dir
        self.transform = transform
        self.files = self.__setup_files()

    def __len__(self):
        """
        compute the length of the dataset
        :return: len => length of dataset
        """
        return len(self.files)

    def __setup_files(self):

        file_names = os.listdir(self.data_dir)
        files = []  # initialize to empty list

        for file_name in file_names:

            possible_file = os.path.join(self.data_dir, file_name)
            if os.path.isfile(possible_file) and (file_name.lower().endswith('.wav') or file_name.lower().endswith('.mp3')): #&& (possible_file.lower().endswith('.wav') or possible_file.lower().endswith('.mp3')):
                files.append(possible_file)

        # return the files list
        return files

    def __getitem__ (self,index):
        sample, _ = librosa.load(self.files[index], 16000)

        if self.transform:
            sample=self.transform(sample)

        sample = torch.from_numpy(sample)
        return sample

from torch.utils.data import DataLoader

    my_dataset=FlatDirectoryAudioDataset(source_directory,source_folder,source_label,transform = None,label=True)

dataloader_my = DataLoader(
        my_dataset,
        batch_size=batch_size,
        num_workers=0,
        shuffle=True)

for (i,batch) in enumerate(dataloader_my,0):
       print(i)
       if batch.shape[0]!=16:
          print(batch.shape)
          assert batch.shape[0]==16,"Something wrong with the batch size"


title: How do I install Pytorch 1.3.1 with CUDA enabled

body:
I have a conda environment on my Ubuntu 16.04 system.

When I install Pytorch using:

conda install pytorch

and I try and run the script I need, I get the error message:

raise AssertionError("Torch not compiled with CUDA enabled")

From looking at forums, I see that this is because I have installed Pytorch without CUDA support.

I then tried:

conda install -c pytorch torchvision cudatoolkit=10.1 pytorch

but now I get the error:

    from torch.utils.cpp_extension import BuildExtension, CUDAExtension
  File "/home/username/miniconda3/envs/super_resolution/lib/python3.6/site-packages/torch/__init__.py", line 81, in <module>
    from torch._C import *
ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found

So it seems that these two installs are installing different versions of Pytorch(?). The first one that seemed to work was Pytorch 1.3.1.

My question: How do I install Pytorch with CUDA enabled, but ensure it is version 1.3.1 so that it works with my system?


title: PyTorch - RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'weight'

body:
I load my previously trained model and want to classify a single (test) image from the disk through this model. All the operations in my model are carried out on my GPU. Hence, I move the numpy array of the test image to GPU by calling cuda() function. When I call the forward() function of my model with the numpy array of the test image, I get the RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'weight'.

Here is the code I use to load the image from disk and call the forward() function:

test_img = imageio.imread('C:\\Users\\talha\\Desktop\\dog.png')
test_img = imresize(test_img, (28, 28))
test_tensor = torch.from_numpy(test_img)
test_tensor = test_tensor.cuda()
test_tensor = test_tensor.type(torch.FloatTensor)
log_results = model.forward(test_tensor)

Software Environment:

torch: 1.0.1

GPU: Nvidia GeForce GTX 1070

OS: Windows 10 64-bit

Python: 3.7.1


title: Different methods for initializing embedding layer weights in Pytorch

body:
There seem to be two ways of initializing embedding layers in Pytorch 1.0 using an uniform distribution.

For example you have an embedding layer:

self.in_embed = nn.Embedding(n_vocab, n_embed)

And you want to initialize its weights with an uniform distribution. The first way you can get this done is:

self.in_embed.weight.data.uniform_(-1, 1)

And another one would be:

nn.init.uniform_(self.in_embed.weight, -1.0, 1.0)

My question is: what is the difference between the first and second initialization form. Do both methods do the same thing?


I try to use pytorch module by conda but I get an error

Traceback (most recent call last):
File "train.py", line 8, in <module>
import torch
ModuleNotFoundError: No module named 'torch'

when I write conda list | findstr torch I see that torch is installed:

What is the problem?

I tried:

conda update conda -n root
conda install mkl=2018

but get:

Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

Could Not Find C:\WINDOWS\TEMP\conda-23721-26470.tmp
Could Not Find C:\WINDOWS\TEMP\tmpry_dlvar.bat

and the same error occurrence


title: Use Pytorch SSIM loss function in my model

body:
I am trying out this SSIM loss implement by this repo for image restoration.

For the reference of original sample code on author's GitHub, I tried:

model.train()
for epo in range(epoch):
    for i, data in enumerate(trainloader, 0):
        inputs = data
        inputs = Variable(inputs)
        optimizer.zero_grad()
        inputs = inputs.view(bs, 1, 128, 128)
        top = model.upward(inputs)
        outputs = model.downward(top, shortcut = True)
        outputs = outputs.view(bs, 1, 128, 128)

        if i % 20 == 0:
            out = outputs[0].view(128, 128).detach().numpy() * 255
            cv2.imwrite("/home/tk/Documents/recover/SSIM/" + str(epo) + "_" + str(i) + "_re.png", out)

        loss = - criterion(inputs, outputs)
        ssim_value = - loss.data.item()
        print (ssim_value)
        loss.backward()
        optimizer.step()

However, the results didn't come out as I expected. After first 10 epochs, the printed outcome image were all black.

loss = - criterion(inputs, outputs) is proposed by the author, however, for classical Pytorch training code this will be loss = criterion(y_pred, target), therefore should be loss = criterion(inputs, outputs) here.

However, I tried loss = criterion(inputs, outputs) but the results are still the same.

Can anyone share some thoughts about how to properly utilize SSIM loss? Thanks.


title: Implementing dropout from scratch

body:
This code attempts to utilize a custom implementation of dropout :

%reset -f

import torch
import torch.nn as nn
# import torchvision
# import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torch.utils.data as data_utils
import numpy as np
import matplotlib.pyplot as plt
import torch.nn.functional as F

num_epochs = 1000

number_samples = 10

from sklearn.datasets import make_moons
from matplotlib import pyplot
from pandas import DataFrame
# generate 2d classification dataset
X, y = make_moons(n_samples=number_samples, noise=0.1)
# scatter plot, dots colored by class value

x_data = [a for a in enumerate(X)]
x_data_train = x_data[:int(len(x_data) * .5)]
x_data_train = [i[1] for i in x_data_train]
x_data_train

y_data = [y[i[0]] for i in x_data]
y_data_train = y_data[:int(len(y_data) * .5)]
y_data_train

x_test = [a[1] for a in x_data[::-1][:int(len(x_data) * .5)]]
y_test = [a for a in y_data[::-1][:int(len(y_data) * .5)]]

x = torch.tensor(x_data_train).float() # <2>
print(x)

y = torch.tensor(y_data_train).long()
print(y)

x_test = torch.tensor(x_test).float()
print(x_test)

y_test = torch.tensor(y_test).long()
print(y_test)

class Dropout(nn.Module):
    def __init__(self, p=0.5, inplace=False):
#         print(p)
        super(Dropout, self).__init__()
        if p < 0 or p > 1:
            raise ValueError("dropout probability has to be between 0 and 1, "
                             "but got {}".format(p))
        self.p = p
        self.inplace = inplace

    def forward(self, input):
        print(list(input.shape))
        return np.random.binomial([np.ones((len(input),np.array(list(input.shape))))],1-dropout_percent)[0] * (1.0/(1-self.p))

    def __repr__(self):
        inplace_str = ', inplace' if self.inplace else ''
        return self.__class__.__name__ + '(' \
            + 'p=' + str(self.p) \
            + inplace_str + ')'

class MyLinear(nn.Linear):
    def __init__(self, in_feats, out_feats, drop_p, bias=True):
        super(MyLinear, self).__init__(in_feats, out_feats, bias=bias)
        self.custom_dropout = Dropout(p=drop_p)

    def forward(self, input):
        dropout_value = self.custom_dropout(self.weight)
        return F.linear(input, dropout_value, self.bias)

my_train = data_utils.TensorDataset(x, y)
train_loader = data_utils.DataLoader(my_train, batch_size=2, shuffle=True)

my_test = data_utils.TensorDataset(x_test, y_test)
test_loader = data_utils.DataLoader(my_train, batch_size=2, shuffle=True)

# Device configuration
device = 'cpu'
print(device)

# Hyper-parameters
input_size = 2
hidden_size = 100
num_classes = 2

learning_rate = 0.0001

pred = []

# Fully connected neural network with one hidden layer
class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes, p):
        super(NeuralNet, self).__init__()
#         self.drop_layer = nn.Dropout(p=p)
#         self.drop_layer = MyLinear()
#         self.fc1 = MyLinear(input_size, hidden_size, p)
        self.fc1 = MyLinear(input_size, hidden_size , p)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
#         out = self.drop_layer(x)
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

model = NeuralNet(input_size, hidden_size, num_classes, p=0.9).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Move tensors to the configured device
        images = images.reshape(-1, 2).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    if (epoch) % 100 == 0:
        print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))

Custom dropout is implemented as :

class Dropout(nn.Module):
    def __init__(self, p=0.5, inplace=False):
#         print(p)
        super(Dropout, self).__init__()
        if p < 0 or p > 1:
            raise ValueError("dropout probability has to be between 0 and 1, "
                             "but got {}".format(p))
        self.p = p
        self.inplace = inplace

    def forward(self, input):
        print(list(input.shape))
        return np.random.binomial([np.ones((len(input),np.array(list(input.shape))))],1-dropout_percent)[0] * (1.0/(1-self.p))

    def __repr__(self):
        inplace_str = ', inplace' if self.inplace else ''
        return self.__class__.__name__ + '(' \
            + 'p=' + str(self.p) \
            + inplace_str + ')'

class MyLinear(nn.Linear):
    def __init__(self, in_feats, out_feats, drop_p, bias=True):
        super(MyLinear, self).__init__(in_feats, out_feats, bias=bias)
        self.custom_dropout = Dropout(p=drop_p)

    def forward(self, input):
        dropout_value = self.custom_dropout(self.weight)
        return F.linear(input, dropout_value, self.bias)

It seems I've implemented the dropout function incorrectly ? :

np.random.binomial([np.ones((len(input),np.array(list(input.shape))))],1-dropout_percent)[0] * (1.0/(1-self.p))

How to modify in order to correctly utilize dropout ?

These posts were useful in getting to this point :

Hinton's Dropout in 3 Lines of Python :
https://iamtrask.github.io/2015/07/28/dropout/

Making a Custom Dropout Function : https://discuss.pytorch.org/t/making-a-custom-dropout-function/14053/2


title: How do you change the dimension of your input pictures in pytorch?

body:
i made a convolutional nuralnetwork and i want it to take input pictures and output pictures but when i turn the pictures into tensors they have the wrong dimension :

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [20, 3, 5, 5], but got 3-dimensional input of size [900, 1440, 3] instead

how do i change the dimension of the pictures ? and why does it need to be changed?
and how do i make the output an picture?
i tryed to use

transform = transforms.Compose(
[transforms.ToTensor(),
 transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

to normilize the img but it didnt change the dimension .
here is my nuralnet

    def __init__(self):
    super(Net, self).__init__()
    torch.nn.Module.dump_patches = True
    self.conv1 = nn.Conv2d(3, 20, 5)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(20, 16, 5)
    self.fc1 = nn.Linear(16*5*5, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 16*5*5)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.view(-1, 16 * 5 )
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)

    return x

here i get the image and put it into a list:

for i in range(4):
l.append(ImageGrab.grab())

and here is the code that turns the img into an tensor

k=torch.from_numpy(np.asarray(l[1],dtype="int32" ))


title: How to downgrade CUDA to 10.0.10 with conda, without conflicts?

body:
I would like to go to CUDA (cudatoolkit) version compatible with
Nvidie-430 driver, i.e., 10.0.130 as recommended by the Nvidias site.
Based on this answer I did,
conda install -c pytorch cudatoolkit=10.0.130
And then I get this error (pastebin
link). (very-short version below):
(fastaiclean) eghx@eghx-nitro:~$ conda install -c pytorch cudatoolkit=10.0.130
...
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: |
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package _libgcc_mutex conflicts for:
pyzmq -> libgcc-ng[version='>=7.3.0'] -> _libgcc_mutex=[build=main]
libgcc-ng -> _libgcc_mutex=[build=main]
lcms2 -> libgcc-ng[version='>=7.3.0'] -> _libgcc_mutex=[build=main]
...
The following specifications were found to be incompatible with your system:

  - feature:/linux-64::__cuda==10.1=0
  - feature:|@/linux-64::__cuda==10.1=0

Your installed version is: 10.1

Why am I getting conflicts? Why does it say 10.1 when cuda toolkit
is 10.2.89 (conda list)? how to handle conflicts? What can I do
with this error? The conflicts are so huge, I don't know where to start.
Other

Nvidia driver 430

current cudatoolkit: 10.2.89


title: How to free up all memory pytorch is taken from gpu memory

body:
I have some kind of high level code, so model training and etc. are wrapped by pipeline_network class. My main goal is to train new model every new fold.

for train_idx, valid_idx in cv.split(meta_train[DEPTH_COLUMN].values.reshape(-1)):

        meta_train_split, meta_valid_split = meta_train.iloc[train_idx], meta_train.iloc[valid_idx]

        pipeline_network = unet(config=CONFIG, suffix = 'fold' + str(fold), train_mode=True)

But then I move on to 2nd fold everything fails out of gpu memory:

RuntimeError: cuda runtime error (2) : out of memory at /pytorch/torch/lib/THC/generic/THCStorage.cu:58

At the end of epoch I tried to manually delete that pipeline with no luck:

 def clean_object_from_memory(obj): #definition
    del obj
    gc.collect()
    torch.cuda.empty_cache()

clean_object_from_memory( clean_object_from_memory) # calling

Calling this didn't help as well:

def dump_tensors(gpu_only=True):
        torch.cuda.empty_cache()
        total_size = 0
        for obj in gc.get_objects():
            try:
                if torch.is_tensor(obj):
                    if not gpu_only or obj.is_cuda:
                        del obj
                        gc.collect()
                elif hasattr(obj, "data") and torch.is_tensor(obj.data):
                    if not gpu_only or obj.is_cuda:
                        del obj
                        gc.collect()
            except Exception as e:
                pass

How can reset pytorch then I move on to the next fold?


title: What is the difference between torch.tensor and torch.Tensor?

body:
Since version 0.4.0, it is possible to use torch.tensor and torch.Tensor

What is the difference? What was the reasoning for providing these two very similar and confusing alternatives?


title: How do I turn a Pytorch Dataloader into a numpy array to display image data with matplotlib?

body:
I am new to Pytorch. I have been trying to learn how to view my input images before I begin training on my CNN. I am having a very hard time changing the images into a form that can be used with matplotlib.

So far I have tried this:

from multiprocessing import freeze_support

import torch
from torch import nn
import torchvision
from torch.autograd import Variable
from torch.utils.data import DataLoader, Sampler
from torchvision import datasets
from torchvision.transforms import transforms
from torch.optim import Adam

import matplotlib.pyplot as plt
import numpy as np
import PIL

num_classes = 5
batch_size = 100
num_of_workers = 5

DATA_PATH_TRAIN = 'C:\\Users\Aeryes\PycharmProjects\simplecnn\images\\train'
DATA_PATH_TEST = 'C:\\Users\Aeryes\PycharmProjects\simplecnn\images\\test'

trans = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.Resize(32),
    transforms.CenterCrop(32),
    transforms.ToPImage(),
    transforms.Normalize((0.5, 0.5, 0.5),(0.5, 0.5, 0.5))
    ])

train_dataset = datasets.ImageFolder(root=DATA_PATH_TRAIN, transform=trans)
train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, num_workers=num_of_workers)

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    print(npimg)
    plt.imshow(np.transpose(npimg, (1, 2, 0, 1)))

def main():
    # get some random training images
    dataiter = iter(train_loader)
    images, labels = dataiter.next()

    # show images
    imshow(images)
    # print labels
    print(' '.join('%5s' % classes[labels[j]] for j in range(4)))

if __name__ == "__main__":
    main()

However, this throws and error:

  [[0.27058825 0.18431371 0.31764707 ... 0.18823528 0.3882353
    0.27450982]
   [0.23137254 0.11372548 0.24313724 ... 0.16862744 0.14117646
    0.40784314]
   [0.25490198 0.19607842 0.30588236 ... 0.27450982 0.25882354
    0.34509805]
   ...
   [0.2784314  0.21960783 0.2352941  ... 0.5803922  0.46666667
    0.25882354]
   [0.26666668 0.16862744 0.23137254 ... 0.2901961  0.29803923
    0.2509804 ]
   [0.30980393 0.39607844 0.28627452 ... 0.1490196  0.10588235
    0.19607842]]

  [[0.2352941  0.06274509 0.15686274 ... 0.09411764 0.3019608
    0.19215685]
   [0.22745097 0.07843137 0.12549019 ... 0.07843137 0.10588235
    0.3019608 ]
   [0.20392156 0.13333333 0.1607843  ... 0.16862744 0.2117647
    0.22745097]
   ...
   [0.18039215 0.16862744 0.1490196  ... 0.45882353 0.36078432
    0.16470587]
   [0.1607843  0.10588235 0.14117646 ... 0.2117647  0.18039215
    0.10980392]
   [0.18039215 0.3019608  0.2117647  ... 0.11372548 0.06274509
    0.04705882]]]

 ...

 [[[0.8980392  0.8784314  0.8509804  ... 0.627451   0.627451
    0.627451  ]
   [0.8509804  0.8235294  0.7921569  ... 0.54901963 0.5568628
    0.56078434]
   [0.7921569  0.7529412  0.7176471  ... 0.47058824 0.48235294
    0.49411765]
   ...
   [0.3764706  0.38431373 0.3764706  ... 0.4509804  0.43137255
    0.39607844]
   [0.38431373 0.39607844 0.3882353  ... 0.4509804  0.43137255
    0.39607844]
   [0.3882353  0.4        0.39607844 ... 0.44313726 0.42352942
    0.39215687]]

  [[0.9254902  0.90588236 0.88235295 ... 0.60784316 0.6
    0.5921569 ]
   [0.88235295 0.85490197 0.8235294  ... 0.5411765  0.5372549
    0.53333336]
   [0.8235294  0.7882353  0.75686276 ... 0.47058824 0.47058824
    0.47058824]
   ...
   [0.50980395 0.5176471  0.5137255  ... 0.58431375 0.5647059
    0.53333336]
   [0.5137255  0.53333336 0.5254902  ... 0.58431375 0.5686275
    0.53333336]
   [0.5176471  0.53333336 0.5294118  ... 0.5764706  0.56078434
    0.5294118 ]]

  [[0.95686275 0.9372549  0.90588236 ... 0.18823528 0.19999999
    0.20784312]
   [0.9098039  0.8784314  0.8352941  ... 0.1607843  0.17254901
    0.18039215]
   [0.84313726 0.7921569  0.7490196  ... 0.1372549  0.14509803
    0.15294117]
   ...
   [0.03921568 0.05490196 0.05098039 ... 0.11764705 0.09411764
    0.02745098]
   [0.04705882 0.07843137 0.06666666 ... 0.12156862 0.10196078
    0.03529412]
   [0.05098039 0.0745098  0.07843137 ... 0.12549019 0.10196078
    0.04705882]]]

 [[[0.30588236 0.28627452 0.24313724 ... 0.2901961  0.26666668
    0.21568626]
   [0.8156863  0.6666667  0.5921569  ... 0.18039215 0.23921567
    0.21568626]
   [0.9019608  0.83137256 0.85490197 ... 0.21960783 0.36862746
    0.23921567]
   ...
   [0.7058824  0.83137256 0.85490197 ... 0.2627451  0.24313724
    0.20784312]
   [0.7137255  0.84313726 0.84705883 ... 0.26666668 0.29803923
    0.21568626]
   [0.7254902  0.8235294  0.8392157  ... 0.2509804  0.27058825
    0.2352941 ]]

  [[0.24705881 0.22745097 0.19215685 ... 0.2784314  0.25490198
    0.19607842]
   [0.59607846 0.37254903 0.29803923 ... 0.16470587 0.22745097
    0.20392156]
   [0.5921569  0.4509804  0.49803922 ... 0.20784312 0.3764706
    0.2352941 ]
   ...
   [0.42352942 0.4627451  0.42352942 ... 0.23921567 0.23137254
    0.19999999]
   [0.45882353 0.5176471  0.35686275 ... 0.23921567 0.26666668
    0.19607842]
   [0.41568628 0.44313726 0.34901962 ... 0.21960783 0.23921567
    0.21568626]]

  [[0.23137254 0.20784312 0.1490196  ... 0.30588236 0.28627452
    0.19607842]
   [0.61960787 0.3764706  0.26666668 ... 0.16470587 0.24313724
    0.21568626]
   [0.57254905 0.43137255 0.48235294 ... 0.2235294  0.40392157
    0.25882354]
   ...
   [0.4        0.42352942 0.37254903 ... 0.25490198 0.24705881
    0.21568626]
   [0.43137255 0.4509804  0.29411766 ... 0.25882354 0.28235295
    0.20392156]
   [0.38431373 0.3529412  0.25490198 ... 0.2352941  0.25490198
    0.23137254]]]

 [[[0.06274509 0.09019607 0.11372548 ... 0.5803922  0.5176471
    0.59607846]
   [0.09411764 0.14509803 0.1372549  ... 0.5294118  0.49803922
    0.5058824 ]
   [0.04705882 0.09411764 0.10196078 ... 0.45882353 0.42352942
    0.38431373]
   ...
   [0.15294117 0.12941176 0.1607843  ... 0.85882354 0.8509804
    0.80784315]
   [0.14509803 0.10588235 0.1607843  ... 0.8666667  0.85882354
    0.8       ]
   [0.1490196  0.10588235 0.16470587 ... 0.827451   0.8156863
    0.7921569 ]]

  [[0.06666666 0.12156862 0.17647058 ... 0.59607846 0.5529412
    0.6039216 ]
   [0.07058823 0.10588235 0.11764705 ... 0.56078434 0.5254902
    0.5372549 ]
   [0.03921568 0.0745098  0.09803921 ... 0.48235294 0.4392157
    0.4117647 ]
   ...
   [0.2117647  0.14509803 0.2784314  ... 0.43137255 0.3529412
    0.34117648]
   [0.2235294  0.11372548 0.2509804  ... 0.4509804  0.39607844
    0.2509804 ]
   [0.25490198 0.12156862 0.24705881 ... 0.38039216 0.36078432
    0.3254902 ]]

  [[0.05490196 0.09803921 0.12549019 ... 0.46666667 0.38039216
    0.45490196]
   [0.06274509 0.09803921 0.10196078 ... 0.44705883 0.41568628
    0.3882353 ]
   [0.03921568 0.06666666 0.0862745  ... 0.3764706  0.33333334
    0.28235295]
   ...
   [0.12156862 0.14509803 0.16862744 ... 0.15686274 0.0745098
    0.09411764]
   [0.10588235 0.11372548 0.16862744 ... 0.25882354 0.18431371
    0.05490196]
   [0.12156862 0.11372548 0.17254901 ... 0.2352941  0.17254901
    0.14117646]]]]
Traceback (most recent call last):
  File "image_loader.py", line 51, in <module>
    main()
  File "image_loader.py", line 46, in main
    imshow(images)
  File "image_loader.py", line 38, in imshow
    plt.imshow(np.transpose(npimg, (1, 2, 0, 1)))
  File "C:\Users\Aeryes\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\core\fromnumeric.py", line 598, in transpose
    return _wrapfunc(a, 'transpose', axes)
  File "C:\Users\Aeryes\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\core\fromnumeric.py", line 51, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
ValueError: repeated axis in transpose

I tried to print out the arrays to get the dimensions but I do not know what to make of this. It is very confusing.

Here is my direct question: How do I view the input images before training using the tensors in my DataLoader object?


title: Using TPUs with PyTorch

body:
I am trying to use Google Cloud's TPU from Colab. I was able to do it following the tutorial by using Tensorflow.

Does anybody know if it is possible to make use of the TPUs using PyTorch?
If so how can I do it? Do you have any example?


title: In PyTorch how are layer weights and biases initialized by default?

body:
I was wondering how are layer weights and biases initialized by default? E.g. if I create the linear layer
     torch.nn.Linear(5,100)
How are weights and biases for this layer initialized by default?


title: How do I split the training dataset into training, validation and test datasets?

body: I have a custom data set of images and its target. I have created a training data set in PyTorch. I want to split it into 3 parts: training, validation and test. How do I do it?