title: How to get rid of checkerboard artifacts

body:
Hello fellow coders,

I am using a fully convolutional autoencoder to color black and white images, however, the output has a checkerboard pattern and I want to get rid of it. The checkerboard artifacts I have seen so far allways have been far smaller than mine and the usual way to get rid of them is replacing all unpooling operations with bilinear upsampling (I have been told that).

But I can not simply replace the unpooling operation because I work with different sized images, thus the unpooling operation is needed, else the output tensor could have a different size than the original.

TLDR:

How can I get rid of these checkerboard-artifacts without replacing the unpooling operations?

class AE(nn.Module):
    def __init__(self):
        super(AE, self).__init__()
        self.leaky_reLU = nn.LeakyReLU(0.2)
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2, padding=1, return_indices=True)
        self.unpool = nn.MaxUnpool2d(kernel_size=2, stride=2, padding=1)
        self.softmax = nn.Softmax2d()

        self.conv1 = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1)
        self.conv2 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, stride=1, padding=1)
        self.conv3 = nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3, stride=1, padding=1)
        self.conv4 = nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3, stride=1, padding=1)
        self.conv5 = nn.Conv2d(in_channels=512, out_channels=1024, kernel_size=3, stride=1, padding=1)
        self.conv6 = nn.ConvTranspose2d(in_channels=1024, out_channels=512, kernel_size=3, stride=1, padding=1)
        self.conv7 = nn.ConvTranspose2d(in_channels=512, out_channels=256, kernel_size=3, stride=1, padding=1)
        self.conv8 = nn.ConvTranspose2d(in_channels=256, out_channels=128, kernel_size=3, stride=1, padding=1)
        self.conv9 = nn.ConvTranspose2d(in_channels=128, out_channels=64, kernel_size=3, stride=1, padding=1)
        self.conv10 = nn.ConvTranspose2d(in_channels=64, out_channels=2, kernel_size=3, stride=1, padding=1)

    def forward(self, x):

        # encoder
        x = self.conv1(x)
        x = self.leaky_reLU(x)
        size1 = x.size()
        x, indices1 = self.pool(x)

        x = self.conv2(x)
        x = self.leaky_reLU(x)
        size2 = x.size()
        x, indices2 = self.pool(x)

        x = self.conv3(x)
        x = self.leaky_reLU(x)
        size3 = x.size()
        x, indices3 = self.pool(x)

        x = self.conv4(x)
        x = self.leaky_reLU(x)
        size4 = x.size()
        x, indices4 = self.pool(x)

        ######################
        x = self.conv5(x)
        x = self.leaky_reLU(x)

        x = self.conv6(x)
        x = self.leaky_reLU(x)
        ######################

        # decoder
        x = self.unpool(x, indices4, output_size=size4)
        x = self.conv7(x)
        x = self.leaky_reLU(x)

        x = self.unpool(x, indices3, output_size=size3)
        x = self.conv8(x)
        x = self.leaky_reLU(x)

        x = self.unpool(x, indices2, output_size=size2)
        x = self.conv9(x)
        x = self.leaky_reLU(x)

        x = self.unpool(x, indices1, output_size=size1)
        x = self.conv10(x)
        x = self.softmax(x)

        return x

.

EDIT - Solution:

Skip-Connections are the way to go!

class AE(nn.Module):
    def __init__(self):
        super(AE, self).__init__()
        self.leaky_reLU = nn.LeakyReLU(0.2)
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2, padding=1, return_indices=True)
        self.unpool = nn.MaxUnpool2d(kernel_size=2, stride=2, padding=1)
        self.softmax = nn.Softmax2d()

        self.conv1 = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1)
        self.conv2 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, stride=1, padding=1)
        self.conv3 = nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3, stride=1, padding=1)
        self.conv4 = nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3, stride=1, padding=1)
        self.conv5 = nn.Conv2d(in_channels=512, out_channels=1024, kernel_size=3, stride=1, padding=1)
        self.conv6 = nn.Conv2d(in_channels=1024, out_channels=512, kernel_size=3, stride=1, padding=1)
        self.conv7 = nn.Conv2d(in_channels=1024, out_channels=256, kernel_size=3, stride=1, padding=1)
        self.conv8 = nn.Conv2d(in_channels=512, out_channels=128, kernel_size=3, stride=1, padding=1)
        self.conv9 = nn.Conv2d(in_channels=256, out_channels=64, kernel_size=3, stride=1, padding=1)
        self.conv10 = nn.Conv2d(in_channels=128, out_channels=2, kernel_size=3, stride=1, padding=1)

    def forward(self, x):

        # encoder
        x = self.conv1(x)
        out1 = self.leaky_reLU(x)
        x = out1
        size1 = x.size()
        x, indices1 = self.pool(x)

        x = self.conv2(x)
        out2 = self.leaky_reLU(x)
        x = out2
        size2 = x.size()
        x, indices2 = self.pool(x)

        x = self.conv3(x)
        out3 = self.leaky_reLU(x)
        x = out3
        size3 = x.size()
        x, indices3 = self.pool(x)

        x = self.conv4(x)
        out4 = self.leaky_reLU(x)
        x = out4
        size4 = x.size()
        x, indices4 = self.pool(x)

        ######################
        x = self.conv5(x)
        x = self.leaky_reLU(x)

        x = self.conv6(x)
        x = self.leaky_reLU(x)
        ######################

        # decoder
        x = self.unpool(x, indices4, output_size=size4)
        x = self.conv7(torch.cat((x, out4), 1))
        x = self.leaky_reLU(x)

        x = self.unpool(x, indices3, output_size=size3)
        x = self.conv8(torch.cat((x, out3), 1))
        x = self.leaky_reLU(x)

        x = self.unpool(x, indices2, output_size=size2)
        x = self.conv9(torch.cat((x, out2), 1))
        x = self.leaky_reLU(x)

        x = self.unpool(x, indices1, output_size=size1)
        x = self.conv10(torch.cat((x, out1), 1))
        x = self.softmax(x)

        return x


title: Got "can't allocate memory" error when I run docker image

body:
I am training an NLP model using the code adapted form https://github.com/huggingface/transformers/blob/master/examples/run_glue.py. I'm working with docker toolbox on windows 10 with CPU only. The code works fine locally, and I built the docker image successfully. However when I tried to do "docker run $IMAGE_URI" I got the following error at the training step:

  File "xlnet/train_config.py", line 318, in <module>
    global_step, tr_loss = train(train_dataset, model, tokenizer)

  File "xlnet/train_config.py", line 214, in train
    outputs = model(**inputs)

...

  File "/usr/local/lib/python3.7/site-packages/pytorch_transformers/modeling_xlnet.py", line 383, in rel_shift
    x = torch.index_select(x, 1, torch.arange(klen, device=x.device, dtype=torch.long))

RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 182452224 bytes. Error code 12 (Cannot allocate memory)

When I run 'docker info', it shows "CPUs: 8, Total Memory: 7.793GiB". It should be enough...

Then I tried to allocate with memory of 10GB. There is no more error message. But it just exits at the same place without continue training.


title: Fixed Gabor Filter Convolutional Neural Networks

body:
I'm trying to build a CNN with some conv layers where half of the filters in the layer are fixed and the other half is learnable while training the model. But I didn't find anything about that.

what I'm trying to do is similar to what they did in this paper https://arxiv.org/pdf/1705.04748.pdf

Is there a way to do that in Keras, Pytorch...


title: DataParallel multi-gpu RuntimeError: chunk expects at least a 1-dimensional tensor

body:
I am trying to run my model on multiple gpus using DataParallel by setting model = nn.DataParallel(model).cuda(), but everytime getting this error -

  RuntimeError: chunk expects at least a 1-dimensional tensor (chunk at
  /pytorch/aten/src/ATen/native/TensorShape.cpp:184).

My code is correct. Does anyone know what's wrong?

I have tried setting device_ids=[0,1] parameter and also CUDA_VISIBLE_DEVICES on the terminal. Also tried different batch sizes.


Getting the following error when trying to use the RMSProp Optimizer with PyTorch:

AttributeError: module 'torch.optim' has no attribute 'RMSProp'

Code:

import torch as T
import torch.nn as nn
import torch.optim as optim

class DeepQNetwork(nn.Module):

  def __init__(self, alpha, ...):
    super(DeepQNetwork, self).__init__()
    ...
    self.optimizer = optim.RMSProp(self.parameters(), lr=alpha)
    ...

PyTorch version is 1.5.1 with Python version 3.6. There's a documentation for torch.optim and its optimizers including RMSProp, but PyCharm only suggests Adam and SGD and it really seems like all other optimizers are missing.

Does anyone have an idea? I did not find a single thing on the internet and it starts driving me crazy.

Suggesstion from PyCharm


title: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number

body:
I'm using PyTorch tutorial to calculate accuracy per class, and it is throw out an error that use tensor.item() which already is in class_correct[target] += c[i].item()

class_correct = list(0. for i in range(15))
class_total = list(0. for i in range(15))

with torch.no_grad():

    for ii, data in enumerate(test_loader):

        t_image, target, classess, image_path  = data

        t_image = t_image.to(device)
        target = target.to(device)

        outputs = model(t_image)

        _, predicted = torch.max(outputs, 1)
        c = (predicted == target).squeeze()

        for i in range(4):
            target = target[i]
            class_correct[target] += c[i].item()
            class_total[target] += 1

for i in range(14):
    print('Accuracy of %5s : %2d %%' % (
        classes[i], 100 * class_correct[i] / class_total[i]))

any comment would be appreciated.


title: How to read from a high IO dataset in pytorch which grows from epoch to epoch

body:
I use Tensorflow, but I'm writing documentation for users that will typically vary across deep learning frameworks.

When working with datasets that don't fit on the local filesystem (TB+) I sample data from a remote data store and write samples locally to a Tensorflow standardtfrecords format.

During the first epoch of training I will have only sampled a few values, therefore an epoch of local data is very small, I train on it. On epoch 2 I re-examine what data files have been produced by my sampling subprocesses (now more) and train on the expanded set of local data files for the next epoch. Repeat the process each epoch. In this way I build up a local cache of samples and can evict older samples as I fill up the local storage. The local samples cache grows at about the time the model needs the variance the most (towards the latter part of training).

In Python/Tensorflow it's crucial that I not deserialize the data in the Python training loop process because the Python GIL can't support the data transfer rates (300-600 MB/sec, the data is raw scientific uncompressible), and thus GPU performance suffers when the Python GIL can't service the training loop fast.

Writing the samples to a tfrecords file from subprocesses (python multiprocessing) allows tensorflow's native TFRecordsDataset to do deserialization outside of Python and thus we sidestep the Python GIL issues, and I can saturate a GPU with high IO data rates.

  I would like to know how I would address this issue in Pytorch. I'm writing about the sampling strategy that's being used, and want to provide specific recommendations to users of both Tensorflow and PyTorch, but I don't know the PyTorch preprocessing ecosystem well enough to write with sufficient detail.

Side note: the only purely Python based solution to support these data transfer rates may come in Python 3.8 with System V shared memory and multiprocessing, but I haven't tried that yet as support for it isn't quite sufficient (soon it will be). Existing multiprocessing solutions aren't sufficient because they require deserialization in the training loop process and thus lock the GIL during deserialization at high IO rates.


title: Installing torchaudio in Windows

body:
I am trying to install torchaudio in Windows from source. I installed sox and added it in the path env variable. Then I run

python setup.py install

cloned from the GitHub. When I import torchaudio, I get warning

  No audio backend is available.

I think this means that the sox is not configured correctly or I have done something very wrong. How to install torchaudio in Windows? My Linux machine is not good and I can't use Linux which has CUDA.


title: How to index specific object in dataloader without enumeration

body:
I have the following code for testing a network in pytorch (taken from pix2pix):

for i, data in enumerate(dataset):
    model.set_input(data)
    model.test()

I have tried altering this to access only one element (any) over iterating a number of elements. the following:

data=dataset[0]

leads to

  TypeError: 'CustomDatasetDataLoader' object does not support indexing

where CustomDatasetDataLoader is a custom class by the author which (if I am understanding correctly) herniates from torch.utils.data.DataLoader.

I am fairly new to python, how do I index a single specific element of dataset if it is at all possible?


title: Predicting sequence of grid coordinates with PyTorch

body:
I have a similar open question here on Cross Validated (though not implementation focused, which I intend this question to be, so I think they are both valid).

I'm working on a project that uses sensors to monitor a persons GPS location. The coordinates will then be converted to a simple-grid representation. What I want to try and do is after recording a users routes, train a neural network to predict the next coordinates, i.e. take the example below where a user repeats only two routes over time, Home->A and Home->B.

I want to train an RNN/LSTM with sequences of varying lengths e.g. (14,3), (13,3), (12,3), (11,3), (10,3), (9,3), (8,3), (7,3), (6,3), (5,3), (4,3), (3,3), (2,3), (1,3) and then also predict with sequences of varying lengths e.g. for this example route if I called

route = [(14,3), (13,3), (12,3), (11,3), (10,3)] //pseudocode
pred = model.predict(route)

pred should give me (9,3) (or ideally even a longer prediction e.g. ((9,3), (8,3), (7,3), (6,3), (5,3), (4,3), (3,3), (2,3), (1,3))

How do I feed such training sequences to the init and forward operations identified below?

self.rnn = nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)
out, hidden = self.rnn(x, hidden)

Also, should the entire route be a tensor or each set of coordinates within the route a tensor?


title: What are the differences between torch.jit.trace and torch.jit.script in torchscript?

body:
Torchscript provides torch.jit.trace and torch.jit.script to convert pytorch code from eager mode to script model. From the documentation, I can understand torch.jit.trace cannot handle control flows and other data structures present in the python. Hence torch.jit.script was developed to overcome the problems in torch.jit.trace.
But it looks like torch.jit.script works for all the cases, then why do we need torch.jit.trace?
Please help me understand the difference between these two methods


title: Interpretation of in_channels and out_channels in Conv2D in Pytorch Convolution Neural Networks (CNN)

body:
Let us suppose I have a CNN with starting 2 layers as:

inp_conv = Conv2D(in_channels=1,out_channels=6,kernel_size=(3,3))

Please correct me if I am wrong but I think what this line if code can be thought as that

There is a single grayscale image coming as input where we have to use 6 different kernals of same size (3,3) to make 6 different feature maps from a single image.

And if I have a second Conv2D layer just after first one as

second_conv_connected_to_inp_conv = Conv2D(in_channels=6,out_channels=12,kernel_size=(3,3))

What does this mean in terms of out_channels? Is there going to be 12 new feature maps for each of the 6 feature maps coming as output from first layer OR are there going to be a total of 12 feature maps from 6 incoming features?


title: Pytorch - TypeError: ToTensor() takes no arguments using torchvision.transform

body:
I’m trying to load in a dataset for super-resolution and I have set up two functions which use Compose to crop and resize the images.

The function I have created for the input images works correctly and they are outputting as expected. The transform function for the target images is basically identical, just omitting the resize part of it.

def input_trans(c_size, sF):
    return Compose([
        CenterCrop(c_size),
        Resize(c_size // sF),
        ToTensor(),
    ])

def goal_trans(c_size):
    return Compose([
        CenterCrop(c_size),
        ToTensor(),
    ])

These functions are used in my dataset class when the images are loaded. I originally had goal = input.Copy() but I have changed it so both input and goal load the image separately. (was testing if the .copy() was the issue

def __getitem__(self, idx):
    input = Image.open(self.image_filenames[idx]).convert('RGB')
    goal = Image.open(self.image_filenames[idx]).convert('RGB')
    if self.input_transform:
        input = self.input_transform(input)
    if self.goal_transform:
        print(goal)
        print(goal.size)
        goal = self.goal_transform(goal)

    return input, goal

The error I receive is the following:

Traceback (most recent call last):
  File "main.py", line 128, in <module>
    main()  # execute this only when run directly, not when imported!
  File "main.py", line 55, in main
    train_model(epoch)
  File "main.py", line 40, in train_model
    for data_item, batch in enumerate(training_data_loader):
  File "C:\Users\[NAME]\anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\Users\[NAME]\anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\[NAME]\anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\[NAME]\anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "main.py", line 118, in __getitem__
    goal = self.goal_transform(goal)
  File "C:\Users\[NAME]\anaconda3\envs\pytorch\lib\site-packages\torchvision\transforms\transforms.py", line 70, in __call__
    img = t(img)
TypeError: ToTensor() takes no arguments

Confuses me because it doesn’t seem to have a problem with the first transformation (Ive checked and it does output before crashing).

I would appreciate any help you guys can give,

Thanks :)


title: Conversion from pytorch to onnx to tensorflow graph definition to tflite - TOCO failed - type check fail

body:
I have a simple network of two convolutional layers and one fully connected one defined in pytorch essentially as follows:

def __init__([...])
    [...]
    self.conv1 = nn.Conv1d(1, channels_conv1, width_conv1)
    self.conv2 = nn.Conv1d(channels_conv1, channels_conv2, width_conv2)
    self.fc1 = nn.Linear(hidden_layer_size, 2)

def forward(self, x):
    x = functional.max_pool1d(functional.relu(self.conv1(x)), 2, stride=2)
    x = functional.max_pool1d(functional.relu(self.conv2(x)), 2, stride=2)
    x = x.view(-1, self.num_flat_features(x))
    x = functional.softmax(self.fc1(x))
    return x

I want to convert it to tflite. So first it is converted to onnx

torch.onnx.export(model, input, "net.onnx",
                  export_params=True,
                  input_names=['input'],
                  output_names=['output'],
                  verbose=true)

Then I convert the result to a tensorflow graph definition with onnx-tf. The resulting net.pb is ok as it produces the same output as the original with prepare(onnx.load('net.onnx')).run(...).

However, I have two problems: a minor one is that the net.pb graph doesn't contain the output node anymore and I have to fish for the output node.
The second is that when I try to perform the final conversion with

tflite_convert --output_file=net.tflite --graph_def_file=net.pb --input_arrays=input --output_arrays=Softmax

I get a TOCO failure at a type check:

tensorflow.lite.python.convert.ConverterError: TOCO failed. See console for info.
2018-11-16 16:11:37.592030: I tensorflow/lite/toco/import_tensorflow.cc:1280] Converting unsupported operation: Where
2018-11-16 16:11:37.601384: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] Before Removing unused ops: 61 operators, 113 arrays (0 quantized)
2018-11-16 16:11:37.602005: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 61 operators, 113 arrays (0 quantized)
2018-11-16 16:11:37.602311: F tensorflow/lite/toco/graph_transformations/resolve_constant_gather.cc:105] Check failed: coords_array.data_type == ArrayDataType::kInt32 Only int32 indices are supported
Aborted (core dumped)

I tried digging in the network but I can't seem to find the vexing object and I didn't find issues obviously related to this problem. Any pointer to a point where this process might have derailed would be a great!

tf-nightly==1.13.0.dev20181116
onnx==1.3.0
torch-nightly==1.0.0.dev201811

and master (commit b5fef1b) of onnx-tensorflow


title: Pytorch installation issue under Anaconda

body:
I followed the link here to install fastai library using pip install git+https://github.com/fastai/fastai.git

It gave me the following error message. These messages keep the same even I installed Pytorch successfully  using  conda install pytorch-cpu -c pytorch
and pip3 install torchvision. What can be the reason?

Collecting torch<0.4 (from fastai==0.7.0)
  Using cached https://files.pythonhosted.org/packages/5f/e9/bac4204fe9cb1a002ec6140b47f51affda1655379fe302a1caef421f9846/torch-0.1.2.post1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\shuxi\AppData\Local\Temp\pip-install-7sjptuad\torch\setup.py", line 11, in <module>
        raise RuntimeError(README)
    RuntimeError: PyTorch does not currently provide packages for PyPI (see status at https://github.com/pytorch/pytorch/issues/566).

    Please follow the instructions at http://pytorch.org/ to install with miniconda instead.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\shuxi\AppData\Local\Temp\pip-install-7sjptuad\torch\


title: Why is looping through pytorch tensors so slow (compared to Numpy)?

body:
I've been working with image transformations recently and came to a situation where I have a large array (shape of 100,000 x 3) where each row represents a point in 3D space like:
pnt = [x y z]

All I'm trying to do is iterating through each point and matrix multiplying each point with a matrix called T (shape = 3 X 3).
Test with Numpy:
def transform(pnt_cloud, T):

    i = 0
    for pnt in pnt_cloud:
        xyz_pnt = np.dot(T, pnt)

        if xyz_pnt[0] > 0:
            arr[i] = xyz_pnt[0]

        i += 1

    return arr

Calling the following code and calculating runtime (using %time) gives the output:
Out[190]: CPU times: user 670 ms, sys: 7.91 ms, total: 678 ms
Wall time: 674 ms

Test with Pytorch Tensor:
import torch

tensor_cld = torch.tensor(pnt_cloud)
tensor_T   = torch.tensor(T)

def transform(pnt_cloud, T):
    depth_array = torch.tensor(np.zeros(pnt_cloud.shape[0]))

    i = 0
    for pnt in pnt_cloud:
        xyz_pnt = torch.matmul(T, pnt)

        if xyz_pnt[0] > 0:
            depth_array[i] = xyz_pnt[0]

        i += 1

    return depth_array

Calling the following code and calculating runtime (using %time) gives the output:
Out[199]: CPU times: user 6.15 s, sys: 28.1 ms, total: 6.18 s
Wall time: 6.09 s

NOTE: Doing the same with torch.jit only reduces 2s
I would have thought that PyTorch tensor computations would be much faster due to the way PyTorch breaks its code down in the compiling stage. What am I missing here?
Would there be any faster way to do this other than using Numba?


title: Combining Neural Networks Pytorch

body:
I have 2 images as input, x1 and x2  and try to use convolution as a similarity measure. The idea is that the learned weights substitute more traditional measure of similarity (cross correlation, NN, ...). Defining my forward function as follows:

def forward(self,x1,x2):
    out_conv1a = self.conv1(x1)
    out_conv2a = self.conv2(out_conv1a)
    out_conv3a = self.conv3(out_conv2a)

    out_conv1b = self.conv1(x2)
    out_conv2b = self.conv2(out_conv1b)
    out_conv3b = self.conv3(out_conv2b)

Now for the similarity measure:

out_cat = torch.cat([out_conv3a, out_conv3b],dim=1)
futher_conv = nn.Conv2d(out_cat)

My question is as follows:

1) Would Depthwise/Separable Convolutions as in the google paper yield any advantage over 2d convolution of the concatenated input. For that matter can convolution be a similarity measure, cross correlation and convolution are very similar.

2) It is my understanding that the groups=2 option in conv2d would provide 2 separate inputs to train weights with, in this case each of the previous networks weights. How are these combined afterwards?

For a basic concept see here.


title: How to convert a torch tensor into a byte string?

body:
I'm trying to serialize a torch tensor using protobuf and it seems using BytesIO along with torch.save() doesn't work. I have tried:
import torch
import io
x = torch.randn(size=(1,20))
buff = io.BytesIO()
torch.save(x, buff)
print(f'buffer: {buff.read()}')

to no avail as it results in b'' in the output! How should I be going about this?


title: Pytorch Unfold and Fold: How do I put this image tensor back together again?

body:
I am trying to filter a single channel 2D image of size 256x256 using unfold to create 16x16 blocks with an overlap of 8. This is shown below:
# I = [256, 256] image
kernel_size = 16
stride = bx/2
patches = I.unfold(1, kernel_size,
int(stride)).unfold(0, kernel_size, int(stride)) # size = [31, 31, 16, 16]

I have started to attempt to put the image back together with fold but I’m not quite there yet. I’ve tried to use view to get the image to ‘fit’ the way it’s supposed to but I don’t see how this would preserve the original image. Perhaps I’m overthinking this.
# patches.shape = [31, 31, 16, 16]
patches = = filt_data_block.contiguous().view(-1, kernel_size*kernel_size) # [961, 256]
patches = patches.permute(1, 0) # size = [951, 256]

Any help would be greatly appreciated. Thanks very much.


title: Cannot run Flask on Docker (ModuleNotFoundError)

body:
I have been trying to run my Python API (using Flask) with Docker for a while, but keep running into this issue:

ModuleNotFoundError: No module named 'flask'

Running this on Mac OS X (10.14.5) with Docker version 19.03.1, build 74b1e89.

My Dockerfile looks like this:

FROM pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime

RUN apt-get update && apt-get install -y python3
RUN apt-get install -y python3-pip
RUN apt-get install -y build-essential
RUN python3 --version
RUN pip3 --version

COPY . /app
WORKDIR /app

RUN pip3 install -r requirements.txt
RUN pip3 install flask # just to be sure Flask gets installed
RUN which flask
RUN ls -alh

ENTRYPOINT ["python3"]
CMD ["api.py"]

Note that I'm extending the PyTorch docker image, because my API will be running an ML model using PyTorch. I tried starting from other images such as ufoym/deepo but with the same Flask error.

I stripped my api.py file to look like this (running it locally is no problem):

from flask import Flask, request, jsonify
#import classifier # commented out for now because I just want the API to work first

app = Flask(__name__)

@app.route('/', methods=['GET', 'POST'])
def process():
    # TODO
    return 'test'

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0')

When I build my Docker image, I get the following output:

$ docker build -t flower-image-classifier:latest .
Sending build context to Docker daemon  269.9MB
Step 1/14 : FROM pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime
 ---> 299bfb9e54db
Step 2/14 : RUN apt-get update && apt-get install -y python3
 ---> Running in f1d7db593022
Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Get:5 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [908 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1558 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [14.1 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [9827 kB]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [12.7 kB]
Get:10 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [573 kB]
Get:11 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [6117 B]
Get:12 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [176 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [1292 kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [13.1 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [980 kB]
Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [19.1 kB]
Get:17 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [7942 B]
Get:18 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [8532 B]
Fetched 16.0 MB in 9s (1744 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  dh-python file libmagic1 libmpdec2 libpython3-stdlib libpython3.5-minimal
  libpython3.5-stdlib mime-support python3-minimal python3.5 python3.5-minimal
Suggested packages:
  python3-doc python3-tk python3-venv python3.5-venv python3.5-doc
  binfmt-support
The following NEW packages will be installed:
  dh-python file libmagic1 libmpdec2 libpython3-stdlib libpython3.5-minimal
  libpython3.5-stdlib mime-support python3 python3-minimal python3.5
  python3.5-minimal
0 upgraded, 12 newly installed, 0 to remove and 29 not upgraded.
Need to get 4885 kB of archives.
After this operation, 28.4 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython3.5-minimal amd64 3.5.2-2ubuntu0~16.04.5 [524 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3.5-minimal amd64 3.5.2-2ubuntu0~16.04.5 [1598 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-minimal amd64 3.5.1-3 [23.3 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial/main amd64 mime-support all 3.59ubuntu1 [31.0 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial/main amd64 libmpdec2 amd64 2.4.2-1 [82.6 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython3.5-stdlib amd64 3.5.2-2ubuntu0~16.04.5 [2134 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3.5 amd64 3.5.2-2ubuntu0~16.04.5 [165 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial/main amd64 libpython3-stdlib amd64 3.5.1-3 [6818 B]
Get:9 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 dh-python all 2.20151103ubuntu1.1 [74.1 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3 amd64 3.5.1-3 [8710 B]
Get:11 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libmagic1 amd64 1:5.25-2ubuntu1.2 [216 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 file amd64 1:5.25-2ubuntu1.2 [21.2 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 4885 kB in 7s (663 kB/s)
Selecting previously unselected package libpython3.5-minimal:amd64.
(Reading database ... 11013 files and directories currently installed.)
Preparing to unpack .../libpython3.5-minimal_3.5.2-2ubuntu0~16.04.5_amd64.deb ...
Unpacking libpython3.5-minimal:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Selecting previously unselected package python3.5-minimal.
Preparing to unpack .../python3.5-minimal_3.5.2-2ubuntu0~16.04.5_amd64.deb ...
Unpacking python3.5-minimal (3.5.2-2ubuntu0~16.04.5) ...
Selecting previously unselected package python3-minimal.
Preparing to unpack .../python3-minimal_3.5.1-3_amd64.deb ...
Unpacking python3-minimal (3.5.1-3) ...
Selecting previously unselected package mime-support.
Preparing to unpack .../mime-support_3.59ubuntu1_all.deb ...
Unpacking mime-support (3.59ubuntu1) ...
Selecting previously unselected package libmpdec2:amd64.
Preparing to unpack .../libmpdec2_2.4.2-1_amd64.deb ...
Unpacking libmpdec2:amd64 (2.4.2-1) ...
Selecting previously unselected package libpython3.5-stdlib:amd64.
Preparing to unpack .../libpython3.5-stdlib_3.5.2-2ubuntu0~16.04.5_amd64.deb ...
Unpacking libpython3.5-stdlib:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Selecting previously unselected package python3.5.
Preparing to unpack .../python3.5_3.5.2-2ubuntu0~16.04.5_amd64.deb ...
Unpacking python3.5 (3.5.2-2ubuntu0~16.04.5) ...
Selecting previously unselected package libpython3-stdlib:amd64.
Preparing to unpack .../libpython3-stdlib_3.5.1-3_amd64.deb ...
Unpacking libpython3-stdlib:amd64 (3.5.1-3) ...
Selecting previously unselected package dh-python.
Preparing to unpack .../dh-python_2.20151103ubuntu1.1_all.deb ...
Unpacking dh-python (2.20151103ubuntu1.1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Setting up libpython3.5-minimal:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Setting up python3.5-minimal (3.5.2-2ubuntu0~16.04.5) ...
Setting up python3-minimal (3.5.1-3) ...
Selecting previously unselected package python3.
(Reading database ... 11957 files and directories currently installed.)
Preparing to unpack .../python3_3.5.1-3_amd64.deb ...
Unpacking python3 (3.5.1-3) ...
Selecting previously unselected package libmagic1:amd64.
Preparing to unpack .../libmagic1_1%3a5.25-2ubuntu1.2_amd64.deb ...
Unpacking libmagic1:amd64 (1:5.25-2ubuntu1.2) ...
Selecting previously unselected package file.
Preparing to unpack .../file_1%3a5.25-2ubuntu1.2_amd64.deb ...
Unpacking file (1:5.25-2ubuntu1.2) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Setting up mime-support (3.59ubuntu1) ...
Setting up libmpdec2:amd64 (2.4.2-1) ...
Setting up libpython3.5-stdlib:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Setting up python3.5 (3.5.2-2ubuntu0~16.04.5) ...
Setting up libpython3-stdlib:amd64 (3.5.1-3) ...
Setting up libmagic1:amd64 (1:5.25-2ubuntu1.2) ...
Setting up file (1:5.25-2ubuntu1.2) ...
Setting up python3 (3.5.1-3) ...
running python rtupdate hooks for python3.5...
running python post-rtupdate hooks for python3.5...
Setting up dh-python (2.20151103ubuntu1.1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Removing intermediate container f1d7db593022
 ---> 562a488914b2
Step 3/14 : RUN apt-get install -y python3-pip
 ---> Running in 3884de8e3194
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libexpat1 libexpat1-dev libpython3-dev libpython3.5 libpython3.5-dev
  python-pip-whl python3-dev python3-pkg-resources python3-setuptools
  python3-wheel python3.5-dev
Suggested packages:
  python-setuptools-doc
The following NEW packages will be installed:
  libexpat1-dev libpython3-dev libpython3.5 libpython3.5-dev python-pip-whl
  python3-dev python3-pip python3-pkg-resources python3-setuptools
  python3-wheel python3.5-dev
The following packages will be upgraded:
  libexpat1
1 upgraded, 11 newly installed, 0 to remove and 28 not upgraded.
Need to get 40.7 MB of archives.
After this operation, 62.4 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libexpat1 amd64 2.1.0-7ubuntu0.16.04.4 [71.4 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libexpat1-dev amd64 2.1.0-7ubuntu0.16.04.4 [115 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython3.5 amd64 3.5.2-2ubuntu0~16.04.5 [1360 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython3.5-dev amd64 3.5.2-2ubuntu0~16.04.5 [37.3 MB]
Get:5 http://archive.ubuntu.com/ubuntu xenial/main amd64 libpython3-dev amd64 3.5.1-3 [6926 B]
Get:6 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 python-pip-whl all 8.1.1-2ubuntu0.4 [1110 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3.5-dev amd64 3.5.2-2ubuntu0~16.04.5 [413 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-dev amd64 3.5.1-3 [1186 B]
Get:9 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 python3-pip all 8.1.1-2ubuntu0.4 [109 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-pkg-resources all 20.7.0-1 [79.0 kB]
Get:11 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-setuptools all 20.7.0-1 [88.0 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial/universe amd64 python3-wheel all 0.29.0-1 [48.1 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 40.7 MB in 1min 5s (624 kB/s)
(Reading database ... 12000 files and directories currently installed.)
Preparing to unpack .../libexpat1_2.1.0-7ubuntu0.16.04.4_amd64.deb ...
Unpacking libexpat1:amd64 (2.1.0-7ubuntu0.16.04.4) over (2.1.0-7ubuntu0.16.04.3) ...
Selecting previously unselected package libexpat1-dev:amd64.
Preparing to unpack .../libexpat1-dev_2.1.0-7ubuntu0.16.04.4_amd64.deb ...
Unpacking libexpat1-dev:amd64 (2.1.0-7ubuntu0.16.04.4) ...
Selecting previously unselected package libpython3.5:amd64.
Preparing to unpack .../libpython3.5_3.5.2-2ubuntu0~16.04.5_amd64.deb ...
Unpacking libpython3.5:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Selecting previously unselected package libpython3.5-dev:amd64.
Preparing to unpack .../libpython3.5-dev_3.5.2-2ubuntu0~16.04.5_amd64.deb ...
Unpacking libpython3.5-dev:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Selecting previously unselected package libpython3-dev:amd64.
Preparing to unpack .../libpython3-dev_3.5.1-3_amd64.deb ...
Unpacking libpython3-dev:amd64 (3.5.1-3) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../python-pip-whl_8.1.1-2ubuntu0.4_all.deb ...
Unpacking python-pip-whl (8.1.1-2ubuntu0.4) ...
Selecting previously unselected package python3.5-dev.
Preparing to unpack .../python3.5-dev_3.5.2-2ubuntu0~16.04.5_amd64.deb ...
Unpacking python3.5-dev (3.5.2-2ubuntu0~16.04.5) ...
Selecting previously unselected package python3-dev.
Preparing to unpack .../python3-dev_3.5.1-3_amd64.deb ...
Unpacking python3-dev (3.5.1-3) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../python3-pip_8.1.1-2ubuntu0.4_all.deb ...
Unpacking python3-pip (8.1.1-2ubuntu0.4) ...
Selecting previously unselected package python3-pkg-resources.
Preparing to unpack .../python3-pkg-resources_20.7.0-1_all.deb ...
Unpacking python3-pkg-resources (20.7.0-1) ...
Selecting previously unselected package python3-setuptools.
Preparing to unpack .../python3-setuptools_20.7.0-1_all.deb ...
Unpacking python3-setuptools (20.7.0-1) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../python3-wheel_0.29.0-1_all.deb ...
Unpacking python3-wheel (0.29.0-1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Setting up libexpat1:amd64 (2.1.0-7ubuntu0.16.04.4) ...
Setting up libexpat1-dev:amd64 (2.1.0-7ubuntu0.16.04.4) ...
Setting up libpython3.5:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Setting up libpython3.5-dev:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Setting up libpython3-dev:amd64 (3.5.1-3) ...
Setting up python-pip-whl (8.1.1-2ubuntu0.4) ...
Setting up python3.5-dev (3.5.2-2ubuntu0~16.04.5) ...
Setting up python3-dev (3.5.1-3) ...
Setting up python3-pip (8.1.1-2ubuntu0.4) ...
Setting up python3-pkg-resources (20.7.0-1) ...
Setting up python3-setuptools (20.7.0-1) ...
Setting up python3-wheel (0.29.0-1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Removing intermediate container 3884de8e3194
 ---> 388a6aa0bb5f
Step 4/14 : RUN apt-get install -y build-essential
 ---> Running in 2e8c8dc95d9e
Reading package lists...
Building dependency tree...
Reading state information...
build-essential is already the newest version (12.1ubuntu2).
0 upgraded, 0 newly installed, 0 to remove and 28 not upgraded.
Removing intermediate container 2e8c8dc95d9e
 ---> 3c08d3467777
Step 5/14 : RUN python3 --version
 ---> Running in 027dcf1dc8b6
Python 3.6.8 :: Anaconda, Inc.
Removing intermediate container 027dcf1dc8b6
 ---> 0a8389f2fe13
Step 6/14 : RUN pip3 --version
 ---> Running in 1df781383970
pip 8.1.1 from /usr/lib/python3/dist-packages (python 3.5)
Removing intermediate container 1df781383970
 ---> ad30a1cd5323
Step 7/14 : COPY . /app
 ---> db6769b7ba80
Step 8/14 : WORKDIR /app
 ---> Running in c234276bf1f6
Removing intermediate container c234276bf1f6
 ---> face402e69b6
Step 9/14 : RUN pip3 install -r requirements.txt
 ---> Running in d89df8c75e5a
Collecting Flask==1.1.1 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/9b/93/628509b8d5dc749656a9641f4caf13540e2cdec85276964ff8f43bbb1d3b/Flask-1.1.1-py2.py3-none-any.whl (94kB)
Collecting numpy==1.16.4 (from -r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/bb/ef/d5a21cbc094d3f4d5b5336494dbcc9550b70c766a8345513c7c24ed18418/numpy-1.16.4-cp35-cp35m-manylinux1_x86_64.whl (17.2MB)
Collecting Pillow==6.1.0 (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/d6/98/0d360dbc087933679398d73187a503533ec0547ba4ffd2115365605559cc/Pillow-6.1.0-cp35-cp35m-manylinux1_x86_64.whl (2.1MB)
Collecting Werkzeug>=0.15 (from Flask==1.1.1->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/d1/ab/d3bed6b92042622d24decc7aadc8877badf18aeca1571045840ad4956d3f/Werkzeug-0.15.5-py2.py3-none-any.whl (328kB)
Collecting itsdangerous>=0.24 (from Flask==1.1.1->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting click>=5.1 (from Flask==1.1.1->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
Collecting Jinja2>=2.10.1 (from Flask==1.1.1->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl (124kB)
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10.1->Flask==1.1.1->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/6e/57/d40124076756c19ff2269678de7ae25a14ebbb3f6314eb5ce9477f191350/MarkupSafe-1.1.1-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: Werkzeug, itsdangerous, click, MarkupSafe, Jinja2, Flask, numpy, Pillow
Successfully installed Flask-1.1.1 Jinja2-2.10.1 MarkupSafe-1.1.1 Pillow-6.1.0 Werkzeug-0.15.5 click-7.0 itsdangerous-1.1.0 numpy-1.16.4
You are using pip version 8.1.1, however version 19.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container d89df8c75e5a
 ---> 95766b1e174e
Step 10/14 : RUN pip3 install flask
 ---> Running in 97e806168780
Requirement already satisfied (use --upgrade to upgrade): flask in /usr/local/lib/python3.5/dist-packages
Requirement already satisfied (use --upgrade to upgrade): itsdangerous>=0.24 in /usr/local/lib/python3.5/dist-packages (from flask)
Requirement already satisfied (use --upgrade to upgrade): Jinja2>=2.10.1 in /usr/local/lib/python3.5/dist-packages (from flask)
Requirement already satisfied (use --upgrade to upgrade): Werkzeug>=0.15 in /usr/local/lib/python3.5/dist-packages (from flask)
Requirement already satisfied (use --upgrade to upgrade): click>=5.1 in /usr/local/lib/python3.5/dist-packages (from flask)
Requirement already satisfied (use --upgrade to upgrade): MarkupSafe>=0.23 in /usr/local/lib/python3.5/dist-packages (from Jinja2>=2.10.1->flask)
You are using pip version 8.1.1, however version 19.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container 97e806168780
 ---> dd8d93e7b8bc
Step 11/14 : RUN which flask
 ---> Running in 0ac4678d5ecd
/usr/local/bin/flask
Removing intermediate container 0ac4678d5ecd
 ---> e9325cc606ec
Step 12/14 : RUN ls -alh
 ---> Running in e249ecafd7fa
total 1.8M
drwxr-xr-x 6 root root  4.0K Aug  4 14:29 .
drwxr-xr-x 1 root root  4.0K Aug  4 14:30 ..
-rw-r--r-- 1 root root   11K Aug  2 11:15 .DS_Store
drwxr-xr-x 5 root root  4.0K Jul 28 11:45 .git
-rw-r--r-- 1 root root    18 Aug  2 13:52 .gitignore
-rw-r--r-- 1 root root   878 Aug  4 14:27 Dockerfile
-rw-r--r-- 1 root root 1004K Jul 24 12:49 Image Classifier Project.html
-rw-r--r-- 1 root root  690K Jul 24 12:50 Image Classifier Project.ipynb
-rw-r--r-- 1 root root  1.1K Feb  6  2018 LICENSE
-rw-r--r-- 1 root root   686 Aug  2 14:04 README.md
drwxr-xr-x 2 root root  4.0K Aug  4 13:05 __pycache__
-rw-r--r-- 1 root root  1018 Aug  4 14:12 api.py
drwxr-xr-x 2 root root  4.0K Jul 20 12:32 assets
-rw-r--r-- 1 root root  2.2K Feb  6  2018 cat_to_name.json
drwxr-xr-x 2 root root  4.0K Jul 24 12:06 checkpoints
-rw-r--r-- 1 root root   245 Aug  2 21:48 environment.yml
-rw-r--r-- 1 root root   921 Aug  4 09:11 environment0.yml
-rw-r--r-- 1 root root  8.6K Aug  4 13:02 flower_classifier.py
-rw-r--r-- 1 root root  1.1K Aug  2 19:17 predict.py
-rw-r--r-- 1 root root    41 Aug  4 12:54 requirements.txt
-rw-r--r-- 1 root root  1.6K Aug  2 18:58 train.py
Removing intermediate container e249ecafd7fa
 ---> dfe9737330ac
Step 13/14 : ENTRYPOINT ["python3"]
 ---> Running in 57782c4a0d03
Removing intermediate container 57782c4a0d03
 ---> 7e6b0d1d5656
Step 14/14 : CMD ["api.py"]
 ---> Running in 8400de10dc2d
Removing intermediate container 8400de10dc2d
 ---> 33733bebf60d
Successfully built 33733bebf60d
Successfully tagged flower-image-classifier:latest

When I run the container, I get the following error:

$ docker run -p 5000:5000 flower-image-classifier
Traceback (most recent call last):
  File "api.py", line 1, in <module>
    from flask import Flask, request, jsonify
ModuleNotFoundError: No module named 'flask'

My question to the Docker/Python experts here: how is it possible that flask is not found, when pip has clearly installed it according to the logs?

Note that I also tried starting from the continuumio/miniconda3 image at first, just like in this issue: Docker Flask ModuleNotFoundError: No module named 'flask'

I created this environment.yml:

name: fic
channels:
  - pytorch
  - defaults
dependencies:
  - flask=1.1.1
  - numpy=1.16.4
  - numpy-base=1.16.4
  - pillow=6.1.0
  - python==3.6.9
  - pytorch=1.1.0
  - torchvision=0.3.0
  - pip:
    - pip==19.2.1
prefix: //anaconda3/envs/fic

Then I modified my Dockerfile a bit to extend from that image, and to run this to install the dependencies with Conda:

RUN echo "source activate $(head -1 ./environment.yml | cut -d' ' -f2)" > ~/.bashrc
ENV PATH /opt/conda/envs/$(head -1 ./environment.yml | cut -d' ' -f2)/bin:$PATH

Unfortunately, this downloads & extracts everything every time I build the Docker image, which takes a while and the downloads often randomly fail: CondaError: Downloaded bytes did not match Content-Length.

And when it finally built successfully, I still got the same ModuleNotFoundError for Flask.

What is wrong here? Thanks for reading


title: Split a multiple dimensional pytorch tensor into "n" smaller tensors

body:
Let's say I have a 5D tensor which has this shape for example : (1, 3, 10, 40, 1). I want to split it into smaller equal tensors (if possible) according to a certain dimension with a step equal to 1 while preserving the other dimensions.

Let's say for example I want to split it according to the fourth dimension (=40) where each tensor will have a size equal to 10. So the first tensor_1 will have values from 0->9, tensor_2 will have values from 1->10 and so on.

The 39 tensors will have these shapes :

Shape of tensor_1 : (1, 3, 10, 10, 1)
Shape of tensor_2 : (1, 3, 10, 10, 1)
Shape of tensor_3 : (1, 3, 10, 10, 1)
...
Shape of tensor_39 : (1, 3, 10, 10, 1)

Here's what I have tried :

a = torch.randn(1, 3, 10, 40, 1)

chunk_dim = 10
a_split = torch.chunk(a, chunk_dim, dim=3)

This gives me 4 tensors. How can I edit this so I'll have 39 tensors with a step = 1 like I explained ?


title: How to sum based off index vector

body:
I have 3 vectors - a sum vector, a contribution vector, and a value vector. I want to sum the value vectors according to their contribution vector and place them in their corresponding index in the sum vector. An example is:

A = [0;0] (sum vector), B = [0,0,1,1] (contribution vector) C=[20,30,40,10] (value vector)

Output:
A = [20+30;40+10]

Such that the B vector is the same length as C and their corresponding index tell us what position in A to be added to.

I am able to achieve this by a for loop as such:

for index,value in enumerate(C):
    A[B[index]]+=value

However, as this will be part of my NN model forward loop it will cause significant performance issue. Specifically I was looking for a vector/matrix sorting approach that will be more efficient. In the example above, something that worked efficiently for me was:

A=torch.zeros(2,1)
C=C.reshape(2,2)
sum=torch.sum(C,1).reshape(2,1)
A += sum

However, I run into issues as it is not always the case that the indexes of A have the same contribution. For example - the case such that B = [0,0,0,1,1] and C=[20,30,40,10,50]. Is there a function or a strategic way to do this for general cases? Thanks!


title: Undefined symbol on torchvision import

body:
pip install torch==0.4.1 -f https://download.pytorch.org/whl/cu100/stable

cat /usr/local/cuda/version.txt
CUDA Version 10.1.168

python -V
Python 3.5.2

python -c "import torch; print(torch.__version__)"
0.4.1

Get error on :

python -c "import torchvision; print(torchvision.__version__)"

    ...
    from torchvision import _C
    ImportError: /my_env/lib/python3.5/site-packages/torchvision/_C.cpython-35m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail36_typeMetaDataInstance_preallocated_7E

Versions:

pip freeze | grep torch
torch==0.4.1
torchfile==0.1.0
torchvision==0.3.0

I have tried:

pip install torchvision==0.4.1 --no-dependencies
Collecting torchvision==0.4.1
  ERROR: Could not find a version that satisfies the requirement torchvision==0.4.1 (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3, 0.3.0)
ERROR: No matching distribution found for torchvision==0.4.1

How to install older version of torchvision?


title: Repeating a pytorch tensor without copying memory

body:
Does pytorch support repeating a tensor without allocating significantly more memory?

Assume we have a tensor

t = torch.ones((1,1000,1000))
t10 = t.repeat(10,1,1)

Repeating t 10 times will require take 10x the memory. Is there a way how I can create a tensor t10 without allocating significantly more memory?

Here is a related question, but without answers.


title: What is the algebraic expression for PyTorch's ConvTranspose2d's output shape?

body:
When using PyTorch's ConvTranspose2d as such:

w = 5 # input width
h = 5 # output height
nn.ConvTranspose2d(in_channels, out_channels, kernel_size=k, stride=s, padding=p)

What is the formula for the dimensions of the output in each channel? I tried a few examples and cannot derive the pattern. For some reason adding padding seems to shrink the output size (example starts with 5 x 5 as above):

# yields an 11 x 11 image
nn.ConvTranspose2d(in_channels, out_channels, kernel_size=3, stride=2, padding=0)

# yields a 7 x 7 image
nn.ConvTranspose2d(in_channels, out_channels, kernel_size=3, stride=2, padding=2)

Using a larger kernel or stride both increase (expected) but not at the rate that I expected:

# yields an 11 x 11 image
nn.ConvTranspose2d(in_channels, out_channels, kernel_size=3, stride=2, padding=0)

# yields a 13 x 13 image
nn.ConvTranspose2d(in_channels, out_channels, kernel_size=5, stride=2, padding=0)

# yields a 15 x 15 image
nn.ConvTranspose2d(in_channels, out_channels, kernel_size=3, stride=3, padding=0)

I'm sure there's a pretty simple math equation involving w, h, k, s, p but I can't find it in the documentation and I haven't been able to derive it myself. Normally I wouldn't ask for a math equation, but it completely affects the ability of a CNN to compile and generate the correct size. Thanks in advance!


title: PyTorch CrossEntropyLoss minibatch size

body:
I am trying to build a simple example with zero CrossEntropyLoss, with three classes and minibatch size = 1 :

ce_loss = nn.CrossEntropyLoss()
outputs = torch.tensor([[1,0,3]], dtype=float)
targets = torch.tensor([1,0,3], dtype=torch.int64)
loss = ce_loss(outputs, targets)

Which results in error:

ValueError: Expected input batch_size (1) to match target batch_size (3).

What is wrong and where does 'target batch_size (3)' comes from?

Update 1

According to my understanding of PyTorch docs I try the following cases. Please indicate where I am wrong.

Example 1:

outputs = torch.tensor([[1,1]], dtype=float)
targets = torch.tensor([1], dtype=torch.int64)
loss = ce_loss(outputs, targets)

Output:

loss: 0.6931471805599453

Questions:

1) I have one class equal to 1 and two predictions for this class each equal to 1 , correct?

2) Why non-zero loss?

Example 2:

outputs = torch.tensor([[1]], dtype=float)
targets = torch.tensor([1], dtype=torch.int64)
loss = ce_loss(outputs, targets)

Error:

IndexError: Target 1 is out of bounds.

Questions:

1) I have one class equal to 1 and one prediction for this class equal to 1,  correct?

2) Why index error?

Example 3:

outputs = torch.tensor([[1,1],[2,2]], dtype=float)
targets = torch.tensor([1, 2], dtype=torch.int64)
loss = ce_loss(outputs, targets)

Error:

IndexError: Target 2 is out of bounds.

Questions:

1) I have two classes equal to 1 and 2 two predictions for these classes equal to 1 and 2 (batch size = 2), correct?

2) Why index error?

--- More questions: ----

What shape should outputs and targets have for the following cases:

1) Binary classification, one time step, one value predicted by one output neuron, batch size = 1

2) Binary classification, five different time steps, 5 values predicted by one output neuron,  batch size = 5

3) Seven classes, one time step, 7 values predicted by 7 output neurons, batch size = 1

4) Seven classes, five different time steps, 5 sets of 7 values are predicted by 7 output neurons, batch size = 5


title: Multi class classifcation with Pytorch

body:
I'm new with Pytorch and I need a clarification on multiclass classification.

I'm fine-tuning the DenseNet neural network, so it can recognize 3 different classes.

Because it's a multiclass problem, I have to replace the classification layer in this way:

kernelCount = self.densenet121.classifier.in_features
self.densenet121.classifier = nn.Sequential(nn.Linear(kernelCount, 3), nn.Softmax(dim=1))

And use CrossEntropyLoss as the loss function:

loss = torch.nn.CrossEntropyLoss(reduction='mean')

By reading on Pytorch forum, I found that CrossEntropyLoss applys the softmax function on the output of the neural network. Is this true? Should I remove the Softmax activation function from the structure of the network?

And what about the test phase? If it's included, I have to call the softmax function on the output of the model?

Thanks in advance for your help.


title: How can I swap axis in a torch tensor?

body:
I have a torch tensor of size torch.Size([1, 128, 56, 128])
1 is channel, 128 is the width, and height. 56 are the stacks of images.
How can I resize it to torch.Size([1, 56, 128, 128]) ?


I am trying to install torch with CUDA support.
Here is the result of my collect_env.py script:
PyTorch version: 1.7.1+cu101
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.9 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: GeForce GTX 1080
Nvidia driver version: 460.39
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.7.1+cu101
[pip3] torchaudio==0.7.2
[pip3] torchvision==0.8.2+cu101
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.1.243             h6bb024c_0
[conda] mkl                       2020.2                      256
[conda] mkl-service               2.3.0            py39he8ac12f_0
[conda] mkl_fft                   1.3.0            py39h54f3939_0
[conda] mkl_random                1.0.2            py39h63df603_0
[conda] numpy                     1.19.2           py39h89c1606_0
[conda] numpy-base                1.19.2           py39h2ae0177_0
[conda] torch                     1.7.1+cu101              pypi_0    pypi
[conda] torchaudio                0.7.2                    pypi_0    pypi
[conda] torchvision               0.8.2+cu101              pypi_0    pypi

Process finished with exit code 0

Here is the output of nvcc - V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Finally, here is the output of nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.39       Driver Version: 460.39       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:01:00.0  On |                  N/A |
|  0%   52C    P0    46W / 180W |    624MiB /  8116MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A       873      G   /usr/lib/xorg/Xorg                101MiB |
|    0   N/A  N/A      1407      G   /usr/lib/xorg/Xorg                419MiB |
|    0   N/A  N/A      2029      G   ...AAAAAAAAA= --shared-files       90MiB |
+-----------------------------------------------------------------------------+

However, when I try to run
print(torch.cuda.is_available())

I get the following error:
UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0

I have performed a reboot, and I have followed the post-installation steps as detailed in here


title: How to convert a tensorflow model to a pytorch model?

body:
I'm new to pytorch. Here's an architecture of a tensorflow model and I'd like to convert it into a pytorch model.

I have done most of the codes but am confused about a few places.

1) In tensorflow, the Conv2D function takes filter as an input. However, in pytorch, the function takes the size of input channels and output channels as inputs. So how do I find the equivalent number of input channels and output channels, provided with the size of the filter.

2) In tensorflow, the dense layer has a parameter called 'nodes'. However, in pytorch, the same layer has 2 different inputs (the size of the input parameters and size of the targeted parameters), how do I determine them based on the number of the nodes.

Here's the tensorflow code.

from keras.utils import to_categorical
from keras.models import Sequential, load_model
from keras.layers import Conv2D, MaxPool2D, Dense, Flatten, Dropout

model = Sequential()
model.add(Conv2D(filters=32, kernel_size=(5,5), activation='relu', input_shape=X_train.shape[1:]))
model.add(Conv2D(filters=32, kernel_size=(5,5), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Dropout(rate=0.25))
model.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))
model.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Dropout(rate=0.25))
model.add(Flatten())
model.add(Dense(256, activation='relu'))
model.add(Dropout(rate=0.5))
model.add(Dense(43, activation='softmax'))

Here's my code.:

import torch.nn.functional as F
import torch

# The network should inherit from the nn.Module
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # Define 2D convolution layers
        # 3: input channels, 32: output channels, 5: kernel size, 1: stride
        self.conv1 = nn.Conv2d(3, 32, 5, 1)   # The size of input channel is 3 because all images are coloured
        self.conv2 = nn.Conv2d(32, 64, 5, 1)
        self.conv3 = nn.Conv2d(64, 128, 3, 1)
        self.conv3 = nn.Conv2d(128, 256, 3, 1)
        # It will 'filter' out some of the input by the probability(assign zero)
        self.dropout1 = nn.Dropout2d(0.25)
        self.dropout2 = nn.Dropout2d(0.5)
        # Fully connected layer: input size, output size
        self.fc1 = nn.Linear(36864, 128)
        self.fc2 = nn.Linear(128, 10)

    # forward() link all layers together,
    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2)
        x = self.dropout1(x)
        x = self.conv3(x)
        x = F.relu(x)
        x = self.conv4(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2)
        x = self.dropout1(x)
        x = torch.flatten(x, 1)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.dropout2(x)
        x = self.fc2(x)
        output = F.log_softmax(x, dim=1)
        return output

Thanks in advance!


title: How to use the Detectron2 .pth model for prediction from storage..?

body:
I have trained the Detectron2 model on the google colab server free server.
from detectron2.engine import DefaultTrainer
from detectron2.config import get_cfg
import os

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-
InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ("Dataset_train",)
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-
InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 2800
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 2
os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

This above code creates an "output" folder in which I have 4 files:
model_final.pth,  metrics.json, last_checkpoint and events.out.file

I can use this model for prediction using
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5
cfg.DATASETS.TEST = ("Datset_test")
predictor = DefaultPredictor(cfg)

Everything works fine when colab local session is not expired.
Problem:

When I mount this output folder somewhere else than the model not imported.
How to use .pth saved model for predictions?
How to reuse detectron2 trained model for prediction..?


title: How can i add a Bi-LSTM layer on top of bert model?

body:
I'm using pytorch and I'm using the base pretrained bert to classify sentences for hate speech.
I want to implement a Bi-LSTM layer that takes as an input all outputs of the latest
transformer encoder from the bert model as a new model (class that implements nn.Module), and i got confused with the nn.LSTM parameters.
I tokenized the data using
bert = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=int(data['class'].nunique()),output_attentions=False,output_hidden_states=False)

My data-set has 2 columns: class(label), sentence.
Can someone help me with this?
Thank you in advance.
Edit:
Also, after processing the input in the bi-lstm, the network sends the  final hidden state to a fully connected network that performs classication using the softmax activation function. how can I do that ?


title: Custom C++ extension : torch/torch.h not found

body:
I am following this tutorial to create a C++ extension for Pytorch. My C++ code is giving following error :

test.cpp:3:10: fatal error: torch/torch.h: No such file or directory
 #include <torch/torch.h>

How to get torch.h header file ? Is there some pytorch-dev version?


title: Can't init the weights of my neural network PyTorch

body:
I can't initialize the weights with the function MyNet.apply(init_weights).

These are my functions:

def init_weights(net):
    if type(net) == torch.nn.Module:
        torch.nn.init.kaiming_uniform_(net.weight)
        net.bias.data.fill_(0.01)  # tots els bias a 0.01

My neural net is the following:

class NeuralNet(torch.nn.Module):
    def __init__(self):
        super().__init__() # Necessary for torch to detect this class as trainable
        # Here define network architecture
        self.layer1 = torch.nn.Linear(28**2, 32).to(device) # Linear layer with 32 neurons
        self.layer2 = torch.nn.Linear(32, 64).to(device) # Linear layer with 64 neurons
        self.layer3 = torch.nn.Linear(64, 128).to(device)  # Linear layer with 128 neurons
        self.output = torch.nn.Linear(128, 1).to(device) # Linear layer with 1 output neuron (binary output)

    def forward(self, x):
        # Here define architecture behavior
        x = torch.sigmoid(self.layer1(x)).to(device) # x = torch.nn.functional.relu(self.layer1(x))
        x = torch.sigmoid(self.layer2(x)).to(device)
        x = torch.sigmoid(self.layer3(x)).to(device)

        return torch.sigmoid(self.output(x)).to(device) # Binary output

The type(net) prints as linear so it never gets inside the if statement, and if I remove it produces the following error:

  AttributeError: 'NeuralNet' object has no attribute 'weight'


title: PyTorch C++ FrontEnd returning multiple Tensors in forward

body:
I was wonder how can I return a std::vector<torch::Tensor> in my forward pass of a Module Class,
I read about the Macro of FORWARD_HAS_DEFAULT_ARGS in the docs, but didn’t really
understand how to use it, and also how to use it for making it possible to return a vector in return.
Thank you in advance.


title: torch.rfft - fft-based convolution creating different output than spatial convolution

body:
I implemented FFT-based convolution in Pytorch and compared the result with spatial convolution via conv2d() function. The convolution filter used is an average filter. The conv2d() function produced smoothened output due to average filtering as expected but the fft-based convolution returned a more blurry output.
I have attached the code and outputs here -

spatial convolution -

from PIL import Image, ImageOps
import torch
from matplotlib import pyplot as plt
from torchvision.transforms import ToTensor
import torch.nn.functional as F
import numpy as np

im = Image.open("/kaggle/input/tiger.jpg")
im = im.resize((256,256))
gray_im = im.convert('L')
gray_im = ToTensor()(gray_im)
gray_im = gray_im.squeeze()

fil = torch.tensor([[1/9,1/9,1/9],[1/9,1/9,1/9],[1/9,1/9,1/9]])

conv_gray_im = gray_im.unsqueeze(0).unsqueeze(0)
conv_fil = fil.unsqueeze(0).unsqueeze(0)

conv_op = F.conv2d(conv_gray_im,conv_fil)

conv_op = conv_op.squeeze()

plt.figure()
plt.imshow(conv_op, cmap='gray')

FFT-based convolution -

def fftshift(image):
    sh = image.shape
    x = np.arange(0, sh[2], 1)
    y = np.arange(0, sh[3], 1)
    xm, ym  = np.meshgrid(x,y)
    shifter = (-1)**(xm + ym)
    shifter = torch.from_numpy(shifter)
    return image*shifter

shift_im = fftshift(conv_gray_im)
padded_fil = F.pad(conv_fil, (0, gray_im.shape[0]-fil.shape[0], 0, gray_im.shape[1]-fil.shape[1]))
shift_fil = fftshift(padded_fil)
fft_shift_im = torch.rfft(shift_im, 2, onesided=False)
fft_shift_fil = torch.rfft(shift_fil, 2, onesided=False)
shift_prod = fft_shift_im*fft_shift_fil
shift_fft_conv = fftshift(torch.irfft(shift_prod, 2, onesided=False))

fft_op = shift_fft_conv.squeeze()
plt.figure('shifted fft')
plt.imshow(fft_op, cmap='gray')

original image -

spatial convolution output -

fft-based convolution output -

Could someone kindly explain the issue?


title: Pytorch is not using GPU even it detects the GPU

body:
I made my windows 10 jupyter notebook as a server and running some trains on it.

I've installed CUDA 9.0 and cuDNN properly, and python detects the GPU. This is what I've got on the anaconda prompt.

>>> torch.cuda.get_device_name(0)
'GeForce GTX 1070'

And I also placed my model and tensors on cuda by .cuda()

model = LogPPredictor(1, 58, 64, 128, 1, 'gsc')

if torch.cuda.is_available():
    torch.set_default_tensor_type(torch.cuda.DoubleTensor)
    model.cuda()
else:
    torch.set_default_tensor_type(torch.FloatTensor)

list_train_loss = list()
list_val_loss = list()
acc = 0
mse = 0

optimizer = args.optim(model.parameters(),
                       lr=args.lr,
                       weight_decay=args.l2_coef)

data_train = DataLoader(args.dict_partition['train'],
                        batch_size=args.batch_size,
                        pin_memory=True,
                        shuffle=args.shuffle)

data_val = DataLoader(args.dict_partition['val'],
                     batch_size=args.batch_size,
                     pin_memory=True,
                     shuffle=args.shuffle)

for epoch in tqdm_notebook(range(args.epoch), desc='Epoch'):
    model.train()
    epoch_train_loss = 0
    for i, batch in enumerate(data_train):
        list_feature = torch.tensor(batch[0]).cuda()
        list_adj = torch.tensor(batch[1]).cuda()
        list_logP = torch.tensor(batch[2]).cuda()
        list_logP = list_logP.view(-1,1)

        optimizer.zero_grad()
        list_pred_logP = model(list_feature, list_adj)
        list_pred_logP.require_grad = False
        train_loss = args.criterion(list_pred_logP, list_logP)
        epoch_train_loss += train_loss.item()
        train_loss.backward()
        optimizer.step()

    list_train_loss.append(epoch_train_loss/len(data_train))

    model.eval()
    epoch_val_loss = 0
    with torch.no_grad():
        for i, batch in enumerate(data_val):
            list_feature = torch.tensor(batch[0]).cuda()
            list_adj = torch.tensor(batch[1]).cuda()
            list_logP = torch.tensor(batch[2]).cuda()
            list_logP = list_logP.view(-1,1)

            list_pred_logP = model(list_feature, list_adj)
            val_loss = args.criterion(list_pred_logP, list_logP)
            epoch_val_loss += val_loss.item()

    list_val_loss.append(epoch_val_loss/len(data_val))

data_test = DataLoader(args.dict_partition['test'],
                   batch_size=args.batch_size,
                   pin_memory=True,
                   shuffle=args.shuffle)

model.eval()
with torch.no_grad():
    logP_total = list()
    pred_logP_total = list()
    for i, batch in enumerate(data_val):
        list_feature = torch.tensor(batch[0]).cuda()
        list_adj = torch.tensor(batch[1]).cuda()
        list_logP = torch.tensor(batch[2]).cuda()
        logP_total += list_logP.tolist()
        list_logP = list_logP.view(-1,1)

    list_pred_logP = model(list_feature, list_adj)

    pred_logP_total += list_pred_logP.tolist()

mse = mean_squared_error(logP_total, pred_logP_total)

But on the Process Manager of Windows, whenever I start training, only CPU usage goes up to 25% and GPU usage remains 0. How can I fix this???


title: Convert dictionary to tensors, and back

body:
I started with a dictionary object:

{"train": [{"input": [[3, 1, 2], [3, 1, 2], [3, 1, 2]], "output": [[4, 5, 6], [4, 5, 6], [4, 5, 6]]}, {"input": [[2, 3, 8], [2, 3, 8], [2, 3, 8]], "output": [[6, 4, 9], [6, 4, 9], [6, 4, 9]]}]}

I then want to add padding to each input and output object in a loop. So I converted each input and output to a tensor so I could then use  F.pad to add padding. Result of the first input:

tensor([[0, 0, 0, 0, 0],
        [0, 3, 1, 2, 0],
        [0, 3, 1, 2, 0],
        [0, 3, 1, 2, 0],
        [0, 0, 0, 0, 0]]).

result of the first output:

tensor([[0, 0, 0, 0, 0],
        [0, 4, 5, 6, 0],
        [0, 4, 5, 6, 0],
        [0, 4, 5, 6, 0],
        [0, 0, 0, 0, 0]])

So that works fine.  Now, I want to reconstruct the generated tensors into the same form as the original dictionary, so that it will look like this:

{"train": [{"input": [[0, 0, 0, 0, 0], [0, 3, 1, 2, 0], [0, 3, 1, 2, 0], [0, 3, 1, 2, 0], [0, 0, 0, 0, 0]], "output": [[0, 0, 0, 0, 0], [0, 4, 5, 6, 0], [0, 4, 5, 6, 0], [0, 4, 5, 6, 0], [0, 0, 0, 0, 0]]}, {"input": [[0, 0, 0, 0, 0],
        [0, 2, 3, 8, 0],
        [0, 2, 3, 8, 0],
        [0, 2, 3, 8, 0],
        [0, 0, 0, 0, 0]], "output": [[0, 0, 0, 0, 0],
        [0, 6, 4, 9, 0],
        [0, 6, 4, 9, 0],
        [0, 6, 4, 9, 0],
        [0, 0, 0, 0, 0]]}]}

I can see a string concatenation way that might work:

composedString = '{"train": [{"input": ' + tensor1 + tensor2

or something like that. But given that there are different numbers of elements in the various arrays, it seems like a loop nightmare. I'm thinking there's got to be a better way. Anyone know what it is?

Thanks.


title: Why does the pytorch model perform poorly after setting eval()?

body:
I used pytorch to build a segmentation model that uses the BatchNormalization layer. I found that when I set model.eval() on the test, the test result will be 0. If I don't set model.eval(), it will perform well.

I tried to search for related questions, but I got the conclusion that model.eval() can fix the parameters of BN, but I am still confused about how to solve this problem.

My batchsize is 1 and this is my model:

import torch
import torch.nn as nn

class Encode_Block(nn.Module):
    def __init__(self, in_feat, out_feat):
        super(Encode_Block, self).__init__()

        self.conv1 = Res_Block(in_feat, out_feat)
        self.conv2 = Res_Block_identity(out_feat, out_feat)

    def forward(self, inputs):
        outputs = self.conv1(inputs)
        outputs = self.conv2(outputs)
        return outputs

class Decode_Block(nn.Module):
    def __init__(self, in_feat, out_feat):
        super(Decode_Block, self).__init__()

        self.conv1 = Res_Block(in_feat, out_feat)
        self.conv2 = Res_Block_identity(out_feat, out_feat)

    def forward(self, inputs):
        outputs = self.conv1(inputs)
        outputs = self.conv2(outputs)
        return outputs

class Conv_Block(nn.Module):
    def __init__(self, in_feat, out_feat):
        super(Conv_Block, self).__init__()

        self.conv1 = nn.Sequential(
            nn.Conv2d(in_feat, out_feat, kernel_size=3, stride=1, padding=1),
            nn.LeakyReLU(),
        )

    def forward(self, inputs):
        outputs = self.conv1(inputs)
        return outputs

def conv3x3(in_planes, out_planes, stride=1):
    """3x3 convolution with padding"""
    return nn.Conv2d(
        in_planes, out_planes, kernel_size=3, stride=stride, padding=1, bias=False
    )

def conv1x1(in_planes, out_planes, stride=1):
    """1x1 convolution with padding"""
    return nn.Conv2d(in_planes, out_planes, kernel_size=1, stride=stride, bias=False)

class Res_Block(nn.Module):
    def __init__(self, inplanes, planes, stride=1):
        super(Res_Block, self).__init__()
        self.conv_input = conv1x1(inplanes, planes)
        self.conv1 = conv3x3(inplanes, planes, stride)
        self.bn = nn.BatchNorm2d(planes)
        self.relu = nn.LeakyReLU(inplace=True)
        self.conv2 = conv3x3(planes, planes)
        self.conv3 = conv1x1(planes, planes)
        self.stride = stride

    def forward(self, x):
        residual = self.conv_input(x)

        out = self.conv1(x)
        out = self.bn(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn(out)
        out = self.relu(out)

        out = self.conv3(out)
        out = self.bn(out)

        out += residual
        out = self.relu(out)

        return out

class Res_Block_identity(nn.Module):
    def __init__(self, inplanes, planes, stride=1):
        super(Res_Block_identity, self).__init__()
        self.conv1 = conv3x3(inplanes, planes, stride)
        self.bn = nn.BatchNorm2d(planes)
        self.relu = nn.LeakyReLU(inplace=True)
        self.conv2 = conv3x3(planes, planes)
        self.conv3 = conv1x1(planes, planes)
        self.stride = stride

    def forward(self, x):
        residual = x

        out = self.conv1(x)
        out = self.bn(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn(out)
        out = self.relu(out)

        out = self.conv3(out)
        out = self.bn(out)

        out += residual
        out = self.relu(out)

        return out

class UpConcat(nn.Module):
    def __init__(self, in_feat, out_feat):
        super(UpConcat, self).__init__()

        self.de_conv = nn.ConvTranspose2d(in_feat, out_feat, kernel_size=2, stride=2)

    def forward(self, inputs, down_outputs):
        outputs = self.de_conv(inputs)
        out = torch.cat([down_outputs, outputs], 1)
        return out

class Res_UNet(nn.Module):
    def __init__(self, num_channels=1, num_classes=1):
        super(Res_UNet, self).__init__()
        flt = 64
        self.down1 = Encode_Block(num_channels, flt)
        self.down2 = Encode_Block(flt, flt * 2)
        self.down3 = Encode_Block(flt * 2, flt * 4)
        self.down4 = Encode_Block(flt * 4, flt * 8)
        self.down_pool = nn.MaxPool2d(kernel_size=2)
        self.bottom = Encode_Block(flt * 8, flt * 16)
        self.up_cat1 = UpConcat(flt * 16, flt * 8)
        self.up_conv1 = Decode_Block(flt * 16, flt * 8)
        self.up_cat2 = UpConcat(flt * 8, flt * 4)
        self.up_conv2 = Decode_Block(flt * 8, flt * 4)
        self.up_cat3 = UpConcat(flt * 4, flt * 2)
        self.up_conv3 = Decode_Block(flt * 4, flt * 2)
        self.up_cat4 = UpConcat(flt * 2, flt)
        self.up_conv4 = Decode_Block(flt * 2, flt)
        self.final = nn.Sequential(
            nn.Conv2d(flt, num_classes, kernel_size=1), nn.Sigmoid()
        )

    def forward(self, inputs):
        down1_feat = self.down1(inputs)
        pool1_feat = self.down_pool(down1_feat)
        down2_feat = self.down2(pool1_feat)
        pool2_feat = self.down_pool(down2_feat)
        down3_feat = self.down3(pool2_feat)
        pool3_feat = self.down_pool(down3_feat)
        down4_feat = self.down4(pool3_feat)
        pool4_feat = self.down_pool(down4_feat)

        bottom_feat = self.bottom(pool4_feat)

        up1_feat = self.up_cat1(bottom_feat, down4_feat)
        up1_feat = self.up_conv1(up1_feat)
        up2_feat = self.up_cat2(up1_feat, down3_feat)
        up2_feat = self.up_conv2(up2_feat)
        up3_feat = self.up_cat3(up2_feat, down2_feat)
        up3_feat = self.up_conv3(up3_feat)
        up4_feat = self.up_cat4(up3_feat, down1_feat)
        up4_feat = self.up_conv4(up4_feat)

        outputs = self.final(up4_feat)

        return outputs

The model completely fails to segmentation after setting model.eval(), but the model is good after model.eval() is removed. I am confused about this, and is model.eval() necessary in the test?


title: How to translate or convert code from Python Pytorch to C++ Libtorch

body:
I couldn't find the equivalent C++ calls in Libtorch (Pytorch C++ Frontend) for my Python Pytorch code.

The documentation according to my searchs (Pytorch Discuss) does not exist yet for my code. I wonder if someone can guide me with the following  pieces (bellow).

I cutted pieces where I've been having more crashes (wrong usage) of Libtorch C++.

import torch as th

th.set_grad_enabled(False)
...
X = th.zeros((nobs, 3+p), device=dev, dtype=th.float32)
y = th.tensor(indata, device=dev, dtype=th.float32)
diffilter = th.tensor([-1., 1.], device=dev, dtype=th.float32).view(1, 1, 2)
dy = th.conv1d(y.view(1, 1, -1), diffilter).view(-1)
z = dy[p:].clone()
...
# X matrix
X[:, 0] = 1
X[:, 1] = th.arange(p+1, n)
X[:, 2] = y[p:-1]
...
# master X
Xm = th.zeros((nobsadf, 3+p), device=th.device('cpu'), dtype=th.float32)
...
# batch matrix, vector and observations
Xbt = th.zeros(batch_size, adfs_count, nobsadf, (3+p), device=th.device('cpu'), dtype=th.float32)
...
t = 0 # start line for master main X OLS matrix/ z vector
for i in range(nbatchs):
    for j in range(batch_size): # assembly batch_size matrixes
        Xm[:] = X[t:t+nobsadf]
        ...
        Xbt[j, :, :, :] = Xm.repeat(adfs_count, 1).view(adfs_count, nobsadf, (3+p))
        for k in range(adfs_count):
            Xbt[j, k, :k, :] = 0
            nobt[j, k] = float(nobsadf-k-(p+3))


title: Hugging face - RuntimeError: Caught RuntimeError in replica 0 on device 0 on Azure Databricks

body:
How do I run the run_language_modeling.py script from hugging face using the pretrained roberta case model to fine-tune  using my own data on the Azure databricks with a GPU cluster.
Using Transformer version 2.9.1 and 3.0 .
Python 3.6
Torch `1.5.0
torchvision 0.6
This is the script I ran below on Azure databricks
%run '/dbfs/FileStore/tables/dev/run_language_modeling.py' \
  --output_dir='/dbfs/FileStore/tables/final_train/models/roberta_base_reduce_n' \
  --model_type=roberta \
  --model_name_or_path=roberta-base \
  --do_train \
  --num_train_epochs 5 \
  --train_data_file='/dbfs/FileStore/tables/final_train/train_data/all_data_desc_list_full.txt' \
  --mlm

This is the error I get after running the above command.
/dbfs/FileStore/tables/dev/run_language_modeling.py in <module>
   279
   280 if __name__ == "__main__":
--> 281     main()

/dbfs/FileStore/tables/dev/run_language_modeling.py in main()
   243             else None
   244         )
--> 245         trainer.train(model_path=model_path)
   246         trainer.save_model()
   247         # For convenience, we also re-save the tokenizer to the same directory,

/databricks/python/lib/python3.7/site-packages/transformers/trainer.py in train(self, model_path)
   497                     continue
   498
--> 499                 tr_loss += self._training_step(model, inputs, optimizer)
   500
   501                 if (step + 1) % self.args.gradient_accumulation_steps == 0 or (

/databricks/python/lib/python3.7/site-packages/transformers/trainer.py in _training_step(self, model, inputs, optimizer)
   620             inputs["mems"] = self._past
   621
--> 622         outputs = model(**inputs)
   623         loss = outputs[0]  # model outputs are always tuple in transformers (see doc)
   624

/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
   548             result = self._slow_forward(*input, **kwargs)
   549         else:
--> 550             result = self.forward(*input, **kwargs)
   551         for hook in self._forward_hooks.values():
   552             hook_result = hook(self, input, result)

/databricks/python/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
   153             return self.module(*inputs[0], **kwargs[0])
   154         replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
--> 155         outputs = self.parallel_apply(replicas, inputs, kwargs)
   156         return self.gather(outputs, self.output_device)
   157

/databricks/python/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in parallel_apply(self, replicas, inputs, kwargs)
   163
   164     def parallel_apply(self, replicas, inputs, kwargs):
--> 165         return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
   166
   167     def gather(self, outputs, output_device):

/databricks/python/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py in parallel_apply(modules, inputs, kwargs_tup, devices)
    83         output = results[i]
    84         if isinstance(output, ExceptionWrapper):
---> 85             output.reraise()
    86         outputs.append(output)
    87     return outputs

/databricks/python/lib/python3.7/site-packages/torch/_utils.py in reraise(self)
   393             # (https://bugs.python.org/issue2651), so we work around it.
   394             msg = KeyErrorMessage(msg)
--> 395         raise self.exc_type(msg)

RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
 File "/databricks/python/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
   output = module(*input, **kwargs)
 File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
   result = self.forward(*input, **kwargs)
 File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 239, in forward
   output_hidden_states=output_hidden_states,
 File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
   result = self.forward(*input, **kwargs)
 File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 762, in forward
   output_hidden_states=output_hidden_states,
 File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
   result = self.forward(*input, **kwargs)
 File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 439, in forward
   output_attentions,
 File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
   result = self.forward(*input, **kwargs)
 File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 371, in forward
   hidden_states, attention_mask, head_mask, output_attentions=output_attentions,
 File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
   result = self.forward(*input, **kwargs)
 File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 315, in forward
   hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, output_attentions,
 File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
   result = self.forward(*input, **kwargs)
 File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 240, in forward
   attention_scores = attention_scores / math.sqrt(self.attention_head_size)
RuntimeError: CUDA out of memory. Tried to allocate 96.00 MiB (GPU 0; 11.17 GiB total capacity; 10.68 GiB already allocated; 95.31 MiB free; 10.77 GiB reserved in total by PyTorch)```

Please how do I resolve this


title: RuntimeError: cuda runtime error (100) . The gpu is enabled but still giving error

body:
I am new to google Colab and pyTorch. I am running a pytorch model but it is giving me the Cuda Runtime Error in google Colab. My gpu is enabled on google colab but it is still giving error, The description of gpu is available in the image below. Can anyone please help me out?
torch GPU
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=47 error=100 : no CUDA-capable device is detected
Traceback (most recent call last):
  File "run.py", line 338, in <module>
    main()
  File "run.py", line 303, in main
    model = model.cuda()
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 458, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 354, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 354, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 354, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 376, in _apply
    param_applied = fn(param)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 458, in <lambda>
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 190, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:47
Read prediction from logs/logs_sparc_editsql/valid_use_predicted_queries_predictions.json


title: what does nn.Linear() do in pytorch's last, and why is it necessary?

body:
I am working with some code that trains an lstm to generate sequences. After training the model, the lstm() method is called:

x = some_input
lstm_output, (h_n, c_n) = lstm(x, hc)
funcc = nn.Linear(in_features=lstm_num_hidden,
                  output_features=vocab_size,
                  bias=True)
func_output = func(lstm_output)

I've looked at the documentation for nn.Linear() but I still don't understand what this transformation is doing and why it is necessary. If the lstm has already been trained, then the output it gives should already have a pre-established dimensionality. This output (lstm_output) would be the generated sequence, or in my case an array of vectors. Am I missing something here?


title: How to load a .bin model file?

body:
Recently, I have been working on a age and gender recognition problem. Upon searching for already existing solution I came across an Open Vino Toolkit.
https://docs.openvinotoolkit.org/

They uses many models for deep learning tasks and one of them is
https://docs.openvinotoolkit.org/latest/_models_intel_age_gender_recognition_retail_0013_description_age_gender_recognition_retail_0013.html

Now these models are free to use and can be easily downloaded. I downloaded the model zip folder and it contains one .xml file. this file shows the layers and dimensions in each layers and other useful things about the structure of the model.
The other file is .bin file and this (i guess) is the real model file but I dont how to load a model saved as .bin file.

The image attached contain the content of the .xml file
Need help.


title: calculate accuracy for each class using CNN and pytorch

body:
I Can calculate accuracy after each epoch using this code . But, I want to calculate the accuracy for each class at the end . how can i do that?
I have two folders train and val . each folder has 7 folders of 7 different classes. the train folder is used for training .otherwise val folder is used for testing
  def train_model(model, criterion, optimizer, lr_scheduler, num_epochs=25):
    since = time.time()

    best_model = model
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                mode='train'
                optimizer = lr_scheduler(optimizer, epoch)
                model.train()  # Set model to training mode
            else:
                model.eval()
                mode='val'

            running_loss = 0.0
            running_corrects = 0

            counter=0
            # Iterate over data.
            for data in dset_loaders[phase]:
                inputs, labels = data
                print(inputs.size())
                # wrap them in Variable
                if use_gpu:
                    try:
                        inputs, labels = Variable(inputs.float().cuda()),
                        Variable(labels.long().cuda())
                    except:
                        print(inputs,labels)
                else:
                    inputs, labels = Variable(inputs), Variable(labels)

                # Set gradient to zero to delete history of computations in previous epoch. Track operations so that differentiation can be done automatically.
                optimizer.zero_grad()
                outputs = model(inputs)
                _, preds = torch.max(outputs.data, 1)

                loss = criterion(outputs, labels)
                # print('loss done')
                # Just so that you can keep track that something's happening and don't feel like the program isn't running.
                # if counter%10==0:
                #     print("Reached iteration ",counter)
                counter+=1

                # backward + optimize only if in training phase
                if phase == 'train':
                    # print('loss backward')
                    loss.backward()
                    # print('done loss backward')
                    optimizer.step()
                    # print('done optim')
                # print evaluation statistics
                try:
                    # running_loss += loss.data[0]
                    running_loss += loss.item()
                    # print(labels.data)
                    # print(preds)
                    running_corrects += torch.sum(preds == labels.data)
                    # print('running correct =',running_corrects)
                except:
                    print('unexpected error, could not calculate loss or do a sum.')
            print('trying epoch loss')
            epoch_loss = running_loss / dset_sizes[phase]
            epoch_acc = running_corrects.item() / float(dset_sizes[phase])
            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val':
                if USE_TENSORBOARD:
                    foo.add_scalar_value('epoch_loss',epoch_loss,step=epoch)
                    foo.add_scalar_value('epoch_acc',epoch_acc,step=epoch)
                if epoch_acc > best_acc:
                    best_acc = epoch_acc
                    best_model = copy.deepcopy(model)
                    print('new best accuracy = ',best_acc)
    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))
    print('returning and looping back')
    return best_model

def exp_lr_scheduler(optimizer, epoch, init_lr=BASE_LR, lr_decay_epoch=EPOCH_DECAY):
    """Decay learning rate by a factor of DECAY_WEIGHT every lr_decay_epoch epochs."""
    lr = init_lr * (DECAY_WEIGHT**(epoch // lr_decay_epoch))

    if epoch % lr_decay_epoch == 0:
        print('LR is set to {}'.format(lr))

    for param_group in optimizer.param_groups:
        param_group['lr'] = lr

    return optimizer


title: Pytorch: Can’t load images using ImageFolder

body:
I’m trying to load images using “ImageFolder”.

data_dir = './train_dog'          # directory structure is
train_dog/image
dset = datasets.ImageFolder(data_dir, transform)
train_loader = torch.utils.data.DataLoader(dset, batch_size=128, shuffle=True)

However, it seems not working. So I checked the stored data as below

print dset[0][0]

Then it shows only 3 tensors(size 64x64).

[torch.FloatTensor of size 3x64x64]

There are more than 10,000 images in the folder. How come it can’t store all data?


title: why is torch.nn.Sigmoid() behaves different than torch.sigmoid?

body:
I am building a RL agent using a neural network to aproximate the Q values of the model.
My first implementation was something like this:

self.l1 = nn.Linear(model.state_dim, 128)
self.l2 = nn.Linear(128, 256)
self.l3 = nn.Linear(256, 256)
self.l4 = nn.Linear(256, model.action_dim)

def forward(self, x):
     x = self.l1(x)
     x = torch.sigmoid(x)
     x = self.l2(x)
     x = torch.sigmoid(x)
     x = self.l3(x)
     x = torch.sigmoid(x)
     return self.l4(x)

I got some output like this over a series of training episodes:

This output made a lot of sense, the Q values should be negative, because the rewards outputted from the model are in the range [-infinity , 0]

Then I was looking the docs of pytorch, and found a much cleaner implementation of this network:

self.fc = nn.Sequential(
            nn.Linear(model.state_dim, 128),
            nn.Sigmoid(),
            nn.Linear(128, 256),
            nn.Sigmoid(),
            nn.Linear(256, 256),
            nn.Sigmoid(),
            nn.Linear(256, model.action_dim)
        )

def forward(self, x):
     return self.fc(x)

This output made no sense, because Q values should be heading toward a negative number, if its like the Sequential model how I configured it could not produce any negative number in the output layer, but to me, the configurations are exactly the same. I read that the nn.Sequential implementation is much more GPU-capable, but I don't want to sacrifice the model accuracy in favor of the additional performance. Why are the two implementations behaving so differently?


title: Pytorch: Set Block-Diagonal Matrix Efficiently?

body:
I have a Tensor A of size [N x 3 x 3], and a Matrix B of size [N*3 x N*3]
I want to copy the contents of A -> B, so that the diagonal elements are filled up basically, and I want to do this efficiently:
It should kind of fill up B to look something filled like this:

So each [i,3,3] fills into each [3x3] part in B diagonally down the line.
How do I do this? As efficiently as possible as this is for a real time application. I could write a CUDA kernel to do this, but I would prefer to do it with some special Pytorch function


title: How to access pytorch model parameters by index

body:
If I have network with let's say 10 layers including biases, how can I access its i'th layer parameters just by index?

Currently, what I am doing is something like this

for parameter in myModel.parameters():
    parameter.data /= 5

How could I access parameter.data with an index? For example I'd like to access 9th layer without iterating, such as myModel.parameter.data[8] or something similar.