title: How to attribute multiple images to the same observation in a neural network?

body:
I have a data set of houses with each house containing 5 - 8 images each.  Each image is a different aspect of the home (roof, right side, left side, etc).  The target variable is binary per house.  How would I feed in each record's multiple images in a way that the neural net knows which record each image is attributed to?

Any ideas in Keras or pytorch is appreciated.

Now that I’m back at my work desk, maybe this record would help. The following is one actual record from the tabular dataset.

ID: 5000686
Age of Home: 34
Zip Code: 49508
Home Value: 200000
Number of families living in home: 1
Target: 1

So for reach photo, the title is the ‘id-photo number.’ I have a csv with three columns that shows the location on my hard drive, full photo ID, and meta label. I have 8 to 28 photos per record.

5000686-136279724.jpg / meta-label = dwelling
5000686-136279725.jpg / meta-label = dwelling
5000686-136279726.jpg / meta-label = roof
5000686-136279727.jpg / meta-label = roof
5000686-136279728.jpg / meta-label = roof
5000686-136279729.jpg / meta-label = Other


title: How to constrain the output of a deep neural network

body:
I would like to preface this by apologizing for the somewhat generalist nature of my question.

I want to fit a DNN classifier that is compiled using a softmax activation with categorical cross-entropy loss to predict the probabilities of 4 classes.

I need to add a constraint on the neural net to take into account one feature value.. for the sake of an example, let's say the 4 classes I want to predict are cat, dog, lion, and ant, and I have a boolean feature "is_pet"  - I want to make sure that the predicted probabilities of lion and ant to be exactly zero ... What is the best way of implementing this constraint? and are there other ways of achieving this by using a different model architecture?


title: AttributeError: ‘RNN’ object has no attribute ‘weight_hh_l’

body:
I want to change weights on my own

class Net(nn.Module):
    def init(self):
        super(Net, self).init()
        self.F = torch.randint(0,255,(len(train_dataset),image_size))
        self.I =torch.ones(image_size,image_size)
        self.w = 0.00001
        self.rnn = nn.RNN(image_size,len(train_dataset),1,‘relu’)
        self.rnn.weight_hh_l[0] = torch.t(self.F)
        self.rnn.weight_ih_l[0] = (self.I - (self.w * torch.mm(torch.t(self.F),self.F)))

And have error

AttributeError: 'RNN' object has no attribute 'weight_hh_l'

What I’m doing wrong?
Thank you in advance


title: tf_encrypted.player how to config?

body:
While training the model federated learning with keras on syft,to start workers on different system as client workers, particular line of command to be exectued on terminal.

python -m tf_encrypted.player --config /tmp/tfe.config client name

this command should be executed on system which is one of my client workers.when executing the code

(PySyft) C:\Users\mades>python -m tf_encrypted.player --config /tmp/tfe.config server1
Falling back to insecure randomness since the required custom op could not be found for the installed version of TensorFlow. Fix this by compiling custom ops. Missing file was 'C:\Users\mades\AppData\Roaming\Python\Python37\site-packages\tf_encrypted/operations/secure_random/secure_random_module_tf_1.15.2.so'
WARNING:tensorflow:From C:\Users\mades\AppData\Roaming\Python\Python37\site-packages\tf_encrypted\session.py:24: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

Traceback (most recent call last):
  File "C:\Users\mades\Anaconda3\envs\PySyft\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\mades\Anaconda3\envs\PySyft\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\mades\AppData\Roaming\Python\Python37\site-packages\tf_encrypted\player\__main__.py", line 22, in <module>
    config = RemoteConfig.load(args.config)
  File "C:\Users\mades\AppData\Roaming\Python\Python37\site-packages\tf_encrypted\config.py", line 221, in load
    with open(filename, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tfe.config'

how to prepare the config file for federated learning on this system?


title: How do i make my PyTorch DCGAN code to run on a GPU?

body:
I am trying to train a DCGAN on a GPU but as I am starting out with PyTorch I have tried to do something from the documentation and it works but I want to confirm if it is the correct way to do it as I have done cause I looked at many other questions about running it on the GPU but they are done in different ways.

import os
import torch
import torchvision.transforms as transforms
from torch.autograd import Variable
from torch.nn import (
    BatchNorm2d,
    BCELoss,
    Conv2d,
    ConvTranspose2d,
    LeakyReLU,
    Module,
    ReLU,
    Sequential,
    Sigmoid,
    Tanh,
)
from torch.optim import Adam
from torch.utils.data import DataLoader
from torchvision.datasets import CIFAR10
from torchvision.utils import save_image
from tqdm import tqdm

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

dataset = DataLoader(
    CIFAR10(
        root="./Data",
        download=True,
        transform=transforms.Compose(
            [
                transforms.Resize(64),
                transforms.ToTensor(),
                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
            ]
        ),
    ),
    batch_size=64,
    shuffle=True,
    num_workers=2,
)

try:
    os.mkdir("./Models")
    os.mkdir("./Results")

except FileExistsError:
    pass

class Gen(Module):
    def __init__(self):
        super(Gen, self).__init__()
        self.main = Sequential(
            ConvTranspose2d(100, 512, 4, 1, 0, bias=False),
            BatchNorm2d(512),
            ReLU(True),
            ConvTranspose2d(512, 256, 4, 2, 1, bias=False),
            BatchNorm2d(256),
            ReLU(True),
            ConvTranspose2d(256, 128, 4, 2, 1, bias=False),
            BatchNorm2d(128),
            ReLU(True),
            ConvTranspose2d(128, 64, 4, 2, 1, bias=False),
            BatchNorm2d(64),
            ReLU(True),
            ConvTranspose2d(64, 3, 4, 2, 1, bias=False),
            Tanh(),
        )

    def forward(self, input):
        output = self.main(input)
        return output

class Dis(Module):
    def __init__(self):
        super(Dis, self).__init__()
        self.main = Sequential(
            Conv2d(3, 64, 4, 2, 1, bias=False),
            LeakyReLU(0.2, inplace=True),
            Conv2d(64, 128, 4, 2, 1, bias=False),
            BatchNorm2d(128),
            LeakyReLU(0.2, inplace=True),
            Conv2d(128, 256, 4, 2, 1, bias=False),
            BatchNorm2d(256),
            LeakyReLU(0.2, inplace=True),
            Conv2d(256, 512, 4, 2, 1, bias=False),
            BatchNorm2d(512),
            LeakyReLU(0.2, inplace=True),
            Conv2d(512, 1, 4, 1, 0, bias=False),
            Sigmoid(),
        )

    def forward(self, input):
        output = self.main(input)
        return output.view(-1)

def weights(obj):
    classname = obj.__class__.__name__
    if classname.find("Conv") != -1:
        obj.weight.data.normal_(0.0, 0.02)

    elif classname.find("BatchNorm") != -1:
        obj.weight.data.normal_(1.0, 0.02)
        obj.bias.data.fill_(0)

gen = Gen()
gen.apply(weights)
gen.cuda().cuda().to(device)

dis = Dis()
dis.apply(weights)
dis.cuda().to(device)

criterion = BCELoss()

optimizerDis = Adam(dis.parameters(), lr=0.0002, betas=(0.5, 0.999))
optimizerGen = Adam(gen.parameters(), lr=0.0002, betas=(0.5, 0.999))

for epoch in range(25):
    for batch, data in enumerate(tqdm(dataset, total=len(dataset)), 0):
        dis.zero_grad()

        input = Variable(data[0]).cuda().to(device)
        target = Variable(torch.ones(input.size()[0])).cuda().to(device)
        output = dis(input).cuda().to(device)

        realError = criterion(output, target)

        noise = Variable(torch.randn(input.size()[0], 100, 1, 1)).cuda().to(device)
        fake = gen(noise).cuda().to(device)
        target = Variable(torch.zeros(input.size()[0])).cuda().to(device)
        output = dis(fake.detach()).cuda().to(device)

        fakeError = criterion(output, target)

        errD = realError + fakeError
        errD.backward()
        optimizerDis.step()

        gen.zero_grad()

        target = Variable(torch.ones(input.size()[0])).cuda().to(device)
        output = dis(fake).cuda().to(device)

        errG = criterion(output, target)
        errG.backward()
        optimizerGen.step()

        print(f"  {epoch+1}/25 Dis Loss: {errD.data:.4f} Gen Loss: {errG.data:.4f}")

save_image(data[0], "./Results/Real.png", normalize=True)
save_image(gen(noise).data, f"./Results/Fake{epoch+1}.png", normalize=True)
torch.save(gen, f"./Models/model{epoch+1}.pth")


title: Bring any PyTorch cuda tensor in the range [0,1]

body:
Suppose I have a PyTorch Cuda Float tensor x of the shape [b,c,h,w] taking on any arbitrary value allowed by Float Tensor range. I want to normalise it in the range [0,1].

I think of the following algorithm (but any other will also do).

Step1: Find minimum in each batch. Call it min and having shape [b,1,1,1].

Step2: Similarly find the maximum and call it max.

Step3: Use y = (x-min)/max. Alternatively use y = (x-min)/(max-min). I don't know which one will be better. y should have the same shape as that of x.

I am using PyTorch 1.3.1.

Specifically I am unable to get the desired min using torch.min(). Same goes for max.

I am going to use it for feeding it to pre-trained VGG for calculating perceptual loss (after the above normalisation i will additionally bring them to ImageNet mean and std). Due to some reason I cannot enforce [0,1] range during data loading part because the previous works in my area have a very specific normalisation algorithm which has to be used but some times does not ensures [0,1] bound but will be somewhere in its vicinity. That is why at the time computing perceptual loss I have to do this explicit normalisation as a precaution. All out of the box implementation of perceptual loss I am aware assume data is in [0,1] or [-1,1] range and so do not do this transformation.

Thankyou very much


title: How to build skip connection between two CNNs (like MSG GAN) using Pytorch framework

body:
MSG-GAN
I am trying to connect two CNNs using the skip connections like MSG-GAN (Multi-Scale Gradients GAN). The only idea come to my mind is to obtain the feature maps outside the 1st CNN then used as the input of the 2nd CNN. However, this method need multi input for the 2nd CNN. It seems like the MSG-GAN doesn't use this method. But the code is too hard for me to understand as a beginner. Is there anyone can help me figure out what did they use to build up the skip connections between two CNNs? Or could you recommend a method for me?
My method:

Net1(nn.Module):
    def __init__(self, kernel_size=5, n_filters=16, n_layers=3):
        xxx
    def forward(self, x):
        x = self.body(self.head(x))
        self.connection1 = x.detach()
        return F.relu(self.fc(x))

cnn1= Net1()
train_model(cnn1)
connection1= cnn1.featuremap1

Net2()
XXX
cnn2 = Net2()
cnn2(input, connection1, connection2, XXX)

MSG-GAN method: https://github.com/akanimax/msg-gan-v1
MSG-GAN


title: Freezing intermediate layers in ALBERT

body:
I want to freeze intermediate sub-layers in ALBERT. Unlike BERT,
ALBERT's encoder looks like this

(encoder): AlbertTransformer(
      (embedding_hidden_mapping_in): Linear(in_features=128, out_features=768, bias=True)
      (albert_layer_groups): ModuleList(
        (0): AlbertLayerGroup(
          (albert_layers): ModuleList(
            (0): AlbertLayer(
              (full_layer_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (attention): AlbertAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0, inplace=False)
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              )
              (ffn): Linear(in_features=768, out_features=3072, bias=True)
              (ffn_output): Linear(in_features=3072, out_features=768, bias=True)
            )
          )
        )
      )
    )

So inside the modulelist, we can easily choose which sublayers to freeze. But here, we have one module. The documentation says that there are 12 repeating layers. Is there any good way to freeze intermediate layers in ALBERT?


title: pytorch autograd.grad get None for some layers occassionally

body:
I'm using the autograd.grad of torch to perform analysis on my network. But I met with the problem that, when I run grad() on different data items, I occassionally get None for some layers on some certain data items, which will further cause errors.
I'm confused why this happens. Why does the same network get None grad on some data items, while does not on other data items? There is not noise items in the dataset, no missing values.
Some part of my network is as follows (to make it short, I did not post the whole network, but it's enough to tell the problem):
def forward(self, input):
    x = input
    out = self.backbone(x)
    predict_up_conv = self.relu2(self.bn2(self.conv_up_conv(out)))
    predict_down_conv = self.relu3(self.bn3(self.conv_down_conv(out)))
    predict_cls_conv = self.relu4(self.bn4(self.conv_cls_conv(out)))
    predict_up = self.conv_up(predict_up_conv)
    predict_down = self.conv_down(predict_down_conv)
    predict_cls = self.conv_cls(predict_cls_conv)
    if self.phase == 'test':
        predict_cls_softmax = self.softmax(predict_cls)
        return predict_up, predict_down, predict_cls_softmax
    else:
        return predict_up,predict_down,predict_cls

the code I use to calculate the gradient is:
grads = grad(y, w, retain_graph=True, create_graph=True, allow_unused=True)

y is the target value, and w is the list of all parameters of the network.
grads is the resulting list of gradients. In normal cases, it is a list of pytorch tensors with shapes matching the feature map of each layer in the network. However, on some data items when I run this code, the grads has some elements which is None. And the associated layers are always the self.conv_up_conv, self.conv_down_conv, self.conv_up, self.conv_down.
I'm confused what's wrong with my code.
Thank you all for helping me!!!


title: How handle variable size sequences in LSTM module of PyTorch?

body:
I know how to handle variable size sequence data in PyTorch for most cases.
For example presume we want to predict next position of a person given its past locations.
In this case we can pad different sequences with pad_sequence function. Then we can use pack_padded_sequence and pad_packed_sequence to calculate a batch of sequences of diverse lengths. Finally, We can use a mask on output of LSTM and ground truth positions and calculate loss and do back propagation.
In case of above example I'm aware of handling variable size sequences during training.
But assume another application of LSTM that we use hidden state of LSTM in last time step and concatenate it with other features and pass it to a deep neural network. Finally the deep neural network gives us a loss. I want to train all networks in an end to end fashion. But in this case I don't know how to handle variable size sequences.
In this case we don't have ground truth for LSTM for different sequences, so we can't use masking after pack_padded_sequence and pad_packed_sequence. Instead we have a loss that back propagated from deep neural network.


title: ‘ImbalancedDatasetSampler’ class can't access TensorDataset labels

body:
I am dealing with imbalanced data (mere 2% minority samples). I tried “WeightedRandomSampler” approach in pytorch for building deep learning model which works OK for my validation set, but it fails in case of independent test set. I came across
https://github.com/ufoym/imbalanced-dataset-sampler
and I wanted to try this approach on my data. The problem is - this ‘ImbalancedDatasetSampler’ module can’t figure out labels in my TensorDataset object.
tensor_x = torch.Tensor(X_TRAIN)     # X_TRAIN shape [16200, 1105]
tensor_y = torch.Tensor(Y_TRAIN.values)     # Y_TRAIN shape [16200, 1]
dataset_train = TensorDataset(tensor_x, tensor_y)
train_loader = data_utils.DataLoader(train_dataset, batch_size = BATCH_SIZE, sampler=ImbalancedDatasetSampler(train_dataset))

it returns an error
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-13-b7cc711025fa> in <module>
      2 train_loader = data_utils.DataLoader(train_dataset,
      3                                      batch_size = BATCH_SIZE,
----> 4                                     sampler=ImbalancedDatasetSampler(train_dataset))

~/py_torch_sampler.py in __init__(self, dataset, indices, num_samples, callback_get_label)
     30         label_to_count = {}
     31         for idx in self.indices:
---> 32             label = self._get_label(dataset, idx)
     33             if label in label_to_count:
     34                 label_to_count[label] += 1

~/py_torch_sampler.py in _get_label(self, dataset, idx)
     51             return self.callback_get_label(dataset, idx)
     52         else:
---> 53             raise NotImplementedError
     54
     55     def __iter__(self):

NotImplementedError:

Could someone tell me how can I solve this problem?


title: How to convert string representation of tensors into numpy array in pandas

body:
I have a pandas df containing prediction probability distributions from a torch model. It looks like this:
    Framecount  Expression                                        Probability
0          0.0         8.0  tensor([6.9263e-06, 6.6337e-10, 8.2442e-03, 4....
11        10.0         8.0  tensor([6.4393e-05, 4.4693e-07, 8.2253e-02, 1....
22        20.0         9.0  tensor([7.5355e-05, 2.4437e-07, 9.7638e-02, 1....
33        30.0         3.0  tensor([4.9751e-05, 1.1386e-06, 4.7163e-03, 7....
44        40.0         9.0  tensor([1.3237e-05, 1.3779e-07, 2.8534e-03, 1....

When I run type(df.Probability.tolist()[0]), I get str. Why is this?
How can I convert this column to contain arrays of floats so that I can do numerical operations on them?
edit
When I create the df, I essentially do the follow:
d = []
framecount = 0

for x in data:
    Expression = model.predict(x)[0] # 8.0
    Probability = model.predict(x)[1] # tensor([6.9263e-06, 6.6337e-10,
...
    d.append([framecount, Expression, Probability])
    framecount += 10

df = pd.DataFrame(d)


title: matplotlib subplots very slow

body:
The following code
from torch.utils.data import DataLoader
from defect_segmentation.data_loading.DatasetSingleImage import dataset_single_image_default
import numpy as np

if __name__ == "__main__":
    import matplotlib.pyplot as plt

    def main():
        dataset = dataset_single_image_default()
        batch_size = 16
        shuffle = True
        num_workers = 0
        loader = DataLoader(dataset, batch_size=batch_size, shuffle=shuffle, num_workers=num_workers)
        for i_batch, sample_batched in enumerate(loader):
            fig, axs = plt.subplots(int(np.sqrt(batch_size)), batch_size // int(np.sqrt(batch_size)))
            fig.suptitle(f"i_batch = {i_batch}")
            for i_sample, ax in zip(range(sample_batched.shape[0]), axs.flat):
                ax.set_title(f"Sample #{i_sample}")
                ax.axis("off")
                ax.imshow(sample_batched[i_sample, :, :])
                plt.pause(0.001)

    main()

works and outputs figures like so

This is fine.
Problem is by the 10th figure, it becomes very slow to populate the figures. I don't know what can be causing this.

For completeness, here is the code for creating the dataset (striding over a single image):
from torch.utils.data import Dataset
from Utils.ConfigProvider import ConfigProvider
import cv2
import os
from overrides import overrides # pip install overrides

class DatasetSingleImage(Dataset):
    def __init__(self, image_path: str, sample_shape: tuple, strides: tuple):
        self._path = image_path
        assert os.path.isfile(self._path)
        self._im = cv2.imread(self._path)
        self._shape = self._im.shape
        self._rows, self._cols = self._shape[0], self._shape[1]

        self._sample_shape = sample_shape
        self._sample_rows, self._sample_cols = self._sample_shape[0], self._sample_shape[1]

        self._strides = strides
        self._stride_rows, self._stride_cols = self._strides[0], self._strides[1]
        # self._rows_start_range = range(0, self._rows, self._stride_rows)
        # self._cols_start_range = range(0, self._cols, self._stride_cols)

        self._rows_tuples_range = \
            [(c, min(c + self._sample_rows, self._rows)) for c in range(0, self._rows - self._sample_rows, self._stride_rows)]
        self._cols_tuples_range = \
            [(r, min(r + self._sample_cols, self._cols)) for r in range(0, self._cols - self._sample_cols, self._stride_cols)]

        self._n_strides_rows = len(self._rows_tuples_range)
        self._n_strides_cols = len(self._cols_tuples_range)
        self._total_strides = self._n_strides_rows * self._n_strides_cols

    def __len__(self):
        return self._total_strides

    @overrides # pip install overrides
    def __getitem__(self, ind):
        row_ind = ind // self._n_strides_cols
        col_ind = ind % self._n_strides_cols
        sample_x = self._rows_tuples_range[row_ind]
        sample_y = self._cols_tuples_range[col_ind]
        sample = self._im[sample_x[0]:sample_x[1], sample_y[0]:sample_y[1]]
        assert sample.shape[:2] == self._sample_shape
        return sample

def dataset_single_image_default():
    path = ConfigProvider.config().data.defective_inspected_path1
    sample_shape = (50, 50)
    strides = (25, 25)
    dataset = DatasetSingleImage(path, sample_shape, strides)
    return dataset

What is making the plotting slow, and how to fix it?


title: Getting impossible predicted labels (all zeroes) when doing mulit-class classification with Bert

body:
I create a dataset with 9 categories:
classDict = {"text/dokujo-tsushin": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001",
"text/it-life-hack": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010",
"text/kaden-channel": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100",
"text/livedoor-homme": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000",
"text/movie-enter": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000",
"text/peachy": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000",
"text/smax": "000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000",
"text/sports-watch": "000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000",
"text/topic-news": "000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000"}

Although I set the number of label to 9 by the following line:
model = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=args.num_labels)

I am getting a tenth label, all zeros. And I have no clue what is wrong. Has anyone run across the same problem or Does anyone know the cause of this?
I have used glue to check the training data that there is not a single example labelled with all zeroes.


title: One instance with multple GPUs or multiple instances with one GPU

body:
I am running multiple models using GPUs and all jobs combined can be run on 4 GPUs, for example. Multiple jobs can be run on the same GPU since the GPU memory can handle it.
Is it a better idea to spin up a powerful instance with all 4 GPUs as part of it and run all the jobs on one instance? Or should I go the route of having multiple instances with 1 GPU on each?
There are a few factors I'm thinking of:

Latency of reading files. Having a local disk on one machine should be faster latency wise, but it would be a quite a few reads from one source. Would this cause any issues?
I would need quite a few vCPU and a lot of memory to scale the IOPS since GPC scales IOPS that way, apparently. What is the best way to approach this? If anyone has any more on this, would appreciate pointers.
If in the future I need to downgrade to save costs/downgrade performance, I could simple stop the instance and change my specs.
Having everything on one machine would be easier to work with. I know in production I would want a more distributed approach, but this is strictly experimentation.

Those are my main thoughts. Am I missing something? Thanks for all of the help.


title: Memory management issue with image upload for dash web app

body:
I built a computer vision web app and deployed it on Heroku.
It processes picture uploaded by a user (through dcc.upload) and should return the processed picture and additional information computed by a PyTorch back-end.
After a few picture uploads, I bumped into memory issue and the app crashed. My understanding is that all pictures passed through dcc.upload are accumulated into memory (see here) and causes the crash.
Here an overview of the code:
from dash.dependencies import Input, Output
import dash_core_components as dcc
import dash_html_components as html

app = dash.Dash(__name__)

app.layout = html.Div([

    ...

        dcc.Upload(
        id='upload-image',
        children=html.Div(['Drag and Drop or ',
                           html.A('Select Files')]),
        ...

        ),
    ...

    ])

@app.callback(
    [Output('output-image-1', 'children'),
      Output('output-image-2', 'children'),
      Output('output-graph', 'children')],
    [Input('upload-image', 'contents')])
def update_output(content):

    if content is not None:

        # Deep Learning
        raw_img, grad_img, d_probs = process(content)
        return parse_image(str_raw_img), parse_image(str_grad_img), parse_graph(d_probs)

    else:
        return None, None, None

if __name__ == '__main__':
    app.run_server()

How could I improve memory management? Ideally, pictures could be deleted whenever a new one is uploaded. I understand I could use dcc.Store but I'm not sure how to proceed. Thanks for any guidance.


title: Pytorch: RuntimeError when trying to alternate optimization steps

body:
I am alternating between 2 optimization steps and each step is using information and gradients from the other. I was able to write a code that works for 1-dimensional parameters but not for 2-dimensional ones.
In the code below if you set dim = 1 it works, whereas for dim = 2 you get

"RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time."

If I set retain_graph=True in backward() then it works. But of course we should avoid retaining the graph (if possible) and the one-dimensional case shows that it can be avoided.
import torch

class f_class():
    def __init__(self, dim):
        self.dim = dim
        if self.dim == 1:
            self.w = torch.tensor((3.), requires_grad=True)
        elif self.dim == 2:
            self.w = [torch.tensor((3.), requires_grad=True), torch.tensor((5.), requires_grad=True)]
        else:
            raise ValueError("dim 1 or 2")

    def forward(self, x):
        if self.dim == 1:
            return torch.mul(self.w, x)
        elif self.dim == 2:
            return torch.mul(torch.mul(self.w[0], self.w[1]), x)

    def set_w(self, w):
        self.w = w

    def get_w(self):
        return self.w

class g_class():
    def __init__(self):
        self.q = torch.tensor((4.), requires_grad=True)

    def forward(self, f):
        return torch.mul(self.q, f)

    def set_q(self, q):
        self.q = q

    def get_q(self):
        return self.q

def w_new(f, g, dim):

    loss_g = g.forward(f.forward(xd))

    if dim == 1:
        grads = torch.autograd.grad(loss_g, f.get_w(), create_graph=True, only_inputs=True)[0]

        temp = f.get_w().detach() + grads
    else:
        grads = torch.autograd.grad(loss_g, f.get_w(), create_graph=True, only_inputs=True)

        temp = [wi.detach() + gi for wi, gi in zip(f.get_w(), grads)]

    return temp

def q_new(f, g):

    loss_f = 2 * f.forward(xd)

    loss_f.backward()

    temp = g.get_q().detach() + g.get_q().grad

    temp.requires_grad = True

    return temp

dim = 2

xd = torch.tensor((2.))

f = f_class(dim)
g = g_class()

for _ in range(3):

    print(f.get_w(), g.get_q())

    wnew = w_new(f, g, dim)

    f.set_w(wnew)
    print(f.get_w(), g.get_q())

    qnew = q_new(f, g)

    g.set_q(qnew)

print(f.get_w(), g.get_q())


title: PyTorch batching at char-level and token-level

body:
I'm trying to train an NLP model that uses a character level and token level LSTM to create and concatenate embeddings. The problem I'm having is batching the data so that each sentence is in the same place in both the token and character batches.
For example, if I had the phases 'Hello there' and 'see you soon', both batches would store them in that order [[Hello, there] [see, you, soon]] (obviously with different tensors).
Thanks in advance


title: PyTotch CIFAR-10 vs Kaggle CIFAR-10 : Totally different result for exactly same architecture on CIFAR-10

body:
I have been learning PyTorch for some weeks. While I was practicing with CIFAR-10 dataset from PyTorch datasets, I also thought of practicing with ImageFolder class, so I found a version of Cifar-10 from Kaggle, where the images were foldered.(I you don't remember PyTorch datasets are in tar.gz format, not in folder structure)
To my utter surprise, in spite of using the same loss function, learning rate and architecture, The Kaggle dataset test set accuracy starts from  0.18 and PyTorch dataset accuracy starts from 0.56 at epoch 1.
Finally after 20 epochs ,one almost saturates near 0.45 and the later one almost fixes near 0.86.
I have checked again and again,but not finding any big difference in those two codes.
I really want to know, if I have done anything deadly wrong, or there is anything fundamentally different about those two datasets.
To clarify, I am using this Pytorch dataset, and this Kaggle dataset .
The codes are too large to be provided here, so I am providing links my notebooks, you  are welcome to take a look at my whole code, and also can run if necessary [You only need to use your Kaggle API key to download the dataset from kaggle, I can't make mine one public...sorry for the inconvinience]
Kaggle Dataset Notebook here and  Pytorch Dataset Notebook here
I am also providing the chunk of code that I think , is mostly different.
Kaggle Dataset:
Epoch 1 score = 0.18
Epoch 20 score = 0.45
from torch.utils.data import DataLoader

def createVal(train_list, root_folder, classes, valid_split ):
    try:
        os.mkdir(os.path.join(root_folder, 'val'))
    except FileExistsError:
        pass
    for cls in classes:
        try:
            os.mkdir(os.path.join(root_folder, 'val', cls))
        except FileExistsError:
            pass

    np.random.shuffle(train_list)

    valid_len = len(train_list) * valid_split
    for i in tqdm(range(int(valid_len))):
        shutil.move(train_list[i], train_list[i].replace('/train/', '/val/'))

valid_split = 0.2
batch_size = 32
num_workers = 4

root_folder = "/content/cifar10/cifar10"
train_folder = os.path.join(root_folder, "train")
test_folder = os.path.join(root_folder, "test")

if valid_split:
  createVal(train_list, root_folder, classes, valid_split = valid_split)

  val_folder = os.path.join(root_folder, "val")

  val_data = datasets.ImageFolder(val_folder, transform = transform)
  val_loader = DataLoader(val_data, batch_size = batch_size, num_workers = num_workers )

train_data = datasets.ImageFolder(train_folder, transform = transform)
train_loader = DataLoader(train_data, shuffle = True, batch_size = batch_size, num_workers = num_workers )

test_data = datasets.ImageFolder(test_folder, transform = transform)
test_loader = DataLoader(test_data, batch_size = batch_size, num_workers = num_workers )

Pytorch Dataset:
Epoch 1 score = 0.18
Epoch 20 score = 0.45
valid_split = 0.2
batch_size = 32
num_workers = 4

if valid_split:
  num_train = len(train_data)
  idx = list(range(num_train))
  np.random.shuffle(idx)
  train_idx = idx[int(valid_split*num_train):]
  val_idx = idx[:int(valid_split*num_train)]

  train_sampler = SubsetRandomSampler(train_idx)
  val_sampler = SubsetRandomSampler(val_idx)

  train_loader = DataLoader(train_data, sampler = train_sampler, batch_size = batch_size, num_workers = num_workers )
  val_loader = DataLoader(train_data, sampler = val_sampler, batch_size = batch_size, num_workers = num_workers )
else:
  train_loader = DataLoader(train_data, batch_size = batch_size, num_workers = num_workers )

test_loader = DataLoader(test_data, batch_size = batch_size, num_workers = num_workers )


title: Buffered data was truncated after reaching the output size limit using tqdm

body:
learning_rate = 1e-4
####### model parameter #######
netG = VAE1().to(device);
netD= Discriminator(nChannels=64,ndf=64).to(device);
####### optimizer  ###########
optimizerD = torch.optim.Adam(netD.parameters(), lr=learning_rate, betas=(0.5, 0.999))
optimizerG = torch.optim.Adam(netG.parameters(), lr=learning_rate, betas=(0.5, 0.999))
########params#######
epochs = 200

##################
# Training and testing the VAE
l1_loss=torch.nn.L1Loss();
means=[];
logvars=[];
costs=[];
print('..starting the training');
for epoch in range(0, epochs + 1):
    # Training
    if epoch > 0:  # test untrained net first
        netG.train()
        netD.train();
        train_lossG = 0.;
        train_lossD = 0.;

        #for samp_real,samp_x in tqdm.tqdm(train_loader, total=int(len(train_loader)),position=0, leave=True):
        with tqdm.tqdm(train_loader, total=int(len(train_loader)), unit="batch",position=0, leave=True) as tepoch:
          for  samp_real,samp_x in tepoch:
            tepoch.set_description(f"Epoch {epoch}")
            samp_real = samp_real.to(device)
            samp_x = samp_x.to(device)
            # ===================forward=====================
            x_hat_real, mu_real, logvar_real,z_real= netG(samp_real)
            x_hat_x, mu_x, logvar_x,z_x= netG(samp_x)
            #print(z_real.shape,z_x.shape)
            ## train Discrimninator########
            out_r = netD(z_real.detach());
            out_x = netD(z_x.detach());
            lossD = 0.5 *(torch.mean((1.0-out_r)**2)
            +torch.mean((out_x)**2)); #discriminator clasify if the image from real domain or not (output=1 for real domain and output=0 x domain)
            train_lossD+=lossD.item();
            '''
            print(np.sum(np.isnan(z_real.detach().cpu().numpy())))
            print(np.sum(np.isnan(z_x.detach().cpu().numpy())))
            print(train_lossD)
            '''

            optimizerD.zero_grad(); #important
            lossD.backward();
            optimizerD.step();

            ## train Generator########
            # VAE loss for real
            loss_real_l1=l1_loss(x_hat_real, samp_x)*256.0
            loss_real_kl=0.5 * torch.mean(logvar_real.exp() - logvar_real - 1.0 + mu_real.pow(2))
            loss_real=loss_real_l1+loss_real_kl;
            # VAE loss for x

            loss_x_l1=l1_loss(x_hat_x, samp_x)*256.0
            loss_x_kl=0.5 * torch.mean(logvar_x.exp() - logvar_x - 1.0 + mu_x.pow(2))
            loss_x=loss_x_l1+loss_x_kl;

            ### GAN loss for genertor (negetive of discriminator loss)
            out_r = netD(z_real);
            out_x = netD(z_x);
            lossGAN = 0.5 *(torch.mean(out_r**2)
            +torch.mean((1.0-out_x)**2))

            ### Total loss
            lossF= loss_real+loss_x+lossGAN;
            train_lossG+=lossF.item();

            optimizerG.zero_grad(); #important
            lossF.backward()
            optimizerG.step();
            costs.append([loss_real.item(), loss_x.item(), lossGAN.item(), lossD.item()])
            tepoch.set_postfix(loss_D=lossD.item(), lossGAN=lossGAN.item(), loss_real=loss_real.item(),loss_x=loss_x.item())

        print(f'====> Epoch: {epoch} Average discriminator loss: {train_lossD / len(train_loader.dataset):.4f} Generator loss: {train_lossG / len(train_loader.dataset):.4f}')

    # Testing
    with torch.no_grad():
        netG.eval()
        test_loss_r = 0
        test_loss_x = 0
        for ix,(samp_real,samp_x) in enumerate(test_loader):
            samp_real = samp_real.to(device)
            samp_x = samp_x.to(device)
            # ===================forward=====================
            x_hat_real, mu_real, logvar_real,z_real= netG(samp_real)
            x_hat_x, mu_x, logvar_x,z_x= netG(samp_x)
            # =====================log=======================
            loss_r=l1_loss(samp_real, x_hat_real)
            loss_x=l1_loss(samp_x, x_hat_x)
            test_loss_r+=loss_r.item();
            test_loss_x+=loss_x.item();
            if(ix<1):
              plt.figure(figsize=(10, 8))
              #plt.suptitle('real test data / reconstructions')
              for i in range(4):
                  plt.subplot(1,8,i+1)
                  plt.imshow(np.clip(x_hat_real.detach().cpu().numpy()[i,:,:,:].transpose(1,2,0)+0.5,0.0,1.0))
                  plt.axis('off')

              for i in range(4):
                  plt.subplot(1,8,i+1+4)
                  plt.imshow(np.clip(samp_real.detach().cpu().numpy()[i,:,:].transpose(1,2,0)+0.5,0.0,1.0))
                  plt.axis('off')
              plt.show()

              plt.figure(figsize=(10, 8))
              #plt.suptitle('synthetic test data / reconstructions')
              for i in range(4):
                  plt.subplot(1,8,i+1)
                  plt.imshow(np.clip(x_hat_x.detach().cpu().numpy()[i,:,:,:].transpose(1,2,0)+0.5,0.0,1.0))
                  plt.axis('off')

              for i in range(4):
                  plt.subplot(1,8,i+1+4)
                  plt.imshow(np.clip(samp_x.detach().cpu().numpy()[i,:,:].transpose(1,2,0)+0.5,0.0,1.0))
                  plt.axis('off')
              plt.show()
            # ===================log========================
    test_loss_r /= len(test_loader.dataset)
    test_loss_x /= len(test_loader.dataset)
    print(f'====> Test set loss_r: {test_loss_r:.4f} test loss_x: {test_loss_x:.4f}')

I was training a model on colab when in the buffer it says : Buffered data was truncated after reaching the output size limit.
  0%|          | 0/282 [00:00<?, ?batch/s]====> Test set loss_r: 0.0163 test loss_x: 0.0110
Epoch 62: 100%|██████████| 282/282 [02:06<00:00,  2.23batch/s, lossGAN=0.292, loss_D=0.246, loss_real=34.1, loss_x=33.7]====> Epoch: 62 Average discriminator loss: 0.0131 Generator loss: 4.3249

  0%|          | 0/282 [00:00<?, ?batch/s]====> Test set loss_r: 0.0167 test loss_x: 0.0112
Epoch 63: 100%|██████████| 282/282 [02:05<00:00,  2.25batch/s, lossGAN=0.354, loss_D=0.236, loss_real=33.9, loss_x=33.4]====> Epoch: 63 Average discriminator loss: 0.0132 Generator loss: 4.3928

  0%|          | 0/282 [00:00<?, ?batch/s]====> Test set loss_r: 0.0164 test loss_x: 0.0113
Epoch 64:  50%|█████     | 142/282 [01:04<01:00,  2.33batch/s, lossGAN=0.375, loss_D=0.208, loss_real=36.4, loss_x=35.9]

Instead on outputting this in colab, can I output the data to an external source? It is annoying because I can't see my results anymore and it took quite some time to train.
I have seen the answer Buffered data was truncated after reaching the output size limit where it says

Even if RAM | GPU | DISK on colab is free, this error still comes
because there is a limited memory for displaying output of a cell on
colab. Assuming the memory limit is around 2Mb to 5Mb when we run many
epochs(148+) during training, it tends to fill that memory and hence
the output is truncated because there is no more memory left free to
display the buffered epochs. However, the machine keeps running in the
background and the output is processed but it is not displayed because
of the buffered limit. You will still get your desired output.
One solution is not to use verbose=1 (use 0 instead).

Not sure how I can use it in my case.
EDIT
I think it is because of tqdm, but I am not sure how to solve that.


title: TensorFlow or Pytorch: Neural Network for classification - Constrain some weights to be chosen from a finite set

body:
When training Neural Networks for classification in TensorFlow/Keras, or Pytorch, is it possible to put constraints on the weights in the output layer such that they are chosen from a specific finite feasible set?
For example, let's say W is the weight in the output layer, is it possible to put constraints on W such that the optimal W is selected from the set S={W_1, W_2, ..., W_n}, where each W_i is a given feasible value for W? i.e. I will give the values of the W_1,...,W_n to the model
If this is not possible in TensorFlow or Pytorch, is there any other ways to achieve this?
Thanks!


I have correct shape of hidden layer for printing as below.
print(h0.shape)
print(x.shape)

torch.Size([2, 8, 256])
torch.Size([8, 300, 300])

But I still have error as  Expected hidden[0] size (2, 8, 256), got [8, 256]
What could be wrong?
The whole code is as follows.
import torch
import torch.nn as nn
import torchvision
import matplotlib.pyplot as plt
import torchvision.transforms as tt
from torchvision.datasets import ImageFolder
from PIL import Image
import numpy as np
from torch.autograd import Variable

seq_len = input_size
hidden_size = 256 #size of hidden layers
num_classes = 5
num_epochs = 20
batch_size = 8
learning_rate = 0.001
# Fully connected neural network with one hidden layer
num_layers = 2 # 2 RNN layers are stacked
class LSTM(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(LSTM, self).__init__()
        self.num_layers = num_layers
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)#batch must have first dimension
        #our inpyt needs to have shape
        #x -> (batch_size, seq, input_size)
        self.fc = nn.Linear(hidden_size, num_classes)#this fc is after RNN. So needs the last hidden size of RNN

    def forward(self, x):
        #according to ducumentation of RNN in pytorch
        #rnn needs input, h_0 for inputs at RNN (h_0 is initial hidden state)

        #the following one is initial hidden layer
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)#first one is number of layers and second one is batch size
        #output has two outputs. The first tensor contains the output features of the hidden last layer for all time steps
        #the second one is hidden state f
        print(h0.shape)
        print(x.shape)
        out, _ = self.lstm(x, h0)
        print(out.shape)
        #output has batch_size, seq_len, hidden size
        #we need to decode hidden state only the last time step
        #out (N, 30, 128)
        #Since we need only the last time step
        #Out (N, 128)
        out = out[:, -1, :] #-1 for last time step, take all for N and 128
        out = self.fc(out)
        return out

stacked_lstm_model = LSTM(input_size, hidden_size, num_layers, num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()#cross entropy has softmax at output
optimizer = torch.optim.Adam(stacked_lstm_model.parameters(), lr=learning_rate) #optimizer used gradient optimization using Adam

# Train the model
n_total_steps = len(train_dl)
for epoch in range(num_epochs):
    t_losses=[]
    for i, (images, labels) in enumerate(train_dl):
        # origin shape: [8, 1, 300, 300]
        # resized: [8, 300, 300]
        images = images.reshape(-1, seq_len, input_size).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = stacked_lstm_model(images)
        loss = criterion(outputs, labels)
        t_losses.append(loss)
        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print (f'Epoch [{epoch+1}/{num_epochs}], Step [{i+1}/{n_total_steps}], Loss: {loss.item():.4f}')
    avgd_trainloss = sum(t_losses)/len(t_losses)
    acc=0
    v_losses=[]
    with torch.no_grad():
      n_correct = 0
      n_samples = 0
      for v_images, v_labels in valid_dl:
          v_images = v_images.reshape(-1, seq_len, input_size).to(device)
          v_labels = v_labels.to(device)
          v_outputs = stacked_lstm_model(v_images)
          v_loss = criterion(v_outputs, v_labels)
          v_losses.append(v_loss)
          # max returns (value ,index)
          _, v_predicted = torch.max(v_outputs.data, 1)
          n_samples += v_labels.size(0)
          n_correct += (v_predicted == v_labels).sum().item()

      acc = 100.0 * n_correct / n_samples
    avgd_validloss = sum(v_losses)/len(v_losses)
    print (f'Epoch [{epoch+1}/{num_epochs}], Train loss: {avgd_trainloss.item():.4f}, Valid loss: {avgd_validloss.item():.4f}, Valid accu: {acc.item():.2f}')

# Test the model
# In test phase, we don't need to compute gradients (for memory efficiency)
with torch.no_grad():
    n_correct = 0
    n_samples = 0
    for images, labels in test_dl:
        images = images.reshape(-1, seq_len, input_size).to(device)
        labels = labels.to(device)
        outputs = stacked_lstm_model(images)
        # max returns (value ,index)
        _, predicted = torch.max(outputs.data, 1)
        n_samples += labels.size(0)
        n_correct += (predicted == labels).sum().item()

    acc = 100.0 * n_correct / n_samples
    print(f'Accuracy of the network on test images: {acc} %')


title: How does PyTorch Tensor.index_select() evaluates tensor output?

body:
I am not able to understand how complex indexing - non contiguous indexing of a tensor works. Here is a sample code and its output
import torch

def describe(x):
  print("Type: {}".format(x.type()))
  print("Shape/size: {}".format(x.shape))
  print("Values: \n{}".format(x))

indices = torch.LongTensor([0,2])
x = torch.arange(6).view(2,3)
describe(torch.index_select(x, dim=1, index=indices))

Returns output as

Type: torch.LongTensor Shape/size: torch.Size([2, 2]) Values:
tensor([[0, 2],
[3, 5]])

Can someone explain how did it arrive to this output tensor?
Thanks!


title: Pytorch ONNX : ONNX export failed on ATen operator _thnn_fused_gru_cell

body:
I am trying to export a Bahdanau Attention RNN model from pytorch to onnx, however I have an issue when trying to convert it. It is most probably caused by the GRUCell layer.
I get this error while trying to convert :
/home/ubuntu/anaconda3/envs/masktextspotter37/lib/python3.7/site-packages/torch/onnx/utils.py:655: UserWarning: ONNX export failed on ATen operator _thnn_fused_gru_cell because torch.onnx.symbolic_opset11._thnn_fused_gru_cell does not exist
  .format(op_name, opset_version, op_name))
Traceback (most recent call last):
  File "tools_onnx/test_net_onnx.py", line 504, in <module>
    main()
  File "tools_onnx/test_net_onnx.py", line 333, in main
    model_decoder_manager.generate_onnx(input_names = ["decoder_input", "decoder_hidden", "encoder_context"])#, output_names = ["decoder_output", "decoder_hidden", "decoder_attention"])#, dynamic_axes = dynamic)
  File "tools_onnx/test_net_onnx.py", line 469, in generate_onnx
    dynamic_axes=dynamic_axes
  File "/home/ubuntu/anaconda3/envs/masktextspotter37/lib/python3.7/site-packages/torch/onnx/__init__.py", line 148, in export
    strip_doc_string, dynamic_axes, keep_initializers_as_inputs)
  File "/home/ubuntu/anaconda3/envs/masktextspotter37/lib/python3.7/site-packages/torch/onnx/utils.py", line 66, in export
    dynamic_axes=dynamic_axes, keep_initializers_as_inputs=keep_initializers_as_inputs)
  File "/home/ubuntu/anaconda3/envs/masktextspotter37/lib/python3.7/site-packages/torch/onnx/utils.py", line 416, in _export
    fixed_batch_size=fixed_batch_size)
  File "/home/ubuntu/anaconda3/envs/masktextspotter37/lib/python3.7/site-packages/torch/onnx/utils.py", line 296, in _model_to_graph
    fixed_batch_size=fixed_batch_size, params_dict=params_dict)
  File "/home/ubuntu/anaconda3/envs/masktextspotter37/lib/python3.7/site-packages/torch/onnx/utils.py", line 135, in _optimize_graph
    graph = torch._C._jit_pass_onnx(graph, operator_export_type)
  File "/home/ubuntu/anaconda3/envs/masktextspotter37/lib/python3.7/site-packages/torch/onnx/__init__.py", line 179, in _run_symbolic_function
    return utils._run_symbolic_function(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/masktextspotter37/lib/python3.7/site-packages/torch/onnx/utils.py", line 656, in _run_symbolic_function
    op_fn = sym_registry.get_registered_op(op_name, '', opset_version)
  File "/home/ubuntu/anaconda3/envs/masktextspotter37/lib/python3.7/site-packages/torch/onnx/symbolic_registry.py", line 91, in get_registered_op
    return _registry[(domain, version)][opname]
KeyError: '_thnn_fused_gru_cell'

I guess it's because the GRUCell is not managed correctly in pytorch onnx, however I saw that the operator "GRU" exists in onnx (documentation available here). I hope there is just a little modification to do in the "symbolic" files.
Here is the model and the code :
class BahdanauAttnDecoderRNN(nn.Module):
    def __init__(
        self,
        hidden_size,
        embed_size,
        output_size,
        n_layers=1,
        dropout_p=0,
        bidirectional=False,
        onehot_size = (8, 32)
    ):
        super(BahdanauAttnDecoderRNN, self).__init__()
        # Define parameters
        self.hidden_size = hidden_size
        self.embed_size = embed_size
        self.output_size = output_size
        self.n_layers = n_layers
        self.dropout_p = dropout_p
        # Define layers
        self.embedding = nn.Embedding(output_size, embed_size)
        self.embedding.weight.data = torch.eye(embed_size)
        # self.dropout = nn.Dropout(dropout_p)
        self.word_linear = nn.Linear(embed_size, hidden_size)
        self.attn = Attn("concat", hidden_size, embed_size, onehot_size[0] + onehot_size[1])
        self.rnn = nn.GRUCell(2 * hidden_size + onehot_size[0] + onehot_size[1], hidden_size)
        self.out = nn.Linear(hidden_size, output_size)

    def forward(self, word_input, last_hidden, encoder_outputs):
        """
        :param word_input:
            word input for current time step, in shape (B)
        :param last_hidden:
            last hidden stat of the decoder, in shape (layers*direction*B, hidden_size)
        :param encoder_outputs:
            encoder outputs in shape (H*W, B, C)
        :return
            decoder output
        """
        # Get the embedding of the current input word (last output word)
        word_embedded_onehot = self.embedding(word_input).view(
            1, word_input.size(0), -1
        )  # (1,B,embed_size)
        word_embedded = self.word_linear(word_embedded_onehot)  # (1, B, hidden_size)
        attn_weights = self.attn(last_hidden, encoder_outputs)  # (B, 1, H*W)
        context = attn_weights.bmm(
            encoder_outputs.transpose(0, 1)
        )  # (B, 1, H*W) * (B, H*W, C) = (B,1,C)
        context = context.transpose(0, 1)  # (1,B,C)
        # Combine embedded input word and attended context, run through RNN
        # 2 * hidden_size + W + H: 256 + 256 + 32 + 8 = 552
        rnn_input = torch.cat((word_embedded, context), 2)
        last_hidden = last_hidden.view(last_hidden.size(0), -1)
        rnn_input = rnn_input.view(word_input.size(0), -1)
        hidden = self.rnn(rnn_input, last_hidden)
        if not self.training:
            output = F.softmax(self.out(hidden), dim=1)
        else:
            output = F.log_softmax(self.out(hidden), dim=1)
        # Return final output, hidden state
        # print(output.shape)
        return output, hidden, attn_weights

BahdanauAttnDecoderRNN(
  (embedding): Embedding(38, 38)
  (word_linear): Linear(in_features=38, out_features=256, bias=True)
  (attn): Attn(
    (attn): Linear(in_features=552, out_features=256, bias=True)
  )
  (rnn): GRUCell(552, 256)
  (out): Linear(in_features=256, out_features=38, bias=True)
)

If the layer needs to be modified by another one in order for the export to work, can you please tell me if it is possible to transfer the weights of the previous training to the new architecture ?
If you know how to deal with that issue, it would be really helpful !


title: How does one implement parallel SGD with the pytorch autograd RPC library so that gradients can be received from different processes without errors?

body:
The main thing I want to do is divide a batch in chunks and have each process (or gpu) process that chunk in parallel. The high level idea is this:
with Poo(100) in pool:
    losses = pool.map(forward, batch)
    torch.mean(losses).backward()
optimizer.step()

I was told that the pytorch RPC library might be a good idea here but I am unable to make it work yet.
My current error message is:
[W ProcessGroupGloo.cpp:558] Warning: Unable to resolve hostname to a (local) address. Using the loopback address as fallback. Manually set the network interface to bind to with GLOO_SOCKET_IFNAME. (function operator())

Although I am not sure why it is hapenning.
At a high level I am starting worker process that process each chunk from the batch and then I try to have them tell the main process (which I call master) what the losses are so that master can the aggregate the losses with torch.distributed.autograd. However, I am unable to make it work with the above error.
Current (completely self contained) code:

import os
from typing import Tuple, List

import torch
from torch import nn, optim, Tensor
from torch.distributed import rpc
import torch.distributed.autograd as dist_autograd
from torch.distributed.optim import DistributedOptimizer
from torch.multiprocessing import Pool, Process
import torch.multiprocessing as mp

world_size = 2  # three chunks, one for each process
num_epochs = 1  # this doesn't really matter, we only need to test if it can process a big batch and a small batch
iters = 5  # iters for 1 epoch
iters = 5
Din, Dout = 10, 5

class ToyModel(nn.Module):

    def __init__(self):
        super().__init__()
        self.lin = nn.Linear(Din, Dout)
        self.criterion = nn.MSELoss()

    def forward(self, batch_x, batch_y):
        y_ = self.lin(batch_x)
        loss = self.criterion(batch_y, y_)
        return loss

def get_ast_batch(batch_size: int) -> List[Tuple[Tensor]]:
    """
    Returns a list of size batch_size with each individual example.
    - 1 example in a batch is a task with K examples with dim=D.

    Note:
        num_proc = 3
        batch_size = 8  # chunk_size = 2 rem 1 (have three chunks of size 2 and one of size 1) 8/3 = 2.666 =  2 rem 1
        batch_size = 2  # chunk_size = 1 for each process since 2/3 <= 1

    :return:
    """
    data_x, data_y = torch.randn(Din), torch.randn(Dout)
    batch = [(data_x, data_y) for _ in range(batch_size)]
    return batch

def get_meta_batch(batch_size: int, k=15) -> Tuple[Tensor]:
    """
    Returns Tuple(torch.Tensor([B, K, D])) where each element in the batch is a task.
    - 1 example in a batch is a task with K examples with dim=D.
    :return:
    """
    data_x, data_y = torch.randn(batch_size, k, Din), torch.randn(batch_size, k, Dout)
    batch = data_x, data_y
    return batch

class Worker:
    def __init__(self, args, kwargs):
        self.args = args
        self.kwargs = kwargs
        self.id = rpc.get_worker_info().id
        # self.env.seed(args.seed)

class Master:

    def __init__(self, world_size):
        self.world_size = world_size
        self.master_rank = 0
        self.num_workers = world_size - 1
        os.environ['MASTER_ADDR'] = '127.0.0.1'
        os.environ['MASTER_PORT'] = '29500'

        rpc.init_rpc(f'worker{self.master_rank}', rank=self.master_rank, world_size=world_size)
        self.model = ToyModel()
        self.optimizer = optim.Adam(self.model.parameters(), lr=1e-2)
        self.saved_losses = {}
        # create rrefs (remote referneces) for calling rpc calls
        self.workers = []
        for worker_rank in range(1, world_size):
            worker_info = rpc.get_worker_info(worker_name=f'worker{worker_rank}')
            # Make a remote call to run func on worker to and return an RRef to the result value immediately.
            # worker_rrf = rpc.remote(to=worker_info, func=Worker, args='args', kwargs='kwargs')
            self.workers.append(worker_info)
            #
            self.saved_losses[worker_info.id] = []

    def forward_parallel(self, batch):
        """
        :param batch:
            - List[Tensor([B, K, D])]
            - List[Tuple[Tensor(D), Tensor(D)]]
        """
        # batch_size = len(batch)
        batch_size = batch.size(0)  # num_tasks
        Sx, Sy = batch
        chunk_size = batch_size // self.num_workers  # the number of examples to give each worker/proc
        futures = []
        if chunk_size <= 0:
            # give each worker a data point and thats it
            for t in range(batch_size):
                worker_info = self.workers[t]
                # makes non blocking rpc call on worker to and immediately returns a future object to wait the result
                future = rpc.rpc_async(to=worker_info, func=self.model.forward, args=(Sx[t], Sy[t]))
                futures.append(future)
        else:
            # each worker receives a chunk of size chunk_size
            chunk_idx = 0
            for worker_idx in range(self.num_workers):
                chunk_x = Sx[chunk_idx:chunk_idx+chunk_size]
                chunk_y = Sy[chunk_idx:chunk_idx+chunk_size]
                chunk_idx += chunk_size
                worker_info = self.workers[worker_idx]
                future = rpc.rpc_async(to=worker_info, func=self.model.forward, args=(chunk_x, chunk_y))
                futures.append(future)
        loss = 0
        for future in futures:
            loss = future.wait()
            loss += (1.0/batch_size)*loss
        return loss

    def finish(self):
        def end_master():
            rpc.shutdown()
        master_proc = Process(target=end_master, args=())
        master_proc.start()

def run_worker_process(rank, world_size):
    print('-- Worker run -- ')
    print(f'current process: {mp.current_process()}')
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    # other ranks are the workers (1 to world_size)
    rpc.init_rpc(f'worker{rank}', rank=rank, world_size=world_size)
    # block until all rpcs finish, and shutdown the RPC instance
    print('running shutdown')
    rpc.shutdown()
    print(f'shutdown complete rank {rank}')
    print(f'shutdown complete for process: {mp.current_process()}')

def master(world_size):
    print('-- Master run -- ')
    print(f'current process: {mp.current_process()}')
    # -- init master process --

    master = Master(world_size=world_size)

    # -- do training loop --
    optimizer = optim.Adam(params=master.model.parameters(), lr=0.1)
    # optimizer = DistributedOptimizer(optim.Adam, )
    for epoch in range(num_epochs):
        for batch_idx in range(iters):
            batch = get_ast_batch(batch_size=batch_idx)
            with dist_autograd.context() as context_id:
                loss = master.forward_parallel(batch)
                dist_autograd.backward(context_id, loss)
            optimizer.step()
            optimizer.zero_grad()

    # block until all rpcs finish, and shutdown the RPC instance, once master gets here everything is shut down
    # rpc.shutdown()
    master.finish()

if __name__ == '__main__':
    print('starting __main__')
    for rank in range(1, world_size):
        worker_proc = Process(target=run_worker_process, args=(rank, world_size))
        print(f'creating process object serially: pro_obj is = {worker_proc}')
        # worker_proc.start()
    master(world_size=2)
    print('Done!\a\n')

Unfortunately, I cannot use DDP because it relies on DistributedSampler which I cannot use since it might not work with the custom dataloaders I am using from torchmeta (their repo says it might not work).
Note that all processes need to be up to date with the new parameters so probably syncing with a distributed optizer is a good idea.

Related questions:

How to parallelize a training loop ever samples of a batch when CPU is only available in pytorch?
https://discuss.pytorch.org/t/how-to-parallelize-a-loop-over-the-samples-of-a-batch/32698/7
sync SGD tutorial https://pytorch.org/tutorials/intermediate/dist_tuto.html#distributed-training
this question has the same error but it lacks an explanation of how to fix it: https://discuss.pytorch.org/t/pytorch-distributed-example-code-hang-deadlock/83335/7


title: which object detection algorithms can extract the QR code from an image efficiently

body:
I am new to Object Detection, for now, I want to predict the QR code in images. I want to extract the QR code from the images, and only predict the QR code without the background information, and finally predict the exact number the QR code is representing, since I am using PyTorch, is there any object detection algorithm which is compatible to PyTorch that I could apply to this task?
(for what I mean by extracting, the raw input is an image,I want to change the input of the image to the QR code in the image).


title: Pytorch Resnet model error if FC layer is changed in Colab

body:
If I simply import the Resnet Model from Pytorch in Colab, and use it to train my dataset, there are no issues. However, when I try to change the last FC layer to change the output features from 1000 to 9, which is the number of classes for my datasets, the following error is obtained.
RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)

Working version:
import torchvision.models as models
#model = Net()
model=models.resnet18(pretrained=True)

# defining the optimizer
optimizer = Adam(model.parameters(), lr=0.07)
# defining the loss function
criterion = CrossEntropyLoss()
# checking if GPU is available
if torch.cuda.is_available():
    model = model.cuda()
    criterion = criterion.cuda()

Version with error:
import torchvision.models as models
#model = Net()
model=models.resnet18(pretrained=True)

# defining the optimizer
optimizer = Adam(model.parameters(), lr=0.07)
# defining the loss function
criterion = CrossEntropyLoss()
# checking if GPU is available
if torch.cuda.is_available():
    model = model.cuda()
    criterion = criterion.cuda()
model.fc = torch.nn.Linear(512, 9)

Error occurs in the stage where training occurs, aka
outputs = model(images)

How should I go about fixing this issue?


title: Pytorch Using nn.linear layer in 3d tensor

body:
I have an input tensor (num_data,channel,num_node)=(10,8,6) and used nn.Linear(6,1) to make output tensor (10,8,1).
I wonder whether like general conv layer with groups=1, all channel output is concatenated to make new output or like general conv layer with groups=8, all channel output separably used make new output.


title: VGG model not converging on CIFAR10 dataset using Pytorch

body:
My code is available at
https://www.kaggle.com/aman0807/vgg-cifar10/execution
As you can see, the validation loss diverges from the start of the training.
I have tried with Adam optimizer as well as SGD optimizer. I cannot figure out what it is that I am doing incorrectly. Please point me in the right direction.
Edit: Included entire code here
# Importing Dependencies

import os
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision.datasets import CIFAR10
from torchvision import transforms
from torch.utils.data import DataLoader
from tqdm import tqdm
from datetime import datetime

# Defining model
arch = [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M']

class VGGNet(nn.Module):
    def __init__(self, in_channels, num_classes):
        super().__init__()
        self.in_channels = in_channels
        self.conv_layers = self.create_conv_layers(arch)
        self.fcs = nn.Sequential(
            nn.Linear(in_features=512*1*1, out_features=4096),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(in_features=4096, out_features=4096),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(4096, num_classes)
        )

    def forward(self, x):
        x = self.conv_layers(x)
        # print(x.shape)
        x = x.reshape(x.shape[0], -1)
        x = self.fcs(x)
        return x

    def create_conv_layers(self, arch):
        layers = []
        in_channels = self.in_channels

        for x in arch:

            if type(x) == int:

                out_channels = x
                layers += [nn.Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
                        nn.BatchNorm2d(x),
                        nn.ReLU(),
                        ]

                in_channels = x

            elif x =='M':
                layers += [nn.MaxPool2d(kernel_size=(2, 2), stride=(2, 2))]

        return nn.Sequential(*layers)

# Hyperparameters and settings

device = "cuda" if torch.cuda.is_available() else "cpu"
print(device)
TRAIN_BATCH_SIZE = 64
VAL_BATCH_SIZE = 16
EPOCHS = 50

train_data = CIFAR10(root=".", train=True,
                    transform=transforms.Compose([transforms.ToTensor()]), download=True)

# print(len(train_data))
val_data = CIFAR10(root=".", train=False,
                    transform=transforms.Compose([transforms.ToTensor()]), download=True)
# print(len(val_data))

train_loader = DataLoader(train_data, batch_size=TRAIN_BATCH_SIZE, shuffle=True, num_workers=8)
val_loader = DataLoader(val_data, batch_size=VAL_BATCH_SIZE, shuffle=True, num_workers=8)
# print(len(train_loader))
# print(len(val_loader))

num_train_batches = int(len(train_data)/TRAIN_BATCH_SIZE)
num_val_batches = int(len(val_data)/VAL_BATCH_SIZE)

# Training and Val Loop

model = VGGNet(3, 10).to(device)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
# optim = torch.optim.Adam(model.parameters(), lr=0.01)

scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, patience=10, verbose=True)

# save_path = os.path.join(r"trained_models", f'{datetime.now().strftime("%m%d_%H%M%S")}.pth')

def train_val():

    for epoch in range(1, EPOCHS+1):
        print(f"Epoch: {epoch}/20")

        model.train()
        total_loss = 0
        for data in train_loader:
            image, target = data[0], data[1]
            image, target = image.to(device), target.to(device)
            optimizer.zero_grad()
            output = model(image)
            loss = criterion(output, target)
            total_loss += loss.item()
            loss.backward()
            optimizer.step()
        print(f"Loss : {total_loss / num_train_batches}")

        save_path = os.path.join(r"trained_models", f'{datetime.now().strftime("%m%d_%H%M%S")}_{epoch}.pth')

        if epoch % 5 == 0:
            torch.save(model.state_dict(), save_path)

        with torch.no_grad():
            model.eval()
            total_val_loss = 0
            for data in val_loader:
                image, target = data[0], data[1]
                image, target = image.to(device), target.to(device)
                output = model(image)
                val_loss = criterion(output, target)
                total_val_loss += val_loss

            total_val_loss = total_val_loss/num_val_batches
            print(f"Val Loss: {total_val_loss}")

            scheduler.step(total_val_loss)

The output is :
> Epoch: 1/20 Loss : 1.3286100650795292 Val Loss: 1.3787670135498047
> Epoch: 2/20 Loss : 0.822020811685832 Val Loss: 0.948610246181488
> Epoch: 3/20 Loss : 0.6018326392113476 Val Loss: 0.9581698775291443
> Epoch: 4/20 Loss : 0.47134833609764004 Val Loss: 1.2446043491363525
> Epoch: 5/20 Loss : 0.35625831704114524 Val Loss: 0.8038020730018616
> Epoch: 6/20 Loss : 0.27602518926566605 Val Loss: 0.6090452075004578
> Epoch: 7/20 Loss : 0.21279048924686128 Val Loss: 0.6626076102256775
> Epoch: 8/20 Loss : 0.16782210255280214 Val Loss: 0.6386368870735168
> Epoch: 9/20 Loss : 0.12904227719518205 Val Loss: 0.8135524988174438
> Epoch: 10/20 Loss : 0.10961572862077902 Val Loss: 0.727300226688385
> Epoch: 11/20 Loss : 0.08377284912137456 Val Loss: 0.7346469163894653
> Epoch: 12/20 Loss : 0.07044737199237916 Val Loss: 0.8241418600082397
> Epoch: 13/20 Loss : 0.06040401630707726 Val Loss: 0.8411757349967957
> Epoch: 14/20 Loss : 0.05157513573171604 Val Loss: 0.9980310201644897
> Epoch: 15/20 Loss : 0.04703645325243019 Val Loss: 0.7441162467002869
> Epoch: 16/20 Loss : 0.039386494244257594 Val Loss: 0.7185537219047546
> Epoch: 17/20 Loss : 0.0361507039006692 Val Loss: 0.7251362800598145
> Epoch    17: reducing learning rate of group 0 to 1.0000e-03. Epoch:
> 18/20 Loss : 0.010131187833331622 Val Loss: 0.6911444067955017 Epoch:
> 19/20 Loss : 0.004273188020082817 Val Loss: 0.6758599877357483 Epoch:
> 20/20 Loss : 0.0023282255553611917 Val Loss: 0.6790934801101685 Epoch:
> 21/20 Loss : 0.002249847758697408 Val Loss: 0.6877240538597107 Epoch:
> 22/20 Loss : 0.0017385115527510854 Val Loss: 0.6955451369285583 Epoch:
> 23/20 Loss : 0.0015288436127294692 Val Loss: 0.6895047426223755 Epoch:
> 24/20 Loss : 0.0013521527944272392 Val Loss: 0.6986600160598755 Epoch:
> 25/20 Loss : 0.001302041847793007 Val Loss: 0.7062056660652161 Epoch:
> 26/20 Loss : 0.0009422088254753626 Val Loss: 0.721610426902771 Epoch:
> 27/20 Loss : 0.00092220353266205 Val Loss: 0.7185065150260925 Epoch:
> 28/20 Loss : 0.000855279816558849 Val Loss: 0.7262870669364929 Epoch
> 28: reducing learning rate of group 0 to 1.0000e-04. Epoch: 29/20 Loss
> : 0.0009319903464029908 Val Loss: 0.7129291892051697 Epoch: 30/20 Loss
> : 0.0009213638452758586 Val Loss: 0.7219230532646179 Epoch: 31/20 Loss
> : 0.000790543920696094 Val Loss: 0.7180697321891785 Epoch: 32/20 Loss
> : 0.0007001494628562334 Val Loss: 0.7132201194763184 Epoch: 33/20 Loss
> : 0.000848956296528779 Val Loss: 0.720444917678833 Epoch: 34/20 Loss :
> 0.0007309111221651813 Val Loss: 0.7152464389801025 Epoch: 35/20 Loss : 0.0008760697004345624 Val Loss: 0.7213227152824402 Epoch: 36/20 Loss : 0.0007820251893755433 Val Loss: 0.7167501449584961 Epoch: 37/20 Loss : 0.0008780398232813891 Val Loss: 0.7176992297172546 Epoch: 38/20 Loss : 0.0006976223276931253 Val Loss: 0.7317324280738831 Epoch: 39/20 Loss : 0.0007082251550411605 Val Loss: 0.7190949320793152 Epoch    39: reducing learning rate of group 0 to 1.0000e-05. Epoch: 40/20 Loss :
> 0.0007200245124846 Val Loss: 0.7223398089408875 Epoch: 41/20 Loss : 0.0007270275670335591 Val Loss: 0.721064031124115 Epoch: 42/20 Loss : 0.0008191512905424633 Val Loss: 0.7175146341323853 Epoch: 43/20 Loss : 0.0007468872463225763 Val Loss: 0.7313894629478455 Epoch: 44/20 Loss : 0.0007858102934361841 Val Loss: 0.7234065532684326 Epoch: 45/20 Loss : 0.0008550570492202036 Val Loss: 0.7210699319839478 Epoch: 46/20 Loss : 0.0007921795028490795 Val Loss: 0.7209402322769165 Epoch: 47/20 Loss : 0.0007907233434224173 Val Loss: 0.7207257747650146 Epoch: 48/20 Loss : 0.0008106642895507929 Val Loss: 0.7174465656280518 Epoch: 49/20 Loss : 0.0008104693277185383 Val Loss: 0.7192382216453552 Epoch: 50/20 Loss : 0.0007664988370204754 Val Loss: 0.722599446773529 Epoch    50: reducing learning rate of group 0 to 1.0000e-06.


title: Difference between numpy.argsort and torch.argsort

body:
I have checked argsort function of pytorch and numpy are same by definition.
However, when I use those functions along different size of random tensor(or ndarray), It makes difference. I have no idea why this difference is observed.
Example code:

and results below,


title: NumPy: pass subset of multi-dimensional indices to array

body:
Suppose I have an array, and a (sub)set of indices:
arr = np.array([[[0,1,2],
                 [3,4,5],
                 [6,7,8]],

                [[9, 10,11],
                 [12,13,14],
                 [15,16,17]]])
idx = [1,2]

And I wish to get, for each element of dim=0 of the array, the respective idx slice, i.e.:
>>> arr[:, idx[0], idx[1]]
array([ 5, 14])

Is there a way to do such slicing without hard-coding each index? Something like:
ar[:, *idx]

Note, the following is my current workaround:
idx = [slice(a.shape[0]), *idx]
a[idx]

but I was wondering if NumPy (or PyTorch, or a related library) supported a more 'elegant'/natural multi-dimensional indexing syntax for such cases.


title: What output_padding does in nn.ConvTranspose2d?

body:
What is the working of Output_padding in Conv2dTranspose? Please Help me to understand this?
Conv2dTranspose(1024, 512, kernel_size=3, stride=2, padding=1, output_padding=1)


I'm trying to combine the CausalConv1d with Conv2d as the encoder of my VAE. But I got this error which is produced on Encoder part. The CausalConv1d is implemented by a nn.Conv1d network, So it should only have 3-dimensional weight, but why the error says expected 4-dimensional?  And I have another question, why I can't use a single int but only tuple in Pycharm when I set the "kernel_size", "stride" etc. parameters in a Convs layer? Although the official document said both int and tuple are valid. Here is the traceback:
Traceback (most recent call last):
  File "training.py", line 94, in <module>
    training(args)
  File "training.py", line 21, in training
    summary(model, torch.zeros(64, 1, 784), show_input=True, show_hierarchical=False)
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\pytorch_model_summary\model_summary.py", line 118, in summary
    model(*inputs)
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "F:\VAE_reservoir_computing\VAE.py", line 34, in forward
    mu, log_var = self.encoder.forward(image)
  File "F:\VAE_reservoir_computing\CausalCnn_Reservoir.py", line 30, in forward
    x = self.conv1d(x.view(x.shape[0], 1, 784))
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\torch\nn\modules\container.py", line 119, in forward
    input = module(input)
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "F:\VAE_reservoir_computing\CausalConv1d.py", line 21, in forward
    conv1d_out = self.conv1d(x)
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\torch\nn\modules\conv.py", line 263, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\torch\nn\modules\conv.py", line 260, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [256, 1, 3, 3], but got 3-dimensional input of size [64, 1, 786] instead

Here is my code of the Encoder-Decoder part:
import torch
import torch.nn as nn
from CausalConv1d import CausalConv1d

class CausalReservoirEncoder(nn.Module):
    def __init__(self, in_channels, out_channels, num_filters, z_dim,  *args):
        super(CausalReservoirEncoder, self).__init__()
        self.num_filters = num_filters
        self.z_dim = z_dim
        hidden_filters = num_filters

        self.Conv1d = nn.Sequential(
            CausalConv1d(in_channels,out_channels,kernel_size=(3,3),dilation=1,A=False),
            nn.LeakyReLU()
        )
        for p in self.parameters():
            p.requires_grad = False

        self.encoder = nn.Sequential(
            nn.Conv2d(out_channels, self.num_filters, kernel_size=(4, 4), padding=(1, 1), stride=(2, 2)), # 28x28 -> 14x14
            nn.LeakyReLU(),
            nn.Conv2d(hidden_filters, 2 * hidden_filters, kernel_size=(4, 4), padding=(1, 1), stride=(2, 2)), # 14x14 -> 7x7
            nn.LeakyReLU(),
            nn.Flatten(),
            nn.Linear(2*hidden_filters*7*7, self.z_dim)

        )

    def forward(self, x):
        x = self.Conv1d(x.view(x.shape[0], 1, 784))
        h_e = self.encoder(x.view(x.shape[0], -1, 28, 28))
        mu, log_var = torch.chunk(h_e, 2, dim=1)

        return mu, log_var

class CausalReservoirDecoder(nn.Module):
    def __init__(self, z_dim, out_channels, num_filters, **kwargs):
        super(CausalReservoirDecoder, self).__init__()
        self.z_dim = z_dim
        self.num_filters = num_filters
        hidden_filters = num_filters

        self.linear = nn.Sequential(
            nn.Linear(self.z_dim, 2*hidden_filters*7*7),
            nn.LeakyReLU()
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(2*hidden_filters, hidden_filters, kernel_size=(4,4), padding=(1,1), stride=(2,2)), #7x7 -> 14x14
            nn.LeakyReLU(),
            nn.ConvTranspose2d(hidden_filters, out_channels, kernel_size=(4,4), padding=(1,1), stride=(2,2)), # 14x14 -> 28x28
            nn.Sigmoid()
        )

    def forward(self, z):
        x = self.linear(z)
        x = x.reshape(x.shape[0], -1, 7, 7)
        x_recon = self.decoder(x)

        return x_recon

Here is the implementation of the CausalConv1d:
import torch.nn as nn
import torch.nn.functional as F

class CausalConv1d(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, dilation, A=False, *args, **kwargs):
        super(CausalConv1d, self).__init__()
        self.kernel_size = kernel_size
        self.dilation = dilation
        self.A = A

        self.padding = (kernel_size[0] - 1) * dilation + A * 1

        self.conv1d = nn.Conv1d(in_channels,out_channels,self.kernel_size,stride=(1,1),padding=(0,0),dilation=dilation,**kwargs)

    def forward(self, x):
        x = F.pad(x, (self.padding, 0))
        conv1d_out = self.conv1d(x)
        if self.A:
            return conv1d_out[:,:,: -1]
        else:
            return conv1d_out

So anyone can give me some suggestions?


title: Why some weights of GPT2Model are not initialized?

body:
I am using the GPT2 pre-trained model for a research project and when I load the pre-trained model with the following code,
from transformers.models.gpt2.modeling_gpt2 import GPT2Model
gpt2 = GPT2Model.from_pretrained('gpt2')

I get the following warning message:

Some weights of GPT2Model were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

From my understanding, it says that the weights of the above layers are not initialized from the pre-trained model. But we all know that attention layers ('attn') are so important in GPT2 and if we can not have their actual weights from the pre-trained model, then what is the point of using a pre-trained model?
I really appreciate it if someone could explain this to me and tell me how I can fix this.


title: Pytorch memory leak when threading

body:
I get significant memory leak when running Pytorch model to evaluate images from dataset.
Every new image evaluation is started in a new thread.
It doesn't matter if the code waits for the thread to finish or not. When the threads are not used (just evaluate function is called), there is no any leak. I've tried to delete the thread variable every iteration, but that doesn't help.
Here is the code:
hidden_sizes = [6336, 1000]
class Net(torch.nn.Module):
    def __init__ (self):
        super(Net, self).__init__()
        self.fc1 = torch.nn.Linear(hidden_sizes[0], hidden_sizes[1])

    def forward(self, x):
        x= x.view(-1,hidden_sizes[0])
        x = torch.nn.functional.log_softmax(self.fc1(x), dim=1)
        return x
#---------------------------------------------------------------
def newThread(i):
   image=cv2.imread(pathx+filenames[i], cv2.IMREAD_GRAYSCALE)

   images = tran(image)
   images = tran1(images)
   images = images.unsqueeze(0)

   #Run model
   with torch.no_grad():
      logps = model(images)
   ps = torch.exp(logps)
   probab = list(ps.numpy()[0])
   pred_label = probab.index(max(probab))
#---------------------------------------------------------------
model = Net ()
model.load_state_dict(torch.load("test_memory_leak.pt"))
#normalize image
tran = transforms.ToTensor()
tran1 = transforms.Normalize((0.5,), (0.5,))

pathx="images\\"
filenames=os.listdir(pathx)

for i in range(len(filenames)):

    thread1  = threading.Thread(target = newThread, args = (i, ))
    thread1.start()
    thread1.join()

What could be the reason for that?
UPD: Tried to detect memory leaks with guppy, but the reason still isn't clear. Here are some stats: First table is program memory usage on the beginning, the second one is when the memory usage increased x2.4 fold (up to 480mb) after analyzing 1000 images:
Partition of a set of 260529 objects. Total size = 33587422 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  72188  28  9511536  28   9511536  28 str
     1  70921  27  5418336  16  14929872  44 tuple
     2  32926  13  2526536   8  17456408  52 bytes
     3  16843   6  2434008   7  19890416  59 types.CodeType
     4   2384   1  2199952   7  22090368  66 type
     5  14785   6  2010760   6  24101128  72 function
     6   4227   2  1631384   5  25732512  77 dict (no owner)
     7    794   0  1399928   4  27132440  81 dict of module
     8   2384   1  1213816   4  28346256  84 dict of type
     9     38   0   704064   2  29050320  86 dict of torch.tensortype
<575 more rows. Type e.g. '_.more' to view.>
-------------------------------------------

Partition of a set of 265841 objects. Total size = 34345930 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  72203  27  9523346  28   9523346  28 str
     1  70924  27  5418584  16  14941930  44 tuple
     2  32928  12  2530722   7  17472652  51 bytes
     3  16844   6  2434152   7  19906804  58 types.CodeType
     4   2384   1  2200488   6  22107292  64 type
     5  14786   6  2010896   6  24118188  70 function
     6   4232   2  1637736   5  25755924  75 dict (no owner)
     7    794   0  1399928   4  27155852  79 dict of module
     8   2384   1  1213816   4  28369668  83 dict of type
     9    265   0   840672   2  29210340  85 set
<577 more rows. Type e.g. '_.more' to view.>


title: Pytorch custom CUDA extension build fails for torch 1.6.0 or higher

body:
I have a custom CUDA extension for pytorch (https://pytorch.org/tutorials/advanced/cpp_extension.html), which used to work fine with pytorch1.4, CUDA10.1, and Titan Xp GPUs. However, recently we changed our system to new A40 GPUs and CUDA11.1. When I try to build my custom pytorch extension using CUDA11.1, pytorch 1.8.1, gcc 9.3.0, and Ubuntu 20.04 I get the following errors:
$ python3 setup.py install
running install
running bdist_egg
running egg_info
creating cuda_test.egg-info
writing cuda_test.egg-info/PKG-INFO
writing dependency_links to cuda_test.egg-info/dependency_links.txt
writing top-level names to cuda_test.egg-info/top_level.txt
writing manifest file 'cuda_test.egg-info/SOURCES.txt'
reading manifest file 'cuda_test.egg-info/SOURCES.txt'
writing manifest file 'cuda_test.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'cuda_test' extension
creating /path/to/code/cuda/test/build
creating /path/to/code/cuda/test/build/temp.linux-x86_64-3.7
Emitting ninja build file /path/to/code/cuda/test/build/temp.linux-x86_64-3.7/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/1] /cm/shared/apps/cuda11.1/toolkit/11.1.1/bin/nvcc --generate-dependencies-with-compile --dependency-output /path/to/code/cuda/test/build/temp.linux-x86_64-3.7/test_cuda.o.d -I/path/to/code/venv/lib/python3.7/site-packages/torch/include -I/path/to/code/venv/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/path/to/code/venv/lib/python3.7/site-packages/torch/include/TH -I/path/to/code/venv/lib/python3.7/site-packages/torch/include/THC -I/cm/shared/apps/cuda11.1/toolkit/11.1.1/include -I/path/to/code/venv/include/python3.7m -c -c /path/to/code/cuda/test/test_cuda.cu -o /path/to/code/cuda/test/build/temp.linux-x86_64-3.7/test_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=cuda_test -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++14
FAILED: /path/to/code/cuda/test/build/temp.linux-x86_64-3.7/test_cuda.o
/cm/shared/apps/cuda11.1/toolkit/11.1.1/bin/nvcc --generate-dependencies-with-compile --dependency-output /path/to/code/cuda/test/build/temp.linux-x86_64-3.7/test_cuda.o.d -I/path/to/code/venv/lib/python3.7/site-packages/torch/include -I/path/to/code/venv/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/path/to/code/venv/lib/python3.7/site-packages/torch/include/TH -I/path/to/code/venv/lib/python3.7/site-packages/torch/include/THC -I/cm/shared/apps/cuda11.1/toolkit/11.1.1/include -I/path/to/code/venv/include/python3.7m -c -c /path/to/code/cuda/test/test_cuda.cu -o /path/to/code/cuda/test/build/temp.linux-x86_64-3.7/test_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=cuda_test -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++14
/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/arithmetic.h(256): error: identifier "FLT_MIN" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/arithmetic.h(274): error: identifier "DBL_MIN" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(190): error: identifier "DBL_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(228): error: identifier "DBL_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(243): error: identifier "DBL_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(293): error: identifier "DBL_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(406): error: identifier "DBL_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(498): error: identifier "DBL_MAX" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(562): error: identifier "DBL_MAX_EXP" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(565): error: identifier "DBL_MANT_DIG" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(630): error: identifier "DBL_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(119): error: identifier "FLT_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(137): error: identifier "FLT_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(147): error: identifier "FLT_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(170): error: identifier "FLT_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(249): error: identifier "FLT_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(327): error: identifier "FLT_MAX" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(375): error: identifier "FLT_MAX_EXP" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(377): error: identifier "FLT_MANT_DIG" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(420): error: identifier "FLT_EPSILON" is undefined

I also wrote a simple test code to verify that my larger CPP/CUDA code isn't the culprit, which produced the same error messages. I also checked if arithmetic.h and catrig.h include <cfloat>, which should provide the {FLT,DBL}_{MIN,MAX,EPSILON,MANT_DIG} definitions but this looks all normal, since it's standard NVIDIA code.
Let me know if anyone has encountered a similar problem or would know a solution.
---- UPDATE ----
Here are a couple of more things that I've tried:

The CUDA code compiles when I use CUDA10.1, pytorch 1.4.0, gcc 9.3.0, and Ubuntu 20.04.
Using pytorch 1.5.1 instead generates the following error:
/usr/include/c++/9/bits/stl_function.h(437): error: identifier "__builtin_is_constant_evaluated" is undefined
but this can be solved by downgrading gcc to version 7.5.
Using pytorch 1.6.0 or higher instead always results in the errors reported in the beginning, even when using gcc-7.


title: Google Colab + Pytorch: RuntimeError: No CUDA GPUs are available

body:
Screenshot of error:

Hello, I am trying to run this Pytorch application, which is a CNN for classifying dog and cat pics.
I am using Google Colab for the GPU, but for some reason, I get RuntimeError: No CUDA GPUs are available. This is weird because I specifically both enabled the GPU in Colab settings, then tested if it was available with torch.cuda.is_available(), which returned true.
The weirdest thing is that this error doesn't appear until about 1.5 minutes after I run the code. You would think that if it couldn't detect the GPU, it would notify me sooner.
I've had no problems using the Colab GPU when running other Pytorch applications using the exact same notebook. I can only imagine it's a problem with this specific code, but the returned error is so bizarre that I had to ask on StackOverflow to make sure.


title: AttributeError: module 'torch' has no attribute 'rfft' with PyTorch

body:
I am getting an error using a code that should work according to the documentation.
The goal is to calculate the Feature Similarity Index Measure (FSIM) using the piq Python library.
Terminal Output:
TiffPage 1: ByteCounts tag is missing
Traceback (most recent call last):
  File "...\.venv\lib\site-packages\IPython\core\interactiveshell.py", line 3441, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-3044cfc208ce>", line 1, in <module>
    runfile('.../stackoverflow.py', wdir='...')
  File "...\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "...\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File ".../stackoverflow.py", line 15, in <module>
    main()
  File "...\.venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File ".../stackoverflow.py", line 10, in main
    fsim_index: torch.Tensor = piq.fsim(x, y, data_range=1., reduction='none')
  File "...\.venv\lib\site-packages\piq\fsim.py", line 84, in fsim
    pc_x = _phase_congruency(
  File "...\.venv\lib\site-packages\piq\fsim.py", line 241, in _phase_congruency
    imagefft = torch.rfft(x, 2, onesided=False)
AttributeError: module 'torch' has no attribute 'rfft'

Code:
from skimage import io
import torch
import piq

@torch.no_grad()
def main():
    x = torch.tensor(io.imread('scikit_image\cover\cover_1.tiff')).permute(2, 0, 1)[None, ...] / 255.
    y = torch.tensor(io.imread('scikit_image\stego\stego_1.tiff')).permute(2, 0, 1)[None, ...] / 255.

    fsim_index: torch.Tensor = piq.fsim(x, y, data_range=1., reduction='none')

    print(fsim_index)

if __name__ == "__main__":
    main()


title: Why a larger neural network back-propagate faster than a smaller one

body:
I wrote the following two NN in pytorch for image segmentation:
The smaller one:

class ConvNetV0(nn.Module):

def __init__(self):
    super(ConvNetV0, self).__init__()
    self.conv1 = nn.Conv2d(3, 30, 4, padding=2)
    self.conv2 = nn.Conv2d(30, 50, 16, padding=7, bias=True)
    self.conv3 = nn.Conv2d(50, 20, 2, stride=2)
    self.conv4 = nn.Conv2d(20, 2, 2, stride=2)

def forward(self, x):
    x = self.conv1(x)
    x = F.relu(x)
    x = self.conv2(x)
    x = F.relu(x)
    x = self.conv3(x)
    x = F.relu(x)
    y = self.conv4(x)
    return y

The larger one:

class ConvNetV1(nn.Module):

def __init__(self):
    super(ConvNetV1, self).__init__()
    self.conv0 = nn.Conv2d(3, 50, 4, padding=1, stride=2)
    self.conv_r1 = nn.Conv2d(50, 40, 15, padding=7, bias=True)
    self.conv_r2 = nn.Conv2d(40, 25, 3, padding=1)
    self.conv_r3 = nn.Conv2d(25, 25, 2, stride=2)
    # self.conv_r3 = nn.MaxPool2d(2, stride=2)
    self.conv_b1 = nn.Conv2d(50, 15, 4, padding=1, stride=2)
    self.conv1 = nn.Conv2d(40, 2, 1)

def forward(self, x):
    x = self.conv0(x)
    x = F.relu(x)

    x1 = self.conv_r1(x)
    x1 = F.relu(x1)
    x1 = self.conv_r2(x1)
    x1 = F.relu(x1)
    x1 = self.conv_r3(x1)

    x2 = self.conv_b1(x)

    y = torch.cat([x1, x2], dim=1)
    y = self.conv1(y)
    return y

However during training at mini-batch size = 8. The smaller net take 2s to complete one iteration while the larger one only take 0.3s to complete.

I also observe that the no of parameters ratio between the two net is around 5:6. However, during training the smaller net only takes 1GB VRAM while the larger takes 3GB. Since My 1050ti have 4GB VRAM. I would like trade off memory for speed. Any idea how I can do so?


title: Problem while loading Images from Training folder having only one folder

body:
I am trying to run Siamese Network implemented in Pytorch for my custom made data set.
Github Link for Code : https://github.com/viral-parmar/Voice_Dissimilarity
When I try to run the code and provide the path to Training data having only one folder with 10 images only. The Data Loading part of Code Take lots of execution time and never gets executed successfully. But When I provide the path to same Training Data, now having like 25 Folders Each having 10 Images, the loading part gets executed successfully. I don't understand the problem with code.

I tried it running on Google Co Lab and also on my local computer, but the results are same.

folder_dataset = dset.ImageFolder(root=Config.training_dir)

siamese_dataset = SiameseNetworkDataset(imageFolderDataset=folder_dataset,
                                    transform=transforms.Compose([transforms.Resize((100,100)),
                                                                  transforms.ToTensor()
                                                                  ])
                                   ,should_invert=False)

#Visualising some of the data
vis_dataloader = DataLoader(siamese_dataset,
                    shuffle=True,
                    num_workers=8,
                    batch_size=8)
dataiter = iter(vis_dataloader)

example_batch = next(dataiter)
concatenated = torch.cat((example_batch[0],example_batch[1]),0)
imshow(torchvision.utils.make_grid(concatenated))
print(example_batch[2].numpy())

It takes long and never gets executed completely.


title: How to prevent buildup of memory usage on GPU PyTorch 0.3

body:
Using this code I get an out of memory error after a few iterations. I've isolated the problem to appending to the self.ground_truth variables, I think for some reason it stays stored on the GPU. x and gt_class_ids start out on the GPU. I'm trying to load it into the CPU so that the self.ground_truth variables don't take up memory on the GPU. I'm kind of new to PyTorch and Python and also I'm restricted to version 0.3

x = x.cpu()[0]
gt_class_ids = gt_class_ids.cpu()[0]

for feature in x:
    self.ground_truth.append(feature)
for id in gt_class_ids:
    self.ground_truth_ids.append(id)

torch.cuda.empty_cache()


title: How can I run pytorch with multiple graphic cards?

body:
I have 4 graphic cards which I want to utilize to pytorch.
I have this net:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5, 1)
        self.conv2 = nn.Conv2d(20, 50, 5, 1)
        self.fc1 = nn.Linear(4*4*50, 500)
        self.fc2 = nn.Linear(500, 10)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2, 2)
        x = x.view(-1, 4*4*50)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

How can I use them on this net?


title: Libtorch build failed. In "qlinear_unpack.cpp", a class has no member "unpack"

body:
Update:
I tried v1.1.0 source that my teammate downloaded before, then this problem disappeared. It seems that the master branch(190604) has the following issue. Try tag v1.1.0 instead of master.

I downloaded pytorch source to build libtorch from github. Using following command:

git clone [pytorch git] --recursive

Then run

python setup.py build

Then get following error msgs:

[ 46%] Building CXX object caffe2/CMakeFiles/caffe2.dir/__/aten/src/TH/THTensorMoreMath.cpp.o
/home/in4ight/libtorch/pytorch/aten/src/ATen/native/quantized/cpu/qlinear_unpack.cpp: In member function ‘at::Tensor at::native::{anonymous}::QLinearUnpackWeightInt8::operator()(at::Tensor)’:
/home/in4ight/libtorch/pytorch/aten/src/ATen/native/quantized/cpu/qlinear_unpack.cpp:37:12: error: ‘class fbgemm::PackBMatrix<signed char>’ has no member named ‘unpack’
     packB->unpack(weight_ptr_int8);

How to build libtorch correctly?