title: unknown tokens appearing when using pack_padded_seq and not appearing without packing

body:
I'm trying to implement a text generation model using Pytorch and LSTM and I have tried to omit padding tokens from the input that goes to LSTM by using pack_padded_sequence. But, for some reason, it produces unk tokens quite frequently which somehow indicates that the model is quite sure that the next token should be unk. When I test the model without packing (Having all instances padded) it does not ever generate a unk token.

class RNNModule(nn.Module):
   def __init__(self, n_vocab, seq_size, embedding_size, lstm_size):
        super(RNNModule, self).__init__()
        self.seq_size = seq_size
        self.lstm_size = lstm_size
        self.embedding = nn.Embedding(n_vocab, embedding_size, padding_idx=0)

        self.lstm = nn.LSTM(embedding_size,
                            lstm_size,
                            num_layers=flags.n_layers,
                            batch_first=True
                            )
        self.dense = nn.Linear(lstm_size, n_vocab)

    def forward(self, x,length,prev_state):
        embed = self.embedding(x)
        packed_input = db.pack_src(embed,length)
        packed_output, state = self.lstm(packed_input,prev_state)
        padded,_ = db.pad_pack(packed_output)
        logits = self.dense(padded)
        return logits, state

    def zero_state(self, batch_size = flags.batch_size):
        return (torch.zeros(flags.n_layers, batch_size, self.lstm_size),
                torch.zeros(flags.n_layers, batch_size, self.lstm_size))```

This is the flow of src into the model and its loss calculation

        trg_batch = torch.t(trg_batch).to(device)
        src_batch = src_batch.to(device)

        logits, (state_h, state_c) = net(src_batch,src_lengths,(state_h,state_c))
        loss = criterion(logits.transpose(1, 2), trg_batch)```

These are the function I call when packing/unpacking

    def pack_src(seld,embed,length):
        return rnn_utils.pack_padded_sequence(embed,length,batch_first=True)

    def pad_pack(self,packed_output):
        return rnn_utils.pad_packed_sequence(packed_output)

    def pad_seq(self,batch):
        return torch.nn.utils.rnn.pad_sequence(batch,batch_first=True).long()


title: Can I extract all indices that correspond to a certain key in a pytorch tensor?

body: Say I have a pytorch tensor tensor([3,5,7,3,9,3,0]). I'd like to extract the indices where 3 appears, i.e. tensor([0,3,5]). Is there a built-in function for this?


title: pyTorch Resnet implementation with two outputs

body:
I am trying to use pyTorch's Resnet to detect an image's object, so by classifying it and by detecting the object position inside the image.
I managed to write the following code, however I am stuck. I don't understand what I should do in order to achieve that.
Any help would be appreciated. Thanks in advance.
class NewResNet(ResNet):
    def __init__(self,pretrained = True):
        super(NewResNet,self).__init__(BasicBlock,[2,2,2,2])
        rect_input = BasicBlock.expansion*512
        self.rectangles = nn.Linear(rect_input, 4)
        self.load_state_dict(load_state_dict_from_url(model_urls['resnet18']),strict=False)

    def _forward_impl(self,x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        xf = torch.flatten(x, 1)

        output1 = ?
        output2 = ?

        return output1, output2


title: Albert Model Display and Copying weights inside Albert

body:
I wanted to try something.
There are 2 experiments, one on Bert an one on Albert. The task is, Train Bert upto Layers 5 , keeping the other 7 fixed, and then during test time, copy the weights of the 5th layer onto layers 6-12. I was successfully able to do this, as Bert has seperate parameters, and I can manipulate them seperately.
Whereas in Albert, the Architecture is such that all the weights are shared from layers 1-12, so, although I can train them till layer 5, they aren't accessible seperately, so, I am unable to copy over the weights onto further layers.
Any suggestions how I might go about this?
I tried printing the entire model, for both Bert and Albert.
(embeddings): BertEmbeddings(
      (word_embeddings): Embedding(30522, 768, padding_idx=0)
      (position_embeddings): Embedding(512, 768)
      (token_type_embeddings): Embedding(2, 768)
      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
(encoder): BertEncoder(
      (layer): ModuleList(
        (0): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
(1): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )

And so on, for layers uptil 11. (They are seperately accessible).
For Albert:
Model(
  (albert): AlbertModel(
    (embeddings): AlbertEmbeddings(
      (word_embeddings): Embedding(30000, 128, padding_idx=0)
      (position_embeddings): Embedding(512, 128)
      (token_type_embeddings): Embedding(2, 128)
      (LayerNorm): LayerNorm((128,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0, inplace=False)
    )
(encoder): AlbertTransformer(
      (embedding_hidden_mapping_in): Linear(in_features=128, out_features=768, bias=True)
      (albert_layer_groups): ModuleList(
        (0): AlbertLayerGroup(
          (albert_layers): ModuleList(
            (0): AlbertLayer(
              (full_layer_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (attention): AlbertAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0, inplace=False)
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              )
              (ffn): Linear(in_features=768, out_features=3072, bias=True)
              (ffn_output): Linear(in_features=3072, out_features=768, bias=True)
            )
          )
        )
      )
    )

The weights can't be copied directly, as they are the same weights. (I want to have weights trained only till layer 5, and then copy layer 5 weights till layer 12, without training it explicitly. How do I go about this?
'''


title: How to normalize input with an unknown distribution?

body:
I am building a Reinforcement Learning agent that operates in a set of unknown environments.
The sensor inputs given by these environments are dynamically generated, so it is very hard for me to tell in advance how large they are going to be. Some inputs might be in the range [0;1], others in [10000; 100000000]. Some might have low variance, but others could have very high variance.
This makes it difficult to normalize the inputs.
Sampling from the environments is expensive, so it would be wasteful to wait for many iterations of sampling before I know how to scale the inputs. Worse, some inputs might only return high values in special circumstances that are only encountered at runtime.
Is there any mechanism I can use as a preprocessor to normalize this completely unknown input data effectively? Preferably, this should not be too expensive to compute, because most of the inputs should be sane already. There are only a few inputs with crazy scaling.
Ideally, I would like to know if there is a standard function I can use that already does this. I am using pytorch, and this feels like a common enough problem that there ought to be a function for it.
There is also a second problem related to this one: Some inputs are categorical strings, and the categories are not known in advance. Currently, I just hash each string into a fixed-size vector, since this also works for data that hasn’t been seen before. I wonder if there is a better way to do it, or if pytorch has a function that already does this.


title: How to link torch from anaconda to python file?

body:
I have got python38,37 and anaconda on my machine. I did install torch via conda:
    conda --version
    conda 4.8.4

However, my python file can't import torch, running pip install torch gives famous no module nn.wrap found error. I can import torch in jupyter(anaconda distro) though.
I tried fixing nn.wrap bug, but failed:
    pip3 install https://download.pytorch.org/whl/cu90/torch-1.1.0-cp38-cp38m-win_amd64.whl

Or
    pip install torch===1.6.0 torchvision===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

Did not work.
I just want to use that torch from anaconda, how to import it instead of searching it in python3.8 dir?


title: syntax pytorch.nn functions neural networks, class or functions?

body:
I am puzzled by the syntax of the functions used in neural networks in pytorch.
Here is an example of how one can define a linear transformation layer: (cf. https://pytorch.org/docs/stable/generated/torch.nn.Linear.html)
m = nn.Linear(20, 30)
input = torch.randn(128, 20)
output = m(input)
print(output.size())
torch.Size([128, 30])

Can someone explain me where the expression nn.Linear(20,30)(input) comes from ? It disturbs me a bit.
Indeed, one can define a class neural network with such cosntructor : (for example)
class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes, p=0):
        super(NeuralNet, self).__init__()
        self.fc1            = nn.Linear(input_size, hidden_size, bias=True)
        self.fc2            = nn.Linear(hidden_size, hidden_size, bias=True)
        self.fc3            = nn.Linear(hidden_size, hidden_size, bias=True)
        self.fc4            = nn.Linear(hidden_size, num_classes, bias=False)
        self.dropout        = nn.Dropout(p=p)

and I was trying to write an attribute using numpy function, like:
  self.enter_reshape  = np.reshape(-1, input_size * input_size)
  self.exit_reshape   = np.reshape(input_size, num_classes / input_size)

or, using the view function from pytorch:
self.reshape = view(-1, self.num_flat_features())

The closest thing I know about is the partial function and closures, where one could write f(z)(x)(y). I looked into the definition of Linear, and I saw that linear is an object, but I don't see where they redefined __call__ magic function, which I thought would be used here when one calls the object.
So basically, can one explain what is up with such writting, and also, would it be possible to give to the neural network the numpy or view functions as attributes?


title: About pytorch, I can't understand why that's the output

body:
The code is as below about pytorch, it's about derivative, I think the output is 18 but it's 4.5, I don'k know why:
import torch
x = torch.ones(2, 2, requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()
out.backward()
print(x.grad)

The output:
tensor([[4.5000, 4.5000],
[4.5000, 4.5000]])
I think the derivative is 2*3*(1+2), so it should be:
tensor([[18, 18],
        [18, 18]])

Why the output is 4.5?
Some people think it's the mean method that make the derivative /4, but when I do the code "print(out)", the output is "tensor(27., grad_fn=)" rather than (4.5., grad_fn=), I'm a new learning of pytorch, so I don't know what it does with "tensor.mean()", but since the output of "print(out)" is 27, so I don't think there is a "/4" process in "tensor.mean()", so I don't think it should include "/4" process in the derivative computation, is that correct?(Please help me~)


title: How do I retain grads and also change device type in pytorch?

body: When I change my input variable from CPU to cuda, it loses all its grad and also loses its is_leaf status. How do I circumvent this? I want to keep the gradients and also change it to another device.


title: About torchmeta BatchMetaDataLoader

body:
dataset = tm_datasets.FC100(root='./data',

            num_classes_per_task=5,
            transform=Compose([ToTensor()])
            meta_train=True,
            download=True)

dataset = ClassSplitter(dataset, shuffle=shuffle,
            num_train_per_class=2,
            num_test_per_class=19)
dataloader = BatchMetaDataLoader(dataset, batch_size=2, drop_last=True)

for batch in dataloader:

    samples, sample_labels = batch["train"]

The samples'shape is torch.Size([2, 10, 3, 32, 32]).
Why sample_labesl is list which length is tow?
sample_labels's values is as follows:
sample_labels[0] = [
    [
        ('large_man-made_outdoor_things', 'trees'),
        ('large_man-made_outdoor_things', 'trees'),
        ('fruit_and_vegetables', 'household_electrical_devices'),
        ('fruit_and_vegetables', 'household_electrical_devices'),
        ('fish', 'trees'),
        ('fish', 'trees'),
        ('vehicles_2', 'fish'),
        ('vehicles_2', 'fish'),
        ('vehicles_1', 'household_furniture'),
        ('vehicles_1', 'household_furniture')
    ],
    [
        ('road', 'oak_tree'),
        ('road', 'oak_tree'),
        ('apple', 'television'),
        ('apple', 'television'),
        ('shark', 'pine_tree'),
        ('shark', 'pine_tree'),
        ('lawn_mower', 'trout'),
        ('lawn_mower', 'trout'),
        ('bus', 'bed'),
        ('bus', 'bed')
    ]
]

sample_labels[1] = tensor(
    [
        [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    ]
)


title: (Efficiently) expanding a feature mask tensor to match embedding dimensions

body:
I have a B (batch size) by F (feature count) mask tensor M, which I'd like to apply (element-wise multiply) to an input x.
...Thing is, my x has had its raw feature columns transformed into embeddings of non-constant width, so its total dimensions are B by E (embedded size).
My draft code is along the lines of:
# Given something like:
M = torch.Tensor([[0.2, 0.8], [0.5, 0.5], [0.6, 0.4]])  # B=3, F=2
x = torch.Tensor([[1, 2, 3, 4, 5], [6, 7, 8, 9, 0], [11, 12, 13, 14, 15]])  # E=5

feature_sizes = [2, 3]  # (Feature 0 embedded to 2 cols, feature 1 to 3)

# In forward() pass:
components = []
for ix, size in enumerate(feature_sizes):
    components.append(M[:, ix].view(-1, 1).expand(-1, size))
M_x = torch.cat(components, dim=1)

# Now M_x is (B, E) and can be mapped with x

> M_x = torch.Tensor([
>     [0.2, 0.4, 2.4, 3.6, 4],
>     [3, 3.5, 4, 4.5, 0],
>     [6.6, 7.2, 5.2, 5.6, 6],
> ])

My question is: are there any obvious optimizations I'm missing here? Is that for loop the right way to go about it, or is there some more direct way to achieve this?
I have control over the embedding process, so can store whatever representation is helpful and am e.g. not tied to feature_sizes list of ints.


title: CNN: accuracy and loss are increasing and decreasing

body:
I'am beginner  in deep learning, I created 3DCNN using Pytorch.
input image: 120 * 120 * 120
the problem that the accuracy and loss decrease and increase in the same interval [45,56].
Can you help me please ?
    def __init__(self):
        super(CNNModel, self).__init__()

        self.conv_layer1 = self._conv_layer_set(3, 32)

        self.conv_layer2 = self._conv_layer_set(32, 64)
        self.conv_layer3 = self._conv_layer_set(64, 128)
        self.conv_layer4 = self._conv_layer_set(128, 256)
        self.conv_layer5 = self._conv_layer_set(256, 512)

        self.fc1 = nn.Linear(512, 128)
        self.fc2 = nn.Linear(128, num_classes)
        self.relu = nn.LeakyReLU()
        self.batch=nn.BatchNorm1d(128)
        self.drop=nn.Dropout(p=0.5, inplace = True)

    def _conv_layer_set(self, in_c, out_c):
        conv_layer = nn.Sequential(
        nn.Conv3d(in_c, out_c, kernel_size=(3, 3, 3), padding=0),
        nn.LeakyReLU(),
        nn.MaxPool3d((2, 2, 2)),
        )
        return conv_layer

    def forward(self, x):
        # Set 1
        out = self.conv_layer1(x)
        out = self.conv_layer2(out)
        out = self.conv_layer3(out)
        out = self.conv_layer4(out)
        out = self.conv_layer5(out)
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        out = self.relu(out)
        out = self.batch(out)
        out = self.drop(out)
        out = self.fc2(out)
        #out = F.softmax(out, dim=1)
        return out

Result :
teration: 1/10  Loss: 0.8040086030960083  Accuracy: 47.023809523809526 %
Iteration: 2/10  Loss: 0.8323351740837097  Accuracy: 45.23809523809524 %
Iteration: 3/10  Loss: 0.8008261322975159  Accuracy: 50.595238095238095 %
Iteration: 4/10  Loss: 0.7527135610580444  Accuracy: 55.95238095238095 %
Iteration: 5/10  Loss: 0.7785584330558777  Accuracy: 51.19047619047619 %
Iteration: 6/10  Loss: 0.7463465929031372  Accuracy: 56.25 %
Iteration: 7/10  Loss: 0.8021382093429565  Accuracy: 52.083333333333336 %
Iteration: 8/10  Loss: 0.7705538868904114  Accuracy: 50.595238095238095 %


title: How to add pytorch to a docker file containing a project written in C++?

body: I am working on a project which has a docker image (I am very new to docker). The project is written in C++ but has some python features. I can easily install python when I start the image by doing apk add python3, for example. However, this does not work for pytorch. How do I add pytorch to the working environment encapsulated in the docker image? Do I necessarily need to modify the docker image or I can install pytorch after the image has started (just like I do with Python)?


title: How to implent tf.nn.in_top_k in pytorch

body:
I want to implent tf.nn.in_top_k in pytorch. Here is the link of tf.nn.in_top_k,
tf.math.in_top_k(
    targets, predictions, k, name=None
)

It computed precision at k as a bool Tensor and will return a Tensor of type bool.
tf.nn.in_top_k
I wonder whether there are similar api in pytorch?


title: Wrapping All Parameters of a Pytorch Model with A Function

body:
I want to be able to wrap all convolutional parameters (weight and bias) of my PyTorch model with a function e.g. softmax(weight), sigmoid(weight), or alpha*weight, etc. Is there any way of making this function part of the graph using hooks etc.?
For instance: Let say I have a model called net which has trainable parameters from those I am interested in say only in convolutional layers:
for module in net.modules():
   if is_conv_layer(module):
      module.weight # This is the weight I am interested in.

So, replacing the value of the weights are easy:
for module in net.modules():
   if is_conv_layer(module):
      module.weight.data = sigmoid(module.weight.data)

But, I want to also add sigmoid to the graph so that when backprop is computed, sigmoid is also taken into account.


title: neural networks in pytorch

body:
I am using pytorch and need to implement this as a part of a neural network. Is there a particular way to code the layers shown in purple (s
def forward(self, images: torch.Tensor) -> torch.Tensor:
x= self.fc1(x)
return x


title: Evaluating Parzen window log-likelihood for GAN

body:
I am trying to reimplement the original GAN paper by Ian Goodfellow et al. And I need to show that my implementation achieves same or similar results as the authors achieved. But I am not sure how to evaluate this metric. I took a look at their implementation but I got some funny results. In the paper they report 225 +- 2 on the MNIST for this metric, while the results I get are bellow -400000000. I thought that maybe the model is bad, but it generates really good images of MNIST digits.
Can someone tell me what am I doing wrong?
Bellow is the code which I used. I copied the part of the code from the official implementation.
Note: valid variable are images taken from the MNIST dataset.
 def get_nll(x, parzen, batch_size=10):
  """
  Credit: Yann N. Dauphin
  """

  inds = range(x.shape[0])
  n_batches = int(numpy.ceil(float(len(inds)) / batch_size))
  print("N batches:", n_batches)

  times = []
  nlls = []
  for i in range(n_batches):
      begin = time.time()
      nll = parzen(x[inds[i::n_batches]])
      end = time.time()
      times.append(end-begin)
      nlls.extend(nll)

      if i % 10 == 0:
          print(i, numpy.mean(times), numpy.mean(nlls))

  return numpy.array(nlls)

 def log_mean_exp(a):
     """
     Credit: Yann N. Dauphin
     """

     max_ = a.max(1)

     return max_ + T.log(T.exp(a - max_.dimshuffle(0, 'x')).mean(1))

  def cross_validate_sigma(samples, data, sigmas, batch_size):
      lls = []
      for sigma in sigmas:
          print("Sigma:", sigma)
          parzen = theano_parzen(samples, sigma)
          tmp = get_nll(data, parzen, batch_size = batch_size)
          lls.append(numpy.asarray(tmp).mean())
          del parzen
          gc.collect()

      ind = numpy.argmax(lls)
      print(max(lls))
      return sigmas[ind]

 noise = torch.randn((10000, 100), device=device)
 gen_model.eval()
 gan_out = gen_model(noise)

 sigma_range = numpy.logspace(-1., 0., num=10)
 sigma = cross_validate_sigma(gan_out.reshape(10000,-1), valid[0:10000], sigma_range, 100)


title: How to interpret Tensor sizes?

body:
I'm having difficulty understanding the difference between the following:
x1 = torch.tensor([1, 2, 3])  # single brackets
x2 = torch.tensor([[1, 2, 3]])  # double brackets
When checking their sizes:
x1.size() and x2.size() we get the following:
torch.Size([3])
torch.Size([1, 3])

Which I interpret as x1 being a (3x1) column vector, while x2 is a (1x3) row vector.
However, when attempting to transpose both vectors:
print(x1.T)
print(x2.T), we get:
tensor([1, 2, 3])
tensor([[1],
        [2],
        [3]])

x1 seems to be unaffected by transposition?
Further when attempting to force x1 to be a (1x3) row vector using ".view()":
print(x1.view(1, -1)) we get:
tensor([[1, 2, 3]])  # double brackets

So how come ".T" didn't do the trick, but ".view(1, -1)" was able to transform x1 into a (1x3) row vector?
What really is x1 when we first assigned it?


title: Binary entropy loss (BCEWithLogitsLoss) do not converge in Pytorch, but does in Tensorflow 1.14

body:
I reimplement my code https://github.com/zhihou7/VCL in Pytorch. However, I find the new code https://github.com/zhihou7/VCL_Pytorch does not converge. I have checked the two projects many times. However, I do not find any problems.
This project is for HOI detection which is a 600 multi-label classification task. The losses of two versions of code will quickly degrade to 0.01. But, I find the version of PyTorch in fact do not converge normally. The program of Pytorch just predicts all 600 logits to a very small number(far less than 0). Due to the sparsity of positive labels, the loss is still very small. However, with the same code (logically), the version of Tensorflow can predict the positive labels rightly (large logits).
I fell very confused about the phenomenon. I have checked different initial parameters and loss function (BCELosswithlogits). I use the same BN strategy (fix all Batchnorm) and the same weight decay. However, the result is the same. The Pytorch code does not converge.
Does anyone ever suffer from the same problem? or Anyone knows the reason.
Thanks in Advance.


title: dim error with pytorch Dataloader, bertModel

body:
I'm using distilBert in an nn.Module class and pytorch Dataset from P3 for dataloaders.
The input into Bert model: outputs = self.bert(**input), is a dictionary of input ids etc. and the input_ids has size (batch_size, sequence_length).
In my get_items() in the custom Dataset class, I return one such dictionary at once and the dimension of input_ids is correct.
    return self.X[index], self.start[index], self.end[index]

    >>> self.X[index]
    >>>{'input_ids': tensor([[  101,  2054,  2001,  2171,  1997, 20773,  2015,  2034,  3948,  2201,
               102, 20773, 21025, 19358, 22815, 10010,  3334, 12170, 23432, 29715,
              3501, 29678, 12325, 29685, 10506, 14001, 24322,  2141,  2244,  1018,
              3261,  2003,  2137,  3220,  6009,  2501,  3135,  1998,  3883,  2141,
              1998,  2992,  1999,  5395, .......

Then in training I did (model is Bert that i defined with pretrained BertModel):
   for (input, expected_start, expected_end) in train_loader:

    optimizer.zero_grad()

    print(input)

    start, end = model(input) // ERROR: too many values to unpack (expected 3)

class Bert(nn.Module):
  def __init__(self, h, output_dim):
    super(Bert, self).__init__()
    self.bert = DistilBertModel.from_pretrained('distilbert-base-uncased')
    self.start_linear = nn.Linear(h, output_dim)
    self.end_linear = nn.Linear(h, output_dim)
    self.softmax = nn.LogSoftmax(dim=1)
    self.loss = nn.NLLLoss()

  def forward(self, input):
    # print(input.size())
    outputs = self.bert(**input) # input = bert tokenizer encoding
    lhs = outputs.last_hidden_state # (batch_size, sequence_length, hidden_size)
    out = outputs[:, -1, :] # (batch_size, hidden_dim)
    st = self.start_linear(lhs)
    end = self.end_linear(lhs) # or (st)?

    predict_start = self.softmax(st)
    predict_end = self.softmax(end)

    return predict_start, predict_end
......

def get_data_loaders(train, val, batch_size = 32):
  dataset = QAData(train + val) # preprocessed

  train_idx = [i for i in range(len(train))]
  val_idx = [i for i in range(len(train), len(train)+len(val))]

  train_sampler = SubsetRandomSampler(train_idx)
  train_loader = DataLoader(dataset, batch_size=batch_size, sampler=train_sampler)

  val_sampler = SubsetRandomSampler(val_idx)
  val_loader = DataLoader(dataset, batch_size=batch_size, sampler=val_sampler)

  return train_loader, val_loader

But the input_ids in "input" in start, end = model(input) here becomes 3D:
    {'input_ids': tensor([[[  101,  2054,  2001,  2171,  1997, 20773,  2015,  2034,  3948,  2201,
               102, 20773, 21025, 19358, 22815, 10010,  3334, 12170, 23432, 29715,
              3501, 29678, 12325, 29685, 10506, 14001, 24322,  2141,  2244,  1018,
              3261,  2003,  2137,  3220,  6009,  2501,  3135,  1998,  3883,  2141,
              1998,  2992,  1999,  5395, .....

I'm wondering if there's any way to solve this? Or am I doing the batching wrong? Is it the right way to feed batches into my model? Thank you!!!
EDIT: error trace
in train_and_eval(number_of_epochs, model, train_loader, val_loader, save_file_name)
      1 def train_and_eval(number_of_epochs, model, train_loader, val_loader, save_file_name = None):
      2   optimizer = AdamW(model.parameters(), lr=1e-5)
----> 3   train_epoch(model, train_loader, optimizer)
      4   return

<ipython-input-31-97b02296ac3b> in train_epoch(model, train_loader, optimizer)
     44     optimizer.zero_grad()
     45     print("****",input)
---> 46     start, end = model(input)
     47
     48     # total += start.size()[0]

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-31-97b02296ac3b> in forward(self, input)
     15   def forward(self, input):
     16     # print(input.size())
---> 17     outputs = self.bert(**input) # input = bert tokenizer encoding
     18     lhs = outputs.last_hidden_state # (batch_size, sequence_length, hidden_size)
     19     out = outputs[:, -1, :] # (batch_size, hidden_dim)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/transformers/models/distilbert/modeling_distilbert.py in forward(self, input_ids, attention_mask, head_mask, inputs_embeds, output_attentions, output_hidden_states, return_dict)
    485             output_attentions=output_attentions,
    486             output_hidden_states=output_hidden_states,
--> 487             return_dict=return_dict,
    488         )
    489

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/transformers/models/distilbert/modeling_distilbert.py in forward(self, x, attn_mask, head_mask, output_attentions, output_hidden_states, return_dict)
    307
    308             layer_outputs = layer_module(
--> 309                 x=hidden_state, attn_mask=attn_mask, head_mask=head_mask[i], output_attentions=output_attentions
    310             )
    311             hidden_state = layer_outputs[-1]

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/transformers/models/distilbert/modeling_distilbert.py in forward(self, x, attn_mask, head_mask, output_attentions)
    254             mask=attn_mask,
    255             head_mask=head_mask,
--> 256             output_attentions=output_attentions,
    257         )
    258         if output_attentions:

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/transformers/models/distilbert/modeling_distilbert.py in forward(self, query, key, value, mask, head_mask, output_attentions)
    158             seq_length, dim) Contextualized layer. Optional: only if `output_attentions=True`
    159         """
--> 160         bs, q_length, dim = query.size()
    161         k_length = key.size(1)
    162         # assert dim == self.dim, 'Dimensions do not match: %s input vs %s configured' % (dim, self.dim)

ValueError: too many values to unpack (expected 3)


title: is there any way to use RAM and VRAM at the same time on Pytorch?

body:
When I use Keras , It loads the input dataset on RAM a
and loads networks on VRAM, so that I could train vary large network.
However I cannot find the way to do so with pytorch
and this makes me using so small memory I guess.
Is there any way to use both RAM on pytorch ?


title: Torchvision transforms nomalize value selection

body:
This is the value for normalizing the kinetics dataset for video classification in the official script.
https://github.com/pytorch/vision/blob/master/references/video_classification/train.py#L115
i.e.,
normalize = T.Normalize(mean=[0.43216, 0.394666, 0.37645],
std=[0.22803, 0.22145, 0.216989]).

And in https://pytorch.org/docs/stable/torchvision/models.html
it says that “All pre-trained models expect input images normalized in the same way”
with this norm transform:
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225]).
My case is to use pre-trained resnet in torchvision to extract features on kinetics video frames.
So which norm should I choose?
And how these values come from?
Thanks.


title: DQN doesn't learn

body:
I'm trying to implement a DQN in CarPole environment using Pytorch. I don't know why, but no matter how long I've tried to train the agent, even though the scores generally increase, they just fluctuate without maintaining high scores. The code was from a DQN tutorial written for tensorflow, which run normally, but when I try to convert to Pytorch, it doesn't learn. Here's the model:
class Net(nn.Module):
def __init__(self, state_size, action_size):
    super(Net, self).__init__()
    self.fc1 = nn.Linear(state_size, 24)
    self.fc2 = nn.Linear(24, 24)
    self.fc3 = nn.Linear(24, action_size)

def forward(self, inputs):
    x = torch.from_numpy(inputs)
    x = F.relu(self.fc1(x.float()))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

class DQNAgent(nn.Module):
def __init__(self, state_size, action_size):
    super(DQNAgent, self).__init__()
    self.state_size = state_size
    self.action_size = action_size
    self.memory = deque(maxlen=2000)
    self.model, self.criterion, self.optimizer = self.build_model()

    self.epsilon = 1.0
    self.epsilon_decay = 0.995
    self.epsilon_min = 0.01
    self.gamma = 0.95

def build_model(self):
    model = Net(state_size, action_size)
    model = model.float()

    criterion = nn.MSELoss()
    optimizer = optim.Adam(model.parameters(), lr=0.001) # might need to return criterion and optimizer

    return model, criterion, optimizer

def remember(self, state, action, reward, next_state, done):
    self.memory.append((state, action, reward, next_state, done))

def act(self, state):
    if np.random.rand() <= self.epsilon:
        return random.randrange(self.action_size)
    act_values = self.model(state)
    return np.argmax(act_values.detach().numpy())

def replay(self, batch_size):
    minibatch = random.sample(self.memory, batch_size)
    for state, action, reward, next_state, done in minibatch:
        if done:
            target = reward
        elif not done:
            target = reward + self.gamma*torch.max(self.model(next_state)) # --> a tensor

        target_f = self.model(state)
        target_f[0][action] = target

        # self.model.fit(state, target_f, epochs=1, verbose=0)
        loss = self.criterion(self.model(state), target_f)
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

    if self.epsilon > self.epsilon_min:
        self.epsilon = self.epsilon*self.epsilon_decay

def load(self, name):
    pass

def save(self, name):
    pass

... and train:
for e in range(n_episodes):
state = env.reset()
state = np.reshape(state, [1, state_size])

for time in range(5000):
    # env.render()
    action = agent.act(state)

    next_state, reward, done, _ = env.step(action)
    reward = reward if not done else -10

    next_state = np.reshape(next_state, [1, state_size])

    agent.remember(state, action, reward, next_state, done)

    state = next_state

    if done:
        print("Episode {}/{}, score: {}, e: {:.2}".format(e, n_episodes, time, agent.epsilon))
        break

if len(agent.memory) > batch_size:
    agent.replay(batch_size)
    memory = agent.memory

If anyone could give me any suggestions/advices, it would be super appreciated!!! Very confused now. Thank you!


title: PyTorch - Shapes Don't Match

body:
Unfortunately, I get the following RuntimeError:

This error hapens at epoch 1 during the last batch (so all other batches run through),
and I don't know what causes the error in my code. Here is a code snippet of my function
def gradient_penalty(critic, real, fake, device):

    BATCH_SIZE, C, H, W = real.shape
    epsilon = torch.rand(size = (BATCH_SIZE, 1, 1, 1)).repeat(1, C, H, W).to(device)

    # generate tensor filles only with ones
    x = torch.ones(size = (BATCH_SIZE, C, H, W), dtype = int)

    # interpolate images
    interpolated_images = real * epsilon + fake * (x - epsilon)

The variable real stands for images and has the shape (128, 3, 64, 64).
I need to admit that I don't find the error message concretely, i. e. where don't the shapes of the tensors coincide?
Any help would be appreciated!


title: Is there a Numpy or pyTorch function for this code?

body:
Basically is there a Numpy or PyTorch function that does this:
vp_sa_s=mdp_data['sa_s'].detach().clone()
dims = vp_sa_s.size()
for i in range(dims[0]):
    for j in range(dims[1]):
        for k in range(dims[2]):
            # to mimic matlab functionality: vp(mdp_data.sa_s)
            try:
                vp_sa_s[i,j,k] = vp[mdp_data['sa_s'][i,j,k]]
            except:
                pass

Given that vp_sa_s is size (10,5,5) and each value is a valid index vp i.e in range 0-9. vp is size (10,1) with a bunch of random values.
Matlab do it elegantly and quickly with vp(mdp_data.sa_s) which will form a new (10,5,5) matrix. If all values in mdp_data.sa_s are 1, the result would be a (10,5,5) tensor with each value being the 1st value in vp.
Does a function or method that exists that can achieve this in less than O(N^3) time as the above code is terribly inefficient.
Thanks!


title: Getting fcntl error when importing PyKeops in Python

body:
I've installed PyKeops package using pip on Windows. The packages seems to be installed properly. Even my pytorch is working well. But when I want to import LazyTensors I get the following error:
from pykeops.torch import LazyTensor
# =====================================
# ModuleNotFoundError: No module named 'fcntl'


title: using ModuleList, still getting ValueError: optimizer got an empty parameter list

body:
With Pytorch I am attempting to use ModuleList to ensure model parameters are detected, and can be optimized. When calling the SGD optimizer I get the following error:

ValueError: optimizer got an empty parameter list

Can you please review the code below and advise?
class LR(nn.Module):
    def ___init___(self):
        super(LR, self).___init___()
        self.linear = nn.ModuleList()
        self.linear.append(nn.Linear(in_features=28*28, out_features=128, bias=True))

    def forward(self, x):
        y_p = torch.sigmoid(self.linear(x))
        return y_p

LR_model = LR()
optimizer = torch.optim.SGD(params = LR_model.parameters(), lr=learn_rate)


title: Pytorch model runtime error when testing U-Net

body:
I've defined a U-Net model using Pytorch but it won't accept my input. I've checked the model layers and they seem to be applying the operations as I would expect them to but I still get an error.
I've just switched to Pytorch after mostly using Keras so I'm not really sure how to debug this issue, the error I get is:

RuntimeError: Given groups=1, weight of size [32, 64, 3, 3], expected
input[1, 128, 65, 65] to have 64 channels, but got 128 channels
instead

Here's the code I'm using:
class UNET(nn.Module):
    def __init__(self, in_channels=2, out_channels=2):
        super().__init__()

        self.conv1 = self.contract_block(in_channels, 32, 3, 1)
        self.conv2 = self.contract_block(32, 64, 3, 1)
        self.conv3 = self.contract_block(64, 128, 3, 1)

        self.upconv3 = self.expand_block(128, 64, 3, 1)
        self.upconv2 = self.expand_block(64, 32, 3, 1)
        self.upconv1 = self.expand_block(32, out_channels, 3, 1)

    def __call__(self, x):

        # downsampling part
        conv1 = self.conv1(x)
        conv2 = self.conv2(conv1)
        conv3 = self.conv3(conv2)

        upconv3 = self.upconv3(conv3)

        upconv2 = self.upconv2(torch.cat([upconv3, conv2], 1))
        upconv1 = self.upconv1(torch.cat([upconv2, conv1], 1))

        return upconv1

    def contract_block(self, in_channels, out_channels, kernel_size, padding):

        contract = nn.Sequential(
            torch.nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, stride=1, padding=padding),
            torch.nn.BatchNorm2d(out_channels),
            torch.nn.Tanh(),
            torch.nn.Conv2d(out_channels, out_channels, kernel_size=kernel_size, stride=1, padding=padding),
            torch.nn.BatchNorm2d(out_channels),
            torch.nn.Tanh(),
            torch.nn.MaxPool2d(kernel_size=2, stride=2, padding=1)
            )

        return contract

    def expand_block(self, in_channels, out_channels, kernel_size, padding):

        expand = nn.Sequential(
            torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=padding),
            torch.nn.BatchNorm2d(out_channels),
            torch.nn.Tanh(),
            torch.nn.Conv2d(out_channels, out_channels, kernel_size, stride=1, padding=padding),
            torch.nn.BatchNorm2d(out_channels),
            torch.nn.Tanh(),
            torch.nn.ConvTranspose2d(out_channels, out_channels, kernel_size=2, stride=2, padding=1, output_padding=1)
            )
        return expand

model = UNET()
x = torch.randn(1, 2, 256, 256)
print(model(x))


title: Simplest of way of reading the content of an .mp3 file as a vector in C++?

body:
What is the simplest of way of reading the content of an .mp3 file into a C++ vector?
Context: I produced a voice recognition model in PyTorch. The model in PyTorch takes tensors as inputs. Those input are read directly from .mp3 files using the torchaudio module.
Goal :I now need to use a TorchScript version of the model in a C++ file. If I understand correctly, the inputs should now be a C++ vector of the type torch::jit::IValue . What is the simplest / most standard / most proper way of getting the input from an .mp3 file? (if any)
In Python, I know there are easy to use modules such as Librosa or Pydub, but I'm really a beginner in C++. I did find this answer but I was wondering if there wasn't a simpler way. Also, what I need is the number of channels,amplitudes and the sampling rate (same as the ouput of the Python function), not the TAG.


title: GAN: Why does adding more layers to my generator cause mode collapse? How can I fix this?

body:
I have a GAN which I am using to generate plasma image data. My mentor has asked me to be able to change the number of layers in my GAN via cmd line argument. With that being said,  anytime I use more than 4 or 5 blocks, my generated images are all almost identical, some batches just having more noise than others. And with larger amounts of layers my images are more likely to just come out wrong(sometimes completely inverted colors). When this happens the model isn’t changing at all even when there is a loss. How can I prevent this mode collapse and why is it happening? My best guess is that all the data has an area with similar values and the GAN is just guessing that image over and over
Helpful details: Using 16000 images for training, have tried adding noise to input data, been trying to train for 100 or 200 epochs, also tried with a wasserstein loss function, seemed to have the same issue but its possible I implemented it wrong
class Block(nn.Module):
   def __init__(self, in_feat, out_feat, normalize=True):
       super(Block, self).__init__()
       self.layers = nn.ModuleList()
       self.layers.append(nn.Linear(in_feat, out_feat))
       if normalize:
           self.layers.append(nn.BatchNorm1d(out_feat, 0.8))
       self.layers.append(nn.LeakyReLU(0.2, inplace=True))

   def forward(self, z):
       img = z
       for layer in self.layers:
           img = layer(img)
       return img

class Generator(nn.Module):
   def __init__(self):
       super(Generator, self).__init__()

       spacing = (1024 - 128)/opt.layers
       self.block_modules = nn.ModuleList()
       self.block_modules.append(Block(opt.latent_dim, 128, normalize=False))

       current_size = 128
       prev_size = current_size
       for i in range(1, opt.layers):
           current_size = 128 + (i * spacing)
           self.block_modules.append(Block(int(prev_size),int(current_size)))
           prev_size = current_size

       self.block_modules.append(nn.Linear(int(current_size), int(np.prod(img_shape))))
       self.block_modules.append(nn.Tanh())

   def forward(self, z):
       img = z
       for layer in self.block_modules:
           img = layer(img)
       img = img.view(img.size(0), *img_shape)
       return img

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()

        self.model = nn.Sequential(
            nn.Linear(int(np.prod(img_shape)), 512),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Linear(512, 256),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Linear(256, 128),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Linear(128, 64),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Linear(64, 1),
            nn.Sigmoid(),
        )

    def forward(self, img):
        img_flat = img.view(img.size(0), -1)
        validity = self.model(img_flat)

        return validity

adversarial_loss = torch.nn.BCELoss()

generator = Generator()
discriminator = Discriminator()

optimizer_G = torch.optim.Adam(generator.parameters(), lr=opt.lr, betas=(opt.b1, opt.b2))
optimizer_D = torch.optim.Adam(discriminator.parameters(), lr=opt.lr, betas=(opt.b1, opt.b2))

Tensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor

for epoch in range(opt.n_epochs):

    for i, (imgs, _) in enumerate(dataloader):

        if opt.noise is not None:
            ns = torch.normal(mean=0.0, std=imgs.detach()*opt.noise)
            imgs = imgs + ns

        valid = Variable(Tensor(imgs.size(0), 1).fill_(1.0), requires_grad=False)
        fake = Variable(Tensor(imgs.size(0), 1).fill_(0.0), requires_grad=False)

        # Configure input
        real_imgs = Variable(imgs.type(Tensor))

        optimizer_G.zero_grad()

        # Sample noise as generator input
        z = Variable(Tensor(np.random.normal(0, 1, (imgs.shape[0], opt.latent_dim))))

        # Generate a batch of images
        gen_imgs = generator(z)

        g_loss = adversarial_loss(discriminator(gen_imgs), valid)

        g_loss.backward()
        optimizer_G.step()

        #  Train Discriminator
        optimizer_D.zero_grad()

        # Measure discriminator's ability to classify real from generated samples
        real_loss = adversarial_loss(discriminator(real_imgs), valid)
        fake_loss = adversarial_loss(discriminator(gen_imgs.detach()), fake)
        d_loss = (real_loss + fake_loss) / 2

        d_loss.backward()
        optimizer_D.step()


title: Convert numpy ndarray to PIL and and convert it to tensor

body:
def camera(transform):
    capture = cv2.VideoCapture(0)
    while True:
        ret, frame = capture.read()
        cv2.imshow('video', frame)
        # esc
        if cv2.waitKey(1) == 27:
            photo = frame
            break
    capture.release()
    cv2.destroyAllWindows()
    img = Image.fromarray(cv2.cvtColor(photo, cv2.COLOR_BGR2RGB))
    img = img.resize([224, 224], Image.LANCZOS)
    if transform is not None:
        img = transform(img).unsqueeze(0)
    return img

This is my code to get image from the camera,
image_tensor = img.to(device)

And I have an error at the line above...
Traceback (most recent call last):
  File "E:/PycharmProjects/ArsElectronica/image_captioning/sample.py", line 126, in <module>
    main(args)
  File "E:/PycharmProjects/ArsElectronica/image_captioning/sample.py", line 110, in main
    caption = Image_Captioning(args, img)
  File "E:/PycharmProjects/ArsElectronica/image_captioning/sample.py", line 88, in Image_Captioning
    image_tensor = img.to(device)
AttributeError: 'Image' object has no attribute 'to'

The error is like this.
If I have the image as png file and reload it with PIL, it works.
But the one I want is to use the image without saving.
Pls... Someone save me...


title: GPU in docker container for deep learning task

body:
Recently, I have started working on using docker images. I want to deploy PyTorch based text classification model which requires GPU to run on. When the docker image is called upon, then it's not able to detect GPU in the VM. Hence, my code is failing by throwing no Cuda device found error.
This is my base image FROM gcr.io/deeplearning-platform-release/pytorch-gpu.1-0. I don't know what all steps to follow to install Nvidia drivers in the docker. Please help me out.


title: How to join two tensors

body:
I have two tensors of dimension [3,1]. I need to join them as a [3,2] tensor.
t = torch.tensor([[3.],[1],[2]], requires_grad=True)
x = torch.tensor([[1.],[4],[5]], requires_grad=True)

I tried torch.cat and torch.stack but neither work for me.


title: How to quickly inverse a permutation by using PyTorch?

body:
I am confused on how to quickly restore an array shuffled by a permutation.
Example #1:

[x, y, z] shuffled by P: [2, 0, 1], we will obtain [z, x, y]
the corresponding inverse should be P^-1: [1, 2, 0]

Example #2:

[a, b, c, d, e, f] shuffled by P: [5, 2, 0, 1, 4, 3], then we will get [f, c, a, b, e, d]
the corresponding inverse should be P^-1: [2, 3, 1, 5, 4, 0]

I wrote the following codes based on matrix multiplication (the transpose of permutation matrix is its inverse), but this approach is too slow when I utilize it on my model training. Does there exisits a faster implementation?
import torch

n = 10
x = torch.Tensor(list(range(n)))
print('Original array', x)

random_perm_indices = torch.randperm(n).long()
perm_matrix = torch.eye(n)[random_perm_indices].t()
x = x[random_perm_indices]
print('Shuffled', x)

restore_indices = torch.Tensor(list(range(n))).view(n, 1)
restore_indices = perm_matrix.mm(restore_indices).view(n).long()
x = x[restore_indices]
print('Restored', x)


title: Can you build a ResNet using the Pytorch Basic block alone?

body:
I have been reading this article: https://dhruvs.space/posts/understanding-resnets/ to understand how ResNet50 is built.
I have seen people implementing ResNet50 using the Basic block alone. Like here: https://analyticsindiamag.com/hands-on-guide-to-implement-resnet50-in-pytorch-with-tpu/
However, pytorch implements the ResNet50 using the Bottleneck block (different than the second link but similar to the first link above) https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py is different than what is cited in the article.
Can you build a ResNet50 using pytorch Basic block alone?


title: Shuffle patches in image batch

body:
I am trying to create a transform that shuffles the patches of each image in a batch.
I aim to use it in the same manner as the rest of the transformations in torchvision:
trans = transforms.Compose([
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
            ShufflePatches(patch_size=(16,16)) # our new transform
        ])

More specifically, the input is a BxCxHxW tensor. I want to split each image in the batch into non-overlapping patches of size patch_size, shuffle them, and regroup into a single image.
Given the image (of size 224x224):

Using ShufflePatches(patch_size=(112,112)) I would like to produce the output image:

I think the solution has to do with torch.unfold and torch.fold, but didn't manage to get any further.
Any help would be appreciated!


title: CNN in pytorch "Expected 4-dimensional input for 4-dimensional weight [32, 1, 5, 5], but got 3-dimensional input of size [16, 64, 64] instead"

body:
I am new to pytorch. I am trying to use chinese mnist dataset to train the neural network that shows in below code. Is that a problem of the neural network input or something else goes wrong in my code. I have tried many ways to fix it but instead it shows me other errors
train_df = chin_mnist_df.groupby('value').apply(lambda x: x.sample(700, random_state=SEED)).reset_index(drop=True)
x_train, y_train  = train_df.iloc[:, :-2], train_df.iloc[:, -2]

valid_df = chin_mnist_df.groupby('value').apply(lambda x: x.sample(200, random_state=SEED)).reset_index(drop=True)
x_valid, y_valid  = valid_df.iloc[:, :-2], valid_df.iloc[:, -2]

test_df  = chin_mnist_df.groupby('value').apply(lambda x: x.sample(100, random_state=SEED)).reset_index(drop=True)
x_test, y_test    = test_df.iloc[:, :-2], test_df.iloc[:, -2]

train_ds = Dataset(x_train, y_train)
train_dataloader = torch.utils.data.DataLoader(train_ds, batch_size=16, shuffle=True)

valid_ds = Dataset(x_valid, y_valid)
valid_dataloader = torch.utils.data.DataLoader(valid_ds, batch_size=16, shuffle=True)

test_ds = Dataset(x_test, y_test)
test_dataloader = torch.utils.data.DataLoader(test_ds, batch_size=16, shuffle=True)

# Convolutional neural network (two convolutional layers)
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(32, 64, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(7 * 7 * 64, 1000)
        self.fc2 = nn.Linear(1000, 15)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.drop_out(out)
        out = self.fc1(out)
        out = self.fc2(out)
        return out

model = ConvNet()
klisi=[]
apoklisi=[]
apoklisi2=[]
klisi2=[]
olatalr=[]
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
total_step = len(train_dataloader)
loss_list = []
acc_list = []
for epoch in range(num_epochs):
    for i,data in enumerate(train_dataloader):#(images, labels)
        batch_inputs, batch_labels = data[0][:].to(device).type(torch.float), data[1][:].to(device)
        # Run the forward pass
        outputs = model(batch_inputs)
        loss = criterion(outputs, batch_labels)


title: Dropout Layer with zero dropping rate

body:
I'm having trouble understanding a certain aspect of dropout layers in PyTorch.
As stated in the Pytorch Documentation the method's signature is torch.nn.Dropout(p=0.5, inplace=False) where p is the dropout rate.
What does this layer do when choosing p=0? Does it change its input in any way?


title: Creating a tensor where each entry is a function of its index

body:
I want to create a matrix D that is defined by D[i,j]=d(i-j) where d is some arbitrary function that I can choose.
It can easily be done with loops, but it is very slow. Is there an efficiant way of creating this matrix with torch or numpy?


title: Fast.ai:ModuleAttributeError: 'Sequential' object has no attribute 'fine_tune'

body:
When I use the fast.ai, T encounter this problem,the follow is my code:
from fastai.vision.all import *
from fastai.text.all import *
from fastai.collab import *
from fastai.tabular.all import *

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

path = untar_data(URLs.PETS)/'images'

def is_cat(x): return x[0].isupper()

dls = ImageDataLoaders.from_name_func(
    path, get_image_files(path), valid_pct=0.2, seed=42,
    label_func=is_cat, item_tfms=Resize(224))

learn = cnn_learner(dls, resnet34, metrics=error_rate).to(device)

learn.fine_tune(1)

It shows :"ModuleAttributeError: 'Sequential' object has no attribute 'fine_tune'"


title: how to store model.state_dict() in a temp var for later use?

body:
I tried to store the state dict of my model in a variable temporarily and wanted to restore it to my model later, but the content of this variable changed automatically as the model updated.
There is a minimal example:
import torch as t
import torch.nn as nn
from torch.optim import Adam

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc = nn.Linear(3, 2)

    def forward(self, x):
        return self.fc(x)

net = Net()
loss_fc = nn.MSELoss()
optimizer = Adam(net.parameters())

weights = net.state_dict()
print(weights)

x = t.rand((5, 3))
y = t.rand((5, 2))
loss = loss_fc(net(x), y)

optimizer.zero_grad()
loss.backward()
optimizer.step()

print(weights)

I thought the two outputs would be the same, but I got (outputs may change due to random initialization)
OrderedDict([('fc.weight', tensor([[-0.5557,  0.0544, -0.2277],
        [-0.0793,  0.4334, -0.1548]])), ('fc.bias', tensor([-0.2204,  0.2846]))])
OrderedDict([('fc.weight', tensor([[-0.5547,  0.0554, -0.2267],
        [-0.0783,  0.4344, -0.1538]])), ('fc.bias', tensor([-0.2194,  0.2856]))])

The content of weights changed, which is so weird.
I also tried .copy() and t.no_grad() as following, but they did not help.
with t.no_grad():
    weights = net.state_dict().copy()

Yes, I know that I can save state dict using t.save(), but I just want to figure out what happened in the previous example.
I'm using Python 3.8.5 and Pytorch 1.8.1
Thanks for any help.


title: How to set GPU count to 0 using os.environ['CUDA_VISIBLE_DEVICES'] =""?

body:
So I have the following GPU configured in my system:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 461.33       Driver Version: 461.33       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100S-PCI...  TCC  | 00000000:3B:00.0 Off |                    0 |
| N/A   30C    P0    25W / 250W |      1MiB / 32642MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100S-PCI...  TCC  | 00000000:D8:00.0 Off |                    0 |
| N/A   31C    P0    25W / 250W |      1MiB / 32642MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Now, via python, I have to set the environment, such that, GPU count = 0.
I have tried the following, after learning from various sources:
import os
os.environ["CUDA_VISIBLE_DEVICES"]=""
import torch
torch.device_count()

But, it still gives me the output as "2" as in for 2 GPUs in the system.
How to set the environment, such that it outputs "0" ?
Any other way, to set the count to "0" is also appreciated but it should be any ML-Library agnostic. (For example, I can't use device = torch.device("cpu") as this will work only for Pytorch and not for other libraries)


title: PyTorch can't use a float type but only long

body:
I am trying to run this very basic neural network:
import os; os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"
import torch
import torchvision
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import time

#####################################################
#             Create the neural network             #
#####################################################

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1, 10)
        self.fc2 = nn.Linear(10, 10)
        self.fc3 = nn.Linear(10, 1)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

#####################################################
#                   Create the datasets             #
#####################################################

trainset = [torch.tensor([1., 1.**2]), torch.tensor([2., 2.**2]), torch.tensor([3., 3.**2]), torch.tensor([4., 4.**2]), torch.tensor([5., 5.**2]), torch.tensor([6., 6.**2]), torch.tensor([7., 7.**2]), torch.tensor([8., 8.**2])]

testset = [torch.tensor([1.1, 1.1**2]), torch.tensor([2.3, 2.3**2]), torch.tensor([3.1, 3.1**2]), torch.tensor([4.5, 4.5**2]), torch.tensor([5.9, 5.9**2]), torch.tensor([6.1, 6.1**2]), torch.tensor([7.3, 7.3**2]), torch.tensor([8.01, 8.01**2])]

#####################################################
#               Optimize the parameters             #
#####################################################

optimizer = torch.optim.Adam(net.parameters(), lr=0.001)

EPOCHS = 3

for epoch in range(EPOCHS):
    for data in trainset:
        x, y = data
        net.zero_grad()
        output = net(x.view(-1,1))
        loss = F.nll_loss(output, y.view(-1,1)[0])
        loss.backward()
        optimizer.step()
    print(loss)

#####################################################
#           Calculate the accuracy rate             #
#####################################################

correct = 0
total = 0

with torch.no_grad():
    for data in trainset:
        x, y = data
        output = net(x)
        if y - 0.01 < output < y + 0.01:
            correct += 1
        total += 1

print("Accuracy: %.2f" % (correct / total))

but I get the following error:

Traceback (most recent call last):   File
"C:\Users\Andrea\Desktop\pythonProject\main.py", line 52, in
loss = F.nll_loss(output, y.view(-1,1)[0])   File "C:\WinPython\python-3.9.1.amd64\lib\site-packages\torch\nn\functional.py",
line 2235, in nll_loss
ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index) RuntimeError: expected scalar type Long but found Float

Why can't I use a float type?


title: Convert keras model architecture to Pytorch

body:
I am trying to convert a keras model to pytorch for human activity recognition. The keras model could achieve up to 98% accuracy, while Pytorch model could achieve only ~60% accuracy. I couldn’t figure out the problem, first I was thinking of the padding=‘same’ of keras, but I have already adjusted the padding of pytorch already. Can you check what is wrong?
The keras code is as above
model = keras.Sequential()
model.add(layers.Input(shape=[100,12]))
model.add(layers.Conv1D(filters=32, kernel_size=3, padding="same"))
model.add(layers.BatchNormalization())
model.add(layers.ReLU())
model.add(layers.Conv1D(filters=64, kernel_size=3, padding="same"))
model.add(layers.BatchNormalization())
model.add(layers.ReLU())
model.add(layers.MaxPool1D(2))
model.add(layers.LSTM(64))
model.add(layers.Dense(units=128, activation='relu'))
model.add(layers.Dense(13, activation='softmax'))
model.summary()

and my pytorch model code is as below
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        seq_len = 100
        # output: [, 32, 100]
        self.conv1 = nn.Conv1d(seq_len, 32, kernel_size=3, stride=1, padding=1)
        self.bn1 = nn.BatchNorm1d(32)
        # output: [, 64, 100]
        self.conv2 = nn.Conv1d(32, 64, kernel_size=3, padding=1)
        self.bn2 = nn.BatchNorm1d(64)
        # output: [, 64, 50]
        self.mp = nn.MaxPool1d(kernel_size=2, stride=2)
        # output: [, 64]
        self.lstm = nn.LSTM(6, 64, 1)
        # output: [, 128]
        self.fc1 = nn.Linear(64, 128)
        # output: [, 13]
        self.fc2 = nn.Linear(128, 13)

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = F.relu(x)

        x = self.conv2(x)
        x = self.bn2(x)
        x = F.relu(x)
        x = self.mp(x)

        out, _ = self.lstm(x)
        x = out[:, -1, :]

        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)

        return x


title: How to implement this non-fully connected structure in Keras?

body:
I am trying to implement a 3-layer non-fully connected network in Keras for classification of output weights from 2 submodels with the same output dimension. This should be accomplished using a binary mask matrix composed of 0s with 1s for each neuron that should be connected to the neuron in the hidden layer (so two 1s for each neuron). I have attached a diagram as well as the PyTorch code I am working from.

class Weight_classifier(nn.Module):
def __init__(self, func):
    super(Weight_classifier, self).__init__()
    # self.weight_layer = nn.Linear(OUT_nodes[func]*3, OUT_nodes[func])
    self.weight_layer = MaskedLinear(OUT_nodes[func]*2, OUT_nodes[func], 'data/{}_maskmatrix.csv'.format(func), func).cuda()
    self.outlayer= nn.Linear(OUT_nodes[func], OUT_nodes[func])

def forward(self, weight_features):
    weight_out = self.weight_layer(weight_features)
    # weight_out = F.sigmoid(weight_out)
    weight_out = F.relu(weight_out)
    weight_out = F.sigmoid(self.outlayer(weight_out))
    return weight_out

class MaskedLinear(nn.Linear):
def __init__(self, in_features, out_features, relation_file, func, bias=True):
    super(MaskedLinear, self).__init__(in_features, out_features, bias)

    mask = self.readRelationFromFile(relation_file, func)
    self.register_buffer('mask', mask)
    self.iter = 0

def forward(self, input):
    masked_weight = self.weight * self.mask
    return F.linear(input, masked_weight, self.bias)

def readRelationFromFile(self, relation_file, func):
    mask = []
    with open(relation_file, 'r') as f:
        for line in f:
            l = [int(x) for x in line.strip().split(',')[OUT_nodes[func]:]]
            assert len(l) == OUT_nodes[func]*2
            for item in l:
                assert item == 1 or item == 0
            mask.append(l)
    return Variable(torch.Tensor(mask))

Thanks


title: Is there a way to load torchvision model by string?

body:
Currently, I load pretrained torchvision model using following code:
import torchvision
torchvision.models.resnet101(pretrained=True)

However, I'd love to have model name as string parameter and then load the pretrained model using that string. A pseudo-code that would do so would be something like:
model_name = 'resnet101'
torchvision.models.get(model_name)(pretrained=True)

Is there a way to accomplish this in a rather simple manner?


title: PyTorch NN : RuntimeError: mat1 dim 1 must match mat2 dim 0

body:
I'm new to the Neural Network domain and I have stuck on a problem.
I'm trying to create a NN with dropout with 0.1 probability for the hidden fully connected layer.
When I code like below:
class ConvNet(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = torch.nn.Sequential(

            #layer1
            torch.nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=1),
            torch.nn.MaxPool2d(kernel_size=2, stride=3),
            torch.nn.BatchNorm2d(num_features=16),
            torch.nn.ReLU(),

            #layer2
            torch.nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, stride=1),
            torch.nn.MaxPool2d(kernel_size=2, stride=3),
            torch.nn.BatchNorm2d(num_features=32),
            torch.nn.ReLU(),

            #layer3
            torch.nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, stride=1),
            torch.nn.MaxPool2d(kernel_size=2, stride=3),
            torch.nn.BatchNorm2d(num_features=32),
            torch.nn.Flatten(),
            torch.nn.Linear(32,16),
            torch.nn.Dropout2d(p=0.1),
            torch.nn.Linear(16, 2)
        )

    def forward(self, x):
        return self.layers(x)

test_convnet = ConvNet().to('cuda')
test_input = torch.randn(16, 3, 100, 100, device='cuda')
test_output = test_convnet(test_input)
print(test_output.shape)

And then I got the error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-27-b5ce1c300266> in <module>()
     48 test_convnet = ConvNet().to('cuda')
     49 test_input = torch.randn(16, 3, 100, 100, device='cuda')
---> 50 test_output = test_convnet(test_input)
     51 print(test_output.shape)

6 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1751     if has_torch_function_variadic(input, weight):
   1752         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1753     return torch._C._nn.linear(input, weight, bias)
   1754
   1755

RuntimeError: mat1 dim 1 must match mat2 dim 0
Thanks in advance for all of your help


title: How to build YOLACT++ using Docker?

body:
I have to build yolact++ in docker enviromment (i'm using sagemaker notebook). Like this
ARG PYTORCH="1.3"
ARG CUDA="10.1"
ARG CUDNN="7"

FROM pytorch/pytorch:${PYTORCH}-cuda${CUDA}-cudnn${CUDNN}-devel

And i want to run this
COPY yolact/external/DCNv2/setup.py /opt/ml/code/external/DCNv2/setup.py
RUN cd /opt/ml/code/external/DCNv2 && \
python setup.py build develop

But i got this error :
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
Traceback (most recent call last):
File "setup.py", line 64, in <module>
ext_modules=get_extensions(),
File "setup.py", line 41, in get_extensions
raise NotImplementedError('Cuda is not available')
NotImplementedError: Cuda is not available

But the enviromment supports CUDA. Anyone have an idea where is the problem ?
Thank you.


title: Access old pytorch release from conda cloud

body:
According to multiple sites, there was a binary release 0.2.1 for pytorch in repo peterjc123 (e.g. https://moodle.di.ens.fr/mod/forum/discuss.php?d=9#p33 ).
I also see a release 0.3.0 when looking at the only snapshot from archive.org.
However, the conda cloud website only shows the latest version (0.3.1; https://anaconda.org/peterjc123/pytorch/files ); the same applies to

conda search pytorch -c peterjc123

The old download links do not work anymore.
How do I access the old version (I need a binary < 0.3 for windows 10; cuda80; py36)?


title: How to Equate dimensions of two tensors of sizes [64, 3, 128, 128] and [64, 64, 127, 127]

body:
I am trying to add two tensors of dimension [64, 64, 127, 127] and [64, 3, 127, 127] in pytorch, but I need to add zeros in the second tensor in order to equate their dimensions to [64, 64, 127, 127].

I have tried adding zeros through slicing but it is not robust


title: Using .hdf5 output from Keras library to use with Pytorch

body:
Fairly new into machine learning - I have a training script that trains a default model with no weights into a classifier using images as an input, and after the training, it gives an output with an .hdf5 file.

I am aware that the .hdf5 file will contain the model's info and the weights (https://www.tinymind.com/learn/terms/hdf5). But Im trying to find a way to use the output .hdf5 file to a certain script that uses pytorch to classify objects.

There has not been much information how to use this output file format as a model in pytorch after searching.

What would I need to do in order to use this output file as a trained model to be used in torch? I would appreciate it if someone points me in the right direction to get started


title: Generalization difference when using `nn.Sequential`?

body:
I have an autoencoder deep network, and I notice that when I use nn.Sequential, the generalization performance is better than when I don't use it (i.e. explicitly pass the inputs through layers). Has anyone else noticed this behavior, or can provide an explanation as to why this is? Does Pytorch handle regularization differently in a sequential block?

Here is a code snippet; I've attached the full code at the end for reference. I am toggling between using nn.Sequential and not using it with the variable use_sequential. I find that when I don't use the sequential module, my test accuracy is always worse than if I do use it.

class Net(nn.Module):
    def __init__(self, hidden_dim, in_dim, use_sequential):
        super(Net, self).__init__()
        self.use_sequential = use_sequential
        self.in_dim = in_dim
        self.hidden_dim = hidden_dim

        self.sig = nn.Sigmoid()
        self.encoder = nn.Sequential(
                nn.Linear(in_dim, in_dim),
                nn.BatchNorm1d(in_dim),
                nn.Linear(in_dim, hidden_dim)
        )
        self.decoder = nn.Sequential(
                nn.Linear(hidden_dim, in_dim),
                nn.BatchNorm1d(in_dim),
                nn.Linear(in_dim, in_dim)
        )

    def encode(self, x):
        if self.use_sequential:
            x = self.encoder(x)
        else:
            x = self.lin1(x)
            x = self.batchnorm(x)
            x = self.lin2(x)
        return x

    def decode(self, x):
        if self.use_sequential:
            x = self.decoder(x)
        else:
            x = self.lin3(x)
            x = self.batchnorm(x)
            x = self.lin4(x)
        return x

    def forward(self, x):
        x = self.encode(x)
        x = self.decode(x)
        x = self.sig(x) # Sigmoid for BCELoss
        return x

Here's the output of the script. As you can see, the test accuracy/loss is worse as the models train, even though their train accuracy/loss remain similar (I initialize all weights myself, see full code at the end):

SEQUENTIAL TRAIN, Epoch  0: loss=0.7185, acc=0.51
NONSEQUENTIAL TRAIN, Epoch  0: loss=0.7185, acc=0.51
---> SEQUENTIAL TEST: Epoch  0: loss=0.7240, acc=0.50080
---> NONSEQUENTIAL TEST: Epoch  0: loss=0.7240, acc=0.50080

SEQUENTIAL TRAIN, Epoch  1: loss=0.7192, acc=0.49
NONSEQUENTIAL TRAIN, Epoch  1: loss=0.7192, acc=0.49
---> SEQUENTIAL TEST: Epoch  1: loss=0.7226, acc=0.49920
---> NONSEQUENTIAL TEST: Epoch  1: loss=0.7221, acc=0.49920

SEQUENTIAL TRAIN, Epoch  2: loss=0.7207, acc=0.50
NONSEQUENTIAL TRAIN, Epoch  2: loss=0.7208, acc=0.50
---> SEQUENTIAL TEST: Epoch  2: loss=0.7183, acc=0.56120
---> NONSEQUENTIAL TEST: Epoch  2: loss=0.7186, acc=0.49920

SEQUENTIAL TRAIN, Epoch  3: loss=0.7032, acc=0.54
NONSEQUENTIAL TRAIN, Epoch  3: loss=0.7033, acc=0.54
---> SEQUENTIAL TEST: Epoch  3: loss=0.7104, acc=0.56120
---> NONSEQUENTIAL TEST: Epoch  3: loss=0.7153, acc=0.49920

SEQUENTIAL TRAIN, Epoch  4: loss=0.7002, acc=0.56
NONSEQUENTIAL TRAIN, Epoch  4: loss=0.7002, acc=0.56
---> SEQUENTIAL TEST: Epoch  4: loss=0.7006, acc=0.56120
---> NONSEQUENTIAL TEST: Epoch  4: loss=0.7119, acc=0.49920

SEQUENTIAL TRAIN, Epoch  5: loss=0.6906, acc=0.55
NONSEQUENTIAL TRAIN, Epoch  5: loss=0.6907, acc=0.55
---> SEQUENTIAL TEST: Epoch  5: loss=0.6903, acc=0.56120
---> NONSEQUENTIAL TEST: Epoch  5: loss=0.7088, acc=0.49920

SEQUENTIAL TRAIN, Epoch  6: loss=0.6807, acc=0.54
NONSEQUENTIAL TRAIN, Epoch  6: loss=0.6811, acc=0.54
---> SEQUENTIAL TEST: Epoch  6: loss=0.6815, acc=0.56120
---> NONSEQUENTIAL TEST: Epoch  6: loss=0.7058, acc=0.49920

SEQUENTIAL TRAIN, Epoch  7: loss=0.6698, acc=0.52
NONSEQUENTIAL TRAIN, Epoch  7: loss=0.6702, acc=0.52
---> SEQUENTIAL TEST: Epoch  7: loss=0.6729, acc=0.56120
---> NONSEQUENTIAL TEST: Epoch  7: loss=0.7033, acc=0.49920

SEQUENTIAL TRAIN, Epoch  8: loss=0.6710, acc=0.67
NONSEQUENTIAL TRAIN, Epoch  8: loss=0.6722, acc=0.60
---> SEQUENTIAL TEST: Epoch  8: loss=0.6643, acc=0.56120
---> NONSEQUENTIAL TEST: Epoch  8: loss=0.7014, acc=0.49920

SEQUENTIAL TRAIN, Epoch  9: loss=0.6642, acc=0.71
NONSEQUENTIAL TRAIN, Epoch  9: loss=0.6659, acc=0.65
---> SEQUENTIAL TEST: Epoch  9: loss=0.6612, acc=0.68860
---> NONSEQUENTIAL TEST: Epoch  9: loss=0.6999, acc=0.49920

SEQUENTIAL TRAIN, Epoch 10: loss=0.6593, acc=0.68
NONSEQUENTIAL TRAIN, Epoch 10: loss=0.6613, acc=0.68
---> SEQUENTIAL TEST: Epoch 10: loss=0.6570, acc=0.68860
---> NONSEQUENTIAL TEST: Epoch 10: loss=0.6988, acc=0.49920

SEQUENTIAL TRAIN, Epoch 11: loss=0.6522, acc=0.68
NONSEQUENTIAL TRAIN, Epoch 11: loss=0.6541, acc=0.68
---> SEQUENTIAL TEST: Epoch 11: loss=0.6540, acc=0.68860
---> NONSEQUENTIAL TEST: Epoch 11: loss=0.6978, acc=0.49920

SEQUENTIAL TRAIN, Epoch 12: loss=0.6651, acc=0.67
NONSEQUENTIAL TRAIN, Epoch 12: loss=0.6679, acc=0.67
---> SEQUENTIAL TEST: Epoch 12: loss=0.6511, acc=0.68860
---> NONSEQUENTIAL TEST: Epoch 12: loss=0.6971, acc=0.49920

SEQUENTIAL TRAIN, Epoch 13: loss=0.6617, acc=0.67
NONSEQUENTIAL TRAIN, Epoch 13: loss=0.6640, acc=0.67
---> SEQUENTIAL TEST: Epoch 13: loss=0.6494, acc=0.68860
---> NONSEQUENTIAL TEST: Epoch 13: loss=0.6964, acc=0.49920

SEQUENTIAL TRAIN, Epoch 14: loss=0.6506, acc=0.67
NONSEQUENTIAL TRAIN, Epoch 14: loss=0.6527, acc=0.67
---> SEQUENTIAL TEST: Epoch 14: loss=0.6470, acc=0.68860
---> NONSEQUENTIAL TEST: Epoch 14: loss=0.6961, acc=0.49920

SEQUENTIAL TRAIN, Epoch 15: loss=0.6479, acc=0.69
NONSEQUENTIAL TRAIN, Epoch 15: loss=0.6500, acc=0.69
---> SEQUENTIAL TEST: Epoch 15: loss=0.6453, acc=0.68860
---> NONSEQUENTIAL TEST: Epoch 15: loss=0.6954, acc=0.49920

SEQUENTIAL TRAIN, Epoch 16: loss=0.6441, acc=0.70
NONSEQUENTIAL TRAIN, Epoch 16: loss=0.6461, acc=0.70
---> SEQUENTIAL TEST: Epoch 16: loss=0.6445, acc=0.68860
---> NONSEQUENTIAL TEST: Epoch 16: loss=0.6950, acc=0.49920

...

I've found that the issue lies in the BatchNorm1d layer, because when I take it out of the model the issue disappears. Is there a difference between BatchNorm1d in a sequential block? Or have I made a mistake I am overlooking? Thanks in advance for any help!

Here's the full code:

import torch
from torch.utils import data
from torch.optim import Adam
from tqdm import tqdm
class Dataset(data.Dataset):
    'Characterizes a dataset for PyTorch'
    def __init__(self, data, labels):
        self.labels = labels
        self.data = data
    def __len__(self):
        'Denotes the total number of samples'
        return len(self.labels)
    def __getitem__(self, index):
        'Generates one sample of data'
        # Load data and get label
        X = self.data[index]
        y = self.labels[index]
        return X, y

class Net(nn.Module):
    def __init__(self, hidden_dim, in_dim, use_sequential):
        super(Net, self).__init__()
        self.use_sequential = use_sequential
        self.in_dim = in_dim
        self.hidden_dim = hidden_dim

        self.sig = nn.Sigmoid()
        self.encoder = nn.Sequential(
                nn.Linear(in_dim, in_dim),
                nn.BatchNorm1d(in_dim),
                nn.Linear(in_dim, hidden_dim)
                )
        self.decoder = nn.Sequential(
                nn.Linear(hidden_dim, in_dim),
                nn.BatchNorm1d(in_dim),
                nn.Linear(in_dim, in_dim)
                )

        self.lin1 = nn.Linear(in_dim, in_dim)
        self.lin1.weight.data.fill_(0.01)
        self.lin1.bias.data.fill_(0.01)

        self.batchnorm = nn.BatchNorm1d(in_dim)
        self.batchnorm.weight.data.fill_(0.01)
        self.batchnorm.bias.data.fill_(0.01)

        self.lin2 = nn.Linear(in_dim, hidden_dim)
        self.lin2.weight.data.fill_(0.01)
        self.lin2.bias.data.fill_(0.01)

        self.lin3 = nn.Linear(hidden_dim, in_dim)
        self.lin3.weight.data.fill_(0.01)
        self.lin3.bias.data.fill_(0.01)

        self.lin4 = nn.Linear(in_dim, in_dim)
        self.lin4.weight.data.fill_(0.01)
        self.lin4.bias.data.fill_(0.01)

    def encode(self, x):
        if self.use_sequential:
            x = self.encoder(x)
        else:
            x = self.lin1(x)
            x = self.batchnorm(x)
            x = self.lin2(x)
        return x

    def decode(self, x):
        if self.use_sequential:
            x = self.decoder(x)
        else:
            x = self.lin3(x)
            x = self.batchnorm(x)
            x = self.lin4(x)
        return x

    def forward(self, x):
        x = self.encode(x)
        x = self.decode(x)
        x = self.sig(x) # Sigmoid for BCELoss
        return x

def accuracy(preds, labels):
    acc2 = 1 - torch.sum(torch.abs(preds-labels)).item() / (list(preds.size())[0]*list(preds.size())[1])
    return acc2

def generate_data(block_size):
    train_data = torch.randint(2, (10000, block_size)).float()
    test_data = torch.randint(2, (2500, block_size)).float()

    train_labels = train_data
    test_labels = test_data
    return train_data, train_labels, test_data, test_labels

def init_weights(m):
    if type(m) == nn.Linear or type(m) == nn.BatchNorm1d:
        m.weight.data.fill_(0.01)
        m.bias.data.fill_(0.01)
    if type(m) == nn.PReLU:
        m.weight.data.fill_(0.01)

########################## Train code ####################
IN_DIM = 4
HIDDEN_DIM = 32
EPOCHS = 200
BATCH_SIZE = 256

# Generate data
train_data, train_labels, test_data, test_labels = generate_data(IN_DIM)

# Data loading
params = {'batch_size': BATCH_SIZE,
          'shuffle': True,
          'num_workers': 8}
training_set = Dataset(train_data, train_labels)
training_loader = torch.utils.data.DataLoader(training_set, **params)

# Sequential and non-sequential models
model_seq = Net(hidden_dim=HIDDEN_DIM, in_dim=IN_DIM, use_sequential=True)
model_non = Net(hidden_dim=HIDDEN_DIM, in_dim=IN_DIM, use_sequential=False)
model_seq.apply(init_weights)
model_non.apply(init_weights)

loss_fn = nn.BCEWithLogitsLoss()

optimizer_seq = Adam(model_seq.parameters(), lr=0.001)
optimizer_non = Adam(model_non.parameters(), lr=0.001)

# Training
for epoch in range(EPOCHS):
    model_seq.train()
    model_non.train()
    for batch_idx, (batch, labels) in enumerate(training_loader):

        # Testing sequential model
        output_seq = model_seq(batch)
        loss_seq = loss_fn(output_seq, labels)
        optimizer_seq.zero_grad()
        loss_seq.backward()
        optimizer_seq.step()

        # Testing non-sequential model
        output_non = model_non(batch)
        loss_non = loss_fn(output_non, labels)
        optimizer_non.zero_grad()
        loss_non.backward()
        optimizer_non.step()

        if batch_idx % (BATCH_SIZE-1) == 0:
            pred_seq = torch.round(output_seq)
            acc_seq = accuracy(pred_seq, labels)
            print('SEQUENTIAL TRAIN, Epoch %2d: loss=%.4f, acc=%.2f' % (epoch, loss_seq.item(), acc_seq))

            pred_non = torch.round(output_non)
            acc_non = accuracy(pred_non, labels)
            print('NONSEQUENTIAL TRAIN, Epoch %2d: loss=%.4f, acc=%.2f' % (epoch, loss_non.item(), acc_non))

            # Sequential Validation
            model_seq.eval()
            val_output_seq = model_seq(test_data)
            val_loss_seq = loss_fn(val_output_seq, test_labels)
            val_pred_seq = torch.round(val_output_seq)
            val_acc_seq = accuracy(val_pred_seq, test_labels)
            print('---> SEQUENTIAL TEST: Epoch %2d: loss=%.4f, acc=%.5f' % (epoch, val_loss_seq.item(), val_acc_seq))
            model_seq.train()

            # Nonsequential Validation
            model_non.eval()
            val_output_non = model_non(test_data)
            val_loss_non = loss_fn(val_output_non, test_labels)
            val_pred_non = torch.round(val_output_non)
            val_acc_non = accuracy(val_pred_non, test_labels)
            print('---> NONSEQUENTIAL TEST: Epoch %2d: loss=%.4f, acc=%.5f' % (epoch, val_loss_non.item(), val_acc_non))
            model_non.train()

            print('\n')


title: axes don't match array / size mismatch, m1: [132096 x 344], m2: [118336 x 128]

body:
This is a linear auto-encoder code, the original picture is 344*344 RGB, after the training process is over, I want to show the decoded picture using the code below, but it has ValueError: axes don't match array

pytorch, google colab(GPU)

enter code here:
EPOCH = 20
BATCH_SIZE = 128
LR = 0.005  # learning rate

torch.cuda.empty_cache()
data_transforms = torchvision.transforms.Compose([
    torchvision.transforms.RandomResizedCrop(344),
    torchvision.transforms.RandomHorizontalFlip(),
    torchvision.transforms.ToTensor()])
path1 = 'drive/My Drive/Colab/image/test/'
train_data = torchvision.datasets.ImageFolder(path1,
transform=data_transforms)

train_loader = Data.DataLoader(dataset=train_data, batch_size=BATCH_SIZE,
shuffle=True)

class AutoEncoder(nn.Module):
    def __init__(self):
        super(AutoEncoder, self).__init__()

        self.encoder = nn.Sequential(
            nn.Linear(3*344*344, 128),
            nn.Tanh(), # 激活
            nn.Linear(128, 64),
            nn.Tanh(),
            nn.Linear(64, 12),
            nn.Tanh(),
            nn.Linear(12, 3),   # compress to 3 features which can be
visualized in plt
        )
        self.decoder = nn.Sequential(
            nn.Linear(3, 12),
            nn.Tanh(),
            nn.Linear(12, 64),
            nn.Tanh(),
            nn.Linear(64, 128),
            nn.Tanh(),
            nn.Linear(128, 3*344*344),
            nn.Sigmoid(),       # compress to a range (0, 1)
        )

    def forward(self, x):
        x = x.view(x.size(0), -1)

        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return encoded, decoded

autoencoder = AutoEncoder()

optimizer = torch.optim.Adam(autoencoder.parameters(), lr=LR)
loss_func = nn.MSELoss()

for epoch in range(EPOCH):
    for step, (x, b_label) in enumerate(train_loader):
        b_x = x.view(-1, 3*344*344)   # batch x, shape (batch, 28*28)
        b_y = x.view(-1, 3*344*344)   # batch y, shape (batch, 28*28)

        encoded, decoded = autoencoder(b_x)

        loss = loss_func(decoded, b_y)      # mean square error
        optimizer.zero_grad()               # clear gradients for this
training step
        loss.backward()                     # backpropagation, compute
gradients
        optimizer.step()                    # apply gradients

###################################################
######## below is used to plot decoded pic ########

with torch.no_grad():
   for img, label in train_loader :
    fig = plt.figure()

)
    imggg = np.transpose(img[0],(1,2,0))
    ax1 = fig.add_subplot(121)
    ax1.imshow(imggg)

    if torch.cuda.is_available():
      img = Variable(img.to())
    else:
      img = Variable(img)

    encoded, decoded = autoencoder(img)

    decodeddd = np.transpose(decoded.cpu()[0],(1,2,0))

    ax2 = fig.add_subplot(122)
    ax2.imshow(decodeddd)

I expect the output of 2 pics but now only the original one shows, the decoded one doesn't show.

The training process works well, but I don't know what's the problem with picture's size.


title: training fit in echostatenet

body:
I am using an Echo-state-net code implemented on PyTorch, and even though the prediction is accurate enough, the training fit looks shifted, can someone help to understand why this can happen? The shift function is deactivated in the code.
Example image


title: Add redundancy to neural network

body:
How do I make my neural network learn same pattern with minor modifications multiple times, that is one neural network learns minor modifications of the same pattern, and stores them in a list, and returns this list to another neural network?

if I do something like this,

class Model(nn.Module):
  def __init__(self):
    self.lin = nn.Linear(3, 3)
  def forward(self, x):
    out = self.lin(x)
    return out, out

then it would return two outputs, but both of them would be same, I want redundancy, but not exactly same patterns, only minor modifications.


title: Pytorch specific operation for the finding dimension wise mean for a list of tensors

body:
I have a list of tensors, each of 300x1 dimensions and I need to find the mean and variance for each dimension based on the list. How exactly this be achieved?. I could make this work in numpy with numpy arrays, with the following code snippet:

for item in embeddingLists:    # embeddingLists is a list of lists
                               # item contains list of numpy arrays
    tempVal =  np.mean(np.array(item),axis=0)
    meanVects.append(tempVal)

    temVar = np.var(np.array(item),axis=0,ddof=1)
    varVects.append(temVar)


title: How can I optimize the 5-layer loop using functions provided by torch?

body:
x is the tensor with the shape of (16, 10, 4, 25, 53), y has the same size as x.
mean's shape is (25, 53), the size of jc and ac are both (16, 10, 4).

How can I optimize the following expression with torch functions?

for k in range(x.size()[0]):
    for s in range(x.size()[1]):
        for u in range(x.size()[2]):
            for i in range(x.size()[3]):
                for j in range(x.size()[4]):
                    num1 += (x[k][s][u][i][j] - mean[i][j] - jc[k][s][u]) * (y[k][s][u][i][j] - mean[i][j] - ac[k][s][u])
                    num2 += (y[k][s][u][i][j] - mean[i][j] - jc[k][s][u]) ** 2
                    num3 += (y[k][s][u][i][j] - mean[i][j] - ac[k][s][u]) ** 2


title: Index Error while trying to run a neural network

body:
I am new to both python and neural networks.

I am trying to run this

First I did

  sh ./create_dataset.sh

And gave the correct path to scenflow_data_path.

Now I am trying to run this:

  python main.py --maxdisp 192 --with_spn

as given, I am getting this following error. Please let me know how to correct this.

The error is:

Traceback (most recent call last):
  File "main.py", line 194, in <module>
    main()
  File "main.py", line 51, in main
    args.datapath)
  File "/home/kbdp5524/Downloads/AnyNet-master/dataloader/listflowfile.py", line 23, in dataloader
    monkaa_path = filepath + [x for x in image if 'monkaa' in x][0]
IndexError: list index out of range


title: RuntimeError, working on IA tryna use a pre-trained BERT model

body:
Hi here is a part of my code to use a pre-trained bert model for classification:

    model = BertForSequenceClassification.from_pretrained(
     "bert-base-uncased", # Use the 12-layer BERT model, with an uncased vocab.
     num_labels = 2, # The number of output labels--2 for binary classification.
                    # You can increase this for multi-class tasks.
     output_attentions = False, # Whether the model returns attentions weights.
     output_hidden_states = False, # Whether the model returns all hidden-states.
    )

...

for step, batch in enumerate(train_dataloader):

b_input_ids = batch[0].to(device)
    b_input_mask = batch[1].to(device)
    b_labels = batch[2].to(device)

outputs = model(b_input_ids,
                token_type_ids=None,
                attention_mask=b_input_mask,
                labels=b_labels)

but then I receive this error message:

  RuntimeError: Expected tensor for argument #1 'indices' to have scalar
  type Long;

but got torch.IntTensor instead (while checking arguments for embedding)
So I think I should transform my b_input_ids to tensor but don't know how to do it.
Thanks a lot in advance for your help everyone !


title: Getting the index of successive minimum comparisons

body:
Say if i have a tensor

values = torch.tensor([5., 4., 8., 3.])

and i want to take the minimum over every 2 successive values, meaning

min(5., 4.) = 4.
min(8., 3.) = 3.

is there a vectorized way of doing it and still obtaining back the relative index of the minimum? Meaning what I want as output is:

min_index = [1, 1]
#min_index[0] == 1 as 4. is the minimum of (5., 4.) and is in index 1 of (5., 4.)
#min_index[1] == 1 as 3. is the minimum of (8., 3.) and is in index 1 of (8., 3.)


title: Converting PyTorch Module to Networkx Graph?

body: What's the simplest way to convert a PyTorch module (e.g. an RNN, LSTM) into a Python networkx graph?


title: conv1D does 2d convolution by default

body:
So most guides to CNNs explain convolution in one dimension as a series of 1D kernels being convolved with your input sequence (Like traditional FIR filters). However, as far as i can understand, the default of conv1d implements convolution across all your channels for each output (essentially 2D convolution).
If the traditional FIR filter implementation is desired, you should specify groups=in_channels.

Inspecting the weights seems to verify this:

from torch import nn

C1 = nn.Conv1d(in_channels=3, out_channels=6, kernel_size=7)
C2 = nn.Conv1d(in_channels=3, out_channels=6, kernel_size=7,groups=3)
C3 = nn.Conv2d(in_channels=3, out_channels=6, kernel_size=7)
C4 = nn.Conv2d(in_channels=3, out_channels=6, kernel_size=7, groups=3)

print(C1.weight.shape, '<-- 6 filters which convolve across two dimensions')
print(C2.weight.shape, '<-- 6 filters which convolve across one dimensions')
print(C3.weight.shape, '<-- 6 filters which convolve across three dimensions')
print(C4.weight.shape, '<-- 6 filters which convolve across two dimensions')

providing the following output:

torch.Size([6, 3, 7]) <-- 6 filters which convolve across two dimensions
torch.Size([6, 1, 7]) <-- 6 filters which convolve across one dimensions
torch.Size([6, 3, 7, 7]) <-- 6 filters which convolve across three dimensions
torch.Size([6, 1, 7, 7]) <-- 6 filters which convolve across two dimensions

Am i wrong in this observation?

If a am correct, i believe the naming of conv1d is quite confusing as it implies 1d convolution.


title: Does PyTorch have implicit functions for element-wise product and sum?

body:
I am trying to implement RCAN for super resolution in Tensorflow by Yulun Zhang et al (the original code published with paper is implemented in PyTorch: https://github.com/yulunzhang/RCAN).

I am trying to understand how they have implemented RCAB. By looking at the diagram they have published of their network architecture, it seems pretty straight forward how the nueral network is built. But the code doesn't seem to match it.

According to the diagram here: https://raw.githubusercontent.com/yulunzhang/RCAN/master/Figs/RCAB.PNG

Each RCAB should have following structure:

Residual Channel Attention Block(RCAB){

--0) Conv2D
--1) Relu
--2) Conv2D
--3) Channel Attention Layer{
----0)Global pooling
----1)Conv2D
----2)Relu
----3)Conv2D
----4)Sigmoid
----5)Element Wise Product (Input of this layer/function would be the output from the Conv2D layer 3)
--}
--4) Element Wise Sum (Input of this layer/function would be the input of layer 1)
}

However, when I print the PyTorch model in the paper's GitHub repo, RCAB looks like this:
(see https://github.com/yulunzhang/RCAN/blob/master/RCAN_TrainCode/experiment/model/Network_RCAN_BIX2_G10R20P48-2018-07-15-20-14-55.txt for the full printed model)

(0)RCAB(
          (body): Sequential(
            (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (1): ReLU(inplace)
            (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (3): CALayer(
              (avg_pool): AdaptiveAvgPool2d(output_size=1)
              (conv_du): Sequential(
                (0): Conv2d(64, 4, kernel_size=(1, 1), stride=(1, 1))
                (1): ReLU(inplace)
                (2): Conv2d(4, 64, kernel_size=(1, 1), stride=(1, 1))
                (3): Sigmoid()
              )
            )
          )
        )

There seems to be no mention of Element Wise sum and Element Wise product in the RCABs of the models that are published along the paper. Signmoid layer is the last layer in each RCAB.

So my question is: does Pytorch have some implicit way of declaring thes element wise sum/product layers? or is it the case that the publishers of the code/model simply haven't added any such layer and therefore have not followed the model architecture diagram that they published?


title: Cuda always get out of memory when training data with colabs

body:
So i have 2DCNN models to classify image, there are just 2 class, i have 300 images each class.
here is my nn module class

class Flatten(Module):

    def forward(self, input):

        return input.view(input.size(0), -1)

class ActionNet(Module):

    def __init__(self, num_class=4):

        super(ActionNet, self).__init__()

        self.cnn_layer = Sequential(

            #conv1

            Conv2d(in_channels=1, out_channels=32, kernel_size=1, bias=False),

            BatchNorm2d(32),

            PReLU(num_parameters=32),

            MaxPool2d(kernel_size=3),

            #conv2

            Conv2d(in_channels=32, out_channels=64, kernel_size=1, bias=False),

            BatchNorm2d(64),

            PReLU(num_parameters=64),

            MaxPool2d(kernel_size=3),

            #flatten

            Flatten(),

            Linear(576, 128),

            BatchNorm1d(128),

            ReLU(inplace=True),

            Dropout(0.5),

            Linear(128, num_class)

        )

    def forward(self, x):

        x = self.cnn_layer(x)

        return x

i split training and test 80%:20%, for more detail here is my training code: https://pastebin.com/uzBFTrDc , when i try to train with colab there is an error:

  RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.90 GiB total capacity; 15.20 GiB already allocated; 1.88 MiB free; 15.20 GiB reserved in total by PyTorch)

i use google colab because i don't have powerfull GPU and implement batch that i dont know whether it's correct or not, the training data is just 400 image with size 32x32,i think colabs is powerful enough to train my data.