title: Pytorch LSTM model's loss not decreasing

body:
I am writing a program that make use of the build in LSTM in the Pytorch, however the loss is always around some numbers and does not decrease significantly.
My model look like this:
class LSTM(nn.Module):
    def __init__(self, input_size, output_size, hidden_dim, n_layers):
        super(LSTM, self).__init__()
        # Defining some parameters
        self.hidden_dim = hidden_dim
        self.n_layers = n_layers
        #Defining the layers
        # LSTM Layer
        self.lstm = nn.LSTM(input_size, hidden_dim, n_layers, batch_first=True)
        for name, param in self.lstm.named_parameters():
            if 'weight_ih' in name:
                torch.nn.init.xavier_uniform_(param.data)
            elif 'weight_hh' in name:
                torch.nn.init.orthogonal_(param.data)
            elif 'bias' in name:
                param.data.fill_(0)
        # Fully connected layer
        self.fc = nn.Linear(hidden_dim, output_size)
        torch.nn.init.xavier_uniform_(self.fc.weight)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, x):
        batch_size = x.size(0)

        # Initializing hidden state for first input using method defined below
        hidden = self.init_hidden(batch_size)

        # Passing in the input and hidden state into the model and obtaining outputs
        out, hidden = self.lstm(x, hidden)

        # print(out[-1])

        # Reshaping the outputs such that it can be fit into the fully connected layer
        out = out.contiguous().view(-1, self.hidden_dim)
        out = self.fc(out)
        out = self.softmax(out)

        return out, hidden

    def init_hidden(self, batch_size):
        # This method generates the first hidden state of zeros which we'll use in the forward pass
        # We'll send the tensor holding the hidden state to the device we specified earlier as well
        device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        hidden = (torch.zeros(self.n_layers, batch_size, self.hidden_dim, dtype=torch.double, device=device),
                  (torch.zeros(self.n_layers, batch_size, self.hidden_dim, dtype=torch.double, device=device)))
        return hidden

lstm = LSTM(300, 2, 2, 7)
lstm = lstm.double()
lstm.to(device)
criterion = nn.NLLLoss()

learning_rate = 0.6
optimizer = torch.optim.Adam(lstm.parameters(), lr=learning_rate)

And here is the function for each training sample
def epoch(x, y):
    global lstm, criterion, learning_rate, optimizer
    optimizer.zero_grad()
    x = torch.unsqueeze(x, 1)
    output, hidden = lstm(x)
    output = torch.unsqueeze(output[-1], 0)
    loss = criterion(output, y)
    loss.backward()
    optimizer.step()
    return output, loss.item()

And the loss in the training looks like this:

Is there anything wrong with these codes? Please help me. Thanks.


title: Load pytorch model from .pts file

body:
I am wondering if someone can explain how to load a pytorch model saved with the .pts extension and extract the weights from the file?
More specifically, I am able to load the model using:
model = torch.jit.load(model_path.pts,torch.device("cuda"))

which is of the following type:
torch.jit._script.RecursiveScriptModule

and I have looked at the following:
https://www.ccoderun.ca/programming/doxygen/pytorch/classtorch_1_1jit_1_1__script_1_1RecursiveScriptModule.html
I can make forward passes with this but trying to figure out how to get the weights so I can also perform updates on the weights.


title: How to prepare and use the sub-directories Names as Labels for a CNN?

body: I have dataset contains of train, val and test sets. I want to use the subdirectories name as labels. For more info. please this What I missing here, using ImageFolder to get the full folder name as labels for MNIST-double dataset images?


title: memory leak in pytorch autograd function when assigning and returning context data

body:
My memory usage steadily increases when I run the program below.
One can avoid that by either not requiring a grad on the data or even by removing the *2 in the forward method.
Note that there is also no leak when one simply returns x*2 explicitly (i.e. reevaluating).
Is this in any way expected? What exactly happens if I assign data to ctx?
tested with torch version 1.8.1 (cpu only)
import torch
print(torch.__version__)

class LeakyBoi(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        ctx.x = x*2
        return ctx.x

EPOCHS = 1000000

data = torch.ones((100,100), requires_grad=True) # set False to avoid leak
lb = LeakyBoi()

for i in range(EPOCHS):
    lb.apply(data)


Device being used: cpu
Training R3D from scratch...
Total params: 33.23M
Training model on ucf101 dataset...
Preprocessing of ucf101 dataset, this will take long, but it will be done only once.
Traceback (most recent call last):
  File "train.py", line 202, in <module>
    train_model()
  File "train.py", li`enter code here`ne 97, in train_model
    train_dataloader = DataLoader(VideoDataset(dataset=dataset, split='train',clip_len=16), batch_size=20, shuffle=True, num_workers=4)
  File "C:\pytorch-video-recognition\dataloaders\dataset.py", line 41, in __init__
    self.preprocess()
  File "C:\pytorch-video-recognition\dataloaders\dataset.py", line 136, in preprocess
    os.mkdir(train_dir)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'dataset/var/UCF-101\\train\\ApplyEyeMakeup'


title: Pytorch nll_loss returning a constant loss during training loop

body:
I've an image binary classification problem which i want to classify weather an image is of an ant or bee. I've scraped the images and i did all the cleaning, reshaping, converting to grayscale. The images are of size 200x200 one channel grayscale. I first wanted to solve this problem using Feed Forwad NN before i jump to Conv Nets..
My problem during the training loop  I am getting a constant loss I am using Adam Optimizer, F.log_softmax for the last layer in the network as well as the nll_loss function. My code so far looks as follows:
FF - Network
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(in_features , 64)
        self.fc2 = nn.Linear(64, 64)
        self.fc3 = nn.Linear(64, 32)
        self.fc4 = nn.Linear(32, 2)

    def forward(self, X):
        X = F.relu(self.fc1(X))
        X = F.relu(self.fc2(X))
        X = F.relu(self.fc3(X))
        X = F.log_softmax(self.fc4(X), dim=1)
        return X

net = Net()

Training loop.
optimizer = torch.optim.Adam(net.parameters(), lr=0.001)
EPOCHS = 10
BATCH_SIZE = 5
for epoch in range(EPOCHS):
    print(f'Epochs: {epoch+1}/{EPOCHS}')
    for i in range(0, len(y_train), BATCH_SIZE):
        X_batch = X_train[i: i+BATCH_SIZE].view(-1,200 * 200)
        y_batch = y_train[i: i+BATCH_SIZE].type(torch.LongTensor)

        net.zero_grad() ## or you can say optimizer.zero_grad()

        outputs = net(X_batch)
        loss = F.nll_loss(outputs, y_batch)
        loss.backward()
        optimizer.step()
    print("Loss", loss)

I am suspecting that the problem maybe with my batching and the loss function. I will appreciate any help. Note: The images are gray-scale images of shape (200, 200).


title: Torchvision model different behavior after torch.hub

body:
I want to fix some odd import behavior I'm seeing
This does not work:
import torchvision
dilated_model = torchvision.models.resnet50(
    pretrained=True,
    replace_stride_with_dilation=[False, False, True]
)

resnet50() got an unexpected keyword argument 'replace_stride_with_dilation'

But this does
import torch
model = torch.hub.load('pytorch/vision:v0.9.0', 'resnet18', pretrained=True)

import torchvision
dilated_model = torchvision.models.resnet50(
    pretrained=True,
    replace_stride_with_dilation=[False, False, True]
)

What's going on? In the torchvision github code, replace_stride_with_dilation should be valid... and my torch/torchvision packages are up to date (torch=1.8.1, torchvision=0.9.1)

Update: no clue what caused the issue, but it went away after pip uninstall torchvision && pip install torchvision.


title: Pytorch apply various functions element-wise (C++ API)

body:
I want to apply an arbitrary function element-wise on a tensor using the pytorch c++ interface, for example:
double myCustomFunction(double p) {
    return p + 3; // arbitrary function
}

Tensor t = torch::tensor({1,2,3});
t = applyElementwise(&myCustomFunction, t);
std::cout << t << std::endl; // Display: [4,5,6]

I did not find any documentation about it. My best shot for now is the following function
    torch::apply(Function, Parameters);

Defined as:
    template <typename Function, typename... Ts>
    void apply(Function function, Ts&&... ts) {
      // https://stackoverflow.com/questions/13978916/inserting-a-variadic-argument-list-into-a-vector
      // Creates a dummy array, so that each function call is evaluated in order.
      // `(function(), 0)` is because `function` should (!) return `void`, so
      // according to the comma operator, it is evaluated and its result (`void`)
      // is discarded. Then the zero is evaluated and used as an element in the
      // array. The first zero ensures the array is not empty.
      int _[]{0, (function(std::forward<Ts>(ts)), 0)...};
      (void)_;                                    // ^ what is this?
    }

My understanding is that:

an array (called "_") is created with first element "0"
the creation of the second element lead to the call of the function sent as parameter and the return of this function is discard by the ", 0".
finally the return is forced to be void by the line "(void)_;"

I don't understand what the "..." after the second element in the array is for, and this function does not seems to do what I want...


title: RuntimeError: CUDA error: invalid argument torch tensor.cuda() raises error after executing a compiled C++ cuda function

body:
All my operators are:

Write a function in .cu file.
Compile .cu using nvcc xxx.cu -o xxx.so -shared -Xcompiler -fPIC
Load it by ctypes.cdll

The error occurred while:
lib = ctypes.cdll.load('xxx.so')

tmp = torch.ones((1, 1)).cuda()

a = np.ones((1, 1))
b = lib.xxx(a)

tmp = torch.ones((1, 1)).cuda()

Error when execute the second tensor.cuda()
RuntimeError: CUDA error: invalid argument
But even the error occurred, torch.cuda.is_available() is still True.
The error disappears if I never load tensor to gpu, that is:
lib = ctypes.cdll.load('xxx.so')

tmp = torch.ones((1, 1))

a = np.ones((1, 1))
b = lib.xxx(a)

tmp = torch.ones((1, 1)).cuda()

runs with no error.
I want to find a method to execute the function in .so anywhere.


title: My Nano is slower than Raspberry Pi when running Pytorch

body:
I am using Raspberry Pi and Nano to test my model, here is the code I used:
import torch
import torchvision
import PIL.Image
import time

class Predict:
    def __init__(self, path):
        print("Loading model : " + path + "...")
        self.model = torch.load(path)
        self.cuda = torch.cuda.is_available()
        if self.cuda:
            print("cuda is available")
            self.model = self.model.cuda()
        self.model.eval()
        self.transforms = torchvision.transforms.Compose([
            torchvision.transforms.ToTensor(),
            torchvision.transforms.Normalize(mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225]),
        ])
        print("Done!")

    def __call__(self, pic):
        time1 = time.time()
        pic = PIL.Image.open(pic)
        pic = self.transforms(pic)
        pic = pic.unsqueeze(0)
        time2 = time.time()
        if self.cuda:
            pic = pic.cuda()

        time3 = time.time()
        with torch.no_grad():
            output = self.model(pic)
        output = output[0].argmax(0)
        time4 = time.time()
        print(time2 - time1, time3 - time2, time4 - time3)
        return output.item()

if __name__ == "__main__":
    start = time.time()
    model = Predict("model/model.pkl")
    end1 = time.time()
    print(model("0.jpg"))
    end = time.time()
    print(end1 - start)
    print(end - end1)

But the results I got surprised me: about 10 seconds on the Raspberry Pi, and about 30 seconds on the Nano. I want to know why there is such a gap.
Cuda version on Nano is 10.2, Pytorch on Nano is officially provided (v1.8.0); the one on the Raspberry Pi is compiled from the code by myself (v1.9.0a0).


title: A Problem aboud using multi-gpu with a two-stage CNN model

body:
I design a CNN model that have two stage. First stage is generating proposal like RPN in Faster RCNN and then feed them into the second stage, but it causes error in the second step.
Accroding the below error information, it seems like the second input is not correctly assigned to the multi GPU. The model works file with single gpu.
  File "/home/f523/guazai/sdb/rsy/cornerPoject/myCornerNet6/exp/train.py", line 212, in run_epoch
    cls, rgr = self.model([proposal, fm], stage='two')
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 156, in forward
    return self.gather(outputs, self.output_device)
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 168, in gather
    return gather(outputs, output_device, dim=self.dim)
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 68, in gather
    res = gather_map(outputs)
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 55, in gather_map
    return Gather.apply(target_device, dim, *outputs)
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 68, in forward
    return comm.gather(inputs, ctx.dim, ctx.target_device)
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/cuda/comm.py", line 166, in gather
    return torch._C._gather(tensors, dim, destination)
RuntimeError: CUDA error: an illegal memory access was encountered

PS
my model sciprt shows below. I want my two-stage model can support for multi batch. e.g. the batch size is 4 and every img output 128 proposal, so the proposal size in here is (4*128, 5)
def _stage2(self, xs):
    proposal, fm = xs
    if proposal.dim()==2 and proposal.size(1) == 5:
        # train mode
        roi = roi_align(fm, proposal, output_size=[15, 15])
    elif proposal.dim()==3 and proposal.size(2) == 4:
        # eval mode
        roi = roi_align(fm, [proposal[0]], output_size=[15, 15])
    else:
        assert AssertionError(" The boxes tensor shape should be Tensor[K, 5] in train or Tensor[N, 4] in eval")
    x = self.big_kernel(roi)
    cls = self.cls_fm(x)
    rgr = self.rgr_fm(x)
    return cls, rgr


title: same LSTM(GRU) implementation different results (pytorch& keras)

body:
hi i am working about time series data. i have a problem that confused me. i am tuned a neural network with same implementation in both keras and pytorch but had different result.
This is not the only problem. The keras model always gives the same results (Every time I do train model). But the Pytorch model gives the results in 10% of the cases consistent with the cross model. And most of the time it has very bad results that I put(And of course not like the results of keras).
Please guide me. thankssssssssssssss
keras model:
model_input = keras.Input(shape=(x_train_T.shape[1], 8))
x_1 = layers.GRU(75,return_sequences=True)(model_input)
x_1 = layers.GRU(90)(x_1)
x_1 = layers.Dense(95)(x_1)
x_1 = layers.Dense(15)(x_1)
model = keras.models.Model(model_input, x_1)
model.compile(optimizer= adam_optim, loss= "mse" , metrics='accuracy')
model.fit(x_train_T, y_train, batch_size=1, epochs = 100)

pytorch model:
class GRU(nn.Module):
    def __init__(self,input_size, hidden_size_1, hidden_size_2, hidden_size_3, output_size, num_layers, device):
        super(GRU, self).__init__()
        self.input_size = input_size
        self.hidden_size_1 = hidden_size_1
        self.hidden_size_2 = hidden_size_2
        self.hidden_size_3 = hidden_size_3
        self.num_layers = num_layers
        self.device = device

        self.gru_1 = nn.GRU(input_size, hidden_size_1, num_layers, batch_first=True)
        self.gru_2 = nn.GRU(hidden_size_1, hidden_size_2, num_layers, batch_first=True)
        self.fc_1 = nn.Linear(hidden_size_2, hidden_size_3)
        self.fc_out = nn.Linear(hidden_size_3, output_dim)

    def forward(self, x):
        input_X = x
        h_1 = torch.zeros(self.num_layers, input_X.size(0), self.hidden_size_1, device=self.device)
        h_2 = torch.zeros(self.num_layers, input_X.size(0), self.hidden_size_2, device=self.device)

        out_gru_1 , h_1 = self.gru_1(input_X, h_1)
        out_gru_2 , h_2 = self.gru_2(out_gru_1, h_2)
        out_Dense_1 = self.fc_1(out_gru_2[:,-1,:])
        out_Dense_out = self.fc_out(out_Dense_1)

        return out_Dense_out
##############################
input_dim = 8
hidden_dim_1 = 75
hidden_dim_2 = 90
hidden_dim_3 = 95
num_layers = 1
output_dim = 15
num_epochs = 100

model = GRU(input_size=input_dim, hidden_size_1 = hidden_dim_1, hidden_size_2 = hidden_dim_2, hidden_size_3 = hidden_dim_3,output_size = output_dim, num_layers=num_layers, device = device)
criterion = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

import time
for t in range(num_epochs ):
  start_time = time.time()
  loss_p = []
  for i in range(x_train_T.size(0)):
    inputs, target = x_train_T[i:i+1] , y_train[i:i+1]
    inputs = torch.tensor(inputs, dtype=torch.float32).to(device)
    target = torch.tensor(target, dtype=torch.float32).to(device)
    y_train_pred = model(inputs)

    loss_ = criterion(y_train_pred, target)

    optimizer.zero_grad()
    loss_.backward()
    optimizer.step()

    loss_p.append(loss_)
  loss_p = np.array(loss_p)
  loss_P = loss_p.sum(0)/loss_p.shape[0]
  end_time = time.time()
  print("Epoch ", t, "MSE: ", loss_P.item() , "///epoch time: {0} seconds".format(round(end_time - start_time, 2)))
##############################

In rare cases, the loss result of both starts at approximately 0.09 and ends at approximately 0.015.
In most cases, the losses is the same for the keras model , but for the pytorch it stays at 0.08.
i.e , sometimes Pytorch is trained and sometimes not
i think should initialize pytorch layers as same as the keras layers.
but how???
lstm initialization in the keras is as follows:
def __init__(units, activation='tanh', recurrent_activation='sigmoid', use_bias=True, kernel_initializer='glorot_uniform', recurrent_initializer='orthogonal', bias_initializer='zeros', kernel_regularizer=None, recurrent_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, recurrent_constraint=None, bias_constraint=None, dropout=0.0, recurrent_dropout=0.0, return_sequences=False, return_state=False, go_backwards=False, stateful=False, unroll=False, time_major=False, reset_after=True, **kwargs)

and linear layers:
def __init__(units, activation=None, use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None, **kwargs)

how i initializing layers in pytorch????


title: Python: BERT Model Pooling Error - mean() received an invalid combination of arguments - got (str, int)

body:
I am writing the code to train a bert model on my dataset. By when I run the code it throws an error in the average pool layer. I am unable to understand what causes this error.
Model
class BERTBaseUncased(nn.Module):
    def __init__(self, bert_path):
        super(BERTBaseUncased, self).__init__()
        self.bert_path = bert_path
        self.bert = transformers.BertModel.from_pretrained(self.bert_path)
        self.bert_drop = nn.Dropout(0.3)
        self.out = nn.Linear(768 * 2, 1)

    def forward(
            self,
            ids,
            mask,
            token_type_ids
    ):
        o1, _ = self.bert(
            ids,
            attention_mask=mask,
            token_type_ids=token_type_ids)

        apool = torch.mean(o1, 1)
        mpool, _ = torch.max(o1, 1)
        cat = torch.cat((apool, mpool), 1)

        bo = self.bert_drop(cat)
        p2 = self.out(bo)
        return p2

Error
Exception in device=TPU:0: mean() received an invalid combination of arguments - got (str, int), but expected one of:
 * (Tensor input, *, torch.dtype dtype)
 * (Tensor input, tuple of names dim, bool keepdim, *, torch.dtype dtype, Tensor out)
 * (Tensor input, tuple of ints dim, bool keepdim, *, torch.dtype dtype, Tensor out)

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 228, in _start_fn
    fn(gindex, *args)
  File "<ipython-input-12-94e926c1f4df>", line 4, in _mp_fn
    a = _run()
  File "<ipython-input-5-ef9fa564682f>", line 146, in _run
    train_loop_fn(para_loader.per_device_loader(device), model, optimizer, device, scheduler=scheduler)
  File "<ipython-input-5-ef9fa564682f>", line 22, in train_loop_fn
    token_type_ids=token_type_ids
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 577, in __call__
    result = self.forward(*input, **kwargs)
  File "<ipython-input-11-9196e0d23668>", line 73, in forward
    apool = torch.mean(o1, 1)
TypeError: mean() received an invalid combination of arguments - got (str, int), but expected one of:
 * (Tensor input, *, torch.dtype dtype)
 * (Tensor input, tuple of names dim, bool keepdim, *, torch.dtype dtype, Tensor out)
 * (Tensor input, tuple of ints dim, bool keepdim, *, torch.dtype dtype, Tensor out)

I am trying to run this on a Kaggle TPU. How to fix this?


title: Why is generator loss 0.00 but still produces white image in GAN?

body:
I have created a GAN (Generative adversarial network) for creating CIFAR-100 images. The model runs fine but produces white images
My code is as follows (Colab notebook):
import torch
import torch.nn as nn
import torchvision
import torchvision.datasets as datasets
from torch.utils.data import DataLoader
from torchvision import transforms
from torch.utils.tensorboard import SummaryWriter

device = torch.device('cuda') if torch.cuda.is_available() else 'cpu'
print(device)

tfs = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]
)
train_data = datasets.CIFAR100('.', train = True, transform = tfs, download = True)
test_data = datasets.CIFAR100('.', train = False, transform = tfs, download = False)

idx = list(i for i in range(0, 10000))
train_data = torch.utils.data.Subset(train_data, idx)

batch_size = 100
train_loader = DataLoader(train_data, batch_size = batch_size, shuffle = True, num_workers = 4)
test_loader = DataLoader(test_data, batch_size = batch_size, shuffle = True, num_workers = 4)

class conv_layer(nn.Module):
  def __init__(self, in_channels, out_channels, ks, strides = 1, padding = 1,
               normalize = True, dropout = 0.0):
    super(conv_layer, self).__init__()
    layer = nn.Conv2d(in_channels, out_channels, ks, strides, padding)

    # Xavier He init
    nn.init.xavier_normal_(layer.weight)

    layers = [layer]
    if normalize:
      layers.append(nn.InstanceNorm2d(out_channels))
    layers.append(nn.ReLU())
    if dropout:
      layers.append(nn.Dropout(dropout))

    self.model = nn.Sequential(*layers)

  def forward(self, x):
    return self.model(x)

class Discriminator(nn.Module):
  def __init__(self, img_channels):
    super(Discriminator, self).__init__()

    self.l1 = conv_layer(img_channels, 64, 3)
    self.l2 = conv_layer(64, 128, 3, 2, 0)
    self.l3 = conv_layer(128, 256, 3, 2, 0)
    self.l4 = conv_layer(256, 512, 3, 2, 0)
    self.l5 = conv_layer(512, 1, 3, 2, 0, dropout = 0.5)
    self.l6 = nn.Sigmoid()

  def forward(self, x):
    out = self.l1(x)
    out = self.l2(out)
    out = self.l3(out)
    out = self.l4(out)
    out = self.l5(out)
    out = self.l6(out)

    return out

class deconv_layer(nn.Module):
  def __init__(self, in_channels, out_channels, ks, strides = 1, padding = 1,
               normalize = True):
    super(deconv_layer, self).__init__()
    conv_layer = nn.ConvTranspose2d(in_channels, out_channels, ks, strides, padding)

    # Xavier He init
    nn.init.xavier_normal_(conv_layer.weight)

    layers = [conv_layer]
    if normalize:
      layers.append(nn.InstanceNorm2d(out_channels))
    layers.append(nn.LeakyReLU(0.2))

    self.model = nn.Sequential(*layers)

  def forward(self, x):
    return self.model(x)

class Generator(nn.Module):
  def __init__(self, latent_dim, out_channel):
    super(Generator, self).__init__()

    self.l1 = nn.Linear(latent_dim, 256*4*4)
    self.l2 = deconv_layer(256, 128, 4, 2)
    self.l3 = deconv_layer(128, 128, 4, 2)
    self.l4 = deconv_layer(128, 128, 4, 2)
    self.l5 = nn.Conv2d(128, out_channel, 3, 1, 1)
    self.l6 = nn.Sigmoid()

  def forward(self, x):
    out = self.l1(x)
    out = out.reshape(-1, 256, 4, 4)
    out = self.l2(out)
    out = self.l3(out)
    out = self.l4(out)
    out = self.l5(out)
    out = self.l6(out)

    return out

image_shape = (3, 32, 32)
num_classes = 100
lr = 0.01
num_epochs = 500

latent_dim = 100
out_channel = image_shape[0]

gen = Generator(latent_dim, out_channel).to(device)
disc = Discriminator(out_channel).to(device)
optim_gen = torch.optim.Adam(gen.parameters(), lr = lr, weight_decay = 5e-4)
optim_disc = torch.optim.Adam(disc.parameters(), lr = lr, weight_decay = 3e-4)
criterion = nn.BCELoss()
criterion_gan = nn.CrossEntropyLoss()

import os
import shutil
if os.path.exists('log/fake'):
  shutil.rmtree('log/fake')
if os.path.exists('log/real'):
  shutil.rmtree('log/real')

writer_fake = SummaryWriter('log/fake')
writer_real = SummaryWriter('log/real')
step = 0

%reload_ext tensorboard
%tensorboard --logdir log

z_dim = 100
for epoch in range(num_epochs):
    for id, (real, label) in enumerate(train_loader):
        real = real.to(device)
        batch_size = real.shape[0]

        noise = torch.randn(batch_size, z_dim).to(device)
        fake = gen(noise)
        disc.zero_grad()

        # output of discriminator(real)
        disc_real = disc(real).view(-1)
        loss_real = criterion(disc_real, torch.ones_like(disc_real))

        # output of discriminator(fake)
        disc_fake = disc(fake.detach()).view(-1)
        loss_fake = criterion(disc_fake, torch.zeros_like(disc_fake))

        loss_disc = (loss_real + loss_fake) / 2

        loss_disc.backward()
        optim_disc.step()

        # Train generator
        gen.zero_grad()
        gen_image = gen(noise)
        loss_gen = criterion(gen_image, torch.ones_like(gen_image))

        loss_gen.backward()
        optim_gen.step()

        if id == 0:
            print(
                f"Epoch [{epoch}/{num_epochs}] Batch {id}/{len(train_loader)} \
                      Loss D: {loss_disc:.4f}, loss G: {loss_gen:.4f}"
            )

            with torch.no_grad():
              fixed_noise = torch.randn((batch_size, z_dim)).to(device)
              fake = gen(fixed_noise)
              data = real
              img_grid_fake = torchvision.utils.make_grid(fake, normalize=True)
              img_grid_real = torchvision.utils.make_grid(data, normalize=True)

              writer_fake.add_image(
                  "Fake Images", img_grid_fake, global_step=step
              )
              writer_real.add_image(
                  "Real Images", img_grid_real, global_step=step
              )
              step += 1

I have ran it for few epochs and here is the log:
Epoch [0/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.7558
Epoch [1/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [2/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0010
Epoch [3/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0002
Epoch [4/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0004
Epoch [5/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [6/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0007
Epoch [7/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [8/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [9/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [10/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [11/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [12/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [13/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [14/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [15/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [16/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [17/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [18/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [19/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [20/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [21/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [22/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [23/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [24/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [25/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [26/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [27/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [28/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0002
Epoch [29/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [30/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [31/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [32/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [33/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [34/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0004
Epoch [35/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [36/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0004
Epoch [37/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [38/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [39/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [40/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [41/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [42/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [43/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0002
Epoch [44/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [45/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [46/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [47/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [48/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [49/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0002
Epoch [50/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [51/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0008
Epoch [52/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [53/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [54/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [55/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [56/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [57/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [58/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [59/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [60/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [61/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [62/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [63/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [64/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [65/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [66/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001
Epoch [67/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [68/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [69/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [70/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [71/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [72/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0002
Epoch [73/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0000
Epoch [74/500] Batch 0/100                       Loss D: 0.6931, loss G: 0.0001

As you can see both discriminator and Generator losses are stagnant. The images produced are white blank images. I'm not able to understand the reason for this result. Kindly explain. Thankyou.


title: Interpolate UV coordinates

body:
I have a list of RGB values and target UV coordinate positions. More specifically, the RGB color is represented by 3 ints and the position is given as floats (u, v) in the range [0,1].
I now want to write in a (texture) image the RGB values at the given location. For example for a texture image size of 10:
T[int(u * 10), int(v * 10)] = RGB
The problem with this approach is, that i always round down to the next integer which introduces artifacts. Ideally I would interpolate between the possible pixels.
I.e. for (0.21, 0.28) I would not only write the entry [2, 2], but linearly interpolate the RGB values between [2,2] ,[2,3], [3,2], [3,3] appropriately.
Is there a function for this in OpenCV or a similar library? How would you otherwise do it? (For example with numpy?)


title: when I `pip install mmseg`, I got the error 'NameError: name 'reload' is not defined'

body:
However, I got the same problem, when I pip install mmseg. So, what should I do?
The log is like that:
(Monocular3D) root@gpu9:~/MonoRUn$ pip3 install mmseg -i http://pypi.douban.com/simple/  --trusted-host pypi.douban.com
Looking in indexes: http://pypi.douban.com/simple/
Collecting mmseg
  Downloading http://pypi.doubanio.com/packages/f8/31/3bc9205f39cc8ab37193a6fbb24693993b2f305aba9f35b09fad882107ee/mmseg-1.3.0.tar.gz (817 kB)
     |████████████████████████████████| 817 kB 4.4 MB/s
    ERROR: Command errored out with exit status 1:
     command: /home/root/miniconda3/envs/Monocular3D/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-sz89ieuw/mmseg_86227a7da2734023a199ab7a10d06662/setup.py'"'"';
__file__='"'"'/tmp/pip-install-sz89ieuw/mmseg_86227a7da2734023a199ab7a10d06662/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-20_hddgd
         cwd: /tmp/pip-install-sz89ieuw/mmseg_86227a7da2734023a199ab7a10d06662/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-sz89ieuw/mmseg_86227a7da2734023a199ab7a10d06662/setup.py", line 6, in <module>
        reload(sys)
    NameError: name 'reload' is not defined
----------------------------------------
WARNING: Discarding http://pypi.doubanio.com/packages/c9/3a/a38acf3f3ae401082d74cd283d29a873dd3d1b77dc0c7fb28b80f0348073/mmseg-1.2.4.tar.gz#sha256=e7ba753b7bbefe54b4a7c2e27ed6714d1349e30ad6db9d7f67ce67b80b818f4c (from http://pypi.doubanio.com/simple/mmseg/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement mmseg
ERROR: No matching distribution found for mmseg


title: Shuffling along a given axis in PyTorch

body:
I have the a dataset that gets loaded in  with the following dimension [batch_size, seq_len, n_features] (e.g. torch.Size([16, 600, 130])).
I want to be able to shuffle this data along the sequence length axis=1 without altering the batch ordering or the feature vector ordering in PyTorch.
Further explanation: For exemplification let's say my batch size is 3, sequence length is 3 and number of features is 2.
example:
tensor([[[1,1],[2,2],[3,3]],[[4,4],[5,5],[6,6]],[[7,7],[8,8],[9,9]]])  I want to be able to randomly shuffle the following way:
tensor([[[3,3],[1,1],[2,2]],[[6,6],[5,5],[4,4]],[[8,8],[7,7],[9,9]]])
Are there any PyTorch functions that will do that automatically for me, or does anyone know what would be a good way to implement this?


title: pytorch LSTM to map series of feature vectors to their labels

body:
currently I have input X with shape (50, 25), where there are 50 feature vectors and each vector has 25 dimensions. The data of X is, for example, like follows:
X = [[0. 0. 0. ... 1. 1. 1.]
 [0. 0. 0. ... 1. 1. 1.]
 [0. 0. 0. ... 1. 1. 1.]
 ...
 [0. 0. 0. ... 1. 1. 1.]
 [0. 0. 0. ... 1. 1. 1.]
 [0. 0. 0. ... 1. 1. 1.]]

And the output label y is [0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0], of length 50. I.e. each feature vector has a label which corresponds to an element in y.
how can I construct a pytorch LSTM, reshape the input object to 3 dimensions, and properly interpret the output object? Thanks so much for the help in advance.
Currently I have a template for LSTM like this, since my input is already numerical, I was thinking to get rid of the encoder / decoder part, is that correct?
class RNNModel(nn.Module):
    """Container module with an encoder, a recurrent module, and a decoder."""

    def __init__(self, rnn_type, ntoken, ninp, nhid, nlayers, dropout=0, tie_weights=False):
        super(RNNModel, self).__init__()
        self.drop = nn.Dropout(dropout)
        self.ntoken = ntoken
        self.decoder = nn.Linear(nhid, self.ntoken)
        if rnn_type in ['LSTM', 'GRU']:
            self.rnn = getattr(nn, rnn_type)(ninp, nhid, nlayers, dropout=dropout)
        else:
            try:
                nonlinearity = {'RNN_TANH': 'tanh', 'RNN_RELU': 'relu'}[rnn_type]
            except KeyError:
                raise ValueError( """An invalid option for `--model` was supplied,
                                 options are ['LSTM', 'GRU', 'RNN_TANH' or 'RNN_RELU']""")
            self.rnn = nn.RNN(ninp, nhid, nlayers, nonlinearity=nonlinearity, dropout=dropout)

        self.init_weights()

        self.rnn_type = rnn_type
        self.nhid = nhid
        self.nlayers = nlayers

    def init_weights(self):
        initrange = 0.1
        nn.init.zeros_(self.decoder.weight)
        nn.init.uniform_(self.decoder.weight, -initrange, initrange)

    def forward(self, input, hidden):
        emb = self.drop(input)
        emb = emb.transpose(1, 0)

        output, hidden = self.rnn(emb, hidden) #output of shape (length, batchsize, nhid)
        output = self.drop(output)
        output = output[-1, :, :] #shape (batchsize, nhid)

        decoded = self.decoder(output) #shape (batchsize, ntoken)
        return F.log_softmax(decoded, dim=1), hidden

    def init_hidden(self, bsz):
        weight = next(self.parameters())
        if self.rnn_type == 'LSTM':
            return (weight.new_zeros(self.nlayers, bsz, self.nhid),
                    weight.new_zeros(self.nlayers, bsz, self.nhid))
        else:
            return weight.new_zeros(self.nlayers, bsz, self.nhid)

Currently the train I wrote is
X = X.reshape((1, 50, 25))
hidden = self.model.init_hidden(1)
for iter in range(0, self.epochs):
    data = torch.from_numpy(X)
    target = torch.LongTensor(y.reshape((1, torch.LongTensor(y).size(0))))
    self.model.zero_grad()
    self.optimizer.zero_grad()
    hidden = self.repackage_hidden(hidden)

    output, hidden = self.model(data.float(), hidden)
    loss = self.criterion(output, target)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(self.model.parameters(), 0.25)
    self.optimizer.step()
    self.model.train()

But I got the error: RuntimeError: multi-target not supported at /tmp/pip-req-build-4baxydiv/aten/src/THNN/generic/ClassNLLCriterion.c:22


title: Compare two segmentation maps predictions

body:
I am using consistency between two predicted segmentation maps on unlabeled data. For labeled data, I’m using nn.BCEwithLogitsLoss and Dice Loss.
I’m working on videos that’s why 5 dimensions output.
(batch_size, channels, frames, height, width)
I want to know how can we compare two predicted segmentation maps.
gmentation maps.

# gt_seg - Ground truth segmentation map. - (8, 1, 8, 112, 112)
# aug_gt_seg - Augmented ground truth segmentation map - (8, 1, 8, 112, 112)

predicted_seg_1 = model(data, targets)       # (8, 1, 8, 112, 112)
predicted_seg_2 = model(augmented_data, augmented_targets) #(8, 1, 8, 112, 112)

# define criterion
seg_criterion_1 = nn.BCEwithLogitsLoss(size_average=True)
seg_criterion_2 = nn.DiceLoss()

# labeled losses
supervised_loss_1 = seg_criterion_1(predicted_seg_1, gt_seg)
supervised_loss_2 = seg_criterion_2(predicted_seg_1, gt_seg)

# Consistency loss
if consistency_loss == "l2":
      consistency_criterion = nn.MSELoss()
      cons_loss = consistency_criterion(predicted_gt_seg_1, predicted_gt_seg_2)

elif consistency_loss == "l1":
      consistency_criterion = nn.L1Loss()
      cons_loss = consistency_criterion(predicted_gt_seg_1, predicted_gt_seg_2)

total_supervised_loss = supervised_loss_1 + supervised_loss_2
total_consistency_loss = cons_loss

Is this the right way to apply consistency between two predicted segmentation maps?
I’m mainly confused due to the definition on the torch website. It’s a comparison with input x with target y. I thought it looks correct since I want both predicted segmentation maps similar. But, 2nd segmentation map is not a target. That’s why I’m confused. Because if this could be valid, then every loss function can be applied in some or another way. That doesn’t look appealing to me. If it’s the correct way to compare, can it be extended to other segmentation-based losses such as Dice Loss, IoU Loss, etc.?
One more query regarding loss computation on labeled data:
# gt_seg - Ground truth segmentation map
# aug_gt_seg - Augmented ground truth segmentation map

predicted_seg_1 = model(data, targets)
predicted_seg_2 = model(augmented_data, augmented_targets)

# define criterion
seg_criterion_1 = nn.BCEwithLogitsLoss(size_average=True)
seg_criterion_2 = nn.DiceLoss()

# labeled losses
supervised_loss_1 = seg_criterion_1(predicted_seg_1, gt_seg)
supervised_loss_2 = seg_criterion_2(predicted_seg_1, gt_seg)

# augmented labeled losses
aug_supervised_loss_1 = seg_criterion_1(predicted_seg_2, aug_gt_seg)
aug_supervised_loss_2 = seg_criterion_2(predicted_seg_2, aug_gt_seg)

total_supervised_loss = supervised_loss_1 + supervised_loss_2 + aug_supervised_loss_1 + aug_supervised_loss_2

Is the calculation of total_supervised_loss correct? Can I apply loss.backward() on this?


title: Execution block when calling DataLoader with self-defined functions on the dataset class

body:
I am running my code in Colab. I use Dataloader to load my dataset as batches for training and validation, and tqdm to visualize the training process. But every time I execute my code, I get stuck at a random point (in epoch 0, meaning the dataset has never been traversed), saying that the program is still running but the progress bar is frozen. An interesting phenomenon lies in the self-defined transformation functions of the inputs since once I drop these transformations, everything went smoothly. The functions are:
class Sharpen(object):
    def __init__(self, p=0.5):
        self.p = p

    def __call__(self, sample):
        if random.uniform(0.0, 1.0) < self.p:
            return sample
        kernel = np.array([[0, -1, 0], [-1, 5, -1], [0, -1, 0]], np.float32)
        for i in range(len(sample['img_list'])):
            sample['img_list'][i] = cv2.filter2D(sample['img_list'][i], -1, kernel=kernel)

        return sample

class Rotation(object):
    def __init__(self, angle=5, fill_value=0, p=0.5):
        self.angle = angle
        self.fill_value = fill_value
        self.p = p

    def __call__(self, sample):
        if random.uniform(0.0, 1.0) < self.p:
            return sample
        for i in range(len(sample['img_list'])):
            ang_rot = np.random.uniform(self.angle) - self.angle / 2
            h, w, _ = sample["img_list"][i].shape
            if h*w == 0:
                continue
            transform = cv2.getRotationMatrix2D((w / 2, h / 2), ang_rot, 1)
            sample["img_list"][i] = cv2.warpAffine(sample["img_list"][i], transform, (w, h),
                              borderValue=self.fill_value)

        return sample

class Translation(object):
    def __init__(self, fill_value=0, p=0.5):
        self.fill_value = fill_value
        self.p = p
        self.SIGMA = 1e-3

    def __call__(self, sample):
        rand_num = random.uniform(0.0, 1.0)
        if rand_num <= self.p:
            return sample

        for i in range(len(sample['img_list'])):
            h, w, _ = sample["img_list"][i].shape
            if h*w == 0:
                continue
            trans_range = (w / 10, h / 10)
            tr_x = trans_range[0] * rand_num - trans_range[0] / 2 + self.SIGMA
            tr_y = trans_range[1] * rand_num - trans_range[1] / 2 + self.SIGMA
            transform = np.float32([[1, 0, tr_x], [0, 1, tr_y]])
            sample["img_list"][i] = cv2.warpAffine(sample["img_list"][i], transform, (w, h),
                                borderValue=self.fill_value)

        return sample

class Normalization(object):
    def __init__(self, mean=(0,0,0), std=(255,255,255)):
        self.mean = mean
        self.std = std

    def __call__(self, sample):
        # norm_func = transforms.Normalize(self.mean, self.std)
        for i in range(len(sample['img_list'])):
            for j in range(3):  # for colored image
                sample['img_list'][i][:,:,j] = np.array(list(map(lambda x: (x-self.mean[j])/self.std[j], sample['img_list'][i][:,:,j])),
                                     dtype=np.float32)
            # sample['img_list'][i] = norm_func(torch.Tensor(sample['img_list'][i]))
        return sample

The parameter sample is a dictionary, and the key: img_list represents a list of images (numpy.ndarray) processed by cv2.imread. The shape of each image is (h, w, c) and the size is not equal across images.
I created a Dataset Class and proceeded input images with Compose([Sharpen(), Rotation(), Translation(), Normalization()]). Now I don't know what's wrong with the transformation code. Any kind of help is appreciated!


title: Loading ImageNet with wrong classes on pytorch

body:
Dataloader not working properaly...
I have downloaded ImageNet 12 to evaluate it on a pretrained model by pytorch. When I load the classes, they have just values between 0 and 1 which is wrong. Because there 1000 different classes
This is the sample code:
import os, sys, pdb
import argparse
import torch
import torch.nn as nn
import torchvision.datasets as datasets
import torch.utils.data as data
import torchvision.transforms as transforms
import torchvision.models as models

sys.path.insert(0,'..')

if __name__ == '__main__'
    model = models.wide_resnet50_2(pretrained=True)
    model.cuda()
    model.eval()

    # load data
    transform_list = [transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor()]
    transform_chain = transforms.Compose(transform_list)

    # data_dir = 'path_to_imagenet'

    item = datasets.ImageNet(data_dir, split='val', transform=transform_chain)
    test_loader = data.DataLoader(item, batch_size=100, shuffle=False, num_workers=4,pin_memory=True)

    inputs, classes = next(iter(test_loader))
    print("classes: ", classes)
    pdb.set_trace()

The output is
classes:  tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 1])

The imagenet folder looks like
ImageNet
 - ILSVRC2012_devkit_t12
 - - dat
 - - - ILSVRC2012_devkit_t12
 - - - meta.mat
 - - evaluation
 - - - COPYING
 - - - readme.txt
 - train
 - - n03738383
 - - ...
 - - n03337373
 - test
 - - 043344.JPEG
 - - ...
 - val
 - - n03738383
 - - ...
 - - n03337373
 - meta.bin

Edit: When I put shuffle True, then I can see different classes


title: Pytorch BCELoss function different outputs for same inputs

body:
I am trying to calculate cross entropy loss using pytorch's BCELoss Function for a binary classification problem. While tinkering I found this weird behaviour.
from torch import nn
sigmoid = nn.Sigmoid()
loss = nn.BCELoss(reduction="sum")
target = torch.tensor([0., 1.])

input1 = torch.tensor([1., 1.], requires_grad=True)
input2 = sigmoid(torch.tensor([10., 10.], requires_grad=True))
print(input2) #tensor([1.0000, 1.0000], grad_fn=<SigmoidBackward>)

print(loss(input1, target)) #tensor(100., grad_fn=<BinaryCrossEntropyBackward>)
print(loss(input2, target)) #tensor(9.9996, grad_fn=<BinaryCrossEntropyBackward>)

Since both input1 and input2 have same value, shouldn't it return the same loss value instead of 100 and 9.9996. The correct loss value should be 100 since I am multiplying log(0) ~-infinity which is capped at -100 in pytorch. https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html
What is going on here and where am I going wrong?


title: Joint training of two embedding models (KGE + GloVe)

body:
How do I create a joint model that shares the parameters of a Knowledge Graph Embedding (KGE) model, TuckER (given below), and GloVe (assume a co-occurrence matrix along with the dimensions is already available) in PyTorch? In other words, the joint model must obey the criterion of the CMTF Framework and the weights from the two embeddings must be tied during training. The problem here is that the KGE expects a triple (subject, relation, object) whereas the GloVe expects a co-occurrence matrix. Additionally, their loss functions are also computed differently.
class TuckER(torch.nn.Module):
    def __init__(self, d, d1, d2, **kwargs):
        super(TuckER, self).__init__()

        self.E = torch.nn.Embedding(len(d.entities), d1)
        self.R = torch.nn.Embedding(len(d.relations), d2)
        self.W = torch.nn.Parameter(torch.tensor(np.random.uniform(-1, 1, (d2, d1, d1)),
                                    dtype=torch.float, device="cuda", requires_grad=True))

        self.input_dropout = torch.nn.Dropout(kwargs["input_dropout"])
        self.hidden_dropout1 = torch.nn.Dropout(kwargs["hidden_dropout1"])
        self.hidden_dropout2 = torch.nn.Dropout(kwargs["hidden_dropout2"])
        self.loss = torch.nn.BCELoss()

        self.bn0 = torch.nn.BatchNorm1d(d1)
        self.bn1 = torch.nn.BatchNorm1d(d1)

    def init(self):
        xavier_normal_(self.E.weight.data)
        xavier_normal_(self.R.weight.data)

    def forward(self, e1_idx, r_idx):
        e1 = self.E(e1_idx)
        x = self.bn0(e1)
        x = self.input_dropout(x)
        x = x.view(-1, 1, e1.size(1))

        r = self.R(r_idx)
        W_mat = torch.mm(r, self.W.view(r.size(1), -1))
        W_mat = W_mat.view(-1, e1.size(1), e1.size(1))
        W_mat = self.hidden_dropout1(W_mat)

        x = torch.bmm(x, W_mat)
        x = x.view(-1, e1.size(1))
        x = self.bn1(x)
        x = self.hidden_dropout2(x)
        x = torch.mm(x, self.E.weight.transpose(1,0))
        pred = torch.sigmoid(x)
        return pred

I know how to jointly train two pre-trained models by loading the state dicts, taking an instance, running them on the two models, and then applying a feedforward layer on top. But I seem to be not able to figure this scenario out. Can you please suggest how I can achieve this?


title: Why is my network changing without me calling the optimizer?

body:
I've got a network in a reinforcement learning program where it plays against itself and updates to improve a valuation and policy. If the game is a draw or longer than a threshold the game is thrown out and the loss is never calculated and the optimizer is never called. Still, even if it does not update it still saves the network. It does this and loads the data with these functions

def load_net(i):
    if not os.path.exists('C:\\Users\\user\\Desktop\\python\\refactor\\nets\\net{}.pth'.format(i)):
        return None
    nn = net.Net()
    nn.load_state_dict(torch.load('C:\\Users\\user\\Desktop\\python\\refactor\\nets\\net{}.pth'.format(i)))
    nn.double()
    nn.train()
    print('loaded net {}'.format(i))
    return nn

def load_latest_net():
    i = 0
    while os.path.exists('C:\\Users\\user\\Desktop\\python\\refactor\\nets\\net{}.pth'.format(i)):
        i += 1
    i -= 1
    nn = net.Net()
    nn.load_state_dict(torch.load('C:\\Users\\user\\Desktop\\python\\refactor\\nets\\net{}.pth'.format(i)))
    nn.double()
    nn.train()
    print('loaded latest net {}'.format(i))
    return nn

def save_net_as_latest(net):
    i = 0
    while os.path.exists('C:\\Users\\user\\Desktop\\python\\refactor\\nets\\net{}.pth'.format(i)):
        i += 1
    print('saving net as {}'.format(i))
    torch.save(net.state_dict(), 'C:\\Users\\user\\Desktop\\python\\refactor\\nets\\net{}.pth'.format(i))

If I compare the output of Networks 18 and 19, say, that have not been updated inbetween, I will get a slightly different result. By the order of like 4 decimal places. I have no idea why. Besides the training which im positive is not happening here there are only calls such as

output = model(input)

I may have some redundant or unnecessary .double() calls. Otherwise I'm perplexed. Here is some output that shows what im talking about

loaded net 31
loaded net 24
0.03457238742588508
{(256, 0): 0.1320085660843816, (256, 1): 0.137342432039808, (16384, 0): 0.17418085554435797, (16384, 1): 0.10271324890344476, (1048576, 0): 0.19074605973319944, (1048576, 1): 0.16643130291875174, (67108864, 0): 0.09657753477605639}
-0.04351635692025382
{(256, 0): 0.12781107650911216, (256, 1): 0.1366737843176031, (16384, 0): 0.17335968033226215, (16384, 1): 0.10237753363808048, (1048576, 0): 0.19300295476929272, (1048576, 1): 0.1690421767620045, (67108864, 0): 0.09773279367164493}


title: RuntimeError: Type ‘Tuple when using Deep Markov Model

body:
I am trying to use deep Markov Model on my dataset. However, when I use it I get the following run time error:

enter image description here

My code is exactly similar to:

https://github.com/pyro-ppl/pyro/blob/dev/examples/dmm/dmm.py

I don't even understand what this error means. Has anyone seen this error before? Insights would be appreciated. This error occurs in guide function (line # 225) I think. But I don't know what is triggering this error.


title: subrandomsapler dataloader pytorch lengh does not seems right

body:
split = int(len(train_dataset) * 0.8)
    print(split)
    index_list = list(range(len(train_dataset)))
    train_idx, valid_idx = index_list[:split], index_list[split:]
    print(len(train_idx),len(valid_idx))
48000 12000

I got this 48000 and 12000 index for train_idx and valid_idx
then I apply this number to the dataloader

train_loader = DataLoader(train_dataset, batch_size=batch_size, sampler= SubsetRandomSampler(train_idx))
valid_loader = DataLoader(train_dataset, batch_size=batch_size, sampler=SubsetRandomSampler(valid_idx))

  print(len(train_loader.dataset),len(valid_loader.dataset))
  60000 60000

but the len does not seemd right

for epoch in range(EPOCHS):
    for i , (train_idx, valid_idx) in enumerate(splits):
        ## create iterator objects for train and valid datasets
        train_loader = DataLoader(train_dataset, batch_size=batch_size, sampler= SubsetRandomSampler(train_idx))
        valid_loader = DataLoader(train_dataset, batch_size=batch_size, sampler=SubsetRandomSampler(valid_idx))
        submit_loader = DataLoader(dataset = test_dataset,batch_size = batch_size, shuffle = True)

        train_loss, train_acc = train(model, device, train_loader, optimizer, criterion)
        valid_loss, valid_acc = evaluate(model, device, valid_loader, criterion)

        if valid_loss < best_valid_loss:
            best_valid_loss = valid_loss
            torch.save(model.state_dict(), MODEL_SAVE_PATH)

        print(f'| Epoch: {epoch+1:02} | Train Loss: {train_loss:.3f} | Train Acc: {train_acc*100:05.2f}% | Val. Loss: {valid_loss:.3f} | Val. Acc: {valid_acc*100:05.2f}% |')


title: chain rule output for matrices after nn.Linear

body:
in the case of applying chain rule, for example, we have,

x = torch.tensor([1.], requires_grad=True)
def square(x):
  return x**2
def exp(x):
  return torch.exp(x)
z = exp(square(x)) # e^x**2
# derivative is e^(x**2) * 2*x

so when we do,

z.backward()
x.grad

it gives

tensor([5.4366])
which is,
torch.exp(torch.tensor([1.])**2) * 2 * torch.tensor([1.])

but when using matrices,

input = torch.randn(3, 3)
a = nn.Linear(3, 3, bias=False)
b = nn.Linear(3, 3, bias=False)
z = b(a(input)).sum()
z.backward()

then, we get these matrices,

for param in a.parameters():
  print(param.grad)

tensor([[-0.9843,  0.5633,  0.4002],
        [ 0.7303, -0.4179, -0.2969],
        [ 1.9923, -1.1402, -0.8101]])

for param in b.parameters():
  print(param.grad)

tensor([[ 1.8401,  0.5115, -1.3559],
        [ 1.8401,  0.5115, -1.3559],
        [ 1.8401,  0.5115, -1.3559]])

how are these two matrices computed?


title: Can't find folder in filesystem on Juypter

body:
I have downloaded files using the following function download_from_url and it seems to download them and put them into a folder called .data however I can't seem to find the folder in the file system. Everything works fine otherwise and the files seem to exist but I can't see them.

Any ideas where they are?

(I tried printing out the path and I got the following: "./.data/yelp_review_polarity_csv/train.csv" but I still can't find that anywhere.)


title: How to assign parameters to Batch Normalizationi layer in Tensorflow1.X？

body:
I have download the parameters of pretrained ResNet34 in Pytorch Models, and looking forward to assign them to a tensorflow1.X ResNet model.

But I don't know how to assign running_var, running_mean, gamma and beta to the corresponding BN layer in Tensorflow1.X, could you please tell me how to implement it?


title: Replacement for datasets.ImageFolder (Pytorch) to work with the RVL-CDIP Dataset

body:
I am currently working on a document recognition project using Deep Learning with Pytorch.

I was using ImageFolder from torchvision.datasets to indicate the path to the folders containing my images and to normalize them, but someone told me that I will have to train my model using the RVL-CDIP Dataset, which is not organized in a directory tree by classes.

What replacement could I use to work with this dataset?


title: Train a classifier with 10 label which only classify two of them

body:
I need to train a classifier on Cifar10, such that it could classify the images with the labels 0 and 1 and for the labels other than these two labels the classifier provide unbiased (neutral) results. I try to train it using the following code with BCE loss:

   input = model(x)
   not_ind_tgt = ~((target==0) | (target==1))
   tgt_hot = F.one_hot(target, 10).float()
   multi_label = 1./8. * torch.ones(1,10).float()
   multi_label[0, 0: 2] = 0.0
   tgt_hot[not_ind_tgt] = multi_label
   loss = nn.BCEWithLogitsLoss()(input, tgt_hot).mean()

However, the accuracy of the trained classifier on 0 and 1 classes is zero.


title: How can I fix the scaling optimization pytorch code error

body:
I want make a scaling optimization system

class GraphFit:
  def __init__(self):
    self.X=torch.tensor([])
    self.Y=torch.tensor([])

    self.scale_x=torch.Tensor([])
    self.scale_y=torch.Tensor([])

    self.opt=0

  def scale(self,imagx=1.,imagy=1):
    self.scale_x=torch.tensor([1.],requires_grad=True)
    self.scale_y=torch.tensor([1.],requires_grad=True)

    self.X=self.scale_x *self.X
    self.Y=self.scale_y *self.Y

  def optimize(self,lr=1e-03):
    params=[self.scale_x,self.scale_y]
    self.opt=torch.optim.Adam(params,lr)

x=torch.linspace(-10,10,100)

yt=0.5*torch.sin(0.2 * x)   ##truth

y=torch.sin(x)   ## try

G1=GraphFit()
G1.X=x

G1.scale()

G1.optimize()

L=nn.MSELoss()

for i in range(1):  ####
  G1.opt.zero_grad()
  G1.Y=torch.sin(G1.X)
  err=L(G1.Y,yt)
  err.backward()
  G1.opt.step()

  if i%100==0:
    print(err)

print(G1.scale_y.grad,G1.scale_x.grad)

When I use  'for i in range(1)' , its output is
tensor(0.1189, grad_fn=<MseLossBackward>)
None tensor([-0.0425])

and when I use 'for i in range(N)' for N >1
it's output is
---> 83   err.backward()

Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

What is the problem and how can I fix it?

########################
dummydummydummydummydummydummydummydummydummydummy
dummydummydummydummydummydummydummydummydummydummy
dummydummydummydummydummydummydummydummydummydummy
dummydummydummydummydummydummydummydummydummydummy


title: CNN Model Definition input and output change with RGB

body:
I am trying to a traffic sign dataset using a CNN, i have done it was a grayscale conversion but want to see the affect of using RGB. I am using a size of 32x32x1 but this will now be 32x32x3.
class ConvNet(nn.Module):
def __init__(self):
    super(ConvNet, self).__init__()
    self.conv1 = nn.Conv2d(in_channels=1,
                           out_channels=6,
                           kernel_size=(5,5),
                           stride=(1,1))
    self.pool2 = nn.MaxPool2d(kernel_size = (2,2))
    self.conv3 = nn.Conv2d (in_channels=6,
                            out_channels=16,
                            kernel_size=(5,5),
                            stride=(1,1))
    self.pool4 = nn.MaxPool2d(kernel_size=(2,2))
    self.conv5 = nn.Conv2d (in_channels=16,
                           out_channels=120,
                           kernel_size=(5,5),
                           stride=(1,1))
    self.fc6 = nn.Linear(in_features=120,
                         out_features=84)
    self.fc7 = nn.Linear(in_features=84,
                         out_features=62)
def forward(self,x):
    x = x.view([-1, 1, 32, 32])
    x = nn.functional.relu(self.conv1(x))
    x = self.pool2(x)
    x = nn.functional.relu(self.conv3(x))
    x = self.pool4(x)
    x = nn.functional.relu(self.conv5(x))
    x = x.view(-1, 120)
    x = nn.functional.relu(self.fc6(x))
    x = self.fc7(x)
    return x

What I cant understand how to do is the definition of the layers with the additional channels. Would appreciate help in defining this.


title: No Parameters in Net class even after inheriting from nn.Module

body:
For some reason, my Class Net has no parameters, even though i have imported everything correctly., its still showing that net has no parameters or attributes and even on printing net its just prints empty list. Can anyone tell me what's wrong ?
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
def ___init__(self):
    super(Net, self).__init__()
    self.fc1 = nn.Linear(784, 64)
    self.fc2 = nn.Parameter(nn.Linear(64, 64))
    self.fc3 = nn.Parameter(nn.Linear(64, 64))
    self.fc4 = nn.Parameter(nn.Linear(64, 10))

def forward(self, x):
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = F.relu(self.fc3(x))
    x = self.fc4(x)
    return F.log_softmax(x, dim=1)

net = Net()
print(net)
print(list(net.parameters()))

Output:
Net()
[]


title: How to assign dtype to a Pyro model parameters?

body:
I am trying to convert myPyTorchModel into a Bayesian Pyro model (see code below).
However, when I execute the line next(myPyTorchModel.parameters()).dtype, StopIteration error pops up. I am wondering if there is any way to prevent Pyro models from generating the StopIteration error.
My code is below:
import pyro
import pyro.infer
import pyro.optim
import pyro.distributions as dist
import pyro.nn.module as module
import torch

# convert myPyTorchModel into a Bayesian Pyro model
module.to_pyro_module_(myPyTorchModel)

# assign distributions to myPyTorchModel parameters
for m in myPyTorchModel.modules():

    for name, value in list(m.named_parameters(recurse=False)):

        options =  dict(dtype=torch.double, device="cpu")
        prior_loc = torch.zeros(1, 1, **options)
        prior_scale = torch.ones(1, 1, **options)
        zs = module.PyroSample(dist.Normal(prior_loc, prior_scale).to_event(1))
        zs.double()
        setattr(m, name, zs)

# this generates an error
next(myPyTorchModel.parameters()).dtype

OUT:
Traceback (most recent call last):

  File "<ipython-input-37-f9ae77a21ef4>", line 1, in <module>
    next(myPyTorchModel.parameters()).dtype

StopIteration

How can I fix this error?


title: Sampling in each region according to a segmentation map in PyTorch

body:
I want to sample a certain number of points in each region of the image according to a segment map(like what the SLIC algorithm produces, it's a map with the same size as the image, containing integers from 0 to num_segment indicating which segment each pixel belongs to).
Currently, I write my own code as follows:

For i in range(0, num_segment):

find the (indexes of the)pixels that belong to the ith segment using torch.where
picking out those pixels and forming a 1-d tensor
use torch.Upsample to uniformly sample n_sample points for the ith segment

stack all the sampled points to form a large 2-d tensor which each row represent selected points belong to one segment, and it has n_sample rows.

I not only want the original value from the image for each selected point, but their indexes are also needed.
I drew a picture to illustrate.
So, my question is, is there a native way to implement this in PyTorch? The above code runs a little bit slow, maybe the For-loop slows down the speed. And if possible, since all the sampling process are independent of each other, how can I speed up the process?
Generally, I have ~400 segments, and I want to sample 20~50 points for each segment.


title: How to use nn.Embeddings for grouped data?

body:
I am new to using Torch and also new to use LSTM, I am working on time series sales forecasting for a dataset with the following data structure:
Data Structure
I wanna do nn.embeddings (as I have read it acts like one hot encoding) to the locations, so that instead of running an individual model for each location, run one model for all the data. So, the final structure should be like that
After Embeddings
what parameters should I pass to nn.embeddings?
Given that my data loading function is as follows:
def sliding_windows(data, seq_length):
    x = []
    y = []

    for i in range(len(data)-seq_length-1):
        _x = data[i:(i+seq_length)]
        _y = data[i+seq_length]
        x.append(_x)
        y.append(_y)

    return np.array(x),np.array(y)

sc = MinMaxScaler()
training_data = sc.fit_transform(training_set)

seq_length = 4
x, y = sliding_windows(training_data, seq_length)

y = (y[:,0]).reshape(y[:,0].shape[0],1)

train_size = int(len(y) * 0.9)
test_size = len(y) - train_size

dataX = Variable(torch.Tensor(np.array(x)))
dataY = Variable(torch.Tensor(np.array(y)))

trainX = Variable(torch.Tensor(np.array(x[0:train_size])))
trainY = Variable(torch.Tensor(np.array(y[0:train_size])))

testX = Variable(torch.Tensor(np.array(x[train_size:len(x)])))
testY = Variable(torch.Tensor(np.array(y[train_size:len(y)])))

and the model is as follows:
class LSTM(nn.Module):
    def __init__(self, num_classes, input_size, hidden_size, num_layers):
        super(LSTM, self).__init__()

        self.num_classes = num_classes
        self.num_layers = num_layers
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.seq_length = seq_length

        self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size,
                            num_layers=num_layers, batch_first=True)

        self.fc = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        h_0 = Variable(torch.zeros(
            self.num_layers, x.size(0), self.hidden_size))

        c_0 = Variable(torch.zeros(
            self.num_layers, x.size(0), self.hidden_size))

        # Propagate input through LSTM
        ula, (h_out, _) = self.lstm(x, (h_0, c_0))

        h_out = h_out.view(-1, self.hidden_size)

        out = self.fc(h_out)

        return out

and finally the training loop is as follows:
num_epochs = 1000
learning_rate = 0.008

input_size = 4
hidden_size = 4
num_layers = 1

num_classes = 1

lstm = LSTM(num_classes, input_size, hidden_size, num_layers)

criterion = torch.nn.MSELoss()    # mean-squared error for regression
optimizer = torch.optim.Adam(lstm.parameters(), lr=learning_rate,weight_decay=0.001)
# Weight_decay to decay the errors == regularization technique
# optimizer = torch.optim.SGD(lstm.parameters(), lr=learning_rate)

# Train the model
for epoch in range(num_epochs):
    outputs = lstm(trainX)
    optimizer.zero_grad()

    # obtain the loss function
    loss = criterion(outputs, trainY)

    loss.backward()

    optimizer.step()
    if epoch % 100 == 0:
      print("Epoch: %d, loss: %1.5f" % (epoch, loss.item()))

and what should I do as well for prediction part?
Thanks in advance


title: Latent vector variance much larger in CoreML than PyTorch

body:
I have a PyTorch model that I've converted to CoreML. In PyTorch, the inferred latent vector's values range to a limit of around ±1.6, but the converted mlmodel varies as much as ±55.0. What might be causing this huge discrepancy? The conversion is pretty straightforward:
encoder = Encoder_CNN(latent_dim, n_c)
enc_fname = os.path.join(models_dir, encoder.name + '_153.pth.tar')
encoder.load_state_dict(torch.load(enc_fname, map_location={'cuda:0': 'cpu'}))
encoder.cpu()
encoder.eval()

img_in = torch.rand(1, 1, 28, 28).cpu()
traced_encoder_model = torch.jit.trace(encoder, img_in)
traced_encoder_model.save("{}_encoder_mobile_{}_{}.pt".format(model_type, latent_dim, n_c))

coreml_encoder = ct.convert(traced_encoder_model, inputs=[ct.ImageType(name=name, shape=img_in.shape)])
coreml_encoder.save("{}_encoder_{}_{}.mlmodel".format(model_type, latent_dim, n_c))


title: How to transform pytorch tensor to view such as ((value0, row0, column0), (value1, row0, column1)...)?

body:
How do I transform PyTorch tensor to view such as ((value0, row0, column0), (value1, row0, column1)...)? For example, if I have a tensor
a = torch.tensor([[5, 7],
                  [9, 11]])

The function should return:
((5, 0, 0), (7, 0, 1), (9, 1, 0), (11, 1, 1)). Also, I want it function to be applied for constants, 1D, 3D, 4D, etc. tensors, e.g. for 3D I want it to output (value0, dim1, dim2, dim3).
I tried to use for loops but didn't realize how to use them for this task: I can make a function only for a constant number of dimensions, such as only for 2D.


title: Training Sagemaker Pytorch model: how to provide an existing S3 bucket with data?

body:
I have data hosted on AWS S3 that I would like to feed into  a Pytorch model.
I saw this example where Sagemaker uploads data from EBS to S3, which makes me think it must be possible.
Anyone aware of a way to feed it a list of pre-upoaded S3 JPEGs or PNGs together with a CSV of labels?


title: Why BatchIterator transforms my label values?

body:
My code is like that:

TEXT = data.Field(batch_first=True,include_lengths=True)
LABEL = data.LabelField(dtype = torch.float,batch_first=True)

fields = [(None, None),('text',TEXT),('label', LABEL)]

training_data=data.TabularDataset(path = 'drive/My Drive/Colab Notebooks/Heyhey/scaled_data_235.csv',format = 'csv',fields = fields,skip_header = True)

print(vars(training_data.examples[0]))

output of training_data.examples[0]:
{'text': ['今夜', 'スパムカレー', 'ﾟ', 'ﾟﾉﾊｲｻｲ', 'tcokSafSSrg', 'フリーランス'...], 'label': '0.7647058823529411'}
import random
train_data, valid_data = training_data.split(split_ratio=0.7, random_state = random.seed(SEED))

#initialize glove embeddings
TEXT.build_vocab(train_data,min_freq=3)
LABEL.build_vocab(train_data)

#No. of unique tokens in text
print("Size of TEXT vocabulary:",len(TEXT.vocab))

#No. of unique tokens in label
print("Size of LABEL vocabulary:",len(LABEL.vocab))

#Commonly used words
print(TEXT.vocab.freqs.most_common(10))

#Word dictionary
print(TEXT.vocab.stoi)

Size of TEXT vocabulary: 13365
Size of LABEL vocabulary: 16
[('ない', 4225), ('れ', 1901), ('なっ', 997), ('', 963), ('なく', 702), ('いい', 658), ('フォロー', 631), ('だっ', 618), ('✨', 607), ('日本', 550)]
defaultdict(<function _default_unk_index at 0x7f2f19709b70>, {'': 0, '': 1, 'ない': 2, ...,\U0001f9d0': 13364})
#check whether cuda is available
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

#set batch size
BATCH_SIZE = 8

#Load an iterator
train_iterator, valid_iterator = data.BucketIterator.splits(
    (train_data, valid_data),
    batch_size = BATCH_SIZE,
    sort_key = lambda x: len(x.text),
    sort_within_batch=True,
    device = device)

for batch in train_iterator:
  print(batch.label)

Batch.label returns like that:
tensor([0., 2., 1., 6., 1., 7., 4., 0.], device='cuda:0')
tensor([2., 0., 8., 0., 6., 2., 4., 3.], device='cuda:0')
...
My question is that: first batch. label value supposed to return as '0.7647058823529411 but it returns an integer between 0 and 16. I try to make regression but it seems but BatchIterator converts it to classifier. How can I return 0.764 instead of [0-16]?


title: How can i realize a heat map from the input and output images of a GAN,to check where the pixel change more

body: I would like to create a function that evaluates the input and output of a U-net (which is the generator of a GAN),and realize a heat map,to check where there has been the greatest variation in pixel values. I'm not being very experienced with python (the network is written in pytorch) I don't know how to do this.


title: how to add L2 regulazation to cost function pytorch?

body:
I try add L1 and L2 regulazation in my Loss function. But I fail.
My code:
criterion = nn.NLLLoss() + nn.L1Loss()
it should be perfect for something like this:
criterion = nn.NLLLoss() + _lambda * nn.L1Loss()
How can i do it?


title: Can't pickle with local class imported

body:
As long as I import class from my own library, my pickle failed, why?
For example:
def fun():
   print(1)
if __name__=="__main__":
   import pickle
   pickle.dump(fun, open('save', 'wb'))

The script above works fine, but if I add from xxx.yyy import zzz (my class from my local library):
_pickle.PicklingError: Can't pickle <function fun at 0x7f5e40665e18>: attribute lookup fun on __main__ failed occurs.
Why? Is there any workaround? By the way, the problem originally occurs when I was trying to use torch.multiprocessing for distributed training, then I reduce the problem to above.


title: Concepts of nn.Embedding and vq-vae codebook

body:
Are these two concepts the same idea?
or does it make sense to implement codebook with the nn.Embedding class (pytorch)?
I know nn.Embedding is used for a trainable lookup table.
And codebook (in vq-vae) seems with the same idea to me (trainable, also a lookup table)
so what's the difference?
I can not clearly distinguish them. Thanks.


title: How to do tokenization from a predifined vocab in tensorflow or pytorch or keras?

body: I have a predefined vocab which build from the common-used 3500 Chinese characters. Now I want to tokenize the Dataset with this vocab to fix each character. Any mature class or function exists I can inherit from to buid the data reading pipline?


title: Keras Custom Generator looping through Multiple Folders

body:
I am trying to build a custom Keras generator that would loop through folders and read a large numpy array. So i have folders like this
data
  --- train
     ---20200506
        ---X
           ---ProductA.npy
           ---ProductB.npy
           ---ProductC.npy
        ---Y
           ---ProductA.npy
           ---ProductB.npy
           ---ProductC.npy
     ---20200507
        ....
  --- test
  --- val

I have a generator now using Keras generator that will read all files (for all dates) for ProductA from training folder, testing folder and validation folder and i return 3 generators like this. However this approach fails when the number of dates are too high, since I first append all data for the several dates for a particular product sometimes i am running out of memory as the number of days get large. Is there some way i can amend my generator to also take this into account? Note that I am using a modified version of this and as below we need list_IDs and labels of all data, i decided to append all data for products. This is now causing memory issues. Any help will be greatly appreciated :)
https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly


title: How to divide the GLUE MNLI dataset into query and support sets for training BERT using MAML?

body:
I am attempting to train the BERT model on the MNLI dataset by using Model Agnostic Meta Learning (MAML). The MNLI dataset has files for train, validation and test sets respectively. Should I use the train set as the query set and the validation set as the support set?
So far I tried to construct the query and support batch from the train set by setting n = 3 and k = 5, to do a 3 way - 5 shot learning. But the performance has been worse than normal training. I was wondering if I am supposed to make the query and and support sets in this manner.


title: PyTorch: torchsummary for loaded model

body:
I have the following code to print the model summary of a previously saved model:
import torch

model = torch.load('results/model_best.pth.tar')
print(model) # works

from torchsummary import summary

summary(model, (3, 224, 224)) # error

The print method works okay, but the summary method has the following error:
Traceback (most recent call last):
  File "ModelSummary.py", line XX, in <module>
    summary(model, (3, 224, 224))
  File "anaconda3/lib/python3.8/site-packages/torchsummary/torchsummary.py", line 68, in summary
    model.apply(register_hook)
AttributeError: 'dict' object has no attribute 'apply'

How to rectify this?
(I tried the summary method on a model from torchvision database. It worked okay).
Thank you.


title: Loop over tensor of vectors (size [batch_size, 1, 28])

body:
I have a network where a vector of size [1,28] is generated for every input in a batch (of size 128). Now I want to access the values in these vectors for the whole batch, so I want to return all 28 values for the 128 inputs in the batch. In other words, I want to loop over a tensor of size [128, 1, 28]. I could use multiple for-loops, but is there a nice pythonic way to achieve this?
Edit:
I'm asking because I want to eventually compute the gradients of each of the z_i in the vector z with respect to one of my network layer's output. I'm working with a variational autoencoder, where the encoder network outputs a mean and variance which is reparameterized to a latent vector z of size [1, 28]. Now given this vector z, I want to compute gradients for each element z_i wrt to the output of an earlier layer in the encoder. I hope this clarifies things!


title: mmdetection cannot get the same results among multi runs after setting the random seed

body: Even though I set the random seed, and deterministic is true, the values of losses are different among multi runs.


title: How to implement randomised log space search of learning rate in PyTorch?

body:
I am looking to fine tune a GNN and my supervisor suggested exploring different learning rates. I came across this tutorial video where he mentions that a randomised log space search of hyper parameters is typically done in practice. For sake of the introductory tutorial this was not covered.
Any help or pointers on how to achieve this in PyTorch is greatly appreciated. Thank you!


title: Dealing with bias in multi-channel auto encoders

body:
The problem
I have a multi-channel 1D signal I want to auto-encode.
I am unable to resonstruct the input when the number of channels increases.

Code
I am using a convolutional encoder, and a convolutional decoder:
latent_dim: 512, frames_per_sample: 128
    self._encoder = nn.Sequential(
        nn.Conv1d(in_channels=self._n_in_features, out_channels=50, kernel_size=15, stride=1, padding=7),
        nn.LeakyReLU(inplace=True),
        nn.Conv1d(in_channels=50, out_channels=50, kernel_size=7, stride=1, padding=3),
        nn.LeakyReLU(inplace=True),
        nn.Conv1d(in_channels=50, out_channels=50, kernel_size=3, stride=1, padding=1),
        nn.LeakyReLU(inplace=True),
        # nn.Flatten(start_dim=1, end_dim=-1)
        nn.Conv1d(in_channels=50, out_channels=1, kernel_size=1, stride=1, padding=0),
        nn.Flatten(start_dim=1, end_dim=-1),
        nn.Linear(frames_per_sample, self._config.case.latent_dim)
    )

and
    start_channels = 256
    start_dim = frames_per_sample // (2 ** 4)

    start_volume = start_dim * start_channels
    self._decoder = nn.Sequential(
            nn.Linear(self._config.case.latent_dim, start_volume),
            nn.LeakyReLU(inplace=True),
            # b, latent
            nn.Unflatten(dim=1, unflattened_size=(start_channels, start_dim)),
            # b, start_channels, start_dim
            nn.Upsample(scale_factor=2, mode='linear', align_corners=False),
            # b, start_channels, start_dim*2
            nn.Conv1d(in_channels=start_channels, out_channels=128, kernel_size=3, stride=1, padding=1),
            # b, 128, start_dim*2
            nn.LeakyReLU(inplace=True),
            # b, 128, start_dim*2
            nn.Upsample(scale_factor=2, mode='linear', align_corners=False),
            # b, 128, start_dim*4
            nn.Conv1d(in_channels=128, out_channels=64, kernel_size=7, stride=1, padding=3),
            # b, 64, start_dim*4
            nn.LeakyReLU(inplace=True),
            # b,64, start_dim*4
            nn.Upsample(scale_factor=2, mode='linear', align_corners=False),
            # b, 64, start_dim*8
            nn.Conv1d(in_channels=64, out_channels=32, kernel_size=11, stride=1, padding=5),
            # b, 32, start_dim*8
            nn.LeakyReLU(inplace=True),
            # b, 32, start_dim*8
            nn.Upsample(scale_factor=2, mode='linear', align_corners=False),
            # b, 32, start_dim*16
            nn.Conv1d(in_channels=32, out_channels=16, kernel_size=21, stride=1, padding=10),
            # b, 16, start_dim*16
            nn.LeakyReLU(inplace=True),
            # b, 16, start_dim*16
            nn.Conv1d(in_channels=16, out_channels=self._n_features, kernel_size=3, stride=1, padding=1),
    )

I am not putting the entire code/data here because this is a theoretical question, and I don't expect anyone to go and run this.

Results
The result (orange) has artifacts on the edges, relative to the input data (blue):
This is easy to see on training examples:

Worse - for unseen examples (validation), reconstruction misses on the bias

Observation
The above only starts to happen when adding more channels, which have different biases.
I am normalizing the entire dataset to sit between -1 and 1, but still each channel has its own typical boundary.
Here is a (nice) result, for a single channel:

What I think
My guess - Multiple channels force filters to have a single bias, which doesn't fit all of them.
The edges problems are due to bias + zero padding, and the validation data is due to bias that doesn't agree with all channels.

Questions:

Does my analysis make sense?
What is a possible way to solve this?

My thoughts:

A distinct bias per channel on at least the last layer. How to do this in Pytorch?
Normalizing per-sample (and not per channel) just before passing it to the model, then de normalizing the reconstructed sample.

I don't know how to correctly implement either of those, nor if they make sense, or how to check.


title: Anaconda installed cx_Freeze can't include libpython3.7m.dylib

body:
My environment.
I'm using mac os high sierra.
and Installed anaconda for using python.
I create virtual environment and using pip for install all of package.
I'm using like this.

Using each of pip in virtual environment.
I think it's good way to build clean environment.
Now I'm working with that situation.
My virtual environment python version is 3.7.3
I created first my gui app with python. gui made with eel.
This is my first app.
Problem
I installed cx_Freeze with pip for build app file for my first mac desktop app.
and Write setup.py and run python setup.py bdist_mac
It was almost good file was created but It didn't work. Run and stop soon.
and out put was
output
running bdist_mac
running build
running build_exe
*** WARNING *** default bin include not found: libpython3.7m.dylib
creating directory build/exe.macosx-10.7-x86_64-3.7
copying /Users/test/anaconda3/envs/youtube/lib/python3.7/site-packages/cx_Freeze/bases/Console -> build/exe.macosx-10.7-x86_64-3.7/app
writing zip file build/exe.macosx-10.7-x86_64-3.7/lib/library.zip
...

more output is here
cx_Freeze couldn't find libpython3.7m.dylib
I'm having libpython here /Users/test/anaconda3/envs/youtube/lib/libpython3.7m.dylib
I'm thinking this is the reason can't run the app file.
So I want to know how to include or tell to cx_Freeze.
I try to solve this problem 6days and seaching a lot but I can't solve this problem.
Please help me I need your help.
I want to end app to build my first desktop app.


title: How to create a tensor by accessing specific values at given indices of a 2 X 2 tensor in pytorch?

body:
Suppose
mat = torch.rand((5,7)) and I want to get values from 1st dimension (here, 7) by passing the indices, say idxs=[0,4,2,3,6]. The way I am able to do it now is by doing mat[[0,1,2,3,4],idxs]. I expected mat[:,idxs] to work, but it didn't. Is the first option the only way or is there a better way?


title: Why need to normalization after made grid for grid sampling?

body:
I have one question about grid sampling process.
When trying to understand grid sampling, made toy grid sampling code.
It just read image and made warped image through pure grid sampling without any operation(=just identical image reproduction)
But if i omit to normalization(x_normalized, y_normalized), then i couldn't get identical image using grid sample.
At this point, Why normalization process essential for grid sample?
Its just made the height of image from -1 ~ 1.
How to understand this normalization ?
Thank you !
import cv2

img = cv2.imread('apple.jpg')
print("input image shape : ", img.shape)
height, width, channel = img.shape
batch = 1

img = torch.FloatTensor(img).unsqueeze(0).permute(0,3,1,2)

y_grid, x_grid = torch.meshgrid([torch.arange(0, height, dtype=torch.float32),
                    torch.arange(0, width, dtype=torch.float32)])

xy = torch.stack((x_grid, y_grid),2)
xy = torch.unsqueeze(xy, 0) #[B,H,W,2]
xy = xy.permute(0,3,1,2) #[B,H,W,2]

x_normalized = xy[:, 0, :, :] / ((width - 1) / 2) - 1
y_normalized = xy[:, 1, :, :] / ((height - 1) / 2) - 1

grid = torch.stack((x_normalized, y_normalized), dim=1)  # [B, 9, H*W, 2]
grid = grid.permute(0,2,3,1)

warped_img = torch.nn.functional.grid_sample(img,grid,mode='bilinear',padding_mode='zeros').permute(0,2,3,1).squeeze(0)
print("warped_img shape : ", warped_img.shape)
cv2.imwrite('warped_apple.jpg',np.array(warped_img))


title: PyTorch torchvision dataset download speed is very slow

body:
I have the following code block in a colab notebook that downloads the EMNIST dataset from torchvision. Sometimes I randomly get an error saying
connectionError: HTTPConnectionPool(host='www.itl.nist.gov', port=80): Max retries exceeded with url: /iaui/vip/cs_links/EMNIST/gzip.zip (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f701d0936d0>: Failed to establish a new connection: [Errno 110] Connection timed out'))

So I have made a code block with a function that calls itself if the attempt to download fails (see bottom of post). Sometimes the download actually starts working, but it's extremely slow. See below screenshot for progress bar

Over two hours to download <1GB of data from the URL in the screenshot. Downloading the dataset directly to my machine took about 60 seconds, so it's not a problem with the server that's serving the data. It seems to be a problem with either colab's internet connection with the server or the way that PyTorch is handling the data download.I don't really know what to do to solve this issue. I tried reconnecting runtime many times but the same issue happened.
Data Download Code:
from torchvision import transforms, datasets

train_data = None
test_data = None
def load_data():
  global train_data, test_data
  try:
    train_data = datasets.EMNIST("./data", split="balanced", train=True, download=True,
                                transform=transforms.Compose([
                                                              transforms.ToTensor()
                                ]))

    test_data = datasets.EMNIST("./data", split="balanced", train=False, download=True,
                                transform=transforms.Compose([
                                                              transforms.ToTensor()
                                ]))
  except:
    load_data()

load_data()
print(train_data)
print(test_data)


title: How to define a “don't care” class in time series classification in Pytorch?

body:
I have a time series classification task in which I should output a classification of 3 classes for every time stamp t.
All data is labeled per frame.
In the data set are more than 3 classes [which are also imbalanced].
My net should see all samples sequentially, because it uses that for historical information.
Thus, I can't just eliminate all irrelevant class samples at preprocessing time.
In case of a prediction on a frame which is labeled differently than those 3 classes, I don't care about the result.

How to do this correctly in Pytorch?


title: Generate the 2 d data

body:
i want to generate the u(N,2) data to feed the NN and also want to save the rx and rt as x_space and t_time to again feed in to get the differential of u wrt to rt and rx.
rx = torch.tensor((np.arange(0,1,.01).reshape(100,1)), requires_grad=True, dtype=torch.float32)
rt = torch.tensor((np.arange(0,30,.3).reshape(100,1)), requires_grad=True, dtype = torch.float32)
def differential(u_pred,x,t):

    z = torch.ones(u_pred.shape).requires_grad_(True)
    dx = torch.autograd.grad(u_pred, x, grad_outputs=z, create_graph=True)[0]
    dt = torch.autograd.grad(u_pred, t, grad_outputs=z, create_graph=True)[0]
    dxx = torch.autograd.grad(dx, z, grad_outputs=torch.ones(dx.shape), create_graph=True)[0]
    f = dxx-dt
    f = pn.MSE(f , torch.zeros(len(f)))

    return f

Is it a better way to do this so that i can save u, t and x to feed in?


title: Why is my AlexNet to train on CIFAR10 is not working?

body:
I'm trying to implement AlexNet and train on CIFAR10. However, the loss is not diminishing. Can you please tell me any problem?
class Alexnet(nn.Module):
  def __init__(self):
    super().__init__()
    self.conv1=nn.Sequential(nn.Conv2d(3,64,3,2,1),nn.ReLU(),nn.MaxPool2d(2))
    self.conv2=nn.Sequential(nn.Conv2d(64,192,3,1,padding=2),nn.MaxPool2d(2))
    self.conv3=nn.Sequential(nn.Conv2d(192,384,3,1,padding=1),nn.ReLU())
    self.conv4=nn.Sequential(nn.Conv2d(384,256,3,1,padding=1),nn.ReLU())
    self.conv5=nn.Sequential(nn.Conv2d(256,256,3,1,padding=1),nn.ReLU(),nn.MaxPool2d(2))
    self.dropout=nn.Dropout(0.3)
    self.fc1=nn.Sequential(nn.Flatten(),nn.Linear(256*2*2,4096),nn.ReLU())
    self.fc2=nn.Sequential(nn.Linear(4096,4096),nn.ReLU())
    self.fc3=nn.Sequential(nn.Linear(4096,10),nn.ReLU())
  def forward(self,x):
    x=self.conv1(x)
    x=self.conv2(x)
    x=self.conv3(x)
    x=self.conv4(x)
    x=self.conv5(x)
    x=self.dropout(x)
    x=self.fc1(x)
    x=self.dropout(x)
    x=self.fc2(x)
    x=self.fc3(x)
    return x


title: Understanding the output of LSTM predictions

body:
It's a 15-class classification model, OUTPUT_DIM = 15. I'm trying to input a frequency vector like this one 'hi my name is' => [1,43,2,56].

When I call predictions = model(x_train[0]) I get an array of size torch.Size([100, 15]), instead of just a 1D array with 15 classes like this: torch.Size([15]). What's happening? Why is this the output? How can I fix it? Thank you in advance, more info below.

The model (from main docs) is the following:
import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim):

        super().__init__()

        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)
        self.hidden2tag = nn.Linear(hidden_dim, output_dim)

    def forward(self, text):

        embeds = self.word_embeddings(text)
        lstm_out, _ = self.lstm(embeds.view(len(text), 1, -1))
        tag_space = self.hidden2tag(lstm_out.view(len(text), -1))
        tag_scores = F.log_softmax(tag_space, dim=1)

        return tag_scores

Parameters:
INPUT_DIM = 62288
EMBEDDING_DIM = 64
HIDDEN_DIM = 128
OUTPUT_DIM = 15


title: pytorch inference lead to memory leak in cpu

body:
I inference using pytorch model, I got memory leak problem, my code as follow:

import torch
import torch.nn as nn
from memory_profiler import profile

@profile(func=None, stream=open('./mem.log', 'w'))
def run(net, x):
    with x:
        net(x)

class ConvVerify(nn.Module):

    def init(self):
        for name, m in self.named_modules():
            # print(name, m)
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out')
                if m.bias is not None:
                    nn.init.constant_(m.bias, 0)
            elif isinstance(m, nn.BatchNorm2d):
                nn.init.constant_(m.running_mean, 0)
                nn.init.constant_(m.running_var, 1)
                if m.weight is not None:
                    nn.init.constant_(m.weight, 1)
                if m.bias is not None:
                    nn.init.constant_(m.bias, 0)

    def __init__(self):
        super(ConvVerify, self).__init__()

        stride = 1
        self.model = nn.Sequential(
            nn.Conv2d(288, 64, (2, 2), stride),
            nn.BatchNorm2d(64),
            nn.ReLU(),
        )
        self.init()

    def forward(self, x):
        return self.model(x)

def begin():
    conv_verify_inst = ConvVerify()
    x = torch.autograd.Variable(torch.rand(1, 288, 7, 7))
    for i in range(3000):
        run(conv_verify_inst, x)

if __name__ == "__main__":
    begin()

When run some iterator, the memory will increate 0.1M or 0.2M. For exmple, base memory is 71.8 MiB, finally, the memory increated to 72 MiB


title: How to use nn.conv2d to do the same function as nn.conv1d?

body:
I'm working on deploying python CNNs on FPGAs, and i'm facing an issue.
I have a 1d CNN which uses torch.conv1d layer function, and it's not supported by the software I'm using :vitis-ai 1.3.
I'm figuring out if there is some way to use nn.conv2d instead to do the same job as nn.conv1d ?
for example :
Conv1d(1, 32, 3, bias = True) => Conv2d(...

Is there a way to do that ? (without loosing too much performance ?)
Thank you.


title: Get a list of tensor from masked indices

body:
I'm trying to get a list of tensors based on different group,
e.g.,
x = tensor([ 0.3018, -0.0079,  1.4995, -1.4422,  1.6007])

indices = torch.tensor([0,0,1,1,2])

res = func(x,indices)

I want my result to be
res= [[0.3018, -0.0079], [1.4995, -1.4422], [1.6007]]

I'm wondering how can I achieve this result, I checked gather and index_select,
but I can't get the result like above.
Thank you!


title: How to reshape in Python by preserving structure of matrix

body:
I am trying to make a transformation that I thought to be trivial but I am not able to find any solution to this. I have an array of dimensions (3, 3, 2), such as the following one:
array([[1,2], [3,4], [5,6]],
[[7,8], [9,0], [1,2]],
[[3,4], [5,6], [7,8]])

I would like to transform it to a (2, 3, 3) array of the following form
array([[1, 3, 5],
[7, 9, 1],
[3, 5, 7]],
[[2, 4, 6],
[8, 0, 2],
[4, 6, 8]])

such that each matrix of this array contains all the elements of the respective indices for each tuple in the first matrix. Is there any way to do this, either with NumPy or preprocessing tools of ML libraries? In a sense this operation corresponds to "decoupling" the channels of an image in two separate matrix.


title: Cant overfit Neural Network

body:
I have a simple encoder-decoder network. The encoder has several layers of conv1d with linear at the end and Relu between them, the decoder is consists of conv1d layers and Relu between them(no batch norm or dropout).
Using this model I try to overfit one example,I work with batch size=1 and always give the same input and same desired output, however no success. The loss indeed goes down until some threshold, but no matter what I do I can't get the loss lower than this bound and the output is useless. I tried more sophisticated encoder/decoder, change hyperparameters, make different preprocessing on my data, but I never can't get the loss lower than that threshold.
Just for the protocol, if I give it as input the desired output(so it will learn the id function) the network works, but that doesn't help me.
I will appreciate any help with it with any idea what might be the problem.