title: Pytorch, slicing tensor causes RuntimeError:: one of the variables needed for gradient computation has been modified by an inplace operation:

body:
I wrote a RNN with LSTM cell with Pycharm. The peculiarity of this network is that the output of the RNN is fed into a integration opeartion, computed with Runge-kutta.
The integration takes some input and propagate that in time one step ahead. In order to do so I need to slice the feature tensor X along the batch dimension, and pass this to the Runge-kutta.

class MyLSTM(torch.nn.Module):
    def __init__(self, ni, no, sampling_interval, nh=10, nlayers=1):
        super(MyLSTM, self).__init__()

        self.device = torch.device("cpu")
        self.dtype = torch.float
        self.ni = ni
        self.no = no
        self.nh = nh
        self.nlayers = nlayers

        self.lstms = torch.nn.ModuleList(
            [torch.nn.LSTMCell(self.ni, self.nh)] + [torch.nn.LSTMCell(self.nh, self.nh) for i in range(nlayers - 1)])
        self.out = torch.nn.Linear(self.nh, self.no)
        self.do = torch.nn.Dropout(p=0.2)
        self.actfn = torch.nn.Sigmoid()
        self.sampling_interval = sampling_interval
        self.scaler_states = None
        # Options

    # description of the whole block
    def forward(self, x, h0, train=False, integrate_ode=True):
        x0 = x.clone().requires_grad_(True)
        hs = x  # initiate hidden state

        if h0 is None:
            h = torch.zeros(hs.shape[0], self.nh, device=self.device)
            c = torch.zeros(hs.shape[0], self.nh, device=self.device)
        else:
            (h, c) = h0

        # LSTM cells
        for i in range(self.nlayers):
            h, c = self.lstms[i](hs, (h, c))
            if train:
                hs = self.do(h)
            else:
                hs = h

        # Output layer
        # y = self.actfn(self.out(hs))
        y = self.out(hs)

        if integrate_ode:
            p = y
            y = self.integrate(x0, p)
        return y, (h, c)

    def integrate(self, x0, p):
        # RK4 steps per interval
        M = 4
        DT = self.sampling_interval / M
        X = x0
        # X = self.scaler_features.inverse_transform(x0)

            for b in range(X.shape[0]):
                xx = X[b, :]
                for j in range(M):
                    k1 = self.ode(xx, p[b, :])
                    k2 = self.ode(xx + DT / 2 * k1, p[b, :])
                    k3 = self.ode(xx + DT / 2 * k2, p[b, :])
                    k4 = self.ode(xx + DT * k3, p[b, :])
                    xx = xx + DT / 6 * (k1 + 2 * k2 + 2 * k3 + k4)
                X_all[b, :] = xx
        return X_all

    def ode(self, x0, y):
        # Here I a dynamic model

I get this error:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor []], which is output 0 of SelectBackward, is at version 64; expected version 63 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

the problem is in the operations xx = X[b, :] and p[b,:]. I know that because I choose batch dimension of 1, then I can replace the previous two equations with xx=X and p, and this works. How can split the tensor without loosing the gradient?


title: Pytorch-some problems about dimension-input must have 3 dimensions, got 2

body:
I need to use LSTM to predict state, specificly my input is 6 dimension with 6-dimension label and output is also 6 dimension. But I come across a problem I have no way to solve.

python tell me as following:
input must have 3 dimensions, got 2

my code is as following:

class lstm(nn.Module):

def __init__(self,input_size,hidden_size,output_size,num_layers):
    super(lstm,self).__init__()
    self.layer1 = nn.LSTM(input_size,hidden_size,num_layers,batch_first=True)
    self.layer2 = nn.Linear(hidden_size,output_size)

def forward(self,x):
    x,_ = self.layer1(x)
    s,b,h = x.size()
    x = x.view(s*b,h)
    x = self.layer2(x)
    x = x.view(s,b,-1)
    return x

where input_size = 6 = output_size

I have tried the following method:
the orignal training data size is [500,6](batch_size is 50)
when I reshape it to [500,1,6], the neural network can run but with large and constant loss.

I will appreciate it if someone can tell how solve it.


title: Using fastai image augmentation reduced my accuracy

body:
I am working on a classifier using the Food-101 dataset. I am trying to react a SOTA score of >=90% using ResNet50. Initially, I was sitting at 75% accuracy. When I included label smoothing, I jumped to a sweet 81% but now when I use fastai’s get_transforms, I dropped to 79%. I also noticed that even though I’m using image augmentation, there wasn’t an increase in my data’s dimensions when I was running the epoch.
Currently, the tfms variable is my augmentation:
np.random.seed(42)

path = '/content/food-101/images/train'
file_parse = r'/([^/]+)_\d+\.(png|jpg|jpeg)$'
tfms = get_transforms(do_flip=True,flip_vert=True, max_rotate=10.0, max_zoom=1.1, max_lighting=0.2, max_warp=0.2, p_affine=0.75, p_lighting=0.75)

data = ImageList.from_folder(path).split_by_rand_pct(valid_pct=0.2).label_from_re(pat=file_parse).transform(tfms, size=224).databunch()

top_1 = partial(top_k_accuracy, k=1)
learn = cnn_learner(data, models.resnet50, metrics=[accuracy, top_1], loss_func = LabelSmoothingCrossEntropy(), callback_fns=ShowGraph)

learn.lr_find()
learn.recorder.plot(suggestion=True)

My learning rate also slightly increased from 1.02e-06 to 1.023e-06 when I applied get_transforms() to my model.
I’m running another 5 epochs. The accuracy is low due to label smoothing. This isn’t an issue:
learn.fit_one_cycle(5, max_lr=slice(1.023e-06/5, 1.023e-06/15))
learn.save('stage-2')

epoch   train_loss  valid_loss  accuracy    top_k_accuracy  time
0       6.549849    5.393574    0.009835    0.009835        18:39
1       6.520850    5.373156    0.009835    0.009835        17:57
2       6.468420    5.362789    0.009901    0.009901        17:49
3       6.461528    5.363743    0.009769    0.009769        17:51
4       6.473554    5.364657    0.009901    0.009901        18:05

Now I’m pulling out another learning rate. It jumped to 1.03e-06.
learn.unfreeze()
learn.lr_find()
learn.recorder.plot(suggestion=True)

Running another 5 epochs using the new LR and saving it. In this round, the label smoothing does not seem to have the same effect as it did on the earlier run. The model is now showing better accuracy than the earlier 5 epochs:
learn.fit_one_cycle(5, max_lr=slice(1e-03/5, 1e-03/15))
learn.save('stage-3')

epoch   train_loss  valid_loss  accuracy    top_k_accuracy  time
0       3.476312    2.645357    0.491683    0.491683        18:11
1       2.781384    2.276255    0.599670    0.599670        18:22
2       2.356208    1.974409    0.677426    0.677426        18:31
3       2.068619    1.836324    0.732409    0.732409        18:26
4       1.943876    1.789893    0.742310    0.742310        18:20

When I was checking the confusions using this block of code:
interp.plot_confusion_matrix()
interp.most_confused(min_val=5)
interp.plot_multi_top_losses()
interp.plot_confusion_matrix(figsize=(20, 20), dpi=200)

I got this output about my validation set:
15081 misclassified samples over 15150 samples in the validation set.

Finally, when I validate the model with the test set, this is what I get:
path = '/content/food-101/images'
data_test = ImageList.from_folder(path).split_by_folder(train='train', valid='test').label_from_re(file_parse).transform(size=224).databunch()

learn.load('stage-3')
learn.validate(data_test.valid_dl)

[1.602944, tensor(0.7987), tensor(0.7987)]


title: Pytorch MaskRCNN training error: too many values to unpack (expected 2)

body:
following the object detection finetuning tutorial (https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html), I want to do the same but, instead of using PennFudan dataset, I want to use another dataset. I have tried to replicate their code with my own data loader (which works fine and loads the data in the same format as in the tutorial), but on the training process, when I execute this code:

    # let's train it for 10 epochs
    num_epochs = 10

    for epoch in range(num_epochs):
        # train for one epoch, printing every 10 iterations
        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        evaluate(model, data_loader_test, device=device)

I get the following error:

ValueErrorTraceback (most recent call last)
<ipython-input-217-1b75405b5b82> in <module>
      8 for epoch in range(num_epochs):
      9     # train for one epoch, printing every 10 iterations
---> 10     train_one_epoch(model, optimizer, data_loader_train, device, epoch, print_freq=10)
     11     # update the learning rate
     12     lr_scheduler.step()

/notebooks/jlabaien/PhD/XAI/GDXRAY/engine.py in train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq)
     28         images = torch.cat([image.to(device) for image in images],dim=0).to(device) #torch.tensor(image.to(device) for image in images).to(device) #images.to(device)
     29         targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
---> 30
     31
     32         loss_dict = model(images, targets)

/notebooks/jlabaien/PhD/XAI/GDXRAY/engine.py in <dictcomp>(.0)
     28         images = torch.cat([image.to(device) for image in images],dim=0).to(device) #torch.tensor(image.to(device) for image in images).to(device) #images.to(device)
     29         targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
---> 30
     31
     32         loss_dict = model(images, targets)

ValueError: too many values to unpack (expected 2)

Does somebody know the reason for what I am getting this?


title: tf.test.is_gpu_available() is False in subprocess but True in main process

body:
I'm currently running a pytorch model which periodically calls out to a tensorflow model for benchmarking purposes. I'd like both of these models to be GPU-enabled and to run in the same script. Since tensorflow benchmarking code claims GPU memory til the end of the process, I've elected to run the benchmarking code in a multiprocessing.Process so that my pytorch model can use the full GPU's memory after the benchmarking script has run.

During this, I've stumbled across an unusual bug (?) in tensorflow's gpu utilization. It seems that tensorflow run in a subprocess doesn't want to use a GPU which has been used ~at all~ by a parent process. I can have tensorflow models and pytorch models in the same GPU and process with no problems, but when I introduce subprocesses tensorflow is ill-behaved.

I'm running
tensorflow-gpu==1.14.0
torch==1.1.0
cudatoolkit=10.0 on an NVIDIA 2080-Ti.

Below is a minimal code snipped to reproduce:

import torch
import tensorflow as tf
from multiprocessing import Process

def f():
    print(tf.test.is_gpu_available())

pa = Process(target=f, args=())
pa.start()
pa.join()

torch.ones(1).cuda()

pb = Process(target=f, args=())
pb.start()
pb.join()

>>> True
>>> False


title: How to use autograd.grad with an optimizer in pytorch?

body:
I need to keep the gradient values of my model (a convolutional neural network). The problem is when I use the autograd.grad function, the optimizer.step() doesn't seem to work (the loss function doesn't change after the optimization step). So my question is how to use the autograd.grad function with the optimizer.step()?
def train(epoch):
  optimizer.zero_grad()

  out = net(gt_data)
  y = criterion(out, gt_onehot_label)
  train_losses.append(y)
  y.backward(retain_graph=True)
  dy_dx = torch.autograd.grad(y, net.parameters())
  optimizer.step()


title: pytorch slow loading speed on dataloader

body:
I am trying this code.
https://github.com/Dannynis/xvector_pytorch/blob/master/xvector%20-%20gpu.ipynb
in this code, I changed to this:
trainloader = torch.utils.data.DataLoader(list(zip(X,y)), shuffle=True, pin_memory=True, batch_size=256, num_workers=8)

and it took very long (~23s) for one step:
loss: 1.8956436561879668 step took 23.29119372367859

I think it is related to this code, copy 256 vectors to cuda took so long?
for i, data in enumerate(trainloader, 0):
    a = time.time()
    # get the inputs; data is a list of [inputs, labels]
    inputs, labels = data
    inputs, labels = inputs.cuda(), labels.cuda()

I wonder if there is anything I can do to speed up the data_loader?


title: Why is Pytorch giving me a datatype error: Float vs Double?

body:
I am working on migrating some working Pytorch code I found online (which is a 2D image classification example using the MNIST data; apologies that I lost track of the original source and am unable to find it) to what I need, which is converting a 1D collection of values into a numerical score.  I created my own Dataset class.  When I call model(), I get an error: RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #2 'mat1' in call to _th_addmm.  My first level of confusion is that I can't find any reference to Python even having a Double datatype.  And my second is why I get the error--when I put in debug code to show the datatype of mat1 and its elements, I am told that it is a Tensor which claims to be float64.  I also wonder why it is expecting a scalar for mat1, which the documentation describes as a matrix/tensor.
The full error dump is
Traceback (most recent call last):
  File "mlalan.py", line 174, in <module>
    outputs = model(images)
  File "/usr/home/adf/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "mlalan.py", line 80, in forward
    x = activate(self.fc1(x))
  File "/usr/home/adf/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/home/adf/.local/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/usr/home/adf/.local/lib/python3.7/site-packages/torch/nn/functional.py", line 1610, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #2 'mat1' in call to _th_addmm

Some of the key code from my Dataset class is
class RandomDataset(Dataset):

    def __init__(self, csv_file, transform=None):
        self.data_frame = pd.read_csv(csv_file, dtype=float)

    def __getitem__(self, idx):
        raw = self.data_frame.values[idx]
        sample = raw[0:6], raw[6:8]
        return sample

The full source code is at http://8wheels.org/mlalan.py.


title: how to add transformation in pytorch object detection

body:
I'm new to PyTorch & going through the PyTorch object detection documentation tutorial pytorch docx.
At their collab version, I made the below changes to add some transformation techniques.

First change to the __getitem__ method of class PennFudanDataset(torch.utils.data.Dataset)

if self.transforms is not None:
   img = self.transforms(img)
   target = T.ToTensor()(target)
   return img, target

In actual documentation it is
if self.transforms is not None:
   img, target = self.transforms(img, target)

Secondly, at the get_transform(train)function.
def get_transform(train):
  if train:
    transformed = T.Compose([
           T.ToTensor(),
           T.GaussianBlur(kernel_size=5, sigma=(0.1, 2.0)),
          T.ColorJitter(brightness=[0.1, 0.2], contrast=[0.1, 0.2], saturation=[0, 0.2], hue=[0,0.5])
    ])
    return transformed

  else:
    return T.ToTensor()

**In the documentation it is-**
def get_transform(train):
    transforms = []
    transforms.append(T.ToTensor())
    if train:
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

While implementing the code, I'm getting the below error. I'm not able to get what I',m doing wrong.
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py", line 272, in __getitem__
    return self.dataset[self.indices[idx]]
  File "<ipython-input-41-94e93ff7a132>", line 72, in __getitem__
    target = T.ToTensor()(target)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py", line 104, in __call__
    return F.to_tensor(pic)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py", line 64, in to_tensor
    raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic)))
TypeError: pic should be PIL Image or ndarray. Got <class 'dict'>


title: Python: fastest way to check in which interval number is in

body:
Suppose I have split the interval [0, 1] into a series of smaller intervals [0, 0.2), [0.2, 0.4), [0.4, 0.9), [0.9, 1.0]. Now I sample a value r in [0, 1]. What is the fastest way I can check in which interval this belongs to using Python / Numpy / Pytorch?
The obvious way is this:
r = np.random.rand()
if 0 <= r < 0.2:
    pass # do something
elif 0.2 <= r < 0.4:
    pass # do something else
elif 0.4 <= r < 0.9:
    pass # do yet something else again
elif 0.9 <= r <= 1.0:
    pass # do some other thing


title: Defining a Torch Class in R package "torch"

body:
this post is related to my earlier How to define a Python Class which uses R code, but called from rTorch? .
I came across the torch package in R (https://torch.mlverse.org/docs/index.html) which allows to Define a DataSet class definition. Yet, I also need to be able to define a model class like class MyModelClass(torch.nn.Module) in Python. Is this possible in the torch package in R?
When I tried to do it with reticulate it did not work - there were conflicts like
  ImportError: /User/homes/mreichstein/miniconda3/envs/r-torch/lib/python3.6/site-packages/torch/lib/libtorch_python.so: undefined symbol: _ZTINSt6thread6_StateE

It also would not make much sense, since torch isn't wrapping Python.
But it is loosing at lot of flexibility, which rTorch has (but see my problem in the upper post).
Thanks for any help!
Markus


title: Torchtext TabularDataset() reads in Datafields incorrectly

body:
Goal: I want to create a text classifier based upon my custom Dataset, simillar (and following) This (now deleted)  Tutorial from mlexplained.
What happened
I sucessfully formatted my data, created a training, validation and test dataset, and formatted it, so that it so that it equals the "toxic tweet" dataset they are using (with a column for each tag, with 1/0 for True/not True). Most of the other parts also worked just as intended, but when it came to iterating i got an Error.
The `device` argument should be set by using `torch.device` or passing a string as an argument.

This behavior will be deprecated soon and currently defaults to cpu.
The `device` argument should be set by using `torch.device` or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
The `device` argument should be set by using `torch.device` or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
The `device` argument should be set by using `torch.device` or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
  0%|          | 0/25517 [00:01<?, ?it/s]
Traceback (most recent call last):
... (trace back messages)
AttributeError: 'Example' object has no attribute 'text'

The lines the Traceback indicated:
opt = optim.Adam(model.parameters(), lr=1e-2)
loss_func = nn.BCEWithLogitsLoss()

epochs = 2

for epoch in range(1, epochs + 1):
    running_loss = 0.0
    running_corrects = 0
    model.train() # turn on training mode
    for x, y in tqdm.tqdm(train_dl): # **THIS LINE CONTAINS THE ERROR**
        opt.zero_grad()

        preds = model(x)
        loss = loss_func(y, preds)
        loss.backward()
        opt.step()

        running_loss += loss.data[0] * x.size(0)

    epoch_loss = running_loss / len(trn)

    # calculate the validation loss for this epoch
    val_loss = 0.0
    model.eval() # turn on evaluation mode
    for x, y in valid_dl:
        preds = model(x)
        loss = loss_func(y, preds)
        val_loss += loss.data[0] * x.size(0)

    val_loss /= len(vld)
    print('Epoch: {}, Training Loss: {:.4f}, Validation Loss: {:.4f}'.format(epoch, epoch_loss, val_loss))

Attempts to solve the problems already made, and what i think was the Reson:
I know that this Problem occured to others, there are even 2 Questions to it on here, bot both had the problem of skipping either Columns or Rows in the dataset (i checked for empty lines/Cokumns, and found none). Another Solution was that the Parameters given the model had to be in the same order (with none Missing) than the parameters in the .csv File.
However, the relevant code (the loading and creating of the tst, trn and vld sets)
def createTestTrain():
    # Create a Tokenizer
    tokenize = lambda x: x.split()

    # Defining Tag and Text
    TEXT = Field(sequential=True, tokenize=tokenize, lower=True)
    LABEL = Field(sequential=False, use_vocab=False)

    # Our Datafield
    tv_datafields = [("ID", None),
                     ("text", TEXT)]

    # Loading our Additional columns we added earlier
    with open(PATH + 'columnList.pickle', 'rb') as handle:
        addColumns = pickle.load(handle)

    # Adding the extra columns, no way we are defining 1000 tags by hand
    for column in addColumns:
        tv_datafields.append((column, LABEL))

    #tv_datafields.append(("split", None))
    # Loading Train/Test Split we created
    trn = TabularDataset(
                   path=PATH+'train.csv',
                   format='csv',
                   skip_header=True,
                   fields=tv_datafields)

    vld = TabularDataset(
            path=PATH+'train.csv',
            format='csv',
            skip_header=True,
            fields=tv_datafields)

    # Creating Test Datafield
    tst_datafields = [("id", None),
              ("text", TEXT)]
    # Using TabularDataset, as we want to Analyse Text on it
    tst = TabularDataset(
               path=PATH+"test.csv", # the file path
               format='csv',
               skip_header=True,
               fields=tst_datafields)

    return(trn, vld, tst)

Has uses the same list and order, like my csv does. tv_datafields is structured exactly like the file. Furthermore, as Datafield objects are just Dicts with Datapoints, i read out the Keys of the dictionary, like the tutorial also did, via:
trn[0].dict_keys()

What Should have happened:
The behaviour of the example was like this
trn[0]
torchtext.data.example.Example at 0x10d3ed3c8
trn[0].__dict__.keys()
dict_keys(['comment_text', 'toxic', 'severe_toxic', 'threat', 'obscene', 'insult', 'identity_hate'])

My result:
trn[0].__dict__.keys()
Out[19]: dict_keys([])

trn[1].__dict__.keys()
Out[20]: dict_keys([])

trn[2].__dict__.keys()
Out[21]: dict_keys([])

trn[3].__dict__.keys()
Out[22]: dict_keys(['text'])

While trn[0] does contain nothing, it is instead spread from 3 to 15, the amount of columns that should normally be there should be way more than that.
Now i am at a loss, as to where i went wrong. The Data fits, the function obviously works, but TabularDataset() seems to read in my columns the wrong way (if at all). Did i classify
# Defining Tag and Text
TEXT = Field(sequential=True, tokenize=tokenize, lower=True)
LABEL = Field(sequential=False, use_vocab=False)

the wrong way? At least that is what my Debuggin seems to indicate.
With the meager Documentation on Torchtext i have problems finding that out, but when im looking at the definitions of Data or Fields i cant see anything wrong with it.
Thank you for your help.


title: How to allow complex inputs, and complex weights to a Pytorch model?

body:
Assume even the simplest model (taken from here)
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3, 1)
        self.conv2 = nn.Conv2d(32, 64, 3, 1)
        self.fc1 = nn.Linear(9216, 128)
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2)
        x = torch.flatten(x, 1)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)

        output = F.log_softmax(x, dim=1)
        return output

When feeding complex data to the model,
output = model(data.complex())

it gives

ret = torch.addmm(bias, input, weight.t())

RuntimeError: expected scalar type Float but found ComplexDouble

(I didn't copy the entire stack trace, nor the entire training code, for question simplicity)

doing self.complex() after the model's __init__, as I normally would do self.double(), doesn't work, with
torch.nn.modules.module.ModuleAttributeError: 'Net' object has no attribute 'complex'

How to allow model's weights to be complex?
How to allow complex input to a model?
Which built-in activation functions support this?
Is anything also supported for 1d operations?

EDIT:
In the meantime, I found
this paper. Still reading it.


title: How to calculate the mean and the std of cifar10 data

body:
Pytorch is using the following values as the mean and std for the cifar10 data:
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
I need to understand the concept behind calculating it because this data is 3 channel image and I do not understand what is summed and divided over what and so on.
Also if someone can share a code for calculating the mean and the std, would be so thankful.


title: Using BERT on cpu: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu

body:
I am trying to use my fine-tuned BERT for visualization on my local machine. The model parameters are saved in a file called trained_model.pt. When I try to load and use it, I get the following error:
import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM

# Load pre-trained model tokenizer (vocabulary)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Tokenized input
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = tokenizer.tokenize(text)

# Mask a token that we will try to predict back with `BertForMaskedLM`
masked_index = 8
tokenized_text[masked_index] = '[MASK]'
assert tokenized_text == ['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer', '[SEP]']

# Convert token to vocabulary indices
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
# Define sentence A and B indices associated to 1st and 2nd sentences (see paper)
segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]

# Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

# Load pre-trained model (weights)
model = torch.load('trained_model.pt', map_location=torch.device('cpu'))
model.eval()

# Predict hidden states features for each layer
with torch.no_grad():
    encoded_layers, _ = model(tokens_tensor, segments_tensors, output_all_encoded_layers=False)

RuntimeError                              Traceback (most recent call last)
<ipython-input-32-65859554cc9e> in <module>
      5 # Predict hidden states features for each layer
      6 with torch.no_grad():
----> 7     encoded_layers, _ = model(tokens_tensor, segments_tensors, output_all_encoded_layers=False)

/opt/anaconda3/envs/bert/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/opt/anaconda3/envs/bert/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
    151         for t in chain(self.module.parameters(), self.module.buffers()):
    152             if t.device != self.src_device_obj:
--> 153                 raise RuntimeError("module must have its parameters and buffers "
    154                                    "on device {} (device_ids[0]) but found one of "
    155                                    "them on device: {}".format(self.src_device_obj, t.device))

RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu

Can anyone explain me, how this is resolved? ..by the way I get problems with DataParallel all the time. Not sure if the error is caused by that, but this feature seems really buggy to me.


title: scikit-learn FeatureUnion with Pytorch

body:
Suppose that I have a neural network written in Pytorch that act as some kind of features extractor and I wrap it in 2 slightly different Scikit-learn transformers so that they can extract different feature. Schematically, the code looks like this:

net = SomePytorchModule()
transformer_1 = Transformer1(net)
transformer_2 = Transformet2(net)

So basically, the 2 transformers are sharing the same neural net but using it in a different way. Now, I want to put them into FeatureUnion

union = FeatureUnion([
            ('transformer_1': transformer_1),
            ('transformer_2': transformer_2)])

It works fine so far. But suppose I want to do a parallel job, that is:

union = FeatureUnion([
            ('transformer_1': transformer_1),
            ('transformer_2': transformer_2)],
        n_jobs = 2)

How can I make sure that the two processes share the same net so that no copy is required because the neural network is quite large


title: Load Validation Set into memory

body:
I'm trying to train a DenseNet model on a set of patches. My Dataset has 20k images for training and 6k for evaluation.

This is my main function :

if __name__ == "__main__":
    for epoch in range(START_EPOCH, START_EPOCH+hp.epoch):
        adjust_learning_rate(optimizer,epoch)
        train(epoch, hp.wrong_save)
        #mining(epoch)
        valid(epoch)

I adjust the learning rate at each epoch due to learning rate decay, I run my training which loads my trainloader to the GPU and then I run a validation function on my validation set to check some metrics at each epoch.

I have a Nvidia GTX1060 (6 GB) and I can load the train set without no problem, but when I load the valloader, I get:

RuntimeError: CUDA out of memory. Tried to allocate 74.12 MiB (GPU 0; 5.93 GiB total capacity; 4.73 GiB already allocated; 75.06 MiB free; 19.57 MiB cached)

I would like to know what do you think about the best approach here. Is it possible to load my evaluation set not to the GPU and run the evaluation? Is there any workaround?

There are my train and valid functions:

# Optimization, Loss Function Init
criterion = nn.BCELoss()
optimizer = optim.SGD(net.parameters(), lr=args.lr, momentum=hp.momentum, weight_decay=hp.weight_decay)

def train(epoch, wrong_save=False):
    ''' trian net using patches of slide.
        save csv file that has patch file name predicted incorrectly.

    Args:
        epoch (int): current epoch
        wrong_save (bool):  If True, save the csv file that has patch file name
                            predicted incorrectly
    '''

    print('\nEpoch: %d' % epoch)

    net.train()
    train_loss = 0
    correct = 0
    total = 0
    wrong_list = []

    for batch_idx, (inputs, targets, filename) in enumerate(trainloader):
        if USE_CUDA:
            inputs = inputs.cuda()
            targets = torch.FloatTensor(np.array(targets).astype(float)).cuda()

        optimizer.zero_grad()
        inputs, targets = Variable(inputs), Variable(targets)
        outputs = net(inputs)
        outputs = torch.squeeze(outputs)
        loss = criterion(outputs, targets)
        loss.backward()
        optimizer.step()
        train_loss += loss.item()
        total += targets.size(0)
        batch_size = targets.shape[0]

        outputs += Variable((torch.ones(batch_size) * (THRESHOLD)).cuda())
        outputs = torch.floor(outputs)
        correct += outputs.data.eq(targets.data).cpu().sum()
        filename_list = filename

        if wrong_save == True:
            for idx in range(len(filename_list)):
                if outputs.data[idx] != targets.data[idx]:
                    wrong_name = filename_list[idx]
                    wrong_list.append(wrong_name)

        progress_bar(batch_idx, len(trainloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
            % (train_loss/(batch_idx+1), 100.*correct/total, correct, total))

    if wrong_save == True:
        wrong_csv = open(cf.wrong_path+'wrong_data_epoch'+str(epoch)+'.csv','w',encoding='utf-8')
        wr = csv.writer(wrong_csv)
        for name in wrong_list:
            wr.writerow([name])
        wrong_csv.close()

CUR_TRA_ACC.append(100.*correct/total)

def valid(epoch):
    ''' valid net using patches of slide.
        Save checkpoint if AUC score is higher than saved checkpoint's.

    Args:
        epoch (int): current epoch
    '''

    global BEST_AUC
    global THRESHOLD
    global LR_CHANCE
    global CK_CHANCE
    global LR_DECAY

    net.eval()
    valid_loss = 0
    total = 0
    correct = 0

    outputs_list = np.array([])
    targets_list = np.array([])

    for batch_idx, (inputs, targets) in enumerate(valloader):
        if USE_CUDA:
            inputs = inputs.cuda()
            targets = torch.FloatTensor(np.array(targets).astype(float)).cuda()

        batch_size = targets.shape[0]
        inputs, targets = Variable(inputs, volatile=True), Variable(targets)
        outputs = net(inputs)
        total += targets.size(0)
        outputs = torch.squeeze(outputs)
        loss = criterion(outputs, targets)
        valid_loss += loss.item()

        _outputs = np.array(outputs.data.cpu()).astype(float)
        _targets = np.array(targets.data.cpu()).astype(float)
        outputs_list = np.append(outputs_list, _outputs)
        targets_list = np.append(targets_list, _targets)

        outputs += Variable((torch.ones(batch_size) * (1-THRESHOLD)).cuda())
        outputs = torch.floor(outputs)
        correct += int(outputs.eq(targets).cpu().sum())

        progress_bar(batch_idx, len(valloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
                    % (valid_loss/(batch_idx+1), 100.*correct/total, correct, total))

    print(tp, tn, fp, fn)
    correct, tp, tn, fp, fn, recall, precision, specificity, f1_score, auc, threshold = stats(outputs_list, targets_list)
    acc = correct/total
    THRESHOLD = threshold

    print(tp, tn, fp, fn)

    print('Acc: %.3f, Recall: %.3f, Prec: %.3f, Spec: %.3f, F1: %.3f, Thres: %.3f, AUC: %.3f'
        %(acc, recall, precision, specificity, f1_score, threshold, auc))
    print('%17s %12s\n%-11s %-8d    %-8d\n%-11s %-8d    %-8d'
        %('Tumor', 'Normal','pos',tp,fp,'neg',fn,tn))
    print("lr: ",args.lr * (0.5 ** (LR_DECAY)), "lr chance:",LR_CHANCE)

    # plot data
    CUR_EPOCH.append(epoch)
    CUR_VAL_ACC.append(acc)
    CUR_LOSS.append(valid_loss/(batch_idx+1))
    CUR_LR.append(args.lr * (0.5 ** (LR_DECAY)))

    # Save checkpoint.
    if auc > BEST_AUC:
        print('saving...')
        BEST_AUC = auc
        state = {
            'net': net if USE_CUDA else net,
            'acc': acc,
            'loss': valid_loss,
            'recall': recall,
            'specificity': specificity,
            'precision': precision,
            'f1_score': f1_score,
            'auc': auc,
            'epoch': epoch,
            'lr': args.lr * (0.5**(LR_DECAY)),
            'threshold': threshold
        }
        torch.save(state, './checkpoint/ckpt.t7')

And this is how I construct my dataset:

def get_dataset(train_transform, test_transform, train_max,
                val_max, subtest_max, ratio=0, mining_mode=False):
    ''' dataset function to get train, valid, subtest, test, mining dataset

    Args:
        train_transform (torchvision.transforms): train set transform for data argumentation
        test_transform (torchvision.transfroms): test set transform for data argumentation
        train_max (int): limit of trian set
        val_max (int): limit of validation set
        subtest_max (int): limit of subtest set
        ratio (int): for mining_mode, inclusion ratio of train set compared mining set
        mining_mode (bool): If true, return mining dataset
    '''
    train_dataset = camel(cf.dataset_path + 'train/', usage='train',
                            limit = train_max, transform=train_transform)

    val_dataset = camel(cf.dataset_path + 'validation/', usage='val',
                            limit = val_max, transform=test_transform)

    subtest_dataset = camel(cf.dataset_path + 'test/', usage='subtest',
                            limit = subtest_max, transform=test_transform)

    test_dataset = camel(cf.test_path, usage ='test',transform=test_transform)

    if mining_mode == True:
        mining_dataset = camel(cf.dataset_path + 'mining/', usage='mining',
                                train_ratio = ratio, transform=train_transform)
        return train_dataset, val_dataset, subtest_dataset, test_dataset, mining_dataset

    else:
        return train_dataset, val_dataset, subtest_dataset, test_dataset


title: How to use DataLoader for PyTorch on iPython Console of Spyder

body:
I check this Tutorial and can't figure out a way to actually use my DataLoader to train a ANN.
When iterating over my DataLoader a cmd prompt pops up and immediately closes itself, afterwards nothing happens. My original data are both np.arrays.

import torch
from torch.utils import data
import numpy as np

class Dataset(data.Dataset):
  'Characterizes a dataset for PyTorch'
  def __init__(self, datax, labels):
        'Initialization'
        self.labels = torch.tensor(labels)
        self.datax = torch.tensor(datax)
        self.len = len(datax)

  def __len__(self):
        'Denotes the total number of samples'
        return self.len

  def __getitem__(self, index):
        'Generates one sample of data'
        # Load data and get label
        X = self.datax[index]
        y = self.labels[index]
        return X, y

params = {'batch_size': 64,
          'shuffle': True,
          'num_workers': 1}
training_set = Dataset(datax=X, labels=labels)
training_generator = data.DataLoader(training_set, **params)

for x in training_generator:
    print(1)

I tried many times and had a glimpse at the commandprompt which says something like

OMP: Info #212: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #210: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0
OMP: Info #156: KMP_AFFINITY: 4 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 1 packages x 2 cores/pkg x 2 threads/core (2 total cores)
OMP: Info #214: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1
OMP: Info #250: KMP_AFFINITY: pid 10264 tid 2388 thread 0 bound to OS proc set 0
OMP: Info #250: KMP_AFFINITY: pid 10264 tid 3288 thread 1 bound to OS proc set 2


title: Is there a more efficient way of retrieving batches from a hdf5 dataset?

body:
I have a dataclass for Pytorch dataloading. It retrieves items from a hdf5 archive (150k samples) before I feed this into a dataloader and train a small one hidden layer autoencoder. However, when I try to train my network, nothing happens, there is no GPU utilization. I am using, 4 CPUs and 2 GPUs to start off with.

My batch size is 128 and I use 8 workers when I start training.

I have also followed Pytorchs dataparallel tutorial. Below is my code for the hdf5 dataclass.

import torch.multiprocessing as mp
mp.set_start_method('fork')

from torch.utils import data
import h5py
import time

class Features_Dataset(data.Dataset):
    def __init__(self, file_path, phase):
        self.file_path = file_path
        self.archive = None
        self.phase = phase
        with h5py.File(file_path, 'r', libver='latest', swmr=True) as f:
           self.length = len(f[(self.phase) + '_labels'])

    def _get_archive(self):
        if self.archive is None:
            self.archive = h5py.File(self.file_path, 'r', libver='latest', swmr=True)
            assert self.archive.swmr_mode
        return self.archive

    def __getitem__(self, index):
        archive = self._get_archive()
        label = archive[str(self.phase) + '_labels']
        datum = archive[str(self.phase) + '_all_arrays']
        path = archive[str(self.phase) + '_img_paths']

        return datum[index], label[index], path[index]

    def __len__(self):
        return self.length

    def close(self):
        self.archive.close()

if __name__ == '__main__':
    train_dataset = Features_Dataset(file_path= "featuresdata/train.hdf5", phase= 'train')
    trainloader = data.DataLoader(train_dataset, num_workers=8, batch_size=1)
    print(len(trainloader))
    myStart = time.time()
    for i, (data, label, path) in enumerate(trainloader):
        print(path)

This is my class for the autoencoder:

import torch
import torch.nn as nn

class AutoEncoder(nn.Module):
    def __init__(self, n_embedded):
        super(AutoEncoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(6144, n_embedded))
        self.decoder = nn.Sequential(nn.Linear(n_embedded, 6144))

    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)

        return encoded, decoded

This is how I initialize the model:

device = torch.device("cuda")
    # Initialize / load checkpoint
    model = AutoEncoder(2048)
    if torch.cuda.device_count() > 1:
        print("Let's use", torch.cuda.device_count(), "GPUs!")
    model= nn.DataParallel(model)
    model.to(device)
    criterion = nn.MSELoss()
    optimizer = torch.optim.Adam(model.parameters(),weight_decay=1e-5)

I make sure that my inputs are put into the device too.

Could the speed of retrieving the batches be the problem? In regards to the hdf5 Features Dataset class, I am attempting to lazily load the hdf5 dataset without using __init__ however, I think maybe calculating the length of the dataset may be the issue...


I am learning about RNNs, and am trying to code one up using PyTorch.
I have some trouble understanding the output dimensions

Here is some code for a simple RNN architecture

class RNN(nn.Module):
    def __init__(self, input_size, hidden_dim, n_layers):
        super(RNN, self).__init__()
        self.hidden_dim=hidden_dim
        self.rnn = nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)

    def forward(self, x, hidden):
        r_out, hidden = self.rnn(x, hidden)

        return r_out, hidden

So, what I understand is the hidden_dim is the number blocks I will have in my hidden layer, and essentially the number of features in the output and in the hidden state.

I create some dummy data, to test it

test_rnn = RNN(input_size=1, hidden_dim=4, n_layers=1)

# generate evenly spaced, test data pts
time_steps = np.linspace(0, 6, 3)
data = np.sin(time_steps)
data.resize((3, 1))

test_input = torch.Tensor(data).unsqueeze(0) # give it a batch_size of 1 as first dimension
print('Input size: ', test_input.size())

# test out rnn sizes
test_out, test_h = test_rnn(test_input, None)
print('Hidden state size: ', test_h.size())
print('Output size: ', test_out.size())

What I get is

Input size:  torch.Size([1, 3, 1])
Hidden state size:  torch.Size([1, 1, 4])
Output size:  torch.Size([1, 3, 4])

So I understand that the shape of x is determined like so
x = (batch_size, seq_length, input_size).. so 1 bath size, and input of 1 feature and 3 time steps(sequence length).

For hidden state, like so hidden = (n_layers, batch_size, hidden_dim).. so I had 1 layer, batch size 1, and 4 blocks in my hidden layer.

What I don't get is the RNN output. From the documentation, r_out = (batch_size, time_step, hidden_size).. Wasn't the output supposed to be the same as the hidden state that was output from the hidden units? That is, if I have 4 units in my hidden layer, I would expect it to output 4 numbers for the hidden state, and 4 numbers for the output. Why is the time step a dimension of the output? Because, each hidden unit, takes in some numbers, outputs a state S and output Y, and both of these are equal, yes? I tried a diagram, this is what I came up with. Help me understand what part of it I'm doing wrong.

So TL;DR
Why does output shape in a simple Elman RNN depend on the sequence length(while hidden state shape doesn't)? For in the diagram I drew, I see both of them being the same.


title: Finetuning embeddings with torchtext - nn.Embedding vs. nn.Embedding.from_pretrained

body:
I have been working with pretrained embeddings (Glove) and would like to allow these to be finetuned. I currently use embeddings like this:
word_embeddingsA = nn.Embedding(vocab_size, embedding_length)
word_embeddingsA.weight = nn.Parameter(TEXT.vocab.vectors, requires_grad=False)

Should I simply set requires_grad=True to allow the embeddings to be trained? Or should I do something like this
word_embeddingsA = nn.Embedding.from_pretrained(TEXT.vocab.vectors, freeze=False)

Are these equivalent, and do I have a way to check that the embeddings are getting trained?


title: How to convert a ByteTensor into an image tensor?

body:
I am converting images into binary files via matlab and try to load the binary matlab vectors (1d) into python and convert them into ByteTensors:
img = np.fromfile(dir_train + image_name)
img = torch.ByteTensor(img)

This works fine. The ByteTensors are of the dimension 2. Afterwards I would like to convert them back into into images, like reshaping them, because the neural network (resnet18) needs 4-dimensional tensors. What is the best way to do that?
At the moment my network needs tensors of the dimension [64, 3, 7, 7], but the ByteTensors are [8, 1914].


title: What does it mean to assert an object in python?

body:
We are getting an assertion error in this line "assert dataset". We are printing the dataset object and the value we got was this '<datasets.TextDataset object at 0x000002531F10E408>'. We are using 'python 3.7' in this code. Why are we getting assertion error on dataset object?
We are basically trying to run the code of AttnGAN (https://github.com/taoxugit/AttnGAN).
The error is happening on Line: 130 in 'code/main.py'.
Code
dataset = TextDataset(cfg.DATA_DIR, split_dir, base_size=cfg.TREE.BASE_SIZE, transform=image_transform)
print(dataset)
assert dataset
dataloader = torch.utils.data.DataLoader(dataset, batch_size=cfg.TRAIN.BATCH_SIZE, drop_last=True, shuffle=bshuffle, num_workers=int(cfg.WORKERS))
Output
Load from:  C:\Users\admin\Desktop\TextToImage\AttnGAN-master (1)\AttnGAN-master\code/../data/birds/captions.pickle
<datasets.TextDataset object at 0x000002531F10E408>
Traceback (most recent call last):
File "./code/main.py", line 131, in
assert dataset
AssertionError
PS C:\Users\admin\Desktop\TextToImage\AttnGAN-master (1)\AttnGAN-master>


Using this UNet implementation: https://github.com/kilgore92/PyTorch-UNet
With this instantiation of the UNet object:
model = UNet(n_channels=1,
             mode='3D',
             num_classes=1,
             use_pooling=True,
             )

Whenever I try running my training script with the dataLoader I get this error:
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 1, 3, 3], but got 5-dimensional input of size [1, 1, 128, 128, 128] instead

Here’s more information:
This is my training script, and it is erroring out at the output = model.forward(images).
def train():
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
    num_steps_train = len(train_loader)

    print(num_steps_train)

    for epoch in range(epochs):
        print(' - training - ')
        for i, (images, masks) in enumerate(train_loader):
            images = images.to(device)
            masks = masks.to(device)
            outputs = model.forward(images)

The size of images and masks is (1, 1, 128, 128) when I print it out. This is because in my getitem_ in the Dataset object, i’m using image = torch.reshape(image, shape=(1, 128, 128, 128)) to reshape my image to the designated size. I tried changing that to image = torch.reshape(image, shape=(128, 128, 128)) and this doesn’t work either.
This is my train_loader:
train_loader = DataLoader(dataset=Dataset(partition['orig'], partition['segment']),
                          batch_size = batch_size, shuffle = True)

partition is just a dictionary that has original and segmentation mask images in it.
I can't seem to figure this out. My shapes are aligning and i've created new axes when needed.


title: Denoising linear autoencoder learns to output a constant instead of denoising

body:
I am trying to create a denoising autoencoder for 1d cyclic signals like cos(x) etc.
The process of creating the dataset is that I pass a list of cyclic functions and for each example generated it rolls random coefficients for each function in the list so every function generated is different yet cyclic. eg - 0.856cos(x) - 1.3cos(0.1x)
Then I add noise and normalize the signal to be between [0, 1).
Next, I train my autoencoder on it but it learns to output a constant (usually 0.5). my guess is that it happens because 0.5 is the usual mean value of the normalized functions. But this is not the result im aspiring to get at all.
I am providing the code I wrote for the autoencoder, the data generator and the training loop as well as two pictures depicting the problem im having.
first example:

second example:

Linear autoencoder:
class LinAutoencoder(nn.Module):
    def __init__(self, in_channels, K, B, z_dim, out_channels):
        super(LinAutoencoder, self).__init__()

        self.in_channels = in_channels
        self.K = K # number of samples per 2pi interval
        self.B = B # how many intervals
        self.out_channels = out_channels

        encoder_layers = []
        decoder_layers = []

        encoder_layers += [
            nn.Linear(in_channels * K * B, 2*z_dim, bias=True),
            nn.ReLU(),
            nn.Linear(2*z_dim, z_dim, bias=True),
            nn.ReLU(),
            nn.Linear(z_dim, z_dim, bias=True),
            nn.ReLU()
        ]

        decoder_layers += [
            nn.Linear(z_dim, z_dim, bias=True),
            nn.ReLU(),
            nn.Linear(z_dim, 2*z_dim, bias=True),
            nn.ReLU(),
            nn.Linear(2*z_dim, out_channels * K * B, bias=True),
            nn.Tanh()
        ]

        self.encoder = nn.Sequential(*encoder_layers)
        self.decoder = nn.Sequential(*decoder_layers)

    def forward(self, x):
        batch_size = x.shape[0]
        x_flat = torch.flatten(x, start_dim=1)
        enc = self.encoder(x_flat)
        dec = self.decoder(enc)
        res = dec.view((batch_size, self.out_channels, self.K * self.B))
        return res

The data generator:
def lincomb_generate_data(batch_size, intervals, sample_length, functions, noise_type="gaussian", **kwargs)->torch.tensor:

    channels = 1
    mul_term = 2 * np.pi / sample_length

    positions = np.arange(0, sample_length * intervals)
    x_axis = positions * mul_term
    X = np.tile(x_axis, (channels, 1))
    y = X

    Y = np.repeat(y[np.newaxis, :], batch_size, axis=0)

    if noise_type == "gaussian":
        # defaults to 0, 0.4
        noise_mean = kwargs.get("noise_mean", 0)
        noise_std  = kwargs.get("noise_std", 0.4)

        noise = np.random.normal(noise_mean, noise_std, Y.shape)

    if noise_type == "uniform":
        # defaults to 0, 1
        noise_low  = kwargs.get("noise_low", 0)
        noise_high = kwargs.get("noise_high", 1)

        noise = np.random.uniform(noise_low, noise_high, Y.shape)

    coef_lo  = -2
    coef_hi  =  2
    coef_mat = np.random.uniform(coef_lo, coef_hi, (batch_size, len(functions))) # creating a matrix of coefficients
    coef_mat = np.where(np.abs(coef_mat) < 10**-1, 0, coef_mat)

    for i in range(batch_size):

        curr_res = np.zeros((channels, sample_length * intervals))
        for func_id, function in enumerate(functions):
            curr_func = functions[func_id]
            curr_coef = coef_mat[i][func_id]
            curr_res += curr_coef * curr_func(Y[i, :, :])

        Y[i, :, :] = curr_res

    clean = Y
    noisy = clean + noise

    # Normalizing
    clean -= clean.min(axis=2, keepdims=2)
    clean /= clean.max(axis=2, keepdims=2) + 1e-5 #avoiding zero division

    noisy -= noisy.min(axis=2, keepdims=2)
    noisy /= noisy.max(axis=2, keepdims=2) + 1e-5 #avoiding zero division

    clean = torch.from_numpy(clean)
    noisy = torch.from_numpy(noisy)

    return x_axis, clean, noisy

Training loop:
functions = [lambda x: np.cos(0.1*x),
             lambda x: np.cos(x),
             lambda x: np.cos(3*x)]

num_epochs = 200

lin_loss_list = []

criterion = torch.nn.MSELoss()

lin_optimizer = torch.optim.SGD(lin_model.parameters(), lr=0.01, momentum=0.9)

_, val_clean, val_noisy = util.lincomb_generate_data(batch_size, B, K, functions, noise_type="gaussian")

print("STARTED TRAINING")

for epoch in range(num_epochs):
    # generate data returns the x-axis used for plotting as well as the clean and noisy data
    _, t_clean, t_noisy = util.lincomb_generate_data(batch_size, B, K, functions, noise_type="gaussian")

    # ===================forward=====================
    lin_output = lin_model(t_noisy.float())
    lin_loss = criterion(lin_output.float(), t_clean.float())
    lin_loss_list.append(lin_loss.data)

    # ===================backward====================
    lin_optimizer.zero_grad()
    lin_loss.backward()
    lin_optimizer.step()

    val_lin_loss = F.mse_loss(lin_model(val_noisy.float()), val_clean.float())

print("DONE TRAINING")

edit: shared the parameters requested
L = 1
K = 512
B = 2

batch_size = 64
z_dim = 64

noise_mean = 0
noise_std  = 0.4


title: Space efficient way to store and read massive 3d dataset?

body:
I am trying to train a neural network on sequential data. My dataset will consist of 3.6 million training examples. Each example will be a 30 x 32 ndarray (32 features observed over 30 days).
My question is what is the most space-efficient way to write and read this data?
Essentially it will have shape (3.6m, 30, 32) and np.save() seems convenient but I can't hold this whole thing in memory so I can't really save it using np.save() (or load it back using np.load()). CSV also won't work because my data has 3 dimensions.
My plan to create the thing is to process entries in batches and append them to some file so that I can keep memory free as I go.
Eventually, I am going to use the data file as an input for a PyTorch IterableDataset so it must be something that can be loaded one line at a time (like a .txt file, but I'm hoping there is some better way to save this data that is more true to its tabular, 3-dimensional nature). Any ideas are appreciated!


title: How to get text from torchtext.data.dataset.Dataset

body:
How can get the training data as text (or list of texts) from PyTorch Dataset(<torchtext.data.dataset.Dataset>) Object ?
Or more specifically from torchtext.datasets.IMDB

For example from torchtext.datasets (IMDB)
# make splits for data
train, test = datasets.IMDB.splits(TEXT, LABEL)

Assuming I did all steps on the documantation untill here, how can I get the first text in Dataset Object train as tuple in form of ("review", classification):
print(first_review)

Output:
('This is a great movie',  1)


title: PyTorch LinearLayer+BatchNorm1d with a 3D input

body: I would like to apply a BatchNorm1d after a Linear. My input is a 3D multivariate time series of shape [batch_size, n_variables, timesteps]. The Linear performs the linear transformation on the third dimension so that the new shape is [batch_size, n_variables, LinearLayer_out_features]. My problem occurs with the BatchNorm1d, I would like to apply it on the third dimension but, for a 3D input, BatchNorm1d operation is done over the second dimension (even for a 3D tensor). Do you have any suggestion on how to do that?


title: How to use Tensorboard's TSNE correctly with pytorch-lightning?

body:
I am running the following code on MNIST
Namely, I am returning from every validation epoch
return {"val_loss": loss, "recon_batch": recon_batch, "label_batch": label_batch, "label_img": orig_batch.view(-1, 1, 28, 28)}

then using
    mat = torch.cat([o["recon_batch"] for o in outputs])
    metadata = torch.cat([o["label_batch"] for o in outputs]).cpu()
    label_img = torch.cat([o["label_img"] for o in outputs]).cpu()
    tb.add_embedding(
        mat=mat,
        metadata=metadata,
        label_img=label_img,
        global_step=self.current_epoch,
    )

and expect it to work, like in the doc.
It seems like only a single batch gets shown, and during validation I am getting logs as follows

Validating:  92%|█████████▏| 49/53 [00:01<00:00, 41.05it/s]warning:
Embedding dir exists, did you set global_step for add_embedding()?

How to get a working TSNE for recon_batch for all the epochs?

Full code for reference:
def validation_step(self, batch, batch_idx):
    if self._config.dataset == "toy":
        (orig_batch, noisy_batch), label_batch = batch
        # TODO put in the noise here and not in the dataset?
    elif self._config.dataset == "mnist":
        orig_batch, label_batch = batch
        orig_batch = orig_batch.reshape(-1, 28 * 28)
        noisy_batch = orig_batch
    else:
        raise ValueError("invalid dataset")

    noisy_batch = noisy_batch.view(noisy_batch.size(0), -1)

    recon_batch, mu, logvar = self.forward(noisy_batch)

    loss = self._loss_function(
        recon_batch,
        orig_batch, mu, logvar,
        reconstruction_function=self._recon_function
    )

    tb = self.logger.experiment
    tb.add_scalars("losses", {"val_loss": loss}, global_step=self.current_epoch)
    if batch_idx == len(self.val_dataloader()) - 2:
        orig_batch -= orig_batch.min()
        orig_batch /= orig_batch.max()
        recon_batch -= recon_batch.min()
        recon_batch /= recon_batch.max()

        orig_grid = torchvision.utils.make_grid(orig_batch.view(-1, 1, 28, 28))
        val_recon_grid = torchvision.utils.make_grid(recon_batch.view(-1, 1, 28, 28))

        tb.add_image("original_val", orig_grid, global_step=self.current_epoch)
        tb.add_image("reconstruction_val", val_recon_grid, global_step=self.current_epoch)
        # f, axarr = plt.subplots(2, 1)
        # axarr[0].imshow(orig_grid.permute(1, 2, 0).cpu())
        # axarr[1].imshow(val_recon_grid.permute(1, 2, 0).cpu())
        # plt.show()
        pass

    return {"val_loss": loss, "recon_batch": recon_batch, "label_batch": label_batch,
            "label_img": orig_batch.view(-1, 1, 28, 28)}

def validation_epoch_end(self, outputs: List[Any]) -> None:
    first_batch_dict = outputs[-1]
    self.log(name="val_epoch_end", value={"val_loss": first_batch_dict["val_loss"]})

    tb = self.logger.experiment
    # assert mat.shape[0] == label_img.shape[0], '#images should equal with #data points'
    mat = torch.cat([o["recon_batch"] for o in outputs])
    metadata = torch.cat([o["label_batch"] for o in outputs]).cpu()
    label_img = torch.cat([o["label_img"] for o in outputs]).cpu()
    tb.add_embedding(
        mat=mat,
        metadata=metadata,
        label_img=label_img,
        global_step=self.current_epoch,
    )


title: Cannot deploy a small transformers model for prediction serving using Google Cloud AI Platform due to "Model requires more memory than allowed"

body:
I have a fine tuned distilgpt2 model that I want to deploy using GCP ai-platform.
I've followed all the documentation for deploying a custom prediction routine on GCP but when creating the model I get the error:

Create Version failed. Bad model detected with error: Model requires more memory than allowed. Please try to decrease the model size and re-deploy.

Here is my setup.py file:
from setuptools import setup

setup(
    name="generator_package",
    version="0.2",
    include_package_data=True,
    scripts=["generator_class.py"],
    install_requires=['transformers==2.8.0']
)

I then create a model version using:
gcloud beta ai-platform versions create v1 --model my_model \
 --origin=gs://my_bucket/model/ \
 --python-version=3.7 \
 --runtime-version=2.3 \
 --package-uris=gs://my_bucket/packages/gpt2-0.1.tar.gz,gs://cloud-ai-pytorch/torch-1.3.1+cpu-cp37-cp37m-linux_x86_64.whl \
 --prediction-class=model_prediction.CustomModelPrediction

Following this answer: PyTorch model deployment in AI Platform, I figured out how to get pytorch installed on my custom prediction routine, but am still getting the above error. I believe it may have something to do with the transformers package, as it has torch as a dependency. Can that be causing the issue?
I have tried every suggested route and cant get this to work and I'm still getting the above error. I'm using the smallest gpt2 model and am well within memory.
Can anyone who have successfully deployed to GCP please give some insight here.
Update:
So to address the above concern of transformers also trying to install torch and that may be causing the problem I rebuilt the .whl file from source with the additional packages removed, below is the edited setup.py file and built using python setup.py bdist_wheel.
I then added this whl to the required dependencies when creating a model version in GCP and removed the transformers==2.8.0 from my own setup.py. But it is still giving the same error of model requires more memory =(
import shutil

from pathlib import Path

from setuptools import find_packages, setup

# Remove stale transformers.egg-info directory to avoid https://github.com/pypa/pip/issues/5466

stale_egg_info = Path(__file__).parent / "transformers.egg-info"

if stale_egg_info.exists():

    print(

        (

            "Warning: {} exists.\n\n"

            "If you recently updated transformers to 3.0 or later, this is expected,\n"

            "but it may prevent transformers from installing in editable mode.\n\n"

            "This directory is automatically generated by Python's packaging tools.\n"

            "I will remove it now.\n\n"

            "See https://github.com/pypa/pip/issues/5466 for details.\n"

        ).format(stale_egg_info)

    )

    shutil.rmtree(stale_egg_info)

extras = {}

extras["mecab"] = ["mecab-python3"]

extras["sklearn"] = ["scikit-learn"]

# extras["tf"] = ["tensorflow"]

# extras["tf-cpu"] = ["tensorflow-cpu"]

# extras["torch"] = ["torch"]

extras["serving"] = ["pydantic", "uvicorn", "fastapi", "starlette"]

extras["all"] = extras["serving"] + ["tensorflow", "torch"]

extras["testing"] = ["pytest", "pytest-xdist"]

extras["docs"] = ["recommonmark", "sphinx", "sphinx-markdown-tables", "sphinx-rtd-theme"]

extras["quality"] = [

    "black",

    "isort",

    "flake8",

]

extras["dev"] = extras["testing"] + extras["quality"] + ["mecab-python3", "scikit-learn", "tensorflow", "torch"]

setup(

    name="transformers",

    version="2.8.0",

    author="Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Sam Shleifer, Google AI Language Team Authors, Open AI team Authors, Facebook AI Authors, Carnegie Mellon University Authors",

    author_email="thomas@huggingface.co",

    description="State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch",

    long_description=open("README.md", "r", encoding="utf-8").read(),

    long_description_content_type="text/markdown",

    keywords="NLP deep learning transformer pytorch tensorflow BERT GPT GPT-2 google openai CMU",

    license="Apache",

    url="https://github.com/huggingface/transformers",

    package_dir={"": "src"},

    packages=find_packages("src"),

    install_requires=[

        "numpy",

        "tokenizers == 0.5.2",

        # dataclasses for Python versions that don't have it

        "dataclasses;python_version<'3.7'",

        # accessing files from S3 directly

        "boto3",

        # filesystem locks e.g. to prevent parallel downloads

        "filelock",

        # for downloading models over HTTPS

        "requests",

        # progress bars in model download and training scripts

        "tqdm >= 4.27",

        # for OpenAI GPT

        "regex != 2019.12.17",

        # for XLNet

        "sentencepiece",

        # for XLM

        "sacremoses",

    ],

    extras_require=extras,

    scripts=["transformers-cli"],

    python_requires=">=3.6.0",

    classifiers=[

        "Development Status :: 5 - Production/Stable",

        "Intended Audience :: Developers",

        "Intended Audience :: Education",

        "Intended Audience :: Science/Research",

        "License :: OSI Approved :: Apache Software License",

        "Operating System :: OS Independent",

        "Programming Language :: Python :: 3",

        "Programming Language :: Python :: 3.6",

        "Programming Language :: Python :: 3.7",

        "Topic :: Scientific/Engineering :: Artificial Intelligence",

    ],

)


title: Scaler type in torch.where?

body:
torch.where documentation states that x and y can be either a tensor or a scaler. However, it doesn't seem to support float32 scaler.
import torch

x = torch.randn(3, 2)  # x is of type torch.float32

torch.where(x>0, 0, x)  # RuntimeError: expected scalar type long long but found float
# torch.where(x>0, 0.0, x)  # RuntimeError: expected scalar type double but found float

My question is how to use float32 scaler?


title: A strange loss descent(looks like a 'cliff') appeared in training 3d Unet with weighted cross-entropy loss or dice score loss

body:
The figure of dice loss:

The figure of weighted cross-entropy loss:

The optimizer is  Adam, lr = 0.0002, beta1=0.5, beta2=0.999. Does anyone have the same problem as me? Can you tell me the solution and the reason for this?

The 3d-unet models show in following.

import torch
import torch.nn as nn
import torch.nn.functional as F

import numpy as np

class UNet3D(nn.Module):
    def name(self):
        return self.name
    def __init__(self, name, in_channels, out_channels, min_filters=2, norm_layer=nn.BatchNorm3d):
        super(UNet3D, self).__init__()
        self.name = name

        activation = nn.LeakyReLU(0.2, True)

        self.input = Conv2Block(in_channels, min_filters, activation=activation, norm_layer=norm_layer)

        self.down1 = DownBlock(min_filters, min_filters*2, activation=activation, norm_layer=norm_layer)
        self.down2 = DownBlock(min_filters*2, min_filters*4, activation=activation, norm_layer=norm_layer)
        self.down3 = DownBlock(min_filters*4, min_filters*8, activation=activation, norm_layer=norm_layer)

        self.up1 = UpBlock(min_filters*8, min_filters*4, activation=activation, norm_layer=norm_layer)
        self.up2 = UpBlock(min_filters*4, min_filters*2, activation=activation, norm_layer=norm_layer)
        self.up3 = UpBlock(min_filters*2, min_filters, activation=activation, norm_layer=norm_layer)

        if self.name == 'netS':
            self.out = OutBlock(min_filters, out_channels, is_image=False)
        elif self.name == 'netG':
            self.out = OutBlock(min_filters, out_channels, is_image=True)

    def forward(self, x):
        # print('x', np.unique(x))
        x1 = self.input(x)
        # print('input', np.unique(x1.detach().numpy()))
        x2 = self.down1(x1)
        # print('down1', np.unique(x2.detach().numpy()))
        x3 = self.down2(x2)
        # print('down2', np.unique(x3.detach().numpy()))
        x = self.down3(x3)
        # print('down3', np.unique(x.detach().numpy()))

        x = self.up1(x, x3)
        # print('up1', np.unique(x.detach().numpy()))
        x = self.up2(x, x2)
        # print('up2', np.unique(x.detach().numpy()))
        x = self.up3(x, x1)
        # print('up3', np.unique(x.detach().numpy()))

        x = self.out(x)
        # print('out', np.unique(x.detach().numpy()))

        return x

class Conv2Block(nn.Module):
    '''
    Two successive Conv3d, each Conv3d followed by norm_layer and activation.
        first one| in: in_channels, out: out_channels
        second one| in: out_channels, out: out_channels
    '''
    def __init__(self, in_channels, out_channels, activation, norm_layer=nn.BatchNorm3d, kernel_size=3):
        super(Conv2Block, self).__init__()
        self.conv1 = nn.Conv3d(in_channels, out_channels, kernel_size, padding=int(np.ceil((kernel_size-1)/2)))
        self.conv2 = nn.Conv3d(out_channels, out_channels, kernel_size, padding=int(np.ceil((kernel_size-1)/2)))

        self.norm_layer = norm_layer
        if(norm_layer != None):
            self.bn1 = norm_layer(out_channels)
            self.bn2 = norm_layer(out_channels)

        self.activation = activation

    def forward(self, x):
        if(self.norm_layer != None):
            x = self.activation(self.bn1(self.conv1(x)))
            x = self.activation(self.bn2(self.conv2(x)))
        else:
            x = self.activation(self.conv1(x))
            x = self.activation(self.conv2(x))
        return x

class DownBlock(nn.Module):
    '''
    MaxPool3d + Conv2Block
    '''
    def __init__(self, in_channels, out_channels, activation, norm_layer=nn.BatchNorm3d, kernel_size=3):
        super(DownBlock, self).__init__()
        self.conv = Conv2Block(in_channels, out_channels, activation, norm_layer=norm_layer, kernel_size=kernel_size)

    def forward(self, x):
        max_pool = nn.MaxPool3d(2)
        x = self.conv(max_pool(x))
        return x

class UpConv(nn.Module):
    '''
    interpolate + conv3d + activation
    '''
    def __init__(self, in_channels, out_channels, activation, kernel_size=3):
        super(UpConv, self).__init__()
        self.conv = nn.Conv3d(in_channels, out_channels, kernel_size, padding=int(np.ceil((kernel_size-1)/2)))
        self.activation = activation

    def forward(self, x, sz):
        x = F.interpolate(x, sz, mode='trilinear', align_corners=True)
        # x = F.interpolate(x, sz, mode='nearest')

        x = self.activation(self.conv(x))           # TODO 未作normalization
        return x

class UpBlock(nn.Module):
    '''
    UpConv + cat + Conv2Block
    '''
    def __init__(self, in_channels, out_channels, activation, norm_layer=nn.BatchNorm3d, kernel_size=3):
        super(UpBlock, self).__init__()
        self.upconv = UpConv(in_channels, out_channels, activation, kernel_size=kernel_size)
        self.conv = Conv2Block(in_channels, out_channels, activation, norm_layer=norm_layer, kernel_size=kernel_size)

    def forward(self, x, x2):
        x = self.upconv(x, (x2.shape[-3], x2.shape[-2], x2.shape[-1]))
        x = torch.cat([x, x2], dim=1)
        x = self.conv(x)
        return x

class OutBlock(nn.Module):
    def __init__(self, in_channels, out_channels, is_image):
        super(OutBlock, self).__init__()
        self.conv = nn.Conv3d(in_channels, out_channels, 1)

        self.out = nn.Tanh()

        if is_image:
            self.out = nn.Tanh()
        else:
            self.out = nn.Sigmoid()

    def forward(self, x):
        x = self.conv(x)

        x = self.out(x)
        return x

Snippets of weighted cross-entropy loss and dice score.

class SoftDiceLoss(nn.Module):
    def __init__(self):
        super(SoftDiceLoss, self).__init__()

    def forward(self, preds, labels):
        preds = F.softmax(preds, dim=1)

        num = labels.size(0)
        m1 = preds.view(num, -1)
        m2 = labels.view(num, -1)
        intersection = (m1 * m2)

        score = 2. * (intersection.sum(1) + 1) / (m1.sum(1) + m2.sum(1) + 1)  # m1.sum = cube size
        score = 1 - score.sum() / num
        return score

class WeightedCrossEntropyLoss(nn.Module):
    '''
    Negative log likelihood loss
    '''
    def __init__(self):
        super(WeightedCrossEntropyLoss, self).__init__()
        # self.loss = nn.NLLLoss()

    def forward(self, pred, gt, alpha=3.0):
        gt = torch.squeeze(gt.argmax(dim=1), dim=1)

        assert gt.dtype == torch.long
        import torch.nn.functional as F
        mtx = F.cross_entropy(pred, gt, reduction='none')

        bg = (gt == 0) + (gt == 5)    # background

        neg = mtx[bg]
        pos = mtx[1-bg]

        Np, Nn = pos.numel(), neg.numel()

        pos = pos.sum()

        k = min(Np*alpha, Nn)
        if k > 0:
            neg, _ = torch.topk(neg, int(k))
            neg = neg.sum()
        else:
            neg = 0.0

        loss = (pos + neg)/(Np + k)

        return loss


title: Issues loading from Drive with pytorch' datasets.DatasetFolder

body:
The loading works great using jupyter and local files, but when I adapted to Colab, fetching data from a Drive folder, datasets.DatasetFolder always loads 9500 odd datapoints, never the full 10 000. Anyone had similar issues?

train_data = datasets.DatasetFolder('/content/drive/My Drive/4 - kaggle/data', np.load, list(('npy'))  )

print(train_data.__len__)

Returns

<bound method DatasetFolder.__len__ of Dataset DatasetFolder
    Number of datapoints: 9554
    Root Location: /content/drive/My Drive/4 - kaggle/data
    Transforms (if any): None
    Target Transforms (if any): None>

Where I would get the full 10 000 elements usually.


title: how to prepare my own data set and transom it using pytorch or tensorflow?

body:
actually what I am trying to do that I have small data of images around 80 pics then I wanna to prepare it correctly so I am not sure about the steps that i fellow is right or not .

class MyCustomDataset(Dataset):
     def __init__(self,path,transform=None):
        self.transformations = transforms.Compose([transforms.Normalize((.5, .5, .5), (.5, .5, .5)),
                                transforms.ToTensor()])
        self.name=os.listdir(path)
        self.count=np.size(self.name)
        self.transform = transform
        self.to_tensor = transforms.ToTensor()
        #transforms.CenterCrop(100),
     def __getitem__(self, index):

         label=self.name[index]

         im3 = Image.open(path2 + '//' + label)

         imge=self.to_tensor(im3)
         if self.transform is not None:
             imge = self.transform(imge)
         return (imge, label)

     def __len__(self):
         return self.count   # of how many data(images?) you have

if __name__ == '__main__':
    # Call the dataset

path=path2
custom_dataset = MyCustomDataset(path)#.(Dataset).__init__(path)
data_loader = torch.utils.data.DataLoader(custom_dataset, batch_size=100, shuffle=True)
num_batches = len(data_loader)


title: Issue with linking torch and hdf5 lib together

body:
I'm trying to set up my project in C++ with HDF5 file usages and pyTorch (C++ API), and there are some linking problem.

So... I run Ubuntu 18.04 LTS. I had installed HDF5 with APT, then I download zip from pyTorch and put it into third_party dir in my project structure.

project/
--data/
----data.h5
--third_party/
----libtorch/
--main.cpp
--CMakeLists.txt
--run.sh

My main.cpp looks like:

#include <iostream>
#include <torch/torch.h>
#include "H5Cpp.h"

#define PRINT(x) std::cout<<x<<std::endl;

const H5std_string FILE_NAME("real/path/to/project/data/data.h5");

int main() {

    // Create a vector of inputs.
    std::vector<torch::jit::IValue> inputs;
    inputs.push_back(torch::ones({1, 3, 3, 3}));

    H5::H5File file(FILE_NAME, H5F_ACC_RDONLY);
}

Then CMakeLists:

project(cpptorch)

set(CMAKE_CXX_STANDARD 14)

# pyTorch
find_package(Torch REQUIRED)

# HDF5
find_package(HDF5 REQUIRED COMPONENTS C CXX HL REQUIRED)
include_directories( ${HDF5_INCLUDE_DIRS} )
link_directories( ${HDF5_LIBRARY_DIRS} )

add_executable(cpptorch main.cpp)

target_link_libraries(cpptorch ${HDF5_CXX_LIBRARIES})
target_link_libraries(cpptorch ${TORCH_LIBRARIES})

And run.sh

parent_path=$( cd "$(dirname "${BASH_SOURCE[0]}")" ; pwd -P )

mkdir -p ${parent_path}/build
cd ${parent_path}/build
echo ${parent_path}
cmake -DCMAKE_PREFIX_PATH=${parent_path}/third_party/libtorch ..
make
./cpptorch
cd ..

And now I'll describe the issue. For example, when in CMake I comment out all lines connected with HDF5 then torch works fine, same when I comment out Torch's lines in CMake - HDF5 runs well. But when I try to run above code I receive errors:

CMakeFiles/cpptorch.dir/main.cpp.o: In function `main':
main.cpp:(.text+0x5b): undefined reference to `H5::H5File::H5File(std::string const&, unsigned int, H5::FileCreatPropList const&, H5::FileAccPropList const&)'
collect2: error: ld returned 1 exit status
CMakeFiles/cpptorch.dir/build.make:92: recipe for target 'cpptorch' failed
make[2]: *** [cpptorch] Error 1
CMakeFiles/Makefile2:72: recipe for target 'CMakeFiles/cpptorch.dir/all' failed
make[1]: *** [CMakeFiles/cpptorch.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2
./run.sh: line 8: ./cpptorch: No such file or directory

If anybody have any idea how to solve it? I find it as linking problem, but I'm not sure.


title: how to feed output from the previous time step along with the input to next rnn cell in pytorch?

body:
for a given input we have got hidden and output state from an RNN cell. Now I want to combine the output with the next input for the next RNN cell. How can I do it?

I tried to concat input with the output from the previous step and dummy initialise the first output which is to be concatenated with the input at t=0?


title: How the function nn.LSTM behaves within the batches/ seq_len?

body:
I’m currently learning to use nn.LSTM with pytorch and had to ask how the function is working.

Basically I’m trying to feed my dataset matrix (M x N).
Since the dataset is a matrix, I wanted to feed the dataset recursively(as timesteps) into the LSTM network with Dataloader(utils.data.Dataset).

The point where i got confused was the size of input(seq_len, batch, input_size)

Let’s say I’m getting my data_loader with batch_size=10.
In order to generate the train_loader with the right form, I had to make the previous size of (M x N) into the size including the sequence_length which could simply be transformed to (M/seq_len, seq_len, N).

Then the input size of my nn.LSTM would be like:
(M/seq_len/batch_size, seq_len, N)

So, my main question comes:

If i feed this data size into the LSTM model nn.LSTM(N, hidden_size),
is the LSTM model already doing the recursive feed-forward within the whole batch?
I'm also confused with the seq_len, while seq_len>1, the output will get the dimension of seq_len.
Would that mean the output contains the recursive operations of sequences?

I’m not sure i made the questions clear, but my understanding is getting quite messed up..lol
Hope somebody could help me organizing the right understanding.


title: Why is there an error (numpy.float64 cannot be interpreted as in integer) in pytorch sample code

body:
I was trying to run the sample code found here:

https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

I get a crash in the class CocoEvaluator() constructor in coco_eval.py where the following line of code:

for iou_type in iou_types:
        self.coco_eval[iou_type] = COCOeval(coco_gt, iouType=iou_type)

will crash with the warning "object of type class 'numpy.float64' cannot be safely interpreted as an integer."

iou_type is a string 'bbox'

COCOeval is a class from pycocotools (pycocotools.cocoeval.COCOeval)

coco_gt is the return value from get_coco_api_from_dataset(data_loader.dataset)

its not clear to me where the numpy.float64 value is being used here, or what I can change to fix this


title: how to match values in two different pytorch tensors using a condition

body:
i have 4 tensors

ib0 = torch.tensor([14,17])
bb0 = torch.tensor([0,1])
ib1 = torch.tensor([17,4,2])
bb1 = torch.tensor([0,0,1])

i need to get another tensor which matches the values in ib0 and ib1 based on the values of bb0 and bb1. the result i am looking for is

torch.tensor([[14,17],[14,4],[17,2]])

because the index of 0 (from bb0) in ib0 has value 14 and the indices of 0 (from bb1) has values 17,4 in ib1 and similar for the index of 1 in bb0.
This is needed in a beam search algorithm to merge together indices from the same beam. The bb0 and bb1 tensors are beam indices while ib0, ib1 are the indices in the 2 beams 0 and 1. So the above procedure is just a way to make sure that indices from different beams do not merge.


title: Using ModuleDict, I have: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

body:
I'm trying in my __init__ function:

        self.downscale_time_conv = np.empty(8, dtype=object)
        for i in range(8):
            self.downscale_time_conv[i] = torch.nn.ModuleDict({})

But in my forward, I have:

        down_out = False
        for i in range(8):
            if not down_out:
                down_out = self.downscale_time_conv[i][side](inputs)
            else:
                down_out += self.downscale_time_conv[i][side](inputs)

and I get:

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same


title: Choice of loss function

body:
Here is a word2vec implementation:

%reset -f

import torch
from torch.autograd import Variable
import numpy as np
import torch.functional as F
import torch.nn.functional as F

corpus = [
    'this test',
    'this separate test'
]

def get_input_layer(word_idx):
    x = torch.zeros(vocabulary_size).float()
    x[word_idx] = 1.0
    return x

def tokenize_corpus(corpus):
    tokens = [x.split() for x in corpus]
    return tokens

tokenized_corpus = tokenize_corpus(corpus)

vocabulary = []
for sentence in tokenized_corpus:
    for token in sentence:
        if token not in vocabulary:
            vocabulary.append(token)

word2idx = {w: idx for (idx, w) in enumerate(vocabulary)}
idx2word = {idx: w for (idx, w) in enumerate(vocabulary)}

window_size = 2
idx_pairs = []
# for each sentence
for sentence in tokenized_corpus:
    indices = [word2idx[word] for word in sentence]
    # for each word, threated as center word
    for center_word_pos in range(len(indices)):
        # for each window position
        for w in range(-window_size, window_size + 1):
            context_word_pos = center_word_pos + w
            # make soure not jump out sentence
            if context_word_pos < 0 or context_word_pos >= len(indices) or center_word_pos == context_word_pos:
                continue
            context_word_idx = indices[context_word_pos]
            idx_pairs.append((indices[center_word_pos], context_word_idx))

idx_pairs = np.array(idx_pairs) # it will be useful to have this as numpy array

vocabulary_size = len(vocabulary)

embedding_dims = 4
W1 = Variable(torch.randn(embedding_dims, vocabulary_size).float(), requires_grad=True)
W2 = Variable(torch.randn(vocabulary_size, embedding_dims).float(), requires_grad=True)
num_epochs = 1
learning_rate = 0.001

for epo in range(num_epochs):
    loss_val = 0
    for data, target in idx_pairs:
        x = Variable(get_input_layer(data)).float()
        y_true = Variable(torch.from_numpy(np.array([target])).long())

        z1 = torch.matmul(W1, x)
        z2 = torch.matmul(W2, z1)

        log_softmax = F.log_softmax(z2, dim=0)

        loss = F.nll_loss(log_softmax.view(1,-1), y_true)
        print(float(loss))

        loss_val += loss.data.item()
        loss.backward()
        W1.data -= learning_rate * W1.grad.data
        W2.data -= learning_rate * W2.grad.data

        W1.grad.data.zero_()
        W2.grad.data.zero_()

        print(W1.shape)
        print(W2.shape)

        print(f'Loss at epo {epo}: {loss_val/len(idx_pairs)}')

This prints:

0.33185482025146484
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.041481852531433105
3.302438735961914
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.45428669452667236
2.3144636154174805
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.7435946464538574
0.33418864011764526
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.7853682264685631
1.0644199848175049
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.9184207245707512
0.4970806837081909
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.980555810034275
3.2861199378967285
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 1.3913208022713661
6.170125961303711
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 2.16258654743433

Modifying the code to use mse_loss change y_true to float :

y_true = Variable(torch.from_numpy(np.array([target])).float())

Use mse_loss :

loss = F.mse_loss(log_softmax.view(1,-1), y_true)

Incorporated updates:

for epo in range(num_epochs):
    loss_val = 0
    for data, target in idx_pairs:
        x = Variable(get_input_layer(data)).float()
        y_true = Variable(torch.from_numpy(np.array([target])).float())

        z1 = torch.matmul(W1, x)
        z2 = torch.matmul(W2, z1)

        log_softmax = F.log_softmax(z2, dim=0)

        loss = F.mse_loss(log_softmax.view(1,-1), y_true)
        print(float(loss))

        loss_val += loss.data.item()
        loss.backward()
        W1.data -= learning_rate * W1.grad.data
        W2.data -= learning_rate * W2.grad.data

        W1.grad.data.zero_()
        W2.grad.data.zero_()

        print(W1.shape)
        print(W2.shape)

        print(f'Loss at epo {epo}: {loss_val/len(idx_pairs)}')

Output is now:

41.75048828125
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 5.21881103515625
16.929386138916016
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 7.334984302520752
50.63690948486328
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 13.664597988128662
36.21110534667969
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 18.190986156463623
5.304859638214111
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 18.854093611240387
9.802173614501953
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 20.07936531305313
15.515325546264648
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 22.018781006336212
30.408292770385742
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 25.81981760263443

-c:12: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 3])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

Why does mse loss not work as well as nll loss ? Is it related to the PyTorch warning:

Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 3])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

?


title: Dealing with missing data for time series data for a custom LSTM in Pytorch

body:
I am coding up a custom LSTM in Pytorch, and would like to train my network on time series data with instances that have quite a few missing time steps. I was looking for a reasonable approach to handling this issue, and based on what I found people seem to use PackedSequence in Pytorch to deal with this issue.
Since I am coding up my own LSTM, I'm not sure if using PackedSequence works in my case, and documentation on PackedSequence seems to be quite lacking so I can't seem to figure out if it'll work. I'm thinking of emulating its functionality, and was wondering how I could do this


title: When extending this class, one of its methods now requires an extra argument. Why?

body:
When I want to do a forward pass with an initialized model = nn.Sequential object, I simply use:

out = model(X)
# OR
out = model.forward(X)

However, I have tried extending the Sequential class, and now both of these methods suddenly require a second argument. For example, note in the following method my call to self(x):

def train(self, trainloader, epochs):
        for e in range(epochs):
            for x, y in trainloader:
                x = x.view(x.shape[0], -1)
                self.optimizer.zero_grad()
                loss = self.criterion(self(x), y) # CALL OCCURS HERE
                loss.backward()
                self.optimizer.step()

This code now gives me TypeError: forward() missing 1 required positional argument: 'target'.

My Question: Since I have done nothing but extend the class, why is this?

Code for full class below:

class Network(nn.Sequential):
    def __init__(self, layers):
        super().__init__(self.init_modules(layers))
        self.criterion = nn.NLLLoss()
        self.optimizer = optim.Adam(self.parameters(), lr=0.003)

    def init_modules(self, layers):

        n_layers = len(layers)
        modules = OrderedDict()

        # Layer definitions for input and inner layers:
        for i in range(n_layers - 2):
            modules[f'fc{i}']   = nn.Linear(layers[i], layers[i+1])
            modules[f'relu{i}'] = nn.ReLU()

        # Definition for output layer:
        modules['fc_out'] = nn.Linear(layers[-2], layers[-1])
        modules['smax_out'] = nn.LogSoftmax(dim=1)

        return modules

    def train(self, trainloader, epochs):
        for e in range(epochs):
            for x, y in trainloader:
                x = x.view(x.shape[0], -1)
                self.optimizer.zero_grad()
                loss = self.criterion(self(x), y)
                loss.backward()
                self.optimizer.step()

Full stack trace:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-63-490e0b9eef22> in <module>
----> 1 model2.train(trainloader, 5, plot_loss=True)

<ipython-input-61-e173e5672f18> in train(self, trainloader, epochs, plot_loss)
     32                 x = x.view(x.shape[0], -1)
     33                 self.optimizer.zero_grad()
---> 34                 loss = self.criterion(self(x), y)
     35                 loss.backward()
     36                 self.optimizer.step()

c:\program files\python38\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

c:\program files\python38\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
     98     def forward(self, input):
     99         for module in self:
--> 100             input = module(input)
    101         return input
    102

c:\program files\python38\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

TypeError: forward() missing 1 required positional argument: 'target'


title: Which parameters of Mask-RCNN control mask recall?

body:
I'm interested in fine-tuning a Mask-RCNN model that I'm using for instance segmentation. Currently I have trained the model for 6 epochs and the various Mask-RCNN losses are as follows:

The reason I'm stopping is that the COCO evaluation metrics seem to have dipped in the last epoch:

I know this is a far reaching question, but I'm looking to gain some intuition of how to understand which parameters are going to be the most impactful in improving the evaluation metrics. I understand there are three places to consider:

Should I be looking at batch size, learning rate and momentum, this uses an SGD optimizer with a learning rate of 1e-4 and batch size 2?
Should I be looking at using more training data or adding augmentation (I don't currently use any) and my dataset is current pretty large 40K images?
Should I be looking at the specific MaskRCNN parameters?

I thing I'll likely be asked to me more specific on what I want to improve so let me say that I would like to improve the recall of the individual masks. The model is performing well but doesn't quite capture the full extend of what I would like it to. I'm also leaving out details of the specific learning problem as I'd like to gain intuition of how to approach this in general.


title: Weight decay loss

body:
I need to write a code to gradually decay the weight of my loss function by computes lambda with given steps, But I don't have any idea. Any help will be appreciated.

This is my Loss function:

loss_A = criterion(recov_A, real_A)
loss_Final = lambda_A * loss_A + #lambda_A is a fixed number: 10

I don't want the lambda_A to be fixed. I need to gradually decay the lambda after passing the specified number of steps

# write function that computes lambda given the steps
cur_lambda  = compute_lambda(step, decay_params, initial_lamdba)
Loss_Final = cur_lambda * loss_A


title: how to resolve the error while installing pytorch

body:
When i am trying to install pytorch getting an error.

All my packages are upgraded to the latest version. The error is

setup.py::build_deps::run()
Failed to run 'bash ../tools/build_pytorch_libs.sh --use-fbgemm --use-nnpack --use-mkldnn --use-qnnpack caffe2'

thank you in Advance


title: How to perform element-wise product in PyTorch?

body:
I have two torch tensors a and b. Tensor a has the shape of [batch_size, emb_size] and Tensor b has the shape of [num_of_words, emb_size]. I want to do the element-wise product on these two tensors instead of dot product.

I noticed that "*" can perform element-wise product but it doesn't fit my case.

For example, batch_size = 3, emb_size = 2, num_of_words = 5.

a = torch.rand((3,2))
b = torch.rand((5,2))

I want to get something like:

torch.cat([a[0]*b, a[1]*b, a[2]*b]).view(3, 5, 2)

but I want to do this in an efficient and elegant way.


title: Recreating char level RNN for generating text

body:
I tied to follow a book on deep learning, where there is an chapter about generating text in the style of an example. They used an char level RNN with two LSTM layers in it to generate text in the style of shakespare. But the code in the book (also online: https://github.com/DOsinga/deep_learning_cookbook/blob/master/05.1%20Generating%20Text%20in%20the%20Style%20of%20an%20Example%20Text.ipynb) is written in keras and I only use pytorch. So i tied to recreate it exactly in pytorch, with same network structure and hyperparameters.

So after recreating it and making it work without errors it trained it and it only learned to write the most common character, a space. Then i tried to overfit it on one realy simple sentence, so I had to decrease the sequence lenght to 8. This also did not work, but when decreasing the hidden size of the LSTMs to only 32 it learned it nearly perfectly.
So then I continued working on the original text and started to play arround with the hidden size, learning rate, optimizer (also tried adam) and trained it even longer. The best I could achieve were some random letters, still with a lot of spaces and somtimes something like "her", but far from readable, with still an quite high loss. I used RMSprop with lr=0.01 and a hidden size of 128 over 20000 epochs. I also tried to initialize the hidden state and cell state to zero.

The problem is, that my results are far worse than those in the book, but I did exactly the same just in pytorch. Can someone please tell me, what I should try or what I have done wrong. Any help is appreciated!
PS: Sorry for my bad english.

Here is my code with the original hyperparameters:

#hyperparameters
batch_size = 256
seq_len = 160
hidden_size = 640
layers = 2

#network structure
class RNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.lstm = nn.LSTM(len(chars),hidden_size,layers)
        self.linear = nn.Linear(hidden_size,len(chars))
        self.softmax = nn.Softmax(dim=2)
    def forward(self,x,h,c):
        x,(h,c) = self.lstm(x,(h,c))
        x = self.softmax(self.linear(x))
        return x,h,c

#create network, optimizer and criterion
rnn = RNN().cuda()
optimizer = torch.optim.RMSprop(rnn.parameters(),lr=0.01)
criterion = nn.CrossEntropyLoss()

#training loop
plt.ion()
losses = []
loss_sum = 0
for epoch in range(10000):
    #generate input and target filled with zeros
    input = numpy.zeros((seq_len,batch_size,len(chars)))
    target = numpy.zeros((seq_len,batch_size))
    for batch in range(batch_size):
        #choose random starting index in text
        start = random.randrange(len(text)-seq_len-1)
        #generate sequences for that batch filled with zeros
        input_seq = numpy.zeros((seq_len+1,len(chars)))
        target_seq = numpy.zeros((seq_len+1))
        for i,char in enumerate(text[start:start+seq_len+1]):
            #convert character to index
            idx = char_to_idx[char]
            #set value of index to one (one-hot-encoding)
            input_seq[i,idx] = 1
            #set value to index (only label)
            target_seq[i] = idx
        #insert sequences into input and target
        input[:,batch,:] = input_seq[:-1]
        target[:,batch] = target_seq[1:]
    #convert input and target from numpy array to pytorch tensor on gpu
    input = torch.from_numpy(input).float().cuda()
    target = torch.from_numpy(target).long().cuda()

    #initialize hidden state and cell state to zero
    h0 = torch.zeros(layers,batch_size,hidden_size).cuda()
    c0 = torch.zeros(layers,batch_size,hidden_size).cuda()
    #run the network on the input
    output,h,c = rnn(input,h0,c0)
    #calculate loss and perform gradient descent
    optimizer.zero_grad()
    loss = criterion(output.view(-1,len(chars)),target.view(-1))
    loss.backward()
    optimizer.step()

Plot of the loss with original hyperparameters:

Example of target and output after training:

Target:  can bring this instrument of honour
    again into his native quarter, be magnanimous in the enterprise,
    and go on; I will grace the attempt for a worthy e
Output:

Plot of the loss with hidden size of 128 over 20000 epochs (best results):