title: Pytorch custom Dataset class giving wrong output

body:
I am trying to use this class I built for a dataset but it saying that it should be a PIL or ndarray. Im not quite sure whats wrong with it.  Here is the class that I am using

class RotateDataset(Dataset):
    def __init__(self, image_list, size,transform = None):
        self.image_list = image_list
        self.size = size
        self.transform = transform
    def __len__(self):
        return len(self.image_list)
    def __getitem__(self, idx):

        img = cv2.imread(self.image_list[idx])
        image_height, image_width = img.shape[:2]
        print("ID: ", idx)
        if idx % 2 == 0:
            label = 0 # Set label
            # chose negative or positive rotation
            rotation_degree = random.randrange(35, 50, 1)
            posnegrot = np.random.randint(2)
            if posnegrot == 0:
                #positive rotation
                #rotation_matrix = cv2.getRotationMatrix2D((num_cols/2, num_rows/2), rotation_degree, 1)
                #img = cv2.warpAffine(img, rotation_matrix, (num_cols, num_rows))

                img = rotate_image(img, rotation_degree)
                img = crop_around_center(img, *largest_rotated_rect(image_width,
                                                                image_height,
                                                                math.radians(rotation_degree)))
            else:
                # Negative rotation
                rotation_degree = -rotation_degree
                img = crop_around_center(img, *largest_rotated_rect(image_width,
                                                                image_height,
                                                                math.radians(rotation_degree)))

        else:
           label = 1
        img = cv2.resize(img, self.size, cv2.INTER_AREA)
        return self.transform(img), self.transform(label)

The error that it is giving me is

TypeError: pic should be PIL Image or ndarray. Got class 'int'

It should give me a img (tensor) and a label (tensor)
but I dont think it is doing it correctly.

TypeError                                 Traceback (most recent call last)
<ipython-input-34-f47943b2600c> in <module>
      2     train_loss = 0.0
      3     net.train()
----> 4     for image, label in enumerate(train_loader):
      5         if train_on_gpu:
      6             image, label = image.cuda(), label.cuda()

~\Anaconda3\envs\TF2\lib\site-packages\torch\utils\data\dataloader.py in __next__(self)
    343
    344     def __next__(self):
--> 345         data = self._next_data()
    346         self._num_yielded += 1
    347         if self._dataset_kind == _DatasetKind.Iterable and \

~\Anaconda3\envs\TF2\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)
    383     def _next_data(self):
    384         index = self._next_index()  # may raise StopIteration
--> 385         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    386         if self._pin_memory:
    387             data = _utils.pin_memory.pin_memory(data)

~\Anaconda3\envs\TF2\lib\site-packages\torch\utils\data\_utils\fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

~\Anaconda3\envs\TF2\lib\site-packages\torch\utils\data\_utils\fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

<ipython-input-28-6c77357ff619> in __getitem__(self, idx)
     35             label = 1
     36         img = cv2.resize(img, self.size, cv2.INTER_AREA)
---> 37         return self.transform(img), self.transform(label)

~\Anaconda3\envs\TF2\lib\site-packages\torchvision\transforms\transforms.py in __call__(self, pic)
     99             Tensor: Converted image.
    100         """
--> 101         return F.to_tensor(pic)
    102
    103     def __repr__(self):

~\Anaconda3\envs\TF2\lib\site-packages\torchvision\transforms\functional.py in to_tensor(pic)
     53     """
     54     if not(_is_pil_image(pic) or _is_numpy(pic)):
---> 55         raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic)))
     56
     57     if _is_numpy(pic) and not _is_numpy_image(pic):

TypeError: pic should be PIL Image or ndarray. Got <class 'int'>


title: Tensorflow to PyTorch - model.predict equivalent

body:
I am attempting to retrieve the mean squared error of my training. within the original code based in TensorFlow, I am moving this code over to PyTorch (for research reasons).
the original TensorFlow code:

    print("Calculating threshold")
    x_opt_predictions = model.predict(x_opt)
    print("Calculating MSE on optimization set...")
    mse = np.mean(np.power(x_opt - x_opt_predictions, 2), axis=1)
    print("mean is %.5f" % mse.mean())
    print("min is %.5f" % mse.min())
    print("max is %.5f" % mse.max())
    print("std is %.5f" % mse.std())
    tr = mse.mean() + mse.std()

the training method of pytorch:
def train(net, x_train, x_opt, BATCH_SIZE, EPOCHS, input_dim):
    outputs = 0
    mse = 0
    optimizer = optim.SGD(net.parameters(), lr=0.001)
    loss_function = nn.MSELoss()
    loss = 0
    for epoch in range(EPOCHS):
        for i in tqdm(range(0, len(x_train), BATCH_SIZE)):
            batch_y = x_opt[i:i + BATCH_SIZE]

            net.zero_grad()

            outputs = net(batch_y)

            loss = loss_function(outputs, batch_y)
            loss.backward()
            optimizer.step()

        print(f"Epoch: {epoch}. Loss: {loss}")
        print("opt", x_opt.size(), "output", outputs.__sizeof__())

    # VVVVVVVVVVVVVVVVVVVVVVVVVVVVVV
    return np.mean(np.power(x_opt - outputs, 2), axis=1)
    # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

as seen above the line "outputs" is not a numpy array of predictions, and acquiring this equivalent to generate the threshold

If there are any other (improved or missing) ways to acquire this value, appreciation in advance.


title: Train Faster RCNN in PyTorch

body:
How can i train Faster RCNN with a different backbone other than ResNet-50 (as implemented in this tutorial https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html) ?
Let's say that i already have a trained CNN that can classify cats and dogs how can i make this a backbone for the Faster RCNN network to detect cats and dogs.

How to construct this network ?
How do i train it?
Also iI would like to implement my own training loop rather the using the one provided by the PyTorch team ? What would a general training loop for this model look like ?


I have written a subclass of torch_fenics. In this, the input is a vector from DG space. I use this input in the weak formulation and then calculate the solution. Further, I need the gradient of the solution with respect to the given input.

I get the following error log on running the same

~/miniconda3/envs/py37/lib/python3.7/site-packages/torch/autograd/function.py in apply(self, *args)
75
76     def apply(self, *args):
---> 77         return self._forward_cls.backward(self, *args)
78
79
~/miniconda3/envs/py37/lib/python3.7/site-packages/torch_fenics/torch_fenics.py in backward(ctx, *grad_outputs)
88         # Check which gradients need to be computed
89         controls = list(map(fenics_adjoint.Control,
---> 90                             (c for g, c in zip(ctx.needs_input_grad[1:], ctx.fenics_inputs) if g)))
91
92         # Compute and accumulate gradient for each output with respect to each input
~/miniconda3/envs/py37/lib/python3.7/site-packages/pyadjoint/control.py in init(self, control)
38     def init(self, control):
39         self.control = control
---> 40         self.block_variable = control.block_variable
41
42     def data(self):
AttributeError: 'Function' object has no attribute 'block_variable'


title: Train network coded with nn.ModuleDict

body:
I was assigned to write a simple network with nn.ModuleDict. So, here it is:
third_model = torch.nn.ModuleDict({
'flatten': torch.nn.Flatten(),

'fc1': torch.nn.Linear(32 * 32 * 3, 1024),
'relu': torch.nn.ReLU(),

'fc2': torch.nn.Linear(1024, 240),
'relu': torch.nn.ReLU(),

'fc3': torch.nn.Linear(240, 10)})

Then I tried to train it (with cuda):
third_model.to(device)

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(third_model.parameters(), lr=0.001, momentum=0.9)

train(third_model, criterion, optimizer, train_dataloader, test_dataloader)

Function "train(model, criterion, optimizer, train_dataloader, test_dataloader)" trains the model and visualizes loss and accuracy of the model. It works properly.
Train:
def train(model, criterion, optimizer, train_dataloader, test_dataloader):
    train_loss_log = []
    train_acc_log = []
    val_loss_log = []
    val_acc_log = []

    for epoch in range(NUM_EPOCH):
        model.train()
        train_loss = 0.
        train_size = 0
        train_acc = 0.
        for imgs, labels in train_dataloader:
            imgs, labels = imgs.to(device), labels.to(device)

            optimizer.zero_grad()
            y_pred = model(imgs)
            loss = criterion(y_pred, labels)
            loss.backward()
            optimizer.step()

            train_loss += loss.item()
            train_size += y_pred.size(0)
            train_loss_log.append(loss.data / y_pred.size(0))

            _, pred_classes = torch.max(y_pred, 1)
            train_acc += (pred_classes == labels).sum().item()
            train_acc_log.append(np.mean((pred_classes == labels).cpu().numpy()))

        val_loss = 0.
        val_size = 0
        val_acc = 0.
        model.eval()
        with torch.no_grad():
            for imgs, labels in test_dataloader:
                imgs, labels = imgs.to(device), labels.to(device)
                pred = model(imgs)
                loss = criterion(pred, labels)
                val_loss += loss.item()
                val_size += pred.size(0)

                _, pred_classes = torch.max(pred, 1)
                val_acc += (pred_classes == labels).sum().item()
        val_loss_log.append(val_loss / val_size)
        val_acc_log.append(val_acc / val_size)

        clear_output()
        plot_history(train_loss_log, val_loss_log, 'loss')
        plot_history(train_acc_log, val_acc_log, 'accuracy')

        print('Train loss:', train_loss / train_size)
        print('Train acc:', train_acc / train_size)
        print('Val loss:', val_loss / val_size)
        print('Val acc:', val_acc / val_size)

I've already trained models coded with nn.Sequential and everything is okay. However, with nn.ModuleDict I get an error:
TypeError                                 Traceback (most recent call last)
<ipython-input-144-8b33ad3aad2c> in <module>()
      2 optimizer = optim.SGD(third_model.parameters(), lr=0.001, momentum=0.9)
      3
----> 4 train(third_model, criterion, optimizer, train_dataloader, test_dataloader)

1 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

TypeError: forward() takes 1 positional argument but 2 were given

Tried to find any documentation on nn.ModuleDict, but it seems like there are no examples of coding networks with it.
It seems like the problem might be with linear layers, although I do not know why.
So, I hope anyone could explain where the mistake is. Would be very grateful for any possible advice.


title: How to have incrementing batch size in pytorch

body:
In pytorch, DataLoader will split a dataset into batches of set size with additional options of shuffling etc, which one can then loop over.

But if I need the batch size to increment, such as first 10 batch of size 50, next 5 batch of size 100 and so on, what's the best way of doing so?

I tried splitting the tensor then concat them:

#10x50 + 5*100
originalTensor = torch.randn(1000, 80)
split1=torch.split(originalTensor, 500, dim=0)
split2=torch.split(list(split1)[0], 100, dim=0)

Thereafter is there a way to pass the concatenated tensor into dataLoader or any other way to directly turn the concat tensor into a generator (which might lose shuffling and other functionalities)?


title: Pytorch loading saved weights but not working

body:
I have this code. I save weights after every epoch and the code saves it. But when I load the weights the loss value starts from the initial loss value which means the loading is failing somehow.

net = torchvision.models.resnet18(pretrained=True)
num_ftrs = net.fc.in_features
net.fc = nn.Linear(num_ftrs, 136)

def train():
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    criterion = L1Loss(reduction='sum')

    lr = 0.0000001
    optimizer = optim.Adam(net.parameters(), lr=lr, weight_decay=0.0005)

    net.to(device)

    state = torch.load('face2.txt')
    net.load_state_dict(state['state_dict'])
    optimizer.load_state_dict(state['optimizer'])

    for epoch in range(int(0), 200000):
        for batch, data in enumerate(trainloader, 0):
            torch.cuda.empty_cache()

            inputs, labels = data
            inputs, labels = inputs.to(device), labels.to(device)

            optimizer.zero_grad()

            outputs = net(inputs).reshape(-1, 68, 2)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()

        state = {
            'epoch': epoch,
            'state_dict': net.state_dict(),
            'optimizer': optimizer.state_dict(),
        }

        torch.save(state, 'face2.txt')

if __name__ == '__main__':
    train()

The initial loss is 50k plus and after some thousand epochs the loss becomes 50-60. Now when I re-run the code I expect it to start from a near loss value but it starts from  around 50k again.


title: Jointly training models in Tensorflow and Pytorch

body:
I have two models, model A in Tensorflow 2.0 and model B in Pytorch 1.3. Model A's output is B's input. I'd like to train the two models end-to-end.

Is it possible to do without porting one of the models to the other framework?


title: Error trying to convert simple convolutional model to CoreML

body:
I'm trying to convert a simple GAN generator (from ClusterGAN):

self.name = 'generator'
self.latent_dim = latent_dim
self.n_c = n_c
self.x_shape = x_shape
self.ishape = (128, 7, 7)
self.iels = int(np.prod(self.ishape))
self.verbose = verbose

self.model = nn.Sequential(
    # Fully connected layers
    torch.nn.Linear(self.latent_dim + self.n_c, 1024),
    nn.BatchNorm1d(1024),
    nn.LeakyReLU(0.2, inplace=True),
    torch.nn.Linear(1024, self.iels),
    nn.BatchNorm1d(self.iels),
    nn.LeakyReLU(0.2, inplace=True),

    # Reshape to 128 x (7x7)
    Reshape(self.ishape),

    # Upconvolution layers
    nn.ConvTranspose2d(128, 64, 4, stride=2, padding=1, bias=True),
    nn.BatchNorm2d(64),
    nn.LeakyReLU(0.2, inplace=True),

    nn.ConvTranspose2d(64, 1, 4, stride=2, padding=1, bias=True),
    nn.Sigmoid()
)

But onnx-coreml fails with Error while converting op of type: BatchNormalization. Error message: provided number axes 2 not supported

I thought it was the BatchNorm2d, so I tried reshaping and applying BatchNorm1d, but I get the same error. Any thoughts? I'm very surprised that I'm having problems converting such a simple model, so I'm assuming that I must be missing something obvious.

I'm targeting iOS 13 and using Opset v10 for the onnx conversion.


title: How to load GloVe embeddings from the local directory than from the URL using Torchtext.vocab.GloVe()

body:
Is there any way to load the vocab from the local file instead of URL? I have the 840B-300D file saved but when I either use name=path_to_file or url=path_to_file, is says KeyError. Ilooked up at the source code of constructor but it goes not help either. I think that there is no support to load from the directory unless copy and paste in the Torchtext'directory. I am using Kaggle and can't access the directories directly.

PATH = '/kaggle/input/glove840b300dtxt/'

When I use
glove = vocab.GloVe(cache=PATH,name='glove.840B.300d.txt')

it throws an KeyError: glove.840B.300d.txt

There is no error wit the PATh or file name as

with open(PATH+'glove.840B.300d.txt') as f:
    print('File exists')

prints the desired output.


title: PyTorch network with customized layer works fine on CPU but get cudaErrorIllegalAddress when moved to GPU

body:
I'm trying to implement my own version of the Graph-Attention-network
The customized GAT layer is as following
class GATLayer(nn.Module):
    def __init__(self, input_dim: int, output_dim: int, adj: torch.tensor):
        super().__init__()
        self.W = nn.Parameter(torch.zeros(size=(output_dim, input_dim)))
        self.a = nn.Parameter(torch.zeros(size=(2 * output_dim,)))
        self.adj = adj
        self.n_points = adj.shape[0]
        #print(f"input dim:{input_dim}")

    def forward(self, h: torch.Tensor):
        B, T, N, F = h.size()
        hh = functional.linear(h, self.W)
        output = torch.zeros_like(hh)
        for i in range(self.n_points):
          #  print(i)
            hhj = hh[:, :, self.adj[i], :]
            hhi = torch.cat([hh[:, :, i:i + 1, :]] * hhj.size(2), 2)
            hhij = torch.cat([hhi, hhj], 3)
            e = torch.mm(hhij.reshape(B * T * hhj.size(2), -1), self.a.reshape(self.a.size(0), 1)).reshape(B, T, -1)
            alpha = functional.softmax(e, dim=2)
            output[:, :, i, :] = torch.sum(hhj * torch.cat([torch.unsqueeze(alpha, 3)] * hhj.size(3), 3), dim=2)
        return output

And the whole network is defined as:
class AQIP(nn.Module):
    def __init__(self, adj: torch.tensor, seq_len: int, with_aqi: bool = True):
        super().__init__()
        self.hid_size = 128
        self.seq_len = seq_len
        self.gat_layers = [
            GATLayer(input_dim=16 + int(with_aqi), output_dim=128, adj=adj),
            GATLayer(input_dim=128, output_dim=128, adj=adj),
        ]
        self.rnns = [
            nn.LSTM(input_size=128, hidden_size=128, num_layers=4, bias=True, batch_first=True),
        ]
        self.linear = nn.Linear(in_features=128 * 4, out_features=1, bias=True)

    def forward(self, x: torch.Tensor, site_idx: int):
        h = torch.zeros(size=(4, x.size(0), 128))
        c = torch.zeros(size=(4, x.size(0), 128))
        for gat in self.gat_layers:
            x = gat(x)
        for rnn in self.rnns:
            x[:, :, site_idx, :], (h, c) = rnn(x[:, :, site_idx, :], (h, c))
        h = h.permute(1, 0, 2)
        h = h.reshape(h.size(0), -1)
        return self.linear(h).squeeze()

When I independently test the customized GAT layer with the following code, it turned out that the GAT layer worked fine even on GPU
model = GATLayer(3, 1024, torch.tensor(np.array([[1, 0, 1], [0, 0, 1], [1, 0, 1]], dtype='bool')))
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = model.to(device)
print(model(torch.randn(5, 5, 3, 3)).shape)

Which outputs torch.Size([5, 5, 3, 1024])
When I test the whole network with CPU and the following code, it worked fine as well
#device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
adj = torch.tensor(np.array([[1, 0, 0], [0, 1, 1], [1, 1, 1]], dtype="bool"))
exp = torch.randn(3, 8, 3, 17)
gpus = [0]
model = AQIP(adj, seq_len=8)
#model = model.to(device, non_blocking=True)
print(model(exp, 1))

Which outputs tensor([-0.0320, -0.0320, -0.0320], grad_fn=<SqueezeBackward0>)
But as long as I try to move the model to GPU and de-comment the device and to lines, I get the following error with trace back to some Formatter class that is irrelevant to my code:
RuntimeError: copy_if failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered

When using CUDA_LAUNCH_BLOCKING=1 to run the code, I get:
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

Which did not help me to locate error at all
I had also run official examples on the same machine and tried to move them to GPU, it turned out that they all worked fine. So I guess it's not about the CUDA and the Cudnn or the GPU driver version incompatible. But I can not locate the problem in my code as well. PLEASE HELP! I'll be much appreciated if you could save me out of this.


title: Where are CUDA toolkit runtimes of pytorch and tensorflow, will they conflict?

body:
I found that we can just install pytorch or tensorflow using pip without CUDA-toolkit runtimes from nvidia or linux package manager.
So where is the runtime they use and will they conflict with each other?
for example, I have 5 python virtual environment:
pytorch==1.6+cu101
pytorch==1.7+cu110
conda pytorch==1.7 cudatoolkit=11.0
conda pytorch==1.6 cudatoolkit=10.2
tensorflow

Will they work together?
If not, where is the conflict and which runtime is shared by all of them?
If I want to use pytorch in different virtual environment, any suggestion for me to follow?


title: where is torch.tensor.item() defined at the Github?

body:
I am torch Newbie.
when I was learning torch tutorial like below,
I was curious about where tensor.item() is defined.
import torch
a = torch.tensor([1])
print( a.item() )  # it works without problem.

In order to find what i dont know,
First, I used VScode. but I got this.
enter image description here
not like this
enter image description here
Second, I searched "def item()" at the torch Github
But, I could not find. T^T
Could you tell me where tensor.item() is defined at the Github?
Or
where Class_torch.tensor's Menber functions(Method function) are defined??


title: Pytorch Resnet CNN only works when test data contains all classes

body:
Hoping to get a hand with a weird CNN training issue.
I am training a Resnet classifier to predict 4 classes of image from a ~10k image dataset. The code is pretty simple. Here's the Resnet/CNN setup part:
####################################
########### LOAD RESNET ############
####################################
device = torch.device("cuda" if torch.cuda.is_available()
                                  else "cpu")
model = models.resnet50(pretrained=True)

#
for param in model.parameters():
    param.requires_grad = False

#
model.fc = nn.Sequential(nn.Linear(2048, 512),
                                 nn.ReLU(),
                                 #nn.Dropout(0.2),
                                 nn.Linear(512, 10),
                                 nn.LogSoftmax(dim=1))

#
criterion = nn.NLLLoss()

#
optimizer = optim.Adam(model.fc.parameters(), lr=0.003)

# move model to gpu
model.to(device)

And here is the training stage (it batches the data in 500 images and shuffles the test datasets) and some accuracy results after some epochs:
   trainloader, testloader, n_batches = make_trainloader(all_data,
                                                          vals,
                                                          batch_size=500,
                                                          randomize=True)
    ...
    for inputs, labels in trainloader:
        ...
        inputs, labels = inputs.to(device), labels.to(device)

        ...
        # PREDICT;
        outputs = model(inputs)

...

epoch #:  12
Loss: 0.1689 Acc: 0.9400
labels:       tensor([0, 0, 1, 0, 3, 0, 0, 2, 1, 2], device='cuda:0')
predictions:  tensor([0, 0, 1, 0, 3, 0, 0, 2, 1, 2], device='cuda:0')

So the weird thing is that I can't seem to predict well on single images but only on large batches of data with mixed classes. For example, if I provide 500 images from class 1, the prediction is random, but if I provide 500 images mixed from the 4 classes (much like during training), the prediction is great (just like during training).
It seems that I'm confused about how to use the ResNet classifier on single images even though it does seem to learn to predict the individual labels of the input data (see labels and prediction output above). Or that my classifier isn't learning single images, but groups of images, not sure.
Any help or direction is appreciated (I can provide more code, but didn't want to make too long of a message).  Here's the prediction code:
# Predict
randomize = False

# load data from above
inputs = test_data[:2000]
vals_inputs = test_vals[:2000]

print ("test data size: ", vals_inputs.shape)

trainloader, testloader, n_batches = make_trainloader(inputs,
                                                      vals_inputs,
                                                      batch_size=500,
                                                      randomize=randomize)
for inputs, labels in trainloader:
    # load to device
    inputs, labels = inputs.to(device), labels.to(device)

    # PREDICT;
    outputs = model(inputs)
    _, preds = torch.max(outputs, 1)

    print ("prediction: ", preds[:10])
    print ("labels: ", labels[:10])
    ...

test data size:  torch.Size([2000])
prediction:  tensor([1, 1, 2, 1, 2, 3, 2, 3, 2, 3], device='cuda:0')
labels:      tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0')
0 Loss: 3.2936 Acc: 0.1420

prediction:  tensor([1, 3, 3, 3, 3, 1, 2, 1, 1, 2], device='cuda:0')
labels:      tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], device='cuda:0')
0 Loss: 2.1462 Acc: 0.2780

prediction:  tensor([3, 3, 1, 2, 0, 1, 2, 1, 3, 2], device='cuda:0')
labels:      tensor([2, 2, 2, 2, 2, 2, 2, 2, 2, 2], device='cuda:0')
0 Loss: 2.1975 Acc: 0.2560

Versus when I simply shuffle the data the accuracy is very high:
# Predict
randomize = True
...
test data size:  torch.Size([2000])
prediction:  tensor([0, 0, 3, 2, 0, 2, 0, 3, 0, 2], device='cuda:0')
labels:      tensor([0, 0, 3, 2, 0, 2, 0, 3, 0, 2], device='cuda:0')
0 Loss: 0.1500 Acc: 0.9580

prediction:  tensor([0, 3, 3, 3, 0, 0, 3, 2, 3, 3], device='cuda:0')
labels:      tensor([0, 2, 3, 0, 0, 0, 3, 2, 0, 3], device='cuda:0')
0 Loss: 0.1714 Acc: 0.9340

prediction:  tensor([3, 3, 2, 2, 3, 1, 3, 0, 2, 2], device='cuda:0')
labels:      tensor([3, 3, 2, 2, 3, 1, 3, 0, 2, 2], device='cuda:0')
0 Loss: 0.1655 Acc: 0.9400


title: How can I swap values in PyTorch Tensor?

body:
I have the following tensor:
vector = torch.tensor([[1,5,3], [2,3,4]])

How can I swap values in the second axis?
e.g.
tensor([[1, 5, 3], [2, 3, 4]])
becomes:
tensor([[1, 3, 3], [2, 5, 4]])


title: Custom NN architecture using Pytorch

body:
I am trying to make a custom CNN architecture using Pytorch. I want to have about the same control as what I would get if I make the architecture using numpy only.

I am new to Pytorch and would like to see some code samples of CNNs implemented without the nn.module class, if possible.


title: Is normalizing each pixel value in a UIImage necessary before passing it to a CoreML model?

body:
We have a converted a pytorch model into CoreML using https://pytorch.org/tutorials/advanced/ONNXLive.html. We have resized our image to the necessary specifications before inputting that image into the CoreML model. Despite doing this, the output of the CoreML model yields different (and incorrect) results from the pytorch model.

Our machine learning model is used to identify a particular object in an image.
Before sending an image into our model, we slice it into an array of UIImage. Using a for loop, we resize each image in the array before sending each of them to the model. The model accepts a UIImage as an input and returns an integer 0 (object not detected) or 1 (object detected) the result. If the return result is 1, we highlight that particular image in the array. After the for loop ends, we merge the array elements toto form a new UIImage with part of its area highlighted (indicating which positions contain the object).

The model works fine in pytorch, but after we converted it into CoreML and use it with Swift, we encountered a problem such that the model identifies most of the images from the array with the output "1" (for example, if we slice an image into 400 smaller pieces and pass each piece into the model, 356 out of 400 results is "1".)

We suspect the reason for this failure is that we didn't normalize each pixel value of the input image before we sent it to the model.
Is it necessary to normalize a UIImage when using CoreML, and if so, how should we go about this?


title: How to train a CNN model?

body:
When trying to train the CNN model, I came across a code shown below:

def train(n_epochs, loaders, model, optimizer, criterion):

    for epoch in range(1,n_epochs):
        train_loss = 0
        valid_loss = 0

        model.train()
        for i, (data,target) in enumerate(loaders['train']):

            # zero the parameter (weight) gradients
            optimizer.zero_grad()
            # forward pass to get outputs
            output = model(data)
            # calculate the loss
            loss = criterion(output, target)
            # backward pass to calculate the parameter gradients
            loss.backward()
            # update the parameters
            optimizer.step()

Can someone please tell me why is the second for loop used?
i.e; for i, (data,target) in enumerate(loaders['train']):

And why optimizer.zero_grad() and optimizer.step() is used?


title: Freeze downstream parameters but allow gradient propagate back for training upstream parameters

body:
Suppose I have a neural network with parameters a and b.

class MyNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.a = torch.tensor(3.0, requires_grad=True)
        self.b = torch.tensor(-1.0, requires_grad=True)

    def forward(self, x):
        return self.a * x + self.b

# The "upstream" model to be trained
model1 = MyNetwork()
# The "downstream" model which is just an alias for model1
# Its parameter should be frozen but allow gradient flow through
model2 = model1

Let's apply the same network twice .

input = 2.0
x1 = model1(input)
x2 = model2(x1)
target = 1.0
loss = (target - x2) ** 2

Normally, the gradient for parameter a is
.

The first part  is the gradient for model1 and the second part  is for model2.

Now I need to treat model2's parameters as constants and train only model1. In the sense that applying only first part of gradients to a. i.e.
.

I need to update a with that gradient
.

I cannot use torch.no_grad() which will stop gradient flowing to model1.

I cannot set torch.optim.Adam's params to be only those of model1 because model1 and model2 are the same.

I can get  by registering hooks but have no idea on getting .

In case of applying the same model multiple rounds, say I want gradients for  if k is not a prime number, how to code this up in PyTorch in a general way for (maybe) large neural networks? Can TensorFlow or other frameworks do this easily?


title: CNN is overfitting to minority class

body:
I'm new to developing CNNs and currently making a binary image classifier using PyTorch. My dataset was heavily unbalanced, and I've manually augmented my a testing split and training split to be balanced. I have a class 0 (Training set has 6500 images), and class 1 (training set has 5200ish images). When I try to use skorch's fit function, I get a validation accuracy equivalent to the percent of class 1 images in the set, and my prediction function only outputs 1 for all images.

This is the tutorial I adapted my CNN from: https://colab.research.google.com/github/dnouri/skorch/blob/master/notebooks/Transfer_Learning.ipynb#scrollTo=cane7VRWW3dO

Here's my CNN: (it's adapted from a tutorial)

import torch
import torch.nn as nn
import torch.optim as optim
import torchvision.transforms as transforms
import torchvision.datasets as datasets
import torchvision.models as models
import numpy as np
from numpy import array
import pandas as pd
from skorch import NeuralNetClassifier
from skorch.helper import predefined_split
from skorch.callbacks import LRScheduler
from skorch.callbacks import Checkpoint
from skorch.callbacks import Freezer

from PIL import Image
import skorch
import os
import cv2
import glob

from sklearn.pipeline import FeatureUnion, Pipeline
from sklearn.preprocessing import StandardScaler, MinMaxScaler, Normalizer
#Define transforms (will be same w/o online transforms)
#Manually augmented earlier
train_transforms = transforms.Compose([
    #transforms.RandomResizedCrop(224),
    #transforms.RandomHorizontalFlip(),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406],
                         [0.229, 0.224, 0.225])
])

val_transforms = transforms.Compose([
    #transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406],
                         [0.229, 0.224, 0.225])
])

train_ds = datasets.ImageFolder(train_split_path, train_transforms)
val_ds = datasets.ImageFolder(test_aug_split_path, val_transforms)

checkpoint = Checkpoint(
    f_params='best_model.pt', monitor='valid_acc_best')

#Using ResNet with some layers for now
class PretrainedModel(nn.Module):

    def __init__(self, output_features):
        super().__init__()
        model = models.resnet152(pretrained=True)
        #Don't want to change pretrained weights
        for param in model.parameters():
            param.requires_grad = False

        num_features = model.fc.in_features
        fc_layers = nn.Sequential(
                nn.Linear(num_features, 4096),
                nn.ReLU(inplace=True),
                nn.Dropout(p=0.3),
                nn.Linear(4096, output_features),
                nn.ReLU(inplace=True),
                nn.Dropout(p=0.3),
            )
        model.fc = fc_layers

        self.model = model

    def forward(self, x):
        return self.model(x)

use_cuda = torch.cuda.is_available()

net = NeuralNetClassifier(
    module=PretrainedModel,
    module__output_features = 2,
    criterion=nn.CrossEntropyLoss,
    batch_size = 16,
    lr=0.0001,
    max_epochs=3,
    optimizer=optim.Adam,
    train_split=predefined_split(val_ds),
    callbacks=[checkpoint],
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
)

net.fit(train_ds, y=None)

Here are the results of the fit function:

Epoch 1: Train loss = .46 Valid_acc = .4301 Valid Loss = .6931

Epoch 2: Train loss = .6931 Valid_acc = .4301 Valid Loss = .6931

Epoch 3: Train loss = .6931 Valid_acc = .4301 Valid Loss = .6931

As it turns out for this particular dataset, exactly 43% of my validation images are Class 1 images.

y_pred = net.predict(val_ds) gives me the following:

array([1, 1, 1, ..., 1, 1, 1], dtype=int64)

I guess I have two questions:

1) Is there anything that I have done incorrectly in my initialization of my CNN that would cause this?

2) What would cause this, and is there anything I can do to correct for it?


title: I cannot install torch with pip3 but I can with pip

body:
So following the guide on pytorchs homepage I should be able to install pytorch by running:

pip3 install torch===1.3.1 torchvision===0.4.2 -f https://download.pytorch.org/whl/torch_stable.html

If I do so, I get the error message

Could not find a version that satisfies the requirement torch===1.3.1 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
No matching distribution found for torch===1.3.1

If I install using pip (and not pip3) it works, but I cannot import torch afterwards - it seems like pip3 is handling my packages.

OS: Windows 10

IDE: VScode


title: pytorch F.cross_entropy does not apply gradient to weights

body:
I'm trying to train an MLP from scratch using torch tensors and some of the built-in loss functions. I have IRIS-data downloaded and stored in tensor (100, 4) and labels (100) (integers 0-2) in data_tr and targets_tr.  I have enabled gradients on the input data data_tr.requires_grad=True

I have a 2-layer MLP initialized like this:

W1 = torch.randn([4, 64], requires_grad=True)
W2 = torch.randn([64, 3], requires_grad=True)
b1 = torch.tensor([1.0], requires_grad=True)
b2 = torch.tensor([1.0], requires_grad=True

I understand that I should train like this:

for epoch in range(num_epochs):
    W1.grad = None
    W2.grad = None
    b1.grad = None
    b2.grad = None

    f = torch.relu(data_tr @ W1 + b1) @ W2 + b2
    error = torch.nn.functional.cross_entropy(f, targets_tr)
    error.backward()

    W1 = W1 - lr * W1.grad
    W2 = W2 - lr * W2.grad
    b1 = b1 - lr * b1.grad
    b2 = b2 - lr * b2.grad

Which makes a 2 layer MLP and cross_entropy applies softmax.
The problem now is that none of the weights or biases (W1, W2, b1, b2) has any gradients after the backward pass. So I get a TypeError: unsupported operand type(s) for *: 'float' and 'NoneType' on the first attempt to updating a weight.


title: How to change some value of a tensor into zero according to another tensor's value in pytorch?

body:
I have two tensors: tensor a and tensor b. How can I change some value of tensor a according to the value of tensor b?

I know the codes following are right, but it runs pretty slow when the tensor is big. Is there any other method?

import torch
a = torch.rand(10).cuda()
b = torch.rand(10).cuda()
a[b > 0.5] = 0.


title: How to ignore loss only some of the time in PyTorch

body:
While training a neural net in PyTorch, I'd like to be able to look at the model outputs and decide whether a particular output should result in a loss or just be ignored (=zero loss).
The usual training loop looks something like this:
for (data, labels) in loader:
    data, labels = data.to(device), labels.to(device)
    optimizer.zero_grad()
    logits = model(data)
    loss = criterion(logits, labels)
    loss.backward()
    optimizer.step()

What is the best way to zero out some of the losses?
Should I make some of the labels equal to the logits before calculating criterion?
Or, can I modify the loss after calculating it before calling loss.backward()? (and how? I could multiply it by a mask of zeros and ones, or round-trip it to a numpy array and modify it, etc)
The use case here is semantic segmentation, sort of.  The labels are a coarse image, where each region (corresponding to a 32x32 patch in the input image) can be either "yes", "no", or "ambiguous".  The "ambiguous" ones should not produce a loss, whatever the network output is.  In this case I know which losses should be ignored right up front, from the labels; but I'd like to also know how to ignore based on the outputs as well as the labels.


title: Pytorch Import Error when deploying django on AWS

body:
My django app works locally. When i try to deploy it and then open it on AWS, i get the following error:

I included torch==1.5.1 in my requirements.txt
Requirements.txt:

When I comment out the parts of the application that require torch and remove it from requirements.txt, it works fine. What can I do to ensure that torch is installed like the other modules in requirements.txt? Here is the link to requirements.txt


title: Does xarray.Dataset.to_array() load the array into memory and how efficiently sample mini batches from an xarray?

body:
I am currently trying to load a big multi-dimensional array (>5 GB) into a python script. Since I use the array as training data for a machine learning model, it is important to efficiently load the data in mini batches but avoid loading the whole data set in memory once.
My idea was to use the xarray library.
I load the data set with X=xarray.open_dataset("Test_file.nc"). To the best of my knowledge, this command does not load the data set in memory - so far, so good. However, I want to convert X to an array with the command X=X.to_array().
My first question is: Does X=X.to_array() load it into memory or not?
If that is done, I wonder how to best load minibatches in memory. The shape of the array is (variable,datetime,x1_position,x2_position). I want to load minibatches per datetime, which would lead to:
ind=np.random.randint(low=0,high=n_times,size=(BATCH_SIZE))
mini_batch=X[:,ind]
The other approach would be to transpose the array before with X.transpose("datetime","variable","x1_position","x2_position") and then sample via:
ind=np.random.randint(low=0,high=n_times,size=(BATCH_SIZE))
mini_batch=X[ind,:]
My second question is:
Does transposing an xarray affect the efficiency of indexing? More specifically, does X[ind,:] take as long as X[:,ind]?


title: Pytorch math calculation ( only one element tensors can be converted to Python scalars)

body:
What does it mean by only one element tensors can be converted to Python scalars in this case and how do I suppose to debug it?
x1 = (max-min)*torch.rand(1, 21) + min
x2 = (max-min)*torch.rand(1, 21) + min
zipped_list = zip(x1, x2)
y = [math.sin(2*x1+2) * math.cos(0.5*x2)+0.5 for (x1, x2) in zipped_list]

output
ValueError: only one element tensors can be converted to Python scalars


title: Lack of gradient when creating tensor from numpy

body:
Can someone please explain to me the following behavior?
import torch
import numpy as np

z = torch.tensor(np.array([1., 1.]), requires_grad=True).float()

def pre_main(z):
  return z * 3.0

x = pre_main(z)
x.backward(torch.tensor([1., 1.]))
print(z.grad)

prints:
None

Meanwhile:
import torch
import numpy as np

z = torch.tensor([1., 1.], requires_grad=True).float()

def pre_main(z):
  return z * 3.0

x = pre_main(z)
x.backward(torch.tensor([1., 1.]))
print(z.grad)

prints:
tensor([3., 3.])

Why are my gradients being destroyed when constructing from a numpy array?  How do I fix this?


title: Run PyTorch in GPU with visual studio code

body:
I am trying to execute code with pytorch in visual studio code, the problem is that I must be able to do it from the CPU. But my idea is that for certain deep learning projects to use the gpu and others not. How can I switch from CPU to GPU
when i run
import torch
torch.cuda.is_available()

the output is
"False".
i have cuda already installed. I'm using Ubuntu 20.04.2 . It is important for me to do it in Visual Studio Code


title: How do I calculate computational complexity of automatic differentiation?

body:
I'm using autogrid implemented in Pytorch to train a Neural Network and I need to calculate the computational complexity of the whole algorithm. Where do I find a complete calculation of computational complexity of the autograd? I've searched in the Pytorch documentation but I'm not finding any answer.
Thank you


title: Vectorized implementation of field-aware factorization

body:
I would like to implement the field-aware factorization model (FFM) in a vectorized way. In FFM, a prediction is made by the following equation

where w are the embeddings that depend on the feature and the field of the other feature. For more info, see equation (4) in FFM.

To do so, I have defined the following parameter:

import torch

W = torch.nn.Parameter(torch.Tensor(n_features, n_fields, n_factors), requires_grad=True)

Now, given an input x of size (batch_size, n_features), I want to be able to compute the previous equation. Here is my current (non-vectorized) implementation:

total_inter = torch.zeros(x.shape[0])
for i in range(n_features):
    for j in range(i + 1, n_features):
        temp1 = torch.mm(
            x[:, i].unsqueeze(1),
            W[i, feature2field[j], :].unsqueeze(0))
        temp2 = torch.mm(
            x[:, j].unsqueeze(1),
            W[j, feature2field[i], :].unsqueeze(0))
        total_inter += torch.sum(temp1 * temp2, dim=1)

Unsurprisingly, this implementation is horribly slow since n_features can easily be as large as 1000! Note however that most of the entries of x are 0. All inputs are appreciated!

Edit:

If it can help in any ways, here are some implementations of this model in PyTorch:

pytorch-fm
ctr_model_zoo

Unfortunately, I cannot figure out exactly how they have done it.

Additional update:

I can now obtain the product of x and W in a more efficient way by doing:

temp = torch.einsum('ij, jkl -> ijkl', x, W)

Thus, my loop is now:

total_inter = torch.zeros(x.shape[0])
for i in range(n_features):
    for j in range(i + 1, n_features):
        temp1 = temp[:, i, feature2field[j], :]
        temp2 = temp[:, j, feature2field[i], :]
        total_inter += 0.5 * torch.sum(temp1 * temp2, dim=1)

It is however still too long since this loop goes over for about 500 000 iterations.


title: Access 3d tensor by 2d index

body:
I am trying to access a 3d tensor matrix by 2d matrix. The return should be an 2d matrix too. And the following is what i try to achieve.

dim1 = 2
dim2 = 3
dim3 = 4
source = torch.FloatTensor(dim1, dim2, dim3)
source.normal_()

check = zn > 0.0
index = torch.argmax(check, dim=0)

for i in range(dim2):
    for j in range(dim3):
        ret[i, j] = source[index[i, j], i, j]


title: Pytorch: Custom Loss involving Norm of End-to-End Jacobian

body:
Cross posting from Pytorch discussion boards

I want to train a network using a modified loss function that has both a typical classification loss (e.g. nn.CrossEntropyLoss) as well as a penalty on the Frobenius norm of the end-to-end Jacobian (i.e. if f(x) is the output of the network, \nabla_x f(x)).

I’ve implemented a model that can successfully learn using nn.CrossEntropyLoss. However, when I try adding the second loss function (by doing two backwards passes), my training loop runs, but the model never learns. Furthermore, if I calculate the end-to-end Jacobian, but don’t include it in the loss function, the model also never learns. At a high level, my code does the following:

Forward pass to get predicted classes, yhat, from inputs x
Call yhat.backward(torch.ones(appropriate shape), retain_graph=True)
Jacobian norm = x.grad.data.norm(2)
Set loss equal to classification loss + scalar coefficient * jacobian norm
Run loss.backward()

I suspect that I’m misunderstanding how backward() works when run twice, but I haven’t been able to find any good resources to clarify this.

Too much is required to produce a working example, so I’ve tried to extract the relevant code:

def train_model(model, train_dataloader, optimizer, loss_fn, device=None):

    if device is None:
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    model.train()
    train_loss = 0
    correct = 0
    for batch_idx, (batch_input, batch_target) in enumerate(train_dataloader):
        batch_input, batch_target = batch_input.to(device), batch_target.to(device)
        optimizer.zero_grad()
        batch_input.requires_grad_(True)
        model_batch_output = model(batch_input)
        loss = loss_fn(model_output=model_batch_output, model_input=batch_input, model=model, target=batch_target)
        train_loss += loss.item()  # sum up batch loss
        loss.backward()
        optimizer.step()

and

    def end_to_end_jacobian_loss(model_output, model_input):
        model_output.backward(
            torch.ones(*model_output.shape),
            retain_graph=True)
        jacobian = model_input.grad.data
        jacobian_norm = jacobian.norm(2)
        return jacobian_norm

Edit 1: I swapped my previous implementation with .backward() to autograd.grad and it apparently works! What's the difference?

    def end_to_end_jacobian_loss(model_output, model_input):
        jacobian = autograd.grad(
            outputs=model_output['penultimate_layer'],
            inputs=model_input,
            grad_outputs=torch.ones(*model_output['penultimate_layer'].shape),
            retain_graph=True,
            only_inputs=True)[0]
        jacobian_norm = jacobian.norm(2)
        return jacobian_norm


title: Creating MLP model to predict the ratings that a user will give to an unseen movie using PyTorch

body:
For my project , i’m trying to predict the ratings that a user will give to an unseen movie, based on the ratings he gave to other movies. I’m using the movielens dataset.The Main folder, which is ml-100k contains informations about 100,000 movies.
Before processing the data, the main data (ratings data) contains user ID, movie ID, user rating from 0 to 5 and timestamps(not considered for this project).I then split the data into Training set(80%) and test data(20%) using sklearn Library.
To create the recommendation systems, the model ‘Stacked-Autoencoder’ is being used. I’m using PyTorch and the code is implemented on Google Colab. The project is based on this https://towardsdatascience.com/stacked-auto-encoder-as-a-recommendation-system-for-movie-rating-prediction-33842386338
I'm new to deep Learning and I want to compare this model(Stacked_Autoencoder) to another Deep Learning model. For Instance,I want to use Multilayer Perception(MLP). This is for research purposes. This is the code below for creating Stacked-Autoencoder model and training  the model.
### Part 1 : Archirecture of the AutoEncoder

#nn.Module is a parent class
# SAE is a child class of the parent class nn.Module
class SAE(nn.Module):
# self is the object of the SAE class

# Archirecture
    def __init__(self, ):
    # self can use alll the methods of the class nn.Module
        super(SAE,self).__init__()
    # Full connected layer  n°1, input and 20 neurons-nodes of the first layer
    # one neuron can be the genre of the movie

    # Encode step
        self.fc1 = nn.Linear(nb_movies,20)
    # Full connected layer n°2
        self.fc2 = nn.Linear(20,10)

    # Decode step
    # Full connected layer n°3
        self.fc3 = nn.Linear(10,20)
    # Full connected layer n°4
        self.fc4 = nn.Linear(20,nb_movies)
    # Sigmoid activation function
        self.activation = nn.Sigmoid()

# Action : activation of the neurons
def forward(self, x) :
        x = self.activation(self.fc1(x))
        x = self.activation(self.fc2(x))
        x = self.activation(self.fc3(x))
        # dont's use the activation function
        # use the linear function only
        x = self.fc4(x)
        # x is th evector of predicted ratings
        return x

# Create the AutoEncoder object
sae=SAE()
#MSE Loss : imported from torch.nn
criterion=nn.MSELoss()
# RMSProp optimizer (update the weights) imported from torch.optim
#sea.parameters() are weights and bias adjusted during the training
optimizer=optim.RMSProp(sae.parameters(),lr=0.01, weight_decay=0.5)

### Part 2 : Training of the SAE
# number of epochs
nb_epochs = 200
# Epoch forloop
for epoch in range(1, nb_epoch+1):
        # at the beginning the loss is at zero
        s=0.
        train_loss = 0

        #Users forloop
        for id_user in range(nb_users)
            # add one dimension to make a two dimension vector.
            # create a new dimension and put it the first position .unsqueeze[0]
            input = Variable(training_set[id_user].unsqueeze[0])

            # clone the input to obtain the target
            target= input.clone()

            # target.data are all the ratings
            # ratings > 0
            if torch.sum(target.data >0) > 0
                output = sae(input)
                # don't compute the gradients regarding the target
                target.require_grad=False
                # only deal with true ratings
                output[target==0]=0

                # Loss Criterion
                loss =criterion(output,target)

                # Average the error of the movies that don't have zero ratings
                mean_corrector=nb_movies/float(torch.sum(target.data>0)+1e-10)

                # Direction of the backpropagation
                loss.backward()
                train_loss+=np.sqrt(loss.data[0]*mean_corrector)
                s+=1.

                # Intensity of the backpropagation
                optimizer.step()

    print('epoch:' +str (epoch)+'loss:' +str(train_loss/s)

)
If I want to train using the MLP model. How can I implement this class model?
Also, What other deep learning model(Beside MLP) that I can use to compare with Stacked-Autoencoder?
Thanks.


title: Distributed training of a GRU network on Spark - PyTorch implementation

body:
I have an implementation of a GRU based network in PyTorch, which I train using a 4 GB GPU present in my laptop, and obviously it takes a lot of time (4+ hrs for 1 epoch). I am looking for ideas/leads on how I can move this deep-learning model to train on a couple of spark clusters instead.
So far, I have only come across this GitHub library called SparkTorch, which unfortunately has limited documentation and the examples provided are way too trivial.
https://github.com/dmmiller612/sparktorch
To summarize, I am looking for answers to the following two questions:

Is it a good idea to train a deep learning model on spark clusters, since I read at places that the communication overhead undermines the gains in training speed
How to convert the PyTorch model (and the underlying dataset) in order to perform a distributed training across the worker nodes.

Any leads appreciated.


title: Unexpected Standard exception from MEX file (pytorch model forward)

body:
When I call mex api from Matlab, I got an unexpected standard exception.
I exported 2 pytorch DNN models to 'A.pt' and 'B.pt' files.
And I implemented c++ functions that load models from the '.pt 'files and run models (forward).
The c++ implementation works fine, I can get proper results from the models.
I built the load & run forward function into '.dll' library,
and I implemented a mex api function can call them.
When I call the mex api in Matlab environment,
the 2 models are loaded normally, and the first model runs forward properly.
However, when run forward the 2nd model, I got the following exception.

Unexpected Standard exception from MEX file
What():The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: error in LoadLibraryA

I have no clue why the c++ implementation works fine but the exception occurs when call it through mex api from Matlab.
Because the load & run forward functions are unchanged I expected the exactly same results.
It is more difficult to debug because there is no call-stack print.
Is there any way to get call-stack ?
Please give me any advice.
Thanks in advance.
-environment-------------------------------
c++ compiler : visual studio 2017 community
matlab : R2020a
libtorch : 1.6
pytorch : 1.5
python : 3.6
cuda : 10.2


title: Different memory allocation on GTX 1080 ti, Tesla k80, Tesla v100 for the same pytorch model

body:
I have tried loading a distilbert model in pytorch over 3 different GPUs (GeForce GTX 1080 ti, tesla k80, tesla v100).  According to the pytorch cuda profiler, the memory consumption is identical in all of these GPUs(534MB).  But "nvidia-smi" shows different memory consumption for each of them (GTX 1080 ti- 1181MB, tesla k80 - 898MB, tesla v100- 1714MB).
I chose v100, hoping to accommodate more processes because of it's extra memory.  Because of this, I am not able accommodate any more processes in v100 compared to k80.
Versions: Python  3.6.11, transformers==2.3.0,
torch==1.6.0
Any help would be appreciated.
Following are the memory consumption in the GPUs.
----------------GTX 1080ti---------------------
2020-10-19 02:11:04,147 - CE - INFO - torch.cuda.max_memory_allocated() : 514.33154296875
2020-10-19 02:11:04,147 - CE - INFO - torch.cuda.memory_allocated() : 514.33154296875
2020-10-19 02:11:04,147 - CE - INFO - torch.cuda.memory_reserved() : 534.0
2020-10-19 02:11:04,148 - CE - INFO - torch.cuda.max_memory_reserved() : 534.0

The output of "nvidia-smi" :
2020-10-19 02:11:04,221 - CE - INFO - | ID | Name                | Serial          | UUID                                     || GPU temp. | GPU util. | Memory util. || Memory total | Memory used | Memory free || Display mode | Display active |
2020-10-19 02:11:04,222 - CE - INFO - |  0 | GeForce GTX 1080 Ti | [Not Supported] | GPU-58d5d4d3-07a1-81b4-ba67-8d6b46e342fb ||       50C |       15% |          11% ||      11178MB |      1181MB |      9997MB || Disabled     | Disabled       |

----------------Tesla k80---------------------
2020-10-19 12:15:37,030 - CE - INFO - torch.cuda.max_memory_allocated() : 514.33154296875
2020-10-19 12:15:37,031 - CE - INFO - torch.cuda.memory_allocated() : 514.33154296875
2020-10-19 12:15:37,031 - CE - INFO - torch.cuda.memory_reserved() : 534.0
2020-10-19 12:15:37,031 - CE - INFO - torch.cuda.max_memory_reserved() : 534.0

The output of "nvidia-smi" :
2020-10-19 12:15:37,081 - CE - INFO - | ID | Name      | Serial        | UUID                                     || GPU temp. | GPU util. | Memory util. || Memory total | Memory used | Memory free || Display mode | Display active |
2020-10-19 12:15:37,081 - CE - INFO - |  0 | Tesla K80 | 0324516191902 | GPU-1e7baee8-174b-2178-7115-cf4a063a8923 ||       50C |        3% |           8% ||      11441MB |       898MB |     10543MB || Disabled     | Disabled       |

----------------Tesla v100---------------------
2020-10-20 08:18:42,952 - CE - INFO - torch.cuda.max_memory_allocated() : 514.33154296875
2020-10-20 08:18:42,952 - CE - INFO - torch.cuda.memory_allocated() : 514.33154296875
2020-10-20 08:18:42,953 - CE - INFO - torch.cuda.memory_reserved() : 534.0
2020-10-20 08:18:42,953 - CE - INFO - torch.cuda.max_memory_reserved() : 534.0

The output of "nvidia-smi" :
2020-10-20 08:18:43,020 - CE - INFO - | ID | Name                 | Serial        | UUID                                     || GPU temp. | GPU util. | Memory util. || Memory total | Memory used | Memory free || Display mode | Display active |
2020-10-20 08:18:43,020 - CE - INFO - |  0 | Tesla V100-SXM2-16GB | 0323617004258 | GPU-849088a3-508a-1737-7611-75a087f18085 ||       29C |        0% |          11% ||      16160MB |      1714MB |     14446MB || Enabled      | Disabled       |


title: Open3D-ML and pytorch

body:
I’m currently trying to work with open3d ML and Pytorch. I followed the installation guide given in the Open3D-ML github. However when I try to import open3d.ml.torch it sends me the following error : Exception: Open3D was not built with PyTorch support!
I’m working with
python 3.8
open3d 0.12.0
pytorch 1.6.0
cuda 10.1
Windows 10

Do you have any idea of where that error comes from ?


title: Pytorch:RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

body:
I set my model and data to the same device,
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
net.to(device)

and I also do this:
inputs, labels = data
inputs, labels = inputs.to(device), labels.to(device)

but the error still exists. When it comes to 5000 times or more, the error will take place.

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

The following is the whole training code, I hope you can answer it. Thanks!
import torch
import os
import torchvision.transforms as transforms
from PIL import Image
from torch import nn
from torch.utils.data import Dataset, DataLoader

captcha_list = list('0123456789abcdefghijklmnopqrstuvwxyz_')
captcha_length = 6

# 验证码文本转为向量
def text2vec(text):
    vector = torch.zeros((captcha_length, len(captcha_list)))
    text_len = len(text)
    if text_len > captcha_length:
        raise ValueError("验证码超过6位啦！")
    for i in range(text_len):
        vector[i,captcha_list.index(text[i])] = 1
    return vector

# 验证码向量转为文本
def vec2text(vec):
    label = torch.nn.functional.softmax(vec, dim =1)
    vec = torch.argmax(label, dim=1)
    for v in vec:
        text_list = [captcha_list[v] for v in vec]
    return ''.join(text_list)

# 加载所有图片，并将验证码向量化
def make_dataset(data_path):
    img_names = os.listdir(data_path)
    samples = []
    for img_name in img_names:
        img_path = data_path+img_name
        target_str = img_name.split('_')[0].lower()
        samples.append((img_path, target_str))
    return samples

class CaptchaData(Dataset):
    def __init__(self, data_path, transform=None):
        super(Dataset, self).__init__()
        self.transform = transform
        self.samples = make_dataset(data_path)

    def __len__(self):
        return len(self.samples)

    def __getitem__(self, index):
        img_path, target = self.samples[index]
        target = text2vec(target)
        target = target.view(1, -1)[0]
        img = Image.open(img_path)
        img = img.resize((140,44))
        img = img.convert('RGB') # img转成向量
        if self.transform is not None:
            img = self.transform(img)
        return img, target
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 第一层神经网络
        # nn.Sequential: 将里面的模块依次加入到神经网络中
        self.layer1 = nn.Sequential(
            nn.Conv2d(3, 16, kernel_size=3, padding=1), # 3通道变成16通道，图片：44*140
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(2)  # 图片：22*70
        )
        # 第2层神经网络
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 64, kernel_size=3), # 16通道变成64通道，图片：20*68
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(2)  # 图片：10*34
        )
        # 第3层神经网络
        self.layer3 = nn.Sequential(
            nn.Conv2d(64, 128, kernel_size=3), # 16通道变成64通道，图片：8*32
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.MaxPool2d(2)  # 图片：4*16
        )
        # 第4层神经网络
        self.fc1 = nn.Sequential(
            nn.Linear(4*16*128, 1024),
            nn.Dropout(0.2),  # drop 20% of the neuron
            nn.ReLU()
        )
        # 第5层神经网络
        self.fc2 = nn.Linear(1024, 6*37) # 6:验证码的长度， 37: 字母列表的长度
    #前向传播
    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        x = self.fc2(x)
        return x
net = Net()

def calculat_acc(output, target):
    output, target = output.view(-1, len(captcha_list)), target.view(-1, len(captcha_list)) # 每37个就是一个字符
    output = nn.functional.softmax(output, dim=1)
    output = torch.argmax(output, dim=1)
    target = torch.argmax(target, dim=1)
    output, target = output.view(-1, captcha_length), target.view(-1, captcha_length) #每6个字符是一个验证码
    c = 0
    for i, j in zip(target, output):
        if torch.equal(i, j):
            c += 1
    acc = c / output.size()[0] * 100
    return acc

def train(epoch_nums):
    # 数据准备
    transform = transforms.Compose([transforms.ToTensor()]) # 不做数据增强和标准化了
    train_dataset = CaptchaData('./sougou_com_Trains/', transform=transform)
    train_data_loader = DataLoader(train_dataset, batch_size=32, num_workers=0, shuffle=True, drop_last=True)

    test_data = CaptchaData('./sougou_com_Trains/', transform=transform)
    test_data_loader = DataLoader(test_data, batch_size=128, num_workers=0, shuffle=True, drop_last=True)
    # 更换设备
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    print('当前设备是:',device)
    net.to(device)

    criterion = nn.MultiLabelSoftMarginLoss() # 损失函数
    optimizer = torch.optim.Adam(net.parameters(), lr=0.001) # 优化器

    # 加载模型
    model_path = './module_build/model.pth'
    if os.path.exists(model_path):
        print('开始加载模型')
        checkpoint = torch.load(model_path)
        net.load_state_dict(checkpoint['model_state_dict'])
        optimizer.load_state_dict(checkpoint['optimizer_state_dict'])

    # 开始训练
    i = 1
    for epoch in range(epoch_nums):
        running_loss = 0.0
        net.train() # 神经网络开启训练模式
        for data in train_data_loader:
            if i%100==0:
                print(i)
            inputs, labels = data
            inputs, labels = inputs.to(device), labels.to(device) #数据发送到指定设备
            #每次迭代都要把梯度置零
            optimizer.zero_grad()
            # 关键步骤
            # 前向传播
            outputs = net(inputs)
            # 计算误差
            loss = criterion(outputs, labels)
            # 后向传播
            loss.backward()
            # 优化参数
            optimizer.step()
            running_loss += loss.item()
            if i % 2000 == 0:
                acc = calculat_acc(outputs, labels)
                print('第%s次训练正确率: %.3f %%, loss: %.3f' % (i,acc,running_loss/2000))
                running_loss = 0
                # 保存模型
                torch.save({
                            'model_state_dict':net.state_dict(),
                            'optimizer_state_dict':optimizer.state_dict(),
                            },model_path)
            i += 1
        # 结束一个epoch,计算测试集的正确率
        net.eval() #测试模式
        with torch.no_grad():
            for inputs, labels in test_data_loader:
                outputs = net(inputs)
                acc = calculat_acc(outputs, labels)
                print('测试集正确率: %.3f %%' % (acc))
                break # 只测试一个batch

        # 每5个epoch 更新学习率
        if epoch % 5 == 4:
            for p in optimizer.param_groups:
                p['lr'] *= 0.9
train(10)


title: Crash when trying to export PyTorch model to ONNX: forward() missing 1 required positional argument

body:
I'm trying to convert pyTorch model to onnx like this:
torch.onnx.export(
  model=modnet.module,
  args=example_input,
  f=ONNX_PATH, # where should it be saved
  verbose=False,
  export_params=True,
  do_constant_folding=False,
  input_names=['input'],
  output_names=['output']
)

modnet is a model from this repo: https://github.com/ZHKKKe/MODNet
example_input is a Tensor of shape [1, 3, 512, 512]
During converting I received that error:
TypeError: forward() missing 1 required positional argument: 'inference'

This is my clone of Colab notebook to reproduce exception: https://colab.research.google.com/drive/1AE1VAXIXkm26krIOoBaFfhoE53hhuEdf?usp=sharing
Save me please! :)


title: RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn for non-neural network implementations

body:
I am working on a project where I am getting some funny error with the automatic differentiator in Pytorch.
I am trying to minimize a function with respect to x values. To do so, I use the code at the bottom of this post. As I understand it, I should be able to make an initial guess, set the requires_grad flag to true and run the forward pass (scores = alpha(Xsamples, model, robustness)) and then get the gradients with scores.backward() and then update my initial guess accordingly with optimizer.step. However, when I try running this I get the following error 'RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn' which I don't understand because I have set my initial guess to requires gradient. I tried looking on forums for help but most of the answers were regarding training neural networks so their fixes did not work in this case. Any guidance on this would be greatly appreciated, thank you.
epoch = 100
learning_rate = 0.01
N = 1
Xsamples = torch.randn(1,2)
Xsamples.requires_grad = True
optimizer = torch.optim.SGD([Xsamples], lr = learning_rate)
for i in range(epoch):
    scores = alpha(Xsamples, model, robustness)
    scores.backward() #dscore/ dx
    optimizer.step()
    optimizer.zero_grad()
return Xsamples


title: Image Classification Pytorch

body:
How to decide number of layers and final model in CNN to increase the accuracy of the prediction.
I am classifying images and currently getting 65% accuracy with simple model how should I enhance it to achieve maximum accuracy.
(Pytorch)


title: Conv 1x1 configuration for feature reduction

body:
I am using 1x1 convolution in the deep network to reduce a feature x: Bx2CxHxW to BxCxHxW. I have three options:

x -> Conv (1x1) ->  Batchnorm-->ReLU. Code will be output = ReLU(BN(Conv(x))). Reference resnet
x -> BN -> ReLU-> Conv. So the code will be output = Conv(ReLU(BN(x))) . Reference densenet
x-> Conv. The code is output = Conv(x)

Which one is most using for feature reduction? Why?


title: Load data from CSV using DataLoader

body:
I have a .csv file with the following format:

x1, x2, y, x3, x4
05  05  0  00  12
01  09  1  00  17
       ...
13  24  0  01  00

I have created the following class that will help me load my data with a DataLoader, so as to shuffle it and read it in batches as well as apply any possible transformations that I might want to (e.g. Normalize) with the end goal of creating a simple RNN.

class MyDataset(Dataset):
    def __init__(self, csv_file, transform=None):
        self.data_frame = pd.read_csv(csv_file)
        self.transform = transform

    def __len__(self):
        return len(self.data_frame)

    def __getitem__(self, idx):
        m_data = self.data_frame.iloc[idx, :].as_matrix()
        m_data = m_data.astype('float')
        sample = {'y': np.array([m_data[2]]), 'x': np.delete(m_data, 2)}

        if self.transform:
            sample = self.transform(sample)

        return sample

I can then load my data with something like

dataset = MyDataset(csv_file=file, transform=transforms.Compose([ToTensor()]))
dataloader = DataLoader(dataset, batch_size=4, shuffle=True, num_workers=4)

This follows this tutorial from PyTorch, that it is for loading a data set of images.

My question is, since I have a completely different data set is what I am doing sensible for my case, or is there some other way I should load and preprocesor my data more efficiently?


title: Compute Optical Flow corresponding to data in the torch.utils.data.DataLoader

body:
I have built a CNN model for action recognition in videos in PyTorch. I'm loading the data for training using the torch dataloader module.

train_loader = torch.utils.data.DataLoader(
            training_data,
            batch_size=8,
            shuffle=True,
            num_workers=4,
            pin_memory=True)

And then passing the train_loader for training the model.

train_epoch(i, train_loader, action_detect_model, criterion, optimizer, opt,
                        train_logger, train_batch_logger)

Now I want to add an additional path which will take the corresponding optical flow of the video frames. To calculate the optical flow I'm using cv2.calcOpticalFlowFarneback.  But the problem is that I'm not sure how to get the images corresponding to the data in the train data loader tensor as they will be shuffled. I don't want to pre-compute the optical flow as the storage requirement will be huge (each frame takes 600 kBs).


title: Finding subtraction of shifted tensor

body:
I'm trying to figure out how to do shifting on a tensor that has b (batch size), d (depth), h (hight) and w (width) represented as following:

b, d, h, w = tensor.size()

So, I need to find the subtract between the shifted tensor and the tensor itself.

I'm thinking of using torch.narrow or torch.concat to do it for each side (shift the right, left, up then down side) and at each time I subtract from the same tensor side (tensor itself side), then at the end I will add/sum the differences/subtractions of each side (so I will have the final subtraction between the shifted and the tensor itself.

I'm new to PyTorch, it's easy to understand but struggling to implemented and maybe there is a simpler way (directly do the subtraction rather than working on each side and so on .....)

Any help on that please?


title: Dataset enumeration (epoch and batchSize) when implementing Ray.Tune PBT hyperparameter optimization

body:
This is my first time trying to use Ray.Tune for hyperparameter optimization. I am confused as to where in the Ray code I should initialize the dataset as well as where to put the for-loops for defining the epoch and enumerating the dataset batches.

Background

In my normal training script, I follow several steps:
1. Parse the model options,
2. Initialize the dataset,
3. Create and initialize the model,
4. For-loop for progressing through the epochs,
5. Nested for-loop for enumerating the dataset batches

The Ray.Tune documentation says that when defining the Trainable class object, that I really only need _setup, _train, _save, and _restore. As I understand it, the _train() is for a single iteration and increments the training_iteration automatically. Given that the dataset_size may not be cleanly divisible by the batchSize, I calculate the total_steps as the training progresses. If I understand it right, my total_steps will not be equal to training_iteration. This is important because the number of steps is supposed to be used to determine when to evaluate the worker.

I also do not want to instantiate the dataset for each worker individually. Ray should instantiate the dataset once, and then the workers can access the data via shared memory.

Original train.py code

self.opt = TrainOptions().parse()
data_loader = CreateDataLoader(self.opt)
self.dataset = data_loader.load_data()
self.dataset_size = len(data_loader)

total_steps = 0
counter = 0
for epoch in range(self.opt.starting_epoch, self.opt.niter + self.opt.niter_decay + 1):
    for i, data in enumerate(self.dataset):
        total_steps += self.opt.batchSize if i<len(self.dataset) else (self.dataset_size * (epoch + 1)) - total_steps
        counter += 1
        self.model.set_input(data, self.opt)
        self.model.optimizeD()
        if counter % self.opt.critic_iters == 0:
            self.model.optimizeG()


title: How to save image paths using PyTorch CNN

body:
I have been working with a CNN in PyTorch and I need to save the image path and its associated predicted probabilities for each class (in this case the classes are pass or fail). This is my code to save the preds to a data frame:

preds_df = pd.DataFrame()
class_labels = []

model_ft.eval()
for i, (inputs, labels) in enumerate(dataloaders['train']):
    inputs = inputs.to(device)
    labels = labels.to(device)
    class_labels.append(labels.tolist())
    output = model_ft(inputs)

    sm = torch.nn.Softmax()
    probabilities = sm(output)
    arr = probabilities.data.cpu().numpy()
    df = pd.DataFrame(arr)

    preds_df = preds_df.append(df)

preds_df['prediction'] = preds_df.idxmax(axis=1)
class_list = [item for sublist in class_labels for item in sublist]
preds_df['label'] = class_list
preds_df.columns = ['pass (0)', 'fail (1)', 'prediction', 'label']

preds_df.to_csv('./zoom17CNN_preds.csv')

How can I save the image path as well for each file in the data loader? Thank you!


title: Removing padding from features extracted from pre-trained VGG16 using PyTorch

body:
I'm using a pre-trained vgg16 as the backbone for a U-Net segmentation model. The code i have is working
fine, but I was wondering if there is a way of removing the padding from the feature layers?

In the source code for the pre-trained vgg16 model it seems that the padding is set to 1 https://pytorch.org/docs/stable/_modules/torchvision/models/vgg.html#vgg16 . So i would like to know if there is a way of altering this after importing the pretrained model?

For example in the code:

import torchvision.models as models

vgg16 = models.vgg16()
conv1 = vgg16.features[0]

print(conv1)

Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))

I would like to remove the padding, to reduce the effect of artifacts at the edges.

Thanks in advance for any advice.


title: Backpropagation in train and eval mode combined

body:
I am trying to backpropagate a network for a loss function defined as follows:
L = L_1 + L_2. Here L_1 is computed using dropout and batchnorm layers active where as L_2 is computed with dropout and batchnorm set to inactive. To update parameters gradients of L_1 and L_2 would have to be summed. I have written the following procedure, but don't know how to check for correctness. Any help will be greatly appreciated.

optimizer.zero_grad()
net.train()
...# Calculation of L_1...
L_1.backward()
net.eval()
...# Calculation of L_2...
L_2.backward()
net.train()
optimizer.step()

My understanding says that L_1.backward() will compute gradients of L_1 with respect to all parameters (including batchnorm), whereas L_2.backward() will compute gradients with respect to parameters only and add it to original gradients. Finally setting the network into train mode followed by optimizer.step(), updates all the parameters (including batchnorm).


title: how to get value 0 or 1 in siamese neural network?

body:
I'm trying to use siamese neural network.
Here I want to compare 2 types of images and get the results of the score,
This is the code for the test model to produce the score

in this case i use pytorch

model = Siamese()

# Load state_dict
model.load_state_dict(torch.load('/Users/tania/Desktop/TA/model/model-batch-1001.pth'))

# Create the preprocessing transformation
from torchvision import transforms
transforms = transforms.ToTensor()

# load image(s)
from PIL import Image
x1 = Image.open('table.PNG')
x2 = Image.open('table.PNG')
# Transform

x1 = transforms(x1)
x2 = transforms(x2)

x1 = torch.stack([x1])
x2 = torch.stack([x2])

model.eval()

# Get prediction
output = model(x1,x2)
print (output)

so i got the score like this,

the score is -14.1640

basically in siamese if the image is the same then it produces a value of 1 and if different it will produce a value of 0

how do I get the value of 0 or 1 so that I know whether the image is the same or not?

please help me, i am newbie in neural network


title: Python multithreaded program crashing without error

body:
I am trying to parallelize the load of the data and the learning in a pytorch project. From main thread I create 2 threads, one loading the next batch while the other is learning from the current batch. The loading threading is transferring the loaded data through a Queue.

My problem is: the program suddenly stops at random state of execution without any error message (debug execution or not). Sometimes at the first epoch, sometimes after 7 epoch (50min)... Is it possible to have an error message somehow ? Does anyone have encountered that problem ?

It makes me think about memory leakage, but I check all the code about the shared data. I have also seen things about prints not being thread safe so I removed them... Please also note that the code didn't had this problem before parallelization.

I am using:

conda environement.
Threading.thread
pytorch
Windows server

Update: Apparently the pytorch code referencing cuda learning doesn't like to be called from a separate thread. If I keep my cuda learning in the main thread, it stays alive...

Code: Since I have less unexpected crashes with the cuda learning in the main thread. I only kept one thread. (it also makes more sense)

part of the main:

        dataQueue = queue.Queue()

        dataAvailable = threading.Event()
        doneComputing = threading.Event()

        # Create new thread
        loadThread = LoadingThread(1,dataQueue)
        #learningThread.daemon = True
        loadThread.daemon=True
        # Add threads to thread list
        threads = []
        threads.append(loadThread)

        print("     >> Parrallel process start")

        # Start new Thread
        loadThread.start()
        doneComputing.set()
        # Learning process
        for i_import in range(import_init['n_import_train']):
            # Wait for data loaded
            dataAvailable.wait()
            data = dataQueue.get()
            dataAvailable.clear()
            # DO learning
            net,net_prev,y_all,yhat_all = doLearning(data, i_import, net, net_prev, y_all, yhat_all)
            doneComputing.set()

        # Wait for all threads to complete
        for t in threads:
            t.join()

One interesting fact is that it seems that the program crashes more often if the model sent to cuda is heavy ... Could tat be only a cuda problem?


title: Pytorch: Overfitting on a small batch: Debugging

body:
I am building a multi-class image classifier.
There is a debugging trick to overfit on a single batch to check if there any deeper bugs in the program.
How to design the code in a way that can do it in a much portable format?
One arduous and a not smart way is to build a holdout train/test folder for a small batch where test class consists of 2 distribution - seen data and unseen data and if the model is performing better on seen data and poorly on unseen data, then we can conclude that our network doesn't have any deeper structural bug.
But, this does not seems like a smart and a portable way, and have to do it with every problem.
Currently, I have a dataset class where I am partitioning the data in train/dev/test in the below way -
def split_equal_into_val_test(csv_file=None, stratify_colname='y',
                              frac_train=0.6, frac_val=0.15, frac_test=0.25,
                              ):
    """
    Split a Pandas dataframe into three subsets (train, val, and test).

    Following fractional ratios provided by the user, where val and
    test set have the same number of each classes while train set have
    the remaining number of left classes
    Parameters
    ----------
    csv_file : Input data csv file to be passed
    stratify_colname : str
        The name of the column that will be used for stratification. Usually
        this column would be for the label.
    frac_train : float
    frac_val   : float
    frac_test  : float
        The ratios with which the dataframe will be split into train, val, and
        test data. The values should be expressed as float fractions and should
        sum to 1.0.
    random_state : int, None, or RandomStateInstance
        Value to be passed to train_test_split().

    Returns
    -------
    df_train, df_val, df_test :
        Dataframes containing the three splits.

    """
    df = pd.read_csv(csv_file).iloc[:, 1:]

    if frac_train + frac_val + frac_test != 1.0:
        raise ValueError('fractions %f, %f, %f do not add up to 1.0' %
                         (frac_train, frac_val, frac_test))

    if stratify_colname not in df.columns:
        raise ValueError('%s is not a column in the dataframe' %
                         (stratify_colname))

    df_input = df

    no_of_classes = 4
    sfact = int((0.1*len(df))/no_of_classes)

    # Shuffling the data frame
    df_input = df_input.sample(frac=1)

    df_temp_1 = df_input[df_input['labels'] == 1][:sfact]
    df_temp_2 = df_input[df_input['labels'] == 2][:sfact]
    df_temp_3 = df_input[df_input['labels'] == 3][:sfact]
    df_temp_4 = df_input[df_input['labels'] == 4][:sfact]

    dev_test_df = pd.concat([df_temp_1, df_temp_2, df_temp_3, df_temp_4])
    dev_test_y = dev_test_df['labels']
    # Split the temp dataframe into val and test dataframes.
    df_val, df_test, dev_Y, test_Y = train_test_split(
        dev_test_df, dev_test_y,
        stratify=dev_test_y,
        test_size=0.5,
        )

    df_train = df[~df['img'].isin(dev_test_df['img'])]

    assert len(df_input) == len(df_train) + len(df_val) + len(df_test)

    return df_train, df_val, df_test

def train_val_to_ids(train, val, test, stratify_columns='labels'): # noqa
    """
    Convert the stratified dataset in the form of dictionary : partition['train] and labels.

    To generate the parallel code according to https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel
    Parameters
    -----------
    csv_file : Input data csv file to be passed
    stratify_columns : The label column

    Returns
    -----------
    partition, labels:
        partition dictionary containing train and validation ids and label dictionary containing ids and their labels # noqa

    """
    train_list, val_list, test_list = train['img'].to_list(), val['img'].to_list(), test['img'].to_list() # noqa
    partition = {"train_set": train_list,
                 "val_set": val_list,
                 }
    labels = dict(zip(train.img, train.labels))
    labels.update(dict(zip(val.img, val.labels)))
    return partition, labels

P.S - I know about the Pytorch lightning and know that they have an overfitting feature which can be used easily but I don't want to move to PyTorch lightning.


title: How to convert cupy data into PyTorch tensor?

body:
Recently I've learned that CuPy would utilize GPU to accelerate the computation in deep learning. However, after the following step I still got an unsolvable error:

I installed CuPy correctly through https://docs.cupy.dev/en/stable/install.html. Other packages are also correctly installed.
The original code is shown below, in file A:

import h5py
import torch
import numpy as np
import cupy as cp
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, utils

class cupy_Dataset(Dataset):

    def __init__(self, file_dir):

        super(cupy_Dataset, self).__init__()
        data = h5py.File(file_dir)

            self.ms = cp.array(data.get("ms"))
            self.lms = cp.array(data.get("lms"))
            self.pan = cp.array(data.get("pan"))
            self.gt = cp.array(data.get("gt"))

            self.ms = (self.ms[0:10, :, :, :])
            self.lms = (self.lms[0:10, :, :, :])
            self.pan = (self.pan[0:10, :, :])
            self.gt = (self.gt[0:10, :, :, :])

            self.ms = torch.utils.dlpack.from_dlpack(self.ms.toDLpack()).float()
            self.lms = torch.utils.dlpack.from_dlpack(self.lms.toDLpack()).float()
            self.pan = torch.utils.dlpack.from_dlpack(self.pan.toDLpack()).float()
            self.gt = torch.utils.dlpack.from_dlpack(self.gt.toDLpack()).float()

    def __len__(self):
        return len(self.gt.shape[0])

    def __getitem__(self, idx):

        ms = self.ms[idx, :, :, :]
        lms = self.lms[idx, :, :, :]
        pan = self.pan[idx, :, :, :]
        gt = self.gt[idx, :, :, :]

        return ms, lms, pan, gt

if __name__ == '__main__':
    train_dataset = cupy_WorldView3_Dataset(file_dir='./training_data/train.mat')
    print(train_dataset.gt.shape)

in file B:
import torch
import cupy
from {file_A} import cupy_Dataset

train_dataset = cupy_Dataset(file_dir='./training_data/train.mat')
gt = train_dataset.gt
pan = train_dataset.pan
lms = train_dataset.lms
ms = train_dataset.ms

print(type(gt))

The outcome in file A is shown as:

Traceback (most recent call last):
  File "{file path}", line 94, in <module>
    train_dataset = cupy_WorldView3_Dataset(file_dir='./training_data/train.mat')
  File "{file path}", line 76, in __init__
    self.ms = torch.utils.dlpack.from_dlpack(self.ms.toDLpack()).float()
AttributeError: module 'torch.utils' has no attribute 'dlpack'

The outcome in file B is shown as:
Using backend: pytorch
Traceback (most recent call last):
  File "/home/office-desktop/.pycharm_helpers/pydev/pydevd.py", line 1477, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/office-desktop/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "{file path}", line 6, in <module>
    train_dataset = cupy_Dataset(file_dir='./training_data/train.mat')
  File "{file path}", line 76, in __init__
    self.ms = torch.utils.dlpack.from_dlpack(self.ms.toDLpack()).float()
AttributeError: 'cupy.core.core.ndarray' object has no attribute 'toDLpack'

Process finished with exit code 1

FYI, my basic thoughts are using DLpack as the interconnection between CuPy and PyTorch tensor. The initial usage and syntax can be found in https://docs.cupy.dev/en/stable/reference/interoperability.html and https://pytorch.org/docs/stable/dlpack.html.
Why is this? Thank you in advance. If you need any additional info, I am willing to provide :)


title: while exporting T5 model to onnx using fastT5 getting "RuntimeError:output with shape [5, 8, 1, 2] doesn't match the broadcast shape [5, 8, 2, 2]"

body:
i'm trying to convert T5 model to onnx using the fastT5 library, but
getting an error while running the following code
from fastT5 import export_and_get_onnx_model
from transformers import AutoTokenizer

model_name = 't5-small'
model = export_and_get_onnx_model(model_name)

tokenizer = AutoTokenizer.from_pretrained(model_name)
t_input = "translate English to French: The universe is a dark forest."
token = tokenizer(t_input, return_tensors='pt')

tokens = model.generate(input_ids=token['input_ids'],
               attention_mask=token['attention_mask'],
               num_beams=2)

output = tokenizer.decode(tokens.squeeze(), skip_special_tokens=True)
print(output)

the error:
/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py:244: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if causal_mask.shape[1] < attention_mask.shape[1]:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-16-80094b7c4f6f> in <module>()
      7                     input_names=decoder_input_names,
      8                     output_names=decoder_output_names,
----> 9                     dynamic_axes=dyn_axis_params,
     10                     )

24 frames
/usr/local/lib/python3.7/dist-packages/transformers/models/t5/modeling_t5.py in forward(self, hidden_states, mask, key_value_states, position_bias, past_key_value, layer_head_mask, query_length, use_cache, output_attentions)
    497                 position_bias = position_bias + mask  # (batch_size, n_heads, seq_length, key_length)
    498
--> 499         scores += position_bias
    500         attn_weights = F.softmax(scores.float(), dim=-1).type_as(
    501             scores

RuntimeError: output with shape [5, 8, 1, 2] doesn't match the broadcast shape [5, 8, 2, 2]

can someone please help me solve the issue?

thank you.


title: How to train BERT with custom (raw text) domain-specific dataset using Huggingface?

body:
I would like to train / fine-tune the BERT model with some own datasets which contain just raw data from a specific domain, similar to BioBERT. In the first step I just want to train BERT with this raw data, without fine-tuning with any specific NLP task. So, I want to build a base which I can potentially use for fine-tuning with annotated train data for specific tasks, such as NER, etc.
Is this possible in the way I plan to do it?
I tried to use the example scripts of Huggingface: https://github.com/huggingface/transformers/tree/master/examples/language-modeling
I used the run_mlm script in the following way:
python run_mlm.py \
    --model_name_or_path bert-base-uncased\
    --train_file /path/to/text_file.txt \
    --do_train \
    --output_dir /out

I get some output but how can I evaluate the resulting model?
There are some intrinsic evaluation methods that I wanted to try but I failed, like word similarity or word analogy.
If you could help me with my approach I would be very thankful.


title: Pytorch getting RuntimeError: Found dtype Double but expected Float

body:
I am trying to implement a neural net in PyTorch but it doesn't seem to work. The problem seems to be in the training loop. I've spend several hours into this but can't get it right. Please help, thanks.
I haven't added the data preprocessing parts.
# importing libraries
import pandas as pd
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
import torch.nn.functional as F

# get x function (dataset related stuff)
def Getx(idx):
    sample = samples[idx]
    vector = Calculating_bottom(sample)
    vector = torch.as_tensor(vector, dtype = torch.float64)

    return vector

# get y function (dataset related stuff)
def Gety(idx):
    y = np.array(train.iloc[idx, 4], dtype = np.float64)
    y = torch.as_tensor(y, dtype = torch.float64)

    return y

# dataset
class mydataset(Dataset):

    def __init__(self):
        super().__init__()

    def __getitem__(self, index):
        x = Getx(index)
        y = Gety(index)

        return x, y

    def __len__(self):
        return len(train)

dataset = mydataset()

# sample dataset value
print(dataset.__getitem__(0))

(tensor([ 5.,  5.,  8., 14.], dtype=torch.float64), tensor(-0.3403, dtype=torch.float64))
# data-loader
dataloader = DataLoader(dataset, batch_size = 1, shuffle = True)

# nn architecture
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(4, 4)
        self.fc2 = nn.Linear(4, 2)
        self.fc3 = nn.Linear(2, 1)

    def forward(self, x):
        x = x.float()
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

model = Net()

# device
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)

# hyper-parameters
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

# training loop

for epoch in range(5):

    for batch in dataloader:

        # unpacking
        x, y = batch
        x.to(device)
        y.to(device)

        # reset gradients
        optimizer.zero_grad()

        # forward propagation through the network
        out = model(x)

        # calculate the loss
        loss = criterion(out, y)

        # backpropagation
        loss.backward()

        # update the parameters
        optimizer.step()

Error:
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/loss.py:446: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-3f68fcee9ff3> in <module>
     20
     21         # backpropagation
---> 22         loss.backward()
     23
     24         # update the parameters

/opt/conda/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    219                 retain_graph=retain_graph,
    220                 create_graph=create_graph)
--> 221         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    222
    223     def register_hook(self, hook):

/opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
--> 132         allow_unreachable=True)  # allow_unreachable flag
    133
    134

RuntimeError: Found dtype Double but expected Float


title: n*m*3 input image to an nxm label in PyTorch

body:
The input to my network is an RGB image with dimensions nm, how can I get the output to have dimensions of nm.

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 20, kernel_size = 5)
        self.conv2 = nn.Conv2d(20, 50, kernel_size = 3)
        self.conv3 = nn.ConvTranspose2d(50,20, kernel_size = 5)
        self.conv4 = nn.ConvTranspose2d(20,1, kernel_size = 3)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        x = F.relu(self.conv4(x))
    return x

I currently output a 1 * n * m. How can I output an n*m?


title: tensor - box plot from a tensor

body:
I'm trying to create a box between two variables probability (y axis) and flowername ( x- axis) .Probability is a tensor. For the flower name, I have to pick the name from a dictionary (flower_dict) where the key is referenced from another tensor, class Index. How do I create box plot ? ANy help is appreciate

print("Probability:", probs)
Probability: tensor([[ 0.9961,  0.0020,  0.0011,  0.0005,  0.0001]], device='cuda:0')

print("Class Index:", classes)
Class Index: tensor([[  21,  3,  45,  34,  27]], device='cuda:0')

print(flower_dict)
{'21': 'fire lily', '3': 'canterbury bells', '45': 'bolero deep blue', '1': 'pink primrose', '34': 'mexican aster', '27': 'prince of wales feathers', '7': 'moon orchid', '16': 'globe-flower', '25': 'grape hyacinth', '26': 'corn poppy', '79': 'toad lily', '39': 'siam tulip', '24': 'red ginger'}


title: How can I create a PyTorch tensor with all zeroes and a 1 in the middle of the third dimension?

body:
I have a tensor, torch.Size([161, 161, 11]) and I want to set it all to zeroes, which I can do with: self.conv1.weight.data = torch.zeros(self.conv1.weight.data.size())

Except, I want column 6 (the middle) of the third dimension to be all ones. How do I do that?


title: How can I interleave 5 PyTorch tensors?

body:
I have 5 tensors of shape torch.Size([7, 20, 180])

I want to interleave them, one after the other along dim=1. So that my final shape will be torch.Size([7, 100, 180]).

Basically, I want the first element from the first tensor, then the first element from the second tensor, and so on.


title: Reset Random Initialization on Pytorch

body:
I want to do some experiments on feedforward neural networks. To make a fair comparison, I need them to have the exact same random initialisation. How can I do it?

Is there a way to save the same initial weights such that I can train a network and then reinitialize it exactly as it was before?

I have been trying to save the initial parameters on a list, called 'init' , and then reassign the parameters  but it did not work:

i = 0
for name, param in model.named_parameters():
    param = init[i]
    i += 1

Any suggestion?