title: BERT error: Cannot squeeze a dimension whose value is not 1

body:
I have an mlmodel based on pytorch-pretrained-BERT, exported via ONNX to CoreML. That process was pretty smooth, so now I'm trying to do some (very) basic testing—i.e., just to make some kind of prediction, and get a rough idea of what performance problems we might encounter.

Howver, when I try to run prediction, I get the following error:

[espresso] [Espresso::handle_ex_plan] exception=Espresso exception: "Invalid state": Cannot squeeze a dimension whose
value is not 1: shape[1]=128 stat2020-02-16
11:36:05.959261-0800 Xxxxx[6725:2140794] [coreml] Error computing NN outputs -5

Is this error indicating a problem with the model itself (i.e., from model conversion), or is there something in Swift/CoreML that I'm doing wrong? My prediction function looks like this:

public func prediction(withInput input: String) -> MLMultiArray? {
        var predictions: MLMultiArray? = nil
        if let bert = bertMLModel {
            var ids = tokenizer.tokenizeToIds(text: input, includeWordpiece: true)
            print("ids: \(ids)")
            while ids.count < 256 {
                ids.append(0)
            }
            let inputMLArray = MLMultiArray.from(ids, dims: 2)
            let modelInput = bert_256_FP16Input(input_ids: inputMLArray)
            var modelOutput: bert_256_FP16Output? = nil
            do {
                modelOutput = try bert.prediction(input: modelInput)
            } catch {
                print("Error running prediction: \(error)")
            }
            if let modelOutput = modelOutput {
                predictions = modelOutput.output_weights
            }
        }
        return predictions
    }

I'm not trying to do anything meaningful, at this stage, only to get it running.

I used the pytorch-pretrained-BERT repo because I was able to find a ground-up pretraining example for that. I have since noticed that HuggingFace has released a "from scratch" training option, but there are still some issues with the tutorial that are being sorted out. So I would like to at least understand what might be going wrong with my current model/approach. However, if the problem is definitely in the PyTorch->ONNX->CoreML conversion, then I don't really want to fight that fight and will just dig into what HuggingFace is offering.

Any thoughts appreciated.

UPDATE: On Matthijs' advice, I'm trying to predict from the model in python:

from coremltools.models import *
import numpy as np

# Just a "dummy" input, but it is a valid series of tokens for my data.
tokens = [3, 68, 45, 68, 45,  5, 45, 68, 45,  4]
tokens_tensor = np.zeros((1, 128))
for i in range(0, 10):
    tokens_tensor[0, i] = tokens[I]
# I'm doing masked token prediction, so one segment.
segments_tensor = np.zeros((1, 128))

mlmodel = 'bert_fp16.mlmodel'

model = MLModel(mlmodel)
spec = model.get_spec()
print("spec: ", spec.description)

predictions = model.predict({'input.1': np.asarray(tokens_tensor, dtype=np.int32), 'input.3': np.asarray(segments_tensor, dtype=np.int32)})

I admit I haven't run an mlmodel from python before, but I think the inputs are correct. The spec indicates:

input {
  name: "input.1"
  type {
    multiArrayType {
      shape: 1
      shape: 128
      dataType: INT32
    }
  }
}
input {
  name: "input.3"
  type {
    multiArrayType {
      shape: 1
      shape: 128
      dataType: INT32
    }
  }
}
...

for my inputs.

For this case, I don't get the cannot squeeze message, or the error code (-5), but it does fail with Error computing NN outputs. So something is definitely wrong. I'm just not at all sure how to go about debugging it.

J.

UPDATE: For comparison I've trained/converted the HuggingFace BERT Model (actually, DistilBert — I've updated the code above accordingly) and have the same error. Looking at the log from onnx, I see that there is one Squeeze added (also clear from the onnx-coreml log, of course), but the only squeeze in the PyTorch code is in BertForQuestionAnswering, not BertForMaskedLM. Perhaps onnx is building the question answering model, not the mlm one (or the question answering one is being saved in the checkpoint)?

Looking at the swift-coreml-transformers sample code I can see that distilBert's input is just let input_ids = MLMultiArray.from(allTokens, dims: 2), which is exactly how I've defined it. So I'm at a dead-end, I think. Has anybody managed to run MLM with Bert/DistilBert in CoreML (via onnx)? If so, an example would be super helpful.

I'm training using the latest run_language_modeling.py from HuggingFace, btw.

Looking at the mlmodel in Netron, I can see the offending squeeze. I confess that I don't know what that branch of the input is for, but I'm guessing it's a mask (and a further guess might be that it has to do with question answering). Could I just remove that somehow?


title: How do I clamp the magnitude of a PyTorch tensor?

body:
I know I can use torch.clamp to clamp a tensor's values within some min / max, but how can I do this if I want to clamp by the magnitude (absolute value)? Example:

import torch
t = torch.tensor([-5.0, -250, -1, 0.003, 7, 1238])
min_mag = 1 / 10
max_mag = 100

# desired output:
tensor([  -5.0000, -100.0000,   -1.0000,    0.1000,    7.0000,  100.0000])


title: Non-deterministic behavior for training a neural network on GPU implemented in PyTorch and with a fixed random seed

body:
I observed a strange behavior of the final Accuracy when I run exactly the same experiment (the same code for training neural net for image classification) with the same random seed on different GPUs (machines). I use only one GPU. Precisely, When I run the experiment on one machine_1 the Accuracy is 86,37. When I run the experiment on machine_2 the Accuracy is 88,0.
There is no variability when I run the experiment multiple times on the same machine. PyTorch and CUDA versions are the same. Could you help me to figure out the reason and fix it?
Machine_1:
NVIDIA-SMI 440.82       Driver Version: 440.82       CUDA Version: 10.2
Machine_2:
NVIDIA-SMI 440.100      Driver Version: 440.100      CUDA Version: 10.2
To fix random seed I use the following code:
random.seed(args.seed)
os.environ['PYTHONHASHSEED'] = str(args.seed)
np.random.seed(args.seed)
torch.manual_seed(args.seed)
torch.cuda.manual_seed(args.seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True


title: Training neural network on gradient of input with pytorch

body:
I am currently trying to train a neural network with pytorch, where I try to match the input on the input derivative. I want to do this because this is ensuring a conservative vector field. (Done in training of neural networks for force matching in molecular dynamics)
This means:
input = torch.rand((n,3),requires_grad=True)
output = torch.rand((n,3),requires_grad=True)
prediction = model(input) # size of the prediction[1]
input_grad = torch.autograd.grad(outputs=prediction, inputs=input, retain_graph=True, create_graph=True)
loss = loss_fn(output,input_grad)
...

The problem is, that if I try to update the parameters of the neural network, the gradient in all parameters is 0. I made sure that the model is working properly; I don't know how to build the graph that the model is training properly.
In Jaxmd, it is possible to train a model like this as shown [Jax Glass Training][1].
I've also tried
input_grad = torch.autograd.grad(outputs=prediction, inputs=input, create_graph=True)

but this yields to similar results and does not make sense.
[1]: https://colab.research.google.com/github/google/jax-md/blob/master/notebooks/neural_networks.ipynb#scrollTo=WNs8v2745Mc3
EDIT:
updated code example for reproduction pytorch version 1.6.0
import torch

class Feedforward(torch.nn.Module):
        def __init__(self, input_size, hidden_size):
            super(Feedforward, self).__init__()
            self.input_size = input_size
            self.hidden_size  = hidden_size
            self.fc1 = torch.nn.Linear(self.input_size, self.hidden_size)
            self.relu = torch.nn.ReLU()
            self.fc2 = torch.nn.Linear(self.hidden_size, 1)

        def forward(self, x):
            hidden = self.fc1(x)
            relu = self.relu(hidden)
            output = self.fc2(relu)
            output = output.sum()
            output =torch.autograd.grad(outputs=output, inputs=x, retain_graph=True, create_graph=True)
            return output[0]

test_input = torch.rand((10,3),requires_grad=True)
test_output = torch.rand((10,3))

model = Feedforward(3,10)
optim = torch.optim.Adam(model.parameters())
optim.zero_grad()
loss_fn = torch.nn.L1Loss()
model.train()
out = model(test_input)
loss = loss_fn(out, test_output)
loss.backward()
optim.step() # if you break here and investigate the gradients
             # of the FFNN, the gradients will be 0


title: Change the precision of torch.sigmoid?

body:
I want my sigmoid to never print a solid 1 or 0, but to actually print the exact value
i tried using
torch.set_printoptions(precision=20)

but it didn't work. here's a sample output of the sigmoid function :
before sigmoid : tensor([[21.2955703735]])
after sigmoid : tensor([[1.]])

but i don't want it to print 1, i want it to print the exact number, how can i force this?


title: No detectible overtraining with pytorch faster-RCNN

body:
I've trained a Faster-RCNN AI for 100 epochs in the hopes of detecting overtraining. I've plotted the training loss and average validation score for every epoch in the image below.

As you can see after 23 epochs the validation score stays constant but never decreases. I expected a decrease in validation score after x epochs which would show overtraining.
So my question is, why can't I detect overtraining or how can I detect it with this information? Or more practical: how can I know from this data which model version is the "best".
The used model is the pytorch faster-RCNN model: https://pytorch.org/docs/stable/torchvision/models.html?highlight=model#faster-r-cnn
The dataset consists of 581 images which are frames taken out of videos every 5s. The goal is to detect heads (not faces, but also the back of the head). Note that the videos are taken from 2 cameras in the same room with the same lighting etc. The validation set consists of 50 images taken at random from these 581 images.
The most important code:
def build_model(num_classes):
    # get moddel Resnet
    model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False)
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
    return model

def main():
    # train on the GPU or on the CPU, if a GPU is not available
    device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
    if torch.cuda.is_available():
        print('Train on GPU.')
    else:
        print('Train on CPU.')

    paths = ... # (= paths to images)

    # our dataset has two classes only - background and heads
    num_classes = 2

    # use our dataset and defined transformations
    dataset = PO3_dataset.PO3Dataset(paths, PO3_dataset.get_transform(train=True))
    dataset_test = PO3_dataset.PO3Dataset(paths, PO3_dataset.get_transform(train=False))

    # split the dataset in train and test set randomly
    indices = torch.randperm(len(dataset)).tolist()
    dataset = torch.utils.data.Subset(dataset, indices[:-50])
    dataset_test = torch.utils.data.Subset(dataset_test, indices[-50:])

    # define training and validation data loaders
    data_loader = torch.utils.data.DataLoader(
        dataset, batch_size=2, shuffle=True, num_workers=4,
        collate_fn=utils.collate_fn)

    data_loader_test = torch.utils.data.DataLoader(
        dataset_test, batch_size=1, shuffle=False, num_workers=4,
        collate_fn=utils.collate_fn)

    # get the model using our helper function
    model = build_model(num_classes)
    model.to(device)

    # construct an optimizer
    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)
    # and a learning rate scheduler
    lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)

    MODEL_PATH = "./saved_models/PO3_v3/"
    try:
        os.mkdir(MODEL_PATH)
    except FileExistsError:
        # directory already exists
        pass

    tr_loss = []
    val_score = []

    # let's train it for a few epochs
    num_epochs = 100

    for epoch in range(num_epochs):
        # train for one epoch, printing every 20 iterations
        training_data = train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=20)
        print("Training Loss: ", str(training_data.loss)[:7])
        tr_loss.append(float(str(training_data.loss)[:7]))
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        _, epoch_val_score = evaluate(model, data_loader_test, device=device)
        print("Validation score:", epoch_val_score)
        val_score.append(epoch_val_score)

        # save current version of the model
        print("Saved the model at:", os.path.join(MODEL_PATH, "training_" + str(epoch) + ".pth"))
        torch.save(model, os.path.join(MODEL_PATH, "training_" + str(epoch) + ".pth"))

    print(tr_loss, val_score)

    print("That's it! The best modelversion is: ", val_score.index(max(val_score)) + 1)
    print("Path: " + os.path.join(MODEL_PATH, "training_" + str(val_score.index(max(val_score)) + 1) + ".pth"))

    plot_losses.plt_losses(num_epochs, tr_loss, val_score, os.path.join(MODEL_PATH, "losses.png"))

if __name__ == "__main__":
    main()

Engine.py, full code at https://github.com/pytorch/vision/blob/master/references/detection/engine.py.
Adjusted method for geting scores out of evaluate():
@torch.no_grad()
def evaluate(model, data_loader, device):
    n_threads = torch.get_num_threads()
    # FIXME remove this and make paste_masks_in_image run on the GPU
    torch.set_num_threads(1)
    cpu_device = torch.device("cpu")
    model.eval()
    metric_logger = utils.MetricLogger(delimiter="  ")
    header = 'Test:'

    coco = get_coco_api_from_dataset(data_loader.dataset)
    iou_types = _get_iou_types(model)
    coco_evaluator = CocoEvaluator(coco, iou_types)

    score = []
    for images, targets in metric_logger.log_every(data_loader, 100, header):
        images = list(img.to(device) for img in images)

        torch.cuda.synchronize()
        model_time = time.time()
        outputs = model(images)
        outputs = [{k: v.to(cpu_device) for k, v in t.items()} for t in outputs]

        # get validation scores
        if len(outputs[0]["scores"]) != 0:
            score.append(float(sum(outputs[0]["scores"]) / len(outputs[0]["scores"])))

        model_time = time.time() - model_time

        res = {target["image_id"].item(): output for target, output in zip(targets, outputs)}
        evaluator_time = time.time()
        coco_evaluator.update(res)
        evaluator_time = time.time() - evaluator_time
        metric_logger.update(model_time=model_time, evaluator_time=evaluator_time)

    # gather the stats from all processes
    metric_logger.synchronize_between_processes()
    print("Averaged stats:", metric_logger)
    coco_evaluator.synchronize_between_processes()

    # accumulate predictions from all images
    coco_evaluator.accumulate()
    coco_evaluator.summarize()
    torch.set_num_threads(n_threads)
    return coco_evaluator, sum(score)/len(score)


title: PyTorch with yolov5: color channel and result display

body:
I have a script that grabs an application's screenshot and displays it. it works quite nicely on my machine like a video with around 60FPS. Now I want to use a yolov5 object detection model on these frames, with TorchHub, as advised here.
The following works:
import os
os.getcwd()
from PIL import ImageGrab
import numpy as np
import cv2
import pyautogui
import win32gui
import time
from mss import mss
from PIL import Image
import tempfile
os.system('calc')
sct = mss()
xx=1
tstart = time.time()
while xx<10000:
    hwnd = win32gui.FindWindow(None, 'Calculator')
    left_x, top_y, right_x, bottom_y = win32gui.GetWindowRect(hwnd)
    #screen = np.array(ImageGrab.grab( bbox = (left_x, top_y, right_x, bottom_y ) ) )
    bbox = {'top': top_y, 'left': left_x, 'width': right_x-left_x, 'height':bottom_y-top_y }
    screen = sct.grab(bbox)
    scr = np.array(screen)

    cv2.imshow('window', scr)
    if cv2.waitKey(25) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break
    xx+=1
cv2.destroyAllWindows()
tend = time.time()
print(xx/(tend-tstart))
print((tend-tstart))
os.system('taskkill /f /im calculator.exe')

Below I try to import torch and use my previously trained model,
screen = sct.grab(bbox)
scr = np.array(screen)
result = model(scr, size=400)
result.save("test.png") #this gives a TypeError: save() takes 1 positional argument but 2 were given
result.show() #this opens a new Paint instance for every frame instead of keeping the same window.
# The shown image is also in a wrong color channel
scr = cv2.imread("test.png")
# How can I use the `result` as argument to cv2.imshow(),
# without saving to disk if possible?

My questions:

result.show() shows an image with wrong color channel compared to cv2.imshow(), how can I ensure that the image being fed to model is on the correct channel?
The performance of classification and detection drastically decrease compared to the training validation, perhaps because of 1?
Do you know how I can display the result model image with bounding boxes in a single window like what cv2.imshow() does ? (result.show() opens a new Paint process instance for each frame) ? How can I save this result image to disk and find more documentation on how to interact with model objects in general?


title: How to convert one-hot vector to label index and back in Pytorch?

body:
How to transform vectors of labels to one-hot encoding and back in Pytorch?
The solution to the question was copied to here after having to go through the entire forum discussion, instead of just finding an easy one from googling.


When I try to train the model I get the following error:

TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not
str

The code I'm using is:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, kernel_size=5)
        self.conv2 = nn.Conv2d(6, 16, kernel_size=5)
        self.dropout = nn.Dropout2d()
        self.fc1 = nn.Linear(256, 64)
        self.fc2 = nn.Linear(64, 1)
        self.hybrid = Hybrid(qiskit.Aer.get_backend('qasm_simulator'), 100, np.pi / 2)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2)
        x = self.dropout(x)
        x = x.view(1, -1)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        x = self.hybrid(x)
        return torch.cat((x, 1 - x), -1)

model = Net()
optimizer = optim.Adam(model.parameters(), lr=0.001)
loss_func = nn.NLLLoss()

epochs = 20
loss_list = []

model.train()

for epoch in range(epochs):
    total_loss = []

    for i, data in enumerate(train_ldr, 0):
        # get the inputs; data is a list of [inputs, labels]
        X_train, Y_train = data
        print(data)
        optimizer.zero_grad()

        # Forward pass
        output = model(X_train)
        # Calculating loss
        loss = loss_func(output, Y_train)
        # Backward pass
        loss.backward()
        # Optimize the weights
        optimizer.step()

        total_loss.append(loss.item())
    loss_list.append(sum(total_loss)/len(total_loss))
    print('Training [{:.0f}%]\tLoss: {:.4f}'.format(
        100. * (epoch + 1) / epochs, loss_list[-1]))

The full traceback is:
{'data': tensor([[715.9147, 679.4994, 131.4772,   9.4777,   9.4777,  13.8722,  85.8577,
           2.5333]]), 'Target': tensor([0])}

TypeError                                 Traceback (most recent call last)
<ipython-input-52-7c8c9f3a38b7> in <module>
     20
     21         # Forward pass
---> 22         output = model(X_train)
     23         # Calculating loss
     24         loss = loss_func(output, Y_train)

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

<ipython-input-39-6b9a402c220d> in forward(self, x)
     10
     11     def forward(self, x):
---> 12         x = F.relu(self.conv1(x))
     13         x = F.max_pool2d(x, 2)
     14         x = F.relu(self.conv2(x))

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

~\anaconda3\lib\site-packages\torch\nn\modules\conv.py in forward(self, input)
    397
    398     def forward(self, input: Tensor) -> Tensor:
--> 399         return self._conv_forward(input, self.weight, self.bias)
    400
    401 class Conv3d(_ConvNd):

~\anaconda3\lib\site-packages\torch\nn\modules\conv.py in _conv_forward(self, input, weight, bias)
    393                             weight, bias, self.stride,
    394                             _pair(0), self.dilation, self.groups)
--> 395         return F.conv2d(input, weight, bias, self.stride,
    396                         self.padding, self.dilation, self.groups)
    397

TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not str

When I try to debug the code, the X_train and Y_train hold 'Data' & 'Target'.
I don't understand why the tensor values are not being taken by the enumerate (dataloader) in the for loop.
As otherwise, values are present in the dataloader. If each tensor row in the dataset has 'Data' and 'Target' prefixed before it, how do I remove that. Please suggest any solution.


title: What is this attribute syntax in Python?

body:
I am current following a tutorial in Pytorch and there is this expression:

grad_h[h < 0] = 0

How does this syntax work and what does it do?


title: Pytorch Fully Connected Feed Forward Network for a regression problem - Gives same result for all inputs

body:
I build a neural network model in Pytorch for a simple regression problem (w1x1+w2x2+w3x3 = y) where I generated 2000 records for training data with random values for x1,x2,x3 and W1=4, W2=6, W3=2. I created a test dataset of 20 records with just values for x1,x2,x3 and I was hoping to get the result for But, the model returns same value for all 20 input rows. I don't know where the issue is. Below is the code snippet.

inputs = df[['x1', 'x2', 'x3']]
target = df['y']
inputs = torch.tensor(inputs.values).float()
target = torch.tensor(target.values).float()

test_data = torch.tensor(test_data.values).float()

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

  def __init__(self):
    super(Net,self).__init__()

    hidden1 = 10
    hidden2 = 15

    self.fc1 = nn.Linear(3,hidden1)
    self.fc2 = nn.Linear(hidden1,hidden2)
    self.fc3 = nn.Linear(hidden2,1)

  def forward(self,x):
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

#instantiate the model

model = Net()
print(model)

criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(),lr=0.01)

model.train()

#epochs
epochs = 500

for x in range(epochs):
  #initialize the training loss to 0
  train_loss = 0
  #clear out gradients
  optimizer.zero_grad()

  #calculate the output
  output = model(inputs)

  #calculate loss
  loss = criterion(output,target)

  #backpropagate
  loss.backward()

  #update parameters
  optimizer.step()

  if ((x%5)==0):
    print('Training Loss after epoch {:2d} is {:2.6f}'.format(x,loss))

#set the model in evaluation mode
model.eval()

#Test the model on unseen data

test_output = model(test_data)

print(test_output)


title: Looking for an efficient implementation of pytorch matrix multiplcation to prevent large memory usage

body:
torch.matmul in pytorch has functions of broadcasting, which may consume too much memory.I am looking for efficient implementations to prevent excessive memory usage.

For example,the input tensor has size as
adj.size()==[1,3000,3000]
s.size()==torch.Size([235, 3000, 10])
s.transpose(1, 2).size()==torch.Size([235, 10, 3000])
The task is to calculate

link_loss = adj - torch.matmul(s, s.transpose(1, 2)) #
link_loss = torch.norm(link_loss, p=2)

The original code lies in the torch extension package torch_geometric.It is in the definition of the  function dense_diff_pool.
torch.matmul(s, s.transpose(1,2)) will consume excessive memory(my computer has only memory space around 2GB) ,raising the error:

  Traceback (most recent call last):

  File "", line 1, in
      torch.matmul(s, s.transpose(1, 2))

  RuntimeError: $ Torch: not enough memory: you tried to allocate 7GB.
  Buy new RAM! at ..\aten\src\TH\THGeneral.cpp:201

The original code by the author of the package contains torch.matmul(s, s.transpose(1, 2)).size()==[235,3000,3000] which is larger than 7GB.

My attemptation is that I tried to use a for iteration

batch_size=235
link_loss=torch.sqrt(torch.stack([torch.norm(adj - torch.matmul(s[i], s[i].transpose(0, 1)), p=2)**2 for i in range(batch_size)]).sum(dim=0))

This for loop is known to be slower than using broadcasting or other pytorch built-in functions.
Question:
Is there any faster implementation, better than using [ ... for ...].
I am a newbie on learning pytorch. Thanks.


title: How to find the index of torch tensor where the maximum value is occurring?

body:
Suppose I have the following tensor called a:

>>> a = torch.randn(10)
Out[32]:
tensor([-0.6936,  1.2862,  0.2096, -0.8629, -3.0095, -0.7957, -0.9336,  0.0622,
        -0.3510, -0.1518])

How can I find the index of this tensor where the maximum value is occurring (which in this case the position 1 (=1.2862))?

Thank you,


title: Trying to mask tensor with another tensor of same dimension getting "index 1 is out of bounds for dimension 0 with size 1"

body:
attn_weights = F.softmax(self.attn(torch.cat((input, hidden_cat), 2)), dim=2)
    attn_weights[mask] = float('-inf')
    attn_applied = torch.bmm(attn_weights.transpose(0,1),encoder_outputs.transpose(0,1)).transpose(0,1)
    attn_output = torch.cat((input, attn_applied), 2)

So I'm trying to set all the indexes in mask that are equal 1 to negative infinity, but that line

attn_weights[mask] = float('-inf')

keeps throwing this exception "index 1 is out of bounds for dimension 0 with size 1" not really sure what's going on attn_weights and mask both have the same dimension, which is 1 x 2048 x 40.


title: Understanding pytorch autograd

body:
I am trying to understand how pytorch autograd works. If I have functions y = 2x and z = y**2, if I do normal differentiation, I get dz/dx at x = 1 as 8 (dz/dx = dz/dy * dy/dx = 2y*2 = 2(2x)*2 = 8x). Or, z = (2x)**2 = 4x^2 and dz/dx = 8x, so at x = 1, it is 8.

If I do the same with pytorch autograd, I get 4

x = torch.ones(1,requires_grad=True)
y = 2*x
z = y**2
x.backward(z)
print(x.grad)

which prints

tensor([4.])

where am I going wrong?


title: Either too little or too many arguments for a nn.Sequential

body:
I am new to PyTorch, so please excuse my silly question.

I define a nn.Sequential in init of my Encoder object like this:

self.list_of_blocks = [EncoderBlock(n_features, n_heads, n_hidden, dropout) for _ in range(n_blocks)]
self.blocks = nn.Sequential(*self.list_of_blocks)

The forward of EncoderBlock looks like this

def forward(self, x, mask):

In the forward() of my Encoder, I try to do:

z0 = self.blocks(z0, mask)

I expect the nn.Sequential to pass these two arguments to individual blocks.

However, I get

TypeError: forward() takes 2 positional arguments but 3 were given

When I try:

z0 = self.blocks(z0)

I get (understandably):

TypeError: forward() takes 2 positional arguments but only 1 was given

When I do not use nn.Sequential and just execute one EncoderBlock after another, it works:

for i in range(self.n_blocks):
     z0 = self.list_of_blocks[i](z0, mask)

Question: What am I doing wrong and how do I use nn.Sequential correctly in this case?


title: Add More Metrics to Ray Tune Status Table (Python, PyTorch)

body:
When running tune.run() on a set of configs to search, is it possible to add more metrics columns (i.e. a, b, etc) to the status table being printed out?

tune.track.log(a=metric1, b=metric2)

will give the following table without columns for the metrics a and b:

== Status ==
Memory usage on this node: 22.1/125.8 GiB
Using FIFO scheduling algorithm.
Resources requested: 1/32 CPUs, 1/4 GPUs, 0.0/65.59 GiB heap, 0.0/22.13 GiB objects
Result logdir: /home/nyxynyx/ray_results/fooba
Number of trials: 4 (3 PENDING, 1 RUNNING)
+--------------+----------+-------+------+-----+
| Trial name   | status   | loc   |   lr |   x |
|--------------+----------+-------+------+-----|
| fooba_00000  | RUNNING  |       | 0.01 |   1 |
| fooba_00001  | PENDING  |       | 0.1  |   1 |
| fooba_00002  | PENDING  |       | 0.01 |   5 |
| fooba_00003  | PENDING  |       | 0.1  |   5 |
+--------------+----------+-------+------+-----+

How can we include a column for every metric that we pass to tune.track.log() other than mean_accuracy?

Using Python 3.7.3 and Ray 0.8.4


title: Cross-validation of neural network: How to treat the number of epochs?

body:
I'm implementing a pytorch neural network (regression) and want to identify the best network topology, optimizer etc.. I use cross validation, because I have x databases of measurements and I want to evaluate whether I can train a neural network with a subset of the x databases and apply the neural network to the unseen databases. Therefore, I also introduce a test database, which I doesn't use in the phase of the hyperparameter identification.
I am confused on how to treat the number of epochs in cross validation, e.g. I have a number of epochs = 100. There are two options:

The number of epochs is a hyperparameter to tune. In each epoch, the mean error across all cross validation iterations is determined. After models are trained with all network topologies, optimizers etc. the model with the smallest mean error is determined and has parameters like: -network topology: 1
-optimizer: SGD
-number of epochs: 54
To calculate the performance on the test set, a model is trained with exactly these parameters (number of epochs = 54) on the training and the validation data. Then it is applied and evaluated on the test set.
The number of epochs is NOT a hyperparameter to tune. Models are trained with all the network topologies, optimizers etc. For each model, the number of epochs, where the error is the smallest, is used. The models are compared and the best model can be determined with parameters like:
-network topology: 1
-optimizer: SGD
To calculate the performance on the test data, a “simple” training and validation split is used (e.g. 80-20). The model is trained with the above parameters and 100 epochs on the training and validation data. Finally, a model with a number of epochs yielding the smallest validation error, is evaluated on the test data.

Which option is the correct or the better one?


title: How do you delete a row using PyTorch?

body: I have been looking on how to delete a row in pytorch(torchtext.data.dataset.TabularDataset) but I'm not able to find anything.


title: Converting 1D CNN from Keras to Pytorch

body:
I am trying to make a switch from Tensorflow Keras to Pytorch.

The shape of my Input dataset is (-1,128,6)

I have the following model in keras:

serverModel = Sequential()
serverModel.add(Conv1D(196,  16, activation='relu',name = 'base1',input_shape=(128,6), padding='same'))
serverModel.add(MaxPool1D(pool_size=4, padding='same',name = 'base2'))
serverModel.add(Flatten(name = 'flatten'))
serverModel.add(Dense(1024, activation = 'relu'))
serverModel.add(Dropout(dropout_rate))
serverModel.add(Dense(activityCount, activation='softmax'))

In words, The convolution layer has 196, 16 by 1 filters. the inputs are tensor of length 128 over 6 dimensions.

I'm a bit confused with the documentation (https://pytorch.org/docs/stable/nn.html#conv1d)

what is the in in_channels, out_channels that I need to specify for the 1DCNN in PyTorch? A full conversion would be very appreciated!


title: Pytorch nn.BCELoss() getting TypeError: 'Tensor' object is not callable

body:
I am using adv_loss = torch.nn.BCELoss() to calculate loss for my network:
When I call the function the first time(first batch), I do not get an error. But, if I call the function 2nd time(2nd batch), I get the error 'Tensor' object is not callable.
I am training GAN, where output_of_nn_layers = D(img1)(output of discriminator network) and
label = torch.ones(batch, 1, device=device, dtype = torch.float32)
I am using adv_loss(BCELoss) in generator training as well, but error occurs in 2nd batch. So total times it is used in training 1st batch is 3 times(2 times for discriminator and 1 time for generator).
for i, (img1, img2) in enumerate(dataloader):
  #do some processing
  loss = adv_loss(output_of_nn_layers, label)
  #do some processing

I am getting error : TypeError: 'Tensor' object is not callable
Full error log:

TypeError                                 Traceback (most recent call last)

<ipython-input-52-bafb0a65d4d9> in <module>()
     73       val_real = D(dis_train_real)
     74       print("check4.2")
---> 75       d_real_loss = adv_loss(val_real, real_label)   #BCELoss
     76       print("check4.3")
     77       val_fake = D(dis_train_fake)

TypeError: 'Tensor' object is not callable

I found similar question on stackoerflow : Recalling function: Tensor 'object' is not callable
It gave solution that called function should not have variable name same as function name. Am i missing something here?
i also tried using nn.L1Loss() and got same error.
what does this error means? what can i do to remove it?


title: PyTorch loop through epoch then output final values for all epoch

body:
The code below calculates the MSE and MAE values but I have an issue where the values for MAE and MSE don't get store_MAE and store MSE after the end of each epoch. It appears to use the values of the last epoch only. Any idea what I need to do in the code to save the values for each epoch I hope this makes sense. Thanks for your help
    global_step = 0
best_test_error = 10000
MAE_for_all_epochs = []
MSE_for_all_epochs = []
for epoch in range(4):
    print("Epoch %d" % epoch)
    model.train()
    for images, paths in tqdm(loader_train):
        images = images.to(device)
        targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
        targets = targets.float().to(device)

        # forward pass:
        output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
        preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)

        # backward pass:
        loss = torch.mean((preds - targets)**2)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # logging:
        count_error = torch.abs(preds - targets).mean()
        writer.add_scalar('train_loss', loss.item(), global_step=global_step)
        writer.add_scalar('train_count_error', count_error.item(), global_step=global_step)

        print("Step %d, loss=%f, count error=%f" % (global_step,loss.item(),count_error.item()))

        global_step += 1

    mean_test_error = 0
    model.eval()
    for images, paths in tqdm(loader_test):
        images = images.to(device)
        targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
        targets = targets.float().to(device)

        # forward pass:
        output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
        preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)

        # logging:
        #error = torch.abs(preds - targets).sum().data
        #squared_error = ((preds - targets)*(preds - targets)).sum().data

        #runnning_mae += error
        #runnning_mse += squared_error

        loss = torch.mean((preds - targets)**2)
        count_error = torch.abs(preds - targets).mean()
        mean_test_error += count_error
        writer.add_scalar('test_loss', loss.item(), global_step=global_step)
        writer.add_scalar('test_count_error', count_error.item(), global_step=global_step)

        global_step += 1
        #store_MAE = 0
        #store_MSE = 0

        mean_test_error = mean_test_error / len(loader_test)
        #store_MAE += mean_test_error
        MAE_for_all_epochs = np.append(MAE_for_all_epochs, mean_test_error)

        mse = math.sqrt(loss / len(loader_test))
        #store_MSE +=mse
        MSE_for_all_epochs = np.append(MSE_for_all_epochs, mse)

        print("Test count error: %f" % mean_test_error)
        print("MSE: %f" % mse)

    if mean_test_error < best_test_error:
        best_test_error = mean_test_error
        torch.save({'state_dict':model.state_dict(),
                    'optimizer_state_dict':optimizer.state_dict(),
                    'globalStep':global_step,
                    'train_paths':dataset_train.files,
                    'test_paths':dataset_test.files},checkpoint_path)

print("MAE Total: %f" % store_MAE)
print("MSE Total: %f" % store_MSE)
model_mae= MAE_for_all_epochs / epoch
model_mse= MSE_for_all_epochs / epoch
print("Model MAE: %f" % model_mae)
print("Model MSE: %f" % model_mse)


title: Adding deep learning model at django

body:
ml: django app folder
test: pytorch python code folder --predict.py(model)
And I wanted to use procees function that i made at predcit.py at views.py
process function
def process():
for img_file in os.listdir('C:\\Users\\user\\Desktop\\model\\project\\ml\\test\\images'):

    img_path = os.path.join('images', img_file)

    predition = predict_breed_transfer(model, img_path)
    print(predition)
    print("image_file_name: {0}, \t predition breed: {1}\n\n".format(img_path, predition))

return (predition)

views.py at ml folder
from django.shortcuts import render
from .test.predict import *

def predict(request):
    pred=process()

    return render(request, "predict.html",{'pred':pred})

When i run the server at my local computer, it gets pred as none.
I don't understand why views.py doesn't run 'predict_breed_transfer'function...


title: AttributeError in creating module

body:
I am getting the following error.

  AttributeError: cannot assign module before Module.init() call

I have a class as follows.

class Classifier(nn.Module):

    def __init__(self, dictionary, embeddings_index, max_seq_length, args):
        self.embedding = EmbeddingLayer(len(dictionary), args.emsize, args.dropout)
        self.drop = nn.Dropout(args.dropout)

What I am doing wrong here? I am beginner in PyTorch, please help!


title: PyTorch is crashing the jupyter notebook, while calculating the loss using nn module?

body:
model = nn.Sequential(nn.Linear(784, 128),
                  nn.ReLU(),
                  nn.Linear(128, 64),
                  nn.ReLU(),
                  nn.Linear(64, 10))

criterion = nn.CrossEntropyLoss()

images, labels = next(iter(trainloader))
images = images.view(images.shape[0], -1)
logits = model(images)
loss = criterion(logits, labels)
print(loss)

The code is crashing at " loss = criterion( logits, labels) " line.
Dataset : MNIST handwritten digits.


title: Building dictionary with GoogleNews-vectors-negative300.bin returns ValueError: could not convert string to float

body:
When trying to load  GoogleNews-vectors-negative300.bin with pytorch Vector struct I am getting

  ValueError: could not convert string to float: b'\x00\x00\x94:\x00\x00k\xba\x00\x00\x

I have tried this post (@robodasha) but without success. My goal is to build a vocabulary with the loaded embedding using build_vocab Any suggestions?


title: How to use tensorboard debugger with pytorch?

body: I know pytorch starts to support tensorboard since the version 1.11. But I am wondering is it possible for us to use the debugger plugin for tensorboard with pytorch? I didn't find any information about this. If pytorch can also support tensorboard debugger, it would be extremely convenient and could save us a lot of time.


title: Is conv2d using random filters? Is that good for the NN?

body:
If I am using nn.Conv2d(in_dim, out_dim, kernel_size, bias=False) command in PyTorch, then the function creates as many filters as specified by out_dim. Are these filters random? Is it not be better to customize these filters? Does someone have a method for this?

Or is it better to use the random filter generated by this function?

Done in PyTorch, python.

Thanks


title: PyTorch CNN: Loss is unchanging

body:
I have tried researching a situation for my unchanging loss, and all the answers I found were specific to the code. I just started learning about CNNs and majority of the CNN is from an example and modified to fit the needs of my dataset. I am trying to classify types of ECGs (normal, atrial fibrillation, other, noisy). When I try to train the CNN the loss remains the same, I think this is because my CNN does not learn and only outputs zeros. So far I have tried changing the learning rate/loss function and have made no difference.

I am doing this on Google Colab so feel free to edit the code, and don't forget to change hardware acceleration under the runtime tab to GPU.

Code:

import os
import cv2
import numpy as np
from tqdm import tqdm
from scipy.io import loadmat
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import matplotlib.pyplot as plt

if torch.cuda.is_available():
  device = torch.device("cuda:0")
  print("Running on GPU -", device )
else:
  device = torch.device("cpu")
  print("Running on CPU -", device )

REBUILD_DATA = True # processing data

class ECG_DATA():
  ECG_LENGTH = 3000
  LABEL_SIZE = 485
  DATA = "physionet.org/files/challenge-2017/1.0.0/training/"
  NORMAL = "physionet.org/files/challenge-2017/1.0.0/training/RECORDS-normal"
  AF = "physionet.org/files/challenge-2017/1.0.0/training/RECORDS-af"
  OTHER = "physionet.org/files/challenge-2017/1.0.0/training/RECORDS-other"
  NOISY = "physionet.org/files/challenge-2017/1.0.0/training/RECORDS-noisy"
  LABELS = {NORMAL: 0, AF: 1, OTHER:2, NOISY: 3}
  trainingData = []
  dataCount = {NORMAL: 0, AF: 0, OTHER: 0, NOISY: 0}

  def make_training_data(self):
    for records in self.LABELS:
      with open(records) as label:
        for ecgFile in tqdm(label):
          ecg = loadmat(self.DATA+ecgFile[:-1]+".mat")["val"][0].tolist()

          if records == self.NOISY:
            #self.zero_padding(ecg)
            for x in range(self.ECG_LENGTH, len(ecg), self.ECG_LENGTH):
              if self.dataCount[records] <= self.LABEL_SIZE and x <= len(ecg):
                self.trainingData.append([np.array(ecg[x-self.ECG_LENGTH:x]), np.eye(len(self.LABELS))[self.LABELS[records]]])
                self.dataCount[records] += 1

          elif self.dataCount[records] <= self.LABEL_SIZE and self.ECG_LENGTH <= len(ecg):
            self.trainingData.append([np.array(ecg[:self.ECG_LENGTH]), np.eye(len(self.LABELS))[self.LABELS[records]]])
            self.dataCount[records] += 1

    print(self.dataCount)
    np.random.shuffle(self.trainingData)
    np.save("training_Data.npy", self.trainingData)

  def zero_padding(self, ecg):
    ecg += [0] * (self.ECG_LENGTH-(len(ecg)%self.ECG_LENGTH))
class Net(nn.Module):
    def __init__(self):
        super().__init__() # just run the init of parent class (nn.Module)
        self.conv1 = nn.Conv1d(1, 32, 5) # input is 1 image, 32 output channels, 5x5 kernel / window
        self.conv2 = nn.Conv1d(32, 64, 5) # input is 32, bc the first layer output 32. Then we say the output will be 64 channels, 5x5 kernel / window
        self.conv3 = nn.Conv1d(64, 128, 5)

        x = torch.randn(1,3000).view(-1,1,3000)
        self._to_linear = None
        self.convs(x)

        self.fc1 = nn.Linear(self._to_linear, 512) #flattening.
        self.fc2 = nn.Linear(512, 4) # 512 in, 2 out bc we're doing 2 classes (dog vs cat).

    def convs(self, x):
        x = F.max_pool1d(F.relu(self.conv1(x)), 1) # adjust shape of pooling?
        x = F.max_pool1d(F.relu(self.conv2(x)), 1) # x = F.max_pool1d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool1d(F.relu(self.conv3(x)), 1)

        if self._to_linear is None:
            self._to_linear = x[0].shape[0]*x[0].shape[1]
        return x

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1, self._to_linear)  # .view is reshape ... this flattens X before
        x = F.relu(self.fc1(x))
        x = self.fc2(x) # bc this is our output layer. No activation here.
        return F.softmax(x, dim=1)

net = Net().to(device)
print(net)

if REBUILD_DATA:
  ECG = ECG_DATA()
  ECG.make_training_data()

training_data = np.load("training_Data.npy", allow_pickle=True)
print(len(training_data))

optimizer = optim.Adam(net.parameters(), lr = 0.01)
loss_function = nn.MSELoss().to(device)

X = torch.Tensor([i[0] for i in training_data])
y = torch.Tensor([i[1] for i in training_data])

VAL_PCT = 0.1
val_size = int(len(X)*VAL_PCT)
print(val_size)

train_X = X[:-val_size]
train_y = y[:-val_size]

test_X = X[-val_size:]
test_y = y[-val_size:]
print(len(train_X), len(test_X))

BATCH_SIZE = 100
EPOCHS = 1
plot = []

for epoch in range(EPOCHS):
    for i in tqdm(range(0, len(train_X), BATCH_SIZE)): # from 0, to the len of x, stepping BATCH_SIZE at a time. [:50] ..for now just to dev
        #print(f"{i}:{i+BATCH_SIZE}")
        batch_X = train_X[i:i+BATCH_SIZE].view(-1,1,3000).to(device)
        batch_y = train_y[i:i+BATCH_SIZE].to(device)

        net.zero_grad()

        outputs = net(batch_X)
        loss = loss_function(outputs, batch_y)
        loss.backward()
        optimizer.step()    # Does the update

    plot.append([epoch, float(loss)])
    print(f"\nEpoch: {epoch}. Loss: {loss}")

plot = list(map(list, zip(*plot)))
plt.plot(plot[0], plot[1])


title: weird problem with Pytorch's mse_loss function

body:
Traceback (most recent call last):
  File "c:/Users/levin/Desktop/programming/nn.py", line 208, in <module>
    agent.train(BATCHSIZE)
  File "c:/Users/levin/Desktop/programming/nn.py", line 147, in train
    output = F.mse_loss(prediction, target)
  File "C:\Users\levin\Anaconda3\lib\site-packages\torch\nn\functional.py", line 2203, in mse_loss
    if not (target.size() == input.size()):
AttributeError: 'NoneType' object has no attribute 'size'

This above is the Error that I'm constantly getting and I really don't know how to fix it.

This some code that might be important

    def train(self, BATCHSIZE):
        trainsample = random.sample(self.memory, BATCHSIZE)

        for state, action, reward, new_state, gameovertemp in trainsample:
            if gameovertemp:
                target = torch.tensor(reward).grad_fn
            else:
                target = reward + self.gamma * torch.max(self.dqn.forward(new_state))

            self.dqn.zero_grad()
            prediction = torch.max(self.dqn.forward(state))
            #print(prediction, "prediction")
            #print(target, "target")
            output = F.mse_loss(prediction, target)
            output.backward()
            self.optimizer.step()


title: Neural Network Input scaling

body:
I trained a simple fully connected network on CIFAR-10 dataset:

import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision
import torchvision.transforms as transforms

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(3*32*32, 300, bias=False)
        self.fc2 = nn.Linear(300, 10, bias=False)

    def forward(self, x):
        x = x.reshape(250, -1)
        self.x2 = F.relu(self.fc1(x))
        x = self.fc2(self.x2)
        return x

def train():
    # The output of torchvision datasets are PILImage images of range [0, 1].
    transform = transforms.Compose([transforms.ToTensor()])
    trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=250, shuffle=True, num_workers=4)
    testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
    testloader = torch.utils.data.DataLoader(testset, batch_size=args.bs, shuffle=False, num_workers=4)

    net = Net()
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(net.parameters(), lr=0.02, momentum=0.9, weight_decay=0.0001)

    for epoch in range(20):
        correct = 0
        total = 0
        for data in trainloader:
            inputs, labels = data
            outputs = net(inputs)
            loss = criterion(outputs, labels)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
        acc = 100. * correct / total

This network gets to ~50% test accuracy with the parameters specified, after 20 epochs.
Note that I didn't do any whitening of the inputs (no per channel mean subtraction)

Next I scaled up the model inputs by 255, by replacing outputs = net(inputs) with outputs = net(inputs*255). After this change, the network no longer converges. I looked at the gradients and they seem to grow explosively after just a few iterations, leading to all model outputs being zero. I'd like to understand why this is happening.

Also, I tried scaling down the learning rate by 255. This helps, but the network only gets to ~43% accuracy. Again, I don't understand why this helps, and more importantly why the accuracy is still degraded compared to the original settings.

EDIT: forgot to mention that I don't use biases in this network.

EDIT2: I can recover the original accuracy if I scale down the initial weights in both layers by 255 (in addition to scaling down the learning rate). I also tried to scale down the initial weights only in the first layer, but the network had trouble learning (even when I did scale down the learning rate in both layers). Then I tried scaling down the learning rate only in the first layer - this also didn't help. Finally I tried reducing learning rate in both layer even  more (by 255*255) and this suddenly worked. This does not make sense to me - scaling down the initial weights by the same factor the inputs have been scaled up should have completely eliminated any difference from the original network, the input to the second layer is identical. At that point the learning rate should be scaled down in the first layer only, but in practice both layers need significantly lower learning rate...


title: Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu

body:
here is my code for lstm network, I instantiated it and passed to Cuda device but still getting the error that hidden and inputs are not in same device
class LSTM_net(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
    super(LSTM_net, self).__init__()
    self.hidden_size = hidden_size
    self.lstm_cell = nn.LSTM(input_size, hidden_size)
    self.h2o = nn.Linear(hidden_size, output_size)
    self.softmax = nn.LogSoftmax(dim=1)

def forward(self, input, hidden_0=None, hidden_1=None, hidden_2=None):
    input=resnet(input)
    input=input.unsqueeze(0)
    out_0, hidden_0 = self.lstm_cell(input, hidden_0)
    out_1, hidden_1 = self.lstm_cell(out_0+input, hidden_1)
    out_2, hidden_2 = self.lstm_cell(out_1+input, hidden_2)
    output = self.h2o(hidden_2[0].view(-1, self.hidden_size))
    output = self.softmax(output)
    return output,hidden_0,hidden_1, hidden_2

def init_hidden(self, batch_size = 1):
    return (torch.zeros(1, batch_size, self.hidden_size), torch.zeros(1, batch_size, self.hidden_size))

net1=LSTM_net(input_size=1000,hidden_size=1000, output_size=100)

net1=net1.to(device)

pic of connections that I want to make, plz guide me to implement it
click here for an image of error massege


title: How to generalize Conv2d to MaxPool2d to ConvTranspose2d?

body:
I have generalized a PyTorch autoencoder implementation. One major strategy I'm using is a rescaling of a  sequence which indicates nodes per layer. This way I can run experiments with various network sizes.
For the sake of an MRE, I will provide the original, non-general code in a case which works, and a case which does not. I am searching for the relationship between Conv2d, MaxPool2d and ConvTranspose2d.
# imports
import torch
import numpy as np
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torchvision import datasets
from torch.utils.data import DataLoader
import torchvision.transforms as transforms

# Original autoencoder class.
class autoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        # encoder layers
        self.enc1 = nn.Conv2d(1, 64, kernel_size=3, padding=1)
        self.enc2 = nn.Conv2d(64, 32, kernel_size=3, padding=1)
        self.enc3 = nn.Conv2d(32, 16, kernel_size=3, padding=1)
        #######################################################
        # The relationship here is beyond me.
        self.enc4 = nn.Conv2d(16, 8, kernel_size=3, padding=1)
        self.pool = nn.MaxPool2d(2, 2)
        # decoder layers
        self.dec1 = nn.ConvTranspose2d(8, 8, kernel_size=3, stride=2)
        #######################################################
        self.dec2 = nn.ConvTranspose2d(8, 16, kernel_size=3, stride=2)
        self.dec3 = nn.ConvTranspose2d(16, 32, kernel_size=2, stride=2)
        self.dec4 = nn.ConvTranspose2d(32, 64, kernel_size=2, stride=2)
        self.out = nn.Conv2d(64, 1, kernel_size=3, padding=1)

    def forward(self, x):
        # encode
        x = F.relu(self.enc1(x))
        x = self.pool(x)
        x = F.relu(self.enc2(x))
        x = self.pool(x)
        x = F.relu(self.enc3(x))
        x = self.pool(x)
        x = F.relu(self.enc4(x))
        x = self.pool(x)
        # decode
        x = F.relu(self.dec1(x))
        x = F.relu(self.dec2(x))
        x = F.relu(self.dec3(x))
        x = F.relu(self.dec4(x))
        x = F.sigmoid(self.out(x))
        return x

def train(network, trainloader):
    train_loss = list()
    for epoch in range(10):
        running_loss = 0.0
        for data in trainloader:
            img, _ = data
            img_noisy = img + 0.5 * torch.randn(img.shape)
            img_noisy = np.clip(img_noisy, 0., 1.)
            img_noisy = img_noisy.to(device)
            optimizer.zero_grad()
            outputs = network(img_noisy)

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
trainset = datasets.FashionMNIST(
    root='./data',
    train=True,
    download=True,
    transform=transform
)
testset = datasets.FashionMNIST(
    root='./data',
    train=False,
    download=True,
    transform=transform
)
trainloader = DataLoader(
    trainset,
    batch_size=16,
    shuffle=True
)
testloader = DataLoader(
    testset,
    batch_size=16,
    shuffle=True
)

net = autoencoder()
device = 'cpu'
net.to(device)
criterion = nn.MSELoss()
optimizer = optim.Adam(net.parameters(), lr = 1e-3)
train_loss = train(net, trainloader)

# MRE for the autoencoder class I generalize.
# I have only added one more layer, and not scaled
# the node per layer sequence, but expanded to 128.
class autoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        # encoder layers
        self.enc0 = nn.Conv2d(1, 128,  kernel_size=3, padding=1)
        self.enc1 = nn.Conv2d(128, 64, kernel_size=3, padding=1)
        self.enc2 = nn.Conv2d(64, 32, kernel_size=3, padding=1)
        self.enc3 = nn.Conv2d(32, 16, kernel_size=3, padding=1)
        self.enc4 = nn.Conv2d(16, 8, kernel_size=3, padding=1)
        self.pool = nn.MaxPool2d(2, 2)
        # decoder layers
        self.dec1 = nn.ConvTranspose2d(8, 8, kernel_size=3, stride=2)
        self.dec2 = nn.ConvTranspose2d(8, 16, kernel_size=3, stride=2)
        self.dec3 = nn.ConvTranspose2d(16, 32, kernel_size=2, stride=2)
        self.dec4 = nn.ConvTranspose2d(32, 64, kernel_size=2, stride=2)
        self.dec5 = nn.ConvTranspose2d(64, 128, kernel_size=2, stride=2)
        self.out = nn.Conv2d(128, 1, kernel_size=3, padding=1)

    def forward(self, x):
        # encode
        x = F.relu(self.enc0(x))
        x = self.pool(x)
        x = F.relu(self.enc1(x))
        x = self.pool(x)
        x = F.relu(self.enc2(x))
        x = self.pool(x)
        x = F.relu(self.enc3(x))
        x = self.pool(x)
        x = F.relu(self.enc4(x))
        x = self.pool(x)
        # decode
        x = F.relu(self.dec1(x))
        x = F.relu(self.dec2(x))
        x = F.relu(self.dec3(x))
        x = F.relu(self.dec4(x))
        x = F.relu(self.dec5(x))
        x = F.sigmoid(self.out(x))
        return x

def train(network, trainloader):
    train_loss = list()
    for epoch in range(10):
        running_loss = 0.0
        for data in trainloader:
            img, _ = data
            img_noisy = img + 0.5 * torch.randn(img.shape)
            img_noisy = np.clip(img_noisy, 0., 1.)
            img_noisy = img_noisy.to(device)
            optimizer.zero_grad()
            outputs = network(img_noisy)

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
trainset = datasets.FashionMNIST(
    root='./data',
    train=True,
    download=True,
    transform=transform
)
testset = datasets.FashionMNIST(
    root='./data',
    train=False,
    download=True,
    transform=transform
)
trainloader = DataLoader(
    trainset,
    batch_size=16,
    shuffle=True
)
testloader = DataLoader(
    testset,
    batch_size=16,
    shuffle=True
)

net = autoencoder()
device = 'cpu'
net.to(device)
criterion = nn.MSELoss()
optimizer = optim.Adam(net.parameters(), lr = 1e-3)
train_loss = train(net, trainloader)

# Error
Traceback (most recent call last):
  File "C:\Users\User\Desktop\ml_paper\mre.py", line 89, in <module>
    train_loss = train(net, trainloader)
  File "C:\Users\User\Desktop\ml_paper\mre.py", line 58, in train
    outputs = network(img_noisy)
  File "C:\Users\User\Desktop\ml_paper\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\User\Desktop\ml_paper\mre.py", line 38, in forward
    x = self.pool(x)
  File "C:\Users\User\Desktop\ml_paper\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\User\Desktop\ml_paper\lib\site-packages\torch\nn\modules\pooling.py", line 159, in forward
    self.return_indices)
  File "C:\Users\User\Desktop\ml_paper\lib\site-packages\torch\_jit_internal.py", line 247, in fn
    return if_false(*args, **kwargs)
  File "C:\Users\User\Desktop\ml_paper\lib\site-packages\torch\nn\functional.py", line 576, in _max_pool2d
    input, kernel_size, stride, padding, dilation, ceil_mode)
RuntimeError: Given input size: (8x1x1). Calculated output size: (8x0x0). Output size is too small

The issue may be how I infer the kernel sizes in the decoding layers.
import math
import torch.nn as nn
# infer kernel sizes for decoding layers.
series = [1, 8, 16, 32, 64, 128]
kernel_set = 3
tmp = nn.ModuleList()
for layer in range(1, len(series)):
    if layer == 1:
        tmp.append(nn.ConvTranspose2d(series[layer], series[layer], kernel_set, stride = 2))
    else:
        if layer > math.floor(len(series)/2):
            tmp.append(nn.ConvTranspose2d(series[layer - 1], series[layer], kernel_set - 1, stride = 2))
        else:
            tmp.append(nn.ConvTranspose2d(series[layer - 1], series[layer], kernel_set, stride = 2))
print(tmp)
ModuleList(
  (0): ConvTranspose2d(8, 8, kernel_size=(3, 3), stride=(2, 2))
  (1): ConvTranspose2d(8, 16, kernel_size=(3, 3), stride=(2, 2))
  (2): ConvTranspose2d(16, 32, kernel_size=(3, 3), stride=(2, 2))
  (3): ConvTranspose2d(32, 64, kernel_size=(2, 2), stride=(2, 2))
  (4): ConvTranspose2d(64, 128, kernel_size=(2, 2), stride=(2, 2))
)


title: Problem loading pre-trained FinBERT model

body:
I am trying to load a pre-trained FinBERT model in python but am getting an error simply trying to load it. I am following the GitHub docs: https://github.com/ProsusAI/finBERT
Following the tutorial, I have created a directory called bert, downloaded the model pytorch_model.bin and config.json file and dropped them in the folder.
I try and call the model using the following code:
import os
os.chdir(r'bert')

from transformers import BertTokenizer, BertForSequenceClassification
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

model = BertForSequenceClassification.from_pretrained(r'bert\pytorch_model.bin',
                                                      config = r'bert\config.json')

The error trying to load is as follows:
RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
    size mismatch for classifier.weight: copying a param with shape torch.Size([3, 768]) from checkpoint, the shape in current model is torch.Size([2, 768]).
    size mismatch for classifier.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).

My knowledge of this is very basic and if anyone understands what is going on here it would be greatly appreciated.
Thanks


title: ValueError: not enough values to unpack (expected 3, got 2). TransformerEncoder

body:
Trying out different models and for some reason I keep getting this error after trying to change everything I can. Can somebody explain where my fault is here or show me how to fix this issue?
I am doing this on mnist_1d dataset.
Model code:
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        from torch.nn import TransformerEncoder, TransformerEncoderLayer
        encoder_layers = TransformerEncoderLayer(40, 8)
        self.transformer_encoder = TransformerEncoder(encoder_layers, 6)
        self.decoder = nn.Linear(40, 10)

    def forward(self, x):
        x = self.transformer_encoder(x)
        x = self.decoder(x)
        return F.log_softmax(x, dim=1)

Training code:
import torch.optim as optim

loss_function = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=0.001)

for epoch in range(3):
    for data in trainset:
        X, y = data
        print(X.view(-1, 40).shape)
        net.zero_grad()
        output = net(X.view(-1,40))
        loss = F.nll_loss(output, y)
        loss.backward()
        optimizer.step()

Error message:
torch.Size([1, 40])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-21-53cd1214bd32> in <module>()
      5         net.zero_grad()
----> 6         output = net(X.view(-1,40))
      7         loss = F.nll_loss(output, y)

<ipython-input-13-fd44a15921d2> in forward(self, x)
      8
      9     def forward(self, x):
---> 10         x = self.transformer_encoder(x)
     11         x = self.decoder(x)
     12         return F.log_softmax(x, dim=1)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/activation.py in forward(self, query, key, value, key_padding_mask, need_weights, attn_mask)
    983                 training=self.training,
    984                 key_padding_mask=key_padding_mask, need_weights=need_weights,
--> 985                 attn_mask=attn_mask)
    986
    987

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in multi_head_attention_forward(query, key, value, embed_dim_to_check, num_heads, in_proj_weight, in_proj_bias, bias_k, bias_v, add_zero_attn, dropout_p, out_proj_weight, out_proj_bias, training, key_padding_mask, need_weights, attn_mask, use_separate_proj_weight, q_proj_weight, k_proj_weight, v_proj_weight, static_k, static_v)
   4132                 q_proj_weight=q_proj_weight, k_proj_weight=k_proj_weight,
   4133                 v_proj_weight=v_proj_weight, static_k=static_k, static_v=static_v)
-> 4134     tgt_len, bsz, embed_dim = query.size()
   4135     assert embed_dim == embed_dim_to_check
   4136     # allow MHA to have different sizes for the feature dimension

ValueError: not enough values to unpack (expected 3, got 2)

Could not post the entire error message


title: Getting error while building PyTorch from source on Linux

body:
I have a problem with building PyTorch from source. I use Anaconda Python 3.6. I installed all the dependencies using conda and issued "python setup.py install" command to build it. It builds all files successfully but then it fails" at line 286 of caffe2/contrib/cuda-convnet2"

Pls help! Thank you!


title: How to transform BERT's network output to readable text?

body:
I am trying to understand, how to use BERT for QnA and found a tutorial on how to start on PyTorch (here). Now I would like to use these snippets to get started, but i do not understand how to project the output back on the example text.

text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"

(...)

# Predict the start and end positions logits
with torch.no_grad():
    start_logits, end_logits = questionAnswering_model(tokens_tensor, segments_tensors)

# Or get the total loss
start_positions, end_positions = torch.tensor([12]), torch.tensor([14])

multiple_choice_loss = questionAnswering_model(
                         tokens_tensor,
                         segments_tensors,
                         start_positions=start_positions,
                         end_positions=end_positions)

  start_logits (shape: [1, 16]):tensor([[ 0.0196,  0.1578,  0.0848,  0.1333, -0.4113, -0.0241, -0.1060, -0.3649,
            0.0955, -0.4644, -0.1548,  0.0967, -0.0659,  0.1055, -0.1488, -0.3649]])

  end_logits (shape: [1, 16]):tensor([[ 0.1828, -0.2691, -0.0594, -0.1618,  0.0441, -0.2574, -0.2883,  0.2526,
           -0.0551, -0.0051, -0.1572, -0.1670, -0.1219, -0.1831, -0.4463,  0.2526]])

If my assumption is correct start_logits and end_logits need to be projected back on text, but how do i compute this?

Additionally do you have any resources/guides/tutorials you could recommend to continue further into QnA (except google-research/bert github and the paper for bert)?

Thank you in advance.


title: How, in pytorch, do I set the second gpu as the default within a juptyer notebook?

body: I have two gpus, and want to open a second juptyer notebook and ensure everything within it runs only on the second gpu rather than the first.  Ideally I'd like to do this by running a cell at the start rather than passing device=1 in multiple places.


title: Fine-tuning Resnet using pre-convoluted features

body:
As we know that convolution layers are expensive to calculate , I would like to compute the output of the convolution layers once and use them to train the fully connected layer of my Resnet, in order to speed up the process.

In the case of a VGG model, we can compute the output of the first convolutional part as follows

x = model_vgg.features(inputs)

But how can I do to extract features from a Resnet?

Thanks in advance


title: torch.cuda.is_available() returning false with Ubuntu 16.04

body:
I have the following configurations

nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17

nvidia-smi

CUDA Toolkit (from Anaconda)

  cudatoolkit               10.2.89              hfd86e86_1    anaconda

PyTorch (from Anaconda)

  pytorch                   1.5.0          py3.7_cuda10.2.89_cudnn7.6.5_0    pytorch

But I am getting torch.cuda.is_available() -> False

Could anyone please tell which component should I upgrade or downgrade in order to bring up the CUDA?


title: Weights are not updated in Pytorch nn.Embedding

body:
I loaded the PyTorch's nn.Embedding module with a pre-trained embedding matrix. I set it to trainable as follows.

self.embedding_layer = nn.Embedding(self.vocab_size, self.embed_size, paddding_idx=self.padding_idx)
self.embedding_layer.weight = nn.Parameter(self.embedding)
self.embedding_layer.weight.requires_grad = True

I processed this output using Bidirectional Gated Recurrent Units network.
After the model is trained, I checked whether the weights of nn.Embedding are updated. The weights were not updated. model.embedding_layer.weight and 'self.embedding' weights are the same.
I checked the gradients of model.embedding_layer. They are all zeros.

Could you please help me? Thank you.


title: How to log custom Pytorch model with Mlflow?

body:
I have been using Presumm https://github.com/nlpyang/PreSumm for text summarization.

However, in src/train_abstractive.py, the model learner trainer is not a torch.nn.Module. However, the input AbsSummarizer is an extension of the torch.nn.Module class.

I want to use mlflow.pytorch.log_model to save the model as a native pytorch model. But trainer is not a torch.nn.Module. How do I go about this?


title: Speeding up tensor concatenation

body:
I need to concatenate a long list of small tensors. Each small tensor is a slice of a given (quite simple) constant matrix. Here is the code:
max_node, counter = 0, 0
batch_size, n_days = (1000, 10)
n_interactions_in = torch.randint(low=100,high=200,size=(batch_size,n_days), dtype=torch.long)
max_interactions = n_interactions_in.max()
delay_table = torch.arange(n_days, device=device, dtype=torch.float).expand([max_interactions, n_days]).t().contiguous()
delay_table = n_days - delay_table - 1
edge_delay_buf = []
for b in range(batch_size):
     delay_vec = [delay_table[d, :n_interactions_in[b, d]] for d in range(n_days)]
     edge_delay_buf.append(torch.cat(delay_vec))
res = torch.cat(edge_delay_buf)

This takes a lot of time. Is there a way to effeciently parrallize the creation of each element in the edge_delay_buf?
I have tried multiple variants, such as replacing the for loop with a list concatenation, where the result is a list of lists, then flattening the list and applying torch.cat on the flattened list. However, it didn’t improve by much. For some reason the slicing operation takes too long.
Is there a way to make the slicing faster? Is there a way to make the loop more efficient / parallel?
Note: while I'm using torch in this example, I can also use numpy.
Note 2: I apologize for a duplicate post in a different forum.


title: Why does regularization in pytorch and scratch code does not match and what is the formula used for regularization in pytorch?

body:
I have been trying to do L2 regularization on a binary classification model in PyTorch but when I match the results of PyTorch and scratch code it doesn't match,
Pytorch code:
class LogisticRegression(nn.Module):
  def __init__(self,n_input_features):
    super(LogisticRegression,self).__init__()
    self.linear=nn.Linear(4,1)
    self.linear.weight.data.fill_(0.0)
    self.linear.bias.data.fill_(0.0)

  def forward(self,x):
    y_predicted=torch.sigmoid(self.linear(x))
    return y_predicted

model=LogisticRegression(4)

criterion=nn.BCELoss()
optimizer=torch.optim.SGD(model.parameters(),lr=0.05,weight_decay=0.1)
dataset=Data()
train_data=DataLoader(dataset=dataset,batch_size=1096,shuffle=False)

num_epochs=1000
for epoch in range(num_epochs):
  for x,y in train_data:
    y_pred=model(x)
    loss=criterion(y_pred,y)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Scratch Code:
def sigmoid(z):
    s = 1/(1+ np.exp(-z))
    return s

def yinfer(X, beta):
  return sigmoid(beta[0] + np.dot(X,beta[1:]))

def cost(X, Y, beta, lam):
    sum = 0
    sum1 = 0
    n = len(beta)
    m = len(Y)
    for i in range(m):
        sum = sum + Y[i]*(np.log( yinfer(X[i],beta)))+ (1 -Y[i])*np.log(1-yinfer(X[i],beta))
    for i in range(0, n):
        sum1 = sum1 + beta[i]**2

    return  (-sum + (lam/2) * sum1)/(1.0*m)

def pred(X,beta):
  if ( yinfer(X, beta) > 0.5):
    ypred = 1
  else :
    ypred = 0
  return ypred

beta = np.zeros(5)
iterations = 1000
arr_cost = np.zeros((iterations,4))
print(beta)
n = len(Y_train)
for i in range(iterations):
    Y_prediction_train=np.zeros(len(Y_train))
    Y_prediction_test=np.zeros(len(Y_test))

    for l in range(len(Y_train)):
        Y_prediction_train[l]=pred(X[l,:],beta)

    for l in range(len(Y_test)):
        Y_prediction_test[l]=pred(X_test[l,:],beta)

    train_acc = format(100 - np.mean(np.abs(Y_prediction_train - Y_train)) * 100)
    test_acc = 100 - np.mean(np.abs(Y_prediction_test - Y_test)) * 100
    arr_cost[i,:] = [i,cost(X,Y_train,beta,lam),train_acc,test_acc]
    temp_beta = np.zeros(len(beta))

    ''' main code from below '''

    for j in range(n):
        temp_beta[0] = temp_beta[0] + yinfer(X[j,:], beta) - Y_train[j]
        temp_beta[1:] = temp_beta[1:] + (yinfer(X[j,:], beta) - Y_train[j])*X[j,:]

    for k in range(0, len(beta)):
        temp_beta[k] = temp_beta[k] +  lam * beta[k]  #regularization here

    temp_beta= temp_beta / (1.0*n)

    beta = beta - alpha*temp_beta

graph of the losses
graph of training accuracy
graph of testing accuracy
Can someone please tell me why this is happening?
L2 value=0.1


title: Invalid pointer error whily running python in C++ using pybind11 and pytorch

body:
While running the following python code in C++ using pybind11, pytorch 1.6.0, I get "Invalid Pointer" error. In python, the code runs successfully without any error. Whats the reason? How can I solve this problem?
import torch
import torch.nn.functional as F
import numpy as np
import cv2
import torchvision
import eval_widerface
import torchvision_model

def resize(image, size):
    image = F.interpolate(image.unsqueeze(0), size=size, mode="nearest").squeeze(0)
    return image

# define constants
model_path = '/path/to/model.pt'
image_path = '/path/to/image_pad.jpg'
scale = 1.0 #Image resize scale (2 for half size)
font = cv2.FONT_HERSHEY_SIMPLEX

MIN_SCORE = 0.9
image_bgr = cv2.imread(image_path)
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)#skimage.io.imread(args.image_path)
cv2.imshow("input image",image_bgr)
cv2.waitKey()
cv2.destroyAllWindows()

# load pre-trained model
return_layers = {'layer2':1,'layer3':2,'layer4':3}
RetinaFace = torchvision_model.create_retinaface(return_layers)

print('RetinaFace.state_dict().')
retina_dict = RetinaFace.state_dict()

the following function generates error.
def create_retinaface(return_layers,backbone_name='resnet50',anchors_num=3,pretrained=True):
    print('In create_retinaface.')
    print(resnet.__dict__)
    backbone = resnet.__dict__[backbone_name](pretrained=pretrained)
    print('backbone.')
    # freeze layer1
    for name, parameter in backbone.named_parameters():
        print('freeze layer 1.');
        # if 'layer2' not in name and 'layer3' not in name and 'layer4' not in name:
        #     parameter.requires_grad_(False)
        if name == 'conv1.weight':
            # print('freeze first conv layer...')
            parameter.requires_grad_(False)
model = RetinaFace(backbone,return_layers,anchor_nums=3)

return model

The statement backbone = resnet.__dict__ [backbone_name](pretrained=pretrained) generated error that looks like
*** Error in `./p': munmap_chunk(): invalid pointer: 0x00007f4461866db0 ***
======= Backtrace: =========
/usr/lib64/libc.so.6(+0x7f3e4)[0x7f44736b43e4]
/usr/local/lib64/libopencv_gapi.so.4.1(_ZNSt10_HashtableISsSsSaISsENSt8__detail9_IdentityESt8equal_toISsESt4hashISsENS1_18_Mod_range_hashingENS1_20_Default_ranged_hashENS1_20_Prime_rehash_policyENS1_17_Hashtable_traitsILb1ELb1ELb1EEEE21_M_insert_unique_nodeEmmPNS1_10_Hash_nodeISsLb1EEE+0xc9)[0x7f4483dee1a9]
/home/20face/.virtualenvs/torch/lib64/python3.6/site-packages/torch/lib/libtorch_python.so(+0x4403b5)[0x7f4460bb73b5]
/home/20face/.virtualenvs/torch/lib64/python3.6/site-packages/torch/lib/libtorch_python.so(+0x44570a)[0x7f4460bbc70a]
/home/20face/.virtualenvs/torch/lib64/python3.6/site-packages/torch/lib/libtorch_python.so(+0x275b20)[0x7f44609ecb20]
/usr/lib64/libpython3.6m.so.1.0(_PyCFunction_FastCallDict+0x147)[0x7f4474307167]
/usr/lib64/libpython3.6m.so.1.0(+0x1507df)[0x7f44743727df]
/usr/lib64/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x3a7)[0x7f44743670f7]
/usr/lib64/libpython3.6m.so.1.0(+0x1505ca)[0x7f44743725ca]
/usr/lib64/libpython3.6m.so.1.0(+0x150903)[0x7f4474372903]
/usr/lib64/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x3a7)[0x7f44743670f7]
/usr/lib64/libpython3.6m.so.1.0(+0x14fb69)[0x7f4474371b69]
/usr/lib64/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x24f)[0x7f44743739ff]
/usr/lib64/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x10e)[0x7f44742ca1de]
/usr/lib64/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0x61)[0x7f44742ca2f1]
/usr/lib64/libpython3.6m.so.1.0(PyObject_Call+0x43)[0x7f44742c9f63]
/usr/lib64/libpython3.6m.so.1.0(+0xfa7e5)[0x7f447431c7e5]
/usr/lib64/libpython3.6m.so.1.0(+0xf71e2)[0x7f44743191e2]
/usr/lib64/libpython3.6m.so.1.0(PyObject_Call+0x43)[0x7f44742c9f63]
/usr/lib64/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x2067)[0x7f4474368db7]
/usr/lib64/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x24f)[0x7f4474372c9f]


title: how to load a (multivariate) timeseries into a neural network /LSTM

body:
How can I load a time series such as:

into a neural network (LSTM).
So far, I have seen approaches where this matrix is transposed (https://github.com/curiousily/Getting-Things-Done-with-Pytorch/blob/master/06.time-series-anomaly-detection-ecg.ipynb) (hours as columns, devices as rows). Others create a custom dataloader: http://www.feeny.org/custom-pytorch-dataset-class-for-timeseries-sequence-windows/ and manually create windows.
Why is there no more native way for the network to directly work on this raw input and learn the patterns / periodicity / anomalies?
How can this multivariate time-series of multiple devices be loaded natively into (pytorch, tensorflow) so that a resulting LSTM would properly learn:

the state of an individual time-series (at least within some window, must not necessarily be the whole potentially infinite time-series)

but also consider information of multiple series /devices / windows when performing a prediction
import pandas as pd
  from pandas import Timestamp
  df = pd.DataFrame({'hour': {0: Timestamp('2020-01-01 00:00:00'), 1: Timestamp('2020-01-01 00:00:00'), 2: Timestamp('2020-01-01 00:00:00'), 3: Timestamp('2020-01-01 00:00:00'), 4: Timestamp('2020-01-01 00:00:00'), 5: Timestamp('2020-01-01 01:00:00'), 6: Timestamp('2020-01-01 01:00:00'), 7: Timestamp('2020-01-01 01:00:00'), 8: Timestamp('2020-01-01 01:00:00'), 9: Timestamp('2020-01-01 01:00:00')}, 'metrik_0': {0: 2.020883621337143, 1: 2.808770093182167, 2: 2.5267618429653402, 3: 3.2709845883575346, 4: 3.7984105853602235, 5: 4.0385160093937795, 6: 4.643267594258785, 7: 1.3012379179114388, 8: 3.509304898336378, 9: 2.8664748765561208}, 'metrik_1': {0: 4.580434685779621, 1: 2.933188328317023, 2: 3.999229120882797, 3: 2.9099857745449706, 4: 4.6302055552849, 5: 4.012670194672169, 6: 3.697352153313931, 7: 4.855210603371005, 8: 2.2197913449032254, 9: 2.393605868973481}, 'metrik_2': {0: 3.680527279150989, 1: 2.511065648719921, 2: 3.8350007982479113, 3: 2.4063786290320333, 4: 3.231433617897482, 5: 3.8505378854180115, 6: 5.359150077287063, 7: 2.8966469424805386, 8: 4.554080028058399, 9: 3.3319064764061914}, 'cohort_id': {0: 1, 1: 2, 2: 1, 3: 2, 4: 2, 5: 1, 6: 2, 7: 2, 8: 1, 9: 2}, 'device_id': {0: 1, 1: 3, 2: 4, 3: 2, 4: 5, 5: 4, 6: 3, 7: 2, 8: 1, 9: 5}})


title: How to solve the NVIDIA driver?

body:
C:\Users\ASUS\anaconda3\lib\site-packages\torch\cuda_init_.py:52: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 9010). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at  ..\c10\cuda\CUDAFunctions.cpp:100.)
return torch._C._cuda_getDeviceCount() > 0enter image description here
My system:
Windows 10
Nvidia GeForce 940 MX


title: Pytorch DataLoader - working with individual batches

body:
I am building an LSTM for part-of-speech tagging, and am currently using this LSTM as reference to understand how to use the Pytorch DataLoader. However, I'm confused as to how to work with individual batches (official Pytorch documentation doesn't shed any light for me). More specifically I need to pad my sentences so that all the sentences in the same batch are of the same length (different batches can have different lengths).

My code currently looks like:

train_loader = DataLoader(train_set, batch_size=BATCH_SIZE)

# other stuff...

for epoch in range(0, EPOCH_COUNT):
    for iter, sentence in enumerate(train_loader):
        model.zero_grad()
        model.hidden = model.init_hidden()
        sentence_in = prep_sentence(sentence, word_to_ix)
        targets = prep_tags(tags, tag_to_ix)
        # do forward pass and then backprop

Since I'm enumerating train_loader and iterating through each sentence, it's not clear to me how/at what point I can get the max sentence length of a single batch and then run my add_padding() function.


title: How to recover from pytroch CUDA out of memory?

body:
I tried the following code. When the code in try failed because of out of CUDA memory, I reduced the batch size to a half in except but it still appear the same issue for running the model in except but I'm sure half of the batch size is runnable since I have tried to directly run the code in except without trying the full batch. It works fine. By the way, is there any way to automatically set the batch size to fully use the CUDA memory without overflow?

try:
    output = model(Variable(torch.LongTensor(np.array(x))).to(device),Variable(torch.LongTensor(np.array(pos))).to(device),Variable(torch.LongTensor(np.array(m))).to(device))
    loss = criterion(output, Variable(torch.LongTensor(y)).to(device))#lb.transform(y)
    loss.backward()
    optimizer.step()
    losses.append(loss.data.mean())
except:
    half = int(len(x) / 2)
    x1 = x[:half]
    x2 = x[half:]
    pos1 = pos[:half]
    pos2 = pos[half:]
    m1 = m[:half]
    m2 = m[half:]
    y1 = y[:half]
    y2 = y[half:]
    optimizer.zero_grad()
    output = model(Variable(torch.LongTensor(np.array(x1))).to(device),Variable(torch.LongTensor(np.array(pos1))).to(device),Variable(torch.LongTensor(np.array(m1))).to(device))
    loss = criterion(output, Variable(torch.LongTensor(y1)).to(device))#lb.transform(y)
    loss.backward()
    optimizer.step()
    losses.append(loss.data.mean())
    output = model(Variable(torch.LongTensor(np.array(x2))).to(device),Variable(torch.LongTensor(np.array(pos2))).to(device),Variable(torch.LongTensor(np.array(m2))).to(device))
    loss = criterion(output, Variable(torch.LongTensor(y2)).to(device))#lb.transform(y)
    loss.backward()
    optimizer.step()
    losses.append(loss.data.mean())


title: C++, Python embedding: Importing torch/sklearn library on embedded python code throws a DEBUG: caught signal to interrupt error

body:
I have written a program in C++ which uses python embeddings in order to call a python module, which utilizes the numpy and pytorch libraries.
When using solely the numpy library I can successfully call and return from the module without any errors.
However, when I add the "import torch" command on my module, my program throws the following error:

  DEBUG: caught signal to interrupt (Child exited).

The problem also occurs when I import torch/sklearn modules using the PyRun_SimpleString() function.

Part of my program where I call the respective imports:

Py_Initialize();
PyRun_SimpleString("import numpy");
cout << "numpy ok" << endl;
PyRun_SimpleString("import torch");
cout << "torch ok" << endl;

When I execute the code above, I get the following output:

numpy ok
DEBUG: caught signal to interrupt (Child exited).

I compile and link my program using the python3-config --cflags and python3-config --ldflags` outputs.
I have also pytorch installed for my python3.6 as can be seen from the code below:

PyRun_SimpleString("import importlib");
PyRun_SimpleString("flag = importlib.util.find_spec('torch')");
PyRun_SimpleString("print(flag is not None)");

This code prints True, showing the torch is actually installed but for some reason can not be loaded.
Finally, the same behavior occurs when trying to import the sklearn library. Thank you.


title: pytorch, Using nn.DataParallel in LSTM

body:
/pytorch/aten/src/ATen/native/cudnn/RNN.cpp:1266: UserWarning: RNN module weights are not part of single contiguous chunk of memory.
This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().

Hello. I am using pytorch.
I am trying to use DataParallel function in pytorch,
but the model is LSTM. I'm warned to flatten the model again,
but I don't know when and where to flatten.
Can you let me know?

This is my model

    import torch.nn as nn
from torchvision import models

class ConvLstm(nn.Module):
    def __init__(self, latent_dim, model, hidden_size, lstm_layers, bidirectional, n_class):
        super(ConvLstm, self).__init__()
        self.conv_model = Pretrained_conv(latent_dim, model)
        self.Lstm = Lstm(latent_dim, hidden_size, lstm_layers, bidirectional)
        self.output_layer = nn.Sequential(
            nn.Linear(2 * hidden_size if bidirectional ==
                    True else hidden_size, n_class),
            nn.Softmax(dim=-1)
        )

    def forward(self, x):
        batch_size, timesteps, channel_x, h_x, w_x = x.shape
        conv_input = x.view(batch_size * timesteps, channel_x, h_x, w_x)
        conv_output = self.conv_model(conv_input)
        lstm_input = conv_output.view(batch_size, timesteps, -1)
        lstm_output = self.Lstm(lstm_input)
        lstm_output = lstm_output[:, -1, :]
        output = self.output_layer(lstm_output)
        return output

class Pretrained_conv(nn.Module):
    def __init__(self, latent_dim, model):
        if model == 'resnet152':
            super(Pretrained_conv, self).__init__()
            self.conv_model = models.resnet152(pretrained=True)
            # ====== freezing all of the layers ======
            for param in self.conv_model.parameters():
                param.requires_grad = False
            # ====== changing the last FC layer to an output with the size we need. this layer is un freezed ======
            self.conv_model.fc = nn.Linear(
                self.conv_model.fc.in_features, latent_dim)

    def forward(self, x):
        return self.conv_model(x)

class Lstm(nn.Module):
    def __init__(self, latent_dim, hidden_size, lstm_layers, bidirectional):
        super(Lstm, self).__init__()
        self.Lstm = nn.LSTM(latent_dim, hidden_size=hidden_size,
                            num_layers=lstm_layers, batch_first=True, bidirectional=bidirectional)
        self.hidden_state = None

    def reset_hidden_state(self):
        self.hidden_state = None

    def forward(self, x):
        output, self.hidden_state = self.Lstm(x, self.hidden_state)
        return output

Enter LSTM and execute the following code.

def foward_step(model, images, labels, criterion, mode=''):

model.module.Lstm.reset_hidden_state()

if mode == 'test':

    with torch.no_grad():

        output = model(images)

else:

    output = model(images)

loss = criterion(output, labels)

# Accuracy calculation

predicted_labels = output.detach().argmax(dim=1)

acc = (predicted_labels == labels).cpu().numpy().sum()

return loss, acc, predicted_labels.cpu()

This is main

model = nn.DataParallel(model, device_ids=[0,1,2,3]).cuda()


title: How update weights of two separate neural network with a computed loss?

body:
I have an encoder and a proxy network that help the encoder to maximize information between its input(an image) and output (feature vector of image). to get this done, I used a loss function that estimate MI and by an optimizer the weights of both networks get updated with computed loss, but I'm not sure that does this get done correctly or not. I used following code (in pytorch):

# Clear the previous gradients
discriminator_net_optim.zero_grad()
encoder_net_optim.zero_grad()

autograd.backward(loss)

torch.nn.utils.clip_grad_norm_(discriminator.parameters(), 2)
torch.nn.utils.clip_grad_norm_(encoder.parameters(), 2)

# adjust weights in discriminator and encoder
discriminator_net_optim.step()
encoder_net_optim.step()

any help or suggestion is appreciated.


title: Convolution in PyTorch with non-trainable pre-defined kernel

body:
I would like to introduce a custom layer to my neural network. The mathematical operation should be a discrete 2D cross correlation (or convolution) with a non-trainable kernel. The values in the kernel depend on three things: kernel shape, strides and padding. I intend to multiply the output element-wise with a weight matrix.

PyTorch already has an implementation of a discrete 2D cross correlation class called 'Conv2d', however it generates a random kernel and trains using the entries of said kernel. If possible I would like a class similar to 'Conv2d' that does what I need, to make sure to use my GPU most effectively. I tried implementing this on my own, but couldn't figure out how to obtain the correct shapes for the input array. 'Conv2d' only uses 'in_channels', if I understood correctly.


title: Traying to run HRNet-Semantic-Segmentation test.py but got a "Command '['ninja', '-v']' returned non-zero exit status 1". So what to do?

body:
I am trying to run the example code to evaluate the Cityscapes validation set from https://github.com/HRNet/HRNet-Semantic-Segmentation/tree/pytorch-v1.1

(pytorch) dev-tojo@devpc01:~/HRNet-Semantic-Segmentation$ python tools/test.py --cfg experiments/cityscapes/seg_hrnet_w18_small_v2_512x1024_sgd_lr1e-2_wd5e-4_bs_12_epoch484.yaml TEST.MODEL_FILE /home/dev-tojo/Downloads/hrnet_w18_small_v2_cityscapes_cls19_1024x2048_trainset.pth TEST.SCALE_LIST 0.5,0.75,1.0,1.25,1.5,1.75 TEST.FLIP_TEST True

but got error like this:

Traceback (most recent call last):
  File "/home/dev-tojo/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 949, in _build_extension_module
    check=True)
  File "/home/dev-tojo/anaconda3/envs/pytorch/lib/python3.7/subprocess.py", line 468, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

This is my env :

Ubuntu 18.04.4 LTS
python 3.7.0
pytorch 1.1.0
ninja 1.9.0
cuda10.0
cudnn 7.5.0
gcc 7.5.0

I use the HRNet-Semantic-Segmentation pythorch v1.1 branch not the master,and i try to evaluate with the hrnet_w18_small_v2_cityscapes_cls19_1024x2048_trainset.pth

I also make some modification in the  config file seg_hrnet_w18_small_v2_512x1024_sgd_lr1e-2_wd5e-4_bs_12_epoch484.yaml to make it work in one GPU like suggest from


title: group rows by specific value in one column and calculate the mean in PyTorch

body:
the sample tensor:

tensor([[ 0.,  1.,  2.,  3.,  4.,  5.],  # class1
        [ 6.,  7.,  8.,  9., 10., 11.],  # class3
        [12., 13., 14., 15., 16., 17.],  # class2
        [18., 19., 20., 21., 22., 23.],  # class0
        [24., 25., 26., 27., 28., 29.].  # class1
])

the expected result:

tensor([[18., 19., 20., 21., 22., 23.], # class0
        [12., 13., 14., 15., 16., 17.], # class1
        [12., 13., 14., 15., 16., 17.], # class2
        [ 6.,  7.,  8.,  9., 10., 11.]. # class3
])

Is there a pure PyTorch method to implement this?


title: How to use the PyTorch Transformer with multi-dimensional sequence-to-seqence?

body:
I'm trying to go seq2seq with a Transformer model. My input and output are the same shape (torch.Size([499, 128]) where 499 is the sequence length and 128 is the number of features.
My input looks like:

My output looks like:

My training loop is:
    for batch in tqdm(dataset):
        optimizer.zero_grad()
        x, y = batch

        x = x.to(DEVICE)
        y = y.to(DEVICE)

        pred = model(x, torch.zeros(x.size()).to(DEVICE))

        loss = loss_fn(pred, y)
        loss.backward()
        optimizer.step()

My model is:
import math
from typing import final
import torch
import torch.nn as nn

class Reconstructor(nn.Module):
    def __init__(self, input_dim, output_dim, dim_embedding, num_layers=4, nhead=8, dim_feedforward=2048, dropout=0.5):
        super(Reconstructor, self).__init__()

        self.model_type = 'Transformer'
        self.src_mask = None
        self.pos_encoder = PositionalEncoding(d_model=dim_embedding, dropout=dropout)
        self.transformer = nn.Transformer(d_model=dim_embedding, nhead=nhead, dim_feedforward=dim_feedforward, num_encoder_layers=num_layers, num_decoder_layers=num_layers)
        self.decoder = nn.Linear(dim_embedding, output_dim)
        self.decoder_act_fn = nn.PReLU()

        self.init_weights()

    def init_weights(self):
        initrange = 0.1
        nn.init.zeros_(self.decoder.weight)
        nn.init.uniform_(self.decoder.weight, -initrange, initrange)

    def forward(self, src, tgt):

        pe_src = self.pos_encoder(src.permute(1, 0, 2))  # (seq, batch, features)
        transformer_output = self.transformer_encoder(pe_src)
        decoder_output = self.decoder(transformer_output.permute(1, 0, 2)).squeeze(2)
        decoder_output = self.decoder_act_fn(decoder_output)
        return decoder_output

My output has a shape of torch.Size([32, 499, 128]) where 32 is batch, 499 is my sequence length and 128 is the number of features. But the output has the same values:
tensor([[[0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         ...,
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017]]],
       grad_fn=<PreluBackward>)

What am I doing wrong? Thank you so much for any help.


title: RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED in pytorch

body:
I am running CNN algorithm using PyTorch on my new machine with 3 Nvidia GPUs and getting the error below:
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

File "code.py", line 342, in <module>
    trainer.fit(model)

  File "/home/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 514, in fit
    self.dispatch()

  File "/home/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 554, in dispatch
    self.accelerator.start_training(self)

  File "/home/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 74, in start_training
    self.training_type_plugin.start_training(trainer)

  File "/home/.local/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 111, in start_training
    self._results = trainer.run_train()

  File "/home/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 615, in run_train
    self.run_sanity_check(self.lightning_module)

  File "/home/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 864, in run_sanity_check
    _, eval_results = self.run_evaluation(max_batches=self.num_sanity_val_batches)

  File "/home/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 733, in run_evaluation
    output = self.evaluation_loop.evaluation_step(batch, batch_idx, dataloader_idx)

  File "/home/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 164, in evaluation_step
    output = self.trainer.accelerator.validation_step(args)

  File "/home/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 178, in validation_step
    return self.training_type_plugin.validation_step(*args)

  File "/home/.local/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 290, in validation_step
    return self.model(*args, **kwargs)

  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)

  File "/home/.local/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 705, in forward
    output = self.module(*inputs[0], **kwargs[0])

  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)

  File "/home/.local/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 63, in forward
    output = self.module.validation_step(*inputs, **kwargs)

  File code.py", line 314, in validation_step
    pred = self.forward(x)

  File code.py", line 259, in forward
    x = self.conv0(x)          #([12, 600, 600])

  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)

  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/container.py", line 119, in forward
    input = module(input)

  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)

  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)

  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 395, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,

NVIDIA-MSI:

The code is running without any issue on another machine with driver version 450.51.06 and Cuda version 11. You can see nvidia-smi of new machine above. I checked different comments on other questions same to this issue and non of them resolved my issue.


title: some problems when installing torchaudio on mac

body:
I use following command to install torchaudio:"conda install -c derickl torchaudio". But when I try to import torchaudio the following error message pops up:

Can someone take a look or propose some other ways to install torchaudio? Thanks in advance!