title: AI Explainabilty: Alibi in PyTorch giving unexpected output

body:
I am trying to use alibi anchors using PyTorch Framework on Income Prediction Model(<=50k or >50k). I have converted numpy arrays into 'torch tensors`. My code doens't show any error but the candidate anchor is wrong.
This is the code block I am using for explainability:
with torch.no_grad():
    predict_fn = lambda x: model(x = torch.from_numpy(x).type(torch.FloatTensor))
    explainer = AnchorTabular(predict_fn, feature_names, seed=1)
    explainer.fit(train_features)
    class_names = ['<=50K', '>50K']
    idx = 4
    pred = class_names[explainer.predictor(train_features[idx].reshape(1, -1))[0]]
    print('Prediction: ', pred)
    explanation = explainer.explain(test_features[idx], threshold=0.95)
    print('Anchor: %s' % (' AND '.join(explanation.anchor)))
    print('Precision: %.2f' % explanation.precision)
    print('Coverage: %.2f' % explanation.coverage)

Code doesnt show any error but the output of the following is:
Prediction:  <=50K
Anchor:
Precision: 1.00
Coverage: 1.00

whereas by following the original documentation in Tensorflow/Keras, output is:
Prediction:  <=50K
Anchor: Age <= 28.00 AND Education = High School grad AND Capital Gain <= 0.00
Precision: 0.98
Coverage: 0.16

Original code can be found here: link
Thanks for any help!


title: PyTorch YOLOv5 - Microsoft C++ Build Tools

body:
I am trying to install PyTorch YOLOv5 from ultralytics from here in Windows 10 x86_64 system. The instructions seem pretty straightforward and I after having installed PyTorch for GPU, I am attempting to install the required requirements by using the command:
pip install -qr https://raw.githubusercontent.com/ultralytics/yolov5/master/requirements.txt
to which I get the following ERROR log:

ERROR: Command errored out with exit status 1:    command:
'C:\Users\arjun\anaconda3\envs\pytorch_object_detection\python.exe' -u
-c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\arjun\AppData\Local\Temp\pip-install-7kbo300l\pycocotools_e5774d8d59d14fa9b3baece40c2b7248\setup.py'"'"';
file='"'"'C:\Users\arjun\AppData\Local\Temp\pip-install-7kbo300l\pycocotools_e5774d8d59d14fa9b3baece40c2b7248\setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))'
bdist_wheel -d 'C:\Users\arjun\AppData\Local\Temp\pip-wheel-kc1jnk9w'
cwd: C:\Users\arjun\AppData\Local\Temp\pip-install-7kbo300l\pycocotools_e5774d8d59d14fa9b3baece40c2b7248
Complete output (16 lines):   running bdist_wheel   running build
running build_py   creating build   creating build\lib.win-amd64-3.8
creating build\lib.win-amd64-3.8\pycocotools   copying
pycocotools\coco.py -> build\lib.win-amd64-3.8\pycocotools   copying
pycocotools\cocoeval.py -> build\lib.win-amd64-3.8\pycocotools
copying pycocotools\mask.py -> build\lib.win-amd64-3.8\pycocotools
copying pycocotools_init_.py -> build\lib.win-amd64-3.8\pycocotools
running build_ext   cythoning pycocotools/_mask.pyx to
pycocotools_mask.c
C:\Users\arjun\anaconda3\envs\pytorch_object_detection\lib\site-packages\Cython\Compiler\Main.py:369:
FutureWarning: Cython directive 'language_level' not set, using 2 for
now (Py2). This will change in a later release! File:
C:\Users\arjun\AppData\Local\Temp\pip-install-7kbo300l\pycocotools_e5774d8d59d14fa9b3baece40c2b7248\pycocotools_mask.pyx
tree = Parsing.p_module(s, pxd, full_module_name)   building 'pycocotools.mask' extension   error: Microsoft Visual C++ 14.0 or
greater is required. Get it with "Microsoft C++ Build Tools":
https://visualstudio.microsoft.com/visual-cpp-build-tools/
----------------------------------------   ERROR: Failed building wheel for pycocotools
ERROR: Command errored out with exit status 1:
command: 'C:\Users\arjun\anaconda3\envs\pytorch_object_detection\python.exe' -u
-c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\arjun\AppData\Local\Temp\pip-install-7kbo300l\pycocotools_e5774d8d59d14fa9b3baece40c2b7248\setup.py'"'"';
file='"'"'C:\Users\arjun\AppData\Local\Temp\pip-install-7kbo300l\pycocotools_e5774d8d59d14fa9b3baece40c2b7248\setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))'
install --record
'C:\Users\arjun\AppData\Local\Temp\pip-record-l60dglwi\install-record.txt'
--single-version-externally-managed --compile --install-headers 'C:\Users\arjun\anaconda3\envs\pytorch_object_detection\Include\pycocotools'
cwd: C:\Users\arjun\AppData\Local\Temp\pip-install-7kbo300l\pycocotools_e5774d8d59d14fa9b3baece40c2b7248
Complete output (14 lines):
running install
running build
running build_py
creating build
creating build\lib.win-amd64-3.8
creating build\lib.win-amd64-3.8\pycocotools
copying pycocotools\coco.py -> build\lib.win-amd64-3.8\pycocotools
copying pycocotools\cocoeval.py -> build\lib.win-amd64-3.8\pycocotools
copying pycocotools\mask.py -> build\lib.win-amd64-3.8\pycocotools
copying pycocotools_init.py -> build\lib.win-amd64-3.8\pycocotools
running build_ext
skipping 'pycocotools_mask.c' Cython extension (up-to-date)
building 'pycocotools._mask' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools":
https://visualstudio.microsoft.com/visual-cpp-build-tools/
---------------------------------------- ERROR: Command errored out with exit status 1:
'C:\Users\arjun\anaconda3\envs\pytorch_object_detection\python.exe' -u
-c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\arjun\AppData\Local\Temp\pip-install-7kbo300l\pycocotools_e5774d8d59d14fa9b3baece40c2b7248\setup.py'"'"';
file='"'"'C:\Users\arjun\AppData\Local\Temp\pip-install-7kbo300l\pycocotools_e5774d8d59d14fa9b3baece40c2b7248\setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))'
install --record
'C:\Users\arjun\AppData\Local\Temp\pip-record-l60dglwi\install-record.txt'
--single-version-externally-managed --compile --install-headers 'C:\Users\arjun\anaconda3\envs\pytorch_object_detection\Include\pycocotools'
Check the logs for full command output.

I have installed Microsoft C++ Build Tools and get the following output in CMD:

** Visual Studio 2019 Developer Command Prompt v16.9.6
** Copyright (c) 2021 Microsoft Corporation

I am trying to reinstall the requirements.txt but the error for Microsoft C++ Build Tools still exist.
What should I do?


title: Starting with a high loss value, but the loss converged at the end. Could start with a loss > 100?

body:
I have been trying to attempt plant disease detection using transfer learning methods. I chose ResNet50 first. I also performed a baseline model which is a CNN model. In resnet50, I used cross entropy loss and trained the model for 30 epochs. I did batch normalization too. Initially epoch 1 loss was 112.5250 and the training loss was 87.512. But, for the last epoch it was 2.1660 and Validation loss was 1.8905 with Validation accuracy as 0.995. The overall accuracy of the model was 98.8% and the model doesn't seem to overfit too.
Before training the model, I performed hyper parameter optimization where I optimized Learning rate and momentum using Bayesian optimization. I optimized weight decay using Cross validation. I performed batch normalization. Before that, I trained the model without any optimization. I just assumed the learning rate to be 0.001, same loss function and trained for 30 epochs. In this case, the model started with a loss value of 114.2 and validation loss was 130.46. It converged to 9.9038 with accuracy of 97%.
So, my question is, it is possible for the model to start with such a high loss value despite giving a good accuracy at the end? Does the magnitude of the loss have nothing to do with the correctness of the model nor the accuracy? If the model started with a huge loss, but gave a good accuracy, is my model bad?
First 5 epochs
EPOCH: 1

Epoch time taken: 28.61398434638977 seconds. Epoch 1 loss: 118.5250 Validation loss: 87.5214 Validation accuracy: 0.753

EPOCH: 2

Epoch time taken: 28.737553358078003 seconds. Epoch 2 loss: 68.8141 Validation loss: 50.3452 Validation accuracy: 0.858

EPOCH: 3

Epoch time taken: 28.45273518562317 seconds. Epoch 3 loss: 41.5181 Validation loss: 31.1520 Validation accuracy: 0.928

EPOCH: 4

Epoch time taken: 28.434630155563354 seconds. Epoch 4 loss: 26.8844 Validation loss: 21.1203 Validation accuracy: 0.948

EPOCH: 5

Epoch time taken: 28.762181282043457 seconds. Epoch 5 loss: 19.3646 Validation loss: 15.4843 Validation accuracy: 0.955

Last 5 epochs
EPOCH: 26

Epoch time taken: 28.35645818710327 seconds. Epoch 26 loss: 2.8168 Validation loss: 2.2771 Validation accuracy: 0.990

EPOCH: 27

Epoch time taken: 28.51978588104248 seconds. Epoch 27 loss: 3.3986 Validation loss: 2.2274 Validation accuracy: 0.992

EPOCH: 28

Epoch time taken: 28.51390767097473 seconds. Epoch 28 loss: 2.2707 Validation loss: 1.9976 Validation accuracy: 0.992

EPOCH: 29

Epoch time taken: 28.456573247909546 seconds. Epoch 29 loss: 2.5344 Validation loss: 1.9272 Validation accuracy: 0.990

EPOCH: 30

Epoch time taken: 28.486974239349365 seconds. Epoch 30 loss: 2.1660 Validation loss: 1.8905 Validation accuracy: 0.995


title: How to convert torch.norm to cosine distance

body:
I want to change norm distance to cosine distance, help me convert this function to cosine distance
def feat_prototype_distance(self, feat):
    N, C, H, W = feat.shape
    feat_proto_distance = -torch.ones((N, self.class_numbers, H, W)).to(feat.device)
    for i in range(self.class_numbers):
        feat_proto_distance[:, i, :, :] = torch.norm(self.objective_vectors[i].reshape(-1,1,1).expand(-1, H, W) - feat, 2, dim=1)
    return feat_proto_distance

This is original function using norm distance with shapes:
self.objective_vectors[i].reshape(-1,1,1).expand(-1, H, W): torch.Size([256, 128, 224])
feat: torch.Size([8, 256, 128, 224]) with 8 is batch_size


title: Define nn.parameters with a for loop

body:
I am interested in defining L weights in a custom neural network with Pytorch.
If L is known it is not a problem to define them one by one, but if L is not known I want to use a for loop to define them. My idea is to do something like this (which does not work)
class Network(nn.Module):
    def __init__(self, ):
        super(Network, self).__init__()
        self.nl = nn.ReLU()

        for i in range(L):
          namew = 'weight'+str(i)
          self.namew = torch.nn.Parameter(data=torch.Tensor(2,2), requires_grad=True)

This should do something like this (which instead works but is limited to a specific number of weights):
class Network(nn.Module):
    def __init__(self, ):
        super(Network, self).__init__()
        self.nl = nn.ReLU()

        self.weight1 = torch.nn.Parameter(data=torch.Tensor(2,2), requires_grad=True)
        self.weight2 = torch.nn.Parameter(data=torch.Tensor(2,2), requires_grad=True)
        self.weight3 = torch.nn.Parameter(data=torch.Tensor(2,2), requires_grad=True)

With what I tried to do, there is the problem that instead of a "dynamic" string for 'namew', Pytorch recognizes just the string 'namew'. Therefore instead of L weights, just 1 weight is defined.
Is there some way to solve this problem?


title: Pytorch data.random_split() doesn't split randomly

body:
I am trying to split my custom dataset randomly into test and train. The code runs and outputs the test and train folders successfully, but I need the test and train sets to be different each time I ran the code. Isn't this what randomly splitting supposed to mean/do?
p.s. just for clarification, the data is images, so I expect to see different images chosen for test and train sets each time I execute the code.
#Set the random seeds for reproducibility
SEED = 1234

random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
torch.cuda.manual_seed(SEED)
torch.backends.cudnn.deterministic = True

TRAIN_RATIO = 0.9

data_dir = 'Data Set 1'
images_dir = os.path.join(data_dir, 'images')
train_dir = os.path.join(data_dir, 'train')
test_dir = os.path.join(data_dir, 'test')

if os.path.exists(train_dir):
    shutil.rmtree(train_dir)
if os.path.exists(test_dir):
    shutil.rmtree(test_dir)

os.makedirs(train_dir)
os.makedirs(test_dir)

classes = os.listdir(images_dir)

for c in classes:

    class_dir = os.path.join(images_dir, c)

    images = os.listdir(class_dir)

    n_train = int(len(images) * TRAIN_RATIO)
    n_test = len(images) - n_train
    train_images, test_images = data.random_split(images,
                                               [n_train, n_test])

    os.makedirs(os.path.join(train_dir, c), exist_ok=True)
    os.makedirs(os.path.join(test_dir, c), exist_ok=True)

    for image in train_images:
        image_src = os.path.join(class_dir, image)
        image_dst = os.path.join(train_dir, c, image)
        shutil.copyfile(image_src, image_dst)

    for image in test_images:
        image_src = os.path.join(class_dir, image)
        image_dst = os.path.join(test_dir, c, image)
        shutil.copyfile(image_src, image_dst)


title: why itertools.zip_longest(*indexes_batch) return an transpose result against itertools.zip_longest(indexes_batch)

body:
I have a tensor like this

indexes_batch
[[28, 633, 1076, 332, 270, 123, 28, 6, 2],
 [47, 171, 120, 12, 261, 58, 1103, 6, 2],
 [1473, 3646, 7, 260, 4, 4, 4, 2],
 [531, 42, 55, 127, 654, 4, 2],
 [639, 6, 2]]

when I use list(itertools.zip_longest(*indexes_batch, fillvalue=0)) I get
[(28, 47, 1473, 531, 639),
 (633, 171, 3646, 42, 6),
 (1076, 120, 7, 55, 2),
 (332, 12, 260, 127, 0),
 (270, 261, 4, 654, 0),
 (123, 58, 4, 4, 0),
 (28, 1103, 4, 2, 0),
 (6, 6, 2, 0, 0),
 (2, 2, 0, 0, 0)]

and use list(itertools.zip_longest(indexes_batch, fillvalue=0)),I get

[([28, 633, 1076, 332, 270, 123, 28, 6, 2],),
 ([47, 171, 120, 12, 261, 58, 1103, 6, 2],),
 ([1473, 3646, 7, 260, 4, 4, 4, 2],),
 ([531, 42, 55, 127, 654, 4, 2],),
 ([639, 6, 2],)]

so waht I confused about is what is the function of * ,and why can * result a Transposed matrix


title: Why does PyTorch gather function require index argument to be of type LongTensor?

body: I'm writing some code in PyTorch and I came across the gather function. Checking the documentation I saw that the index argument takes in a LongTensor, why is that? Why does it need to take in a LongTensor instead of another type such as IntTensor? What are the benefits?


title: Combining multiple boxes

body:
I have trained a network for detection using bounding boxes. During evaluation I remove boxes using IoU technique and thresholding. However, I am getting multiple boxes for a single object (as seen in image below). The bounding boxes are
tensor([[448.00, 219.25, 476.33, 287.26], [453.84, 223.89, 473.85, 279.77], [450.89, 218.27, 467.31, 276.45], [448.76, 219.92, 484.23, 283.18]])

Is there any way i can combine or remove the extra boxes?


title: Performing normal or uniform initializations on Float tensors result in only zeros

body:
import torch
pytorchGPUDirectCreateWEmpty = torch.empty(size=(20000000, 128), dtype=torch.float, device='cuda', requires_grad=False, pin_memory=False).uniform_(-1, 1)
pytorchGPUDirectCreateWEmpty

results in

tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')

and

import torch
torch.set_default_tensor_type('torch.cuda.FloatTensor')
u_embeddings = torch.nn.Embedding(20000000, 128, padding_idx=None, max_norm=None, norm_type=2.0, scale_grad_by_freq=False, sparse=False, _weight=None)
u_embeddings.weight.data.uniform_(-1, 1)
u_embeddings.weight.data

results in

tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]])

If I initialize with double instead of float and the initialization works fine. I could convert to float later, but I am working with limited memory and unable to initialize a double first before converting.

Why is the initialization not working for float tensors?


title: How to set sample input size with different width in different channels to Conv1d in pytorch

body:
I'm new at deep learning and I'm writing a 1D CNN model to train my dataset. Each data in the dataset contains three channels: (raw_data, sex, age). The sample rate of raw_data is 50Hz, and window size is 2 seconds, which means 100 samples in one data. So my data will look like this
([1,2,3,4,...,100],[1],[20]) where [1] means sex and [20] means age.

Now I want to put this data into my 1D CNN model. I know that nn.Conv1d(batch_size, channel, width). If width in all channels are the same, there is no problem. But in my case, raw_data contains 100 elements, sex and age only contain 1 element. How can I put this into my Conv1d layer?

Any help would be appreciated. Thank you!


title: How could I know whether a function in Pytorch allocates new memory or not?

body: Recently, I got stuck in a situation that, in my model, the input data really consumed a lot of memory. And this lead to a lot of memory usage when I operate the data in my network layers. I really want to know whether the operations will allocate new memory block or not. I saw the pytorch doc only found how to use the function. I wonder is there a doc or some websites or anything else official to help me out. For example, will functions like view(), permute() or contiguous() allocate new memory block or not and how do you know that. It really caught me, thanks for helping.


title: Image clustering - allocating memory on GPU

body:
I've written this code for image classification by pretrained googlenet:

gnet = models.googlenet(pretrained=True).cuda()

transform = transforms.Compose([transforms.Resize(256), transforms.CenterCrop(32), transforms.ToTensor()])
images = {}
resultDist = {}
i = 1

for f in glob.iglob("/data/home/student/HW3/trainData/train2014/*"):
    print(i)
    i = i + 1
    image = Image.open(f)
    # transform, create batch and get gnet weights
    img_t = transform(image).cuda()
    batch_t = torch.unsqueeze(img_t, 0).cuda()
    try:
        gnet.eval()
        out = gnet(batch_t)
        resultDist[f[-10:-4]] = out
        del out
    except:
        print(img_t.shape)
    del img_t
    del batch_t
    image.close()
    torch.cuda.empty_cache()
    i = i + 1

torch.save(resultDist, '/data/home/student/HW3/googlenetOutput1.pkl')

I deleted all the possible tensors from the GPU after using them, but after about 8000 images from my dataset the GPU is full. I found the problem to be in:

resultDist[f[-10:-4]] = out

The dictionary taking alot of space and I can't delete it because I want to save my data to pkl file.


title: what is the replacement of the view function in tensorflow?

body:
# Predict localization boxes' bounds (as offsets w.r.t prior-boxes)
    l_conv4_3 = self.loc_conv4_3(conv4_3_feats)  # (N, 16, 38, 38)
    l_conv4_3 = l_conv4_3.permute(0, 2, 3,
                                  1).contiguous()  # (N, 38, 38, 16), to match prior-box order (after .view())
    # (.contiguous() ensures it is stored in a contiguous chunk of memory, needed for .view() below)
    l_conv4_3 = l_conv4_3.view(batch_size, -1, 4)  # (N, 5776, 4), there are a total 5776 boxes on this feature map

I want to change the (N,16,38,38) feature map to the #(N,5764,4) like the above
what is the repladcement of the view function in tensorflow 2.0?


title: Classifier Loss function dimension out of range error

body:
import torch.nn as nn

class MyModel(nn.Module):

    def __init__(self):
        super(MyModel, self).__init__()

        self.conv1 = nn.Conv2d(1, 32, kernel_size=5, stride=2, padding=2)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=5,stride=2)

        self.fc = nn.Linear(884736, 1000)
        self.fc1 = nn.Linear(1000, 600)
        self.fc2 = nn.Linear(600, 200)
        self.fc3 = nn.Linear(200, 6)

        self.pooling = nn.MaxPool2d(2, 2)

    def forward(self, x):

        x = self.conv1(x)
        x = nn.functional.relu(x)
        x= self.pooling(x)

        x= self.conv2(x)

        x = torch.flatten(nn.functional.relu(x))

        x= self.fc(x)
        x = nn.functional.relu(x)
        # import pdb; pdb.set_trace()

        x= self.fc1(x)
        x= self.fc2(x)
        x= self.fc3(x)

        # x = torch.softmax(x)

        return x

# model = torch.nn.Sequential(
# )

model = MyModel()

#Training

dataiter = iter(trainloader)
total_epochs = 5
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters())

for epoch in tqdm(range(total_epochs)):

  #initialize batch
  gc.collect()
  input_, label_ = dataiter.next()

  #forwardd
  out = model.forward(input_)

  #backwardd
  print (out,out.shape,)

  print (label_, label_.shape)
  # out = out.unsqueeze(dim=0)

  # label_ =label_.type_as(out)
  loss = criterion(out, label_)
  loss.backward()

  optimizer.zero_grad()
  optimizer.step()

  print('batch_loss:', str(loss.item()))

  print('Epochs completed:', epoch+1,'\n')
  print('epoch_loss:' + loop_loss/float(batch_size))

I have a dataset of different breed of dogs (120 classes)
http://vision.stanford.edu/aditya86/ImageNetDogs/images.tar

The labels are int values ranging from 1 to 120

I need to make a classifier

Getting an error at loss computation
Dimension out of range (expected to be in range of [-1, 0], but got 1)

What could be wrong?


title: Train inactive class in classification AI problem

body:
I'm facing an issue. I want to classify data in n classes. But, I also want to say that, the answer is none of these n classes.

For example, my classes are : HORSE, CAT, DOG. So I will train with data related to HORSE, CAT and DOG.

But if I give my model something else, just like a car, I would like my model to tell that it is not HORSE or DOG or CAT.
So maybe I have to train a model with these classes : HORSE, CAT, DOG, OTHER ?

But If I train OTHER class with data, how can I be sure that if I give it something new like a spacerocket, the prediction will be active for the OTHER class ?

In other words, I'm in trouble with inactive class. I don't want my model to give "the best prediction" beetween my desired classes, I also want it to tell me if it is none of them.

Thank you !


title: Can we use sequence labels for question answering?

body:
I have some Question, Context and Answer triplets for training and along with that, I have sequences inside the queries that are labelled into a few categories.

Example: Suppose 2 kinds of labels are there A and B.

Query: How is the weather outside?

Label A: How is

Label B: weather outside

I wish to encode the question and context using BERT and then use these labels associated with the tokens to extract the answer span from the context.

My Approach: I thought of defining 3 different weight vectors (dimension: 768*768) or using weight vector of size (768 using elementwise multiplication with the query tokens) for the tokens labelled A, B and others. Then using the new query encoding and the BERT context encoding in an ensemble for the answer span selection task.

Any other ideas to use the sequence labels for the MRC task?


title: Docker container cannot output some information if -t option is not added

body:
I am using pytorch for distributed training, and use docker to containerize the training process.
I use torch.multiprocessing to startup multiprocess training:
import torch.multiprocessing as mp
mp.spawn(train, nprocs=args.gpus, args=(args,))

And I add some print in the train function for tracing:
def train(gpu, args):
    print('starting making group.......')
    ......

print("before train starts")
train()

Then I start the container:
docker run --network host --gpus all <my-image> python test.py

However, the "starting making group" log does not appear, and the output was:
before train starts

Then it is stuck here and I cannot stop it by neither ctrl+C nor other keys.
Then I add -t option:
docker run -t --network host --gpus all <my-image> python test.py

This time the log is as expected.
before train starts
starting making group..........

I think it's strange because any logs appear in the stdout even if -t option is not added, except for those in the train function, which is spawned from torch.multiprocessing.
Can someone tell me why does this happen? Thanks in advance.


title: When does GPU memory get allocated in pytorch?

body:
For example we can have code like this:
model.train()
...
optimizer.zero_grad()
...
trn_x, trn_y = next(train_loader)
val_x, val_y = next(valid_loader)
trn_x, trn_y = trn_x.cuda(), trn_y.cuda()
...
logits_search_train, _ = model_small(trn_x)
loss_weight = criterion(logits_search_train, trn_y)
loss_weight.backward()
optimizer.step()

Where in this code do we have GPU memory allocation/deallocation?


title: Update targets in classification images

body:
Why are we updating targets in the implementation of bayesian cnn with mc dropout here?
https://github.com/sungyubkim/MCDO/blob/master/Bayesian_CNN_with_MCDO.ipynb?fbclid=IwAR18IMLcdUUp90TRoYodsJS7GW1smk-KGYovNpojn8LtRhDQckFI_gnpOYc
def update_target(target, original, update_rate):
        for target_param, param in zip(target.parameters(), original.parameters()):
            target_param.data.copy_((1.0 - update_rate) * target_param.data + update_rate*param.data)


title: Training PyTorch NN but sometimes docker does not run

body:
I am training a NN model, and have made at least 10 sucessful run, but sometimes, my docker container does not run unless I restart my computer, but it either returns a runtime error that was previously nonexistent, or it tells me my CUDA is in wrong version. Although the docker image build CUDA 10.1 my nvidia-smi tells me I am running on CUDA 11. But I have previously managed to have it work and track my GPU. I have also tried to downgrade my CUDA but failing to do so. I know this might be a basic question, but I've searched around for a few weeks but found no close answer.
Screengrab of runtime error and previously successful previously train run in tensorboard


title: How to include an attribute to let the user decide whether to use GPU or not in pytorch?

body:
I am a beginner in Python and pytorch, and my apologies for not making myself clear or not using the proper terms.
I am trying to write a linear regression class that would allow the user to decide whether they want to run the model on GPU or not.
The following are my codes in a pretty dumb way.
class LinearRegression():
    def __init__(self, dtype = torch.float64):
        self.dtype = dtype
        self.bias = None
        self.weight = None

    def fit(self, x, y, std = None, device = "cuda"):
        if x.dtype is not self.dtype:
            x = x.type(dtype = self.dtype)
        if y.dtype is not self.dtype:
            y = y.type(dtype = self.dtype)

        #let the user decide whether they want it to be standardized
        if std == True:
          mean = torch.mean(input = x, dim=0).to(device)
          sd = torch.std(input = x, dim=0).to(device)
          x = x.to(device)
          x = (x-mean)/sd
        u = torch.ones(size = (x.size()[0], 1), dtype = self.dtype).to(device)
        x_design = torch.cat([u, x], dim = 1).to(device)
        y = y.to(device)
        parameter = torch.inverse(
            torch.transpose(x_design, dim0=0, dim1=1) @ x_design).to(device) @ \
                    torch.transpose(x_design, dim0=0, dim1=1).to(device) @ y

        self.bias = parameter[0, 0]
        self.weight = parameter[1:, 0]
        return self

Basically I just added .to(device) to every tensor so that it would be run on the device that the user wishes to use.
However, I am sure there is a better way to do this, perhaps including something in the __int__?
I am not sure how to make writing it more efficiently so that I don't have to include .to(device) anytime I add a new function.


title: XLA rng-bit-generator takes too much memory

body:
XLA allocates 4G of memory to this tensor. The size of which seems to scale with the batch size. Which doesn't make sense to me, it doesn't seem to be part of the model graph to be stored in HBM. I use a TPUv3.
I don't use any random operation apart from initialization of the model. Moreover I declared bfloat16 for all weights, but this is a u32 tensor.
  Largest program allocations in hbm:

  1. Size: 4.00G
     Shape: u32[128,8,1024,1024]{3,2,1,0:T(8,128)}
     Unpadded size: 4.00G
     XLA label: %rng-bit-generator = (u32[2,128]{1,0:T(2,128)}, u32[128,8,1024,1024]{3,2,1,0:T(8,128)}) rng-bit-generator(u32[2,128]{1,0:T(2,128)} %fusion.2446), algorithm=rng_default
     Allocation type: HLO temp
     ==========================

What could be the reason for the above allocation? I use pixelsnail from: https://github.com/kamenbliznashki/pixel_models
Questions:

Why does this tensor have a type of u32, when all my weight / model definitions (including a global enviromental flag) use BF16?
Why is rng-bit-generator used?


title: StepLR Learning Rate Scheduler applying an almost infinitely small decrease and also too early

body:
I am using the StepLR scheduler with the Adam optimizer:
    optimizer = torch.optim.Adam(model.parameters(), lr=LrMax, weight_decay=decay) # , betas=(args.beta1, args.beta2)
    print(f'Optimizer = {repr(optimizer)}')
    scheduler = StepLR(optimizer, step_size=5, gamma=0.2)

The initial learning rate lr is set to 0.1. At the end of the first epoch the situation is stable:
Train: Epoch 0 Batch=601 totalBatches=601 lr=0.1: Loss=2.5451838970184326 accuracy=0.1875
Train: Epoch 0 Batch=651 totalBatches=651 lr=0.1: Loss=3.527266025543213 accuracy=0.1875
Train: Epoch 0 Batch=656 totalBatches=656 lr=0.1: Loss=2.9547425508499146 accuracy=0.1875

But then the clock strikes midnight, we move on to epoch1 and the Learning Rate Scheduler goes bonkers:
Train: Epoch 1 Batch=45 totalBatches=701 lr=2.722258935367529e-93: Loss=2.878746271133423
accuracy=0.25

Why did the learning rate drop by 10^92 ?
Why did the learning rate change on the first epoch instead of epoch 5?


title: Predicting Dance Frames with Pytorch

body:
I created  a VAE archictecture to encode dance frames into latent representations.
Then I planned to use LSTM to take a sequence of those latent vectors to predict the next one. Then decode it and thus generate a new dance sequence.
However, this is not working. The LSTMs prediction is slightly off which makes the predicted frame not entirely accurate. Since that inaccurate frame is used as input to the LSTM next, the prediction of the LSTM becomes even worse off until the predictions are just black.
Does anyone have experience with encoded sequential data prediction who knows how to solve the issue? Or maybe could suggest another architecture that can work to predict dance frames?


title: Tensorflow model reconfiguration from SavedModel from onnx-based model

body:
I had a pretrained Dense-121 model from Pytorch and export it as onnx format. After that I load the onnx file and used prepare function from onnx_tf.backend. I finally used
tf_rep.export_graph('model_var.pb') so that I can convert my onnx file to SavedModel based tensorflow model.
So far so good.
What I wanted to do is loading this SavedModel type file (.pb) in order to obtain the model configuration so that I can reconfigure it in Tensorflow,not Pytorch any more. However I couldn't find any way to do that.
I can't use model.summary() for example.
Is there a way to do this or am I just wasting my time?
If not, it's probably because the way I use these tools are wrong. What can I do to reconfigure a model ,written in Pytorch, in Tensorflow?
Thank you!


title: PyTorch - instruction works in console but not jupyter notebook

body:
I'm learning Deeep Learning with PyTorch on JupyterLab. I'm converting my 2-feature, binary classifier to a one-hot-encoding output layer.
To obtain the loss on the current batch I do:
x1vec, x2vec, yvec = sampleBatch
yvec_pred = classifier(torch.cat( (x1vec,x2vec) ,  1))
lossOnBatch = lossFun(yvec_pred, yvec)

At the last line I got this error:
1D target tensor expected, multi-target not supported

caused by the loss function understanding my yvec being of minibatch_size x 1 as being multi-target.
Problem is that if I open an interactive console with the same python kernel and run:
lossFun(yvec_pred, yvec.squeeze().long())

I get a value tensor(1.4970, grad_fn=<NllLossBackward>) as expected, while substituting the previous line in the jupyterLab environment the error persists.
What am I missing?


title: How do I use my own dataframe for machine translation task?

body:
I've been able to run this notebook successfully in my system. (It is a machine translation task.)
In that notebook, multi30k dataset was used.
I would like to make use of one of these datasets. I am importing this data as a dataframe. How do I make changes to the code so that I get train, valid and test datasets using a dataframe?
Is there a way to convert my dataframe to look like multi30k, because this will not require any changes in code. Or is there a better way? Please assist.


title: How to save trained images without a burden on network

body:
I am training a convolutional neural network in pytorch and want to save trained images. I append each trained image in a data loader loop to save all the trained images into numpy file (train_pred in code below) and works properly. But it is a huge burden on the network and increase running time. Is there any other way to do this?
for epoch in range(epochs):
    mse_train_losses= []
    N_train = []

    train_pred=[]
    model.train()
    for data in train_loader:

        x_train_batch, y_train_batch = data[0].to(device,
            dtype=torch.float), data[1].to(device, dtype=torch.float)

        y_train_pred = model(x_train_batch)
        mse_train_loss = criterion(y_train_batch, y_train_pred, x_train_batch, mse)

        optimizer.zero_grad()
        mse_train_loss.backward()
        optimizer.step()

        mse_train_losses.append(mse_train_loss.item())

        N_train.append(len(x_train_batch))

        train_pred.append(y_train_pred)
        train_pred_de=torch.stack(train_pred).cpu().detach().numpy()


title: Is It Possible to Use Python Code with Pytorch Launched From Node JS?

body:
I manage to launch python code in node js with below code.
const pythonProcess = spawn('python',["./testcode/maincode/test_model/test.py","abc"]);
    pythonProcess.stdout.on("data",data=>{
        console.log(data.toString())
    })

the problem is that I want to use pytourch code with node js
but when I launch with below way,
 const pythonProcess = spawn('python',["./testcode/maincode/test_model/web_output.py","abc"]);
    pythonProcess.stdout.on("data",data=>{
        console.log(data.toString())
    })

it cause     "ModuleNotFoundError: No module named 'torch'\r\n",
I used to launch web_output.py with opening cmd
and command conda activate aip - > python web_output.py.
is it possible to command conda activate aip then command python web_output.py with node js ?


title: Is there any Softmax implementation with sections along the dim (blocky Softmax) in PyTorch?

body:
For example, given logits, dim, and boundary,
boundary = torch.tensor([[0, 3, 4, 8, 0]
                         [1, 3, 5, 7, 9]]

# representing sections look like:
#    [[00012222_]
#     [_00112233]
#  in shape: (2, 9)
# (sections cannot be sliced)

logits = torch.rand(2, 9, 100)
result = blocky_softmax(logits, dim = 1, boundary = boundary)

# result[:, :, 0] may look like:
#   [[0.33, 0.33, 0.33, 1.00, 0.25, 0.25, 0.25, 0.25, 0.0 ]
#    [0.0,  0.50, 0.50, 0.50, 0.50, 0.50, 0.50, 0.50, 0.50]]
# other 99 slices looks similar with each blocks sum to 1.

we hope the Softmax is applied to dim = 1, but sections are also applied to this dim.
My current implementation with PyTorch is using for. It is slow and cost too much memory,
which looks like:
def blocky_softmax(logits, splits, map_inf_to = None):
    _, batch_len, _ = logits.shape
    exp_logits    = logits.exp() # [2, 9, 100]
    batch_seq_idx = torch.arange(batch_len, device = logits.device)[None, :]
    base          = torch.zeros_like(logits)
    _, n_blocks   = splits.shape
    for nid in range(1, n_blocks):
        start = splits[:, nid - 1, None]
        end   = splits[:, nid,     None]
        area = batch_seq_idx >= start
        area &= batch_seq_idx < end
        area.unsqueeze_(dim = 2)
        blocky_z = area * blocky_z
        base = base + blocky_z
    if map_inf_to is not None:
        good_base = base > 0
        ones = torch.ones_like(base)
        base = torch.where(good_base, base, ones)
        exp_logits = torch.where(good_base, exp_logits, ones * map_inf_to)
    return exp_logits / base

This implementation is slowed and fattened by n_blocks times. But it could be parallel with each section.
If there is no off-the-shelf function, should I write a CUDA/C++ library? I hope you could help with my issue.
For further generalization, I hope there are discontinuities in boundary/sections.
sections = torch.tensor([[ 0,  0,  0, -1,  2,  3,  2,  3,  0,  3]
                         [-1,  0,  0,  1,  2,  1,  2,  1, -1,  1]]
# [[000_232303]
#  [_0012121_1]]

Thank you for reading:)


title: Pytorch LSTM predicts too low values

body:
I followed the guide on https://stackabuse.com/time-series-prediction-using-lstm-with-pytorch-in-python/ in order to set up a LSTM model for predictions using torch.
Once imported the needed modules (torch, torch.nn, functional, optim, etc:) I set up my indexlist with quantities for 52 weeks.

idxlist = []

for x in range (0, len(inputslist)):
    idxlist.append(a - datetime.timedelta(weeks = x))

in_data = pd.DataFrame(columns=['date','qty'])
in_data.date = idxlist
in_data.qty = inputslist
all_data = in_data['qty'].values.astype(float)

Then, I defined the test size and set up a scaler to normalize data:

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler(feature_range=(-1,1))
train_data_normalized = scaler.fit_transform(train_data.reshape(-1,1))
train_data_normalized = torch.FloatTensor(train_data_normalized).view(-1)
#train window
train_window = 12

Then I created sequences:
def create_inout_sequences(input_data,tw):
    inout_seq = []
    L = len(input_data) ##40
    for i in range(L-tw): ##32 (se tw=8)
        train_seq = input_data[i:i+tw]
        train_label = input_data[i+tw:i+tw+1]
        inout_seq.append((train_seq ,train_label))
    return inout_seq

train_inout_seq = create_inout_sequences(train_data_normalized, train_window)
print(len(train_inout_seq)) #len(train) - train_w

And set up the Net class for the LSTM model. After optimizing with model=LSTM(), optim.Adam and model.eval(), I obtain the inverse transform with:

actual_predictions = scaler.inverse_transform(np.array(test_inputs[train_window:] ).reshape(-1, 1))

However, plotting the predicted values of last 12 weeks with the actual values I obtain:
Graph.
Why is this prediction so much below the actual values, even if periodic?


title: Does all operators in Pytorch support back-propagation?

body:
I remember that in TensorFlow, there are some operators can only be used for forward operations, and no backward operations being supported/implemented for back-propagation.
My question is, are there any these kind of operators in Pytorch? If yes, could you please give some references? Thanks in advance.


title: NotImplementedError when designing transformer class/nlp/pytorch

body:
I encountered a confusing bug when I'm doing machine translation task using nn.Transformer. Below is my code showing definition of transformer:
class MyTrans(nn.Module):
    def __init__(
        self,
        d_model,
        vocab_size,
        padding_key,
        nhead,
        num_encoder_layers,
        num_decoder_layers,
        dim_feedforward,
        dropout,
        max_len,
        device,
    ):
        super(MyTrans, self).__init__()
        self.device = device
        self.positionalencoding = PositionalEncoding(d_model, dropout=dropout, max_len=max_len)
        self.embedding = nn.Embedding(vocab_size, d_model)
        self.padding_key = padding_key
        self.transfomer = nn.Transformer(
            d_model,
            nhead,
            num_encoder_layers,
            num_decoder_layers,
            dim_feedforward,
            dropout,
        )
        self.fc = nn.Linear(d_model, vocab_size)
        self.dropout = nn.Dropout(dropout)

    def padding_masks(self, src_tgt_memory):
        mask = src_tgt_memory.transpose(0, 1) == self.padding_key
        return mask.to(self.device)

    def square_masks(self, src_tgt):
        mask = self.transformer.generate_square_subsequent_mask(src_tgt.size()[1])
        return mask.to(self.device)

    def forward(self, src, tgt):
        S, N = src.size()
        T, N = tgt.size()
        # N = 64, src_seq = S, tgt_seq = T
        embed_src = self.embedding(src)
        new_src = self.positionalencoding(embed_src)
        embed_tgt = self.embedding(tgt)
        new_tgt = self.positionalencoding(embed_tgt)

        src_key_padding_mask = self.padding_masks(src)
        tgt_mask = self.square_masks(tgt)

        output = self.transfomer(
            src = new_src,
            tgt = new_tgt,
            src_key_padding_mask = src_key_padding_mask,
            tgt_mask = tgt_mask,
        )
        output = self.fc(output)
        output = F.log_softmax(output, dim=-1)

        return output

Then when I run the training process using this model, it gives NotImplementedError:
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _forward_unimplemented(self, *input)
    173         registered hooks while the latter silently ignores them.
    174     """
--> 175     raise NotImplementedError
    176
    177

NotImplementedError:

I know that this error comes up when there is some indentation or typo problem, but I checked in detail that there is no such problem. This bug has been disturbing me the whole day and I still can't fix it.


title: pytorch - visualization of epoch versus error

body:
I'm trying to learn pytorch, moving over from tensorflow.
Following a tutorial that used this code
import torch
import torch.nn as nn
import numpy as np
import matplotlib as plt
from sklearn import datasets
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# 0) Prepare data
bc = datasets.load_breast_cancer()
X, y = bc.data, bc.target

n_samples, n_features = X.shape

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1234)

# scale
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

X_train = torch.from_numpy(X_train.astype(np.float32))
X_test = torch.from_numpy(X_test.astype(np.float32))
y_train = torch.from_numpy(y_train.astype(np.float32))
y_test = torch.from_numpy(y_test.astype(np.float32))

y_train = y_train.view(y_train.shape[0], 1)
y_test = y_test.view(y_test.shape[0], 1)

# 1) Model
# Linear model f = wx + b , sigmoid at the end
class Model(nn.Module):
    def __init__(self, n_input_features):
        super(Model, self).__init__()
        self.linear = nn.Linear(n_input_features, 1)

    def forward(self, x):
        y_pred = torch.sigmoid(self.linear(x))
        return y_pred

model = Model(n_features)

# 2) Loss and optimizer
num_epochs = 100
learning_rate = 0.01
criterion = nn.BCELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

# 3) Training loop
for epoch in range(num_epochs):
    # Forward pass and loss
    y_pred = model(X_train)
    loss = criterion(y_pred, y_train)

    # Backward pass and update
    loss.backward()
    optimizer.step()

    # zero grad before new step
    optimizer.zero_grad()

    if (epoch+1) % 10 == 0:
        print(f'epoch: {epoch+1}, loss = {loss.item():.4f}')

with torch.no_grad():
    y_predicted = model(X_test)
    y_predicted_cls = y_predicted.round()
    acc = y_predicted_cls.eq(y_test).sum() / float(y_test.shape[0])
    print(f'accuracy: {acc.item():.4f}')

Trying to visualize the outputted model, and to visualize how similar the data input to output is, I'm having a bit of problems simply using matploblib,
predicted = model(X_test).detach().numpy()

plt.plot(X_test, y_train, 'ro')
plt.plot(X_test, y_predicted, 'b')
plt.show()

What is a good way to visualize this data? Thanks in advance,
using Ivan's answer - trying to add the train_acc to the function,
train_loss, valid_loss, valid_acc, train_acc = [], [], [],[]
for epoch in range(num_epochs):
    # Forward pass and loss
    y_pred = model(X_train)
    loss = criterion(y_pred, y_train)

    # Backward pass and update
    loss.backward()
    optimizer.step()

    # zero grad before new step
    optimizer.zero_grad()

    train_loss.append(loss.item())

#     acc_init = model(X_test).round().eq(y_train).sum() / float(y_train.shape[0])

    train_acc.append(acc_init)
    with torch.no_grad():
        y_pred = model(X_test)
        loss = criterion(y_pred, y_test)
        y_pred_cls = y_pred.round()
        acc = y_pred_cls.eq(y_test).sum() / float(y_test.shape[0])

        valid_loss.append(loss.item())
        valid_acc.append(acc.item())

    if (epoch+1) % 10 == 0:
        print(f'epoch: {epoch+1}, loss = {loss.item():.4f}')
        print(f'valid loss: {loss.item():.4f} valid acc: {acc.item():.4f}')


title: volatile was removed and now has no effect. Use `with torch.no_grad():` instead. inputs_Var = Variable(inputs, volatile=True)

body:
from torch.autograd import Variable

def test(testloader, net, index, epoch, use_cuda, logger):
    losses = AverageMeter()

net.eval()

for batch_idx, (inputs, targets) in enumerate(testloader):
    if use_cuda:
        inputs = inputs.cuda()

    inputs_Var = Variable(inputs, volatile=True)
    outputs = net(inputs_Var, index)

    # measure accuracy and record loss
    losses.update(outputs.item(), inputs.size(0))

==> UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead.
inputs_Var = Variable(inputs, volatile=True)
Use
with torch.no_grad():
    inputs_Var = Variable(inputs, volatile=True)
    outputs = net(inputs_Var, index)

but ==> UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead. inputs_Var = Variable(inputs, volatile=True)


title: how to load rdf file by virtuoso or other graph database?

body: I want to load freebase rdf file. but no documents. so I want to know how to load rdf file to virtuoso graph database, or other graph database.


title: pytorch custom urdu dataloader (text and images)

body:
I am new on pytorch lib and trying to load a dataset i acquire online. Its a OCR in Urdu language. the structure of the dataset is
-Train_image:
-name1.jpg
-name2.jpg
-train_labels.csv
-column 1 (Files name)
-name1
-name2
-column 2 (text in the corrosponding file)
-label1 (urdu text)
-label2 (urdu text)
now how can i load this data to train a CNN and A RNN for detection and recognition?


title: How to attach list of constants to a run in tensorboard?

body:
I'm logging my pytorch DNN experiments with tensorboard, it would be really nice if there was some way to programmatically attach a list of constants (eg. hyperparams) to each run (and maybe even notes?) so that I can see easily see what effect different constants have had. Is this possible?
Edit: add_text kind of does this but it's still a step-oriented thing rather than a run-oriented thing. so browsing 20 hyperparameters over a dozen experiments would involve a lot of scrolling:

I'd really like to be able to see them all at a glance, maybe in the hover dialog whilst inspecting the curves - maybe this isn't possible and I should be making a feature-request rather than an SO question?


title: How can i detect if a callback is triggered in pytorch?

body: I am fine-tuning a BERT model. First, I want to freeze layers and train a bit. When a certain callback is triggered (let's say ReduceLROnPlateau) I want to unfreeze layers. How can I do it?


title: Scikit one-hot-encoding with very expensive cost function neural network deep learning

body:
I have a client who want me to build a NN/DL app for selecting a model for a very expensive cost (minutes to hours of runtime). The cost function is a non linear regression model of a pretty big data set (1000's - 10'000's of records), quasi-Newton minimization.  The NN/DL would select the structure of the model (e.g.,1, 2,3, 4 5 ODEs)- no closed form solution to the ODEs, parameters estimated by quasi Newton minimization. Can one hot encoding and NN/DL (e.g., Pytorch) do this? Can it be parallelized?
thanks
Mark


trdg -c 1000000 -w 5 -f 64

When I'm trying to run the above command line it's throwing error like
OSError: [Errno 22] Invalid argument: 'out/megaphotographic Okarche angiotonase inevitableness it"ll_3955.jpg'

I'm using python package TextRecognitionDataGenerator to generate images


title: Type Error while using CoNLL data format for NER in simpletransformers

body:
I am trying to build a custom NER with about 530 Mb of data.
I had used following code to implement it using simpletransformers.
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import pandas as pd
from simpletransformers.ner import NERModel,NERArgs
import os

label = ['B-ORG', 'I-ORG', 'B-PER', 'I-PER']

args = NERArgs()
args.num_train_epochs = 10
args.learning_rate = 0.001
args.overwrite_output_dir =True
args.train_batch_size = 32
args.eval_batch_size = 32
args.lazy_loading = True

model = NERModel('roberta', 'roberta-base',labels=label, args =args, use_cuda=True)

model.train_model('a.txt',eval_data = 'b.txt', acc = accuracy_score)

I am using args.lazy_loading=True to resolve memory issue.
But is giving following error:
TypeError: convert_example_to_feature() missing 14 required positional arguments: 'label_map', 'max_seq_length', 'tokenizer', 'cls_token_at_end', 'cls_token', 'cls_token_segment_id', 'sep_token', 'sep_token_extra', 'pad_on_left', 'pad_token', 'pad_token_segment_id', 'pad_token_label_id', 'sequence_a_segment_id', and 'mask_padding_with_zero'

Sample input text in CoNLL format:
a B-PER
b I-PER
c I-PER

f B-ORG
g I-ORG
h I-ORG

Link for reference: https://simpletransformers.ai/docs/ner-specifics/


title: Tensorboard with PyTorch : toggle runs not working

body:
I am using Tensorboard 2.3.0 with PyTorch summary writer.
As I run multiple training, I stored multiple log files in a directory such that
runs
├── test_1
│   └── events.out.tfevents.1614979424.MacBook-Pro-15inch.local.22634.7
├── test_2
│   └── events.out.tfevents.1614979441.MacBook-Pro-15inch.local.22634.8
├── test_3
    └── events.out.tfevents.1614991192.MacBook-Pro-15inch.local.22634.10

...(more logs)

The problem is that when I run tensorboard --logdir runs, it correctly shows all logs but I want to hide certain logs for example only test_1.
I think this should be possible by the toggle button at the left bottom corner but toggling it does not change the graph at all (it keeps showing all logs in the directory).


title: How to build a data loader from list in pytorch

body:
I got one list of images and another list of labels.
Image: shape(64, 64, 3)
Label: shape(64,). actually, labels are 64-dimensional continuous values.
import pickle

file_name = "trainData.pkl"
open_file = open(file_name, "rb")
loaded_list = pickle.load(open_file)
open_file.close()

print(loaded_list[0])
print(loaded_list[0].shape)

file_name = "trainLabel.pkl"
open_file = open(file_name, "rb")
loaded_list = pickle.load(open_file)
open_file.close()

print(loaded_list[0])
print(loaded_list[0].shape)


title: Weight argument of Loss Function

body:
Suppose I have a training set which consists of 4 classes and the number of samples belonging to the 4 classes is 20, 30, 40, 10 respectively. So should I pass the tensor torch.tensor([20,30,40,10]) / 100. to the weight argument of the loss function?
Or should I calculate the values of the weight argument for each batch on the fly in the training loop?


title: tuple unpacking in X_test

body:
i am following a simple tutorial in PyTorch. The tutorial is using the diabetes dataset and built a simple two-layer network. In some part of the tutorial there exist the following part of the code where there exists an iteration on the X_test using enumerate.
predictions=[]
with torch.no_grad():

    for i, data in enumerate(X_test):
        y_pred=model(data)

        predictions.append(y_pred.argmax().item())

        print(y_pred.argmax().item())

while I thought that I was understanding enumerate I am a little confused. What the i and data referring to in the above code?
in general, I understandnd that is a typle unpacking procedure


title: Correct usage of `CrossEntropyLoss` with batches of multidimentional samples in Pytorch

body:
I am trying to train a LSTM for per-frame classification like so
self._lstm1 = nn.LSTM(input_size=16, hidden_size=50, num_layers=1, batch_first=True)
self._log_probs = nn.Linear(50, 4)
self._criterion = nn.CrossEntropyLoss(reduction='mean', weight=self._class_weights_map)

where self._class_weights_map == tensor([0.1340, 0.1181, 0.0000, 0.7479], device='cuda:0', dtype=torch.float64)
The training reaches the lines
label_batch_int = torch.argmax(label_batch_one_hot.squeeze(), dim=-1)
loss = self._criterion(log_probs, label_batch_int)

where label_batch_one_hot.shape == (16, 10, 4) [16 samples per batch, 10 frames per sample, 4 log-probs per frame]. Thus label_batch_int.shape == (16, 10) [16 samples per batch, 10 prediction ints per sample, each in the range of 0 to 3
and where log_probs.shape == (16, 10, 4)
The criterion is then fed log_probs.shape == (16, 10, 4) and label_batch_int.shape == (16, 10) and gives

{ValueError}Expected target size (16, 4), got torch.Size([16, 10])

How to correctly use CrossEntropyLoss in this multi-samples-per-batch case?


title: try to revise the inplace operation error:one of the variables needed for gradient computation has been modified by an inplace operation:

body:
I have two agents a and b:they use the same network stucture.and I try to use replaybuffer to update the parameters of networks.If i update a, then update b would bring the error:one of the variables needed for gradient computation has been modified by an inplace operation. so I try to find the inplace operation.But I couldn't do that.
    while total_timesteps < episodes:
    total_timesteps = total_timesteps + 1
    episodes_reward = []
    state = env.reset()
    state.to(device)
    for j in range(max_steps):
        action = torch.zeros(env.num_agents, env.dim_world)
        action[0] = policy_agent1.select_action(state)
        action[1] = policy_agent2.select_action(state)
        action_agent0 = action[0]
        action_agent1 = action[1]
        next_state, reward, done = env.step(action)
        next_state.to(device)
        action.to(device)
        reward.to(device)
        reward_agent = reward[0]
        reward_agent1 = reward[1]
        replay_buffer1.add((state, next_state, action_agent0, reward_agent, done))
        replay_buffer2.add((state, next_state, action_agent1, reward_agent1, done))
        # print(next_state, reward, action)
        state = next_state
        if done:
            continue
    policy_agent1.train(replay_buffer1.sample(batch_size), gamma)
    policy_agent2.train(replay_buffer2.sample(batch_size), gamma)

and the train part:
    def train(self, replay_buffer, gamma):
    state, next_state, action, reward, done = replay_buffer
    q = torch.zeros(len(reward)).to(device)
    q_ = torch.zeros(len(reward)).to(device)
    q_target = torch.zeros(len(reward)).to(device)
    done = torch.Tensor(done).to(device)

    for j, r in enumerate(reward):
        q1_target = torch.zeros(len(reward)).to(device)
        q_[j] = self.critic_network(torch.transpose(next_state[j].to(device), 0, 1), self.actor_network(torch.transpose(next_state[j].to(device), 0, 1)).view(1, 1))
        q_target[j] = r.to(device) + (done[j] * gamma * q_[j])
        q1_target = q1_target + q_target
        # q_target[j] = r.to(device) + (done[j] * gamma * q_[j]).detach().clone()
        q1 = torch.zeros(len(reward)).to(device)
        q1[j] = self.critic_network(torch.transpose(state[j].to(device), 0, 1), action[j].view(1, 1).to(device))
        q = q + q1
    loss_critic = F.mse_loss(q, q1_target)
    self.critic_optimizer.zero_grad()
    loss_critic.backward(retain_graph=True)
    self.critic_optimizer.step()

    b = torch.zeros(len(reward)).to(device)
    for j, _ in enumerate(reward):
        b1 = torch.zeros(len(reward)).to(device)
        b1[j] =  self.critic_network(torch.transpose(state[j].to(device), 0, 1), self.actor_network(torch.transpose(state[j].to(device), 0, 1)).view(1, 1))
        b = b + b1
        # b[j] = self.critic_network(torch.transpose(state[j].to(device), 0, 1), self.actor_network(torch.transpose(state[j].to(device), 0, 1)).view(1, 1))
    loss_actor = -torch.mean(b)
    self.actor_optimizer.zero_grad()
    loss_actor.backward(retain_graph=True)
    self.actor_optimizer.step()

I also have changed some place where I think have some problems.so it will looks prolix.


title: A language model with only one embedding layer in both encode and decode only predict

body:
I'm trying to make the model predict a word from a sentence using pretrained Huggingface's BERT as feature extractor. The model look like this
class BertAutoEncoder(nn.Module):
    def __init__(self, vocab_size):
        super().__init__()
        decoder_layer = nn.TransformerDecoderLayer(768, 2, 1024, dropout=0.1)
        self.transformer_decoder = nn.TransformerDecoder(decoder_layer, 2)
        self.fc = nn.Linear(768, vocab_size)

    def forward(self, memory, embedded_word):
        output = self.transformer_decoder(embedded_word, memory)
        output = self.fc(output)
        return output

And when train/evaluate I call the model like this
bert = BertModel.from_pretrained('bert-base-uncased')
bert.requires_grad_(False)
...
memory = bert(**src).last_hidden_state.transpose(0, 1)
embeded_word = bert.embeddings(trg.data['input_ids'][:, :-1], token_type_ids=trg.data['token_type_ids'][:, :-1]).transpose(0, 1)
output = model(memory, embeded_word)

The loss reduced nicely but turned out the model only predict <eos> token.
I tried train the model with 1 batch of 32 samples and it did work when loss reduced pass 8e-6 but when I trained it with all data the loss could go way beyond that but none of the saved models work. Even the one with eval or train loss around 4e-6 - 8e-6.
Surprisingly the model would work if I use a separate decoder's Embedding like this
class BertAutoEncoderOld(nn.Module):
    def __init__(self, vocab_size):
        super().__init__()
        decoder_layer = nn.TransformerDecoderLayer(768, 2, 1024, dropout=0.1)
        self.transformer_decoder = nn.TransformerDecoder(decoder_layer, 2)
        self.decoder = nn.Embedding(vocab_size, 768)
        self.pos_decoder = PositionalEncoding(768, 0.5)
        self.fc = nn.Linear(768, vocab_size)

    def forward(self, memory, word):
        tgt = self.decoder(word.data['input_ids'][:, :-1].transpose(0, 1))
        tgt = self.pos_decoder(tgt)
        output = self.transformer_decoder(tgt, memory)
        output = self.fc(output)
        return output

But I was asked to make it work with one Embedding and I have no idea how.
I tried

Reduce/increase batch from 32 to 8-64
Also tried 2 and 1024 batch size
Remove <eos> token and change it's attention mask to 0

But none of those work.
What did I do wrong and how to fix it?
Thanks
Edit per @emily qeustion
I change the data itself in collate function
text.data['attention_mask'][text.data['input_ids'] == 102] = 0
text.data['input_ids'][text.data['input_ids'] == 102] = 0
word.data['attention_mask'][word.data['input_ids'] == 102] = 0
word.data['input_ids'][word.data['input_ids'] == 102] = 0

It only used in Bert though.


title: Pytorch tensor averaging of spitted chunks in an axis

body:
I'm looking for an efficient way of performing averaging of a tensor, by chunking that is predefined by another masking tensor:
We are given a tensor of size [b_size,main_ax,sub_ax] and a masking binary tensor of size [b_size,main_ax] (contains only 1s and 0s values), where the last has the constant sum of number_of_ones along the second axis, meaning the sum of every column is always number_of_ones. The output tensor will be of a size of [b_size,number_of_ones,sub_ax]. Our goal is to average the first tensor according to the chunks defined by the second, meaning each chunk is defined by a sequence, starting from 1, containing 0s, until we encounter the next 1.


title: Pytorch Unfold Causing Unexpected Detach

body:
I implemented this layer that's supposed to be carrying out complex convolution with no shared weights.
import torch
import torch.nn.functional as F
class ComplexLinear(Module):

    def __init__(self, in_features, out_features):
        super(ComplexLinear, self).__init__()
        self.fc_r = Linear(in_features, out_features)
        self.fc_i = Linear(in_features, out_features)

    def forward(self,input_r, input_i):
        return self.fc_r(input_r)-self.fc_i(input_i), \
               self.fc_r(input_i)+self.fc_i(input_r)

class ComplexLocalConv2D(Module):

    def __init__(self, in_size, kernel_size):
        super(ComplexLocalConv2D, self).__init__()
        self.kernels=torch.nn.ModuleList([ComplexLinear(kernel_size**2,1) for _ in range((in_size-kernel_size+1)**2)])
        self.in_size=in_size
        self.kernel_size=kernel_size

    def forward(self,xr,xi):
        xr=F.unfold(xr.view(1,1,self.in_size,self.in_size),(self.in_size-self.kernel_size+1))
        xi=F.unfold(xi.view(1,1,self.in_size,self.in_size),(self.in_size-self.kernel_size+1))
        xr1=[]
        xi1=[]
        for i in range((self.in_size-self.kernel_size+1)**2):
            x1=self.kernels[i].forward(xr[0,i],xi[0,i])
            xr1.append(x1[0])
            xi1.append(x1[1])
        xr1=torch.tensor(xr1)
        xi1=torch.tensor(xi1)
        return xr1,xi1

ComplexLinear is just a wrap-up of Linear to compute complex numbers, taking two tensors for real and imaginary parts as input.
But when I apply this layer in a network, it seems, however, not to preserve gradients, resulting in an error as the code below is executed
class Net(Module):
    def __init__(self):
        super(Net, self).__init__()
        self.linear3=ComplexLocalConv2D(2,2)
        self.linear2=ComplexLocalConv2D(3,2)
        self.linear1=ComplexLocalConv2D(4,2)

    def forward(self,xr,xi):
        xr1,xi1=self.linear1(xr,xi)
        xr2,xi2=self.linear2(xr1,xi1)
        xr3,xi3=self.linear3(xr2,xi2)
        return torch.atan2(xi3,xr3)

model=Net().double()
x=torch.tensor([[1.,2.,3.,4.],[1.,2.,3.,4.],[1.,2.,3.,4.],[1.,2.,3.,4.]]).double()
loss_func=torch.nn.MSELoss()
model.zero_grad()
result=model(x,x)
loss=loss_func(result.unsqueeze(0),torch.tensor([[1.]]).double())
loss.backward()

Resulting in
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

This should be meaning that the layer above did not preserve gradient, which is weird because unfold should have been used in Conv2D and did not lead to this error. Any help is welcome.


title: Problems about realization of VAE with Pytorch

body:
I am learning a VAE pytorch code. The dataset used is mnist.
What the decoder outputs is,
**return**  Independent(Normal(loc=x, scale = self.scale), 1)

So, the dimension of the output is a vector of 28 * 28 * batch_size
While, when it calculates the first term of ELBO, it uses log_prob,
p_x_given_z = self.decoder(z)
reconstruct_loss = -p_x_given_z.log_prob(x.view(shape[0], -1))

Now x_shape is a matrix of batch_size * (28 * 28), however, the p_x_given_z, the decoder output, is a vector of 28 * 28 * batch_size.
The dimensions of both are different, how does the calculation work?


title: unfold().unfold() and unfold() meaning

body:
I found a code and I don't understand these lines :
self.key_conv = nn.Conv2d(in_channels, out_channels, kernel_size=1, bias=bias)
k_out = self.key_conv(padded_x)
k_out = k_out.unfold(2, self.kernel_size, self.stride).unfold(3, self.kernel_size, self.stride)

of course, these codes are not Continuous and they are part of code.
my question is about unfold function, this function whats doing exactly and whats meaning unfold().unfold()?
thanks a lot for your attention.


title: Colab: programmatically restart runtime and run all notebook cells when cuda runs out of memory?

body: Is there a way to programatically catch a cuda out of memory runtime error and then restart and run all? My model can run about 5 runs before cuda runs out of memory and i'm having to manually stop the process and restart & run all cells.


title: Efficient way to convert Pytorch dataset to loader/sampler that samples at least one point from each class

body:
I'm trying to think of a fast and efficient way to convert a Pytorch dataset into a sampler that samples at least one of each class. So far I've iterated through the Pytorch dataset sequentially and creates a 2D tensor for each class (Batch x Feature) then returns random idxes of each tensor, but that's pretty slow and doesn't leverage a dataloader or sampler. I'm trying to think of a way to set this up into a loader or sampler to be faster.
Goal:
So each batch sampled should have at least one representative of each class and it's guaranteed that the batch size given to the sampler will be much larger than the number of classes (Batch size will be at least 2x as large as classes). Ideally each batch shouldn't be sampled with replacement, but a point can be sampled in different batches.


title: Target size and input size not matching despite "minibatches" tells otherwise

body:
I'm relatively new to Pytorch and have been training an AutoEncoder model on the MNIST data set. Before training the model, I have three dataloaders for training-, validation- and test sets.
train_loader = DataLoader(train_set, batch_size=batch_size, shuffle=True)
valid_loader = DataLoader(valid_set, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(test_set, batch_size=batch_size, shuffle=True)

# get minibatch
x_train, _ = next(iter(train_loader))
x_val, _ = next(iter(valid_loader))
x_test, _ = next(iter(test_loader))

The three minibatches have the following sizes:
torch.Size([128, 784])
torch.Size([128, 784])
torch.Size([128, 784])

However, when I run the training loop (in the validation phase), the shapes of validation data does not match and I get the following error:
ValueError: Using a target size (torch.Size([96, 784])) that is different to the input size (torch.Size([128, 784])) is deprecated.
The simple model looks as
class AE(nn.Module):
def __init__(self,latent_dim):
    super(AE, self).__init__()
    ### Encoder layers
    self.fc_enc1 = nn.Linear(784, 32)
    self.fc_enc2 = nn.Linear(32, 16)
    self.fc_enc3 = nn.Linear(16, latent_dim)

    ### Decoder layers
    self.fc_dec1 = nn.Linear(latent_dim, 16)
    self.fc_dec2 = nn.Linear(16,32)
    self.fc_dec3 = nn.Linear(32,784)

def encode(self, x):
    z = F.relu(self.fc_enc1(x))
    z = F.relu(self.fc_enc2(z))
    z = F.relu(self.fc_enc3(z))

    return z

def decode(self, z):
    xHat = F.relu(self.fc_dec1(z))
    xHat = F.relu(self.fc_dec2(xHat))
    xHat = F.sigmoid(self.fc_dec3(xHat))

    return xHat

def forward(self, x):
    ### Autoencoder returns the reconstruction and latent representation
    z = self.encode(x)

    ### decode z
    xHat = self.decode(z)
    return xHat, z

The training loop looks as following:
AEmodel = AE(latent_dim).to(device)
optimizer = optim.Adam(AEmodel.parameters(), lr=lr)
loss_function = nn.BCELoss()

for epoch in range(1, epochs + 1):
AEmodel.train()
train_loss = 0
for batch_idx, (data, _) in enumerate(train_loader):
    data = data.float().to(device)
    optimizer.zero_grad()
    xHat, z = AEmodel(data)
    loss = loss_function(xHat, data)
    loss.backward()
    train_loss += loss.item()
    optimizer.step()

AEmodel.eval()
valid_loss = 0
with torch.no_grad():
    for i, (data, _) in enumerate(valid_loader):
        data = data.float().to(device)
        valid_loss += loss_function(xHat, data).item()

The error occurs in the last line of the above code. I have not been able to figure out, where the reshaping appears, which causes some mismatching. Am I blind and not seeing an obvious mistake??


title: Merging two or more models in PyTorch

body:
Is it possible to merge many models together, whilst maintaining the same overall model structure?
For example, I have three models trained on separate parts of the same dataset. I would like to then merge them into one model that maintains the same structure of the model (so I can use the same scripts etc.).
Is it possible to do so?
Would the model be worse than if I trained one single model?
Would this scale?
Many thanks in advance


title: Does Output channel of conv2d to the next input shape of conv2d can be different?

body:
In the below code, you can see in, first nn.Sequential/conv2d, it has 512 output, but it next nn.Sequential/Convtranspose it takes 1024 input channel. I wonder how it is possible? i didn't really get how this 512 output channel is fed into 1024 of the input channel of the next nn.Seqential.
self.face_decoder_blocks = nn.ModuleList([
        nn.Sequential(Conv2d(512, 512, kernel_size=1, stride=1, padding=0),),   #1, 1

        nn.Sequential(Conv2dTranspose(1024, 512, kernel_size=3, stride=1, padding=0), # 3,3
        Conv2d(512, 512, kernel_size=3, stride=1, padding=1, residual=True),),

        nn.Sequential(Conv2dTranspose(1024, 512, kernel_size=3, stride=2, padding=1, output_padding=1),
        Conv2d(512, 512, kernel_size=3, stride=1, padding=1, residual=True),
        Conv2d(512, 512, kernel_size=3, stride=1, padding=1, residual=True),)]

and inside-forward function,
x = face_sequences
for f in self.face_encoder_blocks:
   x = f(x)

My Main question is stated above, please help me to understand this.
thanks in advance.


title: pytorch - is it possilble to autocalculate shapes of layer?

body:
Ok i got for example this NN:
class Test(nn.Module):
    def __init__(self, pool_size=(4, 4)):
        super(Test, self).__init__()

        self.conv1 = nn.Conv2d(3, 16, 3, padding=1)
        self.conv2 = nn.Conv2d(16, 32, 3, padding=1)
        self.conv3 = nn.Conv2d(32, 64, 3, padding=1)
        self.maxpool = nn.AdaptiveMaxPool2d(pool_size)
        self.fc1 = nn.Linear(64*pool_size[0]*pool_size[1], 500)
        self.fc2 = nn.Linear(500, 10)

    def forward(self, x):
        out = F.relu(self.conv1(x))
        out = F.relu(self.conv2(out))
        out = F.relu(self.conv3(out))
        out = self.maxpool(out)
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        out = self.fc2(out)
        return out

Can i dont calculate each lvl shape? just say 'Ok my input image is 32 * 32 * 3 = x,y,z
and define layers like
self.conv1 = nn.Conv2d(x, y, 3, padding=1)
...
self.fc=self.fc1 = nn.Linear(4*previous_shape, 500)
self.fc=self.fc1 = nn.Linear(2*previous_shape, 10)

I got problem with claculating this paramemters, or there is easy solution about it?


title: pytorch customdataset notimplemented error

body:
Im making my own custom datasets in pytorch.
AND, I would like to visualize the image.
however, i think something wrong is happening in custom datasets.
Please help me.

NotImplementedError                       Traceback (most recent call
last)  in ()
1 import matplotlib.pyplot as plt
2 dat= TrainDataset(transforms.ToTensor())
----> 3 img,label= dat[i]
4 plt.imshow(img.permute(1,2,0))
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataset.py in
getitem(self, index)
31
32     def getitem(self, index) -> T_co:
---> 33         raise NotImplementedError
34
35     def add(self, other: 'Dataset[T_co]') -> 'ConcatDataset[T_co]':
NotImplementedError:

import torch
import torch.nn as nn
import torch.optim as optim
import torchvision.transforms as transforms
import torchvision
from torch.utils.data import DataLoader
import os
import glob

from torch.utils.data import Dataset

import pandas as pd
from PIL import Image

class TrainDataset(Dataset):
  def __init__(self, transform):
    super().__init__()
    self.data = pd.read_csv('/content/drive/MyDrive/cancer/train_labels.csv')
    self.transform = transform

    def __len__(self):
      return len(self.data)

    def __getitem__(self,idx):
      img_name, label = self.data.iloc[idx]
      img = Image.open(f'/content/drive/MyDrive/cancer/test/{image_name}.tif')
      img = self.transform(img)
      return (img, torch.tensor(label).long())

import matplotlib.pyplot as plt
dat= TrainDataset(transforms.ToTensor())
img,label= dat[1]
plt.imshow(img.permute(1,2,0))


title: passing gradient with torch.autograd.backward()

body:
I'm trying to obtain the gradient of loss with respect to combine. The function critic is implemented separately. Without the combine, loss=critic(p) works perfectly with loss.backward(), so my issue is somewhere else. How can I troubleshoot an issue like this?
def ogp(p: Tensor, cp: Tensor, q: Tensor, cq: Tensor, critic: Callable[[Tensor], Tensor]) -> Tensor:
    p.requires_grad = True
    q.requires_grad = True
    eps = torch.randint(0, 1, p.shape, dtype = torch.float32)
    eps.requires_grad = True
    combine = eps * p + (1.0-eps) * q
    combine.retain_grad()
    loss = critic(combine)
    loss.retain_grad()
    torch.autograd.backward(loss, grad_tensors = torch.ones_like(loss), retain_graph=True, create_graph = True)
    grad =  eps * p.grad + (1.0-eps) * q.grad
    norms = (torch.norm(grad) - 1.0)**2
    return norms


title: Image classification, network not learning anything

body:
So i am deshuffleing images, where they have each 4 corners at a random possition.
EPOCHS = 5
activation_function = nn.LeakyReLU()

LossList, AccuracyList, tllist,trlist,bllist,brlist,= [], [], [], [], [], []

model = MyModel(1, activation_function).to("cuda")

optimizer = torch.optim.Adam(model.parameters())
loss_objective = nn.MSELoss()

for epoch in range(EPOCHS):

    model.train()

    for x, y in training_generator:
        optimizer.zero_grad()
        predictions = model(x)
        loss = loss_objective(predictions.to("cuda"),y.type(torch.FloatTensor).to("cuda"))
        loss.backward()
        LossList.append(loss.item())
        optimizer.step()
    with torch.no_grad():

        model.eval()
        tl,tr,bl,br,acc = model.evaluate(test_generator)
        AccuracyList.append(acc)
        tllist.append(tl)
        trlist.append(tr)
        bllist.append(bl)
        brlist.append(br)

    if epoch % 10 == 0 and epoch != 0: clear_output()

    print(
    f'Epoch {epoch+1}, \n '
    f'Loss: {LossList[-1]}, \n '
    f'Full Accuracy: {AccuracyList[-1]}, \n '
    f'Top left Accuracy: {tllist[-1]}, \n '
    f'Top right Accuracy: {trlist[-1]}, \n '
    f'Bottom left Accuracy: {bllist[-1]}, \n '
    f'Bottom right Accuracy: {brlist[-1]}'

    )
    print("-"*50)

The problem i have is that the model(VGG-11 with Batch Normalization after the convolutional layers) does not learn anything, and the accuracy stays at around 4% for correct full image, and 25% for each corner. The size of predictions is (batch_size, 4) and for labels its the same, with inegers between 0-3.
I think the problem is with how i'm suppose to make the model learn anything. Here is the model:
# @title Define model

class MyModel(nn.Module):
  def __init__(self, n_c, ActFunc):
    super(MyModel, self).__init__()

    self.layer1 = nn.Sequential(
        nn.Conv2d(3, 64, 3, padding=1),
        nn.BatchNorm2d(64),
        ActFunc,
        nn.MaxPool2d(2, 2)
    )

    self.layer2 = nn.Sequential(
        nn.Conv2d(64, 128, 3, padding=1),
        nn.BatchNorm2d(128),
        ActFunc,
        nn.MaxPool2d(2, 2)
    )

    self.layer3 = nn.Sequential(
        nn.Conv2d(128, 256, 3, padding=1),
        nn.BatchNorm2d(256),
        ActFunc
    )

    self.layer4 = nn.Sequential(
        nn.Conv2d(256, 256, 3, padding=1),
        nn.BatchNorm2d(256),
        ActFunc,
        nn.MaxPool2d(2, 2)
    )

    self.layer5 = nn.Sequential(
        nn.Conv2d(256, 512, 3, padding=1),
        nn.BatchNorm2d(512),
        ActFunc
    )

    self.layer6 = nn.Sequential(
        nn.Conv2d(512, 512, 3, padding=1),
        nn.BatchNorm2d(512),
        ActFunc,
        nn.MaxPool2d(2, 2)
    )

    self.layer7 = nn.Sequential(
        nn.Conv2d(512, 512, 3, padding=1),
        nn.BatchNorm2d(512),
        ActFunc
    )

    self.layer8 = nn.Sequential(
        nn.Conv2d(512, 512, 3, padding=1),
        nn.BatchNorm2d(512),
        ActFunc,
        nn.MaxPool2d(2, 2)
    )

    self.layer9 = nn.Sequential(
        nn.Conv2d(512, 512, 3, padding=1),
        nn.BatchNorm2d(512),
        ActFunc
    )

    self.layer10 = nn.Sequential(
        nn.Conv2d(512, 512, 3, padding=1),
        nn.BatchNorm2d(512),
        ActFunc,
        nn.MaxPool2d(2, 2),
        nn.Flatten()
    )

    self.layer11 = nn.Sequential(
        nn.Linear(512, 4096),
        ActFunc
    )

    self.layer12 = nn.Sequential(
        nn.Linear(4096, 4096),
        ActFunc
    )

    self.layer12 = nn.Sequential(
        nn.Linear(4096, 4),
        ActFunc
    )

  def forward(self, x):
    l1 = self.layer1(x)
    l2 = self.layer2(l1)
    l3 = self.layer3(l2)
    l4 = self.layer4(l3)
    l5 = self.layer5(l4)
    l6 = self.layer6(l5)
    l7 = self.layer7(l6)
    l8 = self.layer8(l7)
    l9 = self.layer9(l8)
    l10 = self.layer10(l9)
    l11 = self.layer11(l10)
    l12 = self.layer12(l11)
    out = torch.argsort(l12).type(torch.FloatTensor).requires_grad_()

    return out

  def evaluate(self, test_generator):
      batchfullacc = 0
      batchacc = 0
      for images, labels in test_generator:
          predictions = self.forward(images)
          batchacc += (predictions.to("cuda") == labels).sum(dim=0)
          batchfullacc += (predictions.to("cuda") == labels).all(dim=1).sum()
      tl,tr,bl,br = batchacc/len(x_te)
      acc = batchfullacc/len(x_te)

      return tl,tr,bl,br,acc

The training set is on 2500 images. The output i get is(this is only on 5 epochs, but nothing changes with more epochs):
Epoch 5,
 Loss: 2.5899999141693115,
 Full Accuracy: 0.042250003665685654,
 Top left Accuracy: 0.2522500157356262,
 Top right Accuracy: 0.2407500147819519,
 Bottom left Accuracy: 0.25475001335144043,
 Bottom right Accuracy: 0.25600001215934753
--------------------------------------------------


title: Pytorch example from Deep Reinforcement learning in action running too slow

body:
I am running an example from the book Deep reinforcement learning in action; by Alexander Zai, Brandon Brown but my code runs extremely slowly and for the life of me I cannot understand why.
The only difference I can find from my example to its example is the fact that I am using a different game; however, slow move generations should not be the problem as I can use the same game with tabular methods and easily train it on 500000 samples in just a few minutes. But as soon as neural networks are involved Training only on 50000 takes more than a couple hours.
I am using both experience replay and a target network.
My code is as follows:
def train(nGames, lGames, batchSize, replaySize, learning_rate, syncFreq):
    epsilon = .1 # the initial exploration rate
    gamma   = .9 # the discount of the rewards

    replay    = deque(maxlen=replaySize) # initialize the experience replay buffer

    loss_fn = torch.nn.MSELoss()
    # This qFunction is defined further into the question
    optimizer = torch.optim.Adam(qFunction.parameters(), lr=learning_rate)

    j        = 0
    losses   = []

    # Actions that we actually send to the game
    actionIndexed = {0: "U", 1: "D", 2: "L", 3: "R", 4: "PASS"}

    for e in range(nGames):

        progressBar(e, nGames) # Just a progress bar function. Not relevant

        game  = DayGathering() # This is my own game
        state = game.observation() # This just extracts a dictionary with relevant information about the game

        # start playing the game
        i = 0
        done = False
        while not done:
            # agent taking an action with epsilon-greedy strategy
            if np.random.random() < epsilon:
                actionIndex = np.random.choice(range(5))
                action      = actionIndexed[actionIndex]
            else:
                # Select the action with the largest Q-value
                stateC      = converter(state)
                qValues     = qFunction(stateC).data.numpy()
                actionIndex = np.argmax(qValues)
                action      = actionIndexed[actionIndex]

            # advancing the state of the board
            # This functions just move to the next state
            game.playerTurn(action)
            game.boardTurn()
            newState = game.observation()

            # checking for rewards
            reward = rewardFunction(newState)

            # check if the current game is terminal. This happens if i = lGame
            # - 1 or if the agent is caught outside during night fall
            if i == lGames - 1 or outsideAtNight(newState):
                done = True
            else:
                done = False

            # saving the (state, action, newstate, reward) values in the experience replay buffer
            replay.append((stateC, actionIndex, newState, reward, done))

            # when we get to the batch size we train the network
            if len(replay) > batchSize:
                miniBatch     = random.sample(replay, batchSize)
                stateBatch    = torch.cat([sC for sC, _, _, _, _ in miniBatch])
                actionBatch   = torch.Tensor([actionIndex for _, actionIndex, _, _, _ in miniBatch])
                rewardBatch   = torch.Tensor([reward for _, _, _, reward, _ in miniBatch])
                newStateBatch = torch.cat([converter(newState) for _, _, newState, _, _ in miniBatch])
                doneBatch     = torch.Tensor([done for _, _, _, _, done in miniBatch])

                # use the target network to bootstrap
                with torch.no_grad():
                    newStateQ = qFunctionTarget(newStateBatch)

                # compute the predition of the networks for R + gamma*MaxRewards
                Y = rewardBatch + gamma * (1 - doneBatch)*torch.max(newStateQ,dim=1)[0]

                # Compute the discounted rewards using the first original network
                stateQ = qFunction(stateBatch).gather(dim=1, index=actionBatch.long().unsqueeze(dim=1)).squeeze()

                # compute the loss of the model and backpropagate
                loss = loss_fn(stateQ, Y.detach())
                optimizer.zero_grad()
                loss.backward()
                # keeping track of the losses
                losses.append(loss.item())
                optimizer.step()

                # periodically copy parameters to the target network
                if j % syncFreq == 0:
                    qFunctionTarget.load_state_dict(qFunction.state_dict())

            # advance the state of the game
            state = newState

            i += 1
            j += 1 # advance j
    return losses

That was the main training loop. Now for other things that appear in the previous code
The neural network
qFunction = torch.nn.Sequential(torch.nn.Linear(426, 1024),
                                torch.nn.ReLU(),
                                torch.nn.Linear(1024, 512),
                                torch.nn.ReLU(),
                                torch.nn.Linear(512, 5),
                                )
qFunctionTarget = copy.deepcopy(qFunction)
qFunctionTarget.load_state_dict(qFunction.state_dict())

The converter from observations to the ann input: I don't think this is very relevant bu I decided to include it any way. Basically the game is a grid world and we simply convert the observation to an array of zeros and ones indicating positions on the board. After creating this board we perturb it with noise.
def converter(observation):
    px, py = observation["player position"]
    playerBitBoard = torch.zeros([9, 9])
    playerBitBoard[py, px] = 1

    # making the bit boards for all apples
    a1x, a1y = observation["apple 1"]
    apple1BitBoard = torch.zeros([9, 9])
    apple1BitBoard[a1y, a1x] = 1

    a2x, a2y = observation["apple 2"]
    apple2BitBoard = torch.zeros([9, 9])
    apple2BitBoard[a2y, a2x] = 1

    a3x, a3y = observation["apple 3"]
    apple3BitBoard = torch.zeros([9, 9])
    apple3BitBoard[a3y, a3x] = 1

    # making the bit board for the key. Note that the key will have NONE values
    # in its position. They must be dealt with. Current solution put all values
    # in the key board equal to zero
    key = observation["key position"]
    keyBitBoard = torch.zeros([9, 9])
    if key:
        kx, ky = key
        keyBitBoard[ky, kx] = 1

    # making the time features
    t = observation["time until nigth"]
    timeBitBoard = torch.zeros(21)
    timeBitBoard[t] = 1

    # now we join all in a single array
    return torch.cat([playerBitBoard.reshape(81),
                      apple1BitBoard.reshape(81),
                      apple2BitBoard.reshape(81),
                      apple3BitBoard.reshape(81),
                      keyBitBoard.reshape(81),
                      timeBitBoard]).reshape((1, 426)) + torch.rand((1, 426))/100


title: Couldn't open shared file mapping error in Pytorch Dataset

body:
When training a model using a custom dataset in Pytorch 1.4 the following error is thrown after a seemingly random amount of epochs.
RuntimeError: Couldn't open shared file mapping: <torch_15324_2327643205>, error code: <1455>

The dataset is wrapped in a torch.utils.data.DataLoader and uses 4 workers, equal to the amount of physical cores.
class TSNDataSet(data.Dataset):
    def __init__(self, pickle_file_paths, transforms):
        self.pickle_file_paths = pickle_file_paths # list with file paths to pickle files
        self.dataset_size = len(pickle_file_paths)

    def __getitem__(self, index):
        with open(self.pickle_file_paths[index], 'rb') as f:
            mffs = pickle.load(f)

        return mffs, index

    def __len__(self):
        return self.dataset_size

It would be helpful to know what the error means and what the possible solutions are.


title: Semantic Segmentation runtime error at loss function

body:
I am using a costume model for segmentation (SETRModel). The model output shape is (nBatch, 256, 256) and the code below confirms it (note that the channel is squeezed out). The target shape is the same (It’s a PILMask).
When I start training, I get a runtime error (see below) related to the loss function. What am I doing wrong?
```
size = 480
half= (256, 256)
splitter = FuncSplitter(lambda o: Path(o).parent.name == 'validation')

dblock = DataBlock(blocks=(ImageBlock, MaskBlock(codes)),
               get_items=get_relevant_images,
               splitter=splitter,
               get_y=get_mask,
               item_tfms=Resize((size,size)),
               batch_tfms=[*aug_transforms(size=half), Normalize.from_stats(*imagenet_stats)])

dls = dblock.dataloaders(path/'images', bs=4)

model = SETRModel(patch_size=(32, 32),
            in_channels=3,
            out_channels=1,
            hidden_size=1024,
            num_hidden_layers=8,
            num_attention_heads=16,
            decode_features=[512, 256, 128, 64])

# Create a Learner using a custom model
loss = nn.BCEWithLogitsLoss()
learn = Learner(dls, model, loss_func=loss, lr=1.0e-4, cbs=callbacks, metrics=[Dice()])

# Let's test and make sure the loss function is happy with its inputs
learn.eval()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

t1 = torch.rand(4, 3, 256, 256).to(device)
print("input: " + str(t1.shape))

pred = learn.model(t1).to(device)
print("output: " + str(pred.shape))

# prints this:
# input: torch.Size([4, 3, 256, 256])
# output: torch.Size([4, 256, 256])

target = next(iter(learn.dls.train))[1]
target = target.type(torch.float32).to(device)
target.size(), pred.size()

# prints this:
# (torch.Size([4, 256, 256]), torch.Size([4, 256, 256]))

loss(pred, target)

# prints this:
# TensorMask(0.6844, device='cuda:0', grad_fn=<AliasBackward>)

# so, the loss function is happy with its inputs

learn.fine_tune(50)

# prints this:
# ---------------------------------------------------------------------------
# RuntimeError                              Traceback (most recent call last)
# <ipython-input-114-0e514c73651a> in <module>()
# ----> 1 learn.fine_tune(50)

# 19 frames
# /usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in         binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction,     pos_weight)
#    2827 pixel_shuffle = _add_docstr(torch.pixel_shuffle, r"""
#    2828 Rearranges elements in a tensor of shape :math:`(*, C \times r^2, H, W)` to a
# -> 2829 tensor of shape :math:`(*, C, H \times r, W \times r)`.
#    2830
#    2831 See :class:`~torch.nn.PixelShuffle` for details.

# RuntimeError: result type Float can't be cast to the desired output type Long


title: pyTorch: how to reset its state?

body:
I'm using pyTorch and for some reason - after refactoring my code - it's not working anymore. The error is:
  File ".\api\deepmatcher\data\dataset.py", line 420, in load_cache
    cached_data = torch.load(cachefile)
  File "c:\...\lib\site-packages\torch\serialization.py", line 592, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "c:\...\lib\site-packages\torch\serialization.py", line 851, in _load
    result = unpickler.load()
ModuleNotFoundError: No module named 'deepmatcher'

After some research I've learned that pyTorch is saving its state somehow (state_dict?), classes it uses and so on.
How can I reset pyTorch to its plain state, so I can run my code again? Maybe deleting some files on my computer (it's Win10 btw)? Or running some magical python command?


title: How to train a tensor of tensors in PyTorch

body:
I have a class model:
class Model(nn.Module)

That has 2 learnable parameters:
self.a = torch.nn.Parameter(torch.rand(1, requires_grad=True))
self.b = torch.nn.Parameter(torch.rand(1, requires_grad=True))

It also has a neural network class inside:
class Net_x(nn.Module)

So in order to train all the parameters I combine the learnable parameters "a, b" and the NN's params (under the "init" of class Model):
self.net_x = self.Net_X()
self.params = list(self.net_x.parameters())
self.params.extend(list([self.a, self.b]))

This works well and the network is training just fine.
My issue is that now I'm trying to change one of the parameters to be a tensor of tensors of parameters:
self.a = torch.tensor( [[ torch.nn.Parameter(torch.rand(1, requires_grad=True)) ] for i in range(5)] )

That is because at every time step (not epoch), I need to use a different parameter from self.a
Example:
for epoch in range(n_epochs):
    if timestep<5:
        val = 50 - a[0] * b
        loss = 10 - val
    elif timestep >=5 and timestep < 10:
        val = 50 - a[1] * b
        loss = 10 - val

The model runs without issues, but the parameters are not being updated (i.e they stay the same at every epoch).
P.S I would have added my code, but it's really long. I'm hoping the answer is simple (and if not I'll try to reduce my code and attach it)


title: ImportError when working with DynamicEdgeConv

body:
I am trying to move an iPython notebook from Google Collab to Gradient Notebooks.
During this process I encountered the following error:
ImportError: `DynamicEdgeConv` requires `torch-cluster`.

This method belongs to the torch_geometric module.
After looking through the documentation I found the following bit of code:
try:
    from torch_cluster import knn
except ImportError:
    knn = None

which together with
if knn is None:
    raise ImportError('`DynamicEdgeConv` requires `torch-cluster`.')

Further down in the documentation gives us this error.
I am wondering how you can resolve this. I have already downloaded and imported this torch_cluster module at the beginning of my code and it does not give any errors there, so it should be properly installed.


title: Converting a tf.dataset to a PyTorch Dataset?

body:
I'm working on this project where all the data comes preprocessed and ready as a tensorflow datasets which looks like this:
<MapDataset shapes: {input_ids: (128,), input_mask: (128,), label_ids: (), segment_ids: (128,)}, types: {input_ids: tf.int64, input_mask: tf.int64, label_ids: tf.int64, segment_ids: tf.int64}>

The script that I have is in PyTorch and takes in a Dataset object which looks like this:
Dataset({
    features: ['attention_mask', 'input_ids', 'label', 'sentence', 'token_type_ids'],
    num_rows: 12
})

is there anyway to convert one to the other? I'm quite new to both these API's so I'm not too sure how they work? Can I potentially convert one to the other using a dict?
Thank you


title: How does a convolutional layer convert a 3 channel input into a 1 channel output with 1 kernel?

body: I will like to know what kind of calculation does a program do to use a kernel to convert a three channel input into one channel. Thank you.


title: How to map RNN output to tensor of class size?

body:
I'm building a binary text classifier using GLOVE embeddings and RNN. The output of nn.RNN is torch.Size([1, 12, 150]). I need to map this to a dimension of size 2 so that i can calculate loss against the actual class. Im feeding tensor of size 'words' and 100 for glove embeddings of size 100d. E.g a 12 word sentence would be of size [1,12,100]
How is this done?
def forward(self, input, hidden):

    embeds = self.embedding(input) # glove embedding
    embeds = embeds.unsqueeze(0)
    embeds = embeds.float()

    output, hidden = self.rnn(embeds, hidden)
    # output is size [1, 12, 150]

    return output, hidden


title: How to include edge weights in the Pytorch Geometric SageConv layer?

body:
I built a custom graph dataset according to the instructions given here. My graph is a complete graph, every node has a feature vector and every edge has a feature vector.
I'm using the SageConv layer to build my model however in the forward function there doesn't seem to be an option to pass the edge weights.
How can I do this and if the SageConv doesn't support this what's the next best alternative?


title: Clear part of CUDA memory in Pytorch and Jupyter Notebook without resetting the kernel?

body:
I'm investigating the largest batch_size that doesn't cause a CUDA out of memory error. However, everytime this happens, I need to reset the Jupyter Notebook Kernel and re-run all previous code.
Is there a way to clear out just the most recent code execution that caused CUDA out of memory?
For example:
myfunc(TestDataLoader1)
myfunc(TestDataLoader2)
Cuda out of Memory now

How do I go back to being able to run the first function?


title: How do I proceed to load a ga_instance as ".pkl" format in PyGad?

body:
I have been trying to load the PyGad trained instance in another file, in order to make some prediction. But I have been having some problems in the loading process.
After the training phase, I saved the instance like this:
The saving function:
filename = 'GNN_CPTNet' #GNN_CPTNet.pkl
ga_instance.save(filename=filename)
The loading function:
loaded_ga_instance = pygad.load(filename=filename)
loaded_ga_instance.plot_result()
But, when I tried to load the instance in a new notebook or script, I could not load the instance, especially the "GNN_CPT Net.pkl" file.


title: A simple modification of labels prevents model fitting

body:
I am using a neural network (NN, PyTorch) consisting of several FC layers to solve a regression problem. Along with the MSE for assessing model performance, I also use person correlation (predicted vs ground truth) as an independent indicator of model performance. Although a problem is rather difficult, I was able to achieve satisfactory results using proper feature engineering. The loss and correlation are steadily improving for hundreds of epochs for both validation and train sets without any visible overfitting (e.g. correlation reaches up to 0.25). This is reproducible for various architectures and datasets.
The quantity NN predicts is actually a reward which I am planning to use for training an RL agent (for now it is not RL, it is a simple NN - I test if it can predict reward then it will be used for the critic in the RL). For this reason, the reward needs to be further modified depending on the action taken by the agent. The agent will have three binary actions.  For instance, if the agent takes action 0 then the reward becomes zero, if action = 1, then the reward = reward*VAL, and if action is 2, then the reward is = (-reward)*VAL. To summarize, I add three binary values and one real value(VAL) which dictates the linear modification of the reward (value to predict). So, once I append these new values to my NN input and try to predict the modified reward - the NN stops fitting.
I was thinking that this could be due to improper normalization, so I tried to normalize differently and even did minor feature engineering but it does not seem to affect performance. Currently the binary input (actions) takes values ([-0.01,0.01],[-0.01,0.01],[-0.01,0.01]).
Below I included images which demonstrate original and modified reward. The actions are selected randomly.

It feels I might be missing smth obvious. I would highly appreciate any help. Thanks!