title: How can I make this PyTorch tensor (B, C, H, W) tiling & blending code simpler and more efficient?

body:
So, I wrote the code below many months ago, and it's worked pretty well. Though I am struggling on how I can simplify it and make it more efficient.
The functions below split an image tensor (B, C, H, W) into equal sized tiles (B, C, H, W) and then you can do stuff individually to the tiles in order to save memory. Then when rebuilding the tensor from the tiles, it uses masks to ensure that the tiles are seamlessly blended back together. The 'special masks' in the masking function handle when tiles in the right most column or tiles in the bottom row can't use the same overlap as the other tiles. This means that the right edge tiles and the bottom tiles may sometimes have almost none of their content visible. This is done to ensure that the tiles are always the exact specified size, regardless of the original image/tensor's size (important for visualization/DeepDream, neural style transfer, etc...). The adjacent row/column to the edge row/column also has special masks as well for where they overlap with the edge row/column.
There are 8 possible masks for every tile, and 4 of those masks can be used at once. The 4 possible masks are left, right, top, and bottom, with a special version for each mask.
# Improved version of: https://github.com/ProGamerGov/neural-dream/blob/master/neural_dream/dream_tile.py
import torch

# Apply blend masks to tiles
def mask_tile(tile, overlap, side='bottom'):
    c, h, w = tile.size(1), tile.size(2), tile.size(3)
    top_overlap, bottom_overlap, right_overlap, left_overlap = overlap[0], overlap[1], overlap[2], overlap[3]

    base_mask = torch.ones_like(tile)

    if 'left' in side and 'left-special' not in side:
        lin_mask_left = torch.linspace(0,1,left_overlap, device=tile.device).repeat(h,1).repeat(c,1,1).unsqueeze(0)
        base_mask[:,:,:,:left_overlap] = base_mask[:,:,:,:left_overlap] * lin_mask_left
    if 'right' in side and 'right-special' not in side:
        lin_mask_right = torch.linspace(1,0,right_overlap, device=tile.device).repeat(h,1).repeat(c,1,1).unsqueeze(0)
        base_mask[:,:,:,w-right_overlap:] = base_mask[:,:,:,w-right_overlap:] * lin_mask_right
    if 'top' in side and 'top-special' not in side:
        lin_mask_top = torch.linspace(0,1,top_overlap, device=tile.device).repeat(w,1).rot90(3).repeat(c,1,1).unsqueeze(0)
        base_mask[:,:,:top_overlap,:] = base_mask[:,:,:top_overlap,:] * lin_mask_top
    if 'bottom' in side and 'bottom-special' not in side:
        lin_mask_bottom = torch.linspace(1,0,bottom_overlap, device=tile.device).repeat(w,1).rot90(3).repeat(c,1,1).unsqueeze(0)
        base_mask[:,:,h-bottom_overlap:,:] = base_mask[:,:,h-bottom_overlap:,:] * lin_mask_bottom

    if 'left-special' in side:
        lin_mask_left = torch.linspace(0,1,left_overlap, device=tile.device)
        zeros_mask = torch.zeros(w-(left_overlap*2), device=tile.device)
        ones_mask = torch.ones(left_overlap, device=tile.device)
        lin_mask_left = torch.cat([zeros_mask, lin_mask_left, ones_mask], 0).repeat(h,1).repeat(c,1,1).unsqueeze(0)
        base_mask = base_mask * lin_mask_left
    if 'right-special' in side:
        lin_mask_right = torch.linspace(1,0,right_overlap, device=tile.device)
        ones_mask = torch.ones(w-right_overlap, device=tile.device)
        lin_mask_right = torch.cat([ones_mask, lin_mask_right], 0).repeat(h,1).repeat(c,1,1).unsqueeze(0)
        base_mask = base_mask * lin_mask_right
    if 'top-special' in side:
        lin_mask_top = torch.linspace(0,1,top_overlap, device=tile.device)
        zeros_mask = torch.zeros(h-(top_overlap*2), device=tile.device)
        ones_mask = torch.ones(top_overlap, device=tile.device)
        lin_mask_top = torch.cat([zeros_mask, lin_mask_top, ones_mask], 0).repeat(w,1).rot90(3).repeat(c,1,1).unsqueeze(0)
        base_mask = base_mask * lin_mask_top
    if 'bottom-special' in side:
        lin_mask_bottom = torch.linspace(1,0,bottom_overlap, device=tile.device)
        ones_mask = torch.ones(h-bottom_overlap, device=tile.device)
        lin_mask_bottom = torch.cat([ones_mask, lin_mask_bottom], 0).repeat(w,1).rot90(3).repeat(c,1,1).unsqueeze(0)
        base_mask = base_mask * lin_mask_bottom

    # Apply mask to tile and return masked tile
    return tile * base_mask

def add_tiles(tiles, base_img, tile_coords, tile_size, overlap):

    # Check for any tiles that need different overlap values
    r, c = len(tile_coords[0]), len(tile_coords[1])
    f_ovlp = (tile_coords[0][r-1] - tile_coords[0][r-2], tile_coords[1][c-1] - tile_coords[1][c-2])

    h, w = tiles[0].size(2), tiles[0].size(3)
    t=0
    column, row, = 0, 0
    for y in tile_coords[0]:
        for x in tile_coords[1]:
            mask_sides=''
            c_overlap = overlap.copy()
            if row == 0:
                if row == len(tile_coords[0]) - 2:
                    mask_sides += 'bottom-special'
                    c_overlap[1] = f_ovlp[0] # Change bottom overlap
                else:
                    mask_sides += 'bottom'
            elif row > 0 and row < len(tile_coords[0]) -2:
                mask_sides += 'bottom,top'
            elif row == len(tile_coords[0]) - 2:
                if f_ovlp[0] > 0:
                    mask_sides += 'bottom-special,top'
                    c_overlap[1] = f_ovlp[0] # Change bottom overlap
                elif f_ovlp[0] <= 0:
                    mask_sides += 'bottom,top'
            elif row == len(tile_coords[0]) -1:
                if f_ovlp[0] > 0:
                    mask_sides += 'top-special'
                    c_overlap[0] = f_ovlp[0] # Change top overlap
                elif f_ovlp[0] <= 0:
                    mask_sides += 'top'

            if column == 0:
                if column == len(tile_coords[1]) -2:
                    mask_sides += ',right-special'
                    c_overlap[2] = f_ovlp[1] # Change right overlap
                else:
                    mask_sides += ',right'
            elif column > 0 and column < len(tile_coords[1]) -2:
                mask_sides += ',right,left'
            elif column == len(tile_coords[1]) -2:
                if f_ovlp[1] > 0:
                    mask_sides += ',right-special,left'
                    c_overlap[2] = f_ovlp[1] # Change right overlap
                elif f_ovlp[1] <= 0:
                    mask_sides += ',right,left'
            elif column == len(tile_coords[1]) -1:
                if f_ovlp[1] > 0:
                    mask_sides += ',left-special'
                    c_overlap[3] = f_ovlp[1] # Change left overlap
                elif f_ovlp[1] <= 0:
                    mask_sides += ',left'

            tile = mask_tile(tiles[t], c_overlap, side=mask_sides)
            base_img[:, :, y:y+tile_size[0], x:x+tile_size[1]] = base_img[:, :, y:y+tile_size[0], x:x+tile_size[1]] + tile
            t+=1
            column+=1
        row+=1
        column=0
    return base_img

# Calculate the coordinates for tiles
def get_tile_coords(d, tile_dim, overlap=0):
    move = int(tile_dim * (1-overlap))
    c, tile_start, coords = 1, 0, [0]
    while tile_start + tile_dim < d:
        tile_start = move * c
        if tile_start + tile_dim >= d:
            coords.append(d - tile_dim)
        else:
            coords.append(tile_start)
        c += 1
    return coords

# Calculates info required for tiling
def tile_setup(tile_size, overlap_percent, base_size):
    if type(tile_size) is not tuple and type(tile_size) is not list:
        tile_size = (tile_size, tile_size)
    if type(overlap_percent) is not tuple and type(overlap_percent) is not list:
        overlap_percent = (overlap_percent, overlap_percent)
    x_coords = get_tile_coords(base_size[1], tile_size[1], overlap_percent[1])
    y_coords = get_tile_coords(base_size[0], tile_size[0], overlap_percent[0])
    y_ovlp, x_ovlp = int(tile_size[0] * overlap_percent[0]), int(tile_size[1] * overlap_percent[1])
    return (y_coords, x_coords), tile_size, [y_ovlp, y_ovlp, x_ovlp, x_ovlp]

# Split tensor into tiles
def tile_image(img, tile_size, overlap_percent, info_only=False):
    tile_coords, tile_size, _ = tile_setup(tile_size, overlap_percent, (img.size(2), img.size(3)))

    # Cut out tiles
    tile_list = []
    for y in tile_coords[0]:
        for x in tile_coords[1]:
            tile = img[:, :, y:y + tile_size[0], x:x + tile_size[1]]
            tile_list.append(tile)
    return tile_list

# Put tiles back into the original tensor
def rebuild_image(tiles, image_size, tile_size, overlap_percent):
    base_img = torch.zeros(image_size, device=tiles[0].device)
    tile_coords, tile_size, overlap = tile_setup(tile_size, overlap_percent, (base_img.size(2), base_img.size(3)))
    return add_tiles(tiles, base_img, tile_coords, tile_size, overlap)

The above code can be tested with the code below:
import torchvision.transforms as transforms
from PIL import Image
import random

# Load image
def preprocess_simple(image_name, image_size):
    Loader = transforms.Compose([transforms.Resize(image_size), transforms.ToTensor()])
    image = Image.open(image_name).convert('RGB')
    return Loader(image).unsqueeze(0)

# Save image
def deprocess_simple(output_tensor, output_name):
    output_tensor.clamp_(0, 1)
    Image2PIL = transforms.ToPILImage()
    image = Image2PIL(output_tensor.squeeze(0))
    image.save(output_name)

test_input = preprocess_simple('tubingen.jpg', (1024,1024))
tile_size=260
overlap_percent=0.5

img_tiles = tile_image(test_input, tile_size=tile_size, overlap_percent=overlap_percent)

random.shuffle(img_tiles) # Comment this out to not randomize tile positions

output_tensor = rebuild_image(img_tiles, test_input.size(), tile_size=tile_size, overlap_percent=overlap_percent)
deprocess_simple(output_tensor, 'tiled_image.jpg')

I've included an example of what it does below (top is the original image, and the bottom is when I place the tiles back randomly to show off the blending system):


title: How to get number of classes from .pth file without any model information in Python?

body:
I have a .pth file and there is no model information available. How can I know the number of final output classes in .pth file.
For visualization, I can use Netron to see the number of classes. But, how can I get the same output number in Python.


title: Pytorch: Converting a VGG model into a sequential model, but getting different outputs

body:
Background:
I'm working on an adversarial detector method which requires to access the outputs from each hidden layer.
I loaded a pretrained VGG16 from torchvision.models.
To access the output from each hidden layer, I put it into a sequential model:
vgg16 = models.vgg16(pretrained=True)

vgg16_seq = nn.Sequential(*(
    list(list(vgg16.children())[0]) +
    [nn.AdaptiveAvgPool2d((7, 7)), nn.Flatten()] +
    list(list(vgg16.children())[2])))

Without nn.Flatten(), the forward method will complaint about dimensions don't match between mat1 and mat2.
I looked into the torchvision VGG implementation, it uses the [feature..., AvgPool, flatten, classifier...] structure.
Since AdaptiveAvgPool2d layer and Flatten layer have no parameters, I assume this should work, but I have different outputs.
output1 = vgg16(X_small)
print(output1.size())
output2 = vgg16_seq(X_small)
print(output2.size())
torch.equal(output1, output2)

Problem: They are in the same dimension but different outputs.

torch.Size([32, 1000])
torch.Size([32, 1000])
False

I tested the outputs right after the AdaptiveAvgPool2d  layer, the outputs are equal:
output1 = nn.Sequential(*list(vgg16.children())[:2])(X_small)
print(output1.size())
output2 = nn.Sequential(*list(vgg16_seq)[:32])(X_small)
print(output2.size())
torch.equal(output1, output2)

torch.Size([32, 512, 7, 7])
torch.Size([32, 512, 7, 7])
True

Can someone point out what went wrong?
Thank you


title: Unable to use Allennlp biaffine parser model

body:
I am trying to use allennlp predictor for biaffine parser. Here is the code:-
from allennlp.predictors.predictor import Predictor
predictor = Predictor.from_path("https://s3-us-west-2.amazonaws.com/allennlp/models/biaffine-dependency-parser-ptb-2018.08.23.tar.gz")

However, I get this error:-
ConfigurationError                        Traceback (most recent call last)
<ipython-input-3-74e2bcfcbf43> in <module>()
      1 from allennlp.predictors.predictor import Predictor
----> 2 predictor = Predictor.from_path("https://s3-us-west-2.amazonaws.com/allennlp/models/biaffine-dependency-parser-ptb-2018.08.23.tar.gz")

4 frames
/usr/local/lib/python3.6/dist-packages/allennlp/common/params.py in pop_choice(self, key, choices, default_to_first_choice, allow_class_names)
    350                 """{"model": "my_module.models.MyModel"} to have it imported automatically."""
    351             )
--> 352             raise ConfigurationError(message)
    353         return value
    354

ConfigurationError: universal_dependencies not in acceptable choices for dataset_reader.type: ['conll2003', 'interleaving', 'sequence_tagging', 'sharded', 'babi', 'text_classification_json']. You should either use the --include-package flag to make sure the correct module is loaded, or use a fully qualified class name in your config file like {"model": "my_module.models.MyModel"} to have it imported automatically.


title: PyTorch - efficiently apply TopK gradient coordinates in NN

body:
I have a sequential neural network (standard ResNet model), and for a constant k (about 1000, but may potentially increase in the future) I want to do the following:

Find the gradient of NN.
Identify Top-kcoordinates of the gradient (k coordinates with the largest absolute value)
Apply the gradient descent step using only these coordinates (i.e. other gradient coordinates are 0)

What I can do is the following:

Flatten the vector:

param_grads = [param.grad.to(cpu).flatten() for param in model.parameters()]
grad = torch.cat(param_grads)

Identify indices of Top-k coordinates in the sorted vector (I can also use topk function):

sorted_grad = grad.abs().sort()[1]

Now, the question is how to apply only these coordinates. I can write a function that manually translates flatten vector coordinates into the original coordinates (including the corresponding parameter), make a slice for each parameter, and zero gradient outside this slice for each parameter. However, I suspect that it'll be really inefficient. What's the best way to achieve this?


title: Custom transformation on Subset of CIFAR10 - PyTorch

body:
I am trying to create a custom transformation to part of the CIFAR10 data set which superimposing of an image over the dataset. I was able to download the data and divide it into subsets. Using the following code:
transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])

traindata = datasets.CIFAR10('./data', train=True, download=True,
                       transform= transform_train)

partitions = 5
traindata_split = torch.utils.data.random_split(traindata, [int(traindata.data.shape[0] / partitions) for _ in range(partitions)])

then I wanted to modify part of the splits so I created the following class and functions to use as as follows:
class MyDataset(Dataset): # https://discuss.pytorch.org/t/torch-utils-data-dataset-random-split/32209/3
    def __init__(self, subset, transform=None):
        self.subset = subset
        self.transform = transform

    def __getitem__(self, index):
        x, y = self.subset[index]
        if self.transform:
            x = self.transform(x)
        return x, y

    def __len__(self):
        return len(self.subset)

and
class ImageSuperImpose(object):
    """  Image input as PIL and output as PIL
        To be used as part of  torchvision.transforms
       Args: p, a threshold value to control image thinning
    """
    def __init__(self, p=0):
        self.p = p

    def __call__(self, image):
        img = cv2.imread('img.jpg')
        img = img('float32')/255
        imgSm = cv2.resize(img,(32,32))
        np_arr = image.cpu().detach().numpy().T
        sample = cv2.addWeighted(np_arr, 1, imgSm, 1, 0)
        sample = sample.T
        t = torch.from_numpy(sample)
        return sample

transform_train2 = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    ImagePoisoning(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])

datasetA = MyDataset(
    traindata_split[0], transform= transform_train2
)

test_loader = torch.utils.data.DataLoader(datasetA, batch_size=128, shuffle=True)

But when I tried to train the model on the subset I got the following error:
RuntimeError: The size of tensor a (32) must match the size of tensor b (3) at non-singleton dimension 0

** UPDATE**
Here is the full given error
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-20-7428084b03be> in <module>()
----> 1 train(model, opt, test_loader, 3)

9 frames
<ipython-input-14-fcb03e1d7685> in client_update(client_model, optimizer, train_loader, epoch)
      5     client_model.train()
      6     for e in range(epoch):
----> 7         for batch_idx, (data, target) in enumerate(train_loader):
      8             data, target = data.to(device), target.to(device)
      9             optimizer.zero_grad()

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    433         if self._sampler_iter is None:
    434             self._reset()
--> 435         data = self._next_data()
    436         self._num_yielded += 1
    437         if self._dataset_kind == _DatasetKind.Iterable and \

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _next_data(self)
    473     def _next_data(self):
    474         index = self._next_index()  # may raise StopIteration
--> 475         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    476         if self._pin_memory:
    477             data = _utils.pin_memory.pin_memory(data)

/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

<ipython-input-7-1bde43acaff0> in __getitem__(self, index)
      7         x, y = self.subset[index]
      8         if self.transform:
----> 9             x = self.transform(x)
     10         return x, y
     11

/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py in __call__(self, img)
     65     def __call__(self, img):
     66         for t in self.transforms:
---> 67             img = t(img)
     68         return img
     69

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py in forward(self, tensor)
    224             Tensor: Normalized Tensor image.
    225         """
--> 226         return F.normalize(tensor, self.mean, self.std, self.inplace)
    227
    228     def __repr__(self):

/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py in normalize(tensor, mean, std, inplace)
    282     if std.ndim == 1:
    283         std = std.view(-1, 1, 1)
--> 284     tensor.sub_(mean).div_(std)
    285     return tensor
    286

RuntimeError: The size of tensor a (32) must match the size of tensor b (3) at non-singleton dimension 0


title: Pytorch object detection model optimization

body:
I want to reduce the object detection model size. For the same, I tried optimising Faster R-CNN model for object detection using pytorch-mobile optimiser, but the .pt zip file generated is of the same size as that of the original model size.
I used the code mention below
import torch
import torchvision
from torch.utils.mobile_optimizer import optimize_for_mobile

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)

model.eval()
script_model = torch.jit.script(model)
from torch.utils.mobile_optimizer import optimize_for_mobile
script_model_vulkan = optimize_for_mobile(script_model, backend='Vulkan')
torch.jit.save(script_model_vulkan, "frcnn.pth")


title: How to fix a SIGSEGV in pytorch when using distributed training (e.g. DDP)?

body:
I keep getting this issue when running DDP in pytorch:
Traceback (most recent call last):
  File "ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 330, in <module>
    main_distributed()
  File "ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 230, in main_distributed
    mp.spawn(fn=train, args=(opts,), nprocs=opts.world_size)
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
    while not context.join():
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 105, in join
    raise Exception(
Exception: process 1 terminated with signal SIGSEGV

but this error is rather uninformative (dones't tell me what process or what it was trying to access for example) so I am unsure what I need to do do solve it.
Some research takes you that:

SIGSEGV: On a Unix operating system such as Linux, a "segmentation violation" (also known as "signal 11", "SIGSEGV", "segmentation fault" or, abbreviated, "sig11" or "segfault") is a signal sent by the kernel to a process when the system has detected that the process was attempting to access a memory address that does not belong to it. Typically, this results in the offending process being terminated.

yes I do have multiprocessing code as the usual mp.spawn(fn=train, args=(opts,), nprocs=opts.world_size) requires.
First I read the docs on sharing strategies which talks about how tensors are shared in pytorch:

Note that it applies only to CPU tensor - CUDA tensors will always use the CUDA API, as that’s the only way they can be shared.

I was using the file system sharing memory since it seemed to give me less issue when I needed lots of processes but I went down to only 2 processes and 2 gpus and to the share strategy being file descriptor. I thought that perhaps if the processes had their own cached file descriptor then there wouldn't be issues.
I did check the cuda devices availabe:
 $ echo $CUDA_VISIBLE_DEVICES
1,3

all seems fine.
I am unsure what might be causing the issue. There are possible issues like:

two processes are trying to checkpoint at the same time but I always only let rank=0 do the checkpointing so that doesn't make sense.
two processes are writing to tensorboard but I also only allow rank=0 to do the logging (or any of the printing).

So I am unsure what could be causing the issue. It could be that I have my dataset concatenated all 1 single json file causing the issue, but that wasn't causing issues yesterday with multiple gpus...though, if that is the case it would be hard to fix since DDP (distributed data parallel) uses the DistributedSampler which doesn't place any restriction like that on my data-set or dataloaders...or at least as far as I know (afaik).
Last thing is that yesterday I was getting weird error too and somehow it occurred to me to check the gpu type. I was quetting an issue because I was using a k40 gpu. I made sure that was not the case. Yesterday I was using a Quadro 6000 RTX, today it seems these are the GPUs I got:
$ nvidia-smi
Tue Mar  2 12:15:04 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.36.06    Driver Version: 450.36.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  TITAN X (Pascal)    Off  | 00000000:02:00.0 Off |                  N/A |
| 22%   37C    P0    56W / 250W |      0MiB / 12196MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  TITAN X (Pascal)    Off  | 00000000:03:00.0 Off |                  N/A |
| 24%   39C    P0    56W / 250W |      0MiB / 12196MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  TITAN X (Pascal)    Off  | 00000000:82:00.0 Off |                  N/A |
| 53%   84C    P2   244W / 250W |  11935MiB / 12196MiB |     57%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  TITAN X (Pascal)    Off  | 00000000:83:00.0 Off |                  N/A |
| 25%   39C    P0    56W / 250W |      0MiB / 12196MiB |      3%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    2   N/A  N/A     31809      C   python                          11933MiB |
+-----------------------------------------------------------------------------+

not sure if that is causing the issue but it's not always realistic to get the Quadro's so I want it to work for the Titan's too (and anything that isn't the k40s since the k40s seem to not be supported by pytorch anymore).
There are a few pytorch discussion forum posts and gitissues but none seems very helpful (to me at lest - not clear what they did to solve things despite end of discussion):

https://discuss.pytorch.org/t/multiprocessing-using-torch-multiprocessing/11029/3
https://discuss.pytorch.org/t/using-torch-tensor-over-multiprocessing-queue-process-fails/2847/12
https://github.com/pytorch/fairseq/issues/1720
https://github.com/pytorch/fairseq/issues/1308
https://github.com/facebookresearch/SlowFast/issues/118
https://discuss.pytorch.org/t/segmentation-fault/23489

I also noticed this happens when rank 0 ends before rank 1 (that helped me reproduce it, otherwise a sigabort SIGABRT happens).
    # clean up distributed code
    torch.distributed.barrier()
    if rank == 1:
        time.sleep(1)
    print(f'\n----> about to cleanup worker with rank {rank}')
    # cleanup(rank)
    torch.distributed.destroy_process_group()
    print(f'clean up done successfully! {rank}'

crossposted:

https://discuss.pytorch.org/t/how-to-fix-a-sigsegv-in-pytorch-when-using-distributed-training-e-g-ddp/113518
https://www.reddit.com/r/pytorch/comments/lwbb72/how_to_fix_a_sigsegv_in_pytorch_when_using/
https://www.quora.com/unanswered/How-do-I-fix-a-SIGSEGV-in-PyTorch-when-using-distributed-training-e-g-DDP


title: pytorch 0.4.0 broadcasting doesn't work in optimizer

body:
I can't seem to get broadcasting to work with autograd in pytorch 0.4.0! Any help appreciated. Below is a minimal code example that reproduces my problem. I would like to find a single value "bias", which minimizes the loss over the dataset. The understand the error message as it wants to backpropagate a vector with 5 entries into a scalar, which it cannot figure out. However, this is the whole idea of broadcasting. The behavior I expected was that it would propagate the mean of the error back to the broadcasted scalar value (here bias).

Please advice.

import numpy as np
import torch
from torch import nn
import torch.nn.functional as F
from torch.utils.data import Dataset

print(torch.__version__)

class AddBias(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, bias):
        ctx.save_for_backward(input, bias)
        return input - bias
    @staticmethod
    def backward(ctx, grad_out):
        input, bias = ctx.saved_tensors
        grad_in = grad_bias = None
        len_grad = len(ctx.needs_input_grad)
        assert len_grad in {0, 1, 2}
        if ctx.needs_input_grad[0]: grad_in = grad_out
        if len_grad == 2: grad_bias = -1 * grad_out
        return grad_in, grad_bias

class BiasModel(nn.Module):
    def __init__(self, size):
        super(BiasModel, self).__init__()
        self.bias_model = AddBias.apply
        self.bias = nn.Parameter(torch.tensor(0.5, dtype=torch.float, requires_grad=True))
    def forward(self, arr): return self.bias_model(arr[:], self.bias).unsqueeze(-1)

class MyData(Dataset):
    def __init__(self, data): self.data = data
    def __len__(self): return len(self.data)
    def __getitem__(self, i):
        arr = torch.tensor(data[i], dtype=torch.float)
        target = torch.tensor(arr > 0.5, dtype=torch.float).unsqueeze(-1)
        return arr, target

m = 5
data = np.random.random((100, m))
model = BiasModel(m)
my_data = MyData(data)

loss_func = F.binary_cross_entropy_with_logits
with torch.no_grad():
    loss = 0.
    for arr, target in my_data: loss += loss_func(model(arr), target)
    print('loss before', loss / len(my_data))

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

loss_tot = 0.
for arr, target in my_data:
    model.zero_grad()
    loss = loss_func(model(arr), target)
    loss_tot += loss
    loss.backward()
    optimizer.step()

Output:

0.4.0
loss before tensor(0.5735)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-27bce65b553b> in <module>()
     56     loss_tot += loss
     57     loss.backward()
---> 58     optimizer.step()

~/miniconda3/envs/myproject/lib/python3.6/site-packages/torch/optim/sgd.py in step(self, closure)
    105                         d_p = buf
    106
--> 107                 p.data.add_(-group['lr'], d_p)
    108
    109         return loss

RuntimeError: expand(torch.FloatTensor{[5]}, size=[]): the number of sizes provided (0) must be greater or equal to the number of dimensions in the tensor (1)


title: Pytorch: choosing columns from a 3d tensor, according to indices tensor

body:
I have a 3D tensor M of dimensions [BxLxD] and a 1D tensor idx of dimensions [B,1] that contains column indices in the range (0, L-1). I want to create a 2D tensor N of dimensions [BxD] such that N[i,j] = M[i, idx[i], j]. How can this be done efficiently?

Example:

B,L,D = 2,4,2

M = torch.rand(B,L,D)

>

tensor([[[0.0612, 0.7385],
         [0.7675, 0.3444],
         [0.9129, 0.7601],
         [0.0567, 0.5602]],

        [[0.5450, 0.3749],
         [0.4212, 0.9243],
         [0.1965, 0.9654],
         [0.7230, 0.6295]]])

idx = torch.randint(0, L, size = (B,))

>

tensor([3, 0])

N = get_N(M, idx)

Expected output:

>

tensor([[0.0567, 0.5602],
       [0.5450, 0.3749]])

Thanks.


title: Why does the performance drop sharply after replacing DP with DDP？

body:
When I used Distributed dataparallel to replace dataparallel,the result of the validation set becomes very poor, as in the case of overfitting. I used 4 GPUs, one process per GPU, keeping the learning rate and batchsize unchanged.The following is all the code related to DPP：

dist.init_process_group(backend='nccl')
torch.cuda.set_device(args.local_rank)
device = torch.device("cuda", args.local_rank)

train_sampler = torch.utils.data.distributed.DistributedSampler(train_set)
train_loader = torch.utils.data.DataLoader(
        train_set, batch_size=args.batch_size,
        num_workers=args.workers,sampler=train_sampler, pin_memory=True, shuffle=(train_sampler is None))
val_sampler = torch.utils.data.distributed.DistributedSampler(val_set)
val_loader = torch.utils.data.DataLoader(
        val_set, batch_size=args.batch_size,
        num_workers=args.workers, pin_memory=True, shuffle=False,sampler=val_sampler)

model = models.__dict__[args.arch](network_data).to(device)
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.local_rank])
cudnn.benchmark = True
for epoch in tqdm(range(args.start_epoch, args.epochs)):
    # train for one epoch
    train_sampler.set_epoch(epoch)

    train_loss=train(......)
    dist.reduce(train_loss, 0, op=dist.ReduceOp.SUM)
    print(train_loss/nb_gpus)

    test_loss=validate(.....)
    dist.reduce(test_loss, 0, op=dist.ReduceOp.SUM)
    print(test_loss/nb_gpus)

enter image description here

blue curve is the result of validation set

enter image description here

The brown curve is the result of the training set


title: Evaluate CNN model for multiclass image classification

body: i want to ask what metric can be used to evalutate my CNN model for multi class, i have 3 classes for now and i’m just using accuracy and confussion matrix also plot the loss of model, is there any metric can be used to evaluate my model performance?


title: BERT NER Python

body:
I am using BERT model for Named Entity Recognition task.
I have torch version - 1.2.0+cu9.2
torch vision version - 0.4.0+cu9.2
Nvidia drivers compatible with cuda 9.2
when i am trying to train my model using the command
loss, scores = model(b_input_ids.type(torch.cuda.LongTensor), token_type_ids=None,attention_mask=b_input_mask.to(device), labels=b_labels.type(torch.cuda.LongTensor))

i am getting the error below -
C:/w/1/s/windows/pytorch/aten/src/THC/THCTensorIndex.cu:361: block: [35,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.

Can somebody help me with this one?


title: PyTorch - ModuleNotFoundError

body:
I've trained Retinanet for object detection in google colab and now I want to load its .pt file in another python project but I keep getting this error. Any thoughts?
Traceback (most recent call last):
  File "C:\Users\stefan_cepa995\Desktop\breast-mammography-app\app.py", line 522, in <module>
    model = torch.load(os.path.join(".", "models", "retinanet", "retinanet_gwd.pt"))
  File "C:\Users\stefan_cepa995\anaconda3\envs\tensorflow\lib\site-packages\torch\serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\stefan_cepa995\anaconda3\envs\tensorflow\lib\site-packages\torch\serialization.py", line 853, in _load
    result = unpickler.load()
ModuleNotFoundError: No module named 'retinanet'


title: Linking pytorch c++ (prebuilt GPU version) with Qt and GCC

body:
I wrote a simple program in C++ to use my PyTorch model in libtorch 1.7.1. I have CUDA 10.2 on my OS, thus
I downloaded CUDA 10.2 prebuilt version of libtorch from this link.
The program works fine with CMake and CUDA. CMake can load my model on GPU and returns "Yes" when I run this simple program:

Now, I would like to use the model in my Qt Application. It gives this result when I try it :

This is my project QMake file :
QT -= gui
CONFIG +=  c++14 console
CONFIG -= app_bundle
DEFINES += QT_DEPRECATED_WARNINGS
SOURCES += \
    main.cpp
qnx: target.path = /tmp/$${TARGET}/bin
else: unix:!android: target.path = /opt/$${TARGET}/bin
!isEmpty(target.path): INSTALLS += target

QMAKE_CXXFLAGS += -DGLIBCXX_USE_CXX11_ABI=0
CONFIG += no_keywords
QMAKE_CXXFLAGS += -Wno-unused-parameter
QMAKE_LFLAGS += -INCLUDE:?warp_size@cuda@at@@YAHXZ

INCLUDEPATH += /home/eaglesoft/libtorch/include/torch/csrc/api/include
INCLUDEPATH += /home/eaglesoft/libtorch/include
DEPENDPATH +=  /home/eaglesoft/libtorch/include/torch/csrc/api/include
DEPENDPATH +=  /home/eaglesoft/libtorch/include

LIBS += -L/home/eaglesoft/libtorch/lib -lc10 \
        -lc10_cuda \
        -lcaffe2_module_test_dynamic \
        -lcaffe2_observers \
        -ltorch_cpu \
        -ltorch \
        -lc10d_cuda_test \
        -lcaffe2_detectron_ops_gpu \
        -lcaffe2_nvrtc \
        -lfmt \
        -lnvrtc-builtins \
        -lshm \
        -ltensorpipe \
        -ltorch_cuda

Any help would be appreciated.


title: PyTorch, select batches according to label in data column

body:
I have a dataset like such:

index
tag
feature1
feature2
target

1
tag1
1.4342
88.4554
0.5365

2
tag1
2.5656
54.5466
0.1263

3
tag2
5.4561
845.556
0.8613

4
tag3
6.5546
8.52545
0.7864

5
tag3
8.4566
945.456
0.4646

The number of entries in each tag is not always the same.
And my objective is to load only the data with a specific tag or tags, so that I get only the entries in tag1 for one mini-batch and then tag2 for another mini-batch if I set batch_size=1. Or for instance tag1 and tag2 if I set batch_size=2
The code I have so far disregards completely the tag label and just chooses the batches randomly.
I built the datasets like such:
# features is a matrix with all the features columns through all rows
# target is a vector with the target column through all rows
featuresTrain, targetTrain = projutils.get_data(train=True, config=config)
train = torch.utils.data.TensorDataset(featuresTrain, targetTrain)
train_loader = make_loader(train, batch_size=config.batch_size)

And my loader (generically) looks like this:
def make_loader(dataset, batch_size):
loader = torch.utils.data.DataLoader(dataset=dataset,
                                     batch_size=batch_size,
                                     shuffle=True,
                                     pin_memory=True,
                                     num_workers=8)
return loader

Which I then train like this:
for epoch in range(config.epochs):
    for _, (features, target) in enumerate(loader):
        loss = train_batch(features, target, model, optimizer, criterion)

And the train_batch:
def train_batch(features, target, model, optimizer, criterion):
features, target = features.to(device), target.to(device)

# Forward pass ➡
outputs = model(features)
loss = criterion(outputs, target
return loss


title: Half-precision PyTorch float tensors have same performance as single precision float tensors?

body:
Background: I've implemented the antiobject/"field AI" pattern (https://home.cs.colorado.edu/~ralex/papers/PDF/OOPSLA06antiobjects.pdf) for single diffusion using LibTorch/PyTorch.
This works fine, but in the process of running it on the GPU and optimizing it, I've run into a problem. I have a Titan V, which I believe excels at half-precision float math. However, when I make the tensors torch::kHalf, the performance is the same. (I've also tried torch::kFloat16). Any ideas?
The code that I timed is in update():
#define SDL_MAIN_HANDLED

#include <simple2d.h>
#include <torch/torch.h>
#include <c10/cuda/CUDAStream.h>
#include <ATen/cuda/CUDAEvent.h>
#include <math.h>
#include <chrono>

#define DEBUG_NO_DRAW

torch::Device gpu(torch::kCUDA);
torch::Device cpu(torch::kCPU);
torch::Device device = gpu;

const int windowLength = 1000;
const int64_t length = 500;
const float diffusionRate = 0.25;
const int obstacleCount = 4000;
const int entityCount = 1000;

float cellLength = windowLength / length;
torch::Tensor scent = torch::zeros({ length, length }, device).to(torch::kHalf);
torch::Tensor up, down, left, right;
torch::Tensor topWallMask = torch::ones({ length, length }, device).to(torch::kHalf);
torch::Tensor bottomWallMask = torch::ones({ length, length }, device).to(torch::kHalf);
torch::Tensor leftWallMask = torch::ones({ length, length }, device).to(torch::kHalf);
torch::Tensor rightWallMask = torch::ones({ length, length }, device).to(torch::kHalf);
torch::Tensor obstaclesMask = torch::ones({ length, length }, device).to(torch::kHalf);
torch::Tensor entities = torch::zeros({ length, length }, device).to(torch::kHalf);
c10::cuda::CUDAStream stream = c10::cuda::getCurrentCUDAStream();

std::time_t *lastFpsUpdate = NULL;
std::time_t *currentTime = new std::time_t();
int frameAccumulator = 0;
std::vector<long> updateDurations;

void update() {
    torch::NoGradGuard no_grad;

    AT_CUDA_CHECK(cudaStreamSynchronize(stream));
    auto startTime = std::chrono::high_resolution_clock::now();

    down = scent.roll(1, 0) * obstaclesMask * topWallMask;
    up = scent.roll(-1, 0) * obstaclesMask * bottomWallMask;
    right = scent.roll(1, 1) * obstaclesMask * leftWallMask;
    left = scent.roll(-1, 1) * obstaclesMask * rightWallMask;
    scent = scent + ((down - scent) + (up - scent) + (right - scent) + (left - scent)) * diffusionRate;
    scent = torch::max(scent, entities);

    AT_CUDA_CHECK(cudaStreamSynchronize(stream));
    auto endTime = std::chrono::high_resolution_clock::now();
    auto duration = std::chrono::duration_cast<std::chrono::microseconds>(endTime - startTime);
    updateDurations.push_back(duration.count());
}

void render() {
    if (lastFpsUpdate == NULL) {
        lastFpsUpdate = new std::time_t();
        std::time(lastFpsUpdate);
    }

    torch::Tensor sqrtedScent = scent.sqrt().to(torch::kFloat).to(cpu); // just to make darker scents a little brighter for display
    auto obstaclesMaskCPU = obstaclesMask.to(torch::kFloat).to(cpu);

    auto sqrtedScentAccessor = sqrtedScent.accessor<float, 2>();
    auto obstaclesMaskAccessor = obstaclesMaskCPU.accessor<float, 2>();
    float r = 0, g = 0, b = 0, a = 0;

#ifndef DEBUG_NO_DRAW
    S2D_DrawQuad(
        0, 0, 0, 0, 0, 1,
        windowLength, 0, 0, 0, 0, 1,
        windowLength, windowLength, 0, 0, 0, 1,
        0, windowLength, 0, 0, 0, 1);
#endif

    for (int i = 0; i < length; i++) {
        for(int j = 0; j < length; j++) {
            if (obstaclesMaskAccessor[i][j] == 0) {
                r = 1; g = 1; b = 1; a = 1;
            }
            else {
                r = 1; g = 0; b = 0; a = sqrtedScentAccessor[i][j];
            }

#ifndef DEBUG_NO_DRAW
            S2D_DrawQuad(cellLength * j, cellLength * i, r, g, b, a,
                cellLength * (j + 1), cellLength * i, r, g, b, a,
                cellLength * (j + 1), cellLength * (i + 1), r, g, b, a,
                cellLength * j, cellLength * (i + 1), r, g, b, a);
#endif
        }
    }

    frameAccumulator++;

    std::time(currentTime);
    if (std::difftime(*currentTime, *lastFpsUpdate) > 1.0) {
        std::cout << "FPS: " << frameAccumulator << std::endl;
        frameAccumulator = 0;
        *lastFpsUpdate = *currentTime;

        int updateCount = updateDurations.size();
        long totalUpdateTime = 0;
        for (int i = 0; i < updateCount; i++) {
            totalUpdateTime += updateDurations[i];
        }
        long averageUpdateTime = totalUpdateTime / updateCount;
        std::cout << "AverageUpdateTime: " << averageUpdateTime << "us" << std::endl;
        updateDurations.clear();
    }
}

int main() {
    if (torch::cuda::is_available()) {
        std::cout << "CUDA is available!" << std::endl;
    }
    std::cout << "Using " << (device == cpu ? "CPU" : "GPU") << std::endl;

    for (int i = 0; i < length; i++) {
        topWallMask[0][i] = 0;
        bottomWallMask[length - 1][i] = 0;
        leftWallMask[i][0] = 0;
        rightWallMask[i][length - 1] = 0;
    }

    for (int i = 0; i < obstacleCount; i++) {
        int x = rand() % length;
        int y = rand() % length;
        obstaclesMask[x][y] = 0;
    }

    //std::cout << obstaclesMask << std::endl;

    for (int i = 0; i < entityCount; i++) {
        int x = rand() % length;
        int y = rand() % length;
        if (obstaclesMask[x][y].item() == 0)
            continue;
        entities[x][y] = 1;
    }

    S2D_Window* window = S2D_CreateWindow(
        "Collab Diffuse", windowLength, windowLength, update, render, 0
    );
    S2D_Show(window);

    return 0;
}

In both single precision and half precision versions of the code, update() takes about 2700 microseconds.
I'm using PyTorch/LibTorch 1.7.1.
Any other performance tips would be appreciated. (I'm aware drawing pixel by pixel is very slow, so I plan to switch from Simple2D to something else that can draw bitmaps from memory).


title: How to run one batch in pytorch?

body:
I'm new to AI and python and I'm trying to run only one batch to aim to overfit.I found the code:
iter(train_loader).next()
but I'm not sure where to implement it in my code. even if I did, how can I check after each iteration to make sure that I'm training the same batch?
train_loader = torch.utils.data.DataLoader(
    dataset_train,
    batch_size=48,
    shuffle=True,
    num_workers=2
)

net = nn.Sequential(
    nn.Flatten(),
    nn.Linear(128*128*3,10)
)

nepochs = 3
statsrec = np.zeros((3,nepochs))

loss_fn = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=0.001)

for epoch in range(nepochs):  # loop over the dataset multiple times

    running_loss = 0.0
    n = 0
    for i, data in enumerate(train_loader, 0):
        inputs, labels = data

         # Zero the parameter gradients
        optimizer.zero_grad()

        # Forward, backward, and update parameters
        outputs = net(inputs)
        loss = loss_fn(outputs, labels)
        loss.backward()
        optimizer.step()

        # accumulate loss
        running_loss += loss.item()
        n += 1

    ltrn = running_loss/n
    ltst, atst = stats(train_loader, net)
    statsrec[:,epoch] = (ltrn, ltst, atst)
    print(f"epoch: {epoch} training loss: {ltrn: .3f}  test loss: {ltst: .3f} test accuracy: {atst: .1%}")

please give me a hint


title: Pytorch is installed, but while importing getting error

body:
I have installed Anaconda on windows 8, after that done with the PyTorch.
But while importing torch getting error.

  import torch

  AttributeError                            Traceback (most recent
call last)

<ipython-input-4-eb42ca6e4af3> in <module>

----> 1 import torch

I:\ProgramData\Anaconda3\lib\site-packages\torch\__init__.py in
<module>
 51         from ctypes.wintypes import DWORD, HMODULE
 52

---> 53         AddDllDirectory = windll.kernel32.AddDllDirectory
 54         AddDllDirectory.restype = DWORD
 55         AddDllDirectory.argtypes = [c_wchar_p]

I:\ProgramData\Anaconda3\lib\ctypes\__init__.py in __getattr__(self,
name)
367         if name.startswith('__') and name.endswith('__'):
368             raise AttributeError(name)

--> 369         func = self.__getitem__(name)
370         setattr(self, name, func)
371         return func

I:\ProgramData\Anaconda3\lib\ctypes\__init__.py in __getitem__(self,
name_or_ordinal)
372
373     def __getitem__(self, name_or_ordinal):

--> 374         func = self._FuncPtr((name_or_ordinal, self))
375         if not isinstance(name_or_ordinal, int):
376             func.__name__ = name_or_ordinal

AttributeError: function 'AddDllDirectory' not found

Also when I used the following command it is showing the pytorch in
Anaconda command prompt.

(base) C:\Users\rk88>conda list pytorch

# packages in environment at I:\ProgramData\Anaconda3:

#

# Name                    Version                   Build  Channel

pytorch                   1.0.1           py3.7_cuda100_cudnn7_1
PyTorch

Anyone can please help me how to resolve this, if you faced this issue earlier.

Thanks,


title: install/import Pytorch on mac osx / python 3.7

body:
I have torch all installed in a virtual env, but python cannot seem to import it.

(venv) ➜  ParlAI git:(master) pip install torch
Requirement already satisfied: torch in ./venv/lib/python3.7/site-packages (1.3.1)
Requirement already satisfied: numpy in ./venv/lib/python3.7/site-packages (from torch) (1.18.0)

(venv) ➜  ParlAI git:(master) python --version
Python 3.7.6

(venv) ➜  ParlAI git:(master) python
Python 3.7.6 (default, Jan  3 2020, 07:48:25)
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'

>>> ^D

(venv) ➜  ParlAI git:(master) cat venv/lib/python3.7/site-packages/torch/version.py
__version__ = '1.3.1'
debug = False
cuda = None
git_version = 'ee77ccbb6da4e2efd83673e798acf7081bc03564'

followed everything from here:
https://pytorch.org/get-started/locally/

I'm trying out some of the ParlAI samples, eg

https://parl.ai/projects/dialogue_safety/

(venv) ➜  ParlAI git:(master) python examples/display_data.py -t dialogue_safety:adversarial --round 1

Traceback (most recent call last):
  File "examples/display_data.py", line 22, in <module>
    opt = parser.parse_args()
  File "/Volumes/slab/trainingData/ParlAI/parlai/core/params.py", line 1072, in parse_args
    self.add_extra_args(args)
  File "/Volumes/slab/trainingData/ParlAI/parlai/core/params.py", line 889, in add_extra_args
    self.add_task_args(task)
  File "/Volumes/slab/trainingData/ParlAI/parlai/core/params.py", line 812, in add_task_args
    agent = load_teacher_module(t)
  File "/Volumes/slab/trainingData/ParlAI/parlai/core/loader.py", line 217, in load_teacher_module
    task_module = load_task_module(taskname)
  File "/Volumes/slab/trainingData/ParlAI/parlai/core/loader.py", line 187, in load_task_module
    task_module = importlib.import_module(module_name)
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Volumes/slab/trainingData/ParlAI/parlai/tasks/dialogue_safety/agents.py", line 12, in <module>
    from parlai.core.teachers import FixedDialogTeacher
  File "/Volumes/slab/trainingData/ParlAI/parlai/core/teachers.py", line 51, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'

Any suggestions what else I can try with this environment? I don't really want to use anaconda, prefer a bit more control of the system.


title: Two questions on DCGAN: data normalization and fake/real batch

body:
I am analyzing a meta-learning class that uses DCGAN + Reptile within the image generation.

I have two questions about this code.

First question: why during DCGAN training (line 74)

training_batch = torch.cat ([real_batch, fake_batch])

is a training_batch made up of real examples (real_batch) and fake examples (fake_batch) created? Why is training done by mixing real and false images? I have seen many DCGANs, but never with training done in this way.

The second question: why is the normalize_data function (line 49) and the unnormalize_data function (line 55) used during training?

def normalize_data(data):
    data *= 2
    data -= 1
    return data

def unnormalize_data(data):
    data += 1
    data /= 2
    return data

The project uses the Mnist dataset, if I wanted to use a color dataset like CIFAR10, do I have to modify those normalizations?


title: Which output should I use for prediction with LSTM for sequenced data?

body:
I'm still new to machine learning and deep learning. I am currently trying to predict time series data with LSTM in PyTorch. The problem I am having is that I don't understand which output should I use for my final prediction.
My code is given below:

class Model(nn.Module):
  def __init__(self, input_size, hidden_size, output_size, seq_len, dropout):
    super(Model, self).__init__()
    self.input_size = input_size
    self.hidden_size = hidden_size
    self.output_size = output_size
    self.dropout = dropout
    self.seq_len = seq_len
    self.lstm = nn.LSTM(
        input_size = self.input_size,
        hidden_size = self.hidden_size,
        dropout = self.dropout
    )
    self.linear = nn.Linear(self.hidden_size, self.output_size)

  def reset_hidden_state(self):
    self.hidden = (
        torch.zeros(1, self.seq_len, self.hidden_size),
        torch.zeros(1, self.seq_len, self.hidden_size)
    )

  def forward(self, sequences):
    lstm_out, self.hidden = self.lstm(sequences, self.hidden)
    y_pred = self.linear(lstm_out[-1, :, :])
    return y_pred

mymodel = Model(5, 10, 1, 3, 0.0)
inps = torch.randn(10, 3, 5)   #input
#print(inps)
mymodel.reset_hidden_state()
out = mymodel.forward(inps)
print(out.shape)
print(out)

output:

  torch.Size([3, 1])

  tensor([[-0.0996],
          [-0.0587],
          [-0.0421]], grad_fn=)

As you can see, this gives me three outputs, but my output size is 1 as I am trying to predict only 1 variable. So, in this case which variable should I use for my final prediction? Or, is it is even possible to predict only 1 value for sequential data like this?

NB: My python version is 3.7.4
and my PyTorch version is 1.4.0

And, sorry if I have made any mistake while asking the question. This is my first time asking question here.


title: Is there a way to obtain IUV map from image in tensorflow?

body:
I have been using detectron2/densepose to generate IUV map which helped me generate UV texture from the input image. Now, for deployment, I need to have IUV map generation from input image in client-side using JavaScript. I am familiar with TensorFlow but the current dense pose model runs only on PyTorch and interconversion tools are giving errors too.
Any comment or solution to the problem will be very helpful.
IUV image needed:

Final UV map:


title: Combining two pretrained models for Training in pytorch

body:
I am trying to use resnet18 and densenet121 as the pretrained models and have added 1 FC layer at the end of each network two change dimensions to 512 of both network and then have concatenated the outputs and passed to two final FC layers.
this can be seen here:
class classifier(nn.Module):
  def __init__(self,num_classes):
    super(classifier,self).__init__()

    self.resnet=models.resnet18(pretrained=True)

    self.rfc1=nn.Linear(512,512)

    self.densenet=models.densenet121(pretrained=True)
    self.dfc1=nn.Linear(1024,512)

    self.final_fc1=nn.Linear(1024,512)
    self.final_fc2=nn.Linear(512,num_classes)
    self.dropout=nn.Dropout(0.2)

  def forward(self,x):
    y=x.detach().clone()

    x=self.resnet.conv1(x)
    x=self.resnet.bn1(x)
    x=self.resnet.relu(x)
    x=self.resnet.maxpool(x)
    x=self.resnet.layer1(x)
    x=self.resnet.layer2(x)
    x=self.resnet.layer3(x)
    x=self.resnet.layer4(x)
    x=self.resnet.avgpool(x)
    x=x.view(x.size(0),-1)
    x=nn.functional.relu(self.rfc1(x))

    y=self.densenet.features(y)
    y=y.view(y.size(0),-1)
    y=nn.functional.relu(self.dfc1(y))

    x=torch.cat((x,y),0)
    x=nn.functional.relu(self.final_fc1(x))
    x=self.dropout(x)
    x=self.final_fc2(x)

    return x

Error:
size mismatch, m1: [1048576 x 1], m2: [1024 x 512] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:283

but as i could see the densenet outputs 1024 feature to the last layer
Questions:

Is the implementation correct ?
Can this implementation work as a Ensemble


title: RuntimeError: one of the variables needed for gradient computation has been modified: is at version 2; expected version 1 instead

body:
I'm trying the following Kaggle.
TL;DR: I want to classify a sequence (time-series) of measurements to 1 of K classes using LSTM.
I'm trying to overfit the model on 2 sequences:
My input is (B, N, M):

B : batch-size = 1
N : sequence-size = 128
M : num-of-feature = 14 (number of measurements in each timestamp)
My model is a very simple LSTM:

class LSTMClassifier(nn.Module):
    def __init__(self, in_dim, hidden_dim, out_dim, num_layers):
        super(LSTMClassifier, self).__init__()
        self.in_dim = in_dim
        self.hidden_dim = hidden_dim
        self.out_dim = out_dim
        self.num_layers = num_layers

        self.lstm = nn.LSTM(in_dim, hidden_dim, num_layers=num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_dim, out_dim)

    def forward(self, x):
        lstm_out, (ht, ct) = self.lstm(x)
        y = self.fc(ht[-1].reshape(-1, self.hidden_dim))
        return y

And the train process is:
def train_lstm_model(model, data_loader, num_epochs, loss_cls, optimizer_cls, learning_rate):
    start = time.time()

    loss = loss_cls()
    optimizer = optimizer_cls(model.parameters(), lr=learning_rate)

    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model.to(device)

    for epoch in tqdm(range(num_epochs)):
        hidden = (torch.zeros((1, data_loader.batch_size, model.hidden_dim), requires_grad=True).to(device),
                    torch.zeros(1, data_loader.batch_size, model.hidden_dim, requires_grad=True).to(device))
        for i, (batch_x, batch_y) in enumerate(data_loader):
            batch_x = batch_x.to(device).float()
            batch_y = batch_y.to(device).long()

            optimizer.zero_grad()

            y_predicted, hidden = model(batch_x, hidden)
            l = loss(y_predicted, batch_y)

            l.backward()
            optimizer.step()

#            print(f'epoch {epoch+1}, batch {i+1}: loss = {l.item()} |',
#                  f'train accuracy: {eval_lstm_model(model, data_loader.dataset, hidden)}')

    end = time.time()
    print(f'Training took {end-start} seconds.')

And my setup code is:
loss_cls = nn.CrossEntropyLoss
optimizer_cls = torch.optim.SGD
hidden_dim = 100

model_lstm = LSTMClassifier(X_of.shape[-1], hidden_dim, len(np.unique(y_train)))

learning_rate = 0.01
num_epochs = 1000
train_lstm_model(model_lstm, overfit_loader, num_epochs, loss_cls, optimizer_cls, learning_rate)

The overfit_loader is a DataLoader which contains only 2 samples.
But the training process outputs the following error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-87-5f725d0ecc50> in <module>
     27 learning_rate = 0.001
     28 num_epochs = 100
---> 29 train_lstm_model(model_lstm, overfit_loader, num_epochs, loss_cls, optimizer_cls, learning_rate)

<ipython-input-86-ba60b3627f13> in train_lstm_model(model, data_loader, num_epochs, loss_cls, optimizer_cls, learning_rate, test_loader)
     20             l = loss(y_predicted, batch_y)
     21
---> 22             l.backward(retain_graph=True)
     23             optimizer.step()
     24

/usr/local/lib64/python3.6/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    219                 retain_graph=retain_graph,
    220                 create_graph=create_graph)
--> 221         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    222
    223     def register_hook(self, hook):

/usr/local/lib64/python3.6/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
--> 132         allow_unreachable=True)  # allow_unreachable flag
    133
    134

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace
operation: [torch.cuda.FloatTensor [400, 14]] is at version 2; expected version 1 instead. Hint: the
backtrace further above shows the operation that failed to compute its gradient. The variable in question
was changed in there or anywhere later. Good luck!

EDIT: I've removed the loss printing and stop re-using the hidden, according to @SzymonMaszke comment, and the exception gone, but there's still a problem that the loss isn't converges below 0.7
I'd like to get some help please,
Thanks!


title: ResNet family classification layer activation function

body:
I am using the ResNet18 pre-trained model which will be used for a simple binary image classification task. However, all the tutorials including PyTorch itself use nn.Linear(num_of_features, classes) for the final fully connected layer. What I fail to understand is where is the activation function for that module? Also what if I want to use sigmoid/softmax how do I go about that?
Thanks for your help in advance, I am kinda new to Pytorch


title: image segmentation can't generate result

body:
Building an Encoder-Decoder sequence for image segmentation,That’s to say,I use black background and white foreground to train the whole network.and in test process, it generate nothing but black]1
[]2

I check the softmax output, find that it does treat the background pixel to 0 channel, so argmax op generate 0 at this pixel.But foreground pixel’s softmax output is even. just(0.5,0.5),so it can’t distinguish,so the argmax chose the natural order 0 to present the class,and generate 0.
So when I use argmax it generate all black image, while the argmin to genrate
 reverse result,which means it distinguish the background,but at these even pixels,the network fails.
I conjecture that I use a different deconv process.https://distill.pub/2016/deconv-checkerboard/ that’s which inspires me. cause I used to use deconv,but it generate checkerboard artifacts inside the white area.so I follow the blog to use upsampling layer and a size-keeping conv operation to replace the deconv layer.But it fails,too.WHERE to modify?I really confused, and, a bit upset.

the two problem My Image segmentation result map contains black lattice in in the white patch also launched by myself, but it's different.they use different model to generate different result,the former is about deconv layer ,the latter is about checkerboard atrifact and its solution:upsampling,which still fails on my network,cause it produce only black pixels, and unable to categorize white area pixels.


title: Getting rid of maxpooling layer causes running cuda out memory error pytorch

body:
Video card: gtx1070ti 8Gb, batchsize 64, input image size 128*128.
I had such UNET with resnet152 as encoder wich worket pretty fine:

class UNetResNet(nn.Module):

def __init__(self, encoder_depth, num_classes, num_filters=32, dropout_2d=0.2,
             pretrained=False, is_deconv=False):
    super().__init__()
    self.num_classes = num_classes
    self.dropout_2d = dropout_2d

    if encoder_depth == 34:
        self.encoder = torchvision.models.resnet34(pretrained=pretrained)
        bottom_channel_nr = 512
    elif encoder_depth == 101:
        self.encoder = torchvision.models.resnet101(pretrained=pretrained)
        bottom_channel_nr = 2048
    elif encoder_depth == 152:
        self.encoder = torchvision.models.resnet152(pretrained=pretrained)
        bottom_channel_nr = 2048

    else:
        raise NotImplementedError('only 34, 101, 152 version of Resnet are implemented')

    self.pool = nn.MaxPool2d(2, 2)

    self.relu = nn.ReLU(inplace=True)

    self.conv1 = nn.Sequential(self.encoder.conv1,
                               self.encoder.bn1,
                               self.encoder.relu,
                               self.pool) #from that pool layer I would like to get rid off

    self.conv2 = self.encoder.layer1
    self.conv3 = self.encoder.layer2
    self.conv4 = self.encoder.layer3
    self.conv5 = self.encoder.layer4
    self.center = DecoderCenter(bottom_channel_nr, num_filters * 8 *2, num_filters * 8, False)

    self.dec5 =  DecoderBlockV(bottom_channel_nr + num_filters * 8, num_filters * 8 * 2, num_filters * 8,   is_deconv)
    self.dec4 = DecoderBlockV(bottom_channel_nr // 2 + num_filters * 8, num_filters * 8 * 2, num_filters * 8, is_deconv)
    self.dec3 = DecoderBlockV(bottom_channel_nr // 4 + num_filters * 8, num_filters * 4 * 2, num_filters * 2, is_deconv)
    self.dec2 = DecoderBlockV(bottom_channel_nr // 8 + num_filters * 2, num_filters * 2 * 2, num_filters * 2 * 2,
                               is_deconv)
    self.dec1 = DecoderBlockV(num_filters * 2 * 2, num_filters * 2 * 2, num_filters, is_deconv)
    self.dec0 = ConvRelu(num_filters, num_filters)
    self.final = nn.Conv2d(num_filters, num_classes, kernel_size=1)

def forward(self, x):
    conv1 = self.conv1(x)
    conv2 = self.conv2(conv1)
    conv3 = self.conv3(conv2)
    conv4 = self.conv4(conv3)
    conv5 = self.conv5(conv4)
    center = self.center(conv5)
    dec5 = self.dec5(torch.cat([center, conv5], 1))
    dec4 = self.dec4(torch.cat([dec5, conv4], 1))
    dec3 = self.dec3(torch.cat([dec4, conv3], 1))
    dec2 = self.dec2(torch.cat([dec3, conv2], 1))
    dec1 = self.dec1(dec2)
    dec0 = self.dec0(dec1)

    return self.final(F.dropout2d(dec0, p=self.dropout_2d))
# blocks
    class DecoderBlockV(nn.Module):
        def __init__(self, in_channels, middle_channels, out_channels, is_deconv=True):
            super(DecoderBlockV2, self).__init__()
            self.in_channels = in_channels

            if is_deconv:
                self.block = nn.Sequential(
                    ConvRelu(in_channels, middle_channels),
                    nn.ConvTranspose2d(middle_channels, out_channels, kernel_size=4, stride=2,
                                       padding=1),
                    nn.BatchNorm2d(out_channels),
                    nn.ReLU(inplace=True)

                )
            else:
                self.block = nn.Sequential(
                    nn.Upsample(scale_factor=2, mode='bilinear'),
                    ConvRelu(in_channels, middle_channels),
                    ConvRelu(middle_channels, out_channels),
                )

        def forward(self, x):
            return self.block(x)

class DecoderCenter(nn.Module):
    def __init__(self, in_channels, middle_channels, out_channels, is_deconv=True):
        super(DecoderCenter, self).__init__()
        self.in_channels = in_channels

        if is_deconv:
            """
                Paramaters for Deconvolution were chosen to avoid artifacts, following
                link https://distill.pub/2016/deconv-checkerboard/
            """

            self.block = nn.Sequential(
                ConvRelu(in_channels, middle_channels),
                nn.ConvTranspose2d(middle_channels, out_channels, kernel_size=4, stride=2,
                                   padding=1),
        nn.BatchNorm2d(out_channels),
                nn.ReLU(inplace=True)
            )
        else:
            self.block = nn.Sequential(
                ConvRelu(in_channels, middle_channels),
                ConvRelu(middle_channels, out_channels)

            )

    def forward(self, x):
        return self.block(x)

Then I edited my class looks to make it work without pooling layer:

class UNetResNet(nn.Module):
    def __init__(self, encoder_depth, num_classes, num_filters=32, dropout_2d=0.2,
                 pretrained=False, is_deconv=False):
        super().__init__()
        self.num_classes = num_classes
        self.dropout_2d = dropout_2d

        if encoder_depth == 34:
            self.encoder = torchvision.models.resnet34(pretrained=pretrained)
            bottom_channel_nr = 512
        elif encoder_depth == 101:
            self.encoder = torchvision.models.resnet101(pretrained=pretrained)
            bottom_channel_nr = 2048
        elif encoder_depth == 152:
            self.encoder = torchvision.models.resnet152(pretrained=pretrained)
            bottom_channel_nr = 2048
        else:
            raise NotImplementedError('only 34, 101, 152 version of Resnet are implemented')

        self.relu = nn.ReLU(inplace=True)

        self.input_adjust = nn.Sequential(self.encoder.conv1,
                                          self.encoder.bn1,
                                          self.encoder.relu)

        self.conv1 = self.encoder.layer1
        self.conv2 = self.encoder.layer2
        self.conv3 = self.encoder.layer3
        self.conv4 = self.encoder.layer4

        self.dec4 = DecoderBlockV(bottom_channel_nr, num_filters * 8 * 2, num_filters * 8, is_deconv)
        self.dec3 = DecoderBlockV(bottom_channel_nr // 2 + num_filters * 8, num_filters * 8 * 2, num_filters * 8,    is_deconv)
        self.dec2 = DecoderBlockV(bottom_channel_nr // 4 + num_filters * 8, num_filters * 4 * 2, num_filters * 2,    is_deconv)
        self.dec1 = DecoderBlockV(bottom_channel_nr // 8 + num_filters * 2, num_filters * 2 * 2, num_filters * 2 * 2,is_deconv)
        self.final = nn.Conv2d(num_filters * 2 * 2, num_classes, kernel_size=1)

    def forward(self, x):
        input_adjust = self.input_adjust(x)
        conv1 = self.conv1(input_adjust)
        conv2 = self.conv2(conv1)
        conv3 = self.conv3(conv2)
        center = self.conv4(conv3)
        dec4 = self.dec4(center) #now without centblock
        dec3 = self.dec3(torch.cat([dec4, conv3], 1))
        dec2 = self.dec2(torch.cat([dec3, conv2], 1))
        dec1 = F.dropout2d(self.dec1(torch.cat([dec2, conv1], 1)), p=self.dropout_2d)
        return self.final(dec1)

is_deconv - in both cases True. After changing it stop to work with batchsize 64, only with with size of 16 or with batchsize 64 but with resnet16 only - otherwise out of cuda memory. What am I doing wrong?

Full stack of error:

~/Desktop/ml/salt/open-solution-salt-identification-master/common_blocks/unet_models.py in forward(self, x)
    418         conv1 = self.conv1(input_adjust)
    419         conv2 = self.conv2(conv1)
--> 420         conv3 = self.conv3(conv2)
    421         center = self.conv4(conv3)
    422         dec4 = self.dec4(center)

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    355             result = self._slow_forward(*input, **kwargs)
    356         else:
--> 357             result = self.forward(*input, **kwargs)
    358         for hook in self._forward_hooks.values():
    359             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/container.py in forward(self, input)
     65     def forward(self, input):
     66         for module in self._modules.values():
---> 67             input = module(input)
     68         return input
     69

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    355             result = self._slow_forward(*input, **kwargs)
    356         else:
--> 357             result = self.forward(*input, **kwargs)
    358         for hook in self._forward_hooks.values():
    359             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/torchvision-0.2.0-py3.6.egg/torchvision/models/resnet.py in forward(self, x)
     79
     80         out = self.conv2(out)
---> 81         out = self.bn2(out)
     82         out = self.relu(out)


title: My numpy and pytorch codes have totally different results

body:
I wanted to calculate the sum of 1st to K-th power of an array and equally calculate the sum of 1st to k-th power of a tensor. I found out that the following codes and their results are totally different and I don't know why.

I debugged the code and I know that the results are equal in the first round.

Numpy code:

adj_k_prob = adj_prob
adj_k_pow =  adj_prob
for i in range(K):
    adj_k_pow = np.matmul(adj_prob, adj_k_pow)
    adj_k_prob += adj_k_pow

Pytorch code:

adj_k_prob = adj_prob_tensor
adj_k_pow = adj_prob_tensor
for i in range(K):
    adj_k_pow = torch.matmul(adj_prob_tensor, adj_k_pow)
    adj_k_prob += adj_k_pow

The value of adj_prob_tensor and adj_prob at the start of loop are as follow:

tensor([[0.0000, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429],
    [0.2500, 0.0000, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.2500],
    [0.2500, 0.2500, 0.0000, 0.2500, 0.0000, 0.0000, 0.0000, 0.2500],
    [0.2500, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.0000, 0.2500],
    [0.5000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.5000, 0.0000],
    [0.5000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.5000, 0.0000],
    [0.3333, 0.0000, 0.0000, 0.0000, 0.3333, 0.3333, 0.0000, 0.0000],
    [0.2500, 0.2500, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.0000]])

Is there anything that I should check for it?


title: List not populated correctly unless use PyTorch clone()

body:
I'm attempting to add the final weights of each trained model to a list using below code :

%reset -f

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch.utils.data as data_utils
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from matplotlib import pyplot
from pandas import DataFrame
import torchvision.datasets as dset
import os
import torch.nn.functional as F
import time
import random
import pickle
from sklearn.metrics import confusion_matrix
import pandas as pd
import sklearn

trans = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])

root = './data'
if not os.path.exists(root):
    os.mkdir(root)
train_set = dset.MNIST(root=root, train=True, transform=trans, download=True)
test_set = dset.MNIST(root=root, train=False, transform=trans, download=True)

batch_size = 64

train_loader = torch.utils.data.DataLoader(
                 dataset=train_set,
                 batch_size=batch_size,
                 shuffle=True)
test_loader = torch.utils.data.DataLoader(
                dataset=test_set,
                batch_size=batch_size,
shuffle=True)

class NeuralNet(nn.Module):
    def __init__(self):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(28*28, 500)
        self.fc2 = nn.Linear(500, 256)
        self.fc3 = nn.Linear(256, 2)
    def forward(self, x):
        x = x.view(-1, 28*28)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

num_epochs = 2
random_sample_size = 200

values_0_or_1 = [t for t in train_set if (int(t[1]) == 0 or int(t[1]) == 1)]
values_0_or_1_testset = [t for t in test_set if (int(t[1]) == 0 or int(t[1]) == 1)]

print(len(values_0_or_1))
print(len(values_0_or_1_testset))

train_loader_subset = torch.utils.data.DataLoader(
                 dataset=values_0_or_1,
                 batch_size=batch_size,
                 shuffle=True)

test_loader_subset = torch.utils.data.DataLoader(
                 dataset=values_0_or_1_testset,
                 batch_size=batch_size,
                 shuffle=False)

train_loader = train_loader_subset

# Hyper-parameters
input_size = 100
hidden_size = 100
num_classes = 2
# learning_rate = 0.00001
learning_rate = .0001
# Device configuration
device = 'cpu'
print_progress_every_n_epochs = 1

model = NeuralNet().to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

N = len(train_loader)
# Train the model
total_step = len(train_loader)

most_recent_prediction = []
test_actual_predicted_dict = {}

rm = random.sample(list(values_0_or_1), random_sample_size)
train_loader_subset = data_utils.DataLoader(rm, batch_size=4)

weights_without_clone = []
weights_with_clone = []

for i in range(0 , 2) :
    for epoch in range(num_epochs):
        for i, (images, labels) in enumerate(train_loader_subset):
            # Move tensors to the configured device
            images = images.reshape(-1, 2).to(device)
            labels = labels.to(device)

            # Forward pass
            outputs = model(images)
            loss = criterion(outputs, labels)

            # Backward and optimize
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        if (epoch) % print_progress_every_n_epochs == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))

    print('model fc2 weights ' , model.fc2.weight.data)
    weights_without_clone.append(model.fc2.weight.data)
    weights_with_clone.append(model.fc2.weight.data.clone())

Output of model :

12665
2115
Epoch [1/2], Step [50/198], Loss: 0.0968
Epoch [2/2], Step [50/198], Loss: 0.0082
model fc2 weights  tensor([[-3.9507e-02, -4.0454e-02,  3.5576e-03,  ...,  6.2181e-03,
          4.1372e-02, -6.2960e-03],
        [ 1.8778e-02,  2.7049e-02, -3.5624e-02,  ...,  2.6797e-02,
          2.2041e-03, -4.2284e-02],
        [ 1.9571e-02, -3.2545e-02,  2.6618e-02,  ..., -1.6139e-02,
          4.1192e-02, -2.3458e-02],
        ...,
        [-4.6123e-03,  2.6943e-02,  3.9979e-02,  ..., -3.3848e-02,
          3.6096e-02,  2.4211e-02],
        [-1.4698e-02,  9.7528e-04, -2.5244e-03,  ..., -3.3145e-02,
          1.0888e-02,  3.1091e-02],
        [-1.7451e-02, -2.1646e-02,  2.5885e-02,  ...,  4.0453e-02,
         -6.5324e-03, -3.5410e-02]])
Epoch [1/2], Step [50/198], Loss: 0.0025
Epoch [2/2], Step [50/198], Loss: 0.0013
model fc2 weights  tensor(1.00000e-02 *
       [[-3.9891, -4.0454,  0.3558,  ...,  0.7168,  4.1902, -0.6253],
        [ 1.8766,  2.7049, -3.5632,  ...,  2.6785,  0.2192, -4.2297],
        [ 2.1426, -3.2545,  2.6621,  ..., -1.6285,  4.1196, -2.2653],
        ...,
        [-0.4930,  2.6943,  3.9971,  ..., -3.2940,  3.6641,  2.4248],
        [-1.5160,  0.0975, -0.2524,  ..., -3.1938,  1.1753,  3.1065],
        [-1.8116, -2.1646,  2.5883,  ...,  4.1355, -0.5921, -3.5416]])

Printing the values of weights_without_clone :

print(weights_without_clone[0])
print(weights_without_clone[1])

outputs :

tensor(1.00000e-02 *
       [[-3.9891, -4.0454,  0.3558,  ...,  0.7168,  4.1902, -0.6253],
        [ 1.8766,  2.7049, -3.5632,  ...,  2.6785,  0.2192, -4.2297],
        [ 2.1426, -3.2545,  2.6621,  ..., -1.6285,  4.1196, -2.2653],
        ...,
        [-0.4930,  2.6943,  3.9971,  ..., -3.2940,  3.6641,  2.4248],
        [-1.5160,  0.0975, -0.2524,  ..., -3.1938,  1.1753,  3.1065],
        [-1.8116, -2.1646,  2.5883,  ...,  4.1355, -0.5921, -3.5416]])
tensor(1.00000e-02 *
       [[-3.9891, -4.0454,  0.3558,  ...,  0.7168,  4.1902, -0.6253],
        [ 1.8766,  2.7049, -3.5632,  ...,  2.6785,  0.2192, -4.2297],
        [ 2.1426, -3.2545,  2.6621,  ..., -1.6285,  4.1196, -2.2653],
        ...,
        [-0.4930,  2.6943,  3.9971,  ..., -3.2940,  3.6641,  2.4248],
        [-1.5160,  0.0975, -0.2524,  ..., -3.1938,  1.1753,  3.1065],
        [-1.8116, -2.1646,  2.5883,  ...,  4.1355, -0.5921, -3.5416]])

Printing the values of weights_with_clone :

print(weights_with_clone[0])
print(weights_with_clone[1])

outputs :

tensor([[-3.9507e-02, -4.0454e-02,  3.5576e-03,  ...,  6.2181e-03,
          4.1372e-02, -6.2960e-03],
        [ 1.8778e-02,  2.7049e-02, -3.5624e-02,  ...,  2.6797e-02,
          2.2041e-03, -4.2284e-02],
        [ 1.9571e-02, -3.2545e-02,  2.6618e-02,  ..., -1.6139e-02,
          4.1192e-02, -2.3458e-02],
        ...,
        [-4.6123e-03,  2.6943e-02,  3.9979e-02,  ..., -3.3848e-02,
          3.6096e-02,  2.4211e-02],
        [-1.4698e-02,  9.7528e-04, -2.5244e-03,  ..., -3.3145e-02,
          1.0888e-02,  3.1091e-02],
        [-1.7451e-02, -2.1646e-02,  2.5885e-02,  ...,  4.0453e-02,
         -6.5324e-03, -3.5410e-02]])
tensor(1.00000e-02 *
       [[-3.9891, -4.0454,  0.3558,  ...,  0.7168,  4.1902, -0.6253],
        [ 1.8766,  2.7049, -3.5632,  ...,  2.6785,  0.2192, -4.2297],
        [ 2.1426, -3.2545,  2.6621,  ..., -1.6285,  4.1196, -2.2653],
        ...,
        [-0.4930,  2.6943,  3.9971,  ..., -3.2940,  3.6641,  2.4248],
        [-1.5160,  0.0975, -0.2524,  ..., -3.1938,  1.1753,  3.1065],
        [-1.8116, -2.1646,  2.5883,  ...,  4.1355, -0.5921, -3.5416]])

Why is 1.00000e-02 * prepended to the final weight value of the second model ?

Why is using clone() required in order to add final weights for each iteration as omitting clone() the same weights are added to each iteration ? :

weights_without_clone.append(model.fc2.weight.data)
weights_with_clone.append(model.fc2.weight.data.clone())


title: Why does creating a single tensor on the GPU take 2.5 seconds in PyTorch?

body:
I'm just going through the beginner tutorial on PyTorch and noticed that one of the many different ways to put a tensor (basically the same as a numpy array) on the GPU takes a suspiciously long amount compared to the other methods:

import time
import torch

if torch.cuda.is_available():
    print('time =', time.time())
    x = torch.randn(4, 4)
    device = torch.device("cuda")
    print('time =', time.time())
    y = torch.ones_like(x, device=device)  # directly create a tensor on GPU  => 2.5 secs??
    print('time =', time.time())
    x = x.to(device)                       # or just use strings ``.to("cuda")``
    z = x + y
    print(z)
    print(z.to("cpu", torch.double))       # ``.to`` can also change dtype together!
    a = torch.ones(5)
    print(a.cuda())
    print('time =', time.time())
else:
    print('I recommend you get CUDA to work, my good friend!')

Output (just times):

time = 1551809363.28284
time = 1551809363.282943
time = 1551809365.7204516  # (!)
time = 1551809365.7236063

Version details:

1 CUDA device: GeForce GTX 1050, driver version 415.27
CUDA          = 9.0.176
PyTorch       = 1.0.0
cuDNN         = 7401
Python        = 3.5.2
GCC           = 5.4.0
OS            = Linux Mint 18.3
Linux kernel  = 4.15.0-45-generic

As you can see this one operation ("y = ...") takes much longer (2.5 seconds) than the rest combined (.003 seconds). I'm confused about this as I expect all these methods to basically do the same. I've tried making sure the types in this line are 32 bit or have different shapes but that didn't change anything.


title: Multiply a specific column in each row by -1

body:
I have a pytorch tensor A, that's of size (n,m) and a list of indices for size n, such that each entry of 0 <= indices[i] < m. For each row i of A, I want to multiply A[i, indices[i]] *= -1, in a vectorized way. Is there an easy way to do this?

 A = torch.tensor([[1,2,3],[4,5,6]])
 indices = torch.tensor([1, 2])

 #desired result
 A = [[1,-2,3],[4,5,-6]]


title: Deriving the structure of a pytorch network

body:
For my use case, I require to be able to take a pytorch module and interpret the sequence of layers in the module so that I can create a “connection” between the layers in some file format. Now let’s say I have a simple module as below

class mymodel(nn.Module):
    def __init__(self, input_channels):
        super(mymodel, self).__init__()
        self.fc = nn.Linear(input_channels, input_channels)
    def forward(self, x):
        out = self.fc(x)
        out += x
        return out

if __name__ == "__main__":
    net = mymodel(5)

    for mod in net.modules():
        print(mod)

Here the output yields:

mymodel(
  (fc): Linear(in_features=5, out_features=5, bias=True)
)
Linear(in_features=5, out_features=5, bias=True)

as you can see the information about the plus equals operation or plus operation is not captured as it is not a nnmodule in the forward function. My goal is to be able to create a graph connection from the pytorch module object to say something like this in json :

layers {
"fc": {
"inputTensor" : "t0",
"outputTensor": "t1"
}
"addOp" : {
"inputTensor" : "t1",
"outputTensor" : "t2"
}
}

The input tensor names are arbitrary but it captures the essence of the graph and the connections between layers.

My question is, is there a way to extract the information from a pytorch object? I was thinking to use the .modules() but then realized that hand written operations are not captured this way as a module. I guess if everything is an nn.module then the .modules() might give me the network layer arrangement. Looking for some help here. I want to be able to know the connections between tensors to create a format as above.


title: Computing Linear Layer in Tensor/Outer-Product space in PyTorch is Very Slow

body:
I would like to make a PyTorch model that takes the outer product of the input with itself and then does a linear regression on that. As an example, consider the input vector [1,2,3], then I would like to compute w and b to optimize [1*1, 1*2, 1*3, 2*1, 2*2, 2*3, 3*1, 3*2, 3*3] @ w + b.

For a batch input with r rows and c columns, I can do this in PyTorch with

(input.reshape(r,c,1) @ input.reshape(r,1,c)).reshape(r,c**2) @ weigts + b

My problem is that it is extraordinarily slow. Like a factor 1000 times slower and more memory consumptious than Adding a fully connected c*c RELU layer, even though it has the same number of weights.

My question is why this happens?

Is reshape a very expensive operation for PyTorch? Could I reformulate it in a different way that would make things more efficient?

Another equivalent formulation I know is torch.diag(input @ weights @ input.T) + b, but now we are computing way more values than we need (r*r) just to throw them away again.


title: In an adversarial PyTorch model, how to avoid a second forward pass?

body:
The classic PyTorch example/tutorial for a GAN training loop is shown here.
You can find the lines

# Since we just updated D, perform another forward pass of all-fake batch through D
output = netD(fake).view(-1)

but since such an adversary can become quite large this seems like a very unnecessary thing to do. To get more concrete I have a minimal example of my (in some point different) adversarial training where it matters even more (see end of this post). Basically, I have a network X that produces output out1 and a network Y that produces two loss terms out2_x and out2_y. Each of these loss terms should be used to update the weights of the corresponding model. So to update the weights of X I need to do a full backward pass through both networks. If I count the runtime costs in terms of network evaluations (forward or backward) I have now had 2 forward and 2 backward network evaluations. To update the second network, I hoped just to clear the previous gradients of this part and do a backward pass till out1 and stop there. This would add one additional backward network evaluation and I have all my needed gradient updates.

The aforementioned GAN tutorial would suggest doing an additional forward pass on the second network which would result in a total of 6 instead of 5 network evaluations, which seems to me like easily avoidable and unnecessary.

However my idea of stoping somehow autograds backward pass early seems not easily possible. I found a solution that behaves like described above (see example below), but I wonder if it isn't to expensive, as setting require_grads to False for all X weights means that PyTorch also needs to do some additional graph updates to figure out how deep the backward pass needs to go, and as this happens on the Python side (right?) this can be potentially quite slow for large network architectures with many single tensor operations?! Also with having one single bottleneck tensor out1 I was hoping to have a cleaner code by telling autograd specifically to stop the backward pass there, and now I am really curious why this seems so hard to achieve in the framework.

Any insights are very much appreciated!

import torch

# Setting up a dummy graph
x1: torch.Tensor = torch.tensor(1.0, requires_grad=True)
x2: torch.Tensor = torch.tensor(3.0, requires_grad=True)

out1 = x1 * (x2 + 1)
y: torch.Tensor = torch.tensor(11.0, requires_grad=True)

out2_x = out1 * y + y ** 2
out2_y = out1 ** 2 * y ** 2

opt1 = torch.optim.SGD([x1, x2], lr=0.01)
opt2 = torch.optim.SGD([y], lr=0.01)
# Finished setting up a dummy graph

# showcasing some gradients
opt1.zero_grad()
opt2.zero_grad()
out2_x.backward(retain_graph=True)
print("grads wrt out2_x\t\t\t:", x1.grad, x2.grad, y.grad)
opt1.zero_grad()
opt2.zero_grad()
out2_y.backward(retain_graph=True)
print("grads wrt out2_y\t\t\t:", x1.grad, x2.grad, y.grad)

# actual logic for computing gradients
opt1.zero_grad()
opt2.zero_grad()
out2_x.backward(retain_graph=True)
# the following two lines work
x1.requires_grad = False
x2.requires_grad = False
# but I was hoping something like the following would work
#   * out1.requires_grad = False
#   * out1.is_leaf = True
#   * out1.detach() (this would require a second forward pass)
#   * out2_y.backward(tell it to stop at out1)
print("grads wrt out2_x\t\t\t:", x1.grad, x2.grad, y.grad)
opt2.zero_grad()
print("grads after zeroing y\t\t\t:", x1.grad, x2.grad, y.grad)
out2_y.backward()
print("grads after out2_y with no_grads x\t:", x1.grad, x2.grad, y.grad)
opt1.step()
opt2.step()
# x1.requires_grad = True
# x2.requires_grad = True

Output:

grads wrt out2_x            : tensor(44.) tensor(11.) tensor(26.)
grads wrt out2_y            : tensor(3872.) tensor(968.) tensor(352.)
grads wrt out2_x            : tensor(44.) tensor(11.) tensor(26.)
grads after zeroing y           : tensor(44.) tensor(11.) tensor(0.)
grads after out2_y with no_grads x  : tensor(44.) tensor(11.) tensor(352.)


title: Lists and numpy arrays python

body:
I actually have multiple questions that are related.

I have the following class:

class environment_step:
    def __init__(self,solar,battery,allocation,G1,G2):
        self.solar = solar
        self.battery = battery
        self.allocation = allocation
        self.G1 = G1
        self.G2 = G2

    def step(self):
        self.residual = self.solar+self.battery-self.allocation
        self.reward = min(0,max(self.residual*self.G1,self.residual*self.G2))
        self.cost = -self.reward
        self.battery = max(0,self.residual)
        return self.reward, self.cost, self.battery

I use it as follows

reward = []
B = np.zeros(1)
allocation = np.ones(1)*12
G1 = np.ones(24)
G2 = np.ones(24)*2
solar = np.random.rand(24)
for t in range(24):
    environment = environment_step(solar[t], B, allocation, G1[t], G2[t])
    R, C, B = environment.step()
    reward.append(R)

Where solar, battery, allocation, G1, and G2 are numpy arrays. When I check R, if it is 0, then I just get 0, but for something different than zero, I get [[-2.114]] or whatever the number is. Why am I getting that instead of [-2.114]?

I run the environment multiple times with different values for the entries and append R to the list reward. That list is going to look something like the following:
[0, 0, array([[-2.114]], dtype = float32), 0, array([[-1.324]], dtype = float32)]. I convert that list reward to a pytorch tensor by reward = torch.tensor(reward, dtype=torch.float32). When the list begins with 0, as in the example list I showed, it works perfectly. However, if the list starts with non-zero, for instance [array([[-2.114]], dtype =float32) 0, 0, 0, array([[-1.324]], dtype = float32)], then I get the error TypeError: not a sequence. I suspect the problem is the double brackets of array, but I am not sure. That is why I'd like to fix that before (the above paragraph).

Even though, I am working with torch tensors, I think the problem is happening before with the arrays. So, what is the difference between let's say [1.2, 1.42, 3.13] and [array([1.2], dtype=float32), array([1.42], dtype=float32), array([3.13], dtype=float32)]? or even the same but with double brackets (my case). I know that lists are general and admit texts, whereas arrays only admit numbers, but from this there seem to be more to it, I am not sure though. I'd appreciate any help!

Edit: I have added a minimum working example.


title: How to do language model training on BERT

body:
I want to train BERT on a target corpus. I am looking at this HuggingFace implementation.
They are using .raw files for the training data. If I have .txt files of my training data, how can I use their implementation?


title: Is there a function that finds the OR of every row in a boolean matrix in pytorch?

body:
I have a matrix A of size n x m and all the entries are bools.  I want all my calculation to be on the GPU and I have matrix A stored as a tensor with each entry being the pytorch's bool datatype.  I want an output of a single vector, b, to be a 1 x m tensor that stores the OR of all rows in A.

What I want:
A matrix =
[a1,1, a1,2, , a1,3, ... , a1,m]
[a2,1, a2,2, , a2,3, ... , a2,m]
...
[an,1, an,2, , an,3, ... , an,m]

b = [b1, b2, , b3, , ..., bm]

s.t. bi = a1,i | a2,i | a3,i | ... | an,iwhere | is the OR operator in Pytorch

essentially I want a function that applies row or column wise boolean operation. I know the | .__OR__ can be used and Pytorch's OR function takes two boolean tensor as inputs and I would need to loop over all rows to get my desired b vector.
Since OR is commutative (a|(b|c) = (a|b)|c), I would think pytorch would have some nice function that speeds it up by doing the | operations in parallel or in some divide and conquer method, instead of doing this with a loop. Any ideas or references to speed up the process of applying communicative row/column wise boolean operation with pytorch is welcomed.  Best if all operations are done on the GPU.


title: Pytorch - Target and input must have the same number of elements

body:
I'm getting this error in pytorch :
ValueError: Target and input must have the same number of elements. target nelement (16) != input nelement (8388608)
My model :
class DoubleConv(nn.Module):

    def __init__(self, in_channels, out_channels, mid_channels=None):
        super().__init__()
        if not mid_channels:
            mid_channels = out_channels
        self.double_conv = nn.Sequential(
            nn.Conv2d(in_channels, mid_channels, kernel_size=3, padding=1),
            nn.BatchNorm2d(mid_channels),
            nn.ReLU(inplace=True),
            nn.Conv2d(mid_channels, out_channels, kernel_size=3, padding=1),
            nn.BatchNorm2d(out_channels),
            nn.ReLU(inplace=True)
        )

    def forward(self, x):
        return self.double_conv(x)

class Down(nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.maxpool_conv = nn.Sequential(
            nn.MaxPool2d(2),
            DoubleConv(in_channels, out_channels)
        )

    def forward(self, x):
        return self.maxpool_conv(x)

class Up(nn.Module):
    def __init__(self, in_channels, out_channels, bilinear=True):
        super().__init__()

        if bilinear:
            self.up = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)
            self.conv = DoubleConv(in_channels, out_channels, in_channels / 2)
        else:
            self.up = nn.ConvTranspose2d(in_channels , in_channels / 2, kernel_size=2, stride=2)
            self.conv = DoubleConv(in_channels, out_channels)

    def forward(self, x1, x2):
        x1 = self.up(x1)

        diffY = x2.size()[2] - x1.size()[2]
        diffX = x2.size()[3] - x1.size()[3]

        x1 = F.pad(x1, [diffX / 2, diffX - diffX / 2,
                        diffY / 2, diffY - diffY / 2])
        x = torch.cat([x2, x1], dim=1)
        return self.conv(x)

class OutConv(nn.Module):
    def __init__(self, in_channels, out_channels):
        super(OutConv, self).__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size=1)

    def forward(self, x):
        return self.conv(x)

class UNet(nn.Module):
    def __init__(self, n_channels, n_classes, bilinear=True):
        super(UNet, self).__init__()
        self.n_channels = n_channels
        self.n_classes = n_classes
        self.bilinear = bilinear

        self.inc = DoubleConv(n_channels, 64)
        self.down1 = Down(64, 128)
        self.down2 = Down(128, 256)
        self.down3 = Down(256, 512)
        factor = 2 if bilinear else 1
        self.down4 = Down(512, 1024 / factor)
        self.up1 = Up(1024, 512 / factor, bilinear)
        self.up2 = Up(512, 256 / factor, bilinear)
        self.up3 = Up(256, 128 / factor, bilinear)
        self.up4 = Up(128, 64, bilinear)
        self.outc = OutConv(64, n_classes)

    def forward(self, x):
        x1 = self.inc(x)
        x2 = self.down1(x1)
        x3 = self.down2(x2)
        x4 = self.down3(x3)
        x5 = self.down4(x4)
        x = self.up1(x5, x4)
        x = self.up2(x, x3)
        x = self.up3(x, x2)
        x = self.up4(x, x1)
        logits = self.outc(x)
        return logits

model_instance = UNet(n_channels=3, n_classes=8, bilinear=True)

Complete log :
/home/user/.local/lib/python3.6/site-packages/torch/nn/modules/loss.py:529: UserWarning: Using a target size (torch.Size([16])) that is different to the input size (torch.Size([16, 8, 256, 256])) is deprecated. Please ensure they have the same size.
  return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-13-aad65c925500> in <module>()
      1 for _epoch in range(1, epoch):
----> 2     train(model_instance, _epoch)
      3     test(model_instance)

<ipython-input-11-3b706bbe2772> in train(model, epoch)
      6         optimizer.zero_grad()
      7         output = model.forward(sat.float())
----> 8         loss = criterion(output.float(), mask.float())
      9         optimizer.zero_grad()
     10         loss.backward()

/home/user/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/home/user/.local/lib/python3.6/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    527
    528     def forward(self, input: Tensor, target: Tensor) -> Tensor:
--> 529         return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
    530
    531

/home/user/.local/lib/python3.6/site-packages/torch/nn/functional.py in binary_cross_entropy(input, target, weight, size_average, reduce, reduction)
   2475     if input.numel() != target.numel():
   2476         raise ValueError("Target and input must have the same number of elements. target nelement ({}) "
-> 2477                          "!= input nelement ({})".format(target.numel(), input.numel()))
   2478
   2479     if weight is not None:

ValueError: Target and input must have the same number of elements. target nelement (16) != input nelement (8388608)

Is there some problem regarding my model ? Can I enhance it? All sorts of advice is appreciated.
I'm inputting 256x256 3-channel images as input and 256x256 3-channel as mask.