title: Pytorch batchnorm1D with a batch size of 1

body:
I am trying out a Binary Network implementation. And am using a simple perceptron with custom Binarized Layer as shown below
self.w = torch.nn.Parameter((torch.normal(0, 1, size=(param['input_size'], param['hidden_size']))),
                                    requires_grad=False)
self.fc1 = BinarizeLinear(param['hidden_size'], param['n_classes'])
self.bn = nn.BatchNorm1d(param['n_classes'])
self.softmax = nn.LogSoftmax()

However, I want it train it with a batch size of 1. I tried InstanceNorm but instance norm computes the mean and deviation over the Height and width axes. Any recommendations or ideas on how I can perform normalization when training with a batch of 1. Are there any workarounds this or do I have to make a custom Normalization layer?
As a workaround to that, I tried reshaping the input in the forward pass after the fc1 as a way to rescale the inputs so that they pass through pytorch's ``_check_dim` method.
x = self.fc1(x)
x = x.reshape(param['batch_size'],x.shape[0], x.shape[1])
x = self.bn(x)

But I get the error
RuntimeError: running_mean should contain 1 elements not 2
which makes sense. I am trying hacky ways around it, would appreciate it if someone can just direct me regarding how would be the correct way to implement.


title: Integrating PyTorch models with TVM makes it worse from time perspective

body:
I was recently told that integrating/compiling my PyTorch model with the help of Apache TVM should speed inference up, but am very confused about the usage of TVM as there seem to be multiple ways to use it.
This blog on their website says
Usage is simple:
import torch_tvm
torch_tvm.enable()

That’s it! PyTorch will then attempt to convert all operators it can to known Relay operators during its JIT compilation process.
If just this much does the job, why are other specific tutorials so much more comprehensive? After going through a fair few tutorials another thing i assumed (not sure if i am right) was that TVM cannot really use the default way torch code is run and needs the 'jit' compiler (non-default way of running torch code) to work over. This assumption was based off of the fact that the github repo tutorial shows the following snippet
import torch
import torch_tvm

torch_tvm.enable()

# The following function will be compiled with TVM
@torch.jit.script
def my_func(a, b, c):
    return a * b + c

where the function my_func is wrapped with a decorator that looks to be compiling the function with jit. But using this exact same function and timing its usage with and without the wrapper + TVM shows that the normal usage of this function is far more time efficient. If its not speeding things up, what is exactly does jit+tvm help in? and if it is supposed to be speeding things up, why is that not the case here?
P.S. I apologise for the comprehensive description of my problem. I was not able to figure this out and get whats happening even after reading and tinkering around a fair amount with torch_tvm. Would appreciate any link to resources or any explanation that might help me out.


title: Pytorch on Windows : Dataloader problems with numworkers

body:
I have just got a new computer running Windows 10 which has a GPU so I wanted to see if I could sensibly use it for machine learning.
So I tried running an old model which I previously trained on Google Colab.
The answer is that it does do quite well, but I discovered that I could not use more than one worker in the Dataloader. Googling found that this is a known issue with PyTorch on Windows in Jupyter Notebooks so I tried running it in a normal Python program. I found that it did work but that the creation of the DataIterator took a very long time. Below are the times in seconds for 1, 2 and 6 workers, each done twice:

I note that 2 workers seems to be the fastest and there seems to be quite a lot of variation which surprised me as the machine was doing nothing else.
So the first question is:

Is there a way to let PyTorch choose the most efficient number of workers to use?

The second question is:

If I install a version of Linux will I be able to use Jupyter Notebooks with multiple workers which is what I would prefer to do in an ideal world.

The code I ran is below the relevant part is after if __name__ == "__main__":
# -*- coding: utf-8 -*-

import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import Dataset,DataLoader

import os
import numpy as np
#import gym
import pickle
import matplotlib.pyplot as plt
import time

class BasicBlock(nn.Module):
    expansion = 1

    def __init__(self, in_planes, planes, stride=1):
        super(BasicBlock, self).__init__()
        self.conv1 = nn.Conv2d(
            in_planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(planes)
        self.conv2 = nn.Conv2d(planes, planes, kernel_size=3,
                               stride=1, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(planes)

        self.shortcut = nn.Sequential()
        if stride != 1 or in_planes != self.expansion*planes:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_planes, self.expansion*planes,
                          kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(self.expansion*planes)
            )

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += self.shortcut(x)
        out = F.relu(out)
        return out

# create dataset
class C4Dataset(Dataset):
    '''
        the data for the first 12moves is held in a pickles list
        as (key,val)
        The key has to be converted to the pos amd mask which can then be converted to the ones,twos and zeros
        Val is the value for the player playing so needs to be changed for wnen moves  i odd to minus val
    '''

    fileName = r'C:\Users\alan\Desktop\Python\Python36\connect4\Layers\ListAllKeyVal19'
    bottom_mask = 4432676798593
    board_mask  = bottom_mask * ((1 << 6) - 1)
    bNos = 2**np.arange(49, dtype = np.uint64)

    def getData(fileName):
        with open(fileName,'rb') as inFile:
            dict = pickle.load(inFile)
        return dict

    def oneHot(x):
        return np.eye(37,dtype = np.float32)[x]

    def getNoMoves(ones,twos) :
        return np.sum(ones+twos)

    def getPosMask(key):

        binary = ('{:049b}'.format(key))[::-1]
        arr    = np.frombuffer(binary.encode(encoding='utf-8', errors='strict'),'u1') - ord('0')
        outArr = np.reshape(arr,(7,7),order = 'F')

        arr = np.flipud(outArr)
        pos = arr.copy()
        mask =arr.copy()

        for col in range(7):
            res = np.where(arr[:,col]==1)
            topPos = res[0][0]
            pos[topPos,col]  = 0
            mask[topPos,col] = 0
            if topPos<6:
                mask[topPos+1:,col] = 1

        msk = np.flipud(mask)
        msk = np.reshape(msk,(49),order = 'F')
        maskNo  = np.array(msk.dot(C4Dataset.bNos),dtype = np.uint64).item()

        return pos.astype('float32'),(pos ^ mask).astype('float32'),(np.logical_not(mask)).astype('float32'),maskNo

    def possible(mask) :
        poss = (mask + C4Dataset.bottom_mask) & C4Dataset.board_mask
        binary = ('{:049b}'.format(poss))[::-1]
        arr    = np.frombuffer(binary.encode(encoding='utf-8', errors='strict'),'u1') - ord('0')
        outArr = np.reshape(arr,(7,7),order = 'F')

        arr = np.flipud(outArr)

        return arr

    def __init__(self):
        self.lst = C4Dataset.getData(C4Dataset.fileName)

    def __len__(self):
        return len(self.lst)

    def __getitem__(self, idx):
        key,val = self.lst[idx]
        val = int(val)
        ones,twos,zeros,mask = C4Dataset.getPosMask(key)

        arr = np.zeros((5,7,7),dtype = np.float32)
        arr[0,:6,:7] = ones[1:,:]
        arr[1,:6,:7] = twos[1:,:]
        arr[2,:6,:7] = zeros[1:,:]

        moves = int(C4Dataset.getNoMoves(ones,twos))

        p = (moves % 2) + 3

        arr[p,:6,:7] = C4Dataset.possible(mask)[1:,:]

        return arr,val+18 #C4Dataset.oneHot(val+18)

class C4Net(nn.Module):
    def __init__(self, inFilters,outFilters):
        super(C4Net, self).__init__()

        self.conv1 = nn.Conv2d(inFilters, 32, kernel_size=3,
                               stride=1, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(32)
        self.layer1 = BasicBlock(32,32)
        self.layer2 = BasicBlock(32,32)
        self.layer3 = BasicBlock(32,32)
        self.layer4 = BasicBlock(32,32)
        self.layer5 = BasicBlock(32,32)
        self.layer6 = BasicBlock(32,32)
        self.layer7 = BasicBlock(32,32)
        self.layer8 = BasicBlock(32,32)

        self.linear = nn.Linear(32*7*7,outFilters)#1568

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.layer1(out)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = self.layer5(out)
        out = self.layer6(out)
        out = self.layer7(out)
        out = self.layer8(out)

        #out = F.avg_pool2d(out, 2)
        out = out.view(out.size(0), -1)
        out = self.linear(out)
        return out

# show some images
def show(img):
    npimg = img.numpy()[:3,:,:]
    plt.imshow(np.transpose(npimg, (1,2,0)), interpolation='nearest')
# get some random training images

if __name__ == "__main__":

    dirName =r'C:\Users\alan\Desktop\Python\Python36\connect4\Layers'
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

    # Assuming that we are on a CUDA machine, this should print a CUDA device:

    print(device)
    # create dataloader
    max_epochs = 1
    batchSize = 1024#512#256
    learningRate = .00003
    # Parameters
    params = {'batch_size': batchSize,'shuffle': True,'num_workers': 2}
    # Generators
    dataset = C4Dataset()
    start = time.time()
    dataloader = DataLoader(dataset, **params)
    middle = time.time()

    print('create dataloader',middle-start)

    dataiter = iter(dataloader)
    end = time.time()
    print('create data iterator',end-middle)

    images, labels = dataiter.next()
    final = time.time()
    print('get one batch',final-end)
    # show images
    show(torchvision.utils.make_grid(images[:16]))

    #create the weights
    wts =np.array([59, 963, 12406, 148920, 62551, 47281, 55136, 54312, 44465, 31688,
          27912, 37907, 114778, 242800, 394530, 495237, 582174, 163370, 480850,
          201152, 690905, 633937, 721340, 372479, 193375, 84648, 76576, 91087, 130428,
          154184, 157339, 156453, 227696, 1705325, 548155, 44315, 2082],dtype = np.float32)

    maxwt = wts.max()
    weights = wts/maxwt
    weights = torch.from_numpy(weights)
    weights.to(device)

    # create the network

    net = C4Net(5,37)
    net.to(device)
    PATH = r'C:\Users\alan\Desktop\Python\connectX\c4Net37Weights00003.pth'
    net.load_state_dict(torch.load(PATH,map_location=torch.device(device)))

    #create the loss function and optimiser
    criterion = nn.CrossEntropyLoss(weight = weights.to(device) )
    optimizer = optim.Adam(net.parameters(), lr=learningRate)

    #train the network
    import time
    start = time.time()
    for epoch in range(max_epochs):  # loop over the dataset multiple times

        running_loss = 0.0
        for i, data in enumerate(dataloader, 0):
            # get the inputs; data is a list of [inputs, labels]
            inputs, labels = inputs, labels = data[0].to(device), data[1].to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            # print statistics
            running_loss += loss.item()
            if i % 2000 == 1999:    # print every 2000 mini-batches
                print('[%d, %5d] loss: %.3f' %
                      (epoch + 1, i + 1, running_loss / 2000))
                running_loss = 0.0
                torch.save(net.state_dict(),r'C:\Users\alan\Desktop\Python\connectX\tempWeights')
    print('Finished Training')
    # save the weights
    PATH = r'C:\Users\alan\Desktop\Python\connectX\c4Net37Weights00004.pth'

    torch.save(net.state_dict(), PATH)
    end = time.time()
    print('elapsed time',end-start)

PS the machine is a Dell XPS 17 with Intel Core i9-10885H with 8 cores and the GPU is a NVIDIA GeForce RTX 2060 with Max-Q. In this one test it runs 4 times faster than on Google Colab but I do not know what GPU I was allocated.


title: How to merge two torch.utils.data dataloaders with a single operation

body:
I have two dataloaders and I would like to merge them without redefining the datasets, in my case train_dataset and val_dataset.
train_loader = DataLoader(train_dataset, batch_size = 512, drop_last=True,shuffle=True)
val_loader = DataLoader(val_dataset, batch_size = 512, drop_last=False)

Wanted result:
train_loader = train_loader + val_loader


title: Segmentation fault (core dumped) on training the deep neural network /torch/cuda/__init__.py", line 172 in _lazy_init

body:
python3 -q -X faulthandler test.py --model.backbone=resnet18
2021-02-07 20:14:21.198583: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
WARNING:root:Argument whitelist is deprecated. Please use allowlist.
WARNING:root:Argument whitelist is deprecated. Please use allowlist.
WARNING:root:Argument whitelist is deprecated. Please use allowlist.
WARNING:root:Argument whitelist is deprecated. Please use allowlist.
2021-02-07 20:14:22.474523: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-07 20:14:22.474598: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-02-07 20:14:22.474690: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.475028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 3070 computeCapability: 8.6
coreClock: 1.755GHz coreCount: 46 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2021-02-07 20:14:22.475044: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-07 20:14:22.476599: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-02-07 20:14:22.476630: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-02-07 20:14:22.477189: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-07 20:14:22.477333: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-07 20:14:22.478386: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-02-07 20:14:22.478803: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-02-07 20:14:22.478884: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-02-07 20:14:22.478950: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.479304: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.479601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-02-07 20:14:22.479771: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-02-07 20:14:22.480090: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-07 20:14:22.480170: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.480489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 3070 computeCapability: 8.6
coreClock: 1.755GHz coreCount: 46 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2021-02-07 20:14:22.480501: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-07 20:14:22.480512: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-02-07 20:14:22.480521: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-02-07 20:14:22.480529: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-07 20:14:22.480538: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-07 20:14:22.480547: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-02-07 20:14:22.480555: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-02-07 20:14:22.480564: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-02-07 20:14:22.480597: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.480916: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.481206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-02-07 20:14:22.481224: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-07 20:14:22.815420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-02-07 20:14:22.815463: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2021-02-07 20:14:22.815469: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2021-02-07 20:14:22.815656: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.816027: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.816362: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.816685: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5617 MB memory) -> physical GPU (device: 0, name: GeForce RTX 3070, pci bus id: 0000:01:00.0, compute capability: 8.6)
Fatal Python error: Segmentation fault

Current thread 0x00007f52327fa740 (most recent call first):
  File "/home/oem/.local/lib/python3.8/site-packages/torch/cuda/__init__.py", line 172 in _lazy_init
  File "/home/oem/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 610 in convert
  File "/home/oem/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 381 in _apply
  File "/home/oem/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 359 in _apply
  File "/home/oem/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 612 in to
  File "/home/oem/sfuhome/CMPT415/SUR-master/models/model_helpers.py", line 32 in get_model
  File "/home/oem/sfuhome/CMPT415/SUR-master/models/model_helpers.py", line 70 in get_multinet_extractor
  File "/home/oem/sfuhome/CMPT415/SUR-master/models/model_helpers.py", line 61 in get_domain_extractors
  File "test.py", line 39 in main
  File "test.py", line 84 in <module>
Segmentation fault (core dumped)

My guess for this error would be related to CUDA error.
I check my CUDA version
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

But I was installed the most recent CUDA library
/usr/local$ ls
bin  cuda  cuda-11.2  etc  games  include  lib  man  sbin  share  src

But nvidia-smi gives me another version
nvidia-smi
/usr/bin/nvidia-modprobe: unrecognized option: "-s"

ERROR: Invalid commandline, please run `/usr/bin/nvidia-modprobe --help` for usage information.

/usr/bin/nvidia-modprobe: unrecognized option: "-s"

ERROR: Invalid commandline, please run `/usr/bin/nvidia-modprobe --help` for usage information.

Sun Feb  7 20:19:40 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3070    Off  | 00000000:01:00.0  On |                  N/A |
|  0%   51C    P5    23W / 220W |   1628MiB /  7979MiB |     29%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1024      G   /usr/lib/xorg/Xorg                768MiB |
|    0   N/A  N/A      1371      G   /usr/bin/gnome-shell              456MiB |
|    0   N/A  N/A      1857      G   ...gAAAAAAAAA --shared-files      336MiB |
+-----------------------------------------------------------------------------+

And this gives me
cat /usr/local/cuda/version.txt
cat: /usr/local/cuda/version.txt: No such file or directory

Is the problem related to the pytorch or tensorflow because I see this error comes from python3.8/site-package/torch
Anyone can help me? Thank you so much!
And this is the library version
absl-py==0.11.0
apt-clone==0.2.1
apturl==0.5.2
astroid==2.4.2
astunparse==1.6.3
attrs==20.3.0
backcall==0.1.0
bcrypt==3.1.7
bleach==3.1.1
blinker==1.4
Brlapi==0.7.0
cachetools==4.2.1
certifi==2019.11.28
cffi==1.14.4
chardet==3.0.4
Click==7.0
cloudpickle==1.6.0
colorama==0.4.3
command-not-found==0.3
cryptography==2.8
cupshelpers==1.0
cycler==0.10.0
Cython==0.29.21
dataclasses==0.6
dbus-python==1.2.16
decorator==4.4.2
defer==1.0.6
defusedxml==0.6.0
dill==0.3.3
distro==1.4.0
distro-info===0.23ubuntu1
dm-tree==0.1.5
duplicity==0.8.12.0
entrypoints==0.3
fasteners==0.14.1
flatbuffers==1.12
future==0.18.2
gast==0.3.3
gin-config==0.4.0
google-api-core==1.25.1
google-api-python-client==1.12.8
google-auth==1.24.0
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.2
google-cloud-bigquery==2.7.0
google-cloud-core==1.5.0
google-crc32c==1.1.2
google-pasta==0.2.0
google-resumable-media==1.2.0
googleapis-common-protos==1.52.0
grpcio==1.32.0
h5py==2.10.0
html5lib==1.0.1
httplib2==0.18.1
idna==2.8
importlib-metadata==1.5.0
importlib-resources==5.1.0
ipykernel==5.2.0
ipython==7.13.0
ipython-genutils==0.2.0
ipywidgets==6.0.0
isort==5.7.0
jedi==0.15.2
Jinja2==2.10.1
joblib==1.0.0
jsonschema==3.2.0
jupyter-client==6.1.2
jupyter-core==4.6.3
kaggle==1.5.10
Keras-Preprocessing==1.1.2
keyring==18.0.1
kiwisolver==1.3.1
language-selector==0.1
launchpadlib==1.10.13
lazr.restfulclient==0.14.2
lazr.uri==1.0.3
lazy-object-proxy==1.4.3
linecache2==1.0.0
lmdb==1.1.1
lockfile==0.12.2
louis==3.12.0
macaroonbakery==1.3.1
Mako==1.1.0
Markdown==3.3.3
MarkupSafe==1.1.0
matplotlib==3.3.4
mccabe==0.6.1
meta-dataset==0.2.0
mistune==0.8.4
monotonic==1.5
more-itertools==4.2.0
nbconvert==5.6.1
nbformat==5.0.4
netifaces==0.10.4
notebook==6.0.3
numpy==1.19.5
oauth2client==4.1.3
oauthlib==3.1.0
olefile==0.46
opencv-python-headless==4.5.1.48
opt-einsum==3.3.0
PAM==0.4.2
pandas==1.2.1
pandocfilters==1.4.2
paramiko==2.6.0
parso==0.5.2
pbr==5.4.5
pexpect==4.6.0
pickleshare==0.7.5
Pillow==7.0.0
prometheus-client==0.7.1
promise==2.3
prompt-toolkit==2.0.10
proto-plus==1.13.0
protobuf==3.14.0
psutil==5.8.0
py-cpuinfo==7.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycairo==1.16.2
pycocotools==2.0.2
pycparser==2.20
pycups==1.9.73
Pygments==2.3.1
PyGObject==3.36.0
PyICU==2.4.2
PyJWT==1.7.1
pylint==2.6.0
pymacaroons==0.13.0
PyNaCl==1.3.0
pyparsing==2.4.7
pyRFC3339==1.1
pyrsistent==0.15.5
python-apt==2.0.0+ubuntu0.20.4.4
python-dateutil==2.7.3
python-debian===0.1.36ubuntu1
python-slugify==4.0.1
pytz==2019.3
pyxdg==0.26
PyYAML==5.3.1
pyzmq==18.1.1
reportlab==3.5.34
requests==2.22.0
requests-oauthlib==1.3.0
requests-unixsocket==0.2.0
rsa==4.7
scikit-learn==0.24.1
scipy==1.6.0
screen-resolution-extra==0.0.0
SecretStorage==2.3.1
Send2Trash==1.5.0
sentencepiece==0.1.95
seqeval==1.2.2
simplejson==3.16.0
six==1.15.0
sklearn==0.0
systemd-python==234
tabulate==0.8.7
tb-nightly==2.5.0a20210128
tensorboard==2.4.1
tensorboard-plugin-wit==1.8.0
tensorflow==2.4.1
tensorflow-addons==0.12.0
tensorflow-datasets==4.2.0
tensorflow-estimator==2.4.0
tensorflow-gpu==2.4.1
tensorflow-hub==0.11.0
tensorflow-metadata==0.27.0
tensorflow-model-optimization==0.5.0
tensorflow-probability==0.7.0
termcolor==1.1.0
terminado==0.8.2
testpath==0.4.4
testresources==2.0.0
text-unidecode==1.3
tf-estimator-nightly==2.5.0.dev2021012801
tf-models-official==2.4.0
tf-nightly-gpu==2.5.0.dev20210128
tf-slim==1.1.0
threadpoolctl==2.1.0
toml==0.10.2
torch==1.7.1+cu110
torchaudio==0.7.2
torchvision==0.8.2+cu110
tornado==5.1.1
tqdm==4.56.0
traceback2==1.4.0
traitlets==4.3.3
typeguard==2.10.0
typing-extensions==3.7.4.3
ubuntu-advantage-tools==20.3
ubuntu-drivers-common==0.0.0
ufw==0.36
unattended-upgrades==0.1
unittest2==1.1.0
uritemplate==3.0.1
urllib3==1.25.8
usb-creator==0.3.7
wadllib==1.3.3
wcwidth==0.1.8
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==2.0.0
wrapt==1.12.1
xkit==0.0.0
zipp==1.0.0

I try three different versions of Pytorch cuda10.1,cuda11.0. None of them works
And which library is most related to this line?
  File "/home/oem/.local/lib/python3.8/site-packages/torch/cuda/__init__.py", line 172 in _lazy_init


title: Output range for continuous control policy network

body:
I tried to implement the simple vanilla policy gradient (REINFORCE) in a continuous control problem by adapting this pytorch implementation to the continuous case and i stumbled upon the following issue.

Usually, when the action space is discrete, the output of the policy network is bounded in (0,1)^n by the softmax function which gives the probability that the agent would pick a certain action given the state (input to the network). However, when the action space is continuos, for example if we have K action such that each action ak has lower and upper bounds lk anduk, I haven't found a way (empirical or theoretical) to limit the output of the network (wich is usually the means and the stardand deviations of the action probability distribution given the state) using lk and uk.

From the few trials I made, without constraining the output of the policy network, it was very hard, if not impossible, to learn a good policy, but i might be doing something wrong since i am new to reinforcement learning.

My intuition suggests me to limit the means and the standard deviations output of the policy network using, for example, a sigmoid and then scaling them with the absolute difference between lk and uk. I'm not quite sure how to do it properly though, considering also that the sampled action could exceed whatever bound you impose on the distribution parameters when using, for example, a gaussian distribution.

Am i missing something? Are there established ways to limit the output of the policy network for continuous action spaces or there's no need to do that at all?

I am not sure this is the right place for this question, if not i will be glad if you point to me a better place.


title: Summing contiguous non zero tensor values

body:
I am trying to find the summation of contiguous non zero tensor values as shown below

Let’s say, I have a tensor A = [1.3, 0.0, 0.6, 0.7, 0.8]. And I want to

1) sum up the contiguous non-zero values of the tensor to output [1.3, 0.0, 2.1] and then choose the maximum which is 2.1.

2) find the indices as well which were used to sum these values. In this case it will be 2, 3, 4.


title: Unable to optimize function using pytorch

body:
I am trying to write an estimator for a Structural Equation Model. So basically I start with the random parameters for the model B, gamma, phi_diag, psi. And using this I compute the implied covariance matrix sigma. And my optimization function f_ml is computed based on the sigma and the covariance matrix of the data S. Here's my computation code:

device = torch.device('cpu')
dtype = torch.float

B_s = (4, 4)
gamma_s = (4, 1)
phi_s = (1, 1)
psi_s = (4, 4)

# Covariance matrix of data
S = torch.tensor(data.cov().values, dtype=dtype, device=device, requires_grad=False)

# Defining parameters of the model
B = torch.rand(*B_s, dtype=dtype, device=device, requires_grad=True)
B_lower = B.tril(diagonal=-1)

gamma = torch.rand(*gamma_s, dtype=dtype, device=device, requires_grad=True)

phi_diag = torch.rand(phi_s[0], dtype=dtype, device=device, requires_grad=True)
phi = torch.diag(phi_diag)

psi = torch.rand(*psi_s, dtype=dtype, device=device, requires_grad=True)
psi_sym = psi @ psi.t()

B_inv = (torch.eye(*B_s, dtype=dtype, device=device, requires_grad=False) - B_lower).inverse()
sigma_yy = B_inv @ (gamma @ phi @ gamma.t() + psi_sym) @ B_inv.t()
sigma_xy = phi @ gamma.t() @ B_inv.t()
sigma_yx = sigma_xy.t()
sigma_xx = phi

# Computing the covariance matrix from the parameters
sigma = torch.cat((torch.cat((sigma_yy, sigma_yx), 1), torch.cat((sigma_xy, sigma_xx), 1)), 0)

And I am trying to do the optimization as:

optim = torch.optim.Adam([B, gamma, phi_diag, psi], lr=0.01)
for t in range(5000):
    optim.zero_grad()
    f_ml = sigma.logdet() + (S @ sigma.inverse()).trace() - S.logdet() - (4 + 1)
    f_ml.backward(retain_graph=True)
    optim.step()

The problem I am facing is that the values of my parameters aren't updated during the optimization. I tried to debug the problem a bit and what I noticed is that in the first loop of optimization the gradients get calculated but the values of the parameters don't get updated. Here's an example using pdb (breakpoint set right after the for loop):

> <ipython-input-232-c6a6fda6610b>(14)<module>()
-> optim.zero_grad()
(Pdb) B
tensor([[ 6.0198e-01,  8.7188e-01,  5.4234e-01,  6.0800e-01],
        [-4.9971e+03,  9.3324e-01,  8.1482e-01,  8.3517e-01],
        [-1.4002e+04,  2.6706e+04,  2.6412e-01,  4.7804e-01],
        [ 1.1382e+04, -2.1603e+04, -6.0834e+04,  1.2768e-01]],
       requires_grad=True)
(Pdb) c
> <ipython-input-232-c6a6fda6610b>(13)<module>()
-> import pdb; pdb.set_trace()
(Pdb) B.grad
tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00],
        [ 1.6332e+04,  0.0000e+00,  0.0000e+00,  0.0000e+00],
        [ 4.6349e+04, -8.8694e+04,  0.0000e+00,  0.0000e+00],
        [-3.7612e+04,  7.1684e+04,  2.0239e+05,  0.0000e+00]])
(Pdb) B
tensor([[ 6.0198e-01,  8.7188e-01,  5.4234e-01,  6.0800e-01],
        [-4.9971e+03,  9.3324e-01,  8.1482e-01,  8.3517e-01],
        [-1.4002e+04,  2.6706e+04,  2.6412e-01,  4.7804e-01],
        [ 1.1382e+04, -2.1603e+04, -6.0834e+04,  1.2768e-01]],
       requires_grad=True)

I can't figure out what am I doing wrong. Any ideas?


title: Why are there 3 pythons installed on my computer?

body:
When I try to see the version of python installed on my computer, I see the followings:

(base) dhcp76:bin me$ python -V

Python 2.7.16 :: Anaconda, Inc.

(base) dhcp76:bin me$ python2 -V

Python 2.7.16

(base) dhcp76:bin me$ python3 -V

Python 3.7.3

Would this cause any issue? I have also installed anaconda3, but python3 doesn't point to that, and I don't know how to make it point to anaconda3.

So, my questions:

Would having 2 python versions both by brew and anaconda cause problems? If yes, should I remove one of them? (I prefer anaconda)
I installed pytorch using this link: http://deeplizard.com/learn/video/UWlFM0R_x6I, and now when I do import torch in all three version it works!! How is this possible if this links only installs using pip3?

Thanks!


title: Best algorithm for multi agent continuous space path finding using Reinforcement learning

body: I am working on project in which I need to find best optimised path from 1 point to another in continuous space in multi agent scenario.  I am looking for best algorithm which suits this problem using Reinforcement learning. I have tried "Multi-agent actor-critic for mixed cooperative-competitive environment" but it does not seems to reach goals in 10000 epesidoes.  How can I improve this algorithm or is there any other algorithm that can help me with this.


I was following this method
(https://discuss.pytorch.org/t/dynamic-parameter-declaration-in-forward-function/427) to dynamically assign parameters in forward function.

However, my parameter is not just one single weight tensor but it is nn.Sequential.

When I implement below:

class MyModule(nn.Module):
    def __init__(self):
        # you need to register the parameter names earlier
        self.register_parameter('W_di', None)

    def forward(self, input):
        if self.W_di is None:
            self.W_di = nn.Sequential(
                nn.Linear(mL_n * 2, 1024),
                nn.ReLU(),
                nn.Linear(1024, self.hS)).to(device)

I get the following error.

 TypeError: cannot assign 'torch.nn.modules.container.Sequential' as parameter 'W_di' (torch.nn.Parameter or None expected)

Is there any way that I can register nn.Sequential as a whole param? Thanks!


title: Why should the  (huggingface) Transformers library be installed on a virtual environment?

body:
In the huggingface github it is written:

You should install  Transformers in a virtual environment. If you're
unfamiliar with Python virtual environments, check out the user guide.
Create a virtual environment with the version of Python you're going
to use and activate it.
Now, if you want to use  Transformers, you can install it with pip.
If you'd like to play with the examples, you must install it from
source.

Why should it be installed in a virtual python environment?
What are the advantages of doing that rather than installing it on python as is?


title: Visualizing models, data, and training with tensorboard in pytorch

body:
Following this Tutorial
to visualize images in tensorboard using torch.utils.tensorboard got error

from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter('runs/fashion_mnist_experiment_1')
writer.add_embedding(features,
                metadata=class_labels,
                label_img=images.unsqueeze(1))
writer.close()

Error:

        AttributeError: module 'tensorflow_core._api.v2.io.gfile' has no attribute 'get_filesystem'


title: Approximate the q-function with NN in the FrozenLake exercise

body:
import numpy as np
import gym
import random
import time
from IPython.display import clear_output

env = gym.make("FrozenLake-v0")

action_space_size = env.action_space.n
state_space_size = env.observation_space.n

q_table = np.zeros((state_space_size, action_space_size))

num_episodes = 10000
max_steps_per_episode = 100

learning_rate = 0.1
discount_rate = 0.99

exploration_rate = 1
max_exploration_rate = 1
min_exploration_rate = 0.01
exploration_decay_rate = 0.01

reward_all_episodes = []

#Q-learning algorithm
for episode in range(num_episodes):
    state = env.reset()

    done = False
    reward_current_episode = 0

    for step in range(max_steps_per_episode):

        exploration_rate_threshold = random.uniform(0, 1)
        if exploration_rate_threshold > exploration_rate: #Exploit
            action = np.argmax(q_table[state, :])
        else:
            action = env.action_space.sample() #Explore

        new_state, reward, done, info = env.step(action)

        q_table[state, action] = (1 - learning_rate) * q_table[state, action] + \
            learning_rate * (reward + discount_rate * np.max(q_table[new_state]))

        state = new_state

        reward_current_episode += reward

        if done == True:
            break

    exploration_rate = min_exploration_rate + \
        (max_exploration_rate - min_exploration_rate) * np.exp(-exploration_decay_rate * episode)

    reward_all_episodes.append(reward_current_episode)

reward_per_thousand_episodes = np.split(np.array(reward_all_episodes), num_episodes/1000)
count = 1000
print("Average Reward per thousand episode \n")
for r in reward_per_thousand_episodes:
    print(count, ":", str(sum(r/1000)))
    count += 1000

print("\n ***************Q-table****************\n\n")
print(q_table)

I am new in AI and I need a bit of help. I have completed the FrozenLake exercise with MVP / Q-learning. Someone told me I can approximate the q-function using a deep neural network. It explains it is called deep Q-learning. How can I improve that code using deep Q-learning and pytorch? In other words, how can I approximate the q-function in using NN here?


title: Applying a 2D convolution kernel to each channel in Pytorch?

body:
I have a single 2D kernel of size [3,3], and a Tensor of size [B, 64, H, W].
My question is, how can I apply the same 2D kernel to each input channel? Should I reshape/repeat the kernel? I tried to repeat my kernel as follows:
kernel = kernel.repeat((B, 64, 1, 1))

But when I apply it the tensor size changes to [1, 64, H, 1].


title: Accuracy per epoch in PyTorch

body:
I have made a chatbot using pytorch and would like to display accuracy on every epoch. I am not quite understanding how to do that. I can display loss but cant figure out how to display my accuracy
Here is my code :-
from nltk_utils import tokenize, stem, bag_of_words
import json
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
from model import NeuralNet
from torch.autograd import Variable

all_words=[]
tags=[]
xy=[]
questionsP1=[]
questionsP2=[]
questionsP3=[]
questionsP4=[]
questionTag={}

with open('new.json', encoding="utf8") as file:
        data = json.load(file)

for intent in data["intents"]:
    for proficiency in intent["proficiency"]:
        for questions in proficiency["questions"]:
            for responses in questions["responses"]:
                wrds = tokenize(responses)
                all_words.extend(wrds)
                xy.append((wrds, questions["tag"]))

            if questions["tag"] in tags:
                print(questions["tag"])

            if questions["tag"] not in tags:
                tags.append(questions["tag"])

            if proficiency["level"] == "P1":
                questionsP1.append(questions["question"])
                questionTag[questions["question"]]=questions["tag"]

            if proficiency["level"] == "P2":
                questionsP2.append(questions["question"])
                questionTag[questions["question"]]=questions["tag"]

            if proficiency["level"] == "P3":
                questionsP3.append(questions["question"])
                questionTag[questions["question"]]=questions["tag"]

            if proficiency["level"] == "P4":
                questionsP4.append(questions["question"])
                questionTag[questions["question"]]=questions["tag"]

ignore_words = ['?', '!', '.', ',']
all_words = [stem(x) for x in all_words if x not in ignore_words]
all_words = sorted(set(all_words))
tags = sorted(set(tags))

X_train = []
y_train = []

for tokenized_response, tag in xy:
    bag = bag_of_words(tokenized_response, all_words)
    print(bag)
    X_train.append( bag )

    label = tags.index( tag )
    y_train.append( label )
print(y_train)

X_train = np.array( X_train )
y_train = np.array( y_train )

class ChatDataset(Dataset):
    def __init__(self):
        self.n_samples = len(X_train)
        self.x_data = X_train
        self.y_data = y_train

    def __getitem__(self, index):
        return self.x_data[index], self.y_data[index]

    def __len__(self):
        return self.n_samples

#HyperParameters
batch_size = 8
hidden_size = 8
output_size = len(tags)
input_size = len(X_train[0])
learning_rate = 0.001
num_epochs = 994

dataset = ChatDataset()
train_loader = DataLoader(dataset = dataset, batch_size=batch_size, shuffle = True, num_workers = 2)

device = 'cpu'
model = NeuralNet(input_size, hidden_size, output_size).to(device)

#loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr = learning_rate)

for epoch in range( num_epochs ):
    for (words, labels) in train_loader:
        words = words.to(device)
        labels = labels.to(device)

        #Forward
        outputs = model(words)
        loss = criterion(outputs, labels)

        #backward and optimizer step
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        print(f'epoch {epoch + 1}/ {num_epochs}, loss={loss.item(): .4f}')

print(f'final loss, loss={loss.item(): .4f}')

data = {
    "model_state": model.state_dict(),
    "input_size": input_size,
    "output_size": output_size,
    "hidden_size": hidden_size,
    "all_words": all_words,
    "tags": tags,
}

FILE = "data.pth"
torch.save(data, FILE)

with open('new.json', 'r') as f:
    intents = json.load(f)

bot_name = "Sam"
while True:
    sentence = input("You: ")
    if sentence == 'quit':
        break
    sentence = tokenize(sentence)
    X = bag_of_words(sentence, all_words)
    X = X.reshape( 1, X.shape[0])
    X = torch.from_numpy( X )

    output = model( X )
    _, predicted = torch.max(output, dim=1)
    tag = tags[predicted.item()]
    print(tag)

    probs = torch.softmax(output, dim=1)
    probs = probs[0][predicted.item()]

    print( probs.item() )

    if probs.item() > 0.75:
        for intent in intents["intents"]:
            for proficiency in intent["proficiency"]:
                for questions in proficiency["questions"]:
                    if questions["tag"] == tag:
                        print(f'{bot_name}: {questions["question"]}')
    else:
        print(f'{bot_name}: Probability Too Low')

print(f'Training Complete. File saved to {FILE}')

My chatbot is working inverselt... i am trying to map the answer to the right question.
Any help would be appreciated.


title: Masked Matrix multiplication

body:
I have a A = N x M matrix and another array B = N x P x M where P is typically 9 or 15. For each vector a from A, it has to be multiplied with each pi from B of the same row, to get an output of dimensions N x P.
I am using numpy and Python and would be performing this operation on a GPU.
For a tiny example, let N=4, M=5, P=3.
Let A be:
array([[0.18503431, 0.2628188 , 0.26343728, 0.8356702 , 0.47581551],
       [0.70827725, 0.04006919, 0.58975722, 0.90874113, 0.43946412],
       [0.40669507, 0.63328008, 0.95832881, 0.59041436, 0.63578578],
       [0.12129919, 0.74470057, 0.62271405, 0.97760796, 0.6499647 ]])

Let B be:
array([[[4.29031165e-01, 6.17324572e-01, 6.54726975e-02, 1.72218768e-02, 3.53970827e-01],
        [3.38821841e-01, 3.80128792e-01, 7.70995505e-01, 7.38437494e-03, 5.87395036e-02],
        [4.75661932e-01, 3.75617802e-01, 1.28564731e-01, 3.66302247e-01, 6.70953890e-01]],

       [[8.96228996e-02, 1.67135584e-02, 4.56921778e-01, 8.25731354e-01, 7.66242539e-01],
        [5.16651815e-01, 4.27179773e-01, 9.34673912e-01, 2.04687170e-01, 7.68417953e-01],
        [5.90980849e-01, 5.03013376e-01, 8.41765736e-02, 8.08221224e-01, 7.76765422e-01]],

       [[3.25802668e-01, 8.58148960e-01, 9.47505735e-01, 1.01405305e-01, 8.34114717e-01],
        [1.65308159e-01, 9.74572631e-01, 2.69886016e-01, 7.44036253e-02, 4.73350521e-01],
        [8.59030672e-01, 3.96972621e-01, 7.34687493e-01, 2.84647032e-02, 7.19723378e-01]],

       [[1.35751242e-01, 1.74882898e-01, 5.48875709e-01, 7.33443675e-01, 4.05282650e-01],
        [8.41298770e-01, 6.24323279e-01, 5.83482185e-01, 4.28514313e-01, 1.96797205e-01],
        [7.93345700e-04, 3.01441721e-01, 7.59451146e-01, 9.09102382e-01, 7.11518948e-01]]])

This is how I want my output to be:
[[np.dot(a[0], b[0][0]), np.dot(a[0], b[0][1]), np.dot(a[0], b[0][2])],
[np.dot(a[1], b[1][0]), np.dot(a[1], b[1][1]), np.dot(a[1], b[1][2])],
[np.dot(a[2], b[2][0]), np.dot(a[2], b[2][1]), np.dot(a[2], b[2][2])],
[np.dot(a[3], b[3][0]), np.dot(a[3], b[3][1]), np.dot(a[3], b[3][2])]]

Doing this manually gives:
[[0.44169455751462816, 0.3998276862221848, 0.845960080871557],
[1.4207326179275017, 1.4579799277670968, 1.564201768913105],
[2.174162453912622, 1.287925491552765, 1.779226448174152],
[1.4689343122491012, 1.4771555510001255, 2.0487088726424365]]

Since I want to do this on a GPU, this obviously calls for converting my problem into matrix multiplication (this is true if I dont use a GPU as well for that matter).
But I do not exactly know how to convert it to that.
One idea that I had was to reshape B to be Q x M where Q=NxP. And then perform some sort of sparse multiplication, where for every row i of a boolean sparse matrix, I turn on (0:P) + P*ith elements. (Drawing it out makes sense), however I certainly feel that there is a much more elegant way to do this as creating sparse matrices and performing the operations might take time, and that the sparsity of my matrix is not random at all.
How can I solve this elegantly.
Note that I cannot do some operations such as broadcasting/repeating my A matrix P times and performing the huge matrix multiplication and picking out relevant values, since typically N and M will be quite huge (2000ish and 256 respectively), but P will be quite small, hence doing a global matrix multiplication for all vectors means I will be doing >95% unnecessary computations!.


title: CNN for variable sized images in pytorch

body:
I want to make a CNN model in pytorch which can be fed images of different sizes. I am trying to use 2d convolution layer, which takes 4D input shape (pytorch's Conv2d expects its 2D inputs to actually have 4 dimensions).
However, I'm not sure how to set up the input layer that can adjust all the variable sized images into fixed number of feature maps to pass over to remaining layers.
For example, the shape of the input for colored images is [4, 3, 32, 32], which corresponds to batch size, number of channel(RGB), width, and height. If images are grayscale, then it will have [4, 1, 32, 32], which will produce an error when the shape is not what the layer expected.
Error message is "RuntimeError: Given groups=1, weight of size [6, 3, 5, 5], expected input[4, 1, 32, 32] to have 3 channels, but got 1 channels instead"
The architecture of my current CNN is like below.
def __init__(self, num_out, kernel_size, num_input_filters):
        super().__init__()
        self.num_input_filters = num_input_filters
        self.num_out = num_out
        self.kernel_size = kernel_size

        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)

def forward(self, inp):
        inp = self.pool(F.relu(self.conv1(inp)))
        return inp

I have referenced similar questions and Fully convolutional networks (FCN) have no limitations on the input size at all, which could be the solution. And pytorch provides ConvTranspose2d() for FCN, but its parameters still seem to require fixed input size.
Are there any methods that can solve this problem?


title: Not able to install pytorch 1.2 with CUDA 9.2

body:
I am trying to install the lastest version of pytorch with cuda enabled that is version pytorch v1.2 and cuda v9.2 which corresponds with my nvcc version (9.2) on a python 2.7 environment created using conda. (venv)

The command that I used was: conda install pytorch torchvision cudatoolkit=9.2 -c pytorch
which I was given as per my system configurations on the official pytorch website.

But then when I check on the python command line, my pytorch version still shows v0.4.1 and cuda v9.0.


title: How can i do a numerical integration of the following expression?

body:
I am currently trying to evaluate the following expression in Python

The expression comes from a t-distribution and v is a scalar indicating degrees of freedom, mu and sigma is a vector and a matrix respectively, defining the mean and the standard deviation. How would I go on getting a numerical integration of the expression? I have tried the following using PyTorch and Scipy quad function:

def integrand(x, mu, std,v,p):
    return torch.sum((1+(x-mu)**2*std**(-2)/v)**((-v+p)/2)*torch.log(1+x**2/v))

integrate.quad(integrand, -np.inf,np.inf, args=(mu,std,v,p))[0]

However, the results are quite disappointing. The thing that bugs me are the vectors, I don't think I know how to handle such a situation, also the limits are not specified. Help would very much be appreciated, thanks a lot! Let me know if more information is needed.


title: How can I set maximum n elements in one row in pytorch tensor to 1 and оther to zero?

body:
For example, if I have tensor

0.8, 0.1, 0.9, 0.2
0.7, 0.1, 0.4, 0.6

and n = 2, I want to get

1, 0, 1, 0
1, 0, 0, 1

Or maybe it's better to do it numpy, but the question is the same.


title: Pytorch RuntimeError: expected scalar type Long but found Int

body:
I try launch test, but get this error:

*Traceback (most recent call last):
  File "<ipython-input-59-5947c660ace7>", line 26, in test_ClassNLLCriterionUnstable
    torch_layer_output_var = torch_layer(torch.log(layer_input_var),
  File "C:\Program Files\Python38\lib\site-packages\torch\nn\modules\module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Program Files\Python38\lib\site-packages\torch\nn\modules\loss.py", line 205, in forward
    return F.nll_loss(input, target, weight=self.weight, ignore_index=self.ignore_index, reduction=self.reduction)
  File "C:\Program Files\Python38\lib\site-packages\torch\nn\functional.py", line 2115, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: expected scalar type Long but found Int*

Test code:

class TestLayers(unittest.TestCase):
     def test_ClassNLLCriterionUnstable(self):
        np.random.seed(42)
        torch.manual_seed(42)

        batch_size, n_in = 2, 4
        for _ in range(100):
            # layers initialization
            torch_layer = torch.nn.NLLLoss()
            custom_layer = ClassNLLCriterionUnstable()

            layer_input = np.random.uniform(0, 1, (batch_size, n_in)).astype(np.float32)
            layer_input /= layer_input.sum(axis=-1, keepdims=True)
            layer_input = layer_input.clip(custom_layer.EPS, 1. - custom_layer.EPS)  # unifies input
            target_labels = np.random.choice(n_in, batch_size)
            target = np.zeros((batch_size, n_in), np.float32)
            target[np.arange(batch_size), target_labels] = 1  # one-hot encoding

            # 1. check layer output
            custom_layer_output = custom_layer.updateOutput(layer_input, target)
            layer_input_var = Variable(torch.from_numpy(layer_input), requires_grad=True)
            torch_layer_output_var = torch_layer(torch.log(layer_input_var),
                                                 Variable(torch.from_numpy(target_labels), requires_grad=False))
            self.assertTrue(np.allclose(torch_layer_output_var.data.numpy(), custom_layer_output, atol=1e-6))

ClassNLLCriterionUnstable code:

class ClassNLLCriterionUnstable(Criterion):
    EPS = 1e-15
    def __init__(self):
        a = super(ClassNLLCriterionUnstable, self)
        super(ClassNLLCriterionUnstable, self).__init__()

    def updateOutput(self, input, target):
        # Use this trick to avoid numerical errors
        input_clamp = np.clip(input, self.EPS, 1 - self.EPS)
         # Your code goes here. ################################################
        return self.output

    def updateGradInput(self, input, target):

        # Use this trick to avoid numerical errors
        input_clamp = np.clip(input, self.EPS, 1 - self.EPS)

        # Your code goes here. ################################################
        return self.gradInput

    def __repr__(self):
        return "ClassNLLCriterionUnstable"

I don’t know how to fix it. I am new to machine learning, I wanted to implement the negative loglikehood criterion and compare it with the pytorch.


title: Creating pytorch Tensors from `torch` or `numpy` vectors

body:
I am trying to create some testing torch tensors by assembling the dimensions from vectors calculated via basic math functions. As a precursor: assembling Tensors from primitive python arrays does work:
import torch
import numpy as np
torch.Tensor([[1.0, 0.8, 0.6],[0.0, 0.5, 0.75]])
>> tensor([[1.0000, 0.8000, 0.6000],
            [0.0000, 0.5000, 0.7500]])

In addition we can assemble Tensors from numpy arrays
https://pytorch.org/docs/stable/tensors.html:
torch.tensor(np.array([[1, 2, 3], [4, 5, 6]]))
tensor([[ 1,  2,  3],
        [ 4,  5,  6]])

However assembling from calculated vectors is eluding me. Here are some of the attempts made:
X  = torch.arange(0,6.28)
x = X

torch.Tensor([[torch.cos(X),torch.tan(x)]])

torch.Tensor([torch.cos(X),torch.tan(x)])

torch.Tensor([np.cos(X),np.tan(x)])

torch.Tensor([[np.cos(X),np.tan(x)]])

torch.Tensor(np.array([np.cos(X),np.tan(x)]))

All of the above have the following error:

ValueError: only one element tensors can be converted to Python scalars

What is the correct syntax?
Update A comment requested showing x / X.  They're actually set to the same (I changed mind mid-course which to use)
In [56]: x == X
Out[56]: tensor([True, True, True, True, True, True, True])

In [51]:  x
Out[51]: tensor([0., 1., 2., 3., 4., 5., 6.])

In [52]: X
Out[52]: tensor([0., 1., 2., 3., 4., 5., 6.])


title: Training Loss When Resuming From a Checkpoint Explodes

body:
I am trying to implement a function in my algorithm which allows me to resume training from a checkpoint. The problem is that when I resume training, my loss explodes by many orders of magnitude, from the order to 0.001 to 1000. I suspect that the problem may be that when training is resumed, the learning rate is not being set properly.
Here is my training function:
def train_gray(epoch, data_loader, device, model, criterion, optimizer, i, path):
    train_loss = 0.0
    for data in data_loader:
        img, _ = data
        img = img.to(device)
        stand_dev = 0.0392

        noisy_img = add_noise(img, stand_dev, device)

        output = model(noisy_img, stand_dev)
        output = output[:,0:1,:,:]
        loss = criterion(output, img)

        optimizer.zero_grad()

        loss.backward()
        optimizer.step()
        train_loss += loss.item()*img.size(0)

    train_loss = train_loss/len(data_loader)

    print('Epoch: {} Complete \tTraining Loss: {:.6f}'.format(
        epoch,
        train_loss
        ))
    return train_loss

And here is my main function that initialises my variables, loads a checkpoint, calls my training function, and saves a checkpoint after an epoch of training:

def main():
    now = datetime.now()
    current_time = now.strftime("%H_%M_%S")
    path = "/home/bledc/my_remote_folder/denoiser/models/{}_sigma_10_session2".format(current_time)
    os.mkdir(path)

    width = 256
    # height = 256
    num_epochs = 25
    batch_size = 4
    learning_rate = 0.0001

    data_loader = load_dataset(batch_size, width)

    model = UNetWithResnet50Encoder().to(device)
    criterion = nn.MSELoss()
    optimizer = torch.optim.Adam(
        model.parameters(), lr=learning_rate, weight_decay=1e-5)

    ############################################################################################
    # UNCOMMENT CODE BELOW TO RESUME TRAINING FROM A MODEL
    model_path = "/home/bledc/my_remote_folder/denoiser/models/resnet_sigma_10/model_epoch_10.pt"
    save_point = torch.load(model_path)
    model.load_state_dict(save_point['model_state_dict'])
    optimizer.load_state_dict(save_point['optimizer_state_dict'])
    epoch = save_point['epoch']
    train_loss = save_point['train_loss']
    model.train()
    ############################################################################################

    for i in range(epoch, num_epochs+1):
        train_loss = train_gray(i, data_loader, device, model, criterion, optimizer, i, path)
        checkpoint(i, train_loss, model, optimizer, path)

    print("end")

Lastly, here is my function to save checkpoints:
def checkpoint(epoch, train_loss, model, optimizer, path):
    torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'train_loss': train_loss
            }, path+"/model_epoch_{}.pt".format(epoch))
    print("Epoch saved")

If my problem is that I am not saving my learning rate, how would I do this?
Any help would be greatly appreciated,
Clement
Update: I'm fairly certain that the problem lies in my pretrained model. I am saving the optimiser every epoch but the optimiser only holds information for the trainable layers. I hope to solve this soon and post a more thorough answer when I figure out who to save and load the entire model.


title: How to save information about the result of instance segmentation by YOLACT?

body:
Is there any way to save the detected categories, their number, MASK area, etc. to a TXT file or CSV file when performing instance segmentation using YOLACT?
I’m using YOLACT (https://github.com/dbolya/yolact) to challenge instance segmentation.
I was able to use eval.py to do an instance segmentation of my own data and save that image or video.
However, what I really need is the class names and their numbers detected and classified by YOLACT's AI, and the area of MASK.
If we can output this information to a txt file or csv file, we can use YOLACT even more advanced.
If I can achieve that by adding an option in eval.py or modifying the code, please teach me.
Thank you.


title: Validation loss not moving with MLP in Regression

body:
Given input features as such, just raw numbers:

tensor([0.2153, 0.2190, 0.0685, 0.2127, 0.2145, 0.1260, 0.1480, 0.1483, 0.1489,
        0.1400, 0.1906, 0.1876, 0.1900, 0.1925, 0.0149, 0.1857, 0.1871, 0.2715,
        0.1887, 0.1804, 0.1656, 0.1665, 0.1137, 0.1668, 0.1168, 0.0278, 0.1170,
        0.1189, 0.1163, 0.2337, 0.2319, 0.2315, 0.2325, 0.0519, 0.0594, 0.0603,
        0.0586, 0.0067, 0.0624, 0.2691, 0.0617, 0.2790, 0.2805, 0.2848, 0.2454,
        0.1268, 0.2483, 0.2454, 0.2475], device='cuda:0')

And the expected output is a single real number output, e.g.

tensor(-34.8500, device='cuda:0')

Full code on https://www.kaggle.com/alvations/pytorch-mlp-regression

I've tried creating a simple 2 layer network with:

class MLP(nn.Module):
    def __init__(self, input_size, output_size, hidden_size):
        super(MLP, self).__init__()
        self.linear = nn.Linear(input_size, hidden_size)
        self.classifier = nn.Linear(hidden_size, output_size)

    def forward(self, inputs, hidden=None, dropout=0.5):
        inputs = F.dropout(inputs, dropout) # Drop-in.
        # First Layer.
        output = F.relu(self.linear(inputs))

        # Matrix manipulation magic.
        batch_size, sequence_len, hidden_size = output.shape
        # Technically, linear layer takes a 2-D matrix as input, so more manipulation...
        output = output.contiguous().view(batch_size * sequence_len, hidden_size)
        # Apply dropout.
        output = F.dropout(output, dropout)

        # Put it through the classifier
        # And reshape it to [batch_size x sequence_len x vocab_size]
        output = self.classifier(output).view(batch_size, sequence_len, -1)

        return output

And training as such:

# Training routine.
def train(num_epochs, dataloader, valid_dataset, model, criterion, optimizer):
    losses = []
    valid_losses = []
    learning_rates = []
    plt.ion()
    x_valid, y_valid = valid_dataset
    for _e in range(num_epochs):
        for batch in tqdm(dataloader):
            # Zero gradient.
            optimizer.zero_grad()
            #print(batch)
            this_x = torch.tensor(batch['x'].view(len(batch['x']), 1, -1)).to(device)
            this_y = torch.tensor(batch['y'].view(len(batch['y']), 1, 1)).to(device)

            # Feed forward.
            output = model(this_x)

            prediction, _ = torch.max(output, dim=1)
            loss = criterion(prediction, this_y.view(len(batch['y']), -1))
            loss.backward()
            optimizer.step()
            losses.append(torch.sqrt(loss.float()).data)

            with torch.no_grad():
                # Zero gradient.
                optimizer.zero_grad()
                output = model(x_valid.view(len(x_valid), 1, -1))
                prediction, _ = torch.max(output, dim=1)
                loss = criterion(prediction, y_valid.view(len(y_valid), -1))
                valid_losses.append(torch.sqrt(loss.float()).data)

            clear_output(wait=True)
            plt.plot(losses, label='Train')
            plt.plot(valid_losses, label='Valid')
            plt.legend()
            plt.pause(0.05)

Tuning several hyperparameters, it looks like the model doesn't train well, the validation loss doesn't move at all e.g.

hyperparams = Hyperparams(input_size=train_dataset.x.shape[1],
                          output_size=1,
                          hidden_size=150,
                          loss_func=nn.MSELoss,
                          learning_rate=1e-8,
                          optimizer=optim.Adam,
                          batch_size=500)

And it's loss curve:

Any idea what's wrong with the network?

Am I training the regression model with the wrong loss? Or I've just not yet found the right hyperparameters?

Or am I validating the model wrongly?


title: How to get a 5-Dimensional output after torch.nn.Conv2d layer in PyTorch?

body:
I am working on a project based on the OpenPose research paper that I read two weeks ago. In that, the model is supposed to give a 5-dimensional output. For example, torch.nn.conv2d() gives a 4-D output of the following shape: (Batch_size, n_channels, input_width, input_height). What I need is an output of the following shape: (Batch_size, n_channels, input_width, input_height, 2). Here 2 is a fixed number not subject to any changes.
The 2 is there because each entry is a 2-dimensional vector hence for each channel in every pixel position, there are 2 values hence, the added dimension.

What will be the best way to do this?
I thought about having 2 seperate branches for each of the vector values but the network is very deep and I would like to be as Computationally efficient as possible.


title: Unable to reproduce result obtained from hyperparameter tuning using hyperopt

body:
I have built a Pytorch model and performed hyperparameters tuning using library Hyperopt. The result obtained is not reproducible despite I have already call the below seeding function at the beginning of each run:

util.py

def seed_everything(seed=42):
    random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    np.random.seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

Trainer.py

    def train(self, params):
        util.seed_everything()
        #wrote my training code here

Upon futher inspection, I have found that the first result of hyperopt tuning is always reproducible, but the subsequent runs are not. This is unexpected for me as I have already called seed_everthing() at the beginning of the train function.

In addition, if I run the training as below:

    for i in range(2):
        print ("in iteration ", i)
        trainer = Trainer(**configs)
        trainer.train(params)

The result of iteration 1 and 2 is different to each other, but they are always the same(i.e. iteration 1 always gives train_loss of 1.31714 while iteration 2 always gives 4.31235)

I am hoping that iteration 1 and iteration 2 would gives the same result since it should be reproducible.


title: DCGAN where the image size is different for discriminator and generator

body:
I'm working with DCGAN, where my discriminator will have real images of size 128 x 128 feeded in, while the input to the generator network is the 64x64 image (lower resolution) rather than a multinomial gaussian distribution.
When I'm trying to run the training loop, I'm getting dimension errors, and I believe my discriminator and generator dimension architecture might be wrong. However, I can't grasp where the error lies.
ngf = 64, ndf = 128, nz = 745
Here is the code for the generator and discriminator:

class G(nn.Module): # We introduce a class to define the generator.

    def __init__(self): # We introduce the __init__() function that will define the architecture of the generator.
        super(G, self).__init__() # We inherit from the nn.Module tools.
        self.main = nn.Sequential( # We create a meta module of a neural network that will contain a sequence of modules (convolutions, full connections, etc.).
            # input is Z, going into a convolution
            nn.ConvTranspose2d(in_channels=nz, out_channels=ngf*8, kernel_size=4, stride=1, padding=0, bias = False), # first index is HR_data.shape[0] aka nz, second index is ngf * 8, where ngf relates to the size of the feature maps, i.e. 64
            nn.BatchNorm2d(ngf*8), # We normalize all the features along the dimension of the batch.
            nn.ReLU(True), # We apply a ReLU rectification to break the linearity.
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf*8, ngf*4, 4, 2, 1, bias = False), # We add another inversed convolution.
            nn.BatchNorm2d(ngf*4), # We normalize again.
            nn.ReLU(True), # We apply another ReLU.
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d(ngf*4, ngf*2, 4, 2, 1, bias = False), # We add another inversed convolution.
            nn.BatchNorm2d(ngf*2), # We normalize again.
            nn.ReLU(True), # We apply another ReLU.
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d(ngf*2, ngf, 4, 2, 1, bias = False), # We add another inversed convolution.
            nn.BatchNorm2d(ngf), # We normalize again.
            nn.ReLU(True), # We apply another ReLU.
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d(ngf, 3, 4, 2, 1, bias = False), # We add another inversed convolution.
            nn.Tanh() # We apply a Tanh rectification to break the linearity and stay between -1 and +1.
            # state size. 3 x 64 x 64
        )

    def forward(self, input): # We define the forward function that takes as argument an input that will be fed to the neural network, and that will return the output containing the generated images.
        output = self.main(input) # We forward propagate the signal through the whole neural network of the generator defined by self.main.
        return output # We return the output containing the generated images.

# Creating the generator
netG = G() # We create the generator object.
netG.apply(weights_init) # We initialize all the weights of its neural network.

# Defining the discriminator

class D(nn.Module): # We introduce a class to define the discriminator.

    def __init__(self): # We introduce the __init__() function that will define the architecture of the discriminator.
        super(D, self).__init__() # We inherit from the nn.Module tools.
        self.main = nn.Sequential( # We create a meta module of a neural network that will contain a sequence of modules (convolutions, full connections, etc.).
             # input is (nc) x 128 x 128
            nn.Conv2d(3, ndf, 4, 2, 1, bias = False),
            nn.LeakyReLU(0.2, inplace = True), # We apply a LeakyReLU.
            # state size. (ndf) x 64 x 64
            nn.Conv2d(ndf, ndf*2, 4, 2, 1, bias = False), # We add another convolution.
            nn.BatchNorm2d(ndf*2), # ndf * 2
            nn.LeakyReLU(0.2, inplace = True), # We apply another LeakyReLU.
            # state size. (ndf*2) x 32 x 32
            nn.Conv2d(ndf*2, ndf*4, 4, 2, 1, bias = False), # We add another convolution.
            nn.BatchNorm2d(ndf*4), # We normalize again.
            nn.LeakyReLU(0.2, inplace = True), # We apply another LeakyReLU.
            # state size. (ndf*4) x 16 x 16
            nn.Conv2d(ndf*4, ndf*8, 4, 2, 1, bias = False), # We add another convolution.
            nn.BatchNorm2d(ndf*8), # We normalize again.
            nn.LeakyReLU(0.2, inplace = True), # We apply another LeakyReLU.
            # state size. (ndf*8) x 8 x 8
            nn.Conv2d(ndf*8, 1, 4, 1, 0, bias = False), # We add another convolution.
            nn.Sigmoid() # We apply a Sigmoid rectification to break the linearity and stay between 0 and 1.
        )

    def forward(self, input): # We define the forward function that takes as argument an input that will be fed to the neural network, and that will return the output which will be a value between 0 and 1.
        output = self.main(input) # We forward propagate the signal through the whole neural network of the discriminator defined by self.main.
        return output.view(-1) # We return the output which will be a value between 0 and 1.

# Creating the discriminator
netD = D() # We create the discriminator object.
netD.apply(weights_init) # We initialize all the weights of its neural network.


title: argmax_cuda" not implemented for 'Bool

body:
I am facing this error in google collab. I tried other datatypes like bool tensor but didn't work, please help

Code

 def _mask(prev_generated_seq):
        prev_mask = torch.eq(prev_generated_seq, 1)
        lengths = torch.argmax(prev_mask,dim=1)
        #test = torch.max(prev_mask,dim=1)
        #lengths = torch.FloatTensor(test)
        max_len = prev_generated_seq.size(1)
        mask = []
        for i in range(prev_generated_seq.size(0)):
            if lengths[i] == 0:
                mask_line = [0] * max_len
            else:
                mask_line = [0] * lengths[i].item()
                mask_line.extend([1] * (max_len - lengths[i].item()))
            mask.append(mask_line)
        mask = torch.ByteTensor(mask)
        if args.cuda:
            mask = mask.cuda()
        return prev_generated_seq.data.masked_fill_(mask, 0)

Error

File "main.py", line 179, in <module>
    train_epoches(abstracts, model, config.epochs, teacher_forcing_ratio=1)
  File "main.py", line 155, in train_epoches
    target_variables, model, teacher_forcing_ratio)
  File "main.py", line 139, in train_batch
    prev_generated_seq = _mask(prev_generated_seq)
  File "main.py", line 101, in _mask
    lengths = torch.argmax(prev_mask,dim=1)
RuntimeError: "argmax_cuda" not implemented for 'Bool'


title: In pytorch, how can I sum some elements, and get a tensor of smaller shape?

body:
Specifically I have a tensor of dimension 298x160x160 (faces in 298 frames), I need to sum every 4x4 element in last two dimesnion so that I can get a 298x40x40 tensor.

How can I achieve that?


title: Pytorch: Weighted Covariance

body:
I am trying to implement a PyTorch covariance matrix operator. However, I notice the results are not the same between the Numpy implementation and my attempt, yet I do not understand why.

I define the Bessel-corrected weighted covariance matrix as:

I define the weighted mean as:

I compare the NumPy method and my method as follows:

import numpy as np
import torch

torch.set_printoptions(precision=8)

x = np.random.randn(1000, 3)*1000
w = np.abs(np.random.randn(1000))*1000

x_torch = torch.DoubleTensor(x)
w_torch = torch.DoubleTensor(w)

#calculate weighted means
m_w = torch.sum(x_torch.T*w_torch, axis=1)/torch.sum(w_torch)
m_w_np = np.average(x, axis=0, weights=w)

#calculate weighted covariance matrix
Q = (x_torch-m_w).T
cov_w = (1.0 / (torch.sum(w_torch) - 1))*(w_torch*Q).mm(Q.T)
cov_w_np = np.cov(x.T, aweights=w.T)

print("WEIGHTED MEAN")
print("NUMPY = {0}\n\nTORCH = {1}\n\nDIFFERENCE={2}".format(m_w_np, m_w.numpy(), m_w_np-m_w.numpy()))
print("")
print("")
print("WEIGHTED COVARIANCE")
print("NUMPY = {0}\n\nTORCH = {1}\n\nDIFFERENCE={2}".format(cov_w_np, cov_w.numpy(),cov_w_np-cov_w.numpy()))

This yields the following output:

WEIGHTED MEAN
NUMPY = [-21.10537208  -7.70801723  64.4034329 ]

TORCH = [-21.10537208  -7.70801723  64.4034329 ]

DIFFERENCE=[-7.10542736e-15 -1.77635684e-15  1.42108547e-14]

WEIGHTED COVARIANCE
NUMPY = [[ 989468.17457696   13620.54885133   10723.87790683]
         [  13620.54885133  953966.92486133   21407.69378841]
         [  10723.87790683   21407.69378841 1019646.81044077]]

TORCH = [[ 987952.51042915   13599.68493868   10707.45110536]
         [ 13599.68493868  952505.64141296   21374.90155234]
         [ 10707.45110536   21374.90155234 1018084.91875621]]

DIFFERENCE=[[1515.6641478    20.86391265   16.42680147]
            [  20.86391265 1461.28344838   32.79223607]
            [  16.42680147   32.79223607 1561.89168456]]


title: Huggingface language modeling stuck at data reading phase

body:
I have a large file (1 GB+) with a mix of short and long texts (format: wikitext-2) for fine tuning the masked language model with bert-large-uncased
 as baseline model. I followed the instruction at https://github.com/huggingface/transformers/tree/master/examples/language-modeling. The process seems to be stuck at a stage "Creating features from dataset file at <file loc>". I am unsure what is wrong, is it really stuck or does it take really long for file of this size?

Command looks pretty much this:

export TRAIN_FILE=/path/to/dataset/my.train.raw
export TEST_FILE=/path/to/dataset/my.test.raw

python run_language_modeling.py \
    --output_dir=local_output_dir \
    --model_type=bert \
    --model_name_or_path=local_bert_dir \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm

Added: The job is running on CPU


title: My CNN regression predicts the same output for different inputs

body:
I was creating a CNN for prediction of the activity of some molecules. But while I train my data, they give the same prediction whatever I do. I made the linear layers deeper, I changed the learning rate and batches, I even tried to train the network one molecule at a time but also the same. I made sure that the dimensions are matched between the data and the prediction but it always give the same values (as if it only picks one value of the labels to train the whole data set on.)
class Net(nn.Module):
   def __init__(self):
       super().__init__()
       self.conv1 = nn.Conv2d(1, 32, 5)  # (input, features, kernel)
       # that kernel size is the best as it gives highest features
       self.conv2 = nn.Conv2d(32, 64, 5)
       self.conv3 = nn.Conv2d(64, 128, 5)

       x = torch.randn(267, 56).view(-1, 1, 267, 56)
       self._to_linear = None
       # generate a dummy array where our
       # /values to be flattened will be introduced inside
       self.convs(x)
       # Pooling and flattening to prepare for the linear network
       # Linear function

       self.fc1 = nn.Linear(self._to_linear, 5000)  # flattening
       self.fc2 = nn.Linear(5000, 1000)
       self.fc3 = nn.Linear(1000, 500)
       self.fc4 = nn.Linear(500, 256)
       self.fc5 = nn.Linear(256, 128)
       self.fc6 = nn.Linear(128, 32)
       self.fc7 = nn.Linear(32, 1)

   def convs(self, x):
       # Max pooling over 3x3
       x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
       x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
       x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))
       if self._to_linear is None:
           self._to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
       return x

   def forward(self, x):
       x = self.convs(x)
       x = x.view(-1, self._to_linear)
       # .view is reshape ... this flattens X before
       x = F.relu(self.fc1(x))
       x = F.relu(self.fc2(x))
       x = F.relu(self.fc3(x))
       x = F.relu(self.fc4(x))
       x = F.relu(self.fc5(x))
       x = F.relu(self.fc6(x))
       x = self.fc7(x)
       # bc this is our output layer. No activation here.
       return x

net = Net()
print(net)

# Optimizer and Loss Function
optimizer = optim.Adam(net.parameters(), lr=0.01)
loss_function = nn.MSELoss()

epochs = 10
BATCH_SIZE = 100
if torch.cuda.is_available(): #gpu adapted
   net1.cuda()

for epoch in range(epochs):# Converting inputs and labels to Variable (first 50)
   for i in tqdm(range(0, len(x_trial), BATCH_SIZE)):
       if torch.cuda.is_available():
           inputs = Variable(x_inp.cuda())
           labels = Variable(y_inp.cuda())
       else:
           inputs = Variable(x_trial[i:i+BATCH_SIZE])
           labels = Variable(y_trial[i:i+BATCH_SIZE])

       # Clear gradient buffers
       optimizer.zero_grad()

       # get output from the model, given the inputs
       outputs = net(inputs)

       # get loss for the predicted output
       loss = loss_function(outputs, labels)
       print(loss)
       # get gradients w.r.t to parameters
       loss.backward()

       # update parameters
       optimizer.step()

   print('epoch {}, loss {}'.format(epoch, loss.item()))

#The prediction is something like that :-
tensor([[6.1313],
       [6.1313],
       [6.1313],
       ...,
       [6.1313],
       [6.1313],
       [6.1313]])

credits in learning and the model to https://pythonprogramming.net/convnet-model-deep-learning-neural-network-pytorch/
Anyone could help?


title: Pytorch How To Set Every Procees GPU limit?

body:
How can Pytorch set GPU memory limit?
when I start uwsgi and setup 2 workers. then My GPU memory will share all the memory for two.
ex. 4GB GPU - >.  first worker use: 2 GB.  second worker 2GB
how can I setup first worker only use 1GB second worker use 1GB?


title: How to slice 3D torch tensor into 2D slices

body:
I'm working with 3D CT medical data, and I'm trying to slice it into 2D slices that I can input into a UNet model.
I've loaded the data into a torch dataloader, and each iteration currently produces a 4D tensor:
for batch_index, batch_samples in enumerate(train_loader):
    data, target = batch_samples['image'].float().cuda(), batch_samples['label'].float().cuda()
    print(data.size())
torch.Size([1, 333, 512, 512])
torch.Size([1, 356, 512, 512])

such as this one. I want to iterate over the 333 slices, and then the 356 slices, such that the model receives torch sizes [1, 1, 512, 512] each time.
I was hoping something like :
for x in (data[:,x,:,:]):

would work but it says I need to define x first. How can I iterate over a specific dimension in a torch tensor?


title: Why W_q matrix in torch.nn.MultiheadAttention is quadratic

body:
I am trying to implement nn.MultiheadAttention in my network. According to the docs,

embed_dim  – total dimension of the model.

However, according to the source file,

embed_dim must be divisible by num_heads

and

self.q_proj_weight = Parameter(torch.Tensor(embed_dim, embed_dim))

If I understand properly, this means each head takes only a part of features of each query, as the matrix is quadratic. Is it a bug of realization or is my understanding wrong?


title: Implementing BiLSTM-Attention-CRF Model using Pytorch

body:
I am trying to Implement the BiLSTM-Attention-CRF model for the NER task. I am able to perform NER tasks based on the BILSTM-CRF model (code from here) but I need to add attention to improve the performance of the model.
Right now my model is :
BiLSTM  -> Linear Layer (Hidden to tag) -> CRf Layer
The Output from the Linear layer is (seq. length x tagset size) and it is then fed into the CRF layer.
I am trying to replace the Linear layer with Attention layer using the code below:
class SelfAttention(nn.Module):
    def __init__(self, hidden_dim):
            super().__init__()
            self.hidden_dim = hidden_dim
            self.projection = nn.Sequential(
                    nn.Linear(hidden_dim, 64),
                    nn.ReLU(True),
                    nn.Linear(64, 1)
            )

    def forward(self, encoder_outputs):
            batch_size = encoder_outputs.size(0)
            # (B, L, H) -> (B , L, 1)
            energy = self.projection(encoder_outputs)
            weights = F.softmax(energy.squeeze(-1), dim=1)
            # (B, L, H) * (B, L, 1) -> (B, H)
            outputs = (encoder_outputs * weights.unsqueeze(-1)).sum(dim=1)
            return outputs, weights

While doing so I have two issues:

I can not make it work so that the output should come in the shape of (seq. length x tagset size) so that it can be fed into CRF Layer.
According to this paper, we need to initialize and learn word-level context vector which I can not see in this implementation of the attention model.

Kindly help me out.
TIA


title: How to feed a Neural Network with a Complex-Valued Inputs

body: I am building a deep neural network for signal processing, and as you know the real-life signal is represented by a complex number. How can a complex-valued input be feed into a neural network? Is there any standard machine learning library for building a complex-valued neural network? Please help


title: Should I use fp16 tensors to utilize tensor core?

body:
I am trying to utilize the V100 machine on AWS. The official site says that it provides over 100TFLOPS for machine learning application, but it seems not working that faster than the 1080ti I have been used.

Recentely, I found an article that says tensor core operates on fp16 tensor.

So, should I specify the dtype for tensor, like

tensor_a = torch.tensor([1.,2.,3.], dtype=torch.float16)

instead of:

tensor_a = torch.tensor([1.,2.,3])?


title: Empty results from concurrent psycopg2 postgres select queries

body:
I am attempting to retrieve my label and feature datasets from a postgres database using the getitem method from a custom pytorch dataset. When I attempt to sample with random indexes my queries return no results

I have checked to see if my queries work directly on the psql cli. They do.
I have checked my database connection pool for issues. Does not seem to be any.
I have reverted back to sequential sampling and it is still fully functional so it is the random index values that seem to be an issue for query.

The getitem method which performs the queries is place below. This shows both the sequential and attempt to shuffle queries. Both of these are clearly labeled via variable name.

def __getitem__(self, idx):

        query = """SELECT ls.taxonomic_id, it.tensor
                    FROM genomics.tensors2 AS it
                    INNER JOIN genomics.labeled_sequences AS ls
                    ON ls.accession_number = it.accession_number
                    WHERE (%s) <= it.index
                    AND CARDINALITY(tensor) = 89
                    LIMIT (%s) OFFSET (%s)"""

        shuffle_query = """BEGIN
                           SELECT ls.taxonomic_id, it.tensor
                           FROM genomics.tensors2 AS it
                           INNER JOIN genomics.labeled_sequences AS ls
                           ON ls.accession_number = it.accession_number
                           WHERE it.index BETWEEN (%s) AND (%s)
                           END"""

        batch_size = 500
        upper_bound = idx + batch_size

        query_data = (idx, batch_size, batch_size)
        shuffle_query_data = (idx, upper_bound)

        result = None
        results = None

        conn = self.conn_pool.getconn()

        try:
            conn.set_session(readonly=True, autocommit=True)
            cursor = conn.cursor()
            cursor.execute(query, query_data)
            results = cursor.fetchall()
            self.conn_pool.putconn(conn)

            print(idx)
            print(results)
        except Error as conn_pool_error:
            print('Multithreaded __getitem__ query error')
            print(conn_pool_error)

        label_list = []
        sequence_list = []

        for (i,result) in enumerate(results):
            if result is not None:
                (label, sequence) = self.create_batch_stack_element(result)

                label_list.append(label)
                sequence_list.append(sequence)

        label_stack = torch.stack(label_list).to('cuda')
        sequence_stack = torch.stack(sequence_list).to('cuda')

        return (label_stack, sequence_stack)

    def create_batch_stack_element(self, result):
        if result is not None:

            label = np.array(result[0], dtype=np.int64)
            sequence = np.array(result[1], dtype=np.int64)

            label = torch.from_numpy(label)
            sequence = torch.from_numpy(sequence)

            return (label, sequence)

        else:
            return None

The error I receive comes from my attempt to stack my list of tensors after the for loop. This fails because the lists are empty. Since the lists are filled in the loop based off the results of the query. It points to the query being the issue.

I would like some help with my source code to solve this issue and possibly an explanation as to why my concurrent queries with random indexes are failing.

Thanks. Any help is appreciated.

E: I believe I have found the source of the issue and it comes from the pytorch RandomSampler source code. I believe it is providing indexed out of the range of my database keys. This explains why I have no results from the queries. I will have to write my own sampler class to limit this value to the length of my dataset. What an oversight on my part.

E2: The random sampling now works with a customized sampler class but prevents mutlithreaded querying.

E3: I now have the entire problem solved. Using multiple processes to load data to the GPU with a custom random sampler. Will post applicable code when I get a chance and accept it as an answer to close out the thread.


title: Cleaner way to use "with torch.no_grad()" conditioned on an expression

body:
My code looks like:

if no_grad_condition:
  with torch.no_grad():
    out=network(input)
else:
  out=network(input)

Is there a cleaner way to do it, without duplicating the line out=network(input)?

I am looking for something in the spirit of:

  with torch.no_grad(no_grad_condition):
    out=network(input)


title: The loss effects in multitask learning framework

body:
I have designed a multi-task network where the first layers are shared between two output layers. Through investigating multi-task learning principles, I got to know that there should be a weight scalar parameter such as alpha that dampens the two losses outputted from two output layers. My question is about this parameter itself. Does it have effect on the model's final performance? probably yes.

This is the part of my code snippet for computation of losses:

...

mtl_loss = (alpha) * loss_1 + (1-alpha) * loss_2
mtl_loss.backward()

...

Above, loss_1 is MSELoss, and loss_2 is CrossEntropyLoss. As such, picking alpha=0.9, I'm getting the following loss values during training steps:

[2020-05-03 04:46:55,398 INFO] Step 50/150000; loss_1: 0.90 + loss_2: 1.48 = mtl_loss: 2.43 (RMSE: 2.03, F1score: 0.07); lr: 0.0000001;  29 docs/s;     28 sec
[2020-05-03 04:47:23,238 INFO] Step 100/150000; loss_1: 0.40 + loss_2: 1.27 = mtl_loss: 1.72 (RMSE: 1.38, F1score: 0.07); lr: 0.0000002;  29 docs/s;     56 sec
[2020-05-03 04:47:51,117 INFO] Step 150/150000; loss_1: 0.12 + loss_2: 1.19 = mtl_loss: 1.37 (RMSE: 0.81, F1score: 0.08); lr: 0.0000003;  29 docs/s;     84 sec
[2020-05-03 04:48:19,034 INFO] Step 200/150000; loss_1: 0.04 + loss_2: 1.10 = mtl_loss: 1.20 (RMSE: 0.55, F1score: 0.07); lr: 0.0000004;  29 docs/s;    112 sec
[2020-05-03 04:48:46,927 INFO] Step 250/150000; loss_1: 0.02 + loss_2: 0.96 = mtl_loss: 1.03 (RMSE: 0.46, F1score: 0.08); lr: 0.0000005;  29 docs/s;    140 sec
[2020-05-03 04:49:14,851 INFO] Step 300/150000; loss_1: 0.02 + loss_2: 0.99 = mtl_loss: 1.05 (RMSE: 0.43, F1score: 0.08); lr: 0.0000006;  29 docs/s;    167 sec
[2020-05-03 04:49:42,793 INFO] Step 350/150000; loss_1: 0.02 + loss_2: 0.97 = mtl_loss: 1.04 (RMSE: 0.43, F1score: 0.08); lr: 0.0000007;  29 docs/s;    195 sec
[2020-05-03 04:50:10,821 INFO] Step 400/150000; loss_1: 0.01 + loss_2: 0.94 = mtl_loss: 1.00 (RMSE: 0.41, F1score: 0.08); lr: 0.0000008;  29 docs/s;    223 sec
[2020-05-03 04:50:38,943 INFO] Step 450/150000; loss_1: 0.01 + loss_2: 0.86 = mtl_loss: 0.92 (RMSE: 0.40, F1score: 0.08); lr: 0.0000009;  29 docs/s;    252 sec

As training loss shows, it seems that my first network that uses MSELoss converges super fast, while the second network has not been converged yet. RMSE, and F1score are two metrics that I'm using to track the progress of first, and second network, respectively.

I know that picking the optimal alpha is somewhat experimental, but are there hints to make the process of picking it easier? Specifically, I want the networks being trained in line with each other, not like above that the first network converges super duper fast. Can alpha parameter help controlling this?


title: Get gradient of loss w.r.t to sparse matix in pytorch

body:
I'm trying to get the gradient of loss w.r.t to a sparse matrix in pytorch for my neural network. But I get this error calculating the gradient of a sparse Tensor argument to mm is not supported.
The forward function for my network is
class Layer(Module):
   .....
        def forward(self, input, adj):

           support = torch.mm(input, self.weight)
           output = torch.spmm(adj, support)
           if self.bias is not None:
               return output + self.bias, self.weight
           else:
               return output, self.weight

and my model is
class Model(nn.Module):
def __init__(self, nfeat, nhid1, nhid2, nhid3, nclass, dropout, adj_shape, with_relu=False):
    super(GCN, self).__init__()

    self.gc1 = Layer(nfeat, nhid1)
    self.gc2 = Layer(nhid1, nclass)
    self.dropout = dropout
    self.with_relu = with_relu

def forward(self, x, adj):
    features = x
    x , w0 = self.gc1(x, adj)
    x, w1 = self.gc2(x, adj)
    return F.log_softmax(x, dim=1)

and loss will be
output = model(features, adj)
loss_train = F.nll_loss(output, labels)

So if features(this is the input) = X, adj(used in layers) = A
the function will be like
nll_loss(log_softmax(AAXW0W1))

and A is sparse tensor
If I try to get the gradient of loss w.r.t A with torch.autograd.grad(loss_train, adj) I get the error I mentioned at the top.
I know if I change output = torch.spmm(adj, support) to output = torch.sparse.mm(adj, support) I can get the gradient but this is converting the sparse matrix into a dense matrix in the background. So it won't work when A is a huge matrix(100k, 100k) because it takes too much time and also you need a lot of RAM to fit the matrix. Therefore I want A to stay sparse throughout backdrop.
Does anyone have any idea how to do this?
Any suggestions will be appreciated.
Thanks!


title: cannot import name '_C' from 'torchvision' on RaspberryPi 4

body:
I've been trying to run a fruit detection model on my raspberry pi using PyTorch, but this error (below) keeps happening. Did someone had this problem before?
Traceback (most recent call last):
  File "modelLoad.py", line 25, in <module>
    predictions = model.predict_top(image)
  File "/home/pi/.local/lib/python3.7/site-packages/detecto/core.py", line 369, in predict_top
    predictions = self.predict(images)
  File "/home/pi/.local/lib/python3.7/site-packages/detecto/core.py", line 322, in predict
    preds = self._get_raw_predictions(images)
  File "/home/pi/.local/lib/python3.7/site-packages/detecto/core.py", line 278, in _get_raw_predictions
    preds = self._model(images)
  File "/home/pi/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/pi/.local/lib/python3.7/site-packages/torchvision/models/detection/generalized_rcnn.py", line 51, in forward
    proposals, proposal_losses = self.rpn(images, features, targets)
  File "/home/pi/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/pi/.local/lib/python3.7/site-packages/torchvision/models/detection/rpn.py", line 411, in forward
    boxes, scores = self.filter_proposals(proposals, objectness, images.image_sizes, num_anchors_per_level)
  File "/home/pi/.local/lib/python3.7/site-packages/torchvision/models/detection/rpn.py", line 336, in filter_proposals
    keep = box_ops.batched_nms(boxes, scores, lvl, self.nms_thresh)
  File "/home/pi/.local/lib/python3.7/site-packages/torchvision/ops/boxes.py", line 72, in batched_nms
    keep = nms(boxes_for_nms, scores, iou_threshold)
  File "/home/pi/.local/lib/python3.7/site-packages/torchvision/ops/boxes.py", line 32, in nms
    _C = _lazy_import()
  File "/home/pi/.local/lib/python3.7/site-packages/torchvision/extension.py", line 12, in _lazy_import
    from torchvision import _C as C
ImportError: cannot import name '_C' from 'torchvision' (/home/pi/.local/lib/python3.7/site-packages/torchvision/__init__.py)


title: Hugging face, Albert loads model on both CPU and GPU at the same time

body:
While trying to load model on GPU, model also loads into CPU
The below code load the model in both.
MODEL_DEVICE = torch.device('cuda')
MODEL_PATH='./models'
tokenizer = AlbertTokenizer.from_pretrained(MODEL_PATH)
qa_model = AlbertForQuestionAnswering.from_pretrained(MODEL_PATH).to(MODEL_DEVICE)

I have referred to below link for reference.
https://huggingface.co/transformers/quickstart.html
This model occupies both RAM memory(1.5 GB) and GPU memory(650 MB).
I have specified torch device as Cuda, But still, it doesn't behave as expected.
When "cpu" is specified it works well and doesn't load into GPU. But when cuda is specified it load in CPU and GPU as well.
I tried , "cuda", "cuda:0"
I also tried changing hugging face library code from
state_dict = torch.load(resolved_archive_file, map_location="cpu")

to
state_dict = torch.load(resolved_archive_file, map_location="cuda")

Any solution to this ?


title: How to link libtorch libraries statically in CMake?

body:
Following https://pytorch.org/cppdocs/installing.html, I have
cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(example-app)

find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

include_directories(
    "${TORCH_INCLUDE_DIRS}" # libtorch headers
)

add_executable(example-app example-app.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 14)

This works but links to the shared Torch libraries; how can I link to the static ones instead?


title: How to assert torch.float32/64 using isinstance?

body:
I am writing a pipeline for my deep learning project, as a good practice, I try to do some assertions to make sure the data type matches, so it's easier to debug later! However, I do not know how to use isinstance to assert the torch.Tensor's dtype.
For example:
    assert isinstance(image, torch.Tensor) and isinstance(target['boxes'], torch.Tensor)
    assert isinstance(image.dtype, (torch.float32, torch.float64)) and isinstance(target['boxes'].dtype, (torch.float32, torch.float64))

Is using assert image.dtype in [torch.float32, torch.float64] the only method here? Is there an elegant way to do so?


title: How do I reshape pytorch tensor from [a, b*c] to [b,a,c]?

body:
I'm getting this code error, and I am unsure on how to reshape my tensor.
RuntimeError: Expected 3-dimensional input for 3-dimensional weight [32, 35, 2], but got 2-dimensional input of size [35, 64] instead


title: Deep Q-Learning : torch.nn.functional.softmax crash

body:
I am following a tutorial, and the function softmax crashes when I use it.

newSignals = [0.5, 0., 0., -0.7911, 0.7911]
newState = torch.Tensor(newSignals).float().unsqueeze(0)
probs = F.softmax(self.model(newState), dim=1)

self.model is a neural network (torch.nn.module), which return a Tensor like

tensor([[ 0.2699, -0.2176,  0.0333]], grad_fn=<AddmmBackward>)

So, the line probs = F.softmax(self.model(newState), dim=1) crash the program but when dim=0 it works but it is not good.