title: Compare the efficiency of the data loading methods in deep learning

body:
I need to load a time-series dataset to train a network. The dataset was split into many chunks train_x_0.npy, train_x_1.npy, ..., train_x_40.npy (41 chunks) because of memory issue when I extract these .npy files from the raw data. However, their sizes are too large (around 1000 GB) that I couldn't load everything into the RAM. I have been considering two ways to solve this problem.

Loading the data chunks using np.load() with argument mmap_mode='r+'. The memory-mapped chunks are stored in a Python list self.data. In the __getitem__(self, idx) method of Pytorch Dataset class, I convert idx to chunk_idx and sample_idx, then get the sample by self.data[chunk_idx][sample_idx].
Extract .npy files again from raw data, and save the data sample-by-sample, i.e. one .npy file is now one sample, not a data chunk. In the __getitem__(self, idx) method, I will get one sample by loading it using np.load(sample_path).

Assuming the Pytorch DataLoader will be used to iterate through all samples, then which method will be faster?
If you have another suggestion to extract the raw data or to load the .npy files, please share your opinion.


title: How can I get predictions from these pretrained models?

body:
I've been trying to generate human pose estimations, I came across many pretrained models (ex. Pose2Seg, deep-high-resolution-net ), however these models only include scripts for training and testing, this seems to be the norm in code written to implement models from research papers ,in deep-high-resolution-net I have tried to write a script to load the pretrained model and feed it my images, but the output I got was a bunch of tensors and I have no idea how to convert them to the .json annotations that I need.

total newbie here, sorry for my poor English in advance, ANY tips are appreciated.

I would include my script but its over 100 lines.

PS: is it polite to contact the authors and ask them if they can help?

because it seems a little distasteful.


title: How to write a custom loss function in Keras/Tensorflow that uses loops/iterations with reference numpy code

body:
I saw this question: Implementing custom loss function in keras with condition And I need to do the same thing but with code that seems to need loops.

I have a custom numpy function which calculates the mean Euclid distance from the mean vector. I wrote this based on the paper https://arxiv.org/pdf/1801.05365.pdf:

import numpy as np

def mean_euclid_distance_from_mean_vector(n_vectors):

    dists = []

    for (i, v) in enumerate(n_vectors):
        n_vectors_rest = n_vectors[np.arange(len(n_vectors)) != i]

        print("rest of vectors: ")
        print(n_vectors_rest)

        # calculate mean vector
        mean_rest = n_vectors_rest.mean(axis=0)

        print("mean rest vector")
        print(mean_rest)

        dist = v - mean_rest

        print("dist vector")
        print(dist)
        dists.append(dist)

    # dists is now a matrix of distance vectors (distance from the mean vector)
    dists = np.array(dists)

    print("distance vector matrix")
    print(dists)

    # here we matmult each vector
    # sum them up
    # and divide by the total number of elements
    result = np.sum([np.matmul(d, d) for d in dists]) / dists.size

    return result

features = np.array([
    [1,2,3,4],
    [4,3,2,1]
])

c = mean_euclid_distance_from_mean_vector(features)

print(c)

I need this function however to work inside tensorflow with Keras. So a custom lambda https://www.tensorflow.org/api_docs/python/tf/keras/layers/Lambda

However, I'm not sure how to implement the above in Keras/Tensorflow since it has loops, and the way the paper talked about calculating the m_i seems to require loops like the way I implemented the above.

For reference, the PyTorch version of this code is here: https://github.com/PramuPerera/DeepOneClass


title: Having a network output as another network parameters

body:
I have y = Y(x;theta) and theta = M(t;omega), where x and t are input variables (given from the dataset), and theta and omega trainable parameters. I need to have theta as function of omega. Then, I have a loss function over y and need to backpropagate the gradient through M up to Y. How can I create such a structure in pytorch?

Currently, my network is built as follows (sizes is a list of integers, defined as sizes = [input_size, hidden1_size, hidden2_size, ..., output_size])

import torch
import torch.nn as nn
import torch.nn.functional as F

class M(nn.Module):
    def __init__(self, sizes):
        super(Y, self).__init__()
        self.layers = nn.ModuleList()
        for i in range(0, len(sizes) - 1):
            self.layers.append(nn.Linear(sizes[i], sizes[i+1]))

    def forward(self, x):
        for l in self.layers[:-1]:
            x = F.relu(l(x))
        x = self.layers[-1](x)

        return x


title: How does one implemented a parametrized meta-learner in Pytorch's higher library?

body:
I wanted to implement the meta-lstm meta-learner in the paper OPTIMIZATION AS A MODEL FOR FEW-SHOT LEARNING using higher but I found problems. I found that I cannot make it work without removing (what seems to be this crucial line):
https://github.com/facebookresearch/higher/blob/8f0716fb1663218324c02dabdba26b639959cfb6/higher/optim.py#L101
to:
#self.param_groups = _copy.deepcopy(other.param_groups)
self.param_groups = other.param_groups

I provide an extremely simplified self-contained implementation of something similar here:
https://gist.github.com/renesax14/8499e0314351ea4199a17e494bff5c4d
but I will copy paste here to keep the discussion in one place:
# base on the paper "OPTIMIZATION AS A MODEL FOR FEW-SHOT LEARNING": https://openreview.net/pdf?id=rJY0-Kcll

class EmptySimpleMetaLstm(Optimizer):

    def __init__(self, params, trainable_opt_model, trainable_opt_state, *args, **kwargs):
        defaults = {
            'trainable_opt_model':trainable_opt_model,
            'trainable_opt_state':trainable_opt_state,
            'args':args,
            'kwargs':kwargs
        }
        super().__init__(params, defaults)

class SimpleMetaLstm(DifferentiableOptimizer):

    def _update(self, grouped_grads, **kwargs):
        prev_lr = self.param_groups[0]['trainable_opt_state']['prev_lr']
        eta = self.param_groups[0]['trainable_opt_model']['eta']
        # start differentiable & trainable update
        zipped = zip(self.param_groups, grouped_grads)
        for group_idx, (group, grads) in enumerate(zipped):
            for p_idx, (p, g) in enumerate(zip(group['params'], grads)):
                if g is None:
                    continue
                # get gradient as "data"
                g = g.detach() # gradients of gradients are not used (no hessians)
                ## very simplified version of meta-lstm meta-learner
                input_metalstm = torch.stack([p, g, prev_lr.view(1,1)]).view(1,3) # [p, g, prev_lr] note it's missing loss, normalization etc. see original paper
                lr = eta(input_metalstm).view(1)
                fg = 1 - lr # learnable forget rate
                ## update suggested by meta-lstm meta-learner
                p_new = fg*p - lr*g
                group['params'][p_idx] = p_new
        # fake returns
        self.param_groups[0]['trainable_opt_state']['prev_lr'] = lr

higher.register_optim(EmptySimpleMetaLstm, SimpleMetaLstm)

def test_parametrized_inner_optimizer():
    import torch
    import torch.nn as nn
    import torch.optim as optim
    from collections import OrderedDict

    ## training config
    #device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    track_higher_grads = True # if True, during unrolled optimization the graph be retained, and the fast weights will bear grad funcs, so as to permit backpropagation through the optimization process. False during test time for efficiency reasons
    copy_initial_weights = False # if False then we train the base models initial weights (i.e. the base model's initialization)
    episodes = 5
    nb_inner_train_steps = 5
    ## get base model
    base_mdl = nn.Sequential(OrderedDict([
        ('fc', nn.Linear(1,1, bias=False)),
        ('relu', nn.ReLU())
        ]))
    ## parametrization/mdl for the inner optimizer
    opt_mdl = nn.Sequential(OrderedDict([
        ('fc', nn.Linear(3,1, bias=False)), # 3 inputs 1 for parameter, 1 for gradient, 1 for previous lr
        ('sigmoid', nn.Sigmoid())
        ]))
    ## get outer optimizer (not differentiable nor trainable)
    outer_opt = optim.Adam([{'params': base_mdl.parameters()},{'params': opt_mdl.parameters()}], lr=0.01)
    for episode in range(episodes):
        ## get fake support & query data (from a single task and 1 data point)
        spt_x, spt_y, qry_x, qry_y = torch.randn(1), torch.randn(1), torch.randn(1), torch.randn(1)
        ## get differentiable & trainable (parametrized) inner optimizer
        inner_opt = EmptySimpleMetaLstm(base_mdl.parameters(), trainable_opt_model={'eta': opt_mdl}, trainable_opt_state={'prev_lr': 0.9*torch.randn(1)})
        with higher.innerloop_ctx(base_mdl, inner_opt, copy_initial_weights=copy_initial_weights, track_higher_grads=track_higher_grads) as (fmodel, diffopt):
            for i_inner in range(nb_inner_train_steps): # this current version implements full gradient descent on k_shot examples (which is usually small  5)
                fmodel.train()
                # base/child model forward pass
                inner_loss = 0.5*((fmodel(spt_x) - spt_y))**2
                # inner-opt update
                diffopt.step(inner_loss)
            ## Evaluate on query set for current task
            qry_loss = 0.5*((fmodel(qry_x) - qry_y))**2
            qry_loss.backward() # for memory efficient computation
        ## outer update
        print(f'episode = {episode}')
        print(f'base_mdl.grad = {base_mdl.fc.weight.grad}')
        print(f'opt_mdl.grad = {opt_mdl.fc.weight.grad}')
        outer_opt.step()
        outer_opt.zero_grad()

if __name__ == '__main__':
    test_parametrized_inner_optimizer()
    print('Done \a')

"""
output when deep copy is uncommented (parametrized optimizer trains properly):
episode = 0
base_mdl.grad = tensor([[-0.0351]])
opt_mdl.grad = tensor([[0.0085, 0.0000, 0.0204]])
episode = 1
base_mdl.grad = tensor([[0.0311]])
opt_mdl.grad = tensor([[-0.0086, -0.0100,  0.0358]])
episode = 2
base_mdl.grad = tensor([[0.]])
opt_mdl.grad = tensor([[0., 0., 0.]])
episode = 3
base_mdl.grad = tensor([[0.0066]])
opt_mdl.grad = tensor([[-0.0016,  0.0000, -0.0032]])
episode = 4
base_mdl.grad = tensor([[-0.0311]])
opt_mdl.grad = tensor([[0.0077, 0.0000, 0.0130]])
Done
when deep copy is on (paremeters of inner optimizer are not train, sad!):
episode = 0
base_mdl.grad = tensor([[0.]])
opt_mdl.grad = None
episode = 1
base_mdl.grad = tensor([[0.]])
opt_mdl.grad = None
episode = 2
base_mdl.grad = tensor([[0.0069]])
opt_mdl.grad = None
episode = 3
base_mdl.grad = tensor([[0.]])
opt_mdl.grad = None
episode = 4
base_mdl.grad = tensor([[0.]])
opt_mdl.grad = None
Done
The deep copy line in higher I am referencing:
        self.param_groups = _copy.deepcopy(other.param_groups)
        #self.param_groups = other.param_groups
"""

Real Solution
The real solution is if I could pass an arbitrary dictionary to a differentiable optimizer and if I could do whatever I wanted with it.

update:
perhaps this can be implemented with override:

override (optional) – a dictionary mapping optimizer settings (i.e. those which would be passed to the optimizer constructor or provided within parameter groups) to either singleton lists of override values, or to a list of override values of length equal to the number of parameter groups. If a single override is provided for a keyword, it is used for all parameter groups. If a list is provided, the ith element of the list overrides the corresponding setting in the ith parameter group. This permits the passing of tensors requiring gradient to differentiable optimizers for use as optimizer settings.

Didn't work with override:
Exception has occurred: ValueError
Mismatch between the number of override tensors for optimizer parameter trainable_opt_model and the number of parameter groups.
seems like it checks that these lengths match...

def _apply_override(self, override: _OverrideType) -> None:
for k, v in override.items():
# Sanity check
if (len(v) != 1) and (len(v) != len(self.param_groups)

I think this is all I need:
inner_opt = EmptySimpleMetaLstm( base_mdl.parameters() )
with higher.innerloop_ctx(base_mdl, inner_opt, copy_initial_weights=copy_initial_weights, track_higher_grads=track_higher_grads) as (fmodel, diffopt):
     diffopt.override = {'trainable_opt_model': opt_mdl, 'trainable_opt_state': {'prev_lr': 0.9*torch.randn(1)} }

cross-posted:

git issue in higher: https://github.com/facebookresearch/higher/issues/62
https://www.reddit.com/r/pytorch/comments/hbp1n5/how_does_one_implemented_a_parametrized/?
https://discuss.pytorch.org/t/how-does-one-implemented-a-parametrized-meta-learner-in-pytorchs-higher-library/85988

related:

Why does higher need to deep copy the parameters of the base model to create a functional model?
What does the copy_initial_weights documentation mean in the higher library for Pytorch?


File
"/home/jake/venv/lib/python3.7/site-packages/torchvision/models/detection/rpn.py",
line 274, in assign_targets_to_anchors
match_quality_matrix = self.box_similarity(gt_boxes, anchors_per_image)   File
"/home/jake/venv/lib/python3.7/site-packages/torchvision/ops/boxes.py",
line 153, in box_iou
lt = torch.max(boxes1[:, None, :2], boxes2[:, :2])  # [N,M,2] RuntimeError: Expected object of scalar type Double but got scalar
type Float for argument #2 'other' in call to _th_max

got the following error  and here is the type
annotations-> [{'boxes': tensor([[131.5124, 169.4232, 165.8235, 209.7217],
        [157.2659, 185.7604, 199.0850, 210.4478],
        [184.4726, 171.3594, 211.5178, 210.3267],
        [170.5867, 154.1751, 211.7600, 188.7858],
        [162.8364, 156.2324, 182.1313, 186.7285],
        [131.2702, 149.8185, 149.4349, 172.3276],
        [192.0614, 140.5002, 208.4500, 155.2642],
        [165.2583, 140.5002, 188.2670, 158.0476],
        [169.5371, 131.9080, 193.3531, 151.1497],
        [131.1895, 133.2392, 140.1507, 156.3534]], device='cuda:0',
       dtype=torch.float64), 'labels': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], device='cuda:0'), 'area': tensor([1382.6857, 1032.4053, 1053.8795, 1425.0382,  588.4218,  408.8705,
         241.9614,  403.7413,  458.2574,  207.1317], device='cuda:0'), 'iscrowd': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0')}]

I think boxes type is float64 . this cause the problem how to fix it to double?


title: Received a make error while building warp-ctc pytorch

body:
I am trying to build https://github.com/SeanNaren/warp-ctc.git on Google Colab, following this notebook. I am using these commands on Colab:
!git clone https://github.com/SeanNaren/warp-ctc.git;\
cd warp-ctc;\
mkdir build;\
cd build;\
cmake ..;\
make;

but I am receiving an error building it:
[-11%] Building NVCC (Device) object CMakeFiles/warpctc.dir/src/warpctc_generated_ctc_entrypoint.cu.o
/content/drive/My Drive/simple_hwr/warp-ctc/src/ctc_entrypoint.cu(1): error: this declaration has no storage class or type specifier

/content/drive/My Drive/simple_hwr/warp-ctc/src/ctc_entrypoint.cu(1): error: expected a ";"

2 errors detected in the compilation of "/tmp/tmpxft_00000191_00000000-13_ctc_entrypoint.compute_70.cpp1.ii".
CMake Error at warpctc_generated_ctc_entrypoint.cu.o.cmake:279 (message):
  Error generating file /content/drive/My
  Drive/simple_hwr/warp-ctc/build/CMakeFiles/warpctc.dir/src/./warpctc_generated_ctc_entrypoint.cu.o

CMakeFiles/warpctc.dir/build.make:220: recipe for target 'CMakeFiles/warpctc.dir/src/warpctc_generated_ctc_entrypoint.cu.o' failed
make[2]: *** [CMakeFiles/warpctc.dir/src/warpctc_generated_ctc_entrypoint.cu.o] Error 1
CMakeFiles/Makefile2:146: recipe for target 'CMakeFiles/warpctc.dir/all' failed
make[1]: *** [CMakeFiles/warpctc.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

How can this be resolved?


title: PyTorch error when importing another model

body:
I tried to import state-of-the-art EfficientNet model (pytorch implementation):
from efficientnet_pytorch import EfficientNet

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import matplotlib.pyplot as plt
from torchvision.transforms import transforms as T
import torchvision
import torch.optim as optim

model = EfficientNet.from_pretrained('efficientnet-b0')

And I got this chain of errors when downloading model:
Downloading: "https://github.com/lukemelas/EfficientNet-PyTorch/releases/download/1.0/efficientnet-b0-355c32eb.pth" to /Users/telaust/.cache/torch/checkpoints/efficientnet-b0-355c32eb.pth
---------------------------------------------------------------------------

TimeoutError                              Traceback (most recent call last)
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1316                 h.request(req.get_method(), req.selector, req.data, headers,
-> 1317                           encode_chunked=req.has_header('Transfer-encoding'))
   1318             except OSError as err: # timeout error

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1228         """Send a complete request to the server."""
-> 1229         self._send_request(method, url, body, headers, encode_chunked)
   1230

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1274             body = _encode(body, 'body')
-> 1275         self.endheaders(body, encode_chunked=encode_chunked)
   1276

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py in endheaders(self, message_body, encode_chunked)
   1223             raise CannotSendHeader()
-> 1224         self._send_output(message_body, encode_chunked=encode_chunked)
   1225

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py in _send_output(self, message_body, encode_chunked)
   1015         del self._buffer[:]
-> 1016         self.send(msg)
   1017

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py in send(self, data)
    955             if self.auto_open:
--> 956                 self.connect()
    957             else:

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py in connect(self)
   1383
-> 1384             super().connect()
   1385

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py in connect(self)
    927         self.sock = self._create_connection(
--> 928             (self.host,self.port), self.timeout, self.source_address)
    929         self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py in create_connection(address, timeout, source_address)
    726     if err is not None:
--> 727         raise err
    728     else:

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py in create_connection(address, timeout, source_address)
    715                 sock.bind(source_address)
--> 716             sock.connect(sa)
    717             # Break explicitly a reference cycle

TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

URLError                                  Traceback (most recent call last)
<ipython-input-5-5e9bdf5b3e02> in <module>
----> 1 model = EfficientNet.from_pretrained('efficientnet-b0')

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/efficientnet_pytorch/model.py in from_pretrained(cls, model_name, advprop, num_classes, in_channels)
    209     def from_pretrained(cls, model_name, advprop=False, num_classes=1000, in_channels=3):
    210         model = cls.from_name(model_name, override_params={'num_classes': num_classes})
--> 211         load_pretrained_weights(model, model_name, load_fc=(num_classes == 1000), advprop=advprop)
    212         if in_channels != 3:
    213             Conv2d = get_same_padding_conv2d(image_size = model._global_params.image_size)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/efficientnet_pytorch/utils.py in load_pretrained_weights(model, model_name, load_fc, advprop)
    325     # AutoAugment or Advprop (different preprocessing)
    326     url_map_ = url_map_advprop if advprop else url_map
--> 327     state_dict = model_zoo.load_url(url_map_[model_name])
    328     if load_fc:
    329         model.load_state_dict(state_dict)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/hub.py in load_state_dict_from_url(url, model_dir, map_location, progress, check_hash)
    490         sys.stderr.write('Downloading: "{}" to {}\n'.format(url, cached_file))
    491         hash_prefix = HASH_REGEX.search(filename).group(1) if check_hash else None
--> 492         download_url_to_file(url, cached_file, hash_prefix, progress=progress)
    493
    494     # Note: extractall() defaults to overwrite file if exists. No need to clean up beforehand.

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/hub.py in download_url_to_file(url, dst, hash_prefix, progress)
    389     # We use a different API for python2 since urllib(2) doesn't recognize the CA
    390     # certificates in older Python
--> 391     u = urlopen(url)
    392     meta = u.info()
    393     if hasattr(meta, 'getheaders'):

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    220     else:
    221         opener = _opener
--> 222     return opener.open(url, data, timeout)
    223
    224 def install_opener(opener):

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py in open(self, fullurl, data, timeout)
    523             req = meth(req)
    524
--> 525         response = self._open(req, data)
    526
    527         # post-process response

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py in _open(self, req, data)
    541         protocol = req.type
    542         result = self._call_chain(self.handle_open, protocol, protocol +
--> 543                                   '_open', req)
    544         if result:
    545             return result

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
    501         for handler in handlers:
    502             func = getattr(handler, meth_name)
--> 503             result = func(*args)
    504             if result is not None:
    505                 return result

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py in https_open(self, req)
   1358         def https_open(self, req):
   1359             return self.do_open(http.client.HTTPSConnection, req,
-> 1360                 context=self._context, check_hostname=self._check_hostname)
   1361
   1362         https_request = AbstractHTTPHandler.do_request_

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1317                           encode_chunked=req.has_header('Transfer-encoding'))
   1318             except OSError as err: # timeout error
-> 1319                 raise URLError(err)
   1320             r = h.getresponse()
   1321         except:

URLError: <urlopen error [Errno 60] Operation timed out>

I'm so confused about this error, so I see that problem is related to URL, but I can't solve it. Before that I installed this model through pip:
pip3 install efficientnet_pytorch

How can I fix this error? Thank you in advance


hi I have the following code
class GRU(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes ):
        super(GRU, self).__init__()

        #WE STORE THE LAYERS AND HIDDENSIZE
        self.num_layers = num_layers
        self.hidden_size = hidden_size

        self.gru = nn.GRU(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size* sequence_length, num_classes)

    def forward(self, x):
        # Set initial hidden states (and cell states for LSTM)
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)
        # Forward propagate RNN
        out, _ = self.gru(x, h0)
        out=out.reshape(out.shape[0], -1)
        out = self.fc(out)
        return out

and the following hyperparameterers
# Hyper-parameters
num_classes = 6
batch_size = 32
learning_rate = 0.0025
dropout=0.2
epochs = 10

input_size = len(X_train[0][0])
sequence_length = len(X_train[0])
hidden_size = 64
num_layers = 2

print (input_size)
print (sequence_length)
6
100

my data are three dimensions
X_train.shape
torch.Size([4078, 100, 6])
X_test.shape
torch.Size([1020, 100, 6])

however, when I am trying to run the code I take back the error of the description. I check the official site of PyTorch and other same problems but cannot find the solution.
i would appreciate it if someone can help with the code.
this is the full error message
RuntimeError                              Traceback (most recent call last)
<ipython-input-47-21beb733af1e> in <module>
      1 test_accs, confusion_mtxes = [], []
      2 for epoch in range(1, epochs + 1):
----> 3     train(X_train, y_train)
      4     test_acc, confusion_mtx = test(X_test, y_test)
      5     test_accs.append(test_acc)

<ipython-input-45-d10728615dab> in train(X_train, y_train)
      4         data, target = data, target
      5         optimizer.zero_grad()
----> 6         output = model(data)
      7         loss = criterion(output, target)
      8         loss.backward()

~\AppData\Roaming\Python\Python37\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

<ipython-input-42-afe206da09cb> in forward(self, x)
     14         h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)
     15         # Forward propagate RNN
---> 16         out, _ = self.gru(x, h0)
     17         out=out.reshape(out.shape[0], -1)
     18         out = self.fc(out)

~\AppData\Roaming\Python\Python37\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~\AppData\Roaming\Python\Python37\site-packages\torch\nn\modules\rnn.py in forward(self, input, hx)
    711             hx = self.permute_hidden(hx, sorted_indices)
    712
--> 713         self.check_forward_args(input, hx, batch_sizes)
    714         if batch_sizes is None:
    715             result = _VF.gru(input, hx, self._flat_weights, self.bias, self.num_layers,

~\AppData\Roaming\Python\Python37\site-packages\torch\nn\modules\rnn.py in check_forward_args(self, input, hidden, batch_sizes)
    178     def check_forward_args(self, input, hidden, batch_sizes):
    179         # type: (Tensor, Tensor, Optional[Tensor]) -> None
--> 180         self.check_input(input, batch_sizes)
    181         expected_hidden_size = self.get_expected_hidden_size(input, batch_sizes)
    182

~\AppData\Roaming\Python\Python37\site-packages\torch\nn\modules\rnn.py in check_input(self, input, batch_sizes)
    153             raise RuntimeError(
    154                 'input must have {} dimensions, got {}'.format(
--> 155                     expected_input_dim, input.dim()))
    156         if self.input_size != input.size(-1):
    157             raise RuntimeError(

RuntimeError: input must have 3 dimensions, got 2

i also tried to build my model using  keras and works fine
model = Sequential()
model.add(GRU(units=64, return_sequences=True,recurrent_regularizer=l2(0.0015), input_shape=(timesteps, input_dim), activation="tanh"))
model.add(GRU(units=64, recurrent_regularizer=l2(0.0015), input_shape=(timesteps, input_dim), activation="tanh"))
model.add(Dropout(0.5))

model.add(Dense(64, activation='tanh'))
model.add(Dense(64, activation='tanh'))

model.add(Dense(n_classes, activation='softmax'))
model.summary()

model.compile(optimizer=Adam(learning_rate = 0.0025), loss = 'sparse_categorical_crossentropy', metrics = ['accuracy'])

history =model.fit(X_train, y_train, batch_size=32, epochs=100)

I tried to use the same logic here meaning that I multiple the sequence_length with the input_size but still cannot solve the problem


title: Why is data augmentation degrading performance for Mask-RCNN?

body:
I trained a Mask-RCNN for instance segmentation with and without data augmentation. The augmentation was simply a rotation which makes sense for the data concerned. I was very surprised that the augmentation run (dark blue) was worse that the non-augmentation run (light blue).
Since the augmentation plots see to just be shifted down but have the same shape I was wondering if there is something else at play. I am using a batch size of 2 and the dataset have 40K images, could that affect things?


I am trying to implement pretrained model of following repository. I need your assistance to rectify the error.
RuntimeError: unexpected EOF, expected 3302200 more bytes. The file might be corrupted.
I tried to implement pretrained model of CANNet present on following repo using google Collab and followed all steps of (Prerequisites, cloning, Data Preparation, and Testing)
https://github.com/gjy3035/NWPU-Crowd-Sample-Code.git

The detailed error is given below
Traceback (most recent call last):
File "test.py", line 118, in
main()
File "test.py", line 46, in main
test(lines, model_path)
File "test.py", line 55, in test
net.load_state_dict(torch.load(model_path))
File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 593, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 779, in _legacy_load
deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
RuntimeError: unexpected EOF, expected 3302200 more bytes. The file might be corrupted.


title: Concatenation of unequal dimension tensors in tensorflow

body:
Am new to tensorflow and would like to convert a pytorch code - into tensorflow code.. How to do i perform it?

X_train, Y_train = Variable(torch.Tensor(X[:-30])), Variable(torch.Tensor(Y[:-30]))
X_test, Y_test = Variable(torch.Tensor(X[-30:])), Variable(torch.Tensor(Y[-30:]))

X_train = X_train.view(X_train.size(0), -1)
Y_train = Y_train.view(Y_train.size(0), -1)

data = torch.cat([X_train, Y_train], dim=1)
print(type(data))


title: Preprocessing in image recognition

body:
I am a beginner in image recognition and need some help about preprocessing images.

I use transfer learning model resnet18 to do the recognition work. And I get:

    In [3]: pretrainedmodels.pretrained_settings['resnet18']
    Out[3]:
    {'imagenet': {'url':
    'https://download.pytorch.org/models/resnet18-
       5c106cde.pth',
      'input_space': 'RGB',
      'input_size': [3, 224, 224],
      'input_range': [0, 1],
      'mean': [0.485, 0.456, 0.406],
      'std': [0.229, 0.224, 0.225],
      'num_classes': 1000}}

I find that the mean and std is quite different from my image dataset's.

How should I normalize my trainset? Use the mean and std above or use the mean and std I calculate myself?

I divided my dataset into train_set, valid_set and test_set.

I have two methods:

A.calculate their mean and std and normalize them individually

B.calculate the whole dataset's mean and std and then do normalization.

Which one is right?

3.When should i do normalization? Before data_augmentation or after data_augmentation?


title: How can I convert an array containing arrays of different lengths into a tensor?

body:
How can I convert an array containing arrays of different lengths into a pytorch tensor? e.g.

devX = array([
              array([1,2,3,4]),
              array([1,2,3])
              array([1,2,3,4,5])
            ])


title: Does data = batch['data'].cuda().function().cpu() make sense?

body:
I have a dataset, which I call with batch['data'] and get my image output MxM. After I get my image I want to process it with some numpy operations. In this process I want my dataset to give me the image with GPU and changing the outputs device to CPU after that.
My question is, is concetanation of functions in Python being executed in an order? And can I make this process with
base = batch['data'].cuda().function().cpu()

And is this the same as:
base = batch['data'].cuda().function()
base.cpu()

Thanks in advance!


title: How can I set precision when printing a PyTorch tensor with integers?

body:
I have:

mask = mask_model(input_spectrogram)
mask = torch.round(mask).float()
torch.set_printoptions(precision=4)
print(mask.size(), input_spectrogram.size(), masked_input.size())
print(mask[0][-40:], mask[0].size())

This prints:

tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0.], grad_fn=<SliceBackward>) torch.Size([400])

But I want it to print 1.0000 instead of 1.. Why won't my set_precision do it? Even when I converted to float()?


title: Runtime error, shape is invalid for input

body:
i am trying to get the input and output information of a network. When debugging, i got this error, Runtime, shape ‘[-1, 400]’ is invalid for input of size 384. I tried different values, but can’t find the correct value. Is there a way to solve this issue? Thanks.

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16*5*5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16*5*5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

input_shape = (3, 21,21)
        dummy_input = torch.randn(6,*input_shape)
        graph = torch.jit._get_trace_graph(model,  args=dummy_input, _force_outplace=False, _return_inputs_states=False)

Error message:

RuntimeError: shape '[-1, 400]' is invalid for input of size 384


title: Multidimensional random draw without replacement with 'predrawn' samples in pytorch

body:
I have an (N, I) tensor of N rows with I indices beween 0 and Z, e.g.,
N=5, I=3, Z=100:

foo = tensor([[83,  5, 85],
              [ 7, 60, 66],
              [89, 25, 63],
              [58, 67, 47],
              [12, 46, 40]], device='cuda:0')

Now I want to efficiently add X random additional new indices (i.e., not yet included in the tensor!) between 0 and Z to the tensor, e.g.:

foo_new = tensor([[83,  5, 85,  9, 43, 53, 42],
                  [ 7, 60, 66, 85, 64, 22,  1],
                  [89, 25, 63, 38, 24,  4, 75],
                  [58, 67, 47, 83, 43, 29, 55],
                  [12, 46, 40, 74, 21, 11, 52]], device='cuda:0')

The tensor would in the end have in each row I+X unique indices between 0 and Z, where I indices are the ones from the initial tensor, and X indices are uniform randomly drawn without replacement from the remaining indices {0...Z}\{I(n)}, where {I(n)} are the inidices of the nth row.

So it's like a multidimensional random draw without replacement from indices 0 to Z, where the first I draws (in each row) are enforced to result in the indices given by the initial tensor.

How would I do this efficiently, especially with potentially large Z?

What I tried so far (which was quite slow):

device = torch.cuda.current_device()
notinfoo = torch.ones((N,I), device=device).byte()
N_row = torch.arange(N, device=device).unsqueeze(dim=-1)
notinfoo[N_row, foo] = 0
foo_new = torch.stack([torch.cat((f, torch.arange(Z, device=device)[nf][torch.randperm(Z-I, device=device)][:X])) for f,nf in zip(foo,notinfoo)])


title: Converting pytorch model to onnx model on 5-dimensional failed

body:
1.Below picture is the result of a transformation from pth to onnx who are shown in Neutron:

Left is .pth and right is onnx. My network is based on nnUNet so that the input is 5D which is the size of a patch. I figured that the weights in above 2 are also different, which proved that I failed to convert the model. Could someone please help me? I asked on the github repo of onnx and then someone told me converting on multi-dimensional input is not supported by onnx.
2.Below is the script that I used to make this convert run.
img = torch.zeros([1, 1, 96, 160, 160]).to("cuda")
torch.onnx.export(self.network, (img),
                      '/home/qiao/nnUNetFrame/nnUNet/tensorrt/model_final_checkpoint.onnx',
                      input_names=["data"], output_names=["prediction"], verbose=True, opset_version=11,
                      operator_export_type=torch.onnx.OperatorExportTypes.ONNX
                      )
model = onnx.load(
        '/home/qiao/nnUNetFrame/nnUNet/tensorrt/model_final_checkpoint.onnx')
model_simp, check = simplify(model)
assert check, "Simplified ONNX model could not be validated"
onnx.save(model_simp,
              '/home/qiao/nnUNetFrame/nnUNet/tensorrt/model_final_checkpoint.onnx')


title: Fail in PyInstaller to get source code of TorchScript for windows executable file

body:
My question is similar to this, however, the solutions provided for that question did not solve my problem.
I am using:

torch                     1.5.1+cpu
torchvision               0.6.1+cpu
pyinstaller               4.0
pyinstaller-hooks-contrib 2020.7

I ran various commands for Pyinstaller but none of them worked.
Please give any information that may help to solve this problem.
Thank you.
d:\add_yolov3\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py:493: MatplotlibDeprecationWarning:
The MATPLOTLIBDATA environment variable was deprecated in Matplotlib 3.1 and will be removed in 3.3.
  exec(bytecode, module.__dict__)
Traceback (most recent call last):
  File "torch\_utils_internal.py", line 46, in get_source_lines_and_file
    filename = None  # in case getsourcefile throws
  File "inspect.py", line 955, in getsourcelines
  File "inspect.py", line 786, in findsource
OSError: could not get source code

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 4, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "d:\add_yolov3\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "gui\guiUtils.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "d:\add_yolov3\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "detect.py", line 5, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "d:\add_yolov3\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "models.py", line 5, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "d:\add_yolov3\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "utils\utils.py", line 14, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "d:\add_yolov3\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "torchvision\__init__.py", line 3, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "d:\add_yolov3\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "torchvision\models\__init__.py", line 12, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "d:\add_yolov3\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "torchvision\models\detection\__init__.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "d:\add_yolov3\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "torchvision\models\detection\faster_rcnn.py", line 7, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "d:\add_yolov3\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "torchvision\ops\__init__.py", line 8, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "d:\add_yolov3\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "torchvision\ops\poolers.py", line 41, in <module>
  File "torch\jit\__init__.py", line 1280, in script
    _compile_and_register_class(obj, _rcb, qualified_name)
  File "torch\jit\__init__.py", line 1107, in _compile_and_register_class
    ast = get_jit_class_def(obj, obj.__name__)
  File "torch\jit\frontend.py", line 152, in get_jit_class_def
    self_name=self_name) for method in methods]
  File "torch\jit\frontend.py", line 152, in <listcomp>
    self_name=self_name) for method in methods]
  File "torch\jit\frontend.py", line 164, in get_jit_def
    sourcelines, file_lineno, filename = get_source_lines_and_file(fn, torch._C.ErrorReport.call_stack())
  File "torch\_utils_internal.py", line 53, in get_source_lines_and_file
    print('obj:\t', obj)
OSError: Can't get source for <function LevelMapper.__call__ at 0x000001A8DF403708>. TorchScript requires source access in order to carry out compilation, make sure original .py files are available. Original error: could not get source code
[11512] Failed to execute script main


title: how to apply pyannote.audio model on my data without getting "killed"?

body:
following the tutorial Applying pretrained models on your own data gets my process killed when applying the model. RAM is not used fully, but cores go up to 100% and then go back to 0 in a few seconds.
Here's my code (the audio is 2 min, .wav):
import torch
import multiprocessing

torch.set_num_threads(multiprocessing.cpu_count())

print('downloading model')
sad = torch.hub.load('pyannote/pyannote-audio', 'sad_ami', device='cpu', batch_size=32)
print('finished')

test_file = {'uri': 'audio_real.wav', 'audio': './audio.wav'}

print('applying sad')
sad_scores = sad(test_file)
print('finished')

print('binarizing')
from pyannote.audio.utils.signal import Binarize
binarize = Binarize(offset=0.52, onset=0.52, log_scale=True,
                    min_duration_off=0.1, min_duration_on=0.1)
print('finished')

print('applying')
speech = binarize.apply(sad_scores, dimension=1)
print('finished')

print(speech)

If I apply it on audios with 5 sec it works perfectly. I've also ran that on a GCP machine with 64 cores and 64 GB of RAM, and I still got the process killed, with the last message being 'applying sad', and then "Killed".


title: Intercepting CUDA calls

body: I am trying to intercept cudaMemcpy calls from the pytorch library for analysis. I noticed NVIDIA has a cuHook example in the CUDA toolkit samples. However that example requires one to modify the source code of the application itself which I cannot do in this case. So is there a way to write a hook to intercept CUDA calls without modifying the application source code?


title: Pytorch is stuck in middle of training

body:
I have this code:
my_model.py:
num_workers = 1

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=num_workers)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=num_workers)

model_tuils.py:
 def train_network(net, number_of_epoch, trainloader, optimizer, criterion):
        for epoch in range(number_of_epoch):  # loop over the dataset multiple times

        running_loss = 0.0
        for i, data in enumerate(trainloader, 0):
            # get the inputs; data is a list of [inputs, labels]
            inputs, labels = data

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            # print statistics
            running_loss += loss.item()
            if i % 2000 == 1999:  # print every 2000 mini-batches
                print('[%d, %5d] loss: %.3f' %
                      (epoch + 1, i + 1, running_loss / 2000))
                running_loss = 0.0

    print('Finished Training')

When I run my code, it stack inside the training loop on the first iteration, in this line:
for i, data in enumerate(trainloader, 0):
            # get the inputs; data is a list of [inputs, labels]
            inputs, labels = data     <----------------------------this line

            # zero the parameter gradients
            optimizer.zero_grad()

When I try to find what is sticking the program, I get to this file:
/Users/user/.pyenv/versions/3.7.8/lib/python3.7/multiprocessing/queues.py:
 def _feed(buffer, notempty, send_bytes, writelock, close, ignore_epipe,
              onerror, queue_sem):
        debug('starting thread to feed data to pipe')
        nacquire = notempty.acquire
        nrelease = notempty.release
        nwait = notempty.wait
        bpopleft = buffer.popleft
        sentinel = _sentinel
        if sys.platform != 'win32':
            wacquire = writelock.acquire
            wrelease = writelock.release
        else:
            wacquire = None

        while 1:
            try:
                nacquire()
                try:
                    if not buffer:
                        nwait() <------------------------------ This line
                finally:
                    nrelease()

What am i doing wrong?
My num_workers is 1 so it should not have multiple threads


title: How do I have my interactive job and my submission job in condor match 100%?

body:
I want to have my background submission script to be exactly the same as my interactive job. I don't know why I can't make them the same in Condor. I tried:
getenv = True

as suggested here How do I have condor automatically import my conda environment when running my python jobs? (funny I asked something similar before!).
The issue I am running into is that my pytorch script works on interactive mode but it doens't work on submission job. The jobs look the same to me (same pytorch version, same environments, same cuda version) but I can't have them both work. Only interactive job works.
Is there something besides
getenv = True

that I can do for them to work the same?

btw, I can't even run module load to make sure the right cuda versions are loaded on the submission script, it seems the background script really starts from "scratch" and doesn't let me do anything even with that flag. But the interactive seems to be fine.

I also noticed that my PATH's are not the same in the login, interactive and submission job. See:

# export PATH=/home/miranda9/miniconda3/envs/automl-meta-learning/bin:/home/miranda9/miniconda3/condabin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/miranda9/my_bins:/home/miranda9/bin
# export PATH=/usr/local/cuda/bin:/home/miranda9/miniconda3/envs/automl-meta-learning/bin:/home/miranda9/miniconda3/condabin:/usr/local/cuda/bin:/usr/local/bin:/usr/bin:/home/miranda9/my_bins:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/miranda9/my_bins:/home/miranda9/bin
export PATH=/usr/local/cuda/bin:/home/miranda9/miniconda3/envs/metalearning/bin:/home/miranda9/miniconda3/condabin:/usr/local/cuda/bin:/usr/local/bin:/usr/bin:/home/miranda9/my_bins:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/miranda9/my_bins:/home/miranda9/bin

cross-posted:

https://www.quora.com/unanswered/How-do-I-have-my-interactive-job-and-my-submission-job-in-Condor-match-100

In case it was a pytorch issue I found these links (but since it runs in interactive session I doubt it's a pytorch issue):

https://github.com/pytorch/pytorch/issues/54550#issuecomment-806244323

This is my current submission job.sub script:
####################
#
# Experiments script
# Simple HTCondor submit description file
#
# reference: https://gitlab.engr.illinois.edu/Vision/vision-gpu-servers/-/wikis/HTCondor-user-guide#submit-jobs
#
# chmod a+x test_condor.py
# chmod a+x experiments_meta_model_optimization.py
# chmod a+x meta_learning_experiments_submission.py
# chmod a+x download_miniImagenet.py
# chmod a+x ~/meta-learning-lstm-pytorch/main.py
# chmod a+x /home/miranda9/automl-meta-learning/automl-proj/meta_learning/datasets/rand_fc_nn_vec_mu_ls_gen.py
# chmod a+x /home/miranda9/automl-meta-learning/automl-proj/experiments/meta_learning/supervised_experiments_submission.py
# chmod a+x /home/miranda9/automl-meta-learning/results_plots/is_rapid_learning_real.py
# chmod a+x /home/miranda9/automl-meta-learning/test_condor.py
# chmod a+x /home/miranda9/ML4Coq/main.sh
# chmod a+x /home/miranda9/ML4Coq/ml4coq-proj/PosEval/download_data.py
# chmod a+x /home/miranda9/ML4Coq/ml4coq-proj/pos_eval/create_pos_eval_dataset.sh
# chmod a+x /home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py
# chmod a+x /home/miranda9/ML4Coq/main.sh
# condor_submit -i
# condor_submit job.sub
#
####################

# Executable = /home/miranda9/automl-meta-learning/automl-proj/experiments/meta_learning/supervised_experiments_submission.py

# Executable = /home/miranda9/automl-meta-learning/automl-proj/experiments/meta_learning/meta_learning_experiments_submission.py
# SUBMIT_FILE = meta_learning_experiments_submission.py

# Executable = /home/miranda9/meta-learning-lstm-pytorch/main.py
# Executable = /home/miranda9/automl-meta-learning/automl-proj/meta_learning/datasets/rand_fc_nn_vec_mu_ls_gen.py

# Executable = /home/miranda9/automl-meta-learning/results_plots/is_rapid_learning_real.py
# SUBMIT_FILE = is_rapid_learning_real.py

# Executable = /home/miranda9/automl-meta-learning/test_condor.py

# Executable = /home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py
# SUBMIT_FILE = main_brando.py

# Executable = /home/miranda9/ML4Coq/ml4coq-proj/PosEval/download_data.py
# SUBMIT_FILE = ml4coq-proj/PosEval/download_data.py

# Executable = /home/miranda9/ML4Coq/ml4coq-proj/pos_eval/create_pos_eval_dataset.sh
# SUBMIT_FILE = create_pos_eval_dataset.sh

Executable = /home/miranda9/ML4Coq/main.sh
SUBMIT_FILE = main.sh

# Output Files
Log          = $(SUBMIT_FILE).log$(CLUSTER)
Output       = $(SUBMIT_FILE).o$(CLUSTER)
Error        = $(SUBMIT_FILE).o$(CLUSTER)

getenv = True
# cuda_version = 10.2
# cuda_version = 11.0

# Use this to make sure 1 gpu is available. The key words are case insensitive.
# REquest_gpus = 1
REquest_gpus = 2
requirements = (CUDADeviceName != "Tesla K40m")
requirements = (CUDADeviceName != "GeForce GTX TITAN X")
# requirements = (CUDADeviceName == "Quadro RTX 6000")
# requirements = ((CUDADeviceName != "Tesla K40m")) && (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory) && (TARGET.Cpus >= RequestCpus) && (TARGET.gpus >= Requestgpus) && ((TARGET.FileSystemDomain == MY.FileSystemDomain) || (TARGET.HasFileTransfer))
# requirements = (CUDADeviceName == "Tesla K40m")
# requirements = (CUDADeviceName == "GeForce GTX TITAN X")

# Note: to use multiple CPUs instead of the default (one CPU), use request_cpus as well
# Request_cpus = 1
Request_cpus = 4
# Request_cpus = 5
# Request_cpus = 8
# Request_cpus = 16
# Request_cpus = 32

# E-mail option
Notify_user = me@email.com
Notification = always

Environment = MY_CONDOR_JOB_ID= $(CLUSTER)

# "Queue" means add the setup until this line to the queue (needs to be at the end of script).
Queue

and the main.sh script:
#!/bin/bash

echo JOB STARTED

# export PATH=/home/miranda9/miniconda3/envs/automl-meta-learning/bin:/home/miranda9/miniconda3/condabin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/miranda9/my_bins:/home/miranda9/bin
# export PATH=/usr/local/cuda/bin:/home/miranda9/miniconda3/envs/automl-meta-learning/bin:/home/miranda9/miniconda3/condabin:/usr/local/cuda/bin:/usr/local/bin:/usr/bin:/home/miranda9/my_bins:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/miranda9/my_bins:/home/miranda9/bin
export PATH=/usr/local/cuda/bin:/home/miranda9/miniconda3/envs/metalearning/bin:/home/miranda9/miniconda3/condabin:/usr/local/cuda/bin:/usr/local/bin:/usr/bin:/home/miranda9/my_bins:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/miranda9/my_bins:/home/miranda9/bin
# module load cuda-toolkit/10.2
# module load cuda-toolkit/11.1

# echo $PATH
# nvidia-smi
# conda list
which python

# - run script
python ~/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py
# python ~/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py --debug --num_epochs 5 --batch_size 2 --term_encoder_embedding_dim 8

echo JOB ENDED


I am getting the following error.

File "/net/if5/wua4nw/wasi/academic/research_with_prof_chang/projects/question_answering/duplicate_question_detection/source/train.py", line 62, in train
    loss.backward()
  File "/if5/wua4nw/anaconda3/lib/python3.5/site-packages/torch/autograd/variable.py", line 145, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)
  File "/if5/wua4nw/anaconda3/lib/python3.5/site-packages/torch/autograd/function.py", line 208, in _do_backward
    result = super(NestedIOFunction, self)._do_backward(gradients, retain_variables)
  File "/if5/wua4nw/anaconda3/lib/python3.5/site-packages/torch/autograd/function.py", line 216, in backward
    result = self.backward_extended(*nested_gradients)
  File "/if5/wua4nw/anaconda3/lib/python3.5/site-packages/torch/nn/_functions/rnn.py", line 210, in backward_extended
    grad_hx)
  File "/if5/wua4nw/anaconda3/lib/python3.5/site-packages/torch/backends/cudnn/rnn.py", line 360, in backward_grad
    raise RuntimeError('Gradients aren\'t CUDA tensors')
RuntimeError: Gradients aren't CUDA tensors

while running loss.backward() in pytorch. Forward propagation is working fine but when the backpropagation step executes, I am getting the error. Can anyone suggest me how to solve the problem?

P.S: I am not posting the source code since its too long!


title: PyTorch: Render point cloud without CPU transfer?

body:
I have a some data in pytorch on the GPU, say,

x = torch.cuda.FloatTensor(500, 3).uniform_()

Is there any way to render this as a point cloud (ala https://matplotlib.org/gallery/mplot3d/scatter3d.html) without first transferring the data to the CPU?

I have looked at frameworks such as matplotlib, VisPy, mayavi, etc. But all of them seem to want data via the CPU and cannot handle a GPU pointer, say.

Does anyone know of a good method?
Preferably, the plotting should be non-blocking such that I can have a live view of x during training. Any ideas?


title: Output of a CNN doesn't change much with the input

body:
I have been trying to implement Actor Critic with a convolutional neural network. There are two different images as the state for the reinforcement learning agent. The output(actions for Actor) of the CNN is the (almost)same for different inputs after (random)initialisation. Due to this even after training the agent never learns anything useful.

State definitions(2 inputs):

Input 1: [1,1,90,128] image with maximum value for the pixel being 45.

Input 2: [1,1,45,80] image with maximum value for the pixel being 45.

Expected output by the actor: [x,y]: a 2-dimensional vector according to the state. Here x is expected in the range [0,160] and y is expected in the range [0,112]

Different types of modifications tried with the input:

1: Feed both the images as it is.

2: Feed both images with normalisation as (img/45) so that pixel values are from [0,1]

3:Feed both images with normalisation as 2*((img/45)-0.5) so that pixel values are from [-1,1]

4: Feed both images with normalisation as (img-mean)/std

Result: The output of the CNN remains almost same.

The code for the definition of actor has been given below.

import numpy as np
import pandas as pd
from tqdm import tqdm
import time
import cv2
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.nn.functional as F

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

class Actor(nn.Module):
    def __init__(self, action_dim, max_action):
        super(Actor,self).__init__()
        # state image [1,1,90,128]
        self.conv11 = nn.Conv2d(1,16,5)
        self.conv11_bn = nn.BatchNorm2d(16)
        self.conv12 = nn.Conv2d(16,16,5)
        self.conv12_bn = nn.BatchNorm2d(16)
        self.fc11 = nn.Linear(19*29*16,500)
        # dim image [1,1,45,80]
        self.conv21 = nn.Conv2d(1,16,5)
        self.conv21_bn = nn.BatchNorm2d(16)
        self.conv22 = nn.Conv2d(16,16,5)
        self.conv2_bn = nn.BatchNorm2d(16)
        self.fc21 = nn.Linear(8*17*16,250)
        # common pool
        self.pool  = nn.MaxPool2d(2,2)
        # after concatenation
        self.fc2 = nn.Linear(750,100)
        self.fc3 = nn.Linear(100,10)
        self.fc4 = nn.Linear(10,action_dim)
        self.max_action = max_action

    def forward(self,x,y):
        # state image
        x = self.conv11_bn(self.pool(F.relu(self.conv11(x))))
        x = self.conv11_bn(self.pool(F.relu(self.conv12(x))))
        x = x.view(-1,19*29*16)
        x = F.relu(self.fc11(x))
        # state dim
        y = self.conv11_bn(self.pool(F.relu(self.conv21(y))))
        y = self.conv11_bn(self.pool(F.relu(self.conv22(y))))
        y = y.view(-1,8*17*16)
        y = F.relu(self.fc21(y))
        # concatenate
        z = torch.cat((x,y),dim=1)
        z = F.relu(self.fc2(z))
        z = F.relu(self.fc3(z))
        z = self.max_action*torch.tanh(self.fc4(z))
        return z

# to read different sample states for testing
obs = []
for i in range(200):
    obs.append(np.load('eval_episodes/obs_'+str(i)+'.npy',allow_pickle=True))

obs = np.array(obs)

def tensor_from_numpy(state):
    # to add dimensions to tensor to make it [batch_size,channels,height,width]
    state_img = state
    state_img = torch.from_numpy(state_img).float()
    state_img = state_img[np.newaxis, :]
    state_img = state_img[np.newaxis, :].to(device)
    return state_img

actor = Actor(2,torch.FloatTensor([160,112]))
for i in range(20):
    a = tensor_from_numpy(obs[i][0])
    b = tensor_from_numpy(obs[i][2])
    print(actor(a,b))

Output of the above code:

tensor([[28.8616,  3.0934]], grad_fn=<MulBackward0>)
tensor([[27.4125,  3.2864]], grad_fn=<MulBackward0>)
tensor([[28.2210,  2.6859]], grad_fn=<MulBackward0>)
tensor([[27.6312,  3.9528]], grad_fn=<MulBackward0>)
tensor([[25.9290,  4.2942]], grad_fn=<MulBackward0>)
tensor([[26.9652,  4.5730]], grad_fn=<MulBackward0>)
tensor([[27.1342,  2.9612]], grad_fn=<MulBackward0>)
tensor([[27.6494,  4.2218]], grad_fn=<MulBackward0>)
tensor([[27.3122,  1.9945]], grad_fn=<MulBackward0>)
tensor([[29.6915,  1.9938]], grad_fn=<MulBackward0>)
tensor([[28.2001,  2.5967]], grad_fn=<MulBackward0>)
tensor([[26.8502,  4.4917]], grad_fn=<MulBackward0>)
tensor([[28.6489,  3.2022]], grad_fn=<MulBackward0>)
tensor([[28.1455,  2.7610]], grad_fn=<MulBackward0>)
tensor([[27.2369,  3.4243]], grad_fn=<MulBackward0>)
tensor([[25.9513,  5.3057]], grad_fn=<MulBackward0>)
tensor([[28.1400,  3.3242]], grad_fn=<MulBackward0>)
tensor([[28.2049,  2.6622]], grad_fn=<MulBackward0>)
tensor([[26.7446,  2.5966]], grad_fn=<MulBackward0>)
tensor([[25.3867,  5.0346]], grad_fn=<MulBackward0>)

The states(.npy) files can be found here
For different states the actions should vary from [0-160,0-112] but here the outputs just vary a little bit.

Note: the input images are initially sparse(lots of zeros in the image)

Is there a problem with the state pixel values or the network definition?

EDIT: I think the problem has got to do something with the normalisation or the sparsity of the inputs, because I also tried the same network with tensorflow and am facing the same problem there.


title: Numerical equivalence of PyTorch backpropagation

body:
After i 'v written the simple neural network with numpy, i wanted to compare it numerically with PyTorch impementation. Running alone, seems my neural network implementation converges, so it seems to have no errors.
Also i v checked forward pass matches to PyTorch, so basic setup is correct.

But something different happens while backward pass, because the weights after one backpropagation are different.

I dont want to post full code here because its linked over several .py files, and most of the code is irrelevant to the question. I just want to know does PyTorch "basic" gradient descent or something different.

I m viewing the most simle example about full-connected weights of the last layer, cause if it is different, further will be also different:

self.weight +=  self.learning_rate * hidden_layer.T.dot(output_delta )

where

output_delta  = self.expected -  self.output

self.expected  are expected value,
self.output is forward pass result

No activation or further  stuff here.

The torch past is:

optimizer = torch.optim.SGD(nn.parameters() , lr = 1.0)

criterion = torch.nn.MSELoss(reduction='sum')

output = nn.forward(x_train)

loss = criterion(output, y_train)

loss.backward()

optimizer.step()

optimizer.zero_grad()

So it is possible that with SGD optimizer and MSELoss it uses some different delta or backpropagation function, not the basic one mentioned above? If its so i d like to know how to numerically check my numpy solution with pytorch.


title: Implement for transforming rotation matrix to rotation vector in pytorch

body:
I want to implement the function for rotation matrix to rotation vector in pytorch. I have read the document in opencv tutorial Rodrigues and refer the code in c Rodrigues in c. The transformation for rotation vector to rotation matrix is straightforward according to the formulation, but the reverse is obscure and undirect.
Is there any implement for the code in pytorch or any guide for implement this by myself?


title: Comparing two images with ms_ssim on python

body:
I want to compare two images but ms_ssim wants 4D tensor

https://pypi.org/project/pytorch-msssim/

I Tried

from PIL import Image
from tqdm import tqdm
from pytorch_msssim import ssim, ms_ssim, SSIM, MS_SSIM
import torchvision
import numpy as np

topil=torchvision.transforms.ToPILImage()
totensor=torchvision.transforms.ToTensor()

def ssimcompare(path1:str,path2:str)->float:
    image1 = Image.open(path1)
    image2 = Image.open(path2)
    #it1=np.expand_dims(totensor(topil(np.array(image1))), axis=0)
    #it2=np.expand_dims(totensor(topil(np.array(image2))), axis=0)
    #it1=totensor(np.expand_dims(np.array(image1), axis=0))
    #it2=totensor(np.expand_dims(np.array(image2), axis=0))
    it1=totensor(np.array(image1))
    it2=totensor(np.array(image2))
    valor=ms_ssim( it1 , it2, data_range=255, size_average=False )
    return valor

But I get different errors

ValueError: Input images must be 4-d tensors.
TypeError: pic should be Tensor or ndarray. Got <class 'PIL.JpegImagePlugin.JpegImageFile'>.
AttributeError: 'numpy.ndarray' object has no attribute 'type'


title: Where is torch.cholesky and how torch refers to its methods?

body:
I'm doing some research into Cholesky decomposition, which requires some insights into how torch.cholesky works. After a while of grep-ing and searching through ATen, I got stuck at TensorMethods.h, which interestingly has this following code:

inline Tensor Tensor::cholesky(bool upper) const {
#ifdef USE_STATIC_DISPATCH
    return TypeDefault::cholesky(const_cast<Tensor&>(*this), upper);
#else
    static c10::OperatorHandle op = c10::Dispatcher::singleton().findSchema({"aten::cholesky", ""}).value();
    return c10::Dispatcher::singleton().callUnboxed<Tensor, const Tensor &, bool>(
        op, impl::dispatchTypeId(at::detail::multi_dispatch_tensor_type_set(*this)), const_cast<Tensor&>(*this), upper);
#endif
}

This raised the question of how torch locates its methods. Thank you!


title: Compute grads of cloned tensor Pytorch

body:
I am having a hard time with gradient computation using PyTorch.

I have the outputs and the hidden states of the last time step T of an RNN.

I would like to clone my hidden states and compute its grad after backpropagation but it doesn't work.

After reading pytorch how to compute grad after clone a tensor, I used  retain_grad() without any success.

Here's my code

        hidden_copy = hidden.clone()
        hidden.retain_grad()
        hidden_copy.retain_grad()
        outputs_T = outputs[T]
        targets_T = targets[T]
        loss_T = loss(outputs_T,targets_T)
        loss_T.backward()
        print(hidden.grad)
        print(hidden_copy.grad)

hidden_grad gives an array while hidden_copy.grad gives None.

Why does hidden_copy.grad give None ? Is there any way to compute the gradients of a cloned tensor ?


title: How to vectorize matrix inversion while handling runtime error in pytorch

body:
I need to invert some matrices in pytorch. However, some of the matrices are not invertible, which leads to the code throwing runtime error as follows,

matrices = torch.randn([5,3,3])
matrices[[2,3]] = torch.zeros([3,3])
inverses = torch.inverse(matrices)

RuntimeError: inverse_cpu: For batch 2: U(1,1) is zero, singular U.

I have a fallback technique for such situations. However, I can't figure out which of the matrices throw the error. Currently, I have replaced the code with non-vectorized version, but it has become a bottleneck.

Is there a way to handle this without giving up vectorization?


title: Why does the BERT NSP head linear layer have two outputs?

body:
Here's the code in question.

https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491

class BertOnlyNSPHead(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.seq_relationship = nn.Linear(config.hidden_size, 2)

    def forward(self, pooled_output):
        seq_relationship_score = self.seq_relationship(pooled_output)
        return seq_relationship_score

I think it was just ranking how likely one sentence would follow another? Wouldn't it be one score?


title: How to convert Pytorch model to ONNX model?

body:
I have a model that contains two-part  encoder and decoder.
Each of them is a PyTorch model.

I inference them as below.

features = encoder(input)
output = decoder(features)

and I want to convert them into one ONNX model instead of two ONNX models.
How can I do it?


title: PyTorch Deconvolution Layer input dimension mismatch with Linear Layer

body:
I built an autoencoder with some convolutional layers, a fully connected layer in the middle and some deconvolutional layers. I use Atari Frames as input (210x160x3 dimension). My problem at the moment is that I can only process squared images (same side length on all sides). When I use the original side lengths this error occurs:

UserWarning: Using a target size (torch.Size([30, 3, 210, 160])) that is different to the input size (torch.Size([30, 3, 158, 158])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)

I think the problem originates in the change from linear to the deconv Layer, because the tensor size always gets quadratic:

Linear: torch.Size([30, 1000])
Deconv: torch.Size([30, 1000, 1, 1])

However, I want a torch.Size with the shape like: torch.Size([30,1000,11,8]) in the deconv layer.

I am pretty sure there is a simple solution but I didn't find it yet.

Code snippet of the forward function:

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        x = F.relu(self.conv4(x))

        x = x.view(x.size(0), -1)
        x = torch.sigmoid(self.lv(x))
        x = x.unsqueeze(-1).unsqueeze(-1)

        x = F.relu(self.deconv1(x))
        x = F.relu(self.deconv2(x))
        x = F.relu(self.deconv3(x))
        x = F.relu(self.deconv4(x))
        x = F.relu(self.deconv5(x))
        return x

Thank you so much in advance!


title: Can I make my custom pytorch modules behave differently when train() or eval() are called?

body: According to the official documents, using train() or eval() will have effects on certain modules. However, now I wish to achieve a similar thing with my custom module, i.e. it does something when train() is turned on, and something different when eval() is turned on. How can I do this?


title: Is it normal that accuracy decreases when advancing to the next epoch?

body:
I’m training a CNN to predict digits using the MNIST database. I’m doing Data Augmentation and for some reason accuracy sharply decreases when advancing to next epoch (iteration 60 in the image)

It has to do with data augmentation (transform = my_transforms in the code) because when I deactivate augmentation (transform = None) accuracy doesn't decrease when advancing to next epoch. But I can't explain why. Does anyone have an idea why this happens?
my_transforms = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomCrop((25,25)),
    transforms.Resize((28,28)),
    transforms.RandomRotation(degrees=45, fill=255),
    transforms.RandomVerticalFlip(p=0.1),
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.ToTensor()
    ])

dataset = MNISTDataset(transform = my_transforms)
train_loader = DataLoader(dataset = dataset, batch_size = 1000, shuffle=True)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1,10,kernel_size=5)
        self.pool = nn.MaxPool2d(kernel_size=2,stride=2)
        self.conv2 = nn.Conv2d(10,20,kernel_size=5)
        self.fc1 = nn.Linear(20*4*4, 64)
        self.fc2 = nn.Linear(64, 10)

def forward(self,x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1,20*4*4)
        x = F.relu(self.fc1(x))
        x = F.softmax(self.fc2(x), dim=1)

        return x

net = Net()

loss_function=nn.NLLLoss()
optimizer=optim.Adam(net.parameters())

EPOCHS=2
iteracion = 0

for epoch in range(EPOCHS):
    for data in train_loader:
        inputs, labels = data
        inputs = inputs.view(-1,1,28,28)

        net.zero_grad()

        probabilities=net(inputs)

        matches=[torch.argmax(i)==int(j) for i,j in zip(probabilities,labels)]

        in_batch_acc=matches.count(True)/len(matches)

        loss=loss_function(torch.log(probabilities), labels)

        print('Loss:', round(float(loss), 3))
        print('In-batch acc:', round(in_batch_acc, 2))

        iteracion += 1

        loss.backward()
        optimizer.step()


title: Word-sense disambiguation based on sets of words using pre-trained embeddings

body:
I am interested in identifying the WordNet synset IDs for each word in a set of tags.
The words in the set provide the context for the word sense disambiguation, such as:

{mole, skin}
{mole, grass, fur}
{mole, chemistry}
{bank, river, river bank}
{bank, money, building}

I know of the lesk algorithm and libraries, such as pywsd, which is based on 10+ year old tech (which may still be cutting edge -- that is my question).
Are there better performing algorithms by now that make sense of pre-trained embeddings, like GloVe, and maybe the distances of these embeddings to each other?
Are there ready-to-use implementations of such WSD algorithms?
I know this question is close to the danger zone of asking for subjective preferences - as in this 5-year old thread. But I am not asking for an overview of options or the best software for a problem.


title: BERT Binary Text Classifier giving ValueError: Expected input batch_size doesn`t match target

body:
I am working on binary text classification problem and using Bert sequence classification model in pytorch. Here is the link to colab notebook.
After training the model, I am trying to predict on sample text. I have checked the shape of the input_id tensor it is [1,128]. I have used batch_size = 16.
review_text = "I love completing my todos! Best app ever!!!"

encoded_review = tokenizer.encode_plus(
  review_text,
  max_length=MAX_SEQ_LEN,
  add_special_tokens=True,
  return_token_type_ids=False,
  pad_to_max_length=True,
  return_attention_mask=True,
  return_tensors='pt',
)
input_ids = encoded_review['input_ids'].to(device)
print(input_ids.shape)
attention_mask = encoded_review['attention_mask'].to(device)
print(attention_mask.shape)
output = model(input_ids, attention_mask)
output

This produces the error
Truncation was not explicitely activated but `max_length` is provided a specific value, please use `truncation=True` to explicitely truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
torch.Size([1, 128])
torch.Size([1, 128])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-22-74eeba339c4c> in <module>()
     14 attention_mask = encoded_review['attention_mask'].to(device)
     15 print(attention_mask.shape)
---> 16 output = model(input_ids, attention_mask)
     17 output

7 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

<ipython-input-7-71a85d6c1fce> in forward(self, text, label)
      8
      9     def forward(self, text, label):
---> 10         loss, text_fea = self.encoder(text, labels=label)[:2]
     11
     12         return loss, text_fea

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states)
   1282             else:
   1283                 loss_fct = CrossEntropyLoss()
-> 1284                 loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
   1285             outputs = (loss,) + outputs
   1286

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    946     def forward(self, input: Tensor, target: Tensor) -> Tensor:
    947         return F.cross_entropy(input, target, weight=self.weight,
--> 948                                ignore_index=self.ignore_index, reduction=self.reduction)
    949
    950

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2420     if size_average is not None or reduce is not None:
   2421         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2422     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2423
   2424

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2214     if input.size(0) != target.size(0):
   2215         raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
-> 2216                          .format(input.size(0), target.size(0)))
   2217     if dim == 2:
   2218         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

ValueError: Expected input batch_size (1) to match target batch_size (128).


title: Convert .pth Pytorch model to format readable by OpenCv

body:
I'm trying to use this model: https://github.com/zllrunning/face-makeup.PyTorch/blob/master/cp/79999_iter.pth with opencv dnn module but because opencv doesn't read .pth file, I need to convert it to another format.
First I tried to convert to .onnx, I found code to do it but when I call the function "torch.onnx.export", I got "RuntimeError: ONNX symbolic expected a constant value in the trace".
I tried with differrent version of torch but I always got almost the same error message.
I tried to convert to caffe model, there is some available github to do it but none of them worked for me.
I didn't find how to convert directly to tensorflow, they always convert to onnx and then to tensorflow.
If someone can share me a code that works for this model and the version of all packages used. Or if you could share me the model converted to another format, it would be greatly appreciated.


title: How to keep track of GPU memory on google colab and free it in a proper way?

body:
I am trying to train a deep neural network (DNN) on Google Colab with the use of the PyTorch framework.
So far, I am debugging my network, and in order to do this, I reinitialize it each time.
But after doing so several times I am running out of GPU memory. The first thing to think about is to free the memory occupied by the network.
The solution, suggested here https://forums.fast.ai/t/clearing-gpu-memory-pytorch/14637 :
del model
torch.cuda.empty_cache()

Doesn't work. There is definetely option of restarting a session, but doing it each time is rather annoying, because it takes some time to load the data and process it in order to run a network.
Is there some way to clear the GPU memory surely on Google Colab, and keep track of its usage?
In the case I am running out of memory I would like to figure the reason - the exact place, where memory allocation has failed and have a look at the objects, occupying it


title: Q-value keeps stepping down when training a DQN

body:
I am training a DQN and the Q-value keeps going down. The curve looks very weird (see below).

Every step corresponds to an update to target network.
Any possible reason why this happens?


title: About custom operations in Tensorflow and PyTorch

body:
I have to implement an energy function, termed Rigidity Energy, as in Eq 7 of this paper here.
The energy function takes as input two 3D object meshes, and returns the energy between them. The first mesh is the source mesh, and the second mesh is the deformed version of the source mesh. In rough psuedo-code, the computation would go like this:

Iterate over all the vertices in the source mesh.

For every vertex, compute its covariance matrix with its neighboring vertices.
Perform SVD on the computed covariance matrix and find the rotation matrix of the vertex.
Use the computed rotation matrix, the point coordinates in the original mesh and the corresponding coordinates in the deformed mesh, to compute the energy deviation of the vertex.

Thus this energy function requires me to iterate over each point in the mesh, and the mesh could have more than 2k such points. In Tensorflow, there are two ways to do this. I can have 2 tensors of shape (N,3), one representing the points of source and the other of the deformed mesh.

Do it purely using Tensorflow tensors. That is, iterate over elements of the above tensors using tf.gather and perform the computation on each point using only existing TF operations. This method, would be extremely slow. I've tried to define loss functions that iterate over 1000s of points before, and the graph construction itself takes too much time to be practical.
Add a new TF OP as explained in the TF documentation here . This involves writing the function in CPP (and Cuda, for GPU support), and registering the new OP with TF.

The first method is easy to write, but impractically slow. The second method is a pain to write.

I've used TF for 3 years, and have never used PyTorch before, but at this point I'm considering switching to it, if it offers a better alternative for such cases.

Does PyTorch have a way of implementing such loss functions both easily and performs as fast as it would on GPU. i.e, A pythonic way of writing my own loss functions that runs on GPU, without any C or Cuda code on my part?


title: Adding neurons to Adam optimizer state in Pytorch

body:
I've posted the following to the Pytorch discussion board too. I'm trying to keep the learning-rates per parameter for the already existing parameters when adding more neurons (to existing layers, not new layers) to a network. I’ve written the following class which allows me to add neurons to hidden layers during training:

import torch
import torch.nn as nn

class DQN(nn.Module):
    def __init__(self, num_inputs, hidden, num_actions, non_linearity):
        super(DQN, self).__init__()

        self.num_inputs = num_inputs
        self.hidden = hidden
        self.num_actions = num_actions
        self.non_linearity = non_linearity

        self.layers = nn.ModuleList()
        self.layers.append(nn.Linear(num_inputs, self.hidden[0]))

        previous = self.hidden[0]
        for hidden_layer_size in self.hidden[1:]:
            self.layers.append(nn.Linear(previous, hidden_layer_size))
            previous = hidden_layer_size

        self.layers.append(nn.Linear(previous, num_actions))

    def forward(self, x):
        for i in range(len(self.layers) - 1):
            x = self.non_linearity(self.layers[i](x))
        return self.layers[-1](x)

    def increase_capacity(self, increment):
        for i in range(len(self.hidden)):
            self.hidden[i] += increment[i]

        bias = self.layers[0].bias.data
        weight = self.layers[0].weight.data
        self.layers[0] = nn.Linear(self.num_inputs, self.hidden[0])
        if increment[0]>0:
            self.layers[0].weight.data[0:-increment[0],:] = weight
            self.layers[0].bias.data[0:-increment[0]] = bias
        else:
            self.layers[0].weight.data[0:,:] = weight
            self.layers[0].weight.data = bias

        for i in range(1, len(self.layers) - 1):
            bias = self.layers[i].bias.data
            weight = self.layers[i].weight.data
            self.layers[i] = nn.Linear(self.hidden[i-1], self.hidden[i])
            if increment[i] > 0:
                if increment[i-1] >0:
                    self.layers[i].bias.data[0:-increment[i]] = bias
                    self.layers[i].weight.data[0:-increment[i],0:-increment[i-1]] = weight
                else:
                    self.layers[i].bias.data[0:-increment[i]] = bias
                    self.layers[i].weight.data[0:-increment[i],0:] = weight
            else:
                if increment[i-1] >0:
                    self.layers[i].bias.data = bias
                    self.layers[i].weight.data[0:,0:-increment[i-1]] = weight
                else:
                    self.layers[i].bias.data = bias
                    self.layers[i].weight.data[0:,0:] = weight

        bias = self.layers[-1].bias.data
        weight = self.layers[-1].weight.data
        self.layers[-1] = nn.Linear(self.hidden[-1], self.num_actions)
        if increment[-1] >0:
            self.layers[-1].bias.data = bias
            self.layers[-1].weight.data[:,0:-increment[-1]] = weight
        else:
            self.layers[-1].bias.data = bias
            self.layers[-1].weight.data[:,0:] = weight

    def act(self, state, epsilon, mask):
        if np.random.rand() > epsilon:
            state = torch.tensor([state], dtype=torch.float32, device=device)
            mask = torch.tensor([mask], dtype=torch.float32, device=device)
            q_values = self.forward(state) + mask
            action = q_values.max(1)[1].view(1, 1).item()
        else:
            action =  np.random.randint(self.num_actions)
        return action

Now I’ve written a little sanity check (whether it leads to sanity is questionable at this point): a network with 2 layers with both 1 neuron should fail to learn the x-or function, whereas a network where 4 neurons have been added should. If I initialise a new optimiser this indeed works. The optimiser I use is Adam, which keeps track of learning-rates per parameter. I’d like to keep the learning-rates of Adam for the weights and biases that already existed before I add additional neurons. The following is my failed attempt to doing so:

import random
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import numpy as np

# Credits to Alvations
def generate_zero():
    return random.uniform(0, 49) / 100

def generate_one():
    return random.uniform(50, 100) / 100

def generate_xor_XY(num_data_points):
    Xs, Ys = [], []
    for _ in range(num_data_points):
        # xor(0, 0) -> 0
        Xs.append([generate_zero(), generate_zero()]); Ys.append([0])
        # xor(1, 0) -> 1
        Xs.append([generate_one(), generate_zero()]); Ys.append([1])
        # xor(0, 1) -> 1
        Xs.append([generate_zero(), generate_one()]); Ys.append([1])
        # xor(1, 1) -> 0
        Xs.append([generate_one(), generate_one()]); Ys.append([0])
    return Xs, Ys

# Initialisation
network = DQN(2,[1,1],1,F.relu)
# optimizer = optim.Adam(network.parameters(), amsgrad=False)
optimizer = optim.Adam(network.parameters(), amsgrad=True)
criterion = nn.MSELoss()

# Train 50000 steps to show 1 neuron cannot solve x-or task
for i in range(50000):
    optimizer.zero_grad()

    Xs, Ys = generate_xor_XY(1)
    Xs = torch.tensor(Xs)
    Ys = torch.tensor(Ys, dtype=torch.float)

    prediction = network(Xs)
    loss = criterion(prediction, Ys)

    loss.backward()
    optimizer.step()

print(network(torch.tensor([[1,0],[0,1],[1,1],[0,0]], dtype=torch.float)))
print(loss)

# Add 5 neurons to first layer
capacity = [4,4]
network.increase_capacity(capacity)

# Uncomment the following line and comment the lines following it for normal initialisation.
# optimizer = optim.Adam(network.parameters(), amsgrad=True)

nw_param = [p for p in network.parameters()]
new_param_group = []

layer_idx = 0
for idx, group in enumerate(optimizer.param_groups):
        for idx_p, p in enumerate(group['params']):
            # Save previous information
            prev_grad = p.grad
            old_p = copy.deepcopy(p)
            old_state = copy.copy(optimizer.state[p])
            old_step = old_state['step']
            old_exp_avg = old_state['exp_avg']
            old_exp_avg_sq = old_state['exp_avg_sq']
            old_max_exp_avg_sq = old_state['max_exp_avg_sq']

            # Remove old parameter from state
            optimizer.state.pop(p)

            # Weights
            if p.dim()>1:
                p = nn.Parameter(nw_param[layer_idx])
                p.grad = torch.zeros_like(p)
                new_exp_avg = torch.torch.zeros_like(p)
                new_exp_avg_sq = torch.torch.zeros_like(p)
                new_max_exp_avg_sq = torch.torch.zeros_like(p)
                p.grad[0:prev_grad.size(0),0:prev_grad.size(1)] = prev_grad

                optimizer.state[p]['step'] = old_step
                optimizer.state[p]['exp_avg'] = new_exp_avg
                optimizer.state[p]['exp_avg'][0:prev_grad.size(0),0:prev_grad.size(1)] = old_exp_avg
                optimizer.state[p]['exp_avg_sq'] = new_exp_avg_sq
                optimizer.state[p]['exp_avg_sq'][0:prev_grad.size(0),0:prev_grad.size(1)] = old_exp_avg_sq
                optimizer.state[p]['max_exp_avg_sq'] = new_max_exp_avg_sq
                optimizer.state[p]['max_exp_avg_sq'][0:prev_grad.size(0),0:prev_grad.size(1)] = old_max_exp_avg_sq
                new_param_group.append(p)

            # Biases
            else:
                p = nn.Parameter(nw_param[layer_idx])
                p.grad = torch.zeros_like(p)
                new_exp_avg = torch.zeros_like(p)
                new_exp_avg_sq = torch.zeros_like(p)
                new_max_exp_avg_sq = torch.zeros_like(p)
                p.grad[0:prev_grad.size(0)] = prev_grad

                optimizer.state[p]['step'] = old_step
                optimizer.state[p]['exp_avg'] = new_exp_avg
                optimizer.state[p]['exp_avg'][0:prev_grad.size(0)] = old_exp_avg
                optimizer.state[p]['exp_avg_sq'] = new_exp_avg_sq
                optimizer.state[p]['exp_avg_sq'][0:prev_grad.size(0)] = old_exp_avg_sq
                optimizer.state[p]['max_exp_avg_sq'] = new_max_exp_avg_sq
                optimizer.state[p]['max_exp_avg_sq'][0:prev_grad.size(0)] = old_max_exp_avg_sq
                new_param_group.append(p)

            layer_idx += 1

optimizer.param_groups[0]['params'] = new_param_group

print(network)

# Train 50000 steps to show by adding neurons the task can be solved
for i in range(50000):
    optimizer.zero_grad()

    Xs, Ys = generate_xor_XY(1)
    Xs = torch.tensor(Xs)
    Ys = torch.tensor(Ys, dtype=torch.float)

    prediction = network(Xs)
    loss = criterion(prediction, Ys)

    loss.backward()
    optimizer.step()

print(network(torch.tensor([[1,0],[0,1],[1,1],[0,0]], dtype=torch.float)))
print(loss)

I’m trying to get the same optimizer state, but with additional parameters for the added neurons. This seems like a convoluted way of doing it (and it doesn’t work:p). Does anyone know of an (easier) way to do this or see where I’m going wrong?


title: How to score new data after creating fastai model on tabular data?

body:
Creating a learner with

learn.fit_one_cycle(5)

and i can check the prediction from below code on test set

preds_test, _ = learn.get_preds(ds_type=DatasetType.Test)
pred_test_prob, pred_test_class = preds_test.max(1)

How to check prediction on a new dataset which is not a databunch? I am trying below

validateObj = TabularList.from_df(validate, cat_names=cat_names, cont_names=cont_names, procs=procs)

preds, _ = learn.get_preds(validateObj)
pred_prob, pred_class = preds.max(1)

but its giving me an array of 108000 compare to 45000 validation set.

is there a simple way to score new data using above learner?


title: Pycharm complains about using [] operator with pytorch's nn.ModuleList object

body:
Pycharm's type-checking complains about the following block of code (taken from pytorch's docs):

    import torch.nn as nn
    class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.linears = nn.ModuleList([nn.Linear(10, 10) for i in range(10)])

    def forward(self, x):
        # ModuleList can act as an iterable, or be indexed using ints
        for i, l in enumerate(self.linears):
            x = self.linears[i // 2](x) + l(x)
        #                   ^---------- complains here about:
        # Cannot find reference '[' in 'input:(Any, ...), kwargs:dict) -> Any'
        return x

Is there a way to fix this issue without suppressing it?


title: Method for feeding multi-class image data-set where folders name can be used as labels in Pytorch?

body:
I want to feed the multiclass image data-set in Pytorch, in the main folder of data-set I have 15 more folders with different names, I want to use folders names as the labels.
For example, one folder name is Aeroplanes and contain the images (1245 images) other folder name is Cars and contains images of the Cars (997), likewise, each folder has different numbers of images.
Now I want to load them to train my model and to test it, but I don't have separate folders for the training and testing. I want to use folder names as labels and also want to split the data-set into training and testing as an equal ratio.
Your guidance, in this case, will be appreciated. Thanks