title: Pytorch 1.8 hangs by chance when calling loss.backward()

body:
When I was training an LSTM on pytorch the training process hangs by chance and cannot be terminated by Crlt+C. Then I used faulthandler to locate the problem. The training parameters, environment and faulthandler traceback output are listed below. It seems to be some problem with the C++ backend or the CUDA or even my graphic card I do not know.
Trace output batch_size = 64/32/16, num_workers = 2, CUDA 11.1, pytorch 1.8.0, cuDNN 8.0.5/8.0.1:

Thread 0x000036e8 (most recent call first):   File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 302 in wait
File "C:\Users\myUserName\anaconda3\lib\multiprocessing\queues.py",
line 227 in _feed   File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 870 in run
File "C:\Users\myUserName\anaconda3\lib\threading.py", line 932 in
_bootstrap_inner   File "C:\Users\myUserName\anaconda3\lib\threading.py", line 890 in
_bootstrap
Thread 0x00004644 (most recent call first):   File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 302 in wait
File "C:\Users\myUserName\anaconda3\lib\multiprocessing\queues.py",
line 227 in _feed   File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 870 in run
File "C:\Users\myUserName\anaconda3\lib\threading.py", line 932 in
_bootstrap_inner   File "C:\Users\myUserName\anaconda3\lib\threading.py", line 890 in
_bootstrap
Thread 0x00000efc (most recent call first):
Thread 0x00000138 (most recent call first):   File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 306 in wait
File "C:\Users\myUserName\anaconda3\lib\threading.py", line 558 in
wait   File
"C:\Users\myUserName\anaconda3\lib\site-packages\tqdm_monitor.py",
line 59 in run   File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 932 in
_bootstrap_inner   File "C:\Users\myUserName\anaconda3\lib\threading.py", line 890 in
_bootstrap
Thread 0x00001644 (most recent call first):   File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 306 in wait
File "C:\Users\myUserName\anaconda3\lib\queue.py", line 179 in get
File
"C:\Users\myUserName\anaconda3\lib\site-packages\tensorboard\summary\writer\event_file_writer.py",
line 232 in run   File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 932 in
_bootstrap_inner   File "C:\Users\myUserName\anaconda3\lib\threading.py", line 890 in
_bootstrap
Thread 0x0000443c (most recent call first):   File
"C:\Users\myUserName\anaconda3\lib\site-packages\torch\autograd_init_.py",
line 145 in backward   File
"C:\Users\myUserName\anaconda3\lib\site-packages\torch\tensor.py",
line 245 in backward   File "train.py", line 129 in main   File
"train.py", line 246 in
batch_size = 64, num_workers = 0,  CUDA 11.1, pytorch 1.8.0, cuDNN
8.0.5 Thread 0x00003650 (most recent call first):
Thread 0x000043b4 (most recent call first):   File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 306 in wait   File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 558 in wait   File
"C:\Users\myUserName\anaconda3\lib\site-packages\tqdm_monitor.py", line 59
in run   File "C:\Users\myUserName\anaconda3\lib\threading.py", line 932 in
_bootstrap_inner   File "C:\Users\myUserName\anaconda3\lib\threading.py", line 890 in _bootstrap
Thread 0x000017c4 (most recent call first):   File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 306 in wait   File
"C:\Users\myUserName\anaconda3\lib\queue.py", line 179 in get   File
"C:\Users\myUserName\anaconda3\lib\site-packages\tensorboard\summary\writer\event_file_writer.py",
line 232 in run   File "C:\Users\myUserName\anaconda3\lib\threading.py",
line 932 in _bootstrap_inner   File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 890 in _bootstrap
Thread 0x00001458 (most recent call first):   File
"C:\Users\myUserName\anaconda3\lib\site-packages\torch\autograd_init_.py",
line 145 in backward   File
"C:\Users\myUserName\anaconda3\lib\site-packages\torch\tensor.py", line 245
in backward   File "train.py", line 129 in main   File "train.py",
line 246 in

When num_workers=0 the output is the same except for lacking two threads below that I think belong to the dataloader.

Thread 0x000036e8 (most recent call first): File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 302 in wait
File "C:\Users\myUserName\anaconda3\lib\multiprocessing\queues.py",
line 227 in _feed File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 870 in run File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 932 in
_bootstrap_inner File "C:\Users\myUserName\anaconda3\lib\threading.py", line 890 in
_bootstrap
Thread 0x00004644 (most recent call first): File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 302 in wait
File "C:\Users\myUserName\anaconda3\lib\multiprocessing\queues.py",
line 227 in _feed File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 870 in run File
"C:\Users\myUserName\anaconda3\lib\threading.py", line 932 in
_bootstrap_inner File "C:\Users\myUserName\anaconda3\lib\threading.py", line 890 in
_bootstrap

The resource usage is also moderate, with around 20% CPU usage, 16/32GB memory, and 3.8/8GB graphics memory usage. The GPU usage is low when training RNNs. The script was run on Windows 10. The graphic card I use is RTX 3070. The driver version for my graphics card is 461.09.
More Information
When I started debugging code I was using the unmatching versions of CUDA 11.2 with pytorch 1.7.1 and cudnn 8.1.0. At that time I came into CUDA exceptions from time to time, with outputs like kernel launch failed or failed to synchronization, and things just hang without error after I changed my CUDA version.


title: Pytorch pruning on LSTM increases model size?

body:
I am applying pruning using pytorch's torch.nn.utils.prune on a model with LSTM layers. However, when I save the contents of the state_dict, the model is much larger than before pruning. I'm not sure why, as if I print out the sizes of the elements of the state_dict before and after pruning, everything is the same dimension, and there are no additional elements in the state_dict.
My code for pruning is pretty standard, and I make sure to call prune.remove():
        model_state = model.state_dict()
        torch.save(model.state_dict(), 'pre_pruning.pth')
        for param_tensor in model_state:
            print(param_tensor, "\t", model_state[param_tensor].size())

        parameters_to_prune = []
        for param, _ in model.rnn.named_parameters():
            if "weight" in param:
                parameters_to_prune.append((model.rnn, param))
        prune.global_unstructured(parameters_to_prune, pruning_method=prune.L1Unstructured, amount=0.6)
        for module, param in parameters_to_prune:
            prune.remove(module, param)

        model_state = model.state_dict()
        torch.save(model_state, 'pruned.pth') # This file is much larger than the original
        for param_tensor in model_state:
            print(param_tensor, "\t", model_state[param_tensor].size())

When I have tried to prune the linear layers in the model, the saved model does not show the same increase in size as when I prune the LSTM layers. Any idea what could be causing this?


title: Why does an attribute has a method?

body: I use pytorch define a Variable x, compute and get its gradient x.grad, meaning grad is an attribute of instance x. But, I can use x.grad.data.zero_() to set x.grad to zero meaning that data.zero_() is method of x.grad. Why does an attribute has a method? Thanks a lot.


title: How to set a variable as an attribute?

body:
I'm getting error when trying to set variable as an atribute.

parser = argparse.ArgumentParser()

parser.add_argument('--arch', action='store',
                dest='arch', default='alexnet',
                help='Store a simple value')

args = parser.parse_args()

model = models.args.arch(pretrained=True)

I know models.args.arch prodocues an error but how syntax should looke like to set a variable as an attribute? I could do it with if statements but it would be lot's of code and I guess it's possible in 1 line.


title: Backpropogation with self defined loss function and network

body:
I want to create a CNN network for object localization (It is given that there is only one object). For this I am using some general layers and in the end I want to get the nearest and farthest corner to origin. I am also using self defined loss function which is (100 - intersaction over union in %). Loss is not converging. What may be the problem?  Wheather backpropogation will work with this network or some other problem? Below is the code:
For illustraion purpose see .

Network:

class convnet(nn.Module):
def __init__(self):
    super(convnet, self).__init__()
    self.conv1 = nn.Conv2d(1, 4, kernel_size=5)
    self.pool1 = nn.MaxPool2d(kernel_size=3,stride=3)
    self.conv2 = nn.Conv2d(4, 8, kernel_size=5)
    self.pool2 = nn.MaxPool2d(kernel_size=3,stride=3)
    self.conv3 = nn.Conv2d(8, 16, kernel_size=5)
    self.pool3 = nn.MaxPool2d(kernel_size=3,stride=3)
    self.fc1 = nn.Linear(5040, 1000)
    self.fc2 = nn.Linear(1000, 84)
    self.fc3 = nn.Linear(84, 4)

def forward(self, x):
    x = F.relu(self.conv1(x))
    x = self.pool1(x)
    x = F.relu(self.conv2(x))
    x = self.pool2(x)
    x = F.relu(self.conv3(x))
    x = self.pool3(x)
    x = x.view(-1, 5040)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    a = self.sigmoid(self.fc3(x))
    c = torch.zeros(a.shape[0], 2)
    for idx, x in enumerate(a):
      d1 = x[0] ** 2 + x[1] ** 2
      d2 = x[2] ** 2 + x[3] ** 2
      d3 = x[0] ** 2 + x[3] ** 2
      d4 = x[2] ** 2 + x[1] ** 2
      dmin = min(d1, d2, d3, d4)
      if d1 == dmin:
        c[idx] = torch.tensor([x[0], x[1]])
      elif d2 == dmin:
        c[idx] = torch.tensor([x[2], x[3]])
      elif d3 == dmin:
        c[idx] = torch.tensor([x[0], x[3]])
      elif d4 == dmin:
        c[idx] = torch.tensor([x[2], x[1]])
    m = torch.tensor([[640, 480, 640, 480]]).type(torch.DoubleTensor).cuda()
    return c*m
def sigmoid(self, z):
    return 1/(1+torch.exp(-z))

Loss function:

def iou(box_a, box_b):
  A = box_a.size(0)
  B = box_b.size(0)
  max_xy = torch.min(box_a[:, 2:].unsqueeze(1).expand(A, B, 2),
                       box_b[:, 2:].unsqueeze(0).expand(A, B, 2))
  min_xy = torch.max(box_a[:, :2].unsqueeze(1).expand(A, B, 2),
                       box_b[:, :2].unsqueeze(0).expand(A, B, 2))
  inter = torch.clamp((max_xy - min_xy), min=0)
  inter =inter[:, :, 0] * inter[:, :, 1]
  area_a = ((box_a[:, 2]-box_a[:, 0]) *
              (box_a[:, 3]-box_a[:, 1])).unsqueeze(1).expand_as(inter)
  area_b = ((box_b[:, 2]-box_b[:, 0]) *
              (box_b[:, 3]-box_b[:, 1])).unsqueeze(0).expand_as(inter)
  union = area_a + area_b - inter
  return ((inter / union)*100/float(A*A)).sum()

def criterion(output, labels):
  return (100-iou(output, labels))

You can check full code here: link


title: Pyro: Simple inverse graphics example using SVI not working

body:
I'm new to pyro and trying to implement a simple inverse graphics problem involving estimating the coordinates of the points of a triangle rendered on a black & white 32x32 image.

So I defined a generative model that generates 3 uniformly random points, renders them into an image and observes the result.
I then use SVI with an autoguide (AutoMultivariateNormal) to try and estimates the points for a fixed triangle image.
SVI seems to run well and ELBO loss decreases, however when trying to sample from the posterior all I get is uniformly random points with no sign of learning.

My code in a Jupyter notebook with the results:

What am I missing here?


title: CNTK: "inferred dimension cannot be calculated from input and new shape size."

body:
I've set up a model for CIFAR-10 using Pytorch, and saved it as an ONNX file.

But it looks like I can't load it from CNTK.

I've already loaded another ONNX file from the same source code (by mistake), so the dependencies look OK. The problem occurs when I call Function.Load()

var deviceDescriptor = DeviceDescriptor.CPUDevice; ;
var function = Function.Load(ONNX_PATH, deviceDescriptor, ModelFormat.ONNX);

I get this exception (Unhandled exception):

   System.ApplicationException : 'Reshape: inferred dimension cannot be calculated from input and new shape size.

    [CALL STACK]
    - CNTK::TrainingParameterSchedule::  GetMinibatchSize
    - CNTK::  XavierInitializer (x6)
    - CNTK::Function::Load
    - CSharp_CNTK_Function__Load__SWIG_0
    - 00007FFB0C41C307 (SymFromAddr() error: Le module spÃ©cifiÃ© est introuvable.)


title: Python throws error when breakpoint is set, works normal otherwise

body:
I will just list my setup before I give details about my problem:
PyCharm EDU, Python 3.7.3, PyTorch 1.1.0 and a GeForce2080 TI.

I've downloaded an implementation of PointNet for PyTorch and just added a dataloader to fit my data format.
PointNet Code:
https://github.com/fxia22/pointnet.pytorch
PointNet Paper:
https://arxiv.org/abs/1612.00593

It's working fine, however, if I set any breakpoints in my code, python throws an error from time to time before it stops working at all. It seems that some thread breaks for some reason:

Exception ignored in: <function _DataLoaderIter.__del__ at 0x7fdf32f30378>
Traceback (most recent call last):
    File site-packages/torch/utils/data/dataloader.py"
        self._shutdown_workers()
    File site-packages/torch/utils/data/dataloader.py"
        w.join()
    File /multiprocessing/process.py"
        assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process

This ONLY happens if a breakpoint is set.
If I remove the breakpoint, there are no errors at all.
If I start the script without breakpoints and place a breakpoint at some point during the execution, it throws the error message shortly after I did so.

It is not really critical as I can just remove breakpoints and everything works fine, nonetheless I would like to know what's causing this error and how to fix it probably.

I did a search on the net, but I couldn't find anything describing the same problem I'm facing right now


title: What does `(pred[:, 2:4] > min_wh).all(1)` do in YOLO (Deep Learning)?

body:
I am using yoloV3 under pytorch. I encountered this code (pred[:, 2:4] > min_wh).all(1) and do not know its function. Can anyone help? Thanks!

My concern is the use of ().all(1). I know .all() or .any(), but do not know .all(1). Please explain .all(1), thanks.


title: Reinforcement learning converges for mean loss but not for each training data

body:
Here I show a dummy example that represents my actual problem.

My neural network (NN) receives one input and gives the probabilities for two output nodes. The code for the NN is:

class Net(torch.nn.Module):
  def __init__(self, N, M):
      super(Net, self).__init__()
      self.fc1 = torch.nn.Linear(N, 4)
      self.fc2 = torch.nn.Linear(4, 4)
      self.fc3 = torch.nn.Linear(4, M)
  def forward(self, x):
      x = torch.sigmoid(self.fc1(x))
      x = torch.sigmoid(self.fc2(x))
      x = torch.softmax(self.fc3(x),0)
      return x

The ABM class is our model that iteratiely sends calls to Net::forward, and based on the probabilities, chooses an action if it's the first index, increments agent_count. Inputs xx are stored in states which will be used to backward.

class ABM:
    def __init__(self,_nn,_t_data):
        self.nn = nn
        self.iteration_n = _t_data.iteration_n
        self.target_value = _t_data.target_value

    def run(self):
        for jj in range(self.iteration_n):
            xx = self.generate_input();
            self.states.append(xx); # store inputs
            ys = nn.forward(xx);
            action = self.draw(ys);
            if (action == 0):
                self.agent_count+=1

        loss = self.calculateReward();
        return loss;

    def generate_input(self):
        return torch.ones((1),requires_grad = True)
    --some other attributes--

When the run is over, error is calculated as error = (target_value - agent_count)/target_value which is a value between -1 and 1.
In order to train the model, the error is applied to the probability of the first output node of NN. This is to correct the NN in a way that predicts the right probability for the first output. The code is:

class ABM:
    def calculateReward(self):
        error = (self.target_value - self.agent_count)/self.target_value
        reward = torch.tensor((-error), requires_grad = True)
        #since all states are same, we just choose the first one
        state = self.states[0]
        ys = nn.forward(state)
        actionProb = ys[0]
        action_reward = actionProb * reward
        return action_reward;
    --some other members--

Two parameters of iteration_n and target_value used in the ABM are defined in the training data class as:

class Train:
    def __init__(self,tt , tv):
        self.iteration_n = tt
        self.target_value = tv
    target_value =0;
    iteration_n=0;

The harmony between different parts of the code is done as:

#### start optimization ####
nn = Net(1,2)
optimizer = optim.Adam(nn.parameters(), lr=0.01)
# create training data values
training_items = []
training_items.append(Train(1000,800))
training_items.append(Train(500,200))
error_record = []
for ii in range(100):
    print("############ start iteration #%d ################"%ii)
    for t_item in training_items:
        model = ABM(nn,t_item)
        loss = model.run()
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        error_record.append(loss.item())

Now let's present the problem.
If I only define one training item as Train(/*iteration number*/1000,/*target value*/800));, the NN is optimized as expected:

however, by defining two training items, although the average error declines to zero, the error on each training data stays high:

Is there any idea how to solve this issue?

I have omitted some parts of the code here to make it more readable. The full running code is available on minimal ABM


title: GPT-2 language model: multiplying decoder-transformer output with token embedding or another weight matrix

body:
I was reading the code of GPT2 language model. The transformation of hidden states to the probability distribution over the vocabulary has done in the following line:

lm_logits = self.lm_head(hidden_states)

Here,

self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)

However,
In the original paper, they suggested multiplying hidden states with the token embedding matrix whereas huggingface  implementation used another matrix.

Is there any advantage of this? Am I missing something?


title: when I run my network. I got an error,one of the variables needed for gradient computation has been modified by an inplace operation

body:
This is a part of code in my train.py:

output_c1[output_c1 > 0.5] = 1.
output_c1[output_c1 < 0.5] = 0.

dice = DiceLoss()
loss = dice(output_c1,labels)
optimizer.zero_grad()
loss.backward()
optimizer.step()

The error is:

Traceback (most recent call last):
  File "D:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3296, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-78553e2886de>", line 1, in <module>
    runfile('F:/experiment_code/U-net/train.py', wdir='F:/experiment_code/U-net')
  File "D:\pycharm\PyCharm Community Edition 2019.1.1\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "D:\pycharm\PyCharm Community Edition 2019.1.1\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "F:/experiment_code/U-net/train.py", line 99, in <module>
    loss.backward()
  File "D:\Anaconda3\lib\site-packages\torch\tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "D:\Anaconda3\lib\site-packages\torch\autograd\__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [4, 2, 224, 224]], which is output 0 of SigmoidBackward, is at version 2; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

When I comment this two lines of code:

output_c1[output_c1 > 0.5] = 1.
output_c1[output_c1 < 0.5] = 0.

it can run. I think the error comes from here but I don't know how to solve it.


title: Torch - Empty memory cache for a specific GPU

body:
I have multiple GPUs and used one of them for some feature generation on a remote Jupyter Notebook. The process is finished but the network is cut off before I am able to leave the notebook properly. Which results in the memory cache is still living in that GPU, checking nvidia-smi.

I do have another ongoing training process using another GPU. Do I empty the memory cache from the aforementioned GPU w/o affecting the ongoing process? Any help is welcomed thanks!!


title: How to use two seperate dataloaders together?

body:
I have two tensors of shape (16384,3,224,224) each. I need to multiply these two together. Obviously these two tensors are too big to fit in GPU ram. So I want to know, how should I go about this, divide them smaller batches using slicing or should I use two separate dataloaders?(I am confused, how to use two different dataloader together)
What would be the best way to do this?


title: Ptorch3D loss on projected image

body:
Im using Pytorch3D to take a projection of a mesh and running backprop on the loss on that projection.
The simplified version of my task is:

mesh = load_objs_as_meshes([os.path.join(path, 'mesh.obj')], device=device)

criterion = torch.nn.MSELoss()

deform_verts = torch.full(mesh.verts_packed().shape, 0.0, dtype=torch.float32, device=meta.device, requires_grad=True)
mesh = mesh.offset_verts(deform_verts)

projection = project_mesh(smpl_mesh, angle).to(device)[0, :, :, 0]  # gives the projected image at the required angle
loss = criterion(torch.flatten(projection), torch.flatten(ground_truth))

loss.backward()

The error im getting is:

RuntimeError: range.second - range.first == t.size() INTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch_1587428094786/work/torch/csrc/autograd/generated/Functions.cpp:57, please report a bug to PyTorch. inconsistent range for TensorList output (copy_range at /opt/conda/conda-bld/pytorch_1587428094786/work/torch/csrc/autograd/generated/Functions.cpp:57)


title: Duplicate element of tensor while perserving gradient values

body:
So I have a vector that I computed somehow with size k

x = torch.FloatTensor([0.5, 0.3, 0.1, 0.7])
x = x + 2

I want to take its first element x[0] and create a vector of size k-1 filled with the value x[0] so that the gradients that come along with this element are present in the new vector.

I tried using torch.full and filling it up with x[0] but that does not preserve gradients.

Using pytorch 1.4


title: How to use a layer with gradient but without weight adjustment?

body:
Is it possible to mark part of the forward pass to only backpropagate the gradient but not to adjust weights?
In the following example code I have a Module that uses only one layer (one set of parameters) but it is used twice in the forward step. During the optimization I would expect the weights to be adjusted twice as well. If I want to only adjust the weights for one of the layer usages, what can I do?
import torch

class ExampleModel(torch.nn.Module):

    def __init__(self, dim) -> None:
        super(ExampleModel, self).__init__()
        self.linear = torch.nn.Linear(dim, dim)

    def forward(self, x):
        out1 = self.linear(x)  # backprop gradients and adjust weights here
        out2 = self.linear(out1)  # only backprop gradients here
        return out2

# Random input output data for this example
N, D = 64, 100
x = torch.randn(N, D)
y = torch.randn(N, D)

model = ExampleModel(D)

criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.Adam(model.parameters())

y_pred = model(x)
loss = criterion(y_pred, y)

optimizer.zero_grad()
loss.backward()
optimizer.step()

The following will not work since with torch.no_grad() no gradient at all is backpropagated:
def forward(self, x):
    out1 = self.linear(x)  # backprop gradients and adjust weights here
    with torch.no_grad():
        out2 = self.linear(out1)  # only backprop gradients here
    return out2

I can not simply exclude the parameters from the optimization since they need to be optimized in the first part (i.e. out1 = self.linear(x)).
For the same reason I can also not set a learning rate of 0 for these parameters.
What else can I do to achieve this?


title: What should __len__ be for PyTorch when generating unlimited data?

body:
Say I am trying to use PyTorch to learn the equation y = 2x and I want to generate an unlimited amount of data to train my model with. I am supposed to provide a __len__ function. Here's an example below. What should it be in this case? How do I specify the number of mini-batch iterations per epoch? Do I just set a number arbitrarily?
import numpy as np
from torch.utils.data import Dataset

class GenerateUnlimitedData(Dataset):
    def __init__(self):
        pass

    def __getitem__(self, index):
        x = np.random.randint(1,10)
        y = 2 * x
        return x, y

    def __len__(self):
        return 1000000 # This works but is not correct


title: PySyft: dimension out of range on [prediction]

body:
I have a model that I am Testing, this model has undergone training. I have a sy.Virtualworker that I send a tensor to. I send the model to this virtual worker. I get the error IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
I have looked at other errors with this similar error, however, this might have an issue with what i’m doing with the pysyft material.
the goal is to get a prediction from the model based on the input. I believed the input to be of similar dimensions to what the model would work with.
I either have an issue with how I'm dealing with the pysyft material. As I have worked with this code successfully when doing a non-pysyft version and have seen the intended results.
full error list:

Traceback (most recent call last):
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\frameworks\torch\tensors\interpreters\native.py", line 333, in handle_func_command
cmd, args_, kwargs_, return_args_type=True
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\generic\frameworks\hook\hook_args.py", line 157, in unwrap_args_from_function
new_args = hook_args(args_)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\generic\frameworks\hook\hook_args.py", line 356, in
return lambda x: f(lambdas, x)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\generic\frameworks\hook\hook_args.py", line 535, in three_fold
lambdas[1](args_[1], **kwargs),
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\generic\frameworks\hook\hook_args.py", line 331, in
else lambda i: forward_functype(i)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\frameworks\torch\hook\hook_args.py", line 30, in
else (_ for _ in ()).throw(PureFrameworkTensorFoundError),
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\frameworks\torch\hook\hook_args.py", line 30, in
else (_ for _ in ()).throw(PureFrameworkTensorFoundError),
syft.exceptions.PureFrameworkTensorFoundError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "B:\tools and software\PyCharm 2020.1\plugins\python\helpers\pydev_pydev_bundle\pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "B:\tools and software\PyCharm 2020.1\plugins\python\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "B:/projects/GRA/FederatedLearningAnalysis/anomaly-detection/PyTroch_singleworker_combined_train_test.py", line 266, in
app.run(main)
File "B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\absl\app.py", line 299, in run
run_main(main, args)
File "B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\absl\app.py", line 250, in run_main
sys.exit(main(argv))
File "B:/projects/GRA/FederatedLearningAnalysis/anomaly-detection/PyTroch_singleworker_combined_train_test.py", line 261, in main
tr=tr, df_malicious=load_mal_data(), features=features)
File "B:/projects/GRA/FederatedLearningAnalysis/anomaly-detection/PyTroch_singleworker_combined_train_test.py", line 131, in test_with_data
Y_pred = model.predict(torch.from_numpy(X_test_scaled).float())
File "B:/projects/GRA/FederatedLearningAnalysis/anomaly-detection/PyTroch_singleworker_combined_train_test.py", line 204, in predict
x_pred = self.model(x)
File "B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\torch\nn\modules\module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "B:/projects/GRA/FederatedLearningAnalysis/anomaly-detection/PyTroch_singleworker_combined_train_test.py", line 183, in forward
x = torch.tanh(self.fc1(x))
File "B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\torch\nn\modules\module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\torch\nn\modules\linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\generic\frameworks\hook\hook.py", line 336, in overloaded_func
response = handle_func_command(command)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\frameworks\torch\tensors\interpreters\native.py", line 343, in handle_func_command
response = new_type.handle_func_command(new_command)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\generic\pointers\object_pointer.py", line 213, in handle_func_command
response = owner.send_command(location, cmd_name=cmd, args=args, kwargs_=kwargs_)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\workers\base.py", line 626, in send_command
ret_val = self.send_msg(message, location=recipient)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\workers\base.py", line 274, in send_msg
bin_response = self.send_msg(bin_message, location)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\workers\virtual.py", line 16, in send_msg
return location.recv_msg(message)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\workers\virtual.py", line 20, in recv_msg
return self.recv_msg(message)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\workers\base.py", line 310, in recv_msg
response = self.message_routertype(msg)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\workers\base.py", line 451, in execute_tensor_command
return self.execute_computation_action(cmd.action)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\workers\base.py", line 514, in execute_computation_action
response = command(*args, **kwargs)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\generic\frameworks\hook\hook.py", line 336, in overloaded_func
response = handle_func_command(command)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\frameworks\torch\tensors\interpreters\native.py", line 367, in handle_func_command
response = cls.get_response(cmd, args, kwargs)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\frameworks\torch\tensors\interpreters\native.py", line 401, in get_response
response = command_method(*args, **kwargs)
File "B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\torch\nn\functional.py", line 1370, in linear
ret = torch.addmm(bias, input, weight.t())
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\generic\frameworks\hook\hook.py", line 336, in overloaded_func
response = handle_func_command(command)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\frameworks\torch\tensors\interpreters\native.py", line 367, in handle_func_command
response = cls.get_response(cmd, args, kwargs)
File "C:\Users\OMEGA-Money\AppData\Roaming\Python\Python36\site-packages\syft\frameworks\torch\tensors\interpreters\native.py", line 401, in get_response
response = command_method(*args, **kwargs_)
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

CODE bits:

    def predict(self, x):
        x = x.send('testing')
        self.model.send(x.location)
        x_pred = self.model(x)
        mse = np.mean(np.power(x.get().data.numpy() - x_pred.get().data.numpy(), 2), axis=1)
        y_pred = mse > self.threshold
        return y_pred.astype(int)

CODE 2:

X_test = df.drop(columns=["malicious"]).values
    X_test_scaled = scaler.transform(X_test)
    Y_test = df["malicious"]
    Y_pred = model.predict(torch.from_numpy(X_test_scaled).float())


title: In a python iterator select other iterator based on conditions

body:
In python I have an iterator returning an infinite string of indices in a fixed range [0, N] called Sampler. Actually I have a list of those and all they do is return indices in the range [0, N_0], [N_0, N_1], ..., [N_{n-1}, N_n].
What I now want to do is first select one of these iterators based on the length of their range, so I have a weights list [N_0, N_1 - N_0, ...] and I select one of these with:
    iterator_idx = random.choices(range(len(weights)), weights=weights/weights.sum())[0]

Next, what I want to do is create an iterator which randomly selects one of the iterators and selects a batch of M samples.
class BatchSampler:
    def __init__(self, M):
        self.M = M
        self.weights = [weight_list]

        self.samplers = [list_of_iterators]
        ]
        self._batch_samplers = [
            self.batch_sampler(sampler) for sampler in self.samplers
        ]

    def batch_sampler(self, sampler):
        batch = []
        for batch_idx in sampler:
            batch.append(batch_idx)
            if len(batch) == self.M:
                yield batch

        if len(batch) > 0:
            yield batch

    def __iter__(self):
        # First select one of the datasets.
        iterator_idx = random.choices(
            range(len(self.weights)), weights=self.weights / self.weights.sum()
        )[0]
        return self._batch_samplers[iterator_idx]

The issue with this is that iter() only seems to be called once, so only the first time iterator_idx is selected. Obviously this is wrong... What is the way around this?
This is a possible case when you would have multiple datasets in pytorch, but you want to sample only batches from one of the datasets.


title: Pytorch: How to concatenate lists within a tensor?

body:
I have a tensor of size (2, b, h) and I want to change it to the following size: (b, 2*h), where the corresponding lists are concatenated, for example:
a = torch.tensor([[[1, 2, 3], [4, 5, 6], [4, 4, 4]],
                  [[4, 5, 6], [7, 8, 9], [5, 5, 5]]])

I want:
b = tensor([[1, 2, 3, 4, 5, 6],
            [4, 5, 6, 7, 8, 9],
            [4, 4, 4, 5, 5, 5]])


title: How to retain 2D (or more) shape when using pytrorch masked_select

body:
Suppose I have the following two matching shape tensors:
a = tensor([[ 0.0113, -0.1666,  0.5960, -0.0667], [-0.0977, -0.1984,  0.5153,  0.0420]])
selectors = tensor([[ True,  True, False, False], [ True,  False, True, False]])

When using torch.masked_select to find the values in a that match True indices in selectors like this:
torch.masked_select(a, selectors)

The output will be in 1D shape instead of the original 2D shape:
tensor([ 0.0113, -0.1666, -0.0977, 0.5153])

This is consistent with masked_select behavior as it is given in the documentation (torch.masked_select). However, my goal is to get a result that matches the shape of the two original tensors. I.e.:
tensor([[0.0113, -0.1666], [-0.0977, 0.5153]])

Is there a way to get this without having to loop over all the elements in the tensors and find the mask for each one? Please note that I have also looked into using torch.where, but it doesn't fit the case I have as I see it.


title: Exporting forecast data from darts

body:
I am trying to export the forecasts I made with Pytorch models in darts library to some exchangeable format like XLS or CSV.
Is it somehow possible to export the predictions from the Timeseries class? How can I specify output format?


title: Determine whether a model is pytorch model or a tensorflow model or scikit model

body:
If I want to determine the type of model i.e. from which framework was it made programmatically, is there a way to do that?
I have a model in some serialized manner(Eg. a pickle file). For simplicity purposes, assume that my model can be either tensorflow's, pytorch's or scikit learn's. How can I determine programmatically which one of these 3 is the one?


title: Pytorch executeable works while running from Anaconda prompt but not from Cmd or .exe?

body:
I packaged (using Pyinstaller) a small variant of the Minimalistic Yolo github repo, found Here, the packaging was done using pyinstaller to run the object detection as a server using Flask.
So while attempting to run the server, it only works when running from Anaconda Prompt (Which is where i wrote the pyinstaller command) other than that, the following error occur.
Error i Get while running from (exe,Cmd,PowerShell) is:
Traceback (most recent call last):
File "flask\app.py", line 2446, in wsgi_app
File "flask\app.py", line 1951, in full_dispatch_request
File "flask\app.py", line 1820, in handle_user_exception
File "flask\_compat.py", line 39, in reraise
File "flask\app.py", line 1949, in full_dispatch_request
File "flask\app.py", line 1935, in dispatch_request
File "FlaskServerV2.py", line 53, in Hello
File "torch\nn\modules\module.py", line 532, in __call__
File "models.py", line 259, in forward
File "torch\nn\modules\module.py", line 532, in __call__
File "models.py", line 177, in forward
RuntimeError: error in LoadLibraryA
127.0.0.1 - - [19/Nov/2020 10:28:53] "GET /detect HTTP/1.1" 500 -

But when running in conda, the code works fine and runs.
so I am suspecting it's an issue with a PyTorch dependency.
Current Code:
from __future__ import division

from flask import Flask, Response, jsonify
app = Flask(__name__)

from models import *
from utils.utils import *
from utils.datasets import *

import os
import sys
import time
import datetime
import argparse

from PIL import Image

import torch
from torch.utils.data import DataLoader
from torchvision import datasets
from torch.autograd import Variable

import matplotlib.pyplot as plt
import matplotlib.patches as patches
from matplotlib.ticker import NullLocator

import cv2
import time
import json

@app.route('/CheckIfRunning')
def CheckIfRunning():
    return '1'

@app.route('/detect')
def Hello():
    global device
    global model
    global classes
    global colors
    global Tensor
    global a
    img=cv2.imread("temp.jpg")
    PILimg = np.array(Image.fromarray(cv2.cvtColor(img,cv2.COLOR_BGR2RGB)))
    imgTensor = transforms.ToTensor()(PILimg)
    imgTensor, _ = pad_to_square(imgTensor, 0)
    imgTensor = resize(imgTensor, 416)
    #add the batch size
    imgTensor = imgTensor.unsqueeze(0)
    imgTensor = Variable(imgTensor.type(Tensor))
    with torch.no_grad():
        detections = model(imgTensor)
        detections = non_max_suppression(detections,0.8, 0.4)
    a.clear()

    Return={}
    ReturnCounter=0
    if detections is not None:
            a.extend(detections)
            b=len(a)
            if len(a)  :
                for detections in a:
                    if detections is not None:
                        detections = rescale_boxes(detections, 416, PILimg.shape[:2])
                        unique_labels = detections[:, -1].cpu().unique()
                        n_cls_preds = len(unique_labels)
                        for x1, y1, x2, y2, conf, cls_conf, cls_pred in detections:
                            box_w = x2 - x1
                            box_h = y2 - y1
                            color = [int(c) for c in colors[int(cls_pred)]]
                            img = cv2.rectangle(img, (x1, y1 + box_h), (x2, y1), color, 2)
                            cv2.putText(img, classes[int(cls_pred)], (x1, y1),     cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 2)
                            cv2.putText(img, str("%.2f" % float(conf)), (x2, y2 - box_h), cv2.FONT_HERSHEY_SIMPLEX, 0.5,color, 2)
                            Return[ReturnCounter]=    [x1.item(),y1.item(),x2.item(),y2.item(),conf.item(),cls_conf.item(),classes[int(cls_pred)]]
                            ReturnCounter+=1
                        cv2.imwrite("Temp2.jpg",img)
                        return Return

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Set up model
model = Darknet("config/yolov3.cfg", img_size=416).to(device)

model.load_darknet_weights("weights/yolov3.weights")

model.eval()  # Set in evaluation mode

classes = load_classes("data/coco.names")  # Extracts class labels from file
colors = np.random.randint(0, 255, size=(len(classes), 3), dtype="uint8")
Tensor = torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor

a=[]
app.run(threaded=True)


title: How to make graph data in pytorch

body:
I am tring to make graph model (GCN). Since I am not used to PyTorch, I have a problem setting up the dataset. I want to make custom graph data.
I don't know why my code is not working properly:
class MoneyGraph(Dataset):
    def __init__(self):
        # node features
        node_features = np.loadtxt(
            "./datas/node_features.csv",
            encoding="utf-8",
            delimiter=",",
            dtype=np.float32,
            skiprows=1,
        )
        self.x = torch.tensor(node_features, dtype=torch.float)

        # label
        labels = [1, 1, 1, 0, 0, 1, 0, 0, 0, 1]
        self.y = torch.tensor(labels, dtype=torch.float)

        # edge index
        target_nodes = [1, 2, 3, 5, 3, 5, 7, 1, 7]
        source_nodes = [9, 2, 2, 1, 7, 7, 3, 4, 2]
        self.edge_index = torch.tensor([source_nodes, target_nodes], dtype=torch.long)

        # etc info
        self.num_node_features = 5
        self.num_classes = 2
        self.num_nodes = len(self.x)

    def __getitem__(self, index):
        return self.x[index], self.y[index]

    def __len__(self):
        return len(self.x)

    def is_undirected(self):
        return False

And I tried to run this:
loader = DataLoader(dataset, batch_size=5, shuffle=True)

for batch in loader:
    print(batch.x)

I get the following error message:

AttributeError: 'list' object has no attribute 'x'


title: Zero diagonal of a PyTorch tensor?

body:
Is there a simple way to zero the diagonal of a PyTorch tensor?
For example I have:
tensor([[2.7183, 0.4005, 2.7183, 0.5236],
        [0.4005, 2.7183, 0.4004, 1.3469],
        [2.7183, 0.4004, 2.7183, 0.5239],
        [0.5236, 1.3469, 0.5239, 2.7183]])

And I want to get:
tensor([[0.0000, 0.4005, 2.7183, 0.5236],
        [0.4005, 0.0000, 0.4004, 1.3469],
        [2.7183, 0.4004, 0.0000, 0.5239],
        [0.5236, 1.3469, 0.5239, 0.0000]])


title: Switch off CUDA in PyTorch without uninstalling

body:
I am new to PyTorch, and by mistake I have installed PyTorch with CUDA version 10.2. In reality, I do not have CUDA available in my system.
So when I compile my program, I get the following error:
RuntimeError: Detected that PyTorch and torch_cluster were compiled with different CUDA versions. PyTorch has CUDA version 10.2 and torch_cluster has CUDA version 0.0. Please reinstall the torch_cluster that matches your PyTorch install.
I would like to switch my CUDA in PyTorch to CPU (or CUDA 0.0, as far as I understood, it is equivalent term). So is it possible to do without reinstallation of the PyTorch?


title: kmeans clustering python

body:
There are coordinates that I want to cluster. The result of clustering using the kmeans
[[0, 107], [0, 108], [0, 109], [0, 115], [0, 116],
[0, 117], [0, 118], [0, 125], [0, 126], [0, 127],
[0, 128], [0, 135], [0, 136], [0, 194], [0, 195],
[1, 107], [1, 108], [1, 109], [1, 110], [1, 114],
[1, 115], [1, 116], [1, 117], [1, 118], [1, 119]...]

The result of clustering using the kmeans
from sklearn.cluster import KMeans
num_clusters = 9
km = KMeans(n_clusters=num_clusters)
km_fit = km.fit(nonzero_pred_sub)

>>>array([7, 7, 7, 1, 1, 1, 1, 5, 5, 5, 5, 3, 3, 0, 0, 7, 7, 7, 7, 1, 1, 1,
   1, 1, 1, 5, 5, 5...]

I want to know the coordinates of i-th cluster
for example, i need elements of 1st cluster and i can assume [0, 107], [0, 108], [0, 109] was clustered into the 7-th cluster.
How can i get coordinates from cluster?


title: Update elements in torch.zeros with other given tensors

body:
Let’s say, there are a PyTorch torch.zeros tensor (x-dim) and two other tensors 1-dim.
test_t = torch.zeros(2, 1, 1, 1)

output:
tensor([[[[0.]]],

        [[[0.]]]])

t1 = torch.tensor(10.)
t2 = torch.tensor(20.)

I want to modify the original tensor in-place and replace zeros with values in given order.
Required result:
test_t

output:
tensor([[[[10.]]],

        [[[20.]]]])

upd: see solution below


title: Why is using the GPU slower than using the CPU?

body:
Consider the following network:
%%time
import torch
from torch.autograd import grad
import torch.nn as nn
import torch.optim as optim

class net_x(nn.Module):
        def __init__(self):
            super(net_x, self).__init__()
            self.fc1=nn.Linear(1, 20)
            self.fc2=nn.Linear(20, 20)
            self.out=nn.Linear(20, 400) #a,b,c,d

        def forward(self, x):
            x=torch.tanh(self.fc1(x))
            x=torch.tanh(self.fc2(x))
            x=self.out(x)
            return x

nx = net_x()

#input
val = 100
t = torch.rand(val, requires_grad = True) #input vector
t = torch.reshape(t, (val,1)) #reshape for batch

#method
dx = torch.autograd.functional.jacobian(lambda t_: nx(t_), t)

This outputs
CPU times: user 11.1 s, sys: 3.52 ms, total: 11.1 s
Wall time: 11.1 s

However, when I change to using the GPU with .to(device):
%%time
import torch
from torch.autograd import grad
import torch.nn as nn
import torch.optim as optim
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

class net_x(nn.Module):
        def __init__(self):
            super(net_x, self).__init__()
            self.fc1=nn.Linear(1, 20)
            self.fc2=nn.Linear(20, 20)
            self.out=nn.Linear(20, 400) #a,b,c,d

        def forward(self, x):
            x=torch.tanh(self.fc1(x))
            x=torch.tanh(self.fc2(x))
            x=self.out(x)
            return x

nx = net_x()
nx.to(device)
#input
val = 100
t = torch.rand(val, requires_grad = True) #input vector
t = torch.reshape(t, (val,1)).to(device) #reshape for batch

#method
dx = torch.autograd.functional.jacobian(lambda t_: nx(t_), t)

This outputs:
CPU times: user 18.6 s, sys: 1.5 s, total: 20.1 s
Wall time: 19.5 s

Update 1:
Checking the timing of the process of moving the inputs and model to device:
%%time
nx.to(device)
t.to(device)

This outputs:
CPU times: user 2.05 ms, sys: 0 ns, total: 2.05 ms
Wall time: 2.13 ms

Update 2:
Looks like @Gulzar was right. I changed my batch size to 1000 (val=1000) and the CPU outputs:
Wall time: 8min 44s
While the GPU outputs:
Wall time: 3min 12s


I am learning detectron2 from someone's code, but I got an error when I tried to import BoxMode from detectron2.structures.I think BoxMode got removed from the library, I want to know if they change BoxMode with something else that should use instead of it.
error message:
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-6-528200f0b892> in <module>
      6 import numpy as np
      7 import pandas as pd
----> 8 from detectron2.structures import BoxMode
      9 from tqdm import tqdm
     10

/opt/conda/lib/python3.7/site-packages/detectron2/structures/__init__.py in <module>
      5 from .instances import Instances
      6 from .keypoints import Keypoints, heatmaps_to_keypoints
----> 7 from .masks import BitMasks, PolygonMasks, polygons_to_bitmask
      8 from .rotated_boxes import RotatedBoxes
      9 from .rotated_boxes import pairwise_iou as pairwise_iou_rotated

/opt/conda/lib/python3.7/site-packages/detectron2/structures/masks.py in <module>
      7 import torch
      8
----> 9 from detectron2.layers.roi_align import ROIAlign
     10
     11 from .boxes import Boxes

/opt/conda/lib/python3.7/site-packages/detectron2/layers/__init__.py in <module>
      1 # Copyright (c) Facebook, Inc. and its affiliates.
      2 from .batch_norm import FrozenBatchNorm2d, get_norm, NaiveSyncBatchNorm
----> 3 from .deform_conv import DeformConv, ModulatedDeformConv
      4 from .mask_ops import paste_masks_in_image
      5 from .nms import batched_nms, batched_nms_rotated, nms, nms_rotated

/opt/conda/lib/python3.7/site-packages/detectron2/layers/deform_conv.py in <module>
      9 from torchvision.ops import deform_conv2d
     10
---> 11 from detectron2 import _C
     12
     13 from .wrappers import _NewEmptyTensorOp

ImportError: libcudart.so.11.0: cannot open shared object file: No such file or directory


title: sequence to sequence learning for language translation, what about unseen words

body:
sequence to sequence learning is a powerful mechanism for language translation, especially using it locally in a context specific case.

I am following this pytorch tutorial for the task.

However, the tutorial did not split the data into training and testing.
You might think its not a big deal, just split it up, use one chunk for training and the other for testing. But it is not that simple.

Essentially, the tutorial creates the indices of the seen words while leading the dataset. The indices are simply stored in the dictionary. This is before going to the encoder RNN, just a simple conversion kind of task from words to the numbers.

If data is split up at random, what happens is, one of the keyword may not appear in the sentences from the training set, and so may not have an index at all. If it shows up at the time of testing, what should be done?

Extend the dictionary?


title: Import error on Windows10 with pytorch0.4

body:
Discription

I am trying to install pytorch 0.4 on Windows10.
My enviroment settings:
- Windows10
- cuda9.0
- python 3.6
- pytorch 0.4
- anaconda

I tried by using both conda install -n myenv and pip install $path:whl and both failed.

Error

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Anaconda\envs\py3.6_pytorch0.4\lib\site-packages\torch\__init__.py", line 80, in <module>
    from torch._C import *
ImportError: DLL load failed:

I found the related a issue #4518 under pytorch but answers under that issue do not work for me.

What I have tried

add all conda related path to environment path
change the directory (cd)
install vs_runtime under this conda env

None of those works.

But if I install pytorch under conda's base environment, it works well.

So what's going on here?

Update:

When we tried to install pytorch on windows, a lot of packages need to be installed at the same time. We can simply follow the steps on official website [link](https://pytorch.org/），for python 3.6 and cuda9.0 the installation command is as

    conda install pytorch torchvision cudatoolkit=9.0 -c pytorch

If we want to install former version of pytorch, we can assign the version, eg. 0.4 as

    conda install pytorch=0.4 torchvision cudatoolkit=9.0 -c pytorch


title: the model learned, but cannot predict?

body:
I follow the tutorial to write some linear regression code about boston price, it worked very well and the loss became smaller, when I wanted to paint the graph in matplotlib, I found the graph not showed as what in my mind.

I searched,but could not solve my question.

import pandas as pd
from sklearn.datasets import load_boston
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
import torch
from torch.autograd import Variable

import matplotlib.pyplot as plt

if __name__ == '__main__':
    boston = load_boston()
    col_names = ['feature_{}'.format(i) for i in range(boston['data'].shape[1])]
    df_full = pd.DataFrame(boston['data'], columns=col_names)

    scalers_dict = {}
    for col in col_names:
        scaler = StandardScaler()
        df_full[col] = scaler.fit_transform(df_full[col].values.reshape(-1, 1))
        scalers_dict[col] = scaler

    x_train, x_test, y_train, y_test = train_test_split(df_full.values, boston['target'], test_size=0.2, random_state=2)

    model = torch.nn.Sequential(torch.nn.Linear(x_train.shape[1], 1), torch.nn.ReLU())

    criterion = torch.nn.MSELoss(reduction='mean')
    optimizer = torch.optim.Adam(model.parameters(), lr=0.1)

    n_epochs = 2000

    train_loss = []
    test_loss = []
    x_train = Variable(torch.from_numpy(x_train).float(), requires_grad=True)
    y_train = Variable(torch.from_numpy(y_train).float())

    for epoch in range(n_epochs):
        y_hat = model(x_train)
        loss = criterion(y_hat, y_train)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        epoch_loss = loss.data ** (1/2)
        train_loss.append(epoch_loss)
        if (epoch + 1) % 250 == 0:
            print("{}:loss = {}".format(epoch + 1, epoch_loss))

    order = y_train.argsort()
    y_train = y_train[order]
    x_train = x_train[order, :]

    model.eval()

    predicted = model(x_train).detach().numpy()
    actual = y_train.numpy()
    print('predicted:", predicted[:5].flatten(), actual[:5])
    plt.plot(predicted.flatten(), 'r-', label='predicted')
    plt.plot(actual, 'g-', label='actual')
    plt.show()

why the predict were the same result like [22.4413, 22.4413, ...],
in the picture, it's a horizontal line.
I'm a very beginner to deeplearning, thank you very much for your help!


title: How to vectorize the sum? tensor[i,:,:,:] + tensor[i]

body:
I want to vectorize the following code:

def style_noise(self, y, style):
    n = torch.randn(y.shape)
    for i in range(n.shape[0]):
        n[i] = (n[i] - n.mean(dim=(1, 2, 3))[i]) * style.std(dim=(1, 2, 3))[i] / n.std(dim=(1, 2, 3))[i] + style.mean(dim=(1, 2, 3))[i]
    noise = Variable(n, requires_grad=False).to(y.device)
    return noise

I didn't find a way nice way of doing so.

y and style are 4d tensors, say  style.shape = y.shape = [64, 3, 128, 128].

I want to return the noise tensor, noise.shape = [64, 3, 128, 128].

Please let me know in the comments if the question is not clear.


title: How nn.Linear works in shape(batch_size, seq_length, hidden_size)?

body:
In self.classifier, I assume there should be different weights to apply on tokens. Below is the huggingface's implementation.

def __init__(self, config, num_labels=2):
    super(BertForTokenClassification, self).__init__(config)
    self.num_labels = num_labels
    self.bert = BertModel(config)
    self.dropout = nn.Dropout(config.hidden_dropout_prob)
    self.classifier = nn.Linear(config.hidden_size, num_labels)
    self.apply(self.init_bert_weights)

def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None):
    sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
    sequence_output = self.dropout(sequence_output)
    logits = self.classifier(sequence_output)

    if labels is not None:
        loss_fct = CrossEntropyLoss()
        # Only keep active parts of the loss
        if attention_mask is not None:
            active_loss = attention_mask.view(-1) == 1
            active_logits = logits.view(-1, self.num_labels)[active_loss]
            active_labels = labels.view(-1)[active_loss]
            loss = loss_fct(active_logits, active_labels)
        else:
            loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
        return loss
    else:
        return logits


title: Where does Ray.Tune create the model vs implementing the perturbed hyperparameters

body:
I am new to using ray.tune. I already have my network written in a modular format and now I am trying to incorporate ray.tune, but I do not know where to initialize the model (vs updating the perturbed hyperparameters) so that the model and the weights are not re-initialized when a worker is truncated and replaced by a better performing worker.

Background

I am using the PBT scheduler of ray.tune which creates num_samples number of models (workers) each of which are initialized with a different set of sampled hyperparameters. When a model is evaluated, if it is performing poorly, it will be stopped and load the checkpoint of one of the top performing workers. Once it is loaded (this is a deep copy of the network), the hyperparameters are perturbed and then it will train until the next evaluation.
The MyTrainable class should have a _setup, _train, _save, and _restore function. The setup calls for a config variable and this is where the newly sampled hyperparameters are implemented.

My question is where should be the original model be defined? I can easily implement the updated HPs in this section. But I have not seen anywhere in the documentation where I can pass a pre-defined model into the ray.tune.run function. If I keep the create_model() function in the _setup() though, it will eliminate the previously trained weights which is part of the benefit of this method.

Code

Here are the 3 functions I have:

self._hyperparameters(config)    # redefines the self.opt options accoring to the new perturbations
self.model.update_optimizer(self.opt)  # redefines the optimizers using the new learning rates and the beta values for Adam

self.model = create_model(self.opt)   # Original function that defines the initial model and initializes the weights


title: Quantization of linear layer fails

body:
I quantized a MobileNetV2 with linear output. Conv+BN+Relu were fused to ConvBnReLU2d and the Linear+Relu was fused to a LinearReLU layer. When executing it, I get the error message:

Didn't find kernel to dispatch to for operator 'quantized::linear_relu'.

All the conv layers seem to work correctly. According to the documentation LinearReLU is supported:
https://pytorch.org/docs/stable/quantization.html#torch.nn.intrinsic.LinearReLU


title: Manually split and distribute a large tensor for distributed computing with PyTorch

body:
AFAIK, the simplest way to do the distributed training (multiple modes) with Pytorch is something like:

sampler = torch.utils.data.distributed.DistributedSampler(train_data)
data_loader = torch.utils.data.DataLoader(dataset, sampler=sampler)
model = torch.nn.DataParallel(model).cuda()

for data, target in data_loader:
    out = model(data)
    ...

But what if I already have a large tensor data in hand and would like to split and distribute it and get the same output as the above snippet? Specifically,

model = torch.nn.DataParallel(model).cuda()
data = do_sth_fuct(data)
out = model(data)

Is there a PyTorch API to do so? Otherwise, what is the best way to achieve it? Thank you in advance!


title: Using Torchvision ImageFolder with Test Set

body:
I am trying to solve the Dogs-vs-Cats challenge on Kaggle using the Sample Notebook that has been provided in the Udacity course. I have rearranged the files into two folder dogs/ and cats/ in the train/ directory so that the ImageFolder class can pick up the categories, but I don't know what to do in the test folder? I don't have the labels ready.

Do I just not use the ImageFolder API (seems that the course used it, so it should be usable, and obviously very convenient), or is there some option to use it when the classes are not already known. I could not find anything in this vein on the official documentation, but it should be possible seeing the course solution does it that way. Thanks for any help.


title: How can I run this prediction.py and evaluate_performance.py file?

body:
I am a medical student and I am using google colab to learn fastAI.  In this project, https://github.com/QinglingGo/Classification-of-Objects-using-Deep-Learning-Model，
I can achieve the output of the model, but I don't know how to perform the prediction.py and the evaluat_performance.py files.
When I run the evaluat_performance.py, the following message will appear:

python3: can't open file 'prediction.py': [Errno 2] No such file or directory
/usr/local/lib/python3.6/dist-packages/IPython/utils/traitlets.py:5: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package.
  warn ("IPython.utils.traitlets has moved to a top-level traitlets package.")
1. Loading Data ...
ImageDataBunch;

Train: LabelList (942 items)
x: SegmentationItemList
Image (3, 256, 256), Image (3, 256, 256), Image (3, 256, 256), Image (3, 256, 256), Image (3, 256, 256)
y: SegmentationLabelList
ImageSegment (1, 256, 256), ImageSegment (1, 256, 256), ImageSegment (1, 256, 256), ImageSegment (1, 256, 256), ImageSegment (1, 256, 256)
Path: / content / drive / My Drive / Colab Notebooks / bbc_train / images;

Valid: LabelList (0 items)
x: SegmentationItemList

y: SegmentationLabelList

Path: / content / drive / My Drive / Colab Notebooks / bbc_train / images;

Test: None
2. Instantiating Model ...
Traceback (most recent call last):
  File "evaluate_preformance.py", line 66, in <module>
    combined_accuracy, classification_accuracy, bbox_score, segmentation_accuracy = evaluate ()
  File "evaluate_preformance.py", line 29, in evaluate
    M = Model (path = model_dir, file = 'export.pkl')
NameError: name 'Model' is not defined.

And I don't understand the meaning of "from sample_student import Model" in line 6 of the .py file?  Can anyone help me?

Thanks in advance!


title: How to implement a gaussian renderer with mean and variance values as input in any deep modeling framework (needs to be back-propagable)

body:
Imagine a typical auto-encoder-decoder model. However, instead of a general decoder where deconvoutions together with upscaling are used to create/synthesize a tensor similar to the model's input, I need to implement a structured/custom decoder.

Here, I need the decoder to take its input, e.g. a 10x2 tensor where each row represents x,y positions or coordinates, and render a fixed predefined size image where there are 10 gaussian distributions generated at the location specified by the input.

In another way, I need to create an empty fixed sized tensor, fill the locations specified by the 10 coordinates a value 1, and then sweep a gaussian kernel over the whole tensor. For example, imagine the following 1-d scenario. let the input to the whole model be a vector of size 10. if the input to the decoder is [3, 7], which are two x-coordinates (0-indexing), and the gaussian kernel of size 3 that we want to use is [0.28, 0.44, 0.28], then the output of the decoder should look like the following (should be the same size as the original input of the model which is 10):

[0, 0, 0.28, 0.44, 0.28, 0, 0.28, 0.44, 0.28, 0]

which is the same as [0, 0, 0, 1, 0, 0, 0, 1, 0, 0]*[0.28, 0.44, 0.28] where * represents the convolution operator. please note that in the first vector, the 1 or located at positions 3 and 7 considering a 0-indexing format.

Finally a typical pixel loss such as MSE will be calculated. The important part is that this rendering module needs to be able to backpropagate the errors from the loss to its inputs which are the coordinates.

This module itself does not have any trainable parameters. Also, I do not want to change the layers coming before this rendering module and they need to stay as they are. In a more advanced setting, I would also like to provide the 4 covariance values as input too, i.e. the input to the renderer would be in the form of [num_points, 5] where each row is [x_coord, y_coord, cov(x,x), cov(x,y), cov(y,y)].

How can I implement such a module in any of the available deep learning frameworks? a hint towards something similar would also be very useful.


title: Handle large dense vectors generated from universal sentence encoder

body:
I am working on a document classification problem using CNN/LSTM and embeddings generated from universal sentence encoder. I have 10,000 records and each record has about 100~600 sentences. I save all the document matrices into one json file before I feed them into the neural network models. The overall json file is about 20GB which will take too much memory.

I'm not sure if I should save documents in text format and convert them into sentence embeddings during the training process. What's the potential solution?


title: How to properly run my project using GPU?

body:
I am really new to torch and machine learning. I am trying to run my project using GPU. I tried to have such modification to my code:

model = Challenge()
    model = model.to(torch.device('cuda'))

However, I am still having following error:

Traceback (most recent call last):
  File "C:/Users/ruidong/Desktop/YZR temp/Project2/train_challenge.py", line 112, in <module>
    main()
  File "C:/Users/ruidong/Desktop/YZR temp/Project2/train_challenge.py", line 91, in main
    stats)
  File "C:/Users/ruidong/Desktop/YZR temp/Project2/train_challenge.py", line 40, in _evaluate_epoch
    output = model(X)
  File "C:\Users\ruidong\Anaconda3\envs\EECS445\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ruidong\Desktop\YZR temp\Project2\model\challenge.py", line 48, in forward
    z = F.relu(self.conv1(x))
  File "C:\Users\ruidong\Anaconda3\envs\EECS445\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ruidong\Anaconda3\envs\EECS445\lib\site-packages\torch\nn\modules\conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "C:\Users\ruidong\Anaconda3\envs\EECS445\lib\site-packages\torch\nn\modules\conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _thnn_conv2d_forward

Any suggestions? really appreciate.


title: why doesnt Stochastic gradient descent fluctuate

body:
In batch gradient descent the parameters were updated based on the total/average loss of all the points
In Stochastic gradient descent or SGD
we are updating the parameters after every point instead of one epoch.
so lets say if the final point is an outlier woudnt that cause the whole fitted line to fluctuate drastically.
How is it reliable .
or converge on a contour like this SGD contour


title: Update weights dynamically (regarding input) with pytorch

body:
I'm trying to implement an autoencoder for recommendation (autorec) which takes sparse 1d vectors as inputs.

I need to update only the weighs associated to non null values at each itérations.
My model is like

# architecture Autorec
class Model(nn.Module):

    def __init__(self, num_latent, num_items, dropout=0.05):
        super(Model, self).__init__()
        self.encoder = nn.Linear(num_items, num_latent)
        self.decoder = nn.Linear(num_latent, num_items)
    def forward(self, inputs):
        x = self.encoder(inputs)
        x = F.leaky_relu(x)
        outputs = self.decoder(x)
        #outputs = F.linear(x)
        return outputs

And the train function looks like

def train(num_epochs, batch_size, criterion, optimizer, model, dataset):
    dataloader = DataLoader(dataset, batch_size, shuffle=True)
    train_error = []
    test_error = []
    model.train()
    for epoch in range(num_epochs):
        epoch_average_loss = 0.0
        for (X, y) in (dataloader):
            print(X.requires_grad())
            y_pre = model(X)
            loss = criterion(y_pre, y)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            epoch_average_loss += loss.item() * batch_size / len(train_set)
        model.eval()
        with torch.no_grad():
          y_pre_val = model(torch.from_numpy(np.array(w_test)).float())
        error = criterion(y_pre_val, torch.tensor(np.array(w_test_merged)).float()).item()
        test_error.append(error)
        train_error.append(epoch_average_loss)
        if ((epoch+1)%1 == 0):
                print('Epoch [{}/{}], Loss_error: {:.4f}'
                      .format(epoch+1, num_epochs,  epoch_average_loss))
    return train_error, test_error

So first, how to update only weight associated to non nul input ?
I know I have to use requires_grad = False but I don't know how.

Then, is there a way to do it with batch_size > 1?


title: What does "Variable" means in python? Is it a standard function?

body:
I have some python code that I have to read and understand.
In one line I find
 img = Variable(torch.from_numpy(img.transpose(2, 0, 1)[np.newaxis,:,:,:]).cuda().float(), volatile=True)

what is this Variable I am seeing? When I use the IDE to find the definition it says 'No definition found for Variable' which makes me suspect it is a standard function in python. I Obviously can not google "Variable" for python because I will get countless definitions of what a variable is in python.
Has anyone seen a line like this before? Where Variable is being used as a function?
t


title: Pytorch: Lower the parameters in U-net model

body:
can anyone give me some tips on how i would be able to lower the amount of parameters in the following U-net implementation. I'm having trouble with over-fitting on my training data and i would like to lower the parameters in order to see if it improves the validation data accuracy.
Layers:
First2D
layers = [
        nn.Conv2d(in_channels, middle_channels, kernel_size=3, padding=1),
        nn.BatchNorm2d(middle_channels),
        nn.ReLU(inplace=True),
        nn.Conv2d(middle_channels, out_channels, kernel_size=3, padding=1),
        nn.BatchNorm2d(out_channels),
        nn.ReLU(inplace=True)
    ]

Encoder2D
layers = [
        nn.MaxPool2d(kernel_size=downsample_kernel),
        nn.Conv2d(in_channels, middle_channels, kernel_size=3, padding=1),
        nn.BatchNorm2d(middle_channels),
        nn.ReLU(inplace=True),
        nn.Conv2d(middle_channels, out_channels, kernel_size=3, padding=1),
        nn.BatchNorm2d(out_channels),
        nn.ReLU(inplace=True)
    ]

Center2D
layers = [
        nn.MaxPool2d(kernel_size=2),
        nn.Conv2d(in_channels, middle_channels, kernel_size=3, padding=1),
        nn.BatchNorm2d(middle_channels),
        nn.ReLU(inplace=True),
        nn.Conv2d(middle_channels, out_channels, kernel_size=3, padding=1),
        nn.BatchNorm2d(out_channels),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(out_channels, deconv_channels, kernel_size=2, stride=2)
    ]

Decoder2D
layers = [
        nn.Conv2d(in_channels, middle_channels, kernel_size=3, padding=1),
        nn.BatchNorm2d(middle_channels),
        nn.ReLU(inplace=True),
        nn.Conv2d(middle_channels, out_channels, kernel_size=3, padding=1),
        nn.BatchNorm2d(out_channels),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(out_channels, deconv_channels, kernel_size=2, stride=2)
    ]

Last2D
layers = [
        nn.Conv2d(in_channels, middle_channels, kernel_size=3, padding=1),
        nn.BatchNorm2d(middle_channels),
        nn.ReLU(inplace=True),
        nn.Conv2d(middle_channels, middle_channels, kernel_size=3, padding=1),
        nn.BatchNorm2d(middle_channels),
        nn.ReLU(inplace=True),
        nn.Conv2d(middle_channels, out_channels, kernel_size=1),
        nn.Softmax(dim=1)
    ]


title: Training pytorch model in google colab resulted in lower precision model vs training on local gpu, how to fix?

body: I recently discovered Google Colab and I uploaded my Pytorch lightning project related to training models for processing audio. After a few modifications I got it training models using TPUs, but I noticed that the models were less accurate than the ones I trained on my local machine by about half (loss=0.02 vs 0.009). I used the same hyperparams in both cases.  After some searching I read that colab uses their own type called "bfloat16" instead of float32 precision to increase speed, not sure if this is part of the problem.  Is there any obvious reason why the TPU trained model would turn out less accurate? My model uses a stack of 1d convolutional layers, if that matters.


title: The results of tf.pow and torch.pow are different

body:
I am trying to convert PyTorch scripts to Tensorflow.
This is the situation.
>>> a=np.random.rand(500,500)
>>> b=tf.pow(tf.constant(a, tf.float32),2)
>>> c=torch.pow(torch.tensor(a, dtype=torch.float32),2)
>>> np.sum(b.numpy() - c.numpy())
3.2455164e-06

I guess this difference is caused by floating-point representations between the two platform. (I am not sure)
Questions
Do I need to make them exactly the same with each other?
If so how do solve this problem?


title: Randomly select items from two equally sized tensors

body:
Assume that we have two equally sized tensors of size batch_size * 1. For each index in the batch dimension we want to choose randomly between the two tensors. My solution was to create an indices tensor that contains random 0 or 1 indices of size batch_size and use those to index_select from the concatenation of the two tensors. However, to do so I had the "view" that cat tensor and the solution ended up to be quite "ugly":
import torch

bs = 8
a = torch.zeros(bs, 1)
print("a size", a.size())
b = torch.ones(bs, 1)

c = torch.cat([a, b], dim=-1)
print(c)
print("c size", c.size())

# create bs number of random 0 and 1's
indices = torch.randint(0, 2, [bs])
print("idxs size", indices.size())
print("idxs", indices)

# use `indices` to slice the `cat`ted tensor
d = c.view(1, -1).index_select(-1, indices).view(-1, 1)
print("d size", d.size())
print(d)

I am wondering whether there is a prettier and, more importantly, more efficient solution.