title: PyTorch Net.train() fails to establish connection - what for?

body:
I am trying to train a neural network in PyTorch following a git tutorial. When it comes to train the network I get a row of exceptions linked to a failed connection attempt (probably with some google server):

 Traceback (most recent call last):
  File "/home/***/.local/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/***/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/home/***/.conda/envs/cv-nd/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/***/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/***/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/***/.conda/envs/cv-nd/lib/python3.6/http/client.py", line 1262, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/***/.conda/envs/cv-nd/lib/python3.6/http/client.py", line 1308, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/***/.conda/envs/cv-nd/lib/python3.6/http/client.py", line 1257, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/***/.conda/envs/cv-nd/lib/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/home/*.*/.conda/envs/cv-nd/lib/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/home/*.*/.local/lib/python3.6/site-packages/urllib3/connection.py", line 184, in connect
    conn = self._new_conn()
  File "/home/*.*/.local/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff4d4cc87f0>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/*.*/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/*.*/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/*.*/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/attributes/keep_alive_token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff4d4cc87f0>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "NB2.py", line 229, in <module>
    with active_session():
  File "/home/*.*/.conda/envs/cv-nd/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/*.*/Code/faceNNtrain/Facial_Keypoint_Detection/workspace_utils.py", line 31, in active_session
    token = requests.request("GET", TOKEN_URL, headers=TOKEN_HEADERS).text
  File "/home/*.*/.local/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/*.*/.local/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/*.*/.local/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/*.*/.local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/attributes/keep_alive_token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff4d4cc87f0>: Failed to establish a new connection: [Errno -2] Name or service not known',))

I am confused since there is no external call required, and it is nowhere defined in my code (including model definition) - it probably sits somewhere hidden in the libraries, but still, it fails, messes everything up and idk how to fix it. I checked my local firewall and all, but the only host from the errors I could check (metadata.google.internal) is also not reachable from other clients. Any ideas?


title: Is this Neural Net example I'm looking at a mistake or am I not understanding backprop?

body:
Is this model using one relu in two places, or are gradients computed by doing a matrix multiplication of layers on both sides of one layer?

In the last layer of this simple neural net (below) during back prop it calculates the gradient for the last layer w2 by doing a matrix multiplication of y prediction - y and h_relu, which I thought was only between layers w1 and w2 not between w2 and y_pred

The line in question is near the bottom. It is grad_w2 = h_relu.t().mm(grad_y_pred).
I am confused because I thought everything was supposed to go in order forward and go in order backwards. Is this relu being used in two places?

Here is an attempt at a visual illustration of the model.

This example is from the Pytorch website. It is the second block of code on the page.

grad_w2 = h_relu.t().mm(grad_y_pred)

import torch

dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# Randomly initialize weights
w1 = torch.randn(D_in, H, device=device, dtype=dtype)
w2 = torch.randn(H, D_out, device=device, dtype=dtype)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y
    h = x.mm(w1)
    h_relu = h.clamp(min=0)
    y_pred = h_relu.mm(w2)

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum().item()
    if t % 100 == 99:
        print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.t().mm(grad_y_pred)
    grad_h_relu = grad_y_pred.mm(w2.t())
    grad_h = grad_h_relu.clone()
    grad_h[h < 0] = 0
    grad_w1 = x.t().mm(grad_h)

    # Update weights using gradient descent
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2

I appreciate your patience looking at this and trying to clear this up for me.
If you can try adding another layer of whieghts in the middle with another relu that might help me understand. This is what I was trying to do.


title: Accuracy changes drastically on changing from CPU to GPU

body:
So I was working on the SIIM Melanoma classification data on Kaggle. While training a number of networks I found that when I trained them on CPU, accuracy seemed to be appropriate, around 0.75. On switching to GPU, the accuracy would oscillate in and around 0.5. What should I do about this? Here's a code snippet of the training loop. Model trained finally was resnext50.
import cv2
device = "cpu"
import torch.nn.functional as F
epochs=3
#model = torch.load("model.pt")
model.cpu()
#model.cuda()
print("======== Training for ", epochs, "epochs=============")
for epoch in range(epochs):
    total_loss = 0
    model.train()
    print("Training.......")
    print("======== EPOCH #",epoch,"=================")
    tmp_acc = 0
    for i,batch in enumerate(train_loader):

        img,label = batch["images"],batch["labels"]
        #img = img.permute(0,3,1,2)
        #img = torch.Tensor(img)
        label = label.type(torch.FloatTensor)

        img,label = img.to(device),label.to(device)

        model.zero_grad()

        op = model(img)

        label_cpu = label.cpu().numpy()

        op = F.sigmoid(op)

        output = op.detach().cpu().numpy()
        tmp_acc += accuracy_score(output,label_cpu)
        loss = criterion(op,label)
        total_loss = loss.item()
        loss.backward()

        adam.step()
        if(i%10==0 and i>0):
            print("STEP: ",i, "of steps ",len(train_loader))

            print("Current loss: ",total_loss/i)
            print("Training Accuracy ",tmp_acc/i)

    avg_loss = total_loss/len(train_loader)
    print("The loss after ",epoch," epochs is ",avg_loss)
    print("OP",op)
    print("Label",label_cpu)

torch.save(model.state_dict(),"/kaggle/working/model.pt")


title: Volatile was removed and now has no effect. Use `with torch.no_grad():` instead

body:
I tried to using Show And Tell model with COCO dataset. But, at one point my Torch program paused. Can anyone fix my code? Here is my code which causes the training to pause.
def to_var(x, volatile=True):
if torch.cuda.is_available():
    x = x.cuda()
return Variable(x, volatile=volatile)

And the warning was below.

utils.py:114: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead. return Variable(x, volatile=volatile)


title: Shouldn't same neural network weights produce same results?

body:
So I am working with different deep learning frameworks as part of my research and have observed something weird (at least I cannot explain the cause of it).
I trained a fairly simple MLP model (on mnist dataset) in Tensorflow, extracted trained weights, created the same model architecture in PyTorch and applied the trained weights to PyTorch model. Now my expectation is to get same test accuracy from both Tensorflow and PyTorch models but this isn't the case. I get different results.
So my question is: If a model is trained to some optimal value, shouldn't the trained weights produce same results every time testing is done on the same dataset (regardless of the framework used)?
PyTorch Model:
class Net(nn.Module):

def __init__(self) -> None:
    super(Net, self).__init__()
    self.fc1 = nn.Linear(784, 24)
    self.fc2 = nn.Linear(24, 10)

def forward(self, x: Tensor) -> Tensor:
    x = torch.flatten(x, 1)
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return x

Tensorflow Model:
def build_model() -> tf.keras.Model:
    # Build model layers
    model = models.Sequential()
    # Flatten Layer
    model.add(layers.Flatten(input_shape=(28,28)))
    # Fully connected layer
    model.add(layers.Dense(24, activation='relu'))
    model.add(layers.Dense(10))
    # compile the model
    model.compile(
        optimizer='sgd',
        loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=['accuracy']
    )
    # return newly built model
    return model

To extract weights from Tensorflow model and apply them to Pytorch model I use following functions:
Extract Weights:
def get_weights(model):
    # fetch latest weights
    weights = model.get_weights()
    # transpose weights
    t_weights = []
    for w in weights:
        t_weights.append(np.transpose(w))
    # return
    return t_weights

Apply Weights:
def set_weights(model, weights):
    """Set model weights from a list of NumPy ndarrays."""
    state_dict = OrderedDict(
        {k: torch.Tensor(v) for k, v in zip(model.state_dict().keys(), weights)}
    )
    self.load_state_dict(state_dict, strict=True)


title: Switch from LBFGS to ADAM optimizer in the middle of training in pytorch

body:
I want to switch the optimizer in the middle of the training
from LBFGS to Adam.
Below is the code that I wrote.
optimizer = optim.LBFGS(model.parameters(), lr=0.003)
Use_Adam_optim_FirstTime=True
Use_LBFGS_optim=True

for epoch in range(30000):
    loss_SUM = 0
    for i, (x, t) in enumerate(GridLoader):
        x = x.to(device)
        t = t.to(device)

        if Use_LBFGS_optim:
          def closure():
            optimizer.zero_grad()
            lg, lb, li = problem_formulation(x, t, x_Array,t_Array,bndry,pi)
            loss_total=lg+ lb+ li
            loss_total.backward(retain_graph=True)
            return loss_total
          loss_out=optimizer.step(closure)
          loss_SUM+=loss_out.item()

        elif Use_Adam_optim_FirstTime:
          Use_Adam_optim_FirstTime=False
          optimizerAdam = optim.Adam(model.parameters(), lr=0.0003)
          model.load_state_dict(checkpoint['model'])
          optimizerAdam.zero_grad()
          lg, lb, li = problem_formulation(x, t, x_Array,t_Array,bndry,pi)
          lg.backward()
          lb.backward()
          li.backward()
          optimizerAdam.step()
          loss_SUM += lg.item()+lb.item()+li.item()

        else:
          optimizerAdam.zero_grad()
          lg, lb, li = problem_formulation(x, t, x_Array,t_Array,bndry,pi)
          lg.backward()
          lb.backward()
          li.backward()
          optimizerAdam.step()
          loss_SUM += lg.item()+lb.item()+li.item()

    if loss_SUM<.3 and use_LBFGS_optim == True:
      Use_LBFGS_optim=False
      checkpoint = {'model': model.state_dict(),
                    'optimizer': optimizer.state_dict()}

My questions are:
Regarding the closure function:
(1) Is there a way that I can return more than one variable from the closure function?
(2) Is there a way that I can make three backwards in the closure function instead of only one?
(3) Why do I need to set retain_graph to True in loss_total.backward(retain_graph=True) of the closure function
(4) Sometimes, the loss_SUM for the closure function approaches to 1e+29. Is there a way to avoid this problem?
#-----------------------------
Regarding the switch from LBFGS to Adam:
(5) When loss_SUM<0.3 and we go to the "elif" part, the loss reduces however after one epoch,
loss_SUM dramatically increases (e.g., 20 times)
What is the correct way of switching the optimizer from LBFGS to Adam
In general, what are the problems of the above code, and how I can improve it?


title: How to apply distance IoU loss?

body:
I'm currently training custom dataset using this repository: https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch.
The result of training is not satisfactory for me, so I'm gonna change the regression loss, which is L1-smooth loss, into distance IoU loss.
The code for regresssion loss for this repo is below:
                anchor_widths_pi = anchor_widths[positive_indices]
                anchor_heights_pi = anchor_heights[positive_indices]
                anchor_ctr_x_pi = anchor_ctr_x[positive_indices]
                anchor_ctr_y_pi = anchor_ctr_y[positive_indices]

                gt_widths = assigned_annotations[:, 2] - assigned_annotations[:, 0]
                gt_heights = assigned_annotations[:, 3] - assigned_annotations[:, 1]
                gt_ctr_x = assigned_annotations[:, 0] + 0.5 * gt_widths
                gt_ctr_y = assigned_annotations[:, 1] + 0.5 * gt_heights

                # efficientdet style
                gt_widths = torch.clamp(gt_widths, min=1)
                gt_heights = torch.clamp(gt_heights, min=1)

                targets_dx = (gt_ctr_x - anchor_ctr_x_pi) / anchor_widths_pi
                targets_dy = (gt_ctr_y - anchor_ctr_y_pi) / anchor_heights_pi
                targets_dw = torch.log(gt_widths / anchor_widths_pi)
                targets_dh = torch.log(gt_heights / anchor_heights_pi)

                targets = torch.stack((targets_dy, targets_dx, targets_dh, targets_dw))
                targets = targets.t()

                # L1 loss
                regression_diff = torch.abs(targets - regression[positive_indices, :])

                regression_loss = torch.where(
                    torch.le(regression_diff, 1.0 / 9.0),
                    0.5 * 9.0 * torch.pow(regression_diff, 2),
                    regression_diff - 0.5 / 9.0

The code that i'm using as distance IoU is below:
    rows = bboxes1.shape[0]
    cols = bboxes2.shape[0]
    dious = torch.zeros((rows, cols))
    if rows * cols == 0:
        return dious
    exchange = False

    bboxes1 = bboxes1.index_select(1, torch.LongTensor([1, 0, 3, 2]).to('cuda'))

    if bboxes1.shape[0] > bboxes2.shape[0]:
        bboxes1, bboxes2 = bboxes2, bboxes1
        dious = torch.zeros((cols, rows))
        exchange = True

    w1 = bboxes1[:, 2] - bboxes1[:, 0]
    h1 = bboxes1[:, 3] - bboxes1[:, 1]
    w2 = bboxes2[:, 2] - bboxes2[:, 0]
    h2 = bboxes2[:, 3] - bboxes2[:, 1]

    area1 = w1 * h1
    area2 = w2 * h2

    center_x1 = (bboxes1[:, 2] + bboxes1[:, 0]) / 2
    center_y1 = (bboxes1[:, 3] + bboxes1[:, 1]) / 2
    center_x2 = (bboxes2[:, 2] + bboxes2[:, 0]) / 2
    center_y2 = (bboxes2[:, 3] + bboxes2[:, 1]) / 2

    inter_max_xy = torch.min(bboxes1[:, 2:],bboxes2[:, 2:])
    inter_min_xy = torch.max(bboxes1[:, :2],bboxes2[:, :2])
    out_max_xy = torch.max(bboxes1[:, 2:],bboxes2[:, 2:])
    out_min_xy = torch.min(bboxes1[:, :2],bboxes2[:, :2])

    inter = torch.clamp((inter_max_xy - inter_min_xy), min=0)
    inter_area = inter[:, 0] * inter[:, 1]
    inter_diag = (center_x2 - center_x1)**2 + (center_y2 - center_y1)**2
    outer = torch.clamp((out_max_xy - out_min_xy), min=0)
    outer_diag = (outer[:, 0] ** 2) + (outer[:, 1] ** 2)
    union = area1+area2-inter_area
    dious = inter_area / union - (inter_diag) / outer_diag
    dious = torch.clamp(dious,min=-1.0,max = 1.0)
    if exchange:
        dious = dious.T

    loss = 1 - dious
    return loss

The question is here:

Should I apply the distance IoU loss for one target bbox to all pred bbox?
For example, there is 2 annotated bboxes, and 1000 predicted bboxes.
Should I calculate losses for twice like each annotated bbox vs 1000 predicted bboxes?

Should I change the predicted bbox into real coordinates for calculation?


title: DistributedSampler — Expected a ‘cuda’ device type for generator when generating indices

body:
Performing distributed training, I have the following code like this:
training_sampler = DistributedSampler(training_set, num_replicas=2, rank=0)
training_generator = data.DataLoader(training_set, **params, sampler=training_sampler)
for x, y, z in training_generator:  # Error occurs here.
    ...

Overall, I get the following message:
-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/ubuntu/VC/ppg_training_extraction/ppg_training_scripts/train_ASR_trim_scp.py", line 336, in train
    for local_batch_src, local_batch_tgt, lengths in dataloaders[phase]:
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 352, in __iter__
    return self._get_iterator()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 294, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 827, in __init__
    self._reset(loader, first_iter=True)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 857, in _reset
    self._try_put_index()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1091, in _try_put_index
    index = self._next_index()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 427, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 227, in __iter__
    for idx in self.sampler:
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/distributed.py", line 97, in __iter__
    indices = torch.randperm(len(self.dataset), generator=g).tolist()  # type: ignore
RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'

Now at that line, I ran the following instructions in pdb:
(Pdb) g = torch.Generator()
(Pdb) g.manual_seed(0)
<torch._C.Generator object at 0x7ff7f8143110>
(Pdb) indices = torch.randperm(4556, generator=g).tolist()
(Pdb) indices = torch.randperm(455604, generator=g).tolist()
*** RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'

Why am I getting the runtime error when the upperbound integer is high, but not when it’s low enough?
Note, I ran on a clean Python session and found
>>> import torch
>>> g = torch.Generator()
>>> g.manual_seed(0)
<torch._C.Generator object at 0x7f9d2dfb39f0>
>>> indices = torch.randperm(455604, generator=g).tolist()

that this worked fine. Is it some configuration in how I’m handling distributed training among multiple GPUs? Any sort of insights would be appreciated!


title: Multiplying two 3D Pytorch tensors iteratively

body:
I have two 3 dimensional Pytorch tensors, one of dimension (8, 1, 1024) and the other has dimension (8, 59, 77). I wish to multiply these two tnesors.
I know they cannot be multiplied in their current state, so I want to multiply them iteratively and append into a single tensor.
The second tensor can be represented as (8, 59, 1) when we iterate over the 2nd dimension. In this state multiplying it with the first tensor of shape (8, 1, 1024), resulting in a tensor of shape (8, 59, 1024), and finally appending all these 77 outputs into one, resulting in the final shape of (8, 59, 1024, 77).
However I am having issues in it's implementation. Can someone help me here ?


title: How to prevent memory use growth when updating weights and biases in a Pytorch model

body:
I'm trying to build a VGG16 model to make an ONNX export using Pytorch. I want to force the model with my own set of weights and biases. But in this process my computer quickly runs out of memory.
Here is how I want to do it (this is only a test, in the real version I read the weights and biases in a set of files), this example only force all values to 0.5
# Create empty VGG16 model (random weights)
from torchvision import models
from torchsummary import summary

vgg16 = models.vgg16()
# la structure est : vgg16.__dict__
summary(vgg16, (3, 224, 224))

#  convolutive layers
for layer in vgg16.features:
    print()
    print(layer)
    if (hasattr(layer,'weight')):
        dim = layer.weight.shape
        print(dim)
        print(str(dim[0]*(dim[1]*dim[2]*dim[3]+1))+' params')

        # Remplacement des poids et biais
        for i in range (dim[0]):
            layer.bias[i] = 0.5
            for j in range (dim[1]):
                for k in range (dim[2]):
                    for l in range (dim[3]):
                        layer.weight[i][j][k][l] = 0.5

# Dense layers
for layer in vgg16.classifier:
    print()
    print(layer)
    if (hasattr(layer,'weight')):
        dim = layer.weight.shape
        print(str(dim)+' --> '+str(dim[0]*(dim[1]+1))+' params')
        for i in range(dim[0]):
            layer.bias[i] = 0.5
            for j in range(dim[1]):
                layer.weight[i][j] = 0.5

When I look at the memory usage of the computer, it grows linealrly and saturates the 16GB RAM during the first dense layer processing. Then python crashes...
Is there another better way to do this, keeping in mind that I want to onnx export the model afterwards?
Thanks for your help.


title: How to randomly mix two PyTorch tensors

body: I have two same-shaped PyTorch tensors A and B, and I'd like to create a same-shape "randomly mixed" tensor C where C[i,...] = A[i,...] with probability alpha or B[i,...] with probability 1-alpha.  Is there some Pythonic way to do this compactly?


title: Custom loss function depending on batch (Tensorflow, pytorch)

body:
This is a general Question. I know how to write a custom loss function for Tensorflow that works the following way.
Lets say we have a custom loss function L(y_true,y_pred) , a Neural Network NN(x) and some batch {x_1,...,x_n} with  and ground truth {y_true_1,..., y_true_n}.
Now Tensorflow would minimize the following objective with gradient descent:
L(y_true_1,NN(x_1))+...+L(y_true_1,NN(x_n))

My Question is:
Is it also possible (in TensorFlow, PyTorch or wherever) to define a Loss function L_2 that is defined on the batch so that the following objective is minimized by gradient descent:
L(y_true_1,NN(x_1),...,y_true_n,NN(x_n))

So basically I dont want that sum from the first objective and rather define my own function for the batch.
Thanks for help!


title: Flair Elmo Embeddings use in Pytorch model

body:
I am trying to generate the elmo embeddings on a PyTorch model, on every batch iteration, like:
for batch in iterator:
        optimizer.zero_grad()
        embeddings = get_elmo_embeddings(batch.dataset.examples)
        predictions = model(embeddings).squeeze(1)
        target = batch.target

where the get elmo embeddings is something like:

from flair.embeddings import StackedEmbeddings
from flair.embeddings import ELMoEmbeddings
from flair.data import Sentence

elmo_embedding = ELMoEmbeddings('original')
stacked_embeddings = StackedEmbeddings(embeddings = [elmo_embedding])

def get_elmo_embeddings(text):
    flat_list = list()
    for x in text:
      sentence = Sentence(x.text)
      stacked_embeddings.embed(sentence)

    return elmo(stacked_embeddings)

in order to feed that embeddings to the model, but the problem is that for a single batch it takes hours to create them.. am I doing something wrong? any suggestion of how to make it better?
Also, when I feed it to the model can it go directly to a GRU or it needs some more processing?
Thank all for your time :)


title: How do I use a .pickle file to predict an image?

body:
I have trained a CNN model in PyTorch to detect skin diseases in 6 different classes. My model came out with an accuracy of 92% and I saved it in a .pickle file. I wish to use this model for predictions but I don't know how to do so. If anyone can aid me in the necessary steps, I will be grateful. I have tried using Streamlit but apparently, Streamlit does not work anymore so I am opting for an offline solution where I can just upload an image and the model will give me a prediction of such.
Here is the code for my model. I used a pre-trained ResNet18 model and trained it on the Skin Cancer MNIST: HAM10000 dataset from Kaggle.
def set_parameter_requires_grad(model, feature_extracting):
    if feature_extracting:
        for param in model.parameters():
            param.requires_grad = False

def initialize_model(model_name, num_classes, feature_extract, use_pretrained=True):
    # Initialize these variables which will be set in this if statement. Each of these
    #   variables is model specific.
    model_ft = None
    input_size = 0

    if model_name == "resnet":
        """ Resnet18, resnet34, resnet50, resnet101
        """
        model_ft = models.resnet18(pretrained=use_pretrained)
        set_parameter_requires_grad(model_ft, feature_extract)
        num_ftrs = model_ft.fc.in_features
        model_ft.fc = nn.Linear(num_ftrs, num_classes)
        input_size = 224

    elif model_name == "vgg":
        """ VGG11_bn
        """
        model_ft = models.vgg11_bn(pretrained=use_pretrained)
        set_parameter_requires_grad(model_ft, feature_extract)
        num_ftrs = model_ft.classifier[6].in_features
        model_ft.classifier[6] = nn.Linear(num_ftrs,num_classes)
        input_size = 224

    elif model_name == "densenet":
        """ Densenet121
        """
        model_ft = models.densenet121(pretrained=use_pretrained)
        set_parameter_requires_grad(model_ft, feature_extract)
        num_ftrs = model_ft.classifier.in_features
        model_ft.classifier = nn.Linear(num_ftrs, num_classes)
        input_size = 224

    elif model_name == "inception":
        """ Inception v3
        """
        model_ft = models.inception_v3(pretrained=use_pretrained)
        set_parameter_requires_grad(model_ft, feature_extract)
        # Handle the auxilary net
        num_ftrs = model_ft.AuxLogits.fc.in_features
        model_ft.AuxLogits.fc = nn.Linear(num_ftrs, num_classes)
        # Handle the primary net
        num_ftrs = model_ft.fc.in_features
        model_ft.fc = nn.Linear(num_ftrs,num_classes)
        input_size = 299

    else:
        print("Invalid model name, exiting...")
        exit()
    return model_ft, input_size

# resnet,vgg,densenet,inception
model_name = 'resnet'
num_classes = 7
feature_extract = False
# Initialize the model for this run
model_ft, input_size = initialize_model(model_name, num_classes, feature_extract, use_pretrained=True)
# Define the device:
device = torch.device('cuda:0')
# Put the model on the device:
model = model_ft.to(device)

# norm_mean = (0.49139968, 0.48215827, 0.44653124)
# norm_std = (0.24703233, 0.24348505, 0.26158768)
# define the transformation of the train images.
train_transform = transforms.Compose([transforms.Resize((input_size,input_size)),transforms.RandomHorizontalFlip(),
                                      transforms.RandomVerticalFlip(),transforms.RandomRotation(20),
                                      transforms.ColorJitter(brightness=0.1, contrast=0.1, hue=0.1),
                                        transforms.ToTensor(), transforms.Normalize(norm_mean, norm_std)])
# define the transformation of the val images.
val_transform = transforms.Compose([transforms.Resize((input_size,input_size)), transforms.ToTensor(),
                                    transforms.Normalize(norm_mean, norm_std)])

# Define a pytorch dataloader for this dataset
class HAM10000(Dataset):
    def __init__(self, df, transform=None):
        self.df = df
        self.transform = transform

    def __len__(self):
        return len(self.df)

    def __getitem__(self, index):
        # Load data and get label
        X = Image.open(self.df['path'][index])
        y = torch.tensor(int(self.df['cell_type_idx'][index]))

        if self.transform:
            X = self.transform(X)

        return X, y

# Define the training set using the table train_df and using our defined transitions (train_transform)
training_set = HAM10000(df_train, transform=train_transform)
train_loader = DataLoader(training_set, batch_size=64, shuffle=True, num_workers=4)
# Same for the validation set:
validation_set = HAM10000(df_val, transform=train_transform)
val_loader = DataLoader(validation_set, batch_size=64, shuffle=False, num_workers=4)

# we use Adam optimizer, use cross entropy loss as our loss function
optimizer = optim.Adam(model.parameters(), lr=1e-5)
criterion = nn.CrossEntropyLoss().to(device)

Here are the training process and the save file.
class AverageMeter(object):
    def __init__(self):
        self.reset()

    def reset(self):
        self.val = 0
        self.avg = 0
        self.sum = 0
        self.count = 0

    def update(self, val, n=1):
        self.val = val
        self.sum += val * n
        self.count += n
        self.avg = self.sum / self.count

total_loss_train, total_acc_train = [],[]
def train(train_loader, model, criterion, optimizer, epoch):
    model.train()
    train_loss = AverageMeter()
    train_acc = AverageMeter()
    curr_iter = (epoch - 1) * len(train_loader)
    for i, data in enumerate(train_loader):
        images, labels = data
        N = images.size(0)
        # print('image shape:',images.size(0), 'label shape',labels.size(0))
        images = Variable(images).to(device)
        labels = Variable(labels).to(device)

        optimizer.zero_grad()
        outputs = model(images)

        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        prediction = outputs.max(1, keepdim=True)[1]
        train_acc.update(prediction.eq(labels.view_as(prediction)).sum().item()/N)
        train_loss.update(loss.item())
        curr_iter += 1
        if (i + 1) % 100 == 0:
            print('[epoch %d], [iter %d / %d], [train loss %.5f], [train acc %.5f]' % (
                epoch, i + 1, len(train_loader), train_loss.avg, train_acc.avg))
            total_loss_train.append(train_loss.avg)
            total_acc_train.append(train_acc.avg)
    return train_loss.avg, train_acc.avg

def validate(val_loader, model, criterion, optimizer, epoch):
    model.eval()
    val_loss = AverageMeter()
    val_acc = AverageMeter()
    with torch.no_grad():
        for i, data in enumerate(val_loader):
            images, labels = data
            N = images.size(0)
            images = Variable(images).to(device)
            labels = Variable(labels).to(device)

            outputs = model(images)
            prediction = outputs.max(1, keepdim=True)[1]

            val_acc.update(prediction.eq(labels.view_as(prediction)).sum().item()/N)

            val_loss.update(criterion(outputs, labels).item())

    print('------------------------------------------------------------')
    print('[epoch %d], [val loss %.5f], [val acc %.5f]' % (epoch, val_loss.avg, val_acc.avg))
    print('------------------------------------------------------------')
    return val_loss.avg, val_acc.avg

if os.path.exists("Tested_model2.pickle"):
  print("Loading Trained Model")
  model = pickle.load(open("Tested_model2.pickle", "rb"))
  print(model)
else:
  print("Training New Model.")
  print("Training begins.")
  print("********************************************************")
  epoch_num = 25
  load_model = True
  best_val_acc = 0
  total_loss_val, total_acc_val = [],[]
  for epoch in range(1, epoch_num+1):
    loss_train, acc_train = train(train_loader, model, criterion, optimizer, epoch)
    loss_val, acc_val = validate(val_loader, model, criterion, optimizer, epoch)
    total_loss_val.append(loss_val)
    total_acc_val.append(acc_val)
    if acc_val > best_val_acc:
        best_val_acc = acc_val
        print('*****************************************************')
        print('best record: [epoch %d], [val loss %.5f], [val acc %.5f]' % (epoch, loss_val, acc_val))
        print('*****************************************************')

with open ("Tested_model2.pickle", "wb") as file:
  pickle.dump(model, file)

To put it simply, I wanna know how to use the pickle file for predictions.
Edit: I added the evaluation part in the following, please help me to understand how can I proceed further with this code.
model.eval()
y_label = []
y_predict = []
with torch.no_grad():
    for i, data in enumerate(val_loader):
        images, labels = data
        N = images.size(0)
        images = Variable(images).to(device)
        outputs = model(images)
        prediction = outputs.max(1, keepdim=True)[1]
        y_label.extend(labels.cpu().numpy())
        y_predict.extend(np.squeeze(prediction.cpu().numpy().T))

Also, This is the code that I previously used to load and make predictions, however I did not get to know whether the code is correct or not or the method is right.
%%writefile app.py
import streamlit as st
import torch

st.set_option('deprecation.showfileUploaderEncoding', False)
@st.cache(allow_output_mutation=True)

def load_model():
  model = pickle.load(open("Trained_Model_part2.pickle", "rb"))
  return mdoel

model = load_model()
st.write("""
         #Classification of skin disease
         """)

file = st.file_uploader("Please upload the image of the affected area.", type = ["jpg", "png"])

import cv2
from PIL import Image, ImageOps
import numpy as np

def import_and_predict(image_data, model):
  size = (224, 224)
  image = ImageOps.fit(image_data, size, Image.ANTIALIAS)
  img = np.asarray(image)
  image_reshape = img[np.newaxis,...]
  prediction = model.predict(img_reshape)

  return prediction

if file is None:
  st.text("Please upload an image file.")
else:
  image = Image.open(file)
  st.image(image, use_column_width = True)
  predictions = import_and_predict(image, model)
  class_names = ["Melanocytic nevi", "Melanoma", "Benign keratosis-like lesions", "Basal cell carcinoma", "Actinic keratoses", "Vascular lesions", "Dermatofibroma"]
  string = "It is: " + class_names[np.argmax(predictions)]
  st.success(string)

This uses streamlit and the loader is the previous pickle file loader which will be replaced by a .pth loader. I want to know what changes do I have to make so the code will ask for an image input or look for an image in the specific folder and deliver a prediction. Thank you.


title: Python3 Out of Memory Error but Linux not using full RAM

body:
I have a python3 program (that uses pytorch) that is receiving an "Out of Memory" error. When looking at free -t and htop I see that there is still RAM and swap available. On htop I see that my Virtual Memory is larger than my available RAM + swap.
I have looked around for answers, as it seems to be a problem that can be common, and I found some things that helped me understand more and look further, but ultimately I remained stuck.
My System
I am running 64bit Ubuntu 18.04.2 LTS with kernel 4.15.0-45-generic on a cloud VM service.
My program is using Python 3.6.9 and uses pytorch 1.6 (using torchjit) and tornado 6.1 (not sure if relevant at all...).
My python3 program is a tornado server that receives requests intended to run inference of a neural network.
My Problem
All seems to work well for some hours replying to multiple requests but after some hours seemingly randomly it reaches an "Out Of Memory" Error. This will happen to all requests for a few minutes to a few hours, after-which it will start processing new requests properly again and the cycle goes on.
The specific error I get is generated by SOX and is:
ERROR:sox:OSError: SoX failed! [Errno 12] Cannot allocate memory

But I know that SOX itself doesn't use much memory at all, and this happens to multiple requests in a row so it seems like my general python program is just out of memory (and not that SOX has a memory spike that causes this).
Things I noticed
Looking at htop I get:

Looking at free -t I get:

Looking at df -h:

From what I can tell the Virtual Memory (3003M) far exceeds my RAM + swap (~2236M). At the same time still have at least 330M free between my RAM and my swap. I also still have memory available on my HDD so the Virtual Memory should still have room to allocate more memory to if it wanted (and not running into "Out Of Memory" issues.
From reading about this it is normal that the Virtual Memory exceeds my RAM + swap but I still don't really understand when the virtual memory decides to use which resource and where it stores data when it is not using RAM and swap.
What I have tried

I tried testing the garbage collector in python to see if maybe it was not freeing up memory but from my testing it seems like the garbage collector is doing its job as intended and there isn't some build-up of things to be collected that aren't. On average the garbage collector resets itself (frees the memory) after 2 requests to the server.
I have checked if some automatic system updates were taking up memory and thus creating a momentary system memory shortage - this is not the case as far as I can tell looking at system logs
I monitored htop and my logs in real time while receiving requests and did not notice any memory usage spikes.
Restarting the python program fixes the memory problem when it occurs (until it pops up again), but this is not a good solution for me.
Reproducing the problem manually has proved to be a problem. I tried sending the same requests that were sent to the server between 2 of the times the memory error occurred but this did not reproduce it. This has made solving this a lot harder as in order to have a look at the problem I have to wait for it to happen organically and hope it doesn't go away before I figure things out.

Questions
My main question is does this memory issue seem like a python3 problem with my code (pytorch? torchjit? memory leak? even if it is uncommon in python...) or a Linux problem? How do I go about figuring out what the problem actually is and solving it?
Other questions that have been added along the way of trying to solve this problem:

Why isnt my RAM and swap being fully utilized before my VM allocates memory to my program elsewhere
Where does my VM store the memory that it allocates? If I can find this out I can figure out where the memory shortage is, because as it stands I can not find any part of my system that has 100% memory usage.
I read that maybe my program is creating many "empty memory pages" or things of the sort that may have allocated memory to the program that it no longer needs. Is there a way to clean these things in python3 other than using the garbage collector?

Thank you for your time and any help you may be able to provide. If you need more information from me please let me know.


title: Understanding the usage of nn.Linear for Forward Propagation in PyTorch

body:
The purpose of this study is to build a simplified forward propagation model that reproduces the code structure in PyTorch, yet does not use any of the PyTorch libraries. The idea is to do a matrix multiplication while emulating the code structure, including class definitions as in pyTorch.
PyTorch code for Forward Propagation
from torch import nn
#DEFINE THE REQUIRED CLASS

class Network(nn.Module):
    def __init__(self):
        super().__init__()

        # Inputs to hidden layer linear transformation
        self.hidden = nn.Linear(784, 256)
        # Output layer, 10 units - one for each digit
        self.output = nn.Linear(256, 10)

        # Define sigmoid activation and softmax output
        self.sigmoid = nn.Sigmoid()
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):
        # Pass the input tensor through each of our operations
        x = self.hidden(x)
        x = self.sigmoid(x)
        x = self.output(x)
        x = self.softmax(x)

        return x
#use
model = Network()
...
ps = model.forward(some_tensor)

here is my reproducer code:
class my_mul:
        def __init__(self, h, w):
                self.dim1 = h
                self.dim2 = w
                self.layer = LAYER(self.dim1, self.dim2)
        def forward(self, X):
                X =  self.layer.doit( X)
                return X

class LAYER:
        def __init__(self, h, w):
                self.dim1 = h
                self.dim2 = w
        def __call__(self, Z):
                self.matrix2 = Z
        def doit(self, X):
                self.matrix1 = np.random.rand(self.dim1, self.dim2)
                print('matrix 1 in class LAYER ',self.matrix1)
                X =  np.matmul(self.matrix1, self.matrix2)

import numpy as np
#
np.random.seed(0)
# initialize matrix2 which emulates the tensor being passed down to the forward
# propagation step within a deep network training

matrix2 = np.random.rand(2,2)
print('matrix2 ',matrix2)
#
# use the __call__ method of LAYER to pass matrix2 to LL
LL = LAYER(2,2)
LL(matrix2)

MM = my_mul(2,2)
P = MM.forward(matrix2)
print('the product of the 2 matrices is ', P)

The above 'reproducer code' fails as follows. The problem is that the data is not passed correctly and I don't know how to make it work.
matrix2  [[0.5488135  0.71518937]
 [0.60276338 0.54488318]]
matrix 1 in class LAYER  [[0.4236548  0.64589411]
 [0.43758721 0.891773  ]]
Traceback (most recent call last):
  File "all_s.py", line 30, in <module>
    P = MM.forward(matrix2)
  File "all_s.py", line 7, in forward
    X =  self.layer.doit( X)
  File "all_s.py", line 19, in doit
    X =  np.matmul(self.matrix1, self.matrix2)
AttributeError: 'LAYER' object has no attribute 'matrix2'


title: CUDA RuntimeError when trying to use detectron2

body:
I'm trying to run an example of the detectron2 inference. I've installed torch 1.17.1 with cuda 11.0
pip3 install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html. I've also installed the pre-built detecron2 with the following command  python3 -m pip install detectron2 -f   https://dl.fbaipublicfiles.com/detectron2/wheels/cu110/torch1.7/index.html. nvidia-sumi shows the following:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce 920M        Off  | 00000000:04:00.0 N/A |                  N/A |
| N/A   38C    P0    N/A /  N/A |    259MiB /  2004MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

When I try to run the following code:
import tkinter as tk
from tkinter import filedialog
import cv2

from detectron2 import model_zoo
from detectron2.engine import DefaultPredictor
from detectron2.config import get_cfg
from detectron2.utils.visualizer import Visualizer
from detectron2.data import MetadataCatalog

root = tk.Tk()
root.withdraw()

# Open a file dialog and display the selected image
file_path = filedialog.askopenfilename(title="Choose an image")
input_image = cv2.imread(file_path)

# Show image
cv2.imshow('Input', input_image)

cfg = get_cfg()
# add project-specific config (e.g., TensorMask) here if you're not running a model in detectron2's core library
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5  # set threshold for this model
# Find a model from detectron2's model zoo. You can use the https://dl.fbaipublicfiles... url as well
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
predictor = DefaultPredictor(cfg)
outputs = predictor(input_image)

# look at the outputs. See https://detectron2.readthedocs.io/tutorials/models.html#model-output-format for specification
print(outputs["instances"].pred_classes)
print(outputs["instances"].pred_boxes)

# We can use `Visualizer` to draw the predictions on the image.
v = Visualizer(input_image[:, :, ::-1], MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=1.2)
out = v.draw_instance_predictions(outputs["instances"].to("cpu"))
cv2.imshow('Output', out.get_image()[:, :, ::-1])

# wait for exit
if cv2.waitKey(0) & 0xff == 27:
    cv2.destroyAllWindows()

I get the following RuntimeError:
Traceback (most recent call last):
  File "/home/ion/Documents/Detectron2_test/main.py", line 28, in <module>
    outputs = predictor(input_image)
  File "/home/ion/.local/lib/python3.8/site-packages/detectron2/engine/defaults.py", line 223, in __call__
    predictions = self.model([inputs])[0]
  File "/home/ion/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ion/.local/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 149, in forward
    return self.inference(batched_inputs)
  File "/home/ion/.local/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 197, in inference
    images = self.preprocess_image(batched_inputs)
  File "/home/ion/.local/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 222, in preprocess_image
    images = [(x - self.pixel_mean) / self.pixel_std for x in images]
  File "/home/ion/.local/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 222, in <listcomp>
    images = [(x - self.pixel_mean) / self.pixel_std for x in images]
RuntimeError: CUDA error: no kernel image is available for execution on the device

I don't know if there is a problem with my gpu or something else but I could't figure it out by now.


title: import torch OSError

body:
I have a Jupyter notebook open and I tried to run this one line: import torch. The result was this stack trace:
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-3-eb42ca6e4af3> in <module>
----> 1 import torch

~/.local/lib/python3.6/site-packages/torch/__init__.py in <module>
    187     # See Note [Global dependencies]
    188     if USE_GLOBAL_DEPS:
--> 189         _load_global_deps()
    190     from torch._C import *
    191

~/.local/lib/python3.6/site-packages/torch/__init__.py in _load_global_deps()
    140     lib_path = os.path.join(os.path.dirname(here), 'lib', lib_name)
    141
--> 142     ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
    143
    144

/usr/lib/python3.6/ctypes/__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error)
    346
    347         if handle is None:
--> 348             self._handle = _dlopen(self._name, mode)
    349         else:
    350             self._handle = handle

OSError: /home/xianx/.local/lib/python3.6/site-packages/torch/lib/libtorch_global_deps.so: cannot open shared object file: No such file or directory

I tried to import pytorch this way:
xianx@25b6c9724ef7452b9fc3211ab99037e9-master-0:~$ pip install torch
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.2.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch) (1.18.1)

I did find this answer, but they solved it by using an older version of torch and numpy. I'd prefer to use the most updated version if possible.


title: What is torch.randn((1, 5))?

body:
I'm confused as to why there are double parantheses instead of just torch.randn(1,5).
Is torch.randn(1,5) the same thing as torch.randn((1,5))?


title: How to replace specific values in PyTorch tensor along diagonal?

body:
For example, there is a PyTorch matrix A:
A = tensor([[3,2,1],[1,0,2],[2,2,0]])

I need to replace 0 with 1 on the diagonal, so the result should be:
tensor([[3,2,1],[1,1,2],[2,2,1]])


title: Saving a pytoch tensor as a 32-bit grayscale Image

body:
I have manipulated a 32-bit grayscale .tif image which I converted to tensor using PIL. After this I saved it with:
torchvision.utils.save_image(train_img_poac,fp=str(j)+".tif")

This method automatically converts the tensor to an RGB format image. I want my output image to be a 32-bit grayscale image.
I tried to use the arguments in the save_image function but could not find anything. Is converting it to numpy ndarray and then converting it to a 32-bit Image an option?


title: PyTorch DataLoader uses same random seed for batches run in parallel

body:
There is a bug in PyTorch/Numpy where when loading batches in parallel with a DataLoader (i.e. setting num_workers > 1), the same NumPy random seed is used for each worker, resulting in any random functions applied being identical across parallelized batches.
Minimal example:
import numpy as np
from torch.utils.data import Dataset, DataLoader

class RandomDataset(Dataset):
    def __getitem__(self, index):
        return np.random.randint(0, 1000, 2)

    def __len__(self):
        return 9

dataset = RandomDataset()
dataloader = DataLoader(dataset, batch_size=1, num_workers=3)

for batch in dataloader:
    print(batch)

As you can see, for each parallelized set of batches (3), the results are the same:
# First 3 batches
tensor([[891, 674]])
tensor([[891, 674]])
tensor([[891, 674]])
# Second 3 batches
tensor([[545, 977]])
tensor([[545, 977]])
tensor([[545, 977]])
# Third 3 batches
tensor([[880, 688]])
tensor([[880, 688]])
tensor([[880, 688]])

What is the recommended/most elegant way to fix this? i.e. have each batch produce a different randomization, irrespective of the number of workers.


title: Will tensor.backward computes all the gradient in graph?

body:
I read these codes on github:

# loss1, loss2 belong to the same net
net.zero_grad()
loss1 = ...
loss2 = ...
loss1.backward()
loss2.backward()
optim.step()

which is not a mentioned BP method on pytorch official website, and  official documentation mention Computes the gradient of current tensor w.r.t. graph leaves. for tensor.backward.

So, other gradients expect the two loss tensors are not computed? And no tensors are updated?


title: How can I use pytorch pre-trained model without installing pytorch?

body:
I only want to use pre-trained model in pytorch without installing the whole package.
Can I just copy the model module from pytorch?


title: PyTorch doesn't seem to be optimizing correctly

body:
I have posted this question on Data Science StackExchange site since StackOverflow does not support LaTeX. Linking it here because this site is probably more appropriate.

The question with correctly rendered LaTeX is here: https://datascience.stackexchange.com/questions/48062/pytorch-does-not-seem-to-be-optimizing-correctly

The idea is that I am considering sums of sine waves with different phases. The waves are sampled with some sample rate s in the interval [0, 2pi]. I need to select phases in such a way, that the sum of the waves at any sample point is minimized.

Below is the Python code. Optimization does not seem to be computed correctly.

import numpy as np
import torch

def phaseOptimize(n, s = 48000, nsteps = 1000):
    learning_rate = 1e-3

    theta = torch.zeros([n, 1], requires_grad=True)
    l = torch.linspace(0, 2 * np.pi, s)
    t = torch.stack([l] * n)
    T = t + theta

    for jj in range(nsteps):
        loss = T.sin().sum(0).pow(2).sum() / s
        loss.backward()
        theta.data -= learning_rate * theta.grad.data

    print('Optimal theta: \n\n', theta.data)
    print('\n\nMaximum value:', T.sin().sum(0).abs().max().item())

Below is a sample output.

phaseOptimize(5, nsteps=100)

Optimal theta:

 tensor([[1.2812e-07],
        [1.2812e-07],
        [1.2812e-07],
        [1.2812e-07],
        [1.2812e-07]], requires_grad=True)

Maximum value: 5.0

I am assuming this has something to do with broadcasting in

T = t + theta

and/or the way I am computing the loss function.

One way to verify that optimization is incorrect, is to simply evaluate the loss function at random values for the array $\theta_1, \dots, \theta_n$, say uniformly distributed in $[0, 2\pi]$. The maximum value in this case is almost always much lower than the maximum value reported by phaseOptimize(). Much easier in fact is to consider the case with $n = 2$, and simply evaluate at $\theta_1 = 0$ and $\theta_2 = \pi$. In that case we get:

phaseOptimize(2, nsteps=100)

Optimal theta:

 tensor([[2.8599e-08],
        [2.8599e-08]])

Maximum value: 2.0

On the other hand,

theta = torch.FloatTensor([[0], [np.pi]])
l = torch.linspace(0, 2 * np.pi, 48000)
t = torch.stack([l] * 2)
T = t + theta

T.sin().sum(0).abs().max().item()

produces

3.2782554626464844e-07


title: How do I deploy deep reinforcement learning neural network I coded in Pytorch to my website?

body: I have built and trained neural network in Pytorch and is ready for production to a website but how do I deploy it?


title: How do I resolve this error? AttributeError: 'builtin_function_or_method' object has no attribute 'data'

body:
I'm trying to implement neural style transfer by using the code in this article. When I run main.py I get the error in the title of my question referring to line 12. I'm pretty sure that it has something to do with torch.randn(input.data.size()) but I don't know what to do about it. I'm still a beginner with machine learning and I'm still learning things about Python as well. Any help would be appreciated.

from StyleCNN import *
from utils import *

# CUDA Configurations
dtype = torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor

# Content and style
style = image_loader("style.jpg").type(dtype)
content = image_loader("content.jpg").type(dtype)

pastiche = image_loader("content.jpg").type(dtype)
pastiche.data = torch.randn(input.data.size()).type(dtype)

num_epochs = 31

def main():
    style_cnn = StyleCNN(style, content, pastiche)

    for i in range(num_epochs):
        pastiche = style_cnn.train()

        if i % 10 == 0:
            print("Iteration: %d" % (i))

            path = "outputs/%d.png" % (i)
            pastiche.data.clamp_(0, 1)
            save_image(pastiche, path)

main()


title: No GPU Usage apparent in Google Cloud Vm with pytorch already installed and Cuda10

body:
I have been using in my machine a network, that is nothing really special. I wanted to do it faster so I started using google cloud. But I notice something weird that my machine with a GTX 1050 ti was faster than a V100 GPU. This didn't add up so I checked the usage and it seems that even though I put some stress by creating a big network and passing a lot of data to it the gpu by using a simple .cuda() in both the model and the data: there wasn't ussage shown in nvidia-smi command as shown in the image
you can check my code here:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    print("The device is:",device,torch.cuda.get_device_name(0),"and how many are they",torch.cuda.device_count())

    # # We load the training data
    Samples , Ocupancy, num_samples, Samples_per_slice = common.load_samples(args.samples_filename)
    Samples = Samples * args.scaling_todo

    print(Samples_per_slice)
    # Divide into Slices
    Organize_Positions,Orginezed_Ocupancy, batch_size = common.organize_sample_data(Samples,Ocupancy,num_samples,Samples_per_slice,args.num_batches)

    phi = common.MLP(3, 1).cuda()

    x_test = torch.from_numpy(Organize_Positions.astype(np.float32)).cuda()
    y_test = torch.from_numpy(Orginezed_Ocupancy.astype(np.float32)).cuda()

    all_data = common.CustomDataset(x_test, y_test)

    #Dive into Slices the data
    Slice_data = DataLoader(dataset=all_data, batch_size = batch_size, shuffle=False) # only take batch_size = n/b TODO Don't shuffle

    #Chunky_data = DataLoader(dataset=Slice_data, batch_size =  chunch_size, shuffle=False)

    criterion = torch.nn.BCEWithLogitsLoss()
    optimizer = torch.optim.Adam(phi.parameters(), lr = 0.0001)
    epoch = args.num_epochs

    fit_start_time = time.time()

    phi.train()
    for epoch in range(epoch):
        curr_epoch_loss = 0
        batch = 0
        for x_batch, y_batch in Slice_data:
            optimizer.zero_grad()
            x_train = x_batch
            #print(x_train,batch_size)
            y_train = y_batch

            y_pred = phi(x_train)
            #print(y_pred,x_train)

            loss = criterion(y_pred.squeeze(), y_train.squeeze())

            curr_epoch_loss += loss

            print('Batch {}: train loss: {}'.format(batch, loss.item()))    # Backward pass

            loss.backward()
            optimizer.step() # Optimizes only phi parameters
            batch+=1

        print('Epoch {}: train loss: {}'.format(epoch, loss.item()))

    fit_end_time = time.time()
    print("Total time = %f" % (fit_end_time - fit_start_time))

    # Save Model
    torch.save({'state_dict': phi.state_dict()}, args.model_filename)

and the model here:

class MLP(nn.Module):
    def __init__(self, in_dim: int, out_dim: int):
        super().__init__()
        self.in_dim = in_dim
        self.out_dim = out_dim
        self.fc1 = nn.Linear(in_dim, 128)
        self.fc1_bn = nn.BatchNorm1d(128)
        self.fc2 = nn.Linear(128, 256)
        self.fc2_bn = nn.BatchNorm1d(256)
        self.fc3 = nn.Linear(256, 512)
        self.fc3_bn = nn.BatchNorm1d(512)
        self.fc4 = nn.Linear(512, 512)
        self.fc4_bn = nn.BatchNorm1d(512)
        self.fc5 = nn.Linear(512, out_dim,bias=False)

        self.relu = nn.LeakyReLU()

    def forward(self, x):
        x = self.relu(self.fc1_bn(self.fc1(x)))
        x = self.relu(self.fc2_bn(self.fc2(x)))# leaky
        x = self.relu(self.fc3_bn(self.fc3(x)))
        x = self.relu(self.fc4_bn(self.fc4(x)))
        x = self.fc5(x)
        return x

class CustomDataset(Dataset):
    def __init__(self, x_tensor, y_tensor):
        self.x = x_tensor
        self.y = y_tensor

    def __getitem__(self, index):
        return (self.x[index], self.y[index])

    def __len__(self):
        return len(self.x)


I had a doubt with regard to a problem I'm working on . I'm trying to use Alexnet on the At&t ORL face dataset on google colab.

I'm getting the error while training the model-

  RuntimeError: Expected 4-dimensional input for 4-dimensional weight 96 3 11 11, but got 2-dimensional input of size [30, 50176] instead

Here is the Colab file - Here
Can anyone please suggest what I'm doing wrong.


title: Pytorch Partial derivatives like tf.gradients with stop_gradients parameter

body:
Is there a way to calculate partial derivatives instead of total derivative with Pytorch like tf.gradients with parameter stop_gradients https://www.tensorflow.org/api_docs/python/tf/gradients

Example:

a = torch.tensor(1., requires_grad=True)
b = 2 * a
c = a + b
print(torch.autograd.grad(c, a, retain_graph=True))

Output (tensor(3.),), Expected output: 1 for partial derivative.

I know we can use b = 2*(a.detach()). But what if d=2*c. In that case, I want to keep calculating the total derivative dd/da. I want to get dd/da=6 just like there is no a.detach instead of 2.

Background: I am trying to calculate the last hidden layer partial gradient w.r.t the the first hidden in RNN. Then I still need to calculate loss.backward() where I don't want to detach the hidden layer.

Thank you all.


title: How to parallelize iterations of a tensor multiplying which requires extra indexing, or vectorize it without more memory?

body:
I have to calculate this using Pytorch. But it is very slow on GPU, actually not faster than CPU version.

# n, k           10000000         ( 1E7 )
# a        shape: (n, 100)        ( 1E9 )
# index    shape: (k,  10)        ( 1E8 )   used to select the rows of a
# w        shape: (k,  10)        ( 1E8 )
# result   shape: (k, 100)        ( 1E9 )
#
# result[i] = w[i,0]*a[index[i,0]] + ... + w[i,9]*a[index[i,9]]

for i in range(len(result)):
    result[i] = (w[i].unsqueeze(-1) * a[index[i]]).sum(0)  # shape (10, 1) (10, 100) -> (100)

To vectorize it, a simple way is removing [i].

result = (w.unsqueeze(-1) * a[index]).sum(1)  # shape (1E7, 10, 1) (1E7, 10, 100) -> (1E7, 100)

This version is fast. The problem is, a[index] creates a tensor of shape (k, 10, 100) i.e. (1E10), causing CUDA out of memory. I only need to access a, so the extra memory is unnecessary.

So, how to parallelize the for-loop, or how to solve the memory problem in vectorized version?


title: How to solve this question "RuntimeError: CUDA out of memory."?

body:
I'm going to extract a feature from pictures.I first define a tensor data_feature_map, and then use torch.cat to stack the features of one picture.

My code is :

data_feature_map = torch.ones(1,2048)
for i, data in enumerate(train_loader, 0):
    img, _ = data
    img.requires_grad_=False
    if torch.cuda.is_available():
        img = img.cuda()
    out = model(img)
    # out.shape = [1,2048]
    out = out.view(1,-1).cpu()
    data_feature_map = torch.cat((data_feature_map, out), 0)

but when i run it, it will show the error "RuntimeError: CUDA out of memory."

please tell me why this error occurs.Thank you very much.


title: why does import torch for PyTorch stable(1.5) throws error on macOS

body:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/__init__.py", line 136, in <module>
    from torch._C import *
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/_C.cpython-38-darwin.so, 2): Library not loaded: @rpath/libc++.1.dylib
  Referenced from: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/_C.cpython-38-darwin.so
  Reason: image not found


title: calculate gradient of A * relu(A * X * W0) * W1 w.r.t. A

body:
I'm trying to manually calculate the gradient to do backpropagation for my neural network
the formula for my forward function is  A * relu(A * X * W0) * W1
all A, X, W0, W1 are matrices and I want to get the gradient w.r.t A
I'm using pytorch so it would be great if anyone can show how to get the gradient of this function in pytorch( without using autograd).
Thanks!


title: pytorch: Instance norm implemented by basic operations has different result comparing to torch.nn.InstanceNorm2d

body:
I implement instance norm by pytorch basic operations from scratch. But the result is different from torch.nn.InstanceNorm2d. Can anyone help me out? Below is my code:
import torch
import numpy as np
x = torch.rand((8, 16, 32, 32))
a = torch.nn.InstanceNorm2d(256)
a.eval()
with torch.no_grad():
&nbsp;&nbsp;&nbsp;&nbsp;b = a(x)
x_mean = torch.mean(x, axis=(2,3), keepdims=True)
x_var = torch.var(x, axis=(2,3), keepdims=True)
x_norm = (x - x_mean) / torch.sqrt(x_var + 1e-5)
b_numpy = b.numpy()
x_norm_numpy = x_norm.numpy()
\# check if b_numpy and x_norm_numpy close to the torlerance of 1e-3
print(np.allclose(b_numpy, x_norm_numpy, atol=1e-3))
\# check if b_numpy and x_norm_numpy close to the torlerance of 1e-3
print(np.allclose(b_numpy, x_norm_numpy, atol=1e-4))

Result:
True
False

So, the result shows that when precision comes to 1e-4, they are different. I don’t know why. Can anyone help me to get a more close result to torch.nn.InstanceNorm2d?
BTW, the reason why I do not use formula gamma * x_normalized_numpy + beta in the paper is I find that when the first initialization of torch.nn.InstanceNorm2d, all gamma is initialized to [1.0, 1.0, 1.0, …] and all beta is initialized to [0.0, 0.0, 0.0, …]. So, under this condition, x_normalized_numpy = gamma * x_normalized_numpy + beta


title: How to implement KNN in pytorch

body:
I search for both decision tree and KNN algorithm implementation in pytrch
But didn't get anything about it.


title: How to use one original & one augmented batch from torchvision MNIST dataset with shuffeling, "aligned" batch samples, but different batch size?

body:
I want to implement this situation for the torchvision MNIST dataset, loading data with DataLoader:
batch A (unaugmented images): 5, 0, 4, ...
batch B (augmented images): 5*, 5+, 5-, 0*, 0+, 0-, 4*, 4+, 4-, ...

... where for every image of A there are 3 augmentations in batch B. len(B) = 3*len(A) correspondingly. These batches should be used within a single iteration to compare the original images of batch A with those augmented in batch B to build a loss.
class MyMNIST(Dataset):

def __init__(self, mnist_dir, train, augmented, transform=None, repeat=1):

    self.mnist_dir = mnist_dir
    self.train = train
    self.augmented = augmented
    self.repeat = repeat
    self.transform = transform
    self.dataset = None

    if augmented and train:
        self.dataset = datasets.MNIST(self.mnist_dir, train=train, download=True, transform=transform)
        self.dataset.data = torch.repeat_interleave(self.dataset.data, repeats=self.repeat, dim=0)
        self.dataset.targets = torch.repeat_interleave(self.dataset.targets, repeats=self.repeat, dim=0)
    elif augmented and not train:
        raise Exception("Test set should not be augmented.")
    else:
        self.dataset = datasets.MNIST(MNIST_DIR, train=train, download=True, transform=transform)

With this class, I want to initialize two different dataloaders:
orig_train = MyMNIST(MNIST_DIR, train=True, augmented=False, transform=orig_transforms)
orig_train_loader = torch.utils.data.DataLoader(orig_train.dataset, batch_size=100, shuffle=True)

aug_train = MyMNIST(MNIST_DIR, train=True, augmented=True, transform=aug_transforms, repeat=3)
aug_train_loader = torch.utils.data.DataLoader(aug_train.dataset, batch_size=300, shuffle=True)

My problem now is, I also need to shuffle with each iteration while the order between A and B stays in relation. Which is not possible with above code, as both DataLoader yield different orders. So I tried to work with a single DataLoader and to manually copy a repeated batch:
for batch_no, (images, labels) in enumerate(orig_train_loader):
    repeat_images = torch.repeat_interleave(images, 3, dim=0)

This way, I get the order of batch B (repeat_images) right, but now I´m missing the transformations which I would need to apply within a batch/iteration. This seems not to be the paradigm of Pytorch, at least I did not find a way to do that.
I would be happy if somebody can help me - I am quite new to Pytorch (and also to stackoverflow), so please also be welcome to criticize my whole approach, performance issues that could arise etc.
Thanks a lot!


title: Embedding layer in neural machine translation with attention

body:
I am trying to understanding how to implement a seq-to-seq model with attention from this website.
My question: Is nn.embedding just returns some IDs for each word, so the embedding for each word would be the same during whole training? Or are they getting changed during the procedure of training?
My second question is because I am confused whether after training, the output of nn.embedding is something such as word2vec word embeddings or not.
Thanks in advance


title: How to train two models on two different GPUs with the same data in Pytorch?

body:
This might be seen as a somewhat simple question, but I'm new to distributed training in Pytorch.
So, I have two defined models (net1 and net2), and two dataloaders for training and testing (train_dl, test_dl), and also I have access to two GPUs ("cuda:0" and "cuda:1"). I have a function called "training" that receives a model, the two dataloaders, and returns the test accuracy. I just need to train both models at the same time; not sequentially, but simultaneously:
1 net1.to("cuda:0")
2 net2.to("cuda:1")
3 accuracy1 = training(net1, train_dl, test_dl)
4 accuracy2 = training(net2, train_dl, test_dl)

What do I need to do in order to make lines 3 and 4 to execute at the same time? I read something about "spawn" but I have really no clue of how to implement it.
Thank you very much!


title: Importing PyTorch before using pandas.DataFrame.plot() causes Jupyter kernel to crash

body:
I'm running windows 10 with the following library versions:
matplotlib  3.3.3
torch       1.7.0
pandas      1.1.4

When I load from CSV a dataframe and plot its data BEFORE import torch, I get no issues. However, if I put all of my import statements at the top of the notebook, as is tradition, I instead get a crashed kernel with the following pop up message:

The kernel for anom_detect_nn.ipynb appears to have died. It will
restart automatically.

When I look at my shell, I see two error messages:

OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll
already initialized.
OMP: Hint This means that multiple copies of the
OpenMP runtime have been linked into the program. That is dangerous,
since it can degrade performance or cause incorrect results. The best
thing to do is to ensure that only a single OpenMP runtime is linked
into the process, e.g. by avoiding static linking of the OpenMP
runtime in any library. As an unsafe, unsupported, undocumented
workaround you can set the environment variable
KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute,
but that may cause crashes or silently produce incorrect results. For
more information, please see
http://www.intel.com/software/products/support/.

It seems like other users on SO have experienced this before, but all of the solutions apply to MacOS users. I've tried them anyway:

conda install nomkl
pip uninstall everything, pip install everything
I did not use the dangerous workaround KMP_DUPLICATE_LIB_OK=TRUE

Thanks to the above steps entire set-up is now a tangled mess where I can't pip install anything successfully, and modules that are successfully installed can no longer be imported without Module not found errors.
This is a real pain, and I'm at my wits end. I'm currently uninstalling everything Python on my system and starting over. Not a happy camper. Any solutions that are for Windows and not Mac, should this problem persist when I start over?


title: How should I build this LSTM model in Pytorch?

body:
On Pytorch, if you want to build a model like this, ⇓
the code will be:
import torch.nn as nn

BLSTM = nn.LSTM(3, 3, bidirectional=True) # input and hidden sizes are example. it doesn't have to be 3.

But I want to build a model like this. ⇓
This model has two-layerd forward & backward LSTM.
My question is how to build this model in Pytorch.

The code I can think of is this:
import torch.nn

BLSTM = nn.LSTM(3, 3, 2, bidirectional=True) # input and hidden sizes are example. it doesn't have to be 3.

I thought it's wrong because this ⇓ would be the model if I used this code.

I feel that this model is a little different from the model I want to build.


title: I got this error when installing PyTorch in Pycharm. I installed torchvision and torch successfully but the problem is in PyTorch

body:
I try to install PyTorch in cmd to import it in pycharm project. it gives me numerous errors after Running setup.py install for PyTorch ... error.
ERROR: Command errored out with exit status 1:
     command: 'c:\users\sarah\appdata\local\programs\python\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\sarah\\AppData\\Local\\Temp\\pip-install-uejad0d4\\pytorch_87cb825d32c24f6ca6f350ea09c367a4\\setup.py'"'"'; __file__='"'"'C:\\Users\\sarah\\AppData\\Local\\Temp\\pip-install-uejad0d4\\pytorch_87cb825d32c24f6ca6f350ea09c367a4\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\sarah\AppData\Local\Temp\pip-record-k6l1ba85\install-record.txt' --single-version-externally-managed --user --prefix= --compile --install-headers 'C:\Users\sarah\AppData\Roaming\Python\Python38\Include\PyTorch'
         cwd: C:\Users\sarah\AppData\Local\Temp\pip-install-uejad0d4\pytorch_87cb825d32c24f6ca6f350ea09c367a4\
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\sarah\AppData\Local\Temp\pip-install-uejad0d4\pytorch_87cb825d32c24f6ca6f350ea09c367a4\setup.py", line 11, in <module>
        raise Exception(message)
    Exception: You tried to install "pytorch". The package named for PyTorch is "torch"
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\users\sarah\appdata\local\programs\python\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\sarah\\AppData\\Local\\Temp\\pip-install-uejad0d4\\pytorch_87cb825d32c24f6ca6f350ea09c367a4\\setup.py'"'"'; __file__='"'"'C:\\Users\\sarah\\AppData\\Local\\Temp\\pip-install-uejad0d4\\pytorch_87cb825d32c24f6ca6f350ea09c367a4\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\sarah\AppData\Local\Temp\pip-record-k6l1ba85\install-record.txt' --single-version-externally-managed --user --prefix= --compile --install-headers 'C:\Users\sarah\AppData\Roaming\Python\Python38\Include\PyTorch' Check the logs for full command output.


Any help solving this will be highly appreciated.
I have an idea why the error is happening, it is because the xmin == xmax and ymin == ymax which should not be. However I seem not to know how this is happening. Here is how I load my custom dataset with pytorch Dataset class.
`class CustomDataset(torch.utils.data.Dataset):
    def __init__(self, root, transforms=None):
        self.root = root
        self.transforms = transforms
        # load all image files, sorting them to
        # ensure that they are aligned
        self.imgs = list(sorted(os.listdir(os.path.join(root, "seg_image_use"))))
        self.masks = list(sorted(os.listdir(os.path.join(root, "seg_mask_use"))))

    def __getitem__(self, idx):
        # load one image and mask using idx
        img_path = os.path.join(self.root, "seg_image_use", self.imgs[idx])
        mask_path = os.path.join(self.root, "seg_mask_use", self.masks[idx])
        img = Image.open(img_path).convert("RGB")
        # note that we haven't converted the mask to RGB,
        # because each color corresponds to a different instance
        # with 0 being background
        mask = Image.open(mask_path)

        mask = np.asarray(mask)
        # instances are encoded as different colors
        obj_ids = np.unique(mask)[1:] # first id is the background, so remove it
        masks = mask == obj_ids[:, None, None]  # split the color-encoded mask into a set of binary masks
        # get bounding box coordinates for each mask
        num_objs = len(obj_ids)
        boxes = []
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])

       # convert everything into torch.Tensor
        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])

        target = {}
        target["boxes"] = boxes
        target["labels"] = torch.as_tensor(obj_ids, dtype=torch.int64) - 1 # corrected by Rawi
        target["masks"] = torch.as_tensor(masks, dtype=torch.uint8) #uint8
        target["image_id"] = torch.tensor([idx])
        target["area"] = area
        target["iscrowd"] = torch.zeros((num_objs,), dtype=torch.int64) # suppose all instances are not crowd

        if self.transforms is not None:
            img, target = self.transforms(img, target)

        return img, target

    def __len__(self):
        return len(self.imgs)
`
And then when I call it to see the dataset, I get the first index of each dataset showing this; take note of the first index in the 'boxes' tensor. (italic)

`dataset_sample = CustomDataset('C:/Users/LENOVO/Desktop/clothme/Train')
img, target = dataset_sample[2]
print(target)

result: {'boxes': tensor(*[[  0.,   0., 286., 403.]*,
         [ 30., 240.,  52., 241.],
         [ 25., 183.,  31., 204.],
         [ 26., 224.,  34., 240.],
         [ 30., 169.,  88., 181.],
         [ 32., 239.,  85., 251.],
`

Here is the error I get when I try to train the model.
`---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-81-c798930961c1> in <module>
      4 for epoch in range(num_epochs):
      5     # train for one epoch, printing every 10 iterations
----> 6     train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
      7     # update the learning rate
      8     lr_scheduler.step()

~\measurement_model_dev\engine.py in train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq)
     28         targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
     29
---> 30         loss_dict = model(images, targets)
     31
     32         losses = sum(loss for loss in loss_dict.values())

~\anaconda3\envs\measurement_py37\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\anaconda3\envs\measurement_py37\lib\site-packages\torchvision\models\detection\generalized_rcnn.py in forward(self, images, targets)
     92                     raise ValueError("All bounding boxes should have positive height and width."
     93                                      " Found invalid box {} for target at index {}."
---> 94                                      .format(degen_bb, target_idx))
     95
     96         features = self.backbone(images.tensors)

ValueError: All bounding boxes should have positive height and width. Found invalid box [790.0323486328125, 359.0328369140625, 790.0323486328125, 359.0328369140625] for target at index 0.
`


title: Pytorch Lightning duplicates main script in ddp mode

body: When I launch my main script on the cluster with ddp mode (2 GPU's), Pytorch Lightning duplicates whatever is executed in the main script, e.g. prints or other logic. I need some extended training logic, which I would like to handle myself. E.g. do something (once!) after Trainer.fit(). But with the duplication of the main script, this doesn't work as I intend. I also tried to wrap it in if __name__ == "__main__", but it doesn't change behavior. How could one solve this problem? Or, how can I use some logic around my Trainer object, without the duplicates?


title: Why is pytorch softmax function not working?

body:
so this is my code
import torch.nn.functional as F
import torch

inputs = [1,2,3]
input = torch.tensor(inputs)
output = F.softmax(input, dim=1)
print(output)

is the reason why the code not working because of the dim?
the error here:
  File "c:\Users\user\Desktop\AI\pytorch_jovian\linear_reg.py", line 19, in <module>
    output = F.softmax(input, dim=1)
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\functional.py", line 1583, in softmax
    ret = input.softmax(dim)
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)


title: Pytorch RNN with no nonlinearity

body: Is it possible to implement an RNN layer with no nonlinearity in Pytorch like in Keras where one can set the activation to linear? By removing the nonlinearlity, I want to implement a first-order infinite-impulse-response (IIR) filter with a differentiable parameter and integrate it into my model for end-to-end learning. I can obviously implement the filter in Pytorch but I thought using an inbuilt function may be more efficient.


title: RuntimeError (DataLoader worker error) for SimpleTransformers classifier

body:
I'm trying to train a text classifier using SimpleTransformers. I am utilizing a TurkuNLP language model to try and train a classifier (utilizing my computer's GPU).
My code:
model = ClassificationModel('bert', 'TurkuNLP/wikibert-base-he-cased', num_labels=11, use_cuda=True,
                            args={'reprocess_input_data': True, 'overwrite_output_dir': True, 'fp16': False,
                                   "num_train_epochs": 3, 'sliding_window': True, 'train_batch_size': 6})

# You can set class weights by using the optional weight argument

# Train the model
model.train_model(train_df)

# Evaluate the model
result, model_outputs, wrong_predictions = model.eval_model(test_df)

I then start the training of the classifier. However, once the 1st training epoch ends, I receive a warning:

WARNING  py.warnings:99: [JupyterRequire] C:\Users\work\Anaconda3\envs\py35\lib\site-packages\torch\optim\lr_scheduler.py:216:
UserWarning: Please also save or load the state of the optimizer when saving or loading the scheduler.
warnings.warn(SAVE_STATE_WARNING, UserWarning)

Then, as soon as the 2nd epoch begins, I receive the following error:

---------------------------------------------------------------------------
Empty                                     Traceback (most recent call last)
~\Anaconda3\envs\py35\lib\site-packages\torch\utils\data\dataloader.py
in _try_get_data(self, timeout)
    871         try:
--> 872             data = self._data_queue.get(timeout=timeout)
    873             return (True, data)

~\Anaconda3\envs\py35\lib\multiprocessing\queues.py in get(self, block, timeout)
    104                     if not self._poll(timeout):
--> 105                         raise Empty
    106                 elif not self._poll():

Empty:

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
<ipython-input-31-6ab367d014c4> in <module>
     22
     23 # Train the model
---> 24 model.train_model(train_df)
     25
     26 # Evaluate the model

~\AppData\Roaming\Python\Python36\site-packages\simpletransformers\classification\classification_model.py
in train_model(self, train_df, multi_label, output_dir, show_running_loss, args, eval_df, verbose, **kwargs)
    409             eval_df=eval_df,
    410             verbose=verbose,
--> 411             **kwargs,
    412         )
    413

~\AppData\Roaming\Python\Python36\site-packages\simpletransformers\classification\classification_model.py
in train(self, train_dataloader, output_dir, multi_label, show_running_loss, eval_df, verbose, **kwargs)
    627                 mininterval=0,
    628             )
--> 629             for step, batch in enumerate(batch_iterator):
    630                 if steps_trained_in_current_epoch > 0:
    631                     steps_trained_in_current_epoch -= 1

~\Anaconda3\envs\py35\lib\site-packages\tqdm\notebook.py in __iter__(self, *args, **kwargs)
    232     def __iter__(self, *args, **kwargs):
    233         try:
--> 234             for obj in super(tqdm_notebook, self).__iter__(*args, **kwargs):
    235                 # return super(tqdm...) will not catch exception
    236                 yield obj

~\Anaconda3\envs\py35\lib\site-packages\tqdm\std.py in __iter__(self)
   1169
   1170         try:
-> 1171             for obj in iterable:
   1172                 yield obj
   1173                 # Update and possibly print the progressbar.

~\Anaconda3\envs\py35\lib\site-packages\torch\utils\data\dataloader.py
in __next__(self)
    433         if self._sampler_iter is None:
    434             self._reset()
--> 435         data = self._next_data()
    436         self._num_yielded += 1
    437         if self._dataset_kind == _DatasetKind.Iterable and \

~\Anaconda3\envs\py35\lib\site-packages\torch\utils\data\dataloader.py
in _next_data(self)
   1066
   1067             assert not self._shutdown and self._tasks_outstanding > 0
-> 1068             idx, data = self._get_data()
   1069             self._tasks_outstanding -= 1
   1070             if self._dataset_kind == _DatasetKind.Iterable:

~\Anaconda3\envs\py35\lib\site-packages\torch\utils\data\dataloader.py
in _get_data(self)
   1032         else:
   1033             while True:
-> 1034                 success, data = self._try_get_data()
   1035                 if success:
   1036                     return data

~\Anaconda3\envs\py35\lib\site-packages\torch\utils\data\dataloader.py
in _try_get_data(self, timeout)
    883             if len(failed_workers) > 0:
    884                 pids_str = ', '.join(str(w.pid) for w in failed_workers)
--> 885                 raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
    886             if isinstance(e, queue.Empty):
    887                 return (False, None)

RuntimeError: DataLoader worker (pid(s) 15784, 13540, 16356, 15852, 2700, 10548, 21112, 21788, 23504, 23484, 23100, 20344, 22120, 22668, 21924) exited unexpectedly

Any ideas? Thanks :)


title: How to solve the two layers of CNN dimension

body:
Now，I am learning CNN for nlp,some times ago,I try to modify the the code of Yoon Kim,the github link is (https://github.com/Joyce94/cnn-text-classification-pytorch), I try to add a layer of convolution in the model.py.

However, I encounter a dimension problem, in other words, the output of the first convolution layer to make as the input of the second convolution layer, it tells me that  dimension not fit.

Anyone can help me to so solve it?

# one layer
self.conv13 = nn.Conv2d(Ci, 100, (3, D))
self.conv14 = nn.Conv2d(Ci, 100, (4, D))
self.conv15 = nn.Conv2d(Ci, 100, (5, D))

# two layer
self.conv23 = nn.Conv2d(100, Co, (D, Co))
self.conv24 = nn.Conv2d(100, Co, (D, Co))
self.conv25 = nn.Conv2d(100, Co, (D, Co))

x1 = self.conv(x, self.conv13) #(N,Co)
print("convvv", x1)

# two layer
if self.args.cuda:
    self.conv23 = [model.cuda() for model in self.conv23]
    self.conv24 = [model.cuda() for model in self.conv24]
    self.conv25 = [model.cuda() for model in self.conv25]

x1 = x1.unsqueeze(1)
x4 = self.conv(x1, self.conv23) #(N,Co)
x = self.pool(x4)


title: Unable to set sequential to false in a torchtext LABEL field

body:
I'm having trouble setting the sequential attribute for a field to False. My code includes the following.

LABEL = data.Field(sequential=False)

But when I check whether sequential is False or not...

hasattr(LABEL, 'sequential')

I get "True". When I try to set the attribute...

LABEL.sequential = False
hasattr(LABEL, 'sequential')

I still get True!

Any help would be much appreciated!


title: How to load a sprase numerical custom dataset into PyTorch?

body:
I have a custom numerical data set with 100k number of data points and 50M number of sparse features. What is the best way to load this data set into PyTorch?

I am looking for some function that loads a sparse matrix such as scipy CSC matrix into sparse tensor.


title: writing out neural network inference test code

body:
I am trying to modify an inference code for pruned SqueezeNet network

However, I faced the following error. Could anyone comment how to go around this cpu/gpu backend error ?

[kevin@linux SqueezeNet-Pruning]$ python predict.py --image “3_100.jpg” --model “model_prunned” --num_class “2”
prediction in progress
Traceback (most recent call last):
File “predict.py”, line 63, in
prediction = predict_image(imagepath)
File “predict.py”, line 47, in predict_image
output = model(input)
File “/usr/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 477, in call
result = self.forward(*input, **kwargs)
File “/home/kevin/Documents/Grive/Personal/Coursera/Machine_Learning/pruning/Pruning-CNN/SqueezeNet-Pruning/finetune.py”, line 39, in forward
x = self.features(x)
File “/usr/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 477, in call
result = self.forward(*input, **kwargs)
File “/usr/lib/python3.7/site-packages/torch/nn/modules/container.py”, line 92, in forward
input = module(input)
File “/usr/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 477, in call
result = self.forward(*input, **kwargs)
File “/usr/lib/python3.7/site-packages/torch/nn/modules/conv.py”, line 313, in forward
self.padding, self.dilation, self.groups)
RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 ‘weight’
[kevin@linux SqueezeNet-Pruning]$


title: Gradient quantization in data-parallel SGD with a custom quantizer

body:
I would like to perform data-parallel SGD training with my own gradient quantization (in order to reduce the communication). Each worker node sends the encoded (compressed) gradient to the master node. The master node decodes the compressed gradient and updates the parameters. I will use my own encoding (quantization) and decoding functions.

I was wondering under which deep learning framework it is relatively easier to do this. Thank you!


title: Pytorch GPU memory increase after load operation

body:
I have a pytorch model which is of the size 386MB, but when i load the model

state = torch.load(f, flair.device)

My GPU memory takes up to 900MB, why does this happen and is there a way to resolve this?

This is how i save the model

model_state = self._get_state_dict()

# additional fields for model checkpointing
model_state["optimizer_state_dict"] = optimizer_state
model_state["scheduler_state_dict"] = scheduler_state
model_state["epoch"] = epoch
model_state["loss"] = loss

torch.save(model_state, str(model_file), pickle_protocol=4)


title: Number of Conv2d Layers and Filters for Small Image Classification Task

body:
If I'm working with a dataset where I have ~100,000 training images and ~20,000 validation images, each of size 32 x 32 x 3, how does the size and the dimensions of my dataset affect the number of Conv2d layers I have in my CNN? My intuition is to use fewer Conv2d layers, 2-3, because any more than 3 layers will be working with parts of the image that are too small to gain relevant data from.

In addition, does it make sense to have layers with a large number of filters, >128? My thought is that when dealing with small images, it doesn't make sense to have a large number of parameters.


title: Why doesn't nn.Sequential contain a softmax output layer in the example?

body:
The example from PyTorch's official tutorial has the following ConvNet. My understanding is that the output layer uses a softmax to estimate the digit an image corresponds to. Why doesnt the code have a softmax layer or fully connected layer?

model = nn.Sequential(
    nn.Conv2d(1, 16, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.Conv2d(16, 16, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.Conv2d(16, 10, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.AdaptiveAvgPool2d(1),
    Lambda(lambda x: x.view(x.size(0), -1)),
)

opt = optim.SGD(model.parameters(), lr=lr, momentum=0.9)


title: NVIDIA driver too old error running Detectron2

body:
I am newly working with Detectron2 framework on Google Colab.
My code was running perfectly but after one day I faced an error on it.

RuntimeError: The NVIDIA driver on your system is too old (found version 10010). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver.

I don’t know how should I update the NVIDIA driver on colab for Torch.
Can you please help me?