title: Dataloader worker exited unexpectedly while running on Visual Studio. But runs okay on Google Colab

body:
So I have this dataloader that loads data from hdf5 but exits unexpectedly when I am using num_workers>0 (it works ok when 0). More strangely, it works okay with more workers on google colab, but not on my computer.
On my computer I have the following error:
Traceback (most recent call last):
File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 986, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\multiprocessing\queues.py", line 105, in get
raise Empty
_queue.Empty
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "", line 2, in
File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 517, in next
data = self._next_data()
File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 1182, in _next_data
idx, data = self._get_data()
File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 1148, in _get_data
success, data = self._try_get_data()
File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 999, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 12332) exited unexpectedly
Also, my getitem function is:
def __getitem__(self,index):
  desired_file = int(index/self.file_size)
  position = index % self.file_size

  h5_file = h5py.File(self.files[desired_file], 'r')

  image = h5_file['Screenshots'][position]
  rect = h5_file['Rectangles'][position]
  numb = h5_file['Numbers'][position]

  h5_file.close()

  image = torch.from_numpy(image).float()
  rect = torch.from_numpy(rect).float()
  numb = torch.from_numpy( np.asarray(numb) ).float()

  return (image, rect, numb)

Does anyone have any idea what can be causing this empty queue?


title: TensorFlow 2 significantly slower than PyTorch when training only a few batches

body:
Platform: Windows 10 and Colab (reproducible on both platform)
Version: TensorFlow 2.4.1, Pytorch 1.8.1, Python 3.8 (this problem occurs on both CPU and GPU)
When training a small number of batches (for example, train 30 batches, note that similar but slightly different ratio still applies for a smaller number of batches like a single batch, different models with more layers/params, and/or different batch size),

with low-level gradient tape training, TensorFlow is 9 times slower than equivalent model/data on PyTorch
with low-level gradient tape + @tf.function train step, TensorFlow is 18 times slower than equivalent model/data on PyTorch
with high-level fit(), TensorFlow is 19 times slower than equivalent model/data on PyTorch
with high-level train_on_batch(), TensorFlow is 22 times slower than equivalent model/data on PyTorch

A reproducible code example can be found here, which shows a simple model and generated random data being trained with all functions/API discussed above:
https://colab.research.google.com/drive/1VHYLEZ79etKSkK0iWIqHkt7OG8nbU9uU?usp=sharing
above code is partially modified from the official tutorial https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch
and partially written on my own.
training on a single batch or a small number of batches is commonly used in deep reinforcement learning and some deep learning special cases.
This performance difference is also observed on my local Windows 10 machines with PyTorch 1.7.1 vs TensorFlow 2.4.1. When training a Deep Q-Network (a kind of deep reinforcement learning that requires training 1 batch at a time), using TensorFlow train_on_batch results in more than 20 times longer to train the same number of episodes (kind of like reinforcement learning version of epochs).
Anybody know why? Is there a way to write a code that is as fast as PyTorch?
It is reasonable to have gaps among different frameworks, but 9-22 times seems quite large, especially for arguably the two most popular ones.
It is also weird that train_on_batch(), an API that seems obviously designed for training a single batch can be the slowest of all. it is also weird that @tf.function that is designed to accelerate actually slows down training.
Any help and suggestions would be great! Thanks.
Also: is this kind of question eligible to be posted on GitHub Issue?


title: Randomly set some elements in a tensor to zero (with low computational time)

body:
I have a tensor of shape (3072,1000) which represents the weights in my neural network. I want to:

randomly set 60% of its elements to zero.
After updating the weights, keep 60% of the elements equal to zero but again randomly i.e., not the same previous elements.

Note: my network is not the usual artificial neural network which uses backpropagation algorithm but it is a biophysical model of the neurons in the brain so I am using special weight updating rules. Therefore, I think the ready functions in pytorch, if any, might not be helpful.
I tried the follwoing code, it is working but it takes so long because after every time I update my weight tensor, I have to run that code to set the weight tensor again to be 60% zeros
row_indices = np.random.choice(np.size(mytensor.weight, 0),
                       replace=False,size=int(np.size(mytensor.weight, 0)* 0.6))
column_indices = np.random.choice(np. size(mytensor.weight, 1),
                       replace=False, size=int(np. size(mytensor.weight, 1) * 0.6))
for r in row_indices:
    for c in column_indices:
        (mytensor.weight)[r][c] = 0


GLUE_TASKS = ["cola", "mnli", "mnli-mm", "mrpc", "qnli", "qqp", "rte", "sst2", "stsb", "wnli"]

#useful in preprocessing, this sets what each task does.
task_to_keys = {
    "cola": ("sentence", None),
    "mnli": ("premise", "hypothesis"),
    "mnli-mm": ("premise", "hypothesis"),
    "mrpc": ("sentence1", "sentence2"),
    "qnli": ("question", "sentence"),
    "qqp": ("question1", "question2"),
    "rte": ("sentence1", "sentence2"),
    "sst2": ("sentence", None),
    "stsb": ("sentence1", "sentence2"),
    "wnli": ("sentence1", "sentence2"),
}

def set_module_grad_status(module, flag=False):
    if isinstance(module, list):
        for m in module:
            set_module_grad_status(m, flag)
    else:
        for p in module.parameters():
            p.requires_grad = flag

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    if task != "stsb":
        predictions = np.argmax(predictions, axis=1)
    else:
        predictions = predictions[:, 0]
    return metric.compute(predictions=predictions, references=labels)

#setting some model parameters
task = "qnli"
sentence1_key, sentence2_key = task_to_keys[task]
# model_checkpoint = "test-glue/basemodel"

num_labels = 3 if task.startswith("mnli") else 1 if task=="stsb" else 2

metric_name = "pearson" if task == "stsb" else "matthews_correlation" if task == "cola" else "accuracy"
validation_key = "validation_mismatched" if task == "mnli-mm" else "validation_matched" if task == "mnli" else "validation"

model_checkpoint = "google/mobilebert-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)
model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint, num_labels=num_labels)
model = model.to(device)

args = TrainingArguments(
    "test-glue",
    evaluation_strategy = "epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    num_train_epochs=epochs,
    weight_decay=0.01,
    save_steps=0
    # load_best_model_at_end=True,
    # metric_for_best_model=metric_name,
)

#load dataset
actual_task = "mnli" if task == "mnli-mm" else task #some error checking..
dataset = load_dataset("glue", actual_task)
metric = load_metric('glue', actual_task)

# tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)
def preprocess_function(examples):
    if sentence2_key is None:
        return tokenizer(examples[sentence1_key], truncation=True)
    return tokenizer(examples[sentence1_key], examples[sentence2_key], truncation=True)
encoded_dataset = dataset.map(preprocess_function, batched=True)

trainer = Trainer(
    model,
    args,
    train_dataset=encoded_dataset["train"],
    eval_dataset=encoded_dataset[validation_key],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)
trainer.train()

I not quite sure why I am getting the error "RuntimeError: The size of tensor a (549) must match the size of tensor b (512) at non-singleton dimension 1" at the like "trainer.train()". Like isn't the each sample supposed to be consistently size of 512. This only happens with MobileBERT model and not on DistilBERT model. The code that I am using is mostly based on  the huggingface colab tutorial https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/text_classification.ipynb#scrollTo=7k8ge1L1IrJk


title: How to iterate over PyTorch tensor

body:
I have a tensor data of size (1000,110) and I want to iterate over the first index of the tensor and calculate the following.
    data = torch.randn(size=(1000,110)).to(device)

    male_poor = torch.tensor(0).float().to(device)
    male_rich = torch.tensor(0).float().to(device)

    female_poor = torch.tensor(0).float().to(device)
    female_rich = torch.tensor(0).float().to(device)

    for i in data:

        if torch.argmax(i[64:66]) == 0 and torch.argmax(i[108:110]) == 0:
          female_poor += 1
        if torch.argmax(i[64:66]) == 0 and torch.argmax(i[108:110]) == 1:
          female_rich += 1
        if torch.argmax(i[64:66]) == 1 and torch.argmax(i[108:110]) == 0:
          male_poor += 1
        if torch.argmax(i[64:66]) == 1 and torch.argmax(i[108:110]) == 1:
          male_rich += 1

    disparity = ((female_rich/(female_rich + female_poor))) / ((male_rich/(male_rich + male_poor)))

Is there a faster way than for loop to do this?


title: Is there a verctorized approach in PyTorch to get this result?

body:
I would like to get this result with torch functions. Do you have suggestions?
import torch
test_tensor=torch.tensor([[1,  2,  3, 4,  5,  6],
                         [7,  8,  9, 10,  11,  12]]
                        )
print(test_tensor)
'''
I would like to get:
t_1 = torch.tensor([[6], #1+2+3
                  [24]])#7+8+9
t_2 = torch.tensor([[9], #1+3+5
                  [27]])#7+9+11
'''


title: Structure tensor based off index

body:
I'm trying to join multiple tensors based off a corresponding index tensor. As a toy example I have a tensor A = [[0,1,2],[2,3,4]] with a corresponding index vector B = [0,1] and another tensor C = [[5,6,7],[8,9,10],[11,12,13]] with a corresponding index vector D = [0,1,2]. I seek to return a tensor such that E = [[0,1,2],[5,6,7],[2,3,4],[8,9,10],[11,12,13]] where the values with the same indices are grouped together. I'm trying to efficiently do this as it is contained within my NN model and can suffer from performance issues during the backward propagation.

I was able to do achieve this by constructing a dictionary and for looping as such:

E = defaultdict(list)
for vector, indices in zip([A,C],[B,D]):
    for vector, i in enumerate(vector):
        E[indices[i]].append(vector)
E = list(E.values())

This approach however slows down model training significantly. Is there a way to do this more efficiently? Pytorch's various index_ operations don't seem to address this.  Thanks!


title: Pytorch RNN prediction results do not match training results

body:
The RNN code produces good sine approximations during traing, but when it comes to predicting sine values based in previous predictions (with a head start of 20 original values), it fails to approximate sine. (check images https://imgur.com/a/GCQ6QgG)

So either there is a bug in the prediction code or the model in general is not capable of handling refeeded predictions. But I think its more of a bug in my code :)

Code:
https://gist.github.com/saschalippert/36b72313afff86e00f3e10254fb4ff25

Logs:

RNN(
  (rnn): RNN(1, 64, batch_first=True)
  (fc): Linear(in_features=64, out_features=1, bias=True)
)
Training for 10 epoch(s)...
Epoch:    1/10    Loss: 0.05084234103560448
Epoch:    2/10    Loss: 0.006296889390796423
Epoch:    3/10    Loss: 0.005784639157354832
Epoch:    4/10    Loss: 0.0037908838130533695
Epoch:    5/10    Loss: 0.0030788308940827847
Epoch:    6/10    Loss: 0.0027744979597628117
Epoch:    7/10    Loss: 0.0027218328323215246
Epoch:    8/10    Loss: 0.00589225348085165
Epoch:    9/10    Loss: 0.0020567490719258785
Epoch:   10/10    Loss: 0.000807589094620198


title: Why is my texture synthesis algorithm only producing blocky / noisy, non-sensible output?

body:
My end-goal is to create a script for neural style transfer, however, during writing code for said task, I stumbled upon a certain problem: the texture synthesis part of the algorithm seemed to have some problems with reproducing the artistic style. In order to solve this, I decided to create another script where I'd try to solve the task of texture synthesis using a neural network on its own.

TL;DR ... even after tackling the problem on its own, my script still produced blocky / noisy, non-sensible output.

I've tried having a look at how other people have solved this task, but most of what I found were more sophisticated solutions ("fast neural-style-transfer", etc.). Also, I couldn't find too many PyTorch implementations.

Since I've already spent the past couple of days on trying to fix this issue and considering that I'm new to the PyTorch-Framework, I have decided to ask the StackOverflow community for help and advice.

I use the VGG16 network for my model ...

class VGG16(nn.Module):
    def __init__(self):
        super(VGG16, self).__init__()
        vgg_fs = models.vgg16(pretrained=True).features

        self.sl1 = nn.Sequential()
        self.sl2 = nn.Sequential()
        self.sl3 = nn.Sequential()
        self.sl4 = nn.Sequential()
        self.sl5 = nn.Sequential()

        for i in range(4):
            self.sl1.add_module(str(i), vgg_fs[i])
        for i in range(4, 9):
            self.sl2.add_module(str(i), vgg_fs[i])
        for i in range(9, 16):
            self.sl3.add_module(str(i), vgg_fs[i])
        for i in range(16, 23):
            self.sl4.add_module(str(i), vgg_fs[i])
        for i in range(23, 30):
            self.sl5.add_module(str(i), vgg_fs[i])

        for p in self.parameters():
            p.requires_grad_(False)

    def forward(self, x):
        h = self.sl1(x)
        h1 = h
        h = self.sl2(h)
        h2 = h
        h = self.sl3(h)
        h3 = h
        h = self.sl4(h)
        h4 = h
        h = self.sl5(h)
        h5 = h

        return_tuple = namedtuple('hidden_states', ['h1', 'h2', 'h3', 'h4', 'h5'])
        ret = return_tuple(h1, h2, h3, h4, h5)

        return ret

Then, I have some functions for gram-matrix computation and normalization ...

def comp_gram(f):
    (b, c, h, w) = f.shape
    f = f.view(b, c, h * w)
    g = f.bmm(f.transpose(1, 2))
    g = g / (c * h * w)
    return g

def norm(b):
    mean    = torch.Tensor([0.485, 0.456, 0.406]).cuda().view(3, 1, 1)
    std     = torch.Tensor([0.229, 0.224, 0.225]).cuda().view(3, 1, 1)
    return (b - mean) / std

And finally, my training-function ...

def train(model, style_img, w=224, h=224, iters=32, lr=1):
    style = torch.from_numpy(np.array(Image.open(style_img))).cuda().float()
    style = style / 255.
    style = style.view(1, style.size()[2], *style.size()[:2])

    style_fts = model(norm(style))
    style_gms = [comp_gram(f) for f in style_fts]

    img = torch.rand(*style.size()[:2], h, w, requires_grad=True, device='cuda')
    optimizer = optim.Adam([img], lr=lr)
    mse_loss = nn.MSELoss()

    plt.ion()
    for i in range(iters):
        optimizer.zero_grad()
        actvs = model(norm(img))

        lss = 0.
        for f, gm in zip(actvs, style_gms):
            g = comp_gram(f)
            lss += mse_loss(g, gm)

        lss.backward()
        optimizer.step()

        if (i % 5 == 0) or (i == iters - 1):
            plt.title('Iter#{:04d}'.format(i))
            plt.imshow(img.detach().cpu().view(*img.shape[2:], img.shape[1]))
            plt.pause(1e-3)

        print('[iter#{:04d}]: Loss\t-> {:}'.format(i, lss.item()))

    plt.ioff()

    plt.subplot(121)
    plt.title('Original')
    plt.imshow(style.cpu().view(*style.size()[2:], style.size()[1]))

    plt.subplot(122)
    plt.title('Style')
    plt.imshow(img.detach().cpu().view(*img.size()[2:], img.size()[1]))

    plt.show()

I expected the algorithm to produce the style of the painting in some way, shape or form, but that didn't happen. ♂

(Since I sadly don't have enough reputation to post an image, you can find a picture of my non-sensible output here: https://mattmoony.github.io/cdn/conv-net_pytorch-style-transfer/problem.png)

I sincerely hope that you will be able to help me out and that I'm able to use this issue as a learning experience. Thank you!


title: Loss is not decreasing

body:
I am implementing a DNN with, but problem is that the loss is high and not decrease after all Iterations. The lables shape is torch.Size([1124823]), The feat size is torch.Size([1124823, 13]), X train shape torch.Size([719886, 13]), X test shape torch.Size([224965, 13]), X val shape torch.Size([179972, 13]), y train shape torch.Size([719886]), y train shape torch.Size([719886]), y val shape torch.Size([179972])

My Dataloader was implemented as follows:

    X_train, X_test, y_train, y_test = train_test_split(feat, labels, test_size=0.2, random_state=1)
    X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=1)

    train = data_utils.TensorDataset(X_train, y_train)
    train_loader = data_utils.DataLoader(train, batch_size=1000, shuffle=True)

    test = data_utils.TensorDataset(X_test, y_test)
    test_loader = data_utils.DataLoader(test, batch_size=1000, shuffle=False)

input_size = 13
hidden1_size = 13
hidden2_size = 64
hidden3_size = 128
hidden4_size = 256
hidden5_size = 1024
output_size = 3989

class DNN(nn.Module):
    def __init__(self, input_size, hidden1_size, hidden2_size, hidden3_size, hidden4_size, hidden5_size, output_size):
        super(DNN, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden1_size)
        self.drp1 = nn.Dropout(p=0.2, inplace=False)
        self.relu1 = nn.ReLU()
        self.tan1 = nn.Tanh()
        self.fc2 = nn.Linear(hidden1_size, hidden2_size)
        self.drp2 = nn.Dropout(p=0.2, inplace=False)
        self.relu2 = nn.ReLU()
        self.tan2 = nn.Tanh()
        self.fc3 = nn.Linear(hidden2_size, hidden3_size)
        self.drp3 = nn.Dropout(p=0.2, inplace=False)
        self.relu3 = nn.ReLU()
        self.tan3 = nn.Tanh()
        self.fc4 = nn.Linear(hidden3_size, hidden4_size)
        self.drp4 = nn.Dropout(p=0.2, inplace=False)
        self.relu4 = nn.ReLU()
        self.tan4 = nn.Tanh()
        self.fc5 = nn.Linear(hidden4_size, hidden5_size)
        self.drp5 = nn.Dropout(p=0.2, inplace=False)
        self.relu5 = nn.ReLU()
        self.tan5 = nn.Tanh()
        self.fc6 = nn.Linear(hidden5_size, output_size)
        self.tan6 = nn.Tanh()

    def forward(self, x):
        out = self.fc1(x)
        out = self.drp1(out)
        out = self.relu1(out)
        out = self.tan1(out)
        out = self.fc2(out)
        out = self.drp2(out)
        out = self.relu2(out)
        out = self.tan2(out)
        out = self.fc3(out)
        out = self.drp3(out)
        out = self.relu3(out)
        out = self.tan3(out)
        out = self.fc4(out)
        out = self.drp4(out)
        out = self.relu4(out)
        out = self.tan4(out)
        out = self.fc5(out)
        out = self.drp5(out)
        out = self.relu5(out)
        out = self.tan5(out)
        out = self.fc6(out)
        out = self.tan6(out)
        return out
batch_size = 10
n_iterations = 50
no_eps = n_iterations / (13 / batch_size)
no_epochs = int(no_eps)
model = DNN(input_size, hidden1_size, hidden2_size, hidden3_size, hidden4_size, hidden5_size, output_size)

criterion = nn.CrossEntropyLoss()
learning_rate = 0.0001
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

iter = 0
for epoch in range(no_epochs):
    for i, (X_train, y_train) in enumerate(train_loader):
        optimizer.zero_grad()
        outputs = model(Variable(X_train))
        loss = criterion(outputs, Variable(y_train))
        print('Iter %d --> loss %f' % (i, loss.item()))
        loss.backward()
        optimizer.step()
    correct = 0
    total = 0
    print('test')
    for X_test, y_test in test_loader:
        outputs = model(Variable(X_test))
        pred = outputs.argmax(dim=1, keepdim=True)
        total += y_test.size(0)
        correct += (pred.squeeze() == y_test).sum()  # pred.eq(y_test.view_as(pre  d)).sum().item()

    accuracy = 100 * correct / total

    print('Iteration: {}.  Accuracy: {}'.format(epoch, accuracy))


title: Why do I have problem when I train my model in pytorch?

body:
I'm new with PyTorch and AI but I have some trouble when I try to train my model.

I just create my Dataset and my Dataloader

    train_dataset = TensorDataset(tensor_train,tensor_label)
    train_dataloader = DataLoader(train_dataset,batch_size=32,shuffle=True)

And after this my criterion and optimiser

    criterion = nn.CrossEntropyLoss()

    optimiser=optim.Adam(net.parameters(),lr=0.2)

And I try to train it with

    for epoch in range(10):
           for data in train_dataloader:
                inputs,labels = data
                output = net(torch.Tensor(inputs))
                loss = criterion(output,labels.to(device))
                optimiser.zero_grad()
                loss.backward()
                optimiser.step()

But I got this error

    d:\py\lib\site-packages\torch\nn\modules\module.py in <lambda>(t)
321             Module: self
322         """
    --> 323         return self._apply(lambda t: t.type(dst_type))
324
325     def float(self):

TypeError: dtype must be a type, str, or dtype object

I will be happy if someone finds the problem, thanks.


title: Why distillation loss doesn't converge(decrease) compared to cross-entropy loss?

body:
I'm using cross entropy && distillation loss for the continual learning(a.k.a. incremental learning) model.
But, distillation loss doesn't converge while cross-entropy loss converges.

Here is my code for the distillation loss and the training part.

Code for distillation loss:

def loss_fn_distillation(outputs, soft_labels, temperature, current_step, total_step, total_label):
    current_label = (total_label / total_step) * (current_step + 1)
    previous_label = (total_label / total_step) * current_step

    soft_labels = V(soft_labels.data, requires_grad=False).cuda()
    soft_labels = torch.softmax(soft_labels / temperature, dim=1)

     outputs = F.log_softmax(outputs[:,:-int(current_label-previous_label)]/temperature, dim = 1)

     distill_loss = torch.sum(outputs * soft_labels, dim=1, keepdim=False)
     distill_loss = -torch.mean(distill_loss, dim=0, keepdim=False)

      return V(distill_loss, requires_grad=True).cuda()

Code for training part:

    outputs = net(inputs)
    ce_loss = criterion(outputs, targets)

    if(i>0) :

        soft_label = previous_net(inputs)

        distill_loss = loss_fn_distillation(outputs=outputs, soft_labels=soft_label, temperature=2,
                                            current_step=i, total_step=step, total_label=number_label)
        print(ce_loss, distill_loss)
        loss = distill_loss + ce_loss

    else :
        loss = ce_loss

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Result for cross entropy loss && distillation loss for epoch:

I'd appreciate any feedback. Thank you.


title: Confusing results of transfer learning accuracy

body:
I have just started an image classification project according to the tutorial from the documentation on the pytorch website(this).In the tutorial, there is a part of code like this:

model_ft = models.resnet50(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 20)

And I have known the reason why the fc layer is supposed to be changed .Since my project needs to classify 20 classes , so I just changed the parameters from 2 to 20. But , I just get the accuracy of around 60%. When I dont change the fc layer like this:

model_ft = se_resnet50(pretrained = True)

It turned out that the accuracy reaches 93.75% which is much better compared with the former results.

I just couldnt figure out why I get worse classification results when I modify the fc layer. Shouldn't it be modified?


title: Questions regarding image processing for pre-trained image classifier in PyTorch

body:
I am trying to use a popular pre-trained VGG model for image classification in PyTorch but noticed that the image is resized to 256 and crop it to 224 for pre-processing images in most of the programs. I am curious why we do resize it to 256 first and crop it instead of resizing it to 224 directly.

transforms = transforms.Compose([transforms.Resize(256),
                                 transforms.CenterCrop(224),
                                 transforms.ToTensor(),
                                 transforms.Normalize([0.485, 0.456, 0.406],
                                                     [0.229, 0.224, 0.225])])


title: How can I select all values of a multi dimensional tensor based on a mask with PyTorch?

body:
I have a tensor pred, which has a .size() of [4, 53, 161]. I have another tensor mask, which has a .size() of [4, 53].

The mask is only 0's and 1's. What I want to do is get the values of pred where mask has a value of 1. You'll notice that pred has one more dimension than mask. How can I accomplish this?


title: How can I element-wise 2 Tensors with different dimensions with Broadcasting?

body:
I have a tensor called inputs with a size of torch.Size([20, 1, 161, 199]) and another mask with a size of torch.Size([20, 1, 199]). I want to multiply them together.

I tried:

masked_inputs = inputs * mask[..., None]

but get an error:

RuntimeError: The size of tensor a (161) must match the size of tensor b (199) at non-singleton dimension 2

I'm not quite sure what to do?


title: I just installed pytorch on Anaconda but when i try to import it I am getting an error

body: OSError: [WinError 127] The specified procedure could not be found.


title: Manually calculated gradient not matching autograd

body:
I'm trying to calculate a gradient of a function that is A*A*X*W0*W1 w.r.t A in PyTorch.
I have calculated the gradient manually but I'm not getting the same answer as the pytorch's auto grad function. Can anyone help me with what I did wrong?
The manually calculated function is
def torch_grad(A, W0, W1, X):
  # Forward
  W0W1 = torch.mm(W0, W1)
  AX = torch.mm(A, X)
  AXW0W1 = torch.mm(AX, W0W1)
  AAXW0W1 = torch.mm(A, AXW0W1)

  # Backward / Gradients
  rows, cols = AAXW0W1.size()
  grad_AX = torch.mm(torch.eye(rows, cols), W0W1.t())
  grad_AA = torch.mm(grad_AX, X.t())
  grad_A = torch.mm(grad_AA, (2*A).t())
  return grad_A

Thanks!


title: Fast_seg training error - args for data_set giving KeyError: 'CascadeRelatioNet_res50'. Machine learning novice

body:
I've been trying to get this demo of various machine learning models to work. I have no background in machine learning, and essentially what I am trying to do is to analyze my own data against a model trained on the cityscapes dataset. I've been blundering along, and running into lots of problems because I don't really understand what the inputs expected are or what I am supposed to change within the python scripts to run other datasets. They have a fairly extensive list of potential arguments, which I don't understand. I'm also unclear on what is mandatory and where I am supposed to be putting things within the file structure.
The error I'm now bumping up against is:
(base) corey@corona:~/Desktop/pycity/lxtGH-Fast_Seg-682e0e6(2)/lxtGH-Fast_Seg-682e0e6$ python train_distribute.py --save_dir=SAVED --data_set=/home/corey/Desktop/pycity/leftImg8bit_trainvaltest/leftImg8bit

Traceback (most recent call last):
  File "train_distribute.py", line 269, in <module>
    main()
  File "train_distribute.py", line 156, in main
    deeplab = models.__dict__[args.arch](num_classes=args.num_classes, data_set=args.data_set)
KeyError: 'CascadeRelatioNet_res50'

Here's the list of potential arguments within "train_distribute.py" -
def get_arguments():
    """
    Parse all the arguments
    Returns: args
    A list of parsed arguments.
    """
    parser = argparse.ArgumentParser(description="DeepLab-ResNet Network")
    parser.add_argument("--batch_size_per_gpu", type=int, default=1,
                        help="Number of images sent to the network in one step.")
    parser.add_argument("--batch_size", type=int, default=8,
                        help="Number of images sent to the network in one step.")
    parser.add_argument('--gpu_num',type=int, default=8)
    parser.add_argument("--data_dir", type=str, default="./data",
                        help="Path to the directory containing the Cityscapes dataset.")
    parser.add_argument("--data_list", type=str, default="./data/cityscapes/train.txt",
                        help="Path to the file listing the images in the dataset.")
    parser.add_argument("--data_set", type=str, default="cityscapes", help="dataset to train")
    parser.add_argument("--arch", type=str, default="CascadeRelatioNet_res50", help="network architecture")
    parser.add_argument("--ignore_label", type=int, default=255,
                        help="The index of the label to ignore during the training.")
    parser.add_argument("--input_size", type=int, default=832 ,
                        help="Comma-separated string with height and width of images.")
    parser.add_argument("--learning_rate", type=float, default=1e-2,
                        help="Base learning rate for training with polynomial decay.")
    parser.add_argument("--momentum", type=float, default=0.9,
                        help="Momentum component of the optimiser.")
    parser.add_argument("--num_classes", type=int, default=19,
                        help="Number of classes to predict (including background).")
    parser.add_argument("--num_steps", type=int, default=50000,
                        help="Number of training steps.")
    parser.add_argument("--power", type=float, default=0.9,
                        help="Decay parameter to compute the learning rate.")
    parser.add_argument("--weight_decay", type=float, default=5e-4,
                        help="Regularisation parameter for L2-loss.")
    parser.add_argument("--num_workers", type=int, default=8)
    parser.add_argument("--random_mirror", action="store_true", default=True,
                        help="Whether to randomly mirror the inputs during the training.")
    parser.add_argument("--random_scale", action="store_true", default=True,
                        help="Whether to randomly scale the inputs during the training.")
    parser.add_argument("--random_seed", type=int, default=1234,
                        help="Random seed to have reproducible results.")

    # ***** Params for save and load ******
    parser.add_argument("--restore_from", type=str, default="./pretrained",
                        help="Where restore models parameters from.")
    parser.add_argument("--save_pred_every", type=int, default=5000,
                        help="Save summaries and checkpoint every often.")
    parser.add_argument("--save_dir", type=str, default=None,
                        help="Where to save snapshots of the models.")
    parser.add_argument("--save_start",type=int, default=40000)
    parser.add_argument("--gpu", type=str, default=None,
                        help="choose gpu device.")
    parser.add_argument("--ft", type=bool, default=False,
                        help="fine-tune the models with large input size.")
    # **** Params for OHEM **** #
    parser.add_argument("--ohem", type=str2bool, default='False',
                        help="use hard negative mining")
    parser.add_argument("--ohem_thres", type=float, default=0.7,
                        help="choose the samples with correct probability underthe threshold.")
    parser.add_argument("--ohem_keep", type=int, default=100000,
                        help="choose the samples with correct probability underthe threshold.")
    # ***** Params for logging ***** #
    parser.add_argument('--log_level', default="info", type=str,
                        dest='log_level', help='To set the log level to files.')
    parser.add_argument('--log_file', default="./log/train.log", type=str,
                        dest='log_file', help='The path of log files.')
    parser.add_argument("--log_format", default="%(asctime)s %(levelname)-7s %(message)s", type=str,
                        dest="log_format", help="format of log files"
                        )
    parser.add_argument('--stdout_level', default="info", type=str,
                        dest='stdout_level', help='To set the level to print to screen.')
    parser.add_argument("--rewrite", default=False, type=bool,
                        dest="rewrite", help="whether write the file when using log"
                        )
    parser.add_argument("--rgb", type=str2bool, default='False')
    # ***** Params for Distributed Traning ***** #
    parser.add_argument('--apex', action='store_true', default=False,
                        help='Use Nvidia Apex Distributed Data Parallel')
    parser.add_argument("--local_rank", default=0, type=int, help="parameter used by apex library")
    args = parser.parse_args()
    return args

Any advice? I tried feeding in various paths to the cityscape dataset as an arg for "args.data_set" Specifically, beyond this problem, is it clear theres something I'm missing broadly?


title: Loss does not decrease after modifying prediction with torch.floor()

body:
For the following code, the loss decreases.
loss_function=nn.MSELoss()
loss=loss_function(pred,label)

But, the loss remains completely unchanged if I change the pred by floor function. I checked the parameters after opt.step(), they are not changing.
loss_function=nn.MSELoss()
loss=loss_function(torch.floor(pred),label)

Why this might happen?
My guess: This torch.floor(pred) operation breaks the computation graph. Other "real" mathematical operation like pred*3 (for example) does not break the computation graph.


title: How do I avoid increasing the reserved CUDA memory?

body:
Could any one help me, please?)
I try to teach my ConvNetwork in Google Colab by using PyTorch on torchvision.datasets.STL10 (10'000 examples in train, 1'500 in val set, 1'500 in test), but I have the same problem over and over again - the network started learning, after a few epochs the process broke down, and an error appeared.
When I run code below something goes wrong and memory that I use increases and after a while I haven't free memory at all.
I've no idea what's going wrong.
I've tried to figure it out, but I couldn't find a solution to my problem even on stackOverflow.
I tried to use torch.cuda.empty_cache() in the hope that it would clear the memory that CUDA uses, but it didn't change anything at all.
Neural Network:
class ResNet(torch.nn.Module):
  def __init__(self):
    super(ResNet, self).__init__()

    self.conv1 = torch.nn.Conv2d(
        in_channels = 3, out_channels=30, kernel_size=5, padding=1) # 96x96x3 -> 94x94x30
    self.act1  = torch.nn.ReLU()
    self.pool1 = torch.nn.MaxPool2d(kernel_size=2, stride=2) # 94x94x30 -> 47x47x30

    self.conv2 = torch.nn.Conv2d(
        in_channels=30, out_channels=100, kernel_size=5, padding=1) # 47x47x30 -> 45x45x100
    self.act2  = torch.nn.ReLU()
    self.pool2 = torch.nn.MaxPool2d(kernel_size=3, stride=3) # 45x45x100 -> 15x15x100

    self.fc3  = torch.nn.Linear(in_features=15*15*100, out_features = 1000)
    self.act3 = torch.nn.ReLU()

    self.fc4  = torch.nn.Linear(in_features=1000, out_features=1000)
    self.act4 = torch.nn.ReLU()

    self.fc5  = torch.nn.Linear(in_features=1000, out_features=10)

  def forward(self, x):

    x = self.conv1(x)
    x = self.act1(x)
    x = self.pool1(x)

    x = self.conv2(x)
    x = self.act2(x)
    x = self.pool2(x)

    x = x.reshape(x.shape[0], 22500)

    x = self.fc3(x)
    x = self.act3(x)

    x = self.fc4(x)
    x = self.act4(x)

    x = self.fc5(x)
    return x

Next script:
NNet = ResNet()

device = torch.device('cuda:0' if torch.cuda.is_available else 'cpu')
loss = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(NNet.parameters(), lr = 0.001)
NNet = NNet.to(device)

random.seed(k)
np.random.seed(k)
torch.manual_seed(k)
torch.cuda.manual_seed(k)
torch.backends.cudnn.deterministic = True

X_train = X_train.to(device)
y_train = y_train.to(device)

X_dev = X_dev.to(device)
y_dev = y_dev.to(device)

Training script:
batch_size = 1000

for epoch in range(5):

  optimizer.zero_grad()

  order = np.random.permutation(X_train.shape[0])
  for start_batch in range(0, X_train.shape[0], batch_size):

    optimizer.zero_grad()
    batch_index = order[start_batch:start_batch+batch_size]
    X_batch = X_train[batch_index].to(device)
    y_batch = y_train[batch_index].to(device)

    y_pred = NNet.forward(X_batch)

    loss_train = loss(y_pred, y_batch)
    loss_train.backward()
    optimizer.step()

  accuracy_train = (y_pred.argmax(1) == y_batch).data.float().mean().cpu() # Train Accuracy
  y_pred = NNet.forward(X_dev)
  loss_dev = loss(y_pred, y_dev).data.cpu()
  accuracy_dev = (y_pred.argmax(dim=1) == y_dev).data.float().mean().cpu() # Dev Accuracy

  print('epoch #%i' % epoch, end='\t')
  print('loss dev: %.4f' % loss_dev, end='\t')
  print('loss train: %.4f' % loss_train, end='\t')
  print('accuracy dev: %.4f' % accuracy_dev, end='\t')
  print('accuracy train: %.4f' % accuracy_train)

RuntimeError: CUDA out of memory. Tried to allocate 774.00 MiB (GPU 0; 11.17 GiB total capacity; 10.48 GiB already allocated; 275.81 MiB free; 10.54 GiB reserved in total by PyTorch
All thanks!


title: is there pytorch's max operation in tensorflow2.0?

body:
overlap-> tensor([[0.0000, 0.0000, 0.0000,  ..., 0.6466, 0.7945, 0.5389]],
       device='cuda:0')

overlap_for_each_prior, object_for_each_prior = overlap.max(dim=0)  # (8732)

this .max(dim=0) return the two returns.
is there any equivalence method in tensorflow 2.0?


title: Export pytorch model parameters into separate files according to layer hierarchy

body:
Is it possible to export the trained parameters of a Pytorch model into separate binary files (float32/64, not text) under a folder hierarchy reflecting the layers defined by the model's architecture?
I wish to examine a sizeable pretrained model without the framework overhead and also split the checkpoint into manageable chunks.


title: Still get inplace operation error when I use clone()

body:
I tried to add .clone() at anywhere I think it's necessary.
But I still got inplace operation error.
And this error happened in second epoch, rather than first epoch when the beginning I got this error.
class Model(torch.nn.Module):
    def __init__(self, user_M, item_M, feature_M, embedding_M):
        super(Model, self).__init__()
        self.dimension = embedding_M + feature_M + 1

        # Memories
        self.user_memory = torch.randn(user_M, 2, 1, self.dimension)
        self.item_memory = torch.randn(item_M, 2, 1, self.dimension)

        # Writer
        self.user_LSTM = nn.LSTMCell(self.dimension, self.dimension)
        self.item_LSTM = nn.LSTMCell(self.dimension, self.dimension)

    def score(self, user_id, item_id):
        user_em = torch.index_select(self.user_memory[:, 1, :].clone(), 0, user_id[:,0].clone())
        item_em = torch.index_select(self.item_memory[:, 1, :].clone(), 0, item_id[:,0].clone())
        return torch.sigmoid(torch.bmm(user_em, item_em.permute(0,2,1)))

    def forward(self, train_data, user_id, item_id):
        ui_train = train_data[:, 0:1].type(torch.LongTensor).clone()
        ii_train = train_data[:, 1:2].type(torch.LongTensor).clone()

        for row in range(len(train_data)):
            memory = self.user_memory[ui_train[row][0]].clone()
            h, c = self.user_LSTM(train_data[row:row+1].clone(), (memory[0], memory[1]))
            self.user_memory[ui_train[row]] = torch.stack((h, c))

        for row in range(len(train_data)):
            memory = self.item_memory[ii_train[row][0]].clone()
            h, c = self.item_LSTM(train_data[row:row+1].clone(), (memory[0], memory[1]))
            self.item_memory[ii_train[row]] = torch.stack((h, c))

        # Scoring
        y = self.score(user_id, item_id)
        return y.reshape(-1, 1)

model = Model(user_M, item_M, feature_M, 3)
criterion = torch.nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)

for epoch in range(10):
    print(epoch)
    output = model(train_data, user_id, item_id)

    loss = criterion(output, label)

    optimizer.zero_grad()
    loss.backward(retain_graph=True)
    optimizer.step()

`RuntimeError`: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [176, 704]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).


title: trying to run a program on my device ... I still get this problem, I tried to solve it but I never managed to

body:
the program from ( https://github.com/sniklaus/sepconv-slomo ) ...
I tried to run the program using anaconda prompt.. and used "python run.py --model lf --first ./images/first.png --second ./images/second.png --out ./out.png" instruction as said ... but all I had was this
enter image description here
I did everything written in the setup section but it never worked.
Although I installed torch and cupy, but the problem is still the same.
I think it would work if I ran the program on my cpu -I think gpu caused the problem- .. but I did not know how to do so.


title: network values goes to 0 by linear layers

body:
I designed the Graph Attention Network.
However, during the operations inside the layer, the values of features becoming equal.
class GraphAttentionLayer(nn.Module):
    ## in_features = out_features = 1024
    def __init__(self, in_features, out_features, dropout):
        super(GraphAttentionLayer, self).__init__()
        self.dropout = dropout
        self.in_features = in_features
        self.out_features = out_features

        self.W = nn.Parameter(torch.zeros(size=(in_features, out_features)))
        self.a1 = nn.Parameter(torch.zeros(size=(out_features, 1)))
        self.a2 = nn.Parameter(torch.zeros(size=(out_features, 1)))
        nn.init.xavier_normal_(self.W.data, gain=1.414)
        nn.init.xavier_normal_(self.a1.data, gain=1.414)
        nn.init.xavier_normal_(self.a2.data, gain=1.414)
        self.leakyrelu = nn.LeakyReLU()

    def forward(self, input, adj):
        h = torch.mm(input, self.W)
        a_input1 = torch.mm(h, self.a1)
        a_input2 = torch.mm(h, self.a2)
        a_input = torch.mm(a_input1, a_input2.transpose(1, 0))
        e = self.leakyrelu(a_input)

        zero_vec = torch.zeros_like(e)
        attention = torch.where(adj > 0, e, zero_vec) # most of values is close to 0
        attention = F.softmax(attention, dim=1) # all values are 0.0014 which is 1/707 (707^2 is the dimension of attention)
        attention = F.dropout(attention, self.dropout)
        return attention

The dimension of 'attention' is (707 x 707) and I observed the value of attention is near 0 before the softmax.
After the softmax, all values are 0.0014 which is 1/707.
I wonder how to keep the values normalized and prevent this situation.
Thanks


title: How can I extend this object detection pytorch program to detect and classify more than one class?

body:
I followed this tutorial for building an object detector and now I am trying to expand it to recognise more classes.
This object detection algorithm only works for one class (raccoon), however it is possible to train the neural network with bigger datasets to recognise more classes.
It uses a pretrained faster_rcnn_resnet50 network.
Could someone help me adapt the code so that it can distinguish more than one class?
Thanks in advance!
class RaccoonDataset(torch.utils.data.Dataset):

    def __init__(self, root, data_file, transforms=None):
        self.root = root
        self.transforms = transforms
        self.imgs = sorted(os.listdir(os.path.join(root, "images")))
        self.path_to_data_file = data_file

    def __getitem__(self, idx):
        # load images and bounding boxes
        img_path = os.path.join(self.root, "images", self.imgs[idx])
        img = Image.open(img_path).convert("RGB")
        box_list = parse_one_annot(self.path_to_data_file,
                                   self.imgs[idx])
        boxes = torch.as_tensor(box_list, dtype=torch.float32)

        num_objs = len(box_list)
        # there is only one class
        labels = torch.ones((num_objs,), dtype=torch.int64)
        image_id = torch.tensor([idx])
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        # suppose all instances are not crowd
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)
        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["image_id"] = image_id
        target["area"] = area
        target["iscrowd"] = iscrowd

        if self.transforms is not None:
            img, target = self.transforms(img, target)
            return img, target

    def __len__(self):
        return len(self.imgs)


title: Passing batches to PyTorch Model

body:
I am trying to train a PyTorch LSTM model which is defined as:
class RecurrentNet(torch.nn.Module):
    def __init__(self, d_in, d_hidden, sequence_length):
        super(RecurrentNet, self).__init__()
        self.d_in = d_in
        self.d_hidden = d_hidden
        self.sequence_length = sequence_length
        self.lstm = torch.nn.LSTM(input_size=d_in,  hidden_size=d_hidden, num_layers=sequence_length)
        self.fc = nn.Linear(d_hidden, 1)

    def forward(self, x):
        out, hidden =  self.lstm(x)
        y_pred = F.relu(self.fc(out[-1][-1]))
        return y_pred

model = RecurrentNet(28, 3, l)

I have prepared my data in the form of batches of len 32, and with each instance having dimensions of 6 x 28, i.e. torch.Size([6, 28]). This means the total input tensor is size torch.Size([32, 6, 28]), with labels size torch.Size([32, 1]).
When I pass a single instance to the untrained model it returns an integer as expected. When I pass the Tensor of 32 intances, it returns:
tensor([0.], grad_fn=<ReluBackward0>)

I would have expect a 32 x 1 output Tensor for this input. Have I missed something when preparing the training data?


title: Why am I not able to replicate L2 regularization from sklearn's `ridge`?

body:
I am not sure why I cannot get close results for an SGD variant of ridge OLS regression, while the same model, absent L2 regularization, can get exact matches with sklearn.
Prepare Data
import pandas as pd
import numpy as np
import torch
from sklearn.linear_model import Ridge, LinearRegression
from sklearn import datasets
torch.manual_seed(123)
np.random.seed(123)

# load data
X, y, coef = datasets.make_regression(n_samples=1000,
                                      n_features=2,
                                      n_informative=2,
                                      noise=5,
                                      coef=True,
                                      random_state=0)

# set df
df = pd.DataFrame({'x1': X[:, 0], 'x2': X[:, 1], 'y': y})

# assign features and target
X = torch.tensor(df[['x1', 'x2']].values, dtype=torch.float)
y = torch.tensor(df['y'].values, dtype=torch.float)

# Shuffling & train/test split
shuffle_idx = torch.randperm(y.size(0), dtype=torch.long)

# assign shuffle
X, y = X[shuffle_idx], y[shuffle_idx]

# split
percent80 = int(shuffle_idx.size(0)*0.8)
X_train, X_test = X[shuffle_idx[:percent80]], X[shuffle_idx[percent80:]]
y_train, y_test = y[shuffle_idx[:percent80]], y[shuffle_idx[percent80:]]

# Normalize (mean zero, unit variance)
mu, sigma = X_train.mean(dim=0), X_train.std(dim=0)
X_train = (X_train - mu) / sigma
X_test = (X_test - mu) / sigma

Regular OLS
class LM1():
    def __init__(self, num_features, LAMBDA=0.0):
        self.num_features = num_features
        self.weights = torch.zeros(num_features, 1,
                                   dtype=torch.float)
        self.bias = torch.zeros(1, dtype=torch.float)
        self.LAMBDA = LAMBDA

    def forward(self, x):
        netinputs = torch.add(torch.mm(x, self.weights), self.bias)
        activations = netinputs
        return activations.view(-1)

    def backward(self, x, yhat, y):
        grad_loss_yhat = y - yhat
        grad_yhat_weights = x
        grad_yhat_bias = 1.

        grad_loss_weights = (2* -torch.mm(grad_yhat_weights.t(),
                                         grad_loss_yhat.view(-1, 1)) +
                             ( 2 * self.LAMBDA * self.weights)) / y.size(0)

        grad_loss_bias = 2* -torch.sum(grad_yhat_bias*grad_loss_yhat) / y.size(0)

        return (-1)*grad_loss_weights, (-1)*grad_loss_bias

    def loss(self, yhat, y):
        return torch.mean((yhat - y)**2)

    def train(self, x, y, num_epochs, learning_rate=0.01):
        cost = []
        for e in range(num_epochs):

            # forward
            yhat = self.forward(x)

            # backward
            negative_grad_w, negative_grad_b = self.backward(x, yhat, y)

            # update
            self.weights += learning_rate * negative_grad_w
            self.bias += learning_rate * negative_grad_b

            # logs
            curr_loss = self.loss(yhat, y)
            print('Epoch: %03d' % (e+1), end="")
            print(' | MSE: %.5f' % curr_loss)
            cost.append(curr_loss)
        return None

model = LM1(num_features=X_train.size(1), LAMBDA=0.0)
model.train(X_train, y_train, num_epochs=100, learning_rate=0.1)
print('predicted', model.weights)  # [40.1894], [39.6800]
print('true weights:', coef)

# identical to sklearn
reg = LinearRegression().fit(X_train, y_train)
reg.coef_  # [40.1894], [39.6800]

Ridge L2 Regression
# with l2 lambda
model1 = LM1(num_features=X_train.size(1), LAMBDA=125.0)
model1.train(X_train, y_train, num_epochs=100, learning_rate=0.1)
print('predicted', model1.weights)  # [34.7150], [34.2740]

# sklearn ridge
from sklearn.linear_model import Ridge
LAMBDA = 125.0
# C = inverse of lambda
C = 1/LAMBDA
# alpha = 1 / (2C)
alpha = 1 / (2*C)
reg = Ridge(alpha=alpha).fit(X_train, y_train)
print('sklearn ridge:', reg.coef_)  # [37.25, 36.77]


title: How can I change the spatial dimension to the color dimension?(image warping with optical flow in Pytorch)

body:
I’ve some questions of image warping with optical flow.
import torch

from torch.autograd import Variable

import cv2

import numpy as np

import torch.nn as nn

from PIL import Image

import matplotlib.image as img

import matplotlib.pyplot as pp

    def warp(x, flo):
        """
        warp an image/tensor (im2) back to im1, according to the optical flow
        x: [B, C, H, W] (im2)
        flo: [B, 2, H, W] flow
        """
        B, C, H, W = x.size()
        # mesh grid
        xx = torch.arange(0, W).view(1,-1).repeat(H,1)
        yy = torch.arange(0, H).view(-1,1).repeat(1,W)
        xx = xx.view(1,1,H,W).repeat(B,1,1,1)
        yy = yy.view(1,1,H,W).repeat(B,1,1,1)
        grid = torch.cat((xx,yy),1).float()

        if x.is_cuda:
            grid = grid.cuda()
        vgrid = Variable(grid) + flo

        # scale grid to [-1,1]
        vgrid[:,0,:,:] = 2.0*vgrid[:,0,:,:].clone() / max(W-1,1)-1.0
        vgrid[:,1,:,:] = 2.0*vgrid[:,1,:,:].clone() / max(H-1,1)-1.0

        vgrid = vgrid.permute(0,2,3,1)
        output = nn.functional.grid_sample(x.float(), vgrid)
        mask = torch.autograd.Variable(torch.ones(x.size())).cuda()
        mask = nn.functional.grid_sample(mask, vgrid)

        # if W==128:
            # np.save('mask.npy', mask.cpu().data.numpy())
            # np.save('warp.npy', output.cpu().data.numpy())

        mask[mask<0.9999] = 0
        mask[mask>0] = 1

        return output*mask

    optical_path = '.../000000.flo'
    test = cv2.readOpticalFlow(optical_path)

    test = torch.from_numpy(test)
    print(test.size())
    H, W, C = test.size()
    test = test.view(H, W, C, 1)
    # test = test.view(1, 2, H, W)

    test = test.permute(3,2,0,1)
    test = Variable(test).cuda()

    img_path='.../test.png'
    # test_img2 = cv2.imread(img_path)
    im = img.imread(img_path)
    im = np.array(im)
    test_img2 = torch.from_numpy(im)

    print(test_img2.size())
    H, W, C = test_img2.size()
    test_img2 = test_img2.view(1, 3, H, W)
    # test_img2 = test_img2.view(H, W, 3, 1)
    # test_img2 = test_img2.permute(3,2,0,1)

    print(test_img2.size())
    test_img2 = Variable(test_img2).cuda()

    out = warp(test_img2, test)
    out = out.view(H, W, 3)
    out = out.cpu()
    out = np.uint8(out)

    img_name='test3.png'

    image = Image.fromarray(out*255)
    image.save('.../test3.png')
    # cv2.imshow(out*255)

I've run the code above and got a image.
The image is colorless and separated by 9 spaces.
That's not the result that I expected, and I want one image with color.
Please, help me to solve this problem.


title: Customized loss functions PyTorch

body:
I have a regression problem where I want to penalize more on top of a normal MSELoss if the sign of y_true & y_pred is not the same. I am thinking of something like this:
CustomizedLoss(x, y, alpha) = MSELoss(x, y) if sign(x, y) == 1 else alpha * MSELoss(x, y)

I read from https://discuss.pytorch.org/t/custom-loss-function-does-not-converge/87514/3 that when using a sign function - it would return a zero gradient that the model would not be able to learn. I want to know whether the customized loss would make sense to begin with.


title: DataLouder in PyTorch / SpykeTorch: problem of transformed data extraction

body:
Colleagues, I work with Neural Networks in PyTorch and SpykeTorch (based on PyTorch) and I need to create image datasets and place them in DataLouders for further processing.
The complete procedure is as follows:
1.generation of tensors,
2.transform them using torchvision.transforms.ToPILImage (),
3.saving the created images to a directory,
4.creating an ImageFolder based on the directory with image transformation (using filters),
5.create DataLoader from ImageFolder.
image_set = torch.rand([10000, 28, 28], dtype=torch.float)

path = './data/images/'
os.makedirs(path)

tTPI = torchvision.transforms.ToPILImage()

for i in range(n):
    single_image = tTPI(image_set[i])
    image_file = path+f'pic_{i}.jpg'
    saved_image = single_image.save(f'{path}pic_{i}.jpg')

kernels = [ SpykeTorch.utils.DoGKernel(7,1,2),
            SpykeTorch.utils.DoGKernel(7,2,1)]
filter = SpykeTorch.utils.Filter(kernels, padding = 3, thresholds = 50)
s1 = S1Transform(filter)

RandomImageFolder = ImageFolder(root='./data/', transform = s1)
RandomDataLoader = DataLoader(RandomImageFolder, batch_size=len(RandomImageFolder))

Further, the data from the DataLoader is used in work (for example, it is recognized by the neural network).
for data, target in RandomDataLoader:
    prediction_X, prediction_y = predict(model, data, target)

The problem is, when pulling out data and labels from DataLoader, an error occurs:
RuntimeError: Given groups = 1, weight of size [2, 1, 7, 7], expected input [1, 3, 28, 28] to have 1 channels, but got 3 channels instead

Judging by the dimension [1, 2, 7, 7], the error occurs at stage 4, where a set of filters is used for transformation.
However, using a different set of filters in such situations does not cause any error.
How can I solve the problem without changing the filters?


title: How to index with list of indices in a multidimensional array in numpy/pytorch

body:
I have an two multi-dimensional arrays of shape (1,169,5,1+4+4). One contains data, and the other contains zeros that is to be filled.
>>> preds = np.random.random_integers(0,500,(1,169,5,1+4+4))
>>> filtered_preds = np.zeros((1,169,5,1+4+4))

For context the dimensions represent the following:

batch size
number of grid cells
number of anchor boxes
predictions (Confidence score, bounding box coords, number of classes)

I have a third flattened array of shape (169) that contains indicies of the best anchor box
Which looks like the following:
>>> best_bboxes = np.random.random_integers(0,4,(169,))

In order to prevent using loops, my first instinct was to try the following to place the confidence values:
>>> filtered_preds[:,:,best_ab_idx,0:1] = preds[:,:,best_ab_idx,0:1]

However, upon inspecting the array, I do not get the disired output as the shape of filtered_preds[:,:,best_ab_idx,0:1] is (1,169,169,1)
I guess my question so far is, how do I get values from my prediction array with the size of the second dimension and use the value of the third dimension to get the desired value?
I'm still a bit new to array indexing, so any help would be appreciated.
Note: I am using PyTorch but gave my examples in NumPy as it follows the same principle.
Edit:
Here is a solution of what I am trying to acheive using a loop:
for idx,i in enumerate(best_bboxes):
    filtered_preds[:,idx,best_bboxes[idx],0:1] = preds[:,idx,best_bboxes[idx],0:1]
    filtered_preds[:,idx,best_bboxes[idx],1:5] = preds[:,idx,best_bboxes[idx],1:5]
    filtered_preds[:,idx,best_bboxes[idx],5:] = preds[:,idx,best_bboxes[idx],5:]


title: Bitwise shift operator in Pytorch and ONNX export

body:
I have a quantized network from a framework, that I want to export as an ONNX file. The quantization requires adding intermediary layers performing bitwise right shift to avoid overflow. I have to insert these layers in between some other layers of the network.
I think I can use the Bitshift operator from Pytorch and somehow create the additional layers.
My first question is: do such layer already exist or do I have to create them from scratch? For example, could I use a linear layer, with a diagonal weight matrix, and tell the framework not to change the weights?
Then, for the ONNX export, I believe such layers are not supported yet. Is there a simple way to do the export anyway?
Thanks for your help.


title: Confusion about output layer activation function for multi-label classification

body:
I am doing mulit-label classification for tabular data with fastai (implemented using the multi-label helper in the fastai documentation) and the model is performing great! But, when I inspected the model further, I noticed that the output layer is nothing but a linear transformation rather than using a Sigmoid activation function. More specifically, the architecture is
TabularModel(
  (embeds): ModuleList(
    (0): Embedding(25, 10)
    (1): Embedding(9, 5)
    (2): Embedding(33, 11)
    (3): Embedding(32, 11)
    (4): Embedding(8, 5)
    (5): Embedding(207, 32)
    (6): Embedding(3, 3)
    (7): Embedding(3, 3)
    (8): Embedding(3, 3)
    (9): Embedding(3, 3)
    (10): Embedding(3, 3)
    (11): Embedding(3, 3)
    (12): Embedding(5, 4)
  )
  (emb_drop): Dropout(p=0.0, inplace=False)
  (bn_cont): BatchNorm1d(314, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (layers): Sequential(
    (0): LinBnDrop(
      (0): BatchNorm1d(410, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (1): Linear(in_features=410, out_features=200, bias=False)
      (2): ReLU(inplace=True)
    )
    (1): LinBnDrop(
      (0): BatchNorm1d(200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (1): Linear(in_features=200, out_features=100, bias=False)
      (2): ReLU(inplace=True)
    )
    (2): LinBnDrop(
      (0): Linear(in_features=100, out_features=4, bias=True)
    )
  )
)v

I noticed that Jason Brownlee wrote here that for multi-label classification the activation function for the output layer should be a nn.Sigmoid(). I tried replacing the last layer with a LinBnDrop(act=nn.Sigmoid()) layer, but then I get no improvement in validation loss and worse performance overall.
Why is it that that tabular_model implements a linear output layer automatically for multi-label classification?


title: How to understand "torch.randn()" size* parameter arguments?

body:
From what I understand, torch.randn(layers/depth, rows, columns), which can be seen when executing: torch.randn(2, 3, 3) ==> 2 layers (3x3) matrix:
tensor([[[ 1.4838,  1.2926,  1.6147],
     [ 0.7923,  0.6414, -0.2676],
     [-0.1949,  0.3859, -0.6940]],

    [[ 0.2454, -1.9215, -0.3078],
     [ 0.8544,  0.9726,  0.0330],
     [ 0.3579,  0.8247,  2.1288]]])

But what does adding an extra term in the size* parameter imply? As in: torch.randn(2, 1, 3, 3)
tensor([[[[ 0.6206, -1.3697, -0.2267],
      [ 1.0511,  2.3375, -0.9598],
      [-0.8148, -0.0911, -2.1211]]],

    [[[ 0.0659,  1.0764,  0.6150],
      [-1.7226,  0.5038, -0.9544],
      [-0.6447, -0.3325,  0.2048]]]])

What did the "1" add into the Tensor created?


title: how many epoch for training 1k images

body:
am doing training for detecting the objects using yolov3 but i face some problem when i set batch_size > 1 it causes me cuda out of memory so i searched in google to see another solution found it depends on my GPU (GTX 1070 8G) .
may be the number of epoch is  high and it require to be optimized .
maybe the epoch number should be decreased? and for training 1k images and 200 pic for validations .
what is best epoch should i set to avoid overfitting?


title: Managing several versions of Python

body:
Ok, I am relatively new to Python (more of a MATLAB / R /Stata user). I previously installed Python on my computer from the Python website. Everything was running smoothly until I had to install Pytorch too. I tried installing it via pip to no avail, so I had to make a new installation of Python but this time with Anaconda.
However, now I have a mess and I can not load Scypi on Anaconda and I can not load Pytorch in the regular Python I have. Having to run them separately is driving me insane. Is there a way that I can merge the two versions together or should I uninstall and stick to only one?


title: Learning rate finder for CNNLstm model

body:
I have CNNLstm model as follows.
   class CNN(nn.Module):
        def __init__(self):
            super(CNN, self).__init__()
            self.conv1 = nn.Sequential(
                nn.Conv2d(
                    in_channels=3,
                    out_channels=16,
                    kernel_size=5,
                    stride=1,
                    padding=2,
                ),
                nn.ReLU(),
                nn.MaxPool2d(kernel_size=2),
            )
            self.conv2 = nn.Sequential(
                nn.Conv2d(16, 32, 5, 1, 2),
                nn.ReLU(),
                nn.MaxPool2d(kernel_size=2),
            )
            #print(num_classes)
            self.out = nn.Linear(32 * 75 * 75, num_classes)#32 * 75 * 75/64 * 37 * 37/128 * 18 * 18

        def forward(self, x):
            x = self.conv1(x)
            x = self.conv2(x)
            x = x.view(x.size(0), -1)
            output = self.out(x)
            return output, x

    import torch
    from torchvision import datasets, transforms
    import torch.nn.functional as f
    from torch_lr_finder import LRFinder
    class CnnLstm(nn.Module):
        def __init__(self):
            super(CnnLstm, self).__init__()
            self.cnn = CNN()
            self.rnn = nn.LSTM(input_size=180000, hidden_size=256, num_layers=3, batch_first=True)#stacked LSTM with 2 layers
            self.linear = nn.Linear(256, num_classes)

        def forward(self, x):
            batch_size, time_steps, channels, height, width = x.size()
            c_in = x.view(batch_size * time_steps, channels, height, width)
            _, c_out = self.cnn(c_in)
            r_in = c_out.view(batch_size, time_steps, -1)
            r_out, (_, _) = self.rnn(r_in)
            r_out2 = self.linear(r_out[:, -1, :])
            return f.log_softmax(r_out2, dim=1)

class TrainCNNLSTM:
    def __init__(self):
        self.seed = 1
        self.batch_size = 8
        self.validate_batch_size = 8
        self.test_batch_size = 1
        self.epoch = 50
        self.learning_rate = 0.005
        self.step = 100
        self.train_loader = None
        self.validate_loader = None
        self.test_loader = None
        self.modelloaded = False
        self.model = CnnLstm().to(device)
        self.criterion = nn.CrossEntropyLoss()
        #self.optimizer = torch.optim.SGD(self.model.parameters(), lr=self.learning_rate)#self.learning_rate = 0.001
        self.optimizer = torch.optim.AdamW(self.model.parameters())
        #self.scheduler = optim.lr_scheduler.OneCycleLR(self.optimizer, 2e-3, epochs=self.epoch , steps_per_epoch=len(train_loader))

    def load_data(self):
        data_loader = DataLoader()
        self.train_loader = data_loader.get_train_data(self.batch_size)
        self.validate_loader = data_loader.get_validate_data(self.validate_batch_size)
        self.test_loader = data_loader.get_test_data(self.test_batch_size)

    def do_lrfinder(self):
        lr_finder = LRFinder(self.model, self.optimizer, self.criterion, device)
        lr_finder.range_test(self.train_loader, end_lr=1, num_iter=1000)
        lr_finder.plot()
        plt.savefig("LRvsLoss.png")
        plt.close()

    def train(self):
        for epoch in range(0, self.epoch):
            t_losses=[]
            for iteration, (data, target) in enumerate(self.train_loader):
                print(data.shape)
                data = np.expand_dims(data, axis=1)
                print(data.shape)
                data = torch.FloatTensor(data)
                data, target = data.cuda(), target.cuda()
                data, target = Variable(data), Variable(target)
                self.optimizer.zero_grad()

Since it is CNNLstm model, the data input shape to the model are batch_size, time_steps, channels, height, width.
(8, 1, 3, 300, 300)
To use torch_lr_finder, we need to run the following code.
lr_finder = LRFinder(self.model, self.optimizer, self.criterion, device)
lr_finder.range_test(self.train_loader, end_lr=1, num_iter=1000)

self.train_loader output shape is (8, 3, 300, 300). So during finding learning rate, self.model can't be used.
How can I use torch_lr_finder for such model?


title: What is accumulating during training pytorch model

body:
While in the process of training, training went up to 3% and then  Out of Memory occured. But while I was thinking about this error, I wonder about something that, Because learning goes well at first and then became Out of Memory it would be something is accumulated as training progrsses. What is accumulating?
def init_process_group(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    dist.init_process_group("gloo", rank = rank,world_size =world_size)

def build_data_loader(vocab, infile, args, shuffle = True):
    dataset = MovieDataset(vocab, infile)
    if 1< args.n_gpu and shuffle:
        sampler = torch.utils.data.distributed.DistributedSampler(dataset)
        loader = torch.utils.data.DataLoader(dataset, batch_size=config.tconfig.batch_size, sampler = sampler, collate_fn =movie_collate_fn)
    else :
        sampler = None
        loader = torch.utils.data.DataLoader(dataset, batch_size=config.tconfig.batch_size, sampler=sampler, shuffle=shuffle, collate_fn = movie_collate_fn)
    return loader, sampler

def train_model(rank, world_size, args):

    if 1 < args.n_gpu:
        init_process_group(rank, world_size)
    master = (world_size == 0 or rank % world_size ==0)

    config.tconfig.device = torch.device(f"cuda:{rank}" if torch.cuda.is_available() else "cpu")
    print(config.tconfig)

    best_epoch, best_loss, best_score = 0, 0, 0
    model = MovieClassification(config.tconfig)

    if os.path.isfile(args.save):
        print(f"rank:{rank} load state dict from : {args.save}")
    if 1 < args.n_gpu:
        model.to(config.tconfig.device)
        model = DistributedDataParallel(model, device_ids = [rank], find_unused_parameters = True)

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument("--gpu", default = None, type = int, required = False)
    args = parser.parse_args()
    config.tconfig.device = config.device

    if torch.cuda.is_available():
        args.n_gpu = torch.cuda.device_count() if args.gpu is None else 1
    else:
        args.n_gpu = 0
    print("available GPU : ",args.n_gpu)
    if 1 < args.n_gpu:
        mp.spawn(train_model, args= (args.n_gpu,args), nprocs = args.n_gpu, join = True)

and below is the oom error message
-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/fightnyy/anaconda3/envs/selfresearch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/fightnyy/workspace/kotransformer/main.py", line 145, in train_model
    loss = train_epoch(config.tconfig, rank, epoch, model, criterion, optimizer, scheduler, train_loader)
  File "/home/fightnyy/workspace/kotransformer/main.py", line 88, in train_epoch
    outputs = model.forward(enc_inputs, dec_inputs)
  File "/home/fightnyy/anaconda3/envs/selfresearch/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 619, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/fightnyy/anaconda3/envs/selfresearch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/fightnyy/workspace/kotransformer/movieclassifier.py", line 24, in forward
    dec_outputs, enc_self_attn_probs, self_attn_probs, dec_enc_attn_probs = self.transformer.forward(enc_inputs, dec_inputs)
  File "/home/fightnyy/workspace/kotransformer/transformer.py", line 19, in forward
    enc_outputs, enc_self_attn_probs = self.encoder.forward(enc_inputs)
  File "/home/fightnyy/workspace/kotransformer/encoder.py", line 64, in forward
    outputs, attn_prob = layer.forward(outputs, attn_mask)
  File "/home/fightnyy/workspace/kotransformer/encoderlayer.py", line 24, in forward
    pos_output = self.pos_ffnn(attn_output)
  File "/home/fightnyy/anaconda3/envs/selfresearch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/fightnyy/workspace/kotransformer/PFFN.py", line 23, in forward
    output = self.activation(self.conv1(inputs.transpose(1,2)))
  File "/home/fightnyy/anaconda3/envs/selfresearch/lib/python3.8/site-packages/torch/nn/functional.py", line 1383, in gelu
    return torch._C._nn.gelu(input)
RuntimeError: CUDA out of memory. Tried to allocate 118.00 MiB (GPU 0; 10.76 GiB total capacity; 5.05 GiB already allocated; 77.44 MiB free; 5.31 GiB reserved in total by PyTorch)


title: PyTorch: Adding hooks to model for saving the intermediate layers output returns the features two times

body:
I want to implement the Style Transfer paper using Pytorch and the VGG19 network.
For this, I need the intermediate output features for some layers.
I named the convolutional modules: ['conv_1', 'conv_2',..., 'conv_16']
For managing the hooks and features I use this method:
class SaveOutput:
  #Callable object for saving the layers outputs
  def __init__(self):
    self.outputs = []

  def __call__(self, module, module_in, module_out):
    self.outputs.append(module_out)

  def clear(self):
    self.outputs = []

def addHooksToModel(model, layerNames, hookHandles):
  #Remove hooks
  for hook in hookHandles:
    hook.remove()

  hookHandles = []
  features = SaveOutput()

  for name, module in model.named_modules():
    if name in layerNames:
      hookHandles.append(module.register_forward_hook(features))

  return features, hookHandles

I want to store the features of the content and style separately:
CONTENT_LAYERS = ["conv_14"]
STYLE_LAYERS = ["conv_1","conv_3","conv_5","conv_9","conv_13"]

hook_handles_content = []
hook_handles_style = []
content_features, hook_handles_content = addHooksToModel(model, CONTENT_LAYERS, hook_handles_content)
style_features, hook_handles_style = addHooksToModel(model, STYLE_LAYERS, hook_handles_style)

I then pass the contentImage and styleImage through the network and I expect the content_features.outputs to contain 1 tensor and the style_features.outputs to contain 5 tensors.
model(contentImage)
contentImg_content = content_features.outputs
content_features.clear()

model(styleImage)
styleImg_style = style_features.outputs
style_features.clear()

But in reality, I get 1 tensor for the content_features.outputs(as expected), but 10 tensors for the style_features.outputs(two times the expected).
Same thing happens if I first pass the styleImage and then the contentImage. I get 5 tensors for the style_features.outputs(as expected), but 2 tensors for the content_features.outputs(two times the expected).
Could somebody point me in the right direction. I know I'm missing something, probably in the way Pytorch hooks are working, but I can't figure out what. Thank you!


title: Can I access the inner layer outputs of DeepLab in pytorch?

body:
Using Pytorch, I am trying to implement a network that is using the pre=trained DeepLab ResNet-101.
I found two possible methods for using this network:
this one
or
 torchvision.models.segmentation.deeplabv3_resnet101(
     pretrained=False, progress=True, num_classes=21, aux_loss=None, **kwargs)

However, I might not only need this network's output, but also several inside layers' outputs.
Is there a way to access the inner layer outputs using one of these methods?
If not - Is it possible to manually copy the trained resnet's parameters so I can manually recreate it and add those outputs myself? (Hopefully the first option is possible so I won't need to do this)
Thanks!


title: using cv2.UMat() with cifar10 data

body:
I am trying to gabor filter cifar10 data and I get that error
TypeError: UMat() missing required argument 'ranges' (pos 2).

I added double asterisks to the line giving me the error in my code, see below.
I tried np.float32() but got that error
error: OpenCV(4.1.0) c:\projects\opencv-python\opencv\modules\imgproc\src\color.simd_helpers.hpp:92: error: (-2:Unspecified error) in function '__cdecl cv::impl::`anonymous-namespace'::CvtHelper<struct cv::impl::`anonymous namespace'::Set<3,4,-1>,struct cv::impl::A0xe227985e::Set<1,-1,-1>,struct cv::impl::A0xe227985e::Set<0,2,5>,2>::CvtHelper(const class cv::_InputArray &,const class cv::_OutputArray &,int)'
> Invalid number of channels in input image:
>     'VScn::contains(scn)'
> where
>     'scn' is 1

I tried np.array() but got that error
error: OpenCV(4.1.0) c:\projects\opencv-python\opencv\modules\imgproc\src\color.simd_helpers.hpp:92: error: (-2:Unspecified error) in function '__cdecl cv::impl::`anonymous-namespace'::CvtHelper<struct cv::impl::`anonymous namespace'::Set<3,4,-1>,struct cv::impl::A0xe227985e::Set<1,-1,-1>,struct cv::impl::A0xe227985e::Set<0,2,5>,2>::CvtHelper(const class cv::_InputArray &,const class cv::_OutputArray &,int)'
> Invalid number of channels in input image:
>     'VScn::contains(scn)'
> where
>     'scn' is 1

Here is my code:
import numpy as np
import cv2
import matplotlib.pyplot as plt
import torchvision
from torchvision import models, datasets, transforms
import torch

ksize = 15  #Use size that makes sense to the image and fetaure size. Large may not be good.
#On the synthetic image it is clear how ksize affects imgae (try 5 and 50)
sigma = 5 #Large sigma on small features will fully miss the features.
theta = 1*np.pi/2  #/4 shows horizontal 3/4 shows other horizontal. Try other contributions
lamda = 1*np.pi/4  #1/4 works best for angled.
gamma=0.9  #Value of 1 defines spherical. Calue close to 0 has high aspect ratio
#Value of 1, spherical may not be ideal as it picks up features from other regions.
phi = 0.8  #Phase offset. I leave it to 0. (For hidden pic use 0.8)

kernel = cv2.getGaborKernel((ksize, ksize), sigma, theta, lamda, gamma, phi, ktype=cv2.CV_32F)
# Read config to figure out whether to use CUDA.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# CUDA-related parameters. (Got this from pytorch's official tutorials.)
kwargs = {'num_workers': 1, 'pin_memory': True} if torch.cuda.is_available()else {}

# Load Cifar10 dataset using standard torchvision package.
data_transform_train = transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])])

cifar_trainset = torch.utils.data.DataLoader(datasets.CIFAR10(root='./data',
 train=True, download=True,transform=data_transform_train),
 batch_size= 10, shuffle=True, **kwargs)

data_transform_test = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])])

cifar_testset = torch.utils.data.DataLoader( datasets.CIFAR10(root='./data',
 train=False, download=True, transform=data_transform_test),
    batch_size= 10, shuffle=False, **kwargs)
for batch_idx, (x, t) in enumerate(cifar_trainset):
    current_batch_size = x.data.size()[0]
    x, t = x.to(device), t.to(device)
    **x = cv2.cvtColor(cv2.UMat(x), cv2.COLOR_BGR2GRAY)**
    fimg = cv2.filter2D(x, cv2.CV_8UC3, kernel)

    torch.save(cifar_trainset, 'x_trainset/gf_cifar_trainset'+str(batch_idx)+'.pt')
    torch.save (t, 't_trainset/gf_cifar_trainset'+str(batch_idx)+ '.pt')

for batch_idx, (x, t) in enumerate(cifar_testset):
    current_batch_size = x.data.size()[0]
    x, t = x.to(device), t.to(device)
    x = cv2.cvtColor(cv2.UMat(x), cv2.COLOR_BGR2GRAY)
    fimg = cv2.filter2D(x, cv2.CV_8UC3, kernel)

    torch.save(cifar_testset, 'x_testset/gf_cifar_testset'+str(batch_idx)+'.pt')
    torch.save(t, 't_testset/gf_cifar_testset' + str(batch_idx) + '.pt')


title: Automatic Differentiation:numerical or exact?

body:
I have a question about automatic differentiation and especially in Pytorch since i am using this library.I have seen for instance automatic differentiation give the partial derivatives of an expression with respect to a variable.
However,as far as what i have seen ,the result is always given at a specific point,meaning it is a tensor with numerical values.My question is the following: let's say we define a function of two variables:
f(x,y)= x² + y² .
Is Pytorch able to return a function which corresponds to the partial derivative of f with respect to x or y? That is to return the following definitions:
> def partial_f_x:

      return 2*x

  def partial_f_y:

      return 2*y

Because even though the function f here is rather simple,it would be interesting if Pytorch could give us a formula(depending on the different variables) of the derivatives,instead of giving a numerical value at a given point,because in that case we do not know the expression of the derivatives.
So if I summarize: is Pytorch able to return formulas for derivatives of complicated functions? or does it just return a tensor with numerical values for the derivative at a given point?
Thank you so much!


title: forward in pytorch nn.linear gives NaN

body:
I am working on a Pytorch model. The training set looks like this:

Pclass  Sex Age SibSp   Parch   Fare    Embarked
0   3   0   1   1   0   0   1.0
1   1   1   1   1   0   3   2.0
2   3   1   1   0   0   0   1.0
3   1   1   1   1   0   3   1.0
4   3   0   1   0   0   0   1.0
... ... ... ... ... ... ... ...
886 2   0   1   0   0   0   1.0
887 1   1   0   0   0   1   1.0
888 3   1   0   1   2   0   1.0
889 1   0   1   0   0   1   2.0
890 3   0   1   0   0   0   0.0
891 rows × 7 columns

similar is the test set. The training_label is one hot encoded.
The model is:
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(7, 10)
        self.bc1 = nn.BatchNorm1d(10)
        self.fc2 = nn.Linear(10, 10)
        self.bc2 = nn.BatchNorm1d(10)
        self.fc3 = nn.Linear(10, 1)
        self.bc3 = nn.BatchNorm1d(1)

    def forward(self, x):
        x = self.fc1(x)
        x = self.bc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = self.bc2(x)
        x = F.relu(x)
        x = self.fc3(x)
        x = self.bc3(x)
        #x = F.sigmoid(x)
        return x

model = Net()

I am calling the TensorDataset and DataLoader to load (train_data, train_label) of batchsize 100.
I am trying to call the model:
num_epochs = 1
# Repeat for given number of epochs
for epoch in range(num_epochs):

    # Train with batches of data
    for xb,yb in train_dl:
        pred = model(xb)

        yb = torch.unsqueeze(yb, 1)

        print(pred, yb)
        #print('grad', model.fc1.weight.grad)

        l = loss(pred, yb)
        print('loss',l)

        # 3. Compute gradients
        l.backward()

        # 4. Update parameters using gradients
        optimizer.step()

        # 5. Reset the gradients to zero
        optimizer.zero_grad()

After some time, I am getting NaN as output from the pred = model(xb). As you can see, I am running for only 1 epoch, so I am getting the NaN in the first epoch for some batch.
I am not sure why it is happening. I checked and found some solutions to it, like reducing the epoch but my epoch is already very low. I also have preprocessed the dataset so that all the inputs are 0,1,2,3 or 4 (like I preprocessed Age, Gender etc).
I am not getting any error message but I can see the outputs as NaN and so is the loss of course, as I print them. I also tried different optimization, loss. Right now I am using:
#model.cuda()
#loss = nn.BCEWithLogitsLoss()
loss = F.mse_loss

learning_rate = 0.0001
optimizer = torch.optim.SGD(model.parameters(), lr=1e-5)

I also tried BCEWithLogitsLoss. nn.BCELoss gives the error that the inputs should be between 0 and 1 (and here some inputs are becoming Nan)
Can anyone help and explain what is going on?


title: Pytorch DDP get stuck in getting free port

body:
I try to get a free port in DDP initialization of PyTorch. However, my code get stuck. The following snippet could repeat my description:
def get_open_port():
    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s:
        s.bind(('', 0))
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        return s.getsockname()[1]

def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    port = get_open_port()
    os.environ['MASTER_PORT'] = str(port)   # '12345'

    # Initialize the process group.
    dist.init_process_group('NCCL', rank=rank, world_size=world_size)

class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = nn.Linear(10, 5)

    def forward(self, x):
        print(f'x device={x.device}')
        return self.net1(x)

def demo_basic(rank, world_size):
    setup(rank, world_size)

    logger = logging.getLogger('train')
    logger.setLevel(logging.DEBUG)
    logger.info(f'Running DPP on rank={rank}.')

    # Create model and move it to GPU.
    model = ToyModel().to(rank)
    ddp_model = DDP(model, device_ids=[rank])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)  # optimizer takes DDP model.

    optimizer.zero_grad()
    inputs = torch.randn(20, 10)  # .to(rank)

    print(f'inputs device={inputs.device}')
    outputs = ddp_model(inputs)
    print(f'output device={outputs.device}')

    labels = torch.randn(20, 5).to(rank)
    loss_fn(outputs, labels).backward()

    optimizer.step()

    cleanup()

def run_demo(demo_func, world_size):
    mp.spawn(
        demo_func,
        args=(world_size,),
        nprocs=world_size,
        join=True
    )

run_demo(demo_basic, 4)

The function get_open_port is supposed to free the port after invocation. My questions are: 1. How does it happen? 2. How to fix it?


title: How to replace NaN values in the weights of neural network?

body:
I am trying to replace NaN values in the weights with some constant. However I am unaware on how one can do so.
I am aware that to access weights, I have to write.
for i in model.params:
    arr = i.clone()
    arr[torch.isnan(arr)] = 0
    param_list.append(nn.Parameter(arr))

model.params= param_list

But this breaks the computation graph, any better way you can suggest me for the same?


title: loading saved model and optimizer for testing pytorch

body:
I am on ubuntu 18.04 using torch==1.7.1
What I want to do is train a reinforcement learning agent and send it to another server therefore trained agent can play right away.
I've been following https://pytorch.org/tutorials/beginner/saving_loading_models.html however in need of some clarification.
After all training has been finished I've saved agent, optimizer via:
ckpt = {
    'Epoch': epoch,
    'model' : agent.state_dict(),
    'optimizer' : optimizer.state_dict()
}
torch.save(ckpt, filename)

Now, I've sent this saved file to different server and loaded it like following:
ckpt = torch.load(ckpt_FILE_PATH)

agent = Agent()
optimizer = optim.Adam(agent.parameters(), lr=0.0005)

agent.load_state_dict(ckpt["agent"])
optimizer.load_state_dict(ckpt["optimizer"])

agent.eval()

I just want to make sure one thing. during agent instance and optimizer creation in training phase I've instantiated optimizer using agent.parameters()
so my question is do I need to

load_state_dict to agent then instantiate optimizer using agent.parameters()
just instantiate optimizer before loading load_state_dict to agent?
doesn't matter.

Thanks in advance.


title: How do I prevent a lack of VRAM halfway through training a Huggingface Transformers (Pegasus) model?

body:
I'm taking a pre-trained pegasus model through Huggingface transformers, (specifically, google/pegasus-cnn_dailymail, and I'm using Huggingface transformers through Pytorch) and I want to finetune it on my own data. This is however quite a large dataset and I've run into the problem of running out of VRAM halfway through training, which because of the size of the dataset can be a few days after training even started, which makes a trial-and-error approach very inefficient.
I'm wondering how I can make sure ahead of time that it doesn't run out of memory. I would think that the memory usage of the model is in some way proportional to the size of the input, so I've passed truncation=True, padding=True, max_length=1024 to my tokenizer, which if my understanding is correct should make all the outputs of the tokenizer of the same size per line. Considering that the batch size is also a constant, I would think that the amount of VRAM in use should be stable. So I should just be able to cut up the dataset into managable parts, just looking at the ram/vram use of the first run, and infer that it will run smoothly from start to finish.
However, the opposite seems to be true. I've been observing the amount of VRAM used at any time and it can vary wildly, from ~12GB at one time to suddenly requiring more than 24GB and crashing (because I don't have more than 24GB).
So, how do I make sure that the amount of vram in use will stay within reasonable bounds for the full duration of the training process, and avoid it crashing due to a lack of vram when I'm already days into the training process?


title: Why is tensor being converted to numpy array?

body:
For some reason, the tensor datatype is being changed to a numpy array in the second iteration. I can't figure out why.
y_true = torch.tensor([])
y_pred = torch.tensor([])
for i, (x,y) in enumerate(zip(feature_testloader,label_testloader), 0):
    print(i,type(y_true))
    optimizer.zero_grad()
    output = self.forward(x)
    target = y
    y_true = torch.cat((y_true, target))
    _, pred_class = torch.max(output, dim=1)
    y_pred = torch.cat((y_pred, pred_class))

Output from print statement above:
0 <class 'torch.Tensor'>
1 <class 'numpy.ndarray'>


title: Installing pytorch with conda, failed to create dir

body:
I am trying to run some code from github, and having issues. I am trying to download dependencies , which are included in environment.yml. I ran "conda env create -f environment.yml", but got following message.
InvalidArchiveError("Error with archive /home/mchoi/anaconda3/pkgs/pytorch-1.1.0-py3.6_cuda10.0.130_cudnn7.5.1_0.tar.bz2.  You probably need to delete and re-download or re-create this file.  Message from libarchive was:\n\nFailed to create dir 'lib/python3.6/site-packages/torch/for_onnx'")
At first, the terminal asked me to update the conda, so thought thats the problem but im keep getting the same message after the update. I tried to delete and re-download, but same thing happened.


title: NER tagging schema for non-contiguous tokens

body:
The most common tagging procedure for NER is IOB. But it seems that this kind of tagging is limited to cases where tokens from the same entity are contiguous.
So for instance,
Jane Smith is walking in the park would be tagged as: B-PER I-PER O O O O O
And here my PER entity is the concatenation of [Jane, Smith]
If we tweak the example:
Jane and James Smith are walking in the park
B-PER O B-PER I-PER O O O O O
Now the issue is that the entities we would get are [Jane] and [James, Smith] because the IOB tagging does not allow to link Jane to Smith.
Is there any tagging schema that would allow to mark as entities both [Jane, Smith] and [James, Smith]?


title: Spliting the dataset using SubsetRandomSampler not working

body:
I used SubsetRandomSampler to split the training data to train (80%) and validation data (20%). But it is showing the same number of images for both after the split (4996):
>>> print('len(train_data): ', len(train_loader.dataset))
>>> print('len(valid_data): ', len(validation_loader.dataset))

len(train_data):  4996
len(valid_data):  4996

Full code:
import numpy as np
import torch
from torchvision import transforms
from torch.utils.data.sampler import SubsetRandomSampler

train_transforms = transforms.Compose([transforms.ToTensor(),
                                       transforms.Normalize([0.485, 0.456, 0.406],
                                                            [0.229, 0.224, 0.225])])
dataset = datasets.ImageFolder( '/data/images/train', transform=train_transforms )

validation_split = .2
shuffle_dataset = True
random_seed= 42
batch_size = 20

dataset_size = len(dataset) #4996
indices = list(range(dataset_size))
split = int(np.floor(validation_split * dataset_size))

if shuffle_dataset :
    np.random.seed(random_seed)
    np.random.shuffle(indices)
train_indices, val_indices = indices[split:], indices[:split]

train_sampler = SubsetRandomSampler(train_indices)
valid_sampler = SubsetRandomSampler(val_indices)

train_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=train_sampler)
validation_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=valid_sampler)


title: Confusion about the shape of the output logits from Resnet

body: I am trying to understand why the shape of the output logits from the Resnet18 model I am working with are (27, 19). The shape of 19 I understand, that is the number of classes I have set the model to predict, but the shape of 27 is the part that I am confused about. I have a batch size of 64 so I would have thought the shape of the logits would be (64, 19), because that would give me 1 prediction vector for each image in the batch...


title: How to get layer execution time on an AI model saved as .pth file?

body:
I'm trying to run a Resnet-like image classification model on a CPU, and want to know the breakdown of time it takes to run each layer of the model.
The issue I'm facing is the github link https://github.com/facebookresearch/semi-supervised-ImageNet1K-models has the model saved as a .pth file. It is very large (100s of MB), and I don't know exactly how it differs from pytorch except it's binary.
I load the model from this file using the following script. But I don't see a way to modify the model or insert the t = time.time() variables/statements in between model layers to break down the time in each layer.
Questions:

Would running the model in the following script give a correct estimate of end-to-end time (t2-t1) it takes to run the model on the CPU, or would it also include pytorch compilation time?

How to insert time statements between consecutive layers to get a breakdown?

There is no inference/training script at the github link and only has the .pth file. So how exactly is one supposed to run inference or training? How to insert additional layers between consecutive layers of the .pth model and save them?

#!/usr/bin/env python
import torch torchvision time

model=torch.hub.load('facebookresearch/semi-supervised-ImageNet1K-models', 'resnext50_32x4d_swsl', force_reload=False)
in = torch.randn(1, 3, 224, 224)
t1 = time.time()
out = model.forward(in)
t2 = time.time()
```**strong text**


title: TorchServe CORS Settings

body:
To enable CORS(cross origion resoure sharing) in TorchServe, I am using the following settings in the config.properties. Can you please have a look and let me know if I am using the right settings/syntax. My torchserve settings need to return header reponse as  *"Access-Control-Allow-Origin: "
Note: not using a custom header
cors_allowed_origin=*
cors_allowed_methods=GET, POST, PUT, OPTIONS
Reference: https://pytorch.org/serve/configuration.html#configure-cross-origin-resource-sharing-cors


title: Give correct datatype to pytorch, but it does not accept

body:
class Model(nn.Module):
    def __init__(self,
                input_size=12175,
                hidden_size=6,
                num_layers=1,
                batch_size=1,
                sequence_length=1,
                num_classes=6):
        """RNN and FC. hidden_size and num_classes MUST equal."""
        super().__init__()
        self.rnn = nn.RNN(input_size=input_size,
                         hidden_size=hidden_size,
                         batch_first=True)
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.batch_size = batch_size
        self.sequence_length = sequence_length
        self.num_classes = num_classes

        # Fully-Connected layer
        self.fc = nn.Linear(num_classes, num_classes)

    def forward(self, x, hidden):
        import ipdb; ipdb.set_trace()
        # Reshape input in (batch_size, sequence_length, input_size)
        x = x.view(self.batch_size, self.sequence_length, self.input_size)
        x = x.double()
        hidden = hidden.double()
        out, hidden = self.rnn(x, hidden)
        out = self.fc(out) # Add here
        return hidden, out

    def init_hidden(self):
        return torch.zeros(self.num_layers, self.batch_size, self.hidden_size)

And training
import itertools
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix

CLASS_LENGTH = 6

def train(model, device, train_loader, optimizer, epoch, criterion):
    """
    This function has one line different from the ordinary `train()` function
    It has `make_variables()` to convert tuple of names to be a tensor
    """
    model.train()
    hidden = model.init_hidden()
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
#         hidden = hidden.view(model.batch_size, model.sequence_length, CLASS_LENGTH)
        output, hidden = model(data, hidden)

        tmp = output.view(model.batch_size, CLASS_LENGTH)
        loss = criterion(tmp, target)
        loss.backward(retain_graph=True)
        optimizer.step()
        if batch_idx % 1000 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))

def test(model, device, test_loader, criterion):
    model.eval()
    test_loss = 0
    correct = 0
    y_test = []
    y_pred = []
    with torch.no_grad():
        for data, target in tqdm(test_loader):
            data, target = data.to(device), target.to(device)
            output, hidden = model(data, hidden)
            tmp = output.view(-1, COUNTRY_LENGTH)
            test_loss += criterion(tmp, target).item() # sum up batch loss
            pred = tmp.max(1, keepdim=True)[1] # get the index of the max log-probability

            pred_tmp = pred.view(-1)
            pred_list = pred_tmp.tolist()
            target_list = target.tolist()

            y_test += target_list
            y_pred += pred_list
            correct += pred.eq(target.view_as(pred)).sum().item()

    test_loss /= len(test_loader.dataset)

    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))

    # Confusion matrix
    confusion_mtx = confusion_matrix(y_test, y_pred)
    plot_confusion_matrix(confusion_mtx, classes=test_loader.dataset.countries, normalize=True,
                          title='Confusion matrix')

Problem:
I do not understand why it raise error since I am sure that I put correct datatype into it
ipdb> n
> /tmp/ipykernel_16/2018411812.py(30)forward()
     29         hidden = hidden.double()
---> 30         out, hidden = self.rnn(x, hidden)
     31         out = self.fc(out) # Add here

ipdb> n
RuntimeError: expected scalar type Double but found Float
> /tmp/ipykernel_16/2018411812.py(30)forward()
     29         hidden = hidden.double()
---> 30         out, hidden = self.rnn(x, hidden)
     31         out = self.fc(out) # Add here

ipdb> x
tensor([[[-5.6964e-01, -5.1070e-01, -5.9109e-01,  ...,  1.5597e-15,
           1.5597e-15,  1.5597e-15]]], dtype=torch.float64)
ipdb> hidden
tensor([[[0., 0., 0., 0., 0., 0.]]], dtype=torch.float64)

pytorch version: 1.8.1+cu102
OSX 10.15.7
i7
Question:
How to solve this problem?