title: parallel execution of inference of deep learning model which are divided into parts

body:
Here I have a scenario of training a deep neural network(image classification n/w) say of 8 layers.
after training , I split the model in 4 parts each model part having 2 layers each.
Now i want to do the inference on my test image , technically speaking the image passed would move through every block to produce the final output which will be serial processing.
i.e
total_time = time_taken_by_model_1 + time_taken_by_model_2 + time_taken_by_model_3 + time_taken_by_model_4
Some thing like shown in below image

so my question is, instead of this serial computation ,Can I parallelize the inference computation such that time taken should be less than serialized inference from above total_time ( atleast half of the total_time from above case)
Any links/paper/blogs  about these problems can also be helpful
EDIT : all model parts will be on different devices(raspberry pi)


title: Problems identifying images in pytorch

body:
I am really new at deep learning and I am studying how to properly run neural networks using pytorch. Currently I am trying to read a dataset of images using the following code:
from torch.utils.data import Dataset, DataLoader
from torchvision import datasets, transforms
from torchvision import transforms, utils

transformations = transforms.Compose([
    transforms.Resize(255),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])
train_set = datasets.ImageFolder('faces/train', transform = transformations)
test_set = datasets.ImageFolder('faces/test', transform = transformations)

train_loader = DataLoader(train_set, batch_size=60, shuffle = True)
test_loader = DataLoader(test_set, batch_size=60, shuffle = True)

once done that, I am trying to read the images to start running the neural networks on the images, first by separating the imgs from the labels using:
img, labels = next(iter(train_loader))

And then I received the following message:
UnidentifiedImageError: cannot identify image file <_io.BufferedReader name='faces/train/karyadi/karyadi_straight_angry_open.pgm'>

The structure of the folders on my working directory where I have stored the images is as follows:

faces:

Train: 10 folders, each one with around 90 images in pgm format.
Test: 1o folders, each one with around 90 images in pgm format.

does anyone know what the problem might be?
thanks in advance.


title: Output of CNN should be image

body:
I am pretty new to deep learning, so I got one question:
Assume an input Grayscale image of shape (128,128,1). Target (Output) is as well an (128,128,1) sized image, e.g. for segmentation, depth prediction etc.. Usually with valid padding the size of the image shrinks after several convolution layers.
What are decent (maybe not the toughest one) variants to keep the size or predict a same sized image? Is it via same-padding? Is it via tranpose convolution or upsampling? Should I use a FCN at the end and reshape them to the image size? I am using pytorch. I would be glad for any hints, because I didn't find much in the internet.
Best


title: I wonder how to put the parameters in nn.LSTM in this situation

body:
I want to say that I'm not familiar with English.
Hi, I'm recently building LSTM Layers using nn.LSTM.
What I want to do is predicting next 48 values using 48*7=336 sequential inputs.
And here is preprocessed dataframe will be used. It has 384 cols(=336+48) and 457068 rows(=929x492).
In short, there are 929 customers and each customers has 492 sequential data with a length of 384(=336+48)

customerID
col1
col2
...
col380

1
data_{1,1}
data_{1,2}
...
data_{1,380}

1
data_{2,1}
data_{2,2}
...
data_{2,380}

...
...
...
...
...

1
data_{492,1}
data_{492,2}
...
data_{492,380}

2
data_{493,1}
data_{493,2}
...
data_{493,380}

...
...
...
...
...

929
data_{4670687,1}
data_{4670687,2}
...
data_{4670687,380}

929
data_{4670688,1}
data_{4670688,2}
...
data_{4670688,380}

Here, 3 parameters for nn.LSTM : (input_size, hidden_size, num_layers)
Also Input tensor has 3 dimensions : (seq_len, batch, input_size)\
In this situation, please let me know how to enter the above parameters. I will use num_layers=2.
Thank You.


title: MNIST plot first test figure after transform

body:
Here are some of my code, I want to know how to add code to plot the first figure from the test dataset after transforms?
transform=transforms.Compose([
        transforms.ToTensor(),
        AddGaussianNoise(0, 1),
        transforms.Normalize((0.1307,), (0.3081,))
        ])
    dataset1 = datasets.MNIST('../data', train=True, download=True,
                       transform=transform)
    dataset2 = datasets.MNIST('../data', train=False,
                       transform=transform)
    train_loader = torch.utils.data.DataLoader(dataset1,**train_kwargs)
    test_loader = torch.utils.data.DataLoader(dataset2, **test_kwargs)

    model = Net().to(device)
    optimizer = optim.Adadelta(model.parameters(), lr=args.lr)

    scheduler = StepLR(optimizer, step_size=1, gamma=args.gamma)
    for epoch in range(1, args.epochs + 1):
        train(args, model, device, train_loader, optimizer, epoch)
        test(model, device, test_loader)
        scheduler.step()

    if args.save_model:
        torch.save(model.state_dict(), "mnist_cnn.pt")


title: Predicted value in simple LSTM keeps accumulating

body:
After each epoch y_pred simply keeps increasing
input at each batch is 64x10 tensor, trying to predict max of the vector at each row.
I thought the gradient might not be going to 0 between batches, but I that wasn't the case.
I tried changing LR, epoch, LSTM layers (LSTM to RNN), hidden size etc, nothing helped.
BTW, using simple sequential network of dense and relu instead of lstm worked perfectly
Following is the code:
LR = 0.0001
class LSTM(nn.Module):
    def __init__(self, input_size=1, hidden_layer_size=100, output_size=1):
        super().__init__()
        self.hidden_layer_size = hidden_layer_size

        self.lstm = nn.LSTM(input_size, hidden_layer_size)

        self.linear = nn.Linear(hidden_layer_size, output_size)

#         self.hidden_cell = (torch.zeros(1,max_array_len,self.hidden_layer_size),
#                             torch.zeros(1,max_array_len,self.hidden_layer_size))

    def forward(self, input_seq):
#         lstm_out,self.hidden_cell = self.lstm(input_seq.view(len(input_seq),max_array_len, 1),self.hidden_cell)
        lstm_out,self.hidden_cell = self.lstm(input_seq.view(len(input_seq),max_array_len, 1))
        predictions = self.linear(lstm_out[:, -1,:])
        return predictions
model=LSTM()
optimizer = torch.optim.Adam(model.parameters(), lr=LR, weight_decay=0.8)   # optimize all cnn parameters
loss_func = nn.MSELoss()                       # the target label is not one-hotted
print(model)

EPOCHS=2000
for i in range(EPOCHS):
#     model.train()
    for step, (seq,labels) in enumerate(train_data):
        model.zero_grad()
        labels=labels.view(labels.shape[0],1)
        y_pred = model(seq)
        loss = loss_func(y_pred.float(), labels.float())

        loss.backward(retain_graph=True)
        optimizer.step()
    if i%10 == 0:
#         print(y_pred.shape,labels.shape)
        print(y_pred)
        print(f'epoch: {i:3} train_loss: {loss.item():10.8f}')

print('Finished Training')

y_pred i am gettting is:
tensor([[0.2661],
        [0.7536],
        [1.4659],
        [2.4905],
        [3.8662],
        [5.4478],
        [6.8958],
        [7.9347],
        [8.5493],
        [8.8773],
        [9.0486],
        [9.1409],
        [9.1931],
        [9.2244],
        [9.2441],
        [9.2570],
        [9.2657],
        [9.2718],
        [9.2761],
        [9.2792],
        [9.2815],
        [9.2831],
        [9.2843],
        [9.2853],
        [9.2860],
        [9.2865],
        [9.2869],
        [9.2872],
        [9.2874],
        [9.2876],
        [9.2877],
        [9.2878]], grad_fn=<AddmmBackward>)```


title: In pytorch, what is the use case of hooks on Tensors rather than on Modules?

body: In pytorch, we can add hooks to Modules, but also to Tensors. I can see the use cases for adding hooks to Modules. But since any Tensor is generally created by an application of some Module, I don't see why the ability to add hooks to Tensors directly is useful, and every application I've seen so far registers hooks to Modules. Is there any use case where adding hooks directly to Tensors is necessary?


title: How to handle Variable Sized inputs at the end of CNN layers before sending to the Loss Function

body:
I am dealing with variable sized inputs to a CNN and I wanted to know how to feed it to a last FC layer to satisfy the requirement for the CrossEntropy Loss function. Even if taken care for one sample, the subsequent sample would have different dimensions and can't be used in backpropagation. So I wanted to know a way or different ways this can be handled.
(P.S : Cropping the input to make it fixed size is being currently used and the query is for improvising)


title: pytorch KAIR example on Android

body:
I stuck trying to trace/scipt ffdnet KAIR's model to android. Model's forward looks like:
def forward(self, x): #, paddingBottom, paddingRight): #, sigma):

    noise_level_model = 15
    sigma = torch.full((1, 1, 1, 1), noise_level_model / 255.).type_as(x)

    h, w = x.size()[-2:]
    paddingBottom = int(np.ceil(h/2)*2-h)
    paddingRight = int(np.ceil(w/2)*2-w)

    x = torch.nn.ReplicationPad2d((0, paddingRight, 0, paddingBottom))(x)

    x = self.m_down(x)
    # m = torch.ones(sigma.size()[0], sigma.size()[1], x.size()[-2], x.size()[-1]).type_as(x).mul(sigma)
    m = sigma.repeat(1, 1, x.size()[-2], x.size()[-1])
    x = torch.cat((x, m), 1)
    x = self.model(x)
    x = self.m_up(x)

    x = x[..., :h, :w]
    return x

If I trace that I get some warnings about padding arguments but model works on Android. Problem is that it isn't work with input of different sizes, only size same as 'test1.jpeg':
model_name = 'ffdnet_color'
model_pool = 'model_zoo'
model_path = os.path.join(model_pool, model_name + '.pth')
n_channels = 3
nc = 96
nb = 12

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = net(in_nc=n_channels, out_nc=n_channels, nc=nc, nb=nb, act_mode='R')
model.load_state_dict(torch.load(model_path), strict=True)
model.eval()
for k, v in model.named_parameters():
    v.requires_grad = False
model = model.to(device)

img = 'testsets/myset/test1.jpeg'
img_name, ext = os.path.splitext(os.path.basename(img))
img_L = util.imread_uint(img, n_channels=n_channels)
img_L = util.uint2single(img_L)
noise_level_model = 15
img_L = util.single2tensor4(img_L)
img_L = img_L.to(device)
sigma_ = torch.full((1, 1, 1, 1), noise_level_model / 255)
sigma = torch.full((1, 1, 1, 1), noise_level_model / 255.).type_as(img_L)

traced_model = torch.jit.trace(model, img_L)
traced_optimized = optimize_for_mobile(traced_model)

save_path = os.path.splitext(os.path.basename(model_path))[0] + '-mobile.pth'
traced_optimized.save(save_path)

I've tried to script model with traced_model = torch.jit.script(model) but got errors:
TypeError: cannot create weak reference to 'numpy.ufunc' object

What should I do to achieve model works with different input size on mobile?


title: Compute accuracy of TIMIT data set on an RNN model

body:
So I am trying to figure out how to compute the accuracy of an RNN model.
I have a batch size of 192, and every frame (TIMIT frame) is of the size of 490*129.
currently, I compute my accuracy as follows:
    for batch_x, batch_y, len_batch in val_loader:
    batch_x, batch_y, len_batch = batch_x.to(device), batch_y.to(device), len_batch.to(device)
    logits = model(batch_x)
    loss_val = model.loss(logits, batch_y, len_batch)

    preds1 = logits.argmax(dim=-1)
    temp1 = batch_y.argmax(dim=-1)
    acc1 = preds1.squeeze().eq(temp1).sum().item()
    acc_rate1 = 100 * acc1 / (192 * 490)

But the accuracy that I receive seems wrong.
My train:
model.train()
print('number of epochs: {}'.format(args.epochs))

best_test = 1e7
best_validation = 1e7

losses_diagonal = []

for ep in range(1, 2):
    init_time = datetime.now()
    processed = 0
    step = 1

    for batch_idx, (batch_x, batch_y, len_batch) in enumerate(train_loader):
        batch_x, batch_y, len_batch = batch_x.to(device), batch_y.to(device), len_batch.to(device)

        opt.zero_grad()

        logits = model(batch_x)
        loss = model.loss(logits, batch_y, len_batch)
        loss.backward()

        if args.clip > 0:
            nn.utils.clip_grad_norm_(model.parameters(), args.clip)

        opt.step()

        processed += len(batch_x)
        step += 1
        losses_diagonal.append(loss)
        print("   batch_idx {}\tLoss: {:.2f} ".format(batch_idx, loss))

    print("Epoch {}, LR {:.5f} \tLoss: {:.2f} ".format(ep, opt.param_groups[0]['lr'], loss))

My test:
model.eval()
with torch.no_grad():

    for batch_x, batch_y, len_batch in test_loader:
        batch_x, batch_y, len_batch = batch_x.to(device), batch_y.to(device), len_batch.to(device)
        logits = model(batch_x)
        loss_test = model.loss(logits, batch_y, len_batch)

    for batch_x, batch_y, len_batch in val_loader:
        batch_x, batch_y, len_batch = batch_x.to(device), batch_y.to(device), len_batch.to(device)

        logits = model(batch_x)
        loss_val = model.loss(logits, batch_y, len_batch)

    for batch_x, batch_y, len_batch in val_loader:
        batch_x, batch_y, len_batch = batch_x.to(device), batch_y.to(device), len_batch.to(device)
        logits = model(batch_x)
        loss_val = model.loss(logits, batch_y, len_batch)

        preds1 = logits.argmax(dim=-1)
        temp1 = batch_y.argmax(dim=-1)

        acc1 = preds1.squeeze().eq(temp1).sum().item()
        acc_rate1 = 100 * acc1 / (192 * 490)
        acc_rate1
        print('acc = {}, acc_rate = {}'.format(acc1,acc_rate1))

    if loss_val < best_validation:
        best_validation = loss_val.item()
        best_test = loss_test.item()

print()
print("Val:  Loss: {:.2f}\tBest: {:.2f}".format(loss_val, best_validation))
print("Test: Loss: {:.2f}\tBest: {:.2f}".format(loss_test, best_test))
print()

model.train()

I would like to get some help with computing the accuracy of my RNN model, and I am willing to publish some more of my code if it will be required.


title: Adding the tensor values based on indices

body:
I have a tensor of values and a corresponding tensor of indices. I want to sum all the values where the index is not a special token (tokenizers).
For instance:

values=tensor([1.0000, 0.1574, 0.1507, 0.2520, 0.2456, 0.2365, 0.2330, 0.2294, 0.2321,
         0.2339], grad_fn=<MaxBackward0>),
indices=tensor([32099,    12,    24,     3,     3,     3,     3,     3,     3,     3])

I want sum(0.1574, 0.1507)
What is the optimal way to achieve this?


title: Training with parametric partial derivatives in pytorch

body:
Given a neural network with weights theta and inputs x, I am interested in calculating the partial derivatives of the neural network's output w.r.t. x, so that I can use the result when training the weights theta using a loss depending both on the output and the partial derivatives of the output. I figured out how to calculate the partial derivatives following this post. I also found this post that explains how to use sympy to achieve something similar, however, adapting it to a neural network context within pytorch seems like a huge amount of work and a recipee for very slow code.
Thus, I tried something different, which failed. As a minimal example, I created a function (substituting my neural network)
theta = torch.ones([3], requires_grad=True, dtype=torch.float32)
def trainable_function(time):
    return theta[0]*time**3 + theta[1]*time**2 + theta[2]*time

Then, I defined a second function to give me partial derivatives:
def trainable_derivative(time):
    deriv_time = torch.tensor(time, requires_grad=True)
    fun_value = trainable_function(deriv_time)
    gradient = torch.autograd.grad(fun_value, deriv_time, create_graph=True, retain_graph=True)
    deriv_time.requires_grad = False
    return gradient

Given some noisy observations of the derivatives, I now try to train theta. For simplicity, I create a loss that only depends on the derivatives. In this minimal example, the derivatives are used directly as observations, not as regularization, to avoid complicated loss functions that are besides the point.
def objective(train_times, observations):
    predictions = torch.squeeze(torch.tensor([trainable_derivative(a) for a in train_times]))
    return torch.sum((predictions - observations)**2)

optimizer = Adam([theta], lr=0.1)
for iteration in range(200):
    optimizer.zero_grad()
    loss = objective(data_times, noisy_targets)
    loss.backward()
    optimizer.step()

Unfortunately, when running this code, I get the error
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

I suppose that when calculating the partial derivatives in the way I do, I do not really create a computational graph through which autodiff could differentiate through. Thus, the connection to the parameters theta somehow gets lost and now it looks to the optimizer as if the loss is completely independent of the parameters theta. However, I could be totally wrong..
Does anyone know how to fix this?
Is it possible to include this type of derivatives in the loss function in pytorch?
And if so, what would be the most pytorch-style way of doing this?
Many thanks for your help and advise, it is much appreciated.
For completeness:
To run the above code, some training data needs to be generated. I used the following code, which works perfectly and has been tested against the analytical derivatives:
true_a = 1
true_b = 1
true_c = 1

def true_function(time):
    return true_a*time**3 + true_b*time**2 + true_c*time

def true_derivative(time):
    deriv_time = torch.tensor(time, requires_grad=True)
    fun_value = true_function(deriv_time)
    return torch.autograd.grad(fun_value, deriv_time)

data_times = torch.linspace(0, 1, 500)
true_targets = torch.squeeze(torch.tensor([true_derivative(a) for a in data_times]))
noisy_targets = torch.tensor(true_targets) + torch.randn_like(true_targets)*0.1


title: Pycharm can't import torch when I run the import command via script, but can import and use in console

body: As the title states, I am able to successfully import torch via the python console in PyCharm, but when I write a scratch script with just the import command, I get a series of errors. I believe this is likely associated with some sort of issue w/ my selected script path, but I'm not sure which path to choose.


title: Input dimension of MultiheadAttention varies between Pytorch and Tensorflow "Transformer Model for language understanding"

body:
In Pytorch's implementation of MultiheadAttention, TransformerEncoder, the Batch size is in the middle. That is (L,N,E) for query dimension.
But in Tensorflow's  "Transformer Model for language understanding" article, the query dimension is (batch_size,seq_len,depth). That batch_size is at 0th index.
Why this differs?


title: PyTorch Tensors - vectorized slicing with given list of end indices

body:
Suppose I have a 1D PyTorch tensor end_index of length L.
I want to construct a 2D PyTorch tensor T with L lines where
T[i,j] = 2 when j < end_index[i] and T[i,j] = 1 otherwise.
The following works:
T = torch.ones([4,3], dtype=torch.long)
for element in end_index:
    T[:, :element] = 2

Is there a vectorizd way to do this?


title: Convert PyTorch GPU tensor to Tensorflow 1.x tensor without leaving the GPU

body:
I have PyTorch tensors in the GPU, which I would like to convert to Tensorflow 1.x tensors without ever leaving the GPU.
I don't care about passing gradients back and forth. I just want the tensor to stay in the GPU. I don't care if my PyTorch tensor is copied to a TF tensor or modified in place.
I am aware that I can convert to numpy on the CPU and reload onto the GPU. However, I must stay on GPU for speed.


title: How to vectorize loss for a LSTM doing sequential Language modelling

body:
So I have an assignment involving Language Modelling and I passed all the unit tests but my code is too slow to run. I think it's because of the way I compute my loss. The formula we're given is the following:

My naive implementation is the following:
losses_batch_list = []
batch_size = log_probas.size(0)
for b in range(batch_size):

    seq_length = max([i for i, e in enumerate(mask[b,:]) if e != 0]) + 1

    loss_batch = 0
    for t in range(seq_length):
        for n in range(self.vocabulary_size):
            if targets[b, t] == n:
                loss_batch += log_probas[b, t, n].detach()

    loss_batch = - loss_batch / seq_length
    losses_batch_list.append(loss_batch)

loss = torch.tensor(np.mean(losses_batch_list))

return loss

But that loop runs for ever since the vocabulary size is the same approximately as GPT1 (~40 000) and the sequence length is up to 255 (something it is shorter because of padding, hence the mask).
Does anyone have any tips on how to vectorize/speed this up?
I know it's correct but I can't report any results with it...
Thanks!


title: How to shuffle a folder of images that have been converted to tensors

body:
My code:
data_loader = torch.utils.data.DataLoader('/content/gdrive/MyDrive/dataset/test', batch_size = 8, shuffle=True)

The folder above consists of 2 subfolders with images, which have been transformed into tensors. However I can't get them to be shuffled, I'm trying to shuffle all images in both folders. How do I get this to work? Running this line doesn't even shuffle the data within each subfolder.


title: torch: custom dataloader with explicit multiprocessing

body:
I have a custom dataloader. I have explicitly used python's multiprocessing to parallelize data preprocessing in my custom dataloader. I am using 8 workers(num_threads) in multiprocessing in my dataLoader.
I wanted to know, how will that affect my torch.utils.data.DataLoader call?
Will the num_workers argument be set to 8? Or can I leave it at 0?
My custom Loader looks like this
def processParams(params):
        <some operations on params>
        return params

def processParamsParallel(params, pool):
      results = pool.map(processParams, params)
      return results

class DataLoader(object):
      def __init__(self, params, maxId):
             self.params = params
             self.id = 0
             self.maxId = maxId
             self.pool = Pool(processes=8)
      def __iter__(self):
             while self.id<=self.maxId:
                 if self.id==self.maxId:
                       self.id = 0
                 results = processParamsParallel(self.params, self.pool)
                 self.id+=1
                 yield results

It's a very rough example of what I am trying to do.
Now in the torch call
dl = DataLoader(params, 50)
dl_torch = torch.utils.data.Dataloader(dl, num_workers = <what_here?>, prefetch_factor = <what_here?>)

On a similar note, how will prefetch_factor be affected given that the num_workers are not set by torch call but by the custom dataLoader itself?
From torch.utils.data
prefetch_factor - Number of samples loaded in advance by each worker. 2 means there will be a total of 2 * num_workers samples prefetched across all workers. (default: 2 )
Thank you in Advance!


title: BERT: Weights of input embeddings as part of the Masked Language Model

body:
I looked through different implementations of BERT's Masked Language Model.
For pre-training there are two common versions:

Decoder would simply take the final embedding of the [MASK]ed token and pass it throught a linear layer (without any modifications):

    class LMPrediction(nn.Module):
        def __init__(self, hidden_size, vocab_size):
            super().__init__()
            self.decoder = nn.Linear(hidden_size, vocab_size, bias = False)
            self.bias = nn.Parameter(torch.zeros(vocab_size))
            self.decoder.bias = self.bias
        def forward(self, x):
             return self.decoder(x)

Some implementations would use the weights of the input embeddings as weights of the decoder-linear-layer:

    class LMPrediction(nn.Module):
        def __init__(self, hidden_size, vocab_size, embeddings):
            super().__init__()
            self.decoder = nn.Linear(hidden_size, vocab_size, bias = False)
            self.bias = nn.Parameter(torch.zeros(vocab_size))
            self.decoder.weight = embeddings.weight ## <- THIS LINE
            self.decoder.bias = self.bias
        def forward(self, x):
             return self.decoder(x)

Which one is correct? Mostly, I see the first implementation. However, the second one makes sense as well - but I cannot find it mentioned in any papers (I would like to see if the second version is somehow superior to the first one)


title: Using a Pytorch Transformer to model a OR gate

body:
When I'm learning something new from scratch with ML I like to model an OR gate and an XOR gate before I dive in (even when it doesn't really make sense).
I'm learning pytorch and transformer networks currently.
I know that transformers are of course a bit stupid to do such a trivial task, however I like to adapt it and find where I'm going wrong.
I'm using pytorchs nn.Transformer network, I also have made "src" and "target" tensors as such (along with a "y" for predictions:
src = [[0,0], [1,0], [0,1], [1,1]]
tgt= [[0,0], [1,1], [1,1], [1,1]]
y = [0,1,1,1]

The code runs as intended about 20% of the time, however the majority of the time it predicts y = [1,1,1,1].
Can anyone see (besides the obvious "transformers for an OR gate") what might be missing for a consistent result? Code is below
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import torch.optim as optim

X = [[0,0], [1,0], [0,1], [1,1]]
Y = [[0,0], [1,1], [1,1], [1,1]]
Y_true = [0, 1, 1, 1]
num_samples = 4
dimensions = 2
X = torch.tensor(X, dtype=torch.float)
Y = torch.tensor(Y, dtype=torch.float)
Y_true = torch.tensor(Y_true, dtype=torch.float)
X = torch.reshape(X, (1,4,2))
Y = torch.reshape(Y, (1,4,2))

class MyTransformerNetwork(nn.Module):
    def __init__(self):
        super().__init__()

        self.transformer_model = nn.Transformer(d_model = dimensions, nhead = 1, dropout=0)
        self.linear = nn.Linear(dimensions, 1)
        self.sigmoid = nn.Sigmoid()

    def forward(self, src, tgt):
        src = self.transformer_model(src,tgt)
        src = self.sigmoid(src)
        src = self.linear(src)
        src = self.sigmoid(src)

        return src

model = MyTransformerNetwork()
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

epochs = 1000

for e in range(epochs):
    optimizer.zero_grad()
    output = model(X, Y)
    output = torch.squeeze(output)
    rounded_output = torch.round(output)
    print(f"Predicted: {rounded_output}")
    print(f"Expected: {Y_true}")
    loss = criterion(output, Y_true)
    loss.backward()
    optimizer.step()

    print(f"Training loss: {loss.item()}")


title: How to load and test a model in pytorch?

body:
I am loading an already trained dataset. My dataset is a 3D image and i want to test that model now. So I created a different function but is showing an error at. i tried to save using state_dict() first but it showed an error, so I saved the whole model.
My = MyNet(data)
My = torch.load("model_par.pt")

The error is:
File "main_SUDF.py", line 238, in <module>
My = MyNet(data)
File "main_SUDF.py", line 148, in __init__
self.conv1 = nn.Conv2d(input_dim, nChannel, kernel_size=3, stride=1, padding=1)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/conv.py", line 410, in __init__
super(Conv2d, self).__init__(
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/conv.py", line 50, in __init__
if in_channels % groups != 0:
RuntimeError: Boolean value of Tensor with more than one value is ambiguous

and my model is:
class MyNet(nn.Module):
    def __init__(self, input_dim):
        super(MyNet, self).__init__()
        self.conv1 = nn.Conv2d(input_dim, nChannel, kernel_size=3, stride=1, padding=1)
        self.bn1 = nn.BatchNorm2d(nChannel)
        self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv2 = []
        self.bn2 = []
        self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2)
        for i in range(nConv - 1):
            self.conv2.append(nn.Conv2d(nChannel, nChannel, kernel_size=3, stride=1, padding=1))
            self.bn2.append(nn.BatchNorm2d(nChannel))

        self.conv3 = nn.Conv2d(nChannel, nChannel, kernel_size=1, stride=1, padding=0)
        self.bn3 = nn.BatchNorm2d(nChannel)
        self.UB1 = nn.UpsamplingBilinear2d(scale_factor=2)
        self.deconv = nn.ConvTranspose2d(nChannel, nChannel, kernel_size=3, stride=1, padding=1)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.bn1(x)
        x = self.pool1(x)
        for i in range(nConv - 1):
            x = self.conv2[i](x)
            x = F.relu(x)
            x = self.bn2[i](x)
            if i == 0:
                x = self.pool2(x)
        x = self.conv3(x)
        # x = F.relu(x)
        x = self.bn3(x)

        x = self.UB1(x)
        x = self.deconv(x)
        x = F.relu(x)
        x = self.bn3(x)

        x = self.UB1(x)
        x = self.deconv(x)
        x = F.relu(x)
        x = self.bn3(x)

        return x


title: AllenNLP Multi-Task Model: Keep encoder weights for new heads

body:
I have trained a (AllenNLP) multi-task model. I would like to keep the encoder/backbone weights and continue training with new heads on new datasets. How can I do that with AllenNLP?
I have two basic ideas for how to do that:

I followed this AllenNLP tutorial to load the trained model and then instead of just making predictions I wanted to change the configuration and the model-heads to continue training on the new datasets...but I am kinda lost in how to do that.

I guess it should be possible to (a) save the state-dict of the previously trained encoder in a file and then (b) point to those weights in the configuration file for the new model (instead of pointing to "bert-base-cased"-weights for example). But looking at the PretrainedTransformerEmbedder-class I don't see how I could pass my own model-weights to that class.

As an additional question: Is it also possible to save the weights of the heads separately and initialize new heads with those weights?
Any help is appreciated :)


title: Filtering a torch dataset in R

body:
I'm trying to follow along with the book "Deep Learning with PyTorch". I am using the new R packages torch and torchvision.
On page 173, section 7.2.1 I'm just not sure how to filter this dataset to include only labels 1 and 3 (corresponding to the 0 and 2 in the book).
This is my code, and I'd like to know how to filter transformed_cifar10 as per the code in the book. Meaning filter it so that the transformed_cifar10$y labels only include 1 and 3. and then remap {1,3} to {1,2}.
library(dplyr)
library(torch)
library(torchvision)

data_path <- "./ch7/data" # need to change this?

train_transforms <- function (img) {
  img %>%
    transform_to_tensor() %>%
    transform_normalize(mean = c(0.4915, 0.4823, 0.4468),
                        std = c(0.2470, 0.2435, 0.2616))
}

transformed_cifar10 <- cifar10_dataset(data_path,
                                       train = TRUE,
                                       download = TRUE,
                                       transform = train_transforms)

This is the python code in the book:
# In[5]:
label_map = {0: 0, 2: 1}
class_names = ['airplane', 'bird']
cifar2 = [(img, label_map[label])
          for img, label in cifar10
          if label in [0, 2]]

First I thought of trying something like this but clearly it doesn't work...
tensor_cifar10[tensor_cifar10$y == 1]


title: Confused on using dropout in batch gradient descent with Q-learning

body:
I am using PyTorch and adding dropout layers to my inner layers.
class MLP(nn.Module):
  #def __init__(self, n_inputs, n_action, n_hidden_layers=2, hidden_dim=8, drop=0.25):
  def __init__(self, console_file, n_inputs, n_action, layers_list, drop=0.25):
    super(MLP, self).__init__()
    print("Layers structure:")
    console_file.write("Layers structure:\n")
    print(f"inputs: {n_inputs}")
    console_file.write(f"inputs: {n_inputs}\n")
    self.layers = []
    for i, layer_size in enumerate(layers_list):
      if i == 0:
        layer = nn.Linear(n_inputs, layer_size)
      else:
        layer = nn.Linear(layers_list[i-1], layer_size)
      self.layers.append(layer)
      print(f"layer {i}: {layer_size}")
      console_file.write(f"layer {i}: {layer_size}\n")
      self.layers.append(nn.LeakyReLU(0.1))
      if drop > 0.01:
        #self.layers.append(nn.Dropout(p = drop**(len(layers_list)-i)))
        self.layers.append(nn.Dropout(p = drop))
        #print(f"drop {i}: {drop**(len(layers_list)-i)}")
        print(f"drop {i}: {drop}")
        #console_file.write(f"drop {i}: {drop**(len(layers_list)-i)}\n")
        console_file.write(f"drop {i}: {drop}\n")
    # final layer
    self.layers.append(nn.Linear(layers_list[-1], n_action))
    self.layers = nn.Sequential(*self.layers)
    print(f"outputs: {n_action}")
    console_file.write(f"outputs: {n_action}\n")
    print("========= NN structure =========\n")
    console_file.write("========= NN structure =========\n\n")
  def forward(self, X):
    return self.layers(X)

  def save_weights(self, path):
    torch.save(self.state_dict(), path)

  def load_weights(self, path):
    self.load_state_dict(torch.load(path))

I am making sure to turn training on during training and eval mode outside training (removing the droput layers).
    self.model = he.MLP(console_file, state_size, self.action_size, DIMENSION, DROPOUT)
    if DROPOUT > 0.01:
      self.model.train()
...
      if DROPOUT > 0.01: # before testing
        agent.model.eval()
...
    if DROPOUT > 0.01: # after testing
      self.model.train()

My confusion is as to HOW if at all PyTorch keeps track of which neurons it disables between the forward propagation and the later time where a random batch of output-and-reward is selected and backpropagation is performed on a specific case originating in a forward propagation with specific neurons disabled.
def train_one_step(model, criterion, optimizer, inputs, targets):
  # convert to tensors
  inputs = torch.from_numpy(inputs.astype(np.float32))
  targets = torch.from_numpy(targets.astype(np.float32))

  # zero the parameter gradients
  optimizer.zero_grad()

  # Forward pass
  outputs = model(inputs)
  loss = criterion(outputs, targets)

  # Backward and optimize
  loss.backward()
  optimizer.step()

inputs and targets are from random selection (batch over buffered history)
For me it makes sense to have the SAME neurons disabled between the forward and back propagation but since I can find nothing on the subject everything would suggest that the dropout will apply randomly and therefor nonsensically to me.
Either the forward and back propagations need to happen identically (and somehow PyTorch manages to remember by some markers, because I don't seem to pass any markers during batching) or I need to understand why they can be randomly different.


title: How to extract the layers from the source code in `forward()` function of a PyTorch model?

body:
I want to extract the layers of the model that process the input data as defined in the forward function. For example, given the following source code:
 def forward(self, inputs):
     inputs = self.embedding(inputs)
     inputs = F.dropout(inputs, 0.25, self.training)
     return inputs

I want to extract the layers that process the input data i.e:
input -> embedding -> dropout -> output

How do I do this without runing the code?


title: Why isn't my newton's method running slow in pytorch?

body:
I am using Newton's Method as a optimizer for MNIST classifer. But its been over an hour and it has only gone through one epoch. Also I am trying to see how many iterations it takes to get 90% accuracy hence I set the while condition as such. Am I doing something wrong here or is Newton's method just slow. For reference SGD converged in 13 seconds.
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import torchvision
from keras.datasets import mnist
from keras.utils.np_utils import to_categorical
import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score, accuracy_score, confusion_matrix
import time
from torch.utils.data import DataLoader

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
#device = torch.device("cpu")
resnet18 = torchvision.models.resnet18();
resnet18.conv1 = nn.Conv2d(1,resnet18.conv1.out_channels,(7,7))
resnet18.fc = nn.Sequential(
   nn.Dropout(0.5),
   nn.Linear(resnet18.fc.in_features, 10),
);
resnet18 = resnet18.to(device);
model = resnet18;

loss = torch.nn.CrossEntropyLoss();
optimizer = torch.optim.LBFGS(model.parameters(),lr = 0.01);
loss = loss.to(device)

from torchvision.datasets import MNIST
from torchvision.transforms import ToTensor
train_ds = MNIST("mnist", train=True, download=True, transform=ToTensor())
train_dl = DataLoader(train_ds, batch_size=256, shuffle=True)

test_ds = MNIST("mnist", train=False, download=True, transform=ToTensor())
test_dl = DataLoader(test_ds, batch_size=10000)

start_time = time.time()
acc = 0
epochs = 0
while acc < 0.90:
   model.train()
   for data in iter(train_dl):
       x, y = data
       x = x.to(device)
       y = y.to(device)
       def closure():
       #prediction = model(X_train[idx].to(device))
         prediction = model(x);
       #train_loss = loss(prediction.unsqueeze(0), y_train[idx].unsqueeze(0).to(device))

         train_loss = loss(prediction, y);

         optimizer.zero_grad()
         train_loss.backward()
         return train_loss
       optimizer.step(closure)
   epochs += 1;

   model.eval()
   data = iter(test_dl).next()
   x, y = data;
   x = x.to(device)
   y = y.to(device)
   prediction = model(x)
   # print(y.shape)
   # print(torch.max(prediction,1))
   acc = torch.sum(y == torch.max(prediction,1)[1])/len(y)
   print("epoch ", epochs, "accuracy ", acc);

print ("---training time: %s seconds ---" % (time.time() - start_time))
print ("number of epochs is ", epochs)


title: How to clamp output of nueron in pytorch

body:
I am using simple nn linear model(20,64,64,2) for deep reinforcement learning. This model I am using to approximate the policy gradients by the PPO algorithm. Hence the output layer gives 2 values, which are mean and standard deviation. These parameters are further used in the environment to sample more data. For the environment, the parameter values should be between threshold. mean = [max, min] and std = [max, min].
While training, after some iteration, the parameter values of the output layer increases suddenly and because of that the environment fails to sample more data. The environment uses the output from the model and I can not change in environment to make it stable. Hence, is there any way to restrict the parameter value for a threshold or to clamp. (The loss is calculated from the data sampled by environment and then the back propagation)
The sample code for the model you can find below:
    def forward(self, x):
        # feed forwards to layers
        x = F.relu(self.linear_0(x))
        x = F.relu(self.linear_1(x))
        return self.linear_2(x)

I have tried the following structure...
    def forward(self, x):
        # feed forwards to layers
        x = F.relu(self.linear_0(x))
        x = F.relu(self.linear_1(x))
        x = self.linear_2(x)
        x[0] = torch.clamp(x[0], min=min, max=max)
        x[1] = torch.clamp(x[1], min=min, max=max)
        return x

but it gives an error while back propagating:

RuntimeError: one of the variables needed for gradient computation has
been modified by an inplace operation: [torch.DoubleTensor [499, 2]],
which is output 0 of SelectBackward, is at version 3; expected version
2 instead. Hint: enable anomaly detection to find the operation that
failed to compute its gradient, with
torch.autograd.set_detect_anomaly(True).

Thanks in advance!


title: convert Dataframe or list to Tensor using PyTorch

body: I have 25GB .pcap files and want to send the packet bytes to CNN but converting the .pcaps file to CSV makes a large file. I know how to read packets from .pcap files and make a Dataframe or list, is there any way to convert this to PyTorch format? and make train and Test data?


title: are there ways to improve inference results on a pretrained machine learning model?

body:
I am trying to get inference from a pre-trained machine learning object detection model(detr) on to jrdb dataset(https://jrdb.stanford.edu/). I am using the demo code provided by the team that authored the paper. Here is the link to the code: detr_demo
I have changed the model to resnet-101 and the weights file to the best they have reported on the GitHub repo(https://github.com/facebookresearch/detr) for the code. yet, the Average precision score I am getting on the jrdb test dataset is 15.8 AP. while, the best result on the same code using the pre-trained model and weights for another team is 48 AP, as reported on the jrdb leaderboard for 2d detection(https://jrdb.stanford.edu/leaderboards/results/412). What I wish to know is, are there ways to improve inference, without retraining? I have searched on google and most links show to better train the model.


title: os.path.isfile(Path/to/image) gives False while os.path.isdir(The/directory image is in is true)

body:
...
from google.colab import drive
drive.mount('/content/drive', force_remount=True)

# Commented out IPython magic to ensure Python compatibility.
# %cd /content

!ls

!cat ttt.txt

!ls

Path2 = "/content/drive/MyDrive/foop/train/"
Path1 = "/content/drive/MyDrive/food-101/images/"

#print(os.path.isdir(Path2))
for i in range(len(L)):
  #path1 = str(Path1) + str(L[i]) + str('/') #+ str(Train[i][1])
  path2 = str(Path2) + str(L[i]) + str('/') #+ str(Train[i][1])
  print(path2)
  print(os.path.isdir(path2))
  if not os.path.exists(path2):
    os.makedirs(path2)

  print(os.path.exists(path2))

'''
for i in range(len(Train)%100):
  path1 = str(Path1) + str(Train[i][0]) + str('/') #+ str(Train[i][1])
  path2 = str(Path2) + str(Train[i][0]) + str('/') #+ str(Train[i][1])
  path1 = str(path1)
  path2 = str(path2)
  #print(os.path.isdir(path1))
  #print(os.path.isdir(path2))
  if not (os.path.isdir(path2)):
    #pass
    #!mkdir path2
  #print(os.path.isdir(path1))
  print(os.path.isdir(path2))
  #!cp path1 path2
  #shutil.copy(path1, path2)
'''

for i in range(len(Train)):
  path1 = str(Path1) + str(Train[i][0]) + str('/') + str(Train[i][1])
  path2 = str(Path2) + str(Train[i][0]) + str('/') + str(Train[i][1])
  path1 = str(path1)
  path2 = str(path2)
  #print(os.path.isdir(path1))
  #print(os.path.isdir(path2))
  #print(os.path.isdir(path2))
    #pass
    #!mkdir path2
  #print(str(Train[i][1]))
  #print(os.path.isdir(path1))
  if (str(Train[i][0])=="baklava"):
    #print(str(Train[i][1]))
    **print(os.path.isfile(Path1+"/"+"baklava"+"/"+"1117620.jpg"))**
  #!cp path1 path2
  #shutil.copy(path1, path2)

...
The error is that :
os.path.isdir(Path1+"/"+"baklava"+"/") gives True
But-
os.path.isfile(Path1+"/"+"baklava"+"/"+"1117620.jpg") gives False
I have also tried this as suggested in another answer :
...
P2 = os.path.join('/content', 'drive', 'MyDrive', 'foop', 'train')

P1 = os.path.join('/content', 'drive', 'MyDrive', 'food-101', 'images')
for i in range(len(Train)):
  #path1 = str(Path1) + str(Train[i][0]) + str('/') + str(Train[i][1])
  #path2 = str(Path2) + str(Train[i][0]) + str('/') + str(Train[i][1])
  #path1 = str(path1)
  #path2 = str(path2)
  #print(path1)
  #print(os.path.isfile(path1))
  #print(os.path.isdir(path2))
  #print(os.path.isdir(path2))
    #pass
    #!mkdir path2
  #print(str(Train[i][1]))
  #print(os.path.isdir(path1))
    #print(str(Train[i][1]))

  #!cp path1 path2
  #shutil.copy(path1, path2)
  P = os.path.join(P1, str(Train[i][0]), str(str(Train[i][1] + '.jpg')))
  print(os.path.isdir(P1))
  print(P)
  print(os.path.isfile(P))
...

Output -
True
/content/drive/MyDrive/food-101/images/waffles/888311.jpg
False
True
/content/drive/MyDrive/food-101/images/waffles/903293.jpg
False
True
/content/drive/MyDrive/food-101/images/waffles/923472.jpg
False
True
/content/drive/MyDrive/food-101/images/waffles/940186.jpg
False
True
/content/drive/MyDrive/food-101/images/waffles/948929.jpg
False
True
/content/drive/MyDrive/food-101/images/waffles/957965.jpg
False
True
/content/drive/MyDrive/food-101/images/waffles/97524.jpg
False
True
/content/drive/MyDrive/food-101/images/waffles/98131.jpg
False
True
/content/drive/MyDrive/food-101/images/waffles/982668.jpg
False
Also for this
...
P2 = os.path.join('/content', 'drive', 'MyDrive', 'foop', 'train')
P1 = os.path.join('/content', 'drive', 'MyDrive', 'food-101', 'images')
for i in range(len(Train)):
  #path1 = str(Path1) + str(Train[i][0]) + str('/') + str(Train[i][1])
  #path2 = str(Path2) + str(Train[i][0]) + str('/') + str(Train[i][1])
  #path1 = str(path1)
  #path2 = str(path2)
  #print(path1)
  #print(os.path.isfile(path1))
  #print(os.path.isdir(path2))
  #print(os.path.isdir(path2))
    #pass
    #!mkdir path2
  #print(str(Train[i][1]))
  #print(os.path.isdir(path1))
    #print(str(Train[i][1]))

  #!cp path1 path2
  #shutil.copy(path1, path2)
  P = os.path.join(P1, str(Train[i][0]), str(str(Train[i][1] + '.jpg')))
  PP1 = os.path.join(P1, str(Train[i][0]))
  PP2 = os.path.join(P2, str(Train[i][0]))
  print(os.path.isdir(P1))
  print(os.path.isdir(PP1))
  print(os.path.isdir(PP2))
  print(P)
  print(os.path.isfile(P))

...
Output :
True for isdir() false for isfile()
'''
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/1002167.jpg
False
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/1012802.jpg
False
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/102284.jpg
False
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/1034768.jpg
False
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/1045631.jpg
False
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/1068834.jpg
False
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/1096319.jpg
False
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/1113665.jpg
False
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/1126334.jpg
False
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/1140791.jpg
False
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/114959.jpg
False
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/1169507.jpg
False
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/1184113.jpg
False
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/119195.jpg
False
True
True
True
/content/drive/MyDrive/food-101/images/takoyaki/1216488.jpg
False

'''


title: The .data attribute for a dataset in torchvision.datasets doesn't work for ImageFolder?

body:
I can access the training data set of an MNIST object like so:
from torchvision.datasets import MNIST
trainset = MNIST(root='.', train=True, download=True)
print(trainset.data)

However, I'm not able to do the same using ImageFolder. Anybody know the equivalent for ImageFolder?
AttributeError: 'ImageFolder' object has no attribute 'data'


title: pytorch model is giving no output but a weird error message

body:
I am currently getting a really really weird error in pytorch.
When I start my script I first get a error message and then it seems like an endless loop is created.
I don't know what this error is caused by.
Error message when starting the program thru my terminal:
/home/anaconda3/envs/speech/lib/python3.7/site-packages/torch/functional.py:516: UserWarning: stft will require the return_complex parameter be explicitly  specified in a future PyTorch release. Use return_complex=False  to preserve the current behavior or return_complex=True to return  a complex output. (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370152014/work/aten/src/ATen/native/SpectralOps.cpp:653.)
  normalized, onesided, return_complex)
/home/anaconda3/envs/speech/lib/python3.7/site-packages/torch/functional.py:516: UserWarning: The function torch.rfft is deprecated and will be removed in a future PyTorch release. Use the new torch.fft module functions, instead, by importing torch.fft and calling torch.fft.fft or torch.fft.rfft. (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370152014/work/aten/src/ATen/native/SpectralOps.cpp:590.)
  normalized, onesided, return_complex)
[W NNPACK.cpp:80] Could not initialize NNPACK! Reason: Unsupported hardware.

Error message when starting my program thru visual studio code:
Traceback (most recent call last):
  File "/home/Dokumente/VoiceAssistant/engine.py", line 123, in <module>
    asr_engine = SpeechRecognitionEngine(args.model_file, args.ken_lm_file)
  File "/home/Dokumente/VoiceAssistant/engine.py", line 44, in __init__
    self.model = torch.jit.load(model_file)
  File "/home/anaconda3/envs/speech/lib/python3.7/site-packages/torch/jit/_serialization.py", line 151, in load
    raise ValueError("The provided filename {} does not exist".format(f))  # type: ignore
ValueError: The provided filename model.pt does not exist

That's the point where I don't know what to do anymore because two different errors are raised when starting the program in two different ways.
The other problem is that when I run the program thru my terminal I get the first error message(which I've posted at first) and after that the program or something else start an endless loop which is printing every second a new row I can only stop this loop by interrupting or closing the terminal.
Are those errors maybe connected or caused because of the same reason?
My Code:
import pyaudio
import threading
import time
import argparse
import wave
import torchaudio
import torch
import sys
import numpy as np
from neuralnet.dataset import get_featurizer
from decoder import DecodeGreedy, CTCBeamDecoder
from threading import Event

class Listener:

    def __init__(self, sample_rate=8000, record_seconds=2):
        self.chunk = 1024
        self.sample_rate = sample_rate
        self.record_seconds = record_seconds
        self.p = pyaudio.PyAudio()
        self.stream = self.p.open(format=pyaudio.paInt16,
                        channels=1,
                        rate=self.sample_rate,
                        input=True,
                        output=True,
                        frames_per_buffer=self.chunk)

    def listen(self, queue):
        while True:
            data = self.stream.read(self.chunk , exception_on_overflow=False)
            queue.append(data)
            time.sleep(0.01)

    def run(self, queue):
        thread = threading.Thread(target=self.listen, args=(queue,), daemon=True)
        thread.start()
        print("\Speech Recognition engine is now listening... \n")

class SpeechRecognitionEngine:

    def __init__(self, model_file, ken_lm_file, context_length=10):
        self.listener = Listener(sample_rate=8000)
        self.model = torch.jit.load(model_file)
        self.model.eval().to('cpu')  #run on cpu
        self.featurizer = get_featurizer(8000)
        self.audio_q = list()
        self.hidden = (torch.zeros(1, 1, 1024), torch.zeros(1, 1, 1024))
        self.beam_results = ""
        self.out_args = None
        self.beam_search = CTCBeamDecoder(beam_size=100, kenlm_path=ken_lm_file)
        self.context_length = context_length * 50 # multiply by 50 because each 50 from output frame is 1 second
        self.start = False

    def save(self, waveforms, fname="audio_temp"):
        wf = wave.open(fname, "wb")
        # set the channels
        wf.setnchannels(1)
        # set the sample format
        wf.setsampwidth(self.listener.p.get_sample_size(pyaudio.paInt16))
        # set the sample rate
        wf.setframerate(8000)
        # write the frames as bytes
        wf.writeframes(b"".join(waveforms))
        # close the file
        wf.close()
        return fname

    def predict(self, audio):
        with torch.no_grad():
            fname = self.save(audio)
            waveform, _ = torchaudio.load(fname)  # don't normalize on train
            log_mel = self.featurizer(waveform).unsqueeze(1)
            out, self.hidden = self.model(log_mel, self.hidden)
            out = torch.nn.functional.softmax(out, dim=2)
            out = out.transpose(0, 1)
            self.out_args = out if self.out_args is None else torch.cat((self.out_args, out), dim=1)
            results = self.beam_search(self.out_args)
            current_context_length = self.out_args.shape[1] / 50  # in seconds
            if self.out_args.shape[1] > self.context_length:
                self.out_args = None
            return results, current_context_length

    def inference_loop(self, action):
        while True:
            if len(self.audio_q) < 5:
                continue
            else:
                pred_q = self.audio_q.copy()
                self.audio_q.clear()
                action(self.predict(pred_q))
            time.sleep(0.05)

    def run(self, action):
        self.listener.run(self.audio_q)
        thread = threading.Thread(target=self.inference_loop,
                                    args=(action,), daemon=True)
        thread.start()

class DemoAction:

    def __init__(self):
        self.asr_results = ""
        self.current_beam = ""

    def __call__(self, x):
        results, current_context_length = x
        self.current_beam = results
        trascript = " ".join(self.asr_results.split() + results.split())
        print(trascript)
        if current_context_length > 10:
            self.asr_results = trascript

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="demoing the speech recognition engine in terminal.")
    parser.add_argument('--model_file', type=str, default='model.pt', required=False,
                        help='optimized file to load. use optimize_graph.py')
    parser.add_argument('--ken_lm_file', type=str, default=None, required=False,
                        help='If you have an ngram lm use to decode')

    args = parser.parse_args()
    asr_engine = SpeechRecognitionEngine(args.model_file, args.ken_lm_file)
    action = DemoAction()

    asr_engine.run(action)
    threading.Event().wait()

Am I doing something wrong with my code?
With that being said thanks for every suggestion or help in advance:)


title: What is the difference between `on_validation_epoch_end` and `validation_epoch_end` in Pytorch-lightning?

body:
Inside a LightningModule,Pycharm allows 2 auto complete methods:
class MyModel(LightningModule):

    def on_validation_epoch_end(self):

    def validation_epoch_end(self, outs):

with on_validation_epoch_end referenced in hooks.py
def on_validation_epoch_end(self) -> None:
    """
    Called in the validation loop at the very end of the epoch.
    """
    # do something when the epoch ends

and
validation_epoch_end called in evaluation_loop.py as eval_results = model.validation_epoch_end(eval_results) leading to __run_eval_epoch_end.

What is the purpose of each of those?
I can only assume one is deprecated. Could not find any relevant docs.


title: How can I reshape the (1006,19) result of keras regressor predictions into a (1006,1) numpy array?

body:
I'm trying to create a stock prediction model in botch PyTorch and Keras. I have already followed some tutorials online and modified to fit my data and it works fine.
Now I'm translating that code into a compatible Keras model. I've already created the model and did the predictions but the problem is that the regressor.predict() function from Keras returns a (1006,19) numpy array whereas when I do predictions = model(x_test) it returns a (1006,1) which is what I need for my following work so I can plot the results.
Here's my Keras code so far:
from keras.models import Sequential
from keras.layers import LSTM, Dense, Dropout

lookback = 20
x_train_keras, y_train_keras, x_test_keras, y_test_keras = split_data(price, lookback)
print('x_train.shape = ',x_train_keras.shape)  # x_train.shape =  (1006, 19, 1)
print('y_train.shape = ',y_train_keras.shape)  # y_train.shape =  (1006, 1)
print('x_test.shape = ',x_test_keras.shape)  # x_test.shape =  (252, 19, 1)
print('y_test.shape = ',y_test_keras.shape)  # y_test.shape =  (252, 1)

regression = Sequential()
regression.add(LSTM(units=50, return_sequences=True, kernel_initializer='glorot_uniform', input_shape=(x_train_keras.shape[1],1)))
regression.add(Dropout(0.2))

regression.add(LSTM(units=50,kernel_initializer='glorot_uniform',return_sequences=True))
regression.add(Dropout(0.2))

regression.add(LSTM(units=50,kernel_initializer='glorot_uniform',return_sequences=True))
regression.add(Dropout(0.2))

regression.add(LSTM(units=50,kernel_initializer='glorot_uniform',return_sequences=True))
regression.add(Dropout(0.2))

regression.add(Dense(units=1))

regression.compile(optimizer='adam', loss='mean_squared_error')

from keras.callbacks import History

history = History()
history = regression.fit(x_train_keras, y_train_keras, batch_size=30, epochs=100, callbacks=[history])

train_predict_keras = regression.predict(x_train_keras)
train_predict_keras = train_predict_keras.reshape((train_predict_keras.shape[0], train_predict_keras.shape[1]))

predict = pd.DataFrame(scaler.inverse_transform(train_predict_keras))
original = pd.DataFrame(scaler.inverse_transform(y_train_keras))

fig = plt.figure()
fig.subplots_adjust(hspace=0.2, wspace=0.2)

plt.subplot(1,2,1)
ax = sns.lineplot(x=original.index, y=original[0], label='Data', color='royalblue')
ax = sns.lineplot(x=predict.index, y=predict[0], label='Training Prediction', color='tomato')
ax.set_title('Stock Price', size=14, fontweight='bold')
ax.set_xlabel("Days", size = 14)
ax.set_ylabel("Cost (USD)", size = 14)
ax.set_xticklabels('', size=10)

plt.subplot(1,2,2)
ax = sns.lineplot(data=history.history.get('loss'), color='royalblue')
ax.set_xlabel("Epoch", size = 14)
ax.set_ylabel("Loss", size = 14)
ax.set_title("Training Loss", size = 14, fontweight='bold')
fig.set_figheight(6)
fig.set_figwidth(16)

# Make predictions
test_predict_keras = regression.predict(x_test_keras)

# Invert predictions
train_predict_keras = scaler.inverse_transform(train_predict_keras)
y_train_keras = scaler.inverse_transform(y_train_keras)
test_predict_keras = scaler.inverse_transform(test_predict_keras.reshape((test_predict_keras.shape[0], test_predict_keras.shape[1])))
y_test = scaler.inverse_transform(y_test_keras)

# Calculate root MSE
trainScore = math.sqrt(mean_squared_error(y_train[:,0], y_train_pred[:,0]))
print(f'Train score {trainScore:.2f} RMSE')
testScore = math.sqrt(mean_squared_error(y_test[:,0], y_test_pred[:,0]))
print(f'Test score {testScore:.2f} RMSE')

# shift train predictions for plotting
trainPredictPlot_keras = np.empty_like(price)
trainPredictPlot_keras[:, :] = np.nan
trainPredictPlot_keras[lookback:len(train_predict_keras)+lookback, :] = train_predict_keras

# shift test predictions for plotting
testPredictPlot_keras = np.empty_like(price)
testPredictPlot_keras[:, :] = np.nan
testPredictPlot_keras[len(train_predict_keras)+lookback-1:len(price)-1, :] = test_predict_keras

original = scaler.inverse_transform(price['Close'].values.reshape(-1,1))

predictions_keras = np.append(trainPredictPlot_keras, testPredictPlot_keras, axis=1)
predictions_keras = np.append(predictions_keras, original, axis=1)
result_keras = pd.DataFrame(predictions_keras)

The error occurs in the trainPredictPlot_keras[lookback:len(train_predict_keras)+lookback, :] = train_predict_keras line saying could not broadcast input array from shape (1006,19) into shape (1006,1)


title: All values replaced with pool max in pytorch

body:
Given z equal to
tensor([[[[0.0908, 0.1286, 0.6942, 0.5161],
          [0.4227, 0.2154, 0.5990, 0.8666],
          [0.3009, 0.2399, 0.1818, 0.7551],
          [0.2396, 0.4485, 0.4027, 0.5303]],
         [[0.8251, 0.7457, 0.2091, 0.7313],
          [0.3823, 0.7351, 0.3823, 0.2072],
          [0.0863, 0.5489, 0.6515, 0.3855],
          [0.5247, 0.8685, 0.6078, 0.6181]]]])

We have that
torch.nn.MaxUnpool2d(2)(*torch.nn.MaxPool2d(2, return_indices=True)(z))
tensor([[[[0.0000, 0.0000, 0.0000, 0.0000],
          [0.4227, 0.0000, 0.0000, 0.8666],
          [0.0000, 0.0000, 0.0000, 0.7551],
          [0.0000, 0.4485, 0.0000, 0.0000]],
         [[0.8251, 0.0000, 0.0000, 0.7313],
          [0.0000, 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.6515, 0.0000],
          [0.0000, 0.8685, 0.0000, 0.0000]]]])

How can I idiomatically compute a similar inverse where instead of all non-max values being zero, all are equal to the max? In this case I want:
tensor([[[[0.4227, 0.4227, 0.8666, 0.8666],
          [0.4227, 0.4227, 0.8666, 0.8666],
          [0.4485, 0.4485, 0.7551, 0.7551],
          [0.4485, 0.4485, 0.7551, 0.7551]],
         [[0.8251, 0.8251, 0.7313, 0.7313],
          [0.8251, 0.8251, 0.7313, 0.7313],
          [0.8685, 0.8685, 0.6515, 0.6515],
          [0.8685, 0.8685, 0.6515, 0.6515]]]])

The solution should work for all kernel sizes, padding, and dilation, and be fast. In this simple case, for example, I could easily compute what I want as a Kroeneker product with torch.nn.MaxPool2d(2)(z), but this does not generalize.


title: Interleaving a set of channels during concatenation?

body:
I am trying to perform a quaternion space concatenation which requires the four dimensions r,i,j,k to be concatenated. According to quaternion theory, we cannot apply the torch.cat function directly as they would mess up the components as r channels have to be concatenated with r channels and so on. I managed to perform this action using the quaternion_concat function adapted from here. However, when I use this for networks like densenet, this takes a long time due to multiple for loops concatenations.
Code: Let me give an example, tensor_1 and tensor_2 are two tensors that need to be concatenated, and it has 16 channels each. That means it has 4 channels of r,i,j,k respectively stacked together. I used the torch.chunk function to separate these and concatenate them separately and finally combine them back. Is there any way I could efficiently perform this?
import torch

def quarternion_concat(x, dim=2):
    output = [[] for i in range(4)]
    for _x in x:
        sp = torch.chunk(_x, 4, dim=dim)
        for i in range(4):
            output[i].append(sp[i])

    final = []
    for o in output:
        o = torch.cat(o, dim)
        final.append(o)

    return torch.cat(final, dim)

tensor_1 = torch.randn((1, 16, 64, 64), requires_grad=False)
tensor_2 = torch.randn((1, 16, 64, 64), requires_grad=False)

tensor3 = quarternion_concat([tensor_1, tensor_2], dim=1)


title: How do you test a custom dataset in Pytorch?

body:
I've been following tutorials in Pytorch that use datasets from Pytorch that allow you to enable whether you'd like to train using the data or not... But now I'm using a .csv and a custom dataset.
class MyDataset(Dataset):
    def __init__(self, root, n_inp):
        self.df = pd.read_csv(root)
        self.data = self.df.to_numpy()
        self.x , self.y = (torch.from_numpy(self.data[:,:n_inp]),
                           torch.from_numpy(self.data[:,n_inp:]))
    def __getitem__(self, idx):
        return self.x[idx, :], self.y[idx,:]
    def __len__(self):
        return len(self.data)

How can I tell Pytorch not to train my test_dataset so I can use it as a reference of how accurate my model is?
train_dataset = MyDataset("heart.csv", input_size)
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle =True)
test_dataset = MyDataset("heart.csv", input_size)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle =True)


title: Why is my loss not decreasing over training 10 epochs?

body:
My hardware is a Ryzen 5000 series cpu with an nvidia rtx 3060 gpu. I'm currently working on a school assignment involving using a deep learning model (implemented in PyTorch) to predict COVID diagnosis from CT slice images. The dataset can be found at this url on GitHub: https://github.com/UCSD-AI4H/COVID-CT
I've written a custom dataset that takes the images from the dataset and resizes them to 224x224. I've also converted all rgba or grayscale images to rgb using skimage.color. Other transforms include random horizontal and vertical flipping, as well as ToTensor(). To evaluate the model I've used sklearn.metrics to compute the AUC, F1 score, and accuracy of the model.
My trouble is that I can't get the model to train. After 10 epochs the loss has not decreased. I've tried adjusting the learning rate of my optimizer but it hasn't helped. Any recommendations/thoughts would be greatly appreciated. Thanks!
class RONANet(nn.Module):
  def __init__(self, classifier_type=None):
    super(RONANet, self).__init__()
    self.classifier_type = classifier_type
    self.relu = nn.ReLU()
    self.maxpool = nn.MaxPool2d(kernel_size=2, stride=2)
    self.classifier = self.compose_classifier()
    self.conv_layers = nn.Sequential(
        nn.Conv2d(3, 32, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(32),
        self.relu,
        self.maxpool,
        nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(64),
        self.relu,
        self.maxpool,
        nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(128),
        self.relu,
        self.maxpool,
        nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),
        self.relu,
        self.maxpool,
        nn.AdaptiveAvgPool2d(output_size=(1,1)),
    )
  def compose_classifier(self):
    if 'fc' in self.classifier_type:
            classifier = nn.Sequential(
                nn.Flatten(),
                nn.Linear(14**2*256, 256),
                self.relu,
                nn.Linear(256, 128),
                self.relu,
                nn.Linear(128, 2))

    elif 'conv'in self.classifier_type:
        classifier = nn.Sequential(
            nn.Conv2d(256, 1, kernel_size=1, stride=1))
        return classifier

  def forward(self, x):
    features = self.conv_layers(x)
    out = self.classifier(features)
    if 'conv' in self.classifier_type:
        out = out.reshape([-1,])
    return out

RONANetv1 = RONANet(classifier_type='conv')
RONANetv1 = RONANetv1.cuda()
RONANetv2 = RONANet(classifier_type='fc')
RONANetv2 = RONANetv2.cuda()

criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(RONANetv1.parameters(), lr=0.1)
num_epochs = 100
best_auc = 0.5 # set threshold to random model performance
scores = {}

for epoch in range(num_epochs):
RONANetv1.train()
print(f'Current Epoch: {epoch+1}')
epoch_loss = 0

for images, labels in train_dataloader:
    batch_loss = 0
    optimizer.zero_grad()
    with torch.set_grad_enabled(True):
        images = images.cuda()
        labels = labels.cuda()
        out = RONANetv1(images)
        loss = criterion(out, labels)
        batch_loss += loss.item()
        loss.backward()
        optimizer.step()
        epoch_loss += batch_loss
print(f'Loss this epoch: {epoch_loss}\n')

current_val_auc, current_val_f1, current_val_acc = get_scores(RONANetv1, val_dataloader)
if current_val_auc > best_auc:
    best_auc = current_val_auc
    torch.save(RONANetv1.state_dict(), 'RONANetv1.pth')
    scores['AUC'] = current_val_auc
    scores['f1'] = current_val_f1
    scores['Accuracy'] = current_val_acc
    print(scores)

.
Output:

Current Epoch: 1
Loss this epoch: 38.038745045661926

{'AUC': 0.6632183908045978, 'f1': 0.0, 'Accuracy': 0.4915254237288136}

Current Epoch: 2
Loss this epoch: 37.96312761306763

Current Epoch: 3
Loss this epoch: 37.93656861782074

Current Epoch: 4
Loss this epoch: 38.045261442661285

Current Epoch: 5
Loss this epoch: 38.01626980304718

Current Epoch: 6
Loss this epoch: 37.93017905950546

Current Epoch: 7
Loss this epoch: 37.913547694683075

Current Epoch: 8
Loss this epoch: 38.049841582775116

Current Epoch: 9
Loss this epoch: 37.95650988817215


title: Can you use LIME for Object Detection? My image needs a Dim attribute

body:
I want to use the Lime library to evaluate an image for explainability.
I have these lines of code;
iter = 3500
top_labels = 5
num_super_pixels = 10
fasterrcnn_model.eval()
explainer = lime_image.LimeImageExplainer()
explanation = explainer.explain_instance(img, fasterrcnn_model, top_labels = top_labels, hide_color = 0, num_samples = iter)

It seems line where I execute the explain_instance method, it does not like the image being a tensor.
When I convert it from a PILImage to a nump array, I have a problem because I need this image to have a dim attribute, which it does not have.
def get_image(file):
    path = os.path.join(Constants.dir_individual_image, file)
    p_img = PILImage.open(path)
    img = np.array(p_img)
    t_img = transform_img(img)
    return t_img, img, path

I am wondering if I can use LIME for object detection and Pytorch?


title: pytorch: loss.backward out of memory for the second batch

body:
I am learning seq2seq model using https://github.com/keon/seq2seq. I have successfully run the original project. Then, I want to train my-self translation model.
For myself data, the following code is OK. But for second batch, out of memory is reported for loss.backward. The second batch data is smaller than the first one.
    src, trg = src.cuda().T, trg.cuda().T
    optimizer.zero_grad()
    output = model(src, trg)
    loss = F.nll_loss(output[1:].view(-1, vocab_size),
                           trg[1:].contiguous().view(-1),
                           ignore_index=pad)
    loss.backward(retain_graph=True)
    torch.nn.utils.clip_grad_norm_(model.parameters(), grad_clip)
    optimizer.step()
    total_loss += loss.data.item()
    torch.cuda.empty_cache()

For batch size = 16, the above code is OK for first batch, and out of memory is reported at loss.backward for second batch.
The used gpu for first batch is:
2233 MB src, trg = src.cuda().T, trg.cuda().T
2331 MB output = model(src, trg)
4772 MB loss.backward(retain_graph=True)
6471 MB optimizer.step()
5312 MB torch.cuda.empty_cache()

For batch size = 1, the above code is OK for first batch, and out of memory is reported at loss.backward for second batch.
Any suggestion is appreciated!


title: In pytorch , how can I release all gpu memory when the program still run?

body:
import time
from torchvision.models import resnet18
import GPUtil
class AAA:
    def load(self):
        device = torch.device('cuda:0')

        self.dummy_tensor_4 = torch.randn(120, 3, 512, 512).float().to(device)  # 120*3*512*512*4/1000/1000 = 377.48M
        self.dummy_tensor_5 = torch.randn(10, 3, 512, 512).float().to(device)  # 80*3*512*512*4/1000/1000 = 251.64M
        self.model = resnet18(pretrained=True).to(device)
        self.X = self.model(self.dummy_tensor_5)
        time.sleep(5)

    def release(self):
        del self.dummy_tensor_4
        del self.dummy_tensor_5
        del self.model
        del self.X
        torch.cuda.empty_cache()

A = AAA()
A.load()
A.release()
del A

torch.cuda.empty_cache()
print("start waitting")
for i in range(30):
    GPUtil.showUtilization()
    time.sleep(1)

This is my code.
If I use "self.X = self.model(self.dummy_tensor_5) ", my gpu memory can't release clear. Am I missing something to delete?
If I use without "self.X = self.model(self.dummy_tensor_5) ", gpu memory release perfect.


title: Pytorch inference OOM after some batches

body:
I am trying to do inference with a GPT2-like model on a large dataset (26k samples). To speed it up I would like to do it in batches, but trying this it goes in Cuda OOM after some batches. The fact that it goes out only after some batches sounds strange to me, because I suppose the memory use should be more or less constant in different batches.
This is my code:
tokenizer.padding_side = "left"
tokenizer.pad_token = tokenizer.eos_token

sentences = ["<START_TOK>" + s + "<END_TOK>" + tokenizer.eos_token for s in sentences]

inputs = tokenizer(sentences, return_tensors="pt", padding=True, max_length=1024, truncation=True)

device = torch.device("cuda:0")
inputs = inputs.to(device)
model = model.to(device)
model.eval()
res = []
with torch.no_grad():
    output_sequences = model.generate(
            input_ids=inputs['input_ids'],
            attention_mask=inputs['attention_mask'],
            max_length=1024,
            pad_token_id=tokenizer.eos_token_id,
            no_repeat_ngram_size=2,
            do_sample=True,
            top_k=100,
            top_p=0.9,
            temperature=0.85
        )
     output_sequences = output_sequences.cpu() #not really sure this is useful, just tried, but the problem remained
     for i in range(len(sentences)):
         res.append(tokenizer.decode(output_sequences[i]))
model.train()
return res

What could be the problem?


title: Pytorch on GCP: Machine type is not available on this endpoint

body:
I'm new to GCP so pardon for prehaps asking/missing something obvious here.
I'm trying to deploy and create a version resource on GCP with a custom pytorch model. Everything have been working fine until now until I try to create the a new version of the model. Then I keep getting:
INVALID_ARGUMENT: Machine type is not available on this endpoint.
I've tried switching between different types from their list here without luck. What am I missing?
Here's the script I run to deploy:
MODEL_NAME='test_iris'
MODEL_VERSION='v1'
RUNTIME_VERSION='2.4'
MODEL_CLASS='model.PyTorchIrisClassifier'
PYTORCH_PACKAGE='gs://${BUCKET_NAME}/packages/torch-1.8.1+cpu-cp37-cp37m-linux_x86_64.whl'

DIST_PACKAGE='gs://${BUCKET_NAME}/models/Test_model-0.1.tar.gz'
GCS_MODEL_DIR='models/'
REGION="europe-west1"

# Creating model on AI platform
gcloud alpha ai-platform models create ${MODEL_NAME}\
--region=europe-west1 --enable-logging \
--enable-console-logging

gcloud beta ai-platform versions create ${MODEL_VERSION} --model=${MODEL_NAME} \
    --origin=gs://${BUCKET_NAME}/${GCS_MODEL_DIR} \
    --python-version=3.7 \
    --machine-type=mls1-c4-m2\
    --runtime-version=${RUNTIME_VERSION} \
    --package-uris=${DIST_PACKAGE},${PYTORCH_PACKAGE} \
    --prediction-class=${MODEL_CLASS}

Thanks!


title: "Numpy not Available" After installing Pytorch XLA

body:
I am just getting started with using TPUs on kaggle with Pytorch and install it as follows -
!pip3 install mkl

!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py

!python3 pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev

However, after installing Pytorch XLA, I am not able to use NumPy at all. Whenever I call functions like np.uint8 or even NumPy based functions from like torch.from_numpy I get an error who's bottom line says - NumPy not Available. Please note that I am able to import Numpy.
The whole stack trace is as follows -
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-abfcbbc939b0> in <module>
   1026                                  segmentation_Maps='/kaggle/input/pascal-voc/VOC2012/SegmentationClass/')
   1027 dataloader = DataLoader(dataset, batch_size=5)
-> 1028 for _, data in enumerate(dataloader):
   1029     i = data['image']
   1030     gt = data['ground_truth']

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    519             if self._sampler_iter is None:
    520                 self._reset()
--> 521             data = self._next_data()
    522             self._num_yielded += 1
    523             if self._dataset_kind == _DatasetKind.Iterable and \

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    559     def _next_data(self):
    560         index = self._next_index()  # may raise StopIteration
--> 561         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    562         if self._pin_memory:
    563             data = _utils.pin_memory.pin_memory(data)

/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

<ipython-input-1-abfcbbc939b0> in __getitem__(self, item)
    939         print(mask.shape)
    940         image = Image.fromarray(np.uint8(image)).convert('RGB')
--> 941         mask = torch.from_numpy(np.uint8(mask))
    942
    943         image = self.transforms(image)

RuntimeError: Numpy is not available

I have no clue what is going on. Could someone please Help.
PS - please note that pytorch xla updates pytorch to a nightly 1.9 version.
TIA


title: Building CUDA extension fails even after issue in code (deprecated AT_CHECK) is fixed

body:
I'm trying to install neural_renderer. Unfortunately, the original implementation only supports Python 2.7+ and PyTorch 0.4.0, so I'm using a fork that includes some fixes for compatibility with torch 1.7 (here). The main issue was using AT_CHECK(), which was not compatible with newer versions of PyTorch, and was replaced with TORCH_CHECK().
After running pip install neural_renderer_pytorch on the fixed version, using a virtual environment, I get the output (which I truncated to just the error):
 /tmp/pip-install-[somestring]/neural-renderer-pytorch_[somelongstring]/neural_renderer/cuda/load_textures_cuda.cpp:15:23: error: ‘AT_CHECK’ was not declared in this scope; did you mean ‘DCHECK’?
       15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
          |                       ^~~~~~~~

with [somestring] and [somelongstring] being some alphanumeric strings that changed with each compilation.
It looks like AT_CHECK is still being used somewhere in the code, but I don't know where. I know this error is exactly what the fork fixed, so I assume the cpp file is still cached somewhere from a previous compilation. But I can't figure out where exactly. I'm sure I'm on branch pytorch1.7 and running pip in the right repository; with torch==1.7.0 installed.
What I've tried so far, to no avail:

run pip cache purge before attempting to install
running pip with --no-cache-dir
deleting the virtualenv I'm using and making a new one
deleting the entire repository and making a new one

This issue on GitHub suggested just using PyTorch 1.4.0. This worked (i.e. I created a Python3.7 environment and ran conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.0 -c pytorch, then installed), but my goal is still to compile it for PyTorch 1.7.0 (and a newer version of Python).


title: how can i turn to GPU

body:
how can i turn this classifier to run n the GPU
i tryed to add resent18.cuda() but it gave me an error in when I run the train(epochs=20) and when i run show_preds() before train(epochs=20) and the error that desplays it :
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same
can you help me to do it
import os
import shutil
import random
import torch
import torchvision
import numpy as np
from google.colab import files
import time
import copy
import urllib
import torch
import PIL
import json
import numpy as np
import torch.nn.functional as F
from PIL import Image
from matplotlib import pyplot as plt

torch.manual_seed(0)
class_names = ['normal', 'viral', 'covid']
root_dir = 'COVID-19 Radiography Database'
source_dirs = ['NORMAL', 'Viral Pneumonia', 'COVID-19']

if os.path.isdir(os.path.join(root_dir, source_dirs[1])):
    os.mkdir(os.path.join(root_dir, 'test'))

    for i, d in enumerate(source_dirs):
        os.rename(os.path.join(root_dir, d), os.path.join(root_dir, class_names[i]))

    for c in class_names:
        os.mkdir(os.path.join(root_dir, 'test', c))

    for c in class_names:
        images = [x for x in os.listdir(os.path.join(root_dir, c)) if x.lower().endswith('png')]
        selected_images = random.sample(images, 30)
        for image in selected_images:
            source_path = os.path.join(root_dir, c, image)
            target_path = os.path.join(root_dir, 'test', c, image)
            shutil.move(source_path, target_path)
class ChestXRayDataset(torch.utils.data.Dataset):
    def __init__(self, image_dirs, transform):
        def get_images(class_name):
            images = [x for x in os.listdir(image_dirs[class_name]) if x[-3:].lower().endswith('png')]
            print(f'Found {len(images)} {class_name} examples')
            return images

        self.images = {}
        self.class_names = ['normal', 'viral', 'covid']

        for class_name in self.class_names:
            self.images[class_name] = get_images(class_name)

        self.image_dirs = image_dirs
        self.transform = transform

    def __len__(self):
        return sum([len(self.images[class_name]) for class_name in self.class_names])

    def __getitem__(self, index):
        class_name = random.choice(self.class_names)
        index = index % len(self.images[class_name])
        image_name = self.images[class_name][index]
        image_path = os.path.join(self.image_dirs[class_name], image_name)
        image = Image.open(image_path).convert('RGB')
        return self.transform(image), self.class_names.index(class_name)
train_transform = torchvision.transforms.Compose([
    torchvision.transforms.Resize(size=(224, 224)),
    torchvision.transforms.RandomHorizontalFlip(),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

test_transform = torchvision.transforms.Compose([
    torchvision.transforms.Resize(size=(224, 224)),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])
train_dirs = {
    'normal': '/content/gdrive/MyDrive/covidcalssifierdataset/COVID-19 Radiography Database/normal',
    'viral': '/content/gdrive/MyDrive/covidcalssifierdataset/COVID-19 Radiography Database/viral',
    'covid': '/content/gdrive/MyDrive/covidcalssifierdataset/COVID-19 Radiography Database/covid'
}

train_dataset = ChestXRayDataset(train_dirs, train_transform)

test_dirs = {
    'normal': '/content/gdrive/MyDrive/covidcalssifierdataset/COVID-19 Radiography Database/test/normal',
    'viral': '/content/gdrive/MyDrive/covidcalssifierdataset/COVID-19 Radiography Database/test/normal',
    'covid': '/content/gdrive/MyDrive/covidcalssifierdataset/COVID-19 Radiography Database/test/covid'
}

test_dataset = ChestXRayDataset(test_dirs, test_transform)
batch_size = 6

dl_train = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
dl_test = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=True)

print('Number of training batches', len(dl_train))
print('Number of test batches', len(dl_test))
class_names = train_dataset.class_names

def show_images(images, labels, preds):
    plt.figure(figsize=(8, 4))
    for i, image in enumerate(images):
        plt.subplot(1, 6, i + 1, xticks=[], yticks=[])
        image = image.numpy().transpose((1, 2, 0))
        mean = np.array([0.485, 0.456, 0.406])
        std = np.array([0.229, 0.224, 0.225])
        image = image * std + mean
        image = np.clip(image, 0., 1.)
        plt.imshow(image)
        col = 'green'
        if preds[i] != labels[i]:
            col = 'red'

        plt.xlabel(f'{class_names[int(labels[i].numpy())]}')
        plt.ylabel(f'{class_names[int(preds[i].numpy())]}', color=col)
    plt.tight_layout()
    plt.show()
images, labels = next(iter(dl_train))
show_images(images, labels, labels)
images, labels = next(iter(dl_test))
show_images(images, labels, labels)
resnet18 = torchvision.models.resnet18(pretrained=True)
resnet18.fc = torch.nn.Linear(in_features=512, out_features=3)
loss_fn = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(resnet18.parameters(), lr=3e-5)
def show_preds():
    resnet18.eval()
    images, labels = next(iter(dl_test))
    outputs = resnet18(images)
    _, preds = torch.max(outputs, 1)
    show_images(images, labels, press)
def train(epochs):
    print('Starting training..')
    for e in range(0, epochs):
        print('='*20)
        print(f'Starting epoch {e + 1}/{epochs}')
        print('='*20)

        train_loss = 0.
        val_loss = 0.

        resnet18.train() # set model to training phase

        for train_step, (images, labels) in enumerate(dl_train):
            optimizer.zero_grad()
            outputs = resnet18(images)
            loss = loss_fn(outputs, labels)
            loss.backward()
            optimizer.step()
            train_loss += loss.item()
            if train_step % 20 == 0:
                print('Evaluating at step', train_step)

                accuracy = 0

                resnet18.eval() # set model to eval phase

                for val_step, (images, labels) in enumerate(dl_test):
                    outputs = resnet18(images)
                    loss = loss_fn(outputs, labels)
                    val_loss += loss.item()

                    _, preds = torch.max(outputs, 1)
                    accuracy += sum((preds == labels).numpy())

                val_loss /= (val_step + 1)
                accuracy = accuracy/len(test_dataset)
                print(f'Validation Loss: {val_loss:.4f}, Accuracy: {accuracy:.4f}')

                show_preds()

                resnet18.train()

                if accuracy >= 0.95:
                    print('Performance condition satisfied, stopping..')
                    return

        train_loss /= (train_step + 1)

        print(f'Training Loss: {train_loss:.4f}')
    print('Training complete..')

train(epochs=20)


title: Building a confusion matrix in pytorch

body:
I need help to create a confusion matrix 3*3 for this PyTorch classifier so can you help me to build it, please?
I need  to connect the inputs and the outputs with the confusion matrix that we will create it it is an image classifier running in PyTorch by resenet18 model
so I need to creat the confusion matrix for it and if there any issues in my code I want you to teach me and thanks for everything
class_names = ['normal', 'viral', 'covid']
root_dir = 'COVID-19 Radiography Database'
source_dirs = ['NORMAL', 'Viral Pneumonia', 'COVID-19']

if os.path.isdir(os.path.join(root_dir, source_dirs[1])):
    os.mkdir(os.path.join(root_dir, 'test'))

    for i, d in enumerate(source_dirs):
        os.rename(os.path.join(root_dir, d), os.path.join(root_dir, class_names[i]))

    for c in class_names:
        os.mkdir(os.path.join(root_dir, 'test', c))

    for c in class_names:
        images = [x for x in os.listdir(os.path.join(root_dir, c)) if x.lower().endswith('png')]
        selected_images = random.sample(images, 30)
        for image in selected_images:
            source_path = os.path.join(root_dir, c, image)
            target_path = os.path.join(root_dir, 'test', c, image)
            shutil.move(source_path, target_path)
class ChestXRayDataset(torch.utils.data.Dataset):
    def __init__(self, image_dirs, transform):
        def get_images(class_name):
            images = [x for x in os.listdir(image_dirs[class_name]) if x[-3:].lower().endswith('png')]
            print(f'Found {len(images)} {class_name} examples')
            return images

        self.images = {}
        self.class_names = ['normal', 'viral', 'covid']

        for class_name in self.class_names:
            self.images[class_name] = get_images(class_name)

        self.image_dirs = image_dirs
        self.transform = transform

    def __len__(self):
        return sum([len(self.images[class_name]) for class_name in self.class_names])

    def __getitem__(self, index):
        class_name = random.choice(self.class_names)
        index = index % len(self.images[class_name])
        image_name = self.images[class_name][index]
        image_path = os.path.join(self.image_dirs[class_name], image_name)
        image = Image.open(image_path).convert('RGB')
        return self.transform(image), self.class_names.index(class_name)
train_transform = torchvision.transforms.Compose([
    torchvision.transforms.Resize(size=(224, 224)),
    torchvision.transforms.RandomHorizontalFlip(),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

test_transform = torchvision.transforms.Compose([
    torchvision.transforms.Resize(size=(224, 224)),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])
train_dirs = {
    'normal': '/content/gdrive/MyDrive/covidcalssifierdataset/COVID-19 Radiography Database/normal',
    'viral': '/content/gdrive/MyDrive/covidcalssifierdataset/COVID-19 Radiography Database/viral',
    'covid': '/content/gdrive/MyDrive/covidcalssifierdataset/COVID-19 Radiography Database/covid'
}

train_dataset = ChestXRayDataset(train_dirs, train_transform)

test_dirs = {
    'normal': '/content/gdrive/MyDrive/covidcalssifierdataset/COVID-19 Radiography Database/test/normal',
    'viral': '/content/gdrive/MyDrive/covidcalssifierdataset/COVID-19 Radiography Database/test/normal',
    'covid': '/content/gdrive/MyDrive/covidcalssifierdataset/COVID-19 Radiography Database/test/covid'
}

test_dataset = ChestXRayDataset(test_dirs, test_transform)
batch_size = 6

dl_train = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
dl_test = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=True)

print('Number of training batches', len(dl_train))
print('Number of test batches', len(dl_test))
class_names = train_dataset.class_names

def show_images(images, labels, preds):
    plt.figure(figsize=(8, 4))
    for i, image in enumerate(images):
        plt.subplot(1, 6, i + 1, xticks=[], yticks=[])
        image = image.numpy().transpose((1, 2, 0))
        mean = np.array([0.485, 0.456, 0.406])
        std = np.array([0.229, 0.224, 0.225])
        image = image * std + mean
        image = np.clip(image, 0., 1.)
        plt.imshow(image)
        col = 'green'
        if preds[i] != labels[i]:
            col = 'red'

        plt.xlabel(f'{class_names[int(labels[i].numpy())]}')
        plt.ylabel(f'{class_names[int(preds[i].numpy())]}', color=col)
    plt.tight_layout()
    plt.show()
images, labels = next(iter(dl_train))
show_images(images, labels, labels)
images, labels = next(iter(dl_test))
show_images(images, labels, labels)
resnet18 = torchvision.models.resnet18(pretrained=True)
resnet18.fc = torch.nn.Linear(in_features=512, out_features=3)
loss_fn = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(resnet18.parameters(), lr=3e-5)
def show_preds():
    resnet18.eval()
    images, labels = next(iter(dl_test))
    outputs = resnet18(images)
    _, preds = torch.max(outputs, 1)
    show_images(images, labels, press)
def train(epochs):
    print('Starting training..')
    for e in range(0, epochs):
        print('='*20)
        print(f'Starting epoch {e + 1}/{epochs}')
        print('='*20)

        train_loss = 0.
        val_loss = 0.

        resnet18.train() # set model to training phase

        for train_step, (images, labels) in enumerate(dl_train):
            optimizer.zero_grad()
            outputs = resnet18(images)
            loss = loss_fn(outputs, labels)
            loss.backward()
            optimizer.step()
            train_loss += loss.item()
            if train_step % 20 == 0:
                print('Evaluating at step', train_step)

                accuracy = 0

                resnet18.eval() # set model to eval phase

                for val_step, (images, labels) in enumerate(dl_test):
                    outputs = resnet18(images)
                    loss = loss_fn(outputs, labels)
                    val_loss += loss.item()

                    _, preds = torch.max(outputs, 1)
                    accuracy += sum((preds == labels).numpy())

                val_loss /= (val_step + 1)
                accuracy = accuracy/len(test_dataset)
                print(f'Validation Loss: {val_loss:.4f}, Accuracy: {accuracy:.4f}')

                show_preds()

                resnet18.train()

                if accuracy >= 0.95:
                    print('Performance condition satisfied, stopping..')
                    return

        train_loss /= (train_step + 1)

        print(f'Training Loss: {train_loss:.4f}')
    print('Training complete..')

train(epochs=20)


title: Pytorch min and max of tensor

body:
I have following pytorch tensor:
>>> mean_actions
tensor([[-5.7547e-04,  1.4318e-02,  1.9328e-04, -2.5660e-03,  3.5269e-03,
         -1.3797e-02, -6.1871e-04, -2.7425e-03,  1.1661e-03,  1.6873e-03,
          3.9045e-03,  1.8047e-03,  4.8656e-03,  5.7182e-03, -4.8501e-03,
         -5.5913e-03, -4.4574e-03, -3.3154e-03, -4.9826e-03, -1.0071e-02,
         -2.3483e-03, -1.1413e-02, -4.9613e-03, -1.1648e-02,  2.4752e-03,
         -1.9764e-03,  3.1063e-03, -6.3481e-05,  7.6710e-03,  5.7503e-04]])

I am getting following min and max values out of tensor:
>>> th.min(mean_actions)
tensor(-0.0138)
>>> th.max(mean_actions)
tensor(0.0143)

However, I dont see -0.0138 and 0.0143 present in the tensor. What I am missing?
Here are the screenshots from debug session:


title: How to solve PyTorch out of memory allocation errors?

body:
I'm trying to load a model onto the GPU but when I do I receive the following error message:
"CUDA out of memory. Tried to allocate 86.00 MiB (GPU 0; 6.00 GiB total capacity; 84.95 MiB already allocated; 0 bytes free; 86.00 MiB reserved in total by PyTorch)"
The model is
model = BertForSequenceClassification.from_pretrained(
    bert-base-cased,
    num_labels = 5,
    output_attentions = False,
    output_hidden_states = False,
)

model.cuda()

I've checked nvidia-smi and there's nothing using up the memory.


title: VSCode BERT ValueError: Unable to access local path

body:
I have written the code for an entity extraction model using bert but when I run the train.py file I get a value error.
This is the structure of my code with the configuration file in VSCode, I have downloaded bert models from here

Error
>> (myenv) PS D:\Transformers\bert-entity-extraction> python src/train.py

Configuration Complete!
Traceback (most recent call last):
  File "src/train.py", line 83, in <module>
    model = EntityModel(num_tag = num_tag, num_pos = num_pos)
  File "D:\Transformers\bert-entity-extraction\src\model.py", line 25, in __init__
    self.bert = transformers.BertModel.from_pretrained(config.BASE_MODEL_PATH)
  File "C:\Users\hp\anaconda3\envs\myenv\lib\site-packages\transformers\modeling_utils.py", line 1080, in from_pretrained
    **kwargs,
  File "C:\Users\hp\anaconda3\envs\myenv\lib\site-packages\transformers\configuration_utils.py", line 427, in from_pretrained
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "C:\Users\hp\anaconda3\envs\myenv\lib\site-packages\transformers\configuration_utils.py", line 492, in get_config_dict
    user_agent=user_agent,
  File "C:\Users\hp\anaconda3\envs\myenv\lib\site-packages\transformers\file_utils.py", line 1289, in cached_path
    raise ValueError(f"unable to parse {url_or_filename} as a URL or as a local path")
ValueError: unable to parse D:\Transformers\bert-entity-extraction\input\bert-base-uncased_L-12_H-768_A-12\config.json as a URL or as a local path

How to fix this?


title: How to update tensors matching dimensionwise vectors

body:
Let there be two 2D tensors, A (m × c) and B (n × c). Each row vector which belongs to B also belongs to A i.e. . Additionally, row vectors in A are not unique i.e. A may have duplicate rows. However, row vectors in B are unique.
There another pair of tensors P (m × f) and Q (n × f). I am trying to do the following
for i in range(B.shape[0]):
    rv = B[i, :]
    fv = Q[i, :]
    # P[<row indexes of A matching rv>, :] = fv

How to do this correctly?
Is it possible to get rid of the for loop?


title: How to get the anonymizing_video_by_DSFD running

body:
My goal is to use this GitHub repo to blur the faces of my input videos.
Current progress and problems: I have cloned the repo and then get familiar with it. But when I try to get it running on my end, I keep getting error messages like:
Traceback (most recent call last):
  File "demo.py", line 261, in <module>
    test_oneimage()
  File "demo.py", line 230, in test_oneimage
    det0 = infer(net, img, transform, thresh, cuda, shrink)
  File "demo.py", line 107, in infer
    y = net(x)  # forward pass
  File "/home/lakeside/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/lakeside/Desktop/Anonymizing_video_by_DSFD/face_ssd.py", line 342, in forward
    output = self.detect(
  File "/home/lakeside/.local/lib/python3.8/site-packages/torch/autograd/function.py", line 159, in __call__
    raise RuntimeError(
RuntimeError: Legacy autograd function with non-static forward method is deprecated. Please use new-style autograd function with static forward method. (Example: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)

And then I try to search a solution for resolving this, I have tried to add @staticmethod before the forward method and use something like:
y = net.apply(x)

But then it gives me error that:
demo.py:103: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
  x = Variable(x.unsqueeze(0), volatile=True)
Traceback (most recent call last):
  File "demo.py", line 261, in <module>
    test_oneimage()
  File "demo.py", line 230, in test_oneimage
    det0 = infer(net, img, transform, thresh, cuda, shrink)
  File "demo.py", line 107, in infer
    y = net.apply(x)  # forward pass
  File "/home/lakeside/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 473, in apply
    module.apply(fn)
  File "/home/lakeside/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 473, in apply
    module.apply(fn)
  File "/home/lakeside/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 474, in apply
    fn(self)
TypeError: 'Tensor' object is not callable

I have tried to figure this out, but no luck so far. I see on this Reddit post that there are some people get it work successfully.