title: Bit operation of cuda tensor in pytorch

body:
I'd like to do some bit operation with torch.tensor in cuda, such as <<, >>, or extract each bit which represents a float number, such as 0 01101 01010101012 (3555H) for 0.333 in float16.

what I do now is as below:

def _decompose(self, value, exp_bias=None):
    '''
    decompose a single into sign, exp and mant
    '''
    if exp_bias is None:
        exp_bias = self.exp_bias
    # smallest non-zero float point
    descriminator = torch.tensor((2 ** (-exp_bias)) / 2).type_as(value)
    sign = (value > descriminator).type_as(value)
    sign -= (value < -descriminator).type_as(value)
    value = value.abs()
    exp = torch.log2(value).floor()
    mant = value / (2 ** exp)
    return sign, exp, mant

Is there a way to achieve such function? Or is there any wrong with my code? Thanks.


title: Runtime error while Object Detection using SSD and Opencv

body:
Hope everyone is fine. I was writing the following script using Python for Object Detection on a Video:
#Object Detection

#Importing Libraries
import torch
from torch.autograd import Variable
import cv2
from data import BaseTransform, VOC_CLASSES as labelmap
from ssd import build_ssd
import imageio

#Defining the fuction that do the Detection
def detect(frame, net, transform):
    height, width = frame.shape[:2]
    frame_t = transform(frame)[0]
    x = torch.from_numpy(frame_t).permute(2, 0, 1)
    x = Variable(x.unsqueeze(0))
    y = net(x)
    detections = y.data
    scale = torch.Tensor([width, height, width, height])
    for i in range(detections.size(1)):
        j = 0
        while detections[0, i, j, 0] >= 0.6:
            pt = (detections[0, i, j, 1:] * scale).numpy()
            cv2.rectangle(frame, (int(pt[0]), int(pt[1])), (int(pt[2]), int(pt[3])), (255, 0 , 0), 2)
            cv2.putText(frame, labelmap[i - 1], (int(pt[0]), int(pt[1])), cv2.FONT_HERSHEY_SIMPLEX, 2, (255, 255, 255), 2, cv2.LINE_AA)
            j += 1
    return frame

# Creating SSD Neural Networks
net = build_ssd('test')
net.load_state_dict(torch.load('ssd300_mAP_77.43_v2.pth'), map_location = lambda storage, loc: storage)

#Creating transformation
transform = BaseTransform(net.size, (104/256.0, 117/256.0, 123/256.0))

#Doing some Object Detection in the video
reader = imageio.get_reader('funny_dog.mp4')
fps = reader.get_meta_data()['fps']
writer = imageio.get_writer('output.mp4', fps = fps)
for i, frame in enumerate(reader):
    frame = detect(frame, net.eval(), transform)
    writer.append_data(frame)
    print(i)
writer.close()

Whenever I run this code then I get a Runtime Error, i.e.
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

Please help me to correct the problem and explain why this is happening.


title: Any ideas on how to efficiently create this matrix/mask?

body:
I want to make a torch tensor or numpy array efficiently of a matrix that is a shifting window of 1s.
So for example the matrix below would be a window=3.
The diagonal element has 3 1s to it's right, and 3 1s to it's left but it doesn't wrap round like a circulant matrix, so row 1 just has 4 1s.
Has anyone got any ideas, this is to be used as a mask.


title: PySyft Worker overfitting

body:
I try to train a image classification (cifar10) with pysyft. My trainsetup has 10 workers where every worker gets betwen 800 and 1200 images of the dataset.
My Problem is that after about 250-300 epochs, the train loss is at about 0.005 and the model stops improving though the test accuracy is just at about 45% with an increasing loss 1.5 -> 8.5.
I tried the same with 100 workers on 500 images where it stoped at 32%.
Furthrmore the implementation is part of a comparison between models and FL Frameworks and therefor the model can't be changed and the data will be loaded localy and transformed into a Dataloader.
Hence I'm very unexperienced with Pytorch and PySyft it might be that I made some mistakes when training the model though i tried to stay as close as possible with the example.
I trained the model without PySyft and it reached about 85% so I think my dataloader and model should be not the problem. For me it looks like the workers overfit on their own data during the training.
Is there a way to prevent workers to overfit or calculate a loss for the global model instead of the workers?
Trainer:

def fl_train(args, model, device, federated_train_loader, optimizer, epoch, log):
    model.train()
    results = []
    metrics = []
    t1 = time.time()
    cel = nn.CrossEntropyLoss()
    for batch_idx, (data, target) in enumerate(federated_train_loader): # <-- now it is a distributed dataset
        t2 = time.time()
        model.send(data.location) # <-- NEW: send the model to the right location
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, target.long())
        loss.backward()
        optimizer.step()
        model.get() # <-- NEW: get the model back
        if batch_idx % args.log_interval == 0:
            loss = loss.get() # <-- NEW: get the loss back
            results.append(loss.item())
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * BATCH_SIZE, len(federated_train_loader) * BATCH_SIZE,
                100. * batch_idx / len(federated_train_loader), loss.item()))

Model:
class CNN(nn.Module):

    def __init__(self):
        super(CNN, self).__init__()

        self.conv_layer = nn.Sequential(

            # Conv Layer block 1
            nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3),
            nn.ReLU(inplace=True),
            nn.MaxPool2d((2,2)),

            # Conv Layer block 2
            nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3),
            nn.ReLU(inplace=True),
            nn.MaxPool2d((2,2)),

            # Conv Layer block 3
            nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3),
            nn.ReLU(inplace=True),
        )

        self.fc_layer = nn.Sequential(
            nn.Linear(1024, 64),
            nn.ReLU(inplace=True),
            nn.Linear(64, 10)
        )

    def forward(self, x):
        # CNN layers
        x = self.conv_layer(x)

        # flatten
        x = x.view(-1, 1024)

        # NN layer
        x = self.fc_layer(x)
        return F.log_softmax(x, dim=1)

Main:
model = CNN().to(device)
optimizer = optim.SGD(model.parameters(), lr=0.02) # TODO momentum is not supported at the moment
log = {}
for epoch in range(1, args.epochs + 1):
    log = fl_train(args, model, device, f_dataloader, optimizer, epoch, log)
    if epoch % 20 == 0:
      log = test(args, model, device, test_loader, epoch, log)
    if epoch % 100 == 0:
      store_results(log, model)

Log:
....
Train Epoch: 317 [0/10400 (0%)] Loss: 0.005194
Train Epoch: 317 [3000/10400 (29%)] Loss: 0.003882
Train Epoch: 317 [6000/10400 (58%)] Loss: 0.003100
Train Epoch: 317 [9000/10400 (87%)] Loss: 0.004298
Train Epoch: 318 [0/10400 (0%)] Loss: 0.007426
Train Epoch: 318 [3000/10400 (29%)] Loss: 0.002255
Train Epoch: 318 [6000/10400 (58%)] Loss: 0.003835
Train Epoch: 318 [9000/10400 (87%)] Loss: 0.005277
Train Epoch: 319 [0/10400 (0%)] Loss: 0.006207
Train Epoch: 319 [3000/10400 (29%)] Loss: 0.003562
Train Epoch: 319 [6000/10400 (58%)] Loss: 0.001904
Train Epoch: 319 [9000/10400 (87%)] Loss: 0.002644
Train Epoch: 320 [0/10400 (0%)] Loss: 0.007491
Train Epoch: 320 [3000/10400 (29%)] Loss: 0.003794
Train Epoch: 320 [6000/10400 (58%)] Loss: 0.002643
Train Epoch: 320 [9000/10400 (87%)] Loss: 0.002981
Test set: Average loss: 9.1279, Accuracy: 458/1000 (46%)

Train Epoch: 321 [0/10400 (0%)] Loss: 0.007153
Train Epoch: 321 [3000/10400 (29%)] Loss: 0.004265
Train Epoch: 321 [6000/10400 (58%)] Loss: 0.002708
Train Epoch: 321 [9000/10400 (87%)] Loss: 0.002518
Train Epoch: 322 [0/10400 (0%)] Loss: 0.006285
Train Epoch: 322 [3000/10400 (29%)] Loss: 0.002357
Train Epoch: 322 [6000/10400 (58%)] Loss: 0.002465
Train Epoch: 322 [9000/10400 (87%)] Loss: 0.002406
Train Epoch: 323 [0/10400 (0%)] Loss: 0.005361
Train Epoch: 323 [3000/10400 (29%)] Loss: 0.004807
Train Epoch: 323 [6000/10400 (58%)] Loss: 0.001903
Train Epoch: 323 [9000/10400 (87%)] Loss: 0.003711
Train Epoch: 324 [0/10400 (0%)] Loss: 0.006609
....


title: Pytorch - Trick AutoGrad into thinking another output is the final outcome

body:
The scenario:

I have a simple torch CNN network that predicts if a given image input is a dog or a cat.

After getting the output of the neural network, I need to apply a modifier of X to each prediction. For example, if the neural network return [0.6, 0.4], and I want to apply a modifier of [0.05, -0.03], I need the result to be [0.65, 0.37].

The desired result:

I would like AutoGrad to think the final output is [0.65, 0.37]. That is, AutoGrad shouldn't consider the modifier addition at all. In fact, It needs to be tricked into thinking the last operation result is [0.65, 0.37] instead of [0.6, 0.4], and apply the backpropagation with that in mind.

Here's what I need to know:

How am I suppose to do this? I know Torch records every single operation and compute a dynamic graph accordingly. I don't want this operation to be recorded at all, and I can't use the torch.no_grad() wrapper, because I need to be able to do the backpropagation after.

Edit 1
@trsvchn Here's what happen when I do your method. Is this expected? Will autograd use the "T" value instead of the "data" value when doing the backpropagation?


title: Difference between PyTorch and exact analytical expression

body:
While using Multivariate normal distribution in PyTorch I decided to compare it with exact analytical expression.
To my surprise, there was a small difference between them.
Is there any reason for this behaviour?
Firstly, calculate probabilities using MultivariateNormal:
1 from torch.distributions.multivariate_normal import MultivariateNormal
2 import torch
3 sigma = 2
4 m = MultivariateNormal(torch.zeros(2, dtype=torch.float32), torch.eye(2, dtype=torch.float32)*sigma**2)
5 values_temp = torch.zeros(size=(1,2), dtype=torch.float32)
6 out_torch = torch.exp(m.log_prob(values_temp))
7 out_torch
Out: tensor([0.0398])

Secondly, one can write exact formula for this case:
1 import numpy as np
2 out_exact = 1/(2*np.pi*sigma**2) * torch.exp(-torch.pow(values_temp, 2).sum(dim=-1)/(2*sigma**2))
3 out_exact
Out: tensor([0.0398])

There is a difference between them:
1 (out_torch - out_exact).sum()
Out: tensor(3.7253e-09)

Can someone help me understand the behavior of these two snippets? Which of these two expressions is more precise? Maybe someone can underline my mistake in any part of the code?


title: Is NN just Bad at Solving this Simple Linear Problem, or is it because of Bad Training?

body:
I was trying to train a very straightforward (I thought) NN model with PyTorch and skorch, but the bad performance really baffles me, so it would be great if you have any insight into this.

The problem is something like this: there are five objects, A, B, C,
D, E, (labeled by their fingerprint, e.g.(0, 0) is A, (0.2, 0.5) is B,
etc) each correspond to a number, and the problem is trying to find
what number does each correspond to. The training data is a list of
"collections" and the corresponding sum. for example: [A, A, A, B, B]
== [(0,0), (0,0), (0,0), (0.2,0.5), (0.2, 0.5)] --> 15, [B, C, D, E] == [(0.2,0.5), (0.5,0.8), (0.3,0.9), (1,1)] --> 30 .... Note that number of object in one collection is not constant

There is no noise or anything, so it's just a linear system that can be solved directly. So I would thought this would be very easy for a NN for find out. I'm actually using this example as a sanity check for a more complicated problem, but was surprised that NN couldn't even solve this.
Now I'm just trying to pinpoint exactly where it went wrong. The model definition seem to be right, the data input is right, is the bad performance due to bad training? or is NN just bad at these things?
here is the model definition:
class NN(nn.Module):
    def __init__(
        self,
        input_dim,
        num_nodes,
        num_layers,
        batchnorm=False,
        activation=Tanh,
    ):
        super(SingleNN, self).__init__()
        self.get_forces = get_forces
        self.activation_fn = activation

        self.model = MLP(
            n_input_nodes=input_dim,
            n_layers=num_layers,
            n_hidden_size=num_nodes,
            activation=activation,
            batchnorm=batchnorm,
        )

    def forward(self, batch):
        if isinstance(batch, list):
            batch = batch[0]
        with torch.enable_grad():
            fingerprints = batch.fingerprint.float()
            fingerprints.requires_grad = True
            #index of the current "collection" in the training list
            idx = batch.idx
            sorted_idx = torch.unique_consecutive(idx)
            o = self.model(fingerprints)
            total = scatter(o, idx, dim=0)[sorted_idx]

            return total

    @property
    def num_params(self):
        return sum(p.numel() for p in self.parameters())

class MLP(nn.Module):
    def __init__(
        self,
        n_input_nodes,
        n_layers,
        n_hidden_size,
        activation,
        batchnorm,
        n_output_nodes=1,
    ):
        super(MLP, self).__init__()
        if isinstance(n_hidden_size, int):
            n_hidden_size = [n_hidden_size] * (n_layers)
        self.n_neurons = [n_input_nodes] + n_hidden_size + [n_output_nodes]
        self.activation = activation
        layers = []
        for _ in range(n_layers - 1):
            layers.append(nn.Linear(self.n_neurons[_], self.n_neurons[_ + 1]))
            layers.append(activation())
            if batchnorm:
                layers.append(nn.BatchNorm1d(self.n_neurons[_ + 1]))
        layers.append(nn.Linear(self.n_neurons[-2], self.n_neurons[-1]))
        self.model_net = nn.Sequential(*layers)

    def forward(self, inputs):
        return self.model_net(inputs)

and the skorch part is straightforward
model = NN(2, 100, 2)
net = NeuralNetRegressor(
        module=model,
        ...
    )
net.fit(train_dataset, None)

For a test run, the dataset looks like the following (16 collections in total):
[[0.7484336 0.5656401]
 [0.        0.       ]
 [0.        0.       ]
 [0.        0.       ]]
[[1. 1.]
 [0. 0.]
 [0. 0.]]
[[0.51311415 0.67012525]
 [0.51311415 0.67012525]
 [0.         0.        ]
 [0.         0.        ]]
[[0.51311415 0.67012525]
 [0.7484336  0.5656401 ]
 [0.         0.        ]]
[[0.51311415 0.67012525]
 [1.         1.        ]
 [0.         0.        ]
 [0.         0.        ]]
[[0.51311415 0.67012525]
 [0.51311415 0.67012525]
 [0.         0.        ]
 [0.         0.        ]
 [0.         0.        ]
 [0.         0.        ]
 [0.         0.        ]
 [0.         0.        ]]
[[0.51311415 0.67012525]
 [1.         1.        ]
 [0.         0.        ]
 [0.         0.        ]
 [0.         0.        ]
 [0.         0.        ]]
....

with corresponding total:
[10, 11, 14, 14, 17, 18, ...]
It's easy to tell what are the objects/how many of them are in one collection just by eyeballing it
and the training process looks like:
 epoch    train_energy_mae    train_loss    cp     dur
-------  ------------------  ------------  ----  ------
      1              4.9852        0.5425     +  0.1486
      2             16.3659        4.2273        0.0382
      3              6.6945        0.7403        0.0025
      4              7.9199        1.2694        0.0024
      5             12.0389        2.4982        0.0024
      6              9.9942        1.8391        0.0024
      7              5.6733        0.7528        0.0024
      8              5.7007        0.5166        0.0024
      9              7.8929        1.0641        0.0024
     10              9.2560        1.4663        0.0024
     11              8.5545        1.2562        0.0024
     12              6.7690        0.7589        0.0024
     13              5.3769        0.4806        0.0024
     14              5.1117        0.6009        0.0024
     15              6.2685        0.8831        0.0024
....
    290              5.1899        0.4750        0.0024
    291              5.1899        0.4750        0.0024
    292              5.1899        0.4750        0.0024
    293              5.1899        0.4750        0.0024
    294              5.1899        0.4750        0.0025
    295              5.1899        0.4750        0.0025
    296              5.1899        0.4750        0.0025
    297              5.1899        0.4750        0.0025
    298              5.1899        0.4750        0.0025
    299              5.1899        0.4750        0.0025
    300              5.1899        0.4750        0.0025
    301              5.1899        0.4750        0.0024
    302              5.1899        0.4750        0.0025
    303              5.1899        0.4750        0.0024
    304              5.1899        0.4750        0.0024
    305              5.1899        0.4750        0.0025
    306              5.1899        0.4750        0.0024
    307              5.1899        0.4750        0.0025

You can see that it just stopped training after a while.
I can confirm that the NN does give different result for different fingerprint, but somehow the final predicted value is just never good enough.
I have tried different NN size, learning rate, batch size, activation function (tanh, relu, etc) and non of them seem to help. Do you have any insight into this? is there anything I did wrong/could try, or is NN just bad at this kind of task?


title: How to multiply pytorch array elements based on repeated elements in a second array

body:
I have what seems like a simple problem, but I cannot find answers for it anywhere. If I have two arrays, I want to multiply/combine the elements of one of them based on whether or not the elements in the other are sequential or repeated. For instance,
array_with_repeated_elements = tensor([1, 2, 0, 0, 2, 2, 2, 1, 0, 0])
# could just as well be [a, b, c, c, d, d, d, e, f, f]
array_to_be_multiplied = tensor([1., 3., 5., 2., 2., 7., 2., 4., 3., 4.])

desired_output = tensor([1, 3, 10, 28, 4, 12])

In numpy, this can be done easily:
first_index_of_each_sequence = np.hstack([0,np.where(array_with_repeated_elements[1:] != array_with_repeated_elements[0:-1])[0]+1])
# this creates array([0, 1, 2, 4, 7, 8])
desired_output = 1-np.multiply.reduceat(array_to_be_multiplied, first_index_of_each_sequence)

I can't seem to do this in pytorch. The best guess I have is this monster:
first_index_of_each_sequence = torch.cat([torch.LongTensor((0,)), torch.where(array_with_repeated_elementst[1:] != array_with_repeated_elementst[0:-1])[0]+1, torch.LongTensor((len(array_with_repeated_elements),))])
# makes tensor([0, 1, 2, 4, 7, 8, 10])

size_of_each_sequence = first_index_of_each_sequence[1:] - first_index_of_each_sequence[0:-1]
# makes tensor([1, 1, 2, 3, 1, 2])

full_length_array_of_ascending_index_elements = torch.arange(len(size_of_each_sequence)).repeat_interleave(size_of_each_sequence)
desired_output_base = torch.zeros(len(size_of_each_sequence))
# makes tensor([0, 1, 2, 2, 3, 3, 3, 4, 5, 5])

desired_output_base.index_add_(0, full_length_array_of_ascending_index_elements, torch.log(array_to_be_multipliedt))
# does what I want in log space, but ew if I ever have a zero

desired_output = torch.exp(desired_output_base)
# duh

Does anyone have any ideas on how to do this nicely? The easy numpy implementation suggests I've missed something in pytorch...


title: DenseNet, Sizes of tensors must match

body:
would you know how I can adapt this code so that sizes of tensors must match because I have this error: x = torch.cat([x1,x2],1) RuntimeError: Sizes of tensors must match except in dimension 0. Got 32 and 1 (The offending index is 0).
My images are size 416x416.
Thank you in advance for your help,
num_classes = 20
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.inc = models.inception_v3(pretrained=True)
        self.inc.aux_logits = False

        for child in list(self.inc.children())[:-5]:
            for param in child.parameters():
                param.requires_grad = False

        self.inc.fc = nn.Sequential()

        self.dens121 = models.densenet121(pretrained=True)

        for child in list(self.dens121.children())[:-6]:
            for param in child.parameters():
                param.requires_grad = False

        self.dens121 = nn.Sequential(*list(self.dens121.children())[:-1])

        self.SiLU = nn.SiLU()
        self.linear = nn.Linear(4096, num_classes)
        self.dropout = nn.Dropout(0.2)

    def forward(self, x):
        x1 = self.SiLU(self.dens121(x))
        x1 = x1.view(-1, 2048)

        x2 = self.inc(x).view(-1, 2048)
        x = torch.cat([x1,x2],1)

        return self.linear(self.dropout(x))


title: how to separate the low and high frequency components of an image by torch.fft?

body:
I want to separate the low and high frequency components of an image by torch.fft.
It would be better to give me a sample like this:
import cv2 as cv
import numpy as np

img = cv.imread('messi5.jpg',0)
f = np.fft.fft2(img)
fshift = np.fft.fftshift(f)

rows, cols = img.shape
crow,ccol = rows/2 , cols/2
fshift[crow-30:crow+30, ccol-30:ccol+30] = 0
f_ishift = np.fft.ifftshift(fshift)
img_back = np.fft.ifft2(f_ishift)
img_back = np.abs(img_back)


title: torch.unique does not work for float tensors

body:
I am trying to extract the unique elements from a float tensor.
I have tried :
out = torch.unique(my_tensor)

However this method only works for int/long tensor. My tensor is quantizied tensor in a non-uniform way, thus its guaranteed to have a small set of float values.


title: Train in pytorch throws an error, can not figure out what is the issue

body:
So here is the issue. I try to print my statistics about training but somehow i can not get it to work. It gives me a message:

IndexError: invalid index of a 0-dim tensor. Use tensor.item() in Python or tensor.item<T>() in C++ to convert a 0-dim tensor to a number at:                     100. * idx / len(train_data), loss.data[0]))

Tried to use .item but did not help.
also at line : epoch, idx * len(data), len(train_data), i was supposed to use len(train_data.dataset) but it doesnt exist (from ytb tutorial).
   def train(epoch): #model existiert zwar noch nicht, aber wir erschaffen uns unser eigenes model
        model.train() #model auf train setzen, damit es auch wirklich trainiert wird und nicht nur ausgeführt
        for idx, (data, target) in enumerate(train_data) # Daten aus vorgefertigten Daten beziehen, wir brauchen hier die train data
            target = torch.Tensor(target)
            data = Variable(data)
            target = Variable(target) # müssen data und target ja immer noch als Variablen verarbeiten
            optimizer.zero_grad() #setzen unseren Gradienten auf 0, damit wir überhaupt etwas lernen können
            out = model(data) #output soll das model data sein
            criterion = F.binary_cross_entropy #wollen ja Bilder klassifizieren in zwei Klassen
            loss = criterion(out, target) #wollen unser Loss als criterion von output und target berechnen -- zu beachten: target hat zwei Elemente und am Ende soll unser output auch zwei Elemente haben
            loss.backward() # wollen unseren Loss back - propagaten
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, idx * len(data), len(train_data),
                    100. * idx / len(train_data), loss.data[0]))
            optimizer.step() # wollen unseren Optimizer sagen, dass er etwas tun soll
    # das war es mit unserem Trainingsalgorithmus


After converting the pytorch model to coreml, the prediction results are much worse. What do you think, what could be a problem? During conversion I get warnings: WARNING: root: Tuple detected at graph output. This will be flattened in the converted model.
and WARNING: root: Output var reduce_argmax_1 of type i32 in function main is cast to type fp32.
My model:
class Net(nn.Module):
        def __init__(self,NumClasses,PreTrainedModelPath="", UpdateEncoderBatchNormStatistics=True):
            super(Net, self).__init__()
            self.Encoder = densenet_cosine_264_k32.densenet_cosine_264_k32
            if not PreTrainedModelPath=="":
                self.Encoder.load_state_dict(torch.load(PreTrainedModelPath))
                print ("Dense net encoder weights loaded")
            if not UpdateEncoderBatchNormStatistics:
                self.Encoder.eval()
            self.SkipConnectionLayers=[2,12,28,96]#,147]
            self.PSPScales = [1, 1 / 2, 1 / 4, 1 / 8]
            self.PSPLayers = nn.ModuleList()
            for Ps in self.PSPScales:
                self.PSPLayers.append(nn.Sequential(
                    nn.Conv2d(2688, 1024, stride=1, kernel_size=3, padding=1, bias=True)))
            self.PSPSqueeze = nn.Sequential(
                nn.Conv2d(4096, 512, stride=1, kernel_size=1, padding=0, bias=False),
                nn.BatchNorm2d(512),
                nn.ReLU()
            )
            self.SkipConnections = nn.ModuleList()
            self.SkipConnections.append(nn.Sequential(
                nn.Conv2d(1152, 512, stride=1, kernel_size=1, padding=0, bias=False),
                nn.BatchNorm2d(512),
                nn.ReLU()))
            self.SkipConnections.append(nn.Sequential(
                nn.Conv2d(256, 256, stride=1, kernel_size=1, padding=0, bias=False),
                nn.BatchNorm2d(256),
                nn.ReLU()))
            self.SqueezeUpsample = nn.ModuleList()
            self.SqueezeUpsample.append(nn.Sequential(
                nn.Conv2d(1024, 512, stride=1, kernel_size=1, padding=0, bias=False),
                nn.BatchNorm2d(512),
                nn.ReLU()
            ))
            self.SqueezeUpsample.append(nn.Sequential(
                nn.Conv2d(256+512, 256, stride=1, kernel_size=1, padding=0, bias=False),
                nn.BatchNorm2d(256),
                nn.ReLU()
            ))
            self.FinalPrdiction=nn.Conv2d(256, NumClasses, stride=1, kernel_size=3, padding=1, bias=False)
            self=self.cuda()

        def forward(self,Images, EvalMode=False):
                RGBMean = [123.68,116.779,103.939]
                RGBStd = [65,65,65]
                #InpImages = torch.autograd.Variable(torch.from_numpy(Images.astype(float)), requires_grad=False).transpose(2,3).transpose(1, 2).type(torch.FloatTensor)
                #InpImages = torch.autograd.Variable(torch.from_numpy(Images.astype(float)),requires_grad=False).transpose(2, 3).transpose(1, 2).type(torch.HalfTensor)

                InpImages = torch.autograd.Variable(Images)
                InpImages=InpImages.cuda()
                for i in range(len(RGBMean)): InpImages[:, i, :, :]=(InpImages[:, i, :, :]-RGBMean[i])/RGBStd[i]
                x=InpImages
                SkipConFeatures=[]
                for i in range(147):
                    x=self.Encoder[i](x)
                    if i in self.SkipConnectionLayers:
                         SkipConFeatures.append(x)
                PSPSize=(x.shape[2],x.shape[3])
                PSPFeatures=[]
                for i,Layer in enumerate(self.PSPLayers):
                      NewSize=np.ceil(np.array(PSPSize)*self.PSPScales[i]).astype(np.int)
                      y = nn.functional.interpolate(x, tuple(NewSize), mode='bilinear')
                      y = Layer(y)
                      y = nn.functional.interpolate(y, PSPSize, mode='bilinear')
                      PSPFeatures.append(y)
                x=torch.cat(PSPFeatures,dim=1)
                x=self.PSPSqueeze(x)
                for i in range(len(self.SkipConnections)):
                  sp=(SkipConFeatures[-1-i].shape[2],SkipConFeatures[-1-i].shape[3])
                  x=nn.functional.interpolate(x,size=sp,mode='bilinear')
                  x = torch.cat((self.SkipConnections[i](SkipConFeatures[-1-i]),x), dim=1)
                  x = self.SqueezeUpsample[i](x)
                x = self.FinalPrdiction(x)
                x = nn.functional.interpolate(x,size=InpImages.shape[2:4],mode='bilinear')
                Prob=F.softmax(x,dim=1)
                tt,Labels=x.max(1)
                return Prob, Labels

Net = Net(NumClasses=3)
Net.load_state_dict(torch.load(Trained_model_path))
Net.eval()

import coremltools
mlmodel = coremltools.converters.convert(
  traced_model,
  inputs=[coremltools.TensorType(shape=(1, 3, 255, 255))],
)

I look forward to any suggestions


title: When I import torchvision, I get an error that the cuda version of pytorch and torchvision are different

body:
I am getting the following error when importing torchvision.
Detected that PyTorch and torchvision were compiled with different CUDA versions. PyTorch has CUDA Version=11.0 and torchvision has CUDA Version=10.1. Please reinstall the torchvision that matches your PyTorch install.

How can I change the cuda version of pytorch to 10.1?
'conda install pytorch torchvision cudatoolkit=10.1-c pytorch' from anaconda prompt
I get the same error even if I do it.
I am using windows10, and I am using python version 3.7 in the virtual environment of jupyter notebook.


title: accuracy calculation for a binary classification regression model

body:
can someone tell me why my cross entropy Loss function giving this error:
My accuracy method:
def accuracy(outputs, labels):
    _, preds = torch.max(outputs, dim=1)
    return torch.tensor(torch.sum(preds == labels).item() / len(preds))

My class defining the model:
class PulsarLogisticRegression(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear= nn.Linear(input_size,output_size)
    def forward(self,xb):
        xb = xb.view(xb.size(0), -1)
        out= self.linear(xb)
        return out
    def training_step(self, batch):
        inputs, targets = batch
        # Generate predictions
        out = self(inputs)
        # Calcuate loss
        loss = F.cross_entropy(out,targets)
        return loss
    def validation_step(self, batch):
        inputs, targets = batch
        # Generate predictions
        out = self(inputs)
        # Calculate loss
        loss = F.cross_entropy(out,targets)
        acc = accuracy(out, targets)        # Calculate accuracy
        return {'val_loss': loss, 'val_acc': acc} # fill this
    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   # Combine losses
        batch_accs = [x['val_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()      # Combine accuracies
        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}
   def epoch_end(self, epoch, result, num_epochs):
        # Print result every 20th epoch
        print("Epoch [{}], val_loss: {:.4f}, val_acc: {:.4f}".format(epoch, result['val_loss'], result['val_acc']))

error: Cross Entropy Error is expecting 1D target tensor but multi-target is bieng assigned to it.
 RuntimeError                              Traceback (most recent call last)
<ipython-input-88-cd9b8a9a3b02> in <module>()
----> 1 result = evaluate(model, val_loader) # Use the the evaluate function
      2 print(result) 4 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2262                          .format(input.size(0), target.size(0)))
   2263     if dim == 2:
-> 2264         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2265     elif dim == 4:
   2266         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: 1D target tensor expected, multi-target not supported

I am very new to Machine Learning, I am trying to make a model which predicts one column of data based on 5 columns. The values in the column are 0 and 1. So it is basically a binary classification model.
What I tried:
As I said I am fairly new to this field, some explanations suggest to use squeeze function to somehow reduce the shape of target tensor to 1D but that seems to throw some other errors in other methods of the class.
I am looking for an error function which would help me get correct accuracy.


title: How do I plot a contour from a table of values?

body:
I have a table that has 2 features (x,y) - and a vector with the same length that contains their corresponding values (z).
I'm trying to use matplotlib to print this as a 2D plot but I am get an error:
TypeError: Input z must be at least a (2, 2) shaped array, but has shape (5797, 1)
Is there any way to solve this? (since I am trying to use 1d arrays instead of 2d arrays)
The relevant code:
x, y = train_features[:,0], train_features[:,1]
z = train_predictions.detach()
print(x.size())
print(y.size())
print(z.size())

plt.figure()
CS = plt.contour(x, y, z)
CS = plt.contourf(x, y, z)
plt.clabel(CS, fontsize=8, colors='black')
cbar = plt.colorbar(CS)

The prints that result from the prints commands:
torch.Size([5797])
torch.Size([5797])
torch.Size([5797, 1])

EDIT:
I tried to implement this with a second method:
import matplotlib.pyplot as plt
import matplotlib.tri as tri
import numpy as np

npts = 200
ngridx = 100
ngridy = 200
x = train_features[:,0]
y = train_features[:,1]
z = train_predictions.detach().squeeze()

fig, ax1 = plt.subplots()

# -----------------------
# Interpolation on a grid
# -----------------------
# A contour plot of irregularly spaced data coordinates
# via interpolation on a grid.

# Create grid values first.
xi = np.linspace(1, 10, ngridx)
yi = np.linspace(1, 10, ngridy)

# Perform linear interpolation of the data (x,y)
# on a grid defined by (xi,yi)
triang = tri.Triangulation(x, y)
interpolator = tri.LinearTriInterpolator(triang, z)
Xi, Yi = np.meshgrid(xi, yi)
zi = interpolator(Xi, Yi)

ax1.contour(xi, yi, zi, levels=100, linewidths=0.5, colors='k')
cntr1 = ax1.contourf(xi, yi, zi, levels=14, cmap="RdBu_r")

fig.colorbar(cntr1, ax=ax1)
ax1.plot(x, y, 'ko', ms=3)
ax1.set_title('grid and contour (%d points, %d grid points)' %
              (npts, ngridx * ngridy))

But the resulting image was the following:

even though z's values are:
tensor([-0.2434, -0.2155, -0.1900,  ..., 64.7516, 65.2064, 65.6612])


title: Custom small CNN has better accuracy than the pretrained classifiers

body:
I have a dataset of laser welding images of size 300*300 which contains two class of bad and good weld seam. I have followed Pytorch fine-tuning tutorial for an inception-v3 classifier.
on the other hand, I also build a custom CNN with 3 conv layer and 3 fc. What I observed is that the fine tuning showed lots of variation on validation accuracy. basically, I see different maximum accuracy every time I train my model. Plus, my accuracy in fine-tuning is much less than my custom CNN!! for example the accuracy for my synthetic images from a GAN is 86% with inception-v3, while it is 94% with my custom CNN. The real data for both network shows almost similar behaviour and accuracy, however accuracy in custom CNN is about 2% more.
I trained with different training scales of 200, 500 and 1000 train-set images (half of them for each class like for 200 images we have 100 good and 100 bad). I also include a resize transform of 224 in my train_loader; in fine tuning tutorial, this resize is automatically done to 299 for inception-v3. for each trial, the validation-size and its content is constant.
Do you know what cause this behavior? Is it because my dataset is so different from the pretrained model classes? am I not supposed to get better results with fine-tuning?
My custom CNN:
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv3 = nn.Conv2d(16, 24, 5)
        self.fc1 = nn.Linear(13824, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 2)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        #x = x.view(-1, 16 * 5 * 5)
        x = x.view(x.size(0),-1)
        #print(x.shape)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        #x = F.softmax(x, dim=1)
        return x

model = Net()
criterion = nn.CrossEntropyLoss()
#optimizer = optim.Adam(model.parameters(), lr=0.001)
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9, weight_decay=5e-4)
model.to(device)

with training loop of:
epochs = 15
steps = 0
running_loss = 0
print_every = 10
train_losses, test_losses = [], []
train_acc, test_acc = [], []
for epoch in range(epochs):
    for inputs, labels in trainloader:
        steps += 1
        inputs, labels = inputs.to(device), labels.to(device)
        optimizer.zero_grad()
        logps = model.forward(inputs)
        loss = criterion(logps, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

        if steps % print_every == 0:
            test_loss = 0
            accuracy = 0
            model.eval()
            with torch.no_grad():
                for inputs, labels in testloader:
                    inputs, labels = inputs.to(device), labels.to(device)
                    logps = model.forward(inputs)
                    batch_loss = criterion(logps, labels)
                    test_loss += batch_loss.item()

                    ps = torch.exp(logps)
                    top_p, top_class = ps.topk(1, dim=1)
                    equals = top_class == labels.view(*top_class.shape)
                    accuracy += torch.mean(equals.type(torch.FloatTensor)).item()

            train_losses.append(running_loss/len(trainloader))
            test_losses.append(test_loss/len(testloader))

            #train_acc.append(running_loss/len(trainloader))
            test_acc.append(accuracy/len(testloader))
            print(f"Epoch {epoch+1}/{epochs}.. "
                  f"Train loss: {running_loss/print_every:.3f}.. "
                  f"Test loss: {test_loss/len(testloader):.3f}.. "
                  f"Test accuracy: {accuracy/len(testloader):.3f}")
            running_loss = 0
            model.train()


title: 2D times 2D equals a 3d pytorch tensor

body:
Given two 2-D pytorch tensors:
A = torch.FloatTensor([[1,2],[3,4]])
B = torch.FloatTensor([[0,0],[1,1],[2,2]])

Is there an efficient way to calculate a tensor of shape (6, 2, 2) where each entry is a column of A times each row of B?
For example, with A and B above, the 3D tensor should have the following matrices:
[[[0, 0],
  [0, 0]],

 [[1, 1],
  [3, 3]],

 [[2, 2],
  [6, 6]],

 [[0, 0],
  [0, 0]],

 [[2, 2],
  [4, 4]],

 [[4, 4],
  [8, 8]]]

I know how to do it via for-loop but I am wondering if could have an efficient way to save it.


title: Pytorch/cuda : CPU error and map_location

body:
I write this code to download my model :
args = parser.parse_args()

use_cuda = torch.cuda.is_available()

state_dict = torch.load(args.model)
model = Net()
model.load_state_dict(state_dict)
model.eval()

if use_cuda:
    print('Using GPU')
    model.cuda()
else:
    print('Using CPU')

But my terminal returns the following error   RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
So then I tried to write without really understanding too much :
args = parser.parse_args()

map_location=torch.device('cpu')
state_dict = torch.load(args.model)
model = Net()
model.load_state_dict(state_dict)
model.eval()

But I still have the same mistake. Do you see please how I can correct it? (actually I want to load my model with my CPU).


when I am trying to create a DataLoader object with transforms I get this error. I am not sure why there are issues with tensor dimensions as shown here:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-e582b2a128f3> in <module>()
      8 classes = ('none', 'mild', 'moderate', 'severe', 'proliferative')
      9
---> 10 for data, target in train_data_loader:
     11     for i in range(0, 4):
     12         im = data[i]

/share/software/user/open/py-pytorch/1.4.0_py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
    343
    344     def __next__(self):
--> 345         data = self._next_data()
    346         self._num_yielded += 1
    347         if self._dataset_kind == _DatasetKind.Iterable and \

/share/software/user/open/py-pytorch/1.4.0_py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    854             else:
    855                 del self._task_info[idx]
--> 856                 return self._process_data(data)
    857
    858     def _try_put_index(self):

/share/software/user/open/py-pytorch/1.4.0_py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _process_data(self, data)
    879         self._try_put_index()
    880         if isinstance(data, ExceptionWrapper):
--> 881             data.reraise()
    882         return data
    883

/share/software/user/open/py-pytorch/1.4.0_py36/lib/python3.6/site-packages/torch/_utils.py in reraise(self)
    392             # (https://bugs.python.org/issue2651), so we work around it.
    393             msg = KeyErrorMessage(msg)
--> 394         raise self.exc_type(msg)

RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/share/software/user/open/py-pytorch/1.4.0_py36/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/share/software/user/open/py-pytorch/1.4.0_py36/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/share/software/user/open/py-pytorch/1.4.0_py36/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 79, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/share/software/user/open/py-pytorch/1.4.0_py36/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 79, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "/share/software/user/open/py-pytorch/1.4.0_py36/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 634 and 440 in dimension 2 at /pytorch/aten/src/TH/generic/THTensor.cpp:612

Here's how you can reproduce the code. The error comes from creating a DataLoader object and applying the transforms to the images:
def check_cuda():
    is_cuda = False
    if torch.cuda.is_available():
        is_cuda = True
    return is_cuda

def scaleRadius(img, scale):
    x = img[int(img.shape[0] / 2), :, :].sum(1)
    r = (x > x.mean() / 10).sum() / 2
    s = scale * 1.0 / r
    return cv2.resize(img, (0,0), fx=s, fy=s)

class DRDataset(Dataset):
    def __init__(self, csv_path):
        # Transforms
        self.to_tensor = transforms.ToTensor()
        # Read the csv file
        self.data_info = pd.read_csv(csv_path, header=None)
        # First column contains the image paths
        self.image_arr = np.asarray(self.data_info.iloc[:, 0])
        # Second column is the labels
        self.label_arr = np.asarray(self.data_info.iloc[:, 1])
        # Calculate len
        self.data_len = len(self.data_info.index)

    def __getitem__(self, index):
        single_image_name = '/path/to/dataset' + self.image_arr[index] + '.jpeg'
        a = cv2.imread(single_image_name)

        scale = 300
        a = scaleRadius(a, scale)
        a = cv2.addWeighted(a, 4, cv2.GaussianBlur(a, (0,0), scale/30), -4, 128)
        b = np.zeros(a.shape)
        cv2.circle(b, (int(a.shape[1]/2), int(a.shape[0]/2)), int(scale*0.9), (1,1,1), -1, 8, 0)
        a = a*b + 128*(1-b)

        # Transform image to tensor
        img_as_tensor = self.to_tensor(a)

        # Get label(class) of the image based on the cropped pandas column
        single_image_label = self.label_arr[index]

        return (img_as_tensor, single_image_label)

    def __len__(self):
        return self.data_len

full_dataset = DRDataset('/path/to/dataset')

train_size = int(0.7 * len(full_dataset))
val_size = int(0.15 * len(full_dataset))
test_size = len(full_dataset) - train_size - val_size

train_dataset, val_dataset, test_dataset = torch.utils.data.random_split(full_dataset,
                                                                         [train_size, val_size, test_size])

train_data_loader = torch.utils.data.DataLoader(train_dataset, batch_size=8,
                                          shuffle=True, num_workers=4, pin_memory=True)

valid_data_loader = torch.utils.data.DataLoader(val_dataset, batch_size=8,
                                          shuffle=True, num_workers=4, pin_memory=True)

for data, target in train_data_loader:
    for i in range(0, 4):
        im = data[i]
        im = torch.squeeze(im)
        plt.imshow(np.transpose(im.numpy(), (1, 2, 0)), cmap='gray')
        plt.show()
        print(target[i] + ": " + classes[(int)(target[i])])
    break

What am I doing wrong?
--BJ


title: Custom dataset and dataloader

body:
I am new to pytorch .
I have big dataset consist of two txt files one for data and other for target data .
In training file each line is list of length 340, In target each line is list of 136.
I would like to ask how i can define my dataset so I can use Dataloader to load my data to train pytorch model?
I apricate you answers


title: Create a custom dataset from a folder with separate files

body:
I am using Pytorch's custom dataset feature to create a custom dataset from separate files in one folder. Each file contains 123 rows and 123 columns, and all data points are integers.
My issue is that the resources I've come across cater to files in one .csv, mine are not. More so, opening the images after being transformed as an image doesn't run as well. I'm not sure how to proceed from here on as my code gives:
AttributeError: 'Image' object has no attribute 'read'

import os
from torch.utils.data import DataLoader, Dataset
from numpy import genfromtxt

# Custom dataset
class CONCEPTDataset(Dataset):
    """ Concept Dataset """

    def __init__(self, file_dir, transforms=None):
        """
        Args:
            file_dir (string): Directory with all the images.
            transforms (optional): Changes on the data.
        """
        self.file_dir = file_dir
        self.transforms = transforms

        self.concepts = os.listdir(file_dir)
        self.concepts.sort()
        self.concepts = [os.path.join(file_dir, concept) for concept in self.concepts]

    def __len__(self):
        return len(self.concepts)

    def __getitem__(self, idx):
        image = self.concepts[idx]

        # csv file to a numpy array using genfromtxt
        data = genfromtxt(image, delimiter=',')

        data = self.transforms(data.unsqueeze(0))
        return data


title: How to efficiently pairwise pool a tensor by trace value in Pytorch?

body:
I have a pytorch tensor T with shape (batch_size, window_size, filters, 3, 3) and I would like to pool the tensor by trace. Specifically, I would like to obtain a tensor T_pooled of size (batch_size, window_size//2, filters, 3, 3) by comparing the trace of paired frames. For example, if window_size=4, then we would compare the trace of T[i,0,k,3,3] and T[i,1,k,3,3] and select the subtensor with the smaller trace to be T_pooled[i,0,k,3,3]. Similarly, compare T[i,2,k,3,3] and T[i,3,k,3,3] to obtain T_pooled[i,1,k,3,3].
This can be done by looping over i and k, but that is very slow and inefficient. Is there a way to vectorize this pooling operation to speed it up?
Edit:
Here is what I have tried so far. It uses list comprehension and for loops. It takes approximately 2.5s to run on a tensor of size (128,120,22,3,3).
def TPL_Pairwise(x):
    x_pooled=torch.zeros(x.shape[0],x.shape[1]//2,x.shape[2],x.shape[3], x.shape[4])
    #compute tensorized trace
    trace=torch.einsum('ijkll->ijkl', x).sum(-1)
    for i in range(x.shape[0]):
        for j in range(x.shape[2]):
            keep=[ x[i,k,j] if trace[i,k,j] <= trace[i,k+1,j] else x[i,k+1,j] for k in range(0,x.shape[1],2)]
            x_pooled[i,:,j]=torch.stack(keep)
    return x_pooled


title: C++14 : No match for 'operato r='

body:
I am facing this problem ( no match for operator =) with this line and I cannot figure it out.
std ::shared_ptr<torch::jit::script::Module> module;

try {
module = torch::jit::load(argv[1]); //error here
}

please help


title: My OpenCV Live Webcam Demo Doesn't Show Accurate Emotions

body:
I developed a code by following a non-complete tutorial for emotion recognition with using PyTorch. I had so many errors but i fixed them thanks to the other questions here. But i simply stucked at this one. I am running my code, then my webcam opens but i see "Neutral" emotions all the time, even tough i perform different emotions. But on the console i am seeing other emotions as recorded, like:
tensor(0, device='cuda:0')
tensor(6, device='cuda:0')
tensor(4, device='cuda:0')
tensor(4, device='cuda:0')

You have any idea?
Here's my code:
import cv2 ### pip install opencv-python
##pip install open-cv-contrib-python fullpackage
#from deepface import DeepFace ##pip install deepface
import numpy as np

path = "haarcascade_frontalface_default.xml"
font_scale = 1.5
font = cv2.FONT_HERSHEY_PLAIN

#set the rectangle background to white
rectangle_bgr = (255, 255, 255)
# make a black image
img = np.zeros((500, 500))
#set some text
text = "VİDGA Projesi"
#get the width and height of the text box
(text_width, text_height) = cv2.getTextSize(text, font, fontScale=font_scale, thickness=1)[0]
#set the text start position
text_offset_x = 10
text_offset_y = img.shape[0] - 25
#make the coords of the box with a small padding of two pixels
box_coords = ((text_offset_x, text_offset_y), (text_offset_x + text_width + 2, text_offset_y - text_height -2))
cv2.rectangle(img, box_coords[0], box_coords[1], rectangle_bgr, cv2.FILLED)
cv2.putText(img, text, (text_offset_x, text_offset_y), font, fontScale=font_scale, color= (0,0,0), thickness=1)

cap = cv2.VideoCapture(1)
#Check if the webcam is opened correctly
if not cap.isOpened():
    cap = cv2.VideoCapture(0)
if not cap.isOpened():
    raise IOError("Cannot open webcam")

while True:
    ret, frame = cap.read()
    #eye_Cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_eye.xml')
    faceCascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')

    if ret == True:

        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        #print(faceCascade.empty())
        faces = faceCascade.detectMultiScale(gray,1.1,4)
        for x,y,w,h in faces:
            roi_gray = gray[y:y+h, x:x+w]
            roi_color = frame[y:y+h, x:x+w]
            cv2.rectangle(frame, (x,y), (x+w, y+h), (255, 0, 0), 2)
            facess = faceCascade.detectMultiScale(roi_gray)
            if len(facess) == 0:
                print("Face not detected")
            else:
                for(ex,ey,ew,eh) in facess:
                    face_roi = roi_color[ey: ey+eh, ex: ex+ew] ##cropping the face

    graytemp = cv2.cvtColor(face_roi, cv2.COLOR_BGR2GRAY)
    final_image = cv2.resize(graytemp, (48,48))
    final_image = np.expand_dims(final_image, axis =0) #add third dimension
    final_image = np.expand_dims(final_image, axis =0) #add fourth dimension
    final_image = final_image/255.0 # normalization
    dataa = torch.from_numpy(final_image)
    dataa = dataa.type(torch.FloatTensor)
    dataa = dataa.to(device)
    outputs = net(dataa)
    Pred = F.softmax(outputs, dim=1)
    Predictions = torch.argmax(Pred)
    print(Predictions)

    font = cv2.FONT_HERSHEY_SIMPLEX

    font_scale = 1.5
    font = cv2.FONT_HERSHEY_PLAIN

    if ((Predictions)==0):
        status = "Angry"

        x1,y1,w1,h1, = 0,0,175,5
        # Draw black background rectangle
        cv2.rectangle(frame, (x1, x1), (x1 + w1, y1 + h1), (0,0,0), -1)
        # Add text
        cv2.putText(frame, status, (x1 + int(w1/10), y1 + int(h1/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 2)

        cv2.putText(frame, status, (100,150), font, 3,(0,0,255), 2, cv2.LINE_4)

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255))

    elif ((Predictions)==1):
        status = "Disgust"

        x1,y1,w1,h1, = 0,0,175,5
        # Draw black background rectangle
        cv2.rectangle(frame, (x1, x1), (x1 + w1, y1 + h1), (0,0,0), -1)
        # Add text
        cv2.putText(frame, status, (x1 + int(w1/10), y1 + int(h1/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 2)

        cv2.putText(frame, status, (100,150), font, 3,(0,0,255), 2, cv2.LINE_4)

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255))

    elif ((Predictions)==2):
        status = "Fear"

        x1,y1,w1,h1, = 0,0,175,5
        # Draw black background rectangle
        cv2.rectangle(frame, (x1, x1), (x1 + w1, y1 + h1), (0,0,0), -1)
        # Add text
        cv2.putText(frame, status, (x1 + int(w1/10), y1 + int(h1/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 2)

        cv2.putText(frame, status, (100,150), font, 3,(0,0,255), 2, cv2.LINE_4)

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255))

    elif ((Predictions)==3):
        status = "Happy"

        x1,y1,w1,h1, = 0,0,175,5
        # Draw black background rectangle
        cv2.rectangle(frame, (x1, x1), (x1 + w1, y1 + h1), (0,0,0), -1)
        # Add text
        cv2.putText(frame, status, (x1 + int(w1/10), y1 + int(h1/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 2)

        cv2.putText(frame, status, (100,150), font, 3,(0,0,255), 2, cv2.LINE_4)

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255))

    elif ((Predictions)==4):
        status = "Sad"

        x1,y1,w1,h1, = 0,0,175,5
        # Draw black background rectangle
        cv2.rectangle(frame, (x1, x1), (x1 + w1, y1 + h1), (0,0,0), -1)
        # Add text
        cv2.putText(frame, status, (x1 + int(w1/10), y1 + int(h1/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 2)

        cv2.putText(frame, status, (100,150), font, 3,(0,0,255), 2, cv2.LINE_4)

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255))

    elif ((Predictions)==5):
        status = "Surprised"

        x1,y1,w1,h1, = 0,0,175,5
        # Draw black background rectangle
        cv2.rectangle(frame, (x1, x1), (x1 + w1, y1 + h1), (0,0,0), -1)
        # Add text
        cv2.putText(frame, status, (x1 + int(w1/10), y1 + int(h1/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 2)

        cv2.putText(frame, status, (100,150), font, 3,(0,0,255), 2, cv2.LINE_4)

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255))

    elif ((Predictions)==6):
        status = "Neutral"

        x1,y1,w1,h1, = 0,0,175,5
        # Draw black background rectangle
        cv2.rectangle(frame, (x1, x1), (x1 + w1, y1 + h1), (0,0,0), -1)
        # Add text
        cv2.putText(frame, status, (x1 + int(w1/10), y1 + int(h1/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 2)

        cv2.putText(frame, status, (100,150), font, 3,(0,0,255), 2, cv2.LINE_4)

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255))

        if ret == True:
            cv2.imshow('VIDGA Emotion Recognition', frame)
            if cv2.waitKey(2) & 0xFF == ord('q'):
                break

cap.release()
cv2.destroyAllWindows


title: PyTorch How to code Multi Head Self Attention in parallel?

body:
I want to encode the word (embedding) sequence with 16-Head Self-Attention.
Currently I use a nn.ModuleList together with a for loop to generate the output of each head then concatenate all of them. This approach is extremely slow and I wonder if there's way to code MHA in parallel?
To generalize, I would like to know if I can 'stack' multiple nn.Linear, as I feed the input vector, the multiple outputs will be computed in parallel.


title: Is torch.float32 different from numpy's float32?

body:
Setting precision as 30 in PyTorch shows:
>>> torch.set_printoptions(precision=30)
>>> y
tensor([[-0.388252139091491699218750000000, -0.610148549079895019531250000000,
         -1.333969473838806152343750000000, -1.027917861938476562500000000000,
         -0.498563587665557861328125000000, -0.096793495118618011474609375000,
         -0.895992159843444824218750000000, -0.752071321010589599609375000000,
         -0.879653215408325195312500000000],
        [ 1.960780262947082519531250000000,  0.290681242942810058593750000000,
          0.111534759402275085449218750000, -1.412155270576477050781250000000,
          1.015806078910827636718750000000,  0.201809123158454895019531250000,
          0.131465137004852294921875000000, -1.262379050254821777343750000000,
         -0.480409622192382812500000000000],
        [ 0.111068181693553924560546875000, -2.129202365875244140625000000000,
          0.538800299167633056640625000000,  1.165832757949829101562500000000,
          0.194993987679481506347656250000, -1.110693812370300292968750000000,
         -1.451576709747314453125000000000, -3.398952484130859375000000000000,
          2.022404193878173828125000000000],
        [-2.374018669128417968750000000000, -1.442466974258422851562500000000,
         -0.406166225671768188476562500000,  0.045908458530902862548828125000,
         -0.835370421409606933593750000000, -0.302138328552246093750000000000,
         -0.421340197324752807617187500000,  0.931307554244995117187500000000,
          1.061386585235595703125000000000],
        [ 1.166660070419311523437500000000,  0.710260510444641113281250000000,
          1.008558034896850585937500000000,  1.957847237586975097656250000000,
         -1.070753335952758789062500000000,  0.319442749023437500000000000000,
         -1.140496969223022460937500000000, -1.723430752754211425781250000000,
          0.109533369541168212890625000000],
        [ 0.501820147037506103515625000000,  0.349833250045776367187500000000,
         -0.073374643921852111816406250000, -2.459295272827148437500000000000,
         -1.853959321975708007812500000000,  0.153838425874710083007812500000,
         -1.860147237777709960937500000000, -0.880943417549133300781250000000,
         -1.352821707725524902343750000000]])

but in numpy setting precision as 30 shows:
>>> np.set_printoptions(precision=30)
>>> y.numpy()
array([[-0.38825214 , -0.61014855 , -1.3339695  , -1.0279179  ,
        -0.4985636  , -0.096793495, -0.89599216 , -0.7520713  ,
        -0.8796532  ],
       [ 1.9607803  ,  0.29068124 ,  0.11153476 , -1.4121553  ,
         1.0158061  ,  0.20180912 ,  0.13146514 , -1.262379   ,
        -0.48040962 ],
       [ 0.11106818 , -2.1292024  ,  0.5388003  ,  1.1658328  ,
         0.19499399 , -1.1106938  , -1.4515767  , -3.3989525  ,
         2.0224042  ],
       [-2.3740187  , -1.442467   , -0.40616623 ,  0.04590846 ,
        -0.8353704  , -0.30213833 , -0.4213402  ,  0.93130755 ,
         1.0613866  ],
       [ 1.1666601  ,  0.7102605  ,  1.008558   ,  1.9578472  ,
        -1.0707533  ,  0.31944275 , -1.140497   , -1.7234308  ,
         0.10953337 ],
       [ 0.50182015 ,  0.34983325 , -0.073374644, -2.4592953  ,
        -1.8539593  ,  0.15383843 , -1.8601472  , -0.8809434  ,
        -1.3528217  ]], dtype=float32)e

Why are the results different?


title: PyTorch tensors topk for every tensor across a dimension

body:
I have the following tensor
inp = tensor([[[ 0.0000e+00,  5.7100e+02, -6.9846e+00],
     [ 0.0000e+00,  4.4070e+03, -7.1008e+00],
     [ 0.0000e+00,  3.0300e+02, -7.2226e+00],
     [ 0.0000e+00,  6.8000e+01, -7.2777e+00],
     [ 1.0000e+00,  5.7100e+02, -6.9846e+00],
     [ 1.0000e+00,  4.4070e+03, -7.1008e+00],
     [ 1.0000e+00,  3.0300e+02, -7.2226e+00],
     [ 1.0000e+00,  6.8000e+01, -7.2777e+00]],

    [[ 0.0000e+00,  2.1610e+03, -7.0754e+00],
     [ 0.0000e+00,  6.8000e+01, -7.2259e+00],
     [ 0.0000e+00,  1.0620e+03, -7.2920e+00],
     [ 0.0000e+00,  2.9330e+03, -7.3009e+00],
     [ 1.0000e+00,  2.1610e+03, -7.0754e+00],
     [ 1.0000e+00,  6.8000e+01, -7.2259e+00],
     [ 1.0000e+00,  1.0620e+03, -7.2920e+00],
     [ 1.0000e+00,  2.9330e+03, -7.3009e+00]],

    [[ 0.0000e+00,  4.4070e+03, -7.1947e+00],
     [ 0.0000e+00,  3.5600e+02, -7.2958e+00],
     [ 0.0000e+00,  3.0300e+02, -7.3232e+00],
     [ 0.0000e+00,  1.2910e+03, -7.3615e+00],
     [ 1.0000e+00,  4.4070e+03, -7.1947e+00],
     [ 1.0000e+00,  3.5600e+02, -7.2958e+00],
     [ 1.0000e+00,  3.0300e+02, -7.3232e+00],
     [ 1.0000e+00,  1.2910e+03, -7.3615e+00]]])

of shape
torch.Size([3, 8, 3])

and I would like to find the topk(k=4) elements across dim1, where the value to sort by is dim2 (the negative values). The resulting tensor shape should then be:
torch.Size([3, 4, 3])

I know how to do topk for a single tensor, but how do I do this for several batches at once?


title: Meaning of each entry in torch.autograd.profiler.profile

body:
All
I just try using the torch.autograd.profiler.profile
There are several entries
Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls

where could I find the explanation for these? does the time calculated in CPU total also overlap with the time in CUDA total if an operator involves the GPU computation?
Thanks!


title: PyTorch ValueError: Target size (torch.Size([64])) must be the same as input size (torch.Size([15]))

body:
I'm currently using this repo to perform NLP and learn more about CNN's using my own dataset, and I keep running into an error regarding a shape mismatch:
ValueError: Target size (torch.Size([64])) must be the same as input size (torch.Size([15]))

     10 }
     11 for epoch in tqdm(range(params['epochs'])):
---> 12     train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
     13     valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
     14     epoch_mins, epoch_secs = epoch_time(start_time, end_time)

     57         print("PredictionShapeAfter:")
     58         print(predictions.shape)
---> 59         loss = criterion(predictions, batch.l)
     60
     61         acc = binary_accuracy(predictions, batch.l)

Doing some digging, I found that my CNN's prediction is a different size compared to the training data truth it's being compared to:
  Input Shape:
    torch.Size([15, 64])
    Truth Shape:
    torch.Size([64])
    embedded unsqueezed: torch.Size([15, 1, 64, 100])
    cat shape: torch.Size([15, 300])
    Prediction Shape Before Squeeze:
    torch.Size([15, 1])
    PredictionShapeAfter:
    torch.Size([15])

The model is making the prediction shape (the last value in this list) as the first dimension of the inputs. Is this a common problem and is there a way to rectify this issue?
My Model:
class CNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim,
                 dropout, pad_idx):
        super().__init__()

        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx = pad_idx)

        self.convs = nn.ModuleList([
                                    nn.Conv2d(in_channels = 1,
                                              out_channels = n_filters,
                                              kernel_size = (fs, embedding_dim))
                                    for fs in filter_sizes
                                    ])

        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)

        self.dropout = nn.Dropout(dropout)

    def forward(self, text):
        embedded = self.embedding(text)
        embedded = embedded.unsqueeze(1)
        print(f"embedded unsqueezed: {embedded.shape}")
        conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]
        pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved]
        cat = self.dropout(torch.cat(pooled, dim = 1))
        print(f"cat shape: {cat.shape}")
        return self.fc(cat)

My Training function:
def train(model, iterator, optimizer, criterion):

    epoch_loss = 0
    epoch_acc = 0

    model.train()

    for batch in iterator:

        optimizer.zero_grad()

        print("InputShape:")
        print(batch.t.shape)
        print("Truth Shape:")
        print(batch.l.shape)

        predictions = model(batch.t)
        print("Prediction Shape Before Squeeze:")
        print(predictions.shape)

        predictions = predictions.squeeze(1)
        print("PredictionShapeAfter:")
        print(predictions.shape)
        loss = criterion(predictions, batch.l)

        acc = binary_accuracy(predictions, batch.l)

        loss.backward()

        optimizer.step()

        epoch_loss += loss.item()
        epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

My full code can be found at this link.


title: Target value with torch.nn.MultiLabelSoftMarginLoss should be 0 or -1?

body:
I have a multi-label classification problem (A single sample can be classified as several classes at the same time).
I want to use torch.nn.MultiLabelSoftMarginLoss but I got confused with the documentation where the ground truth are written like this :
Target: (N, C)(N,C) , label targets padded by -1 ensuring same shape as the input.

Does that mean the target is in one hot form, but the zero replace with -1?
Let's say I want to classified several attributes for object detection such as : Man, Tall, Long hair.
My first image is a tall woman with long hair, does my Target become 0 1 1 or -1 1 1 ? I can't fathom why use -1 instead of 0
It's quite hard to find example in internet since a lot of people mistook multi-label task as multiple class classification and keep using BCELoss.


title: what is wrong with the following implementation of Conv1d?

body:
I am trying to implement a Conv1d layer with Batch Normalization but I keep getting the following error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-32-ef6e122ea50c> in <module>()
----> 1 test()
      2 for epoch in range(1, n_epochs + 1):
      3   train(epoch)
      4   test()

7 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    258                             _single(0), self.dilation, self.groups)
    259         return F.conv1d(input, weight, bias, self.stride,
--> 260                         self.padding, self.dilation, self.groups)
    261
    262     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Expected 3-dimensional input for 3-dimensional weight [25, 40, 5], but got 2-dimensional input of size [32, 40] instead

The data is passed on in batches of 32 using DataLoader class and it has 40 features and 10 labels. Here is my model:
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        #self.flatten=nn.Flatten()
        self.net_stack=nn.Sequential(
            nn.Conv1d(in_channels=40, out_channels=25, kernel_size=5, stride=2), #applying batch norm
            nn.ReLU(),
            nn.BatchNorm1d(25, affine=True),
            nn.Conv1d(in_channels=25, out_channels=20, kernel_size=5, stride=2), #applying batch norm
            nn.ReLU(),
            nn.BatchNorm1d(20, affine=True),
            nn.Linear(20, 10),
            nn.Softmax(dim=1))

    def forward(self,x):
        #x=torch.reshape(x, (1,-1))
        result=self.net_stack(x)
        return result

I have tried given in other answers like unsqueezing the input tensor, but none of the models in such questions is using Conv1d with batchnorm1d so I am not able to narrow down the problem to which layer must be causing the error. I have just started with using Pytorch and was able to implement a simple linear NN model, but I am facing this error while using a convolutional NN for the same data.


title: How to implement distinctiveness pruning to my neural network built by PyTorch?

body:
I built a neural network with PyTorch to predict the final marks based on students' assignment marks, lab marks, etc.
Although PyTorch has some built-in pruning modules, I am intended to implement distinctiveness pruning to my network, which prunes neurons based on the cosine similarity of pairs of neurons.
I'm really new to deep learning. In fact, I'm not even very familiar with Python itself, but I'm actively learning it as hard as I can. I did a lot of research on the Internet, I now understand the theories and mechanism behind distinctiveness pruning but fail to code it out in Python.
Could you guys please provide me with some help? Anything from pseudo code to a more detailed explanation would be much appreciated.
I think my main problem is that I don't know how to get the activation vector (the output of a neuron as a vector) of each neuron, how to add the weights of one neuron to another if they are similar or get rid of both neurons if they do the opposite thing, how to integrate this pruning process to my existing neural network.
My existing neural network is provided as below:
class Mark_Predict(nn.Module):
    def __init__(self, input_size, hidden_size1, hidden_size2, num_classes):
        super().__init__()
        self.hidden1 = nn.Linear(input_size, hidden_size1)
        self.hidden2 = nn.Linear(hidden_size1, hidden_size2)
        self.relu = nn.ReLU()
        self.out = nn.Linear(hidden_size2, num_classes)

    def forward(self, x):
        out = self.hidden1(x)
        out = self.relu(out)
        out = self.hidden2(out)
        out = self.relu(out)
        out = self.out(out)
        return out

I have 14 features, and the output is either 0, 1, 2, or 3 (A, B, C, or D as the final mark).
I used CrossEntropyLoss as my loss function and Adam as my optimizer.
I implemented 5-Fold cross-validation, and the current average test accuracy is around 70%. I'm looking forward to seeing the outcome after pruning.


title: Correct Validation Loss in Pytorch?

body:
I am a bit confused as to how to calculate Validation loss? Are validation loss to be computed at the end of an epoch OR should the loss be also monitored during iteration through the batches ?
Below I have computer using running_loss which is getting accumulated over batches - but I want to see if its the correct approach?
def validate(loader, model, criterion):
    correct = 0
    total = 0
    running_loss = 0.0
    model.eval()
    with torch.no_grad():
        for i, data in enumerate(loader):
            inputs, labels = data
            inputs = inputs.to(device)
            labels = labels.to(device)

            outputs = model(inputs)
            loss = criterion(outputs, labels)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
            running_loss = running_loss + loss.item()
    mean_val_accuracy = (100 * correct / total)
    mean_val_loss = ( running_loss )
    #mean_val_accuracy = accuracy(outputs,labels)
    print('Validation Accuracy: %d %%' % (mean_val_accuracy))
    print('Validation Loss:'  ,mean_val_loss )

Below is the training block I am using
def train(loader, model, criterion, optimizer, epoch):
    correct = 0
    running_loss = 0.0
    i_max = 0
    for i, data in enumerate(loader):
        total_loss = 0.0
        #print('batch=',i)
        inputs, labels = data
        inputs = inputs.to(device)
        labels = labels.to(device)

        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        if i % 2000 == 1999:
            print('[%d , %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

    print('finished training')
    return mean_val_loss, mean_val_accuracy


title: How to construct batch that return equal number of images for per classes

body:
I am working an image retrieval project, for making model more fair i want to construct batches that return:

5 images per class, and
75 images and per batch

I have total 300 classes in my dataset, so it obvious that only 15 classes of images can be contained in each batch.data is balanced this mean there is equal number of images for per class,I am using pytorch.
I have create pytorch dataset and I want to add above functionality in my ImageFolderLoader class whose code I added below.
IMG_EXTENSIONS = [
   '.jpg', '.JPG', '.jpeg', '.JPEG',
   '.png', '.PNG', '.ppm', '.PPM', '.bmp', '.BMP',
]

def is_image_file(filename):
    return any(filename.endswith(extension) for extension in IMG_EXTENSIONS)

def find_classes(dir):
    classes = os.listdir(dir)
    classes.sort()
    class_to_idx = {classes[i]: i for i in range(len(classes))}
    classes = [clss.split('.')[1] for clss in classes]
    return classes, class_to_idx

def make_dataset(dir, class_to_idx):
    images = []
    for target in os.listdir(dir):
        d = os.path.join(dir, target)
        if not os.path.isdir(d):
            continue
        for filename in os.listdir(d):
            if is_image_file(filename):
                path = '{0}/{1}'.format(target, filename)
                item = (path, class_to_idx[target])
                images.append(item)

    return images

def default_loader(path):
    return Image.open(path).convert('RGB')

class ImageFolderLoader(Dataset):
    def __init__(self, root, transform=None, loader=default_loader,):
        classes, class_to_idx = find_classes(root)
        imgs = make_dataset(root, class_to_idx)

        self.root = root
        self.imgs = imgs
        self.classes = classes
        self.class_to_idx = class_to_idx
        self.transform = transform
        self.loader = loader

    def __getitem__(self, index):
        path, target = self.imgs[index]
        img = self.loader(os.path.join(self.root, path))
        if self.transform is not None:
            img = self.transform(img)

        return img, target

    def __len__(self):
        return len(self.imgs)

if there is way to do this then please let me know>.
edit:- Anyone want to see solution for this, i added the solution below after solving this problem.


title: CUDA Undefined Symbols Error for nvrtc related samples

body:
The following error occurs when I try to test on the CUDA installation samples which come with the installation (e.g. vectorAdd_nvrtc):

clang++ -I../../common/inc -I/Developer/NVIDIA/CUDA-9.2/include  -arch x86_64  -o vectorAdd.o -c vectorAdd.cpp
clang++  -rpath /Developer/NVIDIA/CUDA-9.2/lib -o vectorAdd_nvrtc
vectorAdd.o  -L/Developer/NVIDIA/CUDA-9.2/lib -lnvrtc
Undefined symbols for architecture x86_64:
  "_cuCtxCreate_v2", referenced from:
      loadPTX(char*, int, char**) in vectorAdd.o
  "_cuCtxSynchronize", referenced from:
      _main in vectorAdd.o
  "_cuDeviceGet", referenced from:
      loadPTX(char*, int, char**) in vectorAdd.o
      findCudaDeviceDRV(int, char const**) in vectorAdd.o
      gpuDeviceInitDRV(int, char const**) in vectorAdd.o
  "_cuDeviceGetAttribute", referenced from:
      loadPTX(char*, int, char**) in vectorAdd.o
      gpuGetMaxGflopsDeviceIdDRV() in vectorAdd.o
      void getCudaAttribute<int>(int*, CUdevice_attribute_enum, int) in vectorAdd.o
  "_cuDeviceGetCount", referenced from:
      gpuDeviceInitDRV(int, char const**) in vectorAdd.o
      gpuGetMaxGflopsDeviceIdDRV() in vectorAdd.o
  "_cuDeviceGetName", referenced from:
      loadPTX(char*, int, char**) in vectorAdd.o
      findCudaDeviceDRV(int, char const**) in vectorAdd.o
      gpuDeviceInitDRV(int, char const**) in vectorAdd.o
  "_cuInit", referenced from:
      loadPTX(char*, int, char**) in vectorAdd.o
      gpuDeviceInitDRV(int, char const**) in vectorAdd.o
      gpuGetMaxGflopsDeviceIdDRV() in vectorAdd.o
  "_cuLaunchKernel", referenced from:
      _main in vectorAdd.o
  "_cuMemAlloc_v2", referenced from:
      _main in vectorAdd.o
  "_cuMemFree_v2", referenced from:
      _main in vectorAdd.o
  "_cuMemcpyDtoH_v2", referenced from:
      _main in vectorAdd.o
  "_cuMemcpyHtoD_v2", referenced from:
      _main in vectorAdd.o
  "_cuModuleGetFunction", referenced from:
      _main in vectorAdd.o
  "_cuModuleLoadDataEx", referenced from:
      loadPTX(char*, int, char**) in vectorAdd.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [vectorAdd_nvrtc] Error 1

However, this only happens when the runtime lib nvrtc is utilized, i.e. vectorAdd will build just fine. I tried with several different samples (e.g. clock vs clock_nvrtc, etc.) and obversed the same outcome.

I researched a little online and it seems like to be related to some header file include problem, but I am not very familiar with c++ and have no idea how to fix this. I do see that in the Makefile, it states that nvrtc is not supported on ARMv7 systems, but I do not think that is the issue since my architecture is x86_64? Any help is greatly appreciated!

(CUDA is of version 9.2. I am running on MAC OS 10.13.6, with Xcode 9.2, Apple LLVM version 9.0.0. I need CUDA for pytorch, not sure if this is an relevant issue though.)


title: Pytorch parameter matrix from loss function of transformation

body: I have a pytorch tensor k x (n+k-1) tensor w with requires_grad=True. I want to transform it into a kxn tensor p also with as such: p[i] = w[i][i:i+n]. How do I do this, such that by calling backward() on a loss function of p in the end, I will learn w?


title: Unexpected result of convolution operation

body:
Here is code I wrote to perform a single convolution and output the shape.

Using formula from http://cs231n.github.io/convolutional-networks/ to calculate output size :

  You can convince yourself that the correct formula for calculating how
  many neurons “fit” is given by (W−F+2P)/S+1

The formula for computing the output size has been implemented below as

def output_size(w , f , stride , padding) :
        return (((w - f) + (2 * padding)) / stride) + 1

The issue is output_size computes a size of 2690.5 which differs to the result of the convolution which is 1350 :

%reset -f

import torch
import torch.nn.functional as F
import numpy as np
from PIL import Image
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
from pylab import plt
plt.style.use('seaborn')
%matplotlib inline

width = 60
height = 30
kernel_size_param = 5
stride_param = 2
padding_param = 2

img = Image.new('RGB', (width, height), color = 'red')

in_channels = 3
out_channels = 3

class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(in_channels,
                      out_channels,
                      kernel_size=kernel_size_param,
                      stride=stride_param,
                      padding=padding_param))

    def forward(self, x):
        out = self.layer1(x)

        return out

# w : input volume size
# f : receptive field size of the Conv Layer neurons
# output_size computes spatial size of output volume - spatial dimensions are (width, height)
def output_size(w , f , stride , padding) :
    return (((w - f) + (2 * padding)) / stride) + 1

w = width * height * in_channels
f = kernel_size_param * kernel_size_param

print('output size :' , output_size(w , f , stride_param , padding_param))

model = ConvNet()

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=.001)

img_a = np.array(img)
img_pt = torch.tensor(img_a).float()
result = model(img_pt.view(3, width , height).unsqueeze_(0))
an = result.view(30 , 15 , out_channels).data.numpy()

# print(result.shape)
# print(an.shape)

# print(np.amin(an.flatten('F')))

print(30 * 15 * out_channels)

Have I implemented output_size correctly ? How to amend this model so the result of Conv2d has same shape as result of output_size ?


title: Unable to get an image in the output when using MNIST Database using Pytorch

body:
iter_test = 0

for images, labels in test_loader:

iter_test += 1

images = images.view(-1, 28*28)

outputs = model(images)
_, predicted = torch.max(outputs.data, 1)

if iter_test == 1:
    print('PREDICTION')
    print(predicted[0])

    print('LABEL SIZE')
    print(labels.size())

    print('LABEL FOR IMAGE 0')
    print(labels[0])

I get the output of this code as follows:

    PREDICTION

    tensor(7)

    LABEL SIZE

    torch.Size([100])

    LABEL FOR IMAGE 0

    tensor(7)

The code works perfectly. I was wondering if i could get the "MNIST" like image as output along with prediction?


title: Get corner of rectangle near to origin in batch of tensor given any two diagonal coordinates in pytorch

body:
Let's say I have pytorch tensor of batch of coordinates of off diagonal elements and I want to get coordinate of the corner which is near to origin. coordinates are in (x1, y1, x2, y2) form.

a = torch.tensor([[3,2,2,3], [1,1,2,2])
# expected output
[[2,2], [1,1]]


title: Can A3C be applied for training TD3?

body:
How to use the Asynchronous RL Framework(like A3C) to train the DDPG or TD3?

I have tried each actor-learner as a DDPG agent and asynchronous update the global DDPG agent, but it didn't work.


title: Combine index and value of tenor to from a new tensor

body:
I have a  tensor like a = torch.tensor([1,2,0,1,2]). I want to calculate a tensor b which has indices and values of tensor a such that:

b = tensor([ [0,1], [1,2], [2,0], [3,1], [4,2] ]).

Edit: a[i] is >=  0.


title: How do I reshape this tensor?

body:
I have a torch.tensor that looks like this:

tensor([[[A,B,C],
         [D,E,F],
         [G,H,I]],

        [[J,K,L],
         [M,N,O],
         [P,Q,R]]]

I want to reshape this tensor so that its dimensions are (18, 1). I want the new tensor to look like this:

tensor([A,
        J,
        B,
        K,
        C,
        L,
        D,
        M,
       ...
        I,
        R])

I've tried tensor.view(-1,1) but this doesn't work..


title: Conv2d and the value of padding

body:
I would like to ask about the value of padding we set in the Conv2d function.

I know what zero-padding is. However, what does it mean that the padding is 0 for instance, or 1,2, or 3. What do these values mean? Do they represent the number of columns and rows that will be filled with zeros?

Thanks a lot.


title: Neural Network for bit operation AND

body:
This is probably not the best place to ask this question but I am really struggling.
I want to create a "trivial" neural network which has 2 inputs, 3 hidden neurons and 1 output. The idea is that we feed it two booleans and it outputs the AND result.

I wanted to do it in a somewhat "conventional" way (I am still very new to pytorch). I'm using the Adam optimiser with a learning rate of 0.001 and the nll_loss function (although I need to change it as I need to have only 1 output and this requires the number of outputs to match the number of classes - in this case 1 and 0)

I know this should be a really trivial problem but I am really struggling and can't find anything useful on google.


title: Is it possible to store data in multi-GPUs and calculate the average still on GPU with Pytorch?

body:
I'm learning pytorch recently, and this question comes up to me.

When referring to multi-GPUs, I can see many tutorials about 'DataParallel'. It splits the calculation for different GPUs, then merge the results.

How about this question? I want data stored in different GPUs, then calculate their average still on GPU. For example, if I have 4 tensors with the same shape:

tensor_gpu1=tensor_c1.to(device1)
tensor_gpu2=tensor_c2.to(device2)
tensor_gpu3=tensor_c3.to(device3)
tensor_gpu4=tensor_c4.to(device4)
##then I want the average still calculated on GPU (which means no communication with CPU)
##Will it work if I just sum up and divide?
result=(tensor_gpu1+tensor_gpu2+tensor_gpu3+tensor_gpu4)/4 ##Try to add tensors on different GPU.

After reading some more turorials, now I'm wondering whether this will work. It's said that trying to add a CPU tensor and a GPU tensor will raise up an error. How about tensors on different GPU? It will be a few days before I can get access to an available GPU, so I can't try it out now.


title: Getting the same result at np.gradient with pytorch

body: I'm new to pytorch and I want to calculate the gradient of an image (nrows,ncolumns).  However, torch.grad returns None.  I don't want to backpropagate this gradient, I just want it on the initial image, since I'm implementing the demons algorithm.  How can I get the same type of return as np.gradient()?


title: function as argument of another function

body:
I have a code that in part of it a function is passed to another function as an argument. when I run this code I get this error:

h() missing 1 required positional argument: 'id'

In this part I call h function as a parameter of another function(train of engine class):

engine.train(h, train_loader, opt.epochs, optimizer)

and h is this piece of code:

def h(sample, id):
    inputs = utils.cast(sample[0], opt.dtype).detach()
    targets = utils.cast(sample[1], 'long')
    if opt.teacher_id != '':
        if id==1:
            y_s, y_t, loss_groups = utils.data_parallel(f, inputs, params, sample[2], range(opt.ngpu))
            loss_groups = [v.sum() for v in loss_groups]
            [m.add(v.item()) for m, v in zip(meters_at, loss_groups)]
            return utils.distillation(y_s, y_t, targets, opt.temperature, opt.alpha)+sum(loss_groups), y_s

        if id==2:
            y_s1, y_t1, loss_groups1 = utils.data_parallel(f1, inputs, params1, sample[2], range(opt.ngpu))
            loss_groups1 = [v.sum() for v in loss_groups1]
            [m.add(v.item()) for m, v in zip(meters_at, loss_groups1)]
            return utils.distillation(y_s1, y_t1, targets, opt.temperature, opt.alpha)+sum(loss_groups1), y_s1

        if id==3:
            y_s2, y_t2, loss_groups2 = utils.data_parallel(f2, inputs, params2, sample[2], range(opt.ngpu))
            loss_groups2 = [v.sum() for v in loss_groups2]
            [m.add(v.item()) for m, v in zip(meters_at, loss_groups2)]
            return utils.distillation(y_s2, y_t2, targets, opt.temperature, opt.alpha)+sum(loss_groups2), y_s2

    else:
        if id==1:
            y = utils.data_parallel(f, inputs, params, sample[2], range(opt.ngpu))[0]
        if id==2:
            y = utils.data_parallel(f1, inputs, params1, sample[2], range(opt.ngpu))[0]
        if id==3:
            y = utils.data_parallel(f2, inputs, params2, sample[2], range(opt.ngpu))[0]

        return F.cross_entropy(y, targets), y


title: When trying to use torch's rnn.py, AttributeError: 'builtin_function_or_method' object has no attribute 'size'

body:
Im getting this error when I run the model I'm using.  This is the code that seems to be causing the issue?

  def forward(self, b, xc, xw, lens):
        self.hidden = self.init_state(b)
        x = self.embed(xc, xw)
        x = nn.utils.rnn.pack_padded_sequence(x, lens, batch_first = True)
        h, _ = self.rnn(x, self.hidden)
        h, _ = nn.utils.rnn.pad_packed_sequence(h, batch_first = True)
        return h

Here is the traceback

Traceback (most recent call last):
  File "train.py", line 59, in <module>
    train()
  File "train.py", line 44, in train
    loss = model(xc, xw, y0) # forward pass and compute loss
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/sidu/Downloads/rnn-encoder-decoder-pytorch/model.py", line 18, in forward
    self.dec.hs = self.enc(b, xc, xw, lens)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/sidu/Downloads/rnn-encoder-decoder-pytorch/model.py", line 64, in forward
    h, _ = self.rnn(x, self.hidden)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 567, in forward
    self.check_forward_args(input, hx, batch_sizes)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 522, in check_forward_args
    self.check_hidden_size(hidden[0], expected_hidden_size,
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 186, in check_hidden_size
    if hx.size() != expected_hidden_size:
AttributeError: 'builtin_function_or_method' object has no attribute 'size'

I can't seem to figure out where this error is stemming from, probably due to my inexperience with this stuff.  Can anyone see the issue?  Let me know if there's anything else I need to provide


title: Pytorch: Devolving into nan Batch losses/Log probs

body:
I'm using pytorch as the basis for some machine translation tasks I want to do.
When instantiating my dataset, I have as each torchtext.data.Example a [list(untokenized_string_sentence), untokenized_string_sentence, untokenized_string_sentence]
where untokenized_string_sentence is a just a raw example sentence. I've made sure to be careful about handling the first input as a list in some custom forward functions. However, I don't know if the batch loss computation is treating it accordingly, or if I need to be careful about this. I'm simply calling nn.CrossEntropyLoss() with label smoothing and padding. The result after a couple of batches is a completely NaN forward pass (and subsequently, NaN softmax values and loss).

Does anyone know if there is issue in differing types of inputs using PyTorch?

Here are how my types are defined:

    src_field = data.Field(init_token=None, eos_token=EOS_TOKEN,
                           pad_token=PAD_TOKEN, tokenize=tok_fun,
                           batch_first=True, lower=lowercase,
                           unk_token=UNK_TOKEN,
                           include_lengths=True)

    trg_field = data.Field(init_token=BOS_TOKEN, eos_token=EOS_TOKEN,
                           pad_token=PAD_TOKEN, tokenize=tok_fun,
                           unk_token=UNK_TOKEN,
                           batch_first=True, lower=lowercase,
                           include_lengths=True)
    prev_inputs_field = data.NestedField(data.Field(init_token=CONTEXT_TOKEN, eos_token=CONTEXT_EOS_TOKEN,
                           pad_token=PAD_TOKEN, tokenize=tok_fun,
                           batch_first=True, lower=lowercase,
                           unk_token=UNK_TOKEN,
                           include_lengths=False), tokenize=sent_fun)


title: What does array[...,list([something]) mean?

body:
I am going through the following lines of code but I didn't understand image[...,list()]. What do the three dots mean?
self.probability = 0.5
self.indices = list(permutations(range(3), 3))
if random.random() < self.probability:
            image = np.asarray(image)
            image = Image.fromarray(image[...,list(self.indices[random.randint(0, len(self.indices) - 1)])])

What exactly is happening in the above lines?
I have understood that the list() part is taking random channels from image? Am I correct?


title: Convtranspose2D in tensorflow

body:
I have to use ConvTranspose2d for an input noise of (16,100) of my GAN  generator in tensorflow, without a dense layer. Any suggestions ?
I saw it is possible in pytorch but not in tensorflow.
For background information :
convert a pytorch generator to tensorflow


title: How does Pytorch build the computation graph

body:
Here is example pytorch code from the website:
class Net(nn.Module):

def __init__(self):
    super(Net, self).__init__()
    # 1 input image channel, 6 output channels, 3x3 square convolution
    # kernel
    self.conv1 = nn.Conv2d(1, 6, 3)
    self.conv2 = nn.Conv2d(6, 16, 3)
    # an affine operation: y = Wx + b
    self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 10)

def forward(self, x):
    # Max pooling over a (2, 2) window
    x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
    # If the size is a square you can only specify a single number
    x = F.max_pool2d(F.relu(self.conv2(x)), 2)
    x = x.view(-1, self.num_flat_features(x))
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

In the forward function, we simply apply a series of transformations to x, but never explicitly define which objects are part of that transformation. Yet when computing the gradient and updating the weights, Pytorch 'magically' knows which weights to update and how the gradient should be calculated.
How does this process work? Is there code analysis going on, or something else that I am missing?


title: Neural Networks & ML without GPU. what are my options?

body:
I am starting working with Neural Networks, ML etc. I don't have a GPU, what are my options?
Are Pytorch and Tensorflow good options? If so, which are the pros and cons of each, and which should I choose?
I will be using Python and Linux (ubuntu 20.04) if that makes any difference.
(I cant afford a GPU or cloud services because im broke and 15 y.o.)


title: PyTorch installation problem- package not found using Jupyter notebook and Conda navigator

body:
I have tried to install PyTorch using the installation code form official PyTorch website.
I run it locally in the Jupyter notebook on Conda navigator
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch

I received the following mistake
    PackagesNotFoundError: The following packages are not available from current channels:

  - pytorch
  - cudatoolkit=10.2

Current channels:

  - https://conda.anaconda.org/pytorch/win-32
  - https://conda.anaconda.org/pytorch/noarch
  - https://repo.anaconda.com/pkgs/main/win-32
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/win-32
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/msys2/win-32
  - https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

What to do?


title: How can I create a 2D tensor with one element a sine and the other a cosine wave?

body:
I'm doing
train_data = np.array([np.sin(time), np.cos(time)])

This gives me something with a shape of (2, 4000). I think I need it to be (4000, 2), so for each timestep, I can get the sin and cos.
How can I do that?


title: Why is module.forward()) outputting a 1X1000 tensor?

body:
I am trying to implement a network in an android app. I have been following along with the following demo as I am new to android studio. https://pytorch.org/mobile/android/#custom-build
I am trying to modify it to work with a different network type generated below.
I generated a model.pt file using the following Python script:
import torchvision
from torch import nn

model = torchvision.models.resnet18(pretrained=True)
model.fc = nn.Linear(512, 15)
model.eval()
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("app/src/main/assets/model.pt")

Inside of Android Studio I load the module as follows.
bitmap = BitmapFactory.decodeStream(getAssets().open("ImageName.png"));
module = Module.load(assetFilePath(this, "model.pt"));

The click listener then executes the following when the infer app button is pressed:
final Tensor inputTensor =TensorImageUtils.bitmapToFloat32Tensor(bitmap,
                        TensorImageUtils.TORCHVISION_NORM_MEAN_RGB, TensorImageUtils.TORCHVISION_NORM_STD_RGB);

// running the model
final Tensor outputTensor = module.forward(IValue.from(inputTensor)).toTensor();

// getting tensor content as java array of floats
final float[] scores = outputTensor.getDataAsFloatArray();

The outputTensor has dimensions of 1X1000. I would have expected it to output 1X15 as the final classifier is modified to have 15. I also printed the model in Python and the last layer outputs a tensor of 15. I then loaded model.pt in a C++ file and it also outputs a tensor of size 1X15. Am I misunderstanding something about how module.forward() functions?