title: Backpropagation with PyTorch and autodifferentiation

body:
I have a dataset --X-- comprised of 15 variables and 64 observations and a column vector --Y-- of 64 values representing the target (label).  I try to fit parameters to a quadratic function to return the observed values (Y) using PyTorch but I get an error.  I provide the dataset in the end of the post in json format for reproducibility.

If I had a single example my code possibly could be:

X = torch.from_numpy(X)
X.requires_grad = True
W = np.random.randn(15,15)
W = np.triu(W, k=0)
W = torch.from_numpy(W)
W.requires_grad = True

# define parameters for gradient descent
max_iter=100
lr_rate = 1e-3

# we will do gradient descent for max_iter iteration
for i in range(max_iter):

        # compute the loss
        loss = Y - (X@torch.transpose(X, 1,0) * W).sum()
        # use torch.autograd.grad to compute the gradient
        W = W - lr_rate*torch.autograd.grad(out, W)[0]
        print(f"{i}: {out}")

Could you provide an example of correct implementation using the data I am providing below that would achieve the stated objective (fit parameters to the data) in a vectorized fashion?

Data:
X:

'{"embed_item_dim":{"0":-1.0,"1":1.0,"2":-1.0,"3":1.0,"4":-1.0,"5":1.0,"6":-1.0,"7":1.0,"8":-1.0,"9":1.0,"10":-1.0,"11":1.0,"12":-1.0,"13":1.0,"14":-1.0,"15":1.0,"16":-1.0,"17":1.0,"18":-1.0,"19":1.0,"20":-1.0,"21":1.0,"22":-1.0,"23":1.0,"24":-1.0,"25":1.0,"26":-1.0,"27":1.0,"28":-1.0,"29":1.0,"30":-1.0,"31":1.0,"32":-1.0,"33":1.0,"34":-1.0,"35":1.0,"36":-1.0,"37":1.0,"38":-1.0,"39":1.0,"40":-1.0,"41":1.0,"42":-1.0,"43":1.0,"44":-1.0,"45":1.0,"46":-1.0,"47":1.0,"48":-1.0,"49":1.0,"50":-1.0,"51":1.0,"52":-1.0,"53":1.0,"54":-1.0,"55":1.0,"56":-1.0,"57":1.0,"58":-1.0,"59":1.0,"60":-1.0,"61":1.0,"62":-1.0,"63":1.0},"embed_category_dim":{"0":-1.0,"1":-1.0,"2":1.0,"3":1.0,"4":-1.0,"5":-1.0,"6":1.0,"7":1.0,"8":-1.0,"9":-1.0,"10":1.0,"11":1.0,"12":-1.0,"13":-1.0,"14":1.0,"15":1.0,"16":-1.0,"17":-1.0,"18":1.0,"19":1.0,"20":-1.0,"21":-1.0,"22":1.0,"23":1.0,"24":-1.0,"25":-1.0,"26":1.0,"27":1.0,"28":-1.0,"29":-1.0,"30":1.0,"31":1.0,"32":-1.0,"33":-1.0,"34":1.0,"35":1.0,"36":-1.0,"37":-1.0,"38":1.0,"39":1.0,"40":-1.0,"41":-1.0,"42":1.0,"43":1.0,"44":-1.0,"45":-1.0,"46":1.0,"47":1.0,"48":-1.0,"49":-1.0,"50":1.0,"51":1.0,"52":-1.0,"53":-1.0,"54":1.0,"55":1.0,"56":-1.0,"57":-1.0,"58":1.0,"59":1.0,"60":-1.0,"61":-1.0,"62":1.0,"63":1.0},"embed_shop_dim":{"0":-1.0,"1":-1.0,"2":-1.0,"3":-1.0,"4":1.0,"5":1.0,"6":1.0,"7":1.0,"8":-1.0,"9":-1.0,"10":-1.0,"11":-1.0,"12":1.0,"13":1.0,"14":1.0,"15":1.0,"16":-1.0,"17":-1.0,"18":-1.0,"19":-1.0,"20":1.0,"21":1.0,"22":1.0,"23":1.0,"24":-1.0,"25":-1.0,"26":-1.0,"27":-1.0,"28":1.0,"29":1.0,"30":1.0,"31":1.0,"32":-1.0,"33":-1.0,"34":-1.0,"35":-1.0,"36":1.0,"37":1.0,"38":1.0,"39":1.0,"40":-1.0,"41":-1.0,"42":-1.0,"43":-1.0,"44":1.0,"45":1.0,"46":1.0,"47":1.0,"48":-1.0,"49":-1.0,"50":-1.0,"51":-1.0,"52":1.0,"53":1.0,"54":1.0,"55":1.0,"56":-1.0,"57":-1.0,"58":-1.0,"59":-1.0,"60":1.0,"61":1.0,"62":1.0,"63":1.0},"categorical_dim":{"0":-1.0,"1":-1.0,"2":-1.0,"3":-1.0,"4":-1.0,"5":-1.0,"6":-1.0,"7":-1.0,"8":1.0,"9":1.0,"10":1.0,"11":1.0,"12":1.0,"13":1.0,"14":1.0,"15":1.0,"16":-1.0,"17":-1.0,"18":-1.0,"19":-1.0,"20":-1.0,"21":-1.0,"22":-1.0,"23":-1.0,"24":1.0,"25":1.0,"26":1.0,"27":1.0,"28":1.0,"29":1.0,"30":1.0,"31":1.0,"32":-1.0,"33":-1.0,"34":-1.0,"35":-1.0,"36":-1.0,"37":-1.0,"38":-1.0,"39":-1.0,"40":1.0,"41":1.0,"42":1.0,"43":1.0,"44":1.0,"45":1.0,"46":1.0,"47":1.0,"48":-1.0,"49":-1.0,"50":-1.0,"51":-1.0,"52":-1.0,"53":-1.0,"54":-1.0,"55":-1.0,"56":1.0,"57":1.0,"58":1.0,"59":1.0,"60":1.0,"61":1.0,"62":1.0,"63":1.0},"categorical_dropout":{"0":-1.0,"1":-1.0,"2":-1.0,"3":-1.0,"4":-1.0,"5":-1.0,"6":-1.0,"7":-1.0,"8":-1.0,"9":-1.0,"10":-1.0,"11":-1.0,"12":-1.0,"13":-1.0,"14":-1.0,"15":-1.0,"16":1.0,"17":1.0,"18":1.0,"19":1.0,"20":1.0,"21":1.0,"22":1.0,"23":1.0,"24":1.0,"25":1.0,"26":1.0,"27":1.0,"28":1.0,"29":1.0,"30":1.0,"31":1.0,"32":-1.0,"33":-1.0,"34":-1.0,"35":-1.0,"36":-1.0,"37":-1.0,"38":-1.0,"39":-1.0,"40":-1.0,"41":-1.0,"42":-1.0,"43":-1.0,"44":-1.0,"45":-1.0,"46":-1.0,"47":-1.0,"48":1.0,"49":1.0,"50":1.0,"51":1.0,"52":1.0,"53":1.0,"54":1.0,"55":1.0,"56":1.0,"57":1.0,"58":1.0,"59":1.0,"60":1.0,"61":1.0,"62":1.0,"63":1.0},"numerical_dim":{"0":-1.0,"1":-1.0,"2":-1.0,"3":-1.0,"4":-1.0,"5":-1.0,"6":-1.0,"7":-1.0,"8":-1.0,"9":-1.0,"10":-1.0,"11":-1.0,"12":-1.0,"13":-1.0,"14":-1.0,"15":-1.0,"16":-1.0,"17":-1.0,"18":-1.0,"19":-1.0,"20":-1.0,"21":-1.0,"22":-1.0,"23":-1.0,"24":-1.0,"25":-1.0,"26":-1.0,"27":-1.0,"28":-1.0,"29":-1.0,"30":-1.0,"31":-1.0,"32":1.0,"33":1.0,"34":1.0,"35":1.0,"36":1.0,"37":1.0,"38":1.0,"39":1.0,"40":1.0,"41":1.0,"42":1.0,"43":1.0,"44":1.0,"45":1.0,"46":1.0,"47":1.0,"48":1.0,"49":1.0,"50":1.0,"51":1.0,"52":1.0,"53":1.0,"54":1.0,"55":1.0,"56":1.0,"57":1.0,"58":1.0,"59":1.0,"60":1.0,"61":1.0,"62":1.0,"63":1.0},"numerical_dropout":{"0":-1.0,"1":1.0,"2":1.0,"3":-1.0,"4":1.0,"5":-1.0,"6":-1.0,"7":1.0,"8":-1.0,"9":1.0,"10":1.0,"11":-1.0,"12":1.0,"13":-1.0,"14":-1.0,"15":1.0,"16":-1.0,"17":1.0,"18":1.0,"19":-1.0,"20":1.0,"21":-1.0,"22":-1.0,"23":1.0,"24":-1.0,"25":1.0,"26":1.0,"27":-1.0,"28":1.0,"29":-1.0,"30":-1.0,"31":1.0,"32":-1.0,"33":1.0,"34":1.0,"35":-1.0,"36":1.0,"37":-1.0,"38":-1.0,"39":1.0,"40":-1.0,"41":1.0,"42":1.0,"43":-1.0,"44":1.0,"45":-1.0,"46":-1.0,"47":1.0,"48":-1.0,"49":1.0,"50":1.0,"51":-1.0,"52":1.0,"53":-1.0,"54":-1.0,"55":1.0,"56":-1.0,"57":1.0,"58":1.0,"59":-1.0,"60":1.0,"61":-1.0,"62":-1.0,"63":1.0},"mixed_dim1":{"0":-1.0,"1":1.0,"2":1.0,"3":-1.0,"4":-1.0,"5":1.0,"6":1.0,"7":-1.0,"8":1.0,"9":-1.0,"10":-1.0,"11":1.0,"12":1.0,"13":-1.0,"14":-1.0,"15":1.0,"16":-1.0,"17":1.0,"18":1.0,"19":-1.0,"20":-1.0,"21":1.0,"22":1.0,"23":-1.0,"24":1.0,"25":-1.0,"26":-1.0,"27":1.0,"28":1.0,"29":-1.0,"30":-1.0,"31":1.0,"32":-1.0,"33":1.0,"34":1.0,"35":-1.0,"36":-1.0,"37":1.0,"38":1.0,"39":-1.0,"40":1.0,"41":-1.0,"42":-1.0,"43":1.0,"44":1.0,"45":-1.0,"46":-1.0,"47":1.0,"48":-1.0,"49":1.0,"50":1.0,"51":-1.0,"52":-1.0,"53":1.0,"54":1.0,"55":-1.0,"56":1.0,"57":-1.0,"58":-1.0,"59":1.0,"60":1.0,"61":-1.0,"62":-1.0,"63":1.0},"mixed_dropout1":{"0":-1.0,"1":1.0,"2":1.0,"3":-1.0,"4":-1.0,"5":1.0,"6":1.0,"7":-1.0,"8":-1.0,"9":1.0,"10":1.0,"11":-1.0,"12":-1.0,"13":1.0,"14":1.0,"15":-1.0,"16":1.0,"17":-1.0,"18":-1.0,"19":1.0,"20":1.0,"21":-1.0,"22":-1.0,"23":1.0,"24":1.0,"25":-1.0,"26":-1.0,"27":1.0,"28":1.0,"29":-1.0,"30":-1.0,"31":1.0,"32":-1.0,"33":1.0,"34":1.0,"35":-1.0,"36":-1.0,"37":1.0,"38":1.0,"39":-1.0,"40":-1.0,"41":1.0,"42":1.0,"43":-1.0,"44":-1.0,"45":1.0,"46":1.0,"47":-1.0,"48":1.0,"49":-1.0,"50":-1.0,"51":1.0,"52":1.0,"53":-1.0,"54":-1.0,"55":1.0,"56":1.0,"57":-1.0,"58":-1.0,"59":1.0,"60":1.0,"61":-1.0,"62":-1.0,"63":1.0},"mixed_dim2":{"0":-1.0,"1":1.0,"2":1.0,"3":-1.0,"4":-1.0,"5":1.0,"6":1.0,"7":-1.0,"8":-1.0,"9":1.0,"10":1.0,"11":-1.0,"12":-1.0,"13":1.0,"14":1.0,"15":-1.0,"16":-1.0,"17":1.0,"18":1.0,"19":-1.0,"20":-1.0,"21":1.0,"22":1.0,"23":-1.0,"24":-1.0,"25":1.0,"26":1.0,"27":-1.0,"28":-1.0,"29":1.0,"30":1.0,"31":-1.0,"32":1.0,"33":-1.0,"34":-1.0,"35":1.0,"36":1.0,"37":-1.0,"38":-1.0,"39":1.0,"40":1.0,"41":-1.0,"42":-1.0,"43":1.0,"44":1.0,"45":-1.0,"46":-1.0,"47":1.0,"48":1.0,"49":-1.0,"50":-1.0,"51":1.0,"52":1.0,"53":-1.0,"54":-1.0,"55":1.0,"56":1.0,"57":-1.0,"58":-1.0,"59":1.0,"60":1.0,"61":-1.0,"62":-1.0,"63":1.0},"mixed_dropout2":{"0":-1.0,"1":1.0,"2":-1.0,"3":1.0,"4":1.0,"5":-1.0,"6":1.0,"7":-1.0,"8":1.0,"9":-1.0,"10":1.0,"11":-1.0,"12":-1.0,"13":1.0,"14":-1.0,"15":1.0,"16":-1.0,"17":1.0,"18":-1.0,"19":1.0,"20":1.0,"21":-1.0,"22":1.0,"23":-1.0,"24":1.0,"25":-1.0,"26":1.0,"27":-1.0,"28":-1.0,"29":1.0,"30":-1.0,"31":1.0,"32":-1.0,"33":1.0,"34":-1.0,"35":1.0,"36":1.0,"37":-1.0,"38":1.0,"39":-1.0,"40":1.0,"41":-1.0,"42":1.0,"43":-1.0,"44":-1.0,"45":1.0,"46":-1.0,"47":1.0,"48":-1.0,"49":1.0,"50":-1.0,"51":1.0,"52":1.0,"53":-1.0,"54":1.0,"55":-1.0,"56":1.0,"57":-1.0,"58":1.0,"59":-1.0,"60":-1.0,"61":1.0,"62":-1.0,"63":1.0},"mixed_dim3":{"0":-1.0,"1":1.0,"2":-1.0,"3":1.0,"4":1.0,"5":-1.0,"6":1.0,"7":-1.0,"8":-1.0,"9":1.0,"10":-1.0,"11":1.0,"12":1.0,"13":-1.0,"14":1.0,"15":-1.0,"16":1.0,"17":-1.0,"18":1.0,"19":-1.0,"20":-1.0,"21":1.0,"22":-1.0,"23":1.0,"24":1.0,"25":-1.0,"26":1.0,"27":-1.0,"28":-1.0,"29":1.0,"30":-1.0,"31":1.0,"32":-1.0,"33":1.0,"34":-1.0,"35":1.0,"36":1.0,"37":-1.0,"38":1.0,"39":-1.0,"40":-1.0,"41":1.0,"42":-1.0,"43":1.0,"44":1.0,"45":-1.0,"46":1.0,"47":-1.0,"48":1.0,"49":-1.0,"50":1.0,"51":-1.0,"52":-1.0,"53":1.0,"54":-1.0,"55":1.0,"56":1.0,"57":-1.0,"58":1.0,"59":-1.0,"60":-1.0,"61":1.0,"62":-1.0,"63":1.0},"mixed_dropout3":{"0":-1.0,"1":1.0,"2":-1.0,"3":1.0,"4":1.0,"5":-1.0,"6":1.0,"7":-1.0,"8":-1.0,"9":1.0,"10":-1.0,"11":1.0,"12":1.0,"13":-1.0,"14":1.0,"15":-1.0,"16":-1.0,"17":1.0,"18":-1.0,"19":1.0,"20":1.0,"21":-1.0,"22":1.0,"23":-1.0,"24":-1.0,"25":1.0,"26":-1.0,"27":1.0,"28":1.0,"29":-1.0,"30":1.0,"31":-1.0,"32":1.0,"33":-1.0,"34":1.0,"35":-1.0,"36":-1.0,"37":1.0,"38":-1.0,"39":1.0,"40":1.0,"41":-1.0,"42":1.0,"43":-1.0,"44":-1.0,"45":1.0,"46":-1.0,"47":1.0,"48":1.0,"49":-1.0,"50":1.0,"51":-1.0,"52":-1.0,"53":1.0,"54":-1.0,"55":1.0,"56":1.0,"57":-1.0,"58":1.0,"59":-1.0,"60":-1.0,"61":1.0,"62":-1.0,"63":1.0},"last_layer_dim":{"0":-1.0,"1":1.0,"2":-1.0,"3":1.0,"4":-1.0,"5":1.0,"6":-1.0,"7":1.0,"8":1.0,"9":-1.0,"10":1.0,"11":-1.0,"12":1.0,"13":-1.0,"14":1.0,"15":-1.0,"16":1.0,"17":-1.0,"18":1.0,"19":-1.0,"20":1.0,"21":-1.0,"22":1.0,"23":-1.0,"24":-1.0,"25":1.0,"26":-1.0,"27":1.0,"28":-1.0,"29":1.0,"30":-1.0,"31":1.0,"32":-1.0,"33":1.0,"34":-1.0,"35":1.0,"36":-1.0,"37":1.0,"38":-1.0,"39":1.0,"40":1.0,"41":-1.0,"42":1.0,"43":-1.0,"44":1.0,"45":-1.0,"46":1.0,"47":-1.0,"48":1.0,"49":-1.0,"50":1.0,"51":-1.0,"52":1.0,"53":-1.0,"54":1.0,"55":-1.0,"56":-1.0,"57":1.0,"58":-1.0,"59":1.0,"60":-1.0,"61":1.0,"62":-1.0,"63":1.0},"last_layer_dropout":{"0":-1.0,"1":1.0,"2":-1.0,"3":1.0,"4":-1.0,"5":1.0,"6":-1.0,"7":1.0,"8":1.0,"9":-1.0,"10":1.0,"11":-1.0,"12":1.0,"13":-1.0,"14":1.0,"15":-1.0,"16":-1.0,"17":1.0,"18":-1.0,"19":1.0,"20":-1.0,"21":1.0,"22":-1.0,"23":1.0,"24":1.0,"25":-1.0,"26":1.0,"27":-1.0,"28":1.0,"29":-1.0,"30":1.0,"31":-1.0,"32":1.0,"33":-1.0,"34":1.0,"35":-1.0,"36":1.0,"37":-1.0,"38":1.0,"39":-1.0,"40":-1.0,"41":1.0,"42":-1.0,"43":1.0,"44":-1.0,"45":1.0,"46":-1.0,"47":1.0,"48":1.0,"49":-1.0,"50":1.0,"51":-1.0,"52":1.0,"53":-1.0,"54":1.0,"55":-1.0,"56":-1.0,"57":1.0,"58":-1.0,"59":1.0,"60":-1.0,"61":1.0,"62":-1.0,"63":1.0}}'

Y

'{"0":2.0561221309,"1":2.0649733606,"2":2.0733728925,"3":2.0594125771,"4":2.0949032045,"5":2.0294939058,"6":2.0436441327,"7":2.1209041954,"8":2.0496001055,"9":2.148755921,"10":2.0937250525,"11":2.0629058135,"12":2.0641746866,"13":2.0592979107,"14":2.1166172412,"15":2.1125198086,"16":2.0525522671,"17":2.0687485594,"18":2.0649582587,"19":2.0818384718,"20":2.0839422046,"21":2.043783441,"22":2.05290516,"23":2.0565277924,"24":2.0550897444,"25":2.0663609971,"26":2.0895415003,"27":2.0706054531,"28":2.0639581304,"29":2.0889003421,"30":2.0436977626,"31":2.1350170653,"32":2.0395688425,"33":2.079368626,"34":2.0439947954,"35":2.072433023,"36":2.050665861,"37":2.037977855,"38":2.0527567514,"39":2.050903715,"40":2.0381965719,"41":2.0673631206,"42":2.085004701,"43":2.0458497661,"44":2.0540644062,"45":2.050330556,"46":2.0859451303,"47":2.0323004844,"48":2.05113558,"49":2.046360857,"50":2.0572361143,"51":2.0659940765,"52":2.0583657215,"53":2.0520969623,"54":2.0683284923,"55":2.0491708591,"56":2.0932832342,"57":2.0416396082,"58":2.0703974941,"59":2.0464359665,"60":2.0591405783,"61":2.0527808995,"62":2.0670555565,"63":2.0898413706}'


title: Pytorch Module: Why do we pass the class and object to the parent class initializer, in the __init__ method?

body:
Why do we pass the class and the object (self) to the parent's init method for pytorch Module? For e.g.

class RNN(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
    super(RNN, self).__init__()

Why is the class RNN, as well as the object (self) passed to the parent's init?


title: Save a block diagram of model in PyTorch

body:
Keras has a nice functionality to save the model architecture as a block diagram

keras.utils.plot_model(self.model, plot_filepath.as_posix(), show_shapes=True)

Can this be achieved in PyTorch?

This answer talks about printing summary similar to Keras. Is there something similar to save the block diagram of model architecture?


title: Loading & Freezing a Pretrained Model to Combine with a New Network

body:
I have a pretrained model and would like to build a classifier on top of it. I’m trying to load and freeze the weights of the pretrained model, and pass its outputs to the new classifier, which I’d like to optimise. Here is what I have so far, I’m a little stuck on a TypeError: forward() missing 1 required positional argument: 'x' error from the nn.Sequential line:

import model #model.py contains the architecture of the pretrained model

class Classifier(nn.Module):
    def __init__(self):
        ...
    def forward(self, x):
        ...

net = model.Model()
net.load_state_dict(checkpoint["net"])

for c in net.children():
    for param in child.parameters():
        params.requires_grad = False

model = nn.Sequential(nn.ModuleList(net()), Classifier())


title: Modifying validation function for single image instead of Tencrop

body:
I have a PublicTest function that runs every epoch for validation and there is a transform test variable that transforms the validation data as above:

    transform_test = transforms.Compose([
    transforms.TenCrop(cut_size),
    transforms.Lambda(lambda crops: torch.stack([transforms.ToTensor()(crop) for crop in crops])),
])

def PublicTest(epoch):
    global PublicTest_acc
    global best_PublicTest_acc
    global best_PublicTest_acc_epoch
    net.eval()
    PublicTest_loss = 0
    correct = 0
    total = 0
    for batch_idx, (inputs, targets) in enumerate(PublicTestloader):
        bs, ncrops, c, h, w = np.shape(inputs)
        inputs = inputs.view(-1, c, h, w)
        if use_cuda:
            inputs, targets = inputs.cuda(), targets.cuda()
        inputs, targets = Variable(inputs, volatile=True), Variable(targets)
        outputs = net(inputs)
        outputs_avg = outputs.view(bs, ncrops, -1).mean(1)  # avg over crops
        loss = criterion(outputs_avg, targets)
        PublicTest_loss += loss.item()
        _, predicted = torch.max(outputs_avg.data, 1)
        total += targets.size(0)
        correct += predicted.eq(targets.data).cpu().sum()

        utils.progress_bar(batch_idx, len(PublicTestloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
                           % (PublicTest_loss / (batch_idx + 1), 100. * correct / total, correct, total))

I'd like to modify the code so instead of Tencrop each image would be validated once. I changed the transform_test like this

 transform_test = transforms.Compose([transforms.ToTensor()])

I run the code but had the realized that PublicTest have ncrops as a parameter and received the error:

 File "mainpro_FER.py", line 147, in PublicTest
    bs, ncrops, c, h, w = np.shape(inputs)
ValueError: not enough values to unpack (expected 5, got 4)

removed ncrops parameter in the function, tried again had this error:

  IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

So I need a way to make this function work for single image for every images in batch unlike TenCrop.
Thanks.


title: CNN vs SVM for smile intensity detection training?

body:
I have a dataset made up of images of faces, with the corresponding landmarks that make up the mouth.
These landmarks are sets of 2D points (x,y pixel position).
Each image-landmark set data pair is tagged as either a smile, or neutral.

What i would like to do is train a deep learning model to return a smile intensity for a new image-landmark data pair.

What should I be searching for to help me with the next step?
Is it a CNN that i need? In my limited understanding, the usual training input is just an image, where I would be passing the landmark sets to train with. Or would an SVM approach be more accurate?

I am looking for maximum accuracy, as much as is possible.

What is the approach that I need called?

I am happy to use PyTorch, Dlib or any framework, I am just a little stuck on the search terms to help me move forward.

Thank you.


title: How to use Pytorch to assign multi gpu without cuda_visible_device

body:
I use docker in a server, the GPU is randomly assigned to me, and cuda_visible_device is forbidden in pbs or shell document because it will conflict with server's assignment.

How can I replace os.environ(cuda_visible_device) safely?
Please give me an example


title: Accuracy/train and Loss/train graph by tensorboard

body: i used tensorboard for my pytorch project and got this result for accuracy/train and loss/train but i dont understand what it means


title: Apply a function on sub-tensors in a way that can be parallelized (Pytorch)

body: Lets say I have a Pytorch function f that can be applied to a tensor of shape shape_in=(d1,...,dn) and returns a tensor of shape shape_out. Now I have a tensor T of shape (other_shape,shape_in)=(b1,...,bm,d1,...,dn). How can I apply f to T to return a tensor of shape (other_shape,shape_out) in a way that can be parallelized? I dont think just looping over all elements of a tensor of shape other_shape would be optimal since the computations would probably be done sequentially (is this right?). I think map_fn in Tensorflow does exactly that but I havnt seen an equivalent in torch.


title: How to use GPU while training a model?

body:
I am running a code to train a resnet model on a kaggle notebook. I have chosen the accelerator as GPU so I haven't made any mistakes there. I am training the model using the following code:

model.cuda()
for epoch in range(10):
  model.train(True)
  trainloss=0
  for x,y in trainloader:

    x,y=x.cuda(),y.cuda()

    yhat=model(x)
    optimizer.zero_grad()
    loss=criterion(yhat,y)
    loss.backward()
    optimizer.step()
    trainloss+=loss.item()

  print('Epoch {}  Loss: {}'.format(epoch,(trainloss/len(trainloader.dataset))))
  model.eval()
  testcorrect=0
  with torch.no_grad():
    for test_x,test_y in testloader:
      test_x,test_y=test_x.cuda(),test_y.cuda()
      yhat=model(test_x)
      _,z=yhat.max(1)
      testcorrect+=(test_y==z).sum().item()

print('Model Accuracy: ',(testcorrect/len(testloader.dataset)))

Network Code:

model=torchvision.models.resnet18(pretrained=True)

num_ftrs=model.fc.in_features
model.fc=nn.Sequential(nn.Linear(num_ftrs,1000),
                        nn.ReLU(),
                        nn.Linear(1000,2)
)

If you see I have used the .cuda() function on both my model as well as the tensors(inside the training part as well as validation part). However the GPU usage shown for the kaggle notebook is 0% while my CPU usage is up to 99%. Am I missing any code which is required to train the model using the GPU?


title: Error in Calculating neural network Test Accuracy

body:
I tried to train my neural network, and then evaluate it's testing accuracy. I am using the code at the bottom of this post to train. The fact is that for other neural networks, I can evaluate the testing accuracy with my code without issue. However, for this neural network (which I constructed correctly according to the description of the neural network paper), I can't evaluate the testing accuracy properly and its giving me the traceback below. So maybe something's wrong in my forward pass?

Here is the training and testing code:

//imports including import deepnet.py
cudnn.benchmark = True
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train = X_train.astype('float32')
X_train = np.transpose(X_train, axes=(0, 3, 1, 2))
X_test = X_test.astype('float32')
X_test = np.transpose(X_test, axes=(0, 3, 1, 2))
X_train /= 255
X_test /= 255
device = torch.device('cuda:0')

# This is where you can load any model of your choice.
# I stole PyTorch Vision's VGG network and modified it to work on CIFAR-10.
# You can take this line out and add any other network and the code
# should run just fine.
model = deepnet.cifar10_deep()
#model.to(device)

# Forward pass
opfun = lambda X: model.forward(Variable(torch.from_numpy(X)))

# Forward pass through the network given the input
predsfun = lambda op: np.argmax(op.data.numpy(), 1)

# Do the forward pass, then compute the accuracy
accfun = lambda op, y: np.mean(np.equal(predsfun(op), y.squeeze()))*100

# Initial point
x0 = deepcopy(model.state_dict())

# Number of epochs to train for
# Choose a large value since LB training needs higher values
# Changed from 150 to 30
nb_epochs = 30
batch_range = [25, 40, 50, 64, 80, 128, 256, 512, 625, 1024, 1250, 1750, 2048, 2500, 3125, 4096, 4500, 5000]

# parametric plot (i.e., don't train the network if set to True)
hotstart = False

if not hotstart:
    for batch_size in batch_range:
        optimizer = torch.optim.Adam(model.parameters())
        model.load_state_dict(x0)
        #model.to(device)
        average_loss_over_epoch = '-'
        print('Optimizing the network with batch size %d' % batch_size)
        np.random.seed(1337) #So that both networks see same sequence of batches
        for e in range(nb_epochs):
            model.eval()
            print('Epoch:', e, ' of ', nb_epochs, 'Average loss:', average_loss_over_epoch)
            average_loss_over_epoch = 0

            # Checkpoint the model every epoch
            torch.save(model.state_dict(), "./models/DeepNetC2BatchSize" + str(batch_size) + ".pth")
            array = np.random.permutation(range(X_train.shape[0]))
            slices = X_train.shape[0] // batch_size
            beginning = 0
            end = 1

            # Training loop!
            for _ in range(slices):
                start_index = batch_size * beginning
                end_index = batch_size * end
                smpl = array[start_index:end_index]
                model.train()
                optimizer.zero_grad()
                ops = opfun(X_train[smpl])
                tgts = Variable(torch.from_numpy(y_train[smpl]).long().squeeze())
                loss_fn = F.nll_loss(ops, tgts)
                average_loss_over_epoch += loss_fn.data.numpy() / (X_train.shape[0] // batch_size)
                loss_fn.backward()
                optimizer.step()
                beginning += 1
                end += 1

grid_size = 18 #How many points of interpolation between [0, 5000]
data_for_plotting = np.zeros((grid_size, 3)) #Uncomment this line if running entire code from scratch
sharpnesses1eNeg3 = []
sharpnesses5eNeg4 = []
#data_for_plotting = np.load("DeepNetCIFAR10-intermediate-values.npy") #Uncomment this line to use an existing NumPy array
print(data_for_plotting)
i = 0

# Fill in test accuracy values for `grid_size' points in the interpolation
for batch_size in batch_range:
    mydict = {}
    batchmodel = torch.load("./models/DeepNetC2BatchSize" + str(batch_size) + ".pth")
    for key, value in batchmodel.items():
        mydict[key] = value
    model.load_state_dict(mydict)

    j = 0
    for datatype in [(X_train, y_train), (X_test, y_test)]:
        dataX = datatype[0]
        datay = datatype[1]
        for smpl in np.split(np.random.permutation(range(dataX.shape[0])), 10):
            ops = opfun(dataX[smpl])
            tgts = Variable(torch.from_numpy(datay[smpl]).long().squeeze())
            var = F.nll_loss(ops, tgts).data.numpy() / 10
            if j == 1:
                data_for_plotting[i, j-1] += accfun(ops, datay[smpl]) / 10.
        j += 1
    print(data_for_plotting[i])
    np.save('DeepNetCIFAR10-intermediate-values', data_for_plotting)
    i += 1

And the model code is here and includes the forward pass

import torch
import torch.nn as nn
F = nn.functional
__all__ = ['cifar10_deepnet', 'cifar100_deepnet']

class VGG(nn.Module):
    def __init__(self, num_classes=10):
        super(VGG, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.Dropout(0.3),
            nn.Conv2d(64, 64, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(64, 128, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(128, 128, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(128, 256, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(256, 256, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(256, 256, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(256, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        self.classifier = nn.Sequential(
            nn.Linear(512, 512, bias=False),
            nn.Dropout(0.5),
            nn.BatchNorm1d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(512, num_classes)
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(-1, 512)
        x = self.classifier(x)
        return F.log_softmax(x)

def cifar10_deep(**kwargs):
    num_classes = getattr(kwargs, 'num_classes', 10)
    return VGG(num_classes)

def cifar100_deep(**kwargs):
    num_classes = getattr(kwargs, 'num_classes', 100)
    return VGG(num_classes)


title: Why different coding style obtain different results in Pytorch?

body:
I am new to Pytorch. When I implement a network like this, in init():

...
self.conv1=nn.Conv2d(in_channel,out_channel_mid,kernel_size=1,stride=1,padding=0,bias=False)
        self.relu6=nn.ReLU6()
        self.bn1=nn.BatchNorm2d(out_channel_mid)
...

then in forward():

            x=self.conv1(x)
            x=self.bn1(x)
            x=self.relu6(x)

I obtain results like this:

Batch:0 in Epoch:0,total loss:1.6094366312026978
Batch:30 in Epoch:0,total loss:49.83972668647766
Epoch:0, total loss: 0.025
Epoch:0, accuracy: 0.245
Batch:0 in Epoch:1,total loss:1.59962797164917
Batch:30 in Epoch:1,total loss:49.709712743759155
Epoch:1, total loss: 0.025
Epoch:1, accuracy: 0.245
Batch:0 in Epoch:2,total loss:1.6107732057571411
Batch:30 in Epoch:2,total loss:49.63734531402588
Epoch:2, total loss: 0.188
Epoch:2, accuracy: 0.217
Batch:0 in Epoch:3,total loss:49.33592987060547
Batch:30 in Epoch:3,total loss:474.46153497695923
Epoch:3, total loss: 0.188
Epoch:3, accuracy: 0.173
Batch:0 in Epoch:4,total loss:9.524840354919434
Batch:30 in Epoch:4,total loss:250.68861961364746
Epoch:4, total loss: 0.115
Epoch:4, accuracy: 0.217
...

the model will converge finally but very slow.

But When I coding like this, encapsulation the basic convolution modules into a class:

class BasicBlock(nn.Module):
   def __init__(self,in_channel,out_channel,kernel_size=3,stride=1,padding=1,groups=1):
    super(BasicBlock,self).__init__()
   self.conv=nn.Sequential(nn.Conv2d(in_channel,out_channel,kernel_size,stride,padding,groups=groups,bias=False),                                                          nn.BatchNorm2d(out_channel),                                                           nn.ReLU6())
   def forward(self,x):
         x=self.conv(x)
         return x

then define and call it in the corresponding model:
self.conv1=BasicBlock(in_channel,out_channel_mid,kernel_size=1,stride=1,padding=0)

and the model converges much faster than before:

Batch:0 in Epoch:0,total loss:1.6127870082855225
Batch:30 in Epoch:0,total loss:72.60288417339325
Epoch:0, total loss: 0.031
Epoch:0, accuracy: 0.418
Batch:0 in Epoch:1,total loss:1.387904405593872
Batch:30 in Epoch:1,total loss:43.76062071323395
Epoch:1, total loss: 0.022
Epoch:1, accuracy: 0.533
Batch:0 in Epoch:2,total loss:1.3291308879852295
Batch:30 in Epoch:2,total loss:46.50250172615051
Epoch:2, total loss: 0.023
Epoch:2, accuracy: 0.481
Batch:0 in Epoch:3,total loss:1.2623569965362549
Batch:30 in Epoch:3,total loss:39.24768841266632
Epoch:3, total loss: 0.020
Epoch:3, accuracy: 0.514
Batch:0 in Epoch:4,total loss:1.6293309926986694
Batch:30 in Epoch:4,total loss:40.47891843318939
Epoch:4, total loss: 0.020
Epoch:4, accuracy: 0.538
Batch:0 in Epoch:5,total loss:1.3565115928649902
Batch:30 in Epoch:5,total loss:37.65265315771103
Epoch:5, total loss: 0.019
Epoch:5, accuracy: 0.574
Batch:0 in Epoch:6,total loss:1.2062609195709229
Batch:30 in Epoch:6,total loss:35.79732310771942

I can't figure out the difference between these two styles.

What's more, the use of batch normalization seems also to make the model converge slower, even always obtain the same accuracy in more than 10 epochs. Is there anything wrong when I use these methods?


title: PyTorch: load data / models to a specific CPU

body:
I'm implementing an algorithm which requires a lot of model evaluations. I want to parallelize model evaluations by using CPU. The code segment I want to parallelize is here (simplified for readability):
def test_img(network, datatest):
    network.eval()

    data_loader = DataLoader(datatest, batch_size=args.bs)

    for idx, (data, target) in enumerate(data_loader):
        result = network(data)

And for one evaluation it takes around 1.5s on CPU. The networks for evaluation are different, but the dataset is the same. I tried to use joblib.Parallel to parallelize this process:
results = Parallel(n_jobs=num_cpu, prefer="threads")(delayed(test_img)(network_lst[i], dataset) for i in range(N))

However, it seems that there are no improvement by using this method (it will take 15s for 10 evaluations). I specifically count the runtime for result = network(data) line, and it takes 8 seconds. Therefore I think neither the network evaluation nor the dataloader are parallelized. Are there any way for us to load network and data to a specific CPU core, like data.to('cpu:0')? Is it even possible to use CPU for model evaluation parallelization?
Any suggestions are appreciated!


title: CondaVerificationError: conda.exe showing corrupted

body:
I was trying to download Pytorch where cuda = 10.0 in Anaconda --version: 4.7.12 , Python --version: 4.7.12 .

Preparing transaction: done Verifying transaction: failed

CondaVerificationError: The package for conda located at C:\Users\joyan\.conda\pkgs\conda-4.8.3-py37_0
appears to be corrupted. The path 'Scripts/conda.exe'
specified in the package manifest cannot be found.

What can I do for it?


title: batch_size != eval_batch_size when training and evaluating a transformer based language model

body:
I'm trying to follow pytorch's transformer tutorial. They are training and then evaluating a seq-2-seq language model.
I noted that the batch size (length of input vector) is different when training and evaluating:
batch_size = 20
eval_batch_size = 10

why would I want that? and if I do, how does it work? is it padding the tensor with zeros? where are they doing that?
thanks!


title: How do I release the CUDA memory of an embedding generated with flair?

body:
I have a loop that looks like this:
embedding = TransformerDocumentEmbeddings('bert-base-uncased')
for row in df.itertuples():
    sentence = Sentence(row.TEXT)
    embedding.embed(sentence)
    df.at[row.Index, 'TEXT'] = sentence.embedding

But I run out of CUDA memory almost immediately. I've tried running del sentence and torch.cuda.empty_cache() at the end of each iteration but doesn't work. What should be done here?


title: Manually adjusting parameters of a torch.nn.Module

body:
Suppose I had a simple neural network defined by:
import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(2,2)
        self.fc2 = nn.Linear(2,2)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)

        return x

net = Net()

Doing the following:
for param in net.parameters():
    print(param.data)

results in something like:
tensor([[-0.0776,  0.2409],
        [ 0.3478, -0.6820]])
tensor([-0.6311,  0.2323])
tensor([[-0.5466,  0.0341],
        [ 0.5822,  0.7005]])
tensor([-0.5624,  0.3278])

Let's say that I have a tensor([[0,0],[0,1]]) and I wanted to replace the first
param.data with my custom tensor.
Is this possible and if so, how can I do this?


title: PyTorch find keypoints: output nodes to be in a range and negative loss

body:
I am beginner in deep learning.
I am using this dataset and I want my network to detect keypoints of a hand.
How can I make my output layer's nodes to be in range [-1, 1] (range of normalized 2D points)?
Another problem is when I train for more than 1 epoch the loss gets negative values
criterion: torch.nn.MultiLabelSoftMarginLoss() and optimizer: torch.optim.SGD()
Here u can find my repo
net = nnModel.Net()
net = net.to(device)
criterion = nn.MultiLabelSoftMarginLoss()
optimizer = optim.SGD(net.parameters(), lr=learning_rate)
lr_scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer=optimizer, gamma=decay_rate)


title: Pythonic way to start an iterator from an specific index

body:
I'm trying to change this code so the iterator in the function sample_real_video_batch in line 142 would start from a specific batch number (as opposed to start of the dataset).
Here is the function:
def sample_real_video_batch(self):
    if self.video_enumerator is None:
        self.video_enumerator = enumerate(self.video_sampler)

    batch_idx, batch = next(self.video_enumerator)
    b = batch
    if self.use_cuda:
        for k, v in batch.iteritems():
            b[k] = v.cuda()

    if batch_idx == len(self.video_sampler) - 1:
        self.video_enumerator = enumerate(self.video_sampler)

    return b

An obvious and very inefficient approach would be to call the function in a loop for n times. But generally speaking, is there an efficient way to start the iterator off a specific index in python?
And is it applicable to pytorch ImageDataset?


title: Uniform sampling with replacement non-randomly in PyTorch

body:
Given a bool tensor (mask) of shape 4x4:
x = torch.tensor([
    [ True, True, False, True ],
    [ False, False, False, True ],
    [ False, True, True, True ],
    [ False, True, True, False ],
])

I want to sample (7) accordingly with the following result which is of shape 4x7:
tensor([[0, 0, 0, 1, 1, 3, 3],
        [3, 3, 3, 3, 3, 3, 3],
        [1, 1, 1, 2, 2, 3, 3],
        [1, 1, 1, 1, 2, 2, 2]])

The closest thing I could come is the following implementation:
def uniform_sampling(tensor, count = 1):
    indices = torch.arange(0, tensor.shape[-1], device = tensor.device).expand(tensor.shape)
    samples_count = tensor.long().sum(-1)
    output = tensor.long() * (count // samples_count)[:, None]
    remainder = count - output.sum(-1)

    rem1 = torch.stack((remainder, tensor.sum(-1) - remainder), -1).flatten()
    rem2 = torch.stack((torch.ones_like(remainder), torch.zeros_like(remainder)), -1).flatten()
    remaining = rem2.repeat_interleave(rem1, 0)

    output[tensor > 0] += remaining
    samples = indices[tensor].repeat_interleave(output[tensor], -1).view(-1, count)

    return samples

uniform_sampling(x, count = 7)

Are there any (maybe native) PyTorch functionalities that does the same but faster and more efficiently?


title: Is there a way torch.mode can be applied multiple times without using a loop

body:
let us  say arr1 has size  10 and arr2 is size 5 by 5 .(arr2 contains 5 rows of the target indices to be checked)
is there a way of getting the modes of arr1 using the 5 different indices without a loop
i tried this but it only works if arr2 is one dimensional and it returns the mode of the values of the specific indices in arr2
modes =torch.mode(arr1[arr2]).values


title: Index is out of range when iterating through a ModuleList's Sequentials

body:
I'm writing a YOLO neural network and am fairly new to it.
I have a config file that is used to define the network. It is stored as a nn.ModuleList, with its modules as nn.Sequential.
Examples of these sequential:
  (0): Sequential(
    (conv_0): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (batch_norm_0): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (leaky_0): LeakyReLU(negative_slope=0.1, inplace=True)
  )
  (1): Sequential(
    (conv_1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (batch_norm_1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (leaky_1): LeakyReLU(negative_slope=0.1, inplace=True)
  )
  (2): Sequential(
    (conv_2): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (batch_norm_2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (leaky_2): LeakyReLU(negative_slope=0.1, inplace=True)
  )

I then do some formatting with this, for my output to have the right shape, iterating through every module of the ModuleList.
for i, module in enumerate(modules):
            module_type = (module["type"])

            if module_type == "convolutional" or module_type == "upsample":
                x = self.module_list[i](x)

            elif module_type == "route":
                layers = module["layers"]
                layers = [int(a) for a in layers]

                if (layers[0]) > 0:
                    layers[0] = layers[0] - i

                if len(layers) == 1:
                    x = outputs[i + (layers[0])]

                else:
                    if (layers[1]) > 0:
                        layers[1] = layers[1] - i

                    map1 = outputs[i + layers[0]]
                    map2 = outputs[i + layers[1]]
                    x = torch.cat((map1, map2), 1)

            elif module_type == "shortcut":
                from_ = int(module["from"])
                x = outputs[i-1] + outputs[i+from_]

            elif module_type == 'yolo':
                anchors = self.module_list[i][0].anchors
                #Get the input dimensions
                inp_dim = int (self.net_info["height"])

                #Get the number of classes
                num_classes = int (module["classes"])

                #Transform
                x = x.data
                x = predict_transform(x, inp_dim, anchors, num_classes, CUDA)
                if not write:              #if no collector has been intialised.
                    detections = x
                    write = 1

                else:
                    detections = torch.cat((detections, x), 1)

            outputs[i] = x

However, when I get to the anchors = self.module_list[i][0].anchors line, I get an error that Index 0 is out of range
I do not know why this occurs. I have tried to format the shape of my ModuleList differently.
Any suggestions would be greatly appreciated.


title: How to write the einsum operation: 'i,jk->ijk' in tensorial product form?

body:
I am using pytorch and want to broadcast a vector from 1D (features) to a 3D tensor (features×height×width) to condition the encoding of an image with a Convolutional Neural Network. As is done in Figure S1 of this Deepmind paper.
For now I am using this:
# features: 1D vector
f = torch.einsum('i,jk->ijk',features,torch.ones([5,5]))


title: Creating training data from individual .npy files

body:
I am trying to create training data for my NN project. I have 3 channels I want as input data, representing different spatial fields: Ux, Uy, and Uz. Currently, I've formatted the data as follows:
Ux, Uy, and Uz are all their own folders with ~300-400 .npy files depending on the case. I have 8 different cases, so my total data size is ~3000. Each numpy array is of size 100x60
case_1 >
  Ux >
    Ux_1.npy
    ...
    Ux_350.npy
  Uy >
    Uy_1.npy
    ...
    Uy_350.npy
  Uz >
    Uz_1.npy
    ...
    Uz_350.npy

case_2 >
  Ux >
    Ux_1.npy
    ...
    Ux_330.npy
  Uy >
    Uy_1.npy
    ...
    Uy_330.npy
  Uz >
    Uz_1.npy
    ...
    Uz_330.npy

....

I want my NN input data to have 3 channels as follows:
[[Ux_1, Uy_1, Uz_1], .. [Ux_350, Uy_350, Uz_350]]

for all 8 cases, so basically a shape of (3000, (3, 100, 60)). The cases do not have to be distinguished, they're all part of the same training data. I would love some guidance on how to start doing this. I am used to already-prepared datasets, and I'm not sure how to combine all .npy files into one large tensor for training.


title: Error of horovod load data on each worker node with databrick spark

body:
I am trying to run horovod.torch on gpu clusters (p2.xlarge) from databricks.
Because horovod use AllReduce to communicate parameters among the nodes, each worker node needs to load the whole dataset and work on the different partitions. After each iteration, all nodes get the parameters values from other nodes by AllReduce to update their own parameters by taking averages of them.
My understanding is that this is SPMD (single program multiple data) becauee each worker node needs to load the same whole dataset.
I need to load the whole dataset from each worker node, right ?
My code:
import horovod.torch as hvd
from sparkdl import HorovodRunner

def test1():
    hvd.init()
    train_df = spark.read.parquet("s3://my_data/").cache()
    print("load data done")

hr = HorovodRunner(np=2)
hr.run(test1)

But I got error:

Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.

It seems that spark does not allow multiple context ?
I also tried to create a new local spark session on each worker:
def test1():
    hvd.init()

    from pyspark.sql import SparkSession

    spark = SparkSession \
          .builder \
          .appName("Python Spark SQL example") \
          .config("spark.some.config.option", "some-value") \
          .getOrCreate()

    train_df = spark.read.parquet("s3://my_data/").cache()

hr = HorovodRunner(np=2)
hr.run(test1)

I got error:
[1,1]<stderr>:Error: Could not find or load main class org.apache.spark.launcher.Main
[1,1]<stderr>:/databricks/spark/bin/spark-class: line 101: CMD: bad array subscript

How to use spark to load data on each worker node ?
If spark does not allow other nodes to create their own sparksession, how to load data on each worker node for horovod ?


title: CNN forward function , AutoTuning the number of layers

body:
class ConvolutionalNetwork(nn.Module):
    def __init__(self, in_features, trial):
        # we optimize the number of layers, hidden units and dropout ratio in each layer.
        n_layers = self.trial.suggest_int("n_layers", 1, 5)
        p = self.trial.suggest_uniform("dropout_1{}".format(i), 0, 1.0)
        layers = []
        for i in range(n_layers):
            self.out_features = self.trial.suggest_int("n_units_1{}".format(i), 16, 160,step =2)
            kernel_size = trial.suggest_int('kernel_size', 2, 7)
            layers.append(nn.Conv1d(1, self.out_features,kernel_size,1))
            layers.append(nn.RReLU())
            layers.append(nn.BatchNorm1d(self.out_features)
            layers.append(nn.Dropout(p))

            self.in_features = self.out_features

        layers.append(nn.Conv1d(self.in_features, 16,kernel_size,1))
        layers.append(nn.RReLU())
        return nn.Sequential(*layers)

As you can see above, I did some Optuna tuning of parameters , including tuning on the number of layers.
def forward(self,x):
    # shape x for conv 1d op
    x = x.view(-1, 1, self.in_features)
    x = self.conv1(x)
    x = F.rrelu(x)
    x = F.max_pool1d(x, 64, 64)
    x = self.conv2(x)
    x = F.rrelu(x)
    x = F.max_pool1d(x, 64, 64)
    x = x.view(-1, self.n_conv)

    x = self.dp(x)
    x = self.fc3(x)
    x = F.log_softmax(x, dim=1)
    return x

I now need to do the same for the forward function above, I did the pseudo code below but it won't run, kindly advise how. The main issue is to incorporate the for loop function for the forward function.
def forward(self,x):
    # shape x for conv 1d op
    x = x.view(-1, 1, self.in_features)
    for i in range(n_layers):
        layers.append(self.conv1(x))
        layers.append(F.rrelu(x))
        layers.append(F.max_pool1d(x, 64, 64))
    x = x.view(-1, self.n_conv)
    x = self.dp(x)
    x = self.fc3(x)
    #x = F.sigmoid(x)
    x = F.log_softmax(x, dim=1)
    return x


title: I got value error while training my GAN such as: "ValueError: Target size (torch.Size([10, 1])) must be the same as input size (torch.Size([10]))"

body:
I am new at GANs and also pytorch, studied some repos to developed my skills, but when I tried to implement this repo:

https://github.com/jasonbian97/Deep-Learning-Computer-Vision/blob/master/generative_adversarial_networks.ipynb

Everything goes well, but I got such error when I tried to change these code snippets in
def discriminator_loss(logits_real, logits_fake) function
(I changed it because before this change I got ValueError: Target size (torch.Size([128])) must be the same as input size (torch.Size([128, 1])) ):
  logits_real_loss = bce_loss(logits_real, torch.ones(N, 1).to(dtype).to(device))
  logits_fake_loss = bce_loss(logits_fake, torch.zeros(N, 1).to(dtype).to(device))
  loss = logits_real_loss + logits_fake_loss

and
  N = logits_fake.shape[0]
  device = logits_fake.device
  loss = (bce_loss(logits_fake, torch.ones(N,1).to(dtype).to(device)))

Getting value error:
---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-15-b089f365cd5e> in <module>()
      2   d_loss = discriminator_loss(logits_real, logits_fake)
      3   print("Maximum error in d_loss: %g"%rel_error(d_loss_true, d_loss))
----> 4 test_discriminator_loss(answers['logits_real'], answers['logits_fake'],answers['d_loss_true'])

2 frames

<ipython-input-15-b089f365cd5e> in test_discriminator_loss(logits_real, logits_fake, d_loss_true)
      1 def test_discriminator_loss(logits_real, logits_fake, d_loss_true):
----> 2   d_loss = discriminator_loss(logits_real, logits_fake)
      3   print("Maximum error in d_loss: %g"%rel_error(d_loss_true, d_loss))
      4 test_discriminator_loss(answers['logits_real'], answers['logits_fake'],answers['d_loss_true'])

<ipython-input-14-15d437ae3425> in discriminator_loss(logits_real, logits_fake)
     17   N = logits_real.shape[0]
     18   device = logits_real.device
---> 19   logits_real_loss = bce_loss(logits_real, torch.ones(N, 1).to(dtype).to(device))
     20   logits_fake_loss = bce_loss(logits_fake, torch.zeros(N, 1).to(dtype).to(device))
     21   loss = logits_real_loss + logits_fake_loss

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction, pos_weight)
   2578
   2579     if not (target.size() == input.size()):
-> 2580         raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
   2581
   2582     return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)

ValueError: Target size (torch.Size([10, 1])) must be the same as input size (torch.Size([10]))

Maybe solution is so simple, but I cannot see could anyone help me for fix this issue?


title: How can one use the RemoveIsolatedNodes transform in Pytorch Geometric?

body:
I am trying to run a graph classification problem in pytorch-geometric and I see that some of my graphs contain isolated nodes (which can cause problems). For example, my dataset is a list of pytorch data objects:
dataset = [graph1, graph2, graph3...]

where graph1 is a pytorch-geometric data object, containing the graph's structure, node features and label. I see that pytorch geometric ALREADY HAS A TRANSFORM for precisely this task, however it doesn't say anywhere how to apply it, as it's a class that takes no input.


title: How to load model for inference?

body:
I am using a simple perceptron based classifier to generate sentiment analysis in Pytorch, complete code here Classifying Yelp Reviews.
The example does sentiment analysis, it outputs if the given input string is positive or negative.

example: this is a pretty small old great book -> positive

The application stores the final model along with the vectorizer.json.
So my question is: What should be the pre-requisites to build a separate application just to test the model, so that it can be used in web-application later-on?
Below is my current understanding and queries for this:
I assume to test, we need to load the model, load model parameters and evaluate for inference, please confirm

model = TheModelClass(*args, **kwargs) # Model class must be defined somewhere
model.load_state_dict(torch.load(PATH))
model.eval() # run if you only want to use it for inference

Once step 1 is done, I hope we can deploy the model using Flask and expose a REST API for model inference.


title: How can I create a tensor of batch batch_size of uniformly distributed values between -1 and 1?

body:
The title pretty much sums it, I'm trying to implement a GAN:
How can I create a tensor of batch batch_size of uniformly distributed values between -1 and 1 with pytorch?
def create_latent_batch_vectors(batch_size, latent_vector_size, device):
'''
The function creates a random batch of latent vectors with random values
distributed uniformly between -1 and 1.
Finally, it moves the tensor to the given ```device``` (cpu or gpu).
The output should have a shape of [batch_size, latent_vector_size].
'''
# maybe torch.distributions.uniform.Uniform() somehow?
return z.to(device)

Thanks!


title: Decode BERT embedding sentence

body:
I got an embedding sentence genertated by **bert-base-multilingual-cased** which calculated by the average of the second-and-last layers from hidden_states. tensor size is [768]
My goal is to decode this tensor and get the tokens that the model calculated.
I dont have the input sentence so i need to figure out by myself
My approch
 np_v = np.load('nlp_embedding_sentence.npy')
 v = torch.from_numpy(np_v)

 from transformers import BertTokenizer
 tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
 tokenizer.decode(v)

output
[PAD] [PAD] [PAD] [PAD] [PAD] [PAD]
....
 [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] ...

I need to find the actual sentence from this vector.
I'm new to NLP please help me with this problem.


title: How can I convert the dimension in the model form 2D to 1D?

body:
I am beginner of using pytorch. I would like to classify 2d binary array (17 * 20 ) to 8 classes, I am using cross entropy as loss function . I have 512 batch size . the input is 512 batches of size (17 * 20 )and the final outpu 512 batches of size 8. I applied the following model , I would like to get the final output to be only list of length 8. like [512,8] but I got that dim [512,680,8] (I printed the dimensions i git from the model after the code). How can I get [512,8] from that network as final output.
 def __init__(self, M=1):
        super(PPS, self).__init__()
        #input layer
        self.layer1 = nn.Sequential(
             nn.Conv2d(17, 680,  kernel_size=1, stride=1, padding=0),
             nn.ReLU())
        self.drop1 = nn.Sequential(nn.Dropout())
        self.batch1 = nn.BatchNorm2d(680)
        self.lstm1=nn.Sequential(nn.LSTM(
        input_size=20,
        hidden_size=16,
        num_layers=1,
        bidirectional=True,
        batch_first= True))
        self.gru = nn.Sequential(nn.GRU(
            input_size=16*2,
            hidden_size=16,
            num_layers=2,
            bidirectional=True,
            batch_first=True))
        self.fc1 = nn.Linear(16*2,8)

    def forward(self, x):

        out = self.layer1(x)
        out = self.drop1(out)
        out = self.batch1(out)
        out = out.squeeze()
        out,_ = self.lstm1(out)
        out,_ = self.gru(out)
        out = self.fc1(out)
        return out
cov2d torch.Size([512, 680, 20, 1])
drop torch.Size([512, 680, 20, 1])
batch torch.Size([512, 680, 20])
lstm1 torch.Size([512, 680, 32])
lstm2 torch.Size([512, 680, 32])
linear1 torch.Size([512, 680, 8])


title: Finding the index of peak point ( the first True) in boolean tensor mask

body:
After running a detectron2 model in pytorch, Detectron2 gives me the object masks that it finds as a (true/false) tensor. there are 33 objects found in the image so I have torch.Size([33, 683, 1024]).
tensor([[False, False, False,  ..., False, False, False],
    [False, False, False,  ..., False, False, False],
    [False, False, False,  ..., False, False, False],
    ...,
    [False, False, False,  ..., False, False, False],
    [False, False, False,  ..., False, False, False],
    [False, False, False,  ..., False, False, False]], device='cuda:0')

This is great so far. But I need the peak coordinates in y dimension (height) of those 33 objects. (Lets say the object is baloon, then I need the top of the baloon as (x,y) point)
Any idea how can I get the peak point coordinates as fast as possible
thanks in advance


title: Why does it take so long print the value of a GPU tensor in Pytorch?

body:
I wrote this pytorch program to compute a 5000*5000 matrix multiplication on GPU, 100 iterations.
import torch
import numpy as np
import time

N = 5000
x1 = np.random.rand(N, N)

######## a 5000*5000 matrix multiplication on GPU, 100 iterations #######
x2 = torch.tensor(x1, dtype=torch.float32).to("cuda:0")

start_time = time.time()
for n in range(100):
    G2 = x2.t() @ x2
print(G2.size())
print("It takes", time.time() - start_time, "seconds to compute")
print("G2.device:", G2.device)

start_time2 = time.time()
# G4 = torch.zeros((5,5),device="cuda:0")
G4 = G2[:5, :5]
print("G4.device:", G4.device)
print("G4======", G4)
# G5=G4.cpu()
# print("G5.device:",G5.device)
print("It takes", time.time() - start_time2, "seconds to transfer or display")

Here is the result on my laptop:

torch.Size([5000, 5000])

It takes 0.22243595123291016 seconds to compute

G2.device: cuda:0

G4.device: cuda:0

G4====== tensor([[1636.3195, 1227.1913, 1252.6871, 1242.4584, 1235.8160],
[1227.1913, 1653.0522, 1260.2621, 1246.9526, 1250.2871],
[1252.6871, 1260.2621, 1685.1147, 1257.2373, 1266.2213],
[1242.4584, 1246.9526, 1257.2373, 1660.5951, 1239.5414],
[1235.8160, 1250.2871, 1266.2213, 1239.5414, 1670.0034]],
device='cuda:0')

It takes 60.13639569282532 seconds to transfer or display
Process finished with exit code 0

I am confused why it takes so much time to display the variable G5 on GPU, since is only 5*5 in size.
BTW, I use "G5=G4.cpu()" to transfer the variable on GPU to CPU, it takes so much time too.
My develop enviroment (rather old laptop):

pytorch 1.0.0

CUDA 8.0

Nvidia GeForce GT 730m

Windows 10 Professional

when increasing iteration times, the compute time do not increase obviously, but the transfer or display increased obviously, Why? Can somebody interpret, thanks so much.


title: I've downloaded bert pretrained model 'bert-base-cased'. I'm unable to load the model with help of BertTokenizer

body:
I've downloaded bert pretrained model 'bert-base-cased. I'm unable to load the model with help of BertTokenizer. I'm trying for bert tokenizer. In the bert-pretrained-model folder I have config.json and pytorch_model.bin.
tokenizer = BertTokenizer.from_pretrained(r'C:\Downloads\bert-pretrained-model')

I'm facing error like
OSError                                   Traceback (most recent call last)
<ipython-input-17-bd4c0051c48e> in <module>
----> 1 tokenizer = BertTokenizer.from_pretrained(r'\Downloads\bert-pretrained-model')

~\sentiment_analysis\lib\site-packages\transformers\tokenization_utils_base.py in from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1775                 f"- or '{pretrained_model_name_or_path}' is the correct path to a directory containing relevant tokenizer files\n\n"
   1776             )
-> 1777             raise EnvironmentError(msg)
   1778
   1779         for file_id, file_path in vocab_files.items():

OSError: Can't load tokenizer for 'C:\Downloads\bert-pretrained-model'. Make sure that:

- 'C:\Downloads\bert-pretrained-model' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'C:\Downloads\bert-pretrained-model' is the correct path to a directory containing relevant tokenizer files

When I'm trying load with BertModel, it's loading. But when i'm trying with BertTokenizer it's not loading.


title: Pip is uninstalling pytorch built from source when i use 'pip install '. Is there a way?

body:
I needed to build pytorch from source in order to use the latest cuda toolkit - hence I did that. My environment is a conda enviornment.
The package I am trying to install is allennlp.
Pip list is not showing torch installed, conda is showing it is.
Tried building allennlp from source through:
conda skeleton pypi <package>
conda build <package>

conda build is also causing problems due to incompability with pip packages etc...
I am just wondering if there is a simpler way to do this?
For example: Tell pip that torch is already installed so it stops uninstalling the current torch (why can't it just pick it up as installed).
Appreciate the help!


title: Is there any way to convert .pt to pkl

body: Im trying to make webapp from trained model ı already did using .pkl file but now ı have trained my model using yolo so my weights are on the best.pt file. How can ı implement my best.pt file to webapp or can ı convert it to .pkl file


title: Same weights, implementation but different results n Keras and Pytorch

body:
I have an encoder and a decoder model (monodepth2). I try convert them from Pytorch to Keras using Onnx2Keras, but :

Encoder(ResNet-18) succeeds
I build the decoder myself in Keras (with TF2.3), and copy the weights (numpy array, including weight and bias) for each layer from Pytorch to Keras, without any modification.

But it turns out both Onnx2Keras-converted Encoder and self-built Decoder fails to reproduce the same results. The cross-comparison pictures are below, but I'd first introduce the code of Decoder.
First the core Layer, all the conv2d layer (Conv3x3, ConvBlock) is based on this, but different dims or add an activation:
# Conv3x3 (normal conv2d without BN nor activation)
# There's also a ConvBlock, which is just "Conv3x3 + ELU activation", so I don't list it here.
def TF_Conv3x3(input_channel, filter_num, pad_mode='reflect', activate_type=None):

    # Actually it's 'reflect, but I implement it with tf.pad() outside this
    padding = 'valid'

    # if TF_ConvBlock, then activate_type=='elu
    conv = tf.keras.layers.Conv2D(filters=filter_num, kernel_size=3, activation=activate_type,
                                  strides=1, padding=padding)
    return conv

Then the structure. Note that the definition is EXACTLY the same as the original code. I think it must be some details about the implementation.
def DepthDecoder_keras(num_ch_enc=np.array([64, 64, 128, 256, 512]), channel_first=False,
                       scales=range(4), num_output_channels=1):
    num_ch_dec = np.array([16, 32, 64, 128, 256])
    convs = OrderedDict()
    for i in range(4, -1, -1):
        # upconv_0
        num_ch_in = num_ch_enc[-1] if i == 4 else num_ch_dec[i + 1]
        num_ch_out = num_ch_dec[i]

        # convs[("upconv", i, 0)] = ConvBlock(num_ch_in, num_ch_out)
        convs[("upconv", i, 0)] = TF_ConvBlock(num_ch_in, num_ch_out, pad_mode='reflect')

        # upconv_1
        num_ch_in = num_ch_dec[i]
        if i > 0:
            num_ch_in += num_ch_enc[i - 1]
        num_ch_out = num_ch_dec[i]
        convs[("upconv", i, 1)] = TF_ConvBlock(num_ch_in, num_ch_out, pad_mode='reflect')  # Just Conv3x3 with ELU-activation

    for s in scales:
        convs[("dispconv", s)] = TF_Conv3x3(num_ch_dec[s], num_output_channels, pad_mode='reflect')

    """
    Input_layer dims: (64, 96, 320), (64, 48, 160),  (128, 24, 80), (256, 12, 40), (512, 6, 20)
    """
    x0 = tf.keras.layers.Input(shape=(96, 320, 64))
    # then define the the rest input layers
    input_features = [x0, x1, x2, x3, x4]

    """
    # connect layers
    """
    outputs = []
    ch = 1 if channel_first else 3
    x = input_features[-1]
    for i in range(4, -1, -1):
        x = tf.pad(x, paddings=[[0, 0], [1, 1], [1, 1], [0, 0]], mode='REFLECT')
        x = convs[("upconv", i, 0)](x)
        x = [tf.keras.layers.UpSampling2D()(x)]
        if i > 0:
            x += [input_features[i - 1]]
        x = tf.concat(x, ch)
        x = tf.pad(x, paddings=[[0, 0], [1, 1], [1, 1], [0, 0]], mode='REFLECT')
        x = convs[("upconv", i, 1)](x)
    x = TF_ReflectPad2D_1()(x)
    x = convs[("dispconv", 0)](x)
    disp0 = tf.math.sigmoid(x)

    """
    build keras Model ([input0, ...], [output0, ...])
    """
    # decoder = tf.keras.Model(input_features, outputs)
    decoder = tf.keras.Model(input_features, disp0)

    return decoder

The cross-comparison is as follows... I would really appreciate it if anyone could offer some insights. Thanks!!!
Original results:

Original Encoder + Self-build Decoder:

ONNX-converted Enc + Original Dec (Texture is good, but the contrast is not enough, the car should be very close, i.e. very bright color):

ONNX-converted Enc + Self-built Dec:


title: Training accuracy decrease and loss increase when using pack_padded_sequence - pad_packed_sequence

body:
I'm trying to train a bidirectional lstm with pack_padded_sequence and pad_packed_sequence, but the accuracy keeps decreasing while the loss increasing.
This is my data loader:
X1 (X[0]): tensor([[1408, 1413,   43,  ...,    0,    0,    0],
    [1452, 1415, 2443,  ...,    0,    0,    0],
    [1434, 1432, 2012,  ...,    0,    0,    0],
    ...,
    [1408, 3593, 1431,  ...,    0,    0,    0],
    [1408, 1413, 1402,  ...,    0,    0,    0],
    [1420, 1474, 2645,  ...,    0,    0,    0]]), shape: torch.Size([64, 31])

len_X1 (X[3]): [9, 19, 12, 7, 7, 15, 4, 13, 9, 8, 14, 19, 7, 23, 7, 13, 7, 12, 10, 12, 13, 11, 31, 8, 20, 17, 8, 9, 9, 29, 8, 5, 5, 13, 9, 8, 10, 17, 13, 8, 8, 11, 7, 29, 15, 10, 6, 7, 10, 9, 10, 10, 4, 16, 11, 10, 16, 8, 13, 8, 8, 20, 7, 12]

X2 (X[1]): tensor([[1420, 1415,   51,  ...,    0,    0,    0],
        [1452, 1415, 2376,  ..., 1523, 2770,   35],
        [1420, 1415,   51,  ...,    0,    0,    0],
        ...,
        [1408, 3593, 1474,  ...,    0,    0,    0],
        [1408, 1428, 2950,  ...,    0,    0,    0],
        [1474, 1402, 3464,  ...,    0,    0,    0]]), shape: torch.Size([64, 42])

len_X2 (X[4]): [14, 42, 13, 18, 12, 31, 8, 19, 5, 7, 15, 19, 7, 17, 6, 11, 12, 16, 8, 8, 19, 8, 12, 10, 11, 9, 9, 9, 9, 21, 7, 5, 8, 13, 14, 8, 15, 8, 8, 8, 12, 13, 7, 14, 4, 10, 6, 11, 12, 7, 8, 11, 9, 13, 30, 10, 15, 9, 9, 7, 9, 8, 7, 20]

t (X[2]): tensor([0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1,
        0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0,
        0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1]), shape: torch.Size([64])

This is my model class:
class BiLSTM(nn.Module):
def __init__(self, n_vocabs, embed_dims, n_lstm_units, n_lstm_layers, n_output_classes):
    super(BiLSTM, self).__init__()
    self.v = n_vocabs
    self.e = embed_dims
    self.u = n_lstm_units
    self.l = n_lstm_layers
    self.o = n_output_classes
    self.padd_idx = tokenizer.get_vocab()['[PAD]']
    self.embed = nn.Embedding(
        self.v,
        self.e,
        self.padd_idx
        )
    self.bilstm = nn.LSTM(
        self.e,
        self.u,
        self.l,
        batch_first = True,
        bidirectional = True,
        dropout = 0.5
        )
    self.linear = nn.Linear(
        self.u * 4,
        self.o
        )

  def forward(self, X):
    # initial_hidden
    h0 = torch.zeros(self.l * 2, X[0].size(0), self.u).to(device)
    c0 = torch.zeros(self.l * 2, X[0].size(0), self.u).to(device)

    # embedding
    out1 = self.embed(X[0].to(device))
    out2 = self.embed(X[1].to(device))

    # # pack_padded_sequence
    out1 = nn.utils.rnn.pack_padded_sequence(out1, X[3], batch_first=True, enforce_sorted=False)
    out2 = nn.utils.rnn.pack_padded_sequence(out2, X[4], batch_first=True, enforce_sorted=False)

    # NxTxh, lxNxh
    out1, _ = self.bilstm(out1, (h0, c0))
    out2, _ = self.bilstm(out2, (h0, c0))

    # # pad_packed_sequence
    out1, _ = nn.utils.rnn.pad_packed_sequence(out1, batch_first=True)
    out2, _ = nn.utils.rnn.pad_packed_sequence(out2, batch_first=True)

    # take only the final time step
    out1 = out1[:, -1, :]
    out2 = out2[:, -1, :]

    # concatenate out1&2
    out = torch.cat((out1, out2), 1)

    # linear layer
    out = self.linear(out)

    iout = torch.max(out, 1)[1]
    return iout, out

And if I remove pack_padded_sequence - pad_packed_sequence, the model training works just fine:
class BiLSTM(nn.Module):
def __init__(self, n_vocabs, embed_dims, n_lstm_units, n_lstm_layers, n_output_classes):
    super(BiLSTM, self).__init__()
    self.v = n_vocabs
    self.e = embed_dims
    self.u = n_lstm_units
    self.l = n_lstm_layers
    self.o = n_output_classes
    self.padd_idx = tokenizer.get_vocab()['[PAD]']
    self.embed = nn.Embedding(
        self.v,
        self.e,
        self.padd_idx
        )
    self.bilstm = nn.LSTM(
        self.e,
        self.u,
        self.l,
        batch_first = True,
        bidirectional = True,
        dropout = 0.5
        )
    self.linear = nn.Linear(
        self.u * 4,
        self.o
        )

  def forward(self, X):
    # initial_hidden
    h0 = torch.zeros(self.l * 2, X[0].size(0), self.u).to(device)
    c0 = torch.zeros(self.l * 2, X[0].size(0), self.u).to(device)

    # embedding
    out1 = self.embed(X[0].to(device))
    out2 = self.embed(X[1].to(device))

    # pack_padded_sequence
    # out1 = nn.utils.rnn.pack_padded_sequence(out1, X[3], batch_first=True, enforce_sorted=False)
    # out2 = nn.utils.rnn.pack_padded_sequence(out2, X[4], batch_first=True, enforce_sorted=False)

    # NxTxh, lxNxh
    out1, _ = self.bilstm(out1, (h0, c0))
    out2, _ = self.bilstm(out2, (h0, c0))

    # pad_packed_sequence
    # out1, _ = nn.utils.rnn.pad_packed_sequence(out1, batch_first=True)
    # out2, _ = nn.utils.rnn.pad_packed_sequence(out2, batch_first=True)

    # take only the final time step
    out1 = out1[:, -1, :]
    out2 = out2[:, -1, :]

    # concatenate out1&2
    out = torch.cat((out1, out2), 1)

    # linear layer
    out = self.linear(out)

    iout = torch.max(out, 1)[1]
    return iout, out


title: Pytorch gather question (3D Computer Vision)

body:
I have N groups of C-dimension points. In each groups there are M points. So, there is a tensor of (N, M, C). Let's call it features.
I calculated the maximum element and the index through M dimension, to find the maximum points for each C dimension (a max pooling operation), resulting max tensor (N, 1, C) and index tensor (N, 1, C).
I have another tensor of shape (N, M, 3) storing the geometric coordinates of those N*M high-dimention points. Now, I want to use the index from the maximum points in each C dimension, to get the coordinates of all those maximum points.
For example, N=2, M=4, C=6.
The coordinate tensor, whose shape is (2, 4, 3):
[[[1, 2, 3]
  [4, 5, 6]
  [7, 8, 9]
  [8, 7, 6]]

 [11, 12, 13]
 [14, 15, 16]
 [17, 18, 19]
 [18, 17, 16]]]

The indices tensor, whose shape is (2, 1, 6):
[[[0, 1, 2, 1, 2, 3]]
 [[1, 2, 3, 2, 1, 0]]]

For example, the first element in indices is 0, I want to grab [1, 2, 3] from the coordinate tensor out. For the second element (1), I want to grab [4, 5, 6] out. For the third element in the next dimension (3), I want to grab [18, 17, 16] out.
The result tensor will look like:
[[[1, 2, 3]  # 0
  [4, 5, 6]  # 1
  [7, 8, 9]  # 2
  [4, 5, 6]  # 1
  [7, 8, 9]  # 2
  [8, 7, 6]] # 3

 [[14, 15, 16] # 1
  [17, 18, 19] # 2
  [18, 17, 16] # 3
  [17, 18, 19] # 2
  [14, 15, 16] # 1
  [11, 12, 13]]]# 0

whose shape is (2, 6, 3).
I tried to use torch.gather but I can not get it worked. I wrote a naive algorithm enumerating all N groups, but indeed it is slow, even using TorchScript's JIT. So, how to write this efficiently in pytorch?


title: How to interpolate latent space in a condtional gan setup for image interpolation?

body:
I am trying to interpolate images from 1 label to another however i can't find any codes for this for a condtional GAN
This is the code for a simple interpolation

def get_interpolations(noise1, noise2, num_interpolates):
    x = [((1.0 - (i/num_interpolates)) * noise1) + ((i/num_interpolates) * noise2) for i in range(num_interpolates + 1)]
    return torch.cat(x, dim = 0)

def random_noise(size):
    return torch.normal(0, 1, size, device = device)

with torch.no_grad():
    interpolations = []
    for _ in range(10):
        noise1 = random_noise([1, 100])
        noise2 = random_noise([1, 100])
        interpolations.append(get_interpolations(noise1, noise2, 9).to(device))

    interpolations = torch.cat(interpolations, dim = 0)

    false_images = gen(interpolations).cpu()
    print(f'\nShowing Generated Samples\n')
    visualize_batch(false_images, shape = (10, 10), size = 16, save = None)

however a condtional GAN also takes in label information
xb = torch.randn(BATCH_SIZE, Z_DIM, 1, 1).cuda() # random latent tensors
label = torch.randint(6,7,(BATCH_SIZE,),device=device)
fake_images = gen(xb,label)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-26-439fa6512d5c> in <module>
      8     interpolations = torch.cat(interpolations, dim = 0)
      9     label = torch.randint(0,1,(1,),device=device)
---> 10     false_images = gen(interpolations,label).cpu()
     11     print(f'\nShowing Generated Samples\n')
     12     visualize_batch(false_images, shape = (10, 10), size = 16, save = None)

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-1-ad8eacd654bb> in forward(self, x, labels)
     73
     74         embedding=self.embed(labels).unsqueeze(2).unsqueeze(3)
---> 75         x=torch.cat([x,embedding],dim=1)
     76         return self.net(x)
     77

RuntimeError: Tensors must have same number of dimensions: got 4 and 2

This is the error i get if i try sending in the labels as a second parameter in the first code block


I'm trying to write an assertion statement in my code. The goal of the assertion statement is to check for data leakage, hence I don't want to just delete it.
assert filled_target[:, -forecast_length:, :].any() != trg[:, d -forecast_length:decoder_seq_len, :].any()

However this is raising the following error
RuntimeError: all only supports torch.uint8 and torch.bool dtypes
Both filled_target and trg are type torch.cuda.FloatTensor .  I'm not sure if this is related to this or not.


title: Working of nn.Linear with multiple dimensions

body:
PyTorch's nn.Linear(in_features, out_features) accepts a tensor of size (N_batch, N_1, N_2, ..., N_end), where N_end = in_features. The output is  a tensor of size (N_batch, N_1, N_2, ..., out_features).
It isn't very clear to me how it behaves in the following situations:

If v is a row, the output will be A^Tv+b
If M is a matrix, it is treated as a batch of rows, and for every row v, A^Tv+b is performed, and then everything is put back into matrix form
What if the input tensor is of a higher rank? Say the input tensor has dimensions (N_batch, 4, 5, 6, 7). Is it true that the layer will output a batch of size N_batch of (1, 1, 1, N_out)-shaped vectors, everything shaped into a (N_batch, 4, 5, 6, N_out) tensor?


title: Numpy: get only elements on odd or even diagonal offsets in a matrix, change the rest to zeros

body:
I have a 2D (square) matrix, for example it can be like this:
1 2 3
4 5 6
7 8 9

I want to get only the elements on the odd or even diagonal offsets of it, and let the rest be zeros. For example, with even diagonal offsets (%2 = 0), the resulting matrix is:
1 0 3
0 5 0
7 0 9

Explanation: main diagonal has offset 0, which is 1 5 9. The next diagonal offsets are 2, 6 and 4, 8 thus they are changed to zeros. Repeat the process until we reach the last diagonal.
And with odd diagonal index, the resulting matrix is:
0 2 0
4 0 6
0 8 0

I look at the np.diag(np.diag(x)) but it only returns the main diagonal and the rest are zeros. How can I extend it to odd/even offsets?
I can also use PyTorch.


title: In pytorch, how to train a model with two or more outputs?

body:
output_1, output_2 = model(x)
loss = cross_entropy_loss(output_1, target_1)
loss.backward()
optimizer.step()

loss = cross_entropy_loss(output_2, target_2)
loss.backward()
optimizer.step()

However, when I run this piece of code, I got this error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 4]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Then, I really wanna know what I am supposed to do to train a model with 2 or more outputs


title: How to change and use Pytorch trained Scaled Yolov4 weights to use in Opencv (as .weights file)?

body:
I trained a Scaled YOLO v4 model on my images using this notebook and got the model weights which are in form best_yolov4-csp-results.pt.
With DarkNet weights in Yolov4, I got weights as .weights and I can use Opencv Contrib to read and give predictions as:
net = cv2.dnn.readNet("./yolov4-obj_best_1_class.weights", "./yolov4-custom_1_class.cfg")
model = cv2.dnn_DetectionModel(net)
model.setInputParams(size=(416, 416),scale=1/255.)
classes, scores, boxes = model.detect(img_array, 0.51, 0.51)

My question is, How can I do the same for the current pytorch_weights.pt file?
or if not possible, how can I use Pytorch only to deploy the model for each image?


title: Torch doesnt see gpu on gcloud with deep learning containers

body:
I am trying to run some python code on kubernetes in GCloud, I am using pytorch and for the base image i am using gcr.io/deeplearning-platform-release/pytorch-gpu/.
Everything spins up fine, and my model trains but only using CPU. When i run the following
>>> import torch
>>> torch.cuda.is_available()
/opt/conda/lib/python3.7/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA    initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at  /opt/conda/conda-bld/pytorch_1614378098133/work/c10/cuda/CUDAFunctions.cpp:109.)
return torch._C._cuda_getDeviceCount() > 0
False

But this is the output of nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   62C    P8    32W / 149W |     11MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

The version of pytorch is the one that is given by the base image, so i am not exactly sure how to fix this. Any help would be appreciated.
EDIT:
>>> torch.__version__
'1.8.0'

EDIT TWO:
>>> torch.version.cuda
'11.1'


title: Remove rows from tensor by matching

body:
I'm trying to do some operation like if there is tensor in pytorch
a = torch.tensor([[1,0]
                  ,[0,1]
                  ,[2,0]
                  ,[3,2]])

b = torch.tensor([[0,1]
                  ,[2,0]])

I want to remove the rows [0,1], [2,0] which are the rows of b from a.
Is there any way to do this?
# result
a = torch.tensor([[1,0]
                  ,[3,2]])


title: Creating a model which weights are the sum of weights of 2 different neural networks

body:
I am doing an experiment of transfer learning.
I trained 2 CNNs that have exactly the same structure, one for MNIST and one for SVHN.
I obtained the parameters (weights  and bias) of the 2 models.
Now, I want to combine (sum, or other operations) these weights. A thing like this:
modelMNIST.parameters()
modelSVHN.parameters()

#now the new model
model3 = MyCNN(1)
model3.parameters = modelMNIST.parameters()+modelSVHN.parameters()

If I do in this way, I obtain this error:
SyntaxError: can't assign to function call
And in this way:
model3.block_1[0].weight = modelMNIST.block_1[0].weight + modelSVHN.block_1[0].weight

I get this error:
TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)

Is there any way to combine weights of different models?


title: how to create an augmented dataset in pythorch

body:
I have to add to the original CIFAR dataset, for each image, the corrispondent ones, rotated by 90 deg. The idea is to create the a RotationDateset, a class which extends datasets.VisionDataset, which takes the CIFAR and does what describes above.
from __future__ import print_function, division
import skimage.io

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
import numpy as np
from torchvision.datasets import ImageFolder
import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import time
import os
from sklearn.model_selection import train_test_split
import copy
import cv2
from torchvision.models.resnet import BasicBlock
from torchvision.models.resnet import ResNet
from PIL import Image
import xml.etree.ElementTree as ET
from torch.utils.model_zoo import load_url as load_state_dict_from_url
from torchvision.models.resnet import model_urls

//org_dataset is the CIFAR
//num_rots is 4
//transforms is transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5, 0.5,0.5), (0.5, 0.5, 0.5))])
class RotDataset(datasets.VisionDataset):
    def __init__(self, org_dataset, transforms, num_rots):

        self.samples = org_dataset.data
        self.targets = []
        self.num_rots = num_rots
        self.transforms = transforms

        for k in self.samples:
          self.targets.append(k)

          for i in range(0, self.num_rots):
            tr = torchvision.transforms.Compose([torchvision.transforms.RandomRotation(degrees=90*i),
                                                 torchvision.transforms.ToTensor(),
                                                 torchvision.transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
            # from PIL import Image
            p_i = Image.fromarray(k)
            te = tr(p_i)
            r_im = torch.reshape(te, (k.shape))
            r_im = np.array(r_im)
            self.targets.append(r_im)

    def __len__(self):
        return len(self.samples)

    def __getitem__(self, index):
      imgs = self.targets[index:index + self.num_rots]
      labels = list(range(0, self.num_rots))

      return imgs, labels

here's how i import and transform initially the CIFAR:
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)

testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

here's how i create the CIFAR augmented:
cifar_rot = RotDataset(trainset, trainset.transforms, 4)

rot_train, rot_val= train_test_split(
np.arange(len(cifar_rot.targets)),
test_size=0.2,
shuffle=True,
)

train_sampler_rot = torch.utils.data.SubsetRandomSampler(rot_train)
val_sampler_rot = torch.utils.data.SubsetRandomSampler(rot_val)

dataloaders_rot = {'train': torch.utils.data.DataLoader(cifar_rot, batch_size=8, sampler=train_sampler_rot)
               , 'val':torch.utils.data.DataLoader(cifar_rot, batch_size=8, sampler=val_sampler_rot)}

sizes_rot = {'train':len(rot_train)*4,'val':len(rot_val)*4}

and the model training
model_rot = torchvision.models.resnet34(pretrained=False)

num_ftrs = model_rot.fc.in_features
output_dim_rot = 4 # since are 4 rotations

model_rot.fc = nn.Linear(num_ftrs, output_dim_rot)

model_rot = model_rot.to(device)
criterion = nn.CrossEntropyLoss()

optimizer_conv = optim.SGD(model_rot.parameters(), lr=0.001, momentum=0.9)

exp_lr_scheduler = lr_scheduler.StepLR(optimizer_conv, step_size=7, gamma=0.1)
model_rot = train_model(model_rot,
                        criterion,
                        optimizer_conv,
                        exp_lr_scheduler,
                        dataloaders_rot,
                        sizes_rot,
                        num_epochs=10)

torch.save(model_rot.state_dict(),'rotation_resnet34_10_epochs.pt')

//the problem is that when I launch the model, pythorch throws this error:
Epoch 0/9
----------
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-61-977dbbbef6fe> in <module>()
     23                         dataloaders_rot,
     24                         sizes_rot,
---> 25                         num_epochs=10)
     26 #Save the best trained model, for later use
     27 torch.save(model_rot.state_dict(),'rotation_resnet34_10_epochs.pt')

6 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    394                             _pair(0), self.dilation, self.groups)
    395         return F.conv2d(input, weight, bias, self.stride,
--> 396                         self.padding, self.dilation, self.groups)
    397
    398     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[32, 32, 32, 3] to have 3 channels, but got 32 channels instead

anyone can help me? thanks in advance


title: How do I one hot encode along a specific dimension using PyTorch?

body:
I have a tensor of size [3, 15, 136], where:

3 is batch size
15 - sequence length and
136 is tokens

I want to one-hot my tensor using the probabilities in the tokens dimension (136). To do so I want to extract the tokens dimension for each letter in sequence length and put 1 to the largest possibility and mark all other tokens as 0.


title: How to transfer weight of own model to same network but different number of classin last layer?

body:
I have my own network in Pytorch. It first trained for the binary classifier (2 classes). After 10k epochs, I obtained the trained weight as 10000_model.pth. Now, I want to use the model for 4 classes classifier problem using the same network. Thus, I want to transfer all trained weights in the binary classifier to 4 classes problem, without the lass layer that will random initialization. How could I do it? This is my model

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5, 1)
        self.conv2 = nn.Conv2d(20, 50, 5, 1)
        self.conv_classify= nn.Conv2d(50, 2, 1, 1, bias=True) # number of class

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv_classify(x))
        return x

This what I did

model = Net ()
checkpoint_dict = torch.load('10000_model.pth')
pretrained_dict = checkpoint_dict['state_dict']
model_dict = model.state_dict()
# 1. filter out unnecessary keys
pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}
# 2. overwrite entries in the existing state dict
model_dict.update(pretrained_dict)
# 3. load the new state dict
model.load_state_dict(model_dict)

For now, I have to manually delete the pretrained_dict  by name.

pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}
pretrained_dict.pop('conv_classify.weight', None)
pretrained_dict.pop('conv_classify.bias', None)

It means pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict} does not do anything.

What is wrong? I am using pytorch 1.0. Thanks


title: Pytorch - Are gradients transferred on creation of new Variables?

body:
I have the following code:

A = Tensor of [186,3]

If I create a new empty tensor as follows:

tempTens = torch.tensor(np.zeros((186,3)), requires_grad = True).cuda()

And I apply some operations on a block of A and output it into tempTens, which I use totally for further computation, say like this:

tempTens[20,:] = SomeMatrix * A[20,:]

Will the gradients actually be transferred correctly, lets say I am having a cost function that optimizes for the output of tempTens to some ground truth


title: LSTM Architecture

body:
i'm trying to understand if my two layer LSTM is like the one in the image. Someone can help me?
p.s. i have cleared mean pooling layer.

input layer(green), LSTM layer hidden(blue), linear layer(yellow)

https://ibb.co/1qN6q1m

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.hidden_size = hidden_size
        self.num_layer=2
        self.i2h = nn.LSTM(input_size, hidden_size, self.num_layer, dropout=0.5)

        self.i2o = nn.Linear(hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=1)
        self.hidden = self.initHidden()

    def forward(self, input):
        lstmout, self.hidden= self.i2h(input.view(len(input), 1, -1), self.hidden)
        output = self.i2o(lstmout.view(len(input), -1))
        output = self.softmax(output)
        return output

    def initHidden(self):
        return (torch.zeros(self.num_layer, 1,  self.hidden_size).to(device),torch.zeros(self.num_layer, 1, self.hidden_size).to(device))


title: How to use pytorch dynamic graph property to train a model parallel with different hyperparamters?

body: Is there a standard way to train multiple models with pytorch by leveraging fro dynamic graph property?


title: Why is my classifier not giving me binary results?

body:
I created a linear ReLu network that is supposed to over-fit to my data. I use BCEWithLogisticLoss as the loss function. I was using it to classify 3d points. Since the data were small enough I didn't care to make into batches. And it worked fine. However now that I've implemented batches into it it seems that the predicted values aren't what I expect (i.e 0 or 1) instead it give me numbers like -25.4562 I didn't change anything else from the network only the batches.

I tried the binary loss function BSELoss however it seems to be a bug in the version of pytorch, so I can't use it. You can take a look at my code below:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# We load the training data
Samples, Ocupancy = common.load_samples()
for i in range(0,Ocupancy.shape[0]):
    if Ocupancy[i] > 1 or Ocupancy[i] < 0:
        print("upsie")

max = np.amax(Samples)
min = np.amin(Samples)

x_test = torch.from_numpy(Samples.astype(np.float32)).to(device)
y_test = torch.from_numpy(Ocupancy.astype(np.float32)).to(device)

train_data = CustomDataset(x_test, y_test)

train_loader = DataLoader(dataset=train_data, batch_size= 22500, shuffle=False) # Batches_size equal to the number of points in each slice

phi = common.MLP(3, 1).to(device)
criterion = torch.nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(phi.parameters(), lr = 0.01)
epoch = 10

fit_start_time = time.time()

for epoch in range(epoch):
    for x_batch, y_batch in train_loader:
        #optimizer.zero_grad()

        x_train = x_batch.to(device)

        y_train = y_batch.to(device)

        y_pred = phi(x_batch)
        print(y_pred)

        # Compute Loss
        loss = criterion(y_pred.squeeze(), y_batch.squeeze())

        print('Epoch {}: train loss: {}'.format(epoch, loss.item()))    # Backward pass

        loss.backward()
        optimizer.step()

fit_end_time = time.time()

print("Total time = %f" % (fit_end_time - fit_start_time))
min = -2
max = 2
resolution = 0.05
X,Y,Z  = np.mgrid[min:max:resolution,min:max:resolution,min:max:resolution] # sample way more

xyz = torch.from_numpy(np.vstack([X.ravel(), Y.ravel(),Z.ravel()]).transpose().astype(np.float32)).to(device)

eval = LabelData(xyz)

eval_loader = DataLoader(dataset=eval, batch_size= 22500, shuffle=False) # Make bigger batches

# feed the network bit by bit?
i = 0
for x_batch in eval_loader:
    phi.eval()
    labels =  phi(x_batch).to(device)
print(labels)

visualization_iso(X,Y,Z,labels)

I expect to the predicted values be 0 or 1, or at least be a probability however it gives me big numbers that I don't understand. Like :19.5953
Please take a look at my code and if you spot any big mistakes please tell me so. I'm really confused since it work quite well before I expanded the size of the data I was using.

Regards


title: How to convert data(tensor) across different deep learning frameworks directly in gpu without copying to cpu?

body: For example, I have a cuda tensor in pytorch, how to directly convert it to mxnet/tensorflow in gpu without copying it to cpu? Frequently copying data between cpu and gpu is very time-consuming.


title: Turn list of tuples into binary tensors?

body:
I have a list of tuples in the form below. A given tuple represents a given pair of movies that a given user liked. All tuples, together, capture every combination of movie likes found in my data.

[(movie_a,movie_b),...(movie_a,movie_b)]

My task is to create movie embeddings, similar to word embeddings. The idea is to train a single hidden layer NN to predict the most likely movie which any user might like, given a movie supplied. Much like word embeddings, the task is inconsequential; it's the weight matrix I'm after, which maps movies to vectors.

Reference: https://arxiv.org/vc/arxiv/papers/1603/1603.04259v2.pdf

In total, there are 19,000,000 tuples (training examples.) Likewise, there are 9,000 unique movie IDs in my data. My initial goal was to create an input variable, X where each row represented a unique movie_id, and each column represented a unique observation. In any given column, only one cell would be set to 1, with all other values set to 0.

As an intermediate step, I tried creating a matrix of zeros of the right dimensions

X = np.zeros([9000,19000000])

Understandably, my computer crashed, simply trying to allocate sufficient memory to X.

Is there a memory efficient way to pass my list of values into PyTorch, such that a binary vector is created for every training example?

Likewise, I tried randomly sampling 500,000 observations. But similarly, passing 9000,500000 to np.zeroes() resulted in another crash.

My university has a GPU server available, and that's my next stop. But I'd like to know if there's a memory efficient way that I should be doing this, especially since I'll be using shared resources.


title: customize output vector (instead of category) for neural network in python

body:
I'm trying to build a neural network with TensorFlow, PyTorch, Keras of other ML library.

The input is an image and the output is NOT a category but a vector of 100 elements (with a certain pattern for each image).

Each image has its own distinct pattern that is comprised of a vector of 100 elements, so when I'll get a new image I could predict its pattern.

The problem is that I can't use my own vectors as outputs, the only option I found is to give a category as an output...

Is there a way to establish my own output for each image (input) and let the neural network learn using those output vector (which has 100 elements each)?

I have tried looking for a way to provide my own output vectors by I can't find this option in any ML library.


title: Sparse vectors for training data

body:
I have a training data like this:

x_train = np.random.randint(100, size=(1000, 25))

where each row is a sample and thus we have 1000 samples.

Now I need to have the training data such that for each of the sample/row there can be at max 3 non-zero elements out of 25.

Can you all please suggest how I can implement that? Thanks!


title: Trying to learn how to implement a single Neuron

body:
I have this code in Pytorch but cant get it to work. I have it working with Numpy as return (X.T * W).sum(axis=1) + B

But with Pytorch I keep getting this error...

def neural_network_neurons(W, B, X):

    W = W.view(W.size(0), -1)
    z1 =  torch.matmul(X, W) + B
    return ReLU(z1)

# --------------------------------------------------
W = torch.tensor([[1.2, 0.3, 0.1], [.01, 2.1, 0.7]])
B = torch.tensor([2.1, 0.89])
X = torch.tensor([0.3, 6.8, 0.59])

neural_network_neurons(W, B, X)

---------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-21-8a5d3a425c16> in <module>
      4 X = torch.tensor([0.3, 6.8, 0.59])
      5
----> 6 neural_network_neurons(W, B, X)

<ipython-input-20-7450924eb4a5> in neural_network_neurons(W, B, X)
      5     ###
      6     W = W.view(W.size(0), -1)
----> 7     z1 =  torch.matmul(X, W) + B
      8     return ReLU(z1)
      9     #return (X.T * W).sum(axis=1) + B

RuntimeError: size mismatch, m1: [1 x 3], m2: [2 x 3] at
/pytorch/aten/src/TH/generic/THTensorMath.cpp:197


title: pytorch difference in 1.1.0 and 1.3.0 in Parameter

body:
I am trying to learn PyTorch and have tried running some code that I got from the Kaggle website.

# Get all hidden layers' weights
for i in range(len(hidden_units)):
  fc_layers.extend([
    TrainNet.model.hidden_layers[i].weight.T.tolist(), # weights
    TrainNet.model.hidden_layers[i].bias.tolist() # bias
  ])

This gives the following error:

AttributeError                            Traceback (most recent call last)
<ipython-input-12-65f871b6f0b7> in <module>
  4 for i in range(len(hidden_units)):
  5     fc_layers.extend([
----> 6         TrainNet.model.hidden_layers[i].weight.T.tolist(), # weights
  7         TrainNet.model.hidden_layers[i].bias.tolist() # bias
  8     ])

AttributeError: 'Parameter' object has no attribute 'T'

If I print out the type of 'TrainNet.model.hidden_layers[i].weight' it is indeed type Parameter.
This block of code works without error on the Kaggle website run in there notebook (Google colab I believe) where the version of Torch is 1.3.0.

On my home machine where the error occurs my Anaconda distribution which I have just updated is running Torch 1.1.0.

Is this the source of the error and how do I sort it?

Thanks


title: How to disable crash output in comet ml?

body:
Every time my script is crashing I have such output:

COMET INFO:     sys.cpu.percent.43       : (0.0, 0.0)
COMET INFO:     sys.cpu.percent.44       : (1.8, 1.8)
COMET INFO:     sys.cpu.percent.avg      : (6.579545454545454, 6.579545454545454)
COMET INFO:     sys.gpu.0.free_memory    : (34089664512.0, 34089664512.0)
COMET INFO:     sys.gpu.0.gpu_utilization: (0.0, 0.0)
COMET INFO:     sys.gpu.0.total_memory   : (34089730048.0, 34089730048.0)
COMET INFO:     sys.gpu.0.used_memory    : (65536.0, 65536.0)
COMET INFO:     sys.load.avg             : (39.42, 39.42)
COMET INFO:     sys.ram.total            : (1621711745024.0, 1621711745024.0)
COMET INFO:     sys.ram.used             : (78552326144.0, 78552326144.0)
COMET INFO:   Other [count]:
COMET INFO:     offline_experiment: True

How can I disable it?

In comet-ml docs I found but probably it's not what I look for:

  by setting the environmental variable COMET_DISABLE_AUTO_LOGGING to 1

$ export COMET_DISABLE_AUTO_LOGGING=1
$ echo $COMET_DISABLE_AUTO_LOGGING
1

But it didn't help me.