title: Building pytorch on the NVIDIA Jetson Nano developer kit

body:
I'm working on a Jetson Nano, and trying to install pytorch 1.4.0 onto it to run some toy experiments.
However, I'm running into a lot of trouble with this. After failing to leverage the prebuilt wheels, I've gone the way of building from scratch, but after a couple hours, it fails with the following error.
[3249/3931] Building NVCC (Device) object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/torch_cuda_generated_Unique.cu.o
FAILED: caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/torch_cuda_generated_Unique.cu.o
cd /home/workingdir/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda && /usr/bin/cmake -E make_directory /home/workingdir/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/. && /usr/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=Release -D generated_file:STRING=/home/workingdir/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/./torch_cuda_generated_Unique.cu.o -D generated_cubin_file:STRING=/home/workingdir/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/./torch_cuda_generated_Unique.cu.o.cubin.txt -P /home/workingdir/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/torch_cuda_generated_Unique.cu.o.Release.cmake
Killed
CMake Error at torch_cuda_generated_Unique.cu.o.Release.cmake:281 (message):
  Error generating file
  /home/workingdir/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/./torch_cuda_generated_Unique.cu.o

Does anyone know how to interpret this? Did I run out of memory/swap space?
Additionally, if anyone knows of an easier way to get pytorch>=1.1.0 on my nano, any tips would be appreciated :)
I followed this thread here both for the prebuilt installation and the scratch installation: https://forums.developer.nvidia.com/t/pytorch-for-jetson-nano-version-1-5-0-now-available/72048


title: How to use batch to run RNN model with texts that have different length in each row?

body:
I want to use torch.nn.utils.rnn.pack_padded_sequence in pytorch to deal with this problem, But I dont kown how to get torch variable when it has different size in the last dimension?
torch.nn.utils.rnn.pack_padded_sequence(input, lengths, batch_first=False, enforce_sorted=True)
input can be of size T x B x * where T is the length of the longest sequence (equal to lengths[0]), B is the batch size, and * is any number of dimensions (including 0)

pack_padded_sequence
I will get errors if I run the following code.
x_torch = torch.from_numpy(np.ones((2, 3)))
y_torch = torch.from_numpy(np.ones((2, 5)))
ans_torch = torch.cat([x_torch, y_torch], dim = 0)

# ouput
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 3 and 5 in dimension 1 at c:\a\w\1\s\windows\pytorch\aten\src\th\generic/THTensorMoreMath.cpp:1333

Help!


title: The inference file that goes into the entry point of PyTorchModel to be deployed does not have an effect to the output of the predictor

body:
I am currently running the code on AWS Sagemaker, trying to predict data using an already-trained model, accessed by MODEL_URL.
With the code below, the inference.py as the entry_point does not seem to have an effect on the result of the trained prediction model. Any changes in inference.py does not alter the output (the output is always correct). Is there something I am misunderstanding with how the model works? And how can I incorporate inference.py to the prediction model as the entry point?
role = sagemaker.get_execution_role()

model = PyTorchModel(model_data = MODEL_URL,
                            role = role,
                            framework_version = '0.4.0',
                            entry_point = '/inference.py',
                            source_dir = SOURCE_DIR)

predictor = model.deploy(instance_type = 'ml.c5.xlarge',
                                   initial_instance_count = 1,
                                   endpoint_name = RT_ENDPOINT_NAME)

result = predictor.predict(someData)


title: Im not running anything on GPU. But GPU stat is not clearing my usage

body:
GPUstat image
Im not running anything on gpu.
My ID is suil5044. GPUstat is not clearing my usage.
I think I finished it in the IDE but actually the code is still running on the server.
How do I kill my code still running on the server?
And not affect other users
Thanks!


title: Why does the pytorch .backward() method occupies two more CPU threads when I want to restrict it to use only one CPU thread?

body:
I am trying to deploy my service using CPU and pytorch. I am asked to restrict the single-process service to use only one CPU thread. I have assigned torch.set_num_interop_threads(1) and torch.set_num_threads(1). The question is that when I check the actual CPU thread number using pstree <pid> I got three. Later I found that the problem happened in a.backward(). If I delete this line, I got 1 threads; if I keep this line, I got 3 CPU threads. Why did it happen and what should I do to keep the CPU thread to be one?
I wrote a simple python3 script to reproduce the problem:
import torch
import time
while(1):
    w = torch.Tensor([2])
    w = torch.autograd.Variable(w, requires_grad=True)
    x = torch.rand([1])
    x = torch.autograd.Variable(x, requires_grad=True)
    y = x**w
    y.backward()
    time.sleep(3)

Run it and use pstree <pid> to check the CPU threads, I got python───2*[{python}], but if I delete y.backward() I got only python. I believe the extra two CPU threads in this script is in the same problem with my service code.


title: Custom weight initiatlization in conv1d pytorch

body:
I am trying to create custom weights in conv1d as follows:
import torch
from torch import nn
conv = nn.Conv1d(1,1,kernel_size=2)
K = torch.Tensor([[[0.5, 0.5]]])
with torch.no_grad():
conv.weight = K

But I am getting the error
"File “D:\ProgramData\Miniconda3\envs\pytorchcuda102\lib\site-packages\torch\nn\modules\module.py”, line 611, in setattr
raise TypeError(“cannot assign ‘{}’ as parameter ‘{}’ "
TypeError: cannot assign ‘torch.FloatTensor’ as parameter ‘weight’ (torch.nn.Parameter or None expected)”

What am I doing wrong?


title: Pythonic way of slicing array by list indices

body:
I was wondering what is the best way to perform a batched slice (torch/numpy)?
I know for constant slicing indices it is possible to perform this operation
batch_size = 2
data = torch.zeros((batch_size,1, 256, 256))
x_start = 10
x_stop = 20

y_start = 10
y_stop = 20
data[torch.arange(batch_size), :, y_start:y_stop, x_start:x_stop] = 1

But the question is how do I handle the case if the start and stop values are different?
eg.
batch_size = 2
data = torch.zeros((batch_size,1, 256, 256))
x_start = [10, 5]
x_stop  = [20, 30]

y_start = [10, 5]
y_stop = [20, 30]
data[torch.arange(batch_size), :, y_start:y_stop, x_start:x_stop] = 1 # crashes

I guess I could do it in a for loop, but I was wondering if there is a more pythonic way to do it.


title: How could I get predictions for my PyTorch Image Classifier?

body:
I have implemented this code for My Keras model that worked accurately but now I wanna implement this for my PyTorch Model, but I am unable to configure it for making predictions, Below is my full code, I need help with the classify function:
# importing libraries

import tkinter as tk
from tkinter import *
# import PIL
from tkinter import filedialog

import numpy
from PIL import Image, ImageTk
import torch

# importing model

model=torch.load('corelK_model_0.pt')

classes = {
    0:'africa',
    1: 'beach',
    2: 'tallbuilding',
    3: 'buses',
    4: 'dinosaurs',
    5: 'elephants',
    6: 'Roses',
    7: 'horses',
    8: 'mountains',
    9: 'food'
}

def upload_image():
    file_path = filedialog.askopenfilename()
    uploaded = Image.open(file_path)
    uploaded.thumbnail(((top.winfo_width() / 2.25, (top.winfo_height() / 2.25))))
    im = ImageTk.PhotoImage(uploaded)
    sign_image.configure(image=im)
    sign_image.image = im
    label.configure(text=' ')
    show_classify_button(file_path)

def show_classify_button(file_path):
    classify_btn = Button(top, text="Classify Image", command=lambda: classify(file_path), padx=10, pady=5)
    classify_btn.configure(background="#364156", foreground="white", font=('arial', 10, 'bold'))
    classify_btn.place(relx=0.79, rely=0.46)

def classify(file_path):
    image = Image.open(file_path)
    image = image.resize((32, 32))
    image = numpy.expand_dims(image, axis=0)
    image = numpy.array(image)
    pred = model.predict_classes([image])[0]
    sign = classes[pred]

    print(sign)
    label.configure(foreground='#011638', text=sign)

# initialize GUI
top = tk.Tk()  # calling the constructor or creating the object of tk class
top.geometry('800x600')  # set height and width
top.title("Image Classification CIFAR10")
top.configure(background="#CDCDCD")

# set Heading

heading = Label(top, text="Image Classifier", pady=20, font=('arial', 20, 'bold'))
heading.configure(background="#CDCDCD", foreground='#364156')
heading.pack()

upload = Button(top, text="Upload an image", command=upload_image, padx=10, pady=5)
heading.configure(background="#364156", foreground='white', font=('arial', 10, 'bold'))
upload.pack(side=BOTTOM, pady=50)

# upload image
sign_image = Label(top)
sign_image.pack(side=BOTTOM, expand=True)

# predicted class

label = Label(top, background="#CDCDCD", font=('arial', 15, 'bold'))
label.pack(side=BOTTOM, expand=True)

top.mainloop()

The model I have used is VGG16 after transfer learning, I have saved it using the torch.save .


title: App Crashing whle fetching .pt model file pytorch

body:
I'm trying to run my android app for pytorch model. After I load the image, When Detect button is pressed app is crashing.
Seems like app is crashing while fetching the modle file..
(Model file (.pt)is around 90 mb in size)
Is te issue coming from detectButton.setOnClickListener or fetchModelFile ?
how can i resolve this issue?
Main Activity.java
package com.mytorch.detector;

package com.tckmpsi.objectdetectordemo;

import androidx.appcompat.app.AppCompatActivity;

import android.content.Context;
import android.content.Intent;
import android.database.Cursor;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.graphics.drawable.BitmapDrawable;
import android.net.Uri;
import android.os.Build;
import android.os.Bundle;
import android.provider.MediaStore;
import android.util.Log;
import android.view.View;
import android.widget.Button;
import android.widget.ImageView;
import android.widget.TextView;
import android.widget.Toast;

import org.pytorch.IValue;
import org.pytorch.Module;
import org.pytorch.Tensor;
import org.pytorch.torchvision.TensorImageUtils;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

public class MainActivity extends AppCompatActivity {
    private static int RESULT_LOAD_IMAGE = 1;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        Button buttonLoadImage = (Button) findViewById(R.id.button);
        Button detectButton = (Button) findViewById(R.id.detect);

        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {
            requestPermissions(new String[]{android.Manifest.permission.READ_EXTERNAL_STORAGE}, 1);
        }
        buttonLoadImage.setOnClickListener(new View.OnClickListener() {

            @Override
            public void onClick(View arg0) {
                TextView textView = findViewById(R.id.result_text);
                textView.setText("");
                Intent i = new Intent(
                        Intent.ACTION_PICK,
                        MediaStore.Images.Media.EXTERNAL_CONTENT_URI);

                startActivityForResult(i, RESULT_LOAD_IMAGE);

            }
        });

        detectButton.setOnClickListener(new View.OnClickListener() {

            @Override
            public void onClick(View arg0) {

                Bitmap bitmap = null;
                Module module = null;

                //Getting the image from the image view
                ImageView imageView = (ImageView) findViewById(R.id.image);

                try {
                    //Read the image as Bitmap
                    bitmap = ((BitmapDrawable)imageView.getDrawable()).getBitmap();
                    Toast.makeText(getBaseContext(), "Read the image as Bitmap",
                            Toast.LENGTH_LONG).show();

                    //Here we reshape the image into 400*400
                    bitmap = Bitmap.createScaledBitmap(bitmap, 400, 400, true);

                    //Loading the model file.
                    module = Module.load(fetchModelFile(MainActivity.this, "coco.pt"));
                    Log.d("myTag", "This is fine");

                } catch (IOException e) {
                    Log.d("myTag", "This is my message"+ e);

                    finish();
                }

                //Input Tensor
                final Tensor input = TensorImageUtils.bitmapToFloat32Tensor(
                        bitmap,
                        TensorImageUtils.TORCHVISION_NORM_MEAN_RGB,
                        TensorImageUtils.TORCHVISION_NORM_STD_RGB
                );

                //Calling the forward of the model to run our input
                final Tensor output = module.forward(IValue.from(input)).toTensor();

              final float[] score_arr = output.getDataAsFloatArray();

                // Fetch the index of the value with maximum score
                float max_score = -Float.MAX_VALUE;
                int ms_ix = -1;
                for (int i = 0; i < score_arr.length; i++) {
                    if (score_arr[i] > max_score) {
                        max_score = score_arr[i];
                        ms_ix = i;
                    }
                }

                //Fetching the name from the list based on the index
              //  String detected_class = ModelClasses.MODEL_CLASSES[ms_ix];

                //Writing the detected class in to the text view of the layout
                TextView textView = findViewById(R.id.result_text);

                textView.setText("detected_class");

            }

        });

    }
    @Override
    protected void onActivityResult(int requestCode, int resultCode, Intent data) {
        //This functions return the selected image from gallery
        super.onActivityResult(requestCode, resultCode, data);

        if (requestCode == RESULT_LOAD_IMAGE && resultCode == RESULT_OK && null != data) {
            Uri selectedImage = data.getData();
            String[] filePathColumn = { MediaStore.Images.Media.DATA };

            Cursor cursor = getContentResolver().query(selectedImage,
                    filePathColumn, null, null, null);
            cursor.moveToFirst();

            int columnIndex = cursor.getColumnIndex(filePathColumn[0]);
            String picturePath = cursor.getString(columnIndex);
            cursor.close();

            ImageView imageView = (ImageView) findViewById(R.id.image);
            imageView.setImageBitmap(BitmapFactory.decodeFile(picturePath));

            //Setting the URI so we can read the Bitmap from the image
            imageView.setImageURI(null);
            imageView.setImageURI(selectedImage);

        }

    }

    public static String fetchModelFile(Context context, String modelName) throws IOException {
        File file = new File(context.getFilesDir(), modelName);
        if (file.exists() && file.length() > 0) {
            return file.getAbsolutePath();
        }

        try (InputStream is = context.getAssets().open(modelName)) {
            try (OutputStream os = new FileOutputStream(file)) {
                byte[] buffer = new byte[4 * 1024];
                int read;
                while ((read = is.read(buffer)) != -1) {
                    os.write(buffer, 0, read);
                }

                os.flush();
                Toast.makeText(context, "1",
                        Toast.LENGTH_LONG).show();

            }catch (Exception e){
                Toast.makeText(context, "issue here fuk",
                        Toast.LENGTH_LONG).show();

            }

            return file.getAbsolutePath();
        }

    }

}

dependencies
implementation 'org.pytorch:pytorch_android:1.5.0'
implementation 'org.pytorch:pytorch_android_torchvision:1.5.0'

Stack Trace for crash
2020-12-07 09:08:34.092 15502-15502/com.mytorch.detector E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.tckmpsi.objectdetectordemo, PID: 15502
    com.facebook.jni.CppException: [enforce fail at inline_container.cc:208] . file not found: archive/constants.pkl
    (no backtrace available)
        at org.pytorch.NativePeer.initHybrid(Native Method)
        at org.pytorch.NativePeer.<init>(NativePeer.java:24)
        at org.pytorch.Module.load(Module.java:23)
        at com.tckmpsi.objectdetectordemo.MainActivity$2.onClick(MainActivity.java:88)
        at android.view.View.performClick(View.java:7448)
        at android.view.View.performClickInternal(View.java:7425)
        at android.view.View.access$3600(View.java:810)
        at android.view.View$PerformClick.run(View.java:28305)
        at android.os.Handler.handleCallback(Handler.java:938)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loop(Looper.java:223)
        at android.app.ActivityThread.main(ActivityThread.java:7656)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)


title: Metrics mismatch between BertForSequenceClassification Class and my custom Bert Classification

body:
I implemented my custom Bert Binary Classification Model class, by adding a classifier layer on top of Bert Model (attached below). However, the accuracy/metrics are significantly different when I train with the official BertForSequenceClassification model, which makes me wonder if I am missing somehting in my class.
Few Doubts I have:
While loading the official BertForSequenceClassification from_pretrained are the classifiers weight initialized as well from pretrained model or they are randomly initialized? Because in my custom class they are randomly initialized.
class MyCustomBertClassification(nn.Module):
    def __init__(self, encoder='bert-base-uncased',
                        num_labels,
                        hidden_dropout_prob):

    super(MyCustomBertClassification, self).__init__()
    self.config  = AutoConfig.from_pretrained(encoder)
    self.encoder = AutoModel.from_config(self.config)
    self.dropout = nn.Dropout(hidden_dropout_prob)
    self.classifier = nn.Linear(self.config.hidden_size, num_labels)

def forward(self, input_sent):
    outputs = self.encoder(input_ids=input_sent['input_ids'],
                          attention_mask=input_sent['attention_mask'],
                          token_type_ids=input_sent['token_type_ids'],
                          return_dict=True)

    pooled_output = self.dropout(outputs[1])
    # for both tasks
    logits = self.classifier(pooled_output)

    return logits


title: PyTorch : Aggregate two models

body:
Hello and greetings from Greece
class Model(nn.Module):

def __init__(self, embedding_size, num_numerical_cols, output_size, layers, p=0.4):
    super().__init__()
    self.all_embeddings = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in embedding_size])
    self.embedding_dropout = nn.Dropout(p)
    self.batch_norm_num = nn.BatchNorm1d(num_numerical_cols)

    all_layers = []
    num_categorical_cols = sum((nf for ni, nf in embedding_size))
    input_size = num_categorical_cols + num_numerical_cols

    for i in layers:
        all_layers.append(nn.Linear(input_size, i))
        all_layers.append(nn.ReLU(inplace=True))
        all_layers.append(nn.BatchNorm1d(i))
        all_layers.append(nn.Dropout(p))
        input_size = i

    all_layers.append(nn.Linear(layers[-1], output_size))

    self.layers = nn.Sequential(*all_layers)

def forward(self, x_categorical, x_numerical):
    embeddings = []
    for i,e in enumerate(self.all_embeddings):
        embeddings.append(e(x_categorical[:,i]))
    x = torch.cat(embeddings, 1)
    x = self.embedding_dropout(x)

    x_numerical = self.batch_norm_num(x_numerical)
    x = torch.cat([x, x_numerical], 1)
    x = self.layers(x)
    return x

Suppose I have this nn for classification and I create two instances
model_1=Model(categorical_embedding_sizes, numerical_data.shape[1], 2, [200,100,50], p=0.4)
model_2=Model(categorical_embedding_sizes, numerical_data.shape[1], 2, [200,100,50], p=0.4)

Αnd after I trained these two models  i saved them with torch.save as model_1.pt and model_2.pt
Is there a way to create a new model with the mean parameters of the two models ?
something like
model_new.weight=(model_1.weight+model_2.weight)/2
model_new.bias=(model_1.bias+model_2.bias)/2

Thank you in advance


title: logistic regression model with L1 regularisations

body:
I am trying to apply L1 regularization on a logistic model
class LogisticRegression(nn.Module):

    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(input_size, num_classes)
    def forward(self, x):
        x = x.reshape(-1, 784)
        output = self.linear(x)
        return output

    def training_step(self, batch):
        images, labels = batch
        output = self(images)
        loss = F.cross_entropy(output, labels)

        acc = accuracy(output, labels)
        return {'Training_loss': loss, 'Training_acc': acc}

    def training_epoch_end(self, outputs):
        batch_losses = [x['Training_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()
        batch_accs = [x['Training_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()
        return {'Training_loss': epoch_loss.item(), 'Training_acc': epoch_acc.item()}

    def epoch_end(self, epoch, result):
        print("Epoch [{}], Training_loss: {:.4f}, Training_acc: {:.4f}".format(epoch, result['Training_loss'], result['Training_acc']))
model = LogisticRegression()

But I think I am doing it wrong the accuracy did not change.
L1=0.2
def evaluate(model_b, trainloader):
    outputs = [model_b.training_step(batch) for batch in trainloader]
    return model_b.training_epoch_end(outputs)

def fit(epochs, lr, model_b, trainloader, opt_func=torch.optim.SGD):
    history = []
    optimizer = opt_func(model_b.parameters(), lr)
    for epoch in range(epochs):
        ##### Training Phase
        for batch in trainloader:
            loss = model_b.training_step(batch)['Training_loss']

            loss_Lasso = loss  + 0.5 * L1 #  L1 reg

            loss_Lasso.backward()
            optimizer.step()
            optimizer.zero_grad()
        result = evaluate_b(model_b, trainloader)
        model_b.epoch_end(epoch, result)
        history.append(result)
    return history

Can anyone help me with what I am missing and how I can really apply  L1 regularization?
Also, is  L1 regularization called lasso?


title: How to do with the unbalanced dataset for multi label with bertforsequenceclassification?

body:
I have a BERT model of multi label classification with bertforsequenceclassification, and the dataset is highly unbalanced with 15 classes, as you see here is the pos_weights:
tensor([ 86.5999,   3.3153,   0.8365,   6.6842,  86.5999,   7.0367,   3.3582,
144.9998,  11.5143,  27.2581,  11.0000,  21.4615,  38.8182,   2.7597,
174.1996])
I feed this in the loss function torch.nn.BCEWithLogitsLoss(), however, most of the result turns out to 0 because when I apply sigmoid and make a threshold 0.5 to determine the value, most of them are close to 0.5 but still under 0.5. so I'm not sure if I handle the unbalanced data well,  or probably having other problems, also, when I tried to improve it, I actually don't know if I can simply set 0.5 as the threshold to this unbalanced case.
Did I miss any part of it? or what should I add if I want to deal with this problem?


title: PyTorch - Extremely simple backward() autograd raises CUDA OutOfMemory error?

body:
I turn to you with a problem regarding a custom, extremely simple, PyTorch autograd function I wrote. The autograd function performs a non-differentiable ray-intersection and backpropagates the gradients according to an analytic expression I derived on paper. However, when running it, even with an extremely simple NN (no hidden layer, 2 input nodes, 1 output node), I get a `CUDA Out Of Memory` error, regardless whether I'm running the script on my machine (6GB VRAM) or on Google Colab (16GB (!) VRAM). So there must be something wrong.
MWE to replicate the problem is below. The MLP predicts something, the forward function uses this prediction to perform the intersection and returns the intersection coordinates. Then we do some more stuff with the coordinates, but that's not important. In the backward pass, I want to calculate the gradients of the intersection coordinates w.r.t. the MLP prediction. This is where it fails.
Any help greatly appreciated!
import torch
import torch.nn.functional as F

device = 'cuda' if torch.cuda.is_available() else 'cpu'

class Intersector(torch.autograd.Function):

    @staticmethod
    def forward(ctx, *input):

        (mlp_pred, other_stuff) = input[:2]
        found_intersections = torch.zeros((3, 250, 250), device=device)

        # not differentiable intersection routine
        # (just a dummy, doesn't really interesect anything, normally this would use mlp_pred)
        with torch.no_grad():
            for k in range(100):
                other_stuff += i
                if i > 20:
                    found_intersections += 15.0
                    break

        ctx.mlp_prediction = mlp_pred                   # save stuff for backward pass
        ctx.found_intersections = found_intersections

        return found_intersections

    @staticmethod
    def backward(ctx, grad_output):
        # grad_output contains grad of loss w.r.t the found intersections
        # now: compute gradient of loss w.r.t input, i.e., w.r.t. mlp_pred

        mlp_pred = ctx.mlp_prediction
        intersections = ctx.found_intersections

        with torch.enable_grad():

            # get mlp prediction values at intersection coordinates by sampling the prediction at these values
            grid_x = intersections[0, :, :].unsqueeze(dim=0)
            grid_y = intersections[1, :, :].unsqueeze(dim=0)
            sampling_grid = torch.cat((grid_x, grid_y), dim=0).permute(1, 2, 0).unsqueeze(dim=0)
            mlp_pred_at_intersections = F.grid_sample(mlp_pred, sampling_grid, mode='bilinear', padding_mode='zeros')

            pred_sum = mlp_pred_at_intersections.sum()      # sum bc autograd needs scalar

            ### THIS IS WHERE THE OOM ERROR OCCURS
            gradient_wrt_mlp_pred = torch.autograd.grad(pred_sum, mlp_pred, only_inputs=True, retain_graph=True)

            grad_for_backprop = gradient_wrt_mlp_pred[0].to(device) * grad_output

        # grad. for other_stuff not needed, return None
        return grad_for_backprop, None

class SimpleModel(torch.nn.Module):

    def __init__(self):
        super(SimpleModel, self).__init__()

        # extremely simple MLP
        self.mlp = torch.nn.Sequential(
            torch.nn.Linear(in_features=2, out_features=1),
            torch.nn.LeakyReLU(),
        )

        self.calc_intersections = Intersector.apply
        self.other_stuff = torch.zeros((3, 250, 250), device=device)

    def forward(self, fw_input):

        # get MLP prediction
        mlp_prediction = self.mlp(fw_input).reshape((1, 1, 250, 250))

        # get intersection points w.r.t mlp prediction
        inputs_to_intersection = [mlp_prediction, self.other_stuff]
        intersections = self.calc_intersections(*inputs_to_intersection)

        # do some differentiable stuff w/ predictions
        intersections_altered = intersections + torch.ones_like(intersections, device=device)

        return intersections_altered

if __name__ == '__main__':

    model = SimpleModel().to(device)
    optim = torch.optim.Adam(model.mlp.parameters(), lr=1e-3)
    epochs = 200

    for i in range(epochs):

        # create dummy sample and gt
        sample = torch.zeros((2, 250, 250), device=device).reshape((250**2, 2))     # reshape array of x/y coords to format [batchsize,2]
        gt = torch.zeros((3, 250, 250), device=device)

        optim.zero_grad()

        model_out = model(sample).to(device)
        loss = torch.nn.L1Loss()(model_out, gt)
        loss.backward()

        optim.step()

        print("Epoch {}/{} - Loss: {}".format(i, epochs, loss.item()))


title: BERT- pythorch- regression task - predicting same score for each instance

body:
I have implemented bert model for a regression task. trained the model and predict, however, model predicts 0 for each instance.
In a nutshell, task was predicting the positivity/negativity of a scientific article between -1 and 1.
Obviously the problem turns out from train (model does not learn) because validation loss does not decrease as expected. I will share the code of my model and look forward to hearing any suggestions. Please, let me know if you need to see other parts of the code.
Thanks!
class CustomSciBERTModel(nn.Module):
    def __init__(self):
      super(CustomSciBERTModel, self).__init__()
      self.scibert = AutoModel.from_pretrained('allenai/scibert_scivocab_uncased')
      self.num_labels = 1
      self.linear_layer = nn.Linear(768, 1)

    def forward(self, ids, mask,labels):
      output = self.scibert(
            input_ids=ids,
            attention_mask=mask)

      logits = self.linear_layer(output[1])

      loss = None
      # while training return loss and while validation/ testing resturn logits
      if labels is not None:
        loss_fct = nn.MSELoss()
        #loss_fct = nn.CrossEntropyLoss() *************
        loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
        return loss
      else:
        return logits


title: Is there a good way to modify some values in a pytorch tensor while preserving the autograd functionality?

body:
Sometimes I need to modify some of the values in a pytorch tensor. For example, given a tensor x, I need to multiply its positive part by 2 and multiply its negative part by 3:
import torch

x = torch.randn(1000, requires_grad=True)
x[x>0] = 2 * x[x>0]
x[x<0] = 3 * x[x<0]

y = x.sum()
y.backward()

However such inplace operations always break the graph for autograd:
Traceback (most recent call last):
  File "test_rep.py", line 4, in <module>
    x[x>0] = 2 * x[x>0]
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

Therefore, so far I've been using the following workaround:
import torch

x = torch.randn(1000, requires_grad=True)
y = torch.zeros_like(x, device=x.device)

y[x>0] = 2 * x[x>0]
y[x<0] = 3 * x[x<0]

z = y.sum()
z.backward()

which results in manually creating new tensors. I wonder if there is a better way to do this.


title: Dockerfile: pip install fails with requirements.txt (but succeeds with individual packages)

body:
I'm trying to install some packages in a docker container, and there is a problem when installing from a requirements.txt file. This line:
RUN python3.8 -m pip install -r requirements.txt

fails with the error:
...
Collecting torch
  Downloading torch-1.8.0-cp38-cp38-manylinux1_x86_64.whl (735.5 MB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    torch from https://files.pythonhosted.org/packages/89/c1/72e9050d3e31e4df983f6e06799a1a4c896427c1e5645a6d810940944b60/torch-1.8.0-cp38-cp38-manylinux1_x86_64.whl#sha256=fa1e391cca3937d5dea31f31a1a80a01bd4a8062c039448c254bbf5a58eb0787 (from -r requirements.txt (line 3)):
        Expected sha256 fa1e391cca3937d5dea31f31a1a80a01bd4a8062c039448c254bbf5a58eb0787
             Got        d5466637c17c3ae0c81c00d93a0b7c8d8428cfd216f54953a11d0788ea7b74fb

The requirements.txt file is the following:
numpy
opencv-python
torch

However, when installing these packages one at a time everything works fine:
RUN python3.8 -m pip install numpy
RUN python3.8 -m pip install opencv-python
RUN python3.8 -m pip install torch

Any ideas how to solve this?
*** EDIT ***
Dockerfile up to that point:
FROM public.ecr.aws/lambda/python:3.8
COPY requirements.txt ./


title: ImageNet pretrained ResNet50 backbones are different between Pytorch and TensorFlow

body:
"Obviously!", you might say... But there's one significant difference that I have trouble explaining by the difference in random initialization.
Take the two pre-trained basenets (before the average pooling layer) and feed them with the same image, you will notice that the output features don't follow the same distribution. Specifically, TensorFlow's backbone has more inhibited features by the ReLU compared to Pytorch's backbone. Additionally, as shows in the third figure, the dynamic range is different between the two frameworks.

Of course, this difference is absorbed by the dense layer addressing the classification task, but: Can that difference be explained by randomness in the training process? Or training time? Or is there something else that would explain the difference?
Code to reproduce:
import imageio
import numpy as np
image = imageio.imread("/tmp/image.png").astype(np.float32)/255

import tensorflow as tf
inputs = image[np.newaxis]
model = tf.keras.applications.ResNet50(include_top=False, input_shape=(None, None, 3))
output = model(inputs).numpy()
print(f"TensorFlow features range: [{np.min(output):.02f};{np.max(output):.02f}]")

import torchvision
import torch
model = torch.nn.Sequential(*list(torchvision.models.resnet50(pretrained=True).children())[0:8])
inputs = torch.tensor(image).permute(2,0,1).unsqueeze(0)
output = model(inputs).detach().permute(0,2,3,1).numpy()
print(f"Pytorch features range: [{np.min(output):.02f};{np.max(output):.02f}]")

Outputting
TensorFlow features range: [0.00;25.98]
Pytorch features range: [0.00;12.00]

Note: it's similar to any image.


title: How can I backprop over PyTorch LSTM built from primitive tensor operations (eg not `nn.LSTM`)

body:
EDIT : the problem with my implementation was trying to extract my output, IE a one-hot vector, directly from the hidden state. I added a dense layer on top instead and it works fine.

I'm trying to make an LSTM from more primitive operations in PyTorch, and use torch.autograd features to backprop errors. I'd like it to be "online", in that h and c accumulate their state over time, and at each timestep there's 1 character in, and 1 out.

This is a character-level rnn, so:

my vocablulary is 30 characters (lowercase a-z, and some punctuation)
inp is a onehot vector of length 30.
h and c are of length 30 + 100. The first 30 of h is my "output"
my loss compares the target char, one-hot encoded, against those first 30 indices of h.
I accumulate loss over 10 steps, and then don't know how to properly backprop it. The following is a (poor) attempt.

TL;DR. How do I properly backprop for this LSTM?

    def ff(inp, h, c):
        xh = torch.cat((inp, h), 0)
        f = (xh @ Wf + bf).sigmoid()
        i = (xh @ Wi + bi).sigmoid()
        g = (xh @ Wg + bg).tanh() # C-bar, in some literature
        c = f * c + i * g
        o = (xh @ Wo + bo).sigmoid()
        h = o * c.tanh()
        return h, c

    loss = torch.zeros(1)
    def bp(out, target, lr):
        global Wf, Wi, Wg, Wo
        global bf, bi, bg, bo
        global h, c
        global loss

        # Accumulate loss every step
        loss += (-target * out[:out_n].softmax(dim=0).log()).sum()

        # Every 10 chars, run backprop
        if i % 10 == 0:
            loss.backward()

            with torch.no_grad():
                for param in [Wf, Wi, Wg, Wo, bf, bi, bg, bo]:
                    param -= lr * param.grad
                    param.grad.zero_()

            h.detach_()
            c.detach_()
            loss.detach_()
            loss = torch.zeros(1)

        return loss


title: Common variable name acronym for Pytorch or Tensorflow?

body: I have seen var name like ninp (num_input), nhid (num_hidden), emsize (embedding size) in pytorch example github repo. What are some of other common acronyms and their meaning/context?


title: How can I recover transformation that was applied in the DataLoader?

body:
We define some some DataSet with some randomized transforms that will be reapplied each time some particular image is loaded. Is it possible to also extract the transformation that was applied to the image with the image?
(I'd like to apply the transformation that was applied to the image to some other image too.)

import torchvision
import torchvision.transforms as transforms
import torch.utils.data

path = "/path/to/image/folders/" #should contain at least one folder with some images
transform = transforms.Compose([
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.RandomRotation(degrees=[0, 360]),
    transforms.ToTensor()
])
dataset = torchvision.datasets.ImageFolder(root=path, transform=transform)

dataloader = torch.utils.data.DataLoader(dataset=dataset, shuffle=True)

for input, target in dataloader:
    pass #get transform that was applied to input?


title: Need a vectorized solution in pytorch

body:
I'm doing an experiment using face images in PyTorch framework.  The input x is the given face image of size 5 * 5 (height * width) and there are 192 channels.

Objective: To obtain patches of x of patch_size(given as argument).

I have obtained the required result with the help of two for loops. But I want a better-vectorized solution so that the computation cost will be very less than using two for loops.

Used: PyTorch 0.4.1, (12 GB) Nvidia TitanX GPU.

The following is my implementation using two for loops

def extractpatches( x, patch_size): # x is bsx192x5x5
   patches = x.unfold( 2, patch_size ,  1).unfold(3,patch_size,1)
   bs,c,pi,pj, _, _  = patches.size()  #bs,192,
   cnt = 0
   p = torch.empty((bs,pi*pj,c,patch_size,patch_size)).to(device)
   s = torch.empty((bs,pi*pj, c*patch_size*patch_size)).to(device)
  //Want a vectorized method instead of two for loops below
   for i in range(pi):
      for j in range(pj):
          p[:,cnt,:,:,:] = patches[:,:,i,j,:,:]
          s[:,cnt,:] = p[:,cnt,:,:,:].view(-1,c*patch_size*patch_size)
          cnt  = cnt+1
   return s

Thanks for your help in advance.


title: How to select correct value of learning rate multiplier?

body: I want to manually select correct Learning rate in an image classification problem using Pytorch by running the model for few epochs. I have used LR scheduler to decay the learning rate and also have manipulated Learning rate in optimizer parameter group but i am unable to see any change in Loss.


title: Image deconvolution with a CNN

body:
I have an input tensor of shape (C,H,W), where H=W and C=W^2.  This tensor contains non-linearly transformed information for an image of shape (1,H,W) squeezed to (H,W).  The exact form of the transformation is not important (plus, there is no closed-form expression for it anyway).  I would like to design a CNN to estimate images from such tensors.  I realize that I will have to experiment with CNN architectures (since I don't have the exact form of the transformation), but I'm not exactly sure how to proceed.

The input tensor has both positive and negative values which are important for the image reconstruction, so a ReLU layer probably should not be implemented near the beginning of CNN.  I don't think that pooling layers would be useful, either, at least in the H and W dimensions.  Clearly, I have to collapse the C dimension to get the image, but I don't think that it should be done all at once, e.g., torch.nn.Conv2d( C, 1, kernel_size ) is probably not a good idea.

It seems to me that I should first use a Conv2D layer which produces the same size tensor as the input tensor (to partially unscramble the non-linear transformation), but if the kernel size is greater than one the H and W dimensions will be reduced in size, which I don't want (unless this can be addressed later in the CNN).  On the other hand, if the kernel size is one the shape will stay the same but I don't think that anything happens to the tensor in this case.  Also, I will probably have to include linear layers, but I'm not sure how to use them with 3D tensors.

Any suggestions would be welcome.


title: Resolution independent UNet implementation?

body:
I'm wondering what's the easiest way to make a UNet model work with varying input resolutions? What I mean is that if I downsample 3 times, then with e.g. a 320x320 size image the spatial dimensions of feature layers will be:

320 -> 160 -> 80 -> 40 -> 80 -> 160 -> 320

On the other hand, if I input an image of size, say, 960x540, then I get

540 -> 270 -> 135(!) -> 68 -> 136(!) ->...

The network will blow up with such an input image, because the tensor sizes have a mismatch when concatenating the tensor from the skip.

Is there some standard trick for handling this?


title: How to create categories from image titles when using ImageDataBunch.from_folder()?

body:
When using ImageDataBunch.from_name_re(), fastAI takes care of creating the categories by using the name of each image. We simply have to input a regex string like this:

file_parse = r'/([^/]+)_\d+\.(png|jpg|jpeg)$'

Is there a way to do this when using ImageDataBunch.from_folder()?

np.random.seed(42)
data = ImageDataBunch.from_folder('/content/Food-101/images', train='/content/Food-101/images/train', test='/content/Food-101/images/test', valid_pct=0.2, ds_tfms=get_transforms(), size=224)
data.normalize(imagenet_stats)

Because when I am running epocs, this is what fastAI is giving me:

epoch   train_loss  valid_loss  error_rate  time
0       0.000000    0.000000    0.000000    21:05
1       0.000000    0.000000    0.000000    20:43
2       0.000000    0.000000    0.000000    20:38
3       0.000000    0.000000    0.000000    19:38
4       0.000000    0.000000    0.000000    20:01

I believe this is because my file_parse variable isn’t used anywhere.


title: seq2seq Pytorch- Error while trying to calculate metrics (precision, recall, F1 score)

body:
I run over this tutorial of seq2seq and saved the models every 1000 iterations.
https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation.ipynb
I try to calculate metrics and analyze the results, so I want to calculate Precision, Recall and F1 score for new test set.
I build this script:
import torch
import pandas as pd
from sklearn.metrics import precision_recall_fscore_support
import seq2seq_tutorial
max_length=50

test_vectors = pd.read_pickle("Pickels/first-second.pkl")
y_true = list(test_vectors[1])

test_vectors = [list(map(str, list(test_vectors.iloc[i])[0])) for i in range(len(test_vectors))]
test_vectors = [("".join(sent)).strip() for sent in test_vectors]

results = []
for k in range(1000, 75001, 1000):
    print("K: " + str(k))
    encoder = torch.load("Models/seq2seq/encoder_" + str(k) + "_steps.pth")
    decoder = torch.load("Models/seq2seq/decoder_" + str(k) + "_steps.pth")

    predictions_for_model_k = []
    for i in range(len(test_vectors)):
        try:
            prediction = explained_code.evaluate(encoder, decoder, test_vectors[i],max_length)
            prediction = prediction[0][0]
            predictions_for_model_k.append(prediction)
        except:
            print("Exception at %s" % i)

    accuracy = precision_recall_fscore_support(y_true, predictions_for_model_k, average='weighted')
    precision = accuracy[0]
    recall = accuracy[1]
    f_score = accuracy[2]

    print("Precision: " + str(round(precision * 100, 3)))
    print("Recall: " + str(round(recall * 100, 3)))
    print("F_score: " + str(round(f_score * 100, 3)))

    results.append([str(k) + "steps", predictions_for_model_k, y_true, precision, recall, f_score])

pd.to_pickle(results, "Pickles/Results/results.pkl")

But I got this error:
AttributeError: Can't get attribute 'EncoderRNN' on <module '__main__'

Can you help with this issue?
Thank you!


title: Pytorch Import Error: : object file has no loadable segments

body:
I've been trying to install the Pytorch module for my Ubuntu 16.04 LTS through conda. I used conda install pytorch torchvision cpuonly -c pytorch to install it (non CUDA version). However when I type import torch on the Python shell, this is what I see -
ImportError: /home/student/anaconda2/lib/python2.7/site-packages/torch/_C.so: object file has no loadable segments

I have verified that Pytorch was installed using conda list


title: Pegasus Model for Transfer Learning is generating garbage summaries (Pytorch - transfomers library)

body:
I am using the PegasusForConditionalGeneration from transformers library for transfer learning and generate summaries of chats based on samsum dataset. But, the output generated is repetitive and redundant words are being generated with junk characters.
Also, getting loss as nan. Can someone help me regarding what went wrong?
Please note that the same code works perfectly fine for T5ForConditionalGeneration.
Is there any difference in the implementation method?
Should I try with other Loss Metrics or Optimizer?
Here is a snippet of my code where I am training the model
      for _ in range(epochs):
            self.model.train()
            train_loss = 0
            for idx, data in tqdm(enumerate(self.train_loader)):
                self.optimizer.zero_grad()
                output = self.model(input_ids = data["input_ids"], attention_mask = data["attention_mask"], lm_labels = data["lm_labels"])
                loss, prediction_scores = output[:2]
                train_loss += loss.item()
                loss.backward()
                self.optimizer.step()
                if((idx % 1000) == 0):
                    print("loss: ", loss.item(), " train_loss: ", train_loss/(idx+1))

As the name of each variable suggests, I’ve PegasusForConditionalGeneration in the variable self.model, Adam Optimizer in self.optimizer and self.train_loader is of type DataLoader.\
Thanks in Advance !


title: Large image for semantic segmentation task -- Pytorch runs out of memory

body:
I have large images (256,256) of ImageNet, and I want to train for a semantic segmentation task. The output will thus be (B,Classes,C,H,W). I have 256 classes. For CrossEntropyLoss or NLL as defined by pytorch I need to use dtype long Docs.
However a tensor of (256,3,256,256) of type long is just over 0.4GB of memory making it impossible to train in GPU for batch sizes over 4 (considering that the model takes significant memory as well as backprop algorithm).
I am using Google Cloud T4 with 16GB of memory. Are there any tricks I can do for this problem in order to train for larger batch sizes? For the batch sizes I can train it would take months to finish training.


title: Python3 Pytorch RuntimeError on GCP - no msg

body:
My System
I am running a neural network training on using Python 3.6.9 with pytorch 1.6.0
I am using a google cloud platform N1 Server with a Tesla T4, 2 cores CPU, 12GB RAM.
This is on an Ubuntu 18.04 image.
Problem
When my code reaches the training line I get the following RuntimeError with no real explanation that I can see:
-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/or/.local/share/virtualenvs/or-M3_AaJfY/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/home/or/my_model/train.py", line 88, in train_and_eval
    train(rank, epoch, hps, generator, optimizer_g, train_loader, logger, writer)
  File "/home/or/my_model/train.py", line 117, in train
    scaled_loss.backward()
  File "/home/or/.local/share/virtualenvs/or-M3_AaJfY/lib/python3.6/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/or/.local/share/virtualenvs/or-M3_AaJfY/lib/python3.6/site-packages/torch/autograd/__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError

This happens while the 2 CPU cores are being used at 100% for a long while.
The RAM and GPU, though going up (as expected while training) do not reach close to their
limit.
I checked journalctl to see if this was an operating system issue but there is nothing  there. I also did not find anything relevant in the /var/log/ directory or using dmesg.
I would be happy to provide more log data but I am not aware (after searching) any python logs I can look at, or any other system logs.

Please let me know of ways I can get more information if you have any ideas.
The exact same code works 100% fine on other physical machines I have tested, and a GPU only version of it runs fine on another cloud computing provider
What I am looking for

Ways to get more information about this problem and figure out why it is happening.
Ways to fix this problem

Thanks in advance for your time and any help you may be able to provide.


title: How to load the pth file for DistributedDataparallel model?

body:
I trained the model with DistributedDataParallel and made the pth file
  if args.gpu is not None:
            print('Gpu setting...',args.gpu)
            torch.cuda.set_device(args.gpu)
            model.cuda(args.gpu)
            # When using a single GPU per process and per
            # DistributedDataParallel, we need to divide the batch size
            # ourselves based on the total number of GPUs we have
            args.batch_size = int(args.batch_size / ngpus_per_node)
            args.workers = int((args.workers + ngpus_per_node - 1) / ngpus_per_node)
            model = torch.nn.parallel.DistributedDataParallel(
                model, device_ids=[args.gpu]
                #,output_device=[args.gpu]
                ,find_unused_parameters=True)

Then I try to evaluate the model
 self.model = EfficientDet(num_classes=num_class,
                                  network=network,
                                  W_bifpn=EFFICIENTDET[network]['W_bifpn'],
                                  D_bifpn=EFFICIENTDET[network]['D_bifpn'],
                                  D_class=EFFICIENTDET[network]['D_class'],
                                  is_training=False
                                  )

        #self.model = torch.nn.parallel.DistributedDataParallel(self.model, device_ids=[args.gpu],find_unused_parameters=True)
        #self.model =  torch.nn.parallel.DistributedDataParallel(self.model)

        if(self.weights is not None):
            print('load state dic...',self.weights)
            checkpoint = torch.load(
                self.weights, map_location=lambda storage, loc: storage)
            state_dict = checkpoint['state_dict']
            self.model.load_state_dict(state_dict)
        if torch.cuda.is_available():
            self.model = self.model.cuda()
        self.model.eval()

This gives key missing error. how should I load the DistributedDataParallel trained pth file?


title: Training results are different for Classification using Pytorch APIs and Fast-ai

body:
I have two training python scripts. One using Pytorch's API for classification training and another one is using Fast-ai.
Using Fast-ai has much better results.
Training outcomes are as follows.
Fastai
epoch     train_loss  valid_loss  accuracy  time
0         0.205338    2.318084    0.466482  23:02
1         0.182328    0.041315    0.993334  22:51
2         0.112462    0.064061    0.988932  22:47
3         0.052034    0.044727    0.986920  22:45
4         0.178388    0.081247    0.980883  22:45
5         0.009298    0.011817    0.996730  22:44
6         0.004008    0.003211    0.999748  22:43

Using Pytorch
Epoch [1/10], train_loss : 31.0000 , val_loss : 1.6594, accuracy: 0.3568
Epoch [2/10], train_loss : 7.0000 , val_loss : 1.7065, accuracy: 0.3723
Epoch [3/10], train_loss : 4.0000 , val_loss : 1.6878, accuracy: 0.3889
Epoch [4/10], train_loss : 3.0000 , val_loss : 1.7054, accuracy: 0.4066
Epoch [5/10], train_loss : 2.0000 , val_loss : 1.7154, accuracy: 0.4106
Epoch [6/10], train_loss : 2.0000 , val_loss : 1.7232, accuracy: 0.4144
Epoch [7/10], train_loss : 2.0000 , val_loss : 1.7125, accuracy: 0.4295
Epoch [8/10], train_loss : 1.0000 , val_loss : 1.7372, accuracy: 0.4343
Epoch [9/10], train_loss : 1.0000 , val_loss : 1.6871, accuracy: 0.4441
Epoch [10/10], train_loss : 1.0000 , val_loss : 1.7384, accuracy: 0.4552

Using Pytorch is not converging.
I used the same network (Wideresnet22) and both are trained from scratch without pretrained model.
The network is here.
Training using Pytorch is here.
Using Fastai is as follows.
from fastai.basic_data import DataBunch
from fastai.train import Learner
from fastai.metrics import accuracy

#DataBunch takes data and internall create data loader
data = DataBunch.create(train_ds, valid_ds, bs=batch_size, path='./data')
#Learner uses Adam as default for learning
learner = Learner(data, model, loss_func=F.cross_entropy, metrics=[accuracy])
#Gradient is clipped
learner.clip = 0.1

#learner finds its learning rate
learner.lr_find()

learner.recorder.plot()

#Weight decay helps to lower down weight. Learn in https://towardsdatascience.com/
learner.fit_one_cycle(5, 5e-3, wd=1e-4)

What could be wrong in my training algorithm using Pytorch?


title: Pytorch iOS - torch::jit::load fails with "Unknown Type name Error"

body:
I am trying to implement Fast Neural Style Transfer on iOS application. To do that, I started with an implemented version of Fast Neural Style Transfer from here and attempt to follow Pytorch Mobile tutorial to run the model on my device.
What I did was to change the model.pt in the tutorial file with my model.pt generated. I generate my model.pt following the steps in the Pytorch tutorial. Now, back to the XCode side, when I run my code, I encounter this error

Unknown type name
'torch.torch.classes.xnnpack.TransposeConv2dOpContext': Serialized
File "code/torch/transformer/___torch_mangle_391.py", line 18
annotations["prepack_folding._jit_pass_packed_weight_11"] = torch.torch.classes.xnnpack.Conv2dOpContext   annotations["prepack_folding._jit_pass_packed_weight_12"] = torch.torch.classes.xnnpack.Conv2dOpContext   annotations["prepack_folding._jit_pass_packed_weight_13"] = torch.torch.classes.xnnpack.TransposeConv2dOpContext
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
annotations["prepack_folding._jit_pass_packed_weight_14"] = torch.torch.classes.xnnpack.TransposeConv2dOpContext   annotations["prepack_folding._jit_pass_packed_weight_15"] = torch.torch.classes.xnnpack.Conv2dOpContext (lldb)

Within Xcode, the code fails here
    private lazy var module: TorchModule = {
    // Verified that filePath exist.
    if let filePath = Bundle.main.path(forResource: "mobile_model2", ofType: "pt"),
        // The problem is at TorchModule init setup.
        let module = TorchModule(fileAtPath: filePath) {
        return module
    } else {
        fatalError("Can't find the model file!")
    }
}()

I verified that the Bundle was able to find the mobile_model2.pt file.
Edit**
I found out that the code is failing within TorchModule here
- (nullable instancetype)initWithFileAtPath:(NSString*)filePath {
  self = [super init];
  if (self) {
    try {
      // CODE FAILS HERE, when trying to load!
      _impl = torch::jit::load(filePath.UTF8String);
      _impl.eval();
    } catch (const std::exception& exception) {
      NSLog(@"%s", exception.what());
      return nil;
    }
  }
  return self;
}

This means that the C++ load method call wasn't able to load my model.pt. What could possibly caused this?


title: PyTorch - Convert CIFAR dataset to `TensorDataset`

body:
I train ResNet34 on CIFAR dataset. For a certain reason, I need to convert the dataset into TensorDataset.
My solution is based on this: https://stackoverflow.com/a/44475689/15072863 with some differences (maybe they are critical, but I don't see why).
It looks I'm not doing this correctly.
Train loader:
transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])

train_ds = torchvision.datasets.CIFAR10('/files/', train=True, transform=transform_train, download=True)

xs, ys = [], []
for x, y in train_ds:
  xs.append(x)
  ys.append(y)

# 1) Standard Version
# cifar_train_loader = DataLoader(train_ds, batch_size=batch_size_train, shuffle=True, num_workers=num_workers)

# 2) TensorDataset version, seems to be incorrect
cifar_tensor_ds = TensorDataset(torch.stack(xs), torch.tensor(ys, dtype=torch.long))
cifar_train_loader = DataLoader(cifar_tensor_ds, batch_size=batch_size_train, shuffle=True, num_workers=num_workers)

I don't think it matters, but test loader is defined as usual:
transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])

cifar_test_loader = DataLoader(
  torchvision.datasets.CIFAR10('/files/', train=False, transform=transform_test, download=True),
  batch_size=batch_size_test, shuffle=False, num_workers=num_workers)

I know that something is wrong with how I use TensorDataset, since;

With TensorDataset I achieve 100% train accuracy, 80% test accuracy
With standard Dataset I achieve 99% train accuracy (never 100%), 90% test accuracy.

So, what am I doing wrong?
P.S.: My final goal is to split the dataset into 10 datasets based on their class. Is there a better way to do this? Of course, I can define my subclass of DataSet, but manually splitting it and creating TensorDataset's seemed to be simpler.


title: Binary classification with PyTorch

body:
Below is code I've written for binary classification in PyTorch:
%reset -f

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import numpy as np
import matplotlib.pyplot as plt
import torch.utils.data as data_utils
import torch.nn as nn
import torch.nn.functional as F

device = 'cpu'
num_epochs = 10

hidden_size = 500
num_classes = 2

learning_rate = .001

torch.manual_seed(24)

x1 = np.array([0,0])
x2 = np.array([0,1])
x3 = np.array([1,0])
x4 = np.array([1,1])
x = torch.tensor([x1,x2,x3,x4]).float()
y = torch.tensor([1,0,1,1]).long()

train = data_utils.TensorDataset(x,y)
train_loader = data_utils.DataLoader(train , batch_size=2 , shuffle=True)

input_size = len(x[0])

def weights_init(m):
    if type(m) == nn.Linear:
        m.weight.data.normal_(0.0, 1)

class NeuralNet(nn.Module) :
    def __init__(self, input_size, hidden_size, num_classes) :
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size , hidden_size)
        self.fc2 = nn.Linear(hidden_size , 100)
        self.fc3 = nn.Linear(100 , num_classes)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x) :
        out = self.fc1(x)
        out = self.sigmoid(out)
        out = self.fc2(out)
        out = self.sigmoid(out)
        out = self.fc3(out)
        out = self.sigmoid(out)
        return out

model = NeuralNet(input_size, hidden_size, num_classes).to(device)
model.apply(weights_init)

criterionCE = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
loss_values = []

for i in range(0 , 140) :
    print('in i' , i)
    total_step = len(train_loader)
    for epoch in range(num_epochs) :
        for i,(images , labels) in enumerate(train_loader) :
            images = images.to(device)
            labels = labels.to(device)

            outputs = model(images)
            print('outputs' , outputs)
            loss = criterionCE(outputs , labels)
            loss_values.append(loss)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

    outputs = model(x)

    print(outputs.data.max(1)[1])

This code throws an error :
~/opt/miniconda3/envs/ds1/lib/python3.8/site-packages/torch/nn/functional.py in binary_cross_entropy(input, target, weight, size_average, reduce, reduction)
   2067                       stacklevel=2)
   2068     if input.numel() != target.numel():
-> 2069         raise ValueError("Target and input must have the same number of elements. target nelement ({}) "
   2070                          "!= input nelement ({})".format(target.numel(), input.numel()))
   2071

ValueError: Target and input must have the same number of elements. target nelement (2) != input nelement (4)

I printed where I think is the cause of the error:
print('outputs' , outputs)

renders
outputs tensor([[9.9988e-01, 1.4011e-05],
        [9.9325e-01, 1.2087e-05]], grad_fn=<SigmoidBackward>)

So outputs should be a 2x1 instead of a 2x2 result. Have I not defined the setup of the model correctly? As I'm using sigmoid shouldn't a 2x1 output be created instead of 2x2?


I put together a LSTM model and it works. But only as long as I set num_layers = 1.
If I set it for example to 2 it gives my a long error message that tells me:
RuntimeError: Expected hidden[0] size (2, 1, 100), got (1, 1, 100)
I am pretty new at Python and deep learning in general, so I could need some advice how to fix my problem.
Code:
import torch
import torch.nn as nn
import numpy as np
import dataset
import time

amount_hidden_layers = 2
amount_neurons_hidden_layers = 100

input_layer_size = 12

output_layer_size = 48
small = True
hours = 20
learning_rate = 1/1000 #0,001
early_stoping = False

train_obs_dataset = dataset.return_observation_dataset(hours=hours, split="train", small=small)
val_obs_dataset = dataset.return_observation_dataset(hours=hours, split="val", small=small)
test_obs_dataset = dataset.return_observation_dataset(hours=hours, split="test", small=small)

class LSTM(nn.Module):
    def __init__(self, input_layer_size, hidden_layer_size, output_layer_size,
                 num_layers):
        super().__init__()
        self.hidden_layer_size = hidden_layer_size
        self.input_layer_size = input_layer_size
        self.num_layers = num_layers
        self.output_layer_size = output_layer_size

        # lstm_layers:
        self.lstm = nn.LSTM(input_size=self.input_layer_size, hidden_size=self.hidden_layer_size, num_layers=num_layers)

        self.hidden_cell = (torch.zeros(self.num_layers, 1, self.hidden_layer_size),
                            torch.zeros(self.num_layers, 1, self.hidden_layer_size))
        # output_layer:
        self.linear = nn.Linear(self.hidden_layer_size, self.output_layer_size)

    def forward(self, input_seq):
        lstm_out, self.hidden_cell = self.lstm(input_seq.view(len(input_seq), 1, -1), self.hidden_cell)
        predictions = self.linear(lstm_out.view(len(input_seq), -1))
        return predictions[-1]

model = LSTM(input_layer_size=input_layer_size, hidden_layer_size=amount_neurons_hidden_layers,
             output_layer_size=output_layer_size, num_layers=amount_hidden_layers)
loss_function = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

def training(dataset=train_obs_dataset, epochs=5):
    start_time = time.time()
    val_test = float(100000000)                                                     # for early stoping, saves the last evaluation value
    val_worse_counter = 0                                                           # for early stoping, couts how often tests with validation data get worse
    for i in range(epochs):                                                         # every epoch
        for datapoint in range(dataset[0].__len__()):                               # every datapoint
            optimizer.zero_grad()
            model.hidden_cell = (torch.zeros(1, 1, model.hidden_layer_size),
                                    torch.zeros(1, 1, model.hidden_layer_size))

            y_pred = model(torch.FloatTensor(dataset[0][datapoint]))                # model needs a tensor of shape [sequence][features]
            # squeeze: does not change data, prevents error,
            single_loss = loss_function(y_pred, torch.FloatTensor(dataset[1][datapoint]).squeeze(-1))

            single_loss.backward()
            optimizer.step()
            if datapoint%100 == 1:
                print(f' Time: {time.time() - start_time: 10.1f} sec,   Epoch: {i:4} Datapoint: {datapoint:3} Loss:  {single_loss.item():10.8f} ')
                #print(y_pred[0])
            "early stoping: if tests with the validation data does not get better for 100 steps, the model stops"
            if (early_stoping == True):
                if datapoint%20 == 1:
                    val = test(val_obs_dataset)
                    if (val[0] <= val_test):
                        val_test = val[0]
                        val_worse_counter = 0
                    if(val[0] > val_test):
                        val_worse_counter = val_worse_counter + 1
                    if(val[0] >= 100):
                        break
                    print("Val: ", val[0], val[1][0], "val_test: ", val_test, " val_worse_counter: ", val_worse_counter)

"""
returns a tupel with 1. the average loss and 2. an array with the losses for average losses seperated for all 48 hours
"""
def test(dataset=val_obs_dataset):
    with torch.no_grad():
        losses = []                 #sperated per hour
        loss = 0
        for i in range(48):
            losses.append(0)
        for datapoint in range(dataset.__len__()):
            val_pred = model(torch.FloatTensor(dataset[0][datapoint]))
            "seperates the loss values per hour"
            for i in range(48):
                losses[i] = losses[i] + float(loss_function(val_pred[i], torch.FloatTensor(dataset[1][datapoint]).squeeze(-1)[i]))
            loss = loss + float(loss_function(val_pred, torch.FloatTensor(dataset[1][datapoint]).squeeze(-1)))
        for i in range(48):                                         # calculates the average
            losses[i] = losses[i]/dataset.__len__()
            loss = loss/dataset.__len__()
    return (loss, losses)

print(model)

training(epochs=200)

Thanks in advance for every helpfull comment


title: Pytorch pairwise concatenation of tensors

body:
I'd like to compute a pairwise concatenation over a specific dimension in a batched manner.
For instance,
x = torch.tensor([[[0],[1],[2]],[[3],[4],[5]]])
x.shape = torch.Size([2, 3, 1])

I would like to get y such that y is the concatenation of all pairs of vectors across one dimension, ie:
y = torch.tensor([[[[0,0],[0,1],[0,2]],[[1,0],[1,1],[1,2]], [[2,0], [2,1], [2,2]]],
                 [[[3,3],[3,4],[3,5]],[[4,3],[4,4],[4,5]], [[5,3],[5,4],[5,5]]]])

y.shape = torch.Size([2, 3, 3, 2])

So essentially, for each x[i,:], you generate all pairs of vectors and you concatenate them on the last dimension.
Is there a straightforward way of doing that?


title: tgt and src have to have equal features for a Transformer Network in Pytorch

body:
I am attempting to train EEG data through a transformer network. The input dimensions are 50x16684x60 (seq x batch x features) and the output is 16684x2. Right now I am simply trying to run a basic transformer, and I keep getting an error telling me
RuntimeError: the feature number of src and tgt must be equal to d_model

Why would the source and target feature number ever be equal? Is it possible to run such a dataset through a transformer?
Here is my basic model:
input_size = 60 # seq x batch x features
hidden_size = 32
num_classes = 2
learning_rate = 0.001
batch_size = 64
num_epochs = 2
sequence_length = 50
num_layers = 2
dropout = 0.5

class Transformer(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(Transformer, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        self.transformer = nn.Transformer(60, 2)
        self.fc = nn.Linear(hidden_size * sequence_length, num_classes)

    def forward(self, x, y):

        # Forward Propogation
        out, _ = self.transformer(x,y)
        out = out.reshape(out.shape[0], -1)
        out = self.fc(out)
        return out

model = Transformer(input_size, hidden_size, num_layers, num_classes)

criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

for epoch in range(num_epochs):
    for index in tqdm(range(16684)):
        X, y = (X_train[index], Y_train[index])
        print(X.shape, y.shape)

        output = model(X, y)

        loss = criterion(output, y)

        model.zero_grad()
        loss.backward()

        optimizer.step()

        if index % 500 == 0:
            print(f"Epoch {epoch}, Batch: {index}, Loss: {loss}")


title: Worker timeout when preloading Pytorch model in Flask app on Render.com

body:
In my app.py I have a function that uses a pretrained Pytorch model to generate keywords
@app.route('/get_keywords')
def get_keywords():
    generated_keywords = ml_controller.generate_keywords()
    return jsonify(keywords=generated_keywords)

and in ml_controller.py I have
def generate_keywords():
    model = load_keywords_model()
    output = model.generate()
    return output

This is working fine. Calls to /get_keywords correctly return the generated keywords. However this solution is quite slow since the model gets loaded on each call. Hence I tried to load the model just once by moving it outside my function:
model = load_keywords_model()

def generate_keywords():
    output = model.generate()
    return output

But now all calls to /get_keywords time out when I deploy my app to Render.com. (Locally it's working.) Strangely the problem is not that the model does not get loaded. When I write
model = load_keywords_model()
testOutput = model.generate()
print(testOutput)

def generate_keywords():
    output = model.generate()
    return output

a bunch of keywords are generated when I boot gunicorn. Also, all other endpoints that don't call ml_controller.generate_keywords() work without problems.
For testing purposes I also added a dummy function to ml_controller.py that I can call without problems
def dummy_string():
    return "dummy string"

Based on answers to similar problems I found, I'm starting Gunicorn with
gunicorn app:app --timeout 740 --preload --log-level debug

and in app.py I'm using
if __name__ == '__main__':
    app.run(debug=False, threaded=False)

However, the problem still persists.


title: Pytorch Indexing

body:
I have a tensor [[1,2],[4,5],[7,8]] and a tensor with indices [0,1,0].

I want to apply them to second dimension so that it returns: [1,5,8].

How should I do that?

Thanks!


title: not able to predict using pytorch [MNIST]

body:
pytorch noob here,trying to learn.

link to my notebook:
https://gist.github.com/jagadeesh-kotra/412f371632278a4d9f6cb31a33dfcfeb

I get validation accuracy of 95%.

i use the following to predict:

m.eval()

testset_predictions = []
for batch_id,image in enumerate(test_dataloader):
    image = torch.autograd.Variable(image[0])
    output = m(image)
    _, predictated = torch.max(output.data,1)
    for prediction in predicted:
        testset_predictions.append(prediction.item())

len(testset_predictions)

The problem is i get only 10% accuracy when i submit the result to kaggle competition,which is as good as random prediction.I cant figure out what i'm doing wrong.

please help :)


title: Inplace add with list selectors

body:
I'm facing a somehow inconsistent behaviour of pythorch according to weither the index is a list or an integer. Take a look at this snippet of code :

# First example, integer selector ==> Ok
t = torch.tensor([[0, 1], [1, 0]])
t[0, 0].add_(10)
print(t)
tensor([[10,  1],
        [ 1,  0]])

# Second example, list selector ==> ???
t = torch.tensor([[0, 1], [1, 0]])
t[[0], [0]].add_(10) # notice the list selector
print(t)
tensor([[0, 1],
        [1, 0]])

#Third example, list selector with inplace add operator ==> Ok
t = torch.tensor([[0, 1], [1, 0]])
t[[0], [0]] += 10
print(t)
tensor([[10,  1],
        [ 1,  0]])

I can't understand why pytorch was unable to update tin the second example !


title: Shared weights with model parallelism in PyTorch

body:
Our setup involves initial part of the network (input interface) which run on separate GPU cards. Each GPU gets its own portion of data (model parallelism) and process it separately.

Each input interface, in turn, it itself a complex nn.Module. Every input interface can occupy one or several cards (say, interface_1 runs on GPU 0 and 1, interface_2 - on GPU 2 and 3 and so on).

We need to keep the weights of these input interface the same all over the training. We also need them to run in parallel to save training time which is already weeks for our scenario.

The best idea we can think of was initializing the interfaces with the same weights and then average the gradients for them. As the interfaces are identical, updating same weights with the same gradients should keep them the same all over the training process thus achieving desired “shared weights” mode.

However, I cannot find any good solution for changing values of these weights and their gradients represented as Parameter in PyTorch. Apparently, PyTorch does not allow to do so.

Our current state is: if we copy.deepcopy the ‘parameter.data’ of the “master” interface and assign it to ‘parameter.data’ of the "slave" interface, the values are indeed changed but .to(device_id) does not work and keeps them at the “master” device. However, we need them to move to a “slave” device.

Could someone please tell me if it is possible at all or, if not, if there is a better way to implement shared weights along with the parallel execution for our scenario?


title: Using MIT Indoor scene database in CNN

body:
I'm an engineering student and kind of a noob to programming. I'm taking an AI course, currently trying to do my final project.

I have to create a CNN net, I have to use de MIT Indoor scene database (it can be found here: http://web.mit.edu/torralba/www/indoor.html). I don't have a problem doing the CNN since I've done a few before in the semester using CIFAR10, but I'm having trouble with this one since I don't know who to use this set of images.
I think I need to create a dataset of my own, I've tried with PyTorch using https://pytorch.org/tutorials/beginner/data_loading_tutorial.html, but I get confused because I don't have a .csv with features, I have a lot of .xml files with several features for each picture. Also, I don't have a file just saying "bedroom, bar, etc" as I've seen in other tutorials.

I would rather use PyTorch since I can use the "train_test_split" function, but if anyone could help me to understand how to make those 15620 my input to the net, I would really appreciate it.


title: pytorch load checkpoints to continue training - zero_grad() or not?

body:
I have saved and loaded the checkpoints as per the pytorch manual, and it all seems OK. Now, usually, when I want to start training, I have something like this in pytorch:

    for itr in range(1, args.niters + 1):
        optimizer.zero_grad() # should I or should I not when checkpoints are loaded?

I am unsure if I should do zero_grad() here (which I use when I start training from scratch), since I am reloading all my weights and bias.

Apologies if this is a daft question.


I am trying to use facebook's AX platform to find the optimal values of hyper parameters for my machine learning model.For that I have tried installing the ax platform using:

pip3 install ax-platform

Once I have installed , I have tried to import the following into my spyder notebook

from ax.plot.contour import plot_contour
from ax.plot.trace import optimization_trace_single_method
from ax.service.managed_loop import optimize
from ax.utils.notebook.plotting import render, init_notebook_plotting
from ax.utils.tutorials.cnn_utils import load_mnist, train, evaluate, CNN

I am getting an error in the last line:from ax.utils.tutorials.cnn_utils import load_mnist, train, evaluate, CNN

  from torchvision import _C  ImportError: DLL load failed: The specified module could not be found.

My python version is Python 3.7.4 and my anaconda version is conda 4.7.10
I am new to pytorch and ax platform.I don't have much clue why this is happening.Can someone please help me?


title: Python modify instance attribute with instance method

body:
I am trying to use an instance method to modify one of the instance's attribute like this:

from torch.optim import SGD
from typing import Dict

class ChangeRateSgd(SGD):
    def __init__(self, params, lr: float, lr_change_instructions: Dict):
        super().__init__(params, lr)
        self.lr_change_instructions = lr_change_instructions

    def change_update_rate(self, input_epoch):
        update_mapping = self.lr_change_instructions
        if input_epoch in update_mapping.keys():
            new_lr = self.lr_change_instructions[input_epoch]
            self.lr = new_lr

However, my IDE flags the line self.lr = new_lr as not being ideal coding practice, with the warning Instance attribute lr defined outside __init__. What is the best way to do what I am trying to do with this instance method?


title: Pytorch Custom Module Using Existing CNN Module

body:
I want to access and edit individual modules within a torchvision module and adjust the input.

I know you can edit submodules like this:

import torchvision
resnet18 = torchvision.models.resnet18()
print(resnet18._modules['conv1'])
# Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)

But I want to create a custom Net(nn.Module) kind of class so I can add additional layers later:

class Sonar(resnet18):
    pass

Throws the error:

----> 1 class Sonar(resnet18):
      2     pass

/usr/local/lib/python3.7/dist-packages/torchvision/models/resnet.py in __init__(self, block, layers, num_classes, zero_init_residual, groups, width_per_group, replace_stride_with_dilation, norm_layer)
    142         self.relu = nn.ReLU(inplace=True)
    143         self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
--> 144         self.layer1 = self._make_layer(block, 64, layers[0])
    145         self.layer2 = self._make_layer(block, 128, layers[1], stride=2,
    146                                        dilate=replace_stride_with_dilation[0])

/usr/local/lib/python3.7/dist-packages/torchvision/models/resnet.py in _make_layer(self, block, planes, blocks, stride, dilate)
    176             self.dilation *= stride
    177             stride = 1
--> 178         if stride != 1 or self.inplanes != planes * block.expansion:
    179             downsample = nn.Sequential(
    180                 conv1x1(self.inplanes, planes * block.expansion, stride),

AttributeError: 'str' object has no attribute 'expansion'

Trying again with AlexNet

alexnet = torchvision.models.AlexNet()
class Sonar(alexnet):
    pass

Throws the error:

      1 alexnet = torchvision.models.AlexNet()
----> 2 class Sonar(alexnet):
      3     pass

TypeError: __init__() takes from 1 to 2 positional arguments but 4 were given


title: How is the number of channels adjusted in efficientnet

body:
I was reading the code at efficientnet and was shocked by its clever ideas. But I don't quite understand how it adjusts the number of channels.

def round_filters(filters, width_coefficient, depth_divisor):

  filters *= width_coefficient
  new_filters = int(filters + depth_divisor / 2) // depth_divisor * depth_divisor
  new_filters = max(depth_divisor, new_filters)
  # Make sure that round down does not go down by more than 10%.
  if new_filters < 0.9 * filters:
     new_filters += depth_divisor
  return int(new_filters)

I know the number of channels has to be adjusted by the width factor, but why do I do the following? What is depth_divisor?


title: How to perform advanced indexing in PyTorch?

body:
Is there a way of doing the following without looping?

S, N, H = 9, 7, 4

a = torch.randn(S, N, H)

# tensor with integer values between 1, S of shape (N,)
lens = torch.randint(1, S + 1, (N,))

res = torch.zeros(N, H)

for i in range(N):
    res[i] = a[lens[i] - 1, i, :]


title: Pytorch, INPUT (normal tensor) and WEIGHT (cuda tensor) mismatch

body:
DISCLAIMER I know, this question has already asked multiple times, but i tried their solutions, none of them worked for me, so after all those effort, i can't find anything else and eventually i have to ask again.
I'm doing image classification with cnns (PYTORCH), i wan't to train it on GPU (nvidia gpu, compatible with cuda/cuda installed), i successfully managed to put net on it, but the problem is with data.
if torch.cuda.is_available():
    device = torch.device("cuda:0")
    print("Running on the GPU")
    print("Available GPU", torch.cuda.device_count())

Running on the GPU
Available GPU 1

net = Net()
net.to(device)

for epoch in range(2):
    running_loss=0.0
    for i, data in enumerate(trainloader, 0):
        inputs, labels = data[0].to(device), data[1].to(device) # putting data on gpu
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        if i % 2000 == 1999:
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

dataiter = iter(testloader)
images, labels = dataiter.next()

print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))

out = net(images)

ERROR
----------------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-81-76c52eabb174> in <module>
----> 1 out = net(images)

~/anaconda3/envs/home/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

<ipython-input-57-c74f8361a10b> in forward(self, x)
     11
     12     def forward(self, x):
---> 13         x = self.pool(F.relu(self.conv1(x)))
     14         x = self.pool(F.relu(self.conv2(x)))
     15         x = x.view(-1, 16*5*5)

~/anaconda3/envs/home/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

~/anaconda3/envs/home/lib/python3.8/site-packages/torch/nn/modules/conv.py in forward(self, input)
    351
    352     def forward(self, input):
--> 353         return self._conv_forward(input, self.weight)
    354
    355 class Conv3d(_ConvNd):

~/anaconda3/envs/home/lib/python3.8/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    347                             weight, self.bias, self.stride,
    348                             _pair(0), self.dilation, self.groups)
--> 349         return F.conv2d(input, weight, self.bias, self.stride,
    350                         self.padding, self.dilation, self.groups)
    351

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

inputs.is_cuda

True

and same for labels.
What I've tried:
https://stackoverflow.com/q/59013109/13287412
https://github.com/sksq96/pytorch-summary/issues/57
https://blog.csdn.net/qq_27261889/article/details/86575033
https://blog.csdn.net/public669/article/details/96510293

but nothing worked so far...


title: Pytorch, Gradients of neural network for 3 Dim input - 1 Dim output regression

body:
I want to get the gradients of model(neural network).
In tensorflow, there is a 'tf.gradient'
so the following way is possible.
x = tf.placeholder(shape=[None,1],dtype=tf.float32)
y = tf.placeholder(shape=[None,1],dtype=tf.float32)
inp = tf.concat([x,y],axis=-1)

def network(x):
    x = tf.layers.dense(x,128,activation=tf.nn.relu)
    x = tf.layers.dense(x,128,activation=tf.nn.relu)
    x = tf.layers.dense(x,128,activation=tf.nn.relu)
    x = tf.layers.dense(x,128,activation=tf.nn.relu)
    x = tf.layers.dense(x,128,activation=tf.nn.relu)
    x = tf.layers.dense(x,1)
    return x

pred = network(inp)
dfdx,dfdy = tf.gradients(pred,[x,y])

But how to I get similar results with above code using pytorch?
class net(torch.nn.Module):
  def __init__(self):
    super(net,self).__init__()
    self.linear={}
    self.wei={}
    self.bias={}
    self.para=[]
    self.linear[0] = torch.nn.Linear(3,128)
    self.linear[1] = torch.nn.Linear(128,128)
    self.linear[2] = torch.nn.Linear(128,128)
    self.linear[3] = torch.nn.Linear(128,128)
    self.linear[4] = torch.nn.Linear(128,1)
  def forward(self,x):
    for i in range(4):
      x=F.elu(self.linear[i](x))
    x=self.linear[4](x)
    return x

M=net()

for i in range(5):
  M.para+=list(M.linear[i].parameters())

L=nn.MSELoss()
opt=torch.optim.Adam(M.para,0.00035)

######################################

X=torch.Tensor(data[:,0:3])   ### 3 dim input
X.requires_grad_(True)
Y=torch.Tensor(data[:,3])
Y=Y.reshape(-1,1)

for epoch in range(10000):
    opt.zero_grad()
    idx=torch.randint(0,X.shape[0],(24,))
    M_dev=torch.autograd.grad(M.forward(X),X,torch.ones_like(M.forward(X)))   ????

I tried 'torch.autograd.grad(M.forward(X),X,torch.ones_like(M.forward(X)))'
but it gives wrong answer..
How I can get like 'tf.gradient( M.forward(X), X)'  ?????


title: pytorch does not save pre-trained model weights loaded and the parts of it in the final model

body:
I am currently working on pre-trained model on CIFAR-10 on my data, have removed the final fc layer of the model and have appended my own fc layer and  softmax. There are seven networks which each of them are same as pre-trained part and are combined using appended fc layer. The following is pre-trained Network code :
class Bottleneck(nn.Module):
    def __init__(self, inplanes, expansion=4, growthRate=12, dropRate=0):
        super(Bottleneck, self).__init__()
        planes = expansion * growthRate
        self.bn1 = nn.BatchNorm2d(inplanes)
        self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False)
        self.bn2 = nn.BatchNorm2d(planes)
        self.conv2 = nn.Conv2d(planes, growthRate, kernel_size=3,
                               padding=1, bias=False)
        self.relu = nn.ReLU(inplace=True)
        self.dropRate = dropRate

    def forward(self, x):
        out = self.bn1(x)
        out = self.relu(out)
        out = self.conv1(out)
        out = self.bn2(out)
        out = self.relu(out)
        out = self.conv2(out)
        if self.dropRate > 0:
            out = F.dropout(out, p=self.dropRate, training=self.training)

        out = torch.cat((x, out), 1)

        return out

class BasicBlock(nn.Module):
    def __init__(self, inplanes, expansion=1, growthRate=12, dropRate=0):
        super(BasicBlock, self).__init__()
        planes = expansion * growthRate
        self.bn1 = nn.BatchNorm2d(inplanes)
        self.conv1 = nn.Conv2d(inplanes, growthRate, kernel_size=3,
                               padding=1, bias=False)
        self.relu = nn.ReLU(inplace=True)
        self.dropRate = dropRate

    def forward(self, x):
        out = self.bn1(x)
        out = self.relu(out)
        out = self.conv1(out)
        if self.dropRate > 0:
            out = F.dropout(out, p=self.dropRate, training=self.training)

        out = torch.cat((x, out), 1)

        return out

class Transition(nn.Module):
    def __init__(self, inplanes, outplanes):
        super(Transition, self).__init__()
        self.bn1 = nn.BatchNorm2d(inplanes)
        self.conv1 = nn.Conv2d(inplanes, outplanes, kernel_size=1,
                               bias=False)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        out = self.bn1(x)
        out = self.relu(out)
        out = self.conv1(out)
        out = F.avg_pool2d(out, 2)
        return out

class DenseNet(nn.Module):

    def __init__(self, depth = 22, block = Bottleneck,
        dropRate = 0, num_classes = 10, growthRate = 12, compressionRate = 2):
        super(DenseNet, self).__init__()

        assert (depth - 4) % 3 == 0, 'depth should be 3n+4'
        n = (depth - 4) / 3 if block == BasicBlock else (depth - 4) // 6

        self.growthRate = growthRate
        self.dropRate = dropRate

        # self.inplanes is a global variable used across multiple
        # helper functions
        self.inplanes = growthRate * 2
        self.conv1 = nn.Conv2d(3, self.inplanes, kernel_size = 3, padding = 1,
                               bias = False)
        self.dense1 = self._make_denseblock(block, n)
        self.trans1 = self._make_transition(compressionRate)
        self.dense2 = self._make_denseblock(block, n)
        self.trans2 = self._make_transition(compressionRate)
        self.dense3 = self._make_denseblock(block, n)
        self.bn = nn.BatchNorm2d(self.inplanes)
        self.relu = nn.ReLU(inplace=True)
        self.avgpool = nn.AvgPool2d(8)
        #self.fc = nn.Linear(self.inplanes, num_classes)

        # Weight initialization
#         for m in self.modules():
#             if isinstance(m, nn.Conv2d):
#                 n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
#                 m.weight.data.normal_(0, math.sqrt(2. / n))
#             elif isinstance(m, nn.BatchNorm2d):
#                 m.weight.data.fill_(1)
#                 m.bias.data.zero_()

    def _make_denseblock(self, block, blocks):
        layers = []
        for i in range(blocks):
            # Currently we fix the expansion ratio as the default value
            layers.append(block(self.inplanes, growthRate = self.growthRate, dropRate=self.dropRate))
            self.inplanes += self.growthRate

        return nn.Sequential(*layers)

    def _make_transition(self, compressionRate):
        inplanes = self.inplanes
        outplanes = int(math.floor(self.inplanes // compressionRate))
        self.inplanes = outplanes
        return Transition(inplanes, outplanes)

    def forward(self, x):
        x = self.conv1(x)

        x = self.trans1(self.dense1(x))
        x = self.trans2(self.dense2(x))
        x = self.dense3(x)
        x = self.bn(x)
        x = self.relu(x)

        x = self.avgpool(x)
        #x = x.view(x.size(0), -1)
        #x = self.fc(x)

        return x

    def getParams(self, paramName):
        if paramName == 'inplanes':
            return self.inplanes
        elif paramName == 'growthRate':
            return self.growthRate
        elif paramName == 'dropRate':
            return self.dropRate

def densenet(**kwargs):
    """
    Constructs a DenseNet model.
    """
    return DenseNet(**kwargs)

and the next following is my code:
class Network(nn.Module):

    def __init__(self,pretrained_dict, num_classes = 6, num_channels = 7,
                 expansion = 4, depth = 100, growthRate = 12, dropRate = 0):

        super(Network, self).__init__()

        self.num_channels = num_channels

        # creating 7 channels networks
        self.channels_dnsnets = []

        for ch in range(self.num_channels):
#             print(ch)

            d = densenet(depth = depth)
            d_dict = d.state_dict()

            # 1. filter out unnecessary keys
            pretrained_dict2 = {k[7:]: v for k, v in pretrained_dict.items() if k[7:] in d_dict}
#             print('d_dict_keys :')
#             print(d_dict.keys())
#             print('*'*50)
#             print('pretrained_dict2.keys:')
#             print(pretrained_dict2.keys())
#             print('*'*50)

            # 2. overwrite entries in the existing state dict
            d_dict.update(pretrained_dict2)

            # 3. load the new state dict
            d.load_state_dict(pretrained_dict2)

            # freeze the layers of densenet
            for param in d.parameters():
                param.requires_grad = False

            self.channels_dnsnets.append(d)

        self.inplanes = self.channels_dnsnets[0].getParams(paramName = 'inplanes')
        self.fc = nn.Linear(self.inplanes * self.num_channels, num_classes)
        self.softmax = nn.Softmax(dim = 1)

    def forward(self, x):

        batch_size, channels, ht, wd, in_channels = x.shape
        x = np.reshape(x,(batch_size,channels,in_channels,ht,wd))

        out = []

        for num in range(self.num_channels):
            temp_out = self.channels_dnsnets[0](x[:,num,:])
            temp_out = temp_out.view(temp_out.size(0),-1)
#             print(temp_out.shape)
#             print('*' * 50)
            out.append(temp_out)

        out = torch.stack(out,dim = 1)
#         print(out.shape)
        out = out.view(out.size(0),-1)
        out = self.fc(out)
        out = self.softmax(out)
        return out

I am setting the optimizer as :
optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr = lr,
                                 betas = (0.9, 0.999), eps = 1e-08, weight_decay = wd, amsgrad = False)

However, whenever I save the model the list of densenets and their weights are not saved and only fc layer and softmax layer weights are saved. Is there anything problematic with code? I am new to pytorch.


title: object detection: is object in the photo, python

body:
I am trying to detect plants in the photos, i've already labeled photos with plants (with labelImg), but i don't understand how to train model with only background photos, so that when there is no plant here model can tell me so.
Do I need to set labeled box as the size of image?
p.s. new to ml so don't be rude, please)


I'm working on a pytorch (torch=1.2.0) project considering CNN's pruning. I dropped few filters from a model and tried to retrain the reduced model. However my code crashes while processing backward pass.
My final layer was [294, 768, 1, 1], but was reduced to [294, 592, 1, 1]. This means in the layer before the final one the amount of filters was reduced from 768 to 592. The forward pass runs well, meaning all the model layers are compatible, but it crashed immediately on the last layers backprop with the following message:
Function CudnnConvolutionBackward returned an invalid gradient at index 1 - got [294, 592, 1, 1] but expected shape compatible with [294, 768, 1, 1]
I don't understand it, as the model was pruned, but it looks like it tries to "backprop" on the original layer dimensions.


title: How to get the TopK Accuracy for the classes predictions using tf.keras.metric

body:
I have my y_true labels as [0,1,2,1,0,2] etc and I have my y_pred_probs (probabilities of each classes) as [[45,12,30,80,12,23,45,23],[12,15,47,12,3,55,900,4]] These are logits. I can get softmax() but this should not matter. I get my y_pred_classes = np.argmax(y_pred_probs,1). I can use tf.keras.metrics.TopKCategoricalAccuracy inside model.compile() but when I have my test predictions as given above, how am I supposed to use this one?
I have used to_categorical(test_labels) with tf.keras.metrics.TopKCategoricalAccuracy NS laao used the 1-D array predictions inside tf.keras.metrics.SparseTopKCategoricalAccuracy. everytime there is an error. Sometimes it says that there should be 1-d sometimes there should be 2-d. My y_preds and y_true have same dimension as (n,)
How can I get the Top-K Accuracies on my test data?
I can also any pytorch,numpy or any other function to get this. Please help.
sklearn.metrics.top_k_accuracy_score¶ is still in unstable version.


Here is the data:
https://github.com/GuokaiLiu/MyIssues/blob/main/dataMap.mat
Here is the code:
import torch
import numpy as np
import scipy.io as sio

def pinv1(A,reg):
    pAdd = reg*np.eye(A.shape[1])+A.T.dot(A)
    pInv = np.mat(pAdd).I
    pDot = pInv.dot(A.T)
    return pAdd, pInv, pDot

def pinv2(A,reg):
    pAdd = reg*torch.eye(A.shape[1])+torch.matmul(A.T,A)
    pInv = torch.inverse(pAdd).float()
    pDot = torch.matmul(pInv,A.T.float())
    return pAdd, pInv, pDot

npData = sio.loadmat('dataMap.mat')['dataMap']
ptData = torch.Tensor(dataMap)

a1,a2,a3 = pinv1(npData,2**-30)
b1,b2,b3 = pinv2(ptData,torch.Tensor([2**-30]))

print(torch.dist(torch.Tensor(np.array(a1)),b1,2))
print(torch.dist(torch.Tensor(np.array(a2)),b2,2))
print(torch.dist(torch.Tensor(np.array(a3)),b3,2))

Here is the results:
tensor(0.0022)
tensor(9947374.)
tensor(99862.0469)

Why different inverse results between PyTorch and NumPy


title: Backtransforming a PyTorch Tensor

body:
I have trained a WGAN on the CelebA dataset in PyTorch following this youtube video. Since I do this on Google Cloud Platform where TensorBoard is not availabe, I save one figure of generated images by the GAN every epoch to see how the GAN is actually doing.
Now, the saved pdf files look sth like this: generated images. Unfortunately, this is not really readable, and I suspect this has to do with the preprocessing I do:
trafo = transforms.Compose(
                    [transforms.Resize(size = (64, 64)),
                    transforms.ToTensor(),
                    transforms.Normalize( mean = (0.5,), std = (0.5,))])

Is there any way to kind of undo this transformation when I save the image?
Currently, I save the image every epoch as follows:
visualization = torchvision.utils.make_grid(
                           tensor = gen(fixed_noise),
                           nrow = 8,
                           normalize = False)
plt.savefig("generated_WGAN_" + datetime.now().strftime("%Y%m%d-%H%M%S") + ".pdf")

Also, I should probably mention that in the Jupyter notebook, I get the following warning:
"Clipping input data to the valid range for imshow with RGB data ([0..1]) for floats or [0..255] for integers)."