title: Deal with negative values resulting from my nn model

body:
I have a simple nn model that looks like this

class TestRNN(nn.Module):
    def __init__(self, batch_size, n_steps, n_inputs, n_neurons, n_outputs):
        super(TestRNN, self).__init__()
        ...
        self.basic_rnn = nn.RNN(self.n_inputs, self.n_neurons)
        self.FC = nn.Linear(self.n_neurons, self.n_outputs)

    def forward(self, X):
        ...
        lstm_out, self.hidden = self.basic_rnn(X, self.hidden)
        out = self.FC(self.hidden)

        return out.view(-1, self.n_outputs)

and I am using criterion = nn.CrossEntropyLoss() for calculating my error.  The operation order goes something like this:

# get the inputs
x, y = data

# forward + backward + optimize
outputs = model(x)
loss = criterion(outputs, y)

Where my training data x is normalized and looks like this:

tensor([[[7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [2.6164e-02, 2.6164e-02, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 1.3108e-05],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [9.5062e-01, 3.1036e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [0.0000e+00, 1.3717e-05, 3.2659e-07,  ..., 0.0000e+00,
          0.0000e+00, 3.2659e-07]],

        [[5.1934e-01, 5.4041e-01, 6.8083e-06,  ..., 0.0000e+00,
          0.0000e+00, 6.8083e-06],
         [5.2340e-01, 6.0007e-01, 2.7062e-06,  ..., 0.0000e+00,
          0.0000e+00, 2.7062e-06],
         [8.1923e-01, 5.7346e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00]],

        [[7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0714e-01, 7.0708e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 7.0407e-06],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00]],

        ...,

        [[7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.1852e-01, 2.3411e-02, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0775e-01, 7.0646e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 3.9888e-06],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00]],

        [[5.9611e-01, 5.8796e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0710e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.7538e-01, 2.4842e-01, 1.7787e-06,  ..., 0.0000e+00,
          0.0000e+00, 1.7787e-06],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00]],

        [[5.2433e-01, 5.2433e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [1.3155e-01, 1.3155e-01, 0.0000e+00,  ..., 8.6691e-02,
          9.7871e-01, 0.0000e+00],
         [7.4412e-01, 6.6311e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 9.6093e-07]]])

While a typical output and y passed to the criterion function look like this:

tensor([[-0.0513],
        [-0.0445],
        [-0.0514],
        [-0.0579],
        [-0.0539],
        [-0.0323],
        [-0.0521],
        [-0.0294],
        [-0.0372],
        [-0.0518],
        [-0.0516],
        [-0.0501],
        [-0.0312],
        [-0.0496],
        [-0.0436],
        [-0.0514],
        [-0.0518],
        [-0.0465],
        [-0.0530],
        [-0.0471],
        [-0.0344],
        [-0.0502],
        [-0.0536],
        [-0.0594],
        [-0.0356],
        [-0.0371],
        [-0.0513],
        [-0.0528],
        [-0.0621],
        [-0.0404],
        [-0.0403],
        [-0.0562],
        [-0.0510],
        [-0.0580],
        [-0.0516],
        [-0.0556],
        [-0.0063],
        [-0.0459],
        [-0.0494],
        [-0.0460],
        [-0.0631],
        [-0.0525],
        [-0.0454],
        [-0.0509],
        [-0.0522],
        [-0.0426],
        [-0.0527],
        [-0.0423],
        [-0.0572],
        [-0.0308],
        [-0.0452],
        [-0.0555],
        [-0.0479],
        [-0.0513],
        [-0.0514],
        [-0.0498],
        [-0.0514],
        [-0.0471],
        [-0.0505],
        [-0.0467],
        [-0.0485],
        [-0.0520],
        [-0.0517],
        [-0.0442]], device='cuda:0', grad_fn=<ViewBackward>)
tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0')

When the criterion is being applied I get the following error (running with CUDA_LAUNCH_BLOCKING=1):

/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [7,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [20,0,0] Assertion `t >= 0 && t < n_classes` failed.
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=111 error=59 : device-side assert triggered

The fact that my model outputs negative values is causing the error message above, how can I resolve this issue?


title: "function 'AddDllDirectory' not found" while importing pytorch

body:
I keep getting this error while trying to import pytorch in jupyter notebook

import torch

  AttributeError: function 'AddDllDirectory' not found

this error occur only when I try installing pytorch in anaconda (I used both command to install pytorch pip and conda, I also tried installing only the cpu version and nothing changed, i still get the same error while trying to import it)

I searched the internet and found nothing remotely close to my problem.
is there's a way I can fix this error?

thanks in advance


title: What does 'requires grad' do in PyTorch and should I use it?

body:
I have a network where I need to add my own parameter that I want to be trainable. I am using nn.Parameter() to add this but there's a 'Requires Grad' argument and I can't really make sense as to whether I want this to be true or false from reading the documentation. It makes sense that I would set this to be true, because I want this parameter to be optimised as part of the learning process -- but the need for this argument confuses me: if False means it is not optimised as part of the training process, then why would you use nn.Parameter() rather than just using a normal Tensor?
From the documentation I see that it adds the parameter to the list of utterable parameters that you obtain from the model, but I don't see why you would want this if you're not optimising it.


title: Loss is increasing from first epoch itself

body:
I am training my siamese network for nlp. I have used lstm in it. and BCELoss. My loss is increasing from the first epoch. The first 36 epoch loss is
error after  0 is
 272.4357
[torch.FloatTensor of size 1]

error after  1 is
 271.8972
[torch.FloatTensor of size 1]

error after  2 is
 271.5598
[torch.FloatTensor of size 1]

error after  3 is
 271.6979
[torch.FloatTensor of size 1]

error after  4 is
 271.7315
[torch.FloatTensor of size 1]

error after  5 is
 272.3965
[torch.FloatTensor of size 1]

error after  6 is
 273.3982
[torch.FloatTensor of size 1]

error after  7 is
 275.1197
[torch.FloatTensor of size 1]

error after  8 is
 275.8228
[torch.FloatTensor of size 1]

error after  9 is
 278.3311
[torch.FloatTensor of size 1]

error after  10 is
 277.1054
[torch.FloatTensor of size 1]

error after  11 is
 277.8418
[torch.FloatTensor of size 1]

error after  12 is
 279.0189
[torch.FloatTensor of size 1]

error after  13 is
 278.4090
[torch.FloatTensor of size 1]

error after  14 is
 281.8813
[torch.FloatTensor of size 1]

error after  15 is
 283.4077
[torch.FloatTensor of size 1]

error after  16 is
 286.3093
[torch.FloatTensor of size 1]

error after  17 is
 287.6292
[torch.FloatTensor of size 1]

error after  18 is
 297.2318
[torch.FloatTensor of size 1]

error after  19 is
 307.4176
[torch.FloatTensor of size 1]

error after  20 is
 304.6649
[torch.FloatTensor of size 1]

error after  21 is
 328.9772
[torch.FloatTensor of size 1]

error after  22 is
 300.0669
[torch.FloatTensor of size 1]

error after  23 is
 292.3902
[torch.FloatTensor of size 1]

error after  24 is
 300.8633
[torch.FloatTensor of size 1]

error after  25 is
 305.1822
[torch.FloatTensor of size 1]

error after  26 is
 333.9984
[torch.FloatTensor of size 1]

error after  27 is
 346.2062
[torch.FloatTensor of size 1]

error after  28 is
 354.6148
[torch.FloatTensor of size 1]

error after  29 is
 341.3568
[torch.FloatTensor of size 1]

error after  30 is
 369.7580
[torch.FloatTensor of size 1]

error after  31 is
 366.1615
[torch.FloatTensor of size 1]

error after  32 is
 368.2455
[torch.FloatTensor of size 1]

error after  33 is
 391.4102
[torch.FloatTensor of size 1]

error after  34 is
 394.3190
[torch.FloatTensor of size 1]

error after  35 is
 401.0990
[torch.FloatTensor of size 1]

error after  36 is
 422.3723
[torch.FloatTensor of size 1]


title: Cloud Storage Buckets for PyTorch

body:
For a particular task I'm working on I have a dataset that is about 25 GB. I'm still experimenting with several methods of preprocessing and definitely don't have my data to it's final form yet. I'm not sure what the common workflow is for this sort of problem, so here is what I'm thinking:

Copy dataset from bucket storage to Compute Engine machine SSD (maybe use around 50 GB SSD) using gcsfuse.
Apply various preprocessing operations as an experiment.
Run training with PyTorch on the data stored on the local disk (SSD)
Copy newly processed data back to storage bucket with gcsfuse if it was successful.
Upload results and delete the persistent disk that was used during training.

The alternative approach is this:

Run the processing operations on the data within the Cloud Bucket itself using the mounted directory with gcsfuse
Run training with PyTorch directly on the mounted gcsfuse Bucket directory, using a compute engine instance with very limited storage.
Upload Results and Delete Compute Engine Instance.

Which of these approaches is suggested? Which will incur fewer charges and is used most often when running these kind of operations. Is there a different workflow that I'm not seeing here?


title: Why the resnet110 I train on CIFAR10 dataset only get 77% test acc

body:
I trained the Resnet110 on CIFAR10 dataset, and I got 100% acc on training, but only 77.85% on test dataset. What is the problem probally be?
Otherwise, I use Pytorch framwork.
Thank U very much!

    ------------------------------------------------------------------------
Train Epoch: 200 [0/50000 (0%)] Loss: 0.000811, Accuracy: 100.00
Train Epoch: 200 [12800/50000 (26%)]    Loss: 0.000335, Accuracy: 100.00
Train Epoch: 200 [25600/50000 (51%)]    Loss: 0.000757, Accuracy: 100.00
Train Epoch: 200 [38400/50000 (77%)]    Loss: 0.000334, Accuracy: 100.00
Epoch time:45.98s
Accuracy of plane : 81 %
Accuracy of   car : 88 %
Accuracy of  bird : 65 %
Accuracy of   cat : 60 %
Accuracy of  deer : 73 %
Accuracy of   dog : 69 %
Accuracy of  frog : 81 %
Accuracy of horse : 82 %
Accuracy of  ship : 86 %
Accuracy of truck : 86 %

Test set: Average loss: 1.3605, Accuracy: 77.58%


title: I am having trouble calculating the accuracy, recall, precision and f1-score for my model

body:
I have got my confusion matrix working correctly, just having some trouble producing the scores. A little help would go a long way. I am currently getting the error. "Tensor object is not callable".

def get_confused(model_ft):
    nb_classes = 120
    from sklearn.metrics import precision_recall_fscore_support as score
    confusion_matrix = torch.zeros(nb_classes, nb_classes)
    with torch.no_grad():
        for i, (inputs, classes) in enumerate(dataloaders['val']):
            inputs = inputs.to(device)
            classes = classes.to(device)
            outputs = model_ft(inputs)
            _, preds = torch.max(outputs, 1)
            for t, p in zip(classes.view(-1), preds.view(-1)):
                    confusion_matrix[t.long(), p.long()] += 1

            cm = confusion_matrix(classes, preds)
            recall = np.diag(cm) / np.sum(cm, axis = 1)
            precision = np.diag(cm) / np.sum(cm, axis = 0)
    print(confusion_matrix)
    print(confusion_matrix.diag()/confusion_matrix.sum(1))


title: Changing a Dockerfile from CUDA9 to CUDA10

body:
I need to change the Dockerfile below, coming from pytorch_geometric (a popular PyTorch package), from CUDA9.0 to CUDA10.0.

FROM ubuntu:16.04

RUN apt-get update && apt-get install -y --no-install-recommends apt-utils ca-certificates apt-transport-https gnupg-curl && \
    rm -rf /var/lib/apt/lists/* && \
    NVIDIA_GPGKEY_SUM=d1be581509378368edeec8c1eb2958702feedf3bc3d17011adbf24efacce4ab5 && \
    NVIDIA_GPGKEY_FPR=ae09fe4bbd223a84b2ccfce3f60f4b3d7fa2af80 && \
    apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub && \
    apt-key adv --export --no-emit-version -a $NVIDIA_GPGKEY_FPR | tail -n +5 > cudasign.pub && \
    echo "$NVIDIA_GPGKEY_SUM  cudasign.pub" | sha256sum -c --strict - && rm cudasign.pub && \
    echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 /" > /etc/apt/sources.list.d/cuda.list && \
    echo "deb https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64 /" > /etc/apt/sources.list.d/nvidia-ml.list

ENV CUDA_VERSION 9.0.176
ENV NCCL_VERSION 2.4.2
ENV CUDA_PKG_VERSION 9-0=$CUDA_VERSION-1
ENV CUDNN_VERSION 7.4.2.24

RUN apt-get update && apt-get install -y --no-install-recommends \
        cuda-cudart-$CUDA_PKG_VERSION && \
    ln -s cuda-9.0 /usr/local/cuda && \
    rm -rf /var/lib/apt/lists/*

RUN apt-get update && apt-get install -y --allow-unauthenticated --no-install-recommends \
        cuda-libraries-$CUDA_PKG_VERSION \
        libnccl2=$NCCL_VERSION-1+cuda9.0 && \
    apt-mark hold libnccl2 && \
    rm -rf /var/lib/apt/lists/*

RUN apt-get update && apt-get install -y --allow-unauthenticated --no-install-recommends \
        cuda-libraries-dev-$CUDA_PKG_VERSION \
        cuda-nvml-dev-$CUDA_PKG_VERSION \
        cuda-minimal-build-$CUDA_PKG_VERSION \
        cuda-command-line-tools-$CUDA_PKG_VERSION \
        cuda-core-9-0=9.0.176.3-1 \
        cuda-cublas-dev-9-0=9.0.176.4-1 \
        libnccl-dev=$NCCL_VERSION-1+cuda9.0 && \
    rm -rf /var/lib/apt/lists/*

ENV LIBRARY_PATH /usr/local/cuda/lib64/stubs

# NVIDIA docker 1.0.
LABEL com.nvidia.volumes.needed="nvidia_driver"
LABEL com.nvidia.cuda.version="${CUDA_VERSION}"

RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && \
    echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf

ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64

# NVIDIA container runtime.
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
ENV NVIDIA_REQUIRE_CUDA "cuda>=9.0"

# PyTorch (Geometric) installation
RUN rm /etc/apt/sources.list.d/cuda.list && \
    rm /etc/apt/sources.list.d/nvidia-ml.list

RUN apt-get update &&  apt-get install -y \
    curl \
    ca-certificates \
    vim \
    sudo \
    git \
    bzip2 \
    libx11-6 \
 && rm -rf /var/lib/apt/lists/*

# Create a working directory.
RUN mkdir /app
WORKDIR /app

# Create a non-root user and switch to it.
RUN adduser --disabled-password --gecos '' --shell /bin/bash user \
 && chown -R user:user /app
RUN echo "user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/90-user
USER user

# All users can use /home/user as their home directory.
ENV HOME=/home/user
RUN chmod 777 /home/user

# Install Miniconda.
RUN curl -so ~/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-4.5.12-Linux-x86_64.sh \
 && chmod +x ~/miniconda.sh \
 && ~/miniconda.sh -b -p ~/miniconda \
 && rm ~/miniconda.sh
ENV PATH=/home/user/miniconda/bin:$PATH
ENV CONDA_AUTO_UPDATE_CONDA=false

# Create a Python 3.6 environment.
RUN /home/user/miniconda/bin/conda install conda-build \
 && /home/user/miniconda/bin/conda create -y --name py36 python=3.6.5 \
 && /home/user/miniconda/bin/conda clean -ya
ENV CONDA_DEFAULT_ENV=py36
ENV CONDA_PREFIX=/home/user/miniconda/envs/$CONDA_DEFAULT_ENV
ENV PATH=$CONDA_PREFIX/bin:$PATH

# CUDA 9.0-specific steps.
RUN conda install -y -c pytorch \
    cuda90=1.0 \
    magma-cuda90=2.4.0 \
    "pytorch=1.1.0=py3.6_cuda9.0.176_cudnn7.5.1_0" \
    torchvision=0.2.1 \
 && conda clean -ya

# Install HDF5 Python bindings.
RUN conda install -y h5py=2.8.0 \
 && conda clean -ya
RUN pip install h5py-cache==1.0

# Install TorchNet, a high-level framework for PyTorch.
RUN pip install torchnet==0.0.4

# Install Requests, a Python library for making HTTP requests.
RUN conda install -y requests=2.19.1 \
 && conda clean -ya

# Install Graphviz.
RUN conda install -y graphviz=2.38.0 \
 && conda clean -ya
RUN pip install graphviz==0.8.4

# Install OpenCV3 Python bindings.
RUN sudo apt-get update && sudo apt-get install -y --no-install-recommends \
    libgtk2.0-0 \
    libcanberra-gtk-module \
 && sudo rm -rf /var/lib/apt/lists/*
RUN conda install -y -c menpo opencv3=3.1.0 \
 && conda clean -ya

# Install PyTorch Geometric.
RUN CPATH=/usr/local/cuda/include:$CPATH \
 && LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
 && DYLD_LIBRARY_PATH=/usr/local/cuda/lib:$DYLD_LIBRARY_PATH

RUN pip install --verbose --no-cache-dir torch-scatter \
 && pip install --verbose --no-cache-dir torch-sparse \
 && pip install --verbose --no-cache-dir torch-cluster \
 && pip install --verbose --no-cache-dir torch-spline-conv \
 && pip install torch-geometric

# Set the default command to python3.
CMD ["python3"]

I've tried starting it with FROM pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime and commenting everything up to # PyTorch (Geometric) installation and the section on # CUDA 9.0-specific steps. for

RUN conda install -c pytorch pytorch
RUN conda install -c fragcolor cuda10.0 && conda clean -ya

and commenting out

# Install Graphviz.
RUN conda install -y graphviz=2.38.0 \
 && conda clean -ya
RUN pip install graphviz==0.8.4

which didn't seem to work even with CUDA9.0

This makes the docker work and load, pytorch to be able to be imported and cuda to work as well. However, when I try to import torch_geometric I get ModuleNotFoundError: No module named 'torch_scatter.scatter_cuda'

Since the package is well-maintained (4.5k stars, mentioned in the pytorch website), it seems to me it's likely my fault and something general about how to adapt from CUDA9.0 to CUDA10.0.

I'd appreciate any advice on what I could be doing wrong or a way of changing it without removing so many lines from the original Dockerfile, which is probably what's causing the issue.


title: Pytorch model.train() and model.eval() behave in a weird way

body:
My model is a CNN based one with multiple BN layers and DO layers.
So originally, I accidentally put model.train() outside of the loop just like the following:

model.train()
for e in range(num_epochs):
    # train model
    model.eval()
    # eval model

For the record, the code above trained well and performed decently on validation set:

[CV:02][E:001][I:320/320] avg. Loss: 0.460897, avg. Acc: 0.742746, test. acc: 0.708046(max: 0.708046)
[CV:02][E:002][I:320/320] avg. Loss: 0.389883, avg. Acc: 0.798791, test. acc: 0.823563(max: 0.823563)
[CV:02][E:003][I:320/320] avg. Loss: 0.319034, avg. Acc: 0.825559, test. acc: 0.834914(max: 0.834914)
[CV:02][E:004][I:320/320] avg. Loss: 0.301322, avg. Acc: 0.834254, test. acc: 0.834052(max: 0.834914)
[CV:02][E:005][I:320/320] avg. Loss: 0.292184, avg. Acc: 0.839575, test. acc: 0.835201(max: 0.835201)
[CV:02][E:006][I:320/320] avg. Loss: 0.285467, avg. Acc: 0.842266, test. acc: 0.837931(max: 0.837931)
[CV:02][E:007][I:320/320] avg. Loss: 0.279607, avg. Acc: 0.844917, test. acc: 0.829885(max: 0.837931)
[CV:02][E:008][I:320/320] avg. Loss: 0.275252, avg. Acc: 0.846443, test. acc: 0.827874(max: 0.837931)
[CV:02][E:009][I:320/320] avg. Loss: 0.270719, avg. Acc: 0.848150, test. acc: 0.822989(max: 0.837931)

While reviewing the code, however, I realized I made a mistake because the code above would turn off BN layers and DO layers after the first iteration.

So, I moved the line: model.train() inside the loop:

for e in range(num_epochs):
    model.train()
    #train model
    model.eval()
    #eval model

At this point, the model was learning relatively poorly(seemed like the model overfit as you can see in the following output). It had higher training accuracy, but significantly lower accuracy on validation set(it was starting to get weird at this point considering the usual effect of BN and DO):

[CV:02][E:001][I:320/320] avg. Loss: 0.416946, avg. Acc: 0.750477, test. acc: 0.689080(max: 0.689080)
[CV:02][E:002][I:320/320] avg. Loss: 0.329121, avg. Acc: 0.798992, test. acc: 0.690948(max: 0.690948)
[CV:02][E:003][I:320/320] avg. Loss: 0.305688, avg. Acc: 0.829053, test. acc: 0.719540(max: 0.719540)
[CV:02][E:004][I:320/320] avg. Loss: 0.290048, avg. Acc: 0.840539, test. acc: 0.741954(max: 0.741954)
[CV:02][E:005][I:320/320] avg. Loss: 0.279873, avg. Acc: 0.848872, test. acc: 0.745833(max: 0.745833)
[CV:02][E:006][I:320/320] avg. Loss: 0.270934, avg. Acc: 0.854274, test. acc: 0.742960(max: 0.745833)
[CV:02][E:007][I:320/320] avg. Loss: 0.263515, avg. Acc: 0.856945, test. acc: 0.741667(max: 0.745833)
[CV:02][E:008][I:320/320] avg. Loss: 0.256854, avg. Acc: 0.858672, test. acc: 0.734483(max: 0.745833)
[CV:02][E:009][I:320/320] avg. Loss: 0.252013, avg. Acc: 0.861363, test. acc: 0.723707(max: 0.745833)
[CV:02][E:010][I:320/320] avg. Loss: 0.245525, avg. Acc: 0.865519, test. acc: 0.711494(max: 0.745833)

So I thought to myself: "I guess BN layers and DO layers are having negative effects in my model" and hence, removed them.  However, the model didn't perform well with BN and DO layers removed either(In fact, the model didn't seem to be learning anything):

[CV:02][E:001][I:320/320] avg. Loss: 0.552687, avg. Acc: 0.749071, test. acc: 0.689080(max: 0.689080)
[CV:02][E:002][I:320/320] avg. Loss: 0.506234, avg. Acc: 0.749071, test. acc: 0.689080(max: 0.689080)
[CV:02][E:003][I:320/320] avg. Loss: 0.503373, avg. Acc: 0.749071, test. acc: 0.689080(max: 0.689080)
[CV:02][E:004][I:320/320] avg. Loss: 0.502966, avg. Acc: 0.749071, test. acc: 0.689080(max: 0.689080)
[CV:02][E:005][I:320/320] avg. Loss: 0.502870, avg. Acc: 0.749071, test. acc: 0.689080(max: 0.689080)
[CV:02][E:006][I:320/320] avg. Loss: 0.502832, avg. Acc: 0.749071, test. acc: 0.689080(max: 0.689080)
[CV:02][E:007][I:320/320] avg. Loss: 0.502800, avg. Acc: 0.749071, test. acc: 0.689080(max: 0.689080)
[CV:02][E:008][I:320/320] avg. Loss: 0.502765, avg. Acc: 0.749071, test. acc: 0.689080(max: 0.689080)

I was very confused at this point.  I went further and carried out another experiment. I placed BN layers and DO layers back into the model and tested the following:

for e in range(num_epochs):
    model.eval()
    # train model
    # eval model

It worked poorly:

[CV:02][E:001][I:320/320] avg. Loss: 0.562196, avg. Acc: 0.744774, test. acc: 0.689080(max: 0.689080)
[CV:02][E:002][I:320/320] avg. Loss: 0.506071, avg. Acc: 0.749071, test. acc: 0.689080(max: 0.689080)
[CV:02][E:003][I:320/320] avg. Loss: 0.503234, avg. Acc: 0.749071, test. acc: 0.689080(max: 0.689080)
[CV:02][E:004][I:320/320] avg. Loss: 0.502916, avg. Acc: 0.749071, test. acc: 0.689080(max: 0.689080)
[CV:02][E:005][I:320/320] avg. Loss: 0.502859, avg. Acc: 0.749071, test. acc: 0.689080(max: 0.689080)
[CV:02][E:006][I:320/320] avg. Loss: 0.502838, avg. Acc: 0.749071, test. acc: 0.689080(max: 0.689080)

I performed the above experiments multiple times and the results were not far off the outputs I have posted above. (Data I'm working with is pretty simple).

To summarize, the model works best in a very special setting.

Batch Normalization and Dropout added to the model. (This is good).
Train the model with model.train() on only for the first epoch. (Weird.. combined with 3)
Train the model with model.eval() on for the rest of the iterations. (Weird as well)

To be honest, I wouldn't have set up the training procedure as above(I don't think anyone would), but it works well for some reason.  Has anybody experienced anything similar? or if you could guide me as to why the model behaves this way, it would be much appreciated!!

Thanks in advance!!


I'm writing a neural network to do regression and here is my codes:

class Model(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super().__init__()
        self.h1 = nn.Linear(input_size, hidden_size)
        self.h2 = nn.Linear(hidden_size, hidden_size)
        self.h3 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        x = self.h1(x)
        x = Fuc.tanh(x)
        x = self.h2(x)
        x = Fuc.relu(x)
        x = self.h3(x)
        return x

model = Model(input_size=input_size, hidden_size=hidden_size, num_classes=num_classes)
opt = optim.Adam(params=model.parameters(), lr=learning_rate)

for epoch in range(1000):
    out = model(data)
    print('target', target)
    print('pred', out)
    loss = torch.nn.MSELoss(out, target)
    print('loss', loss)

    model.zero_grad()
    loss.backward()
    opt.step()

my input is of shape (numberOfSample X 2) and out put is of the form [[2],[3],...], namely a list of lists where each inner list contain one number.

Ok so Now I train the neural network and got this error:

       ...
       [-0.1753],
        [-0.1753],
        [-0.1753]], grad_fn=<AddmmBackward>)
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1340: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.
  warnings.warn("nn.functional.tanh is deprecated. Use torch.tanh instead.")
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-26-38e8026bfe54> in <module>()
     68     print('target', target)
     69     print('pred', out)
---> 70     loss = torch.nn.MSELoss(out, target)
     71     print('loss', loss)
     72

2 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/_reduction.py in legacy_get_string(size_average, reduce, emit_warning)
     34         reduce = True
     35
---> 36     if size_average and reduce:
     37         ret = 'mean'
     38     elif reduce:

RuntimeError: bool value of Tensor with more than one value is ambiguous


title: How can I fix the “TypeError: 'Tensor' object is not callable” error in Pytorch?

body:
I am trying to compute a linear function an image's pixels, followed by log softmax (it's for a classification task). I am not sure how to do this without getting errors. Here is the relevant code:
...
torch.nn.functional.nll_loss(output, target) # error happens here
...

def __init__(self):
    super(NetLin, self).__init__()
    self.in_out = torch.nn.Linear(28, 2)

def forward(self, input):
    out_sum = self.in_out(input)
    output = torch.nn.LogSoftmax(out_sum)
    return output

and the full error message I get is:
Traceback (most recent call last):
  File "copy.py", line 98, in <module>
    main()
  File "copy.py", line 94, in main
    train(args, net, device, train_loader, optimizer, epoch)
  File "copy.py", line 21, in train
    loss = torch.nn.functional.nll_loss(output, target)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/functional.py", line 2107, in nll_loss
    dim = input.dim()
TypeError: 'Tensor' object is not callable

I have tried a few different solutions to this based on other answers online but they just result in different error messages. Clearly I am doing something fundamentally wrong here but I haven't used Pytorch before so I'm not sure what it is. Thank you
Edit:
My code is now:
def train(args, model, device, train_loader, optimizer, epoch):
    if args.net == 'lin':
        model = NetLin()
    model.train()
    loss = nn.NLLLoss()
    for batch_idx, (data, target) in enumerate(train_loader):
        data.requires_grad = True
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = loss(model(input), target)
        F.nll_loss(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % 100 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))

class NetLin(nn.Module):
    def __init__(self):
        super(NetLin, self).__init__()
        self.in_out = torch.nn.Linear(28 * 28, 2)

    def forward(self, input):
        input = input.view(-1, 28 * 28)
        out_sum = self.in_out(input)
        output = torch.nn.LogSoftmax(out_sum, dim=1)
        return output

and my error message is now:
Traceback (most recent call last):
  File "copy.py", line 102, in <module>
    main()
  File "copy.py", line 98, in main
    train(args, net, device, train_loader, optimizer, epoch)
  File "copy.py", line 24, in train
    output = loss(model(input), target)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/.../copy.py", line 15, in forward
    input = input.view(-1, 28 * 28)
AttributeError: 'builtin_function_or_method' object has no attribute 'view'

As you can kind of see the data and target are read in from a file (they are from KMNIST actually) so I can't control their format exactly, but I do know the image sizes are all [1,28,28], i.e. a 28*28 greyscale image. Also the batch size is 64 in case that matters.


title: Get LR from cyclical learning rate in PyTorch

body:
I'm trying to implement the cyclical learning rate approach on top of the PyTorch reimplementation of StyleGAN by rosinality. To do so, I am building on what suggested in this blog post.

To check how the loss changes as a function of the learning rate, I need to plot how the (LR, loss) changes. Here you can find my modified version of train.py. These are the main changes I made:

Define cyclical_lr, a function regulating the cyclical learning rate

def cyclical_lr(stepsize, min_lr, max_lr):

    # Scaler: we can adapt this if we do not want the triangular CLR
    scaler = lambda x: 1.

    # Lambda function to calculate the LR
    lr_lambda = lambda it: min_lr + (max_lr - min_lr) * relative(it, stepsize)

    # Additional function to see where on the cycle we are
    def relative(it, stepsize):
        cycle = math.floor(1 + it / (2 * stepsize))
        x = abs(it / stepsize - 2 * cycle + 1)
        return max(0, (1 - x)) * scaler(cycle)

    return lr_lambda

Implement the cyclical learning rate for both the discriminator and the generator

step_size = 5*256
end_lr = 10**-1
factor = 10**5
clr = cyclical_lr(step_size, min_lr=end_lr / factor, max_lr=end_lr)
scheduler_g = torch.optim.lr_scheduler.LambdaLR(g_optimizer, [clr, clr])

d_optimizer = optim.Adam(discriminator.parameters(), lr=args.lr, betas=(0.0, 0.99))
scheduler_d = torch.optim.lr_scheduler.LambdaLR(d_optimizer, [clr])

Do you have suggestions on how to plot how the loss changes as a function of the learning rate? Ideally, I would like to do it using TensorBoard, where for now I am plotting the generator loss, the discriminator loss and the size of the generated images as a function of the iteration number:

    if (i + 1) % 100 == 0:
        writer.add_scalar('Loss/G', gen_loss_val, i * args.batch.get(resolution))
        writer.add_scalar('Loss/D', disc_loss_val, i * args.batch.get(resolution))
        writer.add_scalar('Step/pixel_size', (4 * 2 ** step), i * args.batch.get(resolution))
        print(args.batch.get(resolution))


title: how to implement Grad-CAM on your own network?

body:
I want to implement Grad-CAM on my own network, should I save my model and load it, then treat my saved model like VGG-16, then do similar operations?

I tried to search on the internet, and I found that all methods are based on famous models, not their owns.

So I wonder, maybe I just need to treat my own model as VGG-16, then do similar things.


title: Too slow first run TorchScript model and its implementation in Flask

body:
I'm trying to deploy torchscripted model in Python and Flask. As I realized (at least as mentioned here ) that scripted models need to be "warmed up" before using, so first run of such models takes much longer than subsequent ones. My question is: is there any way to load torchscripted models in Flask route and predict without loss of "worm-up" time? Can I store somewhere "warm-uped" model to avoid warming-up in every request?
I wrote simple code that reproduce the "warm-up" pass:

    import torchvision, torch, time
    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)
    model = torch.jit.script(model)
    model.eval()
    x = [torch.randn((3,224,224))]
    for i in range(3):
      start = time.time()
      model(x)
      print(‘Time elapsed: {}’.format(time.time()-start))

Output:

Time elapsed: 38.29

Time elapsed: 6.65

Time elapsed: 6.65

And Flask code:

    import torch, torchvision, os, time
    from flask import  Flask
    app = Flask(__name__)
    @app.route('/')
    def test_scripted_model(path='/tmp/scripted_model.pth'):
        if os.path.exists(path):
            model = torch.jit.load(path, map_location='cpu')
        else:
            model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)
            model = torch.jit.script(model)
            torch.jit.save(model, path)
        model.eval()
        x = [torch.randn((3, 224, 224))]
        out = ''
        for i in range(3):
            start = time.time()
            model(x)
            out += 'Run {} time: {};\t'.format(i+1, round((time.time() - start), 2))
        return out
    if __name__ == '__main__':
        app.run(host='0.0.0.0', port=5000, debug=False)

Output:

Run 1 time: 46.01; Run 2 time: 8.76; Run 3 time: 8.55;

OS: Ubuntu 18.04 & Windows10
Python version: 3.6.9
Flask: 1.1.1
Torch: 1.4.0
Torchvision: 0.5.0

Update:

Solved "warm-up" problem as:

with torch.jit.optimized_execution(False):
    model(x)

Update2:
Solved Flask problem (as mentioned below) with creating global python model object before server starts and warming up it there. Then in each request the model is ready to use.

    model = torch.jit.load(path, map_location='cpu').eval()
    model(x)
    app = Flask(__name__)

and then in @app.route:

@app.route('/')
def test_scripted_model():
    global model
    ...
    ...


title: Parallel hyperparameter optimization with pytorch on a multi-gpu machine

body:
I have access to a multi-gpu machine and I am running a grid search loop for parameter optimisation. I would like to know if I can distribute several iterations of the loop on multiple gpu at the same time, and if so how do I do it (what me mechanism? threading? how to gather the results if the loop execute asynchronously? etc.)
Thank you.


I have seen this problem the first time, I never encountered such an error in previous Python projects. Here is my training code:
def train(net, opt, criterion,ucf_train, batchsize,i):
    opt.zero_grad()
    total_loss = 0
    net=net.eval()
    net=net.train()
    for vid in range(i*batchsize,i*batchsize+batchsize,1):

        output=infer(net,ucf_train[vid])
        m=get_label_no(ucf_train[vid])
        m=m.cuda( )
        loss = criterion(output,m)
        loss.backward(retain_graph=True)
        total_loss += loss
        opt.step()       #updates wghts and biases

    return total_loss/n_points

code for infer(net,input)
def infer(net, name):
    net.eval()
    hidden_0 = net.init_hidden()
    hidden_1 = net.init_hidden()
    hidden_2 = net.init_hidden()
    video_path = fetch_ucf_video(name)
    cap = cv2.VideoCapture(video_path)
    resize=(224,224)
    T=FrameCapture(video_path)
    print(T)
    lim=T-(T%20)-2
    i=0
    while(1):
      ret, frame2 = cap.read()
      frame2= cv2.resize(frame2, resize)
    #  print(type(frame2))
      if (i%20==0 and i<lim):
          input=normalize(frame2)
          input=input.cuda()
          output,hidden_0,hidden_1, hidden_2  = net(input, hidden_0, hidden_1, hidden_2)
      elif (i>=lim):
          break
      i=i+1
    op=output
    torch.cuda.empty_cache()
    op=op.cuda()
    return op

I am getting this error, I tried with model.train() following this where net is my model:
 RuntimeError                              Traceback (most recent call last)
<ipython-input-62-42238f3f6877> in <module>()
----> 1 train(net1,opt,criterion,ucf_train,1,0)

2 frames
/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    125     Variable._execution_engine.run_backward(
    126         tensors, grad_tensors, retain_graph, create_graph,
--> 127         allow_unreachable=True)  # allow_unreachable flag
    128
    129

RuntimeError: cudnn RNN backward can only be called in training mode


title: Is there another step to importing Pytorch and using it?

body:
I'm trying to test Pytorch but the first step is to get it running and usable on my computer. I have it to an extent but I need it to function in VS code which it does, it's just that it reads as an error despite working.

I've set up pytorch locally on my computer. I can run the test scripts fine in VS Code's integrated terminal.

The issue comes up when I try to do some work normally in VS code.

from __future__ import print_function
import torch
x = torch.rand(5, 3) //This line has an error. torch.rand(5,3) is not callable)
print(x)

It says that module 'torch' has no 'rand' member. But it still outputs correctly.
Running this in the terminal as a python file or in debug mode results in.

(base) c:\Users\Sean\Desktop\Test\hello>D:\Anaconda\python.exe c:\Users\Sean\.vscode\extensions\ms-python.python-2019.10.44104\pythonFiles\ptvsd_launcher.py --default --client --host localhost --port 63625 c:\Users\Sean\Desktop\Test\hello\something.py
tensor([[0.5449, 0.1669, 0.4740],
        [0.3079, 0.0447, 0.9543],
        [0.9137, 0.3987, 0.5736],
        [0.1788, 0.4932, 0.5584],
        [0.1632, 0.6285, 0.4483]])

(base) c:\Users\Sean\Desktop\Test\hello>D:/Anaconda/Scripts/activate

(base) c:\Users\Sean\Desktop\Test\hello>conda activate Anaconda
Could not find conda environment: Anaconda
You can list all discoverable environments with `conda info --envs`.

So it's working, what I want is to not have it coming up as an error. It'll be hard to find the actual bugs later on if it continues and I feel it might cause problems if not addressed. Any help with telling me the root issue and how to go about fixing it would be appreciated.


title: Multi GPU training in Pytorch causes my system to freeze

body:
When I wrap my model in nn.DataParallel(model) and start training my screen freezes and I have to manually restart the computer every time.
I've tried a few variations, like not adding .to(device) to every x and y, but whenever nn.DataParallel is used I seem to cause the computer to freeze.
import random
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torchvision import models, datasets, transforms

import torch.utils.data

DataLoader = torch.utils.data.DataLoader
random_split = torch.utils.data.random_split

global_rank = 0
MNIST = datasets.MNIST

class MLPClassifier(nn.Module):
    def __init__(self):
        super(MLPClassifier, self).__init__()
        self.layer_1 = torch.nn.Linear(28 * 28, 128)
        self.layer_2 = torch.nn.Linear(128, 444)
        self.layer_3 = torch.nn.Linear(444, 333)
        self.layer_4 = torch.nn.Linear(333, 10)

    def forward(self, x):
        x = x.view(x.size(0), -1)
        x = self.layer_1(x)
        x = F.relu(x)
        x = self.layer_2(x)
        x = F.relu(x)
        x = self.layer_3(x)
        x = F.relu(x)
        x = self.layer_4(x)
        return x

# Download data
if global_rank == 0:
    mnist_train = MNIST(os.getcwd(), train=True, download=True)
    mnist_test = MNIST(os.getcwd(), train=False, download=True)

# dist.barrier()

#transforms
transform=transforms.Compose([transforms.ToTensor(),
                             transforms.Normalize((0.1307,), (0.3081,))])
mnist_train = MNIST(os.getcwd(), train=True, transform=transform)

# Split dataset
mnist_train, mnist_val = random_split(mnist_train, [55000, 5000])
mnist_test = MNIST(os.getcwd(), train=False, download=True)

# Build dataloaders
mnist_train = DataLoader(mnist_train, batch_size=256)
mnist_val = DataLoader(mnist_val, batch_size=256)
mnist_test = DataLoader(mnist_test, batch_size=256)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

model = MLPClassifier()
model = nn.DataParallel(model)
model.to(device)

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

# Train loop
model.train()
num_epochs = 1
for epoch in range(num_epochs):
    for train_batch in mnist_train:
        x, y = train_batch
        logits = model(x.to(device))
        loss = F.cross_entropy(logits, y.to(device))
        print('rain loss: ', loss.item())
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

        # EVAL LOOP
        model.eval()
        with torch.no_grad():
            val_loss_a = []
            for val_batch in mnist_val:
                x, y = val_batch
                logits = model(x.to(device))
                val_loss = F.cross_entropy(logits, y.to(device))
                val_loss_a.append(val_loss)
            avg_val_loss = torch.stack(val_loss_a).mean()
            model.train()


title: Pytorch: Randomly subsample loss tensors using `torch.randperm`

body:
I'm trying to randomly subsample the prediction and target array for my loss calculation.

idx = torch.randperm(target.shape[0])
target = target.index_select(0, idx[0, sample_size]

However I'm getting this error message.

index_select(): argument 'index' (position 2) must be Variable, not torch.LongTensor

Does anyone know how to fix this?

Edit:
I got one step closer. It seems like torch.randperm does not return a torch variable, so one has to explicitly convert the output:

idx = torch.randperm(target.shape[0])
idx = Variable(idx).cuda()
target = target.index_select(0, idx[0, sample_size]

only problem is now that the backpropagation fails. Seems like the operation of randomly subsampling is causing an issue with the dimensions.
However the dimensions seem to be fine when calculating the loss:

loss = F.nll_loss(prediction, target.view(-1)) # prediction shape is [Nx12] and target shape is N

Unfortunately when calling loss.backward() I get this error message:

RuntimeError: The expanded size of the tensor (12) must match the existing size (217456) at non-singleton dimension 1


title: How to return intermideate gradients (for non-leaf nodes) in pytorch?

body:
My question is concerning the syntax of pytorch register_hook.

x = torch.tensor([1.], requires_grad=True)
y = x**2
z = 2*y

x.register_hook(print)
y.register_hook(print)

z.backward()

outputs:

tensor([2.])
tensor([4.])

this snippet simply prints the gradient of z w.r.t x and y, respectively.

Now my (most likely trivial) question is how to return the intermediate gradients (rather than only printing)?

UPDATE:

It appears that calling retain_grad() solves the issue for leaf nodes. ex. y.retain_grad().

However, retain_grad does not seem to solve it for non-leaf nodes. Any suggestions?


title: KL Divergence of Normal and Laplace isn't Implemented in TensorFlow Probability and PyTorch

body:
In both TensorFlow Probability (v0.4.0) and PyTorch (v0.4.1) the KL Divergence of the Normal distribution (tfp, PyTorch) and the Laplace distribution (tfp, PyTorch) isn't implemented resulting in a NotImplementedError error being thrown.

>>> import tensorflow as tf
>>> import tensorflow_probability as tfp
>>> tfd = tfp.distributions
>>> import torch
>>>
>>> tf.__version__
'1.11.0'
>>> tfp.__version__
'0.4.0'
>>> torch.__version__
'0.4.1'
>>>
>>> p = tfd.Normal(loc=0., scale=1.)
>>> q = tfd.Laplace(loc=0., scale=1.)
>>> tfd.kl_divergence(p, q)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda/envs/example/lib/python3.6/site-packages/tensorflow/python/ops/distributions/kullback_leibler.py", line 95, in kl_divergence
    % (type(distribution_a).__name__, type(distribution_b).__name__))
NotImplementedError: No KL(distribution_a || distribution_b) registered for distribution_a type Normal and distribution_b type Laplace
>>>
>>> a = torch.distributions.normal.Normal(loc=0., scale=1.)
>>> b = torch.distributions.laplace.Laplace(loc=0., scale=1.)
>>> torch.distributions.kl.kl_divergence(a,b)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda/envs/example/lib/python3.6/site-packages/torch/distributions/kl.py", line 161, in kl_divergence
    raise NotImplementedError
NotImplementedError

I assume as this is missing from both of these libraries that there is some good reason for this and that the user would be expected to implement it themselves with tfp.distributions.RegisterKL in TensorFlow Probability and torch.distributions.kl.register_kl in PyTorch.

Is this the correct assumption? If so, can someone explain why the KL Divergence wouldn't be implemented for given distribution classes? I think I am missing something very basic about this.

If my assumption is wrong, can someone explain how to properly have TensorFlow and PyTorch implement these operations?

For additional reference, using for this example an older version of TensorFlow that works with Edward,

pip install tensorflow==1.7
pip install edward

In this minimal example above, I'm trying to implement the equivalent of the following edward toy example code in tfp (or in torch).

import tensorflow as tf
import edward as ed

p = ed.models.Normal(loc=0., scale=1.)
s = tf.Variable(1.)
q = ed.models.Laplace(loc=0., scale=s)
inference = ed.KLqp({p: q})
inference.run(n_iter=5000)


title: Pytorch: Is it possible to load model when parameters or size have changed?

body:
A Pytorch model (graph, weights, and biases)  is saved with :

torch.save(self.state_dict(), file)

and loaded with :

self.load_state_dict(torch.load(file))

But if parameters are changed, the model won't load with error for example :

RuntimeError: Error(s) in loading state_dict for LeNet5:
    size mismatch for conv1.weight:

Is it possible to load to the model with changed size?
Sort of filling the rest of weights like in initialization (if there are more weights) and clip if there are fewer weights?


title: LSTM many-to-many synced sequence input and output Pytorch loss not decreasing?

body:
I am working on a a project where I built an LSTM model for seq2seq, where I have a synced sequence input and output. My audio time series is 32000 in length and my labels are also 32000 in length. And we wish to make a classification (fake or real audio) decision on each sample of the audio. So my tensors look like this for 1 audio example

time series

tensor([-0.1635, -0.1510, -0.1455,  ..., -0.3338, -0.3163, -0.2944])

labels 0s and 1s corresponding to each sample

tensor([0, 0, 0,  ..., 0, 0, 0])

In all the audios, there is about 45-50% of the audio samples that is labeled 1 for fake, and basically the other half labeled 0 for real. So class imbalance isn't an issue here. Also, the fake audio will always be in a certain region of the audio. So for example in a audio time series of length 32000, the fake region could be from 22000 to 30000, or 5000 to 24000 just to give two examples.

I initially tried this problem with an LSTM that takes a fixed input size of 32000 and trained it. What I noticed is that my model was only predicting one class and the loss was never decreasing, after like 5 epochs even. I then read that LSTMs with long input sequences like the audio that I have, is prone to vanishing gradients and possibly an unlearnable model. So I changed my training method so that for each batch in the Pytorch Dataloader, we split the audio time series of 32000 in length to smaller chunks (I tried 100, 200, 500, 1000...etc) and we do backpropogation on these smaller chunks. With this method my model was able to have some recall during training so it was predicting both classes, but still the LOSS was not decreasing at all, even after like 5 epochs.

class LSTM(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, output_dim = 2, batch_size = batch_size):
        super(LSTM , self).__init__()
        self.num_layers = num_layers
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.batch_size = batch_size
        self.rnn_state = None
        self.lstm = nn.LSTM(self.input_size , self.hidden_size , self.num_layers, batch_first = True)
        self.to_linear = nn.Linear(self.hidden_size, output_dim)
    def repackage_rnn_state(self):

        self.rnn_state = self._detach_rnn_state(self.rnn_state)

    def _detach_rnn_state(self, h):

        if isinstance(h, torch.Tensor):
            return h.detach()
        else:
            return tuple(self._detach_rnn_state(v) for v in h)

    def forward(self, x):
        lstm_out, self.rnn_state = self.lstm(x , self.rnn_state)
        logits = self.to_linear(lstm_out)
        scores = F.softmax(logits, dim = 2)
        return scores

for epoch in range(num_epochs):

    running_loss_train , accuracy_train = 0 , 0

    network.hidden = network.init_hidden()

    for batch_idx, batch in enumerate(train_loader):

        new_time_series , new_labels = batch

        new_time_series, new_labels = new_time_series.to(device), new_labels.to(device)

        optimizer.zero_grad()

        # forward pass feeding the padded sequence

        preds = network(new_time_series.unsqueeze(2))

        preds = preds.view(preds.shape[0]*preds.shape[1], preds.shape[2])

        train_loss = criterion(preds, new_labels.flatten().long())

        train_loss.backward()

        # gradient clipping here just before the parameter updates, clips inplace, clip value = 1

        torch.nn.utils.clip_grad_norm_(network.parameters(),clip)

        optimizer.step()

        running_loss_train+=train_loss.detach().item()/new_time_series.shape[0]

        _ ,predictions = torch.max(preds,1)

        metrics = get_evaluations_ignoring_padding(new_labels, predictions, 2)

        accuracy_train+=metrics['Accuracy']

        if batch_idx%100==0:
            print('Epoch : %d | BatchID : %d | Train Loss : %.3f | Train Acc : %.3f ' %
                  (epoch , batch_idx, train_loss.item() / new_time_series.shape[0] ,
                   metrics['Accuracy']))

The above code is for when I did the "truncated" backpropogation splitting my time series into multiple chunks. So before I didn't have the repackage_rnn_state or _detach_rnn_state when I trained on the 32000 length. Doing it the original way and with the chunk method again both did not decrease loss.

I previously did a many-to-one classification on real vs fake audio, where half of the audio didn't have any fake regions and the other Half has some fake regions. I got 99% accuracy on that CNN model and only after 3 epochs. So I'm thinking my data is pretty easy to train on and just because it's now a many-to-many problem there shouldn't be learning issues.

Is there something wrong with my code??

Is it not appropriate to train an LSTM on my kind of problem?

I'd like some advice about how you would go about training this kind of task.

Please let me know if you need more information and I will update my question and thanks


title: Pytorch dataloader from csv of file paths and labels

body:
I have a csv file for train and test datasets that contains the file location and the label. The head of this data frame is:

df.head()
Out[46]:
             file_path  label
0  \\images\\29771.png      0
1  \\images\\55201.png      0
2  \\images\\00715.png      1
3  \\images\\33214.png      0
4  \\images\\99841.png      1

I have multiple locations for the file paths, and limited space, so I can't copy them into \0 and \1 folder locations. How can I use this data frame to create a pytorch dataloader and/or dataset object?


title: Why Pytorch is slower than Tensorflow in read data

body:
I tried two version code for iterate MNIST data to compare the elapsed time.
Pytorch version
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
import torch
from torchvision import datasets, transforms
import matplotlib.pyplot as plt
import time
train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('~/data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       # transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=30000, shuffle=True,pin_memory=True,num_workers=4)
tic = time.time()
for epoch in range(0, 5):
    for batch_idx, (data, target) in enumerate(train_loader):
        continue
toc=time.time()
print('elapsed time:',toc-tic)

Tensorflow 2.x version
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
import tensorflow as tf
from tensorflow.keras import datasets
import time
(xs, ys), (xs_, ys_) = datasets.mnist.load_data()
# print('datasets:', xs.shape, ys.shape, xs.min(), xs.max())
xs = tf.convert_to_tensor(xs, dtype=tf.float32) / 255.
db = tf.data.Dataset.from_tensor_slices((xs, ys))
db = db.batch(30000)
tic = time.time()
for epoch in range(5):
    for step, (x, y) in enumerate(db):
        continue
toc = time.time()
print('elapsed time:', toc - tic)

And the result are TF elapsed 2s and Pytorch elapsed 15s.
So, why Pytorch is slower than Tensorflow in read data?
Am I setting it wrong?
Thanks!


title: Minimalistic app getting "OSError: [Errno 12] Cannot allocate memory" on deploy - why?

body:
I'm setting up a new Google App Engine with Python 3.7 runtime. However, it fails even with a very minimal setup.

The app.yaml is as simple as:

runtime: python37

The main.py is as simple as:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return 'Hello World!'

And finally the requirements.txt is as simple as:

Flask>=1.0.2
torch==1.0.1
Flask-Compress>=1.4.0
numpy>=1.16.2

I expected that this setup would work when issuing gcloud app deploy, but istead I'm getting an error - this is the last lines of the error log on Google Cloud Build:

(log lines before this omitted as they're on INFO level)
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/bin/ftl.par/__main__.py", line 65, in <module>
File "/usr/local/bin/ftl.par/__main__.py", line 54, in main
File "/usr/local/bin/ftl.par/__main__/ftl/python/builder.py", line 114, in Build
File "/usr/local/bin/ftl.par/__main__/ftl/python/layer_builder.py", line 153, in BuildLayer
File "/usr/local/bin/ftl.par/__main__/ftl/common/single_layer_image.py", line 60, in GetCacheKey
File "/usr/local/bin/ftl.par/__main__/ftl/python/layer_builder.py", line 109, in GetCacheKeyRaw
File "/usr/local/bin/ftl.par/__main__/ftl/python/layer_builder.py", line 332, in GetCacheKeyRaw
File "/usr/local/bin/ftl.par/__main__/ftl/python/layer_builder.py", line 346, in _python_version
File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 938, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

Why does it fail? Is it normal that it uses python 2.7 here, when it uses python 3.7 in the beginning? Cannot seem to find any documentation for this specific bug - I've tried to increase memory in the app.yaml file without any luck.


title: Correct Way to Fine-Tune/Train HuggingFace's Model from scratch (PyTorch)

body:
For example, I want to train a BERT model from scratch but using the existing configuration. Is the following code the correct way to do so?
model = BertModel.from_pretrained('bert-base-cased')
model.init_weights()

Because I think the init_weights method will re-initialize all the weights.
Second question, if I want to change a bit the configuration, such as the number of hidden layers.
model = BertModel.from_pretrained('bert-base-cased', num_hidden_layers=10)
model.init_weights()

I wonder if the above is the correct way to do so. Because they don't appear to have an error when I run the above code.


title: Pytorch not recognizing directory for dataset

body:
I'm trying to run code for a Deep Convolutional GAN from the official PyTorch site (https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html) on my Mac.

When I try loading the data, I keep getting a "FileNotFound" error.

Random Seed:  999
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-6-1019cc990fb4> in <module>()
 78                                transforms.CenterCrop(image_size),
 79                                transforms.ToTensor(),
---> 80                                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
 81                            ]))
 82 # Create the dataloader

/anaconda3/lib/python3.6/site-packages/torchvision/datasets/folder.py in __init__(self, root, transform, target_transform, loader)
176         super(ImageFolder, self).__init__(root, loader, IMG_EXTENSIONS,
177                                           transform=transform,
--> 178                                           target_transform=target_transform)
179         self.imgs = self.samples

/anaconda3/lib/python3.6/site-packages/torchvision/datasets/folder.py in __init__(self, root, loader, extensions, transform, target_transform)
 73
 74     def __init__(self, root, loader, extensions, transform=None, target_transform=None):
---> 75         classes, class_to_idx = find_classes(root)
 76         samples = make_dataset(root, class_to_idx, extensions)
 77         if len(samples) == 0:

/anaconda3/lib/python3.6/site-packages/torchvision/datasets/folder.py in find_classes(dir)
 21
 22 def find_classes(dir):
---> 23     classes = [d for d in os.listdir(dir) if os.path.isdir(os.path.join(dir, d))]
 24     classes.sort()
 25     class_to_idx = {classes[i]: i for i in range(len(classes))}

FileNotFoundError: [Errno 2] No such file or directory: 'Users/user1/Downloads/DCGANs/celeba/'

Here is where I tried loading the dataset where dataroot = "Users/user1/Downloads/DCGANs/celeba/"

The dataset is a folder (named celeba) with about 200,000 images.

dataset = dset.ImageFolder(root=dataroot,
                       transform=transforms.Compose([
                           transforms.Resize(image_size),
                           transforms.CenterCrop(image_size),
                           transforms.ToTensor(),
                           transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                       ]))

I tried this on both Atom and Jupyter Notebook, it didn't make a difference.

All help is highly appreciated :)


title: Vectorize multivariate normal pdf python (PyTorch/NumPy)

body:
I have N Gaussian distributions (multivariate) with N different means (covariance is the same for all of them) in D dimensions.

I also have N evaluation points, where I want to evaluate each of these (log) PDFs.

This means I need to get a NxN matrix, call it "kernels". That is, the (i,j)-th entry is the j-th Gaussian evaluated at the i-th point. A naive approach is:

from torch.distributions.multivariate_normal import MultivariateNormal
import numpy as np

# means contains all N means as rows and is thus N x D
# same for eval_points
# cov is not a problem , just a DxD matrix that is equal for all N Gaussians

kernels = np.empty((N,N))
    for i in range(N):
        for j in range(N):
            kernels[i][j] = MultivariateNormal(means[j], cov).log_prob(eval_points[i])

Now one for loop we can get rid of easily, since for example if we wanted all the evaluations of the first Gaussian , we simply do:

MultivariateNormal(means[0], cov).log_prob(eval_points).squeeze()

and this gives us a N x 1 list of values, that is the first Gaussian evaluated at all N points.

My problem is that , in order to get the full N x N matrix , this doesn't work:

kernels = MultivariateNormal(means, cov).log_prob(eval_points).squeeze()

It doesn't figure out that it should evaluate each mean with all evaluation points in eval_points, and it doesn't return a NxN matrix with these which would be what I want. Therefore, I am not able to get rid of the second for loop, over all N Gaussians.


title: Slicing pytorch tensors and use of data_ptr()

body:
a = tensor([[ 1,  2,  3,  4,  5],
        [ 6,  7,  8,  8, 10],
        [11, 12, 13, 14, 15]])

I have a torch tensor and I need to index a tensor c such that c = [[3], [8], [13]]
So I did c = a[:,[2]] which gave me the expected answer, but it still failed on the autograder.
The autograder uses a check function as follows -
def check(orig, actual, expected):
  expected = torch.tensor(expected)
  same_elements = (actual == expected).all().item() == 1
  same_storage = (orig.storage().data_ptr() == actual.storage().data_ptr())
  return same_elements and same_storage

print('c correct:', check(a, c, [[3], [8], [13]]))

I tried debugging it and it turns out that same_storage is false, I don't understand why orig.storage().data_ptr() == actual.storage().data_ptr() should be True, and how it makes a difference.
Update
I was able to get the correct answer by doing c = a[:, 2:3] instead of c = a[:, [2]] what is the difference?


title: Save and load a Pytorch model

body:
i am trying to train a pytorch model on colab then save the model parameters and load it on my local computer.

After training, the model parameters are stored as below:

torch.save(Model.state_dict(),PATH)

loaded as below:

device = torch.device('cpu')
Model.load_state_dict(torch.load(PATH, map_location=device))

error:

AttributeError: 'Sequential' object has no attribute 'copy'

Does anyone know how to solve this issue?


title: How to load torchvision.datasets.ImageFolder into google colab ram

body:
I'm noticing during training on google colab that getting through the first several thousand images is very fast but after that the training becomes slower.  I've timed other parts of the training to see if they are causing the problem but their time remains consistent even as the training slows down.  I believe that it is because the images are not getting stored in the ram of google colab and it has to keep pulling images from within google drive which is slower.  I also noticed that in re-running the training, the method processes the images that were previously processed very fast but then slows when it reaches images that were not processed by the previous run of the method.

I've trained on colab before using numpy arrays loading my data like this:

train_val_images = np.load("/content/drive/My Drive/Colab Notebooks/datasets/train_data.npy")

I notice when I do this that the ram in use in the notebook increases exactly by the size of the numpy array that I am loading.

In my new project I am trying to use dataloaders and torchvision datasets instead of the numpy arrays like this:

train_dataset = torchvision.datasets.ImageFolder(root="/content/drive/My Drive/Colab Notebooks/datasets/labeled_data/train", transform = transforms.ToTensor())

trainloader = torch.utils.data.DataLoader(train_dataset)

when I load the data this way I notice that the ram in use doesn't really increase.  I am loading roughly 20,000 images from my google drive this way.

I am wondering if there is a way to load the data into the google colab ram like is done when np.load() with the torchvision.datasets.ImageFolder or if there is an another alternative where I still get to use the ImageFolder interface.


title: Using SVM with different kernels as a last layer in CNN network

body:
I'm trying to replace the last fully connected layer of a CNN network with SVM using pytorch in a multi-classification problem. I've done some research and it says, that I should just replace the nn.CrossEntropyLoss with nn.MultiMarginLoss.

How does changing the criterion only actually corresponds with the "replacing fully connected layer with SVM" task? Another thing is that I'd like to use the SVM with different kernel, like for example the quadratic one.


title: Problems of Pytorch installation on Ubuntu 17.10 (GPU)

body:
I would like to use PyTorch and its GPU computations on my computer.

I have a computer running with Ubuntu 17.10. The computer (Alienware m17x) has two graphic cards:

An integrated Intel Ivybridge Mobile
A Nvidia Geforce 675M.

In order to install PyTorch, I followed the instructions on the PyTorch website pytorch.org

1) I installed CUDA 9 with the deb file: https://developer.nvidia.com/cuda-downloads

=> Linux/x86_64/Ubuntu/17.04/deb (local)

2) I installed Pytorch using the conda command line: conda install pytorch torchvision cuda90 -c pytorch

None of these two steps returned me any type of errors.

I restarted my computer. Apparently the two cards are detected:

$ lspci | grep -i vga
00:02.0 VGA compatible controller: Intel Corporation 3rd Gen Core    processor Graphics Controller (rev 09)
01:00.0 VGA compatible controller: NVIDIA Corporation GF114M [GeForce GTX 675M] (rev a1)

But apparently there is something wrong with the drivers or CUDA itself. nvidia-detector does not return me anything:

$ nvidia-detector
none

And pytorch can not use cuda:

[1]: import torch
In [2]: torch.cuda.is_available()
Out[2]: False

Could you help me? I can provide additional informations if necessary, but I am not sure what could be relevant.


title: Making custom non-trivial loss function in pytorch

body:
I'm just started with pytorch and trying to understand how to deal with custom loss functions, especially with some non trivial ones.

Problem 1. I'd like to stimulate my nn to maximize true positive rate and at the same time minimize
false discovery rate. For example increase total score on +2 for true positive, and decrease on -5 for false positive.

def tp_fp_loss(yhat, y):
    total_score = 0
    for i in range(y.size()):
        if is_tp(yhat[i],y[i]):
            total_score += 2
        if is_fp(yhat[i],y[i]):
            total_score -= 5
    return -total_score

Problem 2. In case when y is a list of positive and negative rewards (y = [10,-5, -40, 23, 11, -7]),
stimulate nn to maximize sum of rewards.

def max_reward_loss(yhat,y):
    r = torch.autograd.Variable(torch.Tensor(y[yhat >= .5]), requires_grad=True).sum()
    return -r

Maybe I'm not completely understand some autograd mechanics, functions which I implemented correctly calculate loss but
learning with them doesnt work :( What I'm doing wrong? Can anybody help me with some working solution of any of that problems?


title: Adam optimizer error: one of the variables needed for gradient computation has been modified by an inplace operation

body:
I am trying to implement Actor-Critic learning atuomation algorithm that is not same as basic actor-critic algorithm, it's little bit changed.

Anyway, I used Adam optimizer and implemented with pytorch

when i backward TD-error for Critic first, there's no error.
However, i backward loss for Actor, the error occured.

  --------------------------------------------------------------------------- RuntimeError                              Traceback (most recent call
  last)  in
       46             # update Actor Func
       47             optimizer_M.zero_grad()
  ---> 48             loss.backward()
       49             optimizer_M.step()
       50

  ~\Anaconda3\lib\site-packages\torch\tensor.py in backward(self,
  gradient, retain_graph, create_graph)
      100                 products. Defaults to False.
      101         """
  --> 102         torch.autograd.backward(self, gradient, retain_graph, create_graph)
      103
      104     def register_hook(self, hook):

  ~\Anaconda3\lib\site-packages\torch\autograd__init__.py in
  backward(tensors, grad_tensors, retain_graph, create_graph,
  grad_variables)
       88     Variable._execution_engine.run_backward(
       89         tensors, grad_tensors, retain_graph, create_graph,
  ---> 90         allow_unreachable=True)  # allow_unreachable flag
       91
       92

  RuntimeError: one of the variables needed for gradient computation has
  been modified by an inplace operation

above is the content of error

I tried to find inplace operation, but I haven't found in my written code.
I think i don't know how to handle optimizer.

Here is main code:

        for cur_step in range(1):
        action = M_Agent(state, flag)
        next_state, r = env.step(action)

        # calculate TD Error
        TD_error = M_Agent.cal_td_error(r, next_state)

        # calculate Target
        target = torch.FloatTensor([M_Agent.cal_target(TD_error)])
        logit = M_Agent.cal_logit()
        loss = criterion(logit, target)

        # update value Func
        optimizer_M.zero_grad()
        TD_error.backward()
        optimizer_M.step()

        # update Actor Func
        loss.backward()
        optimizer_M.step()

Here is the agent network

    # Actor-Critic Agent
    self.act_pipe = nn.Sequential(nn.Linear(state, 128),
                            nn.ReLU(),
                            nn.Dropout(0.5),
                            nn.Linear(128, 256),
                            nn.ReLU(),
                            nn.Dropout(0.5),
                            nn.Linear(256, num_action),
                            nn.Softmax()
                            )

     self.val_pipe = nn.Sequential(nn.Linear(state, 128),
                            nn.ReLU(),
                            nn.Dropout(0.5),
                            nn.Linear(128, 256),
                            nn.ReLU(),
                            nn.Dropout(0.5),
                            nn.Linear(256, 1)
                            )

      def forward(self, state, flag, test=None):

          temp_action_prob = self.act_pipe(state)
          self.action_prob = self.cal_prob(temp_action_prob, flag)
          self.action = self.get_action(self.action_prob)
          self.value = self.val_pipe(state)

          return self.action

I wanna update each network respectively.

and I wanna know that Basic TD Actor-Critic method uses TD error for loss??
or squared error between r+V(s') and V(s) ?


title: Saving the dictionary with str keys and a list of pytorch tensors as values

body:
I want to save a list of dictionary in which keys are indices of queries (so the keys of the dictionary are strings). The values of this dictionary are list of sparse tensors which I store as follows :
{'1185869': [tensor(indices=tensor([[   31,    59,   528,  7227, 14981]]),
         values=tensor([1., 1., 1., 1., 1.]),
         size=(30000,), nnz=5, layout=torch.sparse_coo),
  tensor(indices=tensor([[ 39,  74, 233]]),
         values=tensor([1., 1., 1.]),
         size=(30000,), nnz=3, layout=torch.sparse_coo),
  tensor(indices=tensor([[0, 1, 2]]),
         values=tensor([1., 1., 1.]),
         size=(30000,), nnz=3, layout=torch.sparse_coo),
  tensor(indices=tensor([[  36,   99,  243,  428,  513,  514,  741,  751, 1615]]),
         values=tensor([1., 1., 1., 1., 1., 1., 1., 1., 1.]),
         size=(30000,), nnz=9, layout=torch.sparse_coo),
  tensor(indices=tensor([[ 170,  356,  513,  615,  771, 1117]]),
         values=tensor([1., 1., 1., 1., 1., 1.]),
         size=(30000,), nnz=6, layout=torch.sparse_coo),
  tensor(indices=tensor([[ 9, 11]]),
         values=tensor([1., 1.]),
         size=(30000,), nnz=2, layout=torch.sparse_coo)]}

I've not saved any such dictionary before. Generally for normal dictionaries, we pickle it. If I pickle this dictionary, I'm getting a pytorch warning. This makes me doubt the method I'm using to save the dictionary. What extensions should I use to save such dictionary? Is pickling such dictionary safe? What method should I use to save it? Thanks in advance.


title: Pip cannot install PyTorch with Python 3.8.5

body:
This is the error message I get when trying to install PyTorch!
$ pip install torch
Collecting torch
  Using cached torch-0.1.2.post2.tar.gz (128 kB)
 ** On entry to DGEBAL parameter number  3 had an illegal value
 ** On entry to DGEHRD  parameter number  2 had an illegal value
 ** On entry to DORGHR DORGQR parameter number  2 had an illegal value
 ** On entry to DHSEQR parameter number  4 had an illegal value
    ERROR: Command errored out with exit status 1:
     command: 'c:\users\eriku\appdata\local\programs\python\python38\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'C:\\Users\\eriku\\AppData\\Local\\Temp\\pip-install-_q32y4e_\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\eriku\\AppData\\Loc
al\\Temp\\pip-install-_q32y4e_\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\
r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\eriku\AppData\Local\Te
mp\pip-pip-egg-info-095ert_w'
         cwd: C:\Users\eriku\AppData\Local\Temp\pip-install-_q32y4e_\torch\
    Complete output (9 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\eriku\AppData\Local\Temp\pip-install-_q32y4e_\torch\setup.py", line 147, in <module>
        import numpy as np
      File "c:\users\eriku\appdata\local\programs\python\python38\lib\site-packages\numpy\__init__.py", line 305, in <module>
        _win_os_check()
      File "c:\users\eriku\appdata\local\programs\python\python38\lib\site-packages\numpy\__init__.py", line 302, in _win_os_check
        raise RuntimeError(msg.format(__file__)) from None
    RuntimeError: The current Numpy installation ('c:\\users\\eriku\\appdata\\local\\programs\\python\\python38\\lib\\site-packages\\
numpy\\__init__.py') fails to pass a sanity check due to a bug in the windows runtime. See this issue for more information: no tinyurl
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

These are my system specs:
Host Name:                 DESKTOP
OS Name:                   Microsoft Windows 10 Pro
OS Version:                10.0.19041 N/A Build 19041
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Workstation
OS Build Type:             Multiprocessor Free
Registered Owner:          (not leaking my E-Mail here, folks)
Registered Organization:   N/A
Product ID:                (not leaking this either)
Original Install Date:     15/08/2020, 22:55:17
System Boot Time:          07/11/2020, 09:52:30
System Manufacturer:       Micro-Star International Co., Ltd.
System Model:              MS-7B09
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: AMD64 Family 23 Model 1 Stepping 1 AuthenticAMD ~3500 Mhz
BIOS Version:              American Megatrends Inc. A.70, 14/11/2018
Windows Directory:         C:\WINDOWS
System Directory:          C:\WINDOWS\system32
Boot Device:               \Device\HarddiskVolume6
System Locale:             en-us;English (United States)
Input Locale:              (take a wild guess)
Time Zone:                 (Im always awake)
Total Physical Memory:     65.428 MB
Available Physical Memory: 52.078 MB
Virtual Memory: Max Size:  75.156 MB
Virtual Memory: Available: 52.887 MB
Virtual Memory: In Use:    22.269 MB
Page File Location(s):     C:\pagefile.sys
Domain:                    WORKGROUP
Logon Server:              \\DESKTOP

I'm using Python 3.8.5 and the newest version of pip.
I'm doing this within Bash for Windows, I really just want to know what "DGEBAL" and "DGEHRD" etc. means.


title: How to solve the ValueError importing torch in python

body:
After installing pytorch package, I have tried to import pytorch with
import torch

but got the error
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\torch\__init__.py", line 190, in <module>
    from torch._C import *
ValueError: module functions cannot set METH_CLASS or METH_STATIC

from searching, I have read that this could be about numpy. So I have tried uninstalling and installing with those
pip uninstall numpy
pip install numpy -I
pip install numpy -U

It didn't work.
I would be happy to hear from you if you have any idea why such an error occurs and how to correct it?
in case needed: Python version 3.8.6 ,
torch version 1.7.0 ,
numpy version 1.19.4


title: Pytorch Pre-trained RESNET18 Model

body: I have trained a pre-trained RESNET18 model in pytorch and saved it. While testing the model is giving different accuracy for different mini-batch size. Does anyone know why?


title: How to interpret the gradient norms in convolution neural networks?

body:
It might be basic thing, but I am unable to understand the interpretation of computing the gradient norms.

For example, I am working on binary segmentation using architecture by stacking multiple U-Nets with each net has encoder similar to Resnet34. I trained this network and I obtain the gradient mean and norm values as below:

  conv1 has mean grad of -1.77767194275e-14

  conv1 has norm grad of 2.57230658463e-05

  layer1 has mean grad of 1.27381299952e-11

  layer1 has norm grad of 0.000390226632589

  layer2 has mean grad of -2.3107595698e-11

  layer2 has norm grad of 0.000265486567514

All the values seems to be very low. I am not able to get shall I consider them as vanishing gradient problem or gradients can be really low depending upon the problem.

If the gradients can be really low, what is best way to check for vanishing gradient problem?

Any advice is appreciated to analyze the gradients data. Thanks in advance!


title: Multiprocessing code works using numpy but deadlocked using pytorch

body:
I'm hitting what appears to be a deadlock when trying to make use of multiprocessing with pytorch. The equivalent numpy code works like I expect it to.

I've made a simplified version of my code: a pool of 4 workers executing an array-wide broadcast operation 1000 times (so ~250 each worker). The array in question is 100,000 x 3 and the broadcast operation is subtraction of all rows by a single 1 x 3 row array. The large array is a shared/global array, and the row array is different at each iteration.

The code works exactly as expected using numpy, with the pooled workers showing a 4x speedup over the equivalent for loop.

The code in pytorch, however, hits a deadlock (I assume): none of the workers complete the array broadcast operation even once.

The numpy code below prints the following:

  Finished for loop over my_subtractor: took 8.1504 seconds.
  Finished pool over my_subtractor: took 2.2247 seconds.

The pytorch code, on the other hand, prints this then stalls:

  Finished for loop over my_subtractor: took 3.1082 seconds.
  BLA
  BLA
  BLA
  BLA

"BLA" print statements are just to show that each worker is stuck in -- apparently -- a deadlock state. There are exactly 4 of these: one per worker entering -- and getting stuck in -- an iteration.

If you feel ambitious enough to reproduce, note that it doesn't work on Windows because it's not wrapped around if __name__ == '__main__': (I read somewhere that you need this because of the way Windows handles launching processes). Also you will need to create an empty file called my_globals.py.

Here is the numpy code

from time import time
import numpy as np
import my_globals
from multiprocessing import Pool as ThreadPool

# shared memory by virtue of being global
my_globals.minuend = np.random.rand(100000,3)

# array to be iterated over in for loop / pool of workers
subtrahends = np.random.rand(10000,3)

# function called at each iteration (broadcast operation)
def my_subtractor(subtrahend):
    my_globals.minuend - subtrahend
    return 0

# launch for loop
ts = time()
for idx, subtrahend in enumerate(subtrahends):
    my_subtractor(subtrahend)
te = time()
print('Finished for loop over my_subtractor: took %2.4f seconds.' % (te - ts))

# launch equivalent pool of workers
ts = time()
pool = ThreadPool(4)
pool.map(my_subtractor, subtrahends)
pool.close()
pool.join()
te = time()
print('Finished pool over my_subtractor: took %2.4f seconds.' % (te - ts))

Here is the equivalent pytorch code:

from time import time
import torch
import my_globals

from torch.multiprocessing import Pool as ThreadPool

# necessary on my system because it has low limits for number of file descriptors; not recommended for most systems,
# see: https://pytorch.org/docs/stable/multiprocessing.html#file-descriptor-file-descriptor
torch.multiprocessing.set_sharing_strategy('file_system')

# shared memory by virtue of being global
my_globals.minuend = torch.rand(100000,3)

# array to be iterated over in for loop / pool of workers
subtrahends = torch.rand(10000,3)

# function called at each iteration (broadcast operation)
def my_subtractor(subtrahend, verbose=True):
    if verbose:
        print("BLA") # -- prints for every worker in the pool (so 4 times total)
    my_globals.minuend - subtrahend
    if verbose:
        print("ALB") # -- doesn't print for any worker
    return 0

# launch for loop
ts = time()
for idx, subtrahend in enumerate(subtrahends):
    my_subtractor(subtrahend, verbose=False)
te = time()
print('Finished for loop over my_subtractor: took %2.4f seconds.' % (te - ts))

# launch equivalent pool of workers
ts = time()
pool = ThreadPool(4)
pool.map(my_subtractor, subtrahends)
pool.close()
pool.join()
te = time()
print('Finished pool over my_subtractor: took %2.4f seconds.' % (te - ts))


title: Change dimensions of image when creating custom dataloader in pytorch

body:
I was training my CNN on the CIFAR10 dataset. I extracted the 50,000 images of dimensions (32 x 32 x 3) and read them in a list. I converted them into numpy arrays and stored them in a list.  I did the same with the labels for my training and testing.

I then constructed my CNN of two layers and a single FC in pytorch. Before doing this I created my own custom data loader. While doing so, the dimensions of the image that I am feeding in it are changing. The dimensions (32 x 32 x 3) are changing to (3 x 32 x 32) and I am not being able to train my neural network.

tensor_x = torch.stack([torch.Tensor(i) for i in train_images])
tensor_y = torch.stack([torch.Tensor(i) for i in train_labels])
dataset = data_utils.TensorDataset(tensor_x , tensor_y)
train_dataloader = data_utils.DataLoader(dataset=dataset)

tensor_x = torch.stack([torch.Tensor(i) for i in test_images])
tensor_y = torch.stack([torch.Tensor(i) for i in test_labels])
dataset = data_utils.TensorDataset(tensor_x , tensor_y)
test_dataloader = data_utils.DataLoader(dataset=dataset)

RuntimeError: Given groups=1, weight[64, 3, 3, 3], so expected input[1, 32, 32, 3] to have 3 channels, but got 32 channels instead


title: Multiprocessing failed with single GPU

body:
I am trying to parallelly do inference using multi-core CPU and single GPU, however I just got following runtime errors. I have attached a simplified example which can reproduce the errors.

Runtime Error

  THCudaCheck FAIL file=c:\a\w\1\s\tmp_conda_3.6_091443\conda\conda-bld\pytorch_1544087948354\work\torch\csrc\generic\StorageSharing.cpp line=232 error=71 : operation not supported
      File "C:\Users\Anaconda3\lib\site-packages\torch\multiprocessing\reductions.py", line 213, in reduce_tensor
      (device, handle, storage_size_bytes, storage_offset_bytes) = storage._share_cuda_()
      RuntimeError: cuda runtime error (71) : operation not supported at c:\a\w\1\s\tmp_conda_3.6_091443\conda\conda-bld\pytorch_1544087948354\work\torch\csrc\generic\StorageSharing.cpp:232

A simplified example which can reproduce the errors

import torch
from torch import nn

# model used to do inference
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(100,1)

    def forward(self,x):
        return self.fc1(x)

# class running inference
class A(object):
    def __init__(self):
        pass

    def do_something(self, model):
        # do something
        x = torch.randn(100).view(-1)
        print(model.forward(x))

    def run(self):
        mp = torch.multiprocessing.get_context('spawn')
        processes = []

        for i in range(2):
            p = mp.Process(target=self.do_something, args=(Model().cuda(),))
            processes.append(p)

        for p in processes:
            p.start()

if __name__ == '__main__':
    a = A()
    a.run()

It would be greatly appreciated if anyone can help solve this problem. By the way, my PC runs on Windows 10 with one GTX 1070 GPU.


title: huggingface distillbert classification using multiprocessing

body:
I am trying to use torch multiprocessing to parallelize the predictions from two separate huggingface distillbert classification models. It seems to be deadlocked at the prediction step.  I am using python 3.6.5, torch 1.5.0 and huggingface transformers version 2.11.0.
The output from running the code is

Tree enc done
Begin tree prediction<------(Comment: Both begin tree
End tree predictions<-------  and end tree predictions)
0.03125429153442383
Dn prediction
Dn enc done
Begin dn predictions<------(Comment: Both begin dn
End dn predictions<-------  and end dn predictions)
0.029727697372436523
----------Done sequential predictions-------------

--------Start Parallel predictions--------------
Tree prediction
Tree enc done
Begin tree prediction. <------(Comment: Process is deadlocked after this)
Dn prediction
Dn enc done
Begin dn predictions. <-------(Comment: Process is deadlocked after this)

During parallel predictions it seems to be deadlocking and not printing out "End tree predictions" and "End dn predictions". Not sure why this is happening.
The code is

import torch
import torch.multiprocessing as mp
import time
import transformers
from transformers import  DistilBertForSequenceClassification

# Load the BERT tokenizer.
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)

tree_model = DistilBertForSequenceClassification.from_pretrained(
    "distilbert-base-uncased",
    num_labels = 2,
    output_attentions = False,
    output_hidden_states = False
)
tree_model.eval()

dn_model = DistilBertForSequenceClassification.from_pretrained(
    "distilbert-base-uncased",
    num_labels = 2,
    output_attentions = False,
    output_hidden_states = False,
)
dn_model.eval()

tree_model.share_memory()
dn_model.share_memory()

def predict(sentences =[], tokenizer=tokenizer,models=(tree_model,dn_model,None)):
  MAX_SENTENCE_LENGTH = 16
  start = time.time()
  input_ids = []
  attention_masks = []
  predictions = []

  tree_model = models[0]
  dn_model = models[1]

  if models[0]:
      print("Tree prediction")
  if models[1]:
    print("Dn prediction")
  for sent in sentences:
    encoded_dict = tokenizer.encode_plus(
                        sent,
                        add_special_tokens = True,
                        max_length = MAX_SENTENCE_LENGTH,
                        pad_to_max_length = True,
                        return_attention_mask = True,
                        return_tensors = 'pt',
                   )

    # Add the encoded sentence to the list.
    input_ids.append(encoded_dict['input_ids'])

    # And its attention mask (simply differentiates padding from non-padding).
    attention_masks.append(encoded_dict['attention_mask'])

  if tree_model:
      print("Tree enc done")
  if dn_model:
    print("Dn enc done")

  # Convert the lists into tensors.
  new_input_ids = torch.cat(input_ids, dim=0)
  new_attention_masks = torch.cat(attention_masks, dim=0)

  with torch.no_grad():
      # Forward pass, calculate logit predictions
    if tree_model:
      print("Begin tree prediction")
      outputs = tree_model(new_input_ids,
                      attention_mask=new_attention_masks)
      print("End tree predictions")
    else:
      print("Begin dn predictions")
      outputs = dn_model(new_input_ids,
                      attention_mask=new_attention_masks)
      print("End dn predictions")
  logits = outputs[0]
  logits = logits.detach().cpu()
  print(time.time()-start)
  predictions = logits
  return predictions

def get_tree_prediction(sentence, tokenizer=tokenizer,models=(tree_model,dn_model, None)):
    return predict(sentences =[sentence], tokenizer=tokenizer,models=models)

def get_dn_prediction(sentence, tokenizer=tokenizer,models=(tree_model,dn_model, None)):
  return predict(sentences =[sentence], tokenizer=tokenizer,models=models)

if __name__ == '__main__':
    sentence = "hello world"
    processes = []
    get_tree_prediction(sentence, tokenizer, (tree_model,None,None))
    get_dn_prediction(sentence, tokenizer, (None,dn_model,None))
    print("----------Done sequential predictions-------------")

    print('\n--------Start Parallel predictions--------------')
    tr_p = mp.Process(target=get_tree_prediction, args=(sentence, tokenizer,
                                                         (tree_model,None,None)))

    tr_p.start()
    processes.append(tr_p)

    dn_p = mp.Process(target=get_dn_prediction, args=(sentence, tokenizer,
                                                       (None,dn_model,None)))
    dn_p.start()
    processes.append(dn_p)

    for p in processes:
        p.join()


title: How to correctly use CTC Loss with GRU in pytorch?

body:
I am trying to create ASR and I am still learning so, I am just trying with a simple GRU:

MySpeechRecognition(
  (gru): GRU(128, 128, num_layers=5, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.3, inplace=False)
  (fc1): Linear(in_features=128, out_features=512, bias=True)
  (fc2): Linear(in_features=512, out_features=28, bias=True)
)

Classifies each output as one of the possible alphabets + space + blank.

Then I use CTC Loss Function and Adam optimizer:

lr = 5e-4
criterion = nn.CTCLoss(blank=28, zero_infinity=False)
optimizer = torch.optim.Adam(net.parameters(), lr=lr)

In my training loop (I am only showing the problematic area):

output, h = mynet(specs, h)
print(output.size())
output = F.log_softmax(output, dim=2)
output = output.transpose(0,1)
# calculate the loss and perform backprop
loss = criterion(output, labels, input_lengths, label_lengths)
loss.backward()

I get this error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-133-5e47e7b03a46> in <module>
     42         output = output.transpose(0,1)
     43         # calculate the loss and perform backprop
---> 44         loss = criterion(output, labels, input_lengths, label_lengths)
     45         loss.backward()
     46         # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, log_probs, targets, input_lengths, target_lengths)
   1309     def forward(self, log_probs, targets, input_lengths, target_lengths):
   1310         return F.ctc_loss(log_probs, targets, input_lengths, target_lengths, self.blank, self.reduction,
-> 1311                           self.zero_infinity)
   1312
   1313 # TODO: L1HingeEmbeddingCriterion

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in ctc_loss(log_probs, targets, input_lengths, target_lengths, blank, reduction, zero_infinity)
   2050     """
   2051     return torch.ctc_loss(log_probs, targets, input_lengths, target_lengths, blank, _Reduction.get_enum(reduction),
-> 2052                           zero_infinity)
   2053
   2054

RuntimeError: blank must be in label range

I am not sure why I am getting this error. I tried changing to

labels.float()

Thanks.


title: How to compensate if I cant do a large batch size in neural network

body:
I am trying to run an action recognition code from GitHub. The original code used a batch size of 128 with 4 GPUS. I only have two gpus so I cannot match their bacth size number. Is there anyway I can compensate this difference in batch. I saw somewhere that iter_size might compensate according to a formula effective_batchsize= batch_size*iter_size*n_gpu. what is iter_size in this formula?
I am using PYthorch not Caffe.


title: Pytorch. Can autograd be used when the final tensor has more than a single value in it?

body:
Can autograd be used when the final tensor has more than a single value in it?

I tried the following.

x = torch.tensor([4.0, 5.0], requires_grad=True)
y = x ** 2

print(y)

y.backward()

Throws an error

RuntimeError: grad can be implicitly created only for scalar outputs

The following however works.

x = torch.tensor([4.0, 5.0], requires_grad=True)
y = x ** 2
y = torch.sum(y)
print(y)

y.backward()
print(x.grad)

The output is as

tensor(41., grad_fn=<SumBackward0>)
tensor([ 8., 10.])

Am I missing something here or can I proceed with the assumption that autograd only works when the final tensor has a single value in it?


title: Classification with pretrained pytorch vgg16 model and its classes

body:
I wrote a image vgg classification model with pytorch's pretrained vgg16 model.
import matplotlib.pyplot as plt
import numpy as np
import torch
from PIL import Image
import urllib
from skimage.transform import resize
from skimage import io
import yaml

# Downloading imagenet 1000 classes list
file = urllib. request. urlopen("https://gist.githubusercontent.com/yrevar/942d3a0ac09ec9e5eb3a/raw/238f720ff059c1f82f368259d1ca4ffa5dd8f9f5/imagenet1000_clsidx_to_labels.txt")
classes = ''
for f in file:
  classes = classes +  f.decode("utf-8")
classes = yaml.load(classes)

# Downloading pretrained vgg16 model
model = torch.hub.load('pytorch/vision:v0.6.0', 'vgg16', pretrained=True)

print(model)

for param in model.parameters():
    param.requires_grad = False

url, filename = ("https://raw.githubusercontent.com/pytorch/hub/master/dog.jpg", "dog.jpg")

image=io.imread(url)

plt.imshow(image)
plt.show()

# resize to 224x224x3
img = resize(image,(224,224,3))

plt.imshow(img)
plt.show()
# Normalizing input for vgg16
mean = [0.485, 0.456, 0.406]
std = [0.229, 0.224, 0.225]
img1 = mean*img+std
img1 = np.clip(img1,0,1)

img1 = torch.from_numpy(img1).unsqueeze(0)
img1 = img1.permute(0,3,2,1) # batch_size x channels x height x width

model.eval()
pred = model(img1.float())
print(classes[torch.argmax(pred).numpy().tolist()])

The code works fine but its outputting wrong classes. I am not sure where I did wrong but If I have to guess it might be the imagenet yaml classes list or at the normalizing input image. Can anyone tell me where I am making the mistakes?


title: How to get only specific classes from PyTorch's FashionMNIST dataset?

body:
The FashionMNIST dataset has 10 different output classes. How can I get a subset of this dataset with only specific classes? In my case, I only want images of sneaker, pullover, sandal and shirt classes (their classes are 7,2,5 and 6 respectively).
This is how I load my dataset.
train_dataset_full = torchvision.datasets.FashionMNIST(data_folder, train = True, download = True, transform = transforms.ToTensor())
The approach I’ve followed is below.
Iterate through the dataset, one by one, then compare the 1st element (i.e. class) in the returned tuple to my required class. I’m stuck here. If the value returned is true, how can I append/add this observation to an empty dataset?
sneaker = 0
pullover = 0
sandal = 0
shirt = 0
for i in range(60000):
    if train_dataset_full[i][1] == 7:
        sneaker += 1
    elif train_dataset_full[i][1] == 2:
        pullover += 1
    elif train_dataset_full[i][1] == 5:
        sandal += 1
    elif train_dataset_full[i][1] == 6:
        shirt += 1

Now, in place of sneaker += 1, pullover += 1, sandal += 1 and shirt += 1 I want to do something like this empty_dataset.append(train_dataset_full[i]) or something similar.
If the above approach is incorrect, please suggest another method.


title: How to make torchaudio and librosa MFCC calculations equivalent?

body:
I've seen this question concerning the same type of issue between librosa, python_speech_features and tensorflow.signal.
I am trying to make torchaudio and librosa compute MFCC features with the same arguments and underlying methods. This is part of a transition from librosa to torchaudio.
Given:
import numpy as np
import torch

from librosa.feature import mfcc
from torchaudio.transforms import MFCC

sample_rate = 22050
audio = np.ones((sample_rate,), dtype=np.float32)

librosa_mfcc = mfcc(audio, sr=sr, n_mfcc=20, n_fft=2048, hop_length=512, power=2)

mfcc_module = MFCC(sample_rate=sr, n_mfcc=20, melkwargs={"n_fft": 2048, "hop_length": 512, "power": 2})
torch_mfcc = mfcc_module(torch.tensor(audio))

The shapes of librosa_mfcc and torch_mfcc are both (20, 44), but the arrays themselves are different. For example, librosa_mfcc[0][0] is -487.6101, while torch_mfcc[0][0] is -302.7711.
I admit I am lacking a good amount of domain knowledge here, but am working through the librosa and torchaudio documentation and parameters to learn the different routes they take in MFCC calculation as well as the meaning behind each parameter. How do I make torch_mfcc have the same values as librosa_mfcc?